ACK EC2 Controller VPCEndpoint doesn't modify the aws object based on spec definition #2296

stefanescualexandrumihai · 2025-02-12T14:19:19Z

AWS ACK EC2 Controller version : 1.3.1
The ec2 controller has featureGates.ResourceAdoption value set to true.
Kubernetes version v1.29.6

After the object adoption using the following manifest:

apiVersion: ec2.services.k8s.aws/v1alpha1
kind: VPCEndpoint
metadata:
  name: astef8-sbx-va6-test
  namespace: sbx-clusters
  annotations:
    services.k8s.aws/region: us-east-1
    services.k8s.aws/adoption-fields: |
      {
        "vpcEndpointID": "vpce-xxxx"
      }
    services.k8s.aws/adoption-policy: adopt

The spec was populated with this:

spec:
  dnsOptions:
    dnsRecordIPType: ipv4
  ipAddressType: ipv4
  policyDocument: "{\n  \"Statement\": [\n    {\n      \"Action\": \"*\", \n      \"Effect\":
    \"Allow\", \n      \"Principal\": \"*\", \n      \"Resource\": \"*\"\n    }\n
    \ ]\n}"
  privateDNSEnabled: true
  serviceName: com.amazonaws.us-east-1.secretsmanager
  subnetIDs:
  - subnet-xxxx1
  - subnet-xxxx2
  - subnet-xxxx3
  tags:
  - key: services.k8s.aws/controller-version
    value: ec2-1.3.1
  - key: Name
    value: astef8-sbx-va6-test
  - key: services.k8s.aws/namespace
    value: sbx-clusters
  vpcEndpointType: Interface
  vpcID: vpc-xxxx

I did the same thing for a vpcendpoint of gateway type.

spec:
  policyDocument: '{"Version":"2008-10-17","Statement":[{"Effect":"Allow","Principal":"*","Action":"*","Resource":"*"}]}'
  privateDNSEnabled: false
  routeTableIDs:
  - rtb-xxxx1
  - rtb-xxxx2
  - rtb-xxxx3
  - rtb-xxxx4
  serviceName: com.amazonaws.us-east-1.s3
  tags:
  - key: Name
    value: astef8-sbx-va6-s3
  - key: services.k8s.aws/controller-version
    value: ec2-1.3.1
  - key: services.k8s.aws/namespace
    value: sbx-clusters
  vpcEndpointType: Gateway
  vpcID: vpc-xxxx

I removed the annotations: services.k8s.aws/adoption-fields and services.k8s.aws/adoption-policy.
Afterwards, I tried to modify the subnetIDs by deleting one of them. The controller logged the following messages: desired resource state has changed and updated resource . However, in the aws console, the modification was not reflected. The same behaviour for a vpcendpoint of gateway type ( trying to delete one of the routetableIDs entries).

Logs for the gateway type:
kubectl logs -n ack-ec2-controller ack-ec2-controller-5bc4889db9-m64m6 | grep astef8-sbx-va6-s3

{"level":"info","ts":"2025-02-12T13:54:22.931Z","logger":"ackrt","msg":"desired resource state has changed","kind":"VPCEndpoint","namespace":"sbx-clusters","name":"astef8-sbx-va6-s3","account":"xxxx","role":"arn:aws:iam::xxxx:role/ack-ec2-assumedrole","region":"us-east-1","is_adopted":true,"generation":3,"diff":[{"Path":{"Parts":["Spec","RouteTableIDs"]},"A":["rtb-xxxx2","rtb-xxxx3","rtb-xxx4"],"B":["rtb-xxxx1","rtb-xxxx2","rtb-xxxx3","rtb-xxxx4"]}]}
{"level":"info","ts":"2025-02-12T13:54:22.931Z","logger":"ackrt","msg":"updated resource","kind":"VPCEndpoint","namespace":"sbx-clusters","name":"astef8-sbx-va6-s3","account":"xxxx","role":"arn:aws:iam::xxxx:role/ack-ec2-assumedrole","region":"us-east-1","is_adopted":true,"generation":3}

I also tried to delete a routetableid/subnetid from aws console that was present in the object's spec, but the controller didn't add it back ( checked the aws console after a while )

The text was updated successfully, but these errors were encountered:

rushmash91 · 2025-02-12T19:00:49Z

Hi @stefanescualexandrumihai,

Thank you for reporting this issue! To help investigate further, could you please provide a few details:

How long did you wait after making changes in the AWS console before checking if the controller had reconciled the state? It would be helpful to know the specific timeframe.
Could you share your periodicResync configuration for the EC2 controller?

stefanescualexandrumihai · 2025-02-13T09:47:23Z

Reconcile default resync seconds: 600
I waited for more than 10 minutes. After modifying the resources yesterday, I checked them again today and noticed a discrepancy in the AWS console: there are fewer route table IDs (2) compared to what is specified in the AWS ack VPCEndpoint spec (3). The AWS ACK EC2 controller logs messages every 10 minutes:

{"level":"info","ts":"2025-02-13T09:37:29.541Z","logger":"ackrt","msg":"desired resource state has changed","kind":"VPCEndpoint","namespace":"sbx-clusters","name":"astef8-sbx-va6-s3","account":"xxxx","role":"arn:aws:iam::xxxx:role/ack-ec2-assumedrole","region":"us-east-1","is_adopted":true,"generation":3,"diff":[{"Path":{"Parts":["Spec","RouteTableIDs"]},"A":["rtb-0140dfb8fc56f832f","rtb-0665e002ef4ac4e70","rtb-0b6d62b5667f017c5"],"B":["rtb-09ff09131b4a5fe21","rtb-0140dfb8fc56f832f"]}]}
{"level":"info","ts":"2025-02-13T09:37:29.542Z","logger":"ackrt","msg":"updated resource","kind":"VPCEndpoint","namespace":"sbx-clusters","name":"astef8-sbx-va6-s3","account":"xxxx","role":"arn:aws:iam::xxxx:role/ack-ec2-assumedrole","region":"us-east-1","is_adopted":true,"generation":3}

Same thing happened for the interface one.

I retested today

create a vpcendpoint of gateway type (s3) using the aws console.
adopt it using the adoption annotation

spec:
  policyDocument: '{"Version":"2008-10-17","Statement":[{"Effect":"Allow","Principal":"*","Action":"*","Resource":"*"}]}'
  privateDNSEnabled: false
  routeTableIDs:
  - rtb-09ff09131b4a5fe21
  - rtb-0140dfb8fc56f832f
  - rtb-0665e002ef4ac4e70
  - rtb-01f7596b655e26b67
  - rtb-0b6d62b5667f017c5
  serviceName: com.amazonaws.us-east-1.s3
  tags:
  - key: services.k8s.aws/namespace
    value: sbx-clusters
  - key: Name
    value: astef8-sbx-va6-s3-gateway
  - key: services.k8s.aws/controller-version
    value: ec2-1.3.1
  vpcEndpointType: Gateway
  vpcID: vpc-xxxx

check the aws ack ec2 logs

{"level":"info","ts":"2025-02-13T10:05:55.529Z","logger":"ackrt","msg":"Adopting Resource","kind":"VPCEndpoint","namespace":"sbx-clusters","name":"astef8-sbx-va6-s3-gateway","account":"xxxx","role":"arn:aws:iam::xxxx:role/ack-ec2-assumedrole","region":"us-east-1","is_adopted":false,"generation":1}
{"level":"info","ts":"2025-02-13T10:05:57.008Z","logger":"ackrt","msg":"Resource Adopted","kind":"VPCEndpoint","namespace":"sbx-clusters","name":"astef8-sbx-va6-s3-gateway","account":"xxxx","role":"arn:aws:iam::xxxx:role/ack-ec2-assumedrole","region":"us-east-1","is_adopted":false,"is_adopted":"true","generation":1}
{"level":"info","ts":"2025-02-13T10:05:57.127Z","logger":"ackrt","msg":"desired resource state has changed","kind":"VPCEndpoint","namespace":"sbx-clusters","name":"astef8-sbx-va6-s3-gateway","account":"xxxx","role":"arn:aws:iam::xxxx:role/ack-ec2-assumedrole","region":"us-east-1","is_adopted":true,"generation":2,"diff":[{"Path":{"Parts":["Spec","Tags"]},"A":[{"key":"services.k8s.aws/controller-version","value":"ec2-1.3.1"},{"key":"services.k8s.aws/namespace","value":"sbx-clusters"},{"key":"Name","value":"astef8-sbx-va6-s3-gateway"}],"B":[{"key":"Name","value":"astef8-sbx-va6-s3-gateway"}]}]}
{"level":"info","ts":"2025-02-13T10:05:57.420Z","logger":"ackrt","msg":"updated resource","kind":"VPCEndpoint","namespace":"sbx-clusters","name":"astef8-sbx-va6-s3-gateway","account":"xxxx","role":"arn:aws:iam::xxxx:role/ack-ec2-assumedrole","region":"us-east-1","is_adopted":true,"generation":2}

delete those annotations
delete the first three route tables
check the logs ( those 4 from above plus 2 more )

{"level":"info","ts":"2025-02-13T10:11:30.807Z","logger":"ackrt","msg":"desired resource state has changed","kind":"VPCEndpoint","namespace":"sbx-clusters","name":"astef8-sbx-va6-s3-gateway","account":"xxxx","role":"arn:aws:iam::xxxx:role/ack-ec2-assumedrole","region":"us-east-1","is_adopted":true,"generation":3,"diff":[{"Path":{"Parts":["Spec","RouteTableIDs"]},"A":["rtb-01f7596b655e26b67","rtb-0b6d62b5667f017c5"],"B":["rtb-09ff09131b4a5fe21","rtb-0140dfb8fc56f832f","rtb-0665e002ef4ac4e70","rtb-01f7596b655e26b67","rtb-0b6d62b5667f017c5"]}]}
{"level":"info","ts":"2025-02-13T10:11:30.807Z","logger":"ackrt","msg":"updated resource","kind":"VPCEndpoint","namespace":"sbx-clusters","name":"astef8-sbx-va6-s3-gateway","account":"xxxx","role":"arn:aws:iam::xxxx:role/ack-ec2-assumedrole","region":"us-east-1","is_adopted":true,"generation":3}

check the aws console ( nothing changed )
a few more logs appear in the ack ec2 controller after tens of minutes

{"level":"info","ts":"2025-02-13T10:15:59.893Z","logger":"ackrt","msg":"desired resource state has changed","kind":"VPCEndpoint","namespace":"sbx-clusters","name":"astef8-sbx-va6-s3-gateway","account":"xxxx","role":"arn:aws:iam::xxxx:role/ack-ec2-assumedrole","region":"us-east-1","is_adopted":true,"generation":3,"diff":[{"Path":{"Parts":["Spec","RouteTableIDs"]},"A":["rtb-01f7596b655e26b67","rtb-0b6d62b5667f017c5"],"B":["rtb-09ff09131b4a5fe21","rtb-0140dfb8fc56f832f","rtb-0665e002ef4ac4e70","rtb-01f7596b655e26b67","rtb-0b6d62b5667f017c5"]}]}
{"level":"info","ts":"2025-02-13T10:15:59.893Z","logger":"ackrt","msg":"updated resource","kind":"VPCEndpoint","namespace":"sbx-clusters","name":"astef8-sbx-va6-s3-gateway","account":"xxxx","role":"arn:aws:iam::xxxx:role/ack-ec2-assumedrole","region":"us-east-1","is_adopted":true,"generation":3}
{"level":"info","ts":"2025-02-13T10:26:01.453Z","logger":"ackrt","msg":"desired resource state has changed","kind":"VPCEndpoint","namespace":"sbx-clusters","name":"astef8-sbx-va6-s3-gateway","account":"xxxx","role":"arn:aws:iam::xxxx:role/ack-ec2-assumedrole","region":"us-east-1","is_adopted":true,"generation":3,"diff":[{"Path":{"Parts":["Spec","RouteTableIDs"]},"A":["rtb-01f7596b655e26b67","rtb-0b6d62b5667f017c5"],"B":["rtb-09ff09131b4a5fe21","rtb-0140dfb8fc56f832f","rtb-0665e002ef4ac4e70","rtb-01f7596b655e26b67","rtb-0b6d62b5667f017c5"]}]}
{"level":"info","ts":"2025-02-13T10:26:01.453Z","logger":"ackrt","msg":"updated resource","kind":"VPCEndpoint","namespace":"sbx-clusters","name":"astef8-sbx-va6-s3-gateway","account":"xxxx","role":"arn:aws:iam::xxxx:role/ack-ec2-assumedrole","region":"us-east-1","is_adopted":true,"generation":3}
{"level":"info","ts":"2025-02-13T10:36:03.138Z","logger":"ackrt","msg":"desired resource state has changed","kind":"VPCEndpoint","namespace":"sbx-clusters","name":"astef8-sbx-va6-s3-gateway","account":"xxxx","role":"arn:aws:iam::xxxx:role/ec2-assumedrole","region":"us-east-1","is_adopted":true,"generation":3,"diff":[{"Path":{"Parts":["Spec","RouteTableIDs"]},"A":["rtb-01f7596b655e26b67","rtb-0b6d62b5667f017c5"],"B":["rtb-09ff09131b4a5fe21","rtb-0140dfb8fc56f832f","rtb-0665e002ef4ac4e70","rtb-01f7596b655e26b67","rtb-0b6d62b5667f017c5"]}]}
{"level":"info","ts":"2025-02-13T10:36:03.138Z","logger":"ackrt","msg":"updated resource","kind":"VPCEndpoint","namespace":"sbx-clusters","name":"astef8-sbx-va6-s3-gateway","account":"258057316678","role":"arn:aws:iam::xxxx:role/ack-ec2-assumedrole","region":"us-east-1","is_adopted":true,"generation":3}

nothing changed in aws console

stefanescualexandrumihai · 2025-02-13T10:50:06Z

status:
  ackResourceMetadata:
    ownerAccountID: "xxxx"
    region: us-east-1
  conditions:
  - lastTransitionTime: "2025-02-13T10:46:09Z"
    message: Late initialization successful
    reason: Late initialization successful
    status: "True"
    type: ACK.LateInitialized
  - lastTransitionTime: "2025-02-13T10:46:09Z"
    message: Resource synced successfully
    reason: ""
    status: "True"
    type: ACK.ResourceSynced
  creationTimestamp: "2025-02-13T10:05:10Z"
  ownerID: "xxxx"
  requesterManaged: false
  state: available
  vpcEndpointID: vpce-xxxx

rushmash91 · 2025-02-13T17:04:55Z

Thank you for the details! I see that the controller will need custom update to handle the ModifyVpcEndpoint. I will bring this to the teams attention and get back to you.

rushmash91 · 2025-02-14T22:45:35Z

Hi @stefanescualexandrumihai, just updated the method for VPCEndpoint resources that triggers ModifyVpcEndpoint whenever mutable fields such as subnet IDs, route table IDs, and other properties change.

Next release for the controller will address the issue.
Thank you!

…yVpcEndpoint` calls (#240) fixes aws-controllers-k8s/community#2296 Description of changes: Handle updates that require the `ModifyVpcEndpoint` API. Ensure that when fields such as `SubnetIDs`, `RouteTableIDs`, `PolicyDocument`, `PrivateDNSEnabled`, `SecurityGroupIDs`, `DNSOptions`, or `IPAddressType` change, the controller constructs and sends the corresponding Modify calls to AWS. By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

rushmash91 · 2025-02-18T05:07:48Z

Hi @stefanescualexandrumihai ,

The EC2 controller has released version v1.3.5 which includes the fix for this issue.

stefanescualexandrumihai · 2025-02-18T12:54:34Z

Thx so much!

rushmash91 self-assigned this Feb 13, 2025

rushmash91 mentioned this issue Feb 14, 2025

Add custom update support for VPCEndpoint resource to handle ModifyVpcEndpoint calls aws-controllers-k8s/ec2-controller#240

Merged

ack-prow bot closed this as completed in aws-controllers-k8s/ec2-controller#240 Feb 17, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

ACK EC2 Controller VPCEndpoint doesn't modify the aws object based on spec definition #2296

ACK EC2 Controller VPCEndpoint doesn't modify the aws object based on spec definition #2296

stefanescualexandrumihai commented Feb 12, 2025 •

edited

Loading

rushmash91 commented Feb 12, 2025

Uh oh!

stefanescualexandrumihai commented Feb 13, 2025 •

edited

Loading

Uh oh!

stefanescualexandrumihai commented Feb 13, 2025

Uh oh!

rushmash91 commented Feb 13, 2025

Uh oh!

rushmash91 commented Feb 14, 2025

Uh oh!

rushmash91 commented Feb 18, 2025

Uh oh!

stefanescualexandrumihai commented Feb 18, 2025

Uh oh!

ACK EC2 Controller VPCEndpoint doesn't modify the aws object based on spec definition #2296

ACK EC2 Controller VPCEndpoint doesn't modify the aws object based on spec definition #2296

Comments

stefanescualexandrumihai commented Feb 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

rushmash91 commented Feb 12, 2025

Uh oh!

stefanescualexandrumihai commented Feb 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

stefanescualexandrumihai commented Feb 13, 2025

Uh oh!

rushmash91 commented Feb 13, 2025

Uh oh!

rushmash91 commented Feb 14, 2025

Uh oh!

rushmash91 commented Feb 18, 2025

Uh oh!

stefanescualexandrumihai commented Feb 18, 2025

Uh oh!

stefanescualexandrumihai commented Feb 12, 2025 •

edited

Loading

stefanescualexandrumihai commented Feb 13, 2025 •

edited

Loading