Skip to content

bug: unable to deploy version 1.22 on AWS eks; errors thrown by pulumi #137

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
qdzlug opened this issue May 3, 2022 · 3 comments · Fixed by #167
Closed

bug: unable to deploy version 1.22 on AWS eks; errors thrown by pulumi #137

qdzlug opened this issue May 3, 2022 · 3 comments · Fixed by #167
Assignees
@qdzlug

Comments

@qdzlug
Copy link
Contributor

qdzlug commented May 3, 2022

Describe the bug
When trying to run on the most recent version of EKS (1.22) the following errors are thrown:

16:02:26  
16:02:26  #=============================================================================#
16:02:26  #                _     __        __  ____      _____   _  __  ____            #
16:02:26  #               / \    \ \      / / / ___|    | ____| | |/ / / ___|           #
16:02:26  #              / _ \    \ \ /\ / /  \___ \    |  _|   | ' /  \___ \           #
16:02:26  #             / ___ \    \ V  V /    ___) |   | |___  | . \   ___) |          #
16:02:26  #            /_/   \_\    \_/\_/    |____/    |_____| |_|\_\ |____/           #
16:02:26  #                                                                             #
16:02:26  #=============================================================================#
16:02:26  
16:02:26  
16:02:27  Previewing update (marajenkaws3)
16:02:27  
16:02:27  View Live: https://app.pulumi.com/qdzlug/aws-eks/marajenkaws3/previews/9c2cbcf0-eac9-43af-b220-b98c9cb39f9e
16:02:27  
16:02:27  
16:02:28      pulumi:pulumi:Stack aws-eks-marajenkaws3  aws **** profile
16:02:28   +  pulumi:pulumi:Stack aws-eks-marajenkaws3 create aws **** profile
16:02:29   +  aws:iam:Role ec2-nodegroup-iam-role create 
16:02:29   +  aws:iam:Role eks-iam-role create 
16:02:29   +  pulumi:pulumi:Stack aws-eks-marajenkaws3 create read pulumi:pulumi:StackReference qdzlug/aws-vpc/marajenkaws3
16:02:29   +  aws:iam:RolePolicyAttachment eks-workernode-policy-attachment create 
16:02:29   +  aws:iam:RolePolicyAttachment ec2-container-ro-policy-attachment create 
16:02:29   +  aws:iam:RolePolicyAttachment eks-cni-policy-attachment create 
16:02:29   +  aws:iam:InstanceProfile node-group-profile-aws-eks-marajenkaws3 create 
16:02:29   +  aws:iam:RolePolicyAttachment eks-service-policy-attachment create 
16:02:29   +  aws:iam:RolePolicyAttachment eks-cluster-policy-attachment create 
16:02:29   +  pulumi:pulumi:Stack aws-eks-marajenkaws3 create read pulumi:pulumi:StackReference qdzlug/aws-vpc/marajenkaws3
16:02:29   +  pulumi:pulumi:Stack aws-eks-marajenkaws3 create vpc id: vpc-05315b2f4bfb9acce
16:02:31   +  pulumi:pulumi:Stack aws-eks-marajenkaws3 create public subnets: ['subnet-085c683366982fc83', 'subnet-09e52459a6598217d', 'subnet-0f1483979c35c3fe1', 'subnet-0a7c801352bda906c']
16:02:31   +  pulumi:pulumi:Stack aws-eks-marajenkaws3 create public subnets: ['subnet-027f5d25d0b5cdc03', 'subnet-038d95e76e9cdaa9b', 'subnet-0d056c799b48ee6c4', 'subnet-0e5d8659ed51f17ef']
16:02:33   +  eks:index:Cluster aws-eks-marajenkaws3 create 
16:02:35   +  eks:index:ServiceRole aws-eks-marajenkaws3-instanceRole create 
16:02:35   +  aws:ec2:SecurityGroup aws-eks-marajenkaws3-eksClusterSecurityGroup create 
16:02:35   +  eks:index:RandomSuffix aws-eks-marajenkaws3-cfnStackName create 
16:02:35   +  aws:iam:Role aws-eks-marajenkaws3-instanceRole-role create 
16:02:35   +  aws:eks:Cluster aws-eks-marajenkaws3-eksCluster create 
16:02:35   +  aws:ec2:SecurityGroupRule aws-eks-marajenkaws3-eksClusterInternetEgressRule create 
16:02:35   +  pulumi:providers:kubernetes aws-eks-marajenkaws3-eks-k8s create 
16:02:35   +  eks:index:VpcCni aws-eks-marajenkaws3-vpc-cni create 
16:02:35   +  aws:ec2:SecurityGroup aws-eks-marajenkaws3-nodeSecurityGroup create 
16:02:35   +  aws:iam:RolePolicyAttachment aws-eks-marajenkaws3-instanceRole-3eb088f2 create 
16:02:35   +  aws:iam:RolePolicyAttachment aws-eks-marajenkaws3-instanceRole-03516f97 create 
16:02:35   +  aws:iam:RolePolicyAttachment aws-eks-marajenkaws3-instanceRole-e1b295bd create 
16:02:35   +  aws:ec2:SecurityGroupRule aws-eks-marajenkaws3-eksExtApiServerClusterIngressRule create 
16:02:35   +  aws:ec2:SecurityGroupRule aws-eks-marajenkaws3-eksNodeIngressRule create 
16:02:35   +  aws:ec2:SecurityGroupRule aws-eks-marajenkaws3-eksNodeInternetEgressRule create 
16:02:35   +  aws:ec2:SecurityGroupRule aws-eks-marajenkaws3-eksClusterIngressRule create 
16:02:35   +  aws:ec2:SecurityGroupRule aws-eks-marajenkaws3-eksNodeClusterIngressRule create 
16:02:35   +  kubernetes:core/v1:ConfigMap aws-eks-marajenkaws3-nodeAccess create 
16:02:35   +  aws:iam:InstanceProfile aws-eks-marajenkaws3-instanceProfile create 
16:02:35   +  aws:ec2:LaunchConfiguration aws-eks-marajenkaws3-nodeLaunchConfiguration create 
16:02:35   +  aws:cloudformation:Stack aws-eks-marajenkaws3-nodes create 
16:02:35   +  pulumi:providers:kubernetes aws-eks-marajenkaws3-provider create 
16:02:36   +  pulumi:pulumi:Stack aws-eks-marajenkaws3 create 4 messages
16:02:36   
16:02:36  Diagnostics:
16:02:36    pulumi:pulumi:Stack (aws-eks-marajenkaws3):
16:02:36      aws **** profile
16:02:36      vpc id: vpc-05315b2f4bfb9acce
16:02:36      public subnets: ['subnet-085c683366982fc83', 'subnet-09e52459a6598217d', 'subnet-0f1483979c35c3fe1', 'subnet-0a7c801352bda906c']
16:02:36      public subnets: ['subnet-027f5d25d0b5cdc03', 'subnet-038d95e76e9cdaa9b', 'subnet-0d056c799b48ee6c4', 'subnet-0e5d8659ed51f17ef']
16:02:36   
16:02:36  
16:02:36  Updating (marajenkaws3)
16:02:36  
16:02:36  View Live: https://app.pulumi.com/qdzlug/aws-eks/marajenkaws3/updates/1
16:02:36  
16:02:37  
16:02:37      pulumi:pulumi:Stack aws-eks-marajenkaws3  aws **** profile
16:02:38   +  pulumi:pulumi:Stack aws-eks-marajenkaws3 creating aws **** profile
16:02:39   +  aws:iam:Role ec2-nodegroup-iam-role creating 
16:02:39   +  aws:iam:Role eks-iam-role creating 
16:02:39   +  pulumi:pulumi:Stack aws-eks-marajenkaws3 creating read pulumi:pulumi:StackReference qdzlug/aws-vpc/marajenkaws3
16:02:39   +  pulumi:pulumi:Stack aws-eks-marajenkaws3 creating read pulumi:pulumi:StackReference qdzlug/aws-vpc/marajenkaws3
16:02:40   +  pulumi:pulumi:Stack aws-eks-marajenkaws3 creating vpc id: vpc-05315b2f4bfb9acce
16:02:41   +  pulumi:pulumi:Stack aws-eks-marajenkaws3 creating public subnets: ['subnet-085c683366982fc83', 'subnet-09e52459a6598217d', 'subnet-0f1483979c35c3fe1', 'subnet-0a7c801352bda906c']
16:02:41   +  aws:iam:Role ec2-nodegroup-iam-role created 
16:02:41   +  pulumi:pulumi:Stack aws-eks-marajenkaws3 creating public subnets: ['subnet-027f5d25d0b5cdc03', 'subnet-038d95e76e9cdaa9b', 'subnet-0d056c799b48ee6c4', 'subnet-0e5d8659ed51f17ef']
16:02:41   +  aws:iam:RolePolicyAttachment eks-workernode-policy-attachment creating 
16:02:41   +  aws:iam:RolePolicyAttachment ec2-container-ro-policy-attachment creating 
16:02:41   +  aws:iam:RolePolicyAttachment eks-cni-policy-attachment creating 
16:02:41   +  aws:iam:InstanceProfile node-group-profile-aws-eks-marajenkaws3 creating 
16:02:41   +  aws:iam:Role eks-iam-role created 
16:02:41   +  aws:iam:RolePolicyAttachment eks-service-policy-attachment creating 
16:02:41   +  aws:iam:RolePolicyAttachment eks-cluster-policy-attachment creating 
16:02:41   +  aws:iam:RolePolicyAttachment eks-workernode-policy-attachment created 
16:02:42   +  aws:iam:RolePolicyAttachment ec2-container-ro-policy-attachment created 
16:02:42   +  aws:iam:RolePolicyAttachment eks-cni-policy-attachment created 
16:02:42   +  aws:iam:RolePolicyAttachment eks-service-policy-attachment created 
16:02:42   +  aws:iam:RolePolicyAttachment eks-cluster-policy-attachment created 
16:02:42   +  aws:iam:InstanceProfile node-group-profile-aws-eks-marajenkaws3 created 
16:02:44   +  eks:index:Cluster aws-eks-marajenkaws3 creating 
16:02:45   +  eks:index:ServiceRole aws-eks-marajenkaws3-instanceRole creating 
16:02:46   +  aws:ec2:SecurityGroup aws-eks-marajenkaws3-eksClusterSecurityGroup creating 
16:02:46   +  aws:iam:Role aws-eks-marajenkaws3-instanceRole-role creating 
16:02:46   +  eks:index:RandomSuffix aws-eks-marajenkaws3-cfnStackName creating 
16:02:46   +  eks:index:RandomSuffix aws-eks-marajenkaws3-cfnStackName created 
16:02:48   +  aws:iam:Role aws-eks-marajenkaws3-instanceRole-role created 
16:02:48   +  aws:iam:RolePolicyAttachment aws-eks-marajenkaws3-instanceRole-3eb088f2 creating 
16:02:48   +  aws:iam:RolePolicyAttachment aws-eks-marajenkaws3-instanceRole-e1b295bd creating 
16:02:48   +  aws:iam:RolePolicyAttachment aws-eks-marajenkaws3-instanceRole-03516f97 creating 
16:02:48   +  aws:ec2:SecurityGroup aws-eks-marajenkaws3-eksClusterSecurityGroup created 
16:02:48   +  aws:eks:Cluster aws-eks-marajenkaws3-eksCluster creating 
16:02:48   +  aws:ec2:SecurityGroupRule aws-eks-marajenkaws3-eksClusterInternetEgressRule creating 
16:02:49   +  aws:iam:RolePolicyAttachment aws-eks-marajenkaws3-instanceRole-3eb088f2 created 
16:02:49   +  aws:iam:RolePolicyAttachment aws-eks-marajenkaws3-instanceRole-e1b295bd created 
16:02:49   +  aws:iam:RolePolicyAttachment aws-eks-marajenkaws3-instanceRole-03516f97 created 
16:02:49   +  aws:iam:InstanceProfile aws-eks-marajenkaws3-instanceProfile creating 
16:02:49   +  aws:ec2:SecurityGroupRule aws-eks-marajenkaws3-eksClusterInternetEgressRule created 
16:02:50   +  aws:iam:InstanceProfile aws-eks-marajenkaws3-instanceProfile created 
16:12:18  @ Updating...............................
16:12:18   +  aws:eks:Cluster aws-eks-marajenkaws3-eksCluster created 
16:12:18   +  aws:ec2:SecurityGroup aws-eks-marajenkaws3-nodeSecurityGroup creating 
16:12:18   +  aws:eks:Cluster aws-eks-marajenkaws3-eksCluster created Cluster is ready
16:12:18   +  eks:index:VpcCni aws-eks-marajenkaws3-vpc-cni creating 
16:12:18   +  pulumi:providers:kubernetes aws-eks-marajenkaws3-eks-k8s creating 
16:12:18   +  pulumi:providers:kubernetes aws-eks-marajenkaws3-eks-k8s created 
16:12:18  @ Updating....
16:12:18   +  kubernetes:core/v1:ConfigMap aws-eks-marajenkaws3-nodeAccess creating 
16:12:18   +  kubernetes:core/v1:ConfigMap aws-eks-marajenkaws3-nodeAccess creating 
16:12:18   +  kubernetes:core/v1:ConfigMap aws-eks-marajenkaws3-nodeAccess created 
16:12:18   +  aws:ec2:SecurityGroup aws-eks-marajenkaws3-nodeSecurityGroup created 
16:12:18   +  eks:index:VpcCni aws-eks-marajenkaws3-vpc-cni creating error: Command failed: kubectl apply -f /tmp/tmp-23547dNCAQqk1wCDq.tmp
16:12:18   +  eks:index:VpcCni aws-eks-marajenkaws3-vpc-cni **creating failed** error: Command failed: kubectl apply -f /tmp/tmp-23547dNCAQqk1wCDq.tmp
16:12:18   +  pulumi:pulumi:Stack aws-eks-marajenkaws3 creating Warning: spec.template.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].key: beta.kubernetes.io/os is deprecated since v1.14; use "kubernetes.io/os" instead
16:12:18   +  pulumi:pulumi:Stack aws-eks-marajenkaws3 creating Warning: spec.template.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[1].key: beta.kubernetes.io/arch is deprecated since v1.14; use "kubernetes.io/arch" instead
16:12:18   +  pulumi:pulumi:Stack aws-eks-marajenkaws3 creating error: unable to recognize "/tmp/tmp-23547dNCAQqk1wCDq.tmp": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
16:12:18   +  aws:ec2:SecurityGroupRule aws-eks-marajenkaws3-eksExtApiServerClusterIngressRule creating 
16:12:18   +  aws:ec2:SecurityGroupRule aws-eks-marajenkaws3-eksNodeIngressRule creating 
16:12:18   +  aws:ec2:SecurityGroupRule aws-eks-marajenkaws3-eksNodeInternetEgressRule creating 
16:12:18   +  aws:ec2:SecurityGroupRule aws-eks-marajenkaws3-eksClusterIngressRule creating 
16:12:18   +  aws:ec2:SecurityGroupRule aws-eks-marajenkaws3-eksNodeClusterIngressRule creating 
16:12:18   +  aws:ec2:SecurityGroupRule aws-eks-marajenkaws3-eksExtApiServerClusterIngressRule created 
16:12:19   +  aws:ec2:SecurityGroupRule aws-eks-marajenkaws3-eksClusterIngressRule created 
16:12:19   +  aws:ec2:SecurityGroupRule aws-eks-marajenkaws3-eksNodeIngressRule created 
16:12:19   +  aws:ec2:SecurityGroupRule aws-eks-marajenkaws3-eksNodeInternetEgressRule created 
16:12:20   +  aws:ec2:SecurityGroupRule aws-eks-marajenkaws3-eksNodeClusterIngressRule created 
16:12:20   +  pulumi:pulumi:Stack aws-eks-marajenkaws3 creating error: update failed
16:12:20   +  pulumi:pulumi:Stack aws-eks-marajenkaws3 creating error: Resource monitor has terminated, shutting down
16:12:20   +  eks:index:Cluster aws-eks-marajenkaws3 created 
16:12:20   +  pulumi:pulumi:Stack aws-eks-marajenkaws3 **creating failed** 2 errors; 7 messages
16:12:20   
16:12:20  Diagnostics:
16:12:20    eks:index:VpcCni (aws-eks-marajenkaws3-vpc-cni):
16:12:20      error: Command failed: kubectl apply -f /tmp/tmp-23547dNCAQqk1wCDq.tmp
16:12:20      Warning: spec.template.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].key: beta.kubernetes.io/os is deprecated since v1.14; use "kubernetes.io/os" instead
16:12:20      Warning: spec.template.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[1].key: beta.kubernetes.io/arch is deprecated since v1.14; use "kubernetes.io/arch" instead
16:12:20      error: unable to recognize "/tmp/tmp-23547dNCAQqk1wCDq.tmp": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
16:12:20   
16:12:20    pulumi:pulumi:Stack (aws-eks-marajenkaws3):
16:12:20      aws **** profile
16:12:20      vpc id: vpc-05315b2f4bfb9acce
16:12:20      public subnets: ['subnet-085c683366982fc83', 'subnet-09e52459a6598217d', 'subnet-0f1483979c35c3fe1', 'subnet-0a7c801352bda906c']
16:12:20      public subnets: ['subnet-027f5d25d0b5cdc03', 'subnet-038d95e76e9cdaa9b', 'subnet-0d056c799b48ee6c4', 'subnet-0e5d8659ed51f17ef']
16:12:20      error: update failed
16:12:20      error: Resource monitor has terminated, shutting down
16:12:20   
16:12:20      Warning: spec.template.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].key: beta.kubernetes.io/os is deprecated since v1.14; use "kubernetes.io/os" instead
16:12:20      Warning: spec.template.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[1].key: beta.kubernetes.io/arch is deprecated since v1.14; use "kubernetes.io/arch" instead
16:12:20      error: unable to recognize "/tmp/tmp-23547dNCAQqk1wCDq.tmp": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
16:12:20   
16:12:20  Resources:
16:12:20      + 28 created
16:12:20  
16:12:20  Duration: 9m44s

The workaround is to downgrade to EKS 1.21

To Reproduce
Attempt to deploy version 1.22 of EKS.

Expected behavior
Should deploy and run normally

Your environment
Current python modules:

awscli~=1.22.100
grpcio==1.43.0
fart~=0.1.5
lolcat~=1.4
nodeenv~=1.6.0
passlib~=1.7.4
pulumi-aws>=4.37.5
pulumi-docker==3.1.0
pulumi-eks>=0.37.1
pulumi-kubernetes==3.18.2
pycryptodome~=3.14.0
PyYAML~=5.4.1
requests~=2.27.1
setuptools==62.1.0
setuptools-git-versioning==1.9.2
wheel==0.37.1
yamlreader==3.0.4
pulumi-digitalocean==4.12.0
pulumi-linode==3.7.1
linode-cli~=5.17.2
pulumi~=3.30.0

Additional context
None.
@qdzlug qdzlug mentioned this issue May 3, 2022
Merged
7 tasks
@qdzlug
Copy link
Contributor Author

qdzlug commented May 4, 2022 •
edited
Loading

This seemed to be tied into the issue noted in #139 - however, even with the kubectl fix discussed in that issue this version still fails.

@agpenton
Copy link

Any update on this issue? I'm stuck in the version upgrade because of this error.

@qdzlug
Copy link
Contributor Author

qdzlug commented Aug 19, 2022

#167 will close this issue; fix is to go to 1.23.x (but not 1.24.x as that shows other errors #151 and related)

@qdzlug qdzlug linked a pull request Aug 19, 2022 that will close this issue
Add support for the Pulumi Automation API #167
Merged
7 tasks
@qdzlug qdzlug self-assigned this Aug 19, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@qdzlug qdzlug

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add support for the Pulumi Automation API nginxinc/kic-reference-architectures
2 participants
@qdzlug @agpenton