diff --git a/README.md b/README.md index 21c134522e..f5ca0031b5 100644 --- a/README.md +++ b/README.md @@ -18,6 +18,39 @@ Terraform module which creates AWS EKS (Kubernetes) resources - Support for providing maps of node groups/Fargate profiles to the cluster module definition or use separate node group/Fargate profile sub-modules - Provisions to provide node group/Fargate profile "default" settings - useful for when creating multiple node groups/Fargate profiles where you want to set a common set of configurations once, and then individual control only select features +### ℹ️ `Error: Invalid for_each argument ...` + +Users may encounter an error such as `Error: Invalid for_each argument - The "for_each" value depends on resource attributes that cannot be determined until apply, so Terraform cannot predict how many instances will be created. To work around this, use the -target argument to first apply ...` + +This error is due to an upstream issue with [Terraform core](https://github.com/hashicorp/terraform/issues/4149). There are two potential options you can take to help mitigate this issue: + +1. Create the dependent resources before the cluster => `terraform apply -target ` and then `terraform apply` for the cluster (or other similar means to just ensure the referenced resources exist before creating the cluster) + - Note: this is the route users will have to take for adding additonal security groups to nodes since there isn't a separate "security group attachment" resource +2. For addtional IAM policies, users can attach the policies outside of the cluster definition as demonstrated below + +```hcl +resource "aws_iam_role_policy_attachment" "additional" { + for_each = module.eks.eks_managed_node_groups + # you could also do the following or any comibination: + # for_each = merge( + # module.eks.eks_managed_node_groups, + # module.eks.self_managed_node_group, + # module.eks.fargate_profile, + # ) + + # This policy does not have to exist at the time of cluster creation. Terraform can + # deduce the proper order of its creation to avoid errors during creation + policy_arn = aws_iam_policy.node_additional.arn + role = each.value.iam_role_name +} +``` + +The tl;dr for this issue is that the Terraform resource passed into the modules map definition *must* be known before you can apply the EKS module. The variables this potentially affects are: + +- `cluster_security_group_additional_rules` (i.e. - referencing an external security group resource in a rule) +- `node_security_group_additional_rules` (i.e. - referencing an external security group resource in a rule) +- `iam_role_additional_policies` (i.e. - referencing an external policy resource) + ## Usage ```hcl diff --git a/examples/eks_managed_node_group/README.md b/examples/eks_managed_node_group/README.md index 6b662257f9..ca475e572f 100644 --- a/examples/eks_managed_node_group/README.md +++ b/examples/eks_managed_node_group/README.md @@ -53,11 +53,14 @@ Note that this example may create resources which cost money. Run `terraform des | Name | Type | |------|------| +| [aws_iam_policy.node_additional](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_policy) | resource | +| [aws_iam_role_policy_attachment.additional](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy_attachment) | resource | | [aws_key_pair.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/key_pair) | resource | | [aws_kms_key.ebs](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/kms_key) | resource | | [aws_kms_key.eks](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/kms_key) | resource | | [aws_launch_template.external](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/launch_template) | resource | | [aws_security_group.additional](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/security_group) | resource | +| [aws_security_group.remote_access](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/security_group) | resource | | [null_resource.patch](https://registry.terraform.io/providers/hashicorp/null/latest/docs/resources/resource) | resource | | [tls_private_key.this](https://registry.terraform.io/providers/hashicorp/tls/latest/docs/resources/private_key) | resource | | [aws_caller_identity.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/caller_identity) | data source | diff --git a/examples/eks_managed_node_group/main.tf b/examples/eks_managed_node_group/main.tf index ce76f94085..2c540e2ae7 100644 --- a/examples/eks_managed_node_group/main.tf +++ b/examples/eks_managed_node_group/main.tf @@ -76,7 +76,8 @@ module "eks" { # Remote access cannot be specified with a launch template remote_access = { - ec2_ssh_key = aws_key_pair.this.key_name + ec2_ssh_key = aws_key_pair.this.key_name + source_security_group_ids = [aws_security_group.remote_access.id] } } @@ -269,6 +270,18 @@ module "eks" { tags = local.tags } +# References to resources that do not exist yet when creating a cluster will cause a plan failure due to https://github.com/hashicorp/terraform/issues/4149 +# There are two options users can take +# 1. Create the dependent resources before the cluster => `terraform apply -target and then `terraform apply` +# Note: this is the route users will have to take for adding additonal security groups to nodes since there isn't a separate "security group attachment" resource +# 2. For addtional IAM policies, users can attach the policies outside of the cluster definition as demonstrated below +resource "aws_iam_role_policy_attachment" "additional" { + for_each = module.eks.eks_managed_node_groups + + policy_arn = aws_iam_policy.node_additional.arn + role = each.value.iam_role_name +} + ################################################################################ # aws-auth configmap # Only EKS managed node groups automatically add roles to aws-auth configmap @@ -529,3 +542,47 @@ resource "aws_key_pair" "this" { tags = local.tags } + +resource "aws_security_group" "remote_access" { + name_prefix = "${local.name}-remote-access" + description = "Allow remote SSH access" + vpc_id = module.vpc.vpc_id + + ingress { + description = "SSH access" + from_port = 22 + to_port = 22 + protocol = "tcp" + cidr_blocks = ["10.0.0.0/8"] + } + + egress { + from_port = 0 + to_port = 0 + protocol = "-1" + cidr_blocks = ["0.0.0.0/0"] + ipv6_cidr_blocks = ["::/0"] + } + + tags = local.tags +} + +resource "aws_iam_policy" "node_additional" { + name = "${local.name}-additional" + description = "Example usage of node additional policy" + + policy = jsonencode({ + Version = "2012-10-17" + Statement = [ + { + Action = [ + "ec2:Describe*", + ] + Effect = "Allow" + Resource = "*" + }, + ] + }) + + tags = local.tags +} diff --git a/modules/eks-managed-node-group/README.md b/modules/eks-managed-node-group/README.md index 43e4a72bc3..334edb4599 100644 --- a/modules/eks-managed-node-group/README.md +++ b/modules/eks-managed-node-group/README.md @@ -139,7 +139,7 @@ module "eks_managed_node_group" { | [post\_bootstrap\_user\_data](#input\_post\_bootstrap\_user\_data) | User data that is appended to the user data script after of the EKS bootstrap script. Not used when `platform` = `bottlerocket` | `string` | `""` | no | | [pre\_bootstrap\_user\_data](#input\_pre\_bootstrap\_user\_data) | User data that is injected into the user data script ahead of the EKS bootstrap script. Not used when `platform` = `bottlerocket` | `string` | `""` | no | | [ram\_disk\_id](#input\_ram\_disk\_id) | The ID of the ram disk | `string` | `null` | no | -| [remote\_access](#input\_remote\_access) | Configuration block with remote access settings | `map(string)` | `{}` | no | +| [remote\_access](#input\_remote\_access) | Configuration block with remote access settings | `any` | `{}` | no | | [security\_group\_description](#input\_security\_group\_description) | Description for the security group created | `string` | `"EKS managed node group security group"` | no | | [security\_group\_name](#input\_security\_group\_name) | Name to use on security group created | `string` | `null` | no | | [security\_group\_rules](#input\_security\_group\_rules) | List of security group rules to add to the security group created | `any` | `{}` | no | diff --git a/modules/eks-managed-node-group/variables.tf b/modules/eks-managed-node-group/variables.tf index 38d126f269..d35bcc75d6 100644 --- a/modules/eks-managed-node-group/variables.tf +++ b/modules/eks-managed-node-group/variables.tf @@ -334,7 +334,7 @@ variable "launch_template_version" { variable "remote_access" { description = "Configuration block with remote access settings" - type = map(string) + type = any default = {} }