Possible race condition with aws_iam_role.instance recreate

[dzinek]

Hi, I decided to upgrade module from version 5.4.1 to 5.5.0 due to fixed #557.

Plan showed me that ASG Launch Template will be changed (due to changes in user_data with pull policy and gzip from #565).
But also the aws_iam_role.instance has to be recreated:

  # module.gitlab_runner.aws_iam_role.instance will be destroyed
  # (because resource uses count or for_each)
  - resource "aws_iam_role" "instance" {

and then

  # module.gitlab_runner.aws_iam_role.instance[0] will be created
  + resource "aws_iam_role" "instance" {

The plan looked good (first destroy role then create it) so I applied it.
Changes with launch template and ASG passed, but role recreation failed - seems like the race condition - the new role is starting to create, when the old one has not finish destroying:

module.gitlab_runner.aws_iam_role.instance: Destroying... [id=runner-instance]
module.gitlab_runner.aws_iam_role.instance[0]: Creating...
module.gitlab_runner.aws_iam_role.instance: Destruction complete after 3s
╷
│ Error: failed creating IAM Role (runner-instance): EntityAlreadyExists: Role with name runner-instance already exists.
│ 	status code: 409, request id: xxxxxxx
│
│   with module.gitlab_runner.aws_iam_role.instance[0],
│   on .terraform/modules/gitlab_runner/main.tf line 331, in resource "aws_iam_role" "instance":
│  331: resource "aws_iam_role" "instance" {

I need to run apply three times - 1st one to got this error, and also another two times to pass it.

The final state was, that the new runner cannot be registered, due to issue in user_data with fetching runner token from SSM. The instance profile lost the assigned role, so the ec2 could not fetch any data from SSM.
I had to reassign role to instance profile manually to fix the access and have runner registered.

[kayman-mk]

Can't confirm this using Terraform 1.3.2

Plan says:
module.gitlab_runner.module.gitlab_runner_test["subnet-xyz"].aws_iam_role.instance has moved to module.gitlab_runner.module.gitlab_runner_test["subnet-xyz"].aws_iam_role.instance[0].

The resource is not destroyed and recreated.

[dzinek]

Ok, we are in process of migration to 1.3.5 - will test it on the later TF.

[kayman-mk]

Seems to be related to #591

[kayman-mk]

Any news here, @dzinek? Or can we close this issue?

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 15 days.

[github-actions]

This issue was closed because it has been stalled for 15 days with no activity.