fix: do not deregister GitLab tokens which are still in use

[kayman-mk]

Description

So far we haven't tracked the usage of the GitLab Runner token. In case the Runner is stopped and a new one is spawned, race conditions might occur as explained in #1062. In consequence the new Runner used a token deleted by the old Runner. This leads into downtimes as the token is no longer valid and can't be used.

This PR converts the token into a JSON format and ensures that tokens which are still in use, are not deregistered.

Fixes #1062

Migrations needed

In case you want to rollback to a previous version you have to convert the SSM parameter containing the token and usage counter in JSON format back to a plain token string.

Verification

• script changes were tested locally
• Test Runner was started to ensure that the token conversion works

[github-actions]

Hey @kayman-mk! 👋

Thank you for your contribution to the project. Please refer to the contribution rules for a quick overview of the process.

Make sure that this PR clearly explains:

• the problem being solved
• the best way a reviewer and you can test your changes

With submitting this PR you confirm that you hold the rights of the code added and agree that it will published under this LICENSE.

The following ChatOps commands are supported:

• /help: notifies a maintainer to help you out

Simply add a comment with the command in the first line. If you need to pass more information, separate it with a blank line from the command.

This message was generated automatically. You are welcome to improve it.

[github-actions]

🦙 MegaLinter status: ✅ SUCCESS

Descriptor	Linter	Files	Fixed	Errors	Elapsed time
✅ COPYPASTE	jscpd	yes		no	1.72s
✅ REPOSITORY	checkov	yes		no	15.63s
✅ REPOSITORY	dustilock	yes		no	0.36s
✅ REPOSITORY	gitleaks	yes		no	1.09s
✅ REPOSITORY	git_diff	yes		no	0.02s
✅ REPOSITORY	grype	yes		no	11.06s
✅ REPOSITORY	secretlint	yes		no	1.41s
✅ REPOSITORY	syft	yes		no	0.55s
✅ REPOSITORY	trivy-sbom	yes		no	1.63s
✅ REPOSITORY	trufflehog	yes		no	8.97s

See detailed report in MegaLinter reports
Set VALIDATE_ALL_CODEBASE: true in mega-linter.yml to validate all sources, not only the diff

MegaLinter is graciously provided by

[long-wan-ep]

I think it still needs a section for incrementing the counter, but looking good so far.

[kayman-mk]

I think it still needs a section for incrementing the counter, but looking good so far.

@long-wan-ep Yes, missed that and added it a second ago.

[long-wan-ep]

@kayman-mk Thanks for adding that. I tested this out by deploying a test runner and triggering instance refreshes on the ASG. The increment is working, but the decrement is not. I took a look inside the runner instance's /opt/remove_gitlab_registration.sh and the script had all of its variables expanded, this is what it looks like:

#!/bin/bash
json_token={
  "token": "xxxxxxxxxxxxxx",
  "usage_counter": 2
}
deregister_runner=true

# for  module versions >7.4.1 the token is JSON
if [[ "{
  "token": "xxxxxxxxxxxxxx",
  "usage_counter": 2
}" =~ ^\{.* ]]; then
  usage_counter=2

  # ensure that the token is not in use by another Runner
  if [[ 2 -gt 1 ]]; then
    deregister_runner=false
    token="not needed"
  else
    token=xxxxxxxxxxxxxx
  fi
else
  token={
  "token": "xxxxxxxxxxxxxx",
  "usage_counter": 2
}
fi

if [[  == "true" ]]; then
  echo "Removing Gitlab Runner ..."

  aws ssm put-parameter --overwrite --type SecureString  --name "runner-usage-counter-test-runner-token" --region "us-west-2" --value="{\"token\": \"null\", \"in_use_counter\": 0}" 2>&1
  curl -sS  --request DELETE "https://gitlab.com//api/v4/runners" --form "token=xxxxxxxxxxxxxx" 2>&1
else
  usage_counter=2
  usage_counter=1
  json_token={
  "token": "xxxxxxxxxxxxxx",
  "usage_counter": 2
}

  aws ssm put-parameter --overwrite --type SecureString  --name "runner-usage-counter-test-runner-token" --region "us-west-2" --value="{
  "token": "xxxxxxxxxxxxxx",
  "usage_counter": 2
}" 2>&1

  echo "Token still in use. GitLab Runner not removed from GitLab."
fi

[kayman-mk]

The problem was the parameter expansion during the cat operation. Everything fixed now. Spawn a new agent, killed it via console and terraform apply. Decreasing the usage counter works and new instances increase it. Looks good from my side.

@long-wan-ep would appreciate a quick check from you, if you have the time

[long-wan-ep]

Overall looks good, one minor thing.