Clean SSH key pairs in AWS #592
Labels
bug 🐛
Something isn't working
Comments
|
Same problem here. Have a Lambda running to clean up unused resources. |
|
We should add a Lambda function to the module to clean up the unused key pairs every day. This module should not leave unused resources in the AWS account. |
4 tasks
|
I thought this worked in the past, the agent host has permission to delete the key. Could it that we missed something in configuration? Otherwise a lambda would be the only option. Which we we can trigger on docker machine instance termination or just run on a schedule. |
kayman-mk
added a commit
that referenced
this issue
Mar 2, 2023
## Description Whenever an executor is created it needs an SSH key so the agent is able to contact it. In case the executor is killed this SSH key is not removed. Over time they accumulate and reach the AWS limit. From this point no further SSH keys can be created and thus the module is no longer able to create executors. You won't be able to run any pipeline. Some users have created their own housekeeping of SSH keys. But this should be done within the module as the keys are created here. Challenge: Make sure that keys are deleted only which belong to our module. Unfortunately the GitLab Runner does not tag the keys. We use the following logic to determine the keys to delete: - they are unused - their name starts with `runner` - they have the `var.environment`/`var.overrides['name_docker_machine']` somewhere in the name Solves part of #623 Closes #592
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
We recently hit the AWS quota for the maximum number of SSH key pairs:
Around 4000 of the keys were created by the runner when provisioning machines and left there.
It looks like the runner does not remove the associated key pair when removing a worker.
I would like to be able to:
Not use SSH keys at all (only SSM). I do not know if this is possible, as it looks like docker machine uses those keys for keeping runner-worker communication.
OR
Specify one single key for all workers so they do not get created on demand.
As a temporary fix I created a schedule that cleans all unused CI ssh keys: https://gitlab.com/fluidattacks/universe/-/issues/8300
Thank you so much for all the effort you've put in this module!
The text was updated successfully, but these errors were encountered: