feat(spotfleet): add supports spot fleets for spot instances allowing us to use multiple instance types and AZs (#777)

## Description

The
[fork](https://gitlab.com/cki-project/docker-machine/-/tree/v0.16.2-gitlab.19-cki.2)
of docker-machine used in this module has released a version that
supports [AWS Spot
Fleet](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-fleet.html)
3 months ago.

In our case, spot fleets are useful when using spot instances because it
fixes the issue when there is no capacity available for a certain
instance type in a certain AZ. With spot fleets, we can use multiple
instance types and AZ so when there is no capacity on one instance type,
AWS will take the next instance type and so on.

This fixes issues
[#76](#76)
[#435](#435)
[#77](#77)
and
[#474](#474)

## Migrations required
NO our solution is backward compatible so if you don't plan to use spot
fleets, no migration is required.
YES if you want to use spot fleets you will have to do the following
changes:
`docker_machine_instance_type` to `docker_machine_instance_types`
`subnet_id` to `subnet_ids`

You also have to put these 2 new parameters:
`use_fleet` and `key_pair_name` 

### Important
We have seen that the
[fork](https://gitlab.com/cki-project/docker-machine/-/tree/v0.16.2-gitlab.19-cki.2)
of docker-machine this module is using consume more RAM using spot
fleets.
For comparison, if you launch 50 machines in the same time, it consumes
~1.2GB of RAM. In our case, we had to change the `instance_type` of the
runner from `t3.micro` to `t3.small`.

## Verification

```hcl
  docker_machine_instance_types = ["r5a.xlarge", "r5.xlarge", "r5ad.xlarge", "r5d.xlarge", "m5a.2xlarge"]
  use_fleet                     = true
  key_pair_name                 = "<key-pair-name>"
```
We are using this solution in production for a week now with 150+
developers and over 20 000 jobs a day without any issue for now.

---------

Signed-off-by: François Bibron <[email protected]>
Co-authored-by: François Bibron <[email protected]>
Co-authored-by: Marvin Wichmann <[email protected]>
Co-authored-by: Matthias Kay <[email protected]>
Co-authored-by: kayma <[email protected]>

Author: 3 people

README.md

@@ -371,6 +371,48 @@ module "runner" {
 }
 ```
 
+### Scenario: Use of Spot Fleet
+
+Since spot instances can be taken over by AWS depending on the instance type and AZ you are using, you may want multiple instances types in multiple AZs. This is where spot fleets come in, when there is no capacity on one instance type and one AZ, AWS will take the next instance type and so on. This update has been possible since the [fork](https://gitlab.com/cki-project/docker-machine/-/tree/v0.16.2-gitlab.19-cki.2) of docker-machine supports spot fleets.
+
+We have seen that the [fork](https://gitlab.com/cki-project/docker-machine/-/tree/v0.16.2-gitlab.19-cki.2) of docker-machine this module is using consume more RAM using spot fleets.
+For comparison, if you launch 50 machines in the same time, it consumes ~1.2GB of RAM. In our case, we had to change the `instance_type` of the runner from `t3.micro` to `t3.small`.
+
+#### Configuration example
+```hcl
+module "runner" {
+  # https://registry.terraform.io/modules/npalm/gitlab-runner/aws/
+  source  = "npalm/gitlab-runner/aws"
+
+  aws_region  = "eu-west-3"
+  environment = "spot-runners"
+
+  vpc_id                    = module.vpc.vpc_id
+  subnet_id                 = module.vpc.private_subnets[0] # subnet of the agent
+  fleet_executor_subnet_ids = module.vpc.private_subnets
+
+  docker_machine_instance_types             = ["t3a.medium", "t3.medium", "t2.medium"]
+  use_fleet                                 = true
+  key_pair_name                             = "<key_pair_name>"
+
+  runners_name       = "docker-machine"
+  runners_gitlab_url = "https://gitlab.com"
+
+  gitlab_runner_registration_config = {
+    registration_token = "my-token"
+    tag_list           = "docker"
+    description        = "runner default"
+    locked_to_project  = "true"
+    run_untagged       = "false"
+    maximum_timeout    = "3600"
+  }
+
+  overrides = {
+    name_iam_objects = "<region-specific-prefix>-gitlab-runner-iam"
+  }
+}
+```
+
 ## Examples
 
 A few [examples](https://github.com/cattle-ops/terraform-aws-gitlab-runner/tree/main/examples/) are provided. Use the
@@ -435,13 +477,15 @@ Made with [contributors-img](https://contrib.rocks).
 | <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1 |
 | <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 4 |
 | <a name="requirement_local"></a> [local](#requirement\_local) | >= 2.4.0 |
+| <a name="requirement_tls"></a> [tls](#requirement\_tls) | >= 3 |
 
 ## Providers
 
 | Name | Version |
 |------|---------|
 | <a name="provider_aws"></a> [aws](#provider\_aws) | 4.49.0 |
 | <a name="provider_local"></a> [local](#provider\_local) | >= 2.4.0 |
+| <a name="provider_tls"></a> [tls](#provider\_tls) | >= 3 |
 
 ## Modules
 
@@ -479,9 +523,11 @@ Made with [contributors-img](https://contrib.rocks).
 | [aws_iam_role_policy_attachment.service_linked_role](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy_attachment) | resource |
 | [aws_iam_role_policy_attachment.ssm](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy_attachment) | resource |
 | [aws_iam_role_policy_attachment.user_defined_policies](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy_attachment) | resource |
+| [aws_key_pair.fleet_key](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/key_pair) | resource |
 | [aws_kms_alias.default](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/kms_alias) | resource |
 | [aws_kms_key.default](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/kms_key) | resource |
 | [aws_launch_template.gitlab_runner_instance](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/launch_template) | resource |
+| [aws_launch_template.gitlab_runners](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/launch_template) | resource |
 | [aws_security_group.docker_machine](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/security_group) | resource |
 | [aws_security_group.runner](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/security_group) | resource |
 | [aws_security_group_rule.docker_machine_docker_runner](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/security_group_rule) | resource |
@@ -495,6 +541,7 @@ Made with [contributors-img](https://contrib.rocks).
 | [aws_ssm_parameter.runner_sentry_dsn](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/ssm_parameter) | resource |
 | [local_file.config_toml](https://registry.terraform.io/providers/hashicorp/local/latest/docs/resources/file) | resource |
 | [local_file.user_data](https://registry.terraform.io/providers/hashicorp/local/latest/docs/resources/file) | resource |
+| [tls_private_key.runner](https://registry.terraform.io/providers/hashicorp/tls/latest/docs/resources/private_key) | resource |
 | [aws_ami.docker-machine](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/ami) | data source |
 | [aws_ami.runner](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/ami) | data source |
 | [aws_availability_zone.runners](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/availability_zone) | data source |
@@ -537,6 +584,7 @@ Made with [contributors-img](https://contrib.rocks).
 | <a name="input_docker_machine_iam_policy_arns"></a> [docker\_machine\_iam\_policy\_arns](#input\_docker\_machine\_iam\_policy\_arns) | List of policy ARNs to be added to the instance profile of the docker machine runners. | `list(string)` | `[]` | no |
 | <a name="input_docker_machine_instance_metadata_options"></a> [docker\_machine\_instance\_metadata\_options](#input\_docker\_machine\_instance\_metadata\_options) | Enable the docker machine instances metadata service. Requires you use GitLab maintained docker machines. | <pre>object({<br>    http_tokens                 = string<br>    http_put_response_hop_limit = number<br>  })</pre> | <pre>{<br>  "http_put_response_hop_limit": 2,<br>  "http_tokens": "required"<br>}</pre> | no |
 | <a name="input_docker_machine_instance_type"></a> [docker\_machine\_instance\_type](#input\_docker\_machine\_instance\_type) | Instance type used for the instances hosting docker-machine. | `string` | `"m5.large"` | no |
+| <a name="input_docker_machine_instance_types"></a> [docker\_machine\_instance\_types](#input\_docker\_machine\_instance\_types) | Instance types used for the instances hosting docker-machine. This variable is only supported when use\_fleet is set to true. | `list(string)` | `[]` | no |
 | <a name="input_docker_machine_options"></a> [docker\_machine\_options](#input\_docker\_machine\_options) | List of additional options for the docker machine config. Each element of this list must be a key=value pair. E.g. '["amazonec2-zone=a"]' | `list(string)` | `[]` | no |
 | <a name="input_docker_machine_role_json"></a> [docker\_machine\_role\_json](#input\_docker\_machine\_role\_json) | Docker machine runner instance override policy, expected to be in JSON format. | `string` | `""` | no |
 | <a name="input_docker_machine_security_group_description"></a> [docker\_machine\_security\_group\_description](#input\_docker\_machine\_security\_group\_description) | A description for the docker-machine security group | `string` | `"A security group containing docker-machine instances"` | no |
@@ -554,13 +602,15 @@ Made with [contributors-img](https://contrib.rocks).
 | <a name="input_enable_schedule"></a> [enable\_schedule](#input\_enable\_schedule) | Flag used to enable/disable auto scaling group schedule for the runner instance. | `bool` | `false` | no |
 | <a name="input_environment"></a> [environment](#input\_environment) | A name that identifies the environment, used as prefix and for tagging. | `string` | n/a | yes |
 | <a name="input_extra_security_group_ids_runner_agent"></a> [extra\_security\_group\_ids\_runner\_agent](#input\_extra\_security\_group\_ids\_runner\_agent) | Optional IDs of extra security groups to apply to the runner agent. This will not apply to the runners spun up when using the docker+machine executor, which is the default. | `list(string)` | `[]` | no |
+| <a name="input_fleet_executor_subnet_ids"></a> [fleet\_executor\_subnet\_ids](#input\_fleet\_executor\_subnet\_ids) | List of subnets used for executors when the fleet mode is enabled. Must belong to the VPC specified above. | `list(string)` | `[]` | no |
 | <a name="input_gitlab_runner_egress_rules"></a> [gitlab\_runner\_egress\_rules](#input\_gitlab\_runner\_egress\_rules) | List of egress rules for the gitlab runner instance. | <pre>list(object({<br>    cidr_blocks      = list(string)<br>    ipv6_cidr_blocks = list(string)<br>    prefix_list_ids  = list(string)<br>    from_port        = number<br>    protocol         = string<br>    security_groups  = list(string)<br>    self             = bool<br>    to_port          = number<br>    description      = string<br>  }))</pre> | <pre>[<br>  {<br>    "cidr_blocks": [<br>      "0.0.0.0/0"<br>    ],<br>    "description": null,<br>    "from_port": 0,<br>    "ipv6_cidr_blocks": [<br>      "::/0"<br>    ],<br>    "prefix_list_ids": null,<br>    "protocol": "-1",<br>    "security_groups": null,<br>    "self": null,<br>    "to_port": 0<br>  }<br>]</pre> | no |
 | <a name="input_gitlab_runner_registration_config"></a> [gitlab\_runner\_registration\_config](#input\_gitlab\_runner\_registration\_config) | Configuration used to register the runner. See the README for an example, or reference the examples in the examples directory of this repo. | `map(string)` | <pre>{<br>  "access_level": "",<br>  "description": "",<br>  "locked_to_project": "",<br>  "maximum_timeout": "",<br>  "registration_token": "",<br>  "run_untagged": "",<br>  "tag_list": ""<br>}</pre> | no |
 | <a name="input_gitlab_runner_security_group_description"></a> [gitlab\_runner\_security\_group\_description](#input\_gitlab\_runner\_security\_group\_description) | A description for the gitlab-runner security group | `string` | `"A security group containing gitlab-runner agent instances"` | no |
 | <a name="input_gitlab_runner_security_group_ids"></a> [gitlab\_runner\_security\_group\_ids](#input\_gitlab\_runner\_security\_group\_ids) | A list of security group ids that are allowed to access the gitlab runner agent | `list(string)` | `[]` | no |
 | <a name="input_gitlab_runner_version"></a> [gitlab\_runner\_version](#input\_gitlab\_runner\_version) | Version of the [GitLab runner](https://gitlab.com/gitlab-org/gitlab-runner/-/releases). | `string` | `"15.8.2"` | no |
 | <a name="input_instance_role_json"></a> [instance\_role\_json](#input\_instance\_role\_json) | Default runner instance override policy, expected to be in JSON format. | `string` | `""` | no |
 | <a name="input_instance_type"></a> [instance\_type](#input\_instance\_type) | Instance type used for the GitLab runner. | `string` | `"t3.micro"` | no |
+| <a name="input_key_pair_name"></a> [key\_pair\_name](#input\_key\_pair\_name) | The name of the key pair used by the runner to connect to the docker-machine executors. | `string` | `"fleet-key"` | no |
 | <a name="input_kms_alias_name"></a> [kms\_alias\_name](#input\_kms\_alias\_name) | Alias added to the kms\_key (if created and not provided by kms\_key\_id) | `string` | `""` | no |
 | <a name="input_kms_deletion_window_in_days"></a> [kms\_deletion\_window\_in\_days](#input\_kms\_deletion\_window\_in\_days) | Key rotation window, set to 0 for no rotation. Only used when `enable_kms` is set to `true`. | `number` | `7` | no |
 | <a name="input_kms_key_id"></a> [kms\_key\_id](#input\_kms\_key\_id) | KMS key id to encrypted the resources. Ensure CloudWatch and Runner/Executor have access to the provided KMS key. | `string` | `""` | no |
@@ -637,6 +687,7 @@ Made with [contributors-img](https://contrib.rocks).
 | <a name="input_subnet_ids_gitlab_runner"></a> [subnet\_ids\_gitlab\_runner](#input\_subnet\_ids\_gitlab\_runner) | Deprecated! Use subnet\_id instead. Subnet used for hosting the GitLab runner. | `list(string)` | `[]` | no |
 | <a name="input_suppressed_tags"></a> [suppressed\_tags](#input\_suppressed\_tags) | List of tag keys which are removed from tags, agent\_tags and runner\_tags and never added as default tag by the module. | `list(string)` | `[]` | no |
 | <a name="input_tags"></a> [tags](#input\_tags) | Map of tags that will be added to created resources. By default resources will be tagged with name and environment. | `map(string)` | `{}` | no |
+| <a name="input_use_fleet"></a> [use\_fleet](#input\_use\_fleet) | Use the fleet mode for agents. https://gitlab.com/cki-project/docker-machine/-/blob/v0.16.2-gitlab.19-cki.2/docs/drivers/aws.md#fleet-mode | `bool` | `false` | no |
 | <a name="input_userdata_post_install"></a> [userdata\_post\_install](#input\_userdata\_post\_install) | User-data script snippet to insert after GitLab runner install | `string` | `""` | no |
 | <a name="input_userdata_pre_install"></a> [userdata\_pre\_install](#input\_userdata\_pre\_install) | User-data script snippet to insert before GitLab runner install | `string` | `""` | no |
 | <a name="input_vpc_id"></a> [vpc\_id](#input\_vpc\_id) | The target VPC for the docker-machine and runner instances. | `string` | n/a | yes |

main.tf

@@ -77,6 +77,9 @@ locals {
       gitlab_runner_maximum_timeout                = var.gitlab_runner_registration_config["maximum_timeout"]
       gitlab_runner_access_level                   = lookup(var.gitlab_runner_registration_config, "access_level", "not_protected")
       sentry_dsn                                   = var.sentry_dsn
+      public_key                                   = var.use_fleet == true ? tls_private_key.fleet[0].public_key_openssh : ""
+      use_fleet                                    = var.use_fleet
+      private_key                                  = var.use_fleet == true ? tls_private_key.fleet[0].private_key_pem : ""
   })
 
   template_runner_config = templatefile("${path.module}/template/runner-config.tftpl",
@@ -88,8 +91,10 @@ locals {
       runners_extra_hosts               = var.runners_extra_hosts
       runners_vpc_id                    = var.vpc_id
       runners_subnet_id                 = length(var.subnet_id) > 0 ? var.subnet_id : var.subnet_id_runners
+      runners_subnet_ids                = length(var.fleet_executor_subnet_ids) > 0 ? var.fleet_executor_subnet_ids : length(var.subnet_id) > 0 ? [var.subnet_id] : [var.subnet_id_runners]
       runners_aws_zone                  = data.aws_availability_zone.runners.name_suffix
       runners_instance_type             = var.docker_machine_instance_type
+      runners_instance_types            = length(var.docker_machine_instance_types_fleet) > 0 ? var.docker_machine_instance_types_fleet : [var.docker_machine_instance_type]
       runners_spot_price_bid            = var.docker_machine_spot_price_bid == "on-demand-price" || var.docker_machine_spot_price_bid == null ? "" : var.docker_machine_spot_price_bid
       runners_ami                       = var.runners_executor == "docker+machine" ? data.aws_ami.docker-machine[0].id : ""
       runners_security_group_name       = var.runners_executor == "docker+machine" ? aws_security_group.docker_machine[0].name : ""
@@ -138,6 +143,8 @@ locals {
       sentry_dsn                        = var.sentry_dsn
       prometheus_listen_address         = var.prometheus_listen_address
       auth_type                         = var.auth_type_cache_sr
+      use_fleet                         = var.use_fleet
+      launch_template                   = var.use_fleet == true ? aws_launch_template.fleet_gitlab_runner[0].name : ""
     }
   )
 }
@@ -163,7 +170,7 @@ data "aws_ami" "docker-machine" {
 # kics-scan ignore-line
 resource "aws_autoscaling_group" "gitlab_runner_instance" {
   name                      = var.enable_asg_recreation ? "${aws_launch_template.gitlab_runner_instance.name}-asg" : "${var.environment}-as-group"
-  vpc_zone_identifier       = length(var.subnet_id) > 0 ? [var.subnet_id] : var.subnet_ids_gitlab_runner
+  vpc_zone_identifier       = length(var.fleet_executor_subnet_ids) > 0 ? var.fleet_executor_subnet_ids : length(var.subnet_id) > 0 ? [var.subnet_id] : var.subnet_ids_gitlab_runner
   min_size                  = "1"
   max_size                  = "1"
   desired_capacity          = "1"
@@ -319,6 +326,77 @@ resource "aws_launch_template" "gitlab_runner_instance" {
   depends_on = [aws_cloudwatch_log_group.environment]
 }
 
+resource "tls_private_key" "fleet" {
+  count = var.use_fleet == true && var.runners_executor == "docker+machine" ? 1 : 0
+
+  algorithm = "RSA"
+  rsa_bits  = 4096
+}
+
+resource "aws_key_pair" "fleet" {
+  count = var.use_fleet == true && var.runners_executor == "docker+machine" ? 1 : 0
+
+  key_name   = "${var.environment}-${var.fleet_key_pair_name}"
+  public_key = tls_private_key.fleet[0].public_key_openssh
+
+  tags = local.tags
+}
+
+resource "aws_launch_template" "fleet_gitlab_runner" {
+  # checkov:skip=CKV_AWS_88:User can decide to add a public IP.
+  # checkov:skip=CKV_AWS_79:User can decide to enable Metadata service V2. V2 is the default.
+  count       = var.use_fleet == true && var.runners_executor == "docker+machine" ? 1 : 0
+  name_prefix = "${local.name_runner_agent_instance}-worker-"
+
+  key_name               = aws_key_pair.fleet[0].key_name
+  image_id               = data.aws_ami.docker-machine[0].id
+  user_data              = base64gzip(var.runners_userdata)
+  instance_type          = var.docker_machine_instance_types_fleet[0] # it will be override by the fleet
+  update_default_version = true
+  ebs_optimized          = var.runners_ebs_optimized
+  monitoring {
+    enabled = var.runners_monitoring
+  }
+  block_device_mappings {
+    device_name = "/dev/sda1"
+
+    ebs {
+      volume_size = var.runners_root_size
+      volume_type = var.runners_volume_type
+    }
+  }
+
+  iam_instance_profile {
+    name = aws_iam_instance_profile.docker_machine[0].name
+  }
+
+  network_interfaces {
+    security_groups             = [aws_security_group.docker_machine[0].id]
+    associate_public_ip_address = !var.runners_use_private_address
+  }
+
+  tag_specifications {
+    resource_type = "instance"
+    tags          = local.tags
+  }
+  tag_specifications {
+    resource_type = "volume"
+    tags          = local.tags
+  }
+
+  tags = local.tags
+
+  metadata_options {
+    http_tokens                 = var.docker_machine_instance_metadata_options.http_tokens
+    http_put_response_hop_limit = var.docker_machine_instance_metadata_options.http_put_response_hop_limit
+    instance_metadata_tags      = "enabled"
+  }
+
+  lifecycle {
+    create_before_destroy = true
+  }
+}
+
 ################################################################################
 ### Create cache bucket
 ################################################################################

policies/instance-docker-machine-policy.json

@@ -17,7 +17,8 @@
           "ec2:RequestSpotInstances",
           "ec2:CancelSpotInstanceRequests",
           "ec2:DescribeSubnets",
-          "ec2:AssociateIamInstanceProfile"
+          "ec2:AssociateIamInstanceProfile",
+          "ec2:CreateFleet"
         ],
         "Effect": "Allow",
         "Resource": "*"

template/gitlab-runner.tftpl

@@ -48,6 +48,14 @@ fi
 
 sed -i.bak s/__REPLACED_BY_USER_DATA__/$token/g /etc/gitlab-runner/config.toml
 
+if [[ "${use_fleet}" == "true" ]]
+then
+  echo "${public_key}" > /root/.ssh/id_rsa.pub
+  echo "${private_key}" > /root/.ssh/id_rsa
+
+  chmod 600 /root/.ssh/id_rsa
+fi
+
 ssm_sentry_dsn=$(aws ssm get-parameters --names "${secure_parameter_store_runner_sentry_dsn}" --with-decryption --region "${secure_parameter_store_region}" | jq -r ".Parameters | .[0] | .Value")
 if [[ "${sentry_dsn}" == "__SENTRY_DSN_REPLACED_BY_USER_DATA__" && "$ssm_sentry_dsn" == "null" ]]
 then

template/runner-config.tftpl

@@ -50,24 +50,33 @@ listen_address = "${prometheus_listen_address}"
     MachineDriver = "amazonec2"
     MachineName = "${docker_machine_name}"
     MachineOptions = [
-      "amazonec2-instance-type=${runners_instance_type}",
+      %{~ for instance_type in runners_instance_types ~}
+      "amazonec2-instance-type=${instance_type}",
+      %{~ endfor ~}
       "amazonec2-region=${aws_region}",
       "amazonec2-zone=${runners_aws_zone}",
       "amazonec2-vpc-id=${runners_vpc_id}",
-      "amazonec2-subnet-id=${runners_subnet_id}",
+      %{~ for subnet_id in runners_subnet_ids ~}
+      "amazonec2-subnet-id=${subnet_id}",
+      %{~ endfor ~}
       "amazonec2-private-address-only=${runners_use_private_address_only}",
       "amazonec2-use-private-address=${runners_use_private_address}",
       "amazonec2-request-spot-instance=${runners_request_spot_instance}",
-      %{ if runners_spot_price_bid != ""}"amazonec2-spot-price=${runners_spot_price_bid}",%{ endif ~}
+      %{~ if runners_spot_price_bid != "" ~}"amazonec2-spot-price=${runners_spot_price_bid}",%{~ endif ~}
       "amazonec2-security-group=${runners_security_group_name}",
       "amazonec2-tags=${runners_tags},__PARENT_TAG__",
       "amazonec2-use-ebs-optimized-instance=${runners_ebs_optimized}",
       "amazonec2-monitoring=${runners_monitoring}",
       "amazonec2-iam-instance-profile=%{ if runners_iam_instance_profile_name != "" }${runners_iam_instance_profile_name}%{ else }${runners_instance_profile}%{ endif ~}",
       "amazonec2-root-size=${runners_root_size}",
       "amazonec2-volume-type=${runners_volume_type}",
-      "amazonec2-userdata=%{ if runners_userdata != "" }/etc/gitlab-runner/runners_userdata.sh%{ endif ~}",
+      "amazonec2-userdata=%{~ if runners_userdata != "" ~}/etc/gitlab-runner/runners_userdata.sh%{~ endif ~}",
       "amazonec2-ami=${runners_ami}"
+      %{~ if use_fleet == true ~}
+      ,"amazonec2-ssh-keypath=/root/.ssh/id_rsa",
+      "amazonec2-use-fleet=${use_fleet}",
+      "amazonec2-launch-template=${launch_template}"
+      %{~ endif ~}
       ${docker_machine_options}
     ]
 

variables.tf

@@ -31,6 +31,12 @@ variable "subnet_id" {
   default     = "" # TODO remove as soon as subnet_id_runners and subnet_ids_gitlab_runner are gone. Variable is mandatory now.
 }
 
+variable "fleet_executor_subnet_ids" {
+  description = "List of subnets used for executors when the fleet mode is enabled. Must belong to the VPC specified above."
+  type        = list(string)
+  default     = []
+}
+
 variable "extra_security_group_ids_runner_agent" {
   description = "Optional IDs of extra security groups to apply to the runner agent. This will not apply to the runners spun up when using the docker+machine executor, which is the default."
   type        = list(string)
@@ -43,6 +49,12 @@ variable "metrics_autoscaling" {
   default     = null
 }
 
+variable "fleet_key_pair_name" {
+  description = "The name of the key pair used by the runner to connect to the docker-machine executors."
+  type        = string
+  default     = "fleet-key"
+}
+
 variable "instance_type" {
   description = "Instance type used for the GitLab runner."
   type        = string
@@ -101,6 +113,12 @@ variable "docker_machine_instance_type" {
   default     = "m5.large"
 }
 
+variable "docker_machine_instance_types_fleet" {
+  description = "Instance types used for the instances hosting docker-machine. This variable is only supported when use_fleet is set to true."
+  type        = list(string)
+  default     = []
+}
+
 variable "docker_machine_spot_price_bid" {
   description = "Spot price bid. The maximum price willing to pay. By default the price is limited by the current on demand price for the instance type chosen."
   type        = string
@@ -763,6 +781,12 @@ variable "kms_deletion_window_in_days" {
   default     = 7
 }
 
+variable "use_fleet" {
+  description = "Use the fleet mode for agents. https://gitlab.com/cki-project/docker-machine/-/blob/v0.16.2-gitlab.19-cki.2/docs/drivers/aws.md#fleet-mode"
+  type        = bool
+  default     = false
+}
+
 variable "enable_eip" {
   description = "Enable the assignment of an EIP to the gitlab runner instance"
   default     = false

versions.tf

@@ -10,5 +10,9 @@ terraform {
       source  = "hashicorp/local"
       version = ">= 2.4.0"
     }
+    tls = {
+      source  = "hashicorp/tls"
+      version = ">= 3"
+    }
   }
 }