Skip to content

Commit 583700c

Browse files
authored
feat: Support multi-region deployments (#437)
* fix: add override for IAM objects name to all IAM resources * fix: input variable for IAM object name override in cache module * chore: update comment * chore: revert unnecessary name changes * chore: fix typo in description of overrides variable * feat: add example for multi-region deployment * docs: update readme * docs: fix typo * chore: apply review changes * feat: remove protected runner setting from configuration
1 parent 54c10f3 commit 583700c

File tree

19 files changed

+409
-10
lines changed

19 files changed

+409
-10
lines changed

.github/workflows/ci.yml

+1-1
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ jobs:
2626
fail-fast: false
2727
matrix:
2828
terraform: [0.13.0, 0.14.0, 0.15.0, 1.0.8]
29-
example: ["runner-default", "runner-docker", "runner-pre-registered", "runner-public"]
29+
example: ["runner-default", "runner-docker", "runner-multi-region", "runner-pre-registered", "runner-public"]
3030
defaults:
3131
run:
3232
working-directory: examples/${{ matrix.example }}

README.md

+45-2
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@ overrides = {
3232
name_sg = ""
3333
name_runner_agent_instance = "Gitlab Runner Agent"
3434
name_docker_machine_runners = "Gitlab Runner Terraform"
35+
name_iam_objects = "gitlab-runner"
3536
}
3637
3738
//this doesn't work
@@ -171,6 +172,8 @@ By default the module creates a a cache for the runner in S3. Old objects are au
171172

172173
Creation of the bucket can be disabled and managed outside this module. A good use case is for sharing the cache across multiple runners. For this purpose the cache is implemented as a sub module. For more details see the [cache module](https://github.com/npalm/terraform-aws-gitlab-runner/tree/develop/cache). An example implementation of this use case can be found in the [runner-public](https://github.com/npalm/terraform-aws-gitlab-runner/tree/__GIT_REF__/examples/runner-public) example.
173174

175+
176+
174177
## Usage
175178

176179
### Configuration
@@ -185,7 +188,7 @@ runner_token = "RUNNER_TOKEN"
185188

186189
The base image used to host the GitLab Runner agent is the latest available Amazon Linux 2 HVM EBS AMI. In previous versions of this module a hard coded list of AMIs per region was provided. This list has been replaced by a search filter to find the latest AMI. Setting the filter to `amzn2-ami-hvm-2.0.20200207.1-x86_64-ebs` will allow you to version lock the target AMI.
187190

188-
### Usage module
191+
### Scenario: Basic usage
189192

190193
Below is a basic examples of usages of the module. Regarding the dependencies such as a VPC and SSH keys, have a look at the [default example](https://github.com/npalm/terraform-aws-gitlab-runner/tree/develop/examples/runner-default).
191194

@@ -218,6 +221,46 @@ module "runner" {
218221
}
219222
```
220223

224+
### Scenario: Multi-region deployment
225+
226+
Name clashes due to multi-region deployments for global AWS ressources create by this module (IAM, S3) can be avoided by including a distinguishing region specific prefix via the _cache_bucket_prefix_ string respectively via _name_iam_objects_ in the _overrides_ map. A simple example for this would be to set _region-specific-prefix_ to the AWS region the module is deployed to.
227+
228+
229+
230+
```hcl
231+
module "runner" {
232+
# https://registry.terraform.io/modules/npalm/gitlab-runner/aws/
233+
source = "npalm/gitlab-runner/aws"
234+
235+
aws_region = "eu-west-1"
236+
environment = "spot-runners"
237+
238+
ssh_public_key = local_file.public_ssh_key.content
239+
240+
vpc_id = module.vpc.vpc_id
241+
subnet_ids_gitlab_runner = module.vpc.private_subnets
242+
subnet_id_runners = element(module.vpc.private_subnets, 0)
243+
244+
runners_name = "docker-default"
245+
runners_gitlab_url = "https://gitlab.com"
246+
247+
gitlab_runner_registration_config = {
248+
registration_token = "my-token
249+
tag_list = "docker"
250+
description = "runner default"
251+
locked_to_project = "true"
252+
run_untagged = "false"
253+
maximum_timeout = "3600"
254+
}
255+
256+
overrides = {
257+
name_iam_objects = "<region-specific-prefix>-gitlab-runner-iam"
258+
}
259+
260+
cache_bucket_prefix = "<region-specific-prefix>"
261+
}
262+
```
263+
221264
## Examples
222265

223266
A few [examples](https://github.com/npalm/terraform-aws-gitlab-runner/tree/develop/examples/) are provided. Use the following steps to deploy. Ensure your AWS and Terraform environment is set up correctly. All commands below should be run from the `terraform-aws-gitlab-runner/examples/<example-dir>` directory.
@@ -396,7 +439,7 @@ terraform destroy
396439
| <a name="input_kms_key_id"></a> [kms\_key\_id](#input\_kms\_key\_id) | KMS key id to encrypted the CloudWatch logs. Ensure CloudWatch has access to the provided KMS key. | `string` | `""` | no |
397440
| <a name="input_log_group_name"></a> [log\_group\_name](#input\_log\_group\_name) | Option to override the default name (`environment`) of the log group, requires `enable_cloudwatch_logging = true`. | `string` | `null` | no |
398441
| <a name="input_metrics_autoscaling"></a> [metrics\_autoscaling](#input\_metrics\_autoscaling) | A list of metrics to collect. The allowed values are GroupDesiredCapacity, GroupInServiceCapacity, GroupPendingCapacity, GroupMinSize, GroupMaxSize, GroupInServiceInstances, GroupPendingInstances, GroupStandbyInstances, GroupStandbyCapacity, GroupTerminatingCapacity, GroupTerminatingInstances, GroupTotalCapacity, GroupTotalInstances. | `list(string)` | `null` | no |
399-
| <a name="input_overrides"></a> [overrides](#input\_overrides) | This maps provides the possibility to override some defaults. The following attributes are supported: `name_sg` overwrite the `Name` tag for all security groups created by this module. `name_runner_agent_instance` override the `Name` tag for the ec2 instance defined in the auto launch configuration. `name_docker_machine_runners` ovverrid the `Name` tag spot instances created by the runner agent. | `map(string)` | <pre>{<br> "name_docker_machine_runners": "",<br> "name_iam_objects": "",<br> "name_runner_agent_instance": "",<br> "name_sg": ""<br>}</pre> | no |
442+
| <a name="input_overrides"></a> [overrides](#input\_overrides) | This map provides the possibility to override some defaults. <br>The following attributes are supported: <br> * `name_sg` set the name prefix and overwrite the `Name` tag for all security groups created by this module. <br> * `name_runner_agent_instance` set the name prefix and override the `Name` tag for the EC2 gitlab runner instances defined in the auto launch configuration. <br> * `name_docker_machine_runners` override the `Name` tag of EC2 instances created by the runner agent. <br> * `name_iam_objects` set the name prefix of all AWS IAM resources created by this module. | `map(string)` | <pre>{<br> "name_docker_machine_runners": "",<br> "name_iam_objects": "",<br> "name_runner_agent_instance": "",<br> "name_sg": ""<br>}</pre> | no |
400443
| <a name="input_permissions_boundary"></a> [permissions\_boundary](#input\_permissions\_boundary) | Name of permissions boundary policy to attach to AWS IAM roles | `string` | `""` | no |
401444
| <a name="input_role_tags"></a> [role\_tags](#input\_role\_tags) | Map of tags that will be added to the role created. Useful for tag based authorization. | `map(string)` | `{}` | no |
402445
| <a name="input_runner_agent_uses_private_address"></a> [runner\_agent\_uses\_private\_address](#input\_runner\_agent\_uses\_private\_address) | Restrict the runner agent to the use of a private IP address. If `runner_agent_uses_private_address` is set to `false` it will override the `runners_use_private_address` for the agent. | `bool` | `true` | no |
+84
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
# Example - Spot Runner - Multi-region
2+
3+
In this scenario, we create multiple runner agents similarly to the _runner-public_ example but in different regions. Runners can be created with different configuration by instantiating the module multiple times. Runners will scale automatically based on configuration. The S3 cache can be shared cross runners by managing the cache outside the module.
4+
5+
![runners-cache](https://github.com/npalm/assets/raw/master/images/terraform-aws-gitlab-runner/runner-cache.png)
6+
7+
This examples shows:
8+
9+
- Multi-region deployment.
10+
- Usages of public subnets.
11+
- Usages of multiple runner instances sharing a common cache.
12+
- Overrides for tag naming.
13+
- Registration via GitLab token.
14+
- Auto scaling using `docker+machine` executor.
15+
16+
Note that global AWS resources like IAM policies and S3 buckets must be unique across regions.
17+
To duplicate the Gitlab runner deployment to multiple regions, we therefore have to use the name overrides for the IAM resources (_overrides.name_iam_objects_) respectively for the S3 cache bucket (_cache_bucket_prefix_) in the modules _runner_main_region_ and _runner_alternate_region_.
18+
19+
```hcl
20+
# examples/runner-multi-region/main.tf
21+
# ...
22+
23+
module "runner_main_region" {
24+
# ...
25+
26+
overrides = {
27+
name_sg = "my-security-group"
28+
name_runner_agent_instance = "my-runner-agent"
29+
name_docker_machine_runners = "my-runners-dm"
30+
name_iam_objects = local.name_iam_objects_main_region # <--
31+
}
32+
33+
# ...
34+
35+
cache_bucket_prefix = local.cache_bucket_prefix_main_region # <--
36+
cache_bucket_name_include_account_id = false
37+
}
38+
39+
# ...
40+
41+
module "runner_alternate_region" {
42+
# ...
43+
44+
overrides = {
45+
name_sg = "my-security-group"
46+
name_runner_agent_instance = "my-runner-agent"
47+
name_docker_machine_runners = "my-runners-dm"
48+
name_iam_objects = local.name_iam_objects_alternate_region # <--
49+
}
50+
51+
# ...
52+
53+
cache_bucket_prefix = local.cache_bucket_prefix_alternate_region # <--
54+
cache_bucket_name_include_account_id = false
55+
}
56+
```
57+
58+
## Prerequisite
59+
60+
The Terraform version is managed using [tfenv](https://github.com/Zordrak/tfenv). If you are not using `tfenv` please check `.terraform-version` for the tested version.
61+
62+
## Providers
63+
64+
| Name | Version |
65+
|------|---------|
66+
| aws | 2.56 |
67+
| local | 1.4 |
68+
| null | 2.1.2 |
69+
| tls | 2.1.1 |
70+
71+
## Inputs
72+
73+
| Name | Description | Type | Default | Required |
74+
|------|-------------|------|---------|:-----:|
75+
| aws\_main\_region | Main AWS region to deploy to. | `string` | `"eu-west-1"` | no |
76+
| aws\_alternate\_region | Main AWS region to deploy to. | `string` | `"eu-central-1"` | no |
77+
| environment | A name that identifies the environment, will used as prefix and for tagging. | `string` | `"runner-public"` | no |
78+
| gitlab\_url | URL of the gitlab instance to connect to. | `string` | `"https://gitlab.com"` | no |
79+
| registration\_token | n/a | `any` | n/a | yes |
80+
| runner\_name | Name of the runner, will be used in the runner config.toml | `string` | `"public-auto"` | no |
81+
82+
## Outputs
83+
84+
No output.
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
# Example - Spot Runner - Public subnets
2+
3+
In this scenario the multiple runner agents can be created with different configuration by instantiating the module multiple times. Runners will scale automatically based on configuration. The S3 cache can be shared cross runners by managing the cache outside the module.
4+
5+
![runners-cache](https://github.com/npalm/assets/raw/master/images/terraform-aws-gitlab-runner/runner-cache.png)
6+
7+
This examples shows:
8+
- Usages of public subnets.
9+
- Usages of multiple runner instances sharing a common cache.
10+
- Overrides for tag naming.
11+
- Registration via GitLab token.
12+
- Auto scaling using `docker+machine` executor.
13+
- Register runner as [protected](https://docs.gitlab.com/ee/ci/runners/configure_runners.html#prevent-runners-from-revealing-sensitive-information).
14+
15+
16+
## Prerequisite
17+
18+
The Terraform version is managed using [tfenv](https://github.com/Zordrak/tfenv). If you are not using `tfenv` please check `.terraform-version` for the tested version.
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
## Providers
2+
3+
| Name | Version |
4+
|------|---------|
5+
| aws | 2.56 |
6+
| local | 1.4 |
7+
| null | 2.1.2 |
8+
| tls | 2.1.1 |
9+
10+
## Inputs
11+
12+
| Name | Description | Type | Default | Required |
13+
|------|-------------|------|---------|:-----:|
14+
| aws\_region | AWS region. | `string` | `"eu-west-1"` | no |
15+
| environment | A name that identifies the environment, will used as prefix and for tagging. | `string` | `"runner-public"` | no |
16+
| gitlab\_url | URL of the gitlab instance to connect to. | `string` | `"https://gitlab.com"` | no |
17+
| private\_ssh\_key\_filename | n/a | `string` | `"generated/id_rsa"` | no |
18+
| public\_ssh\_key\_filename | n/a | `string` | `"generated/id_rsa.pub"` | no |
19+
| registration\_token | n/a | `any` | n/a | yes |
20+
| runner\_name | Name of the runner, will be used in the runner config.toml | `string` | `"public-auto"` | no |
21+
22+
## Outputs
23+
24+
No output.
25+
+6
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
locals {
2+
name_iam_objects_main_region = "${var.aws_main_region}-my-runner-iam-objects"
3+
cache_bucket_prefix_main_region = "${var.aws_main_region}-my-runner-cache-bucket"
4+
name_iam_objects_alternate_region = "${var.aws_alternate_region}-my-runner-iam-objects"
5+
cache_bucket_prefix_alternate_region = "${var.aws_alternate_region}-my-runner-cache-bucket"
6+
}

examples/runner-multi-region/main.tf

+131
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,131 @@
1+
data "aws_availability_zones" "available_main_region" {
2+
state = "available"
3+
}
4+
5+
module "vpc_main_region" {
6+
source = "terraform-aws-modules/vpc/aws"
7+
version = "2.70"
8+
9+
name = "vpc-${var.environment}"
10+
cidr = "10.1.0.0/16"
11+
12+
azs = [data.aws_availability_zones.available_main_region.names[0]]
13+
public_subnets = ["10.1.101.0/24"]
14+
15+
map_public_ip_on_launch = "false"
16+
17+
tags = {
18+
Environment = var.environment
19+
}
20+
}
21+
22+
module "runner_main_region" {
23+
source = "../../"
24+
25+
aws_region = var.aws_main_region
26+
environment = var.environment
27+
28+
runners_use_private_address = false
29+
30+
vpc_id = module.vpc_main_region.vpc_id
31+
subnet_id = element(module.vpc_main_region.public_subnets, 0)
32+
33+
docker_machine_spot_price_bid = "on-demand-price"
34+
35+
runners_name = var.runner_name
36+
runners_gitlab_url = var.gitlab_url
37+
runners_environment_vars = ["KEY=Value", "FOO=bar"]
38+
39+
runners_privileged = "false"
40+
runners_additional_volumes = ["/var/run/docker.sock:/var/run/docker.sock"]
41+
42+
gitlab_runner_registration_config = {
43+
registration_token = var.registration_token
44+
tag_list = "docker_spot_runner"
45+
description = "runner public - auto"
46+
locked_to_project = "false"
47+
run_untagged = "false"
48+
maximum_timeout = "3600"
49+
}
50+
51+
overrides = {
52+
name_sg = "my-security-group"
53+
name_runner_agent_instance = "my-runner-agent"
54+
name_docker_machine_runners = "my-runners-dm"
55+
name_iam_objects = local.name_iam_objects_main_region
56+
}
57+
58+
cache_shared = "true"
59+
60+
cache_bucket_prefix = local.cache_bucket_prefix_main_region
61+
cache_bucket_name_include_account_id = false
62+
}
63+
64+
module "vpc_alternate_region" {
65+
providers = {
66+
aws = aws.alternate_region
67+
}
68+
69+
source = "terraform-aws-modules/vpc/aws"
70+
version = "2.70"
71+
72+
name = "vpc-${var.environment}"
73+
cidr = "10.1.0.0/16"
74+
75+
# Have to construct the zone names here because the data source always uses the main region
76+
azs = ["${var.aws_alternate_region}a", "${var.aws_alternate_region}b", "${var.aws_alternate_region}c"]
77+
public_subnets = ["10.1.101.0/24"]
78+
79+
map_public_ip_on_launch = "false"
80+
81+
tags = {
82+
Environment = var.environment
83+
}
84+
}
85+
86+
module "runner_alternate_region" {
87+
providers = {
88+
aws = aws.alternate_region
89+
}
90+
91+
source = "../../"
92+
93+
aws_region = var.aws_alternate_region
94+
environment = var.environment
95+
96+
runners_use_private_address = false
97+
98+
vpc_id = module.vpc_alternate_region.vpc_id
99+
subnet_id = element(module.vpc_alternate_region.public_subnets, 0)
100+
101+
docker_machine_spot_price_bid = "on-demand-price"
102+
103+
runners_name = var.runner_name
104+
runners_gitlab_url = var.gitlab_url
105+
runners_environment_vars = ["KEY=Value", "FOO=bar"]
106+
107+
runners_privileged = "false"
108+
runners_additional_volumes = ["/var/run/docker.sock:/var/run/docker.sock"]
109+
110+
gitlab_runner_registration_config = {
111+
registration_token = var.registration_token
112+
tag_list = "docker_spot_runner"
113+
description = "runner public - auto"
114+
locked_to_project = "true"
115+
run_untagged = "false"
116+
maximum_timeout = "3600"
117+
access_level = "ref_protected"
118+
}
119+
120+
overrides = {
121+
name_sg = "my-security-group"
122+
name_runner_agent_instance = "my-runner-agent"
123+
name_docker_machine_runners = "my-runners-dm"
124+
name_iam_objects = local.name_iam_objects_alternate_region
125+
}
126+
127+
cache_shared = "true"
128+
129+
cache_bucket_prefix = local.cache_bucket_prefix_alternate_region
130+
cache_bucket_name_include_account_id = false
131+
}
+16
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
provider "aws" {
2+
region = var.aws_main_region
3+
}
4+
5+
provider "aws" {
6+
region = var.aws_alternate_region
7+
alias = "alternate_region"
8+
}
9+
10+
provider "local" {}
11+
12+
provider "null" {}
13+
14+
provider "tls" {}
15+
16+
provider "random" {}

0 commit comments

Comments
 (0)