Skip to content

Commit 3faf81b

Browse files
csantanaprcsantanapr
authored
restore irsa multi-cluster hub-spoke example (#62)
Signed-off-by: Carlos Santana <[email protected]>
1 parent e2b4968 commit 3faf81b

File tree

18 files changed

+1172
-0
lines changed

18 files changed

+1172
-0
lines changed

‎argocd/iac/terraform/examples/eks/multi-cluster/hub-spoke-irsa/README.md

Lines changed: 156 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,156 @@
1+
# Multi-Cluster centralized hub-spoke topology
2+
3+
This example uses IRSA
4+
5+
This example deploys ArgoCD on the Hub cluster (ie. management/control-plane cluster).
6+
The spoke clusters are registered as remote clusters in the Hub Cluster's ArgoCD
7+
The ArgoCD on the Hub Cluster deploy addons and workloads to the spoke clusters
8+
9+
Each spoke cluster gets deployed an app of apps ArgoCD Application with the name `workloads-${env}`
10+
11+
## Prerequisites
12+
Before you begin, make sure you have the following command line tools installed:
13+
- git
14+
- terraform
15+
- kubectl
16+
- argocd
17+
18+
## Fork the Git Repositories
19+
20+
### Fork the Addon GitOps Repo
21+
1. Fork the git repository for addons [here](https://github.com/gitops-bridge-dev/gitops-bridge-argocd-control-plane-template).
22+
2. Update the following environment variables to point to your fork by changing the default values:
23+
```shell
24+
export TF_VAR_gitops_addons_org=https://github.com/gitops-bridge-dev
25+
export TF_VAR_gitops_addons_repo=gitops-bridge-argocd-control-plane-template
26+
```
27+
28+
## Deploy the Hub EKS Cluster
29+
Change Director to `hub`
30+
```shell
31+
cd hub
32+
```
33+
Initialize Terraform and deploy the EKS cluster:
34+
```shell
35+
terraform init
36+
terraform apply -auto-approve
37+
```
38+
Retrieve `kubectl` config, then execute the output command:
39+
```shell
40+
terraform output -raw configure_kubectl
41+
```
42+
43+
### Monitor GitOps Progress for Addons
44+
Wait until **all** the ArgoCD applications' `HEALTH STATUS` is `Healthy`. Use Crl+C to exit the `watch` command
45+
```shell
46+
watch kubectl get applications -n argocd
47+
```
48+
49+
## Access ArgoCD on Hub Cluster
50+
Access ArgoCD's UI, run the command from the output:
51+
```shell
52+
terraform output -raw access_argocd
53+
```
54+
55+
## Verify that ArgoCD Service Accouts has the annotation for IRSA
56+
```shell
57+
kubectl get sa -n argocd argocd-application-controller -o json | jq '.metadata.annotations."eks.amazonaws.com/role-arn"'
58+
kubectl get sa -n argocd argocd-server -o json | jq '.metadata.annotations."eks.amazonaws.com/role-arn"'
59+
```
60+
The output should match the `arn` for the IAM Role that will assume the IAM Role in spoke/remote clusters
61+
```text
62+
"arn:aws:iam::0123456789:role/hub-spoke-control-plane-argocd-hub"
63+
```
64+
65+
## Deploy the Spoke EKS Cluster
66+
Initialize Terraform and deploy the EKS clusters:
67+
```shell
68+
cd ../spokes
69+
./deploy.sh dev
70+
./deploy.sh staging
71+
./deploy.sh prod
72+
```
73+
Each environment uses a Terraform workspace
74+
75+
To access Terraform output run the following commands for the particular environment
76+
```shell
77+
terraform workspace select dev
78+
terraform output
79+
```
80+
```shell
81+
terraform workspace select staging
82+
terraform output
83+
```
84+
```shell
85+
terraform workspace select prod
86+
terraform output
87+
```
88+
89+
Retrieve `kubectl` config, then execute the output command:
90+
```shell
91+
terraform output -raw configure_kubectl
92+
```
93+
94+
### Verify ArgoCD Cluster Secret for Spoke has the correct IAM Role to be assume by Hub Cluster
95+
```shell
96+
kubectl get secret -n argocd hub-spoke-dev --template='{{index .data.config | base64decode}}'
97+
```
98+
Do the same for the other cluster replaced `dev` in `hub-spoke-dev`
99+
The output have a section `awsAuthConfig` with the `clusterName` and the `roleARN` that has write access to the spoke cluster
100+
```json
101+
{
102+
"tlsClientConfig": {
103+
"insecure": false,
104+
"caData" : "LS0tL...."
105+
},
106+
"awsAuthConfig" : {
107+
"clusterName": "hub-spoke-dev",
108+
"roleARN": "arn:aws:iam::0123456789:role/hub-spoke-dev-argocd-spoke"
109+
}
110+
}
111+
```
112+
113+
114+
### Verify the Addons on Spoke Clusters
115+
Verify that the addons are ready:
116+
```shell
117+
kubectl get deployment -n kube-system \
118+
metrics-server
119+
```
120+
121+
122+
### Monitor GitOps Progress for Workloads from Hub Cluster (run on Hub Cluster context)
123+
Watch until **all* the Workloads ArgoCD Applications are `Healthy`
124+
```shell
125+
watch kubectl get -n argocd applications
126+
```
127+
Wait until the ArgoCD Applications `HEALTH STATUS` is `Healthy`. Crl+C to exit the `watch` command
128+
129+
130+
### Verify the Application
131+
Verify that the application configuration is present and the pod is running:
132+
```shell
133+
kubectl get all -n workload
134+
```
135+
136+
### Container Metrics
137+
Check the application's CPU and memory metrics:
138+
```shell
139+
kubectl top pods -n workload
140+
```
141+
142+
## Destroy the Spoke EKS Clusters
143+
To tear down all the resources and the EKS cluster, run the following command:
144+
```shell
145+
./destroy.sh dev
146+
./destroy.sh staging
147+
./destroy.sh prod
148+
```
149+
150+
## Destroy the Hub EKS Clusters
151+
To tear down all the resources and the EKS cluster, run the following command:
152+
Destroy Hub Clusters
153+
```shell
154+
cd ../hub
155+
./destroy.sh
156+
```

‎argocd/iac/terraform/examples/eks/multi-cluster/hub-spoke-irsa/hub/bootstrap/addons.yaml

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
apiVersion: argoproj.io/v1alpha1
2+
kind: ApplicationSet
3+
metadata:
4+
name: cluster-addons
5+
namespace: argocd
6+
spec:
7+
syncPolicy:
8+
preserveResourcesOnDeletion: true
9+
generators:
10+
- clusters:
11+
selector:
12+
matchExpressions:
13+
- key: akuity.io/argo-cd-cluster-name
14+
operator: NotIn
15+
values: [in-cluster]
16+
template:
17+
metadata:
18+
name: cluster-addons
19+
spec:
20+
project: default
21+
source:
22+
repoURL: '{{metadata.annotations.addons_repo_url}}'
23+
path: '{{metadata.annotations.addons_repo_basepath}}{{metadata.annotations.addons_repo_path}}'
24+
targetRevision: '{{metadata.annotations.addons_repo_revision}}'
25+
directory:
26+
recurse: true
27+
exclude: exclude/*
28+
destination:
29+
namespace: 'argocd'
30+
name: '{{name}}'
31+
syncPolicy:
32+
automated: {}

‎argocd/iac/terraform/examples/eks/multi-cluster/hub-spoke-irsa/hub/bootstrap/workloads.yaml

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
apiVersion: argoproj.io/v1alpha1
2+
kind: ApplicationSet
3+
metadata:
4+
name: workloads
5+
namespace: argocd
6+
spec:
7+
syncPolicy:
8+
preserveResourcesOnDeletion: false
9+
generators:
10+
- clusters:
11+
selector:
12+
matchExpressions:
13+
- key: akuity.io/argo-cd-cluster-name
14+
operator: NotIn
15+
values: [in-cluster]
16+
- key: environment
17+
operator: NotIn
18+
values: [control-plane]
19+
template:
20+
metadata:
21+
name: 'workload-{{metadata.labels.environment}}'
22+
spec:
23+
project: default
24+
source:
25+
repoURL: '{{metadata.annotations.workload_repo_url}}'
26+
path: '{{metadata.annotations.workload_repo_basepath}}{{metadata.annotations.workload_repo_path}}'
27+
targetRevision: '{{metadata.annotations.workload_repo_revision}}'
28+
destination:
29+
namespace: 'workload'
30+
name: '{{name}}'
31+
syncPolicy:
32+
automated: {}
33+
syncOptions:
34+
- CreateNamespace=true

‎argocd/iac/terraform/examples/eks/multi-cluster/hub-spoke-irsa/hub/destroy.sh

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
#!/bin/bash
2+
3+
set -uo pipefail
4+
5+
SCRIPTDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
6+
ROOTDIR="$(cd ${SCRIPTDIR}/../..; pwd )"
7+
[[ -n "${DEBUG:-}" ]] && set -x
8+
9+
# Delete the Ingress/SVC before removing the addons
10+
TMPFILE=$(mktemp)
11+
terraform -chdir=$SCRIPTDIR output -raw configure_kubectl > "$TMPFILE"
12+
# check if TMPFILE contains the string "No outputs found"
13+
if [[ ! $(cat $TMPFILE) == *"No outputs found"* ]]; then
14+
source "$TMPFILE"
15+
kubectl delete -n argocd applicationset workloads
16+
kubectl delete -n argocd applicationset cluster-addons
17+
kubectl delete -n argocd applicationset addons-argocd
18+
kubectl delete -n argocd svc argo-cd-argocd-server
19+
fi
20+
21+
terraform destroy -target="module.gitops_bridge_bootstrap" -auto-approve
22+
terraform destroy -target="module.eks_blueprints_addons" -auto-approve
23+
terraform destroy -target="module.eks" -auto-approve
24+
terraform destroy -target="module.vpc" -auto-approve
25+
terraform destroy -auto-approve

0 commit comments

Comments
 (0)