GitOps Bridge v2 poc

argocd/iac/terraform/examples/eks/single-cluster-v2/README.md

@@ -0,0 +1,161 @@
+# ArgoCD on Amazon EKS
+
+This pattern shows how to use the new stack platform concept
+
+This tutorial guides you through deploying an Amazon EKS cluster with addons configured via ArgoCD, employing the [GitOps Bridge Pattern](https://github.com/gitops-bridge-dev).
+
+
+## Prerequisites
+Before you begin, make sure you have the following command line tools installed:
+- git
+- terraform
+- kubectl
+- argocd
+
+## Fork the Git Repositories
+
+### Fork the Addon GitOps Repo
+1. Fork the git repository for addons [here](https://github.com/gitops-bridge-dev/gitops-bridge-argocd-control-plane-template).
+2. Update the following environment variables to point to your fork by changing the default values:
+```shell
+export TF_VAR_gitops_addons_org=https://github.com/gitops-bridge-dev
+export TF_VAR_gitops_addons_repo=gitops-bridge-argocd-control-plane-template
+```
+
+### Fork the Workloads GitOps Repo
+1. Fork the git repository for this pattern [here](https://github.com/gitops-bridge-dev/gitops-bridge)
+2. Update the following environment variables to point to your fork by changing the default values:
+```shell
+export TF_VAR_gitops_workload_org=https://github.com/gitops-bridge-dev
+export TF_VAR_gitops_workload_repo=gitops-bridge
+```
+
+## Deploy the EKS Cluster
+Initialize Terraform and deploy the EKS cluster:
+```shell
+terraform init
+terraform apply -auto-approve
+```
+Retrieve `kubectl` config, then execute the output command:
+```shell
+terraform output -raw configure_kubectl
+```
+
+Terraform will add GitOps Bridge Metadata to the ArgoCD secret.
+The annotations contain metadata for the addons' Helm charts and ArgoCD ApplicationSets.
+```shell
+kubectl get secret -n argocd -l argocd.argoproj.io/secret-type=cluster -o json | jq '.items[0].metadata.annotations'
+```
+The output looks like the following:
+```json
+{
+  "addons_repo_basepath": "",
+  "addons_repo_path": "bootstrap/control-plane/addons",
+  "addons_repo_revision": "main",
+  "addons_repo_url": "https://github.com/gitops-bridge-dev/gitops-bridge-argocd-control-plane-template",
+  "aws_account_id": "0123456789",
+  "aws_cluster_name": "getting-started-gitops",
+  "aws_load_balancer_controller_iam_role_arn": "arn:aws:iam::0123456789:role/alb-controller",
+  "aws_load_balancer_controller_namespace": "kube-system",
+  "aws_load_balancer_controller_service_account": "aws-load-balancer-controller-sa",
+  "aws_region": "us-west-2",
+  "aws_vpc_id": "vpc-001d3f00151bbb731",
+  "cluster_name": "in-cluster",
+  "environment": "dev",
+  "workload_repo_basepath": "argocd/iac/terraform/examples/eks/",
+  "workload_repo_path": "getting-started/k8s",
+  "workload_repo_revision": "main",
+  "workload_repo_url": "https://github.com/gitops-bridge-dev/gitops-bridge"
+}
+```
+The labels offer a straightforward way to enable or disable an addon in ArgoCD for the cluster.
+```shell
+kubectl get secret -n argocd -l argocd.argoproj.io/secret-type=cluster -o json | jq '.items[0].metadata.labels'
+```
+The output looks like the following:
+```json
+{
+  "aws_cluster_name": "getting-started-gitops",
+  "enable_argocd": "true",
+  "enable_aws_load_balancer_controller": "true",
+  "enable_metrics_server": "true",
+  "kubernetes_version": "1.28",
+}
+```
+
+## Deploy the Addons
+Bootstrap the addons using ArgoCD:
+```shell
+kubectl apply -f bootstrap/addons.yaml
+```
+
+### Monitor GitOps Progress for Addons
+Wait until all the ArgoCD applications' `HEALTH STATUS` is `Healthy`. Use Crl+C to exit the `watch` command
+```shell
+watch kubectl get applications -n argocd
+```
+
+### Verify the Addons
+Verify that the addons are ready:
+```shell
+kubectl get deployment -n kube-system \
+  aws-load-balancer-controller \
+  metrics-server
+```
+
+## Access ArgoCD
+Access ArgoCD's UI, run the command from the output:
+```shell
+terraform output -raw access_argocd
+```
+
+
+## Deploy the Workloads
+Deploy a sample application located in [k8s/game-2048.yaml](k8s/game-2048.yaml) using ArgoCD:
+```shell
+kubectl apply -f bootstrap/workloads.yaml
+```
+
+### Monitor GitOps Progress for Workloads
+Watch until the Workloads ArgoCD Application is `Healthy`
+```shell
+watch kubectl get -n argocd applications workloads
+```
+Wait until the ArgoCD Applications `HEALTH STATUS` is `Healthy`. Crl+C to exit the `watch` command
+
+### Verify the Application
+Verify that the application configuration is present and the pod is running:
+```shell
+kubectl get -n game-2048 deployments,service,ep,ingress
+```
+Wait until the Ingress/game-2048 `MESSAGE` column value is `Successfully reconciled`. Crl+C to exit the `watch` command
+```shell
+kubectl events -n game-2048 --for ingress/game-2048 --watch
+```
+
+
+
+### Access the Application using AWS Load Balancer
+Verify the application endpoint health using `curl`:
+```shell
+curl -I $(kubectl get -n game-2048 ingress game-2048 -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
+```
+The first line of the output should have `HTTP/1.1 200 OK`.
+
+Retrieve the ingress URL for the application, and access in the browser:
+```shell
+echo "Application URL: http://$(kubectl get -n game-2048 ingress game-2048 -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')"
+```
+
+
+### Container Metrics
+Check the application's CPU and memory metrics:
+```shell
+kubectl top pods -n game-2048
+```
+
+## Destroy the EKS Cluster
+To tear down all the resources and the EKS cluster, run the following command:
+```shell
+./destroy.sh
+```

argocd/iac/terraform/examples/eks/single-cluster-v2/argocd-initial-values.yaml

@@ -0,0 +1,8 @@
+global:
+  tolerations:
+  - key: "CriticalAddonsOnly"
+    operator: "Exists"
+controller:
+  env:
+    - name: ARGOCD_SYNC_WAVE_DELAY
+      value: '30'

argocd/iac/terraform/examples/eks/single-cluster-v2/bootstrap/addons.tpl.yaml

@@ -0,0 +1,58 @@
+apiVersion: argoproj.io/v1alpha1
+kind: ApplicationSet
+metadata:
+  name: cluster-addons
+  namespace: argocd
+spec:
+  syncPolicy:
+    preserveResourcesOnDeletion: true
+  goTemplate: true
+  goTemplateOptions:
+    - missingkey=error
+  generators:
+    - clusters: {}
+  template:
+    metadata:
+      name: cluster-addons
+    spec:
+      project: default
+      source:
+        repoURL: '{{.metadata.annotations.addons_repo_url}}'
+        path: '{{.metadata.annotations.addons_repo_basepath}}charts/gitops-bridge'
+        targetRevision: '{{.metadata.annotations.addons_repo_revision}}'
+        helm:
+          valuesObject:
+            repoURLValuesBasePath: '{{.metadata.annotations.addons_repo_basepath}}'
+            repoURLValuesRevision: '{{.metadata.annotations.addons_repo_revision}}'
+            repoURLGitBasePath: '{{.metadata.annotations.addons_repo_basepath}}stacks/'
+            repoURLGitRevision: '{{.metadata.annotations.addons_repo_revision}}'
+            useSelector: false
+            useStack: true
+            addons:
+%{ for key, value in addons ~}
+%{ if substr(key, 0, 7) == "enable_" && value == true ~}
+              ${replace(key, "enable_", "")}:
+                enabled: ${value}
+%{ endif ~}
+%{ endfor ~}
+          ignoreMissingValueFiles: true
+          valueFiles:
+            - '{{.metadata.annotations.addons_repo_basepath}}default/addons/gitops-bridge/values.yaml'
+            - '{{.metadata.annotations.addons_repo_basepath}}environments/{{.metadata.labels.environment}}/gitops-bridge/values.yaml'
+            - '{{.metadata.annotations.addons_repo_basepath}}clusters/{{.nameNormalized}}/addons/values.yaml'
+            - '{{.metadata.annotations.addons_repo_basepath}}tenants/{{.metadata.labels.tenant}}/default/addons/gitops-bridge/values.yaml'
+            - '{{.metadata.annotations.addons_repo_basepath}}tenants/{{.metadata.labels.tenant}}/environments/{{.metadata.labels.environment}}/gitops-bridge/values.yaml'
+            - '{{.metadata.annotations.addons_repo_basepath}}tenants/{{.metadata.labels.tenant}}/clusters/{{.nameNormalized}}/addons/values.yaml'
+      destination:
+        namespace: argocd
+        name: '{{.name}}'
+      syncPolicy:
+        automated:
+          selfHeal: true
+          allowEmpty: true
+          prune: false
+        retry:
+          limit: 100
+        syncOptions:
+          - CreateNamespace=true
+          - ServerSideApply=true  # Big CRDs.

argocd/iac/terraform/examples/eks/single-cluster-v2/bootstrap/addons.yaml

@@ -0,0 +1,51 @@
+apiVersion: argoproj.io/v1alpha1
+kind: ApplicationSet
+metadata:
+  name: cluster-addons
+  namespace: argocd
+spec:
+  syncPolicy:
+    preserveResourcesOnDeletion: true
+  goTemplate: true
+  goTemplateOptions:
+    - missingkey=error
+  generators:
+    - clusters: {}
+  template:
+    metadata:
+      name: cluster-addons
+    spec:
+      project: default
+      source:
+        repoURL: '{{.metadata.annotations.addons_repo_url}}'
+        path: '{{.metadata.annotations.addons_repo_basepath}}charts/gitops-bridge'
+        targetRevision: '{{.metadata.annotations.addons_repo_revision}}'
+        helm:
+          valuesObject:
+            repoURLValuesBasePath: '{{.metadata.annotations.addons_repo_basepath}}'
+            repoURLValuesRevision: '{{.metadata.annotations.addons_repo_revision}}'
+            repoURLGitBasePath: '{{.metadata.annotations.addons_repo_basepath}}stacks/'
+            repoURLGitRevision: '{{.metadata.annotations.addons_repo_revision}}'
+            useSelector: true
+            useStack: true
+          ignoreMissingValueFiles: true
+          valueFiles:
+            - '{{.metadata.annotations.addons_repo_basepath}}default/addons/gitops-bridge/values.yaml'
+            - '{{.metadata.annotations.addons_repo_basepath}}environments/{{.metadata.labels.environment}}/gitops-bridge/values.yaml'
+            - '{{.metadata.annotations.addons_repo_basepath}}clusters/{{.nameNormalized}}/addons/values.yaml'
+            - '{{.metadata.annotations.addons_repo_basepath}}tenants/{{.metadata.labels.tenant}}/default/addons/gitops-bridge/values.yaml'
+            - '{{.metadata.annotations.addons_repo_basepath}}tenants/{{.metadata.labels.tenant}}/environments/{{.metadata.labels.environment}}/gitops-bridge/values.yaml'
+            - '{{.metadata.annotations.addons_repo_basepath}}tenants/{{.metadata.labels.tenant}}/clusters/{{.nameNormalized}}/addons/values.yaml'
+      destination:
+        namespace: argocd
+        name: '{{.name}}'
+      syncPolicy:
+        automated:
+          selfHeal: true
+          allowEmpty: true
+          prune: false
+        retry:
+          limit: 100
+        syncOptions:
+          - CreateNamespace=true
+          - ServerSideApply=true  # Big CRDs.

argocd/iac/terraform/examples/eks/single-cluster-v2/bootstrap/workloads.yaml

@@ -0,0 +1,35 @@
+apiVersion: argoproj.io/v1alpha1
+kind: ApplicationSet
+metadata:
+  name: workloads
+  namespace: argocd
+spec:
+  syncPolicy:
+    preserveResourcesOnDeletion: false
+  generators:
+    - clusters: {}
+  template:
+    metadata:
+      name: workloads
+      finalizers:
+        # This finalizer is for demo purposes, in production remove apps using argocd CLI "argocd app delete workload --cascade"
+        # When you invoke argocd app delete with --cascade, the finalizer is added automatically.
+        - resources-finalizer.argocd.argoproj.io
+    spec:
+      project: default
+      source:
+        repoURL: '{{metadata.annotations.workload_repo_url}}'
+        path: '{{metadata.annotations.workload_repo_basepath}}'
+        targetRevision: '{{metadata.annotations.workload_repo_revision}}'
+      destination:
+        name: '{{name}}'
+      syncPolicy:
+        automated:
+          selfHeal: true
+          allowEmpty: true
+          prune: false
+        retry:
+          limit: 100
+        syncOptions:
+          - CreateNamespace=true
+          - ServerSideApply=true  # Big CRDs.

argocd/iac/terraform/examples/eks/single-cluster-v2/destroy.sh

@@ -0,0 +1,78 @@
+#!/bin/bash
+
+set -uo pipefail
+
+SCRIPTDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+ROOTDIR="$(cd ${SCRIPTDIR}/../..; pwd )"
+[[ -n "${DEBUG:-}" ]] && set -x
+
+scale_down_karpenter_nodes() {
+  # Get all nodes with the label karpenter.sh/registered=true
+  nodes=$(kubectl get nodes -l karpenter.sh/registered=true -o jsonpath='{.items[*].metadata.name}')
+
+  # Iterate over each node
+  for node in $nodes; do
+    # Get all pods running on the current node
+    pods=$(kubectl get pods --all-namespaces --field-selector spec.nodeName=$node -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{"\n"}{end}')
+
+    # Iterate over each pod
+    while IFS= read -r pod; do
+      namespace=$(echo $pod | awk '{print $1}')
+      pod_name=$(echo $pod | awk '{print $2}')
+
+      # Get the owner references of the pod
+      owner_refs=$(kubectl get pod $pod_name -n $namespace -o jsonpath='{.metadata.ownerReferences[*]}')
+
+      # Check if the owner is a ReplicaSet (which is part of a deployment) or a StatefulSet and scale down
+      if echo $owner_refs | grep -q "ReplicaSet"; then
+      replicaset_name=$(kubectl get pod $pod_name -n $namespace -o jsonpath='{.metadata.ownerReferences[?(@.kind=="ReplicaSet")].name}')
+      deployment_name=$(kubectl get replicaset $replicaset_name -n $namespace -o jsonpath='{.metadata.ownerReferences[?(@.kind=="Deployment")].name}')
+      if [[ $(kubectl get deployment $deployment_name -n $namespace -o jsonpath='{.spec.replicas}') -gt 0 ]]; then
+        echo kubectl scale deployment $deployment_name -n $namespace --replicas=0
+        kubectl scale deployment $deployment_name -n $namespace --replicas=0
+      fi
+      elif echo $owner_refs | grep -q "StatefulSet"; then
+      statefulset_name=$(kubectl get pod $pod_name -n $namespace -o jsonpath='{.metadata.ownerReferences[?(@.kind=="StatefulSet")].name}')
+      if [[ $(kubectl get statefulset $statefulset_name -n $namespace -o jsonpath='{.spec.replicas}') -gt 0 ]]; then
+        echo kubectl scale statefulset $statefulset_name -n $namespace --replicas=0
+        kubectl scale statefulset $statefulset_name -n $namespace --replicas=0
+      fi
+      fi
+    done <<< "$pods"
+  done
+
+  # Loop through each node and delete it
+  for node in $nodes; do
+      echo "Deleting node: $node"
+      kubectl delete node $node
+  done
+  # do a final check to make sure the nodes are gone, loop sleep 60 in between checks
+  nodes=$(kubectl get nodes -l karpenter.sh/registered=true -o jsonpath='{.items[*].metadata.name}')
+  while [[ ! -z $nodes ]]; do
+    echo "Waiting for nodes to be deleted: $nodes"
+    sleep 60
+    nodes=$(kubectl get nodes -l karpenter.sh/registered=true -o jsonpath='{.items[*].metadata.name}')
+  done
+  sleep 60
+
+
+}
+
+# Delete the Ingress/SVC before removing the addons
+TMPFILE=$(mktemp)
+terraform -chdir=$SCRIPTDIR output -raw configure_kubectl > "$TMPFILE"
+# check if TMPFILE contains the string "No outputs found"
+if [[ ! $(cat $TMPFILE) == *"No outputs found"* ]]; then
+  source "$TMPFILE"
+  scale_down_karpenter_nodes
+  kubectl delete ing -A --all
+  # delete all the kuberneters service of type LoadBalancer, without using jq
+  kubectl get svc --all-namespaces -o json | grep -E '"type": "LoadBalancer"' | awk '{print "kubectl delete svc " $1 " -n " $2}' | bash
+  sleep 60
+fi
+
+terraform destroy -target="module.gitops_bridge_bootstrap" -auto-approve
+terraform destroy -target="module.eks_blueprints_addons" -auto-approve
+terraform destroy -target="module.eks" -auto-approve
+terraform destroy -target="module.vpc" -auto-approve
+terraform destroy -auto-approve