[installer-preview] add werft flags for triggering self-hosted preview

[nandajavarma]

Description

This PR adds a werft flag to create self-hosted preview environments from PRs. The idea is that, users will be able to make the following comment to the PR and a werft job will get triggered from the build job (with status written back to Github) creating a preview environment:

/werft run with-sh-preview

The above command will do the following:

1. Publish the branch to KOTS
2. Create a replicated license for the KOTS channel created in the previous step
3. Start a self-hosted preview on a newly created k3s cluster(we are planning to work on reducing waste here)
4. Delete the created license
5. The URL of the preview setup will be available under the Results tab of the werft job

If the user wants to test on cluster other than k3s, they can run:

werft run with-sh-preview cluster=aks # for Azure k8s cluster
werft run with-sh-preview cluster=eks # for AWS k8s cluster
werft run with-sh-preview cluster=gke # for GCP k8s cluster

The created cluster will be deleted in 10 hours.

Related Issue(s)

Fixes #7824

How to test

Currently, since this change is not merged with main, one cannot add comments to the PR to trigger the builds. Instead you will have to open a gitpod workspace from this PR and run:

werft run github -j .werft/build.yaml -a with-sh-preview=true -a cluster=k3s
# or
werft run github -j .werft/build.yaml -a with-sh-preview=true -a cluster=aks
# or
werft run github -j .werft/build.yaml -a with-sh-preview=true -a cluster=eks
# or
werft run github -j .werft/build.yaml -a with-sh-preview=true -a cluster=gke

Release Notes

NONE

Documentation

Werft options:

• /werft with-preview

[nandajavarma]

/werft run with-sh-preview cluster=k3s

👍 started the job as gitpod-build-nvn-sh-preview-flag.1
(with .werft/ from main)

[nandajavarma]

/werft run with-sh-preview cluster=k3s channel=unstable

👍 started the job as gitpod-build-nvn-sh-preview-flag.2
(with .werft/ from main)

[meysholdt]

hey @nandajavarma ! Thx for this PR - can you give more context on the goal behind it? is this a new way to run automated tests or do you intend to improve dev loops? I'm asking to understand if this is related to https://www.notion.so/gitpod/Gitpod-Preview-Environments-65606379c3724734b28115cc3e48a13c :)

[nandajavarma]

Hey @meysholdt ! Yes! it is related to the RFC, yes. The goal of this PR is to do 2 things: 1) being able to start-up preview setups from PRs to improve the dev velocity especially for the self-hosted team and 2) workflow to create and delete replicated licence that can be used to try out newer changes.

Once this PR is merged, folks should be able to comment /werft run with-sh-preview to create self-hosted preview setup on k3s, which will get cleaned up after 10 hours. Goal was to get it merged before the release testing starts. Basically a big part of #7824 will be addressed by this. Let me know if that sheds a bit more light on the PR.

[werft-gitpod-dev-com]

started the job as gitpod-build-nvn-sh-preview-flag.12 because the annotations in the pull request description changed
(with .werft/ from main)

[adrienthebo]

/werft run with-sh-preview cluster=k3s

👍 started the job as gitpod-build-nvn-sh-preview-flag.14
(with .werft/ from main)

[nandajavarma]

@adrienthebo since this is not a part of main yet, the flags as comment doesn't work yet. You will have to follow the How to test section to trigger them

[adrienthebo]

I believe I've found an issue - though it might be outside of the juristiction of this pull request. Error is as follows (from gitpod-self-hosted-installer-tests-head.77)

k3s

Error: Activated service account credentials for: [[email protected]]
Fetching cluster endpoint and auth data.
ERROR: (gcloud.container.clusters.get-credentials) ResponseError: code=404, message=Not found: projects/sh-automated-tests/zones/europe-west1-d/clusters/gp-nvn-sh-prev-k3s.
No cluster named 'gp-nvn-sh-prev-k3s' in sh-automated-tests.
Activated service account credentials for: [[email protected]]
CommandException: No URLs matched: gs://nightly-tests/tf-state/nvn-sh-prev-k3s-kubeconfig
╷
│ Error: Error, failed to create instance sql-nvn-sh-prev-k3s: googleapi: Error 409: The Cloud SQL instance already exists. When you delete an instance, you can't reuse the name of the deleted instance until one week from the deletion date., instanceAlreadyExists
│
│   with module.k3s.google_sql_database_instance.gitpod,
│   on ../infra/modules/k3s/main.tf line 175, in resource "google_sql_database_instance" "gitpod":
│  175: resource "google_sql_database_instance" "gitpod" {
│
╵
make: *** [Makefile:129: k3s-standard-cluster] Error 1

Trying to send slack alert
tracing warning: No slice span by name create-gcp-infra

aks

AKS is failing, again looks unrelated

Error: ╷
│ Error: creating Cluster: (Managed Cluster Name "gitpod-test-nor-primary-nvnshprevaks" / Resource Group "gitpod-test-nor-nvnshprevaks"): containerservice.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="UnsupportedAliasMinorVersion" Message="1.21 is not a supported Kubernetes version. To see our supported AKS version list please visit https://docs.microsoft.com/azure/aks/supported-kubernetes-versions?tabs=azure-cli for more details."
│
│   with module.aks.azurerm_kubernetes_cluster.k8s,
│   on ../infra/modules/aks/kubernetes.tf line 17, in resource "azurerm_kubernetes_cluster" "k8s":
│   17: resource "azurerm_kubernetes_cluster" "k8s" {
│
╵
make: *** [Makefile:97: aks-standard-cluster] Error 1

Trying to send slack alert
tracing warning: No slice span by name create-azure-infra

eks

Error: ╷
│ Error: expected length of name_prefix to be in the range (1 - 38), got service-nvn-sh-prev-eks-eks-node-group-
│
│   with module.eks.module.eks.module.eks_managed_node_group["Services"].aws_iam_role.this[0],
│   on .terraform/modules/eks.eks/modules/eks-managed-node-group/main.tf line 426, in resource "aws_iam_role" "this":
│  426:   name_prefix = var.iam_role_use_name_prefix ? "${local.iam_role_name}-" : null
│
╵
make: *** [Makefile:89: eks-standard-cluster] Error 1

Trying to send slack alert
tracing warning: No slice span by name create-aws-infra

GKE

Error: ╷
│ Error: Error, failed to create instance sql-gp-nvn-sh-prev-gke-dBnz: googleapi: Error 400: Invalid request: instance name (sql-gp-nvn-sh-prev-gke-dBnz)., invalid
│
│   with module.gke.google_sql_database_instance.gitpod[0],
│   on ../infra/modules/gke/main.tf line 160, in resource "google_sql_database_instance" "gitpod":
│  160: resource "google_sql_database_instance" "gitpod" {
│
╵
make: *** [Makefile:68: gke-standard-cluster] Error 1

Trying to send slack alert
tracing warning: No slice span by name create-gcp-infra

[adrienthebo]

As noted in an earlier comment, we have test failures on all platforms that prevent this from reliably working out of the gate. I've opened https://github.com/gitpod-io/gitpod/pull/12321/files targeting this branch with the fixups, let's give those fixes a try (or equivalent changes) and see if that improves the reliability. Looking forward to getting this merged!

Dismissing blocker as changes have landed so others can potentially land a review

[nandajavarma]

@adrienthebo All the dependent PRs has been reviewed and merged. Can we try and merge this one ?

[mrsimonemms]

/hold

Looks good. I have one question - if the answer is "no", then feel free to remove the hold and get this merged

[mrsimonemms]

Is there any way of making this a function that can be invoked? This is a repetition of what's in .github/workflows/delete-kots-channel.yml

[nandajavarma]

we ideally could, but I don't know if it is much of an optimization considering it means we will have to pass the API token, license name and in the very near future appname since I have an issue open to use gitpod-dev as published channel. Do you mind if I think about this better when I deal with that issue?

[mrsimonemms]

Absolutely - let's get it merged

[mrsimonemms]

/unhold