[Preview env] TF Workflow #12981

vulkoingim · 2022-09-15T06:09:29Z

Description

Introduce functions + set of scripts to manage TF
Most basic TF for a Harvester preview env - creates only a namespace in the Harvester cluster to show how it works
Written in bash, so they can be executed from any environment and have dependency solely on terraform binary being available.
Standalone scripts for [init, plan, apply, workspace], so they can be executed in any context.

Other PRs will follow with werft job and harvester VM will follow.

Related Issue(s)

Fixes https://github.com/gitpod-io/ops/issues/5093

How to test

TF_VAR_preview_name=temp ./workflow/preview/deploy-harvester.sh

Initializing the backend...

Initializing provider plugins...
- Reusing previous version of hashicorp/kubernetes from the dependency lock file
- Reusing previous version of harvester/harvester from the dependency lock file
- Using previously-installed hashicorp/kubernetes v2.13.1
- Using previously-installed harvester/harvester v0.5.1

Terraform has been successfully initialized!

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # kubernetes_namespace.example will be created
  + resource "kubernetes_namespace" "example" {
      + id = (known after apply)

      + metadata {
          + generation       = (known after apply)
          + name             = "preview-temp"
          + resource_version = (known after apply)
          + uid              = (known after apply)
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to apply the plan? [y/n]: y

kubernetes_namespace.example: Creating...
kubernetes_namespace.example: Creation complete after 1s [id=preview-temp]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

TF_VAR_preview_name=temp DESTROY=true ./workflow/preview/deploy-harvester.sh

Initializing the backend...
Terraform has been successfully initialized!

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  # kubernetes_namespace.example will be destroyed
  - resource "kubernetes_namespace" "example" {
      - id = "preview-temp" -> null

      - metadata {
          - annotations      = {} -> null
          - generation       = 0 -> null
          - labels           = {} -> null
          - name             = "preview-temp" -> null
          - resource_version = "347998054" -> null
          - uid              = "9939dd8c-bf8c-4456-8e81-b90f28a18707" -> null
        }
    }

Do you want to apply the plan? [y/n]: y

kubernetes_namespace.example: Destroying... [id=preview-temp]
kubernetes_namespace.example: Destruction complete after 8s

Apply complete! Resources: 0 added, 0 changed, 1 destroyed.

Release Notes

NONE

Documentation

Werft options:

/werft with-preview

mads-hartmann

Really awesome to see the first steps for hits 🎉 I have some high-level comments.

Written in bash, so they can be executed from any environment and have dependency solely on terraform binary being available.

I was really hoping to reduce the amount of bash we have for critical components. I'd be okay having two system dependencies of "go and terraform" just to avoid adding more bash. But that's my personal preference.

I didn't dive into the bash implementation in this review as I wanted to discuss this first ☺️

This is currently introducing a new top-level folder named preview. Previously we have put everything related to our "dev experience" in ./dev and we already have a ./dev/preview folder so I'd prefer to move everything there. So that would be:

dev
  preview
      previewctl
      infrastructure
      workflow

mads-hartmann · 2022-09-16T08:16:41Z

preview/infrastructure/harvester/provider.tf

+terraform {
+
+  backend "gcs" {
+    bucket = "3f4745df-preview-tf-state"


What's 3f4745df? ☺️

It's a random string - bucket names are public and globally unique, so I prefer to add some randomness to each.

mads-hartmann · 2022-09-16T08:17:43Z

preview/infrastructure/harvester/provider.tf

+  }
+}
+
+# https://registry.terraform.io/providers/harvester/harvester/latest/docs


I know we link to the docs in many places, but I find it bit redundant so would prefer to not have them unless there's a specific reason for it.

mads-hartmann · 2022-09-16T08:29:42Z

preview/infrastructure/harvester/provider.tf

+provider "k8s" {
+  alias       = "harvester"
+  config_path = var.harvester_kube_path
+}


How come our pre-commit hook end-of-file-fixer didn't remove these missing newlines? ☺️

Speaking of pre-commit-hooks it would be nice to add one for terraform fmt https://github.com/antonbabenko/pre-commit-terraform#3-add-configs-and-hooks

Ah I commit with --no-verify by default and usually at the end I run the pre-commit manually to fix anything outstanding 😅 Weird that we don't run the pre-commit as a step in the CI so it fails in cases like this.

I'll add the terraform fmt in another PR with the rest of the TF.

vulkoingim

I was really hoping to reduce the amount of bash we have for critical components. I'd be okay having two system dependencies of "go and terraform" just to avoid adding more bash. But that's my personal preference.

IMO wrapping terraform in be it go, or ts, or anything else makes very little (next to none) sense. I've purposefully written it in bash as it's quite a thin layer (albeit a little opinionated), and you can run it from anywhere and requires only terraform to exist. As we discussed, this is also a foundation that we'll be using for access, and ops - and I would probably separate it into a dedicated Action once we get to that point.

This is currently introducing a new top-level folder named preview. Previously we have put everything related to our "dev experience" in ./dev and we already have a ./dev/preview folder so I'd prefer to move everything there. So that would be:

👍 I don't have a strong opinion on that. Will move it in a bit.

vulkoingim · 2022-09-16T09:32:54Z

preview/infrastructure/harvester/provider.tf

+  }
+}
+
+# https://registry.terraform.io/providers/harvester/harvester/latest/docs


vulkoingim · 2022-09-16T09:34:51Z

preview/infrastructure/harvester/provider.tf

+provider "k8s" {
+  alias       = "harvester"
+  config_path = var.harvester_kube_path
+}


Ah I commit with --no-verify by default and usually at the end I run the pre-commit manually to fix anything outstanding 😅 Weird that we don't run the pre-commit as a step in the CI so it fails in cases like this.

I'll add the terraform fmt in another PR with the rest of the TF.

mads-hartmann · 2022-09-16T12:21:44Z

IMO wrapping terraform in be it go, or ts, or anything else makes very little (next to none) sense. I've purposefully written it in bash as it's quite a thin layer (albeit a little opinionated), and you can run it from anywhere and requires only terraform to exist. As we discussed, this is also a foundation that we'll be using for access, and ops - and I would probably separate it into a dedicated Action once we get to that point.

That's fair, if we don't expect these scripts to grow beyond just invoking a bit of TF I agree it's overkill to start using a "proper language" for it.

mads-hartmann

I added the hold label so you can update the CODEOWNERS but otherwise this looks like a great start to me. Would love a demo on during the Platform Sync on Monday.

mads-hartmann · 2022-09-16T12:31:41Z

.github/CODEOWNERS

@@ -100,6 +100,9 @@
 /.werft/*installer-tests* @gitpod-io/engineering-self-hosted
 /.werft/jobs/build/self-hosted-* @gitpod-io/engineering-self-hosted

+/preview/infrastructure/harvester @gitpod-io/platform


We can remove these now that the files have been moved as we already own /dev/preview ☺️

I've updated them, but I would like to keep them explicit, as we would like to separate them in the future.

mads-hartmann · 2022-09-20T10:20:09Z

@vulkoingim the build is failing so added the hold label

roboquat added release-note-none do-not-merge/work-in-progress size/XL labels Sep 15, 2022

vulkoingim force-pushed the aa/tf-workflow branch 2 times, most recently from 3f98ce0 to 5bc1456 Compare September 15, 2022 11:21

vulkoingim marked this pull request as ready for review September 15, 2022 11:22

roboquat removed the do-not-merge/work-in-progress label Sep 15, 2022

vulkoingim force-pushed the aa/tf-workflow branch from e7d0266 to a2ec2c5 Compare September 15, 2022 11:54

vulkoingim requested a review from a team September 15, 2022 12:44

github-actions bot added the team: devx label Sep 15, 2022

mads-hartmann reviewed Sep 16, 2022

View reviewed changes

mads-hartmann requested a review from a team September 16, 2022 08:32

vulkoingim commented Sep 16, 2022

View reviewed changes

vulkoingim force-pushed the aa/tf-workflow branch from 328ec90 to 710739c Compare September 16, 2022 10:32

mads-hartmann added the do-not-merge/hold label Sep 16, 2022

mads-hartmann approved these changes Sep 16, 2022

View reviewed changes

vulkoingim added do-not-merge/hold and removed do-not-merge/hold labels Sep 20, 2022

vulkoingim force-pushed the aa/tf-workflow branch from 7d70e55 to f364419 Compare September 20, 2022 09:17

vulkoingim removed the do-not-merge/hold label Sep 20, 2022

mads-hartmann added the do-not-merge/hold label Sep 20, 2022

Terraform workflow for preview envs

47a6f15

vulkoingim force-pushed the aa/tf-workflow branch from f364419 to 47a6f15 Compare September 20, 2022 10:20

vulkoingim removed the do-not-merge/hold label Sep 20, 2022

roboquat merged commit 85ab504 into main Sep 20, 2022

roboquat deleted the aa/tf-workflow branch September 20, 2022 10:45

vulkoingim mentioned this pull request Oct 7, 2022

Preview vm create and delete uses the new TF flow #13658

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Preview env] TF Workflow #12981

[Preview env] TF Workflow #12981

vulkoingim commented Sep 15, 2022 •

edited

Loading

mads-hartmann left a comment

mads-hartmann Sep 16, 2022

vulkoingim Sep 16, 2022

mads-hartmann Sep 16, 2022

vulkoingim Sep 16, 2022

mads-hartmann Sep 16, 2022

mads-hartmann Sep 16, 2022

vulkoingim Sep 16, 2022

vulkoingim left a comment

vulkoingim Sep 16, 2022

vulkoingim Sep 16, 2022

mads-hartmann commented Sep 16, 2022

mads-hartmann left a comment

mads-hartmann Sep 16, 2022

vulkoingim Sep 16, 2022

mads-hartmann commented Sep 20, 2022

[Preview env] TF Workflow #12981

[Preview env] TF Workflow #12981

Conversation

vulkoingim commented Sep 15, 2022 • edited Loading

Description

Related Issue(s)

How to test

Release Notes

Documentation

Werft options:

mads-hartmann left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vulkoingim left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mads-hartmann commented Sep 16, 2022

mads-hartmann left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mads-hartmann commented Sep 20, 2022

vulkoingim commented Sep 15, 2022 •

edited

Loading