From 23c517f15662093ed0456b188515382c890fd194 Mon Sep 17 00:00:00 2001 From: Joe Lanford Date: Tue, 23 Oct 2018 17:08:56 -0400 Subject: [PATCH 1/5] doc/proposals: adding helm operator proposal --- doc/proposals/helm-operator.md | 127 +++++++++++++++++++++++++++++++++ 1 file changed, 127 insertions(+) create mode 100644 doc/proposals/helm-operator.md diff --git a/doc/proposals/helm-operator.md b/doc/proposals/helm-operator.md new file mode 100644 index 00000000000..fe7feafed2d --- /dev/null +++ b/doc/proposals/helm-operator.md @@ -0,0 +1,127 @@ +## Helm Operator Proposal + +### Background + +As was mentioned in the [Ansible Operator Proposal](./ansible-operator.md), not everyone is a golang developer, so the SDK needs to support other types of operators to gain adoption across a wider community of users. + +[Helm](https://helm.sh/) is one of the most widely-used tools for Kubernetes application management, and it bills itself as the "package manager for Kubernetes." Operators serve a nearly identical function, but they improve on Helm's concepts by incorporating an always-on reconciliation loop rather than relying on an imperative user-driven command line tool. By integrating Helm's templating engine and release management into an operator, the SDK will further increase the number of potential users by adding the ability to deploy Charts (e.g. from Helm's [large catalog of existing Charts](https://github.com/helm/charts)) as operators with very little extra effort. + +### Goals + +The goal of the Helm Operator will be to create a fully functional framework for Helm Chart developers to create operators. It will also expose a library for golang users to use Helm in their operator if they so choose. These two goals in conjunction will allow users to select the best technology for their project or skillset. + +### New Operator Type + +This proposal creates a new type of operator called `helm`. The new type is used to tell the tooling to act on that type of operator. + +### Package Structure + +Packages will be added to the operator-sdk. These packages are designed to be usable by the end user if they choose to and should have a well documented public API. The proposed packages are: + +* /operator-sdk/pkg/helm/client + * Will contain a helper function to create a Helm client from `controller-runtime` manager. + +* /operator-sdk/pkg/helm/controller + * Will contain the Helm controller. + * Will contain an exposed reconciler. The default `Add` method will use this reconciler. + +* /operator-sdk/pkg/helm/engine + * Will contain a Helm Engine implementation that adds owner references to generated Kubernetes resource assets, which is necessary for garbage collection of Helm chart resources. + +* /operator-sdk/pkg/helm/internal + * Will contain types and utilities used by other Helm packages in the SDK. + +* /operator-sdk/pkg/helm/release + * Will contain the Manager types and interfaces. A Manager is responsible for: + * Implementing Helm's Tiller functions that are necessary to install, update, and uninstall releases. + * Reconciling an existing release's resources. + * A default Manager implementation is provided in this package but is not exported. + * Package functions: + * NewManager - method that returns a new Manager for a provided helm chart. + * NewManagersFromEnv - method that returns a map of GVK to Manager types based on environment variables. + * NewManagersFromFile - method that returns a map of GVK to Manager types based on a provided config file. + +### Commands + +We are adding and updating existing commands to accommodate the helm operator. Changes to the `cmd` package as well as changes to the generator are needed. + +#### New + +New functionality will be updates to allow Helm operator developers to create a new boilerplate operator structure with everything necessary to get started developing and deploying a Helm operator with the SDK. + +``` +operator-sdk new --type=helm --kind= --api-version= +``` + +This will be new scaffolding for the above command under the hood. We will: +* Create a `./` directory. +* Create a `.//charts` directory. +* Generate a simple default chart at `.//charts/`. +* Create a new watches file at `.//watches.yaml`. The chart and GVK will be defaulted based on input to the `new` command. +* Create a `.//deploy` with the Kubernetes resource files necessary to run the operator. +* Create a `./build/Dockerfile` that uses the watches file and the helm chart. It will use the Helm operator as its base image. + +The resulting structure will be: + +``` + +| watches.yaml +| +|-- build +| | Dockerfile +| +|-- charts +| |-- +| | Chart.yaml +| | ... +| +|-- deploy +| | operator.yaml +| | role_binding.yaml +| | role.yaml +| | service_account.yaml +| | +| |-- crds +| | _crd.yaml +| | _cr.yaml +``` + +#### Add + +Add functionality will be updated to allow Helm operator developers to add new CRDs/CRs and to update the watches.yaml file for additional Helm charts. The command helps when a user wants to watch more than one CRD for their operator. + +``` +operator-sdk add api --api-version=/ --kind= +``` + +Flags: +* **Required:** --kind - the kind for the CRD. +* **Required:** --api-version - the group/version for the CRD. + +**NOTE:** `operator-sdk add controller` will not be supported, since it doesn't make sense for a Helm operator. + +#### Up + +Up functionality will be updated to allow Helm operator developers to run their operator locally, using the `operator-sdk` binary's built-in helm operator implementation. + +``` +operator-sdk up local +``` + +This should use the known structure and the helm operator code to run the operator from this location. The existing code will need to be updated with a new operator type check for `helm` (in addition to existing `go` and `ansible` types). The command works by running the operator-sdk binary, which includes the Helm operator code, as the operator process. + +#### Build + +Build functionality will be updated to support building a docker image from the Helm operator directory structure. + +``` +operator-sdk build +``` + +### Observations and open questions + +* There will be a large amount of overlap in the `operator-sdk` commands for the Ansible and Helm operators. We should take care to extract the resusable features of the Ansible operator commands into a shared library, usable by both Helm and Ansible commands. + +* This proposal assumes that a Helm Operator base image will be available for building Helm operator projects. What generates the Helm operator base image and what is the registry, image name, versioning, etc.? + +* There is a moderate amount of complexity already related to how operator types are handled between the `go` and `ansible` types. With the addition of a third type, there may need to be a larger design proposal for operator types. For example, do we need to define an `Operator` interface that each of the operator types can implement for flag verification, scaffolding, project detection, etc.? From ce43f82d0cc8990887c85590cc8d36597d051522 Mon Sep 17 00:00:00 2001 From: Joe Lanford Date: Tue, 30 Oct 2018 12:50:49 -0400 Subject: [PATCH 2/5] doc/proposals: rename helm chart dir to 'helm-charts' --- doc/proposals/helm-operator.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/proposals/helm-operator.md b/doc/proposals/helm-operator.md index fe7feafed2d..aa2e142844b 100644 --- a/doc/proposals/helm-operator.md +++ b/doc/proposals/helm-operator.md @@ -55,8 +55,8 @@ operator-sdk new --type=helm --kind= --api-version=` directory. -* Create a `.//charts` directory. -* Generate a simple default chart at `.//charts/`. +* Create a `.//helm-charts` directory. +* Generate a simple default chart at `.//helm-charts/`. * Create a new watches file at `.//watches.yaml`. The chart and GVK will be defaulted based on input to the `new` command. * Create a `.//deploy` with the Kubernetes resource files necessary to run the operator. * Create a `./build/Dockerfile` that uses the watches file and the helm chart. It will use the Helm operator as its base image. @@ -70,7 +70,7 @@ The resulting structure will be: |-- build | | Dockerfile | -|-- charts +|-- helm-charts | |-- | | Chart.yaml | | ... From 62f76ae806d183c553a01b080315978a309f5826 Mon Sep 17 00:00:00 2001 From: Joe Lanford Date: Wed, 31 Oct 2018 12:13:27 -0400 Subject: [PATCH 3/5] doc/proposals: updates based on PR feedback * change `add api` to `add crd` * clarify that `helm-charts` directory will be used to detect that operator type is `helm` * add base image build and tag plan --- doc/proposals/helm-operator.md | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-) diff --git a/doc/proposals/helm-operator.md b/doc/proposals/helm-operator.md index aa2e142844b..d7e784e980e 100644 --- a/doc/proposals/helm-operator.md +++ b/doc/proposals/helm-operator.md @@ -2,7 +2,7 @@ ### Background -As was mentioned in the [Ansible Operator Proposal](./ansible-operator.md), not everyone is a golang developer, so the SDK needs to support other types of operators to gain adoption across a wider community of users. +As was mentioned in the [Ansible Operator Proposal](./ansible-operator.md), not everyone is a golang developer, so the SDK needs to support other types of operators to gain adoption across a wider community of users. [Helm](https://helm.sh/) is one of the most widely-used tools for Kubernetes application management, and it bills itself as the "package manager for Kubernetes." Operators serve a nearly identical function, but they improve on Helm's concepts by incorporating an always-on reconciliation loop rather than relying on an imperative user-driven command line tool. By integrating Helm's templating engine and release management into an operator, the SDK will further increase the number of potential users by adding the ability to deploy Charts (e.g. from Helm's [large catalog of existing Charts](https://github.com/helm/charts)) as operators with very little extra effort. @@ -86,19 +86,21 @@ The resulting structure will be: | | _cr.yaml ``` +The SDK CLI will use the presence of the `helm-charts` directory to detect a `helm` type project. + #### Add Add functionality will be updated to allow Helm operator developers to add new CRDs/CRs and to update the watches.yaml file for additional Helm charts. The command helps when a user wants to watch more than one CRD for their operator. ``` -operator-sdk add api --api-version=/ --kind= +operator-sdk add crd --api-version=/ --kind= ``` Flags: * **Required:** --kind - the kind for the CRD. * **Required:** --api-version - the group/version for the CRD. -**NOTE:** `operator-sdk add controller` will not be supported, since it doesn't make sense for a Helm operator. +**NOTE:** `operator-sdk add` subcommands `api` and `controller` will not be supported, since they are only valid for Go operators. #### Up @@ -118,10 +120,18 @@ Build functionality will be updated to support building a docker image from the operator-sdk build ``` +### Base Image + +The SDK team will maintain a build job for the `helm-operator` base image with the following tagging methodology: +* Builds on the master branch that pass nightly CI tests will be tagged with `:master` +* Builds for tags that pass CI will be tagged with `:`. If the tag is also the greatest semantic version for the repository, the image will also be tagged with `:latest`. + +The go binary included in the base image will be built with `GOOS=linux` and `GOARCH=amd64`. + +The base image repository will be `quay.io/water-hole/helm-operator`. + ### Observations and open questions * There will be a large amount of overlap in the `operator-sdk` commands for the Ansible and Helm operators. We should take care to extract the resusable features of the Ansible operator commands into a shared library, usable by both Helm and Ansible commands. -* This proposal assumes that a Helm Operator base image will be available for building Helm operator projects. What generates the Helm operator base image and what is the registry, image name, versioning, etc.? - * There is a moderate amount of complexity already related to how operator types are handled between the `go` and `ansible` types. With the addition of a third type, there may need to be a larger design proposal for operator types. For example, do we need to define an `Operator` interface that each of the operator types can implement for flag verification, scaffolding, project detection, etc.? From cd09a9f7fd7199d2bf78ad0245082ba7179ff29b Mon Sep 17 00:00:00 2001 From: Joe Lanford Date: Fri, 2 Nov 2018 10:05:57 -0400 Subject: [PATCH 4/5] doc/proposals/helm-operator.md: rename Release to ReleaseManager, clarify function vs. method --- doc/proposals/helm-operator.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/doc/proposals/helm-operator.md b/doc/proposals/helm-operator.md index d7e784e980e..2672d7dab58 100644 --- a/doc/proposals/helm-operator.md +++ b/doc/proposals/helm-operator.md @@ -23,7 +23,7 @@ Packages will be added to the operator-sdk. These packages are designed to be us * /operator-sdk/pkg/helm/controller * Will contain the Helm controller. - * Will contain an exposed reconciler. The default `Add` method will use this reconciler. + * Will contain an exposed reconciler. The default `Add` function will use this reconciler. * /operator-sdk/pkg/helm/engine * Will contain a Helm Engine implementation that adds owner references to generated Kubernetes resource assets, which is necessary for garbage collection of Helm chart resources. @@ -32,14 +32,14 @@ Packages will be added to the operator-sdk. These packages are designed to be us * Will contain types and utilities used by other Helm packages in the SDK. * /operator-sdk/pkg/helm/release - * Will contain the Manager types and interfaces. A Manager is responsible for: + * Will contain the ReleaseManager types and interfaces. A ReleaseManager is responsible for: * Implementing Helm's Tiller functions that are necessary to install, update, and uninstall releases. * Reconciling an existing release's resources. - * A default Manager implementation is provided in this package but is not exported. + * A default ReleaseManager implementation is provided in this package but is not exported. * Package functions: - * NewManager - method that returns a new Manager for a provided helm chart. - * NewManagersFromEnv - method that returns a map of GVK to Manager types based on environment variables. - * NewManagersFromFile - method that returns a map of GVK to Manager types based on a provided config file. + * NewReleaseManager - function that returns a new ReleaseManager for a provided helm chart. + * NewReleaseManagersFromEnv - function that returns a map of GVK to ReleaseManager types based on environment variables. + * NewReleaseManagersFromFile - function that returns a map of GVK to ReleaseManager types based on a provided config file. ### Commands From aeb87e8abe4e16e3618e3fbc58b138ca28f42c00 Mon Sep 17 00:00:00 2001 From: Joe Lanford Date: Fri, 2 Nov 2018 16:05:18 -0400 Subject: [PATCH 5/5] doc/proposals/helm-operator.md: cleanup and PR feedback updates * Change `release.ReleaseManager` back to `release.Manager` * Improve description of `pkg/helm/controller` package * Clarifying flags and features of `add crd` subcommand * Clarifying that `test` command is not supproted * Clarifying `new` command flag requirements and support --- doc/proposals/helm-operator.md | 30 ++++++++++++++++++++---------- 1 file changed, 20 insertions(+), 10 deletions(-) diff --git a/doc/proposals/helm-operator.md b/doc/proposals/helm-operator.md index 2672d7dab58..d05d576fbf8 100644 --- a/doc/proposals/helm-operator.md +++ b/doc/proposals/helm-operator.md @@ -22,8 +22,8 @@ Packages will be added to the operator-sdk. These packages are designed to be us * Will contain a helper function to create a Helm client from `controller-runtime` manager. * /operator-sdk/pkg/helm/controller - * Will contain the Helm controller. - * Will contain an exposed reconciler. The default `Add` function will use this reconciler. + * Will contain an exported `HelmOperatorReconciler` that implements the `controller-runtime` `reconcile.Reconciler` interface. + * Will contain an exported `Add` function that creates a controller using the `HelmOperatorReconciler` and adds watches based on a set of watch options passed to the `Add` function. * /operator-sdk/pkg/helm/engine * Will contain a Helm Engine implementation that adds owner references to generated Kubernetes resource assets, which is necessary for garbage collection of Helm chart resources. @@ -32,18 +32,18 @@ Packages will be added to the operator-sdk. These packages are designed to be us * Will contain types and utilities used by other Helm packages in the SDK. * /operator-sdk/pkg/helm/release - * Will contain the ReleaseManager types and interfaces. A ReleaseManager is responsible for: + * Will contain the `Manager` types and interfaces. A `Manager` is responsible for: * Implementing Helm's Tiller functions that are necessary to install, update, and uninstall releases. * Reconciling an existing release's resources. - * A default ReleaseManager implementation is provided in this package but is not exported. + * A default `Manager` implementation is provided in this package but is not exported. * Package functions: - * NewReleaseManager - function that returns a new ReleaseManager for a provided helm chart. - * NewReleaseManagersFromEnv - function that returns a map of GVK to ReleaseManager types based on environment variables. - * NewReleaseManagersFromFile - function that returns a map of GVK to ReleaseManager types based on a provided config file. + * `NewManager` - function that returns a new Manager for a provided helm chart. + * `NewManagersFromEnv` - function that returns a map of GVK to Manager types based on environment variables. + * `NewManagersFromFile` - function that returns a map of GVK to Manager types based on a provided config file. ### Commands -We are adding and updating existing commands to accommodate the helm operator. Changes to the `cmd` package as well as changes to the generator are needed. +We are adding and updating existing commands to accommodate the Helm operator. Changes to the `cmd` package as well as changes to the generator are needed. #### New @@ -53,6 +53,11 @@ New functionality will be updates to allow Helm operator developers to create a operator-sdk new --type=helm --kind= --api-version= ``` +Flags: +* `--type=helm` is required to create Helm operator project. +* **Required:** --kind - the kind for the CRD. +* **Required:** --api-version - the group/version for the CRD. + This will be new scaffolding for the above command under the hood. We will: * Create a `./` directory. * Create a `.//helm-charts` directory. @@ -93,14 +98,15 @@ The SDK CLI will use the presence of the `helm-charts` directory to detect a `he Add functionality will be updated to allow Helm operator developers to add new CRDs/CRs and to update the watches.yaml file for additional Helm charts. The command helps when a user wants to watch more than one CRD for their operator. ``` -operator-sdk add crd --api-version=/ --kind= +operator-sdk add crd --api-version=/ --kind= --update-watches= ``` Flags: * **Required:** --kind - the kind for the CRD. * **Required:** --api-version - the group/version for the CRD. +* **Optional:** --update-watches - whether or not to update watches.yaml file (default: false). -**NOTE:** `operator-sdk add` subcommands `api` and `controller` will not be supported, since they are only valid for Go operators. +**NOTE:** `operator-sdk add` subcommands `api` and `controller` will not be supported, since they are only valid for Go operators. Running these subcommands in a Helm operator project will result in an error. #### Up @@ -120,6 +126,10 @@ Build functionality will be updated to support building a docker image from the operator-sdk build ``` +#### Test + +The SDK `test` command currently only supports Go projects, so there will be no support for the `operator-sdk test` subcommand in the initial integration of the Helm operator. + ### Base Image The SDK team will maintain a build job for the `helm-operator` base image with the following tagging methodology: