Standard way to add manifests used for implementing the operator (ex: Tekton Pipeline)

[VannTen]

What is the URL of the document?

https://sdk.operatorframework.io/docs/overview/project-layout/

Which section(s) is the issue in?

Common Base, I guess.

What needs fixing?

Precise how to ship resources which are part of the operator implementation.

Additional context

For some context, we're currently working on a operator which use Pipeline (CR provied by Tekton) to implement some parts of its functionality (we create PipelineRun from within the operator code).
The question is, where should those Pipelines lives in our repo ? Is there a standard location ?

The goal would be to be able to use the manifests target to deploy the Pipelines with the rest of the resources composing the operator.

bundle: manifests | $(KUSTOMIZE) ## Generate bundle manifests and metadata, then validate generated files.
	operator-sdk generate kustomize manifests -q
	cd config/manager && $(KUSTOMIZE) edit set image controller=$(IMG)
	$(KUSTOMIZE) build config/manifests | operator-sdk generate bundle $(BUNDLE_GEN_FLAGS)
	operator-sdk bundle validate ./bundle

I looked through the documentation but didn't find (might be by my mistake) an answer

[everettraven]

@VannTen does your operator depend on Tekton being installed on the cluster? If so, then you shouldn't have to install the CRD as part of your operator installation.

If you are just using the CRD for something custom and it doesn't depend on Tekton being installed on the cluster, then I believe the best place for it would be to add it under the config/crd/ directory in the project.

Under the Common Base section, in the table there is this entry:

|config/crd/ | Contains the Custom Resources Definitions.|

[VannTen]

On Mon, Nov 21, 2022 at 07:44:06AM -0800, Bryce Palmer wrote: @VannTen does your operator depend on Tekton being installed on the cluster? If so, then you shouldn't have to install the CRD as part of your operator installation.

Yes, it does. But I don't want to install the CRD, I want to install the CR (aka, `Pipeline`). To schematise, my operator reconcile loop does something like this: ``` my_cr = kubectl get my_cr $current_cr_reconciled data = extract_stuff from $my_cr type = extract_type from $my_cr corresponding_pipeline_run = create_pipeline_run(type, data) echo $corresponding_pipeline_run | kubectl create -f - ``` So I want to ship one Pipeline object per type (but the problem is the same using only one) ; those implements part of our operator logic. The operator code itself only has the responsability to create objects and update status depending (among other things) PipelineRuns results. Is that clearer ? EDIT: Sorry, I forgot md does not work in email replies :/

[everettraven]

Ah okay, I think I understand now. Just to check my understanding, you are wanting to create a Pipeline CR as part of your reconciliation loop?

Also a follow up question, what kind of operator is this (Go, Ansible, Helm)?

[VannTen]

On Mon, Nov 21, 2022 at 08:05:51AM -0800, Bryce Palmer wrote: Ah okay, I think I understand now. Just to check my understanding, you are wanting to create a `Pipeline` CR as part of your reconciliation loop?

(a PipelineRun is the instantiation of a Pipeline) Almost. I create a PipelineRun as part of the reconcile loop. That part is done. But to create a PipelineRun, I need it's "blueprint" which is the Pipeline.

[varshaprasad96]

@VannTen If I understand right, the goal is to include an external CR manifest in the operator project and not the definition for the name. manifests target uses KB markers and generates a bunch of manifests with the help of controller-tools. Additionally, when the make build target is called, the yaml files under config/manifests are used by kustomize to build the required files. If you are looking to include these files in the bundle, I would suggest to utilize the --input-dir flag in the operator-sdk generate bundle command (ref: https://sdk.operatorframework.io/docs/cli/operator-sdk_generate_bundle/). This way you can have the external CRs in any location inside config/ (which seems apt for your porject) and specify it in the command to include them in the bundle.

[jberkhahn]

So, it looks like you're trying to create a single Pipeline object when your operator is installed, and then run stuff against it when an instance of your operator is created?

There (currently) isn't really a good way to include instances of a CR in your operator's bundle via OLM. This is a known issue, and is an feature explicitly being added in Rukpak.

However, if instances of your operator type are going to interact with that resource and do stuff, I'm not sure if statically specifying it is a good idea at all. I think I would include the information you need to instantiate the Pipeline (basically all the stuff it the Pipeline.spec), and check to see if the pipeline exists at the top of your reconciliation loop, and make sure it does there.

[VannTen]

So, it looks like you're trying to create a single Pipeline object when your operator is installed, and then run stuff against it when an instance of your operator is created?

Exactly.

There (currently) isn't really a good way to include instances of a CR in your operator's bundle via OLM. This is a known issue, and is an feature explicitly being added in [Rukpak](https://github.com/operator-framework/rukpak).

Thanks for the link, I'll take a look.

However, if instances of your operator type are going to interact with that resource and do stuff, I'm not sure if statically specifying it is a good idea at all. I think I would include the information you need to instantiate the Pipeline (basically all the stuff it the Pipeline.spec), and check to see if the pipeline exists at the top of your reconciliation loop, and make sure it does there.

So basically, embed the definition of the Pipeline into the operator code, right ? That's an idea, but it makes the Pipelines a bit harder to test. What reasons make static Pipelines a bad idea ? (Thanks for the discussion in any case :) )

[everettraven]

@VannTen,

So basically, embed the definition of the Pipeline into the operator code, right ?

Yep that is the suggestion @jberkhahn is making. The logic for your reconciliation loop would essentially be updated to have the first step be:

• Create the Pipeline CR if it does not already exist

What reasons make static Pipelines a bad idea ?

I think the problem here with making the Pipeline a static resource that is only applied as part of the installation of the operator is that your operator depends on the existence of that Pipeline resource to work.

If you create it only during the installation of the operator, what happens if someone unaware of what that specific Pipeline resource is for deletes it from the cluster? Having a check at the beginning of your reconciliation loop that says "if the Pipeline resource I depend on does not exist, then create it" helps to make your operator more resilient in a situation like this because you will just recreate it if it is removed, ensuring your operator always has what it needs to run properly.

I hope this helps a bit!

[VannTen]

I think the problem here with making the `Pipeline` a static resource that is only applied as part of the installation of the operator is that your operator depends on the existence of that `Pipeline` resource to work.

But how does that differ from removing the operator itself ? (like someone deleting the Deployment which runs the controller, or the RBAC needed for its operation) IMO, this is the responsibility of the cluster operator, or eventually of OLM. My operator should not have the rights to install itself, don't you think ?

[everettraven]

But how does that differ from removing the operator itself ? (like someone deleting the Deployment which runs the controller, or the RBAC needed for its operation)
IMO, this is the responsibility of the cluster operator, or eventually of OLM.

I believe OLM does handle the case of Deployment or RBAC deletion (just deleting this stuff isn't the proper way to remove an operator). As mentioned by @jberkhahn:

There (currently) isn't really a good way to include instances of a CR in your operator's bundle via OLM. This is a known issue, and is an feature explicitly being added in Rukpak.

Due to this limitation unfortunately this route would likely have to be taken for the time being (and if you wanted to have it created once during operator startup that is an option as well. It is your operator after all).

Alternatively you could begin looking into using Rukpak but it is not a direct replacement to OLM. I would recommend really weighing the pros & cons of a switch like this prior to actually switching.

My operator should not have the rights to install itself, don't you think ?

I don't think in this case the operator would be "installing" itself. It would rather be creating the resources that it is dependent on existing in order to operate as expected.

In my opinion, it is your responsibility as the developer of the operator to make clear what happens when the operator is installed on the cluster and it is the responsibility of the cluster admin (or whoever is installing the operator) to have an understanding and accept what will happen on the cluster when they install the operator. This includes what permissions the operator requires and will be created, any dependent resources that are created on the cluster, and what happens when a CR is created and reconciled by the operator.

[VannTen]

On Mon, Nov 28, 2022 at 07:54:40AM -0800, Bryce Palmer wrote: > But how does that differ from removing the operator itself ? (like someone deleting the Deployment which runs the controller, or the RBAC needed for its operation) IMO, this is the responsibility of the cluster operator, or eventually of OLM.

Agreed. But I would have liked to have OLM handle the rest as well. (seems to be what operator-framework/operator-lifecycle-manager#1996 is about right ?)

Alternatively you could begin looking into using [Rukpak](https://github.com/operator-framework/rukpak) but it is not a direct replacement to OLM. I would recommend really weighing the pros & cons of a switch like this prior to actually switching.

We'll take a took. - Thanks for your answers, I now understand what I can and can not do with OLM and the operator-sdk. I'm ok to close this, unless you want to keep it open as a documentation issue.

[jberkhahn]

Thanks. Feel free to reopen this if you do have any further questions.