✨ Add support for customizing additional provider deployments #538

willie-yao · 2024-05-31T00:31:25Z

What this PR does / why we need it:
This PR adds support for customizing additional provider deployments via the additionalDeployments field in the provider spec. additionalDeployments is a map of the name of the provider to both the deployment and manager spec which is used for customizing the deployment/management container.

This PR also adds allows CRDPattern to be configurable in deployment customization. This is to allow for installing additional CRDs through the ASO deployment.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

netlify · 2024-05-31T00:31:44Z

✅ Deploy Preview for kubernetes-sigs-cluster-api-operator ready!

Name	Link
🔨 Latest commit	`d1112d3`
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-sigs-cluster-api-operator/deploys/667df1d13ef2ce0008206184
😎 Deploy Preview	https://deploy-preview-538--kubernetes-sigs-cluster-api-operator.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

willie-yao · 2024-05-31T18:45:13Z

I'm a bit confused on how to add a conversion function for CRDPattern. @Fedosin @alexander-demicev is there an example on how to do this? I tried adding something like this to the conversion funcs which didn't seem to work: https://github.com/willie-yao/cluster-api-operator/blob/upstream-install-aso-crds/api/v1alpha1/provider_conversion.go#L48

alexander-demicev · 2024-06-03T11:25:32Z

@willie-yao You can find an example here https://github.com/kubernetes-sigs/cluster-api-provider-aws/pull/4228/files, this PR added a new field to API and conversion to older versions

willie-yao · 2024-06-03T20:59:15Z

Thanks @alexander-demicev! I fixed up the conversion and added a unit test!

alexander-demicev · 2024-06-04T08:59:06Z

@willie-yao This change is too specific to capz. What do you think about making it more generic? Can we introduce a new field to customize additional deployments that come with providers? I think something like AdditionalDeployments might work, where we can store the deployment names to customize and DeploymentSpec structure to apply these customizations. Do you think it makes sense?
I can see that in the future there might be more providers like CAPZ with more than one Deployment so it might be better to try to fix the problem in a bigger scope.

willie-yao · 2024-06-04T20:09:36Z

@alexander-demicev That is a great idea. I did notice that this may be too specific to CAPZ to be implemented like this. I'll work on adding a field like AdditionalDeployments. Thanks for the suggestion!

willie-yao · 2024-06-05T22:05:41Z

api/v1alpha2/provider_types.go

+
+	// Deployment defines the properties that can be enabled on the deployment for the provider.
+	// +optional
+	Deployment *DeploymentSpec `json:"deployment,omitempty"`


@alexander-demicev not sure if this is the best way to do it, but I refactored the code to include an AdditionalDeployments field that maps the name of the deployment to the Manager and Deployment spec, where the additional deployments can be customized. Is this a good way to go about it?

willie-yao · 2024-06-06T22:31:11Z

/retest

willie-yao · 2024-06-07T00:11:25Z

hack/charts/cluster-api-operator/templates/infra.yaml

+  {{- include "recursivePrinter" $value | indent 2 }}
+{{- else }}
+  {{ $key }}: {{ $value }}
+{{- end }}


This function allows the chart to print out additionalDeployments correctly in yaml format as both deployments and manager has nested fields.

willie-yao · 2024-06-07T00:12:18Z

internal/controller/component_customizer.go

@@ -73,25 +73,41 @@ func customizeObjectsFn(provider operatorv1.GenericProvider) func(objs []unstruc
 					}))
 			}

+			//nolint:nestif


Not sure if a nolint is allowed here. I'll have to create a few helper functions if not. lmk which approach is better!

alexander-demicev

looks great, couple of minor questions

api/v1alpha2/provider_types.go

alexander-demicev · 2024-06-07T15:22:30Z

api/v1alpha2/provider_types.go

@@ -123,6 +141,11 @@ type ManagerSpec struct {
 	// in as container args to the provider's controller manager.
 	// Controller Manager flag is --feature-gates.
 	FeatureGates map[string]bool `json:"featureGates,omitempty"`
+
+	// CRDPattern is a regular expression that matches the CRDs that the provider


do we still need to keep this field? can it be replaced with the container arguments in the deployment spec?

Yup, confirmed it works via container arguments in the deployment spec!

alexander-demicev · 2024-06-21T09:04:53Z

test/e2e/resources/only-infra-with-additional-deployments.yaml

+  additionalDeployments:
+    fake-controller-manager:
+      manager:
+        crdPattern: ".*"


Should crdPattern be removed since we can achieve the same with container arguments?

alexander-demicev · 2024-06-21T09:05:11Z

PR looks good, just one small comment

furkatgofurov7 · 2024-06-24T09:17:08Z

/retest

willie-yao · 2024-06-25T22:25:27Z

Looks like there's a problem with the E2E test output. Will be looking into this.

willie-yao · 2024-06-25T22:34:02Z

The hyphen in --crd-pattern messed up the test helm command, and since we're just testing if something can be set in the additionalDeployments rather than a specific field, I removed that field.

willie-yao · 2024-06-25T23:15:42Z

Actually it looks like the error is coming from the recursivePrinter: https://github.com/willie-yao/cluster-api-operator/blob/25328eb918edd7eb4b2e79cf10ab73cf0ca4f9e1/hack/charts/cluster-api-operator/templates/infra.yaml#L1

I'm not sure how to adjust that function to allow array values to be passed inline. When specifying container customizations for additionalDeployments, using a values file works. Do we have to add support for setting this inline?

willie-yao · 2024-06-27T23:13:26Z

I went ahead and removed the test case from helm_test.go due to the complicated nature of setting additionalDeployments via inline --set. If this is something we should support, I'll look into another way to scaffold out the additional customizations.

alexander-demicev · 2024-07-08T16:17:34Z

@willie-yao We can look into setting additionalDeployments in tests in a follow-up PR

alexander-demicev

/approve

k8s-ci-robot · 2024-07-08T16:18:16Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alexander-demicev

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [alexander-demicev]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Fedosin

/lgtm

k8s-ci-robot · 2024-07-08T16:55:46Z

LGTM label has been added.

Git tree hash: 35e1c27d5d4bc9aba5487fb18b15f2309573f2ff

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label May 31, 2024

k8s-ci-robot requested review from Fedosin and furkatgofurov7 May 31, 2024 00:31

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label May 31, 2024

willie-yao force-pushed the upstream-install-aso-crds branch from 909dceb to 15a3e6c Compare May 31, 2024 17:20

willie-yao force-pushed the upstream-install-aso-crds branch 3 times, most recently from 7888168 to 7c0f00d Compare June 3, 2024 20:58

willie-yao force-pushed the upstream-install-aso-crds branch from 7c0f00d to 3ae6a36 Compare June 3, 2024 21:02

willie-yao changed the title ~~✨ Add configurable CRDPattern field for manager~~ ✨ WIP: Add configurable CRDPattern field for manager Jun 4, 2024

willie-yao changed the title ~~✨ WIP: Add configurable CRDPattern field for manager~~ WIP: ✨ Add configurable CRDPattern field for manager Jun 4, 2024

willie-yao force-pushed the upstream-install-aso-crds branch 2 times, most recently from de8275a to a40a7c8 Compare June 5, 2024 22:02

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 5, 2024

willie-yao commented Jun 5, 2024

View reviewed changes

willie-yao force-pushed the upstream-install-aso-crds branch 4 times, most recently from 9a75480 to 1306800 Compare June 6, 2024 22:15

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 6, 2024

willie-yao force-pushed the upstream-install-aso-crds branch from 1306800 to 450c0c0 Compare June 6, 2024 22:50

willie-yao changed the title ~~✨ Add configurable CRDPattern field for manager~~ ✨ Add support for customizing additional provider deployments Jun 6, 2024

willie-yao commented Jun 7, 2024

View reviewed changes

willie-yao force-pushed the upstream-install-aso-crds branch from ad064f1 to 728860c Compare June 7, 2024 00:13

alexander-demicev reviewed Jun 7, 2024

View reviewed changes

willie-yao force-pushed the upstream-install-aso-crds branch 2 times, most recently from 0cec2c0 to b6684d7 Compare June 10, 2024 18:48

alexander-demicev reviewed Jun 21, 2024

View reviewed changes

willie-yao force-pushed the upstream-install-aso-crds branch 2 times, most recently from 151f8ee to 9e2c3ed Compare June 21, 2024 22:43

willie-yao force-pushed the upstream-install-aso-crds branch from 9e2c3ed to 25328eb Compare June 25, 2024 22:33

Add support for customizing additional provider deployments

d1112d3

willie-yao force-pushed the upstream-install-aso-crds branch from 25328eb to d1112d3 Compare June 27, 2024 23:12

alexander-demicev approved these changes Jul 8, 2024

View reviewed changes

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 8, 2024

Fedosin approved these changes Jul 8, 2024

View reviewed changes

k8s-ci-robot assigned Fedosin Jul 8, 2024

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 8, 2024

k8s-ci-robot merged commit 771e8a3 into kubernetes-sigs:main Jul 8, 2024
14 checks passed

✨ Add support for customizing additional provider deployments #538

✨ Add support for customizing additional provider deployments #538

Uh oh!

Conversation

willie-yao commented May 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

netlify bot commented May 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for kubernetes-sigs-cluster-api-operator ready!

Uh oh!

willie-yao commented May 31, 2024

Uh oh!

alexander-demicev commented Jun 3, 2024

Uh oh!

willie-yao commented Jun 3, 2024

Uh oh!

alexander-demicev commented Jun 4, 2024

Uh oh!

willie-yao commented Jun 4, 2024

Uh oh!

willie-yao Jun 5, 2024

Choose a reason for hiding this comment

Uh oh!

willie-yao commented Jun 6, 2024

Uh oh!

willie-yao Jun 7, 2024

Choose a reason for hiding this comment

Uh oh!

willie-yao Jun 7, 2024

Choose a reason for hiding this comment

Uh oh!

alexander-demicev left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alexander-demicev Jun 7, 2024

Choose a reason for hiding this comment

Uh oh!

willie-yao Jun 10, 2024

Choose a reason for hiding this comment

Uh oh!

alexander-demicev Jun 21, 2024

Choose a reason for hiding this comment

Uh oh!

alexander-demicev commented Jun 21, 2024

Uh oh!

furkatgofurov7 commented Jun 24, 2024

Uh oh!

willie-yao commented Jun 25, 2024

Uh oh!

willie-yao commented Jun 25, 2024

Uh oh!

willie-yao commented Jun 25, 2024

Uh oh!

willie-yao commented Jun 27, 2024

Uh oh!

alexander-demicev commented Jul 8, 2024

Uh oh!

alexander-demicev left a comment

Choose a reason for hiding this comment

Uh oh!

k8s-ci-robot commented Jul 8, 2024

Uh oh!

Fedosin left a comment

Choose a reason for hiding this comment

Uh oh!

k8s-ci-robot commented Jul 8, 2024

Uh oh!

Uh oh!

Uh oh!

willie-yao commented May 31, 2024 •

edited

Loading

netlify bot commented May 31, 2024 •

edited

Loading