Skip to content

Commit d8ff5bd

Browse files
acornett21acornett21
committed
adding changelog and migrations for k8s 1.30 and kubebuilderv4 work
Signed-off-by: Adam D. Cornett <[email protected]>
1 parent 28c1bcb commit d8ff5bd

File tree

1 file changed

+351
-0
lines changed

1 file changed

+351
-0
lines changed

Diff for: ‎changelog/fragments/02-document-k8s-1-30-changes.yaml

+351
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,351 @@
1+
# entries is a list of entries to include in
2+
# release notes and/or the migration guide
3+
entries:
4+
- description: >
5+
For Go-based, Helm-based and Ansible-based operators this release moves to Kubernetes 1.30 API's and Kubebuilder
6+
v4 Scaffolding, specifically utilizing the v4.1.1 version. The update to Kubebuiler results in some scaffolding
7+
changes which more information can be found below:
8+
- Discontinue usage of [kube-rbac-proxy](https://github.com/brancz/kube-rbac-proxy) in the default
9+
scaffolding of new projects. For further information,
10+
see: [Action Required: Ensure that you no longer use gcr.io/kubebuilder images](https://github.com/kubernetes-sigs/kubebuilder/discussions/3907)
11+
- The `go/v2` or `go/v3` layouts have been removed, you must upgrade to `go/v4` to be compatible with this release and future updates.
12+
To know how to upgrade,check the [migration documentation](https://book.kubebuilder.io/migration/v3vsv4).
13+
- Re-introduces authn/authz protection for the metrics endpoint using [`WithAuthenticationAndAuthorization`](https://github.com/kubernetes-sigs/controller-runtime/blob/v0.18.4/pkg/metrics/filters/filters.go#L35)
14+
provided by controller-runtime instead of kube-rbac-proxy; which usage was [discontinued in the project](https://github.com/kubernetes-sigs/kubebuilder/discussions/3907).
15+
Please, ensure that you no longer use the image `gcr.io/kubebuilder/kube-rbac-proxy`. Images provided under `gcr.io/kubebuilder/` will be unavailable from **March 18, 2025**.
16+
To learn more about any of the metrics changes please look at the Kubebuilder book [metrics](https://book.kubebuilder.io/reference/metrics) page.
17+
18+
For `Helm-based` and `Ansible-based` operators, a new flag called `metrics-require-rbac` was introduced into the runtime/binary, to control adding
19+
[`WithAuthenticationAndAuthorization`](https://github.com/kubernetes-sigs/controller-runtime/blob/v0.18.4/pkg/metrics/filters/filters.go#L35)
20+
to `Metrics.FilterProvider` of controller-runtime. This was done to ensure forwards and backwards compatibility of the binary and images with any scaffolded content.
21+
22+
# kind is one of:
23+
# - addition
24+
# - change
25+
# - deprecation
26+
# - removal
27+
# - bugfix
28+
kind: "change"
29+
30+
# Is this a breaking change?
31+
breaking: false
32+
33+
# NOTE: ONLY USE `pull_request_override` WHEN ADDING THIS
34+
# FILE FOR A PREVIOUSLY MERGED PULL_REQUEST!
35+
#
36+
# The generator auto-detects the PR number from the commit
37+
# message in which this file was originally added.
38+
#
39+
# What is the pull request number (without the "#")?
40+
# pull_request_override: 0
41+
42+
43+
# Migration can be defined to automatically add a section to
44+
# the migration guide. This is required for breaking changes.
45+
migration:
46+
header: Upgrade K8s versions to use 1.30 and Kubebuilder v4
47+
body: |
48+
This update has a lot of scaffolding changes due to the removal of [kube-rbac-proxy](https://github.com/brancz/kube-rbac-proxy),
49+
if these migrations become difficult to follow, it might be beneficial to scaffold a net new sample project to compare.
50+
51+
1) [helm/v1, ansible/v1] Update the kustomize version in your Makefile
52+
```diff
53+
- curl -sSLo - https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize/v5.3.0/kustomize_v5.3.0_$(OS)_$(ARCH).tar.gz | \
54+
+ curl -sSLo - https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize/v5.4.2/kustomize_v5.4.2_$(OS)_$(ARCH).tar.gz | \
55+
```
56+
57+
2) [go/v4] Update your `go.mod` file to upgrade the dependencies and run `go mod tidy` to download them
58+
```go
59+
go 1.22.0
60+
61+
github.com/onsi/ginkgo/v2 v2.17.1
62+
github.com/onsi/gomega v1.32.0
63+
k8s.io/api v0.30.1
64+
k8s.io/apimachinery v0.30.1
65+
k8s.io/client-go v0.30.1
66+
sigs.k8s.io/controller-runtime v0.18.4
67+
```
68+
3) [go/v4] Update your `Makefile` with the below changes:
69+
```diff
70+
- ENVTEST_K8S_VERSION = 1.29.0
71+
+ ENVTEST_K8S_VERSION = 1.30.0
72+
```
73+
74+
```diff
75+
- KUSTOMIZE ?= $(LOCALBIN)/kustomize-$(KUSTOMIZE_VERSION)
76+
- CONTROLLER_GEN ?= $(LOCALBIN)/controller-gen-$(CONTROLLER_TOOLS_VERSION)
77+
- ENVTEST ?= $(LOCALBIN)/setup-envtest-$(ENVTEST_VERSION)
78+
- GOLANGCI_LINT = $(LOCALBIN)/golangci-lint-$(GOLANGCI_LINT_VERSION)
79+
+ KUSTOMIZE ?= $(LOCALBIN)/kustomize
80+
+ CONTROLLER_GEN ?= $(LOCALBIN)/controller-gen
81+
+ ENVTEST ?= $(LOCALBIN)/setup-envtest
82+
+ GOLANGCI_LINT = $(LOCALBIN)/golangci-lint
83+
```
84+
85+
```diff
86+
- KUSTOMIZE_VERSION ?= v5.3.0
87+
- CONTROLLER_TOOLS_VERSION ?= v0.14.0
88+
- ENVTEST_VERSION ?= release-0.17
89+
- GOLANGCI_LINT_VERSION ?= v1.57.2
90+
+ KUSTOMIZE_VERSION ?= v5.4.2
91+
+ CONTROLLER_TOOLS_VERSION ?= v0.15.0
92+
+ ENVTEST_VERSION ?= release-0.18
93+
+ GOLANGCI_LINT_VERSION ?= v1.59.1
94+
```
95+
96+
```diff
97+
- $(call go-install-tool,$(GOLANGCI_LINT),github.com/golangci/golangci-lint/cmd/golangci-lint,${GOLANGCI_LINT_VERSION})
98+
+ $(call go-install-tool,$(GOLANGCI_LINT),github.com/golangci/golangci-lint/cmd/golangci-lint,$(GOLANGCI_LINT_VERSION))
99+
```
100+
101+
```diff
102+
- @[ -f $(1) ] || { \
103+
+ @[ -f "$(1)-$(3)" ] || { \
104+
echo "Downloading $${package}" ;\
105+
+ rm -f $(1) || true ;\
106+
- mv "$$(echo "$(1)" | sed "s/-$(3)$$//")" $(1) ;\
107+
- }
108+
+ mv $(1) $(1)-$(3) ;\
109+
+ } ;\
110+
+ ln -sf $(1)-$(3) $(1)
111+
```
112+
113+
4) [go/v4] Update your `.golangci.yml` with the below changes:
114+
```diff
115+
- exportloopref
116+
+ - ginkgolinter
117+
- prealloc
118+
+ - revive
119+
+
120+
+linters-settings:
121+
+ revive:
122+
+ rules:
123+
+ - name: comment-spacings
124+
```
125+
126+
5) [go/v4] Update your `Dockerfile` file with the below changes:
127+
```diff
128+
- FROM golang:1.21 AS builder
129+
+ FROM golang:1.22 AS builder
130+
```
131+
132+
6) [go/v4] Update your `main.go` file with the below changes:
133+
```diff
134+
"sigs.k8s.io/controller-runtime/pkg/log/zap"
135+
+ "sigs.k8s.io/controller-runtime/pkg/metrics/filters"
136+
137+
var enableHTTP2 bool
138+
- flag.StringVar(&metricsAddr, "metrics-bind-address", ":8080", "The address the metric endpoint binds to.")
139+
+ var tlsOpts []func(*tls.Config)
140+
+ flag.StringVar(&metricsAddr, "metrics-bind-address", "0", "The address the metrics endpoint binds to. "+
141+
+ "Use :8443 for HTTPS or :8080 for HTTP, or leave as 0 to disable the metrics service.")
142+
flag.StringVar(&probeAddr, "health-probe-bind-address", ":8081", "The address the probe endpoint binds to.")
143+
flag.BoolVar(&enableLeaderElection, "leader-elect", false,
144+
"Enable leader election for controller manager. "+
145+
"Enabling this will ensure there is only one active controller manager.")
146+
- flag.BoolVar(&secureMetrics, "metrics-secure", false,
147+
- "If set the metrics endpoint is served securely")
148+
+ flag.BoolVar(&secureMetrics, "metrics-secure", true,
149+
+ "If set, the metrics endpoint is served securely via HTTPS. Use --metrics-secure=false to use HTTP instead.")
150+
151+
- tlsOpts := []func(*tls.Config){}
152+
153+
+ // Metrics endpoint is enabled in 'config/default/kustomization.yaml'. The Metrics options configure the server.
154+
+ // More info:
155+
+ // - https://pkg.go.dev/sigs.k8s.io/[email protected]/pkg/metrics/server
156+
+ // - https://book.kubebuilder.io/reference/metrics.html
157+
+ metricsServerOptions := metricsserver.Options{
158+
+ BindAddress: metricsAddr,
159+
+ SecureServing: secureMetrics,
160+
+ // TODO(user): TLSOpts is used to allow configuring the TLS config used for the server. If certificates are
161+
+ // not provided, self-signed certificates will be generated by default. This option is not recommended for
162+
+ // production environments as self-signed certificates do not offer the same level of trust and security
163+
+ // as certificates issued by a trusted Certificate Authority (CA). The primary risk is potentially allowing
164+
+ // unauthorized access to sensitive metrics data. Consider replacing with CertDir, CertName, and KeyName
165+
+ // to provide certificates, ensuring the server communicates using trusted and secure certificates.
166+
+ TLSOpts: tlsOpts,
167+
+ }
168+
+
169+
+ if secureMetrics {
170+
+ // FilterProvider is used to protect the metrics endpoint with authn/authz.
171+
+ // These configurations ensure that only authorized users and service accounts
172+
+ // can access the metrics endpoint. The RBAC are configured in 'config/rbac/kustomization.yaml'. More info:
173+
+ // https://pkg.go.dev/sigs.k8s.io/[email protected]/pkg/metrics/filters#WithAuthenticationAndAuthorization
174+
+ metricsServerOptions.FilterProvider = filters.WithAuthenticationAndAuthorization
175+
+ }
176+
+
177+
mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
178+
- Scheme: scheme,
179+
- Metrics: metricsserver.Options{
180+
- BindAddress: metricsAddr,
181+
- SecureServing: secureMetrics,
182+
- TLSOpts: tlsOpts,
183+
- },
184+
+ Scheme: scheme,
185+
+ Metrics: metricsServerOptions,
186+
```
187+
7) [go/v4, helm/v1, ansible/v1] Update your `/config/default/kustomization.yaml` file with the below changes:
188+
```diff
189+
# [PROMETHEUS] To enable prometheus monitor, uncomment all sections with 'PROMETHEUS'.
190+
#- ../prometheus
191+
+# [METRICS] Expose the controller manager metrics service.
192+
+- metrics_service.yaml
193+
194+
+# Uncomment the patches line if you enable Metrics, and/or are using webhooks and cert-manager
195+
patches:
196+
-# Protect the /metrics endpoint by putting it behind auth.
197+
-# If you want your controller-manager to expose the /metrics
198+
-# endpoint w/o any authn/z, please comment the following line.
199+
-- path: manager_auth_proxy_patch.yaml
200+
+# [METRICS] The following patch will enable the metrics endpoint using HTTPS and the port :8443.
201+
+# More info: https://book.kubebuilder.io/reference/metrics
202+
+- path: manager_metrics_patch.yaml
203+
+ target:
204+
+ kind: Deployment
205+
```
206+
207+
8) [go/v4, helm/v1, ansible/v1] Remove `/config/default/manager_auth_proxy_patch.yaml` and `/config/default/manager_config_patch.yaml` files.
208+
209+
9) [go/v4, helm/v1, ansible/v1] Add `/config/default/manager_metrics_patch.yaml` file with the below changes:
210+
```diff
211+
# This patch adds the args to allow exposing the metrics endpoint using HTTPS
212+
- op: add
213+
path: /spec/template/spec/containers/0/args/0
214+
value: --metrics-bind-address=:8443
215+
```
216+
217+
10) [helm/v1, ansible/v1] Update `/config/default/manager_metrics_patch.yaml` file with the below changes:
218+
```diff
219+
# This patch adds the args to allow securing the metrics endpoint
220+
- op: add
221+
path: /spec/template/spec/containers/0/args/0
222+
value: --metrics-secure
223+
# This patch adds the args to allow RBAC-based authn/authz the metrics endpoint
224+
- op: add
225+
path: /spec/template/spec/containers/0/args/0
226+
value: --metrics-require-rbac
227+
```
228+
229+
11) [go/v4, helm/v1, ansible/v1] Add `/config/default/metrics_service.yaml` file with the below changes:
230+
```diff
231+
apiVersion: v1
232+
kind: Service
233+
metadata:
234+
labels:
235+
control-plane: controller-manager
236+
app.kubernetes.io/name: <operator-name>
237+
app.kubernetes.io/managed-by: kustomize
238+
name: controller-manager-metrics-service
239+
namespace: system
240+
spec:
241+
ports:
242+
- name: https
243+
port: 8443
244+
protocol: TCP
245+
targetPort: 8443
246+
selector:
247+
control-plane: controller-manager
248+
```
249+
250+
12) [go/v4, helm/v1, ansible/v1] Update your `/config/manager/manager.yaml` file with the below changes (**Note: The port for ansible is 6789**):
251+
```diff
252+
- --leader-elect
253+
+ - --health-probe-bind-address=:8081
254+
```
255+
256+
13) [go/v4, helm/v1, ansible/v1] Update your `/config/prometheus/monitor/yaml` file with the below changes:
257+
```diff
258+
- path: /metrics
259+
- port: https
260+
+ port: https # Ensure this is the name of the port that exposes HTTPS metrics
261+
tlsConfig:
262+
+ # TODO(user): The option insecureSkipVerify: true is not recommended for production since it disables
263+
+ # certificate verification. This poses a significant security risk by making the system vulnerable to
264+
+ # man-in-the-middle attacks, where an attacker could intercept and manipulate the communication between
265+
+ # Prometheus and the monitored services. This could lead to unauthorized access to sensitive metrics data,
266+
+ # compromising the integrity and confidentiality of the information.
267+
+ # Please use the following options for secure configurations:
268+
+ # caFile: /etc/metrics-certs/ca.crt
269+
+ # certFile: /etc/metrics-certs/tls.crt
270+
+ # keyFile: /etc/metrics-certs/tls.key
271+
insecureSkipVerify: true
272+
```
273+
274+
14) [go/v4, helm/v1, ansible/v1] Remove the following files from `/config/rbac`
275+
```diff
276+
- auth_proxy_client_clusterrole.yaml
277+
- auth_proxy_role.yaml
278+
- auth_proxy_role_binding.yaml
279+
- auth_proxy_service.yaml
280+
```
281+
282+
15) [go/v4, helm/v1, ansible/v1] Update your `/config/rbac/kustomization.yaml` file with the below changes:
283+
```diff
284+
- leader_election_role_binding.yaml
285+
- # Comment the following 4 lines if you want to disable
286+
- # the auth proxy (https://github.com/brancz/kube-rbac-proxy)
287+
- # which protects your /metrics endpoint.
288+
- - auth_proxy_service.yaml
289+
- - auth_proxy_role.yaml
290+
- - auth_proxy_role_binding.yaml
291+
- - auth_proxy_client_clusterrole.yaml
292+
+ # The following RBAC configurations are used to protect
293+
+ # the metrics endpoint with authn/authz. These configurations
294+
+ # ensure that only authorized users and service accounts
295+
+ # can access the metrics endpoint. Comment the following
296+
+ # permissions if you want to disable this protection.
297+
+ # More info: https://book.kubebuilder.io/reference/metrics.html
298+
+ - metrics_auth_role.yaml
299+
+ - metrics_auth_role_binding.yaml
300+
+ - metrics_reader_role.yaml
301+
```
302+
303+
16) [go/v4, helm/v1, ansible/v1] Add `/config/rbac/metrics_auth_role.yaml` file with the below changes:
304+
```diff
305+
apiVersion: rbac.authorization.k8s.io/v1
306+
kind: ClusterRole
307+
metadata:
308+
name: metrics-auth-role
309+
rules:
310+
- apiGroups:
311+
- authentication.k8s.io
312+
resources:
313+
- tokenreviews
314+
verbs:
315+
- create
316+
- apiGroups:
317+
- authorization.k8s.io
318+
resources:
319+
- subjectaccessreviews
320+
verbs:
321+
- create
322+
```
323+
324+
17) [go/v4, helm/v1, ansible/v1] Add `/config/rbac/metrics_auth_role_binding.yaml` file with the below changes:
325+
```diff
326+
apiVersion: rbac.authorization.k8s.io/v1
327+
kind: ClusterRoleBinding
328+
metadata:
329+
name: metrics-auth-rolebinding
330+
roleRef:
331+
apiGroup: rbac.authorization.k8s.io
332+
kind: ClusterRole
333+
name: metrics-auth-role
334+
subjects:
335+
- kind: ServiceAccount
336+
name: controller-manager
337+
namespace: system
338+
```
339+
340+
18) [go/v4, helm/v1, ansible/v1] Add `/config/rbac/metrics_reader_role.yaml` file with the below changes:
341+
```diff
342+
apiVersion: rbac.authorization.k8s.io/v1
343+
kind: ClusterRole
344+
metadata:
345+
name: metrics-reader
346+
rules:
347+
- nonResourceURLs:
348+
- "/metrics"
349+
verbs:
350+
- get
351+
```

0 commit comments

Comments
 (0)