Skip to content

Commit 08d311a

Browse files
acornett21acornett21
committed
adding changelog and migrations for k8s 1.30 and kubebuilderv4 work
Signed-off-by: Adam D. Cornett <[email protected]>
1 parent 28c1bcb commit 08d311a

File tree

1 file changed

+354
-0
lines changed

1 file changed

+354
-0
lines changed

Diff for: ‎changelog/fragments/02-document-k8s-1-30-changes.yaml

+354
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,354 @@
1+
# entries is a list of entries to include in
2+
# release notes and/or the migration guide
3+
entries:
4+
- description: >
5+
For Go-based, Helm-based and Ansible-based operators this release moves to Kubernetes 1.30 API's and Kubebuilder
6+
v4 Scaffolding, specifically utilizing the v4.1.1 version. The update to Kubebuiler results in some scaffolding
7+
changes which more information can be found below:
8+
- Discontinue usage of [kube-rbac-proxy](https://github.com/brancz/kube-rbac-proxy) in the default
9+
scaffolding of new projects. For further information,
10+
see: [Action Required: Ensure that you no longer use gcr.io/kubebuilder images](https://github.com/kubernetes-sigs/kubebuilder/discussions/3907)
11+
- The `go/v2` or `go/v3` layouts have been removed, you must upgrade to `go/v4` to be compatible with this release and future updates.
12+
To know how to upgrade,check the [migration documentation](https://book.kubebuilder.io/migration/v3vsv4).
13+
- Re-introduces authn/authz protection for the metrics endpoint using [`WithAuthenticationAndAuthorization`](https://github.com/kubernetes-sigs/controller-runtime/blob/v0.18.4/pkg/metrics/filters/filters.go#L35)
14+
provided by controller-runtime. which usage was [discontinued in the project](https://github.com/kubernetes-sigs/kubebuilder/discussions/3907).
15+
Please, ensure that you no longer use the image `gcr.io/kubebuilder/kube-rbac-proxy`. Images provided under `gcr.io/kubebuilder/` will be unavailable from **March 18, 2025**.
16+
To learn more about any of the metrics changes please look at the Kubebuilder book [metrics](https://book.kubebuilder.io/reference/metrics) page.
17+
18+
For `Helm-based` and `Ansible-based` operators, a new flag called `metrics-require-rbac` was introduced into the runtime/binary, to control adding
19+
[`WithAuthenticationAndAuthorization`](https://github.com/kubernetes-sigs/controller-runtime/blob/v0.18.4/pkg/metrics/filters/filters.go#L35)
20+
to `Metrics.FilterProvider` of controller-runtime. This was done to ensure forwards and backwards compatibility of the binary and images with any scaffolded content.
21+
22+
23+
24+
# kind is one of:
25+
# - addition
26+
# - change
27+
# - deprecation
28+
# - removal
29+
# - bugfix
30+
kind: "change"
31+
32+
# Is this a breaking change?
33+
breaking: false
34+
35+
# NOTE: ONLY USE `pull_request_override` WHEN ADDING THIS
36+
# FILE FOR A PREVIOUSLY MERGED PULL_REQUEST!
37+
#
38+
# The generator auto-detects the PR number from the commit
39+
# message in which this file was originally added.
40+
#
41+
# What is the pull request number (without the "#")?
42+
# pull_request_override: 0
43+
44+
45+
# Migration can be defined to automatically add a section to
46+
# the migration guide. This is required for breaking changes.
47+
migration:
48+
header: Upgrade K8s versions to use 1.30 and Kubebuilder v4
49+
body: |
50+
This update has a lot of scaffolding changes due to the removal of [kube-rbac-proxy](https://github.com/brancz/kube-rbac-proxy),
51+
if these migrations become difficult to follow, it might be beneficial to scaffold a net new sample project to compare, or
52+
look at the Kubebuilder book [metrics](https://book.kubebuilder.io/reference/metrics) page.
53+
54+
1) [helm/v1, ansible/v1] Update the kustomize version in your Makefile
55+
```diff
56+
- curl -sSLo - https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize/v5.3.0/kustomize_v5.3.0_$(OS)_$(ARCH).tar.gz | \
57+
+ curl -sSLo - https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize/v5.4.2/kustomize_v5.4.2_$(OS)_$(ARCH).tar.gz | \
58+
```
59+
60+
2) [go/v4] Update your `go.mod` file to upgrade the dependencies and run `go mod tidy` to download them
61+
```go
62+
go 1.22.0
63+
64+
github.com/onsi/ginkgo/v2 v2.17.1
65+
github.com/onsi/gomega v1.32.0
66+
k8s.io/api v0.30.1
67+
k8s.io/apimachinery v0.30.1
68+
k8s.io/client-go v0.30.1
69+
sigs.k8s.io/controller-runtime v0.18.4
70+
```
71+
3) [go/v4] Update your `Makefile` with the below changes:
72+
```diff
73+
- ENVTEST_K8S_VERSION = 1.29.0
74+
+ ENVTEST_K8S_VERSION = 1.30.0
75+
```
76+
77+
```diff
78+
- KUSTOMIZE ?= $(LOCALBIN)/kustomize-$(KUSTOMIZE_VERSION)
79+
- CONTROLLER_GEN ?= $(LOCALBIN)/controller-gen-$(CONTROLLER_TOOLS_VERSION)
80+
- ENVTEST ?= $(LOCALBIN)/setup-envtest-$(ENVTEST_VERSION)
81+
- GOLANGCI_LINT = $(LOCALBIN)/golangci-lint-$(GOLANGCI_LINT_VERSION)
82+
+ KUSTOMIZE ?= $(LOCALBIN)/kustomize
83+
+ CONTROLLER_GEN ?= $(LOCALBIN)/controller-gen
84+
+ ENVTEST ?= $(LOCALBIN)/setup-envtest
85+
+ GOLANGCI_LINT = $(LOCALBIN)/golangci-lint
86+
```
87+
88+
```diff
89+
- KUSTOMIZE_VERSION ?= v5.3.0
90+
- CONTROLLER_TOOLS_VERSION ?= v0.14.0
91+
- ENVTEST_VERSION ?= release-0.17
92+
- GOLANGCI_LINT_VERSION ?= v1.57.2
93+
+ KUSTOMIZE_VERSION ?= v5.4.2
94+
+ CONTROLLER_TOOLS_VERSION ?= v0.15.0
95+
+ ENVTEST_VERSION ?= release-0.18
96+
+ GOLANGCI_LINT_VERSION ?= v1.59.1
97+
```
98+
99+
```diff
100+
- $(call go-install-tool,$(GOLANGCI_LINT),github.com/golangci/golangci-lint/cmd/golangci-lint,${GOLANGCI_LINT_VERSION})
101+
+ $(call go-install-tool,$(GOLANGCI_LINT),github.com/golangci/golangci-lint/cmd/golangci-lint,$(GOLANGCI_LINT_VERSION))
102+
```
103+
104+
```diff
105+
- @[ -f $(1) ] || { \
106+
+ @[ -f "$(1)-$(3)" ] || { \
107+
echo "Downloading $${package}" ;\
108+
+ rm -f $(1) || true ;\
109+
- mv "$$(echo "$(1)" | sed "s/-$(3)$$//")" $(1) ;\
110+
- }
111+
+ mv $(1) $(1)-$(3) ;\
112+
+ } ;\
113+
+ ln -sf $(1)-$(3) $(1)
114+
```
115+
116+
4) [go/v4] Update your `.golangci.yml` with the below changes:
117+
```diff
118+
- exportloopref
119+
+ - ginkgolinter
120+
- prealloc
121+
+ - revive
122+
+
123+
+linters-settings:
124+
+ revive:
125+
+ rules:
126+
+ - name: comment-spacings
127+
```
128+
129+
5) [go/v4] Update your `Dockerfile` file with the below changes:
130+
```diff
131+
- FROM golang:1.21 AS builder
132+
+ FROM golang:1.22 AS builder
133+
```
134+
135+
6) [go/v4] Update your `main.go` file with the below changes:
136+
```diff
137+
"sigs.k8s.io/controller-runtime/pkg/log/zap"
138+
+ "sigs.k8s.io/controller-runtime/pkg/metrics/filters"
139+
140+
var enableHTTP2 bool
141+
- flag.StringVar(&metricsAddr, "metrics-bind-address", ":8080", "The address the metric endpoint binds to.")
142+
+ var tlsOpts []func(*tls.Config)
143+
+ flag.StringVar(&metricsAddr, "metrics-bind-address", "0", "The address the metrics endpoint binds to. "+
144+
+ "Use :8443 for HTTPS or :8080 for HTTP, or leave as 0 to disable the metrics service.")
145+
flag.StringVar(&probeAddr, "health-probe-bind-address", ":8081", "The address the probe endpoint binds to.")
146+
flag.BoolVar(&enableLeaderElection, "leader-elect", false,
147+
"Enable leader election for controller manager. "+
148+
"Enabling this will ensure there is only one active controller manager.")
149+
- flag.BoolVar(&secureMetrics, "metrics-secure", false,
150+
- "If set the metrics endpoint is served securely")
151+
+ flag.BoolVar(&secureMetrics, "metrics-secure", true,
152+
+ "If set, the metrics endpoint is served securely via HTTPS. Use --metrics-secure=false to use HTTP instead.")
153+
154+
- tlsOpts := []func(*tls.Config){}
155+
156+
+ // Metrics endpoint is enabled in 'config/default/kustomization.yaml'. The Metrics options configure the server.
157+
+ // More info:
158+
+ // - https://pkg.go.dev/sigs.k8s.io/[email protected]/pkg/metrics/server
159+
+ // - https://book.kubebuilder.io/reference/metrics.html
160+
+ metricsServerOptions := metricsserver.Options{
161+
+ BindAddress: metricsAddr,
162+
+ SecureServing: secureMetrics,
163+
+ // TODO(user): TLSOpts is used to allow configuring the TLS config used for the server. If certificates are
164+
+ // not provided, self-signed certificates will be generated by default. This option is not recommended for
165+
+ // production environments as self-signed certificates do not offer the same level of trust and security
166+
+ // as certificates issued by a trusted Certificate Authority (CA). The primary risk is potentially allowing
167+
+ // unauthorized access to sensitive metrics data. Consider replacing with CertDir, CertName, and KeyName
168+
+ // to provide certificates, ensuring the server communicates using trusted and secure certificates.
169+
+ TLSOpts: tlsOpts,
170+
+ }
171+
+
172+
+ if secureMetrics {
173+
+ // FilterProvider is used to protect the metrics endpoint with authn/authz.
174+
+ // These configurations ensure that only authorized users and service accounts
175+
+ // can access the metrics endpoint. The RBAC are configured in 'config/rbac/kustomization.yaml'. More info:
176+
+ // https://pkg.go.dev/sigs.k8s.io/[email protected]/pkg/metrics/filters#WithAuthenticationAndAuthorization
177+
+ metricsServerOptions.FilterProvider = filters.WithAuthenticationAndAuthorization
178+
+ }
179+
+
180+
mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
181+
- Scheme: scheme,
182+
- Metrics: metricsserver.Options{
183+
- BindAddress: metricsAddr,
184+
- SecureServing: secureMetrics,
185+
- TLSOpts: tlsOpts,
186+
- },
187+
+ Scheme: scheme,
188+
+ Metrics: metricsServerOptions,
189+
```
190+
7) [go/v4, helm/v1, ansible/v1] Update your `/config/default/kustomization.yaml` file with the below changes:
191+
```diff
192+
# [PROMETHEUS] To enable prometheus monitor, uncomment all sections with 'PROMETHEUS'.
193+
#- ../prometheus
194+
+# [METRICS] Expose the controller manager metrics service.
195+
+- metrics_service.yaml
196+
197+
+# Uncomment the patches line if you enable Metrics, and/or are using webhooks and cert-manager
198+
patches:
199+
-# Protect the /metrics endpoint by putting it behind auth.
200+
-# If you want your controller-manager to expose the /metrics
201+
-# endpoint w/o any authn/z, please comment the following line.
202+
-- path: manager_auth_proxy_patch.yaml
203+
+# [METRICS] The following patch will enable the metrics endpoint using HTTPS and the port :8443.
204+
+# More info: https://book.kubebuilder.io/reference/metrics
205+
+- path: manager_metrics_patch.yaml
206+
+ target:
207+
+ kind: Deployment
208+
```
209+
210+
8) [go/v4, helm/v1, ansible/v1] Remove `/config/default/manager_auth_proxy_patch.yaml` and `/config/default/manager_config_patch.yaml` files.
211+
212+
9) [go/v4, helm/v1, ansible/v1] Add `/config/default/manager_metrics_patch.yaml` file with the below changes:
213+
```diff
214+
# This patch adds the args to allow exposing the metrics endpoint using HTTPS
215+
- op: add
216+
path: /spec/template/spec/containers/0/args/0
217+
value: --metrics-bind-address=:8443
218+
```
219+
220+
10) [helm/v1, ansible/v1] Update `/config/default/manager_metrics_patch.yaml` file with the below changes:
221+
```diff
222+
# This patch adds the args to allow securing the metrics endpoint
223+
- op: add
224+
path: /spec/template/spec/containers/0/args/0
225+
value: --metrics-secure
226+
# This patch adds the args to allow RBAC-based authn/authz the metrics endpoint
227+
- op: add
228+
path: /spec/template/spec/containers/0/args/0
229+
value: --metrics-require-rbac
230+
```
231+
232+
11) [go/v4, helm/v1, ansible/v1] Add `/config/default/metrics_service.yaml` file with the below changes:
233+
```diff
234+
apiVersion: v1
235+
kind: Service
236+
metadata:
237+
labels:
238+
control-plane: controller-manager
239+
app.kubernetes.io/name: <operator-name>
240+
app.kubernetes.io/managed-by: kustomize
241+
name: controller-manager-metrics-service
242+
namespace: system
243+
spec:
244+
ports:
245+
- name: https
246+
port: 8443
247+
protocol: TCP
248+
targetPort: 8443
249+
selector:
250+
control-plane: controller-manager
251+
```
252+
253+
12) [go/v4, helm/v1, ansible/v1] Update your `/config/manager/manager.yaml` file with the below changes (**Note: The port for ansible is 6789**):
254+
```diff
255+
- --leader-elect
256+
+ - --health-probe-bind-address=:8081
257+
```
258+
259+
13) [go/v4, helm/v1, ansible/v1] Update your `/config/prometheus/monitor/yaml` file with the below changes:
260+
```diff
261+
- path: /metrics
262+
- port: https
263+
+ port: https # Ensure this is the name of the port that exposes HTTPS metrics
264+
tlsConfig:
265+
+ # TODO(user): The option insecureSkipVerify: true is not recommended for production since it disables
266+
+ # certificate verification. This poses a significant security risk by making the system vulnerable to
267+
+ # man-in-the-middle attacks, where an attacker could intercept and manipulate the communication between
268+
+ # Prometheus and the monitored services. This could lead to unauthorized access to sensitive metrics data,
269+
+ # compromising the integrity and confidentiality of the information.
270+
+ # Please use the following options for secure configurations:
271+
+ # caFile: /etc/metrics-certs/ca.crt
272+
+ # certFile: /etc/metrics-certs/tls.crt
273+
+ # keyFile: /etc/metrics-certs/tls.key
274+
insecureSkipVerify: true
275+
```
276+
277+
14) [go/v4, helm/v1, ansible/v1] Remove the following files from `/config/rbac`
278+
```diff
279+
- auth_proxy_client_clusterrole.yaml
280+
- auth_proxy_role.yaml
281+
- auth_proxy_role_binding.yaml
282+
- auth_proxy_service.yaml
283+
```
284+
285+
15) [go/v4, helm/v1, ansible/v1] Update your `/config/rbac/kustomization.yaml` file with the below changes:
286+
```diff
287+
- leader_election_role_binding.yaml
288+
- # Comment the following 4 lines if you want to disable
289+
- # the auth proxy (https://github.com/brancz/kube-rbac-proxy)
290+
- # which protects your /metrics endpoint.
291+
- - auth_proxy_service.yaml
292+
- - auth_proxy_role.yaml
293+
- - auth_proxy_role_binding.yaml
294+
- - auth_proxy_client_clusterrole.yaml
295+
+ # The following RBAC configurations are used to protect
296+
+ # the metrics endpoint with authn/authz. These configurations
297+
+ # ensure that only authorized users and service accounts
298+
+ # can access the metrics endpoint. Comment the following
299+
+ # permissions if you want to disable this protection.
300+
+ # More info: https://book.kubebuilder.io/reference/metrics.html
301+
+ - metrics_auth_role.yaml
302+
+ - metrics_auth_role_binding.yaml
303+
+ - metrics_reader_role.yaml
304+
```
305+
306+
16) [go/v4, helm/v1, ansible/v1] Add `/config/rbac/metrics_auth_role.yaml` file with the below changes:
307+
```diff
308+
apiVersion: rbac.authorization.k8s.io/v1
309+
kind: ClusterRole
310+
metadata:
311+
name: metrics-auth-role
312+
rules:
313+
- apiGroups:
314+
- authentication.k8s.io
315+
resources:
316+
- tokenreviews
317+
verbs:
318+
- create
319+
- apiGroups:
320+
- authorization.k8s.io
321+
resources:
322+
- subjectaccessreviews
323+
verbs:
324+
- create
325+
```
326+
327+
17) [go/v4, helm/v1, ansible/v1] Add `/config/rbac/metrics_auth_role_binding.yaml` file with the below changes:
328+
```diff
329+
apiVersion: rbac.authorization.k8s.io/v1
330+
kind: ClusterRoleBinding
331+
metadata:
332+
name: metrics-auth-rolebinding
333+
roleRef:
334+
apiGroup: rbac.authorization.k8s.io
335+
kind: ClusterRole
336+
name: metrics-auth-role
337+
subjects:
338+
- kind: ServiceAccount
339+
name: controller-manager
340+
namespace: system
341+
```
342+
343+
18) [go/v4, helm/v1, ansible/v1] Add `/config/rbac/metrics_reader_role.yaml` file with the below changes:
344+
```diff
345+
apiVersion: rbac.authorization.k8s.io/v1
346+
kind: ClusterRole
347+
metadata:
348+
name: metrics-reader
349+
rules:
350+
- nonResourceURLs:
351+
- "/metrics"
352+
verbs:
353+
- get
354+
```

0 commit comments

Comments
 (0)