Removal of old apiVersions in CRDs

[sbueringer]

Problem statement

As we evolve our CRDs over time we regularly bump our apiVersions. While we have an established process to add new apiVersions in Cluster API, removal of apiVersions is still problematic.

Before we can remove an apiVersion from a CRD we have to go through the following steps:

1. Ensure all custom resources can still be read from etcd after the apiVersion is removed ("Storage version migration")

This is typically done by writing all custom resources with the current storage version. More information can be found in Versions in CRDs.

In Cluster API today, this is only implemented as part of the clusterctl upgrade command. If clusterctl is not used, folks have to write their own full custom implementation or build on top of the builtin storage migration (alpha since v1.30).

Note: Storage version migration should be run as soon as a new apiVersion becomes storage version to minimize conversion webhook calls.

2. Remove the old apiVersion from managedFields of all custom resources ("ManagedField cleanup")

Kubernetes stores field ownership information by apiVersion in managedFields. Unfortunately, there is no builtin logic that removes managedFields of an apiVersion when that apiVersion is removed from the CRD. If there are still managedFields with a removed apiVersion any subsequent apply requests will fail:

• server-side apply will result in request to convert CR to an invalid group/version errors (see SSA failure after removal of old API versions #10051 & managedField entry for a removed API version prevent further resource updates kubernetes/kubernetes#111937)
• client-side apply will lead to a removal of all managedFields and the apiserver will log this error: SHOULD NOT HAPPEN failed to update managedFields

Note: managedField cleanup should be run as soon as an apiVersion is not served anymore to minimize conversion webhook calls. As long as an apiVersion is still served, users can still apply with this apiVersion and then the corresponding managedFields are needed to properly execute the apply.

Why is this important now

Expand for more details

For the following reasons we want to remove old apiVersions as soon as possible:

Maintenance effort

We have to keep the Go types of the old apiVersions around. We also have to adjust the conversion implementation whenever we add a new field to our current API.

Increased resource usage through conversion requests

As long as the old apiVersions are part of our CRDs we will get a significant number of requests on the conversion webhooks:

• kube-apiserver creates watches for every apiVersion (even if not served) and is calling our conversion webhooks. For more information see: CRD conversion webhooks should not be called for unused apiVersions kubernetes/kubernetes#129979
• kube-apiserver calls the conversion webhook if a custom resource has managedField entries with old apiVersions

Implementation

A few notes:

• The solution should be also available for folks that don't use clusterctl
• It should be easy for providers to re-use the implementation for their own CRDs
• It should be possible to disable storage version migration and/or managedField cleanup for cases where folks want to take care of these themselves

Idea:

• Implement a controller / reconciler that can be embedded in core CAPI / providers to run storage version migration and managedField cleanup for the providers' CRDs

For more implementation details, see: #11889

Removal of v1alpha3 & v1alpha4 and v1beta1 apiVersions

Context:

• CAPI supports 3 versions: for 2 versions we regularly release patch releases, for the 3rd one we create emergency patches on demand
• CAPI currently tests up to n-3 => n upgrades.
  • Note: This is not well documented at the moment, but we should follow-up (see 📖 Cleanup and improve version support page #11914).
  • Note: The n-3 => n policy in CAPI aligns to the corresponding policy in Kubernetes
• We want to make sure that removal of old apiVersions do not break the n-3 => n upgrade path. This means that we have to keep apiVersions around long enough that "storage version migration" and "managedField cleanup" are run.

v1alpha3 & v1alpha4

CAPI	Release date	v1alpha3 + v1alpha4	Notes
v1.9	Dec 24	Served: false
v1.10	April 25	Served: false	CRD migrator added
v1.11	August 25	Served: false
v1.12	December 25	Served: false
v1.13	April 26		v1alpha3 + v1alpha4 removed
v1.14	August 26
v1.15	December 26

Notes:

• v1.10-v1.12: We have to keep v1alpha3 + v1alpha4 around for 3 versions after the CRD migrator has been added, to ensure that managedField cleanup is run even if someone upgrades from n-3 => n

v1beta1

CAPI	Release date	v1beta1	v1beta2	Notes
v1.9	Dec 24	Served: true, Storage
v1.10	April 25	Served: true, Storage
v1.11	August 25	Served: true	Served: true, Storage	v1beta2 added
v1.12	December 25	Served: true	Served: true, Storage
v1.13	April 26	Served: true	Served: true, Storage
v1.14	August 26	Served: false	Served: true, Storage	v1beta1 unserved
v1.15	December 26	Served: false	Served: true, Storage
v1.16	April 27	Served: false	Served: true, Storage
v1.17	August 27	Served: false	Served: true, Storage
v1.18	December 27		Served: true, Storage	v1beta1 removed

Notes:

• v1.11-v1.13: We have to keep v1beta1 served for 3 versions after introduction of v1beta2 according to the Kubernetes deprecation policy
• v1.14-v1.17: We have to keep v1beta1 around for 3 versions after it was unserved, to ensure that managedField cleanup is run even if someone upgrades from n-3 => n.
  • Note: We want to keep v1beta1 around for one additional release so that folks have 1 buffer release where they can revert v1beta1 back to served if they need more time to pick up v1beta2 (we did the same for v1alpha3 + v1alpha4 in the past).

Tasks:

• ✨ Add CRD migrator, deprecate clusterctl upgrade CRD storage version migration #11889
  • Follow-up: 🐛 CRD migration: Fix cases where update validation fails #11991
• Move v1alpha3 & v1alpha4 + v1beta1 timelines to their own issues so we can track it over the years
  • Created Remove v1alpha3 + v1alpha4 apiVersions #11919
  • Created Remove v1beta1 apiVersion #11920
• Follow-up: Improve version documentation: 📖 Cleanup and improve version support page #11914
• Bring up our points to k/k where it wasn't already done (we can't rely on k/k improvements for now as we would have to wait a few years until we can assume they are GA & available)
  • [Stefan] I will track this on my side. For now we shared on the managedField issue what we are doing.

[sbueringer]

cc @fabriziopandini @chrischdi @enxebre @vincepri @JoelSpeed

(I'll bring it up on Wednesday in the office hours)

[fabriziopandini]

Thanks for doing the hard work on this issue, +1 for me

[chrischdi]

Also +1 from my side.

[sbueringer]

I created issues for v1alpha3 / v1alpha4 / v1beta1 removal and updated the task list. I think we can keep this issue closed.

Thx everyone for the quick reviews!