Skip to content

Commit 80f592e

Browse files
estrozestroz
committed
doc/user/migrating-existing-apis.md: document API version migrations
1 parent 560208d commit 80f592e

File tree

1 file changed

+382
-0
lines changed

1 file changed

+382
-0
lines changed

‎doc/user/migrating-existing-apis.md

+382
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,382 @@
1+
# Migrating Existing Kubernetes API's
2+
3+
Kubernetes API's are assumed to evolve over time, hence the well-defined API [versioning scheme][k8s-versioning]. Upgrading your operator's API's can be a non-trivial task, one that will involve changing quite a few source files and manifests. This document aims to identify the complexities of migrating an operator project's API using examples from existing operators.
4+
5+
While examples in this guide follow particular types of API migrations, most of the documented migration steps can be generalized to all migration types. A thorough discussion of migration types for a particular project type (Go, Ansible, Helm) is found at the end of each project type's section.
6+
7+
## Golang: Upgrading one Kind to a new Version from a Version with multiple Kind's
8+
9+
**Scenario:** your Go operator test-operator has one API version `v1` for group `operators.example.com`. You would like to migrate (upgrade) one kind `CatalogSourceConfig` to `v2` while keeping the other `v1` kind `OperatorGroup` in `v1`. These kind's will remain in group `operators.example.com`. Your project structure looks like the following:
10+
11+
```console
12+
$ tree pkg/apis
13+
pkg/apis/
14+
├── addtoscheme_operators_v1.go
15+
├── apis.go
16+
└── operators
17+
└── v1
18+
├── catalogsourceconfig_types.go
19+
├── catalogsourceconfig.go
20+
├── doc.go
21+
├── operatorgroup_types.go
22+
├── operatorgroup.go
23+
├── phase.go
24+
├── phase_types.go
25+
├── register.go
26+
├── shared.go
27+
├── zz_generated.deepcopy.go
28+
├── zz_generated.openapi.go
29+
```
30+
31+
Relevant files:
32+
33+
- `catalogsourceconfig_types.go` and `catalogsourceconfig.go` contain types and functions used by API kind type `CatalogSourceConfig`.
34+
- `operatorgroup_types.go` and `operatorgroup.go` contain types and functions used by API kind type `OperatorGroup`.
35+
- `phase_types.go` and `phase.go` contain types and functions used by *non-API* type `Phase`, which is used by both `CatalogSourceConfig` and `OperatorGroup` types.
36+
- `shared.go` contain types and functions used by both `CatalogSourceConfig` and `OperatorGroup` types.
37+
38+
#### Questions to ask yourself
39+
1. **Scope:** what files, Go source and YAML, must I modify when migrating?
40+
1. **Shared code:** do I have shared types and functions between `CatalogSourceConfig` and `OperatorGroup`? How do I want shard code refactored?
41+
1. **Imports:** which packages import those I am migrating? How do I modify these packages to import `v2` and possible new shared package(s)?
42+
1. **Backwards-compatibility:** do I want to remove code being migrated from `v1` entirely, forcing use of `v2`, or support both `v1` and `v2` going forward?
43+
44+
---
45+
46+
### Setting up to upgrade versions
47+
48+
Creating the new version `v2` is the first step in upgrading your kind `CatalogSourceConfig`. Use the `operator-sdk` to do so by running the following command:
49+
50+
```console
51+
$ operator-sdk add api --api-version operators.example.com/v2 --kind CatalogSourceConfig
52+
```
53+
54+
This command creates a new API version `v2` under group `operators`:
55+
56+
```console
57+
$ tree pkg/apis
58+
pkg/apis/
59+
├── addtoscheme_operators_v1.go
60+
├── addtoscheme_operators_v2.go # new addtoscheme source file for v2
61+
├── apis.go
62+
└── operators
63+
└── v1
64+
| ├── catalogsourceconfig_types.go
65+
| ├── catalogsourceconfig.go
66+
| ├── doc.go
67+
| ├── operatorgroup_types.go
68+
| ├── operatorgroup.go
69+
| ├── phase.go
70+
| ├── phase_types.go
71+
| ├── register.go
72+
| ├── shared.go
73+
| ├── zz_generated.deepcopy.go
74+
| ├── zz_generated.openapi.go
75+
└── v2 # new version dir with source files for v2
76+
├── catalogsourceconfig_types.go
77+
├── doc.go
78+
├── register.go
79+
├── zz_generated.deepcopy.go
80+
├── zz_generated.openapi.go
81+
```
82+
83+
In addition to creating a new API version, the command creates an `addtoscheme_operators_v2.go` file that exposes an `AddToScheme()` function for registering `v2.CatalogSourceConfig` and `v2.CatalogSourceConfigList`.
84+
85+
### Moving shared type definitions and functions to a separate package
86+
87+
Now that `v2` and all related project structure exist, we can begin moving types and functions around. First we must move anything shared between `CatalogSourceConfig` and `OperatorGroup` to a separate package that can be imported by both `v1` and `v2`. We've identified the files containing these types above: `phase.go`, `phase_types.go`, and `shared.go`.
88+
89+
#### Creating a new 'shared' package
90+
91+
Lets create a new package `shared` at `pkg/apis/operators/shared` for these files:
92+
93+
```console
94+
$ pwd
95+
/home/user/projects/test-operator
96+
$ mkdir pkg/apis/operators/shared
97+
```
98+
99+
This package is not a typical API because it contains types only to be used as parts of larger schema, and therefore should not be created with `operator-sdk add api`. It should contain a `doc.go` file with some package-level documentation and annotations:
100+
101+
```console
102+
$ cat > pkg/apis/operators/shared/doc.go <<EOF
103+
// +k8s:deepcopy-gen=package,register
104+
105+
// Package shared contains types and functions used by API definitions in the
106+
// operators package
107+
// +groupName=operators.example.com
108+
package shared
109+
EOF
110+
```
111+
112+
Global annotations necessary for using `shared` types in API type fields:
113+
114+
- `+k8s:deepcopy-gen=package,register`: directs [`deepcopy-gen`][deepcopy-gen] to generate `DeepCopy()` functions for all types in the `shared` package.
115+
- `+groupName=operators.example.com`: defines the fully qualified API group name for [`client-gen`][client-gen]. Note: this annotation *must* be on the line above `package shared`.
116+
117+
We recommend adding more comments explaining what types and functions exist in `shared` and how they are intended to be used. If you have any comments in `pkg/apis/operators/v1/doc.go` related to copied source code, ensure they are moved into `pkg/apis/operators/v2/doc.go`.
118+
119+
#### Moving types to package shared
120+
121+
The three files containing shared code (`phase.go`, `phase_types.go`, and `shared.go`) can *almost* be copied as-is from `v1` to `shared`. The only changes necessary are:
122+
123+
- Changing the package statements in each file: `package v1` -> `package shared`.
124+
- Exporting types, their methods, and functions used by external API types.
125+
126+
This will regenerate deepcopy code for all tagged types in `pkg/apis`.
127+
128+
Additionally, `deepcopy-gen` must be run on the new package to generate `DeepCopy()` and `DeepCopyInto()` methods, which are necessary for all Kubernetes API code. Doing so will also remove deepcopy code for the now `shared` types from `v1`. To do so, run the following command:
129+
130+
```console
131+
$ operator-sdk generate k8s
132+
```
133+
134+
Now that shared types and functions have their own package we can update types in package `v1`, and any other package that imports them from `v1`, to use those in `shared`. The source file `catalogsourceconfig_types.go` imports and uses a type defined in `shared`, `ObjectPhase`, as a field in `CatalogSourceConfigSpec`.
135+
136+
```Go
137+
import (
138+
"github.com/test-org/test-operator/pkg/apis/operators/shared"
139+
140+
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
141+
)
142+
143+
...
144+
145+
type CatalogSourceConfig struct {
146+
metav1.TypeMeta `json:",inline"`
147+
metav1.ListMeta `json:"metadata,omitempty"`
148+
149+
Spec CatalogSourceConfigSpec `json:"spec,omitempty"`
150+
Status CatalogSourceConfigStatus `json:"status,omitempty"`
151+
}
152+
153+
type CatalogSourceConfigSpec struct {
154+
...
155+
}
156+
157+
type CatalogSourceConfigStatus struct {
158+
// The type was previously ObjectPhase, now shared.ObjectPhase.
159+
CurrentPhase shared.ObjectPhase `json:"currentPhase,omitempty"`
160+
...
161+
}
162+
```
163+
164+
Do this for all instances of types previously in `v1` that are now in `shared`. Once done, remove `phase.go`, `phase_types.go`, and `shared.go` from `pkg/apis/operators/v1`.
165+
166+
### Updating empty v2 types using v1 types
167+
168+
The `CatalogSourceConfig` type and schema code were generated by `operator-sdk add api`, but the types are not populated. We need to move existing type data from `v1` to `v2`. This process is similar to migrating shared code, except we do not need to export any types or functions.
169+
170+
Remove `pkg/apis/operators/v2/catalogsourceconfig_types.go` and copy `catalogsourceconfig.go` and `catalogsourceconfig_types.go` from `pkg/apis/operators/v1` to `pkg/apis/operators/v2`:
171+
172+
```console
173+
$ rm pkg/apis/operators/v2/catalogsourceconfig_types.go
174+
$ cp pkg/apis/operators/v1/catalogsourceconfig*.go pkg/apis/operators/v2
175+
```
176+
177+
If you have any comments or custom code in `pkg/apis/operators/v1` related to source code in either copied file, ensure that is copied to `doc.go` or `register.go` in `pkg/apis/operators/v2`.
178+
179+
You can now run `operator-sdk generate k8s` to generate deepcopy code for the migrated `v2` types. Once this is done, update all packages that import the migrated `v1` types to use those in `v2`.
180+
181+
The final step is to remove `catalogsourceconfig.go` and `catalogsourceconfig_types.go` and any related comments or custom code in `doc.go` or `register.go` from `pkg/apis/operators/v1`.
182+
183+
**Note:** updating package import paths will likely be the most pervasive change lines-of-code-wise in this process. Luckily the Go compiler will tell you which import path's you have missed once `CatalogSourceConfig` types are removed from `v1`!
184+
185+
### Updating CustomResourceDefinition manifests and generating OpenAPI code
186+
187+
Now that we've migrated all Go types to their destination packages, we must update the corresponding CustomResourceDefinition (CRD) manifests in `deploy/crds` and generate [OpenAPI][openapi-gen] validation source.
188+
189+
Doing so can be as simple as running the following command:
190+
191+
```console
192+
$ operator-sdk generate openapi
193+
```
194+
195+
This command will automatically update all CRD manifests.
196+
197+
#### CRD Versioning
198+
199+
Kubernetes 1.11+ supports CRD [`spec.versions`][crd-versions] and `spec.version` is [deprecated][crd-version-deprecated] as of Kubernetes 1.12. SDK versions `v0.9.x` and below leverage [`controller-tools`][controller-tools]' CRD generator `v0.1.x` which generates a now-deprecated `spec.version` value based on the version contained in an API's import path. Names of CRD manifest files generated by those SDK versions contain the `spec.version`, i.e. one CRD manifest is created *per version in a group* with the form `<group>_<version>_<kind>_crd.yaml`. The SDK is in the process of upgrading to `controller-tools` `v0.2.x`, which generates `spec.versions` but not `spec.version` by default. Once the upgrade is complete, future SDK versions will place all versions in a group in `spec.versions`. File names will then have the format `<group>_<resource>_crd.yaml`.
200+
201+
**Note:** If your operator does not have custom data manually added to its CRD's, you can skip to the [following section](#golang-api-migrations-types-and-commonalities); `operator-sdk generate openapi` will handle CRD updates in that case.
202+
203+
Upgrading from `spec.version` to `spec.versions` will be demonstrated using the following CRD manifest example:
204+
205+
`deploy/crds/operators_v1_catalogsourceconfig_crd.yaml`:
206+
```yaml
207+
apiVersion: apiextensions.k8s.io/v1beta1
208+
kind: CustomResourceDefinition
209+
metadata:
210+
name: catalogsourceconfigs.operators.coreos.com
211+
spec:
212+
group: operators.coreos.com
213+
names:
214+
kind: CatalogSourceConfig
215+
listKind: CatalogSourceConfigList
216+
plural: catalogsourceconfigs
217+
singular: catalogsourceconfig
218+
scope: Namespaced
219+
validation:
220+
openAPIV3Schema:
221+
properties:
222+
apiVersion:
223+
type: string
224+
kind:
225+
type: string
226+
metadata:
227+
type: object
228+
spec:
229+
properties:
230+
size:
231+
format: int32
232+
type: integer
233+
test:
234+
type: string
235+
required:
236+
- size
237+
type: object
238+
status:
239+
properties:
240+
nodes:
241+
items:
242+
type: string
243+
type: array
244+
required:
245+
- nodes
246+
type: object
247+
version: v1
248+
subresources:
249+
status: {}
250+
```
251+
252+
Steps to upgrade the above CRD:
253+
254+
1. Rename your CRD manifest file from `deploy/crds/operators_v1_catalogsourceconfig_crd.yaml` to `deploy/crds/catalogsourceconfigs.operators.coreos.com_catalogsourceconfigs_crd.yaml`
255+
256+
```console
257+
$ mv deploy/crds/cache_v1alpha1_memcached_crd.yaml deploy/crds/catalogsourceconfigs.operators.coreos.com_catalogsourceconfigs_crd.yaml
258+
```
259+
260+
1. Create a `spec.versions` list that contains two elements for each version that now exists (`v1` and `v2`):
261+
262+
```yaml
263+
spec:
264+
...
265+
# version is now v2, as it must match the first element in versions.
266+
version: v2
267+
versions:
268+
- name: v2
269+
# Set to true for this CRD version to be enabled in-cluster.
270+
served: true
271+
# Exactly one CRD version should be a storage version.
272+
storage: true
273+
- name: v1
274+
served: true
275+
storage: false
276+
```
277+
278+
The first version in `spec.versions` *must* match that in `spec.version` if `spec.version` exists in the manifest.
279+
280+
1. *Optional:* `spec.versions` elements have a `schema` field that holds a version-specific OpenAPIV3 validation block to override the global `spec.validation` block. `spec.validation` will be used by the API server to validate one or more version in `spec.versions` that does not have a `schema` block. If all versions have the same schema, leave `spec.validation` as-is and skip to the [following section](#golang-api-migrations-types-and-commonalities). If your CRD versions differ in scheme, copy `spec.validation` YAML to the `schema` field in each `spec.versions` element, then modify as needed:
281+
282+
```yaml
283+
spec:
284+
...
285+
version: v2
286+
versions:
287+
- name: v2
288+
served: true
289+
storage: true
290+
schema: # v2-specific OpenAPIV3 validation block.
291+
openAPIV3Schema:
292+
properties:
293+
apiVersion:
294+
type: string
295+
...
296+
- name: v1
297+
served: true
298+
storage: false
299+
schema: # v1-specific OpenAPIV3 validation block.
300+
openAPIV3Schema:
301+
properties:
302+
apiVersion:
303+
type: string
304+
...
305+
```
306+
307+
The API server will validate each version by its own `schema` if the global `spec.validation` block is removed. No validation will be performed if a `schema` does not exist for a version and `spec.validation` does not exist.
308+
309+
If the CRD targets a Kubernetes 1.13+ cluster with the `CustomResourceWebhookConversion` feature enabled, converting between multiple versions can be done using a [conversion webhooks][crd-conv-webhook].
310+
311+
**Note:** read the [CRD versioning][crd-versions] docs for detailed CRD information, notes on conversion webhooks, and CRD versioning case studies.
312+
313+
1. *Optional:* `spec.versions` elements have a `subresources` field that holds CR subresource information to override the global `spec.subresources` block. `spec.subresources` will be used by the API server to assess subresource requirements of any version in `spec.versions` that does not have a `subresources` block. If all versions have the same requirements, leave `spec.subresources` as-is and skip to the [following section](#golang-api-migrations-types-and-commonalities). If CRD versions differ in subresource requirements, add a `subresources` section in each `spec.versions` entry with differing requirements and add each subresource's spec and status as needed:
314+
315+
```yaml
316+
spec:
317+
...
318+
version: v2
319+
versions:
320+
- name: v2
321+
served: true
322+
storage: true
323+
subresources:
324+
...
325+
- name: v1
326+
served: true
327+
storage: false
328+
subresources:
329+
...
330+
```
331+
332+
Remove the global `spec.subresources` block if all versions have different subresource requirements.
333+
334+
1. *Optional:* remove `spec.version`, as it is deprecated in favor of `spec.versions`.
335+
336+
### Golang API Migrations: Types and Commonalities
337+
338+
This version upgrade walkthrough demonstrates only one of several possible migration scenarios:
339+
340+
- Group migration, ex. moving an API from group `operators.example.com/v1` to `new-group.example.com/v1alpha1`.
341+
- Kind change, ex. `CatalogSourceConfig` to `CatalogSourceConfigurer`.
342+
- Some combination of group, version, and kind migration.
343+
344+
Each case is different; one may require many more changes than others. However there are several themes common to all:
345+
346+
1. Using `operator-sdk add api` to create the necessary directory structure and files used in migration.
347+
- Group migration using the same version, for each kind in the old group `operators.example.com` you want to migrate:
348+
349+
```console
350+
$ operator-sdk add api --api-version new-group.example.com/v1 --kind YourKind
351+
```
352+
353+
- Kind migration, using the same group and version as `CatalogSourceConfig`:
354+
355+
```console
356+
$ operator-sdk add api --api-version operators.example.com/v1 --kind CatalogSourceConfigurer
357+
```
358+
359+
1. Moving code from one Go package to another, ex. from `v1` to `v2` and `shared`.
360+
1. Changing import paths in project Go source files to those of new packages.
361+
1. Updating CRD manifests.
362+
- In many cases, having sufficient [code annotations][kubebuilder-api-annotations] and running `operator-sdk generate openapi` will be enough.
363+
364+
The Go toolchain can be your friend here too. Running `go vet ./...` can tell you what import paths require changing and what type instantiations are using fields incorrectly.
365+
366+
## Helm
367+
368+
TODO
369+
370+
## Ansible
371+
372+
TODO
373+
374+
[k8s-versioning]:https://kubernetes.io/docs/concepts/overview/kubernetes-api/#api-versioning
375+
[deepcopy-gen]:https://godoc.org/k8s.io/gengo/examples/deepcopy-gen
376+
[client-gen]:https://github.com/kubernetes/community/blob/master/contributors/devel/sig-api-machinery/generating-clientset.md
377+
[openapi-gen]:https://github.com/kubernetes/kube-openapi
378+
[controller-tools]:https://github.com/kubernetes-sigs/controller-tools
379+
[crd-versions]:https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definition-versioning/
380+
[crd-conv-webhook]:https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definition-versioning/#webhook-conversion
381+
[kubebuilder-api-annotations]:https://book-v1.book.kubebuilder.io/beyond_basics/generating_crd.html
382+
[crd-version-deprecated]:https://github.com/kubernetes/apiextensions-apiserver/commit/d1c6536f26319513417b12245c6e3aee5ca005ca

0 commit comments

Comments
 (0)