Skip to content

Commit 78b1795

Browse files
sethp-nrmnguyendetibervincepri
committed
✨ KCP adopts existing machines
The KCP controller identifies Machines that belong to the control plane of an existing cluster and adopts them, including finding PKI materials that may be owned by the machine's bootstrap config and pivoting their ownership to the KCP as well. Prior to adopting machines (which, if unsuccessful, will block the KCP from taking any management actions), it runs a number of safety checks including: - Ensuring the KCP has not been deleted (to prevent re-adoption of orphans, though this process races with the garbage collector) - Checking that the machine's bootstrap provider was KubeadmConfig - Verifying that the Machine is no further than one minor version off of the KCP's spec Additionally, we set set a "best guess" value for the kubeadm.controlplane.cluster.x-k8s.io/hash on the adopted machine as if it were generated by a KCP in the past. The intent is that a KCP will adopt machines matching its "spec" (to the best of its ability) without modification, which in practice works well for adopting machines with the same spec'd version. We make an educated guess at what parts of the kubeadm config the operator considers important, and feed that into the hash function. The goal here is to be able to: 1. Create a KCP over existing machines with ~the same configuration (for then new api/equality's package definition of the same) and have the KCP make no changes 2. Create a KCP over existing machines with a different configuration and have the KCP upgrade those machines to the new config Finally, replaces PointsTo and introduces IsControlledBy, both of which search through the owner reference list of their first argument (similar to metav1.IsControlledBy). Unlike the metav1 function, they're looking for a match on name plus api group (not version) and kind. PointsTo returns true if any OwnerReference matches the pointee, whereas IsControlledBy only returns true if the sole permitted (by the API) controller reference matches. fix: handle JoinConfiguration hashing refactor: util should not check UIDs This is a behavior change not least for tests, which typically do not set either Kind or GroupVersion on various objects that end up acting as referents. Co-Authored-By: mnguyen <[email protected]> Co-Authored-By: Jason DeTiberus <[email protected]> Co-authored-by: Vince Prignano <[email protected]>
1 parent 69ebae6 commit 78b1795

36 files changed

+1720
-142
lines changed

.dockerignore

+29-1
Original file line numberDiff line numberDiff line change
@@ -12,8 +12,36 @@ tilt-settings.json
1212
tilt.d/
1313
Tiltfile
1414
**/.tiltbuild
15-
test/infrastructure/docker/e2e/logs/**
1615
**/config/**/*.yaml
16+
**/config/**/*.yaml-e
1717
_artifacts
1818
Makefile
1919
**/Makefile
20+
21+
# ignores changes to test-only code to avoid extra rebuilds
22+
test/e2e/**
23+
test/framework/**
24+
test/infrastructure/docker/e2e/**
25+
26+
.dockerignore
27+
# We want to ignore any frequently modified files to avoid cache-busting the COPY ./ ./
28+
# Binaries for programs and plugins
29+
**/*.exe
30+
**/*.dll
31+
**/*.so
32+
**/*.dylib
33+
cmd/clusterctl/clusterctl/**
34+
**/bin/**
35+
**/out/**
36+
37+
# Test binary, build with `go test -c`
38+
**/*.test
39+
40+
# Output of the go coverage tool, specifically when used with LiteIDE
41+
**/*.out
42+
43+
# Common editor / temporary files
44+
**/*~
45+
**/*.tmp
46+
**/.DS_Store
47+
**/*.swp

api/v1alpha3/cluster_types.go

+8
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ package v1alpha3
1818

1919
import (
2020
"fmt"
21+
"strings"
2122

2223
corev1 "k8s.io/api/core/v1"
2324
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
@@ -90,6 +91,13 @@ type NetworkRanges struct {
9091
CIDRBlocks []string `json:"cidrBlocks"`
9192
}
9293

94+
func (n *NetworkRanges) String() string {
95+
if n == nil {
96+
return ""
97+
}
98+
return strings.Join(n.CIDRBlocks, "")
99+
}
100+
93101
// ANCHOR_END: NetworkRanges
94102

95103
// ANCHOR: ClusterStatus

bootstrap/kubeadm/api/equality/doc.go

+28
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
/*
2+
Copyright 2020 The Kubernetes Authors.
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
http://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
*/
16+
17+
/*
18+
Package equality defines equality semantics for KubeadmConfigs, and utility tools for identifying
19+
equivalent configurations.
20+
21+
There are a number of distinct but not different ways to express the "same" Kubeadm configuration,
22+
and so this package attempts to sort out what differences are likely meaningful or intentional.
23+
24+
It is inspired by the observation that k/k no longer relies on hashing to identify "current" versions
25+
ReplicaSets, instead using a semantic equality check that's more amenable to field modifications and
26+
deletions: https://github.com/kubernetes/kubernetes/blob/0bb125e731/pkg/controller/deployment/util/deployment_util.go#L630-L634
27+
*/
28+
package equality
+128
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,128 @@
1+
/*
2+
Copyright 2020 The Kubernetes Authors.
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
http://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
*/
16+
17+
package equality
18+
19+
import (
20+
clusterv1 "sigs.k8s.io/cluster-api/api/v1alpha3"
21+
bootstrapv1 "sigs.k8s.io/cluster-api/bootstrap/kubeadm/api/v1alpha3"
22+
kubeadmv1 "sigs.k8s.io/cluster-api/bootstrap/kubeadm/types/v1beta1"
23+
"sigs.k8s.io/cluster-api/util/secret"
24+
)
25+
26+
// SemanticMerge takes two KubeConfigSpecs and produces a third that is semantically equivalent to the other two
27+
// by way of merging non-behavior-changing fields from incoming into current.
28+
func SemanticMerge(current, incoming bootstrapv1.KubeadmConfigSpec, cluster *clusterv1.Cluster) bootstrapv1.KubeadmConfigSpec {
29+
merged := bootstrapv1.KubeadmConfigSpec{}
30+
current.DeepCopyInto(&merged)
31+
32+
merged = mergeClusterConfiguration(merged, incoming, cluster)
33+
34+
// Prefer non-nil init configurations: in most cases the init configuration is ignored and so this is purely representative
35+
if merged.InitConfiguration == nil && incoming.InitConfiguration != nil {
36+
merged.InitConfiguration = incoming.InitConfiguration.DeepCopy()
37+
}
38+
39+
// The type meta for embedded types is currently ignored
40+
if merged.ClusterConfiguration != nil && incoming.ClusterConfiguration != nil {
41+
merged.ClusterConfiguration.TypeMeta = incoming.ClusterConfiguration.TypeMeta
42+
}
43+
44+
if merged.InitConfiguration != nil && incoming.InitConfiguration != nil {
45+
merged.InitConfiguration.TypeMeta = incoming.InitConfiguration.TypeMeta
46+
}
47+
48+
if merged.JoinConfiguration != nil && incoming.JoinConfiguration != nil {
49+
merged.JoinConfiguration.TypeMeta = incoming.JoinConfiguration.TypeMeta
50+
}
51+
52+
// The default discovery injected by the kubeadm bootstrap controller is a unique, time-bounded token. However, any
53+
// valid token has the same effect of joining the machine to the cluster. We consider the following scenarios:
54+
// 1. current has no join configuration (meaning it was never reconciled) -> do nothing
55+
// 2. current has a bootstrap token, and incoming has some configured discovery mechanism -> prefer incoming
56+
// 3. current has a bootstrap token, and incoming has no discovery set -> delete current's discovery config
57+
// 4. in all other cases, prefer current's configuration
58+
if merged.JoinConfiguration == nil || merged.JoinConfiguration.Discovery.BootstrapToken == nil {
59+
return merged
60+
}
61+
62+
// current has a bootstrap token, check incoming
63+
switch {
64+
case incoming.JoinConfiguration == nil:
65+
merged.JoinConfiguration.Discovery = kubeadmv1.Discovery{}
66+
case incoming.JoinConfiguration.Discovery.BootstrapToken != nil:
67+
fallthrough
68+
case incoming.JoinConfiguration.Discovery.File != nil:
69+
incoming.JoinConfiguration.Discovery.DeepCopyInto(&merged.JoinConfiguration.Discovery)
70+
default:
71+
// Neither token nor file is set on incoming's Discovery config
72+
merged.JoinConfiguration.Discovery = kubeadmv1.Discovery{}
73+
}
74+
75+
return merged
76+
}
77+
78+
func mergeClusterConfiguration(merged, incoming bootstrapv1.KubeadmConfigSpec, cluster *clusterv1.Cluster) bootstrapv1.KubeadmConfigSpec {
79+
if merged.ClusterConfiguration == nil && incoming.ClusterConfiguration != nil {
80+
merged.ClusterConfiguration = incoming.ClusterConfiguration.DeepCopy()
81+
} else if merged.ClusterConfiguration == nil {
82+
// incoming's cluster configuration is also nil
83+
return merged
84+
}
85+
86+
// Attempt to reconstruct a cluster config in reverse of reconcileTopLevelObjectSettings
87+
newCfg := incoming.ClusterConfiguration
88+
if newCfg == nil {
89+
newCfg = &kubeadmv1.ClusterConfiguration{}
90+
}
91+
92+
if merged.ClusterConfiguration.ControlPlaneEndpoint == cluster.Spec.ControlPlaneEndpoint.String() &&
93+
newCfg.ControlPlaneEndpoint == "" {
94+
merged.ClusterConfiguration.ControlPlaneEndpoint = ""
95+
}
96+
97+
if newCfg.ClusterName == "" {
98+
merged.ClusterConfiguration.ClusterName = ""
99+
}
100+
101+
if cluster.Spec.ClusterNetwork != nil {
102+
if merged.ClusterConfiguration.Networking.DNSDomain == cluster.Spec.ClusterNetwork.ServiceDomain && newCfg.Networking.DNSDomain == "" {
103+
merged.ClusterConfiguration.Networking.DNSDomain = ""
104+
}
105+
106+
if merged.ClusterConfiguration.Networking.ServiceSubnet == cluster.Spec.ClusterNetwork.Services.String() &&
107+
newCfg.Networking.ServiceSubnet == "" {
108+
merged.ClusterConfiguration.Networking.ServiceSubnet = ""
109+
}
110+
111+
if merged.ClusterConfiguration.Networking.PodSubnet == cluster.Spec.ClusterNetwork.Pods.String() &&
112+
newCfg.Networking.PodSubnet == "" {
113+
merged.ClusterConfiguration.Networking.PodSubnet = ""
114+
}
115+
}
116+
117+
// We defer to other sources for the version, such as the Machine
118+
if newCfg.KubernetesVersion == "" {
119+
merged.ClusterConfiguration.KubernetesVersion = ""
120+
}
121+
122+
if merged.ClusterConfiguration.CertificatesDir == secret.DefaultCertificatesDir &&
123+
newCfg.CertificatesDir == "" {
124+
merged.ClusterConfiguration.CertificatesDir = ""
125+
}
126+
127+
return merged
128+
}

bootstrap/kubeadm/controllers/kubeadmconfig_controller.go

+2-3
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,6 @@ import (
2020
"context"
2121
"fmt"
2222
"strconv"
23-
"strings"
2423
"time"
2524

2625
"github.com/go-logr/logr"
@@ -688,13 +687,13 @@ func (r *KubeadmConfigReconciler) reconcileTopLevelObjectSettings(cluster *clust
688687
if config.Spec.ClusterConfiguration.Networking.ServiceSubnet == "" &&
689688
cluster.Spec.ClusterNetwork.Services != nil &&
690689
len(cluster.Spec.ClusterNetwork.Services.CIDRBlocks) > 0 {
691-
config.Spec.ClusterConfiguration.Networking.ServiceSubnet = strings.Join(cluster.Spec.ClusterNetwork.Services.CIDRBlocks, "")
690+
config.Spec.ClusterConfiguration.Networking.ServiceSubnet = cluster.Spec.ClusterNetwork.Services.String()
692691
log.Info("Altering ClusterConfiguration", "ServiceSubnet", config.Spec.ClusterConfiguration.Networking.ServiceSubnet)
693692
}
694693
if config.Spec.ClusterConfiguration.Networking.PodSubnet == "" &&
695694
cluster.Spec.ClusterNetwork.Pods != nil &&
696695
len(cluster.Spec.ClusterNetwork.Pods.CIDRBlocks) > 0 {
697-
config.Spec.ClusterConfiguration.Networking.PodSubnet = strings.Join(cluster.Spec.ClusterNetwork.Pods.CIDRBlocks, "")
696+
config.Spec.ClusterConfiguration.Networking.PodSubnet = cluster.Spec.ClusterNetwork.Pods.String()
698697
log.Info("Altering ClusterConfiguration", "PodSubnet", config.Spec.ClusterConfiguration.Networking.PodSubnet)
699698
}
700699
}

controllers/cluster_controller.go

+1-1
Original file line numberDiff line numberDiff line change
@@ -404,7 +404,7 @@ func (c clusterDescendants) filterOwnedDescendants(cluster *clusterv1.Cluster) (
404404
return nil
405405
}
406406

407-
if util.PointsTo(acc.GetOwnerReferences(), &cluster.ObjectMeta) {
407+
if util.IsOwnedByObject(acc, cluster) {
408408
ownedDescendants = append(ownedDescendants, o)
409409
}
410410

controllers/cluster_controller_test.go

+4
Original file line numberDiff line numberDiff line change
@@ -703,6 +703,10 @@ func TestFilterOwnedDescendants(t *testing.T) {
703703
g := NewWithT(t)
704704

705705
c := clusterv1.Cluster{
706+
TypeMeta: metav1.TypeMeta{
707+
APIVersion: clusterv1.GroupVersion.String(),
708+
Kind: "Cluster",
709+
},
706710
ObjectMeta: metav1.ObjectMeta{
707711
Name: "c",
708712
},

controlplane/kubeadm/config/rbac/role.yaml

+1
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,7 @@ rules:
6161
- get
6262
- list
6363
- patch
64+
- update
6465
- watch
6566

6667
---

0 commit comments

Comments
 (0)