You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: enhancements/machine-api/out-of-tree-provider-support.md
+10-6
Original file line number
Diff line number
Diff line change
@@ -203,6 +203,8 @@ Without this information, the `scheduler` cannot schedule `Pods` that have any s
203
203
To ensure that cluster disaster recovery procedures can still operate smoothly, we will ensure that core control plane components and their operators tolerate the uninitialized taint, to prevent `CCM` blocking new control plane hosts being added if `CCM` is non-functional.
204
204
This will include, but is not limited to: Kube Controller Manager, Etcd, Kube API Server, Networking, Cluster Machine Approver.
205
205
206
+
*Note: Windows Machine Config Operator (WMCO) manages `kubelet` configuration for Windows nodes, for which described changes in this chapter would still apply.*
207
+
206
208
##### Example flag changes for kubelet
207
209
208
210
Current flag configuration for kubelet in AWS provider:
@@ -332,18 +334,19 @@ Once the provider is moved to out-of-tree, the migration mechanism will be disab
332
334
333
335
#### Bootstrap changes
334
336
335
-
One of the responsibilities of the initialisation process for Kubelet is to set the `Node`’s IP addresses within the status of the `Node` object. The remaining responsibilities are not important for this document bar the removal of a taint which prevents workloads running on the `Node` until the initialisation has completed.
337
+
One of the responsibilities of the initialisation process for Kubelet is to set the `Node`’s IP addresses within the status of the `Node` object. The remaining responsibilities are not important for bootstrapping, bar the removal of a taint which prevents workloads running on the `Node` until the initialisation has completed.
336
338
337
339
A second part of the bootstrap process for a new `Node`, is to initialise the `CNI` (networking). Typically in an OpenShift cluster, this is handled once the Networking Operator starts.
338
-
The Networking operator will create the `CNI` pods (typically OpenShift SDN), which schedule on the `Node`, use the `Node` IP addresses to create a `HostSubnet` resource within Kubernetes and then mark then complete the initialisation process for the `CNI`, in doing so, marking the `Node` as ready and allowing the remaining workloads to start.
340
+
The Networking operator will create the `CNI` pods (typically OpenShift SDN), which schedule on the `Node`, use the `Node` IP addresses to create a `HostSubnet` resource within Kubernetes and then complete the initialisation process for the `CNI`, in doing so, marking the `Node` as ready and allowing the remaining workloads to start.
339
341
340
-
Before the `CNI` is initialized on a `Node`, in-cluster networking such as Service IPs, in particular the API server Service, will not work for any `Pod` on the `Node`. Additionally, any `Pod` that requires the Pod Networking implemented by `CNI`, cannot start. For this reason, `Pods` such as the Networking Operator must use host networking and the “API Int” load balancer to contact the Kube API Server.
342
+
Before the `CNI` is initialized on a `Node`, in-cluster networking such as Service IPs, in particular the API server Service, will not work for any `Pod` on the `Node`. Additionally, any `Pod` that requires the Pod Networking implemented by `CNI`, cannot start.
343
+
For this reason, `Pods` such as the Networking Operator must use host networking and the “API Int” load balancer to contact the Kube API Server.
341
344
342
345
Because the `CCM` is taking over the responsibility of setting the `Node` IP addresses, `CCM` will become a prerequisite for networking to become functional within any Cluster. Because the `CNI` is not initialised, we must ensure that the `CCCMO` and `CCM` Pods tolerate the scenario where `CNI` is non-functional.
343
346
344
347
To do so, we must tolerate the not-ready taint for these pods and they must all run with host networking and use the API load balancer, rather than using the internal Service. This will ensure that the cluster can bootstrap successfully and recover from any disaster recovery scenario.
345
348
346
-
Our operator will take precedence for CNI operator. It will tolerate `NotReady` `NoSchedule` taint and `CCM` specific `Uninitialized` taint. Operator would start as the first operator in the cluster when first `control-plane` is created, and be responsible for initializing `Nodes` which will allow latter operators to start.
349
+
Our operator will become a prerequisite for the Network Operator. CCCMO will tolerate the `Node` `NotReady:NoSchedule` and `CCM` specific `Uninitialized` taints. CCCMO will start as the first operator on the controlplane hosts, and be responsible for initializing `Nodes`, allowing other operators to start.
347
350
348
351
#### Metrics
349
352
@@ -553,7 +556,8 @@ This functionality should not be required as OpenShift handles certificate appro
553
556
554
557
Q: Does every node need a CCM?
555
558
556
-
- A: No. The cluster only needs one active `CCM` at any time. A `Deployment` will manage the `CCM` pod and will have 2 replicas which will use leader election to nominate an active leader.
559
+
- A: No. The cluster only needs one active `CCM` at any time. A `Deployment` will manage the `CCM` pod and will have 2 replicas which will use leader election to nominate an active leader and maintain HA by scheduling on control-plane nodes located in the different regions.
560
+
Only in some scenarios (depending on the cloud provider implementation) like Azure `cloud-node-manager` has to run on all the `Nodes` due to 1:1 relation betwen `Node` and a `CNM` replica in their case.
557
561
[Source](https://kubernetes.io/docs/concepts/overview/components/#cloud-controller-manager) This assumption may change in the future, as `CCM` may run in worker nodes to determine the state of the instance.
0 commit comments