You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
= Binding infrastructure node workloads using taints and tolerations
9
9
10
-
If you have an infra node that has the `infra` and `worker` roles assigned, you must configure the node so that user workloads are not assigned to it.
10
+
If you have an infrastructure node that has the `infra` and `worker` roles assigned, you must configure the node so that user workloads are not assigned to it.
11
11
12
12
[IMPORTANT]
13
13
====
14
-
It is recommended that you preserve the dual `infra,worker` label that is created for infra nodes and use taints and tolerations to manage nodes that user workloads are scheduled on. If you remove the `worker` label from the node, you must create a custom pool to manage it. A node with a label other than `master` or `worker` is not recognized by the MCO without a custom pool. Maintaining the `worker` label allows the node to be managed by the default worker machine config pool, if no custom pools that select the custom label exists. The `infra` label communicates to the cluster that it does not count toward the total number of subscriptions.
14
+
It is recommended that you preserve the dual `infra,worker` label that is created for infrastructure nodes and use taints and tolerations to manage nodes that user workloads are scheduled on. If you remove the `worker` label from the node, you must create a custom pool to manage it. A node with a label other than `master` or `worker` is not recognized by the MCO without a custom pool. Maintaining the `worker` label allows the node to be managed by the default worker machine config pool, if no custom pools that select the custom label exists. The `infra` label communicates to the cluster that it does not count toward the total number of subscriptions.
15
15
====
16
16
17
17
.Prerequisites
@@ -20,7 +20,7 @@ It is recommended that you preserve the dual `infra,worker` label that is create
20
20
21
21
.Procedure
22
22
23
-
. Add a taint to the infra node to prevent scheduling user workloads on it:
23
+
. Add a taint to the infrastructure node to prevent scheduling user workloads on it:
You can alternatively apply the following YAML to add the taint:
61
+
You can alternatively edit the pod specification to add the taint:
62
62
63
63
[source,yaml]
64
64
----
65
-
kind: Node
66
65
apiVersion: v1
66
+
kind: Node
67
67
metadata:
68
-
name: <node_name>
69
-
labels:
70
-
...
68
+
name: node1
69
+
# ...
71
70
spec:
72
71
taints:
73
72
- key: node-role.kubernetes.io/infra
74
-
effect: NoExecute
75
73
value: reserved
76
-
...
74
+
effect: NoSchedule
75
+
# ...
77
76
----
78
77
====
79
78
+
80
-
This example places a taint on `node1` that has key`node-role.kubernetes.io/infra` and taint effect `NoSchedule`. Nodes with the `NoSchedule` effect schedule only pods that tolerate the taint, but allow existing pods to remain scheduled on the node.
79
+
These examples place a taint on `node1` that has the`node-role.kubernetes.io/infra`key and the `NoSchedule` taint effect. Nodes with the `NoSchedule` effect schedule only pods that tolerate the taint, but allow existing pods to remain scheduled on the node.
81
80
+
82
81
[NOTE]
83
82
====
84
83
If a descheduler is used, pods violating node taints could be evicted from the cluster.
85
84
====
86
85
87
-
. Add tolerations for the pod configurations you want to schedule on the infra node, like router, registry, and monitoring workloads. Add the following code to the `Pod` object specification:
86
+
. Add tolerations to the pods that you want to schedule on the infrastructure node, such as the router, registry, and monitoring workloads. Referencing the previous examples, add the following tolerations to the `Pod` object specification:
88
87
+
89
88
[source,yaml]
90
89
----
91
-
tolerations:
92
-
- effect: NoExecute <1>
93
-
key: node-role.kubernetes.io/infra <2>
94
-
operator: Equal <3>
95
-
value: reserved <4>
90
+
apiVersion: v1
91
+
kind: Pod
92
+
metadata:
93
+
annotations:
94
+
95
+
# ...
96
+
spec:
97
+
# ...
98
+
tolerations:
99
+
- key: node-role.kubernetes.io/infra <1>
100
+
value: reserved <2>
101
+
effect: NoSchedule <3>
102
+
operator: Equal <4>
96
103
----
97
-
<1> Specify the effect that you added to the node.
98
-
<2> Specify the key that you added to the node.
99
-
<3> Specify the `Equal` Operator to require a taint with the key `node-role.kubernetes.io/infra` to be present on the node.
100
-
<4> Specify the value of the key-value pair taint that you added to the node.
104
+
<1> Specify the key that you added to the node.
105
+
<2> Specify the value of the key-value pair taint that you added to the node.
106
+
<3> Specify the effect that you added to the node.
107
+
<4> Specify the `Equal` Operator to require a taint with the key `node-role.kubernetes.io/infra` to be present on the node.
101
108
+
102
-
This toleration matches the taint created by the `oc adm taint` command. A pod with this toleration can be scheduled onto the infra node.
109
+
This toleration matches the taint created by the `oc adm taint` command. A pod with this toleration can be scheduled onto the infrastructure node.
103
110
+
104
111
[NOTE]
105
112
====
106
-
Moving pods for an Operator installed via OLM to an infra node is not always possible. The capability to move Operator pods depends on the configuration of each Operator.
113
+
Moving pods for an Operator installed via OLM to an infrastructure node is not always possible. The capability to move Operator pods depends on the configuration of each Operator.
107
114
====
108
115
109
-
. Schedule the pod to the infra node using a scheduler. See the documentation for _Controlling pod placement onto nodes_ for details.
116
+
. Schedule the pod to the infrastructure node by using a scheduler. See the documentation for "Controlling pod placement using the scheduler" for details.
117
+
118
+
. Remove any workloads that you do not want, or that do not belong, on the new infrastructure node. See the list of workloads supported for use on infrastructure nodes in "{product-title} infrastructure components".
Copy file name to clipboardExpand all lines: modules/creating-an-infra-node.adoc
+15-20Lines changed: 15 additions & 20 deletions
Original file line number
Diff line number
Diff line change
@@ -13,16 +13,20 @@
13
13
See Creating infrastructure machine sets for installer-provisioned infrastructure environments or for any cluster where the control plane nodes are managed by the machine API.
14
14
====
15
15
16
-
Requirements of the cluster dictate that infrastructure, also called `infra` nodes, be provisioned. The installer only provides provisions for control plane and worker nodes. Worker nodes can be designated as infrastructure nodes or application, also called `app`, nodes through labeling.
16
+
Requirements of the cluster dictate that infrastructure (infra) nodes, be provisioned. The installation program provisions only control plane and worker nodes. Worker nodes can be designated as infrastructure nodes through labeling. You can then use taints and tolerations to move appropriate workloads to the infrastructure nodes. For more information, see "Moving resources to infrastructure machine sets".
17
17
18
-
.Procedure
18
+
You can optionally create a default cluster-wide node selector. The default node selector is applied to pods created in all namespaces and creates an intersection with any existing node selectors on a pod, which additionally constrains the pod's selector.
19
19
20
-
. Add a label to the worker node that you want to act as application node:
If the default node selector key conflicts with the key of a pod's label, then the default node selector is not applied.
23
+
24
+
However, do not set a default node selector that might cause a pod to become unschedulable. For example, setting the default node selector to a specific node role, such as `node-role.kubernetes.io/infra=""`, when a pod's label is set to a different node role, such as `node-role.kubernetes.io/master=""`, can cause the pod to become unschedulable. For this reason, use caution when setting the default node selector to specific node roles.
25
+
26
+
You can alternatively use a project node selector to avoid cluster-wide node selector key conflicts.
27
+
====
28
+
29
+
.Procedure
26
30
27
31
. Add a label to the worker nodes that you want to act as infrastructure nodes:
. Check to see if applicable nodes now have the `infra` role and `app` roles:
38
+
. Check to see if applicable nodes now have the `infra` role:
35
39
+
36
40
[source,terminal]
37
41
----
38
42
$ oc get nodes
39
43
----
40
44
41
-
. Create a default cluster-wide node selector. The default node selector is applied to pods created in all namespaces. This creates an intersection with any existing node selectors on a pod, which additionally constrains the pod's selector.
42
-
+
43
-
[IMPORTANT]
44
-
====
45
-
If the default node selector key conflicts with the key of a pod's label, then the default node selector is not applied.
46
-
47
-
However, do not set a default node selector that might cause a pod to become unschedulable. For example, setting the default node selector to a specific node role, such as `node-role.kubernetes.io/infra=""`, when a pod's label is set to a different node role, such as `node-role.kubernetes.io/master=""`, can cause the pod to become unschedulable. For this reason, use caution when setting the default node selector to specific node roles.
48
-
49
-
You can alternatively use a project node selector to avoid cluster-wide node selector key conflicts.
50
-
====
45
+
. Optional: Create a default cluster-wide node selector:
51
46
52
47
.. Edit the `Scheduler` object:
53
48
+
@@ -72,4 +67,4 @@ spec:
72
67
73
68
.. Save the file to apply the changes.
74
69
75
-
You can now move infrastructure resources to the newly labeled `infra`nodes.
70
+
You can now move infrastructure resources to the new infrastructure nodes. Also, remove any workloads that you do not want, or that do not belong, on the new infrastructure node. See the list of workloads supported for use on infrastructure nodes in "{product-title} infrastructure components".
Copy file name to clipboardExpand all lines: modules/infrastructure-moving-monitoring.adoc
+1-25Lines changed: 1 addition & 25 deletions
Original file line number
Diff line number
Diff line change
@@ -40,81 +40,57 @@ data:
40
40
- key: node-role.kubernetes.io/infra
41
41
value: reserved
42
42
effect: NoSchedule
43
-
- key: node-role.kubernetes.io/infra
44
-
value: reserved
45
-
effect: NoExecute
46
43
prometheusK8s:
47
44
nodeSelector:
48
45
node-role.kubernetes.io/infra: ""
49
46
tolerations:
50
47
- key: node-role.kubernetes.io/infra
51
48
value: reserved
52
49
effect: NoSchedule
53
-
- key: node-role.kubernetes.io/infra
54
-
value: reserved
55
-
effect: NoExecute
56
50
prometheusOperator:
57
51
nodeSelector:
58
52
node-role.kubernetes.io/infra: ""
59
53
tolerations:
60
54
- key: node-role.kubernetes.io/infra
61
55
value: reserved
62
56
effect: NoSchedule
63
-
- key: node-role.kubernetes.io/infra
64
-
value: reserved
65
-
effect: NoExecute
66
57
k8sPrometheusAdapter:
67
58
nodeSelector:
68
59
node-role.kubernetes.io/infra: ""
69
60
tolerations:
70
61
- key: node-role.kubernetes.io/infra
71
62
value: reserved
72
63
effect: NoSchedule
73
-
- key: node-role.kubernetes.io/infra
74
-
value: reserved
75
-
effect: NoExecute
76
64
kubeStateMetrics:
77
65
nodeSelector:
78
66
node-role.kubernetes.io/infra: ""
79
67
tolerations:
80
68
- key: node-role.kubernetes.io/infra
81
69
value: reserved
82
70
effect: NoSchedule
83
-
- key: node-role.kubernetes.io/infra
84
-
value: reserved
85
-
effect: NoExecute
86
71
telemeterClient:
87
72
nodeSelector:
88
73
node-role.kubernetes.io/infra: ""
89
74
tolerations:
90
75
- key: node-role.kubernetes.io/infra
91
76
value: reserved
92
77
effect: NoSchedule
93
-
- key: node-role.kubernetes.io/infra
94
-
value: reserved
95
-
effect: NoExecute
96
78
openshiftStateMetrics:
97
79
nodeSelector:
98
80
node-role.kubernetes.io/infra: ""
99
81
tolerations:
100
82
- key: node-role.kubernetes.io/infra
101
83
value: reserved
102
84
effect: NoSchedule
103
-
- key: node-role.kubernetes.io/infra
104
-
value: reserved
105
-
effect: NoExecute
106
85
thanosQuerier:
107
86
nodeSelector:
108
87
node-role.kubernetes.io/infra: ""
109
88
tolerations:
110
89
- key: node-role.kubernetes.io/infra
111
90
value: reserved
112
91
effect: NoSchedule
113
-
- key: node-role.kubernetes.io/infra
114
-
value: reserved
115
-
effect: NoExecute
116
92
----
117
-
<1> Add a `nodeSelector` parameter with the appropriate value to the component you want to move. You can use a `nodeSelector` in the format shown or use `<key>: <value>` pairs, based on the value specified for the node. If you added a taint to the infrasructure node, also add a matching toleration.
93
+
<1> Add a `nodeSelector` parameter with the appropriate value to the component you want to move. You can use a `nodeSelector`parameter in the format shown or use `<key>: <value>` pairs, based on the value specified for the node. If you added a taint to the infrastructure node, also add a matching toleration.
118
94
119
95
. Watch the monitoring pods move to the new machines:
<1> Add a `nodeSelector` parameter with the appropriate value to the component you want to move. You can use a `nodeSelector` in the format shown or use `<key>: <value>` pairs, based on the value specified for the node. If you added a taint to the infrasructure node, also add a matching toleration.
79
+
<1> Add a `nodeSelector` parameter with the appropriate value to the component you want to move. You can use a `nodeSelector`parameter in the format shown or use `<key>: <value>` pairs, based on the value specified for the node. If you added a taint to the infrasructure node, also add a matching toleration.
86
80
87
81
. Verify the registry pod has been moved to the infrastructure node.
<1> Add a `nodeSelector` parameter with the appropriate value to the component you want to move. You can use a `nodeSelector` in the format shown or use `<key>: <value>` pairs, based on the value specified for the node. If you added a taint to the infrastructure node, also add a matching toleration.
<1> Add a `nodeSelector` parameter with the appropriate value to the component you want to move. You can use a `nodeSelector` parameter in the format shown or use `<key>: <value>` pairs, based on the value specified for the node. If you added a taint to the infrastructure node, also add a matching toleration.
76
83
77
84
. Confirm that the router pod is running on the `infra` node.
78
85
.. View the list of router pods and note the node name of the running pod:
0 commit comments