Skip to content

Commit 19b6b33

Browse files
aireillyaireilly
committed
low latency tuning refactor
1 parent c2de709 commit 19b6b33

File tree

38 files changed

+493
-603
lines changed

38 files changed

+493
-603
lines changed

‎_topic_maps/_topic_map.yml

+14-3
Original file line numberDiff line numberDiff line change
@@ -2916,10 +2916,21 @@ Topics:
29162916
File: what-huge-pages-do-and-how-they-are-consumed-by-apps
29172917
Distros: openshift-origin,openshift-enterprise
29182918
- Name: Low latency tuning
2919-
File: cnf-low-latency-tuning
2919+
Dir: low_latency_tuning
29202920
Distros: openshift-origin,openshift-enterprise
2921-
- Name: Performing latency tests for platform verification
2922-
File: cnf-performing-platform-verification-latency-tests
2921+
Topics:
2922+
- Name: Understanding low latency
2923+
File: cnf-understanding-low-latency
2924+
- Name: Provisioning real-time and low latency workloads
2925+
File: cnf-provisioning-low-latency-workloads
2926+
- Name: Tuning nodes for low latency with the performance profile
2927+
File: cnf-tuning-low-latency-nodes-with-perf-profile
2928+
- Name: Reducing NIC queues using the Node Tuning Operator
2929+
File: cnf-reducing-nic-queues-with-nto
2930+
- Name: Debugging low latency tuning
2931+
File: cnf-debugging-low-latency-tuning-status
2932+
- Name: Performing latency tests for platform verification
2933+
File: cnf-performing-platform-verification-latency-tests
29232934
- Name: Improving cluster stability in high latency environments using worker latency profiles
29242935
File: scaling-worker-latency-profiles
29252936
- Name: Creating a performance profile

‎installing/installing-preparing.adoc

+1-1
Original file line numberDiff line numberDiff line change
@@ -114,7 +114,7 @@ For a production cluster, you must configure the following integrations:
114114
[id="installing-preparing-cluster-for-workloads"]
115115
== Preparing your cluster for workloads
116116

117-
Depending on your workload needs, you might need to take extra steps before you begin deploying applications. For example, after you prepare infrastructure to support your application xref:../cicd/builds/build-strategies.adoc#build-strategies[build strategy], you might need to make provisions for xref:../scalability_and_performance/cnf-low-latency-tuning.adoc#cnf-low-latency-tuning[low-latency] workloads or to xref:../nodes/pods/nodes-pods-secrets.adoc#nodes-pods-secrets[protect sensitive workloads]. You can also configure xref:../monitoring/enabling-monitoring-for-user-defined-projects.adoc#enabling-monitoring-for-user-defined-projects[monitoring] for application workloads.
117+
Depending on your workload needs, you might need to take extra steps before you begin deploying applications. For example, after you prepare infrastructure to support your application xref:../cicd/builds/build-strategies.adoc#build-strategies[build strategy], you might need to make provisions for xref:../scalability_and_performance/low_latency_tuning/cnf-provisioning-low-latency-workloads.adoc#cnf-provisioning-low-latency-workloads[low-latency] workloads or to xref:../nodes/pods/nodes-pods-secrets.adoc#nodes-pods-secrets[protect sensitive workloads]. You can also configure xref:../monitoring/enabling-monitoring-for-user-defined-projects.adoc#enabling-monitoring-for-user-defined-projects[monitoring] for application workloads.
118118
If you plan to run xref:../windows_containers/enabling-windows-container-workloads.adoc#enabling-windows-container-workloads[Windows workloads], you must enable xref:../networking/ovn_kubernetes_network_provider/configuring-hybrid-networking.adoc#configuring-hybrid-networking[hybrid networking with OVN-Kubernetes] during the installation process; hybrid networking cannot be enabled after your cluster is installed.
119119

120120
[id="supported-installation-methods-for-different-platforms"]

‎modules/cnf-about-irq-affinity-setting.adoc

+3-2
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
// Module included in the following assemblies:
22
//
3-
// scalability_and_performance/cnf-low-latency-tuning.adoc
3+
// * scalability_and_performance/cnf-low-latency-tuning.adoc
4+
// * scalability_and_performance/low_latency_tuning/cnf-tuning-low-latency-nodes-with-perf-profile.adoc
45

56
:_mod-docs-content-type: CONCEPT
67
[id="about_irq_affinity_setting_{context}"]
@@ -60,4 +61,4 @@ $ find /proc/irq -name effective_affinity -printf "%p: " -exec cat {} \;
6061
/proc/irq/34/effective_affinity: 2
6162
----
6263

63-
Some drivers use `managed_irqs`, whose affinity is managed internally by the kernel and userspace cannot change the affinity. In some cases, these IRQs might be assigned to isolated CPUs. For more information about `managed_irqs`, see link:https://access.redhat.com/solutions/4819541[Affinity of managed interrupts cannot be changed even if they target isolated CPU].
64+
Some drivers use `managed_irqs`, whose affinity is managed internally by the kernel and userspace cannot change the affinity. In some cases, these IRQs might be assigned to isolated CPUs. For more information about `managed_irqs`, see link:https://access.redhat.com/solutions/4819541[Affinity of managed interrupts cannot be changed even if they target isolated CPU].

‎modules/cnf-about_hyperthreading_for_low_latency_and_real_time_applications.adoc

+2-1
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
// Module included in the following assemblies:
22
//
3-
// scalability_and_performance/cnf-low-latency-tuning.adoc
3+
// * scalability_and_performance/cnf-low-latency-tuning.adoc
4+
// * scalability_and_performance/low_latency_tuning/cnf-understanding-low-latency.adoc
45

56
:_mod-docs-content-type: CONCEPT
67
[id="about_hyperthreading_for_low_latency_and_real_time_applications_{context}"]

‎modules/cnf-adjusting-nic-queues-with-the-performance-profile.adoc

+3-2
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
// Module included in the following assemblies:
2-
//CNF-1483 (4.8)
3-
// * scalability_and_performance/low-latency-tuning.adoc
2+
//
3+
// * scalability_and_performance/cnf-low-latency-tuning.adoc
4+
// * scalability_and_performance/low_latency_tuning/cnf-reducing-nic-queues-with-nto.adoc
45

56
:_mod-docs-content-type: PROCEDURE
67
[id="adjusting-nic-queues-with-the-performance-profile_{context}"]

‎modules/cnf-allocating-multiple-huge-page-sizes.adoc

+3-3
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
1-
// CNF-538 Promote Multiple Huge Pages Sizes for Pods and Containers to beta
21
// Module included in the following assemblies:
32
//
4-
// *scalability_and_performance/cnf-low-latency-tuning.adoc
3+
// * scalability_and_performance/cnf-low-latency-tuning.adoc
4+
// * scalability_and_performance/low_latency_tuning/cnf-tuning-low-latency-nodes-with-perf-profile.adoc
55

66
[id="cnf-allocating-multiple-huge-page-sizes_{context}"]
77
= Allocating multiple huge page sizes
@@ -22,4 +22,4 @@ spec:
2222
- count: 4
2323
node: 1
2424
size: 1G
25-
----
25+
----

‎modules/cnf-collecting-low-latency-tuning-debugging-data-for-red-hat-support.adoc

+2-2
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
1-
// CNF-643 Support and debugging tools for CNF
21
// Module included in the following assemblies:
32
//
4-
// *scalability_and_performance/cnf-low-latency-tuning.adoc
3+
// * scalability_and_performance/cnf-low-latency-tuning.adoc
4+
// * scalability_and_performance/low_latency_tuning/cnf-debugging-low-latency-tuning-status.adoc
55

66
:_mod-docs-content-type: PROCEDURE
77
[id="cnf-collecting-low-latency-tuning-debugging-data-for-red-hat-support_{context}"]

‎modules/cnf-configure_for_irq_dynamic_load_balancing.adoc

+3-2
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
// Module included in the following assemblies:
22
//
3-
// scalability_and_performance/cnf-low-latency-tuning.adoc
3+
// * scalability_and_performance/cnf-low-latency-tuning.adoc
4+
// * scalability_and_performance/low_latency_tuning/cnf-provisioning-low-latency-workloads.adoc
45

56
:_mod-docs-content-type: PROCEDURE
67
[id="configuring_for_irq_dynamic_load_balancing_{context}"]
@@ -176,4 +177,4 @@ find /proc/irq/ -name smp_affinity_list -exec sh -c 'i="$1"; mask=$(cat $i); fil
176177
/proc/irq/28/smp_affinity_list: 1
177178
/proc/irq/29/smp_affinity_list: 0
178179
/proc/irq/30/smp_affinity_list: 0-5
179-
----
180+
----

‎modules/cnf-configuring-huge-pages.adoc

+2-1
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
// Module included in the following assemblies:
2-
//CNF-78 (4.4)
2+
//
33
// * scalability_and_performance/cnf-low-latency-tuning.adoc
4+
// * scalability_and_performance/low_latency_tuning/cnf-tuning-low-latency-nodes-with-perf-profile.adoc
45

56
[id="cnf-configuring-huge-pages_{context}"]
67
= Configuring huge pages

‎modules/cnf-configuring-workload-hints.adoc

+3-2
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
// Module included in the following assemblies:
22
//
3-
// scalability_and_performance/cnf-low-latency-tuning.adoc
3+
// * scalability_and_performance/cnf-low-latency-tuning.adoc
4+
// * scalability_and_performance/low_latency_tuning/cnf-tuning-low-latency-nodes-with-perf-profile.adoc
45

56
:_mod-docs-content-type: CONCEPT
67
[id="configuring-workload-hints_{context}"]
@@ -30,4 +31,4 @@
3031
[NOTE]
3132
====
3233
When the `realTime` workload hint flag is set to `true` in a performance profile, add the `cpu-quota.crio.io: disable` annotation to every guaranteed pod with pinned CPUs. This annotation is necessary to prevent the degradation of the process performance within the pod. If the `realTime` workload hint is not explicitly set then it defaults to `true`.
33-
====
34+
====

‎modules/cnf-cpu-infra-container.adoc

+2-1
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
// Module included in the following assemblies:
22
//
3-
// scalability_and_performance/cnf-low-latency-tuning.adoc
3+
// * scalability_and_performance/cnf-low-latency-tuning.adoc
4+
// * scalability_and_performance/low_latency_tuning/cnf-tuning-low-latency-nodes-with-perf-profile.adoc
45

56
:_mod-docs-content-type: PROCEDURE
67
[id="cnf-cpu-infra-container_{context}"]

‎modules/cnf-debugging-low-latency-cnf-tuning-status.adoc

+3-5
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,7 @@
11
// Module included in the following assemblies:
2-
// Epic CNF-303 (4.5)
3-
// scalability_and_performance/cnf-low-latency-tuning.adoc
4-
//CNF-303 Performance add-ons status CNF-372
5-
//Performance Addon Operator Detailed Status
6-
//See: https://issues.redhat.com/browse/CNF-379 (Yanir Quinn)
2+
//
3+
// * scalability_and_performance/cnf-low-latency-tuning.adoc
4+
// * scalability_and_performance/low_latency_tuning/cnf-debugging-low-latency-tuning-status.adoc
75

86
[id="cnf-debugging-low-latency-cnf-tuning-status_{context}"]
97
= Debugging low latency CNF tuning status

‎modules/cnf-logging-associated-with-adjusting-nic-queues.adoc

+2-1
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
// Module included in the following assemblies:
2-
//CNF-1483 (4.8)
2+
//
33
// * scalability_and_performance/cnf-low-latency-tuning.adoc
4+
// * scalability_and_performance/low_latency_tuning/cnf-reducing-nic-queues-with-nto.adoc
45

56
[id="logging-associated-with-adjusting-nic-queues_{context}"]
67
= Logging associated with adjusting NIC queues

‎modules/cnf-managing-device-interrupt-processing-for-guaranteed-pod-isolated-cpus.adoc

+3-3
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
1-
// CNF-802 Infrastructure-provided interrupt processing for guaranteed pod CPUs
21
// Module included in the following assemblies:
32
//
4-
// *cnf-low-latency-tuning.adoc
3+
// * scalability_and_performance/cnf-low-latency-tuning.adoc
4+
// * scalability_and_performance/low_latency_tuning/cnf-provisioning-low-latency-workloads.adoc
55

66
[id="managing-device-interrupt-processing-for-guaranteed-pod-isolated-cpus_{context}"]
77
= Managing device interrupt processing for guaranteed pod isolated CPUs
@@ -54,7 +54,7 @@ spec:
5454
...
5555
----
5656

57-
[id="disabling_interrupt_processing_for_individual_pods_{context}"]
57+
[id="disabling-interrupt-processing-for-individual-pods_{context}"]
5858
== Disabling interrupt processing for individual pods
5959

6060
To disable interrupt processing for individual pods, ensure that `globallyDisableIrqLoadBalancing` is set to `false` in the performance profile. Then, in the pod specification, set the `irq-load-balancing.crio.io` pod annotation to `disable`. The following pod specification contains this annotation:

‎modules/cnf-node-tuning-operator-creating-pod-with-guaranteed-qos-class.adoc

+63
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * scalability_and_performance/low_latency_tuning/cnf-provisioning-low-latency-workloads.adoc
4+
5+
:_mod-docs-content-type: PROCEDURE
6+
[id="cnf-node-tuning-operator-creating-pod-with-guaranteed-qos-class_{context}"]
7+
= Creating a pod with a QoS class of `Guaranteed`
8+
9+
Keep the following in mind when you create a pod that is given a QoS class of `Guaranteed`:
10+
11+
* Every container in the pod must have a memory limit and a memory request, and they must be the same.
12+
* Every container in the pod must have a CPU limit and a CPU request, and they must be the same.
13+
14+
The following example shows the configuration file for a pod that has one container. The container has a memory limit and a memory request, both equal to 200 MiB. The container has a CPU limit and a CPU request, both equal to 1 CPU.
15+
16+
[source,yaml]
17+
----
18+
apiVersion: v1
19+
kind: Pod
20+
metadata:
21+
name: qos-demo
22+
namespace: qos-example
23+
spec:
24+
containers:
25+
- name: qos-demo-ctr
26+
image: <image-pull-spec>
27+
resources:
28+
limits:
29+
memory: "200Mi"
30+
cpu: "1"
31+
requests:
32+
memory: "200Mi"
33+
cpu: "1"
34+
----
35+
36+
. Create the pod:
37+
+
38+
[source,terminal]
39+
----
40+
$ oc apply -f qos-pod.yaml --namespace=qos-example
41+
----
42+
43+
. View detailed information about the pod:
44+
+
45+
[source,terminal]
46+
----
47+
$ oc get pod qos-demo --namespace=qos-example --output=yaml
48+
----
49+
+
50+
.Example output
51+
[source,yaml]
52+
----
53+
spec:
54+
containers:
55+
...
56+
status:
57+
qosClass: Guaranteed
58+
----
59+
+
60+
[NOTE]
61+
====
62+
If a container specifies its own memory limit, but does not specify a memory request, {product-title} automatically assigns a memory request that matches the limit. Similarly, if a container specifies its own CPU limit, but does not specify a CPU request, {product-title} automatically assigns a CPU request that matches the limit.
63+
====

‎modules/cnf-node-tuning-operator-creating-workload-that-works-in-real-time.adoc

+40
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * scalability_and_performance/low_latency_tuning/cnf-provisioning-low-latency-workloads.adoc
4+
5+
:_mod-docs-content-type: PROCEDURE
6+
[id="cnf-node-tuning-operator-creating-workload-that-works-in-real-time_{context}"]
7+
= Creating a workload that works in real-time
8+
9+
Use the following procedure to prepare a workload that uses real-time capabilities.
10+
11+
.Procedure
12+
13+
. Create a pod with a QoS class of `Guaranteed`.
14+
15+
. Optional: Disable CPU load balancing for DPDK.
16+
17+
. Assign a proper node selector.
18+
The preferred way to assign a pod to nodes is to use the same node selector the performance profile used, as shown here:
19+
+
20+
[source,yaml]
21+
----
22+
apiVersion: v1
23+
kind: Pod
24+
metadata:
25+
name: example
26+
spec:
27+
# ...
28+
nodeSelector:
29+
node-role.kubernetes.io/worker-rt: ""
30+
----
31+
+
32+
[NOTE]
33+
====
34+
For more information, see xref:../../nodes/pods/nodes-pods-node-selectors.adoc#nodes-pods-node-selectors[Placing pods on specific nodes using node selectors]
35+
====
36+
37+
[NOTE]
38+
====
39+
When writing your applications, follow the general recommendations described in link:https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux_for_real_time/8/html-single/tuning_guide/index#chap-Application_Tuning_and_Deployment[Application tuning and deployment].
40+
====

‎modules/cnf-node-tuning-operator-disabling-cpu-load-balancing-for-dpdk.adoc

+52
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * scalability_and_performance/low_latency_tuning/cnf-provisioning-low-latency-workloads.adoc
4+
5+
:_mod-docs-content-type: PROCEDURE
6+
[id="cnf-node-tuning-operator-disabling-cpu-load-balancing-for-dpdk_{context}"]
7+
= Disabling CPU load balancing for DPDK
8+
9+
Functionality to disable or enable CPU load balancing is implemented on the CRI-O level. The code under the CRI-O disables or enables CPU load balancing only when the following requirements are met.
10+
11+
* The pod must use the `performance-<profile-name>` runtime class. You can get the proper name by looking at the status of the performance profile, as shown here:
12+
+
13+
[source,yaml]
14+
----
15+
apiVersion: performance.openshift.io/v2
16+
kind: PerformanceProfile
17+
...
18+
status:
19+
...
20+
runtimeClass: performance-manual
21+
----
22+
23+
[NOTE]
24+
====
25+
Currently, disabling CPU load balancing is not supported with cgroup v2.
26+
====
27+
28+
The Node Tuning Operator is responsible for the creation of the high-performance runtime handler config snippet under relevant nodes and for creation of the high-performance runtime class under the cluster. It will have the same content as default runtime handler except it enables the CPU load balancing configuration functionality.
29+
30+
To disable the CPU load balancing for the pod, the `Pod` specification must include the following fields:
31+
32+
[source,yaml]
33+
----
34+
apiVersion: v1
35+
kind: Pod
36+
metadata:
37+
...
38+
annotations:
39+
...
40+
cpu-load-balancing.crio.io: "disable"
41+
...
42+
...
43+
spec:
44+
...
45+
runtimeClassName: performance-<profile_name>
46+
...
47+
----
48+
49+
[NOTE]
50+
====
51+
Only disable CPU load balancing when the CPU manager static policy is enabled and for pods with guaranteed QoS that use whole CPUs. Otherwise, disabling CPU load balancing can affect the performance of other containers in the cluster.
52+
====

0 commit comments

Comments
 (0)