Skip to content

Commit e643d77

Browse files
aireillyaireilly
committed
low latency tuning refactor
Changes for Martin Martin's 3rd review comments Martin's 4th review Final comments from Martin More updates for Martin Tweaks for Martin Martin's comments 22-Apr - workload pods Apr 30 review comments final review comments Michael's comments
1 parent f685ca7 commit e643d77

File tree

67 files changed

+898
-843
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

67 files changed

+898
-843
lines changed

‎_topic_maps/_topic_map.yml

+12-5
Original file line numberDiff line numberDiff line change
@@ -2901,14 +2901,21 @@ Topics:
29012901
File: what-huge-pages-do-and-how-they-are-consumed-by-apps
29022902
Distros: openshift-origin,openshift-enterprise
29032903
- Name: Low latency tuning
2904-
File: cnf-low-latency-tuning
2904+
Dir: low_latency_tuning
29052905
Distros: openshift-origin,openshift-enterprise
2906-
- Name: Performing latency tests for platform verification
2907-
File: cnf-performing-platform-verification-latency-tests
2906+
Topics:
2907+
- Name: Understanding low latency
2908+
File: cnf-understanding-low-latency
2909+
- Name: Tuning nodes for low latency with the performance profile
2910+
File: cnf-tuning-low-latency-nodes-with-perf-profile
2911+
- Name: Provisioning real-time and low latency workloads
2912+
File: cnf-provisioning-low-latency-workloads
2913+
- Name: Debugging low latency tuning
2914+
File: cnf-debugging-low-latency-tuning-status
2915+
- Name: Performing latency tests for platform verification
2916+
File: cnf-performing-platform-verification-latency-tests
29082917
- Name: Improving cluster stability in high latency environments using worker latency profiles
29092918
File: scaling-worker-latency-profiles
2910-
- Name: Creating a performance profile
2911-
File: cnf-create-performance-profiles
29122919
Distros: openshift-origin,openshift-enterprise
29132920
- Name: Workload partitioning
29142921
File: enabling-workload-partitioning

‎installing/installing-preparing.adoc

+1-1
Original file line numberDiff line numberDiff line change
@@ -112,7 +112,7 @@ For a production cluster, you must configure the following integrations:
112112
[id="installing-preparing-cluster-for-workloads"]
113113
== Preparing your cluster for workloads
114114

115-
Depending on your workload needs, you might need to take extra steps before you begin deploying applications. For example, after you prepare infrastructure to support your application xref:../cicd/builds/build-strategies.adoc#build-strategies[build strategy], you might need to make provisions for xref:../scalability_and_performance/cnf-low-latency-tuning.adoc#cnf-low-latency-tuning[low-latency] workloads or to xref:../nodes/pods/nodes-pods-secrets.adoc#nodes-pods-secrets[protect sensitive workloads]. You can also configure xref:../observability/monitoring/enabling-monitoring-for-user-defined-projects.adoc#enabling-monitoring-for-user-defined-projects[monitoring] for application workloads.
115+
Depending on your workload needs, you might need to take extra steps before you begin deploying applications. For example, after you prepare infrastructure to support your application xref:../cicd/builds/build-strategies.adoc#build-strategies[build strategy], you might need to make provisions for xref:../scalability_and_performance/low_latency_tuning/cnf-tuning-low-latency-nodes-with-perf-profile.adoc#cnf-low-latency-perf-profile[low-latency] workloads or to xref:../nodes/pods/nodes-pods-secrets.adoc#nodes-pods-secrets[protect sensitive workloads]. You can also configure xref:../observability/monitoring/enabling-monitoring-for-user-defined-projects.adoc#enabling-monitoring-for-user-defined-projects[monitoring] for application workloads.
116116
If you plan to run xref:../windows_containers/enabling-windows-container-workloads.adoc#enabling-windows-container-workloads[Windows workloads], you must enable xref:../networking/ovn_kubernetes_network_provider/configuring-hybrid-networking.adoc#configuring-hybrid-networking[hybrid networking with OVN-Kubernetes] during the installation process; hybrid networking cannot be enabled after your cluster is installed.
117117

118118
[id="supported-installation-methods-for-different-platforms"]

‎installing/installing_openstack/installing-openstack-nfv-preparing.adoc

+2-1
Original file line numberDiff line numberDiff line change
@@ -42,4 +42,5 @@ After you perform preinstallation tasks, install your cluster by following the m
4242
* Consult the following references after you deploy your cluster to improve its performance:
4343
** xref:../../networking/hardware_networks/using-dpdk-and-rdma.adoc#nw-openstack-ovs-dpdk-testpmd-pod_using-dpdk-and-rdma[A test pod template for clusters that use OVS-DPDK on OpenStack].
4444
** xref:../../networking/hardware_networks/add-pod.adoc#nw-openstack-sr-iov-testpmd-pod_add-pod[A test pod template for clusters that use SR-IOV on OpenStack].
45-
** xref:../../scalability_and_performance/cnf-create-performance-profiles.adoc#installation-openstack-ovs-dpdk-performance-profile_cnf-create-performance-profiles[A performance profile template for clusters that use OVS-DPDK on OpenStack].
45+
** xref:../../scalability_and_performance/low_latency_tuning/cnf-tuning-low-latency-nodes-with-perf-profile.adoc#installation-openstack-ovs-dpdk-performance-profile_cnf-low-latency-perf-profile[A performance profile template for clusters that use OVS-DPDK on OpenStack]
46+
.

‎modules/cnf-about-hyperthreading-for-low-latency-and-real-time-applications.adoc

+16
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * scalability_and_performance/low_latency_tuning/cnf-understanding-low-latency.adoc
4+
5+
:_mod-docs-content-type: CONCEPT
6+
[id="cnf-about-hyper-threading-for-low-latency-and-real-time-applications_{context}"]
7+
= About Hyper-Threading for low latency and real-time applications
8+
9+
Hyper-Threading is an Intel processor technology that allows a physical CPU processor core to function as two logical cores, executing two independent threads simultaneously. Hyper-Threading allows for better system throughput for certain workload types where parallel processing is beneficial. The default {product-title} configuration expects Hyper-Threading to be enabled.
10+
11+
For telecommunications applications, it is important to design your application infrastructure to minimize latency as much as possible. Hyper-Threading can slow performance times and negatively affect throughput for compute-intensive workloads that require low latency. Disabling Hyper-Threading ensures predictable performance and can decrease processing times for these workloads.
12+
13+
[NOTE]
14+
====
15+
Hyper-Threading implementation and configuration differs depending on the hardware you are running {product-title} on. Consult the relevant host hardware tuning information for more details of the Hyper-Threading implementation specific to that hardware. Disabling Hyper-Threading can increase the cost per core of the cluster.
16+
====

‎modules/cnf-about-irq-affinity-setting.adoc

+4-3
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,11 @@
11
// Module included in the following assemblies:
22
//
3-
// scalability_and_performance/cnf-low-latency-tuning.adoc
3+
// * scalability_and_performance/cnf-low-latency-tuning.adoc
4+
// * scalability_and_performance/low_latency_tuning/cnf-tuning-low-latency-nodes-with-perf-profile.adoc
45

56
:_mod-docs-content-type: CONCEPT
67
[id="about_irq_affinity_setting_{context}"]
7-
= About support of IRQ affinity setting
8+
= Finding the effective IRQ affinity setting for a node
89

910
Some IRQ controllers lack support for IRQ affinity setting and will always expose all online CPUs as the IRQ mask. These IRQ controllers effectively run on CPU 0.
1011

@@ -60,4 +61,4 @@ $ find /proc/irq -name effective_affinity -printf "%p: " -exec cat {} \;
6061
/proc/irq/34/effective_affinity: 2
6162
----
6263

63-
Some drivers use `managed_irqs`, whose affinity is managed internally by the kernel and userspace cannot change the affinity. In some cases, these IRQs might be assigned to isolated CPUs. For more information about `managed_irqs`, see link:https://access.redhat.com/solutions/4819541[Affinity of managed interrupts cannot be changed even if they target isolated CPU].
64+
Some drivers use `managed_irqs`, whose affinity is managed internally by the kernel and userspace cannot change the affinity. In some cases, these IRQs might be assigned to isolated CPUs. For more information about `managed_irqs`, see link:https://access.redhat.com/solutions/4819541[Affinity of managed interrupts cannot be changed even if they target isolated CPU].

‎modules/cnf-about-the-profile-creator-tool.adoc

+2-2
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
// Module included in the following assemblies:
2-
// Epic CNF-792 (4.8)
3-
// * scalability_and_performance/cnf-create-performance-profiles.adoc
2+
//
3+
// * scalability_and_performance/low_latency_tuning/cnf-tuning-low-latency-nodes-with-perf-profile.adoc
44

55
:_mod-docs-content-type: CONCEPT
66
[id="cnf-about-the-profile-creator-tool_{context}"]

‎modules/cnf-about_hyperthreading_for_low_latency_and_real_time_applications.adoc

-16
This file was deleted.

‎modules/cnf-adjusting-nic-queues-with-the-performance-profile.adoc

+3-3
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
// Module included in the following assemblies:
2-
//CNF-1483 (4.8)
3-
// * scalability_and_performance/low-latency-tuning.adoc
2+
//
3+
// * scalability_and_performance/low_latency_tuning/cnf-tuning-low-latency-nodes-with-perf-profile.adoc
44

55
:_mod-docs-content-type: PROCEDURE
66
[id="adjusting-nic-queues-with-the-performance-profile_{context}"]
@@ -165,4 +165,4 @@ spec:
165165
[source,terminal]
166166
----
167167
$ oc apply -f <your_profile_name>.yaml
168-
----
168+
----

‎modules/cnf-allocating-multiple-huge-page-sizes.adoc

+3-3
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
1-
// CNF-538 Promote Multiple Huge Pages Sizes for Pods and Containers to beta
21
// Module included in the following assemblies:
32
//
4-
// *scalability_and_performance/cnf-low-latency-tuning.adoc
3+
// * scalability_and_performance/cnf-low-latency-tuning.adoc
4+
// * scalability_and_performance/low_latency_tuning/cnf-tuning-low-latency-nodes-with-perf-profile.adoc
55

66
[id="cnf-allocating-multiple-huge-page-sizes_{context}"]
77
= Allocating multiple huge page sizes
@@ -22,4 +22,4 @@ spec:
2222
- count: 4
2323
node: 1
2424
size: 1G
25-
----
25+
----

‎modules/cnf-collecting-low-latency-tuning-debugging-data-for-red-hat-support.adoc

+2-2
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
1-
// CNF-643 Support and debugging tools for CNF
21
// Module included in the following assemblies:
32
//
4-
// *scalability_and_performance/cnf-low-latency-tuning.adoc
3+
// * scalability_and_performance/cnf-low-latency-tuning.adoc
4+
// * scalability_and_performance/low_latency_tuning/cnf-debugging-low-latency-tuning-status.adoc
55

66
:_mod-docs-content-type: PROCEDURE
77
[id="cnf-collecting-low-latency-tuning-debugging-data-for-red-hat-support_{context}"]

‎modules/cnf-configure_for_irq_dynamic_load_balancing.adoc

+1-1
Original file line numberDiff line numberDiff line change
@@ -176,4 +176,4 @@ find /proc/irq/ -name smp_affinity_list -exec sh -c 'i="$1"; mask=$(cat $i); fil
176176
/proc/irq/28/smp_affinity_list: 1
177177
/proc/irq/29/smp_affinity_list: 0
178178
/proc/irq/30/smp_affinity_list: 0-5
179-
----
179+
----

‎modules/cnf-configuring-high-priority-workload-pods.adoc

+55
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * scalability_and_performance/low_latency_tuning/cnf-provisioning-low-latency-workloads.adoc
4+
5+
:_mod-docs-content-type: PROCEDURE
6+
[id="cnf-configuring-high-priority-workload-pods_{context}"]
7+
= Disabling power saving mode for high priority pods
8+
9+
You can configure pods to ensure that high priority workloads are unaffected when you configure power saving for the node that the workloads run on.
10+
11+
When you configure a node with a power saving configuration, you must configure high priority workloads with performance configuration at the pod level, which means that the configuration applies to all the cores used by the pod.
12+
13+
By disabling P-states and C-states at the pod level, you can configure high priority workloads for best performance and lowest latency.
14+
15+
.Configuration for high priority workloads
16+
[cols="1,2,3", options="header"]
17+
18+
|===
19+
| Annotation | Possible Values | Description
20+
21+
|`cpu-c-states.crio.io:` a| * `"enable"`
22+
* `"disable"`
23+
* `"max_latency:microseconds"` | This annotation allows you to enable or disable C-states for each CPU. Alternatively, you can also specify a maximum latency in microseconds for the C-states. For example, enable C-states with a maximum latency of 10 microseconds with the setting `cpu-c-states.crio.io`: `"max_latency:10"`. Set the value to `"disable"` to provide the best performance for a pod.
24+
25+
| `cpu-freq-governor.crio.io:` | Any supported `cpufreq governor`. | Sets the `cpufreq` governor for each CPU. The `"performance"` governor is recommended for high priority workloads.
26+
|===
27+
28+
.Prerequisites
29+
30+
* You have configured power saving in the performance profile for the node where the high priority workload pods are scheduled.
31+
32+
.Procedure
33+
34+
. Add the required annotations to your high priority workload pods. The annotations override the `default` settings.
35+
+
36+
.Example high priority workload annotation
37+
[source,yaml]
38+
----
39+
apiVersion: v1
40+
kind: Pod
41+
metadata:
42+
#...
43+
annotations:
44+
#...
45+
cpu-c-states.crio.io: "disable"
46+
cpu-freq-governor.crio.io: "performance"
47+
#...
48+
#...
49+
spec:
50+
#...
51+
runtimeClassName: performance-<profile_name>
52+
#...
53+
----
54+
55+
. Restart the pods to apply the annotation.

‎modules/cnf-configuring-huge-pages.adoc

+2-1
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
// Module included in the following assemblies:
2-
//CNF-78 (4.4)
2+
//
33
// * scalability_and_performance/cnf-low-latency-tuning.adoc
4+
// * scalability_and_performance/low_latency_tuning/cnf-tuning-low-latency-nodes-with-perf-profile.adoc
45

56
[id="cnf-configuring-huge-pages_{context}"]
67
= Configuring huge pages

‎modules/configuring_hyperthreading_for_a_cluster.adoc renamed to ‎modules/cnf-configuring-hyperthreading-for-a-cluster.adoc

+14-11
Original file line numberDiff line numberDiff line change
@@ -1,21 +1,21 @@
11
// Module included in the following assemblies:
22
//
3-
// scalability_and_performance/cnf-low-latency-tuning.adoc
3+
// * scalability_and_performance/low_latency_tuning/cnf-tuning-low-latency-nodes-with-perf-profile.adoc
44

55
:_mod-docs-content-type: PROCEDURE
6-
[id="configuring_hyperthreading_for_a_cluster_{context}"]
7-
= Configuring hyperthreading for a cluster
6+
[id="cnf-configuring-hyperthreading-for-a-cluster_{context}"]
7+
= Configuring Hyper-Threading for a cluster
88

9-
To configure hyperthreading for an {product-title} cluster, set the CPU threads in the performance profile to the same cores that are configured for the reserved or isolated CPU pools.
9+
To configure Hyper-Threading for an {product-title} cluster, set the CPU threads in the performance profile to the same cores that are configured for the reserved or isolated CPU pools.
1010

1111
[NOTE]
1212
====
13-
If you configure a performance profile, and subsequently change the hyperthreading configuration for the host, ensure that you update the CPU `isolated` and `reserved` fields in the `PerformanceProfile` YAML to match the new configuration.
13+
If you configure a performance profile, and subsequently change the Hyper-Threading configuration for the host, ensure that you update the CPU `isolated` and `reserved` fields in the `PerformanceProfile` YAML to match the new configuration.
1414
====
1515

1616
[WARNING]
1717
====
18-
Disabling a previously enabled host hyperthreading configuration can cause the CPU core IDs listed in the `PerformanceProfile` YAML to be incorrect. This incorrect configuration can cause the node to become unavailable because the listed CPUs can no longer be found.
18+
Disabling a previously enabled host Hyper-Threading configuration can cause the CPU core IDs listed in the `PerformanceProfile` YAML to be incorrect. This incorrect configuration can cause the node to become unavailable because the listed CPUs can no longer be found.
1919
====
2020

2121
.Prerequisites
@@ -51,7 +51,7 @@ CPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE MAXMHZ MINMHZ
5151
+
5252
In this example, there are eight logical CPU cores running on four physical CPU cores. CPU0 and CPU4 are running on physical Core0, CPU1 and CPU5 are running on physical Core 1, and so on.
5353
+
54-
Alternatively, to view the threads that are set for a particular physical CPU core (`cpu0` in the example below), open a command prompt and run the following:
54+
Alternatively, to view the threads that are set for a particular physical CPU core (`cpu0` in the example below), open a shell prompt and run the following:
5555
+
5656
[source,terminal]
5757
----
@@ -83,20 +83,23 @@ The reserved and isolated CPU pools must not overlap and together must span all
8383
8484
[IMPORTANT]
8585
====
86-
Hyperthreading is enabled by default on most Intel processors. If you enable hyperthreading, all threads processed by a particular core must be isolated or processed on the same core.
86+
Hyper-Threading is enabled by default on most Intel processors. If you enable Hyper-Threading, all threads processed by a particular core must be isolated or processed on the same core.
87+
88+
When Hyper-Threading is enabled, all guaranteed pods must use multiples of the simultaneous multi-threading (SMT) level to avoid a "noisy neighbor" situation that can cause the pod to fail.
89+
See link:https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/#static-policy-options[Static policy options] for more information.
8790
====
8891
8992
[id="disabling_hyperthreading_for_low_latency_applications_{context}"]
90-
== Disabling hyperthreading for low latency applications
93+
== Disabling Hyper-Threading for low latency applications
9194
92-
When configuring clusters for low latency processing, consider whether you want to disable hyperthreading before you deploy the cluster. To disable hyperthreading, do the following:
95+
When configuring clusters for low latency processing, consider whether you want to disable Hyper-Threading before you deploy the cluster. To disable Hyper-Threading, perform the following steps:
9396
9497
. Create a performance profile that is appropriate for your hardware and topology.
9598
. Set `nosmt` as an additional kernel argument. The following example performance profile illustrates this setting:
9699
+
97100
[source,yaml]
98101
----
99-
apiVersion: performance.openshift.io/v2
102+
apiVersion: performance.openshift.io/v2
100103
kind: PerformanceProfile
101104
metadata:
102105
name: example-performanceprofile

0 commit comments

Comments
 (0)