Skip to content

Commit 88edcde

Browse files
JSticklerJStickler
authored
Merge pull request #45700 from openshift-cherrypick-robot/cherry-pick-45007-to-enterprise-4.11
[enterprise-4.11] RHDEVDOCS-3919 - document size-based configuration for Prometheus metrics retention
2 parents fdcf49b + 0c32a72 commit 88edcde

3 files changed

+134
-115
lines changed

‎modules/monitoring-modifying-retention-time-and-size-for-prometheus-metrics-data.adoc

+133
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,133 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * monitoring/configuring-the-monitoring-stack.adoc
4+
5+
:_content-type: PROCEDURE
6+
[id="modifying-retention-time-and-size-for-prometheus-metrics-data_{context}"]
7+
= Modifying the retention time and size for Prometheus metrics data
8+
9+
By default, Prometheus automatically retains metrics data for 15 days.
10+
You can modify the retention time to change how soon data is deleted by specifying a time value in the `retention` field.
11+
You can also configure the maximum amount of disk space the retained metrics data uses by specifying a size value in the `retentionSize` field.
12+
If the data reaches this size limit, Prometheus deletes the oldest data first until the disk space used is again below the limit.
13+
14+
Note the following behaviors of these data retention settings:
15+
16+
* The size-based retention policy applies to all data block directories in the `/prometheus` directory, including persistent blocks, write-ahead log (WAL) data, and m-mapped chunks.
17+
* Data in the `/wal` and `/head_chunks` directories counts toward the retention size limit, but Prometheus never purges data from these directories based on size- or time-based retention policies.
18+
Thus, if you set a retention size limit lower than the maximum size set for the `/wal` and `/head_chunks` directories, you have configured the system not to retain any data blocks in the `/prometheus` data directories.
19+
* The size-based retention policy is applied only when Prometheus cuts a new data block, which occurs every two hours after the WAL contains at least three hours of data.
20+
* If you do not explicitly define values for either `retention` or `retentionSize`, retention time defaults to 15 days, and retention size is not set.
21+
* If you define values for both `retention` and `retentionSize`, both values apply.
22+
If any data blocks exceed the defined retention time or the defined size limit, Prometheus purges these data blocks.
23+
* If you define a value for `retentionSize` and do not define `retention`, only the `retentionSize` value applies.
24+
* If you do not define a value for `retentionSize` and only define a value for `retention`, only the `retention` value applies.
25+
26+
.Prerequisites
27+
28+
* *If you are configuring core {product-title} monitoring components*:
29+
** You have access to the cluster as a user with the `cluster-admin` role.
30+
** You have created the `cluster-monitoring-config` `ConfigMap` object.
31+
* *If you are configuring components that monitor user-defined projects*:
32+
** A cluster administrator has enabled monitoring for user-defined projects.
33+
** You have access to the cluster as a user with the `cluster-admin` role, or as a user with the `user-workload-monitoring-config-edit` role in the `openshift-user-workload-monitoring` project.
34+
** You have created the `user-workload-monitoring-config` `ConfigMap` object.
35+
* You have installed the OpenShift CLI (`oc`).
36+
37+
[WARNING]
38+
====
39+
Saving changes to a monitoring config map might restart monitoring processes and redeploy the pods and other resources in the related project.
40+
The running monitoring processes in that project might also restart.
41+
====
42+
43+
.Procedure
44+
45+
. Edit the `ConfigMap` object:
46+
** *To modify the retention time and size for the Prometheus instance that monitors core {product-title} projects*:
47+
.. Edit the `cluster-monitoring-config` `ConfigMap` object in the `openshift-monitoring` project:
48+
+
49+
[source,terminal]
50+
----
51+
$ oc -n openshift-monitoring edit configmap cluster-monitoring-config
52+
----
53+
54+
.. Add the retention time and size configuration under `data/config.yaml`:
55+
+
56+
[source,yaml]
57+
----
58+
apiVersion: v1
59+
kind: ConfigMap
60+
metadata:
61+
name: cluster-monitoring-config
62+
namespace: openshift-monitoring
63+
data:
64+
config.yaml: |
65+
prometheusK8s:
66+
retention: <time_specification> <1>
67+
retentionSize: <size_specification> <2>
68+
----
69+
+
70+
<1> The retention time: a number directly followed by `ms` (milliseconds), `s` (seconds), `m` (minutes), `h` (hours), `d` (days), `w` (weeks), or `y` (years). You can also combine time values for specific times, such as `1h30m15s`.
71+
<2> The retention size: a number directly followed by `B` (bytes), `KB` (kilobytes), `MB` (megabytes), `GB` (gigabytes), `TB` (terabytes), `PB` (petabytes), and `EB` (exabytes).
72+
+
73+
The following example sets the retention time to 24 hours and the retention size to 10 gigabytes for the Prometheus instance that monitors core {product-title} components:
74+
+
75+
[source,yaml,subs=quotes]
76+
----
77+
apiVersion: v1
78+
kind: ConfigMap
79+
metadata:
80+
name: cluster-monitoring-config
81+
namespace: openshift-monitoring
82+
data:
83+
config.yaml: |
84+
prometheusK8s:
85+
retention: *24h*
86+
retentionSize: *10GB*
87+
----
88+
89+
** *To modify the retention time and size for the Prometheus instance that monitors user-defined projects*:
90+
.. Edit the `user-workload-monitoring-config` `ConfigMap` object in the `openshift-user-workload-monitoring` project:
91+
+
92+
[source,terminal]
93+
----
94+
$ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config
95+
----
96+
97+
.. Add the retention time and size configuration under `data/config.yaml`:
98+
+
99+
[source,yaml]
100+
----
101+
apiVersion: v1
102+
kind: ConfigMap
103+
metadata:
104+
name: user-workload-monitoring-config
105+
namespace: openshift-user-workload-monitoring
106+
data:
107+
config.yaml: |
108+
prometheus:
109+
retention: <time_specification> <1>
110+
retentionSize: <size_specification> <2>
111+
----
112+
+
113+
<1> The retention time: a number directly followed by `ms` (milliseconds), `s` (seconds), `m` (minutes), `h` (hours), `d` (days), `w` (weeks), or `y` (years).
114+
You can also combine time values for specific times, such as `1h30m15s`.
115+
<2> The retention size: a number directly followed by `B` (bytes), `KB` (kilobytes), `MB` (megabytes), `GB` (gigabytes), `TB` (terabytes), `PB` (petabytes), or `EB` (exabytes).
116+
+
117+
The following example sets the retention time to 24 hours and the retention size to 10 gigabytes for the Prometheus instance that monitors user-defined projects:
118+
+
119+
[source,yaml,subs=quotes]
120+
----
121+
apiVersion: v1
122+
kind: ConfigMap
123+
metadata:
124+
name: user-workload-monitoring-config
125+
namespace: openshift-user-workload-monitoring
126+
data:
127+
config.yaml: |
128+
prometheus:
129+
retention: *24h*
130+
retentionSize: *10GB*
131+
----
132+
133+
. Save the file to apply the changes. The pods affected by the new configuration restart automatically.

‎modules/monitoring-modifying-retention-time-for-prometheus-metrics-data.adoc

-114
This file was deleted.

‎monitoring/configuring-the-monitoring-stack.adoc

+1-1
Original file line numberDiff line numberDiff line change
@@ -96,7 +96,7 @@ If you use a local volume for persistent storage, do not use a raw block volume,
9696
====
9797

9898
include::modules/monitoring-configuring-a-local-persistent-volume-claim.adoc[leveloffset=+2]
99-
include::modules/monitoring-modifying-retention-time-for-prometheus-metrics-data.adoc[leveloffset=+2]
99+
include::modules/monitoring-modifying-retention-time-and-size-for-prometheus-metrics-data.adoc[leveloffset=+2]
100100

101101
[role="_additional-resources"]
102102
.Additional resources

0 commit comments

Comments
 (0)