Skip to content

Commit 443ce25

Browse files
libanderlibander
committed
Logging 6x - POC WIP
1 parent db7835c commit 443ce25

File tree

6 files changed

+371
-0
lines changed

6 files changed

+371
-0
lines changed

‎_topic_maps/_topic_map.yml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2709,6 +2709,11 @@ Topics:
27092709
File: logging-5-8-release-notes
27102710
- Name: Logging 5.7
27112711
File: logging-5-7-release-notes
2712+
- Name: Logging 6.0
2713+
Dir: logging-6.0
2714+
Topics:
2715+
- Name: Introducing Logging 6.0
2716+
File: logging-poc
27122717
- Name: Support
27132718
File: cluster-logging-support
27142719
- Name: Troubleshooting logging

‎modules/logging-6x-about.adoc

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
// Module included in the following assemblies:
2+
//
3+
4+
5+
:_mod-docs-content-type: CONCEPT
6+
[id="logging-6x-about_{context}"]
7+
8+
9+
= What is OpenShift Logging?
10+
11+
OpenShift Logging provides a comprehensive log management solution integrated into your OpenShift clusters. It utilizes:
12+
13+
* **Loki:** A horizontally scalable, highly available log aggregation system optimized for efficient storage and retrieval of log data in dynamic OpenShift environments.
14+
* **Vector:** A lightweight, high-performance log forwarding agent that connects your application and infrastructure logs to Loki, offering a flexible and customizable way to collect, process, and route log data.
15+
16+
== How OpenShift Logging Works
17+
18+
1. **Log Collection:** Vector agents, deployed on each node in your cluster, collect logs from various sources, including containers, system components, and applications.
19+
2. **Log Processing (Optional):** Vector can filter, enrich, and transform log data before forwarding it to Loki.
20+
3. **Log Aggregation:** Loki receives the processed log data from Vector and stores it efficiently for later retrieval.
21+
4. **Log Querying & Analysis:** Use Loki's powerful query language to search, filter, and analyze logs, gaining valuable insights into your OpenShift environment.
22+
23+
== Visualizing Logs in OpenShift
24+
25+
OpenShift Logging provides multiple ways to visualize and explore your log data:
26+
27+
* **OpenShift Web Console:** The integrated Logs page offers a user-friendly interface for searching, filtering, and viewing logs directly within the OpenShift console.
28+
* **Grafana:** OpenShift Logging integrates seamlessly with Grafana, enabling you to create rich, customizable dashboards to visualize and correlate log data with metrics and other observability data.

‎modules/logging-6x-loki-architecture.adoc

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
// Module included in the following assemblies:
2+
//
3+
4+
5+
:_mod-docs-content-type: CONCEPT
6+
[id="logging-6x-loki-architecture_{context}"]
7+
= Loki Architecture: Understanding the Components
8+
Loki is a distributed system with several core components, each serving a specific purpose in the logging pipeline:
9+
10+
* **Distributor:** Receives incoming log streams, validates them, and distributes them to the appropriate ingesters.
11+
* **Ingester:** Builds chunks of log data and creates an index to enable efficient querying.
12+
* **Querier:** Executes queries against the log data, retrieving relevant log entries from the ingesters and storage backend.
13+
* **Query Frontend:** Manages and coordinates incoming queries, distributes the workload across queriers, and returns the query results.
14+
* **Storage Backend:** Persists the log data and index, typically using an object storage like Amazon S3 or Google Cloud Storage.
15+
* **Compactor:** Optimizes the storage of log data by compacting older log chunks into a more efficient format.
16+
* **Ruler:** Allows you to define rules for alerting and recording based on log data patterns.
17+
18+
19+
20+
= LokiStack Custom Resource (CR): Mapping Fields to Components
21+
22+
Custom Resource (CR) files are used to configure Operators.
23+
24+
* **`spec`:** This is effectively the root of the CR
25+
26+
* **Component-Specific Sections:** Within `spec`, you'll find subsections for each component (e.g., `distributor`, `ingester`, `querier`). The fields in each section directly control that component's behavior.
27+
28+
[source,yaml]
29+
----
30+
spec:
31+
distributor:
32+
replicas: 3 #<1>
33+
ingester:
34+
replicas: 2
35+
resources:
36+
limits:
37+
memory: 1Gb #<2>
38+
querier:
39+
replicas: 2
40+
queryFrontend:
41+
replicas: 1
42+
...
43+
----
44+
45+
<1> `spec.distributor.replicas: 3` means you want 3 pods running the Distributor component.
46+
<2> `spec.ingester.resources.limits.memory: 1Gb` gives each Ingester pod the memory limit specified.
47+
48+
49+
== Key Points:
50+
51+
* **Hierarchy:** `spec` is the top level, then components, then specific settings for that component.
52+
* **Direct Mapping:** Each field in a component section directly affects that component's configuration in your Loki cluster.
53+
* **Customization:** By tweaking these values, you tailor Loki to your workload and resource constraints.
54+
55+
56+
57+
== Distributor:
58+
59+
* `spec.distributor.replicas`: Controls the number of distributor replicas.
60+
* `spec.distributor.resources`: Sets resource limits (CPU, memory) for the distributor pods.
61+
* `spec.distributor.ring`: Configures the hash ring for distributing log streams among ingesters.
62+
63+
== Ingester:
64+
65+
* `spec.ingester.replicas`: Controls the number of ingester replicas.
66+
* `spec.ingester.resources`: Sets resource limits (CPU, memory) for the ingester pods.
67+
* `spec.ingester.lifecycler`: Configures the ingester lifecycle, including chunk lifecycle and table manager settings.
68+
* `spec.ingester.persistence`: Configures the persistent storage for ingester data (e.g., volume claims, storage class).
69+
70+
== Querier:
71+
72+
* `spec.querier.replicas`: Controls the number of querier replicas.
73+
* `spec.querier.resources`: Sets resource limits (CPU, memory) for the querier pods.
74+
* `spec.querier.queryRange`: Configures the default query range (time duration) for queriers.
75+
* `spec.querier.storeGateway`: Configures the interaction between queriers and the storage backend.
76+
77+
== Query Frontend:
78+
79+
* `spec.queryFrontend.replicas`: Controls the number of query frontend replicas.
80+
* `spec.queryFrontend.resources`: Sets resource limits (CPU, memory) for the query frontend pods.
81+
* `spec.queryFrontend.grpcServer`: Configures the gRPC server for the query frontend.
82+
83+
== Storage Backend:
84+
85+
* `spec.storageConfig`: Specifies the type of storage backend (e.g., S3, GCS, local filesystem) and its configuration parameters (e.g., bucket name, credentials).
86+
87+
== Compactor:
88+
89+
* `spec.compactor.replicas`: Controls the number of compactor replicas.
90+
* `spec.compactor.resources`: Sets resource limits (CPU, memory) for the compactor pods.
91+
* `spec.compactor.workingDirectory`: Specifies the working directory for the compactor.
92+
* `spec.compactor.sharedStore`: Configures the shared storage for compactor data.
93+
94+
== Ruler:
95+
96+
* `spec.ruler.replicas`: Controls the number of ruler replicas.
97+
* `spec.ruler.resources`: Sets resource limits (CPU, memory) for the ruler pods.
98+
* `spec.ruler.alertmanagerURL`: Specifies the URL of the Alertmanager instance to send alerts to.
99+
* `spec.ruler.ruleNamespaceSelector`: Selects the namespaces from which to load ruler rules.
100+
* `spec.ruler.storage`: Configures the storage for ruler data.
101+
102+
== Other Components:
103+
104+
The LokiStack CR also allows you to configure other aspects of the Loki deployment, such as:
105+
106+
* `spec.limits`: Global limits for Loki components.
107+
* `spec.storageClassName`: Default storage class for persistent volumes.
108+
* `spec.serviceMonitor`: Configuration for Prometheus service monitors.
109+
* `spec.tenants`: Configuration for multi-tenancy.
110+
111+
By modifying the values in the LokiStack CR, you can fine-tune the behavior and resource allocation of each Loki component to match your specific requirements and workload.

‎modules/logging-6x-v-5x.adoc

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
// Module included in the following assemblies:
2+
//
3+
4+
5+
:_mod-docs-content-type: PROCEDURE
6+
[id="logging-6x-v-5x_{context}"]
7+
= OpenShift Logging 6.0: Key Differences from Previous Versions
8+
9+
OpenShift Logging 6.0 is the culmination of efforts to modernize the underlying components of the logging solution. Over the past few releases, Loki and Vector were gradually introduced with a focus on improving performance and enhancing features while maintaining the stability required for business continuity.
10+
11+
12+
* **Horizontal Scalability:** Loki's architecture allows for seamless horizontal scaling, ensuring that the logging system can handle the growing demands of large-scale OpenShift deployments.
13+
14+
* **Improved Query Performance:** Loki's indexing and query mechanisms are optimized for speed, enabling faster log retrieval and analysis.
15+
16+
* **Streamlined Configuration:** The removal of Elasticsearch and Fluentd simplify setup and configuration.
17+
18+
* **Improved Integration:** Loki and Vector offer tighter integration with OpenShift, providing a more seamless and native logging experience.
19+
20+
* **Expanded Log Sources:** Vector's flexibility allows you to collect logs from an even wider range of sources, providing greater visibility into your OpenShift environment.
21+
22+
* **Enhanced Log Processing:** Vector's processing capabilities have been expanded, enabling you to perform more complex log transformations and filtering.
23+
24+
* **OpenShift Web Console:** The integrated Logs page offers a user-friendly interface for searching, filtering, and viewing logs directly within the OpenShift console.
25+
26+
* **Grafana:** OpenShift Logging integrates seamlessly with Grafana, enabling you to create rich, customizable dashboards to visualize and correlate log data with metrics and other observability data.

‎modules/logging-elastic-to-loki-migration.adoc

Lines changed: 186 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,186 @@
1+
// Module included in the following assemblies:
2+
//
3+
4+
5+
:_mod-docs-content-type: PROCEDURE
6+
[id="logging-elastic-to-loki-migration_{context}"]
7+
= Migrating the Default Log Store from Elasticsearch to Loki in OpenShift
8+
9+
This guide describes how to switch the OpenShift Logging storage service from Elasticsearch to LokiStack. It focuses on log forwarding, not data migration. After following these steps, old logs will remain in Elasticsearch (accessible via Kibana), while new logs will go to LokiStack (visible in the OpenShift Console).
10+
11+
.Prerequisites
12+
13+
* Red Hat OpenShift Logging Operator (v5.5.5+)
14+
* OpenShift Elasticsearch Operator (v5.5.5+)
15+
* Red Hat Loki Operator (v5.5.5+)
16+
* Sufficient resources on target nodes to run both Elasticsearch and LokiStack (see LokiStack Deployment Sizing Table in the documentation).
17+
18+
== Installing LokiStack
19+
20+
. **Install the Loki Operator:** Follow the official guide to install the Loki Operator using the OpenShift web console.
21+
.. **Create a Secret for Loki Object Storage:** Create a secret for Loki object storage (e.g., AWS S3). Refer to the documentation for other object storage types.
22+
23+
[source,bash]
24+
----
25+
$ cat << EOF |oc create -f -
26+
apiVersion: v1
27+
kind: Secret
28+
metadata:
29+
name: logging-loki-s3
30+
namespace: openshift-logging
31+
data:
32+
access_key_id: $(echo "PUT_S3_ACCESS_KEY_ID_HERE" | base64 -w0)
33+
access_key_secret: $(echo "PUT_S3_ACCESS_KEY_SECRET_HERE" | base64 -w0)
34+
bucketnames: $(echo "s3-bucket-name" | base64 -w0)
35+
endpoint: $(echo "https://s3.eu-central-1.amazonaws.com" | base64 -w0)
36+
region: $(echo "eu-central-1" | base64 -w0)
37+
EOF
38+
----
39+
40+
41+
... **Deploy LokiStack Custom Resource (CR):**
42+
43+
[NOTE]
44+
====
45+
Change the `spec.size` if needed.
46+
====
47+
48+
[source,bash]
49+
----
50+
$ cat << EOF |oc create -f -
51+
apiVersion: loki.grafana.com/v1
52+
kind: LokiStack
53+
metadata:
54+
name: logging-loki
55+
namespace: openshift-logging
56+
spec:
57+
size: 1x.small
58+
storage:
59+
schemas:
60+
- version: v12
61+
effectiveDate: '2022-06-01'
62+
secret:
63+
name: logging-loki-s3
64+
type: s3
65+
storageClassName: gp2
66+
tenants:
67+
mode: openshift-logging
68+
EOF
69+
----
70+
71+
== Disconnecting Elasticsearch and Kibana
72+
73+
To keep Elasticsearch and Kibana running while transitioning:
74+
75+
. **Set `ClusterLogging` to Unmanaged:**
76+
77+
[source,bash]
78+
----
79+
oc -n openshift-logging patch clusterlogging/instance -p '{"spec":{"managementState": "Unmanaged"}}' --type=merge
80+
----
81+
82+
.. **Remove Owner References:** Remove `ClusterLogging` owner references from Elasticsearch and Kibana resources:
83+
84+
[source,bash]
85+
----
86+
$ oc -n openshift-logging patch elasticsearch/elasticsearch -p '{"metadata":{"ownerReferences": []}}' --type=merge
87+
----
88+
89+
[source,bash]
90+
----
91+
$ oc -n openshift-logging patch kibana/kibana -p '{"metadata":{"ownerReferences": []}}' --type=merge
92+
----
93+
94+
... **Back Up Elasticsearch and Kibana Resources:** Use `yq` to back up these resources to prevent accidental deletion: link:https://github.com/mikefarah/yq[yq utility]
95+
96+
For Elasticsearch:
97+
98+
[source,bash]
99+
----
100+
$ oc -n openshift-logging get elasticsearch elasticsearch -o yaml \
101+
| yq 'del(.metadata.resourceVersion) | del(.metadata.uid)' \
102+
| yq 'del(.metadata.generation) | del(.metadata.creationTimestamp)' \
103+
| yq 'del(.metadata.selfLink) | del(.status)' > /tmp/cr-elasticsearch.yaml
104+
----
105+
106+
For Kibana:
107+
108+
[source,bash]
109+
----
110+
$ oc -n openshift-logging get kibana kibana -o yaml \
111+
| yq 'del(.metadata.resourceVersion) | del(.metadata.uid)' \
112+
| yq 'del(.metadata.generation) | del(.metadata.creationTimestamp)' \
113+
| yq 'del(.metadata.selfLink) | del(.status)' > /tmp/cr-kibana.yaml
114+
----
115+
116+
== Switching to LokiStack
117+
118+
. Switch Log Storage to LokiStack
119+
The following manifest will apply several changes to the `ClusterLogging` resource:
120+
* Re-instate the management state to `Managed`.
121+
* Switch the `logStore` spec from `elasticsearch` to `lokistack`, restarting the collector pods to start forwarding logs to `lokistack`.
122+
* Remove the `visualization` spec, prompting the cluster-logging-operator to install the `logging-view-plugin` for observing `lokistack` logs in the OpenShift Console.
123+
* If the collection type is not `fluentd`, replace it with `vector`.
124+
125+
[source,yaml]
126+
----
127+
$ cat << EOF |oc replace -f -
128+
apiVersion: logging.openshift.io/v1
129+
kind: ClusterLogging
130+
metadata:
131+
name: instance
132+
namespace: openshift-logging
133+
spec:
134+
managementState: Managed
135+
logStore:
136+
type: lokistack
137+
lokistack:
138+
name: logging-loki
139+
collection:
140+
logs:
141+
type: fluentd
142+
fluentd: {}
143+
visualization:
144+
type: kibana
145+
kibana:
146+
replicas: 1
147+
EOF
148+
----
149+
150+
. Re-instantiate Kibana Resource
151+
152+
In the previous step, removing the `visualization` field prompted the operator to remove the `Kibana` resource. Re-instantiate the `Kibana` resource using the backup created earlier.
153+
154+
[source,bash]
155+
----
156+
$ oc -n openshift-logging apply -f /tmp/cr-kibana.yaml
157+
----
158+
159+
. Enable the Console View Plugin
160+
161+
Enable the console view plugin to view the logs integrated from the OpenShift Console (Observe > Logs).
162+
163+
[source,bash]
164+
----
165+
$ oc patch consoles.operator.openshift.io cluster --type=merge --patch '{ "spec": { "plugins": ["logging-view-plugin"] } }'
166+
----
167+
168+
== Delete the Elasticsearch Stack
169+
170+
Once the retention period for logs stored in Elasticsearch expires and no more logs are visible in Kibana, remove the old stack to release resources.
171+
172+
=== Step 1: Delete Elasticsearch and Kibana Resources
173+
174+
[source,bash]
175+
----
176+
$ oc -n openshift-logging delete kibana/kibana elasticsearch/elasticsearch
177+
----
178+
179+
===
180+
181+
Step 2: Delete the PVCs Used by Elasticsearch Instances
182+
183+
[source,bash]
184+
----
185+
$ oc delete -n openshift-logging pvc -l logging-cluster=elasticsearch
186+
----

‎observability/logging/logging-6.0/logging-poc.adoc

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
:_mod-docs-content-type: ASSEMBLY
2+
include::_attributes/common-attributes.adoc[]
3+
[id="logging-6x"]
4+
= Logging 6.0
5+
:context: logging-6x
6+
7+
toc::[]
8+
9+
include::modules/logging-6x-v-5x.adoc[leveloffset=+1]
10+
11+
//include::modules/logging-elastic-to-loki-migration.adoc[leveloffset=+1]
12+
13+
include::modules/logging-6x-about.adoc[leveloffset=+1]
14+
15+
include::modules/logging-6x-loki-architecture.adoc[leveloffset=+1]

0 commit comments

Comments
 (0)