Skip to content

Commit 650434d

Browse files
libanderlibander
committed
Logging 6x - POC WIP
1 parent db7835c commit 650434d

File tree

6 files changed

+282
-0
lines changed

6 files changed

+282
-0
lines changed

‎_topic_maps/_topic_map.yml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2709,6 +2709,11 @@ Topics:
27092709
File: logging-5-8-release-notes
27102710
- Name: Logging 5.7
27112711
File: logging-5-7-release-notes
2712+
- Name: Logging 6.0
2713+
Dir: logging-6.0
2714+
Topics:
2715+
- Name: Introducing Logging 6.0
2716+
File: logging-poc
27122717
- Name: Support
27132718
File: cluster-logging-support
27142719
- Name: Troubleshooting logging

‎modules/logging-6x-about.adoc

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
// Module included in the following assemblies:
2+
//
3+
4+
5+
:_mod-docs-content-type: CONCEPT
6+
[id="logging-6x-about_{context}"]
7+
8+
9+
= What is OpenShift Logging?
10+
11+
OpenShift Logging provides a comprehensive log management solution integrated into your OpenShift clusters. It utilizes:
12+
13+
* **Loki:** A horizontally scalable, highly available log aggregation system optimized for efficient storage and retrieval of log data in dynamic OpenShift environments.
14+
* **Vector:** A lightweight, high-performance log forwarding agent that connects your application and infrastructure logs to Loki, offering a flexible and customizable way to collect, process, and route log data.
15+
16+
== How OpenShift Logging Works
17+
18+
1. **Log Collection:** Vector agents, deployed on each node in your cluster, collect logs from various sources, including containers, system components, and applications.
19+
2. **Log Processing (Optional):** Vector can filter, enrich, and transform log data before forwarding it to Loki.
20+
3. **Log Aggregation:** Loki receives the processed log data from Vector and stores it efficiently for later retrieval.
21+
4. **Log Querying & Analysis:** Use Loki's powerful query language to search, filter, and analyze logs, gaining valuable insights into your OpenShift environment.
22+
23+
== Visualizing Logs in OpenShift
24+
25+
OpenShift Logging provides multiple ways to visualize and explore your log data:
26+
27+
* **OpenShift Web Console:** The integrated Logs page offers a user-friendly interface for searching, filtering, and viewing logs directly within the OpenShift console.
28+
* **Grafana:** OpenShift Logging integrates seamlessly with Grafana, enabling you to create rich, customizable dashboards to visualize and correlate log data with metrics and other observability data.

‎modules/logging-6x-loki-architecture.adoc

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
// Module included in the following assemblies:
2+
//
3+
4+
5+
:_mod-docs-content-type: CONCEPT
6+
[id="logging-6x-loki-architecture_{context}"]
7+
= Loki Architecture: Understanding the Components
8+
9+
Loki's architecture is comprised of several key components, each playing a specific role in log processing and storage:
10+
11+
* **Distributor:** Receives incoming logs from Vector agents and distributes them across multiple Ingesters.
12+
* **Ingester:** Responsible for building chunks of log data and creating an index for efficient retrieval.
13+
* **Querier:** Processes incoming log queries and fetches relevant log data from the storage backend.
14+
* **Query Frontend:** Manages and coordinates incoming queries, distributes the workload across Queriers, and returns the results to the user.
15+
* **Storage Backend:** Stores the log chunks and indexes, often using object storage services like S3 or GCS.

‎modules/logging-6x-v-5x.adoc

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
// Module included in the following assemblies:
2+
//
3+
4+
5+
:_mod-docs-content-type: PROCEDURE
6+
[id="logging-6x-v-5x_{context}"]
7+
= OpenShift Logging 6.0: Key Differences from Previous Versions (5.0-5.9)
8+
9+
OpenShift Logging 6.0 introduces several significant changes compared to previous versions, streamlining the logging architecture and enhancing the overall logging experience.
10+
11+
== Simplified Architecture
12+
13+
* **Removal of Elasticsearch and Fluentd:** OpenShift Logging 6.0 eliminates Elasticsearch and Fluentd as default components, streamlining the architecture for improved performance and reduced complexity.
14+
* **Focus on Loki and Vector:** The logging stack now centers around Loki for log aggregation and Vector for log collection and forwarding, providing a more unified and efficient solution.
15+
16+
== Enhanced Scalability and Performance
17+
18+
* **Horizontal Scalability:** Loki's architecture allows for seamless horizontal scaling, ensuring that the logging system can handle the growing demands of large-scale OpenShift deployments.
19+
* **Improved Query Performance:** Loki's indexing and query mechanisms are optimized for speed, enabling faster log retrieval and analysis.
20+
21+
== Enhanced User Experience
22+
23+
* **Streamlined Configuration:** The removal of Elasticsearch and Fluentd simplifies the configuration process, making it easier to set up and manage your logging infrastructure.
24+
* **Improved Integration:** Loki and Vector offer tighter integration with OpenShift, providing a more seamless and native logging experience.
25+
26+
== Additional Enhancements
27+
28+
* **Expanded Log Sources:** Vector's flexibility allows you to collect logs from an even wider range of sources, providing greater visibility into your OpenShift environment.
29+
* **Enhanced Log Processing:** Vector's processing capabilities have been expanded, enabling you to perform more complex log transformations and filtering.
30+
31+
== Transitioning to OpenShift Logging 6.0
32+
33+
If you're upgrading from a previous version of OpenShift Logging, the transition to 6.0 will involve migrating your existing log data and reconfiguring your logging pipeline to utilize Loki and Vector.

‎modules/logging-elastic-to-loki-migration.adoc

Lines changed: 186 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,186 @@
1+
// Module included in the following assemblies:
2+
//
3+
4+
5+
:_mod-docs-content-type: PROCEDURE
6+
[id="logging-elastic-to-loki-migration_{context}"]
7+
= Migrating the Default Log Store from Elasticsearch to Loki in OpenShift
8+
9+
This guide describes how to switch the OpenShift Logging storage service from Elasticsearch to LokiStack. It focuses on log forwarding, not data migration. After following these steps, old logs will remain in Elasticsearch (accessible via Kibana), while new logs will go to LokiStack (visible in the OpenShift Console).
10+
11+
.Prerequisites
12+
13+
* Red Hat OpenShift Logging Operator (v5.5.5+)
14+
* OpenShift Elasticsearch Operator (v5.5.5+)
15+
* Red Hat Loki Operator (v5.5.5+)
16+
* Sufficient resources on target nodes to run both Elasticsearch and LokiStack (see LokiStack Deployment Sizing Table in the documentation).
17+
18+
== Installing LokiStack
19+
20+
. **Install the Loki Operator:** Follow the official guide to install the Loki Operator using the OpenShift web console.
21+
.. **Create a Secret for Loki Object Storage:** Create a secret for Loki object storage (e.g., AWS S3). Refer to the documentation for other object storage types.
22+
23+
[source,bash]
24+
----
25+
$ cat << EOF |oc create -f -
26+
apiVersion: v1
27+
kind: Secret
28+
metadata:
29+
name: logging-loki-s3
30+
namespace: openshift-logging
31+
data:
32+
access_key_id: $(echo "PUT_S3_ACCESS_KEY_ID_HERE" | base64 -w0)
33+
access_key_secret: $(echo "PUT_S3_ACCESS_KEY_SECRET_HERE" | base64 -w0)
34+
bucketnames: $(echo "s3-bucket-name" | base64 -w0)
35+
endpoint: $(echo "https://s3.eu-central-1.amazonaws.com" | base64 -w0)
36+
region: $(echo "eu-central-1" | base64 -w0)
37+
EOF
38+
----
39+
40+
41+
... **Deploy LokiStack Custom Resource (CR):**
42+
43+
[NOTE]
44+
====
45+
Change the `spec.size` if needed.
46+
====
47+
48+
[source,bash]
49+
----
50+
$ cat << EOF |oc create -f -
51+
apiVersion: loki.grafana.com/v1
52+
kind: LokiStack
53+
metadata:
54+
name: logging-loki
55+
namespace: openshift-logging
56+
spec:
57+
size: 1x.small
58+
storage:
59+
schemas:
60+
- version: v12
61+
effectiveDate: '2022-06-01'
62+
secret:
63+
name: logging-loki-s3
64+
type: s3
65+
storageClassName: gp2
66+
tenants:
67+
mode: openshift-logging
68+
EOF
69+
----
70+
71+
== Disconnecting Elasticsearch and Kibana
72+
73+
To keep Elasticsearch and Kibana running while transitioning:
74+
75+
. **Set `ClusterLogging` to Unmanaged:**
76+
77+
[source,bash]
78+
----
79+
oc -n openshift-logging patch clusterlogging/instance -p '{"spec":{"managementState": "Unmanaged"}}' --type=merge
80+
----
81+
82+
.. **Remove Owner References:** Remove `ClusterLogging` owner references from Elasticsearch and Kibana resources:
83+
84+
[source,bash]
85+
----
86+
$ oc -n openshift-logging patch elasticsearch/elasticsearch -p '{"metadata":{"ownerReferences": []}}' --type=merge
87+
----
88+
89+
[source,bash]
90+
----
91+
$ oc -n openshift-logging patch kibana/kibana -p '{"metadata":{"ownerReferences": []}}' --type=merge
92+
----
93+
94+
... **Back Up Elasticsearch and Kibana Resources:** Use `yq` to back up these resources to prevent accidental deletion: link:https://github.com/mikefarah/yq[yq utility]
95+
96+
For Elasticsearch:
97+
98+
[source,bash]
99+
----
100+
$ oc -n openshift-logging get elasticsearch elasticsearch -o yaml \
101+
| yq 'del(.metadata.resourceVersion) | del(.metadata.uid)' \
102+
| yq 'del(.metadata.generation) | del(.metadata.creationTimestamp)' \
103+
| yq 'del(.metadata.selfLink) | del(.status)' > /tmp/cr-elasticsearch.yaml
104+
----
105+
106+
For Kibana:
107+
108+
[source,bash]
109+
----
110+
$ oc -n openshift-logging get kibana kibana -o yaml \
111+
| yq 'del(.metadata.resourceVersion) | del(.metadata.uid)' \
112+
| yq 'del(.metadata.generation) | del(.metadata.creationTimestamp)' \
113+
| yq 'del(.metadata.selfLink) | del(.status)' > /tmp/cr-kibana.yaml
114+
----
115+
116+
== Switching to LokiStack
117+
118+
. Switch Log Storage to LokiStack
119+
The following manifest will apply several changes to the `ClusterLogging` resource:
120+
* Re-instate the management state to `Managed`.
121+
* Switch the `logStore` spec from `elasticsearch` to `lokistack`, restarting the collector pods to start forwarding logs to `lokistack`.
122+
* Remove the `visualization` spec, prompting the cluster-logging-operator to install the `logging-view-plugin` for observing `lokistack` logs in the OpenShift Console.
123+
* If the collection type is not `fluentd`, replace it with `vector`.
124+
125+
[source,yaml]
126+
----
127+
$ cat << EOF |oc replace -f -
128+
apiVersion: logging.openshift.io/v1
129+
kind: ClusterLogging
130+
metadata:
131+
name: instance
132+
namespace: openshift-logging
133+
spec:
134+
managementState: Managed
135+
logStore:
136+
type: lokistack
137+
lokistack:
138+
name: logging-loki
139+
collection:
140+
logs:
141+
type: fluentd
142+
fluentd: {}
143+
visualization:
144+
type: kibana
145+
kibana:
146+
replicas: 1
147+
EOF
148+
----
149+
150+
. Re-instantiate Kibana Resource
151+
152+
In the previous step, removing the `visualization` field prompted the operator to remove the `Kibana` resource. Re-instantiate the `Kibana` resource using the backup created earlier.
153+
154+
[source,bash]
155+
----
156+
$ oc -n openshift-logging apply -f /tmp/cr-kibana.yaml
157+
----
158+
159+
. Enable the Console View Plugin
160+
161+
Enable the console view plugin to view the logs integrated from the OpenShift Console (Observe > Logs).
162+
163+
[source,bash]
164+
----
165+
$ oc patch consoles.operator.openshift.io cluster --type=merge --patch '{ "spec": { "plugins": ["logging-view-plugin"] } }'
166+
----
167+
168+
== Delete the Elasticsearch Stack
169+
170+
Once the retention period for logs stored in Elasticsearch expires and no more logs are visible in Kibana, remove the old stack to release resources.
171+
172+
=== Step 1: Delete Elasticsearch and Kibana Resources
173+
174+
[source,bash]
175+
----
176+
$ oc -n openshift-logging delete kibana/kibana elasticsearch/elasticsearch
177+
----
178+
179+
===
180+
181+
Step 2: Delete the PVCs Used by Elasticsearch Instances
182+
183+
[source,bash]
184+
----
185+
$ oc delete -n openshift-logging pvc -l logging-cluster=elasticsearch
186+
----

‎observability/logging/logging-6.0/logging-poc.adoc

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
:_mod-docs-content-type: ASSEMBLY
2+
include::_attributes/common-attributes.adoc[]
3+
[id="logging-6x"]
4+
= Logging 6.0
5+
:context: logging-6x
6+
7+
toc::[]
8+
9+
include::modules/logging-6x-v-5x.adoc[leveloffset=+1]
10+
11+
include::modules/logging-elastic-to-loki-migration.adoc[leveloffset=+1]
12+
13+
include::modules/logging-6x-about.adoc[leveloffset=+1]
14+
15+
include::modules/logging-6x-loki-architecture.adoc[leveloffset=+1]

0 commit comments

Comments
 (0)