Skip to content

Commit 2e24f09

Browse files
lcawllcawl
authored
[DOCS] Describe setup for monitoring logs (#42655)
1 parent f45cf90 commit 2e24f09

File tree

2 files changed

+190
-0
lines changed

2 files changed

+190
-0
lines changed

‎docs/reference/monitoring/configuring-filebeat.asciidoc

+187
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,187 @@
1+
[role="xpack"]
2+
[testenv="basic"]
3+
[[configuring-filebeat]]
4+
=== Collecting {es} log data with {filebeat}
5+
6+
[subs="attributes"]
7+
++++
8+
<titleabbrev>Collecting log data with {filebeat}</titleabbrev>
9+
++++
10+
11+
You can use {filebeat} to monitor the {es} log files, collect log events, and
12+
ship them to the monitoring cluster. Your recent logs are visible on the
13+
*Monitoring* page in {kib}.
14+
15+
//NOTE: The tagged regions are re-used in the Stack Overview.
16+
17+
. Verify that {es} is running and that the monitoring cluster is ready to
18+
receive data from {filebeat}.
19+
+
20+
--
21+
TIP: In production environments, we strongly recommend using a separate cluster
22+
(referred to as the _monitoring cluster_) to store the data. Using a separate
23+
monitoring cluster prevents production cluster outages from impacting your
24+
ability to access your monitoring data. It also prevents monitoring activities
25+
from impacting the performance of your production cluster. See
26+
{stack-ov}/monitoring-production.html[Monitoring in a production environment].
27+
28+
--
29+
30+
. Enable the collection of monitoring data on your cluster.
31+
+
32+
--
33+
include::configuring-metricbeat.asciidoc[tag=enable-collection]
34+
35+
For more information, see <<monitoring-settings>> and <<cluster-update-settings>>.
36+
--
37+
38+
. Identify which logs you want to monitor.
39+
+
40+
--
41+
The {filebeat} {es} module can handle
42+
{stack-ov}/audit-log-output.html[audit logs],
43+
{ref}/logging.html#deprecation-logging[deprecation logs],
44+
{ref}/gc-logging.html[gc logs], {ref}/logging.html[server logs], and
45+
{ref}/index-modules-slowlog.html[slow logs].
46+
For more information about the location of your {es} logs, see the
47+
{ref}/path-settings.html[path.logs] setting.
48+
49+
IMPORTANT: If there are both structured (`*.json`) and unstructured (plain text)
50+
versions of the logs, you must use the structured logs. Otherwise, they might
51+
not appear in the appropriate context in {kib}.
52+
53+
--
54+
55+
. {filebeat-ref}/filebeat-installation.html[Install {filebeat}] on the {es}
56+
nodes that contain logs that you want to monitor.
57+
58+
. Identify where to send the log data.
59+
+
60+
--
61+
// tag::output-elasticsearch[]
62+
For example, specify {es} output information for your monitoring cluster in
63+
the {filebeat} configuration file (`filebeat.yml`):
64+
65+
[source,yaml]
66+
----------------------------------
67+
output.elasticsearch:
68+
# Array of hosts to connect to.
69+
hosts: ["http://es-mon-1:9200", "http://es-mon2:9200"] <1>
70+
71+
# Optional protocol and basic auth credentials.
72+
#protocol: "https"
73+
#username: "elastic"
74+
#password: "changeme"
75+
----------------------------------
76+
<1> In this example, the data is stored on a monitoring cluster with nodes
77+
`es-mon-1` and `es-mon-2`.
78+
79+
If you configured the monitoring cluster to use encrypted communications, you
80+
must access it via HTTPS. For example, use a `hosts` setting like
81+
`https://es-mon-1:9200`.
82+
83+
IMPORTANT: The {es} {monitor-features} use ingest pipelines, therefore the
84+
cluster that stores the monitoring data must have at least one
85+
<<ingest,ingest node>>.
86+
87+
If {es} {security-features} are enabled on the monitoring cluster, you must
88+
provide a valid user ID and password so that {filebeat} can send metrics
89+
successfully.
90+
91+
For more information about these configuration options, see
92+
{filebeat-ref}/elasticsearch-output.html[Configure the {es} output].
93+
// end::output-elasticsearch[]
94+
--
95+
96+
. Optional: Identify where to visualize the data.
97+
+
98+
--
99+
// tag::setup-kibana[]
100+
{filebeat} provides example {kib} dashboards, visualizations and searches. To
101+
load the dashboards into the appropriate {kib} instance, specify the
102+
`setup.kibana` information in the {filebeat} configuration file
103+
(`filebeat.yml`) on each node:
104+
105+
[source,yaml]
106+
----------------------------------
107+
setup.kibana:
108+
host: "localhost:5601"
109+
#username: "my_kibana_user"
110+
#password: "YOUR_PASSWORD"
111+
----------------------------------
112+
113+
TIP: In production environments, we strongly recommend using a dedicated {kib}
114+
instance for your monitoring cluster.
115+
116+
If {security-features} are enabled, you must provide a valid user ID and
117+
password so that {filebeat} can connect to {kib}:
118+
119+
.. Create a user on the monitoring cluster that has the
120+
{stack-ov}/built-in-roles.html[`kibana_user` built-in role] or equivalent
121+
privileges.
122+
123+
.. Add the `username` and `password` settings to the {es} output information in
124+
the {filebeat} configuration file. The example shows a hard-coded password, but
125+
you should store sensitive values in the
126+
{filebeat-ref}/keystore.html[secrets keystore].
127+
128+
See {filebeat-ref}/setup-kibana-endpoint.html[Configure the {kib} endpoint].
129+
130+
// end::setup-kibana[]
131+
--
132+
133+
. Enable the {es} module and set up the initial {filebeat} environment on each
134+
node.
135+
+
136+
--
137+
// tag::enable-es-module[]
138+
For example:
139+
140+
["source","sh",subs="attributes,callouts"]
141+
----------------------------------------------------------------------
142+
filebeat modules enable elasticsearch
143+
filebeat setup -e
144+
----------------------------------------------------------------------
145+
146+
For more information, see
147+
{filebeat-ref}/filebeat-module-elasticsearch.html[{es} module].
148+
149+
// end::enable-es-module[]
150+
--
151+
152+
. Configure the {es} module in {filebeat} on each node.
153+
+
154+
--
155+
// tag::configure-es-module[]
156+
If the logs that you want to monitor aren't in the default location, set the
157+
appropriate path variables in the `modules.d/elasticsearch.yml` file. See
158+
{filebeat-ref}/filebeat-module-elasticsearch.html#configuring-elasticsearch-module[Configure the {es} module].
159+
160+
IMPORTANT: If there are JSON logs, configure the `var.paths` settings to point
161+
to them instead of the plain text logs.
162+
163+
// end::configure-es-module[]
164+
--
165+
166+
. {filebeat-ref}/filebeat-starting.html[Start {filebeat}] on each node.
167+
+
168+
--
169+
NOTE: Depending on how you’ve installed {filebeat}, you might see errors related
170+
to file ownership or permissions when you try to run {filebeat} modules. See
171+
{beats-ref}/config-file-permissions.html[Config file ownership and permissions].
172+
173+
--
174+
175+
. Check whether the appropriate indices exist on the monitoring cluster.
176+
+
177+
--
178+
For example, use the {ref}/cat-indices.html[cat indices] command to verify
179+
that there are new `filebeat-*` indices.
180+
181+
TIP: If you want to use the *Monitoring* UI in {kib}, there must also be
182+
`.monitoring-*` indices. Those indices are generated when you collect metrics
183+
about {stack} products. For example, see <<configuring-metricbeat>>.
184+
185+
--
186+
187+
. {kibana-ref}/monitoring-data.html[View the monitoring data in {kib}].

‎docs/reference/monitoring/configuring-monitoring.asciidoc

+3
Original file line numberDiff line numberDiff line change
@@ -12,9 +12,12 @@ methods to collect metrics about {es}:
1212
* <<collecting-monitoring-data>>
1313
* <<configuring-metricbeat>>
1414

15+
You can also <<configuring-filebeat,use {filebeat} to collect {es} logs>>.
16+
1517
To learn about monitoring in general, see
1618
{stack-ov}/xpack-monitoring.html[Monitoring the {stack}].
1719

1820
include::collecting-monitoring-data.asciidoc[]
1921
include::configuring-metricbeat.asciidoc[]
22+
include::configuring-filebeat.asciidoc[]
2023
include::indices.asciidoc[]

0 commit comments

Comments
 (0)