[DOCS] Describe setup for monitoring logs #42655
Conversation
lcawl
added
>docs
|
Pinging @elastic/es-core-features |
lcawl
changed the title
| For more information, see <<monitoring-settings>> and <<cluster-update-settings>>. | ||
| -- | ||
|
|
||
| . Optional: Collect metrics about {es}. See <<configuring-metricbeat>> or |
There was a problem hiding this comment.
Is this an attempt to convert users over to using Metricbeat (which I'm totally for btw)? I feel it's a bit of place considering this is a guide around filebeat, and not metricbeat. cc @elastic/stack-monitoring for thoughts
There was a problem hiding this comment.
I do like the "cross sell" here but I also found it a bit out-of-place as step 3 while setting up logs collection. How about making this a note or tip at the bottom of the page?
There was a problem hiding this comment.
I had it in my brain somehow that the Monitoring UI wouldn't show up on the monitoring cluster until the .monitoring* indices existed so that's why I tested it that way. And it seemed to me that the Filebeat output went to filebeat* indices. i.e. I needed to do something to get other metrics onto the monitoring cluster before any of these log-related pieces will appear. I'll retest without sending any other monitoring data to the monitoring cluster to see if I was wrong about all that.
There was a problem hiding this comment.
You're right. There are steps required to see logs within the Monitoring UI that are separate from these filebeat steps, but maybe we can just link off to some enabling monitoring doc, or leave this as-is, but add a little more explanation of why it's necessary?
There was a problem hiding this comment.
Thanks for the feedback. I've moved this information to the second-last step.
| . Optional: Collect metrics about {es}. See <<configuring-metricbeat>> or | ||
| <<collecting-monitoring-data>>. | ||
|
|
||
| . Identify which logs you want to monitor. |
There was a problem hiding this comment.
Should we include any guidance on whether to point FIlebeat at unstructured versus structured ES logs?
There was a problem hiding this comment.
To be more clear, this is about whether or not we should explicitly tell users to use the *.json logs versus the free-text logs. However, I am not sure if there is an advantage from the Filebeat perspective. @ycombinator do you know?
There was a problem hiding this comment.
Good point, @cachedout.
We should require users to use *.json logs (as opposed to plaintext logs). The JSON logs are the only ones guaranteed to contain the cluster_uuid field, without which the log lines won't be shown in the correct context in the Stack Monitoring UI.
There was a problem hiding this comment.
Thanks for the clarification! I've added tips to the steps related to choosing the logs and configuring the filebeat module.
|
Also, there is currently a limit of |
| ++++ | ||
|
|
||
| You can use {filebeat} to monitor the {es} log files, collect log events, and | ||
| ship them to the monitoring cluster. In 7.2 and later, your recent logs are |
There was a problem hiding this comment.
Is this availability date information is really necessary? This page in this form will only exist in 7.2 and later documentation, right? So, it might be redundant to add this information.
There was a problem hiding this comment.
I think we've included that in monitoring instructions in the past because your monitoring cluster can be at a different version.release than your production cluster. But I think you're right that it's not really necessary.
| must access it via HTTPS. For example, use a `hosts` setting like | ||
| `https://es-mon-1:9200`. | ||
|
|
||
| IMPORTANT: The {es} {monitor-features} use ingest pipelines, therefore the |
There was a problem hiding this comment.
It might be worth to consider to add a link to ingest pipelines to have an explanation at hand.
https://www.elastic.co/guide/en/elasticsearch/client/net-api/6.x/pipelines.html
There was a problem hiding this comment.
Thanks, I added a link to the Elasticsearch ingest node page.
|
@elasticmachine update branch |
@chrisronline Unless I'm mistaken, this seems like a configuration method that is broader than just the Filebeat scenario. Therefore, I've created elastic/stack-docs#378 to follow up on those instructions. |
I think this must be a Kibana setting, so I've opened a PR to add it there: elastic/kibana#39139 |
Related to elastic/stack-docs#348
This PR adds documentation related to configuring Filebeat to collect and send log data to the monitoring cluster.