Skip to content

Clarifying current limitation with forwarding to Logstash #3228

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Dec 15, 2016
Merged

Clarifying current limitation with forwarding to Logstash #3228

merged 1 commit into from
Dec 15, 2016

Conversation

ewolinetz
Copy link

ahardin-rh
ahardin-rh reviewed Nov 15, 2016
View reviewed changes
install_config/aggregate_logging.adoc
@@ -877,11 +877,16 @@ interpret messages from
link:https://www.elastic.co/guide/en/logstash/current/plugins-codecs-fluent.html[Fluentd]
and can be configured to use
link:https://www.elastic.co/guide/en/logstash/current/plugins-inputs-tcp.html#ssl_enable[SSL].
Currently however, the `secure_forward` plugin is required to provide a `shared_key`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently, however, the
s/plugin/plug-in

ahardin-rh
ahardin-rh reviewed Nov 15, 2016
View reviewed changes
install_config/aggregate_logging.adoc
@@ -877,11 +877,16 @@ interpret messages from
link:https://www.elastic.co/guide/en/logstash/current/plugins-codecs-fluent.html[Fluentd]
and can be configured to use
link:https://www.elastic.co/guide/en/logstash/current/plugins-inputs-tcp.html#ssl_enable[SSL].
Currently however, the `secure_forward` plugin is required to provide a `shared_key`
which Logstash is unable to interpret and provide. In order to forward logs from
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shared_key, which

ahardin-rh
ahardin-rh reviewed Nov 15, 2016
View reviewed changes
install_config/aggregate_logging.adoc
@@ -877,11 +877,16 @@ interpret messages from
link:https://www.elastic.co/guide/en/logstash/current/plugins-codecs-fluent.html[Fluentd]
and can be configured to use
link:https://www.elastic.co/guide/en/logstash/current/plugins-inputs-tcp.html#ssl_enable[SSL].
Currently however, the `secure_forward` plugin is required to provide a `shared_key`
which Logstash is unable to interpret and provide. In order to forward logs from
Fluentd to Logstash the `forward` Fluentd plugin will need to be used instead.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to Logstash,
s/plugin/plug-in
s/will need to be/must be

@ewolinetz ewolinetz force-pushed the update_secure_forward_docs branch from 8adfec4 to 3b891fc Compare November 15, 2016 20:56
ahardin-rh
ahardin-rh reviewed Nov 15, 2016
View reviewed changes
install_config/aggregate_logging.adoc

Further explanation of how to set up the `fluent-plugin-secure-forward` plug-in
can be link:https://github.com/tagomoris/fluent-plugin-secure-forward[found
here].

Further explanation of how to set up the `forward` plug-in can be link:http://docs.fluentd.org/articles/out_forward[found here].
Copy link
Contributor

@ahardin-rh ahardin-rh Nov 15, 2016 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider rewording to:

link:http://docs.fluentd.org/articles/out_forward[Learn more about the `forward` plug-in.]

or something like that 🤔

@ahardin-rh
Copy link
Contributor

Just a heads-up that we will have to coordinate with #3211 before merging.

cc @danmacpherson

@ewolinetz ewolinetz force-pushed the update_secure_forward_docs branch from 3b891fc to e691909 Compare November 15, 2016 21:23
@ewolinetz
Copy link
Author

Thanks @ahardin-rh, addressed comments.

@elyscape
Copy link

Copying my comment from openshift/origin-aggregated-logging#279:

@ewolinetz I did some research and it turns out that the out_forward plugin won't work for connecting to Logstash either. Logstash's tcp input assumes that you'll open a socket and then push a bunch of messages separated by newlines. Fluentd's out_forward plugin opens a socket and then sends MessagePack arrays with no delimiters between them. Additionally, due to the way Fluentd is configured here, there are often newlines at the end of the log message, which happens midway through the MessagePack object. This causes Logstash to chop the messages in the wrong location (or just assemble huge megamessages) which then fail to parse in the fluent codec. Furthermore, there is no way to tell Fluentd to put a delimiter between messages and no way to tell Logstash not to look for delimiters. As such, there does not appear to be any way to make Fluentd talk to Logstash without some sort of middle-man (e.g. Kafka, RabbitMQ).

@ewolinetz
Copy link
Author

@elyscape thanks for the confirmation. Since it seems that Fluentd does not allow us to forward to Logstash, despite Logstash having a Fluentd codec, i'll update the docs to remove mention of this.

@ewolinetz ewolinetz force-pushed the update_secure_forward_docs branch from e691909 to 32e6011 Compare November 16, 2016 21:22
@ahardin-rh ahardin-rh added this to the Next Release milestone Dec 15, 2016
@ahardin-rh
Copy link
Contributor

Sorry for the delay. Thank you!

@ahardin-rh ahardin-rh merged commit 180a163 into openshift:master Dec 15, 2016
@ahardin-rh
Copy link
Contributor

No revision history needed.

@bfallonf bfallonf modified the milestones: Next Release, Staging, Published - 20/12/2016 Dec 20, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ahardin-rh ahardin-rh ahardin-rh left review comments

Assignees
No one assigned
Labels
branch/enterprise-3.3
Projects
None yet
Milestone
Published - 12/20/16
Development

Successfully merging this pull request may close these issues.
4 participants
@ewolinetz @ahardin-rh @elyscape @bfallonf