enable the configuration of the line delimiter for the json output

[rabam]

Hi,

I tried to ship messages to a graylog gelf server over the TCP GELF input. Unfortunately it didn't work because Graylog did not accept the messages logstash was sending, both with NUL delimiters and \n delimiters. Might be a bug in graylog-inputs, not sure.
So I tried to patch the logstash codec and it worked like a charm.
You can now do something like this to use \0 as delimiter:

tcp {
    codec => json_lines {
            line_delimiter => 0
        }
    }
}

I'm not a ruby star so there might be many better ways to do it.
Can you please take a look at it and eventually merge it or provide a similar functionality.
Thank you!

Max

[lennartkoopmann]

Logstash does not support GELF via TCP AFAIK. Did you try to build TCP GELF manually using logstash JSON encoding and sending via TCP?

[rabam]

This is exaclty what I did and it works with the patch I provided in the pull request.
By taking the pull request or a similar implementation you can send messages to your graylog server like this:

filter {
  mutate {
    add_field => {
      "version" => "1.1"
      "short_message" => "%{message}"
    }
  }
  ruby {
      code => "event['timestamp'] = '%10.3f' % event['@timestamp'].to_f"
    }


}
output {
  tcp {
    host => "127.0.0.1"
    port => 12201
    codec => json_lines {
               line_delimiter => 0
             }
  }
}

Graylog needs the NUL delimiter. I thought the codec would be a goos place to place it.

[lennartkoopmann]

Interesting. You basically built GELF TCP support using built-in logstash functionality. Smart!

[rabam]

Is it possible to integrate the change? I signed the CLA.
Thanks
Max

[rabam]

Is it possible to integrate the change? I signed the CLA.
Thanks
Max

[jordansissel]

I like the idea! I'm not sure about the implementation. Can GELF over TCP be compressed? If so, then this patch will not help us move towards compressed-gelf-over-tcp behavior. Though, if gelf-tcp doesn't support compression, then maybe this is ok to modify in the json_lines codec.

Additionally, having the line_delimiter setting be a number feels a bit weird. I understand it's hard to type a null character, but this also prevents folks from using, say, \r\n as the delimiter (is that desirable?).

I'm not requiring any of the above changes, but want to at least discuss them a bit before we move forward if that's ok :)

[jordansissel]

Once we get through that small discussion, It'd be oh so lovely to have some tests to help us keep this behavior functioning in the future :)

[elasticsearch-release]

Jenkins standing by to test this. If you aren't a maintainer, you can ignore this comment. Someone with commit access, please review this and clear it for Jenkins to run; then say 'jenkins, test it'.

[Hermain]

I want to use logstash to receive gelf logs from docker over TCP. Having support for NUL delimiter in json and jsonLines codec would be nice for that

[colinsurprenant]

Very late response I know - I am ok with the addition of a delimiter option but it should be the same as with the line codec see https://github.com/logstash-plugins/logstash-codec-line/blob/802904335a6c1a88dfb968fa42afe51f05e322ae/lib/logstash/codecs/line.rb#L25-L26 and a null delimiter could be specified as "\0".

@rabam are you still around and would you like to make that change?