Also facing this issue when using logstash-to-logstash integration which internally uses this codec. I think #43 completely broke logstash-input-http. The pipeline basically stops within a few minutes after startup. I'm not entirely sure how the current default 500MB buffer gets filled so fast, the forwarded data is nowhere near that size (at least it's much smaller at rest).

Maybe the integration plugin is using this codec wrong? (If that's even a possible thing.)

Looking into this, I'll post here soon with findings.

I think I know what's wrong. The logstash integration's "logstash input" is using the json_lines codec instead of the normal json codec.
On the "client side" the logstash output will take the batch of data and transform it into a JSON array with the individual json objects (events) inside, so the receiving end (logstash input) should just parse the object as a whole, it shouldn't be using the buffered json_lines.

This is coupled with a separate bug in the BufferedTokenizer when the tokenizer doesn't properly reset its state when data is only extracted from it through flushing:

jruby-9.4.9.0 :001 > data = [{"message" => "a"*20 }]
 => [{"message"=>"aaaaaaaaaaaaaaaaaaaa"}] 
jruby-9.4.9.0 :002 > data_json = data.to_json
 => "[{\"message\":\"aaaaaaaaaaaaaaaaaaaa\"}]" 
jruby-9.4.9.0 :003 > codec = LogStash::Plugin.lookup("codec", "json_lines").new("delimiter" => "\n", "decode_size_limit_bytes" => 100)
 => <LogStash::Codecs::JSONLines [...] >
jruby-9.4.9.0 :004 > codec.decode(data_json) {|event| puts "decode got 1 event" }; codec.flush {|event| puts "flush got 1 event" }
flush got 1 event
 => [#<LogStash::Event:0x450d0da9>] 
jruby-9.4.9.0 :005 > codec.decode(data_json) {|event| puts "decode got 1 event" }; codec.flush {|event| puts "flush got 1 event" }
flush got 1 event
 => [#<LogStash::Event:0x6da63893>] 
jruby-9.4.9.0 :006 > codec.decode(data_json) {|event| puts "decode got 1 event" }; codec.flush {|event| puts "flush got 1 event" }
org.logstash.common.BufferedTokenizerExt.extract(BufferedTokenizerExt.java:85): input buffer full (Java::JavaLang::IllegalStateException)

Because the logstash output sends json payloads without a delimiter (they're just json arrays), then data only comes out of the codec through flushing, triggering this bug.
The bug is cased by the internal state variable inputSize not being reset when it accepts data that doesn't immediately produce any entity (given its not "\n" delimited data).
So inputSize keeps increasing until it raises an exception stating the buffer is full when in fact it's not, it's empty (data is being flushed by .flush calls from the logstash input plugin).

Fixes are therefore needed in:

1. logstash integration switch to json codec instead of json_lines in the "logstash input" plugin
2. change the BufferedTokenizer .flush method to reset the inputSize.

[edit] after chatting with @yaauie it makes more sense to fix the logstash output to properly emit ndjson like it was supposed to in the first place.

Fixes in progress:

1. output: emit ndjson payloads logstash-integration-logstash#25
2. ensure inputSize state value is reset during buftok.flush elastic/logstash#16760

@andsel Can this be closed?

No can't be closed because the feature is still need to be implemented. As it's at version 3.2.2 it uses a big limit (512Mb) and when not configured logs a deprecation message.
This issue is to:

• remove the limit
• logs a deprecation if the user doesn't define a value for decode_size_limit_bytes
• if the decode_size_limit_bytes is set and an "buffer full" condition, it has to recover from the error, create an event and tag it, resume from the next token at

  logstash-codec-json_lines/lib/logstash/codecs/json_lines.rb

  Lines 69 to 71 in f4e4e00

  	@buffer.extract(data).each do |line|
  	parse_json(@converter.convert(line), &block)
  	end