Default buffer type to 'heap' for 9.0

[andsel]

Release notes

Switch the default value of pipeline.buffer.type to use the heap memory instead of direct one.

What does this PR do?

Change the default value of the setting pipeline.buffer.type from direct to heap and update consequently the documentation.

Why is it important/What is the impact to the user?

It's part of the work to make more explicit the usage of memory used by Netty based plugins. Using heap instead of direct for Netty buffers, make easier the debug of memory issues at the cost of bigger heap size.

Checklist

• My code follows the style guidelines of this project
• [ ] I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files (and/or docker env variables)
• [ ] I have added tests that prove my fix is effective or that my feature works

Author's Checklist

• [ ]

How to test this PR locally

Related issues

• Relates Set pipeline.buffer.type to heap by default #16353

Use cases

As a developer of Netty based plugin I want to have better insight in memory usage patterns.

[jsvd]

It is important to note that the Java heap sizing requirements will be impacted by this change since
allocations that previously resided on the direct memory will use heap instead. 
Performance-wise there shouldn't be a noticeable impact, since while direct memory IO is faster, Logstash Event objects produced by these plugins end up being allocated on the Java Heap, incurring the cost of copying from direct memory to heap memory regardless of the setting.

These must be changed as well, same with:

* When you set `pipeline.buffer.type` to `heap`, consider incrementing the Java heap by the 
amount of memory that had been reserved for direct space. 

[jsvd]

Please review the rest of the docs on the config-details asciidoc as it assumes direct memory is the default, that heap will be come the default, and that the user will be changing it to heap (which doesn't make sense for someone starting fresh with 9.x)

[yaauie]

Don't we use pooled buffers anyway? I'm under the impression that allocations of buffers may be faster in direct, but that our pool usage is a primary driver for our buffer use being performant. We also know that large buffer allocations can succeed in heap where they would fail in direct because the GC will rearrange the existing object space to ensure that it can allocate while the direct allocation will simply not have an appropriately-sized continuous chunk. I would rather a small performance cost in this case than a crashing OOM :)

Suggested change

	Performance-wise there shouldn't be a noticeable impact in switching to `direct`, since while direct memory IO is faster,
	Logstash Event objects produced by these plugins end up being allocated on the Java Heap,
	incurring the cost of copying from direct memory to heap memory regardless of the setting.
	Performance-wise there shouldn't be a noticeable impact in switching to `direct`.
	While allocating direct memory for an individual buffer is faster, these plugins use buffer pools to reduce allocations, and heap-managed allocations are significantly safer.
	Logstash Event objects produced by these plugins also end up being allocated on the Java Heap, incurring the cost of copying from buffers to heap memory regardless of the setting.

[andsel]

We always use the Netty PooledAllocator, but instead of using direct memory buffers it uses Java heap byte[]

[andsel]

Hi @yaauie

While allocating direct memory for an individual buffer is faster,

The is not the allocation that's faster, it's the transfer from OS buffers to direct buffers that's faster, that's usually needed when data is moved around in OS, so from network to filesystem, or network to network. In our case that direct buffers content always flow into Java heap space to inflate Logstash's events, so we loose the benefit of direct buffer, or at least is not so dominant.

[jsvd]

Minor tweaks, otherwise LGTM, but I'd like @karenzone's input here as well.

[karenzone]

Left some suggestions for your consideration. Please LMKWYT.

[andsel]

Thanks @karen for all your precious suggestions. I've integrated all because results in more fluent wording without changing the meaning.

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 756602f

History

• 💚 Build #1720 succeeded d64d834
• 💚 Build #1701 succeeded a9098a4
• 💛 Build #1672 was flaky 44fdfaf
• 💛 Build #1650 was flaky d851770
• 💚 Build #1647 succeeded 6f4878e
• 💚 Build #1619 succeeded 54d1315

cc @andsel

[karenzone]

LGTM. Thanks!