Always compress cluster state on transport layer #80104
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
DaveCTurner
added
>enhancement
:Distributed Coordination/Cluster Coordination
DaveCTurner
force-pushed
the
2021-10-29-compress-cluster-state-responses
branch
2 times, most recently
from
d889c5b to
f6ce9d3
Compare
We send cluster states over the wire in response to cluster state requests and when validating join requests. Today there's no special treatment of these messages so we just materialize the whole cluster state into a sequence of buffers which can be tens or hundreds of MBs in a big cluster, and it's quite possible that a really big cluster would exceed the 2GiB limit on message size. This commit forces transport compression on these messages even if it's usually disabled, which means that the materialized cluster state is always compressed. It tends to compress well, so this gives us a bit more headroom in terms of maximum cluster size. Relates elastic#77466 Closes elastic#79906
DaveCTurner
force-pushed
the
2021-10-29-compress-cluster-state-responses
branch
from
f6ce9d3 to
a4fa12a
Compare
|
Hi @DaveCTurner, I've created a changelog YAML for you. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We send cluster states over the wire in response to cluster state
requests and when validating join requests. Today there's no special
treatment of these messages so we just materialize the whole cluster
state into a sequence of buffers which can be tens or hundreds of MBs in
a big cluster, and it's quite possible that a really big cluster would
exceed the 2GiB limit on message size.
This commit forces transport compression on these messages even if it's
usually disabled, which means that the materialized cluster state is
always compressed. It tends to compress well, so this gives us a bit
more headroom in terms of maximum cluster size.
Relates #77466
Closes #79906