Transparent compression of requests and responses #20128
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Added options `--compress-transfer` and `--compress-request-threshold` to all client tools. These options can be used to enable transparent compression of the data that is sent between the client tools and the ArangoDB server. If the option `--compress-transfer` is set to `true`, the client-tools will add an extra HTTP header `Accept-Encoding: deflate` to all requests they make to the server. This allows the server to compress its responses before sending them back to the client tools. If the option is set to `true`, the client will also transparently compress their own requests to the server if the size of the request body (in bytes) is at least the value of the startup option `--compress-request-treshold`. The default value for this option is `0`, which disables compression of the request bodies in the client tools. To opt in to sending compressed data, the option must be set to a value greater than zero. Request body compression will be performed using the deflate compression algorithm. The client tools will also add a `Content-Encoding: deflate` header to the request when the request body was compressed. As compression will use CPU cycles, it should be activated only in case the network communication between the client tools and the server is slow and there is enough CPU capacity left for the extra compression/decompression work. Furthermore, requests should only be compressed when they exceed a certain minimum size, e.g. 250 bytes. Too small requests are typically not compressible very well. Request compression is only supported for HTTP/1.1 and HTTP/2 requests, but not when using the VST protocol. * Added startup option `--http.compress-response-threshold` to the ArangoDB server. The value of this specifies the threshold value (in bytes) from which on response bodies will be sent out compressed by the server. The default value for this option is `0`, which disables sending out compressed response bodies. To enable compression, the option should be set to a value greater than 0. The selected value should be large enough to justify the compression overhead. Too small response bodies are typically not well compressible. Regardless of the value of this option, response body compression will only happen in case the client advertised that it expects a compressed response body by sending an `Accept-Encoding: gzip` or `Accept-Encoding: deflate` header with its request. If that header is missing, no response compression will be performed by the server. As compression will use CPU cycles, it should be activated only in case the network communication between the server and clients is slow and there is enough CPU capacity left for the extra compression/decompression work. work. Furthermore, responses should only be compressed when they exceed a certain minimum size, e.g. 250 bytes. Response compression is only supported for HTTP/1.1 and HTTP/2 responses, but not when using the VST protocol.
…ngodb into feature/compression-support
…ngodb into feature/compression-support
…ngodb into feature/compression-support
MBkkt
approved these changes
Dec 5, 2023
markuspf
pushed a commit
that referenced
this pull request
Dec 6, 2023
* Transparent compression of requests and responses * Added options `--compress-transfer` and `--compress-request-threshold` to all client tools. These options can be used to enable transparent compression of the data that is sent between the client tools and the ArangoDB server. If the option `--compress-transfer` is set to `true`, the client-tools will add an extra HTTP header `Accept-Encoding: deflate` to all requests they make to the server. This allows the server to compress its responses before sending them back to the client tools. If the option is set to `true`, the client will also transparently compress their own requests to the server if the size of the request body (in bytes) is at least the value of the startup option `--compress-request-treshold`. The default value for this option is `0`, which disables compression of the request bodies in the client tools. To opt in to sending compressed data, the option must be set to a value greater than zero. Request body compression will be performed using the deflate compression algorithm. The client tools will also add a `Content-Encoding: deflate` header to the request when the request body was compressed. As compression will use CPU cycles, it should be activated only in case the network communication between the client tools and the server is slow and there is enough CPU capacity left for the extra compression/decompression work. Furthermore, requests should only be compressed when they exceed a certain minimum size, e.g. 250 bytes. Too small requests are typically not compressible very well. Request compression is only supported for HTTP/1.1 and HTTP/2 requests, but not when using the VST protocol. * Added startup option `--http.compress-response-threshold` to the ArangoDB server. The value of this specifies the threshold value (in bytes) from which on response bodies will be sent out compressed by the server. The default value for this option is `0`, which disables sending out compressed response bodies. To enable compression, the option should be set to a value greater than 0. The selected value should be large enough to justify the compression overhead. Too small response bodies are typically not well compressible. Regardless of the value of this option, response body compression will only happen in case the client advertised that it expects a compressed response body by sending an `Accept-Encoding: gzip` or `Accept-Encoding: deflate` header with its request. If that header is missing, no response compression will be performed by the server. As compression will use CPU cycles, it should be activated only in case the network communication between the server and clients is slow and there is enough CPU capacity left for the extra compression/decompression work. work. Furthermore, responses should only be compressed when they exceed a certain minimum size, e.g. 250 bytes. Response compression is only supported for HTTP/1.1 and HTTP/2 responses, but not when using the VST protocol. * fix compilation of mocks * fix compilation of tests * fix tests * transparently uncompress in arangosh as well * fix test error * fix test * fix cluster-internal request decompression * adjust compression limits to cover even more cases * added startup option * add some test for compression * some updates * fix test expectations --------- Co-authored-by: Max Neunhöffer <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Scope & Purpose
Enterprise companion PR: https://github.com/arangodb/enterprise/pull/1410
Added options
--compress-transferand--compress-request-thresholdto all client tools. These options can be used to enable transparent compression of the data that is sent between the client tools and the ArangoDB server.If the option
--compress-transferis set totrue, the client-tools will add an extra HTTP headerAccept-Encoding: deflateto all requests they make to the server. This allows the server to compress its responses before sending them back to the client tools.If the option is set to
true, the client will also transparently compress their own requests to the server if the size of the request body (in bytes) is at least the value of the startup option--compress-request-treshold. The default value for this option is0, which disables compression of the request bodies in the client tools. To opt in to sending compressed data, the option must be set to a value greater than zero. Request body compression will be performed using the deflate compression algorithm. The client tools will also add aContent-Encoding: deflateheader to the request when the request body was compressed. As compression will use CPU cycles, it should be activated only in case the network communication between the client tools and the server is slow and there is enough CPU capacity left for the extra compression/decompression work. Furthermore, requests should only be compressed when they exceed a certain minimum size, e.g. 250 bytes. Too small requests are typically not compressible very well.Request compression is only supported for HTTP/1.1 and HTTP/2 requests, but not when using the VST protocol.
Added startup option
--http.compress-response-thresholdto the ArangoDB server. The value of this specifies the threshold value (in bytes) from which on response bodies will be sent out compressed by the server.The default value for this option is
0, which disables sending out compressed response bodies. To enable compression, the option should be set to a value greater than 0. The selected value should be large enough to justify the compression overhead. Too small response bodies are typically not well compressible.Regardless of the value of this option, response body compression will only happen in case the client advertised that it expects a compressed response body by sending an
Accept-Encoding: gziporAccept-Encoding: deflateheader with its request. If that header is missing, no response compression will be performed by the server. As compression will use CPU cycles, it should be activated only in case the network communication between the server and clients is slow and there is enough CPU capacity left for the extra compression/decompression work. work. Furthermore, responses should only be compressed when they exceed a certain minimum size, e.g. 250 bytes. Response compression is only supported for HTTP/1.1 and HTTP/2 responses, but not when using the VST protocol.Checklist
Related Information