Always compress based on the settings

[Tim-Brooks]

Currently TransportRequestOptions allows specific requests to request
compression. This commit removes this and always compresses based on the
settings. Additionally, it removes TransportResponseOptions as they
are unused.

This closes #36399.

[elasticmachine]

Pinging @elastic/es-distributed

[Tim-Brooks]

Background

This approach is based on the discussion in #36399. I want to point out that it is kind of different than the approaches discussed in #33844. In #33844 there was agreement that the TransportRequestOptions should be respected. However, this removes the compression option.

Just so the rules are understood here:

Behavior

Requests

transport.tcp.compress - is the default setting. It defines whether the local cluster requests are compressed. It is also the fallback setting for remote clusters. This setting will probably change to transport.compress in 7.0.

cluster.remote.name.transport.compress - is the setting for remote clusters. If this is set for a remote cluster, it is the setting that is used.

Responses

We compress the response IF the request was compressed. This is a change from prior (6.5) behavior which would only compress responses if transport.tcp.compress was set to true. This is because we do not know what setting to apply, as we do not know if an inbound connection is remote or local.

Also, we will ALWAYS compress responses if transport.tcp.compress is set to true. Do we want this behavior? This is to be compatible with the prior behavior where if transport.tcp.compress was set to true, responses would be compressed. But maybe we should move to a world where we always do what the request sender is doing?

Blocker

This should be considered a blocker for 6.6 as the current behavior will lead to many more requests being compressed than in 6.5. And that will lead to unexpected performance degradation.

[jasontedor]

Looks good, left one comment twice. No need for another round.

[ywelsch]

Thanks @tbrooks8