High level REST client test framework encourages using randomness for test coverage

[jasontedor]

We want to avoid using randomness for test coverage. Randomness is good in cases when the values are not expected to impact the behavior of the functionality under test (e.g., randomizing the number of shards, whether or not replicas are present, or the store type in most cases is not expected to have any impact on functionality). In these cases, using randomness ensures that we are in fact not relying on implementation details of components unrelated to the functionality under test. However we want to avoid using randomness to ensure branch coverage, as it leads to tests that only run a fraction of the time, and so we can end up with a test passing (e.g., on a PR), only to fail later in CI. Instead, we want distinct test cases.

The high level REST client test framework encourages using randomness to ensure test coverage of sync versus async method execution. That is, the test framework randomly tests sync as opposed to async methods, only half the time covering one and half the time the other. Some tests that are invoking multiple high level REST clients in a single test end up with a confusing mix of sync and async requests.

Instead, we would want to refactor these tests so that on every test run we are covering both sync and async methods.

[elasticmachine]

Pinging @elastic/es-core-features

[hub-cap]

++. Ive commented out this bit of code more than once when i was testing something I was modifying in the framework. Ill turn this issue into a meta that has more than just the sync/async randomness portion.

[hub-cap]

I looked into the framework for the rest tests, and this sync vs async is the only randomized offender for branch coverage. Ill be putting up a commit shortly that uses gradle to run the tests twice, once with all sync and once with all async.