Still OOM in the processing of search reduce phase

[boicehuang]

Bucket aggregation will consume a lot of memory on the coordinate node if it has a huge number of resulting buckets. search.max_buckets setting could limit maximum number of buckets allowed in a single response. Sometimes user may increase this setting to get more buckets, but it also increases the risk of OOM. It's hard to evaluate a suitable value for max_buckets.

PR #46751 introduces search.check_buckets_step_size setting and does a parent circuit breaker checking whenever allocating a certain number of new buckets. But it won't work after the change of PR #57042. Originally, consumeBucketsAndMaybeBreak was checked every time a bucket was generated. But now it would only be checked after reduce phase produces the result. It is easier to cause OOM during the reduce phase.

[elasticmachine]

Pinging @elastic/es-analytics-geo (Team:Analytics)

[boicehuang]

@imotov can you please have a look? Any help will be appreciated.