7.6.0 composite aggregation on index-sorting fields: out-of-bounds exception

[itizir]

Elasticsearch version (bin/elasticsearch --version): 7.6.0

Description of the problem including expected versus actual behavior:

Was trying to test the optimisation #48399 released in ES 7.6.0, and encountered this bug.

It's failing here https://github.com/elastic/elasticsearch/blob/v7.6.0/server/src/main/java/org/elasticsearch/search/aggregations/bucket/composite/CompositeAggregator.java#L206
when the aggregation buckets are a subset of the fields on which the index is sorted.

Steps to reproduce:

Minimal example highlighting the problem:

PUT /myindex
{"settings":{"index":{"sort.field":["foo","bar"]}},"mappings":{"properties":{"foo":{"type":"keyword"},"bar":{"type":"keyword"}}}}

POST /myindex/_doc?refresh
{"foo":"foo","bar":"bar"}

POST /myindex/_search
{"aggs":{"agg":{"composite":{"sources":[{"agg-foo":{"terms":{"field":"foo"}}}]}}}}

Results in failure

{
  "type" : "array_index_out_of_bounds_exception",
  "reason" : "Index 1 out of bounds for length 1"
}

Provide logs (if relevant):

"Caused by: org.elasticsearch.search.query.QueryPhaseExecutionException: Query Failed [Failed to execute main query]",
"at org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:312) ~[elasticsearch-7.6.0.jar:7.6.0]",
"at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:134) ~[elasticsearch-7.6.0.jar:7.6.0]",
"at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:338) ~[elasticsearch-7.6.0.jar:7.6.0]",
"at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:358) ~[elasticsearch-7.6.0.jar:7.6.0]",
"at org.elasticsearch.search.SearchService.lambda$executeQueryPhase$1(SearchService.java:343) ~[elasticsearch-7.6.0.jar:7.6.0]",
"at org.elasticsearch.action.ActionListener.lambda$map$2(ActionListener.java:146) ~[elasticsearch-7.6.0.jar:7.6.0]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.6.0.jar:7.6.0]",
"at org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:58) [elasticsearch-7.6.0.jar:7.6.0]",
"at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73) [elasticsearch-7.6.0.jar:7.6.0]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.0.jar:7.6.0]",
"at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.6.0.jar:7.6.0]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.0.jar:7.6.0]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.0.jar:7.6.0]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
"at java.lang.Thread.run(Thread.java:830) [?:?]",
"Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 1 out of bounds for length 1",
"at org.elasticsearch.search.aggregations.bucket.composite.CompositeAggregator.buildIndexSortPrefix(CompositeAggregator.java:206) ~[elasticsearch-7.6.0.jar:7.6.0]",
"at org.elasticsearch.search.aggregations.bucket.composite.CompositeAggregator.getLeafCollector(CompositeAggregator.java:286) ~[elasticsearch-7.6.0.jar:7.6.0]",
"at org.elasticsearch.search.aggregations.AggregatorBase.getLeafCollector(AggregatorBase.java:169) ~[elasticsearch-7.6.0.jar:7.6.0]",
"at org.elasticsearch.search.aggregations.AggregatorBase.getLeafCollector(AggregatorBase.java:42) ~[elasticsearch-7.6.0.jar:7.6.0]",
"at org.apache.lucene.search.MultiCollector.getLeafCollector(MultiCollector.java:124) ~[lucene-core-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
"at org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:186) ~[elasticsearch-7.6.0.jar:7.6.0]",
"at org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:171) ~[elasticsearch-7.6.0.jar:7.6.0]",
"at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445) ~[lucene-core-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
"at org.elasticsearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:333) ~[elasticsearch-7.6.0.jar:7.6.0]",
"at org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:295) ~[elasticsearch-7.6.0.jar:7.6.0]",
"at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:134) ~[elasticsearch-7.6.0.jar:7.6.0]",
"at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:338) ~[elasticsearch-7.6.0.jar:7.6.0]",
"at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:358) ~[elasticsearch-7.6.0.jar:7.6.0]",
"at org.elasticsearch.search.SearchService.lambda$executeQueryPhase$1(SearchService.java:343) ~[elasticsearch-7.6.0.jar:7.6.0]",
"at org.elasticsearch.action.ActionListener.lambda$map$2(ActionListener.java:146) ~[elasticsearch-7.6.0.jar:7.6.0]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.6.0.jar:7.6.0]",
"at org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:58) ~[elasticsearch-7.6.0.jar:7.6.0]",
"at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73) ~[elasticsearch-7.6.0.jar:7.6.0]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.6.0.jar:7.6.0]",
"at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) ~[elasticsearch-7.6.0.jar:7.6.0]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) ~[elasticsearch-7.6.0.jar:7.6.0]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.6.0.jar:7.6.0]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]",
"at java.lang.Thread.run(Thread.java:830) ~[?:?]"] }

[elasticmachine]

Pinging @elastic/es-analytics-geo (:Analytics/Aggregations)

[matriv]

Followed the steps and it's reproducable on current master as well.

[itizir]

Any news regarding this? It's a very unfortunate deal breaker...

[nik9000]

Thanks for the ping @itizir! I'm just coming up to speed on composite aggs and hadn't seen this.

[itizir]

Thanks for the quick fix!

[Oupsla]

Hello folks !

Do you have any alternative that we can set in the query (while waiting for the next version) ?
I have tried to specify the sort but seems to have no effect.

Thanks !

[nik9000]

Do you have any alternative that we can set in the query (while waiting for the next version) ?
I have tried to specify the sort but seems to have no effect.

I believe you should be able to replace the direct reference to the field with a script that just reads the value of the field. When you pass a script we can't enable the optimization that was broken here because we can't be sure that your script won't jumble the order of the terms. So a script should "escape" from the bug.

[Oupsla]

Thanks @nik9000 , will try it out !