Mixing field collapsing and suggestions can lead to errors

[cbuescher]

There seem to be certain circumstances when requesting completion suggestions along with a regular search that uses the "collapse" feature can lead to "array_index_out_of_bounds_exception" in the fetch phase.

I was able to reproduce the issue on master and 7.11 with the following script:

DELETE test

PUT test
{
  "mappings": {
    "properties": {
      "collapse_field": {
        "type": "keyword"
      },
      "suggest_field": {
        "type": "completion",
        "analyzer": "whitespace"
      }
    }
  },
  "settings": {
    "number_of_shards": 2
  }
}

POST /test/_bulk?refresh=true
{ "index" : { "_id" : 1 } }
{ "collapse_field" : "aaa", "suggest_field" : "suggestion1" }
{ "index" : { "_id" : 2 } }
{ "collapse_field" : "bbb", "suggest_field" : "suggestion2" }
{ "index" : { "_id" : 3 } }
{ "collapse_field" : "bbb", "suggest_field" : "suggestion3" }
{ "index" : { "_id" : 4 } }
{ "collapse_field" : "ccc", "suggest_field" : "suggestion4" }
{ "index" : { "_id" : 5 } }
{ "collapse_field" : "ccc", "suggest_field" : "suggestion5" }

POST /test/_search
{
  "query": {"match_all": {}}, 
  "suggest": {
    "my_suggestion": {
      "prefix": "sug",
      "completion": {
        "field": "suggest_field",
        "size": 1
      }
    }
  },
  "collapse": {
    "field": "collapse_field"
  },
  "from": 3,
  "size": 1
}

The request returns

{
  "error" : {
    "root_cause" : [ ],
    "type" : "search_phase_execution_exception",
    "reason" : "",
    "phase" : "fetch",
    "grouped" : true,
    "failed_shards" : [ ],
    "caused_by" : {
      "type" : "array_index_out_of_bounds_exception",
      "reason" : "Index 1 out of bounds for length 1"
    }
  },
  "status" : 500
}

My guess is this only happens with at least 2 shards, but I might be mistaken. When I omitted the document ids, reproduction sometimes worked, sometimes failed, so I assume the distribution of documents on the shards also matters.

Stack trace of the exception is coming from SearchPhaseController.merge where we are exceeding the array bounds:

Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 1 out of bounds for length 1
        at org.elasticsearch.action.search.SearchPhaseController.merge(SearchPhaseController.java:272) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.search.FetchSearchPhase.moveToNextPhase(FetchSearchPhase.java:217) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.search.FetchSearchPhase.lambda$innerRun$1(FetchSearchPhase.java:101) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.search.CountedCollector.countDown(CountedCollector.java:40) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.search.ArraySearchPhaseResults.consumeResult(ArraySearchPhaseResults.java:35) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.search.CountedCollector.onResult(CountedCollector.java:48) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.search.FetchSearchPhase$2.innerOnResponse(FetchSearchPhase.java:171) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]

[elasticmachine]

Pinging @elastic/es-search (Team:Search)