Improve scroll search by using Lucene's IndexSearcher#searchAfter(...)

[martijnvg]

Improve the regular scroll search by using Lucene's searchAfter, which allows subsequent scroll request to always have a priority queue size equal to the specified size in the first search request. (priority queue is used to collect the competitive hits that match with a query)

Currently the priority queue size grows with each subsequent scroll request with what has been specified in from of the first search request.

Note: scan scroll is unaffected by this issue, which already is a highly optimized search to fetch a large part or all docs from a cluster. Scan scroll forcefully sort the hits always by the Lucene docids, while with the regular scroll can now support any sort efficiently.

[nik9000]

Is the idea to be able to scroll a non-scan search?

[martijnvg]

This is already possible, the scroll parameter can also be used on non scan search requests.

[nik9000]

Hey, neat. That is kinda documented on the scroll page but it is implied that scroll is a scan thing. So this enhancement will make it more efficient to scroll without scan?

[martijnvg]

Yes, this enhancement will make scroll without scan more efficient.

The memory usage will be improved from O(from+size) to O(size) and also collecting the competitive hits for a query will be improved from O(numHits + log(from+size)) to O(numHits + (log(size)). This improvement becomes really noticeable when scrolling deep into a result set.

[tlrx]

nice, thanks for this optimisation!