Add support for extracting ranges from percolator queries

[martijnvg]

Also stop support percolator queries containing range queries with ranges based on now.

[jpountz]

maybe remove the dot after time?

[jpountz]

I don't understand how the percolator can correlate the ranges of values with the name of the field. For instance if I index a query on a foo field and try to percolate a document that contains a bar field, how does it know it should not try ot use the value of bar in the approximation query since it is a different field name? Similarly how does it work if a query has ranges on multiple field names?

[martijnvg]

I don't understand how the percolator can correlate the ranges of values with the name of the field. For instance if I index a query on a foo field and try to percolate a document that contains a bar field, how does it know it should not try ot use the value of bar in the approximation query since it is a different field name?

So it doesn't know and in the case of your example it the approximation query will match and then we the MemoryIndex verifies that match it will not verify it and so it will never match.

The goal here is to do better for range queries than we do right now (in many cases marking the entire percolator query as failed). The should clause with range queries that we add to the approximation query will yield more false matches, then the should clause we add for evaluating the percolator queries that failed query extraction. So I think in general it will be better?

Similarly how does it work if a query has ranges on multiple field names?

If the document has at least one range that matches (not necessarily matching with the field) with what we recorded for the percolator query at index time the approximation query will match it. Depending if the range matches with the field and if it needs to match with the other ranges too (depending if ranges are part of a conjunction or disjunction) it will be become a match after it has been verified by the MemoryIndex.

So in the previous approach we did record with what field a range belongs to, so it was less likely to be a false positive. The big down side is that we didn't have control over the amount of fields the percolator field type created, because for each number type + range field name combination we would introduce a two new field (from and to fields.

With this approach we will always have 12 fields for range queries. In retrospect I'm still not happy with the facts that we add 12 fields (it is still a lot) (field per number type) and that the approximation query is less accurate. I don't feel there is rush to get this in, there must be a better approach the handle range queries.

[jpountz]

Thanks for the explanation, it makes sense.

With this approach we will always have 12 fields for range queries. In retrospect I'm still not happy with the facts that we add 12 fields (it is still a lot) (field per number type) and that the approximation query is less accurate.

I have mixed feelings about it too. On the one hand I don't want to have one field per query field in order to index range queries, but on the other hand, using a field per field type introduces fuzziness that makes me a bit worried, both because it could be a nest for bugs, and also because there could certainly be patterns of range queries and/or documents that defeat this optimization by making queries always returned by the approximation.

I think we should put it on hold for now.

[martijnvg]

@jpountz I do want to extract the now range validation from PercolatorFieldMapper and some of the refactoring here (mainly the move of ExtractQueryTermsService#extractQueryTerms to PercolatorFieldMapper) and put it into a new pr. Does that make sense?

[jpountz]

+1

[martijnvg]

Closing this PR, we should look into this optimization another time. Opened #19327 for the now range validation.