Completion Suggester V2

[areek]

Completion Suggester V2

The completion suggester provides auto-complete/search-as-you-type functionality.
This is a navigational feature to guide users to relevant results as they are typing, improving search precision. It is not meant for spell correction or did-you-mean functionality like the term or phrase suggesters.

The completions are indexed as a weighted FST (finite state transducer) to provide fast Top N prefix-based
searches suitable for serving relevant results as a user types.

Notable Features:

• Document oriented suggestions:
  • Near-real time.
  • Deleted document filtering.
  • Multiple Context support.
  • Return document field values via payload.
• Query Interface:
  • Regular expression support via regex.
  • Typo tolerance via fuzzy.
  • Context boosting at query time.

Completion Suggester V2 is based on LUCENE-6339 and LUCENE-6459, the first iteration of Lucene's new suggest API.

Mapping

The completion fields are indexed in a special way, hence a field mapping has to be defined.
Following shows a field mapping for a completion field named title_suggest:

PUT {INDEX_NAME}
{
 "mappings": {
  {TYPE_NAME}: {
   "properties": {
    "title_suggest": {
     "type": "completion"
   }
  }
 }
}

You can choose index and search time analyzer for the completion field by adding analyzer
and search_analyzer options.

Context Mappings

Adding a contexts option in the field mapping defines a context-enabled completion field. You may want
a context-enabled completion field, if you require filtering or boosting suggestions by a criteria other than
just its prefix. Note that adding high-cardinality context values will increase the size of the in-memory
index significantly.

There are two types of supported context types: category and geo.

Category Context Mapping

Category contexts are indexed as prefixes to the completion field value.

The following adds a category context named genre:

...
"contexts": [
 {
   "name": "genre",
   "type": "category"
 }
]

You can also pull context values from another field in a document by using a path option specifying the field name.

Geo Context Mapping

Geo points are encoded as geohash strings and prefixed to the completion field value.
The following adds a geo context named location:

...
"contexts": [
 {
   "name": "location",
   "type": "geo"
 }
]

You can also set precision option to choose the geohash length and path to pull context values from another
field in the document.

Indexing

Just like any other field, you can add multiple completion fields to a document. You can also index multiple completions
for a completion field per document. Each completion value is tied to its document and can be assigned an index-time
weight, which determines its relative rank among other completion values which share a common prefix.

The following indexes a completion value and its weight for the title_suggest completion field:

POST {INDEX_NAME}/{TYPE_NAME}
{
 "title_suggest": {
  "input": "title1",
  "weight": 7
 }
}

You can use the short-form, if you prefer not to add weight to the completions:

POST {INDEX_NAME}/{TYPE_NAME}
{
 "title_suggest": "title1",
}

Arrays are also supported to index multiple values,

The following indexes multiple completion entries (input and weight) for a single document:

POST {INDEX_NAME}/{TYPE_NAME}
{
 "title_suggest": [
  {
   "input": "title1",
   "weight": 14
  },
  {
   "input": "alternate_title",
   "weight": 7
  }
 ]
}

Indexing context-enabled fields

You can use the path option previously mentioned to pull context values from another field
in the document or add contexts option to the completion entry while indexing.

The following explicitly indexes context values along with completions:

POST {INDEX_NAME}/{TYPE_NAME}
{
 "genre_title_suggest": {
  "input": "title1",
  "contexts": {
   "genre": ["genre1", "genre2"]
  },
  "weight": 7
 }
}

You can also configure the path option in the context mapping to pull values from another
field as follows (assuming path for the genre context has been set to genre field):

POST {INDEX_NAME}/{TYPE_NAME}
{
 "genre_title_suggest": "title1",
 "genre": ["genre1", "genre2"]
}

Query Interface

The point of indexing values as completions is to be able to run fast prefix-based searches on them.
You can run Prefix, Fuzzy and Regex queries on all completion fields. In case of a context-
enabled completion field, providing no context indicates all contexts will be considered. But you
can not run a Context query on a completion field with no contexts. When a query is run on a context-
enabled field, the contexts for a completion is returned with the suggestion.

Prefix Query

The following suggests completions from the field title_suggest that start with titl:

POST {INDEX}/_suggest
{
 "suggest-namespace" : {
  "prefix" : "titl",
  "completion" : {
   "field" : "title_suggest"
  }
 }
}

The suggestions are sorted by their index-time weight.

Fuzzy Prefix Query

A fuzzy prefix query can serve typo-tolerant suggestions. It scores suggestions closer (based on its edit distance)
to the provided prefix higher, regardless of their weight.

POST {INDEX}/_suggest
{
 "suggest-namespace" : {
  "prefix" : "sug",
  "completion" : {
   "field" : "suggest",
   "fuzzy" : {        (1)
    "fuzziness" : 2
   }
  }
 }
}

Specify fuzzy as shown in (1) to use typo-tolerant suggester. Full options for fuzzy

Regex Prefix Query

A regex prefix query matches all the term prefixes that match a regular expression. Regex is anchored at the begining but not at the end.
The suggestions are sorted by their index-time weight.

POST {INDEX}/_suggest
{
 "suggest-namespace" : {
  "regex" : "s[u|a]g",    (1)
  "completion" : {
   "field" : "suggest"
  }
 }
}

Specify regex as shown in (1), instead of prefix to use regular expressions. Supported regular expression syntax

Context Query

Adding contexts (1) option to the query enables filtering and/or boosting suggestions based on their context values.
This query scores suggestions by multiplying the query-time boost withe the suggestion weight.

POST {INDEX}/_suggest
{
 "suggest-namespace" : {
  "prefix" : "sug",
  "completion" : {
   "field" : "genre_title_suggest",
   "contexts": {           (1)
    "genre": [
     {
      "value" : "rock", 
      "boost" : 3
     },
     {
      "value" : "indie",
      "boost" : 2
     }
    ]
   }
  }
 }
}

The contexts can also be specified without any boost:

  ...
  "contexts": {
    "genre" : ["rock", "indie"]
  }

Geo Context Query:

The result will be scored such that the suggestions are first sorted by the distance between the corresponding geo context and the provided
geo location and then by the weight of the suggestions.

  ...
  "contexts" : {
    "location" : {
      "context" : {
        "lat" : ..,
        "lon" : ..
      },
      "precision" : ..
    }
  }

Example

The following performs a Fuzzy Prefix Query combined with a Context Query on a context-enabled completion field named genre_song_suggest.

POST {INDEX}/_suggest
{
 "suggest-namespace" : {
  "prefix" : "like a roling st",
  "completion" : {
   "field" : "genre_song_suggest",
   "fuzzy" : {
    "fuzziness" : 2
   },
   "contexts" : {
    "genre": [
     {
      "context" : "rock", 
      "boost" : 3
     },
     {
      "context" : "indie",
      "boost" : 2
     }
    ]
   }
  }
 }
}

This query will return all song names for the genre rock and indie that are within an edit distance of 2 from the prefix like a roling st.
The song names with genre of rock will be boosted higher then that of indie.
The completion field values that share the longest prefix with like a roling st will be additionally boosted higher.

Payload

You can retrieve any document field values along with its completions using the payload option.
The following returns the url field with each suggestion entry:

POST {INDEX}/_suggest
{
 "suggest-namespace" : {
  "prefix" : "titl",
  "completion" : {
   "field" : "title_suggest",
   "payload" : ["url"]
  }
 }
}

The response format is as follows:

{
 ...
 "suggest-namespace" : [ 
  {
   "prefix" : "sugg",
   "offset" : 0,
   "length" : 4,
   "options" : [ 
    {
     "text" : "suggestion",
     "score" : 34.0, 
     "payload": {
       "url" : [ "url_1" ]
     }
    }
   ]
  } 
 ]
}

[clintongormley]

Hi @areek

I like the look of this a lot. Btw, at the top you say we can use an arbitrary filter, but then later you say only a prefix/regex/etc filter. Which is it?

Also, under "Context Query" you say "Prefix, Regex and Fuzzy queries can be used with this query." Can you provide an example of this?

[clintongormley]

@s1monw please could you review this as well.

[s1monw]

I like this a lot the only thing that I think is confusing is that we use text as the property name if it's prefix / fuzzy prefix, should we just name it prefix since we also use regex? I wonder if we finally plan to return the entire document here?

[areek]

@clintongormley & @s1monw thanks for taking a look!

I have updated the Query Interface section to clarify how multiple queries can be used together (includes example).

text has been renamed to prefix for prefix queries and regex for regex prefix queries. At the moment, _source and fields can be used to retrieve field values for all resulting documents (like the _search API).

Also updated the issue with link to LUCENE-6459, query interface for suggest API.

Let me know if there is still any confusion with the API.

[clintongormley]

Looking awesome. A few thoughts:

• The search_analyzer should default to the analyzer setting - just checking that this is the case.
• We should probably remove the fuzzy wrapper and just allow specifying fuzziness directly?
• For geo-context searching, you sort on distance and THEN on weight? Feels like we should be able to combine these two factors in a user configurable way.

[areek]

Thanks @clintongormley for the feedback!

The search_analyzer should default to the analyzer setting - just checking that this is the case.

That is the case :)

We should probably remove the fuzzy wrapper and just allow specifying fuzziness directly?

If we remove the fuzzy wrapper, there would be no way for the user to specify transpositions, min_length, prefix_length and unicode_aware. I think it is useful to expose at least the min_length and prefix_length to the user?

For geo-context searching, you sort on distance and THEN on weight? Feels like we should be able to combine these two factors in a user configurable way.

We can allow this to be configurable (distance then weight or weight then distance) in the future. I think sorting the suggestions by distance and then weight for the first iteration is good?

[clintongormley]

If we remove the fuzzy wrapper, there would be no way for the user to specify transpositions, min_length, prefix_length and unicode_aware. I think it is useful to expose at least the min_length and prefix_length to the user?

OK, that's fine. Btw, do we really need unicode_aware? We use unicode everywhere, not sure why we would ever disable that?

We can allow this to be configurable (distance then weight or weight then distance) in the future. I think sorting the suggestions by distance and then weight for the first iteration is good?

OK