Update Edge NGram Tokenizer documentation

[Stefanqn]

Elasticsearch version (bin/elasticsearch --version): current

Plugins installed: []

JVM version (java -version): -

OS version (uname -a if on a Unix-like system): -

Description of the problem including expected versus actual behavior: bad behaviour

Steps to reproduce:

1. read the elasticsearch documentation regarding Edge NGram and use the 'Example configuration' with split search_analyzer and analyzer
2. this will result in unexpected behavior when exceeding the max_gram length, e.g. for "max_gram": 3 searching for an existing, indexed "aaaa" returns an empty result set.
3. please update the documentation, e.g. with following configuration

 "settings": {
        "analysis": {
            "filter": {
                "truncate_search": {
                    "type": "truncate",
                    "length": 10
                }
            },
            "analyzer": {
                "autocomplete": {
                    "tokenizer": "autocomplete",
                    "filter": [
                        "lowercase"
                    ]
                },
                "autocomplete_search": {
                    "tokenizer": "lowercase",
                    "filter": "truncate_search"
                }
            },
            "tokenizer": {
                "autocomplete": {
                    "type": "edge_ngram",
                    "min_gram": 1,
                    "max_gram": 10,
                    "token_chars": [
                        "letter"
                    ]
                }
            }
        }
    }

showing the need for a truncate filter.

Provide logs (if relevant): -

[elasticmachine]

Pinging @elastic/es-search (:Search/Analysis)

[elasticmachine]

Pinging @elastic/es-docs (:Docs)

[jrodewig]

Hi @Stefanqn

Thanks for reaching out. Unfortunately, I wasn't able to reproduce your reported problem using the current snippet setup. I've outlined my steps below.

Can you highlight the expected behavior or where your steps differed?

1. Create an index with split search and index analyzers.

PUT my_index
{
  "settings": {
    "analysis": {
      "analyzer": {
        "autocomplete": {
          "tokenizer": "autocomplete",
          "filter": [
            "lowercase"
          ]
        },
        "autocomplete_search": {
          "tokenizer": "lowercase"
        }
      },
      "tokenizer": {
        "autocomplete": {
          "type": "edge_ngram",
          "min_gram": 2,
          "max_gram": 10,
          "token_chars": [
            "letter"
          ]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "title": {
        "type": "text",
        "analyzer": "autocomplete",
        "search_analyzer": "autocomplete_search"
      }
    }
  }
}

2. Index a document with a title of aaaa.

PUT my_index/_doc/1
{
  "title": "aaaa" 
}

3. Refresh the index.

POST my_index/_refresh

4. Search for aaaa in the title field.

GET my_index/_search
{
  "query": {
    "match": {
      "title": {
        "query": "aaaa", 
        "operator": "and"
      }
    }
  }
}

Response:

{
  "took" : 3,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 1,
      "relation" : "eq"
    },
    "max_score" : 0.2876821,
    "hits" : [
      {
        "_index" : "my_index",
        "_type" : "_doc",
        "_id" : "1",
        "_score" : 0.2876821,
        "_source" : {
          "title" : "aaaa"
        }
      }
    ]
  }
}

[Stefanqn]

please set "max_gram": 3 or use 11x 'a' = "aaaaaaaaaaa"

[jrodewig]

Thanks @Stefanqn. I was able to reproduce. I'll work on getting this fixed in our docs.

Thanks for reporting.

[Stefanqn]

thanks!