Term vectors query returns nothing in nested inner objects if term vectors are stored in the index

[gronostajo]

Elasticsearch version: 5.0.0, 2.4.1

Plugins installed: []

JVM version: 1.8.0_112-b15 64-bit

OS version: Windows 10 v. 1607 (build 14393.447)

Description of the problem including expected versus actual behavior:

When making a _termvectors query to an object in a "type": "nested" field, response is returned with "term_vectors" : { } if vectors are stored ("store": true in mapping). If the object isn't nested or term vectors aren't stored, the object returned is filled properly (examples below). When TV storing is enabled then despite the fact that I'm unable to retrieve them, highlighting seems to be significantly faster, as if term vectors were present.

Steps to reproduce:

1. PUT /mcve

2.  PUT /mcve/ex/_mapping
    {
      "properties": {
        "parent": {
          "type": "nested",
          "properties": {
            "value": {"type": "text", "term_vector": "with_positions_offsets", "store": true}
          }
        }
      }
    }

3.  PUT /mcve/ex/ample
    {
      "parent": {
        "value": "hello world"
      }
    }

4.  PUT /mcve/ex/ample/_termvectors
    {
      "fields": ["parent.value"]
    }

Response:

{
  "_index" : "mcve",
  "_type" : "ex",
  "_id" : "ample",
  "_version" : 1,
  "found" : true,
  "took" : 1,
  "term_vectors" : { }
}

However, if I disable storing term vectors by changing this line in mapping:

"value": {"type": "text", "term_vector": "with_positions_offsets", "store": true}

to this:

"value": {"type": "text"}

the result will be:

{
  "_index" : "mcve",
  "_type" : "ex",
  "_id" : "ample",
  "_version" : 1,
  "found" : true,
  "took" : 60,
  "term_vectors" : {
    "parent.value" : {
      "field_statistics" : {
        "sum_doc_freq" : 2,
        "doc_count" : 1,
        "sum_ttf" : 2
      },
      "terms" : {
        /* ...snip... */
      }
    }
  }
}

But highlighting queries will be slower (tested on a ~80 MB index with ~600 documents), as if term vectors were in fact built previously.

Provide logs (if relevant):

--

Related SO question

[jpountz]

This is a tricky problem. The reason why the term vectors API works when term vectors are disabled is that it ignores the fact that the sum documents are nested, which is probably a bug. I don't think we thought about how the term vectors API should work with nested documents. Out of curiosity, what is your use-case for retrieving term vectors on nested docs?

[jacool]

It seems the issue is not related to storing term vectors but to the "with_positions_offsets" option. As can be seen by this sequence:

PUT /mcve

PUT /mcve/ex/_mapping
{
  "properties": {
    "parent": {
      "type": "nested",
      "properties": {
        "value": { "type": "text",  "store": true }
      } } } }

PUT /mcve/ex/ample
{ "parent": { "value": "hello world" } }

GET /mcve/ex/ample/_termvectors
{ "fields": [ "parent.value" ] }

The last command produces the expected term-vectors.
However adding "term_vector": "with_positions_offsets" breaks the _termvectors functionality.
Reproducible with ES 5.4.0

[jpountz]

The issue is that the term vectors API has been designed with the idea in mind that there would be a single document composed of 0 or more fields. However when indexing with nested documents, you have multiple sub-documents that each have their own term vectors, se we'd have to return an array of term vectors for each nested path.

[jacool]

This is not what happens with the simple example we used above. Try to replace the indexing command above with this

PUT /mcve/ex/ample
{ "parent": [{ "value": "hello world " }, { "value": "having world" } ] }

And then request the _termvectors, you'll see some kind of merging is performed between the results representing inner documents. For us accidentally this merging makes sense, however the fact the functionality disappears once with_positions_offsets is declared seems like a bug.

[fbaligand]

I'm really interested by this issue fix.

[bkazez]

I'm interested in this too. At the very least, it would be helpful to return an error if someone tries to get term vectors for a nested document, since the documentation never mentions that this isn't supported.

Meanwhile, is there another way to debug analyzers for nested docs?

[mayya-sharipova]

cc @elastic/es-search-aggs

We currently still have this issue.

Stored in the index term vectors on nested inner values, are not returned when requested.
While generated on the fly term vectors on nested inner values are returned.

A way to reproduce

PUT tm
{
  "mappings": {
    "doc": {
      "properties": {
        "users1" : {
          "type" : "nested",
          "properties": {
            "name" : {"type": "text"}
          }
        },
        "users2" : {
          "type" : "nested",
          "properties": {
            "name" : {"type": "text", "term_vector": "yes"}
          }
        }
      }
    }
  }
}

PUT tm/doc/1
{
  "users1": {"name": "John Snow"},
  "users2": {"name": "John Snow"}
}

GET tm/doc/1/_termvectors?fields=users1.name
//will return term vectors

GET tm/doc/1/_termvectors?fields=users2.name
//will NOT return term vectors

Possible resolutions:

• accept this as a feature and document this behaviour
• return stored term vectors of nested inner values
• not return generated term vectors of nested inner values