Stored fields are getting deleted on partial update of the document.

[kiljag]

Elasticsearch Version

Version: 8.9.0

Installed Plugins

No response

Java Version

openjdk 17.0.8 2023-07-18

OS Version

Linux ip-172-31-15-10 ...aws #44~20.04.1-Ubuntu SMP Thu Jun 22 12:21:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Problem Description

I have an index created using a template where few fields are excluded from _source, but stored as stored fields by setting "store" to true. But when I try to update (partial update) the document, existing stored fields are getting deleted.

Steps to Reproduce

1. Create an index.

PUT itf_test_index
{
  "mappings": {
    "_source": {
        "excludes": [ "inference*", "metric*"]
      },
    "dynamic_templates": [
      {
          "inference_fields": {
            "match": "inference*",
            "mapping": {
              "index": false,
              "store": true,
              "type": "text"
            }
          }
        },
        {
          "metric_fields": {
            "match": "metric*",
            "mapping": {
              "index": false,
              "store": true,
              "type": "text"
            }
          }
        }
    ]
  }
}

2. Upload a document to the index

POST itf_test_index/_create/1
{
  "imageName": "123.jpg",
  "attribute_1": "Cat",
  "inference_1": "inference_1"
}

3. Retrieve the document

GET itf_test_index/_search
{
  "_source": ["*"],
  "stored_fields": ["*"]
}

######

"hits": [
{
    "_index": "itf_test_index",
    "_id": "1",
    "_score": 1,
    "_source": {
        "imageName": "123.jpg",
        "attribute_1": "Cat"
    },
    "fields": {
        "inference_1": [
            "inference_1"
        ]
    }
}
]

At this point, everything is as expected.

4. Update the document

POST itf_test_index/_update/1
{
  "doc": {
    "metric_1" : "metric_1"
  }
}

5. Retrieve the updated document.

GET itf_test_index/_search
{
"_source": [""],
"stored_fields": [""]
}

"hits": [
{
"_index": "itf_test_index",
"_id": "1",
"_score": 1,
"_source": {
"imageName": "123.jpg",
"attribute_1": "Cat"
},
"fields": {
"metric_1": [
"metric_1"
]
}
}
]

Now, previously stored "inference_1" field is not available anymore. Is this the expected behaviour and am I missing something in template creation of updating the document ?

Thank you.

Logs (if relevant)

No response

[gwbrown]

I got a bad feeling about this and took the liberty of reproducing it. The (excellent, thank you @kiljag) reproduction instructions worked exactly as described, but on 8.8.2, the stored field survives the update. It looks to me like we shipped a regression in 8.9.0. Nevermind, I got something mixed up in initial testing. The behavior is as described on 8.9.0, 8.8.2, and 8.1.3.

Edit: I'm not sure exactly what to tag this - document updates are a bit in an overlapping area - but I'm going to ping appropriate folks internally.

[elasticsearchmachine]

Pinging @elastic/es-search (Team:Search)

[gwbrown]

Update to the above: I think I got mixed up in my testing. The behavior IS as described but is IS NOT a regression (at least from recent versions).

[kiljag]

Hi @gwbrown, Thanks for coming back on this.
So, what should be the ideal behaviour in this situation? stored fields are supposed to be vanished or to be retained post doc update?

[gwbrown]

After getting some feedback internally, this is apparently expected and documented behavior - see this docs page. That said, I don't think this is a very intuitive way for this to work and would like to see it change.

However, we also don't have a concrete proposal for what it should do instead. Since we don't have enough here to really propose a fix, I'm going to close this issue, but I would be open to a new one proposing a different behavior.

[S-Dragon0302]

I think the stored field should be retained after the document is updated because the updated characters are not those of _source. The result should remain unchanged

[S-Dragon0302]

I also have such a scenario. I only want to update some frequently changing fields. Most of these fields are of numerical type. I have closed the source of text-type data. After only updating the numerical fields, the index of text type is affected. I think this is incorrect. If full document coverage is required, the update cost is too high.

[DaveCTurner]

Thanks very much for your interest in Elasticsearch @S-Dragon0302.

This appears to be a user question, and we'd like to direct these kinds of things to the Elasticsearch forum. If you can stop by there, we'd appreciate it. This allows us to use GitHub for verified bug reports, feature requests, and pull requests.

There's an active community in the forum that should be able to help get an answer to your question.