Doc values (doc['field'] notation) does not work on update

[OlivierPetri]

Elasticsearch version: 6.2

Plugins installed: []

JVM version: 8 (presumably any)

OS version: Any

Description of the problem including expected versus actual behavior:

The doc['field'] notation works well when applied to script_fields but it does not in _update and _update_by_query. It will rather give a null_pointer_exception, no matter what input document.

The notation is introduced as "doc values" in the reference documentation at "Modules/Scripting/Accessing document fields and special variables" as a more performant mean to access document fields.

Steps to reproduce:

PUT sumtest
{
	"mappings" : {
	"doc" : {
	"properties" : {
		"NRA" : { "type" : "integer" },
		"NRB" : { "type" : "integer" },
		"SUMAB" : { "type" : "integer" }
	}
	}
	}
}

PUT sumtest/doc/1
{
	"NRA" : 12,
	"NRB" : 34,
	"SUMAB" : 0,
}

------------

POST sumtest/_update_by_query
{
  "script" : {
      "source" : "doc['SUMAB'] = doc['NRA'] + doc['NRB']",
    "lang": "painless"
  },
  "query": {
    "match_all": {}
  }
}

Provide logs (if relevant):

{
    "error": {
        "root_cause": [
            {
                "type": "script_exception",
                "reason": "runtime error",
                "script_stack": [
                    "doc['SUMAB'] = doc['NRA'] + doc['NRB']",
                    "                   ^---- HERE"
                ],
                "script": "doc['SUMAB'] = doc['NRA'] + doc['NRB']",
                "lang": "painless"
            }
        ],
        "type": "script_exception",
        "reason": "runtime error",
        "script_stack": [
            "doc['SUMAB'] = doc['NRA'] + doc['NRB']",
            "                   ^---- HERE"
        ],
        "script": "doc['SUMAB'] = doc['NRA'] + doc['NRB']",
        "lang": "painless",
        "caused_by": {
            "type": "null_pointer_exception",
            "reason": null
        }
    },
    "status": 500
}

[OlivierPetri]

You can vary this using other field types, using _update instead of _update_by_query and using doc['SUMAB'].value instead of doc['SUMAB']. The result ist all the same.

[elasticmachine]

Pinging @elastic/es-core-infra

[rjernst]

This is a long known issue, most recently discussed in #29290. Update scripts simply do not have access to doc values, since the doc may be coming from the translog. Eventually using doc in an update script will cause a compilation failure instead of a runtime failure, thanks to script contexts now allowing to have different script signatures, but this has not yet been done.