Audit request.id propagate to CRUD actions

[albertzaharovits]

In #35536 we have added a new field for every audit event record, namely request.id. The goal was to use this id to correlate audit events.
For example, in:

{"@timestamp":"2019-01-28T12:23:24,952", "node.id":"BhChKPecQi6-S2GwCYaTJg", "event.type":"rest", "event.action":"authentication_success", "user.name":"elastic-admin", "origin.type":"rest", "origin.address":"[::1]:62149", "realm":"default_file", "url.path":"/_xpack/security/user/jacknich", "request.method":"POST", "request.id":"DnbbcktES8ubfXgbBsdjbw"}
{"@timestamp":"2019-01-28T12:23:25,030", "node.id":"BhChKPecQi6-S2GwCYaTJg", "event.type":"transport", "event.action":"access_granted", "user.name":"elastic-admin", "user.realm":"default_file", "user.roles":["superuser"], "origin.type":"rest", "origin.address":"[::1]:62149", "request.id":"DnbbcktES8ubfXgbBsdjbw", "action":"cluster:admin/xpack/security/user/put", "request.name":"PutUserRequest"}

The two authentication_success and access_granted events are correlated by the same DnbbcktES8ubfXgbBsdjbw id.

The problem is that this link gets broken when the transport action does a CRUD operation on an index. This is because these internal CRUD operations are guarded by executeAsyncWithOrigin which has the purpose of overriding the authorization of the internal operation by an internal, more restricted, user. But while doing this, we also clear the threadContext holding the request.id, so even if these CRUD actions are related (by causation) to the transport action they would be having different request.ids.

For example:

{"@timestamp":"2019-01-28T12:29:17,715", "node.id":"BhChKPecQi6-S2GwCYaTJg", "event.type":"transport", "event.action":"access_granted", "user.name":"_xpack_security", "user.realm":"__attach", "user.roles":["superuser"], "origin.type":"local_node", "origin.address":"127.0.0.1:9300", "request.id":"UB2oDlpXQsqYA-5NEO8G4A", "action":"indices:data/write/index", "request.name":"IndexRequest", "indices":[".security"]}
{"@timestamp":"2019-01-28T12:29:17,737", "node.id":"BhChKPecQi6-S2GwCYaTJg", "event.type":"transport", "event.action":"access_granted", "user.name":"_xpack_security", "user.realm":"__attach", "user.roles":["superuser"], "origin.type":"local_node", "origin.address":"127.0.0.1:9300", "request.id":"UB2oDlpXQsqYA-5NEO8G4A", "action":"indices:data/write/bulk", "request.name":"BulkRequest"}
{"@timestamp":"2019-01-28T12:29:17,759", "node.id":"BhChKPecQi6-S2GwCYaTJg", "event.type":"transport", "event.action":"access_granted", "user.name":"_xpack_security", "user.realm":"__attach", "user.roles":["superuser"], "origin.type":"local_node", "origin.address":"127.0.0.1:9300", "request.id":"UB2oDlpXQsqYA-5NEO8G4A", "action":"indices:data/write/bulk[s]", "request.name":"BulkShardRequest", "indices":[".security"]}
{"@timestamp":"2019-01-28T12:29:17,768", "node.id":"BhChKPecQi6-S2GwCYaTJg", "event.type":"transport", "event.action":"access_granted", "user.name":"_xpack_security", "user.realm":"__attach", "user.roles":["superuser"], "origin.type":"local_node", "origin.address":"127.0.0.1:9300", "request.id":"UB2oDlpXQsqYA-5NEO8G4A", "action":"indices:data/write/bulk[s][p]", "request.name":"BulkShardRequest", "indices":[".security"]}

These are all related to the first two events in the previous block, but have a different request.id that has been generated anew, after switching context.

I suggest we keep the request.id as well as the opaque_id across the executeAsyncWithOrigin context switch.

[elasticmachine]

Pinging @elastic/es-security

[albertzaharovits]

oops, I just realized this is a duplicate of #36492 . Closing...