Add X-Opaque-Id to root level application.log

[D-shaw]

In #31521 X-Opaque-Id was added to access logs and slow logs. It would be very useful for investigation if whenever Elasticsearch throws an exception, X-Opaque-Id is appended to the exception log which locates at root level application.log.

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Logging)

[pgomulka]

we discussed this on a core-infra sync and we thought it would be a good feature.
However I am not sure if we could make it work easily.
We propagate X-Opaque-ID header directly from request onto the ThreadContext and from there it goes into a deprecation logger or search task context.

In order to have it available everywhere we would have to introduce an ExceptionLogger that would have access to ThreadContext. This would however mean we would have to find all usages of loggers which log an exception and use ExceptionLogger there.

[mosche]

Can't this be done using a LogEventPatternConverter, as done for trace_id?

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)