Search profiler mode focused on ES queries

[jtibshirani]

The search profiler is a really helpful tool in answering "what part of my search is slow"? But because it profiles Lucene queries, its output can be hard to understand for non-Lucene experts. For example, you might see that a PointInSetQuery or IndexOrDocValuesQuery are taking substantial time, but have no idea how this corresponds to your search request.

I wonder if we could have a profiler 'mode' that operates at the level of Elasticsearch queries. Its timings would be grouped by the queries the user actually sent, like terms or range.

Another option would be to expand the documentation on search profiling with more guidance on how to interpret the query output. This is more likely to get out-of-date though, and is still not as user-friendly. If we looked into a new profile 'mode' we could make sure to trade-off the added complexity vs. improving docs.

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[nik9000]

Could we include the "source" of the lucene query in the profiled output so folks could attach it to some part of their request?

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)

[javanna]

Closing in favour of #88370 .