[ES|QL] Double parameter markers for identifiers

[fang-xing-esql]

#115061 introduces the support of (named)parameters for identifiers - field and function names with a single parameter marker ? in ES|QL queries, however it requires users to provide the context whether the provided parameter is a value, identifier or pattern in the params string in a request to remove ambiguity, as they are all represented by ? and there are multiple places in the ES|QL syntax where several of these variations are valid options, that causes ambiguity in the query language.

This PR use double parameter markers - ?? to represent field or function name in ES|QL queries, so that the params string can be simplified, and users don't have to annotate a field or function name with identifier.

For now, this PR doesn't remove the old syntax of a single ? for identifier just for backward compatibility, and ?? is under snapshot, both syntax coexist so that Kibana have time to adapt to ??. This is related to elastic/kibana#209731.

There are three variations of ??, similar with ?, it can be a named parameter, a positional parameter or an anonymous parameter, below are some examples to show the usage of ?? with different variations.

named parameters - ?? followed by a named defined in params

POST /_query?format=txt
{
 "query": """
   FROM sample_data
   | STATS x = ??param1(??param2) by ??param3
   | SORT ??param3
   """,
 "params": [ {"param1" : "avg"}, {"param2" : "event.duration"}, {"param3" : "client.ip"}]
}

POST /_query?format=txt
{
 "query": """
   FROM persons
   | WHERE ??param1.??param2 == ?param3
   | KEEP ??param1.??param2, ??param1.??param4, ??param5
   | SORT ??param5
   """,
   "params": [{"param1" : "car"}, {"param2" : "make"}, {"param3" : "Tesla"}, {"param4" : "model"}, {"param5" : "first_name"}]
}

positional parameters - ?? followed by an integer number indicating the position of the parameter in params

POST /_query?format=txt
{
 "query": """
   FROM sample_data
   | STATS x = ??1(??2) by ??3
   | SORT ??3
   """,
 "params": [ {"param1" : "avg"}, {"param2" : "event.duration"}, {"param3" : "client.ip"}]
}

POST /_query?format=txt
{
 "query": """
   FROM sample_data
   | STATS x = ??1(??2) by ??3
   | SORT ??3
   """,
 "params": [ "avg", "event.duration", "client.ip"]
}

anonymous parameters - just a ??

POST /_query?format=txt
{
 "query": """
   FROM sample_data
   | STATS x = ??(??) by ??
   | SORT ??
   """,
 "params": [ "avg", "event.duration", "client.ip", "client.ip"]
}

[elasticsearchmachine]

Hi @fang-xing-esql, I've created a changelog YAML for you.

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[elasticsearchmachine]

Pinging @elastic/kibana-esql (ES|QL-ui)

[fang-xing-esql]

@elasticmachine update branch

[fang-xing-esql]

@elasticmachine update branch

[stratoula]

Which is the plan for keeping this only on snapshot releases? I am going to work on it immediately when this is merged and ideally we want to release it asap. Does it make sense for you?

[fang-xing-esql]

Which is the plan for keeping this only on snapshot releases? I am going to work on it immediately when this is merged and ideally we want to release it asap. Does it make sense for you?

I'll test and try to merge this(??) without being behind snapshot, and keep the old approach(?) coexist until Kibana switch to ?? successfully. Hope it will make it easier for Kibana.

[stratoula]

Totally 🙌

You can also merge with snapshot, give me time to create the PR (which means that I will also test the feature) and when I am ready to merge I can ping you to remove the snaphsot. I am also fine with this approch!

[costin]

LGTM

[costin]

👍 for the extensive tests

[elasticsearchmachine]

💔 Backport failed

Status	Branch	Result
❌	8.x	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 122459

[fang-xing-esql]

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Questions ?

Please refer to the Backport tool documentation