ESQL: Enable LOOKUP JOIN in non-snapshot builds #121193

alex-spies · 2025-01-29T15:09:27Z

This effectively releases LOOKUP JOIN into tech preview. Docs will follow in a separate PR.

Enable the lexing/grammar for LOOKUP JOIN in non-snapshot builds.
Remove the grammar for the unsupported | JOIN ... command (without LOOKUP as first keyword). The way the lexer modes work, otherwise we'd also have to enable | JOIN ... syntax on non-snapshot builds and would have to add additional validation to provide appropriate error messages.
Remove grammar for LOOKUP JOIN index AS ... because qualifiers are not yet supported. Otherwise we'd have to put in additional validation as well to prevent such queries.

For this, we need to remove `| JOIN ...` (without `LOOKUP`) from the lexer/grammar to avoid enabling this in release mode as well.

Qualifiers are not yet supported. If we supported this grammar, we'd also need to validate against usage of AS.

The tables are only required for LOOKUP_🐔, and they were sent whenever the query contained just LOOKUP (ignoring case). Require the whole (disabled) command to be contained in the query to avoid sending it for LOOKUP JOIN tests.

elasticsearchmachine · 2025-01-29T15:09:51Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

elasticsearchmachine · 2025-01-29T15:09:52Z

Hi @alex-spies, I've created a changelog YAML for you. Note that since this PR is labelled release highlight, you need to update the changelog YAML to fill out the extended information sections.

This was meant to refer to lookup_chicken, not lookup join.

Drop check for snapshot build.

alex-spies · 2025-01-30T15:06:00Z

Ok, CI is running nearly green, but it somehow doesn't show in the PR's checks on GitHub:

The remaining failures in the release tests are all unrelated IMO. Here's the two build scans with the failures:

The failures are either in x-pack.inference, or in ESQL yaml tests. The former is more obviously unrelated. For the latter, the failures are in

test {p0=esql/40_tsdb/from doc with aggregate_metric_double}
FAILED	3.297s			
test {p0=esql/46_downsample/Query stats on downsampled index}
FAILED	0.900s			
test {p0=esql/40_tsdb/from index pattern unsupported counter}
FAILED	0.626s			
test {p0=esql/40_tsdb/stats on aggregate_metric_double}
FAILED	0.626s			
test {p0=esql/40_unsupported_types/unsupported}
FAILED	0.604s

And seem to be related to this PR from yesterday #120343, which also made changes to all the failing tests. /cc @limotova . None of the tests use LOOKUP JOIN and the test failures don't seem related to the grammar changes here at all.

alex-spies · 2025-01-30T15:06:45Z

Because CI is fine except for unrelated test failures, I will exceptionally merge this because fixing the failing, unrelated release tests will likely take a while.

elasticsearchmachine · 2025-01-30T15:10:08Z

💔 Backport failed

Status	Branch	Result
❌	8.19	The branch "8.19" is invalid or doesn't exist

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 121193

alex-spies · 2025-01-30T15:17:31Z

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Questions ?

Please refer to the Backport tool documentation

This effectively releases LOOKUP JOIN into tech preview. Docs will follow in a separate PR. * Enable the lexing/grammar for LOOKUP JOIN in non-snapshot builds. * Remove the grammar for the unsupported | JOIN ... command (without LOOKUP as first keyword). The way the lexer modes work, otherwise we'd also have to enable | JOIN ... syntax on non-snapshot builds and would have to add additional validation to provide appropriate error messages. * Remove grammar for LOOKUP JOIN index AS ... because qualifiers are not yet supported. Otherwise we'd have to put in additional validation as well to prevent such queries. --------- Co-authored-by: Bogdan Pintea <[email protected]> Co-authored-by: elasticsearchmachine <[email protected]> Co-authored-by: Nik Everett <[email protected]> (cherry picked from commit 70b397c) # Conflicts: # muted-tests.yml # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/parser/EsqlBaseParser.interp # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/parser/EsqlBaseParser.java

This effectively releases LOOKUP JOIN into tech preview. Docs will follow in a separate PR. * Enable the lexing/grammar for LOOKUP JOIN in non-snapshot builds. * Remove the grammar for the unsupported | JOIN ... command (without LOOKUP as first keyword). The way the lexer modes work, otherwise we'd also have to enable | JOIN ... syntax on non-snapshot builds and would have to add additional validation to provide appropriate error messages. * Remove grammar for LOOKUP JOIN index AS ... because qualifiers are not yet supported. Otherwise we'd have to put in additional validation as well to prevent such queries. --------- Co-authored-by: Bogdan Pintea <[email protected]> Co-authored-by: elasticsearchmachine <[email protected]> Co-authored-by: Nik Everett <[email protected]>

limotova · 2025-01-30T18:17:41Z

Hi! Looked into the test failures, I think this should address it

) This effectively releases LOOKUP JOIN into tech preview. Docs will follow in a separate PR. * Enable the lexing/grammar for LOOKUP JOIN in non-snapshot builds. * Remove the grammar for the unsupported | JOIN ... command (without LOOKUP as first keyword). The way the lexer modes work, otherwise we'd also have to enable | JOIN ... syntax on non-snapshot builds and would have to add additional validation to provide appropriate error messages. * Remove grammar for LOOKUP JOIN index AS ... because qualifiers are not yet supported. Otherwise we'd have to put in additional validation as well to prevent such queries.

This effectively releases LOOKUP JOIN into tech preview. Docs will follow in a separate PR. * Enable the lexing/grammar for LOOKUP JOIN in non-snapshot builds. * Remove the grammar for the unsupported | JOIN ... command (without LOOKUP as first keyword). The way the lexer modes work, otherwise we'd also have to enable | JOIN ... syntax on non-snapshot builds and would have to add additional validation to provide appropriate error messages. * Remove grammar for LOOKUP JOIN index AS ... because qualifiers are not yet supported. Otherwise we'd have to put in additional validation as well to prevent such queries.

alex-spies added 5 commits January 29, 2025 15:49

Enable LOOKUP JOIN + tests in release mode

6484509

For this, we need to remove `| JOIN ...` (without `LOOKUP`) from the lexer/grammar to avoid enabling this in release mode as well.

Remove DEV_ prefix from lexer rules

4152a7b

Disallow LOOKUP JOIN index AS

de150d9

Qualifiers are not yet supported. If we supported this grammar, we'd also need to validate against usage of AS.

Unmute LogicalPlanOptimizerTests

d986f26

Avoid sending tables in the request in tests

33f00a0

The tables are only required for LOOKUP_🐔, and they were sent whenever the query contained just LOOKUP (ignoring case). Require the whole (disabled) command to be contained in the query to avoid sending it for LOOKUP JOIN tests.

alex-spies added >enhancement release highlight auto-backport Automatically create backport pull requests when merged :Analytics/ES|QL AKA ESQL v9.0.0 v8.18.0 labels Jan 29, 2025

alex-spies requested a review from costin January 29, 2025 15:09

elasticsearchmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Jan 29, 2025

Update docs/changelog/121193.yaml

47de3c0

alex-spies added the test-release Trigger CI checks against release build label Jan 29, 2025

idegtiarenko approved these changes Jan 29, 2025

View reviewed changes

alex-spies and others added 3 commits January 29, 2025 18:06

Update GrammarInDevelopmentParsingTests

6857bee

This was meant to refer to lookup_chicken, not lookup join.

Update telemetry test using lookup join

c2358ae

Drop check for snapshot build.

Merge branch 'main' into unsnapshot_lookup_join

5164c24

bpintea added the auto-merge-without-approval Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) label Jan 29, 2025

elasticsearchmachine and others added 5 commits January 29, 2025 18:47

[CI] Auto commit changes from spotless

e68819b

Merge branch 'main' into unsnapshot_lookup_join

7ba9099

Merge branch 'main' into unsnapshot_lookup_join

fbe82a5

Merge branch 'main' into unsnapshot_lookup_join

6584706

Merge branch 'main' into unsnapshot_lookup_join

d29855e

alex-spies removed the auto-merge-without-approval Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) label Jan 30, 2025

Merge branch 'main' into unsnapshot_lookup_join

35e7771

alex-spies added the auto-merge-without-approval Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) label Jan 30, 2025

elasticsearchmachine added v8.19.0 and removed v8.18.0 labels Jan 30, 2025

alex-spies merged commit 70b397c into elastic:main Jan 30, 2025
15 of 19 checks passed

alex-spies deleted the unsnapshot_lookup_join branch January 30, 2025 15:09

elasticsearchmachine added the backport pending label Jan 30, 2025

alex-spies added v8.18.0 v9.1.0 labels Jan 30, 2025

alex-spies mentioned this pull request Jan 30, 2025

[8.x] ESQL: Enable LOOKUP JOIN in non-snapshot builds (#121193) #121297

Merged

alex-spies mentioned this pull request Jan 30, 2025

[9.0] ESQL: Enable LOOKUP JOIN in non-snapshot builds (#121193) #121303

Merged

alex-spies added >feature and removed >enhancement labels Jan 30, 2025

alex-spies mentioned this pull request Jan 30, 2025

[8.18] ESQL: Enable LOOKUP JOIN in non-snapshot builds (#121193) #121306

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

ESQL: Enable LOOKUP JOIN in non-snapshot builds #121193

ESQL: Enable LOOKUP JOIN in non-snapshot builds #121193

Uh oh!

alex-spies commented Jan 29, 2025 •

edited

Loading

Uh oh!

elasticsearchmachine commented Jan 29, 2025

Uh oh!

elasticsearchmachine commented Jan 29, 2025

Uh oh!

alex-spies commented Jan 30, 2025

Uh oh!

alex-spies commented Jan 30, 2025

Uh oh!

Uh oh!

elasticsearchmachine commented Jan 30, 2025

Uh oh!

alex-spies commented Jan 30, 2025

Uh oh!

limotova commented Jan 30, 2025

Uh oh!

Uh oh!

ESQL: Enable LOOKUP JOIN in non-snapshot builds #121193

ESQL: Enable LOOKUP JOIN in non-snapshot builds #121193

Uh oh!

Conversation

alex-spies commented Jan 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented Jan 29, 2025

Uh oh!

elasticsearchmachine commented Jan 29, 2025

Uh oh!

alex-spies commented Jan 30, 2025

Uh oh!

alex-spies commented Jan 30, 2025

Uh oh!

Uh oh!

elasticsearchmachine commented Jan 30, 2025

💔 Backport failed

Uh oh!

alex-spies commented Jan 30, 2025

💚 All backports created successfully

Questions ?

Uh oh!

limotova commented Jan 30, 2025

Uh oh!

Uh oh!

alex-spies commented Jan 29, 2025 •

edited

Loading