Supporting EQL in Elasticsearch

[colings86]

This is a meta issue to track progress of adding EQL support to Elasticsearch. EQL will be supported via a new Elastic licensed plugin which will provide execution of EQL rules.

The language reference for EQL can be found here

Scope of first iteration

Language features

In Scope

• Sequences
• Joins (Note this is specifically EQL joins which are most like a merge join)
• Pipes
• Most functions (see EQL: Add support for existing functions #51556)

Out of Scope

• Lineage, i.e.:
  • Descendant of
  • Child of
  • Event of
• Array functions
  • arrayContains
  • arraySearch
  • arrayCount
• fork in sequences

Things to consider during the design and implementation

• EQL is a language that’s already used in the wild. Any changes we make to the language or the way EQL rules are written will have implications on the existing users and migration of users to future versions so these should be very carefully considered
• A separate implementation of EQL will still exist that queries the endpoint devices directly. This means that we should avoid differences between writing EQL for Elasticsearch and EQL to run on the endpoints as any differences would have the potential to confuse users
• We see EQL being useful in many use cases. The Elasticsearch implementation need to be able to be used on generic data not just Endpoint data. EQL should default to expecting data in ECS format, but should have the ability to be overridden so it can be used on other schemas as well

High level tasks

Each task here has its own issue and some bigger tasks might have their own meta issue:

• Create EQL plugin Create EQL Plugin projects #49583
• Extract reusable components from SQL into common project Extract common/reusable components from SQL for EQL #49773
• Synchronous EQL REST API for ad-hoc querying Create Synchronous EQL querying REST API #49634
• Build query parser and plan on reusable SQL components Build EQL Query Parser and planner on top of reusable xQL components #49997
• EQL transpiler for stateless EQL expressions to ES Search DSL Transpile EQL stateless expressions into ES Search DSL #49589
• MILESTONE 1: Ability to run stateless EQL queries from a synchronous request/response API
  • This means running queries of the form event_type where CRITERIA where CRITERIA can use field_name=value, together with and/or/not
  • The REST API will be a synchronous request/response so the client will have to wait on the connection for the response
• Response format enhancements EQL: Response format enhancements #52845
• Support existing EQL functions EQL: Add support for existing functions #51556
• Map EQL sequence parts to ES requests Map EQL sequence/join parts to ES requests #49590
• Sequence and join execution EQL sequence and join execution #49594
• Implement pipes logic in the plugin (not using aggregations) Implement EQL Pipes in EQL Plugin #49627
• Add support for EQL functions EQL: Add support for existing functions #51556
• High Level REST client support for EQL API EQL: Add High Level Rest Client #51961
• Cancelling a task's grandchildren when the task is cancelled Cancelling a task's grandchildren when the task is cancelled #50990
• Convert EQL REST API to be async (similar to async search) Make EQL REST Querying API async #49638
• Create documentation for using EQL [DOCS] Document EQL support in Elasticsearch #51057
• MILESTONE 2: Supports all in scope EQL language features for first release in an asynchronous API
  • This means the plugin will functionally support all the elements int he "Language Features" section above"
  • The API will be asynchronous so a client can send a request and have a task ID returned, then check back for progress and final results later
  • Although functionally complete there are still items below we would want to implement before the first release is GA ready
• Develop Point-in-time reader @jimczi Move the state of search requests to the coordinator node #46523
• Modify EQL execution to use point in time reader Modify EQL execution to use point in time reader #49628
• Push down pipes to aggregations where possible Pushdown EQL pipes to aggregations where possible #49629
• Telemetry for EQL usage EQL usage telemetry #49630
• Improvements to search_after performance Can we use top hits optimizations when sorting by a field? #37043
• Improve wildcard search performance Faster wildcard search #48852
• Escape non-alphanumeric fields for EQL Escape non-alphanumeric fields for EQL #51443
• MILESTONE 3: Ready for first release
• Remove feature flags and prepare for release Prepare EQL for release #51613

[elasticmachine]

Pinging @elastic/es-search (:Search/EQL)

[alexbrasetvik]

It seems that some of the infrastructure required for this (notably the multiple search request executor and limited join-handling) could also eventually help support SQL WITH-queries (common table expressions) in some distant future :)