diff --git a/docs/reference/eql/eql-search-api.asciidoc b/docs/reference/eql/eql-search-api.asciidoc index 3eaca341691e9..aecefb0037b50 100644 --- a/docs/reference/eql/eql-search-api.asciidoc +++ b/docs/reference/eql/eql-search-api.asciidoc @@ -81,6 +81,72 @@ Defaults to `open`. include::{docdir}/rest-api/common-parms.asciidoc[tag=index-ignore-unavailable] +`keep_alive`:: ++ +-- +(Optional, <>) +Period for which the search and its results are stored on the cluster. Defaults +to `5d` (five days). + +When this period expires, the search and its results are deleted, even if the +search is still ongoing. + +If the <> parameter is +`false`, {es} only stores <<> that do not +complete within the period set by the +<> +parameter, regardless of this value. + +[IMPORTANT] +==== +You can also specify this value using the `keep_alive` request body parameter. +If both parameters are specified, only the query parameter is used. +==== +-- + +`keep_on_completion`:: ++ +-- +(Optional, boolean) +If `true`, the search and its results are stored on the cluster. + +If `false`, the search and its results are stored on the cluster only if the +request does not complete during the period set by the +<> +parameter. Defaults to `false`. + +[IMPORTANT] +==== +You can also specify this value using the `keep_on_completion` request body +parameter. If both parameters are specified, only the query parameter is used. +==== +-- + +`wait_for_completion_timeout`:: ++ +-- +(Optional, <>) +Timeout duration to block other requests while waiting for the search request to +finish. Defaults to no timeout, meaning the search request waits for complete +search results. + +If this parameter is specified and the request completes before this period +ends, complete search results are returned. + +If the request does not complete before this period ends, the search becomes an +<>. A +<> is returned at the end of +the period. This ID can be used to monitor the progress of the async search as +it runs. + +[IMPORTANT] +==== +You can also specify this value using the `wait_for_completion_timeout` request +body parameter. If both parameters are specified, only the query parameter is +used. +==== +-- + [[eql-search-api-request-body]] ==== {api-request-body-title} @@ -102,6 +168,48 @@ runs. (Optional, string) Reserved for future use. +`keep_alive`:: ++ +-- +(Optional, <>) +Period for which the search and its results are stored on the cluster. Defaults +to `5d` (five days). + +When this period expires, the search and its results are deleted, even if the +search is still ongoing. + +If the <> parameter is +`false`, {es} only stores <<> that do not +complete within the period set by the +<> +parameter, regardless of this value. + +[IMPORTANT] +==== +You can also specify this value using the `keep_alive` query parameter. +If both parameters are specified, only the query parameter is used. +==== +-- + +[[eql-search-api-keep-on-completion]] +`keep_on_completion`:: ++ +-- +(Optional, boolean) +If `true`, the search and its results are stored on the cluster. + +If `false`, the search and its results are stored on the cluster only if the +request does not complete during the period set by the +<> +parameter. Defaults to `false`. + +[IMPORTANT] +==== +You can also specify this value using the `keep_on_completion` query parameter. +If both parameters are specified, only the query parameter is used. +==== +-- + `query`:: (Required, string) <> query you wish to run. @@ -134,10 +242,73 @@ milliseconds since the https://en.wikipedia.org/wiki/Unix_time[Unix epoch], in ascending order. -- +[[eql-search-api-wait-for-completion-timeout]] +`wait_for_completion_timeout`:: ++ +-- +(Optional, <>) +Timeout duration to block other requests while waiting for the search request to +finish. Defaults to no timeout, meaning the search request waits for complete +search results. + +If this parameter is specified and the request completes before this period +ends, complete search results are returned. + +If the request does not complete before this period ends, the search becomes an +<>. A +<> is returned at the end of +the period. This ID can be used to monitor the ongoing progress of the search as +it runs. + +[IMPORTANT] +==== +You can also specify this value using the `wait_for_completion_timeout` query +parameter. If both parameters are specified, only the query parameter is used. +==== +-- + [role="child_attributes"] [[eql-search-api-response-body]] ==== {api-response-body-title} +[[eql-search-api-response-body-search-id]] +`id`:: ++ +-- +Identifier for the search. + +This search ID is only provided if a search request does not return complete +results during the timeout period set by the +<> +parameter, becoming an <>. + +You can use this ID to +monitor the progress of an ongoing async search and retrieve complete results +when the search finishes. +-- + +`is_partial`:: +(boolean) +If `true`, the response does not contain complete search results. + +`is_running`:: ++ +-- +(boolean) +If `true`, the search request is still executing. + +[IMPORTANT] +==== +If this parameter and the `is_partial` parameter are `true`, the search is an +<>. If the `keep_alive` period does not +pass, the complete search results will be available when the search completes. + +If `is_partial` is `true` but `is_running` is `false`, the search returned +partial results due to a failure. Only some shards returned results or the node +coordinating the search failed. +==== +-- + `took`:: + -- @@ -168,7 +339,7 @@ If `true`, the request timed out before completion. `hits`:: (object) -Contains returned events and metadata. +Contains matching events and metadata. + .Properties of `hits` [%collapsible%open] @@ -176,7 +347,7 @@ Contains returned events and metadata. `total`:: (object) -Metadata about the number of returned events. +Metadata about the number of matching events. + .Properties of `total` [%collapsible%open] @@ -184,7 +355,7 @@ Metadata about the number of returned events. `value`:: (integer) -Total number of returned events. +Total number of matching events. `relation`:: + @@ -199,29 +370,80 @@ Returned values are: -- ===== +`sequences`:: +(array of objects) +Contains event sequences matching the query. Each object represents a +matching sequence. This parameter is only returned for EQL queries containing +a <>. ++ +.Properties of `sequences` objects +[%collapsible%open] +===== +`join_keys`:: +(array of strings) +Shared field values used to constrain matches in the sequence. These are defined +using the <> in the EQL query syntax. + `events`:: (array of objects) -Contains returned events matching the query. Each object represents a +Contains events matching the query. Each object represents a matching event. + .Properties of `events` objects [%collapsible%open] +====== +`_index`:: +(string) +Name of the index containing the event. + +`_id`:: +(string) +(string) +Unique identifier for the event. +This ID is only unique within the index. + +`_score`:: +(float) +Positive 32-bit floating point number used to determine the relevance of the + event. See <>. + +`_source`:: +(object) +Original JSON body passed for the event at index time. + +`sort`:: +(array) +Integer used as the sort value for the event. ++ +By default, this is the event's <>, converted to milliseconds since the +https://en.wikipedia.org/wiki/Unix_time[Unix epoch]. +====== ===== +[[eql-search-api-response-events]] +`events`:: +(array of objects) +Contains events matching the query. Each object represents a +matching event. ++ +.Properties of `events` objects +[%collapsible%open] +===== `_index`:: (string) -Name of the index containing the returned event. +Name of the index containing the event. `_id`:: (string) (string) -Unique identifier for the returned event. -This ID is only unique within the returned index. +Unique identifier for the event. +This ID is only unique within the index. `_score`:: (float) Positive 32-bit floating point number used to determine the relevance of the -returned event. See <>. + event. See <>. `_source`:: (object) @@ -240,11 +462,14 @@ https://en.wikipedia.org/wiki/Unix_time[Unix epoch]. [[eql-search-api-example]] ==== {api-examples-title} +[[eql-search-api-basic-query-ex]] +==== Basic query example + The following EQL search request searches for events with an `event.category` of `file` that meet the following conditions: * A `file.name` of `cmd.exe` -* An `agent.id` that is _not_ `my_user` +* An `agent.id` other than `my_user` [source,console] ---- @@ -256,13 +481,16 @@ GET my_index/_eql/search } ---- -The API returns the following response. Events in the response are sorted by -<>, converted to milliseconds since -the https://en.wikipedia.org/wiki/Unix_time[Unix epoch], in ascending order. +The API returns the following response. Matching events in the `hits.events` +property are sorted by <>, converted +to milliseconds since the https://en.wikipedia.org/wiki/Unix_time[Unix epoch], +in ascending order. [source,console-result] ---- { + "is_partial": false, + "is_running": false, "took": 6, "timed_out": false, "hits": { @@ -328,9 +556,132 @@ the https://en.wikipedia.org/wiki/Unix_time[Unix epoch], in ascending order. ] } ] - }, + } +} +---- +// TESTRESPONSE[s/"took": 6/"took": $body.took/] + +[[eql-search-api-sequence-ex]] +==== Sequence query example + +The following EQL search request matches a <> of events +that: + +. Start with an event with: ++ +-- +* An `event.category` of `file` +* A `file.name` of `cmd.exe` +* An `agent.id` other than `my_user` +-- +. Followed by an event with: ++ +-- +* An `event.category` of `process` +* A `process.path` that contains the substring `regsvr32` +-- + +These events must also share the same `agent.id` value. + +[source,console] +---- +GET my_index/_eql/search +{ + "query": """ + sequence by agent.id + [ file where file.name == "cmd.exe" and agent.id != "my_user" ] + [ process where stringContains(process.path, "regsvr32") ] + """ +} +---- + +The API returns the following response. The `hits.sequences.join_keys` property +contains the shared `agent.id` value for each matching event. Matching events in +the `hits.sequences.events` property are sorted by +<>, converted to milliseconds since +the https://en.wikipedia.org/wiki/Unix_time[Unix epoch], in ascending order. + +[source,console-result] +---- +{ "is_partial": false, - "is_running": false + "is_running": false, + "took": 6, + "timed_out": false, + "hits": { + "total": { + "value": 1, + "relation": "eq" + }, + "sequences": [ + { + "join_keys": [ + "8a4f500d" + ], + "events": [ + { + "_index": "my_index", + "_id": "4", + "_score": null, + "_source": { + "@timestamp": "2020-12-07T11:07:08.000Z", + "agent": { + "id": "8a4f500d" + }, + "event": { + "category": "file" + }, + "file": { + "accessed": "2020-12-07T11:07:08.000Z", + "name": "cmd.exe", + "path": "C:\\Windows\\System32\\cmd.exe", + "type": "file", + "size": 16384 + }, + "process": { + "name": "cmd.exe", + "path": "C:\\Windows\\System32\\cmd.exe" + } + }, + "fields": { + "@timestamp": [ + "1607339228000" + ] + }, + "sort": [ + 1607339228000 + ] + }, + { + "_index": "my_index", + "_id": "5", + "_score": null, + "_source": { + "@timestamp": "2020-12-07T11:07:09.000Z", + "agent": { + "id": "8a4f500d" + }, + "event": { + "category": "process" + }, + "process": { + "name": "regsvr32.exe", + "path": "C:\\Windows\\System32\\regsvr32.exe" + } + }, + "fields": { + "@timestamp": [ + "1607339229000" + ] + }, + "sort": [ + 1607339229000 + ] + } + ] + } + ] + } } ---- // TESTRESPONSE[s/"took": 6/"took": $body.took/] diff --git a/docs/reference/eql/limitations.asciidoc b/docs/reference/eql/limitations.asciidoc index 122d5ce2fe971..11513574a9226 100644 --- a/docs/reference/eql/limitations.asciidoc +++ b/docs/reference/eql/limitations.asciidoc @@ -37,4 +37,6 @@ queries that contain: * {eql-ref}/pipes.html[Pipes] -* {eql-ref}/sequences.html[Sequences] \ No newline at end of file +* {eql-ref}/sequences.html[State and timespan-related sequence keywords]: +** `with maxspan` +** `until` \ No newline at end of file diff --git a/docs/reference/eql/search.asciidoc b/docs/reference/eql/search.asciidoc index 84f3e1f28f0b6..a1fd0e7abd6f5 100644 --- a/docs/reference/eql/search.asciidoc +++ b/docs/reference/eql/search.asciidoc @@ -70,6 +70,8 @@ https://en.wikipedia.org/wiki/Unix_time[Unix epoch], in ascending order. [source,console-result] ---- { + "is_partial": false, + "is_running": false, "took": 60, "timed_out": false, "hits": { @@ -121,9 +123,7 @@ https://en.wikipedia.org/wiki/Unix_time[Unix epoch], in ascending order. ] } ] - }, - "is_partial": false, - "is_running": false + } } ---- // TESTRESPONSE[s/"took": 60/"took": $body.took/] @@ -215,3 +215,194 @@ GET /sec_logs/_eql/search } ---- ==== + +[discrete] +[[eql-search-async]] +=== Run an async EQL search + +EQL searches in {es} are designed to run on large volumes of data quickly, +often returning results in milliseconds. Because of this, the EQL search API +runs _synchronous_ searches by default. This means the search request +blocks other requests and waits for complete results before returning a +response. + +However, complete results can take longer for searches across: + +* <> +* <> +* Many shards + +To avoid long waits, you can use the EQL search API's +`wait_for_completion_timeout` parameter to run an _asynchronous_, or _async_, +search. + +Set the `wait_for_completion_timeout` parameter to a duration you'd like to wait +for complete search results. If the search request does not finish within this +period, the search becomes an async search. The EQL search +API returns a response that includes: + +* A search ID, which can be used to monitor the progress of the async search and + retrieve complete results when it finishes. +* An `is_partial` value of `true`, indicating the response does not contain + complete search results. +* An `is_running` value of `true`, indicating the search is async and ongoing. + +The async search continues to run in the background without blocking +other requests. + +[%collapsible] +.*Example* +==== +The following request searches the `frozen_sec_logs` index, which has been +<> for storage and is rarely searched. + +Because searches on frozen indices are expected to take longer to complete, the +request contains a `wait_for_completion_timeout` query parameter value of `2s` +(two seconds). + +If the request does not return complete results in two seconds, the search +becomes an async search and a search ID is returned. + +[source,console] +---- +GET /frozen_sec_logs/_eql/search +{ + "wait_for_completion_timeout": "2s", + "query": """ + process where process.name == "cmd.exe" + """ +} +---- +// TEST[s/frozen_sec_logs/sec_logs/] +// TEST[s/"wait_for_completion_timeout": "2s"/"wait_for_completion_timeout": "2s"/] + +After two seconds, the request returns the following response. + +[source,console-result] +---- +{ + "id": "FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=", + "is_partial": true, + "is_running": true, + "took": 2000, + "timed_out": false, + "hits": { + "total": { + "value": 2, + "relation": "eq" + }, + "events": [ + { + "_index": "frozen_sec_logs", + "_id": "1", + "_score": null, + "_source": { + "@timestamp": "2020-12-06T11:04:05.000Z", + "agent": { + "id": "8a4f500d" + }, + "event": { + "category": "process" + }, + "process": { + "name": "cmd.exe", + "path": "C:\\Windows\\System32\\cmd.exe" + } + }, + "sort": [ + 1607252645000 + ] + }, + { + "_index": "frozen_sec_logs", + "_id": "3", + "_score": null, + "_source": { + "@timestamp": "2020-12-07T11:06:07.000Z", + "agent": { + "id": "8a4f500d" + }, + "event": { + "category": "process" + }, + "process": { + "name": "cmd.exe", + "path": "C:\\Windows\\System32\\cmd.exe" + } + }, + "sort": [ + 1607339167000 + ] + } + ] + } +} +---- +// TESTRESPONSE[s/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=/$body.id/] +// TESTRESPONSE[s/"is_partial": true/"is_partial": $body.is_partial/] +// TESTRESPONSE[s/"is_running": true/"is_running": $body.is_running/] +// TESTRESPONSE[s/"took": 2000/"took": $body.took/] +// TESTRESPONSE[s/"_index": "frozen_sec_logs"/"_index": "sec_logs"/] +==== + +[discrete] +[[eql-search-store-async-eql-search]] +=== Change the search retention period + +By default, the EQL search API only stores async searches and their results for +five days. After this period, any ongoing searches or saved results are deleted. + +You can use the `keep_alive` parameter to change the duration of this period. + +[%collapsible] +.*Example* +==== +In the following EQL search API request, the `keep_alive` parameter is `2d` (two +days). This means that if the search becomes async, its results +are stored on the cluster for two days. After two days, the async +search and its results are deleted, even if it's still ongoing. + +[source,console] +---- +GET /sec_logs/_eql/search +{ + "keep_alive": "2d", + "wait_for_completion_timeout": "2s", + "query": """ + process where process.name == "cmd.exe" + """ +} +---- +==== + +[discrete] +[[eql-search-store-sync-eql-search]] +=== Store synchronous EQL searches + +By default, the EQL search API only stores async searches that cannot be +completed within the period set by the `wait_for_completion_timeout` parameter. + +To save the results of searches that complete during this period, set the +`keep_on_completion` parameter to `true`. Note these saved searches are still +subject to the storage retention period set by the `keep_alive` parameter. + +[%collapsible] +.*Example* +==== +In the following EQL search API request, the `keep_on_completion` parameter is +`true`. This means the search results are stored on the cluster, even if +the search completes within the `2s` (two-second) period set by the +`wait_for_completion_timeout` parameter. + +[source,console] +---- +GET /sec_logs/_eql/search +{ + "keep_on_completion": true, + "wait_for_completion_timeout": "2s", + "query": """ + process where process.name == "cmd.exe" + """ +} +---- +==== \ No newline at end of file diff --git a/docs/reference/eql/syntax.asciidoc b/docs/reference/eql/syntax.asciidoc index de11476c64117..299829409c0de 100644 --- a/docs/reference/eql/syntax.asciidoc +++ b/docs/reference/eql/syntax.asciidoc @@ -341,6 +341,84 @@ dots (`.`), hyphens (`-`), or spaces, must be escaped using backticks (+++`+++). `my field` ---- +[discrete] +[[eql-sequences]] +=== Sequences + +You can use EQL sequences to describe and match an ordered series of events. +Each item in a sequence is an event category and event condition, +surrounded by square brackets. Events are listed in ascending chronological +order, with the most recent event listed last. + +[source,eql] +---- +sequence + [ event_category_1 where condition_1 ] + [ event_category_2 where condition_2 ] + ... +---- + +.*Example* +[%collapsible] +==== +The following EQL query matches this series of ordered events: + +. Start with an event with: ++ +-- +* An event category of `file` +* A `file.extension` of `exe` +-- +. Followed by an event with an event category of `process` + +[source,eql] +---- +sequence + [ file where file.extension == "exe" ] + [ process where true ] +---- +==== + +You can use the `by` keyword with sequences to only match events that share the +same field values. If a field value should be shared across all events, you +can use `sequence by`. + +[source,eql] +---- +sequence by field_foo + [ event_category_1 where condition_1 ] by field_baz + [ event_category_2 where condition_2 ] by field_bar + ... +---- + +.*Example* +[%collapsible] +==== +The following sequence uses the `by` keyword to constrain matching events to: + +* Events with the same `user.name` value +* `file` events with a `file.path` value equal to the following `process` + event's `process.path` value. + +[source,eql] +---- +sequence + [ file where file.extension == "exe" ] by user.name, file.path + [ process where true ] by user.name, process.path +---- + +Because the `user.name` field is shared across all events in the sequence, it +can be included using `sequence by`. The following sequence is equivalent to the +prior one. + +[source,eql] +---- +sequence by user.name + [ file where file.extension == "exe" ] by file.path + [ process where true ] by process.path +---- +==== + [discrete] [[eql-functions]] === Functions @@ -394,4 +472,4 @@ file where file.extension in ("exe", "dll") We recommend testing and benchmarking any indexing changes before deploying them in production. See <> and <>. -==== \ No newline at end of file +====