diff --git a/docs/reference/eql/eql-search-api.asciidoc b/docs/reference/eql/eql-search-api.asciidoc index 5cd3d24797927..7e0f63ca70a3f 100644 --- a/docs/reference/eql/eql-search-api.asciidoc +++ b/docs/reference/eql/eql-search-api.asciidoc @@ -168,7 +168,7 @@ If `true`, the request timed out before completion. `hits`:: (object) -Contains returned events and metadata. +Contains matching events and metadata. + .Properties of `hits` [%collapsible%open] @@ -176,7 +176,7 @@ Contains returned events and metadata. `total`:: (object) -Metadata about the number of returned events. +Metadata about the number of matching events. + .Properties of `total` [%collapsible%open] @@ -184,7 +184,7 @@ Metadata about the number of returned events. `value`:: (integer) -Total number of returned events. +Total number of matching events. `relation`:: + @@ -199,29 +199,80 @@ Returned values are: -- ===== +`sequences`:: +(array of objects) +Contains event sequences matching the query. Each object represents a +matching sequence. This parameter is only returned for EQL queries containing +a <>. ++ +.Properties of `sequences` objects +[%collapsible%open] +===== +`join_keys`:: +(array of strings) +Shared field values used to constrain matches in the sequence. These are defined +using the <> in the EQL query syntax. + `events`:: (array of objects) -Contains returned events matching the query. Each object represents a +Contains events matching the query. Each object represents a matching event. + .Properties of `events` objects [%collapsible%open] +====== +`_index`:: +(string) +Name of the index containing the event. + +`_id`:: +(string) +(string) +Unique identifier for the event. +This ID is only unique within the index. + +`_score`:: +(float) +Positive 32-bit floating point number used to determine the relevance of the + event. See <>. + +`_source`:: +(object) +Original JSON body passed for the event at index time. + +`sort`:: +(array) +Integer used as the sort value for the event. ++ +By default, this is the event's <>, converted to milliseconds since the +https://en.wikipedia.org/wiki/Unix_time[Unix epoch]. +====== ===== +[[eql-search-api-response-events]] +`events`:: +(array of objects) +Contains events matching the query. Each object represents a +matching event. ++ +.Properties of `events` objects +[%collapsible%open] +===== `_index`:: (string) -Name of the index containing the returned event. +Name of the index containing the event. `_id`:: (string) (string) -Unique identifier for the returned event. -This ID is only unique within the returned index. +Unique identifier for the event. +This ID is only unique within the index. `_score`:: (float) Positive 32-bit floating point number used to determine the relevance of the -returned event. See <>. + event. See <>. `_source`:: (object) @@ -240,11 +291,14 @@ https://en.wikipedia.org/wiki/Unix_time[Unix epoch]. [[eql-search-api-example]] ==== {api-examples-title} +[[eql-search-api-basic-query-ex]] +==== Basic query example + The following EQL search request searches for events with an `event.category` of `file` that meet the following conditions: * A `file.name` of `cmd.exe` -* An `agent.id` that is _not_ `my_user` +* An `agent.id` other than `my_user` [source,console] ---- @@ -256,9 +310,10 @@ GET my_index/_eql/search } ---- -The API returns the following response. Events in the response are sorted by -<>, converted to milliseconds since -the https://en.wikipedia.org/wiki/Unix_time[Unix epoch], in ascending order. +The API returns the following response. Matching events in the `hits.events` +property are sorted by <>, converted +to milliseconds since the https://en.wikipedia.org/wiki/Unix_time[Unix epoch], +in ascending order. [source,console-result] ---- @@ -331,4 +386,127 @@ the https://en.wikipedia.org/wiki/Unix_time[Unix epoch], in ascending order. } } ---- -// TESTRESPONSE[s/"took": 6/"took": $body.took/] \ No newline at end of file +// TESTRESPONSE[s/"took": 6/"took": $body.took/] + +[[eql-search-api-sequence-ex]] +==== Sequence query example + +The following EQL search request matches a <> of events +that: + +. Start with an event with: ++ +-- +* An `event.category` of `file` +* A `file.name` of `cmd.exe` +* An `agent.id` other than `my_user` +-- +. Followed by an event with: ++ +-- +* An `event.category` of `process` +* A `process.path` that contains the substring `regsvr32` +-- + +These events must also share the same `agent.id` value. + +[source,console] +---- +GET my_index/_eql/search +{ + "query": """ + sequence by agent.id + [ file where file.name == "cmd.exe" and agent.id != "my_user" ] + [ process where stringContains(process.path, "regsvr32") ] + """ +} +---- + +The API returns the following response. The `hits.sequences.join_keys` property +contains the shared `agent.id` value for each matching event. Matching events in +the `hits.sequences.events` property are sorted by +<>, converted to milliseconds since +the https://en.wikipedia.org/wiki/Unix_time[Unix epoch], in ascending order. + +[source,console-result] +---- +{ + "took": 6, + "timed_out": false, + "hits": { + "total": { + "value": 1, + "relation": "eq" + }, + "sequences": [ + { + "join_keys": [ + "8a4f500d" + ], + "events": [ + { + "_index": "my_index", + "_id": "4", + "_score": null, + "_source": { + "@timestamp": "2020-12-07T11:07:08.000Z", + "agent": { + "id": "8a4f500d" + }, + "event": { + "category": "file" + }, + "file": { + "accessed": "2020-12-07T11:07:08.000Z", + "name": "cmd.exe", + "path": "C:\\Windows\\System32\\cmd.exe", + "type": "file", + "size": 16384 + }, + "process": { + "name": "cmd.exe", + "path": "C:\\Windows\\System32\\cmd.exe" + } + }, + "fields": { + "@timestamp": [ + "1607339228000" + ] + }, + "sort": [ + 1607339228000 + ] + }, + { + "_index": "my_index", + "_id": "5", + "_score": null, + "_source": { + "@timestamp": "2020-12-07T11:07:09.000Z", + "agent": { + "id": "8a4f500d" + }, + "event": { + "category": "process" + }, + "process": { + "name": "regsvr32.exe", + "path": "C:\\Windows\\System32\\regsvr32.exe" + } + }, + "fields": { + "@timestamp": [ + "1607339229000" + ] + }, + "sort": [ + 1607339229000 + ] + } + ] + } + ] + } +} +---- +// TESTRESPONSE[s/"took": 6/"took": $body.took/] diff --git a/docs/reference/eql/limitations.asciidoc b/docs/reference/eql/limitations.asciidoc index 122d5ce2fe971..11513574a9226 100644 --- a/docs/reference/eql/limitations.asciidoc +++ b/docs/reference/eql/limitations.asciidoc @@ -37,4 +37,6 @@ queries that contain: * {eql-ref}/pipes.html[Pipes] -* {eql-ref}/sequences.html[Sequences] \ No newline at end of file +* {eql-ref}/sequences.html[State and timespan-related sequence keywords]: +** `with maxspan` +** `until` \ No newline at end of file diff --git a/docs/reference/eql/syntax.asciidoc b/docs/reference/eql/syntax.asciidoc index de11476c64117..299829409c0de 100644 --- a/docs/reference/eql/syntax.asciidoc +++ b/docs/reference/eql/syntax.asciidoc @@ -341,6 +341,84 @@ dots (`.`), hyphens (`-`), or spaces, must be escaped using backticks (+++`+++). `my field` ---- +[discrete] +[[eql-sequences]] +=== Sequences + +You can use EQL sequences to describe and match an ordered series of events. +Each item in a sequence is an event category and event condition, +surrounded by square brackets. Events are listed in ascending chronological +order, with the most recent event listed last. + +[source,eql] +---- +sequence + [ event_category_1 where condition_1 ] + [ event_category_2 where condition_2 ] + ... +---- + +.*Example* +[%collapsible] +==== +The following EQL query matches this series of ordered events: + +. Start with an event with: ++ +-- +* An event category of `file` +* A `file.extension` of `exe` +-- +. Followed by an event with an event category of `process` + +[source,eql] +---- +sequence + [ file where file.extension == "exe" ] + [ process where true ] +---- +==== + +You can use the `by` keyword with sequences to only match events that share the +same field values. If a field value should be shared across all events, you +can use `sequence by`. + +[source,eql] +---- +sequence by field_foo + [ event_category_1 where condition_1 ] by field_baz + [ event_category_2 where condition_2 ] by field_bar + ... +---- + +.*Example* +[%collapsible] +==== +The following sequence uses the `by` keyword to constrain matching events to: + +* Events with the same `user.name` value +* `file` events with a `file.path` value equal to the following `process` + event's `process.path` value. + +[source,eql] +---- +sequence + [ file where file.extension == "exe" ] by user.name, file.path + [ process where true ] by user.name, process.path +---- + +Because the `user.name` field is shared across all events in the sequence, it +can be included using `sequence by`. The following sequence is equivalent to the +prior one. + +[source,eql] +---- +sequence by user.name + [ file where file.extension == "exe" ] by file.path + [ process where true ] by process.path +---- +==== + [discrete] [[eql-functions]] === Functions @@ -394,4 +472,4 @@ file where file.extension in ("exe", "dll") We recommend testing and benchmarking any indexing changes before deploying them in production. See <> and <>. -==== \ No newline at end of file +====