From b29d6f726e2fea9d7bddf9445c618fa61d00d977 Mon Sep 17 00:00:00 2001 From: James Rodewig Date: Wed, 13 May 2020 15:10:00 -0400 Subject: [PATCH 1/9] [DOCS] EQL: Document sequences --- docs/reference/eql/eql-search-api.asciidoc | 204 +++++++++++++++++++-- docs/reference/eql/limitations.asciidoc | 4 +- docs/reference/eql/syntax.asciidoc | 78 ++++++++ 3 files changed, 273 insertions(+), 13 deletions(-) diff --git a/docs/reference/eql/eql-search-api.asciidoc b/docs/reference/eql/eql-search-api.asciidoc index 5cd3d24797927..0405edf3d1bca 100644 --- a/docs/reference/eql/eql-search-api.asciidoc +++ b/docs/reference/eql/eql-search-api.asciidoc @@ -168,7 +168,7 @@ If `true`, the request timed out before completion. `hits`:: (object) -Contains returned events and metadata. +Contains matching events and metadata. + .Properties of `hits` [%collapsible%open] @@ -176,7 +176,7 @@ Contains returned events and metadata. `total`:: (object) -Metadata about the number of returned events. +Metadata about the number of matching events. + .Properties of `total` [%collapsible%open] @@ -184,7 +184,7 @@ Metadata about the number of returned events. `value`:: (integer) -Total number of returned events. +Total number of matching events. `relation`:: + @@ -199,29 +199,82 @@ Returned values are: -- ===== +`sequences`:: +(array of objects) +Contains event sequences matching the query. Each object represents a +matching sequence. This parameter is only returned for EQL queries containing +a <>. ++ +.Properties of `sequences` objects +[%collapsible%open] +===== +`join_keys`:: +(array of strings) +Shared field values used to constrain matches in the sequence. These are defined +using the <> in the EQL query syntax. This +parameter is only returned for EQL queries containing a +<>. + `events`:: (array of objects) -Contains returned events matching the query. Each object represents a +Contains events matching the query. Each object represents a matching event. + .Properties of `events` objects [%collapsible%open] +====== +`_index`:: +(string) +Name of the index containing the event. + +`_id`:: +(string) +(string) +Unique identifier for the event. +This ID is only unique within the index. + +`_score`:: +(float) +Positive 32-bit floating point number used to determine the relevance of the + event. See <>. + +`_source`:: +(object) +Original JSON body passed for the event at index time. + +`sort`:: +(array) +Integer used as the sort value for the event. ++ +By default, this is the event's <>, converted to milliseconds since the +https://en.wikipedia.org/wiki/Unix_time[Unix epoch]. +====== ===== +[[eql-search-api-response-events]] +`events`:: +(array of objects) +Contains events matching the query. Each object represents a +matching event. ++ +.Properties of `events` objects +[%collapsible%open] +===== `_index`:: (string) -Name of the index containing the returned event. +Name of the index containing the event. `_id`:: (string) (string) -Unique identifier for the returned event. -This ID is only unique within the returned index. +Unique identifier for the event. +This ID is only unique within the index. `_score`:: (float) Positive 32-bit floating point number used to determine the relevance of the -returned event. See <>. + event. See <>. `_source`:: (object) @@ -240,11 +293,14 @@ https://en.wikipedia.org/wiki/Unix_time[Unix epoch]. [[eql-search-api-example]] ==== {api-examples-title} +[[eql-search-api-basic-query-ex]] +==== Basic query example + The following EQL search request searches for events with an `event.category` of `file` that meet the following conditions: * A `file.name` of `cmd.exe` -* An `agent.id` that is _not_ `my_user` +* An `agent.id` other than `my_user` [source,console] ---- @@ -256,9 +312,10 @@ GET my_index/_eql/search } ---- -The API returns the following response. Events in the response are sorted by -<>, converted to milliseconds since -the https://en.wikipedia.org/wiki/Unix_time[Unix epoch], in ascending order. +The API returns the following response. Matching events in the `hits.events` +property are sorted by <>, converted +to milliseconds since the https://en.wikipedia.org/wiki/Unix_time[Unix epoch], +in ascending order. [source,console-result] ---- @@ -331,4 +388,127 @@ the https://en.wikipedia.org/wiki/Unix_time[Unix epoch], in ascending order. } } ---- +// TESTRESPONSE[s/"took": 6/"took": $body.took/] + +[[eql-search-api-sequence-ex]] +==== Sequence query example + +The following EQL search request matches a <> of events +that: + +. Start with an event with: ++ +-- +* An `event.category` of `file` +* A `file.name` of `cmd.exe` +* An `agent.id` other than `my_user` +-- +. Followed by an event with: ++ +-- +* An `event.category` of `process` +* A `process.path` of `regsvr32` +-- + +These events must also share the same `agent.id` value. + +[source,console] +---- +GET my_index/_eql/search +{ + "query": """ + sequence by agent.id + [ file where (file.name == "cmd.exe" and agent.id != "my_user") ] + [ process where stringContains(process.path, "regsvr32") ] + """ +} +---- + +The API returns the following response. The `hits.sequences.join_keys` property +contains the shared `agent.id` value for each matching event. Matching events in +the `hits.sequences.events` property are sorted by +<>, converted to milliseconds since +the https://en.wikipedia.org/wiki/Unix_time[Unix epoch], in ascending order. + +[source,console-result] +---- +{ + "took": 6, + "timed_out": false, + "hits": { + "total": { + "value": 1, + "relation": "eq" + }, + "sequences": [ + { + "join_keys": [ + "8a4f500d" + ], + "events": [ + { + "_index": "my_index", + "_id": "4", + "_score": null, + "_source": { + "@timestamp": "2020-12-07T11:07:08.000Z", + "agent": { + "id": "8a4f500d" + }, + "event": { + "category": "file" + }, + "file": { + "accessed": "2020-12-07T11:07:08.000Z", + "name": "cmd.exe", + "path": "C:\\Windows\\System32\\cmd.exe", + "type": "file", + "size": 16384 + }, + "process": { + "name": "cmd.exe", + "path": "C:\\Windows\\System32\\cmd.exe" + } + }, + "fields": { + "@timestamp": [ + "1607339228000" + ] + }, + "sort": [ + 1607339228000 + ] + }, + { + "_index": "my_index", + "_id": "5", + "_score": null, + "_source": { + "@timestamp": "2020-12-07T11:07:09.000Z", + "agent": { + "id": "8a4f500d" + }, + "event": { + "category": "process" + }, + "process": { + "name": "regsvr32.exe", + "path": "C:\\Windows\\System32\\regsvr32.exe" + } + }, + "fields": { + "@timestamp": [ + "1607339229000" + ] + }, + "sort": [ + 1607339229000 + ] + } + ] + } + ] + } +} +---- // TESTRESPONSE[s/"took": 6/"took": $body.took/] \ No newline at end of file diff --git a/docs/reference/eql/limitations.asciidoc b/docs/reference/eql/limitations.asciidoc index 122d5ce2fe971..11513574a9226 100644 --- a/docs/reference/eql/limitations.asciidoc +++ b/docs/reference/eql/limitations.asciidoc @@ -37,4 +37,6 @@ queries that contain: * {eql-ref}/pipes.html[Pipes] -* {eql-ref}/sequences.html[Sequences] \ No newline at end of file +* {eql-ref}/sequences.html[State and timespan-related sequence keywords]: +** `with maxspan` +** `until` \ No newline at end of file diff --git a/docs/reference/eql/syntax.asciidoc b/docs/reference/eql/syntax.asciidoc index de11476c64117..c66510a6be3cb 100644 --- a/docs/reference/eql/syntax.asciidoc +++ b/docs/reference/eql/syntax.asciidoc @@ -341,6 +341,84 @@ dots (`.`), hyphens (`-`), or spaces, must be escaped using backticks (+++`+++). `my field` ---- +[discrete] +[[eql-sequences]] +=== Sequences + +You can use EQL sequences to describe and match an ordered series of events. +Each item in a sequence is an event category and event condition, +surrounded by square brackets. Events are listed in ascending chronological +order, with the most recent event listed last. + +[source,eql] +---- +sequence + [ event_category_1 where condition_1 ] + [ event_category_2 where condition_2 ] + ... +---- + +.*Example* +[%collapsible] +==== +The following EQL query matches this series of ordered events: + +. Start with an event with: ++ +-- +* An event category of `file` +* A `file.name` that ends with `.exe` +-- +. Followed by an event with an event category of `process` + +[source,eql] +---- +sequence + [ file where file.name == "*.exe" ] + [ process where true ] +---- +==== + +You can use the `by` keyword with sequences to only match events that share the +same field values. If a field value should be shared across all events, you +can use `sequence by`. + +[source,eql] +---- +sequence by field_foo + [ event_category_1 where condition_1 ] by field_baz + [ event_category_2 where condition_2 ] by field_bar + ... +---- + +.*Example* +[%collapsible] +==== +The following sequence uses the `by` keyword to constrain matching events to: + +* Events with same `user.name` value +* `file` events with a `file.path` value equal to the following `process` + event's `process.path` value. + +[source,eql] +---- +sequence + [ file where file.name == "*.exe" ] by user.name, file.path + [ process where true ] by user.name, process.path +---- + +Because the `user.name` field is shared across all events in the sequence, it +can be included using `sequence by`. The following sequence is equivalent to the +prior one. + +[source,eql] +---- +sequence by user.name + [ file where file.name == "*.exe" ] by file.path + [ process where true ] by process.path +---- +==== + [discrete] [[eql-functions]] === Functions From 08bea0f58e77d653d35c1f751beb894b7cfecd1e Mon Sep 17 00:00:00 2001 From: James Rodewig Date: Wed, 13 May 2020 17:35:45 -0400 Subject: [PATCH 2/9] Update docs/reference/eql/eql-search-api.asciidoc Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com> --- docs/reference/eql/eql-search-api.asciidoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/reference/eql/eql-search-api.asciidoc b/docs/reference/eql/eql-search-api.asciidoc index 0405edf3d1bca..0deb4076f4b1f 100644 --- a/docs/reference/eql/eql-search-api.asciidoc +++ b/docs/reference/eql/eql-search-api.asciidoc @@ -418,7 +418,7 @@ GET my_index/_eql/search { "query": """ sequence by agent.id - [ file where (file.name == "cmd.exe" and agent.id != "my_user") ] + [ file where file.name == "cmd.exe" and agent.id != "my_user" ] [ process where stringContains(process.path, "regsvr32") ] """ } @@ -511,4 +511,4 @@ the https://en.wikipedia.org/wiki/Unix_time[Unix epoch], in ascending order. } } ---- -// TESTRESPONSE[s/"took": 6/"took": $body.took/] \ No newline at end of file +// TESTRESPONSE[s/"took": 6/"took": $body.took/] From f22873a40870a5cde83d0b6e781104b6e8fdcd3d Mon Sep 17 00:00:00 2001 From: James Rodewig Date: Wed, 13 May 2020 17:36:30 -0400 Subject: [PATCH 3/9] Update docs/reference/eql/syntax.asciidoc Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com> --- docs/reference/eql/syntax.asciidoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/reference/eql/syntax.asciidoc b/docs/reference/eql/syntax.asciidoc index c66510a6be3cb..4798b0c6b61af 100644 --- a/docs/reference/eql/syntax.asciidoc +++ b/docs/reference/eql/syntax.asciidoc @@ -414,7 +414,7 @@ prior one. [source,eql] ---- sequence by user.name - [ file where file.name == "*.exe" ] by file.path + [ file where file.extension == "exe" ] by file.path [ process where true ] by process.path ---- ==== @@ -472,4 +472,4 @@ file where file.extension in ("exe", "dll") We recommend testing and benchmarking any indexing changes before deploying them in production. See <> and <>. -==== \ No newline at end of file +==== From a5c2d70c8975563f60a6847e9689b689a5348914 Mon Sep 17 00:00:00 2001 From: James Rodewig Date: Wed, 13 May 2020 17:36:55 -0400 Subject: [PATCH 4/9] Update docs/reference/eql/syntax.asciidoc Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com> --- docs/reference/eql/syntax.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/eql/syntax.asciidoc b/docs/reference/eql/syntax.asciidoc index 4798b0c6b61af..f504094f7620d 100644 --- a/docs/reference/eql/syntax.asciidoc +++ b/docs/reference/eql/syntax.asciidoc @@ -403,7 +403,7 @@ The following sequence uses the `by` keyword to constrain matching events to: [source,eql] ---- sequence - [ file where file.name == "*.exe" ] by user.name, file.path + [ file where file.extension == "exe" ] by user.name, file.path [ process where true ] by user.name, process.path ---- From 82c326f3d7e0811f7147f1be503cf72b2b164197 Mon Sep 17 00:00:00 2001 From: James Rodewig Date: Wed, 13 May 2020 17:43:19 -0400 Subject: [PATCH 5/9] Update docs/reference/eql/syntax.asciidoc Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com> --- docs/reference/eql/syntax.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/eql/syntax.asciidoc b/docs/reference/eql/syntax.asciidoc index f504094f7620d..7b15d89e742bd 100644 --- a/docs/reference/eql/syntax.asciidoc +++ b/docs/reference/eql/syntax.asciidoc @@ -374,7 +374,7 @@ The following EQL query matches this series of ordered events: [source,eql] ---- sequence - [ file where file.name == "*.exe" ] + [ file where file.extension == "exe" ] [ process where true ] ---- ==== From aa174ccc068b0f907b4d62b8a86eb083250908de Mon Sep 17 00:00:00 2001 From: James Rodewig Date: Wed, 13 May 2020 17:43:27 -0400 Subject: [PATCH 6/9] Update docs/reference/eql/syntax.asciidoc Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com> --- docs/reference/eql/syntax.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/eql/syntax.asciidoc b/docs/reference/eql/syntax.asciidoc index 7b15d89e742bd..4e19de0c8b3d5 100644 --- a/docs/reference/eql/syntax.asciidoc +++ b/docs/reference/eql/syntax.asciidoc @@ -367,7 +367,7 @@ The following EQL query matches this series of ordered events: + -- * An event category of `file` -* A `file.name` that ends with `.exe` +* A `file.extension` of `exe` -- . Followed by an event with an event category of `process` From 1c3073f05e733f0b524a45beb387f2876245f0ef Mon Sep 17 00:00:00 2001 From: James Rodewig Date: Wed, 13 May 2020 17:44:30 -0400 Subject: [PATCH 7/9] typo --- docs/reference/eql/syntax.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/eql/syntax.asciidoc b/docs/reference/eql/syntax.asciidoc index 4e19de0c8b3d5..299829409c0de 100644 --- a/docs/reference/eql/syntax.asciidoc +++ b/docs/reference/eql/syntax.asciidoc @@ -396,7 +396,7 @@ sequence by field_foo ==== The following sequence uses the `by` keyword to constrain matching events to: -* Events with same `user.name` value +* Events with the same `user.name` value * `file` events with a `file.path` value equal to the following `process` event's `process.path` value. From f593aa667a402372ca48dc8f2282e979736bfa86 Mon Sep 17 00:00:00 2001 From: James Rodewig Date: Wed, 13 May 2020 17:46:07 -0400 Subject: [PATCH 8/9] Update docs/reference/eql/eql-search-api.asciidoc Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com> --- docs/reference/eql/eql-search-api.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/eql/eql-search-api.asciidoc b/docs/reference/eql/eql-search-api.asciidoc index 0deb4076f4b1f..21ab41380b2e9 100644 --- a/docs/reference/eql/eql-search-api.asciidoc +++ b/docs/reference/eql/eql-search-api.asciidoc @@ -407,7 +407,7 @@ that: + -- * An `event.category` of `process` -* A `process.path` of `regsvr32` +* A `process.path` that contains the substring `regsvr32` -- These events must also share the same `agent.id` value. From adbe73b14cdf4fb9217b0b3722232b38905937a6 Mon Sep 17 00:00:00 2001 From: James Rodewig Date: Thu, 14 May 2020 09:58:45 -0400 Subject: [PATCH 9/9] remove redundant sentence --- docs/reference/eql/eql-search-api.asciidoc | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/reference/eql/eql-search-api.asciidoc b/docs/reference/eql/eql-search-api.asciidoc index 21ab41380b2e9..7e0f63ca70a3f 100644 --- a/docs/reference/eql/eql-search-api.asciidoc +++ b/docs/reference/eql/eql-search-api.asciidoc @@ -211,9 +211,7 @@ a <>. `join_keys`:: (array of strings) Shared field values used to constrain matches in the sequence. These are defined -using the <> in the EQL query syntax. This -parameter is only returned for EQL queries containing a -<>. +using the <> in the EQL query syntax. `events`:: (array of objects)