From 8dbb2539d80eae7590190af2aecdd5e4520e46ca Mon Sep 17 00:00:00 2001 From: James Rodewig Date: Mon, 3 Feb 2020 11:57:36 -0500 Subject: [PATCH 01/10] [DOCS] Add EQL syntax page --- docs/reference/eql/index.asciidoc | 4 +- docs/reference/eql/syntax.asciidoc | 224 +++++++++++++++++++++++++++++ 2 files changed, 227 insertions(+), 1 deletion(-) create mode 100644 docs/reference/eql/syntax.asciidoc diff --git a/docs/reference/eql/index.asciidoc b/docs/reference/eql/index.asciidoc index 8c4b0e07ce21b..8de75449607d6 100644 --- a/docs/reference/eql/index.asciidoc +++ b/docs/reference/eql/index.asciidoc @@ -29,6 +29,8 @@ Consider using EQL if you: [[eql-toc]] === In this section -* <> +* <> +* <> include::requirements.asciidoc[] +include::syntax.asciidoc[] diff --git a/docs/reference/eql/syntax.asciidoc b/docs/reference/eql/syntax.asciidoc new file mode 100644 index 0000000000000..05022ddc7f818 --- /dev/null +++ b/docs/reference/eql/syntax.asciidoc @@ -0,0 +1,224 @@ +[role="xpack"] +[testenv="basic"] +[[eql-syntax]] +== EQL syntax reference + +experimental::[] + +[IMPORTANT] +==== +{es} supports a subset of EQL syntax. +==== + +[discrete] +[[eql-basic-syntax]] +=== Basic syntax + +EQL queries require an event type and a matching condition. The `where` keyword connects them. + +[source,eql] +---- +event_type where condition +---- + +For example, the following EQL query matches `process` events with a `name` +field value of `svchost.exe`: + +[source,eql] +---- +process where name == "svchost.exe" +---- + +[discrete] +[[eql-syntax-conditions]] +==== Conditions + +A condition consists of one or more criteria an event must match. +You can specify and combine these criteria using the following operators: + +[discrete] +[[eql-syntax-comparison-operators]] +===== Comparison operators + +[source,eql] +---- +< <= == != >= > +---- + +.*Definitions* +[%collapsible] +==== +`<` (less than):: +Returns `true` if the value to the left of the operator is less than the value +to the right. Otherwise returns `false`. + +`<=` (less than or equal) :: +Returns `true` if the value to the left of the operator is less than or equal to +the value to the right. Otherwise returns `false`. + +`==` (equal):: +Returns `true` if the values to the left and right of the operator are equal. +Otherwise returns `false`. + +`!=` (not equal):: +Returns `true` if the values to the left and right of the operator are not +equal. Otherwise returns `false`. + +`>=` (greater than or equal) :: +Returns `true` if the value to the left of the operator is greater than or equal +to the value to the right. Otherwise returns `false`. + +`>` (greater than):: +Returns `true` if the value to the left of the operator is greater than the +value to the right. Otherwise returns `false`. +==== + +[discrete] +[[eql-syntax-logical-operators]] +===== Logical operators + +[source,eql] +---- +and or not +---- + +.*Definitions* +[%collapsible] +==== +`and`:: +Returns `true` only if the condition to the left and right _both_ return `true`. +Otherwise returns `false. + +`or`:: +Returns `true` if one of the conditions to the left or right `true`. +Otherwise returns `false. + +`not`:: +Returns `true` if the condition to the right is `false`. +==== + +[discrete] +[[eql-syntax-lookup-operators]] +===== Lookup operators + +[source,eql] +---- +user.name in ("Administrator", "SYSTEM", "NETWORK SERVICE") +user.name not in ("Administrator", "SYSTEM", "NETWORK SERVICE") +---- + +.*Definitions* +[%collapsible] +==== +`in`:: +Returns `true` if the value is contained in the provided list. + +`not in`:: +Returns `true` if the value is not contained in the provided list. +==== + +[discrete] +[[eql-syntax-math-operators]] +===== Math operators + +[source,eql] +---- ++ - * / % +---- + +.*Definitions* +[%collapsible] +==== +`+` (add):: +Adds the values to the left and right of the operator. + +`-` (Subtract):: +Subtracts the value to the right of the operator from the value to the left. + +`*` (Subtract):: +Multiplies the values to the left and right of the operator. + +`/` (Divide):: +Divides the value to the left of the operator by the value to the right. + +`%` (modulo):: +Divides the value to the left of the operator by the value to the right. Returns only the remainder. +==== + +[discrete] +[[eql-syntax-strings]] +==== Strings + +Strings are enclosed with double quotes (`"`) or single quotes (`'`). + +[source,eql] +---- +"hello world" +"hello world with 'substring'" +---- + +[discrete] +[[eql-syntax-wildcards]] +===== Wildcards + +You can use the wildcard operator (`*`) within a string to match specific +patterns. + +The following example string matches any value that ends with `.txt`. + +[source,eql] +---- +"*.txt" +---- + +[discrete] +[[eql-syntax-escaped-characters]] +===== Escaped characters + +When used within a string, special characters, such as a carriage return or +double quote (`"`), must be escaped with a preceding `\`. + +[source,eql] +---- +"example \t of \n escaped \r characters" +---- + +.*Escape sequences* +[%collapsible] +==== +[options="header"] +|==== +| Escape sequence | Literal character +|`\n` | A newline (linefeed) character +|`\r` | A carriage return character +|`\t` | A tab character +|`\\` | A backslash (`\`) character +|`\"` | A double quote (`"`) character +|`\'` | A single quote (`'`) character +|`\*` | A wildcard (`*`) character +|==== +==== + +[discrete] +[[eql-syntax-raw-strings]] +===== Raw strings + +Raw strings are preceded by a question mark (`?`), which escapes all special +characters in the string except single quote (`'`) and double quote (`"`). + +[source,eql] +---- +?"String with literal 'slash' \ characters included" +---- + +[discrete] +[[eql-syntax-nested-fields]] +==== Nested fields + +When using dot notation to specify nested object fields, the field name and path +are escaped using backticks (+++`+++). + +[source,eql] +---- +`obj1.name` +---- \ No newline at end of file From eae9c10cc5de12e0822ab837af08bcad2cd50fda Mon Sep 17 00:00:00 2001 From: James Rodewig Date: Mon, 3 Feb 2020 14:46:17 -0500 Subject: [PATCH 02/10] Update docs/reference/eql/syntax.asciidoc Co-Authored-By: Ross Wolf <31489089+rw-access@users.noreply.github.com> --- docs/reference/eql/syntax.asciidoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/reference/eql/syntax.asciidoc b/docs/reference/eql/syntax.asciidoc index 05022ddc7f818..107bc7304a735 100644 --- a/docs/reference/eql/syntax.asciidoc +++ b/docs/reference/eql/syntax.asciidoc @@ -26,7 +26,7 @@ field value of `svchost.exe`: [source,eql] ---- -process where name == "svchost.exe" +process where process.name == "svchost.exe" ---- [discrete] @@ -221,4 +221,4 @@ are escaped using backticks (+++`+++). [source,eql] ---- `obj1.name` ----- \ No newline at end of file +---- From 93ddab1982836201384477c1ca091086d7f701e5 Mon Sep 17 00:00:00 2001 From: James Rodewig Date: Mon, 3 Feb 2020 14:46:42 -0500 Subject: [PATCH 03/10] Update docs/reference/eql/syntax.asciidoc Co-Authored-By: Ross Wolf <31489089+rw-access@users.noreply.github.com> --- docs/reference/eql/syntax.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/eql/syntax.asciidoc b/docs/reference/eql/syntax.asciidoc index 107bc7304a735..21b8d8d3c1461 100644 --- a/docs/reference/eql/syntax.asciidoc +++ b/docs/reference/eql/syntax.asciidoc @@ -142,7 +142,7 @@ Multiplies the values to the left and right of the operator. Divides the value to the left of the operator by the value to the right. `%` (modulo):: -Divides the value to the left of the operator by the value to the right. Returns only the remainder. +Divides the value to the left of the operator by the value to the right. Returns only the remainder. ==== [discrete] From 2827c20aed32602c0182b49cdb32c5fc91aa8862 Mon Sep 17 00:00:00 2001 From: James Rodewig Date: Mon, 3 Feb 2020 14:47:43 -0500 Subject: [PATCH 04/10] Remove escaped wildcard --- docs/reference/eql/syntax.asciidoc | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/reference/eql/syntax.asciidoc b/docs/reference/eql/syntax.asciidoc index 21b8d8d3c1461..91e746d28a73b 100644 --- a/docs/reference/eql/syntax.asciidoc +++ b/docs/reference/eql/syntax.asciidoc @@ -195,7 +195,6 @@ double quote (`"`), must be escaped with a preceding `\`. |`\\` | A backslash (`\`) character |`\"` | A double quote (`"`) character |`\'` | A single quote (`'`) character -|`\*` | A wildcard (`*`) character |==== ==== From e76900ea8e9ab736b1821786424d891efd6ff1cf Mon Sep 17 00:00:00 2001 From: James Rodewig Date: Mon, 3 Feb 2020 14:55:35 -0500 Subject: [PATCH 05/10] Field name correction --- docs/reference/eql/syntax.asciidoc | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/docs/reference/eql/syntax.asciidoc b/docs/reference/eql/syntax.asciidoc index 91e746d28a73b..5a3f87f52b0b4 100644 --- a/docs/reference/eql/syntax.asciidoc +++ b/docs/reference/eql/syntax.asciidoc @@ -211,13 +211,16 @@ characters in the string except single quote (`'`) and double quote (`"`). ---- [discrete] -[[eql-syntax-nested-fields]] -==== Nested fields +[[eql-syntax-non-alpha-field-namess]] +==== Non-alphanumeric field names -When using dot notation to specify nested object fields, the field name and path -are escaped using backticks (+++`+++). +Field names containing non-alphanumeric characters, such as underscores (`_`), +dots (`.`), hyphens (`-`), or spaces, must be escaped using backticks (+++`+++). [source,eql] ---- -`obj1.name` +`my_field` +`my.field` +`my-field` +`my field` ---- From c529f73ff08d43c1a07b1361072b5de4c41944dc Mon Sep 17 00:00:00 2001 From: James Rodewig Date: Mon, 3 Feb 2020 15:26:03 -0500 Subject: [PATCH 06/10] Correct raw string section --- docs/reference/eql/syntax.asciidoc | 22 ++++++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-) diff --git a/docs/reference/eql/syntax.asciidoc b/docs/reference/eql/syntax.asciidoc index 5a3f87f52b0b4..2026160a7d8f5 100644 --- a/docs/reference/eql/syntax.asciidoc +++ b/docs/reference/eql/syntax.asciidoc @@ -176,7 +176,7 @@ The following example string matches any value that ends with `.txt`. ===== Escaped characters When used within a string, special characters, such as a carriage return or -double quote (`"`), must be escaped with a preceding `\`. +double quote (`"`), must be escaped with a preceding backslash (`\`). [source,eql] ---- @@ -202,14 +202,28 @@ double quote (`"`), must be escaped with a preceding `\`. [[eql-syntax-raw-strings]] ===== Raw strings -Raw strings are preceded by a question mark (`?`), which escapes all special -characters in the string except single quote (`'`) and double quote (`"`). +Raw strings are preceded by a question mark (`?`) and treat backslashes (`\`) as +literal characters. [source,eql] ---- -?"String with literal 'slash' \ characters included" +?"String with a literal 'blackslash' \ character included" ---- +You can escape single quotes (`'`) and double quotes (`"`) with a backslash, but +the backslash remains in the resulting string. + +[source,eql] +---- +?"\"" +---- + +[NOTE] +==== +Raw strings cannot contain only a single backslash. Additionally, raw strings +cannot end in an odd number of backslashes. +==== + [discrete] [[eql-syntax-non-alpha-field-namess]] ==== Non-alphanumeric field names From 8097b414be6f45b54b35191ef799d380037581ce Mon Sep 17 00:00:00 2001 From: James Rodewig Date: Mon, 3 Feb 2020 15:39:17 -0500 Subject: [PATCH 07/10] fix anchor --- docs/reference/eql/syntax.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/eql/syntax.asciidoc b/docs/reference/eql/syntax.asciidoc index 2026160a7d8f5..4fc1f4f5c11e8 100644 --- a/docs/reference/eql/syntax.asciidoc +++ b/docs/reference/eql/syntax.asciidoc @@ -225,7 +225,7 @@ cannot end in an odd number of backslashes. ==== [discrete] -[[eql-syntax-non-alpha-field-namess]] +[[eql-syntax-non-alpha-field-names]] ==== Non-alphanumeric field names Field names containing non-alphanumeric characters, such as underscores (`_`), From 3a1021c6a3198384a19bedcf0f11e50d1c3d1ec7 Mon Sep 17 00:00:00 2001 From: James Rodewig Date: Mon, 3 Feb 2020 16:47:48 -0500 Subject: [PATCH 08/10] Update docs/reference/eql/syntax.asciidoc Co-Authored-By: Ross Wolf <31489089+rw-access@users.noreply.github.com> --- docs/reference/eql/syntax.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/eql/syntax.asciidoc b/docs/reference/eql/syntax.asciidoc index 4fc1f4f5c11e8..ccc17bc7ed76b 100644 --- a/docs/reference/eql/syntax.asciidoc +++ b/docs/reference/eql/syntax.asciidoc @@ -21,7 +21,7 @@ EQL queries require an event type and a matching condition. The `where` keyword event_type where condition ---- -For example, the following EQL query matches `process` events with a `name` +For example, the following EQL query matches `process` events with a `process.name` field value of `svchost.exe`: [source,eql] From a4db068dbd8a8750fd79af1b0adaa10904576415 Mon Sep 17 00:00:00 2001 From: James Rodewig Date: Mon, 3 Feb 2020 16:56:35 -0500 Subject: [PATCH 09/10] change wildcard example --- docs/reference/eql/syntax.asciidoc | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/reference/eql/syntax.asciidoc b/docs/reference/eql/syntax.asciidoc index ccc17bc7ed76b..e17631c75ac7f 100644 --- a/docs/reference/eql/syntax.asciidoc +++ b/docs/reference/eql/syntax.asciidoc @@ -162,13 +162,13 @@ Strings are enclosed with double quotes (`"`) or single quotes (`'`). ===== Wildcards You can use the wildcard operator (`*`) within a string to match specific -patterns. - -The following example string matches any value that ends with `.txt`. +patterns. Wildcards are interpreted if they are used with `==` (equal) or `!=` +(not equal): [source,eql] ---- -"*.txt" +field == "example*wildcard" +field != "example*wildcard" ---- [discrete] From 27986af49ac2ab8f81e6237773c7e322123017cf Mon Sep 17 00:00:00 2001 From: James Rodewig Date: Tue, 4 Feb 2020 11:03:47 -0500 Subject: [PATCH 10/10] reword --- docs/reference/eql/syntax.asciidoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/reference/eql/syntax.asciidoc b/docs/reference/eql/syntax.asciidoc index e17631c75ac7f..d4753ff7d7764 100644 --- a/docs/reference/eql/syntax.asciidoc +++ b/docs/reference/eql/syntax.asciidoc @@ -162,8 +162,8 @@ Strings are enclosed with double quotes (`"`) or single quotes (`'`). ===== Wildcards You can use the wildcard operator (`*`) within a string to match specific -patterns. Wildcards are interpreted if they are used with `==` (equal) or `!=` -(not equal): +patterns. You can use wildcards with the `==` (equal) or `!=` (not equal) +operators: [source,eql] ----