Skip to content

[7.x] [DOCS] EQL: Add case-insensitive ~ operator (#68217) #68236

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jan 29, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 35 additions & 11 deletions docs/reference/eql/functions.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,7 @@
<titleabbrev>Function reference</titleabbrev>
++++

{es} supports the following <<eql-functions,EQL functions>>. Most EQL functions
are case-sensitive by default.
{es} supports the following <<eql-functions,EQL functions>>.

[discrete]
[[eql-fn-add]]
Expand Down Expand Up @@ -61,27 +60,31 @@ If using a field as the argument, this parameter supports only
=== `between`

Extracts a substring that's between a provided `left` and `right` text in a
source string. Matching is case-sensitive.
source string. Matching is case-sensitive by default.

*Example*
[source,eql]
----
// file.path = "C:\\Windows\\System32\\cmd.exe"
between(file.path, "system32\\\\", ".exe") // returns "cmd"
between(file.path, "System32\\\\", ".exe") // returns "cmd"
between(file.path, "system32\\\\", ".exe") // returns ""
between(file.path, "workspace\\\\", ".exe") // returns ""

// Make matching case-insensitive
between~(file.path, "system32\\\\", ".exe") // returns "cmd"

// Greedy matching defaults to false.
between(file.path, "\\\\", "\\\\", false) // returns "Windows"

// Sets greedy matching to true
between(file.path, "\\\\", "\\\\", true) // returns "Windows\\System32"

// empty source string
between("", "system32\\\\", ".exe") // returns ""
between("", "System32\\\\", ".exe") // returns ""
between("", "", "") // returns ""

// null handling
between(null, "system32\\\\", ".exe") // returns null
between(null, "System32\\\\", ".exe") // returns null
----

*Syntax*
Expand Down Expand Up @@ -340,15 +343,19 @@ If using a field as the argument, this parameter supports only
=== `endsWith`

Returns `true` if a source string ends with a provided substring. Matching is
case-sensitive.
case-sensitive by default.

*Example*
[source,eql]
----
endsWith("regsvr32.exe", ".exe") // returns true
endsWith("regsvr32.exe", ".EXE") // returns false
endsWith("regsvr32.exe", ".dll") // returns false
endsWith("", "") // returns true

// Make matching case-insensitive
endsWith~("regsvr32.exe", ".EXE") // returns true

// file.name = "regsvr32.exe"
endsWith(file.name, ".exe") // returns true
endsWith(file.name, ".dll") // returns false
Expand Down Expand Up @@ -405,7 +412,7 @@ field data types:
=== `indexOf`

Returns the first position of a provided substring in a source string. Matching
is case-sensitive.
is case-sensitive by default.

If an optional start position is provided, this function returns the first
occurrence of the substring at or after the start position.
Expand All @@ -414,11 +421,16 @@ occurrence of the substring at or after the start position.
[source,eql]
----
// url.domain = "subdomain.example.com"
indexOf(url.domain, "d") // returns 3
indexOf(url.domain, "D") // returns null
indexOf(url.domain, ".") // returns 9
indexOf(url.domain, ".", 9) // returns 9
indexOf(url.domain, ".", 10) // returns 17
indexOf(url.domain, ".", -6) // returns 9

// Make matching case-insensitive
indexOf~(url.domain, "D") // returns 4

// empty strings
indexOf("", "") // returns 0
indexOf(url.domain, "") // returns 0
Expand Down Expand Up @@ -748,15 +760,19 @@ Fields are not supported as arguments.
=== `startsWith`

Returns `true` if a source string begins with a provided substring. Matching is
case-sensitive.
case-sensitive by default.

*Example*
[source,eql]
----
startsWith("regsvr32.exe", "regsvr32") // returns true
startsWith("regsvr32.exe", "Regsvr32") // returns false
startsWith("regsvr32.exe", "explorer") // returns false
startsWith("", "") // returns true

// Make matching case-insensitive
startsWith~("regsvr32.exe", "Regsvr32") // returns true

// process.name = "regsvr32.exe"
startsWith(process.name, "regsvr32") // returns true
startsWith(process.name, "explorer") // returns false
Expand Down Expand Up @@ -848,16 +864,20 @@ If using a field as the argument, this parameter does not support the
=== `stringContains`

Returns `true` if a source string contains a provided substring. Matching is
case-sensitive.
case-sensitive by default.

*Example*
[source,eql]
----
// process.command_line = "start regsvr32.exe"
stringContains(process.command_line, "regsvr32") // returns true
stringContains(process.command_line, "Regsvr32") // returns false
stringContains(process.command_line, "start ") // returns true
stringContains(process.command_line, "explorer") // returns false

// Make matching case-insensitive
stringContains~(process.command_line, "Regsvr32") // returns false

// process.name = "regsvr32.exe"
stringContains(command_line, process.name) // returns true

Expand Down Expand Up @@ -1008,18 +1028,22 @@ If using a field as the argument, this parameter supports only
=== `wildcard`

Returns `true` if a source string matches one or more provided wildcard
expressions. Matching is case-sensitive.
expressions. Matching is case-sensitive by default.

*Example*
[source,eql]
----
// The * wildcard matches zero or more characters.
// process.name = "regsvr32.exe"
wildcard(process.name, "*regsvr32*") // returns true
wildcard(process.name, "*Regsvr32*") // returns false
wildcard(process.name, "*regsvr32*", "*explorer*") // returns true
wildcard(process.name, "*explorer*") // returns false
wildcard(process.name, "*explorer*", "*scrobj*") // returns false

// Make matching case-insensitive
wildcard~(process.name, "*Regsvr32*") // returns true

// The ? wildcard matches exactly one character.
// process.name = "regsvr32.exe"
wildcard(process.name, "regsvr32.e?e") // returns true
Expand Down
38 changes: 31 additions & 7 deletions docs/reference/eql/syntax.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -220,18 +220,28 @@ Returns `true` if the condition to the right is `false`.
[source,eql]
----
user.name in ("Administrator", "SYSTEM", "NETWORK SERVICE")
user.name in~ ("administrator", "system", "network service")
user.name not in ("Administrator", "SYSTEM", "NETWORK SERVICE")
user.name not in~ ("administrator", "system", "network service")
user.name : ("administrator", "system", "network service")
----

`in` (case-sensitive)::
Returns `true` if the value is contained in the provided list. For strings,
matching is case-sensitive.

`in~` (case-insensitive)::
Returns `true` if the value is contained in the provided list. For strings,
matching is case-insensitive.

`not in` (case-sensitive)::
Returns `true` if the value is not contained in the provided list. For strings,
matching is case-sensitive.

`not in~` (case-insensitive)::
Returns `true` if the value is not contained in the provided list. For strings,
matching is case-insensitive.

`:` (case-insensitive)::
Returns `true` if the value is contained in the provided list. Can only be used
to compare strings.
Expand Down Expand Up @@ -627,12 +637,25 @@ until [ process where event.type == "stop" ]
=== Functions

You can use EQL functions to convert data types, perform math, manipulate
strings, and more. Most functions are case-sensitive by default.
strings, and more. For a list of supported functions, see <<eql-function-ref>>.

For a list of supported functions, see <<eql-function-ref>>.
[discrete]
[[eql-case-insensitive-functions]]
=== Case-insensitive functions

Most EQL functions are case-sensitive by default. To make a function
case-insensitive, use the `~` operator after the function name:

[source,eql]
----
stringContains(process.name,".exe") // Matches ".exe" but not ".EXE" or ".Exe"
stringContains~(process.name,".exe") // Matches ".exe", ".EXE", or ".Exe"
----

[discrete]
[[eql-how-functions-impact-search-performance]]
=== How functions impact search performance

[TIP]
====
Using functions in EQL queries can result in slower search speeds. If you
often use functions to transform indexed data, you can speed up search by making
these changes during indexing instead. However, that often means slower index
Expand Down Expand Up @@ -672,7 +695,6 @@ file where file.extension in ("exe", "dll")

We recommend testing and benchmarking any indexing changes before deploying them
in production. See <<tune-for-indexing-speed>> and <<tune-for-search-speed>>.
====

[discrete]
[[eql-pipes]]
Expand Down Expand Up @@ -741,8 +763,10 @@ sub-fields of a `nested` field. However, data streams and indices containing
{es} EQL differs from the {eql-ref}/index.html[Elastic Endgame EQL syntax] as
follows:

* Most operators and functions in {es} EQL are case-sensitive. For
case-insensitive equality comparisons, use the `:` operator.
* Most operators and functions in {es} EQL are case-sensitive by default. For
case-insensitive equality comparisons, use the `:` operator. To make a function
case-insensitive, use the `~` operator after the function name. See
<<eql-case-insensitive-functions>>.

* Comparisons using the `==` and `!=` operators do not expand wildcard
characters. For example, `process_name == "cmd*.exe"` interprets `*` as a
Expand Down