Skip to content

[DOCS] EQL/SQL: Document runtime_fields parameter #71487

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Apr 19, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/reference/eql/eql-search-api.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -294,6 +294,8 @@ command].
NOTE: This parameter may change the set of returned hits. However, it does not
change the sort order of hits in the response.

include::{es-repo-dir}/search/search.asciidoc[tag=runtime-mappings-def]

`size`::
(Optional, integer or float)
For <<eql-basic-syntax,basic queries>>, the maximum number of matching events to
Expand Down
71 changes: 70 additions & 1 deletion docs/reference/eql/eql.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -442,7 +442,7 @@ GET /my-data-stream/_eql/search?filter_path=-hits.events._source
"process.*", <1>
{
"field": "@timestamp", <2>
"format": "epoch_millis"
"format": "epoch_millis"
}
]
}
Expand Down Expand Up @@ -532,6 +532,75 @@ The values are returned as a flat list in the `fields` section of each hit:
// TESTRESPONSE[s/"_id": "OQmfCaduce8zoHT93o4H"/"_id": $body.hits.events.0._id/]
// TESTRESPONSE[s/"_id": "xLkCaj4EujzdNSxfYLbO"/"_id": $body.hits.events.1._id/]

[discrete]
[[eql-use-runtime-fields]]
=== Use runtime fields

Use the `runtime_mappings` parameter to extract and create <<runtime,runtime
fields>> during a search. Use the `fields` parameter to include runtime fields
in the response.

The following search creates a `day_of_week` runtime field from the `@timestamp`
and returns it in the response.

[source,console]
----
GET /my-data-stream/_eql/search?filter_path=-hits.events._source
{
"runtime_mappings": {
"day_of_week": {
"type": "keyword",
"script": "emit(doc['@timestamp'].value.dayOfWeekEnum.toString())"
}
},
"query": """
process where process.name == "regsvr32.exe"
""",
"fields": [
"@timestamp",
"day_of_week"
]
}
----
// TEST[setup:sec_logs]

The API returns:

[source,console-result]
----
{
"is_partial": false,
"is_running": false,
"took": 60,
"timed_out": false,
"hits": {
"total": {
"value": 2,
"relation": "eq"
},
"events": [
{
"_index": ".ds-my-data-stream-2099.12.07-000001",
"_id": "OQmfCaduce8zoHT93o4H",
"fields": {
"@timestamp": [
"2099-12-07T11:07:09.000Z"
],
"day_of_week": [
"MONDAY"
]
}
},
...
]
}
}
----
// TESTRESPONSE[s/"took": 60/"took": $body.took/]
// TESTRESPONSE[s/"_index": ".ds-my-data-stream-2099.12.07-000001"/"_index": $body.hits.events.0._index/]
// TESTRESPONSE[s/"_id": "OQmfCaduce8zoHT93o4H"/"_id": $body.hits.events.0._id/]
// TESTRESPONSE[s/\.\.\./$body.hits.events.1/]

[discrete]
[[specify-a-timestamp-or-event-category-field]]
=== Specify a timestamp or event category field
Expand Down
27 changes: 19 additions & 8 deletions docs/reference/search/search.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -428,29 +428,37 @@ Period of time used to extend the life of the PIT.
<<query-dsl,Query DSL>>.

[[search-api-body-runtime]]
// tag::runtime-mappings-def[]
`runtime_mappings`::
(Optional, object)
Defines a <<runtime-search-request,runtime field>> in the search request that
exist only as part of the query. Fields defined in the search request take
precedence over fields defined with the same name in the index mappings.
(Optional, object of objects)
Defines one or more <<runtime-search-request,runtime fields>> in the search
request. These fields take precedence over mapped fields with the same name.
+
.Properties of `runtime_mappings` objects
[%collapsible%open]
====

`<field-name>`::
(Required, object)
Configuration for the runtime field. The key is the field name.
+
.Properties of `<field-name>`
[%collapsible%open]
=====

`type`::
(Required, string)
<<mapping-types,Data type>> of the object, which can be any of the following:
<<mapping-types,Field type>>, which can be any of the following:
+
include::{es-ref-dir}/mapping/runtime.asciidoc[tag=runtime-data-types]

`script`::
(Optional, string)
<<modules-scripting-using,Painless script>> that is executed at query time. The
<<modules-scripting-using,Painless script>> executed at query time. The
script has access to the entire context of a document, including the original
`_source` and any mapped fields plus their values.
+
Your script must include `emit` to return calculated values. For
example:
This script must include `emit` to return calculated values. For example:
+
[source,js]
----
Expand All @@ -459,7 +467,10 @@ example:
}
----
// NOTCONSOLE

=====
====
// end::runtime-mappings-def[]

[[request-body-search-seq-no-primary-term]]
`seq_no_primary_term`::
Expand Down
46 changes: 45 additions & 1 deletion docs/reference/sql/endpoints/rest.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
* <<sql-rest-filtering>>
* <<sql-rest-columnar>>
* <<sql-rest-params>>
* <<sql-runtime-fields>>
* <<sql-rest-fields>>

[[sql-rest-overview]]
Expand Down Expand Up @@ -522,6 +523,44 @@ POST /_sql?format=txt
[IMPORTANT]
The recommended way of passing values to a query is with question mark placeholders, to avoid any attempts of hacking or SQL injection.

[[sql-runtime-fields]]
=== Use runtime fields

Use the `runtime_mappings` parameter to extract and create <<runtime,runtime
fields>>, or columns, from existing ones during a search.

The following search creates a `release_day_of_week` runtime field from
`release_date` and returns it in the response.

[source,console]
----
POST _sql?format=txt
{
"runtime_mappings": {
"release_day_of_week": {
"type": "keyword",
"script": """
emit(doc['release_date'].value.dayOfWeekEnum.toString())
"""
}
},
"query": """
SELECT * FROM library WHERE page_count > 300 AND author = 'Frank Herbert'
"""
}
----
// TEST[setup:library]

The API returns:

[source,txt]
----
author | name | page_count | release_date |release_day_of_week
---------------+---------------+---------------+------------------------+-------------------
Frank Herbert |Dune |604 |1965-06-01T00:00:00.000Z|TUESDAY
----
// TESTRESPONSE[non_json]

[[sql-rest-fields]]
=== Supported REST parameters

Expand All @@ -530,7 +569,7 @@ the request time-outs or localization information (such as timezone).

The table below lists the supported parameters:

[cols="^m,^m,^5"]
[cols="<m,<m,<5"]

|===

Expand Down Expand Up @@ -579,6 +618,11 @@ More information available https://docs.oracle.com/javase/8/docs/api/java/time/Z
|none
|Optional list of parameters to replace question mark (`?`) placeholders inside the query.

|runtime_mappings
|none
|Defines one or more <<runtime-search-request,runtime fields>> in the search
request. These fields take precedence over mapped fields with the same name.

|===

Do note that most parameters (outside the timeout and `columnar` ones) make sense only during the initial query - any follow-up pagination request only requires the `cursor` parameter as explained in the <<sql-pagination, pagination>> chapter.
Expand Down