Skip to content

Commit 34b13c2

Browse files
romseygeekromseygeek
committed
Add index-time scripts to date field mapper (#71633)
This commit allows you to set 'script' and 'on_script_error' parameters on date field mappers, meaning that runtime date fields can be made indexed simply by moving their definitions from the runtime section of the mappings to the properties section.
1 parent 55e003d commit 34b13c2

File tree

15 files changed

+435
-47
lines changed

15 files changed

+435
-47
lines changed

‎docs/reference/mapping/types/date.asciidoc

Lines changed: 25 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -137,7 +137,8 @@ The following parameters are accepted by `date` fields:
137137
<<ignore-malformed,`ignore_malformed`>>::
138138

139139
If `true`, malformed numbers are ignored. If `false` (default), malformed
140-
numbers throw an exception and reject the whole document.
140+
numbers throw an exception and reject the whole document. Note that this
141+
cannot be set if the `script` parameter is used.
141142

142143
<<mapping-index,`index`>>::
143144

@@ -147,7 +148,29 @@ The following parameters are accepted by `date` fields:
147148

148149
Accepts a date value in one of the configured +format+'s as the field
149150
which is substituted for any explicit `null` values. Defaults to `null`,
150-
which means the field is treated as missing.
151+
which means the field is treated as missing. Note that this cannot be
152+
set of the `script` parameter is used.
153+
154+
155+
`on_script_error`::
156+
157+
Defines what to do if the script defined by the `script` parameter
158+
throws an error at indexing time. Accepts `reject` (default), which
159+
will cause the entire document to be rejected, and `ignore`, which
160+
will register the field in the document's
161+
<<mapping-ignored-field,`_ignored`>> metadata field and continue
162+
indexing. This parameter can only be set if the `script` field is
163+
also set.
164+
165+
`script`::
166+
167+
If this parameter is set, then the field will index values generated
168+
by this script, rather than reading the values directly from the
169+
source. If a value is set for this field on the input document, then
170+
the document will be rejected with an error.
171+
Scripts are in the same format as their
172+
<<runtime-mapping-fields,runtime equivalent>>, and should emit
173+
long-valued timestamps.
151174

152175
<<mapping-store,`store`>>::
153176

‎modules/runtime-fields-common/src/yamlRestTest/resources/rest-api-spec/test/runtime_fields/43_date_calculated_at_index.yml

Lines changed: 178 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,178 @@
1+
---
2+
setup:
3+
- do:
4+
indices.create:
5+
index: sensor
6+
body:
7+
settings:
8+
number_of_shards: 1
9+
number_of_replicas: 0
10+
mappings:
11+
properties:
12+
tomorrow:
13+
type: date
14+
script:
15+
source: |
16+
for (def dt : doc['timestamp']) {
17+
emit(dt.plus(params.days, ChronoUnit.DAYS).toEpochMilli());
18+
}
19+
params:
20+
days: 1
21+
# Test fetching from _source and parsing
22+
tomorrow_from_source:
23+
type: date
24+
script:
25+
source: |
26+
Instant instant = Instant.ofEpochMilli(parse(params._source.timestamp));
27+
ZonedDateTime dt = ZonedDateTime.ofInstant(instant, ZoneId.of("UTC"));
28+
emit(dt.plus(1, ChronoUnit.DAYS).toEpochMilli());
29+
# Test returning millis
30+
the_past:
31+
type: date
32+
script:
33+
source: |
34+
for (def dt : doc['timestamp']) {
35+
emit(dt.toInstant().toEpochMilli() - 1000);
36+
}
37+
# Test fetching many values
38+
all_week:
39+
type: date
40+
script:
41+
source: |
42+
for (def dt : doc['timestamp']) {
43+
for (int i = 0; i < 7; i++) {
44+
emit(dt.plus(i, ChronoUnit.DAYS).toEpochMilli());
45+
}
46+
}
47+
# Test format parameter
48+
formatted_tomorrow:
49+
type: date
50+
format: yyyy-MM-dd
51+
script: |
52+
for (def dt : doc['timestamp']) {
53+
emit(dt.plus(1, ChronoUnit.DAYS).toEpochMilli());
54+
}
55+
timestamp:
56+
type: date
57+
temperature:
58+
type: long
59+
voltage:
60+
type: double
61+
node:
62+
type: keyword
63+
64+
- do:
65+
bulk:
66+
index: sensor
67+
refresh: true
68+
body: |
69+
{"index":{}}
70+
{"timestamp": 1516729294000, "temperature": 200, "voltage": 5.2, "node": "a"}
71+
{"index":{}}
72+
{"timestamp": 1516642894000, "temperature": 201, "voltage": 5.8, "node": "b"}
73+
{"index":{}}
74+
{"timestamp": 1516556494000, "temperature": 202, "voltage": 5.1, "node": "a"}
75+
{"index":{}}
76+
{"timestamp": 1516470094000, "temperature": 198, "voltage": 5.6, "node": "b"}
77+
{"index":{}}
78+
{"timestamp": 1516383694000, "temperature": 200, "voltage": 4.2, "node": "c"}
79+
{"index":{}}
80+
{"timestamp": "2018-01-18T17:41:34.000Z", "temperature": 202, "voltage": 4.0, "node": "c"}
81+
82+
---
83+
"get mapping":
84+
- do:
85+
indices.get_mapping:
86+
index: sensor
87+
- match: {sensor.mappings.properties.tomorrow.type: date }
88+
- match:
89+
sensor.mappings.properties.tomorrow.script.source: |
90+
for (def dt : doc['timestamp']) {
91+
emit(dt.plus(params.days, ChronoUnit.DAYS).toEpochMilli());
92+
}
93+
- match: {sensor.mappings.properties.tomorrow.script.params: {days: 1} }
94+
- match: {sensor.mappings.properties.tomorrow.script.lang: painless }
95+
96+
- match: {sensor.mappings.properties.formatted_tomorrow.type: date }
97+
- match:
98+
sensor.mappings.properties.formatted_tomorrow.script.source: |
99+
for (def dt : doc['timestamp']) {
100+
emit(dt.plus(1, ChronoUnit.DAYS).toEpochMilli());
101+
}
102+
- match: {sensor.mappings.properties.formatted_tomorrow.script.lang: painless }
103+
- match: {sensor.mappings.properties.formatted_tomorrow.format: yyyy-MM-dd }
104+
105+
---
106+
"fetch fields":
107+
- do:
108+
search:
109+
index: sensor
110+
body:
111+
sort: timestamp
112+
fields: [tomorrow, tomorrow_from_source, the_past, all_week, formatted_tomorrow]
113+
- match: {hits.total.value: 6}
114+
- match: {hits.hits.0.fields.tomorrow: ["2018-01-19T17:41:34.000Z"] }
115+
- match: {hits.hits.0.fields.tomorrow_from_source: ["2018-01-19T17:41:34.000Z"] }
116+
- match: {hits.hits.0.fields.the_past: ["2018-01-18T17:41:33.000Z"] }
117+
- match:
118+
hits.hits.0.fields.all_week:
119+
- 2018-01-18T17:41:34.000Z
120+
- 2018-01-19T17:41:34.000Z
121+
- 2018-01-20T17:41:34.000Z
122+
- 2018-01-21T17:41:34.000Z
123+
- 2018-01-22T17:41:34.000Z
124+
- 2018-01-23T17:41:34.000Z
125+
- 2018-01-24T17:41:34.000Z
126+
- match: {hits.hits.0.fields.formatted_tomorrow: [2018-01-19] }
127+
128+
---
129+
"docvalue_fields":
130+
- do:
131+
search:
132+
index: sensor
133+
body:
134+
sort: timestamp
135+
docvalue_fields: [tomorrow, tomorrow_from_source, the_past, all_week, formatted_tomorrow]
136+
- match: {hits.total.value: 6}
137+
- match: {hits.hits.0.fields.tomorrow: ["2018-01-19T17:41:34.000Z"] }
138+
- match: {hits.hits.0.fields.tomorrow_from_source: ["2018-01-19T17:41:34.000Z"] }
139+
- match: {hits.hits.0.fields.the_past: ["2018-01-18T17:41:33.000Z"] }
140+
- match:
141+
hits.hits.0.fields.all_week:
142+
- 2018-01-18T17:41:34.000Z
143+
- 2018-01-19T17:41:34.000Z
144+
- 2018-01-20T17:41:34.000Z
145+
- 2018-01-21T17:41:34.000Z
146+
- 2018-01-22T17:41:34.000Z
147+
- 2018-01-23T17:41:34.000Z
148+
- 2018-01-24T17:41:34.000Z
149+
- match: {hits.hits.0.fields.formatted_tomorrow: [2018-01-19] }
150+
151+
---
152+
"terms agg":
153+
- do:
154+
search:
155+
index: sensor
156+
body:
157+
aggs:
158+
v10:
159+
terms:
160+
field: tomorrow
161+
format: strict_date_optional_time
162+
- match: {hits.total.value: 6}
163+
- match: {aggregations.v10.buckets.0.key_as_string: "2018-01-19T17:41:34.000Z"}
164+
- match: {aggregations.v10.buckets.0.doc_count: 1}
165+
- match: {aggregations.v10.buckets.1.key_as_string: "2018-01-20T17:41:34.000Z"}
166+
- match: {aggregations.v10.buckets.1.doc_count: 1}
167+
168+
---
169+
"term query":
170+
- do:
171+
search:
172+
index: sensor
173+
body:
174+
query:
175+
term:
176+
tomorrow: 2018-01-19T17:41:34Z
177+
- match: {hits.total.value: 1}
178+
- match: {hits.hits.0._source.voltage: 4.0}

0 commit comments

Comments
 (0)