Skip to content

Commit 05551dd

Browse files
romseygeekromseygeek
authored
Add index-time scripts to date field mapper (#71633)
This commit allows you to set 'script' and 'on_script_error' parameters on date field mappers, meaning that runtime date fields can be made indexed simply by moving their definitions from the runtime section of the mappings to the properties section.
1 parent 6823b8e commit 05551dd

File tree

15 files changed

+435
-47
lines changed

15 files changed

+435
-47
lines changed

‎docs/reference/mapping/types/date.asciidoc

Lines changed: 25 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -132,7 +132,8 @@ The following parameters are accepted by `date` fields:
132132
<<ignore-malformed,`ignore_malformed`>>::
133133

134134
If `true`, malformed numbers are ignored. If `false` (default), malformed
135-
numbers throw an exception and reject the whole document.
135+
numbers throw an exception and reject the whole document. Note that this
136+
cannot be set if the `script` parameter is used.
136137

137138
<<mapping-index,`index`>>::
138139

@@ -142,7 +143,29 @@ The following parameters are accepted by `date` fields:
142143

143144
Accepts a date value in one of the configured +format+'s as the field
144145
which is substituted for any explicit `null` values. Defaults to `null`,
145-
which means the field is treated as missing.
146+
which means the field is treated as missing. Note that this cannot be
147+
set of the `script` parameter is used.
148+
149+
150+
`on_script_error`::
151+
152+
Defines what to do if the script defined by the `script` parameter
153+
throws an error at indexing time. Accepts `reject` (default), which
154+
will cause the entire document to be rejected, and `ignore`, which
155+
will register the field in the document's
156+
<<mapping-ignored-field,`_ignored`>> metadata field and continue
157+
indexing. This parameter can only be set if the `script` field is
158+
also set.
159+
160+
`script`::
161+
162+
If this parameter is set, then the field will index values generated
163+
by this script, rather than reading the values directly from the
164+
source. If a value is set for this field on the input document, then
165+
the document will be rejected with an error.
166+
Scripts are in the same format as their
167+
<<runtime-mapping-fields,runtime equivalent>>, and should emit
168+
long-valued timestamps.
146169

147170
<<mapping-store,`store`>>::
148171

‎modules/runtime-fields-common/src/yamlRestTest/resources/rest-api-spec/test/runtime_fields/43_date_calculated_at_index.yml

Lines changed: 178 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,178 @@
1+
---
2+
setup:
3+
- do:
4+
indices.create:
5+
index: sensor
6+
body:
7+
settings:
8+
number_of_shards: 1
9+
number_of_replicas: 0
10+
mappings:
11+
properties:
12+
tomorrow:
13+
type: date
14+
script:
15+
source: |
16+
for (def dt : doc['timestamp']) {
17+
emit(dt.plus(params.days, ChronoUnit.DAYS).toEpochMilli());
18+
}
19+
params:
20+
days: 1
21+
# Test fetching from _source and parsing
22+
tomorrow_from_source:
23+
type: date
24+
script:
25+
source: |
26+
Instant instant = Instant.ofEpochMilli(parse(params._source.timestamp));
27+
ZonedDateTime dt = ZonedDateTime.ofInstant(instant, ZoneId.of("UTC"));
28+
emit(dt.plus(1, ChronoUnit.DAYS).toEpochMilli());
29+
# Test returning millis
30+
the_past:
31+
type: date
32+
script:
33+
source: |
34+
for (def dt : doc['timestamp']) {
35+
emit(dt.toInstant().toEpochMilli() - 1000);
36+
}
37+
# Test fetching many values
38+
all_week:
39+
type: date
40+
script:
41+
source: |
42+
for (def dt : doc['timestamp']) {
43+
for (int i = 0; i < 7; i++) {
44+
emit(dt.plus(i, ChronoUnit.DAYS).toEpochMilli());
45+
}
46+
}
47+
# Test format parameter
48+
formatted_tomorrow:
49+
type: date
50+
format: yyyy-MM-dd
51+
script: |
52+
for (def dt : doc['timestamp']) {
53+
emit(dt.plus(1, ChronoUnit.DAYS).toEpochMilli());
54+
}
55+
timestamp:
56+
type: date
57+
temperature:
58+
type: long
59+
voltage:
60+
type: double
61+
node:
62+
type: keyword
63+
64+
- do:
65+
bulk:
66+
index: sensor
67+
refresh: true
68+
body: |
69+
{"index":{}}
70+
{"timestamp": 1516729294000, "temperature": 200, "voltage": 5.2, "node": "a"}
71+
{"index":{}}
72+
{"timestamp": 1516642894000, "temperature": 201, "voltage": 5.8, "node": "b"}
73+
{"index":{}}
74+
{"timestamp": 1516556494000, "temperature": 202, "voltage": 5.1, "node": "a"}
75+
{"index":{}}
76+
{"timestamp": 1516470094000, "temperature": 198, "voltage": 5.6, "node": "b"}
77+
{"index":{}}
78+
{"timestamp": 1516383694000, "temperature": 200, "voltage": 4.2, "node": "c"}
79+
{"index":{}}
80+
{"timestamp": "2018-01-18T17:41:34.000Z", "temperature": 202, "voltage": 4.0, "node": "c"}
81+
82+
---
83+
"get mapping":
84+
- do:
85+
indices.get_mapping:
86+
index: sensor
87+
- match: {sensor.mappings.properties.tomorrow.type: date }
88+
- match:
89+
sensor.mappings.properties.tomorrow.script.source: |
90+
for (def dt : doc['timestamp']) {
91+
emit(dt.plus(params.days, ChronoUnit.DAYS).toEpochMilli());
92+
}
93+
- match: {sensor.mappings.properties.tomorrow.script.params: {days: 1} }
94+
- match: {sensor.mappings.properties.tomorrow.script.lang: painless }
95+
96+
- match: {sensor.mappings.properties.formatted_tomorrow.type: date }
97+
- match:
98+
sensor.mappings.properties.formatted_tomorrow.script.source: |
99+
for (def dt : doc['timestamp']) {
100+
emit(dt.plus(1, ChronoUnit.DAYS).toEpochMilli());
101+
}
102+
- match: {sensor.mappings.properties.formatted_tomorrow.script.lang: painless }
103+
- match: {sensor.mappings.properties.formatted_tomorrow.format: yyyy-MM-dd }
104+
105+
---
106+
"fetch fields":
107+
- do:
108+
search:
109+
index: sensor
110+
body:
111+
sort: timestamp
112+
fields: [tomorrow, tomorrow_from_source, the_past, all_week, formatted_tomorrow]
113+
- match: {hits.total.value: 6}
114+
- match: {hits.hits.0.fields.tomorrow: ["2018-01-19T17:41:34.000Z"] }
115+
- match: {hits.hits.0.fields.tomorrow_from_source: ["2018-01-19T17:41:34.000Z"] }
116+
- match: {hits.hits.0.fields.the_past: ["2018-01-18T17:41:33.000Z"] }
117+
- match:
118+
hits.hits.0.fields.all_week:
119+
- 2018-01-18T17:41:34.000Z
120+
- 2018-01-19T17:41:34.000Z
121+
- 2018-01-20T17:41:34.000Z
122+
- 2018-01-21T17:41:34.000Z
123+
- 2018-01-22T17:41:34.000Z
124+
- 2018-01-23T17:41:34.000Z
125+
- 2018-01-24T17:41:34.000Z
126+
- match: {hits.hits.0.fields.formatted_tomorrow: [2018-01-19] }
127+
128+
---
129+
"docvalue_fields":
130+
- do:
131+
search:
132+
index: sensor
133+
body:
134+
sort: timestamp
135+
docvalue_fields: [tomorrow, tomorrow_from_source, the_past, all_week, formatted_tomorrow]
136+
- match: {hits.total.value: 6}
137+
- match: {hits.hits.0.fields.tomorrow: ["2018-01-19T17:41:34.000Z"] }
138+
- match: {hits.hits.0.fields.tomorrow_from_source: ["2018-01-19T17:41:34.000Z"] }
139+
- match: {hits.hits.0.fields.the_past: ["2018-01-18T17:41:33.000Z"] }
140+
- match:
141+
hits.hits.0.fields.all_week:
142+
- 2018-01-18T17:41:34.000Z
143+
- 2018-01-19T17:41:34.000Z
144+
- 2018-01-20T17:41:34.000Z
145+
- 2018-01-21T17:41:34.000Z
146+
- 2018-01-22T17:41:34.000Z
147+
- 2018-01-23T17:41:34.000Z
148+
- 2018-01-24T17:41:34.000Z
149+
- match: {hits.hits.0.fields.formatted_tomorrow: [2018-01-19] }
150+
151+
---
152+
"terms agg":
153+
- do:
154+
search:
155+
index: sensor
156+
body:
157+
aggs:
158+
v10:
159+
terms:
160+
field: tomorrow
161+
format: strict_date_optional_time
162+
- match: {hits.total.value: 6}
163+
- match: {aggregations.v10.buckets.0.key_as_string: "2018-01-19T17:41:34.000Z"}
164+
- match: {aggregations.v10.buckets.0.doc_count: 1}
165+
- match: {aggregations.v10.buckets.1.key_as_string: "2018-01-20T17:41:34.000Z"}
166+
- match: {aggregations.v10.buckets.1.doc_count: 1}
167+
168+
---
169+
"term query":
170+
- do:
171+
search:
172+
index: sensor
173+
body:
174+
query:
175+
term:
176+
tomorrow: 2018-01-19T17:41:34Z
177+
- match: {hits.total.value: 1}
178+
- match: {hits.hits.0._source.voltage: 4.0}

0 commit comments

Comments
 (0)