Skip to content

Commit 5913b79

Browse files
dnhatndnhatn
authored
Allow specify dynamic templates in bulk request (#71518)
This change allows users to specify dynamic templates in a bulk request. ``` PUT myindex { "mappings": { "dynamic_templates": [{ "time_histograms": { "mapping": { "type": "histogram", "meta": { "unit": "s" } } } }] } } ``` ``` POST myindex/_bulk { "index": { "dynamic_templates": { "response_times": "time_histograms" } } } { "@timestamp": "2020-08-12", "response_times": { "values": [1, 10], "counts": [5, 1] }} ``` Closes #61939
1 parent 77bb387 commit 5913b79

File tree

28 files changed

+934
-102
lines changed

28 files changed

+934
-102
lines changed

‎docs/reference/docs/bulk.asciidoc

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -283,6 +283,8 @@ include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=bulk-index]
283283
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=bulk-id]
284284

285285
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=bulk-require-alias]
286+
287+
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=bulk-dynamic-templates]
286288
--
287289

288290
`delete`::
@@ -311,6 +313,8 @@ include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=bulk-index]
311313
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=bulk-id]
312314

313315
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=bulk-require-alias]
316+
317+
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=bulk-dynamic-templates]
314318
--
315319

316320
`update`::
@@ -750,3 +754,39 @@ The API returns the following result.
750754
}
751755
----
752756
// TESTRESPONSE[s/"index_uuid": "aAsFqTI0Tc2W0LCWgPNrOA"/"index_uuid": $body.$_path/]
757+
758+
759+
[discrete]
760+
[[bulk-dynamic-templates]]
761+
===== Example with dynamic templates parameter
762+
763+
The below example creates a dynamic template, then performs a bulk request
764+
consisting of index/create requests with the `dynamic_templates` parameter.
765+
766+
[source,console]
767+
----
768+
PUT my-index/
769+
{
770+
"mappings": {
771+
"dynamic_templates": [
772+
{
773+
"geo_point": {
774+
"mapping": {
775+
"type" : "geo_point"
776+
}
777+
}
778+
}
779+
]
780+
}
781+
}
782+
783+
POST /_bulk
784+
{ "index" : { "_index" : "my_index", "_id" : "1", "dynamic_templates": {"work_location": "geo_point"}} }
785+
{ "field" : "value1", "work_location": "41.12,-71.34", "raw_location": "41.12,-71.34"}
786+
{ "create" : { "_index" : "my_index", "_id" : "2", "dynamic_templates": {"home_location": "geo_point"}} }
787+
{ "field" : "value2", "home_location": "41.12,-71.34"}
788+
----
789+
790+
The bulk request creates two new fields `work_location` and `home_location` with type `geo_point` according
791+
to the `dynamic_templates` parameter; however, the `raw_location` field is created using default dynamic mapping
792+
rules, as a `text` field in that case since it is supplied as a string in the JSON document.

‎docs/reference/mapping/dynamic/templates.asciidoc

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,10 @@ name
1414
* <<path-match-unmatch,`path_match` and `path_unmatch`>> operate on the full
1515
dotted path to the field
1616

17+
* if a dynamic template does not define `match_mapping_type` nor `match` nor
18+
`path_match`, then it won't match any field, but it can be referred to by
19+
name in `dynamic_templates` as part of a <bulk, bulk request>.
20+
1721
Use the `{name}` and `{dynamic_type}` <<template-variables,template variables>>
1822
in the mapping specification as placeholders.
1923

‎docs/reference/rest-api/common-parms.asciidoc

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -640,6 +640,15 @@ If `true`, the destination must be an <<indices-aliases,index alias>>. Defaults
640640
`false`.
641641
end::require-alias[]
642642

643+
tag::bulk-dynamic-templates[]
644+
`dynamic_templates`::
645+
(Optional, map)
646+
A map from the full name of fields to the name of <<dynamic-templates, dynamic templates>.
647+
Defaults to an empty map. If a name matches a dynamic template, then that template will be
648+
applied regardless of other match predicates defined in the template. And if a field is
649+
already defined in the mapping, then this parameter won't be used.
650+
end::bulk-dynamic-templates[]
651+
643652
tag::node-filter[]
644653
`<node_filter>`::
645654
(Optional, string)

‎rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/bulk/11_dynamic_templates.yml

Lines changed: 171 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,171 @@
1+
---
2+
"Dynamic templates":
3+
- skip:
4+
version: " - 7.12.99"
5+
reason: "Dynamic templates parameter is added to bulk requests in 7.13"
6+
7+
- do:
8+
indices.create:
9+
index: test_index
10+
body:
11+
mappings:
12+
dynamic_templates:
13+
- location:
14+
mapping:
15+
type: geo_point
16+
- my_location:
17+
match: my*
18+
mapping:
19+
type: geo_point
20+
- string:
21+
mapping:
22+
type: keyword
23+
- do:
24+
bulk:
25+
refresh: true
26+
body:
27+
- index:
28+
_index: test_index
29+
_id: id_1
30+
dynamic_templates:
31+
location: location
32+
- { "location": [ -71.34, 41.12 ]}
33+
- index:
34+
_index: test_index
35+
_id: id_2
36+
dynamic_templates:
37+
location: location
38+
- { "location": "41.12,-71.34"}
39+
- match: { errors: false }
40+
- match: { items.0.index.result: created }
41+
- match: { items.1.index.result: created }
42+
43+
- do:
44+
search:
45+
index: test_index
46+
body:
47+
query:
48+
geo_bounding_box:
49+
location:
50+
top_left:
51+
lat: 42
52+
lon: -72
53+
bottom_right:
54+
lat: 40
55+
lon: -74
56+
- match: { hits.total.value: 2 }
57+
- match: { hits.hits.0._id: id_1 }
58+
- match: { hits.hits.1._id: id_2 }
59+
60+
- do:
61+
bulk:
62+
refresh: true
63+
body:
64+
- index:
65+
_index: test_index
66+
_id: id_3
67+
- { "my_location": "41.12,-71.34" } # matches the field name defined in the `my_location` template
68+
- index:
69+
_index: test_index
70+
_id: id_4
71+
dynamic_templates:
72+
my_location: my_location
73+
- { "my_location": "41.12,-71.34" } # use dynamic_templates parameter
74+
- do:
75+
search:
76+
index: test_index
77+
body:
78+
query:
79+
geo_bounding_box:
80+
my_location:
81+
top_left:
82+
lat: 42
83+
lon: -72
84+
bottom_right:
85+
lat: 40
86+
lon: -74
87+
- match: { hits.total.value: 2 }
88+
- match: { hits.hits.0._id: id_3 }
89+
- match: { hits.hits.1._id: id_4 }
90+
91+
- do:
92+
bulk:
93+
refresh: true
94+
body:
95+
- index:
96+
_index: test_index
97+
_id: id_5
98+
dynamic_templates:
99+
location: foo_bar # ok as fields are defined
100+
- { "location": [ -71.34, 41.12 ]}
101+
- index:
102+
_index: test_index
103+
_id: id_6
104+
dynamic_templates:
105+
my_location: foo_bar # ok as fields are defined
106+
- { "my_location": "41.12,-71.34" }
107+
- index:
108+
_index: test_index
109+
_id: id_7
110+
dynamic_templates:
111+
location: bar_foo # ok as fields are defined
112+
- { "location": "41.12,-71.34" }
113+
- match: { errors: false }
114+
- match: { items.0.index.result: created }
115+
- match: { items.1.index.result: created }
116+
- match: { items.2.index.result: created }
117+
118+
- do:
119+
bulk:
120+
refresh: true
121+
body:
122+
- index:
123+
_index: test_index
124+
_id: id_8
125+
dynamic_templates:
126+
foo_location: bar_foo
127+
- { "foo_location": [ -71.34, 41.12 ] } # failed because dynamic template is not found
128+
- index:
129+
_index: test_index
130+
_id: id_9
131+
dynamic_templates:
132+
foo_location: foo_bar
133+
- { "foo_location": "41.12,-71.34" } # failed because dynamic template is not found
134+
- index:
135+
_index: test_index
136+
_id: id_10
137+
dynamic_templates:
138+
new_location: foo
139+
- { "location": "41.12,-71.34"} # ok as fields are defined
140+
- match: { errors: true }
141+
- match: { items.0.index.status: 400 }
142+
- match: { items.0.index.error.type: mapper_parsing_exception }
143+
- match: { items.0.index.error.reason: "Can't find dynamic template for dynamic template name [bar_foo] of field [foo_location]"}
144+
- match: { items.1.index.status: 400 }
145+
- match: { items.1.index.error.type: mapper_parsing_exception }
146+
- match: { items.1.index.error.reason: "Can't find dynamic template for dynamic template name [foo_bar] of field [foo_location]"}
147+
- match: { items.2.index.status: 201 }
148+
- match: { items.2.index.result: created }
149+
150+
# Dynamic template has a wrong type
151+
- do:
152+
bulk:
153+
body:
154+
- index:
155+
_index: test_index
156+
_id: id_11
157+
dynamic_templates:
158+
foo: string
159+
- { "foo.bar": "hello world" } # failed because dynamic template has a wrong type
160+
- index:
161+
_index: test_index
162+
_id: id_12
163+
dynamic_templates:
164+
foo.bar: string
165+
- { "foo.bar": "hello world" } # ok
166+
- match: { errors: true }
167+
- match: { items.0.index.status: 400 }
168+
- match: { items.0.index.error.type: mapper_parsing_exception }
169+
- match: { items.0.index.error.reason: "Field [foo] must be an object; but it's configured as [keyword] in dynamic template [string]"}
170+
- match: { items.1.index.status: 201 }
171+
- match: { items.1.index.result: created }

0 commit comments

Comments
 (0)