Skip to content

Commit 81a6c1d

Browse files
jpountzjpountz
authored
Give precedence to index creation when mixing typed templates with typeless index creation and vice-versa. (#38055)
This is a backport of #37871 and #38032.
1 parent 9c2d1e6 commit 81a6c1d

File tree

6 files changed

+308
-3
lines changed

6 files changed

+308
-3
lines changed

‎docs/reference/mapping/removal_of_types.asciidoc

+73
Original file line numberDiff line numberDiff line change
@@ -424,3 +424,76 @@ POST _reindex
424424
----
425425
// NOTCONSOLE
426426

427+
[float]
428+
=== Index templates
429+
430+
It is recommended to make index templates typeless before upgrading to 7.0 by
431+
re-adding them with `include_type_name` set to `false`.
432+
433+
In case typeless templates are used with typed index creation calls or typed
434+
templates are used with typeless index creation calls, the template will still
435+
be applied but the index creation call decides whether there should be a type
436+
or not. For instance in the below example, `index-1-01` will have a type in
437+
spite of the fact that it matches a template that is typeless, and `index-2-01`
438+
will be typeless in spite of the fact that it matches a template that defines
439+
a type. Both `index-1-01` and `index-2-01` will inherit the `foo` field from
440+
the template that they match.
441+
442+
[source,js]
443+
--------------------------------------------------
444+
PUT _template/template1?include_type_name=false
445+
{
446+
"index_patterns":[ "index-1-*" ],
447+
"mappings": {
448+
"properties": {
449+
"foo": {
450+
"type": "keyword"
451+
}
452+
}
453+
}
454+
}
455+
456+
PUT _template/template2?include_type_name=true
457+
{
458+
"index_patterns":[ "index-2-*" ],
459+
"mappings": {
460+
"type": {
461+
"properties": {
462+
"foo": {
463+
"type": "keyword"
464+
}
465+
}
466+
}
467+
}
468+
}
469+
470+
PUT index-1-01?include_type_name=true
471+
{
472+
"mappings": {
473+
"type": {
474+
"properties": {
475+
"bar": {
476+
"type": "long"
477+
}
478+
}
479+
}
480+
}
481+
}
482+
483+
PUT index-2-01?include_type_name=false
484+
{
485+
"mappings": {
486+
"properties": {
487+
"bar": {
488+
"type": "long"
489+
}
490+
}
491+
}
492+
}
493+
--------------------------------------------------
494+
// CONSOLE
495+
496+
In case of implicit index creation, because of documents that get indexed in
497+
an index that doesn't exist yet, the template is always honored. This is
498+
usually not a problem due to the fact that typless index calls work on typed
499+
indices.

‎rest-api-spec/src/main/resources/rest-api-spec/test/indices.create/20_mix_typeless_typeful.yml

+129
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,129 @@
1+
---
2+
"Create a typeless index while there is a typed template":
3+
4+
- skip:
5+
version: " - 6.6.99"
6+
reason: Merging typeless/typed mappings/templates was added in 6.7
7+
8+
- do:
9+
indices.put_template:
10+
include_type_name: true
11+
name: test_template
12+
body:
13+
index_patterns: test-*
14+
mappings:
15+
my_type:
16+
properties:
17+
foo:
18+
type: keyword
19+
20+
- do:
21+
indices.create:
22+
include_type_name: false
23+
index: test-1
24+
body:
25+
mappings:
26+
properties:
27+
bar:
28+
type: "long"
29+
30+
- do:
31+
indices.get_mapping:
32+
include_type_name: true
33+
index: test-1
34+
35+
- is_true: test-1.mappings._doc # the index creation call won
36+
- is_false: test-1.mappings.my_type
37+
- is_true: test-1.mappings._doc.properties.foo
38+
- is_true: test-1.mappings._doc.properties.bar
39+
40+
---
41+
"Create a typed index while there is a typeless template":
42+
43+
- skip:
44+
version: " - 6.6.99"
45+
reason: Merging typeless/typed mappings/templates was added in 6.7
46+
47+
- do:
48+
indices.put_template:
49+
include_type_name: false
50+
name: test_template
51+
body:
52+
index_patterns: test-*
53+
mappings:
54+
properties:
55+
foo:
56+
type: keyword
57+
58+
- do:
59+
indices.create:
60+
include_type_name: true
61+
index: test-1
62+
body:
63+
mappings:
64+
my_type:
65+
properties:
66+
bar:
67+
type: "long"
68+
69+
- do:
70+
indices.get_mapping:
71+
include_type_name: true
72+
index: test-1
73+
74+
- is_true: test-1.mappings.my_type # the index creation call won
75+
- is_false: test-1.mappings._doc
76+
- is_true: test-1.mappings.my_type.properties.foo
77+
- is_true: test-1.mappings.my_type.properties.bar
78+
79+
---
80+
"Implicitly create a typed index while there is a typeless template":
81+
82+
- skip:
83+
version: " - 6.6.99"
84+
reason: include_type_name only supported as of 6.7
85+
86+
- do:
87+
indices.put_template:
88+
include_type_name: false
89+
name: test_template
90+
body:
91+
index_patterns: test-*
92+
mappings:
93+
properties:
94+
foo:
95+
type: keyword
96+
97+
- do:
98+
catch: /the final mapping would have more than 1 type/
99+
index:
100+
index: test-1
101+
type: my_type
102+
body: { bar: 42 }
103+
104+
---
105+
"Implicitly create a typeless index while there is a typed template":
106+
107+
- skip:
108+
version: " - 6.6.99"
109+
reason: include_type_name only supported as of 6.7
110+
111+
- do:
112+
indices.put_template:
113+
include_type_name: true
114+
name: test_template
115+
body:
116+
index_patterns: test-*
117+
mappings:
118+
my_type:
119+
properties:
120+
foo:
121+
type: keyword
122+
123+
- do:
124+
catch: /the final mapping would have more than 1 type/
125+
index:
126+
index: test-1
127+
type: _doc
128+
body: { bar: 42 }
129+

‎rest-api-spec/src/main/resources/rest-api-spec/test/indices.put_mapping/20_mix_typeless_typeful.yml

+31-1
Original file line numberDiff line numberDiff line change
@@ -42,11 +42,41 @@
4242
type: "keyword" # also test no-op updates that trigger special logic wrt the mapping version
4343

4444
- do:
45-
catch: bad_request
45+
catch: /the final mapping would have more than 1 type/
4646
indices.put_mapping:
4747
index: index
48+
type: some_other_type
4849
body:
4950
some_other_type:
5051
properties:
5152
bar:
5253
type: "long"
54+
55+
56+
---
57+
"PUT mapping with _doc on an index that has types":
58+
59+
- skip:
60+
version: " - 5.99.99"
61+
reason: 5.x indices can have types that start with an `_`
62+
63+
- do:
64+
indices.create:
65+
index: index
66+
body:
67+
mappings:
68+
my_type:
69+
properties:
70+
foo:
71+
type: "keyword"
72+
73+
- do:
74+
catch: /the final mapping would have more than 1 type/
75+
indices.put_mapping:
76+
index: index
77+
type: _doc
78+
body:
79+
_doc:
80+
properties:
81+
bar:
82+
type: "long"

‎server/src/main/java/org/elasticsearch/cluster/metadata/MetaDataCreateIndexService.java

+22
Original file line numberDiff line numberDiff line change
@@ -321,6 +321,28 @@ public ClusterState execute(ClusterState currentState) throws Exception {
321321
if (mappings.containsKey(cursor.key)) {
322322
XContentHelper.mergeDefaults(mappings.get(cursor.key),
323323
MapperService.parseMapping(xContentRegistry, mappingString));
324+
} else if (mappings.size() == 1 && cursor.key.equals(MapperService.SINGLE_MAPPING_NAME)) {
325+
// Typeless template with typed mapping
326+
Map<String, Object> templateMapping = MapperService.parseMapping(xContentRegistry, mappingString);
327+
assert templateMapping.size() == 1 : templateMapping;
328+
assert cursor.key.equals(templateMapping.keySet().iterator().next()) :
329+
cursor.key + " != " + templateMapping;
330+
Map.Entry<String, Map<String, Object>> mappingEntry = mappings.entrySet().iterator().next();
331+
templateMapping = Collections.singletonMap(
332+
mappingEntry.getKey(), // reuse type name from the mapping
333+
templateMapping.values().iterator().next()); // but actual mappings from the template
334+
XContentHelper.mergeDefaults(mappingEntry.getValue(), templateMapping);
335+
} else if (template.mappings().size() == 1 && mappings.containsKey(MapperService.SINGLE_MAPPING_NAME)) {
336+
// Typed template with typeless mapping
337+
Map<String, Object> templateMapping = MapperService.parseMapping(xContentRegistry, mappingString);
338+
assert templateMapping.size() == 1 : templateMapping;
339+
assert cursor.key.equals(templateMapping.keySet().iterator().next()) :
340+
cursor.key + " != " + templateMapping;
341+
Map<String, Object> mapping = mappings.get(MapperService.SINGLE_MAPPING_NAME);
342+
templateMapping = Collections.singletonMap(
343+
MapperService.SINGLE_MAPPING_NAME, // make template mapping typeless
344+
templateMapping.values().iterator().next());
345+
XContentHelper.mergeDefaults(mapping, templateMapping);
324346
} else {
325347
mappings.put(cursor.key,
326348
MapperService.parseMapping(xContentRegistry, mappingString));

‎server/src/main/java/org/elasticsearch/cluster/metadata/MetaDataMappingService.java

+23-2
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,8 @@
3838
import org.elasticsearch.common.compress.CompressedXContent;
3939
import org.elasticsearch.common.inject.Inject;
4040
import org.elasticsearch.common.unit.TimeValue;
41+
import org.elasticsearch.common.xcontent.XContentHelper;
42+
import org.elasticsearch.common.xcontent.XContentType;
4143
import org.elasticsearch.core.internal.io.IOUtils;
4244
import org.elasticsearch.index.Index;
4345
import org.elasticsearch.index.IndexService;
@@ -273,7 +275,10 @@ private ClusterState applyRequest(ClusterState currentState, PutMappingClusterSt
273275
updateList.add(indexMetaData);
274276
// try and parse it (no need to add it here) so we can bail early in case of parsing exception
275277
DocumentMapper newMapper;
276-
DocumentMapper existingMapper = getMapperForUpdate(mapperService, mappingType);
278+
DocumentMapper existingMapper = mapperService.documentMapper(mappingType);
279+
if (existingMapper == null && isMappingSourceTyped(mapperService, mappingUpdateSource, request.type()) == false) {
280+
existingMapper = getMapperForUpdate(mapperService, mappingType);
281+
}
277282
String typeForUpdate = existingMapper == null ? mappingType : existingMapper.type();
278283

279284
if (MapperService.DEFAULT_MAPPING.equals(typeForUpdate)) {
@@ -325,9 +330,16 @@ private ClusterState applyRequest(ClusterState currentState, PutMappingClusterSt
325330
// we use the exact same indexService and metadata we used to validate above here to actually apply the update
326331
final Index index = indexMetaData.getIndex();
327332
final MapperService mapperService = indexMapperServices.get(index);
333+
334+
// If the _type name is _doc and there is no _doc top-level key then this means that we
335+
// are handling a typeless call. In such a case, we override _doc with the actual type
336+
// name in the mappings. This allows to use typeless APIs on typed indices.
328337
String typeForUpdate = mappingType;
329338
CompressedXContent existingSource = null;
330-
DocumentMapper existingMapper = getMapperForUpdate(mapperService, mappingType);
339+
DocumentMapper existingMapper = mapperService.documentMapper(mappingType);
340+
if (existingMapper == null && isMappingSourceTyped(mapperService, mappingUpdateSource, request.type()) == false) {
341+
existingMapper = getMapperForUpdate(mapperService, mappingType);
342+
}
331343
if (existingMapper != null) {
332344
typeForUpdate = existingMapper.type();
333345
existingSource = existingMapper.mappingSource();
@@ -388,6 +400,15 @@ public String describeTasks(List<PutMappingClusterStateUpdateRequest> tasks) {
388400
}
389401
}
390402

403+
/**
404+
* Returns {@code true} if the given {@code mappingSource} includes a type
405+
* as a top-level object.
406+
*/
407+
private static boolean isMappingSourceTyped(MapperService mapperService, CompressedXContent mappingSource, String type) {
408+
Map<String, Object> root = XContentHelper.convertToMap(mappingSource.compressedReference(), true, XContentType.JSON).v2();
409+
return root.size() == 1 && root.keySet().iterator().next().equals(type);
410+
}
411+
391412
public void putMapping(final PutMappingClusterStateUpdateRequest request, final ActionListener<ClusterStateUpdateResponse> listener) {
392413
clusterService.submitStateUpdateTask("put-mapping",
393414
request,

‎server/src/test/java/org/elasticsearch/cluster/metadata/IndexCreationTaskTests.java

+30
Original file line numberDiff line numberDiff line change
@@ -348,6 +348,36 @@ public void testWriteIndexValidationException() throws Exception {
348348
+ "you must manage this on the create index request or with an index template");
349349
}
350350

351+
public void testTypelessTemplateWithTypedIndexCreation() throws Exception {
352+
reqSettings.put(SETTING_NUMBER_OF_SHARDS, 1);
353+
addMatchingTemplate(builder -> builder.putMapping("type", "{\"type\": {}}"));
354+
setupRequestMapping(MapperService.SINGLE_MAPPING_NAME, new CompressedXContent("{\"_doc\":{}}"));
355+
executeTask();
356+
assertThat(getMappingsFromResponse(), Matchers.hasKey(MapperService.SINGLE_MAPPING_NAME));
357+
}
358+
359+
public void testTypedTemplateWithTypelessIndexCreation() throws Exception {
360+
reqSettings.put(SETTING_NUMBER_OF_SHARDS, 1);
361+
addMatchingTemplate(builder -> builder.putMapping(MapperService.SINGLE_MAPPING_NAME, "{\"_doc\": {}}"));
362+
setupRequestMapping("type", new CompressedXContent("{\"type\":{}}"));
363+
executeTask();
364+
assertThat(getMappingsFromResponse(), Matchers.hasKey("type"));
365+
}
366+
367+
public void testTypedTemplate() throws Exception {
368+
reqSettings.put(SETTING_NUMBER_OF_SHARDS, 1);
369+
addMatchingTemplate(builder -> builder.putMapping("type", "{\"type\": {}}"));
370+
executeTask();
371+
assertThat(getMappingsFromResponse(), Matchers.hasKey("type"));
372+
}
373+
374+
public void testTypelessTemplate() throws Exception {
375+
reqSettings.put(SETTING_NUMBER_OF_SHARDS, 1);
376+
addMatchingTemplate(builder -> builder.putMapping(MapperService.SINGLE_MAPPING_NAME, "{\"_doc\": {}}"));
377+
executeTask();
378+
assertThat(getMappingsFromResponse(), Matchers.hasKey(MapperService.SINGLE_MAPPING_NAME));
379+
}
380+
351381
private IndexRoutingTable createIndexRoutingTableWithStartedShards(Index index) {
352382
final IndexRoutingTable idxRoutingTable = mock(IndexRoutingTable.class);
353383

0 commit comments

Comments
 (0)