component template cannot update after ES7.8 reboot #58643
Comments
tvernum
added
:Data Management/Indices APIs
|
Pinging @elastic/es-core-features (:Core/Features/Indices APIs) |
elasticmachine
added
the
Team:Data Management
|
I was able to reproduce this, I'll work on a fix for it, thanks for reporting it! |
|
I found that it also causes ILM running error
|
This adds rolling upgrade tests that component and composable templates can be read from older versions of the cluster. Relates to elastic#58643
This commit adds an integration test that component templates used to form a composite template can still be updated after a cluster restart. In elastic#58643 an issue arose where mappings were causing problems because of the way we unwrap `_doc` in template mappings. This was also related to the mappings being merged manually rather than using the `MapperService` to do the merging. elastic#58643 was fixed in 7.9 and master with the elastic#58521 change, since mappings now are read and digested by the actual mapper service. This test passes for 7.x and master, and I intend to open a separate PR including this test for 7.8.1 along with a bug fix for elastic#58643. This test is to ensure we don't have any regression in the future.
This commit adds an integration test that component templates used to form a composite template can still be updated after a cluster restart. In #58643 an issue arose where mappings were causing problems because of the way we unwrap `_doc` in template mappings. This was also related to the mappings being merged manually rather than using the `MapperService` to do the merging. #58643 was fixed in 7.9 and master with the #58521 change, since mappings now are read and digested by the actual mapper service. This test passes for 7.x and master, and I intend to open a separate PR including this test for 7.8.1 along with a bug fix for #58643. This test is to ensure we don't have any regression in the future.
…ic#58883) This commit adds an integration test that component templates used to form a composite template can still be updated after a cluster restart. In elastic#58643 an issue arose where mappings were causing problems because of the way we unwrap `_doc` in template mappings. This was also related to the mappings being merged manually rather than using the `MapperService` to do the merging. elastic#58643 was fixed in 7.9 and master with the elastic#58521 change, since mappings now are read and digested by the actual mapper service. This test passes for 7.x and master, and I intend to open a separate PR including this test for 7.8.1 along with a bug fix for elastic#58643. This test is to ensure we don't have any regression in the future.
…58883) (#58914) This commit adds an integration test that component templates used to form a composite template can still be updated after a cluster restart. In #58643 an issue arose where mappings were causing problems because of the way we unwrap `_doc` in template mappings. This was also related to the mappings being merged manually rather than using the `MapperService` to do the merging. #58643 was fixed in 7.9 and master with the #58521 change, since mappings now are read and digested by the actual mapper service. This test passes for 7.x and master, and I intend to open a separate PR including this test for 7.8.1 along with a bug fix for #58643. This test is to ensure we don't have any regression in the future.
Due to the way that our `toXContent` of mappings works, we reduce the type to remove `_doc` from template mappings when serializing them to disk as part of cluster state. This causes problems however, as we always expected there to be a `_doc` field in the mappings. This was resolved in later versions of ES (7.9+) by elastic#58521 which uses a real mapper to handle the mappings. For 7.8.1, however, we need to ensure that we can still read the state after it has been persisted to disk. Resolves elastic#58643 Relates to elastic#58883 Relates to elastic#58521
) This adds rolling upgrade tests that component and composable templates can be read from older versions of the cluster. Relates to #58643
…stic#58867) This adds rolling upgrade tests that component and composable templates can be read from older versions of the cluster. Relates to elastic#58643
Due to the way that our `toXContent` of mappings works, we reduce the type to remove `_doc` from template mappings when serializing them to disk as part of cluster state. This causes problems however, as we always expected there to be a `_doc` field in the mappings. This was resolved in later versions of ES (7.9+) by #58521 which uses a real mapper to handle the mappings. For 7.8.1, however, we need to ensure that we can still read the state after it has been persisted to disk. Resolves #58643 Relates to #58883 Relates to #58521
) Backport of #58867 to 7.x branch. This adds rolling upgrade tests that component and composable templates can be read from older versions of the cluster. Relates to #58643 * Added skip versions * Make `tests.upgrade_from_version` available to `UpgradeClusterClientYamlTestSuiteIT` and use that to skip tests based on versions. * added jdocs Co-authored-by: Elastic Machine <[email protected]> Co-authored-by: Martijn van Groningen <[email protected]>
|
what is the fix for this issue? |
Elasticsearch version (
bin/elasticsearch --version): 7.8.0Plugins installed: []
JVM version (
java -version):openjdk 14.0.1 2020-04-14
OpenJDK Runtime Environment AdoptOpenJDK (build 14.0.1+7)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 14.0.1+7, mixed mode, sharing)
OS version (
uname -aif on a Unix-like system):Linux ubuntu 4.15.0-106-generic #107-Ubuntu SMP Thu Jun 4 11:27:52 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Description of the problem including expected versus actual behavior:
Running 7.8.0 official docker image. Put an index template that using one component template. Restart the container.
Use the same configuration to update the existed component template. You will get "updating component template [x] results in invalid composable template [x] after templates are merged"
Steps to reproduce:
1.Start a es7.8.0 nodel with docker-compose.
compose file:
version: '3'
services:
es_master:
image: elasticsearch:7.8.0
container_name: es_master
restart: always
environment:
- cluster.name=shotecorps
- network.host=192.168.55.3
- http.port=9200
- transport.port=9300
- node.name=master
- node.remote_cluster_client=false
- node.ml=false
- discovery.seed_hosts=192.168.55.3:9300
- cluster.initial_master_nodes=master1
- "ES_JAVA_OPTS=-Xms2g -Xmx2g"
- "action.auto_create_index=-write-,-act-,-read-,"
- "node.attr.hot=true"
- "node.attr.warm=true"
- path.repo=/usr/share/elasticsearch/data/backup
- bootstrap.memory_lock=true
ports:
- "9300:9300"
volumes:
- /opt/elasticsearch/config/:/usr/share/elasticsearch/config/
- /opt/elasticsearch/data/:/usr/share/elasticsearch/data/
- /opt/elasticsearch/log/:/usr/share/elasticsearch/logs/
logging:
driver: "json-file"
options:
max-size: "100m"
network_mode: "host"
kibana:
image: kibana:7.8.0
container_name: kibana
restart: always
container_name: kibana
restart: always
environment:
- ELASTICSEARCH_URL=http://192.168.55.3:9200
- ELASTICSEARCH_HOSTS=http://192.168.55.3:9200
ports:
- "5601:5601"
put component template
PUT _component_template/a
{
"template": {
"settings": {
"index": {
"codec": "default",
"routing": {
"allocation": {
"require": {
"hot": "true"
}
}
},
"refresh_interval": "60s",
"number_of_shards": "1",
"number_of_replicas": "0"
}
},
"mappings": {
"dynamic": "false"
}
}
}
put index template
PUT _index_template/template_2
{
"index_patterns": ["test*"],
"template": {
"mappings": {
"properties": {
"host_name": {
"type": "keyword"
}
}
}
},
"priority": 10,
"composed_of": ["a"]
}
(Now you can update the component/index template, or create new index template that uses the existed component template successfully)
kill es container and restart it
shotecorps@ubuntu:
/tmp$ docker ps/tmp$ docker rm 56a923fdf7c2 -fCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1743ca1ce44b kibana:7.8.0 "/usr/local/bin/dumb…" 5 minutes ago Up 5 minutes 0.0.0.0:5601->5601/tcp kibana
56a923fdf7c2 elasticsearch:7.8.0 "/tini -- /usr/local…" 5 minutes ago Up 5 minutes es_master
shotecorps@ubuntu:
56a923fdf7c2
shotecorps@ubuntu:
/tmp$ docker ps/tmp$ docker-compose -f docker-compose.yml up -dCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1743ca1ce44b kibana:7.8.0 "/usr/local/bin/dumb…" 5 minutes ago Up 5 minutes 0.0.0.0:5601->5601/tcp kibana
shotecorps@ubuntu:
kibana is up-to-date
Creating es_master ... done
After es started, update old component template will get error response.
And you cannot put new index template that uses the old component template either.
PUT _component_template/a
{
"template": {
"settings": {
"index": {
"codec": "default",
"routing": {
"allocation": {
"require": {
"hot": "true"
}
}
},
"refresh_interval": "60s",
"number_of_shards": "1",
"number_of_replicas": "0"
}
},
"mappings": {
"dynamic": "false"
}
}
}
{
"error" : {
"root_cause" : [
{
"type" : "illegal_argument_exception",
"reason" : "updating component template [a] results in invalid composable template [template_2] after templates are merged"
}
],
"type" : "illegal_argument_exception",
"reason" : "updating component template [a] results in invalid composable template [template_2] after templates are merged",
"caused_by" : {
"type" : "illegal_argument_exception",
"reason" : "invalid composite mappings for [template_2]",
"caused_by" : {
"type" : "illegal_state_exception",
"reason" : "invalid mapping definition, expected a single map underneath [_doc] but it was: [{properties={host_name={type=keyword}}}]"
}
}
},
"status" : 400
}
PUT _index_template/template_3
{
"index_patterns": ["new*"],
"template": {
"mappings": {
"properties": {
"host_name": {
"type": "keyword"
}
}
}
},
"priority": 10,
"composed_of": ["a"]
}
{
"error" : {
"root_cause" : [
{
"type" : "illegal_argument_exception",
"reason" : "composable template [template_3] template after composition with component templates [a] is invalid"
}
],
"type" : "illegal_argument_exception",
"reason" : "composable template [template_3] template after composition with component templates [a] is invalid",
"caused_by" : {
"type" : "illegal_argument_exception",
"reason" : "invalid composite mappings for [template_3]",
"caused_by" : {
"type" : "illegal_state_exception",
"reason" : "invalid mapping definition, expected a single map underneath [_doc] but it was: [{dynamic=false}]"
}
}
},
"status" : 400
}
The text was updated successfully, but these errors were encountered: