ES 6.4 unable to start after rollover #33316
Assignees
Labels
Comments
jasontedor
added
>bug
:Data Management/Indices APIs
|
Pinging @elastic/es-core-infra |
|
Thanks for the report, this looks serious. We will take a look. |
nik9000
added a commit
to nik9000/elasticsearch
that referenced
this issue
Sep 4, 2018
When we rollover and index we write the conditions of the rollover that the old index met into the old index. Loading this index metadata requires a working `NamedXContentRegistry` that has been populated with parsers from the rollover infrastructure. We had a few loads that didn't use a working `NamedXContentRegistry` and so would fail if they ever encountered an index that had been rolled over. Here are the locations of the loads and how I fixed them: * IndexFolderUpgrader - removed entirely. It existed to support opening indices made in Elasticsearch 2.x. Since we only need this change as far back as 6.4.1 which will supports reading from indices created as far back as 5.0.0 we should be good here. * TransportNodesListGatewayStartedShards - wired the `NamedXContentRegistry` into place. * TransportNodesListShardStoreMetaData - wired the `NamedXContentRegistry` into place. * OldIndexUtils - removed entirely. It existed to support the zip based index backwards compatibility tests which we've since replaced with code that actually runs old versions of Elasticsearch. In addition to fixing the actual problem I added full cluster restart integration tests for rollover which would have caught this problem and I added an extra assertion to IndexMetaData's deserialization code which will trip if we try to deserialize and index's metadata without a fully formed `NamedXContentRegistry`. It won't catch if use the *wrong* `NamedXContentRegistry` but it is better than nothing. Closes elastic#33316
nik9000
added a commit
that referenced
this issue
Sep 6, 2018
When we rollover and index we write the conditions of the rollover that the old index met into the old index. Loading this index metadata requires a working `NamedXContentRegistry` that has been populated with parsers from the rollover infrastructure. We had a few loads that didn't use a working `NamedXContentRegistry` and so would fail if they ever encountered an index that had been rolled over. Here are the locations of the loads and how I fixed them: * IndexFolderUpgrader - removed entirely. It existed to support opening indices made in Elasticsearch 2.x. Since we only need this change as far back as 6.4.1 which will supports reading from indices created as far back as 5.0.0 we should be good here. * TransportNodesListGatewayStartedShards - wired the `NamedXContentRegistry` into place. * TransportNodesListShardStoreMetaData - wired the `NamedXContentRegistry` into place. * OldIndexUtils - removed entirely. It existed to support the zip based index backwards compatibility tests which we've since replaced with code that actually runs old versions of Elasticsearch. In addition to fixing the actual problem I added full cluster restart integration tests for rollover which would have caught this problem and I added an extra assertion to IndexMetaData's deserialization code which will trip if we try to deserialize and index's metadata without a fully formed `NamedXContentRegistry`. It won't catch if use the *wrong* `NamedXContentRegistry` but it is better than nothing. Closes #33316
nik9000
added a commit
that referenced
this issue
Sep 7, 2018
When we rollover and index we write the conditions of the rollover that the old index met into the old index. Loading this index metadata requires a working `NamedXContentRegistry` that has been populated with parsers from the rollover infrastructure. We had a few loads that didn't use a working `NamedXContentRegistry` and so would fail if they ever encountered an index that had been rolled over. Here are the locations of the loads and how I fixed them: * IndexFolderUpgrader - removed entirely. It existed to support opening indices made in Elasticsearch 2.x. Since we only need this change as far back as 6.4.1 which will supports reading from indices created as far back as 5.0.0 we should be good here. * TransportNodesListGatewayStartedShards - wired the `NamedXContentRegistry` into place. * TransportNodesListShardStoreMetaData - wired the `NamedXContentRegistry` into place. * OldIndexUtils - removed entirely. It existed to support the zip based index backwards compatibility tests which we've since replaced with code that actually runs old versions of Elasticsearch. In addition to fixing the actual problem I added full cluster restart integration tests for rollover which would have caught this problem and I added an extra assertion to IndexMetaData's deserialization code which will trip if we try to deserialize and index's metadata without a fully formed `NamedXContentRegistry`. It won't catch if use the *wrong* `NamedXContentRegistry` but it is better than nothing. Closes #33316
nik9000
added a commit
that referenced
this issue
Sep 7, 2018
When we rollover and index we write the conditions of the rollover that the old index met into the old index. Loading this index metadata requires a working `NamedXContentRegistry` that has been populated with parsers from the rollover infrastructure. We had a few loads that didn't use a working `NamedXContentRegistry` and so would fail if they ever encountered an index that had been rolled over. Here are the locations of the loads and how I fixed them: * IndexFolderUpgrader - removed entirely. It existed to support opening indices made in Elasticsearch 2.x. Since we only need this change as far back as 6.4.1 which will supports reading from indices created as far back as 5.0.0 we should be good here. * TransportNodesListGatewayStartedShards - wired the `NamedXContentRegistry` into place. * TransportNodesListShardStoreMetaData - wired the `NamedXContentRegistry` into place. * OldIndexUtils - removed entirely. It existed to support the zip based index backwards compatibility tests which we've since replaced with code that actually runs old versions of Elasticsearch. In addition to fixing the actual problem I added full cluster restart integration tests for rollover which would have caught this problem and I added an extra assertion to IndexMetaData's deserialization code which will trip if we try to deserialize and index's metadata without a fully formed `NamedXContentRegistry`. It won't catch if use the *wrong* `NamedXContentRegistry` but it is better than nothing. Closes #33316
|
Is there as work around for this in 6.4? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Elasticsearch version (
bin/elasticsearch --version):elasticsearch.noarch 6.4.0-1
Plugins installed: []
n/a
JVM version (
java -version):java version "1.8.0_91"
Java(TM) SE Runtime Environment (build 1.8.0_91-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)
OS version (
uname -aif on a Unix-like system):Linux 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
Description of the problem including expected versus actual behavior:
Statefiles are being corrupted by, I suspect, rollover actions from curator (5.5.4-1). After a successful rollover, and the es service is restarted, ES fails to come back up.
Steps to reproduce:
Please include a minimal but complete recreation of the problem, including
(e.g.) index creation, mappings, settings, query etc. The easier you make for
us to reproduce it, the more likely that somebody will take the time to look at it.
rollover alias/index using max_age condition via curator.
index template
{ "template" : "*", "version" : 3, "settings" : { "index.refresh_interval" : "5s" }, "mappings" : { "_default_" : { "dynamic_templates" : [ { "message_field" : { "path_match" : "message", "match_mapping_type" : "string", "mapping" : { "type" : "text", "norms" : false } } }, { "string_fields" : { "match" : "*", "match_mapping_type" : "string", "mapping" : { "type" : "text", "norms" : false, "fields" : { "keyword" : { "type": "keyword", "ignore_above": 256 } } } } } ], "properties" : { "@timestamp": { "type": "date" }, "@version": { "type": "keyword"}, "dest_ip": { "type": "ip" }, "src_ip": { "type": "ip" }, "remote-address": { "type": "ip" }, "security_event" : { "dynamic": true, "properties" : { "src_ip": { "type": "ip" } } }, "geoip" : { "dynamic": true, "properties" : { "ip": { "type": "ip" }, "location" : { "type" : "geo_point" }, "latitude" : { "type" : "half_float" }, "longitude" : { "type" : "half_float" } } } } } } }restart elasticsearch service
systemctl restart elasticsearchProvide logs (if relevant):
[2018-08-31T09:00:35,520][ERROR][o.e.b.Bootstrap ] Exception
org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read [id:14, file:/db/elasticsearch/nodes/0/indices/oEgzNrS3QFuzZVafv2Rhzg/_state/state-14.st]
at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:199) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:304) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.common.util.IndexFolderUpgrader.upgrade(IndexFolderUpgrader.java:89) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.common.util.IndexFolderUpgrader.upgradeIndicesIfNeeded(IndexFolderUpgrader.java:127) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.gateway.GatewayMetaState.(GatewayMetaState.java:87) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.node.Node.(Node.java:447) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.node.Node.(Node.java:256) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:213) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) [elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) [elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) [elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.4.0.jar:6.4.0]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.4.0.jar:6.4.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) [elasticsearch-6.4.0.jar:6.4.0]
Caused by: java.io.IOException: failed to read [id:14, file:/db/elasticsearch/nodes/0/indices/oEgzNrS3QFuzZVafv2Rhzg/_state/state-14.st]
at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:298) ~[elasticsearch-6.4.0.jar:6.4.0]
... 15 more
Caused by: org.elasticsearch.common.xcontent.XContentParseException: [-1:2031] [rollover_info] failed to parse field [met_conditions]
at org.elasticsearch.common.xcontent.ObjectParser.parseValue(ObjectParser.java:316) ~[elasticsearch-x-content-6.4.0.jar:6.4.0]
at org.elasticsearch.common.xcontent.ObjectParser.parseSub(ObjectParser.java:326) ~[elasticsearch-x-content-6.4.0.jar:6.4.0]
at org.elasticsearch.common.xcontent.ObjectParser.parse(ObjectParser.java:168) ~[elasticsearch-x-content-6.4.0.jar:6.4.0]
at org.elasticsearch.common.xcontent.ConstructingObjectParser.parse(ConstructingObjectParser.java:169) ~[elasticsearch-x-content-6.4.0.jar:6.4.0]
at org.elasticsearch.common.xcontent.ConstructingObjectParser.apply(ConstructingObjectParser.java:161) ~[elasticsearch-x-content-6.4.0.jar:6.4.0]
at org.elasticsearch.action.admin.indices.rollover.RolloverInfo.parse(RolloverInfo.java:72) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.cluster.metadata.IndexMetaData$Builder.fromXContent(IndexMetaData.java:1266) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.cluster.metadata.IndexMetaData$2.fromXContent(IndexMetaData.java:1369) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.cluster.metadata.IndexMetaData$2.fromXContent(IndexMetaData.java:1360) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:196) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:294) ~[elasticsearch-6.4.0.jar:6.4.0]
... 15 more
Caused by: org.elasticsearch.common.xcontent.XContentParseException: [-1:2031] [met_conditions] failed to parse field [max_age]
at org.elasticsearch.common.xcontent.ObjectParser.lambda$declareNamedObjects$3(ObjectParser.java:244) ~[elasticsearch-x-content-6.4.0.jar:6.4.0]
at org.elasticsearch.common.xcontent.ObjectParser.lambda$declareNamedObjects$4(ObjectParser.java:256) ~[elasticsearch-x-content-6.4.0.jar:6.4.0]
at org.elasticsearch.common.xcontent.ObjectParser.parseValue(ObjectParser.java:314) ~[elasticsearch-x-content-6.4.0.jar:6.4.0]
at org.elasticsearch.common.xcontent.ObjectParser.parseSub(ObjectParser.java:326) ~[elasticsearch-x-content-6.4.0.jar:6.4.0]
at org.elasticsearch.common.xcontent.ObjectParser.parse(ObjectParser.java:168) ~[elasticsearch-x-content-6.4.0.jar:6.4.0]
at org.elasticsearch.common.xcontent.ConstructingObjectParser.parse(ConstructingObjectParser.java:169) ~[elasticsearch-x-content-6.4.0.jar:6.4.0]
at org.elasticsearch.common.xcontent.ConstructingObjectParser.apply(ConstructingObjectParser.java:161) ~[elasticsearch-x-content-6.4.0.jar:6.4.0]
at org.elasticsearch.action.admin.indices.rollover.RolloverInfo.parse(RolloverInfo.java:72) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.cluster.metadata.IndexMetaData$Builder.fromXContent(IndexMetaData.java:1266) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.cluster.metadata.IndexMetaData$2.fromXContent(IndexMetaData.java:1369) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.cluster.metadata.IndexMetaData$2.fromXContent(IndexMetaData.java:1360) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:196) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:294) ~[elasticsearch-6.4.0.jar:6.4.0]
... 15 more
Caused by: org.elasticsearch.common.xcontent.NamedObjectNotFoundException: named objects are not supported for this parser
at org.elasticsearch.common.xcontent.NamedXContentRegistry.parseNamedObject(NamedXContentRegistry.java:126) ~[elasticsearch-x-content-6.4.0.jar:6.4.0]
at org.elasticsearch.common.xcontent.support.AbstractXContentParser.namedObject(AbstractXContentParser.java:433) ~[elasticsearch-x-content-6.4.0.jar:6.4.0]
at org.elasticsearch.action.admin.indices.rollover.RolloverInfo.lambda$static$1(RolloverInfo.java:51) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.common.xcontent.ObjectParser.lambda$declareNamedObjects$3(ObjectParser.java:242) ~[elasticsearch-x-content-6.4.0.jar:6.4.0]
at org.elasticsearch.common.xcontent.ObjectParser.lambda$declareNamedObjects$4(ObjectParser.java:256) ~[elasticsearch-x-content-6.4.0.jar:6.4.0]
at org.elasticsearch.common.xcontent.ObjectParser.parseValue(ObjectParser.java:314) ~[elasticsearch-x-content-6.4.0.jar:6.4.0]
at org.elasticsearch.common.xcontent.ObjectParser.parseSub(ObjectParser.java:326) ~[elasticsearch-x-content-6.4.0.jar:6.4.0]
at org.elasticsearch.common.xcontent.ObjectParser.parse(ObjectParser.java:168) ~[elasticsearch-x-content-6.4.0.jar:6.4.0]
at org.elasticsearch.common.xcontent.ConstructingObjectParser.parse(ConstructingObjectParser.java:169) ~[elasticsearch-x-content-6.4.0.jar:6.4.0]
at org.elasticsearch.common.xcontent.ConstructingObjectParser.apply(ConstructingObjectParser.java:161) ~[elasticsearch-x-content-6.4.0.jar:6.4.0]
at org.elasticsearch.action.admin.indices.rollover.RolloverInfo.parse(RolloverInfo.java:72) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.cluster.metadata.IndexMetaData$Builder.fromXContent(IndexMetaData.java:1266) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.cluster.metadata.IndexMetaData$2.fromXContent(IndexMetaData.java:1369) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.cluster.metadata.IndexMetaData$2.fromXContent(IndexMetaData.java:1360) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:196) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:294) ~[elasticsearch-6.4.0.jar:6.4.0]
... 15 more
[2018-08-31T09:00:49,455][INFO ][o.e.x.m.j.p.NativeController] Native controller process has stopped - no new native processes can be started
The text was updated successfully, but these errors were encountered: