Shard IP filtering does not support wildcards (which breaks 2.x to 5.5.1 upgrade)

[joedj]

Elasticsearch version (bin/elasticsearch --version):
Version: 5.5.1, Build: 19c13d0/2017-07-18T20:44:24.823Z, JVM: 1.8.0_141

Plugins installed:
discovery-ec2 ingest-geoip ingest-user-agent x-pack

JVM version (java -version):
openjdk version "1.8.0_141"
OpenJDK Runtime Environment (build 1.8.0_141-b16)
OpenJDK 64-Bit Server VM (build 25.141-b16, mixed mode)

OS version (uname -a if on a Unix-like system):
Linux cd24f177cfe3 4.9.38-16.33.amzn1.x86_64 #1 SMP Thu Jul 20 01:31:29 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:

The docs at https://www.elastic.co/guide/en/elasticsearch/reference/current/shard-allocation-filtering.html and https://www.elastic.co/guide/en/elasticsearch/reference/current/allocation-filtering.html specify that these settings support wildcards (in our case, we're concerned specifically with index.routing.allocation.include._ip).

This is not currently true. Attempting to use the example from the docs results in:

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"invalid IP address [192.168.2.*] for [_ip]"}],"type":"illegal_argument_exception","reason":"invalid IP address [192.168.2.*] for [_ip]"},"status":400}

This is preventing us from upgrading a 2.x cluster to 5.5.1, because the value is currently set to "*". In 2.x, I don't believe it's possible to delete these settings.

I'm sure we can think of some nasty workaround that will allow us to upgrade our old cluster, but suggestions are welcome

Steps to reproduce:

curl -s -XPUT localhost:9200/_cluster/settings --data-binary @- <<"END" | jq .
{
    "transient": {
        "cluster.routing.allocation.include._ip" : "*"
    }
}
END

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "invalid IP address [*] for [_ip]"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "invalid IP address [*] for [_ip]"
  },
  "status": 400
}

Provide logs:

Logs from an attempt to start a node using ES 5.5.1, with an index from ES 2.4.1:

[2017-08-14T00:43:25,522][ERROR][org.elasticsearch.gateway.GatewayMetaState] [cd24f177cfe3] failed to read local state, exiting...
org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read [id:5, legacy:false, file:/var/lib/elasticsearch/data/nodes/0/indices/.kibana/_state/state-5.st]
	at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:150) ~[elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:334) ~[elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.common.util.IndexFolderUpgrader.upgrade(IndexFolderUpgrader.java:90) ~[elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.common.util.IndexFolderUpgrader.upgradeIndicesIfNeeded(IndexFolderUpgrader.java:128) ~[elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:91) [elasticsearch-5.5.1.jar:5.5.1]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) [?:1.8.0_141]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) [?:1.8.0_141]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) [?:1.8.0_141]
	at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:49) [elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86) [elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:116) [elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:47) [elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:825) [elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:43) [elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:59) [elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:50) [elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.common.inject.SingleParameterInjector.inject(SingleParameterInjector.java:42) [elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.common.inject.SingleParameterInjector.getAll(SingleParameterInjector.java:66) [elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:85) [elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:116) [elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:47) [elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:825) [elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:43) [elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:59) [elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:50) [elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:191) [elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:183) [elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:818) [elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:183) [elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:173) [elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:161) [elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:96) [elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:96) [elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:70) [elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:42) [elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.node.Node.<init>(Node.java:497) [elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.node.Node.<init>(Node.java:244) [elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:232) [elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:232) [elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:351) [elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) [elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114) [elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:67) [elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) [elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.cli.Command.main(Command.java:88) [elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) [elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) [elasticsearch-5.5.1.jar:5.5.1]
Caused by: java.io.IOException: failed to read [id:5, legacy:false, file:/var/lib/elasticsearch/data/nodes/0/indices/.kibana/_state/state-5.st]
	at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:327) ~[elasticsearch-5.5.1.jar:5.5.1]
	... 46 more
Caused by: java.lang.IllegalArgumentException: invalid IP address [*] for [_ip]
	at org.elasticsearch.cluster.node.DiscoveryNodeFilters.lambda$static$0(DiscoveryNodeFilters.java:58) ~[elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.common.settings.Setting$3.get(Setting.java:908) ~[elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.common.settings.Setting$3.get(Setting.java:885) ~[elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.cluster.metadata.IndexMetaData$Builder.build(IndexMetaData.java:1026) ~[elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.cluster.metadata.IndexMetaData$Builder.fromXContent(IndexMetaData.java:1240) ~[elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.cluster.metadata.IndexMetaData$1.fromXContent(IndexMetaData.java:1302) ~[elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.cluster.metadata.IndexMetaData$1.fromXContent(IndexMetaData.java:1293) ~[elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:202) ~[elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:322) ~[elasticsearch-5.5.1.jar:5.5.1]
	... 46 more

[ywelsch]

Thank you for the bug report. This is caused by the changes in #22591 and affects ES versions >= 5.3.0.

[ywelsch]

I've opened #26187 that will fix this.

[joedj]

Thanks @ywelsch !

[joedj]

FYI, setting routing.allocation.include._ip to "" instead of "*" appears to be a valid workaround.