Watcher attempts to use ILM when ILM is disabled

[gwbrown]

Found in 7.0.0 (BC2), I think this impacts 6.7 as well.

To reproduce

Start a node with a trial license and xpack.ilm.enabled: false.

During startup, this message will appear in the logs:

java.lang.IllegalStateException: failed to find action [org.elasticsearch.xpack.core.indexlifecycle.action.PutLifecycleAction@d06806ea] to execute
	at org.elasticsearch.client.node.NodeClient.transportAction(NodeClient.java:116) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:393) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.xpack.core.indexlifecycle.client.ILMClient.putLifecyclePolicy(ILMClient.java:42) ~[?:?]
	at org.elasticsearch.xpack.watcher.support.WatcherIndexTemplateRegistry.lambda$putPolicy$3(WatcherIndexTemplateRegistry.java:195) ~[?:?]
	at org.elasticsearch.xpack.core.ClientHelper.executeAsyncWithOrigin(ClientHelper.java:82) ~[?:?]
	at org.elasticsearch.xpack.watcher.support.WatcherIndexTemplateRegistry.lambda$putPolicy$4(WatcherIndexTemplateRegistry.java:178) ~[?:?]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) ~[elasticsearch-7.0.0.jar:7.0.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:835) [?:?]

Additionally, the index template will contain index.lfiecycle.name:

GET _template/.watch-history-9

{
  ".watch-history-9": {
    "order": 2147483647,
    "index_patterns": [
      ".watcher-history-9*"
    ],
    "settings": {
      "index": {
        "format": "6",
        "lifecycle": {
          "name": "watch-history-ilm-policy"
        },
        "number_of_shards": "1",
        "auto_expand_replicas": "0-1",
        "number_of_replicas": "0"
      }
    },
    "mappings": {[...elided for brevity...]},
    "aliases": {}
  }
}

The watch history index is created and records history just fine, although it too has index.lifecycle.name set (even though ILM is disabled):

GET .watcher-history-*

{
  ".watcher-history-9-2019.04.09": {
    "aliases": {},
    "mappings": {[...elided for brevity...]},
    "settings": {
      "index": {
        "lifecycle": {
          "name": "watch-history-ilm-policy"
        },
        "number_of_shards": "1",
        "auto_expand_replicas": "0-1",
        "provided_name": ".watcher-history-9-2019.04.09",
        "format": "6",
        "creation_date": "1554852639758",
        "number_of_replicas": "0",
        "uuid": "b4Mj6GgkQHSP77kyxH2DHA",
        "version": {
          "created": "7000099"
        }
      }
    }
  }
}

Root cause

I believe this is caused by the same root cause as #40803 - we check this setting via the cluster state:

elasticsearch/x-pack/plugin/watcher/src/main/java/org/elasticsearch/xpack/watcher/support/WatcherIndexTemplateRegistry.java

Line 156 in 9c6231f

boolean ilmSupported = XPackSettings.INDEX_LIFECYCLE_ENABLED.get(state.metaData().settings());

But since xpack.ilm.enabled can only be set via elasticsearch.yml, reading it from the cluster state will always result in the default value - it must be read from the Settings object created from elasticsearch.yml at startup.

[elasticmachine]

Pinging @elastic/es-core-features