Data frame transforms doc test leaks notifications index

[DaveCTurner]

This PR build failed:

https://gradle-enterprise.elastic.co/s/gl7xfw7xuevbm/console-log?task=:docs:integTestRunner

It looks like we expected there to be a single active shard (out of 2, i.e. 50%) but in fact there were two active shards (out of 3, i.e. 66⅔%):

org.elasticsearch.smoketest.DocsClientYamlTestSuiteIT > test {yaml=reference/cluster/health/line_140} FAILED
    java.lang.AssertionError: Failure at [reference/cluster/health:36]: $body didn't match expected value:
                             $body: 
               active_primary_shards: expected Integer [1] but was Integer [2]
                       active_shards: expected Integer [1] but was Integer [2]
      active_shards_percent_as_number: expected Double [50.0] but was Double [66.66666666666666]
                        cluster_name: same [integTest]
           delayed_unassigned_shards: same [0]
                 initializing_shards: same [0]
                number_of_data_nodes: same [1]
           number_of_in_flight_fetch: same [0]
                     number_of_nodes: same [1]
             number_of_pending_tasks: same [0]
                   relocating_shards: same [0]
                              status: same [yellow]
      task_max_waiting_in_queue_millis: same [0]
                           timed_out: same [false]
                   unassigned_shards: same [1]

The failing test runs just after a test from reference/transform/apis/put-transform:

1> [2019-09-24T06:09:01,547][INFO ][o.e.s.DocsClientYamlTestSuiteIT] [test] [yaml=reference/transform/apis/put-transform/line_143] before test
1> [2019-09-24T06:09:01,925][INFO ][o.e.s.DocsClientYamlTestSuiteIT] [test] [yaml=reference/transform/apis/put-transform/line_143] after test
1> [2019-09-24T06:09:01,967][INFO ][o.e.s.DocsClientYamlTestSuiteIT] [test] [yaml=reference/cluster/health/line_140] before test
1> [2019-09-24T06:09:02,332][INFO ][o.e.s.DocsClientYamlTestSuiteIT] [test] [yaml=reference/cluster/health/line_140] after test

Looking at the node logs I see that the .data-frame-notifications-1 index was created, presumably by the previous test, but not cleaned up before the cluster health test started:

[2019-09-24T12:09:01,554][INFO ][o.e.c.m.MetaDataCreateIndexService] [node-0] [kibana_sample_data_ecommerce] creating index, cause [api], templates [], shards [1]/[0], mappings []
[2019-09-24T12:09:01,589][INFO ][o.e.c.r.a.AllocationService] [node-0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[kibana_sample_data_ecommerce][0]]]).
[2019-09-24T12:09:01,621][INFO ][o.e.c.m.MetaDataCreateIndexService] [node-0] [.data-frame-internal-2] creating index, cause [auto(bulk api)], templates [.data-frame-internal-2], shards [1]/[1], mappings [_doc]
[2019-09-24T12:09:01,621][INFO ][o.e.c.r.a.AllocationService] [node-0] updating number_of_replicas to [0] for indices [.data-frame-internal-2]
[2019-09-24T12:09:01,657][INFO ][o.e.c.r.a.AllocationService] [node-0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.data-frame-internal-2][0]]]).
[2019-09-24T12:09:01,705][INFO ][o.e.c.m.MetaDataCreateIndexService] [node-0] [.data-frame-notifications-1] creating index, cause [auto(bulk api)], templates [.data-frame-notifications-1], shards [1]/[1], mappings [_doc]
[2019-09-24T12:09:01,706][INFO ][o.e.c.r.a.AllocationService] [node-0] updating number_of_replicas to [0] for indices [.data-frame-notifications-1]
[2019-09-24T12:09:01,731][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node-0] [kibana_sample_data_ecommerce/PGkpPb14TrmCHStV6yBaZQ] deleting index
[2019-09-24T12:09:01,731][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node-0] [.data-frame-internal-2/NFmRTZ3fSwedxfFHnHoSaw] deleting index
[2019-09-24T12:09:01,820][INFO ][o.e.c.r.a.AllocationService] [node-0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.data-frame-notifications-1][0]]]).
[2019-09-24T12:09:01,830][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node-0] removing template [.logstash-management]
[2019-09-24T12:09:01,842][INFO ][o.e.c.m.TemplateUpgradeService] [node-0] Starting template upgrade to version 8.0.0, 1 templates will be updated and 0 will be removed
[2019-09-24T12:09:01,850][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node-0] adding template [.logstash-management] for index patterns [.logstash]
[2019-09-24T12:09:01,862][INFO ][o.e.c.m.TemplateUpgradeService] [node-0] Templates were upgraded successfully to version 8.0.0
[2019-09-24T12:09:01,882][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [node-0] adding index lifecycle policy [watch-history-ilm-policy]
[2019-09-24T12:09:01,905][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [node-0] adding index lifecycle policy [slm-history-ilm-policy]
[2019-09-24T12:09:01,972][INFO ][o.e.c.m.MetaDataCreateIndexService] [node-0] [test1] creating index, cause [api], templates [], shards [1]/[1], mappings []
[2019-09-24T12:09:02,201][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node-0] [.data-frame-notifications-1/tf0A8w8-Syib0TWrxJTgDg] deleting index
[2019-09-24T12:09:02,201][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node-0] [test1/23XD040xTSCqHS31Yw0AzA] deleting index

[elasticmachine]

Pinging @elastic/ml-core

[droberts195]

Hopefully #47016 will fix this

[ywelsch]

There are more failures of this on the 7.4 branch, see https://gradle-enterprise.elastic.co/s/q64jghrniuykc/tests/jbm2x6tnwtxku-amcrja7w2kuqe

Can you backport #47016 to all active branches, in particular 7.4? Thank you

[droberts195]

Backported the fix to 7.4 in b28fd92 and 7.3 in 997ed60. (And transforms doesn't exist in 6.8.)