[CI] Failure in MixedClusterClientYamlTestSuiteIT p0=indices.data_stream/10_basic/Create data stream

[cbuescher]

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob+fast+bwc/4517/
https://gradle-enterprise.elastic.co/s/jos32xdpfyopu

I cannot reproduce locally so far, but this is the REPRODUCE line from the failure:

./gradlew ':qa:mixed-cluster:v7.8.0#mixedClusterTest' --tests "org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT" -Dtests.method="test {p0=indices.data_stream/10_basic/Create data stream}" -Dtests.seed=85EE8414B315A65E -Dtests.security.manager=true -Dtests.locale=ro-RO -Dtests.timezone=Asia/Samarkand -Dcompiler.java=13

Error:

[2020-03-26T18:21:51,645][INFO ][o.e.b.MixedClusterClientYamlTestSuiteIT] [test] Stash dump on test failure [{
  "stash" : {
    "body" : {
      "error" : {
        "root_cause" : [
          {
            "type" : "illegal_argument_exception",
            "reason" : "data_stream [simple-data-stream2] already exists",
            "stack_trace" : "org.elasticsearch.ElasticsearchException$1: data_stream [simple-data-stream2] already exists
	at org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:644)
	at org.elasticsearch.ElasticsearchException.generateFailureXContent(ElasticsearchException.java:572)
	at org.elasticsearch.rest.BytesRestResponse.build(BytesRestResponse.java:138)
	at org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:96)
	at org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:91)
	at org.elasticsearch.rest.action.RestActionListener.onFailure(RestActionListener.java:58)
	at org.elasticsearch.client.node.NodeClient.lambda$executeLocally$1(NodeClient.java:91)
	at org.elasticsearch.tasks.TaskManager$1.onFailure(TaskManager.java:153)
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.lambda$doStart$2(TransportMasterNodeAction.java:163)
	at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:94)
	at org.elasticsearch.action.admin.indices.datastream.CreateDataStreamAction$TransportAction$1.onFailure(CreateDataStreamAction.java:148)
	at org.elasticsearch.cluster.service.MasterService$SafeClusterStateTaskListener.onFailure(MasterService.java:508)
	at org.elasticsearch.cluster.service.MasterService$TaskOutputs.notifyFailedTasks(MasterService.java:441)
	at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:215)
	at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151)
	at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150)
	at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:629)
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252)
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.IllegalArgumentException: data_stream [simple-data-stream2] already exists
	at org.elasticsearch.action.admin.indices.datastream.CreateDataStreamAction$TransportAction.createDataStream(CreateDataStreamAction.java:165)
	at org.elasticsearch.action.admin.indices.datastream.CreateDataStreamAction$TransportAction$1.execute(CreateDataStreamAction.java:153)
	at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:47)
	at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:697)
	at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:319)
	at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:214)
	... 9 more
"
          }
        ],
        "type" : "illegal_argument_exception",
        "reason" : "data_stream [simple-data-stream2] already exists",
        "stack_trace" : "java.lang.IllegalArgumentException: data_stream [simple-data-stream2] already exists
	at org.elasticsearch.action.admin.indices.datastream.CreateDataStreamAction$TransportAction.createDataStream(CreateDataStreamAction.java:165)
	at org.elasticsearch.action.admin.indices.datastream.CreateDataStreamAction$TransportAction$1.execute(CreateDataStreamAction.java:153)
	at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:47)
	at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:697)
	at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:319)
	at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:214)
	at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151)
	at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150)
	at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:629)
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252)
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
"
      },
      "status" : 400
    }
  }
}]
[2020-03-26T18:21:51,683][INFO ][o.e.b.MixedClusterClientYamlTestSuiteIT] [test] [p0=indices.data_stream/10_basic/Create data stream] after test

This yml test seems to have recently unmuted by #54074, doesn't seem to fail consistently so I leave it unmuted for now.

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/Data streams)

[DaveCTurner]

https://gradle-enterprise.elastic.co/s/jos32xdpfyopu looks to be another failure. No, that's the one linked above, nvm.

[martijnvg]

This test doesn't fail with the error originally provided here. There are failures, but these are related to #54415, which is unrelated to data streams.