[CI] DataStreamsRestIT.test {p0=data_stream/90_reindex/Reindex from index into data stream} failure

[probakowski]

Build scan:
https://gradle-enterprise.elastic.co/s/y6ykknsazcvp6
Repro line:
./gradlew ':x-pack:plugin:data-streams:qa:rest:yamlRestTest' --tests "org.elasticsearch.xpack.datastreams.DataStreamsRestIT.test {p0=data_stream/90_reindex/Reindex from index into data stream}" -Dtests.seed=8F6FB95F9DB7D007 -Dtests.security.manager=true -Dbuild.snapshot=false -Dtests.jvm.argline="-Dbuild.snapshot=false" -Dtests.locale=et-EE -Dtests.timezone=America/Indiana/Knox -Druntime.java=11
Reproduces locally?:
No
Applicable branches:
master, 7.x, 7.9
Failure history:
Failures started on 9/7, it seems it only fails on release job
https://gradle-enterprise.elastic.co/scans/tests?failures.failureClassification=non_verification&list.offset=0&list.size=50&list.sortColumn=startTime&list.sortOrder=desc&search.buildToolTypes=gradle&search.buildToolTypes=maven&search.relativeStartTime=P7D&search.timeZoneId=Europe/Warsaw&tests.container=org.elasticsearch.xpack.datastreams.DataStreamsRestIT&tests.sortField=FAILED&tests.test=test%20%7Bp0%3Ddata_stream/90_reindex/Reindex%20from%20index%20into%20data%20stream%7D&tests.unstableOnly=true
Failure excerpt:

[2020-09-08T03:50:41,333][INFO ][o.e.x.d.DataStreamsRestIT] [test] Stash dump on test failure [{ |  
-- | --
  | "stash" : { |  
  | "body" : { |  
  | "error" : { |  
  | "root_cause" : [ |  
  | { |  
  | "type" : "illegal_argument_exception", |  
  | "reason" : "index template [generic_logs_template] has index patterns [logs-*] matching patterns from existing templates [my-template] with patterns (my-template => [logs-*]) that have the same priority [0], multiple index templates may not match during index creation, please use a different priority", |  
  | "stack_trace" : "org.elasticsearch.ElasticsearchException$1: index template [generic_logs_template] has index patterns [logs-*] matching patterns from existing templates [my-template] with patterns (my-template => [logs-*]) that have the same priority [0], multiple index templates may not match during index creation, please use a different priority |  
  | at org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:644) |  
  | at org.elasticsearch.ElasticsearchException.generateFailureXContent(ElasticsearchException.java:572) |  
  | at org.elasticsearch.rest.BytesRestResponse.build(BytesRestResponse.java:149) |  
  | at org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:110) |  
  | at org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:93) |  
  | at org.elasticsearch.rest.action.RestActionListener.onFailure(RestActionListener.java:58) |  
  | at org.elasticsearch.client.node.NodeClient.lambda$executeLocally$1(NodeClient.java:94) |  
  | at org.elasticsearch.tasks.TaskManager$1.onFailure(TaskManager.java:184) |  
  | at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.lambda$doStart$2(TransportMasterNodeAction.java:158) |  
  | at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:94) |  
  | at org.elasticsearch.cluster.metadata.MetadataIndexTemplateService$4.onFailure(MetadataIndexTemplateService.java:412) |  
  | at org.elasticsearch.cluster.service.MasterService$SafeClusterStateTaskListener.onFailure(MasterService.java:508) |  
  | at org.elasticsearch.cluster.service.MasterService$TaskOutputs.notifyFailedTasks(MasterService.java:441) |  
  | at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:215) |  
  | at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151) |  
  | at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) |  
  | at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) |  
  | at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:647) |  
  | at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) |  
  | at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) |  
  | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) |  
  | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) |  
  | at java.base/java.lang.Thread.run(Thread.java:834) |  
  | Caused by: java.lang.IllegalArgumentException: index template [generic_logs_template] has index patterns [logs-*] matching patterns from existing templates [my-template] with patterns (my-template => [logs-*]) that have the same priority [0], multiple index templates may not match during index creation, please use a different priority |  
  | at org.elasticsearch.cluster.metadata.MetadataIndexTemplateService.addIndexTemplateV2(MetadataIndexTemplateService.java:468) |  
  | at org.elasticsearch.cluster.metadata.MetadataIndexTemplateService$4.execute(MetadataIndexTemplateService.java:417) |  
  | at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:47) |  
  | at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:697) |  
  | at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:319) |  
  | at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:214) |  
  | ... 9 more |  
  | " |  
  | } |  
  | ], |  
  | "type" : "illegal_argument_exception", |  
  | "reason" : "index template [generic_logs_template] has index patterns [logs-*] matching patterns from existing templates [my-template] with patterns (my-template => [logs-*]) that have the same priority [0], multiple index templates may not match during index creation, please use a different priority", |  
  | "stack_trace" : "java.lang.IllegalArgumentException: index template [generic_logs_template] has index patterns [logs-*] matching patterns from existing templates [my-template] with patterns (my-template => [logs-*]) that have the same priority [0], multiple index templates may not match during index creation, please use a different priority |  
  | at org.elasticsearch.cluster.metadata.MetadataIndexTemplateService.addIndexTemplateV2(MetadataIndexTemplateService.java:468) |  
  | at org.elasticsearch.cluster.metadata.MetadataIndexTemplateService$4.execute(MetadataIndexTemplateService.java:417) |  
  | at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:47) |  
  | at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:697) |  
  | at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:319) |  
  | at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:214) |  
  | at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151) |  
  | at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) |  
  | at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) |  
  | at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:647) |  
  | at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) |  
  | at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) |  
  | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) |  
  | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) |  
  | at java.base/java.lang.Thread.run(Thread.java:834) |  
  | " |  
  | }, |  
  | "status" : 400 |  
  | } |  
  | } |  
  | }]

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/Data streams)

[probakowski]

Likely caused by #62038

[ywangd]

Failed again https://gradle-enterprise.elastic.co/s/n76foygie4xbg

This can be reproduced locally with:

./gradlew ':x-pack:plugin:data-streams:qa:rest:yamlRestTest' --tests "org.elasticsearch.xpack.datastreams.DataStreamsRestIT.test {p0=data_stream/10_data_stream_resolvability/Verify data stream resolvability for xpack apis}" -Dtests.seed
=8AA6124395032DFC -Dtests.security.manager=true -Dbuild.snapshot=false -Dtests.jvm.argline="-Dbuild.snapshot=false" -Dtests.locale=lv -Dtests.timezone=Europe/Helsinki -Druntime.java=11 -Dlicense.key=x-pack/license-tools/src/test/resources
/public.key

Note: since this is a release test, I add -Dlicense.key=x-pack/license-tools/src/test/resources /public.key at the end of the reproduction line so it runs locally.

Failure message excerpt:

java.lang.AssertionError: Failure at [data_stream/10_data_stream_resolvability:34]: expected [404] status code but api [searchable_snapshots.clear_cache] returned [400 Bad Request] [{"error":"no handler found for uri [/logs-foobar/_searchable_snapshots/cache/clear?error_trace=true] and method [POST]"}] |  
-- | --
  | Expected: <404> |  
  | but: was <400>

[ywangd]

Failed for 7.9 as well: https://gradle-enterprise.elastic.co/s/7wifjfnj6pdeo

[ywangd]

The failure is because the feature of searchable snapshots is not enabled in release builds (thanks @tvernum). I raised #62137 for the fix.

[ywangd]

Manually closing this since I had a typo in the fixing PR #62137 ...