[CI] CoreFullClusterRestartIT.testNewReplicasWork fails with security exception

[romseygeek]

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.3+bwc-tests/49/console

19:44:18 ERROR   1.14s | CoreFullClusterRestartIT.testNewReplicasWork <<< FAILURES!
19:44:18    > Throwable #1: org.elasticsearch.client.ResponseException: method [POST], host [http://[::1]:46861], URI [/testnewreplicaswork/doc/159], status line [HTTP/1.1 403 Forbidden]
19:44:18    > {"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:data/write/index[p]] is unauthorized for user [_system]"}],"type":"security_exception","reason":"action [indices:data/write/index[p]] is unauthorized for user [_system]"},"status":403}
19:44:18    > 	at __randomizedtesting.SeedInfo.seed([1A4895C4F873CE3C:60CC608FA07927E0]:0)
19:44:18    > 	at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:705)
19:44:18    > 	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235)
19:44:18    > 	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:198)
19:44:18    > 	at org.elasticsearch.upgrades.FullClusterRestartIT.indexRandomDocuments(FullClusterRestartIT.java:1015)
19:44:18    > 	at org.elasticsearch.upgrades.FullClusterRestartIT.testNewReplicasWork(FullClusterRestartIT.java:206)
19:44:18    > 	at java.lang.Thread.run(Thread.java:748)

[elasticmachine]

Pinging @elastic/es-security

[bizybot]

This test has not failed since October 4th, 2018, I have not found any related commit.
I think we can reopen this if it gets reproduced again.

[jkakavas]

This failed again today https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.8+bwc-tests/186/console

[jkakavas]

This hasn't failed again since, list of latest failures until July is here

[romseygeek]

Another failure here: https://gradle-enterprise.elastic.co/s/c2hf2k7kfq3i6/console-log?task=:x-pack:qa:full-cluster-restart:with-system-key:v5.1.1%23oldClusterTestRunner

The failures only ever appear to happen on the 6x branch

[cbuescher]

Similar looking one on 6.8 from today:

https://gradle-enterprise.elastic.co/s/zpiswvdz5tiru

org.elasticsearch.client.ResponseException: method [POST], host [http://[::1]:45969], URI [/testnewreplicaswork/doc/854], status line [HTTP/1.1 403 Forbidden]
{"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:data/write/index[p]] is unauthorized for user [_system]"}],"type":"security_exception","reason":"action [indices:data/write/index[p]] is unauthorized for user [_system]"},"status":403}
	at __randomizedtesting.SeedInfo.seed([9D31C1324F47AD19:E7B53479174D44C5]:0)
	at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:936)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:233)
	at org.elasticsearch.upgrades.FullClusterRestartIT.indexRandomDocuments(FullClusterRestartIT.java:1217)
	at org.elasticsearch.upgrades.FullClusterRestartIT.testNewReplicasWork(FullClusterRestartIT.java:177)

Didn't reproduce locally though:

 ./gradlew ':x-pack:qa:full-cluster-restart:without-system-key:v5.1.2#oldClusterTestRunner' \
  -Dtests.seed=9D31C1324F47AD19 \
  -Dtests.class=org.elasticsearch.xpack.restart.CoreFullClusterRestartIT \
  -Dtests.method="testNewReplicasWork" \
  -Dtests.security.manager=true \
  -Dtests.locale=es-EC \
  -Dtests.timezone=Pacific/Auckland \
  -Dcompiler.java=11 \
  -Druntime.java=8

[Luegg]

One more today https://gradle-enterprise.elastic.co/s/agmqz6vcuevu2/

[jdconrad]

And another https://gradle-enterprise.elastic.co/s/ina3rttb3nm44/

[tvernum]

Closing this issue because it only seem to have affected 6.x