[CI] BasicWatcherTests#testPutWatchWithNegativeSchedule failure

[elasticmachine]

Original comment by @davidkyle:

java.lang.AssertionError: Expected watcher to be started, but state was [Tuple [v1=node_sm0, v2=STARTED], Tuple [v1=node_sd1, v2=STARTED], Tuple [v1=node_sd3, v2=STARTED], Tuple [v1=node_sd2, v2=STOPPED]]
Expected: every item is is <STARTED>
     but: an item was <STOPPED>
	at __randomizedtesting.SeedInfo.seed([CCFE7F7B7BF0353A:49A59D6F1069F01D]:0)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
	at org.junit.Assert.assertThat(Assert.java:956)
	at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase.lambda$startWatcher$6(AbstractWatcherIntegrationTestCase.java:473)
	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:734)
	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:708)
	at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase.startWatcher(AbstractWatcherIntegrationTestCase.java:459)
	at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase._setup(AbstractWatcherIntegrationTestCase.java:209)

This appears very similar to LINK REDACTED as in both cases the assertion failure is in AbstractWatcherIntegrationTestCase.startWatcher

Does not reproduce

./gradlew :x-pack-elasticsearch:plugin:watcher:test \
  -Dtests.seed=CCFE7F7B7BF0353A \
  -Dtests.class=org.elasticsearch.xpack.watcher.test.integration.BasicWatcherTests \
  -Dtests.method="testPutWatchWithNegativeSchedule" \
  -Dtests.security.manager=true \
  -Dtests.locale=ar-EG \
  -Dtests.timezone=America/Ojinaga

LINK REDACTED

[elasticmachine]

Original comment by @cbuescher:

Also happened in PR testing, just to mention another datapoint to estimate the frequency of this failure.

[elasticmachine]

Original comment by @tbrooks8:

This failure seems similar:

LINK REDACTED

18:12:59    > Throwable LINK REDACTED: java.lang.AssertionError: Expected watcher to be started, but state was [Tuple [v1=node_s1, v2=STARTED], Tuple [v1=node_s0, v2=STOPPED]]
18:12:59    > Expected: every item is is <STARTED>
18:12:59    >      but: an item was <STOPPED>
18:12:59    > 	at __randomizedtesting.SeedInfo.seed([9AC531CD2C811B67:A597656FA8B305A8]:0)
18:12:59    > 	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
18:12:59    > 	at 

[elasticmachine]

Original comment by @tbrooks8:

This failure seems similar:

LINK REDACTED

java.lang.AssertionError: Expected watcher to be started, but state was [Tuple [v1=node_s1, v2=STOPPED], Tuple [v1=node_s0, v2=STARTED], Tuple [v1=node_s2, v2=STARTED]]
Expected: every item is is <STARTED>
     but: an item was <STOPPED>
	at __randomizedtesting.SeedInfo.seed([222EECB28B15ED5B:23B3C115992D7AC2]:0)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
	at org.junit.Assert.assertThat(Assert.java:956)
	at 

[elasticmachine]

Original comment by @tbrooks8:

This failure seems similar:

LINK REDACTED

java.lang.AssertionError: 
Expected: is "started"
     but: was "stopped"
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
	at org.junit.Assert.assertThat(Assert.java:956)
	at org.junit.Assert.assertThat(Assert.java:923)
	at 

[elasticmachine]

Original comment by @tbrooks8:

This failure seems similar:

LINK REDACTED

[jasontedor]

Closed by #30118