[CI] AsyncSearchActionTests.testCancellation fails on CI

[tlrx]

The test AsyncSearchActionTests.testCancellation failed on CI on the 7.x branch:

org.elasticsearch.xpack.search.AsyncSearchActionTests > testCancellation FAILED
    java.lang.AssertionError: Shard [.async-search][0] is still locked after 5 sec waiting
        at __randomizedtesting.SeedInfo.seed([DD4D761E3C3EE063:B7A2968A281D7E5]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.elasticsearch.test.InternalTestCluster.assertAfterTest(InternalTestCluster.java:2483)
        at org.elasticsearch.test.ESIntegTestCase.afterInternal(ESIntegTestCase.java:600)
        at org.elasticsearch.test.ESIntegTestCase.cleanUpCluster(ESIntegTestCase.java:2143)

Build scan: https://gradle-enterprise.elastic.co/s/4ksqilghmk6ii

This is the first time this test fails on CI. I'm not sure this is really related to async search as the logs mentioned some version conflict exception:

[2020-03-25T03:22:06,439][ERROR][o.e.x.s.TransportSubmitAsyncSearchAction] [node_s0] failed to store async-search [FkdmWEtzRnhjUVZ1cE9ya090RUE1TVEaWWR1SU5FRV9RcTZuYTBqa0kyYVVDZzo1NTQ=]
  org.elasticsearch.transport.RemoteTransportException: [node_s0][127.0.0.1:63916][indices:data/write/update[s]]
  Caused by: org.elasticsearch.index.engine.VersionConflictEngineException: [GfXKsFxcQVupOrkOtEA5MQ]: version conflict, required seqNo [0], primary term [1]. but no document was found
       at org.elasticsearch.index.engine.InternalEngine.planIndexingAsPrimary(InternalEngine.java:1065) ~[elasticsearch-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT]

It does not reproduce locally with:

./gradlew ':x-pack:plugin:async-search:test' --tests "org.elasticsearch.xpack.search.AsyncSearchActionTests.testCancellation" \
  -Dtests.seed=DD4D761E3C3EE063 \
  -Dtests.security.manager=true \
  -Dtests.locale=es-NI \
  -Dtests.timezone=Atlantic/Faroe \
  -Dcompiler.java=13

This also looks different from #53360.

[elasticmachine]

Pinging @elastic/es-search (:Search/Search)

[DaveCTurner]

I saw another failure of this test. It looks like there's a problem during test cleanup where the .async-search index is deleted and then re-created:

1> [2020-03-26T21:29:32,687][INFO ][o.e.x.s.AsyncSearchActionTests] [testCancellation] [AsyncSearchActionTests#testCancellation]: cleaning up after test |  
-- | --
  | 1> [2020-03-26T21:29:32,736][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_s0] [test-async/4xCsJ1nESwa4J36TnnOulA] deleting index |  
  | 1> [2020-03-26T21:29:32,736][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_s0] [.async-search/WeVwFd18SPWmCrHQx0qgSg] deleting index |  
  | 1> [2020-03-26T21:29:32,848][WARN ][o.e.d.c.m.MetaDataCreateIndexService] [node_s0] index name [.async-search] starts with a dot '.', in the next major version, index names starting with a dot are reserved for hidden indices and system indices |  
  | 1> [2020-03-26T21:29:32,851][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s0] [.async-search] creating index, cause [api], templates [random_index_template], shards [1]/[0], mappings [_doc] |  
  | 1> [2020-03-26T21:29:32,851][INFO ][o.e.c.r.a.AllocationService] [node_s0] updating number_of_replicas to [1] for indices [.async-search] |  
  | 1> [2020-03-26T21:29:32,876][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_s0] removing template [random_index_template] |  
  | 1> [2020-03-26T21:29:32,894][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_s0] removing template [ilm-history] |  
  | 1> [2020-03-26T21:29:32,931][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_s0] adding template [ilm-history] for index patterns [ilm-history-2*] |  
  | 1> [2020-03-26T21:29:32,945][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_s0] removing template [.slm-history] |  
  | 1> [2020-03-26T21:29:32,972][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_s0] adding template [.slm-history] for index patterns [.slm-history-2*] |  
  | 1> [2020-03-26T21:29:33,068][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.async-search][0]]]). |  
  | 1> [2020-03-26T21:29:38,202][INFO ][o.e.x.s.AsyncSearchActionTests] [testCancellation] after test

[DaveCTurner]

Muted in master, 7.x and 7.7: 1d3a8de f48e8f3 7491f7f

[mark-vieira]

There are a number of failures that seem to be specific to our MacOS agents. They typically look like this:

org.elasticsearch.xpack.search.AsyncSearchActionTests > testCleanupOnFailure FAILED
    java.lang.AssertionError: 6 channels still being tracked in RestCancellableNodeClient while there should be none expected:<0> but was:<6>
        at __randomizedtesting.SeedInfo.seed([4C0451DF11A87EF0:C71070CDB7AD6138]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.junit.Assert.failNotEquals(Assert.java:834)
        at org.junit.Assert.assertEquals(Assert.java:645)
        at org.elasticsearch.test.ESIntegTestCase.lambda$clearClusters$2(ESIntegTestCase.java:551)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:917)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:890)
        at org.elasticsearch.test.ESIntegTestCase.clearClusters(ESIntegTestCase.java:549)
        at org.elasticsearch.test.ESIntegTestCase.buildAndPutCluster(ESIntegTestCase.java:523)
        at org.elasticsearch.test.ESIntegTestCase.beforeInternal(ESIntegTestCase.java:381)
        at org.elasticsearch.test.ESIntegTestCase.setupTestCluster(ESIntegTestCase.java:2192)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

This doesn't happen locally for me on my MBP so it might be infra related but I don't know enough about these tests to infer what that might be. Seems isolated to AsyncSearchActionTests.

Let me know if I should open a separate issue for this.

[javanna]

thanks for your comment @mark-vieira I will look into this.

[mark-vieira]

Ah, I forgot to include a link to an example build. Here you are @javanna.

https://gradle-enterprise.elastic.co/s/msim2zdegwhzg

[benwtrent]

More MacOS failures like this comment:

https://gradle-enterprise.elastic.co/s/7rni7zuujym2m (7.x)
https://gradle-enterprise.elastic.co/s/wr5bjpe6hejig (7.7)

[jimczi]

I seems that we have two types of failure here. One that fails with:

java.lang.AssertionError: 6 channels still being tracked in RestCancellableNodeClient while there should be none expected:<0> but was:<6>

and another one that fails with:

org.elasticsearch.xpack.search.AsyncSearchActionTests > testCancellation FAILED
    java.lang.AssertionError: Shard [.async-search][0] is still locked after 5 sec waiting

They are maybe linked but the former is weird since it fails with a rest layer assertion on tests that should only use the transport client. I suspect that the leaked channels are coming from other tests that failed to cleanup their connections. @javanna were you able to reproduce ?