InternalEngineTests.testLookupSeqNoByIdInLucene fails on master

[albertzaharovits]

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-unix-compatibility/os=centos/74/console
Reproduces, but only if you leave out the method name:

./gradlew :server:test -Dtests.seed=9C4E17D15B9D4A8E -Dtests.class=org.elasticsearch.index.engine.InternalEngineTests -Dtests.security.manager=true -Dtests.locale=fr-CH -Dtests.timezone=Pacific/Kiritimati -Dcompiler.java=11 -Druntime.java=8

Throwable #1: java.lang.AssertionError: latestOps={22=org.elasticsearch.index.engine.Engine$Index@143161b5, 44=org.elasticsearch.index.engine.Engine$Delete@68d260d, 45=org.elasticsearch.index.engine.Engine$Index@685ed2f2, 24=org.elasticsearch.index.engine.Engine$Index@81e4dc9, 46=org.elasticsearch.index.engine.Engine$Index@2b6f6fec, 47=org.elasticsearch.index.engine.Engine$Index@4238abb0, 25=org.elasticsearch.index.engine.Engine$Delete@54512e13, 26=org.elasticsearch.index.engine.Engine$Delete@4063cb48, 27=org.elasticsearch.index.engine.Engine$Delete@6389d662, 29=org.elasticsearch.index.engine.Engine$Delete@4ec2db88, 50=org.elasticsearch.index.engine.Engine$Delete@3547df80, 30=org.elasticsearch.index.engine.Engine$Delete@4a1db814, 31=org.elasticsearch.index.engine.Engine$Index@324fdb12, 10=org.elasticsearch.index.engine.Engine$Delete@156db115, 11=org.elasticsearch.index.engine.Engine$Index@195e85f4, 33=org.elasticsearch.index.engine.Engine$Index@1dd38453, 13=org.elasticsearch.index.engine.Engine$Delete@3e708295, 35=org.elasticsearch.index.engine.Engine$Delete@1ab1c972, 14=org.elasticsearch.index.engine.Engine$Delete@339a01a, 36=org.elasticsearch.index.engine.Engine$Delete@1ced1406, 37=org.elasticsearch.index.engine.Engine$Delete@664a114, 15=org.elasticsearch.index.engine.Engine$Index@51226618, 16=org.elasticsearch.index.engine.Engine$Delete@69b4d8fa, 39=org.elasticsearch.index.engine.Engine$Delete@3a860784, 17=org.elasticsearch.index.engine.Engine$Delete@61250fb9, 18=org.elasticsearch.index.engine.Engine$Delete@73127eef, 19=org.elasticsearch.index.engine.Engine$Delete@7796af48, 1=org.elasticsearch.index.engine.Engine$Index@28d1cdc4, 2=org.elasticsearch.index.engine.Engine$Index@674e61c4, 4=org.elasticsearch.index.engine.Engine$Delete@2e6ef728, 5=org.elasticsearch.index.engine.Engine$Delete@769720b0, 6=org.elasticsearch.index.engine.Engine$Delete@1181ae7b, 7=org.elasticsearch.index.engine.Engine$Index@367b7947, 8=org.elasticsearch.index.engine.Engine$Delete@22ca6d15, 9=org.elasticsearch.index.engine.Engine$Delete@424f9aa5, 40=org.elasticsearch.index.engine.Engine$Delete@2f684ea4, 41=org.elasticsearch.index.engine.Engine$Delete@6840a5d2, 42=org.elasticsearch.index.engine.Engine$Delete@370b82c7, 20=org.elasticsearch.index.engine.Engine$Delete@1180b04e, 43=org.elasticsearch.index.engine.Engine$Index@70d51ed3, 21=org.elasticsearch.index.engine.Engine$Index@1535cfff} op=47
   > Expected: <61L>
   >      but: was <30L>
   >    at __randomizedtesting.SeedInfo.seed([9C4E17D15B9D4A8E:49B4D7CDBCAA346B]:0)
   >    at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   >    at org.elasticsearch.index.engine.InternalEngineTests.lambda$testLookupSeqNoByIdInLucene$40(InternalEngineTests.java:3849)
   >    at org.elasticsearch.index.engine.InternalEngineTests.testLookupSeqNoByIdInLucene(InternalEngineTests.java:3880)
   >    at java.lang.Thread.run(Thread.java:748)

[elasticmachine]

Pinging @elastic/es-distributed

[jpountz]

We looked into it during test triage power hour. It does indeed reproduce, but doesn't fail anymore if the call to flush() is replaced with a call to refresh(). @ywelsch suggested trying engine.flush(true, false) to force the flush to occur but unfortunately it didn't help. We stopped looking into it since we have limited time to look into it but someone should check why flush is somehow not making changes visible.