[CI] InternalEngineTests.testRestoreLocalHistoryFromTranslog

[astefan]

This one reproduces for me with ./gradlew ':server:test' --tests "org.elasticsearch.index.engine.InternalEngineTests.testRestoreLocalHistoryFromTranslog" -Dtests.seed=9E40590F007B5A62 -Dtests.security.manager=true -Dbuild.snapshot=false -Dtests.jvm.argline="-Dbuild.snapshot=false" -Dtests.locale=da -Dtests.timezone=America/Adak -Dcompiler.java=13

03:27:37   1> [2020-02-16T14:26:51,941][INFO ][o.e.i.e.InternalEngineTests] [testRestoreLocalHistoryFromTranslog] before test
03:27:37   1> [2020-02-16T14:26:52,385][INFO ][o.e.i.e.InternalEngineTests] [testRestoreLocalHistoryFromTranslog] after test
03:27:37   2> REPRODUCE WITH: ./gradlew ':server:test' --tests "org.elasticsearch.index.engine.InternalEngineTests.testRestoreLocalHistoryFromTranslog" -Dtests.seed=9E40590F007B5A62 -Dtests.security.manager=true -Dbuild.snapshot=false -Dtests.jvm.argline="-Dbuild.snapshot=false" -Dtests.locale=da -Dtests.timezone=America/Adak -Dcompiler.java=13
03:27:37   2> java.lang.AssertionError: restore from local translog must not add operations to translog
03:27:37     Expected: <23>
03:27:37          but: was <20>
03:27:37         at __randomizedtesting.SeedInfo.seed([9E40590F007B5A62:1BAA42410B03C763]:0)
03:27:37         at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
03:27:37         at org.junit.Assert.assertThat(Assert.java:956)
03:27:37         at org.elasticsearch.index.engine.InternalEngineTests.testRestoreLocalHistoryFromTranslog(InternalEngineTests.java:4243)

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+release-tests/907/console

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Engine)

[ywelsch]

What's happening here is that when reopening the engine, recoverFromTranslog is issuing a flush, which creates a new safe commit, and thereby has the translog retention policy clean up old generations.

I'm not sure what kind of assertion we want to go for here. Perhaps just that the number of operations recovered is less than or equal to the number of operations we previously had in the translog?