Write next cluster state fully on all failures

[DaveCTurner]

Today we do not set the LucenePersistedState#writeNextStateFully flag
on all failures, notably on an OutOfMemoryError. Since we don't exit
immediately on an OOME we may have failed part-way through writing a
full state but still proceed with another apparently-incremental write.

With this commit we ensure LucenePersistedState#writeNextStateFully is
only set if the previous write was successful.

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[henningandersen]

LGTM.

Ideally we would add a test that also exercises an Error in writeIncrementalStateAndCommit and writeIncrementalTermUpdateAndCommit. The former is probably "easy" since it also uses big arrays, whereas I can see the latter needing to mock more. I think this can go in without it though if you prefer.

[DaveCTurner]

👍 I generalised the test to behave a bit more randomly to cover a few more branches, see f04ea2f.

There's no easy way to test the term update. I pondered the failure modes of commit() and think it might have a similar problem, although much rarer because it does almost nothing outside of Lucene code so an OOME is more likely tragic to the writer. If however it hits a non-tragic OOME then we might have called prepareCommit() and then the next time round call prepareCommit() again, perhaps after writing some more docs. I think this is ultimately ok, the second time round it'll throw an IllegalStateException and close the writer rather than breaking anything.