Assertion trips in SearchHits$Total#writeTo in CI

[jaymode]

A CI failure https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+intake/542/console has exposed the tripping of a new assertion when running in a mixed cluster. The assertion that trips is:

elasticsearch/server/src/main/java/org/elasticsearch/search/SearchHits.java

Line 313 in 2d03eeb

assert in.relation == Relation.EQUAL_TO;

When this assertion is hit, node-2 terminates in the test which results in the search failure due to the node being disconnected.

[2018-12-05T20:44:21,496][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-2] fatal error in thread [elasticsearch[node-2][search][T#21]], exiting
java.lang.AssertionError: null
	at org.elasticsearch.search.SearchHits$Total.writeTo(SearchHits.java:313) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.search.SearchHits.writeTo(SearchHits.java:213) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.search.fetch.FetchSearchResult.writeTo(FetchSearchResult.java:102) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.search.fetch.QueryFetchSearchResult.writeTo(QueryFetchSearchResult.java:101) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TcpTransport.buildMessage(TcpTransport.java:946) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TcpTransport.sendResponse(TcpTransport.java:899) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TcpTransport.sendResponse(TcpTransport.java:872) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TcpTransportChannel.sendResponse(TcpTransportChannel.java:70) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:54) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.HandledTransportAction$ChannelActionListener.onResponse(HandledTransportAction.java:93) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.HandledTransportAction$ChannelActionListener.onResponse(HandledTransportAction.java:77) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:357) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:353) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.search.SearchService$4.doRun(SearchService.java:1094) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:759) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]

I have not been able to reproduce this, but the reproduce line is:

./gradlew :qa:mixed-cluster:v6.6.0#mixedClusterTestRunner \
  -Dtests.seed=D53E935D92AC375C \
  -Dtests.class=org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT \
  -Dtests.method="test {p0=indices.sort/10_basic/Index Sort}" \
  -Dtests.security.manager=true \
  -Dtests.locale=cs-CZ \
  -Dtests.timezone=America/Santiago \
  -Dcompiler.java=11 \
  -Druntime.java=8

[elasticmachine]

Pinging @elastic/es-search