RestHighLevelClient: indicesOptions cannot be used with point in time, but is not nullable in SearchRequest

[lucboruta]

Elasticsearch version (bin/elasticsearch --version): 7.11.1

Plugins installed: [mapper-size]

JVM version (java -version): 1.8.0_51

OS version (uname -a if on a Unix-like system): macOS Big Sur 11.0.1

Description of the problem including expected versus actual behavior:

Points in time (PIT) are incompatible with a few parameters of the search API, e.g. ccs_minimize_roundtrips and indicesOptions. While ccs_minimize_roundtrips can be set to false with the appropriate setter in org.elasticsearch.action.search.SearchRequest, indices options has a non-null default value, and it cannot be set to null because of the call to java.util.Objects#requireNonNull on line 354 of the SearchRequest class. But this non-null default then triggers a validation exception saying that [indicesOptions] cannot be used with point in time.

We see in the suppressed org.elasticsearch.client.ResponseException exception (see stack trace below) that the URI that is built from the SearchRequest includes the parameters that correspond to the default indicesOptions.

Two observations:

• The request converter which is used when org.elasticsearch.client.RestHighLevelClient#search is called seems to handle null indices options just fine, see org.elasticsearch.client#RequestConverters line 1017.
• If I force indicesOptions to null using reflection, the request completes with no issue.

Steps to reproduce:

		final RestHighLevelClient client = ...
		final String pit = ...
		final TimeValue keepAlive = ...
		final SearchSourceBuilder searchSourceBuilder = SearchSourceBuilder.searchSource()
				.pointInTimeBuilder(new PointInTimeBuilder(pit).setKeepAlive(keepAlive));
		final SearchRequest searchRequest = new SearchRequest();
		// searchRequest.indicesOptions(null); // Necessary but impossible
		searchRequest.setCcsMinimizeRoundtrips(false);
		searchRequest.source(searchSourceBuilder);
		final SearchResponse response = client.search(searchRequest, RequestOptions.DEFAULT);

Provide logs (if relevant):

ElasticsearchStatusException[Elasticsearch exception [type=action_request_validation_exception, reason=Validation Failed: 1: [indicesOptions] cannot be used with point in time;]]
	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:176)
	at org.elasticsearch.client.RestHighLevelClient$$Lambda$847/204421373.apply(Unknown Source)
	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1900)
	at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1877)
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1634)
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1591)
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1561)
	at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:1077)
	...
	Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [http://localhost:9200], URI [/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=false], status line [HTTP/1.1 400 Bad Request]
{"error":{"root_cause":[{"type":"action_request_validation_exception","reason":"Validation Failed: 1: [indicesOptions] cannot be used with point in time;"}],"type":"action_request_validation_exception","reason":"Validation Failed: 1: [indicesOptions] cannot be used with point in time;"},"status":400}
		at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:326)
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:296)
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:270)
		at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1621)
		... 12 more

[elasticmachine]

Pinging @elastic/es-core-features (Team:Core/Features)

[TanveshT]

Getting the same issue while paginating the hits. Have you reached a solution @lucboruta ?

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[TanveshT]

Hey, the issue is with the line 407 of SearchRequest. It accepts Objects.requireNonNull()

ElasticSearch library version: 7.12

A workaround solution for this would be extending the SearchRequest and then Overriding these methods like this:

public class PITAwareSearchRequest extends SearchRequest {

    private IndicesOptions indicesOptions;

    @Override
    public SearchRequest indicesOptions(IndicesOptions indicesOptions) {
        this.indicesOptions = indicesOptions;
        return this;
    }

    @Override
    public IndicesOptions indicesOptions() {
        return indicesOptions;
    }
}

[lucboruta]

@TanveshT, the quickest workaround for us was to use reflection, namely FieldUtils from org.apache.commons.lang3.reflect, to force-set the indicesOptions field to null when we use PITs, e.g. FieldUtils.writeDeclaredField(searchRequest, "indicesOptions", null, true).

I don't know if removing the call to java.util.Objects#requireNonNull would have unintended consequences on other parts of the API.

[jimczi]

I opened #71688 to fix the bug at the validation level. The comparison with the default value is too strict so it only works if the option is not explicitly set.