Got Request execution cancelled for every requests after an OutOfMemoryError

[OlivierToth-itsf]

Hi,

I found that if I have an OutOfMemoryError when querying "too big object" from elasticsearch, my application does not crash, but any further request will be cancelled with this error :

Unexpected exception thrown: java.lang.RuntimeException: Request execution cancelled
org.opentest4j.AssertionFailedError: Unexpected exception thrown: java.lang.RuntimeException: Request execution cancelled
	at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:152)
	at org.junit.jupiter.api.AssertDoesNotThrow.createAssertionFailedError(AssertDoesNotThrow.java:84)
	at org.junit.jupiter.api.AssertDoesNotThrow.assertDoesNotThrow(AssertDoesNotThrow.java:75)
	at org.junit.jupiter.api.AssertDoesNotThrow.assertDoesNotThrow(AssertDoesNotThrow.java:58)
	at org.junit.jupiter.api.Assertions.assertDoesNotThrow(Assertions.java:3228)
	at test.es_crash.service.StubServiceTest.test(StubServiceTest.java:81)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
Caused by: java.lang.RuntimeException: Request execution cancelled
	at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:942)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:304)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:292)
	at co.elastic.clients.transport.rest_client.RestClientHttpClient.performRequest(RestClientHttpClient.java:91)
	at co.elastic.clients.transport.ElasticsearchTransportBase.performRequest(ElasticsearchTransportBase.java:144)
	at co.elastic.clients.elasticsearch.ElasticsearchClient.get(ElasticsearchClient.java:910)
	at org.springframework.data.elasticsearch.client.elc.ElasticsearchTemplate.lambda$get$0(ElasticsearchTemplate.java:140)
	at org.springframework.data.elasticsearch.client.elc.ElasticsearchTemplate.execute(ElasticsearchTemplate.java:683)
	at org.springframework.data.elasticsearch.client.elc.ElasticsearchTemplate.get(ElasticsearchTemplate.java:140)
	at org.springframework.data.elasticsearch.core.AbstractElasticsearchTemplate.get(AbstractElasticsearchTemplate.java:272)
	at test.es_crash.service.StubService.get(StubService.java:27)
	at test.es_crash.service.StubServiceTest.lambda$test$2(StubServiceTest.java:81)
	at org.junit.jupiter.api.AssertDoesNotThrow.assertDoesNotThrow(AssertDoesNotThrow.java:71)
	... 6 more
Caused by: java.util.concurrent.CancellationException: Request execution cancelled
	at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.execute(CloseableHttpAsyncClientBase.java:114)
	at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:138)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:300)
	... 17 more

I created a sample project here https://github.com/OlivierToth-itsf/Spring-and-Es-crash .
If you run the test case StubServiceTest.test you will see my problem.

(I made many call in parallel in order to force the OOM).

Is this a bug or I misconfigured my project ?

Configuration
Springboot: 3.3.3
Java: 17

[sothawo]

I don't know with how much memory the application is running. But these OOM errors crash the underlying I/O system

2024-09-06T18:52:22.738+02:00 ERROR 3377 --- [es-crash] [ient-0-thread-1] o.a.h.i.n.c.InternalHttpAsyncClient      : I/O reactor terminated abnormally

org.apache.http.nio.reactor.IOReactorException: I/O dispatch worker terminated abnormally
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:359) ~[httpcore-nio-4.4.16.jar:4.4.16]
	at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221) ~[httpasyncclient-4.1.5.jar:4.1.5]
	at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64) ~[httpasyncclient-4.1.5.jar:4.1.5]
	at java.base/java.lang.Thread.run(Thread.java:840) ~[na:na]
Caused by: java.lang.OutOfMemoryError: Java heap space
	at java.base/java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:64) ~[na:na]

For me it's working when increasing the memory (just using some high value, not checking where the limit is that makes it working):

tasks.named('test') {
	useJUnitPlatform()
	jvmArgs '-Xmx16G'
}

So you could increase the heap space, reduce the size of data retrieved from the index (I assume that your documents are large with with multiple smaller properties) by specifying only the needed fields with a source filter and check otherwise the memory consumption of your app to potentially reduce it.

Spring Data Elasticsearch has no means to influence either the heap memory or what happens within the Elastichsearch rest client in this case.

As for recovery it would be nice to have a mechanism to discard the current low level rest client and create a new one, but this is currently not possible (and I don't know if this would really in your case)

[OlivierToth-itsf]

Thanks for your reply
In fact, I forced the OOM in order to have an IOReactorException.
My problem is that, if I have this type of exception, my application doesn't crash, but blocks all calls to Elasticsearch.
As a result, the healthcheck endpoint responds correctly, and so K8S is unable to restart it.

I was wondering if there was a way to recover the connection to Elasticsearch, or if I've made a configuration error.

[sothawo]

Last weekend I tried to find a possibility to recreate the RestClient which is exposed as a Spring bean and then injected (indirectly) in the ElasticsearchTemplate. Although it is possible to close/destroy the RestClient ans register a new instance in the Spring application context, this does not exchange it at the places where it was injected before, so at these places the old instance will be used.

With the current implementation I don't see a possibility to close and recreate a RestClient in the normal program flow; I have an idea but need to check this out first.