calling hotThreads fails because of text/plain response format vs json

[dadoonet]

Java API client version

8.15.0

Java version

21.0

Elasticsearch Version

8.15.0

Problem description

Reported at https://discuss.elastic.co/t/365144 and reproduced loally.

With the 8.15.0 client, run:

HotThreadsResponse hotThreadsResponse = client.nodes().hotThreads();

This gives:

co.elastic.clients.transport.TransportException: node: https://localhost:63624/, status: 200, [es/nodes.hot_threads] Expecting JSON data but response content-type is: text/plain; charset=UTF-8

	at co.elastic.clients.transport.ElasticsearchTransportBase.checkJsonContentType(ElasticsearchTransportBase.java:471)
	at co.elastic.clients.transport.ElasticsearchTransportBase.decodeTransportResponse(ElasticsearchTransportBase.java:394)
	at co.elastic.clients.transport.ElasticsearchTransportBase.getApiResponse(ElasticsearchTransportBase.java:363)
	at co.elastic.clients.transport.ElasticsearchTransportBase.performRequest(ElasticsearchTransportBase.java:147)
	at co.elastic.clients.elasticsearch.nodes.ElasticsearchNodesClient.hotThreads(ElasticsearchNodesClient.java:199)
	at fr.pilato.test.elasticsearch.hlclient.EsClientIT.callHotThreads(EsClientIT.java:956)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)

[dadoonet]

Note that calling the hot threads API from the CLI is indeed sending back a text response:

curl -k -u elastic "https://localhost:63624/_nodes/hot_threads?pretty" 

::: {e519681496ff}{0Ir-BDJfT1icSesuJE3bsA}{euSvsAQOTlel-6xsE0ybKw}{e519681496ff}{172.17.0.2}{172.17.0.2:9300}{cdfhilmrstw}{8.15.0}{7000099-8512000}{ml.machine_memory=8221806592, ml.allocated_processors=12, ml.allocated_processors_double=12.0, ml.max_jvm_size=2147483648, ml.config_version=12.0.0, xpack.installed=true, transform.config_version=10.0.0}
   Hot threads at 2024-08-19T15:00:59.008Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
   
    0.1% [cpu=0.1%, other=0.0%] (299micros out of 500ms) cpu usage by thread 'elasticsearch[e519681496ff][inference_utility][T#1]'
     10/10 snapshots sharing following 16 elements
       java.base@22.0.1/java.lang.Thread.sleepNanos0(Native Method)
       java.base@22.0.1/java.lang.Thread.sleepNanos(Thread.java:491)
       java.base@22.0.1/java.lang.Thread.sleep(Thread.java:559)
       java.base@22.0.1/java.util.concurrent.TimeUnit.sleep(TimeUnit.java:446)
       org.elasticsearch.inference@8.15.0/org.elasticsearch.xpack.inference.external.http.sender.RequestExecutorService.lambda$static$0(RequestExecutorService.java:66)
       org.elasticsearch.inference@8.15.0/org.elasticsearch.xpack.inference.external.http.sender.RequestExecutorService$$Lambda/0x00001ffe01d00b68.sleep(Unknown Source)
       org.elasticsearch.inference@8.15.0/org.elasticsearch.xpack.inference.external.http.sender.RequestExecutorService.handleTasks(RequestExecutorService.java:240)
       org.elasticsearch.inference@8.15.0/org.elasticsearch.xpack.inference.external.http.sender.RequestExecutorService.start(RequestExecutorService.java:192)
       org.elasticsearch.inference@8.15.0/org.elasticsearch.xpack.inference.external.http.sender.AmazonBedrockRequestExecutorService.start(AmazonBedrockRequestExecutorService.java:19)
       org.elasticsearch.inference@8.15.0/org.elasticsearch.xpack.inference.external.amazonbedrock.AmazonBedrockRequestSender.lambda$start$0(AmazonBedrockRequestSender.java:89)
       org.elasticsearch.inference@8.15.0/org.elasticsearch.xpack.inference.external.amazonbedrock.AmazonBedrockRequestSender$$Lambda/0x00001ffe01d0c9c0.run(Unknown Source)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:917)
       java.base@22.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
       java.base@22.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
       java.base@22.0.1/java.lang.Thread.runWith(Thread.java:1583)
       java.base@22.0.1/java.lang.Thread.run(Thread.java:1570)

[dadoonet]

Sounds like it's caused by elastic/elasticsearch-specification#1206

[dadoonet]

Hey @l-trotta

I'm not sure about the status of this one.
I tried today against 9.0.2:

client.nodes().hotThreads();

And this gives:

co.elastic.clients.transport.TransportException: node: https://localhost:52391/, status: 200, [es/nodes.hot_threads] Expecting JSON data but response content-type is: text/plain; charset=UTF-8

	at co.elastic.clients.transport.ElasticsearchTransportBase.checkJsonContentType(ElasticsearchTransportBase.java:506)
	at co.elastic.clients.transport.ElasticsearchTransportBase.decodeTransportResponse(ElasticsearchTransportBase.java:426)
	at co.elastic.clients.transport.ElasticsearchTransportBase.getApiResponse(ElasticsearchTransportBase.java:392)
	at co.elastic.clients.transport.ElasticsearchTransportBase.performRequest(ElasticsearchTransportBase.java:154)
	at co.elastic.clients.elasticsearch.nodes.ElasticsearchNodesClient.hotThreads(ElasticsearchNodesClient.java:199)
	at fr.pilato.test.elasticsearch.hlclient.EsClientIT.callHotThreads(EsClientIT.java:1058)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)

Is it supposed to be fixed?

[l-trotta]

not yet actually, it needs further updates to the spec to mark non json endpoints, I'm waiting for that