[CI] InferenceRestIT test {p0=inference/30_semantic_text_inference/Calculates embeddings using the default ELSER 2 endpoint} failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic-platform-support #4737 / sles-12_platform-support-unix
• elasticsearch-periodic-platform-support #4727 / rocky-9_platform-support-unix
• elasticsearch-periodic-platform-support #4727 / sles-12_platform-support-unix
• elasticsearch-periodic-platform-support #4722 / rhel-9_platform-support-unix
• elasticsearch-periodic-platform-support #4717 / oraclelinux-8_platform-support-unix
• elasticsearch-periodic-platform-support #4717 / sles-12_platform-support-unix
• elasticsearch-periodic #4695 / graalvm-ce17_checkpart2_java-matrix
• elasticsearch-periodic #4690 / encryption-at-rest
• elasticsearch-periodic #4690 / single-processor-node-tests
• elasticsearch-periodic-platform-support #4635 / rhel-9_platform-support-unix

Reproduction Line:

./gradlew ":x-pack:plugin:inference:yamlRestTest" --tests "org.elasticsearch.xpack.inference.InferenceRestIT.test {p0=inference/30_semantic_text_inference/Calculates embeddings using the default ELSER 2 endpoint}" -Dtests.seed=2322785AFCE90B0E -Dtests.locale=en-PR -Dtests.timezone=Antarctica/McMurdo -Druntime.java=23

Applicable branches:
8.16

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: Failure at [inference/30_semantic_text_inference:570]: expected [2xx] status code but api [index] returned [500 Internal Server Error] [{"error":{"root_cause":[{"type":"status_exception","reason":"Model importing failed due to an IOException [java.net.SocketException: Connection reset]","stack_trace":"org.elasticsearch.ElasticsearchStatusException: Model importing failed due to an IOException [java.net.SocketException: Connection reset]\n\tat org.elasticsearch.xpack.ml.packageloader.action.TransportLoadTrainedModelPackage.recordError(TransportLoadTrainedModelPackage.java:222)\n\tat org.elasticsearch.xpack.ml.packageloader.action.TransportLoadTrainedModelPackage.processException(TransportLoadTrainedModelPackage.java:173)\n\tat org.elasticsearch.xpack.ml.packageloader.action.TransportLoadTrainedModelPackage.lambda$importModel$1(TransportLoadTrainedModelPackage.java:158)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64)\n\tat [email protected]/org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:265)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:75)\n\tat [email protected]/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:32)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations$RunAfterActionListener.onFailure(ActionListenerImplementations.java:307)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:75)\n\tat [email protected]/org.elasticsearch.action.ActionListener$3.onFailure(ActionListener.java:410)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:75)\n\tat [email protected]/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:32)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations$RunAfterActionListener.onFailure(ActionListenerImplementations.java:307)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:75)\n\tat [email protected]/org.elasticsearch.action.ActionListener$3.onFailure(ActionListener.java:410)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64)\n\tat [email protected]/org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:265)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:75)\n\tat [email protected]/org.elasticsearch.action.ActionListener$3.onFailure(ActionListener.java:410)\n\tat [email protected]/org.elasticsearch.action.support.RefCountingListener.finish(RefCountingListener.java:128)\n\tat [email protected]/org.elasticsearch.core.AbstractRefCounted$1.closeInternal(AbstractRefCounted.java:125)\n\tat [email protected]/org.elasticsearch.core.AbstractRefCounted.decRef(AbstractRefCounted.java:77)\n\tat [email protected]/org.elasticsearch.action.support.RefCountingRunnable.close(RefCountingRunnable.java:113)\n\tat [email protected]/org.elasticsearch.core.Releasables$4.close(Releasables.java:161)\n\tat [email protected]/org.elasticsearch.action.support.RefCountingListener$1.onFailure(RefCountingListener.java:159)\n\tat org.elasticsearch.xpack.ml.packageloader.action.ModelImporter.downloadPartInRange(ModelImporter.java:197)\n\tat org.elasticsearch.xpack.ml.packageloader.action.ModelImporter.lambda$downloadPartInRange$6(ModelImporter.java:203)\n\tat [email protected]/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:956)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1575)\nCaused by: java.net.SocketException: Connection reset\n\tat java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:318)\n\tat java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:346)\n\tat java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:796)\n\tat java.base/java.net.Socket$SocketInputStream.implRead(Socket.java:1116)\n\tat java.base/java.net.Socket$SocketInputStream.read(Socket.java:1103)\n\tat java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:489)\n\tat java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:483)\n\tat java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70)\n\tat java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1462)\n\tat java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1068)\n\tat java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:291)\n\tat java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:347)\n\tat java.base/java.io.BufferedInputStream.implRead(BufferedInputStream.java:420)\n\tat java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:399)\n\tat java.base/sun.net.www.MeteredStream.read(MeteredStream.java:131)\n\tat java.base/java.io.FilterInputStream.read(FilterInputStream.java:119)\n\tat java.base/sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3649)\n\tat org.elasticsearch.xpack.ml.packageloader.action.ModelLoaderUtils$HttpStreamChunker.next(ModelLoaderUtils.java:113)\n\tat org.elasticsearch.xpack.ml.packageloader.action.ModelImporter.downloadPartInRange(ModelImporter.java:192)\n\t... 5 more\n"}],"type":"exception","reason":"Exception when running inference id [.elser-2-elasticsearch] on field [sparse_field]","caused_by":{"type":"status_exception","reason":"Model importing failed due to an IOException [java.net.SocketException: Connection reset]","caused_by":{"type":"socket_exception","reason":"Connection reset","stack_trace":"java.net.SocketException: Connection reset\n\tat java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:318)\n\tat java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:346)\n\tat java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:796)\n\tat java.base/java.net.Socket$SocketInputStream.implRead(Socket.java:1116)\n\tat java.base/java.net.Socket$SocketInputStream.read(Socket.java:1103)\n\tat java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:489)\n\tat java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:483)\n\tat java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70)\n\tat java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1462)\n\tat java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1068)\n\tat java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:291)\n\tat java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:347)\n\tat java.base/java.io.BufferedInputStream.implRead(BufferedInputStream.java:420)\n\tat java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:399)\n\tat java.base/sun.net.www.MeteredStream.read(MeteredStream.java:131)\n\tat java.base/java.io.FilterInputStream.read(FilterInputStream.java:119)\n\tat java.base/sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3649)\n\tat org.elasticsearch.xpack.ml.packageloader.action.ModelLoaderUtils$HttpStreamChunker.next(ModelLoaderUtils.java:113)\n\tat org.elasticsearch.xpack.ml.packageloader.action.ModelImporter.downloadPartInRange(ModelImporter.java:192)\n\tat org.elasticsearch.xpack.ml.packageloader.action.ModelImporter.lambda$downloadPartInRange$6(ModelImporter.java:203)\n\tat [email protected]/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:956)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1575)\n"},"stack_trace":"org.elasticsearch.ElasticsearchStatusException: Model importing failed due to an IOException [java.net.SocketException: Connection reset]\n\tat org.elasticsearch.xpack.ml.packageloader.action.TransportLoadTrainedModelPackage.recordError(TransportLoadTrainedModelPackage.java:222)\n\tat org.elasticsearch.xpack.ml.packageloader.action.TransportLoadTrainedModelPackage.processException(TransportLoadTrainedModelPackage.java:173)\n\tat org.elasticsearch.xpack.ml.packageloader.action.TransportLoadTrainedModelPackage.lambda$importModel$1(TransportLoadTrainedModelPackage.java:158)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64)\n\tat [email protected]/org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:265)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:75)\n\tat [email protected]/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:32)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations$RunAfterActionListener.onFailure(ActionListenerImplementations.java:307)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:75)\n\tat [email protected]/org.elasticsearch.action.ActionListener$3.onFailure(ActionListener.java:410)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:75)\n\tat [email protected]/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:32)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations$RunAfterActionListener.onFailure(ActionListenerImplementations.java:307)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:75)\n\tat [email protected]/org.elasticsearch.action.ActionListener$3.onFailure(ActionListener.java:410)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64)\n\tat [email protected]/org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:265)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:75)\n\tat [email protected]/org.elasticsearch.action.ActionListener$3.onFailure(ActionListener.java:410)\n\tat [email protected]/org.elasticsearch.action.support.RefCountingListener.finish(RefCountingListener.java:128)\n\tat [email protected]/org.elasticsearch.core.AbstractRefCounted$1.closeInternal(AbstractRefCounted.java:125)\n\tat [email protected]/org.elasticsearch.core.AbstractRefCounted.decRef(AbstractRefCounted.java:77)\n\tat [email protected]/org.elasticsearch.action.support.RefCountingRunnable.close(RefCountingRunnable.java:113)\n\tat [email protected]/org.elasticsearch.core.Releasables$4.close(Releasables.java:161)\n\tat [email protected]/org.elasticsearch.action.support.RefCountingListener$1.onFailure(RefCountingListener.java:159)\n\tat org.elasticsearch.xpack.ml.packageloader.action.ModelImporter.downloadPartInRange(ModelImporter.java:197)\n\tat org.elasticsearch.xpack.ml.packageloader.action.ModelImporter.lambda$downloadPartInRange$6(ModelImporter.java:203)\n\tat [email protected]/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:956)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1575)\nCaused by: java.net.SocketException: Connection reset\n\tat java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:318)\n\tat java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:346)\n\tat java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:796)\n\tat java.base/java.net.Socket$SocketInputStream.implRead(Socket.java:1116)\n\tat java.base/java.net.Socket$SocketInputStream.read(Socket.java:1103)\n\tat java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:489)\n\tat java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:483)\n\tat java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70)\n\tat java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1462)\n\tat java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1068)\n\tat java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:291)\n\tat java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:347)\n\tat java.base/java.io.BufferedInputStream.implRead(BufferedInputStream.java:420)\n\tat java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:399)\n\tat java.base/sun.net.www.MeteredStream.read(MeteredStream.java:131)\n\tat java.base/java.io.FilterInputStream.read(FilterInputStream.java:119)\n\tat java.base/sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3649)\n\tat org.elasticsearch.xpack.ml.packageloader.action.ModelLoaderUtils$HttpStreamChunker.next(ModelLoaderUtils.java:113)\n\tat org.elasticsearch.xpack.ml.packageloader.action.ModelImporter.downloadPartInRange(ModelImporter.java:192)\n\t... 5 more\n"},"stack_trace":"org.elasticsearch.ElasticsearchException: Exception when running inference id [.elser-2-elasticsearch] on field [sparse_field]\n\tat [email protected]/org.elasticsearch.xpack.inference.action.filter.ShardBulkInferenceActionFilter$AsyncBulkShardInferenceAction$2.onResponse(ShardBulkInferenceActionFilter.java:290)\n\tat [email protected]/org.elasticsearch.xpack.inference.action.filter.ShardBulkInferenceActionFilter$AsyncBulkShardInferenceAction$2.onResponse(ShardBulkInferenceActionFilter.java:277)\n\tat [email protected]/org.elasticsearch.xpack.inference.chunking.EmbeddingRequestChunker$DebatchingListener.sendResponse(EmbeddingRequestChunker.java:345)\n\tat [email protected]/org.elasticsearch.xpack.inference.chunking.EmbeddingRequestChunker$DebatchingListener.onFailure(EmbeddingRequestChunker.java:331)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:75)\n\tat [email protected]/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:32)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:75)\n\tat [email protected]/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:32)\n\tat [email protected]/org.elasticsearch.action.support.SubscribableListener$FailureResult.complete(SubscribableListener.java:399)\n\tat [email protected]/org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:311)\n\tat [email protected]/org.elasticsearch.action.support.SubscribableListener.setResult(SubscribableListener.java:336)\n\tat [email protected]/org.elasticsearch.action.support.SubscribableListener.onFailure(SubscribableListener.java:255)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:75)\n\tat [email protected]/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:32)\n\tat [email protected]/org.elasticsearch.action.support.SubscribableListener$FailureResult.complete(SubscribableListener.java:399)\n\tat [email protected]/org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:311)\n\tat [email protected]/org.elasticsearch.action.support.SubscribableListener.setResult(SubscribableListener.java:336)\n\tat [email protected]/org.elasticsearch.action.support.SubscribableListener.onFailure(SubscribableListener.java:255)\n\tat [email protected]/org.elasticsearch.xpack.inference.services.elasticsearch.BaseElasticsearchInternalService.lambda$putBuiltInModel$5(BaseElasticsearchInternalService.java:197)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64)\n\tat [email protected]/org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:265)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:75)\n\tat [email protected]/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:32)\n\tat [email protected]/org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:40)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:75)\n\tat [email protected]/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:32)\n\tat [email protected]/org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:40)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:75)\n\tat [email protected]/org.elasticsearch.action.ActionListener$3.onFailure(ActionListener.java:410)\n\tat [email protected]/org.elasticsearch.tasks.TaskManager$1.onFailure(TaskManager.java:215)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:75)\n\tat [email protected]/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:32)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onFailure(ActionListenerImplementations.java:346)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:75)\n\tat [email protected]/org.elasticsearch.action.ActionListener$3.onFailure(ActionListener.java:410)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:75)\n\tat [email protected]/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:32)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onFailure(ActionListenerImplementations.java:346)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:75)\n\tat [email protected]/org.elasticsearch.action.ActionListener$3.onFailure(ActionListener.java:410)\n\tat [email protected]/org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.lambda$doStart$2(TransportMasterNodeAction.java:235)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.acceptException(ActionListenerImplementations.java:188)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.onFailure(ActionListenerImplementations.java:193)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:75)\n\tat [email protected]/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:32)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:75)\n\tat [email protected]/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:32)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:75)\n\tat [email protected]/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:32)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64)\n\tat [email protected]/org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:265)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:75)\n\tat [email protected]/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:32)\n\tat [email protected]/org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:40)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:75)\n\tat [email protected]/org.elasticsearch.action.ActionListener$3.onFailure(ActionListener.java:410)\n\tat [email protected]/org.elasticsearch.tasks.TaskManager$1.onFailure(TaskManager.java:215)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:75)\n\tat [email protected]/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:32)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onFailure(ActionListenerImplementations.java:346)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:75)\n\tat [email protected]/org.elasticsearch.action.ActionListener$3.onFailure(ActionListener.java:410)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:75)\n\tat [email protected]/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:32)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onFailure(ActionListenerImplementations.java:346)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:75)\n\tat [email protected]/org.elasticsearch.action.ActionListener$3.onFailure(ActionListener.java:410)\n\tat [email protected]/org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.lambda$doStart$2(TransportMasterNodeAction.java:235)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.acceptException(ActionListenerImplementations.java:188)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.onFailure(ActionListenerImplementations.java:193)\n\tat org.elasticsearch.xpack.ml.packageloader.action.TransportLoadTrainedModelPackage.lambda$importModel$1(TransportLoadTrainedModelPackage.java:158)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64)\n\tat [email protected]/org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:265)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:75)\n\tat [email protected]/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:32)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations$RunAfterActionListener.onFailure(ActionListenerImplementations.java:307)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:75)\n\tat [email protected]/org.elasticsearch.action.ActionListener$3.onFailure(ActionListener.java:410)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:75)\n\tat [email protected]/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:32)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations$RunAfterActionListener.onFailure(ActionListenerImplementations.java:307)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:75)\n\tat [email protected]/org.elasticsearch.action.ActionListener$3.onFailure(ActionListener.java:410)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64)\n\tat [email protected]/org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:265)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:75)\n\tat [email protected]/org.elasticsearch.action.ActionListener$3.onFailure(ActionListener.java:410)\n\tat [email protected]/org.elasticsearch.action.support.RefCountingListener.finish(RefCountingListener.java:128)\n\tat [email protected]/org.elasticsearch.core.AbstractRefCounted$1.closeInternal(AbstractRefCounted.java:125)\n\tat [email protected]/org.elasticsearch.core.AbstractRefCounted.decRef(AbstractRefCounted.java:77)\n\tat [email protected]/org.elasticsearch.action.support.RefCountingRunnable.close(RefCountingRunnable.java:113)\n\tat [email protected]/org.elasticsearch.core.Releasables$4.close(Releasables.java:161)\n\tat [email protected]/org.elasticsearch.action.support.RefCountingListener$1.onFailure(RefCountingListener.java:159)\n\tat org.elasticsearch.xpack.ml.packageloader.action.ModelImporter.downloadPartInRange(ModelImporter.java:197)\n\tat org.elasticsearch.xpack.ml.packageloader.action.ModelImporter.lambda$downloadPartInRange$6(ModelImporter.java:203)\n\tat [email protected]/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:956)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1575)\nCaused by: org.elasticsearch.ElasticsearchStatusException: Model importing failed due to an IOException [java.net.SocketException: Connection reset]\n\tat org.elasticsearch.xpack.ml.packageloader.action.TransportLoadTrainedModelPackage.recordError(TransportLoadTrainedModelPackage.java:222)\n\tat org.elasticsearch.xpack.ml.packageloader.action.TransportLoadTrainedModelPackage.processException(TransportLoadTrainedModelPackage.java:173)\n\t... 34 more\nCaused by: java.net.SocketException: Connection reset\n\tat java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:318)\n\tat java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:346)\n\tat java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:796)\n\tat java.base/java.net.Socket$SocketInputStream.implRead(Socket.java:1116)\n\tat java.base/java.net.Socket$SocketInputStream.read(Socket.java:1103)\n\tat java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:489)\n\tat java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:483)\n\tat java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70)\n\tat java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1462)\n\tat java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1068)\n\tat java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:291)\n\tat java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:347)\n\tat java.base/java.io.BufferedInputStream.implRead(BufferedInputStream.java:420)\n\tat java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:399)\n\tat java.base/sun.net.www.MeteredStream.read(MeteredStream.java:131)\n\tat java.base/java.io.FilterInputStream.read(FilterInputStream.java:119)\n\tat java.base/sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3649)\n\tat org.elasticsearch.xpack.ml.packageloader.action.ModelLoaderUtils$HttpStreamChunker.next(ModelLoaderUtils.java:113)\n\tat org.elasticsearch.xpack.ml.packageloader.action.ModelImporter.downloadPartInRange(ModelImporter.java:192)\n\t... 5 more\n"},"status":500}]

Issue Reasons:

• [8.16] 10 failures in test test {p0=inference/30_semantic_text_inference/Calculates embeddings using the default ELSER 2 endpoint} (1.5% fail rate in 652 executions)
• [8.16] 3 failures in step sles-12_platform-support-unix (14.3% fail rate in 21 executions)
• [8.16] 2 failures in step rhel-9_platform-support-unix (10.5% fail rate in 19 executions)
• [8.16] 5 failures in pipeline elasticsearch-periodic-platform-support (23.8% fail rate in 21 executions)
• [8.16] 2 failures in pipeline elasticsearch-periodic (9.5% fail rate in 21 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 3 consecutive failures in step release-tests
• [main] 5 failures in test test {p0=inference/30_semantic_text_inference/Calculates embeddings using the default ELSER 2 endpoint} (2.3% fail rate in 221 executions)
• [main] 3 failures in step release-tests (100.0% fail rate in 3 executions)
• [main] 3 failures in pipeline elasticsearch-periodic (75.0% fail rate in 4 executions)
• [main] 2 failures in pipeline elasticsearch-pull-request (1.9% fail rate in 106 executions)

Build Scans:

• elasticsearch-periodic #4354 / openjdk21_checkpart2_java-fips-matrix
• elasticsearch-pull-request #35400 / part-2
• elasticsearch-pull-request #35350 / release-tests
• elasticsearch-periodic #4350 / release-tests
• elasticsearch-periodic #4346 / release-tests

[elasticsearchmachine]

This has been muted on branch 8.x

Mute Reasons:

• [8.x] 6 consecutive failures in step release-tests
• [8.x] 12 failures in test test {p0=inference/30_semantic_text_inference/Calculates embeddings using the default ELSER 2 endpoint} (5.4% fail rate in 224 executions)
• [8.x] 6 failures in step release-tests (100.0% fail rate in 6 executions)
• [8.x] 2 failures in step part-2 (3.4% fail rate in 58 executions)
• [8.x] 6 failures in pipeline elasticsearch-periodic (100.0% fail rate in 6 executions)
• [8.x] 3 failures in pipeline elasticsearch-periodic-platform-support (50.0% fail rate in 6 executions)
• [8.x] 2 failures in pipeline elasticsearch-pull-request (3.4% fail rate in 58 executions)

Build Scans:

• elasticsearch-periodic #4367 / release-tests
• elasticsearch-periodic-platform-support #4341 / ubuntu-2004-aarch64_checkpart2_platform-support-arm
• elasticsearch-periodic #4363 / release-tests
• elasticsearch-periodic #4359 / release-tests
• elasticsearch-periodic-platform-support #4333 / opensuse-leap-15_platform-support-unix
• elasticsearch-periodic #4355 / release-tests
• elasticsearch-periodic-platform-support #4329 / amazonlinux-2023_platform-support-aws
• elasticsearch-pull-request #35442 / part-2
• elasticsearch-pull-request #35439 / part-2
• elasticsearch-periodic #4351 / release-tests

[elasticsearchmachine]

Pinging @elastic/es-search-relevance (Team:Search Relevance)

[elasticsearchmachine]

Pinging @elastic/search-eng (Team:SearchOrg)

[elasticsearchmachine]

Pinging @elastic/search-relevance (Team:Search - Relevance)

[andreidan]

@Mikep86 @davidkyle this seems related to #113563 ?

[Mikep86]

This test is failing because the required cluster feature is defined only when the DefaultElserFeatureFlag is enabled. See here: https://github.com/elastic/elasticsearch/blob/8.16/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/InferenceFeatures.java#L31-L33

Exclusion of this cluster feature is by design, the idea was to use its existence to make the YAML test conditional on the default ELSER feature flag. How do we do that without causing an IllegalArgumentException?

[Mikep86]

This test has been fixed in the 8.16 branch, so it should not be an issue for that release. Fixes in 8.17+ are pending; they are currently blocked by unrelated broken TSDB BwC tests.

[Mikep86]

Fixes have been merged into main and 8.x

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 2 consecutive failures in test test {p0=inference/30_semantic_text_inference/Calculates embeddings using the default ELSER 2 endpoint}
• [main] 2 failures in test test {p0=inference/30_semantic_text_inference/Calculates embeddings using the default ELSER 2 endpoint} (6.1% fail rate in 33 executions)

Build Scans:

• elasticsearch-periodic-platform-support #4654 / almalinux-8_platform-support-unix
• elasticsearch-intake #12325 / part2

[Mikep86]

... and it's failing again now, but this time due to timeouts. Can someone from the ML team take a look?

cc: @davidkyle @maxhniebergall

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[davidkyle]

The recent failures are timeouts where the model does not download and deploy in 60 seconds. I've opened #116247 to increase the client timeout