[CI] CoreWithMultipleProjectsClientYamlTestSuiteIT test {yaml=search.vectors/41_knn_search_bbq_hnsw/Vector rescoring has same scoring as exact search for kNN section} failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic #6789 / openjdk21_entitlements_true_checkpart2_java-matrix
• elasticsearch-pull-request #60458 / part-2
• elasticsearch-pull-request #60204 / part-2

Reproduction Line:

./gradlew ":x-pack:qa:multi-project:core-rest-tests-with-multiple-projects:yamlRestTest" --tests "org.elasticsearch.multiproject.test.CoreWithMultipleProjectsClientYamlTestSuiteIT" -Dtests.method="test {yaml=search.vectors/41_knn_search_bbq_hnsw/Vector rescoring has same scoring as exact search for kNN section}" -Dtests.seed=DF1DDE5C4B469CD5 -Dtests.jvm.argline="-Des.entitlements.enabled=true" -Dtests.locale=ksf-Latn-CM -Dtests.timezone=Etc/GMT+3 -Druntime.java=21

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: Failure at [search.vectors/41_knn_search_bbq_hnsw:171]: 
Expected: <3>
     but: was <2>

Issue Reasons:

• [main] 3 failures in test test {yaml=search.vectors/41_knn_search_bbq_hnsw/Vector rescoring has same scoring as exact search for kNN section} (1.1% fail rate in 278 executions)
• [main] 2 failures in step part-2 (1.7% fail rate in 116 executions)
• [main] 2 failures in pipeline elasticsearch-pull-request (1.7% fail rate in 118 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 3 failures in test test {yaml=search.vectors/41_knn_search_bbq_hnsw/Vector rescoring has same scoring as exact search for kNN section} (1.1% fail rate in 278 executions)
• [main] 2 failures in step part-2 (1.7% fail rate in 116 executions)
• [main] 2 failures in pipeline elasticsearch-pull-request (1.7% fail rate in 118 executions)

Build Scans:

• elasticsearch-periodic #6789 / openjdk21_entitlements_true_checkpart2_java-matrix
• elasticsearch-pull-request #60458 / part-2
• elasticsearch-pull-request #60204 / part-2

[elasticsearchmachine]

Pinging @elastic/es-delivery (Team:Delivery)

[elasticsearchmachine]

Pinging @elastic/es-search-relevance (Team:Search Relevance)

[thecoop]

@carlosdelest This is from #116663 - it doesnt reproduce locally...

[benwtrent]

This is really weird. There is something wrong here. This failure is on the scriptScore query, not returning 3 hits. This indicates a very weird race condition bug.

This also means that from the bbq search, there were indeed 3 hits. But now at the script query search there are only 2?!

[benwtrent]

I think we are just missing a refresh call after force merge. That might be enough.

[benwtrent]

@tvernum You might be interested in this test failure. Its weird that the script query didn't "see" all the docs.

I am adding a refresh here to just double check everything. But if this fails a bunch of times again but only with the CoreWithMultipleProjectsClientYamlTestSuiteIT runner, y'all might have an interesting bug ;)

[tvernum]

From memory, the MP tests run with 2 nodes, while the standard YAML executor has a single node. Is it possible that node count would make a difference?

[benwtrent]

Is it possible that node count would make a difference?

I really shouldn't both queries are hitting the same index and are attempting to match all docs. Its really strange that the first query sees 3 docs, but then the next query only sees 2. Both of them are "match_all".

I am doing a "refresh" now and maybe this will help.

Also, CCS, CCR tests all will end up using more nodes as well and split the index as well.

[benwtrent]

Gonna go ahead and close: #125255

If this fails a bunch again, the robot will let us know quickly ;)