[CI] DocsClientYamlTestSuiteIT test {yaml=reference/cat/nodes/line_361} failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic-platform-support #6878 / windows-2019_checkpart1_platform-support-windows
• elasticsearch-periodic-platform-support #6878 / windows-2022_checkpart1_platform-support-windows
• elasticsearch-pull-request #61486 / part-1-windows
• elasticsearch-periodic-platform-support #6871 / windows-2019_checkpart1_platform-support-windows
• elasticsearch-periodic-platform-support #6871 / windows-2022_checkpart1_platform-support-windows
• elasticsearch-pull-request #61413 / part-1-windows
• elasticsearch-pull-request #61413 / part-1-windows
• elasticsearch-periodic-platform-support #6864 / windows-2022_checkpart1_platform-support-windows
• elasticsearch-periodic-platform-support #6864 / windows-2019_checkpart1_platform-support-windows
• elasticsearch-periodic-platform-support #6857 / windows-2019_checkpart1_platform-support-windows

Reproduction Line:

gradlew ":docs:yamlRestTest" --tests "org.elasticsearch.smoketest.DocsClientYamlTestSuiteIT.test {yaml=reference/cat/nodes/line_361}" -Dtests.seed=245967704EFCFB2C -Dtests.locale=rn-BI -Dtests.timezone=ACT -Druntime.java=24

Applicable branches:
8.x

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: Failure at [reference/cat/nodes:15]: field [$body] was expected to match the provided regex but didn't
Expected: ip        \s+heap.percent \s+ram.percent \s+cpu \s+load_1m \s+load_5m \s+load_15m \s+node.role \s+master \s+name\s* 127.0.0.1           \s+\d+ \s+\d+ \s+\d+    \s+(\d+\.\d+( \s+\d+\.\d+ \s+(\d+\.\d+)?)?)?                  \s+.+       \s+[*]      \s+.+\s*
     but: was "ip        heap.percent ram.percent cpu load_1m load_5m load_15m node.role   master name\n127.0.0.1           59          39  -1                          cdfhilmrstw *      node-0\n"

Issue Reasons:

• [8.x] 2 consecutive failures in test test {yaml=reference/cat/nodes/line_361}
• [8.x] 8 consecutive failures in step windows-2019_checkpart1_platform-support-windows
• [8.x] 9 consecutive failures in step windows-2022_checkpart1_platform-support-windows
• [8.x] 9 consecutive failures in step part-1-windows
• [8.x] 26 failures in test test {yaml=reference/cat/nodes/line_361} (6.5% fail rate in 401 executions)
• [8.x] 8 failures in step windows-2019_checkpart1_platform-support-windows (100.0% fail rate in 8 executions)
• [8.x] 9 failures in step windows-2022_checkpart1_platform-support-windows (100.0% fail rate in 9 executions)
• [8.x] 9 failures in step part-1-windows (100.0% fail rate in 9 executions)
• [8.x] 9 failures in pipeline elasticsearch-periodic-platform-support (100.0% fail rate in 9 executions)
• [8.x] 5 failures in pipeline elasticsearch-pull-request (8.6% fail rate in 58 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch 9.0

Mute Reasons:

• [9.0] 2 failures in test test {yaml=reference/cat/nodes/line_361} (1.7% fail rate in 118 executions)

Build Scans:

• elasticsearch-periodic-platform-support #6794 / windows-2022_checkpart1_platform-support-windows
• elasticsearch-pull-request #60741 / part-1-windows

[elasticsearchmachine]

Pinging @elastic/es-delivery (Team:Delivery)

[elasticsearchmachine]

This has been muted on branch 8.18

Mute Reasons:

• [8.18] 6 consecutive failures in step windows-2019_checkpart1_platform-support-windows
• [8.18] 6 consecutive failures in step part-1-windows
• [8.18] 5 consecutive failures in step windows-2022_checkpart1_platform-support-windows
• [8.18] 17 failures in test test {yaml=reference/cat/nodes/line_361} (7.2% fail rate in 236 executions)
• [8.18] 6 failures in step windows-2019_checkpart1_platform-support-windows (100.0% fail rate in 6 executions)
• [8.18] 6 failures in step part-1-windows (100.0% fail rate in 6 executions)
• [8.18] 5 failures in step windows-2022_checkpart1_platform-support-windows (100.0% fail rate in 5 executions)
• [8.18] 6 failures in pipeline elasticsearch-periodic-platform-support (100.0% fail rate in 6 executions)
• [8.18] 4 failures in pipeline elasticsearch-pull-request (12.1% fail rate in 33 executions)

Build Scans:

• elasticsearch-periodic-platform-support #6856 / windows-2019_checkpart1_platform-support-windows
• elasticsearch-pull-request #61414 / part-1-windows
• elasticsearch-pull-request #61372 / part-1-windows
• elasticsearch-pull-request #61372 / part-1-windows
• elasticsearch-pull-request #61335 / part-1-windows
• elasticsearch-pull-request #61335 / part-1-windows
• elasticsearch-pull-request #61298 / part-1-windows
• elasticsearch-periodic-platform-support #6844 / windows-2019_checkpart1_platform-support-windows
• elasticsearch-periodic-platform-support #6844 / windows-2022_checkpart1_platform-support-windows
• elasticsearch-periodic-platform-support #6834 / windows-2019_checkpart1_platform-support-windows

[elasticsearchmachine]

This has been muted on branch 8.x

Mute Reasons:

• [8.x] 9 consecutive failures in step windows-2022_checkpart1_platform-support-windows
• [8.x] 9 consecutive failures in step part-1-windows
• [8.x] 7 consecutive failures in step windows-2019_checkpart1_platform-support-windows
• [8.x] 25 failures in test test {yaml=reference/cat/nodes/line_361} (6.3% fail rate in 400 executions)
• [8.x] 9 failures in step windows-2022_checkpart1_platform-support-windows (100.0% fail rate in 9 executions)
• [8.x] 9 failures in step part-1-windows (100.0% fail rate in 9 executions)
• [8.x] 7 failures in step windows-2019_checkpart1_platform-support-windows (100.0% fail rate in 7 executions)
• [8.x] 9 failures in pipeline elasticsearch-periodic-platform-support (100.0% fail rate in 9 executions)
• [8.x] 5 failures in pipeline elasticsearch-pull-request (8.6% fail rate in 58 executions)

Build Scans:

• elasticsearch-periodic-platform-support #6878 / windows-2022_checkpart1_platform-support-windows
• elasticsearch-pull-request #61486 / part-1-windows
• elasticsearch-periodic-platform-support #6871 / windows-2019_checkpart1_platform-support-windows
• elasticsearch-periodic-platform-support #6871 / windows-2022_checkpart1_platform-support-windows
• elasticsearch-pull-request #61413 / part-1-windows
• elasticsearch-pull-request #61413 / part-1-windows
• elasticsearch-periodic-platform-support #6864 / windows-2022_checkpart1_platform-support-windows
• elasticsearch-periodic-platform-support #6864 / windows-2019_checkpart1_platform-support-windows
• elasticsearch-periodic-platform-support #6857 / windows-2019_checkpart1_platform-support-windows
• elasticsearch-periodic-platform-support #6857 / windows-2022_checkpart1_platform-support-windows

[nielsbauman]

Reopening this because the failure in the Failure Message is still relevant.

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[ldematte]

This never worked with OpenJDK on Windows, but we bundle the Oracle JDK (at least on Windows), and we use that for tests launched via gradle.

This is the OpenJDK implementation:

// Windows doesn't provide a loadavg primitive so this is stubbed out for now.
// It does have primitives (PDH API) to get CPU usage and run queue length.
// "\\Processor(_Total)\\% Processor Time", "\\System\\Processor Queue Length"
// If we wanted to implement loadavg on Windows, we have a few options:
//
// a) Query CPU usage and run queue length and "fake" an answer by
//    returning the CPU usage if it's under 100%, and the run queue
//    length otherwise.  It turns out that querying is pretty slow
//    on Windows, on the order of 200 microseconds on a fast machine.
//    Note that on the Windows the CPU usage value is the % usage
//    since the last time the API was called (and the first call
//    returns 100%), so we'd have to deal with that as well.
//
// b) Sample the "fake" answer using a sampling thread and store
//    the answer in a global variable.  The call to loadavg would
//    just return the value of the global, avoiding the slow query.
//
// c) Sample a better answer using exponential decay to smooth the
//    value.  This is basically the algorithm used by UNIX kernels.
//
// Note that sampling thread starvation could affect both (b) and (c).
int os::loadavg(double loadavg[], int nelem) {
  return -1;
}

Apparently, the Oracle JDK has a different (more complete, proprietary?) management bean for Windows.
But there is no Oracle JDK 24 yet, so we always use the OpenJDK 24 in tests now, and this test fails on Windows now.

I think it's reasonable to adjust the test to allow -1. We wouldn't catch cases where the CPU load percentage is unexpectedly -1, but I don't think these CAT tests should be the ones asserting that.

[ldematte]

This is not reproducible anymore, as we are transitioning to the new doc system; as we discussed on Slack, the fix would be in the test: -1 is an acceptable value, and we already accept it in related cat API tests (see for example rest-api-spec/test/cat.nodes/10_basic.yml)

[nielsbauman]

@ldematte should we add the -1 fix on the 8.x branches to prevent the bot from reopening this (and related) issues?

[ldematte]

The only branch that still have the old docs is 8.18. We can add the fix to that branch if we want.