Change severity of negative stats messages from WARN to DEBUG

[danhermann]

Because the incidence of negative stats values reported by the JVM has increased significantly as of JDK14 (at least partially due to this JVM bug: https://bugs.openjdk.java.net/browse/JDK-8242480), this message is ending up in logs fairly frequently and causing some concern and confusion among users. The message is largely harmless because the only effect is that the stats value, usually either free memory or free swap space, reported as negative by the OS will instead be recorded as zero by ES and no user action is necessary or possible. Should the JVM bug be fixed in the future, upgrading to an unaffected JVM may reduce the incidence of these occurrences.

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/Stats)

[danhermann]

@elasticmachine update branch

[jbaiera]

LGTM

[danhermann]

Thanks, @jbaiera!

[RickyLau]

Why is this patch not backported to 6.8

[lhzw]

elasticsearch 6.8.18 docker image, on one of my clusters, repeats this again and again, memory locked, so annoying:

curl -s 'localhost:9200/_nodes?filter_path=**.mlockall' | jq
{
  "nodes": {
    "NHmeHf3VTqWzHYk8qD4iDg": {
      "process": {
        "mlockall": true
      }
    },
    "hU4oOeWcQSCtfJe9yTrmFQ": {
      "process": {
        "mlockall": true
      }
    }
  }
}

root@cluster3:20211109 17:03:11:itp# kubectl logs --tail 20 -f es-2
[2021-11-09T09:00:00,825][WARN ][o.e.m.o.O.Swap           ] [hU4oOeW] cannot compute used swap when total swap is 0 and free swap is 0
[2021-11-09T09:00:10,697][WARN ][o.e.m.o.O.Swap           ] [hU4oOeW] cannot compute used swap when total swap is 0 and free swap is 0
[2021-11-09T09:00:20,671][WARN ][o.e.m.o.O.Swap           ] [hU4oOeW] cannot compute used swap when total swap is 0 and free swap is 0
[2021-11-09T09:00:30,934][WARN ][o.e.m.o.O.Swap           ] [hU4oOeW] cannot compute used swap when total swap is 0 and free swap is 0
[2021-11-09T09:00:40,653][WARN ][o.e.m.o.O.Swap           ] [hU4oOeW] cannot compute used swap when total swap is 0 and free swap is 0
[2021-11-09T09:00:50,677][WARN ][o.e.m.o.O.Swap           ] [hU4oOeW] cannot compute used swap when total swap is 0 and free swap is 0
[2021-11-09T09:01:01,043][WARN ][o.e.m.o.O.Swap           ] [hU4oOeW] cannot compute used swap when total swap is 0 and free swap is 0
[2021-11-09T09:01:10,678][WARN ][o.e.m.o.O.Swap           ] [hU4oOeW] cannot compute used swap when total swap is 0 and free swap is 0
[2021-11-09T09:01:20,669][WARN ][o.e.m.o.O.Swap           ] [hU4oOeW] cannot compute used swap when total swap is 0 and free swap is 0
[2021-11-09T09:01:30,946][WARN ][o.e.m.o.O.Swap           ] [hU4oOeW] cannot compute used swap when total swap is 0 and free swap is 0
[2021-11-09T09:01:40,677][WARN ][o.e.m.o.O.Swap           ] [hU4oOeW] cannot compute used swap when total swap is 0 and free swap is 0
[2021-11-09T09:01:50,682][WARN ][o.e.m.o.O.Swap           ] [hU4oOeW] cannot compute used swap when total swap is 0 and free swap is 0
[2021-11-09T09:02:00,964][WARN ][o.e.m.o.O.Swap           ] [hU4oOeW] cannot compute used swap when total swap is 0 and free swap is 0
[2021-11-09T09:02:10,695][WARN ][o.e.m.o.O.Swap           ] [hU4oOeW] cannot compute used swap when total swap is 0 and free swap is 0
[2021-11-09T09:02:20,673][WARN ][o.e.m.o.O.Swap           ] [hU4oOeW] cannot compute used swap when total swap is 0 and free swap is 0
[2021-11-09T09:02:31,013][WARN ][o.e.m.o.O.Swap           ] [hU4oOeW] cannot compute used swap when total swap is 0 and free swap is 0
[2021-11-09T09:02:40,695][WARN ][o.e.m.o.O.Swap           ] [hU4oOeW] cannot compute used swap when total swap is 0 and free swap is 0
[2021-11-09T09:02:50,737][WARN ][o.e.m.o.O.Swap           ] [hU4oOeW] cannot compute used swap when total swap is 0 and free swap is 0
[2021-11-09T09:03:00,997][WARN ][o.e.m.o.O.Swap           ] [hU4oOeW] cannot compute used swap when total swap is 0 and free swap is 0
[2021-11-09T09:03:10,707][WARN ][o.e.m.o.O.Swap           ] [hU4oOeW] cannot compute used swap when total swap is 0 and free swap is 0
^C                                                                                                                                                  

[dalei2019]

[2021-11-09T09:00:10,697][WARN ][o.e.m.o.O.Swap ] [hU4oOeW] cannot compute used swap when total swap is 0 and free swap is 0

same issue!

[brianlukoff]

I'm also seeing this in a 6.8.21 cloud deployment.

[StijnArnauts]

We are suffering from this as well on a 6.8 ElasticCloud environment

[danhermann]

The 6.8.22 and later releases of Elasticsearch have this change.