Stats based on LongAdder can return unexpected values

[ywelsch]

We expect certain stats values to be non-negative when serializing them over the wire. While investigating a test failure (#52406), I noticed that, due to concurrency, counts from LongAdder can be inexact, and even be negative even though in a properly serialized execution no such negative value should be observable.

Here's a test illustrating the issue:

    public void testConcurrentLongAdderAccess() throws InterruptedException {
        AtomicBoolean stopped = new AtomicBoolean();
        LongAdder longAdder = new LongAdder();

        Thread[] writers = new Thread[10];
        for (int i = 0; i < writers.length; i++) {
            writers[i] = new Thread(() -> {
                while(stopped.get() == false) {
                    longAdder.increment();
                    longAdder.decrement();
                }
            });
            writers[i].start();
        }

        Thread[] readers = new Thread[10];
        for (int i = 0; i < writers.length; i++) {
            readers[i] = new Thread(() -> {
                while(stopped.get() == false) {
                    long val = longAdder.longValue();
                    assert val >= 0L : "expected positive but got " + val;
                }
            });
            readers[i].start();
        }

        Thread.sleep(10000);

        stopped.set(true);

        for (int i = 0; i < writers.length; i++) {
            writers[i].join();
        }

        for (int i = 0; i < readers.length; i++) {
            readers[i].join();
        }
    }

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/Stats)

[jasontedor]

It appears that is documented. The Javadocs for LongAdder#longValue point to the Javadocs for LongAdder#sum:

    /**
     * Equivalent to {@link #sum}.
     *
     * @return the sum
     */
    public long longValue();

And here's the Javadocs for LongAdder#sum:

    /**
     * Returns the current sum.  The returned value is <em>NOT</em> an
     * atomic snapshot; invocation in the absence of concurrent
     * updates returns an accurate result, but concurrent updates that
     * occur while the sum is being calculated might not be
     * incorporated.
     *
     * @return the sum
     */
    public long sum();