[clang][analysis] Fix flaky clang/test/Analysis/live-stmts.cpp test #126913

steakhal · 2025-02-12T13:38:01Z

Multiple people reported flaky bot failures tied to clang/test/Analysis/live-stmts.cpp I tried reproducing the flaky behavior on my Linux x86_64 system, but the tests appears to be stable in my context.

Only by looking at the failures reported, I could formulate a potential diagnosis.
The output always looked almost the same, except that the Exprs dumped per Basic block were shuffled compared to my expectation. This suggests to me some ordering issue.

If you look at the backing storage of blocksEndToLiveness[B].liveExprs,
it uses llvm::ImmutableSet<const Expr *>.
That container likely uses the pointer values as keys, thus the runtime values of the addresses influence the iteration order.

To fix this, before dumping, I sort the expressions by their "beginLocs". It should be efficient enough for a debug checker, where there is no performance constraint.

This should hopefully fix the flaky behavior on systems where ASLR works differently than (my) Linux system.

Hopefully fixes #126619
Hopefully fixes #126804

Multiple people reported flaky bot failures tied to clang/test/Analysis/live-stmts.cpp I tried reproducing the flaky behavior on my Linux x86_64 system, but the tests appears to be stable in my context. Only by looking at the failures reported, I could formulate a potential diagnosis. The output always looked almost the same, except that the Exprs dumped per Basic block were shuffled compared to my expectation. This suggests to me some ordering issue. If you look at the backing storage of `blocksEndToLiveness[B].liveExprs`, it uses `llvm::ImmutableSet<const Expr *>`. That container likely uses the pointer values as keys, thus the runtime values of the addresses influence the iteration order. To fix this, before dumping, I sort the expressions by their "beginLocs". It should be efficient enough for a debug checker, where there is no performance constraint. This should hopefully fix the flaky behavior on systems where ASLR works differently than (my) Linux system. Hopefully fixes llvm#126619 Hopefully fixes llvm#126804

llvmbot · 2025-02-12T13:38:23Z

@llvm/pr-subscribers-clang
@llvm/pr-subscribers-clang-static-analyzer-1

@llvm/pr-subscribers-clang-analysis

Author: Balazs Benics (steakhal)

Changes

Multiple people reported flaky bot failures tied to clang/test/Analysis/live-stmts.cpp I tried reproducing the flaky behavior on my Linux x86_64 system, but the tests appears to be stable in my context.

Only by looking at the failures reported, I could formulate a potential diagnosis.
The output always looked almost the same, except that the Exprs dumped per Basic block were shuffled compared to my expectation. This suggests to me some ordering issue.

If you look at the backing storage of blocksEndToLiveness[B].liveExprs,
it uses llvm::ImmutableSet<const Expr *>.
That container likely uses the pointer values as keys, thus the runtime values of the addresses influence the iteration order.

To fix this, before dumping, I sort the expressions by their "beginLocs". It should be efficient enough for a debug checker, where there is no performance constraint.

This should hopefully fix the flaky behavior on systems where ASLR works differently than (my) Linux system.

Hopefully fixes #126619
Hopefully fixes #126804

Full diff: https://github.com/llvm/llvm-project/pull/126913.diff

2 Files Affected:

(modified) clang/lib/Analysis/LiveVariables.cpp (+10-1)
(modified) clang/test/Analysis/live-stmts.cpp (+22-23)

diff --git a/clang/lib/Analysis/LiveVariables.cpp b/clang/lib/Analysis/LiveVariables.cpp
index 481932ee59c8e..af563702b77bf 100644
--- a/clang/lib/Analysis/LiveVariables.cpp
+++ b/clang/lib/Analysis/LiveVariables.cpp
@@ -16,7 +16,9 @@
 #include "clang/Analysis/AnalysisDeclContext.h"
 #include "clang/Analysis/CFG.h"
 #include "clang/Analysis/FlowSensitive/DataflowWorklist.h"
+#include "clang/Basic/SourceManager.h"
 #include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/STLExtras.h"
 #include "llvm/Support/raw_ostream.h"
 #include <algorithm>
 #include <optional>
@@ -662,12 +664,19 @@ void LiveVariables::dumpExprLiveness(const SourceManager &M) {
 }
 
 void LiveVariablesImpl::dumpExprLiveness(const SourceManager &M) {
+  auto ByBeginLoc = [&M](const Expr *L, const Expr *R) {
+    return M.isBeforeInTranslationUnit(L->getBeginLoc(), R->getBeginLoc());
+  };
+
   // Don't iterate over blockEndsToLiveness directly because it's not sorted.
   for (const CFGBlock *B : *analysisContext.getCFG()) {
 
     llvm::errs() << "\n[ B" << B->getBlockID()
                  << " (live expressions at block exit) ]\n";
-    for (const Expr *E : blocksEndToLiveness[B].liveExprs) {
+    std::vector<const Expr *> LiveExprs;
+    llvm::append_range(LiveExprs, blocksEndToLiveness[B].liveExprs);
+    llvm::sort(LiveExprs, ByBeginLoc);
+    for (const Expr *E : LiveExprs) {
       llvm::errs() << "\n";
       E->dump();
     }
diff --git a/clang/test/Analysis/live-stmts.cpp b/clang/test/Analysis/live-stmts.cpp
index 33b8d59305d3d..8034d3a30436e 100644
--- a/clang/test/Analysis/live-stmts.cpp
+++ b/clang/test/Analysis/live-stmts.cpp
@@ -1,6 +1,3 @@
-// Flaky on aarch64: http://llvm.org/PR126619
-// UNSUPPORTED: target=aarch64{{.*}}
-
 // RUN: %clang_analyze_cc1 -w -analyzer-checker=debug.DumpLiveExprs %s 2>&1\
 // RUN:   | FileCheck %s
 
@@ -29,34 +26,36 @@ int testThatDumperWorks(int x, int y, int z) {
 // CHECK-EMPTY:
 // CHECK: [ B2 (live expressions at block exit) ]
 // CHECK-EMPTY:
-// CHECK-NEXT: DeclRefExpr {{.*}} 'y' 'int'
-// CHECK-EMPTY:
-// CHECK-NEXT: DeclRefExpr {{.*}} 'z' 'int'
-// CHECK-EMPTY:
 // CHECK-NEXT: ImplicitCastExpr {{.*}} <IntegralToBoolean>
 // CHECK-NEXT: `-ImplicitCastExpr {{.*}} <LValueToRValue>
 // CHECK-NEXT:   `-DeclRefExpr {{.*}} 'x' 'int'
 // CHECK-EMPTY:
-// CHECK-EMPTY:
-// CHECK: [ B3 (live expressions at block exit) ]
-// CHECK-EMPTY:
 // CHECK-NEXT: DeclRefExpr {{.*}} 'y' 'int'
 // CHECK-EMPTY:
 // CHECK-NEXT: DeclRefExpr {{.*}} 'z' 'int'
 // CHECK-EMPTY:
+// CHECK-EMPTY:
+// CHECK: [ B3 (live expressions at block exit) ]
+// CHECK-EMPTY:
 // CHECK-NEXT: ImplicitCastExpr {{.*}} <IntegralToBoolean>
 // CHECK-NEXT: `-ImplicitCastExpr {{.*}} <LValueToRValue>
 // CHECK-NEXT:   `-DeclRefExpr {{.*}} 'x' 'int'
-// CHECK: [ B4 (live expressions at block exit) ]
 // CHECK-EMPTY:
 // CHECK-NEXT: DeclRefExpr {{.*}} 'y' 'int'
 // CHECK-EMPTY:
 // CHECK-NEXT: DeclRefExpr {{.*}} 'z' 'int'
 // CHECK-EMPTY:
+// CHECK-EMPTY:
+// CHECK: [ B4 (live expressions at block exit) ]
+// CHECK-EMPTY:
 // CHECK-NEXT: ImplicitCastExpr {{.*}} <IntegralToBoolean>
 // CHECK-NEXT: `-ImplicitCastExpr {{.*}} <LValueToRValue>
 // CHECK-NEXT:   `-DeclRefExpr {{.*}} 'x' 'int'
 // CHECK-EMPTY:
+// CHECK-NEXT: DeclRefExpr {{.*}} 'y' 'int'
+// CHECK-EMPTY:
+// CHECK-NEXT: DeclRefExpr {{.*}} 'z' 'int'
+// CHECK-EMPTY:
 // CHECK-EMPTY:
 // CHECK: [ B5 (live expressions at block exit) ]
 // CHECK-EMPTY:
@@ -226,15 +225,15 @@ int logicalOpInTernary(bool b) {
 // CHECK: ImplicitCastExpr {{.*}} '_Bool' <LValueToRValue>
 // CHECK: `-DeclRefExpr {{.*}} '_Bool' lvalue ParmVar {{.*}} 'b' '_Bool'
 // CHECK-EMPTY:
-// CHECK: ImplicitCastExpr {{.*}} '_Bool' <LValueToRValue>
-// CHECK: `-DeclRefExpr {{.*}} '_Bool' lvalue ParmVar {{.*}} 'b' '_Bool'
-// CHECK-EMPTY:
 // CHECK: BinaryOperator {{.*}} '_Bool' '||'
 // CHECK: |-ImplicitCastExpr {{.*}} '_Bool' <LValueToRValue>
 // CHECK: | `-DeclRefExpr {{.*}} '_Bool' lvalue ParmVar {{.*}} 'b' '_Bool'
 // CHECK: `-ImplicitCastExpr {{.*}} '_Bool' <LValueToRValue>
 // CHECK:   `-DeclRefExpr {{.*}} '_Bool' lvalue ParmVar {{.*}} 'b' '_Bool'
 // CHECK-EMPTY:
+// CHECK: ImplicitCastExpr {{.*}} '_Bool' <LValueToRValue>
+// CHECK: `-DeclRefExpr {{.*}} '_Bool' lvalue ParmVar {{.*}} 'b' '_Bool'
+// CHECK-EMPTY:
 // CHECK: IntegerLiteral {{.*}} 'int' 0
 // CHECK-EMPTY:
 // CHECK: IntegerLiteral {{.*}} 'int' 1
@@ -245,15 +244,15 @@ int logicalOpInTernary(bool b) {
 // CHECK: ImplicitCastExpr {{.*}} '_Bool' <LValueToRValue>
 // CHECK: `-DeclRefExpr {{.*}} '_Bool' lvalue ParmVar {{.*}} 'b' '_Bool'
 // CHECK-EMPTY:
-// CHECK: ImplicitCastExpr {{.*}} '_Bool' <LValueToRValue>
-// CHECK: `-DeclRefExpr {{.*}} '_Bool' lvalue ParmVar {{.*}} 'b' '_Bool'
-// CHECK-EMPTY:
 // CHECK: BinaryOperator {{.*}} '_Bool' '||'
 // CHECK: |-ImplicitCastExpr {{.*}} '_Bool' <LValueToRValue>
 // CHECK: | `-DeclRefExpr {{.*}} '_Bool' lvalue ParmVar {{.*}} 'b' '_Bool'
 // CHECK: `-ImplicitCastExpr {{.*}} '_Bool' <LValueToRValue>
 // CHECK:   `-DeclRefExpr {{.*}} '_Bool' lvalue ParmVar {{.*}} 'b' '_Bool'
 // CHECK-EMPTY:
+// CHECK: ImplicitCastExpr {{.*}} '_Bool' <LValueToRValue>
+// CHECK: `-DeclRefExpr {{.*}} '_Bool' lvalue ParmVar {{.*}} 'b' '_Bool'
+// CHECK-EMPTY:
 // CHECK: IntegerLiteral {{.*}} 'int' 0
 // CHECK-EMPTY:
 // CHECK: IntegerLiteral {{.*}} 'int' 1
@@ -264,15 +263,15 @@ int logicalOpInTernary(bool b) {
 // CHECK: ImplicitCastExpr {{.*}} '_Bool' <LValueToRValue>
 // CHECK: `-DeclRefExpr {{.*}} '_Bool' lvalue ParmVar {{.*}} 'b' '_Bool'
 // CHECK-EMPTY:
-// CHECK: ImplicitCastExpr {{.*}} '_Bool' <LValueToRValue>
-// CHECK: `-DeclRefExpr {{.*}} '_Bool' lvalue ParmVar {{.*}} 'b' '_Bool'
-// CHECK-EMPTY:
 // CHECK: BinaryOperator {{.*}} '_Bool' '||'
 // CHECK: |-ImplicitCastExpr {{.*}} '_Bool' <LValueToRValue>
 // CHECK: | `-DeclRefExpr {{.*}} '_Bool' lvalue ParmVar {{.*}} 'b' '_Bool'
 // CHECK: `-ImplicitCastExpr {{.*}} '_Bool' <LValueToRValue>
 // CHECK:   `-DeclRefExpr {{.*}} '_Bool' lvalue ParmVar {{.*}} 'b' '_Bool'
 // CHECK-EMPTY:
+// CHECK: ImplicitCastExpr {{.*}} '_Bool' <LValueToRValue>
+// CHECK: `-DeclRefExpr {{.*}} '_Bool' lvalue ParmVar {{.*}} 'b' '_Bool'
+// CHECK-EMPTY:
 // CHECK: IntegerLiteral {{.*}} 'int' 0
 // CHECK-EMPTY:
 // CHECK: IntegerLiteral {{.*}} 'int' 1
@@ -283,15 +282,15 @@ int logicalOpInTernary(bool b) {
 // CHECK: ImplicitCastExpr {{.*}} '_Bool' <LValueToRValue>
 // CHECK: `-DeclRefExpr {{.*}} '_Bool' lvalue ParmVar {{.*}} 'b' '_Bool'
 // CHECK-EMPTY:
-// CHECK: ImplicitCastExpr {{.*}} '_Bool' <LValueToRValue>
-// CHECK: `-DeclRefExpr {{.*}} '_Bool' lvalue ParmVar {{.*}} 'b' '_Bool'
-// CHECK-EMPTY:
 // CHECK: BinaryOperator {{.*}} '_Bool' '||'
 // CHECK: |-ImplicitCastExpr {{.*}} '_Bool' <LValueToRValue>
 // CHECK: | `-DeclRefExpr {{.*}} '_Bool' lvalue ParmVar {{.*}} 'b' '_Bool'
 // CHECK: `-ImplicitCastExpr {{.*}} '_Bool' <LValueToRValue>
 // CHECK:   `-DeclRefExpr {{.*}} '_Bool' lvalue ParmVar {{.*}} 'b' '_Bool'
 // CHECK-EMPTY:
+// CHECK: ImplicitCastExpr {{.*}} '_Bool' <LValueToRValue>
+// CHECK: `-DeclRefExpr {{.*}} '_Bool' lvalue ParmVar {{.*}} 'b' '_Bool'
+// CHECK-EMPTY:
 // CHECK: IntegerLiteral {{.*}} 'int' 0
 // CHECK-EMPTY:
 // CHECK: IntegerLiteral {{.*}} 'int' 1

steakhal · 2025-02-12T13:39:56Z

FYI @necto, #100745.

necto

thank you for fixing the test

mikaelholmen · 2025-02-13T06:47:12Z

Unfortunately the test still seems to be flaky. With EXPENSIVE_CHECKS I often see it fail on the RHEL8 x86_64 machines I use.
E.g.

FAIL: Clang :: Analysis/live-stmts.cpp (1 of 1)
******************** TEST 'Clang :: Analysis/live-stmts.cpp' FAILED ********************
Exit Code: 1

Command Output (stderr):
--
RUN: at line 1: /repo/uabelho/main-github/llvm/build-all-expensive/bin/clang -cc1 -internal-isystem /repo/uabelho/main-github/llvm/build-all-expensive/lib/clang/21/include -nostdsysteminc -analyze -analyzer-constraints=range -setup-static-analyzer -w -analyzer-checker=debug.DumpLiveExprs /repo/uabelho/main-github/clang/test/Analysis/live-stmts.cpp 2>&1   | /repo/uabelho/main-github/llvm/build-all-expensive/bin/FileCheck /repo/uabelho/main-github/clang/test/Analysis/live-stmts.cpp
+ /repo/uabelho/main-github/llvm/build-all-expensive/bin/clang -cc1 -internal-isystem /repo/uabelho/main-github/llvm/build-all-expensive/lib/clang/21/include -nostdsysteminc -analyze -analyzer-constraints=range -setup-static-analyzer -w -analyzer-checker=debug.DumpLiveExprs /repo/uabelho/main-github/clang/test/Analysis/live-stmts.cpp
+ /repo/uabelho/main-github/llvm/build-all-expensive/bin/FileCheck /repo/uabelho/main-github/clang/test/Analysis/live-stmts.cpp
/repo/uabelho/main-github/clang/test/Analysis/live-stmts.cpp:227:16: error: CHECK-EMPTY: is not on the line after the previous match
// CHECK-EMPTY:
               ^
<stdin>:160:1: note: 'next' match was here

^
<stdin>:157:81: note: previous match ended here
| `-DeclRefExpr 0x55e1600cec58 '_Bool' lvalue ParmVar 0x55e1600cead8 'b' '_Bool'
                                                                                ^
<stdin>:158:1: note: non-matching line after previous match is here
`-ImplicitCastExpr 0x55e1600cecb0 '_Bool' <LValueToRValue>
^

Input file: <stdin>
Check file: /repo/uabelho/main-github/clang/test/Analysis/live-stmts.cpp

-dump-input=help explains the following input dump.

Input was:
<<<<<<
           .
           .
           .
         155: BinaryOperator 0x55e1600cecc8 '_Bool' '||' 
         156: |-ImplicitCastExpr 0x55e1600cec98 '_Bool' <LValueToRValue> 
         157: | `-DeclRefExpr 0x55e1600cec58 '_Bool' lvalue ParmVar 0x55e1600cead8 'b' '_Bool' 
         158: `-ImplicitCastExpr 0x55e1600cecb0 '_Bool' <LValueToRValue> 
         159:  `-DeclRefExpr 0x55e1600cec78 '_Bool' lvalue ParmVar 0x55e1600cead8 'b' '_Bool' 
         160:  
empty:227     ! error: match on wrong line
         161: ImplicitCastExpr 0x55e1600cec98 '_Bool' <LValueToRValue> 
         162: `-DeclRefExpr 0x55e1600cec58 '_Bool' lvalue ParmVar 0x55e1600cead8 'b' '_Bool' 
         163:  
         164: ImplicitCastExpr 0x55e1600cecb0 '_Bool' <LValueToRValue> 
         165: `-DeclRefExpr 0x55e1600cec78 '_Bool' lvalue ParmVar 0x55e1600cead8 'b' '_Bool' 
           .
           .
           .
>>>>>>

--

********************
********************
Failed Tests (1):
  Clang :: Analysis/live-stmts.cpp

steakhal · 2025-02-13T06:53:55Z

Unfortunately the test still seems to be flaky. With EXPENSIVE_CHECKS I often see it fail on the RHEL8 x86_64 machines I use.
E.g.

FAIL: Clang :: Analysis/live-stmts.cpp (1 of 1)
******************** TEST 'Clang :: Analysis/live-stmts.cpp' FAILED ********************
Exit Code: 1

Command Output (stderr):
--
RUN: at line 1: /repo/uabelho/main-github/llvm/build-all-expensive/bin/clang -cc1 -internal-isystem /repo/uabelho/main-github/llvm/build-all-expensive/lib/clang/21/include -nostdsysteminc -analyze -analyzer-constraints=range -setup-static-analyzer -w -analyzer-checker=debug.DumpLiveExprs /repo/uabelho/main-github/clang/test/Analysis/live-stmts.cpp 2>&1   | /repo/uabelho/main-github/llvm/build-all-expensive/bin/FileCheck /repo/uabelho/main-github/clang/test/Analysis/live-stmts.cpp
+ /repo/uabelho/main-github/llvm/build-all-expensive/bin/clang -cc1 -internal-isystem /repo/uabelho/main-github/llvm/build-all-expensive/lib/clang/21/include -nostdsysteminc -analyze -analyzer-constraints=range -setup-static-analyzer -w -analyzer-checker=debug.DumpLiveExprs /repo/uabelho/main-github/clang/test/Analysis/live-stmts.cpp
+ /repo/uabelho/main-github/llvm/build-all-expensive/bin/FileCheck /repo/uabelho/main-github/clang/test/Analysis/live-stmts.cpp
/repo/uabelho/main-github/clang/test/Analysis/live-stmts.cpp:227:16: error: CHECK-EMPTY: is not on the line after the previous match
// CHECK-EMPTY:
               ^
<stdin>:160:1: note: 'next' match was here

^
<stdin>:157:81: note: previous match ended here
| `-DeclRefExpr 0x55e1600cec58 '_Bool' lvalue ParmVar 0x55e1600cead8 'b' '_Bool'
                                                                                ^
<stdin>:158:1: note: non-matching line after previous match is here
`-ImplicitCastExpr 0x55e1600cecb0 '_Bool' <LValueToRValue>
^

Input file: <stdin>
Check file: /repo/uabelho/main-github/clang/test/Analysis/live-stmts.cpp

-dump-input=help explains the following input dump.

Input was:
<<<<<<
           .
           .
           .
         155: BinaryOperator 0x55e1600cecc8 '_Bool' '||' 
         156: |-ImplicitCastExpr 0x55e1600cec98 '_Bool' <LValueToRValue> 
         157: | `-DeclRefExpr 0x55e1600cec58 '_Bool' lvalue ParmVar 0x55e1600cead8 'b' '_Bool' 
         158: `-ImplicitCastExpr 0x55e1600cecb0 '_Bool' <LValueToRValue> 
         159:  `-DeclRefExpr 0x55e1600cec78 '_Bool' lvalue ParmVar 0x55e1600cead8 'b' '_Bool' 
         160:  
empty:227     ! error: match on wrong line
         161: ImplicitCastExpr 0x55e1600cec98 '_Bool' <LValueToRValue> 
         162: `-DeclRefExpr 0x55e1600cec58 '_Bool' lvalue ParmVar 0x55e1600cead8 'b' '_Bool' 
         163:  
         164: ImplicitCastExpr 0x55e1600cecb0 '_Bool' <LValueToRValue> 
         165: `-DeclRefExpr 0x55e1600cec78 '_Bool' lvalue ParmVar 0x55e1600cead8 'b' '_Bool' 
           .
           .
           .
>>>>>>

--

********************
********************
Failed Tests (1):
  Clang :: Analysis/live-stmts.cpp

Oh cmmon. :(
Could you send me your nonpassing output, the whole one? @mikaelholmen

mikaelholmen · 2025-02-13T07:24:35Z

Unfortunately the test still seems to be flaky. With EXPENSIVE_CHECKS I often see it fail on the RHEL8 x86_64 machines I use.
E.g.

FAIL: Clang :: Analysis/live-stmts.cpp (1 of 1)
******************** TEST 'Clang :: Analysis/live-stmts.cpp' FAILED ********************
Exit Code: 1

Command Output (stderr):
--
RUN: at line 1: /repo/uabelho/main-github/llvm/build-all-expensive/bin/clang -cc1 -internal-isystem /repo/uabelho/main-github/llvm/build-all-expensive/lib/clang/21/include -nostdsysteminc -analyze -analyzer-constraints=range -setup-static-analyzer -w -analyzer-checker=debug.DumpLiveExprs /repo/uabelho/main-github/clang/test/Analysis/live-stmts.cpp 2>&1   | /repo/uabelho/main-github/llvm/build-all-expensive/bin/FileCheck /repo/uabelho/main-github/clang/test/Analysis/live-stmts.cpp
+ /repo/uabelho/main-github/llvm/build-all-expensive/bin/clang -cc1 -internal-isystem /repo/uabelho/main-github/llvm/build-all-expensive/lib/clang/21/include -nostdsysteminc -analyze -analyzer-constraints=range -setup-static-analyzer -w -analyzer-checker=debug.DumpLiveExprs /repo/uabelho/main-github/clang/test/Analysis/live-stmts.cpp
+ /repo/uabelho/main-github/llvm/build-all-expensive/bin/FileCheck /repo/uabelho/main-github/clang/test/Analysis/live-stmts.cpp
/repo/uabelho/main-github/clang/test/Analysis/live-stmts.cpp:227:16: error: CHECK-EMPTY: is not on the line after the previous match
// CHECK-EMPTY:
               ^
<stdin>:160:1: note: 'next' match was here

^
<stdin>:157:81: note: previous match ended here
| `-DeclRefExpr 0x55e1600cec58 '_Bool' lvalue ParmVar 0x55e1600cead8 'b' '_Bool'
                                                                                ^
<stdin>:158:1: note: non-matching line after previous match is here
`-ImplicitCastExpr 0x55e1600cecb0 '_Bool' <LValueToRValue>
^

Input file: <stdin>
Check file: /repo/uabelho/main-github/clang/test/Analysis/live-stmts.cpp

-dump-input=help explains the following input dump.

Input was:
<<<<<<
           .
           .
           .
         155: BinaryOperator 0x55e1600cecc8 '_Bool' '||' 
         156: |-ImplicitCastExpr 0x55e1600cec98 '_Bool' <LValueToRValue> 
         157: | `-DeclRefExpr 0x55e1600cec58 '_Bool' lvalue ParmVar 0x55e1600cead8 'b' '_Bool' 
         158: `-ImplicitCastExpr 0x55e1600cecb0 '_Bool' <LValueToRValue> 
         159:  `-DeclRefExpr 0x55e1600cec78 '_Bool' lvalue ParmVar 0x55e1600cead8 'b' '_Bool' 
         160:  
empty:227     ! error: match on wrong line
         161: ImplicitCastExpr 0x55e1600cec98 '_Bool' <LValueToRValue> 
         162: `-DeclRefExpr 0x55e1600cec58 '_Bool' lvalue ParmVar 0x55e1600cead8 'b' '_Bool' 
         163:  
         164: ImplicitCastExpr 0x55e1600cecb0 '_Bool' <LValueToRValue> 
         165: `-DeclRefExpr 0x55e1600cec78 '_Bool' lvalue ParmVar 0x55e1600cead8 'b' '_Bool' 
           .
           .
           .
>>>>>>

--

********************
********************
Failed Tests (1):
  Clang :: Analysis/live-stmts.cpp

Oh cmmon. :( Could you send me your nonpassing output, the whole one? @mikaelholmen

Yep
out.log

steakhal · 2025-02-13T08:25:24Z

What the heck, your logs refer to B7. But I can't see a 7th basic block anywhere :D

I'll just disable this test on all platforms, and try to debug this with the parties who reported flaky behavior.
Would you be up to pilot testing fixes @mikaelholmen if I would find a plausable theory and fix what's causing this flakyness?

mikaelholmen · 2025-02-13T08:33:02Z

@steakhal : Sure I can do some tests if you come up with a fix. Right now, with an EXPENSIVE_CHECKS build it fails in 30-40% of the runs for me so it should be pretty quick to see if a fix works or not.
I've never seen it fail without EXPENSIVE_CHECKS.

steakhal · 2025-02-13T09:33:58Z

@steakhal : Sure I can do some tests if you come up with a fix. Right now, with an EXPENSIVE_CHECKS build it fails in 30-40% of the runs for me so it should be pretty quick to see if a fix works or not. I've never seen it fail without EXPENSIVE_CHECKS.

I could reproduce the flakyness on my system using expensive checks. Thanks for the hint.

I had previous attempts for fixing this flaky test. Let's admit I failed so far, and disable this until we have a permanent fix. See the discussion at: llvm#126913 (comment)

I had previous attempts for fixing this flaky test. Let's admit I failed so far, and disable this until we have a permanent fix. See the discussion at: #126913 (comment)

…ky (#127034) I had previous attempts for fixing this flaky test. Let's admit I failed so far, and disable this until we have a permanent fix. See the discussion at: llvm/llvm-project#126913 (comment)

…lvm#126913) Multiple people reported flaky bot failures tied to `clang/test/Analysis/live-stmts.cpp` I tried reproducing the flaky behavior on my Linux x86_64 system, but the tests appears to be stable in my context. Only by looking at the failures reported, I could formulate a potential diagnosis. The output always looked almost the same, except that the Exprs dumped per Basic block were shuffled compared to my expectation. This suggests to me some ordering issue. If you look at the backing storage of `blocksEndToLiveness[B].liveExprs`, it uses `llvm::ImmutableSet<const Expr *>`. That container likely uses the pointer values as keys, thus the runtime values of the addresses influence the iteration order. To fix this, before dumping, I sort the expressions by their "beginLocs". It should be efficient enough for a debug checker, where there is no performance constraint. This should hopefully fix the flaky behavior on systems where ASLR works differently than (my) Linux system. Hopefully fixes llvm#126619 Hopefully fixes llvm#126804

…7034) I had previous attempts for fixing this flaky test. Let's admit I failed so far, and disable this until we have a permanent fix. See the discussion at: llvm#126913 (comment)

metaflow · 2025-02-14T06:11:44Z

For what it worth this change made it persistently failing in my setup.

steakhal · 2025-02-14T06:15:08Z

For what it worth this change made it persistently failing in my setup.

I'm not sure I follow you.
The test is disabled on all platforms unconditionally on main.
Since #127034

metaflow · 2025-02-14T09:30:28Z

For what it worth this change made it persistently failing in my setup.

I'm not sure I follow you. The test is disabled on all platforms unconditionally on main. Since #127034

yea, I know it was disabled. I meant that this very be25d61 made it constantly failing.

edit: fixed commit hash

whisperity · 2025-02-14T09:36:32Z

For what it worth this change made it persistently failing in my setup.

I'm not sure I follow you. The test is disabled on all platforms unconditionally on main. Since #127034

yea, I know it was disabled. I meant that this very 55f3df8 made it constantly failing.

The commit you linked belongs to a backend code change in LLVM, not this patch. Also, "unsupported platform" should show up as unsupported in LIT, not as failures. Are you using some custom LLVM_LIT_ARGS? If so, could you please share?

metaflow · 2025-02-14T10:25:12Z

oh sorry, I meant be25d61 :)

steakhal · 2025-02-14T10:36:16Z

oh sorry, I meant be25d61 :)

Let me know if this test still causes any issues on main. Otherwise, I'll consider this thread concluded.

…lvm#126913) Multiple people reported flaky bot failures tied to `clang/test/Analysis/live-stmts.cpp` I tried reproducing the flaky behavior on my Linux x86_64 system, but the tests appears to be stable in my context. Only by looking at the failures reported, I could formulate a potential diagnosis. The output always looked almost the same, except that the Exprs dumped per Basic block were shuffled compared to my expectation. This suggests to me some ordering issue. If you look at the backing storage of `blocksEndToLiveness[B].liveExprs`, it uses `llvm::ImmutableSet<const Expr *>`. That container likely uses the pointer values as keys, thus the runtime values of the addresses influence the iteration order. To fix this, before dumping, I sort the expressions by their "beginLocs". It should be efficient enough for a debug checker, where there is no performance constraint. This should hopefully fix the flaky behavior on systems where ASLR works differently than (my) Linux system. Hopefully fixes llvm#126619 Hopefully fixes llvm#126804

…7034) I had previous attempts for fixing this flaky test. Let's admit I failed so far, and disable this until we have a permanent fix. See the discussion at: llvm#126913 (comment)

…2nd attempt) In my previous attempt (llvm#126913) of fixing the flaky case was on a good track when I used the begin locations as a stable ordering. However, I forgot to consider the case when the begin locations are the same among the Exprs. In an `EXPENSIVE_CHECKS` build, arrays are randomly shuffled prior to sorting them. This exposed the flaky behavior much more often basically breaking the "stability" of the vector - as it should. To fix this, I I use this time `Expr::getID` for a stable ID for an Expr. Hopefully fixes llvm#126619 Hopefully fixes llvm#126804

…2nd attempt) (#127406) In my previous attempt (#126913) of fixing the flaky case was on a good track when I used the begin locations as a stable ordering. However, I forgot to consider the case when the begin locations are the same among the Exprs. In an `EXPENSIVE_CHECKS` build, arrays are randomly shuffled prior to sorting them. This exposed the flaky behavior much more often basically breaking the "stability" of the vector - as it should. Because of this, I had to revert the previous fix attempt in #127034. To fix this, I use this time `Expr::getID` for a stable ID for an Expr. Hopefully fixes #126619 Hopefully fixes #126804

steakhal · 2025-02-17T10:13:46Z

FYI I just merged my second attempt of enabling and fixing the flaky test. Let me know if it works. @metaflow @mikaelholmen

mikaelholmen · 2025-02-17T11:45:24Z

FYI I just merged my second attempt of enabling and fixing the flaky test. Let me know if it works. @metaflow @mikaelholmen

No failures so far :)

…lvm#126913) Multiple people reported flaky bot failures tied to `clang/test/Analysis/live-stmts.cpp` I tried reproducing the flaky behavior on my Linux x86_64 system, but the tests appears to be stable in my context. Only by looking at the failures reported, I could formulate a potential diagnosis. The output always looked almost the same, except that the Exprs dumped per Basic block were shuffled compared to my expectation. This suggests to me some ordering issue. If you look at the backing storage of `blocksEndToLiveness[B].liveExprs`, it uses `llvm::ImmutableSet<const Expr *>`. That container likely uses the pointer values as keys, thus the runtime values of the addresses influence the iteration order. To fix this, before dumping, I sort the expressions by their "beginLocs". It should be efficient enough for a debug checker, where there is no performance constraint. This should hopefully fix the flaky behavior on systems where ASLR works differently than (my) Linux system. Hopefully fixes llvm#126619 Hopefully fixes llvm#126804

…7034) I had previous attempts for fixing this flaky test. Let's admit I failed so far, and disable this until we have a permanent fix. See the discussion at: llvm#126913 (comment)

…2nd attempt) (llvm#127406) In my previous attempt (llvm#126913) of fixing the flaky case was on a good track when I used the begin locations as a stable ordering. However, I forgot to consider the case when the begin locations are the same among the Exprs. In an `EXPENSIVE_CHECKS` build, arrays are randomly shuffled prior to sorting them. This exposed the flaky behavior much more often basically breaking the "stability" of the vector - as it should. Because of this, I had to revert the previous fix attempt in llvm#127034. To fix this, I use this time `Expr::getID` for a stable ID for an Expr. Hopefully fixes llvm#126619 Hopefully fixes llvm#126804

…2nd attempt) (llvm#127406) In my previous attempt (llvm#126913) of fixing the flaky case was on a good track when I used the begin locations as a stable ordering. However, I forgot to consider the case when the begin locations are the same among the Exprs. In an `EXPENSIVE_CHECKS` build, arrays are randomly shuffled prior to sorting them. This exposed the flaky behavior much more often basically breaking the "stability" of the vector - as it should. Because of this, I had to revert the previous fix attempt in llvm#127034. To fix this, I use this time `Expr::getID` for a stable ID for an Expr. Hopefully fixes llvm#126619 Hopefully fixes llvm#126804 (cherry picked from commit f378e52)

steakhal added clang:static analyzer clang:analysis labels Feb 12, 2025

steakhal requested review from nico, Xazax-hun and NagyDonat February 12, 2025 13:38

llvmbot added the clang Clang issues not falling into any other category label Feb 12, 2025

steakhal mentioned this pull request Feb 12, 2025

Clang test Analysis/live-stmts.cpp randomly fails on MacOS #126804

Closed

necto approved these changes Feb 12, 2025

View reviewed changes

Xazax-hun approved these changes Feb 12, 2025

View reviewed changes

steakhal merged commit be25d61 into llvm:main Feb 12, 2025
12 checks passed

steakhal deleted the bb/fix-flaky-live-stmts-test branch February 12, 2025 15:07

steakhal mentioned this pull request Feb 13, 2025

[analyzer] Disable a flaky test while triaging why its flaky #127034

Merged

steakhal mentioned this pull request Feb 16, 2025

[clang][analysis] Fix flaky clang/test/Analysis/live-stmts.cpp test (2nd attempt) #127406

Merged

steakhal mentioned this pull request May 12, 2025

[clang][analysis] Fix flaky clang/test/Analysis/live-stmts.cpp test (2nd attempt) (#127406) #139591

Merged

[clang][analysis] Fix flaky clang/test/Analysis/live-stmts.cpp test #126913

[clang][analysis] Fix flaky clang/test/Analysis/live-stmts.cpp test #126913

Uh oh!

Conversation

steakhal commented Feb 12, 2025

Uh oh!

llvmbot commented Feb 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

steakhal commented Feb 12, 2025

Uh oh!

necto left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mikaelholmen commented Feb 13, 2025

Uh oh!

steakhal commented Feb 13, 2025

Uh oh!

mikaelholmen commented Feb 13, 2025

Uh oh!

steakhal commented Feb 13, 2025

Uh oh!

mikaelholmen commented Feb 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

steakhal commented Feb 13, 2025

Uh oh!

metaflow commented Feb 14, 2025

Uh oh!

steakhal commented Feb 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

metaflow commented Feb 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

whisperity commented Feb 14, 2025

Uh oh!

metaflow commented Feb 14, 2025

Uh oh!

steakhal commented Feb 14, 2025

Uh oh!

steakhal commented Feb 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mikaelholmen commented Feb 17, 2025

Uh oh!

Uh oh!

llvmbot commented Feb 12, 2025 •

edited

Loading

mikaelholmen commented Feb 13, 2025 •

edited

Loading

steakhal commented Feb 14, 2025 •

edited

Loading

metaflow commented Feb 14, 2025 •

edited

Loading

steakhal commented Feb 17, 2025 •

edited

Loading