clang/test/Analysis/live-stmts.cpp frequently flaky on aarch64 #126619

nico · 2025-02-10T23:00:38Z

I reported a failure of clang/test/Analysis/live-stmts.cpp here: #125840 (comment)

@nikic replied:

Pretty sure I see spurious failures of that test on aarch64 all the time.

So I'm filing this, to make sure we have an issue for tracking the flaky test.

failure log

-- Testing: 20876 tests, 8 workers --
Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90
FAIL: Clang :: Analysis/live-stmts.cpp (12643 of 20876)
******************** TEST 'Clang :: Analysis/live-stmts.cpp' FAILED ********************
Exit Code: 1

Command Output (stderr):
--
RUN: at line 1: /Users/thakis/src/llvm-project/out/gn/bin/clang -cc1 -internal-isystem /Users/thakis/src/llvm-project/out/gn/lib/clang/21/include -nostdsysteminc -analyze -analyzer-constraints=range -setup-static-analyzer -w -analyzer-checker=debug.DumpLiveExprs /Users/thakis/src/llvm-project/clang/test/Analysis/live-stmts.cpp 2>&1   | /Users/thakis/src/llvm-project/out/gn/bin/FileCheck /Users/thakis/src/llvm-project/clang/test/Analysis/live-stmts.cpp
+ /Users/thakis/src/llvm-project/out/gn/bin/clang -cc1 -internal-isystem /Users/thakis/src/llvm-project/out/gn/lib/clang/21/include -nostdsysteminc -analyze -analyzer-constraints=range -setup-static-analyzer -w -analyzer-checker=debug.DumpLiveExprs /Users/thakis/src/llvm-project/clang/test/Analysis/live-stmts.cpp
+ /Users/thakis/src/llvm-project/out/gn/bin/FileCheck /Users/thakis/src/llvm-project/clang/test/Analysis/live-stmts.cpp
/Users/thakis/src/llvm-project/clang/test/Analysis/live-stmts.cpp:239:16: error: CHECK-EMPTY: is not on the line after the previous match
// CHECK-EMPTY:
               ^
<stdin>:180:1: note: 'next' match was here

^
<stdin>:177:1: note: previous match ended here

^
<stdin>:178:1: note: non-matching line after previous match is here
ImplicitCastExpr 0x11280d178 '_Bool' <LValueToRValue>
^

Input file: <stdin>
Check file: /Users/thakis/src/llvm-project/clang/test/Analysis/live-stmts.cpp

-dump-input=help explains the following input dump.

Input was:
<<<<<<
           .
           .
           .
         175:  
         176: IntegerLiteral 0x110810820 'int' 1 
         177:  
         178: ImplicitCastExpr 0x11280d178 '_Bool' <LValueToRValue> 
         179: `-DeclRefExpr 0x11280d138 '_Bool' lvalue ParmVar 0x11280cfb8 'b' '_Bool' 
         180:  
empty:239     ! error: match on wrong line
         181: ImplicitCastExpr 0x11280d190 '_Bool' <LValueToRValue> 
         182: `-DeclRefExpr 0x11280d158 '_Bool' lvalue ParmVar 0x11280cfb8 'b' '_Bool' 
         183:  
         184: BinaryOperator 0x11280d1a8 '_Bool' '||' 
         185: |-ImplicitCastExpr 0x11280d178 '_Bool' <LValueToRValue> 
           .
           .
           .
>>>>>>

--

********************
Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90..
********************
Failed Tests (1):
  Clang :: Analysis/live-stmts.cpp

The text was updated successfully, but these errors were encountered:

llvmbot · 2025-02-10T23:00:55Z

@llvm/issue-subscribers-clang-static-analyzer

Author: Nico Weber (nico)

I reported a failure of clang/test/Analysis/live-stmts.cpp here: https://github.com//pull/125840#issuecomment-2648420648

@nikic replied:

> Pretty sure I see spurious failures of that test on aarch64 all the time.

So I'm filing this, to make sure we have an issue for tracking the flaky test.

Failure log, from #125840 (comment):

<details>
<summary>failure log</summary

-- Testing: 20876 tests, 8 workers --
Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90
FAIL: Clang :: Analysis/live-stmts.cpp (12643 of 20876)
******************** TEST 'Clang :: Analysis/live-stmts.cpp' FAILED ********************
Exit Code: 1

Command Output (stderr):
--
RUN: at line 1: /Users/thakis/src/llvm-project/out/gn/bin/clang -cc1 -internal-isystem /Users/thakis/src/llvm-project/out/gn/lib/clang/21/include -nostdsysteminc -analyze -analyzer-constraints=range -setup-static-analyzer -w -analyzer-checker=debug.DumpLiveExprs /Users/thakis/src/llvm-project/clang/test/Analysis/live-stmts.cpp 2&gt;&amp;1   | /Users/thakis/src/llvm-project/out/gn/bin/FileCheck /Users/thakis/src/llvm-project/clang/test/Analysis/live-stmts.cpp
+ /Users/thakis/src/llvm-project/out/gn/bin/clang -cc1 -internal-isystem /Users/thakis/src/llvm-project/out/gn/lib/clang/21/include -nostdsysteminc -analyze -analyzer-constraints=range -setup-static-analyzer -w -analyzer-checker=debug.DumpLiveExprs /Users/thakis/src/llvm-project/clang/test/Analysis/live-stmts.cpp
+ /Users/thakis/src/llvm-project/out/gn/bin/FileCheck /Users/thakis/src/llvm-project/clang/test/Analysis/live-stmts.cpp
/Users/thakis/src/llvm-project/clang/test/Analysis/live-stmts.cpp:239:16: error: CHECK-EMPTY: is not on the line after the previous match
// CHECK-EMPTY:
               ^
&lt;stdin&gt;:180:1: note: 'next' match was here

^
&lt;stdin&gt;:177:1: note: previous match ended here

^
&lt;stdin&gt;:178:1: note: non-matching line after previous match is here
ImplicitCastExpr 0x11280d178 '_Bool' &lt;LValueToRValue&gt;
^

Input file: &lt;stdin&gt;
Check file: /Users/thakis/src/llvm-project/clang/test/Analysis/live-stmts.cpp

-dump-input=help explains the following input dump.

Input was:
&lt;&lt;&lt;&lt;&lt;&lt;
           .
           .
           .
         175:  
         176: IntegerLiteral 0x110810820 'int' 1 
         177:  
         178: ImplicitCastExpr 0x11280d178 '_Bool' &lt;LValueToRValue&gt; 
         179: `-DeclRefExpr 0x11280d138 '_Bool' lvalue ParmVar 0x11280cfb8 'b' '_Bool' 
         180:  
empty:239     ! error: match on wrong line
         181: ImplicitCastExpr 0x11280d190 '_Bool' &lt;LValueToRValue&gt; 
         182: `-DeclRefExpr 0x11280d158 '_Bool' lvalue ParmVar 0x11280cfb8 'b' '_Bool' 
         183:  
         184: BinaryOperator 0x11280d1a8 '_Bool' '||' 
         185: |-ImplicitCastExpr 0x11280d178 '_Bool' &lt;LValueToRValue&gt; 
           .
           .
           .
&gt;&gt;&gt;&gt;&gt;&gt;

--

********************
Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90..
********************
Failed Tests (1):
  Clang :: Analysis/live-stmts.cpp

</details>

See #126619 and discussion on #125840

See llvm/llvm-project#126619 and discussion on llvm/llvm-project#125840

llvmbot · 2025-02-11T12:59:14Z

@llvm/issue-subscribers-clang-static-analyzer

Author: Nico Weber (nico)

I reported a failure of clang/test/Analysis/live-stmts.cpp here: https://github.com//pull/125840#issuecomment-2648420648

@nikic replied:

> Pretty sure I see spurious failures of that test on aarch64 all the time.

So I'm filing this, to make sure we have an issue for tracking the flaky test.

Failure log, from #125840 (comment):

<details>
<summary>failure log</summary

-- Testing: 20876 tests, 8 workers --
Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90
FAIL: Clang :: Analysis/live-stmts.cpp (12643 of 20876)
******************** TEST 'Clang :: Analysis/live-stmts.cpp' FAILED ********************
Exit Code: 1

Command Output (stderr):
--
RUN: at line 1: /Users/thakis/src/llvm-project/out/gn/bin/clang -cc1 -internal-isystem /Users/thakis/src/llvm-project/out/gn/lib/clang/21/include -nostdsysteminc -analyze -analyzer-constraints=range -setup-static-analyzer -w -analyzer-checker=debug.DumpLiveExprs /Users/thakis/src/llvm-project/clang/test/Analysis/live-stmts.cpp 2&gt;&amp;1   | /Users/thakis/src/llvm-project/out/gn/bin/FileCheck /Users/thakis/src/llvm-project/clang/test/Analysis/live-stmts.cpp
+ /Users/thakis/src/llvm-project/out/gn/bin/clang -cc1 -internal-isystem /Users/thakis/src/llvm-project/out/gn/lib/clang/21/include -nostdsysteminc -analyze -analyzer-constraints=range -setup-static-analyzer -w -analyzer-checker=debug.DumpLiveExprs /Users/thakis/src/llvm-project/clang/test/Analysis/live-stmts.cpp
+ /Users/thakis/src/llvm-project/out/gn/bin/FileCheck /Users/thakis/src/llvm-project/clang/test/Analysis/live-stmts.cpp
/Users/thakis/src/llvm-project/clang/test/Analysis/live-stmts.cpp:239:16: error: CHECK-EMPTY: is not on the line after the previous match
// CHECK-EMPTY:
               ^
&lt;stdin&gt;:180:1: note: 'next' match was here

^
&lt;stdin&gt;:177:1: note: previous match ended here

^
&lt;stdin&gt;:178:1: note: non-matching line after previous match is here
ImplicitCastExpr 0x11280d178 '_Bool' &lt;LValueToRValue&gt;
^

Input file: &lt;stdin&gt;
Check file: /Users/thakis/src/llvm-project/clang/test/Analysis/live-stmts.cpp

-dump-input=help explains the following input dump.

Input was:
&lt;&lt;&lt;&lt;&lt;&lt;
           .
           .
           .
         175:  
         176: IntegerLiteral 0x110810820 'int' 1 
         177:  
         178: ImplicitCastExpr 0x11280d178 '_Bool' &lt;LValueToRValue&gt; 
         179: `-DeclRefExpr 0x11280d138 '_Bool' lvalue ParmVar 0x11280cfb8 'b' '_Bool' 
         180:  
empty:239     ! error: match on wrong line
         181: ImplicitCastExpr 0x11280d190 '_Bool' &lt;LValueToRValue&gt; 
         182: `-DeclRefExpr 0x11280d158 '_Bool' lvalue ParmVar 0x11280cfb8 'b' '_Bool' 
         183:  
         184: BinaryOperator 0x11280d1a8 '_Bool' '||' 
         185: |-ImplicitCastExpr 0x11280d178 '_Bool' &lt;LValueToRValue&gt; 
           .
           .
           .
&gt;&gt;&gt;&gt;&gt;&gt;

--

********************
Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90..
********************
Failed Tests (1):
  Clang :: Analysis/live-stmts.cpp

</details>

See llvm#126619 and discussion on llvm#125840

Multiple people reported flaky bot failures tied to clang/test/Analysis/live-stmts.cpp I tried reproducing the flaky behavior on my Linux x86_64 system, but the tests appears to be stable in my context. Only by looking at the failures reported, I could formulate a potential diagnosis. The output always looked almost the same, except that the Exprs dumped per Basic block were shuffled compared to my expectation. This suggests to me some ordering issue. If you look at the backing storage of `blocksEndToLiveness[B].liveExprs`, it uses `llvm::ImmutableSet<const Expr *>`. That container likely uses the pointer values as keys, thus the runtime values of the addresses influence the iteration order. To fix this, before dumping, I sort the expressions by their "beginLocs". It should be efficient enough for a debug checker, where there is no performance constraint. This should hopefully fix the flaky behavior on systems where ASLR works differently than (my) Linux system. Hopefully fixes llvm#126619 Hopefully fixes llvm#126804

…lvm#126913) Multiple people reported flaky bot failures tied to `clang/test/Analysis/live-stmts.cpp` I tried reproducing the flaky behavior on my Linux x86_64 system, but the tests appears to be stable in my context. Only by looking at the failures reported, I could formulate a potential diagnosis. The output always looked almost the same, except that the Exprs dumped per Basic block were shuffled compared to my expectation. This suggests to me some ordering issue. If you look at the backing storage of `blocksEndToLiveness[B].liveExprs`, it uses `llvm::ImmutableSet<const Expr *>`. That container likely uses the pointer values as keys, thus the runtime values of the addresses influence the iteration order. To fix this, before dumping, I sort the expressions by their "beginLocs". It should be efficient enough for a debug checker, where there is no performance constraint. This should hopefully fix the flaky behavior on systems where ASLR works differently than (my) Linux system. Hopefully fixes llvm#126619 Hopefully fixes llvm#126804

See llvm#126619 and discussion on llvm#125840

…lvm#126913) Multiple people reported flaky bot failures tied to `clang/test/Analysis/live-stmts.cpp` I tried reproducing the flaky behavior on my Linux x86_64 system, but the tests appears to be stable in my context. Only by looking at the failures reported, I could formulate a potential diagnosis. The output always looked almost the same, except that the Exprs dumped per Basic block were shuffled compared to my expectation. This suggests to me some ordering issue. If you look at the backing storage of `blocksEndToLiveness[B].liveExprs`, it uses `llvm::ImmutableSet<const Expr *>`. That container likely uses the pointer values as keys, thus the runtime values of the addresses influence the iteration order. To fix this, before dumping, I sort the expressions by their "beginLocs". It should be efficient enough for a debug checker, where there is no performance constraint. This should hopefully fix the flaky behavior on systems where ASLR works differently than (my) Linux system. Hopefully fixes llvm#126619 Hopefully fixes llvm#126804

…2nd attempt) In my previous attempt (llvm#126913) of fixing the flaky case was on a good track when I used the begin locations as a stable ordering. However, I forgot to consider the case when the begin locations are the same among the Exprs. In an `EXPENSIVE_CHECKS` build, arrays are randomly shuffled prior to sorting them. This exposed the flaky behavior much more often basically breaking the "stability" of the vector - as it should. To fix this, I I use this time `Expr::getID` for a stable ID for an Expr. Hopefully fixes llvm#126619 Hopefully fixes llvm#126804

…2nd attempt) (#127406) In my previous attempt (#126913) of fixing the flaky case was on a good track when I used the begin locations as a stable ordering. However, I forgot to consider the case when the begin locations are the same among the Exprs. In an `EXPENSIVE_CHECKS` build, arrays are randomly shuffled prior to sorting them. This exposed the flaky behavior much more often basically breaking the "stability" of the vector - as it should. Because of this, I had to revert the previous fix attempt in #127034. To fix this, I use this time `Expr::getID` for a stable ID for an Expr. Hopefully fixes #126619 Hopefully fixes #126804

See llvm#126619 and discussion on llvm#125840

…lvm#126913) Multiple people reported flaky bot failures tied to `clang/test/Analysis/live-stmts.cpp` I tried reproducing the flaky behavior on my Linux x86_64 system, but the tests appears to be stable in my context. Only by looking at the failures reported, I could formulate a potential diagnosis. The output always looked almost the same, except that the Exprs dumped per Basic block were shuffled compared to my expectation. This suggests to me some ordering issue. If you look at the backing storage of `blocksEndToLiveness[B].liveExprs`, it uses `llvm::ImmutableSet<const Expr *>`. That container likely uses the pointer values as keys, thus the runtime values of the addresses influence the iteration order. To fix this, before dumping, I sort the expressions by their "beginLocs". It should be efficient enough for a debug checker, where there is no performance constraint. This should hopefully fix the flaky behavior on systems where ASLR works differently than (my) Linux system. Hopefully fixes llvm#126619 Hopefully fixes llvm#126804

…2nd attempt) (llvm#127406) In my previous attempt (llvm#126913) of fixing the flaky case was on a good track when I used the begin locations as a stable ordering. However, I forgot to consider the case when the begin locations are the same among the Exprs. In an `EXPENSIVE_CHECKS` build, arrays are randomly shuffled prior to sorting them. This exposed the flaky behavior much more often basically breaking the "stability" of the vector - as it should. Because of this, I had to revert the previous fix attempt in llvm#127034. To fix this, I use this time `Expr::getID` for a stable ID for an Expr. Hopefully fixes llvm#126619 Hopefully fixes llvm#126804

…2nd attempt) (llvm#127406) In my previous attempt (llvm#126913) of fixing the flaky case was on a good track when I used the begin locations as a stable ordering. However, I forgot to consider the case when the begin locations are the same among the Exprs. In an `EXPENSIVE_CHECKS` build, arrays are randomly shuffled prior to sorting them. This exposed the flaky behavior much more often basically breaking the "stability" of the vector - as it should. Because of this, I had to revert the previous fix attempt in llvm#127034. To fix this, I use this time `Expr::getID` for a stable ID for an Expr. Hopefully fixes llvm#126619 Hopefully fixes llvm#126804 (cherry picked from commit f378e52)

nico added the clang:static analyzer label Feb 10, 2025

nico added a commit that referenced this issue Feb 10, 2025

[clang] Disaqble test/Analysis/live-stmts.cpp on aarch64

44fcc5c

See #126619 and discussion on #125840

nico mentioned this issue Feb 10, 2025

SCEV: thread samesign in isBasicBlockEntryGuardedByCond (NFC) #125840

Merged

EugeneZelenko added test-suite clang:analysis and removed clang:static analyzer labels Feb 10, 2025

github-actions bot pushed a commit to arm/arm-toolchain that referenced this issue Feb 10, 2025

Automerge: [clang] Disaqble test/Analysis/live-stmts.cpp on aarch64

e2b41c4

See llvm/llvm-project#126619 and discussion on llvm/llvm-project#125840

steakhal added the clang:static analyzer label Feb 11, 2025

EugeneZelenko removed the clang:analysis label Feb 11, 2025

shafik mentioned this issue Feb 11, 2025

Clang test Analysis/live-stmts.cpp randomly fails on MacOS #126804

Closed

Icohedron pushed a commit to Icohedron/llvm-project that referenced this issue Feb 11, 2025

[clang] Disaqble test/Analysis/live-stmts.cpp on aarch64

56dbc47

See llvm#126619 and discussion on llvm#125840

steakhal mentioned this issue Feb 12, 2025

[clang][analysis] Fix flaky clang/test/Analysis/live-stmts.cpp test #126913

Merged

steakhal closed this as completed in #126913 Feb 12, 2025

steakhal closed this as completed in be25d61 Feb 12, 2025

EugeneZelenko added clang:analysis and removed test-suite clang:static analyzer labels Feb 12, 2025

joaosaffran pushed a commit to joaosaffran/llvm-project that referenced this issue Feb 14, 2025

[clang] Disaqble test/Analysis/live-stmts.cpp on aarch64

84f6fa8

See llvm#126619 and discussion on llvm#125840

steakhal mentioned this issue Feb 16, 2025

[clang][analysis] Fix flaky clang/test/Analysis/live-stmts.cpp test (2nd attempt) #127406

Merged

sivan-shani pushed a commit to sivan-shani/llvm-project that referenced this issue Feb 24, 2025

[clang] Disaqble test/Analysis/live-stmts.cpp on aarch64

654dc11

See llvm#126619 and discussion on llvm#125840

steakhal mentioned this issue May 12, 2025

[clang][analysis] Fix flaky clang/test/Analysis/live-stmts.cpp test (2nd attempt) (#127406) #139591

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

clang/test/Analysis/live-stmts.cpp frequently flaky on aarch64 #126619

clang/test/Analysis/live-stmts.cpp frequently flaky on aarch64 #126619

nico commented Feb 10, 2025

llvmbot commented Feb 10, 2025

Uh oh!

llvmbot commented Feb 11, 2025

Uh oh!

clang/test/Analysis/live-stmts.cpp frequently flaky on aarch64 #126619

clang/test/Analysis/live-stmts.cpp frequently flaky on aarch64 #126619

Comments

nico commented Feb 10, 2025

llvmbot commented Feb 10, 2025

Uh oh!

llvmbot commented Feb 11, 2025

Uh oh!