Skip to content

Clang test Analysis/live-stmts.cpp randomly fails on MacOS #126804

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
dyung opened this issue Feb 11, 2025 · 6 comments · Fixed by #126913 or #127406
Closed

Clang test Analysis/live-stmts.cpp randomly fails on MacOS #126804

dyung opened this issue Feb 11, 2025 · 6 comments · Fixed by #126913 or #127406
Assignees

Comments

@dyung
Copy link
Collaborator

dyung commented Feb 11, 2025

I have 2 build bots setup to build/test LLVM running on MacOS and I have noticed that the test clang/test/Analysis/live-stmts.cpp randomly seems to fail on the bot. I have two identically configured workers running the job, and it has failed on both so it doesn't seem specific to one machine configuration.

Here is a sample of the failing test output:

******************** TEST 'Clang :: Analysis/live-stmts.cpp' FAILED ********************
Exit Code: 1
Command Output (stderr):
--
RUN: at line 1: /Users/buildbot/buildbot-root/aarch64-darwin/build/bin/clang -cc1 -internal-isystem /Users/buildbot/buildbot-root/aarch64-darwin/build/lib/clang/21/include -nostdsysteminc -analyze -analyzer-constraints=range -setup-static-analyzer -w -analyzer-checker=debug.DumpLiveExprs /Users/buildbot/buildbot-root/aarch64-darwin/llvm-project/clang/test/Analysis/live-stmts.cpp 2>&1   | /Users/buildbot/buildbot-root/aarch64-darwin/build/bin/FileCheck /Users/buildbot/buildbot-root/aarch64-darwin/llvm-project/clang/test/Analysis/live-stmts.cpp
+ /Users/buildbot/buildbot-root/aarch64-darwin/build/bin/clang -cc1 -internal-isystem /Users/buildbot/buildbot-root/aarch64-darwin/build/lib/clang/21/include -nostdsysteminc -analyze -analyzer-constraints=range -setup-static-analyzer -w -analyzer-checker=debug.DumpLiveExprs /Users/buildbot/buildbot-root/aarch64-darwin/llvm-project/clang/test/Analysis/live-stmts.cpp
+ /Users/buildbot/buildbot-root/aarch64-darwin/build/bin/FileCheck /Users/buildbot/buildbot-root/aarch64-darwin/llvm-project/clang/test/Analysis/live-stmts.cpp
/Users/buildbot/buildbot-root/aarch64-darwin/llvm-project/clang/test/Analysis/live-stmts.cpp:239:16: error: CHECK-EMPTY: is not on the line after the previous match
// CHECK-EMPTY:
               ^
<stdin>:180:1: note: 'next' match was here
^
<stdin>:177:1: note: previous match ended here
^
<stdin>:178:1: note: non-matching line after previous match is here
ImplicitCastExpr 0x151009d78 '_Bool' <LValueToRValue>
^

<many lines skipped>

172: [ B3 (live expressions at block exit) ] 
         173:  
         174: IntegerLiteral 0x15082f800 'int' 0 
check:235     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
         175:  
empty:236     ^
         176: IntegerLiteral 0x15082f820 'int' 1 
check:237     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
         177:  
empty:238     ^
         178: ImplicitCastExpr 0x151009d78 '_Bool' <LValueToRValue> 
         179: `-DeclRefExpr 0x151009d38 '_Bool' lvalue ParmVar 0x151009bb8 'b' '_Bool' 
         180:  
empty:239     ! error: match on wrong line

Here are some recent runs of the build bot where the test failure occurred:

@llvmbot llvmbot added the clang Clang issues not falling into any other category label Feb 11, 2025
@shafik
Copy link
Collaborator

shafik commented Feb 11, 2025

Maybe a duplicate of: #126619

CC @haoNoQ @necto

@EugeneZelenko EugeneZelenko added test-suite clang:static analyzer and removed clang Clang issues not falling into any other category labels Feb 11, 2025
@llvmbot
Copy link
Member

llvmbot commented Feb 11, 2025

@llvm/issue-subscribers-clang-static-analyzer

Author: None (dyung)

I have 2 build bots setup to build/test LLVM running on MacOS and I have noticed that the test `clang/test/Analysis/live-stmts.cpp` randomly seems to fail on the bot. I have two identically configured workers running the job, and it has failed on both so it doesn't seem specific to one machine configuration.

Here is a sample of the failing test output:

******************** TEST 'Clang :: Analysis/live-stmts.cpp' FAILED ********************
Exit Code: 1
Command Output (stderr):
--
RUN: at line 1: /Users/buildbot/buildbot-root/aarch64-darwin/build/bin/clang -cc1 -internal-isystem /Users/buildbot/buildbot-root/aarch64-darwin/build/lib/clang/21/include -nostdsysteminc -analyze -analyzer-constraints=range -setup-static-analyzer -w -analyzer-checker=debug.DumpLiveExprs /Users/buildbot/buildbot-root/aarch64-darwin/llvm-project/clang/test/Analysis/live-stmts.cpp 2&gt;&amp;1   | /Users/buildbot/buildbot-root/aarch64-darwin/build/bin/FileCheck /Users/buildbot/buildbot-root/aarch64-darwin/llvm-project/clang/test/Analysis/live-stmts.cpp
+ /Users/buildbot/buildbot-root/aarch64-darwin/build/bin/clang -cc1 -internal-isystem /Users/buildbot/buildbot-root/aarch64-darwin/build/lib/clang/21/include -nostdsysteminc -analyze -analyzer-constraints=range -setup-static-analyzer -w -analyzer-checker=debug.DumpLiveExprs /Users/buildbot/buildbot-root/aarch64-darwin/llvm-project/clang/test/Analysis/live-stmts.cpp
+ /Users/buildbot/buildbot-root/aarch64-darwin/build/bin/FileCheck /Users/buildbot/buildbot-root/aarch64-darwin/llvm-project/clang/test/Analysis/live-stmts.cpp
/Users/buildbot/buildbot-root/aarch64-darwin/llvm-project/clang/test/Analysis/live-stmts.cpp:239:16: error: CHECK-EMPTY: is not on the line after the previous match
// CHECK-EMPTY:
               ^
&lt;stdin&gt;:180:1: note: 'next' match was here
^
&lt;stdin&gt;:177:1: note: previous match ended here
^
&lt;stdin&gt;:178:1: note: non-matching line after previous match is here
ImplicitCastExpr 0x151009d78 '_Bool' &lt;LValueToRValue&gt;
^

&lt;many lines skipped&gt;

172: [ B3 (live expressions at block exit) ] 
         173:  
         174: IntegerLiteral 0x15082f800 'int' 0 
check:235     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
         175:  
empty:236     ^
         176: IntegerLiteral 0x15082f820 'int' 1 
check:237     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
         177:  
empty:238     ^
         178: ImplicitCastExpr 0x151009d78 '_Bool' &lt;LValueToRValue&gt; 
         179: `-DeclRefExpr 0x151009d38 '_Bool' lvalue ParmVar 0x151009bb8 'b' '_Bool' 
         180:  
empty:239     ! error: match on wrong line

Here are some recent runs of the build bot where the test failure occurred:

@steakhal
Copy link
Contributor

Duplicate of #126619

@steakhal steakhal marked this as a duplicate of #126619 Feb 12, 2025
@steakhal steakhal self-assigned this Feb 12, 2025
@steakhal
Copy link
Contributor

Ill have a look at his

@steakhal
Copy link
Contributor

I tried running this test about half a million times, and it always passed. I'm on linux x86_64.
The flaky failure may have something to do with ASLR and other platform-specific behavior.

After looking at the output you observed, my guess is that there is some nondeterminism somewhere, making the dumps unstable wrt. their ordering.

If you look at each live expressions at block exit section, the expectation specifies a fixed ordering, yet, in your observed failure they appear in a different permutation.

I think this shouldn't be too hard to fix while dumping these, but we should maybe go to the bottom of this and consider making the backing implementation deterministic instead/in addition to sorting in the dumps.

I don't believe it's the #100745 fault. I think it just exposed this behavior more often due to the additional tests.

steakhal added a commit to steakhal/llvm-project that referenced this issue Feb 12, 2025
Multiple people reported flaky bot failures tied to clang/test/Analysis/live-stmts.cpp
I tried reproducing the flaky behavior on my Linux x86_64 system, but
the tests appears to be stable in my context.

Only by looking at the failures reported, I could formulate a potential
diagnosis.
The output always looked almost the same, except that the Exprs dumped
per Basic block were shuffled compared to my expectation.
This suggests to me some ordering issue.

If you look at the backing storage of
`blocksEndToLiveness[B].liveExprs`, it uses `llvm::ImmutableSet<const Expr *>`.
That container likely uses the pointer values as keys, thus the runtime
values of the addresses influence the iteration order.

To fix this, before dumping, I sort the expressions by their
"beginLocs". It should be efficient enough for a debug checker, where
there is no performance constraint.

This should hopefully fix the flaky behavior on systems where ASLR works
differently than (my) Linux system.

Hopefully fixes llvm#126619
Hopefully fixes llvm#126804
@steakhal
Copy link
Contributor

Find the fix at #126913

flovent pushed a commit to flovent/llvm-project that referenced this issue Feb 13, 2025
…lvm#126913)

Multiple people reported flaky bot failures tied to
`clang/test/Analysis/live-stmts.cpp` I tried reproducing the flaky
behavior on my Linux x86_64 system, but the tests appears to be stable
in my context.

Only by looking at the failures reported, I could formulate a potential
diagnosis.
The output always looked almost the same, except that the Exprs dumped
per Basic block were shuffled compared to my expectation. This suggests
to me some ordering issue.

If you look at the backing storage of
`blocksEndToLiveness[B].liveExprs`,
it uses `llvm::ImmutableSet<const Expr *>`.
That container likely uses the pointer values as keys, thus the runtime
values of the addresses influence the iteration order.

To fix this, before dumping, I sort the expressions by their
"beginLocs". It should be efficient enough for a debug checker, where
there is no performance constraint.

This should hopefully fix the flaky behavior on systems where ASLR works
differently than (my) Linux system.

Hopefully fixes llvm#126619
Hopefully fixes llvm#126804
joaosaffran pushed a commit to joaosaffran/llvm-project that referenced this issue Feb 14, 2025
…lvm#126913)

Multiple people reported flaky bot failures tied to
`clang/test/Analysis/live-stmts.cpp` I tried reproducing the flaky
behavior on my Linux x86_64 system, but the tests appears to be stable
in my context.

Only by looking at the failures reported, I could formulate a potential
diagnosis.
The output always looked almost the same, except that the Exprs dumped
per Basic block were shuffled compared to my expectation. This suggests
to me some ordering issue.

If you look at the backing storage of
`blocksEndToLiveness[B].liveExprs`,
it uses `llvm::ImmutableSet<const Expr *>`.
That container likely uses the pointer values as keys, thus the runtime
values of the addresses influence the iteration order.

To fix this, before dumping, I sort the expressions by their
"beginLocs". It should be efficient enough for a debug checker, where
there is no performance constraint.

This should hopefully fix the flaky behavior on systems where ASLR works
differently than (my) Linux system.

Hopefully fixes llvm#126619
Hopefully fixes llvm#126804
steakhal added a commit to steakhal/llvm-project that referenced this issue Feb 16, 2025
…2nd attempt)

In my previous attempt (llvm#126913) of fixing the flaky case was on a good track
when I used the begin locations as a stable ordering. However, I forgot to
consider the case when the begin locations are the same among the Exprs.

In an `EXPENSIVE_CHECKS` build, arrays are randomly shuffled prior to sorting them.
This exposed the flaky behavior much more often basically breaking the
"stability" of the vector - as it should.

To fix this, I I use this time `Expr::getID` for a stable ID for an Expr.

Hopefully fixes llvm#126619
Hopefully fixes llvm#126804
steakhal added a commit that referenced this issue Feb 17, 2025
…2nd attempt) (#127406)

In my previous attempt (#126913) of fixing the flaky case was on a good
track when I used the begin locations as a stable ordering. However, I
forgot to consider the case when the begin locations are the same among
the Exprs.

In an `EXPENSIVE_CHECKS` build, arrays are randomly shuffled prior to
sorting them. This exposed the flaky behavior much more often basically
breaking the "stability" of the vector - as it should.
Because of this, I had to revert the previous fix attempt in #127034.

To fix this, I use this time `Expr::getID` for a stable ID for an Expr.

Hopefully fixes #126619
Hopefully fixes #126804
sivan-shani pushed a commit to sivan-shani/llvm-project that referenced this issue Feb 24, 2025
…lvm#126913)

Multiple people reported flaky bot failures tied to
`clang/test/Analysis/live-stmts.cpp` I tried reproducing the flaky
behavior on my Linux x86_64 system, but the tests appears to be stable
in my context.

Only by looking at the failures reported, I could formulate a potential
diagnosis.
The output always looked almost the same, except that the Exprs dumped
per Basic block were shuffled compared to my expectation. This suggests
to me some ordering issue.

If you look at the backing storage of
`blocksEndToLiveness[B].liveExprs`,
it uses `llvm::ImmutableSet<const Expr *>`.
That container likely uses the pointer values as keys, thus the runtime
values of the addresses influence the iteration order.

To fix this, before dumping, I sort the expressions by their
"beginLocs". It should be efficient enough for a debug checker, where
there is no performance constraint.

This should hopefully fix the flaky behavior on systems where ASLR works
differently than (my) Linux system.

Hopefully fixes llvm#126619
Hopefully fixes llvm#126804
sivan-shani pushed a commit to sivan-shani/llvm-project that referenced this issue Feb 24, 2025
…2nd attempt) (llvm#127406)

In my previous attempt (llvm#126913) of fixing the flaky case was on a good
track when I used the begin locations as a stable ordering. However, I
forgot to consider the case when the begin locations are the same among
the Exprs.

In an `EXPENSIVE_CHECKS` build, arrays are randomly shuffled prior to
sorting them. This exposed the flaky behavior much more often basically
breaking the "stability" of the vector - as it should.
Because of this, I had to revert the previous fix attempt in llvm#127034.

To fix this, I use this time `Expr::getID` for a stable ID for an Expr.

Hopefully fixes llvm#126619
Hopefully fixes llvm#126804
steakhal added a commit to steakhal/llvm-project that referenced this issue May 12, 2025
…2nd attempt) (llvm#127406)

In my previous attempt (llvm#126913) of fixing the flaky case was on a good
track when I used the begin locations as a stable ordering. However, I
forgot to consider the case when the begin locations are the same among
the Exprs.

In an `EXPENSIVE_CHECKS` build, arrays are randomly shuffled prior to
sorting them. This exposed the flaky behavior much more often basically
breaking the "stability" of the vector - as it should.
Because of this, I had to revert the previous fix attempt in llvm#127034.

To fix this, I use this time `Expr::getID` for a stable ID for an Expr.

Hopefully fixes llvm#126619
Hopefully fixes llvm#126804

(cherry picked from commit f378e52)
swift-ci pushed a commit to swiftlang/llvm-project that referenced this issue May 13, 2025
…2nd attempt) (llvm#127406)

In my previous attempt (llvm#126913) of fixing the flaky case was on a good
track when I used the begin locations as a stable ordering. However, I
forgot to consider the case when the begin locations are the same among
the Exprs.

In an `EXPENSIVE_CHECKS` build, arrays are randomly shuffled prior to
sorting them. This exposed the flaky behavior much more often basically
breaking the "stability" of the vector - as it should.
Because of this, I had to revert the previous fix attempt in llvm#127034.

To fix this, I use this time `Expr::getID` for a stable ID for an Expr.

Hopefully fixes llvm#126619
Hopefully fixes llvm#126804

(cherry picked from commit f378e52)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment