Skip to content

enable ASAN/UBSAN in pandas CI #55102

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 57 commits into from
Dec 21, 2023
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
57 commits
Select commit Hold shift + click to select a range
66d83d1
enable ASAN/UBSAN in pandas CI
WillAyd Sep 11, 2023
7aa2e7a
try input
WillAyd Sep 11, 2023
a5b3808
try removing sanitize
WillAyd Sep 12, 2023
7b58c6d
try no CFLAGS
WillAyd Sep 12, 2023
18111b0
try GH string substituion
WillAyd Sep 12, 2023
438cdfa
change flags in build script
WillAyd Sep 12, 2023
b18cf9d
quotes
WillAyd Sep 12, 2023
69cb6f6
update script run
WillAyd Sep 12, 2023
6f5fb11
single_cpu updates
WillAyd Sep 12, 2023
eb258ca
Merge branch 'main' into pandas-asan
WillAyd Sep 14, 2023
663d6d4
asan checks for datetime funcs
WillAyd Sep 14, 2023
466056d
try smaller config
WillAyd Sep 15, 2023
91f2e17
Merge remote-tracking branch 'upstream/main' into pandas-asan
WillAyd Sep 15, 2023
d4074ca
checkpoint
WillAyd Sep 15, 2023
aeff50e
Merge branch 'main' into pandas-asan
WillAyd Oct 26, 2023
e303ba1
bool fixup
WillAyd Oct 27, 2023
4220d82
Merge branch 'main' into pandas-asan
WillAyd Nov 16, 2023
46d1034
reverts
WillAyd Nov 16, 2023
89706a4
Merge branch 'main' into pandas-asan
WillAyd Nov 17, 2023
929c731
known UB marker
WillAyd Nov 17, 2023
b01242b
Merge branch 'main' into pandas-asan
lithomas1 Nov 28, 2023
6483e07
Finished marking tests with known UB
WillAyd Dec 2, 2023
de13605
Merge remote-tracking branch 'upstream/main' into pandas-asan
WillAyd Dec 2, 2023
b87a210
dedicated CI job
WillAyd Dec 2, 2023
77d1e00
Merge remote-tracking branch 'upstream/main' into pandas-asan
WillAyd Dec 2, 2023
46ec023
identifier fix
WillAyd Dec 2, 2023
8695dca
fixes
WillAyd Dec 2, 2023
05319ae
more test skip
WillAyd Dec 2, 2023
6d76a57
try quotes
WillAyd Dec 2, 2023
f5dd440
simplify ci
WillAyd Dec 2, 2023
12aa1d1
try CFLAGS
WillAyd Dec 2, 2023
628d1c2
preload args
WillAyd Dec 2, 2023
1de633e
skip single_cpu tests
WillAyd Dec 2, 2023
3e295c5
wording
WillAyd Dec 2, 2023
252197e
Merge remote-tracking branch 'upstream/main' into pandas-asan
WillAyd Dec 5, 2023
d5809b8
removed unneeded marker
WillAyd Dec 5, 2023
6266422
float set implementations
WillAyd Dec 5, 2023
b68a533
Revert "float set implementations"
WillAyd Dec 5, 2023
47dc305
Merge branch 'main' into pandas-asan
WillAyd Dec 6, 2023
636b8dd
Merge remote-tracking branch 'upstream/main' into pandas-asan
WillAyd Dec 13, 2023
a03ad1e
change marker name
WillAyd Dec 15, 2023
656edb1
dedicated actions file
WillAyd Dec 15, 2023
2aabda1
consolidated into matrix
WillAyd Dec 15, 2023
a9f2419
Merge remote-tracking branch 'upstream/main' into pandas-asan
WillAyd Dec 15, 2023
3056e5f
fixup
WillAyd Dec 15, 2023
89b2b80
typos
WillAyd Dec 15, 2023
d591b78
fixups
WillAyd Dec 16, 2023
6442066
add qt?
WillAyd Dec 16, 2023
c59703d
Merge branch 'main' into pandas-asan
WillAyd Dec 19, 2023
02bf20d
intentional UB with verbose
WillAyd Dec 19, 2023
01070f3
disable pytest-xdist
WillAyd Dec 20, 2023
9f1adbc
Merge remote-tracking branch 'upstream/main' into pandas-asan
WillAyd Dec 20, 2023
57ed286
original issue
WillAyd Dec 20, 2023
677da0e
remove UB
WillAyd Dec 20, 2023
af0150a
Revert "remove UB"
WillAyd Dec 21, 2023
4647f12
merge fixup
WillAyd Dec 21, 2023
cba79f6
remove UB
WillAyd Dec 21, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 13 additions & 2 deletions .github/actions/build_pandas/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,9 @@ inputs:
editable:
description: Whether to build pandas in editable mode (default true)
default: true
sanitize:
description: Whether sanitizers should be used or not
default: false
runs:
using: composite
steps:
Expand All @@ -25,8 +28,16 @@ runs:
- name: Build Pandas
run: |
if [[ ${{ inputs.editable }} == "true" ]]; then
pip install -e . --no-build-isolation -v
if [[ ${{ inputs.sanitize }} == "true" ]]; then
pip install -e . --no-build-isolation -v --config-settings=setup-args="-Db_sanitize=address,undefined"
else
pip install -e . --no-build-isolation -v
fi
else
pip install . --no-build-isolation -v
if [[ ${{ inputs.sanitize }} == "true" ]]; then
pip install . --no-build-isolation -v --config-settings=setup-args="-Db_sanitize=address,undefined"
else
pip install . --no-build-isolation -v
fi
fi
shell: bash -el {0}
14 changes: 14 additions & 0 deletions .github/workflows/unit-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -154,22 +154,36 @@ jobs:
with:
environment-file: ci/deps/${{ matrix.env_file }}

- name: Set sanitizer flags
run: |
echo "CFLAGS=$CFLAGS -fno-sanitize-recover=all" >> "$GITHUB_ENV"
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having trouble setting this with GHA. There may also be a way to pass the flag directly through meson python? @lithomas1 any idea?

- name: echo flags
run: |
printf '%s\n' "CFLAGS"
- name: Build Pandas
id: build
uses: ./.github/actions/build_pandas
env:
CFLAGS: "$CFLAGS"
Copy link
Member

@lithomas1 lithomas1 Sep 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIRC, this doesn't set CFLAGS inside of the action. It just makes CFLAGS available under env.CFLAGS .

Why don't you try setting CFLAGs in action.yml directly if sanitize = true?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah great idea - much cleaner

with:
sanitize: true

- name: Test (not single_cpu)
uses: ./.github/actions/run-tests
if: ${{ matrix.name != 'Pypy' }}
env:
# Set pattern to not single_cpu if not already set
PATTERN: ${{ env.PATTERN == '' && 'not single_cpu' || matrix.pattern }}
ASAN_OPTIONS: detect_leaks=0
LD_PRELOAD: $(gcc -print-file-name=libasan.so)

- name: Test (single_cpu)
uses: ./.github/actions/run-tests
env:
PATTERN: 'single_cpu'
PYTEST_WORKERS: 0
ASAN_OPTIONS: detect_leaks=0
LD_PRELOAD: $(gcc -print-file-name=libasan.so)
if: ${{ matrix.pattern == '' && (always() && steps.build.outcome == 'success')}}

macos-windows:
Expand Down