Skip to content

PYTHON-4409 Streamline Wheel Building #1624

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 15 commits into from
Apr 30, 2024
116 changes: 30 additions & 86 deletions .github/workflows/release-python.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ defaults:

jobs:
build_wheels:
name: Build wheel for ${{ matrix.python }}-${{ matrix.buildplat[1] }}
name: Build wheels for ${{ matrix.buildplat[1] }}
runs-on: ${{ matrix.buildplat[0] }}
strategy:
# Ensure that a wheel builder finishes even if another fails
Expand All @@ -29,43 +29,25 @@ jobs:
# Github Actions doesn't support pairing matrix values together, let's improvise
# https://github.com/github/feedback/discussions/7835#discussioncomment-1769026
buildplat:
- [ubuntu-20.04, manylinux_x86_64]
- [ubuntu-20.04, manylinux_aarch64]
- [ubuntu-20.04, manylinux_ppc64le]
- [ubuntu-20.04, manylinux_s390x]
- [ubuntu-20.04, manylinux_i686]
- [windows-2019, win_amd64]
- [windows-2019, win32]
python: ["cp37", "cp38", "cp39", "cp310", "cp311", "cp312"]
include:
- buildplat: [macos-11, macosx_*]
python: "cp37"
- buildplat: [macos-11, macosx_*]
python: "cp38"
- buildplat: [macos-11, macosx_*]
python: "cp39"
- buildplat: [macos-14, macosx_*]
python: "cp310"
- buildplat: [macos-14, macosx_*]
python: "cp311"
- buildplat: [macos-14, macosx_*]
python: "cp312"
Comment on lines -39 to -52
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why can we get away with not specifying the python version anymore Is It that the way we build wheels now will automatically create wheels packages for 3.7 - 3.12?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, and we verify it later. :)

- [ubuntu-20.04, "manylinux_x86_64", "cp3*-manylinux_x86_64"]
- [ubuntu-20.04, "manylinux_aarch64", "cp3*-manylinux_aarch64"]
- [ubuntu-20.04, "manylinux_ppc64le", "cp3*-manylinux_ppc64le"]
- [ubuntu-20.04, "manylinux_s390x", "cp3*-manylinux_s390x"]
- [ubuntu-20.04, "manylinux_i686", "cp3*-manylinux_i686"]
- [windows-2019, "win_amd6", "cp3*-win_amd64"]
- [windows-2019, "win32", "cp3*-win32"]
- [macos-14, "macos", "cp*-macosx_*"]

steps:
- name: Checkout pymongo
uses: actions/checkout@v4
with:
fetch-depth: 0

- name: Set up python version
run: |
export PYTHON_VERSION=$(sed 's/^cp3/3./' <<< ${{ matrix.python }} )
echo "PYTHON_VERSION=$PYTHON_VERSION" >> $GITHUB_ENV

- uses: actions/setup-python@v5
with:
python-version: ${{env.PYTHON_VERSION}}
cache: 'pip'
python-version: 3.8
cache-dependency-path: 'pyproject.toml'
allow-prereleases: true

Expand All @@ -79,75 +61,37 @@ jobs:
# Note: the default manylinux is manylinux2014
run: |
python -m pip install -U pip
python -m pip install "cibuildwheel>=2.4,<3"
python -m pip install "cibuildwheel>=2.17,<3"

- name: Build wheel
if: ${{ matrix.buildplat[0] != 'macos-14' }}
- name: Build wheels
env:
CIBW_BUILD: ${{ matrix.python }}-${{ matrix.buildplat[1] }}
MACOS_TEST_SKIP: "*arm64"
CIBW_BUILD: ${{ matrix.buildplat[2] }}
Comment on lines +66 to +68
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this step exclude manylinux1?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need to exclude it. We have a separate manylinux1 to ensure that we have wheels that include that arch, but it is fine if we have duplicate wheels.

run: python -m cibuildwheel --output-dir wheelhouse

- name: Build wheel on M1
if: ${{ matrix.buildplat[0] == 'macos-14' }}
- name: Build manylinux1 wheels
if: ${{ matrix.buildplat[1] == 'manylinux_x86_64' || matrix.buildplat[1] == 'manylinux_i686' }}
env:
CIBW_BUILD: ${{ matrix.python }}-${{ matrix.buildplat[1] }}
CIBW_MANYLINUX_X86_64_IMAGE: manylinux1
CIBW_MANYLINUX_I686_IMAGE: manylinux1
CIBW_BUILD: "cp37-${{ matrix.buildplat[1] }} cp38-${{ matrix.buildplat[1] }} cp39-${{ matrix.buildplat[1] }}"
run: python -m cibuildwheel --output-dir wheelhouse

- name: Assert all versions in wheelhouse
if: ${{ ! startsWith(matrix.buildplat[1], 'macos') }}
run: |
ls wheelhouse/*cp37*.whl
ls wheelhouse/*cp38*.whl
ls wheelhouse/*cp39*.whl
ls wheelhouse/*cp310*.whl
ls wheelhouse/*cp311*.whl
ls wheelhouse/*cp312*.whl

- uses: actions/upload-artifact@v4
with:
name: wheel-${{ matrix.python }}-${{ startsWith(matrix.buildplat[1], 'macosx') && 'macosx' || matrix.buildplat[1] }}
name: wheel-${{ matrix.buildplat[1] }}
path: ./wheelhouse/*.whl
if-no-files-found: error

build_manylinux1_wheels:
runs-on: ubuntu-latest
strategy:
# Ensure that a wheel builder finishes even if another fails
fail-fast: false
matrix:
python: ["cp37", "cp38", "cp39"]
steps:
- name: Checkout pymongo
uses: actions/checkout@v4
with:
fetch-depth: 0

- name: Set up python version
run: |
export PYTHON_VERSION=$(sed 's/^cp3/3./' <<< ${{ matrix.python }} )
echo "PYTHON_VERSION=$PYTHON_VERSION" >> $GITHUB_ENV

- uses: actions/setup-python@v5
with:
python-version: ${{env.PYTHON_VERSION}}
cache: 'pip'
cache-dependency-path: 'pyproject.toml'
allow-prereleases: true

- name: Install cibuildwheel
run: |
python -m pip install -U pip
python -m pip install "cibuildwheel>=2.4,<3"

- name: Build manylinux_x86_64
env:
CIBW_BUILD: ${{ matrix.python }}-manylinux_x86_64
CIBW_MANYLINUX_X86_64_IMAGE: manylinux1
run: python -m cibuildwheel --output-dir wheelhouse

- name: Build manylinux_i686
env:
CIBW_BUILD: ${{ matrix.python }}-manylinux_i686
CIBW_MANYLINUX_I686_IMAGE: manylinux1
run: python -m cibuildwheel --output-dir wheelhouse

- uses: actions/upload-artifact@v4
with:
name: wheel-${{ matrix.python }}-manylinux1
path: ./wheelhouse/*.whl
if-no-files-found: error

make_sdist:
name: Make SDist
runs-on: macos-13
Expand Down Expand Up @@ -180,7 +124,7 @@ jobs:

collect_dist:
runs-on: ubuntu-latest
needs: [build_wheels, build_manylinux1_wheels, make_sdist]
needs: [build_wheels, make_sdist]
name: Download Wheels
steps:
- name: Download all workflow run artifacts
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/test-python.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ name: Python Tests

on:
push:
branches: ["master", "v**"]
pull_request:
workflow_dispatch:

Expand Down
Loading