Skip to content

ref(ci): Create a separate test group for AI #3198

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jun 25, 2024
Merged

ref(ci): Create a separate test group for AI #3198

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
40bb62b
ref(ci): Create a separate test group for AI
sentrivana Jun 25, 2024
52e90f9
Merge branch 'master' into ivana/test-groups
sentrivana Jun 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
135 changes: 135 additions & 0 deletions .github/workflows/test-integrations-ai.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,135 @@
name: Test AI
on:
push:
branches:
- master
- release/**
- sentry-sdk-2.0
pull_request:
# Cancel in progress workflows on pull_requests.
# https://docs.github.com/en/actions/using-jobs/using-concurrency#example-using-a-fallback-value
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
cancel-in-progress: true
permissions:
contents: read
env:
BUILD_CACHE_KEY: ${{ github.sha }}
CACHED_BUILD_PATHS: |
${{ github.workspace }}/dist-serverless
jobs:
test-ai-latest:
name: AI (latest)
timeout-minutes: 30
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
python-version: ["3.7","3.9","3.11","3.12"]
# python3.6 reached EOL and is no longer being supported on
# new versions of hosted runners on Github Actions
# ubuntu-20.04 is the last version that supported python3.6
# see https://github.com/actions/setup-python/issues/544#issuecomment-1332535877
os: [ubuntu-20.04]
steps:
- uses: actions/[email protected]
- uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Setup Test Env
run: |
pip install coverage tox
- name: Erase coverage
run: |
coverage erase
- name: Test anthropic latest
run: |
set -x # print commands that are executed
./scripts/runtox.sh "py${{ matrix.python-version }}-anthropic-latest" --cov=tests --cov=sentry_sdk --cov-report= --cov-branch
- name: Test cohere latest
run: |
set -x # print commands that are executed
./scripts/runtox.sh "py${{ matrix.python-version }}-cohere-latest" --cov=tests --cov=sentry_sdk --cov-report= --cov-branch
- name: Test langchain latest
run: |
set -x # print commands that are executed
./scripts/runtox.sh "py${{ matrix.python-version }}-langchain-latest" --cov=tests --cov=sentry_sdk --cov-report= --cov-branch
- name: Test openai latest
run: |
set -x # print commands that are executed
./scripts/runtox.sh "py${{ matrix.python-version }}-openai-latest" --cov=tests --cov=sentry_sdk --cov-report= --cov-branch
- name: Test huggingface_hub latest
run: |
set -x # print commands that are executed
./scripts/runtox.sh "py${{ matrix.python-version }}-huggingface_hub-latest" --cov=tests --cov=sentry_sdk --cov-report= --cov-branch
- name: Generate coverage XML
run: |
coverage combine .coverage*
coverage xml -i
- uses: codecov/codecov-action@v4
with:
token: ${{ secrets.CODECOV_TOKEN }}
files: coverage.xml
test-ai-pinned:
name: AI (pinned)
timeout-minutes: 30
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
python-version: ["3.7","3.9","3.11","3.12"]
# python3.6 reached EOL and is no longer being supported on
# new versions of hosted runners on Github Actions
# ubuntu-20.04 is the last version that supported python3.6
# see https://github.com/actions/setup-python/issues/544#issuecomment-1332535877
os: [ubuntu-20.04]
steps:
- uses: actions/[email protected]
- uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Setup Test Env
run: |
pip install coverage tox
- name: Erase coverage
run: |
coverage erase
- name: Test anthropic pinned
run: |
set -x # print commands that are executed
./scripts/runtox.sh --exclude-latest "py${{ matrix.python-version }}-anthropic" --cov=tests --cov=sentry_sdk --cov-report= --cov-branch
- name: Test cohere pinned
run: |
set -x # print commands that are executed
./scripts/runtox.sh --exclude-latest "py${{ matrix.python-version }}-cohere" --cov=tests --cov=sentry_sdk --cov-report= --cov-branch
- name: Test langchain pinned
run: |
set -x # print commands that are executed
./scripts/runtox.sh --exclude-latest "py${{ matrix.python-version }}-langchain" --cov=tests --cov=sentry_sdk --cov-report= --cov-branch
- name: Test openai pinned
run: |
set -x # print commands that are executed
./scripts/runtox.sh --exclude-latest "py${{ matrix.python-version }}-openai" --cov=tests --cov=sentry_sdk --cov-report= --cov-branch
- name: Test huggingface_hub pinned
run: |
set -x # print commands that are executed
./scripts/runtox.sh --exclude-latest "py${{ matrix.python-version }}-huggingface_hub" --cov=tests --cov=sentry_sdk --cov-report= --cov-branch
- name: Generate coverage XML
run: |
coverage combine .coverage*
coverage xml -i
- uses: codecov/codecov-action@v4
with:
token: ${{ secrets.CODECOV_TOKEN }}
files: coverage.xml
check_required_tests:
name: All AI tests passed
needs: test-ai-pinned
# Always run this, even if a dependent job failed
if: always()
runs-on: ubuntu-20.04
steps:
- name: Check for failures
if: contains(needs.test-ai-pinned.result, 'failure') || contains(needs.test-ai-pinned.result, 'skipped')
run: |
echo "One of the dependent jobs has failed. You may need to re-run it." && exit 1
42 changes: 1 addition & 41 deletions .github/workflows/test-integrations-data-processing.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: ["3.6","3.7","3.8","3.9","3.11","3.12"]
python-version: ["3.6","3.7","3.8","3.11","3.12"]
# python3.6 reached EOL and is no longer being supported on
# new versions of hosted runners on Github Actions
# ubuntu-20.04 is the last version that supported python3.6
Expand All @@ -44,10 +44,6 @@ jobs:
- name: Erase coverage
run: |
coverage erase
- name: Test anthropic latest
run: |
set -x # print commands that are executed
./scripts/runtox.sh "py${{ matrix.python-version }}-anthropic-latest" --cov=tests --cov=sentry_sdk --cov-report= --cov-branch
- name: Test arq latest
run: |
set -x # print commands that are executed
Expand All @@ -60,26 +56,10 @@ jobs:
run: |
set -x # print commands that are executed
./scripts/runtox.sh "py${{ matrix.python-version }}-celery-latest" --cov=tests --cov=sentry_sdk --cov-report= --cov-branch
- name: Test cohere latest
run: |
set -x # print commands that are executed
./scripts/runtox.sh "py${{ matrix.python-version }}-cohere-latest" --cov=tests --cov=sentry_sdk --cov-report= --cov-branch
- name: Test huey latest
run: |
set -x # print commands that are executed
./scripts/runtox.sh "py${{ matrix.python-version }}-huey-latest" --cov=tests --cov=sentry_sdk --cov-report= --cov-branch
- name: Test langchain latest
run: |
set -x # print commands that are executed
./scripts/runtox.sh "py${{ matrix.python-version }}-langchain-latest" --cov=tests --cov=sentry_sdk --cov-report= --cov-branch
- name: Test openai latest
run: |
set -x # print commands that are executed
./scripts/runtox.sh "py${{ matrix.python-version }}-openai-latest" --cov=tests --cov=sentry_sdk --cov-report= --cov-branch
- name: Test huggingface_hub latest
run: |
set -x # print commands that are executed
./scripts/runtox.sh "py${{ matrix.python-version }}-huggingface_hub-latest" --cov=tests --cov=sentry_sdk --cov-report= --cov-branch
- name: Test rq latest
run: |
set -x # print commands that are executed
Expand Down Expand Up @@ -118,10 +98,6 @@ jobs:
- name: Erase coverage
run: |
coverage erase
- name: Test anthropic pinned
run: |
set -x # print commands that are executed
./scripts/runtox.sh --exclude-latest "py${{ matrix.python-version }}-anthropic" --cov=tests --cov=sentry_sdk --cov-report= --cov-branch
- name: Test arq pinned
run: |
set -x # print commands that are executed
Expand All @@ -134,26 +110,10 @@ jobs:
run: |
set -x # print commands that are executed
./scripts/runtox.sh --exclude-latest "py${{ matrix.python-version }}-celery" --cov=tests --cov=sentry_sdk --cov-report= --cov-branch
- name: Test cohere pinned
run: |
set -x # print commands that are executed
./scripts/runtox.sh --exclude-latest "py${{ matrix.python-version }}-cohere" --cov=tests --cov=sentry_sdk --cov-report= --cov-branch
- name: Test huey pinned
run: |
set -x # print commands that are executed
./scripts/runtox.sh --exclude-latest "py${{ matrix.python-version }}-huey" --cov=tests --cov=sentry_sdk --cov-report= --cov-branch
- name: Test langchain pinned
run: |
set -x # print commands that are executed
./scripts/runtox.sh --exclude-latest "py${{ matrix.python-version }}-langchain" --cov=tests --cov=sentry_sdk --cov-report= --cov-branch
- name: Test openai pinned
run: |
set -x # print commands that are executed
./scripts/runtox.sh --exclude-latest "py${{ matrix.python-version }}-openai" --cov=tests --cov=sentry_sdk --cov-report= --cov-branch
- name: Test huggingface_hub pinned
run: |
set -x # print commands that are executed
./scripts/runtox.sh --exclude-latest "py${{ matrix.python-version }}-huggingface_hub" --cov=tests --cov=sentry_sdk --cov-report= --cov-branch
- name: Test rq pinned
run: |
set -x # print commands that are executed
Expand Down
12 changes: 7 additions & 5 deletions scripts/split-tox-gh-actions/split-tox-gh-actions.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,13 @@
"Common": [
"common",
],
"AI": [
"anthropic",
"cohere",
"langchain",
"openai",
"huggingface_hub",
],
"AWS Lambda": [
# this is separate from Cloud Computing because only this one test suite
# needs to run with access to GitHub secrets
Expand All @@ -70,15 +77,10 @@
"gcp",
],
"Data Processing": [
"anthropic",
"arq",
"beam",
"celery",
"cohere",
"huey",
"langchain",
"openai",
"huggingface_hub",
"rq",
],
"Databases": [
Expand Down
Loading