Skip to content

Add performance regression tests and performance reports to CI #75

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 16 commits into from
Aug 26, 2019
Merged

Add performance regression tests and performance reports to CI #75

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
b02e2ed
add regression test
Aug 8, 2019
411063c
pass correct args; add graph generation
Aug 8, 2019
8ae4db8
output test stdout
Aug 8, 2019
33a60a1
no need to make graphs folder
Aug 8, 2019
8e47624
break graph generation into separate jobs
Aug 9, 2019
5eb310c
rename regression tests
Aug 20, 2019
2e5a629
clone from new repo
Aug 21, 2019
49471c1
clone safe & current lightstep-benchmarks with semantic versioning
Aug 21, 2019
a9ad1f3
clone safe & current lightstep-benchmarks with semantic versioning
Aug 21, 2019
bf0380a
use xlarge resource class for all CI jobs
Aug 21, 2019
4e1a91b
only check lightstep-benchmarks major version
Aug 21, 2019
77eeb55
resolve merge conflict
Aug 21, 2019
56b2b09
checkout branch
Aug 21, 2019
b53e7f8
properly clone lightstep-benchmarks
Aug 22, 2019
85d92c5
add some debugging info
Aug 22, 2019
b6fece8
run more CPU trials
Aug 26, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
214 changes: 159 additions & 55 deletions .circleci/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,18 +7,18 @@ jobs:
CIRCLE_ARTIFACTS: /tmp/circleci-artifacts
CIRCLE_TEST_REPORTS: /tmp/circleci-test-results
docker:
- image: circleci/python:2.7
- image: circleci/python:2.7
steps:
- checkout
- run: mkdir -p $CIRCLE_ARTIFACTS $CIRCLE_TEST_REPORTS
- run: sudo pip install tox
- run: make test27
- store_test_results:
path: /tmp/circleci-test-results
- store_artifacts:
path: /tmp/circleci-artifacts
- store_artifacts:
path: /tmp/circleci-test-results
- checkout
- run: mkdir -p $CIRCLE_ARTIFACTS $CIRCLE_TEST_REPORTS
- run: sudo pip install tox
- run: make test27
- store_test_results:
path: /tmp/circleci-test-results
- store_artifacts:
path: /tmp/circleci-artifacts
- store_artifacts:
path: /tmp/circleci-test-results

test-python3.4:
working_directory: ~/lightstep/lightstep-tracer-python
Expand All @@ -27,18 +27,18 @@ jobs:
CIRCLE_ARTIFACTS: /tmp/circleci-artifacts
CIRCLE_TEST_REPORTS: /tmp/circleci-test-results
docker:
- image: circleci/python:3.4
- image: circleci/python:3.4
steps:
- checkout
- run: mkdir -p $CIRCLE_ARTIFACTS $CIRCLE_TEST_REPORTS
- run: sudo pip install tox
- run: make test34
- store_test_results:
path: /tmp/circleci-test-results
- store_artifacts:
path: /tmp/circleci-artifacts
- store_artifacts:
path: /tmp/circleci-test-results
- checkout
- run: mkdir -p $CIRCLE_ARTIFACTS $CIRCLE_TEST_REPORTS
- run: sudo pip install tox
- run: make test34
- store_test_results:
path: /tmp/circleci-test-results
- store_artifacts:
path: /tmp/circleci-artifacts
- store_artifacts:
path: /tmp/circleci-test-results

test-python3.5:
working_directory: ~/lightstep/lightstep-tracer-python
Expand All @@ -47,18 +47,18 @@ jobs:
CIRCLE_ARTIFACTS: /tmp/circleci-artifacts
CIRCLE_TEST_REPORTS: /tmp/circleci-test-results
docker:
- image: circleci/python:3.5
- image: circleci/python:3.5
steps:
- checkout
- run: mkdir -p $CIRCLE_ARTIFACTS $CIRCLE_TEST_REPORTS
- run: sudo pip install tox
- run: make test35
- store_test_results:
path: /tmp/circleci-test-results
- store_artifacts:
path: /tmp/circleci-artifacts
- store_artifacts:
path: /tmp/circleci-test-results
- checkout
- run: mkdir -p $CIRCLE_ARTIFACTS $CIRCLE_TEST_REPORTS
- run: sudo pip install tox
- run: make test35
- store_test_results:
path: /tmp/circleci-test-results
- store_artifacts:
path: /tmp/circleci-artifacts
- store_artifacts:
path: /tmp/circleci-test-results

test-python3.6:
working_directory: ~/lightstep/lightstep-tracer-python
Expand All @@ -67,18 +67,18 @@ jobs:
CIRCLE_ARTIFACTS: /tmp/circleci-artifacts
CIRCLE_TEST_REPORTS: /tmp/circleci-test-results
docker:
- image: circleci/python:3.6
- image: circleci/python:3.6
steps:
- checkout
- run: mkdir -p $CIRCLE_ARTIFACTS $CIRCLE_TEST_REPORTS
- run: sudo pip install tox
- run: make test36
- store_test_results:
path: /tmp/circleci-test-results
- store_artifacts:
path: /tmp/circleci-artifacts
- store_artifacts:
path: /tmp/circleci-test-results
- checkout
- run: mkdir -p $CIRCLE_ARTIFACTS $CIRCLE_TEST_REPORTS
- run: sudo pip install tox
- run: make test36
- store_test_results:
path: /tmp/circleci-test-results
- store_artifacts:
path: /tmp/circleci-artifacts
- store_artifacts:
path: /tmp/circleci-test-results

test-python3.7:
working_directory: ~/lightstep/lightstep-tracer-python
Expand All @@ -87,18 +87,107 @@ jobs:
CIRCLE_ARTIFACTS: /tmp/circleci-artifacts
CIRCLE_TEST_REPORTS: /tmp/circleci-test-results
docker:
- image: circleci/python:3.7
- image: circleci/python:3.7
steps:
- checkout
- run: mkdir -p $CIRCLE_ARTIFACTS $CIRCLE_TEST_REPORTS
- run: sudo pip install tox
- run: make test37
- store_test_results:
path: /tmp/circleci-test-results
- store_artifacts:
path: /tmp/circleci-artifacts
- store_artifacts:
path: /tmp/circleci-test-results
- checkout
- run: mkdir -p $CIRCLE_ARTIFACTS $CIRCLE_TEST_REPORTS
- run: sudo pip install tox
- run: make test37
- store_test_results:
path: /tmp/circleci-test-results
- store_artifacts:
path: /tmp/circleci-artifacts
- store_artifacts:
path: /tmp/circleci-test-results

regression_test:
resource_class: xlarge
docker:
- image: circleci/python:3.7.4
working_directory: ~/repo
steps:
- checkout
- run: cd ~; ~/repo/ci/clone_lightstep_benchmarks.sh
- run: ~/lightstep-benchmarks/scripts/ci_setup.sh
- run: sudo pip install -e ~/repo
- run: |
cd ~/lightstep-benchmarks/
pytest -s --client_name python regression_tests.py
- store_artifacts:
path: ~/lightstep-benchmarks/logs
destination: logs

make_cpu_graphs:
resource_class: xlarge
docker:
- image: circleci/python:3.7.4
working_directory: ~/repo
steps:
- checkout
- run: cd ~; ~/repo/ci/clone_lightstep_benchmarks.sh
- run: ~/lightstep-benchmarks/scripts/ci_setup.sh
- run: sudo pip install -e ~/repo
- run: python ~/lightstep-benchmarks/cpu_graphs.py python --runtime 10 --trials 20
- store_artifacts:
path: ~/lightstep-benchmarks/graphs
destination: graphs
- store_artifacts:
path: ~/lightstep-benchmarks/logs
destination: logs

make_dropped_graphs:
resource_class: xlarge
docker:
- image: circleci/python:3.7.4
working_directory: ~/repo
steps:
- checkout
- run: cd ~; ~/repo/ci/clone_lightstep_benchmarks.sh
- run: ~/lightstep-benchmarks/scripts/ci_setup.sh
- run: sudo pip install -e ~/repo
- run: python ~/lightstep-benchmarks/dropped_graphs.py python
- store_artifacts:
path: ~/lightstep-benchmarks/graphs
destination: graphs
- store_artifacts:
path: ~/lightstep-benchmarks/logs
destination: logs

make_memory_graphs:
resource_class: xlarge
docker:
- image: circleci/python:3.7.4
working_directory: ~/repo
steps:
- checkout
- run: cd ~; ~/repo/ci/clone_lightstep_benchmarks.sh
- run: ~/lightstep-benchmarks/scripts/ci_setup.sh
- run: sudo pip install -e ~/repo
- run: python ~/lightstep-benchmarks/memory_graphs.py python
- store_artifacts:
path: ~/lightstep-benchmarks/graphs
destination: graphs
- store_artifacts:
path: ~/lightstep-benchmarks/logs
destination: logs

make_disconnect_graphs:
resource_class: xlarge
docker:
- image: circleci/python:3.7.4
working_directory: ~/repo
steps:
- checkout
- run: cd ~; ~/repo/ci/clone_lightstep_benchmarks.sh
- run: ~/lightstep-benchmarks/scripts/ci_setup.sh
- run: sudo pip install -e ~/repo
- run: python ~/lightstep-benchmarks/disconnect_graphs.py python
- store_artifacts:
path: ~/lightstep-benchmarks/graphs
destination: graphs
- store_artifacts:
path: ~/lightstep-benchmarks/logs
destination: logs

workflows:
version: 2
Expand All @@ -109,3 +198,18 @@ workflows:
- test-python3.5
- test-python3.6
- test-python3.7
- regression_test
- approve_make_graphs:
type: approval
- make_dropped_graphs:
requires:
- approve_make_graphs
- make_memory_graphs:
requires:
- approve_make_graphs
- make_disconnect_graphs:
requires:
- approve_make_graphs
- make_cpu_graphs:
requires:
- approve_make_graphs
27 changes: 27 additions & 0 deletions ci/clone_lightstep_benchmarks.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
#!/bin/bash

set -e

REPO_URL="https://github.com/lightstep/lightstep-benchmarks.git"
MAJOR_VERSION="0"

# clone the repo
git clone ${REPO_URL}
cd lightstep-benchmarks

# copy the tags to the local repo (from all remotes)
git fetch --all --tags

# find most recent release whose tag matches the specified major / minor
# version
MATCHING_VERSIONS=`git tag --sort -version:refname --list "v${MAJOR_VERSION}.*.*" | tr '\n' ','`
NEWEST_VERSION=`echo ${MATCHING_VERSIONS} | cut -d ',' -f 1`

echo "versions, newest to oldest: ${MATCHING_VERSIONS}"
echo "newest version: ${NEWEST_VERSION}"

# checkout the code corresponding to the newest tag
git checkout tags/${NEWEST_VERSION}

# so that we can confirm everything is a-okay
git branch; git tag