Skip to content

Commit aeb0191

Browse files
docs: write guide on adding new test suites to circleci (#6773)
## Motivation This PR aims to explain how to add new test suites in the contributing docs. This can be highly useful for new contributors as a reference to follow when writing new integrations for `dd-trace-py`. ## Checklist - [X] Change(s) are motivated and described in the PR description. - [X] Testing strategy is described if automated tests are not included in the PR. - [X] Risk is outlined (performance impact, potential for breakage, maintainability, etc). - [X] Change is maintainable (easy to change, telemetry, documentation). - [X] [Library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) are followed. If no release note is required, add label `changelog/no-changelog`. - [X] Documentation is included (in-code, generated user docs, [public corp docs](https://github.com/DataDog/documentation/)). - [X] Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Title is accurate. - [x] No unnecessary changes are introduced. - [x] Description motivates each change. - [x] Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes unless absolutely necessary. - [x] Testing strategy adequately addresses listed risk(s). - [x] Change is maintainable (easy to change, telemetry, documentation). - [x] Release note makes sense to a user of the library. - [x] Reviewer has explicitly acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment. - [x] Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) - [x] If this PR touches code that signs or publishes builds or packages, or handles credentials of any kind, I've requested a review from `@DataDog/security-design-and-guidance`. - [x] This PR doesn't touch any of that. --------- Co-authored-by: Emmett Butler <[email protected]>
1 parent 086132e commit aeb0191

File tree

2 files changed

+65
-2
lines changed

2 files changed

+65
-2
lines changed

docs/contributing-integrations.rst

+14-2
Original file line numberDiff line numberDiff line change
@@ -173,8 +173,9 @@ Many of the tests are based on "snapshots": saved copies of actual traces sent t
173173
`APM test agent <../README.md#use-the-apm-test-agent>`_.
174174

175175
To update the snapshots expected by a test, first update the library and test code to generate
176-
new traces. Then, delete the snapshot file corresponding to your test. Use `docker-compose up -d testagent`
177-
to start the APM test agent, and re-run the test. Use `--pass-env` as described
176+
new traces. Then, delete the snapshot file corresponding to your test at ``tests/snapshots/<snapshot_file>``.
177+
178+
Use `docker-compose up -d testagent` to start the APM test agent, and then re-run the test. Use `--pass-env` as described
178179
`here <../README.md#use-the-apm-test-agent>`_ to ensure that your test run can talk to the
179180
test agent. Once the run finishes, the snapshot file will have been regenerated.
180181

@@ -248,6 +249,17 @@ are not yet any expected spans stored for it, so we need to create some.
248249
not match.
249250
13. Repeat steps 7 through 9 until you've achieved test coverage for the entire "happy path" of normal usage
250251
for the library you're integrating with, as well as coverage of any known likely edge cases.
252+
14. Enable the `snapshot` option in `.circleci/config.templ.yml` and run the test as a `machine_executor` at ``.circleci/config.templ.yml``
253+
just like:
254+
255+
.. code-block:: yaml
256+
257+
<test_suite_name>:
258+
<<: *machine_executor
259+
steps:
260+
- run_test:
261+
pattern: '<test_suite_name>'
262+
snapshot: true
251263
252264
253265
Trace Examples

docs/contributing-testing.rst

+51
Original file line numberDiff line numberDiff line change
@@ -117,3 +117,54 @@ when the riotfile changes. Thus, if you make changes to the riotfile, you need t
117117
118118
You can commit and pull request the resulting changes to files in ``.riot/requirements`` alongside the
119119
changes you made to ``riotfile.py``.
120+
121+
How do I add a new test suite?
122+
------------------------------
123+
124+
We use `riot <https://ddriot.readthedocs.io/en/latest/>`_, a Python virtual environment constructor, to run the test suites.
125+
It is necessary to create a new ``Venv`` instance in ``riotfile.py`` if it does not exist already. It can look like this:
126+
127+
.. code-block:: python
128+
129+
Venv(
130+
name="asyncio",
131+
command="pytest {cmdargs} tests/contrib/asyncio",
132+
pys=select_pys(),
133+
pkgs={
134+
"pytest-asyncio": latest,
135+
},
136+
env={
137+
"DD_ENV_VARIABLE": "1", # if needed
138+
},
139+
)
140+
141+
Once a ``Venv`` instance has been created, you will be able to run it as explained in the section below.
142+
Next, we will need to add a new CircleCI job to run the newly added test suite at ``.circleci/config.templ.yml`` just like:
143+
144+
.. code-block:: python
145+
146+
asyncio:
147+
<<: *contrib_job
148+
steps:
149+
- run_test:
150+
pattern: 'asyncio'
151+
152+
153+
After this, a new component must be added to ``tests/.suitespec.json`` under ``"components":`` like:
154+
155+
.. code-block:: JSON
156+
157+
"asyncio": [
158+
"ddtrace/contrib/asyncio/*"
159+
],
160+
161+
Lastly, we will register it as a suite in the same file under ``"suites":``:
162+
163+
.. code-block:: JSON
164+
165+
"asyncio": [
166+
"@asyncio",
167+
"tests/contrib/asyncio/*"
168+
],
169+
170+
Once you've completed these steps, CircleCI will run the new test suite.

0 commit comments

Comments
 (0)