Skip to content

docs: write guide on adding new test suites to circleci #6773

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Aug 29, 2023
Merged
16 changes: 14 additions & 2 deletions docs/contributing-integrations.rst
Original file line number Diff line number Diff line change
Expand Up @@ -173,8 +173,9 @@ Many of the tests are based on "snapshots": saved copies of actual traces sent t
`APM test agent <../README.md#use-the-apm-test-agent>`_.

To update the snapshots expected by a test, first update the library and test code to generate
new traces. Then, delete the snapshot file corresponding to your test. Use `docker-compose up -d testagent`
to start the APM test agent, and re-run the test. Use `--pass-env` as described
new traces. Then, delete the snapshot file corresponding to your test at ``tests/snapshots/<snapshot_file>``.

Use `docker-compose up -d testagent` to start the APM test agent, and then re-run the test. Use `--pass-env` as described
`here <../README.md#use-the-apm-test-agent>`_ to ensure that your test run can talk to the
test agent. Once the run finishes, the snapshot file will have been regenerated.

Expand Down Expand Up @@ -248,6 +249,17 @@ are not yet any expected spans stored for it, so we need to create some.
not match.
13. Repeat steps 7 through 9 until you've achieved test coverage for the entire "happy path" of normal usage
for the library you're integrating with, as well as coverage of any known likely edge cases.
14. Enable the `snapshot` option in `.circleci/config.templ.yml` and run the test as a `machine_executor` at ``.circleci/config.templ.yml``
just like:

.. code-block:: yaml

<test_suite_name>:
<<: *machine_executor
steps:
- run_test:
pattern: '<test_suite_name>'
snapshot: true


Trace Examples
Expand Down
51 changes: 51 additions & 0 deletions docs/contributing-testing.rst
Original file line number Diff line number Diff line change
Expand Up @@ -117,3 +117,54 @@ when the riotfile changes. Thus, if you make changes to the riotfile, you need t

You can commit and pull request the resulting changes to files in ``.riot/requirements`` alongside the
changes you made to ``riotfile.py``.

How do I add a new test suite?
------------------------------

We use `riot <https://ddriot.readthedocs.io/en/latest/>`_, a Python virtual environment constructor, to run the test suites.
It is necessary to create a new ``Venv`` instance in ``riotfile.py`` if it does not exist already. It can look like this:

.. code-block:: python

Venv(
name="asyncio",
command="pytest {cmdargs} tests/contrib/asyncio",
pys=select_pys(),
pkgs={
"pytest-asyncio": latest,
},
env={
"DD_ENV_VARIABLE": "1", # if needed
},
)

Once a ``Venv`` instance has been created, you will be able to run it as explained in the section below.
Next, we will need to add a new CircleCI job to run the newly added test suite at ``.circleci/config.templ.yml`` just like:

.. code-block:: python

asyncio:
<<: *contrib_job
steps:
- run_test:
pattern: 'asyncio'


After this, a new component must be added to ``tests/.suitespec.json`` under ``"components":`` like:

.. code-block:: JSON

"asyncio": [
"ddtrace/contrib/asyncio/*"
],

Lastly, we will register it as a suite in the same file under ``"suites":``:

.. code-block:: JSON

"asyncio": [
"@asyncio",
"tests/contrib/asyncio/*"
],

Once you've completed these steps, CircleCI will run the new test suite.