Skip to content

Commit cbebfc9

Browse files
gabalafoudrammocktrallard
authored
Accessibility test Kitchen Sink with Playwright (#1260)
* Add core-js so dev build works Install pytest-playwright and (npm) axe-core Write test that scans admonitions with Axe How to run test: nox -s compile nox -s docs python -m http.server -d docs/_build/html/ nox -s test -- -k accessibility * add more pages, test server * add documentation * add prerequisites to nox test * update package-lock.json via stb * update package-lock.json via nox -s compile * streamline, update docs * typo * sync lockfile (remove core-js) * Update tests/utils.py Co-authored-by: Daniel McCloy <[email protected]> * Update tests/README.md Co-authored-by: Tania Allard <[email protected]> * Update docs/community/topics/accessibility.md Co-authored-by: Tania Allard <[email protected]> * Update tests/README.md Co-authored-by: Tania Allard <[email protected]> * Update tests/utils.py Co-authored-by: Tania Allard <[email protected]> * Update tests/utils.py Co-authored-by: Tania Allard <[email protected]> * - nox session a11y - add pytest-base-url plugin - push more test setup into page fixture - add license header to pretty_axe_results - move pretty_axe_results and check_warnings to utils/ * update docs * uncomment commented-out tests * regenerate package-lock.json via: rm package-lock.json rm -r node_modules stb npm install --include=dev * oops finish updating tests * do not include a11y tests in main test session * remove redundant decorator * Update tests/README.md Co-authored-by: Daniel McCloy <[email protected]> * Update tests/README.md Co-authored-by: Daniel McCloy <[email protected]> * Update tests/README.md Co-authored-by: Tania Allard <[email protected]> * Remove base-url Pytest plugin * Update tests/utils/pretty_axe_results.py Co-authored-by: Tania Allard <[email protected]> * Apply suggestions from code review Co-authored-by: Tania Allard <[email protected]> * fix pretty_axe_results * Generate each test run from parameters * remove unused marker url_path * undo multiline * update doc string * Add CI check for --with-deps suggested by Tony * oops, rm extra playwright install line * update comment * update docstring * testing the pytest-axe mark * run prettier * Revert "testing the pytest-axe mark" This reverts commit f7b92ea. * Fix D104 * skip a11y tests on CI runs * run a11y tests on one CI worker only * ✅ Register a11y mark and separate dependencies * 👷 Update CI - delegate playwright install and cache * 📝 Update test docs * ✅ Ensure a11y tests are skipped unles called * 👷 Ensure CI installs all deps * ⏪ Revert to installing playwright * 👷 Forgot to re-add a11y deps * ✏️ Fix typo * 🔧 Fix path - seems to have been messed up when moving things * Update tests/test_a11y.py --------- Co-authored-by: Daniel McCloy <[email protected]> Co-authored-by: Tania Allard <[email protected]>
1 parent d939eda commit cbebfc9

14 files changed

+327
-63
lines changed

.github/workflows/prerelease.yml

+1-1
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ jobs:
3636
3737
- name: Check that there are no unexpected Sphinx warnings
3838
if: matrix.python-version == '3.10'
39-
run: python tests/check_warnings.py
39+
run: python tests/utils/check_warnings.py
4040

4141
- name: Run the tests
4242
run: |

.github/workflows/tests.yml

+21-2
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,9 @@ jobs:
5656
# windows test
5757
- os: windows-latest
5858
python-version: "3.11"
59+
# needed to cache the browsers for the accessibility tests
60+
env:
61+
PLAYWRIGHT_BROWSERS_PATH: ${{ github.workspace }}/pw-browsers
5962
runs-on: ${{ matrix.os }}
6063
steps:
6164
- uses: actions/checkout@v3
@@ -90,14 +93,30 @@ jobs:
9093
pip install nox
9194
nox -s compile
9295
- name: Run tests
93-
run: pytest --color=yes --cov pydata_sphinx_theme --cov-branch --cov-report xml:cov.xml --cov-report= --cov-fail-under ${{ env.COVERAGE_THRESHOLD }}
96+
run: pytest -m "not a11y" --color=yes --cov pydata_sphinx_theme --cov-branch --cov-report xml:cov.xml --cov-report= --cov-fail-under ${{ env.COVERAGE_THRESHOLD }}
9497
- name: Upload to Codecov
9598
if: matrix.python-version == '3.11' && matrix.os == 'ubuntu-latest'
9699
uses: codecov/[email protected]
97100
with:
98101
files: cov.xml
99102
fail_ci_if_error: true
100103

104+
# note I am setting this on top of the Python cache as I could not find
105+
# how to set the hash key on the python one
106+
- name: Set up browser cache (for accessibility tests)
107+
uses: actions/cache@v3
108+
with:
109+
path: |
110+
${{ github.workspace }}/pw-browsers
111+
key: ${{ runner.os }}-pw-${{ hashFiles('pyproject.toml') }}
112+
restore-keys: |
113+
${{ runner.os }}-pw-
114+
115+
- name: Run accessibility tests with playwright
116+
if: matrix.python-version == '3.11' && matrix.os == 'ubuntu-latest'
117+
run: |
118+
nox -s a11y
119+
101120
# Build our site on the 3 major OSes and check for Sphinx warnings
102121
build-site:
103122
needs: [lint]
@@ -124,7 +143,7 @@ jobs:
124143
- name: Build docs
125144
run: sphinx-build -b html docs/ docs/_build/html --keep-going -w warnings.txt
126145
- name: Check for unexpected Sphinx warnings
127-
run: python tests/check_warnings.py
146+
run: python tests/utils/check_warnings.py
128147

129148
# Run local Lighthouse audit against built site
130149
audit:

docs/community/setup.md

+32-18
Original file line numberDiff line numberDiff line change
@@ -22,10 +22,10 @@ The sections below cover the steps to do this in more detail.
2222

2323
## Clone the repository
2424

25-
First off you'll need your own copy of the `pydata-sphinx-theme` codebase.
25+
First off you'll need your copy of the `pydata-sphinx-theme` codebase.
2626
You can clone it for local development like so:
2727

28-
1. **Fork the repository** so you have your own copy on GitHub.
28+
1. **Fork the repository**, so you have your own copy on GitHub.
2929
See [the GitHub forking guide](https://docs.github.com/en/get-started/quickstart/fork-a-repo) for more information.
3030
2. **Clone the repository locally** so that you have a local copy to work from:
3131

@@ -36,14 +36,14 @@ You can clone it for local development like so:
3636

3737
## Install your tools
3838

39-
Building a Sphinx site uses a combination of Python and Jinja to manage HTML, SCSS, and Javascript.
39+
Building a Sphinx site uses a combination of Python and Jinja to manage HTML, SCSS, and JavaScript.
4040
To simplify this process, we use a few helper tools:
4141

42-
- [The Sphinx Theme Builder](https://sphinx-theme-builder.readthedocs.io/en/latest/) to automatically perform compilation of web assets.
42+
- [The Sphinx Theme Builder](https://sphinx-theme-builder.readthedocs.io/en/latest/) compiles web assets in an automated way.
4343
- [pre-commit](https://pre-commit.com/) for automatically enforcing code standards and quality checks before commits.
44-
- [nox](https://nox.thea.codes/), for automating common development tasks.
44+
- [nox](https://nox.thea.codes/) for automating common development tasks.
4545

46-
In particular, `nox` can be used to automatically create isolated local development environments with all of the correct packages installed to work on the theme.
46+
In particular, `nox` can be used to automatically create isolated local development environments with all the correct packages installed to work on the theme.
4747
The rest of this guide focuses on using `nox` to start with a basic environment.
4848

4949
```{seealso}
@@ -58,9 +58,9 @@ To start, install `nox`:
5858
$ pip install nox
5959
```
6060

61-
You can call `nox` from the command line in order to perform common actions that are needed in building the theme.
61+
You can call `nox` from the command line to perform common actions that are needed in building the theme.
6262
`nox` operates with isolated environments, so each action has its own packages installed in a local directory (`.nox`).
63-
For common development actions, you'll simply need to use `nox` and won't need to set up any other packages.
63+
For common development actions, you'll only need to use `nox` and won't need to set up any other packages.
6464

6565
### Setup `pre-commit`
6666

@@ -116,7 +116,7 @@ Now that you've built the documentation, edit one of the source files to see how
116116
1. **Make an edit to a page**. For example, add a word or fix a typo on any page.
117117
2. **Rebuild the documentation** with `nox -s docs`
118118

119-
It should go much faster this time, because `nox` is re-using the old environment, and because Sphinx has cached the pages that you didn't change.
119+
It should go much faster this time because `nox` is re-using the previously created environment, and because Sphinx has cached the pages that you didn't change.
120120

121121
## Compile the CSS/JS assets
122122

@@ -138,15 +138,15 @@ The `sphinx-theme-builder` will bundle these assets automatically when we make a
138138

139139
## Run a development server
140140

141-
You can combine the above two actions and run a development server so that changes to `src/` are automatically bundled with the package, and the documentation is immediately reloaded in a live preview window.
141+
You can combine the above two actions (build the docs and compile JS/CSS assets) and run a development server so that changes to `src/` are automatically bundled with the package, and the documentation is immediately reloaded in a live preview window.
142142

143143
To run the development server with `nox`, run the following command:
144144

145145
```console
146146
$ nox -s docs-live
147147
```
148148

149-
When working on the theme, saving changes to any of these directories:
149+
When working on the theme, making changes to any of these directories:
150150

151151
- `src/js/index.js`
152152
- `src/scss/index.scss`
@@ -161,19 +161,33 @@ will cause the development server to do the following:
161161

162162
## Run the tests
163163

164-
This theme uses `pytest` for its testing, with a lightweight fixture defined
165-
in the `test_build.py` script that makes it easy to run a Sphinx build using
166-
this theme and inspect the results.
164+
This theme uses `pytest` for its testing. There is a lightweight fixture defined
165+
in the `test_build.py` script that makes it straightforward to run a Sphinx build using
166+
this theme and inspect the results. There are also several automated accessibility checks in
167+
`test_a11y.py`.
167168

168-
In addition, we use [pytest-regressions](https://pytest-regressions.readthedocs.io/en/latest/)
169-
to ensure that the HTML generated by the theme is what we'd expect. This module
169+
```{warning}
170+
Currently, the automated accessibility tests check the Kitchen Sink page only.
171+
We are working on extending coverage to the rest of the theme.
172+
```
173+
174+
In addition, we use
175+
[pytest-regressions](https://pytest-regressions.readthedocs.io/en/latest/) to
176+
ensure that the HTML generated by the theme is what we'd expect. This module
170177
provides a `file_regression` fixture that will check the contents of an object
171178
against a reference file on disk. If the structure of the two differs, then the
172179
test will fail. If we _expect_ the structure to differ, then delete the file on
173-
disk and run the test. A new file will be created, and subsequent tests will pass.
180+
disk and run the test. A new file will be created, and subsequent tests will
181+
pass.
174182

175-
To run the tests with `nox`, run the following command:
183+
To run the build tests with `nox`, run the following command:
176184

177185
```console
178186
$ nox -s test
179187
```
188+
189+
To run the accessibility checks:
190+
191+
```console
192+
$ nox -s a11y
193+
```
+12-5
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,18 @@
11
# Accessibility checks
22

3-
The accessibility checking tools can find a number of common HTML patterns which
3+
```{note}
4+
April-2023: we are currently
5+
[re-evaluating how we do accessibility checks](https://github.com/pydata/pydata-sphinx-theme/issues/1168)
6+
and reporting, so this may change soon.
7+
```
8+
9+
In general, accessibility-checking tools can find a limited number of common HTML patterns which
410
assistive technology can't help users understand.
5-
We run a [Lighthouse](https://developers.google.com/web/tools/lighthouse) job in our CI/CD, which generates a "score" for all pages in our **Kitchen Sink** example documentation.
6-
The configuration for Lighthouse is in:
711

8-
- `.github/workflows/lighthouserc.json`
12+
## Accessibility checks as part of our development process
13+
14+
We run a [Lighthouse](https://developers.google.com/web/tools/lighthouse) job in our CI/CD, which generates a "score" for all pages in our **Kitchen Sink** example documentation.
15+
The configuration for Lighthouse can be found in the `.github/workflows/lighthouserc.json` file.
916

10-
For more information about configuring lighthouse, see [the lighthouse documentation](https://github.com/GoogleChrome/lighthouse-ci/blob/main/docs/configuration.md).
17+
For more information about configuring Lighthouse, see [the Lighthouse documentation](https://github.com/GoogleChrome/lighthouse-ci/blob/main/docs/configuration.md).
1118
For more information about Accessibility in general, see [](../../user_guide/accessibility.rst).

noxfile.py

+23-1
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@
55
66
nox -s docs -- -r
77
"""
8+
import os
89
import shutil as sh
910
import tempfile
1011
from pathlib import Path
@@ -87,7 +88,28 @@ def test(session: nox.Session) -> None:
8788
if _should_install(session):
8889
session.install("-e", ".[test]")
8990
session.run(*split("pybabel compile -d src/pydata_sphinx_theme/locale -D sphinx"))
90-
session.run("pytest", *session.posargs)
91+
session.run("pytest", "-m", "not a11y", *session.posargs)
92+
93+
94+
@nox.session()
95+
def a11y(session: nox.Session) -> None:
96+
"""Run the accessibility test suite only."""
97+
if _should_install(session):
98+
session.install("-e", ".[test, a11y]")
99+
# Install the drivers that Playwright needs to control the browsers.
100+
if os.environ.get("CI") or os.environ.get("GITPOD_WORKSPACE_ID"):
101+
# CI and other cloud environments are potentially missing system
102+
# dependencies, so we tell Playwright to also install the system
103+
# dependencies
104+
session.run("playwright", "install", "--with-deps")
105+
else:
106+
# But most dev environments have the needed system dependencies
107+
session.run("playwright", "install")
108+
# Build the docs so we can run accessibility tests against them.
109+
session.run("nox", "-s", "docs")
110+
# The next step would be to open a server to the docs for Playwright, but
111+
# that is done in the test file, along with the accessibility checks.
112+
session.run("pytest", "-m", "a11y", *session.posargs)
91113

92114

93115
@nox.session(name="test-sphinx")

package-lock.json

+38-18
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

package.json

+1
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@
55
"build": "webpack"
66
},
77
"devDependencies": {
8+
"axe-core": "^4.6.3",
89
"copy-webpack-plugin": "^11.0.0",
910
"css-loader": "^3.4.2",
1011
"css-minimizer-webpack-plugin": "^4.2.2",

0 commit comments

Comments
 (0)