Accessibility test Kitchen Sink with Playwright

.github/workflows/prerelease.yml

@@ -36,7 +36,7 @@ jobs:
 
       - name: Check that there are no unexpected Sphinx warnings
         if: matrix.python-version == '3.10'
-        run: python tests/check_warnings.py
+        run: python tests/utils/check_warnings.py
 
       - name: Run the tests
         run: |

.github/workflows/tests.yml

@@ -56,6 +56,9 @@ jobs:
           # windows test
           - os: windows-latest
             python-version: "3.11"
+    # needed to cache the browsers for the accessibility tests
+    env:
+      PLAYWRIGHT_BROWSERS_PATH: ${{ github.workspace }}/pw-browsers
     runs-on: ${{ matrix.os }}
     steps:
       - uses: actions/checkout@v3
@@ -90,14 +93,34 @@ jobs:
           pip install nox
           nox -s compile
       - name: Run tests
-        run: pytest --color=yes --cov pydata_sphinx_theme --cov-branch --cov-report xml:cov.xml --cov-report= --cov-fail-under ${{ env.COVERAGE_THRESHOLD }}
+        run: pytest -m "not a11y" --color=yes --cov pydata_sphinx_theme --cov-branch --cov-report xml:cov.xml --cov-report= --cov-fail-under ${{ env.COVERAGE_THRESHOLD }}
       - name: Upload to Codecov
         if: matrix.python-version == '3.11' && matrix.os == 'ubuntu-latest'
         uses: codecov/[email protected]
         with:
           files: cov.xml
           fail_ci_if_error: true
 
+      # note I am setting this on top of the Python cache as I could not find
+      # how to set the hash key on the python one
+      - name: Set up browser cache (for accessibility tests)
+        uses: actions/cache@v3
+        with:
+          path: |
+            ${{ github.workspace }}/pw-browsers
+          key: ${{ runner.os }}-pw-${{ hashFiles('pyproject.toml') }}
+          restore-keys: |
+            ${{ runner.os }}-pw-
+
+      - name: Run accessibility tests with playwright
+        if: matrix.python-version == '3.11' && matrix.os == 'ubuntu-latest'
+        run: |
+          pip install playwright
+          python -m playwright install --with-deps
+          stb compile
+          sphinx-build -b=html docs/ docs/_build/html
+          pytest -m a11y
+
   # Build our site on the 3 major OSes and check for Sphinx warnings
   build-site:
     needs: [lint]
@@ -124,7 +147,7 @@ jobs:
       - name: Build docs
         run: sphinx-build -b html docs/ docs/_build/html --keep-going -w warnings.txt
       - name: Check for unexpected Sphinx warnings
-        run: python tests/check_warnings.py
+        run: python tests/utils/check_warnings.py
 
   # Run local Lighthouse audit against built site
   audit:

docs/community/setup.md

@@ -22,10 +22,10 @@ The sections below cover the steps to do this in more detail.
 
 ## Clone the repository
 
-First off you'll need your own copy of the `pydata-sphinx-theme` codebase.
+First off you'll need your copy of the `pydata-sphinx-theme` codebase.
 You can clone it for local development like so:
 
-1. **Fork the repository** so you have your own copy on GitHub.
+1. **Fork the repository**, so you have your own copy on GitHub.
    See [the GitHub forking guide](https://docs.github.com/en/get-started/quickstart/fork-a-repo) for more information.
 2. **Clone the repository locally** so that you have a local copy to work from:
 
@@ -36,14 +36,14 @@ You can clone it for local development like so:
 
 ## Install your tools
 
-Building a Sphinx site uses a combination of Python and Jinja to manage HTML, SCSS, and Javascript.
+Building a Sphinx site uses a combination of Python and Jinja to manage HTML, SCSS, and JavaScript.
 To simplify this process, we use a few helper tools:
 
-- [The Sphinx Theme Builder](https://sphinx-theme-builder.readthedocs.io/en/latest/) to automatically perform compilation of web assets.
+- [The Sphinx Theme Builder](https://sphinx-theme-builder.readthedocs.io/en/latest/) compiles web assets in an automated way.
 - [pre-commit](https://pre-commit.com/) for automatically enforcing code standards and quality checks before commits.
-- [nox](https://nox.thea.codes/), for automating common development tasks.
+- [nox](https://nox.thea.codes/) for automating common development tasks.
 
-In particular, `nox` can be used to automatically create isolated local development environments with all of the correct packages installed to work on the theme.
+In particular, `nox` can be used to automatically create isolated local development environments with all the correct packages installed to work on the theme.
 The rest of this guide focuses on using `nox` to start with a basic environment.
 
 ```{seealso}
@@ -58,9 +58,9 @@ To start, install `nox`:
 $ pip install nox
 ```
 
-You can call `nox` from the command line in order to perform common actions that are needed in building the theme.
+You can call `nox` from the command line to perform common actions that are needed in building the theme.
 `nox` operates with isolated environments, so each action has its own packages installed in a local directory (`.nox`).
-For common development actions, you'll simply need to use `nox` and won't need to set up any other packages.
+For common development actions, you'll only need to use `nox` and won't need to set up any other packages.
 
 ### Setup `pre-commit`
 
@@ -116,7 +116,7 @@ Now that you've built the documentation, edit one of the source files to see how
 1. **Make an edit to a page**. For example, add a word or fix a typo on any page.
 2. **Rebuild the documentation** with `nox -s docs`
 
-It should go much faster this time, because `nox` is re-using the old environment, and because Sphinx has cached the pages that you didn't change.
+It should go much faster this time because `nox` is re-using the previously created environment, and because Sphinx has cached the pages that you didn't change.
 
 ## Compile the CSS/JS assets
 
@@ -138,15 +138,15 @@ The `sphinx-theme-builder` will bundle these assets automatically when we make a
 
 ## Run a development server
 
-You can combine the above two actions and run a development server so that changes to `src/` are automatically bundled with the package, and the documentation is immediately reloaded in a live preview window.
+You can combine the above two actions (build the docs and compile JS/CSS assets) and run a development server so that changes to `src/` are automatically bundled with the package, and the documentation is immediately reloaded in a live preview window.
 
 To run the development server with `nox`, run the following command:
 
 ```console
 $ nox -s docs-live
 ```
 
-When working on the theme, saving changes to any of these directories:
+When working on the theme, making changes to any of these directories:
 
 - `src/js/index.js`
 - `src/scss/index.scss`
@@ -161,19 +161,33 @@ will cause the development server to do the following:
 
 ## Run the tests
 
-This theme uses `pytest` for its testing, with a lightweight fixture defined
-in the `test_build.py` script that makes it easy to run a Sphinx build using
-this theme and inspect the results.
+This theme uses `pytest` for its testing. There is a lightweight fixture defined
+in the `test_build.py` script that makes it straightforward to run a Sphinx build using
+this theme and inspect the results. There are also several automated accessibility checks in
+`test_a11y.py`.
 
-In addition, we use [pytest-regressions](https://pytest-regressions.readthedocs.io/en/latest/)
-to ensure that the HTML generated by the theme is what we'd expect. This module
+```{warning}
+Currently, the automated accessibility tests check the Kitchen Sink page only.
+We are working on extending coverage to the rest of the theme.
+```
+
+In addition, we use
+[pytest-regressions](https://pytest-regressions.readthedocs.io/en/latest/) to
+ensure that the HTML generated by the theme is what we'd expect. This module
 provides a `file_regression` fixture that will check the contents of an object
 against a reference file on disk. If the structure of the two differs, then the
 test will fail. If we _expect_ the structure to differ, then delete the file on
-disk and run the test. A new file will be created, and subsequent tests will pass.
+disk and run the test. A new file will be created, and subsequent tests will
+pass.
 
-To run the tests with `nox`, run the following command:
+To run the build tests with `nox`, run the following command:
 
 ```console
 $ nox -s test
 ```
+
+To run the accessibility checks:
+
+```console
+$ nox -s a11y
+```

docs/community/topics/accessibility.md

@@ -1,11 +1,18 @@
 # Accessibility checks
 
-The accessibility checking tools can find a number of common HTML patterns which
+```{note}
+April-2023: we are currently
+[re-evaluating how we do accessibility checks](https://github.com/pydata/pydata-sphinx-theme/issues/1168)
+and reporting, so this may change soon.
+```
+
+In general, accessibility-checking tools can find a limited number of common HTML patterns which
 assistive technology can't help users understand.
-We run a [Lighthouse](https://developers.google.com/web/tools/lighthouse) job in our CI/CD, which generates a "score" for all pages in our **Kitchen Sink** example documentation.
-The configuration for Lighthouse is in:
 
-- `.github/workflows/lighthouserc.json`
+## Accessibility checks as part of our development process
+
+We run a [Lighthouse](https://developers.google.com/web/tools/lighthouse) job in our CI/CD, which generates a "score" for all pages in our **Kitchen Sink** example documentation.
+The configuration for Lighthouse can be found in the `.github/workflows/lighthouserc.json` file.
 
-For more information about configuring lighthouse, see [the lighthouse documentation](https://github.com/GoogleChrome/lighthouse-ci/blob/main/docs/configuration.md).
+For more information about configuring Lighthouse, see [the Lighthouse documentation](https://github.com/GoogleChrome/lighthouse-ci/blob/main/docs/configuration.md).
 For more information about Accessibility in general, see [](../../user_guide/accessibility.rst).

noxfile.py

@@ -5,6 +5,7 @@
 
     nox -s docs -- -r
 """
+import os
 import shutil as sh
 import tempfile
 from pathlib import Path
@@ -87,7 +88,24 @@ def test(session: nox.Session) -> None:
     if _should_install(session):
         session.install("-e", ".[test]")
     session.run(*split("pybabel compile -d src/pydata_sphinx_theme/locale -D sphinx"))
-    session.run("pytest", *session.posargs)
+    session.run("pytest", "-m", "not a11y", *session.posargs)
+
+
+@nox.session()
+def a11y(session: nox.Session) -> None:
+    """Run the accessibility test suite only."""
+    if _should_install(session):
+        session.install("-e", ".[test, a11y]")
+        # Install the drivers that Playwright needs to control the browsers.
+        if not (os.environ.get("CI") or os.environ.get("GITPOD_WORKSPACE_ID")):
+            # Assuming that most local dev environments have the needed system dependencies
+            # for playwright to work
+            session.run("playwright", "install")
+    # Build the docs so we can run accessibility tests against them.
+    session.run("nox", "-s", "docs")
+    # The next step would be to open a server to the docs for Playwright, but
+    # that is done in the test file, along with the accessibility checks.
+    session.run("pytest", "-m", "a11y", *session.posargs)
 
 
 @nox.session(name="test-sphinx")

package.json

@@ -5,6 +5,7 @@
     "build": "webpack"
   },
   "devDependencies": {
+    "axe-core": "^4.6.3",
     "copy-webpack-plugin": "^11.0.0",
     "css-loader": "^3.4.2",
     "css-minimizer-webpack-plugin": "^4.2.2",