diff --git a/README.rst b/README.rst index a4057557..409f4b73 100644 --- a/README.rst +++ b/README.rst @@ -1,369 +1,66 @@ -About ------ - -This is a plugin to facilitate image comparison for -`Matplotlib `__ figures in pytest. - -For each figure to test, the reference image is subtracted from the -generated image, and the RMS of the residual is compared to a -user-specified tolerance. If the residual is too large, the test will -fail (this is implemented using helper functions from -``matplotlib.testing``). - -For more information on how to write tests to do this, see the **Using** -section below. - -Installing ----------- +``pytest-mpl`` +============== -This plugin is compatible with Python 3.6 and later, and -requires `pytest `__ and -`matplotlib `__ to be installed. +``pytest-mpl`` is a `pytest `__ plugin to facilitate image comparison for `Matplotlib `__ figures. -To install, you can do:: +For each figure to test, an image is generated and then subtracted from an existing reference image. +If the RMS of the residual is larger than a user-specified tolerance, the test will fail. +Alternatively, the generated image can be hashed and compared to an expected value. - pip install pytest-mpl +For more information, see the `pytest-mpl documentation `__. -You can check that the plugin is registered with pytest by doing:: +Installation +------------ +.. code-block:: bash - pytest --version + pip install pytest-mpl -which will show a list of plugins: +For detailed instructions, see the `installation guide `__ in the ``pytest-mpl`` docs. -:: - - This is pytest version 2.7.1, imported from ... - setuptools registered plugins: - pytest-mpl-0.1 at ... - -Using +Usage ----- +First, write test functions that create a figure. +These image comparison tests are decorated with ``@pytest.mark.mpl_image_compare`` and return the figure for testing: -With Baseline Images -^^^^^^^^^^^^^^^^^^^^ - -To use, you simply need to mark the function where you want to compare -images using ``@pytest.mark.mpl_image_compare``, and make sure that the -function returns a Matplotlib figure (or any figure object that has a -``savefig`` method): - -.. code:: python - - import pytest - import matplotlib.pyplot as plt - - @pytest.mark.mpl_image_compare - def test_succeeds(): - fig = plt.figure() - ax = fig.add_subplot(1,1,1) - ax.plot([1,2,3]) - return fig - -To generate the baseline images, run the tests with the -``--mpl-generate-path`` option with the name of the directory where the -generated images should be placed:: - - pytest --mpl-generate-path=baseline - -If the directory does not exist, it will be created. The directory will -be interpreted as being relative to where you are running ``pytest``. -Once you are happy with the generated images, you should move them to a -sub-directory called ``baseline`` relative to the test files (this name -is configurable, see below). You can also generate the baseline image -directly in the right directory. - -With a Hash Library -^^^^^^^^^^^^^^^^^^^ - -Instead of comparing to baseline images, you can instead compare against a JSON -library of SHA-256 hashes. This has the advantage of not having to check baseline -images into the repository with the tests, or download them from a remote -source. - -The hash library can be generated with -``--mpl-generate-hash-library=path_to_file.json``. The hash library to be used -can either be specified via the ``--mpl-hash-library=`` command line argument, -or via the ``hash_library=`` keyword argument to the -``@pytest.mark.mpl_image_compare`` decorator. - -When generating a hash library, the tests will also be run as usual against the -existing hash library specified by ``--mpl-hash-library`` or the keyword argument. -However, generating baseline images will always result in the tests being skipped. - -Hybrid Mode: Hashes and Images -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +.. code-block:: python -It is possible to configure both hashes and baseline images. In this scenario -only the hash comparison can determine the test result. If the hash comparison -fails, the test will fail, however a comparison to the baseline image will be -carried out so the actual difference can be seen. If the hash comparison passes, -the comparison to the baseline image is skipped (unless **results always** is -configured). + import matplotlib.pyplot as plt + import pytest -This is especially useful if the baseline images are external to the repository -containing the tests, and are accessed via HTTP. In this situation, if the hashes -match, the baseline images won't be retrieved, saving time and bandwidth. Also, it -allows the tests to be modified and the hashes updated to reflect the changes -without having to modify the external images. + @pytest.mark.mpl_image_compare + def test_plot(): + fig, ax = plt.subplots() + ax.plot([1, 2]) + return fig +Then, generate reference images by running the test suite with the ``--mpl-generate-path`` option: -Running Tests -^^^^^^^^^^^^^ +.. code-block:: bash -Once tests are written with baseline images, a hash library, or both to compare -against, the tests can be run with:: + pytest --mpl-generate-path=baseline - pytest --mpl +Then, run the test suite as usual, but pass ``--mpl`` to compare the returned figures to the reference images: -and the tests will pass if the images are the same. If you omit the -``--mpl`` option, the tests will run but will only check that the code -runs, without checking the output images. +.. code-block:: bash -If pytest-mpl is not installed, the image comparison tests will cause pytest -to show a warning, ``PytestReturnNotNoneWarning``. Installing pytest-mpl will -solve this issue. Alternativly, the image comparison tests can be deselected -by running pytest with ``-m "not mpl_image_compare"``. + pytest --mpl - -Generating a Test Summary -^^^^^^^^^^^^^^^^^^^^^^^^^ - -By specifying the ``--mpl-generate-summary=html`` CLI argument, a HTML summary -page will be generated showing the test result, log entry and generated result -image. When in the (default) image comparison mode, the baseline image, diff -image and RMS (if any), and tolerance of each test will also be shown. -When in the hash comparison mode, the baseline hash and result hash will -also be shown. When in hybrid mode, all of these are included. - -When generating a HTML summary, the ``--mpl-results-always`` option is -automatically applied (see section below). Therefore images for passing -tests will also be shown. +By also passing ``--mpl-generate-summary=html``, a summary of the image comparison results will be generated in HTML format: +---------------+---------------+---------------+ | |html all| | |html filter| | |html result| | +---------------+---------------+---------------+ -As well as ``html``, ``basic-html`` can be specified for an alternative HTML -summary which does not rely on JavaScript or external resources. A ``json`` -summary can also be saved. Multiple options can be specified comma-separated. - -Options -------- - -Tolerance -^^^^^^^^^ - -The RMS tolerance for the image comparison (which defaults to 2) can be -specified in the ``mpl_image_compare`` decorator with the ``tolerance`` -argument: - -.. code:: python - - @pytest.mark.mpl_image_compare(tolerance=20) - def test_image(): - ... - -Savefig options -^^^^^^^^^^^^^^^ - -You can pass keyword arguments to ``savefig`` by using -``savefig_kwargs`` in the ``mpl_image_compare`` decorator: - -.. code:: python - - @pytest.mark.mpl_image_compare(savefig_kwargs={'dpi':300}) - def test_image(): - ... - -Baseline images -^^^^^^^^^^^^^^^ - -The baseline directory (which defaults to ``baseline`` ) and the -filename of the plot (which defaults to the name of the test with a -``.png`` suffix) can be customized with the ``baseline_dir`` and -``filename`` arguments in the ``mpl_image_compare`` decorator: - -.. code:: python - - @pytest.mark.mpl_image_compare(baseline_dir='baseline_images', - filename='other_name.png') - def test_image(): - ... - -The baseline directory in the decorator above will be interpreted as -being relative to the test file. Note that the baseline directory can -also be a URL (which should start with ``http://`` or ``https://`` and -end in a slash). If you want to specify mirrors, set ``baseline_dir`` to -a comma-separated list of URLs (real commas in the URL should be encoded -as ``%2C``). - -Finally, you can also set a custom baseline directory globally when -running tests by running ``pytest`` with:: - - pytest --mpl --mpl-baseline-path=baseline_images - -This directory will be interpreted as being relative to where pytest -is run. However, if the ``--mpl-baseline-relative`` option is also -included, this directory will be interpreted as being relative to -the current test directory. -In addition, if both this option and the ``baseline_dir`` -option in the ``mpl_image_compare`` decorator are used, the one in the -decorator takes precedence. - -Results always -^^^^^^^^^^^^^^ - -By default, result images are only saved for tests that fail. -Passing ``--mpl-results-always`` to pytest will force result images -to be saved for all tests, even for tests that pass. - -When in **hybrid mode**, even if a test passes hash comparison, -a comparison to the baseline image will also be carried out, -with the baseline image and diff image (if image comparison fails) -saved for all tests. This secondary comparison will not affect -the success status of the test. - -This option is useful for always *comparing* the result images against -the baseline images, while only *assessing* the tests against the -hash library. -If you only update your baseline images after merging a PR, this -option means that the generated summary will always show how the -PR affects the baseline images, with the success status of each -test (based on the hash library) also shown in the generated -summary. This option is applied automatically when generating -a HTML summary. - -When the ``--mpl-results-always`` option is active, and some hash -comparison tests are performed, a hash library containing all the -result hashes will also be saved to the root of the results directory. -The filename will be extracted from ``--mpl-generate-hash-library``, -``--mpl-hash-library`` or ``hash_library=`` in that order. - -Base style -^^^^^^^^^^ - -By default, tests will be run using the Matplotlib 'classic' style -(ignoring any locally defined RC parameters). This can be overridden by -using the ``style`` argument: - -.. code:: python - - @pytest.mark.mpl_image_compare(style='fivethirtyeight') - def test_image(): - ... - -Package version dependencies -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Different versions of Matplotlib and FreeType may result in slightly -different images. When testing on multiple platforms or as part of a -pipeline, it is important to ensure that the versions of these -packages match the versions used to generate the images used for -comparison. It can be useful to pin versions of Matplotlib and FreeType -so as to avoid automatic updates that fail tests. - -Removing text -^^^^^^^^^^^^^ - -If you are running a test for which you are not interested in comparing -the text labels, you can use the ``remove_text`` argument to the -decorator: - -.. code:: python - - @pytest.mark.mpl_image_compare(remove_text=True) - def test_image(): - ... - -This will make the test insensitive to changes in e.g. the freetype -library. - -Supported formats and deterministic output ------------------------------------------- - -By default, pytest-mpl will save and compare figures in PNG format. However, -it is possible to set the format to use by setting e.g. ``savefig_kwargs={'format': 'pdf'}`` -in ``mpl_image_compare``. Supported formats are ``'eps'``, ``'pdf'``, ``'png'``, and ``'svg'``. -Note that Ghostscript is required to be installed for comparing PDF and EPS figures, while -Inkscape is required for SVG comparison. - -By default, Matplotlib does not produce deterministic output that will have a -consistent hash every time it is run, or over different Matplotlib versions. In -order to enforce that the output is deterministic, you can set the ``deterministic`` -keyword argument in ``mpl_image_compare``: - -.. code:: python - - @pytest.mark.mpl_image_compare(deterministic=True) - -This does a number of things such as e.g., setting the creation date in the -metadata to be constant, and avoids hard-coding the Matplotlib in the files. - -Test failure example --------------------- - -If the images produced by the tests are correct, then the test will -pass, but if they are not, the test will fail with a message similar to -the following:: - - E Exception: Error: Image files did not match. - E RMS Value: 142.2287807767823 - E Expected: - E /var/folders/zy/t1l3sx310d3d6p0kyxqzlrnr0000gr/T/tmp4h4oxr7y/baseline-coords_overlay_auto_coord_meta.png - E Actual: - E /var/folders/zy/t1l3sx310d3d6p0kyxqzlrnr0000gr/T/tmp4h4oxr7y/coords_overlay_auto_coord_meta.png - E Difference: - E /var/folders/zy/t1l3sx310d3d6p0kyxqzlrnr0000gr/T/tmp4h4oxr7y/coords_overlay_auto_coord_meta-failed-diff.png - E Tolerance: - E 10 - -The image paths included in the exception are then available for -inspection: - -+----------------+----------------+-------------+ -| Expected | Actual | Difference | -+================+================+=============+ -| |expected| | |actual| | |diff| | -+----------------+----------------+-------------+ - -In this case, the differences are very clear, while in some cases it may -be necessary to use the difference image, or blink the expected and -actual images, in order to see what changed. - -The default tolerance is 2, which is very strict. In some cases, you may -want to relax this to account for differences in fonts across different -systems. - -By default, the expected, actual and difference files are written to a -temporary directory with a non-deterministic path. If you want to instead -write them to a specific directory, you can use:: - - pytest --mpl --mpl-results-path=results - -The ``results`` directory will then contain one sub-directory per test, and each -sub-directory will contain the three files mentioned above. If you are using a -continuous integration service, you can then use the option to upload artifacts -to upload these results to somewhere where you can view them. For more -information, see: - -* `Uploading artifacts on Travis-CI `_ -* `Build Artifacts (CircleCI) `_ -* `Packaging Artifacts (AppVeyor) `_ - -Running the tests for pytest-mpl --------------------------------- - -If you are contributing some changes and want to run the tests, first -install the latest version of the plugin then do:: +For more information on how to configure and use ``pytest-mpl``, see the `pytest-mpl documentation `__. - cd tests - pytest --mpl +Contributing +------------ +``pytest-mpl`` is a community project maintained for and by its users. +There are many ways you can help! -The reason for having to install the plugin first is to ensure that the -plugin is correctly loaded as part of the test suite. +- Report a bug or request a feature `on GitHub `__ +- Improve the documentation or code -.. |html all| image:: images/html_all.png -.. |html filter| image:: images/html_filter.png -.. |html result| image:: images/html_result.png -.. |expected| image:: images/baseline-coords_overlay_auto_coord_meta.png -.. |actual| image:: images/coords_overlay_auto_coord_meta.png -.. |diff| image:: images/coords_overlay_auto_coord_meta-failed-diff.png +.. |html all| image:: docs/images/html_all.png +.. |html filter| image:: docs/images/html_filter.png +.. |html result| image:: docs/images/html_result.png diff --git a/docs/conf.py b/docs/conf.py index c9cd50d2..f372bbf8 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -38,6 +38,7 @@ # ones. extensions = [ 'sample_summaries', + 'sphinx.ext.intersphinx', 'sphinx_design', ] @@ -49,6 +50,9 @@ # This pattern also affects html_static_path and html_extra_path. exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store'] +intersphinx_mapping = { + "matplotlib": ("https://matplotlib.org/stable", None), +} # -- Options for HTML output ------------------------------------------------- diff --git a/docs/configuration.rst b/docs/configuration.rst index 7bc41a13..9e6914fb 100644 --- a/docs/configuration.rst +++ b/docs/configuration.rst @@ -4,184 +4,424 @@ Configuration ############# -Tolerance -^^^^^^^^^ +This section defines all the ``pytest-mpl`` configuration options. -The RMS tolerance for the image comparison (which defaults to 2) can be -specified in the ``mpl_image_compare`` decorator with the ``tolerance`` -argument: +There are three ways to configure the plugin: + +1. Passing kwargs to the ``pytest.mark.mpl_image_compare`` decorator (kwarg) +2. Passing options to pytest from the command line (CLI) +3. Setting INI options in pytest configuration files (INI) + +The CLI and INI options are global, and will apply to all tests. +The kwarg options are local, and will only apply to the test function that they are specified in. + +If set, the kwarg options will override the equivalent CLI and INI options. +Furthermore, CLI options will override equivalent INI options. + +See the `pytest documentation `__ for more information on how to set INI options. + +Enabling the plugin +=================== + +Enable testing +-------------- +| **kwarg**: --- +| **CLI**: ``--mpl`` +| **INI**: --- +| Default: ``False`` + +To enable image comparison testing, pass ``--mpl`` when running pytest. + +.. code:: bash + + pytest --mpl + +By default, this option will enable :doc:`baseline image comparison `. +:doc:`Baseline hash comparison ` can be enabled by configuring the :ref:`hash library configuration option `. + +Without this option, the tests will still run. +However, the returned figures will be closed without being compared to a baseline image or hash. + +Enable baseline image generation +-------------------------------- +| **kwarg**: --- +| **CLI**: ``--mpl-generate-path=`` +| **INI**: --- +| Default: ``None`` + +Baseline images will be generated and saved to the specified directory path, relative to where pytest was run. + +.. code:: bash + + pytest --mpl-generate-path=baseline + +The baseline directory specified by the :ref:`baseline directory configuration option ` will be ignored. +However, the filename of the baseline image will still be determined by the :ref:`filename configuration option ` and the :ref:`use full test name configuration option `. +This option overrides the ``--mpl`` option. + +Enable baseline hash generation +------------------------------- +| **kwarg**: --- +| **CLI**: ``--mpl-generate-hash-library=`` +| **INI**: --- +| Default: ``None`` + +Baseline hashes will be generated and saved to the specified JSON file path, relative to where pytest was run. + +.. code:: bash + + pytest --mpl-generate-hash-library=hashes.json + +Enabling this option will also set the ``--mpl`` option, as it is important to visually inspect the figures before generating baseline hashes. +The hash library specified by the :ref:`hash library configuration option ` will be ignored. + +Locating baseline images +======================== + +.. _baseline-dir: + +Directory containing baseline images +------------------------------------ +| **kwarg**: ``baseline_dir=`` +| **CLI**: ``--mpl-baseline-path=`` +| **INI**: --- +| Default: ``baseline/`` *(relative to the test file)* + +The directory containing the baseline images that will be compared to the test figures. +The kwarg option (``baseline_dir``) is relative to the test file, while the CLI option (``--mpl-baseline-path``) is relative to where pytest was run. +Absolute paths can also be used. +If the directory does not exist, it will be created along with any missing parent directories. + +.. code:: bash + + pytest --mpl --mpl-baseline-path=baseline_images + +The baseline directory can also be a URL, which should start with ``http://`` or ``https://`` and end in a slash. +Alternative URLs, or mirrors, can be configured by specifying a comma-separated list of URLs. +Baseline images will be searched for in the order that the URLs are specified, and the first successful download will be used. +Real commas in URLs should be encoded as ``%2C``. + +.. code:: bash + + pytest --mpl --mpl-baseline-path=https://example.com/baseline/,https://mirror.example.com/baseline/ .. code:: python - @pytest.mark.mpl_image_compare(tolerance=20) - def test_image(): + @pytest.mark.mpl_image_compare(baseline_dir="https://example.com/baseline/", + filename="other_name.png") + def test_plot(): ... -Savefig options -^^^^^^^^^^^^^^^ +Whether ``--mpl-baseline-path`` should also be relative to the test file +------------------------------------------------------------------------ +| **kwarg**: --- +| **CLI**: ``--mpl-baseline-relative`` +| **INI**: --- +| Default: ``False`` + +If this option is set, the baseline directory specified by ``--mpl-baseline-path`` will be interpreted as being relative to the test file. +This option is only relevant if ``--mpl-baseline-path`` refers to a directory and not a URL. + +.. code:: bash + + pytest --mpl --mpl-baseline-path=baseline_images --mpl-baseline-relative + +.. _filename: -You can pass keyword arguments to ``savefig`` by using -``savefig_kwargs`` in the ``mpl_image_compare`` decorator: +Filename of the baseline image +------------------------------ +| **kwarg**: ``filename=`` +| **CLI**: --- +| **INI**: --- +| Default: *name of the test with a file extension suffix* + +The filename of the baseline image that will be compared to the test figure. +The default file extension is ``png``, unless overridden by :ref:`savefig_kwargs["format"] `. +This option has no effect if the :ref:`use full test name configuration option ` is enabled. .. code:: python - @pytest.mark.mpl_image_compare(savefig_kwargs={'dpi':300}) - def test_image(): + @pytest.mark.mpl_image_compare(baseline_dir="baseline_images", + filename="other_name.png") + def test_plot(): ... -Baseline images -^^^^^^^^^^^^^^^ - -The baseline directory (which defaults to ``baseline`` ) and the -filename of the plot (which defaults to the name of the test with a -``.png`` suffix) can be customized with the ``baseline_dir`` and -``filename`` arguments in the ``mpl_image_compare`` decorator: +If you specify a filename that has an extension other than ``png``, you must also specify it in :ref:`savefig_kwargs["format"] `. .. code:: python - @pytest.mark.mpl_image_compare(baseline_dir='baseline_images', - filename='other_name.png') - def test_image(): + @pytest.mark.mpl_image_compare(filename="plot.pdf", + savefig_kwargs={"format": "pdf"}) + def test_plot(): ... -The baseline directory in the decorator above will be interpreted as -being relative to the test file. Note that the baseline directory can -also be a URL (which should start with ``http://`` or ``https://`` and -end in a slash). If you want to specify mirrors, set ``baseline_dir`` to -a comma-separated list of URLs (real commas in the URL should be encoded -as ``%2C``). +.. _full-test-name: -Finally, you can also set a custom baseline directory globally when -running tests by running ``pytest`` with:: +Whether to include the module name in the filename +-------------------------------------------------- +| **kwarg**: --- +| **CLI**: --- +| **INI**: ``mpl-use-full-test-name`` +| Default: ``False`` - pytest --mpl --mpl-baseline-path=baseline_images +Whether to include the module name (and class name) in the baseline image filename. -This directory will be interpreted as being relative to where pytest -is run. However, if the ``--mpl-baseline-relative`` option is also -included, this directory will be interpreted as being relative to -the current test directory. -In addition, if both this option and the ``baseline_dir`` -option in the ``mpl_image_compare`` decorator are used, the one in the -decorator takes precedence. +This option is useful if you have multiple tests with the same name in different modules. +Or have multiple tests with the same name in the same module, but in different classes. +If this option is set, the baseline image filename will be ``[.]..``. +The file extension is the default extension as documented in the :ref:`filename option documentation `. -Results always -^^^^^^^^^^^^^^ +Enabling this should ensure baseline image filenames are unique. +The :ref:`filename configuration option ` can also be used to fix the filename of the baseline image. -By default, result images are only saved for tests that fail. -Passing ``--mpl-results-always`` to pytest will force result images -to be saved for all tests, even for tests that pass. - -When in **hybrid mode**, even if a test passes hash comparison, -a comparison to the baseline image will also be carried out, -with the baseline image and diff image (if image comparison fails) -saved for all tests. This secondary comparison will not affect -the success status of the test. - -This option is useful for always *comparing* the result images against -the baseline images, while only *assessing* the tests against the -hash library. -If you only update your baseline images after merging a PR, this -option means that the generated summary will always show how the -PR affects the baseline images, with the success status of each -test (based on the hash library) also shown in the generated -summary. This option is applied automatically when generating -a HTML summary. - -When the ``--mpl-results-always`` option is active, and some hash -comparison tests are performed, a hash library containing all the -result hashes will also be saved to the root of the results directory. -The filename will be extracted from ``--mpl-generate-hash-library``, -``--mpl-hash-library`` or ``hash_library=`` in that order. - -Base style -^^^^^^^^^^ - -By default, tests will be run using the Matplotlib 'classic' style -(ignoring any locally defined RC parameters). This can be overridden by -using the ``style`` argument: +.. note:: + + Filename collisions are permitted. + This is useful if, for example, you want to verify that two tests produce the same figure. + However, unexpected collisions should become apparent when the tests are run and failures are reported. + +This option overrides the :ref:`filename configuration option `. + +Locating baseline hashes +======================== + +.. _hash-library: + +File containing baseline hashes +------------------------------- +| **kwarg**: ``hash_library=`` +| **CLI**: ``--mpl-hash-library=`` +| **INI**: --- +| Default: *no hash comparison* + +The file containing the baseline hashes that will be compared to the test figures. +Both the kwarg option (``hash_library``) and the CLI option (``--mpl-hash-library``) are relative to the test file. +In this case, the CLI option takes precedence over the kwarg option. +The file must be a JSON file in the same format as one generated by ``--mpl-generate-hash-library``. +If its directory does not exist, it will be created along with any missing parent directories. + +Configuring this option disables baseline image comparison. +If you want to enable both hash and baseline image comparison, which we call :doc:`"hybrid mode" `, you must explicitly set the :ref:`baseline directory configuration option `. + +.. _controlling-sensitivity: + +Controlling the sensitivity of the comparison +============================================= + +.. rubric:: Package version dependencies + +Different versions of Matplotlib and FreeType may result in slightly different images. +When testing on multiple platforms or as part of a pipeline, it is important to ensure that the versions of these packages match the versions used to generate the images and/or hashes used for comparison. +It can be useful to pin versions of Matplotlib and FreeType so as to avoid automatic updates that fail tests. + +The ``pytest-mpl`` configuration options in this section allow you to control the sensitivity of the comparison. +Adjusting these options *may* allow tests to pass across a range of Matplotlib and FreeType versions. + +.. _tolerance: + +RMS tolerance +------------- +| **kwarg**: ``tolerance=`` +| **CLI**: --- +| **INI**: --- +| Default: ``2`` + +The maximum RMS difference between the result image and the baseline image before the test fails. +The specified tolerance value can be a float or an integer between 0 and 255. .. code:: python - @pytest.mark.mpl_image_compare(style='fivethirtyeight') - def test_image(): + @pytest.mark.mpl_image_compare(tolerance=20) + def test_plot(): ... -Package version dependencies -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Different versions of Matplotlib and FreeType may result in slightly -different images. When testing on multiple platforms or as part of a -pipeline, it is important to ensure that the versions of these -packages match the versions used to generate the images used for -comparison. It can be useful to pin versions of Matplotlib and FreeType -so as to avoid automatic updates that fail tests. +.. rubric:: How the RMS difference is calculated + +Result images and baseline images are *always* converted to PNG files before comparison. +Each are read as an array of RGBA pixels (or just RGB if fully opaque) with values between 0 and 255. +If the result image and the baseline image have different aspect ratios, the test will always fail. +The RMS difference is calculated as the square root of the mean of the squared differences between the result image and the baseline image. +If the RMS difference is greater than the tolerance, the test will fail. -Removing text -^^^^^^^^^^^^^ +Whether to make metadata deterministic +-------------------------------------- +| **kwarg**: ``deterministic=`` +| **CLI**: --- +| **INI**: --- +| Default: ``True`` (PNG: ``False``) -If you are running a test for which you are not interested in comparing -the text labels, you can use the ``remove_text`` argument to the -decorator: +Whether to make the image file metadata deterministic. + +By default, Matplotlib does not produce deterministic output that will have a consistent hash every time it is run, or over different Matplotlib versions. +Depending on the file format, enabling this option does a number of things such as, e.g., setting the creation date in the metadata to be constant, and avoids hard-coding the Matplotlib version in the file. +Supported formats for deterministic metadata are ``"eps"``, ``"pdf"``, ``"png"``, and ``"svg"``. + +.. code:: python + + @pytest.mark.mpl_image_compare(deterministic=True) + def test_plot(): + ... + +By default, ``pytest-mpl`` will save and compare figures in PNG format. +However, it is possible to set the format to use by setting, e.g., ``savefig_kwargs={"format": "pdf"}`` when configuring the :ref:`savefig_kwargs configuration option `. +Note that Ghostscript is required to be installed for comparing PDF and EPS figures, while Inkscape is required for SVG comparison. + +Whether to remove titles and axis tick labels +--------------------------------------------- +| **kwargs**: ``remove_text=`` +| **CLI**: --- +| **INI**: --- +| Default: ``False`` + +Enabling this option will remove titles and axis tick labels from the figure before saving and comparing. +This will make the test less sensitive to changes in the FreeType library version. +This feature, provided by :func:`matplotlib.testing.decorators.remove_ticks_and_titles`, will not remove any other text such as axis labels and annotations. .. code:: python @pytest.mark.mpl_image_compare(remove_text=True) - def test_image(): + def test_plot(): ... -This will make the test insensitive to changes in e.g. the freetype -library. - -Test failure example --------------------- - -If the images produced by the tests are correct, then the test will -pass, but if they are not, the test will fail with a message similar to -the following:: - - E Exception: Error: Image files did not match. - E RMS Value: 142.2287807767823 - E Expected: - E /var/folders/zy/t1l3sx310d3d6p0kyxqzlrnr0000gr/T/tmp4h4oxr7y/baseline-coords_overlay_auto_coord_meta.png - E Actual: - E /var/folders/zy/t1l3sx310d3d6p0kyxqzlrnr0000gr/T/tmp4h4oxr7y/coords_overlay_auto_coord_meta.png - E Difference: - E /var/folders/zy/t1l3sx310d3d6p0kyxqzlrnr0000gr/T/tmp4h4oxr7y/coords_overlay_auto_coord_meta-failed-diff.png - E Tolerance: - E 10 - -The image paths included in the exception are then available for -inspection: - -+----------------+----------------+-------------+ -| Expected | Actual | Difference | -+================+================+=============+ -| |expected| | |actual| | |diff| | -+----------------+----------------+-------------+ - -In this case, the differences are very clear, while in some cases it may -be necessary to use the difference image, or blink the expected and -actual images, in order to see what changed. - -The default tolerance is 2, which is very strict. In some cases, you may -want to relax this to account for differences in fonts across different -systems. - -By default, the expected, actual and difference files are written to a -temporary directory with a non-deterministic path. If you want to instead -write them to a specific directory, you can use:: - - pytest --mpl --mpl-results-path=results - -The ``results`` directory will then contain one sub-directory per test, and each -sub-directory will contain the three files mentioned above. If you are using a -continuous integration service, you can then use the option to upload artifacts -to upload these results to somewhere where you can view them. For more -information, see: - -* `Uploading artifacts on Travis-CI `_ -* `Build Artifacts (CircleCI) `_ -* `Packaging Artifacts (AppVeyor) `_ - -.. |expected| image:: images/baseline-coords_overlay_auto_coord_meta.png -.. |actual| image:: images/coords_overlay_auto_coord_meta.png -.. |diff| image:: images/coords_overlay_auto_coord_meta-failed-diff.png +Modifying the figure before saving +================================== + +.. _savefig-kwargs: + +Matplotlib savefig kwargs +------------------------- +| **kwarg**: ``savefig_kwargs=`` +| **CLI**: --- +| **INI**: --- +| Default: ``{}`` + +A dictionary of keyword arguments to pass to :func:`matplotlib.pyplot.savefig`. + +.. code:: python + + @pytest.mark.mpl_image_compare(savefig_kwargs={"dpi": 300}) + def test_plot(): + ... + +Matplotlib style +---------------- +| **kwarg**: ``style=`` +| **CLI**: --- +| **INI**: --- +| Default: ``"classic"`` + +The Matplotlib style to use when saving the figure. +See the :func:`matplotlib.style.context` ``style`` documentation for the options available. +``pytest-mpl`` will ignore any locally defined :class:`~matplotlib.RcParams`. + +.. code:: python + + @pytest.mark.mpl_image_compare(style="fivethirtyeight") + def test_plot(): + ... + +.. note:: + + It is recommended to use the ``"default"`` style for new code. + + .. code:: python + + @pytest.mark.mpl_image_compare(style="default") + def test_plot(): + ... + + The ``"classic"`` style (which ``pytest-mpl`` currently uses by default) was the default style for Matplotlib versions prior to 2.0. + A future major release of ``pytest-mpl`` *may* change the default style to ``"default"``. + +Matplotlib backend +------------------ +| **kwarg**: ``backend=`` +| **CLI**: --- +| **INI**: --- +| Default: ``"agg"`` + +The Matplotlib backend to use when saving the figure. +See the :ref:`Matplotlib backend documentation ` for the options available. +``pytest-mpl`` will ignore any locally defined :class:`~matplotlib.RcParams`. + +Recording test results +====================== + +.. _results-path: + +Directory to write testing artifacts to +--------------------------------------- +| **kwarg**: --- +| **CLI**: ``--mpl-results-path=`` +| **INI**: ``mpl-results-path = `` +| Default: *temporary directory* + +The directory to write result images and test summary reports to. +The path is relative to where pytest was run. +Absolute paths are also supported. +If the directory does not exist, it will be created along with any missing parent directories. + +.. _results-always: + +Whether to save result images for passing tests +----------------------------------------------- +| **kwarg**: --- +| **CLI**: ``--mpl-results-always`` +| **INI**: ``mpl-results-always = `` +| Default: ``False`` (``True`` if generating a HTML summary) + +By default, result images are only saved for tests that fail. +Enabling this option will force result images to be saved for all tests, even for tests that pass. + +When this option is enabled, and some hash comparison tests are performed, a hash library containing all the result hashes will also be saved to the root of the results directory. +The filename will be extracted from ``--mpl-generate-hash-library``, ``--mpl-hash-library``, or ``hash_library=`` in that order. + +This option is applied automatically when generating a HTML summary. + +.. rubric:: Relevance to "hybrid mode" + +When in :doc:`"hybrid mode" `, a baseline image comparison is only performed if the test fails hash comparison. +However, enabling this option will force a comparison to the baseline image even if the test passes hash comparison. +This option is useful for always *comparing* the result images against the baseline images, while only *assessing* the tests against the hash library. +This secondary comparison will **not** affect the success status of the test, but any failures (including diff images) will be included in generated summary reports. + +Some projects store their baseline images in a separate repository, and only keep the baseline hash library in the main repository. +This means that they cannot update the baseline images until after the PR is merged. +Enabling this option allows them to ensure the hashes are correct before merging the PR, but also see how the PR affects the baseline images, as the diff images will always be shown in the HTML summary. + +.. _generate-summary: + +Generate test summaries +----------------------- +| **kwarg**: --- +| **CLI**: ``--mpl-generate-summary={html,json,basic-html}`` +| **INI**: --- +| Default: ``None`` + +This option specifies the format of the test summary report to generate, if any. +Multiple options can be specified comma-separated. +The available options are: + +``html`` + Generate a HTML summary report showing the test result, log entry and generated result image. + Results can be searched and filtered. + When in the (default) image comparison mode, the baseline image, diff image and RMS difference (if any), and RMS tolerance of each test will also be shown. + When in the hash comparison mode, the baseline hash and result hash will also be shown. + When in hybrid mode, all of these are included. +``json`` + Generate a JSON summary report. + This format includes the same information as the HTML summary, but is more suitable for automated processing. +``basic-html`` + Generate a HTML summary report with a simplified layout. + This format does not include any JavaScript or need internet access to load web resources. + +Summary reports can also be produced when generating baseline images and hash libraries. +The summaries will be written to the :ref:`results directory `. +When generating a HTML summary, the ``--mpl-results-always`` option is automatically applied. +Therefore images for passing tests will also be shown. + +For examples of how the summary reports look in different operating modes, see: + +* :doc:`image_mode` +* :doc:`hash_mode` +* :doc:`hybrid_mode` diff --git a/docs/hash_mode.rst b/docs/hash_mode.rst new file mode 100644 index 00000000..13fa7de8 --- /dev/null +++ b/docs/hash_mode.rst @@ -0,0 +1,83 @@ +.. title:: Hash comparison mode + +##################### +Hash Comparison Mode +##################### + +This how-to guide will show you how to use the hash comparison mode of ``pytest-mpl``. + +In this mode, the hash of the image is compared to the hash of the baseline image. +Only the hash value of the baseline image, rather than the full image, needs to be stored in the repository. +This means that the repository size is reduced, and the images can be regenerated if necessary. +This approach does however make it more difficult to visually inspect any changes to the images. + +If your goal is to not commit any images to the code repository, then you should consider using :doc:`hybrid mode ` instead. +In this mode, the hashes can be stored in the code repository, while the baseline images are stored in a separate repository and accessed through a URL when testing. + +Generating baseline hashes +========================== + +Once a suite of image comparison tests have been written, baseline hashes should be generated by setting ``--mpl-generate-hash-library``: + +.. code-block:: bash + + pytest --mpl-generate-hash-library=your_project/tests/hashes.json + +It is important to visually inspect the figures before generating baseline hashes. +So, as well as generating baseline hashes, this command runs baseline image comparison tests. +If no baseline images exist in the default directory, this command will fail. + +A better option is to generate baseline images along with the baseline hashes to ensure that the images are as expected, even if you do not wish to use them for comparison: + +.. code-block:: bash + + pytest \ + --mpl-generate-hash-library=your_project/tests/mpl35_ft261.json \ + --mpl-generate-path=baseline + +To assist with inspecting the generated images (and hashes), a HTML summary report can be generated by setting ``--mpl-generate-summary``: + +.. code-block:: bash + + pytest \ + --mpl-generate-hash-library=test_hashes.json \ + --mpl-generate-path=baseline \ + --mpl-results-path=results \ + --mpl-generate-summary=html,json + +:summary:`test_html_generate` + +You should choose a directory within you repository to store the baseline hashes. +It's usually a good idea to encode the Matplotlib version and the FreeType version in the filename, e.g. ``mpl35_ft261.json``. +The hash library file should then be committed to the repository. + +Running hash comparison tests +============================= + +When running the tests, the ``--mpl`` flag should be used along with a :ref:`configured hash library path ` to enable baseline hash comparison testing: + +.. code-block:: bash + + pytest --mpl \ + --mpl-hash-library=your_project/tests/mpl35_ft261.json + +Optionally, a HTML summary report can be generated by setting ``--mpl-generate-summary``: + +.. code-block:: bash + + pytest --mpl \ + --mpl-hash-library=your_project/tests/mpl35_ft261.json \ + --mpl-results-path=results \ + --mpl-generate-summary=html,json + +:summary:`test_html_hashes_only` + +The ``--mpl-results-path`` flag can be used to set the directory where the generated HTML summary will be stored. +If this is not set, the images will be stored in a temporary directory. + +Continue reading +================ + +``pytest-mpl`` has many configuration options that can be used to customize the behavior of the hash comparison mode. +Only a few of the most commonly used options are covered in this guide. +See the :doc:`configuration options documentation ` for full details. diff --git a/docs/hybrid_mode.rst b/docs/hybrid_mode.rst new file mode 100644 index 00000000..f5b1ebfb --- /dev/null +++ b/docs/hybrid_mode.rst @@ -0,0 +1,90 @@ +.. title:: Hybrid mode + +############################## +Hybrid Mode: Hashes and Images +############################## + +This how-to guide will show you how to use the hybrid mode of ``pytest-mpl``. + +For a full description of the hybrid mode, see the :ref:`hybrid mode section of the get started guide `. +In summary, hybrid mode uses both baseline images and hashes. +First, the hash of the image is compared to the hash of the baseline image. +If the hashes match, the test passes. +If the hashes do not match, the test fails. + +The difference with hybrid mode is that a baseline image comparison will also be carried out if the hashes do not match, or always :ref:`if this has been configured `. +The purpose of the additional image comparison (which does not affect the test result) is to allow the user to visually inspect the difference between the baseline image and the image generated by the test. + +In order to keep the code repository size small, it is recommended to store the baseline hashes in the code repository, and the baseline images in a separate repository. +The baseline hashes should be updated where appropriate in PRs to the code repository. +However, the baseline images are not updated in these PRs. +Instead, they should be updated once the PR has been merged, preferably by a CI job. + +Another benefit of only updating the baseline images once the PR has been merged is that the PR tests will show the difference between the remote baseline images and the images generated by the PR. +Even though the tests will pass when the baseline hash matches, the images will still be compared and the difference will be shown in the HTML test summary report, which is useful when reviewing the PR. + +Generating baseline hashes and images +===================================== + +Once a suite of image comparison tests have been written, baseline hashes and images should be generated by setting ``--mpl-generate-path`` and ``--mpl-generate-hash-library``: + +.. code-block:: bash + + pytest \ + --mpl-generate-hash-library=your_project/tests/test_hashes.json \ + --mpl-generate-path=baseline \ + --mpl-results-path=results \ + --mpl-generate-summary=html,json + +:summary:`test_html_generate` + +Open the HTML summary file and inspect the figures to ensure that the baseline images are correct. +If they are, the baseline hashes can be committed to the code repository. +It's usually a good idea to encode the Matplotlib version and the FreeType version in the filename, e.g. ``mpl35_ft261.json``. +The baseline images should be copied to a separate repository; preferably within a version specific directory, e.g. ``mpl35_ft261/``. + +Running hash comparison tests +============================= + +When running the tests, the ``--mpl`` flag should be used along with a configured :ref:`hash library path ` and :ref:`baseline image path ` to enable hybrid mode testing: + +.. code-block:: bash + + pytest --mpl \ + --mpl-hash-library=your_project/tests/mpl35_ft261.json \ + --mpl-baseline-path=https://raw.githubusercontent.com/your-org/your-project-figure-tests/mpl35_ft261/ \ + --mpl-results-path=results \ + --mpl-generate-summary=html,json + +:summary:`test_html` + +The ``--mpl-results-path`` flag can be used to set the directory where the generated HTML summary will be stored. +If this is not set, the images will be stored in a temporary directory. + +Notice that the baseline image path is set to a URL, which is the location of the baseline images in the separate repository. +When the baseline image comparison is carried out, the baseline images will be downloaded from this URL. + +It is recommended to create a CI job that updates the baseline images in the separate repository once the PR has been merged. +The CI job should run when new commits are pushed to the default branch. +The baseline images should only be regenerated and updated if the tests pass in the hash comparison mode. + +.. rubric:: Aside: basic HTML summary + +This is what the basic HTML summary looks like for the same test as above: + +.. code-block:: bash + + pytest --mpl \ + --mpl-hash-library=your_project/tests/mpl35_ft261.json \ + --mpl-baseline-path=https://raw.githubusercontent.com/your-org/your-project-figure-tests/mpl35_ft261/ \ + --mpl-results-path=results \ + --mpl-generate-summary=basic-html,json + +:summary:`test_basic_html` + +Continue reading +================ + +``pytest-mpl`` has many configuration options that can be used to customize the behavior of the hybrid mode. +Only a few of the most commonly used options are covered in this guide. +See the :doc:`configuration options documentation ` for full details. diff --git a/docs/image_mode.rst b/docs/image_mode.rst new file mode 100644 index 00000000..13a907f5 --- /dev/null +++ b/docs/image_mode.rst @@ -0,0 +1,70 @@ +.. title:: Image comparison mode + +##################### +Image Comparison Mode +##################### + +This how-to guide will show you how to use the image comparison mode of ``pytest-mpl``. +This is the default mode when ``pytest-mpl`` is invoked with ``--mpl``. + +In this mode, ``pytest-mpl`` will compare the images generated by the test with a baseline image. +If the images are different, :ref:`to a specified RMS tolerance `, the test will fail. +The test will also fail if the baseline image is not found or if the test does not generate a figure. +If the figure has a different aspect ratio to the baseline image, the test will also fail. + +Generating baseline images +========================== + +Once a suite of image comparison tests have been written, baseline images should be generated by setting ``--mpl-generate-path``: + +.. code-block:: bash + + pytest --mpl-generate-path=your_project/tests/baseline + +You should choose a directory within you repository to store the baseline images. +The generated images should be checked to ensure they are as expected. +The images should then be committed to the repository. + +To assist with inspecting the generated images, a HTML summary report can be generated by setting ``--mpl-generate-summary``: + +.. code-block:: bash + + pytest \ + --mpl-generate-path=your_project/tests/baseline \ + --mpl-generate-summary=html,json + +:summary:`test_html_generate_images_only` + +Running image comparison tests +============================== + +When running the tests, the ``--mpl`` flag should be used to enable baseline image comparison testing: + +.. code-block:: bash + + pytest --mpl \ + --mpl-baseline-path=your_project/tests/baseline + +The :ref:`baseline path ` should be set to the directory containing the baseline images. +If the baseline images are not found, the test will fail. + +Optionally, a HTML summary report can be generated by setting ``--mpl-generate-summary``: + +.. code-block:: bash + + pytest --mpl \ + --mpl-baseline-path=your_project/tests/baseline \ + --mpl-results-path=results \ + --mpl-generate-summary=html,json + +:summary:`test_html_images_only` + +The ``--mpl-results-path`` flag can be used to set the directory where the generated HTML summary will be stored. +If this is not set, the images will be stored in a temporary directory. + +Continue reading +================ + +``pytest-mpl`` has many configuration options that can be used to customize the behavior of the image comparison mode. +Only a few of the most commonly used options are covered in this guide. +See the :doc:`configuration options documentation ` for full details. diff --git a/docs/index.rst b/docs/index.rst index c55ab66f..f48b5448 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -7,14 +7,20 @@ installing usage + image_mode + hash_mode + hybrid_mode configuration - summaries ################################## pytest-mpl |release| documentation ################################## -This is a plugin to facilitate image comparison for Matplotlib figures in pytest. +``pytest-mpl`` is a `pytest `__ plugin to facilitate image comparison for `Matplotlib `__ figures. + +For each figure to test, an image is generated and then subtracted from an existing reference image. +If the RMS of the residual is larger than a user-specified tolerance, the test will fail. +Alternatively, the generated image can be hashed and compared to an expected value. ************ Installation @@ -53,7 +59,7 @@ Learning resources Tutorials ^^^ - - :doc:`Basic usage ` + - :doc:`Get started ` .. grid-item-card:: :padding: 2 @@ -61,6 +67,9 @@ Learning resources How-tos ^^^ + - :doc:`Image comparison mode ` + - :doc:`Hash comparison mode ` + - :doc:`Hybrid mode ` .. grid-item-card:: :padding: 2 @@ -68,6 +77,7 @@ Learning resources Understand how pytest-mpl works ^^^ + Explanatory information is included where relevant throughout the documentation. .. grid-item-card:: :padding: 2 @@ -76,14 +86,12 @@ Learning resources ^^^ - :doc:`Configuration ` - - :doc:`Summary Reports ` - ************ Contributing ************ -pytest-mpl is a community project maintained for and by its users. +``pytest-mpl`` is a community project maintained for and by its users. There are many ways you can help! - Report a bug or request a feature `on GitHub `__ diff --git a/docs/summaries.rst b/docs/summaries.rst deleted file mode 100644 index add3e6c9..00000000 --- a/docs/summaries.rst +++ /dev/null @@ -1,95 +0,0 @@ -.. title:: Summary Reports - -############### -Summary Reports -############### - -Generating a Test Summary -^^^^^^^^^^^^^^^^^^^^^^^^^ - -By specifying the ``--mpl-generate-summary=html`` CLI argument, a HTML summary -page will be generated showing the test result, log entry and generated result -image. When in the (default) image comparison mode, the baseline image, diff -image and RMS (if any), and tolerance of each test will also be shown. -When in the hash comparison mode, the baseline hash and result hash will -also be shown. When in hybrid mode, all of these are included. - -When generating a HTML summary, the ``--mpl-results-always`` option is -automatically applied (see section below). Therefore images for passing -tests will also be shown. - -+---------------+---------------+---------------+ -| |html all| | |html filter| | |html result| | -+---------------+---------------+---------------+ - -As well as ``html``, ``basic-html`` can be specified for an alternative HTML -summary which does not rely on JavaScript or external resources. A ``json`` -summary can also be saved. Multiple options can be specified comma-separated. - -.. card:: Image comparison only - - .. code-block:: bash - - pytest --mpl --mpl-results-path=results --mpl-generate-summary=html,json - - :summary:`test_html_images_only` - -.. card:: Hash comparison only - - .. code-block:: bash - - pytest --mpl --mpl-hash-library=mpl35_ft261.json --mpl-results-path=results --mpl-generate-summary=html,json - - :summary:`test_html_hashes_only` - -.. card:: Hybrid mode: hash and image comparison - - .. code-block:: bash - - pytest --mpl --mpl-hash-library=mpl35_ft261.json --mpl-baseline-path=baseline --mpl-results-path=results --mpl-generate-summary=html,json - - :summary:`test_html` - -.. card:: Generating baseline images and hashes (With no testing) - - .. code-block:: bash - - pytest --mpl --mpl-generate-path=baseline --mpl-generate-hash-library=test_hashes.json --mpl-results-path=results --mpl-generate-summary=html,json - - :summary:`test_html_generate` - -.. card:: Generating baseline images (With no testing) - - .. code-block:: bash - - pytest --mpl --mpl-generate-path=baseline --mpl-results-path=results --mpl-generate-summary=html,json - - :summary:`test_html_generate_images_only` - -.. card:: Generating baseline hashes (With image comparison) - - .. code-block:: bash - - pytest --mpl --mpl-generate-hash-library=test_hashes.json --mpl-results-path=results --mpl-generate-summary=html,json - - :summary:`test_html_generate_hashes_only` - -.. card:: Generating baseline hashes (With hash comparison) - - .. code-block:: bash - - pytest --mpl --mpl-generate-hash-library=test_hashes.json --mpl-hash-library=mpl35_ft261.json --mpl-results-path=results --mpl-generate-summary=html,json - - :summary:`test_html_run_generate_hashes_only` - -.. card:: Hybrid mode: hash and image comparison - - .. code-block:: bash - - pytest --mpl --mpl-hash-library=mpl35_ft261.json --mpl-baseline-path=baseline --mpl-results-path=results --mpl-generate-summary=basic-html,json - - :summary:`test_basic_html` - -.. |html all| image:: images/html_all.png -.. |html filter| image:: images/html_filter.png -.. |html result| image:: images/html_result.png diff --git a/docs/usage.rst b/docs/usage.rst index ff6e282a..0c944171 100644 --- a/docs/usage.rst +++ b/docs/usage.rst @@ -1,93 +1,247 @@ -.. title:: Basic Usage +.. title:: Get Started ########### -Basic Usage +Get Started ########### -For each figure to test, the reference image is subtracted from the generated image, and the RMS of the residual is compared to a user-specified tolerance. If the residual is too large, the test will fail (this is implemented using helper functions from ``matplotlib.testing``). +This section describes how to configure simple image comparison testing with ``pytest-mpl``. +It assumes that you already have a `pytest `__ test suite set up for your project. -With Baseline Images -^^^^^^^^^^^^^^^^^^^^ +Install ``pytest-mpl`` +^^^^^^^^^^^^^^^^^^^^^^ + +First, install ``pytest-mpl`` and also include it in your project's "test" dependencies: + +.. code-block:: python + + pip install pytest-mpl + +Write image comparison tests +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Then, alongside your existing tests, write test functions that create a figure. +These image comparison tests must be decorated with ``@pytest.mark.mpl_image_compare`` and return the figure for testing: + +.. code-block:: python -To use, you simply need to mark the function where you want to compare -images using ``@pytest.mark.mpl_image_compare``, and make sure that the -function returns a Matplotlib figure (or any figure object that has a -``savefig`` method): + import matplotlib.pyplot as plt + import pytest -.. code:: python + @pytest.mark.mpl_image_compare + def test_plot(): + fig, ax = plt.subplots() + ax.plot([1, 2]) + return fig - import pytest - import matplotlib.pyplot as plt +Generate baseline images +^^^^^^^^^^^^^^^^^^^^^^^^ - @pytest.mark.mpl_image_compare - def test_succeeds(): - fig = plt.figure() - ax = fig.add_subplot(1,1,1) - ax.plot([1,2,3]) - return fig +Then, generate reference images by running the test suite with the ``--mpl-generate-path`` option: -To generate the baseline images, run the tests with the -``--mpl-generate-path`` option with the name of the directory where the -generated images should be placed:: +.. code-block:: bash - pytest --mpl-generate-path=baseline + pytest --mpl-generate-path=baseline -If the directory does not exist, it will be created. The directory will -be interpreted as being relative to where you are running ``pytest``. -Once you are happy with the generated images, you should move them to a -sub-directory called ``baseline`` relative to the test files (this name -is configurable, see below). You can also generate the baseline image -directly in the right directory. +If, for example, you are using a directory structure like this: -With a Hash Library -^^^^^^^^^^^^^^^^^^^ +.. code-block:: -Instead of comparing to baseline images, you can instead compare against a JSON -library of SHA-256 hashes. This has the advantage of not having to check baseline -images into the repository with the tests, or download them from a remote -source. + . + └── tests + ├── plotting + │ └── test_plotting.py + └── utils + └── test_utils.py -The hash library can be generated with -``--mpl-generate-hash-library=path_to_file.json``. The hash library to be used -can either be specified via the ``--mpl-hash-library=`` command line argument, -or via the ``hash_library=`` keyword argument to the -``@pytest.mark.mpl_image_compare`` decorator. +Then, the generated images will be placed in a new directory called ``baseline`` located where you ran pytest: -When generating a hash library, the tests will also be run as usual against the -existing hash library specified by ``--mpl-hash-library`` or the keyword argument. -However, generating baseline images will always result in the tests being skipped. +.. code-block:: + . + ├── baseline + │ ├── test_plot.png + │ └── test_util.png + └── tests + ├── plotting + │ └── test_plotting.py + └── utils + └── test_utils.py -Hybrid Mode: Hashes and Images +Take a look at the generated images inside the new ``baseline`` directory. +If they are correct, move each baseline image to a sub-directory called ``baseline`` relative to the test files: + +.. code-block:: + + . + └── tests + ├── plotting + │ ├── baseline + │ │ └── test_plot.png + │ └── test_plotting.py + └── utils + ├── baseline + │ └── test_util.png + └── test_utils.py + +Then, commit these baseline images to your repository. + +Run the image comparison tests ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -It is possible to configure both hashes and baseline images. In this scenario -only the hash comparison can determine the test result. If the hash comparison -fails, the test will fail, however a comparison to the baseline image will be -carried out so the actual difference can be seen. If the hash comparison passes, -the comparison to the baseline image is skipped (unless **results always** is -configured). +You now have a set of baseline images that can be used to verify the figures generated by your code. +You can now run the test suite as usual. +To enable image comparison testing, pass ``--mpl``: + +.. code-block:: bash + + pytest --mpl + +You should observe that the tests pass. + +Try modifying the test function and run the test suite again. +The test should now fail, because the generated image does not match the reference image. +Try running the test suite without ``--mpl``. +Even through the figure has changed, the test will pass, because image comparison testing is disabled. + +.. rubric:: Running pytest without pytest-mpl installed + +If ``pytest-mpl`` is not installed, the image comparison tests will cause pytest to show a warning, ``PytestReturnNotNoneWarning``. +Installing pytest-mpl will solve this issue. +When ``pytest-mpl`` is installed but not enabled, it will intercept the returned figure and close it without doing any comparison. + +Alternatively, the image comparison tests can be deselected by running pytest with ``-m "not mpl_image_compare"``. +Or the following can be included in your test functions to skip if ``pytest-mpl`` is not installed: + +.. code-block:: python + + @pytest.mark.mpl_image_compare + def test_plot(): + pytest.importorskip("pytest_mpl") + ... + +.. rubric:: Tests can fail when Matplotlib and FreeType versions change + +If the Matplotlib version changes, or if the FreeType version changes, the generated images may change. +This is mostly because the text rendering in Matplotlib is dependent on the FreeType version. +It is recommended to pin the Matplotlib and FreeType versions in your testing environments to avoid this issue. +There are also a number of :ref:`configuration options for controlling the sensitivity of the comparison `. + +Image comparison mode +^^^^^^^^^^^^^^^^^^^^^ + +The above example uses the image comparison mode, which is the default when just ``--mpl`` is set. +Pros and cons of this mode are: + +- :octicon:`diff-added;1em;sd-text-success` Easy to configure +- :octicon:`diff-added;1em;sd-text-success` Easy to run the tests and see the results +- :octicon:`diff-removed;1em;sd-text-danger` Baseline images usually need to be checked into the repository with the tests (larger repo) + +For a more detailed example of image comparison testing, see the :doc:`image comparison mode how-to guide `. +Also see the :doc:`configuration guide ` for more information on configuring image comparison testing. + +Hash comparison mode +^^^^^^^^^^^^^^^^^^^^ + +Instead of comparing to baseline images, you can instead compare against a JSON library of SHA-256 hashes of the baseline image files. +Pros and cons of this mode are: + +- :octicon:`diff-added;1em;sd-text-success` Easy to configure +- :octicon:`diff-removed;1em;sd-text-danger` Difficult to *see* how a failing test differs from the baseline +- :octicon:`diff-added;1em;sd-text-success` Baseline images do not need to be checked into the repository with the tests (smaller repo) + +See the :doc:`hash comparison mode how-to guide ` for more information on how to use this mode. +Also see the :doc:`configuration guide ` for more information on configuring hash comparison testing. + +.. _hybrid-usage: + +Hybrid mode: hash, then image, comparison +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +You can also use a "hybrid mode", which uses both baseline images and hashes. +A figure is first *assessed* against the baseline hash. +If the hash does not match, the figure will then be *compared* to the baseline image. +This mode can also be configured to :ref:`always compare the figure to the baseline image `, even if the hash matches. + +In this mode, only the hash assessment will affect the success status of the test. +If the hash assessment fails, the test will fail, even if the image comparison passes. + +This mode is intended for projects which have a large number of tests, and where it is impractical to store all of the baseline images in the repository. +These projects can use this mode to store the baseline images in a separate repository, and only store hashes in the main repository. +In PRs, contributors only need to update the hashes, and the CI tests will pass if the hashes match. +For the baseline image comparison, ``pytest-mpl`` will download the baseline image from a URL and compare it to the generated image. + +Pros and cons of this mode are: + +- :octicon:`diff-removed;1em;sd-text-danger` Usually more complex to configure (managing a separate baseline image repository) +- :octicon:`diff-added;1em;sd-text-success` Easy to run the tests and see the results +- :octicon:`diff-added;1em;sd-text-success` Baseline images can be stored in a separate repository (smaller main repo) + +See the :doc:`hybrid mode how-to guide ` for more information on how to use this mode. +Also see the :doc:`configuration guide ` for more information on configuring hybrid comparison testing. + +Test results +^^^^^^^^^^^^ + +By default, the expected, actual, and difference files are written to a temporary directory with a non-deterministic path. +You can :ref:`configure the results directory ` to save to a specific location:: + + pytest --mpl --mpl-results-path=results + +The ``results`` directory will then contain one sub-directory per test, and each sub-directory will contain the files mentioned above. +If you are using a continuous integration (CI) service, you can upload this directory as an artifact. + +HTML summary reports +-------------------- + +``pytest-mpl`` can also generate HTML reports of the image comparison results, allowing you to see the results of the image comparison tests in a web browser. +See the :ref:`configuration documentation ` for more information on how to generate the HTML report. +Some CI services, such as CircleCI, can host the HTML summary on a web server so it can be viewed directly in the browser. +On other CI services, such as GitHub Actions, you can download the artifact and open the local HTML file in a web browser. + ++---------------+---------------+---------------+ +| |html all| | |html filter| | |html result| | ++---------------+---------------+---------------+ + +Test failure example +-------------------- + +If the images produced by the tests are correct, then the test will pass. +If the images are not correct, the test will fail and a message similar to the following will be shown in the pytest logs:: + + E Exception: Error: Image files did not match. + E RMS Value: 142.2287807767823 + E Expected: + E /var/folders/zy/t1l3sx310d3d6p0kyxqzlrnr0000gr/T/tmp4h4oxr7y/baseline-coords_overlay_auto_coord_meta.png + E Actual: + E /var/folders/zy/t1l3sx310d3d6p0kyxqzlrnr0000gr/T/tmp4h4oxr7y/coords_overlay_auto_coord_meta.png + E Difference: + E /var/folders/zy/t1l3sx310d3d6p0kyxqzlrnr0000gr/T/tmp4h4oxr7y/coords_overlay_auto_coord_meta-failed-diff.png + E Tolerance: + E 10 -This is especially useful if the baseline images are external to the repository -containing the tests, and are accessed via HTTP. In this situation, if the hashes -match, the baseline images won't be retrieved, saving time and bandwidth. Also, it -allows the tests to be modified and the hashes updated to reflect the changes -without having to modify the external images. +The image paths included in the exception are then available for inspection: ++----------------+----------------+-------------+ +| Expected | Actual | Difference | ++================+================+=============+ +| |expected| | |actual| | |diff| | ++----------------+----------------+-------------+ -Running Tests -^^^^^^^^^^^^^ +In this case, the differences are very clear, while in some cases it may be necessary to use the difference image, or blink the expected and actual images, in order to see what changed. -Once tests are written with baseline images, a hash library, or both to compare -against, the tests can be run with:: +Continue reading +^^^^^^^^^^^^^^^^ - pytest --mpl +See the :doc:`configuration guide ` for more information on configuring ``pytest-mpl``. +For examples of how to configure the different operating modes, see the following how-to guides: -and the tests will pass if the images are the same. If you omit the -``--mpl`` option, the tests will run but will only check that the code -runs, without checking the output images. +* :doc:`image_mode` +* :doc:`hash_mode` +* :doc:`hybrid_mode` -If pytest-mpl is not installed, the image comparison tests will cause pytest -to show a warning, ``PytestReturnNotNoneWarning``. Installing pytest-mpl will -solve this issue. Alternativly, the image comparison tests can be deselected -by running pytest with ``-m "not mpl_image_compare"``. +.. |html all| image:: images/html_all.png +.. |html filter| image:: images/html_filter.png +.. |html result| image:: images/html_result.png +.. |expected| image:: images/baseline-coords_overlay_auto_coord_meta.png +.. |actual| image:: images/coords_overlay_auto_coord_meta.png +.. |diff| image:: images/coords_overlay_auto_coord_meta-failed-diff.png diff --git a/images/baseline-coords_overlay_auto_coord_meta.png b/images/baseline-coords_overlay_auto_coord_meta.png deleted file mode 100644 index 06970ddd..00000000 Binary files a/images/baseline-coords_overlay_auto_coord_meta.png and /dev/null differ diff --git a/images/coords_overlay_auto_coord_meta-failed-diff.png b/images/coords_overlay_auto_coord_meta-failed-diff.png deleted file mode 100644 index 0d1b7e2d..00000000 Binary files a/images/coords_overlay_auto_coord_meta-failed-diff.png and /dev/null differ diff --git a/images/coords_overlay_auto_coord_meta.png b/images/coords_overlay_auto_coord_meta.png deleted file mode 100644 index b6c1a5bc..00000000 Binary files a/images/coords_overlay_auto_coord_meta.png and /dev/null differ diff --git a/images/html_all.png b/images/html_all.png deleted file mode 100644 index 82e3eec4..00000000 Binary files a/images/html_all.png and /dev/null differ diff --git a/images/html_filter.png b/images/html_filter.png deleted file mode 100644 index 9fc6f998..00000000 Binary files a/images/html_filter.png and /dev/null differ diff --git a/images/html_result.png b/images/html_result.png deleted file mode 100644 index d8ef5c44..00000000 Binary files a/images/html_result.png and /dev/null differ