Major update to "using mypy with existing codebase"

See python#13681

In particular, it's really common to want to make progress towards
`--strict`, and we currently don't really have any guidance on doing so.
Per-module ignore_errors is also a really useful tool for adopting mypy.

Also try to link more to existing documentation elsewhere.

Author: hauntsaninja

docs/source/existing_code.rst

@@ -7,46 +7,94 @@ This section explains how to get started using mypy with an existing,
 significant codebase that has little or no type annotations. If you are
 a beginner, you can skip this section.
 
-These steps will get you started with mypy on an existing codebase:
+Start small
+-----------
 
-1. Start small -- get a clean mypy build for some files, with few
-   annotations
+If your codebase is large, pick a subset of your codebase (say, 5,000 to 50,000
+lines) and get mypy to run successfully only on this subset at first, *before
+adding annotations*. This should be doable in a day or two. The sooner you get
+some form of mypy passing on your codebase, the sooner you benefit.
 
-2. Write a mypy runner script to ensure consistent results
+You'll likely need to fix some mypy errors, either by inserting
+annotations requested by mypy or by adding ``# type: ignore``
+comments to silence errors you don't want to fix now.
 
-3. Run mypy in Continuous Integration to prevent type errors
+We'll mention some tips for getting mypy passing on your codebase in various
+sections below.
 
-4. Gradually annotate commonly imported modules
+Run mypy consistently and prevent regressions
+---------------------------------------------
 
-5. Write annotations as you modify existing code and write new code
+Make sure all developers on your codebase run mypy the same way.
+One way to ensure this is adding a small script with your mypy
+invocation to your codebase, or adding your mypy invocation to
+existing tools you use to run tests, like ``tox``.
 
-6. Use :doc:`monkeytype:index` or `PyAnnotate`_ to automatically annotate legacy code
+* Make sure everyone runs mypy with the same options. Checking a mypy
+  :ref:`configuration file <config-file>` into your codebase can help
+  with this.
 
-We discuss all of these points in some detail below, and a few optional
-follow-up steps.
+* Make sure everyone type checks the same set of files. See
+  :ref:`specifying-code-to-be-checked` for details.
 
-Start small
------------
+* Make sure everyone runs mypy with the same version of mypy, for instance
+  by pinning mypy with the rest of your dev requirements.
 
-If your codebase is large, pick a subset of your codebase (say, 5,000
-to 50,000 lines) and run mypy only on this subset at first,
-*without any annotations*. This shouldn't take more than a day or two
-to implement, so you start enjoying benefits soon.
+In particular, you'll want to make sure to run mypy as part of your
+Continuous Integration (CI) system as soon as possible. This will
+prevent new type errors from being introduced into your codebase.
 
-You'll likely need to fix some mypy errors, either by inserting
-annotations requested by mypy or by adding ``# type: ignore``
-comments to silence errors you don't want to fix now.
+A simple CI script could look something like this:
+
+.. code-block:: text
+
+    python3 -m pip install mypy==0.971
+    # Run your standardised mypy invocation, e.g.
+    mypy my_project
+    # This could also look like `scripts/run_mypy.sh`, `tox -e mypy`, `make mypy`, etc
+
+Ignoring errors from certain modules
+------------------------------------
 
-In particular, mypy often generates errors about modules that it can't
-find or that don't have stub files:
+By default mypy will follow imports in your code and try to check everything.
+This means even if you only pass in a few files to mypy, it may still process a
+large number of imported files. This could potentially result in lots of errors
+you don't want to deal with at the moment.
+
+One way to deal with this is to ignore errors in modules you aren't yet ready to
+type check. The :confval:`ignore_errors` option is useful for this, for instance,
+if you aren't yet ready to deal with errors from ``package_to_fix_later``:
+
+.. code-block:: text
+
+   [mypy-package_to_fix_later.*]
+   ignore_errors = True
+
+You could even invert this, by setting ``ignore_errors = True`` in your global
+config section and only enabling error reporting with ``ignore_errors = False``
+for the set of modules you are ready to type check.
+
+Fixing errors related to imports
+--------------------------------
+
+A common class of error you will encounter is errors from mypy about modules
+that it can't find, that don't have types, or don't have stub files:
 
 .. code-block:: text
 
     core/config.py:7: error: Cannot find implementation or library stub for module named 'frobnicate'
     core/model.py:9: error: Cannot find implementation or library stub for module named 'acme'
     ...
 
-This is normal, and you can easily ignore these errors. For example,
+Sometimes these can be fixed by installing the relevant packages or
+stub libraries in the environment you're running ``mypy`` in.
+
+See :ref:`ignore-missing-imports` for a complete reference on these errors
+and the ways in which you can fix them.
+
+You'll likely find that you want to suppress all errors from importing
+a given module that doesn't have types. If you only import that module
+in one or two places, you can use ``# type: ignore`` comments. For example,
 here we ignore an error about a third-party module ``frobnicate`` that
 doesn't have stubs using ``# type: ignore``:
 
@@ -56,9 +104,9 @@ doesn't have stubs using ``# type: ignore``:
    ...
    frobnicate.initialize()  # OK (but not checked)
 
-You can also use a mypy configuration file, which is convenient if
-there are a large number of errors to ignore. For example, to disable
-errors about importing ``frobnicate`` and ``acme`` everywhere in your
+But if you import the module in many places, this becomes unwieldy. In this
+case, we recommend using a :ref:`configuration file <config-file>`. For example,
+to disable errors about importing ``frobnicate`` and ``acme`` everywhere in your
 codebase, use a config like this:
 
 .. code-block:: text
@@ -69,69 +117,33 @@ codebase, use a config like this:
    [mypy-acme.*]
    ignore_missing_imports = True
 
-You can add multiple sections for different modules that should be
-ignored.
-
-If your config file is named ``mypy.ini``, this is how you run mypy:
-
-.. code-block:: text
-
-   mypy --config-file mypy.ini mycode/
-
 If you get a large number of errors, you may want to ignore all errors
-about missing imports.  This can easily cause problems later on and
-hide real errors, and it's only recommended as a last resort.
-For more details, look :ref:`here <follow-imports>`.
+about missing imports, for instance by setting :confval:`ignore_missing_imports`
+to true globally. This can hide errors later on, so we recommend avoiding this
+if possible.
 
-Mypy follows imports by default. This can result in a few files passed
-on the command line causing mypy to process a large number of imported
-files, resulting in lots of errors you don't want to deal with at the
-moment. There is a config file option to disable this behavior, but
-since this can hide errors, it's not recommended for most users.
+Finally, mypy allows fine-grained control over specific import following
+behaviour. It's very easy to silently shoot yourself in the foot when playing
+around with these, so it's mostly recommended as a last resort. For more
+details, look :ref:`here <follow-imports>`.
 
-Mypy runner script
-------------------
-
-Introduce a mypy runner script that runs mypy, so that every developer
-will use mypy consistently. Here are some things you may want to do in
-the script:
-
-* Ensure that the correct version of mypy is installed.
-
-* Specify mypy config file or command-line options.
-
-* Provide set of files to type check. You may want to implement
-  inclusion and exclusion filters for full control of the file
-  list.
-
-Continuous Integration
-----------------------
-
-Once you have a clean mypy run and a runner script for a part
-of your codebase, set up your Continuous Integration (CI) system to
-run mypy to ensure that developers won't introduce bad annotations.
-A simple CI script could look something like this:
-
-.. code-block:: text
-
-    python3 -m pip install mypy==0.790  # Pinned version avoids surprises
-    scripts/mypy  # Run the mypy runner script you set up
-
-Annotate widely imported modules
---------------------------------
+Prioritise annotating widely imported modules
+---------------------------------------------
 
 Most projects have some widely imported modules, such as utilities or
 model classes. It's a good idea to annotate these pretty early on,
 since this allows code using these modules to be type checked more
-effectively. Since mypy supports gradual typing, it's okay to leave
-some of these modules unannotated. The more you annotate, the more
-useful mypy will be, but even a little annotation coverage is useful.
+effectively.
+
+Mypy is designed to support gradual typing, i.e. letting you add annotations at
+your own pace, so it's okay to leave some of these modules unannotated. The more
+you annotate, the more useful mypy will be, but even a little annotation
+coverage is useful.
 
 Write annotations as you go
 ---------------------------
 
-Now you are ready to include type annotations in your development
-workflows. Consider adding something like these in your code style
+Consider adding something like these in your code style
 conventions:
 
 1. Developers should add annotations for any new code.
@@ -143,9 +155,9 @@ codebase without much effort.
 Automate annotation of legacy code
 ----------------------------------
 
-There are tools for automatically adding draft annotations
-based on type profiles collected at runtime.  Tools include
-:doc:`monkeytype:index` (Python 3) and `PyAnnotate`_.
+There are tools for automatically adding draft annotations based on simple
+static analysis or on type profiles collected at runtime.  Tools include
+:doc:`monkeytype:index`, `autotyping`_ and `PyAnnotate`_.
 
 A simple approach is to collect types from test runs. This may work
 well if your test coverage is good (and if your tests aren't very
@@ -156,6 +168,68 @@ fraction of production network requests.  This clearly requires more
 care, as type collection could impact the reliability or the
 performance of your service.
 
+Introduce stricter options
+--------------------------
+
+Mypy is very configurable. Once you get started with static typing, you may want
+to explore the various strictness options mypy provides to catch more bugs. For
+example, you can ask mypy to require annotations for all functions in certain
+modules to avoid accidentally introducing code that won't be type checked using
+:confval:`disallow_untyped_defs`. Refer to :ref:`config-file` for the details.
+
+An excellent goal to aim for is to have your codebase pass when run against ``mypy --strict``.
+This basically ensures that you will never have a type related error without an explicit
+circumvention somewhere (such as a ``# type: ignore`` comment).
+
+The following config is equivalent to ``--strict``:
+
+.. code-block:: text
+
+   # Start off with these
+   warn_unused_configs = True
+   warn_redundant_casts = True
+   warn_unused_ignores = True
+   no_implicit_optional = True
+
+   # Getting these passing should be easy
+   strict_equality = True
+   strict_concatenate = True
+
+   # Strongly recommend enabling this one as soon as you can
+   check_untyped_defs = True
+
+   # These shouldn't be too much additional work, but may be tricky to
+   # get passing if you use a lot of untyped libraries
+   disallow_subclassing_any = True
+   disallow_untyped_decorators = True
+   disallow_any_generics = True
+
+   # These next few are various gradations of forcing use of type annotations
+   disallow_untyped_calls = True
+   disallow_incomplete_defs = True
+   disallow_untyped_defs = True
+
+   # This one isn't too hard to get passing, but return on investment is lower
+   no_implicit_reexport = True
+
+   # This one can be tricky to get passing if you use a lot of untyped libraries
+   warn_return_any = True
+
+Note that you can also start with ``--strict`` and subtract, for instance:
+
+.. code-block:: text
+
+   strict = True
+   warn_return_any = False
+
+Remember that many of these options can be enabled on a per-module basis. For instance,
+you may want to enable ``disallow_untyped_defs`` for modules which you've completed
+annotations for, in order to prevent new code from being added without annotations.
+
+And if you want, it doesn't stop at ``--strict``. Mypy has additional checks
+that are not part of ``--strict`` that can be useful. See the complete
+:ref:`command-line` reference and :ref:`error-codes-optional`.
+
 Speed up mypy runs
 ------------------
 
@@ -165,14 +239,5 @@ this will be.  If your project has at least 100,000 lines of code or
 so, you may also want to set up :ref:`remote caching <remote-cache>`
 for further speedups.
 
-Introduce stricter options
---------------------------
-
-Mypy is very configurable. Once you get started with static typing, you may want
-to explore the various strictness options mypy provides to catch more bugs. For
-example, you can ask mypy to require annotations for all functions in certain
-modules to avoid accidentally introducing code that won't be type checked using
-:confval:`disallow_untyped_defs`, or type check code without annotations as well
-with :confval:`check_untyped_defs`. Refer to :ref:`config-file` for the details.
-
 .. _PyAnnotate: https://github.com/dropbox/pyannotate
+.. _autotyping: https://github.com/JelleZijlstra/autotyping