Initial import of PEP 592

Author: Till Varoquaux

pep-0592.rst

@@ -0,0 +1,295 @@
+PEP: 592
+Title: External annotations in the typing module
+Author: Till Varoquaux <[email protected]>, Konstantin Kashin <[email protected]>
+Sponsor: Ivan Levkivskyi <[email protected]>
+Discussions-To: [email protected]
+Status: Draft
+Type: Standards Track
+Content-Type: text/x-rst
+Created: 26-April-2019
+Python-Version:
+Post-History:
+
+Abstract
+--------
+
+This PEP introduces a mechanism to extend the type annotations from PEP
+484 with arbitrary metadata.
+
+Motivation
+----------
+
+PEP 484 provides a standard semantic for the annotations introduced in
+PEP 3107. PEP 484 is prescriptive but it is the de-facto standard
+for most of the consumers of annotations; in many statically checked
+code bases, where type annotations are widely used, they have
+effectively crowded out any other form of annotation. Some of the use
+cases for annotations described in PEP 3107 (database mapping,
+foreign languages bridge) are not currently realistic given the
+prevalence of type annotations. Furthermore the standardisation of type
+annotations rules out advanced features only supported by specific type
+checkers.
+
+Rationale
+---------
+
+We propose adding an ``Annotated`` type to the typing module to decorate
+existing types with context-specific metadata. Specifically, a type
+``T`` can be annotated with metadata ``x`` via the typehint
+``Annotated[T, x]``. This metadata can be used for either static
+analysis or at runtime. If a library (or tool) encounters a typehint
+``Annotated[T, x]`` and has no special logic for metadata ``x``, it
+should ignore it and simply treat the type as ``T``. Unlike the
+``no_type_check`` functionality that current exists in the ``typing``
+module which completely disables typechecking annotations on a function
+or a class, the ``Annotated`` type allows for both static typechecking
+of ``T`` (e.g., via mypy [mypy]_ or Pyre [pyre]_, which can safely ignore ``x``)
+together with runtime access to ``x`` within a specific application. We
+believe that the introduction of this type would address a diverse set
+of use cases of interest to the broader Python community.
+
+This was originally brought up as issue 600 [issue-600]_ in the typing github
+and then discussed in Python ideas [python-ideas]_.
+
+Motivating examples
+-------------------
+
+Combining runtime and static uses of annotations
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+There's an emerging trend of libraries leveraging the typing annotations at
+runtime (e.g.: dataclasses); having the ability to extend the typing annotations
+with external data would be a great boon for those libraries.
+
+Example::
+
+    UnsignedShort = Annotated[int, cstruct.ctype('H')]
+    SignedChar = Annotated[int, cstruct.ctype('b')]
+
+    class Student(cstruct.Packed):
+        # mypy typechecks 'name' field as 'str'
+        name: Annotated[str, cstruct.ctype("<10s")]
+        serialnum: UnsignedShort
+        school: SignedChar
+        gradelevel: SignedChar
+
+    # 'unpack' only uses the metadata within the type annotations
+    Student.unpack(record)
+    # Student(name=b'raymond   ', serialnum=4658, school=264, gradelevel=8)
+
+Lowering barriers to developing new typing constructs
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Typically when adding a new type, we need to upstream that type to the
+typing module and change mypy, PyCharm [pycharm]_, Pyre,
+pytype [pytype]_, etc...
+This is particularly important when working on open-source code that
+makes use of our new types, seeing as the code would not be immediately
+transportable to other developers' tools without additional logic. As a result,
+there is a high cost to developing and trying out new types in a codebase.
+Ideally, we should be able to introduce new types in a manner that allows for
+graceful degradation when clients do not have a custom mypy plugin
+[mypy-plugin]_, which would lower the barrier to development and ensure some
+degree of backward compatibility.
+
+For example, suppose that we wanted to add support for tagged unions
+[tagged-union]_ to Python. One way to accomplish would be to annotate
+``TypedDict`` [typed-dict]_ in Python such that only one field is allowed to be
+set::
+
+    Currency = Annotated[
+        TypedDict('Currency', {'dollars': float, 'pounds': float}, total=False),
+        TaggedUnion,
+    ]
+
+This is a somewhat cumbersome syntax but it allows us to iterate on this
+proof-of-concept and have people with type checkers (or other tools) that don't
+yet support this feature work in a codebase with tagged unions. We could easily
+test this proposal and iron out the kinks before trying to upstream tagged union
+to ``typing``, mypy, etc. Moreover, tools that do not have support for parsing
+the ``TaggedUnion`` annotation would still be able able to treat ``Currency`` as
+a ``TypedDict``, which is still a close approximation (slightly less strict).
+
+Specification
+-------------
+
+Syntax
+~~~~~~
+
+``Annotated`` is parameterized with a type and an arbitrary list of
+Python values that represent the annotations. Here are the specific
+details of the syntax:
+
+* The first argument to ``Annotated`` must be a valid type
+
+* Multiple type annotations are supported (``Annotated`` supports variadic
+  arguments)::
+
+    Annotated[int, ValueRange(3, 10), ctype("char")]
+
+* ``Annotated`` must be called with at least two arguments (
+  ``Annotated[int]`` is not valid)
+
+* The order of the annotations is preserved and matters for equality
+  checks::
+
+    Annotated[int, ValueRange(3, 10), ctype("char")] != Annotated[
+        int, ctype("char"), ValueRange(3, 10)
+    ]
+
+* Nested ``Annotated`` types are flattened, with metadata ordered
+  starting with the innermost annotation::
+
+    Annotated[Annotated[int, ValueRange(3, 10)], ctype("char")] == Annotated[
+        int, ValueRange(3, 10), ctype("char")
+    ]
+
+* Duplicated annotations are not removed::
+
+    Annotated[int, ValueRange(3, 10)] != Annotated[
+        int, ValueRange(3, 10), ValueRange(3, 10)
+    ]
+
+* ``Annotated`` can be used as a higher order aliases::
+
+    Typevar T = ...
+    Vec = Annotated[List[Tuple[T, T]], MaxLen(10)]
+    V = Vec[int]
+
+    V == Annotated[List[Tuple[int, int]], MaxLen(10)]
+
+Consuming annotations
+~~~~~~~~~~~~~~~~~~~~~
+
+Ultimately, the responsibility of how to interpret the annotations (if
+at all) is the responsibility of the tool or library encountering the
+``Annotated`` type. A tool or library encountering an ``Annotated`` type
+can scan through the annotations to determine if they are of interest
+(e.g., using ``isinstance()``).
+
+**Unknown annotations:** When a tool or a library does not support
+annotations or encounters an unknown annotation it should just ignore it
+and treat annotated type as the underlying type. For example, if we were
+to add an annotation that is not an instance of ``new_struct.ctype`` to the
+annotation for name (e.g.,
+``Annotated[str, 'foo', new_struct.ctype("<10s")]``), the unpack method
+should ignore it.
+
+**Namespacing annotations:** We do not need namespaces for annotations
+since the class used by the annotations acts as a namespace.
+
+**Multiple annotations:** It's up to the tool consuming the annotations
+to decide whether the client is allowed to have several annotations on
+one type and how to merge those annotations.
+
+Since the ``Annotated`` type allows you to put several annotations of
+the same (or different) type(s) on any node, the tools or libraries
+consuming those annotations are in charge of dealing with potential
+duplicates. For example, if you are doing value range analysis you might
+allow this::
+
+    T1 = Annotated[int, ValueRange(-10, 5)]
+    T2 = Annotated[T1, ValueRange(-20, 3)]
+
+Flattening nested annotations, this translates to::
+
+    T2 = Annotated[int, ValueRange(-10, 5), ValueRange(-20, 3)]
+
+Interaction with ``get_type_hints()``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``typing.get_type_hints()`` will take a new argument ``include_extras`` that
+defaults to ``False`` to preserve backward compatibility. When
+``include_extras`` is ``False``, the extra annotations will be stripped
+out of the returned value. Otherwise, the annotations will be returned
+unchanged::
+
+    @struct.packedclass Student(NamedTuple):
+        name: Annotated[str, struct.ctype("<10s")]
+
+    get_type_hints(Student) == {'name': str}
+    get_type_hints(Student, include_extras=False) == {'name': str}
+    get_type_hints(Student, include_extras=True) == {
+        'name': Annotated[str, struct.ctype("<10s")]
+    }
+
+Aliases & Concerns over verbosity
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Writing ``typing.Annotated`` everywhere can be quite verbose;
+fortunately, the ability to alias annotations means that in practice we
+don't expect clients to have to write lots of boilerplate code::
+
+    T = TypeVar('T')
+    Const = Annotated[T, my_annotations.CONST]
+
+    Class C:
+        def const_method(self: Const[List[int]]) -> int:
+            ...
+
+Rejected ideas
+--------------
+
+Some of the proposed ideas were rejected from this PEP because they would
+cause ``Annotated`` to not integrate cleanly with the other typing annotations:
+
+* ``Annotated`` cannot infer the decorated type. You could imagine that
+  ``Annotated[..., Immutable]`` could be used to mark a value as immutable
+  while still infering its type. Typing does not support support using the
+  inferred type anywhere else [issue-276]_; it's best to not add this as a
+  special case.
+
+* We could use ``(Type, Ann1, Ann2, ...)`` instead of
+  ``Annotated[Type, Ann1, Ann2, ...]``. This would cause confusion when
+  annotations appear in nested positions (``Callable[[A, B], C]`` is too similar
+  to ``Callable[[(A, B)], C]``) and would make it impossible for constructors to
+  be passthrough (``T(5) == C(5)`` when ``C = Annotation[T, Ann]``).
+
+This feature was left out to keep the design simple:
+
+* ``Annotated`` cannot be called with a single argument. Annotated could support
+  returning the underlying value when called with a single argument (e.g.:
+  ``Annotated[int] == int``). This complicates the specifications and adds
+  little benefit.
+
+
+References
+----------
+
+.. [issue-600]
+   https://github.com/python/typing/issues/600
+
+.. [python-ideas]
+   https://mail.python.org/pipermail/python-ideas/2019-January/054908.html
+
+.. [struct-doc]
+   https://docs.python.org/3/library/struct.html#examples
+
+.. [mypy]
+	http://www.mypy-lang.org/
+
+.. [pyre]
+   https://pyre-check.org/
+
+.. [pycharm]
+   https://www.jetbrains.com/pycharm/
+
+.. [pytype]
+   https://github.com/google/pytype
+
+.. [mypy-plugin]
+   https://github.com/python/mypy_extensions
+
+.. [tagged-union]
+   https://en.wikipedia.org/wiki/Tagged_union
+
+.. [typed-dict]
+   https://mypy.readthedocs.io/en/latest/more_types.html#typeddict
+
+.. [issue-276]
+   https://github.com/python/typing/issues/276
+
+Copyright
+---------
+
+This document has been placed in the public domain.