From 822133a29166860869154a7eb3e172d8dda57bdb Mon Sep 17 00:00:00 2001 From: Jukka Lehtosalo Date: Mon, 11 Dec 2017 17:08:57 +0000 Subject: [PATCH 1/5] Update docs to reflect the fact that Iterable etc. are protocols Also update the discussion of ABCs. --- docs/source/class_basics.rst | 255 ++++++++++++++++++++++++++++++++--- docs/source/faq.rst | 13 +- 2 files changed, 240 insertions(+), 28 deletions(-) diff --git a/docs/source/class_basics.rst b/docs/source/class_basics.rst index 3dc116cfb089..cfff89b6a363 100644 --- a/docs/source/class_basics.rst +++ b/docs/source/class_basics.rst @@ -117,11 +117,15 @@ annotations have no effect at runtime: Abstract base classes and multiple inheritance ********************************************** -Mypy uses Python abstract base classes for protocol types. There are -several built-in abstract base classes types (for example, -``Sequence``, ``Iterable`` and ``Iterator``). You can define abstract -base classes using the ``abc.ABCMeta`` metaclass and the -``abc.abstractmethod`` function decorator. +Mypy supports Python abstract base classes (ABCs). Abstract classes +have at least one abstract method or property that must be implemented +by a subclass. You can define abstract base classes using the +``abc.ABCMeta`` metaclass, and the ``abc.abstractmethod`` and +``abc.abstractproperty`` function decorators. + +Note that mypy performs checking for unimplemented abstract methods +even if you omit the ``ABCMeta`` metaclass. This can be useful if the +metaclass would cause runtime metaclass conflicts. .. code-block:: python @@ -143,25 +147,19 @@ base classes using the ``abc.ABCMeta`` metaclass and the a = A() # Error: A is abstract b = B() # OK -Unlike most Python code, abstract base classes are likely to play a -significant role in many complex mypy programs. - A class can inherit any number of classes, both abstract and concrete. As with normal overrides, a dynamically typed method can -implement a statically typed abstract method defined in an abstract -base class. +implement a statically typed method defined in any base class, +including an abstract method defined in an abstract base class. + +You can implement an abstract property using either a normal +property or an instance variable. .. _protocol-types: Protocols and structural subtyping ********************************** -.. note:: - - Structural subtyping is experimental. Some things may not - work as expected. Mypy may pass unsafe code or it can reject - valid code. - Mypy supports two ways of deciding whether two classes are compatible as types: nominal subtyping and structural subtyping. *Nominal* subtyping is strictly based on the class hierarchy. If class ``D`` @@ -174,16 +172,231 @@ a structural subtype of class ``C`` if the former has all attributes and methods of the latter, and with compatible types. Structural subtyping can be seen as a static equivalent of duck -typing, which is well known to Python programmers. Mypy provides an -opt-in support for structural subtyping via protocol classes described +typing, which is well known to Python programmers. Mypy provides +support for structural subtyping via protocol classes described below. See `PEP 544 `_ for the detailed specification of protocols and structural subtyping in Python. +Built-in protocols +****************** + +The ``typing`` module defines various protocol classes that correspond +to common Python protocols, such as ``Iterable[T]``. If a class +defines a suitable ``__iter__`` method, mypy understands that it +implements the iterable protocol and is compatible with ``Iterable[T]``. +For example, ``IntList`` below is iterable, over ``int`` values: + +.. code-block:: python + + from typing import Iterator, Iterable, Optional + + class IntList: + def __init__(self, value: int, next: Optional[IntList]) -> None: + self.value = value + self.next = next + + def __iter__(self) -> Iterator[int]: + current = self + while current: + yield current.value + current = current.next + + def print_numbered(items: Iterable[int]) -> None: + for n, x in enumerate(items): + print(n + 1, x) + + x = IntList(3, IntList(5, None)) + print_numbered(x) # Okay + print_numbered([4, 5]) # Also okay + +The subsections below introduce all built-in protocols defined in +``typing`` and the corresponding methods you need to define to +implement each protocol. + +Iteration protocols +................... + +The iteration protocols are useful is numerous contexts. For example, they allow +iteration of objects in for loops. + +``Iterable[T]`` +--------------- + +.. code-block:: python + + def __iter__(self) -> Iterator[T] + +``Iterator[T]`` +--------------- + +.. code-block:: python + + def __next__(self) -> T + def __iter__(self) -> Iterator[T] + +Collection protocols +.................... + +Many of these are implemented by built-in container types such as +``list`` and ``dict``, and these are also useful for user-defined +collection objects. + +``Sized`` +--------- + +This defined for objects that support ``len(x)``. + +.. code-block:: python + + def __len__(self) -> int + +``Container[T]`` +---------------- + +.. code-block:: python + + def __contains__(self, x: object) -> bool + +``Collection[T]`` +----------------- + +.. code-block:: python + + def __len__(self) -> int + def __iter__(self) -> Iterator[T] + def __contains__(self, x: object) -> bool + +One-off protocols +................. + +These protocols are typically only useful with a single standard +library function or class. + +``Reversible[T]`` +----------------- + +This is defined for objects that support ``reversed(x)``. + +.. code-block:: python + + def __reversed__(self) -> Iterator[T] + +``SupportsAbs[T]`` +------------------ + +This is defined for objects that support ``abs(x)``. ``T`` is the type of +value returned by ``abs(x)``. + +.. code-block:: python + + def __abs__(self) -> T + +``SupportsBytes`` +----------------- + +This is defined for objects that support ``bytes(x)``. + +.. code-block:: python + + def __bytes__(self) -> bytes + +``SupportsComplex`` +------------------- + +This is defined for objects that support ``complex(x)``. + +.. code-block:: python + + def __complex__(self) -> complex + +``SupportsFloat`` +----------------- + +This is defined for objects that support ``float(x)``. + +.. code-block:: python + + def __float__(self) -> float + +``SupportsInt`` +--------------- + +This is defined for objects that support ``int(x)``. + +.. code-block:: python + + def __int__(self) -> int + +``SupportsRound[T]`` +-------------------- + +This is defined for objects that support ``round(x)``. + +.. code-block:: python + + def __round__(self) -> T + +Async protocols +............... + +These protocols are only useful in async code. + +``Awaitable[T]`` +---------------- + +.. code-block:: python + + def __await__(self) -> Generator[Any, None, T] + +``AsyncIterable[T]`` +-------------------- + +.. code-block:: python + + def __aiter__(self) -> AsyncIterator[T] + +``AsyncIterator[T]`` +-------------------- + +.. code-block:: python + + def __anext__(self) -> Awaitable[T] + def __aiter__(self) -> AsyncIterator[T] + +Context manager protocols +......................... + +There are two protocols for context managers -- one for regular context +managers and one for async ones. These allow defining objects that can +be used in with and async with statements. + +``ContextManager[T]`` +--------------------- + +.. code-block:: python + + def __enter__(self) -> T + def __exit__(self, + exc_type: Optional[Type[BaseException]], + exc_value: Optional[BaseException], + traceback: Optional[TracebackType]) -> Optional[bool] + +``AsyncContextManager[T]`` +-------------------------- + +.. code-block:: python + + def __aenter__(self) -> Awaitable[T] + def __aexit__(self, + exc_type: Optional[Type[BaseException]], + exc_value: Optional[BaseException], + traceback: Optional[TracebackType]) -> Awaitable[Optional[bool]] + Simple user-defined protocols ***************************** -You can define a protocol class by inheriting the special +You can define your own protocol class by inheriting the special ``typing_extensions.Protocol`` class: .. code-block:: python @@ -216,9 +429,7 @@ similarly compatible with the protocol, as they support ``close()``. The ``Protocol`` base class is currently provided in the ``typing_extensions`` package. Once structural subtyping is mature and `PEP 544 `_ has been accepted, - ``Protocol`` will be included in the ``typing`` module. Several library - types such as ``typing.Sized`` and ``typing.Iterable`` will also be changed - into protocols. They are currently treated as regular ABCs by mypy. + ``Protocol`` will be included in the ``typing`` module. Defining subprotocols ********************* diff --git a/docs/source/faq.rst b/docs/source/faq.rst index b131e3f7e2a2..ff1b8018c917 100644 --- a/docs/source/faq.rst +++ b/docs/source/faq.rst @@ -108,10 +108,11 @@ Mypy provides support for both `nominal subtyping `_ and `structural subtyping `_. -Support for structural subtyping is considered experimental. Some argue that structural subtyping is better suited for languages with duck typing such as Python. Mypy however primarily uses nominal subtyping, -leaving structural subtyping opt-in. Here are some reasons why: +leaving structural subtyping mostly opt-in (except for built-in protocols +such as ``Iterable`` that always support structural subtyping). Here are some +reasons why: 1. It is easy to generate short and informative error messages when using a nominal type system. This is especially important when @@ -119,12 +120,12 @@ leaving structural subtyping opt-in. Here are some reasons why: 2. Python provides built-in support for nominal ``isinstance()`` tests and they are widely used in programs. Only limited support for structural - ``isinstance()`` exists for ABCs in ``collections.abc`` and ``typing`` - standard library modules. + ``isinstance()`` is available, and it's less type safe than + nominal type tests. -3. Many programmers are already familiar with nominal subtyping and it +3. Many programmers are already familiar with static, nominal subtyping and it has been successfully used in languages such as Java, C++ and - C#. Only few languages use structural subtyping. + C#. Fewer languages use structural subtyping. However, structural subtyping can also be useful. For example, a "public API" may be more flexible if it is typed with protocols. Also, using protocol types From 3b03130091404de325e033c0da2107875a5bf0ad Mon Sep 17 00:00:00 2001 From: Jukka Lehtosalo Date: Tue, 12 Dec 2017 12:12:45 +0000 Subject: [PATCH 2/5] Small updates --- docs/source/class_basics.rst | 47 ++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/docs/source/class_basics.rst b/docs/source/class_basics.rst index cfff89b6a363..3a0ae9e36f3f 100644 --- a/docs/source/class_basics.rst +++ b/docs/source/class_basics.rst @@ -121,16 +121,11 @@ Mypy supports Python abstract base classes (ABCs). Abstract classes have at least one abstract method or property that must be implemented by a subclass. You can define abstract base classes using the ``abc.ABCMeta`` metaclass, and the ``abc.abstractmethod`` and -``abc.abstractproperty`` function decorators. - -Note that mypy performs checking for unimplemented abstract methods -even if you omit the ``ABCMeta`` metaclass. This can be useful if the -metaclass would cause runtime metaclass conflicts. +``abc.abstractproperty`` function decorators. Example: .. code-block:: python from abc import ABCMeta, abstractmethod - import typing class A(metaclass=ABCMeta): @abstractmethod @@ -144,8 +139,12 @@ metaclass would cause runtime metaclass conflicts. def bar(self) -> str: return 'x' - a = A() # Error: A is abstract - b = B() # OK + a = A() # Error: A is abstract + b = B() # OK + +Note that mypy performs checking for unimplemented abstract methods +even if you omit the ``ABCMeta`` metaclass. This can be useful if the +metaclass would cause runtime metaclass conflicts. A class can inherit any number of classes, both abstract and concrete. As with normal overrides, a dynamically typed method can @@ -163,7 +162,8 @@ Protocols and structural subtyping Mypy supports two ways of deciding whether two classes are compatible as types: nominal subtyping and structural subtyping. *Nominal* subtyping is strictly based on the class hierarchy. If class ``D`` -inherits class ``C``, it's also a subtype of ``C``. This form of +inherits class ``C``, it's also a subtype of ``C``, and instances of +``D`` can be used when ``C`` instances are expected. This form of subtyping is used by default in mypy, since it's easy to understand and produces clear and concise error messages, and since it matches how the native ``isinstance()`` check works -- based on class @@ -178,8 +178,10 @@ below. See `PEP 544 `_ for the detailed specification of protocols and structural subtyping in Python. -Built-in protocols -****************** +.. _predefined_protocols: + +Predefined protocols +******************** The ``typing`` module defines various protocol classes that correspond to common Python protocols, such as ``Iterable[T]``. If a class @@ -207,22 +209,26 @@ For example, ``IntList`` below is iterable, over ``int`` values: print(n + 1, x) x = IntList(3, IntList(5, None)) - print_numbered(x) # Okay - print_numbered([4, 5]) # Also okay + print_numbered(x) # OK + print_numbered([4, 5]) # Also OK The subsections below introduce all built-in protocols defined in -``typing`` and the corresponding methods you need to define to -implement each protocol. +``typing`` and the signatures of the corresponding methods you need to define +to implement each protocol (the signatures can be left out, as always, but mypy +won't type check unannotated methods). Iteration protocols ................... -The iteration protocols are useful is numerous contexts. For example, they allow +The iteration protocols are useful in many contexts. For example, they allow iteration of objects in for loops. ``Iterable[T]`` --------------- +The :ref:`example above ` has a simple implementation of an +``__iter__`` method. + .. code-block:: python def __iter__(self) -> Iterator[T] @@ -254,6 +260,8 @@ This defined for objects that support ``len(x)``. ``Container[T]`` ---------------- +This is defined for objects that support the ``in`` operator. + .. code-block:: python def __contains__(self, x: object) -> bool @@ -340,7 +348,7 @@ This is defined for objects that support ``round(x)``. Async protocols ............... -These protocols are only useful in async code. +These protocols can be useful in async code. ``Awaitable[T]`` ---------------- @@ -369,7 +377,7 @@ Context manager protocols There are two protocols for context managers -- one for regular context managers and one for async ones. These allow defining objects that can -be used in with and async with statements. +be used in ``with`` and ``async with`` statements. ``ContextManager[T]`` --------------------- @@ -537,6 +545,9 @@ adds support for basic runtime structural checks: if isinstance(mug, Portable): use(mug.handles) # Works statically and at runtime +``isinstance()`` also works with the :ref:`predefined protocols ` +in ``typing`` such as ``Iterable``. + .. note:: ``isinstance()`` with protocols is not completely safe at runtime. For example, signatures of methods are not checked. The runtime From bf6d5a7e6d78acbcb4d3bd555a707a5a33325fb5 Mon Sep 17 00:00:00 2001 From: Jukka Lehtosalo Date: Tue, 12 Dec 2017 12:23:40 +0000 Subject: [PATCH 3/5] Update discussion of subprotocols --- docs/source/class_basics.rst | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/docs/source/class_basics.rst b/docs/source/class_basics.rst index 3a0ae9e36f3f..4b7f3beeeb32 100644 --- a/docs/source/class_basics.rst +++ b/docs/source/class_basics.rst @@ -439,8 +439,8 @@ similarly compatible with the protocol, as they support ``close()``. `PEP 544 `_ has been accepted, ``Protocol`` will be included in the ``typing`` module. -Defining subprotocols -********************* +Defining subprotocols and subclassing protocols +*********************************************** You can also define subprotocols. Existing protocols can be extended and merged using multiple inheritance. Example: @@ -468,7 +468,7 @@ and merged using multiple inheritance. Example: Note that inheriting from an existing protocol does not automatically turn the subclass into a protocol -- it just creates a regular -(non-protocol) ABC that implements the given protocol (or +(non-protocol) class or ABC that implements the given protocol (or protocols). The ``typing_extensions.Protocol`` base class must always be explicitly present if you are defining a protocol: @@ -486,6 +486,13 @@ be explicitly present if you are defining a protocol: # Error: nominal subtyping used by default x: NewProtocol = Concrete() # Error! +You can also include default implementations of methods in +protocols. If you explicitly subclass these protocols you can inherit +these default implementations. Explicitly including a protocol as a +base class is also a way of documenting that your class implements a +particular protocol, and it forces mypy to verify that your class +implementation is actually compatible with the protocol. + .. note:: You can use Python 3.6 variable annotations (`PEP 526 From 274af7689bf0337528f2eb8c65f511eddd44dccc Mon Sep 17 00:00:00 2001 From: Jukka Lehtosalo Date: Tue, 12 Dec 2017 12:38:25 +0000 Subject: [PATCH 4/5] Update discussion of generics --- docs/source/generics.rst | 42 ++++++++++++++++++++++++---------------- 1 file changed, 25 insertions(+), 17 deletions(-) diff --git a/docs/source/generics.rst b/docs/source/generics.rst index 2ea9b424e139..00e80e92314d 100644 --- a/docs/source/generics.rst +++ b/docs/source/generics.rst @@ -112,21 +112,27 @@ non-generic. For example: .. code-block:: python - from typing import Generic, TypeVar, Iterable + from typing import Generic, TypeVar, Mapping, Iterator, Dict - T = TypeVar('T') + KT = TypeVar('KT') + VT = TypeVar('VT') - class Stream(Iterable[T]): # This is a generic subclass of Iterable - def __iter__(self) -> Iterator[T]: + class MyMap(Mapping[KT, VT]]): # This is a generic subclass of Mapping + def __getitem__(self, k: KT) -> VT: + ... # Implementations omitted + def __iter__(self) -> Iterator[KT]: + ... + def __len__(self) -> int: ... - input: Stream[int] # Okay + items: MyMap[str, int] # Okay - class Codes(Iterable[int]): # This is a non-generic subclass of Iterable - def __iter__(self) -> Iterator[int]: - ... + class StrDict(Dict[str, str]): # This is a non-generic subclass of Dict + def __str__(self) -> str: + return 'StrDict({})'.format(super().__str__()) - output: Codes[int] # Error! Codes is not generic + data: StrDict[int, int] # Error! StrDict is not generic + data2: StrDict # OK class Receiver(Generic[T]): def accept(self, value: T) -> None: @@ -137,15 +143,15 @@ non-generic. For example: .. note:: - You have to add an explicit ``Iterable`` (or ``Iterator``) base class - if you want mypy to consider a user-defined class as iterable (and - ``Sequence`` for sequences, etc.). This is because mypy doesn't support - *structural subtyping* and just having an ``__iter__`` method defined is - not sufficient to make mypy treat a class as iterable. + You have to add an explicit ``Mapping`` base class + if you want mypy to consider a user-defined class as a mapping (and + ``Sequence`` for sequences, etc.). This is because mypy doesn't use + *structural subtyping* for these ABCs, unlike simpler protocols + like ``Iterable``, which use :ref:`structural subtyping `. ``Generic[...]`` can be omitted from bases if there are -other base classes that include type variables, such as ``Iterable[T]`` in -the above example. If you include ``Generic[...]`` in bases, then +other base classes that include type variables, such as ``Mapping[KT, VT]`` +in the above example. If you include ``Generic[...]`` in bases, then it should list all type variables present in other bases (or more, if needed). The order of type variables is defined by the following rules: @@ -549,7 +555,9 @@ problem. This is also the reason for the ``cast()`` call in the Generic protocols ***************** -Mypy supports generic protocols (see also :ref:`protocol-types`). Generic +Mypy supports generic protocols (see also :ref:`protocol-types`). Several +:ref:`predefined protocols ` are generic, such as +``Iterable[T]``, and you can define additional generic protocols. Generic protocols mostly follow the normal rules for generic classes. Example: .. code-block:: python From 7adc68416e664ca16a8c4d9f8116d66c5e66b39e Mon Sep 17 00:00:00 2001 From: Jukka Lehtosalo Date: Tue, 12 Dec 2017 13:19:53 +0000 Subject: [PATCH 5/5] Minor rewording --- docs/source/class_basics.rst | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/source/class_basics.rst b/docs/source/class_basics.rst index 4b7f3beeeb32..b3027c193ae4 100644 --- a/docs/source/class_basics.rst +++ b/docs/source/class_basics.rst @@ -251,7 +251,7 @@ collection objects. ``Sized`` --------- -This defined for objects that support ``len(x)``. +This is a type for objects that support ``len(x)``. .. code-block:: python @@ -260,7 +260,7 @@ This defined for objects that support ``len(x)``. ``Container[T]`` ---------------- -This is defined for objects that support the ``in`` operator. +This is a type for objects that support the ``in`` operator. .. code-block:: python @@ -284,7 +284,7 @@ library function or class. ``Reversible[T]`` ----------------- -This is defined for objects that support ``reversed(x)``. +This is a type for objects that support ``reversed(x)``. .. code-block:: python @@ -293,7 +293,7 @@ This is defined for objects that support ``reversed(x)``. ``SupportsAbs[T]`` ------------------ -This is defined for objects that support ``abs(x)``. ``T`` is the type of +This is a type for objects that support ``abs(x)``. ``T`` is the type of value returned by ``abs(x)``. .. code-block:: python @@ -303,7 +303,7 @@ value returned by ``abs(x)``. ``SupportsBytes`` ----------------- -This is defined for objects that support ``bytes(x)``. +This is a type for objects that support ``bytes(x)``. .. code-block:: python @@ -312,7 +312,7 @@ This is defined for objects that support ``bytes(x)``. ``SupportsComplex`` ------------------- -This is defined for objects that support ``complex(x)``. +This is a type for objects that support ``complex(x)``. .. code-block:: python @@ -321,7 +321,7 @@ This is defined for objects that support ``complex(x)``. ``SupportsFloat`` ----------------- -This is defined for objects that support ``float(x)``. +This is a type for objects that support ``float(x)``. .. code-block:: python @@ -330,7 +330,7 @@ This is defined for objects that support ``float(x)``. ``SupportsInt`` --------------- -This is defined for objects that support ``int(x)``. +This is a type for objects that support ``int(x)``. .. code-block:: python @@ -339,7 +339,7 @@ This is defined for objects that support ``int(x)``. ``SupportsRound[T]`` -------------------- -This is defined for objects that support ``round(x)``. +This is a type for objects that support ``round(x)``. .. code-block:: python