Skip to content

PEP 727: Review #3316

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 14 commits into from
Sep 8, 2023
Merged

PEP 727: Review #3316

Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
788eea8
Strip thread name
AA-Turner Aug 31, 2023
bc12352
Link to Python docs
AA-Turner Aug 31, 2023
599733a
Reference PEP 484
AA-Turner Aug 31, 2023
9efc614
Fix heading levels
AA-Turner Aug 31, 2023
aac18fe
Add missing sections (as a comment)
AA-Turner Aug 31, 2023
57e8dcc
Rename EAaOPV to Reference Implementation
AA-Turner Aug 31, 2023
4d72db4
Code block formatting
AA-Turner Aug 31, 2023
75d66e0
Remove leading elipses
AA-Turner Aug 31, 2023
014a316
Tighten usage criteria
AA-Turner Aug 31, 2023
492a73d
Remove non-specification content
AA-Turner Aug 31, 2023
ee92cae
Rewrite the Specification
AA-Turner Aug 31, 2023
b45c5ac
Simplify Examples and move DocInfo to the reference implementation
AA-Turner Aug 31, 2023
69e1baa
Improve the Reference Implementation
AA-Turner Aug 31, 2023
5905f98
Syntax
AA-Turner Aug 31, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
242 changes: 119 additions & 123 deletions pep-0727.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,22 +2,22 @@ PEP: 727
Title: Documentation Metadata in Typing
Author: Sebastián Ramírez <[email protected]>
Sponsor: Jelle Zijlstra <[email protected]>
Discussions-To: https://discuss.python.org/t/pep-727-documentation-metadata-in-typing/32566
Discussions-To: https://discuss.python.org/t/32566
Status: Draft
Type: Standards Track
Topic: Typing
Content-Type: text/x-rst
Created: 28-Aug-2023
Python-Version: 3.13
Post-History: `30-Aug-2023 <https://discuss.python.org/t/pep-727-documentation-metadata-in-typing/32566>`__
Post-History: `30-Aug-2023 <https://discuss.python.org/t/32566>`__


Abstract
========

This document proposes a way to complement docstrings to add additional documentation
to Python symbols using type annotations with ``Annotated`` (in class attributes,
function and method parameters, return values, and variables).
to Python symbols using type annotations with :py:class:`~typing.Annotated`
(in class attributes, function and method parameters, return values, and variables).


Motivation
Expand Down Expand Up @@ -60,7 +60,7 @@ documentation in some other way (e.g. an API, a CLI, etc).
Some of these previous formats tried to account for the lack of type annotations
in older Python versions by including typing information in the docstrings,
but now that information doesn't need to be in docstrings as there is now an official
syntax for type annotations.
:pep:`syntax for type annotations <484>`.


Rationale
Expand All @@ -84,79 +84,56 @@ like to adopt it.
Specification
=============

The main proposal is to introduce a new function, ``typing.doc()``,
to be used when documenting Python objects.
This function MUST only be used within :py:class:`~typing.Annotated` annotations.
The function takes a single string argument, ``documentation``,
and returns an instance of ``typing.DocInfo``,
which stores the input string unchanged.

``typing.doc``
--------------
Any tool processing ``typing.DocInfo`` objects SHOULD interpret the string as
a docstring, and therefore SHOULD normalize whitespace
as if ``inspect.cleandoc()`` were used.

The main proposal is to have a new function ``doc()`` in the ``typing`` module.
Even though this is not strictly related to the type annotations, it's expected
to go in ``Annotated`` type annotations, and to interact with type annotations.
The string passed to ``typing.doc()`` SHOULD be of the form that would be a valid docstring.
This means that `f-strings`__ and string operations SHOULD NOT be used.
As this cannot be enforced by the Python runtime,
tools SHOULD NOT rely on this behaviour,
and SHOULD exit with an error if such a prohibited string is encountered.

There's also the particular benefit that it could be implemented in the
``typing_extensions`` package to have support for older versions of Python and
early adopters of this proposal.

This ``doc()`` function would receive one single parameter ``documentation`` with
a documentation string.

This string could be a multi-line string, in which case, when extracted by tools,
should be interpreted cleaning up indentation as if using ``inspect.cleandoc()``,
the same procedure used for docstrings.

This string could probably contain markup, like Markdown or reST. As that could
be highly debated, that decision is left for a future proposal, to focus here
on the main functionality.

This specification targets static analysis tools and editors, and as such, the
value passed to ``doc()`` should allow static evaluation and analysis. If a
developer passes as the value something that requires runtime execution
(e.g. a function call) the behavior of static analysis tools is unspecified
and they could omit it from their process and results. For static analysis
tools to be conformant with this specification they need only to support
statically accessible values.

An example documenting the attributes of a class, or in this case, the keys
of a ``TypedDict``, could look like this:

.. code-block::

from typing import Annotated, TypedDict, NotRequired, doc


class User(TypedDict):
firstname: Annotated[str, doc("The user's first name")]
lastname: Annotated[str, doc("The user's last name")]
__ https://docs.python.org/3/reference/lexical_analysis.html#formatted-string-literals

Examples
--------

An example documenting the parameters of a function could look like this:
Class attributes may be documented:

.. code-block::
.. code:: python

from typing import Annotated, doc
from typing import Annotated, doc

class User:
first_name: Annotated[str, doc("The user's first name")]
last_name: Annotated[str, doc("The user's last name")]

def create_user(
lastname: Annotated[str, doc("The **last name** of the newly created user")],
firstname: Annotated[str | None, doc("The user's **first name**")] = None,
) -> Annotated[User, doc("The created user after saving in the database")]:
"""
Create a new user in the system, it needs the database connection to be already
initialized.
"""
pass
...

As can function or method parameters:

The return of the ``doc()`` function is an instance of a class that can be checked
and used at runtime, defined similar to:
.. code:: python

.. code-block::
from typing import Annotated, doc

class DocInfo:
def __init__(self, documentation: str):
self.documentation = documentation
def create_user(
first_name: Annotated[str, doc("The user's first name")],
last_name: Annotated[str, doc("The user's last name")],
cursor: DatabaseConnection | None = None,
) -> Annotated[User, doc("The created user after saving in the database")]:
"""Create a new user in the system.

...where the attribute ``documentation`` contains the same value string passed to
the function ``doc()``.
It needs the database connection to be already initialized.
"""
pass


Additional Scenarios
Expand All @@ -171,52 +148,48 @@ but implementers are not required to support them.


Type Alias
----------
''''''''''

When creating a type alias, like:

.. code-block::
.. code:: python

Username = Annotated[str, doc("The name of a user in the system")]
Username = Annotated[str, doc("The name of a user in the system")]


...the documentation would be considered to be carried by the parameter annotated
The documentation would be considered to be carried by the parameter annotated
with ``Username``.

So, in a function like:

.. code-block::
.. code:: python

def hi(
to: Username,
) -> None: ...
def hi(to: Username) -> None: ...


...it would be equivalent to:
It would be equivalent to:

.. code-block::
.. code:: python

def hi(
to: Annotated[str, doc("The name of a user in the system")],
) -> None: ...
def hi(to: Annotated[str, doc("The name of a user in the system")]) -> None: ...

Nevertheless, implementers would not be required to support type aliases outside
of the final type annotation to be conformant with this specification, as it
could require more complex dereferencing logic.


Annotating Type Parameters
--------------------------
''''''''''''''''''''''''''

When annotating type parameters, as in:

.. code-block::
.. code:: python

def hi(
to: list[Annotated[str, doc("The name of a user in a list")]],
) -> None: ...
def hi(
to: list[Annotated[str, doc("The name of a user in a list")]],
) -> None: ...

...the documentation in ``doc()`` would refer to what it is annotating, in this
The documentation in ``doc()`` would refer to what it is annotating, in this
case, each item in the list, not the list itself.

There are currently no practical use cases for documenting type parameters,
Expand All @@ -225,17 +198,17 @@ conformant, but it's included for completeness.


Annotating Unions
-----------------
'''''''''''''''''

If used in one of the parameters of a union, as in:

.. code-block::
.. code:: python

def hi(
to: str | Annotated[list[str], doc("List of user names")],
) -> None: ...
def hi(
to: str | Annotated[list[str], doc("List of user names")],
) -> None: ...

...again, the documentation in ``doc()`` would refer to what it is annotating,
Again, the documentation in ``doc()`` would refer to what it is annotating,
in this case, this documents the list itself, not its items.

In particular, the documentation would not refer to a single string passed as a
Expand All @@ -247,22 +220,23 @@ included for completeness.


Nested ``Annotated``
--------------------
''''''''''''''''''''

Continuing with the same idea above, if ``Annotated`` was used nested and used
multiple times in the same parameter, ``doc()`` would refer to the type it
is annotating.

So, in an example like:

.. code-block::
.. code:: python

def hi(
to: Annotated[
Annotated[str, doc("A user name")] | Annotated[list, doc("A list of user names")],
doc("Who to say hi to"),
],
) -> None: ...
def hi(
to: Annotated[
Annotated[str, doc("A user name")]
| Annotated[list, doc("A list of user names")],
doc("Who to say hi to"),
],
) -> None: ...


The documentation for the whole parameter ``to`` would be considered to be
Expand All @@ -281,16 +255,16 @@ of the parameter passed is of one type or another, but they are not required to


Duplication
-----------
'''''''''''

If ``doc()`` is used multiple times in a single ``Annotated``, it would be
considered invalid usage from the developer, for example:

.. code-block::
.. code:: python

def hi(
to: Annotated[str, doc("A user name"), doc("The current user name")],
) -> None: ...
def hi(
to: Annotated[str, doc("A user name"), doc("The current user name")],
) -> None: ...


Implementers can consider this invalid and are not required to support this to be
Expand All @@ -302,46 +276,68 @@ can opt to support one of the ``doc()`` declarations.
In that case, the suggestion would be to support the last one, just because
this would support overriding, for example, in:

.. code-block::
.. code:: python

User = Annotated[str, doc("A user name")]
User = Annotated[str, doc("A user name")]

CurrentUser = Annotated[User, doc("The current user name")]
CurrentUser = Annotated[User, doc("The current user name")]


Internally, in Python, ``CurrentUser`` here is equivalent to:

.. code-block::
.. code:: python

CurrentUser = Annotated[str, doc("A user name"), doc("The current user name")]
CurrentUser = Annotated[str,
doc("A user name"),
doc("The current user name")]


For an implementation that supports the last ``doc()`` appearance, the above
example would be equivalent to:

.. code-block::
.. code:: python

def hi(to: Annotated[str, doc("The current user name")]) -> None: ...


.. you need to fill these in:

Backwards Compatibility
=======================

[Describe potential impact and severity on pre-existing code.]


Security Implications
=====================

[How could a malicious user take advantage of this new feature?]


How to Teach This
=================

[How to teach users, new and experienced, how to apply the PEP to their work.]

def hi(
to: Annotated[str, doc("The current user name")],
) -> None: ...

Reference Implementation
========================

Early Adopters and Older Python Versions
========================================
``typing.doc`` and ``typing.DocInfo`` are implemented as follows:

For older versions of Python and early adopters of this proposal, ``doc()`` and
``DocInfo`` can be imported from the ``typing_extensions`` package.
.. code:: python

.. code-block::
def doc(documentation: str, /) -> DocInfo:
return DocInfo(documentation)

from typing import Annotated
class DocInfo:
def __init__(self, documentation: str, /):
self.documentation = documentation

from typing_extensions import doc

These have been implemented in the `typing_extensions`__ package.

def hi(
to: Annotated[str, doc("The current user name")],
) -> None: ...
__ https://pypi.org/project/typing-extensions/


Rejected Ideas
Expand Down Expand Up @@ -407,8 +403,8 @@ to be used by those that are willing to take the extra verbosity in exchange
for the benefits.


Doc is not Typing
-----------------
Documentation is not Typing
---------------------------

It could also be argued that documentation is not really part of typing, or that
it should live in a different module. Or that this information should not be part
Expand Down