gh-119180: More documentation for PEP 649/749 (#133552)

The SC asked that the Appendix in PEP-749 be added to the docs.

Co-authored-by: Adam Turner <[email protected]>

Author: JelleZijlstra

Doc/library/annotationlib.rst

@@ -485,3 +485,117 @@ annotations from the class and puts them in a separate attribute:
          typ.classvars = classvars  # Store the ClassVars in a separate attribute
          return typ
 
+
+Limitations of the ``STRING`` format
+------------------------------------
+
+The :attr:`~Format.STRING` format is meant to approximate the source code
+of the annotation, but the implementation strategy used means that it is not
+always possible to recover the exact source code.
+
+First, the stringifier of course cannot recover any information that is not present in
+the compiled code, including comments, whitespace, parenthesization, and operations that
+get simplified by the compiler.
+
+Second, the stringifier can intercept almost all operations that involve names looked
+up in some scope, but it cannot intercept operations that operate fully on constants.
+As a corollary, this also means it is not safe to request the ``STRING`` format on
+untrusted code: Python is powerful enough that it is possible to achieve arbitrary
+code execution even with no access to any globals or builtins. For example:
+
+.. code-block:: pycon
+
+  >>> def f(x: (1).__class__.__base__.__subclasses__()[-1].__init__.__builtins__["print"]("Hello world")): pass
+  ...
+  >>> annotationlib.get_annotations(f, format=annotationlib.Format.SOURCE)
+  Hello world
+  {'x': 'None'}
+
+.. note::
+   This particular example works as of the time of writing, but it relies on
+   implementation details and is not guaranteed to work in the future.
+
+Among the different kinds of expressions that exist in Python,
+as represented by the :mod:`ast` module, some expressions are supported,
+meaning that the ``STRING`` format can generally recover the original source code;
+others are unsupported, meaning that they may result in incorrect output or an error.
+
+The following are supported (sometimes with caveats):
+
+* :class:`ast.BinOp`
+* :class:`ast.UnaryOp`
+
+  * :class:`ast.Invert` (``~``), :class:`ast.UAdd` (``+``), and :class:`ast.USub` (``-``) are supported
+  * :class:`ast.Not` (``not``) is not supported
+
+* :class:`ast.Dict` (except when using ``**`` unpacking)
+* :class:`ast.Set`
+* :class:`ast.Compare`
+
+  * :class:`ast.Eq` and :class:`ast.NotEq` are supported
+  * :class:`ast.Lt`, :class:`ast.LtE`, :class:`ast.Gt`, and :class:`ast.GtE` are supported, but the operand may be flipped
+  * :class:`ast.Is`, :class:`ast.IsNot`, :class:`ast.In`, and :class:`ast.NotIn` are not supported
+
+* :class:`ast.Call` (except when using ``**`` unpacking)
+* :class:`ast.Constant` (though not the exact representation of the constant; for example, escape
+  sequences in strings are lost; hexadecimal numbers are converted to decimal)
+* :class:`ast.Attribute` (assuming the value is not a constant)
+* :class:`ast.Subscript` (assuming the value is not a constant)
+* :class:`ast.Starred` (``*`` unpacking)
+* :class:`ast.Name`
+* :class:`ast.List`
+* :class:`ast.Tuple`
+* :class:`ast.Slice`
+
+The following are unsupported, but throw an informative error when encountered by the
+stringifier:
+
+* :class:`ast.FormattedValue` (f-strings; error is not detected if conversion specifiers like ``!r``
+  are used)
+* :class:`ast.JoinedStr` (f-strings)
+
+The following are unsupported and result in incorrect output:
+
+* :class:`ast.BoolOp` (``and`` and ``or``)
+* :class:`ast.IfExp`
+* :class:`ast.Lambda`
+* :class:`ast.ListComp`
+* :class:`ast.SetComp`
+* :class:`ast.DictComp`
+* :class:`ast.GeneratorExp`
+
+The following are disallowed in annotation scopes and therefore not relevant:
+
+* :class:`ast.NamedExpr` (``:=``)
+* :class:`ast.Await`
+* :class:`ast.Yield`
+* :class:`ast.YieldFrom`
+
+
+Limitations of the ``FORWARDREF`` format
+----------------------------------------
+
+The :attr:`~Format.FORWARDREF` format aims to produce real values as much
+as possible, with anything that cannot be resolved replaced with
+:class:`ForwardRef` objects. It is affected by broadly the same Limitations
+as the :attr:`~Format.STRING` format: annotations that perform operations on
+literals or that use unsupported expression types may raise exceptions when
+evaluated using the :attr:`~Format.FORWARDREF` format.
+
+Below are a few examples of the behavior with unsupported expressions:
+
+.. code-block:: pycon
+
+   >>> from annotationlib import get_annotations, Format
+   >>> def zerodiv(x: 1 / 0): ...
+   >>> get_annotations(zerodiv, format=Format.STRING)
+   Traceback (most recent call last):
+     ...
+   ZeroDivisionError: division by zero
+   >>> get_annotations(zerodiv, format=Format.FORWARDREF)
+   Traceback (most recent call last):
+     ...
+   ZeroDivisionError: division by zero
+   >>> def ifexp(x: 1 if y else 0): ...
+   >>> get_annotations(ifexp, format=Format.STRING)
+   {'x': '1'}

Doc/reference/compound_stmts.rst

@@ -1885,7 +1885,7 @@ expressions. The presence of annotations does not change the runtime semantics o
 the code, except if some mechanism is used that introspects and uses the annotations
 (such as :mod:`dataclasses` or :func:`functools.singledispatch`).
 
-By default, annotations are lazily evaluated in a :ref:`annotation scope <annotation-scopes>`.
+By default, annotations are lazily evaluated in an :ref:`annotation scope <annotation-scopes>`.
 This means that they are not evaluated when the code containing the annotation is evaluated.
 Instead, the interpreter saves information that can be used to evaluate the annotation later
 if requested. The :mod:`annotationlib` module provides tools for evaluating annotations.
@@ -1898,6 +1898,12 @@ all annotations are instead stored as strings::
    >>> f.__annotations__
    {'param': 'annotation'}
 
+This future statement will be deprecated and removed in a future version of Python,
+but not before Python 3.13 reaches its end of life (see :pep:`749`).
+When it is used, introspection tools like
+:func:`annotationlib.get_annotations` and :func:`typing.get_type_hints` are
+less likely to be able to resolve annotations at runtime.
+
 
 .. rubric:: Footnotes
 

Doc/whatsnew/3.14.rst

@@ -82,7 +82,7 @@ and improvements in user-friendliness and correctness.
 
 .. PEP-sized items next.
 
-* :ref:`PEP 649: deferred evaluation of annotations <whatsnew314-pep649>`
+* :ref:`PEP 649 and 749: deferred evaluation of annotations <whatsnew314-pep649>`
 * :ref:`PEP 741: Python Configuration C API <whatsnew314-pep741>`
 * :ref:`PEP 750: Template strings <whatsnew314-pep750>`
 * :ref:`PEP 758: Allow except and except* expressions without parentheses <whatsnew314-pep758>`
@@ -362,18 +362,19 @@ Check :pep:`758` for more details.
 
 .. _whatsnew314-pep649:
 
-PEP 649: deferred evaluation of annotations
--------------------------------------------
+PEP 649 and 749: deferred evaluation of annotations
+---------------------------------------------------
 
 The :term:`annotations <annotation>` on functions, classes, and modules are no
 longer evaluated eagerly. Instead, annotations are stored in special-purpose
 :term:`annotate functions <annotate function>` and evaluated only when
-necessary. This is specified in :pep:`649` and :pep:`749`.
+necessary (except if ``from __future__ import annotations`` is used).
+This is specified in :pep:`649` and :pep:`749`.
 
 This change is designed to make annotations in Python more performant and more
 usable in most circumstances. The runtime cost for defining annotations is
 minimized, but it remains possible to introspect annotations at runtime.
-It is usually no longer necessary to enclose annotations in strings if they
+It is no longer necessary to enclose annotations in strings if they
 contain forward references.
 
 The new :mod:`annotationlib` module provides tools for inspecting deferred
@@ -409,7 +410,8 @@ writing annotations the same way you did with previous versions of Python.
 You will likely be able to remove quoted strings in annotations, which are frequently
 used for forward references. Similarly, if you use ``from __future__ import annotations``
 to avoid having to write strings in annotations, you may well be able to
-remove that import. However, if you rely on third-party libraries that read annotations,
+remove that import once you support only Python 3.14 and newer.
+However, if you rely on third-party libraries that read annotations,
 those libraries may need changes to support unquoted annotations before they
 work as expected.
 
@@ -422,6 +424,11 @@ annotations. For example, you may want to use :func:`annotationlib.get_annotatio
 with the :attr:`~annotationlib.Format.FORWARDREF` format, as the :mod:`dataclasses`
 module now does.
 
+The external :pypi:`typing_extensions` package provides partial backports of some of the
+functionality of the :mod:`annotationlib` module, such as the :class:`~annotationlib.Format`
+enum and the :func:`~annotationlib.get_annotations` function. These can be used to
+write cross-version code that takes advantage of the new behavior in Python 3.14.
+
 Related changes
 ^^^^^^^^^^^^^^^
 
@@ -433,6 +440,10 @@ functions in the standard library, there are many ways in which your code may
 not work in Python 3.14. To safeguard your code against future changes,
 use only the documented functionality of the :mod:`annotationlib` module.
 
+In particular, do not read annotations directly from the namespace dictionary
+attribute of type objects. Use :func:`annotationlib.get_annotate_from_class_namespace`
+during class construction and :func:`annotationlib.get_annotations` afterwards.
+
 ``from __future__ import annotations``
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -444,8 +455,10 @@ Python without deferred evaluation of annotations, reaches its end of life in 20
 In Python 3.14, the behavior of code using ``from __future__ import annotations``
 is unchanged.
 
+(Contributed by Jelle Zijlstra in :gh:`119180`; :pep:`649` was written by Larry Hastings.)
+
 .. seealso::
-   :pep:`649`.
+   :pep:`649` and :pep:`749`.
 
 
 Improved error messages