Merge branch 'master' into sqlite-no-hasattr

Author: serhiy-storchaka

.travis.yml

@@ -40,6 +40,10 @@ matrix:
       # compiler here and the other to run the coverage build. Clang is preferred
       # in this instance for its better error messages.
       env: TESTING=cpython
+      addons:
+        apt:
+          packages:
+            - xvfb
     - os: linux
       language: python
       # Build the docs against a stable version of Python so code bugs don't hold up doc-related PRs.
@@ -70,6 +74,7 @@ matrix:
         apt:
           packages:
             - lcov
+            - xvfb
       before_script:
         - ./configure
         - make coverage -s -j4
@@ -79,7 +84,7 @@ matrix:
         - ./venv/bin/python -m test.pythoninfo
       script:
         # Skip tests that re-run the entire test suite.
-        - ./venv/bin/python -m coverage run --pylib -m test --fail-env-changed -uall,-cpu -x test_multiprocessing_fork -x test_multiprocessing_forkserver -x test_multiprocessing_spawn -x test_concurrent_futures
+        - xvfb-run ./venv/bin/python -m coverage run --pylib -m test --fail-env-changed -uall,-cpu -x test_multiprocessing_fork -x test_multiprocessing_forkserver -x test_multiprocessing_spawn -x test_concurrent_futures
       after_script:  # Probably should be after_success once test suite updated to run under coverage.py.
         # Make the `coverage` command available to Codecov w/ a version of Python that can parse all source files.
         - source ./venv/bin/activate
@@ -150,7 +155,7 @@ script:
   # Check that all symbols exported by libpython start with "Py" or "_Py"
   - make smelly
   # `-r -w` implicitly provided through `make buildbottest`.
-  - make buildbottest TESTOPTS="-j4 -uall,-cpu"
+  - if [[ "$TRAVIS_OS_NAME" == "linux" ]]; then XVFB_RUN=xvfb-run; fi; $XVFB_RUN make buildbottest TESTOPTS="-j4 -uall,-cpu"
 
 notifications:
   email: false

.github/CODE_OF_CONDUCT.rst renamed to CODE_OF_CONDUCT.rst

@@ -273,23 +273,24 @@ dictionary's keys::
 
     >>> m = {'Jan': 1, 'Feb': 2, 'Mar': 3, 'Apr': 4, 'May': 5, 'Jun': 6,
     ...      'Jul': 7, 'Aug': 8, 'Sep': 9, 'Oct': 10, 'Nov': 11, 'Dec': 12}
-    >>> for key in m:  #doctest: +SKIP
+    >>> for key in m:
     ...     print(key, m[key])
-    Mar 3
+    Jan 1
     Feb 2
-    Aug 8
-    Sep 9
+    Mar 3
     Apr 4
+    May 5
     Jun 6
     Jul 7
-    Jan 1
-    May 5
+    Aug 8
+    Sep 9
+    Oct 10
     Nov 11
     Dec 12
-    Oct 10
 
-Note that the order is essentially random, because it's based on the hash
-ordering of the objects in the dictionary.
+Note that starting with Python 3.7, dictionary iteration order is guaranteed
+to be the same as the insertion order. In earlier versions, the behaviour was
+unspecified and could vary between implementations.
 
 Applying :func:`iter` to a dictionary always loops over the keys, but
 dictionaries have methods that return other iterators.  If you want to iterate
@@ -301,8 +302,8 @@ The :func:`dict` constructor can accept an iterator that returns a finite stream
 of ``(key, value)`` tuples:
 
     >>> L = [('Italy', 'Rome'), ('France', 'Paris'), ('US', 'Washington DC')]
-    >>> dict(iter(L))  #doctest: +SKIP
-    {'Italy': 'Rome', 'US': 'Washington DC', 'France': 'Paris'}
+    >>> dict(iter(L))
+    {'Italy': 'Rome', 'France': 'Paris', 'US': 'Washington DC'}
 
 Files also support iteration by calling the :meth:`~io.TextIOBase.readline`
 method until there are no more lines in the file.  This means you can read each

Doc/howto/functional.rst

@@ -152,10 +152,22 @@ Functions and classes provided:
 
 .. function:: nullcontext(enter_result=None)
 
-   Return a context manager that returns enter_result from ``__enter__``, but
+   Return a context manager that returns *enter_result* from ``__enter__``, but
    otherwise does nothing. It is intended to be used as a stand-in for an
    optional context manager, for example::
 
+      def myfunction(arg, ignore_exceptions=False):
+          if ignore_exceptions:
+              # Use suppress to ignore all exceptions.
+              cm = contextlib.suppress(Exception)
+          else:
+              # Do not ignore any exceptions, cm has no effect.
+              cm = contextlib.nullcontext()
+          with cm:
+              # Do something
+
+   An example using *enter_result*::
+
       def process_file(file_or_path):
           if isinstance(file_or_path, str):
               # If string, open file

Doc/library/contextlib.rst

@@ -1025,6 +1025,22 @@ As we can easily check, our array is sorted now::
    1 5 7 33 99
    >>>
 
+The function factories can be used as decorator factories, so we may as well
+write::
+
+   >>> @CFUNCTYPE(c_int, POINTER(c_int), POINTER(c_int))
+   ... def py_cmp_func(a, b):
+   ...     print("py_cmp_func", a[0], b[0])
+   ...     return a[0] - b[0]
+   ...
+   >>> qsort(ia, len(ia), sizeof(c_int), py_cmp_func)
+   py_cmp_func 5 1
+   py_cmp_func 33 99
+   py_cmp_func 7 33
+   py_cmp_func 1 7
+   py_cmp_func 5 7
+   >>>
+
 .. note::
 
    Make sure you keep references to :func:`CFUNCTYPE` objects as long as they
@@ -1577,7 +1593,9 @@ Foreign functions can also be created by instantiating function prototypes.
 Function prototypes are similar to function prototypes in C; they describe a
 function (return type, argument types, calling convention) without defining an
 implementation.  The factory functions must be called with the desired result
-type and the argument types of the function.
+type and the argument types of the function, and can be used as decorator
+factories, and as such, be applied to functions through the ``@wrapper`` syntax.
+See :ref:`ctypes-callback-functions` for examples.
 
 
 .. function:: CFUNCTYPE(restype, *argtypes, use_errno=False, use_last_error=False)

Doc/library/ctypes.rst

@@ -532,7 +532,7 @@ Mutable default values
      class C:
          x = []
          def add(self, element):
-             self.x += element
+             self.x.append(element)
 
      o1 = C()
      o2 = C()

Doc/library/dataclasses.rst

@@ -246,7 +246,7 @@ in the top-level :mod:`email` package namespace.
       Removed the *strict* argument.  Added the *policy* keyword.
 
 
-.. function:: message_from_binary_file(fp, _class=None, *,
+.. function:: message_from_binary_file(fp, _class=None, *, \
                                        policy=policy.compat32)
 
    Return a message object structure tree from an open binary :term:`file

Doc/library/email.parser.rst

@@ -525,7 +525,7 @@ The following exceptions are the exceptions that are usually raised.
 
 .. exception:: ValueError
 
-   Raised when a built-in operation or function receives an argument that has the
+   Raised when an operation or function receives an argument that has the
    right type but an inappropriate value, and the situation is not described by a
    more precise exception such as :exc:`IndexError`.
 

Doc/library/exceptions.rst

@@ -241,8 +241,8 @@ are always available.  They are listed here in alphabetical order.
    interactive statement (in the latter case, expression statements that
    evaluate to something other than ``None`` will be printed).
 
-   The optional arguments *flags* and *dont_inherit* control which future
-   statements (see :pep:`236`) affect the compilation of *source*.  If neither
+   The optional arguments *flags* and *dont_inherit* control which :ref:`future
+   statements <future>` affect the compilation of *source*.  If neither
    is present (or both are zero) the code is compiled with those future
    statements that are in effect in the code that is calling :func:`compile`.  If the
    *flags* argument is given and *dont_inherit* is not (or is zero) then the

Doc/library/functions.rst

@@ -243,7 +243,7 @@ Mac OS Platform
 Unix Platforms
 --------------
 
-.. function:: libc_ver(executable=sys.executable, lib='', version='', chunksize=2048)
+.. function:: libc_ver(executable=sys.executable, lib='', version='', chunksize=16384)
 
    Tries to determine the libc version against which the file executable (defaults
    to the Python interpreter) is linked.  Returns a tuple of strings ``(lib,

Doc/library/platform.rst

@@ -346,7 +346,7 @@ Connection Objects
       called as the SQL function. If *deterministic* is true, the created function
       is marked as `deterministic <https://sqlite.org/deterministic.html>`_, which
       allows SQLite to perform additional optimizations. This flag is supported by
-      SQLite 3.8.3 or higher, ``sqlite3.NotSupportedError`` will be raised if used
+      SQLite 3.8.3 or higher, :exc:`NotSupportedError` will be raised if used
       with older versions.
 
       The function can return any of the types supported by SQLite: bytes, str, int,
@@ -835,6 +835,13 @@ Exceptions
    disconnect occurs, the data source name is not found, a transaction could
    not be processed, etc.  It is a subclass of :exc:`DatabaseError`.
 
+.. exception:: NotSupportedError
+
+   Exception raised in case a method or database API was used which is not
+   supported by the database, e.g. calling the :meth:`~Connection.rollback`
+   method on a connection that does not support transaction or has
+   transactions turned off.  It is a subclass of :exc:`DatabaseError`.
+
 
 .. _sqlite3-types:
 

Doc/library/sqlite3.rst

@@ -4220,12 +4220,17 @@ pairs within braces, for example: ``{'jack': 4098, 'sjoerd': 4127}`` or ``{4098:
 
    .. method:: popitem()
 
-      Remove and return an arbitrary ``(key, value)`` pair from the dictionary.
+      Remove and return a ``(key, value)`` pair from the dictionary.
+      Pairs are returned in :abbr:`LIFO (last-in, first-out)` order.
 
       :meth:`popitem` is useful to destructively iterate over a dictionary, as
       often used in set algorithms.  If the dictionary is empty, calling
       :meth:`popitem` raises a :exc:`KeyError`.
 
+      .. versionchanged:: 3.7
+         LIFO order is now guaranteed. In prior versions, :meth:`popitem` would
+         return an arbitrary key/value pair.
+
    .. method:: setdefault(key[, default])
 
       If *key* is in the dictionary, return its value.  If not, insert *key*

Doc/library/stdtypes.rst

@@ -1825,12 +1825,12 @@ sentinel
 
 .. data:: sentinel
 
-    The ``sentinel`` object provides a convenient way of providing unique
-    objects for your tests.
+   The ``sentinel`` object provides a convenient way of providing unique
+   objects for your tests.
 
-    Attributes are created on demand when you access them by name. Accessing
-    the same attribute will always return the same object. The objects
-    returned have a sensible repr so that test failure messages are readable.
+   Attributes are created on demand when you access them by name. Accessing
+   the same attribute will always return the same object. The objects
+   returned have a sensible repr so that test failure messages are readable.
 
    .. versionchanged:: 3.7
       The ``sentinel`` attributes now preserve their identity when they are
@@ -2070,22 +2070,22 @@ mock_open
 
 .. function:: mock_open(mock=None, read_data=None)
 
-    A helper function to create a mock to replace the use of :func:`open`. It works
-    for :func:`open` called directly or used as a context manager.
-
-    The *mock* argument is the mock object to configure. If ``None`` (the
-    default) then a :class:`MagicMock` will be created for you, with the API limited
-    to methods or attributes available on standard file handles.
-
-    *read_data* is a string for the :meth:`~io.IOBase.read`,
-    :meth:`~io.IOBase.readline`, and :meth:`~io.IOBase.readlines` methods
-    of the file handle to return.  Calls to those methods will take data from
-    *read_data* until it is depleted.  The mock of these methods is pretty
-    simplistic: every time the *mock* is called, the *read_data* is rewound to
-    the start.  If you need more control over the data that you are feeding to
-    the tested code you will need to customize this mock for yourself.  When that
-    is insufficient, one of the in-memory filesystem packages on `PyPI
-    <https://pypi.org>`_ can offer a realistic filesystem for testing.
+   A helper function to create a mock to replace the use of :func:`open`. It works
+   for :func:`open` called directly or used as a context manager.
+
+   The *mock* argument is the mock object to configure. If ``None`` (the
+   default) then a :class:`MagicMock` will be created for you, with the API limited
+   to methods or attributes available on standard file handles.
+
+   *read_data* is a string for the :meth:`~io.IOBase.read`,
+   :meth:`~io.IOBase.readline`, and :meth:`~io.IOBase.readlines` methods
+   of the file handle to return.  Calls to those methods will take data from
+   *read_data* until it is depleted.  The mock of these methods is pretty
+   simplistic: every time the *mock* is called, the *read_data* is rewound to
+   the start.  If you need more control over the data that you are feeding to
+   the tested code you will need to customize this mock for yourself.  When that
+   is insufficient, one of the in-memory filesystem packages on `PyPI
+   <https://pypi.org>`_ can offer a realistic filesystem for testing.
 
    .. versionchanged:: 3.4
       Added :meth:`~io.IOBase.readline` and :meth:`~io.IOBase.readlines` support.

Doc/library/unittest.mock.rst

@@ -41,8 +41,8 @@ printing space-separated values. There are several ways to format output.
   ::
 
      >>> yes_votes = 42_572_654 ; no_votes = 43_132_495
-     >>> percentage = (yes_votes/(yes_votes+no_votes)
-     >>> '{:-9} YES votes  {:2.2%}'.format(yes_votes, percentage))
+     >>> percentage = yes_votes/(yes_votes+no_votes)
+     >>> '{:-9} YES votes  {:2.2%}'.format(yes_votes, percentage)
      ' 42572654 YES votes  49.67%'
 
 * Finally, you can do all the string handling yourself by using string slicing and
@@ -338,7 +338,7 @@ automatically fail. ::
    >>> f.read()
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
-   ValueError: I/O operation on closed file
+   ValueError: I/O operation on closed file.
 
 
 .. _tut-filemethods:

Doc/tutorial/inputoutput.rst

@@ -1304,8 +1304,8 @@ Host name validation can be customized with
 .. note::
    The improved host name check requires a *libssl* implementation compatible
    with OpenSSL 1.0.2 or 1.1.  Consequently, OpenSSL 0.9.8 and 1.0.1 are no
-   longer supported.  The ssl module is mostly compatible with LibreSSL 2.7.2
-   and newer.
+   longer supported (see :ref:`37-platform-support-removals` for more details).
+   The ssl module is mostly compatible with LibreSSL 2.7.2 and newer.
 
 The ``ssl`` module no longer sends IP addresses in SNI TLS extension.
 (Contributed by Christian Heimes in :issue:`32185`.)
@@ -2069,10 +2069,33 @@ or higher.  (Contributed by Serhiy Storchaka in :issue:`27867`.)
 (Contributed by Antoine Pitrou in :issue:`16500`.)
 
 
+.. _37-platform-support-removals:
+
 Platform Support Removals
 =========================
 
-FreeBSD 9 and older are no longer officially supported.
+* FreeBSD 9 and older are no longer officially supported.
+* For full Unicode support, including within extension modules, \*nix platforms
+  are now expected to provide at least one of ``C.UTF-8`` (full locale),
+  ``C.utf8`` (full locale) or ``UTF-8`` (``LC_CTYPE``-only locale) as an
+  alternative to the legacy ``ASCII``-based ``C`` locale.
+* OpenSSL 0.9.8 and 1.0.1 are no longer supported, which means building CPython
+  3.7 with SSL/TLS support on older platforms still using these versions
+  requires custom build options that link to a more recent version of OpenSSL.
+
+  Notably, this issue affects the Debian 8 (aka "jessie") and Ubuntu 14.04
+  (aka "Trusty") LTS Linux distributions, as they still use OpenSSL 1.0.1 by
+  default.
+
+  Debian 9 ("stretch") and Ubuntu 16.04 ("xenial"), as well as recent releases
+  of other LTS Linux releases (e.g. RHEL/CentOS 7.5, SLES 12-SP3), use OpenSSL
+  1.0.2 or later, and remain supported in the default build configuration.
+
+  CPython's own :source:`CI configuration file <.travis.yml>` provides an
+  example of using the SSL
+  :source:`compatibility testing infrastructure <Tools/ssl/multissltests.py>` in
+  CPython's test suite to build and link against OpenSSL 1.1.0 rather than an
+  outdated system provided OpenSSL.
 
 
 API and Feature Removals

Doc/whatsnew/3.7.rst

@@ -127,6 +127,10 @@ Optimizations
   first introduced in Python 3.4.  It offers better performance and smaller
   size compared to Protocol 3 available since Python 3.0.
 
+* Removed one ``Py_ssize_t`` member from ``PyGC_Head``.  All GC tracked
+  objects (e.g. tuple, list, dict) size is reduced 4 or 8 bytes.
+  (Contributed by Inada Naoki in :issue:`33597`)
+
 
 Build and C API Changes
 =======================
@@ -135,6 +139,21 @@ Build and C API Changes
   ``const char *`` rather of ``char *``.
   (Contributed by Serhiy Storchaka in :issue:`33818`.)
 
+* The duality of ``Modules/Setup.dist`` and ``Modules/Setup`` has been
+  removed.  Previously, when updating the CPython source tree, one had
+  to manually copy ``Modules/Setup.dist`` (inside the source tree) to
+  ``Modules/Setup`` (inside the build tree) in order to reflect any changes
+  upstream.  This was of a small benefit to packagers at the expense of
+  a frequent annoyance to developers following CPython development, as
+  forgetting to copy the file could produce build failures.
+
+  Now the build system always reads from ``Modules/Setup`` inside the source
+  tree.  People who want to customize that file are encouraged to maintain
+  their changes in a git fork of CPython or as patch files, as they would do
+  for any other change to the source tree.
+
+  (Contributed by Antoine Pitrou in :issue:`32430`.)
+
 
 Deprecated
 ==========
@@ -201,6 +220,10 @@ Changes in the Python API
 * :func:`shutil.copyfile` default buffer size on Windows was changed from
   16 KiB to 1 MiB.
 
+* ``PyGC_Head`` struct is changed completely.  All code touched the
+  struct member should be rewritten.  (See :issue:`33597`)
+
+
 CPython bytecode changes
 ------------------------