PEP 522: BlockingIOError in security sensitive APIs on Linux

pep-0522.txt

@@ -0,0 +1,244 @@
+PEP: 522
+Title: Raise BlockingIOError in security sensitive APIs on Linux
+Version: $Revision$
+Last-Modified: $Date$
+Author: Nick Coghlan <[email protected]>
+Status: Draft
+Type: Standards Track
+Content-Type: text/x-rst
+Created: 16 June 2016
+Python-Version: 3.6
+
+
+Abstract
+========
+
+On Linux systems, the documentation for ``os.urandom`` currently makes the
+following contradictory promises:
+
+* to provide random numbers that are suitable for security sensitive
+  operations (such as client authentication and cryptography)
+* to provide access to the best available randomness source provided by
+  the underlying operating system
+* to present a relatively thin wrapper around the system ``/dev/urandom``
+  device
+
+This PEP proposes that in Python 3.6+ the 3rd guarantee be dropped in order to
+preserve the first two: on Linux systems that provide the ``getrandom()``
+syscall, ``os.urandom()`` would become a wrapper around that API, and raise
+``BlockingIOError`` in cases where directly accessing ``/dev/urandom/`` would
+instead return random data that may not be adequately unpredictable for use in
+security sensitive operations.
+
+As higher level abstractions over the lower level ``os.urandom()`` API, both
+``random.SystemRandom()`` and the ``secrets`` would also be documented as
+potentially raising ``BlockingIOError``.
+
+In all cases, as soon as a call to ``os.urandom()`` succeeds, all future
+calls to ``os.urandom()`` in that process will succeed (once the operating
+system random number generator is ready after system boot, it remains ready).
+
+
+Proposal
+========
+
+This PEP proposes that in Python 3.6+, ``os.urandom()`` be updated to call
+the new Linux ``getrandom()``` syscall in non-blocking mode if available and
+raise ``BlockingIOError: system random number generator is not ready`` if
+the kernel reports that the call would block.
+
+No changes are proposed for Windows or Mac OS X systems, as neither of those
+platforms provides any mechanism to run Python code before the operating
+system random number generator has been initialised. Mac OS X goes so far as
+to kernel panic and abort the boot process if it can't properly initialise the
+random number generator (although Apple's restrictions on the supported
+hardware platforms make that exceedingly unlikely in practice).
+
+Other \*nix systems that offer a non-blocking API for requesting random numbers
+suitable for use in security sensitive applications could potentially receive
+a similar update, but such changes are out of scope for this particular
+proposal.
+
+
+Rationale
+=========
+
+For several years now, the security community's guidance has been to use
+``os.urandom()`` (or the ``random.SystemRandom()`` wrapper) when implementing
+security sensitive operations in Python.
+
+To help improve API discoverability and make it clearer that secrecy and
+simulation are not the same problem (even though they both involve
+random numbers), PEP 506 collected several of the one line recipes based
+on the lower level ``os.urandom()`` API into a new ``secrets`` module.
+
+However, this guidance has also come with a longstanding caveat: developers
+writing security sensitive software at least for Linux, and potentially for
+some other \*BSD systems, may need to wait until the operating system's
+random number generator is ready before relying on it for security sensitive
+operations.
+
+Unfortunately, there's currently no clear indicator to developers that their
+software may not be working as expected when run early in the Linux boot
+process, or on hardware without good sources of entropy to seed the operating
+system's random number generator: due to the behaviour of the underlying
+``/dev/urandom`` device, ``os.urandom()`` on Linux returns a result either way,
+and it takes extensive statistical analysis to show that a security
+vulnerability exists.
+
+By contrast, if ``BlockingIOError`` is raised in those situations, then
+developers can easily choose their desired behaviour:
+
+1. Loop until the call succeeds (security sensitive)
+2. Switch to using the random module (non-security sensitive)
+3. Switch to reading ``/dev/urandom`` directly (non-security sensitive)
+
+
+Why now?
+--------
+
+The main reason is because the 3.5 SipHash initialisation bug causing a deadlock
+when attempting to run Python scripts during the Linux init process resulted in
+a rash of proposals to add *new* APIs like ``getrandom()``, ``urandom_block()``,
+``pseudorandom()`` and ``cryptorandom()`` to the ``os`` module and to start
+trying to educate users on when they should call those APIs instead of
+``os.urandom()``.
+
+This is a *really* obscure problem, and we definitely shouldn't clutter up the
+standard library with new APIs without a compelling reason, especially with the
+``secrets`` module already being added as the "use this and don't worry about
+the low level details" for developers that don't need to worry about versions
+prior to Python 3.6.
+
+However, it's also the case that low cost ARM devices are becoming increasingly
+prevalent, with a lot of them running Linux, and a lot of folks writing
+Python applications that run on those devices. That creates an opportunity to
+take an obscure security problem that requires a lot of knowledge about
+Linux boot processes and secure random number generation and turn it into a
+relatively mundane and easy-to-find-in-an-internet-search runtime exception.
+
+
+Background
+==========
+
+On operating systems other than Linux, ``os.urandom()`` may already block
+waiting for the operating system's random number generator to be ready.
+
+On Linux, even when the operating system's random number generator doesn't
+consider itself ready for use in security sensitive operations, it will return
+random values based on the entropy it as available.
+
+This behaviour is potentially problematic, so Linux 3.17 added a new
+``getrandom()`` syscall that (amongst other benefits) allows callers to
+either block waiting for the random number generator to be ready, or
+else request an error return if the random number generator is not ready.
+Notably, the new API does *not* support the old behaviour of returning
+data that is not suitable for security sensitive use cases.
+
+Versions of Python prior up to and including Python 3.4 access the
+Linux ``/dev/urandom`` device directly.
+
+Python 3.5.0 and 3.5.1 called ``getrandom()`` in blocking mode in order to
+avoid the use of a file descriptor to access ``/dev/urandom``. While there
+were no specific problems reported due to ``os.urandom()`` blocking in user
+code, there *were* problems due to CPython implicitly invoking the blocking
+behaviour during interpreter startup.
+
+Rather than trying to decouple SipHash initialisation from the
+``os.urandom()`` implementation, Python 3.5.2 switched to calling
+``getrandom()`` in non-blocking mode, and falling back to reading from
+``/dev/urandom`` if the syscall indicates it will block.
+
+
+Backwards Compatibility Impact Assessment
+=========================================
+
+Similar to PEP 476, this is a proposal to turn a previously silent security
+failure into a noisy exception that requires the application developer to
+make an explicit decision regarding the behaviour they desire.
+
+As no changes are proposed for operating systems other than Linux,
+``os.urandom()`` retains its existing behaviour as a nominally blocking API
+that is non-blocking in practice due to the difficulty of scheduling Python
+code to run before the operating system random number generator is ready. We
+believe it may be possible on \*BSD, but nobody has explicitly demonstrated
+that. On Mac OS X and Windows, it appears to be straight up impossible to
+even try to run a Python interpreter that early in the boot process.
+
+On Linux, ``os.urandom()`` retains its status as a guaranteed non-blocking API.
+However, the means of achieving that status changes in the specific case of
+the operating system random number generator not being ready for use in security
+sensitive operations: historically it would return potentially predictable
+random data, with this PEP it would change to raise ``BlockingIOError``.
+
+Developers of affected applications would then be required to make one of the
+following changes to forward compatibility with Python 3.6, based on the kind
+of application they're developing.
+
+
+Unaffected Applications
+-----------------------
+
+The following kinds of applications would be entirely unaffected by the change,
+regardless of whether or not they perform security sensitive operations:
+
+- applications that don't support Linux
+- applications that are only run on desktops or conventional servers
+- applications that are only run after the system RNG is ready
+
+
+Affected security sensitive applications
+----------------------------------------
+
+Security sensitive applications would need to either change their system
+configuration so the application is only started after the operating system
+random number generator is ready for security sensitive operations, or else
+change their code to busy loop until the operating system is ready::
+
+    def blocking_urandom(num_bytes):
+        while True:
+            try:
+                return os.urandom(num_bytes)
+            except BlockingIOError:
+                pass
+
+
+Affected Linux specific non-security sensitive applications
+-----------------------------------------------------------
+
+Non-security sensitive applications that don't need to worry about cross
+platform compatibility can be updated to access ``/dev/urandom`` directly::
+
+    def dev_urandom(num_bytes):
+        with open("/dev/urandom", "rb") as f:
+            return f.read(num_bytes)
+
+
+Affected portable non-security sensitive applications
+-----------------------------------------------------
+
+Non-security sensitive applications that don't want to assume access to
+``/dev/urandom`` can be updated to use the ``random`` module instead::
+
+    def pseudorandom(num_bytes):
+        random.getrandbits(num_bytes*8).to_bytes(num_bytes, "little")
+
+
+References
+==========
+
+* Victor's summary: http://haypo-notes.readthedocs.io/pep_random.html
+
+Copyright
+=========
+
+This document has been placed into the public domain.
+
+
+..
+   Local Variables:
+   mode: indented-text
+   indent-tabs-mode: nil
+   sentence-end-double-space: t
+   fill-column: 70
+   coding: utf-8