pythongh-126925: Modify how iOS test results are gathered (python#127592)

Adds a `use_system_log` config item to enable stdout/stderr redirection for
Apple platforms. This log streaming is then used by a new iOS test runner
script, allowing the display of test suite output at runtime. The iOS test
runner script can be used by any Python project, not just the CPython test
suite.

Author: freakboy3742

Doc/c-api/init_config.rst

@@ -1279,6 +1279,15 @@ PyConfig
 
       Default: ``1`` in Python config and ``0`` in isolated config.
 
+   .. c:member:: int use_system_logger
+
+      If non-zero, ``stdout`` and ``stderr`` will be redirected to the system
+      log.
+
+      Only available on macOS 10.12 and later, and on iOS.
+
+      Default: ``0`` (don't use system log).
+
    .. c:member:: int user_site_directory
 
       If non-zero, add the user site directory to :data:`sys.path`.

Doc/using/ios.rst

@@ -292,10 +292,12 @@ To add Python to an iOS Xcode project:
 10. Add Objective C code to initialize and use a Python interpreter in embedded
     mode. You should ensure that:
 
-   * :c:member:`UTF-8 mode <PyPreConfig.utf8_mode>` is *enabled*;
-   * :c:member:`Buffered stdio <PyConfig.buffered_stdio>` is *disabled*;
-   * :c:member:`Writing bytecode <PyConfig.write_bytecode>` is *disabled*;
-   * :c:member:`Signal handlers <PyConfig.install_signal_handlers>` are *enabled*;
+   * UTF-8 mode (:c:member:`PyPreConfig.utf8_mode`) is *enabled*;
+   * Buffered stdio (:c:member:`PyConfig.buffered_stdio`) is *disabled*;
+   * Writing bytecode (:c:member:`PyConfig.write_bytecode`) is *disabled*;
+   * Signal handlers (:c:member:`PyConfig.install_signal_handlers`) are *enabled*;
+   * System logging (:c:member:`PyConfig.use_system_logger`) is *enabled*
+     (optional, but strongly recommended);
    * ``PYTHONHOME`` for the interpreter is configured to point at the
      ``python`` subfolder of your app's bundle; and
    * The ``PYTHONPATH`` for the interpreter includes:
@@ -324,6 +326,49 @@ modules in your app, some additional steps will be required:
 * If you're using a separate folder for third-party packages, ensure that folder
   is included as part of the ``PYTHONPATH`` configuration in step 10.
 
+Testing a Python package
+------------------------
+
+The CPython source tree contains :source:`a testbed project <iOS/testbed>` that
+is used to run the CPython test suite on the iOS simulator. This testbed can also
+be used as a testbed project for running your Python library's test suite on iOS.
+
+After building or obtaining an iOS XCFramework (See :source:`iOS/README.rst`
+for details), create a clone of the Python iOS testbed project by running:
+
+.. code-block:: bash
+
+    $ python iOS/testbed clone --framework <path/to/Python.xcframework> --app <path/to/module1> --app <path/to/module2> app-testbed
+
+You will need to modify the ``iOS/testbed`` reference to point to that
+directory in the CPython source tree; any folders specified with the ``--app``
+flag will be copied into the cloned testbed project. The resulting testbed will
+be created in the ``app-testbed`` folder. In this example, the ``module1`` and
+``module2`` would be importable modules at runtime. If your project has
+additional dependencies, they can be installed into the
+``app-testbed/iOSTestbed/app_packages`` folder (using ``pip install --target
+app-testbed/iOSTestbed/app_packages`` or similar).
+
+You can then use the ``app-testbed`` folder to run the test suite for your app,
+For example, if ``module1.tests`` was the entry point to your test suite, you
+could run:
+
+.. code-block:: bash
+
+    $ python app-testbed run -- module1.tests
+
+This is the equivalent of running ``python -m module1.tests`` on a desktop
+Python build. Any arguments after the ``--`` will be passed to the testbed as
+if they were arguments to ``python -m`` on a desktop machine.
+
+You can also open the testbed project in Xcode by running:
+
+.. code-block:: bash
+
+    $ open app-testbed/iOSTestbed.xcodeproj
+
+This will allow you to use the full Xcode suite of tools for debugging.
+
 App Store Compliance
 ====================
 

Doc/whatsnew/3.14.rst

@@ -205,6 +205,13 @@ Other language changes
   making it a :term:`generic type`.
   (Contributed by Brian Schubert in :gh:`126012`.)
 
+* iOS and macOS apps can now be configured to redirect ``stdout`` and
+  ``stderr`` content to the system log. (Contributed by Russell Keith-Magee in
+  :gh:`127592`.)
+
+* The iOS testbed is now able to stream test output while the test is running.
+  The testbed can also be used to run the test suite of projects other than
+  CPython itself. (Contributed by Russell Keith-Magee in :gh:`127592`.)
 
 New modules
 ===========

Include/cpython/initconfig.h

@@ -179,6 +179,9 @@ typedef struct PyConfig {
     int use_frozen_modules;
     int safe_path;
     int int_max_str_digits;
+#ifdef __APPLE__
+    int use_system_logger;
+#endif
 
     int cpu_count;
 #ifdef Py_GIL_DISABLED

Lib/_apple_support.py

@@ -0,0 +1,66 @@
+import io
+import sys
+
+
+def init_streams(log_write, stdout_level, stderr_level):
+    # Redirect stdout and stderr to the Apple system log. This method is
+    # invoked by init_apple_streams() (initconfig.c) if config->use_system_logger
+    # is enabled.
+    sys.stdout = SystemLog(log_write, stdout_level, errors=sys.stderr.errors)
+    sys.stderr = SystemLog(log_write, stderr_level, errors=sys.stderr.errors)
+
+
+class SystemLog(io.TextIOWrapper):
+    def __init__(self, log_write, level, **kwargs):
+        kwargs.setdefault("encoding", "UTF-8")
+        kwargs.setdefault("line_buffering", True)
+        super().__init__(LogStream(log_write, level), **kwargs)
+
+    def __repr__(self):
+        return f"<SystemLog (level {self.buffer.level})>"
+
+    def write(self, s):
+        if not isinstance(s, str):
+            raise TypeError(
+                f"write() argument must be str, not {type(s).__name__}")
+
+        # In case `s` is a str subclass that writes itself to stdout or stderr
+        # when we call its methods, convert it to an actual str.
+        s = str.__str__(s)
+
+        # We want to emit one log message per line, so split
+        # the string before sending it to the superclass.
+        for line in s.splitlines(keepends=True):
+            super().write(line)
+
+        return len(s)
+
+
+class LogStream(io.RawIOBase):
+    def __init__(self, log_write, level):
+        self.log_write = log_write
+        self.level = level
+
+    def __repr__(self):
+        return f"<LogStream (level {self.level!r})>"
+
+    def writable(self):
+        return True
+
+    def write(self, b):
+        if type(b) is not bytes:
+            try:
+                b = bytes(memoryview(b))
+            except TypeError:
+                raise TypeError(
+                    f"write() argument must be bytes-like, not {type(b).__name__}"
+                ) from None
+
+        # Writing an empty string to the stream should have no effect.
+        if b:
+            # Encode null bytes using "modified UTF-8" to avoid truncating the
+            # message. This should not affect the return value, as the caller
+            # may be expecting it to match the length of the input.
+            self.log_write(self.level, b.replace(b"\x00", b"\xc0\x80"))
+
+        return len(b)

Lib/test/test_apple.py

@@ -0,0 +1,155 @@
+import unittest
+from _apple_support import SystemLog
+from test.support import is_apple
+from unittest.mock import Mock, call
+
+if not is_apple:
+    raise unittest.SkipTest("Apple-specific")
+
+
+# Test redirection of stdout and stderr to the Apple system log.
+class TestAppleSystemLogOutput(unittest.TestCase):
+    maxDiff = None
+
+    def assert_writes(self, output):
+        self.assertEqual(
+            self.log_write.mock_calls,
+            [
+                call(self.log_level, line)
+                for line in output
+            ]
+        )
+
+        self.log_write.reset_mock()
+
+    def setUp(self):
+        self.log_write = Mock()
+        self.log_level = 42
+        self.log = SystemLog(self.log_write, self.log_level, errors="replace")
+
+    def test_repr(self):
+        self.assertEqual(repr(self.log), "<SystemLog (level 42)>")
+        self.assertEqual(repr(self.log.buffer), "<LogStream (level 42)>")
+
+    def test_log_config(self):
+        self.assertIs(self.log.writable(), True)
+        self.assertIs(self.log.readable(), False)
+
+        self.assertEqual("UTF-8", self.log.encoding)
+        self.assertEqual("replace", self.log.errors)
+
+        self.assertIs(self.log.line_buffering, True)
+        self.assertIs(self.log.write_through, False)
+
+    def test_empty_str(self):
+        self.log.write("")
+        self.log.flush()
+
+        self.assert_writes([])
+
+    def test_simple_str(self):
+        self.log.write("hello world\n")
+
+        self.assert_writes([b"hello world\n"])
+
+    def test_buffered_str(self):
+        self.log.write("h")
+        self.log.write("ello")
+        self.log.write(" ")
+        self.log.write("world\n")
+        self.log.write("goodbye.")
+        self.log.flush()
+
+        self.assert_writes([b"hello world\n", b"goodbye."])
+
+    def test_manual_flush(self):
+        self.log.write("Hello")
+
+        self.assert_writes([])
+
+        self.log.write(" world\nHere for a while...\nGoodbye")
+        self.assert_writes([b"Hello world\n", b"Here for a while...\n"])
+
+        self.log.write(" world\nHello again")
+        self.assert_writes([b"Goodbye world\n"])
+
+        self.log.flush()
+        self.assert_writes([b"Hello again"])
+
+    def test_non_ascii(self):
+        # Spanish
+        self.log.write("ol\u00e9\n")
+        self.assert_writes([b"ol\xc3\xa9\n"])
+
+        # Chinese
+        self.log.write("\u4e2d\u6587\n")
+        self.assert_writes([b"\xe4\xb8\xad\xe6\x96\x87\n"])
+
+        # Printing Non-BMP emoji
+        self.log.write("\U0001f600\n")
+        self.assert_writes([b"\xf0\x9f\x98\x80\n"])
+
+        # Non-encodable surrogates are replaced
+        self.log.write("\ud800\udc00\n")
+        self.assert_writes([b"??\n"])
+
+    def test_modified_null(self):
+        # Null characters are logged using "modified UTF-8".
+        self.log.write("\u0000\n")
+        self.assert_writes([b"\xc0\x80\n"])
+        self.log.write("a\u0000\n")
+        self.assert_writes([b"a\xc0\x80\n"])
+        self.log.write("\u0000b\n")
+        self.assert_writes([b"\xc0\x80b\n"])
+        self.log.write("a\u0000b\n")
+        self.assert_writes([b"a\xc0\x80b\n"])
+
+    def test_nonstandard_str(self):
+        # String subclasses are accepted, but they should be converted
+        # to a standard str without calling any of their methods.
+        class CustomStr(str):
+            def splitlines(self, *args, **kwargs):
+                raise AssertionError()
+
+            def __len__(self):
+                raise AssertionError()
+
+            def __str__(self):
+                raise AssertionError()
+
+        self.log.write(CustomStr("custom\n"))
+        self.assert_writes([b"custom\n"])
+
+    def test_non_str(self):
+        # Non-string classes are not accepted.
+        for obj in [b"", b"hello", None, 42]:
+            with self.subTest(obj=obj):
+                with self.assertRaisesRegex(
+                    TypeError,
+                    fr"write\(\) argument must be str, not "
+                    fr"{type(obj).__name__}"
+                ):
+                    self.log.write(obj)
+
+    def test_byteslike_in_buffer(self):
+        # The underlying buffer *can* accept bytes-like objects
+        self.log.buffer.write(bytearray(b"hello"))
+        self.log.flush()
+
+        self.log.buffer.write(b"")
+        self.log.flush()
+
+        self.log.buffer.write(b"goodbye")
+        self.log.flush()
+
+        self.assert_writes([b"hello", b"goodbye"])
+
+    def test_non_byteslike_in_buffer(self):
+        for obj in ["hello", None, 42]:
+            with self.subTest(obj=obj):
+                with self.assertRaisesRegex(
+                    TypeError,
+                    fr"write\(\) argument must be bytes-like, not "
+                    fr"{type(obj).__name__}"
+                ):
+                    self.log.buffer.write(obj)

Lib/test/test_capi/test_config.py

@@ -110,6 +110,10 @@ def test_config_get(self):
             options.extend((
                 ("_pystats", bool, None),
             ))
+        if support.is_apple:
+            options.extend((
+                ("use_system_logger", bool, None),
+            ))
 
         for name, option_type, sys_attr in options:
             with self.subTest(name=name, option_type=option_type,

Lib/test/test_embed.py

@@ -649,6 +649,8 @@ class InitConfigTests(EmbeddingTestsMixin, unittest.TestCase):
         CONFIG_COMPAT.update({
             'legacy_windows_stdio': False,
         })
+    if support.is_apple:
+        CONFIG_COMPAT['use_system_logger'] = False
 
     CONFIG_PYTHON = dict(CONFIG_COMPAT,
         _config_init=API_PYTHON,

Makefile.pre.in

@@ -2132,7 +2132,6 @@ testuniversal: all
 # This must be run *after* a `make install` has completed the build. The
 # `--with-framework-name` argument *cannot* be used when configuring the build.
 XCFOLDER:=iOSTestbed.$(MULTIARCH).$(shell date +%s)
-XCRESULT=$(XCFOLDER)/$(MULTIARCH).xcresult
 .PHONY: testios
 testios:
 	@if test "$(MACHDEP)" != "ios"; then \
@@ -2151,29 +2150,12 @@ testios:
 		echo "Cannot find a finalized iOS Python.framework. Have you run 'make install' to finalize the framework build?"; \
 		exit 1;\
 	fi
-	# Copy the testbed project into the build folder
-	cp -r $(srcdir)/iOS/testbed $(XCFOLDER)
-	# Copy the framework from the install location to the testbed project.
-	cp -r $(PYTHONFRAMEWORKPREFIX)/* $(XCFOLDER)/Python.xcframework/ios-arm64_x86_64-simulator
-
-	# Run the test suite for the Xcode project, targeting the iOS simulator.
-	# If the suite fails, touch a file in the test folder as a marker
-	if ! xcodebuild test -project $(XCFOLDER)/iOSTestbed.xcodeproj -scheme "iOSTestbed" -destination "platform=iOS Simulator,name=iPhone SE (3rd Generation)" -resultBundlePath $(XCRESULT) -derivedDataPath $(XCFOLDER)/DerivedData ; then \
-	 	touch $(XCFOLDER)/failed; \
-	fi
 
-	# Regardless of success or failure, extract and print the test output
-	xcrun xcresulttool get --path $(XCRESULT) \
-		--id $$( \
-			xcrun xcresulttool get --path $(XCRESULT) --format json | \
-			$(PYTHON_FOR_BUILD) -c "import sys, json; result = json.load(sys.stdin); print(result['actions']['_values'][0]['actionResult']['logRef']['id']['_value'])" \
-		) \
-		--format json | \
-		$(PYTHON_FOR_BUILD) -c "import sys, json; result = json.load(sys.stdin); print(result['subsections']['_values'][1]['subsections']['_values'][0]['emittedOutput']['_value'])"
+	# Clone the testbed project into the XCFOLDER
+	$(PYTHON_FOR_BUILD) $(srcdir)/iOS/testbed clone --framework $(PYTHONFRAMEWORKPREFIX) "$(XCFOLDER)"
 
-	@if test -e $(XCFOLDER)/failed ; then \
-		exit 1; \
-	fi
+	# Run the testbed project
+	$(PYTHON_FOR_BUILD) "$(XCFOLDER)" run -- test -uall --single-process --rerun -W
 
 # Like test, but using --slow-ci which enables all test resources and use
 # longer timeout. Run an optional pybuildbot.identify script to include

Misc/NEWS.d/next/Library/2024-12-04-15-04-12.gh-issue-126821.lKCLVV.rst

@@ -0,0 +1,2 @@
+macOS and iOS apps can now choose to redirect stdout and stderr to the
+system log during interpreter configuration.

Misc/NEWS.d/next/Tests/2024-12-04-15-03-24.gh-issue-126925.uxAMK-.rst

@@ -0,0 +1,2 @@
+iOS test results are now streamed during test execution, and the deprecated
+xcresulttool is no longer used.