Pasting 🏳️‍🌈 crashes the new Python REPL

[treyhunner]

Crash report

What happened?

To reproduce, start the new Python REPL in 3.13.0b3 and start to assign a string:

$ python3.13
Python 3.13.0b3 (main, Jul  2 2024, 13:24:06) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> flag = '

Then paste 🏳️‍🌈 and this happens:

$ python3.13
Python 3.13.0b3 (main, Jul  2 2024, 13:24:06) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> flag = 'Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/trey/.pyenv/versions/3.13.0b3/lib/python3.13/_pyrepl/__main__.py", line 3, in <module>
    __pyrepl_interactive_console()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/trey/.pyenv/versions/3.13.0b3/lib/python3.13/_pyrepl/main.py", line 55, in interactive_console
    run_interactive(namespace)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/home/trey/.pyenv/versions/3.13.0b3/lib/python3.13/_pyrepl/simple_interact.py", line 144, in run_multiline_interactive_console
    statement = multiline_input(more_lines, ps1, ps2)
  File "/home/trey/.pyenv/versions/3.13.0b3/lib/python3.13/_pyrepl/readline.py", line 385, in multiline_input
    return reader.readline()
           ~~~~~~~~~~~~~~~^^
  File "/home/trey/.pyenv/versions/3.13.0b3/lib/python3.13/_pyrepl/reader.py", line 768, in readline
    self.handle1()
    ~~~~~~~~~~~~^^
  File "/home/trey/.pyenv/versions/3.13.0b3/lib/python3.13/_pyrepl/reader.py", line 751, in handle1
    self.do_cmd(cmd)
    ~~~~~~~~~~~^^^^^
  File "/home/trey/.pyenv/versions/3.13.0b3/lib/python3.13/_pyrepl/reader.py", line 695, in do_cmd
    self.refresh()
    ~~~~~~~~~~~~^^
  File "/home/trey/.pyenv/versions/3.13.0b3/lib/python3.13/_pyrepl/reader.py", line 672, in refresh
    self.screen = self.calc_screen()
                  ~~~~~~~~~~~~~~~~^^
  File "/home/trey/.pyenv/versions/3.13.0b3/lib/python3.13/_pyrepl/completing_reader.py", line 261, in calc_screen
    screen = super().calc_screen()
  File "/home/trey/.pyenv/versions/3.13.0b3/lib/python3.13/_pyrepl/reader.py", line 360, in calc_screen
    l, l2 = disp_str(line)
            ~~~~~~~~^^^^^^
  File "/home/trey/.pyenv/versions/3.13.0b3/lib/python3.13/_pyrepl/reader.py", line 61, in disp_str
    b.append(str_width(c))
             ~~~~~~~~~^^^
  File "/home/trey/.pyenv/versions/3.13.0b3/lib/python3.13/_pyrepl/utils.py", line 10, in str_width
    if ord(c) < 128:
       ~~~^^^
TypeError: ord() expected a character, but string of length 6 found

This looks like an issue with multiple-character glyphs, as 🏴󠁧󠁢󠁥󠁮󠁧󠁿 causes the same issue.

I found myself at my system command prompt after this (Python had exited).

I initially tested this on Python 3.13.0b3, but I also reproduced the issue on the current main branch.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.13.0b3 (main, Jul 2 2024, 13:24:06) [GCC 11.4.0]

Linked PRs

• gh-121609: Fix pasting of characters containing unicode character joiner #121667
• [3.13] gh-121609: Fix pasting of characters containing unicode character joiner (GH-121667) #121733

[itamaro]

confirmed the reproducer indeed crashes the repl on 3.14 (main branch) on macOS too (with a debug build and GIL disabled, ftr)

[itamaro]

for comparison, this is how it behaves on 3.12:

$ python3                                                                                                                                  [18:48:47]
Python 3.12.3 (v3.12.3:f6650f9ad7, Apr  9 2024, 08:18:47) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> flag = '🏳️\U+200D🌈'
>>> flag
'🏳️\u200d🌈'
>>> print(flag)
🏳️‍🌈

the '🏳️\U+200D🌈' sequence is what gets pasted when I paste the pride flag, not something I typed manually

[Eclips4]

Hello! The type-crash label is intended for bugs where the interpreter exits with a segmentation fault or something similar.
The type-bug label is more applicable here since there is no hard crash of the interpreter.

[mgmacias95]

I'm working on this :)

[ods]

@mgmacias95 It's just a typo, one extra copy-pasted line

--- i/Lib/_pyrepl/reader.py
+++ w/Lib/_pyrepl/reader.py
@@ -58,7 +58,6 @@ def disp_str(buffer: str) -> tuple[str, list[int]]:
         elif unicodedata.category(c).startswith("C"):
             c = r"\u%04x" % ord(c)
             s.append(c)
-            b.append(str_width(c))
             b.extend([0] * (len(c) - 1))
         else:
             s.append(c)

[mgmacias95]

If that line is removed the emoji is pasted as:

mgmacias@mgmacias:~/cpython$ ./python
Python 3.14.0a0 (remotes/cpython/main:dc03ce797ae, Jul 13 2024, 08:30:47) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 🏳️\u200d🌈🏳️\u200d🌈🏳️\u200d🌈🏳️\u200d🌈🏳️\u200d🌈

In prior versions of python it was pasted correctly:

Python 3.12.2 (main, Feb  7 2024, 20:47:03) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 🏳️‍🌈🏳️‍🌈🏳️‍🌈🏳️‍🌈🏳️‍🌈🏳️‍🌈

[ods]

It looks like intended behaviour: the code explicitly replaces unprintable control/format/etc characters with their unicode escape sequence.

[hugovk]

Thank you @treyhunner for the report, @mgmacias95 for the fix, and everyone else for the investigation!