IDLE: Handle Shell input SyntaxWarning & DeprecationWarning

[serhiy-storchaka]

BPO	37824
Nosy	@rhettinger, @terryjreedy, @taleinat, @serhiy-storchaka, @csabella, @miss-islington
PRs	gh-82005: Properly handle user input warnings in IDLE shell. #15311 bpo-37824: Properly handle user input warnings in IDLE shell. #15500 [3.8] bpo-37824: Properly handle user input warnings in IDLE shell. (GH-15500) #15504 [3.7] bpo-37824: Properly handle user input warnings in IDLE shell. (GH-15500) #15505

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/terryjreedy'
closed_at = None
created_at = <Date 2019-08-11.16:32:12.488>
labels = ['3.8', 'expert-IDLE', 'type-bug', '3.7', '3.9']
title = 'IDLE: Handle Shell input SyntaxWarning & DeprecationWarning'
updated_at = <Date 2021-11-12.19:21:29.683>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2021-11-12.19:21:29.683>
actor = 'terry.reedy'
assignee = 'terry.reedy'
closed = False
closed_date = None
closer = None
components = ['IDLE']
creation = <Date 2019-08-11.16:32:12.488>
creator = 'serhiy.storchaka'
dependencies = []
files = []
hgrepos = []
issue_num = 37824
keywords = ['patch']
message_count = 19.0
messages = ['349396', '349441', '349846', '349858', '349869', '349884', '349896', '350483', '350487', '350492', '350494', '369929', '369934', '370163', '370212', '370357', '370358', '386622', '406234']
nosy_count = 6.0
nosy_names = ['rhettinger', 'terry.reedy', 'taleinat', 'serhiy.storchaka', 'cheryl.sabella', 'miss-islington']
pr_nums = ['15311', '15500', '15504', '15505']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue37824'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9']

[serhiy-storchaka]

DeprecationWarning at compile time is output multiple times to the stderr (on console from which IDLE was ran), but not to IDLE Shell window.

For example, when enter '\e':

$ ./python -m idlelib

Warning (from warnings module):
  File "<pyshell#0>", line 1
    '\e'
DeprecationWarning: invalid escape sequence \e
>>> 
Warning (from warnings module):
  File "<pyshell#0>", line 1
    '\e'
DeprecationWarning: invalid escape sequence \e
>>> 
Warning (from warnings module):
  File "<pyshell#0>", line 1
    '\e'
DeprecationWarning: invalid escape sequence \e

And when close IDLE, additional output is printed in 3.8+:

>>> Exception ignored in: <idlelib.run.PseudoInputFile object at 0x7f9e3b4a15a0>
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/idlelib/run.py", line 488, in close
  File "/home/serhiy/py/cpython/Lib/idlelib/pyshell.py", line 1019, in close
  File "/home/serhiy/py/cpython/Lib/idlelib/editor.py", line 1058, in close
  File "/home/serhiy/py/cpython/Lib/idlelib/outwin.py", line 94, in maybesave
  File "/home/serhiy/py/cpython/Lib/idlelib/editor.py", line 991, in get_saved
AttributeError: 'NoneType' object has no attribute 'get_saved'

[terryjreedy]

None.get_saved is bpo-35623. I believe it is related to bpo-35379, where I requested your comment on what I found so far.
---

1. Compiler warnings for shell input go to sys.__stderr, which only exists when IDLE is started from a console. This will include SyntaxWarnings when not turned into errors. pyshell 67+, warning_stream and idle_showwarning. I have no idea why. Compiler warnings for editor contents and runtime warnings go to shell. For now, all should go to Shell.

2. Warnings are formatted by run 42, idle_formatwarning. Someone must have decided that "<stdin>:1: DeprecationWarning: invalid escape sequence \e" is newbie unfriendly. When printed in an editor-derived window, the IDLE format will enable context menu 'goto file/line'. But 4 lines is a lot and the top line only serves to imitate the 'Traceback ...' line. I will think about this. Perhaps

DeprecationWarning: invalid escape sequence \e
File "<pyshell#1>", line 1
'\e'

3. Warnings (at least this one) for shell input are printed 3 times. I do not know why at present. ModifiedInterpreter.runsource calls code.InteractiveInterpreter(self, source, filename (pyshell 669). Perhaps the latter is buggy in somehow calling compile 3 times. I will look another time.

[terryjreedy]

I am combining the trivial 'leave Shell input SyntaxWarnings alone (instead of making them SyntaxErrors)' part of bpo-34857 with this issue, 'print Shell input (and the very rare internal IDLE Warnings occurring after Shell exists) in Shell, and changing the title accordingly.

The PR is a WIP with at least two bugs. Help wanted.

1. the triple output. (Before the patch, when I started IDLE from command prompt, I only saw one.
2. Warnings are inserted before the text generating the warning. I interpret this as intercepted \n causing compilation and printing of the warning before the \n is inserted at the end of the line and the iomark moved to the beginning of the next. I don't know what moves the iomark.

[taleinat]

Il take a look at this. I've very recently gone through the relevant parts of the code and I should be able to figure it it efficiently.

[terryjreedy]

The triple (or double, see below) printing seems like a pure bug to be fixed. A fix could be a separate PR to be merged 'immediately'.

But there is a complication with insertion location. My bug claim was based on a one line statement, where 'before' looks wrong and 'after', as in REPL, right.

>>> <IDLE>
Warning (from warnings module):  # Printed 3 times, 2 deleted.
  File "<pyshell#0>", line 1
    0 is 0
SyntaxWarning: "is" with a literal. Did you mean "=="?
0 is 0
True

>>> 0 is 0  # REPL
<stdin>:1: SyntaxWarning: "is" with a literal. Did you mean "=="?
True

But now consider a more realistic compound statement.

>>> if 0 is 0:  # REPL
...   print('true')
...
<stdin>:1: SyntaxWarning: "is" with a literal. Did you mean "=="?
true

With IDLE, a warning is printed *before* the user enters a blank line to signal end of statement.

>>> 
Warning (from warnings module):  # Only printed TWICE!.
  File "<pyshell#2>", line 1
    if 0 is 0:
SyntaxWarning: "is" with a literal. Did you mean "=="?
if 0 is 0:
	print('true')
	|< cursor location

A user could edit the condition, add more to the suite, or hit return. In the latter 2 cases, the warnings are repeated. In this case, the insertion seems like the best that can be done.

Perhaps the msg349845 comment about run.show_warnings collecting warnings for possible delayed display applies to pyshell.show_warnings. Or the warning could be put in a popup, though I do not especially like the idea of that. Or it could be logged to a separate Warnings window, and a red "Warning logged' flashed on the status bar.

I suspect that this case is why a fake prompt was previously tacked on to the end of the warning, though I also suspect these complications are why shell input warnings were relegated to a possibly non-existent console (where the fake prompt is nonsense).

Behavior is the same with '\e' and DeprecationWarning.

This example further convinces me that multiline shell input should be isolated from both prompts and intermixed outputs. I plan to open that issue later today.

[taleinat]

I see none of this on latest master (0567786) on Windows 10. Can you specify on what OS and Python branch/version this happens?

[terryjreedy]

My first post described master at the time, when SyntaxWarnings were converted to SyntexErrors and shell input warnings, in particular, Deprecation warnings were sent to the console or nowhere. My third post describes the situation with the PR, where SyntaxWarnings are left as warnings and shell input warnings go to shell. An improvement, but still buggy. I just reconfirmed after updating, compiling Python, and checking out my 'warn' branch.

[terryjreedy]

I pulled the part of PR 15311 that works, stop turning Shell SyntaxWarnings into SyntaxErrors, into PR-15500, to get it into b4. I will update the former after merging the latter.

[terryjreedy]

New changeset 1039f39 by Terry Jan Reedy in branch 'master':
bpo-37824: Properly handle user input warnings in IDLE shell. (GH-15500)
1039f39

[terryjreedy]

New changeset 0778870 by Terry Jan Reedy (Miss Islington (bot)) in branch '3.8':
bpo-37824: Properly handle user input warnings in IDLE shell. (GH-15500)
0778870

[miss-islington]

New changeset 1b15914 by Miss Islington (bot) in branch '3.7':
bpo-37824: Properly handle user input warnings in IDLE shell. (GH-15500)
1b15914

[csabella]

Terry,

I put this into debug and found the reason it's printing the warning three times. In codeop.py, it's running _maybe_compile and there are three try statements:

def _maybe_compile(compiler, source, filename, symbol):
    # Check for source consisting of only blank lines and comments
    for line in source.split("\n"):
        line = line.strip()
        if line and line[0] != '#':
            break               # Leave it alone
    else:
        if symbol != "eval":
            source = "pass"     # Replace it with a 'pass' statement

    err = err1 = err2 = None
    code = code1 = code2 = None

    try:
        code = compiler(source, filename, symbol)
    except SyntaxError:
        pass

    try:
        code1 = compiler(source + "\n", filename, symbol)
    except SyntaxError as e:
        err1 = e

    try:
        code2 = compiler(source + "\n\n", filename, symbol)
    except SyntaxError as e:
        err2 = e

    try:
        if code:
            return code
        if not code1 and repr(err1) == repr(err2):
            raise err1
    finally:
        err1 = err2 = None

It also has this in the module docstring:

Compile three times: as is, with \n, and with \n\n appended.  If it
compiles as is, it's complete.  If it compiles with one \n appended,
we expect more.  If it doesn't compile either way, we compare the
error we get when compiling with \n or \n\n appended.  If the errors
are the same, the code is broken.  But if the errors are different, we
expect more.  Not intuitive; not even guaranteed to hold in future
releases; but this matches the compiler's behavior from Python 1.4
through 2.2, at least.

[csabella]

I added a commit to the PR to show warnings once.

[terryjreedy]

codeop._maybe_compile wraps each compile in try ... except SyntaxError. It can later reraise just once. I think it a bug that it is not similarly careful about SyntaxWarning and DeprecationWarning to only emit a particular warning just once.

Cheryl, would you like to open a new issue and submit a patch?

[csabella]

Thanks, Terry. I created bpo-40807 for the codeop warnings.

[terryjreedy]

To review: Serhiy reported 3 'IDLE' bugs: tripled DeprecationWarning (now joined by SyntaxWarning), printing to console (if available) instead of Shell, and an exit exception.

1. Codeop generates the tripicates; bpo-40807 will fix that.
2. PR-15311 prints warnings to Shell, but I held off pending some fix. 3. The exception (a duplicate report) was fixed on bpo-35623.
3. PR-15500 stopped turning a SyntaxWarning into a SyntaxError.

Additional issues, semi-independent.
5. REPL prints warning after code, IDLE before
6. REPL prints 1 line, IDLE 4, but this is related to Shell warnings going to the console. IDLE should use 1 and highlight.
7. Should IDLE extend scope of deduplication? Yes with 4 lines, maybe not with 1 line.
8. REPL partially deduplicates within compound statement; see bpo-40807.

[terryjreedy]

The new items can be new issues, with 6 maybe the highest priority. Currently
<stdin>:1: DeprecationWarning: invalid escape sequence \e
becomes
Warning (from warnings module):
File "<pyshell#0>", line 1
'\e'
DeprecationWarning: invalid escape sequence \e

For both Shell and Editor input, the Warning Line is not really needed. For Shell, the fake filename is not needed, nor is the line # and line if somehow otherwise marked.

[terryjreedy]

bpo-43151 asked about 'is <literal>' syntax warnings in REPL. Discussion noted that IDLE Shell does note print it for "x is 'a'" (known here) and that for "if x is 'a': print('executed')" there is a syntax error and no execution. The conversion to SyntaxError was supposedly fixed by PR-15500, but it appears that the patch for bpo-40807 introduced it into codeop._maybe_compile.

msg386621 has summary and discussion. I need to add debug prints or breakpoints into _maybe_compile to understand it better, and maybe improve it.

Before printing syntax warnings in Shell, I want to stop expanding them from 1 to 4 lines. We could just not import the replacement from run.py, but I think the default format could be improved even with 1 or 2 lines.

[terryjreedy]

Might want to special case running certain console commands in Shell, such as 'pip install'.
https://stackoverflow.com/questions/69936921/packages-not-installing-in-idle-shell
SyntaxError: invalid syntax. Perhaps you forgot a comma?