frame.f_locals for list/dict/set comprehension in module/class scope doesn't contain iteration variables

[chgnrdv]

Bug report

Checklist

• I am confident this is a bug in CPython, not a bug in a third-party project
• I have searched the CPython issue tracker,
  and am confident this bug has not been reported before

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.13.0a0 (heads/main:dd32611f4f, Aug 31 2023, 20:23:23) [GCC 10.2.1 20210110]

A clear and concise description of the bug:

The following assertion fails with PEP 709 (c3b595e):

import sys
f_locs = None
[f_locs := sys._getframe().f_locals for a in range(1)]
assert 'a' in f_locs

This was discovered when trying to inspect listcomp iteration variable under pdb, as it makes inspection of these variables impossible:

> /home/radislav/projects/cpython/listcomp.py(1)<module>()
-> [print(a) for a in range(1)]
(Pdb) s
0
> /home/radislav/projects/cpython/listcomp.py(1)<module>()
-> [print(a) for a in range(1)]
(Pdb) print(a)
*** NameError: name 'a' is not defined
(Pdb) s
/home/radislav/projects/cpython/listcomp.py:1: RuntimeWarning: assigning None to unbound local 'a'
  [print(a) for a in range(1)]
--Return--
> /home/radislav/projects/cpython/listcomp.py(1)<module>()->None
-> [print(a) for a in range(1)]

As I can see, it happens because iteration vars are now stored in "fast hidden locals" that aren't included into frame->f_locals on _PyFrame_FastToLocalsWithError(frame) call, and thus are not accessible through PyFrameObject f_locals attribute.

cc @carljm

Linked PRs

• gh-108732: include comprehension locals in frame.f_locals #109026
• [3.12] gh-108732: include comprehension locals in frame.f_locals (GH-109026) #109097

[carljm]

Thanks for the report! Attached PR fixes this. Sorry I wasn't able to get the PR up and merged in time for 3.12.0.rc2; was traveling over the long (US) holiday weekend. I think this probably means the fix will need to wait until 3.12.1, unless @Yhg1s wants to merge it between rc2 and final .0. I think the fix is low-risk, but the bug is also an edge case that is unlikely to affect a lot of users, so I think either way is probably OK.

[P403n1x87]

Just stumbled upon this and I have noticed something similar with frame stacks. Not sure whether this is considered expected in 3.12

❯ python3.12 -c "import sys;from traceback import print_stack; [print_stack() for _ in range(1)]"
  File "<string>", line 1, in <module>

❯ python3.11 -c "import sys;from traceback import print_stack; [print_stack() for _ in range(1)]"
  File "<string>", line 1, in <module>
  File "<string>", line 1, in <listcomp>

Essentially, frame stacks in 3.12 seem to be missing list (and probably other, not tested) comprehensions.

[carljm]

@P403n1x87 Yes, that's expected and called out in PEP 709 as an expected impact. There are not language guarantees about what frames show up in Python frame stacks.