Parser stack overflow on WASI with --with-pydebug

[FFY00]

Bug report

Bug description:

$ python Tools/wasm/wasi.py configure-build-python -- --with-pydebug
...
$ python Tools/wasm/wasi.py make-build-python
...
$ python Tools/wasm/wasi.py configure-host -- --with-pydebug
...
$ python Tools/wasm/wasi.py make-host
...
$ wasmtime run --dir .::/ cross-build/wasm32-wasip1/python.wasm -m test.test_compile
Traceback (most recent call last):
  File "/Lib/runpy.py", line 189, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
                               ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/Lib/runpy.py", line 159, in _get_module_details
    code = loader.get_code(mod_name)
  File "<frozen importlib._bootstrap_external>", line 896, in get_code
  File "<frozen importlib._bootstrap_external>", line 826, in source_to_code
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
MemoryError: Parser stack overflowed - Python source too complex to parse

CPython versions tested on:

CPython main branch

Operating systems tested on:

No response

Linked PRs

• GH-131770: increase assumed WASI stack size to 131072 (wasi-sdk default) #131844

[FFY00]

I bisected this to 0142236, specifically, the following call, which uses the new _Py_ReachedRecursionLimitWithMargin function.

cpython/Tools/peg_generator/pegen/c_generator.py

Lines 382 to 386 in 4b3d5b6

	def add_level(self) -> None:
	self.print("if (p->level++ == MAXSTACK || _Py_ReachedRecursionLimitWithMargin(PyThreadState_Get(), 1)) {")
	with self.indent():
	self.print("_Pypegen_stack_overflow(p);")
	self.print("}")

PYOS_STACK_MARGIN_BYTES seems a bit low, causing it to fail. In my last run, for example, I am seeing p->level at 209, which is quite far from the MAXSTACK limit. I don't know exactly how the WASM double stack model works, but I suspect that our PYOS_STACK_MARGIN value might be a bit low. I think expecting WASI to have a big enough stack for the parser to be able to parse all regular Python source files in the repo should be a fairly reasonable expectation.

cc @markshannon