[WIP] A workaround for Python 3.6 WindowsConsoleIO breaking with FDCapture #2462
Conversation
Python 3.6 implemented unicode console handling for Windows. This works
by reading/writing to the raw console handle using
``{Read,Write}ConsoleW``.
The problem is that we are going to ``dup2`` over the stdio file
descriptors when doing ``FDCapture`` and this will ``CloseHandle`` the
handles used by Python to write to the console. Though there is still some
weirdness and the console handle seems to only be closed randomly and not
on the first call to ``CloseHandle``, or maybe it gets reopened with the
same handle value when we suspend capturing.
The workaround in this case will reopen stdio with a different fd which
also means a different handle by replicating the logic in
"Py_lifecycle.c:initstdio/create_stdio".
See pytest-dev/py#103
|
@RonnyPfannschmidt Not so fast - it is marked as WIP, breaks the checks, and would be nice if someone could test this on Windows 😉 |
|
@RonnyPfannschmidt OK if you like it as is. Though lets make sure it at least passed CI first. (I had a wrong if condition). I will push an update soon. Do note that any code doing |
|
hmm, oh, i was indeed a bit quick - but i like the basic setup, lets sort out the technical issues |
|
@RonnyPfannschmidt @The-Compiler I do think it would be nice if we can first figure this out completely before merging. Like I mentioned It probably also changes the fd numbers of the EDIT: Note that appveyor is likely not using a Windows console to run the tests. (Something like winpty). So this code probably doesn't run there. |
|
Agree, I would like to take this for a spin on my computer before merging this. |
|
See http://bugs.python.org/issue30555. If this is solved in CPython's side this hack will not be necessary anymore. Since it will likely be a while until it's solved, if it's even decided to solve this in CPython's side. We can still implement a workaround and remove it/gate it on an upper CPython version, when and if, CPython implements it's own fix. |
|
@segevfiner first of all thanks for all the work you've put into this. I've read briefly your bug reports both here, in Hopefully your python/cpython#1927 will get accepted as it affects other projects. Other than that I've verified that this fixes the problem for me on my Windows system, so I'm 👍 to add this workaround to pytest: it only gets into effect for Python 3.6 users on Windows and is cleanly written so as to separate it from the normal code. @segevfiner anything else you want to do with it, or can I merge it as it is? |
|
@nicoddemus It should be okay to merge. We just need to remember to disable it conditionally or remove it when and if this is fixed in CPython. 😉 |
|
Awesome, thanks again! 👍 |
On Windows with Python >= 3.6.0, the console handling seems to have changed and this seems to break our capturing the file descriptor to suppress the compilation messages on the console. pytest has a fix here: pytest-dev/pytest#2462 but we essentially do not do any capturing on windows for these Python versions as this fix doesn't seem to help us here. Hopefully this will be resolved in Python itself at some point but for now this error was breaking any compilation using compyle on Windows.
Python 3.6 implemented unicode console handling for Windows. This works by reading/writing to the raw console handle using
{Read,Write}ConsoleW.The problem is that we are going to
dup2over the stdio file descriptors when doingFDCaptureand this willCloseHandlethe handles used by Python to write to the console. Though there is still some weirdness and the console handle seems to only be closed randomly and not on the first call toCloseHandle, or maybe it gets reopened with the same handle value when we suspend capturing.The workaround in this case will reopen stdio with a different fd which also means a different handle by replicating the logic in "Py_lifecycle.c:initstdio/create_stdio".
Obviously not very pretty, I don't know what other side effects this might have and I'm not even sure this is the right way to go about this. Since the std handles are getting clobbered, other code which does
GetStdHandle, or something similar, is also affected. Meaning that this is only a partial fix.For example: colorama breaks just as randomly. But it ignores the errors and all colorama features just randomly stop working (e.g. Colors stop working). Adding error handling to
colorama.wintermreveals the very same exception. Essentially colorama caches the std handles incolorama.win32.handlesand it breaks, even without_WindowsConsoleIObeing used by Python, due toFDCapture.Of course, who knows what other stuff might get broken by this. And I'm still baffled by why it's random and only have a guess as to why (Described above).
Also note that I currently ignored the case if
sys.stdout != sys.__stdout__and similarly for the otherstd.std*. I'm not sure what is the right thing to do in such a case...I'm using the Pytest test suite and this to reproduce the issue (Running with
pytest -v):See also
GetLastError()mishap in_io.WindowsConsoleIO.write.PR Checklist
Thanks for submitting a PR, your contribution is really appreciated!
Here's a quick checklist that should be present in PRs:
$issue_id.$typefor example (588.bug)removal,feature,bugfix,vendor,docortrivialbugfix,vendor,docortrivialfixes, targetmaster; for removals or features targetfeatures;Unless your change is a trivial or a documentation fix (e.g., a typo or reword of a small section) please:
AUTHORS;