mintty issues

[kblees]

Playing around with the new development environment a bit, I found the following issues with mintty:

Issues with git / msys programs:

• PR Better MSys PTY detection #102: automatic paging doesn't work in a second shell window (opened via Alt+F2, 'start mintty', 'mintty &', 'git-bash' etc.)
• d56d2f8: password prompt doesn't work in a second shell window (or goes to the wrong window)
• d56d2f8: password prompt no longer works in a native console window (due to new "mintty-aware" git_terminal_prompt checking for TERM=xterm)

Issues with other (native) console programs:

• non-ASCII characters don't work (e.g. cmd /? mangles äöü...)
• automatic paging doesn't work (e.g. cmd /?)
• cursor keys don't work (e.g. in cmd subshell)
• console screen buffer doesn't work (e.g. Far Manager)

[elyscape]

Unfortunately, most of the issues with native console programs are an ongoing and long-standing issue with mintty due to the way it implements standard streams and pipes and unlikely to be resolved any time soon.

[elyscape]

The short version, as I understand it, is that, in lieu of actual pseudo-terminal support on Windows, mintty implements the standard streams as named Windows pipes that programs can arbitrarily attach to. The problem is that programs that haven't been specifically compiled to support treating those particular streams as terminal streams will cause issues by treating them as pipes. See for example the patch in de0d35d to get Git to work properly in mintty. Most applications compiled under MSYS2 should automatically handle that, as it's part of the provided _isatty() function, but anything else is liable to have issues.

[dscho]

Given that I frequently use git log in two separate mintty windows and it Works Just Fine, there is something really fishy going on on the reporter's side.

[dscho]

Oh, maybe the problem is something else entirely: @kblees by "git/msys" do you refer to Git for Windows 1.x? Sorry, it is not at all clear to me what your setup is, after reading the report again.

[kblees]

Given that I frequently use git log in two separate mintty windows and it Works Just Fine, there is something really fishy going on on the reporter's side.

Steps to reproduce:

• install fresh MSys2 development environment (no, I'm not using msys1)
• open mintty (via git-bash.exe or mingw64-shell.bat)
• open second window (via Alt+F2, or start mintty sh -li or start /git-bash or /git-bash &...)
• git log in second window doesn't use pager
• git credential fill <credfile prompt goes to the wrong window, and only reads a single character instead of an entire line of input

Note that these issues are specific to mintty, native console windows work just fine (e.g. start sh -li (+ export TERM=cygwin to disable the mintty-specific git_terminal_prompt code)).

[dscho]

open second window

The only difference I see is that I open the second window by double-clicking the desktop icon of the Git SDK. And voilà, it works here. Maybe it would work that way on your side, too?

[kblees]

The only difference I see is that I open the second window by double-clicking the desktop icon of the Git SDK.

Double-clicking the desktop icon starts a fresh shell, it loses the current directory and any additional environment settings that may be necessary to work on a project.

[dscho]

The only difference I see is that I open the second window by double-clicking the desktop icon of the Git SDK.

Double-clicking the desktop icon starts a fresh shell, it loses the current directory and any additional environment settings that may be necessary to work on a project.

That still does not answer my question whether that makes the pager work for you?

[elyscape]

Hmm, it looks like the issues with native console programs might be at least partially resolved if mintty adopts something similar to this code from winansi.c. It modifies the internal file handles such that native isatty() functions return true.

[dscho]

Interesting hack. But I think console programs running inside mintty would need that hack, not mintty itself. For example, if you want to run an interactive Python session in mintty, you would need to modify Python's source code...

[tkelman]

We've made this work fine for Julia's REPL by specifically checking for the pipe names that mintty uses here: https://github.com/JuliaLang/julia/blob/9ad3aa0813f8e4652c761ab010bbc537ac807b2a/src/jl_uv.c#L876-L904

It can be made to work, but as @dscho said it needs some special patching in the terminal/pipe handling of any program you want to fix.

[elyscape]

I wonder if it would be possible for mintty to hook the handle creation function for child processes and run that code before returning the handle.

[tkelman]

Maybe? It would probably need to be forked and experimented with. Last I checked with some Cygwin folks, mintty's author has been MIA for a while.

[dscho]

I do not see how this could be changed on mintty's side because all the presented workarounds know that isatty() returns the "wrong thing" for mintty's stdin/stdout/stderr and work around that on their side.

A solution on mintty's side would magically make MSVCRT's isatty() report "this is a terminal" when it is not, as far as MSVCRT is concerned: MSVCRT's isatty() basically says: "is this attached to a Win32 console?"...

[dscho]

It would probably need to be forked and experimented with. Last I checked with some Cygwin folks, mintty's author has been MIA for a while.

That is not a problem, thanks to the awesomeness called GitHub.

[dscho]

automatic paging doesn't work in a second shell window (opened via Alt+F2, 'start mintty', 'mintty &', 'git-bash' etc.)

I got a chance to test this (I actually only tested /git-bash &). The workaround is awfully ridiculous: call unset MSYS_TTY_HANDLES and everything works again. I guess the second true in this line should be a false instead.

[dscho]

I guess the second true in this line should be a false instead.

Sadly, this does not fix the issue. I fear we need to force a reinitialization of that environment variable somehow.

[kblees]

I fear we need to force a reinitialization of that environment variable somehow.

...or go a different route: check if the name of the pipe matches '...msyspty' (at least that's how MSys2.dll does it). Sadly this requires the internal NtQueryObject API from ntdll.dll

Considering that handles can be duplicated and closed (and closed handles can be reused for different kernel objects), the MSYS_TTY_HANDLES solution sounds a bit fragile.

[dscho]

Considering that handles can be duplicated and closed (and closed handles can be reused for different kernel objects), the MSYS_TTY_HANDLES solution sounds a bit fragile.

The idea of MSYS_TTY_HANDLES is that it is only set for processes that were spawned off of a running terminal. And while that terminal is running, the handles are still in use and cannot be reassigned.

The problem we are facing here is that this environment variable is not reinitialized correctly. I have a hunch that the code path I am using is actually hit only once in the call stack (read: if a mintty launches another mintty, the code may not even be executed to set MSYS_TTY_HANDLES).

[kblees]

And while that terminal is running, the handles are still in use and cannot be reassigned.

I don't think so. Handles are private to a process. The only exception would be inherited handles, which initially have the same value in parent and child process. But if the child process closes them, they can be reused for anything.

if a mintty launches another mintty, the code may not even be executed to set MSYS_TTY_HANDLES

Perhaps MSys fork emulation uses DuplicateHandle to make extra sure that the handles are inheritable, but the static unsigned initialized variable in cygheap_user::env_tty_handles survives the fork, so MSYS_TTY_HANDLES is not re-built? Just a hunch...

If git launches another git, though (which I think is what the !git alias in issue #85 does), the code in MSys2.dll will definitely not be executed :-)

[dscho]

if the child process closes them, they can be reused for anything.

Hmm. In that case, we would have no way to determine whether a given stdin/stdout/stderr refers to a stream that is connected to mintty. Hmm.

the static unsigned initialized variable in cygheap_user::env_tty_handles survives the fork

good point. I'll look into that tomorrow.

If git launches another git, though (which I think is what the !git alias in issue #85 does), the code in MSys2.dll will definitely not be executed :-)

Right. My current plan is to adjust MSYS_TTY_HANDLES appropriately in this case (if it exists).

[dscho]

@rcdailey you could investigate yourself by comparing the behavior and /etc/inputrc between Git for Windows 1.x and 2.x.

[dscho]

@rcdailey you could investigate yourself by comparing the behavior and /etc/inputrc between Git for Windows 1.x and 2.x.

... and if you find something interesting, you might want to open a new Pull Request right away (this here ticket is about an entirely unrelated issue than you raised in your latest comment).

[elyscape]

@rcdailey you could investigate yourself by comparing the behavior and /etc/inputrc between Git for Windows 1.x and 2.x.

The only differences between the /etc/inputrc files in Git for WIndows 1.x and 2.x are a few of the key bindings and this setting:

set mark-symlinked-directories on

I just tested on my system by typing "cd /c/ProgramD" and hitting tab. It completed immediately to /c/ProgramData/. I suspect something with @rcdailey's environment.

[jan-hudec]

There is a terminal emulator for windows, http://conemu.github.io/, that spawns native console hosts and hacks into them to manage the actual display. I've been using it for both cygwin and msysgit (1.x) for couple of years, basically since cygwin started pushing mintty, because I always needed native (compiled by somebody else with MSC++) programs to work.

Perhaps it would be an option to use that terminal, some other terminal that implements the hack or point people at that terminal if they need to use native programs from git bash instead of using mintty.

[dscho]

@jan-hudec @rcdailey this is really the wrong ticket to discuss these issues. The correct ticket to do so, in fact the ticket that already has this discussion, is here: #96.

Let's keep discussions organized and head over to the appropriate place, and stay on topic in this ticket's discussion.

[jan-hudec]

I mean, since native programs have, and always will have, problems with mintty, why use it at all? What is the problem with using default console host (cmd.exe) like version 1 does?

I mentioned ConEmu to suggest that if the problem with cmd.exe is that it is crap, it would be better solved by pointing people to ConEmu then by using mintty, which has it's own problems.

[dscho]

I mean, since native programs have, and always will have, problems with mintty, why use it at all?

This statement is so broad that it is necessarily incorrect.

So let me restate what I think you intended to say:

Compared to other terminal emulators, mintty causes more problems because it does not offer a Win32 Console to the programs running inside. Why is it the default?

That statement is correct and the question is valid.

The answer is two-fold: mintty is the default because it is small, much smaller than any alternative.

If you have to run a program that requires a Win32 Console, you can do so by using the winpty helper which does offer a Win32 Console, with all the incurred performance penalties.

By the way, since the issue tracker together with VM struggles and actual development keeps me too busy to update the FAQ myself, I would appreciate it if you took the time to chisel what you learned into a kick-ass answer here: https://github.com/git-for-windows/git/wiki/FAQ

[jan-hudec]

@dscho, you still didn't say why use any replacement for the win32 console at all. For me, the win32 console is preferred because of encoding issues (msys and cygwin use utf-8, native programs use legacy codepages, so winpty is needed for anything that prints non-ascii, which is a lot of things for me).

[dscho]

you still didn't say why use any replacement for the win32 console at all.

Ah, you're correct, of course. Sorry.

Of the top of my head:

• My workflow with Git requires frequent copy-pasting, say, of commit names and command-lines. The rectangular-only selection in cmd drives me crazy.
• When I resize cmd, it does not let me resize horizontally, only vertically.
• The default for cmd's scroll-back buffer is ridiculously small and cannot be adjusted for Git for Windows only.
• Shift+PageUp does not scroll back in cmd.
• There is no keyboard shortcut for copying or pasting.
• The default for cmd is to have a non-Unicode font.

These are just a couple of really annoying things.

To be sure, before releasing a Git for Windows 2.x, I want to add a page to the installer that lets you switch to the cmd-based terminal emulator instead of mintty (and hopefully also other terminal emulators, at least at some stage).

If mintty is hated universally enough, we will make the cmd-backed Git Bash the default again. So far, I am convinced that more users are in favor of mintty, though.

[jan-hudec]

Thanks.

The scroll-back, size and font are set in the shortcut, so that should be possible to set for Git Bash and I've learned to live with the mouse-only copy&paste. The rectangular selection is silly, I agree.

I suppose use case like ours with a bunch of bash scripts calling native tools (for which msys is better then cygwin) is not that typical to make mintty universally hated.

[dscho]

By all means please bundle winpty;

I forgot that, didn't I? Could you please patch https://github.com/git-for-windows/build-extra/blob/master/make-file-list.sh#L29-L32 and open a PR after testing that this does what you need? (I can't do that right now because my machine is running the Git test suite for another 1.5h or so)

[be5invis]

@dscho have you ever considered ConsoleZ?

[dscho]

@dscho have you ever considered ConsoleZ?

Yes. See #96 (comment)

[dscho]

@kblees could you check whether the current master works for you regarding password reading? I am pretty certain that I addressed this correctly now via d56d2f8

[kblees]

@dscho Yes, d56d2f8 works fine for me, thanks.

The shell solution can even handle non-ASCII usernames and passwords correctly. With TERM=cygwin in a native console window, these are read in GetConsoleCP() encoding, which is cp850 for me. So we could actually drop the old $CONIN / $CONOUT stuff, right?

And of course the MSYS_TTY_HANDLES patch could be removed from MSys2.dll

[kblees]

By all means please bundle winpty;

...and probably also the /usr/bin/start script, to be able to run native programs in a real native console window, if all else fails...

[dscho]

By all means please bundle winpty;

...and probably also the /usr/bin/start script, to be able to run native programs in a real native console window, if all else fails...

git-for-windows/build-extra@36be391 and git-for-windows/build-extra@fd0095c.

The shell solution can even handle non-ASCII usernames and passwords correctly. With TERM=cygwin in a native console window, these are read in GetConsoleCP() encoding, which is cp850 for me.

Great! Thanks for testing.

So we could actually drop the old $CONIN / $CONOUT stuff, right?

I am actually reluctant to do that. There is a really good reason why the shell stuff is guarded behind an if (starts_with(getenv("TERM"), "xterm")) test: I am really uncomfortable with enabling this for anything but mintty.

And of course the MSYS_TTY_HANDLES patch could be removed from MSys2.dll

Whoa, not so fast. First I need to release a new mingw-w64-git package... Then I can remove that hack. But yes, my urge to remove that hack was the primary reason why I spent a considerable effort to rethink the compat/terminal.c code.

[kblees]

I am really uncomfortable with enabling this for anything but mintty.

Even though it works much better than the CONIN$ / CONOUT$ stuff? See PR #141 how I think this could be done.

But perhaps I'm missing a really important use case (MSVC perhaps?).

[elyscape]

Even though it works much better than the CONIN$ / CONOUT$ stuff?

Does it work even when invoked from git-cmd?

[dscho]

Let's discuss the CONIN$/CONOUT$ issue in #141, not here.

[sschuberth]

Another item for the list of "Issues with other (native) console programs" from the first post: For many native console programs the is-a-tty detection to determine whether they should output color does not work as expected inside mintty, i.e. no color is used. Examples are the Rake / Ruby and Gradle. Note that I'm not talking about ANSI color codes, but color using SetConsoleTextAttribute() etc.

[dscho]

For many native console programs the is-a-tty detection to determine whether they should output color does not work as expected inside mintty

It works if you use winpty with the command, which is why I ship it with Git for Windows.

Anyway, with the new wizard page of the installer that lets you choose the ConHost-based Git Bash, I consider this ticket to be addressed, because it was never about MinTTY per se, but always about having MinTTY as the only option for Git Bash.

[dscho]

I heard no objections to my statement that this ticket is addressed by the new wizard page that let's the user choose whether to use MinTTY or ConHost for Git Bash.

[linquize]

Current User, Current directory, git branch name are colored.
ls, git diff are colored.
But no color with grunt. Is it the same problem?

[maoueh]

@linquize probably yeah, any program that query console to know if it is interactive will get a non-interactive result and will usually disable colors. As a workaround, most program have a switch to force color. In those cases, I set up an alias.

Otherwise, you need to use winpty on the executable directly, in this case, it would be node.exe. I "hacked" a bit node installation by adding a node script before my node root path reading like:

#!/usr/bin/bash

exec winpty "$NODE_ROOT/node.exe" "$@"

For git, it works out of the box because git executable has been taught to determine if console is interactive using a different trick if it determines that execution is being performed on an MSYS2 base system.