gitea hanging with git 2.43.1

[tycho]

Description

I noticed that gitea was hanging when attempting to view commits (via /<team>/<repo>/commit/<hash>), but only with git 2.43.1 installed. "ps" showed a bunch of cat-file --batch-check, cat-file --batch, and check-attr invocations of git, which were making no progress. I attached strace to one of the git child processes and it was simply blocked on reading from stdin. I downgraded to git 2.43.0 and gitea started being able to view commits and index repositories.

I tried cranking the log level up to Debug and other than reporting "router: slow" messages for the commit URLs, it didn't seem to have any complaints, e.g. here's what happens on attempting to view a commit URL:

Feb 12 01:25:10 korriban gitea[14813]: 2024/02/12 01:25:10 ...dules/git/command.go:290:Run() [D] git.Command.RunDir(/var/lib/gitea/repos/organization/repo.git): /usr/bin/git -c protocol.version=2 -c credential.helper= read-tree ca9da37ae9c977b9ab06fe21d515b1cd9f1ba563
Feb 12 01:25:10 korriban gitea[14813]: 2024/02/12 01:25:10 ...dules/git/command.go:290:Run() [D] git.Command.RunDir(/var/lib/gitea/repos/organization/repo.git): /usr/bin/git -c protocol.version=2 -c credential.helper= check-attr --stdin -z --cached linguist-vendored linguist-generated linguist-language gitlab-language
Feb 12 01:25:13 korriban gitea[14813]: 2024/02/12 01:25:13 ...eb/routing/logger.go:68:func1() [I] router: polling   GET /user/events for [2604:4080:xxxx]:0, elapsed 3211.7ms @ events/events.go:18(events.Events)
Feb 12 01:25:14 korriban gitea[14813]: 2024/02/12 01:25:14 ...eb/routing/logger.go:68:func1() [W] router: slow      GET /Organization/Repo/commit/ca9da37ae9c977b9ab06fe21d515b1cd9f1ba563 for [2604:4080:xxxx]:0, elapsed 3675.9ms @ repo/commit.go:271(repo.Diff)

And again this only seems to happen with git 2.43.1 installed. Any idea what could be going on? Is this an upstream git regression?

Gitea Version

All

Git Version

2.43.1

Workaround

Downgrade git to <=2.43.0 or upgrade to >= 2.43.2

[Bodenhaltung]

Same here but as DB sqlite3, updated Git to git 2.43.1-1 and running Gitea 1.21.4-1 (Gitea version 1.21.5 built with GNU Make 4.4.1, go1.21.6 : bindata, sqlite, sqlite_unlock_notify, pam).

Feb 12 11:36:37 srv01 gitea[1566129]: 2024/02/12 11:36:37 ...eb/routing/logger.go:68:func1() [W] router: slow      GET /**removed**/**removed**/commit/38c1d2267aaf3d03675b3700b06226d7e9c4e5ec for 127.0.0.1:0, elapsed 3046.2ms @ repo/commit.go:270(repo.Diff)
Feb 12 11:42:34 srv01 gitea[1566129]: 2024/02/12 11:42:34 ...it/repo_attribute.go:306:func2() [E] Unable to open checker for 38c1d2267aaf3d03675b3700b06226d7e9c4e5ec. Error: failed to run attr-check. Error: context deadline exceeded
Feb 12 11:42:34 srv01 gitea[1566129]: Stderr:
Feb 12 11:42:34 srv01 gitea[1566129]: 2024/02/12 11:42:34 ...eb/routing/logger.go:102:func1() [I] router: completed GET /**removed**/**removed**/commit/38c1d2267aaf3d03675b3700b06226d7e9c4e5ec for 127.0.0.1:0, 200 OK in 360178.3ms @ repo/commit.go:270(repo.Diff)

[wxiaoguang]

I think it is a regression bug of Git. I have sent an email to the git mail list, quote it below.

(the only workaround at the moment is to downgrade the git to <=2.43.0 , well, I am not sure whether we should propose a dirty patch to bypass the regression).

---

Incorrect GIT_FLUSH behavior in 2.43.1 (regression and breaking)

Ref: git/git@556e680

If GIT_FLUSH=true, it should mean to "do the flush". But that commit made skip_stdout_flush=true when GIT_FLUSH=true.

And by the way, only accepting GIT_FLUSH=true is quite breaking, it drops the compatibility of GIT_FLUSH=1: it causes the existing programs which depend on the "flushing(GIT_FLUSH=1)" behavior would hang forever if they use the new git binary, because the program won't see any flushed output .... (not a problem if the filpping bug could be fixed)

[tycho]

I am not sure whether we should propose a dirty patch to bypass the regression

Probably shouldn't, as long as the git team resolves it quickly with a revert/patch and new release.

But is there an opportunity for better logging on the gitea side for this issue? Even with Debug level logging there wasn't much to report aside from the "router: slow" messages, which were at least somewhat helpful in figuring it out. But maybe there should be something more here?

[wxiaoguang]

But is there an opportunity for better logging on the gitea side for this issue?

IMO there is no general/easy way to do the log for such hanging at the moment (feel free to propose some approaches), because the problem is that Gitea couldn't read the git program's output. Actually, there are many clues, eg: the "running process" (in the admin panel), the log about "failed to run attr-check". The real difficult step is how to figure out why it hangs.

Probably shouldn't, as long as the git team resolves it quickly with a revert/patch and new release.

Maybe we could do this: Refactor git version functions and check compatibility #29155

Tell end users to upgrade or downgrade the git if there is some known issues.

[VitaliyAT]

I have the same problems with git version 2.43.0.windows.1

[milosivanovic]

@tycho @wxiaoguang thanks for figuring this out - the new hangs were baffling me. I didn't immediately use gitea after updating git from 2.43.0 to 2.43.1, so I didn't correlate updating git with the hangs that occurred while trying to view diffs until I found this issue.

I am running gitea behind an nginx reverse proxy, and after clicking a commit to view a diff, nginx returned HTTP 504 after gitea failed to respond within 60s. @wxiaoguang It would be useful if gitea could be configured to time out the git command after a shorter time so that it could return a meaningful error for users who are still trying to figure this issue out. Would that be possible?

[wxiaoguang]

I am running gitea behind an nginx reverse proxy, and after clicking a commit to view a diff, nginx returned HTTP 504 after gitea failed to respond within 60s. @wxiaoguang It would be useful if gitea could be configured to time out the git command after a shorter time so that it could return a meaningful error for users who are still trying to figure this issue out. Would that be possible?

Actually there is one: [git.timeout].DEFAULT https://github.com/go-gitea/gitea/blob/6d4dc16c726dd0be8d0f56405ba396d44dfd04ac/custom/conf/app.example.ini#L708C2-L708C15

But I think you shouldn't touch it ..... because it's difficult to fine-tune a proper timeout for various cases at the moment.

[wxiaoguang]

I have the same problems with git version 2.43.0.windows.1

Does an old version (say , git 2.42.x) work for you? If no, then it could be a different problem (not the GIT_FLUSH regression)

[tycho]

Also 2.43.2 is out, which fixes this bug. Just have to wait for vendors to build binary packages for it.