async git status update blocks diagnostic and nvim quit

[cseickel]

Hi @cseickel

Update this issue:

1. Open any file in this bare repo with a very large working treee
2. Open neotree window
3. Type something in the file buffer. The diagnostics information will not be shown (seems that the git running in background asynchronously to fetch status blocks the diagnostics). Then we run :qa! to quit, but vim will be stuck there for a while. In shell, an error appears (screenshot below)
   

I am using the latest nightly neovim and latest neotree with default config. I also found nvim-tree is very fast when fetching git status and diagnostics no matter how large the repo is. Probably you could investigate it and find a workaround.

Thank you.

Originally posted by @rockyzhang24 in #147 (comment)

[cseickel]

@rockyzhang24 I moved this to a new issue. I guess the first priority is to make sure that the status update can be cancelled so it doesn't block nvim from exiting.

Blocking diagnostics happens because I limit the number of refreshes that can happen, so it's intentional that it won't start a new one while another one is still in progress. I could think about a git status update as a separate process from the rest of the refresh which has its own separate throttling.

[rockyzhang24]

Okay. Thank you so much. As you know (hope you could remember me 😀), I am using a bare repo to manager my dotfiles as we deeply discussed in the original issue. This problem troubles me a lot and hope it could be easily to fix. Thanks again.

[rockyzhang24]

Hi @cseickel
Just FYI, I noticed that nvim-tree provides an option git.timeout and if the asynchronous git related operation is longer than timeout, the process will be killed. Probably this can be used as a workaround but it is not ideal. I don't know whether there could be a perfect solution so that no matter how large the repo is, the git status can be fetched and showed without any blocking and freezing.

Thanks.

[cseickel]

@rockyzhang24 I have done some work to reduce the amount of refreshes that happen as part of other issues that may also help with your problem. In addition to that, I just added in the ability to cancel out of the git status async update when nvim is closing.

I haven't really tested it because I don't naturally have any repo slow enough to hit these issues.

Can you try teh latest and let me know if there is an improvement?

[rockyzhang24]

Thank you so much for your hard work 👍

I just updated neo-tree. The problem about blocking nvim quit is solved. Now when I run :qa! immediately after I open neo-tree window, nvim will quit right away.

However, for the problem on blocking diagnostics, it is tricky. Sometimes the diagnostic will show immediately, but sometimes it won't, i.e., I have to wait a couple of seconds.

I will demonstrate my situation and test in details below.

I have a bare repo like this git --git-dir=/Users/rockyzhang/dotfiles/ --work-tree=/Users/rockyzhang/, i.e., a bare repo with a very large working tree. Then I run the test below 4 times in a row and I attached a demo video.

1. open /Users/rockyzhang/.config/nvim/init.vim
2. run :Neotree immediately to open neo-tree window
3. in init.vim buffer, type something.

Screen.Recording.2022-04-30.at.21.41.02-2.mov

It can be observed that in the first run and the 4th run, diagnostic information was blocked and I had to wait a few seconds, but the cursor movement was still smooth, however, in the 2nd and 3rd run, the diagnostic information showed immediately. That's so weird. Any thoughts?

I know my case is rather niche. I apologize sincerely if I bother you too much. I think more or less my case can be a corner case of neo-tree, and could help neo-tree more robust and shining 🤣.

Thanks a lot.

[cseickel]

I didn't realize it was blocking all diagnostics, I originally assumed it was just not updating diagnostics in the tree. I guess it could be filling the queue that the vim.schedule function sends work to.

I can try introducing pauses into the job so that other background processes can work. Can you tell me how many files each of these commands will return for you?

git diff --staged --name-status
git diff --name-status
git ls-files --exclude-standard --others

[cseickel]

In preparation for further updates, I refactored the git status async code a little. It's possible that those changes could have a positive (or negative) effect on your problem. Could you give it a try and let me know?

If it is not magically fixed, the file counts will help me to determine a good batch size if I try to break up the work and add pauses.

[rockyzhang24]

Hi @cseickel , I just got up. I will turn on my laptop and run those commands now.

[rockyzhang24]

So you mean run those commands in my bare repo, right? The commands I actually should run are

git --git-dir=/Users/rockyzhang/dotfiles/ --work-tree=/Users/rockyzhang/ diff --staged --name-status
git --git-dir=/Users/rockyzhang/dotfiles/ --work-tree=/Users/rockyzhang/ diff --name-status
git --git-dir=/Users/rockyzhang/dotfiles/ --work-tree=/Users/rockyzhang/ ls-files --exclude-standard --others

1. After running git --git-dir=/Users/rockyzhang/dotfiles/ --work-tree=/Users/rockyzhang/ diff --staged --name-status, I got

2. For git --git-dir=/Users/rockyzhang/dotfiles/ --work-tree=/Users/rockyzhang/ diff --name-status, I got

3. For git --git-dir=/Users/rockyzhang/dotfiles/ --work-tree=/Users/rockyzhang/ ls-files --exclude-standard --others, it output lots of files. I piped into wc -l and the number was 447430.

Thank you.

[cseickel]

3. it output lots of files. I piped into wc -l and the number was 447430.

🤯

My git status is not adding those extra arguments. If they are needed then maybe we have another issue. What happens if you just run the commands I gave you in the directory that neo-tree is working in?

[rockyzhang24]

Because the bare repo is my whole $HOME, thus in my nvim config, I set the environment variables $GIT_DIR = "~/dotfiles" and $GIT_WORK_TREE = "~" when a file under $HOME is open, and reset these environment variables when a file under a normal git repo is open. By this setting, other git related plugins like gitsigns and fugitive will recognize the tracked files in the bare repo. I attached this part of my nvim config below:

" I manage my dotfiles using a bare repository. To make Vim recognize them and git related plugins
" work on them, the environment variables should be set to indicate the locations of git-dir and
" work-tree when we enter the dotfile buffer. Don't forget to reset them when we enter other buffers,
" otherwise the normal repository will not be recognized.
function! s:SetGitEnv() abort
  let cur_file = expand('%')
  " Only set the Git env for the buffer containing a real file
  if !filereadable(cur_file)
    return
  endif
  let git_dir = expand('~/dotfiles')
  let work_tree = expand('~')
  let jib = jobstart(["git", "--git-dir", git_dir, "--work-tree", work_tree, "ls-files", "--error-unmatch", cur_file])
  let ret = jobwait([jib])[0]
  if ret == 0
    let $GIT_DIR = git_dir
    let $GIT_WORK_TREE = work_tree
  else
    unlet $GIT_DIR
    unlet $GIT_WORK_TREE
  endif
endfunction

augroup personal
  autocmd!
  autocmd BufNewFile,BufRead,BufEnter * call s:SetGitEnv()
augroup END

After setting this, when I open a file under $HOME, git related commands in nvim will use those environment variables, so if I run git status, it means git --git-dir=/Users/rockyzhang/dotfiles/ --work-tree=/Users/rockyzhang/ status. However, if I open a file under a normal git repo, git commands in nvim become normal.

If I run the git commands under $HOME in shell directly, because those environment variables are not set, git will say this is not a git repo, so arguments --git-dir and --work-tree should be used. Or set the environment variables in zshrc.

[cseickel]

Got it. I am adding the ability to batch the processing of these items and will include some options to fine tune the amount per batch and pause between batches.

[cseickel]

The latest commit changes the processing strategy from scheduling each individual line, to processing them in batches with small delays in between each batch. Here are the new options you can use to control this:

    require("neo-tree").setup({
      git_status_async = true,
      -- These options are for people with VERY large git repos
      git_status_async_options = {
        batch_size = 1000, -- how many lines of git status results to process at a time
        batch_delay = 10,  -- delay in ms between batches. Spreads out the workload to let other processes run.
        max_lines = 100000, -- How many lines of git status results to process. Anything after this will be dropped.
                            -- Anything before this will be used. The last items to be processed are the untracked files.
      },
    })

You can tweak those options to figure out what works best for you. If you want to actually process everything, you'll need to increase your max_lines to at least 450000. It's also possible that you don't really need to process all of those untracked files and you can fix everything by setting max_lines to 1000.

Give it a try and let me know how it works out and what settings you end up with.

[rockyzhang24]

Sure. I’m outside and gonna try it tonight as soon as I get home. Thank you.

[rockyzhang24]

Hi, sorry for my late response. I just tested it and the issue still exists.

Firstly, I used the default values, i.e., 1000, 10, and 100000.

Screen.Recording.2022-05-02.at.14.58.19.mov

The issue happened in the last run, i.e., the diagnostic information was not displayed immediately and I have to wait for a couple of seconds.

Next, I change the values to 100, 10, and 1000.

Screen.Recording.2022-05-02.at.15.03.08.mov

Still see my last run please. The diagnostic info was blocked again. Also, :qa! is still kinda sluggish.

What's more, FYI I tried nvim-tree again and I found the git status and diagnostics are displayed immediately after I open nvim-tree window, or type something in the buffer, and cannot feel any delay. And no sluggishness for :qa!. See the demo below.

Screen.Recording.2022-05-02.at.15.15.49.mov

Actually I am not care about untracked file for my bare repo. In my bare repo config, I have

[status]
	showUntrackedFiles = no

Thanks a lot.

[cseickel]

@rockyzhang24 It would be helpful if there was actually a modified file in these tests so that I can see when the git status update is complete. Could you first save a change to the README or init.vim, and create a new file but don't add it to the index so they show a status in the tree, and then repeat these tests?

[rockyzhang24]

Sure. I added a new line at the end of init.vim, and touched a new file named new_file.

Screen.Recording.2022-05-02.at.15.35.17.mov

[cseickel]

Actually I am not care about untracked file for my bare repo. In my bare repo config, I have

[status]
	showUntrackedFiles = no

This option is not respected by neo-tree and I think the only real solution is to start respecting it. I just need to figure out a good way to determine if the option is set and if so, skip that check for untracked files.

[rockyzhang24]

I think we can have an option to enable/disable the check for untracked files.

What's more, I tested nvim-tree further. As I mentioned before, nvim-tree has an option git.timeout to set a period and it will kill the git process after this period if git takes too long. Its default value is 400ms. I deleted showUntrackedFiles = no from my bare repo config, i.e., let git to check all untracked files. Then I opened nvim-tree window. It is opened immediately but no git status was shown because the git process was killed. Next, I increased the timeout value to 1000000 (large enough). Then I tried to open nvim-tree window but it would take a couple of seconds to open and nvim was blocked.

So I think for neo-tree we can add such a timeout option as well. Once we add these two options (one is enable/disable checking untracked file, and the other is timeout), we could freely control the behavior for large git repo. What do you think?

Thank you very much.

[cseickel]

I really don't like the timeout option, that makes it unpredictable by design. To me if you show git status you have to show it every time. I would hate to operate under the impression that there were no changed items and find out later that there were but my tree is unreliable. I'd rather it be slow and predictable than fast and inaccurate.

The least I will do is offer an option to disable showing untracked files. A better choice is to add an option to disable untracked files for specific repos.

My preferred option is to obey what you have configured in your gitconfig for each repo. I think that is very doable. Now that I know about the showUntrackedFiles option, that should solve everything.

[rockyzhang24]

I would hate to operate under the impression that there were no changed items and find out later that there were but my tree is unreliable

It makes sense. I think this is more reasonable.

My preferred option is to obey what you have configured in your gitconfig for each repo. I think that is very doable. Now that I know about the showUntrackedFiles option, that should solve everything.

This is really nice and what I want.

Thank you.

[cseickel]

The latest in main will actually skip untracked files if showUntrackedFiles is set to no for the current repo. If this works, git status should be fast for you now.

[rockyzhang24]

Thank you. I tried it and now the git status is reflected on the tree immediately after I open the tree, and the diagnostic information and nvim quit are not blocked any more.