Async scan is blocking?

[musjj]

I have async_directory_scan set to always in my configuration, but toggling a neo-tree window still blocks neovim for me.
I tested by opening neo-tree in a very large directory on my slow hard drive, and neovim blocks, until the neo-tree window is fully rendered.
I looked through the code for async_scan, but it seems completely synchronous to me.
Anything I'm missing here?

[musjj]

Looked through it again, and it seems that it uses that async API from luv, so I'm not sure why it blocks.

[cseickel]

If you look here: https://github.com/nvim-neo-tree/neo-tree.nvim/wiki/Troubleshooting

It will describe how to enable logging. You can then tail the log file in a separate window for clues as to what is happening during the hang.

It's always possible that the hang is from some other plugin that is responding to the new text being added to the buffer.

[cseickel]

It is possible that checking git ignore is the hang. Is the large directory a git project?

[musjj]

No, it was a plain directory of ~/Downloads. Aren't git operations supposed to be async by default anyways?
I don't think there's any interference of other plugins, it blocked for quite a long time and I don't see any plugins that could be responsible for this.
I also tried out https://github.com/lambdalisue/fern.vim on the same directory and it felt completely asynchronous to me. After running the command to open the tree, I could freely edit and operate however I want as the tree loads in the background.

[musjj]

I also tried to log it, and I can't really find any thing suspicious, but I snipped some parts where there is a big gap between the log times:

[TRACE 2022-11-17 11:11:19 AM] ...ree.nvim/lua/neo-tree/sources/filesystem/lib/fs_scan.lua:117: async_scan:  C:\Users\user\Downloads
[TRACE 2022-11-17 11:11:42 AM] ...pack\packer\opt\neo-tree.nvim/lua/neo-tree/git/utils.lua:13: GIT ROOT ERROR  {}
[TRACE 2022-11-17 11:11:42 AM] ...ack\packer\opt\neo-tree.nvim/lua/neo-tree/git/status.lua:208: status_async: not a git folder:  C:\Users\user\Downloads

[trace 2022-11-17 11:11:42 am] ...t\neo-tree.nvim/lua/neo-tree/sources/filesystem/init.lua:32: follow called
[trace 2022-11-17 11:12:11 am] ...k\packer\opt\neo-tree.nvim/lua/neo-tree/events/queue.lua:213: firing event:  vim_win_enter  with args:  {
  afile = "tmp.txt"
}

[TRACE 2022-11-17 11:12:17 AM] ...k\packer\opt\neo-tree.nvim/lua/neo-tree/events/queue.lua:213: Firing event:  after_render  with args:  {
  async_directory_scan = "always",
  bind_to_cwd = true,
  bufnr = 3,
  commands = {
    add = <function 1>,
    add_directory = <function 2>,
    cancel = <function 3>,
    clear_filter = <function 4>,
    close_all_nodes = <function 5>,
    close_node = <function 6>,
    close_window = <function...
[TRACE 2022-11-17 11:12:24 AM] ...k\packer\opt\neo-tree.nvim/lua/neo-tree/events/queue.lua:213: Firing event:  vim_buffer_enter  with args:  {
  afile = "neo-tree.nvim.log"
}

[TRACE 2022-11-17 11:12:24 AM] ...t\neo-tree.nvim/lua/neo-tree/sources/filesystem/init.lua:32: follow called
[TRACE 2022-11-17 11:12:42 AM] ...k\packer\opt\neo-tree.nvim/lua/neo-tree/events/queue.lua:213: Firing event:  vim_win_enter  with args:  {
  afile = "neo-tree.nvim.log"
}

[cseickel]

Are you on Windows?

[musjj]

Yes, I'm on Windows.

[musjj]

Alright, I figured out the problem.
neo-tree uses uv.fs_scandir, which is async, but the function that is used to iterate on the returned handle (uv.fs_scandir_next) has no async version.
The solution is to use uv.fs_opendir and uv.fs_readdir, because both of them has an async version.
So the funny thing is that this means that neo-tree was never async no matter how it is configured, but for some reason no one noticed...
But anyways, I made a rough patch for async_scan that makes it actually async. I might clean it up into a PR later if no else does.

Rough patch

local function async_scan(context, path)
  log.trace("async_scan: ", path)
  -- prepend the root path
  table.insert(context.paths_to_load, 1, path)

  context.directories_scanned = 0
  context.directories_to_scan = #context.paths_to_load

  context.on_exit = vim.schedule_wrap(function()
    job_complete(context)
  end)

  -- from https://github.com/nvim-lua/plenary.nvim/blob/master/lua/plenary/scandir.lua
  local function read_dir(current_dir, ctx)
    local function on_fs_opendir(err, dir)
      if err then
        log.error(current_dir, ": ", err)
      else
        local function on_fs_readdir(err, entries)
          if err then
            log.error(current_dir, ": ", err)
          else
            if entries then
              for _, entry in ipairs(entries) do
                local success, item = pcall(
                  file_items.create_item,
                  ctx,
                  utils.path_join(current_dir, entry.name),
                  entry.type
                )
                if success then
                  if ctx.recursive and item.type == "directory" then
                    ctx.directories_to_scan = ctx.directories_to_scan + 1
                    table.insert(ctx.paths_to_load, item.path)
                  end
                else
                  log.error("error creating item for ", path)
                end
              end

              uv.fs_readdir(dir, on_fs_readdir)
            else
              uv.fs_closedir(dir)
              on_directory_loaded(ctx, current_dir)
              ctx.directories_scanned = ctx.directories_scanned + 1
              if ctx.directories_scanned == #ctx.paths_to_load then
                ctx.on_exit()
              end
            end
          end
        end

        uv.fs_readdir(dir, on_fs_readdir)
      end
    end

    uv.fs_opendir(current_dir, on_fs_opendir)
  end

  for i = 1, context.directories_to_scan do
    read_dir(context.paths_to_load[i], context)
  end
end

[cseickel]

So the funny thing is that this means that neo-tree was never async no matter how it is configured, but for some reason no one noticed...

Huh. I think that the original implementation allowed other work in between because it scanned one folder at a time in a chain. So it wasn't fully async, but effectively good enough. Then it was changed to scan them in parallel to make the whole process faster, which accidentally caused it to be effectively blocking.

Anyway, good catch! Please do submit a PR if you have the time.

[nyngwang]

@musjj Do you know whether the issue is related to this change #619? Thanks in advance!