EMFILE: too many open files

[MunifTanjim]

When too many nodes are expanded and then neo-tree is toggled, getting this:

[Neo-tree ERROR] debounce  filesystem_navigate  error:  .../.local/share/nvim/lazy/plenary.nvim/lua/plenary/job.lua:404: Failed to spawn process: {                                                                                                                                         
  _additional_on_exit_callbacks = {},                                                                                                                                                                                                                                                       
  _shutdown_check = <userdata 1>,                                                                                                                                                                                                                                                           
  _stderr_results = {},                                                                                                                                                                                                                                                                     
  _stdout_results = {},                                                                                                                                                                                                                                                                     
  _user_on_exit = <function 1>,                                                                                                                                                                                                                                                             
  _user_on_start = <function 2>,                                                                                                                                                                                                                                                            
  args = { "-C", "/Users/munif.tanjim/Dev/github/qmk/qmk_firmware/keyboards/tunks", "check-ignore", "--stdin" },                                                                                                                                                                            
  command = "git",                                                                                                                                                                                                                                                                          
  enable_handlers = true,                                                                                                                                                                                                                                                                   
  enable_recording = true,                                                                                                                                                                                                                                                                  
  interactive = true,                                                                                                                                                                                                                                                                       
  pid = "EMFILE: too many open files",                                                                                                                                                                                                                                                      
  stderr = <userdata 2>,                                                                                                                                                                                                                                                                    
  stdin = <userdata 3>,                                                                                                                                                                                                                                                                     
  stdout = <userdata 4>,                                                                                                                                                                                                                                                                    
  user_data = {},                                                                                                                                                                                                                                                                           
  writer = { "/Users/munif.tanjim/Dev/github/qmk/qmk_firmware/keyboards/tunks/ergo33" },                                                                                                                                                                                                    
  <metatable> = <1>{                                                                                                                                                                                                                                                                        
    __index = <table 1>,                                                                                                                                                                                                                                                                    
    _create_uv_options = <function 3>,                                                                                                                                                                                                                                                      
    _execute = <function 4>,                                                                                                                                                                                                                                                                
    _pipes_are_closed = <function 5>,                                                                                                                                                                                                                                                       
    _prepare_pipes = <function 6>,                                                                                                                                                                                                                                                          
    _reset = <function 7>,                                                                                                                                                                                                                                                                  
    _shutdown = <function 8>,                                                                                                                                                                                                                                                               
    _stop = <function 9>,                                                                                                                                                                                                                                                                   
    add_on_exit_callback = <function 10>,                                                                                                                                                                                                                                                   
    after = <function 11>,                                                                                                                                                                                                                                                                  
    after_failure = <function 12>,                                                                                                                                                                                                                                                          
    after_success = <function 13>,                                                                                                                                                                                                                                                          
    and_then = <function 14>,                                                                                                                                                                                                                                                               
    and_then_on_failure = <function 15>,                                                                                                                                                                                                                                                    
    and_then_on_failure_wrap = <function 16>,                                                                                                                                                                                                                                               
    and_then_on_success = <function 17>,                                                                                                                                                                                                                                                    
    and_then_on_success_wrap = <function 18>,                                                                                                                                                                                                                                               
    and_then_wrap = <function 19>,                                                                                                                                                                                                                                                          
    chain = <function 20>,                                                                                                                                                                                                                                                                  
    chain_status = <function 21>,                                                                                                                                                                                                                                                           
    co_wait = <function 22>,                                                                                                                                                                                                                                                                
    is_job = <function 23>,                                                                                                                                                                                                                                                                 
    join = <function 24>,                                                                                                                                                                                                                                                                   
    new = <function 25>,                                                                                                                                                                                                                                                                    
    pid = <function 26>,                                                                                                                                                                                                                                                                    
    result = <function 27>,                                                                                                                                                                                                                                                                 
    send = <function 28>,                                                                                                                                                                                                                                                                   
    shutdown = <function 29>,                                                                                                                                                                                                                                                               
    start = <function 30>,                                                                                                                                                                                                                                                                  
    stderr_result = <function 31>,                                                                                                                                                                                                                                                          
    sync = <function 32>,                                                                                                                                                                                                                                                                   
    wait = <function 33>                                                                                                                                                                                                                                                                    
  }                                                                                                                                                                                                                                                                                         
}                                                                                                                                                                                                                                                                                           
stack traceback:                                                                                                                                                                                                                                                                            
^I.../.local/share/nvim/lazy/plenary.nvim/lua/plenary/job.lua:404: in function '_execute'                                                                                                                                                                                                   
^I.../.local/share/nvim/lazy/plenary.nvim/lua/plenary/job.lua:447: in function 'start'                                                                                                                                                                                                      
^I...are/nvim/lazy/neo-tree.nvim/lua/neo-tree/git/ignored.lua:138: in function 'mark_ignored'                                                                                                                                                                                               
^I...ree.nvim/lua/neo-tree/sources/filesystem/lib/fs_scan.lua:119: in function 'job_complete'                                                                                                                                                                                               
^I...ree.nvim/lua/neo-tree/sources/filesystem/lib/fs_scan.lua:352: in function 'sync_scan'                                                                                                                                                                                                  
^I...ree.nvim/lua/neo-tree/sources/filesystem/lib/fs_scan.lua:350: in function 'sync_scan'                                                                                                                                                                                                  
^I...ree.nvim/lua/neo-tree/sources/filesystem/lib/fs_scan.lua:350: in function 'sync_scan'                                                                                                                                                                                                  
^I...ree.nvim/lua/neo-tree/sources/filesystem/lib/fs_scan.lua:350: in function 'sync_scan'                                                                                                                                                                                                  
^I...ree.nvim/lua/neo-tree/sources/filesystem/lib/fs_scan.lua:350: in function 'sync_scan'                                                                                                                                                                                                  
^I...ree.nvim/lua/neo-tree/sources/filesystem/lib/fs_scan.lua:350: in function 'sync_scan'              

^I...
^I...y/neo-tree.nvim/lua/neo-tree/sources/filesystem/init.lua:164: in function '_navigate_internal'
^I...y/neo-tree.nvim/lua/neo-tree/sources/filesystem/init.lua:184: in function <...y/neo-tree.nvim/lua/neo-tree/sources/filesystem/init.lua:183>
^I[C]: in function 'pcall'
^I...cal/share/nvim/lazy/neo-tree.nvim/lua/neo-tree/utils.lua:117: in function 'debounce'
^I...y/neo-tree.nvim/lua/neo-tree/sources/filesystem/init.lua:183: in function 'navigate'
^I...nvim/lazy/neo-tree.nvim/lua/neo-tree/sources/manager.lua:488: in function 'navigate'
^I...re/nvim/lazy/neo-tree.nvim/lua/neo-tree/command/init.lua:177: in function 'do_show_or_focus'
^I...re/nvim/lazy/neo-tree.nvim/lua/neo-tree/command/init.lua:135: in function 'execute'
^I...re/nvim/lazy/neo-tree.nvim/lua/neo-tree/command/init.lua:143: in function '_command'
^I[string ":lua"]:1: in main chunk

[cseickel]

I am aware of this possibility. The problem is that we spawn a new scanner for every sub folder. We should really have a limit on the number of fs scanning jobs and add in a queue to manage any folders that need scanning beyond that number.

There is a refactor already under way for this section of the code. If no one else volunteers, I may make this change after that lands.

[cseickel]

I would like to work on this issue but I have never actually hit it myself and don't have a way to recreate it. If someone could provide me with reliable steps to reproduce it I will work on it.

[MunifTanjim]

When I created the issue I was testing with https://github.com/qmk/qmk_firmware and expanded a lot of folders inside the https://github.com/qmk/qmk_firmware/tree/master/keyboards directory.

Then I closed neotree, and opened it again - that triggered the error.