[BUG] Running make html-noplot yields errors.

[phonokoye]

Add Link

I ran the following command about 10 hours ago, around 12:20:00 utc and it gave me errors. (I am being specific about the time, because I was unable to find a release that I could point to).
git clone --depth 1 https://github.com/pytorch/tutorials.git

Describe the bug

What errors did you encounter?

generating gallery for beginner... [  6%] saving_loading_models.py
Extension error (sphinx_gallery.gen_gallery):
Handler <function generate_gallery_rst at 0x000001C0A8C3AB00> for event 'builder-inited' threw an exception (exception: Can't pickle <function call_fn at 0x000001C088543010>: attribute lookup call_fn on __main__ failed)
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Python\Python310\lib\multiprocessing\spawn.py", line 107, in spawn_main
    new_handle = reduction.duplicate(pipe_handle,
  File "C:\Python\Python310\lib\multiprocessing\reduction.py", line 79, in duplicate
    return _winapi.DuplicateHandle(
OSError: [WinError 6] The handle is invalid
make: *** [html-noplot] Error 2

What did you expect to happen?

As stated in the README.md file, I expected a basic html version of the tutorial to be built at _build/html

Steps to Reproduce the error

1. Run the git command below
   git clone --depth 1 https://github.com/pytorch/tutorials.git

2. Run pip install -r .ci/docker/requirements.txt. I am aware the instruction was to pip install -r requirements.txt. But
   I keep encountering the errors below, so I improvised.

ERROR: Invalid requirement: '.ci/docker/requirements.txt': Expected package name at the start of dependency specifier
    .ci/docker/requirements.txt
    ^ (from line 1 of requirements.txt)

3. Run make html-noplot. For this one, I gnuWin32 make. This is what is available on Windows.

   I noticed that this error is similar to that found when I run re.compile('\c'). I am familiar with this scenario and so I looked further and traced the error to the code here. I was able to move on from this error by modifying my local version of the code to
   SPHINX_SHOULD_RUN = "|".join(get_files_for_sphinx()).replace('\\', '\\\\')
   I want to note that I do not feel confident in that action because I notice that code was last modified 2 years ago, unless I have a wrong interpretation of what the "2 years ago" that I see around there means. It was last modified 2 years ago! That means that working tutorials have been built with that piece of code. This makes me feel very strongly that something is wrong with my setup. But I resisted raising any issues because I considered that it might not be worth it to distract the attention of our dear conscientious developers whose efforts to maintain this codebase does not go unnoticed.

4. Run make html-noplot once more.
   The error above appears. I look at the error and I see multiprocessing.py there. I do not know how to do anything with code that runs on more than one thread or process. I would appreciate knowing what I have done wrong in my environment because surely the code in this repository works as it has been tested as required.

Describe your environment

Environment

• Python 3.10.5
• pip 25.1.1
• All commands were run in the top directory of the cloned repository
• All pip-installing was done in a fresh virtual environment created using venv and located in the top directory of the cloned repository. The command used for that was python -m venv doc-env.
• GPU (not cuda): Intel Iris Xe (Not sure this is relevant)

[svekars]

Hi, @phonokoye . Virtualenv should work but you can also try conda. I just ran this on my machine and it worked w/o issues:

1. Install miniconda as described: https://www.anaconda.com/docs/getting-started/miniconda/install
2. From the root of the tutorials repo

   conda create --name myenv python=3.10
   conda activate myenv
   pip install -r requirements.txt
   make-html noplot 
   

We will update instructions for conda as well.

[svekars]

sorry, I just realized that you are on Win. Can you use something like WSL?

[phonokoye]

Hello, @svekars. Thank you for responding. Regarding your suggestion above, you are going to think that I am being a stubborn billy goat.

I cannot change environment at this point. I will be using a python and venv environment on Windows 10 to build the tutorial.

[phonokoye]

I initially wrote that I do not dig into any code that runs on more than one process or thread, but while I was waiting, I had a peek. It didn't have much to do with Locks, Semaphores, and the likes which I have not yet gotten my hands dirty with. It seems to be a bug with Windows. Let me explain

I modified the makefile to make more verbose outputs during the build and recognized that the following code was the source of the problem. From lines 65 to 88, we have the following:

def call_fn(func, args, kwargs, result_queue):
    try:
        result = func(*args, **kwargs)
        result_queue.put((True, result))
    except Exception as e:
        result_queue.put((False, str(e)))

def call_in_subprocess(func):
    def wrapper(*args, **kwargs):
        result_queue = multiprocessing.Queue()
        p = multiprocessing.Process(
            target=call_fn,
            args=(func, args, kwargs, result_queue)
        )
        p.start()
        p.join()
        success, result = result_queue.get()
        if success:
            return result
        else:
            raise RuntimeError(f"Error in subprocess: {result}")
    return wrapper

sphinx_gallery.gen_rst.generate_file_rst = call_in_subprocess(sphinx_gallery.gen_rst.generate_file_rst)

Since we are already hinted that the code runs fine on Ubuntu/Linux, one has to guess that the error is as a result of a different implementation in Windows. The error which I initially posted reveals that _winapi.DuplicateHandle gets called at some point. To avoid scrolling, I will repaste the error below

generating gallery for beginner... [  6%] saving_loading_models.py
Extension error (sphinx_gallery.gen_gallery):
Handler <function generate_gallery_rst at 0x000001C0A8C3AB00> for event 'builder-inited' threw an exception (exception: Can't pickle <function call_fn at 0x000001C088543010>: attribute lookup call_fn on __main__ failed)
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Python\Python310\lib\multiprocessing\spawn.py", line 107, in spawn_main
    new_handle = reduction.duplicate(pipe_handle,
  File "C:\Python\Python310\lib\multiprocessing\reduction.py", line 79, in duplicate
    return _winapi.DuplicateHandle(
OSError: [WinError 6] The handle is invalid
make: *** [html-noplot] Error 2

With a little more looking around, one finds that there is a bug in the python implementation and that it has been reported here. It has to do with an incorrect order of the arguments passed to DuplicateHandle.

Well, then I should not be getting errors. It's a known bug since September 2019 and would have been fixed by now, so what's the catch?
> Hmm... what version of python are you using again?
< 3.10.5. Why?
> Take a look at the image below and tell me what you see

< My current python version was built in 2022. Yeah but that is after the bug was known
> So when was that bug fixed?
< ... Oh... I need an update. I'll get back to you later.

[phonokoye]

Hello @svekars. I came across hud.pytorch.org some time ago. I suspect that I should be able to get a built documentation there as an artifact. Is this inaccurate?