pytest with multiprocessing & "--import-mode=importlib" EOFError

[schperplata]

Issue Type: Bug

Pytest raises:

    def test_funcInTestFile():
        p = multiprocessing.Process(target=sleepSomeTimeInTestFile)
>       p.start()

test_example.py:24: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
W:\Python38-64bit\lib\multiprocessing\process.py:121: in start
    self._popen = self._Popen(self)
W:\Python38-64bit\lib\multiprocessing\context.py:224: in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
W:\Python38-64bit\lib\multiprocessing\context.py:327: in _Popen
    return Popen(process_obj)
W:\Python38-64bit\lib\multiprocessing\popen_spawn_win32.py:93: in __init__
    reduction.dump(process_obj, to_child)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

obj = <Process name='Process-1' parent=19324 initial>
file = <_io.BufferedWriter name=5>, protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       _pickle.PicklingError: Can't pickle <function sleepSomeTimeInTestFile at 0x0000019EF622A3A0>: it's not the same object as test_example.sleepSomeTimeInTestFile

W:\Python38-64bit\lib\multiprocessing\reduction.py:60: PicklingError
  File "W:\Python38-64bit\lib\multiprocessing\spawn.py", line 116, in spawn_main
- generated xml file: C:\Users\DOMENJ~1.ISY\AppData\Local\Temp\tmp-1116815mH4iv3Rg16.xml -
    exitcode = _main(fd, parent_sentinel)

... error when:

• "--import-mode=importlib" arg is used
• multiprocessing Process() target=<function> function is defined in the same (test) file

Without "--import-mode=importlib" test works (but not when test files have the same names.

pytestImporlibTest.zip

Extension version: 2020.8.105045
VS Code version: Code 1.48.2 (a0479759d6e9ea56afa657e454193f72aef85bd0, 2020-08-25T10:13:11.295Z)
OS version: Windows_NT x64 10.0.19041

System Info

Item	Value
CPUs	Intel(R) Core(TM) i5-4570 CPU @ 3.20GHz (4 x 3193)
GPU Status	2d_canvas: enabled flash_3d: enabled flash_stage3d: enabled flash_stage3d_baseline: enabled gpu_compositing: enabled multiple_raster_threads: enabled_on oop_rasterization: disabled_off protected_video_decode: unavailable_off rasterization: enabled skia_renderer: disabled_off_ok video_decode: enabled viz_display_compositor: enabled_on viz_hit_test_surface_layer: disabled_off_ok webgl: enabled webgl2: enabled
Load (avg)	undefined
Memory (System)	15.89GB (6.98GB free)
Process Argv	W:\AsystLib\isys\python\VSCodeDevWorkspace_isys.code-workspace
Screen Reader	no
VM	0%

[nicoddemus]

I assume multiprocessing either tries to:

1. Dump only module name + function name, and in the other process, it uses that information to obtain the function object and call it.

2. If the function object doesn't have a module attribute (which is the case with --import-mode=importlib), then it tries to pickle the actual function object, but then fails.

(I don't have time to look up the code, so I'm just assuming the above is true)

If that's the case, it is not something that is possible to fix: the importlib module's purpose is to not pollute the sys.modules dict with the test modules being imported.

[karthiknadig]

@nicoddemus Thank you for the response.

@schperplata The python extension is not involved in this at all. If you do pickle.dumps on sleepSomeTimeInTestFile it does not match what it expects in the test process. Actual __main__.sleepSomeTimeInTestFile, pickle expects test_example.sleepSomeTimeInTestFile.

Closing this since it occurs without the python extension being involved.