Test failure: test_typed_pkg

[dmoisset]

I'm running mypy tests from master ( af7e834 ) with python 3.6.2; When running tests I get a failure at test_typed_package

====================================================== FAILURES =======================================================
______________________________________________ TestPEP561.test_typed_pkg ______________________________________________
[gw1] linux -- Python 3.6.2 /home/machinalis/.virtualenvs/mypy-dev/bin/python3

self = <mypy.test.testpep561.TestPEP561 testMethod=test_typed_pkg>

    def test_typed_pkg(self) -> None:
        """Tests type checking based on installed packages.
    
            This test CANNOT be split up, concurrency means that simultaneously
            installing/uninstalling will break tests.
            """
        test_file = 'simple.py'
        if not os.path.isdir('test-packages-data'):
            os.mkdir('test-packages-data')
        old_cwd = os.getcwd()
        os.chdir('test-packages-data')
        with open(test_file, 'w') as f:
            f.write(SIMPLE_PROGRAM)
        try:
            with self.install_package('typedpkg-stubs'):
                check_mypy_run(
                    [test_file],
                    "simple.py:4: error: Revealed type is 'builtins.list[builtins.str]'\n"
                )
    
            # The Python 2 tests are intentionally placed after a Python 3 test to check
            # the package_dir_cache is behaving correctly.
            python2 = try_find_python2_interpreter()
            if python2:
                with self.install_package('typedpkg-stubs', python2):
                    check_mypy_run(
                        ['--python-executable={}'.format(python2), test_file],
                        "simple.py:4: error: Revealed type is 'builtins.list[builtins.str]'\n"
                    )
                with self.install_package('typedpkg', python2):
                    check_mypy_run(
                        ['--python-executable={}'.format(python2), 'simple.py'],
                        "simple.py:4: error: Revealed type is 'builtins.tuple[builtins.str]'\n"
                    )
    
                with self.install_package('typedpkg', python2):
                    with self.install_package('typedpkg-stubs', python2):
                        check_mypy_run(
                            ['--python-executable={}'.format(python2), test_file],
                            "simple.py:4: error: Revealed type is 'builtins.list[builtins.str]'\n"
                        )
    
            with self.install_package('typedpkg'):
                check_mypy_run(
                    [test_file],
>                   "simple.py:4: error: Revealed type is 'builtins.tuple[builtins.str]'\n"
                )

mypy/test/testpep561.py:103: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

cmd_line = ['simple.py'], expected_out = "simple.py:4: error: Revealed type is 'builtins.tuple[builtins.str]'\n"
expected_err = '', expected_returncode = 1

    def check_mypy_run(cmd_line: List[str],
                       expected_out: str,
                       expected_err: str = '',
                       expected_returncode: int = 1) -> None:
        """Helper to run mypy and check the output."""
        out, err, returncode = mypy.api.run(cmd_line)
>       assert out == expected_out, err
E       AssertionError: 
E       assert "simple.py:4:...ltins.str]'\n" == "simple.py:4: ...ltins.str]'\n"
E         Skipping 37 identical leading characters in diff, use -v to show
E         - 'builtins.list[builtins.str]'
E         ?            ^^^
E         + 'builtins.tuple[builtins.str]'
E         ?           +++ ^

mypy/test/testpep561.py:27: AssertionError
============================================== 1 failed in 26.85 seconds ==============================================

[emmatyping]

I cannot seem to repro this on either Arch or Ubuntu 17.10. This failure happens if it is finding the stub package before the typed package.

Could you give the output of python -m mypy.sitepkgs and tell me if typedpkg or typedpkg-stubs are listed in a pip freeze?

[gvanrossum]

Maybe checking the source could help? It looks like a simple confusion between list and tuple.

[emmatyping]

One package has the return type List, the other returning a Tuple so that the tests fail if the wrong package is found. This error means that the wrong package is being found first.

[emmatyping]

Additionally, in this specific case, based on the check being run, and the output, I can surmise that there is typedpkg-stubs installed, when it shouldn't be, and it is being picked up before the installed typedpkg package. It is possible that the uninstall failed, which I realized I don't check for (PR incoming).

[gvanrossum]

So maybe it's a race between different threads/subprocesses running tests?

[emmatyping]

All of the package install/uninstall happen in a single test case so that shouldn't happen. I designed the test case intentionally to avoid race conditions.

[ilevkivskyi]

Just a random idea: Maybe this is outdated info, but my instinct was always that pip uninstall doesn't actually uninstall everything (in the sense that it doesn't revert the state of everything to the same as before install). Maybe a cleaner solution would be to create a virtualenv that will be destroyed completely after the test?

[emmatyping]

Pip doesn't always remove everything for egg installations AIUI. However, for packages installed by pip, it should remove mostly everything. One thing it absolutely removes is the package directory, so I don't think it is an issue.

[gvanrossum]

It also doesn't uninstall dependencies.

…

On Tue, Apr 10, 2018, 19:51 Ethan Smith ***@***.***> wrote: Pip doesn't always remove everything for egg installations AIUI. However, for packages installed by pip, it should remove mostly everything. One thing it absolutely removes is the package directory, so I don't think it is an issue. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#4883 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACwrMhFOtSKOLc1pYrYpJYPTqxyKiqdlks5tnW-ugaJpZM4TOlOC> .

[emmatyping]

I just ran the test on my friends Mac in a virtualenv and this test case did not fail...

@gvanrossum and @dmoisset since you two have been able to hit this bug, could you give me some information?

• the output of python -m mypy.sitepkgs, to make sure it is finding the correct package directory
• tell me if pip freeze lists typedpkg or typedpkg-stubs, so I can make sure there isn't a transient package laying around.

It also doesn't uninstall dependencies.

Yes this is true, fortunately for my sanity, these test packages have no dependencies :)

[dmoisset]

@ethanhs this is my output:

(mypy-dev) machinalis@ogion ~/oss/mypy master [1]$ python -m mypy.sitepkgs
['/home/machinalis/.virtualenvs/mypy-dev/lib/python3.6/site-packages', '/home/machinalis/.virtualenvs/mypy-dev/local/lib/python3.6/dist-packages', '/home/machinalis/.virtualenvs/mypy-dev/lib/python3/dist-packages', '/home/machinalis/.virtualenvs/mypy-dev/lib/python3.6/dist-packages', '/home/machinalis/.local/lib/python3.6/site-packages']

(mypy-dev) machinalis@ogion ~/oss/mypy master $ pip freeze
apipkg==1.4
attrs==17.4.0
coverage==4.5.1
execnet==1.5.0
flake8==3.5.0
flake8-bugbear==18.2.0
flake8-pyi==18.3.1
lxml==4.1.1
mccabe==0.6.1
more-itertools==4.1.0
pluggy==0.6.0
psutil==5.4.0
py==1.5.3
pycodestyle==2.3.1
pyflakes==1.6.0
pytest==3.5.0
pytest-cov==2.5.1
pytest-forked==0.2
pytest-xdist==1.22.2
six==1.11.0
typed-ast==1.1.0

I don't have mypy installed in my virtualenv, I'm just running the tests from the checked out repo, I'm not sure if that might be relevant.

[gvanrossum]

I can still repro it, both on master and on release-0.590, using my Python 3.6.5-based venv.

alabaster==0.7.10
apipkg==1.4
attrs==17.4.0
Babel==2.5.1
certifi==2017.11.5
chardet==3.0.4
cherry-picker==0.2.6
click==6.7
coverage==4.5.1
docutils==0.14
execnet==1.5.0
flake8==3.5.0
flake8-bugbear==18.2.0
flake8-pyi==18.3.1
gidgethub==2.4.1
idna==2.6
imagesize==0.7.1
Jinja2==2.10
junit-xml==1.8
lxml==4.1.1
MarkupSafe==1.0
mccabe==0.6.1
MonkeyType==17.12.4.dev1
more-itertools==4.1.0
mypy===0.590-dev-1fd5a30086d535d57ca87deba532ab5f36f83e45-dirty
mypy-extensions==0.3.0
mypy-protobuf==1.0
pkginfo==1.4.1
pluggy==0.6.0
psutil==5.4.0
py==1.5.3
pyannotate==1.0.5
pycodestyle==2.3.1
pyflakes==1.6.0
Pygments==2.2.0
pytest==3.5.0
pytest-cov==2.5.1
pytest-forked==0.2
pytest-xdist==1.22.2
pytz==2017.3
requests==2.18.4
requests-toolbelt==0.8.0
retype==17.12.0
six==1.11.0
snowballstemmer==1.2.1
Sphinx==1.6.5
sphinxcontrib-websupport==1.0.1
tqdm==4.19.5
twine==1.9.1
typed-ast==1.1.0
typing==3.6.4
typing-extensions==3.6.2.1
typycal==0.0.1
uritemplate==3.0.0
urllib3==1.22
virtualenv==15.1.0

(Note that the mypy refers to the mypy package in the current directory.)

[JelleZijlstra]

I can also repro on master on my Mac.

$ python -m mypy.sitepkgs
['/Users/jzijlstra-mpbt/py/mypy/.venv/lib/python3.6/site-packages', '/Users/jzijlstra-mpbt/Library/Python/3.6/lib/python/site-packages']
$ pip freeze
apipkg==1.4
attrs==17.4.0
coverage==4.5.1
execnet==1.5.0
flake8==3.5.0
flake8-bugbear==18.2.0
flake8-pyi==18.3.1
lxml==4.1.1
mccabe==0.6.1
more-itertools==4.1.0
pluggy==0.6.0
psutil==5.4.0
py==1.5.3
pycodestyle==2.3.1
pyflakes==1.6.0
pytest==3.5.0
pytest-cov==2.5.1
pytest-forked==0.2
pytest-xdist==1.22.2
six==1.11.0
typed-ast==1.1.0
You are using pip version 9.0.1, however version 9.0.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

[gvanrossum]

So I had typedpkg and typedpkg-stubs installed in my non-venv Python 3.6.5 site-packages. No idea how that happened, but there it was. Somehow this didn't show up in the above list but it did when I deactivated the venv:

$ python3.6 -m pip freeze
certifi==2018.1.18
mypy===0.590-dev-1fd5a30086d535d57ca87deba532ab5f36f83e45-dirty
typedpkg==0.1
typedpkg-stubs==0.1
virtualenv==15.0.2

So I uninstalled these, re-activated the venv, ran the test again, and it still failed, and after deactivating the venv, typedpkg and typedpkg-stubs were there again, in the "root" python3.6!

[gvanrossum]

Since I don't have pytest installed in my root Python 3.6, I can't test with that, but at this point it does look like something funky's going on with my venv or with my root Python or with the relationship between the two. Note that my root Python 3.6 lives in /usr/local/bin/python3.6 which is a symlink to ../../../Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6 (where the initial ../../../ is equivalent to /) and the typedpkg I found was in /Users/guido/Library/Python/3.6/lib/python/site-packages/typedpkg/

[JelleZijlstra]

That's a great find! I looked a bit more and found that in this code:

        # if we aren't in a virtualenv, install in the
        # user package directory so we don't need sudo
        # In a virtualenv, real_prefix is patched onto
        # sys
        if not hasattr(sys, 'real_prefix') or python_executable != sys.executable:
            install_cmd.append('--user')

The real_prefix check doesn't actually work: we call pip with --user even if I run the test inside a virtualenv. If I just comment out those lines and manually remove the typedpkg global install, the test passes.

So it looks like we need a better way to figure out whether we're in a virtualenv, or perhaps the test should just always create a new virtualenv for itself.

[dmoisset]

Thanks for the fix, I can confirm this has been fixed in Ubuntu too