tox fails to detect extras installation failure and proceed running tests

[ssbarnea]

In certain conditions tox 3.14.0 seems to ignore failure to install extras and proceeds running the test commands. Even worse, on subsequent executions it believes that all extras were correctly installed and goes directly to running tests.

This can easily be reproduce on a machine that does not nave python headers (so it cannot compile some reqs).

git clone https://github.com/ansible/molecule.git
cd molecule
tox -e py27-ansible27-functional --notest

That environment lists most of extras from https://github.com/ansible/molecule/blob/master/setup.cfg#L97-L138 and in my case installing lxc extra failed due to being unable to compile lxc-python2. At a result none of the requirements from test extras was installed, not even on extra runs.

I tried commenting the custom install_command but the same behavior remained.

tox should never assume that an environment is setup correctly if it failed to install requirements initially. I am not sure how it works internally but this could easily be obtained by using a placeholder that is added only on success execution, so when run again, it will know if it could bypass it or not.

[gaborbernat]

this could be a pip issue, as long as pip installation fails... tox should fail... however pip might consider extras optional and not fail.... 😢

[asottile]

this is just how pip works yeah -- so not a bug from tox's side

$ pip install astpretty[lolwut-this-does-not-exist]
Collecting astpretty[lolwut-this-does-not-exist]
  Using cached https://files.pythonhosted.org/packages/16/6c/3814ad265e04ae9cd3fc2bd3b72e80c122adffc3f2b8147d9d2897246d25/astpretty-1.6.0-py2.py3-none-any.whl
  WARNING: astpretty 1.6.0 does not provide the extra 'lolwut-this-does-not-exist'
Installing collected packages: astpretty
Successfully installed astpretty-1.6.0
WARNING: You are using pip version 19.1.1, however version 19.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
$ echo $?
0

[gaborbernat]

If anything I would file this as a bug under pip 👍 if pip would fail we would fail 👍

[ssbarnea]

As is not a good practice to close bugs without links to upstream ones, I added pypa/pip#7122 -- lets follow the issue there,... hopefully getting it fixed.