[nox] Test fails when nox fails to download dependencies

[tmatsuo]

Maybe we should retry few times.

Example log:

 Using noxfile-template from parent folder (/tmpfs/src/github/python-docs-samples).

Running session py-3.7
Creating virtual environment (virtualenv) using python3.7 in .nox/py-3-7
pip install -r requirements.txt
Command pip install -r requirements.txt failed with exit code 2:
Ignoring tensorflow: markers 'python_version < "3.0"' don't match your environment
Collecting tensorflow==2.0.1
  Downloading tensorflow-2.0.1-cp37-cp37m-manylinux2010_x86_64.whl (86.3 MB)
ERROR: Exception:
Traceback (most recent call last):
  File "/tmpfs/src/github/python-docs-samples/ml_engine/online_prediction/.nox/py-3-7/lib/python3.7/site-packages/pip/_vendor/urllib3/response.py", line 425, in _error_catcher
    yield
  File "/tmpfs/src/github/python-docs-samples/ml_engine/online_prediction/.nox/py-3-7/lib/python3.7/site-packages/pip/_vendor/urllib3/response.py", line 507, in read
    data = self._fp.read(amt) if not fp_closed else b""
  File "/tmpfs/src/github/python-docs-samples/ml_engine/online_prediction/.nox/py-3-7/lib/python3.7/site-packages/pip/_vendor/cachecontrol/filewrapper.py", line 62, in read
    data = self.__fp.read(amt)
  File "/usr/local/lib/python3.7/http/client.py", line 457, in read
    n = self.readinto(b)
  File "/usr/local/lib/python3.7/http/client.py", line 501, in readinto
    n = self.fp.readinto(b)
  File "/usr/local/lib/python3.7/socket.py", line 589, in readinto
    return self._sock.recv_into(b)
  File "/usr/local/lib/python3.7/ssl.py", line 1071, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/local/lib/python3.7/ssl.py", line 929, in read
    return self._sslobj.read(len, buffer)
ConnectionResetError: [Errno 104] Connection reset by peer

[tmatsuo]

The first option came to mind is to use backoff in our noxfile-template.py.
I think we can retry upon nox.command.CommandFailed.

If we do this, we have to install backoff locally, and also install backoff alongside with nox in the run-tests.sh.

Maybe we want to retry only for transient errors, but we need to determine this by looking at the error log (through nox.command.CommandFailed I guess).

@kurtisvg @gguuss
WDYT?

[kurtisvg]

Are these happening frequently enough that's it's worth adding logic for this? Or was this a one off error?

[tmatsuo]

I have only seen one time. It's one off, but potentially it can happen anytime.
It's worth adding in my opinion, but there is also a downside of requiring backoff locally.

Other option is to add retry to nox itself.

[tmatsuo]

Let's see how often we'll see this. I'll add occurrences to this issue whenever I see the same error.

[tmatsuo]

According to https://pip.pypa.io/en/stable/reference/pip/

It seems that pip already retries 5 times. In that case, I don't think more retry would improve the situation. Closing.