requests.exceptions.ConnectionError retry

[nitishxp]

Hi,

Could we add ConnectionError in list of retry so the client libraries which depend over api-core don't need to add.

"/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/bigquery/client.py", line 561, in create_table api_response = self._call_api( File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/bigquery/client.py", line 574, in _call_api return call() File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/api_core/retry.py", line 281, in retry_wrapped_func return retry_target( File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/api_core/retry.py", line 184, in retry_target return target() File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/_http.py", line 472, in api_request response = self._make_request( File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/_http.py", line 336, in _make_request return self._do_request( File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/_http.py", line 374, in _do_request return self.http.request( File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/auth/transport/requests.py", line 464, in request response = super(AuthorizedSession, self).request( File "/layers/google.python.pip/pip/lib/python3.8/site-packages/requests/sessions.py", line 530, in request resp = self.send(prep, **send_kwargs) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/requests/sessions.py", line 643, in send r = adapter.send(request, **kwargs) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/requests/adapters.py", line 498, in send raise ConnectionError(err, request=request) requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

I have reached out to individually to storage,bigquery and auth sdk page to report about this issue and they have acknowledged it googleapis/google-auth-library-python#739, googleapis/google-auth-library-python#737, googleapis/python-storage#414

[busunkim96]

Hi,

I don't think we should retry at every level that shows up in the stack trace (that might result in retrying more times than is desirable). I think addressing googleapis/google-auth-library-python#737 should be sufficient. If it turns out that isn't enough, we can revisit tweaking the rtries in google-api-core and/or google-cloud-core.

[parthea]

I'm planning to add

    requests.exceptions.ConnectionError,
    auth_exceptions.TransportError,

to the list of errors that can be retried here as we have had success with this configuration in the BigQuery client and it follows the spirit of client side retries in AIP-4221. Googlers see b/182321177 for internal discussion.

[nitishxp]

@parthea Could you also add https://github.com/googleapis/python-bigquery/blob/6a48e80bc7d347f381b181f4cf81fef105d0ad0d/google/cloud/bigquery/retry.py#L26 in the list of retry

[busunkim96]

@nitishxp ConnectionError was added, it's just referred to as requests.exceptions.ConnectionError here.

[nitishxp]

@busunkim96 ConnectionError is specfic to python https://docs.python.org/3/library/exceptions.html#ConnectionError which is an OS error big query sdk https://github.com/googleapis/python-bigquery/blob/6a48e80bc7d347f381b181f4cf81fef105d0ad0d/google/cloud/bigquery/retry.py#L26 has added os connection error in retry along with requests.ConnectionError

[busunkim96]

Ah I see. Do you have a stack trace that shows an un-retried ConnectionError? I'm wondering if that's specific to BIgQuery.

[nitishxp]

@busunkim96 Naah , But i have faced this issue in Cloud function when there r too many instances of CF running and same instance is used for another request