Add ChunkedEncodingError to the list of resumable stream errors

[nitishxp]

Hi I faced this error in production.

Lib version:
google-cloud-bigquery==2.15.0

[POST]
Traceback (most recent call last):
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/urllib3/response.py", line 696, in _update_chunk_length
    self.chunk_left = int(line, 16)
  ValueError: invalid literal for int() with base 16: b''

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/urllib3/response.py", line 436, in _error_catcher
    yield
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/urllib3/response.py", line 763, in read_chunked
    self._update_chunk_length()
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/urllib3/response.py", line 700, in _update_chunk_length
    raise httplib.IncompleteRead(line)    
  http.client.IncompleteRead: IncompleteRead(0 bytes read)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/requests/models.py", line 751, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/urllib3/response.py", line 571, in stream
    for line in self.read_chunked(amt, decode_content=decode_content):
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/urllib3/response.py", line 792, in read_chunked
    self._original_response.close()
  File "/opt/python3.9/lib/python3.9/contextlib.py", line 135, in __exit__
    self.gen.throw(type, value, traceback)
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/urllib3/response.py", line 454, in _error_catcher
    raise ProtocolError("Connection broken: %r" % e, e)
  urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/functions_framework/__init__.py", line 152, in view_func
    function(data, context)
  File "/workspace/common/functions.py", line 61, in wrapper
    raise e
  File "/workspace/common/functions.py", line 54, in wrapper
    response = wrapped_func(*args, **kwargs)
  File "/workspace/main.py", line 533, in controller
    status = create_files(contex_bdp)
  File "/workspace/main.py", line 346, in create_files
    for row in result:
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/api_core/page_iterator.py", line 214, in _items_iter
    for page in self._page_iter(increment=False):
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/api_core/page_iterator.py", line 253, in _page_iter
    page = self._next_page()
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/api_core/page_iterator.py", line 382, in _next_page
    response = self._get_next_page_response()
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/cloud/bigquery/table.py", line 1541, in _get_next_page_response
    return self.api_request(
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/cloud/bigquery/client.py", line 686, in _call_api
    return call()
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/api_core/retry.py", line 285, in retry_wrapped_func
    return retry_target(
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/api_core/retry.py", line 188, in retry_target
    return target()
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/cloud/_http.py", line 473, in api_request
    response = self._make_request(
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/cloud/_http.py", line 337, in _make_request
    return self._do_request(
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/cloud/_http.py", line 375, in _do_request
    return self.http.request(
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/auth/transport/requests.py", line 482, in request
    response = super(AuthorizedSession, self).request(
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/requests/sessions.py", line 685, in send
    r.content
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/requests/models.py", line 829, in content
    self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/requests/models.py", line 754, in generate
    raise ChunkedEncodingError(e) requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))

[plamut]

@nitishxp I reformatted the posted traceback for better clarity.

It appears that the connection got broken part-way through downloading the results with ChunkedEncodingError. That error is not retryable by default, and I'm not even sure it can be reliably retried - I don't think the server keeps track of partially downloaded results and allows resuming the download if it fails.

Did this happen after executing a query, i.e. is related to the other issue you opened recently?

The nature of some errors is that the client cannot recover from them, it seems that the way to handle this in your app would be to re-issue the query?

Edit: If you have a reference to the query_job instance and the query executed successfully, but it was just downloading the results that failed, it might work to call query_job.result() again to fetch them.

[nitishxp]

Hi @plamut

While using gbq.list_rows(result.destination) i got the error

[plamut]

@nitishxp I see, there's no job object you could leverage then. Seems like there's no other option but to call list_rows() again if it fails the first time, because the ChunkedEncodingError is not recoverable.

[plamut]

Update: I asked around and got feedback that retrying ChunkedEncodingError should actually work in this case when tabledata.list is used. Result page tokens should be good for quite some time, so even if fetching a page fails, retrying that request can actually succeed.

The ChunkedEncodingError should be added to the list of retryable stream errors in bigquery-storage. I'll thus move the issue there.

[nitishxp]

Will that resolve the above issue?

[plamut]

@nitishxp It should, as the request to fetch table data that resulted in ChunkedEncodingError would be retried, which should work fine.

[nitishxp]

But in traceback i am not finding any storage components being used is it being used internally?

[plamut]

From what I was told, that makes sense if downloading results, and retrying such download request should work. But I'll re-check, as I had the same question.

[nitishxp]

@plamut any update did u find anything related to google-storage in traceback?

[plamut]

@nitishxp Didn't have a chance to have a deeper look at it yet, but generally speaking the retries are controlled by the Retry object used. That object can (but not necessarily does) use retryable exception types defined in BQ Storage (and other places).

I'll need to see what kind of retry is used for list_rows() and other code paths (e.g. to_dataframe()). The ChunkedEncodingError should be treated as a transport error and if it is indeed missing in the Retry object used, we'll need to fix it.

[plamut]

I checked the code and bigquery.Client.list_rows() uses bigquery.DEFAULT_RETRY. The latter does not retry ChunkedEncodingError, as it is not a subclass of requests.exceptions.ConnectionError , thus it needs to be added (in google-cloud-bigquery). And possibly to _STREAM_RESUMPTION_EXCEPTIONS here in BigQuery Storage.

[plamut]

@tswast Hm, BigQuery storage does not depend on requests, it downloads data over gRPC, thus adding requests.exceptions.ChunkedEncodingError to _STREAM_RESUMPTION_EXCEPTIONS would not make sense.

The issue the user observed seems specific to the bigquery.Client.list_rows() path, which will be fixed directly in the bigquery repo.

[tswast]

We can close this one out, since it doesn't seem to affect the gRPC code path.