Description
There are a few questions similar to this, but I don't think any are the same. I apologise if I didn't recognise it.
I am using Datastore inside a docker on a Google Compute Engine instance. The docker container is based off of python:3.4-slim. The python version is Python 3.4.6. I have used pip to install google-cloud-datastore==1.0.0. My GCE instance explicitly has Cloud API access scope Cloud Datastore enabled. I have a worker that regularly (10s interval) checks my datastore for new tasks to process. This makes a query like:
query = ds_client.query(kind=RepeatingJob.KIND)
query.add_filter("deleted", "=", False)
query.add_filter("next_process", "<", datetime.now())
jobs = list(query.fetch(limit=1))
where ds_client is initialised once in the application:
from google.cloud import datastore
ds_client = datastore.Client()
If I understand correctly, because this is GCE instance and I have enabled the datastore scope, I should not have to do anything more for authorization.
This system works fine for tens of minutes at a time (40 minutes 21 seconds) in my last test. Then an error is raised:
Traceback (most recent call last):
File "/usr/local/lib/python3.4/site-packages/google/gax/retry.py", line 120, in inner
return to_call(*args)
File "/usr/local/lib/python3.4/site-packages/google/gax/retry.py", line 68, in inner
return a_func(*updated_args, **kwargs)
File "/usr/local/lib/python3.4/site-packages/grpc/_channel.py", line 507, in __call__
return _end_unary_response_blocking(state, call, False, deadline)
File "/usr/local/lib/python3.4/site-packages/grpc/_channel.py", line 455, in _end_unary_response_blocking
raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNAUTHENTICATED, Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.)>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.4/site-packages/google/cloud/datastore/_gax.py", line 74, in _catch_remap_gax_error
yield
File "/usr/local/lib/python3.4/site-packages/google/cloud/datastore/_gax.py", line 134, in run_query
return super(GAPICDatastoreAPI, self).run_query(*args, **kwargs)
File "/usr/local/lib/python3.4/site-packages/google/cloud/gapic/datastore/v1/datastore_client.py", line 258, in run_query
return self._run_query(request, options)
File "/usr/local/lib/python3.4/site-packages/google/gax/api_callable.py", line 419, in inner
return api_caller(api_call, this_settings, request)
File "/usr/local/lib/python3.4/site-packages/google/gax/api_callable.py", line 407, in base_caller
return api_call(*args)
File "/usr/local/lib/python3.4/site-packages/google/gax/api_callable.py", line 368, in inner
return a_func(*args, **kwargs)
File "/usr/local/lib/python3.4/site-packages/google/gax/retry.py", line 126, in inner
' classified as transient', exception)
google.gax.errors.RetryError: GaxError(Exception occurred in retry method that was not classified as transient, caused by <_Rendezvous of RPC that terminated with (StatusCode.UNAUTHENTICATED, Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.)>)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/app/application/jobs/models/repeating_job.py", line 199, in needs_processing
jobs = list(query.fetch(limit=1))
File "/usr/local/lib/python3.4/site-packages/google/cloud/iterator.py", line 218, in _items_iter
for page in self._page_iter(increment=False):
File "/usr/local/lib/python3.4/site-packages/google/cloud/iterator.py", line 247, in _page_iter
page = self._next_page()
File "/usr/local/lib/python3.4/site-packages/google/cloud/datastore/query.py", line 497, in _next_page
query=query_pb,
File "/usr/local/lib/python3.4/site-packages/google/cloud/datastore/_gax.py", line 134, in run_query
return super(GAPICDatastoreAPI, self).run_query(*args, **kwargs)
File "/usr/local/lib/python3.4/contextlib.py", line 77, in __exit__
self.gen.throw(type, value, traceback)
File "/usr/local/lib/python3.4/site-packages/google/cloud/datastore/_gax.py", line 82, in _catch_remap_gax_error
six.reraise(error_class, new_exc, sys.exc_info()[2])
File "/usr/local/lib/python3.4/site-packages/six.py", line 685, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.4/site-packages/google/cloud/datastore/_gax.py", line 74, in _catch_remap_gax_error
yield
File "/usr/local/lib/python3.4/site-packages/google/cloud/datastore/_gax.py", line 134, in run_query
return super(GAPICDatastoreAPI, self).run_query(*args, **kwargs)
File "/usr/local/lib/python3.4/site-packages/google/cloud/gapic/datastore/v1/datastore_client.py", line 258, in run_query
return self._run_query(request, options)
File "/usr/local/lib/python3.4/site-packages/google/gax/api_callable.py", line 419, in inner
return api_caller(api_call, this_settings, request)
File "/usr/local/lib/python3.4/site-packages/google/gax/api_callable.py", line 407, in base_caller
return api_call(*args)
File "/usr/local/lib/python3.4/site-packages/google/gax/api_callable.py", line 368, in inner
return a_func(*args, **kwargs)
File "/usr/local/lib/python3.4/site-packages/google/gax/retry.py", line 126, in inner
' classified as transient', exception)
google.cloud.exceptions.Unauthorized: 401 Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.
At this point, if I immediately retry the request, it fails with the same error. At some point a short (maybe 1 minute) period of time later, the same command starts working again.
I also use Sentry error reporting to track issues. Perhaps interestingly, I noticed that in the tracked variables at this point (/usr/local/lib/python3.4/site-packages/google/gax/retry.py in inner at line 126) that env = RetryError('Retry total timeout exceeded before anyresponse was received',). I don't think I can share that report, but if it is relevant, let me know and I will see what I can do.