API ignores user defined timeout and applies default timeout #591
Comments
|
Are the custom values you're passing down FWIW, None isn't technically a supported value for the Retry object type hints, which is probably why we had the gap in the tests. But we'll definitely fix it up if there's a regression here |
|
We ran into the same issue and downgrading to 2.15 fixed it. 900 second timeouts in DBT that ignore manual override with 2.16. |
|
Can you confirm that the manual override you're doing is setting the timeout to None? I think I found the solution for that, but if you're seeing other overrides not working, let me know and I'll keep looking |
|
Originally we didn't set anything in the DBT config for timeout, but once we started hitting the timeout issues, we manually set the timeouts in DBT: This didn't work, and downgrading google-api-core was just a bit of trial and error/guesswork. |
|
I see. I opened a PR here to fix the bug I found. If you can let me know if that seems to fix your issue, that would be great Or if you can link to the place in the DBT code that configures the retry, that could be helpful too |
|
@daniel-sanche As far as I know, dbt-bigquery didn't pass # only use async logic if user specifies a timeout
if job_execution_timeout:
loop = asyncio.new_event_loop()
future_iterator = asyncio.wait_for(
loop.run_in_executor(None, functools.partial(query_job.result, max_results=limit)),
timeout=job_execution_timeout,
)
try:
iterator = loop.run_until_complete(future_iterator)
except asyncio.TimeoutError:
query_job.cancel()
raise DbtRuntimeError(
f"Query exceeded configured timeout of {job_execution_timeout}s"
)
finally:
loop.close()
else:
iterator = query_job.result(max_results=limit)According to my research, python-api-core/google/api_core/future/polling.py Lines 126 to 142 in 5922f2b Set
|
|
@yu-iskw & @daniel-sanche for context our async wrapper as a band-aid because, at the time, the bq python client wasn't actually passing the timeout method to the api call. Sounds like we can now remove that async logic and rely on python-api-core to handle it |
|
@colin-rogers-dbt I got it. Can we tentatively set the [MODIFIED] |
|
Ok, if someone could The |
|
@daniel-sanche I am going to try it out locally. I will get back to you. |
|
@daniel-sanche I tested the patch branch locally. That looks good as we expected. Here is the situation to test it. Set upI set up an environment to test it with Pyhon codeI implemented the subsequent python code to test the patch. The query of BigQuery script enables us to wait for 1000 seconds which is larger than 900. from pprint import pprint
from datetime import datetime
from google.cloud import bigquery
def main():
client = bigquery.Client()
try:
query = """
DECLARE DELAY_TIME DATETIME;
DECLARE WAIT BOOL;
BEGIN
SET WAIT = TRUE;
SET DELAY_TIME = DATE_ADD(CURRENT_DATETIME, INTERVAL 1000 SECOND);
WHILE WAIT DO
IF (DELAY_TIME < CURRENT_DATETIME) THEN
SET WAIT = FALSE;
END IF;
END WHILE;
END
"""
job_config = bigquery.QueryJobConfig()
pprint(f"[{datetime.utcnow()}] Creating query job")
query_job = client.query(query=query, job_config=job_config)
pprint(f"[{datetime.utcnow()}] Waiting for query job to complete")
query_result = query_job.result()
pprint(f"[{datetime.utcnow()}] Query job completed")
pprint(query_result)
except Exception as e:
pprint(f"[{datetime.utcnow()}] Exception occurred")
pprint(e)
if __name__ == "__main__":
main() |
|
Thank you for verifying, we'll get this fix merged and into the next release |
parthea
added
type: bug
When we updated to
dbtversion 1.7.5 anddbt-bigqueryversion 1.7.3, we noticed that our BigQuery queries were always timing out and cancelling after 900 seconds with the below error message, regardless of our own timeout settings:Those two
dbtlibs depend on thegoogle-api-coreversion 2.16.0, which seems to always use the default timeout of 900 seconds instead of honouring the user defined timeout setting. This appears to be happening somewhere here: https://github.com/googleapis/python-api-core/blob/v2.16.0/google/api_core/future/polling.py#L126Once we reverted to using
google-api-coreversion 2.15.0, this problem disappeared and dbt went back to honouring our custom defined timeout settings.The text was updated successfully, but these errors were encountered: