Repository navigation
BigQuery: Add QueryResults.reload() function which calls getQueryResults #3506
Description
https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/getQueryResults
One of the things I learned when talking to the BigQuery API folks is that the recommended way to run a query is to create a query job and then call getQueryResults() until the jobComplete property is true.
The reason for this is that a call to get the job resource will return immediately, whereas getQueryResults will return after a timeout or the query is complete, whichever comes first. With getQueryResults users get their data faster.
This means the code for waiting for a query job to complete would change from
while True:
query_job.reload()
if query_job.state == 'DONE':
return
time.sleep(1)to something like
query_results = query_job.results()
while not query_results.complete:
query_results.reload()
# No need for sleep, since reload should wait for timeout
Activity
tswast commentedon Jun 15, 2017
I figured out that I can force a reload with
but obviously, that's not ideal for the purpose of waiting for a job to finish.
tseaver commentedon Jun 19, 2017
If you are going to iterate over the rows anyway, then that second example becomes:
That certainly seems like "idiomatic" usage. Does that accomplish what you need?
tswast commentedon Jun 19, 2017
No. The results might not all be there. That's why I need a reload function to check if the query is complete or not. (A bit hard to test, since it requires a query that takes longer than the timeout value, which defaults to 10 seconds).
[-]BigQuery: Add QueryResults.reload() function which calls getQueryResults[/-][+]BigQuery: make QueryResults implement the futures interface[/+]tswast commentedon Aug 1, 2017
A reload function probably isn't the best solution. After thinking about this, I think
QueryResultsshould also implement the futures interface.theacodes commentedon Aug 1, 2017
I don't think
QueryResultsshould implement the future interface, rather, I think thatQueryJob(which already has the futures interface) should use thegetQueryResultsAPI method instead ofgetJob.[-]BigQuery: make QueryResults implement the futures interface[/-][+]BigQuery: Add QueryResults.reload() function which calls getQueryResults[/+]tswast commentedon Aug 11, 2017
After I've played around with some proposed designs for GA, I agree with you, Jon. QueryJob's implementation of futures interface will be sufficient.
I've changed this request back to asking for
.reload()method since there does need to be a more straightforward way to actually callgetQueryResults.Closing this issue since I'm tracking internally in GA redesign doc.