You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Bigquery Jobs should match the Futures interface #3556
Presently, bigquery uses a custom Job class (and subclasses) to deal with long-running operations such as queries and data import.
The Job class causes the proliferation of code like this around all of our samples and user code:
defwait_for_job(job):
whileTrue:
job.reload() # Refreshes the state via a GET request.ifjob.state=='DONE':
ifjob.error_result:
raiseRuntimeError(job.errors)
returntime.sleep(1)
Which in terms means our samples require a level on indirection to accomplish simple tasks:
However, our gax-based clients share a common "Long Running Operation" strategy that surfaces these types of on-going operations as a class that implements the concurrent.futures.Future interface. This is implemented as gax._OperationFuture. bigquery unfortunately can not use this as it's an http-only API and gax can only currently be used for gRPC APIs.
We should make the _BaseJob class and its subclasses closely conform to the Future interface so that the usage is similar to other API client and so that we can simplify usage to:
client=bigquery.Client()
query_job=client.run_async_query(str(uuid.uuid4()), query)
query_job.use_legacy_sql=Falsequery_job.begin()
# Wait for the job to finishquery_result=query.result()
rows=query_result.fetch_data(max_results=10)
forrowinrows:
print(row)
Activity
tswast commentedon Jun 30, 2017
When you call
.result()does it actually block? The Video Intelligence API sample, which uses that gax OperationFuture, I believe, shows some polling.theacodes commentedon Jun 30, 2017
theacodes commentedon Jul 17, 2017
Related #3617