Repository navigation
Random Auth Errors with BigQuery #2139
Description
Insert calls to BigQuery randomly result in auth errors.
I have a job that is responsible for sending data in bulk to BigQuery by using .insert([ 'lots', 'of', 'items' ]) repeatedly in synchronous fashion. This process goes on for hours at a time, calling insert after the prior operation finishes.
After a few hours, my calls to BigQuery abruptly fail. I'm not sure if this issue is time-related or the call itself randomly fails. It does seem to occur after a few hours. This happens repeatedly.
Error:
Unhandled promise rejection (rejection id: 1): ApiError: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.
Environment details
- OS: OSX El Capitan
- Node.js version: 7.7.3
- npm version: 4.1.2
@google-cloud/bigqueryversion: 0.8.0
Steps to reproduce
The above description tells it all, but I'm working on a script. I'll update this if I can find a way to reproduce in a more predictable manner.
Activity
lukesneeringer commentedon Mar 27, 2017
@stephenplusplus This smells like a rate limit to me, other than the error message. Might this be a case where we are clobbering an error?
stephenplusplus commentedon Mar 27, 2017
I don't think we're clobbering it, just judging by the error message not being one that we've written. Before every request, we do a token grab, refreshing if necessary (that work is done by google-auth-library).
With this large amount of requests, maybe we need a larger safety window around the expiration timestamp for when the auth client makes the grab. @jmdobry does that sound possible? (please re-tag if someone else is closer to the auth library code)
stephenplusplus commentedon Mar 29, 2017
@ofrobots wdyt?
stephenplusplus commentedon Apr 7, 2017
Friendly re-ping.
stephenplusplus commentedon Apr 13, 2017
This issue was moved to googleapis/google-auth-library-nodejs#133
stephenplusplus commentedon Apr 13, 2017
We haven't been able to attract the right attention over here, so I've moved the issue to the source of the error (I think). googleapis/google-auth-library-nodejs#133
MarkHerhold commentedon Apr 13, 2017
Can please we keep this open so I can easily track it?
stephenplusplus commentedon Apr 13, 2017
For tracking, just subscribe to the new issue with the button in the right column: googleapis/google-auth-library-nodejs#133 -- I assumed you'd be automatically subscribed because of the
@mention of your name over there, but it wouldn't hurt to confirm.