Repository navigation
ResourceExhaustedException #3343
Description
Hello,
I am reading a pubsub subscription queue length and publish rate continuously using a Timer. But I am getting this exception time to time.
Exception in thread "Timer-0" com.google.api.gax.rpc.ResourceExhaustedException: io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: The query rate is too high. at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:57) at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:72) at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:60) at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:95) at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:61) at com.google.common.util.concurrent.Futures$4.run(Futures.java:1123) at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:435) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:900) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:811) at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:675) at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:492) at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:467) at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:41) at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:684) at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:41) at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:392) at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:475) at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:557) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:478) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:590) at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: The query rate is too high. at io.grpc.Status.asRuntimeException(Status.java:526) ... 19 more
my code is as follows.
final MetricServiceClient client = MetricServiceClient.create();
ProjectName name = ProjectName.of(projectId);
long startMillis = System.currentTimeMillis() - ((60 * 1) * 1000);
TimeInterval interval = TimeInterval.newBuilder()
.setStartTime(Timestamps.fromMillis(startMillis))
.setEndTime(Timestamps.fromMillis(System.currentTimeMillis()))
.build();
Aggregation aggregation = Aggregation.newBuilder()
.setAlignmentPeriod(Duration.newBuilder().setSeconds(300).build())
.setPerSeriesAligner(Aggregation.Aligner.ALIGN_MEAN)
.build();
ListTimeSeriesRequest.Builder requestBuilder1 = ListTimeSeriesRequest.newBuilder()
.setName(name.toString())
.setFilter(filter1)
.setInterval(interval);
ListTimeSeriesRequest request1 = requestBuilder1.build();
MetricServiceClient.ListTimeSeriesPagedResponse response1 = client.listTimeSeries(request1);
Activity
andreamlin commentedon Jun 5, 2018
The
MetricServiceClientis part of the Monitoring API, which is subject to request both daily quotas and QPS quotas (see https://developers.google.com/analytics/devguides/reporting/core/v3/limits-quotas for details). It is likely that requests are made too often through your project or your IP address.SachithW commentedon Jun 6, 2018
According to the API dashboard my requests/sec is always under 8 requests/sec. And in my project
Requests per day quota is set to unlimited. Anyway I will further reduces the timer frequency and let you know.
By the way with the above exception there is another exception throws.
andreamlin commentedon Jun 6, 2018
Is the INTERNAL error intermittent also? Does it only happen when the RESOURCE_EXHAUSTED error is also thrown?
SachithW commentedon Jun 7, 2018
sduskis commentedon Dec 3, 2018
At this point, I think that there isn't enough information to answer the question. Pub/Sub quotas are described here: https://cloud.google.com/pubsub/quotas
The INTERNAL issue is potentially a grpc issue. We would need a bit more information before proceeding. Also, we'd likely request that you use a newer version of the client, since these types of issues get resolved with newer versions of the library.
If there's any new information, please reopen the issue.
deps: update dependency com.google.cloud:google-cloud-datacatalog-bom…
chore(main): release 2.40.3 (#3325)
test(deps): update dependency io.github.classgraph:classgraph to v4.8…