Hitting quota limit when starting a large number of pub/sub subscribers simultaneously

I have been spinning up a lot of pods within large node-pools in ContainerEngine. When the pods start subscribing to pub/sub I often get
RESOURCE_EXHAUSTED: Insufficient tokens for quota 'administrator' and limit 'CLIENT_PROJECT-100s'
I think this is due to the subscriber starting multiple channels per core. First, isn't core-count improperly biased in kubernetes - I have a machine that has 16 cores but might be sharing them with other pods. My use case results in 10-20 minute of processing per pub/sub message when using 16 cores. Isn't 160 channels overkill? I am assuming that starting 10,000 to 20,000 channels simultaneously the reason I'm hitting the quota. Is the large number of channels for use cases that need to consume large number of messages per second?

I'd like to add a configuration parameter to limit the number of channels. Are the other considerations for having large numbers of channels? Perhaps a minimum and/or maximum number of channels?

I can create a pull request but I'm relatively new to the pub/sub code base and wanted to make sure I was understanding things appropriately.

@davidtorres

I am assuming that starting 10,000 to 20,000 channels simultaneously the reason I'm hitting the quota. Is the large number of channels for use cases that need to consume large number of messages per second?

@davidtorres @haakonringberg do these sound like reasonable assumptions to you? And if so, what do you think about the suggestion for a channel-limiting configuration parameter?

No, this is not due to # channels; you're hitting the administrative operations quota limit: https://cloud.google.com/pubsub/quotas My guess would be that you're having each of your nodes do some subset of "Get, List, Create, Delete, ModifyPushConfig, SetIamPolicy, GetIamPolicy, TestIamPermissions". I'd advise you to have each worker node focus on the data plane operations (publish and subscribe) and then have an administrative node that issues the administrative operations (e.g., don't have each of your nodes do a CreateSubscription on startup).

It was certainly my intent that I was only subscribing - all workers use the same subscription so there isn't any admin calls needed. Perhaps I'm missing the proper use of the API - I am doing the following:

  def newSubscriber(receiver: MessageReceiver) = {
    val flowControl = FlowControlSettings.newBuilder
      .setMaxOutstandingElementCount(1L)
      .setLimitExceededBehavior(LimitExceededBehavior.Ignore)
      .build
    Subscriber.defaultBuilder(SubscriptionName.create(project, subscription), receiver)
      .setExecutorProvider(pubsubThreadPool)
      .setFlowControlSettings(flowControl)
      .setChannelProvider(channelProvider)
      .build()
  }

which is the only pub/sub actively (other than the ack calls). The "SubscriptionName.create" implementation looks to be client-side only (creating the object not the server-side subscription itself). Perhaps pertinent, I had been using 0.13.0-alpha, switching to 0.18.0-alpha to see if it makes any difference.

Am I missing something?

@davidtorres

Yes please upgrade and let us know if you still see issues. Thank you.

@davidtorres @pongad fyi

pubsub: GetSubscription only once (#2127)

chore(deps): Update gapic-generator-java to 2.21.0 (#2111)

test: remove some Emulator restrictions from tests (googleapis#2111)

build(deps): update dependency org.apache.maven.plugins:maven-surefir…

chore(deps): update dependency com.google.cloud:google-cloud-firestor…

chore(main): release 2.33.1-SNAPSHOT (googleapis#2111)

Navigation Menu

Repository navigation

Hitting quota limit when starting a large number of pub/sub subscribers simultaneously #2111

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Hitting quota limit when starting a large number of pub/sub subscribers simultaneously #2111

Description

Activity

tcoffee-google commented on Jun 5, 2017

haakonringberg commented on Jun 5, 2017

rdmoore commented on Jun 7, 2017

haakonringberg commented on Jun 7, 2017

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions