Skip to content

feat: retry coordinator-to-worker channel setup + fix 2 bugs - #712

Merged
gabotechs merged 7 commits into
mainfrom
gabrielmusat/add-retry-mechanism
Sep 11, 2026
Merged

gabotechs merged 7 commits into
mainfrom
gabrielmusat/add-retry-mechanism

Conversation

@gabotechs

@gabotechs gabotechs commented Sep 7, 2026 •

Copy link
Copy Markdown
Collaborator

Builds on top of

And adds 4 individual commits:

  1. Some improvements to the chaos test, to try to make results a bit more catastrophic
  2. Adds some retry mechanism that reacts to gRPC status codes and retries the coordinator->worker channels
  3. Fix one bug surfaced by the chaos test: when retrying, the same TaskKey might be attempted to be written multiple times, making the SingleWriteMultipleRead struct fail on the second write. With this change, the second write just overrides the previous one.
  4. Fix another bug surfaced by the chaos test: when retrying, there's a chance that two tasks fall on the same worker. If that happens there's a caching collision in the WorkerConnectionPool, where retrieving two different non-overlapping ranges of partitions will go through the same cached stream, trying to consume it multiple times.

Comment on lines +83 to +95
// https://grpc.io/docs/guides/status-codes/#deadline-exceeded
// The worker may be slow or wedged, so retry a different URL.
Code::DeadlineExceeded => RetryOutcome::OtherUrl.tag(err),
// https://grpc.io/docs/guides/status-codes/#resource-exhausted
// Admission pressure is local to this worker, so retry a different URL.
Code::ResourceExhausted => RetryOutcome::OtherUrl.tag(err),
// https://grpc.io/docs/guides/status-codes/#aborted
// Routing retries this task setup at the higher level on the same URL.
Code::Aborted => RetryOutcome::SameUrl.tag(err),
// https://grpc.io/docs/guides/status-codes/#unavailable
// This is a transient failure; retry the task setup on the same URL.
Code::Unavailable => RetryOutcome::SameUrl.tag(err),
Code::Ok => err,

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tbh, I don't know what I'm doing here... there's probably a better retry policy for this.

Invoking @hermeGarcia, in the hopes that you have better suggestions about how to transparently handle retries here.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This all seems reasonable to me.

I was suspicious about aborted. This description makes it sound like an optimistic locking error and the canonical thing to do in that situation is retry.

The operation was aborted, typically due to a concurrency issue such as a sequencer check failure or transaction abort. See the guidelines above for deciding between FAILED_PRECONDITION, ABORTED, and UNAVAILABLE.

Comment on lines 59 to 61
self.tx.send_modify(|v| {
if v.is_none() {
if v.is_none() || !*read_started {
*v = Some(item);

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fix for bug 1: if nobody has read yet the value, we are good to replace it.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks right. I would have called it a bug, not a feature though :)

Comment on lines -24 to +27
/// - Lazy inits connections to a remote worker on first call to [WorkerConnectionPool::execute].
/// - Lazily initializes connections per remote worker and requested partition range.
pub(crate) struct WorkerConnectionPool {
lazy_stream_groups: Vec<OnceLock<SharedBoxFuture<StreamGroup>>>,
lazy_stream_groups: Vec<DashMap<Range<usize>, SharedBoxFuture<StreamGroup>>>,

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fix for bug 2: if two requests come for the same target task, we still need to discriminate between requested partition ranges.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also think this was a feature, prior to adding retry capabilities :)

@gabotechs gabotechs changed the title feat: retry coordinator-to-worker channel setup feat: retry coordinator-to-worker channel setup + fix 2 bugs Sep 7, 2026
let mut url = url;
let mut attempted_urls = HashSet::from([url.clone()]);
retry_urls.retain(|candidate| attempted_urls.insert(candidate.clone()));
retry_urls.shuffle(&mut rand::rng());

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably makes sense to re-use the seed used in stage_contiguous_rand_choice

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I wanted to avoid is the same worker to get overloaded. Let's say for example that these are all the available URLs:

[http://1, http://2, http://3, http://4, http://5, http://6, http://7, http://8, http://9, http://10]

And the random stage-offseted choice is:

[http://5, http://6, http://7, http://8]

But it turns out that all workers 5, 6 and 7 are down. Then we'd be overloading the worker 8 with all tasks, so I thought we might as well just go truly random in case of retries.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that we should shuffle. I meant that we should continue using the seed here:

We have an instantiated rng:

let mut seed = [0; 32];
seed[..16].copy_from_slice(key.query_id.as_bytes());
seed[16..24].copy_from_slice(&(key.stage_id as u64).to_le_bytes());
let mut rng = StdRng::from_seed(seed);

and this can be re-used here

retry_urls.retain(|candidate| attempted_urls.insert(candidate.clone()));
retry_urls.shuffle(&mut rng); // don't use rand::rng(), which uses a new seed

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah ok, I thought you were referring to something different. TBH, just for the sake of avoiding threading one extra argument across functions, I'd just regenerated another random seed and call it a day, but if you feel strongly about it I can just do it.

MetricBuilder::new(metrics)
.global_counter("coordinator_to_worker_other_url_retries")
})
.add(1);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to work pretty well.

You don't get to see how many times a specific worker was retried, but you can see how many times we tried a new worker, the total number of retries, and the total number retries to the same worker.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well there's the coordinator_to_worker_same_url_retries metric as well, right next to this one, that should reflect this as well.

Comment on lines +83 to +95
// https://grpc.io/docs/guides/status-codes/#deadline-exceeded
// The worker may be slow or wedged, so retry a different URL.
Code::DeadlineExceeded => RetryOutcome::OtherUrl.tag(err),
// https://grpc.io/docs/guides/status-codes/#resource-exhausted
// Admission pressure is local to this worker, so retry a different URL.
Code::ResourceExhausted => RetryOutcome::OtherUrl.tag(err),
// https://grpc.io/docs/guides/status-codes/#aborted
// Routing retries this task setup at the higher level on the same URL.
Code::Aborted => RetryOutcome::SameUrl.tag(err),
// https://grpc.io/docs/guides/status-codes/#unavailable
// This is a transient failure; retry the task setup on the same URL.
Code::Unavailable => RetryOutcome::SameUrl.tag(err),
Code::Ok => err,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This all seems reasonable to me.

I was suspicious about aborted. This description makes it sound like an optimistic locking error and the canonical thing to do in that situation is retry.

The operation was aborted, typically due to a concurrency issue such as a sequencer check failure or transaction abort. See the guidelines above for deciding between FAILED_PRECONDITION, ABORTED, and UNAVAILABLE.

Comment on lines 59 to 61
self.tx.send_modify(|v| {
if v.is_none() {
if v.is_none() || !*read_started {
*v = Some(item);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks right. I would have called it a bug, not a feature though :)

Comment on lines -24 to +27
/// - Lazy inits connections to a remote worker on first call to [WorkerConnectionPool::execute].
/// - Lazily initializes connections per remote worker and requested partition range.
pub(crate) struct WorkerConnectionPool {
lazy_stream_groups: Vec<OnceLock<SharedBoxFuture<StreamGroup>>>,
lazy_stream_groups: Vec<DashMap<Range<usize>, SharedBoxFuture<StreamGroup>>>,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also think this was a feature, prior to adding retry capabilities :)

Comment thread tests/chaos.rs
const TOTAL_QUERIES: usize = 100;
const CONCURRENT_QUERIES_PER_CLIENT_RANDOM_RANGE: Range<usize> = 1..4;
const CONCURRENT_CLIENTS: usize = 10;
const QUERIES: &[&str] = &[

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Revert before merging?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This actually is something I changed in the original PR, but I forgot that there was a reason for using just 1 specific query rather than a random distribution:

At the end of the test, we store in memory all the results from all the queries that have ever happened during the test, and we assert the results from all of them are exactly the same. This is to ensure that even if retries work, we do not produce duplicate or omitted data.

For this reason, we use just 1 relatively convoluted query, and then expect the results for all queries that have happened during the test to be the same. Forgot about this in the original PR, and got surfaced now.

Comment thread tests/chaos.rs
let mut batch = vec![];
for _ in 0..rng.random_range(CONCURRENT_QUERIES_PER_CLIENT_RANDOM_RANGE) {
let query = tpch::get_query(QUERIES.choose(&mut rng).unwrap())?;
let query = tpch::get_query("q20")?;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto. Should revert?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above

@gabotechs
gabotechs force-pushed the gabrielmusat/add-retry-mechanism branch from 4669347 to 56aa985 Compare September 10, 2026 07:23
Base automatically changed from gabrielmusat/add-dialer-based-route-task-handler to main September 10, 2026 09:14
@gabotechs
gabotechs force-pushed the gabrielmusat/add-retry-mechanism branch from 56aa985 to ea30735 Compare September 10, 2026 09:16
Comment thread src/protocol/grpc/worker_service.rs Resolved
…n retrieving it manually in WorkerService implementations.
@gabotechs
gabotechs force-pushed the gabrielmusat/add-retry-mechanism branch from 7d24ba7 to a69a623 Compare September 10, 2026 09:44
@gabotechs
gabotechs merged commit 344d152 into main Sep 11, 2026
34 checks passed
@gabotechs
gabotechs deleted the gabrielmusat/add-retry-mechanism branch September 11, 2026 06:21
gabotechs added a commit that referenced this pull request Sep 22, 2026
Follow up on:
- #712

This PR introduces a new `SamplerKickOff` message that is sent only
after a worker has committed to drive work forward.

### Before

As soon as a plan was set in a worker, sampling kicks off. This means
that side effects will happen even before the coordinator has received
the OK from the worker to be part of the query.

### Now

Sampling kicks-off given a specific "go ahead" signal from the
coordinator, so that side effects only happens if the coordinator wants
to.
gabotechs pushed a commit that referenced this pull request Oct 19, 2026
## Problem

When a coordinator control channel closes, the worker sends its final
reports and invalidates the task's cache entry. Cache maintenance can
run later, so the invalidated entry may continue holding the task's plan
and session context. Those resources can remain alive after the task has
finished.

## Fix

After sending the final reports, the worker replaces the cached task
payload with a lightweight terminal error, then invalidates the entry.
This releases the cache's ownership of the plan and context immediately.
Streams and contexts already handed to consumers keep their own
references and remain usable.

The slot uses `None` for setup that has not arrived yet: readers wait
and writers may populate it. Clearing the payload would put it back in
that state. A terminal `Err` provides a finished state within the
existing `Result` type: it releases the cached resources immediately,
lets reads of that slot fail promptly, and prevents further writes.

The worker only finalizes a slot if it still contains the setup
belonging to the closing channel. An unread setup can be replaced during
a connection retry, as supported by
[#712](#712);
closing the older channel must preserve that replacement.

The change uses the slot's existing synchronization and runs during task
teardown. It adds no public API or dependencies.

## Validation

A worker API regression checks that closing an unexecuted task releases
its context. It observes destruction through a weak reference, without
forcing cache maintenance.

A small synchronous test verifies that finalization preserves a newer
value, releases the matching cached reference, and prevents further
writes.

## Documentation

The worker guide now explains when cached task resources are released
and how existing consumers retain their references.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants