Repository navigation
feat: retry coordinator-to-worker channel setup + fix 2 bugs - #712
Conversation
| // https://grpc.io/docs/guides/status-codes/#deadline-exceeded | ||
| // The worker may be slow or wedged, so retry a different URL. | ||
| Code::DeadlineExceeded => RetryOutcome::OtherUrl.tag(err), | ||
| // https://grpc.io/docs/guides/status-codes/#resource-exhausted | ||
| // Admission pressure is local to this worker, so retry a different URL. | ||
| Code::ResourceExhausted => RetryOutcome::OtherUrl.tag(err), | ||
| // https://grpc.io/docs/guides/status-codes/#aborted | ||
| // Routing retries this task setup at the higher level on the same URL. | ||
| Code::Aborted => RetryOutcome::SameUrl.tag(err), | ||
| // https://grpc.io/docs/guides/status-codes/#unavailable | ||
| // This is a transient failure; retry the task setup on the same URL. | ||
| Code::Unavailable => RetryOutcome::SameUrl.tag(err), | ||
| Code::Ok => err, |
There was a problem hiding this comment.
tbh, I don't know what I'm doing here... there's probably a better retry policy for this.
Invoking @hermeGarcia, in the hopes that you have better suggestions about how to transparently handle retries here.
There was a problem hiding this comment.
This all seems reasonable to me.
I was suspicious about aborted. This description makes it sound like an optimistic locking error and the canonical thing to do in that situation is retry.
The operation was aborted, typically due to a concurrency issue such as a sequencer check failure or transaction abort. See the guidelines above for deciding between FAILED_PRECONDITION, ABORTED, and UNAVAILABLE.
| self.tx.send_modify(|v| { | ||
| if v.is_none() { | ||
| if v.is_none() || !*read_started { | ||
| *v = Some(item); |
There was a problem hiding this comment.
Fix for bug 1: if nobody has read yet the value, we are good to replace it.
There was a problem hiding this comment.
Looks right. I would have called it a bug, not a feature though :)
| /// - Lazy inits connections to a remote worker on first call to [WorkerConnectionPool::execute]. | ||
| /// - Lazily initializes connections per remote worker and requested partition range. | ||
| pub(crate) struct WorkerConnectionPool { | ||
| lazy_stream_groups: Vec<OnceLock<SharedBoxFuture<StreamGroup>>>, | ||
| lazy_stream_groups: Vec<DashMap<Range<usize>, SharedBoxFuture<StreamGroup>>>, |
There was a problem hiding this comment.
Fix for bug 2: if two requests come for the same target task, we still need to discriminate between requested partition ranges.
There was a problem hiding this comment.
I also think this was a feature, prior to adding retry capabilities :)
| let mut url = url; | ||
| let mut attempted_urls = HashSet::from([url.clone()]); | ||
| retry_urls.retain(|candidate| attempted_urls.insert(candidate.clone())); | ||
| retry_urls.shuffle(&mut rand::rng()); |
There was a problem hiding this comment.
Probably makes sense to re-use the seed used in stage_contiguous_rand_choice
There was a problem hiding this comment.
What I wanted to avoid is the same worker to get overloaded. Let's say for example that these are all the available URLs:
[http://1, http://2, http://3, http://4, http://5, http://6, http://7, http://8, http://9, http://10]
And the random stage-offseted choice is:
[http://5, http://6, http://7, http://8]
But it turns out that all workers 5, 6 and 7 are down. Then we'd be overloading the worker 8 with all tasks, so I thought we might as well just go truly random in case of retries.
There was a problem hiding this comment.
I agree that we should shuffle. I meant that we should continue using the seed here:
We have an instantiated rng:
let mut seed = [0; 32];
seed[..16].copy_from_slice(key.query_id.as_bytes());
seed[16..24].copy_from_slice(&(key.stage_id as u64).to_le_bytes());
let mut rng = StdRng::from_seed(seed);and this can be re-used here
retry_urls.retain(|candidate| attempted_urls.insert(candidate.clone()));
retry_urls.shuffle(&mut rng); // don't use rand::rng(), which uses a new seedThere was a problem hiding this comment.
Ah ok, I thought you were referring to something different. TBH, just for the sake of avoiding threading one extra argument across functions, I'd just regenerated another random seed and call it a day, but if you feel strongly about it I can just do it.
| MetricBuilder::new(metrics) | ||
| .global_counter("coordinator_to_worker_other_url_retries") | ||
| }) | ||
| .add(1); |
There was a problem hiding this comment.
This seems to work pretty well.
You don't get to see how many times a specific worker was retried, but you can see how many times we tried a new worker, the total number of retries, and the total number retries to the same worker.
There was a problem hiding this comment.
Well there's the coordinator_to_worker_same_url_retries metric as well, right next to this one, that should reflect this as well.
| // https://grpc.io/docs/guides/status-codes/#deadline-exceeded | ||
| // The worker may be slow or wedged, so retry a different URL. | ||
| Code::DeadlineExceeded => RetryOutcome::OtherUrl.tag(err), | ||
| // https://grpc.io/docs/guides/status-codes/#resource-exhausted | ||
| // Admission pressure is local to this worker, so retry a different URL. | ||
| Code::ResourceExhausted => RetryOutcome::OtherUrl.tag(err), | ||
| // https://grpc.io/docs/guides/status-codes/#aborted | ||
| // Routing retries this task setup at the higher level on the same URL. | ||
| Code::Aborted => RetryOutcome::SameUrl.tag(err), | ||
| // https://grpc.io/docs/guides/status-codes/#unavailable | ||
| // This is a transient failure; retry the task setup on the same URL. | ||
| Code::Unavailable => RetryOutcome::SameUrl.tag(err), | ||
| Code::Ok => err, |
There was a problem hiding this comment.
This all seems reasonable to me.
I was suspicious about aborted. This description makes it sound like an optimistic locking error and the canonical thing to do in that situation is retry.
The operation was aborted, typically due to a concurrency issue such as a sequencer check failure or transaction abort. See the guidelines above for deciding between FAILED_PRECONDITION, ABORTED, and UNAVAILABLE.
| self.tx.send_modify(|v| { | ||
| if v.is_none() { | ||
| if v.is_none() || !*read_started { | ||
| *v = Some(item); |
There was a problem hiding this comment.
Looks right. I would have called it a bug, not a feature though :)
| /// - Lazy inits connections to a remote worker on first call to [WorkerConnectionPool::execute]. | ||
| /// - Lazily initializes connections per remote worker and requested partition range. | ||
| pub(crate) struct WorkerConnectionPool { | ||
| lazy_stream_groups: Vec<OnceLock<SharedBoxFuture<StreamGroup>>>, | ||
| lazy_stream_groups: Vec<DashMap<Range<usize>, SharedBoxFuture<StreamGroup>>>, |
There was a problem hiding this comment.
I also think this was a feature, prior to adding retry capabilities :)
| const TOTAL_QUERIES: usize = 100; | ||
| const CONCURRENT_QUERIES_PER_CLIENT_RANDOM_RANGE: Range<usize> = 1..4; | ||
| const CONCURRENT_CLIENTS: usize = 10; | ||
| const QUERIES: &[&str] = &[ |
There was a problem hiding this comment.
Revert before merging?
There was a problem hiding this comment.
This actually is something I changed in the original PR, but I forgot that there was a reason for using just 1 specific query rather than a random distribution:
At the end of the test, we store in memory all the results from all the queries that have ever happened during the test, and we assert the results from all of them are exactly the same. This is to ensure that even if retries work, we do not produce duplicate or omitted data.
For this reason, we use just 1 relatively convoluted query, and then expect the results for all queries that have happened during the test to be the same. Forgot about this in the original PR, and got surfaced now.
| let mut batch = vec![]; | ||
| for _ in 0..rng.random_range(CONCURRENT_QUERIES_PER_CLIENT_RANDOM_RANGE) { | ||
| let query = tpch::get_query(QUERIES.choose(&mut rng).unwrap())?; | ||
| let query = tpch::get_query("q20")?; |
There was a problem hiding this comment.
Ditto. Should revert?
4669347 to
56aa985
Compare
56aa985 to
ea30735
Compare
…n retrieving it manually in WorkerService implementations.
7d24ba7 to
a69a623
Compare
Follow up on: - #712 This PR introduces a new `SamplerKickOff` message that is sent only after a worker has committed to drive work forward. ### Before As soon as a plan was set in a worker, sampling kicks off. This means that side effects will happen even before the coordinator has received the OK from the worker to be part of the query. ### Now Sampling kicks-off given a specific "go ahead" signal from the coordinator, so that side effects only happens if the coordinator wants to.
## Problem When a coordinator control channel closes, the worker sends its final reports and invalidates the task's cache entry. Cache maintenance can run later, so the invalidated entry may continue holding the task's plan and session context. Those resources can remain alive after the task has finished. ## Fix After sending the final reports, the worker replaces the cached task payload with a lightweight terminal error, then invalidates the entry. This releases the cache's ownership of the plan and context immediately. Streams and contexts already handed to consumers keep their own references and remain usable. The slot uses `None` for setup that has not arrived yet: readers wait and writers may populate it. Clearing the payload would put it back in that state. A terminal `Err` provides a finished state within the existing `Result` type: it releases the cached resources immediately, lets reads of that slot fail promptly, and prevents further writes. The worker only finalizes a slot if it still contains the setup belonging to the closing channel. An unread setup can be replaced during a connection retry, as supported by [#712](#712); closing the older channel must preserve that replacement. The change uses the slot's existing synchronization and runs during task teardown. It adds no public API or dependencies. ## Validation A worker API regression checks that closing an unexecuted task releases its context. It observes destruction through a weak reference, without forcing cache maintenance. A small synchronous test verifies that finalization preserves a newer value, releases the matching cached reference, and prevents further writes. ## Documentation The worker guide now explains when cached task resources are released and how existing consumers retain their references.
Builds on top of
And adds 4 individual commits:
SingleWriteMultipleReadstruct fail on the second write. With this change, the second write just overrides the previous one.WorkerConnectionPool, where retrieving two different non-overlapping ranges of partitions will go through the same cached stream, trying to consume it multiple times.