Repository navigation
Conversation
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
squiidz
marked this pull request as ready for review
October 21, 2026 15:21
Contributor
Connect docs checkThe docs build passed with this PR's generated docs. Preview: https://connect-pr-5001--redpanda-connect.netlify.app/connect/home/ This PR changes 1 published page (2 generated files: 2 changed, 0 added, 0 removed). Review the rendered pages, not only the Go strings.
Per-file diffs and the rendered-HTML findings are in the job summary. Checked at c87547a. |
…parents, forget finished shard state
…HORIZON so it cannot stall its children
…oint position survives
The Checkpointing section now says the principal needs dynamodb:UpdateItem on the checkpoint table (MarkShardCompleted is an UpdateItem), and describes the behaviour this PR adds: closed shards read to the end are marked completed and skipped after a restart, a child shard is read only after its parent is read to the end (except on the first discovery under start_from: latest), and a checkpoint pointing at trimmed stream data resumes from TRIM_HORIZON with a warning. max_tracked_shards now says a shard is normally forgotten once read to the end and acknowledged.
…hildren A child shard was held back for as long as its parent stayed listed and unfinished, so a parent that could not be prepared for a reason other than trimmed data kept its whole lineage unread until it aged out of the listing (up to 24h). Discovery now remembers failed prepares across refresh cycles: a parent whose GetShardIterator returns ResourceNotFoundException is treated as done for ordering right away, and a parent that fails to prepare on 5 consecutive refresh cycles releases its children with a single warning. Neither writes a completion marker for the parent, which is still retried and read if it becomes readable.
GetShardIterator returns no iterator for a closed shard with nothing left to read, and an iterator refresh can do the same. The reader treated a nil iterator as "stop" and returned silently, leaving the shard registered but unread: it was never marked completed and its children stayed held back behind it. A registered, unfinished reader holding a nil iterator now takes the closed-shard path (completeShard / completeTableShard), which persists the completion marker and releases the children.
…utdown A failed write of a shard's completion marker was logged but not counted, so dynamodb_cdc_checkpoint_failures missed it. completeShard and completeTableShard now increment the metric when the write fails. A write that failed only because the reader's context was canceled (shutdown) is expected: it is logged at debug instead of error and not counted.
…rdinator Adds coverage for two paths the PR changed without a direct test: syncShardReaders restarting a reader that gave up and was re-registered under the same shard ID (the old goroutine is canceled and a fresh one is started for the new reader), and a multi-table variant of TestShardCoordinatorReadsClosedShardOnce driving the table stream coordinator. The in-memory checkpoint store shared by both coordinator tests moves into lockedStoreCheckpointer.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds an in-memory completedShards set to the single-table input and to each tableStream. A reader adds its shard when it reaches the end of a closed shard, and refreshShards/refreshTableShards skip shards in the set, so the 5-minute cleanup no longer causes closed shards still in retention to be rediscovered and re-read. Entries are pruned once DescribeStream stops listing the shard. The reader also now delivers the records returned on a closed shard's final page (nil NextShardIterator) before exiting; previously they were dropped. The set is in memory only, so a restart re-reads each retained closed shard once; the closed-shard tail checkpoint lag is tracked separately in CON-598.
Jira: https://redpandadata.atlassian.net/browse/CON-599