Repository navigation
fix(s3): use lenient CRT health checks for S3 control operations - #3695
Merged
superhx merged 3 commits intoOct 23, 2026
Merged
Conversation
Committed-By-Agent: codex Co-authored-by: codex <noreply@openai.com>
ashsamantaray-stripe
requested review from
1sonofqiu,
Gezi-lzq and
superhx
as code owners
October 23, 2026 06:35
superhx
reviewed
Oct 23, 2026
| if (usesCrt) { | ||
| CRT.releaseShutdownRef(); | ||
| try { | ||
| if (listS3Client != null && listS3Client != readS3Client && listS3Client != writeS3Client) { |
Collaborator
There was a problem hiding this comment.
We should put the listS3Client close to the try block
Committed-By-Agent: codex Co-authored-by: codex <noreply@openai.com>
Collaborator
|
@ashsamantaray-stripe Should we also let delete / createMultiPartUpload / completeMultiPartUpload / uploadPartCopy use this client? |
ashsamantaray-stripe
changed the title
Contributor
Author
|
@superhx should we use that for delete and upload part? Or would they benefit from the existing client? |
| void doClose() { | ||
| if (!closed.compareAndSet(false, true)) { | ||
| synchronized void doClose() { | ||
| if (closed) { |
Contributor
Author
There was a problem hiding this comment.
synchronizing closing the client with client creation, happy to revert, to make a more targeted fix
Collaborator
delete, createMultiPartUpload, completeMultiPartUpload, and uploadPartCopy are also low-throughput APIs, but they may can take a relatively long time to process internally. uploadPart should use the writeClient. |
Committed-By-Agent: codex Co-authored-by: codex <noreply@openai.com>
ashsamantaray-stripe
changed the title
superhx
approved these changes
Oct 23, 2026
This was referenced Oct 23, 2026
superhx
added a commit
that referenced
this pull request
Oct 23, 2026
… backport) (#3697) ## Why Backport #3695 (commit dd35f8c) to 1.8. Low-throughput S3 control requests can exceed CRT connection health limits while waiting for S3 processing. LIST failures during WAL recovery can prevent broker startup. ## What Route LIST, DeleteObjects, CreateMultipartUpload, CompleteMultipartUpload, and UploadPartCopy through one lazily created CRT control client with a minimum throughput threshold of at most 1 B/s and a window of at least 30 seconds, preserving already more lenient settings. UploadPart remains on the write client and Netty retains its existing routing. ## How Reuse the control client across requests and LIST pages, synchronize lazy creation with shutdown, and close distinct clients with the storage instance. Preserve endpoint, credentials, retry, and timeout configuration. This is a clean cherry-pick of the original change, including its regression tests. Co-authored-by: ashsamantaray-stripe <ashsamantaray@stripe.com> Co-authored-by: codex <noreply@openai.com>
superhx
added a commit
that referenced
this pull request
Oct 23, 2026
… backport) (#3696) ## Why Backport #3695 (commit dd35f8c) to 1.7. Low-throughput S3 control requests can exceed CRT connection health limits while waiting for S3 processing. LIST failures during WAL recovery can prevent broker startup. ## What Route LIST, DeleteObjects, CreateMultipartUpload, CompleteMultipartUpload, and UploadPartCopy through one lazily created CRT control client with a minimum throughput threshold of at most 1 B/s and a window of at least 30 seconds, preserving already more lenient settings. UploadPart remains on the write client and Netty retains its existing routing. ## How Reuse the control client across requests and LIST pages, synchronize lazy creation with shutdown, and close distinct clients with the storage instance. Preserve endpoint, credentials, retry, and timeout configuration. Adapt the backport to 1.7's existing string-prefix LIST API and native CRT implementation. Retain the pagination retry regression test with separate data and control clients, and adapt the control-operation routing test to the native CRT builder. CRT HTTP and bounded LIST APIs are not introduced. --------- Co-authored-by: ashsamantaray-stripe <ashsamantaray@stripe.com> Co-authored-by: codex <noreply@openai.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
S3 LIST, DeleteObjects, CreateMultipartUpload, CompleteMultipartUpload, and UploadPartCopy move little data on the client connection, even when S3 spends time processing the operation. The CRT client applies the same connection health threshold to them as to object transfers. The default threshold is 32 MiB/s for 2 seconds. A delayed response with little traffic can cause CRT to close its HTTP connection with a minimum throughput failure. LIST is used during WAL recovery, so this failure can prevent broker startup.
The AWS Java SDK configures this health threshold per CRT client, not per request.
What
Use one separate, lazily created S3 client for LIST, DeleteObjects, CreateMultipartUpload, CompleteMultipartUpload, and UploadPartCopy when the selected SDK implementation uses CRT. Its connection health threshold is at most 1 B/s, and its low throughput window is at least 30 seconds. Preserve more lenient settings when already configured. UploadPart stays on the write client. Netty retains its existing read/write routing.
How
Create the control client on the first control request, reuse it for later requests and LIST pagination, and close it with the storage instance. Synchronize lazy creation with close so the client cannot be created after shutdown. Keep the existing endpoint, credentials, retry, and API timeout settings, including the UploadPartCopy request override.
Reviewer notes
Review
AwsObjectStorage.Configuration.forControl()and client construction first, then the low-throughput request routing inAwsObjectStorage.AwsObjectStorageTestcovers both CRT builder configurations, lazy creation and sharing of the control client, LIST pagination, Netty routing, UploadPart staying on the write client, and closing distinct clients.Verified with:
./gradlew :s3stream:test --tests 'com.automq.stream.s3.operator.AwsObjectStorageTest' --no-daemon./gradlew --build-cache rat checkstyleMain checkstyleTest spotlessJavaCheck --no-daemon