Repository navigation
feat: Handle MRAP URI - #19609
feat: Handle MRAP URI#19609vivek807 wants to merge 8 commits into
Conversation
Signed-off-by: Vivek Dhiman <approach2vivek@gmail.com>
FrankChen021
left a comment
There was a problem hiding this comment.
| Severity | Findings |
|---|---|
| P0 | 0 |
| P1 | 0 |
| P2 | 1 |
| P3 | 0 |
| Total | 1 |
Reviewed 6 of 6 changed files.
This is an automated review by Codex GPT-5.5
FrankChen021
left a comment
There was a problem hiding this comment.
| Severity | Findings |
|---|---|
| P0 | 0 |
| P1 | 0 |
| P2 | 1 |
| P3 | 0 |
| Total | 1 |
Reviewed 6 of 6 changed files.
This is an automated review by Codex GPT-5.5
Signed-off-by: Vivek Dhiman <approach2vivek@gmail.com>
9a40fd3 to
e06de77
Compare
FrankChen021
left a comment
There was a problem hiding this comment.
Follow-up handled: slash-form S3 Access Point and MRAP ARNs are now explicitly rejected by CloudObjectLocation, with coverage for both direct construction and URI parsing. Callers accepting slash form normalize it through S3Utils.normalizeBucketName.
Reviewed 6 of 6 changed files.
This is an automated review by Codex GPT-5.6-Sol
|
This pull request has been marked as stale due to 60 days of inactivity. |
|
Gentle follow-up on this PR. The changes are ready from my side and still relevant. Please let me know if there are any concerns or additional changes needed. Happy to address them. |
…r strategy; added SigV4A signing, ACL and move-source checks for MRAPs. Signed-off-by: Vivek Dhiman <approach2vivek@gmail.com>
FrankChen021
left a comment
There was a problem hiding this comment.
🟡 Changes recommended
Follow-up assessment: The original provider-specific-regex concern is resolved at this head: CloudObjectLocation is provider-neutral, while S3-specific validation is centralized in S3BucketValidator and S3Utils. Regional Access Point ARNs can still fail before transmission when their ARN region differs from the configured client region; fix that before merging.
Reviewed 22 of 22 changed files; findings in 1 file.
Validation: No Druid build or test commands were run. A narrow AWS SDK 2.57.2 endpoint-resolution check reproduced the regional ARN failure when useArnRegion is disabled.
| Severity | Findings |
|---|---|
| P0 | 0 |
| P1 | 0 |
| P2 | 1 |
| P3 | 0 |
| Total | 1 |
This is an automated review by Codex GPT-5.6-Luna(max)
After addressing the findings or replying to the comments, you can request another review from me to trigger a new automated review.
FrankChen021
left a comment
There was a problem hiding this comment.
🟢 Approval recommended
No actionable issues were found. The regional ARN endpoint-resolution finding is resolved: both sync and async clients now enable ARN-region routing, with coverage in the incremental tests.
Reviewed 23 of 23 changed files. Coverage included the 4-file incremental diff, full diff reconciliation, URI parsing, normalization, ACL handling, S3 clients, data-segment lifecycle, task logs, input sources, dependencies, documentation, and tests.
Validation: Narrow offline Maven dependency-tree checks only; no Druid build or test commands were run.
This is an automated review by Codex GPT-5.6-Luna(max)
amaechler
left a comment
There was a problem hiding this comment.
Thanks for the work on this! Some comments from mostly automated Claude reviews.
The inline comments cover the regional Access Point ACL check, path-style access, two remaining URI.getHost() call sites, and the default bucket validator for the other providers. Two smaller points outside the diff:
- MSQ durable storage (
S3OutputConfig) and export keep the raw bucket, so the slash-form ARN that AWS documents fails there:S3StorageConnector.listDirbuilds aCloudObjectLocation, which rejects/. Normalizing in theS3OutputConfigconstructor (this.bucket = S3Utils.normalizeBucketName(bucket);) covers durable storage. Alternatively, the docs could say that only the colon form works for those. - The PR description still describes an ARN pattern check inside
CloudObjectLocation, but that logic now lives inS3BucketValidator. Worth refreshing so the description and release note match the code.
Signed-off-by: Vivek Dhiman <approach2vivek@gmail.com>
FrankChen021
left a comment
There was a problem hiding this comment.
🟡 Changes recommended
Fix MRAP cleanup before merging. Reconciliation of the incremental and full diffs confirms that the earlier regional-ARN routing, provider-neutral validation, slash-form normalization, and ACL-scope findings are addressed. However, normalized MRAP ARNs still enter existing DeleteObjects cleanup paths. AWS’s MRAP supported-operations list does not include DeleteObjects. Consequently, unzipped deep-storage pushes can fail after upload, while task-log, MSQ, and multi-segment cleanup can fail or leave objects behind.
Reviewed 28 of 28 changed files; finding in 1 file.
| Severity | Findings |
|---|---|
| P0 | 0 |
| P1 | 1 |
| P2 | 0 |
| P3 | 0 |
| Total | 1 |
This is an automated review by Codex GPT-5.6-Luna(max)
After addressing the findings or replying to the comments, you can request another review from me to trigger a new automated review.
amaechler
left a comment
There was a problem hiding this comment.
Thanks for the quick turnaround! Two small follow-ups inline, plus a concrete fix proposal on the DeleteObjects thread.
FrankChen021
left a comment
There was a problem hiding this comment.
🟡 Changes recommended
The supplied reply is not reflected in head b5391acc: S3Utils.deleteBucketKeys still sends non-empty batches through DeleteObjects, which MRAPs do not support. Fix this before merging because unzipped pushes and other bulk cleanup paths can fail or leave objects behind.
Reviewed 28 of 28 changed files; finding in 1 file. Coverage included the supplied review thread, full diff, S3 cleanup callers, ARN normalization and URI parsing, client setup, dependencies, documentation, and tests.
Validation: git diff --check passed. No Druid tests or builds were run.
| Severity | Findings |
|---|---|
| P0 | 0 |
| P1 | 1 |
| P2 | 0 |
| P3 | 0 |
| Total | 1 |
This is an automated review by Codex GPT-5.6-Luna(max)
After addressing the findings or replying to the comments, you can request another review from me to trigger a new automated review.
Multi-Region Access Points support DeleteObject but not DeleteObjects, so deleteBucketKeys now deletes MRAP keys one request at a time while keeping the retry-failed-keys behaviour. Also names the bucket in the default validator error and documents the colon form for allowedExportPaths. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Updates the MRAP pusher test for the two-argument DeepStorageSegmentConfig constructor introduced upstream in apache#20806.
FrankChen021
left a comment
There was a problem hiding this comment.
🟡 Changes recommended
Follow-up assessment
The original DeleteObjects blocker is resolved for normalized MRAP buckets: cleanup now uses per-key DeleteObject calls with retry-failed-keys handling. One P1 remains because slash-form ARNs are normalized only for branch selection, while the actual delete call still receives the unnormalized value.
Reviewed 29 of 29 changed files; findings in 1 file. Coverage included cleanup callers, raw load specs, ARN normalization and URI parsing, client configuration, ACL handling, dependencies, documentation, and tests.
Validation: No Druid tests or builds were run. A narrow offline Maven dependency-tree check was blocked by an existing local Maven lock.
| Severity | Findings |
|---|---|
| P0 | 0 |
| P1 | 1 |
| P2 | 0 |
| P3 | 0 |
| Total | 1 |
This is an automated review by Codex GPT-5.6-Luna(max)
After addressing the findings or replying to the comments, you can request another review from me to trigger a new automated review.
| if (!response.hasErrors()) { | ||
| final List<S3Error> errors; | ||
| if (deleteIndividually) { | ||
| errors = deleteKeysIndividually(s3Client, bucket, remaining); |
There was a problem hiding this comment.
[P1] Pass normalized bucket to MRAP deletes
Finding: The MRAP branch normalizes the bucket only to choose per-object deletion, then passes the original value to deleteKeysIndividually. For the slash-form ARN accepted by this PR and carried by raw load specs, the AWS SDK receives the slash form instead of the colon-form ARN and can reject the request before any object is deleted.
Suggestion: Compute the normalized bucket once and pass it to deleteKeysIndividually, with the slash-form test asserting the normalized request bucket.
There was a problem hiding this comment.
I don't think the SDK treats the slash form differently. AWS's own SDK for Java examples for Multi-Region Access Points pass arn:aws:s3::123456789012:accesspoint/abcdef0123456.mrap as the bucket (MrapOperations), and I checked it against the SDK version this PR uses (2.57.2). I captured the DeleteObject request for each form with an interceptor that aborts before transmission. Apart from the timestamp and signature value, the two requests are identical:
DELETE https://abcdef0123456.mrap.accesspoint.s3-global.amazonaws.com:443/druid/segments/foo/index.zip- same headers, including
X-Amz-Region-Set: * - same signed headers, both signed with SigV4A (
AWS4-ECDSA-P256-SHA256)
S3 receives the same request either way, so the current code deletes correctly for both forms and I don't see a P1 here. Passing the normalized bucket to deleteKeysIndividually would still be fine for consistency with the rest of the PR, but it doesn't change behaviour.
Fixes #19608.
Description
Druid's S3 extension now accepts S3 Access Point and Multi-Region Access Point (MRAP) ARNs wherever it takes a bucket. That covers deep storage, task logs, the S3 input source, MSQ durable storage and export. Before this change,
CloudObjectLocationrejected any ARN because its DNS-naming check fails on:.Provider-neutral bucket validation
CloudObjectLocationno longer knows about S3. Its check still requires a bucket to URL-encode to itself, but now permits:. Provider rules live behind a new strategy interface,CloudBucketValidator, whichCloudObjectInputSourceapplies to every bucket inuris,prefixesandobjects.CloudBucketValidator.REJECT_COLONis the default. GCS, Azure and OSS input sources use it, so they still reject:at spec validation, as before.S3InputSourcepassesS3BucketValidator, which allows:only when the bucket is a valid Access Point ARN in colon form. Anything else fails with a message that suggests the colon form.CloudObjectLocation.extractBucket(URI)returns the host, or the authority when the host is null.URI.getHost()returns null for an ARN authority, so all bucket reads from a URI go through this helper, includingS3ExportStorageProvider.validateUriandS3DataSegmentPusher.makeLoadSpec(URI).I also considered a regex in
CloudObjectLocationshared by all providers. That puts S3 knowledge in a shared class and drifts from the S3 pattern, so the S3 rules now live only inS3Utils.ARN parsing and normalization in
S3UtilsisS3Arnmatches regional Access Point and MRAP ARNs in theaws,aws-cnandaws-us-govpartitions.isMultiRegionAccessPointArnmatches only ARNs with an empty Region field.normalizeBucketNameconverts theaccesspoint/<name>form that AWS documents to theaccesspoint:<name>form that the SDK expects. Plain bucket names pass through unchanged.Normalization happens where config enters Druid:
S3DataSegmentPusherConfig,S3TaskLogsConfig,S3LoadSpec,S3OutputConfig,S3ExportStorageProvider, andS3DataSegmentMover.move, which receives load specs as raw maps. ThesetBucketandsetS3Bucketsetters now carry@JsonProperty. Druid'sObjectMapperdisables setter auto-detection, so without the annotation Jackson wrote the field directly and skipped normalization.Request routing and signing
software.amazon.awssdk:http-auth-aws-crtis visible to the classloader that holdshttp-auth-aws, which islib/. Placing the module in the extension directory is not enough, so this PR adds it as a runtime dependency ofdruid-aws-common, next tos3andaws-crt-client.licenses.yamlis updated.S3Utils.configureAccessPointArnRegionsetsuseArnRegion(true)on the sync and async clients. Without it, the SDK rejects a regional Access Point ARN whose Region differs from the client Region before sending the request.Fail-fast checks for MRAP limits
GetBucketAcl, which MRAPs do not support.S3DataSegmentPusher,S3TaskLogsandS3DataSegmentMoverfail at construction or before the copy if the bucket is an MRAP anddisableAclis false. Regional Access Points supportGetBucketAcland are allowed.CopyObject.S3DataSegmentMoverfails with aSegmentLoadingExceptionwhen the source is an MRAP, instead of sending a request that S3 rejects.Deletes through an MRAP
MRAPs support
DeleteObjectbut not the multi-objectDeleteObjects. Every bulk delete in the extension goes throughS3Utils.deleteBucketKeys: stale-object cleanup after an unzipped push, segment kill, task-log cleanup, and MSQ durable storage cleanup. For an MRAP bucket it now sends oneDeleteObjectper key. A key that S3 refuses is reported as anS3Error, so the existing retry-only-failed-keys loop andS3MultiObjectDeleteExceptionbehave the same as for a bucket.Docs
s3.mdhas a new "Access Points and Multi-Region Access Points" section. It covers both ARN forms, thedisableAclrequirement for MRAPs, ARN Region routing, path-style access, STS credentials for SigV4A, and the move limitation.Release note
Druid's S3 extension accepts S3 Access Point and Multi-Region Access Point ARNs as a bucket. This works for deep storage, task logs, the S3 input source, MSQ durable storage and export. Both the
accesspoint/<name>andaccesspoint:<name>forms are accepted in runtime properties; input sourceurisandprefixesneed the colon form. Multi-Region Access Points requiredruid.storage.disableAcl=true(anddruid.indexer.logs.disableAcl=trueif task logs use one) and can't be the source of a segment move. ARN buckets don't work withdruid.s3.enablePathStyleAccess=true. Plain bucket names are unaffected.Key changed/added classes in this PR
CloudBucketValidator(new)CloudObjectLocationCloudObjectInputSourceS3BucketValidator(new)S3UtilsS3InputSourceS3DataSegmentPusher,S3DataSegmentPusherConfigS3DataSegmentMoverS3TaskLogs,S3TaskLogsConfigS3OutputConfig,S3ExportStorageProviderServerSideEncryptingAmazonS3This PR has: