Skip to content

feat: Handle MRAP URI - #19609

Open
vivek807 wants to merge 8 commits into
apache:masterfrom
deep-bi:deep/feature/Add-AWS-MRAP-support
Open

vivek807 wants to merge 8 commits into
apache:masterfrom
deep-bi:deep/feature/Add-AWS-MRAP-support

Conversation

@vivek807

@vivek807 vivek807 commented Jun 22, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #19608.

Description

Druid's S3 extension now accepts S3 Access Point and Multi-Region Access Point (MRAP) ARNs wherever it takes a bucket. That covers deep storage, task logs, the S3 input source, MSQ durable storage and export. Before this change, CloudObjectLocation rejected any ARN because its DNS-naming check fails on :.

Provider-neutral bucket validation

CloudObjectLocation no longer knows about S3. Its check still requires a bucket to URL-encode to itself, but now permits :. Provider rules live behind a new strategy interface, CloudBucketValidator, which CloudObjectInputSource applies to every bucket in uris, prefixes and objects.

  • CloudBucketValidator.REJECT_COLON is the default. GCS, Azure and OSS input sources use it, so they still reject : at spec validation, as before.
  • S3InputSource passes S3BucketValidator, which allows : only when the bucket is a valid Access Point ARN in colon form. Anything else fails with a message that suggests the colon form.
  • CloudObjectLocation.extractBucket(URI) returns the host, or the authority when the host is null. URI.getHost() returns null for an ARN authority, so all bucket reads from a URI go through this helper, including S3ExportStorageProvider.validateUri and S3DataSegmentPusher.makeLoadSpec(URI).

I also considered a regex in CloudObjectLocation shared by all providers. That puts S3 knowledge in a shared class and drifts from the S3 pattern, so the S3 rules now live only in S3Utils.

ARN parsing and normalization in S3Utils

  • isS3Arn matches regional Access Point and MRAP ARNs in the aws, aws-cn and aws-us-gov partitions. isMultiRegionAccessPointArn matches only ARNs with an empty Region field.
  • normalizeBucketName converts the accesspoint/<name> form that AWS documents to the accesspoint:<name> form that the SDK expects. Plain bucket names pass through unchanged.

Normalization happens where config enters Druid: S3DataSegmentPusherConfig, S3TaskLogsConfig, S3LoadSpec, S3OutputConfig, S3ExportStorageProvider, and S3DataSegmentMover.move, which receives load specs as raw maps. The setBucket and setS3Bucket setters now carry @JsonProperty. Druid's ObjectMapper disables setter auto-detection, so without the annotation Jackson wrote the field directly and skipped normalization.

Request routing and signing

  • MRAP requests need SigV4A signing. The AWS SDK enables it only when software.amazon.awssdk:http-auth-aws-crt is visible to the classloader that holds http-auth-aws, which is lib/. Placing the module in the extension directory is not enough, so this PR adds it as a runtime dependency of druid-aws-common, next to s3 and aws-crt-client. licenses.yaml is updated.
  • S3Utils.configureAccessPointArnRegion sets useArnRegion(true) on the sync and async clients. Without it, the SDK rejects a regional Access Point ARN whose Region differs from the client Region before sending the request.

Fail-fast checks for MRAP limits

  • ACLs: with ACLs enabled, Druid looks up the bucket owner with GetBucketAcl, which MRAPs do not support. S3DataSegmentPusher, S3TaskLogs and S3DataSegmentMover fail at construction or before the copy if the bucket is an MRAP and disableAcl is false. Regional Access Points support GetBucketAcl and are allowed.
  • Moves: S3 accepts an MRAP only as the destination of CopyObject. S3DataSegmentMover fails with a SegmentLoadingException when the source is an MRAP, instead of sending a request that S3 rejects.

Deletes through an MRAP

MRAPs support DeleteObject but not the multi-object DeleteObjects. Every bulk delete in the extension goes through S3Utils.deleteBucketKeys: stale-object cleanup after an unzipped push, segment kill, task-log cleanup, and MSQ durable storage cleanup. For an MRAP bucket it now sends one DeleteObject per key. A key that S3 refuses is reported as an S3Error, so the existing retry-only-failed-keys loop and S3MultiObjectDeleteException behave the same as for a bucket.

Docs

s3.md has a new "Access Points and Multi-Region Access Points" section. It covers both ARN forms, the disableAcl requirement for MRAPs, ARN Region routing, path-style access, STS credentials for SigV4A, and the move limitation.

Release note

Druid's S3 extension accepts S3 Access Point and Multi-Region Access Point ARNs as a bucket. This works for deep storage, task logs, the S3 input source, MSQ durable storage and export. Both the accesspoint/<name> and accesspoint:<name> forms are accepted in runtime properties; input source uris and prefixes need the colon form. Multi-Region Access Points require druid.storage.disableAcl=true (and druid.indexer.logs.disableAcl=true if task logs use one) and can't be the source of a segment move. ARN buckets don't work with druid.s3.enablePathStyleAccess=true. Plain bucket names are unaffected.


Key changed/added classes in this PR
  • CloudBucketValidator (new)
  • CloudObjectLocation
  • CloudObjectInputSource
  • S3BucketValidator (new)
  • S3Utils
  • S3InputSource
  • S3DataSegmentPusher, S3DataSegmentPusherConfig
  • S3DataSegmentMover
  • S3TaskLogs, S3TaskLogsConfig
  • S3OutputConfig, S3ExportStorageProvider
  • ServerSideEncryptingAmazonS3

This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • a release note entry in the PR description.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths. The code coverage threshold is met.
  • been tested in a test Druid cluster.

Signed-off-by: Vivek Dhiman <approach2vivek@gmail.com>

@FrankChen021 FrankChen021 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity Findings
P0 0
P1 0
P2 1
P3 0
Total 1

Reviewed 6 of 6 changed files.


This is an automated review by Codex GPT-5.5

@vivek807
vivek807 requested a review from FrankChen021 June 26, 2026 09:42

@FrankChen021 FrankChen021 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity Findings
P0 0
P1 0
P2 1
P3 0
Total 1

Reviewed 6 of 6 changed files.


This is an automated review by Codex GPT-5.5

Signed-off-by: Vivek Dhiman <approach2vivek@gmail.com>
@vivek807
vivek807 force-pushed the deep/feature/Add-AWS-MRAP-support branch from 9a40fd3 to e06de77 Compare July 17, 2026 11:35
@vivek807
vivek807 requested a review from FrankChen021 July 17, 2026 11:52

@FrankChen021 FrankChen021 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up handled: slash-form S3 Access Point and MRAP ARNs are now explicitly rejected by CloudObjectLocation, with coverage for both direct construction and URI parsing. Callers accepting slash form normalize it through S3Utils.normalizeBucketName.

Reviewed 6 of 6 changed files.


This is an automated review by Codex GPT-5.6-Sol

@github-actions

Copy link
Copy Markdown

This pull request has been marked as stale due to 60 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If you think
that's incorrect or this pull request should instead be reviewed, please simply
write any comment. Even if closed, you can still revive the PR at any time or
discuss it on the dev@druid.apache.org list.
Thank you for your contributions.

@github-actions github-actions Bot added the stale label Sep 17, 2026
@vivek807

vivek807 commented Oct 7, 2026

Copy link
Copy Markdown
Contributor Author

Gentle follow-up on this PR. The changes are ready from my side and still relevant. Please let me know if there are any concerns or additional changes needed. Happy to address them.

@github-actions github-actions Bot removed the stale label Oct 8, 2026
…r strategy;

added SigV4A signing, ACL and move-source checks for MRAPs.

Signed-off-by: Vivek Dhiman <approach2vivek@gmail.com>

@FrankChen021 FrankChen021 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Follow-up assessment: The original provider-specific-regex concern is resolved at this head: CloudObjectLocation is provider-neutral, while S3-specific validation is centralized in S3BucketValidator and S3Utils. Regional Access Point ARNs can still fail before transmission when their ARN region differs from the configured client region; fix that before merging.

Reviewed 22 of 22 changed files; findings in 1 file.

Validation: No Druid build or test commands were run. A narrow AWS SDK 2.57.2 endpoint-resolution check reproduced the regional ARN failure when useArnRegion is disabled.

Severity Findings
P0 0
P1 0
P2 1
P3 0
Total 1

This is an automated review by Codex GPT-5.6-Luna(max)

After addressing the findings or replying to the comments, you can request another review from me to trigger a new automated review.

@FrankChen021 FrankChen021 added the Approval Recommended Used when AI completes a review and reports no findings label Nov 12, 2026

@FrankChen021 FrankChen021 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No actionable issues were found. The regional ARN endpoint-resolution finding is resolved: both sync and async clients now enable ARN-region routing, with coverage in the incremental tests.

Reviewed 23 of 23 changed files. Coverage included the 4-file incremental diff, full diff reconciliation, URI parsing, normalization, ACL handling, S3 clients, data-segment lifecycle, task logs, input sources, dependencies, documentation, and tests.

Validation: Narrow offline Maven dependency-tree checks only; no Druid build or test commands were run.


This is an automated review by Codex GPT-5.6-Luna(max)

@amaechler amaechler left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the work on this! Some comments from mostly automated Claude reviews.

The inline comments cover the regional Access Point ACL check, path-style access, two remaining URI.getHost() call sites, and the default bucket validator for the other providers. Two smaller points outside the diff:

  • MSQ durable storage (S3OutputConfig) and export keep the raw bucket, so the slash-form ARN that AWS documents fails there: S3StorageConnector.listDir builds a CloudObjectLocation, which rejects /. Normalizing in the S3OutputConfig constructor (this.bucket = S3Utils.normalizeBucketName(bucket);) covers durable storage. Alternatively, the docs could say that only the colon form works for those.
  • The PR description still describes an ARN pattern check inside CloudObjectLocation, but that logic now lives in S3BucketValidator. Worth refreshing so the description and release note match the code.

Comment thread docs/development/extensions-core/s3.md Resolved
Comment thread docs/development/extensions-core/s3.md Resolved
Signed-off-by: Vivek Dhiman <approach2vivek@gmail.com>
@vivek807
vivek807 requested a review from amaechler November 13, 2026 05:28

@FrankChen021 FrankChen021 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Fix MRAP cleanup before merging. Reconciliation of the incremental and full diffs confirms that the earlier regional-ARN routing, provider-neutral validation, slash-form normalization, and ACL-scope findings are addressed. However, normalized MRAP ARNs still enter existing DeleteObjects cleanup paths. AWS’s MRAP supported-operations list does not include DeleteObjects. Consequently, unzipped deep-storage pushes can fail after upload, while task-log, MSQ, and multi-segment cleanup can fail or leave objects behind.

Reviewed 28 of 28 changed files; finding in 1 file.

Severity Findings
P0 0
P1 1
P2 0
P3 0
Total 1

This is an automated review by Codex GPT-5.6-Luna(max)

After addressing the findings or replying to the comments, you can request another review from me to trigger a new automated review.

@FrankChen021 FrankChen021 removed the Approval Recommended Used when AI completes a review and reports no findings label Nov 13, 2026

@amaechler amaechler left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick turnaround! Two small follow-ups inline, plus a concrete fix proposal on the DeleteObjects thread.

Comment thread docs/development/extensions-core/s3.md Resolved

@FrankChen021 FrankChen021 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The supplied reply is not reflected in head b5391acc: S3Utils.deleteBucketKeys still sends non-empty batches through DeleteObjects, which MRAPs do not support. Fix this before merging because unzipped pushes and other bulk cleanup paths can fail or leave objects behind.

Reviewed 28 of 28 changed files; finding in 1 file. Coverage included the supplied review thread, full diff, S3 cleanup callers, ARN normalization and URI parsing, client setup, dependencies, documentation, and tests.

Validation: git diff --check passed. No Druid tests or builds were run.

Severity Findings
P0 0
P1 1
P2 0
P3 0
Total 1

This is an automated review by Codex GPT-5.6-Luna(max)

After addressing the findings or replying to the comments, you can request another review from me to trigger a new automated review.

vivek807 and others added 2 commits November 16, 2026 10:21
Multi-Region Access Points support DeleteObject but not DeleteObjects,
so deleteBucketKeys now deletes MRAP keys one request at a time while
keeping the retry-failed-keys behaviour. Also names the bucket in the
default validator error and documents the colon form for
allowedExportPaths.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Updates the MRAP pusher test for the two-argument DeepStorageSegmentConfig
constructor introduced upstream in apache#20806.

@FrankChen021 FrankChen021 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Follow-up assessment

The original DeleteObjects blocker is resolved for normalized MRAP buckets: cleanup now uses per-key DeleteObject calls with retry-failed-keys handling. One P1 remains because slash-form ARNs are normalized only for branch selection, while the actual delete call still receives the unnormalized value.

Reviewed 29 of 29 changed files; findings in 1 file. Coverage included cleanup callers, raw load specs, ARN normalization and URI parsing, client configuration, ACL handling, dependencies, documentation, and tests.

Validation: No Druid tests or builds were run. A narrow offline Maven dependency-tree check was blocked by an existing local Maven lock.

Severity Findings
P0 0
P1 1
P2 0
P3 0
Total 1

This is an automated review by Codex GPT-5.6-Luna(max)

After addressing the findings or replying to the comments, you can request another review from me to trigger a new automated review.

if (!response.hasErrors()) {
final List<S3Error> errors;
if (deleteIndividually) {
errors = deleteKeysIndividually(s3Client, bucket, remaining);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Pass normalized bucket to MRAP deletes

Finding: The MRAP branch normalizes the bucket only to choose per-object deletion, then passes the original value to deleteKeysIndividually. For the slash-form ARN accepted by this PR and carried by raw load specs, the AWS SDK receives the slash form instead of the colon-form ARN and can reject the request before any object is deleted.

Suggestion: Compute the normalized bucket once and pass it to deleteKeysIndividually, with the slash-form test asserting the normalized request bucket.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think the SDK treats the slash form differently. AWS's own SDK for Java examples for Multi-Region Access Points pass arn:aws:s3::123456789012:accesspoint/abcdef0123456.mrap as the bucket (MrapOperations), and I checked it against the SDK version this PR uses (2.57.2). I captured the DeleteObject request for each form with an interceptor that aborts before transmission. Apart from the timestamp and signature value, the two requests are identical:

  • DELETE https://abcdef0123456.mrap.accesspoint.s3-global.amazonaws.com:443/druid/segments/foo/index.zip
  • same headers, including X-Amz-Region-Set: *
  • same signed headers, both signed with SigV4A (AWS4-ECDSA-P256-SHA256)

S3 receives the same request either way, so the current code deletes correctly for both forms and I don't see a P1 here. Passing the normalized bucket to deleteKeysIndividually would still be fine for consistency with the rest of the PR, but it doesn't change behaviour.

@amaechler amaechler left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🐻

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add AWS S3 Multi-Region Access Point (MRAP) support to the S3 extension

3 participants