Skip to content

[spark] JOINs on masked partition keys fail with DPP - #11014

Draft
taoran92 wants to merge 1 commit into
apache:masterfrom
taoran92:fix/spark-query-auth-runtime-filtering
Draft

taoran92 wants to merge 1 commit into
apache:masterfrom
taoran92:fix/spark-query-auth-runtime-filtering

Conversation

@taoran92

@taoran92 taoran92 commented Nov 6, 2026 •

Copy link
Copy Markdown
Member

Purpose

Fixes #11013.

A Spark JOIN on a masked partition key can fail when dynamic partition pruning (DPP) is enabled. For example, if two date partitions both expose dt as '***', both can match a JOIN on '***'. Spark may push that value down as a physical partition filter, which Paimon rejects.

Changes

For ordinary query-authorized tables, exclude masked partition keys from the runtime filter attributes advertised to Spark. If a table is partitioned by (region, dt) and only dt is masked, Spark can still prune on region through the existing filter() and scan re-planning path. The JOIN evaluates dt using its visible value.

Fallback and chain scans skip DPP when any participating branch has query authorization enabled. Spark evaluates partition-key IS NOT NULL checks, including those inferred from an equality JOIN, after masking. Scans whose branches all have query authorization disabled retain their existing runtime filter attributes.

Tests

  • On Spark 3.5.8, the masked-key JOIN returned IDs 1 and 2. Adding the unmasked region key returned ID 1, and the executed fact-table scan reported Equal(region, CN) as its only runtime partition filter.
  • Enabling authorization only on the backup or delta branch disabled DPP for the corresponding scan. A fallback JOIN matched the backup row whose stored NULL partition key was masked to '***'.
  • All 27 cases in SparkPartitionMetadataAuthTest passed on Spark 3.5.9. The Spark 3.2 and 3.3 connector modules compiled successfully.

Follow-up work

Selective DPP for authorized multi-branch reads will be addressed in follow-up PRs. Fallback scans must identify keys that are safe across branches and preserve partition read priority. Chain scans must also retain the snapshot and delta files needed to reconstruct each selected logical partition. These changes extend into scan planning. If they require pruning cached splits, a separate follow-up will cover Spark and JMX file counts for shared files.

@taoran92
taoran92 marked this pull request as ready for review November 6, 2026 09:45

@Akash3121 Akash3121 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@taoran92

Copy link
Copy Markdown
Member Author

@Akash3121 Thanks for reviewing!

@JingsongLi Could you also take a look when you have time? This PR fixes Dynamic Partition Pruning (DPP) failures when joining on masked partition keys. Thanks!

@taoran92
taoran92 marked this pull request as draft November 18, 2026 12:01
@taoran92
taoran92 force-pushed the fix/spark-query-auth-runtime-filtering branch from 3ecd512 to 02a0b54 Compare November 18, 2026 12:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Spark JOINs on masked partition keys fail with DPP

2 participants