Repository navigation
Conversation
taoran92
marked this pull request as ready for review
November 6, 2026 09:45
Member
Author
|
@Akash3121 Thanks for reviewing! @JingsongLi Could you also take a look when you have time? This PR fixes Dynamic Partition Pruning (DPP) failures when joining on masked partition keys. Thanks! |
taoran92
marked this pull request as draft
November 18, 2026 12:01
taoran92
force-pushed
the
fix/spark-query-auth-runtime-filtering
branch
from
November 18, 2026 12:07
3ecd512 to
02a0b54
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Fixes #11013.
A Spark JOIN on a masked partition key can fail when dynamic partition pruning (DPP) is enabled. For example, if two date partitions both expose
dtas'***', both can match a JOIN on'***'. Spark may push that value down as a physical partition filter, which Paimon rejects.Changes
For ordinary query-authorized tables, exclude masked partition keys from the runtime filter attributes advertised to Spark. If a table is partitioned by
(region, dt)and onlydtis masked, Spark can still prune onregionthrough the existingfilter()and scan re-planning path. The JOIN evaluatesdtusing its visible value.Fallback and chain scans skip DPP when any participating branch has query authorization enabled. Spark evaluates partition-key
IS NOT NULLchecks, including those inferred from an equality JOIN, after masking. Scans whose branches all have query authorization disabled retain their existing runtime filter attributes.Tests
1and2. Adding the unmaskedregionkey returned ID1, and the executed fact-table scan reportedEqual(region, CN)as its only runtime partition filter.NULLpartition key was masked to'***'.SparkPartitionMetadataAuthTestpassed on Spark 3.5.9. The Spark 3.2 and 3.3 connector modules compiled successfully.Follow-up work
Selective DPP for authorized multi-branch reads will be addressed in follow-up PRs. Fallback scans must identify keys that are safe across branches and preserve partition read priority. Chain scans must also retain the snapshot and delta files needed to reconstruct each selected logical partition. These changes extend into scan planning. If they require pruning cached splits, a separate follow-up will cover Spark and JMX file counts for shared files.