Repository navigation
[fix](paimon) Fail scans of binary partitions with legacy partition names #69571
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
morningman
merged 3 commits into
apache:master
from
morningman:b2m-t14h-paimon-legacy-binary-partition-read
Oct 26, 2026
Merged
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
a77603c
[fix](paimon) Fail scans of binary partitions with legacy partition n…
morningman 65a1682
[fix](paimon) Check deletion vectors and confirm missing files of bin…
morningman cf79a1a
[fix](paimon) Check binary partitions of fallback splits under the op…
morningman File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[P2] Resolve fallback options for audit-log binary partitions
For
$audit_log,resolveSchemaDictTablereturns null, so this selects the system wrapper's options. Paimon's audit wrapper exposes the main branch's options while still planning and reading the fallback branch. For example, write only a NULL binary partition on main with the defaultpartition.legacy-name=true; create abackupbranch, set its persisted naming option tofalse, and write a non-NULL binary partition there. After setting main'sscan.fallback-branch=backup,SELECT id FROM t$audit_lognow throws the legacy-name error for the backup split, even though its byte-named file exists and the JNI reader selects backup correctly. Resolve the pinned system source and the split's producing branch independently of native-schema-dictionary availability, and add this wrapper/fallback case to the tests. This uses ordinary persisted branch options, with no dynamic writer override or schema-ID provenance ambiguity.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Confirmed, and fixed in
cf79a1ac228.resolveSchemaDictTablehas no answer for$audit_log, since anAuditLogTableis aDataTablerather than aFileStoreTable. The check then fell back to the wrapper's own options, andAuditLogTable.options()returns those of the fallback pair, which are the main branch's. The splits still come fromFallbackReadScan, which plans a partition that is missing on main from the fallback branch and reads it with that branch's read. I replayed your history with the Paimon 2.1.0 API. Main keeps the defaultpartition.legacy-name = trueand holds only a NULL partition, andbackuphasfalseand a row inkey = 0x4142.$audit_logreportstrue, while the fallback split's file is atkey=AB/bucket-0/…, exists, and Paimon's own read returns the row. The check rejected that scan although master reads it through JNI.For the system tables read through JNI, the check now resolves the table that produced the splits from the wrapper's pinned source, the same source the timestamp-history checks already use (
resolveTimestampHistoryTable). It then picks each split's branch as it does for data tables, so a fallback split of$audit_logor$row_trackingis judged by the naming and schemas of its own branch. In the reverse history (mainfalse,backuptrue) the scan now fails at planning with the legacy-name error. Before, it planned and Paimon's reader failed at execution with aFileNotFoundException.$binlogis not affected: it exposes its partition columns as arrays, so it has no binary partition column to check.fallbackBinaryPartitionsUseTheNamingOfTheirBranchcovers both directions for the data table and$audit_log, and reads the planned$audit_logsplits back through the wrapper.While there: a
scan.file-creation-time-millisread plans a fallback table through the pair'sSnapshotReader, which returns plainDataSplits of the wrapped branch.legacyOrcSplitTablecast every split toFallbackSplit, so such a scan of a table with a non-NULL binary partition failed with aClassCastException. A split that is not aFallbackSplitnow maps to the wrapped branch. The same cast on the native timestamp path predates this PR and is fixed with it.fileCreationTimeScanOfFallbackTableChecksBinaryPartitionsOfTheMainBranchplans such a read.