Repository navigation
[improvement](external) Refine Parquet splits by row group in BE - #66802
Merged
yiguolei merged 14 commits intoAug 21, 2026
Merged
Conversation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed changes
This PR implements two-level external-file split scheduling:
Split-size policy
file_split_size_on_fe, default 512 MiBfile_split_size_on_be, default 64 MiBfile_split_size_on_bereplacesfile_scanner_v2_split_size. The new name describes the generic target for BE-side file splitting rather than tying the policy to a scanner implementation.The existing
file_split_sizevariable is unchanged. It remains part of the legacy FE split policy and is used when BE refinement is ineligible. Eligible paths use the dedicated 512 MiB FE coarse target in both batch and non-batch planning.FE coarse-grained planning
FE selects the coarse target per split instead of applying 512 MiB globally. Hive, Iceberg, native Paimon RawFile, and file TVF planning use
file_split_size_on_feonly for a native Parquet/ORC range that BE can refine.FE retains the original connector-specific split target and does not attach
TFileRangeDesc.target_split_sizefor:max_file_scanners_concurrency = 1;file_split_size_on_feorfile_split_size_on_be.The fallback preserves existing
file_split_size, adaptive initial/max split sizing, batch-mode behavior, andmax_file_split_numhandling. This prevents FE from enlarging a range when BE cannot recover the parallelism through fine-grained splitting.For Iceberg, the decision is made per
FileScanTask, so a scan containing both eligible and ineligible files preserves the correct policy for each file. Iceberg task planning remains lazy. For Paimon, only native RawFile Parquet/ORC splits use the coarse target; JNI and unsupported semantic paths remain unchanged.BE fine-grained refinement
file_split_size_on_beinstead of creating one scanner task per physical unit.Footer and tail reuse scope
FileContextand passes the sameshared_ptrto all physical children it generates.FileContext, even when they refer to the same physical file. There is noFileScanLocalStateregistry or cross-parent single-flight coordination.Eligibility and limitations
End-to-end flow
Verification
FileQueryScanNodeTest,HiveScanNodeTest,IcebergScanNodeTest,PaimonScanNodeTest, andTVFScanNodeTest.PaimonScanNodeTesttests passing.doris_be_testtarget compiled and linked successfully.TableReaderTestASAN tests passed.