Skip to content

[improvement](external) Refine Parquet splits by row group in BE - #66802

Merged
yiguolei merged 14 commits into
apache:branch-4.1from
Gabriel39:dev/file-split-rowgroup-context-4.1
Aug 21, 2026
Merged

yiguolei merged 14 commits into
apache:branch-4.1from
Gabriel39:dev/file-split-rowgroup-context-4.1

Conversation

@Gabriel39

@Gabriel39 Gabriel39 commented Aug 16, 2026 •

Copy link
Copy Markdown
Contributor

Proposed changes

This PR implements two-level external-file split scheduling:

  • FE performs coarse-grained split planning and assigns each source split to a scan instance.
  • BE refines an eligible Parquet source split by row group or an eligible ORC source split by stripe, then schedules the generated tasks concurrently inside that instance.

Split-size policy

Scenario FE source split target BE refinement target
BE refinement is eligible file_split_size_on_fe, default 512 MiB file_split_size_on_be, default 64 MiB
BE refinement is ineligible Existing connector-specific split policy Not set

file_split_size_on_be replaces file_scanner_v2_split_size. The new name describes the generic target for BE-side file splitting rather than tying the policy to a scanner implementation.

The existing file_split_size variable is unchanged. It remains part of the legacy FE split policy and is used when BE refinement is ineligible. Eligible paths use the dedicated 512 MiB FE coarse target in both batch and non-batch planning.

FE coarse-grained planning

FE selects the coarse target per split instead of applying 512 MiB globally. Hive, Iceberg, native Paimon RawFile, and file TVF planning use file_split_size_on_fe only for a native Parquet/ORC range that BE can refine.

FE retains the original connector-specific split target and does not attach TFileRangeDesc.target_split_size for:

  • non-Parquet/ORC formats;
  • File Scanner V2-disabled scans;
  • max_file_scanners_concurrency = 1;
  • small-LIMIT scans that BE intentionally executes with one scanner;
  • transactional Hive scans;
  • table-level metadata COUNT ranges;
  • Iceberg data files carrying delete files;
  • serialized logical/JNI Paimon splits;
  • non-positive file_split_size_on_fe or file_split_size_on_be.

The fallback preserves existing file_split_size, adaptive initial/max split sizing, batch-mode behavior, and max_file_split_num handling. This prevents FE from enlarging a range when BE cannot recover the parallelism through fine-grained splitting.

For Iceberg, the decision is made per FileScanTask, so a scan containing both eligible and ineligible files preserves the correct policy for each file. Iceberg task planning remains lazy. For Paimon, only native RawFile Parquet/ORC splits use the coarse target; JNI and unsupported semantic paths remain unchanged.

BE fine-grained refinement

  • Publish BE-generated physical tasks through both local and remote split-source connectors, allowing multiple scanners to consume children produced from one FE source split.
  • Refine only the row groups or stripes owned by the FE source range after format-level pruning.
  • Reuse immutable footer or ORC-tail context only among physical children generated from the same parent source split.
  • Share only immutable file metadata. Each generated child retains independent reader, predicate, decoding, and progress state.
  • Coalesce consecutive row groups or stripes near file_split_size_on_be instead of creating one scanner task per physical unit.
  • Preserve exact physical ownership with format-local first/end IDs. Padding and pruned gaps cannot make a child read an unowned row group or stripe.
  • Leave an oversized row group or stripe indivisible.
  • Keep the initialized planning reader when refinement is unsafe or produces only one child.
  • Keep cache behavior unchanged.

Footer and tail reuse scope

  • Each parent source split resolves its own immutable FileContext and passes the same shared_ptr to all physical children it generates.
  • Parquet children reuse the parent metadata and adaptive context; ORC children create independent readers from the parent serialized file tail. Reader, predicate, decoding, and progress state remain child-local.
  • Two parent source splits do not share a FileContext, even when they refer to the same physical file. There is no FileScanLocalState registry or cross-parent single-flight coordination.
  • The existing process-level Parquet metadata cache is unchanged. It may satisfy independent parent reads, but it is separate from parent-to-child context reuse.
  • Consequently, ORC parents for the same file may parse and serialize the tail independently. This keeps context ownership bounded to one parent split and avoids scan-instance registry lifetime and synchronization costs.

Eligibility and limitations

  • Native Parquet and ORC ranges can be refined when File Scanner V2 is enabled and at least two scanners can consume generated work.
  • Native Paimon RawFile ranges support Parquet and ORC. Serialized logical/JNI Paimon splits remain unsplit because they can represent semantics beyond one physical file.
  • Iceberg ranges carrying delete files remain unsplit. Delete state is currently source-split-local and must not be rebuilt independently for generated physical children.
  • A stable file identity is required before immutable metadata can be shared with generated readers. Mutable or unversioned objects keep the initialized source reader.
  • Row groups and stripes are indivisible, so one physical unit may exceed the 64 MiB target.

End-to-end flow

  1. FE determines whether a connector split can be refined on BE.
  2. An eligible split uses the 512 MiB FE coarse target; an ineligible split retains its legacy FE target.
  3. FE assigns the source split and attaches the 64 MiB BE target only to an eligible range.
  4. BE opens and prunes the source range metadata.
  5. BE selects the exact row groups or stripes owned by that range.
  6. BE coalesces consecutive physical units near the requested target.
  7. Multiple scanners in the same instance consume the generated tasks; children from the same parent share its immutable file context.

Verification

  • FE Checkstyle passed with 0 violations.
  • 129 focused FE tests passed across FileQueryScanNodeTest, HiveScanNodeTest, IcebergScanNodeTest, PaimonScanNodeTest, and TVFScanNodeTest.
  • The final Paimon fallback guard was re-verified with all 45 PaimonScanNodeTest tests passing.
  • clang-format 16 passed for every affected C/C++ source and header.
  • The complete ASAN doris_be_test target compiled and linked successfully.
  • 56 focused ASAN tests passed for split-source coordination, Parquet/ORC physical planning, parent-scoped file-context reuse, ownership boundaries, target-size grouping, Paimon native eligibility, and Iceberg delete-file fallback.
  • After correcting the BE UT fixture lifecycle, all 112 TableReaderTest ASAN tests passed.

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants