Repository navigation
[improvement](external) Refine external file splits in BE - #70664
morningman wants to merge 10 commits into
Conversation
|
run buildall |
|
/review |
|
Codex automated review failed and did not complete. Error: Review setup stopped before context preparation ran; inspect the first failed or cancelled setup step. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
334885e to
b39a9d0
Compare
|
run buildall |
### Background FE plans external-file ranges and BE refines eligible native ranges into physical row-group or stripe tasks. Predicate snapshots, metadata aggregates and condition cache state must remain reusable across these tasks without losing the existing FE planning safeguards. ### What problem does this PR solve? Issue Number: None Related PR: apache#70664 (review follow-up) PR Type: Bugfix Problem Summary: Address the five performance regressions identified in the review of two-level external-file splitting. Reuse lazy predicate keys, retain metadata-only ORC readers, assemble child cache bitmaps once, preserve FE split-count heuristics, and retain legacy HTTP TVF range parallelism. ### Details - **Change map:** 416 changed production lines and 480 changed test lines across 35 files; no generated files, dependencies or wire changes. - **Frequency:** Scans with repeated runtime-filter payloads, metadata-only ORC aggregates, many cache-enabled physical children, a binding max_file_split_num, or HTTP TVF files eligible for legacy byte splitting. - **Root cause:** Per-split digest construction reaches nonconsuming readers; ORC omits the metadata aggregate hook; child bitmap merging repeatedly copies its prefix; coarse targets replace count-derived bounds; HTTP uses coarse sizing despite an unstable file identity. - **Why existing tests missed it:** Existing tests cover results and isolated sizing helpers, but omit digest call counts, real ORC planning-reader retention, multi-child bitmap ownership, and integrated coarse-target range counts. - **Fix:** Share an immutable scanner predicate snapshot with a lazy key requested through a reader capability. Delegate ORC metadata aggregation to its existing implementation. Transfer completed bitmap owners under the coordinator mutex and assemble their union outside it after all matching EOFs. Apply count-derived bounds after coarse sizing and opt HTTP TVFs out of coarse planning. - **Other paths and branches:** Forward snapshots through Hudi, Paimon, Fluss and JNI wrappers. Cover synchronous SDK and manifest-cache Iceberg planning; keep streaming lazy. Keep native Paimon COUNT and existing unsupported aggregate fallbacks. Adapt the encryption fixture to the shared-metadata ownership change and initialize the scanner refresh-error fixture with a valid local state. ### Testing - Standard FE script: 400 tests passed across TVFScanNodeTest, PaimonScanPlanProviderTest and IcebergScanPlanProviderTest. The native Paimon and synchronous Iceberg range-count regressions fail on the original production code and pass after the fixes. - Standard FE build including Paimon and Iceberg completed with Checkstyle and plugin packaging. - Standard ASAN BE script: 257 tests in 10 suites passed, including all TableReaderTest and FileScannerV2Test cases, real ORC metadata aggregation, hybrid forwarding, the encryption fixture and binary column iterator coverage. - Changed C++ lines passed clang-tidy; clang-format v16 and build hygiene passed. - No live external-cluster regression or latency benchmark was run. ### Release note Preserve split-count safeguards and HTTP TVF parallelism during external scans, and avoid repeated runtime-filter hashing, ORC metadata reader creation and child condition-cache bitmap copying. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: No; restores the intended planning and cache behavior of the pending feature without adding settings. - Pre-submission status: Ready ### AI Disclosure - Tool: Codex (GPT-6) - Generated: Code, tests and this commit description. - Human review: Not stated.
### Background Standalone ORC readers can leave ReaderOptions metrics unset. Statistics loading and predicate pruning support those callers. ### What problem does this PR solve? Issue Number: None Related PR: apache/doris#70664 PR Type: Bugfix Problem Summary: The new statistics counters dereferenced the optional metrics pointer. Keep instrumentation conditional so default ReaderOptions continues to support statistics reads and SARG pruning. ### Details - **Frequency:** Always when the new counters run without configured metrics. - **Root cause:** The metrics pointer defaults to nullptr, but the added counters assumed it was configured. - **Why existing tests missed it:** The new counter tests supplied metrics explicitly; running the existing standalone-reader cases exposed the missing guards. - **Fix:** Follow SargsApplier's existing optional-metrics pattern for both Metadata parsing and stripe evaluation. - **Other paths and branches:** Both full-file and interval APIs use the guarded counters. No format or config changes. ### Testing All 238 selected Doris ORC and SDK reader/predicate tests passed locally under ASAN, including default ReaderOptions, concurrent cold/warm statistics sharing, and exact stripe ownership. ### Release note None ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Ready ### AI Disclosure - Tool: Codex - Generated: Metrics guards and this message. - Human review: not stated
### Background The shared statistics loader uses std::call_once. ORC already discovers the platform thread dependency during configuration. ### What problem does this PR solve? Issue Number: None Related PR: apache/doris#70664 PR Type: Build/CI/Chore Problem Summary: Declare Threads::Threads on the ORC library so standalone native consumers inherit the loader's thread linkage. ### Details - **Affected:** ORC library link interface. - **Developer and release impact:** Standalone consumers receive the required platform thread flags; Doris already links threads. - **License and third-party:** No new dependency; use the existing discovered system thread target. - **Rollback:** Remove the target dependency together with the shared loader. ### Testing Reconfigured through the standard Doris BE unit script and passed all 238 selected ORC reader/predicate tests under ASAN. ### Release note None ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Ready ### AI Disclosure - Tool: Codex - Generated: Link dependency and this message. - Human review: not stated
### Background Independent readers share lazily parsed immutable stripe statistics. ### What problem does this PR solve? Issue Number: None Related PR: apache/doris#70664 (downstream physical stripe readers) PR Type: Bugfix Problem Summary: A throwing call_once initializer relies on runtime exception-reset behavior. Replace it with a mutex and an explicit loaded state so failed loaders cannot publish success and other readers can retry with their own streams. ### Details - **Frequency:** Reported in SDK review on an affected runtime combination; not reproduced on the local macOS runtime or in a running Doris BE. - **Root cause:** pthread_once-backed call_once implementations can fail to reset initialization state after an exception. - **Why existing tests missed it:** Successful cold/warm sharing tests did not inject Metadata load failures. - **Fix:** Protect shared state with a mutex, parse locally, and mark loaded only after success. Keep successful empty Metadata distinct from an unloaded handle. - **Other paths and branches:** All Metadata accessors use this loader. This corrects code introduced in the same unmerged SDK PR; no public API or file-format changes in this followup. ### Testing All 239 selected Doris/SDK reader and predicate tests passed under ASAN, including 196 Doris ORC tests. The new test injects both a read exception and an invalid protobuf tag after exporting a cold handle and destroying its parent. It checks retry through a different reader and cache reuse by the failed reader and a later sibling after both loaders are destroyed. The same new test passes on the unchanged call_once implementation locally; this is a conforming-runtime control, not a reproduced failing baseline. ### Release note None ### Check List (For Author) - Behavior changed: No - Does this need documentation: No; existing API contracts remain unchanged. - Pre-submission status: Partially verified: the affected runtime combination is unavailable locally. ### AI Disclosure - Tool: Codex - Generated: implementation, tests and description. - Human review: not stated - Prompts: Fix the review finding on apache#435 and update that PR.
* [fix](orc) Share stripe statistics between range readers ### Background Independent ORC readers use separate streams and predicate appliers. Stripe statistics live in a file-level Metadata protobuf, separate from the serialized file tail. ### What problem does this PR solve? Issue Number: None Related PR: apache/doris#70664 (physical stripe readers) PR Type: Bugfix Problem Summary: Physical stripe readers repeat complete Metadata parsing and all-file SARG evaluation. Eight one-stripe readers plus their planning reader can parse Metadata nine times and evaluate 72 stripe statistics. Share lazy immutable Metadata and add bounded stripe evaluation while preserving reader-local predicates and row-group filtering. ### Details - **Frequency:** Found by code review; triggered by independent readers with SARG over the same multi-stripe file. - **Root cause:** Serialized tails omit Metadata, and getNeedReadStripes evaluates the whole file regardless of the reader's range. - **Why existing tests missed it:** Predicate tests check pruning results without counting repeated setup across independent readers. - **Fix:** Add an opaque thread-safe lazy statistics handle, an interval overload and operation counters. A successful load publishes immutable Metadata; failed loads remain retryable. The handle retains no streams, allocators, metrics or SARG appliers. - **Other paths and branches:** The existing full-file API delegates to the interval implementation. No format or default changes; native consumers must rebuild for the extended interface. ### Testing Local Doris ASAN BE build passed with this SDK candidate. Standard BE unit tests, including the new concurrent cold/warm-handle test, are running; results will be updated before merge. The test checks one parse after parent destruction, bounded evaluation, independent predicates, row-group pruning and absolute row positions. ### Release note None ### Check List (For Author) - Behavior changed: No - Does this need documentation: No; SDK interface contracts are documented in the header. - Pre-submission status: Partially verified: unit tests are running. ### AI Disclosure - Tool: Codex - Generated: SDK implementation, tests and description. - Human review: not stated - Prompts: Fix F-01 from the review of apache/doris#70664 and update that PR. * [fix](orc) Preserve optional reader metrics ### Background Standalone ORC readers can leave ReaderOptions metrics unset. Statistics loading and predicate pruning support those callers. ### What problem does this PR solve? Issue Number: None Related PR: apache/doris#70664 PR Type: Bugfix Problem Summary: The new statistics counters dereferenced the optional metrics pointer. Keep instrumentation conditional so default ReaderOptions continues to support statistics reads and SARG pruning. ### Details - **Frequency:** Always when the new counters run without configured metrics. - **Root cause:** The metrics pointer defaults to nullptr, but the added counters assumed it was configured. - **Why existing tests missed it:** The new counter tests supplied metrics explicitly; running the existing standalone-reader cases exposed the missing guards. - **Fix:** Follow SargsApplier's existing optional-metrics pattern for both Metadata parsing and stripe evaluation. - **Other paths and branches:** Both full-file and interval APIs use the guarded counters. No format or config changes. ### Testing All 238 selected Doris ORC and SDK reader/predicate tests passed locally under ASAN, including default ReaderOptions, concurrent cold/warm statistics sharing, and exact stripe ownership. ### Release note None ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Ready ### AI Disclosure - Tool: Codex - Generated: Metrics guards and this message. - Human review: not stated * [chore](orc) Declare thread linkage for statistics loading ### Background The shared statistics loader uses std::call_once. ORC already discovers the platform thread dependency during configuration. ### What problem does this PR solve? Issue Number: None Related PR: apache/doris#70664 PR Type: Build/CI/Chore Problem Summary: Declare Threads::Threads on the ORC library so standalone native consumers inherit the loader's thread linkage. ### Details - **Affected:** ORC library link interface. - **Developer and release impact:** Standalone consumers receive the required platform thread flags; Doris already links threads. - **License and third-party:** No new dependency; use the existing discovered system thread target. - **Rollback:** Remove the target dependency together with the shared loader. ### Testing Reconfigured through the standard Doris BE unit script and passed all 238 selected ORC reader/predicate tests under ASAN. ### Release note None ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Ready ### AI Disclosure - Tool: Codex - Generated: Link dependency and this message. - Human review: not stated * [fix](orc) Keep shared statistics retryable after load failures ### Background Independent readers share lazily parsed immutable stripe statistics. ### What problem does this PR solve? Issue Number: None Related PR: apache/doris#70664 (downstream physical stripe readers) PR Type: Bugfix Problem Summary: A throwing call_once initializer relies on runtime exception-reset behavior. Replace it with a mutex and an explicit loaded state so failed loaders cannot publish success and other readers can retry with their own streams. ### Details - **Frequency:** Reported in SDK review on an affected runtime combination; not reproduced on the local macOS runtime or in a running Doris BE. - **Root cause:** pthread_once-backed call_once implementations can fail to reset initialization state after an exception. - **Why existing tests missed it:** Successful cold/warm sharing tests did not inject Metadata load failures. - **Fix:** Protect shared state with a mutex, parse locally, and mark loaded only after success. Keep successful empty Metadata distinct from an unloaded handle. - **Other paths and branches:** All Metadata accessors use this loader. This corrects code introduced in the same unmerged SDK PR; no public API or file-format changes in this followup. ### Testing All 239 selected Doris/SDK reader and predicate tests passed under ASAN, including 196 Doris ORC tests. The new test injects both a read exception and an invalid protobuf tag after exporting a cold handle and destroying its parent. It checks retry through a different reader and cache reuse by the failed reader and a later sibling after both loaders are destroyed. The same new test passes on the unchanged call_once implementation locally; this is a conforming-runtime control, not a reproduced failing baseline. ### Release note None ### Check List (For Author) - Behavior changed: No - Does this need documentation: No; existing API contracts remain unchanged. - Pre-submission status: Partially verified: the affected runtime combination is unavailable locally. ### AI Disclosure - Tool: Codex - Generated: implementation, tests and description. - Human review: not stated - Prompts: Fix the review finding on #435 and update that PR. --------- Co-authored-by: morningman <moringman@apache.org>
### Background FE plans external-file ranges and BE refines eligible native ranges into physical row-group or stripe tasks. Predicate snapshots, metadata aggregates and condition cache state must remain reusable across these tasks without losing the existing FE planning safeguards. ### What problem does this PR solve? Issue Number: None Related PR: apache#70664 (review follow-up) PR Type: Bugfix Problem Summary: Address the five performance regressions identified in the review of two-level external-file splitting. Reuse lazy predicate keys, retain metadata-only ORC readers, assemble child cache bitmaps once, preserve FE split-count heuristics, and retain legacy HTTP TVF range parallelism. ### Details - **Change map:** 416 changed production lines and 480 changed test lines across 35 files; no generated files, dependencies or wire changes. - **Frequency:** Scans with repeated runtime-filter payloads, metadata-only ORC aggregates, many cache-enabled physical children, a binding max_file_split_num, or HTTP TVF files eligible for legacy byte splitting. - **Root cause:** Per-split digest construction reaches nonconsuming readers; ORC omits the metadata aggregate hook; child bitmap merging repeatedly copies its prefix; coarse targets replace count-derived bounds; HTTP uses coarse sizing despite an unstable file identity. - **Why existing tests missed it:** Existing tests cover results and isolated sizing helpers, but omit digest call counts, real ORC planning-reader retention, multi-child bitmap ownership, and integrated coarse-target range counts. - **Fix:** Share an immutable scanner predicate snapshot with a lazy key requested through a reader capability. Delegate ORC metadata aggregation to its existing implementation. Transfer completed bitmap owners under the coordinator mutex and assemble their union outside it after all matching EOFs. Apply count-derived bounds after coarse sizing and opt HTTP TVFs out of coarse planning. - **Other paths and branches:** Forward snapshots through Hudi, Paimon, Fluss and JNI wrappers. Cover synchronous SDK and manifest-cache Iceberg planning; keep streaming lazy. Keep native Paimon COUNT and existing unsupported aggregate fallbacks. Adapt the encryption fixture to the shared-metadata ownership change and initialize the scanner refresh-error fixture with a valid local state. ### Testing - Standard FE script: 400 tests passed across TVFScanNodeTest, PaimonScanPlanProviderTest and IcebergScanPlanProviderTest. The native Paimon and synchronous Iceberg range-count regressions fail on the original production code and pass after the fixes. - Standard FE build including Paimon and Iceberg completed with Checkstyle and plugin packaging. - Standard ASAN BE script: 257 tests in 10 suites passed, including all TableReaderTest and FileScannerV2Test cases, real ORC metadata aggregation, hybrid forwarding, the encryption fixture and binary column iterator coverage. - Changed C++ lines passed clang-tidy; clang-format v16 and build hygiene passed. - No live external-cluster regression or latency benchmark was run. ### Release note Preserve split-count safeguards and HTTP TVF parallelism during external scans, and avoid repeated runtime-filter hashing, ORC metadata reader creation and child condition-cache bitmap copying. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: No; restores the intended planning and cache behavior of the pending feature without adding settings. - Pre-submission status: Ready ### AI Disclosure - Tool: Codex (GPT-6) - Generated: Code, tests and this commit description. - Human review: Not stated.
### Background Physical ORC children use independent readers for their owned stripe intervals. The serialized file tail shares layout metadata, but omits the stripe-statistics Metadata protobuf used by SARG pruning. ### What problem does this PR solve? Issue Number: None Related PR: apache/doris-thirdparty#435 PR Type: Bugfix Problem Summary: Every physical child repeats full-file Metadata parsing and stripe selection. Share the SDK's lazy immutable statistics handle and evaluate only each reader's owned stripe interval while retaining reader-local predicates and row-group pruning. ### Details - **Frequency:** Always for multi-child ORC scans with a supported SARG; reproduced with eight selected stripes. - **Root cause:** Each SDK reader loaded its own Metadata and getNeedReadStripes evaluated every stripe regardless of ownership. - **Why existing tests missed it:** The physical-child test bounded Doris directory visits but did not count SDK parsing or evaluation; external regression tests compare results only. - **Fix:** Carry the SDK statistics handle in the existing file context and use bounded stripe selection. Keep streams, metrics and predicate appliers private to each reader. The handle remains lazy when planning has no SARG. - **Other paths and branches:** Individual and coalesced children, stronger child predicates, default SDK options and standalone full-file calls are covered. No FE, wire or file-format changes; release-branch behavior is not audited in this followup. - **Behavior change:** Add OrcStripeStatisticsParses and OrcStripeStatisticsEvaluations profile counters. The latter measures explicit stripe-selection work, excluding normal RowReader PPD. ### Testing All 239 selected native ORC reader/predicate tests passed under ASAN, including all 196 Doris ORC reader tests. Coverage includes concurrent cold/warm SDK readers, independent predicates, row-group filtering, absolute positions and read/parse failure followed by retry through another reader after parent destruction. An earlier negative control restoring unshared/full-file SDK calls failed the real-factory test with nine parses and 72 evaluations; the fixed path records one parse and 16 evaluations. These are operation counts, not elapsed-time measurements. ASAN BE build, clang-format16, build hygiene, changed-line clang-tidy and git diff --check passed. Pin the merged SDK apache#435 commit 4ec25901b1266235f56ebe5c948277d6f3c2c616, whose complete source tree matches the tested SDK candidate. External end-to-end suites were not run. The affected pthread_once runtime combination is unavailable locally; its reported hang was not reproduced here. ### Release note None ### Check List (For Author) - Behavior changed: Yes (profile counters, see Details) - Does this need documentation: No additional user-facing setting; existing two-level splitting documentation plan remains in the PR. - Pre-submission status: Partially verified: external end-to-end suites and the affected pthread_once runtime combination were not exercised locally. ### AI Disclosure - Tool: Codex - Generated: Implementation, tests, verification and this description. - Human review: not stated - Prompts: Fix F-01 from the review of PR apache#70664 and update that PR.
add70a4 to
a1ad4bb
Compare
* [opt](orc) Share parsed file tails across range readers ### Background ORC range readers need file-wide footer metadata and absolute stripe row prefixes. Physical split scheduling creates independent readers for one immutable file. ### What problem does this PR solve? Issue Number: None Related PR: apache/doris#70664 (introduces physical child readers) PR Type: Performance Problem Summary: Each physical child currently reparses the serialized file tail and rebuilds all stripe row prefixes. Reuse a parsed immutable tail and initialize each RowReader from its byte interval, keeping streams, schema, predicates and metrics private. ### Details - **Change map:** 225 production lines implement parsed-tail sharing, bounded range/seek lookup, operation counters and corrupt-directory rejection; 142 test lines cover concurrency, lifetime and edge cases. - **Why it was slow:** createReader reparses the FileTail protobuf and RowReaderImpl scans every stripe for each child. - **Optimization:** Share immutable Footer/PostScript and a memory-pool-backed absolute row-prefix buffer; binary-search byte and row boundaries. Reject an unordered stripe directory before binary search can misselect data. - **Results:** Static operation counts for an eight-stripe source plus eight individual children: tail parses 9 to 1, prefix entries built 72 to 8, RowReader range-accumulation visits 72 to 16. Binary-search comparisons are additional logarithmic work. These counts are source-derived; no elapsed-time or executed-test result is claimed. - **Applicability and cost:** Helps sibling readers of one immutable file, particularly uncoalesced stripes. Independent standalone readers retain their existing API. A parsed tail retains one prefix buffer in its source memory pool, which must outlive every shared handle. The new Reader API requires consumers to rebuild. Metadata-only opens now build the prefix when the tail is parsed; that path is not benchmarked. ### Testing clang-format 16.0.6 and git diff --check passed. Added concurrent range/seek, parent-destruction, serialized-tail compatibility, empty-file and malformed-directory SDK tests. Build and runtime tests have not run because the current worktree's installed dependencies point into a repository excluded by AGENTS.md; permission to read them is pending. ### Release note Reduce repeated ORC file-metadata initialization for physical split readers. ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Partially verified: compilation, runtime tests and elapsed-time performance remain unverified ### AI Disclosure - Tool: Codex - Generated: Code, tests and this description - Human review: Pending author review * [opt](orc) Build shared stripe prefixes only when reading rows ### Background Physical ORC children reuse parsed file tails and absolute stripe row prefixes. Readers that only inspect metadata do not need row prefixes. ### What problem does this PR solve? Issue Number: None Related PR: apache/doris#70664 (review follow-up) PR Type: Performance Problem Summary: Keep the shared prefix lazy so metadata-only opens avoid stripe traversal and allocation. Concurrent children build one immutable prefix with std::call_once, and an initialization failure remains retryable. ### Details - **Why it was slow:** Eager prefix construction adds file-wide work to metadata-only opens. Before parsed-tail sharing, each physical child also reparsed the tail and rebuilt every prefix. - **Optimization:** Build the source-pool-backed prefix only when a RowReader needs it and publish only after success. Keep streams, mutable schemas, predicates and metrics private. - **Results:** The eight-stripe real TableReader fixture passes at one tail parse, eight prefix entries and sixteen range visits. Temporarily restoring repeated setup makes the same test fail at nine, seventy-two and seventy-two, including the nonpositive-target case. SDK metadata-only opens build zero prefix entries; cold concurrent children build eight in total and warm children build none. These are measured operation counts, not elapsed-time claims. - **Applicability and cost:** Reuses work within one immutable file context. Byte/row range lookup uses logarithmic comparisons in addition to the counted accumulation visits. The source memory pool must outlive its shared handles. Serialized-tail callers remain supported; no session, wire or storage format change is introduced by this follow-up. ### Testing Local BE ASAN build through build.sh passed. run-be-ut.sh ran 255 related tests with zero failures or skips after restoring the candidate from the counterfactual. Coverage includes reader values/absolute row positions, read intent, physical splitting, shared statistics, cold concurrent initialization, warm reuse, parent destruction, seeks, empty files/ranges and repeated malformed-directory errors. Changed-line BE clang-tidy, clang-format 16, build-header hygiene and diff checks passed. SDK formatting passed separately; SDK code is excluded from the BE clang-tidy script by policy. The counterfactual is removed and all candidate source hashes match. No cluster regression or RELEASE timing benchmark was run. ### Release note Reduce repeated ORC physical-child metadata setup while preserving lazy metadata-only opens. ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Ready ### AI Disclosure - Tool: Codex - Generated: Code, tests and this description - Human review: Pending author review --------- Co-authored-by: morningman <moringman@apache.org>
### Background Doris plans external-file byte ranges in FE and assigns them to BE scanners. Native Parquet and ORC readers then select the row groups or stripes belonging to each range. FE range sizing determines the initial work distribution, while the physical file layout determines what a reader can scan independently. ### What problem does this PR solve? Issue Number: None Related PR: apache#66802 (branch-4.2 source), apache#68143 (Iceberg missing-file policy in split planning) PR Type: Enhancement Problem Summary: This ports the two-level external-file splitting from branch-4.2 apache#66802 to the current scanner and connector architecture. FE selects coarse ranges for eligible native scans, and BE refines them into row-group or stripe tasks that scanners in the same instance can consume. The port also handles source completion, shared immutable metadata, runtime filters, condition-cache publication, and the plugin request paths. Eligible scans use dedicated FE and BE size targets after upgrading. Original author: @Gabriel39 ### Details - **Change map:** 2,247 changed production lines across 49 files; 2,562 test lines across 25 files; two build-metadata lines in one file; no generated outputs or documentation. BE adds physical split scheduling, exact ownership and parent-scoped metadata/cache state. FE and SPI propagate coarse sizing through synchronous, partition-batch and streaming planning; connectors retain their source-specific exclusions. Tests port the original BE cases and cover the new engine/plugin seams. - **Before and after:** FE previously used each source's existing split policy. Eligible Parquet/ORC scans now use a 512 MiB FE target; BE groups physical children toward 64 MiB. For a 96 MiB Iceberg fixture without explicit split offsets, tests verify three ranges with a 32 MiB coarse hint and two with 64 MiB, including streaming. Files with Iceberg deletes, logical/JNI Paimon splits, metadata COUNT and serial scans retain their applicable fallback. BE may refine eligible ranges even when FE omits the target; a positive target controls coalescing only. - **Configs and defaults:** Add session variables `file_split_size_on_fe=536870912` and `file_split_size_on_be=67108864`, both forwarded and changeable through SET. Nonpositive values keep legacy FE planning; they do not disable BE refinement. The deprecated `enable_file_scanner_v2` compatibility variable remains ineffective. - **Impact on existing users:** Eligible ranges use the dedicated targets instead of `file_split_size`; that variable remains the fallback policy. Results and table-format delete semantics are preserved. Load retains FE range boundaries. The optional Thrift field uses the same id 17 as 4.1/4.2. Older BE ignores it and cannot recover coarse-range parallelism, so upgrade BE first. Connector API becomes 35.0; plugins must be rebuilt together with FE. ### Testing - Standard FE test script: 790 tests in 52 classes passed after rebasing onto current master, including the 533 affected engine/connector tests, the complete 163-test SPI suite and 94 predicate/converter tests covering the newly merged changes. Both recorded SPI surface contracts passed. - Five targeted temporary mutants lost either synchronous/batch request propagation or one connector's reuse-key input. Every control passed and every mutant failed its intended assertion. - Before the final rebase, all 27 modified BE translation units passed syntax checks against the generated compilation database; a deliberate failing probe confirmed detection. BE and protocol sources are unchanged since. - Five changed regression scripts passed Groovy compilation before the final rebase and are unchanged since. Actual BE unit tests and external end-to-end suites have not run: the installed third-party package lacks the BSON/MongoDB 2.0 include directories required to generate the BE build. No running-cluster evidence is claimed. ### Notes for Reviewers Start with source-split publication and completion, then physical ownership and cache finalization. Children share only their parent's immutable metadata and synchronized hints; decoding, predicates and progress remain reader-local. Keep the refinement-planning missing-file classifier aligned with normal Iceberg reads. Iceberg preserves SDK split offsets, so an explicit offset list can override the requested size. Existing tests that deliberately force tiny FE ranges now disable coarse FE planning to retain their coverage. The new regression compares parallel configurations with a single-scanner reference and requires no handwritten expected output. ### Release note Native Parquet and ORC external scans gain FE coarse splitting and BE physical refinement, controlled by `file_split_size_on_fe` (512 MiB) and `file_split_size_on_be` (64 MiB). Eligible scans use these targets instead of `file_split_size`. Set a nonpositive dedicated size to retain legacy FE sizing; BE can still refine eligible ranges without a coalescing target. Upgrade BE before FE and rebuild connector plugins for API 35.0. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: Yes; update the external scan split policy and both session-variable entries in apache/doris-website development documentation. - Pre-submission status: Partially verified: BE unit execution and external end-to-end regression are blocked by missing third-party BSON/MongoDB 2.0 dependencies. ### AI Disclosure - Tool: Codex (GPT-6) - Generated: Port adaptations, tests, evidence collection, and this description. - Human review: not stated - Prompts: Start the next item from the batch handoff. Follow the settled decision to retain branch-4.2 BE refinement without an FE authorization gate, and use engine-coordinated FE/SPI planning in one PR.
### Background The binary column iterator test helper limits each batch to the remaining source rows. std::min requires both arguments to have the same type. ### What problem does this PR solve? Issue Number: None Related PR: None PR Type: Bugfix Problem Summary: The helper does not compile on macOS because size_t is unsigned long while the ordinal_t subtraction yields unsigned long long. Specify the size_t template argument so both supported 64-bit platforms use the existing size_t result type. ### Details - **Frequency:** Compiling the BE unit-test executable on macOS. - **Root cause:** Template deduction sees different integer types despite their equal width on the supported platform. - **Why existing tests missed it:** Matching Linux typedefs do not exercise the Darwin compilation mismatch. - **Fix:** Use std::min<size_t> and explain the platform type difference beside the expression. - **Other paths and branches:** Test helper only; no production behavior or wire changes. ### Testing - The original expression fails to compile on Darwin arm64 with LLVM 22. - The standard ASAN BE run passes 257 tests, including all 13 BinaryColumnExtractIteratorV2Test cases and the related VariantAssemblerTest and SparseColumnMergeIteratorTest cases. - The edited lines pass clang-tidy and formatting checks. ### Release note None ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Ready ### AI Disclosure - Tool: Codex (GPT-6) - Generated: Test compilation fix and this description. - Human review: Not stated.
### Background FE plans external-file ranges and BE refines eligible native ranges into physical row-group or stripe tasks. Predicate snapshots, metadata aggregates and condition cache state must remain reusable across these tasks without losing the existing FE planning safeguards. ### What problem does this PR solve? Issue Number: None Related PR: apache#70664 (review follow-up) PR Type: Bugfix Problem Summary: Address the five performance regressions identified in the review of two-level external-file splitting. Reuse lazy predicate keys, retain metadata-only ORC readers, assemble child cache bitmaps once, preserve FE split-count heuristics, and retain legacy HTTP TVF range parallelism. ### Details - **Change map:** 416 changed production lines and 480 changed test lines across 35 files; no generated files, dependencies or wire changes. - **Frequency:** Scans with repeated runtime-filter payloads, metadata-only ORC aggregates, many cache-enabled physical children, a binding max_file_split_num, or HTTP TVF files eligible for legacy byte splitting. - **Root cause:** Per-split digest construction reaches nonconsuming readers; ORC omits the metadata aggregate hook; child bitmap merging repeatedly copies its prefix; coarse targets replace count-derived bounds; HTTP uses coarse sizing despite an unstable file identity. - **Why existing tests missed it:** Existing tests cover results and isolated sizing helpers, but omit digest call counts, real ORC planning-reader retention, multi-child bitmap ownership, and integrated coarse-target range counts. - **Fix:** Share an immutable scanner predicate snapshot with a lazy key requested through a reader capability. Delegate ORC metadata aggregation to its existing implementation. Transfer completed bitmap owners under the coordinator mutex and assemble their union outside it after all matching EOFs. Apply count-derived bounds after coarse sizing and opt HTTP TVFs out of coarse planning. - **Other paths and branches:** Forward snapshots through Hudi, Paimon, Fluss and JNI wrappers. Cover synchronous SDK and manifest-cache Iceberg planning; keep streaming lazy. Keep native Paimon COUNT and existing unsupported aggregate fallbacks. Adapt the encryption fixture to the shared-metadata ownership change and initialize the scanner refresh-error fixture with a valid local state. ### Testing - Standard FE script: 400 tests passed across TVFScanNodeTest, PaimonScanPlanProviderTest and IcebergScanPlanProviderTest. The native Paimon and synchronous Iceberg range-count regressions fail on the original production code and pass after the fixes. - Standard FE build including Paimon and Iceberg completed with Checkstyle and plugin packaging. - Standard ASAN BE script: 257 tests in 10 suites passed, including all TableReaderTest and FileScannerV2Test cases, real ORC metadata aggregation, hybrid forwarding, the encryption fixture and binary column iterator coverage. - Changed C++ lines passed clang-tidy; clang-format v16 and build hygiene passed. - No live external-cluster regression or latency benchmark was run. ### Release note Preserve split-count safeguards and HTTP TVF parallelism during external scans, and avoid repeated runtime-filter hashing, ORC metadata reader creation and child condition-cache bitmap copying. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: No; restores the intended planning and cache behavior of the pending feature without adding settings. - Pre-submission status: Ready ### AI Disclosure - Tool: Codex (GPT-6) - Generated: Code, tests and this commit description. - Human review: Not stated.
### Background ORC physical children share immutable file metadata and carry exact stripe ids. SARG setup and timestamp stripe readers use that metadata to select stripes and retain file-global row positions. ### What problem does this PR solve? Issue Number: None Related PR: None PR Type: Bugfix Problem Summary: Each ORC child searched the whole file directory to rediscover its stripe ownership. Timestamp children also reopened all preceding stripes to recover their first row. Consume the planned stripe-id interval and cache row prefixes once in the shared file context. ### Details - **Change map:** 108 changed production lines across three files and 211 changed test lines in one file; no generated files or documentation. - **Frequency:** Eligible ORC scans with many physical children, especially tiny split targets and timestamp files requiring stripe-specific reader settings. - **Root cause:** The ORC factory dropped the physical id/id-end fields, and timestamp setup rebuilt file-global row prefixes for every child. - **Why existing tests missed it:** Tests checked ownership and output rows but did not bound directory visits across all children. - **Fix:** Forward both ids, enumerate only the inclusive child interval, and store each stripe's first row alongside its shared metadata. Operation counters exist only in BE_TEST builds. - **Other paths and branches:** Raw FE ranges retain byte-window selection. Coalesced children retain inclusive id-end semantics. Parquet already consumes exact physical ids. Query results, timestamp conversion, FE-BE protocol and on-disk formats do not change. ### Testing - Local ASAN BE build: DISABLE_BUILD_UI=ON ./build.sh --be -j8 passed. - ./run-be-ut.sh --run --filter='NewOrcReaderTest.*' -j8: all 196 tests passed under ASAN on macOS. - Negative controls restored the original algorithms while retaining test instrumentation. The SARG test failed with 64 visits for eight one-stripe children and 24 for coalesced children, versus eight after the fix. The timestamp test failed with 1024 ownership visits and 528 prefix visits for 32 children, versus 32 each after the fix. Both tests also cover coalesced intervals and output correctness. - clang-format 16, check-format.sh, check-build-hygiene.sh, git diff --check and run-clang-tidy.sh passed for the modified files. - No running-cluster regression or elapsed-time benchmark was performed. ### Release note ORC scans with many physical child splits avoid repeated full-directory ownership searches and timestamp row-prefix reconstruction during child initialization. ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Partially verified: local build and ORC unit tests passed; external end-to-end suites were not run. ### AI Disclosure - Tool: Codex - Generated: Code, tests and this description. - Human review: not stated
### Background Physical ORC children use independent readers for their owned stripe intervals. The serialized file tail shares layout metadata, but omits the stripe-statistics Metadata protobuf used by SARG pruning. ### What problem does this PR solve? Issue Number: None Related PR: apache/doris-thirdparty#435 PR Type: Bugfix Problem Summary: Every physical child repeats full-file Metadata parsing and stripe selection. Share the SDK's lazy immutable statistics handle and evaluate only each reader's owned stripe interval while retaining reader-local predicates and row-group pruning. ### Details - **Frequency:** Always for multi-child ORC scans with a supported SARG; reproduced with eight selected stripes. - **Root cause:** Each SDK reader loaded its own Metadata and getNeedReadStripes evaluated every stripe regardless of ownership. - **Why existing tests missed it:** The physical-child test bounded Doris directory visits but did not count SDK parsing or evaluation; external regression tests compare results only. - **Fix:** Carry the SDK statistics handle in the existing file context and use bounded stripe selection. Keep streams, metrics and predicate appliers private to each reader. The handle remains lazy when planning has no SARG. - **Other paths and branches:** Individual and coalesced children, stronger child predicates, default SDK options and standalone full-file calls are covered. No FE, wire or file-format changes; release-branch behavior is not audited in this followup. - **Behavior change:** Add OrcStripeStatisticsParses and OrcStripeStatisticsEvaluations profile counters. The latter measures explicit stripe-selection work, excluding normal RowReader PPD. ### Testing All 239 selected native ORC reader/predicate tests passed under ASAN, including all 196 Doris ORC reader tests. Coverage includes concurrent cold/warm SDK readers, independent predicates, row-group filtering, absolute positions and read/parse failure followed by retry through another reader after parent destruction. An earlier negative control restoring unshared/full-file SDK calls failed the real-factory test with nine parses and 72 evaluations; the fixed path records one parse and 16 evaluations. These are operation counts, not elapsed-time measurements. ASAN BE build, clang-format16, build hygiene, changed-line clang-tidy and git diff --check passed. Pin the merged SDK apache#435 commit 4ec25901b1266235f56ebe5c948277d6f3c2c616, whose complete source tree matches the tested SDK candidate. External end-to-end suites were not run. The affected pthread_once runtime combination is unavailable locally; its reported hang was not reproduced here. ### Release note None ### Check List (For Author) - Behavior changed: Yes (profile counters, see Details) - Does this need documentation: No additional user-facing setting; existing two-level splitting documentation plan remains in the PR. - Pre-submission status: Partially verified: external end-to-end suites and the affected pthread_once runtime combination were not exercised locally. ### AI Disclosure - Tool: Codex - Generated: Implementation, tests, verification and this description. - Human review: not stated - Prompts: Fix F-01 from the review of PR apache#70664 and update that PR.
### Background ORC range readers need file-wide footer metadata and absolute stripe row prefixes. Physical split scheduling creates independent readers for one immutable file. ### What problem does this PR solve? Issue Number: None Related PR: apache#70664 (review follow-up) PR Type: Performance Problem Summary: Each physical child currently reparses the serialized file tail and rebuilds all stripe row prefixes. Reuse a parsed immutable tail and initialize each RowReader from its byte interval, keeping streams, schema, predicates and metrics private. ### Details - **Change map:** 50 Doris production lines connect parsed-tail sharing and profile counters; 83 Doris test lines cover the real TableReader path. The SDK pin carries 225 production and 142 test lines, plus the two-line gitlink change. - **Why it was slow:** createReader reparses the FileTail protobuf and RowReaderImpl scans every stripe for each child. - **Optimization:** Share immutable Footer/PostScript and a memory-pool-backed absolute row-prefix buffer; binary-search byte and row boundaries. Reject an unordered stripe directory before binary search can misselect data. - **Results:** Static operation counts for an eight-stripe source plus eight individual children: tail parses 9 to 1, prefix entries built 72 to 8, RowReader range-accumulation visits 72 to 16. Binary-search comparisons are additional logarithmic work. These counts are source-derived; no elapsed-time or executed-test result is claimed. - **Applicability and cost:** Helps sibling readers of one immutable file, particularly uncoalesced stripes. The process-wide metadata cache still stores serialized tails; physical children use the parsed SDK handle. A parsed tail retains one prefix buffer in its source memory pool, which must outlive every shared handle. The new Reader API requires consumers to rebuild. Metadata-only opens now build the prefix when the tail is parsed; that path is not benchmarked. ### Testing clang-format 16.0.6 and git diff --check passed. Added a real no-SARG TableReader operation-count test for individual and coalesced children, plus SDK concurrency, seek, parent-destruction, serialized-tail, empty-file and malformed-directory tests. BE header hygiene also passed. Build and runtime tests have not run because the current worktree's installed dependencies point into a repository excluded by AGENTS.md; permission to read them is pending. ### Release note Reduce repeated ORC file-metadata initialization for physical split readers. ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Partially verified: compilation, runtime tests and elapsed-time performance remain unverified ### AI Disclosure - Tool: Codex - Generated: Code, tests and this description - Human review: Pending author review
### Background Physical ORC children reuse parsed file tails and absolute stripe row prefixes. Readers that only inspect metadata do not need row prefixes. ### What problem does this PR solve? Issue Number: None Related PR: apache#70664 (review follow-up) PR Type: Performance Problem Summary: Keep the shared prefix lazy so metadata-only opens avoid stripe traversal and allocation. Concurrent children build one immutable prefix with std::call_once, and an initialization failure remains retryable. ### Details - **Why it was slow:** Eager prefix construction adds file-wide work to metadata-only opens. Before parsed-tail sharing, each physical child also reparsed the tail and rebuilt every prefix. - **Optimization:** Build the source-pool-backed prefix only when a RowReader needs it and publish only after success. Keep streams, mutable schemas, predicates and metrics private. - **Results:** The eight-stripe real TableReader fixture passes at one tail parse, eight prefix entries and sixteen range visits. Temporarily restoring repeated setup makes the same test fail at nine, seventy-two and seventy-two, including the nonpositive-target case. SDK metadata-only opens build zero prefix entries; cold concurrent children build eight in total and warm children build none. These are measured operation counts, not elapsed-time claims. - **Applicability and cost:** Reuses work within one immutable file context. Byte/row range lookup uses logarithmic comparisons in addition to the counted accumulation visits. The source memory pool must outlive its shared handles. Serialized-tail callers remain supported; no session, wire or storage format change is introduced by this follow-up. ### Testing Local BE ASAN build through build.sh passed. run-be-ut.sh ran 255 related tests with zero failures or skips after restoring the candidate from the counterfactual. Coverage includes reader values/absolute row positions, read intent, physical splitting, shared statistics, cold concurrent initialization, warm reuse, parent destruction, seeks, empty files/ranges and repeated malformed-directory errors. Changed-line BE clang-tidy, clang-format 16, build-header hygiene and diff checks passed. SDK formatting passed separately; SDK code is excluded from the BE clang-tidy script by policy. The counterfactual is removed and all candidate source hashes match. No cluster regression or RELEASE timing benchmark was run. ### Release note Reduce repeated ORC physical-child metadata setup while preserving lazy metadata-only opens. ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Not ready: the SDK pin is local and must be published and merged upstream before submission ### AI Disclosure - Tool: Codex - Generated: Code, tests and this description - Human review: Pending author review
### Background ORC physical stripe readers reuse parsed immutable file metadata and lazy row prefixes. The SDK implementation has merged into the upstream orc branch. ### What problem does this PR solve? Issue Number: None Related PR: apache/doris-thirdparty#436 (merged SDK), apache#70664 (consumer) PR Type: Release/Dependency Problem Summary: Replace the local SDK pin with its upstream squash-merge commit so the Doris dependency is available from the maintained branch. ### Details - **Versions:** contrib/apache-orc 89c98c2 to upstream 00fa6d6. - **Upstream changes:** Both commits have identical complete source trees; apache#436 combines parsed file-tail sharing with lazy prefix initialization. - **Compatibility:** Source and test contents are unchanged. Native consumers build against the extended SDK interface already used by this branch. - **Rollback:** Restore the prior pin if the dependency update must be reverted. ### Testing Verified identical SDK tree hashes. The prior local ASAN build and 255 selected tests cover these exact SDK source bytes; they were not rerun for the hash-only pin update. ### Release note None ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Ready ### AI Disclosure - Tool: Codex - Generated: Dependency update and commit description - Human review: not stated
a1ad4bb to
0430b14
Compare
Local pipeline review — ✅ PASSschema: doris-repo-review/v1
status: PASS
pr: apache/doris#70664
commit: 0430b1447f22394e8e95c9b577bc2256ba20639c
base: 098d0bb62bbc82d9bd999d391efe1c20df89930b
reviewed_at: 2026-11-18T17:06+08:00
reviewer: morningman
model: gpt-6.1-sol
effort: xhigh
findings: {blocker: 0, major: 0, minor: 0, nit: 0}
rounds: 1
converged: trueNotes for maintainers
Reviewed locally with the |
### Background Source split progress uses DORIS_CHECK to assert positive child counts and batch-width samples. The macro supports streaming error messages through an intentionally unbraced failure branch. ### What problem does this PR solve? Issue Number: None Related PR: apache#70664 (source split progress) PR Type: Build/CI/Chore Problem Summary: Clang-tidy reports the macro expansion at three call sites as inconsistent if/else braces. Document and narrowly suppress those expansion-only diagnostics without changing the macro or any executable tokens. ### Details - **Affected:** Three DORIS_CHECK call sites in file_scan_context.h. - **Developer and release impact:** Explain the existing streaming macro contract and keep changed-line static checks actionable. Runtime behavior is unchanged. - **License and third-party:** None. - **Rollback:** Remove the NOLINT comments if the checker learns to ignore this macro expansion. ### Testing Verified that removing line comments makes the old and new headers identical. Project clang-format16 and header hygiene passed. The current ASAN BE+FE build predates this comment-only cleanup; focused BE/FE tests and final static checks are in progress. ### Release note None ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Partially verified: final static checking is in progress ### AI Disclosure - Tool: Codex - Generated: Narrow style-suppression comments and commit description - Human review: not stated
Local pipeline review — ✅ PASSschema: doris-repo-review/v1
status: PASS
pr: apache/doris#70664
commit: feec2880ae7e16646fe6b759e632ef302aac5177
base: 098d0bb62bbc82d9bd999d391efe1c20df89930b
reviewed_at: 2026-11-18T17:44+08:00
reviewer: morningman
model: gpt-6.1-sol
effort: xhigh
findings: {blocker: 0, major: 0, minor: 0, nit: 0}
rounds: 1
converged: trueNotes for maintainers
Reviewed locally with the |
|
run buildall |
### Background Iceberg manifest planning tests verify query-wide read budgets and cancellation before scheduler acquisition. FEUT runs these tests through both materialized and streaming scan planning. ### What problem does this PR solve? Issue Number: None Related PR: apache#70664 (two-level external file splitting) PR Type: Test Problem Summary: Manifest planning gained a coarse split size argument, but its existing reflection-based tests still use the previous signatures. Update the method lookup and invocation arguments so the tests reach their original cancellation and shared-budget assertions. ### Details - **Failure:** Full FEUT reports four NoSuchMethodException errors and four wrong-number-of-arguments assertion failures in IcebergManifestReadContextTest. - **Cause:** The planFileScanTask and streamingFileScanTasks test calls omit the new long coarseSplitSize parameter. Reflection defers the mismatch until test execution. - **Stability evidence:** The unchanged 14-case test class passes locally after correcting the signatures. All 389 selected tests in 10 related Iceberg classes pass with zero failures, errors or skips. The fixture uses 0L to preserve legacy split sizing; assertions, synchronization and timeouts are unchanged. ### Testing - Local run-fe-ut.sh --run 'IcebergManifest*Test,IcebergScanPlanProvider*Test,IcebergScanRangeTest,IcebergStatementScopeTest': 389 tests passed, including all 14 IcebergManifestReadContextTest cases. - Full FEUT on the updated head was not rerun locally. ### Release note None ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Ready ### AI Disclosure - Tool: Codex - Generated: Test signature updates, verification and this description. - Human review: not stated
|
run buildall |
Local pipeline review — ✅ PASSschema: doris-repo-review/v1
status: PASS
pr: apache/doris#70664
commit: 2998c21b6ba46b1e3501457aed8a48bf5dabef25
base: 098d0bb62bbc82d9bd999d391efe1c20df89930b
reviewed_at: 2026-11-18T23:23+08:00
reviewer: morningman
model: gpt-6.1-sol
effort: xhigh
findings: {blocker: 0, major: 0, minor: 0, nit: 0}
rounds: 1
converged: trueNotes for maintainers
Reviewed locally with the |
Background
FE assigns external-file byte ranges to BE scanners. Native Parquet and ORC readers select each range's row groups or stripes, whose physical layout determines the safe unit of parallel work.
What problem does this PR solve?
Issue Number: None
Related PR: #66802 (branch-4.2 source), #68143 (Iceberg missing-file policy in split planning), apache/doris-thirdparty#435 (merged ORC statistics sharing), apache/doris-thirdparty#436 (merged parsed-tail sharing)
PR Type: Enhancement
Problem Summary:
This backport adds FE coarse ranges and BE row-group/stripe tasks for eligible native scans, preserving runtime filters, source completion and condition-cache publication. Eligible scans use dedicated FE/BE size targets after upgrading.
Original author: @Gabriel39
Details
file_split_size_on_fe=536870912andfile_split_size_on_be=67108864, both forwarded and changeable through SET. Nonpositive values keep legacy FE planning; they do not disable BE refinement.file_split_size(still the fallback). Load keeps FE boundaries. Optional Thrift field 17 matches 4.1/4.2; upgrade BE first because older BE cannot recover coarse-range parallelism. Connector API 35.0 requires rebuilt plugins. Results and delete semantics are preserved.Testing
The ASAN BE+FE build passed at
0430b14. Laterfeec288adds six comment lines, and the FEUT follow-up only updates existing Iceberg test reflection signatures and arguments; non-comment production source is unchanged.run-fe-ut.sh: 389 tests in 10 Iceberg manifest, planning, range and statement-scope classes passed, with zero failures, errors or skips. All 14IcebergManifestReadContextTestcases pass after adding the missing coarse split argument; the original cancellation and shared-budget assertions remain unchanged.run-be-ut.sh: 482 tests in 10 suites passed, covering Fluss, ORC/SDK, scanners and TableReader; zero failures, errors or skips. The driver began at0430b14and covered identical non-comment source.run-fe-ut.shatfeec288: 448 tests in 75 classes passed, including the complete 37-class SPI suite, both surface baselines/API35 and engine planning/path-partition tests; zero failures, errors or skips.Notes for Reviewers
Three narrow
NOLINTannotations coverDORIS_CHECK's intentional streaming-macroelse; the assertions are unchanged.API 35.0 extends master's existing 34.0 surface, preserving its path-partition-key API alongside the split-planning additions. The planning missing-file classifier follows normal Iceberg reads.
Children retain exact stripe-ID intervals and share parsed immutable file tails and lazy row prefixes. Merged apache/doris-thirdparty#435 shares lazy Metadata and evaluates owned stripes; streams, metrics and SARG appliers remain private. Metadata-only opens and no-SARG planning remain lazy.
OrcFileTailParses,OrcStripeDirectoryEntriesandOrcRowReaderStripeVisitstrack SDK tail/prefix/range setup.OrcStripeStatisticsParsesandOrcStripeStatisticsEvaluationsexpose parse attempts and explicit selection work, excluding RowReader PPD.The ORC pin moves from
5962bb0to merged apache/doris-thirdparty#436 commit00fa6d6, whose complete source tree equals the reviewed and tested SDK89c98c2. The range also includes shared statistics #435, canonical seek fix #431 (the old pin already has equivalent Reader.cc behavior), and Maven output ignore #434: eight files, +644/-100 lines. Shared loaders publish only after success and permit failure retries.Iceberg SDK offsets can override size targets. Small-range fixtures disable coarse FE planning; the new regression compares parallel scans with a single-scanner reference without handwritten output.
Release note
Native Parquet/ORC scans gain FE coarse splitting and BE physical refinement through
file_split_size_on_fe(512 MiB) andfile_split_size_on_be(64 MiB), replacingfile_split_size. Nonpositive sizes retain legacy FE sizing; BE can still refine. Upgrade BE first and rebuild API 35.0 plugins.Check List (For Author)
AI Disclosure