From 1c474c3fa17389d360feda03df5469b96eeadbe6 Mon Sep 17 00:00:00 2001 From: morningman Date: Tue, 17 Nov 2026 13:14:07 +0800 Subject: [PATCH 01/10] [improvement](external) Refine external file splits in BE ### Background Doris plans external-file byte ranges in FE and assigns them to BE scanners. Native Parquet and ORC readers then select the row groups or stripes belonging to each range. FE range sizing determines the initial work distribution, while the physical file layout determines what a reader can scan independently. ### What problem does this PR solve? Issue Number: None Related PR: #66802 (branch-4.2 source), #68143 (Iceberg missing-file policy in split planning) PR Type: Enhancement Problem Summary: This ports the two-level external-file splitting from branch-4.2 #66802 to the current scanner and connector architecture. FE selects coarse ranges for eligible native scans, and BE refines them into row-group or stripe tasks that scanners in the same instance can consume. The port also handles source completion, shared immutable metadata, runtime filters, condition-cache publication, and the plugin request paths. Eligible scans use dedicated FE and BE size targets after upgrading. Original author: @Gabriel39 ### Details - **Change map:** 2,247 changed production lines across 49 files; 2,562 test lines across 25 files; two build-metadata lines in one file; no generated outputs or documentation. BE adds physical split scheduling, exact ownership and parent-scoped metadata/cache state. FE and SPI propagate coarse sizing through synchronous, partition-batch and streaming planning; connectors retain their source-specific exclusions. Tests port the original BE cases and cover the new engine/plugin seams. - **Before and after:** FE previously used each source's existing split policy. Eligible Parquet/ORC scans now use a 512 MiB FE target; BE groups physical children toward 64 MiB. For a 96 MiB Iceberg fixture without explicit split offsets, tests verify three ranges with a 32 MiB coarse hint and two with 64 MiB, including streaming. Files with Iceberg deletes, logical/JNI Paimon splits, metadata COUNT and serial scans retain their applicable fallback. BE may refine eligible ranges even when FE omits the target; a positive target controls coalescing only. - **Configs and defaults:** Add session variables `file_split_size_on_fe=536870912` and `file_split_size_on_be=67108864`, both forwarded and changeable through SET. Nonpositive values keep legacy FE planning; they do not disable BE refinement. The deprecated `enable_file_scanner_v2` compatibility variable remains ineffective. - **Impact on existing users:** Eligible ranges use the dedicated targets instead of `file_split_size`; that variable remains the fallback policy. Results and table-format delete semantics are preserved. Load retains FE range boundaries. The optional Thrift field uses the same id 17 as 4.1/4.2. Older BE ignores it and cannot recover coarse-range parallelism, so upgrade BE first. Connector API becomes 35.0; plugins must be rebuilt together with FE. ### Testing - Standard FE test script: 790 tests in 52 classes passed after rebasing onto current master, including the 533 affected engine/connector tests, the complete 163-test SPI suite and 94 predicate/converter tests covering the newly merged changes. Both recorded SPI surface contracts passed. - Five targeted temporary mutants lost either synchronous/batch request propagation or one connector's reuse-key input. Every control passed and every mutant failed its intended assertion. - Before the final rebase, all 27 modified BE translation units passed syntax checks against the generated compilation database; a deliberate failing probe confirmed detection. BE and protocol sources are unchanged since. - Five changed regression scripts passed Groovy compilation before the final rebase and are unchanged since. Actual BE unit tests and external end-to-end suites have not run: the installed third-party package lacks the BSON/MongoDB 2.0 include directories required to generate the BE build. No running-cluster evidence is claimed. ### Notes for Reviewers Start with source-split publication and completion, then physical ownership and cache finalization. Children share only their parent's immutable metadata and synchronized hints; decoding, predicates and progress remain reader-local. Keep the refinement-planning missing-file classifier aligned with normal Iceberg reads. Iceberg preserves SDK split offsets, so an explicit offset list can override the requested size. Existing tests that deliberately force tiny FE ranges now disable coarse FE planning to retain their coverage. The new regression compares parallel configurations with a single-scanner reference and requires no handwritten expected output. ### Release note Native Parquet and ORC external scans gain FE coarse splitting and BE physical refinement, controlled by `file_split_size_on_fe` (512 MiB) and `file_split_size_on_be` (64 MiB). Eligible scans use these targets instead of `file_split_size`. Set a nonpositive dedicated size to retain legacy FE sizing; BE can still refine eligible ranges without a coalescing target. Upgrade BE before FE and rebuild connector plugins for API 35.0. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: Yes; update the external scan split policy and both session-variable entries in apache/doris-website development documentation. - Pre-submission status: Partially verified: BE unit execution and external end-to-end regression are blocked by missing third-party BSON/MongoDB 2.0 dependencies. ### AI Disclosure - Tool: Codex (GPT-6) - Generated: Port adaptations, tests, evidence collection, and this description. - Human review: not stated - Prompts: Start the next item from the batch handoff. Follow the settled decision to retain branch-4.2 BE refinement without an FE authorization gate, and use engine-coordinated FE/SPI planning in one PR. --- .../block/adaptive_block_size_predictor.cpp | 7 + .../block/adaptive_block_size_predictor.h | 3 + be/src/exec/operator/file_scan_operator.cpp | 62 +- be/src/exec/operator/file_scan_operator.h | 15 + be/src/exec/scan/file_scanner_v2.cpp | 229 ++++- be/src/exec/scan/file_scanner_v2.h | 49 ++ be/src/exec/scan/split_source_connector.cpp | 110 ++- be/src/exec/scan/split_source_connector.h | 24 + be/src/format_v2/column_mapper.cpp | 21 + be/src/format_v2/file_reader.cpp | 5 +- be/src/format_v2/file_reader.h | 40 + be/src/format_v2/file_scan_context.h | 117 +++ be/src/format_v2/jni/jni_table_reader.cpp | 7 +- be/src/format_v2/jni/jni_table_reader.h | 4 +- be/src/format_v2/orc/orc_reader.cpp | 252 ++++-- be/src/format_v2/orc/orc_reader.h | 23 +- .../parquet/parquet_file_context.cpp | 106 ++- .../format_v2/parquet/parquet_file_context.h | 31 +- be/src/format_v2/parquet/parquet_reader.cpp | 141 ++- be/src/format_v2/parquet/parquet_reader.h | 12 +- be/src/format_v2/parquet/parquet_scan.cpp | 116 ++- be/src/format_v2/parquet/parquet_scan.h | 38 +- .../format_v2/table/fluss_hybrid_reader.cpp | 10 +- be/src/format_v2/table/fluss_hybrid_reader.h | 4 +- .../table/fluss_union_lake_reader.cpp | 10 +- .../format_v2/table/fluss_union_lake_reader.h | 4 +- be/src/format_v2/table/hudi_reader.cpp | 21 +- be/src/format_v2/table/hudi_reader.h | 6 +- be/src/format_v2/table/load_reader.cpp | 2 +- be/src/format_v2/table/load_reader.h | 4 +- be/src/format_v2/table/paimon_reader.cpp | 17 +- be/src/format_v2/table/paimon_reader.h | 6 +- be/src/format_v2/table_reader.cpp | 369 +++++++- be/src/format_v2/table_reader.h | 40 +- be/src/storage/segment/condition_cache.h | 16 + be/test/exec/scan/file_scanner_v2_test.cpp | 520 ++++++++++- .../format_v2/jni/jni_table_reader_test.cpp | 14 + be/test/format_v2/orc/orc_reader_test.cpp | 123 ++- .../parquet/parquet_reader_control_test.cpp | 47 + .../format_v2/parquet/parquet_reader_test.cpp | 416 ++++++++- .../format_v2/parquet/parquet_scan_test.cpp | 66 ++ .../table/fluss_union_lake_reader_test.cpp | 7 +- be/test/format_v2/table/hudi_reader_test.cpp | 7 +- .../format_v2/table/paimon_reader_test.cpp | 7 +- be/test/format_v2/table_reader_test.cpp | 818 ++++++++++++++++++ .../connector/hive/HiveScanPlanProvider.java | 41 +- .../doris/connector/hive/HiveScanRange.java | 5 + .../connector/hive/HiveScanBatchModeTest.java | 55 ++ .../doris/connector/hudi/HudiScanRange.java | 5 + .../iceberg/IcebergScanPlanProvider.java | 90 +- .../connector/iceberg/IcebergScanRange.java | 5 + .../iceberg/IcebergScanPlanProviderTest.java | 60 ++ .../iceberg/IcebergScanRangeTest.java | 15 + .../paimon/PaimonScanPlanProvider.java | 43 +- .../connector/paimon/PaimonScanRange.java | 5 + .../paimon/PaimonScanPlanProviderTest.java | 21 +- .../spi/scan/ConnectorScanRange.java | 9 + .../spi/scan/ConnectorScanRequest.java | 23 +- .../spi/ConnectorPluginSurfaceTest.java | 4 +- ...onnectorScanPlanProviderBatchScanTest.java | 12 +- .../resources/connector-plugin-surface.txt | 3 + fe/fe-connector/pom.xml | 2 +- .../datasource/scan/FileQueryScanNode.java | 53 ++ .../datasource/scan/PluginDrivenScanNode.java | 7 + .../datasource/tvf/source/TVFScanNode.java | 7 +- .../org/apache/doris/qe/SessionVariable.java | 29 + .../scan/FileQueryScanNodeTest.java | 102 +++ .../scan/PluginDrivenScanNodeBeSplitTest.java | 176 ++++ .../tvf/source/TVFScanNodeTest.java | 19 + gensrc/thrift/PlanNodes.thrift | 2 + .../test_parquet_join_runtime_filter.groovy | 1 + .../hive/test_two_level_file_splits.groovy | 63 ++ .../test_iceberg_missing_data_file.groovy | 2 + .../paimon/test_paimon_deletion_vector.groovy | 2 + .../test_paimon_file_metadata_columns.groovy | 2 + 75 files changed, 4544 insertions(+), 265 deletions(-) create mode 100644 be/src/format_v2/file_scan_context.h create mode 100644 fe/fe-core/src/test/java/org/apache/doris/datasource/scan/PluginDrivenScanNodeBeSplitTest.java create mode 100644 regression-test/suites/external_table_p0/hive/test_two_level_file_splits.groovy diff --git a/be/src/core/block/adaptive_block_size_predictor.cpp b/be/src/core/block/adaptive_block_size_predictor.cpp index 3d7fb6e5a8b70a..1a7456c1e71963 100644 --- a/be/src/core/block/adaptive_block_size_predictor.cpp +++ b/be/src/core/block/adaptive_block_size_predictor.cpp @@ -20,6 +20,7 @@ #include #include +#include "common/check.h" #include "core/block/block.h" namespace doris { @@ -46,6 +47,12 @@ void AdaptiveBlockSizePredictor::update(const Block& block) { } } +void AdaptiveBlockSizePredictor::seed_history(double bytes_per_row) { + DORIS_CHECK(bytes_per_row > 0); + _bytes_per_row = bytes_per_row; + _has_history = true; +} + size_t AdaptiveBlockSizePredictor::predict_next_rows() const { if (_block_size_bytes == 0) { return _block_size_rows; diff --git a/be/src/core/block/adaptive_block_size_predictor.h b/be/src/core/block/adaptive_block_size_predictor.h index 4ee85117517392..d156088202ef9d 100644 --- a/be/src/core/block/adaptive_block_size_predictor.h +++ b/be/src/core/block/adaptive_block_size_predictor.h @@ -58,6 +58,9 @@ class AdaptiveBlockSizePredictor { bool has_history() const { return _has_history; } + // Restore the measured row width shared by physical children of one source split. + void seed_history(double bytes_per_row); + private: // EWMA weight for historical estimate (0.9) and current sample (0.1). static constexpr double kAlpha = 0.9; diff --git a/be/src/exec/operator/file_scan_operator.cpp b/be/src/exec/operator/file_scan_operator.cpp index 8aec598787016a..4ef9d183ec7e48 100644 --- a/be/src/exec/operator/file_scan_operator.cpp +++ b/be/src/exec/operator/file_scan_operator.cpp @@ -117,6 +117,35 @@ ScannerScheduler* FileScanLocalState::scan_scheduler(RuntimeState* state) const return state->get_query_ctx()->get_remote_scan_scheduler(); } +bool FileScanLocalState::_can_generate_physical_splits(bool is_load, + const TFileScanRangeParams& scan_params, + const TFileRangeDesc& range) { + if (is_load || (scan_params.__isset.table_format_params && + scan_params.table_format_params.table_format_type == "transactional_hive")) { + return false; + } + const auto format = range.__isset.format_type ? range.format_type : scan_params.format_type; + if (format == TFileFormatType::FORMAT_PARQUET || format == TFileFormatType::FORMAT_ORC) { + // Keep scanner creation aligned with the downstream refinement guard. Otherwise an + // Iceberg delete split creates idle scanners even though it can never publish children. + return FileScannerV2::can_refine_source_split(range); + } + if (format != TFileFormatType::FORMAT_JNI || !range.__isset.table_format_params || + range.table_format_params.table_format_type != "paimon" || + !range.table_format_params.__isset.paimon_params) { + return false; + } + const auto& paimon = range.table_format_params.paimon_params; + return paimon.__isset.file_format && + (paimon.file_format == "parquet" || paimon.file_format == "orc") && + !paimon.__isset.paimon_split; +} + +int FileScanLocalState::_adjust_scanner_count(int requested, int initial_ranges, + bool can_generate_physical_splits) { + return can_generate_physical_splits ? requested : std::min(requested, initial_ranges); +} + Status FileScanLocalState::_init_scanners(std::list* scanners) { if (_split_source->num_scan_ranges() == 0) { _eos = true; @@ -196,9 +225,36 @@ void FileScanLocalState::set_scan_ranges(RuntimeState* state, if (_split_source == nullptr) { _split_source = std::make_shared(scan_ranges, _max_scanners); } - // currently the total number of splits in the bach split mode cannot be accurately obtained, - // so we don't do it in the batch split mode. - _max_scanners = std::min(_max_scanners, _split_source->num_scan_ranges()); + // One FE columnar split can publish many format-local children after metadata planning. + // Keep the requested scanner concurrency so those children do not run serially. + bool can_generate_physical_splits = false; + const TFileScanRangeParams* common_params = nullptr; + if (state->get_query_ctx() != nullptr && + state->get_query_ctx()->file_scan_range_params_map.contains(parent_id())) { + common_params = &state->get_query_ctx()->file_scan_range_params_map[parent_id()]; + } + for (const auto& scan_range_params : scan_ranges) { + const auto& file_scan_range = + scan_range_params.scan_range.ext_scan_range.file_scan_range; + const auto* params = + file_scan_range.__isset.params ? &file_scan_range.params : common_params; + if (params == nullptr) { + continue; + } + const bool is_load = + state->desc_tbl().get_tuple_descriptor(params->src_tuple_id) != nullptr; + can_generate_physical_splits = + std::ranges::any_of(file_scan_range.ranges, [&](const auto& range) { + return _can_generate_physical_splits(is_load, *params, range); + }); + if (can_generate_physical_splits) { + break; + } + } + // Currently the total number of remote splits cannot be accurately obtained, so batch + // mode already skips this cap. + _max_scanners = _adjust_scanner_count(_max_scanners, _split_source->num_scan_ranges(), + can_generate_physical_splits); } if (!scan_ranges.empty() && diff --git a/be/src/exec/operator/file_scan_operator.h b/be/src/exec/operator/file_scan_operator.h index 8c5ebda44f47b6..7431bcc5d28015 100644 --- a/be/src/exec/operator/file_scan_operator.h +++ b/be/src/exec/operator/file_scan_operator.h @@ -57,6 +57,17 @@ class FileScanLocalState final : public ScanLocalState { int max_scanners_concurrency(RuntimeState* state) const override; int min_scanners_concurrency(RuntimeState* state) const override; ScannerScheduler* scan_scheduler(RuntimeState* state) const override; +#ifdef BE_TEST + static bool TEST_can_generate_physical_splits(bool is_load, + const TFileScanRangeParams& scan_params, + const TFileRangeDesc& range) { + return _can_generate_physical_splits(is_load, scan_params, range); + } + static int TEST_adjust_scanner_count(int requested, int initial_ranges, + bool can_generate_physical_splits) { + return _adjust_scanner_count(requested, initial_ranges, can_generate_physical_splits); + } +#endif private: friend class FileScannerV2; @@ -67,6 +78,10 @@ class FileScanLocalState final : public ScanLocalState { return PushDownType::PARTIAL_ACCEPTABLE; } bool _push_down_topn(const RuntimePredicate& predicate) override; + static bool _can_generate_physical_splits(bool is_load, const TFileScanRangeParams& scan_params, + const TFileRangeDesc& range); + static int _adjust_scanner_count(int requested, int initial_ranges, + bool can_generate_physical_splits); PushDownType _should_push_down_is_null_predicate(VectorizedFnCall* fn_call) const override { return fn_call->fn().name.function_name == "is_null_pred" || diff --git a/be/src/exec/scan/file_scanner_v2.cpp b/be/src/exec/scan/file_scanner_v2.cpp index 86d261b3397cb0..cc11b281ba263e 100644 --- a/be/src/exec/scan/file_scanner_v2.cpp +++ b/be/src/exec/scan/file_scanner_v2.cpp @@ -193,6 +193,27 @@ Status rewrite_slot_refs_to_global_index( } // namespace +int64_t FileScannerV2::_cumulative_profile_delta(int64_t current, int64_t* reported) { + DORIS_CHECK(reported != nullptr); + DORIS_CHECK(current >= *reported); + const int64_t delta = current - *reported; + *reported = current; + return delta; +} + +FileScannerV2::FileReaderProfileDeltas FileScannerV2::_collect_file_reader_profile_deltas( + const io::FileReaderStats& stats, int64_t* reported_bytes, int64_t* reported_calls, + int64_t* reported_time) { + return { + .read_bytes = + _cumulative_profile_delta(cast_set(stats.read_bytes), reported_bytes), + .read_calls = + _cumulative_profile_delta(cast_set(stats.read_calls), reported_calls), + .read_time_ns = + _cumulative_profile_delta(cast_set(stats.read_time_ns), reported_time), + }; +} + #ifdef BE_TEST FileScannerV2::FileScannerV2(RuntimeState* state, RuntimeProfile* profile, std::unique_ptr table_reader) @@ -252,7 +273,8 @@ FileScannerV2::FileScannerV2(RuntimeState* state, FileScanLocalState* local_stat : Scanner(state, local_state, limit, profile), _split_source(std::move(split_source)), _kv_cache(kv_cache), - _parquet_profile_cache(local_state->_parquet_profile_cache.get()) { + _parquet_profile_cache(local_state->_parquet_profile_cache.get()), + _constructed_scanners(local_state->_max_scanners) { (void)colname_to_slot_id; if (state->get_query_ctx() != nullptr && state->get_query_ctx()->file_scan_range_params_map.count(local_state->parent_id()) > 0) { @@ -328,6 +350,17 @@ Status FileScannerV2::_open_impl(RuntimeState* state) { Status FileScannerV2::_open_wrapped(RuntimeState* state) { RETURN_IF_CANCELLED(state); RETURN_IF_ERROR(Scanner::_open_impl(state)); + // The first split may open its physical reader eagerly while refining physical children. + // Synchronize ready filters first so COUNT(*) never treats a future RF carrier as a disposable + // placeholder. + const auto rf_status = try_append_late_arrival_runtime_filter(); + if (!rf_status.ok()) { + // Late RF pushdown is optional and the scheduler already treats refresh failures as + // best-effort. Keep the original conjunct snapshot instead of failing scanner open after + // a consumer has transitioned the ready filter to APPLIED. + LOG(WARNING) << "Failed to append late arrival runtime filter before file scanner open: " + << rf_status.to_string(); + } RETURN_IF_ERROR(_get_next_scan_range(&_first_scan_range)); if (_first_scan_range) { if (_is_load) { @@ -340,6 +373,11 @@ Status FileScannerV2::_open_wrapped(RuntimeState* state) { DORIS_CHECK(_table_reader != nullptr); RETURN_IF_ERROR(_init_expr_ctxes()); RETURN_IF_ERROR(_init_table_reader(_current_range)); + // Refine the first source split before yielding the scanner worker. Other scanners may be + // waiting for its physical children, so deferring publication until a later get_block() + // turn could let those waiters occupy the scan thread pool ahead of the producer. + bool eos = false; + RETURN_IF_ERROR(_prepare_next_split(&eos)); } return Status::OK(); } @@ -347,9 +385,39 @@ Status FileScannerV2::_open_wrapped(RuntimeState* state) { Status FileScannerV2::_get_next_scan_range(bool* has_next) { SCOPED_TIMER(_get_next_range_timer); DORIS_CHECK(has_next != nullptr); - RETURN_IF_ERROR(_split_source->get_next(has_next, &_current_range, &_should_stop)); + RETURN_IF_ERROR(_split_source->get_next_split(has_next, &_current_split, &_should_stop)); if (*has_next) { + _current_range = _current_split.materialize_range(); RETURN_IF_ERROR(validate_file_scan_support(*_params, &_current_range)); + } else { + _current_split = {}; + // Final counter publication happens after EOS. Keep the last materialized range so its + // range-only file type still controls storage and load accounting during that publication. + } + return Status::OK(); +} + +Status FileScannerV2::_retire_current_source_split(std::vector generated_splits) { + if (!_current_split.is_source_split) { + return Status::OK(); + } + const bool transferred_progress = !generated_splits.empty(); + RETURN_IF_ERROR( + _split_source->finish_source_split(_current_split, std::move(generated_splits))); + _current_split.is_source_split = false; + if (transferred_progress) { + _current_split.source_progress.reset(); + } + return Status::OK(); +} + +Status FileScannerV2::_complete_current_split() { + RETURN_IF_ERROR(_retire_current_source_split()); + if (_current_split.source_progress != nullptr) { + if (_current_split.source_progress->complete_one()) { + _state->update_num_finished_scan_range(1); + } + _current_split.source_progress.reset(); } return Status::OK(); } @@ -376,7 +444,9 @@ Status FileScannerV2::_get_block_wrapped(RuntimeState* state, Block* block, bool if (_table_reader_rf_num != _applied_rf_num) { VExprContextSPtrs refreshed_conjuncts; RETURN_IF_ERROR(_build_table_conjuncts(&refreshed_conjuncts)); - RETURN_IF_ERROR(_table_reader->refresh_conjuncts(std::move(refreshed_conjuncts))); + RETURN_IF_ERROR(_table_reader->refresh_conjuncts(std::move(refreshed_conjuncts), + _current_condition_cache_digest(), + _applied_rf_num == _total_rf_num)); _table_reader_rf_num = _applied_rf_num; } if (_should_run_adaptive_batch_size()) { @@ -391,7 +461,7 @@ Status FileScannerV2::_get_block_wrapped(RuntimeState* state, Block* block, bool COUNTER_UPDATE(_not_found_file_counter, 1); LOG(INFO) << "Skip missing file: query_id=" << print_id(_state->query_id()) << ", path=" << _current_range.path << ", reason=NOT_FOUND"; - _state->update_num_finished_scan_range(1); + RETURN_IF_ERROR(_complete_current_split()); _has_prepared_split = false; block->clear_column_data(cast_set(_projected_columns.size())); *eof = false; @@ -404,7 +474,7 @@ Status FileScannerV2::_get_block_wrapped(RuntimeState* state, Block* block, bool // this range, discard partial reader state, and let the loop fetch the next split. RETURN_IF_ERROR(_table_reader->abort_split()); COUNTER_UPDATE(_empty_file_counter, 1); - _state->update_num_finished_scan_range(1); + RETURN_IF_ERROR(_complete_current_split()); _has_prepared_split = false; block->clear_column_data(cast_set(_projected_columns.size())); *eof = false; @@ -413,7 +483,7 @@ Status FileScannerV2::_get_block_wrapped(RuntimeState* state, Block* block, bool RETURN_IF_ERROR(status); } if (*eof) { - _state->update_num_finished_scan_range(1); + RETURN_IF_ERROR(_complete_current_split()); _has_prepared_split = false; *eof = false; continue; @@ -454,6 +524,7 @@ Status FileScannerV2::_prepare_next_split(bool* eos) { } _first_scan_range = false; if (!has_next || _should_stop) { + RETURN_IF_ERROR(_complete_current_split()); *eos = true; return Status::OK(); } @@ -481,7 +552,7 @@ Status FileScannerV2::_prepare_next_split(bool* eos) { COUNTER_UPDATE(_not_found_file_counter, 1); LOG(INFO) << "Skip missing file: query_id=" << print_id(_state->query_id()) << ", path=" << _current_range.path << ", reason=NOT_FOUND"; - _state->update_num_finished_scan_range(1); + RETURN_IF_ERROR(_complete_current_split()); continue; } if (_should_skip_empty(status, _should_stop || _io_ctx->should_stop)) { @@ -491,15 +562,55 @@ Status FileScannerV2::_prepare_next_split(bool* eos) { // advance exactly one scan range and preserve later files in the same scan. RETURN_IF_ERROR(_table_reader->abort_split()); COUNTER_UPDATE(_empty_file_counter, 1); - _state->update_num_finished_scan_range(1); + RETURN_IF_ERROR(_complete_current_split()); continue; } RETURN_IF_ERROR(status); if (_table_reader->current_split_pruned()) { - _state->update_num_finished_scan_range(1); + RETURN_IF_ERROR(_complete_current_split()); continue; } - COUNTER_UPDATE(_file_counter, 1); + _update_file_counter(_file_counter, _current_split); + if (!_is_load && _current_split.is_source_split && + _should_refine_source_split(_current_range, _constructed_scanners)) { + std::vector generated_splits; + bool was_split = false; + const auto split_status = _table_reader->build_physical_splits( + _current_split, &generated_splits, &was_split); + const auto ignored_split_status = _classify_ignored_split_status( + split_status, + can_ignore_not_found_file(_current_range, + config::ignore_not_found_file_in_external_table), + _should_stop || _io_ctx->should_stop); + if (ignored_split_status == IgnoredSplitStatus::NOT_FOUND) { + RETURN_IF_ERROR(_table_reader->abort_split()); + COUNTER_UPDATE(_not_found_file_counter, 1); + RETURN_IF_ERROR(_complete_current_split()); + continue; + } + if (ignored_split_status == IgnoredSplitStatus::EMPTY) { + RETURN_IF_ERROR(_table_reader->abort_split()); + COUNTER_UPDATE(_empty_file_counter, 1); + RETURN_IF_ERROR(_complete_current_split()); + continue; + } + RETURN_IF_ERROR(split_status); + if (was_split) { + RETURN_IF_ERROR(_table_reader->abort_split()); + const bool has_children = !generated_splits.empty(); + RETURN_IF_ERROR(_retire_current_source_split(std::move(generated_splits))); + if (!has_children) { + RETURN_IF_ERROR(_complete_current_split()); + } + continue; + } + } + if (_current_split.is_source_split) { + // Non-refinable and metadata-only source splits are already fully prepared and cannot + // publish children. Release the source reservation before yielding the scanner worker + // so waiters never depend on a later get_block() turn to observe completion. + RETURN_IF_ERROR(_retire_current_source_split()); + } _has_prepared_split = true; _table_reader_rf_num = _applied_rf_num; *eos = false; @@ -707,6 +818,10 @@ Status FileScannerV2::_prepare_table_reader_split(const TFileRangeDesc& range, // keeps safe partition pruning enabled independently of the legacy session gate. RETURN_IF_ERROR(_build_table_conjuncts(&partition_prune_conjuncts)); } + const auto& source_identity = _current_split.source_identity_range(); + const auto source_start = + source_identity.__isset.start_offset ? source_identity.start_offset : 0; + const auto source_size = source_identity.__isset.size ? source_identity.size : -1; RETURN_IF_ERROR(_table_reader->prepare_split({ .partition_values = std::move(partition_values), .conjuncts = std::move(conjuncts), @@ -718,7 +833,13 @@ Status FileScannerV2::_prepare_table_reader_split(const TFileRangeDesc& range, .cache = _kv_cache, .current_range = range, .current_split_format = current_split_format, - .global_rowid_context = _create_global_rowid_context(range), + .file_context = _current_split.file_context, + .condition_cache_source_range = std::pair(source_start, source_size), + .condition_cache_split_context = _current_split.condition_cache_split_context, + .format_split_id = _current_split.format_split_id, + .format_split_id_end = _current_split.format_split_id_end, + .global_rowid_context = + _create_global_rowid_context(_current_split.source_identity_range()), .row_ids = std::nullopt, })); return Status::OK(); @@ -736,6 +857,42 @@ bool FileScannerV2::_should_skip_empty(const Status& status, bool stopped) { return !stopped && status.is(); } +FileScannerV2::IgnoredSplitStatus FileScannerV2::_classify_ignored_split_status( + const Status& status, bool ignore_not_found, bool stopped) { + if (_should_skip_not_found(status, ignore_not_found)) { + return IgnoredSplitStatus::NOT_FOUND; + } + if (_should_skip_empty(status, stopped)) { + return IgnoredSplitStatus::EMPTY; + } + return IgnoredSplitStatus::NONE; +} + +bool FileScannerV2::can_refine_source_split(const TFileRangeDesc& range) { + if (!range.__isset.table_format_params || !range.table_format_params.__isset.iceberg_params) { + return true; + } + const auto& iceberg = range.table_format_params.iceberg_params; + // Iceberg delete readers build split-local delete state. Refining the data file would rebuild + // that state for every physical child, so keep the already prepared source reader instead. + return !iceberg.__isset.delete_files || iceberg.delete_files.empty(); +} + +bool FileScannerV2::_should_refine_source_split(const TFileRangeDesc& range, + int constructed_scanners) { + // Row-group children only reduce wall time when another scanner can consume them. Keeping the + // source reader avoids repeated reader setup in serial and explicitly single-scanner scans. + return constructed_scanners >= 2 && can_refine_source_split(range); +} + +void FileScannerV2::_update_file_counter(RuntimeProfile::Counter* counter, + const FileScanSplit& split) { + if (split.is_source_split) { + // FileNumber describes FE source ranges, so BE-local physical children must not change it. + COUNTER_UPDATE(counter, 1); + } +} + bool FileScannerV2::_should_enable_file_meta_cache() const { return ExecEnv::GetInstance()->file_meta_cache()->enabled() && _split_source->num_scan_ranges() < config::max_external_file_meta_cache_num / 3; @@ -1012,6 +1169,14 @@ void FileScannerV2::_init_adaptive_batch_size_state(TFileFormatType::type format _block_size_predictor = std::make_unique( _state->preferred_block_size_bytes(), 0.0, ADAPTIVE_BATCH_INITIAL_PROBE_ROWS, _state->batch_size()); + if (_current_split.source_progress != nullptr) { + const auto source_history = _current_split.source_progress->adaptive_batch_bytes_per_row(); + if (source_history.has_value()) { + // Generated children share one FE source budget. Seed the per-scanner predictor from + // that source so physical refinement does not restart the probe for every child. + _block_size_predictor->seed_history(*source_history); + } + } } bool FileScannerV2::_should_enable_adaptive_batch_size(TFileFormatType::type format_type) const { @@ -1073,10 +1238,17 @@ void FileScannerV2::_update_adaptive_batch_size(const Block& block) { // The sample is taken after TableReader has finalized file-local columns to table columns. // This matches the memory shape seen by upstream operators and catches very wide nested // columns, such as map/string payloads, after the first probe batch. - if (!_block_size_predictor->has_history()) { + bool first_history_sample = !_block_size_predictor->has_history(); + _block_size_predictor->update(block); + if (_current_split.source_progress != nullptr) { + first_history_sample = _current_split.source_progress->update_adaptive_batch_bytes_per_row( + static_cast(block.bytes()) / static_cast(block.rows())); + } + if (first_history_sample) { + // Concurrent children may all start before the first sample is visible. Count the source's + // synchronized first sample once instead of reporting one cold probe per racing child. COUNTER_UPDATE(_adaptive_batch_probe_count_counter, 1); } - _block_size_predictor->update(block); } Status FileScannerV2::close(RuntimeState* state) { @@ -1088,19 +1260,25 @@ Status FileScannerV2::close(RuntimeState* state) { if (_table_reader != nullptr) { const auto close_status = _table_reader->close(); if (!close_status.ok()) { - // Reserve the close attempt with _try_close(), but commit the scanner-level closed - // state only after the retained table reader has completed its retryable cleanup. + // A failed reader close remains retryable, but this scanner can no longer publish + // useful children. Retire its source reservation so peer scanners are not stranded. + const auto split_status = _complete_current_split(); _is_closed.store(false); + if (!split_status.ok()) { + return split_status; + } return close_status; } _report_condition_cache_profile(); _table_reader.reset(); } + RETURN_IF_ERROR(_complete_current_split()); return Scanner::close(state); } void FileScannerV2::try_stop() { Scanner::try_stop(); + _split_source->stop(); if (_io_ctx) { _io_ctx->should_stop = true; } @@ -1111,7 +1289,6 @@ void FileScannerV2::update_realtime_counters() { return; } DORIS_CHECK(_file_cache_statistics != nullptr); - const int64_t bytes_read = cast_set(_file_reader_stats->read_bytes); auto* local_state = static_cast(_local_state); const auto file_type = _current_range.__isset.file_type @@ -1133,9 +1310,14 @@ void FileScannerV2::update_realtime_counters() { _state->get_query_ctx()->resource_ctx()->io_context()->update_scan_bytes_from_remote_storage( deltas.scan_bytes_from_remote_storage); - COUNTER_SET(_file_read_bytes_counter, bytes_read); - COUNTER_SET(_file_read_calls_counter, cast_set(_file_reader_stats->read_calls)); - COUNTER_SET(_file_read_time_counter, cast_set(_file_reader_stats->read_time_ns)); + // Scanner instances share these profile counters. Publish only scanner-private deltas so one + // scanner cannot erase or double count values already contributed by a sibling. + const auto profile_deltas = _collect_file_reader_profile_deltas( + *_file_reader_stats, &_reported_file_read_bytes, &_reported_file_read_calls, + &_reported_file_read_time); + COUNTER_UPDATE(_file_read_bytes_counter, profile_deltas.read_bytes); + COUNTER_UPDATE(_file_read_calls_counter, profile_deltas.read_calls); + COUNTER_UPDATE(_file_read_time_counter, profile_deltas.read_time_ns); DorisMetrics::instance()->query_scan_bytes->increment(deltas.scan_bytes); DorisMetrics::instance()->query_scan_rows->increment(deltas.scan_rows); @@ -1236,9 +1418,12 @@ void FileScannerV2::_collect_profile_before_close() { _reported_file_cache_statistics = *_file_cache_statistics; } if (_file_reader_stats != nullptr) { - COUNTER_SET(_file_read_bytes_counter, cast_set(_file_reader_stats->read_bytes)); - COUNTER_SET(_file_read_calls_counter, cast_set(_file_reader_stats->read_calls)); - COUNTER_SET(_file_read_time_counter, cast_set(_file_reader_stats->read_time_ns)); + const auto profile_deltas = _collect_file_reader_profile_deltas( + *_file_reader_stats, &_reported_file_read_bytes, &_reported_file_read_calls, + &_reported_file_read_time); + COUNTER_UPDATE(_file_read_bytes_counter, profile_deltas.read_bytes); + COUNTER_UPDATE(_file_read_calls_counter, profile_deltas.read_calls); + COUNTER_UPDATE(_file_read_time_counter, profile_deltas.read_time_ns); const auto read_time = cast_set(_file_reader_stats->read_time_ns); DORIS_CHECK(read_time >= _reported_io_read_time); // Some transports (for example Arrow Flight) record directly into IO, while filesystem diff --git a/be/src/exec/scan/file_scanner_v2.h b/be/src/exec/scan/file_scanner_v2.h index a8612aeda676b4..83dd924ff3f03d 100644 --- a/be/src/exec/scan/file_scanner_v2.h +++ b/be/src/exec/scan/file_scanner_v2.h @@ -64,8 +64,17 @@ class FileScannerV2 final : public Scanner { int64_t scan_bytes_from_remote_storage = 0; }; + struct FileReaderProfileDeltas { + int64_t read_bytes = 0; + int64_t read_calls = 0; + int64_t read_time_ns = 0; + }; + + enum class IgnoredSplitStatus { NONE, NOT_FOUND, EMPTY }; + enum class UncachedReaderBytesStorage { LOCAL, REMOTE, NONE }; + static bool can_refine_source_split(const TFileRangeDesc& range); #ifdef BE_TEST FileScannerV2(RuntimeState* state, RuntimeProfile* profile, std::unique_ptr table_reader); @@ -86,6 +95,27 @@ class FileScannerV2 final : public Scanner { int64_t* last_bytes_read_from_remote); static void TEST_report_file_cache_profile( RuntimeProfile* profile, const io::FileCacheStatistics& file_cache_statistics); + static int64_t TEST_cumulative_profile_delta(int64_t current, int64_t* reported) { + return _cumulative_profile_delta(current, reported); + } + static FileReaderProfileDeltas TEST_collect_file_reader_profile_deltas( + const io::FileReaderStats& stats, int64_t* reported_bytes, int64_t* reported_calls, + int64_t* reported_time) { + return _collect_file_reader_profile_deltas(stats, reported_bytes, reported_calls, + reported_time); + } + static IgnoredSplitStatus TEST_classify_ignored_split_status(const Status& status, + bool ignore_not_found, + bool stopped) { + return _classify_ignored_split_status(status, ignore_not_found, stopped); + } + static bool TEST_can_refine_source_split(const TFileRangeDesc& range) { + return can_refine_source_split(range); + } + static bool TEST_should_refine_source_split(const TFileRangeDesc& range, + int constructed_scanners) { + return _should_refine_source_split(range, constructed_scanners); + } static bool TEST_should_skip_not_found(const Status& status, bool ignore_not_found); static bool TEST_should_skip_empty(const Status& status, bool stopped); @@ -98,6 +128,10 @@ class FileScannerV2 final : public Scanner { return _should_run_adaptive_batch_size(predictor_initialized, current_split_uses_metadata_count); } + static void TEST_update_file_counter(RuntimeProfile::Counter* counter, + const FileScanSplit& split) { + _update_file_counter(counter, split); + } #endif FileScannerV2(RuntimeState* state, FileScanLocalState* parent, int64_t limit, @@ -141,6 +175,8 @@ class FileScannerV2 final : public Scanner { Block* result_block, int64_t* init_reader_ms, int64_t* get_block_ms); Status _get_next_scan_range(bool* has_next); + Status _retire_current_source_split(std::vector generated_splits = {}); + Status _complete_current_split(); TFileFormatType::type _get_current_format_type() const; Status _init_io_ctx(); Status _init_expr_ctxes(); @@ -152,8 +188,16 @@ class FileScannerV2 final : public Scanner { std::map partition_values); static bool _should_skip_not_found(const Status& status, bool ignore_not_found); static bool _should_skip_empty(const Status& status, bool stopped); + static IgnoredSplitStatus _classify_ignored_split_status(const Status& status, + bool ignore_not_found, bool stopped); static Status _contextualize_output_filter_status(Status status, TFileFormatType::type format_type); + static bool _should_refine_source_split(const TFileRangeDesc& range, int constructed_scanners); + static int64_t _cumulative_profile_delta(int64_t current, int64_t* reported); + static void _update_file_counter(RuntimeProfile::Counter* counter, const FileScanSplit& split); + static FileReaderProfileDeltas _collect_file_reader_profile_deltas( + const io::FileReaderStats& stats, int64_t* reported_bytes, int64_t* reported_calls, + int64_t* reported_time); bool _should_enable_file_meta_cache() const; std::optional _create_global_rowid_context( const TFileRangeDesc& range) const; @@ -198,6 +242,7 @@ class FileScannerV2 final : public Scanner { bool _has_prepared_split = false; int _table_reader_rf_num = 0; TFileRangeDesc _current_range; + FileScanSplit _current_split; std::unique_ptr _table_reader; std::vector _projected_columns; @@ -217,6 +262,7 @@ class FileScannerV2 final : public Scanner { std::shared_ptr _io_ctx; ShardedKVCache* _kv_cache = nullptr; format::parquet::ParquetProfileCache* _parquet_profile_cache = nullptr; + int _constructed_scanners = 1; RuntimeProfile::Counter* _scanner_total_timer = nullptr; RuntimeProfile::Counter* _init_timer = nullptr; @@ -246,6 +292,9 @@ class FileScannerV2 final : public Scanner { int64_t _last_bytes_read_from_local = 0; int64_t _last_bytes_read_from_remote = 0; int64_t _reported_io_read_time = 0; + int64_t _reported_file_read_bytes = 0; + int64_t _reported_file_read_calls = 0; + int64_t _reported_file_read_time = 0; }; } // namespace doris diff --git a/be/src/exec/scan/split_source_connector.cpp b/be/src/exec/scan/split_source_connector.cpp index ebcac31662d762..240f33ba51732f 100644 --- a/be/src/exec/scan/split_source_connector.cpp +++ b/be/src/exec/scan/split_source_connector.cpp @@ -26,6 +26,114 @@ namespace doris { using apache::thrift::transport::TTransportException; +Status SplitSourceConnector::get_next_split(bool* has_next, FileScanSplit* split, + const std::atomic_bool* scanner_stop) { + DORIS_CHECK(has_next != nullptr); + DORIS_CHECK(split != nullptr); + std::unique_lock lock(_split_lock); + while (true) { + *has_next = false; + if (_stopped) { + return Status::OK(); + } + if (!_generated_splits.empty()) { + *split = std::move(_generated_splits.front()); + _generated_splits.pop_front(); + *has_next = true; + return Status::OK(); + } + if (_source_exhausted) { + if (_active_source_splits.empty()) { + return Status::OK(); + } + _split_ready.wait(lock); + continue; + } + if (_source_claim_in_progress) { + _split_ready.wait(lock); + continue; + } + + TFileRangeDesc range; + bool has_source = false; + // Record the in-flight claim before dropping the queue lock, so another scanner cannot + // report raw EOS before this claim becomes an active source. The potentially blocking + // remote RPC must not hold the queue lock because a footer producer may publish children + // while that fetch is in flight. + _source_claim_in_progress = true; + lock.unlock(); + const auto source_status = + get_next(&has_source, &range, scanner_stop == nullptr ? &_stopped : scanner_stop); + lock.lock(); + _source_claim_in_progress = false; + if (source_status.ok() && !has_source) { + // Both local exhaustion and an empty final remote batch are terminal. Remember EOS so + // parent waiters do not wake each other into repeated empty source fetches. + _source_exhausted = true; + } + _split_ready.notify_all(); + RETURN_IF_ERROR(source_status); + if (_stopped) { + return Status::OK(); + } + if (has_source) { + // A scanner reuses its output envelope across files. Reset child-only shared range and + // context state before installing a raw FE range, or materialization can read the + // previous child's file instead of this source. + *split = {}; + split->range = std::move(range); + split->is_source_split = true; + split->source_split_id = _next_source_split_id++; + split->source_progress = std::make_shared(); + _active_source_splits.insert(split->source_split_id); + *has_next = true; + return Status::OK(); + } + // A footer producer can publish children while the final remote fetch runs without the + // queue lock. Re-enter the queue checks before deciding EOS so the fetching scanner can + // claim those children even when no third scanner is waiting to be notified. + } +} + +Status SplitSourceConnector::finish_source_split(const FileScanSplit& source_split, + std::vector generated_splits) { + if (!source_split.is_source_split || source_split.source_split_id == 0) { + return Status::InvalidArgument("Only an active source split can publish generated splits"); + } + { + std::scoped_lock lock(_split_lock); + if (_active_source_splits.erase(source_split.source_split_id) != 1) { + return Status::InvalidArgument("Source split {} is not active", + source_split.source_split_id); + } + if (!_stopped) { + if (!generated_splits.empty()) { + DORIS_CHECK(source_split.source_progress != nullptr); + source_split.source_progress->reset_for_children(generated_splits.size()); + } + for (auto& split : generated_splits) { + split.is_source_split = false; + split.source_split_id = 0; + split.source_progress = source_split.source_progress; + _generated_splits.push_back(std::move(split)); + } + } + } + _split_ready.notify_all(); + return Status::OK(); +} + +void SplitSourceConnector::stop() { + { + std::scoped_lock lock(_split_lock); + _stopped = true; + // Cancellation must release queued descriptors and footer contexts immediately; source + // producers finishing later will only retire their reservation without publishing work. + _generated_splits.clear(); + } + _split_ready.notify_all(); +} + Status LocalSplitSourceConnector::get_next(bool* has_next, TFileRangeDesc* range, const std::atomic_bool* /*scanner_stop*/) { std::lock_guard l(_range_lock); @@ -58,7 +166,7 @@ Status RemoteSplitSourceConnector::get_next(bool* has_next, TFileRangeDesc* rang RETURN_IF_CANCELLED(_state); // A local LIMIT stops this scanner without cancelling the query. Do not keep a scan // worker in the retry loop while another branch of the same query remains active. - if (scanner_stop->load(std::memory_order_acquire)) { + if (is_stopped() || scanner_stop->load(std::memory_order_acquire)) { return Status::OK(); } Status coord_status; diff --git a/be/src/exec/scan/split_source_connector.h b/be/src/exec/scan/split_source_connector.h index 332c6736b92834..b0ae61f375444a 100644 --- a/be/src/exec/scan/split_source_connector.h +++ b/be/src/exec/scan/split_source_connector.h @@ -19,9 +19,13 @@ #include #include +#include +#include +#include #include "common/config.h" #include "core/custom_allocator.h" +#include "format_v2/file_scan_context.h" #include "runtime/runtime_state.h" #include "util/client_cache.h" @@ -46,6 +50,16 @@ class SplitSourceConnector { virtual Status get_next(bool* has_next, TFileRangeDesc* range, const std::atomic_bool* scanner_stop) = 0; + // FileScannerV2 may replace one FE split with several physical children after inspecting file + // metadata. These methods are shared by local and remote sources so neither source can report + // EOS while a claimed source split is still producing children. + Status get_next_split(bool* has_next, FileScanSplit* split, + const std::atomic_bool* scanner_stop = nullptr); + bool is_stopped() const { return _stopped.load(std::memory_order_acquire); } + Status finish_source_split(const FileScanSplit& source_split, + std::vector generated_splits); + void stop(); + virtual int num_scan_ranges() = 0; virtual TFileScanRangeParams* get_params() = 0; @@ -96,6 +110,16 @@ class SplitSourceConnector { protected: int _max_scanners; + +private: + std::mutex _split_lock; + std::condition_variable _split_ready; + std::deque _generated_splits; + std::unordered_set _active_source_splits; + uint64_t _next_source_split_id = 1; + bool _source_claim_in_progress = false; + bool _source_exhausted = false; + std::atomic_bool _stopped {false}; }; /** diff --git a/be/src/format_v2/column_mapper.cpp b/be/src/format_v2/column_mapper.cpp index f1eb3d07b93980..a1073d691ed676 100644 --- a/be/src/format_v2/column_mapper.cpp +++ b/be/src/format_v2/column_mapper.cpp @@ -2605,6 +2605,7 @@ Status TableColumnMapper::create_scan_request( file_request->non_predicate_positions = *fixed_non_predicate_positions; } file_request->conjuncts.clear(); + file_request->residual_predicate_columns.clear(); file_request->metadata_pruning_safe_conjunct_count = 0; file_request->constant_pruning_safe_table_filter_count = 0; file_request->delete_conjuncts.clear(); @@ -2905,6 +2906,26 @@ Status TableColumnMapper::localize_filters(const std::vector& table } } + for (size_t table_filter_idx = 0; table_filter_idx < table_filters.size(); ++table_filter_idx) { + if (localized_table_filters[table_filter_idx]) { + continue; + } + for (const auto global_index : table_filters[table_filter_idx].global_indices) { + const auto* mapping = _find_filter_mapping(global_index); + if (mapping == nullptr || !mapping->file_local_id.has_value()) { + continue; + } + const auto local_id = LocalColumnId(*mapping->file_local_id); + if (std::ranges::find(file_request->residual_predicate_columns, local_id) == + file_request->residual_predicate_columns.end()) { + // Scanner re-evaluates every table predicate after materialization. A predicate + // that could not be localized therefore proves its input is semantic data, not a + // disposable COUNT(*) carrier. + file_request->residual_predicate_columns.push_back(local_id); + } + } + } + // Candidate columns are added before expression rewriting because their file-block positions // are needed to localize slot refs. If rewriting rejects every filter that references a visible // column, merge any independent output/filter subtrees and move the result to the lazy diff --git a/be/src/format_v2/file_reader.cpp b/be/src/format_v2/file_reader.cpp index 95c272a49181c5..7ce26e61308c3f 100644 --- a/be/src/format_v2/file_reader.cpp +++ b/be/src/format_v2/file_reader.cpp @@ -79,7 +79,10 @@ std::string FileScanRequest::debug_string() const { } else { out << "nullopt"; } - out << ", conjunct_count=" << conjuncts.size() + out << ", conjunct_count=" << conjuncts.size() << ", residual_predicate_columns=" + << join_debug_strings( + residual_predicate_columns, + [](LocalColumnId column_id) { return std::to_string(column_id.value()); }) << ", metadata_pruning_safe_conjunct_count=" << metadata_pruning_safe_conjunct_count << ", constant_pruning_safe_table_filter_count=" << constant_pruning_safe_table_filter_count << ", delete_conjunct_count=" << delete_conjuncts.size() << ", variant_schema_overrides=" diff --git a/be/src/format_v2/file_reader.h b/be/src/format_v2/file_reader.h index aa3a4756dc9c34..688b4e9be53d13 100644 --- a/be/src/format_v2/file_reader.h +++ b/be/src/format_v2/file_reader.h @@ -32,6 +32,7 @@ #include "core/field.h" #include "exprs/vexpr_fwd.h" #include "format_v2/column_data.h" +#include "format_v2/file_scan_context.h" #include "gen_cpp/PlanNodes_types.h" #include "io/file_factory.h" #include "io/fs/file_reader_writer_fwd.h" @@ -65,6 +66,16 @@ enum class FileFormat { LANCE, }; +// Format-local description of one independently readable part of a file. Scanner scheduling +// policy is attached only after this descriptor crosses the TableReader boundary. +struct PhysicalFileSplit { + int64_t start_offset = 0; + int64_t size = -1; + std::shared_ptr file_context; + int64_t format_split_id = -1; + int64_t format_split_id_end = -1; +}; + struct FileScanRequest { virtual ~FileScanRequest() = default; @@ -88,6 +99,10 @@ struct FileScanRequest { std::map non_predicate_positions; // Row-level filters converted to file-local expressions from table-level predicates. VExprContextSPtrs conjuncts; + // File-local columns still consumed by table-level predicates that could not be localized. + // Their values must survive physical reading even when COUNT(*) otherwise permits a synthetic + // carrier column. + std::vector residual_predicate_columns; // Only this leading subset may participate in footer/page metadata pruning. The boundary is // inherited from table-conjunct order so an omitted slotless unsafe expression remains a fence. size_t metadata_pruning_safe_conjunct_count = std::numeric_limits::max(); @@ -116,11 +131,20 @@ struct FileScanRequest { // Variant. Keeping this explicit prevents generic Parquet scans from guessing based on names. std::vector variant_schema_overrides; + // Digest of the exact table-predicate snapshot used to build this immutable request. Format + // readers use it to prevent late runtime filters from inheriting stale adaptive state. + std::optional predicate_snapshot_digest; + bool is_count_star_placeholder(LocalColumnId column_id) const { return std::ranges::find(count_star_placeholder_columns, column_id) != count_star_placeholder_columns.end(); } + bool is_residual_predicate_column(LocalColumnId column_id) const { + return std::ranges::find(residual_predicate_columns, column_id) != + residual_predicate_columns.end(); + } + bool is_predicate_only(LocalColumnId column_id) const { return std::ranges::find(predicate_only_columns, column_id) != predicate_only_columns.end(); } @@ -365,6 +389,17 @@ class FileReader { // Initialize file reader and parse file metadata. virtual Status init(RuntimeState* state); + // Optionally refine the prepared file range into format-specific physical children after + // metadata initialization. The default keeps non-columnar formats on their original range. + virtual Status build_physical_splits(std::vector* splits, + bool* was_split) const { + DORIS_CHECK(splits != nullptr); + DORIS_CHECK(was_split != nullptr); + splits->clear(); + *was_split = false; + return Status::OK(); + } + // Set the maximum row count for the next physical read batch. Readers that do not batch by // rows may ignore it. virtual void set_batch_size(size_t batch_size) { (void)batch_size; } @@ -436,6 +471,11 @@ class FileReader { return Status::NotSupported("FileReader does not support aggregate pushdown"); } + virtual Status get_metadata_aggregate_result(const FileAggregateRequest& request, + FileAggregateResult* result) { + return Status::NotSupported("FileReader does not support metadata-only aggregate pushdown"); + } + // Condition cache is managed by TableReader and consumed by physical file readers. // On cache HIT, readers may skip granules whose cached bit is false before doing column IO. // On cache MISS, readers mark a granule true when row-level predicates keep at least one row diff --git a/be/src/format_v2/file_scan_context.h b/be/src/format_v2/file_scan_context.h new file mode 100644 index 00000000000000..68a2898727aed2 --- /dev/null +++ b/be/src/format_v2/file_scan_context.h @@ -0,0 +1,117 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#pragma once + +#include +#include +#include +#include +#include +#include +#include + +#include "common/check.h" +#include "gen_cpp/PlanNodes_types.h" + +namespace doris { + +struct ConditionCacheSplitContext; + +// Opaque immutable metadata shared by physical children generated from one parent split. Concrete +// file formats own the derived type so scanner and split-source layers do not depend on internals. +class FileContext { +public: + virtual ~FileContext() = default; +}; + +// Tracks completion of one FE source range after BE refinement. Generated children share this +// object, so the FE range is reported finished only when its final physical child completes. +class SourceSplitProgress { +public: + void reset_for_children(size_t children) { + DORIS_CHECK(children > 0); + _remaining.store(children, std::memory_order_release); + } + + bool complete_one() { + const size_t previous = _remaining.fetch_sub(1, std::memory_order_acq_rel); + DORIS_CHECK(previous > 0); + return previous == 1; + } + + std::optional adaptive_batch_bytes_per_row() const { + std::scoped_lock lock(_adaptive_batch_lock); + return _adaptive_batch_bytes_per_row; + } + + bool update_adaptive_batch_bytes_per_row(double bytes_per_row) { + DORIS_CHECK(bytes_per_row > 0); + std::scoped_lock lock(_adaptive_batch_lock); + // Sibling scanners can finish probes concurrently. Merge their samples under the source + // lock so later physical children never regress to a fresh small probe. + const bool first_source_sample = !_adaptive_batch_bytes_per_row.has_value(); + _adaptive_batch_bytes_per_row = + _adaptive_batch_bytes_per_row.has_value() + ? 0.9 * *_adaptive_batch_bytes_per_row + 0.1 * bytes_per_row + : bytes_per_row; + return first_source_sample; + } + +private: + std::atomic _remaining {1}; + mutable std::mutex _adaptive_batch_lock; + std::optional _adaptive_batch_bytes_per_row; +}; + +// BE-local scheduling envelope. It deliberately carries no Thrift fields: FE ranges remain the +// source of table-format semantics, while generated physical children add only opaque metadata and +// format-local subrange ids. +struct FileScanSplit { + TFileRangeDesc range; + std::shared_ptr source_range; + int64_t start_offset = 0; + int64_t size = -1; + bool clear_table_level_row_count = false; + std::shared_ptr file_context; + std::shared_ptr condition_cache_split_context; + int64_t format_split_id = -1; + int64_t format_split_id_end = -1; + bool is_source_split = false; + uint64_t source_split_id = 0; + std::shared_ptr source_progress; + + // GLOBAL_ROWID second-phase reads batch by the FE source mapping, while start_offset/size above + // identify only this first-phase physical child. + const TFileRangeDesc& source_identity_range() const { + return source_range == nullptr ? range : *source_range; + } + + TFileRangeDesc materialize_range() const { + auto materialized = source_range == nullptr ? range : *source_range; + if (source_range != nullptr) { + materialized.__set_start_offset(start_offset); + materialized.__set_size(size); + } + if (clear_table_level_row_count && materialized.__isset.table_format_params) { + materialized.table_format_params.__isset.table_level_row_count = false; + } + return materialized; + } +}; + +} // namespace doris diff --git a/be/src/format_v2/jni/jni_table_reader.cpp b/be/src/format_v2/jni/jni_table_reader.cpp index e454d0b11b6e4a..f56de9c51d4462 100644 --- a/be/src/format_v2/jni/jni_table_reader.cpp +++ b/be/src/format_v2/jni/jni_table_reader.cpp @@ -77,7 +77,9 @@ Status JniTableReader::prepare_split(const SplitReadOptions& options) { return _open_jni_scanner(); } -Status JniTableReader::refresh_conjuncts(VExprContextSPtrs conjuncts) { +Status JniTableReader::refresh_conjuncts(VExprContextSPtrs conjuncts, + std::optional condition_cache_digest, + bool all_runtime_filters_applied) { if (_scanner_opened) { SCOPED_TIMER(_profile.total_timer); SCOPED_TIMER(_profile.refresh_conjuncts_timer); @@ -91,7 +93,8 @@ Status JniTableReader::refresh_conjuncts(VExprContextSPtrs conjuncts) { RETURN_IF_ERROR(conjunct->open(_runtime_state)); } } - return TableReader::refresh_conjuncts(std::move(conjuncts)); + return TableReader::refresh_conjuncts(std::move(conjuncts), condition_cache_digest, + all_runtime_filters_applied); } Status JniTableReader::get_block(Block* output_block, bool* eos) { diff --git a/be/src/format_v2/jni/jni_table_reader.h b/be/src/format_v2/jni/jni_table_reader.h index ab94b2702cad03..ac8c14670388e4 100644 --- a/be/src/format_v2/jni/jni_table_reader.h +++ b/be/src/format_v2/jni/jni_table_reader.h @@ -49,7 +49,9 @@ class JniTableReader : public TableReader { Status init(TableReadOptions&& options) override; Status prepare_split(const SplitReadOptions& options) override; - Status refresh_conjuncts(VExprContextSPtrs conjuncts) override; + Status refresh_conjuncts(VExprContextSPtrs conjuncts, + std::optional condition_cache_digest = std::nullopt, + bool all_runtime_filters_applied = false) override; Status get_block(Block* block, bool* eos) override; Status abort_split() override; Status close() override; diff --git a/be/src/format_v2/orc/orc_reader.cpp b/be/src/format_v2/orc/orc_reader.cpp index 5534d415068cb5..f6e06212ef1b92 100644 --- a/be/src/format_v2/orc/orc_reader.cpp +++ b/be/src/format_v2/orc/orc_reader.cpp @@ -858,6 +858,7 @@ struct OrcReaderScanState { }; std::unique_ptr<::orc::Reader> reader; + std::shared_ptr file_context; const ::orc::Type* root_type = nullptr; ::orc::ReaderMetrics reader_metrics; int64_t footer_read_calls = 0; @@ -1136,11 +1137,13 @@ OrcReader::OrcReader(std::shared_ptr& system_propertie std::unique_ptr& file_description, std::shared_ptr io_ctx, RuntimeProfile* profile, std::optional global_rowid_context, - bool enable_mapping_timestamp_tz, bool enable_mapping_varbinary) + bool enable_mapping_timestamp_tz, bool enable_mapping_varbinary, + std::shared_ptr file_context) : FileReader(system_properties, file_description, io_ctx, profile), _global_rowid_context(std::move(global_rowid_context)), _enable_mapping_timestamp_tz(enable_mapping_timestamp_tz), - _enable_mapping_varbinary(enable_mapping_varbinary) {} + _enable_mapping_varbinary(enable_mapping_varbinary), + _file_context(std::move(file_context)) {} OrcReader::~OrcReader() = default; @@ -1299,10 +1302,6 @@ Status OrcReader::init(RuntimeState* state) { } } - ::orc::ReaderOptions options; - options.setMemoryPool(*ExecEnv::GetInstance()->orc_memory_pool()); - options.setReaderMetrics(&_state->reader_metrics); - auto* meta_cache = ExecEnv::GetInstance()->file_meta_cache(); // Preserve the V1 identity policy, including its file-size fallback when mtime is absent. auto cache_key = FileMetaCache::get_key(_file_reader, *_file_description); @@ -1310,15 +1309,6 @@ Status OrcReader::init(RuntimeState* state) { // Keep serialized V2 tails in their own domain so a hit always has the expected value type. const bool cache_enabled = meta_cache != nullptr && meta_cache->enabled(); cache_key.append("\0orc-v2-tail", 12); - ObjLRUCache::CacheHandle cache_handle; - const bool cache_hit = cache_enabled && meta_cache->lookup(cache_key, &cache_handle); - // A warm open must use the serialized tail before createReader issues its footer read. - if (cache_hit) { - options.setSerializedFileTail(*cache_handle.data()); - ++_reader_statistics.file_footer_hit_cache; - ++_state->footer_cache_hits; - } - OrcFileInputStreamOptions input_stream_options; const auto natural_read_size_mb = config::orc_natural_read_size_mb; if (natural_read_size_mb <= 0 || natural_read_size_mb > 1024) { @@ -1332,37 +1322,122 @@ Status OrcReader::init(RuntimeState* state) { input_stream_options.max_merge_distance_bytes = state->query_options().orc_max_merge_distance_bytes; } - auto input_stream = std::make_unique( - _file_description->path, _file_reader, _io_ctx.get(), _profile, input_stream_options); - try { - _state->reader = ::orc::createReader(std::move(input_stream), options); - _state->root_type = &_state->reader->getType(); - // Publish only successfully parsed tails, and only when the file has a stable identity. - if (!cache_hit) { - ++_reader_statistics.file_footer_read_calls; - // close() resets per-open state; cumulative reader statistics would publish twice - // when close is repeated or the same reader is initialized for another file. - ++_state->footer_read_calls; - if (cache_enabled) { - meta_cache->insert(cache_key, - new std::string(_state->reader->getSerializedFileTail()), - &cache_handle); + auto create_reader = [&](const std::string* serialized_file_tail, + std::unique_ptr<::orc::Reader>* reader) -> Status { + DORIS_CHECK(reader != nullptr); + ::orc::ReaderOptions options; + ObjLRUCache::CacheHandle cache_handle; + const bool cache_hit = serialized_file_tail == nullptr && cache_enabled && + meta_cache->lookup(cache_key, &cache_handle); + options.setMemoryPool(*ExecEnv::GetInstance()->orc_memory_pool()); + options.setReaderMetrics(&_state->reader_metrics); + if (serialized_file_tail != nullptr) { + options.setSerializedFileTail(*serialized_file_tail); + } else if (cache_hit) { + options.setSerializedFileTail(*cache_handle.data()); + } + auto input_stream = + std::make_unique(_file_description->path, _file_reader, + _io_ctx.get(), _profile, input_stream_options); + try { + *reader = ::orc::createReader(std::move(input_stream), options); + } catch (const std::exception& e) { + if (is_orc_stop(_io_ctx.get(), e)) { + return Status::EndOfFile("stop"); + } + const std::string err_msg = e.what(); + if (err_msg.find("No such file or directory") != std::string::npos || + err_msg.find("NoSuchKey") != std::string::npos) { + return Status::NotFound(err_msg); + } + return Status::InternalError("Failed to open ORC file {}: {}", _file_description->path, + err_msg); + } + // Parent contexts retain the existing process-wide tail cache policy. A child uses its + // parent's tail without reporting another physical footer read or process-cache lookup. + if (serialized_file_tail == nullptr) { + if (cache_hit) { + ++_reader_statistics.file_footer_hit_cache; + ++_state->footer_cache_hits; + } else { + ++_reader_statistics.file_footer_read_calls; + ++_state->footer_read_calls; + if (cache_enabled) { + meta_cache->insert(cache_key, + new std::string((*reader)->getSerializedFileTail()), + &cache_handle); + } } } - } catch (const std::exception& e) { - if (is_orc_stop(_io_ctx.get(), e)) { - return Status::EndOfFile("stop"); + return Status::OK(); + }; + + const int64_t resolved_mtime = + _file_description->mtime != 0 ? _file_description->mtime : _file_reader->mtime(); + const int64_t resolved_file_size = _file_description->file_size >= 0 + ? _file_description->file_size + : cast_set(_file_reader->size()); + const bool has_stable_identity = resolved_mtime != 0 || _file_description->is_immutable; + const auto reader_path = _file_reader->path().native(); + const std::string file_identity = fmt::format( + "orc::fs[{}]={}::path[{}]={}::mtime={}::size={}::immutable={}", + _file_description->fs_name.size(), _file_description->fs_name, reader_path.size(), + reader_path, resolved_mtime, resolved_file_size, _file_description->is_immutable); + auto build_context = [&](const ::orc::Reader& reader, + std::shared_ptr* result) -> Status { + DORIS_CHECK(result != nullptr); + auto context = std::make_shared(); + context->file_identity = file_identity; + context->has_stable_identity = has_stable_identity; + try { + if (has_stable_identity) { + // Mutable files cannot publish children, so avoid serializing a tail that no other + // reader may safely consume. + context->serialized_file_tail = reader.getSerializedFileTail(); + } + const auto stripe_count = reader.getNumberOfStripes(); + context->stripes.reserve(stripe_count); + for (uint64_t stripe_id = 0; stripe_id < stripe_count; ++stripe_id) { + const auto stripe = reader.getStripe(stripe_id); + DORIS_CHECK(stripe != nullptr); + context->stripes.push_back({.offset = stripe->getOffset(), + .length = stripe->getLength(), + .rows = stripe->getNumberOfRows()}); + } + } catch (const std::exception& e) { + return Status::InternalError("Failed to cache ORC file context for {}: {}", + _file_description->path, e.what()); + } + *result = std::move(context); + return Status::OK(); + }; + + std::shared_ptr resolved_context = std::move(_file_context); + std::unique_ptr<::orc::Reader> loaded_reader; + if (resolved_context == nullptr) { + RETURN_IF_ERROR(create_reader(nullptr, &loaded_reader)); + RETURN_IF_ERROR(build_context(*loaded_reader, &resolved_context)); + } + + if (resolved_context != nullptr) { + _state->file_context = + std::dynamic_pointer_cast(resolved_context); + if (_state->file_context == nullptr) { + return Status::InvalidArgument("ORC split has an incompatible file context"); + } + if (_state->file_context->file_identity != file_identity) { + return Status::InvalidArgument("ORC split file context does not match file identity"); } - // invoker maybe just skip Status.NotFound and continue - // so we need distinguish between it and other kinds of errors - const std::string err_msg = e.what(); - if (err_msg.find("No such file or directory") != std::string::npos || - err_msg.find("NoSuchKey") != std::string::npos) { - return Status::NotFound(err_msg); + if (loaded_reader != nullptr) { + // The parent keeps the reader that produced the context. Generated children rebuild + // independent readers from the shared serialized tail. + _state->reader = std::move(loaded_reader); + } else { + RETURN_IF_ERROR( + create_reader(&_state->file_context->serialized_file_tail, &_state->reader)); } - return Status::InternalError("Failed to open ORC file {}: {}", _file_description->path, - err_msg); } + _state->root_type = &_state->reader->getType(); return Status::OK(); } @@ -1944,7 +2019,8 @@ void OrcReader::_split_byte_window(uint64_t* start, uint64_t* end) const { Status OrcReader::_collect_split_stripes(std::vector* stripe_indices) const { DORIS_CHECK(stripe_indices != nullptr); stripe_indices->clear(); - const auto stripe_count = _state->reader->getNumberOfStripes(); + DORIS_CHECK(_state->file_context != nullptr); + const auto stripe_count = _state->file_context->stripes.size(); stripe_indices->reserve(stripe_count); uint64_t split_start = 0; @@ -1952,14 +2028,7 @@ Status OrcReader::_collect_split_stripes(std::vector* stripe_indices) _split_byte_window(&split_start, &split_end); for (uint64_t stripe_index = 0; stripe_index < stripe_count; ++stripe_index) { - std::unique_ptr<::orc::StripeInformation> stripe_information; - try { - stripe_information = _state->reader->getStripe(stripe_index); - } catch (const std::exception& e) { - return Status::InternalError("Failed to read ORC stripe info: {}", e.what()); - } - DORIS_CHECK(stripe_information != nullptr); - const auto stripe_offset = stripe_information->getOffset(); + const auto stripe_offset = _state->file_context->stripes[stripe_index].offset; if (stripe_offset >= split_start && stripe_offset < split_end) { stripe_indices->push_back(stripe_index); } @@ -1967,6 +2036,73 @@ Status OrcReader::_collect_split_stripes(std::vector* stripe_indices) return Status::OK(); } +Status OrcReader::_collect_selected_stripes(std::vector* stripe_indices) const { + DORIS_CHECK(stripe_indices != nullptr); + stripe_indices->clear(); + if (!_state->stripe_ranges_applied) { + return _collect_split_stripes(stripe_indices); + } + const auto stripe_count = _state->reader->getNumberOfStripes(); + for (const auto& stripe_range : _state->selected_stripe_ranges) { + if (stripe_range.last_stripe < stripe_range.first_stripe || + stripe_range.last_stripe > stripe_count) { + return Status::InternalError("Invalid ORC stripe range {}-{}", + stripe_range.first_stripe, stripe_range.last_stripe); + } + for (uint64_t stripe_id = stripe_range.first_stripe; stripe_id < stripe_range.last_stripe; + ++stripe_id) { + stripe_indices->push_back(stripe_id); + } + } + return Status::OK(); +} + +Status OrcReader::build_physical_splits(std::vector* splits, + bool* was_split) const { + DORIS_CHECK(splits != nullptr); + DORIS_CHECK(was_split != nullptr); + splits->clear(); + *was_split = false; + if (_state == nullptr || _state->reader == nullptr || _state->file_context == nullptr) { + return Status::Uninitialized("OrcReader is not open"); + } + if (!_state->file_context->has_stable_identity) { + // Generated readers outlive the planning reader. Without a versioned identity, sharing a + // serialized tail could pair stale stripe metadata with bytes from an overwritten object. + return Status::OK(); + } + + std::vector selected_stripes; + RETURN_IF_ERROR(_collect_selected_stripes(&selected_stripes)); + splits->reserve(selected_stripes.size()); + for (const auto stripe_id : selected_stripes) { + if (stripe_id >= _state->file_context->stripes.size()) { + return Status::InternalError("Invalid ORC physical stripe {}", stripe_id); + } + const auto& stripe = _state->file_context->stripes[stripe_id]; + if (stripe_id > static_cast(std::numeric_limits::max()) || + stripe.length == 0 || + stripe.offset > static_cast(std::numeric_limits::max()) || + stripe.length > static_cast(std::numeric_limits::max()) || + stripe.offset + stripe.length < stripe.offset || + stripe.offset + stripe.length > + static_cast(std::numeric_limits::max())) { + // A malformed envelope must stay on the initialized reader, whose ORC validation is + // authoritative and cannot accidentally publish an overlapping child range. + splits->clear(); + return Status::OK(); + } + PhysicalFileSplit split; + split.start_offset = static_cast(stripe.offset); + split.size = static_cast(stripe.length); + split.file_context = _state->file_context; + split.format_split_id = static_cast(stripe_id); + splits->push_back(std::move(split)); + } + *was_split = true; + return Status::OK(); +} + void OrcReader::_apply_split_range() { uint64_t start = 0; uint64_t end = std::numeric_limits::max(); @@ -2722,21 +2858,7 @@ Status OrcReader::get_aggregate_result(const format::FileAggregateRequest& reque } std::vector selected_stripes; - if (_state->stripe_ranges_applied) { - for (const auto& stripe_range : _state->selected_stripe_ranges) { - if (stripe_range.last_stripe < stripe_range.first_stripe || - stripe_range.last_stripe > _state->reader->getNumberOfStripes()) { - return Status::InternalError("Invalid ORC stripe range {}-{}", - stripe_range.first_stripe, stripe_range.last_stripe); - } - for (uint64_t stripe_index = stripe_range.first_stripe; - stripe_index < stripe_range.last_stripe; ++stripe_index) { - selected_stripes.push_back(stripe_index); - } - } - } else { - RETURN_IF_ERROR(_collect_split_stripes(&selected_stripes)); - } + RETURN_IF_ERROR(_collect_selected_stripes(&selected_stripes)); for (const auto stripe_index : selected_stripes) { std::unique_ptr<::orc::StripeInformation> stripe_information; diff --git a/be/src/format_v2/orc/orc_reader.h b/be/src/format_v2/orc/orc_reader.h index a98de35c8f920d..0c8e9011badcbf 100644 --- a/be/src/format_v2/orc/orc_reader.h +++ b/be/src/format_v2/orc/orc_reader.h @@ -17,6 +17,7 @@ #pragma once +#include #include #include #include @@ -37,6 +38,21 @@ namespace doris::format::orc { struct OrcReaderScanState; +struct OrcStripeMetadata { + uint64_t offset = 0; + uint64_t length = 0; + uint64_t rows = 0; +}; + +// ORC readers keep independent row/stripe execution state. Only the immutable serialized tail and +// stripe layout cross physical children, so concurrent scanners never share an ORC RowReader. +struct OrcSharedFileContext final : public FileContext { + std::string file_identity; + bool has_stable_identity = false; + std::string serialized_file_tail; + std::vector stripes; +}; + namespace detail { bool valid_statistics_bounds(const Field& min_value, const Field& max_value); } // namespace detail @@ -49,13 +65,16 @@ class OrcReader final : public format::FileReader { std::unique_ptr& file_description, std::shared_ptr io_ctx, RuntimeProfile* profile, std::optional global_rowid_context = std::nullopt, - bool enable_mapping_timestamp_tz = false, bool enable_mapping_varbinary = false); + bool enable_mapping_timestamp_tz = false, bool enable_mapping_varbinary = false, + std::shared_ptr file_context = nullptr); ~OrcReader() override; static format::ColumnDefinition row_position_column_definition(); Status init(RuntimeState* state) override; void set_batch_size(size_t batch_size) override; + Status build_physical_splits(std::vector* splits, + bool* was_split) const override; Status get_schema(std::vector* const file_schema) const override; std::unique_ptr create_column_mapper( format::TableColumnMapperOptions options) const override; @@ -133,6 +152,7 @@ class OrcReader final : public format::FileReader { // Collect stripes whose offset falls in the current split window, matching ORC // RowReader's own range inclusion test. Status _collect_split_stripes(std::vector* stripe_indices) const; + Status _collect_selected_stripes(std::vector* stripe_indices) const; // Seed row_reader_options with the split byte window so each split only reads its own // stripes. Skips the call for a whole-file window to keep ORC library defaults. void _apply_split_range(); @@ -200,6 +220,7 @@ class OrcReader final : public format::FileReader { bool _enable_mapping_timestamp_tz = false; bool _enable_mapping_varbinary = false; size_t _batch_size = 0; + std::shared_ptr _file_context; }; } // namespace doris::format::orc diff --git a/be/src/format_v2/parquet/parquet_file_context.cpp b/be/src/format_v2/parquet/parquet_file_context.cpp index d164d8783692f9..b709c843df8ac7 100644 --- a/be/src/format_v2/parquet/parquet_file_context.cpp +++ b/be/src/format_v2/parquet/parquet_file_context.cpp @@ -28,6 +28,7 @@ #include "common/check.h" #include "common/config.h" #include "format_v2/parquet/parquet_crypto.h" +#include "format_v2/parquet/parquet_scan.h" #include "format_v2/parquet/parquet_statistics.h" #include "format_v2/parquet/reader/native/column_chunk_reader.h" #include "io/cache/cached_remote_file_reader.h" @@ -49,7 +50,10 @@ constexpr size_t V2_PARQUET_FOOTER_SIZE = 8; NativeParquetMetadata::NativeParquetMetadata(tparquet::FileMetaData metadata, size_t parsed_size, std::shared_ptr crypto) - : _metadata(std::move(metadata)), _parsed_size(parsed_size), _crypto(std::move(crypto)) { + : _metadata(std::move(metadata)), + _row_group_first_rows(_metadata.row_groups.size()), + _parsed_size(parsed_size), + _crypto(std::move(crypto)) { ExecEnv::GetInstance()->parquet_meta_tracker()->consume(get_mem_size()); } @@ -67,12 +71,20 @@ Status NativeParquetMetadata::init_schema(bool enable_mapping_varbinary, // Native readers address projected leaves by stable DFS IDs. Assign them only on the private // v2 schema object so v1's cached schema lifecycle and numbering remain untouched. _schema.assign_ids(); + int64_t next_first_row = 0; for (size_t row_group_idx = 0; row_group_idx < _metadata.row_groups.size(); ++row_group_idx) { const auto& row_group = _metadata.row_groups[row_group_idx]; if (row_group.num_rows < 0) { return Status::Corruption("Parquet row group {} has negative row count {}", row_group_idx, row_group.num_rows); } + if (row_group.num_rows > std::numeric_limits::max() - next_first_row) { + return Status::Corruption("Parquet row counts overflow at row group {}", row_group_idx); + } + // Generated children share this immutable prefix through the footer context. Recomputing + // all prefixes in every exact-row-group child would turn R-way refinement into O(R^2). + _row_group_first_rows[row_group_idx] = next_first_row; + next_first_row += row_group.num_rows; if (row_group.columns.size() != _schema.physical_fields_size()) { // All v2 planners index chunks by the native DFS leaf order, so validate cardinality // once before any projection, prefetch, or decoder can perform indexed access. @@ -306,11 +318,19 @@ std::string build_page_cache_file_key(const io::FileReader& file_reader, } // namespace +bool ParquetFileContext::can_refine_physical_splits() const { + // InMemoryFileReader lazily owns one whole-file buffer per reader. Publishing children would + // reload and copy that complete HTTP object once per child, so keep its initialized reader. + return native_file != nullptr && + typeid_cast(native_file.get()) == nullptr; +} + Status ParquetFileContext::open(io::FileReaderSPtr input_file_reader, io::IOContext* io_ctx, bool enable_page_cache, const io::FileDescription& file_description, bool enable_mapping_timestamp_tz, bool enable_mapping_varbinary, bool preserve_binary_uuid, - const TFileEncryptionInfo* encryption_info) { + const TFileEncryptionInfo* encryption_info, + std::shared_ptr file_context) { DORIS_CHECK(input_file_reader != nullptr); contains_variant = false; if (detail::should_stage_small_http_file(input_file_reader->path().native(), @@ -343,31 +363,66 @@ Status ParquetFileContext::open(io::FileReaderSPtr input_file_reader, io::IOCont meta_cache_key.push_back(static_cast(enable_mapping_timestamp_tz)); meta_cache_key.push_back(static_cast(preserve_binary_uuid)); } - size_t native_footer_size = 0; - if (has_stable_meta_cache_identity && meta_cache != nullptr && meta_cache->enabled() && - meta_cache->lookup(meta_cache_key, &native_meta_cache_handle)) { - native_metadata = native_meta_cache_handle.data(); - ++native_footer_cache_hits; - } else { - auto footer_status = parse_native_parquet_footer( - native_file, &native_metadata_owner, &native_footer_size, io_ctx, - enable_mapping_varbinary, enable_mapping_timestamp_tz, preserve_binary_uuid, - encryption_info); - if (!footer_status.ok()) { - // Schema validators have no file identity. Attach it before leaving the file - // layer, preserving the original code used by callers to classify the failure. - return footer_status.prepend(fmt::format("Failed to read Parquet footer for '{}': ", - native_file->path().native())); - } - ++native_footer_read_calls; - if (has_stable_meta_cache_identity && meta_cache != nullptr && meta_cache->enabled()) { - meta_cache->insert(meta_cache_key, native_metadata_owner.release(), - &native_meta_cache_handle); - native_metadata = native_meta_cache_handle.data(); + // A child must never consume a context from another physical file. Normalize optional FE + // identity fields with reader values before validating the parent-provided context. + const int64_t identity_mtime = + file_description.mtime != 0 ? file_description.mtime : native_file->mtime(); + const int64_t identity_file_size = file_description.file_size >= 0 + ? file_description.file_size + : cast_set(native_file->size()); + const auto identity_path = native_file->path().native(); + const std::string file_identity = fmt::format( + "fs[{}]={}::path[{}]={}::mtime={}::size={}::immutable={}::varbinary={}::timestamp_tz={" + "}::preserve_uuid={}", + file_description.fs_name.size(), file_description.fs_name, identity_path.size(), + identity_path, identity_mtime, identity_file_size, file_description.is_immutable, + enable_mapping_varbinary, enable_mapping_timestamp_tz, preserve_binary_uuid); + auto load_context = [&](std::shared_ptr* result) -> Status { + auto loaded = std::make_shared(); + loaded->adaptive_scan_context = std::make_shared(); + loaded->file_identity = file_identity; + loaded->has_stable_identity = has_stable_meta_cache_identity; + if (has_stable_meta_cache_identity && meta_cache != nullptr && meta_cache->enabled() && + meta_cache->lookup(meta_cache_key, &loaded->metadata_cache_handle)) { + loaded->metadata = loaded->metadata_cache_handle.data(); + ++native_footer_cache_hits; } else { - native_metadata = native_metadata_owner.get(); + size_t native_footer_size = 0; + auto footer_status = parse_native_parquet_footer( + native_file, &loaded->metadata_owner, &native_footer_size, io_ctx, + enable_mapping_varbinary, enable_mapping_timestamp_tz, preserve_binary_uuid, + encryption_info); + if (!footer_status.ok()) { + return footer_status.prepend(fmt::format("Failed to read Parquet footer for '{}': ", + native_file->path().native())); + } + ++native_footer_read_calls; + if (has_stable_meta_cache_identity && meta_cache != nullptr && meta_cache->enabled()) { + meta_cache->insert(meta_cache_key, loaded->metadata_owner.release(), + &loaded->metadata_cache_handle); + loaded->metadata = loaded->metadata_cache_handle.data(); + } else { + loaded->metadata = loaded->metadata_owner.get(); + } } + DORIS_CHECK(loaded->metadata != nullptr); + *result = std::move(loaded); + return Status::OK(); + }; + + std::shared_ptr resolved_context = std::move(file_context); + if (resolved_context == nullptr) { + RETURN_IF_ERROR(load_context(&resolved_context)); + } + shared_file_context = + std::dynamic_pointer_cast(resolved_context); + if (shared_file_context == nullptr) { + return Status::InvalidArgument("Parquet split has an incompatible file context"); + } + if (shared_file_context->file_identity != file_identity) { + return Status::InvalidArgument("Parquet split file context does not match file identity"); } + native_metadata = shared_file_context->metadata; DORIS_CHECK(native_metadata != nullptr); auto page_cache_file_key = build_page_cache_file_key(*native_file, file_description); @@ -723,8 +778,7 @@ Status ParquetFileContext::close() { } native_row_group_file.reset(); native_metadata = nullptr; - native_metadata_owner.reset(); - native_meta_cache_handle = {}; + shared_file_context.reset(); native_file.reset(); native_io_ctx = nullptr; native_page_cache_enabled = false; diff --git a/be/src/format_v2/parquet/parquet_file_context.h b/be/src/format_v2/parquet/parquet_file_context.h index 6de380e3d54dd7..f2f25ee2d5578a 100644 --- a/be/src/format_v2/parquet/parquet_file_context.h +++ b/be/src/format_v2/parquet/parquet_file_context.h @@ -27,6 +27,7 @@ #include #include "common/status.h" +#include "format_v2/file_scan_context.h" #include "format_v2/parquet/native_schema_desc.h" #include "io/fs/file_reader.h" #include "util/obj_lru_cache.h" @@ -45,6 +46,7 @@ namespace doris::format::parquet { struct NativeParquetPageIndex; class ParquetFileCrypto; +class ParquetAdaptiveContext; // V2-owned footer/schema tree. Production planning and decoding consume this object directly; // Arrow metadata is intentionally not materialized from the serialized footer. @@ -58,16 +60,32 @@ class NativeParquetMetadata { bool preserve_binary_uuid = false); const tparquet::FileMetaData& to_thrift() const { return _metadata; } const NativeFieldDescriptor& schema() const { return _schema; } - size_t get_mem_size() const { return _parsed_size; } const std::shared_ptr& encryption() const { return _crypto; } + const std::vector& row_group_first_rows() const { return _row_group_first_rows; } + size_t get_mem_size() const { + return _parsed_size + _row_group_first_rows.capacity() * sizeof(int64_t); + } private: tparquet::FileMetaData _metadata; NativeFieldDescriptor _schema; + std::vector _row_group_first_rows; size_t _parsed_size = 0; std::shared_ptr _crypto; }; +// Physical children of one parent split share immutable footer metadata and synchronized adaptive +// hints. Every ParquetReader still owns its file, page-index, merge-reader, and row-group state. +struct ParquetSharedFileContext final : public FileContext { + std::string file_identity; + bool has_stable_identity = false; + const NativeParquetMetadata* metadata = nullptr; + std::unique_ptr metadata_owner; + ObjLRUCache::CacheHandle metadata_cache_handle; + // Predicate-keyed adaptive state is shared only by children of one parent split. + std::shared_ptr adaptive_scan_context; +}; + struct ParquetPageCacheRange { int64_t offset = 0; int64_t size = 0; @@ -135,11 +153,10 @@ struct ParquetFileContext { // large chunks and in-memory files keep native_file. io::FileReaderSPtr native_row_group_file; io::IOContext* native_io_ctx = nullptr; - // V2-owned Thrift footer/schema used to construct native page/encoding readers. A cache hit is - // owned by native_meta_cache_handle; a miss without cache is owned by native_metadata_owner. + // V2-owned Thrift footer/schema used to construct native page/encoding readers. The shared + // parent context owns either the parsed value or its process-cache handle. const NativeParquetMetadata* native_metadata = nullptr; - std::unique_ptr native_metadata_owner; - ObjLRUCache::CacheHandle native_meta_cache_handle; + std::shared_ptr shared_file_context; int64_t native_footer_read_calls = 0; int64_t native_footer_cache_hits = 0; bool native_page_cache_enabled = false; @@ -152,7 +169,9 @@ struct ParquetFileContext { const io::FileDescription& file_description, bool enable_mapping_timestamp_tz = false, bool enable_mapping_varbinary = false, bool preserve_binary_uuid = false, - const TFileEncryptionInfo* encryption_info = nullptr); + const TFileEncryptionInfo* encryption_info = nullptr, + std::shared_ptr file_context = nullptr); + bool can_refine_physical_splits() const; Status load_native_offset_indexes( int row_group_id, const std::unordered_set& leaf_column_ids, std::unordered_map* offset_indexes) const; diff --git a/be/src/format_v2/parquet/parquet_reader.cpp b/be/src/format_v2/parquet/parquet_reader.cpp index d9ad0c8df03119..96ed98d3d94c35 100644 --- a/be/src/format_v2/parquet/parquet_reader.cpp +++ b/be/src/format_v2/parquet/parquet_reader.cpp @@ -20,6 +20,7 @@ #include #include +#include #include #include #include @@ -43,6 +44,7 @@ #include "format_v2/parquet/parquet_scan.h" #include "format_v2/parquet/parquet_statistics.h" #include "format_v2/parquet/reader/count_column_reader.h" +#include "format_v2/parquet/reader/native/column_chunk_reader.h" #include "io/io_common.h" #include "runtime/runtime_state.h" #include "util/timezone_utils.h" @@ -578,7 +580,9 @@ ParquetReader::ParquetReader(std::shared_ptr& system_p bool enable_mapping_timestamp_tz, bool enable_mapping_varbinary, std::optional hive_parquet_time_zone, bool preserve_binary_uuid, ParquetProfileCache* profile_cache, - std::optional encryption_info) + std::optional encryption_info, + std::shared_ptr file_context, + int64_t format_split_id, int64_t format_split_id_end) : FileReader(system_properties, file_description, io_ctx, profile), _profile_cache(profile_cache), _global_rowid_context(global_rowid_context), @@ -586,7 +590,10 @@ ParquetReader::ParquetReader(std::shared_ptr& system_p _enable_mapping_varbinary(enable_mapping_varbinary), _preserve_binary_uuid(preserve_binary_uuid), _hive_parquet_time_zone(std::move(hive_parquet_time_zone)), - _encryption_info(std::move(encryption_info)) {} + _encryption_info(std::move(encryption_info)), + _file_context(std::move(file_context)), + _format_split_id(format_split_id), + _format_split_id_end(format_split_id_end) {} ParquetReader::~ParquetReader() { _clear_decryption_parameters(); @@ -644,26 +651,29 @@ Status ParquetReader::init(RuntimeState* state) { _state->scheduler.set_batch_size(_batch_size); // Opening the file parses the footer before any row group can be scheduled. Keep this timer // around the whole operation so footer/cache latency cannot disappear from a slow profile. + Status file_context_status; { SCOPED_TIMER(_parquet_profile.parse_footer_time); - auto footer_status = _state->file_context.open( + file_context_status = _state->file_context.open( _tracing_file_reader, _io_ctx.get(), _state->enable_page_cache, *_file_description, _enable_mapping_timestamp_tz, _enable_mapping_varbinary, _preserve_binary_uuid, - _encryption_info.has_value() ? &*_encryption_info : nullptr); + _encryption_info.has_value() ? &*_encryption_info : nullptr, _file_context); // The file context owns a SecureString after opening; discard the temporary wire copy // on success and failure, before any scan request can retain it. _clear_decryption_parameters(); - RETURN_IF_ERROR(footer_status); - if (_state->file_context.native_metadata->encryption() != nullptr) { - _state->enable_page_cache = false; - } } + // Publish physical-footer outcomes even when opening the file fails so a failing footer path + // remains visible in the profile that must diagnose it. if (_profile != nullptr) { COUNTER_UPDATE(_parquet_profile.file_footer_read_calls, _state->file_context.native_footer_read_calls); COUNTER_UPDATE(_parquet_profile.file_footer_hit_cache, _state->file_context.native_footer_cache_hits); } + RETURN_IF_ERROR(file_context_status); + if (_state->file_context.native_metadata->encryption() != nullptr) { + _state->enable_page_cache = false; + } // Build file schema from parquet metadata. // A file reader may expose raw file identifiers, such as Parquet field_id, through ColumnDefinition::identifier { @@ -684,6 +694,97 @@ Status ParquetReader::init(RuntimeState* state) { return Status::OK(); } +Status ParquetReader::build_physical_splits(std::vector* splits, + bool* was_split) const { + DORIS_CHECK(splits != nullptr); + DORIS_CHECK(was_split != nullptr); + splits->clear(); + *was_split = false; + if (_state == nullptr || _state->file_context.native_metadata == nullptr || + _state->file_context.shared_file_context == nullptr) { + return Status::Uninitialized("ParquetReader is not open"); + } + if (!_state->file_context.shared_file_context->has_stable_identity) { + // A path and size do not identify a mutable remote object. Keep the initialized parent + // reader instead of publishing children whose shared footer could become stale. + return Status::OK(); + } + if (!_state->file_context.can_refine_physical_splits()) { + return Status::OK(); + } + + ParquetScanRange scan_range { + .start_offset = _file_description->range_start_offset, + .size = _file_description->range_size, + .file_size = _file_description->file_size, + }; + std::vector selected_row_groups; + if (_state->scan_plan != nullptr) { + selected_row_groups.reserve(_state->scan_plan->row_groups.size()); + for (const auto& row_group_plan : _state->scan_plan->row_groups) { + selected_row_groups.push_back(row_group_plan.row_group_id); + } + } else { + RETURN_IF_ERROR(detail::select_native_row_groups_by_scan_range( + _state->file_context.native_metadata->to_thrift(), scan_range, + _state->file_context.native_metadata->row_group_first_rows(), + &selected_row_groups)); + } + const auto& metadata = _state->file_context.native_metadata->to_thrift(); + const auto compat = native::parquet_reader_compat( + metadata.__isset.created_by ? metadata.created_by : std::string {}); + const size_t file_size = _state->file_context.native_file->size(); + splits->reserve(selected_row_groups.size()); + for (const int row_group_id : selected_row_groups) { + const auto& row_group = metadata.row_groups[row_group_id]; + if (row_group.num_rows == 0) { + // Empty row groups are valid and the ordinary scan planner ignores them. Refinement + // must preserve that behavior before inspecting their potentially empty chunks. + continue; + } + if (row_group.columns.empty()) { + // A root-only schema can still carry rows for metadata COUNT(*), but it has no byte + // envelope that can identify a child. Keep the initialized source reader so those + // rows are not turned into a corruption error or discarded after tentative children. + splits->clear(); + return Status::OK(); + } + size_t group_start = std::numeric_limits::max(); + size_t group_end = 0; + for (size_t column_id = 0; column_id < row_group.columns.size(); ++column_id) { + const auto& chunk = row_group.columns[column_id]; + if (!chunk.__isset.meta_data) { + // Scheduling envelopes span every chunk, including columns the current request + // does not read. If an unused chunk cannot prove a safe envelope, retain the + // initialized source reader so projected-column validation remains authoritative. + splits->clear(); + return Status::OK(); + } + native::ColumnChunkRange chunk_range; + const auto range_status = native::compute_column_chunk_range( + chunk.meta_data, file_size, compat.parquet_816_padding, &chunk_range); + if (!range_status.ok()) { + splits->clear(); + return Status::OK(); + } + group_start = std::min(group_start, chunk_range.offset); + group_end = std::max(group_end, chunk_range.offset + chunk_range.length); + } + if (group_end <= group_start) { + splits->clear(); + return Status::OK(); + } + PhysicalFileSplit child; + child.start_offset = cast_set(group_start); + child.size = cast_set(group_end - group_start); + child.file_context = _state->file_context.shared_file_context; + child.format_split_id = row_group_id; + splits->push_back(std::move(child)); + } + *was_split = true; + return Status::OK(); +} + void ParquetReader::set_batch_size(size_t batch_size) { _batch_size = std::max(1, batch_size); if (_state != nullptr) { @@ -803,6 +904,8 @@ Status ParquetReader::open(std::shared_ptr request) { scan_range.start_offset = _file_description->range_start_offset; scan_range.size = _file_description->range_size; scan_range.file_size = _file_description->file_size; + scan_range.row_group_id = _format_split_id; + scan_range.row_group_id_end = _format_split_id_end; // Get selected ranges in row groups according to metadata (Row-Group level index and Page Index including Zonemap, Dictionary, Bloom Filter). RETURN_IF_ERROR(plan_parquet_row_groups( *_state->file_context.native_metadata, _state->file_schema, *request_snapshot, @@ -821,6 +924,9 @@ Status ParquetReader::open(std::shared_ptr request) { } _state->scheduler.set_global_rowid_context(_global_rowid_context); _state->scheduler.set_scan_profile(_parquet_profile.scan_profile()); + DORIS_CHECK(_state->file_context.shared_file_context != nullptr); + _state->scheduler.set_adaptive_context( + _state->file_context.shared_file_context->adaptive_scan_context); _state->scheduler.set_plan(std::move(row_group_plan)); _state->scheduler.set_scan_request(request_snapshot); _eof = _state->scheduler.empty(); @@ -1113,6 +1219,25 @@ Status ParquetReader::get_aggregate_result(const format::FileAggregateRequest& r return Status::OK(); } +Status ParquetReader::get_metadata_aggregate_result(const format::FileAggregateRequest& request, + format::FileAggregateResult* result) { + if (request.agg_type == TPushAggOp::type::COUNT && !request.columns.empty()) { + if (request.columns.size() != 1 || _state == nullptr || + _state->file_context.native_metadata == nullptr) { + return Status::NotSupported("Parquet COUNT requires row data"); + } + const auto& projection = request.columns.front().projection; + const auto& root_schema = projected_root_schema(_state->file_schema, projection); + if (root_schema.kind != ParquetColumnSchemaKind::PRIMITIVE || + root_schema.max_definition_level != 0) { + // Nullable and nested COUNT semantics come from definition/repetition levels. Do not + // suppress row-group children unless the already-open reader can stay footer-only. + return Status::NotSupported("Parquet COUNT requires definition levels"); + } + } + return get_aggregate_result(request, result); +} + Status ParquetReader::close() { _clear_decryption_parameters(); SCOPED_TIMER(_parquet_profile.total_time); diff --git a/be/src/format_v2/parquet/parquet_reader.h b/be/src/format_v2/parquet/parquet_reader.h index 603ddf1cbec749..94b4e40cbb3ec5 100644 --- a/be/src/format_v2/parquet/parquet_reader.h +++ b/be/src/format_v2/parquet/parquet_reader.h @@ -50,11 +50,16 @@ class ParquetReader : public format::FileReader { bool enable_mapping_timestamp_tz = false, bool enable_mapping_varbinary = false, std::optional hive_parquet_time_zone = std::nullopt, bool preserve_binary_uuid = false, ParquetProfileCache* profile_cache = nullptr, - std::optional encryption_info = std::nullopt); + std::optional encryption_info = std::nullopt, + std::shared_ptr file_context = nullptr, + int64_t format_split_id = -1, int64_t format_split_id_end = -1); ~ParquetReader() override; Status init(RuntimeState* state) override; + Status build_physical_splits(std::vector* splits, + bool* was_split) const override; + void set_batch_size(size_t batch_size) override; Status get_schema(std::vector* file_schema) const override; @@ -73,6 +78,8 @@ class ParquetReader : public format::FileReader { Status get_aggregate_result(const format::FileAggregateRequest& request, format::FileAggregateResult* result) override; + Status get_metadata_aggregate_result(const format::FileAggregateRequest& request, + format::FileAggregateResult* result) override; void set_condition_cache_context(std::shared_ptr ctx) override; @@ -105,6 +112,9 @@ class ParquetReader : public format::FileReader { // nullopt preserves legacy session conversion; an engaged empty value explicitly disables it. std::optional _hive_parquet_time_zone; std::optional _encryption_info; + std::shared_ptr _file_context; + int64_t _format_split_id = -1; + int64_t _format_split_id_end = -1; }; } // namespace doris::format::parquet diff --git a/be/src/format_v2/parquet/parquet_scan.cpp b/be/src/format_v2/parquet/parquet_scan.cpp index 4ecea4ab08ec60..d7c344dd3d70a0 100644 --- a/be/src/format_v2/parquet/parquet_scan.cpp +++ b/be/src/format_v2/parquet/parquet_scan.cpp @@ -657,12 +657,54 @@ Status select_native_row_groups_by_scan_range(const tparquet::FileMetaData& meta std::vector* row_group_first_rows, std::vector* selected_row_groups) { DORIS_CHECK(row_group_first_rows != nullptr && selected_row_groups != nullptr); - if (scan_range.start_offset < 0 || scan_range.size < -1 || + row_group_first_rows->assign(metadata.row_groups.size(), 0); + int64_t next_first_row = 0; + for (size_t row_group_idx = 0; row_group_idx < metadata.row_groups.size(); ++row_group_idx) { + (*row_group_first_rows)[row_group_idx] = next_first_row; + const auto row_group_rows = metadata.row_groups[row_group_idx].num_rows; + if (row_group_rows < 0) { + return Status::Corruption("Invalid negative row count in parquet row group {}", + row_group_idx); + } + if (row_group_rows > std::numeric_limits::max() - next_first_row) { + return Status::Corruption("Parquet row counts overflow at row group {}", row_group_idx); + } + next_first_row += row_group_rows; + } + return select_native_row_groups_by_scan_range(metadata, scan_range, *row_group_first_rows, + selected_row_groups); +} + +Status select_native_row_groups_by_scan_range(const tparquet::FileMetaData& metadata, + const ParquetScanRange& scan_range, + const std::vector& row_group_first_rows, + std::vector* selected_row_groups) { + DORIS_CHECK(selected_row_groups != nullptr); + DORIS_CHECK(row_group_first_rows.size() == metadata.row_groups.size()); + if (scan_range.start_offset < 0 || scan_range.size < -1 || scan_range.row_group_id < -1 || + scan_range.row_group_id_end < -1 || + scan_range.row_group_id >= cast_set(metadata.row_groups.size()) || + (scan_range.row_group_id_end >= 0 && + (scan_range.row_group_id < 0 || scan_range.row_group_id_end < scan_range.row_group_id || + scan_range.row_group_id_end >= cast_set(metadata.row_groups.size()))) || (scan_range.size >= 0 && scan_range.start_offset > std::numeric_limits::max() - scan_range.size)) { return Status::Corruption("Invalid Parquet scan range [{}, {})", scan_range.start_offset, scan_range.size); } + selected_row_groups->clear(); + if (scan_range.row_group_id >= 0) { + const int64_t row_group_id_end = scan_range.row_group_id_end >= 0 + ? scan_range.row_group_id_end + : scan_range.row_group_id; + selected_row_groups->reserve( + cast_set(row_group_id_end - scan_range.row_group_id + 1)); + for (int64_t row_group_id = scan_range.row_group_id; row_group_id <= row_group_id_end; + ++row_group_id) { + selected_row_groups->push_back(cast_set(row_group_id)); + } + return Status::OK(); + } const uint64_t range_start = static_cast(scan_range.start_offset); const uint64_t range_end = scan_range.size < 0 ? std::numeric_limits::max() @@ -674,21 +716,9 @@ Status select_native_row_groups_by_scan_range(const tparquet::FileMetaData& meta range_end >= static_cast(scan_range.file_size)); const auto compat = native::parquet_reader_compat( metadata.__isset.created_by ? metadata.created_by : std::string {}); - row_group_first_rows->assign(metadata.row_groups.size(), 0); - selected_row_groups->clear(); selected_row_groups->reserve(metadata.row_groups.size()); - int64_t next_first_row = 0; for (size_t row_group_idx = 0; row_group_idx < metadata.row_groups.size(); ++row_group_idx) { - (*row_group_first_rows)[row_group_idx] = next_first_row; const auto& row_group = metadata.row_groups[row_group_idx]; - if (row_group.num_rows < 0) { - return Status::Corruption("Invalid negative row count in parquet row group {}", - row_group_idx); - } - if (row_group.num_rows > std::numeric_limits::max() - next_first_row) { - return Status::Corruption("Parquet row counts overflow at row group {}", row_group_idx); - } - next_first_row += row_group.num_rows; bool selected = full_file_range; if (!full_file_range) { if (row_group.columns.empty()) { @@ -724,6 +754,30 @@ Status select_native_row_groups_by_scan_range(const tparquet::FileMetaData& meta } // namespace detail +detail::ParquetAdaptiveSnapshot ParquetAdaptiveContext::restore(uint64_t predicate_digest) const { + std::scoped_lock lock(_lock); + const auto snapshot = _snapshots.find(predicate_digest); + return snapshot == _snapshots.end() ? detail::ParquetAdaptiveSnapshot {} : snapshot->second; +} + +void ParquetAdaptiveContext::publish(uint64_t predicate_digest, + const detail::ParquetAdaptiveSnapshot& incoming_snapshot) { + std::scoped_lock lock(_lock); + auto& current = _snapshots[predicate_digest]; + for (const auto& [position, incoming_stats] : incoming_snapshot.predicate_runtime_stats) { + auto& current_stats = current.predicate_runtime_stats[position]; + if (incoming_stats.samples >= current_stats.samples) { + current_stats = incoming_stats; + } + } + if (incoming_snapshot.predicate_batch_sequence >= current.predicate_batch_sequence) { + current.predicate_batch_sequence = incoming_snapshot.predicate_batch_sequence; + current.predicate_survival_ratio = incoming_snapshot.predicate_survival_ratio; + } + current.empty_predicate_batch_rows = std::max(current.empty_predicate_batch_rows, + incoming_snapshot.empty_predicate_batch_rows); +} + namespace { std::vector intersect_row_ranges(const std::vector& left, @@ -867,10 +921,10 @@ Status plan_parquet_row_groups(const NativeParquetMetadata& metadata, plan->pruning_stats = {}; plan->requested_leaf_column_ids = request_leaf_column_ids(file_schema, request); plan->enable_bloom_filter = enable_bloom_filter; - std::vector row_group_first_rows; + const auto& row_group_first_rows = metadata.row_group_first_rows(); std::vector scan_range_selected; RETURN_IF_ERROR(detail::select_native_row_groups_by_scan_range( - metadata.to_thrift(), scan_range, &row_group_first_rows, &scan_range_selected)); + metadata.to_thrift(), scan_range, row_group_first_rows, &scan_range_selected)); RETURN_IF_ERROR(build_native_row_group_read_plans(metadata, file_schema, request, scan_range_selected, row_group_first_rows, plan, timezone, runtime_state, file_context)); @@ -1251,6 +1305,7 @@ void ParquetScanScheduler::set_scan_request(std::shared_ptr request) { @@ -1276,9 +1331,35 @@ void ParquetScanScheduler::activate_pending_scan_request_at_row_group_boundary() _predicate_indices_by_position_scratch.clear(); _materialized_predicate_positions_scratch.clear(); _ordered_predicate_positions_scratch.clear(); + _restore_adaptive_state(*_active_request); +} + +void ParquetScanScheduler::_restore_adaptive_state(const format::FileScanRequest& request) { _predicate_runtime_stats.clear(); _predicate_batch_sequence = 0; _predicate_survival_ratio = -1; + _empty_predicate_batch_rows = 0; + if (_adaptive_context == nullptr || !request.predicate_snapshot_digest.has_value()) { + return; + } + const auto snapshot = _adaptive_context->restore(*request.predicate_snapshot_digest); + _predicate_runtime_stats = snapshot.predicate_runtime_stats; + _predicate_batch_sequence = snapshot.predicate_batch_sequence; + _predicate_survival_ratio = snapshot.predicate_survival_ratio; + _empty_predicate_batch_rows = snapshot.empty_predicate_batch_rows; +} + +void ParquetScanScheduler::_publish_adaptive_state(const format::FileScanRequest& request) { + if (_adaptive_context == nullptr || !request.predicate_snapshot_digest.has_value()) { + return; + } + // Concurrent children may publish in either order. The adaptive context keeps only monotonic + // sample progress for this exact predicate digest, so a cold sibling cannot erase warm state. + _adaptive_context->publish(*request.predicate_snapshot_digest, + {.predicate_runtime_stats = _predicate_runtime_stats, + .predicate_survival_ratio = _predicate_survival_ratio, + .predicate_batch_sequence = _predicate_batch_sequence, + .empty_predicate_batch_rows = _empty_predicate_batch_rows}); } void ParquetScanScheduler::reset_current_row_group() { @@ -3077,6 +3158,7 @@ Status ParquetScanScheduler::read_current_row_group_batch( _predicate_survival_ratio = _predicate_survival_ratio < 0 ? batch_survival : 0.25 * batch_survival + 0.75 * _predicate_survival_ratio; + _publish_adaptive_state(request); if (_scan_profile.selected_rows != nullptr) { COUNTER_UPDATE(_scan_profile.selected_rows, selected_rows); } @@ -3290,7 +3372,7 @@ Status ParquetScanScheduler::read_next_batch( *eof = false; return Status::OK(); } - int64_t predicate_batch_rows = _batch_size; + int64_t predicate_batch_rows = std::max(_batch_size, _empty_predicate_batch_rows); const int64_t max_predicate_batch_rows = std::min( std::numeric_limits::max(), std::max(DEFAULT_READ_BATCH_SIZE, _runtime_state == nullptr @@ -3360,6 +3442,8 @@ Status ParquetScanScheduler::read_next_batch( // long empty prefixes cheaply; a later non-empty probe is sliced before lazy columns // are materialized, so this internal width cannot escape the caller's row cap. predicate_batch_rows = grow_empty_predicate_batch(predicate_batch_rows); + _empty_predicate_batch_rows = predicate_batch_rows; + _publish_adaptive_state(*_active_request); continue; } *eof = false; diff --git a/be/src/format_v2/parquet/parquet_scan.h b/be/src/format_v2/parquet/parquet_scan.h index a42203042863d3..c9214c168967f2 100644 --- a/be/src/format_v2/parquet/parquet_scan.h +++ b/be/src/format_v2/parquet/parquet_scan.h @@ -21,6 +21,7 @@ #include #include #include +#include #include #include #include @@ -78,6 +79,13 @@ struct AdaptivePredicateStats { size_t samples = 0; }; +struct ParquetAdaptiveSnapshot { + std::unordered_map predicate_runtime_stats; + double predicate_survival_ratio = -1; + size_t predicate_batch_sequence = 0; + int64_t empty_predicate_batch_rows = 0; +}; + size_t finalize_variant_leaf_projection_for_row_group(const tparquet::RowGroup& row_group, const ParquetColumnSchema& schema, format::LocalColumnIndex* projection, @@ -105,19 +113,38 @@ Status select_native_row_groups_by_scan_range(const tparquet::FileMetaData& meta const ParquetScanRange& scan_range, std::vector* row_group_first_rows, std::vector* selected_row_groups); +Status select_native_row_groups_by_scan_range(const tparquet::FileMetaData& metadata, + const ParquetScanRange& scan_range, + const std::vector& row_group_first_rows, + std::vector* selected_row_groups); #ifdef BE_TEST void reset_physical_leaf_set_build_count(); size_t physical_leaf_set_build_count(); #endif } // namespace detail +// Children of one parent split share this context. Snapshots are keyed by predicate digest because +// late runtime filters can change both order and selectivity. +class ParquetAdaptiveContext final { +public: + detail::ParquetAdaptiveSnapshot restore(uint64_t predicate_digest) const; + void publish(uint64_t predicate_digest, + const detail::ParquetAdaptiveSnapshot& incoming_snapshot); + +private: + mutable std::mutex _lock; + std::unordered_map _snapshots; +}; + // ============================================================================ // ============================================================================ struct ParquetScanRange { int64_t start_offset = 0; - int64_t size = -1; // -1 means read the whole file - int64_t file_size = -1; // -1 means unknown + int64_t size = -1; // -1 means read the whole file + int64_t file_size = -1; // -1 means unknown + int64_t row_group_id = -1; // First exact row group selected by a BE-generated split. + int64_t row_group_id_end = -1; // Inclusive end for a coalesced consecutive row-group range. }; struct RowGroupReadPlan { @@ -217,6 +244,9 @@ class ParquetScanScheduler { void set_runtime_state(RuntimeState* runtime_state) { _runtime_state = runtime_state; } void set_scan_request(std::shared_ptr request); void queue_scan_request(std::shared_ptr request); + void set_adaptive_context(std::shared_ptr adaptive_context) { + _adaptive_context = std::move(adaptive_context); + } // Release row-group readers before the owning RuntimeProfile is reported. Native readers // publish their accumulated page/decode statistics from their destructor. void close() { reset_current_row_group(); } @@ -244,6 +274,8 @@ class ParquetScanScheduler { const std::vector>& file_schema, const format::FileScanRequest& request); void activate_pending_scan_request_at_row_group_boundary(); + void _restore_adaptive_state(const format::FileScanRequest& request); + void _publish_adaptive_state(const format::FileScanRequest& request); void flush_current_reader_profiles(); bool finish_current_reader_batch_profiles(); const detail::PredicateConjunctSchedule& predicate_conjunct_schedule( @@ -370,6 +402,8 @@ class ParquetScanScheduler { size_t _batches_since_profile_flush = 0; std::unordered_map _predicate_runtime_stats; double _predicate_survival_ratio = -1; + int64_t _empty_predicate_batch_rows = 0; + std::shared_ptr _adaptive_context; std::shared_ptr _condition_cache_ctx; int64_t _condition_cache_filtered_rows = 0; int64_t _predicate_filtered_rows = 0; diff --git a/be/src/format_v2/table/fluss_hybrid_reader.cpp b/be/src/format_v2/table/fluss_hybrid_reader.cpp index 5bd6a9a062032d..acd7ef899a8cb2 100644 --- a/be/src/format_v2/table/fluss_hybrid_reader.cpp +++ b/be/src/format_v2/table/fluss_hybrid_reader.cpp @@ -81,8 +81,11 @@ Status FlussHybridReader::prepare_split(const format::SplitReadOptions& options) return _current_split_reader->prepare_split(options); } -Status FlussHybridReader::refresh_conjuncts(VExprContextSPtrs conjuncts) { - RETURN_IF_ERROR(format::TableReader::refresh_conjuncts(std::move(conjuncts))); +Status FlussHybridReader::refresh_conjuncts(VExprContextSPtrs conjuncts, + std::optional condition_cache_digest, + bool all_runtime_filters_applied) { + RETURN_IF_ERROR(format::TableReader::refresh_conjuncts( + std::move(conjuncts), condition_cache_digest, all_runtime_filters_applied)); if (_current_split_reader == nullptr) { return Status::OK(); } @@ -94,7 +97,8 @@ Status FlussHybridReader::refresh_conjuncts(VExprContextSPtrs conjuncts) { RETURN_IF_ERROR(_clone_conjuncts(&child_conjuncts)); // The hybrid wrapper owns no physical reader; forward a clone so the active child, rather than // only the wrapper snapshot, observes late predicates for the remainder of this split. - return _current_split_reader->refresh_conjuncts(std::move(child_conjuncts)); + return _current_split_reader->refresh_conjuncts( + std::move(child_conjuncts), condition_cache_digest, all_runtime_filters_applied); } Status FlussHybridReader::get_block(Block* block, bool* eos) { diff --git a/be/src/format_v2/table/fluss_hybrid_reader.h b/be/src/format_v2/table/fluss_hybrid_reader.h index 070d435198336e..095050a5d4359d 100644 --- a/be/src/format_v2/table/fluss_hybrid_reader.h +++ b/be/src/format_v2/table/fluss_hybrid_reader.h @@ -53,7 +53,9 @@ class FlussHybridReader final : public format::TableReader { Status init(format::TableReadOptions&& options) override; Status prepare_split(const format::SplitReadOptions& options) override; - Status refresh_conjuncts(VExprContextSPtrs conjuncts) override; + Status refresh_conjuncts(VExprContextSPtrs conjuncts, + std::optional condition_cache_digest = std::nullopt, + bool all_runtime_filters_applied = false) override; Status get_block(Block* block, bool* eos) override; bool current_split_pruned() const override; bool current_split_uses_metadata_count() const override; diff --git a/be/src/format_v2/table/fluss_union_lake_reader.cpp b/be/src/format_v2/table/fluss_union_lake_reader.cpp index 4eaa81a8eaedea..ba303c6dc4c327 100644 --- a/be/src/format_v2/table/fluss_union_lake_reader.cpp +++ b/be/src/format_v2/table/fluss_union_lake_reader.cpp @@ -482,14 +482,18 @@ Status FlussUnionLakeReader::prepare_split(const format::SplitReadOptions& optio return _prepare_suppression(options); } -Status FlussUnionLakeReader::refresh_conjuncts(VExprContextSPtrs conjuncts) { - RETURN_IF_ERROR(format::TableReader::refresh_conjuncts(std::move(conjuncts))); +Status FlussUnionLakeReader::refresh_conjuncts(VExprContextSPtrs conjuncts, + std::optional condition_cache_digest, + bool all_runtime_filters_applied) { + RETURN_IF_ERROR(format::TableReader::refresh_conjuncts( + std::move(conjuncts), condition_cache_digest, all_runtime_filters_applied)); if (_lake_reader == nullptr) { return Status::OK(); } VExprContextSPtrs child_conjuncts; RETURN_IF_ERROR(_clone_conjuncts(&child_conjuncts)); - return _lake_reader->refresh_conjuncts(std::move(child_conjuncts)); + return _lake_reader->refresh_conjuncts(std::move(child_conjuncts), condition_cache_digest, + all_runtime_filters_applied); } Status FlussUnionLakeReader::_prepare_suppression(const format::SplitReadOptions& options) { diff --git a/be/src/format_v2/table/fluss_union_lake_reader.h b/be/src/format_v2/table/fluss_union_lake_reader.h index c1d031bbb447ee..bdd95223daf229 100644 --- a/be/src/format_v2/table/fluss_union_lake_reader.h +++ b/be/src/format_v2/table/fluss_union_lake_reader.h @@ -144,7 +144,9 @@ class FlussUnionLakeReader final : public format::TableReader { Status init(format::TableReadOptions&& options) override; Status prepare_split(const format::SplitReadOptions& options) override; - Status refresh_conjuncts(VExprContextSPtrs conjuncts) override; + Status refresh_conjuncts(VExprContextSPtrs conjuncts, + std::optional condition_cache_digest = std::nullopt, + bool all_runtime_filters_applied = false) override; Status get_block(Block* block, bool* eos) override; bool current_split_pruned() const override; bool current_split_uses_metadata_count() const override; diff --git a/be/src/format_v2/table/hudi_reader.cpp b/be/src/format_v2/table/hudi_reader.cpp index 44d7f8818418ce..e4489a7f5a7552 100644 --- a/be/src/format_v2/table/hudi_reader.cpp +++ b/be/src/format_v2/table/hudi_reader.cpp @@ -81,7 +81,9 @@ Status HudiReader::create_file_reader(std::unique_ptr* reade *reader = std::make_unique( _system_properties, _current_task->data_file, _io_ctx, _scanner_profile, _global_rowid_context, enable_mapping_timestamp_tz, enable_mapping_varbinary, - parquet_int96_time_zone(), false, _parquet_profile_cache); + parquet_int96_time_zone(), false, _parquet_profile_cache, std::nullopt, + _current_task->file_context, _current_task->format_split_id, + _current_task->format_split_id_end); return Status::OK(); } @@ -112,8 +114,18 @@ Status HudiHybridReader::prepare_split(const format::SplitReadOptions& options) return _current_split_reader->prepare_split(options); } -Status HudiHybridReader::refresh_conjuncts(VExprContextSPtrs conjuncts) { - RETURN_IF_ERROR(format::TableReader::refresh_conjuncts(std::move(conjuncts))); +Status HudiHybridReader::build_physical_splits(const FileScanSplit& source_split, + std::vector* splits, + bool* was_split) { + DORIS_CHECK(_current_split_reader != nullptr); + return _current_split_reader->build_physical_splits(source_split, splits, was_split); +} + +Status HudiHybridReader::refresh_conjuncts(VExprContextSPtrs conjuncts, + std::optional condition_cache_digest, + bool all_runtime_filters_applied) { + RETURN_IF_ERROR(format::TableReader::refresh_conjuncts( + std::move(conjuncts), condition_cache_digest, all_runtime_filters_applied)); if (_current_split_reader == nullptr) { return Status::OK(); } @@ -121,7 +133,8 @@ Status HudiHybridReader::refresh_conjuncts(VExprContextSPtrs conjuncts) { RETURN_IF_ERROR(_clone_conjuncts(&child_conjuncts)); // The hybrid wrapper owns no physical reader; forward a clone so the active child, rather than // only the wrapper snapshot, observes late predicates for the remainder of this split. - return _current_split_reader->refresh_conjuncts(std::move(child_conjuncts)); + return _current_split_reader->refresh_conjuncts( + std::move(child_conjuncts), condition_cache_digest, all_runtime_filters_applied); } Status HudiHybridReader::get_block(Block* block, bool* eos) { diff --git a/be/src/format_v2/table/hudi_reader.h b/be/src/format_v2/table/hudi_reader.h index e562f36ea209bb..d46e6fa43c2e65 100644 --- a/be/src/format_v2/table/hudi_reader.h +++ b/be/src/format_v2/table/hudi_reader.h @@ -62,7 +62,11 @@ class HudiHybridReader final : public format::TableReader { Status init(format::TableReadOptions&& options) override; Status prepare_split(const format::SplitReadOptions& options) override; - Status refresh_conjuncts(VExprContextSPtrs conjuncts) override; + Status build_physical_splits(const FileScanSplit& source_split, + std::vector* splits, bool* was_split) override; + Status refresh_conjuncts(VExprContextSPtrs conjuncts, + std::optional condition_cache_digest = std::nullopt, + bool all_runtime_filters_applied = false) override; Status get_block(Block* block, bool* eos) override; bool current_split_pruned() const override; bool current_split_uses_metadata_count() const override; diff --git a/be/src/format_v2/table/load_reader.cpp b/be/src/format_v2/table/load_reader.cpp index 83896dc8caa5ca..27231155a9e25e 100644 --- a/be/src/format_v2/table/load_reader.cpp +++ b/be/src/format_v2/table/load_reader.cpp @@ -186,7 +186,7 @@ Status LoadTableReader::prepare_split(const SplitReadOptions& options) { return TableReader::prepare_split(source_options); } -Status LoadTableReader::refresh_conjuncts(VExprContextSPtrs) { +Status LoadTableReader::refresh_conjuncts(VExprContextSPtrs, std::optional, bool) { return Status::OK(); } diff --git a/be/src/format_v2/table/load_reader.h b/be/src/format_v2/table/load_reader.h index 73c9b1944511ec..f2098e046929b3 100644 --- a/be/src/format_v2/table/load_reader.h +++ b/be/src/format_v2/table/load_reader.h @@ -30,7 +30,9 @@ class LoadTableReader : public TableReader { Status init(TableReadOptions&& options) override; Status prepare_split(const SplitReadOptions& options) override; - Status refresh_conjuncts(VExprContextSPtrs conjuncts) override; + Status refresh_conjuncts(VExprContextSPtrs conjuncts, + std::optional condition_cache_digest = std::nullopt, + bool all_runtime_filters_applied = false) override; bool current_split_uses_metadata_count() const override { return false; } Status abort_split() override; Status close() override; diff --git a/be/src/format_v2/table/paimon_reader.cpp b/be/src/format_v2/table/paimon_reader.cpp index 408da1286698b6..d8e0ace378ff0b 100644 --- a/be/src/format_v2/table/paimon_reader.cpp +++ b/be/src/format_v2/table/paimon_reader.cpp @@ -400,8 +400,18 @@ Status PaimonHybridReader::prepare_split(const format::SplitReadOptions& options return _current_split_reader->prepare_split(native_options); } -Status PaimonHybridReader::refresh_conjuncts(VExprContextSPtrs conjuncts) { - RETURN_IF_ERROR(format::TableReader::refresh_conjuncts(std::move(conjuncts))); +Status PaimonHybridReader::build_physical_splits(const FileScanSplit& source_split, + std::vector* splits, + bool* was_split) { + DORIS_CHECK(_current_split_reader != nullptr); + return _current_split_reader->build_physical_splits(source_split, splits, was_split); +} + +Status PaimonHybridReader::refresh_conjuncts(VExprContextSPtrs conjuncts, + std::optional condition_cache_digest, + bool all_runtime_filters_applied) { + RETURN_IF_ERROR(format::TableReader::refresh_conjuncts( + std::move(conjuncts), condition_cache_digest, all_runtime_filters_applied)); if (_current_split_reader == nullptr) { return Status::OK(); } @@ -409,7 +419,8 @@ Status PaimonHybridReader::refresh_conjuncts(VExprContextSPtrs conjuncts) { RETURN_IF_ERROR(_clone_conjuncts(&child_conjuncts)); // The hybrid wrapper owns no physical reader; forward a clone so the active child, rather than // only the wrapper snapshot, observes late predicates for the remainder of this split. - return _current_split_reader->refresh_conjuncts(std::move(child_conjuncts)); + return _current_split_reader->refresh_conjuncts( + std::move(child_conjuncts), condition_cache_digest, all_runtime_filters_applied); } Status PaimonHybridReader::get_block(Block* block, bool* eos) { diff --git a/be/src/format_v2/table/paimon_reader.h b/be/src/format_v2/table/paimon_reader.h index 65d0a8d1d50f24..db1afdac4107dc 100644 --- a/be/src/format_v2/table/paimon_reader.h +++ b/be/src/format_v2/table/paimon_reader.h @@ -87,7 +87,11 @@ class PaimonHybridReader final : public format::TableReader { Status init(format::TableReadOptions&& options) override; Status prepare_split(const format::SplitReadOptions& options) override; - Status refresh_conjuncts(VExprContextSPtrs conjuncts) override; + Status build_physical_splits(const FileScanSplit& source_split, + std::vector* splits, bool* was_split) override; + Status refresh_conjuncts(VExprContextSPtrs conjuncts, + std::optional condition_cache_digest = std::nullopt, + bool all_runtime_filters_applied = false) override; Status get_block(Block* block, bool* eos) override; bool current_split_pruned() const override; bool current_split_uses_metadata_count() const override; diff --git a/be/src/format_v2/table_reader.cpp b/be/src/format_v2/table_reader.cpp index 49921d7b702f6c..9877935c9c9984 100644 --- a/be/src/format_v2/table_reader.cpp +++ b/be/src/format_v2/table_reader.cpp @@ -22,6 +22,7 @@ #include #include +#include #include #include #include @@ -81,6 +82,103 @@ std::optional TableReader::_get_int96_timezone_override( namespace { +std::optional build_predicate_snapshot_digest(const VExprContextSPtrs& conjuncts) { + // Adaptive state must remain independent of the optional Condition Cache seed. A zero result + // means an expression cannot provide a stable semantic digest, so sharing is disabled. + uint64_t digest = 0xcbf29ce484222325ULL; + for (const auto& conjunct : conjuncts) { + digest = conjunct->get_digest(digest); + if (digest == 0) { + return std::nullopt; + } + } + return digest; +} + +void extend_format_split_id_range(PhysicalFileSplit* destination, const PhysicalFileSplit& source) { + DORIS_CHECK(destination != nullptr); + DORIS_CHECK(destination->format_split_id >= 0 && source.format_split_id >= 0); + const int64_t destination_end = destination->format_split_id_end >= 0 + ? destination->format_split_id_end + : destination->format_split_id; + const int64_t source_end = + source.format_split_id_end >= 0 ? source.format_split_id_end : source.format_split_id; + DORIS_CHECK(destination_end < std::numeric_limits::max()); + DORIS_CHECK(source.format_split_id == destination_end + 1); + destination->format_split_id_end = source_end; +} + +std::vector coalesce_physical_splits(std::vector splits, + int64_t target_split_size) { + if (target_split_size <= 0 || splits.size() < 2) { + return splits; + } + + const uint64_t target_size = static_cast(target_split_size); + std::vector merged; + merged.reserve(splits.size()); + auto current = std::move(splits.front()); + for (size_t index = 1; index < splits.size(); ++index) { + auto& next = splits[index]; + const bool valid_ranges = current.start_offset >= 0 && current.size >= 0 && + next.start_offset >= 0 && next.size >= 0; + const uint64_t current_start = + valid_ranges ? static_cast(current.start_offset) : 0; + const uint64_t current_end = valid_ranges ? current_start + current.size : 0; + const uint64_t next_start = valid_ranges ? static_cast(next.start_offset) : 0; + const uint64_t next_end = valid_ranges ? next_start + next.size : 0; + const int64_t current_id_end = current.format_split_id_end >= 0 + ? current.format_split_id_end + : current.format_split_id; + const bool consecutive_ids = current_id_end >= 0 && + current_id_end < std::numeric_limits::max() && + next.format_split_id == current_id_end + 1; + // Coalescing changes only scheduling granularity. Exact format ids remain attached so byte + // padding or skipped physical granules cannot change which rows a child owns. + const bool fits_target = valid_ranges && current.file_context == next.file_context && + consecutive_ids && next_start >= current_start && + next_end >= current_end && next_end - current_start <= target_size; + if (fits_target) { + current.size = cast_set(next_end - current_start); + extend_format_split_id_range(¤t, next); + } else { + merged.push_back(std::move(current)); + current = std::move(next); + } + } + merged.push_back(std::move(current)); + + if (merged.size() > 1) { + auto& previous = merged[merged.size() - 2]; + auto& tail = merged.back(); + const bool valid_ranges = previous.start_offset >= 0 && previous.size >= 0 && + tail.start_offset >= 0 && tail.size >= 0; + const uint64_t previous_start = + valid_ranges ? static_cast(previous.start_offset) : 0; + const uint64_t previous_end = valid_ranges ? previous_start + previous.size : 0; + const uint64_t tail_start = valid_ranges ? static_cast(tail.start_offset) : 0; + const uint64_t tail_end = valid_ranges ? tail_start + tail.size : 0; + const uint64_t combined_size = valid_ranges ? tail_end - previous_start : 0; + const int64_t previous_id_end = previous.format_split_id_end >= 0 + ? previous.format_split_id_end + : previous.format_split_id; + const bool consecutive_ids = previous_id_end >= 0 && + previous_id_end < std::numeric_limits::max() && + tail.format_split_id == previous_id_end + 1; + // Avoid a tiny final task only when the ownership envelopes touch. A pruned gap must not + // inflate its predecessor far beyond the FE target. + if (valid_ranges && previous.file_context == tail.file_context && consecutive_ids && + previous.size <= target_split_size && tail.size <= (target_split_size - 1) / 2 && + tail_start <= previous_end && tail_end >= previous_end && + combined_size <= static_cast(std::numeric_limits::max())) { + previous.size = static_cast(combined_size); + extend_format_split_id_range(&previous, tail); + merged.pop_back(); + } + } + return merged; +} + template std::string join_table_reader_debug_strings(const std::vector& values, Formatter formatter) { std::ostringstream out; @@ -986,6 +1084,7 @@ Status TableReader::init(TableReadOptions&& options) { _system_properties = create_system_properties(_scan_params); _mapper_options.mode = TableColumnMappingMode::BY_NAME; _conjuncts = std::move(options.conjuncts); + _predicate_snapshot_digest = build_predicate_snapshot_digest(_conjuncts); return Status::OK(); } @@ -1076,10 +1175,27 @@ bool same_physical_scan_layout(const FileScanRequest& lhs, const FileScanRequest } // namespace -Status TableReader::refresh_conjuncts(VExprContextSPtrs conjuncts) { +Status TableReader::refresh_conjuncts(VExprContextSPtrs conjuncts, + std::optional condition_cache_digest, + bool all_runtime_filters_applied) { SCOPED_TIMER(_profile.total_timer); SCOPED_TIMER(_profile.refresh_conjuncts_timer); _conjuncts = std::move(conjuncts); + _predicate_snapshot_digest = build_predicate_snapshot_digest(_conjuncts); + // A prepared footer result belongs to the prior immutable predicate snapshot. Discard it + // before a refresh can make the planning reader resume ordinary row production. + _metadata_aggregate_result.reset(); + if (all_runtime_filters_applied) { + // A refresh can prove the last pending RF has arrived, but a later partial refresh must + // never make the same split incomplete again. + _all_runtime_filters_applied_for_split = true; + } + if (condition_cache_digest.has_value()) { + // A runtime filter can arrive after a physical child is prepared but before its reader is + // created. Keep that child's cache key tied to the same refreshed predicate snapshot. + _condition_cache_digest = *condition_cache_digest; + _condition_cache_digest_covers_current_split = true; + } if (_data_reader.reader == nullptr) { // The split is prepared but its physical reader has not opened yet. open_reader() will use // this newest snapshot directly, so no pending request is needed. @@ -1104,12 +1220,15 @@ Status TableReader::refresh_conjuncts(VExprContextSPtrs conjuncts) { _file_scan_request == nullptr ? nullptr : &_file_scan_request->non_predicate_positions)); refreshed_request->row_ids = _row_ids; + refreshed_request->predicate_snapshot_digest = _predicate_snapshot_digest; // A refresh does not prove that every future runtime filter has arrived. Keep carrier values // available whenever the split started with pending filters. if (_push_down_agg_type == TPushAggOp::type::COUNT && _push_down_count_columns.has_value() && _push_down_count_columns->empty() && _all_runtime_filters_applied_for_split) { for (const auto& column : refreshed_request->non_predicate_columns) { - refreshed_request->count_star_placeholder_columns.push_back(column.column_id()); + if (!refreshed_request->is_residual_predicate_column(column.column_id())) { + refreshed_request->count_star_placeholder_columns.push_back(column.column_id()); + } } } RETURN_IF_ERROR(customize_file_scan_request(refreshed_request.get())); @@ -1127,6 +1246,7 @@ Status TableReader::refresh_conjuncts(VExprContextSPtrs conjuncts) { if (_condition_cache_ctx != nullptr && !_condition_cache_ctx->is_hit) { // Rows before and after a late RF were evaluated by different predicate snapshots. Such a // partial MISS bitmap must never be published under either snapshot's cache key. + _condition_cache_split_invalid = _condition_cache_split_participating; _condition_cache = nullptr; _condition_cache_ctx = nullptr; _data_reader.reader->set_condition_cache_context(nullptr); @@ -1192,11 +1312,13 @@ Status TableReader::_init_reader_condition_cache(const FileScanRequest& file_req _condition_cache = nullptr; _condition_cache_ctx = nullptr; if (!_should_enable_condition_cache(file_request)) { + _condition_cache_split_invalid = _condition_cache_split_participating; return Status::OK(); } auto* cache = segment_v2::ConditionCache::instance(); if (cache == nullptr) { + _condition_cache_split_invalid = _condition_cache_split_participating; return Status::OK(); } const auto& file = *_current_file_description; @@ -1214,10 +1336,16 @@ Status TableReader::_init_reader_condition_cache(const FileScanRequest& file_req HashUtil::xxHash64WithSeed(timezone->data(), timezone->size(), cache_digest); } } + const auto cache_start = _condition_cache_source_range.has_value() + ? _condition_cache_source_range->first + : file.range_start_offset; + const auto cache_size = _condition_cache_source_range.has_value() + ? _condition_cache_source_range->second + : file.range_size; _condition_cache_key = segment_v2::ConditionCache::ExternalCacheKey( - file.path, file.mtime, file.file_size, cache_digest, file.range_start_offset, - file.range_size, + file.path, file.mtime, file.file_size, cache_digest, cache_start, cache_size, segment_v2::ConditionCache::ExternalCacheKey::BASE_GRANULE_AWARE_VERSION); + _condition_cache_initialized = true; segment_v2::ConditionCacheHandle handle; const bool condition_cache_hit = cache->lookup(_condition_cache_key, &handle); @@ -1251,6 +1379,91 @@ Status TableReader::_init_reader_condition_cache(const FileScanRequest& file_req } void TableReader::_finalize_reader_condition_cache() { + if (_condition_cache_split_participating) { + DORIS_CHECK(_condition_cache_split_context != nullptr); + const bool cache_hit = _condition_cache_ctx != nullptr && _condition_cache_ctx->is_hit; + const bool complete_miss = _condition_cache_initialized && + !_condition_cache_split_invalid && _current_reader_reached_eof; + std::shared_ptr> published_filter; + int64_t published_base_granule = 0; + { + std::scoped_lock lock(_condition_cache_split_context->lock); + auto& split_context = *_condition_cache_split_context; + if (!_condition_cache_initialized) { + split_context.valid = false; + } else { + const auto encoded_key = _condition_cache_key.encode(); + if (!split_context.encoded_key.has_value()) { + split_context.encoded_key = encoded_key; + } else if (*split_context.encoded_key != encoded_key) { + // Children can observe different late-RF snapshots. Never combine their + // partial bitmaps under either digest's source-level cache key. + split_context.valid = false; + } + } + if (cache_hit) { + split_context.cache_hit_seen = true; + } else if (!complete_miss) { + split_context.valid = false; + } else if (_condition_cache_ctx != nullptr && _condition_cache != nullptr) { + DORIS_CHECK(_condition_cache_ctx->num_granules <= _condition_cache->size()); + const int64_t local_base = _condition_cache_ctx->base_granule; + const size_t local_size = _condition_cache_ctx->num_granules; + if (split_context.merged_filter_result.empty()) { + split_context.base_granule = local_base; + split_context.merged_filter_result.assign(local_size, false); + } else { + const int64_t merged_end = + split_context.base_granule + + cast_set(split_context.merged_filter_result.size()); + const int64_t local_end = local_base + cast_set(local_size); + const int64_t combined_base = std::min(split_context.base_granule, local_base); + const int64_t combined_end = std::max(merged_end, local_end); + if (combined_base != split_context.base_granule || combined_end != merged_end) { + std::vector combined(cast_set(combined_end - combined_base), + false); + for (size_t index = 0; index < split_context.merged_filter_result.size(); + ++index) { + combined[cast_set(split_context.base_granule - combined_base) + + index] = split_context.merged_filter_result[index]; + } + split_context.base_granule = combined_base; + split_context.merged_filter_result = std::move(combined); + } + } + for (size_t index = 0; index < local_size; ++index) { + const auto merged_index = + cast_set(local_base - split_context.base_granule) + index; + split_context.merged_filter_result[merged_index] = + split_context.merged_filter_result[merged_index] || + (*_condition_cache)[index]; + } + } + + DORIS_CHECK(split_context.remaining_children > 0); + --split_context.remaining_children; + if (split_context.remaining_children == 0 && split_context.valid && + !split_context.cache_hit_seen && !split_context.merged_filter_result.empty()) { + // A source-level entry is visible only after every physical child reaches EOF; + // publishing an earlier partial MISS would let a sibling HIT skip valid rows. + published_base_granule = split_context.base_granule; + published_filter = std::make_shared>( + std::move(split_context.merged_filter_result)); + } + } + if (published_filter != nullptr) { + if (auto* cache = segment_v2::ConditionCache::instance(); cache != nullptr) { + cache->insert(_condition_cache_key, std::move(published_filter), + published_base_granule); + } + } + _condition_cache = nullptr; + _condition_cache_ctx = nullptr; + _condition_cache_split_context.reset(); + _condition_cache_split_participating = false; + _condition_cache_initialized = false; + return; + } if (_condition_cache_ctx == nullptr || _condition_cache_ctx->is_hit) { _condition_cache = nullptr; _condition_cache_ctx = nullptr; @@ -1338,13 +1551,16 @@ Status TableReader::create_file_reader(std::unique_ptr* reader) { _system_properties, _current_task->data_file, _io_ctx, _scanner_profile, _global_rowid_context, enable_mapping_timestamp_tz, enable_mapping_varbinary, hive_parquet_time_zone, preserve_binary_uuid(), _parquet_profile_cache, - encryption == nullptr ? std::nullopt : std::optional(*encryption)); + encryption == nullptr ? std::nullopt : std::optional(*encryption), + _current_task->file_context, _current_task->format_split_id, + _current_task->format_split_id_end); return Status::OK(); } if (_format == FileFormat::ORC) { *reader = std::make_unique( _system_properties, _current_task->data_file, _io_ctx, _scanner_profile, - _global_rowid_context, enable_mapping_timestamp_tz, enable_mapping_varbinary); + _global_rowid_context, enable_mapping_timestamp_tz, enable_mapping_varbinary, + _current_task->file_context); return Status::OK(); } if (_format == FileFormat::CSV) { @@ -1408,6 +1624,11 @@ Status TableReader::prepare_split(const SplitReadOptions& options) { SCOPED_TIMER(_profile.prepare_split_timer); _current_split_pruned = false; _all_runtime_filters_applied_for_split = options.all_runtime_filters_applied; + _condition_cache_source_range = options.condition_cache_source_range; + _condition_cache_split_context = options.condition_cache_split_context; + _condition_cache_split_participating = _condition_cache_split_context != nullptr; + _condition_cache_split_invalid = false; + _condition_cache_initialized = false; _condition_cache_digest_covers_current_split = options.condition_cache_digest.has_value(); if (options.condition_cache_digest.has_value()) { // The split snapshot may include RFs that arrived after TableReader::init(). Use the digest @@ -1421,6 +1642,7 @@ Status TableReader::prepare_split(const SplitReadOptions& options) { } if (options.conjuncts.has_value()) { _conjuncts = *options.conjuncts; + _predicate_snapshot_digest = build_predicate_snapshot_digest(_conjuncts); } // Update to current split format to handle ORC/PARQUET files in one table. _format = options.current_split_format; @@ -1439,6 +1661,7 @@ Status TableReader::prepare_split(const SplitReadOptions& options) { _delete_rows = nullptr; _deletion_vector = nullptr; _aggregate_pushdown_tried = false; + _metadata_aggregate_result.reset(); _remaining_table_level_count = -1; _remaining_file_level_count = -1; _current_split_uses_metadata_count = false; @@ -1451,6 +1674,9 @@ Status TableReader::prepare_split(const SplitReadOptions& options) { } _current_task = std::make_unique(); _current_task->data_file = create_file_description(options.current_range); + _current_task->file_context = options.file_context; + _current_task->format_split_id = options.format_split_id; + _current_task->format_split_id_end = options.format_split_id_end; _current_file_description = *_current_task->data_file; // A table-level row count is only equivalent to scanning the split when no row predicate is // active and no predicate can arrive later. The metadata path can return several batches for @@ -1476,6 +1702,137 @@ Status TableReader::prepare_split(const SplitReadOptions& options) { return _parse_delete_predicates(options); } +Status TableReader::build_physical_splits(const FileScanSplit& source_split, + std::vector* splits, bool* was_split) { + SCOPED_TIMER(_profile.total_timer); + DORIS_CHECK(splits != nullptr); + DORIS_CHECK(was_split != nullptr); + splits->clear(); + *was_split = false; + if ((_format != FileFormat::PARQUET && _format != FileFormat::ORC) || _current_split_pruned || + _current_split_uses_metadata_count || _current_task == nullptr) { + return Status::OK(); + } + SCOPED_TIMER(_profile.create_reader_timer); + + std::unique_ptr reader; + RETURN_IF_ERROR(create_file_reader(&reader)); + DORIS_CHECK(reader != nullptr); + auto close_planning_reader = [&]() { + SCOPED_TIMER(_profile.file_reader_total_timer); + SCOPED_TIMER(_profile.file_reader_close_timer); + return reader->close(); + }; + Status init_status; + { + SCOPED_TIMER(_profile.file_reader_total_timer); + SCOPED_TIMER(_profile.file_reader_init_timer); + init_status = reader->init(_runtime_state); + } + if (!init_status.ok()) { + // A failed init may still own partially opened resources. Close it through the same + // lifecycle path while preserving the initialization error returned to the scanner. + static_cast(close_planning_reader()); + return init_status; + } + + _data_reader.reader = std::move(reader); + if (_batch_size > 0) { + _data_reader.reader->set_batch_size(_batch_size); + } + const auto open_status = open_reader(); + if (!open_status.ok()) { + if (_data_reader.reader != nullptr) { + static_cast(close_current_reader()); + } + return open_status; + } + if (_data_reader.reader == nullptr) { + // Constant pruning can close the eagerly opened planning reader. Publish an empty refined + // source so scanners do not recreate and reopen the same already-rejected file. + *was_split = true; + return Status::OK(); + } + + if (_supports_aggregate_pushdown(_push_down_agg_type)) { + FileAggregateRequest aggregate_request; + const auto request_status = + _build_file_aggregate_request(_push_down_agg_type, &aggregate_request); + if (!request_status.ok()) { + static_cast(close_current_reader()); + return request_status; + } + FileAggregateResult aggregate_result; + Status aggregate_status; + { + SCOPED_TIMER(_profile.file_reader_total_timer); + SCOPED_TIMER(_profile.file_reader_aggregate_timer); + aggregate_status = _data_reader.reader->get_metadata_aggregate_result( + aggregate_request, &aggregate_result); + } + if (aggregate_status.ok()) { + // The planning reader already owns the exact pruned physical-granule set. Retaining it + // avoids replacing one metadata-only aggregate with N children that repeat the + // reduction. + _metadata_aggregate_result = std::move(aggregate_result); + return Status::OK(); + } + if (!aggregate_status.is()) { + static_cast(close_current_reader()); + return aggregate_status; + } + } + + std::vector physical_splits; + Status status; + { + SCOPED_TIMER(_profile.file_reader_total_timer); + status = _data_reader.reader->build_physical_splits(&physical_splits, was_split); + } + if (!status.ok()) { + static_cast(close_current_reader()); + return status; + } + const auto& source_range = source_split.source_identity_range(); + if (source_range.__isset.target_split_size && source_range.target_split_size > 0) { + physical_splits = coalesce_physical_splits(std::move(physical_splits), + source_range.target_split_size); + } + if (!*was_split || physical_splits.size() == 1) { + // Reuse the fully planned reader when refinement is unnecessary. Besides avoiding another + // footer parse, this preserves the request snapshot whose metadata pruning selected the + // single surviving physical granule. + splits->clear(); + *was_split = false; + return Status::OK(); + } + // FileReader descriptors deliberately carry no scanner/table policy. Attach the FE source + // identity and shared child coordination here so format readers cannot accidentally own range + // progress, table metadata semantics, or Condition Cache publication. + const auto shared_source_range = std::make_shared(source_split.range); + std::shared_ptr condition_cache_split_context; + if (physical_splits.size() > 1) { + condition_cache_split_context = + std::make_shared(physical_splits.size()); + } + splits->reserve(physical_splits.size()); + for (auto& physical_split : physical_splits) { + FileScanSplit child; + child.source_range = shared_source_range; + child.start_offset = physical_split.start_offset; + child.size = physical_split.size; + // A source-level count is not valid for one generated physical child. Child readers can + // still derive an exact count from shared format metadata when pushdown is eligible. + child.clear_table_level_row_count = true; + child.file_context = std::move(physical_split.file_context); + child.condition_cache_split_context = condition_cache_split_context; + child.format_split_id = physical_split.format_split_id; + child.format_split_id_end = physical_split.format_split_id_end; + splits->push_back(std::move(child)); + } + return close_current_reader(); +} + Status TableReader::_evaluate_partition_prune_conjuncts(const VExprContextSPtrs& conjuncts, bool* can_filter_all) { DORIS_CHECK(can_filter_all != nullptr); diff --git a/be/src/format_v2/table_reader.h b/be/src/format_v2/table_reader.h index 213dbcd347d991..e4c09b131fb215 100644 --- a/be/src/format_v2/table_reader.h +++ b/be/src/format_v2/table_reader.h @@ -60,6 +60,7 @@ #include "format_v2/expr/delete_predicate.h" #include "format_v2/expr/evolution_cast.h" #include "format_v2/file_reader.h" +#include "format_v2/file_scan_context.h" #include "format_v2/parquet/reader/column_reader.h" #include "format_v2/schema_projection.h" #include "gen_cpp/PlanNodes_types.h" @@ -94,6 +95,9 @@ struct ScanTask { virtual ~ScanTask() = default; std::unique_ptr data_file; + std::shared_ptr file_context = nullptr; + int64_t format_split_id = -1; + int64_t format_split_id_end = -1; }; struct ProjectedColumnBuildContext { @@ -191,6 +195,13 @@ struct SplitReadOptions { ShardedKVCache* cache = nullptr; TFileRangeDesc current_range; FileFormat current_split_format = FileFormat::PARQUET; + std::shared_ptr file_context = nullptr; + // Generated physical children keep the FE source coordinates in the cache key so serial and + // refined scans address the same predicate bitmap. + std::optional> condition_cache_source_range = std::nullopt; + std::shared_ptr condition_cache_split_context = nullptr; + int64_t format_split_id = -1; + int64_t format_split_id_end = -1; std::optional global_rowid_context; // Optional absolute file-row selection used by TopN two-phase materialization. TableReader // carries it unchanged into the format-neutral FileScanRequest. @@ -241,13 +252,22 @@ class TableReader { // Refresh row-level predicates for an already prepared split. Physical readers that support // this operation decide the safe boundary at which the new immutable request becomes active. - virtual Status refresh_conjuncts(VExprContextSPtrs conjuncts); + // A supplied digest describes this exact conjunct snapshot for condition-cache isolation. RF + // completeness is monotonic within one split and enables safe COUNT(*) placeholder elision. + virtual Status refresh_conjuncts(VExprContextSPtrs conjuncts, + std::optional condition_cache_digest = std::nullopt, + bool all_runtime_filters_applied = false); virtual bool current_split_pruned() const { return _current_split_pruned; } virtual bool current_split_uses_metadata_count() const { return _current_split_uses_metadata_count; } + // Refine a prepared source split into format-specific physical children. The default + // implementation currently handles native Parquet; wrappers dispatch to their active child. + virtual Status build_physical_splits(const FileScanSplit& source_split, + std::vector* splits, bool* was_split); + // Discard the active split after the caller decides an error is ignorable, for example a // stale external-table file listing that returns NOT_FOUND. The next prepare_split() must start // with no concrete reader or split-local state left from the failed split. @@ -466,6 +486,7 @@ class TableReader { RETURN_IF_ERROR(_data_reader.column_mapper->create_scan_request( _table_filters, _projected_columns, file_request.get(), _runtime_state)); file_request->row_ids = _row_ids; + file_request->predicate_snapshot_digest = _predicate_snapshot_digest; _constant_pruning_safe_filter_count = std::min(_constant_pruning_safe_filter_count, file_request->constant_pruning_safe_table_filter_count); @@ -489,7 +510,9 @@ class TableReader { file_request->count_star_placeholder_columns.reserve( file_request->non_predicate_columns.size()); for (const auto& column : file_request->non_predicate_columns) { - file_request->count_star_placeholder_columns.push_back(column.column_id()); + if (!file_request->is_residual_predicate_column(column.column_id())) { + file_request->count_star_placeholder_columns.push_back(column.column_id()); + } } } RETURN_IF_ERROR(customize_file_scan_request(file_request.get())); @@ -1078,7 +1101,11 @@ class TableReader { RETURN_IF_ERROR(_build_file_aggregate_request(_push_down_agg_type, &file_request)); FileAggregateResult file_result; Status status; - { + if (_metadata_aggregate_result.has_value()) { + file_result = std::move(*_metadata_aggregate_result); + _metadata_aggregate_result.reset(); + status = Status::OK(); + } else { SCOPED_TIMER(_profile.file_reader_total_timer); SCOPED_TIMER(_profile.file_reader_aggregate_timer); status = _data_reader.reader->get_aggregate_result(file_request, &file_result); @@ -2269,6 +2296,7 @@ class TableReader { FileFormat _format; TPushAggOp::type _push_down_agg_type = TPushAggOp::type::NONE; std::optional> _push_down_count_columns; + std::optional _predicate_snapshot_digest; size_t _batch_size = 0; uint64_t _initial_condition_cache_digest = 0; uint64_t _condition_cache_digest = 0; @@ -2279,6 +2307,11 @@ class TableReader { segment_v2::ConditionCache::ExternalCacheKey _condition_cache_key; std::shared_ptr> _condition_cache; std::shared_ptr _condition_cache_ctx; + std::optional> _condition_cache_source_range; + std::shared_ptr _condition_cache_split_context; + bool _condition_cache_split_participating = false; + bool _condition_cache_split_invalid = false; + bool _condition_cache_initialized = false; int64_t _condition_cache_hit_count = 0; bool _current_reader_reached_eof = false; int64_t _remaining_table_level_count = -1; @@ -2293,6 +2326,7 @@ class TableReader { std::optional _global_rowid_context; std::optional> _row_ids; bool _aggregate_pushdown_tried = false; + std::optional _metadata_aggregate_result; bool _current_split_pruned = false; TableColumnMapperOptions _mapper_options; diff --git a/be/src/storage/segment/condition_cache.h b/be/src/storage/segment/condition_cache.h index 41e0c0b1f73c6f..876f91afcc70bf 100644 --- a/be/src/storage/segment/condition_cache.h +++ b/be/src/storage/segment/condition_cache.h @@ -24,6 +24,8 @@ #include #include +#include +#include #include #include #include @@ -53,6 +55,20 @@ struct ConditionCacheContext { static constexpr int GRANULE_SIZE = 2048; }; +// Coordinates condition-cache MISS results produced by row-group children of one BE-local source +// split. The entry is published only after every child contributes the same source-level key. +struct ConditionCacheSplitContext { + explicit ConditionCacheSplitContext(size_t child_count) : remaining_children(child_count) {} + + std::mutex lock; + size_t remaining_children = 0; + bool valid = true; + bool cache_hit_seen = false; + std::optional encoded_key; + int64_t base_granule = 0; + std::vector merged_filter_result; +}; + namespace segment_v2 { class ConditionCacheHandle; diff --git a/be/test/exec/scan/file_scanner_v2_test.cpp b/be/test/exec/scan/file_scanner_v2_test.cpp index c5d936c73df19a..8961879d46607b 100644 --- a/be/test/exec/scan/file_scanner_v2_test.cpp +++ b/be/test/exec/scan/file_scanner_v2_test.cpp @@ -21,9 +21,14 @@ #include #include +#include +#include +#include #include +#include #include #include +#include #include #include #include @@ -51,7 +56,9 @@ #include "exprs/vslot_ref.h" #include "format/table/iceberg_scan_semantics.h" #include "format_v2/expr/cast.h" +#include "format_v2/file_scan_context.h" #include "format_v2/table/fluss_hybrid_reader.h" +#include "storage/id_manager.h" #include "testutil/mock/mock_runtime_state.h" #include "util/timezone_utils.h" @@ -175,7 +182,9 @@ class CapturingSplitTableReader final : public format::TableReader { class ErrorContextTableReader final : public format::TableReader { public: Status prepare_split(const format::SplitReadOptions&) override { return prepare_status; } - Status refresh_conjuncts(VExprContextSPtrs) override { return refresh_status; } + Status refresh_conjuncts(VExprContextSPtrs, std::optional, bool) override { + return refresh_status; + } Status get_block(Block*, bool* eof) override { *eof = false; return read_status; @@ -1085,4 +1094,513 @@ TEST(FileScannerV2Test, RewriteSlotRefsToGlobalIndexMatrix) { } } +TScanRangeParams scan_range_with_path(std::string path) { + TScanRangeParams params; + TFileRangeDesc range; + range.__set_path(std::move(path)); + params.scan_range.ext_scan_range.file_scan_range.ranges.push_back(std::move(range)); + return params; +} + +class RemoteStyleSplitSourceConnector final : public SplitSourceConnector { +public: + explicit RemoteStyleSplitSourceConnector(std::vector ranges) + : _ranges(std::move(ranges)) {} + + Status get_next(bool* has_next, TFileRangeDesc* range, const std::atomic_bool*) override { + std::scoped_lock lock(_lock); + ++_get_next_calls; + *has_next = _next < _ranges.size(); + if (*has_next) { + *range = _ranges[_next++]; + } + return Status::OK(); + } + + int num_scan_ranges() override { return cast_set(_ranges.size()); } + + TFileScanRangeParams* get_params() override { return &_params; } + + int get_next_calls() const { return _get_next_calls.load(); } + +private: + std::mutex _lock; + std::vector _ranges; + size_t _next = 0; + std::atomic _get_next_calls = 0; + TFileScanRangeParams _params; +}; + +class BlockingRemoteStyleSplitSourceConnector final : public SplitSourceConnector { +public: + explicit BlockingRemoteStyleSplitSourceConnector(TFileRangeDesc range) + : _range(std::move(range)) {} + + Status get_next(bool* has_next, TFileRangeDesc* range, const std::atomic_bool*) override { + std::unique_lock lock(_lock); + if (!_source_returned) { + _source_returned = true; + *has_next = true; + *range = _range; + return Status::OK(); + } + _fetch_started = true; + _fetch_cv.notify_all(); + _fetch_cv.wait(lock, [&]() { return _release_fetch; }); + *has_next = false; + return Status::OK(); + } + + void wait_for_fetch() { + std::unique_lock lock(_lock); + _fetch_cv.wait(lock, [&]() { return _fetch_started; }); + } + + void release_fetch() { + { + std::scoped_lock lock(_lock); + _release_fetch = true; + } + _fetch_cv.notify_all(); + } + + int num_scan_ranges() override { return 1; } + + TFileScanRangeParams* get_params() override { return &_params; } + +private: + std::mutex _lock; + std::condition_variable _fetch_cv; + TFileRangeDesc _range; + bool _source_returned = false; + bool _fetch_started = false; + bool _release_fetch = false; + TFileScanRangeParams _params; +}; + +void expect_generated_splits_keep_source_alive(SplitSourceConnector* connector) { + FileScanSplit source; + bool has_next = false; + ASSERT_TRUE(connector->get_next_split(&has_next, &source).ok()); + ASSERT_TRUE(has_next); + ASSERT_TRUE(source.is_source_split); + + auto waiter = std::async(std::launch::async, [&]() { + FileScanSplit split; + bool waiter_has_next = false; + auto status = connector->get_next_split(&waiter_has_next, &split); + return std::make_tuple(status, waiter_has_next, std::move(split)); + }); + EXPECT_EQ(waiter.wait_for(std::chrono::milliseconds(50)), std::future_status::timeout); + + FileScanSplit child; + child.range = source.range; + child.range.__set_start_offset(101); + child.range.__set_size(17); + ASSERT_TRUE(connector->finish_source_split(source, {child}).ok()); + + ASSERT_EQ(waiter.wait_for(std::chrono::seconds(2)), std::future_status::ready); + auto [status, waiter_has_next, generated] = waiter.get(); + ASSERT_TRUE(status.ok()) << status; + ASSERT_TRUE(waiter_has_next); + EXPECT_FALSE(generated.is_source_split); + EXPECT_EQ(generated.range.start_offset, 101); + EXPECT_EQ(generated.range.size, 17); + + FileScanSplit end; + ASSERT_TRUE(connector->get_next_split(&has_next, &end).ok()); + EXPECT_FALSE(has_next); +} + +TEST(FileScannerV2Test, LocalSplitSourceWaitsForGeneratedRowGroupSplits) { + LocalSplitSourceConnector connector({scan_range_with_path("local.parquet")}, 4); + expect_generated_splits_keep_source_alive(&connector); +} + +TEST(FileScannerV2Test, RemoteStyleSplitSourceWaitsForGeneratedRowGroupSplits) { + TFileRangeDesc range; + range.__set_path("remote.parquet"); + RemoteStyleSplitSourceConnector connector({range}); + expect_generated_splits_keep_source_alive(&connector); + EXPECT_EQ(connector.get_next_calls(), 2); +} + +TEST(FileScannerV2Test, RemoteFetchDoesNotBlockGeneratedSplitPublication) { + TFileRangeDesc range; + range.__set_path("remote.parquet"); + BlockingRemoteStyleSplitSourceConnector connector(std::move(range)); + FileScanSplit source; + bool has_next = false; + ASSERT_TRUE(connector.get_next_split(&has_next, &source).ok()); + ASSERT_TRUE(has_next); + + auto remote_fetch = std::async(std::launch::async, [&]() { + FileScanSplit split; + bool fetch_has_next = false; + auto status = connector.get_next_split(&fetch_has_next, &split); + return std::make_pair(status, fetch_has_next); + }); + connector.wait_for_fetch(); + auto child_waiter = std::async(std::launch::async, [&]() { + FileScanSplit split; + bool child_available = false; + auto status = connector.get_next_split(&child_available, &split); + return std::make_tuple(status, child_available, std::move(split)); + }); + EXPECT_EQ(child_waiter.wait_for(std::chrono::milliseconds(50)), std::future_status::timeout); + + FileScanSplit child; + child.range = source.range; + child.range.__set_start_offset(101); + auto publish = std::async(std::launch::async, + [&]() { return connector.finish_source_split(source, {child}); }); + const bool publish_completed_while_fetch_blocked = + publish.wait_for(std::chrono::seconds(2)) == std::future_status::ready; + const bool child_published_while_fetch_blocked = + child_waiter.wait_for(std::chrono::seconds(2)) == std::future_status::ready; + connector.release_fetch(); + + const auto publish_status = publish.get(); + ASSERT_TRUE(publish_status.ok()) << publish_status; + ASSERT_TRUE(publish_completed_while_fetch_blocked); + ASSERT_TRUE(child_published_while_fetch_blocked); + auto [child_status, child_available, generated] = child_waiter.get(); + ASSERT_TRUE(child_status.ok()) << child_status; + ASSERT_TRUE(child_available); + EXPECT_EQ(generated.range.start_offset, 101); + auto [fetch_status, fetch_has_next] = remote_fetch.get(); + ASSERT_TRUE(fetch_status.ok()) << fetch_status; + EXPECT_FALSE(fetch_has_next); +} + +TEST(FileScannerV2Test, FinalRemoteFetchRechecksChildrenPublishedWhileBlocked) { + TFileRangeDesc range; + range.__set_path("remote.parquet"); + BlockingRemoteStyleSplitSourceConnector connector(std::move(range)); + FileScanSplit source; + bool has_next = false; + ASSERT_TRUE(connector.get_next_split(&has_next, &source).ok()); + ASSERT_TRUE(has_next); + + auto remote_fetch = std::async(std::launch::async, [&]() { + FileScanSplit split; + bool fetch_has_next = false; + auto status = connector.get_next_split(&fetch_has_next, &split); + return std::make_tuple(status, fetch_has_next, std::move(split)); + }); + connector.wait_for_fetch(); + + FileScanSplit child; + child.range = source.range; + child.range.__set_start_offset(101); + child.range.__set_size(17); + ASSERT_TRUE(connector.finish_source_split(source, {child}).ok()); + connector.release_fetch(); + + ASSERT_EQ(remote_fetch.wait_for(std::chrono::seconds(2)), std::future_status::ready); + auto [fetch_status, fetch_has_next, generated] = remote_fetch.get(); + ASSERT_TRUE(fetch_status.ok()) << fetch_status; + ASSERT_TRUE(fetch_has_next); + EXPECT_EQ(generated.range.start_offset, 101); + EXPECT_EQ(generated.range.size, 17); + + FileScanSplit end; + ASSERT_TRUE(connector.get_next_split(&has_next, &end).ok()); + EXPECT_FALSE(has_next); +} + +TEST(FileScannerV2Test, RawSourceClearsGeneratedChildEnvelope) { + LocalSplitSourceConnector connector({scan_range_with_path("next-local.parquet")}, 1); + FileScanSplit reused; + auto stale_source_range = std::make_shared(); + stale_source_range->__set_path("stale-child.parquet"); + reused.source_range = std::move(stale_source_range); + reused.start_offset = 101; + reused.size = 17; + reused.clear_table_level_row_count = true; + reused.file_context = std::make_shared(); + reused.condition_cache_split_context = std::make_shared(1); + reused.format_split_id = 3; + + bool has_next = false; + ASSERT_TRUE(connector.get_next_split(&has_next, &reused).ok()); + ASSERT_TRUE(has_next); + EXPECT_TRUE(reused.is_source_split); + EXPECT_EQ(reused.materialize_range().path, "next-local.parquet"); + EXPECT_EQ(reused.source_range, nullptr); + EXPECT_EQ(reused.file_context, nullptr); + EXPECT_EQ(reused.condition_cache_split_context, nullptr); + EXPECT_EQ(reused.format_split_id, -1); + ASSERT_TRUE(connector.finish_source_split(reused, {}).ok()); +} + +TEST(FileScannerV2Test, EosKeepsLastRangeIdentityForFinalAccounting) { + TFileRangeDesc range; + range.__set_path("remote.parquet"); + range.__set_format_type(TFileFormatType::FORMAT_PARQUET); + range.__set_file_type(TFileType::FILE_S3); + auto split_source = + std::make_shared(std::vector {range}); + + TFileScanRangeParams params; + params.__set_format_type(TFileFormatType::FORMAT_PARQUET); + MockRuntimeState state; + RuntimeProfile profile("last_range_accounting"); + FileScannerV2 scanner(&state, &profile, nullptr); + scanner._split_source = std::move(split_source); + scanner._params = ¶ms; + + bool has_next = false; + ASSERT_TRUE(scanner._get_next_scan_range(&has_next).ok()); + ASSERT_TRUE(has_next); + ASSERT_TRUE(scanner._split_source->finish_source_split(scanner._current_split, {}).ok()); + ASSERT_TRUE(scanner._get_next_scan_range(&has_next).ok()); + EXPECT_FALSE(has_next); + EXPECT_TRUE(scanner._current_range.__isset.file_type); + EXPECT_EQ(scanner._current_range.file_type, TFileType::FILE_S3); + EXPECT_EQ(scanner._uncached_reader_bytes_storage(scanner._current_range.file_type), + FileScannerV2::UncachedReaderBytesStorage::REMOTE); + + range.__set_file_type(TFileType::FILE_STREAM); + scanner._split_source = + std::make_shared(std::vector {range}); + ASSERT_TRUE(scanner._get_next_scan_range(&has_next).ok()); + ASSERT_TRUE(has_next); + // EOS is observable only after the claimed source is retired; otherwise the connector must + // keep waiting for that source to publish generated children. + ASSERT_TRUE(scanner._split_source->finish_source_split(scanner._current_split, {}).ok()); + ASSERT_TRUE(scanner._get_next_scan_range(&has_next).ok()); + EXPECT_FALSE(has_next); + EXPECT_TRUE(scanner._should_update_load_counters()); +} + +TEST(FileScannerV2Test, GeneratedChildrenCompleteSourceProgressOnlyOnce) { + LocalSplitSourceConnector connector({scan_range_with_path("two-groups.parquet")}, 1); + FileScanSplit source; + bool has_next = false; + ASSERT_TRUE(connector.get_next_split(&has_next, &source).ok()); + ASSERT_TRUE(has_next); + ASSERT_NE(source.source_progress, nullptr); + + FileScanSplit first_child; + first_child.range = source.range; + first_child.range.__set_start_offset(0); + FileScanSplit second_child; + second_child.range = source.range; + second_child.range.__set_start_offset(100); + ASSERT_TRUE(connector.finish_source_split(source, {first_child, second_child}).ok()); + + FileScanSplit first; + ASSERT_TRUE(connector.get_next_split(&has_next, &first).ok()); + ASSERT_TRUE(has_next); + FileScanSplit second; + ASSERT_TRUE(connector.get_next_split(&has_next, &second).ok()); + ASSERT_TRUE(has_next); + ASSERT_NE(first.source_progress, nullptr); + EXPECT_EQ(first.source_progress, second.source_progress); + EXPECT_FALSE(first.source_progress->complete_one()); + EXPECT_TRUE(second.source_progress->complete_one()); +} + +TEST(FileScannerV2Test, GeneratedChildrenShareAdaptiveBatchHistory) { + SourceSplitProgress source_progress; + EXPECT_FALSE(source_progress.adaptive_batch_bytes_per_row().has_value()); + + EXPECT_TRUE(source_progress.update_adaptive_batch_bytes_per_row(128.0)); + ASSERT_TRUE(source_progress.adaptive_batch_bytes_per_row().has_value()); + EXPECT_DOUBLE_EQ(*source_progress.adaptive_batch_bytes_per_row(), 128.0); + + EXPECT_FALSE(source_progress.update_adaptive_batch_bytes_per_row(256.0)); + ASSERT_TRUE(source_progress.adaptive_batch_bytes_per_row().has_value()); + EXPECT_DOUBLE_EQ(*source_progress.adaptive_batch_bytes_per_row(), 140.8); + + SourceSplitProgress concurrent_progress; + std::atomic first_samples = 0; + std::vector publishers; + for (int index = 0; index < 8; ++index) { + publishers.emplace_back([&] { + first_samples += concurrent_progress.update_adaptive_batch_bytes_per_row(64.0); + }); + } + for (auto& publisher : publishers) { + publisher.join(); + } + EXPECT_EQ(first_samples.load(), 1); +} + +TEST(FileScannerV2Test, PhysicalChildrenDoNotChangeFileNumber) { + RuntimeProfile profile("source_file_counter"); + auto* file_counter = ADD_COUNTER(&profile, "FileNumber", TUnit::UNIT); + FileScanSplit source; + source.is_source_split = true; + FileScannerV2::TEST_update_file_counter(file_counter, source); + + FileScanSplit child; + child.is_source_split = false; + FileScannerV2::TEST_update_file_counter(file_counter, child); + FileScannerV2::TEST_update_file_counter(file_counter, child); + + EXPECT_EQ(file_counter->value(), 1); +} + +TEST(FileScannerV2Test, GeneratedChildrenKeepOneGlobalRowIdSourceMapping) { + auto source_range = std::make_shared(); + source_range->__set_path("shared.parquet"); + source_range->__set_start_offset(64); + source_range->__set_size(1024); + + FileScanSplit first; + first.source_range = source_range; + first.start_offset = 128; + first.size = 256; + FileScanSplit second; + second.source_range = source_range; + second.start_offset = 512; + second.size = 256; + + EXPECT_NE(first.materialize_range().start_offset, second.materialize_range().start_offset); + EXPECT_EQ(first.source_identity_range().start_offset, source_range->start_offset); + EXPECT_EQ(second.source_identity_range().start_offset, source_range->start_offset); + + IdFileMap id_file_map(0); + const auto first_id = id_file_map.get_file_mapping_id( + std::make_shared(7, first.source_identity_range(), false)); + const auto second_id = id_file_map.get_file_mapping_id( + std::make_shared(7, second.source_identity_range(), false)); + EXPECT_EQ(first_id, second_id); +} + +TEST(FileScannerV2Test, IcebergDeleteSplitKeepsInitialScannerCountCap) { + TFileScanRangeParams params; + params.__set_format_type(TFileFormatType::FORMAT_PARQUET); + TFileRangeDesc range; + range.__set_format_type(TFileFormatType::FORMAT_PARQUET); + TTableFormatFileDesc table_format; + table_format.__set_table_format_type("iceberg"); + TIcebergFileDesc iceberg; + TIcebergDeleteFileDesc delete_file; + iceberg.__set_delete_files({delete_file}); + table_format.__set_iceberg_params(iceberg); + range.__set_table_format_params(table_format); + + EXPECT_FALSE(FileScanLocalState::TEST_can_generate_physical_splits(false, params, range)); + EXPECT_EQ(FileScanLocalState::TEST_adjust_scanner_count(16, 1, false), 1); + + range.table_format_params.iceberg_params.__set_delete_files({}); + EXPECT_TRUE(FileScanLocalState::TEST_can_generate_physical_splits(false, params, range)); + EXPECT_EQ(FileScanLocalState::TEST_adjust_scanner_count(16, 1, true), 16); + + params.__set_format_type(TFileFormatType::FORMAT_ORC); + range.__set_format_type(TFileFormatType::FORMAT_ORC); + EXPECT_TRUE(FileScanLocalState::TEST_can_generate_physical_splits(false, params, range)); + range.table_format_params.iceberg_params.__set_delete_files({delete_file}); + EXPECT_FALSE(FileScanLocalState::TEST_can_generate_physical_splits(false, params, range)); +} + +TEST(FileScannerV2Test, NativePaimonOrcCanGeneratePhysicalSplits) { + TFileScanRangeParams params; + params.__set_format_type(TFileFormatType::FORMAT_JNI); + TFileRangeDesc range; + range.__set_format_type(TFileFormatType::FORMAT_JNI); + TTableFormatFileDesc table_format; + table_format.__set_table_format_type("paimon"); + TPaimonFileDesc paimon; + paimon.__set_file_format("orc"); + table_format.__set_paimon_params(paimon); + range.__set_table_format_params(table_format); + + EXPECT_TRUE(FileScanLocalState::TEST_can_generate_physical_splits(false, params, range)); +} + +TEST(FileScannerV2Test, FileReadBytesProfilePublishesOnlyNewScannerDelta) { + int64_t reported = 0; + EXPECT_EQ(FileScannerV2::TEST_cumulative_profile_delta(100, &reported), 100); + EXPECT_EQ(FileScannerV2::TEST_cumulative_profile_delta(150, &reported), 50); + EXPECT_EQ(FileScannerV2::TEST_cumulative_profile_delta(150, &reported), 0); + EXPECT_EQ(reported, 150); +} + +TEST(FileScannerV2Test, FileReaderProfilePublishesOnlyNewScannerDeltas) { + io::FileReaderStats stats; + int64_t reported_bytes = 0; + int64_t reported_calls = 0; + int64_t reported_time = 0; + stats.read_bytes = 100; + stats.read_calls = 10; + stats.read_time_ns = 1000; + auto deltas = FileScannerV2::TEST_collect_file_reader_profile_deltas( + stats, &reported_bytes, &reported_calls, &reported_time); + EXPECT_EQ(deltas.read_bytes, 100); + EXPECT_EQ(deltas.read_calls, 10); + EXPECT_EQ(deltas.read_time_ns, 1000); + + stats.read_bytes = 150; + stats.read_calls = 12; + stats.read_time_ns = 1400; + deltas = FileScannerV2::TEST_collect_file_reader_profile_deltas( + stats, &reported_bytes, &reported_calls, &reported_time); + EXPECT_EQ(deltas.read_bytes, 50); + EXPECT_EQ(deltas.read_calls, 2); + EXPECT_EQ(deltas.read_time_ns, 400); +} + +TEST(FileScannerV2Test, SplitPlanningUsesTheSameIgnoredErrorPolicyAsOpen) { + EXPECT_EQ(FileScannerV2::TEST_classify_ignored_split_status(Status::NotFound("missing"), true, + false), + FileScannerV2::IgnoredSplitStatus::NOT_FOUND); + EXPECT_EQ(FileScannerV2::TEST_classify_ignored_split_status(Status::EndOfFile("empty"), false, + false), + FileScannerV2::IgnoredSplitStatus::EMPTY); + EXPECT_EQ(FileScannerV2::TEST_classify_ignored_split_status(Status::EndOfFile("stopped"), false, + true), + FileScannerV2::IgnoredSplitStatus::NONE); +} + +TEST(FileScannerV2Test, IcebergDeleteFilesDisablePhysicalSplitRefinement) { + TFileRangeDesc range; + EXPECT_TRUE(FileScannerV2::TEST_can_refine_source_split(range)); + + TTableFormatFileDesc table_format; + TIcebergFileDesc iceberg; + iceberg.__set_delete_files({}); + table_format.__set_iceberg_params(iceberg); + range.__set_table_format_params(table_format); + EXPECT_TRUE(FileScannerV2::TEST_can_refine_source_split(range)); + + TIcebergDeleteFileDesc delete_file; + delete_file.__set_path("delete.parquet"); + iceberg.__set_delete_files({delete_file}); + table_format.__set_iceberg_params(iceberg); + range.__set_table_format_params(table_format); + EXPECT_FALSE(FileScannerV2::TEST_can_refine_source_split(range)); +} + +TEST(FileScannerV2Test, LoadRangesKeepTheirFeBoundaries) { + TFileScanRangeParams params; + params.__set_format_type(TFileFormatType::FORMAT_PARQUET); + TFileRangeDesc range; + range.__set_format_type(TFileFormatType::FORMAT_PARQUET); + EXPECT_TRUE(FileScanLocalState::TEST_can_generate_physical_splits(false, params, range)); + EXPECT_FALSE(FileScanLocalState::TEST_can_generate_physical_splits(true, params, range)); +} + +TEST(FileScannerV2Test, PhysicalSplitRefinementRequiresMultipleConstructedScanners) { + TFileRangeDesc range; + EXPECT_FALSE(FileScannerV2::TEST_should_refine_source_split(range, 1)); + EXPECT_TRUE(FileScannerV2::TEST_should_refine_source_split(range, 2)); + for (int64_t target : {0, -1, 64 << 20}) { + range.__set_target_split_size(target); + EXPECT_TRUE(FileScannerV2::TEST_should_refine_source_split(range, 2)); + } + + TIcebergDeleteFileDesc delete_file; + delete_file.__set_path("delete.parquet"); + TIcebergFileDesc iceberg; + iceberg.__set_delete_files({delete_file}); + TTableFormatFileDesc table_format; + table_format.__set_iceberg_params(iceberg); + range.__set_table_format_params(table_format); + EXPECT_FALSE(FileScannerV2::TEST_should_refine_source_split(range, 2)); +} + } // namespace doris diff --git a/be/test/format_v2/jni/jni_table_reader_test.cpp b/be/test/format_v2/jni/jni_table_reader_test.cpp index ef524450f5d696..d50bf628c3a98b 100644 --- a/be/test/format_v2/jni/jni_table_reader_test.cpp +++ b/be/test/format_v2/jni/jni_table_reader_test.cpp @@ -392,6 +392,8 @@ TEST(JniTableReaderTest, AdaptiveProbeSetBeforePrepareControlsFirstJniOpen) { ASSERT_TRUE(init_reader(&reader, nullptr).ok()); reader.set_batch_size(32); + // Keep this aggregate initializer exhaustive because BE UT promotes omitted split-contract + // fields to compile errors, catching JNI callers that drift from scanner-created options. ASSERT_TRUE(reader.prepare_split({ .partition_values = {}, .conjuncts = std::nullopt, @@ -401,6 +403,10 @@ TEST(JniTableReaderTest, AdaptiveProbeSetBeforePrepareControlsFirstJniOpen) { .cache = nullptr, .current_range = {}, .current_split_format = FileFormat::JNI, + .file_context = nullptr, + .condition_cache_source_range = std::nullopt, + .condition_cache_split_context = nullptr, + .format_split_id = -1, .global_rowid_context = std::nullopt, }) .ok()); @@ -421,6 +427,10 @@ TEST(JniTableReaderTest, RefreshedConjunctIsReadyBeforeFilteringOpenScanner) { .cache = nullptr, .current_range = {}, .current_split_format = FileFormat::JNI, + .file_context = nullptr, + .condition_cache_source_range = std::nullopt, + .condition_cache_split_context = nullptr, + .format_split_id = -1, .global_rowid_context = std::nullopt, }) .ok()); @@ -451,6 +461,10 @@ TEST(JniTableReaderTest, CommonLifecycleTimersContainJniLifecycleWork) { .cache = nullptr, .current_range = {}, .current_split_format = FileFormat::JNI, + .file_context = nullptr, + .condition_cache_source_range = std::nullopt, + .condition_cache_split_context = nullptr, + .format_split_id = -1, .global_rowid_context = std::nullopt, }) .ok()); diff --git a/be/test/format_v2/orc/orc_reader_test.cpp b/be/test/format_v2/orc/orc_reader_test.cpp index 9872e5cfe564f5..55a81e7929b8cc 100644 --- a/be/test/format_v2/orc/orc_reader_test.cpp +++ b/be/test/format_v2/orc/orc_reader_test.cpp @@ -4706,7 +4706,8 @@ class NewOrcReaderTest : public testing::Test { // whole file (unset sentinel), matching how FE leaves the range unspecified. std::unique_ptr create_reader_with_range( const std::string& file_path, int64_t range_start_offset, int64_t range_size, - RuntimeProfile* profile = nullptr) const { + RuntimeProfile* profile = nullptr, int64_t mtime = 0, + std::shared_ptr file_context = nullptr) const { auto system_properties = std::make_shared(); system_properties->system_type = TFileType::FILE_LOCAL; auto file_description = std::make_unique(); @@ -4714,8 +4715,10 @@ class NewOrcReaderTest : public testing::Test { file_description->file_size = static_cast(std::filesystem::file_size(file_path)); file_description->range_start_offset = range_start_offset; file_description->range_size = range_size; + file_description->mtime = mtime; return std::make_unique(system_properties, file_description, - nullptr, profile, std::nullopt); + nullptr, profile, std::nullopt, false, + false, std::move(file_context)); } Status scan_direct_in_filter(const std::string& file_path, int32_t in_list_size, @@ -4941,6 +4944,111 @@ TEST_F(NewOrcReaderTest, AggregatePushdownCountUsesOnlySplitStripes) { EXPECT_EQ(first_split_count + second_split_count, layout[0].rows + layout[1].rows); } +TEST_F(NewOrcReaderTest, PhysicalSplitPlanningReturnsOwnedStripes) { + const auto multi_stripe_file_path = (_test_dir / "physical_splits.orc").string(); + write_multi_stripe_orc_int_file(multi_stripe_file_path, {1, 1000, 2000}); + const auto layout = get_orc_stripe_layout(multi_stripe_file_path); + ASSERT_EQ(layout.size(), 3); + + constexpr int64_t TEST_MTIME = 828282; + auto reader = create_reader_with_range(multi_stripe_file_path, 0, -1, nullptr, TEST_MTIME); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + ASSERT_TRUE(reader->init(&state).ok()); + ASSERT_TRUE(reader->open(std::make_shared()).ok()); + + std::vector splits; + bool was_split = false; + ASSERT_TRUE(reader->build_physical_splits(&splits, &was_split).ok()); + ASSERT_TRUE(was_split); + ASSERT_EQ(splits.size(), layout.size()); + for (size_t stripe_id = 0; stripe_id < layout.size(); ++stripe_id) { + EXPECT_EQ(splits[stripe_id].start_offset, cast_set(layout[stripe_id].offset)); + EXPECT_EQ(splits[stripe_id].size, cast_set(layout[stripe_id].length)); + EXPECT_EQ(splits[stripe_id].format_split_id, cast_set(stripe_id)); + EXPECT_NE(splits[stripe_id].file_context, nullptr); + EXPECT_EQ(splits[stripe_id].file_context, splits[0].file_context); + } + + auto partial_reader = + create_reader_with_range(multi_stripe_file_path, cast_set(layout[1].offset), + cast_set(layout[1].length), nullptr, TEST_MTIME); + ASSERT_TRUE(partial_reader->init(&state).ok()); + ASSERT_TRUE(partial_reader->open(std::make_shared()).ok()); + splits.clear(); + was_split = false; + ASSERT_TRUE(partial_reader->build_physical_splits(&splits, &was_split).ok()); + ASSERT_TRUE(was_split); + ASSERT_EQ(splits.size(), 1); + EXPECT_EQ(splits[0].format_split_id, 1); + + auto mutable_reader = create_reader_with_range(multi_stripe_file_path, 0, -1); + ASSERT_TRUE(mutable_reader->init(&state).ok()); + ASSERT_TRUE(mutable_reader->open(std::make_shared()).ok()); + splits.clear(); + was_split = true; + ASSERT_TRUE(mutable_reader->build_physical_splits(&splits, &was_split).ok()); + EXPECT_FALSE(was_split); + EXPECT_TRUE(splits.empty()); +} + +TEST_F(NewOrcReaderTest, PhysicalStripeChildrenReuseOnlyParentTailAndReadExactRows) { + const auto multi_stripe_file_path = (_test_dir / "shared_physical_splits.orc").string(); + write_multi_stripe_orc_int_file(multi_stripe_file_path, {1, 1000, 2000}); + const auto layout = get_orc_stripe_layout(multi_stripe_file_path); + ASSERT_EQ(layout.size(), 3); + + constexpr int64_t TEST_MTIME = 838383; + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + auto build_splits = [&](std::vector* splits) -> Status { + auto reader = create_reader_with_range(multi_stripe_file_path, 0, -1, nullptr, TEST_MTIME); + RETURN_IF_ERROR(reader->init(&state)); + RETURN_IF_ERROR(reader->open(std::make_shared())); + bool was_split = false; + RETURN_IF_ERROR(reader->build_physical_splits(splits, &was_split)); + if (!was_split) { + return Status::InternalError("ORC reader did not refine the source split"); + } + return Status::OK(); + }; + + std::vector parent_splits; + ASSERT_TRUE(build_splits(&parent_splits).ok()); + ASSERT_EQ(parent_splits.size(), layout.size()); + const auto parent_context = parent_splits[0].file_context; + ASSERT_NE(parent_context, nullptr); + for (const auto& split : parent_splits) { + EXPECT_EQ(split.file_context, parent_context); + } + std::vector sibling_splits; + ASSERT_TRUE(build_splits(&sibling_splits).ok()); + ASSERT_EQ(sibling_splits.size(), layout.size()); + ASSERT_NE(sibling_splits[0].file_context, parent_splits[0].file_context); + for (const auto& split : sibling_splits) { + EXPECT_EQ(split.file_context, sibling_splits[0].file_context); + } + + auto child = + create_reader_with_range(multi_stripe_file_path, parent_splits[0].start_offset, + parent_splits[0].size, nullptr, TEST_MTIME, parent_context); + ASSERT_TRUE(child->init(&state).ok()); + std::vector schema; + ASSERT_TRUE(child->get_schema(&schema).ok()); + auto request = std::make_shared(); + request->non_predicate_columns.push_back(field_projection(0)); + request->local_positions.emplace(format::LocalColumnId(0), format::LocalIndex(0)); + ASSERT_TRUE(child->open(request).ok()); + + size_t rows_read = 0; + bool eof = false; + while (!eof) { + auto block = build_file_block({schema[0]}); + size_t rows = 0; + ASSERT_TRUE(child->get_block(&block, &rows, &eof).ok()); + rows_read += rows; + } + EXPECT_EQ(rows_read, layout[0].rows); +} + TEST_F(NewOrcReaderTest, OpenAcceptsDorisOffsetTimezone) { auto reader = create_reader(); RuntimeState state {TQueryOptions(), TQueryGlobals()}; @@ -10648,7 +10756,8 @@ TEST_F(NewOrcReaderTest, SargConjunctReadsNonAdjacentStripeRangesAfterPruning) { write_multi_stripe_orc_int_file(multi_stripe_file_path, {1, 1000, 201}); ASSERT_EQ(get_orc_stripe_count(multi_stripe_file_path), 3); - auto reader = create_reader_for_path(multi_stripe_file_path); + constexpr int64_t TEST_MTIME = 848484; + auto reader = create_reader_with_range(multi_stripe_file_path, 0, -1, nullptr, TEST_MTIME); RuntimeState state {TQueryOptions(), TQueryGlobals()}; ASSERT_TRUE(reader->init(&state).ok()); reader->set_batch_size(32); @@ -10664,6 +10773,14 @@ TEST_F(NewOrcReaderTest, SargConjunctReadsNonAdjacentStripeRangesAfterPruning) { VExprContext::create_shared(std::make_shared(0, 500))); ASSERT_TRUE(reader->open(request).ok()); + std::vector splits; + bool was_split = false; + ASSERT_TRUE(reader->build_physical_splits(&splits, &was_split).ok()); + ASSERT_TRUE(was_split); + ASSERT_EQ(splits.size(), 2); + EXPECT_EQ(splits[0].format_split_id, 0); + EXPECT_EQ(splits[1].format_split_id, 2); + bool eof = false; std::vector result_ids; while (!eof) { diff --git a/be/test/format_v2/parquet/parquet_reader_control_test.cpp b/be/test/format_v2/parquet/parquet_reader_control_test.cpp index fde32ee81ad025..baf00570f5df7e 100644 --- a/be/test/format_v2/parquet/parquet_reader_control_test.cpp +++ b/be/test/format_v2/parquet/parquet_reader_control_test.cpp @@ -447,6 +447,53 @@ TEST(ParquetColumnReaderControlTest, PendingRequestActivatesOnlyAtRowGroupBounda EXPECT_TRUE(scheduler._predicate_runtime_stats.empty()); } +TEST(ParquetColumnReaderControlTest, PhysicalChildrenRestoreAdaptiveStateByPredicateSnapshot) { + auto adaptive_context = std::make_shared(); + auto first_request = std::make_shared(); + first_request->predicate_snapshot_digest = 17; + + ParquetScanScheduler first; + first.set_adaptive_context(adaptive_context); + first.set_scan_request(first_request); + first._predicate_runtime_stats.emplace( + 3, detail::AdaptivePredicateStats { + .cost_per_input_row_ns = 11, .survival_ratio = 0.25, .samples = 8}); + first._predicate_survival_ratio = 0.25; + first._predicate_batch_sequence = 8; + first._empty_predicate_batch_rows = 4096; + first._publish_adaptive_state(*first_request); + + ParquetScanScheduler sibling; + sibling.set_adaptive_context(adaptive_context); + sibling.set_scan_request(first_request); + ASSERT_EQ(sibling._predicate_runtime_stats.at(3).samples, 8); + EXPECT_DOUBLE_EQ(sibling._predicate_survival_ratio, 0.25); + EXPECT_EQ(sibling._predicate_batch_sequence, 8); + EXPECT_EQ(sibling._empty_predicate_batch_rows, 4096); + + adaptive_context->publish(17, {.predicate_runtime_stats = {{3, + detail::AdaptivePredicateStats { + .cost_per_input_row_ns = 99, + .survival_ratio = 0.9, + .samples = 1}}}, + .predicate_survival_ratio = 0.9, + .predicate_batch_sequence = 1, + .empty_predicate_batch_rows = 256}); + const auto monotonic_snapshot = adaptive_context->restore(17); + EXPECT_EQ(monotonic_snapshot.predicate_runtime_stats.at(3).samples, 8); + EXPECT_DOUBLE_EQ(monotonic_snapshot.predicate_survival_ratio, 0.25); + EXPECT_EQ(monotonic_snapshot.empty_predicate_batch_rows, 4096); + + auto refreshed_request = std::make_shared(); + refreshed_request->predicate_snapshot_digest = 19; + sibling.queue_scan_request(refreshed_request); + sibling.activate_pending_scan_request_at_row_group_boundary(); + EXPECT_TRUE(sibling._predicate_runtime_stats.empty()); + EXPECT_EQ(sibling._predicate_survival_ratio, -1); + EXPECT_EQ(sibling._predicate_batch_sequence, 0); + EXPECT_EQ(sibling._empty_predicate_batch_rows, 0); +} + TEST(ParquetColumnReaderControlTest, PendingOutputDrainsBeforePageCrossingSample) { ParquetScanScheduler scheduler; scheduler._batch_size = 1; diff --git a/be/test/format_v2/parquet/parquet_reader_test.cpp b/be/test/format_v2/parquet/parquet_reader_test.cpp index 306dd9bdefa048..5abc3ea4603ac2 100644 --- a/be/test/format_v2/parquet/parquet_reader_test.cpp +++ b/be/test/format_v2/parquet/parquet_reader_test.cpp @@ -30,6 +30,7 @@ #include #include #include +#include #include #include #include @@ -70,6 +71,7 @@ #include "format_v2/column_mapper.h" #include "format_v2/expr/delete_predicate.h" #include "format_v2/file_reader.h" +#include "format_v2/file_scan_context.h" #include "format_v2/parquet/parquet_column_schema.h" #include "format_v2/parquet/parquet_scan.h" #include "format_v2/parquet/reader/column_reader.h" @@ -77,7 +79,9 @@ #include "format_v2/schema_reader.h" #include "format_v2/table_reader.h" #include "gen_cpp/Types_types.h" +#include "io/fs/buffered_reader.h" #include "io/fs/file_meta_cache.h" +#include "io/fs/local_file_system.h" #include "io/io_common.h" #include "runtime/exec_env.h" #include "runtime/runtime_state.h" @@ -138,6 +142,94 @@ void annotate_variant_schema(const std::string& file_path) { DORIS_CHECK(output.good()); } +void rewrite_as_root_only_parquet_file(const std::string& file_path) { + std::ifstream input(file_path, std::ios::binary | std::ios::ate); + DORIS_CHECK(input.good()); + const auto input_size = static_cast(input.tellg()); + DORIS_CHECK(input_size >= static_cast(8)); + std::vector file_bytes(cast_set(input_size)); + input.seekg(0); + input.read(reinterpret_cast(file_bytes.data()), cast_set(input_size)); + DORIS_CHECK(input.good()); + DORIS_CHECK(memcmp(file_bytes.data() + file_bytes.size() - 4, "PAR1", 4) == 0); + + const uint32_t footer_size = decode_fixed32_le(file_bytes.data() + file_bytes.size() - 8); + DORIS_CHECK(footer_size <= file_bytes.size() - 8); + const size_t footer_offset = file_bytes.size() - 8 - footer_size; + uint32_t thrift_size = footer_size; + tparquet::FileMetaData metadata; + DORIS_CHECK( + deserialize_thrift_msg(file_bytes.data() + footer_offset, &thrift_size, true, &metadata) + .ok()); + input.close(); + + tparquet::SchemaElement root; + root.__set_name("schema"); + root.__set_num_children(0); + root.__set_repetition_type(tparquet::FieldRepetitionType::REQUIRED); + metadata.__set_schema({root}); + for (auto& row_group : metadata.row_groups) { + row_group.columns.clear(); + } + + file_bytes.resize(footer_offset); + std::vector footer; + ThriftSerializer serializer(/*compact=*/true, 1024); + DORIS_CHECK(serializer.serialize(&metadata, &footer).ok()); + file_bytes.insert(file_bytes.end(), footer.begin(), footer.end()); + std::array encoded_footer_size {}; + encode_fixed32_le(encoded_footer_size.data(), cast_set(footer.size())); + file_bytes.insert(file_bytes.end(), encoded_footer_size.begin(), encoded_footer_size.end()); + file_bytes.insert(file_bytes.end(), {'P', 'A', 'R', '1'}); + + std::ofstream output(file_path, std::ios::binary | std::ios::trunc); + output.write(reinterpret_cast(file_bytes.data()), + cast_set(file_bytes.size())); + output.close(); + DORIS_CHECK(output.good()); +} + +void rewrite_parquet_metadata( + const std::string& file_path, + const std::function& rewrite_metadata) { + std::ifstream input(file_path, std::ios::binary | std::ios::ate); + DORIS_CHECK(input.good()); + const auto input_size = static_cast(input.tellg()); + DORIS_CHECK(input_size >= static_cast(8)); + std::vector file_bytes(cast_set(input_size)); + input.seekg(0); + input.read(reinterpret_cast(file_bytes.data()), cast_set(input_size)); + DORIS_CHECK(input.good()); + DORIS_CHECK(memcmp(file_bytes.data() + file_bytes.size() - 4, "PAR1", 4) == 0); + + const uint32_t footer_size = decode_fixed32_le(file_bytes.data() + file_bytes.size() - 8); + DORIS_CHECK(footer_size <= file_bytes.size() - 8); + const size_t footer_offset = file_bytes.size() - 8 - footer_size; + uint32_t thrift_size = footer_size; + tparquet::FileMetaData metadata; + DORIS_CHECK( + deserialize_thrift_msg(file_bytes.data() + footer_offset, &thrift_size, true, &metadata) + .ok()); + input.close(); + rewrite_metadata(&metadata); + + file_bytes.resize(footer_offset); + std::vector footer; + ThriftSerializer serializer(/*compact=*/true, 1024); + DORIS_CHECK(serializer.serialize(&metadata, &footer).ok()); + file_bytes.insert(file_bytes.end(), footer.begin(), footer.end()); + std::array encoded_footer_size {}; + encode_fixed32_le(encoded_footer_size.data(), cast_set(footer.size())); + file_bytes.insert(file_bytes.end(), encoded_footer_size.begin(), encoded_footer_size.end()); + file_bytes.insert(file_bytes.end(), {'P', 'A', 'R', '1'}); + + std::ofstream output(file_path, std::ios::binary | std::ios::trunc); + output.write(reinterpret_cast(file_bytes.data()), + cast_set(file_bytes.size())); + output.close(); + DORIS_CHECK(output.good()); +} + format::LocalColumnIndex field_projection(int32_t column_id) { return format::LocalColumnIndex {.index = column_id}; } @@ -873,6 +965,58 @@ void write_parquet_file(const std::string& file_path, int64_t row_group_size = R row_group_size, builder.build())); } +void write_parquet_file_with_explicit_row_groups( + const std::string& file_path, const std::vector>& row_groups) { + auto file_result = arrow::io::FileOutputStream::Open(file_path); + ASSERT_TRUE(file_result.ok()) << file_result.status(); + std::shared_ptr out = *file_result; + + const auto id = ::parquet::schema::PrimitiveNode::Make("id", ::parquet::Repetition::REQUIRED, + ::parquet::LogicalType::None(), + ::parquet::Type::INT32); + const auto schema_node = + ::parquet::schema::GroupNode::Make("schema", ::parquet::Repetition::REQUIRED, {id}); + const auto schema = std::static_pointer_cast<::parquet::schema::GroupNode>(schema_node); + ::parquet::WriterProperties::Builder properties; + properties.compression(::parquet::Compression::UNCOMPRESSED); + properties.disable_dictionary(); + auto writer = ::parquet::ParquetFileWriter::Open(out, schema, properties.build()); + for (const auto& values : row_groups) { + auto* row_group = writer->AppendRowGroup(); + auto* id_writer = static_cast<::parquet::Int32Writer*>(row_group->NextColumn()); + if (!values.empty()) { + EXPECT_EQ(id_writer->WriteBatch(values.size(), nullptr, nullptr, values.data()), + values.size()); + } + id_writer->Close(); + row_group->Close(); + } + writer->Close(); +} + +class HttpPathFileReader final : public io::FileReader { +public: + explicit HttpPathFileReader(io::FileReaderSPtr delegate) + : _delegate(std::move(delegate)), + _path("https://example.test/multi-row-group.parquet") {} + + Status close() override { return _delegate->close(); } + const io::Path& path() const override { return _path; } + size_t size() const override { return _delegate->size(); } + bool closed() const override { return _delegate->closed(); } + int64_t mtime() const override { return _delegate->mtime(); } + +protected: + Status read_at_impl(size_t offset, Slice result, size_t* bytes_read, + const io::IOContext* io_ctx) override { + return _delegate->read_at(offset, result, bytes_read, io_ctx); + } + +private: + io::FileReaderSPtr _delegate; + io::Path _path; +}; + void write_mixed_variant_row_groups(const std::string& file_path) { auto n_type = arrow::struct_({arrow::field("value", arrow::binary(), true), arrow::field("typed_value", arrow::int32(), true)}); @@ -1851,7 +1995,9 @@ class NewParquetReaderTest : public testing::Test { bool is_immutable = false, bool enable_mapping_varbinary = false, std::string fs_name = {}, int64_t mtime = 0, std::optional hive_parquet_time_zone = std::nullopt, - bool preserve_binary_uuid = false) const { + bool preserve_binary_uuid = false, + std::shared_ptr file_context = nullptr, + int64_t format_split_id = -1) const { auto system_properties = std::make_shared(); system_properties->system_type = TFileType::FILE_LOCAL; auto file_description = std::make_unique(); @@ -1865,7 +2011,8 @@ class NewParquetReaderTest : public testing::Test { return std::make_unique( system_properties, file_description, std::move(io_ctx), profile, global_rowid_context, enable_mapping_timestamp_tz, enable_mapping_varbinary, - std::move(hive_parquet_time_zone), preserve_binary_uuid); + std::move(hive_parquet_time_zone), preserve_binary_uuid, nullptr, std::nullopt, + std::move(file_context), format_split_id); } std::filesystem::path _test_dir; @@ -2075,6 +2222,251 @@ TEST_F(NewParquetReaderTest, ReusesProfileHandlesAcrossFiles) { } } +TEST_F(NewParquetReaderTest, PhysicalSplitPlanningReturnsFormatLocalDescriptors) { + write_parquet_file(_file_path, 2); + constexpr int64_t TEST_MTIME = 414141; + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + auto reader = create_reader(0, -1, nullptr, false, nullptr, std::nullopt, false, false, {}, + TEST_MTIME); + ASSERT_TRUE(reader->init(&state).ok()); + + std::vector splits; + bool was_split = false; + ASSERT_TRUE(reader->build_physical_splits(&splits, &was_split).ok()); + ASSERT_TRUE(was_split); + ASSERT_EQ(splits.size(), 3); + for (size_t index = 0; index < splits.size(); ++index) { + EXPECT_EQ(splits[index].format_split_id, cast_set(index)); + EXPECT_GT(splits[index].size, 0); + EXPECT_NE(splits[index].file_context, nullptr); + } +} + +TEST_F(NewParquetReaderTest, RowGroupSplitsShareOnlyParentFooterContext) { + write_parquet_file(_file_path, 2); + constexpr int64_t TEST_MTIME = 424242; + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + RuntimeProfile parent_profile("row_group_split_parent"); + auto parent = create_reader(0, -1, &parent_profile, false, nullptr, std::nullopt, false, false, + {}, TEST_MTIME); + ASSERT_TRUE(parent->init(&state).ok()); + + std::vector children; + bool was_split = false; + ASSERT_TRUE(parent->build_physical_splits(&children, &was_split).ok()); + ASSERT_TRUE(was_split); + ASSERT_EQ(children.size(), 3); + ASSERT_NE(children[0].file_context, nullptr); + for (size_t index = 0; index < children.size(); ++index) { + EXPECT_EQ(children[index].file_context, children[0].file_context); + EXPECT_EQ(children[index].format_split_id, cast_set(index)); + EXPECT_GT(children[index].size, 0); + } + EXPECT_EQ(parent_profile.get_counter("FileFooterReadCalls")->value(), 1); + + RuntimeProfile sibling_profile("same_file_source_split"); + auto sibling = create_reader(0, -1, &sibling_profile, false, nullptr, std::nullopt, false, + false, {}, TEST_MTIME); + ASSERT_TRUE(sibling->init(&state).ok()); + std::vector sibling_children; + ASSERT_TRUE(sibling->build_physical_splits(&sibling_children, &was_split).ok()); + ASSERT_TRUE(was_split); + ASSERT_EQ(sibling_children.size(), children.size()); + EXPECT_NE(sibling_children[0].file_context, children[0].file_context); + EXPECT_EQ(sibling_profile.get_counter("FileFooterReadCalls")->value(), 1); + EXPECT_EQ(sibling_profile.get_counter("FileFooterHitCache")->value(), 0); + + RuntimeProfile child_profile("row_group_split_child"); + auto child = create_reader(children[0].start_offset, children[0].size, &child_profile, false, + nullptr, std::nullopt, false, false, {}, TEST_MTIME, std::nullopt, + false, children[0].file_context, children[0].format_split_id); + ASSERT_TRUE(child->init(&state).ok()); + EXPECT_EQ(child_profile.get_counter("FileFooterReadCalls")->value(), 0); + EXPECT_EQ(child_profile.get_counter("FileFooterHitCache")->value(), 0); + + std::vector schema; + ASSERT_TRUE(child->get_schema(&schema).ok()); + auto request = std::make_shared(); + for (size_t index = 0; index < schema.size(); ++index) { + request->non_predicate_columns.push_back(field_projection(cast_set(index))); + } + use_schema_order_positions(request.get(), schema); + ASSERT_TRUE(child->open(request).ok()); + size_t total_rows = 0; + bool eof = false; + while (!eof) { + auto block = build_file_block(schema); + size_t rows = 0; + ASSERT_TRUE(child->get_block(&block, &rows, &eof).ok()); + total_rows += rows; + } + EXPECT_EQ(total_rows, 2); +} + +TEST_F(NewParquetReaderTest, PhysicalSplitsApplyOpenedRequestFooterPruning) { + write_parquet_file(_file_path, 1); + constexpr int64_t TEST_MTIME = 434343; + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + auto reader = create_reader(0, -1, nullptr, false, nullptr, std::nullopt, false, false, {}, + TEST_MTIME); + ASSERT_TRUE(reader->init(&state).ok()); + + std::vector schema; + ASSERT_TRUE(reader->get_schema(&schema).ok()); + auto request = std::make_shared(); + request->predicate_columns.push_back(field_projection(0)); + request->non_predicate_columns.push_back(field_projection(1)); + use_schema_order_positions(request.get(), schema); + request->conjuncts.push_back(create_int32_greater_than_conjunct(0, 3)); + ASSERT_TRUE(reader->open(request).ok()); + + std::vector children; + bool was_split = false; + ASSERT_TRUE(reader->build_physical_splits(&children, &was_split).ok()); + ASSERT_TRUE(was_split); + ASSERT_EQ(children.size(), 2); + EXPECT_EQ(children[0].format_split_id, 3); + EXPECT_EQ(children[1].format_split_id, 4); + + auto all_pruned = create_reader(0, -1, nullptr, false, nullptr, std::nullopt, false, false, {}, + TEST_MTIME); + ASSERT_TRUE(all_pruned->init(&state).ok()); + auto all_pruned_request = std::make_shared(*request); + all_pruned_request->conjuncts = {create_int32_greater_than_conjunct(0, 10)}; + ASSERT_TRUE(all_pruned->open(all_pruned_request).ok()); + children.clear(); + was_split = false; + ASSERT_TRUE(all_pruned->build_physical_splits(&children, &was_split).ok()); + EXPECT_TRUE(was_split); + EXPECT_TRUE(children.empty()); +} + +TEST_F(NewParquetReaderTest, PhysicalSplitsDeclineWhenUnusedChunkHasInvalidEnvelope) { + write_parquet_file(_file_path, 2); + rewrite_parquet_metadata(_file_path, [&](tparquet::FileMetaData* metadata) { + const auto invalid_offset = cast_set(std::filesystem::file_size(_file_path) + 1); + for (auto& row_group : metadata->row_groups) { + ASSERT_GE(row_group.columns.size(), 2); + auto& unused = row_group.columns[1].meta_data; + unused.__set_data_page_offset(invalid_offset); + unused.__set_dictionary_page_offset(invalid_offset); + } + }); + + constexpr int64_t TEST_MTIME = 444444; + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + auto reader = create_reader(0, -1, nullptr, false, nullptr, std::nullopt, false, false, {}, + TEST_MTIME); + ASSERT_TRUE(reader->init(&state).ok()); + + std::vector children; + bool was_split = false; + ASSERT_TRUE(reader->build_physical_splits(&children, &was_split).ok()); + EXPECT_FALSE(was_split); + EXPECT_TRUE(children.empty()); + + auto request = std::make_shared(); + request->non_predicate_columns.push_back(field_projection(0)); + request->local_positions.emplace(format::LocalColumnId(0), format::LocalIndex(0)); + ASSERT_TRUE(reader->open(request).ok()); + std::vector schema; + ASSERT_TRUE(reader->get_schema(&schema).ok()); + size_t total_rows = 0; + bool eof = false; + while (!eof) { + auto block = build_file_block({schema[0]}); + size_t rows = 0; + ASSERT_TRUE(reader->get_block(&block, &rows, &eof).ok()); + total_rows += rows; + } + EXPECT_EQ(total_rows, ROW_COUNT); +} + +TEST_F(NewParquetReaderTest, PhysicalSplitsSkipEmptyRowGroupsBeforeAndBetweenData) { + write_parquet_file_with_explicit_row_groups(_file_path, {{}, {10}, {}, {20}}); + constexpr int64_t TEST_MTIME = 515151; + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + auto reader = create_reader(0, -1, nullptr, false, nullptr, std::nullopt, false, false, {}, + TEST_MTIME); + ASSERT_TRUE(reader->init(&state).ok()); + + std::vector children; + bool was_split = false; + ASSERT_TRUE(reader->build_physical_splits(&children, &was_split).ok()); + ASSERT_TRUE(was_split); + ASSERT_EQ(children.size(), 2); + EXPECT_EQ(children[0].format_split_id, 1); + EXPECT_EQ(children[1].format_split_id, 3); +} + +TEST_F(NewParquetReaderTest, PhysicalSplitsCompleteAnAllEmptyFileWithoutChildren) { + write_parquet_file_with_explicit_row_groups(_file_path, {{}, {}}); + constexpr int64_t TEST_MTIME = 616161; + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + auto reader = create_reader(0, -1, nullptr, false, nullptr, std::nullopt, false, false, {}, + TEST_MTIME); + ASSERT_TRUE(reader->init(&state).ok()); + + std::vector children; + bool was_split = false; + ASSERT_TRUE(reader->build_physical_splits(&children, &was_split).ok()); + EXPECT_TRUE(was_split); + EXPECT_TRUE(children.empty()); +} + +TEST_F(NewParquetReaderTest, PositiveRowRootOnlyFileKeepsInitializedReaderForCount) { + write_parquet_file(_file_path, 2); + rewrite_as_root_only_parquet_file(_file_path); + constexpr int64_t TEST_MTIME = 717171; + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + auto reader = create_reader(0, -1, nullptr, false, nullptr, std::nullopt, false, false, {}, + TEST_MTIME); + ASSERT_TRUE(reader->init(&state).ok()); + + std::vector children; + bool was_split = false; + ASSERT_TRUE(reader->build_physical_splits(&children, &was_split).ok()); + EXPECT_FALSE(was_split); + EXPECT_TRUE(children.empty()); + + auto request = std::make_shared(); + ASSERT_TRUE(reader->open(request).ok()); + size_t total_rows = 0; + bool eof = false; + while (!eof) { + Block block; + size_t rows = 0; + ASSERT_TRUE(reader->get_block(&block, &rows, &eof).ok()); + total_rows += rows; + } + EXPECT_EQ(total_rows, ROW_COUNT); +} + +TEST_F(NewParquetReaderTest, InMemoryStagedFilesDeclinePhysicalSplitRefinement) { + write_parquet_file(_file_path, 2); + auto parquet_reader = ::parquet::ParquetFileReader::OpenFile(_file_path, false); + ASSERT_GT(parquet_reader->metadata()->num_row_groups(), 1); + io::FileReaderSPtr local_reader; + ASSERT_TRUE(io::global_local_filesystem()->open_file(_file_path, &local_reader).ok()); + + format::parquet::ParquetFileContext file_context; + io::FileDescription file_description; + file_description.path = "https://example.test/multi-row-group.parquet"; + file_description.file_size = static_cast(local_reader->size()); + file_description.mtime = 123456; + io::IOContext io_context; + ASSERT_TRUE(file_context + .open(std::make_shared(std::move(local_reader)), + &io_context, false, file_description) + .ok()); + EXPECT_FALSE(file_context.can_refine_physical_splits()); + + io::FileReaderSPtr ordinary_reader; + ASSERT_TRUE(io::global_local_filesystem()->open_file(_file_path, &ordinary_reader).ok()); + file_context.native_file = std::move(ordinary_reader); + EXPECT_TRUE(file_context.can_refine_physical_splits()); +} + TEST_F(NewParquetReaderTest, GetSchemaReturnsFileLocalColumns) { auto reader = create_reader(); RuntimeState state {TQueryOptions(), TQueryGlobals()}; @@ -4022,16 +4414,17 @@ TEST_F(NewParquetReaderTest, NativeFooterCacheDoesNotReuseMutableUnknownVersion) _file_path = (_test_dir / "mutable_footer_cache.parquet").string(); write_parquet_file(_file_path); RuntimeState state {TQueryOptions(), TQueryGlobals()}; - RuntimeProfile first_profile("native_footer_cache_mutable_first"); - auto first = create_reader(0, -1, &first_profile); + auto first = + create_reader(0, -1, &first_profile, false, nullptr, std::nullopt, false, false, {}, 0); ASSERT_TRUE(first->init(&state).ok()); EXPECT_EQ(first_profile.get_counter("FileFooterReadCalls")->value(), 1); EXPECT_EQ(first_profile.get_counter("FileFooterHitCache")->value(), 0); write_parquet_file(_file_path); RuntimeProfile second_profile("native_footer_cache_mutable_second"); - auto second = create_reader(0, -1, &second_profile); + auto second = create_reader(0, -1, &second_profile, false, nullptr, std::nullopt, false, false, + {}, 0); ASSERT_TRUE(second->init(&state).ok()); EXPECT_EQ(second_profile.get_counter("FileFooterReadCalls")->value(), 1); EXPECT_EQ(second_profile.get_counter("FileFooterHitCache")->value(), 0); @@ -4047,6 +4440,19 @@ TEST_F(NewParquetReaderTest, NativeFooterAcceptsMetadataAboveThriftMessageLimit) format::parquet::detail::validate_native_footer_size(metadata_size, file_size).ok()); } +TEST_F(NewParquetReaderTest, MutableUnknownVersionDeclinesPhysicalSplitRefinement) { + write_parquet_file(_file_path, 2); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + auto reader = create_reader(0, -1, nullptr, false, nullptr, std::nullopt, false, false, {}, 0); + ASSERT_TRUE(reader->init(&state).ok()); + + std::vector children; + bool was_split = false; + ASSERT_TRUE(reader->build_physical_splits(&children, &was_split).ok()); + EXPECT_FALSE(was_split); + EXPECT_TRUE(children.empty()); +} + TEST_F(NewParquetReaderTest, NativeFooterSizeIsBoundedBeforeMetadataAllocation) { constexpr size_t file_size = 512UL << 20; const auto status = format::parquet::detail::validate_native_footer_size( diff --git a/be/test/format_v2/parquet/parquet_scan_test.cpp b/be/test/format_v2/parquet/parquet_scan_test.cpp index 0b5d9b503f4c9f..9f5739dcea016f 100644 --- a/be/test/format_v2/parquet/parquet_scan_test.cpp +++ b/be/test/format_v2/parquet/parquet_scan_test.cpp @@ -183,6 +183,55 @@ TEST(ParquetScanMetadataSafetyTest, CheckedChunkRangesDrivePrefetchAndSplitAssig .ok()); } +TEST(ParquetScanMetadataSafetyTest, ExactRowGroupUsesPrecomputedFirstRows) { + tparquet::FileMetaData metadata; + tparquet::RowGroup unrelated_before; + unrelated_before.__set_num_rows(-1); + tparquet::RowGroup selected_group; + selected_group.__set_num_rows(7); + tparquet::RowGroup unrelated_after; + unrelated_after.__set_num_rows(-1); + metadata.__set_row_groups({unrelated_before, selected_group, unrelated_after}); + + const std::vector first_rows {0, 10, 17}; + const format::parquet::ParquetScanRange exact_group { + .start_offset = 0, .size = -1, .file_size = 1, .row_group_id = 1}; + std::vector selected; + ASSERT_TRUE(format::parquet::detail::select_native_row_groups_by_scan_range( + metadata, exact_group, first_rows, &selected) + .ok()); + EXPECT_EQ(selected, std::vector({1})); +} + +TEST(ParquetScanMetadataSafetyTest, CoalescedSplitSelectsExactRowGroups) { + tparquet::FileMetaData metadata; + tparquet::RowGroup row_group; + row_group.__set_num_rows(1); + metadata.__set_row_groups({row_group, row_group, row_group, row_group}); + + const std::vector first_rows {0, 1, 2, 3}; + const format::parquet::ParquetScanRange coalesced {.start_offset = 0, + .size = -1, + .file_size = 1, + .row_group_id = 1, + .row_group_id_end = 3}; + std::vector selected; + ASSERT_TRUE(format::parquet::detail::select_native_row_groups_by_scan_range( + metadata, coalesced, first_rows, &selected) + .ok()); + EXPECT_EQ(selected, std::vector({1, 2, 3})); + + auto invalid = coalesced; + invalid.row_group_id_end = 0; + EXPECT_FALSE(format::parquet::detail::select_native_row_groups_by_scan_range( + metadata, invalid, first_rows, &selected) + .ok()); + invalid.row_group_id_end = 4; + EXPECT_FALSE(format::parquet::detail::select_native_row_groups_by_scan_range( + metadata, invalid, first_rows, &selected) + .ok()); +} + class Int32ZoneMapExpr final : public VExpr { public: enum class Op { GE, GT, LT }; @@ -2568,6 +2617,9 @@ TEST_F(ParquetScanTest, AggregateCountAndMinMaxUseAllSelectedRowGroups) { format::FileAggregateResult count_result; format::FileAggregateRequest count_request; count_request.agg_type = TPushAggOp::COUNT; + format::FileAggregateResult metadata_count_result; + ASSERT_TRUE(reader->get_metadata_aggregate_result(count_request, &metadata_count_result).ok()); + EXPECT_EQ(metadata_count_result.count, 6); ASSERT_TRUE(reader->get_aggregate_result(count_request, &count_result).ok()); EXPECT_EQ(count_result.count, 6); EXPECT_TRUE(count_result.columns.empty()); @@ -2576,6 +2628,11 @@ TEST_F(ParquetScanTest, AggregateCountAndMinMaxUseAllSelectedRowGroups) { format::FileAggregateRequest required_count_request; required_count_request.agg_type = TPushAggOp::COUNT; required_count_request.columns.push_back({.projection = field_projection(0)}); + format::FileAggregateResult metadata_required_count_result; + ASSERT_TRUE(reader->get_metadata_aggregate_result(required_count_request, + &metadata_required_count_result) + .ok()); + EXPECT_EQ(metadata_required_count_result.count, 6); ASSERT_TRUE(reader->get_aggregate_result(required_count_request, &required_count_result).ok()); // The required scalar projection is retained for logical-type validation, but after that // validation COUNT(id) can reuse the same selected-row-group footer count as COUNT(*). @@ -2586,6 +2643,10 @@ TEST_F(ParquetScanTest, AggregateCountAndMinMaxUseAllSelectedRowGroups) { minmax_request.agg_type = TPushAggOp::MINMAX; minmax_request.columns.push_back({.projection = field_projection(0)}); minmax_request.columns.push_back({.projection = field_projection(1)}); + format::FileAggregateResult metadata_minmax_result; + ASSERT_TRUE( + reader->get_metadata_aggregate_result(minmax_request, &metadata_minmax_result).ok()); + ASSERT_EQ(metadata_minmax_result.columns.size(), 2); ASSERT_TRUE(reader->get_aggregate_result(minmax_request, &minmax_result).ok()); EXPECT_EQ(minmax_result.count, 6); ASSERT_EQ(minmax_result.columns.size(), 2); @@ -2777,6 +2838,11 @@ TEST_F(ParquetScanTest, AggregateCountOnStructRecordsSelectedRowsRead) { format::FileAggregateRequest aggregate_request; aggregate_request.agg_type = TPushAggOp::COUNT; aggregate_request.columns.push_back({.projection = field_projection(0)}); + format::FileAggregateResult metadata_result; + const auto metadata_status = + reader->get_metadata_aggregate_result(aggregate_request, &metadata_result); + EXPECT_TRUE(metadata_status.is()) << metadata_status; + EXPECT_EQ(file_reader_stats.read_rows, 0); format::FileAggregateResult result; ASSERT_TRUE(reader->get_aggregate_result(aggregate_request, &result).ok()); EXPECT_EQ(result.count, 4); diff --git a/be/test/format_v2/table/fluss_union_lake_reader_test.cpp b/be/test/format_v2/table/fluss_union_lake_reader_test.cpp index f4e776d9e35bce..a6cada38380942 100644 --- a/be/test/format_v2/table/fluss_union_lake_reader_test.cpp +++ b/be/test/format_v2/table/fluss_union_lake_reader_test.cpp @@ -69,9 +69,12 @@ class RecordingLakeReader final : public format::TableReader { return Status::OK(); } - Status refresh_conjuncts(VExprContextSPtrs conjuncts) override { + Status refresh_conjuncts(VExprContextSPtrs conjuncts, + std::optional condition_cache_digest = std::nullopt, + bool all_runtime_filters_applied = false) override { ++refresh_calls; - return format::TableReader::refresh_conjuncts(std::move(conjuncts)); + return format::TableReader::refresh_conjuncts(std::move(conjuncts), condition_cache_digest, + all_runtime_filters_applied); } Status close() override { diff --git a/be/test/format_v2/table/hudi_reader_test.cpp b/be/test/format_v2/table/hudi_reader_test.cpp index aaa684be5138a9..8baf3f89887fc5 100644 --- a/be/test/format_v2/table/hudi_reader_test.cpp +++ b/be/test/format_v2/table/hudi_reader_test.cpp @@ -147,9 +147,12 @@ class RefreshTrackingTableReader final : public TableReader { public: Status prepare_split(const SplitReadOptions&) override { return Status::OK(); } - Status refresh_conjuncts(VExprContextSPtrs conjuncts) override { + Status refresh_conjuncts(VExprContextSPtrs conjuncts, + std::optional condition_cache_digest = std::nullopt, + bool all_runtime_filters_applied = false) override { ++refresh_count; - return TableReader::refresh_conjuncts(std::move(conjuncts)); + return TableReader::refresh_conjuncts(std::move(conjuncts), condition_cache_digest, + all_runtime_filters_applied); } int refresh_count = 0; diff --git a/be/test/format_v2/table/paimon_reader_test.cpp b/be/test/format_v2/table/paimon_reader_test.cpp index 354e608e48f999..a1df88536b86b5 100644 --- a/be/test/format_v2/table/paimon_reader_test.cpp +++ b/be/test/format_v2/table/paimon_reader_test.cpp @@ -96,9 +96,12 @@ class RefreshTrackingTableReader final : public TableReader { public: Status prepare_split(const SplitReadOptions&) override { return Status::OK(); } - Status refresh_conjuncts(VExprContextSPtrs conjuncts) override { + Status refresh_conjuncts(VExprContextSPtrs conjuncts, + std::optional condition_cache_digest = std::nullopt, + bool all_runtime_filters_applied = false) override { ++refresh_count; - return TableReader::refresh_conjuncts(std::move(conjuncts)); + return TableReader::refresh_conjuncts(std::move(conjuncts), condition_cache_digest, + all_runtime_filters_applied); } int refresh_count = 0; diff --git a/be/test/format_v2/table_reader_test.cpp b/be/test/format_v2/table_reader_test.cpp index 4ea240f3cc2c7d..876ba4297b34db 100644 --- a/be/test/format_v2/table_reader_test.cpp +++ b/be/test/format_v2/table_reader_test.cpp @@ -26,11 +26,13 @@ #include #include +#include #include #include #include #include #include +#include #include #include @@ -457,6 +459,48 @@ class NonDeterministicPartitionPredicate final : public VExpr { const std::string _expr_name = "NonDeterministicPartitionPredicate"; }; +class NonLocalizableInt32Predicate final : public VExpr { +public: + explicit NonLocalizableInt32Predicate(int column_id) + : VExpr(std::make_shared(), false), _column_id(column_id) { + // The production dependency collector walks slot children. Keep the input in the tree so + // a rejected file-local rewrite still marks its value as required by the residual filter. + add_child(table_int32_slot_ref(column_id, column_id, "non_localizable_input")); + } + + Status execute_column_impl(VExprContext*, const Block* block, const Selector* selector, + size_t count, ColumnPtr& result_column) const override { + const auto& input = assert_cast( + assert_cast(*block->get_by_position(_column_id).column) + .get_nested_column()); + auto result = ColumnUInt8::create(); + result->get_data().resize(count); + for (size_t row = 0; row < count; ++row) { + result->get_data()[row] = + input.get_element(selector == nullptr ? row : (*selector)[row]) > 0; + } + result_column = std::move(result); + return Status::OK(); + } + + const std::string& expr_name() const override { return _expr_name; } + bool is_deterministic() const override { return false; } + + void collect_slot_column_ids(std::set& column_ids) const override { + column_ids.insert(_column_id); + } + + Status clone_node(VExprSPtr* cloned_expr) const override { + DORIS_CHECK(cloned_expr != nullptr); + *cloned_expr = std::make_shared(_column_id); + return Status::OK(); + } + +private: + const int _column_id; + const std::string _expr_name = "NonLocalizableInt32Predicate"; +}; + class NullableArrayBigintDefaultExpr final : public VExpr { public: explicit NullableArrayBigintDefaultExpr(DataTypePtr data_type) @@ -1430,6 +1474,7 @@ struct FakeFileReaderState { int refresh_count = 0; int64_t total_rows = 2; int64_t aggregate_count = -1; + int64_t metadata_aggregate_count = -1; int64_t condition_cache_base_granule = 0; size_t condition_cache_num_granules = 0; bool eof_with_first_batch = true; @@ -1437,6 +1482,14 @@ struct FakeFileReaderState { bool stop_during_aggregate = false; bool stop_during_read = false; bool not_found_during_init = false; + bool physical_split_error = false; + int physical_split_count = -1; + std::vector physical_split_sizes; + std::vector physical_split_ids; + bool physical_split_saw_open_request = false; + std::optional unsupported_projection; + int build_physical_splits_count = 0; + std::chrono::milliseconds lifecycle_delay {0}; std::shared_ptr last_request; std::shared_ptr pending_request; std::optional last_aggregate_request; @@ -1455,6 +1508,9 @@ class FakeFileReader final : public FileReader { Status init(RuntimeState* state) override { (void)state; + if (_state->lifecycle_delay.count() > 0) { + std::this_thread::sleep_for(_state->lifecycle_delay); + } ++_state->init_count; if (_state->not_found_during_init) { return Status::NotFound("fake table reader input is missing"); @@ -1473,8 +1529,15 @@ class FakeFileReader final : public FileReader { } Status open(std::shared_ptr request) override { + if (_state->lifecycle_delay.count() > 0) { + std::this_thread::sleep_for(_state->lifecycle_delay); + } RETURN_IF_ERROR(FileReader::open(std::move(request))); _state->last_request = _request; + if (_state->unsupported_projection.has_value() && + !_request->is_count_star_placeholder(*_state->unsupported_projection)) { + return Status::NotSupported("fake unsupported projection requires a placeholder"); + } ++_state->open_count; _returned_batch = false; return Status::OK(); @@ -1588,6 +1651,19 @@ class FakeFileReader final : public FileReader { return Status::OK(); } + Status get_metadata_aggregate_result(const FileAggregateRequest& request, + FileAggregateResult* result) override { + if (_state->metadata_aggregate_count < 0) { + return FileReader::get_metadata_aggregate_result(request, result); + } + if (request.agg_type != TPushAggOp::type::COUNT) { + return Status::NotSupported("fake reader only supports metadata COUNT pushdown"); + } + result->count = _state->metadata_aggregate_count; + result->columns.clear(); + return Status::OK(); + } + void set_condition_cache_context(std::shared_ptr ctx) override { _state->condition_cache_ctx = std::move(ctx); if (_state->condition_cache_ctx != nullptr && !_state->condition_cache_ctx->is_hit) { @@ -1600,7 +1676,47 @@ class FakeFileReader final : public FileReader { int64_t get_total_rows() const override { return _state->total_rows; } + Status build_physical_splits(std::vector* splits, + bool* was_split) const override { + if (_state->physical_split_count < 0) { + return FileReader::build_physical_splits(splits, was_split); + } + if (_state->lifecycle_delay.count() > 0) { + std::this_thread::sleep_for(_state->lifecycle_delay); + } + ++_state->build_physical_splits_count; + _state->physical_split_saw_open_request = _request != nullptr; + if (_state->physical_split_error) { + return Status::InternalError("injected physical split planning failure"); + } + splits->clear(); + DORIS_CHECK(_state->physical_split_sizes.empty() || + _state->physical_split_sizes.size() == + cast_set(_state->physical_split_count)); + DORIS_CHECK(_state->physical_split_ids.empty() || + _state->physical_split_ids.size() == + cast_set(_state->physical_split_count)); + int64_t next_start_offset = 0; + for (int index = 0; index < _state->physical_split_count; ++index) { + PhysicalFileSplit child; + child.start_offset = next_start_offset; + child.size = _state->physical_split_sizes.empty() + ? 100 + : _state->physical_split_sizes[cast_set(index)]; + child.format_split_id = _state->physical_split_ids.empty() + ? index + : _state->physical_split_ids[cast_set(index)]; + next_start_offset += child.size; + splits->push_back(std::move(child)); + } + *was_split = true; + return Status::OK(); + } + Status close() override { + if (_state->lifecycle_delay.count() > 0) { + std::this_thread::sleep_for(_state->lifecycle_delay); + } ++_state->close_count; _request.reset(); _eof = true; @@ -2048,6 +2164,345 @@ TEST(TableReaderTest, CanUseInjectedFileReaderForStandaloneUnitTest) { EXPECT_TRUE(eos); } +TEST(TableReaderTest, SinglePhysicalSplitReusesPlanningReader) { + std::vector file_schema; + file_schema.push_back(make_file_column(0, "id", std::make_shared())); + std::vector projected_columns; + projected_columns.push_back(make_table_column(0, "id", std::make_shared())); + set_name_identifiers(&projected_columns); + + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + RuntimeProfile profile("single_physical_split"); + auto fake_state = std::make_shared(); + fake_state->physical_split_count = 1; + fake_state->lifecycle_delay = std::chrono::milliseconds(2); + FakeTableReader reader(file_schema, fake_state); + ASSERT_TRUE(reader.init({ + .projected_columns = projected_columns, + .conjuncts = {}, + .format = FileFormat::PARQUET, + .scan_params = nullptr, + .io_ctx = nullptr, + .runtime_state = &state, + .scanner_profile = &profile, + }) + .ok()); + + SplitReadOptions split_options; + split_options.current_range.__set_path("fake-table-reader-input"); + ASSERT_TRUE(reader.prepare_split(split_options).ok()); + FileScanSplit source_split; + source_split.range = split_options.current_range; + source_split.is_source_split = true; + std::vector children; + bool was_split = false; + ASSERT_TRUE(reader.build_physical_splits(source_split, &children, &was_split).ok()); + + EXPECT_FALSE(was_split); + EXPECT_TRUE(children.empty()); + EXPECT_EQ(fake_state->init_count, 1); + EXPECT_EQ(fake_state->close_count, 0); + for (const auto* counter_name : + {"TableReader", "CreateReaderTime", "FileReader", "FileReaderInitTime", "OpenReaderTime", + "FileReaderOpenTime"}) { + const auto* counter = profile.get_counter(counter_name); + ASSERT_NE(counter, nullptr) << counter_name; + EXPECT_GT(counter->value(), 0) << counter_name; + } + + Block block = build_table_block(projected_columns); + bool eos = false; + ASSERT_TRUE(reader.get_block(&block, &eos).ok()); + EXPECT_EQ(fake_state->init_count, 1); + EXPECT_EQ(fake_state->open_count, 1); + ASSERT_TRUE(reader.close().ok()); +} + +TEST(TableReaderTest, MetadataAggregateKeepsPlanningReaderInsteadOfGeneratingChildren) { + std::vector file_schema; + file_schema.push_back(make_file_column(0, "id", std::make_shared())); + std::vector projected_columns; + projected_columns.push_back(make_table_column(0, "id", std::make_shared())); + set_name_identifiers(&projected_columns); + + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + auto fake_state = std::make_shared(); + fake_state->physical_split_count = 3; + fake_state->metadata_aggregate_count = 7; + FakeTableReader reader(file_schema, fake_state); + ASSERT_TRUE(reader.init({ + .projected_columns = projected_columns, + .conjuncts = {}, + .format = FileFormat::PARQUET, + .scan_params = nullptr, + .io_ctx = nullptr, + .runtime_state = &state, + .scanner_profile = nullptr, + .push_down_agg_type = TPushAggOp::type::COUNT, + .push_down_count_columns = std::vector {}, + }) + .ok()); + + SplitReadOptions split_options; + split_options.current_range.__set_path("fake-table-reader-input"); + ASSERT_TRUE(reader.prepare_split(split_options).ok()); + FileScanSplit source_split; + source_split.range = split_options.current_range; + source_split.is_source_split = true; + std::vector children; + bool was_split = false; + ASSERT_TRUE(reader.build_physical_splits(source_split, &children, &was_split).ok()); + EXPECT_FALSE(was_split); + EXPECT_TRUE(children.empty()); + EXPECT_EQ(fake_state->build_physical_splits_count, 0); + EXPECT_EQ(fake_state->close_count, 0); + + Block block = build_table_block(projected_columns); + bool eos = false; + ASSERT_TRUE(reader.get_block(&block, &eos).ok()); + EXPECT_EQ(block.rows(), 7); + EXPECT_EQ(fake_state->close_count, 1); +} + +TEST(TableReaderTest, PhysicalSplitPlanningProfilesZeroAndManyChildren) { + for (const int child_count : {0, 2}) { + SCOPED_TRACE(child_count); + std::vector file_schema; + file_schema.push_back(make_file_column(0, "id", std::make_shared())); + std::vector projected_columns; + projected_columns.push_back(make_table_column(0, "id", std::make_shared())); + set_name_identifiers(&projected_columns); + + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + RuntimeProfile profile("temporary_physical_split_reader"); + auto fake_state = std::make_shared(); + fake_state->physical_split_count = child_count; + fake_state->lifecycle_delay = std::chrono::milliseconds(2); + FakeTableReader reader(file_schema, fake_state); + ASSERT_TRUE(reader.init({ + .projected_columns = projected_columns, + .conjuncts = {}, + .format = FileFormat::PARQUET, + .scan_params = nullptr, + .io_ctx = nullptr, + .runtime_state = &state, + .scanner_profile = &profile, + }) + .ok()); + + SplitReadOptions split_options; + split_options.current_range.__set_path("fake-table-reader-input"); + split_options.current_range.__set_start_offset(17); + split_options.current_range.__set_size(400); + split_options.current_range.__set_file_size(400); + TTableFormatFileDesc table_format; + table_format.__set_table_level_row_count(123); + split_options.current_range.__set_table_format_params(table_format); + ASSERT_TRUE(reader.prepare_split(split_options).ok()); + FileScanSplit source_split; + source_split.range = split_options.current_range; + source_split.is_source_split = true; + std::vector children; + bool was_split = false; + ASSERT_TRUE(reader.build_physical_splits(source_split, &children, &was_split).ok()); + EXPECT_TRUE(was_split); + EXPECT_EQ(children.size(), child_count); + if (child_count > 1) { + ASSERT_NE(children[0].source_range, nullptr); + ASSERT_NE(children[0].condition_cache_split_context, nullptr); + for (size_t index = 0; index < children.size(); ++index) { + EXPECT_EQ(children[index].source_range, children[0].source_range); + EXPECT_EQ(children[index].condition_cache_split_context, + children[0].condition_cache_split_context); + EXPECT_EQ(children[index].start_offset, cast_set(index * 100)); + EXPECT_EQ(children[index].size, 100); + EXPECT_EQ(children[index].format_split_id, cast_set(index)); + EXPECT_FALSE(children[index].is_source_split); + EXPECT_FALSE(children[index] + .materialize_range() + .table_format_params.__isset.table_level_row_count); + } + } + EXPECT_TRUE(fake_state->physical_split_saw_open_request); + EXPECT_EQ(fake_state->close_count, 1); + for (const auto* counter_name : {"TableReader", "CreateReaderTime", "FileReader", + "FileReaderInitTime", "FileReaderCloseTime"}) { + const auto* counter = profile.get_counter(counter_name); + ASSERT_NE(counter, nullptr) << counter_name; + EXPECT_GT(counter->value(), 0) << counter_name; + } + } +} + +TEST(TableReaderTest, PhysicalSplitsUseFeTargetAndMergeSmallTail) { + std::vector file_schema; + file_schema.push_back(make_file_column(0, "id", std::make_shared())); + std::vector projected_columns; + projected_columns.push_back(make_table_column(0, "id", std::make_shared())); + set_name_identifiers(&projected_columns); + + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + auto fake_state = std::make_shared(); + fake_state->physical_split_count = 5; + FakeTableReader reader(file_schema, fake_state); + ASSERT_TRUE(reader.init({ + .projected_columns = projected_columns, + .conjuncts = {}, + .format = FileFormat::PARQUET, + .scan_params = nullptr, + .io_ctx = nullptr, + .runtime_state = &state, + .scanner_profile = nullptr, + }) + .ok()); + + SplitReadOptions split_options; + split_options.current_range.__set_path("fake-table-reader-input"); + split_options.current_range.__set_start_offset(0); + split_options.current_range.__set_size(500); + split_options.current_range.__set_file_size(500); + split_options.current_range.__set_target_split_size(250); + ASSERT_TRUE(reader.prepare_split(split_options).ok()); + + FileScanSplit source_split; + source_split.range = split_options.current_range; + source_split.is_source_split = true; + std::vector children; + bool was_split = false; + ASSERT_TRUE(reader.build_physical_splits(source_split, &children, &was_split).ok()); + ASSERT_TRUE(was_split); + ASSERT_EQ(children.size(), 2); + EXPECT_EQ(children[0].start_offset, 0); + EXPECT_EQ(children[0].size, 200); + EXPECT_EQ(children[0].format_split_id, 0); + EXPECT_EQ(children[0].format_split_id_end, 1); + EXPECT_EQ(children[1].start_offset, 200); + EXPECT_EQ(children[1].size, 300); + EXPECT_EQ(children[1].format_split_id, 2); + EXPECT_EQ(children[1].format_split_id_end, 4); + + fake_state->physical_split_count = 2; + fake_state->physical_split_sizes = {300, 50}; + split_options.current_range.__set_size(350); + split_options.current_range.__set_file_size(350); + ASSERT_TRUE(reader.prepare_split(split_options).ok()); + source_split.range = split_options.current_range; + children.clear(); + ASSERT_TRUE(reader.build_physical_splits(source_split, &children, &was_split).ok()); + // A row group is indivisible; exceeding the target must not pull the following group into it. + ASSERT_EQ(children.size(), 2); + EXPECT_EQ(children[0].size, 300); + EXPECT_EQ(children[1].size, 50); + + fake_state->physical_split_sizes = {100, 100}; + fake_state->physical_split_ids = {0, 2}; + split_options.current_range.__set_size(200); + split_options.current_range.__set_file_size(200); + ASSERT_TRUE(reader.prepare_split(split_options).ok()); + source_split.range = split_options.current_range; + children.clear(); + ASSERT_TRUE(reader.build_physical_splits(source_split, &children, &was_split).ok()); + // Non-consecutive ids represent a pruned row-group gap and must remain separate ownership. + ASSERT_EQ(children.size(), 2); + EXPECT_EQ(children[0].format_split_id, 0); + EXPECT_EQ(children[1].format_split_id, 2); +} + +TEST(TableReaderTest, OrcPhysicalSplitsUseFeTargetSize) { + std::vector file_schema; + file_schema.push_back(make_file_column(0, "id", std::make_shared())); + std::vector projected_columns; + projected_columns.push_back(make_table_column(0, "id", std::make_shared())); + set_name_identifiers(&projected_columns); + + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + auto fake_state = std::make_shared(); + fake_state->physical_split_count = 5; + FakeTableReader reader(file_schema, fake_state); + ASSERT_TRUE(reader.init({ + .projected_columns = projected_columns, + .conjuncts = {}, + .format = FileFormat::ORC, + .scan_params = nullptr, + .io_ctx = nullptr, + .runtime_state = &state, + .scanner_profile = nullptr, + }) + .ok()); + + SplitReadOptions split_options; + split_options.current_range.__set_path("fake-orc-table-reader-input"); + split_options.current_range.__set_start_offset(0); + split_options.current_range.__set_size(500); + split_options.current_range.__set_file_size(500); + split_options.current_range.__set_target_split_size(250); + ASSERT_TRUE(reader.prepare_split(split_options).ok()); + + FileScanSplit source_split; + source_split.range = split_options.current_range; + source_split.is_source_split = true; + std::vector children; + bool was_split = false; + ASSERT_TRUE(reader.build_physical_splits(source_split, &children, &was_split).ok()); + ASSERT_TRUE(was_split); + ASSERT_EQ(children.size(), 2); + EXPECT_EQ(children[0].start_offset, 0); + EXPECT_EQ(children[0].size, 200); + EXPECT_EQ(children[0].format_split_id, 0); + EXPECT_EQ(children[0].format_split_id_end, 1); + EXPECT_EQ(children[1].start_offset, 200); + EXPECT_EQ(children[1].size, 300); + EXPECT_EQ(children[1].format_split_id, 2); + EXPECT_EQ(children[1].format_split_id_end, 4); +} + +TEST(TableReaderTest, PhysicalSplitPlanningProfilesAndClosesFailedReaders) { + for (const bool fail_during_init : {true, false}) { + SCOPED_TRACE(fail_during_init ? "init" : "build"); + std::vector file_schema; + file_schema.push_back(make_file_column(0, "id", std::make_shared())); + std::vector projected_columns; + projected_columns.push_back(make_table_column(0, "id", std::make_shared())); + set_name_identifiers(&projected_columns); + + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + RuntimeProfile profile("failed_physical_split_reader"); + auto fake_state = std::make_shared(); + fake_state->physical_split_count = 2; + fake_state->not_found_during_init = fail_during_init; + fake_state->physical_split_error = !fail_during_init; + fake_state->lifecycle_delay = std::chrono::milliseconds(2); + FakeTableReader reader(file_schema, fake_state); + ASSERT_TRUE(reader.init({ + .projected_columns = projected_columns, + .conjuncts = {}, + .format = FileFormat::PARQUET, + .scan_params = nullptr, + .io_ctx = nullptr, + .runtime_state = &state, + .scanner_profile = &profile, + }) + .ok()); + + SplitReadOptions split_options; + split_options.current_range.__set_path("fake-table-reader-input"); + ASSERT_TRUE(reader.prepare_split(split_options).ok()); + FileScanSplit source_split; + source_split.range = split_options.current_range; + source_split.is_source_split = true; + std::vector children; + bool was_split = false; + EXPECT_FALSE(reader.build_physical_splits(source_split, &children, &was_split).ok()); + EXPECT_EQ(fake_state->close_count, 1); + for (const auto* counter_name : {"TableReader", "CreateReaderTime", "FileReader", + "FileReaderInitTime", "FileReaderCloseTime"}) { + const auto* counter = profile.get_counter(counter_name); + ASSERT_NE(counter, nullptr) << counter_name; + EXPECT_GT(counter->value(), 0) << counter_name; + } + } +} + TEST(TableReaderTest, PrepareSplitReplacesInitialConjunctSnapshot) { std::vector file_schema; file_schema.push_back(make_file_column(0, "id", std::make_shared())); @@ -2321,6 +2776,154 @@ TEST(TableReaderTest, CountStarFallbackKeepsLateRuntimeFilterCarrierValues) { ASSERT_TRUE(reader.close().ok()); } +TEST(TableReaderTest, FinalRuntimeFilterRefreshRestoresCountStarPlaceholder) { + std::vector file_schema; + file_schema.push_back( + make_file_column(0, "planner_carrier", std::make_shared())); + file_schema.push_back(make_file_column(1, "rf_key", std::make_shared())); + + std::vector projected_columns; + projected_columns.push_back( + make_table_column(0, "planner_carrier", std::make_shared())); + projected_columns.push_back(make_table_column(1, "rf_key", std::make_shared())); + set_name_identifiers(&projected_columns); + + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + auto fake_state = std::make_shared(); + fake_state->unsupported_projection = LocalColumnId(0); + FakeTableReader reader(file_schema, fake_state); + ASSERT_TRUE(reader.init({ + .projected_columns = projected_columns, + .conjuncts = {}, + .format = FileFormat::PARQUET, + .scan_params = nullptr, + .io_ctx = nullptr, + .runtime_state = &state, + .scanner_profile = nullptr, + .push_down_agg_type = TPushAggOp::type::COUNT, + .push_down_count_columns = std::vector {}, + }) + .ok()); + + SplitReadOptions split_options; + split_options.current_range.__set_path("fake-table-reader-input"); + split_options.all_runtime_filters_applied = false; + ASSERT_TRUE(reader.prepare_split(split_options).ok()); + + VExprContextSPtrs refreshed {VExprContext::create_shared( + runtime_filter_wrapper_expr(table_int32_greater_than_expr(1, 1, 1)))}; + ASSERT_TRUE(reader.refresh_conjuncts(std::move(refreshed), std::nullopt, + /*all_runtime_filters_applied=*/true) + .ok()); + + // create_next_reader() establishes the concrete-reader lifecycle that open_reader() requires. + bool eos = false; + ASSERT_TRUE(reader.create_next_reader(&eos).ok()); + ASSERT_FALSE(eos); + ASSERT_NE(fake_state->last_request, nullptr); + EXPECT_TRUE(fake_state->last_request->is_count_star_placeholder(LocalColumnId(0))); + EXPECT_FALSE(fake_state->last_request->is_count_star_placeholder(LocalColumnId(1))); + ASSERT_TRUE(reader.close().ok()); +} + +TEST(TableReaderTest, FinalRuntimeFilterKeepsResidualPredicateInputMaterialized) { + std::vector file_schema; + file_schema.push_back(make_file_column(0, "residual_key", std::make_shared())); + file_schema.push_back(make_file_column(1, "rf_key", std::make_shared())); + + std::vector projected_columns; + projected_columns.push_back( + make_table_column(0, "residual_key", std::make_shared())); + projected_columns.push_back(make_table_column(1, "rf_key", std::make_shared())); + set_name_identifiers(&projected_columns); + + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + auto residual = prepared_conjunct(&state, std::make_shared(0)); + auto fake_state = std::make_shared(); + FakeTableReader reader(file_schema, fake_state); + ASSERT_TRUE(reader.init({ + .projected_columns = projected_columns, + .conjuncts = {residual}, + .format = FileFormat::PARQUET, + .scan_params = nullptr, + .io_ctx = nullptr, + .runtime_state = &state, + .scanner_profile = nullptr, + .push_down_agg_type = TPushAggOp::type::COUNT, + .push_down_count_columns = std::vector {}, + }) + .ok()); + + SplitReadOptions split_options; + split_options.current_range.__set_path("fake-table-reader-input"); + split_options.all_runtime_filters_applied = false; + ASSERT_TRUE(reader.prepare_split(split_options).ok()); + + VExprContextSPtrs refreshed {residual, VExprContext::create_shared(runtime_filter_wrapper_expr( + table_int32_greater_than_expr(1, 1, 1)))}; + ASSERT_TRUE(reader.refresh_conjuncts(std::move(refreshed), std::nullopt, + /*all_runtime_filters_applied=*/true) + .ok()); + // Keep the late-filter test on the same create/init/open lifecycle used by FileScannerV2. + bool eos = false; + ASSERT_TRUE(reader.create_next_reader(&eos).ok()); + ASSERT_FALSE(eos); + ASSERT_NE(fake_state->last_request, nullptr); + EXPECT_FALSE(fake_state->last_request->is_count_star_placeholder(LocalColumnId(0))); + EXPECT_FALSE(fake_state->last_request->is_count_star_placeholder(LocalColumnId(1))); + ASSERT_TRUE(reader.close().ok()); +} + +TEST(TableReaderTest, SinglePhysicalSplitUsesFinalRuntimeFilterPlaceholderSnapshot) { + std::vector file_schema; + file_schema.push_back( + make_file_column(0, "planner_carrier", std::make_shared())); + file_schema.push_back(make_file_column(1, "rf_key", std::make_shared())); + + std::vector projected_columns; + projected_columns.push_back( + make_table_column(0, "planner_carrier", std::make_shared())); + projected_columns.push_back(make_table_column(1, "rf_key", std::make_shared())); + set_name_identifiers(&projected_columns); + + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + auto fake_state = std::make_shared(); + fake_state->physical_split_count = 1; + fake_state->unsupported_projection = LocalColumnId(0); + FakeTableReader reader(file_schema, fake_state); + ASSERT_TRUE(reader.init({ + .projected_columns = projected_columns, + .conjuncts = {}, + .format = FileFormat::PARQUET, + .scan_params = nullptr, + .io_ctx = nullptr, + .runtime_state = &state, + .scanner_profile = nullptr, + .push_down_agg_type = TPushAggOp::type::COUNT, + .push_down_count_columns = std::vector {}, + }) + .ok()); + + SplitReadOptions split_options; + split_options.current_range.__set_path("fake-table-reader-input"); + split_options.conjuncts = VExprContextSPtrs {VExprContext::create_shared( + runtime_filter_wrapper_expr(table_int32_greater_than_expr(1, 1, 1)))}; + split_options.all_runtime_filters_applied = true; + ASSERT_TRUE(reader.prepare_split(split_options).ok()); + + FileScanSplit source_split; + source_split.range = split_options.current_range; + source_split.is_source_split = true; + std::vector children; + bool was_split = false; + ASSERT_TRUE(reader.build_physical_splits(source_split, &children, &was_split).ok()); + EXPECT_FALSE(was_split); + ASSERT_NE(fake_state->last_request, nullptr); + EXPECT_TRUE(fake_state->last_request->is_count_star_placeholder(LocalColumnId(0))); + EXPECT_FALSE(fake_state->last_request->is_count_star_placeholder(LocalColumnId(1))); + ASSERT_TRUE(reader.close().ok()); +} + TEST(TableReaderTest, PendingRuntimeFilterDisablesMinMaxPushdown) { const auto test_dir = std::filesystem::temp_directory_path() / "doris_table_reader_pending_rf_minmax_test"; @@ -3977,6 +4580,221 @@ TEST(TableReaderTest, ConditionCacheAllowsRuntimeFilterCoveredBySplitDigest) { ASSERT_TRUE(reader.close().ok()); } +TEST(TableReaderTest, ConditionCacheRefinedChildrenPublishOneSourceRangeEntry) { + ScopedConditionCacheForTest cache; + std::vector file_schema; + file_schema.push_back(make_file_column(0, "id", std::make_shared())); + std::vector projected_columns; + projected_columns.push_back(make_table_column(0, "id", std::make_shared())); + set_name_identifiers(&projected_columns); + + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + auto predicate = prepared_conjunct(&state, table_int32_greater_than_expr(0, 0, 0)); + auto split_context = std::make_shared(2); + constexpr int64_t FILE_SIZE = 4096; + constexpr int64_t MTIME = 17; + constexpr uint64_t DIGEST = 23; + segment_v2::ConditionCache::ExternalCacheKey source_key( + "fake-table-reader-input", MTIME, FILE_SIZE, DIGEST, 0, FILE_SIZE, + segment_v2::ConditionCache::ExternalCacheKey::BASE_GRANULE_AWARE_VERSION); + + for (int child = 0; child < 2; ++child) { + auto fake_state = std::make_shared(); + fake_state->total_rows = ConditionCacheContext::GRANULE_SIZE; + fake_state->condition_cache_base_granule = child; + fake_state->condition_cache_num_granules = 1; + FakeTableReader reader(file_schema, fake_state); + ASSERT_TRUE(reader.init({ + .projected_columns = projected_columns, + .conjuncts = {predicate}, + .format = FileFormat::PARQUET, + .scan_params = nullptr, + .io_ctx = nullptr, + .runtime_state = &state, + .scanner_profile = nullptr, + .condition_cache_digest = DIGEST, + }) + .ok()); + SplitReadOptions options; + options.current_range.__set_path("fake-table-reader-input"); + options.current_range.__set_file_size(FILE_SIZE); + options.current_range.__set_modification_time(MTIME); + options.current_range.__set_start_offset(100 + child * 100); + options.current_range.__set_size(100); + options.condition_cache_digest = DIGEST; + options.condition_cache_source_range = std::pair(0, FILE_SIZE); + options.condition_cache_split_context = split_context; + ASSERT_TRUE(reader.prepare_split(options).ok()); + + Block block = build_table_block(projected_columns); + bool eos = false; + ASSERT_TRUE(reader.get_block(&block, &eos).ok()); + segment_v2::ConditionCacheHandle handle; + EXPECT_EQ(cache.get()->lookup(source_key, &handle), child == 1); + ASSERT_TRUE(reader.close().ok()); + } + + segment_v2::ConditionCacheHandle handle; + ASSERT_TRUE(cache.get()->lookup(source_key, &handle)); + ASSERT_NE(handle.get_filter_result(), nullptr); + EXPECT_EQ(*handle.get_filter_result(), std::vector({true, true})); + EXPECT_EQ(handle.get_base_granule(), 0); +} + +TEST(TableReaderTest, ConditionCacheSerialEntryIsReusedByRefinedChild) { + ScopedConditionCacheForTest cache; + std::vector file_schema; + file_schema.push_back(make_file_column(0, "id", std::make_shared())); + std::vector projected_columns; + projected_columns.push_back(make_table_column(0, "id", std::make_shared())); + set_name_identifiers(&projected_columns); + + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + auto predicate = prepared_conjunct(&state, table_int32_greater_than_expr(0, 0, 0)); + constexpr int64_t FILE_SIZE = 4096; + constexpr int64_t MTIME = 29; + constexpr uint64_t DIGEST = 31; + auto serial_state = std::make_shared(); + serial_state->total_rows = ConditionCacheContext::GRANULE_SIZE * 2; + serial_state->condition_cache_num_granules = 2; + FakeTableReader serial_reader(file_schema, serial_state); + ASSERT_TRUE(serial_reader + .init({ + .projected_columns = projected_columns, + .conjuncts = {predicate}, + .format = FileFormat::PARQUET, + .scan_params = nullptr, + .io_ctx = nullptr, + .runtime_state = &state, + .scanner_profile = nullptr, + .condition_cache_digest = DIGEST, + }) + .ok()); + SplitReadOptions serial_options; + serial_options.current_range.__set_path("fake-table-reader-input"); + serial_options.current_range.__set_file_size(FILE_SIZE); + serial_options.current_range.__set_modification_time(MTIME); + serial_options.current_range.__set_start_offset(0); + serial_options.current_range.__set_size(FILE_SIZE); + serial_options.condition_cache_digest = DIGEST; + ASSERT_TRUE(serial_reader.prepare_split(serial_options).ok()); + Block block = build_table_block(projected_columns); + bool eos = false; + ASSERT_TRUE(serial_reader.get_block(&block, &eos).ok()); + ASSERT_TRUE(serial_reader.close().ok()); + + auto child_state = std::make_shared(); + child_state->total_rows = ConditionCacheContext::GRANULE_SIZE; + FakeTableReader child_reader(file_schema, child_state); + ASSERT_TRUE(child_reader + .init({ + .projected_columns = projected_columns, + .conjuncts = {predicate}, + .format = FileFormat::PARQUET, + .scan_params = nullptr, + .io_ctx = nullptr, + .runtime_state = &state, + .scanner_profile = nullptr, + .condition_cache_digest = DIGEST, + }) + .ok()); + SplitReadOptions child_options = serial_options; + child_options.current_range.__set_start_offset(100); + child_options.current_range.__set_size(100); + child_options.condition_cache_source_range = std::pair(0, FILE_SIZE); + child_options.condition_cache_split_context = std::make_shared(1); + ASSERT_TRUE(child_reader.prepare_split(child_options).ok()); + block = build_table_block(projected_columns); + eos = false; + ASSERT_TRUE(child_reader.get_block(&block, &eos).ok()); + EXPECT_EQ(child_reader.condition_cache_hit_count(), 1); + ASSERT_TRUE(child_reader.close().ok()); +} + +TEST(TableReaderTest, ConditionCacheLateRefreshBeforeOpenUsesRefreshedDigest) { + ScopedConditionCacheForTest cache; + std::vector file_schema; + file_schema.push_back(make_file_column(0, "id", std::make_shared())); + std::vector projected_columns; + projected_columns.push_back(make_table_column(0, "id", std::make_shared())); + set_name_identifiers(&projected_columns); + + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + auto fake_state = std::make_shared(); + fake_state->total_rows = ConditionCacheContext::GRANULE_SIZE; + FakeTableReader reader(file_schema, fake_state); + ASSERT_TRUE(reader.init({ + .projected_columns = projected_columns, + .conjuncts = {prepared_conjunct( + &state, table_int32_greater_than_expr(0, 0, 0))}, + .format = FileFormat::PARQUET, + .scan_params = nullptr, + .io_ctx = nullptr, + .runtime_state = &state, + .scanner_profile = nullptr, + .condition_cache_digest = 7, + }) + .ok()); + + SplitReadOptions split_options; + split_options.current_range.__set_path("late-runtime-filter-input"); + split_options.condition_cache_digest = 11; + ASSERT_TRUE(reader.prepare_split(split_options).ok()); + VExprContextSPtrs refreshed { + prepared_conjunct(&state, table_int32_greater_than_expr(0, 0, 0)), + prepared_conjunct(&state, + runtime_filter_wrapper_expr(table_int32_greater_than_expr(0, 0, 1)))}; + ASSERT_TRUE(reader.refresh_conjuncts(std::move(refreshed), 19).ok()); + + Block block = build_table_block(projected_columns); + bool eos = false; + ASSERT_TRUE(reader.get_block(&block, &eos).ok()); + segment_v2::ConditionCacheHandle handle; + segment_v2::ConditionCache::ExternalCacheKey stale_key( + "late-runtime-filter-input", 0, -1, 11, 0, -1, + segment_v2::ConditionCache::ExternalCacheKey::BASE_GRANULE_AWARE_VERSION); + EXPECT_FALSE(cache.get()->lookup(stale_key, &handle)); + segment_v2::ConditionCache::ExternalCacheKey refreshed_key( + "late-runtime-filter-input", 0, -1, 19, 0, -1, + segment_v2::ConditionCache::ExternalCacheKey::BASE_GRANULE_AWARE_VERSION); + EXPECT_TRUE(cache.get()->lookup(refreshed_key, &handle)); + ASSERT_TRUE(reader.close().ok()); + + auto hit_state = std::make_shared(); + hit_state->total_rows = ConditionCacheContext::GRANULE_SIZE; + FakeTableReader hit_reader(file_schema, hit_state); + ASSERT_TRUE( + hit_reader + .init({ + .projected_columns = projected_columns, + .conjuncts = + { + prepared_conjunct( + &state, table_int32_greater_than_expr(0, 0, 0)), + prepared_conjunct(&state, + runtime_filter_wrapper_expr( + table_int32_greater_than_expr( + 0, 0, 1))), + }, + .format = FileFormat::PARQUET, + .scan_params = nullptr, + .io_ctx = nullptr, + .runtime_state = &state, + .scanner_profile = nullptr, + .condition_cache_digest = 7, + }) + .ok()); + split_options.condition_cache_digest = 19; + ASSERT_TRUE(hit_reader.prepare_split(split_options).ok()); + block = build_table_block(projected_columns); + eos = false; + ASSERT_TRUE(hit_reader.get_block(&block, &eos).ok()); + ASSERT_NE(hit_state->condition_cache_ctx, nullptr); + EXPECT_TRUE(hit_state->condition_cache_ctx->is_hit); + EXPECT_EQ(hit_reader.condition_cache_hit_count(), 1); + ASSERT_TRUE(hit_reader.close().ok()); +} + // Scenario: table-format delete files/deletion vectors are outside the data-file cache key. When // TableReader injects delete conjuncts into the file scan request, condition cache must be disabled // for that split. diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java index 1355ef75af9a8d..e19c4842086a12 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java @@ -144,7 +144,8 @@ public List planScan(ConnectorSession session, ConnectorScan String memoKey = SCAN_REUSE_NAMESPACE + ":" + session.getCatalogId() + ":" + session.getQueryId(); Map> scanReuse = session.getStatementScope().computeIfAbsentForPlanning(memoKey, () -> new ConcurrentHashMap<>()); - HiveScanReuseKey reuseKey = new HiveScanReuseKey(hiveHandle, request.getRequiredPartitions()); + HiveScanReuseKey reuseKey = new HiveScanReuseKey( + hiveHandle, request.getRequiredPartitions(), coarseSplitSize(request)); AtomicReference> uncached = new AtomicReference<>(); List cached = scanReuse.computeIfAbsent(reuseKey, key -> { PlanCompleteness completeness = new PlanCompleteness(); @@ -201,7 +202,8 @@ private List doPlanScan(ConnectorSession session, ConnectorS HiveFileFormat partFormat = partition.fileFormat != null ? partition.fileFormat : fileFormat; listAndSplitFiles(dbName, tableName, partition, partFormat, - splittable, isLzo, targetSplitSize, fs, ranges, completeness, request.getScanContext()); + splittable, isLzo, selectFeSplitSize(targetSplitSize, partFormat, coarseSplitSize(request)), + fs, ranges, completeness, request.getScanContext()); } LOG.info("Hive scan plan: table={}.{}, partitions={}, splits={}", @@ -276,7 +278,8 @@ private long cachedFileCount(HiveTableHandle handle) { public ConnectorSplitSource streamSplits(ConnectorSession session, ConnectorScanRequest request) { HiveTableHandle hiveHandle = (HiveTableHandle) request.getTableHandle(); rejectTransactionalRead(hiveHandle); - return new HiveSplitSource(session, hiveHandle, request.getRequiredPartitions(), request.getScanContext()); + return new HiveSplitSource(session, hiveHandle, request.getRequiredPartitions(), request.getScanContext(), + coarseSplitSize(request)); } @Override @@ -309,7 +312,7 @@ public boolean hasNext() { } if (source == null) { source = new HiveSplitSource(session, handle, names, request.getScanContext(), - partitionsPerTask); + coarseSplitSize(request), partitionsPerTask); source.partitionLoader = loader; } return source.hasNext(); @@ -378,6 +381,7 @@ private final class HiveSplitSource implements ConnectorSplitSource { private final boolean isLzo; private final boolean splittable; private final long targetSplitSize; + private final long coarseSplitSize; private final int partitionBatchSize; private HivePartitionBatchLoader partitionLoader; private CompletableFuture blockedOnPartitions; @@ -392,13 +396,13 @@ private final class HiveSplitSource implements ConnectorSplitSource { private volatile boolean closed; HiveSplitSource(ConnectorSession session, HiveTableHandle handle, List requiredPartitions, - ConnectorScanContext scanContext) { - this(session, handle, requiredPartitions, scanContext, Math.max(1, + ConnectorScanContext scanContext, long coarseSplitSize) { + this(session, handle, requiredPartitions, scanContext, coarseSplitSize, Math.max(1, sessionInt(session, "split_partition_batch_size", DEFAULT_PARTITION_BATCH_SIZE))); } HiveSplitSource(ConnectorSession session, HiveTableHandle handle, List requiredPartitions, - ConnectorScanContext scanContext, int partitionsPerTask) { + ConnectorScanContext scanContext, long coarseSplitSize, int partitionsPerTask) { rejectTransactionalRead(handle); this.handle = handle; this.scanContext = scanContext; @@ -411,6 +415,7 @@ private final class HiveSplitSource implements ConnectorSplitSource { this.isLzo = isLzoInputFormat(handle.getInputFormat()); this.splittable = tableFormat.isSplittable() && !isLzo; this.targetSplitSize = getTargetSplitSize(session); + this.coarseSplitSize = coarseSplitSize; this.partitionBatchSize = partitionsPerTask; recordPruningProfile(handle); } @@ -483,7 +488,7 @@ private void loadNextFileRanges() { } HiveFileFormat format = partition.fileFormat == null ? tableFormat : partition.fileFormat; ranges = splitFileRanges(file.getPath(), file.getLength(), file.getModificationTime(), - partition, format, splittable, targetSplitSize); + partition, format, splittable, selectFeSplitSize(targetSplitSize, format, coarseSplitSize)); } private void loadNextPartitionFiles() { @@ -625,7 +630,8 @@ public List planScanForPartitionBatch( } List ranges = new ArrayList<>(); try (HiveSplitSource source = new HiveSplitSource( - session, (HiveTableHandle) request.getTableHandle(), partitionBatch, request.getScanContext())) { + session, (HiveTableHandle) request.getTableHandle(), partitionBatch, request.getScanContext(), + coarseSplitSize(request))) { while (source.hasNext()) { ranges.add(source.next()); } @@ -1148,6 +1154,15 @@ private static String formatNanos(long nanos) { * pruned partition set (each partition's location and values). ACID tables are excluded * upstream, and session variables are statement-constant, so both stay out of the key. */ + private static long coarseSplitSize(ConnectorScanRequest request) { + return request.isCountPushdown() ? -1 : request.getCoarseSplitSize(); + } + + static long selectFeSplitSize(long fallbackSize, HiveFileFormat format, long coarseSize) { + return coarseSize > 0 && (format == HiveFileFormat.PARQUET || format == HiveFileFormat.ORC) + ? coarseSize : fallbackSize; + } + private static final class HiveScanReuseKey { private final String dbName; private final String tableName; @@ -1158,8 +1173,10 @@ private static final class HiveScanReuseKey { private final List partitionKeyNames; private final List prunedPartitions; private final List requiredPartitions; + private final long coarseSplitSize; - private HiveScanReuseKey(HiveTableHandle handle, List requiredPartitions) { + private HiveScanReuseKey(HiveTableHandle handle, List requiredPartitions, long coarseSplitSize) { + this.coarseSplitSize = coarseSplitSize; // Catalog and query isolation are provided by the statement-scope memo key. The table // location identifies the data source of unpartitioned tables, whose prunedPartitions // is null. @@ -1187,7 +1204,7 @@ public boolean equals(Object object) { return false; } HiveScanReuseKey that = (HiveScanReuseKey) object; - return firstColumnIsString == that.firstColumnIsString + return coarseSplitSize == that.coarseSplitSize && firstColumnIsString == that.firstColumnIsString && Objects.equals(dbName, that.dbName) && Objects.equals(tableName, that.tableName) && Objects.equals(location, that.location) @@ -1202,7 +1219,7 @@ public boolean equals(Object object) { public int hashCode() { return Objects.hash(dbName, tableName, location, inputFormat, serializationLib, firstColumnIsString, - partitionKeyNames, prunedPartitions, requiredPartitions); + partitionKeyNames, prunedPartitions, requiredPartitions, coarseSplitSize); } @Override diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanRange.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanRange.java index 6995ed4366cc2c..65f34588836ef0 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanRange.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanRange.java @@ -105,6 +105,11 @@ public String getFileFormat() { return fileFormat; } + @Override + public boolean supportsBeSplitRefinement() { + return "parquet".equals(fileFormat) || "orc".equals(fileFormat); + } + @Override public String getTableFormatType() { return tableFormatType; diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveScanBatchModeTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveScanBatchModeTest.java index 4a0caa0deea8e4..faf9bd9a11fdfa 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveScanBatchModeTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveScanBatchModeTest.java @@ -102,6 +102,61 @@ public class HiveScanBatchModeTest { private static final String LAZY_SIMPLE_SERDE = "org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"; private static final String OPEN_CSV_SERDE = "org.apache.hadoop.hive.serde2.OpenCSVSerde"; + @Test + public void statementReuseSeparatesScanLocalCoarseSizes() { + HiveScanPlanProvider provider = provider(null, (location, fs) -> Collections.singletonList( + new HiveFileStatus(location + "/large.parquet", 35, 1))); + ConnectorSession session = new ScopeSession(7L, "coarse-size", new TestStatementScope(), + Map.of("enable_external_scan_task_reuse", "true", "file_split_size", "8")); + HiveTableHandle handle = parquetHandle(Collections.emptyList()); + ConnectorScanRequest smaller = ConnectorScanRequest.builder(handle, Collections.emptyList()) + .coarseSplitSize(16).build(); + List first = provider.planScan(session, smaller); + List second = provider.planScan(session, + ConnectorScanRequest.builder(handle, Collections.emptyList()).coarseSplitSize(32).build()); + Assertions.assertEquals(3, first.size()); + Assertions.assertEquals(2, second.size()); + Assertions.assertSame(first, provider.planScan(session, smaller)); + } + + @Test + public void coarseSizesReachSynchronousBatchAndStreamingPlans() throws Exception { + HiveScanPlanProvider provider = provider(new FakeHmsClient(), (location, fs) -> Collections.singletonList( + new HiveFileStatus(location + "/large.parquet", 35, 1))); + FakeSession session = new FakeSession(Collections.singletonMap("file_split_size", "8")); + String partition = "year=2024/month=01"; + HiveTableHandle handle = parquetHandle(PART_KEYS).toBuilder() + .prunedPartitions(Collections.singletonList(part(partition))).build(); + ConnectorScanRequest request = ConnectorScanRequest.builder(handle, Collections.emptyList()) + .requiredPartitions(Collections.singletonList(partition)).coarseSplitSize(16).build(); + List synchronous = provider.planScan(session, request); + List batch = provider.planScanForPartitionBatch(session, request, + Collections.singletonList(partition)); + List streaming = new ArrayList<>(); + try (ConnectorSplitSource source = provider.streamSplits(session, request)) { + while (source.hasNext()) { + streaming.add(source.next()); + } + } + List cooperative = new ArrayList<>(); + for (ConnectorSplitSource source : provider.streamPartitionSources(session, request, 1, 1).get()) { + try (ConnectorSplitSource current = source) { + while (current.hasNext()) { + cooperative.add(current.next()); + } + } + } + for (List ranges : Arrays.asList(synchronous, batch, streaming, cooperative)) { + Assertions.assertEquals(3, ranges.size()); + Assertions.assertEquals(16, ranges.get(0).getLength()); + Assertions.assertEquals(16, ranges.get(1).getStart()); + Assertions.assertEquals(3, ranges.get(2).getLength()); + Assertions.assertTrue(ranges.get(0).supportsBeSplitRefinement()); + } + Assertions.assertEquals(8, HiveScanPlanProvider.selectFeSplitSize(8, HiveFileFormat.TEXT, 16)); + Assertions.assertEquals(16, HiveScanPlanProvider.selectFeSplitSize(8, HiveFileFormat.ORC, 16)); + } + @Test public void completedMetadataDependencyMustNotLookLikeEndOfInput() throws Exception { Queue metadata = new ArrayDeque<>(); diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanRange.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanRange.java index ba268e8e22e1ca..8252e2477c40ba 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanRange.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanRange.java @@ -163,6 +163,11 @@ public String toString() { + ", start=" + start + ", length=" + length + "}"; } + @Override + public boolean supportsBeSplitRefinement() { + return deltaLogs.isEmpty() && !forceJni; + } + @Override public void populateRangeParams(TTableFormatFileDesc formatDesc, TFileRangeDesc rangeDesc) { diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java index 8b243d3147009b..74434f2f7ae50b 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java @@ -44,6 +44,8 @@ import org.apache.doris.thrift.TTableFormatFileDesc; import com.google.common.annotations.VisibleForTesting; +import com.google.common.base.Preconditions; +import com.google.common.collect.FluentIterable; import org.apache.iceberg.BaseFileScanTask; import org.apache.iceberg.BaseTable; import org.apache.iceberg.BatchScan; @@ -433,12 +435,13 @@ public List planScan(ConnectorSession session, ConnectorScan validateMetadataColumnReader(session, request.getColumns()); if (session == null || !session.isExternalScanTaskReuseEnabled()) { return rejectRowIdScanWithoutPartitionData(request.getColumns(), planScanInternal(session, - icebergHandle, request.getColumns(), request.getFilter(), request.isCountPushdown())); + icebergHandle, request.getColumns(), request.getFilter(), request.isCountPushdown(), + request.getCoarseSplitSize())); } if (icebergHandle.isSystemTable()) { // System tables read connector metadata through their own readers; never reuse them. return planScanInternal(session, icebergHandle, request.getColumns(), - request.getFilter(), request.isCountPushdown()); + request.getFilter(), request.isCountPushdown(), request.getCoarseSplitSize()); } // Statement-scoped reuse: within one statement the identical scan (same table, same // snapshot/ref/schema pin, same filter, same COUNT pushdown) plans once and every @@ -463,7 +466,7 @@ public List planScan(ConnectorSession session, ConnectorScan return rejectRowIdScanWithoutPartitionData(request.getColumns(), scanReuse.computeIfAbsent(reuseKey, k -> Collections.unmodifiableList(planScanInternal(session, icebergHandle, request.getColumns(), request.getFilter(), - request.isCountPushdown())))); + request.isCountPushdown(), request.getCoarseSplitSize())))); } catch (RuntimeException e) { // Normal data scans and native position_deletes run on File Scanner V2. Keep the serialized JNI // system-table route untouched because its deferred reads belong to the V1 scanner contract. @@ -595,9 +598,10 @@ public ConnectorSplitSource streamSplits(ConnectorSession session, ConnectorScan try { ManifestReadControl readControl = new ManifestReadControl(false); CloseableIterable tasks = streamingFileScanTasks( - scan, session, table, filter, sliceSize, readControl, iceHandle, request.getScanContext()); + scan, session, table, filter, sliceSize, request.getCoarseSplitSize(), + readControl, iceHandle, request.getScanContext()); return new IcebergStreamingSplitSource(tasks, table, scanSchema, formatVersion, partitioned, - orderedPartitionKeys, zone, uriNormalizer, sliceSize, + orderedPartitionKeys, zone, uriNormalizer, sliceSize, request.getCoarseSplitSize(), iceHandle.getRewriteFileScope(), iceHandle, readControl, requiresRowId(columns), request.getScanContext()); } catch (RuntimeException e) { @@ -632,15 +636,15 @@ public void close() { * streaming source's {@code hasNext}. Cache disabled -> the SDK path, byte-unchanged. */ private CloseableIterable streamingFileScanTasks(TableScan scan, ConnectorSession session, - Table table, Optional filter, long sliceSize, + Table table, Optional filter, long sliceSize, long coarseSplitSize, ManifestReadControl readControl, IcebergTableHandle readHandle, ConnectorScanContext scanContext) { scanContext.checkCancelled(); if (isManifestCacheEnabled()) { try { - return TableScanUtil.splitFiles( + return splitFileTasks( cacheBackedFileScanTasks(scan, session, table, filter, null, readControl, readHandle, scanContext), - sliceSize); + sliceSize, coarseSplitSize); } catch (CancellationException e) { // A stopped query must not restart file planning through the SDK fallback. throw e; @@ -653,7 +657,7 @@ private CloseableIterable streamingFileScanTasks(TableScan scan, C } } scanContext.checkCancelled(); - return TableScanUtil.splitFiles(sdkStreamingFileScanTasks(scan, scanContext), sliceSize); + return splitFileTasks(sdkStreamingFileScanTasks(scan, scanContext), sliceSize, coarseSplitSize); } private CloseableIterable sdkStreamingFileScanTasks(TableScan scan, @@ -754,6 +758,7 @@ private final class IcebergStreamingSplitSource implements ConnectorSplitSource private final ZoneId zone; private final UnaryOperator uriNormalizer; private final long sliceSize; + private final long coarseSplitSize; private final Set rewriteScope; private final IcebergTableHandle handle; private final boolean requiresRowId; @@ -775,7 +780,7 @@ private final class IcebergStreamingSplitSource implements ConnectorSplitSource IcebergStreamingSplitSource(CloseableIterable tasks, Table table, Schema scanSchema, int formatVersion, boolean partitioned, List orderedPartitionKeys, ZoneId zone, - UnaryOperator uriNormalizer, long sliceSize, Set rewriteScope, + UnaryOperator uriNormalizer, long sliceSize, long coarseSplitSize, Set rewriteScope, IcebergTableHandle handle, ManifestReadControl readControl, boolean requiresRowId, ConnectorScanContext scanContext) { this.tasks = tasks; @@ -789,6 +794,7 @@ private final class IcebergStreamingSplitSource implements ConnectorSplitSource this.zone = zone; this.uriNormalizer = uriNormalizer; this.sliceSize = sliceSize; + this.coarseSplitSize = coarseSplitSize; this.rewriteScope = rewriteScope; this.handle = handle; this.requiresRowId = requiresRowId; @@ -820,7 +826,8 @@ public boolean hasNext() { scanContext.checkCancelled(); IcebergScanRange range = buildRangeForTask( iterator.next(), table, scanSchema, formatVersion, partitioned, - orderedPartitionKeys, zone, uriNormalizer, sliceSize, rewriteScope, null, scratch); + orderedPartitionKeys, zone, uriNormalizer, sliceSize, coarseSplitSize, + rewriteScope, null, scratch); if (range != null) { if (requiresRowId) { rejectRowIdRangeWithoutPartitionData(range); @@ -868,7 +875,7 @@ private List planScanInternal( ConnectorTableHandle handle, List columns, Optional filter, - boolean countPushdown) { + boolean countPushdown, long coarseSplitSize) { IcebergTableHandle iceHandle = (IcebergTableHandle) handle; if (iceHandle.isResolvedEmptySnapshot() && !iceHandle.isSystemTable()) { // Empty pins still need admission: otherwise COUNT/empty scans bypass the encryption fence. @@ -961,13 +968,13 @@ private List planScanInternal( // once per file, not per slice. The streaming source below holds its own. PerFileScratch scratch = new PerFileScratch(); List ranges = new ArrayList<>(); - try (SplitPlan plan = planFileScanTask(scan, session, table, filter, iceHandle)) { + try (SplitPlan plan = planFileScanTask(scan, session, table, filter, iceHandle, coarseSplitSize)) { for (FileScanTask task : plan.tasks) { // Shared per-task mapping (rewrite-scope skip + M-2 weight denominator + v3 stash side-effect), // identical to the streaming path's IcebergStreamingSplitSource so both produce the same ranges. IcebergScanRange range = buildRangeForTask( task, table, scanSchema, formatVersion, partitioned, orderedPartitionKeys, zone, - uriNormalizer, plan.targetSplitSize, rewriteScope, rewritableDeleteSupply, + uriNormalizer, plan.targetSplitSize, coarseSplitSize, rewriteScope, rewritableDeleteSupply, scratch); if (range != null) { ranges.add(range); @@ -1026,7 +1033,7 @@ private static String rootCauseMessage(Throwable t) { */ private IcebergScanRange buildRangeForTask(FileScanTask task, Table table, Schema scanSchema, int formatVersion, boolean partitioned, List orderedPartitionKeys, ZoneId zone, - UnaryOperator uriNormalizer, long targetSplitSize, Set rewriteScope, + UnaryOperator uriNormalizer, long targetSplitSize, long coarseSplitSize, Set rewriteScope, Map> rewritableDeleteSupply, PerFileScratch scratch) { DataFile dataFile = task.file(); if (rewriteScope != null && !rewriteScope.contains(dataFile.path().toString())) { @@ -1037,11 +1044,11 @@ private IcebergScanRange buildRangeForTask(FileScanTask task, Table table, Schem // for every slice of a file, so record it exactly ONCE per file — on the file's first slice, keyed to // the new file, so the LAST file in the stream is never dropped. boolean firstSliceOfFile = scratch.file != dataFile; - // targetSplitSize is the scan-level weight denominator (M-2): each data-file range carries a - // size-proportional BE scheduling weight (selfSplitWeight computed inside buildRange). + // Use the same per-file target for splitting and proportional scheduling. A file with + // deletes retains its fallback size even when its neighbors use coarse FE ranges. IcebergScanRange range = buildRange( table, scanSchema, dataFile, task, formatVersion, partitioned, orderedPartitionKeys, - zone, uriNormalizer, -1, targetSplitSize, scratch); + zone, uriNormalizer, -1, selectFeSplitSize(task, targetSplitSize, coarseSplitSize), scratch); if (rewritableDeleteSupply != null && firstSliceOfFile) { // Record this data file's non-equality delete supply keyed on its RAW path (the exact string the BE // matches a rewritable set against). An empty list (no old non-eq deletes) contributes nothing. @@ -3135,14 +3142,29 @@ public void close() throws IOException { * (M-2): the forced {@code file_split_size} (that path slices to it) or the heuristic {@code targetSplitSize} * (legacy {@code IcebergScanNode.targetSplitSize}, the same value used to slice — legacy reuses it for both). */ - private SplitPlan splitFiles(TableScan scan, ConnectorSession session) { + static long selectFeSplitSize(FileScanTask task, long fallbackSize, long coarseSize) { + if (coarseSize <= 0 || !task.deletes().isEmpty()) { + return fallbackSize; + } + FileFormat format = task.file().format(); + return format == FileFormat.PARQUET || format == FileFormat.ORC ? coarseSize : fallbackSize; + } + + private static CloseableIterable splitFileTasks( + CloseableIterable tasks, long fallbackSize, long coarseSize) { + Preconditions.checkArgument(fallbackSize > 0, "Split size must be positive: %s", fallbackSize); + // Preserve the SDK iterable's lazy enumeration and close ownership while selecting per file. + Iterable splits = FluentIterable.from(tasks) + .transformAndConcat(task -> task.split(selectFeSplitSize(task, fallbackSize, coarseSize))); + return CloseableIterable.combine(splits, tasks); + } + + private SplitPlan splitFiles(TableScan scan, ConnectorSession session, long coarseSplitSize) { long fileSplitSize = sessionLong(session, FILE_SPLIT_SIZE, 0L); if (fileSplitSize > 0) { - // The split granularity IS fileSplitSize, so it is the correct weight denominator. (Legacy left its - // targetSplitSize field at 0 here → divide-by-zero → weight clamped to 1.0; the generic - // PluginDrivenSplit guards target>0, so reproducing that is impossible without un-guarding division - // for all connectors. Using fileSplitSize gives proper proportional weighting — strictly better.) - return new SplitPlan(TableScanUtil.splitFiles(scan.planFiles(), fileSplitSize), fileSplitSize); + // This remains the fallback granularity and weight denominator for ineligible files. + // Eligible files select the dedicated coarse target for both splitting and weighting. + return new SplitPlan(splitFileTasks(scan.planFiles(), fileSplitSize, coarseSplitSize), fileSplitSize); } List fileScanTasks = new ArrayList<>(); try (CloseableIterable planned = scan.planFiles()) { @@ -3155,7 +3177,7 @@ private SplitPlan splitFiles(TableScan scan, ConnectorSession session) { } long targetSplitSize = determineTargetFileSplitSize(fileScanTasks, session); return new SplitPlan( - TableScanUtil.splitFiles(CloseableIterable.withNoopClose(fileScanTasks), targetSplitSize), + splitFileTasks(CloseableIterable.withNoopClose(fileScanTasks), targetSplitSize, coarseSplitSize), targetSplitSize); } @@ -3168,12 +3190,12 @@ private SplitPlan splitFiles(TableScan scan, ConnectorSession session) { */ private SplitPlan planFileScanTask(TableScan scan, ConnectorSession session, Table table, Optional filter, - IcebergTableHandle readHandle) { + IcebergTableHandle readHandle, long coarseSplitSize) { if (!isManifestCacheEnabled()) { - return splitFiles(scan, session); + return splitFiles(scan, session, coarseSplitSize); } try { - return planFileScanTaskWithManifestCache(scan, session, table, filter, readHandle); + return planFileScanTaskWithManifestCache(scan, session, table, filter, readHandle, coarseSplitSize); } catch (CancellationException e) { // A stopped query must not restart file planning through the SDK fallback. throw e; @@ -3182,7 +3204,7 @@ private SplitPlan planFileScanTask(TableScan scan, ConnectorSession session, Tab LOG.warn("Iceberg plan with manifest cache failed, falling back to SDK scan: {}", e.getMessage(), e); // Mirror the legacy manifestCacheFailures bump so VERBOSE EXPLAIN can report the fallback. manifestCache.recordFailure(session.getQueryId()); - return splitFiles(scan, session); + return splitFiles(scan, session, coarseSplitSize); } } @@ -3197,7 +3219,7 @@ private SplitPlan planFileScanTask(TableScan scan, ConnectorSession session, Tab */ private SplitPlan planFileScanTaskWithManifestCache(TableScan scan, ConnectorSession session, Table table, Optional filter, - IcebergTableHandle readHandle) throws IOException { + IcebergTableHandle readHandle, long coarseSplitSize) throws IOException { // Null-safe queryId (offline tests pass a null session): a null id selects the no-stats cache overload, // matching the pre-PERF-04 body, which read scan.snapshot() and returned early for an empty table BEFORE // ever calling session.getQueryId(). @@ -3211,7 +3233,8 @@ private SplitPlan planFileScanTaskWithManifestCache(TableScan scan, } long targetSplitSize = determineTargetFileSplitSize(tasks, session); return new SplitPlan( - TableScanUtil.splitFiles(CloseableIterable.withNoopClose(tasks), targetSplitSize), targetSplitSize); + splitFileTasks(CloseableIterable.withNoopClose(tasks), targetSplitSize, coarseSplitSize), + targetSplitSize); } /** @@ -3839,6 +3862,7 @@ private static final class IcebergScanReuseKey { private final List rewriteFileScope; private final Optional filter; private final boolean countPushdown; + private final long coarseSplitSize; private IcebergScanReuseKey(IcebergTableHandle handle, ConnectorScanRequest request) { this.dbName = handle.getDbName(); @@ -3854,6 +3878,7 @@ private IcebergScanReuseKey(IcebergTableHandle handle, ConnectorScanRequest requ .collect(Collectors.toList()); this.filter = request.getFilter(); this.countPushdown = request.isCountPushdown(); + this.coarseSplitSize = request.getCoarseSplitSize(); } @Override @@ -3868,6 +3893,7 @@ public boolean equals(Object object) { return snapshotId == that.snapshotId && schemaId == that.schemaId && countPushdown == that.countPushdown + && coarseSplitSize == that.coarseSplitSize && Objects.equals(dbName, that.dbName) && Objects.equals(tableName, that.tableName) && Objects.equals(ref, that.ref) @@ -3878,7 +3904,7 @@ public boolean equals(Object object) { @Override public int hashCode() { return Objects.hash(dbName, tableName, snapshotId, ref, schemaId, - rewriteFileScope, filter, countPushdown); + rewriteFileScope, filter, countPushdown, coarseSplitSize); } @Override diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanRange.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanRange.java index 4e296d96518738..7ac251d14ac4ce 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanRange.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanRange.java @@ -128,6 +128,11 @@ public class IcebergScanRange implements ConnectorScanRange { private final Long positionDeleteContentOffset; private final Long positionDeleteContentSizeInBytes; + @Override + public boolean supportsBeSplitRefinement() { + return serializedSplit == null && !positionDeleteSystemTableSplit && deleteFiles.isEmpty(); + } + private IcebergScanRange(Builder builder) { this.path = builder.path; // Default the raw original path to the (possibly already-raw) path when a caller does not split them diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java index f528c82c6d30d2..06e65bd868262c 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java @@ -151,6 +151,66 @@ public class IcebergScanPlanProviderTest { @TempDir Path tempDir; + @Test + public void selectFeSplitSizeRetainsLegacySizeForDeleteFilesAndUnsupportedFormats() throws Exception { + Table table = createTable("t1", SCHEMA, PartitionSpec.unpartitioned()); + DataFile file = dataFile(table.spec(), "s3://b/db/t1/data.parquet", 96L << 20, null, null); + table.newAppend().appendFile(file).commit(); + DeleteFile deletion = FileMetadata.deleteFileBuilder(table.spec()).ofPositionDeletes() + .withPath("s3://b/db/t1/delete.parquet").withFormat(FileFormat.PARQUET) + .withFileSizeInBytes(100).withRecordCount(1).build(); + table.newRowDelta().addDeletes(deletion).commit(); + try (org.apache.iceberg.io.CloseableIterable tasks = + table.newScan().planFiles()) { + org.apache.iceberg.FileScanTask task = tasks.iterator().next(); + Assertions.assertFalse(task.deletes().isEmpty()); + Assertions.assertEquals(16L << 20, + IcebergScanPlanProvider.selectFeSplitSize(task, 16L << 20, 64L << 20)); + } + Table avro = createTable("avro_table", SCHEMA, PartitionSpec.unpartitioned()); + avro.newAppend().appendFile(DataFiles.builder(avro.spec()).withPath("s3://b/db/a.avro") + .withFormat(FileFormat.AVRO).withFileSizeInBytes(100).withRecordCount(1).build()).commit(); + try (org.apache.iceberg.io.CloseableIterable tasks = + avro.newScan().planFiles()) { + Assertions.assertEquals(16L << 20, + IcebergScanPlanProvider.selectFeSplitSize(tasks.iterator().next(), 16L << 20, 64L << 20)); + } + } + + @Test + public void coarseSizesOverrideLegacySplitSizeInSynchronousAndStreamingPlans() throws Exception { + Table table = createTable("t1", SCHEMA, PartitionSpec.unpartitioned()); + long mb = 1024L * 1024L; + table.newAppend().appendFile(dataFile(table.spec(), "s3://b/db/t1/big.parquet", 96 * mb, + null, null)).commit(); + IcebergScanPlanProvider provider = new IcebergScanPlanProvider( + IcebergCatalogProperties.of(Collections.emptyMap()), opsReturning(table)); + ConnectorSession session = new FakeScanSession("UTC", Map.of( + "file_split_size", Long.toString(16 * mb), "enable_external_scan_task_reuse", "true")) + .withScope(new TestStatementScope()); + ConnectorScanRequest request = ConnectorScanRequest.builder( + new IcebergTableHandle("db1", "t1"), Collections.emptyList()).coarseSplitSize(64 * mb).build(); + List synchronous = provider.planScan(session, request); + List streaming = new ArrayList<>(); + try (ConnectorSplitSource source = provider.streamSplits(session, request)) { + while (source.hasNext()) { + streaming.add(source.next()); + } + } + List smaller = provider.planScan(session, ConnectorScanRequest.builder( + new IcebergTableHandle("db1", "t1"), Collections.emptyList()).coarseSplitSize(32 * mb).build()); + Assertions.assertEquals(3, smaller.size()); + Assertions.assertSame(synchronous, provider.planScan(session, request)); + for (List ranges : Arrays.asList(synchronous, streaming)) { + Assertions.assertEquals(2, ranges.size()); + Assertions.assertEquals(64 * mb, ranges.get(0).getLength()); + Assertions.assertEquals(64 * mb, ranges.get(0).getTargetSplitSize()); + Assertions.assertEquals(64 * mb, ranges.get(1).getStart()); + Assertions.assertEquals(32 * mb, ranges.get(1).getLength()); + Assertions.assertTrue(ranges.get(0).supportsBeSplitRefinement()); + } + } + @Test public void scanReuseNamespaceUsesConnectorType() { String prefix = new IcebergConnectorProvider().getType() + "."; diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanRangeTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanRangeTest.java index e657951100b4f6..4bb16a0b0e16e7 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanRangeTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanRangeTest.java @@ -42,6 +42,21 @@ */ public class IcebergScanRangeTest { + @Test + public void onlyOrdinaryDeleteFreeRangesSupportBeRefinement() { + IcebergScanRange normal = new IcebergScanRange.Builder().path("s3://b/data.parquet").build(); + Assertions.assertTrue(normal.supportsBeSplitRefinement()); + Assertions.assertFalse(new IcebergScanRange.Builder().path("s3://b/data.parquet") + .deleteFiles(Collections.singletonList(IcebergScanRange.DeleteFile.positionDelete("s3://b/delete.parquet", + TFileFormatType.FORMAT_PARQUET, 0L, 1L, 10L))) + .build().supportsBeSplitRefinement()); + Assertions.assertFalse(new IcebergScanRange.Builder().path("/dummyPath") + .serializedSplit("system-task").build().supportsBeSplitRefinement()); + Assertions.assertFalse(new IcebergScanRange.Builder().path("s3://b/delete.parquet") + .positionDeleteSysTableSplit(1, TFileFormatType.FORMAT_PARQUET, "s3://b/delete.parquet") + .build().supportsBeSplitRefinement()); + } + @Test public void metadataTasksUseTaskIdentityAndNativeRangesKeepFileIdentity() { IcebergScanRange first = new IcebergScanRange.Builder().path("/dummyPath") diff --git a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java index 4d43cc80770f6a..d194c547668f9c 100644 --- a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java @@ -555,7 +555,8 @@ public List planScan(ConnectorSession session, ConnectorScan return planScanInternal(session, request.getTableHandle(), request.getColumns(), request.getFilter(), request.getLimit(), request.isCountPushdown(), request.isAggregatePushdown(), - request.allBackendsSupport(PaimonRustReaderSelector.BACKEND_CAPABILITY)); + request.allBackendsSupport(PaimonRustReaderSelector.BACKEND_CAPABILITY), + request.getCoarseSplitSize()); } if (paimonHandle.isSystemTable()) { // System tables resolve their snapshot on the BE and carry deferred side effects @@ -563,7 +564,8 @@ public List planScan(ConnectorSession session, ConnectorScan return planScanInternal(session, request.getTableHandle(), request.getColumns(), request.getFilter(), request.getLimit(), request.isCountPushdown(), request.isAggregatePushdown(), - request.allBackendsSupport(PaimonRustReaderSelector.BACKEND_CAPABILITY)); + request.allBackendsSupport(PaimonRustReaderSelector.BACKEND_CAPABILITY), + request.getCoarseSplitSize()); } // Resolve the table ONCE at the statement scope so both the scan-planning path (here) and // the properties path (getScanNodeProperties) observe the SAME table generation. Without @@ -585,7 +587,8 @@ public List planScan(ConnectorSession session, ConnectorScan key -> Collections.unmodifiableList(planScanInternal(session, request.getTableHandle(), request.getColumns(), request.getFilter(), request.getLimit(), request.isCountPushdown(), request.isAggregatePushdown(), - request.allBackendsSupport(PaimonRustReaderSelector.BACKEND_CAPABILITY), table))); + request.allBackendsSupport(PaimonRustReaderSelector.BACKEND_CAPABILITY), + request.getCoarseSplitSize(), table))); } private long resolvePaimonGeneration(PaimonTableHandle handle, Table table) { @@ -768,9 +771,10 @@ private List planScanInternal( long limit, boolean countPushdown, boolean aggregatePushdown, - boolean backendsSupportRust) { + boolean backendsSupportRust, long coarseSplitSize) { return planScanInternal(session, handle, columns, filter, limit, countPushdown, - aggregatePushdown, backendsSupportRust, resolveScanTable((PaimonTableHandle) handle)); + aggregatePushdown, backendsSupportRust, coarseSplitSize, + resolveScanTable((PaimonTableHandle) handle)); } private List planScanInternal( @@ -782,7 +786,7 @@ private List planScanInternal( boolean countPushdown, boolean aggregatePushdown, boolean backendsSupportRust, - Table table) { + long coarseSplitSize, Table table) { PaimonTableHandle paimonHandle = (PaimonTableHandle) handle; boolean requiresMetadataColumns = requiresMetadataColumns(columns); @@ -1050,18 +1054,22 @@ && hasTrustworthyLimitAccounting(table) // DV with null cardinality) is kept WHOLE. We mirror that by passing target size 0, which // makes buildNativeRanges emit a single whole-file range; the target heuristic is then not // needed (and not computed) under count pushdown. - if (!countPushdown && targetSplitSize < 0) { - targetSplitSize = resolveTargetSplitSize(session, dataSplits); - } - long effectiveSplitSize = countPushdown ? 0L : targetSplitSize; List rawFiles = optRawFiles.get(); for (int i = 0; i < rawFiles.size(); i++) { RawFile file = rawFiles.get(i); + long nativeCoarseSize = coarseSplitSizeForRawFile(file.path(), defaultFileFormat, + countPushdown ? -1 : coarseSplitSize); + if (!countPushdown && nativeCoarseSize <= 0 && targetSplitSize < 0) { + targetSplitSize = resolveTargetSplitSize(session, dataSplits); + } + long effectiveSplitSize = countPushdown ? 0L + : nativeCoarseSize > 0 ? nativeCoarseSize : targetSplitSize; + long nativeWeightTarget = nativeCoarseSize > 0 ? nativeCoarseSize : weightDenominator; DeletionFile deletionFile = (optDeletionFiles.isPresent() && i < optDeletionFiles.get().size()) ? optDeletionFiles.get().get(i) : null; ranges.addAll(buildNativeRanges(file, deletionFile, defaultFileFormat, - partitionValues, vendedToken, effectiveSplitSize, weightDenominator, + partitionValues, vendedToken, effectiveSplitSize, nativeWeightTarget, dataSplit.bucket())); } } else { @@ -3240,6 +3248,14 @@ public int hashCode() { * routing even when the projected root-column names are identical. System tables are excluded * upstream, and session variables are statement-constant, so both stay out of the key. */ + static long coarseSplitSizeForRawFile(String path, String defaultFileFormat, long coarseSize) { + if (coarseSize <= 0) { + return -1; + } + String format = getFileFormatBySuffix(path).orElse(defaultFileFormat); + return "parquet".equals(format) || "orc".equals(format) ? coarseSize : -1; + } + private static final class PaimonScanReuseKey { private final String databaseName; private final String tableName; @@ -3253,6 +3269,7 @@ private static final class PaimonScanReuseKey { private final long limit; private final boolean countPushdown; private final boolean aggregatePushdown; + private final long coarseSplitSize; private PaimonScanReuseKey(PaimonTableHandle handle, ConnectorScanRequest request, long generation) { @@ -3278,6 +3295,7 @@ private PaimonScanReuseKey(PaimonTableHandle handle, ConnectorScanRequest reques this.limit = request.getLimit(); this.countPushdown = request.isCountPushdown(); this.aggregatePushdown = request.isAggregatePushdown(); + this.coarseSplitSize = request.getCoarseSplitSize(); } private static String toPaimonColumnName(ConnectorColumnHandle column) { @@ -3309,6 +3327,7 @@ public boolean equals(Object object) { && limit == that.limit && countPushdown == that.countPushdown && aggregatePushdown == that.aggregatePushdown + && coarseSplitSize == that.coarseSplitSize && rustBackendCapable == that.rustBackendCapable && Objects.equals(databaseName, that.databaseName) && Objects.equals(tableName, that.tableName) @@ -3323,7 +3342,7 @@ public boolean equals(Object object) { public int hashCode() { return Objects.hash(databaseName, tableName, branchName, scanOptions, generation, columnNames, nestedProjections, - rustBackendCapable, filter, limit, countPushdown, aggregatePushdown); + rustBackendCapable, filter, limit, countPushdown, aggregatePushdown, coarseSplitSize); } @Override diff --git a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanRange.java b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanRange.java index 949713fdeb6068..4ca08a718e5eeb 100644 --- a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanRange.java +++ b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanRange.java @@ -207,6 +207,11 @@ public boolean isNativeReadRange() { return !properties.containsKey("paimon.split") && path != null; } + @Override + public boolean supportsBeSplitRefinement() { + return isNativeReadRange(); + } + @Override public long getSelfSplitWeight() { return selfSplitWeight; diff --git a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonScanPlanProviderTest.java b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonScanPlanProviderTest.java index 4f56f1f87ca6b4..6fafc1103543aa 100644 --- a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonScanPlanProviderTest.java +++ b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonScanPlanProviderTest.java @@ -129,6 +129,18 @@ public class PaimonScanPlanProviderTest { private static final String PAIMON_FILE_PATH_COL = "__paimon_file_path"; private static final String PAIMON_ROW_POSITION_COL = "__paimon_row_index"; + @Test + public void coarseSizeIsLimitedToNativeColumnarFiles() { + Assertions.assertEquals(512 * MB, + PaimonScanPlanProvider.coarseSplitSizeForRawFile("a.parquet", "orc", 512 * MB)); + Assertions.assertEquals(512 * MB, + PaimonScanPlanProvider.coarseSplitSizeForRawFile("a.orc", "parquet", 512 * MB)); + Assertions.assertEquals(-1, + PaimonScanPlanProvider.coarseSplitSizeForRawFile("a.avro", "avro", 512 * MB)); + Assertions.assertEquals(-1, + PaimonScanPlanProvider.coarseSplitSizeForRawFile("a.parquet", "parquet", 0)); + } + @Test public void scanReuseNamespaceUsesConnectorType() { String prefix = new PaimonConnectorProvider().getType() + "."; @@ -4026,7 +4038,8 @@ public void nativeFileIsSubSplitWhenFileSplitSizeForcesIt(@TempDir Path warehous // WHY: this is the whole fix — one scanner per large file becomes N parallel sub-ranges. // MUTATION: neuter computeFileSplitOffsets to a single whole-file range -> nativeRanges==1 -> red. ConnectorSession splitting = sessionWithProps( - Collections.singletonMap("file_split_size", String.valueOf(splitSize))); + Map.of("file_split_size", String.valueOf(splitSize), "enable_external_scan_task_reuse", "true"), + new TestStatementScope()); List ranges = provider.planScan(splitting, ConnectorScanRequest.builder(handle, noColumns) .countPushdown(false).build()); @@ -4052,6 +4065,12 @@ public void nativeFileIsSubSplitWhenFileSplitSizeForcesIt(@TempDir Path warehous Assertions.assertEquals(fileLength, expectedStart, "native sub-ranges must cover exactly [0, fileLength)"); + List coarse = provider.planScan(splitting, + ConnectorScanRequest.builder(handle, noColumns).coarseSplitSize(fileLength * 2).build()); + Assertions.assertEquals(1, coarse.size()); + Assertions.assertEquals(fileLength * 2, coarse.get(0).getTargetSplitSize()); + Assertions.assertTrue(coarse.get(0).supportsBeSplitRefinement()); + // Contrast: with the default (large) split size the small file stays a SINGLE native range. List whole = provider.planScan(sessionWithProps(Collections.emptyMap()), ConnectorScanRequest.builder(handle, noColumns) diff --git a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/scan/ConnectorScanRange.java b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/scan/ConnectorScanRange.java index a4ac22e52c0d73..91434ff1db1e29 100644 --- a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/scan/ConnectorScanRange.java +++ b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/scan/ConnectorScanRange.java @@ -123,6 +123,15 @@ default long getTargetSplitSize() { return -1; } + /** + * Whether FE may use coarse byte ranges and supply a physical-child coalescing target for + * this range. Opt in only when native Parquet/ORC children preserve the range's table semantics. + * BE independently determines whether to refine; this declaration does not gate BE execution. + */ + default boolean supportsBeSplitRefinement() { + return false; + } + /** Returns preferred host locations for data locality. */ default List getHosts() { return Collections.emptyList(); diff --git a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/scan/ConnectorScanRequest.java b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/scan/ConnectorScanRequest.java index 39c2f580115b9d..01cd02a3b95e9b 100644 --- a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/scan/ConnectorScanRequest.java +++ b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/scan/ConnectorScanRequest.java @@ -48,6 +48,7 @@ public final class ConnectorScanRequest { private final List columns; private final Optional filter; private final long limit; + private final long coarseSplitSize; private final List requiredPartitions; private final boolean partitionsPrunedToEmpty; private final boolean countPushdown; @@ -57,13 +58,14 @@ public final class ConnectorScanRequest { private final ConnectorScanContext scanContext; private ConnectorScanRequest(ConnectorTableHandle tableHandle, List columns, - Optional filter, long limit, List requiredPartitions, + Optional filter, long limit, long coarseSplitSize, List requiredPartitions, boolean partitionsPrunedToEmpty, boolean countPushdown, boolean aggregatePushdown, boolean explainOnly, Set backendCapabilities, ConnectorScanContext scanContext) { this.tableHandle = tableHandle; this.columns = columns; this.filter = filter; this.limit = limit; + this.coarseSplitSize = coarseSplitSize; this.requiredPartitions = requiredPartitions; this.partitionsPrunedToEmpty = partitionsPrunedToEmpty; this.countPushdown = countPushdown; @@ -104,6 +106,15 @@ public long getLimit() { return limit; } + /** + * FE's coarse byte target for native ranges that support BE refinement, or {@code -1} to keep + * the connector's existing split policy. Connectors still decide eligibility per range, for + * example excluding Iceberg delete files. This is a planning hint, not a BE authorization gate. + */ + public long getCoarseSplitSize() { + return coarseSplitSize; + } + /** * The partitions the engine's pruning left, as metastore-rendered spec strings * (e.g. {@code "pt=1/region=cn"}); EMPTY means "not pruned — scan every partition". @@ -176,7 +187,7 @@ public ConnectorScanContext getScanContext() { /** This request with the partition set replaced — the batched scan's per-batch request. */ public ConnectorScanRequest withRequiredPartitions(List partitions) { - return new ConnectorScanRequest(tableHandle, columns, filter, limit, + return new ConnectorScanRequest(tableHandle, columns, filter, limit, coarseSplitSize, normalizePartitions(partitions), partitionsPrunedToEmpty, countPushdown, aggregatePushdown, explainOnly, backendCapabilities, scanContext); } @@ -192,6 +203,7 @@ public static final class Builder { private final List columns; private Optional filter = Optional.empty(); private long limit = -1; + private long coarseSplitSize = -1; private List requiredPartitions = Collections.emptyList(); private boolean partitionsPrunedToEmpty; private boolean countPushdown; @@ -215,6 +227,11 @@ public Builder limit(long limit) { return this; } + public Builder coarseSplitSize(long coarseSplitSize) { + this.coarseSplitSize = coarseSplitSize; + return this; + } + /** {@code null} is accepted and means the same as empty: not pruned, scan every partition. */ public Builder requiredPartitions(List requiredPartitions) { this.requiredPartitions = normalizePartitions(requiredPartitions); @@ -257,7 +274,7 @@ public Builder scanContext(ConnectorScanContext scanContext) { } public ConnectorScanRequest build() { - return new ConnectorScanRequest(tableHandle, columns, filter, limit, + return new ConnectorScanRequest(tableHandle, columns, filter, limit, coarseSplitSize, requiredPartitions, partitionsPrunedToEmpty, countPushdown, aggregatePushdown, explainOnly, backendCapabilities, scanContext); } diff --git a/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java b/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java index dbe7a76f33f7b5..29d96c98ed0b21 100644 --- a/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java +++ b/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java @@ -88,8 +88,8 @@ public void connectorApiMajorTracksTheRecordedSurfaceChange() throws IOException Assertions.assertNotNull(in, "missing connector plugin API version resource"); version.load(in); } - // Major 34 adds path partition keys after the upstream API 33 endpoint-composition changes. - Assertions.assertEquals("34.0", version.getProperty("api.version")); + // Major 35 adds coarse-split targets and BE refinement after API 34 path partition keys. + Assertions.assertEquals("35.0", version.getProperty("api.version")); } /** Root entry points plus provider/handle types returned to connector plugins. */ diff --git a/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/scan/ConnectorScanPlanProviderBatchScanTest.java b/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/scan/ConnectorScanPlanProviderBatchScanTest.java index f9f5ae31267b9f..282dc58fee7ab2 100644 --- a/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/scan/ConnectorScanPlanProviderBatchScanTest.java +++ b/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/scan/ConnectorScanPlanProviderBatchScanTest.java @@ -62,6 +62,14 @@ public List planScan(ConnectorSession session, ConnectorScan private static final ConnectorTableHandle HANDLE = new ConnectorTableHandle() { }; + @Test + public void refinementHintsDefaultToExistingPolicy() { + ConnectorScanRequest request = ConnectorScanRequest.builder(HANDLE, Collections.emptyList()).build(); + Assertions.assertEquals(-1L, request.getCoarseSplitSize()); + ConnectorScanRange range = Collections::emptyMap; + Assertions.assertFalse(range.supportsBeSplitRefinement()); + } + @Test public void partitionBatchSourceDefersScopedPlanningAndReleasesItsRanges() throws Exception { AtomicInteger calls = new AtomicInteger(); @@ -69,13 +77,14 @@ public void partitionBatchSourceDefersScopedPlanningAndReleasesItsRanges() throw ConnectorScanRequest request = ConnectorScanRequest.builder(HANDLE, Collections.emptyList()) .requiredPartitions(Arrays.asList("pt=1", "pt=2")) .backendCapabilities(Collections.singleton("capability")) - .aggregatePushdown(true).build(); + .aggregatePushdown(true).coarseSplitSize(512L << 20).build(); ConnectorScanPlanProvider provider = new ConnectorScanPlanProvider() { @Override public List planScan(ConnectorSession session, ConnectorScanRequest scoped) { calls.incrementAndGet(); Assertions.assertEquals(Collections.singletonList("pt=2"), scoped.getRequiredPartitions()); Assertions.assertTrue(scoped.isAggregatePushdown()); + Assertions.assertEquals(512L << 20, scoped.getCoarseSplitSize()); Assertions.assertEquals(request.getBackendCapabilities(), scoped.getBackendCapabilities()); return Collections.singletonList(range); } @@ -150,6 +159,7 @@ public void testPlanScanForPartitionBatchRescopesTheRequestToTheBatch() { ConnectorScanRequest request = ConnectorScanRequest.builder(HANDLE, columns) .filter(Optional.of(filter)) .limit(7L) + .coarseSplitSize(1234L) .partitionsPrunedToEmpty(true) .countPushdown(true) .aggregatePushdown(true) diff --git a/fe/fe-connector/fe-connector-spi/src/test/resources/connector-plugin-surface.txt b/fe/fe-connector/fe-connector-spi/src/test/resources/connector-plugin-surface.txt index e52e7eb25d45bf..6f6c64dbdbb428 100644 --- a/fe/fe-connector/fe-connector-spi/src/test/resources/connector-plugin-surface.txt +++ b/fe/fe-connector/fe-connector-spi/src/test/resources/connector-plugin-surface.txt @@ -256,9 +256,11 @@ org.apache.doris.connector.spi.scan.ConnectorScanRange#isNativeReadRange():boole org.apache.doris.connector.spi.scan.ConnectorScanRange#isPartitionBearing():boolean org.apache.doris.connector.spi.scan.ConnectorScanRange#isSingleUse():boolean org.apache.doris.connector.spi.scan.ConnectorScanRange#populateRangeParams(org.apache.doris.thrift.TTableFormatFileDesc,org.apache.doris.thrift.TFileRangeDesc):void +org.apache.doris.connector.spi.scan.ConnectorScanRange#supportsBeSplitRefinement():boolean org.apache.doris.connector.spi.scan.ConnectorScanRequest#allBackendsSupport(java.lang.String):boolean org.apache.doris.connector.spi.scan.ConnectorScanRequest#builder(org.apache.doris.connector.spi.handle.ConnectorTableHandle,java.util.List):org.apache.doris.connector.spi.scan.ConnectorScanRequest$Builder org.apache.doris.connector.spi.scan.ConnectorScanRequest#getBackendCapabilities():java.util.Set +org.apache.doris.connector.spi.scan.ConnectorScanRequest#getCoarseSplitSize():long org.apache.doris.connector.spi.scan.ConnectorScanRequest#getColumns():java.util.List org.apache.doris.connector.spi.scan.ConnectorScanRequest#getFilter():java.util.Optional org.apache.doris.connector.spi.scan.ConnectorScanRequest#getLimit():long @@ -273,6 +275,7 @@ org.apache.doris.connector.spi.scan.ConnectorScanRequest#withRequiredPartitions( org.apache.doris.connector.spi.scan.ConnectorScanRequest$Builder#aggregatePushdown(boolean):org.apache.doris.connector.spi.scan.ConnectorScanRequest$Builder org.apache.doris.connector.spi.scan.ConnectorScanRequest$Builder#backendCapabilities(java.util.Set):org.apache.doris.connector.spi.scan.ConnectorScanRequest$Builder org.apache.doris.connector.spi.scan.ConnectorScanRequest$Builder#build():org.apache.doris.connector.spi.scan.ConnectorScanRequest +org.apache.doris.connector.spi.scan.ConnectorScanRequest$Builder#coarseSplitSize(long):org.apache.doris.connector.spi.scan.ConnectorScanRequest$Builder org.apache.doris.connector.spi.scan.ConnectorScanRequest$Builder#countPushdown(boolean):org.apache.doris.connector.spi.scan.ConnectorScanRequest$Builder org.apache.doris.connector.spi.scan.ConnectorScanRequest$Builder#explainOnly(boolean):org.apache.doris.connector.spi.scan.ConnectorScanRequest$Builder org.apache.doris.connector.spi.scan.ConnectorScanRequest$Builder#filter(java.util.Optional):org.apache.doris.connector.spi.scan.ConnectorScanRequest$Builder diff --git a/fe/fe-connector/pom.xml b/fe/fe-connector/pom.xml index 861de4ab5772b2..bf038cbe57e31a 100644 --- a/fe/fe-connector/pom.xml +++ b/fe/fe-connector/pom.xml @@ -55,7 +55,7 @@ under the License. of the latter two means bumping this property as well (and fe-extension-spi means bumping all five families). --> - 34.0 + 35.0 diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FileQueryScanNode.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FileQueryScanNode.java index 164b791298a448..10bb8eeeccaed6 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FileQueryScanNode.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FileQueryScanNode.java @@ -532,6 +532,8 @@ private TScanRangeLocations splitToScanRange( // override it with the actual format carried by an individual split. rangeDesc.setFormatType(params.getFormatType()); setScanParams(rangeDesc, fileSplit); + setTargetSplitSize(rangeDesc, sessionVariable, + !wouldRunSerialOnBe() && supportsBeSplitRefinement(fileSplit)); rangeDesc.setFileCacheAdmission(admissionResult); curLocations.getScanRange().getExtScanRange().getFileScanRange().addToRanges(rangeDesc); @@ -543,6 +545,57 @@ private TScanRangeLocations splitToScanRange( return curLocations; } + protected boolean supportsBeSplitRefinement(FileSplit split) { + return true; + } + + protected long coarseSplitSizeForBeRefinement() { + return isCoarseSplitPlanningEnabled(sessionVariable) && !wouldRunSerialOnBe() + ? sessionVariable.getFileSplitSizeOnFe() : -1; + } + + protected long selectFeSplitSizeForBe(long fallbackSize, TFileFormatType format, boolean supportsBeSplit) { + long coarseSize = coarseSplitSizeForBeRefinement(); + return supportsBeSplit && isColumnarFormat(format) && coarseSize > 0 ? coarseSize : fallbackSize; + } + + static void setTargetSplitSize(TFileRangeDesc rangeDesc, SessionVariable sessionVariable) { + setTargetSplitSize(rangeDesc, sessionVariable, true); + } + + private static void setTargetSplitSize(TFileRangeDesc rangeDesc, SessionVariable sessionVariable, + boolean supportsBeSplit) { + if (supportsBeSplit && canSetBeSplitTarget(rangeDesc, sessionVariable)) { + rangeDesc.setTargetSplitSize(sessionVariable.getFileSplitSizeOnBe()); + } + } + + private boolean wouldRunSerialOnBe() { + // Match BE ScanLocalState's external small-LIMIT rule before enlarging FE ranges. + return sessionVariable.enableAdaptivePipelineTaskSerialReadOnLimit && conjuncts.isEmpty() + && getLimit() > 0 && getLimit() <= sessionVariable.adaptivePipelineTaskSerialReadOnLimit; + } + + static boolean canSetBeSplitTarget(TFileRangeDesc rangeDesc, SessionVariable sessionVariable) { + if (!isCoarseSplitPlanningEnabled(sessionVariable) || !rangeDesc.isSetFormatType() + || !isColumnarFormat(rangeDesc.getFormatType())) { + return false; + } + return !rangeDesc.isSetTableFormatParams() || !rangeDesc.getTableFormatParams().isSetTableLevelRowCount() + || rangeDesc.getTableFormatParams().getTableLevelRowCount() < 0; + } + + // These FE planning gates do not authorize BE refinement: an omitted target only disables coalescing. + private static boolean isCoarseSplitPlanningEnabled(SessionVariable sessionVariable) { + // enable_file_scanner_v2 is a deprecated compatibility variable; all file scans use V2. + return sessionVariable.maxFileScannersConcurrency != 1 && sessionVariable.getFileSplitSizeOnFe() > 0 + && sessionVariable.getFileSplitSizeOnBe() > 0; + } + + private static boolean isColumnarFormat(TFileFormatType format) { + return format == TFileFormatType.FORMAT_PARQUET || format == TFileFormatType.FORMAT_ORC; + } + private void setLocationPropertiesIfNecessary(Backend selectedBackend, TFileType locationType, Map locationProperties) throws UserException { if (locationType == TFileType.FILE_HDFS || locationType == TFileType.FILE_BROKER) { diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java index 8e0b978352c0e7..e6e6439af6323f 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java @@ -1948,6 +1948,7 @@ public List getSplits(int numBackends) throws UserException { .requiredPartitions(requiredPartitions) .partitionsPrunedToEmpty(partitionsPrunedToEmpty) .countPushdown(countPushdown) + .coarseSplitSize(coarseSplitSizeForBeRefinement()) .aggregatePushdown(getPushDownAggNoGroupingOp() != TPushAggOp.NONE) .backendCapabilities(commonBackendCapabilities(backendPolicy.getBackends())) // EXPLAIN plans the scan for real -- that is where its inputSplitNum comes from -- so a @@ -2359,6 +2360,7 @@ private ConnectorScanRequest prepareBatchScan(QueryFinishCleanup queryFinishClea pinRewriteFileScope(); queryFinishCleanup.bind(resolveScanProvider()); return ConnectorScanRequest.builder(currentHandle, columns).filter(remainingFilter) + .coarseSplitSize(isTableLevelCountStarPushdown() ? -1 : coarseSplitSizeForBeRefinement()) .scanContext(splitAssignment.getScanContext()).build(); } @@ -2601,6 +2603,11 @@ public void close() { }; } + @Override + protected boolean supportsBeSplitRefinement(FileSplit split) { + return ((PluginDrivenSplit) split).getConnectorScanRange().supportsBeSplitRefinement(); + } + @Override protected void setScanParams(TFileRangeDesc rangeDesc, Split split) { if (!(split instanceof PluginDrivenSplit)) { diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/tvf/source/TVFScanNode.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/tvf/source/TVFScanNode.java index 15c7fe83ea79af..0d14fabb85b40e 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/tvf/source/TVFScanNode.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/tvf/source/TVFScanNode.java @@ -179,9 +179,10 @@ protected TColumnCategory classifyColumn(SlotDescriptor slot, List parti return super.classifyColumn(slot, partitionKeys); } - private long determineTargetFileSplitSize(List fileStatuses) { + private long determineTargetFileSplitSize(List fileStatuses) throws UserException { if (sessionVariable.getFileSplitSize() > 0) { - return sessionVariable.getFileSplitSize(); + return selectFeSplitSizeForBe(sessionVariable.getFileSplitSize(), getFileFormatType(), + !isTableLevelCountStarPushdown()); } long result = sessionVariable.getMaxInitialSplitSize(); long totalFileSize = 0; @@ -195,7 +196,7 @@ private long determineTargetFileSplitSize(List fileStatuses) } result = exceedInitialThreshold ? sessionVariable.getMaxSplitSize() : result; result = applyMaxFileSplitNumLimit(result, totalFileSize); - return result; + return selectFeSplitSizeForBe(result, getFileFormatType(), !isTableLevelCountStarPushdown()); } @Override diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java b/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java index 9e07d31223c8a5..7e0ae70ff46421 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java +++ b/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java @@ -577,6 +577,10 @@ public String toString() { // Split size for ExternalFileScanNode. Default value 0 means use the block size of HDFS/S3. public static final String FILE_SPLIT_SIZE = "file_split_size"; + public static final String FILE_SPLIT_SIZE_ON_FE = "file_split_size_on_fe"; + + public static final String FILE_SPLIT_SIZE_ON_BE = "file_split_size_on_be"; + public static final String MAX_INITIAL_FILE_SPLIT_SIZE = "max_initial_file_split_size"; public static final String MAX_FILE_SPLIT_SIZE = "max_file_split_size"; @@ -2590,6 +2594,15 @@ public void checkParquetWriteDataPageSize(String value) throws Exception { @VarAttrDef.VarAttr(name = FILE_SPLIT_SIZE, needForward = true) public long fileSplitSize = 0; + @VarAttrDef.VarAttr(name = FILE_SPLIT_SIZE_ON_FE, needForward = true, + description = "Target bytes for FE coarse file splits when BE refinement is supported. Default: 512 MiB.") + public long fileSplitSizeOnFe = 512L * 1024L * 1024L; + + @VarAttrDef.VarAttr(name = FILE_SPLIT_SIZE_ON_BE, needForward = true, + description = "Target bytes for coalescing BE physical splits. Default: 64 MiB. " + + "A nonpositive value does not disable BE refinement.") + public long fileSplitSizeOnBe = 64L * 1024L * 1024L; + @VarAttrDef.VarAttr( name = MAX_INITIAL_FILE_SPLIT_SIZE, description = "For each table scan, The maximum initial file split size. " @@ -5078,6 +5091,22 @@ public long getFileSplitSize() { return fileSplitSize; } + public long getFileSplitSizeOnFe() { + return fileSplitSizeOnFe; + } + + public void setFileSplitSizeOnFe(long fileSplitSizeOnFe) { + this.fileSplitSizeOnFe = fileSplitSizeOnFe; + } + + public long getFileSplitSizeOnBe() { + return fileSplitSizeOnBe; + } + + public void setFileSplitSizeOnBe(long fileSplitSizeOnBe) { + this.fileSplitSizeOnBe = fileSplitSizeOnBe; + } + public void setFileSplitSize(long fileSplitSize) { this.fileSplitSize = fileSplitSize; } diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/scan/FileQueryScanNodeTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/scan/FileQueryScanNodeTest.java index 9fe589eecfd068..1336bdec930a29 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/scan/FileQueryScanNodeTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/scan/FileQueryScanNodeTest.java @@ -40,8 +40,10 @@ import org.apache.doris.thrift.TExpr; import org.apache.doris.thrift.TExprNodeType; import org.apache.doris.thrift.TFileFormatType; +import org.apache.doris.thrift.TFileRangeDesc; import org.apache.doris.thrift.TFileScanRangeParams; import org.apache.doris.thrift.TFileScanSlotInfo; +import org.apache.doris.thrift.TTableFormatFileDesc; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.BeforeEach; @@ -126,6 +128,106 @@ public void setUp() { Mockito.when(table.getName()).thenReturn("test_table"); } + @Test + public void testEligibleFileRangeCarriesBeSplitSize() { + SessionVariable sv = new SessionVariable(); + sv.setFileSplitSize(16 * MB); + sv.setFileSplitSizeOnBe(96 * MB); + + TFileRangeDesc range = new TFileRangeDesc(); + range.setFormatType(TFileFormatType.FORMAT_PARQUET); + FileQueryScanNode.setTargetSplitSize(range, sv); + Assertions.assertTrue(range.isSetTargetSplitSize()); + Assertions.assertEquals(96 * MB, range.getTargetSplitSize()); + } + + @Test + public void testIneligibleFileRangeOmitsBeSplitSize() { + SessionVariable sv = new SessionVariable(); + + TFileRangeDesc range = new TFileRangeDesc(); + range.setFormatType(TFileFormatType.FORMAT_CSV_PLAIN); + FileQueryScanNode.setTargetSplitSize(range, sv); + Assertions.assertFalse(range.isSetTargetSplitSize()); + } + + @Test + public void testMetadataCountRangesOmitBeSplitSize() { + SessionVariable sv = new SessionVariable(); + TFileRangeDesc countRange = new TFileRangeDesc(); + countRange.setFormatType(TFileFormatType.FORMAT_PARQUET); + TTableFormatFileDesc metadataCount = new TTableFormatFileDesc(); + metadataCount.setTableLevelRowCount(10); + countRange.setTableFormatParams(metadataCount); + FileQueryScanNode.setTargetSplitSize(countRange, sv); + Assertions.assertFalse(countRange.isSetTargetSplitSize()); + } + + @Test + public void testEligibleFileUsesDedicatedFeCoarseSplitSize() { + SessionVariable sv = new SessionVariable(); + sv.setFileSplitSize(16 * MB); + sv.setFileSplitSizeOnFe(512 * MB); + TestFileQueryScanNode node = new TestFileQueryScanNode(sv); + + Assertions.assertEquals(512 * MB, + node.selectFeSplitSizeForBe(16 * MB, TFileFormatType.FORMAT_ORC, true)); + } + + @Test + public void testIneligibleFileKeepsLegacyFeSplitSize() { + SessionVariable sv = new SessionVariable(); + sv.setFileSplitSizeOnFe(512 * MB); + TestFileQueryScanNode node = new TestFileQueryScanNode(sv); + + Assertions.assertEquals(16 * MB, + node.selectFeSplitSizeForBe(16 * MB, TFileFormatType.FORMAT_CSV_PLAIN, true)); + Assertions.assertEquals(16 * MB, + node.selectFeSplitSizeForBe(16 * MB, TFileFormatType.FORMAT_PARQUET, false)); + + sv.enableFileScannerV2 = false; + Assertions.assertEquals(512 * MB, + node.selectFeSplitSizeForBe(16 * MB, TFileFormatType.FORMAT_PARQUET, true)); + + sv.enableFileScannerV2 = true; + sv.maxFileScannersConcurrency = 1; + Assertions.assertEquals(16 * MB, + node.selectFeSplitSizeForBe(16 * MB, TFileFormatType.FORMAT_PARQUET, true)); + + TFileRangeDesc range = new TFileRangeDesc(); + range.setFormatType(TFileFormatType.FORMAT_PARQUET); + FileQueryScanNode.setTargetSplitSize(range, sv); + Assertions.assertFalse(range.isSetTargetSplitSize()); + + sv.maxFileScannersConcurrency = 16; + node.setLimit(100); + Assertions.assertEquals(16 * MB, + node.selectFeSplitSizeForBe(16 * MB, TFileFormatType.FORMAT_PARQUET, true)); + } + + @Test + public void testDedicatedSplitSizesHaveCoarseAndFineDefaults() { + SessionVariable sv = new SessionVariable(); + Assertions.assertEquals(512 * MB, sv.getFileSplitSizeOnFe()); + Assertions.assertEquals(64 * MB, sv.getFileSplitSizeOnBe()); + } + + @Test + public void nonpositiveTargetsKeepLegacyFePlanning() { + SessionVariable sv = new SessionVariable(); + TestFileQueryScanNode node = new TestFileQueryScanNode(sv); + for (long value : new long[] {0, -1}) { + sv.setFileSplitSizeOnFe(value); + Assertions.assertEquals(16 * MB, + node.selectFeSplitSizeForBe(16 * MB, TFileFormatType.FORMAT_PARQUET, true)); + sv.setFileSplitSizeOnFe(512 * MB); + sv.setFileSplitSizeOnBe(value); + Assertions.assertEquals(16 * MB, + node.selectFeSplitSizeForBe(16 * MB, TFileFormatType.FORMAT_ORC, true)); + sv.setFileSplitSizeOnBe(64 * MB); + } + } + @Test public void testApplyMaxFileSplitNumLimitRaisesTargetSize() { SessionVariable sv = new SessionVariable(); diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/scan/PluginDrivenScanNodeBeSplitTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/scan/PluginDrivenScanNodeBeSplitTest.java new file mode 100644 index 00000000000000..0ff04b4e8035c4 --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/scan/PluginDrivenScanNodeBeSplitTest.java @@ -0,0 +1,176 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.datasource.scan; + +import org.apache.doris.analysis.TupleDescriptor; +import org.apache.doris.analysis.TupleId; +import org.apache.doris.catalog.TableIf; +import org.apache.doris.common.jmockit.Deencapsulation; +import org.apache.doris.common.util.DebugUtil; +import org.apache.doris.connector.spi.Connector; +import org.apache.doris.connector.spi.ConnectorMetadata; +import org.apache.doris.connector.spi.ConnectorSession; +import org.apache.doris.connector.spi.ConnectorStatementScope; +import org.apache.doris.connector.spi.handle.ConnectorTableHandle; +import org.apache.doris.connector.spi.scan.ConnectorScanContext; +import org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider; +import org.apache.doris.connector.spi.scan.ConnectorScanRange; +import org.apache.doris.connector.spi.scan.ConnectorScanRequest; +import org.apache.doris.datasource.split.PluginDrivenSplit; +import org.apache.doris.datasource.split.SplitAssignment; +import org.apache.doris.planner.PlanNodeId; +import org.apache.doris.planner.ScanContext; +import org.apache.doris.qe.QeProcessorImpl; +import org.apache.doris.qe.SessionVariable; +import org.apache.doris.system.Backend; +import org.apache.doris.thrift.TFileFormatType; +import org.apache.doris.thrift.TFileRangeDesc; +import org.apache.doris.thrift.TFileScanRangeParams; +import org.apache.doris.thrift.TPushAggOp; +import org.apache.doris.thrift.TScanRangeLocations; +import org.apache.doris.thrift.TTableFormatFileDesc; +import org.apache.doris.thrift.TUniqueId; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; +import org.mockito.Mockito; + +import java.util.Collections; +import java.util.Map; +import java.util.Optional; +import java.util.concurrent.atomic.AtomicLong; +import java.util.concurrent.atomic.AtomicReference; + +public class PluginDrivenScanNodeBeSplitTest { + private static final AtomicLong NEXT_QUERY = new AtomicLong(); + + private static final class Fixture implements AutoCloseable { + private final TUniqueId queryId = new TUniqueId(66802, NEXT_QUERY.incrementAndGet()); + private final ConnectorSession session = Mockito.mock(ConnectorSession.class); + private final AtomicReference planned = new AtomicReference<>(); + private final PluginDrivenScanNode node; + + private Fixture(SessionVariable variables) { + ConnectorTableHandle handle = Mockito.mock(ConnectorTableHandle.class); + ConnectorScanPlanProvider provider = Mockito.mock(ConnectorScanPlanProvider.class, + Mockito.CALLS_REAL_METHODS); + Mockito.when(provider.planScan(Mockito.any(), Mockito.any())).thenAnswer(invocation -> { + planned.set(invocation.getArgument(1)); + return Collections.emptyList(); + }); + Connector connector = Mockito.mock(Connector.class); + Mockito.when(connector.getScanPlanProvider(handle)).thenReturn(provider); + Mockito.when(session.getQueryId()).thenReturn(DebugUtil.printId(queryId)); + Mockito.when(session.getStatementScope()).thenReturn(ConnectorStatementScope.NONE); + TupleDescriptor descriptor = new TupleDescriptor(new TupleId(0)); + descriptor.setTable(Mockito.mock(TableIf.class)); + node = new PluginDrivenScanNode(new PlanNodeId(0), descriptor, false, variables, + ScanContext.EMPTY, connector, session, handle); + ConnectorMetadata metadata = Mockito.mock(ConnectorMetadata.class, Mockito.CALLS_REAL_METHODS); + Mockito.doReturn(Collections.emptyMap()).when(metadata).getColumnHandles(session, handle); + Deencapsulation.setField(node, "cachedMetadata", metadata); + SplitAssignment assignment = Mockito.mock(SplitAssignment.class); + Mockito.when(assignment.getScanContext()).thenReturn(new ConnectorScanContext()); + Deencapsulation.setField(node, "splitAssignment", assignment); + } + + private ConnectorScanRequest batchRequest() { + return Deencapsulation.invoke(node, "prepareBatchScan", + PluginDrivenScanNode.registerQueryFinishCleanup(session)); + } + + @Override + public void close() { + QeProcessorImpl.INSTANCE.unregisterQuery(queryId); + } + } + + @Test + public void synchronousAndStreamingRequestsUseActualNodeSerialLimit() throws Exception { + SessionVariable variables = new SessionVariable(); + variables.enableAdaptivePipelineTaskSerialReadOnLimit = true; + try (Fixture fixture = new Fixture(variables)) { + fixture.node.getSplits(1); + Assertions.assertEquals(512L << 20, fixture.planned.get().getCoarseSplitSize()); + Assertions.assertEquals(512L << 20, fixture.batchRequest().getCoarseSplitSize()); + fixture.node.setLimit(1); + fixture.node.getSplits(1); + Assertions.assertEquals(-1, fixture.planned.get().getCoarseSplitSize()); + Assertions.assertEquals(-1, fixture.batchRequest().getCoarseSplitSize()); + fixture.node.setLimit(-1); + variables.maxFileScannersConcurrency = 1; + fixture.node.getSplits(1); + Assertions.assertEquals(-1, fixture.planned.get().getCoarseSplitSize()); + Assertions.assertEquals(-1, fixture.batchRequest().getCoarseSplitSize()); + } + } + + @Test + public void metadataCountBatchKeepsLegacyPlanningHint() throws Exception { + try (Fixture fixture = new Fixture(new SessionVariable())) { + fixture.node.setPushDownAggNoGrouping(TPushAggOp.COUNT); + fixture.node.setPushDownCountSlotIds(Collections.emptyList()); + Assertions.assertEquals(-1, fixture.batchRequest().getCoarseSplitSize()); + } + } + + @Test + public void finalConnectorFormatAndCapabilityControlTheCoalescingTarget() throws Exception { + try (Fixture fixture = new Fixture(new SessionVariable())) { + TFileScanRangeParams params = new TFileScanRangeParams(); + params.setFormatType(TFileFormatType.FORMAT_JNI); + Deencapsulation.setField(fixture.node, "params", params); + for (boolean supported : new boolean[] {false, true}) { + for (TFileFormatType format : new TFileFormatType[] { + TFileFormatType.FORMAT_PARQUET, TFileFormatType.FORMAT_ORC, TFileFormatType.FORMAT_JNI}) { + ConnectorScanRange range = new ConnectorScanRange() { + @Override + public Map getProperties() { + return Collections.emptyMap(); + } + + @Override + public Optional getPath() { + return Optional.of("file:///tmp/two-level.parquet"); + } + + @Override + public boolean supportsBeSplitRefinement() { + return supported; + } + + @Override + public void populateRangeParams(TTableFormatFileDesc table, TFileRangeDesc file) { + file.setFormatType(format); + } + }; + TScanRangeLocations locations = Deencapsulation.invoke(fixture.node, "splitToScanRange", + new Backend(1, "127.0.0.1", 9050), Collections.emptyMap(), new PluginDrivenSplit(range), + Collections.emptyList(), true); + TFileRangeDesc file = locations.getScanRange().getExtScanRange() + .getFileScanRange().getRanges().get(0); + Assertions.assertEquals(supported && format != TFileFormatType.FORMAT_JNI, + file.isSetTargetSplitSize()); + if (file.isSetTargetSplitSize()) { + Assertions.assertEquals(64L << 20, file.getTargetSplitSize()); + } + } + } + } + } +} diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/tvf/source/TVFScanNodeTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/tvf/source/TVFScanNodeTest.java index 83ab587f0266fc..1dc4042020b2c1 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/tvf/source/TVFScanNodeTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/tvf/source/TVFScanNodeTest.java @@ -43,6 +43,25 @@ public class TVFScanNodeTest { private static final long MB = 1024L * 1024L; + @Test + public void testDetermineTargetFileSplitSizeUsesCoarseSizeForParquet() throws Exception { + SessionVariable sv = new SessionVariable(); + sv.setFileSplitSize(16 * MB); + TupleDescriptor desc = new TupleDescriptor(new TupleId(0)); + FunctionGenTable table = Mockito.mock(FunctionGenTable.class); + ExternalFileTableValuedFunction tvf = Mockito.mock(ExternalFileTableValuedFunction.class); + Mockito.when(table.getTvf()).thenReturn(tvf); + Mockito.when(tvf.getTFileFormatType()).thenReturn(org.apache.doris.thrift.TFileFormatType.FORMAT_PARQUET); + desc.setTable(table); + TVFScanNode node = new TVFScanNode(new PlanNodeId(0), desc, false, sv, ScanContext.EMPTY); + Method method = TVFScanNode.class.getDeclaredMethod("determineTargetFileSplitSize", List.class); + method.setAccessible(true); + List statuses = List.of(splittableFile("s3://bucket/large.parquet", 1024 * MB)); + Assertions.assertEquals(512 * MB, (long) method.invoke(node, statuses)); + sv.maxFileScannersConcurrency = 1; + Assertions.assertEquals(16 * MB, (long) method.invoke(node, statuses)); + } + @Test public void testCountColumnKeepsNormalFileSplitting() throws Exception { SessionVariable sv = new SessionVariable(); diff --git a/gensrc/thrift/PlanNodes.thrift b/gensrc/thrift/PlanNodes.thrift index 2c3df7796bbf41..2f8a93a1bec5e7 100644 --- a/gensrc/thrift/PlanNodes.thrift +++ b/gensrc/thrift/PlanNodes.thrift @@ -979,6 +979,8 @@ struct TFileRangeDesc { // whether the value of columns_from_path is null 15: optional list columns_from_path_is_null; 16: optional bool file_cache_admission; + // Target size for coalescing physical children; absent/nonpositive does not disable refinement. + 17: optional i64 target_split_size; } struct TSplitSource { diff --git a/regression-test/suites/external_table_p0/hive/test_parquet_join_runtime_filter.groovy b/regression-test/suites/external_table_p0/hive/test_parquet_join_runtime_filter.groovy index a9a1ca1e175721..bd66beaf4b38ba 100644 --- a/regression-test/suites/external_table_p0/hive/test_parquet_join_runtime_filter.groovy +++ b/regression-test/suites/external_table_p0/hive/test_parquet_join_runtime_filter.groovy @@ -50,6 +50,7 @@ suite("test_parquet_join_runtime_filter", "p0,external") { sql "set enable_profile=true;" sql " set parallel_pipeline_task_num = 1;" sql " set file_split_size = 100000;" + sql "set file_split_size_on_fe = 0;" String enabled = context.config.otherConfigs.get("enableHiveTest") if (!"true".equalsIgnoreCase(enabled)) { diff --git a/regression-test/suites/external_table_p0/hive/test_two_level_file_splits.groovy b/regression-test/suites/external_table_p0/hive/test_two_level_file_splits.groovy new file mode 100644 index 00000000000000..20d7779a32c11a --- /dev/null +++ b/regression-test/suites/external_table_p0/hive/test_two_level_file_splits.groovy @@ -0,0 +1,63 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +suite("test_two_level_file_splits", "p0,external") { + if (!"true".equalsIgnoreCase(context.config.otherConfigs.get("enableHiveTest"))) { + return + } + String host = context.config.otherConfigs.get("externalEnvIp") + String hmsPort = context.config.otherConfigs.get("hive2HmsPort") + sql "drop catalog if exists test_two_level_file_splits" + sql """create catalog test_two_level_file_splits properties ( + "type"="hms", "hadoop.username"="hadoop", + "hive.metastore.uris"="thrift://${host}:${hmsPort}" + )""" + sql "switch test_two_level_file_splits" + sql "use `default`" + sql "set enable_sql_cache=false" + sql "set enable_query_cache=false" + sql "set file_split_size=1048576" + sql "set file_split_size_on_fe=0" + sql "set file_split_size_on_be=0" + sql "set max_file_scanners_concurrency=1" + sql "set enable_condition_cache=false" + def queries = [ + "select k, count(*) from fact_big where k is not null group by k order by k", + "select count(*) from fact_big", + "select a.k, count(*) from fact_big a join dim_small b on a.k=b.k where b.c1=5 group by a.k order by a.k", + "select col1, col2, col3 from orc_tiny_stripes where col1%100=0 order by col1,col2,col3", + "select count(*) from orc_tiny_stripes" + ] + // The single-scanner run cannot publish physical children. Compare the same data and SQL + // against parallel refinement, including absent/nonpositive and positive coalescing targets. + def reference = queries.collect { query -> sql(query) } + sql "set max_file_scanners_concurrency=16" + sql "set enable_condition_cache=true" + for (int batchThreshold : [0, Integer.MAX_VALUE]) { + sql "set num_files_in_batch_mode=${batchThreshold}" + for (List sizes : [[0, 0], [536870912, -1], [536870912, 1048576], [536870912, 67108864]]) { + sql "set file_split_size_on_fe=${sizes[0]}" + sql "set file_split_size_on_be=${sizes[1]}" + // The second pass exercises the condition-cache lifecycle after sibling completion. + for (int pass = 0; pass < 2; pass++) { + queries.eachWithIndex { query, index -> + assertEquals(reference[index], sql(query)) + } + } + } + } +} diff --git a/regression-test/suites/external_table_p0/iceberg/test_iceberg_missing_data_file.groovy b/regression-test/suites/external_table_p0/iceberg/test_iceberg_missing_data_file.groovy index f71e1449cfb865..0119f1e82fab1e 100644 --- a/regression-test/suites/external_table_p0/iceberg/test_iceberg_missing_data_file.groovy +++ b/regression-test/suites/external_table_p0/iceberg/test_iceberg_missing_data_file.groovy @@ -72,6 +72,8 @@ suite("test_iceberg_missing_data_file", client.shutdown() } + // The refinement planner must honor the same Iceberg missing-file policy as ordinary reads. + sql "set max_file_scanners_concurrency = 16" sql "set enable_file_cache = false" sql "set enable_sql_cache = false" sql "set enable_query_cache = false" diff --git a/regression-test/suites/external_table_p0/paimon/test_paimon_deletion_vector.groovy b/regression-test/suites/external_table_p0/paimon/test_paimon_deletion_vector.groovy index 33bd56210a96b2..5fe2d76fab9bf8 100644 --- a/regression-test/suites/external_table_p0/paimon/test_paimon_deletion_vector.groovy +++ b/regression-test/suites/external_table_p0/paimon/test_paimon_deletion_vector.groovy @@ -41,6 +41,7 @@ suite("test_paimon_deletion_vector", "p0,external") { // Force tiny splits so both native and JNI paths read Paimon DV tables through multiple // scanner ranges while still returning the same filtered rows. sql """ set file_split_size=1 """ + sql """ set file_split_size_on_fe=0 """ qt_1 """select count(*) from deletion_vector_orc;""" qt_2 """select count(*) from deletion_vector_parquet;""" qt_3 """select count(*) from deletion_vector_orc where id > 2;""" @@ -102,6 +103,7 @@ suite("test_paimon_deletion_vector", "p0,external") { } finally { sql """ set enable_count_push_down_for_external_table=true; """ sql """unset variable file_split_size""" + sql """unset variable file_split_size_on_fe""" sql """set force_jni_scanner=false""" } diff --git a/regression-test/suites/external_table_p0/paimon/test_paimon_file_metadata_columns.groovy b/regression-test/suites/external_table_p0/paimon/test_paimon_file_metadata_columns.groovy index 7091a5fa0d2acb..4cdd70f2f5ee81 100644 --- a/regression-test/suites/external_table_p0/paimon/test_paimon_file_metadata_columns.groovy +++ b/regression-test/suites/external_table_p0/paimon/test_paimon_file_metadata_columns.groovy @@ -117,6 +117,7 @@ suite("test_paimon_file_metadata_columns", "p0,external") { sql """set force_jni_scanner=false""" // A one-byte split forces one native source file through multiple scanner ranges. sql """set file_split_size=1""" + sql """set file_split_size_on_fe=0""" verifyMetadata("deletion_vector_orc") verifyMetadata("deletion_vector_parquet") @@ -161,6 +162,7 @@ suite("test_paimon_file_metadata_columns", "p0,external") { assertEquals(metadataRows, sql(metadataQuery)) } finally { sql """unset variable file_split_size""" + sql """unset variable file_split_size_on_fe""" sql """set force_jni_scanner=false""" sql """set enable_file_scanner_v2=true""" sql """drop catalog if exists ${crossFileCatalogName}""" From f6eb0621044ef51cef3b238693455d0b7ef93148 Mon Sep 17 00:00:00 2001 From: morningman Date: Tue, 17 Nov 2026 19:35:41 +0800 Subject: [PATCH 02/10] [fix](test) Fix binary column iterator compilation on macOS ### Background The binary column iterator test helper limits each batch to the remaining source rows. std::min requires both arguments to have the same type. ### What problem does this PR solve? Issue Number: None Related PR: None PR Type: Bugfix Problem Summary: The helper does not compile on macOS because size_t is unsigned long while the ordinal_t subtraction yields unsigned long long. Specify the size_t template argument so both supported 64-bit platforms use the existing size_t result type. ### Details - **Frequency:** Compiling the BE unit-test executable on macOS. - **Root cause:** Template deduction sees different integer types despite their equal width on the supported platform. - **Why existing tests missed it:** Matching Linux typedefs do not exercise the Darwin compilation mismatch. - **Fix:** Use std::min and explain the platform type difference beside the expression. - **Other paths and branches:** Test helper only; no production behavior or wire changes. ### Testing - The original expression fails to compile on Darwin arm64 with LLVM 22. - The standard ASAN BE run passes 257 tests, including all 13 BinaryColumnExtractIteratorV2Test cases and the related VariantAssemblerTest and SparseColumnMergeIteratorTest cases. - The edited lines pass clang-tidy and formatting checks. ### Release note None ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Ready ### AI Disclosure - Tool: Codex (GPT-6) - Generated: Test compilation fix and this description. - Human review: Not stated. --- .../storage/segment/binary_column_extract_iterator_test.cpp | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/be/test/storage/segment/binary_column_extract_iterator_test.cpp b/be/test/storage/segment/binary_column_extract_iterator_test.cpp index bf2c53335089e2..c1419198536f5d 100644 --- a/be/test/storage/segment/binary_column_extract_iterator_test.cpp +++ b/be/test/storage/segment/binary_column_extract_iterator_test.cpp @@ -74,7 +74,8 @@ class FixedSparseIterator final : public ColumnIterator { Status next_batch(size_t* n, MutableColumnPtr& dst, bool* has_null) override { ++_counters->next_batch; - const size_t rows = std::min(*n, _source->size() - _ordinal); + // size_t and ordinal_t are distinct 64-bit integer types on macOS. + const size_t rows = std::min(*n, _source->size() - _ordinal); if (_ordinal == 0 && rows == _source->size()) { // Preserve deliberately invalid offsets in corruption tests. dst = std::move(_source); From f8950813af1c2a114b374c3cca12db8fe6c6fb4a Mon Sep 17 00:00:00 2001 From: morningman Date: Tue, 17 Nov 2026 19:36:37 +0800 Subject: [PATCH 03/10] [fix](external) Preserve efficient two-level external file splitting ### Background FE plans external-file ranges and BE refines eligible native ranges into physical row-group or stripe tasks. Predicate snapshots, metadata aggregates and condition cache state must remain reusable across these tasks without losing the existing FE planning safeguards. ### What problem does this PR solve? Issue Number: None Related PR: #70664 (review follow-up) PR Type: Bugfix Problem Summary: Address the five performance regressions identified in the review of two-level external-file splitting. Reuse lazy predicate keys, retain metadata-only ORC readers, assemble child cache bitmaps once, preserve FE split-count heuristics, and retain legacy HTTP TVF range parallelism. ### Details - **Change map:** 416 changed production lines and 480 changed test lines across 35 files; no generated files, dependencies or wire changes. - **Frequency:** Scans with repeated runtime-filter payloads, metadata-only ORC aggregates, many cache-enabled physical children, a binding max_file_split_num, or HTTP TVF files eligible for legacy byte splitting. - **Root cause:** Per-split digest construction reaches nonconsuming readers; ORC omits the metadata aggregate hook; child bitmap merging repeatedly copies its prefix; coarse targets replace count-derived bounds; HTTP uses coarse sizing despite an unstable file identity. - **Why existing tests missed it:** Existing tests cover results and isolated sizing helpers, but omit digest call counts, real ORC planning-reader retention, multi-child bitmap ownership, and integrated coarse-target range counts. - **Fix:** Share an immutable scanner predicate snapshot with a lazy key requested through a reader capability. Delegate ORC metadata aggregation to its existing implementation. Transfer completed bitmap owners under the coordinator mutex and assemble their union outside it after all matching EOFs. Apply count-derived bounds after coarse sizing and opt HTTP TVFs out of coarse planning. - **Other paths and branches:** Forward snapshots through Hudi, Paimon, Fluss and JNI wrappers. Cover synchronous SDK and manifest-cache Iceberg planning; keep streaming lazy. Keep native Paimon COUNT and existing unsupported aggregate fallbacks. Adapt the encryption fixture to the shared-metadata ownership change and initialize the scanner refresh-error fixture with a valid local state. ### Testing - Standard FE script: 400 tests passed across TVFScanNodeTest, PaimonScanPlanProviderTest and IcebergScanPlanProviderTest. The native Paimon and synchronous Iceberg range-count regressions fail on the original production code and pass after the fixes. - Standard FE build including Paimon and Iceberg completed with Checkstyle and plugin packaging. - Standard ASAN BE script: 257 tests in 10 suites passed, including all TableReaderTest and FileScannerV2Test cases, real ORC metadata aggregation, hybrid forwarding, the encryption fixture and binary column iterator coverage. - Changed C++ lines passed clang-tidy; clang-format v16 and build hygiene passed. - No live external-cluster regression or latency benchmark was run. ### Release note Preserve split-count safeguards and HTTP TVF parallelism during external scans, and avoid repeated runtime-filter hashing, ORC metadata reader creation and child condition-cache bitmap copying. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: No; restores the intended planning and cache behavior of the pending feature without adding settings. - Pre-submission status: Ready ### AI Disclosure - Tool: Codex (GPT-6) - Generated: Code, tests and this commit description. - Human review: Not stated. --- be/src/exec/scan/file_scanner_v2.cpp | 20 +- be/src/exec/scan/file_scanner_v2.h | 4 +- be/src/format_v2/file_reader.cpp | 18 ++ be/src/format_v2/file_reader.h | 4 + be/src/format_v2/jni/jni_table_reader.cpp | 10 +- be/src/format_v2/jni/jni_table_reader.h | 8 +- be/src/format_v2/orc/orc_reader.h | 4 + be/src/format_v2/parquet/parquet_reader.h | 1 + be/src/format_v2/predicate_snapshot.h | 43 +++ .../format_v2/table/fluss_hybrid_reader.cpp | 14 +- be/src/format_v2/table/fluss_hybrid_reader.h | 8 +- .../table/fluss_union_lake_reader.cpp | 14 +- .../format_v2/table/fluss_union_lake_reader.h | 8 +- be/src/format_v2/table/hudi_reader.cpp | 14 +- be/src/format_v2/table/hudi_reader.h | 8 +- be/src/format_v2/table/load_reader.cpp | 3 +- be/src/format_v2/table/load_reader.h | 8 +- be/src/format_v2/table/paimon_reader.cpp | 14 +- be/src/format_v2/table/paimon_reader.h | 8 +- be/src/format_v2/table_reader.cpp | 115 ++++---- be/src/format_v2/table_reader.h | 19 +- be/src/storage/segment/condition_cache.h | 5 +- be/test/exec/scan/file_scanner_v2_test.cpp | 49 +++- be/test/format_v2/orc/orc_reader_test.cpp | 58 ++++ .../parquet/parquet_encryption_test.cpp | 4 +- .../table/fluss_union_lake_reader_test.cpp | 12 +- be/test/format_v2/table/hudi_reader_test.cpp | 13 +- .../format_v2/table/paimon_reader_test.cpp | 13 +- be/test/format_v2/table_reader_test.cpp | 254 ++++++++++++++++++ .../iceberg/IcebergScanPlanProvider.java | 31 ++- .../iceberg/IcebergScanPlanProviderTest.java | 20 ++ .../paimon/PaimonScanPlanProvider.java | 21 +- .../paimon/PaimonScanPlanProviderTest.java | 7 + .../datasource/tvf/source/TVFScanNode.java | 14 +- .../tvf/source/TVFScanNodeTest.java | 50 ++++ 35 files changed, 751 insertions(+), 145 deletions(-) create mode 100644 be/src/format_v2/predicate_snapshot.h diff --git a/be/src/exec/scan/file_scanner_v2.cpp b/be/src/exec/scan/file_scanner_v2.cpp index cc11b281ba263e..1a07a5e70bc587 100644 --- a/be/src/exec/scan/file_scanner_v2.cpp +++ b/be/src/exec/scan/file_scanner_v2.cpp @@ -59,6 +59,7 @@ #include "format_v2/jni/jdbc_reader.h" #include "format_v2/jni/max_compute_jni_reader.h" #include "format_v2/jni/trino_connector_jni_reader.h" +#include "format_v2/predicate_snapshot.h" #include "format_v2/table/adbc_reader.h" #include "format_v2/table/es_reader.h" #include "format_v2/table/fluss_hybrid_reader.h" @@ -91,9 +92,6 @@ const std::string FileScannerV2::FileReadTimeProfile = "FileReadTime"; namespace { -constexpr int kIcebergPositionDeleteContent = 1; -constexpr int kIcebergDeletionVectorContent = 3; - std::string table_format_name(const TFileRangeDesc& range); std::string table_format_name(const TFileRangeDesc& range) { @@ -444,9 +442,9 @@ Status FileScannerV2::_get_block_wrapped(RuntimeState* state, Block* block, bool if (_table_reader_rf_num != _applied_rf_num) { VExprContextSPtrs refreshed_conjuncts; RETURN_IF_ERROR(_build_table_conjuncts(&refreshed_conjuncts)); - RETURN_IF_ERROR(_table_reader->refresh_conjuncts(std::move(refreshed_conjuncts), - _current_condition_cache_digest(), - _applied_rf_num == _total_rf_num)); + RETURN_IF_ERROR(_table_reader->refresh_conjuncts( + std::move(refreshed_conjuncts), _current_condition_cache_digest(), + _applied_rf_num == _total_rf_num, _predicate_snapshot)); _table_reader_rf_num = _applied_rf_num; } if (_should_run_adaptive_batch_size()) { @@ -659,6 +657,7 @@ Status FileScannerV2::_init_table_reader(const TFileRangeDesc& range) { .push_down_count_columns = std::move(push_down_count_columns), .condition_cache_digest = _local_state != nullptr ? _local_state->get_condition_cache_digest() : 0, + .predicate_snapshot = _predicate_snapshot, })); return Status::OK(); } @@ -841,6 +840,7 @@ Status FileScannerV2::_prepare_table_reader_split(const TFileRangeDesc& range, .global_rowid_context = _create_global_rowid_context(_current_split.source_identity_range()), .row_ids = std::nullopt, + .predicate_snapshot = _predicate_snapshot, })); return Status::OK(); } @@ -1075,7 +1075,7 @@ format::ColumnDefinition FileScannerV2::_build_table_column(const SlotDescriptor return column; } -Status FileScannerV2::_build_table_conjuncts(VExprContextSPtrs* conjuncts) const { +Status FileScannerV2::_build_table_conjuncts(VExprContextSPtrs* conjuncts) { DORIS_CHECK(conjuncts != nullptr); conjuncts->clear(); if (_is_load) { @@ -1088,6 +1088,12 @@ Status FileScannerV2::_build_table_conjuncts(VExprContextSPtrs* conjuncts) const RETURN_IF_ERROR(rewrite_slot_refs_to_global_index(&root, _slot_id_to_global_index)); conjuncts->push_back(VExprContext::create_shared(std::move(root))); } + // Ready RF payloads are immutable; only a new arrival replaces Scanner's conjunct set. + // Keep one lazy key across source/physical splits, including transitions between formats. + if (_predicate_snapshot_rf_num != _applied_rf_num) { + _predicate_snapshot = std::make_shared(*conjuncts); + _predicate_snapshot_rf_num = _applied_rf_num; + } return Status::OK(); } diff --git a/be/src/exec/scan/file_scanner_v2.h b/be/src/exec/scan/file_scanner_v2.h index 83dd924ff3f03d..ffb466d645d2a8 100644 --- a/be/src/exec/scan/file_scanner_v2.h +++ b/be/src/exec/scan/file_scanner_v2.h @@ -208,7 +208,7 @@ class FileScannerV2 final : public Scanner { Status _build_projected_columns(const format::TableReader& table_reader); Status _build_default_expr(const TFileScanSlotInfo& slot_info, VExprContextSPtr* ctx) const; static format::ColumnDefinition _build_table_column(const SlotDescriptor* slot_desc); - Status _build_table_conjuncts(VExprContextSPtrs* conjuncts) const; + Status _build_table_conjuncts(VExprContextSPtrs* conjuncts); static Status _to_file_format(TFileFormatType::type format_type, format::FileFormat* file_format); void _reset_adaptive_batch_size_state(); @@ -241,6 +241,8 @@ class FileScannerV2 final : public Scanner { bool _first_scan_range = false; bool _has_prepared_split = false; int _table_reader_rf_num = 0; + int _predicate_snapshot_rf_num = -1; + std::shared_ptr _predicate_snapshot; TFileRangeDesc _current_range; FileScanSplit _current_split; diff --git a/be/src/format_v2/file_reader.cpp b/be/src/format_v2/file_reader.cpp index 7ce26e61308c3f..df406a5fd49b54 100644 --- a/be/src/format_v2/file_reader.cpp +++ b/be/src/format_v2/file_reader.cpp @@ -19,7 +19,9 @@ #include +#include "exprs/vexpr_context.h" #include "format_v2/column_mapper.h" +#include "format_v2/predicate_snapshot.h" #include "io/fs/buffered_reader.h" #include "io/fs/tracing_file_reader.h" #include "runtime/runtime_state.h" @@ -43,6 +45,22 @@ std::string join_debug_strings(const std::vector& values, Formatter formatter } // namespace +std::optional PredicateSnapshot::digest() const { + std::call_once(_digest_once, [this] { + // Independent of the optional Condition Cache seed; zero disables adaptive sharing for + // predicates whose full semantics cannot be represented. Cache that outcome as well. + uint64_t digest = 0xcbf29ce484222325ULL; + for (const auto& conjunct : _conjuncts) { + digest = conjunct->get_digest(digest); + if (digest == 0) { + return; + } + } + _digest = digest; + }); + return _digest; +} + std::string FileScanRequest::debug_string() const { std::ostringstream out; out << "FileScanRequest{predicate_columns=" diff --git a/be/src/format_v2/file_reader.h b/be/src/format_v2/file_reader.h index 688b4e9be53d13..f86e77cf805059 100644 --- a/be/src/format_v2/file_reader.h +++ b/be/src/format_v2/file_reader.h @@ -450,6 +450,10 @@ class FileReader { // granule and switch a newer snapshot at a well-defined boundary. virtual bool supports_scan_request_refresh() const { return false; } + // Only adaptive consumers need a semantic snapshot key. Other readers must not traverse + // large Bloom/IN runtime-filter payloads just to prepare another file. + virtual bool uses_predicate_snapshot_digest() const { return false; } + virtual Status queue_scan_request(std::shared_ptr request) { (void)request; return Status::NotSupported("FileReader does not support scan request refresh"); diff --git a/be/src/format_v2/jni/jni_table_reader.cpp b/be/src/format_v2/jni/jni_table_reader.cpp index f56de9c51d4462..36658ca04671bc 100644 --- a/be/src/format_v2/jni/jni_table_reader.cpp +++ b/be/src/format_v2/jni/jni_table_reader.cpp @@ -77,9 +77,10 @@ Status JniTableReader::prepare_split(const SplitReadOptions& options) { return _open_jni_scanner(); } -Status JniTableReader::refresh_conjuncts(VExprContextSPtrs conjuncts, - std::optional condition_cache_digest, - bool all_runtime_filters_applied) { +Status JniTableReader::refresh_conjuncts( + VExprContextSPtrs conjuncts, std::optional condition_cache_digest, + bool all_runtime_filters_applied, + std::shared_ptr predicate_snapshot) { if (_scanner_opened) { SCOPED_TIMER(_profile.total_timer); SCOPED_TIMER(_profile.refresh_conjuncts_timer); @@ -94,7 +95,8 @@ Status JniTableReader::refresh_conjuncts(VExprContextSPtrs conjuncts, } } return TableReader::refresh_conjuncts(std::move(conjuncts), condition_cache_digest, - all_runtime_filters_applied); + all_runtime_filters_applied, + std::move(predicate_snapshot)); } Status JniTableReader::get_block(Block* output_block, bool* eos) { diff --git a/be/src/format_v2/jni/jni_table_reader.h b/be/src/format_v2/jni/jni_table_reader.h index ac8c14670388e4..76dfa4c9d4c909 100644 --- a/be/src/format_v2/jni/jni_table_reader.h +++ b/be/src/format_v2/jni/jni_table_reader.h @@ -49,9 +49,11 @@ class JniTableReader : public TableReader { Status init(TableReadOptions&& options) override; Status prepare_split(const SplitReadOptions& options) override; - Status refresh_conjuncts(VExprContextSPtrs conjuncts, - std::optional condition_cache_digest = std::nullopt, - bool all_runtime_filters_applied = false) override; + Status refresh_conjuncts( + VExprContextSPtrs conjuncts, + std::optional condition_cache_digest = std::nullopt, + bool all_runtime_filters_applied = false, + std::shared_ptr predicate_snapshot = nullptr) override; Status get_block(Block* block, bool* eos) override; Status abort_split() override; Status close() override; diff --git a/be/src/format_v2/orc/orc_reader.h b/be/src/format_v2/orc/orc_reader.h index 0c8e9011badcbf..381e95a64a3ad2 100644 --- a/be/src/format_v2/orc/orc_reader.h +++ b/be/src/format_v2/orc/orc_reader.h @@ -83,6 +83,10 @@ class OrcReader final : public format::FileReader { Status get_block(Block* file_block, size_t* rows, bool* eof) override; Status get_aggregate_result(const format::FileAggregateRequest& request, format::FileAggregateResult* result) override; + Status get_metadata_aggregate_result(const format::FileAggregateRequest& request, + format::FileAggregateResult* result) override { + return get_aggregate_result(request, result); + } void set_condition_cache_context(std::shared_ptr ctx) override; int64_t get_total_rows() const override; Status close() override; diff --git a/be/src/format_v2/parquet/parquet_reader.h b/be/src/format_v2/parquet/parquet_reader.h index 94b4e40cbb3ec5..1fe3aa0698bb4f 100644 --- a/be/src/format_v2/parquet/parquet_reader.h +++ b/be/src/format_v2/parquet/parquet_reader.h @@ -71,6 +71,7 @@ class ParquetReader : public format::FileReader { Status open(std::shared_ptr request) override; bool supports_scan_request_refresh() const override { return true; } + bool uses_predicate_snapshot_digest() const override { return true; } Status queue_scan_request(std::shared_ptr request) override; diff --git a/be/src/format_v2/predicate_snapshot.h b/be/src/format_v2/predicate_snapshot.h new file mode 100644 index 00000000000000..6a0f7dfd204c92 --- /dev/null +++ b/be/src/format_v2/predicate_snapshot.h @@ -0,0 +1,43 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#pragma once + +#include +#include +#include +#include + +#include "exprs/vexpr_fwd.h" + +namespace doris::format { + +// The scanner replaces this snapshot when its ready runtime-filter set changes. Splits keep +// independent executable conjuncts, while digest consumers share this immutable semantic key. +class PredicateSnapshot { +public: + explicit PredicateSnapshot(VExprContextSPtrs conjuncts) : _conjuncts(std::move(conjuncts)) {} + + std::optional digest() const; + +private: + const VExprContextSPtrs _conjuncts; + mutable std::once_flag _digest_once; + mutable std::optional _digest; +}; + +} // namespace doris::format diff --git a/be/src/format_v2/table/fluss_hybrid_reader.cpp b/be/src/format_v2/table/fluss_hybrid_reader.cpp index acd7ef899a8cb2..e5bc7c024fcc14 100644 --- a/be/src/format_v2/table/fluss_hybrid_reader.cpp +++ b/be/src/format_v2/table/fluss_hybrid_reader.cpp @@ -81,11 +81,13 @@ Status FlussHybridReader::prepare_split(const format::SplitReadOptions& options) return _current_split_reader->prepare_split(options); } -Status FlussHybridReader::refresh_conjuncts(VExprContextSPtrs conjuncts, - std::optional condition_cache_digest, - bool all_runtime_filters_applied) { +Status FlussHybridReader::refresh_conjuncts( + VExprContextSPtrs conjuncts, std::optional condition_cache_digest, + bool all_runtime_filters_applied, + std::shared_ptr predicate_snapshot) { RETURN_IF_ERROR(format::TableReader::refresh_conjuncts( - std::move(conjuncts), condition_cache_digest, all_runtime_filters_applied)); + std::move(conjuncts), condition_cache_digest, all_runtime_filters_applied, + predicate_snapshot)); if (_current_split_reader == nullptr) { return Status::OK(); } @@ -98,7 +100,8 @@ Status FlussHybridReader::refresh_conjuncts(VExprContextSPtrs conjuncts, // The hybrid wrapper owns no physical reader; forward a clone so the active child, rather than // only the wrapper snapshot, observes late predicates for the remainder of this split. return _current_split_reader->refresh_conjuncts( - std::move(child_conjuncts), condition_cache_digest, all_runtime_filters_applied); + std::move(child_conjuncts), condition_cache_digest, all_runtime_filters_applied, + std::move(predicate_snapshot)); } Status FlussHybridReader::get_block(Block* block, bool* eos) { @@ -286,6 +289,7 @@ Status FlussHybridReader::_init_child_reader(format::TableReader* reader, .push_down_agg_type = _push_down_agg_type, .push_down_count_columns = _push_down_count_columns, .condition_cache_digest = _condition_cache_digest, + .predicate_snapshot = _predicate_snapshot, })); // Zero means no adaptive prediction has been produced yet. Preserve the child's normal // runtime default until FileScannerV2 supplies the first positive prediction. diff --git a/be/src/format_v2/table/fluss_hybrid_reader.h b/be/src/format_v2/table/fluss_hybrid_reader.h index 095050a5d4359d..cfdff8e62f1c4a 100644 --- a/be/src/format_v2/table/fluss_hybrid_reader.h +++ b/be/src/format_v2/table/fluss_hybrid_reader.h @@ -53,9 +53,11 @@ class FlussHybridReader final : public format::TableReader { Status init(format::TableReadOptions&& options) override; Status prepare_split(const format::SplitReadOptions& options) override; - Status refresh_conjuncts(VExprContextSPtrs conjuncts, - std::optional condition_cache_digest = std::nullopt, - bool all_runtime_filters_applied = false) override; + Status refresh_conjuncts( + VExprContextSPtrs conjuncts, + std::optional condition_cache_digest = std::nullopt, + bool all_runtime_filters_applied = false, + std::shared_ptr predicate_snapshot = nullptr) override; Status get_block(Block* block, bool* eos) override; bool current_split_pruned() const override; bool current_split_uses_metadata_count() const override; diff --git a/be/src/format_v2/table/fluss_union_lake_reader.cpp b/be/src/format_v2/table/fluss_union_lake_reader.cpp index ba303c6dc4c327..caf17ae01de862 100644 --- a/be/src/format_v2/table/fluss_union_lake_reader.cpp +++ b/be/src/format_v2/table/fluss_union_lake_reader.cpp @@ -375,6 +375,7 @@ Status FlussUnionLakeReader::init(format::TableReadOptions&& options) { // and it would do so without ever producing a block to suppress them from. .push_down_agg_type = TPushAggOp::type::NONE, .condition_cache_digest = _condition_cache_digest, + .predicate_snapshot = _predicate_snapshot, })); if (_batch_size > 0) { _lake_reader->set_batch_size(_batch_size); @@ -482,18 +483,21 @@ Status FlussUnionLakeReader::prepare_split(const format::SplitReadOptions& optio return _prepare_suppression(options); } -Status FlussUnionLakeReader::refresh_conjuncts(VExprContextSPtrs conjuncts, - std::optional condition_cache_digest, - bool all_runtime_filters_applied) { +Status FlussUnionLakeReader::refresh_conjuncts( + VExprContextSPtrs conjuncts, std::optional condition_cache_digest, + bool all_runtime_filters_applied, + std::shared_ptr predicate_snapshot) { RETURN_IF_ERROR(format::TableReader::refresh_conjuncts( - std::move(conjuncts), condition_cache_digest, all_runtime_filters_applied)); + std::move(conjuncts), condition_cache_digest, all_runtime_filters_applied, + predicate_snapshot)); if (_lake_reader == nullptr) { return Status::OK(); } VExprContextSPtrs child_conjuncts; RETURN_IF_ERROR(_clone_conjuncts(&child_conjuncts)); return _lake_reader->refresh_conjuncts(std::move(child_conjuncts), condition_cache_digest, - all_runtime_filters_applied); + all_runtime_filters_applied, + std::move(predicate_snapshot)); } Status FlussUnionLakeReader::_prepare_suppression(const format::SplitReadOptions& options) { diff --git a/be/src/format_v2/table/fluss_union_lake_reader.h b/be/src/format_v2/table/fluss_union_lake_reader.h index bdd95223daf229..84bc811a501360 100644 --- a/be/src/format_v2/table/fluss_union_lake_reader.h +++ b/be/src/format_v2/table/fluss_union_lake_reader.h @@ -144,9 +144,11 @@ class FlussUnionLakeReader final : public format::TableReader { Status init(format::TableReadOptions&& options) override; Status prepare_split(const format::SplitReadOptions& options) override; - Status refresh_conjuncts(VExprContextSPtrs conjuncts, - std::optional condition_cache_digest = std::nullopt, - bool all_runtime_filters_applied = false) override; + Status refresh_conjuncts( + VExprContextSPtrs conjuncts, + std::optional condition_cache_digest = std::nullopt, + bool all_runtime_filters_applied = false, + std::shared_ptr predicate_snapshot = nullptr) override; Status get_block(Block* block, bool* eos) override; bool current_split_pruned() const override; bool current_split_uses_metadata_count() const override; diff --git a/be/src/format_v2/table/hudi_reader.cpp b/be/src/format_v2/table/hudi_reader.cpp index e4489a7f5a7552..e3a8c31ab85d10 100644 --- a/be/src/format_v2/table/hudi_reader.cpp +++ b/be/src/format_v2/table/hudi_reader.cpp @@ -121,11 +121,13 @@ Status HudiHybridReader::build_physical_splits(const FileScanSplit& source_split return _current_split_reader->build_physical_splits(source_split, splits, was_split); } -Status HudiHybridReader::refresh_conjuncts(VExprContextSPtrs conjuncts, - std::optional condition_cache_digest, - bool all_runtime_filters_applied) { +Status HudiHybridReader::refresh_conjuncts( + VExprContextSPtrs conjuncts, std::optional condition_cache_digest, + bool all_runtime_filters_applied, + std::shared_ptr predicate_snapshot) { RETURN_IF_ERROR(format::TableReader::refresh_conjuncts( - std::move(conjuncts), condition_cache_digest, all_runtime_filters_applied)); + std::move(conjuncts), condition_cache_digest, all_runtime_filters_applied, + predicate_snapshot)); if (_current_split_reader == nullptr) { return Status::OK(); } @@ -134,7 +136,8 @@ Status HudiHybridReader::refresh_conjuncts(VExprContextSPtrs conjuncts, // The hybrid wrapper owns no physical reader; forward a clone so the active child, rather than // only the wrapper snapshot, observes late predicates for the remainder of this split. return _current_split_reader->refresh_conjuncts( - std::move(child_conjuncts), condition_cache_digest, all_runtime_filters_applied); + std::move(child_conjuncts), condition_cache_digest, all_runtime_filters_applied, + std::move(predicate_snapshot)); } Status HudiHybridReader::get_block(Block* block, bool* eos) { @@ -242,6 +245,7 @@ Status HudiHybridReader::_init_child_reader(format::TableReader* reader, .push_down_agg_type = _push_down_agg_type, .push_down_count_columns = _push_down_count_columns, .condition_cache_digest = _condition_cache_digest, + .predicate_snapshot = _predicate_snapshot, })); // Zero means no adaptive prediction has been produced yet. Preserve the child's normal // runtime default until FileScannerV2 supplies the first positive prediction. diff --git a/be/src/format_v2/table/hudi_reader.h b/be/src/format_v2/table/hudi_reader.h index d46e6fa43c2e65..dc5cfeb29844a2 100644 --- a/be/src/format_v2/table/hudi_reader.h +++ b/be/src/format_v2/table/hudi_reader.h @@ -64,9 +64,11 @@ class HudiHybridReader final : public format::TableReader { Status prepare_split(const format::SplitReadOptions& options) override; Status build_physical_splits(const FileScanSplit& source_split, std::vector* splits, bool* was_split) override; - Status refresh_conjuncts(VExprContextSPtrs conjuncts, - std::optional condition_cache_digest = std::nullopt, - bool all_runtime_filters_applied = false) override; + Status refresh_conjuncts( + VExprContextSPtrs conjuncts, + std::optional condition_cache_digest = std::nullopt, + bool all_runtime_filters_applied = false, + std::shared_ptr predicate_snapshot = nullptr) override; Status get_block(Block* block, bool* eos) override; bool current_split_pruned() const override; bool current_split_uses_metadata_count() const override; diff --git a/be/src/format_v2/table/load_reader.cpp b/be/src/format_v2/table/load_reader.cpp index 27231155a9e25e..96b451b78ca346 100644 --- a/be/src/format_v2/table/load_reader.cpp +++ b/be/src/format_v2/table/load_reader.cpp @@ -186,7 +186,8 @@ Status LoadTableReader::prepare_split(const SplitReadOptions& options) { return TableReader::prepare_split(source_options); } -Status LoadTableReader::refresh_conjuncts(VExprContextSPtrs, std::optional, bool) { +Status LoadTableReader::refresh_conjuncts(VExprContextSPtrs, std::optional, bool, + std::shared_ptr) { return Status::OK(); } diff --git a/be/src/format_v2/table/load_reader.h b/be/src/format_v2/table/load_reader.h index f2098e046929b3..bd5fa653911626 100644 --- a/be/src/format_v2/table/load_reader.h +++ b/be/src/format_v2/table/load_reader.h @@ -30,9 +30,11 @@ class LoadTableReader : public TableReader { Status init(TableReadOptions&& options) override; Status prepare_split(const SplitReadOptions& options) override; - Status refresh_conjuncts(VExprContextSPtrs conjuncts, - std::optional condition_cache_digest = std::nullopt, - bool all_runtime_filters_applied = false) override; + Status refresh_conjuncts( + VExprContextSPtrs conjuncts, + std::optional condition_cache_digest = std::nullopt, + bool all_runtime_filters_applied = false, + std::shared_ptr predicate_snapshot = nullptr) override; bool current_split_uses_metadata_count() const override { return false; } Status abort_split() override; Status close() override; diff --git a/be/src/format_v2/table/paimon_reader.cpp b/be/src/format_v2/table/paimon_reader.cpp index d8e0ace378ff0b..439321be6d15b6 100644 --- a/be/src/format_v2/table/paimon_reader.cpp +++ b/be/src/format_v2/table/paimon_reader.cpp @@ -407,11 +407,13 @@ Status PaimonHybridReader::build_physical_splits(const FileScanSplit& source_spl return _current_split_reader->build_physical_splits(source_split, splits, was_split); } -Status PaimonHybridReader::refresh_conjuncts(VExprContextSPtrs conjuncts, - std::optional condition_cache_digest, - bool all_runtime_filters_applied) { +Status PaimonHybridReader::refresh_conjuncts( + VExprContextSPtrs conjuncts, std::optional condition_cache_digest, + bool all_runtime_filters_applied, + std::shared_ptr predicate_snapshot) { RETURN_IF_ERROR(format::TableReader::refresh_conjuncts( - std::move(conjuncts), condition_cache_digest, all_runtime_filters_applied)); + std::move(conjuncts), condition_cache_digest, all_runtime_filters_applied, + predicate_snapshot)); if (_current_split_reader == nullptr) { return Status::OK(); } @@ -420,7 +422,8 @@ Status PaimonHybridReader::refresh_conjuncts(VExprContextSPtrs conjuncts, // The hybrid wrapper owns no physical reader; forward a clone so the active child, rather than // only the wrapper snapshot, observes late predicates for the remainder of this split. return _current_split_reader->refresh_conjuncts( - std::move(child_conjuncts), condition_cache_digest, all_runtime_filters_applied); + std::move(child_conjuncts), condition_cache_digest, all_runtime_filters_applied, + std::move(predicate_snapshot)); } Status PaimonHybridReader::get_block(Block* block, bool* eos) { @@ -559,6 +562,7 @@ Status PaimonHybridReader::_init_child_reader(format::TableReader* reader, .push_down_agg_type = _push_down_agg_type, .push_down_count_columns = _push_down_count_columns, .condition_cache_digest = _condition_cache_digest, + .predicate_snapshot = _predicate_snapshot, })); // Zero means no adaptive prediction has been produced yet. Preserve the child's normal // runtime default until FileScannerV2 supplies the first positive prediction. diff --git a/be/src/format_v2/table/paimon_reader.h b/be/src/format_v2/table/paimon_reader.h index db1afdac4107dc..61e105f8affd02 100644 --- a/be/src/format_v2/table/paimon_reader.h +++ b/be/src/format_v2/table/paimon_reader.h @@ -89,9 +89,11 @@ class PaimonHybridReader final : public format::TableReader { Status prepare_split(const format::SplitReadOptions& options) override; Status build_physical_splits(const FileScanSplit& source_split, std::vector* splits, bool* was_split) override; - Status refresh_conjuncts(VExprContextSPtrs conjuncts, - std::optional condition_cache_digest = std::nullopt, - bool all_runtime_filters_applied = false) override; + Status refresh_conjuncts( + VExprContextSPtrs conjuncts, + std::optional condition_cache_digest = std::nullopt, + bool all_runtime_filters_applied = false, + std::shared_ptr predicate_snapshot = nullptr) override; Status get_block(Block* block, bool* eos) override; bool current_split_pruned() const override; bool current_split_uses_metadata_count() const override; diff --git a/be/src/format_v2/table_reader.cpp b/be/src/format_v2/table_reader.cpp index 9877935c9c9984..eaf2196957b32c 100644 --- a/be/src/format_v2/table_reader.cpp +++ b/be/src/format_v2/table_reader.cpp @@ -53,6 +53,7 @@ #include "format_v2/json/json_reader.h" #include "format_v2/orc/orc_reader.h" #include "format_v2/parquet/parquet_reader.h" +#include "format_v2/predicate_snapshot.h" #include "format_v2/table/schema_history_util.h" // get_field_ptr #include "runtime/file_scan_profile.h" #include "storage/segment/condition_cache.h" @@ -82,19 +83,6 @@ std::optional TableReader::_get_int96_timezone_override( namespace { -std::optional build_predicate_snapshot_digest(const VExprContextSPtrs& conjuncts) { - // Adaptive state must remain independent of the optional Condition Cache seed. A zero result - // means an expression cannot provide a stable semantic digest, so sharing is disabled. - uint64_t digest = 0xcbf29ce484222325ULL; - for (const auto& conjunct : conjuncts) { - digest = conjunct->get_digest(digest); - if (digest == 0) { - return std::nullopt; - } - } - return digest; -} - void extend_format_split_id_range(PhysicalFileSplit* destination, const PhysicalFileSplit& source) { DORIS_CHECK(destination != nullptr); DORIS_CHECK(destination->format_split_id >= 0 && source.format_split_id >= 0); @@ -1084,7 +1072,9 @@ Status TableReader::init(TableReadOptions&& options) { _system_properties = create_system_properties(_scan_params); _mapper_options.mode = TableColumnMappingMode::BY_NAME; _conjuncts = std::move(options.conjuncts); - _predicate_snapshot_digest = build_predicate_snapshot_digest(_conjuncts); + _predicate_snapshot = options.predicate_snapshot != nullptr + ? options.predicate_snapshot + : std::make_shared(_conjuncts); return Status::OK(); } @@ -1175,13 +1165,23 @@ bool same_physical_scan_layout(const FileScanRequest& lhs, const FileScanRequest } // namespace -Status TableReader::refresh_conjuncts(VExprContextSPtrs conjuncts, - std::optional condition_cache_digest, - bool all_runtime_filters_applied) { +std::optional TableReader::_current_predicate_snapshot_digest() const { + // CSV, ORC and JNI never consume adaptive predicate keys. In particular, preparing 10,000 + // CSV files must not hash the same multi-MiB Bloom filter 10,000 times. + return _data_reader.reader->uses_predicate_snapshot_digest() ? _predicate_snapshot->digest() + : std::nullopt; +} + +Status TableReader::refresh_conjuncts( + VExprContextSPtrs conjuncts, std::optional condition_cache_digest, + bool all_runtime_filters_applied, + std::shared_ptr predicate_snapshot) { SCOPED_TIMER(_profile.total_timer); SCOPED_TIMER(_profile.refresh_conjuncts_timer); _conjuncts = std::move(conjuncts); - _predicate_snapshot_digest = build_predicate_snapshot_digest(_conjuncts); + _predicate_snapshot = predicate_snapshot != nullptr + ? std::move(predicate_snapshot) + : std::make_shared(_conjuncts); // A prepared footer result belongs to the prior immutable predicate snapshot. Discard it // before a refresh can make the planning reader resume ordinary row production. _metadata_aggregate_result.reset(); @@ -1220,7 +1220,7 @@ Status TableReader::refresh_conjuncts(VExprContextSPtrs conjuncts, _file_scan_request == nullptr ? nullptr : &_file_scan_request->non_predicate_positions)); refreshed_request->row_ids = _row_ids; - refreshed_request->predicate_snapshot_digest = _predicate_snapshot_digest; + refreshed_request->predicate_snapshot_digest = _current_predicate_snapshot_digest(); // A refresh does not prove that every future runtime filter has arrived. Keep carrier values // available whenever the split started with pending filters. if (_push_down_agg_type == TPushAggOp::type::COUNT && _push_down_count_columns.has_value() && @@ -1384,8 +1384,7 @@ void TableReader::_finalize_reader_condition_cache() { const bool cache_hit = _condition_cache_ctx != nullptr && _condition_cache_ctx->is_hit; const bool complete_miss = _condition_cache_initialized && !_condition_cache_split_invalid && _current_reader_reached_eof; - std::shared_ptr> published_filter; - int64_t published_base_granule = 0; + std::vector> completed_fragments; { std::scoped_lock lock(_condition_cache_split_context->lock); auto& split_context = *_condition_cache_split_context; @@ -1407,54 +1406,46 @@ void TableReader::_finalize_reader_condition_cache() { split_context.valid = false; } else if (_condition_cache_ctx != nullptr && _condition_cache != nullptr) { DORIS_CHECK(_condition_cache_ctx->num_granules <= _condition_cache->size()); - const int64_t local_base = _condition_cache_ctx->base_granule; - const size_t local_size = _condition_cache_ctx->num_granules; - if (split_context.merged_filter_result.empty()) { - split_context.base_granule = local_base; - split_context.merged_filter_result.assign(local_size, false); - } else { - const int64_t merged_end = - split_context.base_granule + - cast_set(split_context.merged_filter_result.size()); - const int64_t local_end = local_base + cast_set(local_size); - const int64_t combined_base = std::min(split_context.base_granule, local_base); - const int64_t combined_end = std::max(merged_end, local_end); - if (combined_base != split_context.base_granule || combined_end != merged_end) { - std::vector combined(cast_set(combined_end - combined_base), - false); - for (size_t index = 0; index < split_context.merged_filter_result.size(); - ++index) { - combined[cast_set(split_context.base_granule - combined_base) + - index] = split_context.merged_filter_result[index]; - } - split_context.base_granule = combined_base; - split_context.merged_filter_result = std::move(combined); - } - } - for (size_t index = 0; index < local_size; ++index) { - const auto merged_index = - cast_set(local_base - split_context.base_granule) + index; - split_context.merged_filter_result[merged_index] = - split_context.merged_filter_result[merged_index] || - (*_condition_cache)[index]; - } + // A child owns its MISS bitmap through EOF. Transfer that owner in O(1) instead + // of reallocating/copying the source prefix for every consecutive row group. + split_context.fragments.push_back(std::move(_condition_cache_ctx)); } DORIS_CHECK(split_context.remaining_children > 0); --split_context.remaining_children; if (split_context.remaining_children == 0 && split_context.valid && - !split_context.cache_hit_seen && !split_context.merged_filter_result.empty()) { + !split_context.cache_hit_seen && !split_context.fragments.empty()) { // A source-level entry is visible only after every physical child reaches EOF; // publishing an earlier partial MISS would let a sibling HIT skip valid rows. - published_base_granule = split_context.base_granule; - published_filter = std::make_shared>( - std::move(split_context.merged_filter_result)); + completed_fragments = std::move(split_context.fragments); } } - if (published_filter != nullptr) { - if (auto* cache = segment_v2::ConditionCache::instance(); cache != nullptr) { - cache->insert(_condition_cache_key, std::move(published_filter), - published_base_granule); + if (!completed_fragments.empty()) { + // Only the final child owns these immutable fragments now. Allocate the full span + // once and OR every child bit once, outside the coordinator lock. Adjacent row groups + // may share a granule, so concatenation alone would lose surviving rows. + int64_t base_granule = completed_fragments.front()->base_granule; + int64_t end_granule = base_granule; + for (const auto& fragment : completed_fragments) { + base_granule = std::min(base_granule, fragment->base_granule); + end_granule = + std::max(end_granule, fragment->base_granule + + cast_set(fragment->num_granules)); + } + auto published_filter = std::make_shared>( + cast_set(end_granule - base_granule), false); + for (const auto& fragment : completed_fragments) { + const auto offset = cast_set(fragment->base_granule - base_granule); + for (size_t index = 0; index < fragment->num_granules; ++index) { + (*published_filter)[offset + index] = (*published_filter)[offset + index] || + (*fragment->filter_result)[index]; + } + } + // Zero authoritative granules must not create an empty source-level cache entry. + if (!published_filter->empty()) { + if (auto* cache = segment_v2::ConditionCache::instance(); cache != nullptr) { + cache->insert(_condition_cache_key, std::move(published_filter), base_granule); + } } } _condition_cache = nullptr; @@ -1642,7 +1633,9 @@ Status TableReader::prepare_split(const SplitReadOptions& options) { } if (options.conjuncts.has_value()) { _conjuncts = *options.conjuncts; - _predicate_snapshot_digest = build_predicate_snapshot_digest(_conjuncts); + _predicate_snapshot = options.predicate_snapshot != nullptr + ? options.predicate_snapshot + : std::make_shared(_conjuncts); } // Update to current split format to handle ORC/PARQUET files in one table. _format = options.current_split_format; diff --git a/be/src/format_v2/table_reader.h b/be/src/format_v2/table_reader.h index e4c09b131fb215..d68afacdc69d94 100644 --- a/be/src/format_v2/table_reader.h +++ b/be/src/format_v2/table_reader.h @@ -82,6 +82,7 @@ class ParquetProfileCache; } using DeleteRows = std::vector; +class PredicateSnapshot; // Row-level predicates on table/global schema. They are rewritten to file-local expressions when // possible, and remain the source of row-level filtering after localization. @@ -171,6 +172,9 @@ struct TableReadOptions { // with SplitReadOptions::condition_cache_digest after collecting late-arrival runtime filters. // A zero digest disables condition cache. uint64_t condition_cache_digest = 0; + // Scanner-owned immutable semantics, shared across executable clones and physical splits. + // Standalone callers may omit it; only readers opting into adaptive sharing request its key. + std::shared_ptr predicate_snapshot = nullptr; }; struct SplitReadOptions { @@ -206,6 +210,8 @@ struct SplitReadOptions { // Optional absolute file-row selection used by TopN two-phase materialization. TableReader // carries it unchanged into the format-neutral FileScanRequest. std::optional> row_ids = std::nullopt; + // Must describe conjuncts above. A late RF replaces this owner rather than mutating its key. + std::shared_ptr predicate_snapshot = nullptr; }; // Base class for table-level readers. @@ -254,9 +260,11 @@ class TableReader { // this operation decide the safe boundary at which the new immutable request becomes active. // A supplied digest describes this exact conjunct snapshot for condition-cache isolation. RF // completeness is monotonic within one split and enables safe COUNT(*) placeholder elision. - virtual Status refresh_conjuncts(VExprContextSPtrs conjuncts, - std::optional condition_cache_digest = std::nullopt, - bool all_runtime_filters_applied = false); + virtual Status refresh_conjuncts( + VExprContextSPtrs conjuncts, + std::optional condition_cache_digest = std::nullopt, + bool all_runtime_filters_applied = false, + std::shared_ptr predicate_snapshot = nullptr); virtual bool current_split_pruned() const { return _current_split_pruned; } virtual bool current_split_uses_metadata_count() const { @@ -486,7 +494,7 @@ class TableReader { RETURN_IF_ERROR(_data_reader.column_mapper->create_scan_request( _table_filters, _projected_columns, file_request.get(), _runtime_state)); file_request->row_ids = _row_ids; - file_request->predicate_snapshot_digest = _predicate_snapshot_digest; + file_request->predicate_snapshot_digest = _current_predicate_snapshot_digest(); _constant_pruning_safe_filter_count = std::min(_constant_pruning_safe_filter_count, file_request->constant_pruning_safe_table_filter_count); @@ -582,6 +590,7 @@ class TableReader { Status _open_local_filter_exprs(const FileScanRequest& file_request); Status _init_reader_condition_cache(const FileScanRequest& file_request); void _finalize_reader_condition_cache(); + std::optional _current_predicate_snapshot_digest() const; // The condition cache key covers the file identity plus the table-level predicate/session // digest only. A table format whose file-column binding can change without changing that digest // (Iceberg's positional fallback resolves an ID-less file by physical ordinals) must override @@ -2296,7 +2305,7 @@ class TableReader { FileFormat _format; TPushAggOp::type _push_down_agg_type = TPushAggOp::type::NONE; std::optional> _push_down_count_columns; - std::optional _predicate_snapshot_digest; + std::shared_ptr _predicate_snapshot; size_t _batch_size = 0; uint64_t _initial_condition_cache_digest = 0; uint64_t _condition_cache_digest = 0; diff --git a/be/src/storage/segment/condition_cache.h b/be/src/storage/segment/condition_cache.h index 876f91afcc70bf..16f03e52982d8b 100644 --- a/be/src/storage/segment/condition_cache.h +++ b/be/src/storage/segment/condition_cache.h @@ -65,8 +65,9 @@ struct ConditionCacheSplitContext { bool valid = true; bool cache_hit_seen = false; std::optional encoded_key; - int64_t base_granule = 0; - std::vector merged_filter_result; + // Keep completed child bitmaps without copying their growing prefix. The last child takes + // ownership and assembles their union once, after releasing lock. + std::vector> fragments; }; namespace segment_v2 { diff --git a/be/test/exec/scan/file_scanner_v2_test.cpp b/be/test/exec/scan/file_scanner_v2_test.cpp index 8961879d46607b..1cc253b6073003 100644 --- a/be/test/exec/scan/file_scanner_v2_test.cpp +++ b/be/test/exec/scan/file_scanner_v2_test.cpp @@ -57,6 +57,7 @@ #include "format/table/iceberg_scan_semantics.h" #include "format_v2/expr/cast.h" #include "format_v2/file_scan_context.h" +#include "format_v2/predicate_snapshot.h" #include "format_v2/table/fluss_hybrid_reader.h" #include "storage/id_manager.h" #include "testutil/mock/mock_runtime_state.h" @@ -182,7 +183,8 @@ class CapturingSplitTableReader final : public format::TableReader { class ErrorContextTableReader final : public format::TableReader { public: Status prepare_split(const format::SplitReadOptions&) override { return prepare_status; } - Status refresh_conjuncts(VExprContextSPtrs, std::optional, bool) override { + Status refresh_conjuncts(VExprContextSPtrs, std::optional, bool, + std::shared_ptr) override { return refresh_status; } Status get_block(Block*, bool* eof) override { @@ -231,11 +233,29 @@ TEST(FileScannerV2Test, ReaderErrorsRetainCurrentPathAndOrcContext) { TEST(FileScannerV2Test, RefreshErrorsRetainCurrentPath) { RuntimeState state {TQueryOptions(), TQueryGlobals()}; RuntimeProfile profile("refresh_error_context"); + ObjectPool pool; + TDescriptorTable thrift_descriptors; + TTupleDescriptor tuple; + tuple.__set_id(0); + thrift_descriptors.tupleDescriptors.push_back(tuple); + DescriptorTbl* descriptors = nullptr; + ASSERT_TRUE(DescriptorTbl::create(&pool, thrift_descriptors, &descriptors).ok()); + TPlanNode plan; + plan.__set_node_id(0); + plan.__set_node_type(TPlanNodeType::FILE_SCAN_NODE); + plan.__set_limit(-1); + plan.__set_row_tuples({0}); + TFileScanNode scan; + scan.__set_tuple_id(0); + plan.__set_file_scan_node(scan); + FileScanOperatorX parent(&pool, plan, 0, *descriptors, 1); + FileScanLocalState local_state(&state, &parent); TFileScanRangeParams params; auto reader = std::make_unique(); reader->refresh_status = Status::InvalidArgument("injected refresh failure"); FileScannerV2 scanner(&state, &profile, std::move(reader)); scanner._params = ¶ms; + scanner._local_state = &local_state; scanner._has_prepared_split = true; scanner._applied_rf_num = 1; scanner._current_range.__set_path("s3://test-bucket/refresh.parquet"); @@ -493,6 +513,33 @@ TEST(FileScannerV2Test, ConditionCacheDigestIncludesRuntimeFilterPayload) { 0); } +TEST(FileScannerV2Test, PredicateSnapshotSurvivesSplitsAndChangesOnLateRuntimeFilters) { + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + RuntimeProfile profile("predicate_snapshot"); + FileScannerV2 scanner(&state, &profile, nullptr); + scanner._is_load = false; + scanner._slot_id_to_global_index.emplace(1, format::GlobalIndex(0)); + scanner._conjuncts = {int_in_runtime_filter({7, 9}, 1)}; + scanner._applied_rf_num = 1; + VExprContextSPtrs conjuncts; + ASSERT_TRUE(scanner._build_table_conjuncts(&conjuncts).ok()); + const auto first = scanner._predicate_snapshot; + ASSERT_NE(first, nullptr); + const auto digest = first->digest(); + ASSERT_TRUE(digest.has_value()); + for (int split = 0; split < 10; ++split) { + ASSERT_TRUE(scanner._build_table_conjuncts(&conjuncts).ok()); + EXPECT_EQ(scanner._predicate_snapshot, first); + EXPECT_EQ(scanner._predicate_snapshot->digest(), digest); + } + scanner._conjuncts.push_back(int_minmax_runtime_filter(10, 2)); + scanner._applied_rf_num = 2; + ASSERT_TRUE(scanner._build_table_conjuncts(&conjuncts).ok()); + EXPECT_NE(scanner._predicate_snapshot, first); + EXPECT_NE(scanner._predicate_snapshot->digest(), digest); + EXPECT_EQ(first->digest(), digest); +} + TEST(FileScannerV2Test, RemoteSplitSourceStopsLocallyWithoutQueryCancellation) { RuntimeState state {TQueryOptions(), TQueryGlobals()}; RemoteSplitSourceConnector source(&state, nullptr, 1, 1, 1); diff --git a/be/test/format_v2/orc/orc_reader_test.cpp b/be/test/format_v2/orc/orc_reader_test.cpp index 55a81e7929b8cc..50270de1f86edc 100644 --- a/be/test/format_v2/orc/orc_reader_test.cpp +++ b/be/test/format_v2/orc/orc_reader_test.cpp @@ -79,6 +79,7 @@ #include "format_v2/file_reader.h" #include "format_v2/parquet/parquet_profile.h" #include "format_v2/schema_reader.h" +#include "format_v2/table_reader.h" #include "gen_cpp/Types_types.h" #include "io/fs/buffered_reader.h" #include "io/fs/file_meta_cache.h" @@ -4911,6 +4912,60 @@ TEST_F(NewOrcReaderTest, InitRestoresNotFoundFromReadFailure) { EXPECT_TRUE(st.is()) << st; } +TEST_F(NewOrcReaderTest, MetadataAggregatesKeepTablePlanningReader) { + const auto path = (_test_dir / "metadata_aggregate.orc").string(); + write_multi_stripe_orc_int_file(path); + ASSERT_EQ(get_orc_stripe_count(path), 2); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + format::ColumnDefinition column; + column.identifier = Field::create_field("id"); + column.name = "id"; + column.type = make_nullable(std::make_shared()); + for (auto agg : {TPushAggOp::COUNT, TPushAggOp::MINMAX}) { + format::TableReader reader; + ASSERT_TRUE( + reader.init({ + .projected_columns = {column}, + .conjuncts = {}, + .format = format::FileFormat::ORC, + .scan_params = nullptr, + .io_ctx = nullptr, + .runtime_state = &state, + .scanner_profile = nullptr, + .push_down_agg_type = agg, + .push_down_count_columns = std::vector {}, + }) + .ok()); + format::SplitReadOptions options; + options.current_split_format = format::FileFormat::ORC; + options.current_range.__set_path(path); + options.current_range.__set_file_size(std::filesystem::file_size(path)); + options.current_range.__set_start_offset(0); + options.current_range.__set_size(std::filesystem::file_size(path)); + options.current_range.__set_modification_time(17); + options.current_range.__set_target_split_size(1); + ASSERT_TRUE(reader.prepare_split(options).ok()); + FileScanSplit source; + source.range = options.current_range; + source.is_source_split = true; + std::vector children; + bool was_split = false; + ASSERT_TRUE(reader.build_physical_splits(source, &children, &was_split).ok()); + EXPECT_FALSE(was_split); + EXPECT_TRUE(children.empty()); + ASSERT_NE(reader._data_reader.reader, nullptr); + ASSERT_TRUE(reader._metadata_aggregate_result.has_value()); + EXPECT_EQ(reader._metadata_aggregate_result->count, 400); + if (agg == TPushAggOp::MINMAX) { + const auto& result = reader._metadata_aggregate_result->columns; + ASSERT_EQ(result.size(), 1); + EXPECT_EQ(result[0].min_value.get(), 1); + EXPECT_EQ(result[0].max_value.get(), 1199); + } + ASSERT_TRUE(reader.close().ok()); + } +} + TEST_F(NewOrcReaderTest, AggregatePushdownCountUsesOnlySplitStripes) { const auto multi_stripe_file_path = (_test_dir / "aggregate_count_split.orc").string(); write_multi_stripe_orc_int_file(multi_stripe_file_path); @@ -5387,6 +5442,9 @@ TEST_F(NewOrcReaderTest, AggregatePushdownTimestampMinMaxFallsBackWithoutExactNa format::FileAggregateResult aggregate_result; const auto status = reader->get_aggregate_result(aggregate_request, &aggregate_result); EXPECT_TRUE(status.is()) << status; + const auto metadata_status = + reader->get_metadata_aggregate_result(aggregate_request, &aggregate_result); + EXPECT_TRUE(metadata_status.is()) << metadata_status; }; assert_fallback("aggregate_timestamp_no_nanos_tail.orc", false); diff --git a/be/test/format_v2/parquet/parquet_encryption_test.cpp b/be/test/format_v2/parquet/parquet_encryption_test.cpp index 52b626189eeec0..f7fb204d70c8c4 100644 --- a/be/test/format_v2/parquet/parquet_encryption_test.cpp +++ b/be/test/format_v2/parquet/parquet_encryption_test.cpp @@ -1220,9 +1220,9 @@ TEST(ParquetEncryptionTest, EncryptedIndexRangesRejectInvalidOffsetsBeforeIo) { auto metadata = context.native_metadata->to_thrift(); metadata.row_groups[0].columns[0].__set_offset_index_offset(-1); auto crypto = context.native_metadata->encryption(); - context.native_metadata_owner = + auto invalid_metadata = std::make_unique(std::move(metadata), 0, crypto); - context.native_metadata = context.native_metadata_owner.get(); + context.native_metadata = invalid_metadata.get(); const auto reads = file->read_count(); std::unordered_map pages; std::unordered_map offsets; diff --git a/be/test/format_v2/table/fluss_union_lake_reader_test.cpp b/be/test/format_v2/table/fluss_union_lake_reader_test.cpp index a6cada38380942..422e75567e212a 100644 --- a/be/test/format_v2/table/fluss_union_lake_reader_test.cpp +++ b/be/test/format_v2/table/fluss_union_lake_reader_test.cpp @@ -71,10 +71,13 @@ class RecordingLakeReader final : public format::TableReader { Status refresh_conjuncts(VExprContextSPtrs conjuncts, std::optional condition_cache_digest = std::nullopt, - bool all_runtime_filters_applied = false) override { + bool all_runtime_filters_applied = false, + std::shared_ptr predicate_snapshot = + nullptr) override { ++refresh_calls; return format::TableReader::refresh_conjuncts(std::move(conjuncts), condition_cache_digest, - all_runtime_filters_applied); + all_runtime_filters_applied, + std::move(predicate_snapshot)); } Status close() override { @@ -905,7 +908,10 @@ TEST(FlussUnionLakeReaderTest, ForwardsLateConjunctsToTheLakeReader) { ASSERT_TRUE(init_reader(&reader, &scan_params).ok()); auto* lake = install_lake_reader(&reader); - ASSERT_TRUE(reader.refresh_conjuncts({}).ok()); + const auto snapshot = reader._predicate_snapshot; + ASSERT_NE(snapshot, nullptr); + ASSERT_TRUE(reader.refresh_conjuncts({}, std::nullopt, false, snapshot).ok()); + EXPECT_EQ(lake->_predicate_snapshot, snapshot); EXPECT_EQ(lake->refresh_calls, 1); } diff --git a/be/test/format_v2/table/hudi_reader_test.cpp b/be/test/format_v2/table/hudi_reader_test.cpp index 8baf3f89887fc5..510ad60bbf6458 100644 --- a/be/test/format_v2/table/hudi_reader_test.cpp +++ b/be/test/format_v2/table/hudi_reader_test.cpp @@ -149,10 +149,13 @@ class RefreshTrackingTableReader final : public TableReader { Status refresh_conjuncts(VExprContextSPtrs conjuncts, std::optional condition_cache_digest = std::nullopt, - bool all_runtime_filters_applied = false) override { + bool all_runtime_filters_applied = false, + std::shared_ptr predicate_snapshot = + nullptr) override { ++refresh_count; return TableReader::refresh_conjuncts(std::move(conjuncts), condition_cache_digest, - all_runtime_filters_applied); + all_runtime_filters_applied, + std::move(predicate_snapshot)); } int refresh_count = 0; @@ -371,7 +374,11 @@ TEST(HudiHybridReaderTest, ForwardsLatePredicatesToActiveChild) { split.current_range.__set_format_type(TFileFormatType::FORMAT_PARQUET); ASSERT_TRUE(reader.prepare_split(split).ok()); ASSERT_NE(child, nullptr); - ASSERT_TRUE(reader.refresh_conjuncts({}).ok()); + const auto snapshot = reader._predicate_snapshot; + ASSERT_NE(snapshot, nullptr); + EXPECT_EQ(child->_predicate_snapshot, snapshot); + ASSERT_TRUE(reader.refresh_conjuncts({}, std::nullopt, false, snapshot).ok()); + EXPECT_EQ(child->_predicate_snapshot, snapshot); EXPECT_EQ(child->refresh_count, 1); } diff --git a/be/test/format_v2/table/paimon_reader_test.cpp b/be/test/format_v2/table/paimon_reader_test.cpp index a1df88536b86b5..3f721ccd498818 100644 --- a/be/test/format_v2/table/paimon_reader_test.cpp +++ b/be/test/format_v2/table/paimon_reader_test.cpp @@ -98,10 +98,13 @@ class RefreshTrackingTableReader final : public TableReader { Status refresh_conjuncts(VExprContextSPtrs conjuncts, std::optional condition_cache_digest = std::nullopt, - bool all_runtime_filters_applied = false) override { + bool all_runtime_filters_applied = false, + std::shared_ptr predicate_snapshot = + nullptr) override { ++refresh_count; return TableReader::refresh_conjuncts(std::move(conjuncts), condition_cache_digest, - all_runtime_filters_applied); + all_runtime_filters_applied, + std::move(predicate_snapshot)); } int refresh_count = 0; @@ -1343,7 +1346,11 @@ TEST(PaimonHybridReaderTest, ForwardsLatePredicatesToActiveChild) { split.current_range = make_paimon_native_range(TFileFormatType::FORMAT_PARQUET); ASSERT_TRUE(reader.prepare_split(split).ok()); ASSERT_NE(child, nullptr); - ASSERT_TRUE(reader.refresh_conjuncts({}).ok()); + const auto snapshot = reader._predicate_snapshot; + ASSERT_NE(snapshot, nullptr); + EXPECT_EQ(child->_predicate_snapshot, snapshot); + ASSERT_TRUE(reader.refresh_conjuncts({}, std::nullopt, false, snapshot).ok()); + EXPECT_EQ(child->_predicate_snapshot, snapshot); EXPECT_EQ(child->refresh_count, 1); } diff --git a/be/test/format_v2/table_reader_test.cpp b/be/test/format_v2/table_reader_test.cpp index 876ba4297b34db..08d20a7585c0a5 100644 --- a/be/test/format_v2/table_reader_test.cpp +++ b/be/test/format_v2/table_reader_test.cpp @@ -62,6 +62,7 @@ #include "exprs/vslot_ref.h" #include "format/table/iceberg_scan_semantics.h" #include "format_v2/expr/cast.h" +#include "format_v2/predicate_snapshot.h" #include "format_v2/table/iceberg_reader.h" #include "gen_cpp/Exprs_types.h" #include "gen_cpp/ExternalTableSchema_types.h" @@ -427,6 +428,35 @@ VExprSPtr runtime_filter_wrapper_expr(VExprSPtr impl) { return RuntimeFilterExpr::create_shared(node, std::move(impl), 0, false, /*filter_id=*/1); } +class RecordingDigestPredicate final : public VExpr { +public: + RecordingDigestPredicate(size_t* calls, uint64_t value) + : VExpr(std::make_shared(), false), _calls(calls), _value(value) { + set_node_type(TExprNodeType::FUNCTION_CALL); + } + + Status execute_column_impl(VExprContext*, const Block*, const Selector*, size_t count, + ColumnPtr& result_column) const override { + result_column = ColumnUInt8::create(count, 1); + return Status::OK(); + } + + const std::string& expr_name() const override { return _name; } + uint64_t get_digest(uint64_t seed) const override { + ++*_calls; + return _value == 0 ? 0 : seed ^ _value; + } + Status clone_node(VExprSPtr* cloned_expr) const override { + *cloned_expr = std::make_shared(_calls, _value); + return Status::OK(); + } + +private: + size_t* _calls; + uint64_t _value; + const std::string _name = "RecordingDigestPredicate"; +}; + class NonDeterministicPartitionPredicate final : public VExpr { public: explicit NonDeterministicPartitionPredicate(bool* executed) @@ -1483,6 +1513,7 @@ struct FakeFileReaderState { bool stop_during_read = false; bool not_found_during_init = false; bool physical_split_error = false; + bool uses_predicate_snapshot_digest = false; int physical_split_count = -1; std::vector physical_split_sizes; std::vector physical_split_ids; @@ -1544,6 +1575,9 @@ class FakeFileReader final : public FileReader { } bool supports_scan_request_refresh() const override { return true; } + bool uses_predicate_snapshot_digest() const override { + return _state->uses_predicate_snapshot_digest; + } Status queue_scan_request(std::shared_ptr request) override { _state->pending_request = std::move(request); @@ -2164,6 +2198,91 @@ TEST(TableReaderTest, CanUseInjectedFileReaderForStandaloneUnitTest) { EXPECT_TRUE(eos); } +TEST(TableReaderTest, PredicateSnapshotDigestIsLazyAndSharedAcrossSplitsAndRefresh) { + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + std::vector file_schema { + make_file_column(0, "id", std::make_shared())}; + std::vector projected_columns { + make_table_column(0, "id", std::make_shared())}; + set_name_identifiers(&projected_columns); + size_t first_calls = 0; + const auto recording_conjunct = [&] { + auto root = std::make_shared(&first_calls, 17); + root->add_child(table_int32_greater_than_expr(0, 0, 0)); + return prepared_conjunct(&state, std::move(root)); + }; + auto snapshot = + std::make_shared(VExprContextSPtrs {VExprContext::create_shared( + std::make_shared(&first_calls, 17))}); + auto fake_state = std::make_shared(); + FakeTableReader reader(file_schema, fake_state); + ASSERT_TRUE(reader.init({ + .projected_columns = projected_columns, + .conjuncts = {recording_conjunct()}, + .format = FileFormat::PARQUET, + .scan_params = nullptr, + .io_ctx = nullptr, + .runtime_state = &state, + .scanner_profile = nullptr, + .predicate_snapshot = snapshot, + }) + .ok()); + // Nonconsuming readers must not hash even the first payload. Changing the executable clones + // at each split must not invalidate the scanner's unchanged semantic snapshot. + for (auto format : {FileFormat::CSV, FileFormat::ORC, FileFormat::PARQUET}) { + fake_state->uses_predicate_snapshot_digest = format == FileFormat::PARQUET; + for (int split = 0; split < 3; ++split) { + SplitReadOptions options; + options.current_range.__set_path("fake-table-reader-input"); + options.current_split_format = format; + options.conjuncts = VExprContextSPtrs {recording_conjunct()}; + options.predicate_snapshot = snapshot; + ASSERT_TRUE(reader.prepare_split(options).ok()); + Block block = build_table_block(projected_columns); + bool eos = false; + ASSERT_TRUE(reader.get_block(&block, &eos).ok()); + EXPECT_EQ(first_calls, format == FileFormat::PARQUET ? 1 : 0); + } + } + const auto first_digest = fake_state->last_request->predicate_snapshot_digest; + ASSERT_TRUE(first_digest.has_value()); + size_t refreshed_calls = 0; + auto refreshed_snapshot = + std::make_shared(VExprContextSPtrs {VExprContext::create_shared( + std::make_shared(&refreshed_calls, 19))}); + SplitReadOptions options; + options.current_range.__set_path("fake-table-reader-input"); + ASSERT_TRUE(reader.prepare_split(options).ok()); + ASSERT_TRUE(reader.refresh_conjuncts( + {prepared_conjunct(&state, table_int32_greater_than_expr(0, 0, 1))}, + std::nullopt, true, refreshed_snapshot) + .ok()); + EXPECT_EQ(refreshed_calls, 0); + Block block = build_table_block(projected_columns); + bool eos = false; + ASSERT_TRUE(reader.get_block(&block, &eos).ok()); + EXPECT_EQ(refreshed_calls, 1); + EXPECT_NE(fake_state->last_request->predicate_snapshot_digest, first_digest); + ASSERT_TRUE(reader.refresh_conjuncts( + {prepared_conjunct(&state, table_int32_greater_than_expr(0, 0, 1))}, + std::nullopt, true, refreshed_snapshot) + .ok()); + EXPECT_EQ(refreshed_calls, 1); + EXPECT_EQ(first_calls, 1); + ASSERT_TRUE(reader.close().ok()); +} + +TEST(TableReaderTest, PredicateSnapshotMemoizesUndigestibleResult) { + size_t calls = 0; + PredicateSnapshot snapshot( + {VExprContext::create_shared(std::make_shared(&calls, 0))}); + EXPECT_EQ(calls, 0); + for (int split = 0; split < 10; ++split) { + EXPECT_FALSE(snapshot.digest().has_value()); + } + EXPECT_EQ(calls, 1); +} + TEST(TableReaderTest, SinglePhysicalSplitReusesPlanningReader) { std::vector file_schema; file_schema.push_back(make_file_column(0, "id", std::make_shared())); @@ -4641,6 +4760,141 @@ TEST(TableReaderTest, ConditionCacheRefinedChildrenPublishOneSourceRangeEntry) { EXPECT_EQ(handle.get_base_granule(), 0); } +TEST(TableReaderTest, ConditionCacheRefinedChildrenAssembleFragmentsOnceInAnyOrder) { + constexpr int CHILDREN = 128; + constexpr int64_t FILE_SIZE = CHILDREN * 100; + constexpr uint64_t DIGEST = 23; + for (int order = 0; order < 3; ++order) { + ScopedConditionCacheForTest cache; + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + std::vector file_schema { + make_file_column(0, "id", std::make_shared())}; + std::vector projected_columns { + make_table_column(0, "id", std::make_shared())}; + set_name_identifiers(&projected_columns); + auto predicate = prepared_conjunct(&state, table_int32_greater_than_expr(0, 0, 0)); + auto context = std::make_shared(CHILDREN); + segment_v2::ConditionCache::ExternalCacheKey key( + "fragment-input", 17, FILE_SIZE, DIGEST, 0, FILE_SIZE, + segment_v2::ConditionCache::ExternalCacheKey::BASE_GRANULE_AWARE_VERSION); + std::vector*> buffers; + for (int completed = 0; completed < CHILDREN; ++completed) { + // Growing prefixes, growing suffixes, and alternating extremes all need linear + // assembly. Two neighboring children overlap in their boundary granule. + int child = order == 0 ? completed : CHILDREN - completed - 1; + if (order == 2) { + child = completed % 2 == 0 ? completed / 2 : CHILDREN - 1 - completed / 2; + } + auto fake = std::make_shared(); + fake->total_rows = 2 * ConditionCacheContext::GRANULE_SIZE; + fake->condition_cache_base_granule = 10 + 2 * child; + fake->condition_cache_num_granules = 3; + FakeTableReader reader(file_schema, fake); + ASSERT_TRUE(reader.init({ + .projected_columns = projected_columns, + .conjuncts = {predicate}, + .format = FileFormat::PARQUET, + .scan_params = nullptr, + .io_ctx = nullptr, + .runtime_state = &state, + .scanner_profile = nullptr, + .condition_cache_digest = DIGEST, + }) + .ok()); + SplitReadOptions options; + options.current_range.__set_path("fragment-input"); + options.current_range.__set_file_size(FILE_SIZE); + options.current_range.__set_modification_time(17); + options.current_range.__set_start_offset(100 * child); + options.current_range.__set_size(100); + options.condition_cache_digest = DIGEST; + options.condition_cache_source_range = std::pair(0, FILE_SIZE); + options.condition_cache_split_context = context; + ASSERT_TRUE(reader.prepare_split(options).ok()); + Block block = build_table_block(projected_columns); + bool eos = false; + ASSERT_TRUE(reader.get_block(&block, &eos).ok()); + buffers.push_back(fake->condition_cache_ctx->filter_result.get()); + segment_v2::ConditionCacheHandle handle; + EXPECT_EQ(cache.get()->lookup(key, &handle), completed == CHILDREN - 1); + if (completed != CHILDREN - 1) { + ASSERT_EQ(context->fragments.size(), completed + 1); + // No completed bitmap is copied or rewritten while the source is incomplete. + for (size_t index = 0; index < buffers.size(); ++index) { + EXPECT_EQ(context->fragments[index]->filter_result.get(), buffers[index]); + EXPECT_EQ(*context->fragments[index]->filter_result, + std::vector({true, false, false})); + } + } + ASSERT_TRUE(reader.close().ok()); + } + EXPECT_TRUE(context->fragments.empty()); + segment_v2::ConditionCacheHandle handle; + ASSERT_TRUE(cache.get()->lookup(key, &handle)); + EXPECT_EQ(handle.get_base_granule(), 10); + const auto& result = *handle.get_filter_result(); + ASSERT_EQ(result.size(), 2 * CHILDREN + 1); + for (size_t index = 0; index < result.size(); ++index) { + EXPECT_EQ(result[index], index < 2 * CHILDREN && index % 2 == 0); + } + } +} + +TEST(TableReaderTest, ConditionCacheRefinedChildrenRejectMismatchedOrIncompleteFragments) { + for (int failure = 0; failure < 3; ++failure) { + ScopedConditionCacheForTest cache; + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + std::vector schema { + make_file_column(0, "id", std::make_shared())}; + std::vector columns { + make_table_column(0, "id", std::make_shared())}; + set_name_identifiers(&columns); + auto context = std::make_shared(2); + for (int child = 0; child < 2; ++child) { + auto fake = std::make_shared(); + fake->total_rows = ConditionCacheContext::GRANULE_SIZE; + fake->condition_cache_base_granule = child; + fake->condition_cache_num_granules = 1; + fake->eof_with_first_batch = !(child == 1 && failure == 1); + fake->stop_during_read = child == 1 && failure == 2; + fake->io_ctx = std::make_shared(); + FakeTableReader reader(schema, fake); + ASSERT_TRUE( + reader.init({ + .projected_columns = columns, + .conjuncts = {prepared_conjunct( + &state, table_int32_greater_than_expr(0, 0, 0))}, + .format = FileFormat::PARQUET, + .scan_params = nullptr, + .io_ctx = fake->io_ctx, + .runtime_state = &state, + .scanner_profile = nullptr, + .condition_cache_digest = 23, + }) + .ok()); + SplitReadOptions options; + options.current_range.__set_path("incomplete-fragment-input"); + options.condition_cache_digest = child == 1 && failure == 0 ? 29 : 23; + options.condition_cache_source_range = std::pair(0, 200); + options.condition_cache_split_context = context; + ASSERT_TRUE(reader.prepare_split(options).ok()); + Block block = build_table_block(columns); + bool eos = false; + ASSERT_TRUE(reader.get_block(&block, &eos).ok()); + ASSERT_TRUE(reader.close().ok()); + } + EXPECT_EQ(context->remaining_children, 0); + EXPECT_FALSE(context->valid); + for (uint64_t digest : {23, 29}) { + segment_v2::ConditionCache::ExternalCacheKey key( + "incomplete-fragment-input", 0, -1, digest, 0, 200, + segment_v2::ConditionCache::ExternalCacheKey::BASE_GRANULE_AWARE_VERSION); + segment_v2::ConditionCacheHandle handle; + EXPECT_FALSE(cache.get()->lookup(key, &handle)); + } + } +} + TEST(TableReaderTest, ConditionCacheSerialEntryIsReusedByRefinedChild) { ScopedConditionCacheForTest cache; std::vector file_schema; diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java index 74434f2f7ae50b..bc1ac2a97c9cf9 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java @@ -974,7 +974,7 @@ private List planScanInternal( // identical to the streaming path's IcebergStreamingSplitSource so both produce the same ranges. IcebergScanRange range = buildRangeForTask( task, table, scanSchema, formatVersion, partitioned, orderedPartitionKeys, zone, - uriNormalizer, plan.targetSplitSize, coarseSplitSize, rewriteScope, rewritableDeleteSupply, + uriNormalizer, plan.targetSplitSize, plan.coarseSplitSize, rewriteScope, rewritableDeleteSupply, scratch); if (range != null) { ranges.add(range); @@ -3121,10 +3121,12 @@ private static int getFormatVersion(Table table) { private static final class SplitPlan implements Closeable { private final CloseableIterable tasks; private final long targetSplitSize; + private final long coarseSplitSize; - SplitPlan(CloseableIterable tasks, long targetSplitSize) { + SplitPlan(CloseableIterable tasks, long targetSplitSize, long coarseSplitSize) { this.tasks = tasks; this.targetSplitSize = targetSplitSize; + this.coarseSplitSize = coarseSplitSize; } @Override @@ -3164,7 +3166,8 @@ private SplitPlan splitFiles(TableScan scan, ConnectorSession session, long coar if (fileSplitSize > 0) { // This remains the fallback granularity and weight denominator for ineligible files. // Eligible files select the dedicated coarse target for both splitting and weighting. - return new SplitPlan(splitFileTasks(scan.planFiles(), fileSplitSize, coarseSplitSize), fileSplitSize); + return new SplitPlan(splitFileTasks(scan.planFiles(), fileSplitSize, coarseSplitSize), + fileSplitSize, coarseSplitSize); } List fileScanTasks = new ArrayList<>(); try (CloseableIterable planned = scan.planFiles()) { @@ -3176,9 +3179,13 @@ private SplitPlan splitFiles(TableScan scan, ConnectorSession session, long coar + e.getMessage(), e); } long targetSplitSize = determineTargetFileSplitSize(fileScanTasks, session); + // Apply the non-batch count bound after choosing the dedicated target. Otherwise a + // 2 GiB file with max_file_split_num=1 would still produce four 512 MiB FE ranges. + long boundedCoarseSize = coarseSplitSize > 0 + ? determineTargetFileSplitSize(fileScanTasks, session, coarseSplitSize) : coarseSplitSize; return new SplitPlan( - splitFileTasks(CloseableIterable.withNoopClose(fileScanTasks), targetSplitSize, coarseSplitSize), - targetSplitSize); + splitFileTasks(CloseableIterable.withNoopClose(fileScanTasks), targetSplitSize, boundedCoarseSize), + targetSplitSize, boundedCoarseSize); } /** @@ -3232,9 +3239,11 @@ private SplitPlan planFileScanTaskWithManifestCache(TableScan scan, } } long targetSplitSize = determineTargetFileSplitSize(tasks, session); + long boundedCoarseSize = coarseSplitSize > 0 + ? determineTargetFileSplitSize(tasks, session, coarseSplitSize) : coarseSplitSize; return new SplitPlan( - splitFileTasks(CloseableIterable.withNoopClose(tasks), targetSplitSize, coarseSplitSize), - targetSplitSize); + splitFileTasks(CloseableIterable.withNoopClose(tasks), targetSplitSize, boundedCoarseSize), + targetSplitSize, boundedCoarseSize); } /** @@ -3606,6 +3615,11 @@ private boolean isManifestCacheEnabled() { */ private long determineTargetFileSplitSize(List> tasks, ConnectorSession session) { + return determineTargetFileSplitSize(tasks, session, -1); + } + + private long determineTargetFileSplitSize(List> tasks, + ConnectorSession session, long coarseSplitSize) { long maxInitialSplitSize = sessionLong(session, MAX_INITIAL_FILE_SPLIT_SIZE, DEFAULT_MAX_INITIAL_FILE_SPLIT_SIZE); long maxSplitSize = sessionLong(session, MAX_FILE_SPLIT_SIZE, DEFAULT_MAX_FILE_SPLIT_SIZE); @@ -3620,7 +3634,8 @@ private long determineTargetFileSplitSize(List> tas exceedInitialThreshold = true; } } - long result = exceedInitialThreshold ? maxSplitSize : maxInitialSplitSize; + long result = coarseSplitSize > 0 ? coarseSplitSize + : exceedInitialThreshold ? maxSplitSize : maxInitialSplitSize; if (maxFileSplitNum > 0 && total > 0) { long minSplitSizeForMaxNum = (total + maxFileSplitNum - 1) / maxFileSplitNum; result = Math.max(result, minSplitSizeForMaxNum); diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java index 06e65bd868262c..ea662e454849ae 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java @@ -211,6 +211,26 @@ public void coarseSizesOverrideLegacySplitSizeInSynchronousAndStreamingPlans() t } } + @Test + public void coarseSizeRetainsMaxFileSplitNumInSdkAndManifestPlans() { + long mb = 1024L * 1024L; + Table table = createTable("t1", SCHEMA, PartitionSpec.unpartitioned()); + table.newAppend().appendFile(dataFile(table.spec(), "s3://b/db/t1/big.parquet", + 2048 * mb, null, null)).commit(); + ConnectorSession session = new FakeScanSession("UTC", Map.of("max_file_split_num", "1")); + ConnectorScanRequest request = ConnectorScanRequest.builder( + new IcebergTableHandle("db1", "t1"), Collections.emptyList()).coarseSplitSize(512 * mb).build(); + for (IcebergScanPlanProvider provider : Arrays.asList( + new IcebergScanPlanProvider(IcebergCatalogProperties.of(Collections.emptyMap()), opsReturning(table)), + manifestProvider(manifestCacheProps(), table, new IcebergManifestCache()))) { + List ranges = provider.planScan(session, request); + Assertions.assertEquals(1, ranges.size()); + Assertions.assertEquals(0, ranges.get(0).getStart()); + Assertions.assertEquals(2048 * mb, ranges.get(0).getLength()); + Assertions.assertEquals(2048 * mb, ranges.get(0).getTargetSplitSize()); + } + } + @Test public void scanReuseNamespaceUsesConnectorType() { String prefix = new IcebergConnectorProvider().getType() + "."; diff --git a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java index d194c547668f9c..582491ab9b0cbe 100644 --- a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java @@ -966,6 +966,7 @@ && hasTrustworthyLimitAccounting(table) // FIX-NATIVE-SUBSPLIT: target file split size for native ORC/Parquet sub-splitting, computed // lazily ONCE on the first native split (legacy hasDeterminedTargetFileSplitSize parity). long targetSplitSize = -1; + long coarseTargetSplitSize = -1; Set physicalVariantSchemaIds = hasVariantProjection ? physicalVariantSchemaIds(table, paimonHandle, rowType, columns, dataSplits) @@ -1060,10 +1061,15 @@ && hasTrustworthyLimitAccounting(table) long nativeCoarseSize = coarseSplitSizeForRawFile(file.path(), defaultFileFormat, countPushdown ? -1 : coarseSplitSize); if (!countPushdown && nativeCoarseSize <= 0 && targetSplitSize < 0) { - targetSplitSize = resolveTargetSplitSize(session, dataSplits); + targetSplitSize = resolveTargetSplitSize(session, dataSplits, -1); + } + // A small coarse target must not bypass max_file_split_num and expand one + // large native file into an unbounded list of FE ranges. + if (!countPushdown && nativeCoarseSize > 0 && coarseTargetSplitSize < 0) { + coarseTargetSplitSize = resolveTargetSplitSize(session, dataSplits, nativeCoarseSize); } long effectiveSplitSize = countPushdown ? 0L - : nativeCoarseSize > 0 ? nativeCoarseSize : targetSplitSize; + : nativeCoarseSize > 0 ? coarseTargetSplitSize : targetSplitSize; long nativeWeightTarget = nativeCoarseSize > 0 ? nativeCoarseSize : weightDenominator; DeletionFile deletionFile = (optDeletionFiles.isPresent() && i < optDeletionFiles.get().size()) @@ -2060,7 +2066,7 @@ static long determineTargetSplitSize(long fileSplitSize, long maxInitialSplitSiz * once-per-scan computation (summing every {@code supportNativeReader}-eligible RawFile, like * {@code PaimonScanNode.java:552-564}). */ - private long resolveTargetSplitSize(ConnectorSession session, List dataSplits) { + private long resolveTargetSplitSize(ConnectorSession session, List dataSplits, long coarseSplitSize) { long totalNativeFileSize = 0; for (DataSplit dataSplit : dataSplits) { Optional> rawFiles = dataSplit.convertToRawFiles(); @@ -2071,12 +2077,19 @@ private long resolveTargetSplitSize(ConnectorSession session, List da totalNativeFileSize += file.fileSize(); } } + long maxFileSplitNum = sessionLong(session, MAX_FILE_SPLIT_NUM, DEFAULT_MAX_FILE_SPLIT_NUM); + if (coarseSplitSize > 0) { + // The dedicated target replaces file_split_size, but keeps the count-derived bound. + return maxFileSplitNum > 0 && totalNativeFileSize > 0 + ? Math.max(coarseSplitSize, (totalNativeFileSize + maxFileSplitNum - 1L) / maxFileSplitNum) + : coarseSplitSize; + } return determineTargetSplitSize( sessionLong(session, FILE_SPLIT_SIZE, 0L), sessionLong(session, MAX_INITIAL_FILE_SPLIT_SIZE, DEFAULT_MAX_INITIAL_FILE_SPLIT_SIZE), sessionLong(session, MAX_FILE_SPLIT_SIZE, DEFAULT_MAX_FILE_SPLIT_SIZE), sessionLong(session, MAX_INITIAL_FILE_SPLIT_NUM, DEFAULT_MAX_INITIAL_FILE_SPLIT_NUM), - sessionLong(session, MAX_FILE_SPLIT_NUM, DEFAULT_MAX_FILE_SPLIT_NUM), + maxFileSplitNum, totalNativeFileSize); } diff --git a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonScanPlanProviderTest.java b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonScanPlanProviderTest.java index 6fafc1103543aa..a64c888f94d470 100644 --- a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonScanPlanProviderTest.java +++ b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonScanPlanProviderTest.java @@ -4071,6 +4071,13 @@ public void nativeFileIsSubSplitWhenFileSplitSizeForcesIt(@TempDir Path warehous Assertions.assertEquals(fileLength * 2, coarse.get(0).getTargetSplitSize()); Assertions.assertTrue(coarse.get(0).supportsBeSplitRefinement()); + ConnectorSession cappedSession = sessionWithProps(Map.of("max_file_split_num", "1")); + List capped = provider.planScan(cappedSession, + ConnectorScanRequest.builder(handle, noColumns).coarseSplitSize(splitSize).build()); + Assertions.assertEquals(1, capped.size(), "the count bound must survive coarse target selection"); + Assertions.assertEquals(0, capped.get(0).getStart()); + Assertions.assertEquals(fileLength, capped.get(0).getLength()); + // Contrast: with the default (large) split size the small file stays a SINGLE native range. List whole = provider.planScan(sessionWithProps(Collections.emptyMap()), ConnectorScanRequest.builder(handle, noColumns) diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/tvf/source/TVFScanNode.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/tvf/source/TVFScanNode.java index 0d14fabb85b40e..cc832c98c7856f 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/tvf/source/TVFScanNode.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/tvf/source/TVFScanNode.java @@ -179,6 +179,14 @@ protected TColumnCategory classifyColumn(SlotDescriptor slot, List parti return super.classifyColumn(slot, partitionKeys); } + @Override + protected long coarseSplitSizeForBeRefinement() { + // HTTP discovery has no stable mtime/version. BE refuses refinement for that identity, + // so a coarse FE range would lose the parallelism supplied by legacy byte ranges. + return tableValuedFunction.getTFileType() == TFileType.FILE_HTTP + ? -1 : super.coarseSplitSizeForBeRefinement(); + } + private long determineTargetFileSplitSize(List fileStatuses) throws UserException { if (sessionVariable.getFileSplitSize() > 0) { return selectFeSplitSizeForBe(sessionVariable.getFileSplitSize(), getFileFormatType(), @@ -195,8 +203,10 @@ private long determineTargetFileSplitSize(List fileStatuses) } } result = exceedInitialThreshold ? sessionVariable.getMaxSplitSize() : result; - result = applyMaxFileSplitNumLimit(result, totalFileSize); - return selectFeSplitSizeForBe(result, getFileFormatType(), !isTableLevelCountStarPushdown()); + result = selectFeSplitSizeForBe(result, getFileFormatType(), !isTableLevelCountStarPushdown()); + // Coarse sizing must not replace the independent non-batch FE memory bound. For example, + // max_file_split_num=1 still keeps a 2 GiB file whole with the 512 MiB coarse target. + return applyMaxFileSplitNumLimit(result, totalFileSize); } @Override diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/tvf/source/TVFScanNodeTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/tvf/source/TVFScanNodeTest.java index 1dc4042020b2c1..56db999894b5d2 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/tvf/source/TVFScanNodeTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/tvf/source/TVFScanNodeTest.java @@ -29,6 +29,7 @@ import org.apache.doris.spi.Split; import org.apache.doris.tablefunction.ExternalFileTableValuedFunction; import org.apache.doris.thrift.TBrokerFileStatus; +import org.apache.doris.thrift.TFileFormatType; import org.apache.doris.thrift.TFileType; import org.apache.doris.thrift.TPushAggOp; @@ -43,6 +44,55 @@ public class TVFScanNodeTest { private static final long MB = 1024L * 1024L; + @Test + public void testCoarseRangesHonorMaxFileSplitNum() throws Exception { + for (TFileFormatType format : List.of(TFileFormatType.FORMAT_PARQUET, TFileFormatType.FORMAT_ORC)) { + SessionVariable sv = new SessionVariable(); + sv.setMaxFileSplitNum(1); + TBrokerFileStatus file = splittableFile("s3://bucket/large", 2048 * MB); + file.setModificationTime(17); + TVFScanNode node = fileScanNode(sv, TFileType.FILE_S3, format, file); + List splits = node.getSplits(1); + Assertions.assertEquals(1, splits.size()); + Assertions.assertEquals(2048 * MB, splits.get(0).getLength()); + } + } + + @Test + public void testHttpRetainsFineRangesWhileStableStorageUsesCoarseRanges() throws Exception { + for (TFileFormatType format : List.of(TFileFormatType.FORMAT_PARQUET, TFileFormatType.FORMAT_ORC)) { + for (long explicitSize : new long[] {0, 16 * MB}) { + SessionVariable sv = new SessionVariable(); + sv.setFileSplitSize(explicitSize); + TBrokerFileStatus httpFile = splittableFile("http://example.test/data", 256 * MB); + TVFScanNode http = fileScanNode(sv, TFileType.FILE_HTTP, format, httpFile); + Assertions.assertEquals(explicitSize > 0 ? 16 : 8, http.getSplits(1).size()); + for (TFileType type : List.of(TFileType.FILE_S3, TFileType.FILE_HDFS)) { + String path = type == TFileType.FILE_S3 ? "s3://bucket/data" : "hdfs://host/data"; + TBrokerFileStatus stableFile = splittableFile(path, 256 * MB); + stableFile.setModificationTime(17); + TVFScanNode stable = fileScanNode(sv, type, format, stableFile); + Assertions.assertEquals(1, stable.getSplits(1).size()); + } + } + } + } + + private static TVFScanNode fileScanNode(SessionVariable sv, TFileType type, + TFileFormatType format, TBrokerFileStatus file) throws Exception { + TupleDescriptor desc = new TupleDescriptor(new TupleId(0)); + FunctionGenTable table = Mockito.mock(FunctionGenTable.class); + ExternalFileTableValuedFunction tvf = Mockito.mock(ExternalFileTableValuedFunction.class); + Mockito.when(table.getTvf()).thenReturn(tvf); + Mockito.when(tvf.getTFileType()).thenReturn(type); + Mockito.when(tvf.getTFileFormatType()).thenReturn(format); + Mockito.when(tvf.getFileStatuses()).thenReturn(List.of(file)); + desc.setTable(table); + TVFScanNode node = new TVFScanNode(new PlanNodeId(0), desc, false, sv, ScanContext.EMPTY); + setFileSplitter(node, new FileSplitter(32 * MB, 32 * MB, 0)); + return node; + } + @Test public void testDetermineTargetFileSplitSizeUsesCoarseSizeForParquet() throws Exception { SessionVariable sv = new SessionVariable(); From 8980ecf967c30092a9df06f4a3c2fca75343da62 Mon Sep 17 00:00:00 2001 From: morningman Date: Tue, 17 Nov 2026 23:00:48 +0800 Subject: [PATCH 04/10] [fix](orc) Bound physical child stripe metadata traversal ### Background ORC physical children share immutable file metadata and carry exact stripe ids. SARG setup and timestamp stripe readers use that metadata to select stripes and retain file-global row positions. ### What problem does this PR solve? Issue Number: None Related PR: None PR Type: Bugfix Problem Summary: Each ORC child searched the whole file directory to rediscover its stripe ownership. Timestamp children also reopened all preceding stripes to recover their first row. Consume the planned stripe-id interval and cache row prefixes once in the shared file context. ### Details - **Change map:** 108 changed production lines across three files and 211 changed test lines in one file; no generated files or documentation. - **Frequency:** Eligible ORC scans with many physical children, especially tiny split targets and timestamp files requiring stripe-specific reader settings. - **Root cause:** The ORC factory dropped the physical id/id-end fields, and timestamp setup rebuilt file-global row prefixes for every child. - **Why existing tests missed it:** Tests checked ownership and output rows but did not bound directory visits across all children. - **Fix:** Forward both ids, enumerate only the inclusive child interval, and store each stripe's first row alongside its shared metadata. Operation counters exist only in BE_TEST builds. - **Other paths and branches:** Raw FE ranges retain byte-window selection. Coalesced children retain inclusive id-end semantics. Parquet already consumes exact physical ids. Query results, timestamp conversion, FE-BE protocol and on-disk formats do not change. ### Testing - Local ASAN BE build: DISABLE_BUILD_UI=ON ./build.sh --be -j8 passed. - ./run-be-ut.sh --run --filter='NewOrcReaderTest.*' -j8: all 196 tests passed under ASAN on macOS. - Negative controls restored the original algorithms while retaining test instrumentation. The SARG test failed with 64 visits for eight one-stripe children and 24 for coalesced children, versus eight after the fix. The timestamp test failed with 1024 ownership visits and 528 prefix visits for 32 children, versus 32 each after the fix. Both tests also cover coalesced intervals and output correctness. - clang-format 16, check-format.sh, check-build-hygiene.sh, git diff --check and run-clang-tidy.sh passed for the modified files. - No running-cluster regression or elapsed-time benchmark was performed. ### Release note ORC scans with many physical child splits avoid repeated full-directory ownership searches and timestamp row-prefix reconstruction during child initialization. ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Partially verified: local build and ORC unit tests passed; external end-to-end suites were not run. ### AI Disclosure - Tool: Codex - Generated: Code, tests and this description. - Human review: not stated --- be/src/format_v2/orc/orc_reader.cpp | 95 ++++++---- be/src/format_v2/orc/orc_reader.h | 10 +- be/src/format_v2/table_reader.cpp | 3 +- be/test/format_v2/orc/orc_reader_test.cpp | 211 +++++++++++++++++++++- 4 files changed, 277 insertions(+), 42 deletions(-) diff --git a/be/src/format_v2/orc/orc_reader.cpp b/be/src/format_v2/orc/orc_reader.cpp index f6e06212ef1b92..4eb90290816260 100644 --- a/be/src/format_v2/orc/orc_reader.cpp +++ b/be/src/format_v2/orc/orc_reader.cpp @@ -1138,12 +1138,15 @@ OrcReader::OrcReader(std::shared_ptr& system_propertie std::shared_ptr io_ctx, RuntimeProfile* profile, std::optional global_rowid_context, bool enable_mapping_timestamp_tz, bool enable_mapping_varbinary, - std::shared_ptr file_context) + std::shared_ptr file_context, int64_t format_split_id, + int64_t format_split_id_end) : FileReader(system_properties, file_description, io_ctx, profile), _global_rowid_context(std::move(global_rowid_context)), _enable_mapping_timestamp_tz(enable_mapping_timestamp_tz), _enable_mapping_varbinary(enable_mapping_varbinary), - _file_context(std::move(file_context)) {} + _file_context(std::move(file_context)), + _format_split_id(format_split_id), + _format_split_id_end(format_split_id_end) {} OrcReader::~OrcReader() = default; @@ -1397,12 +1400,15 @@ Status OrcReader::init(RuntimeState* state) { } const auto stripe_count = reader.getNumberOfStripes(); context->stripes.reserve(stripe_count); + uint64_t first_row = 0; for (uint64_t stripe_id = 0; stripe_id < stripe_count; ++stripe_id) { const auto stripe = reader.getStripe(stripe_id); DORIS_CHECK(stripe != nullptr); context->stripes.push_back({.offset = stripe->getOffset(), .length = stripe->getLength(), - .rows = stripe->getNumberOfRows()}); + .rows = stripe->getNumberOfRows(), + .first_row = first_row}); + first_row += context->stripes.back().rows; } } catch (const std::exception& e) { return Status::InternalError("Failed to cache ORC file context for {}: {}", @@ -2021,13 +2027,29 @@ Status OrcReader::_collect_split_stripes(std::vector* stripe_indices) stripe_indices->clear(); DORIS_CHECK(_state->file_context != nullptr); const auto stripe_count = _state->file_context->stripes.size(); - stripe_indices->reserve(stripe_count); - + uint64_t first_stripe = 0; + uint64_t stripe_end = stripe_count; uint64_t split_start = 0; uint64_t split_end = std::numeric_limits::max(); - _split_byte_window(&split_start, &split_end); + // A generated child already owns an exact inclusive stripe-id interval. For example, child + // [17, 17] of a 64-stripe file must visit/reserve one entry, not rediscover it among all 64. + if (_format_split_id >= 0) { + const auto last_stripe = + _format_split_id_end >= 0 ? _format_split_id_end : _format_split_id; + DORIS_CHECK(last_stripe >= _format_split_id); + DORIS_CHECK(static_cast(last_stripe) < stripe_count); + first_stripe = static_cast(_format_split_id); + stripe_end = static_cast(last_stripe) + 1; + } else { + // Original FE ranges have no physical ids and still resolve ownership by byte offsets. + _split_byte_window(&split_start, &split_end); + } + stripe_indices->reserve(stripe_end - first_stripe); - for (uint64_t stripe_index = 0; stripe_index < stripe_count; ++stripe_index) { + for (uint64_t stripe_index = first_stripe; stripe_index < stripe_end; ++stripe_index) { +#ifdef BE_TEST + ++_stripe_ownership_visits; +#endif const auto stripe_offset = _state->file_context->stripes[stripe_index].offset; if (stripe_offset >= split_start && stripe_offset < split_end) { stripe_indices->push_back(stripe_index); @@ -2247,38 +2269,39 @@ Status OrcReader::_prepare_timestamp_stripe_ranges() { return Status::OK(); } try { - uint64_t first_row = 0; - size_t selected = 0; std::string previous_timezone; - for (uint64_t index = 0; index <= stripes.back(); ++index) { + // Row prefixes belong to the shared immutable directory. A child starting at stripe 17 + // must not reopen stripes 0..16 merely to recover its file-global row position. + for (const auto index : stripes) { +#ifdef BE_TEST + ++_timestamp_stripe_visits; +#endif const auto stripe = _state->reader->getStripe(index); - const auto end_row = first_row + stripe->getNumberOfRows(); - if (index == stripes[selected]) { - const auto& name = stripe->getWriterTimezone(); - const std::string timezone = name.empty() ? "GMT" : name; - auto& ranges = _state->selected_stripe_ranges; - // Each SDK RowReader scans the entire stripe directory. Reuse it across - // adjacent stripes with the same timezone to avoid quadratic setup work. - if (!ranges.empty() && ranges.back().last_stripe == index && - previous_timezone == timezone) { - auto& range = ranges.back(); - range.last_stripe = index + 1; - range.length = stripe->getOffset() + stripe->getLength() - range.offset; - range.end_row = end_row; - } else { - ranges.push_back({ - .first_stripe = index, - .last_stripe = index + 1, - .offset = stripe->getOffset(), - .length = stripe->getLength(), - .first_row = first_row, - .end_row = end_row, - }); - } - previous_timezone = timezone; - ++selected; + const auto& metadata = _state->file_context->stripes[index]; + const auto first_row = metadata.first_row; + const auto end_row = first_row + metadata.rows; + const auto& name = stripe->getWriterTimezone(); + const std::string timezone = name.empty() ? "GMT" : name; + auto& ranges = _state->selected_stripe_ranges; + // Each SDK RowReader scans the entire stripe directory. Reuse it across + // adjacent stripes with the same timezone to avoid quadratic setup work. + if (!ranges.empty() && ranges.back().last_stripe == index && + previous_timezone == timezone) { + auto& range = ranges.back(); + range.last_stripe = index + 1; + range.length = stripe->getOffset() + stripe->getLength() - range.offset; + range.end_row = end_row; + } else { + ranges.push_back({ + .first_stripe = index, + .last_stripe = index + 1, + .offset = stripe->getOffset(), + .length = stripe->getLength(), + .first_row = first_row, + .end_row = end_row, + }); } - first_row = end_row; + previous_timezone = timezone; } } catch (const std::exception& e) { return Status::InternalError("Failed to prepare ORC timestamp stripe ranges: {}", e.what()); diff --git a/be/src/format_v2/orc/orc_reader.h b/be/src/format_v2/orc/orc_reader.h index 381e95a64a3ad2..5f5bcd06acd435 100644 --- a/be/src/format_v2/orc/orc_reader.h +++ b/be/src/format_v2/orc/orc_reader.h @@ -42,6 +42,7 @@ struct OrcStripeMetadata { uint64_t offset = 0; uint64_t length = 0; uint64_t rows = 0; + uint64_t first_row = 0; }; // ORC readers keep independent row/stripe execution state. Only the immutable serialized tail and @@ -66,7 +67,8 @@ class OrcReader final : public format::FileReader { std::shared_ptr io_ctx, RuntimeProfile* profile, std::optional global_rowid_context = std::nullopt, bool enable_mapping_timestamp_tz = false, bool enable_mapping_varbinary = false, - std::shared_ptr file_context = nullptr); + std::shared_ptr file_context = nullptr, + int64_t format_split_id = -1, int64_t format_split_id_end = -1); ~OrcReader() override; static format::ColumnDefinition row_position_column_definition(); @@ -225,6 +227,12 @@ class OrcReader final : public format::FileReader { bool _enable_mapping_varbinary = false; size_t _batch_size = 0; std::shared_ptr _file_context; + int64_t _format_split_id = -1; + int64_t _format_split_id_end = -1; +#ifdef BE_TEST + mutable size_t _stripe_ownership_visits = 0; + size_t _timestamp_stripe_visits = 0; +#endif }; } // namespace doris::format::orc diff --git a/be/src/format_v2/table_reader.cpp b/be/src/format_v2/table_reader.cpp index eaf2196957b32c..6e94fb17658fb2 100644 --- a/be/src/format_v2/table_reader.cpp +++ b/be/src/format_v2/table_reader.cpp @@ -1551,7 +1551,8 @@ Status TableReader::create_file_reader(std::unique_ptr* reader) { *reader = std::make_unique( _system_properties, _current_task->data_file, _io_ctx, _scanner_profile, _global_rowid_context, enable_mapping_timestamp_tz, enable_mapping_varbinary, - _current_task->file_context); + _current_task->file_context, _current_task->format_split_id, + _current_task->format_split_id_end); return Status::OK(); } if (_format == FileFormat::CSV) { diff --git a/be/test/format_v2/orc/orc_reader_test.cpp b/be/test/format_v2/orc/orc_reader_test.cpp index 50270de1f86edc..50dc3a7c65f409 100644 --- a/be/test/format_v2/orc/orc_reader_test.cpp +++ b/be/test/format_v2/orc/orc_reader_test.cpp @@ -4708,7 +4708,8 @@ class NewOrcReaderTest : public testing::Test { std::unique_ptr create_reader_with_range( const std::string& file_path, int64_t range_start_offset, int64_t range_size, RuntimeProfile* profile = nullptr, int64_t mtime = 0, - std::shared_ptr file_context = nullptr) const { + std::shared_ptr file_context = nullptr, int64_t format_split_id = -1, + int64_t format_split_id_end = -1) const { auto system_properties = std::make_shared(); system_properties->system_type = TFileType::FILE_LOCAL; auto file_description = std::make_unique(); @@ -4717,9 +4718,9 @@ class NewOrcReaderTest : public testing::Test { file_description->range_start_offset = range_start_offset; file_description->range_size = range_size; file_description->mtime = mtime; - return std::make_unique(system_properties, file_description, - nullptr, profile, std::nullopt, false, - false, std::move(file_context)); + return std::make_unique( + system_properties, file_description, nullptr, profile, std::nullopt, false, false, + std::move(file_context), format_split_id, format_split_id_end); } Status scan_direct_in_filter(const std::string& file_path, int32_t in_list_size, @@ -5104,6 +5105,118 @@ TEST_F(NewOrcReaderTest, PhysicalStripeChildrenReuseOnlyParentTailAndReadExactRo EXPECT_EQ(rows_read, layout[0].rows); } +TEST_F(NewOrcReaderTest, PhysicalStripeChildrenBoundSargDirectoryVisits) { + const auto path = (_test_dir / "bounded_stripe_directory.orc").string(); + constexpr size_t STRIPES = 8; + constexpr int64_t ROWS_PER_STRIPE = 200; + std::vector first_values; + for (size_t i = 0; i < STRIPES; ++i) { + first_values.push_back(1 + i * ROWS_PER_STRIPE); + } + write_multi_stripe_orc_int_file(path, first_values); + const auto layout = get_orc_stripe_layout(path); + ASSERT_EQ(layout.size(), STRIPES); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + format::ColumnDefinition column; + column.identifier = Field::create_field("id"); + column.name = "id"; + column.type = make_nullable(std::make_shared()); + const auto init_reader = [&](format::TableReader* reader) { + const auto int_type = std::make_shared(); + const auto bool_type = make_nullable(std::make_shared()); + TFunctionName name; + name.__set_function_name("lt"); + TFunction function; + function.__set_name(name); + function.__set_binary_type(TFunctionBinaryType::BUILTIN); + function.__set_arg_types({column.type->to_thrift(), int_type->to_thrift()}); + function.__set_ret_type(bool_type->to_thrift()); + function.__set_has_var_args(false); + TExprNode node; + node.__set_node_type(TExprNodeType::BINARY_PRED); + node.__set_opcode(TExprOpcode::LT); + node.__set_type(bool_type->to_thrift()); + node.__set_fn(function); + node.__set_num_children(2); + node.__set_is_nullable(true); + auto predicate = VectorizedFnCall::create_shared(node); + predicate->add_child(VSlotRef::create_shared(0, 0, 0, column.type, "id")); + predicate->add_child( + VLiteral::create_shared(int_type, Field::create_field(1000000))); + return reader->init({ + .projected_columns = {column}, + .conjuncts = {VExprContext::create_shared(std::move(predicate))}, + .format = format::FileFormat::ORC, + .scan_params = nullptr, + .io_ctx = nullptr, + .runtime_state = &state, + .scanner_profile = nullptr, + }); + }; + // Exercise both one-stripe children and the inclusive id-end of coalesced children through + // TableReader's real factory, so dropping either id argument reintroduces full-file visits. + for (const int64_t target : {int64_t {1}, cast_set(layout[0].length * 4)}) { + format::TableReader source_reader; + ASSERT_TRUE(init_reader(&source_reader).ok()); + format::SplitReadOptions options; + options.current_split_format = format::FileFormat::ORC; + options.current_range.__set_path(path); + options.current_range.__set_file_size(std::filesystem::file_size(path)); + options.current_range.__set_start_offset(0); + options.current_range.__set_size(std::filesystem::file_size(path)); + options.current_range.__set_modification_time(17); + options.current_range.__set_target_split_size(target); + ASSERT_TRUE(source_reader.prepare_split(options).ok()); + FileScanSplit source; + source.range = options.current_range; + source.is_source_split = true; + std::vector children; + bool was_split = false; + const auto split_status = + source_reader.build_physical_splits(source, &children, &was_split); + ASSERT_TRUE(split_status.ok()) << split_status; + ASSERT_TRUE(was_split); + ASSERT_GT(children.size(), 1); + if (target == 1) { + ASSERT_EQ(children.size(), STRIPES); + } else { + ASSERT_LT(children.size(), STRIPES); + } + size_t directory_visits = 0; + size_t total_rows = 0; + for (const auto& child : children) { + format::TableReader reader; + ASSERT_TRUE(init_reader(&reader).ok()); + options.current_range = child.materialize_range(); + options.file_context = child.file_context; + options.format_split_id = child.format_split_id; + options.format_split_id_end = child.format_split_id_end; + ASSERT_TRUE(reader.prepare_split(options).ok()); + bool eof = false; + ASSERT_TRUE(reader.create_next_reader(&eof).ok()); + ASSERT_FALSE(eof); + const auto* orc_reader = + dynamic_cast(reader._data_reader.reader.get()); + ASSERT_NE(orc_reader, nullptr); + directory_visits += orc_reader->_stripe_ownership_visits; + while (!eof) { + auto block = build_file_block({column}); + ASSERT_TRUE(reader.get_block(&block, &eof).ok()); + const auto& ids = assert_cast( + assert_cast(*block.get_by_position(0).column) + .get_nested_column()); + for (size_t row = 0; row < block.rows(); ++row) { + EXPECT_EQ(ids.get_element(row), ++total_rows); + } + } + ASSERT_TRUE(reader.close().ok()); + } + EXPECT_EQ(total_rows, STRIPES * ROWS_PER_STRIPE); + EXPECT_EQ(directory_visits, STRIPES); + ASSERT_TRUE(source_reader.close().ok()); + } +} + TEST_F(NewOrcReaderTest, OpenAcceptsDorisOffsetTimezone) { auto reader = create_reader(); RuntimeState state {TQueryOptions(), TQueryGlobals()}; @@ -11431,6 +11544,96 @@ TEST_F(NewOrcReaderTest, AdjacentNanosStripesReuseOneRowReader) { EXPECT_EQ(profile.get_counter("OrcRowReadersCreated")->value(), 1); } +TEST_F(NewOrcReaderTest, PhysicalTimestampChildrenUseSharedRowPrefixes) { + const auto path = (_test_dir / "timestamp_child_prefixes.orc").string(); + auto type = ::orc::createStructType(); + auto ns = ::orc::createPrimitiveType(::orc::TIMESTAMP); + ns->setAttribute("iceberg.timestamp-unit", "NANOS"); + type->addStructField("ns", std::move(ns)); + MemoryOutputStream stream(1024 * 1024); + ::orc::WriterOptions options; + options.setTimezoneName("UTC"); + options.setStripeSize(0); + auto writer = ::orc::createWriter(*type, &stream, options); + constexpr size_t STRIPES = 32; + size_t file_rows = 0; + for (size_t i = 0; i < STRIPES; ++i) { + const size_t rows = i % 3 + 1; + auto batch = writer->createRowBatch(rows); + auto& root = dynamic_cast<::orc::StructVectorBatch&>(*batch); + root.numElements = rows; + auto& ts = dynamic_cast<::orc::TimestampVectorBatch&>(*root.fields[0]); + ts.numElements = rows; + for (size_t row = 0; row < rows; ++row) { + ts.data[row] = 0; + ts.nanoseconds[row] = cast_set(file_rows++); + } + writer->add(*batch); + } + writer->close(); + { + std::ofstream out(path, std::ios::binary); + out.write(stream.getData(), static_cast(stream.getLength())); + } + const auto layout = get_orc_stripe_layout(path); + ASSERT_EQ(layout.size(), STRIPES); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + auto parent = create_reader_with_range(path, 0, -1, nullptr, 17); + ASSERT_TRUE(parent->init(&state).ok()); + ASSERT_TRUE(parent->open(std::make_shared()).ok()); + std::vector splits; + bool was_split = false; + ASSERT_TRUE(parent->build_physical_splits(&splits, &was_split).ok()); + ASSERT_TRUE(was_split); + ASSERT_EQ(splits.size(), STRIPES); + const auto context = splits[0].file_context; + ASSERT_TRUE(parent->close().ok()); + for (const size_t group_size : {size_t {1}, size_t {4}}) { + size_t directory_visits = 0; + size_t timestamp_visits = 0; + size_t total_rows = 0; + for (size_t first = 0; first < STRIPES; first += group_size) { + const auto last = first + group_size - 1; + const auto offset = layout[first].offset; + const auto size = layout[last].offset + layout[last].length - offset; + auto reader = create_reader_with_range( + path, cast_set(offset), cast_set(size), nullptr, 17, context, + cast_set(first), cast_set(last)); + ASSERT_TRUE(reader->init(&state).ok()); + std::vector schema; + ASSERT_TRUE(reader->get_schema(&schema).ok()); + schema.push_back(format::orc::OrcReader::row_position_column_definition()); + auto request = std::make_shared(); + request->non_predicate_columns = {field_projection(0), + field_projection(format::ROW_POSITION_COLUMN_ID)}; + request->local_positions = { + {format::LocalColumnId(0), format::LocalIndex(0)}, + {format::LocalColumnId(format::ROW_POSITION_COLUMN_ID), format::LocalIndex(1)}}; + ASSERT_TRUE(reader->open(request).ok()); + directory_visits += reader->_stripe_ownership_visits; + timestamp_visits += reader->_timestamp_stripe_visits; + bool eof = false; + while (!eof) { + auto block = build_file_block(schema); + size_t rows = 0; + ASSERT_TRUE(reader->get_block(&block, &rows, &eof).ok()); + const auto& values = assert_cast( + assert_cast(*block.get_by_position(0).column) + .get_nested_column()); + const auto& positions = int64_data_column(*block.get_by_position(1).column); + for (size_t row = 0; row < rows; ++row, ++total_rows) { + EXPECT_EQ(values.get_element(row).epoch_nanos(), total_rows); + EXPECT_EQ(positions.get_element(row), total_rows); + } + } + ASSERT_TRUE(reader->close().ok()); + } + EXPECT_EQ(total_rows, file_rows); + EXPECT_EQ(directory_visits, STRIPES); + EXPECT_EQ(timestamp_visits, STRIPES); + } +} + TEST_F(NewOrcReaderTest, JavaNanosSiblingPreservesHistoricalCivilTime) { const auto file = find_repo_file("be/test/format_v2/orc/test_data/java_timestamp_nanos.orc"); const std::array plain {"1800-01-01 00:00:00.123457", From a33ac34c447d8e57354ea35414280d399abe40d2 Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 18 Nov 2026 06:52:52 +0800 Subject: [PATCH 05/10] [fix](orc) Reuse shared statistics in physical stripe readers ### Background Physical ORC children use independent readers for their owned stripe intervals. The serialized file tail shares layout metadata, but omits the stripe-statistics Metadata protobuf used by SARG pruning. ### What problem does this PR solve? Issue Number: None Related PR: apache/doris-thirdparty#435 PR Type: Bugfix Problem Summary: Every physical child repeats full-file Metadata parsing and stripe selection. Share the SDK's lazy immutable statistics handle and evaluate only each reader's owned stripe interval while retaining reader-local predicates and row-group pruning. ### Details - **Frequency:** Always for multi-child ORC scans with a supported SARG; reproduced with eight selected stripes. - **Root cause:** Each SDK reader loaded its own Metadata and getNeedReadStripes evaluated every stripe regardless of ownership. - **Why existing tests missed it:** The physical-child test bounded Doris directory visits but did not count SDK parsing or evaluation; external regression tests compare results only. - **Fix:** Carry the SDK statistics handle in the existing file context and use bounded stripe selection. Keep streams, metrics and predicate appliers private to each reader. The handle remains lazy when planning has no SARG. - **Other paths and branches:** Individual and coalesced children, stronger child predicates, default SDK options and standalone full-file calls are covered. No FE, wire or file-format changes; release-branch behavior is not audited in this followup. - **Behavior change:** Add OrcStripeStatisticsParses and OrcStripeStatisticsEvaluations profile counters. The latter measures explicit stripe-selection work, excluding normal RowReader PPD. ### Testing All 239 selected native ORC reader/predicate tests passed under ASAN, including all 196 Doris ORC reader tests. Coverage includes concurrent cold/warm SDK readers, independent predicates, row-group filtering, absolute positions and read/parse failure followed by retry through another reader after parent destruction. An earlier negative control restoring unshared/full-file SDK calls failed the real-factory test with nine parses and 72 evaluations; the fixed path records one parse and 16 evaluations. These are operation counts, not elapsed-time measurements. ASAN BE build, clang-format16, build hygiene, changed-line clang-tidy and git diff --check passed. Pin the merged SDK #435 commit 4ec25901b1266235f56ebe5c948277d6f3c2c616, whose complete source tree matches the tested SDK candidate. External end-to-end suites were not run. The affected pthread_once runtime combination is unavailable locally; its reported hang was not reproduced here. ### Release note None ### Check List (For Author) - Behavior changed: Yes (profile counters, see Details) - Does this need documentation: No additional user-facing setting; existing two-level splitting documentation plan remains in the PR. - Pre-submission status: Partially verified: external end-to-end suites and the affected pthread_once runtime combination were not exercised locally. ### AI Disclosure - Tool: Codex - Generated: Implementation, tests, verification and this description. - Human review: not stated - Prompts: Fix F-01 from the review of PR #70664 and update that PR. --- be/src/format_v2/orc/orc_reader.cpp | 20 ++- be/src/format_v2/orc/orc_reader.h | 6 +- be/test/format_v2/orc/orc_reader_test.cpp | 141 +++++++++++++--------- contrib/apache-orc | 2 +- 4 files changed, 105 insertions(+), 64 deletions(-) diff --git a/be/src/format_v2/orc/orc_reader.cpp b/be/src/format_v2/orc/orc_reader.cpp index 4eb90290816260..5c735cf5bb7a5b 100644 --- a/be/src/format_v2/orc/orc_reader.cpp +++ b/be/src/format_v2/orc/orc_reader.cpp @@ -1184,6 +1184,10 @@ void OrcReader::_init_profile() { _profile, "OrcFileFooterReadCalls", TUnit::UNIT, orc_profile, 1); _orc_profile.file_footer_cache_hits = ADD_CHILD_COUNTER_WITH_LEVEL( _profile, "OrcFileFooterHitCache", TUnit::UNIT, orc_profile, 1); + _orc_profile.stripe_statistics_parses = ADD_CHILD_COUNTER_WITH_LEVEL( + _profile, "OrcStripeStatisticsParses", TUnit::UNIT, orc_profile, 1); + _orc_profile.stripe_statistics_evaluations = ADD_CHILD_COUNTER_WITH_LEVEL( + _profile, "OrcStripeStatisticsEvaluations", TUnit::UNIT, orc_profile, 1); _orc_profile.reader_call = ADD_CHILD_COUNTER_WITH_LEVEL(_profile, "ReaderCall", TUnit::UNIT, orc_profile, 1); _orc_profile.reader_inclusive_latency_us = ADD_CHILD_COUNTER_WITH_LEVEL( @@ -1236,6 +1240,10 @@ void OrcReader::_collect_profile() const { const auto& reader_metrics = _state->reader_metrics; const uint64_t read_row_count = orc_read_row_count(reader_metrics); if (_profile != nullptr) { + COUNTER_UPDATE(_orc_profile.stripe_statistics_parses, + orc_metric_value(reader_metrics.MetadataParseCount)); + COUNTER_UPDATE(_orc_profile.stripe_statistics_evaluations, + orc_metric_value(reader_metrics.StripeStatisticsEvaluationCount)); COUNTER_UPDATE(_orc_profile.reader_call, orc_metric_value(reader_metrics.ReaderCall)); COUNTER_UPDATE(_orc_profile.reader_inclusive_latency_us, orc_metric_value(reader_metrics.ReaderInclusiveLatencyUs)); @@ -1336,6 +1344,9 @@ Status OrcReader::init(RuntimeState* state) { options.setReaderMetrics(&_state->reader_metrics); if (serialized_file_tail != nullptr) { options.setSerializedFileTail(*serialized_file_tail); + // The SDK handle shares only lazily parsed immutable statistics. Each child keeps + // its own stream, metrics and SARG, including filters received after planning. + options.setSharedStripeStatistics(_state->file_context->stripe_statistics); } else if (cache_hit) { options.setSerializedFileTail(*cache_handle.data()); } @@ -1392,6 +1403,7 @@ Status OrcReader::init(RuntimeState* state) { auto context = std::make_shared(); context->file_identity = file_identity; context->has_stable_identity = has_stable_identity; + context->stripe_statistics = reader.getSharedStripeStatistics(); try { if (has_stable_identity) { // Mutable files cannot publish children, so avoid serializing a tail that no other @@ -2157,7 +2169,11 @@ Status OrcReader::_select_stripe_ranges_by_statistics() { std::vector sarg_needed_stripes; try { - sarg_needed_stripes = _state->reader->getNeedReadStripes(_state->row_reader_options); + // Byte-window parents and exact-id children both evaluate only their owned interval. + // Eight one-stripe children must not each re-evaluate all eight file stripes. + sarg_needed_stripes = _state->reader->getNeedReadStripes( + _state->row_reader_options, split_stripes.front(), + split_stripes.back() - split_stripes.front() + 1); } catch (const Exception& e) { if (is_orc_stop(_io_ctx.get(), e)) { return Status::EndOfFile("stop"); @@ -2181,7 +2197,7 @@ Status OrcReader::_select_stripe_ranges_by_statistics() { int64_t filtered_bytes = 0; for (const auto stripe_index : split_stripes) { bool drop = false; - if (stripe_index < sarg_needed_stripes.size() && sarg_needed_stripes[stripe_index] == 0) { + if (sarg_needed_stripes[stripe_index - split_stripes.front()] == 0) { drop = true; } if (!drop) { diff --git a/be/src/format_v2/orc/orc_reader.h b/be/src/format_v2/orc/orc_reader.h index 5f5bcd06acd435..a9020d076883b5 100644 --- a/be/src/format_v2/orc/orc_reader.h +++ b/be/src/format_v2/orc/orc_reader.h @@ -46,11 +46,13 @@ struct OrcStripeMetadata { }; // ORC readers keep independent row/stripe execution state. Only the immutable serialized tail and -// stripe layout cross physical children, so concurrent scanners never share an ORC RowReader. +// stripe layout and lazy immutable stripe statistics cross physical children. Streams and +// SARG appliers remain reader-local, so concurrent scanners never share an ORC RowReader. struct OrcSharedFileContext final : public FileContext { std::string file_identity; bool has_stable_identity = false; std::string serialized_file_tail; + std::shared_ptr<::orc::SharedStripeStatistics> stripe_statistics; std::vector stripes; }; @@ -104,6 +106,8 @@ class OrcReader final : public format::FileReader { RuntimeProfile::Counter* dictionary_values_evaluated = nullptr; RuntimeProfile::Counter* file_footer_read_calls = nullptr; RuntimeProfile::Counter* file_footer_cache_hits = nullptr; + RuntimeProfile::Counter* stripe_statistics_parses = nullptr; + RuntimeProfile::Counter* stripe_statistics_evaluations = nullptr; RuntimeProfile::Counter* total_time = nullptr; RuntimeProfile::Counter* reader_call = nullptr; // ReaderCall RuntimeProfile::Counter* reader_inclusive_latency_us = nullptr; // ReaderInclusiveLatencyUs diff --git a/be/test/format_v2/orc/orc_reader_test.cpp b/be/test/format_v2/orc/orc_reader_test.cpp index 50dc3a7c65f409..70b718b708e561 100644 --- a/be/test/format_v2/orc/orc_reader_test.cpp +++ b/be/test/format_v2/orc/orc_reader_test.cpp @@ -5121,7 +5121,8 @@ TEST_F(NewOrcReaderTest, PhysicalStripeChildrenBoundSargDirectoryVisits) { column.identifier = Field::create_field("id"); column.name = "id"; column.type = make_nullable(std::make_shared()); - const auto init_reader = [&](format::TableReader* reader) { + const auto init_reader = [&](format::TableReader* reader, RuntimeProfile* profile, + int32_t cutoff = 1000000, bool with_sarg = true) { const auto int_type = std::make_shared(); const auto bool_type = make_nullable(std::make_shared()); TFunctionName name; @@ -5142,78 +5143,98 @@ TEST_F(NewOrcReaderTest, PhysicalStripeChildrenBoundSargDirectoryVisits) { auto predicate = VectorizedFnCall::create_shared(node); predicate->add_child(VSlotRef::create_shared(0, 0, 0, column.type, "id")); predicate->add_child( - VLiteral::create_shared(int_type, Field::create_field(1000000))); + VLiteral::create_shared(int_type, Field::create_field(cutoff))); + std::vector conjuncts; + if (with_sarg) { + conjuncts.push_back(VExprContext::create_shared(std::move(predicate))); + } return reader->init({ .projected_columns = {column}, - .conjuncts = {VExprContext::create_shared(std::move(predicate))}, + .conjuncts = std::move(conjuncts), .format = format::FileFormat::ORC, .scan_params = nullptr, .io_ctx = nullptr, .runtime_state = &state, - .scanner_profile = nullptr, + .scanner_profile = profile, }); }; // Exercise both one-stripe children and the inclusive id-end of coalesced children through // TableReader's real factory, so dropping either id argument reintroduces full-file visits. - for (const int64_t target : {int64_t {1}, cast_set(layout[0].length * 4)}) { - format::TableReader source_reader; - ASSERT_TRUE(init_reader(&source_reader).ok()); - format::SplitReadOptions options; - options.current_split_format = format::FileFormat::ORC; - options.current_range.__set_path(path); - options.current_range.__set_file_size(std::filesystem::file_size(path)); - options.current_range.__set_start_offset(0); - options.current_range.__set_size(std::filesystem::file_size(path)); - options.current_range.__set_modification_time(17); - options.current_range.__set_target_split_size(target); - ASSERT_TRUE(source_reader.prepare_split(options).ok()); - FileScanSplit source; - source.range = options.current_range; - source.is_source_split = true; - std::vector children; - bool was_split = false; - const auto split_status = - source_reader.build_physical_splits(source, &children, &was_split); - ASSERT_TRUE(split_status.ok()) << split_status; - ASSERT_TRUE(was_split); - ASSERT_GT(children.size(), 1); - if (target == 1) { - ASSERT_EQ(children.size(), STRIPES); - } else { - ASSERT_LT(children.size(), STRIPES); - } - size_t directory_visits = 0; - size_t total_rows = 0; - for (const auto& child : children) { - format::TableReader reader; - ASSERT_TRUE(init_reader(&reader).ok()); - options.current_range = child.materialize_range(); - options.file_context = child.file_context; - options.format_split_id = child.format_split_id; - options.format_split_id_end = child.format_split_id_end; - ASSERT_TRUE(reader.prepare_split(options).ok()); - bool eof = false; - ASSERT_TRUE(reader.create_next_reader(&eof).ok()); - ASSERT_FALSE(eof); - const auto* orc_reader = - dynamic_cast(reader._data_reader.reader.get()); - ASSERT_NE(orc_reader, nullptr); - directory_visits += orc_reader->_stripe_ownership_visits; - while (!eof) { - auto block = build_file_block({column}); - ASSERT_TRUE(reader.get_block(&block, &eof).ok()); - const auto& ids = assert_cast( - assert_cast(*block.get_by_position(0).column) - .get_nested_column()); - for (size_t row = 0; row < block.rows(); ++row) { - EXPECT_EQ(ids.get_element(row), ++total_rows); + // A child may receive a stronger predicate after planning (for example, a late RF). + // Share file statistics, never the parent's predicate applier or selection result. + // Planning without a SARG must leave the handle lazy for filters received by children. + for (const bool parent_sarg : {false, true}) { + for (const int32_t child_cutoff : {1000000, 700}) { + for (const int64_t target : {int64_t {1}, cast_set(layout[0].length * 4)}) { + SCOPED_TRACE(fmt::format("parent_sarg={}, child_cutoff={}, target={}", parent_sarg, + child_cutoff, target)); + RuntimeProfile profile("shared_stripe_statistics"); + format::TableReader source_reader; + ASSERT_TRUE(init_reader(&source_reader, &profile, 1000000, parent_sarg).ok()); + format::SplitReadOptions options; + options.current_split_format = format::FileFormat::ORC; + options.current_range.__set_path(path); + options.current_range.__set_file_size(std::filesystem::file_size(path)); + options.current_range.__set_start_offset(0); + options.current_range.__set_size(std::filesystem::file_size(path)); + options.current_range.__set_modification_time(17); + options.current_range.__set_target_split_size(target); + ASSERT_TRUE(source_reader.prepare_split(options).ok()); + FileScanSplit source; + source.range = options.current_range; + source.is_source_split = true; + std::vector children; + bool was_split = false; + const auto split_status = + source_reader.build_physical_splits(source, &children, &was_split); + ASSERT_TRUE(split_status.ok()) << split_status; + ASSERT_TRUE(was_split); + ASSERT_GT(children.size(), 1); + if (target == 1) { + ASSERT_EQ(children.size(), STRIPES); + } else { + ASSERT_LT(children.size(), STRIPES); + } + size_t directory_visits = 0; + size_t total_rows = 0; + for (const auto& child : children) { + format::TableReader reader; + ASSERT_TRUE(init_reader(&reader, &profile, child_cutoff).ok()); + options.current_range = child.materialize_range(); + options.file_context = child.file_context; + options.format_split_id = child.format_split_id; + options.format_split_id_end = child.format_split_id_end; + ASSERT_TRUE(reader.prepare_split(options).ok()); + bool eof = false; + ASSERT_TRUE(reader.create_next_reader(&eof).ok()); + ASSERT_FALSE(eof); + const auto* orc_reader = dynamic_cast( + reader._data_reader.reader.get()); + ASSERT_NE(orc_reader, nullptr); + directory_visits += orc_reader->_stripe_ownership_visits; + while (!eof) { + auto block = build_file_block({column}); + ASSERT_TRUE(reader.get_block(&block, &eof).ok()); + const auto& ids = assert_cast( + assert_cast(*block.get_by_position(0).column) + .get_nested_column()); + for (size_t row = 0; row < block.rows(); ++row) { + EXPECT_EQ(ids.get_element(row), ++total_rows); + } + } + ASSERT_TRUE(reader.close().ok()); } + EXPECT_EQ(total_rows, + std::min(STRIPES * ROWS_PER_STRIPE, child_cutoff - 1)); + EXPECT_EQ(directory_visits, STRIPES); + ASSERT_TRUE(source_reader.close().ok()); + ASSERT_NE(profile.get_counter("OrcStripeStatisticsParses"), nullptr); + ASSERT_NE(profile.get_counter("OrcStripeStatisticsEvaluations"), nullptr); + EXPECT_EQ(profile.get_counter("OrcStripeStatisticsParses")->value(), 1); + EXPECT_EQ(profile.get_counter("OrcStripeStatisticsEvaluations")->value(), + STRIPES * (parent_sarg ? 2 : 1)); } - ASSERT_TRUE(reader.close().ok()); } - EXPECT_EQ(total_rows, STRIPES * ROWS_PER_STRIPE); - EXPECT_EQ(directory_visits, STRIPES); - ASSERT_TRUE(source_reader.close().ok()); } } diff --git a/contrib/apache-orc b/contrib/apache-orc index 5962bb071d6345..4ec25901b12662 160000 --- a/contrib/apache-orc +++ b/contrib/apache-orc @@ -1 +1 @@ -Subproject commit 5962bb071d6345cc29c5353d64d85a203525e4d9 +Subproject commit 4ec25901b1266235f56ebe5c948277d6f3c2c616 From 7e1fed6605189b661284563513023632bd374d8d Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 18 Nov 2026 14:11:13 +0800 Subject: [PATCH 06/10] [opt](orc) Reuse parsed SDK metadata in physical children ### Background ORC range readers need file-wide footer metadata and absolute stripe row prefixes. Physical split scheduling creates independent readers for one immutable file. ### What problem does this PR solve? Issue Number: None Related PR: #70664 (review follow-up) PR Type: Performance Problem Summary: Each physical child currently reparses the serialized file tail and rebuilds all stripe row prefixes. Reuse a parsed immutable tail and initialize each RowReader from its byte interval, keeping streams, schema, predicates and metrics private. ### Details - **Change map:** 50 Doris production lines connect parsed-tail sharing and profile counters; 83 Doris test lines cover the real TableReader path. The SDK pin carries 225 production and 142 test lines, plus the two-line gitlink change. - **Why it was slow:** createReader reparses the FileTail protobuf and RowReaderImpl scans every stripe for each child. - **Optimization:** Share immutable Footer/PostScript and a memory-pool-backed absolute row-prefix buffer; binary-search byte and row boundaries. Reject an unordered stripe directory before binary search can misselect data. - **Results:** Static operation counts for an eight-stripe source plus eight individual children: tail parses 9 to 1, prefix entries built 72 to 8, RowReader range-accumulation visits 72 to 16. Binary-search comparisons are additional logarithmic work. These counts are source-derived; no elapsed-time or executed-test result is claimed. - **Applicability and cost:** Helps sibling readers of one immutable file, particularly uncoalesced stripes. The process-wide metadata cache still stores serialized tails; physical children use the parsed SDK handle. A parsed tail retains one prefix buffer in its source memory pool, which must outlive every shared handle. The new Reader API requires consumers to rebuild. Metadata-only opens now build the prefix when the tail is parsed; that path is not benchmarked. ### Testing clang-format 16.0.6 and git diff --check passed. Added a real no-SARG TableReader operation-count test for individual and coalesced children, plus SDK concurrency, seek, parent-destruction, serialized-tail, empty-file and malformed-directory tests. BE header hygiene also passed. Build and runtime tests have not run because the current worktree's installed dependencies point into a repository excluded by AGENTS.md; permission to read them is pending. ### Release note Reduce repeated ORC file-metadata initialization for physical split readers. ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Partially verified: compilation, runtime tests and elapsed-time performance remain unverified ### AI Disclosure - Tool: Codex - Generated: Code, tests and this description - Human review: Pending author review --- be/src/format_v2/orc/orc_reader.cpp | 41 ++++++----- be/src/format_v2/orc/orc_reader.h | 9 ++- be/test/format_v2/orc/orc_reader_test.cpp | 83 +++++++++++++++++++++++ contrib/apache-orc | 2 +- 4 files changed, 114 insertions(+), 21 deletions(-) diff --git a/be/src/format_v2/orc/orc_reader.cpp b/be/src/format_v2/orc/orc_reader.cpp index 5c735cf5bb7a5b..de4e50f2778a90 100644 --- a/be/src/format_v2/orc/orc_reader.cpp +++ b/be/src/format_v2/orc/orc_reader.cpp @@ -1188,6 +1188,12 @@ void OrcReader::_init_profile() { _profile, "OrcStripeStatisticsParses", TUnit::UNIT, orc_profile, 1); _orc_profile.stripe_statistics_evaluations = ADD_CHILD_COUNTER_WITH_LEVEL( _profile, "OrcStripeStatisticsEvaluations", TUnit::UNIT, orc_profile, 1); + _orc_profile.file_tail_parses = ADD_CHILD_COUNTER_WITH_LEVEL(_profile, "OrcFileTailParses", + TUnit::UNIT, orc_profile, 1); + _orc_profile.stripe_directory_entries = ADD_CHILD_COUNTER_WITH_LEVEL( + _profile, "OrcStripeDirectoryEntries", TUnit::UNIT, orc_profile, 1); + _orc_profile.row_reader_stripe_visits = ADD_CHILD_COUNTER_WITH_LEVEL( + _profile, "OrcRowReaderStripeVisits", TUnit::UNIT, orc_profile, 1); _orc_profile.reader_call = ADD_CHILD_COUNTER_WITH_LEVEL(_profile, "ReaderCall", TUnit::UNIT, orc_profile, 1); _orc_profile.reader_inclusive_latency_us = ADD_CHILD_COUNTER_WITH_LEVEL( @@ -1240,6 +1246,12 @@ void OrcReader::_collect_profile() const { const auto& reader_metrics = _state->reader_metrics; const uint64_t read_row_count = orc_read_row_count(reader_metrics); if (_profile != nullptr) { + COUNTER_UPDATE(_orc_profile.file_tail_parses, + orc_metric_value(reader_metrics.FileTailParseCount)); + COUNTER_UPDATE(_orc_profile.stripe_directory_entries, + orc_metric_value(reader_metrics.StripeDirectoryEntryCount)); + COUNTER_UPDATE(_orc_profile.row_reader_stripe_visits, + orc_metric_value(reader_metrics.RowReaderStripeVisitCount)); COUNTER_UPDATE(_orc_profile.stripe_statistics_parses, orc_metric_value(reader_metrics.MetadataParseCount)); COUNTER_UPDATE(_orc_profile.stripe_statistics_evaluations, @@ -1333,20 +1345,20 @@ Status OrcReader::init(RuntimeState* state) { input_stream_options.max_merge_distance_bytes = state->query_options().orc_max_merge_distance_bytes; } - auto create_reader = [&](const std::string* serialized_file_tail, + auto create_reader = [&](const OrcSharedFileContext* shared_context, std::unique_ptr<::orc::Reader>* reader) -> Status { DORIS_CHECK(reader != nullptr); ::orc::ReaderOptions options; ObjLRUCache::CacheHandle cache_handle; - const bool cache_hit = serialized_file_tail == nullptr && cache_enabled && + const bool cache_hit = shared_context == nullptr && cache_enabled && meta_cache->lookup(cache_key, &cache_handle); options.setMemoryPool(*ExecEnv::GetInstance()->orc_memory_pool()); options.setReaderMetrics(&_state->reader_metrics); - if (serialized_file_tail != nullptr) { - options.setSerializedFileTail(*serialized_file_tail); - // The SDK handle shares only lazily parsed immutable statistics. Each child keeps - // its own stream, metrics and SARG, including filters received after planning. - options.setSharedStripeStatistics(_state->file_context->stripe_statistics); + if (shared_context != nullptr) { + // Re-parsing a serialized tail in every one-stripe child turns file-wide SDK setup + // quadratic. Share parsed metadata and prefixes; streams, schema and SARG stay private. + options.setSharedFileTail(shared_context->file_tail); + options.setSharedStripeStatistics(shared_context->stripe_statistics); } else if (cache_hit) { options.setSerializedFileTail(*cache_handle.data()); } @@ -1369,7 +1381,7 @@ Status OrcReader::init(RuntimeState* state) { } // Parent contexts retain the existing process-wide tail cache policy. A child uses its // parent's tail without reporting another physical footer read or process-cache lookup. - if (serialized_file_tail == nullptr) { + if (shared_context == nullptr) { if (cache_hit) { ++_reader_statistics.file_footer_hit_cache; ++_state->footer_cache_hits; @@ -1404,12 +1416,8 @@ Status OrcReader::init(RuntimeState* state) { context->file_identity = file_identity; context->has_stable_identity = has_stable_identity; context->stripe_statistics = reader.getSharedStripeStatistics(); + context->file_tail = reader.getSharedFileTail(); try { - if (has_stable_identity) { - // Mutable files cannot publish children, so avoid serializing a tail that no other - // reader may safely consume. - context->serialized_file_tail = reader.getSerializedFileTail(); - } const auto stripe_count = reader.getNumberOfStripes(); context->stripes.reserve(stripe_count); uint64_t first_row = 0; @@ -1447,12 +1455,11 @@ Status OrcReader::init(RuntimeState* state) { return Status::InvalidArgument("ORC split file context does not match file identity"); } if (loaded_reader != nullptr) { - // The parent keeps the reader that produced the context. Generated children rebuild - // independent readers from the shared serialized tail. + // The parent keeps the reader that produced the context. Children retain the parsed + // tail without retaining the parent reader or its stream. _state->reader = std::move(loaded_reader); } else { - RETURN_IF_ERROR( - create_reader(&_state->file_context->serialized_file_tail, &_state->reader)); + RETURN_IF_ERROR(create_reader(_state->file_context.get(), &_state->reader)); } } _state->root_type = &_state->reader->getType(); diff --git a/be/src/format_v2/orc/orc_reader.h b/be/src/format_v2/orc/orc_reader.h index a9020d076883b5..eed32ddc87ce8f 100644 --- a/be/src/format_v2/orc/orc_reader.h +++ b/be/src/format_v2/orc/orc_reader.h @@ -45,13 +45,13 @@ struct OrcStripeMetadata { uint64_t first_row = 0; }; -// ORC readers keep independent row/stripe execution state. Only the immutable serialized tail and -// stripe layout and lazy immutable stripe statistics cross physical children. Streams and +// ORC readers keep independent row/stripe execution state. Only the immutable parsed tail, +// row prefixes, stripe layout and lazy stripe statistics cross physical children. Streams and // SARG appliers remain reader-local, so concurrent scanners never share an ORC RowReader. struct OrcSharedFileContext final : public FileContext { std::string file_identity; bool has_stable_identity = false; - std::string serialized_file_tail; + std::shared_ptr file_tail; std::shared_ptr<::orc::SharedStripeStatistics> stripe_statistics; std::vector stripes; }; @@ -108,6 +108,9 @@ class OrcReader final : public format::FileReader { RuntimeProfile::Counter* file_footer_cache_hits = nullptr; RuntimeProfile::Counter* stripe_statistics_parses = nullptr; RuntimeProfile::Counter* stripe_statistics_evaluations = nullptr; + RuntimeProfile::Counter* file_tail_parses = nullptr; + RuntimeProfile::Counter* stripe_directory_entries = nullptr; + RuntimeProfile::Counter* row_reader_stripe_visits = nullptr; RuntimeProfile::Counter* total_time = nullptr; RuntimeProfile::Counter* reader_call = nullptr; // ReaderCall RuntimeProfile::Counter* reader_inclusive_latency_us = nullptr; // ReaderInclusiveLatencyUs diff --git a/be/test/format_v2/orc/orc_reader_test.cpp b/be/test/format_v2/orc/orc_reader_test.cpp index 70b718b708e561..3386dcc13b28cd 100644 --- a/be/test/format_v2/orc/orc_reader_test.cpp +++ b/be/test/format_v2/orc/orc_reader_test.cpp @@ -5238,6 +5238,89 @@ TEST_F(NewOrcReaderTest, PhysicalStripeChildrenBoundSargDirectoryVisits) { } } +TEST_F(NewOrcReaderTest, PhysicalChildrenBoundSdkSetupWithoutPredicates) { + const auto path = (_test_dir / "shared_sdk_setup.orc").string(); + constexpr size_t STRIPES = 8; + constexpr int64_t ROWS_PER_STRIPE = 200; + std::vector first_values; + for (size_t stripe = 0; stripe < STRIPES; ++stripe) { + first_values.push_back(1 + stripe * ROWS_PER_STRIPE); + } + write_multi_stripe_orc_int_file(path, first_values); + const auto layout = get_orc_stripe_layout(path); + ASSERT_EQ(layout.size(), STRIPES); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + format::ColumnDefinition column; + column.identifier = Field::create_field("id"); + column.name = "id"; + column.type = make_nullable(std::make_shared()); + + // No SARG: statistics counters cannot detect repeated footer parsing or prefix construction. + // Target 0 preserves the legacy FE range and emits one child per stripe; positive targets + // also cover explicitly small and coalesced child intervals. + for (const int64_t target : + {int64_t {0}, int64_t {1}, cast_set(layout[0].length * 4)}) { + SCOPED_TRACE(target); + RuntimeProfile profile("shared_sdk_setup"); + const auto init_reader = [&](format::TableReader* reader) { + return reader->init({.projected_columns = {column}, + .format = format::FileFormat::ORC, + .runtime_state = &state, + .scanner_profile = &profile}); + }; + format::TableReader source_reader; + ASSERT_TRUE(init_reader(&source_reader).ok()); + format::SplitReadOptions options; + options.current_split_format = format::FileFormat::ORC; + options.current_range.__set_path(path); + options.current_range.__set_file_size(std::filesystem::file_size(path)); + options.current_range.__set_start_offset(0); + options.current_range.__set_size(std::filesystem::file_size(path)); + options.current_range.__set_modification_time(19); + options.current_range.__set_target_split_size(target); + ASSERT_TRUE(source_reader.prepare_split(options).ok()); + FileScanSplit source; + source.range = options.current_range; + source.is_source_split = true; + std::vector children; + bool was_split = false; + ASSERT_TRUE(source_reader.build_physical_splits(source, &children, &was_split).ok()); + ASSERT_TRUE(was_split); + ASSERT_GT(children.size(), 1); + // Children must retain parsed metadata even after their planning reader is closed. + ASSERT_TRUE(source_reader.close().ok()); + size_t total_rows = 0; + for (const auto& child : children) { + format::TableReader reader; + ASSERT_TRUE(init_reader(&reader).ok()); + options.current_range = child.materialize_range(); + options.file_context = child.file_context; + options.format_split_id = child.format_split_id; + options.format_split_id_end = child.format_split_id_end; + ASSERT_TRUE(reader.prepare_split(options).ok()); + bool eof = false; + ASSERT_TRUE(reader.create_next_reader(&eof).ok()); + ASSERT_FALSE(eof); + while (!eof) { + auto block = build_file_block({column}); + ASSERT_TRUE(reader.get_block(&block, &eof).ok()); + const auto& ids = assert_cast( + assert_cast(*block.get_by_position(0).column) + .get_nested_column()); + for (size_t row = 0; row < block.rows(); ++row) { + EXPECT_EQ(ids.get_element(row), ++total_rows); + } + } + ASSERT_TRUE(reader.close().ok()); + } + EXPECT_EQ(total_rows, STRIPES * ROWS_PER_STRIPE); + EXPECT_EQ(profile.get_counter("OrcFileTailParses")->value(), 1); + EXPECT_EQ(profile.get_counter("OrcStripeDirectoryEntries")->value(), STRIPES); + EXPECT_EQ(profile.get_counter("OrcRowReaderStripeVisits")->value(), 2 * STRIPES); + EXPECT_EQ(profile.get_counter("OrcStripeStatisticsParses")->value(), 0); + } +} + TEST_F(NewOrcReaderTest, OpenAcceptsDorisOffsetTimezone) { auto reader = create_reader(); RuntimeState state {TQueryOptions(), TQueryGlobals()}; diff --git a/contrib/apache-orc b/contrib/apache-orc index 4ec25901b12662..e410e9b969392e 160000 --- a/contrib/apache-orc +++ b/contrib/apache-orc @@ -1 +1 @@ -Subproject commit 4ec25901b1266235f56ebe5c948277d6f3c2c616 +Subproject commit e410e9b969392ed6a887e90e92abfb61721895aa From 50c71c300154558e45a185924cbc11e58f47d024 Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 18 Nov 2026 15:30:58 +0800 Subject: [PATCH 07/10] [opt](orc) Preserve lazy shared row-prefix initialization ### Background Physical ORC children reuse parsed file tails and absolute stripe row prefixes. Readers that only inspect metadata do not need row prefixes. ### What problem does this PR solve? Issue Number: None Related PR: #70664 (review follow-up) PR Type: Performance Problem Summary: Keep the shared prefix lazy so metadata-only opens avoid stripe traversal and allocation. Concurrent children build one immutable prefix with std::call_once, and an initialization failure remains retryable. ### Details - **Why it was slow:** Eager prefix construction adds file-wide work to metadata-only opens. Before parsed-tail sharing, each physical child also reparsed the tail and rebuilt every prefix. - **Optimization:** Build the source-pool-backed prefix only when a RowReader needs it and publish only after success. Keep streams, mutable schemas, predicates and metrics private. - **Results:** The eight-stripe real TableReader fixture passes at one tail parse, eight prefix entries and sixteen range visits. Temporarily restoring repeated setup makes the same test fail at nine, seventy-two and seventy-two, including the nonpositive-target case. SDK metadata-only opens build zero prefix entries; cold concurrent children build eight in total and warm children build none. These are measured operation counts, not elapsed-time claims. - **Applicability and cost:** Reuses work within one immutable file context. Byte/row range lookup uses logarithmic comparisons in addition to the counted accumulation visits. The source memory pool must outlive its shared handles. Serialized-tail callers remain supported; no session, wire or storage format change is introduced by this follow-up. ### Testing Local BE ASAN build through build.sh passed. run-be-ut.sh ran 255 related tests with zero failures or skips after restoring the candidate from the counterfactual. Coverage includes reader values/absolute row positions, read intent, physical splitting, shared statistics, cold concurrent initialization, warm reuse, parent destruction, seeks, empty files/ranges and repeated malformed-directory errors. Changed-line BE clang-tidy, clang-format 16, build-header hygiene and diff checks passed. SDK formatting passed separately; SDK code is excluded from the BE clang-tidy script by policy. The counterfactual is removed and all candidate source hashes match. No cluster regression or RELEASE timing benchmark was run. ### Release note Reduce repeated ORC physical-child metadata setup while preserving lazy metadata-only opens. ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Not ready: the SDK pin is local and must be published and merged upstream before submission ### AI Disclosure - Tool: Codex - Generated: Code, tests and this description - Human review: Pending author review --- be/test/format_v2/orc/orc_reader_test.cpp | 3 +++ contrib/apache-orc | 2 +- 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/be/test/format_v2/orc/orc_reader_test.cpp b/be/test/format_v2/orc/orc_reader_test.cpp index 3386dcc13b28cd..bca6e1484a8f2c 100644 --- a/be/test/format_v2/orc/orc_reader_test.cpp +++ b/be/test/format_v2/orc/orc_reader_test.cpp @@ -5264,7 +5264,10 @@ TEST_F(NewOrcReaderTest, PhysicalChildrenBoundSdkSetupWithoutPredicates) { RuntimeProfile profile("shared_sdk_setup"); const auto init_reader = [&](format::TableReader* reader) { return reader->init({.projected_columns = {column}, + .conjuncts = {}, .format = format::FileFormat::ORC, + .scan_params = nullptr, + .io_ctx = nullptr, .runtime_state = &state, .scanner_profile = &profile}); }; diff --git a/contrib/apache-orc b/contrib/apache-orc index e410e9b969392e..89c98c2a1befea 160000 --- a/contrib/apache-orc +++ b/contrib/apache-orc @@ -1 +1 @@ -Subproject commit e410e9b969392ed6a887e90e92abfb61721895aa +Subproject commit 89c98c2a1befeabe5e9bfae30ba394db48223611 From 0430b1447f22394e8e95c9b577bc2256ba20639c Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 18 Nov 2026 16:45:57 +0800 Subject: [PATCH 08/10] [deps](orc) Consume merged parsed-tail sharing SDK ### Background ORC physical stripe readers reuse parsed immutable file metadata and lazy row prefixes. The SDK implementation has merged into the upstream orc branch. ### What problem does this PR solve? Issue Number: None Related PR: apache/doris-thirdparty#436 (merged SDK), #70664 (consumer) PR Type: Release/Dependency Problem Summary: Replace the local SDK pin with its upstream squash-merge commit so the Doris dependency is available from the maintained branch. ### Details - **Versions:** contrib/apache-orc 89c98c2 to upstream 00fa6d6. - **Upstream changes:** Both commits have identical complete source trees; #436 combines parsed file-tail sharing with lazy prefix initialization. - **Compatibility:** Source and test contents are unchanged. Native consumers build against the extended SDK interface already used by this branch. - **Rollback:** Restore the prior pin if the dependency update must be reverted. ### Testing Verified identical SDK tree hashes. The prior local ASAN build and 255 selected tests cover these exact SDK source bytes; they were not rerun for the hash-only pin update. ### Release note None ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Ready ### AI Disclosure - Tool: Codex - Generated: Dependency update and commit description - Human review: not stated --- contrib/apache-orc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/contrib/apache-orc b/contrib/apache-orc index 89c98c2a1befea..00fa6d69e8ee1b 160000 --- a/contrib/apache-orc +++ b/contrib/apache-orc @@ -1 +1 @@ -Subproject commit 89c98c2a1befeabe5e9bfae30ba394db48223611 +Subproject commit 00fa6d69e8ee1bfca05aa98037634bc42720d0db From feec2880ae7e16646fe6b759e632ef302aac5177 Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 18 Nov 2026 17:13:54 +0800 Subject: [PATCH 09/10] [chore](be) Explain assertion macro style suppressions ### Background Source split progress uses DORIS_CHECK to assert positive child counts and batch-width samples. The macro supports streaming error messages through an intentionally unbraced failure branch. ### What problem does this PR solve? Issue Number: None Related PR: #70664 (source split progress) PR Type: Build/CI/Chore Problem Summary: Clang-tidy reports the macro expansion at three call sites as inconsistent if/else braces. Document and narrowly suppress those expansion-only diagnostics without changing the macro or any executable tokens. ### Details - **Affected:** Three DORIS_CHECK call sites in file_scan_context.h. - **Developer and release impact:** Explain the existing streaming macro contract and keep changed-line static checks actionable. Runtime behavior is unchanged. - **License and third-party:** None. - **Rollback:** Remove the NOLINT comments if the checker learns to ignore this macro expansion. ### Testing Verified that removing line comments makes the old and new headers identical. Project clang-format16 and header hygiene passed. The current ASAN BE+FE build predates this comment-only cleanup; focused BE/FE tests and final static checks are in progress. ### Release note None ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Partially verified: final static checking is in progress ### AI Disclosure - Tool: Codex - Generated: Narrow style-suppression comments and commit description - Human review: not stated --- be/src/format_v2/file_scan_context.h | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/be/src/format_v2/file_scan_context.h b/be/src/format_v2/file_scan_context.h index 68a2898727aed2..37408e5c65715f 100644 --- a/be/src/format_v2/file_scan_context.h +++ b/be/src/format_v2/file_scan_context.h @@ -44,12 +44,16 @@ class FileContext { class SourceSplitProgress { public: void reset_for_children(size_t children) { + // DORIS_CHECK's streaming failure branch deliberately expands to an unbraced else. + // NOLINTNEXTLINE(readability-inconsistent-ifelse-braces) DORIS_CHECK(children > 0); _remaining.store(children, std::memory_order_release); } bool complete_one() { const size_t previous = _remaining.fetch_sub(1, std::memory_order_acq_rel); + // DORIS_CHECK's streaming failure branch deliberately expands to an unbraced else. + // NOLINTNEXTLINE(readability-inconsistent-ifelse-braces) DORIS_CHECK(previous > 0); return previous == 1; } @@ -60,6 +64,8 @@ class SourceSplitProgress { } bool update_adaptive_batch_bytes_per_row(double bytes_per_row) { + // DORIS_CHECK's streaming failure branch deliberately expands to an unbraced else. + // NOLINTNEXTLINE(readability-inconsistent-ifelse-braces) DORIS_CHECK(bytes_per_row > 0); std::scoped_lock lock(_adaptive_batch_lock); // Sibling scanners can finish probes concurrently. Merge their samples under the source From 2998c21b6ba46b1e3501457aed8a48bf5dabef25 Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 18 Nov 2026 20:53:51 +0800 Subject: [PATCH 10/10] [test](iceberg) Align manifest planning tests with coarse split sizing ### Background Iceberg manifest planning tests verify query-wide read budgets and cancellation before scheduler acquisition. FEUT runs these tests through both materialized and streaming scan planning. ### What problem does this PR solve? Issue Number: None Related PR: #70664 (two-level external file splitting) PR Type: Test Problem Summary: Manifest planning gained a coarse split size argument, but its existing reflection-based tests still use the previous signatures. Update the method lookup and invocation arguments so the tests reach their original cancellation and shared-budget assertions. ### Details - **Failure:** Full FEUT reports four NoSuchMethodException errors and four wrong-number-of-arguments assertion failures in IcebergManifestReadContextTest. - **Cause:** The planFileScanTask and streamingFileScanTasks test calls omit the new long coarseSplitSize parameter. Reflection defers the mismatch until test execution. - **Stability evidence:** The unchanged 14-case test class passes locally after correcting the signatures. All 389 selected tests in 10 related Iceberg classes pass with zero failures, errors or skips. The fixture uses 0L to preserve legacy split sizing; assertions, synchronization and timeouts are unchanged. ### Testing - Local run-fe-ut.sh --run 'IcebergManifest*Test,IcebergScanPlanProvider*Test,IcebergScanRangeTest,IcebergStatementScopeTest': 389 tests passed, including all 14 IcebergManifestReadContextTest cases. - Full FEUT on the updated head was not rerun locally. ### Release note None ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Ready ### AI Disclosure - Tool: Codex - Generated: Test signature updates, verification and this description. - Human review: not stated --- .../iceberg/IcebergManifestReadContextTest.java | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergManifestReadContextTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergManifestReadContextTest.java index 2ab89bae81e70c..277ab3182c9a49 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergManifestReadContextTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergManifestReadContextTest.java @@ -139,7 +139,8 @@ public boolean tryAcquire(long timeout, TimeUnit unit) throws InterruptedExcepti IcebergCatalogProperties.of(Map.of(IcebergConnector.MANIFEST_CACHE_ENABLE, "true")), new RecordingIcebergCatalogOps(), null, cache); Method plan = IcebergScanPlanProvider.class.getDeclaredMethod("planFileScanTask", - TableScan.class, ConnectorSession.class, Table.class, Optional.class, IcebergTableHandle.class); + TableScan.class, ConnectorSession.class, Table.class, Optional.class, IcebergTableHandle.class, + long.class); plan.setAccessible(true); ExecutorService worker = Executors.newSingleThreadExecutor(); budget.acquire(); // The first scan is currently loading a cold data manifest. @@ -148,7 +149,7 @@ public boolean tryAcquire(long timeout, TimeUnit unit) throws InterruptedExcepti ? provider.getScanNodeProperties(session, new IcebergTableHandle("db1", "t1"), Collections.emptyList(), Optional.empty()) : plan.invoke(provider, scan, session, table, Optional.empty(), - new IcebergTableHandle("db1", "t1"))); + new IcebergTableHandle("db1", "t1"), 0L)); Assertions.assertTrue(waiting.await(5, TimeUnit.SECONDS)); Assertions.assertEquals(0, deleteReads.get(), "delete reads cannot bypass the shared query budget"); if (cancel) { @@ -234,12 +235,12 @@ void cancellationBeforeSchedulerAcquisitionCannotRestartSdkPlanning(String path, switch (path) { case "streaming": methodName = "streamingFileScanTasks"; - arguments = new Object[] {scan, session, table, Optional.empty(), 128L, null, handle, + arguments = new Object[] {scan, session, table, Optional.empty(), 128L, 0L, null, handle, new ConnectorScanContext()}; break; case "materialized": methodName = "planFileScanTask"; - arguments = new Object[] {scan, session, table, Optional.empty(), handle}; + arguments = new Object[] {scan, session, table, Optional.empty(), handle, 0L}; break; case "representative": methodName = "planManifestCountRange";