Repository navigation
Conversation
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
### What problem does this PR solve? Issue Number: close apache#69884 Related PR: apache#69904 Problem Summary: Ordinary external-table queries only consulted the table-keyed row-count cache, which could keep a cold estimate after versioned connector statistics completed. Prefer statistics at the statement's pinned snapshot for ordinary and versioned queries, reuse the existing size/row-width fallback, and forward the pinned snapshot through the Hive sibling gateway. This covers STAT-02. ### Release note External-table optimizer estimates can use completed statistics for the query's pinned snapshot without waiting for the table-only row-count cache to refresh. ### Check List (For Author) - Test: Full FE build and Checkstyle; 114 FE core and 267 Hive tests; focused compatibility run including 22 Paimon and 7 Iceberg tests; seven Delta p0 suites on the combined change, including new filesystem/HMS/Hive cardinality coverage. - Behavior changed: Yes, ordinary pinned queries prefer snapshot statistics. - Does this need documentation: Yes, Delta connector README and prepared EN/ZH website patch are updated in the accompanying statistics change; website PR pending.
### What problem does this PR solve? Issue Number: close apache#69884 Related PR: apache#69904 Problem Summary: The native Delta connector did not supply basic table statistics to the optimizer. Cache logical row counts by user, table identity and version, summing active AddFile numRecords minus deletion-vector cardinality. Collect in a bounded background pool and return same-version CRC size while cold; missing row statistics stay unknown. Shared cache invalidation cancels pending work, and exact COUNT remains independent of optimizer estimates. Add shared-snapshot concurrency, real checkpoint/DV, lifecycle, authentication and million-file tests (STAT-01/03), plus filesystem/HMS/Hive regression coverage and connector documentation. ### Release note Native Delta Lake catalogs and Hive gateway queries gain basic versioned row/size statistics with asynchronous collection and a CRC size fallback. ### Check List (For Author) - Test: Full FE build and Checkstyle; 87 Delta and 2 metastore tests, including the opt-in million-file fixture in a 512 MiB JVM; focused compatibility/authentication rerun; seven Delta p0 suites passed on the final build. New .out generated by the official regression script and then verified without generation. Preserved all 1,227 MinIO objects, checked by path, size and SHA-256. - Behavior changed: Yes, Delta exposes basic optimizer statistics. Exact COUNT semantics and Delta data are unchanged; column NDV/histograms are not supplied. - Does this need documentation: Yes, connector README included; local design, tasklist and EN/ZH website patch updated. Website PR/full site build pending.
|
run buildall |
…stics ### What problem does this PR solve? Issue Number: close apache#69884 Related PR: apache#69904 Problem Summary: Delta table row counts load asynchronously, so a post-statistics benchmark needs to distinguish ready statistics from a cold metadata lookup. Add an optional retained-manifest check that waits for Delta EXPLAIN cardinalities before timing and records Hive estimates. Keep the existing query texts, common controls, result comparator and three alternating measured pairs unchanged. Document the complete local TPC-DS rerun: the summed median gap narrows from 26.1% to 16.5%, with remaining regressions retained rather than replaced by focused samples. No production code or fixture data changes. ### Release note None ### Check List (For Author) - Test: Official regression script passed all 103 TPC-DS queries (824 executions) with matching results; all 24 Delta cardinalities match the independent manifest. A separate Q4 rerun passed, and eight complete diagnostic profiles were retained. All 1,227 MinIO objects match their original path/size/SHA-256 before and after. Source whitespace checks and code-review checkpoints passed. - Behavior changed: No production change. Optional benchmark warmup is disabled unless deltalakeBenchmarkStatisticsManifest is provided. - Does this need documentation: Yes, benchmark README includes warmup usage, before/after measurements, controls and remaining performance limitations.
|
run buildall |
1 similar comment
|
run buildall |
### What problem does this PR solve? Issue Number: close apache#69884 Related PR: apache#69904 Problem Summary: COMPILE, FE UT, Cloud UT and BE UT all failed while merging the PR into master 5af2df3, before their build or test commands ran. Keep master's unified FileScannerV2 and split-producer lifecycle. Route Delta through the request-based streaming SPI, honor cancellation and partition selection, and validate the physical format on every Delta scan range. Propagate missing snapshot files and scheduling errors. Include Delta in the default FE unit-test reactor so the CI invocation exercises the connector. ### Release note Delta scans fail if a committed data or deletion-vector file cannot be found, instead of silently skipping a range when ignore_not_found_file is enabled. ### Check List (For Author) - Test: Full FE and ASAN BE builds; standard FE unit-test selection (857 passed, 2 opt-in cases skipped); FE Checkstyle, clang-format 16 and BE header hygiene. ASAN BE unit tests and a final FE lifecycle fixture recheck were still running or queued at commit time. - Behavior changed: Yes. Delta uses master's common scanner and streamed split lifecycle and fails on missing snapshot files. - Does this need documentation: Yes (connector README updated in this PR; the end-user documentation PR is tracked in apache#69884).
FE UT Coverage ReportIncrement line coverage |
TPC-H: Total hot run time: 28313 ms |
TPC-DS: Total hot run time: 151400 ms |
ClickBench: Total hot run time: 25.2 s |
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
### What problem does this PR solve? Issue Number: close apache#69884 Related PR: apache#69904 Problem Summary: File scans always use FileScannerV2. The basic and features Delta suites still exercised the same reader twice for both filesystem and HMS catalogs. Run them with enable_file_scanner_v2=true and preserve the retained query tags. The dedicated Hive gateway and native reader dispatch checks still exercise the deprecated false setting. ### Release note None ### Check List (For Author) - Test: Regenerated expected outputs with run-regression-test.sh and reran test_deltalake_basic and test_deltalake_features in comparison mode. Both suites passed both runs. All 146 retained query output blocks were byte-identical. Removed 32 redundant blocks (526 lines). - Behavior changed: No - Does this need documentation: No
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
TPC-H: Total hot run time: 28448 ms |
TPC-H: Total hot run time: 28753 ms |
TPC-DS: Total hot run time: 152082 ms |
ClickBench: Total hot run time: 24.35 s |
### What problem does this PR solve? Issue Number: close apache#69884 Related PR: apache#69904 Problem Summary: Commit 598756e made ordinary plugin table reads resolve their pinned snapshot statistics on the planning thread. This changed the row-count path for Iceberg, Paimon and Hive as part of a Delta feature. Revert that commit's FE table and Hive statistics changes. Delta still collects logical rows asynchronously per version and supplies same-version CRC bytes when its count is unavailable. Exercise cold estimates in p0 and wait for the standard FE cache refresh before timing the p2 benchmark. ### Release note Ordinary catalog queries use the standard FE external-table row-count cache. Delta estimates in EXPLAIN may retain their initial value until an access triggers a refresh (10 minutes by default). Exact COUNT is independent. ### Check List (For Author) - Test: Full FE build and Checkstyle. All 102 targeted FE tests passed. test_deltalake_statistics passed for filesystem, HMS and Hive. With the default refresh interval, all 24 TPC-DS manifest counts matched and Q1 passed for both readers in test_deltalake_benchmark. - Behavior changed: Yes. Restore master's row-count policy for ordinary plugin table reads and its original Hive statistics delegation. - Does this need documentation: Yes (connector README and benchmark guide updated in this PR; website documentation is tracked in apache#69884).
|
run buildall |
### What problem does this PR solve? Issue Number: close apache#69884 Related PR: apache#69904 Problem Summary: Merging master 7acfd9b left two Delta gaps. First, apache#69717 replaced table_reader.h's full ShardedKVCache definition with a forward declaration, but deltalake_reader_test.cpp constructs a ShardedKVCache and relied on the transitive include. Include format/format_common.h directly, as hudi_reader_test.cpp does. Second, apache#69300 made Parquet readers in one scan share their profile counters through TableReader's parquet profile cache. Hudi, Iceberg and the base reader pass that cache, but DeltaReader still built its ParquetReader without it, so every Delta file created its own set of counters. Pass the cache, and keep binary-UUID preservation disabled as before. ### Release note None ### Check List (For Author) - Test: No local build. The merge was conflict-free. Both changes follow the matching master code. COMPILE and BE UT are left to CI. - Behavior changed: No. Delta scan profiles reuse Parquet counters across files, as other formats do. - Does this need documentation: No
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
### What problem does this PR solve? Issue Number: close apache#69884 Related PR: apache#69904 Problem Summary: Extracting decode_portable_deletion_vector_payload for inline Delta DVs left the framed Iceberg/Delta decoder checking the portable magic twice. Its first check runs before the CRC, and the payload function checked it again. The magic constant was also defined twice. Move the constant, the magic check and the Roaring decode into file-local helpers. The framed decoder again validates magic, then CRC, then decodes the bitmap once, with the same offsets and length it used before this PR. The inline payload decoder keeps its own size check and uses the same helpers. ### Release note None ### Check List (For Author) - Test: No local build. Existing Iceberg DV unit tests cover the magic and CRC rejection order. The Delta reader unit tests compare inline and on-disk decoding. Both are left to CI. - Behavior changed: No. Iceberg DV decoding is again identical to master's. - Does this need documentation: No
…n validation ### What problem does this PR solve? Issue Number: close apache#69884 Related PR: apache#69904 Problem Summary: validate_file_scan_support is the shared range gate for table markers that are no longer supported. The Delta integration also added a rule there that rejects non-Parquet Delta ranges. That state cannot occur. The Delta protocol stores data only as Parquet, and DeltaLakeScanRange always sends FORMAT_PARQUET. DeltaReader also asserts Parquet in create_file_reader. That runs for every data file, after prepare_split resets the reader format from the current range. Remove the shared rule and its unit test, so validate_file_scan_support is identical to master again. Keep the Delta entry in can_ignore_not_found_file. A missing file in a committed Delta snapshot must still fail the scan, as for Iceberg. ### Release note None ### Check List (For Author) - Test: No local build. The remaining Delta dispatch and missing-file unit tests are left to CI. - Behavior changed: No for FE-generated plans. A malformed non-Parquet Delta range now fails the DeltaReader assertion instead of returning NotSupported. - Does this need documentation: No
… the Delta PR ### What problem does this PR solve? Issue Number: close apache#69884 Related PR: apache#69904 Problem Summary: The Delta integration widened two fe-core catch blocks to Error. One is in executeWithProfileFinalization, on the batch planning path. The other is in the streaming profile finalizer. A lazily loaded plugin dependency then reported NoClassDefFoundError like any other failure. These are general hardening changes for every plugin connector, and Delta does not depend on them. Master's SplitProducer already catches Throwable while it iterates, closes the source and runs completion. The scan still fails without hanging and still calls finishSchedule. Of the four streaming tests added with the change, three cover existing master behavior. Only the profile-phase Error case needs the widened catch. Restore PluginDrivenScanNode and its scan profile test to master so the Delta PR leaves fe-core unchanged. The hardening can be proposed on its own. ### Release note None ### Check List (For Author) - Test: No local build. Both files are identical to master; left to CI. - Behavior changed: No relative to master. A Delta-only branch would have reported a profile-phase Error as a UserException. - Does this need documentation: No
…catalog ### Background An HMS catalog can delegate different tables in one statement to the Delta and Iceberg connectors. Its scan nodes share catalog metadata, while Delta's scan provider rejects CAST-bearing file-pruning predicates and Iceberg retains its existing policy. ### What problem does this PR solve? Issue Number: apache#69884 Related PR: None PR Type: Test Problem Summary: The existing Delta CAST regression tested each catalog entrance separately and did not mix Delta and Iceberg scans in one statement. Add mixed HMS queries and shared-metadata unit coverage so a catalog-wide CAST opt-out or a policy leaking between table scans cannot pass unnoticed. ### Details - **Scenarios:** UNION ALL and JOIN across Delta and Iceberg tables in one HMS catalog, both table orders, synchronous and streaming scans, and an independent ordinary Delta predicate. The FE test alternates two scan nodes sharing the same metadata instance in both orders. - **What existing tests miss:** A blanket HMS CAST opt-out returns correct rows for these fixtures but changes Iceberg's file-pruning behavior. Separate-catalog queries do not exercise the shared metadata boundary. - **Fails when broken:** Pushing the Delta CAST loses the matching 123.45 row. Disabling Iceberg's existing CAST pushdown removes its EXPLAIN predicate and makes the synchronous scan select both files instead of one. The ordinary Delta predicate must still prune one of its two files. ### Testing - Local run-regression-test.sh: test_deltalake_hms_mixed_cast with -genOut and an ordinary rerun both passed. Independently checked all 14 generated result blocks; the golden SHA-256 stayed unchanged. - Local run-regression-test.sh: test_deltalake_cast_pruning passed all 24 result blocks and test_deltalake_basic passed all 36 result blocks. - Local run-fe-ut.sh: 108 tests across PluginDrivenScanNodeCastPredicateTest, Hive routing, Delta, Iceberg, Paimon, JDBC and SPI passed, with no failures, errors or skipped tests. - Local FE Checkstyle reported zero violations; git diff --check passed. ### Notes for Reviewers Iceberg fixtures use integral decimal values to test its existing policy without relying on the separately known lossy CAST bug. Iceberg publishes SDK file counters only for synchronous planning; both modes check its EXPLAIN predicate and actual query results. Production code is unchanged in this commit. ### Release note None ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Ready ### AI Disclosure - Tool: OpenAI Codex - Generated: Tests, generated regression output, and this description - Human review: Pending
|
run buildall |
### Background Delta's scan provider rejects CAST-bearing predicates before the shared converter strips their casts. The mixed HMS regression reads Delta and Iceberg tables in one statement, and originally asserted Iceberg's existing CAST pushdown behavior as well as query results. ### What problem does this PR solve? Issue Number: apache#69884 Related PR: None PR Type: Test Problem Summary: The Delta connector unit tests did not directly guard its scan provider's CAST rejection. Meanwhile, the mixed regression would reject a future Iceberg correctness fix that disables unsafe CAST pushdown. Add a real-provider unit assertion and keep the mixed regression focused on query results and Delta pruning. ### Details - **Scenarios:** Obtain the real scan provider from the Delta connector and assert that it implements ConnectorPushdownOps and returns false for CAST pushdown. Keep the mixed HMS JOIN, UNION ALL, table-order and synchronous/streaming result checks. - **What existing tests miss:** Shared FE tests use provider stubs, so removing the real Delta override could pass them. Iceberg's current CAST pushdown policy is not a permanent correctness contract for a Delta regression. - **Fails when broken:** Removing Delta's interface opt-in or its false-returning override fails the connector unit test. Losing matching rows still fails the mixed golden results. Iceberg can change its own CAST policy without failing assertions about its plan text or selected file count. ### Testing - run-fe-ut.sh: DeltaLakeScanPlanProviderTest and PluginDrivenScanNodeCastPredicateTest passed all 32 tests, with no failures, errors or skips. - run-regression-test.sh: test_deltalake_hms_mixed_cast passed an ordinary rerun with all 14 result blocks. Its generated golden file is unchanged. - Delta connector Checkstyle reported zero violations; git diff --check passed. ### Release note None ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Ready ### AI Disclosure - Tool: OpenAI Codex - Generated: Unit test, regression revisions, and this description - Human review: Pending
Resolve three conflicts with current master: - be/src/format_v2/table_reader.cpp: keep master's comment on the existing format_common.h include. - build.sh: keep master's MongoDB packaging step and the connector list that includes deltalake. - docker/thirdparties/docker-compose/iceberg/README.md: keep master's encrypted fixture text, followed by the Delta Lake regression section. Test: No local build. Build hygiene, git diff --check and bash -n build.sh passed. Compilation and tests are left to CI.
|
run buildall |
### Background Delta chooses synchronous or streaming split planning from the current snapshot's file count. Writers without a current checksum leave that count unknown, so even a two-file table streams and can start far more scanners than the selected file needs. ### What problem does this PR solve? Issue Number: apache#69884 Related PR: apache#69904 (Delta M1 reader) PR Type: Performance Problem Summary: Reuse complete file counts already available in the statement or in Delta's user/table/version-scoped statistics cache when no current checksum is available. Existing statistics collection counts files, and exhaustive normal scans publish their already-observed total without submitting a statistics task. This lets subsequent small-table queries use synchronous planning while keeping large and unknown tables streaming. Changes are confined to the Delta plugin, its tests, and its README. ### Details - **Why it was slow:** The streaming scanner path does not cap scanner creation by the actual split count. On the retained two-file fixture, the automatic path started 144 scanners to read one selected file and one row; forced synchronous planning started one. - **Optimization:** Check the current CRC, then the statement's complete count, then the same user/table identity/version cache entry. The gate never enumerates files, starts statistics work, or waits for storage. A fully consumed scan without partition-pruned replay publishes its total into the entry captured before scanning, so late reports cannot repopulate invalidated entries. Background row-count sessions can have a different identity from SQL sessions, making normal-scan publication necessary without relaxing user isolation. - **Results:** On local macOS ARM64 with the same ASAN BE and retained data, 16 alternating warm pairs per entrance reduced automatic median latency from 61.614 to 41.420 ms (filesystem), 66.370 to 38.509 ms (HMS), and 57.911 to 32.874 ms (HMS gateway). Scanner counts became one; all returned 123.45. Forced-sync controls were 35.978/41.056, 44.302/37.627, and 32.914/32.370 ms before/after. Million-file metadata enumeration medians were 1493.563/1451.023 ms with a 512 MiB heap; both scale tests passed. - **Applicability and cost:** Warm snapshots with an already-known complete small file count benefit. Cold unknown counts still stream. Reuse existing cache capacity, TTL, invalidation and executor ownership; add one count, not a file list. These are small local ASAN measurements, not a TPC-DS claim. Sampled scan heap peaks increased from 133327520 to 238172480 bytes while remaining within 512 MiB; actual million-file data IO was not measured. - **Behavior change:** Small Delta tables may select synchronous split planning after a complete count is available. Row results, public row-count cache behavior, BE scheduling, other connectors, and wire/storage formats are unchanged. ### Testing - Local `./build.sh --fe -j4`: passed, including Checkstyle. Existing ASAN BE binary and local source configurations were unchanged by SHA-256. - Official FE UT runner for `DeltaLake*Test,DeltaKernel*Test`: 92 actual tests passed, zero failures/errors; two optional classes skipped in that invocation. Separately enabled `DeltaLakePlanningScaleTest` with the retained million-file checkpoint and `-Xmx512m`: two tests passed. The diagnostic PlanningMetrics class remained disabled. - Tests cover cached/unknown/current-CRC counts, threshold equality, missing row statistics, user/version/UUID/path isolation, no work submission or IO wait, late invalidated reports, partial and partition-pruned scans, large-file splitting, and large-table streaming after count reuse. - Official regression runner: basic, statistics, cast_pruning, parquet_deletion_vector, hms_mixed_cast, and planning passed: six suites, 129 result blocks. Existing golden files were unchanged. The planning suite initially targeted the wrong local MinIO; using the retained fixture's endpoint resolved the missing-database error without changing the test or data. - `git diff --check`: passed. ### Release note Delta Lake queries can avoid excessive scanner startup for small tables without a checksum once a complete file count is available. Large tables and snapshots with unknown file counts continue to use streaming planning. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: Yes; the Delta connector README documents selection, cache reuse and cold-query limitations. No new user configuration is introduced. - Pre-submission status: Partially verified: local tests and targeted performance checks passed; full TPC-DS and large-table data IO performance were not rerun. ### AI Disclosure - Tool: Codex (GPT-6) - Generated: Implementation, tests, README, local evidence report and this description. - Human review: Pending.
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
TPC-H: Total hot run time: 28591 ms |
TPC-DS: Total hot run time: 154395 ms |
ClickBench: Total hot run time: 23.94 s |
### Background Delta fixes a snapshot for each statement, then enumerates its active files to plan native Parquet ranges. Repeated scans currently reopen the same checkpoint and reconstruct the same ranges. Other connectors already use the existing statement planning scope and external scan task reuse session flag. ### What problem does this PR solve? Issue Number: apache#69884 Related PR: apache#69904 (Delta M1 implementation) PR Type: Performance Problem Summary: Repeated identical synchronous Delta scans redo checkpoint IO and planning. Reuse completed immutable ranges and metrics within the existing planning scope, with typed request keys, bounded retention and cancellable single-flight loading. The change applies to dedicated Delta and HMS gateway reads. Production changes are confined to the Delta plugin; most added lines are tests and generated output. ### Details - **Change map:** Four production Java files (269 added / 50 removed lines), five test files (549 added / 4 removed), 244 generated expected-output lines and 33 README lines. The new memo owns keys, admission and concurrency; the existing scan body is extracted without changing its pruning semantics. Provider/scale tests and a three-route SQL suite cover reuse and bypass. - **Why it was slow:** Each scan independently enumerated the pinned snapshot, including repeated checkpoint range GETs, decode and range construction. - **Optimization:** Use the existing enable_external_scan_task_reuse flag and planning-only statement scope. Key by catalog/query/user, table coordinates/UUID/version, typed filter, partition selection, COUNT, split size and time zone. Coalesce identical loads outside scope/memo locks; publish only completed results and per-node metrics. Retry failed/cancelled loads; let waiters cancel independently. Retain at most 128 entries, 16,384 ranges and 16 MiB estimated range payload; oversized synchronous results are returned without retention. Streaming never enters the memo. - **Results:** Same-build alternating local ASAN/SF1 A/B: Q88 EXPLAIN 323.616 to 152.279 ms and execution 2268.353 to 2050.009 ms (-9.6%); Q80 EXPLAIN -23.9%, execution -1.9%; Q4 EXPLAIN -49.4%, execution essentially flat; Q22 has no hits and execution is flat (+0.5%). Three independent traces each show checkpoint GETs 160 to 55 / 90 to 60 / 90 to 30 for Q88/Q80/Q4; Q22 stays at 15. A million-file streaming fixture remains bounded in a 512 MiB JVM: enumeration median 1493.940 to 1484.218 ms. Sampled heap peak 112.921 to 125.744 MB is recorded, not claimed unchanged. - **Applicability and cost:** Helps identical synchronous scans within one statement. Does not share content across statements or change large-table streaming. The byte budget covers retained range payload, not whole-query or in-flight heap. Query results and FE/BE wire formats are unchanged; all four targeted queries match Hive reference results. This is not a full TPC-DS or RELEASE measurement. - **Behavior change:** Repeated Delta plans may share immutable ranges until planning ends when the existing flag is enabled. PlanningReuseHit identifies each reuse. Disable the session flag to bypass the memo. Shared FE/SPI/BE and other connectors are unchanged. ### Testing Local macOS ARM64: - build.sh --fe -j4: full FE build and Checkstyle passed; final test-only refinements also passed reactor validate/Checkstyle. - run-fe-ut.sh --run 'DeltaLake*Test,DeltaKernel*Test', with follow-up memo/provider and opt-in scale runs: 102 distinct executed tests passed; the optional PlanningMetrics diagnostic class was not run. - Official regression runner: test_deltalake_basic (36 blocks), test_deltalake_scan_reuse (27), test_deltalake_parquet_deletion_vector (32), test_deltalake_hms_mixed_cast (14), all passed. The new .out was script-generated, independently audited, then compared without -genOut; existing .out files are unchanged. - Retained-data A/B: 10 alternating EXPLAIN pairs and 7 execution pairs per query, after three warmups per flag and readiness of all 24 table counts. All 88 Delta executions (including warmups/profiles) matched four Hive references. Separate three-sample storage traces verified requests and bytes. - Million-file streaming before/after: independent 512 MiB JVMs, three complete enumerations each, with FE/BE/Docker stopped for the matched comparison. - All 96 retained TPC-DS objects matched path/size/SHA-256 before and after. Temporary services were stopped and fixtures/volumes retained. ### Release note Repeated identical Delta Lake scans within one statement can plan faster, including Delta tables queried through an HMS catalog. The optimization uses the existing external scan task reuse setting and preserves streaming for large tables. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: Yes; the connector README and local design/tasklist are updated. The broader Delta website documentation remains tracked by apache#69884. - Pre-submission status: Partially verified: local build, relevant tests and targeted ASAN measurements passed; Linux CI, RELEASE and a full TPC-DS rerun are not claimed for this commit. ### AI Disclosure - Tool: Codex - Generated: Implementation, tests, documentation and this description. - Human review: Pending review of this change.
|
run buildall |
1 similar comment
|
run buildall |
Master added `16: optional TMongoScanDesc mongodb_params` to TTableFormatFileDesc (apache#70327), so `deltalake_params` moves to field 17. Field 17 is unused on master, branch-4.2, branch-4.1, branch-4.0 and branch-3.1. Generated code accesses the field by name, so no Java or C++ code changes. build.sh keeps master's mongodb plugin entries and adds deltalake to both connector lists.
|
run buildall |
### Background The Delta reader can stream file enumeration. While enumeration is in progress, EXPLAIN can show a file-count estimate for scan scheduling. The completed query profile records the selected files and generated splits. ### What problem does this PR solve? Issue Number: apache#69884 Related PR: apache#69904 (Delta reader) PR Type: Test Problem Summary: The basic partition-pruning and Parquet deletion-vector regressions treated the streaming EXPLAIN estimate as the final split count. Read the counters from the completed profile of the query whose rows are checked. ### Details - **Failure:** With the default streaming threshold of 1, the estimate for the basic query was 3 when one split was generated. For the DV byte-range scan the estimate was 1 when five splits were generated. - **Cause:** Split scheduling uses a file-count hint before enumeration, including when one file generates multiple ranges. - **Stability evidence:** The basic check retains the partition fraction. The DV check retains the deletion-vector, row group, page-pruning, payload and row checks. Each scan is located by its unique SQL token and waits for complete Delta enumeration. ### Testing - Built the PR1 FE and regression framework and ran the existing golden outputs for test_deltalake_basic and test_deltalake_parquet_deletion_vector on local ASAN Doris, Hive Metastore and MinIO. - Both suites passed with default batch scheduling and threshold 1, and passed again with only session batch scheduling disabled. - Filesystem and HMS each reported one selected file for both queries, one generated split for the filtered basic query and five for the DV cross-split query. ### Release note None ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Ready ### AI Disclosure - Tool: OpenAI Codex (GPT-6) - Generated: Test changes and commit description - Human review: Pending
### Background Apache Doris PR apache#69904 adds the Delta Lake reader for the latest table state. Delta records each committed change as a numbered version of its transaction log. Reading an older version requires using that version for column binding, file selection and deletion vectors. ### What problem does this PR solve? Issue Number: apache#69884 Related PR: apache#69904 (required base, merge first) PR Type: Feature Problem Summary: Allow an ordinary Delta table reference to select a numeric version using `FOR VERSION AS OF`. This is a stacked draft based on `8ba8e67f29` from apache#69904. GitHub's Files changed tab includes the base PR until it merges. [Review only the changes for version reads](8ba8e67...codex/delta-version-as-of). ### Details - **Change map:** The Delta FE connector accounts for 65 added and 8 removed production lines. Unit tests and fixtures add 136 lines and remove 2. The Spark comparison adds 172 test lines and the framework generated 91 output lines. - **Scope:** Numeric versions, including zero, through Delta filesystem, Delta HMS and the HMS gateway. - **Design:** Load the requested Kernel snapshot, verify the target table UUID and reader features, and retain its schema. Reuse the checked handle for that table and version within the query. The file scan, projected columns and exact COUNT operate on the same version. - **Usage:** `SELECT * FROM catalog.db.t FOR VERSION AS OF 3`. - **Default behavior:** Ordinary reads continue to use the latest snapshot and the configured latest cache. - **Compatibility:** The table's current protocol must be readable when opening its handle. An unavailable log version fails explicitly. Timestamp selection and the history system table remain open in apache#69884. The HMS gateway still uses its existing statistics delegation for CBO estimates. Kernel reads a version while its required log and data files are available. Spark Delta 4.4.1 adds a separate, default-on age guard based on deleted-file retention, which can reject readable data. [Spark implementation](https://github.com/delta-io/delta/blob/v4.4.1/spark/src/main/scala/org/apache/spark/sql/delta/SnapshotManagement.scala#L1656-L1689) ### Testing - Full local macOS arm64 ASAN build with `FE_MAVEN_THREADS=4 ./build.sh --fe --be -j10`. Started the resulting FE and BE and verified a distributed query. - `bash run-fe-ut.sh --run 'DeltaLake*Test,DeltaKernel*Test'` passed. The Delta connector and Delta metastore report 106 tests executed with zero failures, including all six new history tests. Two opt-in diagnostics were skipped. - Generated the golden output for `test_deltalake_version_as_of` via `run-regression-test.sh` and reran the unchanged suite without `-genOut`. Both runs passed with Spark 4.0.0, Delta 4.4.1, Hive Metastore and MinIO. All three catalog routes matched Spark on historical columns, types, ordered rows, counts, column mapping by name/ID, a physical deletion vector and multi-version joins. The new test exercises synchronous and streaming scheduling with the existing default threshold of 1. - The existing `test_deltalake_hive` passed. After rebasing on the split-profile test fix from apache#69904, reran `test_deltalake_basic`, `test_deltalake_parquet_deletion_vector` and `test_deltalake_version_as_of` with default streaming scheduling and `num_files_in_batch_mode=1`. All three passed against their existing golden outputs. ### Release note Delta Lake tables can be queried at a numeric version with `FOR VERSION AS OF`, including the schema and deletion vectors recorded at that version. ### Check List (For Author) - Behavior changed: Yes (see Details). - Does this need documentation: Yes, to accompany the Delta reader website documentation tracked in apache#69884. - Pre-submission status: Locally verified. Draft pending apache#69904. ### AI Disclosure - Tool: OpenAI Codex (GPT-6). - Generated: Implementation, tests and PR description with AI assistance. - Human review: Pending.
|
run buildall |
### Background The M1 Delta reader uses Delta Kernel to plan external scans and the native Parquet reader to execute them. Unity Catalog also coordinates the commits of managed tables, which can be visible before their numbered log files are backfilled. ### What problem does this PR solve? Issue Number: apache#69884 Related PR: apache#69904 (M1 reader) PR Type: Feature Problem Summary: Provide read-only Unity Catalog access alongside filesystem discovery. Retire the branch's dedicated HMS resolver and native Hive delegation. Use UC to authorize table reads and to resolve managed versions. ### Details - **Change map:** Production and build files +933/-342, tests and fixtures +714/-1311, generated regression output +25/-2773, and documentation +49/-3. The removed golden blocks belong to the retired HMS routes. - **Scope:** List UC schemas and Delta tables and read external and managed tables. Reject unsupported governance and storage modes. - **Design:** Carry the UC registration identity through snapshot pinning, caching and statistics. Resolve managed versions through Kernel's UC client. Bind token or OAuth M2M to UC and table READ grants to storage. Use short-lived table credentials on FE and request a grant for each BE scan. Keep credentials out of serialized handles and the UC identity out of BE tasks. - **Usage:** Set deltalake.catalog.type=uc, deltalake.uc.uri and deltalake.uc.catalog. Configured storage credentials can be used for a cloud EXTERNAL table after UC authorizes READ. - **Default behavior:** Filesystem remains the default discovery mode. UC defaults to vended storage credentials and supports S3 sessions. - **Compatibility:** Contains no BE or wire-format changes. Hive rejects Delta-marked Parquet registrations. Other Hive formats retain their routing. Restore the master CAST pushdown interface and use the Delta metadata override for Delta queries. - **Behavior change:** This M1 branch stops accepting the dedicated HMS flavor and stops forwarding Delta tables from Hive. ### Testing Built FE and ASAN BE on macOS. The local BE toolchain required disabling the precompiled header with ENABLE_PCH=OFF. Ran 128 selected FE unit tests, including opt-in million-file planning tests, with zero failures or skips. Ran all 12 Delta P0 suites and checked all 190 retained filesystem output blocks against the prior revision. Generated and reran the seven UC blocks. Spark 4.0 / Delta 4.4.1 and Doris read the same two rows through open-source Unity Catalog 0.7.0 when managed v3 had only a ratified staged commit. Both Doris scheduling modes returned the expected rows and count; INSERT failed. Read a registered external MinIO table with a scoped S3 session through UC and with configured storage credentials. Confirmed the retired HMS flavor, the Hive Delta registration and path-based managed reads were rejected, and an ordinary Hive Parquet table remained readable. Verified masking of both UC secret properties in SHOW CREATE CATALOG. ### Notes for Reviewers Creating a catalog requires CREATE on the catalog resource; changing its properties requires ALTER on that catalog. Such a user can name outbound FE hosts in deltalake.uc.uri and deltalake.uc.oauth.token-uri. The Delta provider declares both SSRF endpoints for the existing operator-configured CREATE/ALTER gate. Inspect DeltaLakeUnityCatalog and the staged-commit fixture for the managed-table read path. ### Release note M1 can read filesystem Delta tables and Unity Catalog external and managed Delta tables. A UC catalog is read-only. ### Check List (For Author) - Behavior changed: Yes (see Details). - Does this need documentation: Yes. Update the M1 website catalog page to cover filesystem and UC, authentication and verified storage modes. - Pre-submission status: Partially verified. Hosted Databricks with a configured Azure Shared Key needs a workspace and authorized table. UC-vended Azure SAS is pending. Linux CI has not run on this commit. ### AI Disclosure - Tool: OpenAI Codex (GPT-6). - Generated: Implementation, tests, fixture, documentation and this description. - Human review: Pending review of this revision.
|
run buildall |
### Background The filesystem and read-only Unity Catalog implementations share the native Delta reader. Both must use Parquet field IDs and interpret partition values from the Delta log. ### What problem does this PR solve? Issue Number: apache#69884 Related PR: apache#69904 (M1 reader) PR Type: Test Problem Summary: Add correctness checks for a reused physical name, a widened partition, an empty partition value and a comma in a logical partition name. Keep the expected rows independent of the Doris results. ### Details - **Scenarios:** Read ten retained tables through filesystem and open-source UC with synchronous and streaming scheduling. Verify that ID mapping distinguishes matching, missing and different IDs, that DATE partitions survive widening to TIMESTAMP_NTZ, that empty partition values are null and that a mapped partition name can contain a comma. - **What existing tests miss:** The existing suites use unambiguous mapped IDs, compatible partition text and simple partition names. Spark and the Delta protocol also differ on the empty STRING fixture. - **Fails when broken:** Checks compare against typed SQL literals and log each mismatch. An untagged Parquet file may produce null or an explicit field-ID error. A wrong row, rejected valid partition or dropped matching row fails the suite. ### Testing Built FE and BE at 84ddcbd. The basic and UC regression suites passed. The new suite ran 56 comparisons against MinIO and local OSS UC: all 20 controls passed, and 36 comparisons reproduced the four open findings. A separate filesystem/UC run over identical local files agreed. Spark 4.0.1 with Delta 4.4.1 independently read the same files. ### Release note None ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Not ready: the four product findings remain open and the new regression is red. ### AI Disclosure - Tool: OpenAI Codex - Generated: Regression checks, fixture registration and setup, and the commit description. - Human review: Pending.
### Background A Delta snapshot specifies whether Parquet columns are identified by their display names, physical names or field IDs. Files written by Spark can carry field IDs in both mapped modes. ### What problem does this PR solve? Issue Number: apache#69884 Related PR: apache#69904 (M1 reader) PR Type: Bugfix Problem Summary: A Delta table requesting field ID 7 could read the value of a same-named field with ID 3. Send the mapping mode with the scan and let the native reader use the column identity selected by the snapshot. ### Details - **Change map:** Production code and Thrift +85/-5, unit tests, regression and fixture metadata +119/-17, generated regression output +85, and fixture documentation +7/-1. All three 608-byte Parquet fixtures reuse the same existing blob. - **Frequency:** An ID-mapped file has a physical name and field ID that identify different table fields, or the file lacks field IDs. - **Root cause:** The scan represented name and ID mapping with the same schema dictionary. BE assigned field IDs to the file by physical name, overwriting its original identifiers. - **Why existing tests missed it:** Spark's ordinary rename fixtures have consistent physical names and IDs. The retained reader-edge regression reproduced the wrong-ID and untagged cases. The new crossed fields also verify that the value reaches the differently named column with the matching ID. - **Fix:** Carry an optional Delta-specific mode on each scan. Preserve Parquet field IDs in ID mode, ignore file IDs when resolving physical names in name mode, and use display names in none mode. - **Other paths and branches:** Filesystem and Unity Catalog use the same Delta reader. A BE executing an older FE plan without the mode keeps the existing behavior. - **Behavior change:** A missing requested ID yields NULL when the file has other field IDs. Reading a data column by ID from an entirely untagged file reports the file path. ### Testing - Built FE and BE on macOS arm64 with ASAN. Delta connector FE tests passed (101 passed, 2 skipped), including both mapped modes and the none-mode scan parameters. The two opt-in scale/profile tests were skipped. - All 18 Delta reader and deletion-vector BE tests passed under ASAN (15 reader, 3 DV), including nested conflicts and a full untagged-file read. The monolithic unit-test build hit an unrelated macOS std::min type mismatch in binary_column_extract_iterator_test.cpp. Used the same BE libraries, test harness and run-be-ut.sh with the Delta test translation unit linked separately. - Compared three Delta tables sharing identical retained Parquet bytes with Spark 4.0.1 and Delta 4.4.1. Before the fix, ID mode returned (703, NULL). After the fix it returns (NULL, 703) through filesystem/MinIO and open-source UC under synchronous and streaming scheduling. Name and none return (703, NULL). - Generated and verified all 28 output blocks in test_deltalake_column_mapping and exercised four untagged-file refusals. The new suite and the existing basic, features, UC and Parquet DV suites passed. On the retained Spark DROP/ADD tables, both mapped modes read NULL for the old row and 900 for the new row. - The combined reader_edges suite passes all field-ID checks. Its 28 outstanding failures out of 56 comparisons concern the separate widened partition, empty partition and comma-in-partition-name findings. ### Release note Doris resolves Delta ID-mapped columns by their Parquet field IDs, including when physical names and IDs disagree. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: No - Pre-submission status: Partially verified: macOS ASAN and the retained Delta suites. The combined reader_edges suite still reproduces three separate partition issues. ### AI Disclosure - Tool: Codex (GPT-6) - Generated: Implementation, tests, fixture metadata, verification and commit description. - Human review: Pending.
### Background The Delta reader checks both the Parquet schema and the table projection for field IDs before reporting a file with no field IDs. ### What problem does this PR solve? Issue Number: apache#69884 Related PR: apache#69904 (M1 reader) PR Type: Refactor Problem Summary: The previous helper name implied that every ID came from Parquet. Projected table fields get their IDs from the Delta snapshot. ### Details - **Moves vs. logic changes:** Rename the private C++ helper and its three uses to `has_data_field_id`. - **Behavior change:** None. - **Compatibility:** The helper has translation-unit scope. - **Performance impact:** None. - **Rollback:** Revert this commit. ### Testing Four focused Delta reader tests passed in a local macOS ASAN build, covering an untagged file, a renamed physical column, nested renames, and conflicting field IDs. The clang-format 16 check and clang-tidy passed. ### Release note None ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Ready ### AI Disclosure - Tool: Codex (GPT-6) - Generated: Helper rename and commit description. - Human review: Pending.
### Background Delta stores each file's partition values as strings in the transaction log. Changing a partition column from DATE to TIMESTAMP_NTZ updates the snapshot schema and leaves old files with date-only strings. ### What problem does this PR solve? Issue Number: apache#69884 Related PR: apache#69904 (M1 reader) PR Type: Bugfix Problem Summary: Reading old partitions failed during planning after Spark widened a DATE partition column to TIMESTAMP_NTZ. Even a query that projected only a data column could fail during partition listing. ### Details - **Frequency:** A snapshot reads an active file with a date-only partition value for a column widened from DATE to TIMESTAMP_NTZ. - **Root cause:** Partition normalization used the current data type without consulting the field's type changes. The timestamp parser requires a time component. - **Why existing tests missed it:** Data-file widening and ordinary timestamp partitions had coverage separately. The reader-edge regression reproduced the planning failure but had no predicate on the old partition. - **Fix:** Consult the current field's recorded date-to-NTZ transition for a date-only value. Parse the date strictly and render its local midnight with the same formatter as a full NTZ timestamp. The predicate converter reuses the same type-change identity. - **Other paths and branches:** Filesystem and read-only UC share the Delta partition normalization path. The change needs no extra log reads or BE support. - **Behavior change:** Old date partitions can be listed, filtered and read after widening. A native NTZ column still requires a timestamp, and an invalid calendar date still raises an error. ### Testing - Built FE and passed Checkstyle on macOS. Delta connector FE tests passed (102 passed, two opt-in tests skipped), including strict date parsing and the native NTZ control. - Reused the two-file Spark table in MinIO filesystem and local open-source UC. On the previous FE, all 20 checks failed during planning. On the fixed FE, all 60 checks across both catalogs, sync/stream scheduling and UTC, Asia/Shanghai and America/Los_Angeles matched Spark. Equality at the old midnight returned id 1, the range excluding it returned id 2, and equality one microsecond later counted zero rows. - Ran test_deltalake_reader_edges. All 20 R4 checks passed, as did 28 controls. Its 20 remaining failures reproduce the open R5 empty-partition and R7 comma-in-column-name issues. - Ran the basic, features, UC, column mapping, skipping and Parquet deletion-vector regression suites. All six passed. ### Release note Doris can read and filter existing Delta DATE partitions after they are widened to TIMESTAMP_NTZ. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: No - Pre-submission status: Partially verified: local macOS validation. The combined reader-edge suite also exercises separate open partition bugs. ### AI Disclosure - Tool: Codex (GPT-6) - Generated: Implementation, tests, validation and commit description. - Human review: Pending.
### Background The Delta connector uses AddFile partition values both to plan files and to supply Doris with the values of partition columns. Delta Kernel can also evaluate partition predicates on the serialized values before returning files. ### What problem does this PR solve? Issue Number: apache#69884 Related PR: apache#69904 (Delta M1 reader) PR Type: Bugfix Problem Summary: A DATE partition serialized as an empty string failed to plan. Kernel 4.4.1 could also discard its file for IS NULL or fail a date comparison before the connector decoded the value. Read that encoding as NULL and use the normalized partitions to select its files. ### Details - **Change map:** 50 implementation lines added and seven removed in the Delta FE plugin, 86 unit-test lines added and two removed, 54 regression suite lines, and 129 generated output lines. - **Frequency:** A Delta transaction log must contain an explicit empty string for a DATE partition. The Spark 4.0.1 / Delta 4.4.1 writes tested locally emitted JSON null for a null DATE. - **Root cause:** The connector passed the empty value to LocalDate.parse. Kernel used the serialized null bit for IS NULL and attempted to parse the empty value for date comparisons. - **Why existing tests missed it:** Ordinary DATE and JSON-null fixtures did not exercise the empty encoding. The existing reader-edge reproducer exposed a projection failure but did not test filtered file selection for this case. - **Fix:** Decode only an empty DATE as NULL. The existing full partition listing records the affected logical DATE columns for that immutable Kernel snapshot. The scan uses normalized partition selection for those columns. Before a complete listing, DATE partition predicates remain residual. A listed clean DATE column retains Kernel pruning. - **Other paths and branches:** This applies to the Delta filesystem and Unity Catalog paths in both scheduling modes. STRING semantics remain under review. The common FE, BE, and other connectors are unchanged. - **Behavior change:** Empty DATE partitions are readable and participate in IS NULL and date comparisons. Malformed non-empty dates still raise an error. ### Testing A local FE build with Checkstyle succeeded. The Delta connector FE tests passed 105 tests with two opt-in tests skipped. On the retained MinIO and open-source Unity Catalog fixtures, 20 pre-fix DATE queries failed. All 40 post-fix DATE, JSON-null and STRING probe queries matched their expected rows. Twelve filtered plans each selected one of two files. The new P0 suite generated and then verified 40 expected output blocks across both catalogs and both scheduling modes. Six other Delta P0 suites passed. The existing reader-edge suite passed 52 of 68 checks. Its 16 remaining failures cover the pending STRING interpretation and partition column names containing commas. All DATE checks passed. ### Release note Doris now reads Delta DATE partitions serialized as an empty string as NULL, including when filtering on the partition column. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: No - Pre-submission status: Partially verified: the local Delta suites above ran. CI has not run on this commit. ### AI Disclosure - Tool: OpenAI Codex (GPT-6) - Generated: Delta connector changes, unit tests, regression suite, and this description. The regression runner generated the expected output. - Human review: Pending.
### Background Doris masks catalog credentials in SHOW CREATE CATALOG. The frontend now discovers connector-specific sensitive keys when it loads each connector provider. ### What problem does this PR solve? Issue Number: apache#69884 Related PR: apache#69904 (M1 Delta reader) PR Type: Build/CI/Chore Problem Summary: The Delta branch added two UC keys to the frontend's masking list before master introduced registration through the connector. Merge master at e04a785 and register the keys through the Delta provider so both credential types remain masked. ### Details - **Affected:** The Delta FE connector. Its security surface includes the sensitive properties declared by its metastore providers and the existing UC endpoints. - **Developer and release impact:** The shared masking implementation matches master. The connector test verifies registration of the bearer token, OAuth client secret and endpoints. - **License and third-party:** None. - **Rollback:** Revert this merge relative to its first parent to restore the previous branch state. ### Testing - The FE build with Checkstyle passed on macOS arm64. Targeted FE unit tests reported 164 passes and two opt-in skips, with no failures or errors. - SHOW CREATE CATALOG masked the UC bearer token and OAuth client secret for newly created catalogs and again after restarting the FE. The catalog name and OAuth client ID remained visible. ### Release note None ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Ready ### AI Disclosure - Tool: Codex (GPT-6) - Generated: Conflict resolution, provider test and merge description. - Human review: Pending.
|
run buildall |
Background
Delta Lake records a table snapshot in its transaction log. Unity Catalog (UC) also coordinates managed-table commits, which can become visible before a numbered log file appears in storage.
What problem does this PR solve?
Issue Number: #69884
Related PR: #70536 (stacked work on historical reads)
PR Type: Feature
Problem Summary: Add native read-only Delta Lake support through a Kernel 4.4.1 FE connector and FileScannerV2's native Parquet reader. M1 has two catalog modes, filesystem and read-only Unity Catalog. Both use the same Delta reader for schema mapping, snapshot planning, pruning and deletion vectors. This revision withdraws the dedicated HMS catalog and native Hive-to-Delta forwarding.
Details
type=deltalake. A filesystem warehouse discovers tables underwarehouse/<database>/<table>. A UC catalog lists its schemas and Delta tables and resolves registered external and managed tables. Unsupported reader features, UC row filters and column masks fail during planning.warehousefor filesystem discovery. For UC, setdeltalake.catalog.type=uc,deltalake.uc.urianddeltalake.uc.catalog. Theconfiguredstorage mode can use the storage owner's credentials for a cloud external table after UC grantsREAD. Configuration and fixture instructions1.Reader features
timestamps/NTZ and the tested type-widening combinations.
across byte-range splits. Legacy native bitmap encoding is rejected.
unfiltered metadata COUNT when all required file statistics are present,
subtracting deletion-vector cardinality. Other cases scan normally.
Fixed latest-version TTL (hits do not extend it), user-scoped snapshot caching,
REFRESH invalidation and lazy streaming split enumeration.
by user/table identity/version and exclude deletion-vector cardinality. A bounded
background pool collects active-file statistics; cold lookups use same-version
CRC bytes when available. Delta reads use the standard FE table-level row-count
cache. Metadata COUNT uses the snapshot's file statistics. Shared real
checkpoint/DV snapshots have concurrent scan coverage.
Streaming sources close before scheduling completion; close failures propagate.
shared-file Hive/Delta benchmark. Normal restarts reuse initialized fixtures.
Delta validates the physical format of every range and fails a query if a file referenced by the snapshot is missing. Time travel, Glue, Variant, writes and column-level NDV/histogram statistics are tracked in later milestones. Delta roadmap
Compatibility: all new Thrift fields are optional.
deltalake_paramsusesTTableFormatFileDescfield 17: master took field 16 formongodb_params(#70327),so the merge of master at
e321ede256moved Delta to 17. Field 17 is unused on master andbranch-4.2/4.1/4.0/3.1 (highest existing IDs: 16/13/12/10/9); no ID is reused and
existing meanings are preserved. Native Delta queries require compatible FE/BE
versions; older BE versions do not implement this reader.
Testing
ENABLE_PCH=OFF. All 128 selected FE unit tests and 12 Delta P0 regression suites passed with zero skips. The 190 retained filesystem result blocks matched the previous revision. The UC suite added seven generated blocks and passed on replay.(1, before)and(3, after), count 2. Deleted id 2 stayed absent. A filesystem read of the managed table and an INSERT through UC both failed as expected.SHOW CREATE CATALOGmasked both UC secret properties.1. Time to the first split was 19.3–22.5 ms and full enumeration took 1.54–1.57 s. The fixture measures metadata planning.Earlier reader builds, test coverage and CI history (before the UC scope change)
Integrated master
5af2df3231infcc32e2885after the original TeamCity COMPILE, FE UT, Cloud UT and BE UT jobs all stopped at the same merge conflicts. Delta now uses the request-based streaming SPI and master's unified scanner; closing sources, stopping failed queries and publishing successful scan profiles follow the current split lifecycle. The standard FE UT reactor now includes Delta.Local integration checks: full FE/Checkstyle and ASAN BE builds passed. Of 859 selected FE tests, 857 passed and two opt-in cases were skipped; the final real SplitAssignment fixture was also rerun (39/39 overlapping lifecycle/producer tests passed). The ASAN BE runner passed all 239 selected tests, including 3 scanner-dispatch/format/missing-file cases, 14 Delta reader and 3 deletion-vector cases. C++ clang-format 16, header hygiene and changed-line clang-tidy passed.
On integration commit
fcc32e2885, COMPILE, FE UT, Cloud UT and BE UT all passed. The separate FE coverage check failed on that commit.The November 3 test change runs the Delta basic and features suites with the deprecated scanner option set to true. At that revision, the Hive gateway and native dispatch tests retained false-setting coverage. Both suites passed generation and comparison against the retained fixtures. Removing 32 duplicate result blocks dropped 526 expected-output lines. All 146 retained blocks were byte-identical. Full CI requested on
338108051b.The row-count change in
3d4899eb94restores master’s FE table-level row-count cache for ordinary reads and removes the extra Hive snapshot-statistics forwarding. Delta keeps versioned logical counts and same-version CRC fallback. An access after the FE refresh interval (10 minutes by default) triggers an asynchronous update. The full FE build and 102 targeted unit tests passed. The statistics p0 passed for filesystem, HMS and Hive. Under the default refresh interval all three routes converged to 3 and 35 rows for the two reference tables. All 24 TPC-DS manifest counts matched and Q1 passed on both readers in the focused p2 run. Full CI requested.CI diagnosis and verification. Benchmark, SQL and scale figures below are historical results at or before measurement head
f0a229f7b9.Docker environment; million-file planning p2 passed; MinIO and ordinary
HDFS 3.2.1 storage p2 passed using the same generated expected output.
75 Delta connector, 2 metastore, 25 FE core and 274 Hive-related tests passed.
clang-format 16, header hygiene and clang-tidy passed. Thirteen physical
Parquet fixtures matched PyArrow; semantic golden fixtures matched Spark.
Metadata COUNT profiles verified zero data reads. Three million-file
source passes completed in a 512 MiB JVM (first split 22–60 ms, enumeration
1.46–1.86 s, sampled heap peak about 200 MiB). This synthetic fixture has
no physical data objects and measures planning/count only. Restart checks
preserved all 1,227 MinIO objects by path, size and SHA-256.
cc12b56819): final full FE buildand Checkstyle; 87 Delta + 2 metastore tests, broad 267 Hive / 114 FE core
tests, and focused 22 Paimon / 7 Iceberg compatibility tests passed. Eight
concurrent scans share one real checkpoint/DV snapshot. Seven Delta p0
suites passed on the final FE build, including the new statistics suite
through filesystem/HMS/Hive; expected output was generated by the official
script and then verified without generation. Background auth/classloader,
invalidation, cancellation, retry, identity/version isolation and missing
stats are covered. Existing 1,227 MinIO objects still match their retained
path/size/SHA-256 manifest.
completion 2.170 s, sampled heap peak 100,255,200 bytes (about 96 MiB).
This synthetic fixture measures metadata planning/statistics, not data scans.
executions passed in one official suite, with three alternating measured
pairs and unchanged controls. All 24 Delta cardinalities matched the
independent fixture manifest before timing. Eight complete diagnostic
profiles and a separate Q4 repeat are retained without replacing the full
samples. All 1,227 data objects still match the retained checksum manifest.
Notes for Reviewers
Outbound HTTP / SSRF. A user with
CREATE_PRIVorALTER_PRIVon a catalog can configure the UC origin (deltalake.uc.uri) and OAuth token endpoint (deltalake.uc.oauth.token-uri). FE contacts these hosts for metadata and authentication. The Delta provider declares both endpoint properties for the CREATE/ALTER SSRF check whensecurity_checker_class_nameis configured. Object-store endpoints also drive FE log/checkpoint reads and BE Parquet/deletion-vector reads through the existing filesystem clients.SELECT_PRIVcan trigger those reads. The connector and its dependencies are packaged with Doris.Cloud verification. Hosted Databricks external tables with the owner's Azure Shared Key still need a live workspace, authorized table and ADLS credentials. UC-vended Azure SAS requires FE/BE read and renewal support. The open-source UC and MinIO results above cover different environments.
Performance. The latest full SF1 comparison on the same Parquet files measured Hive at 240.236 s and Delta at 251.496 s (+4.69% across 103 TPC-DS queries, local ASAN,
num_files_in_batch_mode=1024). At the default of1four targeted queries were compared. A full default-setting run, TPC-H rerun and RELEASE measurements remain open. Full comparison · Default-setting assessmentHistorical TPC-DS and TPC-H measurements and controls
On the November 2 build with the earlier FE statistics path, all 103 TPC-DS queries
passed and the measured gap narrowed from 26.1% to 16.5%. Performance acceptance remains open.
The full rerun uses the same retained SF1 data, ASAN build, query settings and
Q85/Q30 controls. An optional independent-manifest check waits for Delta's
asynchronous statistics before timing and records Hive estimates. Medians still
come from three alternating pairs, excluding first executions. Delta's total
falls 10.46% and Hive's 3.07%; this before/after observation does not isolate the
cause of every timing change.
Separate complete profiles confirm Q43's build side falls from 2.75M rows to
366 and Q72 no longer builds from 11.745M inventory rows. Q22 is still about 40%
slower; Q4/Q11/Q74 lost earlier Delta advantages. A separate Q4 repeat still shows
a 44.2% gap and is excluded from the 103-query aggregate. Detailed samples and
before/after CSV/JSON remain in the local benchmark report; the benchmark README
contains the reproducible setup and result summary.
Earlier baseline, retained below: TPC-H has not been rerun. The following
125-query comparison predates basic statistics; it is not a new result.
All 22 TPC-H and 103 TPC-DS queries completed cross-reader result comparison and
three alternating paired repetitions on the same retained SF1 Parquet objects.
This combines several official regression invocations, with original failures
retained; it is not an unmodified full-suite pass. Both paths use FileScannerV2,
four pipeline tasks, disabled SQL/file caches, UTC, 8 GiB/query and 120 seconds.
These are local relative observations, not RELEASE performance, concurrent
throughput or certified TPC results. TPC-DS uses the repository's SF100 query
parameters on SF1 data; 12 queries have empty results. Excluding them, the
TPC-DS ratio is 1.215.
Explicit controls: Q7 and Q85 use the same fixed join order on both readers after
default Hive plans exceeded memory; Q15 uses an equivalent CTE; Q30 projects the
current generator's review-date string column. Results use the framework's typed
comparator (exact integer/decimal, floating tolerance 1e-8). Q39's original exact
comparison differed by one ULP in one cell; the original difference is retained.
Nine complete profile pairs show different join build sides, runtime-filter
placement and scanner parallelism. For example, Q72's Delta plan builds from
11.745M inventory rows versus Hive's 62,699-row filtered build side; Q43 builds
from 2.750M sales rows versus Hive's 366 date rows. Missing statistics were one observed planning limitation; basic row/size
statistics are now available. Column statistics remain absent, and the full gap
has not been attributed solely to this omission. Q22 variability and small-table streaming startup still
need isolation; performance parity is not accepted yet.
Release note
Apache Doris can read Delta Lake tables from filesystem warehouses and read-only Unity Catalog catalogs. UC supports external Delta tables and catalog-managed snapshots, including commits awaiting log backfill. Native scans apply deletion vectors, schema evolution and Parquet pruning.
Check List (For Author)
Check List (For Reviewer who merge this PR)
AI Disclosure