Skip to content

[feature](external) Support reading Delta Lake tables natively with Delta Kernel - #69904

Open
xylaaaaa wants to merge 52 commits into
apache:masterfrom
xylaaaaa:codex/delta-m1-reader
Open

xylaaaaa wants to merge 52 commits into
apache:masterfrom
xylaaaaa:codex/delta-m1-reader

Conversation

@xylaaaaa

@xylaaaaa xylaaaaa commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Background

Delta Lake records a table snapshot in its transaction log. Unity Catalog (UC) also coordinates managed-table commits, which can become visible before a numbered log file appears in storage.

What problem does this PR solve?

Issue Number: #69884
Related PR: #70536 (stacked work on historical reads)
PR Type: Feature

Problem Summary: Add native read-only Delta Lake support through a Kernel 4.4.1 FE connector and FileScannerV2's native Parquet reader. M1 has two catalog modes, filesystem and read-only Unity Catalog. Both use the same Delta reader for schema mapping, snapshot planning, pruning and deletion vectors. This revision withdraws the dedicated HMS catalog and native Hive-to-Delta forwarding.

Details

  • Change map: The PR adds 5,359 lines across production and build files, 7,996 across tests and fixtures, 2,303 in generated regression results, and 593 in documentation. It also includes 19 binary fixtures. The FE plugin discovers tables and plans scans. The BE applies mapping and deletion vectors in the native Parquet reader.
  • Scope: Read the latest snapshot through type=deltalake. A filesystem warehouse discovers tables under warehouse/<database>/<table>. A UC catalog lists its schemas and Delta tables and resolves registered external and managed tables. Unsupported reader features, UC row filters and column masks fail during planning.
  • Design: UC authorizes table reads. Kernel's UC client resolves managed snapshots, including commits awaiting log backfill. The table registration ID travels with pinned versions and cache keys. Static tokens and OAuth M2M authenticate UC requests. FE refreshes vended S3 storage sessions and requests a fresh table grant when preparing the BE scan.
  • Usage: Set warehouse for filesystem discovery. For UC, set deltalake.catalog.type=uc, deltalake.uc.uri and deltalake.uc.catalog. The configured storage mode can use the storage owner's credentials for a cloud external table after UC grants READ. Configuration and fixture instructions
  • Default behavior: Filesystem is the default catalog, and UC defaults to vended storage credentials. The UC implementation supports table-scoped S3 sessions. The FE batch threshold retains its default of 1.
  • Compatibility: Existing Hive catalogs continue to read ordinary Hive/Parquet tables. A Hive registration marked as Delta is rejected so that directory scanning cannot expose files or rows removed by Delta. Delta applies its CAST-pushdown policy in its own metadata provider.

Reader features

  • Native Parquet scans; none/name/id column mapping, nested schema evolution,
    timestamps/NTZ and the tested type-widening combinations.
  • Portable disk and inline deletion vectors, including file-global row positions
    across byte-range splits. Legacy native bitmap encoding is rejected.
  • Partition/file skipping, native Parquet row-group/page pruning, and exact
    unfiltered metadata COUNT when all required file statistics are present,
    subtracting deletion-vector cardinality. Other cases scan normally.
  • Classic/multipart/V2 JSON and Parquet checkpoints, sidecars and log compaction.
    Fixed latest-version TTL (hits do not extend it), user-scoped snapshot caching,
    REFRESH invalidation and lazy streaming split enumeration.
  • Basic optimizer row/size statistics, added on 2026-11-02: logical rows are cached
    by user/table identity/version and exclude deletion-vector cardinality. A bounded
    background pool collects active-file statistics; cold lookups use same-version
    CRC bytes when available. Delta reads use the standard FE table-level row-count
    cache. Metadata COUNT uses the snapshot's file statistics. Shared real
    checkpoint/DV snapshots have concurrent scan coverage.
  • EXPLAIN/profile diagnostics for snapshot version, files, partitions and splits.
    Streaming sources close before scheduling completion; close failures propagate.
  • Optional Spark/HMS/MinIO fixtures, retained test data, storage regression and a
    shared-file Hive/Delta benchmark. Normal restarts reuse initialized fixtures.

Delta validates the physical format of every range and fails a query if a file referenced by the snapshot is missing. Time travel, Glue, Variant, writes and column-level NDV/histogram statistics are tracked in later milestones. Delta roadmap

Compatibility: all new Thrift fields are optional. deltalake_params uses
TTableFormatFileDesc field 17: master took field 16 for mongodb_params (#70327),
so the merge of master at e321ede256 moved Delta to 17. Field 17 is unused on master and
branch-4.2/4.1/4.0/3.1 (highest existing IDs: 16/13/12/10/9); no ID is reused and
existing meanings are preserved. Native Delta queries require compatible FE/BE
versions; older BE versions do not implement this reader.

Testing

  • On this UC revision, the macOS FE build and ASAN BE build passed. The local BE build used ENABLE_PCH=OFF. All 128 selected FE unit tests and 12 Delta P0 regression suites passed with zero skips. The 190 retained filesystem result blocks matched the previous revision. The UC suite added seven generated blocks and passed on replay.
  • Spark 4.0 with Delta 4.4.1 wrote a managed table through open-source UC 0.7.0. UC had accepted version 3 while the numbered log copy and its optional CRC were withheld. Spark and both Doris scheduling modes returned (1, before) and (3, after), count 2. Deleted id 2 stayed absent. A filesystem read of the managed table and an INSERT through UC both failed as expected.
  • For an external Delta table on MinIO, UC-vended table credentials and explicitly configured storage credentials yielded the same three rows. The vended run gave Doris only the UC configuration and storage transport settings. A separate Hive query read an ordinary Parquet table successfully. SHOW CREATE CATALOG masked both UC secret properties.
  • The retained million-file checkpoint produced 1,000,000 splits in each of three streaming runs at threshold 1. Time to the first split was 19.3–22.5 ms and full enumeration took 1.54–1.57 s. The fixture measures metadata planning.
Earlier reader builds, test coverage and CI history (before the UC scope change)

Integrated master 5af2df3231 in fcc32e2885 after the original TeamCity COMPILE, FE UT, Cloud UT and BE UT jobs all stopped at the same merge conflicts. Delta now uses the request-based streaming SPI and master's unified scanner; closing sources, stopping failed queries and publishing successful scan profiles follow the current split lifecycle. The standard FE UT reactor now includes Delta.

Local integration checks: full FE/Checkstyle and ASAN BE builds passed. Of 859 selected FE tests, 857 passed and two opt-in cases were skipped; the final real SplitAssignment fixture was also rerun (39/39 overlapping lifecycle/producer tests passed). The ASAN BE runner passed all 239 selected tests, including 3 scanner-dispatch/format/missing-file cases, 14 Delta reader and 3 deletion-vector cases. C++ clang-format 16, header hygiene and changed-line clang-tidy passed.

On integration commit fcc32e2885, COMPILE, FE UT, Cloud UT and BE UT all passed. The separate FE coverage check failed on that commit.

The November 3 test change runs the Delta basic and features suites with the deprecated scanner option set to true. At that revision, the Hive gateway and native dispatch tests retained false-setting coverage. Both suites passed generation and comparison against the retained fixtures. Removing 32 duplicate result blocks dropped 526 expected-output lines. All 146 retained blocks were byte-identical. Full CI requested on 338108051b.

The row-count change in 3d4899eb94 restores master’s FE table-level row-count cache for ordinary reads and removes the extra Hive snapshot-statistics forwarding. Delta keeps versioned logical counts and same-version CRC fallback. An access after the FE refresh interval (10 minutes by default) triggers an asynchronous update. The full FE build and 102 targeted unit tests passed. The statistics p0 passed for filesystem, HMS and Hive. Under the default refresh interval all three routes converged to 3 and 35 rows for the two reference tables. All 24 TPC-DS manifest counts matched and Q1 passed on both readers in the focused p2 run. Full CI requested.

CI diagnosis and verification. Benchmark, SQL and scale figures below are historical results at or before measurement head f0a229f7b9.

  • Test
    • Regression test: eight Delta p0 suites passed twice against the optional
      Docker environment; million-file planning p2 passed; MinIO and ordinary
      HDFS 3.2.1 storage p2 passed using the same generated expected output.
    • The basic partition and Parquet deletion-vector suites passed using the default streaming threshold of 1 and passed again with synchronous scheduling. The completed query profiles for filesystem and HMS each showed one file and one split for the filtered partition, and one file and five splits for the cross-split deletion-vector query.
    • Unit Test: 165 ASAN BE tests (including existing Iceberg/Paimon readers),
      75 Delta connector, 2 metastore, 25 FE core and 274 Hive-related tests passed.
    • Manual test: full FE and ASAN BE builds on local macOS ARM64;
      clang-format 16, header hygiene and clang-tidy passed. Thirteen physical
      Parquet fixtures matched PyArrow; semantic golden fixtures matched Spark.
      Metadata COUNT profiles verified zero data reads. Three million-file
      source passes completed in a 512 MiB JVM (first split 22–60 ms, enumeration
      1.46–1.86 s, sampled heap peak about 200 MiB). This synthetic fixture has
      no physical data objects and measures planning/count only. Restart checks
      preserved all 1,227 MinIO objects by path, size and SHA-256.
    • Statistics follow-up (2026-11-02, head cc12b56819): final full FE build
      and Checkstyle; 87 Delta + 2 metastore tests, broad 267 Hive / 114 FE core
      tests, and focused 22 Paimon / 7 Iceberg compatibility tests passed. Eight
      concurrent scans share one real checkpoint/DV snapshot. Seven Delta p0
      suites passed on the final FE build, including the new statistics suite
      through filesystem/HMS/Hive; expected output was generated by the official
      script and then verified without generation. Background auth/classloader,
      invalidation, cancellation, retry, identity/version isolation and missing
      stats are covered. Existing 1,227 MinIO objects still match their retained
      path/size/SHA-256 manifest.
    • Million-file statistics in a 512 MiB JVM: cold lookup 0.491 ms, background
      completion 2.170 s, sampled heap peak 100,255,200 bytes (about 96 MiB).
      This synthetic fixture measures metadata planning/statistics, not data scans.
    • TPC-DS statistics-ready rerun (2026-11-02): all 103 query files / 824
      executions passed in one official suite, with three alternating measured
      pairs and unchanged controls. All 24 Delta cardinalities matched the
      independent fixture manifest before timing. Eight complete diagnostic
      profiles and a separate Q4 repeat are retained without replacing the full
      samples. All 1,227 data objects still match the retained checksum manifest.

Notes for Reviewers

Outbound HTTP / SSRF. A user with CREATE_PRIV or ALTER_PRIV on a catalog can configure the UC origin (deltalake.uc.uri) and OAuth token endpoint (deltalake.uc.oauth.token-uri). FE contacts these hosts for metadata and authentication. The Delta provider declares both endpoint properties for the CREATE/ALTER SSRF check when security_checker_class_name is configured. Object-store endpoints also drive FE log/checkpoint reads and BE Parquet/deletion-vector reads through the existing filesystem clients. SELECT_PRIV can trigger those reads. The connector and its dependencies are packaged with Doris.

Cloud verification. Hosted Databricks external tables with the owner's Azure Shared Key still need a live workspace, authorized table and ADLS credentials. UC-vended Azure SAS requires FE/BE read and renewal support. The open-source UC and MinIO results above cover different environments.

Performance. The latest full SF1 comparison on the same Parquet files measured Hive at 240.236 s and Delta at 251.496 s (+4.69% across 103 TPC-DS queries, local ASAN, num_files_in_batch_mode=1024). At the default of 1 four targeted queries were compared. A full default-setting run, TPC-H rerun and RELEASE measurements remain open. Full comparison · Default-setting assessment

Historical TPC-DS and TPC-H measurements and controls

On the November 2 build with the earlier FE statistics path, all 103 TPC-DS queries
passed and the measured gap narrowed from 26.1% to 16.5%. Performance acceptance remains open.

TPC-DS summed per-query medians Hive Delta Delta / Hive
Before statistics 232.545 s 293.214 s 1.2609
After statistics, 103 queries 225.408 s 262.556 s 1.1648
After, 91 nonempty-result queries 212.654 s 246.057 s 1.1571

The full rerun uses the same retained SF1 data, ASAN build, query settings and
Q85/Q30 controls. An optional independent-manifest check waits for Delta's
asynchronous statistics before timing and records Hive estimates. Medians still
come from three alternating pairs, excluding first executions. Delta's total
falls 10.46% and Hive's 3.07%; this before/after observation does not isolate the
cause of every timing change.

Separate complete profiles confirm Q43's build side falls from 2.75M rows to
366 and Q72 no longer builds from 11.745M inventory rows. Q22 is still about 40%
slower; Q4/Q11/Q74 lost earlier Delta advantages. A separate Q4 repeat still shows
a 44.2% gap and is excluded from the 103-query aggregate. Detailed samples and
before/after CSV/JSON remain in the local benchmark report; the benchmark README
contains the reproducible setup and result summary.

Earlier baseline, retained below: TPC-H has not been rerun. The following
125-query comparison predates basic statistics; it is not a new result.

All 22 TPC-H and 103 TPC-DS queries completed cross-reader result comparison and
three alternating paired repetitions on the same retained SF1 Parquet objects.
This combines several official regression invocations, with original failures
retained; it is not an unmodified full-suite pass. Both paths use FileScannerV2,
four pipeline tasks, disabled SQL/file caches, UTC, 8 GiB/query and 120 seconds.

Local ASAN -O0 measurement Hive sum of per-query medians Delta sum Delta / Hive
TPC-H, 22 queries 134.565 s 96.342 s 0.716
TPC-DS, 103 queries 232.545 s 293.214 s 1.261

These are local relative observations, not RELEASE performance, concurrent
throughput or certified TPC results. TPC-DS uses the repository's SF100 query
parameters on SF1 data; 12 queries have empty results. Excluding them, the
TPC-DS ratio is 1.215.

Explicit controls: Q7 and Q85 use the same fixed join order on both readers after
default Hive plans exceeded memory; Q15 uses an equivalent CTE; Q30 projects the
current generator's review-date string column. Results use the framework's typed
comparator (exact integer/decimal, floating tolerance 1e-8). Q39's original exact
comparison differed by one ULP in one cell; the original difference is retained.

Nine complete profile pairs show different join build sides, runtime-filter
placement and scanner parallelism. For example, Q72's Delta plan builds from
11.745M inventory rows versus Hive's 62,699-row filtered build side; Q43 builds
from 2.750M sales rows versus Hive's 366 date rows. Missing statistics were one observed planning limitation; basic row/size
statistics are now available. Column statistics remain absent, and the full gap
has not been attributed solely to this omission. Q22 variability and small-table streaming startup still
need isolation; performance parity is not accepted yet.

Release note

Apache Doris can read Delta Lake tables from filesystem warehouses and read-only Unity Catalog catalogs. UC supports external Delta tables and catalog-managed snapshots, including commits awaiting log backfill. Native scans apply deletion vectors, schema evolution and Parquet pruning.

Check List (For Author)

  • Behavior changed: Yes. Adds native Delta reads through filesystem and UC, with versioned table statistics and scan diagnostics.
  • Does this need documentation: Yes. The English and Chinese website pages for the two catalog modes and a full website build are pending.
  • Pre-submission status: Partially verified. Local test results are listed above. Linux CI on the UC commit, maintainer review, the remaining cloud storage matrix and performance acceptance are open. M1 acceptance checklist

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

AI Disclosure

  • Tool: OpenAI Codex (GPT-6).
  • Generated: The UC revision, its tests and fixture, and this PR description update.
  • Human review: Maintainer review of the current head is pending.

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@xylaaaaa xylaaaaa mentioned this pull request Oct 30, 2026
5 of 36 tasks
@xylaaaaa

Copy link
Copy Markdown
Contributor Author

run buildall

xylaaaaa pushed a commit to xylaaaaa/doris that referenced this pull request Nov 2, 2026
### What problem does this PR solve?

Issue Number: close apache#69884

Related PR: apache#69904

Problem Summary: Ordinary external-table queries only consulted the table-keyed
row-count cache, which could keep a cold estimate after versioned connector
statistics completed. Prefer statistics at the statement's pinned snapshot for
ordinary and versioned queries, reuse the existing size/row-width fallback, and
forward the pinned snapshot through the Hive sibling gateway. This covers STAT-02.

### Release note

External-table optimizer estimates can use completed statistics for the query's
pinned snapshot without waiting for the table-only row-count cache to refresh.

### Check List (For Author)

- Test: Full FE build and Checkstyle; 114 FE core and 267 Hive tests; focused
  compatibility run including 22 Paimon and 7 Iceberg tests; seven Delta p0 suites
  on the combined change, including new filesystem/HMS/Hive cardinality coverage.
- Behavior changed: Yes, ordinary pinned queries prefer snapshot statistics.
- Does this need documentation: Yes, Delta connector README and prepared EN/ZH
  website patch are updated in the accompanying statistics change; website PR pending.
xylaaaaa pushed a commit to xylaaaaa/doris that referenced this pull request Nov 2, 2026
### What problem does this PR solve?

Issue Number: close apache#69884

Related PR: apache#69904

Problem Summary: The native Delta connector did not supply basic table statistics
to the optimizer. Cache logical row counts by user, table identity and version,
summing active AddFile numRecords minus deletion-vector cardinality. Collect in a
bounded background pool and return same-version CRC size while cold; missing row
statistics stay unknown. Shared cache invalidation cancels pending work, and exact
COUNT remains independent of optimizer estimates. Add shared-snapshot concurrency,
real checkpoint/DV, lifecycle, authentication and million-file tests (STAT-01/03),
plus filesystem/HMS/Hive regression coverage and connector documentation.

### Release note

Native Delta Lake catalogs and Hive gateway queries gain basic versioned row/size
statistics with asynchronous collection and a CRC size fallback.

### Check List (For Author)

- Test: Full FE build and Checkstyle; 87 Delta and 2 metastore tests, including the
  opt-in million-file fixture in a 512 MiB JVM; focused compatibility/authentication
  rerun; seven Delta p0 suites passed on the final build. New .out generated by the
  official regression script and then verified without generation. Preserved all
  1,227 MinIO objects, checked by path, size and SHA-256.
- Behavior changed: Yes, Delta exposes basic optimizer statistics. Exact COUNT
  semantics and Delta data are unchanged; column NDV/histograms are not supplied.
- Does this need documentation: Yes, connector README included; local design,
  tasklist and EN/ZH website patch updated. Website PR/full site build pending.
@xylaaaaa

xylaaaaa commented Nov 2, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

xylaaaaa pushed a commit to xylaaaaa/doris that referenced this pull request Nov 2, 2026
…stics

### What problem does this PR solve?

Issue Number: close apache#69884

Related PR: apache#69904

Problem Summary: Delta table row counts load asynchronously, so a post-statistics
benchmark needs to distinguish ready statistics from a cold metadata lookup.
Add an optional retained-manifest check that waits for Delta EXPLAIN cardinalities
before timing and records Hive estimates. Keep the existing query texts, common
controls, result comparator and three alternating measured pairs unchanged.
Document the complete local TPC-DS rerun: the summed median gap narrows from
26.1% to 16.5%, with remaining regressions retained rather than replaced by
focused samples. No production code or fixture data changes.

### Release note

None

### Check List (For Author)

- Test: Official regression script passed all 103 TPC-DS queries (824 executions)
  with matching results; all 24 Delta cardinalities match the independent manifest.
  A separate Q4 rerun passed, and eight complete diagnostic profiles were retained.
  All 1,227 MinIO objects match their original path/size/SHA-256 before and after.
  Source whitespace checks and code-review checkpoints passed.
- Behavior changed: No production change. Optional benchmark warmup is disabled
  unless deltalakeBenchmarkStatisticsManifest is provided.
- Does this need documentation: Yes, benchmark README includes warmup usage,
  before/after measurements, controls and remaining performance limitations.
@xylaaaaa

xylaaaaa commented Nov 2, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

1 similar comment
@xylaaaaa

xylaaaaa commented Nov 2, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

xylaaaaa pushed a commit to xylaaaaa/doris that referenced this pull request Nov 2, 2026
### What problem does this PR solve?

Issue Number: close apache#69884

Related PR: apache#69904

Problem Summary: COMPILE, FE UT, Cloud UT and BE UT all failed while merging
the PR into master 5af2df3, before their build or test commands ran.
Keep master's unified FileScannerV2 and split-producer lifecycle. Route Delta
through the request-based streaming SPI, honor cancellation and partition
selection, and validate the physical format on every Delta scan range.
Propagate missing snapshot files and scheduling errors. Include Delta in the
default FE unit-test reactor so the CI invocation exercises the connector.

### Release note

Delta scans fail if a committed data or deletion-vector file cannot be found,
instead of silently skipping a range when ignore_not_found_file is enabled.

### Check List (For Author)

- Test: Full FE and ASAN BE builds; standard FE unit-test selection (857
  passed, 2 opt-in cases skipped); FE Checkstyle, clang-format 16 and BE
  header hygiene. ASAN BE unit tests and a final FE lifecycle fixture
  recheck were still running or queued at commit time.
- Behavior changed: Yes. Delta uses master's common scanner and streamed
  split lifecycle and fails on missing snapshot files.
- Does this need documentation: Yes (connector README updated in this PR;
  the end-user documentation PR is tracked in apache#69884).
@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 100.00% (17/17) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28313 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fcc32e28850c5f1f60f583d2613fa390d1a8f7e7, data reload: false

------ Round 1 ----------------------------------
============================================
q1	16629	5111	5029	5029
q2	2107	307	245	245
q3	9910	1360	824	824
q4	4554	466	337	337
q5	7327	795	528	528
q6	186	181	150	150
q7	748	786	599	599
q8	9441	1333	1050	1050
q9	5551	4215	4154	4154
q10	6831	1294	1031	1031
q11	438	245	227	227
q12	639	393	279	279
q13	17923	3589	3038	3038
q14	282	274	248	248
q15	q16	713	711	667	667
q17	1183	748	564	564
q18	7050	6217	6152	6152
q19	1104	943	594	594
q20	385	342	228	228
q21	4996	2116	2240	2116
q22	345	296	253	253
Total cold run time: 98342 ms
Total hot run time: 28313 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5600	5593	5634	5593
q2	683	557	546	546
q3	5209	5387	5016	5016
q4	1990	2141	1575	1575
q5	5276	5171	5283	5171
q6	341	204	147	147
q7	2421	2065	1866	1866
q8	3007	2665	2658	2658
q9	8151	8077	8022	8022
q10	3729	3685	3451	3451
q11	603	447	409	409
q12	707	697	521	521
q13	3492	3844	3278	3278
q14	317	315	283	283
q15	q16	696	743	653	653
q17	8268	7277	6880	6880
q18	13074	12254	12877	12254
q19	895	767	782	767
q20	2203	2157	1932	1932
q21	5670	4548	4676	4548
q22	518	469	422	422
Total cold run time: 72850 ms
Total hot run time: 65992 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 151400 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fcc32e28850c5f1f60f583d2613fa390d1a8f7e7, data reload: false

query5	4329	604	436	436
query6	435	208	195	195
query7	4815	438	234	234
query8	343	175	154	154
query9	8781	3139	3144	3139
query10	470	312	264	264
query11	5980	3844	3600	3600
query12	147	101	86	86
query13	1279	460	330	330
query14	6093	3911	3618	3618
query14_1	3441	3417	3393	3393
query15	208	199	182	182
query16	947	423	408	408
query17	842	668	532	532
query18	2422	427	312	312
query19	197	161	136	136
query20	83	85	82	82
query21	208	128	106	106
query22	12888	13175	12799	12799
query23	13500	12872	12378	12378
query23_1	12628	12629	12461	12461
query24	7230	821	475	475
query24_1	494	486	498	486
query25	535	419	369	369
query26	1266	255	144	144
query27	2780	432	272	272
query28	4531	2044	2057	2044
query29	1497	611	457	457
query30	300	221	184	184
query31	1018	930	816	816
query32	135	93	89	89
query33	528	311	244	244
query34	899	851	510	510
query35	730	747	639	639
query36	823	801	713	713
query37	130	99	82	82
query38	1767	1727	1651	1651
query39	797	814	748	748
query39_1	729	738	742	738
query40	215	116	101	101
query41	77	67	67	67
query42	88	85	86	85
query43	351	375	317	317
query44	1301	707	712	707
query45	181	189	164	164
query46	848	958	563	563
query47	2992	2787	2811	2787
query48	309	309	268	268
query49	560	385	287	287
query50	681	287	219	219
query51	10676	10433	10579	10433
query52	78	79	67	67
query53	181	215	156	156
query54	242	190	176	176
query55	68	66	60	60
query56	224	219	200	200
query57	1620	1626	1572	1572
query58	286	264	257	257
query59	2197	2224	2035	2035
query60	295	238	211	211
query61	153	147	137	137
query62	381	332	306	306
query63	190	161	170	161
query64	2678	964	785	785
query65	3319	3283	3262	3262
query66	1743	412	296	296
query67	20157	20259	21602	20259
query68	3090	1006	608	608
query69	377	293	250	250
query70	889	780	771	771
query71	303	235	208	208
query72	2676	2575	2227	2227
query73	504	521	317	317
query74	4601	4499	4345	4345
query75	2306	2308	1928	1928
query76	2143	1003	603	603
query77	345	361	274	274
query78	9740	9645	9129	9129
query79	1039	860	531	531
query80	1226	423	360	360
query81	567	332	272	272
query82	572	132	111	111
query83	299	193	186	186
query84	310	127	105	105
query85	855	431	375	375
query86	416	238	236	236
query87	1983	1959	1804	1804
query88	3827	2778	2772	2772
query89	329	281	242	242
query90	1841	168	165	165
query91	159	153	132	132
query92	110	89	85	85
query93	1090	997	576	576
query94	633	312	270	270
query95	585	324	366	324
query96	681	490	236	236
query97	2461	2458	2306	2306
query98	159	151	152	151
query99	766	756	654	654
Total cold run time: 234945 ms
Total hot run time: 151400 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 25.2 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fcc32e28850c5f1f60f583d2613fa390d1a8f7e7, data reload: false

query1	0.00	0.00	0.01
query2	0.14	0.08	0.08
query3	0.34	0.13	0.13
query4	1.61	0.13	0.14
query5	0.28	0.26	0.25
query6	1.15	0.65	0.64
query7	0.04	0.01	0.01
query8	0.09	0.07	0.08
query9	0.46	0.39	0.37
query10	0.58	0.60	0.58
query11	0.31	0.18	0.17
query12	0.32	0.19	0.21
query13	0.51	0.52	0.51
query14	0.94	0.93	0.92
query15	0.66	0.58	0.57
query16	0.36	0.38	0.37
query17	1.04	1.03	1.05
query18	0.28	0.28	0.27
query19	1.96	1.82	1.90
query20	0.02	0.02	0.01
query21	15.39	0.36	0.30
query22	4.93	0.12	0.12
query23	15.88	0.49	0.30
query24	1.99	0.58	0.39
query25	0.14	0.10	0.10
query26	0.79	0.27	0.21
query27	0.10	0.10	0.10
query28	3.46	0.83	0.39
query29	12.59	4.43	3.39
query30	0.35	0.18	0.14
query31	2.76	0.57	0.32
query32	3.23	0.61	0.49
query33	2.94	3.00	3.04
query34	15.90	4.09	3.44
query35	3.40	3.41	3.38
query36	0.64	0.49	0.47
query37	0.12	0.09	0.09
query38	0.07	0.07	0.07
query39	0.06	0.06	0.05
query40	0.19	0.17	0.17
query41	0.11	0.05	0.06
query42	0.06	0.06	0.06
query43	0.06	0.06	0.05
Total cold run time: 96.25 s
Total hot run time: 25.2 s

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 82.35% (28/34) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 77.70% (35894/46198)
Line Coverage 63.04% (403760/640441)
Region Coverage 59.67% (341911/572969)
Branch Coverage 60.74% (158134/260349)

xylaaaaa pushed a commit to xylaaaaa/doris that referenced this pull request Nov 3, 2026
### What problem does this PR solve?

Issue Number: close apache#69884

Related PR: apache#69904

Problem Summary: File scans always use FileScannerV2. The basic and features
Delta suites still exercised the same reader twice for both filesystem and
HMS catalogs. Run them with enable_file_scanner_v2=true and preserve the
retained query tags. The dedicated Hive gateway and native reader dispatch
checks still exercise the deprecated false setting.

### Release note

None

### Check List (For Author)

- Test: Regenerated expected outputs with run-regression-test.sh and reran
  test_deltalake_basic and test_deltalake_features in comparison mode.
  Both suites passed both runs. All 146 retained query output blocks were
  byte-identical. Removed 32 redundant blocks (526 lines).
- Behavior changed: No
- Does this need documentation: No
@xylaaaaa

xylaaaaa commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.65% (2101/2741)
Line Coverage 66.07% (38830/58772)
Region Coverage 53.84% (36475/67743)
Branch Coverage 57.28% (11870/20722)

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28448 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 338108051bffd01074b8d4f488da1211b44c5d00, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17392	5197	5076	5076
q2	2118	313	251	251
q3	10161	1386	838	838
q4	4672	462	333	333
q5	7439	801	524	524
q6	182	185	154	154
q7	733	797	605	605
q8	9253	1375	1095	1095
q9	5597	4213	4197	4197
q10	6831	1293	1022	1022
q11	424	244	223	223
q12	638	407	291	291
q13	18105	3695	3038	3038
q14	272	277	259	259
q15	q16	721	714	663	663
q17	1213	750	569	569
q18	7004	6177	6116	6116
q19	1105	970	579	579
q20	382	336	231	231
q21	4797	2128	2247	2128
q22	348	294	256	256
Total cold run time: 99387 ms
Total hot run time: 28448 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5643	5584	5621	5584
q2	688	581	533	533
q3	5183	5372	4929	4929
q4	1997	2141	1561	1561
q5	5242	5177	5553	5177
q6	268	210	150	150
q7	2350	2022	1836	1836
q8	3009	2682	2730	2682
q9	8094	8005	7934	7934
q10	3761	3910	3205	3205
q11	618	462	406	406
q12	680	704	525	525
q13	3668	3894	3291	3291
q14	345	325	283	283
q15	q16	672	716	642	642
q17	8230	7187	6842	6842
q18	12967	12156	12976	12156
q19	908	787	786	786
q20	2157	2151	1942	1942
q21	5564	4560	4636	4560
q22	496	454	391	391
Total cold run time: 72540 ms
Total hot run time: 65415 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28753 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 338108051bffd01074b8d4f488da1211b44c5d00, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17608	5156	5027	5027
q2	2120	320	257	257
q3	10230	1398	844	844
q4	4675	461	346	346
q5	7435	793	531	531
q6	184	188	152	152
q7	738	814	604	604
q8	9253	1310	1089	1089
q9	5453	4196	4179	4179
q10	6822	1294	1035	1035
q11	430	248	226	226
q12	653	411	277	277
q13	18117	3665	3052	3052
q14	280	272	249	249
q15	q16	704	712	658	658
q17	1196	753	578	578
q18	7116	6180	6124	6124
q19	1104	971	594	594
q20	384	338	223	223
q21	5634	2496	2425	2425
q22	392	330	283	283
Total cold run time: 100528 ms
Total hot run time: 28753 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5981	5871	6024	5871
q2	741	619	590	590
q3	5685	5787	5265	5265
q4	2022	2169	1571	1571
q5	5698	5917	5694	5694
q6	246	189	148	148
q7	2264	2015	1789	1789
q8	2965	2596	2607	2596
q9	7929	7535	7492	7492
q10	3649	3587	3125	3125
q11	575	406	410	406
q12	663	663	517	517
q13	3373	3637	3055	3055
q14	284	299	261	261
q15	q16	658	695	612	612
q17	7743	7051	6814	6814
q18	12877	12223	12874	12223
q19	901	787	791	787
q20	2113	2122	1928	1928
q21	5633	4543	4644	4543
q22	482	451	404	404
Total cold run time: 72482 ms
Total hot run time: 65691 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 152082 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 338108051bffd01074b8d4f488da1211b44c5d00, data reload: false

query5	4305	617	451	451
query6	428	217	199	199
query7	4806	453	229	229
query8	326	170	153	153
query9	8791	3262	3255	3255
query10	482	304	275	275
query11	6066	3891	3576	3576
query12	151	91	86	86
query13	1270	444	326	326
query14	6143	3920	3730	3730
query14_1	3443	3411	3422	3411
query15	204	213	183	183
query16	956	425	404	404
query17	846	659	531	531
query18	2416	423	308	308
query19	200	171	136	136
query20	86	85	84	84
query21	210	134	112	112
query22	12909	13000	12795	12795
query23	13484	12902	12483	12483
query23_1	12578	12614	12680	12614
query24	7363	843	468	468
query24_1	480	499	502	499
query25	504	391	338	338
query26	1250	254	133	133
query27	2778	437	282	282
query28	4612	2048	2061	2048
query29	1543	589	439	439
query30	299	221	190	190
query31	1000	929	810	810
query32	151	90	87	87
query33	519	285	234	234
query34	896	853	502	502
query35	720	747	625	625
query36	816	802	736	736
query37	129	106	82	82
query38	1784	1717	1713	1713
query39	788	786	771	771
query39_1	737	740	739	739
query40	229	115	92	92
query41	65	64	64	64
query42	89	82	81	81
query43	348	368	323	323
query44	1308	705	718	705
query45	177	180	162	162
query46	837	960	567	567
query47	3019	2897	2785	2785
query48	298	318	228	228
query49	571	416	290	290
query50	700	268	220	220
query51	10672	10689	10542	10542
query52	76	81	71	71
query53	184	213	151	151
query54	266	199	177	177
query55	70	68	67	67
query56	241	219	196	196
query57	1640	1643	1560	1560
query58	292	259	254	254
query59	2161	2132	1956	1956
query60	283	227	225	225
query61	152	154	147	147
query62	385	334	281	281
query63	189	159	163	159
query64	2836	1080	883	883
query65	3367	3314	3283	3283
query66	1796	428	314	314
query67	20563	20303	20160	20160
query68	3307	1029	621	621
query69	416	309	254	254
query70	824	796	773	773
query71	308	242	221	221
query72	2932	2662	2329	2329
query73	531	531	307	307
query74	4607	4516	4367	4367
query75	2357	2296	1907	1907
query76	2402	1008	589	589
query77	359	355	281	281
query78	9865	9809	9131	9131
query79	996	863	539	539
query80	1215	416	350	350
query81	598	333	275	275
query82	620	127	103	103
query83	305	194	183	183
query84	313	128	95	95
query85	889	459	379	379
query86	396	238	235	235
query87	2005	1962	1806	1806
query88	3841	2790	2785	2785
query89	360	285	240	240
query90	1901	169	170	169
query91	165	149	132	132
query92	102	96	96	96
query93	1079	1010	581	581
query94	653	289	266	266
query95	578	333	377	333
query96	699	486	243	243
query97	2430	2422	2313	2313
query98	166	158	148	148
query99	773	759	641	641
Total cold run time: 236972 ms
Total hot run time: 152082 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.35 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 338108051bffd01074b8d4f488da1211b44c5d00, data reload: false

query1	0.01	0.01	0.01
query2	0.09	0.05	0.04
query3	0.26	0.08	0.09
query4	1.61	0.12	0.11
query5	0.27	0.24	0.24
query6	1.17	0.68	0.68
query7	0.04	0.01	0.01
query8	0.05	0.04	0.04
query9	0.39	0.35	0.35
query10	0.57	0.57	0.57
query11	0.19	0.14	0.15
query12	0.17	0.14	0.15
query13	0.49	0.51	0.48
query14	1.02	1.03	1.01
query15	0.62	0.61	0.60
query16	0.36	0.36	0.35
query17	1.12	1.12	1.15
query18	0.20	0.19	0.20
query19	2.12	2.05	2.02
query20	0.02	0.02	0.01
query21	15.51	0.20	0.13
query22	4.89	0.06	0.05
query23	16.26	0.33	0.13
query24	2.86	0.52	0.37
query25	0.12	0.06	0.05
query26	1.23	0.21	0.17
query27	0.04	0.04	0.05
query28	3.40	0.69	0.35
query29	12.49	4.15	3.23
query30	0.29	0.13	0.09
query31	2.77	0.55	0.33
query32	3.22	0.60	0.50
query33	3.25	3.21	3.23
query34	15.33	4.09	3.37
query35	3.35	3.36	3.31
query36	0.55	0.45	0.45
query37	0.13	0.08	0.08
query38	0.05	0.05	0.03
query39	0.04	0.04	0.03
query40	0.16	0.15	0.16
query41	0.09	0.04	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.03
Total cold run time: 96.88 s
Total hot run time: 24.35 s

xylaaaaa pushed a commit to xylaaaaa/doris that referenced this pull request Nov 3, 2026
### What problem does this PR solve?

Issue Number: close apache#69884

Related PR: apache#69904

Problem Summary: Commit 598756e made ordinary plugin table reads resolve
their pinned snapshot statistics on the planning thread. This changed the
row-count path for Iceberg, Paimon and Hive as part of a Delta feature.
Revert that commit's FE table and Hive statistics changes. Delta still
collects logical rows asynchronously per version and supplies same-version
CRC bytes when its count is unavailable. Exercise cold estimates in p0
and wait for the standard FE cache refresh before timing the p2 benchmark.

### Release note

Ordinary catalog queries use the standard FE external-table row-count cache.
Delta estimates in EXPLAIN may retain their initial value until an access
triggers a refresh (10 minutes by default). Exact COUNT is independent.

### Check List (For Author)

- Test: Full FE build and Checkstyle. All 102 targeted FE tests passed.
  test_deltalake_statistics passed for filesystem, HMS and Hive. With the
  default refresh interval, all 24 TPC-DS manifest counts matched and Q1
  passed for both readers in test_deltalake_benchmark.
- Behavior changed: Yes. Restore master's row-count policy for ordinary
  plugin table reads and its original Hive statistics delegation.
- Does this need documentation: Yes (connector README and benchmark guide
  updated in this PR; website documentation is tracked in apache#69884).
@xylaaaaa

xylaaaaa commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

xylaaaaa added a commit to xylaaaaa/doris that referenced this pull request Nov 3, 2026
### What problem does this PR solve?

Issue Number: close apache#69884

Related PR: apache#69904

Problem Summary: Merging master 7acfd9b left two Delta gaps. First,
apache#69717 replaced table_reader.h's full ShardedKVCache definition with a
forward declaration, but deltalake_reader_test.cpp constructs a
ShardedKVCache and relied on the transitive include. Include
format/format_common.h directly, as hudi_reader_test.cpp does. Second,
apache#69300 made Parquet readers in one scan share their profile counters
through TableReader's parquet profile cache. Hudi, Iceberg and the base
reader pass that cache, but DeltaReader still built its ParquetReader
without it, so every Delta file created its own set of counters. Pass
the cache, and keep binary-UUID preservation disabled as before.

### Release note

None

### Check List (For Author)

- Test: No local build. The merge was conflict-free. Both changes follow
  the matching master code. COMPILE and BE UT are left to CI.
- Behavior changed: No. Delta scan profiles reuse Parquet counters across files, as other formats do.
- Does this need documentation: No
@xylaaaaa

xylaaaaa commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.65% (2101/2741)
Line Coverage 66.03% (38808/58772)
Region Coverage 53.78% (36430/67743)
Branch Coverage 57.23% (11859/20722)

xylaaaaa added a commit to xylaaaaa/doris that referenced this pull request Nov 5, 2026
### What problem does this PR solve?

Issue Number: close apache#69884

Related PR: apache#69904

Problem Summary: Extracting decode_portable_deletion_vector_payload for
inline Delta DVs left the framed Iceberg/Delta decoder checking the
portable magic twice. Its first check runs before the CRC, and the
payload function checked it again. The magic constant was also defined
twice. Move the constant, the magic check and the Roaring decode into
file-local helpers. The framed decoder again validates magic, then
CRC, then decodes the bitmap once, with the same offsets and length it
used before this PR. The inline payload decoder keeps its own size
check and uses the same helpers.

### Release note

None

### Check List (For Author)

- Test: No local build. Existing Iceberg DV unit tests cover the magic
  and CRC rejection order. The Delta reader unit tests compare inline
  and on-disk decoding. Both are left to CI.
- Behavior changed: No. Iceberg DV decoding is again identical to master's.
- Does this need documentation: No
xylaaaaa added a commit to xylaaaaa/doris that referenced this pull request Nov 5, 2026
…n validation

### What problem does this PR solve?

Issue Number: close apache#69884

Related PR: apache#69904

Problem Summary: validate_file_scan_support is the shared range gate for
table markers that are no longer supported. The Delta integration also
added a rule there that rejects non-Parquet Delta ranges. That state
cannot occur. The Delta protocol stores data only as Parquet, and
DeltaLakeScanRange always sends FORMAT_PARQUET. DeltaReader also asserts
Parquet in create_file_reader. That runs for every data file, after
prepare_split resets the reader format from the current range.
Remove the shared rule and its unit test, so validate_file_scan_support
is identical to master again. Keep the Delta entry in
can_ignore_not_found_file. A missing file in a committed Delta snapshot
must still fail the scan, as for Iceberg.

### Release note

None

### Check List (For Author)

- Test: No local build. The remaining Delta dispatch and missing-file
  unit tests are left to CI.
- Behavior changed: No for FE-generated plans. A malformed non-Parquet
  Delta range now fails the DeltaReader assertion instead of returning
  NotSupported.
- Does this need documentation: No
xylaaaaa added a commit to xylaaaaa/doris that referenced this pull request Nov 5, 2026
… the Delta PR

### What problem does this PR solve?

Issue Number: close apache#69884

Related PR: apache#69904

Problem Summary: The Delta integration widened two fe-core catch blocks
to Error. One is in executeWithProfileFinalization, on the batch planning
path. The other is in the streaming profile finalizer. A lazily loaded
plugin dependency then reported NoClassDefFoundError like any other
failure.

These are general hardening changes for every plugin connector, and
Delta does not depend on them. Master's SplitProducer already catches
Throwable while it iterates, closes the source and runs completion. The
scan still fails without hanging and still calls finishSchedule. Of the
four streaming tests added with the change, three cover existing master
behavior. Only the profile-phase Error case needs the widened catch.
Restore PluginDrivenScanNode and its scan profile test to master so the
Delta PR leaves fe-core unchanged. The hardening can be proposed on its
own.

### Release note

None

### Check List (For Author)

- Test: No local build. Both files are identical to master; left to CI.
- Behavior changed: No relative to master. A Delta-only branch would have
  reported a profile-phase Error as a UserException.
- Does this need documentation: No
…catalog

### Background

An HMS catalog can delegate different tables in one statement to the Delta and Iceberg connectors. Its scan nodes share catalog metadata, while Delta's scan provider rejects CAST-bearing file-pruning predicates and Iceberg retains its existing policy.

### What problem does this PR solve?

Issue Number: apache#69884
Related PR: None
PR Type: Test

Problem Summary:
The existing Delta CAST regression tested each catalog entrance separately and did not mix Delta and Iceberg scans in one statement. Add mixed HMS queries and shared-metadata unit coverage so a catalog-wide CAST opt-out or a policy leaking between table scans cannot pass unnoticed.

### Details

- **Scenarios:** UNION ALL and JOIN across Delta and Iceberg tables in one HMS catalog, both table orders, synchronous and streaming scans, and an independent ordinary Delta predicate. The FE test alternates two scan nodes sharing the same metadata instance in both orders.
- **What existing tests miss:** A blanket HMS CAST opt-out returns correct rows for these fixtures but changes Iceberg's file-pruning behavior. Separate-catalog queries do not exercise the shared metadata boundary.
- **Fails when broken:** Pushing the Delta CAST loses the matching 123.45 row. Disabling Iceberg's existing CAST pushdown removes its EXPLAIN predicate and makes the synchronous scan select both files instead of one. The ordinary Delta predicate must still prune one of its two files.

### Testing

- Local run-regression-test.sh: test_deltalake_hms_mixed_cast with -genOut and an ordinary rerun both passed. Independently checked all 14 generated result blocks; the golden SHA-256 stayed unchanged.
- Local run-regression-test.sh: test_deltalake_cast_pruning passed all 24 result blocks and test_deltalake_basic passed all 36 result blocks.
- Local run-fe-ut.sh: 108 tests across PluginDrivenScanNodeCastPredicateTest, Hive routing, Delta, Iceberg, Paimon, JDBC and SPI passed, with no failures, errors or skipped tests.
- Local FE Checkstyle reported zero violations; git diff --check passed.

### Notes for Reviewers

Iceberg fixtures use integral decimal values to test its existing policy without relying on the separately known lossy CAST bug. Iceberg publishes SDK file counters only for synchronous planning; both modes check its EXPLAIN predicate and actual query results. Production code is unchanged in this commit.

### Release note

None

### Check List (For Author)

- Behavior changed: No
- Does this need documentation: No
- Pre-submission status: Ready

### AI Disclosure

- Tool: OpenAI Codex
- Generated: Tests, generated regression output, and this description
- Human review: Pending
@xylaaaaa

xylaaaaa commented Nov 6, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

chenjunwei and others added 2 commits November 6, 2026 11:56
### Background

Delta's scan provider rejects CAST-bearing predicates before the shared converter strips their casts. The mixed HMS regression reads Delta and Iceberg tables in one statement, and originally asserted Iceberg's existing CAST pushdown behavior as well as query results.

### What problem does this PR solve?

Issue Number: apache#69884
Related PR: None
PR Type: Test

Problem Summary:
The Delta connector unit tests did not directly guard its scan provider's CAST rejection. Meanwhile, the mixed regression would reject a future Iceberg correctness fix that disables unsafe CAST pushdown. Add a real-provider unit assertion and keep the mixed regression focused on query results and Delta pruning.

### Details

- **Scenarios:** Obtain the real scan provider from the Delta connector and assert that it implements ConnectorPushdownOps and returns false for CAST pushdown. Keep the mixed HMS JOIN, UNION ALL, table-order and synchronous/streaming result checks.
- **What existing tests miss:** Shared FE tests use provider stubs, so removing the real Delta override could pass them. Iceberg's current CAST pushdown policy is not a permanent correctness contract for a Delta regression.
- **Fails when broken:** Removing Delta's interface opt-in or its false-returning override fails the connector unit test. Losing matching rows still fails the mixed golden results. Iceberg can change its own CAST policy without failing assertions about its plan text or selected file count.

### Testing

- run-fe-ut.sh: DeltaLakeScanPlanProviderTest and PluginDrivenScanNodeCastPredicateTest passed all 32 tests, with no failures, errors or skips.
- run-regression-test.sh: test_deltalake_hms_mixed_cast passed an ordinary rerun with all 14 result blocks. Its generated golden file is unchanged.
- Delta connector Checkstyle reported zero violations; git diff --check passed.

### Release note

None

### Check List (For Author)

- Behavior changed: No
- Does this need documentation: No
- Pre-submission status: Ready

### AI Disclosure

- Tool: OpenAI Codex
- Generated: Unit test, regression revisions, and this description
- Human review: Pending
Resolve three conflicts with current master:

- be/src/format_v2/table_reader.cpp: keep master's comment on the existing
  format_common.h include.
- build.sh: keep master's MongoDB packaging step and the connector list that
  includes deltalake.
- docker/thirdparties/docker-compose/iceberg/README.md: keep master's encrypted
  fixture text, followed by the Delta Lake regression section.

Test: No local build. Build hygiene, git diff --check and bash -n build.sh
passed. Compilation and tests are left to CI.
@xylaaaaa

xylaaaaa commented Nov 6, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

### Background

Delta chooses synchronous or streaming split planning from the current snapshot's file count. Writers without a current checksum leave that count unknown, so even a two-file table streams and can start far more scanners than the selected file needs.

### What problem does this PR solve?

Issue Number: apache#69884
Related PR: apache#69904 (Delta M1 reader)
PR Type: Performance

Problem Summary:

Reuse complete file counts already available in the statement or in Delta's user/table/version-scoped statistics cache when no current checksum is available. Existing statistics collection counts files, and exhaustive normal scans publish their already-observed total without submitting a statistics task. This lets subsequent small-table queries use synchronous planning while keeping large and unknown tables streaming. Changes are confined to the Delta plugin, its tests, and its README.

### Details

- **Why it was slow:** The streaming scanner path does not cap scanner creation by the actual split count. On the retained two-file fixture, the automatic path started 144 scanners to read one selected file and one row; forced synchronous planning started one.
- **Optimization:** Check the current CRC, then the statement's complete count, then the same user/table identity/version cache entry. The gate never enumerates files, starts statistics work, or waits for storage. A fully consumed scan without partition-pruned replay publishes its total into the entry captured before scanning, so late reports cannot repopulate invalidated entries. Background row-count sessions can have a different identity from SQL sessions, making normal-scan publication necessary without relaxing user isolation.
- **Results:** On local macOS ARM64 with the same ASAN BE and retained data, 16 alternating warm pairs per entrance reduced automatic median latency from 61.614 to 41.420 ms (filesystem), 66.370 to 38.509 ms (HMS), and 57.911 to 32.874 ms (HMS gateway). Scanner counts became one; all returned 123.45. Forced-sync controls were 35.978/41.056, 44.302/37.627, and 32.914/32.370 ms before/after. Million-file metadata enumeration medians were 1493.563/1451.023 ms with a 512 MiB heap; both scale tests passed.
- **Applicability and cost:** Warm snapshots with an already-known complete small file count benefit. Cold unknown counts still stream. Reuse existing cache capacity, TTL, invalidation and executor ownership; add one count, not a file list. These are small local ASAN measurements, not a TPC-DS claim. Sampled scan heap peaks increased from 133327520 to 238172480 bytes while remaining within 512 MiB; actual million-file data IO was not measured.
- **Behavior change:** Small Delta tables may select synchronous split planning after a complete count is available. Row results, public row-count cache behavior, BE scheduling, other connectors, and wire/storage formats are unchanged.

### Testing

- Local `./build.sh --fe -j4`: passed, including Checkstyle. Existing ASAN BE binary and local source configurations were unchanged by SHA-256.
- Official FE UT runner for `DeltaLake*Test,DeltaKernel*Test`: 92 actual tests passed, zero failures/errors; two optional classes skipped in that invocation. Separately enabled `DeltaLakePlanningScaleTest` with the retained million-file checkpoint and `-Xmx512m`: two tests passed. The diagnostic PlanningMetrics class remained disabled.
- Tests cover cached/unknown/current-CRC counts, threshold equality, missing row statistics, user/version/UUID/path isolation, no work submission or IO wait, late invalidated reports, partial and partition-pruned scans, large-file splitting, and large-table streaming after count reuse.
- Official regression runner: basic, statistics, cast_pruning, parquet_deletion_vector, hms_mixed_cast, and planning passed: six suites, 129 result blocks. Existing golden files were unchanged. The planning suite initially targeted the wrong local MinIO; using the retained fixture's endpoint resolved the missing-database error without changing the test or data.
- `git diff --check`: passed.

### Release note

Delta Lake queries can avoid excessive scanner startup for small tables without a checksum once a complete file count is available. Large tables and snapshots with unknown file counts continue to use streaming planning.

### Check List (For Author)

- Behavior changed: Yes (see Details)
- Does this need documentation: Yes; the Delta connector README documents selection, cache reuse and cold-query limitations. No new user configuration is introduced.
- Pre-submission status: Partially verified: local tests and targeted performance checks passed; full TPC-DS and large-table data IO performance were not rerun.

### AI Disclosure

- Tool: Codex (GPT-6)
- Generated: Implementation, tests, README, local evidence report and this description.
- Human review: Pending.
@xylaaaaa

xylaaaaa commented Nov 6, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.51% (2101/2746)
Line Coverage 65.88% (38770/58848)
Region Coverage 53.62% (36353/67797)
Branch Coverage 57.11% (11851/20750)

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28591 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fb4e695560bafc1862ac398fe22bfe6338c7f02e, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17591	5223	5122	5122
q2	2165	315	254	254
q3	10234	1455	843	843
q4	4686	479	340	340
q5	7438	809	541	541
q6	181	185	152	152
q7	764	812	612	612
q8	9264	1364	1064	1064
q9	5724	4251	4215	4215
q10	6850	1307	1019	1019
q11	435	245	234	234
q12	638	403	283	283
q13	18106	3684	3051	3051
q14	283	266	245	245
q15	q16	730	718	669	669
q17	1210	757	588	588
q18	7141	6216	6124	6124
q19	1111	981	607	607
q20	383	340	226	226
q21	4777	2141	2310	2141
q22	346	295	261	261
Total cold run time: 100057 ms
Total hot run time: 28591 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5644	5618	5622	5618
q2	700	570	562	562
q3	5190	5445	4971	4971
q4	2011	2177	1531	1531
q5	5304	5127	5867	5127
q6	253	198	149	149
q7	2379	2088	1922	1922
q8	3166	2946	2723	2723
q9	8208	8035	8117	8035
q10	3773	3670	3278	3278
q11	677	459	399	399
q12	703	722	511	511
q13	3489	4073	3235	3235
q14	323	301	296	296
q15	q16	708	717	661	661
q17	8439	7728	7143	7143
q18	12962	12136	12963	12136
q19	921	776	808	776
q20	2142	2151	1969	1969
q21	5869	4822	4887	4822
q22	507	456	399	399
Total cold run time: 73368 ms
Total hot run time: 66263 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 154395 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fb4e695560bafc1862ac398fe22bfe6338c7f02e, data reload: false

query5	4313	528	411	411
query6	437	214	206	206
query7	4821	442	233	233
query8	327	175	158	158
query9	8802	3295	3273	3273
query10	486	320	260	260
query11	6032	3872	3578	3578
query12	152	91	94	91
query13	1286	447	337	337
query14	6142	3961	3679	3679
query14_1	3471	3410	3450	3410
query15	207	205	184	184
query16	945	425	413	413
query17	852	662	537	537
query18	2418	424	315	315
query19	199	169	145	145
query20	88	85	87	85
query21	213	124	110	110
query22	13029	13342	12808	12808
query23	13411	12894	12338	12338
query23_1	12620	12550	12686	12550
query24	7366	826	475	475
query24_1	493	485	490	485
query25	533	431	373	373
query26	1248	265	142	142
query27	2768	449	280	280
query28	4633	2065	2050	2050
query29	1546	614	473	473
query30	302	237	189	189
query31	1008	940	836	836
query32	149	99	97	97
query33	538	300	244	244
query34	900	852	504	504
query35	735	752	645	645
query36	818	801	734	734
query37	127	100	88	88
query38	1767	1713	1647	1647
query39	811	789	756	756
query39_1	740	728	724	724
query40	224	117	102	102
query41	74	69	66	66
query42	93	95	88	88
query43	341	376	313	313
query44	1352	722	746	722
query45	182	176	179	176
query46	836	964	564	564
query47	2744	2880	2629	2629
query48	308	310	228	228
query49	578	415	286	286
query50	674	306	214	214
query51	10522	10657	10543	10543
query52	87	79	72	72
query53	187	214	161	161
query54	232	192	187	187
query55	77	70	62	62
query56	229	220	212	212
query57	1667	1656	1549	1549
query58	288	266	267	266
query59	2135	2173	1972	1972
query60	296	250	236	236
query61	156	147	172	147
query62	388	333	281	281
query63	187	168	159	159
query64	2766	940	864	864
query65	3358	3274	3270	3270
query66	1746	428	302	302
query67	22726	22653	22581	22581
query68	3069	1015	625	625
query69	400	301	302	301
query70	841	778	760	760
query71	303	234	212	212
query72	2739	2663	2330	2330
query73	513	528	300	300
query74	4593	4529	4396	4396
query75	2359	2296	1947	1947
query76	2321	1032	629	629
query77	355	368	274	274
query78	9968	9764	9168	9168
query79	2104	865	551	551
query80	1505	430	353	353
query81	590	318	275	275
query82	588	135	107	107
query83	317	198	184	184
query84	308	120	99	99
query85	854	429	387	387
query86	378	241	231	231
query87	2002	1956	1825	1825
query88	3860	2798	2768	2768
query89	327	290	252	252
query90	1903	178	169	169
query91	157	153	136	136
query92	102	91	94	91
query93	1133	1017	593	593
query94	581	272	266	266
query95	571	401	302	302
query96	684	489	241	241
query97	2493	2483	2341	2341
query98	159	156	146	146
query99	783	762	658	658
Total cold run time: 239864 ms
Total hot run time: 154395 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 23.94 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fb4e695560bafc1862ac398fe22bfe6338c7f02e, data reload: false

query1	0.01	0.01	0.01
query2	0.09	0.05	0.05
query3	0.26	0.09	0.09
query4	1.61	0.10	0.10
query5	0.25	0.24	0.24
query6	1.17	0.68	0.68
query7	0.04	0.01	0.00
query8	0.06	0.04	0.04
query9	0.39	0.35	0.36
query10	0.57	0.57	0.58
query11	0.19	0.16	0.15
query12	0.19	0.16	0.15
query13	0.48	0.49	0.48
query14	1.04	1.01	0.99
query15	0.62	0.60	0.60
query16	0.35	0.34	0.36
query17	1.15	1.15	1.18
query18	0.21	0.18	0.19
query19	2.08	2.01	1.93
query20	0.02	0.02	0.01
query21	15.47	0.20	0.13
query22	4.90	0.05	0.05
query23	16.29	0.31	0.11
query24	1.05	0.23	0.17
query25	0.10	0.04	0.04
query26	0.75	0.21	0.14
query27	0.02	0.03	0.04
query28	3.69	0.79	0.36
query29	12.48	4.12	3.17
query30	0.28	0.14	0.10
query31	2.78	0.55	0.33
query32	3.23	0.59	0.50
query33	3.17	3.19	3.20
query34	15.30	4.03	3.39
query35	3.32	3.35	3.34
query36	0.57	0.45	0.42
query37	0.08	0.06	0.07
query38	0.05	0.04	0.04
query39	0.04	0.03	0.03
query40	0.16	0.14	0.14
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 94.67 s
Total hot run time: 23.94 s

### Background

Delta fixes a snapshot for each statement, then enumerates its active files to
plan native Parquet ranges. Repeated scans currently reopen the same checkpoint
and reconstruct the same ranges. Other connectors already use the existing
statement planning scope and external scan task reuse session flag.

### What problem does this PR solve?

Issue Number: apache#69884
Related PR: apache#69904 (Delta M1 implementation)
PR Type: Performance

Problem Summary:

Repeated identical synchronous Delta scans redo checkpoint IO and planning.
Reuse completed immutable ranges and metrics within the existing planning scope,
with typed request keys, bounded retention and cancellable single-flight loading.
The change applies to dedicated Delta and HMS gateway reads. Production changes
are confined to the Delta plugin; most added lines are tests and generated output.

### Details

- **Change map:** Four production Java files (269 added / 50 removed lines), five test files (549 added / 4 removed), 244 generated expected-output lines and 33 README lines. The new memo owns keys, admission and concurrency; the existing scan body is extracted without changing its pruning semantics. Provider/scale tests and a three-route SQL suite cover reuse and bypass.
- **Why it was slow:** Each scan independently enumerated the pinned snapshot, including repeated checkpoint range GETs, decode and range construction.
- **Optimization:** Use the existing enable_external_scan_task_reuse flag and planning-only statement scope. Key by catalog/query/user, table coordinates/UUID/version, typed filter, partition selection, COUNT, split size and time zone. Coalesce identical loads outside scope/memo locks; publish only completed results and per-node metrics. Retry failed/cancelled loads; let waiters cancel independently. Retain at most 128 entries, 16,384 ranges and 16 MiB estimated range payload; oversized synchronous results are returned without retention. Streaming never enters the memo.
- **Results:** Same-build alternating local ASAN/SF1 A/B: Q88 EXPLAIN 323.616 to 152.279 ms and execution 2268.353 to 2050.009 ms (-9.6%); Q80 EXPLAIN -23.9%, execution -1.9%; Q4 EXPLAIN -49.4%, execution essentially flat; Q22 has no hits and execution is flat (+0.5%). Three independent traces each show checkpoint GETs 160 to 55 / 90 to 60 / 90 to 30 for Q88/Q80/Q4; Q22 stays at 15. A million-file streaming fixture remains bounded in a 512 MiB JVM: enumeration median 1493.940 to 1484.218 ms. Sampled heap peak 112.921 to 125.744 MB is recorded, not claimed unchanged.
- **Applicability and cost:** Helps identical synchronous scans within one statement. Does not share content across statements or change large-table streaming. The byte budget covers retained range payload, not whole-query or in-flight heap. Query results and FE/BE wire formats are unchanged; all four targeted queries match Hive reference results. This is not a full TPC-DS or RELEASE measurement.
- **Behavior change:** Repeated Delta plans may share immutable ranges until planning ends when the existing flag is enabled. PlanningReuseHit identifies each reuse. Disable the session flag to bypass the memo. Shared FE/SPI/BE and other connectors are unchanged.

### Testing

Local macOS ARM64:
- build.sh --fe -j4: full FE build and Checkstyle passed; final test-only refinements also passed reactor validate/Checkstyle.
- run-fe-ut.sh --run 'DeltaLake*Test,DeltaKernel*Test', with follow-up memo/provider and opt-in scale runs: 102 distinct executed tests passed; the optional PlanningMetrics diagnostic class was not run.
- Official regression runner: test_deltalake_basic (36 blocks), test_deltalake_scan_reuse (27), test_deltalake_parquet_deletion_vector (32), test_deltalake_hms_mixed_cast (14), all passed. The new .out was script-generated, independently audited, then compared without -genOut; existing .out files are unchanged.
- Retained-data A/B: 10 alternating EXPLAIN pairs and 7 execution pairs per query, after three warmups per flag and readiness of all 24 table counts. All 88 Delta executions (including warmups/profiles) matched four Hive references. Separate three-sample storage traces verified requests and bytes.
- Million-file streaming before/after: independent 512 MiB JVMs, three complete enumerations each, with FE/BE/Docker stopped for the matched comparison.
- All 96 retained TPC-DS objects matched path/size/SHA-256 before and after. Temporary services were stopped and fixtures/volumes retained.

### Release note

Repeated identical Delta Lake scans within one statement can plan faster,
including Delta tables queried through an HMS catalog. The optimization uses
the existing external scan task reuse setting and preserves streaming for large tables.

### Check List (For Author)

- Behavior changed: Yes (see Details)
- Does this need documentation: Yes; the connector README and local design/tasklist are updated. The broader Delta website documentation remains tracked by apache#69884.
- Pre-submission status: Partially verified: local build, relevant tests and targeted ASAN measurements passed; Linux CI, RELEASE and a full TPC-DS rerun are not claimed for this commit.

### AI Disclosure

- Tool: Codex
- Generated: Implementation, tests, documentation and this description.
- Human review: Pending review of this change.
@xylaaaaa

xylaaaaa commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

1 similar comment
@xylaaaaa

xylaaaaa commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@xylaaaaa xylaaaaa changed the title [feature](fe) Add native Delta Lake M1 reader [feature](external) Support reading Delta Lake tables natively with Delta Kernel Nov 10, 2026
Master added `16: optional TMongoScanDesc mongodb_params` to
TTableFormatFileDesc (apache#70327), so `deltalake_params` moves to field 17.
Field 17 is unused on master, branch-4.2, branch-4.1, branch-4.0 and
branch-3.1. Generated code accesses the field by name, so no Java or C++
code changes. build.sh keeps master's mongodb plugin entries and adds
deltalake to both connector lists.
@xylaaaaa

Copy link
Copy Markdown
Contributor Author

run buildall

### Background

The Delta reader can stream file enumeration. While enumeration is in progress, EXPLAIN can show a file-count estimate for scan scheduling. The completed query profile records the selected files and generated splits.

### What problem does this PR solve?

Issue Number: apache#69884
Related PR: apache#69904 (Delta reader)
PR Type: Test

Problem Summary: The basic partition-pruning and Parquet deletion-vector regressions treated the streaming EXPLAIN estimate as the final split count. Read the counters from the completed profile of the query whose rows are checked.

### Details

- **Failure:** With the default streaming threshold of 1, the estimate for the basic query was 3 when one split was generated. For the DV byte-range scan the estimate was 1 when five splits were generated.
- **Cause:** Split scheduling uses a file-count hint before enumeration, including when one file generates multiple ranges.
- **Stability evidence:** The basic check retains the partition fraction. The DV check retains the deletion-vector, row group, page-pruning, payload and row checks. Each scan is located by its unique SQL token and waits for complete Delta enumeration.

### Testing

- Built the PR1 FE and regression framework and ran the existing golden outputs for test_deltalake_basic and test_deltalake_parquet_deletion_vector on local ASAN Doris, Hive Metastore and MinIO.
- Both suites passed with default batch scheduling and threshold 1, and passed again with only session batch scheduling disabled.
- Filesystem and HMS each reported one selected file for both queries, one generated split for the filtered basic query and five for the DV cross-split query.

### Release note

None

### Check List (For Author)

- Behavior changed: No
- Does this need documentation: No
- Pre-submission status: Ready

### AI Disclosure

- Tool: OpenAI Codex (GPT-6)
- Generated: Test changes and commit description
- Human review: Pending
xylaaaaa pushed a commit to xylaaaaa/doris that referenced this pull request Nov 12, 2026
### Background

Apache Doris PR apache#69904 adds the Delta Lake reader for the latest table state. Delta records each committed change as a numbered version of its transaction log. Reading an older version requires using that version for column binding, file selection and deletion vectors.

### What problem does this PR solve?

Issue Number: apache#69884  
Related PR: apache#69904 (required base, merge first)  
PR Type: Feature

Problem Summary: Allow an ordinary Delta table reference to select a numeric version using `FOR VERSION AS OF`. This is a stacked draft based on `8ba8e67f29` from apache#69904. GitHub's Files changed tab includes the base PR until it merges. [Review only the changes for version reads](8ba8e67...codex/delta-version-as-of).

### Details

- **Change map:** The Delta FE connector accounts for 65 added and 8 removed production lines. Unit tests and fixtures add 136 lines and remove 2. The Spark comparison adds 172 test lines and the framework generated 91 output lines.
- **Scope:** Numeric versions, including zero, through Delta filesystem, Delta HMS and the HMS gateway.
- **Design:** Load the requested Kernel snapshot, verify the target table UUID and reader features, and retain its schema. Reuse the checked handle for that table and version within the query. The file scan, projected columns and exact COUNT operate on the same version.
- **Usage:** `SELECT * FROM catalog.db.t FOR VERSION AS OF 3`.
- **Default behavior:** Ordinary reads continue to use the latest snapshot and the configured latest cache.
- **Compatibility:** The table's current protocol must be readable when opening its handle. An unavailable log version fails explicitly. Timestamp selection and the history system table remain open in apache#69884. The HMS gateway still uses its existing statistics delegation for CBO estimates. Kernel reads a version while its required log and data files are available. Spark Delta 4.4.1 adds a separate, default-on age guard based on deleted-file retention, which can reject readable data. [Spark implementation](https://github.com/delta-io/delta/blob/v4.4.1/spark/src/main/scala/org/apache/spark/sql/delta/SnapshotManagement.scala#L1656-L1689)

### Testing

- Full local macOS arm64 ASAN build with `FE_MAVEN_THREADS=4 ./build.sh --fe --be -j10`. Started the resulting FE and BE and verified a distributed query.
- `bash run-fe-ut.sh --run 'DeltaLake*Test,DeltaKernel*Test'` passed. The Delta connector and Delta metastore report 106 tests executed with zero failures, including all six new history tests. Two opt-in diagnostics were skipped.
- Generated the golden output for `test_deltalake_version_as_of` via `run-regression-test.sh` and reran the unchanged suite without `-genOut`. Both runs passed with Spark 4.0.0, Delta 4.4.1, Hive Metastore and MinIO. All three catalog routes matched Spark on historical columns, types, ordered rows, counts, column mapping by name/ID, a physical deletion vector and multi-version joins. The new test exercises synchronous and streaming scheduling with the existing default threshold of 1.
- The existing `test_deltalake_hive` passed. After rebasing on the split-profile test fix from apache#69904, reran `test_deltalake_basic`, `test_deltalake_parquet_deletion_vector` and `test_deltalake_version_as_of` with default streaming scheduling and `num_files_in_batch_mode=1`. All three passed against their existing golden outputs.

### Release note

Delta Lake tables can be queried at a numeric version with `FOR VERSION AS OF`, including the schema and deletion vectors recorded at that version.

### Check List (For Author)

- Behavior changed: Yes (see Details).
- Does this need documentation: Yes, to accompany the Delta reader website documentation tracked in apache#69884.
- Pre-submission status: Locally verified. Draft pending apache#69904.

### AI Disclosure

- Tool: OpenAI Codex (GPT-6).
- Generated: Implementation, tests and PR description with AI assistance.
- Human review: Pending.
@xylaaaaa

Copy link
Copy Markdown
Contributor Author

run buildall

### Background

The M1 Delta reader uses Delta Kernel to plan external scans and the native
Parquet reader to execute them. Unity Catalog also coordinates the commits
of managed tables, which can be visible before their numbered log files are
backfilled.

### What problem does this PR solve?

Issue Number: apache#69884
Related PR: apache#69904 (M1 reader)
PR Type: Feature

Problem Summary: Provide read-only Unity Catalog access alongside filesystem
discovery. Retire the branch's dedicated HMS resolver and native Hive
delegation. Use UC to authorize table reads and to resolve managed versions.

### Details

- **Change map:** Production and build files +933/-342, tests and fixtures
  +714/-1311, generated regression output +25/-2773, and documentation +49/-3.
  The removed golden blocks belong to the retired HMS routes.
- **Scope:** List UC schemas and Delta tables and read external and managed
  tables. Reject unsupported governance and storage modes.
- **Design:** Carry the UC registration identity through snapshot pinning,
  caching and statistics. Resolve managed versions through Kernel's UC
  client. Bind token or OAuth M2M to UC and table READ grants to storage.
  Use short-lived table credentials on FE and request a grant for each BE
  scan. Keep credentials out of serialized handles and the UC identity out
  of BE tasks.
- **Usage:** Set deltalake.catalog.type=uc, deltalake.uc.uri and
  deltalake.uc.catalog. Configured storage credentials can be used for a
  cloud EXTERNAL table after UC authorizes READ.
- **Default behavior:** Filesystem remains the default discovery mode. UC
  defaults to vended storage credentials and supports S3 sessions.
- **Compatibility:** Contains no BE or wire-format changes. Hive rejects
  Delta-marked Parquet registrations. Other Hive formats retain their
  routing. Restore the master CAST pushdown interface and use the Delta
  metadata override for Delta queries.
- **Behavior change:** This M1 branch stops accepting the dedicated HMS
  flavor and stops forwarding Delta tables from Hive.

### Testing

Built FE and ASAN BE on macOS. The local BE toolchain required disabling
the precompiled header with ENABLE_PCH=OFF. Ran 128 selected FE unit tests,
including opt-in million-file planning tests, with zero failures or skips.
Ran all 12 Delta P0 suites and checked all 190 retained filesystem output
blocks against the prior revision. Generated and reran the seven UC blocks.

Spark 4.0 / Delta 4.4.1 and Doris read the same two rows through open-source
Unity Catalog 0.7.0 when managed v3 had only a ratified staged commit. Both
Doris scheduling modes returned the expected rows and count; INSERT failed.
Read a registered external MinIO table with a scoped S3 session through UC
and with configured storage credentials. Confirmed the retired HMS flavor,
the Hive Delta registration and path-based managed reads were rejected,
and an ordinary Hive Parquet table remained readable. Verified masking of
both UC secret properties in SHOW CREATE CATALOG.

### Notes for Reviewers

Creating a catalog requires CREATE on the catalog resource; changing its
properties requires ALTER on that catalog. Such a user can name outbound
FE hosts in deltalake.uc.uri and deltalake.uc.oauth.token-uri. The Delta
provider declares both SSRF endpoints for the existing operator-configured
CREATE/ALTER gate. Inspect DeltaLakeUnityCatalog and the staged-commit
fixture for the managed-table read path.

### Release note

M1 can read filesystem Delta tables and Unity Catalog external and managed
Delta tables. A UC catalog is read-only.

### Check List (For Author)

- Behavior changed: Yes (see Details).
- Does this need documentation: Yes. Update the M1 website catalog page to cover filesystem and UC, authentication and verified storage modes.
- Pre-submission status: Partially verified. Hosted Databricks with a configured Azure Shared Key needs a workspace and authorized table. UC-vended Azure SAS is pending. Linux CI has not run on this commit.

### AI Disclosure

- Tool: OpenAI Codex (GPT-6).
- Generated: Implementation, tests, fixture, documentation and this description.
- Human review: Pending review of this revision.
@xylaaaaa

Copy link
Copy Markdown
Contributor Author

run buildall

chenjunwei added 6 commits November 17, 2026 10:31
### Background

The filesystem and read-only Unity Catalog implementations share the native Delta reader. Both must use Parquet field IDs and interpret partition values from the Delta log.

### What problem does this PR solve?

Issue Number: apache#69884
Related PR: apache#69904 (M1 reader)
PR Type: Test

Problem Summary:

Add correctness checks for a reused physical name, a widened partition, an empty partition value and a comma in a logical partition name. Keep the expected rows independent of the Doris results.

### Details

- **Scenarios:** Read ten retained tables through filesystem and open-source UC with synchronous and streaming scheduling. Verify that ID mapping distinguishes matching, missing and different IDs, that DATE partitions survive widening to TIMESTAMP_NTZ, that empty partition values are null and that a mapped partition name can contain a comma.
- **What existing tests miss:** The existing suites use unambiguous mapped IDs, compatible partition text and simple partition names. Spark and the Delta protocol also differ on the empty STRING fixture.
- **Fails when broken:** Checks compare against typed SQL literals and log each mismatch. An untagged Parquet file may produce null or an explicit field-ID error. A wrong row, rejected valid partition or dropped matching row fails the suite.

### Testing

Built FE and BE at 84ddcbd. The basic and UC regression suites passed. The new suite ran 56 comparisons against MinIO and local OSS UC: all 20 controls passed, and 36 comparisons reproduced the four open findings. A separate filesystem/UC run over identical local files agreed. Spark 4.0.1 with Delta 4.4.1 independently read the same files.

### Release note

None

### Check List (For Author)

- Behavior changed: No
- Does this need documentation: No
- Pre-submission status: Not ready: the four product findings remain open and the new regression is red.

### AI Disclosure

- Tool: OpenAI Codex
- Generated: Regression checks, fixture registration and setup, and the commit description.
- Human review: Pending.
### Background

A Delta snapshot specifies whether Parquet columns are identified by their display names, physical names or field IDs. Files written by Spark can carry field IDs in both mapped modes.

### What problem does this PR solve?

Issue Number: apache#69884
Related PR: apache#69904 (M1 reader)
PR Type: Bugfix

Problem Summary:

A Delta table requesting field ID 7 could read the value of a same-named field with ID 3. Send the mapping mode with the scan and let the native reader use the column identity selected by the snapshot.

### Details

- **Change map:** Production code and Thrift +85/-5, unit tests, regression and fixture metadata +119/-17, generated regression output +85, and fixture documentation +7/-1. All three 608-byte Parquet fixtures reuse the same existing blob.
- **Frequency:** An ID-mapped file has a physical name and field ID that identify different table fields, or the file lacks field IDs.
- **Root cause:** The scan represented name and ID mapping with the same schema dictionary. BE assigned field IDs to the file by physical name, overwriting its original identifiers.
- **Why existing tests missed it:** Spark's ordinary rename fixtures have consistent physical names and IDs. The retained reader-edge regression reproduced the wrong-ID and untagged cases. The new crossed fields also verify that the value reaches the differently named column with the matching ID.
- **Fix:** Carry an optional Delta-specific mode on each scan. Preserve Parquet field IDs in ID mode, ignore file IDs when resolving physical names in name mode, and use display names in none mode.
- **Other paths and branches:** Filesystem and Unity Catalog use the same Delta reader. A BE executing an older FE plan without the mode keeps the existing behavior.
- **Behavior change:** A missing requested ID yields NULL when the file has other field IDs. Reading a data column by ID from an entirely untagged file reports the file path.

### Testing

- Built FE and BE on macOS arm64 with ASAN. Delta connector FE tests passed (101 passed, 2 skipped), including both mapped modes and the none-mode scan parameters. The two opt-in scale/profile tests were skipped.
- All 18 Delta reader and deletion-vector BE tests passed under ASAN (15 reader, 3 DV), including nested conflicts and a full untagged-file read. The monolithic unit-test build hit an unrelated macOS std::min type mismatch in binary_column_extract_iterator_test.cpp. Used the same BE libraries, test harness and run-be-ut.sh with the Delta test translation unit linked separately.
- Compared three Delta tables sharing identical retained Parquet bytes with Spark 4.0.1 and Delta 4.4.1. Before the fix, ID mode returned (703, NULL). After the fix it returns (NULL, 703) through filesystem/MinIO and open-source UC under synchronous and streaming scheduling. Name and none return (703, NULL).
- Generated and verified all 28 output blocks in test_deltalake_column_mapping and exercised four untagged-file refusals. The new suite and the existing basic, features, UC and Parquet DV suites passed. On the retained Spark DROP/ADD tables, both mapped modes read NULL for the old row and 900 for the new row.
- The combined reader_edges suite passes all field-ID checks. Its 28 outstanding failures out of 56 comparisons concern the separate widened partition, empty partition and comma-in-partition-name findings.

### Release note

Doris resolves Delta ID-mapped columns by their Parquet field IDs, including when physical names and IDs disagree.

### Check List (For Author)

- Behavior changed: Yes (see Details)
- Does this need documentation: No
- Pre-submission status: Partially verified: macOS ASAN and the retained Delta suites. The combined reader_edges suite still reproduces three separate partition issues.

### AI Disclosure

- Tool: Codex (GPT-6)
- Generated: Implementation, tests, fixture metadata, verification and commit description.
- Human review: Pending.
### Background

The Delta reader checks both the Parquet schema and the table projection for field IDs before reporting a file with no field IDs.

### What problem does this PR solve?

Issue Number: apache#69884
Related PR: apache#69904 (M1 reader)
PR Type: Refactor

Problem Summary:

The previous helper name implied that every ID came from Parquet. Projected table fields get their IDs from the Delta snapshot.

### Details

- **Moves vs. logic changes:** Rename the private C++ helper and its three uses to `has_data_field_id`.
- **Behavior change:** None.
- **Compatibility:** The helper has translation-unit scope.
- **Performance impact:** None.
- **Rollback:** Revert this commit.

### Testing

Four focused Delta reader tests passed in a local macOS ASAN build, covering an untagged file, a renamed physical column, nested renames, and conflicting field IDs. The clang-format 16 check and clang-tidy passed.

### Release note

None

### Check List (For Author)

- Behavior changed: No
- Does this need documentation: No
- Pre-submission status: Ready

### AI Disclosure

- Tool: Codex (GPT-6)
- Generated: Helper rename and commit description.
- Human review: Pending.
### Background

Delta stores each file's partition values as strings in the transaction log. Changing a partition column from DATE to TIMESTAMP_NTZ updates the snapshot schema and leaves old files with date-only strings.

### What problem does this PR solve?

Issue Number: apache#69884
Related PR: apache#69904 (M1 reader)
PR Type: Bugfix

Problem Summary:

Reading old partitions failed during planning after Spark widened a DATE partition column to TIMESTAMP_NTZ. Even a query that projected only a data column could fail during partition listing.

### Details

- **Frequency:** A snapshot reads an active file with a date-only partition value for a column widened from DATE to TIMESTAMP_NTZ.
- **Root cause:** Partition normalization used the current data type without consulting the field's type changes. The timestamp parser requires a time component.
- **Why existing tests missed it:** Data-file widening and ordinary timestamp partitions had coverage separately. The reader-edge regression reproduced the planning failure but had no predicate on the old partition.
- **Fix:** Consult the current field's recorded date-to-NTZ transition for a date-only value. Parse the date strictly and render its local midnight with the same formatter as a full NTZ timestamp. The predicate converter reuses the same type-change identity.
- **Other paths and branches:** Filesystem and read-only UC share the Delta partition normalization path. The change needs no extra log reads or BE support.
- **Behavior change:** Old date partitions can be listed, filtered and read after widening. A native NTZ column still requires a timestamp, and an invalid calendar date still raises an error.

### Testing

- Built FE and passed Checkstyle on macOS. Delta connector FE tests passed (102 passed, two opt-in tests skipped), including strict date parsing and the native NTZ control.
- Reused the two-file Spark table in MinIO filesystem and local open-source UC. On the previous FE, all 20 checks failed during planning. On the fixed FE, all 60 checks across both catalogs, sync/stream scheduling and UTC, Asia/Shanghai and America/Los_Angeles matched Spark. Equality at the old midnight returned id 1, the range excluding it returned id 2, and equality one microsecond later counted zero rows.
- Ran test_deltalake_reader_edges. All 20 R4 checks passed, as did 28 controls. Its 20 remaining failures reproduce the open R5 empty-partition and R7 comma-in-column-name issues.
- Ran the basic, features, UC, column mapping, skipping and Parquet deletion-vector regression suites. All six passed.

### Release note

Doris can read and filter existing Delta DATE partitions after they are widened to TIMESTAMP_NTZ.

### Check List (For Author)

- Behavior changed: Yes (see Details)
- Does this need documentation: No
- Pre-submission status: Partially verified: local macOS validation. The combined reader-edge suite also exercises separate open partition bugs.

### AI Disclosure

- Tool: Codex (GPT-6)
- Generated: Implementation, tests, validation and commit description.
- Human review: Pending.
### Background

The Delta connector uses AddFile partition values both to plan files and to
supply Doris with the values of partition columns. Delta Kernel can also
evaluate partition predicates on the serialized values before returning files.

### What problem does this PR solve?

Issue Number: apache#69884
Related PR: apache#69904 (Delta M1 reader)
PR Type: Bugfix

Problem Summary: A DATE partition serialized as an empty string failed to
plan. Kernel 4.4.1 could also discard its file for IS NULL or fail a date
comparison before the connector decoded the value. Read that encoding as
NULL and use the normalized partitions to select its files.

### Details

- **Change map:** 50 implementation lines added and seven removed in the Delta FE plugin, 86 unit-test lines added and two removed, 54 regression suite lines, and 129 generated output lines.
- **Frequency:** A Delta transaction log must contain an explicit empty string for a DATE partition. The Spark 4.0.1 / Delta 4.4.1 writes tested locally emitted JSON null for a null DATE.
- **Root cause:** The connector passed the empty value to LocalDate.parse. Kernel used the serialized null bit for IS NULL and attempted to parse the empty value for date comparisons.
- **Why existing tests missed it:** Ordinary DATE and JSON-null fixtures did not exercise the empty encoding. The existing reader-edge reproducer exposed a projection failure but did not test filtered file selection for this case.
- **Fix:** Decode only an empty DATE as NULL. The existing full partition listing records the affected logical DATE columns for that immutable Kernel snapshot. The scan uses normalized partition selection for those columns. Before a complete listing, DATE partition predicates remain residual. A listed clean DATE column retains Kernel pruning.
- **Other paths and branches:** This applies to the Delta filesystem and Unity Catalog paths in both scheduling modes. STRING semantics remain under review. The common FE, BE, and other connectors are unchanged.
- **Behavior change:** Empty DATE partitions are readable and participate in IS NULL and date comparisons. Malformed non-empty dates still raise an error.

### Testing

A local FE build with Checkstyle succeeded. The Delta connector FE tests
passed 105 tests with two opt-in tests skipped. On the retained MinIO and
open-source Unity Catalog fixtures, 20 pre-fix DATE queries failed. All 40
post-fix DATE, JSON-null and STRING probe queries matched their expected
rows. Twelve filtered plans each selected one of two files.

The new P0 suite generated and then verified 40 expected output blocks
across both catalogs and both scheduling modes. Six other Delta P0 suites
passed. The existing reader-edge suite passed 52 of 68 checks. Its 16
remaining failures cover the pending STRING interpretation and partition
column names containing commas. All DATE checks passed.

### Release note

Doris now reads Delta DATE partitions serialized as an empty string as NULL,
including when filtering on the partition column.

### Check List (For Author)

- Behavior changed: Yes (see Details)
- Does this need documentation: No
- Pre-submission status: Partially verified: the local Delta suites above ran. CI has not run on this commit.

### AI Disclosure

- Tool: OpenAI Codex (GPT-6)
- Generated: Delta connector changes, unit tests, regression suite, and this description. The regression runner generated the expected output.
- Human review: Pending.
### Background

Doris masks catalog credentials in SHOW CREATE CATALOG. The frontend now discovers connector-specific sensitive keys when it loads each connector provider.

### What problem does this PR solve?

Issue Number: apache#69884
Related PR: apache#69904 (M1 Delta reader)
PR Type: Build/CI/Chore

Problem Summary:

The Delta branch added two UC keys to the frontend's masking list before master introduced registration through the connector. Merge master at e04a785 and register the keys through the Delta provider so both credential types remain masked.

### Details

- **Affected:** The Delta FE connector. Its security surface includes the sensitive properties declared by its metastore providers and the existing UC endpoints.
- **Developer and release impact:** The shared masking implementation matches master. The connector test verifies registration of the bearer token, OAuth client secret and endpoints.
- **License and third-party:** None.
- **Rollback:** Revert this merge relative to its first parent to restore the previous branch state.

### Testing

- The FE build with Checkstyle passed on macOS arm64. Targeted FE unit tests reported 164 passes and two opt-in skips, with no failures or errors.
- SHOW CREATE CATALOG masked the UC bearer token and OAuth client secret for newly created catalogs and again after restarting the FE. The catalog name and OAuth client ID remained visible.

### Release note

None

### Check List (For Author)

- Behavior changed: No
- Does this need documentation: No
- Pre-submission status: Ready

### AI Disclosure

- Tool: Codex (GPT-6)
- Generated: Conflict resolution, provider test and merge description.
- Human review: Pending.
@xylaaaaa

Copy link
Copy Markdown
Contributor Author

run buildall

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants