Repository navigation
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Issue Number: N/A Related PR: apache#67470, apache#66782 (separate Iceberg/Azure work; not included as feature changes here) Problem Summary: Doris currently exposes Databricks interoperability through the Iceberg catalog path. This draft adds an experimental native Delta Lake catalog that reads Delta metadata directly with Delta Kernel, rather than relying on an Iceberg mirror. The draft contains: - A new `type=delta` connector with path and Unity Catalog adapters; Delta Kernel 4.3.1, Unity Catalog client integration, snapshot/time-travel planning, partition pruning, and an explicit supported-feature gate. - Native Parquet scans, Delta deletion vectors and column mapping, plus table-scoped AWS, Azure SAS and GCS credential handling. Shared Azure SAS/GCS backend prerequisites are included because the Delta adapter emits those credentials. - Opt-in writes (`delta.write.enabled=true`): create, append, overwrite, restricted copy-on-write DELETE/UPDATE/MERGE, TRUNCATE and Unity catalog registration deletion. Catalog-managed snapshots/commits use the Unity/Kernel path. - Connector SPI 13.0 changes, snapshot-pinned COW routing and write transactions, typed file-commit reports integrated with the current final-report acknowledgment path, and focused FE/BE tests and Delta regression suites. This is a standalone, single-commit branch based on master `0c29961f45e`, extracted from local development commit `1d957e6c8bb`. Course/dbt changes, local configuration, independent Iceberg ADLS normalization and Iceberg Azure dependency additions are excluded. This is **not ready to merge** and does not claim production Databricks support. Static checks passed on the source development checkout: SPI/Delta/FE-core Checkstyle, connector import and metadata-funnel gates, BE header hygiene, scoped clang-format 16 checks, and Java generation for the modified Thrift definitions. The extracted branch passed code whitespace and scope/content checks. Auto-generated regression .out files retain their original tab-separated formatting (including empty fields and section separators); their results were not edited by hand. Full FE/BE compilation and FE/BE unit tests have **not been rerun on this extracted branch**. Full compilation is being handled by the author. Prior local fixture/SQL results are historical evidence only; they are not validation of this draft. A stale BE test enum reference found while extracting the branch is updated to the current `ObjStorageProvider` name. The separate Azure auth source formatting is normalized with clang-format 16. Neither change is a claim of successful compilation. Still required before merge: - Standard FE/BE builds and relevant unit tests. - Path and Unity SQL regression comparison with default FileScannerV2. - Real Databricks workspace and cloud-storage E2E verification. - Long-query credential renewal, feature/version compatibility and supported DML semantics review. ABAC row filters/masks are rejected, not executed by this implementation. Unity REST and object-storage requests are outbound HTTP/SSRF surfaces to endpoints supplied through catalog properties or returned table/storage metadata. Creating or changing the catalog configuration requires catalog CREATE or ALTER privilege; scanning/writing requires the corresponding table privileges and Unity/storage credentials. This connector introduces no URL-based artifact download or dynamic code loading. Add an experimental native Delta Lake catalog with path and Unity Catalog adapters. Databricks and multi-cloud production readiness remains unverified. - Test: - [x] Manual static validation (details and limitations above) - [ ] Unit Test — rerun FE SPI/Delta/COW/write-report tests and relevant BE tests on this branch - [ ] Regression test — rerun native Delta path/Unity suites with the default scanner - Behavior changed: - [ ] No. - [x] Yes. Add an experimental external Delta catalog and shared storage prerequisites. - Does this need documentation? - [ ] No. - [x] Yes. A progress/validation record is included; user-facing website documentation is still pending. - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
### What problem does this PR solve? Issue Number: N/A Related PR: apache#68702 Problem Summary: Native Delta writes reuse the Hive file sink, whose partitioned and unpartitioned file-rotation paths discarded close failures. A failed file could therefore disappear from the commit reports while later successful files were committed, losing rows during append or copy-on-write replacement. Propagate close errors before replacing the writer. Delta snapshot loading also looked up logical partition names in add-file maps that use physical names under column mapping. Resolve those names through the pinned schema using Delta Kernel, preserve logical scan order and null values, and keep remove-action maps physical. Add failure-injection tests for both sink kinds and partition layouts, plus name/id mapping fixtures covering renames and historical snapshots. Update two existing BE test constructor calls for the Delta partition-report arguments. ### Release note Prevent partial Delta writes from committing after file-rotation failures, and support reading partitioned Delta tables with name or id column mapping. ### Check List (For Author) - Test: - Unit Test: All 75 Delta FE tests passed via run-fe-ut.sh with EXTRA_FE_MODULES=delta=fe-connector/fe-connector-delta; the wider reactor later failed in unrelated fe-grpc protoc-gen-grpc-java execution. - Unit Test: Added four BE rollover-close failure cases. Attempted run-be-ut.sh under ASAN; the existing Core wide_integer_to_string.h/fmt incomplete-type errors stopped the build before tests ran. - Manual test: Delta reactor Checkstyle validate, full clang-format 16 check, BE build hygiene, connector import and metadata-funnel gates, and git diff whitespace checks passed. clang-tidy and supplementary test-TU syntax checks were blocked by the local shell/gtest toolchain errors; no BE test pass is claimed. - Behavior changed: Yes. Failed file rotations abort writes; mapped partition values resolve using physical log names. - Does this need documentation: No. Correctness fixes for the existing experimental connector.
### What problem does this PR solve? Issue Number: N/A Related PR: apache#68702 Problem Summary: Native Delta fails to compile against current connector APIs and on ARM macOS. Use the supported batch mask-policy and session/namespace APIs, preserve opaque commit reports alongside Paimon, remove duplicate declarations, and adapt affected test fixtures to the current sink and reader contracts. Select an operator-installed native grpc-java generator when available while retaining the Maven fallback. Keep floating-point limits available without disabling Clang PCH reuse, isolate shaded GCS dependencies, and preserve existing connector configuration when repeating a BSD build. ### Release note None ### Check List (For Author) - Test: Unit Test / Manual test - BE ASAN: 415 related tests passed - FE: 75 Delta, 154 SPI and 148 related core tests passed - Standard ASAN BE and FE builds, FE checkstyle and deployed JNI plugin isolation check passed - clang-format 16, clang-tidy changed-line checks, header hygiene and FE architecture gates passed - Repeated FE package deployment preserves existing JDBC configuration - Behavior changed: No; restore intended native Delta behavior and build portability - Does this need documentation: No
### What problem does this PR solve? Issue Number: N/A Related PR: apache#68702 Problem Summary: The current connector SPI preserves DELETE, UPDATE and MERGE as the statement operation even after their native Delta plans become full-table overwrites. Delta rejected these valid sink handles because it accepted only INSERT and OVERWRITE operations. Accept row-level intents when the handle carries the copy-on-write overwrite mechanism, while continuing to reject direct row-level writes and unsupported file rewrites. Update negative DDL regression assertions to the connector-owned messages used by current master. ### Release note Native Delta DELETE, UPDATE and MERGE execute their snapshot-pinned copy-on-write plans with the current connector SPI. ### Check List (For Author) - Test: Unit Test / Regression test - Delta module: 83 tests passed, including 8 new write-intent cases - Native Delta path and local Unity mock-service regression suites passed, with no skipped suites - Standard FE build and checkstyle passed - Existing regression result files were verified without modification - Behavior changed: Yes; restore supported native Delta row-level DML after the connector SPI update - Does this need documentation: No
xylaaaaa
force-pushed
the
codex/native-delta-catalog
branch
from
October 29, 2026 14:27
d3cd7e9 to
3c7dfc9
Compare
added 2 commits
October 29, 2026 23:18
### What problem does this PR solve? Issue Number: None Related PR: apache#68702 Problem Summary: Native Delta scans use the generic TableReader directly, which puts Delta deletion-vector descriptors and inline decoding in the shared reader. Add DeltaTableReader and route Delta scans to it. Move descriptor validation, path and cache-key handling, inline loading and sidecar decoding into the adapter. Keep common file I/O, decoded-vector caching, profiling, filtering and V2 Parquet decoding in the existing shared layers. Remove Delta-only fields from the shared DeleteFileDesc without changing the FE/BE protocol or query behavior. Increase the RLE reverse-reach baseline by three for the new adapter header, implementation and unit test, all legitimate TableReader users. No heavy include is added to a hub header; build hygiene and all gate self-tests pass. ### Release note None ### Check List (For Author) - Test: ASAN BE build; 475 BE unit tests; all 3 native Delta SQL regression suites - Add 21 unit test instances and SQL coverage for inline, absolute-URI and UUID deletion vectors; generate results with the standard regression runner - clang-format 16, clang-tidy, build hygiene and gate self-tests pass - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#68702 Problem Summary: Official Delta Golden Tables expose two native-read failures. Percent-escaped catalog locations are passed directly to Kernel's unescaped Hadoop-path API, so a valid GMT%2B9 URI cannot find the GMT+9 table. Convert location URIs exactly once for latest, version and timestamp snapshots and create/append transactions. Also apply the existing committed-snapshot missing-file rule to Delta: both scanners must propagate NOT_FOUND instead of silently returning incomplete rows under the generic external-file ignore option. Add functional URI tests and an offline copy of the official Golden Tables at 9cb15d26156ac42e646e69d26652976fdb72f8df, preserving the upstream license. The regression covers 108 Delta directories, a non-Delta negative fixture and 126 historical versions, plus both scanners' missing-file behavior and real create/append/history reads at escaped locations. Independently compare 203 result blocks with Spark 4.1.1 / Delta Spark 4.3.1 in UTC. Large decimal tables check every row against the official generator and the complete unique ID domain. Keep unsupported features and malformed/missing-file fixtures as explicit errors. ### Release note Native Delta catalogs correctly resolve percent-escaped table locations and report missing committed data files instead of silently returning incomplete data. ### Check List (For Author) - Test: Unit Test / Regression test - Standard ASAN BE build and FE build/package with Checkstyle passed - Standard BE runner: 482 tests passed; Delta FE runner: 88 tests passed - All 4 native Delta SQL suites passed in normal comparison mode, no skips - Independent Spark comparison: 203 result blocks, including 201 official reads - clang-format 16, clang-tidy and build hygiene passed - Behavior changed: Yes, resolve URI escapes once and fail Delta scans on missing committed files - Does this need documentation: Yes, fixture provenance and reproduction instructions included
morningman
added a commit
to morningman/doris
that referenced
this pull request
Nov 13, 2026
…shdown ### Background Connector plugins learn which filters they may push down from a `ConnectorExpression` tree that fe-core's `ExprToConnectorExpressionConverter` builds from a scan's conjuncts. A function call arrives as a `ConnectorFunctionCall` with the function's name, return type and arguments, and connectors that translate calls match it by name: JDBC writes it into the remote SQL when `enable_ext_func_pred_pushdown` is on (the default) and the name passes the database's rules (for MySQL, a blocklist of three names), and Elasticsearch runs `esquery` as a native query. ### What problem does this PR solve? Issue Number: None Related PR: apache#70369 (changes the same converter and SPI package; independent), apache#68702 (also changes the connector SPI; whichever merges second takes the next plugin API major) PR Type: Bugfix Problem Summary: A filter that calls a Java or Python UDF on a JDBC table is sent to the remote database as a call of the same name. On MySQL, `WHERE mydb.my_udf(id) = 113` fails with `FUNCTION doris_test.my_udf does not exist`, and a UDF named like a MySQL function, `mydb.abs(id) = 113`, is computed by MySQL instead, which silently drops rows. The converter hands connectors only the function's name, so no connector can tell a UDF from a builtin. This PR adds `ConnectorFunctionCall#isBuiltin()`, set from the catalog function's binary type; JDBC and Elasticsearch no longer translate a call that is not a builtin, and Doris evaluates it. ### Details - **Change map:** +373 −26 in 13 files, of which tests are +303: a regression suite +89 with its `.out` +10, and unit tests +204. The production code is +69 −23: 1. SPI (+42 −13): the flag on `ConnectorFunctionCall`, and Rule 6 of `pushdown/package-info.java` rewritten; plus the plugin API major 30.0 → 31.0 in `fe/fe-connector/pom.xml`. 2. fe-core (+12 −4): `ExprToConnectorExpressionConverter` sets the flag. 3. Connectors (+15 −6): the checks in `JdbcQueryBuilder` and `EsQueryDslBuilder`. - **Frequency:** Always, with function pushdown on: any filter that calls a UDF on a JDBC catalog over MySQL, and over ClickHouse or Oracle when the UDF's name is on their allowlist or in `function_rules`. On Elasticsearch, a UDF named `esquery`. - **Root cause:** `ExprToConnectorExpressionConverter.convertFunctionCall` keeps only `getFnName().getFunction()`, the return type and the arguments, and drops `Function.BinaryType`. `JdbcQueryBuilder.functionCallToSql` then renders any call that `JdbcFunctionPushdownConfig.canPushDown` accepts by name, and `EsQueryDslBuilder` takes any call named `esquery` for the builtin. The SPI documentation said calls with real arguments "are safe to match by name". - **Why existing tests missed it:** `JdbcQueryBuilderTest` and `EsScanPlanProviderTest` build calls by hand, all of them builtins, and no unit test converts a UDF. The only regression suite that uses a UDF on a JDBC table, `test_mysql_varbinary_with_udf`, calls it in a projection; no suite filters with one. - **Fix:** The converter sets the flag where the binary type is still known, and each connector that matches calls by name checks it. JDBC refuses a non-builtin call before rendering anything, so a fallback (an expression the converter does not model, sent as SQL text) that calls a UDF stays local too; a fallback is a builtin only when it calls no UDF. Elasticsearch requires the builtin for `esquery`, and the converter's LIKE / REGEXP shortcut applies only to builtins. The alternative, the engine keeping every conjunct with a UDF away from all connectors as it does for CASTs, was not chosen: the flag leaves the decision to each connector. Changing the SPI surface moves the plugin API major to 31.0. - **Other paths and branches:** Iceberg drops every function call, and Hive, Hudi, Paimon, MaxCompute and Trino translate none by name, so they are not affected. On branch-4.1 and branch-4.2 the legacy `JdbcScanNode` has the same gap (`JdbcFunctionPushDownRule` checks names only); they have no connector SPI, so this PR does not apply there. - **Behavior change:** Such a filter no longer reaches the remote database, so the remote query returns what the other conjuncts select, without LIMIT, and Doris filters the rest. Connector plugins built against plugin API 30 are refused by this FE until they are rebuilt. ### Testing - New regression suite `external_table_p0/jdbc/test_jdbc_mysql_udf_filter` (a Java UDF registered as `udf_add_one` and as `abs`, on MySQL 5.7): on the unfixed engine its first check fails, MySQL receiving `WHERE (udf_add_one(`id`) = 113)`; by hand on that engine, the query fails with MySQL error 1305, and `mydb.abs(id) = 113` returns no row instead of 112. With the fix both return 112 with no WHERE in the remote query, and the builtin `abs` is still pushed. Run locally against a standalone `mysql:5.7.36` container; the External pipeline runs it (`enableJdbcTest`). - Unit tests, locally: new `ExprToConnectorFunctionCallTest` (fe-core) and `ConnectorFunctionCallTest` (SPI), and new cases in `JdbcQueryBuilderTest` and `EsQueryDslBuilderTest`. All of fe-connector-spi (154), fe-connector-jdbc (275), fe-connector-es (164), and in fe-core the converter tests with `PluginDrivenScanNode*Test` (282), pass. - Seven mutations, each failing a new test: the converter taking every call for a builtin, the LIKE shortcut or the fallback ignoring the flag, `equals` ignoring it, the 3-argument constructor meaning a UDF, and each connector check removed. ### Notes for Reviewers - The 3-argument `ConnectorFunctionCall` constructor still means a builtin; its callers are the converter's arithmetic operators and connector tests. - Rule 6's old example, `my_udf(a, b) = 1` arriving as one fallback, never happened: a UDF call converts to a named call with arguments. The new example is a CASE expression, which the converter does not model. - `getFn()` is not null-checked: plugin scan conjuncts come from Nereids' `ExpressionTranslator`, which always sets the catalog function. - The Lance connector being ported from branch-4.2 (not yet opened) will check the flag for `starts_with`, `ends_with`, `array_contains` and `arrays_overlap`, as 4.2 checks the binary type. ### Release note JDBC catalogs over MySQL, ClickHouse or Oracle no longer send a filter that calls a Java or Python UDF to the remote database, where it failed with "FUNCTION ... does not exist" or was computed by a remote function of the same name; Doris evaluates it. On Elasticsearch catalogs, a UDF named esquery is no longer run as an Elasticsearch query. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: No - Pre-submission status: Partially verified: the regression suite ran against MySQL 5.7 only, not ClickHouse or Oracle; the other JDBC and Elasticsearch suites were not run locally ### AI Disclosure - Tool: Claude Code (Claude Opus 5.5) - Generated: code, tests, this description - Human review: The author reviewed part of the changes. - Prompts: While porting the Lance pushdown from branch-4.2, the agent reported that connectors cannot tell a Java UDF from a builtin of the same name; the author chose to fix it at the SPI boundary in its own PR, with a flag on the function call that each connector checks, and approved an end-to-end run against a MySQL container. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
morningman
added a commit
that referenced
this pull request
Nov 16, 2026
…shdown (#70372) ### Background Connector plugins learn which filters they may push down from a `ConnectorExpression` tree that fe-core's `ExprToConnectorExpressionConverter` builds from a scan's conjuncts. A function call arrives as a `ConnectorFunctionCall` with the function's name, return type and arguments, and connectors that translate calls match it by name: JDBC writes it into the remote SQL when `enable_ext_func_pred_pushdown` is on (the default) and the name passes the database's rules (for MySQL, a blocklist of three names), and Elasticsearch runs `esquery` as a native query. ### What problem does this PR solve? Issue Number: None Related PR: #70369 (changes the same converter and SPI package; independent), #68702 (also changes the connector SPI; whichever merges second takes the next plugin API major) PR Type: Bugfix Problem Summary: A filter that calls a Java or Python UDF on a JDBC table is sent to the remote database as a call of the same name. On MySQL, `WHERE mydb.my_udf(id) = 113` fails with `FUNCTION doris_test.my_udf does not exist`, and a UDF named like a MySQL function, `mydb.abs(id) = 113`, is computed by MySQL instead, which silently drops rows. The converter hands connectors only the function's name, so no connector can tell a UDF from a builtin. This PR adds `ConnectorFunctionCall#isBuiltin()`, set from the catalog function's binary type; JDBC and Elasticsearch no longer translate a call that is not a builtin, and Doris evaluates it. ### Details - **Change map:** +373 −26 in 13 files, of which tests are +303: a regression suite +89 with its `.out` +10, and unit tests +204. The production code is +69 −23: 1. SPI (+42 −13): the flag on `ConnectorFunctionCall`, and Rule 6 of `pushdown/package-info.java` rewritten; plus the plugin API major 30.0 → 31.0 in `fe/fe-connector/pom.xml`. 2. fe-core (+12 −4): `ExprToConnectorExpressionConverter` sets the flag. 3. Connectors (+15 −6): the checks in `JdbcQueryBuilder` and `EsQueryDslBuilder`. - **Frequency:** Always, with function pushdown on: any filter that calls a UDF on a JDBC catalog over MySQL, and over ClickHouse or Oracle when the UDF's name is on their allowlist or in `function_rules`. On Elasticsearch, a UDF named `esquery`. - **Root cause:** `ExprToConnectorExpressionConverter.convertFunctionCall` keeps only `getFnName().getFunction()`, the return type and the arguments, and drops `Function.BinaryType`. `JdbcQueryBuilder.functionCallToSql` then renders any call that `JdbcFunctionPushdownConfig.canPushDown` accepts by name, and `EsQueryDslBuilder` takes any call named `esquery` for the builtin. The SPI documentation said calls with real arguments "are safe to match by name". - **Why existing tests missed it:** `JdbcQueryBuilderTest` and `EsScanPlanProviderTest` build calls by hand, all of them builtins, and no unit test converts a UDF. The only regression suite that uses a UDF on a JDBC table, `test_mysql_varbinary_with_udf`, calls it in a projection; no suite filters with one. - **Fix:** The converter sets the flag where the binary type is still known, and each connector that matches calls by name checks it. JDBC refuses a non-builtin call before rendering anything, so a fallback (an expression the converter does not model, sent as SQL text) that calls a UDF stays local too; a fallback is a builtin only when it calls no UDF. Elasticsearch requires the builtin for `esquery`, and the converter's LIKE / REGEXP shortcut applies only to builtins. The alternative, the engine keeping every conjunct with a UDF away from all connectors as it does for CASTs, was not chosen: the flag leaves the decision to each connector. Changing the SPI surface moves the plugin API major to 31.0. - **Other paths and branches:** Iceberg drops every function call, and Hive, Hudi, Paimon, MaxCompute and Trino translate none by name, so they are not affected. On branch-4.1 and branch-4.2 the legacy `JdbcScanNode` has the same gap (`JdbcFunctionPushDownRule` checks names only); they have no connector SPI, so this PR does not apply there. - **Behavior change:** Such a filter no longer reaches the remote database, so the remote query returns what the other conjuncts select, without LIMIT, and Doris filters the rest. Connector plugins built against plugin API 30 are refused by this FE until they are rebuilt. ### Testing - New regression suite `external_table_p0/jdbc/test_jdbc_mysql_udf_filter` (a Java UDF registered as `udf_add_one` and as `abs`, on MySQL 5.7): on the unfixed engine its first check fails, MySQL receiving `WHERE (udf_add_one(`id`) = 113)`; by hand on that engine, the query fails with MySQL error 1305, and `mydb.abs(id) = 113` returns no row instead of 112. With the fix both return 112 with no WHERE in the remote query, and the builtin `abs` is still pushed. Run locally against a standalone `mysql:5.7.36` container; the External pipeline runs it (`enableJdbcTest`). - Unit tests, locally: new `ExprToConnectorFunctionCallTest` (fe-core) and `ConnectorFunctionCallTest` (SPI), and new cases in `JdbcQueryBuilderTest` and `EsQueryDslBuilderTest`. All of fe-connector-spi (154), fe-connector-jdbc (275), fe-connector-es (164), and in fe-core the converter tests with `PluginDrivenScanNode*Test` (282), pass. - Seven mutations, each failing a new test: the converter taking every call for a builtin, the LIKE shortcut or the fallback ignoring the flag, `equals` ignoring it, the 3-argument constructor meaning a UDF, and each connector check removed. ### Notes for Reviewers - The 3-argument `ConnectorFunctionCall` constructor still means a builtin; its callers are the converter's arithmetic operators and connector tests. - Rule 6's old example, `my_udf(a, b) = 1` arriving as one fallback, never happened: a UDF call converts to a named call with arguments. The new example is a CASE expression, which the converter does not model. - `getFn()` is not null-checked: plugin scan conjuncts come from Nereids' `ExpressionTranslator`, which always sets the catalog function. - The Lance connector being ported from branch-4.2 (not yet opened) will check the flag for `starts_with`, `ends_with`, `array_contains` and `arrays_overlap`, as 4.2 checks the binary type.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: N/A
Related PR: #67470, #66782 (separate Iceberg/Azure work; not included as feature changes here)
Problem Summary:
Doris currently exposes Databricks interoperability through the Iceberg catalog path. This draft adds an experimental native Delta Lake catalog that reads Delta metadata directly with Delta Kernel, rather than relying on an Iceberg mirror.
The draft contains:
type=deltaconnector with path and Unity Catalog adapters; Delta Kernel 4.3.1, Unity Catalog client integration, snapshot/time-travel planning, partition pruning, and an explicit supported-feature gate.delta.write.enabled=true): create, append, overwrite, restricted copy-on-write DELETE/UPDATE/MERGE, TRUNCATE and Unity catalog registration deletion. Catalog-managed snapshots/commits use the Unity/Kernel path.This is a standalone, single-commit branch based on master
0c29961f45e, extracted from local development commit1d957e6c8bb. Course/dbt changes, local configuration, independent Iceberg ADLS normalization and Iceberg Azure dependency additions are excluded.Draft status / validation boundary
This is not ready to merge and does not claim production Databricks support.
Static checks passed on the source development checkout: SPI/Delta/FE-core Checkstyle, connector import and metadata-funnel gates, BE header hygiene, scoped clang-format 16 checks, and Java generation for the modified Thrift definitions. The extracted branch passed code whitespace and scope/content checks. Auto-generated regression .out files retain their original tab-separated formatting (including empty fields and section separators); their results were not edited by hand.
Full FE/BE compilation and FE/BE unit tests have not been rerun on this extracted branch. Full compilation is being handled by the author. Prior local fixture/SQL results are historical evidence only; they are not validation of this draft.
A stale BE test enum reference found while extracting the branch is updated to the current
ObjStorageProvidername. The separate Azure auth source formatting is normalized with clang-format 16. Neither change is a claim of successful compilation.Still required before merge:
Outbound HTTP / privilege surface
Unity REST and object-storage requests are outbound HTTP/SSRF surfaces to endpoints supplied through catalog properties or returned table/storage metadata. Creating or changing the catalog configuration requires catalog CREATE or ALTER privilege; scanning/writing requires the corresponding table privileges and Unity/storage credentials. This connector introduces no URL-based artifact download or dynamic code loading.
Release note
Add an experimental native Delta Lake catalog with path and Unity Catalog adapters. Databricks and multi-cloud production readiness remains unverified.
Check List (For Author)
Check List (For Reviewer who merge this PR)