Skip to content

[fix](catalog) Keep UDF calls out of JDBC and Elasticsearch filter pushdown - #70372

Merged
morningman merged 1 commit into
apache:masterfrom
morningman:b2m-t38h-keep-udf-conjuncts-local
Nov 16, 2026
Merged

morningman merged 1 commit into
apache:masterfrom
morningman:b2m-t38h-keep-udf-conjuncts-local

Conversation

@morningman

Copy link
Copy Markdown
Contributor

Background

Connector plugins learn which filters they may push down from a ConnectorExpression tree that fe-core's ExprToConnectorExpressionConverter builds from a scan's conjuncts. A function call arrives as a ConnectorFunctionCall with the function's name, return type and arguments, and connectors that translate calls match it by name: JDBC writes it into the remote SQL when enable_ext_func_pred_pushdown is on (the default) and the name passes the database's rules (for MySQL, a blocklist of three names), and Elasticsearch runs esquery as a native query.

What problem does this PR solve?

Issue Number: None
Related PR: #70369 (changes the same converter and SPI package; independent), #68702 (also changes the connector SPI; whichever merges second takes the next plugin API major)
PR Type: Bugfix

Problem Summary:

A filter that calls a Java or Python UDF on a JDBC table is sent to the remote database as a call of the same name. On MySQL, WHERE mydb.my_udf(id) = 113 fails with FUNCTION doris_test.my_udf does not exist, and a UDF named like a MySQL function, mydb.abs(id) = 113, is computed by MySQL instead, which silently drops rows. The converter hands connectors only the function's name, so no connector can tell a UDF from a builtin. This PR adds ConnectorFunctionCall#isBuiltin(), set from the catalog function's binary type; JDBC and Elasticsearch no longer translate a call that is not a builtin, and Doris evaluates it.

Details

  • Change map: +373 −26 in 13 files, of which tests are +303: a regression suite +89 with its .out +10, and unit tests +204. The production code is +69 −23:
    1. SPI (+42 −13): the flag on ConnectorFunctionCall, and Rule 6 of pushdown/package-info.java rewritten; plus the plugin API major 30.0 → 31.0 in fe/fe-connector/pom.xml.
    2. fe-core (+12 −4): ExprToConnectorExpressionConverter sets the flag.
    3. Connectors (+15 −6): the checks in JdbcQueryBuilder and EsQueryDslBuilder.
  • Frequency: Always, with function pushdown on: any filter that calls a UDF on a JDBC catalog over MySQL, and over ClickHouse or Oracle when the UDF's name is on their allowlist or in function_rules. On Elasticsearch, a UDF named esquery.
  • Root cause: ExprToConnectorExpressionConverter.convertFunctionCall keeps only getFnName().getFunction(), the return type and the arguments, and drops Function.BinaryType. JdbcQueryBuilder.functionCallToSql then renders any call that JdbcFunctionPushdownConfig.canPushDown accepts by name, and EsQueryDslBuilder takes any call named esquery for the builtin. The SPI documentation said calls with real arguments "are safe to match by name".
  • Why existing tests missed it: JdbcQueryBuilderTest and EsScanPlanProviderTest build calls by hand, all of them builtins, and no unit test converts a UDF. The only regression suite that uses a UDF on a JDBC table, test_mysql_varbinary_with_udf, calls it in a projection; no suite filters with one.
  • Fix: The converter sets the flag where the binary type is still known, and each connector that matches calls by name checks it. JDBC refuses a non-builtin call before rendering anything, so a fallback (an expression the converter does not model, sent as SQL text) that calls a UDF stays local too; a fallback is a builtin only when it calls no UDF. Elasticsearch requires the builtin for esquery, and the converter's LIKE / REGEXP shortcut applies only to builtins. The alternative, the engine keeping every conjunct with a UDF away from all connectors as it does for CASTs, was not chosen: the flag leaves the decision to each connector. Changing the SPI surface moves the plugin API major to 31.0.
  • Other paths and branches: Iceberg drops every function call, and Hive, Hudi, Paimon, MaxCompute and Trino translate none by name, so they are not affected. On branch-4.1 and branch-4.2 the legacy JdbcScanNode has the same gap (JdbcFunctionPushDownRule checks names only); they have no connector SPI, so this PR does not apply there.
  • Behavior change: Such a filter no longer reaches the remote database, so the remote query returns what the other conjuncts select, without LIMIT, and Doris filters the rest. Connector plugins built against plugin API 30 are refused by this FE until they are rebuilt.

Testing

  • New regression suite external_table_p0/jdbc/test_jdbc_mysql_udf_filter (a Java UDF registered as udf_add_one and as abs, on MySQL 5.7): on the unfixed engine its first check fails, MySQL receiving WHERE (udf_add_one(id) = 113); by hand on that engine, the query fails with MySQL error 1305, and mydb.abs(id) = 113 returns no row instead of 112. With the fix both return 112 with no WHERE in the remote query, and the builtin abs is still pushed. Run locally against a standalone mysql:5.7.36 container; the External pipeline runs it (enableJdbcTest).
  • Unit tests, locally: new ExprToConnectorFunctionCallTest (fe-core) and ConnectorFunctionCallTest (SPI), and new cases in JdbcQueryBuilderTest and EsQueryDslBuilderTest. All of fe-connector-spi (154), fe-connector-jdbc (275), fe-connector-es (164), and in fe-core the converter tests with PluginDrivenScanNode*Test (282), pass.
  • Seven mutations, each failing a new test: the converter taking every call for a builtin, the LIKE shortcut or the fallback ignoring the flag, equals ignoring it, the 3-argument constructor meaning a UDF, and each connector check removed.

Notes for Reviewers

  • The 3-argument ConnectorFunctionCall constructor still means a builtin; its callers are the converter's arithmetic operators and connector tests.
  • Rule 6's old example, my_udf(a, b) = 1 arriving as one fallback, never happened: a UDF call converts to a named call with arguments. The new example is a CASE expression, which the converter does not model.
  • getFn() is not null-checked: plugin scan conjuncts come from Nereids' ExpressionTranslator, which always sets the catalog function.
  • The Lance connector being ported from branch-4.2 (not yet opened) will check the flag for starts_with, ends_with, array_contains and arrays_overlap, as 4.2 checks the binary type.

Release note

JDBC catalogs over MySQL, ClickHouse or Oracle no longer send a filter that calls a Java or Python UDF to the remote database, where it failed with "FUNCTION ... does not exist" or was computed by a remote function of the same name; Doris evaluates it. On Elasticsearch catalogs, a UDF named esquery is no longer run as an Elasticsearch query.

Check List (For Author)

  • Behavior changed: Yes (see Details)
  • Does this need documentation: No
  • Pre-submission status: Partially verified: the regression suite ran against MySQL 5.7 only, not ClickHouse or Oracle; the other JDBC and Elasticsearch suites were not run locally

AI Disclosure

  • Tool: Claude Code (Claude Opus 5.5)
  • Generated: code, tests, this description
  • Human review: The author reviewed part of the changes.
  • Prompts: While porting the Lance pushdown from branch-4.2, the agent reported that connectors cannot tell a Java UDF from a builtin of the same name; the author chose to fix it at the SPI boundary in its own PR, with a flag on the function call that each connector checks, and approved an end-to-end run against a MySQL container.

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morningman

Copy link
Copy Markdown
Contributor Author

run buildall

@morningman

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all 13 changed files at head 0f9ca4e4cda734dec5e4ec90661b373b08469fb5, including upstream function binding and downstream filter/residual/LIMIT handling. No new actionable findings or existing blocking inline findings. Review converged in 2 rounds; all three reviewers returned NO_NEW_VALUABLE_FINDINGS in the final round and all risk focuses have evidence-based dispositions.

Critical checkpoint conclusions:

  • Goal and data correctness: builtin/UDF identity survives FE conversion, including LIKE/REGEXP and opaque fallback expressions. JDBC keeps UDF predicates local and suppresses remote LIMIT; Elasticsearch keeps same-named UDFs local while preserving native builtin esquery behavior. Transaction visibility, visible versions, delete bitmaps and BE memory accounting have no changed mechanism.
  • Scope and reuse: the changes are focused on the expression carrier, converter and two name-based consumers, and reuse existing recursive rendering and residual tracking.
  • Concurrency and locks: no new threads, shared mutable state or lock operations; the added identity flag is final.
  • Lifecycle and initialization: no new resources, ownership relationships or static initialization dependencies.
  • Configuration: no configuration item or dynamic-update behavior is added.
  • Compatibility: connector plugin API moves from 30 to 31; the shared version property feeds the SPI resource and plugin manifests, and the loader rejects mismatched majors. Both recorded baseline renderers exclude this carrier, so their rendered surfaces remain unchanged.
  • Thrift/Protobuf: no definitions, field IDs or enum values change.
  • Parallel paths: audited all production carrier constructors and name-based consumers. Arithmetic remains builtin; ordinary functions and fallback carriers preserve identity. Other connector conversion paths do not translate these calls by name.
  • Conditional checks and errors: the guards address concrete UDF execution/name-collision failures and follow existing rejection conventions. Upstream translation binds catalog Functions before the new binary-type read. Native esquery still fails explicitly when it would require unsupported local evaluation.
  • Test coverage: inspected SPI/converter tests for constructor defaults, equality, Java/Python identity, builtin-name collisions, LIKE/REGEXP and CASE fallback; JDBC tests for local predicates and LIMIT; ES tests for native/UDF distinction and residual OR handling. The regression exercises Java UDFs with unique and builtin-colliding names plus a builtin control.
  • Expected results: the new ordered outputs 112, 112, 113 match MySQL fixture IDs 111–115 and IntTest adding one. Setup follows existing UDF jar-copy conventions and retains fixtures for debugging.
  • Observability: existing remote SQL/DSL EXPLAIN output supports diagnosis; no new distributed operation requires metrics or logging.
  • Persistence and failover: no EditLog, replay or transaction state changes.
  • Data writes and atomicity: production changes affect read planning only; no new crash cleanup or write-atomicity path.
  • FE/BE transmission: no new wire field or distributed variable; local residual expressions remain available for BE evaluation.
  • Performance: ordinary calls add constant-time identity checks; opaque fallbacks add one recursive expression walk. Broader remote reads when a UDF stays local are the intended correctness tradeoff.
  • Other risks: rechecked nested Boolean/scalar expressions, arithmetic/time/fallback shortcuts, custom function rules, CASE/lambda descendants, cloned binding and SlotRef source expansion; no unresolved static issue was found.

User focus: no additional focus points were supplied. No existing inline threads required deduplication.

Validation limits: static inspection only; no builds, tests, benchmarks or runtime checks were executed. Python UDFs are covered by converter unit-test source, and the new end-to-end suite targets Java UDFs on MySQL; ClickHouse, Oracle and Elasticsearch end-to-end behavior was traced in code, not executed. No unresolved static coverage gaps remain.

Review converged after 2 round(s).

@morningman

Copy link
Copy Markdown
Contributor Author

run buildall

@morningman

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No new actionable findings at head 0f9ca4e4cda734dec5e4ec90661b373b08469fb5. Static review covered all 13 changed files and the upstream/downstream planning paths. The review converged in 1 round: both complete-scope reviewers and the separate risk reviewer returned NO_NEW_VALUABLE_FINDINGS, and every risk focus has an evidence-based disposition. There are no existing blocking inline findings.

The inspected flow is: Nereids bound catalog function → legacy scan expression → connector expression with builtin identity → JDBC SQL / Elasticsearch DSL → remote subset plus original local residual. Rejected predicates also prevent premature source LIMIT.

Critical checkpoint conclusions:

  1. Goal and correctness: the flag distinguishes UDFs from same-named builtins. Structured calls, LIKE/REGEXP shortcuts and opaque fallback descendants preserve this distinction. JDBC leaves UDF evaluation local; ES preserves native builtin esquery semantics while keeping same-named UDFs local. No changed transaction visibility, partition-version or delete-bitmap mechanism.
  2. Scope and reuse: the carrier, converter and two name-based consumers form a focused change and reuse recursive rejection/residual handling.
  3. Concurrency and locks: no new threads, shared mutable state, lock operations or lock-order changes; the new field is final.
  4. Lifecycle and initialization: no new resources, ownership cycles or static initialization dependencies.
  5. Configuration: no setting added; existing function pushdown configuration remains in use.
  6. Compatibility: plugin API major 30 → 31 is propagated through the shared Maven property into the SPI version resource and plugin manifests; the loader rejects mismatched majors. The carrier is outside both baseline renderers' explicit frozen-type sets, so their recorded surfaces do not change. No storage format or BE function symbol changes.
  7. Thrift/Protobuf: no schema, field ID or enum value changes.
  8. Parallel paths: audited carrier producers and name-based consumers, including scan and EXPLAIN. Other connector translators do not translate these calls by name; arithmetic remains builtin.
  9. Conditions and errors: guards address concrete remote UDF/name-collision failures. Production translation supplies resolved catalog Functions before the binary-type read; nested CASE/lambda traversal was checked. Builtin esquery still fails explicitly when local evaluation would be required.
  10. Test coverage: inspected constructor/equality tests; Java/Python converter identity, LIKE/REGEXP and CASE tests; JDBC local-filter/LIMIT tests; ES native/UDF and residual-OR tests; and the MySQL Java-UDF regression with unique and builtin-colliding names plus a builtin control.
  11. Expected results: ordered outputs 112, 112, 113 match the seeded MySQL IDs 111–115 and IntTest adding one. Regression setup follows existing jar-copy conventions, drops before use and retains debugging state. Output-generation provenance was not independently executed.
  12. Observability: existing SQL/DSL EXPLAIN output, debug logging and explicit ES rejection cover diagnosis; no new distributed operation needs separate instrumentation.
  13. Persistence and failover: no EditLog, replay or transaction-state changes.
  14. Writes and atomicity: production changes affect read planning; no new write/crash cleanup path.
  15. FE/BE transmission: builtin identity stays within the FE connector SPI; original residual expressions remain available for BE evaluation. No wire variable added.
  16. Performance: constant-time identity checks for ordinary calls and one short-circuit tree walk for opaque fallbacks; broader remote reads for local UDF evaluation are the intended correctness tradeoff. No new I/O or per-row allocation.
  17. Further risks: rechecked nested scalar/Boolean calls, arithmetic/time/fallback branches, residual index mapping, source LIMIT and unsafe ES fields. No unresolved source-code candidate remains.

User focus: no additional focus points were supplied. Current inline comments were refreshed before submission; none required deduplication.

Validation limits: static inspection only. No builds, tests, benchmarks, runtime services or plugin loading were executed. The new end-to-end suite targets Java UDFs on MySQL; Python UDF coverage and ClickHouse, Oracle and Elasticsearch behavior were inspected in test/source code, not run. No unresolved static coverage gap remains.

Review converged after 1 round(s).

…shdown

### Background

Connector plugins learn which filters they may push down from a `ConnectorExpression` tree that fe-core's `ExprToConnectorExpressionConverter` builds from a scan's conjuncts. A function call arrives as a `ConnectorFunctionCall` with the function's name, return type and arguments, and connectors that translate calls match it by name: JDBC writes it into the remote SQL when `enable_ext_func_pred_pushdown` is on (the default) and the name passes the database's rules (for MySQL, a blocklist of three names), and Elasticsearch runs `esquery` as a native query.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#70369 (changes the same converter and SPI package; independent), apache#68702 (also changes the connector SPI; whichever merges second takes the next plugin API major)
PR Type: Bugfix

Problem Summary:

A filter that calls a Java or Python UDF on a JDBC table is sent to the remote database as a call of the same name. On MySQL, `WHERE mydb.my_udf(id) = 113` fails with `FUNCTION doris_test.my_udf does not exist`, and a UDF named like a MySQL function, `mydb.abs(id) = 113`, is computed by MySQL instead, which silently drops rows. The converter hands connectors only the function's name, so no connector can tell a UDF from a builtin. This PR adds `ConnectorFunctionCall#isBuiltin()`, set from the catalog function's binary type; JDBC and Elasticsearch no longer translate a call that is not a builtin, and Doris evaluates it.

### Details

- **Change map:** +373 −26 in 13 files, of which tests are +303: a regression suite +89 with its `.out` +10, and unit tests +204. The production code is +69 −23:
  1. SPI (+42 −13): the flag on `ConnectorFunctionCall`, and Rule 6 of `pushdown/package-info.java` rewritten; plus the plugin API major 30.0 → 31.0 in `fe/fe-connector/pom.xml`.
  2. fe-core (+12 −4): `ExprToConnectorExpressionConverter` sets the flag.
  3. Connectors (+15 −6): the checks in `JdbcQueryBuilder` and `EsQueryDslBuilder`.
- **Frequency:** Always, with function pushdown on: any filter that calls a UDF on a JDBC catalog over MySQL, and over ClickHouse or Oracle when the UDF's name is on their allowlist or in `function_rules`. On Elasticsearch, a UDF named `esquery`.
- **Root cause:** `ExprToConnectorExpressionConverter.convertFunctionCall` keeps only `getFnName().getFunction()`, the return type and the arguments, and drops `Function.BinaryType`. `JdbcQueryBuilder.functionCallToSql` then renders any call that `JdbcFunctionPushdownConfig.canPushDown` accepts by name, and `EsQueryDslBuilder` takes any call named `esquery` for the builtin. The SPI documentation said calls with real arguments "are safe to match by name".
- **Why existing tests missed it:** `JdbcQueryBuilderTest` and `EsScanPlanProviderTest` build calls by hand, all of them builtins, and no unit test converts a UDF. The only regression suite that uses a UDF on a JDBC table, `test_mysql_varbinary_with_udf`, calls it in a projection; no suite filters with one.
- **Fix:** The converter sets the flag where the binary type is still known, and each connector that matches calls by name checks it. JDBC refuses a non-builtin call before rendering anything, so a fallback (an expression the converter does not model, sent as SQL text) that calls a UDF stays local too; a fallback is a builtin only when it calls no UDF. Elasticsearch requires the builtin for `esquery`, and the converter's LIKE / REGEXP shortcut applies only to builtins. The alternative, the engine keeping every conjunct with a UDF away from all connectors as it does for CASTs, was not chosen: the flag leaves the decision to each connector. Changing the SPI surface moves the plugin API major to 31.0.
- **Other paths and branches:** Iceberg drops every function call, and Hive, Hudi, Paimon, MaxCompute and Trino translate none by name, so they are not affected. On branch-4.1 and branch-4.2 the legacy `JdbcScanNode` has the same gap (`JdbcFunctionPushDownRule` checks names only); they have no connector SPI, so this PR does not apply there.
- **Behavior change:** Such a filter no longer reaches the remote database, so the remote query returns what the other conjuncts select, without LIMIT, and Doris filters the rest. Connector plugins built against plugin API 30 are refused by this FE until they are rebuilt.

### Testing

- New regression suite `external_table_p0/jdbc/test_jdbc_mysql_udf_filter` (a Java UDF registered as `udf_add_one` and as `abs`, on MySQL 5.7): on the unfixed engine its first check fails, MySQL receiving `WHERE (udf_add_one(`id`) = 113)`; by hand on that engine, the query fails with MySQL error 1305, and `mydb.abs(id) = 113` returns no row instead of 112. With the fix both return 112 with no WHERE in the remote query, and the builtin `abs` is still pushed. Run locally against a standalone `mysql:5.7.36` container; the External pipeline runs it (`enableJdbcTest`).
- Unit tests, locally: new `ExprToConnectorFunctionCallTest` (fe-core) and `ConnectorFunctionCallTest` (SPI), and new cases in `JdbcQueryBuilderTest` and `EsQueryDslBuilderTest`. All of fe-connector-spi (154), fe-connector-jdbc (275), fe-connector-es (164), and in fe-core the converter tests with `PluginDrivenScanNode*Test` (282), pass.
- Seven mutations, each failing a new test: the converter taking every call for a builtin, the LIKE shortcut or the fallback ignoring the flag, `equals` ignoring it, the 3-argument constructor meaning a UDF, and each connector check removed.

### Notes for Reviewers

- The 3-argument `ConnectorFunctionCall` constructor still means a builtin; its callers are the converter's arithmetic operators and connector tests.
- Rule 6's old example, `my_udf(a, b) = 1` arriving as one fallback, never happened: a UDF call converts to a named call with arguments. The new example is a CASE expression, which the converter does not model.
- `getFn()` is not null-checked: plugin scan conjuncts come from Nereids' `ExpressionTranslator`, which always sets the catalog function.
- The Lance connector being ported from branch-4.2 (not yet opened) will check the flag for `starts_with`, `ends_with`, `array_contains` and `arrays_overlap`, as 4.2 checks the binary type.

### Release note

JDBC catalogs over MySQL, ClickHouse or Oracle no longer send a filter that calls a Java or Python UDF to the remote database, where it failed with "FUNCTION ... does not exist" or was computed by a remote function of the same name; Doris evaluates it. On Elasticsearch catalogs, a UDF named esquery is no longer run as an Elasticsearch query.

### Check List (For Author)

- Behavior changed: Yes (see Details)
- Does this need documentation: No
- Pre-submission status: Partially verified: the regression suite ran against MySQL 5.7 only, not ClickHouse or Oracle; the other JDBC and Elasticsearch suites were not run locally

### AI Disclosure

- Tool: Claude Code (Claude Opus 5.5)
- Generated: code, tests, this description
- Human review: The author reviewed part of the changes.
- Prompts: While porting the Lance pushdown from branch-4.2, the agent reported that connectors cannot tell a Java UDF from a builtin of the same name; the author chose to fix it at the SPI boundary in its own PR, with a flag on the function call that each connector checks, and approved an end-to-end run against a MySQL container.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@morningman
morningman force-pushed the b2m-t38h-keep-udf-conjuncts-local branch from 0f9ca4e to 20f2c3c Compare November 13, 2026 08:32
@morningman

Copy link
Copy Markdown
Contributor Author

run buildall

@morningman

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all 13 changed files at head 20f2c3c5d780ecdddb601ae85b18007ce08192ac, including the converter's producers and consumers, JDBC/Elasticsearch scan integration, SPI compatibility, and tests. No new actionable findings or existing blocking inline findings were identified. No additional user focus points were supplied.

The traced execution flow is:

Nereids scalar / Java UDF / Python UDF
  -> bound catalog FunctionCallExpr
  -> ConnectorFunctionCall(name, arguments, builtin)
  -> JDBC SQL / Elasticsearch DSL translation
  -> rejected UDF remains a Doris predicate; source early-stop is disabled

Critical checkpoint conclusions:

Checkpoint Conclusion
Goal and correctness The provenance flag distinguishes UDFs from same-name builtins. Named calls, LIKE/REGEXP shortcuts, and SQL fallbacks preserve that distinction; the added tests exercise the intended behavior.
Scope and reuse Focused changes reuse JDBC's existing refusal path and one shared ES predicate; no unrelated production behavior was added.
Concurrency and locking The added flag is final, per-expression data. No new threads, shared mutable state, locks, heavy work under locks, or lock-order changes.
Lifecycle and initialization No new resources or static initialization dependencies. Existing expression ownership and per-plan lifetimes remain applicable.
Configuration No new configuration items or dynamic-update semantics. Existing function pushdown rules cannot bypass the UDF guard.
Compatibility and upgrades Connector API major moves from 30 to 31 and feeds both the SPI resource and plugin manifests. Older-major directory plugins require rebuilding. Recorded baselines remain accurate because their enumerated surfaces do not include ConnectorFunctionCall and no metadata method changed. No storage-format changes.
Thrift/protobuf No gensrc definitions, field IDs, or enum values changed.
Parallel paths All production ConnectorFunctionCall producers and name-based consumers were checked. JDBC and ES honor the flag; other connector paths do not translate these calls by name. Nested builtin wrappers cannot bypass JDBC refusal.
Conditional checks and errors BinaryType identifies provenance before name matching. JDBC null means unpushable and propagates through operands; ES still fails explicitly when a real builtin esquery would require unsupported local evaluation.
Test coverage Inspected converter tests for Java/Python provenance, LIKE/REGEXP, and CASE fallback; SPI constructor/equality tests; JDBC fallback/AND/LIMIT and builtin controls; ES same-name UDF and OR residual cases; and the MySQL regression suite. Shared dialect and NOT/nested paths were also traced statically.
Changed expected results Both UDF results of 112 and builtin abs result of 113 agree with IntTest's input-plus-one implementation and the checked-in MySQL fixture IDs 111 through 115. Ordered output is used. Output generation was not independently repeated.
Observability Existing EXPLAIN remote SQL/DSL, scan properties, and explicit esquery error provide useful visibility; this change adds no new distributed operation requiring metrics.
Transactions and persistence No transaction, EditLog, failover, or persistence path changes.
Writes, atomicity, and crash behavior No production data-write or resource-management path changes. Read correctness is preserved by retaining local UDF predicates and disabling premature remote LIMIT/early-stop.
FE/BE propagation No new FE/BE variable or wire field; the flag stays at the FE connector boundary.
Performance and allocation Constant-time provenance checks and bounded expression traversal add no material allocation or locking concern. Keeping UDF filters local can increase source reads, as required for correctness.
Other risks and core invariants Function-bearing production inputs carry bound catalog metadata; fallback traversal sees nested Java/Python calls. No new visibility, version, delete-bitmap, memory-safety, or error-handling defect was established.

Validation was static inspection only: no builds, style gates, unit/regression tests, benchmarks, or external database executions were run. Runtime behavior and the external database matrix were not independently validated. All assigned static scopes and risk questions are covered, with no unresolved candidates or static coverage gaps. Review converged in two rounds.

Review converged after 2 round(s).

@morningman

Copy link
Copy Markdown
Contributor Author

run external

@morningman
morningman merged commit e9caad5 into apache:master Nov 16, 2026
34 of 35 checks passed
OIiveirra added a commit to OIiveirra/doris that referenced this pull request Nov 16, 2026
### Background

The spatial-type branch adds typed Geography evaluation and connector metadata.
Current master adds relation-function constant caching and connector API 31.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#69032 (follow-up), apache#70302 (constant shape caching), apache#70372 (connector API 31)
PR Type: Build/CI/Chore

Problem Summary:

Merge current master and reconcile its constant shape cache with typed WKB
admission and NULL evaluation. Keep the spatial connector surface on API 32
after upstream API 31.

### Details

- **Affected:** The spatial relation implementation, geometry state, conflict-local tests, and connector API version pins.
- **Developer and release impact:** The merged branch retains constant caching and the spatial calculation contracts. Connector plugins use API 32.
- **License and third-party:** This merge introduces no extra dependency changes.
- **Rollback:** Revert this merge while retaining the branch's existing commits.

### Testing

- Ran 399 BE ASAN tests across VGeoFunctionsTest, GeoTypesTest, IcebergV2ReaderTest and NewParquetReaderTest.
- Ran 155 connector SPI tests and 14 PluginApiVersionWiringTest tests.
- Checked changed C++ lines with clang-format 16 and clang-tidy, plus build hygiene and FE Checkstyle.

### Release note

None

### Check List (For Author)

- Behavior changed: Yes (see Details)
- Does this need documentation: No
- Pre-submission status: Ready

### AI Disclosure

- Tool: OpenAI Codex
- Generated: Merge resolution, focused regression coverage and commit message
- Human review: Pending
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants