Repository navigation
[improvement](external) Hand ARRAY literals to connectors as element lists - #70369
Conversation
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
|
/review |
There was a problem hiding this comment.
Reviewed all eight changed files at head 5fcad09. One new P2 finding: recursive ARRAY conversion rejects a supported binary-prepared zero-date value before local BE evaluation. No P0/P1 findings or existing blocking inline findings were identified.
Flow inspected: planner/parameter literals -> legacy ArrayLiteral -> shared connector expression conversion -> connector pushdown or Doris residual. Nereids DML/conflict and unbound EXECUTE literal paths share the converter.
Critical checkpoint conclusions:
- Goal and correctness: structured, unmodifiable ARRAY element values are implemented, including nested arrays and nulls; the prepared zero-date exception is the verified limitation described inline.
- Scope and reuse: the change is focused; recursion reuses existing scalar conversion and both connector guards reuse unsupported-predicate handling.
- Concurrency and locking: conversion allocates query-local lists; no new threads, shared mutable state, locks or lock ordering changes.
- Lifecycle and initialization: fresh lists and scalar values are garbage collected; no new external resources, cycles or static initialization dependencies.
- Configuration: no configuration is added; the finding concerns existing allow_zero_date=true behavior.
- Compatibility: no public signatures, wire fields or persisted formats change. The expanded Object value domain was traced through built-in readers and plugin surface gates; no additional supported-reader regression was confirmed.
- Thrift/protobuf: no definitions or field IDs change.
- Parallel paths: scan, Nereids and unbound converters use the shared encoding; built-in ADBC, Iceberg, Paimon, MaxCompute and Trino reject unsupported ARRAY predicates, while Hive/Hudi partition paths use scalar keys.
- Conditional checks: JDBC rejects List values centrally, including function arguments; ES rejects whole-array comparisons. Planner legality rejects ARRAY IN/NOT IN before normal ES scan conversion.
- Residuals and LIMIT: ES retains failed Boolean conjuncts in Doris; JDBC preserves safe sibling conjuncts and suppresses remote LIMIT/TOP/ROWNUM when any filter remains local.
- Test coverage: new converter tests cover ambiguous strings, scalar shapes and precision, LARGEINT fallback, null/nested/empty arrays and immutability; JDBC/ES tests cover local fallback and partial AND. Binary-prepared zero-date coverage is missing.
- Test results: assertions were inspected for correctness; no regression .out files change. Author-reported unit, mutation and end-to-end results were not independently executed.
- Observability: existing predicate/EXPLAIN and error reporting are adequate for the pure conversion and fallback paths; no additional metric is required.
- Persistence and failover: no journal, metadata persistence or transaction changes.
- Writes and atomicity: no write or transaction lifecycle changes; conflict/procedure literal encoding was traced for consistency.
- FE/BE propagation: no new transported variables or send paths.
- Performance and memory: conversion is linear in total elements and owns fresh recursively unmodifiable lists; the guards are constant-time. Additional Doris filtering is necessary for whole-array semantics.
- Core invariants and further review: visible versions, committed visibility and MoW delete bitmaps are unaffected. Child literal casts are guaranteed by array producers. Error handling for the newly reached zero-date conversion needs the inline fix; no other verified issue remains.
No additional user focus was supplied. Review converged in round 2 after independent complete-scope and risk-focused review plus challenge of this frozen payload. Static inspection only: no builds, tests, benchmarks or live JDBC/Elasticsearch execution were performed; source files were not modified.
Review converged after 2 round(s).
| private static List<Object> arrayElements(ArrayLiteral literal) { | ||
| List<Object> elements = new ArrayList<>(literal.getChildren().size()); | ||
| for (Expr element : literal.getChildren()) { | ||
| elements.add(convertLiteral((LiteralExpr) element).getValue()); |
There was a problem hiding this comment.
[P2] Keep zero-date array parameters on the local evaluation path
With BE allow_zero_date=true, a binary prepared DATE parameter encoded with length 4 and zero year/month/day is supported: Literal.handleDateLiteral creates DateV2Literal(0,0,0), and FoldConstantRuleOnFE.visitArray preserves it in array(?). For a Hive scan with WHERE array_contains(array(?), date_col), the previous conversion left this generic predicate for BE, whose date decoder normalizes 0000-00-00 to 0000-01-01. This recursive call now reaches convertDateLiteral / LocalDate.of(0,0,0) and throws during scan planning, before the connector can decline pushdown. Preserve local evaluation when an array element cannot be represented in the SPI scalar domain, and cover binary-bound zero dates; the new date tests use canonical parsed dates only.
There was a problem hiding this comment.
Thanks, the path is real, but I'm leaving it as is in this PR. The array form now fails at the same call where the scalar form of the same parameter already fails on master, and the one query that passed before needs a non-default BE setting.
Reproduced on a local cluster whose FE has master's handleDateLiteral, converter and PluginDrivenScanNode: an Iceberg table with a DATE column, COM_STMT_EXECUTE with a 4-byte DATE whose year, month and day are 0.
| Predicate | BE allow_zero_date |
master | this PR |
|---|---|---|---|
d = ? |
false or true | FE: Invalid value for MonthOfYear (valid values 1 - 12): 0, from convertDateLiteral → LocalDate.of |
same |
array_contains(array(?), d) |
false (default) | BE: [E33] Invalid value: 0000-00-00 for type DateV2 |
the FE error above |
array_contains(array(?), d) |
true | returns the 0000-01-01 row | the FE error above |
Only the last row changes, and config.h recommends keeping allow_zero_date false. A 4-byte 2024-00-00 fails on the BE even with it on. libmysqlclient sends an all-zero MYSQL_TIME with length 0, which handleDateLiteral decodes as 0000-01-01, and the text form cast('0000-00-00' as date) folds to NULL.
After this PR an element converts exactly as the same literal does on its own, which is what the PR sets out to do. Keeping dates that LocalDate cannot represent in Doris has to cover d = ? as well, so it belongs with the binary DATE decoding (it accepts dates the text parser turns into NULL), not in this PR.
|
run buildall |
1 similar comment
|
run buildall |
|
/review |
There was a problem hiding this comment.
Reviewed all 12 changed files at head 21ba4db. Two new P2 findings: the new ARRAY value domain breaks older JDBC plugins admitted by the unchanged API major, and a cast BOOLEAN array can wrongly prune a matching Hudi string partition. No new P0/P1 findings or existing P0/P1 inline findings were identified. Existing P2 thread 4502548727 remains applicable to binary zero-date arrays; the author's reply and non-default allow_zero_date condition were considered, and that issue is not duplicated.
Flow inspected: planner and parameter literals -> legacy ArrayLiteral -> shared connector expression converter -> connector predicate consumer -> remote filter plus Doris residual and LIMIT handling. The Nereids DML/conflict and unbound EXECUTE paths share literal conversion.
Critical checkpoint conclusions:
- Goal and correctness: ARRAY values now carry recursively unmodifiable element lists, preserving commas, quotes, nulls, nested and empty arrays. The inline findings concern older plugin compatibility and Hudi partition correctness after CAST stripping; the known zero-date exception remains separate.
- Scope and reuse: production changes are focused and reuse scalar conversion and existing unsupported-predicate handling.
- Concurrency and locks: fresh query-local lists introduce no shared mutable state, threads, locks or lock-order changes.
- Lifecycle and initialization: lists and scalar values are garbage collected; no new external resources, reference cycles or static initialization dependencies.
- Configuration: no configuration is added or changed; the existing zero-date setting only qualifies the already-known concern.
- Compatibility: no public method, wire or storage signature changes, but the Object payload domain changes. Same-major plugin loading remains allowed and an older JDBC builder cannot render the new List safely; see the new inline finding.
- Thrift/protobuf: no definitions, IDs or enum values change.
- Parallel paths: scan, Nereids conflict/delete and unbound EXECUTE conversions share encoding. Current ADBC, Iceberg, Paimon, MaxCompute and Trino readers decline unsupported ARRAY values; Hive/Hudi partition paths use scalar keys, but CAST stripping can expose a new List to Hudi string-key extraction; see the second finding.
- Conditions and special cases: JDBC rejects List centrally across operands/functions/Boolean forms. ES rejects whole-array comparisons on either side, including negated forms. ARRAY IN is rejected by planner legality checks before normal scan conversion.
- Residuals and LIMIT: JDBC retains local predicates and disables remote LIMIT/TOP/ROWNUM when any filter fails rendering. ES not-pushed indices preserve local conjuncts and disable terminate_after. Independent scalar AND conjuncts remain pushable.
- Test coverage: converter unit tests cover scalar shapes and precision, LARGEINT text, ambiguous strings, null/nested/empty arrays and immutability. Builder unit tests and both regression suites cover rejection, partial AND, OR/function paths and LIMIT. Older-plugin artifact compatibility, cast BOOLEAN-array Hudi pruning, and binary zero-date parameters are not covered by the added tests.
- Test results: all new .out rows were checked against source fixtures: remote Doris INSERT rows, PostgreSQL/ClickHouse initialization data and both ES versions' documents. Ordered queries and test setup were inspected. Output regeneration and author-reported results were not independently executed.
- Observability: existing EXPLAIN remote query/predicate reporting and conversion errors support diagnosis; no new metric or logging path is needed for this pure conversion change.
- Persistence and failover: no journal or persisted metadata changes; transaction/version ownership is unchanged.
- Writes and atomicity: no row-write or commit lifecycle changes. Shared conflict/procedure literal conversion and unsupported-value fallback were traced.
- FE/BE propagation: no new transported variables or send paths; BE still evaluates rejected predicates.
- Performance and memory: conversion performs one pass over elements with fresh Java list storage; guards are constant-time. Additional local filtering preserves ARRAY semantics. No BE allocation tracking change.
- Core invariants and other issues: transaction visibility/version management and MoW delete-bitmap mechanics are unchanged; Hudi query-result correctness is affected as described in the second finding. Array producers guarantee LiteralExpr children. No additional substantiated issue beyond the two new findings and known zero-date thread.
No additional user focus was supplied. Static inspection only: no builds, tests, benchmarks or live external-source queries were run, and no repository source files were modified.
Review converged after 3 round(s).
| * javadoc (Rule 4) for the full Doris-type-to-Java-class table: {@code null}, {@code Boolean}, {@code Long}, | ||
| * {@code Double}, {@code BigDecimal}, {@code String}, {@code LocalDate}, {@code LocalDateTime}.</p> | ||
| * {@code Double}, {@code BigDecimal}, {@code String}, {@code LocalDate}, {@code LocalDateTime}, and for an | ||
| * {@code ARRAY} an unmodifiable {@code List} whose elements take these same shapes.</p> |
There was a problem hiding this comment.
[P2] Bump the plugin API for the new ARRAY value shape
Changing the value from String to List breaks already-built connectors while connector.plugin.api.version remains 30.0. ApiVersionGate therefore still admits a JDBC plugin built from the base, and the parent-first SPI supplies it these new values. That plugin's old literalToSql falls through to val.toString(): a previously working remote-Doris predicate tags = ['a', 'b'] becomes tags = [a, b] with unresolved identifiers, instead of the quoted '[a, b]' that Doris casts to the array. The List guard fixes only rebuilt plugins. Bump the connector API major and its pinned test, or negotiate the literal shape, so an FE upgrade cannot silently load this incompatible reader.
There was a problem hiding this comment.
Thanks. I'm keeping 30.0: the major tracks the SPI surface, and this PR does not change it.
- The pom comment and
ApiVersionGatedefine a major as a change to the set of SPI elements a plugin can see, additions included, andConnectorPluginSurfaceTest#connectorApiMajorTracksTheRecordedSurfaceChangeties it to the recorded baseline; master bumps it exactly when that baseline changes. This PR adds, removes and changes no type, method or enum constant. What changes is the documented value ofConnectorLiteral#getValue()for ARRAY, which used to be display text. - The pair you describe needs a JDBC plugin built before this PR next to an FE built after it, and no release can ship one: the connector SPI exists only on master (branch-4.x has no fe-connector), and build.sh builds every in-tree connector with the FE and deploys it under
fe/plugins/connector/. - That text was not a working encoding to preserve: PostgreSQL and ClickHouse reject
'[a, b]', and a remote Doris reads elements containing,or NULL back as other elements;test_jdbc_array_literal_predicatefails on master for each.
| } else if (literal instanceof DateLiteral) { | ||
| return convertDateLiteral((DateLiteral) literal, ct); | ||
| } else if (literal instanceof ArrayLiteral) { | ||
| return new ConnectorLiteral(ct, arrayElements((ArrayLiteral) literal)); |
There was a problem hiding this comment.
[P2] Keep cast ARRAY predicates out of Hudi partition pruning
For a Hudi table in an HMS catalog with a STRING partition p='[1]', this plan should retain the matching rows:
Filter(p = CAST([TRUE] AS STRING))
HudiScan
The cast remains after FE folding (BE folding defaults off). During physical scan planning, convertPredicate calls this converter, which strips CastExpr; this branch now produces List<Boolean> instead of the previous text [1]. HudiConnectorMetadata.extractLiteralValue stringifies it as [true] and excludes the [1] partition. Hudi stores the empty pruned paths and plans zero ranges, so the residual cannot recover the rows; the actual BE cast renders [1]. Refuse this cast predicate before partition pushdown, or make the scalar partition reader reject ARRAY literals, and cover this case.
There was a problem hiding this comment.
Thanks, confirmed, and fixed in 3162b0e. PluginDrivenScanNode.convertPredicate hands every conjunct to applyFilter (the cast switch does not apply there), the converter strips the CAST, and Hudi's extractLiteralValue turned the element list into [true]. Master's text [1] matched the BE cast only by accident: for CAST(['a'] AS STRING) both forms differ from BE's ["a"], so master already pruned that partition away.
A literal of type ARRAY, MAP or STRUCT is never a partition value, so Hudi and its Hive twin now skip it: an EQ with one is not a partition predicate, and an IN prunes only when every item is a scalar literal, since dropping just that item would narrow the list. The conjunct stays with Doris like any predicate the pruner does not understand. HudiPartitionPruningTest#complexLiteralIsNeverAPartitionValue and HiveConnectorMetadataPartitionPruningTest#testComplexLiteralIsNeverAPartitionValue fail without the check: the [1] partition is dropped.
…lists ### Background A catalog connector learns which filters it may push down from an engine-neutral expression tree (`ConnectorExpression`) that fe-core's `ExprToConnectorExpressionConverter` builds from the planner's conjuncts. A literal in that tree carries a plain Java value whose shape the SPI documents per Doris type (Rule 4 in `connector/spi/pushdown/package-info.java`): `Long` for integers, `LocalDate` for dates, and so on. Types outside that table, ARRAY among them, arrive as the literal's display text. ### What problem does this PR solve? Issue Number: None Related PR: apache#68687 (the Lance array-membership pushdown on branch-4.2, whose master port needs the elements) PR Type: Enhancement Problem Summary: An ARRAY literal reaches a connector as text such as `[red, a, b]`, with unquoted elements, so `['a, b']` and `['a', 'b']` cannot be told apart. The Lance connector needs the elements to push down `arrays_overlap`, which the planner makes from three or more ORed `array_contains`. This PR makes the value an unmodifiable `List` of element values, each in the shape the SPI gives its element type. An audit of the connectors that read literal values found two that would treat the list as text: JDBC would write it into remote SQL unquoted, and Elasticsearch already pushes array comparisons as a term query on the text. Both now leave such predicates to Doris. ### Details - **Before and after:** For `['red', 'a, b', NULL]`, `ConnectorLiteral#getValue()` was the String `[red, a, b, NULL]` and is now the list `["red", "a, b", null]`; nested arrays are nested lists, and a NULL array is still a NULL literal. An element has the value it would have as a literal of its own (a LARGEINT element is still a decimal String). JDBC: `tags = ['a', 'b']`, or on MySQL `arrays_overlap(tags, ['a', 'b'])`, is no longer sent to the source, where it went as the string `'[a, b]'`. Elasticsearch (docvalue mode): `tags = ['red', 'blue']` is no longer pushed as `{"term":{"tags":"[red, blue]"}}`. - **Configs and defaults:** None. - **Impact on existing users:** JDBC catalogs filter such rows in Doris instead of the source, and the query no longer pushes LIMIT. Elasticsearch drops a pushed comparison from what Doris evaluates, and a term query matches any one value of a multi-valued field rather than the whole array, so `=` could lose rows and `!=` could add some; this follows from the term query's semantics and was not reproduced on a live cluster. ADBC, Iceberg, Paimon, MaxCompute and Trino already refuse ARRAY literals, and Hive and Hudi partition pruning never sees one. No SPI type or method changes, so the connector plugin API version stays. ### Testing All run locally on this commit's code: - New `ExprToConnectorExpressionConverterTest` (fe-core): elements with commas and quotes, every scalar shape, NULL elements, nested and empty arrays, an unmodifiable list, a NULL array. 4 tests pass; with the new branch removed (master's behavior) all 4 fail. - `JdbcQueryBuilderTest#testArrayLiteralsRemainLocalWithoutLimit` (MySQL, PostgreSQL, ClickHouse, Trino): no WHERE or LIMIT for a comparison, NOT or `arrays_overlap` with an ARRAY literal; an AND keeps its other conjunct. Without the new check the builder emits ``WHERE (`tags` = [a, b, x') OR ('1'='1]) LIMIT 1``. - `EsQueryDslBuilderTest#arrayLiteralComparisonsRemainLocal`: `=`, `!=`, `<` on either side stay in Doris; in an AND the other conjunct is still pushed. - Six mutations (text instead of the list, a modifiable list, elements as text, NULL elements dropped, each connector check removed), each against its test class beside an unmutated control: every one fails a new test. - Existing suites pass: fe-connector-spi (149, with `ConnectorPluginSurfaceTest`), fe-connector-es (165), fe-connector-jdbc (276), and in fe-core the converters that route through this one plus `PluginDrivenScanNode*Test` (317). - End to end, on a local cluster whose FE also carries the Lance connector PR that builds on this one: `test_lance_array_predicate_pushdown` pushes `arrays_overlap(labels, [...])` into Lance, up to the 64-needle index budget, and its BE profile shows the scalar index segments searched. ### Notes for Reviewers - `NereidsToConnectorExpressionConverter` (DELETE / UPDATE / MERGE) and `UnboundExpressionToConnectorPredicateConverter` (`EXECUTE ... WHERE`) convert literals through `ExprToConnectorExpressionConverter`, so all three paths get the list. - `convertLiteral` and `convertDateLiteral` now declare `ConnectorLiteral`, which they always returned, so the element loop needs no cast. ### Release note For JDBC and Elasticsearch catalogs, a filter that compares a column with an ARRAY constant, such as `tags = ['a', 'b']`, and for JDBC a function call that takes one, is now evaluated by Doris instead of being sent to the source. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: No - Pre-submission status: Partially verified: not run against live JDBC sources or Elasticsearch ### AI Disclosure - Tool: Claude Code (Claude Opus 5.5) - Generated: code, tests, this description - Human review: The author reviewed part of the changes. - Prompts: While porting the Lance pushdown from branch-4.2, the agent proposed handing ARRAY literals to connectors as element lists; the author approved that plan: change the value, document it in the SPI, audit every connector first, and send it as its own PR.
…DBC and ES ### Background A catalog connector gets the filters of a scan as a `ConnectorExpression` and sends the part its source can evaluate along with the scan: JDBC writes it into the remote SQL, which EXPLAIN shows as `QUERY:`, and Elasticsearch into the query DSL of its scroll requests. Doris evaluates the rest itself, shown as `PREDICATES:`. ### What problem does this PR solve? Issue Number: None Related PR: None PR Type: Test Problem Summary: The previous commit leaves a filter with an ARRAY literal to Doris on JDBC and Elasticsearch catalogs, and its tests checked only the query builders. These suites run such filters against live sources. On master most of them return too few rows or fail; with the change every one returns the rows the source holds. ### Details - **Change map:** 328 added lines, all tests: two regression suites (228) and the `.out` files the framework generated for them (100). - **Scenarios:** `test_jdbc_array_literal_predicate` (p0, external) reads three JDBC sources. A remote Doris, which is the cluster itself and needs no environment: comparisons whose elements contain `, ` or NULL, INT and nested arrays, `arrays_overlap`, the three ORed `array_contains` the planner rewrites into it, an AND whose other conjunct still reaches the source, an OR, and a LIMIT. Under `enableJdbcTest`, PostgreSQL's `catalog_pg_test.test_array_null` and ClickHouse's `doris_test.arr`, which the docker fixtures already create. EXPLAIN checks that the remote query carries no array filter and no LIMIT. `test_es_array_literal_predicate` (p2, like the other ES suites) compares the keyword and long arrays of index `test1` on ES 7 and 8 in a docvalue scan: `=`, `!=`, an AND, and a LIMIT that must not become `terminate_after`. - **What existing tests miss:** no suite filtered an external ARRAY column with an ARRAY constant. - **Fails when broken:** with master's converter, which hands the literal over as text, the remote Doris part fails at its first block (no row instead of row 1), the PostgreSQL part with `malformed array literal: "[hello, world]"`, the ClickHouse part with `Cannot parse quoted string ... while converting '[string]' to Array(String)`, and the ES suite at its first block (no row instead of five). With only the JDBC check removed, the remote Doris receives ``WHERE (`tags` = [a, b])`` and rejects it; with only the ES check removed, the ES suite fails as it does with master's converter. ### Testing Locally, on a single-FE, single-BE cluster whose FE was built from this branch: both suites generated their `.out` and pass again in compare mode. Each failure above came from the same build with one class replaced by its master or check-less version, recompiled with javac; the unchanged source recompiles to the jar's bytecode (`javap -c -p`, no difference). PostgreSQL 14 and ClickHouse 23.8 ran as single containers loaded with the two tables copied from the docker init scripts, and ES 7.17.5 and 8.3.3 as single-node containers loaded with `test1` the way `es_init.sh` does. ### Release note None ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Ready ### AI Disclosure - Tool: Claude Code (Claude Opus 5.5) - Generated: these suites and this message - Human review: not stated Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…shdown ### Background Connector plugins learn which filters they may push down from a `ConnectorExpression` tree that fe-core's `ExprToConnectorExpressionConverter` builds from a scan's conjuncts. A function call arrives as a `ConnectorFunctionCall` with the function's name, return type and arguments, and connectors that translate calls match it by name: JDBC writes it into the remote SQL when `enable_ext_func_pred_pushdown` is on (the default) and the name passes the database's rules (for MySQL, a blocklist of three names), and Elasticsearch runs `esquery` as a native query. ### What problem does this PR solve? Issue Number: None Related PR: apache#70369 (changes the same converter and SPI package; independent), apache#68702 (also changes the connector SPI; whichever merges second takes the next plugin API major) PR Type: Bugfix Problem Summary: A filter that calls a Java or Python UDF on a JDBC table is sent to the remote database as a call of the same name. On MySQL, `WHERE mydb.my_udf(id) = 113` fails with `FUNCTION doris_test.my_udf does not exist`, and a UDF named like a MySQL function, `mydb.abs(id) = 113`, is computed by MySQL instead, which silently drops rows. The converter hands connectors only the function's name, so no connector can tell a UDF from a builtin. This PR adds `ConnectorFunctionCall#isBuiltin()`, set from the catalog function's binary type; JDBC and Elasticsearch no longer translate a call that is not a builtin, and Doris evaluates it. ### Details - **Change map:** +373 −26 in 13 files, of which tests are +303: a regression suite +89 with its `.out` +10, and unit tests +204. The production code is +69 −23: 1. SPI (+42 −13): the flag on `ConnectorFunctionCall`, and Rule 6 of `pushdown/package-info.java` rewritten; plus the plugin API major 30.0 → 31.0 in `fe/fe-connector/pom.xml`. 2. fe-core (+12 −4): `ExprToConnectorExpressionConverter` sets the flag. 3. Connectors (+15 −6): the checks in `JdbcQueryBuilder` and `EsQueryDslBuilder`. - **Frequency:** Always, with function pushdown on: any filter that calls a UDF on a JDBC catalog over MySQL, and over ClickHouse or Oracle when the UDF's name is on their allowlist or in `function_rules`. On Elasticsearch, a UDF named `esquery`. - **Root cause:** `ExprToConnectorExpressionConverter.convertFunctionCall` keeps only `getFnName().getFunction()`, the return type and the arguments, and drops `Function.BinaryType`. `JdbcQueryBuilder.functionCallToSql` then renders any call that `JdbcFunctionPushdownConfig.canPushDown` accepts by name, and `EsQueryDslBuilder` takes any call named `esquery` for the builtin. The SPI documentation said calls with real arguments "are safe to match by name". - **Why existing tests missed it:** `JdbcQueryBuilderTest` and `EsScanPlanProviderTest` build calls by hand, all of them builtins, and no unit test converts a UDF. The only regression suite that uses a UDF on a JDBC table, `test_mysql_varbinary_with_udf`, calls it in a projection; no suite filters with one. - **Fix:** The converter sets the flag where the binary type is still known, and each connector that matches calls by name checks it. JDBC refuses a non-builtin call before rendering anything, so a fallback (an expression the converter does not model, sent as SQL text) that calls a UDF stays local too; a fallback is a builtin only when it calls no UDF. Elasticsearch requires the builtin for `esquery`, and the converter's LIKE / REGEXP shortcut applies only to builtins. The alternative, the engine keeping every conjunct with a UDF away from all connectors as it does for CASTs, was not chosen: the flag leaves the decision to each connector. Changing the SPI surface moves the plugin API major to 31.0. - **Other paths and branches:** Iceberg drops every function call, and Hive, Hudi, Paimon, MaxCompute and Trino translate none by name, so they are not affected. On branch-4.1 and branch-4.2 the legacy `JdbcScanNode` has the same gap (`JdbcFunctionPushDownRule` checks names only); they have no connector SPI, so this PR does not apply there. - **Behavior change:** Such a filter no longer reaches the remote database, so the remote query returns what the other conjuncts select, without LIMIT, and Doris filters the rest. Connector plugins built against plugin API 30 are refused by this FE until they are rebuilt. ### Testing - New regression suite `external_table_p0/jdbc/test_jdbc_mysql_udf_filter` (a Java UDF registered as `udf_add_one` and as `abs`, on MySQL 5.7): on the unfixed engine its first check fails, MySQL receiving `WHERE (udf_add_one(`id`) = 113)`; by hand on that engine, the query fails with MySQL error 1305, and `mydb.abs(id) = 113` returns no row instead of 112. With the fix both return 112 with no WHERE in the remote query, and the builtin `abs` is still pushed. Run locally against a standalone `mysql:5.7.36` container; the External pipeline runs it (`enableJdbcTest`). - Unit tests, locally: new `ExprToConnectorFunctionCallTest` (fe-core) and `ConnectorFunctionCallTest` (SPI), and new cases in `JdbcQueryBuilderTest` and `EsQueryDslBuilderTest`. All of fe-connector-spi (154), fe-connector-jdbc (275), fe-connector-es (164), and in fe-core the converter tests with `PluginDrivenScanNode*Test` (282), pass. - Seven mutations, each failing a new test: the converter taking every call for a builtin, the LIKE shortcut or the fallback ignoring the flag, `equals` ignoring it, the 3-argument constructor meaning a UDF, and each connector check removed. ### Notes for Reviewers - The 3-argument `ConnectorFunctionCall` constructor still means a builtin; its callers are the converter's arithmetic operators and connector tests. - Rule 6's old example, `my_udf(a, b) = 1` arriving as one fallback, never happened: a UDF call converts to a named call with arguments. The new example is a CASE expression, which the converter does not model. - `getFn()` is not null-checked: plugin scan conjuncts come from Nereids' `ExpressionTranslator`, which always sets the catalog function. - The Lance connector being ported from branch-4.2 (not yet opened) will check the flag for `starts_with`, `ends_with`, `array_contains` and `arrays_overlap`, as 4.2 checks the binary type. ### Release note JDBC catalogs over MySQL, ClickHouse or Oracle no longer send a filter that calls a Java or Python UDF to the remote database, where it failed with "FUNCTION ... does not exist" or was computed by a remote function of the same name; Doris evaluates it. On Elasticsearch catalogs, a UDF named esquery is no longer run as an Elasticsearch query. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: No - Pre-submission status: Partially verified: the regression suite ran against MySQL 5.7 only, not ClickHouse or Oracle; the other JDBC and Elasticsearch suites were not run locally ### AI Disclosure - Tool: Claude Code (Claude Opus 5.5) - Generated: code, tests, this description - Human review: The author reviewed part of the changes. - Prompts: While porting the Lance pushdown from branch-4.2, the agent reported that connectors cannot tell a Java UDF from a builtin of the same name; the author chose to fix it at the SPI boundary in its own PR, with a flag on the function call that each connector checks, and approved an end-to-end run against a MySQL container. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
21ba4db to
ddbcd42
Compare
|
run buildall |
|
/review |
There was a problem hiding this comment.
Static review of all 12 changed files at ddbcd42 found one additional P2 correctness issue: Elasticsearch's IN/pattern literal readers can still consume the new ARRAY List shape after a scalar cast is stripped. The inline finding gives the reachable plan and wrong-row consequence. Changes are warranted.
Already-known issues were independently revalidated and suppressed: plugin API compatibility, Hudi cast partition pruning, and zero-date binary ARRAY parameters. The last issue remains limited to the acknowledged nondefault allow_zero_date scenario; the author's reply was considered. All three are P2, so there are no existing P0/P1 IDs to carry forward. The new finding concerns a separate Elasticsearch IN/pattern consumer, rather than the already-reported partition reader.
Applicable code-review checkpoints:
| Checkpoint | Conclusion |
|---|---|
| Goal and correctness | Typed, immutable ARRAY element lists preserve scalar shapes, NULL elements, nesting and element boundaries. JDBC/ES comparison fallback is implemented, but ES IN/pattern fallback remains incomplete as reported inline. |
| Scope and reuse | Small production change; recursive conversion reuses the scalar converter, and parallel literal entry points reuse it too. |
| Concurrency and locks | New lists are request-local and unmodifiable; no new shared mutation, threads, lock acquisitions or lock-order change. |
| Lifecycle and initialization | Lists own fresh storage and immutable scalar values; no external-resource, static-initialization or release-order change. |
| Configuration | No configuration added or default changed. |
| Compatibility and rolling upgrades | The String-to-List plugin contract changes while API major stays 30; the existing compatibility thread still applies. No on-disk format change. |
| Thrift/protobuf | No gensrc definitions or wire field IDs changed. |
| Parallel paths | Scan, DELETE/UPDATE/MERGE and EXECUTE literals share conversion; write/EXECUTE conversion exceptions retain their conservative/fail-loud behavior. Other connector readers and scalar cast paths were traced. |
| Conditional checks and errors | JDBC's null render propagates through composite/function paths; ES comparison failures preserve residual indices. Recursive invalid-date conversion remains the known exception case. The missing ES IN/pattern check is the new finding. |
| Test coverage | Inspected converter shape/NULL/nesting/immutability unit cases, JDBC/ES builder cases and both end-to-end suites. Scalar cast options/patterns in ES are missing. |
| Expected results | Both added .out files agree statically with the corresponding Doris inserts and PostgreSQL, ClickHouse and ES fixtures; ordering is deterministic. Generation provenance and runtime success were not independently verified. |
| Observability | Existing EXPLAIN remote SQL/DSL and residual/LIMIT facts expose these pushdown decisions; no additional metric is needed for this local conversion. |
| Transactions and persistence | No journal, replay, failover or transaction-state logic changed. |
| Data writes and crash atomicity | No data-writing path or atomicity/resource protocol changed; shared write-side literal consumers were inspected. |
| FE/BE propagation | No new transmitted variable or serializer field; BE ARRAY-to-STRING rendering was traced to validate the reported semantics. |
| Performance and memory | Conversion is proportional to ARRAY elements, as was recursive display rendering; fresh lists add bounded per-literal allocation. No new redundant scan or unbounded cache was identified. |
| Other issues and focus | No additional user focus was supplied. Complete changed-path, consumer, special-case and existing-thread sweeps found no further independent issue. |
Validation was static inspection only: no builds, tests, benchmarks or runtime experiments were executed. No remaining static coverage questions are open; runtime behavior and plugin deployment were not exercised in this review.
Review converged after 2 round(s).
| ConnectorLiteral literal = (ConnectorLiteral) valueExpr; | ||
| // An ARRAY literal compares whole arrays, but a term or range query matches any one value of a | ||
| // multi-valued field, so Doris evaluates the comparison itself. | ||
| if (literal.getValue() instanceof List) { |
There was a problem hiding this comment.
[P2] Reject ARRAY values in the other Elasticsearch literal readers
For a native scalar keyword field p read through docvalues, this valid plan survives FE folding (BE folding is off by default):
Filter(p IN (CAST([TRUE] AS STRING), 'other'))
ElasticsearchScan(p: STRING)
ES allows cast pushdown, and the neutral converter strips CastExpr. This PR therefore hands inToDsl a List<Boolean> option; unlike comparisonToDsl, it does not reject it, and extractLiteralValue stringifies it as "[true]". The previous ARRAY text and the actual BE cast both render "[1]". ES now receives a terms query for [true]/other, drops matching [1] documents, and reports the conjunct pushed so Doris removes the residual and can also accept wrong [true] rows. likeToDsl has the same unguarded String.valueOf path.
Apply the List rejection to IN/pattern readers too, or keep these casts local before conversion, and cover scalar cast constants. This is a separate ES predicate reader from the existing Hudi partition-pruning thread.
There was a problem hiding this comment.
Thanks, confirmed. Fixed in 3162b0e once for every residual reader rather than per builder: buildRemainingFilter now keeps a conjunct holding a cast into or out of ARRAY, MAP or STRUCT (other than one between two types of the same kind) in Doris whatever supportsCastPredicatePushdown returns, as it already does for TIMESTAMPTZ casts. The converter strips the CAST, and the operand of such a cast is not the value it yields (BE renders [1], ["a"]), so no reader can push it faithfully. Elasticsearch inToDsl, likeToDsl and REGEXP no longer see such a value, and neither does Paimon behind a Fluss union read, which inherits the default true and stringifies VARCHAR values the same way. A cast between two ARRAY types still reaches connectors, so arrays_overlap(col, [...]) and the JDBC / ES comparison checks are unchanged. PluginDrivenScanNodeCastPredicateTest covers =, IN and LIKE on CAST([true] AS STRING) and casts of ARRAY, MAP and STRUCT columns with casts accepted (each fails without the check), and keeps the ARRAY-to-ARRAY case pushed.
…nector filters
### Background
A catalog connector receives a scan's filters as a `ConnectorExpression` that fe-core's `ExprToConnectorExpressionConverter` builds, dropping every CAST and passing its operand on. `applyFilter` gets every conjunct: Hive and Hudi read its EQ and IN predicates on partition columns and prune partitions by comparing the values' text with the partition values. The remaining filter, which the query builders of Elasticsearch, JDBC and other connectors read, loses the conjuncts holding a CAST only for a connector whose `supportsCastPredicatePushdown` is false.
### What problem does this PR solve?
Issue Number: None
Related PR: None
PR Type: Bugfix
Problem Summary:
The previous commit hands an ARRAY literal over as its element list. When a predicate casts one to a scalar type, as in `p = CAST([TRUE] AS STRING)`, Doris compares `p` with the cast's text `[1]`, but the connector gets `p` against the list `[true]`. Hudi and Hive read the list as the text `[true]` and pruned the partition `p='[1]'` away; Elasticsearch pushed IN, LIKE and REGEXP with that text and stopped evaluating them. A conjunct holding a cast into or out of ARRAY, MAP or STRUCT, other than one between two types of the same kind, now stays in Doris on the remaining filter for every connector, and Hive and Hudi prune neither by a literal of those types nor by an IN list holding anything but scalar literals.
### Details
- **Frequency:** Not observed; found by review. Triggers whenever such a cast survives planning, which it does: FE cannot fold a cast between kinds, and `enable_fold_constant_by_be` is off by default.
- **Root cause:** Without its CAST, the operand has another shape than the value the predicate compares: an element list, or for `CAST(arr AS STRING) = '[1]'` an ARRAY column. `PluginDrivenScanNode.buildRemainingFilter` kept such conjuncts in Doris only for connectors that refuse every cast, and `extractPredicatesRecursive` in Hive and Hudi read any literal's text as a partition value. Their IN loop also skipped the items it could not read, so the rest narrowed the list.
- **Why existing tests missed it:** `PluginDrivenScanNodeCastPredicateTest` covered scalar casts for a connector that refuses casts, and the pruning tests of Hive and Hudi use STRING literals only. No regression suite filters with a cast between a complex type and another: the 28 WHERE clauses that cast to a string type in the external suites all cast scalar columns.
- **Fix:** The remaining filter has many readers (the Elasticsearch IN, LIKE and REGEXP builders, Paimon behind a Fluss union read, JDBC, Iceberg, ADBC), so the check sits once in `buildRemainingFilter`, beside the TIMESTAMPTZ one, instead of in each of them. `applyFilter` gets every conjunct by design, so its readers check the literal: Hive and Hudi are the only ones, as Trino's returns no result. Rule 4 of the SPI package documentation and `supportsCastPredicatePushdown` now state both.
- **Other paths and branches:** A cast between two ARRAY types still reaches connectors, so `arrays_overlap(col, [...])` and the JDBC and Elasticsearch checks of the previous commit are unchanged. The same check also keeps in Doris a cast of an ARRAY, MAP or STRUCT column, such as `CAST(arr AS STRING) = 'x'`, which connectors that accept casts received as `arr = 'x'`, and the Hive and Hudi guard stops an IN with a call the planner did not fold from pruning as if the call were not there; both predate this PR. Not on branch-4.x, which has no connector SPI.
### Testing
All run locally on this commit's code:
- `PluginDrivenScanNodeCastPredicateTest`: for a connector that accepts casts, `code = CAST([true] AS STRING)`, an IN and a LIKE with such a cast, `CAST(arr AS STRING) = '[1]'`, `arr = CAST('[1]' AS ARRAY<INT>)`, and MAP and STRUCT columns cast to STRING stay in Doris, each with the CAST the planner kept, while `CAST(arr AS ARRAY<BIGINT>) = [1, 2]` is still pushed. Without the check all seven are pushed.
- `HudiPartitionPruningTest` and `HiveConnectorMetadataPartitionPruningTest`: an EQ or IN with an ARRAY, MAP or STRUCT literal, or an IN holding a function call, prunes nothing, and Hive asks HMS for no filter; an IN with a NULL item still prunes. Without the guards the `p=[1]` partition is dropped.
- 15 mutations (the guard off; its ARRAY, MAP or STRUCT clause off; a cast between two ARRAY types treated as a change of kind; in Hive and in Hudi the EQ guard, the IN guard and each type name off), each beside an unmutated control: every one fails exactly the cases it targets.
- Suites: fe-connector-hive (580), fe-connector-hudi (272), fe-connector-spi (152), fe-connector-es (164), fe-connector-jdbc (275), and in fe-core the converters plus `PluginDrivenScanNode*Test` (291).
### Release note
None
### Check List (For Author)
- Behavior changed: No
- Does this need documentation: No
- Pre-submission status: Ready
### AI Disclosure
- Tool: Claude Code (Claude Opus 5.5)
- Generated: code, tests, this message
- Human review: not stated
- Prompts: After the CI review bot found that a stripped CAST exposed the element list to Hudi and Elasticsearch, the author chose to fix the whole class in this PR and to keep the plugin API version.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Addressed in 3162b0e:
Rule 4 of the SPI package documentation now states both, and the description's Impact and Testing cover CAST-wrapped ARRAY literals. |
|
run buildall |
|
/review |
There was a problem hiding this comment.
Reviewed head 3162b0e5eab040f693c8d25e0586e9bbf6c72d9e: no additional verified findings. All 19 changed files, related conversion and consumer paths, unit tests, regression suites and expected outputs were inspected. Both complete-review agents and the separate risk reviewer returned NO_NEW_VALUABLE_FINDINGS; every identified risk has an evidence-backed disposition.
The relevant flow is Nereids -> legacy Expr -> neutral literal -> applyFilter/partition pruning and, separately, buildRemainingFilter -> connector SQL/DSL -> residual-index mapping -> local filtering/LIMIT. ARRAY values now preserve element boundaries, NULLs and nesting. Unsupported remote interpretations retain local evaluation.
Existing review context:
- Hudi partition pruning: the concrete reported case is addressed. Hive/Hudi refuse complex partition literals and incomplete IN lists and retain the original residual.
- ES IN/LIKE readers: the concrete reported case is addressed by excluding the whole complex-kind CAST conjunct before neutral conversion, preserving local filtering and LIMIT behavior.
- Zero-date ARRAY parameters: the previously reported nondefault
allow_zero_datecase remains; recursive DATE conversion still reachesLocalDate.of(0,0,0). The author's deferral is acknowledged; no duplicate inline is posted. - Plugin value-shape compatibility: ARRAY semantics change despite unchanged public signatures. The same-major gate does not protect an independently retained old plugin from the new value shape. This remains already-reported context, including the author's matched-build/version-policy response; no duplicate is posted.
There are no existing P0/P1 inline findings to reconfirm. No additional user review focus was supplied.
Required checkpoint conclusions:
| Checkpoint | Conclusion |
|---|---|
| Goal and evidence | Immutable element lists implement the stated representation goal; inspected tests cover scalar shapes, punctuation, NULL, nested and empty arrays. |
| Scope and clarity | Changes remain focused on literal representation, its contract and conservative predicate handling. |
| Concurrency and locking | No new threads, shared mutable state or lock ordering; newly constructed lists have private immutable backing. |
| Lifecycle and initialization | No new external resources or static initialization dependencies; catalog/reset and plugin ownership mechanisms are unchanged. |
| Configuration | No settings or defaults added. |
| Compatibility | No public SPI symbol, storage-format or FE/BE protocol change; semantic mixed-plugin compatibility is the known concern above. |
| Thrift/protobuf | Neither directory is changed. |
| Parallel paths | SELECT, DML conflict filters and EXECUTE share literal conversion; all in-tree consumer families were traced for unsupported ARRAY handling. |
| Special conditions | Guards explain concrete remote misinterpretation and partition-loss scenarios; IN refuses an incomplete domain rather than dropping an unsupported item. |
| Test coverage | Unit and regression additions cover converter structure, cast directions/kinds, JDBC functions and boolean combinations, ES comparisons, partition pruning and LIMIT. Tests were inspected, not run. |
| Expected results | Both complete output files agree with inspected Doris/PostgreSQL/ClickHouse/ES fixtures; result queries use ordered comparison. Generation provenance was not independently verified. |
| Observability | Existing EXPLAIN SQL/DSL, residual predicates and LIMIT facts expose the changed behavior; no additional instrumentation appears necessary. |
| Transactions and persistence | No EditLog, metadata persistence or failover behavior changes; shared write-conflict conversion remains conservative. |
| Data writes and atomicity | No write/commit protocol changes; no new crash cleanup or atomicity mechanism. |
| FE/BE values | No new transmitted variables or serialization fields. |
| Performance | Conversion is linear in literal contents; moving unsupported filtering and LIMIT local is intentional and documented. No new asymptotic or redundant heavy operation identified. |
| Further issues | Rechecked constructor invariants, nested shapes/casts, TRY_CAST, NULLs, scalar-column casts, all literal readers, residual caches and index mapping; no distinct new candidate remains. |
Static-only validation: no builds, tests, benchmarks, runtime checks, source changes or submodule initialization were performed. Author-reported executions and separately built external plugins were not independently validated. No unresolved source-level candidate or uncovered changed file remains in this review.
Review converged after 1 round(s).
…shdown (#70372) ### Background Connector plugins learn which filters they may push down from a `ConnectorExpression` tree that fe-core's `ExprToConnectorExpressionConverter` builds from a scan's conjuncts. A function call arrives as a `ConnectorFunctionCall` with the function's name, return type and arguments, and connectors that translate calls match it by name: JDBC writes it into the remote SQL when `enable_ext_func_pred_pushdown` is on (the default) and the name passes the database's rules (for MySQL, a blocklist of three names), and Elasticsearch runs `esquery` as a native query. ### What problem does this PR solve? Issue Number: None Related PR: #70369 (changes the same converter and SPI package; independent), #68702 (also changes the connector SPI; whichever merges second takes the next plugin API major) PR Type: Bugfix Problem Summary: A filter that calls a Java or Python UDF on a JDBC table is sent to the remote database as a call of the same name. On MySQL, `WHERE mydb.my_udf(id) = 113` fails with `FUNCTION doris_test.my_udf does not exist`, and a UDF named like a MySQL function, `mydb.abs(id) = 113`, is computed by MySQL instead, which silently drops rows. The converter hands connectors only the function's name, so no connector can tell a UDF from a builtin. This PR adds `ConnectorFunctionCall#isBuiltin()`, set from the catalog function's binary type; JDBC and Elasticsearch no longer translate a call that is not a builtin, and Doris evaluates it. ### Details - **Change map:** +373 −26 in 13 files, of which tests are +303: a regression suite +89 with its `.out` +10, and unit tests +204. The production code is +69 −23: 1. SPI (+42 −13): the flag on `ConnectorFunctionCall`, and Rule 6 of `pushdown/package-info.java` rewritten; plus the plugin API major 30.0 → 31.0 in `fe/fe-connector/pom.xml`. 2. fe-core (+12 −4): `ExprToConnectorExpressionConverter` sets the flag. 3. Connectors (+15 −6): the checks in `JdbcQueryBuilder` and `EsQueryDslBuilder`. - **Frequency:** Always, with function pushdown on: any filter that calls a UDF on a JDBC catalog over MySQL, and over ClickHouse or Oracle when the UDF's name is on their allowlist or in `function_rules`. On Elasticsearch, a UDF named `esquery`. - **Root cause:** `ExprToConnectorExpressionConverter.convertFunctionCall` keeps only `getFnName().getFunction()`, the return type and the arguments, and drops `Function.BinaryType`. `JdbcQueryBuilder.functionCallToSql` then renders any call that `JdbcFunctionPushdownConfig.canPushDown` accepts by name, and `EsQueryDslBuilder` takes any call named `esquery` for the builtin. The SPI documentation said calls with real arguments "are safe to match by name". - **Why existing tests missed it:** `JdbcQueryBuilderTest` and `EsScanPlanProviderTest` build calls by hand, all of them builtins, and no unit test converts a UDF. The only regression suite that uses a UDF on a JDBC table, `test_mysql_varbinary_with_udf`, calls it in a projection; no suite filters with one. - **Fix:** The converter sets the flag where the binary type is still known, and each connector that matches calls by name checks it. JDBC refuses a non-builtin call before rendering anything, so a fallback (an expression the converter does not model, sent as SQL text) that calls a UDF stays local too; a fallback is a builtin only when it calls no UDF. Elasticsearch requires the builtin for `esquery`, and the converter's LIKE / REGEXP shortcut applies only to builtins. The alternative, the engine keeping every conjunct with a UDF away from all connectors as it does for CASTs, was not chosen: the flag leaves the decision to each connector. Changing the SPI surface moves the plugin API major to 31.0. - **Other paths and branches:** Iceberg drops every function call, and Hive, Hudi, Paimon, MaxCompute and Trino translate none by name, so they are not affected. On branch-4.1 and branch-4.2 the legacy `JdbcScanNode` has the same gap (`JdbcFunctionPushDownRule` checks names only); they have no connector SPI, so this PR does not apply there. - **Behavior change:** Such a filter no longer reaches the remote database, so the remote query returns what the other conjuncts select, without LIMIT, and Doris filters the rest. Connector plugins built against plugin API 30 are refused by this FE until they are rebuilt. ### Testing - New regression suite `external_table_p0/jdbc/test_jdbc_mysql_udf_filter` (a Java UDF registered as `udf_add_one` and as `abs`, on MySQL 5.7): on the unfixed engine its first check fails, MySQL receiving `WHERE (udf_add_one(`id`) = 113)`; by hand on that engine, the query fails with MySQL error 1305, and `mydb.abs(id) = 113` returns no row instead of 112. With the fix both return 112 with no WHERE in the remote query, and the builtin `abs` is still pushed. Run locally against a standalone `mysql:5.7.36` container; the External pipeline runs it (`enableJdbcTest`). - Unit tests, locally: new `ExprToConnectorFunctionCallTest` (fe-core) and `ConnectorFunctionCallTest` (SPI), and new cases in `JdbcQueryBuilderTest` and `EsQueryDslBuilderTest`. All of fe-connector-spi (154), fe-connector-jdbc (275), fe-connector-es (164), and in fe-core the converter tests with `PluginDrivenScanNode*Test` (282), pass. - Seven mutations, each failing a new test: the converter taking every call for a builtin, the LIKE shortcut or the fallback ignoring the flag, `equals` ignoring it, the 3-argument constructor meaning a UDF, and each connector check removed. ### Notes for Reviewers - The 3-argument `ConnectorFunctionCall` constructor still means a builtin; its callers are the converter's arithmetic operators and connector tests. - Rule 6's old example, `my_udf(a, b) = 1` arriving as one fallback, never happened: a UDF call converts to a named call with arguments. The new example is a CASE expression, which the converter does not model. - `getFn()` is not null-checked: plugin scan conjuncts come from Nereids' `ExpressionTranslator`, which always sets the catalog function. - The Lance connector being ported from branch-4.2 (not yet opened) will check the flag for `starts_with`, `ends_with`, `array_contains` and `arrays_overlap`, as 4.2 checks the binary type.
Background
A catalog connector learns which filters it may push down from an engine-neutral expression tree (
ConnectorExpression) that fe-core'sExprToConnectorExpressionConverterbuilds from the planner's conjuncts. A literal in that tree carries a plain Java value whose shape the SPI documents per Doris type (Rule 4 inconnector/spi/pushdown/package-info.java):Longfor integers,LocalDatefor dates, and so on. Types outside that table, ARRAY among them, arrive as the literal's display text.What problem does this PR solve?
Issue Number: None
Related PR: #68687 (the Lance array-membership pushdown on branch-4.2, whose master port needs the elements)
PR Type: Enhancement
Problem Summary:
An ARRAY literal reaches a connector as text such as
[red, a, b], with unquoted elements, so['a, b']and['a', 'b']cannot be told apart. The Lance connector needs the elements to push downarrays_overlap, which the planner makes from three or more ORedarray_contains. This PR makes the value an unmodifiableListof element values, each in the shape the SPI gives its element type. An audit of the connectors that read literal values found two that already mishandled the text and would mishandle the list: JDBC sent the text to the source, which PostgreSQL and ClickHouse reject and a remote Doris reads back as other elements, and would write the list unquoted; Elasticsearch pushed array comparisons as a term query on the text and stopped evaluating them. Both now leave such predicates to Doris. Since the converter drops CASTs, a conjunct that casts into or out of ARRAY, MAP or STRUCT, such asp = CAST([TRUE] AS STRING), now stays in Doris, and Hive and Hudi do not prune by it.Details
.out, in three commits. The code is the converter's ARRAY branch (19 lines) and the JDBC and ES checks (10) in the first commit, then the third commit's guards for casts the converter drops: one inPluginDrivenScanNode.buildRemainingFilter(28) and one each in Hive and Hudi partition pruning (42); the SPI documentation of both (29) is split between them. The tests are unit tests of the converter, both query builders, the scan node and the two pruners (326), and the second commit's two regression suites (228).['red', 'a, b', NULL],ConnectorLiteral#getValue()was the String[red, a, b, NULL]and is now the list["red", "a, b", null]; nested arrays are nested lists, and a NULL array is still a NULL literal. An element has the value it would have as a literal of its own (a LARGEINT element is still a decimal String). JDBC:tags = ['a', 'b'], or on MySQLarrays_overlap(tags, ['a', 'b']), is no longer sent to the source, where it went as the string'[a, b]'. Elasticsearch (docvalue mode):tags = ['red', 'blue']is no longer pushed as{"term":{"tags":"[red, blue]"}}. A conjunct that casts into or out of ARRAY, MAP or STRUCT, such asp = CAST([TRUE] AS STRING)orCAST(arr AS STRING) = '[1]', reached connectors that accept casts as its bare operand and now stays in Doris; a cast between two ARRAY types still reaches them. Hive and Hudi read the operand ofp = CAST([TRUE] AS STRING)as the partition value[true](master read[1], right for this element type only); they now take no partition value from a complex-typed literal, nor prune by an IN holding anything but scalar literals.malformed array literal; ClickHouse string arrays:Cannot parse quoted string; a remote Doris: no signature forarrays_overlap(ARRAY<TEXT>, VARCHAR)), and a remote Doris, which parsed the text into other elements, no longer loses rows whose elements contain,or NULL. Elasticsearch dropped a pushed comparison from what Doris evaluates, while a term query matches any one value of a multi-valued field: on ES 7.17.5 and 8.3.3 a docvalue scan returned no row for=, every row for!=, and failed the query for a long array. ADBC, Iceberg, Paimon, MaxCompute and Trino already refuse ARRAY literals. Hive and Hudi partition pruning sees one only behind a CAST the converter drops, and now prunes nothing by it, and any filter with a cast into or out of a complex type, with its LIMIT, stays in Doris. No SPI type or method changes, so the connector plugin API version stays.Testing
All run locally, the first commit's tests and the regression suites before the rebase onto
ee73acc6131, the rest on the head:ExprToConnectorExpressionConverterTest(fe-core): elements with commas and quotes, every scalar shape, NULL elements, nested and empty arrays, an unmodifiable list, a NULL array. 4 tests pass; with the new branch removed (master's behavior) all 4 fail.JdbcQueryBuilderTest#testArrayLiteralsRemainLocalWithoutLimit(MySQL, PostgreSQL, ClickHouse, Trino): no WHERE or LIMIT for a comparison, NOT orarrays_overlapwith an ARRAY literal; an AND keeps its other conjunct. Without the new check the builder emitsWHERE (`tags` = [a, b, x') OR ('1'='1]) LIMIT 1.EsQueryDslBuilderTest#arrayLiteralComparisonsRemainLocal:=,!=,<on either side stay in Doris; in an AND the other conjunct is still pushed.PluginDrivenScanNodeCastPredicateTest(fe-core, planned queries, casts accepted):code = CAST([true] AS STRING), an IN and a LIKE with such a cast,CAST(arr AS STRING) = '[1]',arr = CAST('[1]' AS ARRAY<INT>), and MAP and STRUCT columns cast to STRING stay in Doris, andCAST(arr AS ARRAY<BIGINT>) = [1, 2]is still pushed. Each case also shows that FE keeps the CAST.HudiPartitionPruningTestandHiveConnectorMetadataPartitionPruningTest: an EQ or IN with an ARRAY, MAP or STRUCT literal, or an IN holding a function call, prunes nothing, and Hive asks HMS for no filter; an IN with a NULL item still prunes.ConnectorPluginSurfaceTest), fe-connector-es (164), fe-connector-jdbc (275), and in fe-core the converters that route through this one plusPluginDrivenScanNode*Test(291).test_jdbc_array_literal_predicate(p0, external) runs such filters against a remote Doris (the cluster itself), PostgreSQL and ClickHouse:,and NULL elements, INT and nested arrays,arrays_overlapand the ORedarray_containsthe planner rewrites into it, AND, OR and LIMIT. Each block returns the source's rows and the remote query carries no array filter or LIMIT. With master's converter each source's part fails; without the JDBC check the remote Doris rejectsWHERE (`tags` = [a, b]).test_es_array_literal_predicate(p2, like the other ES suites) does the same on ES 7 and 8 in a docvalue scan. With master's converter or without the ES check,c_keyword = ['a', 'b', 'c']returns no row instead of five.test_lance_array_predicate_pushdownpushesarrays_overlap(labels, [...])into Lance, up to the 64-needle index budget, and its BE profile shows the scalar index segments searched.Notes for Reviewers
NereidsToConnectorExpressionConverter(DELETE / UPDATE / MERGE) andUnboundExpressionToConnectorPredicateConverter(EXECUTE ... WHERE) convert literals throughExprToConnectorExpressionConverter, so all three paths get the list.convertLiteralandconvertDateLiteralnow declareConnectorLiteral, which they always returned, so the element loop needs no cast.buildRemainingFilter, beside the TIMESTAMPTZ one, because the remaining filter has many readers;applyFilterreceives every conjunct by design, so its readers, Hive and Hudi (Trino's returns no result), check the literal. Wrapping such a cast in the converter'sfallback()would not be safe: JDBC writes an argument-less call into the remote SQL as text.Release note
For JDBC and Elasticsearch catalogs, a filter that compares a column with an ARRAY constant, such as
tags = ['a', 'b'], and for JDBC a function call that takes one, is now evaluated by Doris instead of being sent to the source. Such queries no longer fail on PostgreSQL and ClickHouse sources, and no longer return wrong rows from a remote Doris or Elasticsearch.Check List (For Author)
AI Disclosure