Repository navigation
[feature](mongodb) Complete M1 type coverage and safe scan pushdown - #70604
Conversation
|
run buildall |
|
/review |
The native MongoDB Catalog on master already provides single-cursor reads and first-release BSON SerDes. SQL filters and source projection have remained local, and the remaining M1 type and pushdown contracts need end-to-end qualification. Issue Number: apache#70251 (D04b and D05) Related PR: apache#70327 (native read path), apache#70456 (M0 acceptance) PR Type: Feature Problem Summary: Complete the D04b qualification coverage and connect conservative predicate/LIKE candidates and column pruning to the native MongoDB reader. Every SQL predicate remains a Doris residual; the default strict type policy, strict SQL casts and standard views keep filtering local. This PR does not complete the separate D06 or later snapshot/parallel-scan work. - **Change map:** 449 added production lines, 1,187 test/fixture/generated-result lines, and 47 documentation lines across 19 files. The production changes comprise a connector-local translator and scan planning, three session controls, two optional descriptor fields, and native find options/validation. Most additions qualify types and compare execution modes. - **Scope:** D04b covers Decimal128 extremes, exact integer/floating-point behavior, UTC dates and scales, binary ownership/subtypes, missing/null, recursive arrays/structs, rollback and existing document limits. D05 covers typed scalar-root comparisons, positive IN/BETWEEN, IS NOT NULL, safe AND/OR, necessary-literal LIKE candidates, root projection and time-series Date ranges. - **Design:** Candidates are classified conservatively as SUPERSET; unsupported expressions remain local, and the EXACT category is reserved for proven equivalence. No conjunct is consumed through either SPI path. CASTs are rejected before the neutral converter can erase them. The translator reconciles the schema/expression SPI date-scale and decimal-width representations, including FE's DECIMAL128 name. Native BSON is bounded and validated before client acquisition; views cannot receive a predicate/collation override. Complete root projection retains residual columns and scalar-ancestor errors. - **Usage:** Use `SET` or `SET_VAR` with `enable_mongodb_predicate_pushdown`, `enable_mongodb_like_pushdown` and `enable_mongodb_projection_pushdown`. Predicate candidates require the explicit catalog policy `mongodb.type_mismatch=null`. - **Default behavior:** All three switches default to true. The default `type_mismatch=error` policy preserves local filtering; `enable_strict_cast=true` and views also preserve local filtering. Source projection is eligible independently. DOUBLE/NaN comparisons, NOT/!=/NOT IN, mismatch-sensitive IS NULL, functions, casts, nested predicate paths and existing direct VARBINARY operator restrictions stay local. LIKE keeps the complete SQL residual. LIMIT, aggregation, sort, joins, snapshots and parallel scans are unchanged. - **Compatibility:** The connector SPI and stored schemas are unchanged. Descriptor fields 23/24 are optional; absent fields preserve the prior full scan. The touched descriptor is absent from the four supported release branches, so no field id is reused. Qualification uses matching FE/BE versions. All testing below ran locally against MongoDB 8.0.20 with authentication enabled and an ASAN BE: - 59 MongoDB connector FE tests, 25 connector-session/forwarding tests and 55 BSON/native-reader BE tests passed. - All three MongoDB SQL suites passed: 3 suites, 0 failed, 0 fatal, 0 skipped. This includes the existing native and M0 acceptance suites plus the new M1 suite. - The generated M1 output contains 38 identical sorted result pairs with predicate/LIKE/projection disabled and enabled. It covers precision, NULL/arrays, casts, collations, residual LIMIT, aggregates/joins, views and time-series reads. The completed Profile check reports `MongoRowsRead` of 10 with candidates disabled and 1 with them enabled. - Source profiling independently confirmed actual BSON int64, Decimal128 and Date filters, projection, simple collection collation, and view reads without a collation override. - FE and ASAN BE builds passed; the final MongoDB plugin was subsequently packaged/tested and deployed from its final source. Full FE Checkstyle and the final affected-core-file Checkstyle run passed with zero violations. All affected C/C++ files pass clang-format 16, build hygiene and clang-tidy. - Red/green checks reproduced missing candidate planning, acceptance of malformed BSON descriptors, missing strict-cast gating, command-budget gaps and schema/expression type representation gaps before the corresponding implementations. Native MongoDB scans can prune source fields and apply conservative WHERE/LIKE candidates under the explicit null-on-type-mismatch policy. SQL residuals remain active, and default strict scans and views preserve local filtering. New session switches allow statement-level comparisons and disabling these optimizations. - Behavior changed: Yes (see Details). - Does this need documentation: Yes; the connector and fixture READMEs are updated here. - Pre-submission status: Ready. Local UT, authenticated SQL regression, builds and style/static checks passed; CI is pending. - Tool: Codex (GPT-6). - Generated: Implementation, tests, fixture, documentation and this description. - Human review: Pending maintainer review.
### Background MongoDB null comparisons traverse array elements, while SQL tests the nullability of the complete array or JSON value. ### What problem does this PR solve? Issue Number: apache#70251 Related PR: apache#70604 PR Type: Bugfix Problem Summary: Restrict the IS NOT NULL candidate to scalar targets so a valid array or JSON array containing null cannot be omitted remotely. ### Details - Frequency: Complex mappings whose BSON array contains null and is filtered with IS NOT NULL under the null-on-mismatch policy. - Root cause: MongoDB's $ne:null excludes arrays containing null, but ARRAY[NULL] and JSON [null] are non-NULL SQL values. - Why existing tests missed it: Scalar null tests and array conversion tests did not combine IS NOT NULL with valid array null elements. - Fix: Keep complex-target null tests local. Add translator UT and ARRAY/JSON result pairs to the SQL differential suite. - Other paths and branches: Scalar null-test candidates retain their existing superset behavior. Every SQL residual remains active. ### Testing The new translator UT failed before the guard and passes afterward. A real-driver SQL probe reproduced the missing row before the fix. All 60 connector tests and Checkstyle pass. All three authenticated MongoDB SQL suites pass with zero failed/fatal/skipped suites; all 40 sorted off/on result pairs agree. ### Release note Native MongoDB candidate filtering preserves non-NULL arrays and JSON arrays containing null. ### Check List (For Author) - Behavior changed: Yes; corrected candidate eligibility as described above. - Does this need documentation: Yes; scalar eligibility is clarified in the connector README. - Pre-submission status: Ready for conflict integration and final validation. ### AI Disclosure - Tool: Codex (GPT-6). - Generated: Fix, tests and this description. - Human review: Pending maintainer review.
There was a problem hiding this comment.
Request changes: one verified P1 wrong-result issue in IS NOT NULL candidate translation for non-null arrays containing null elements. The inline includes a concrete ARRAY trigger and a conservative fix direction.
The review converged after 2 rounds. All three reviewers returned NO_NEW_VALUABLE_FINDINGS in round 2; all 19 changed paths and the related FE-to-BE invocation chain were covered. R1 resulted in the inline finding; R2 projection concerns were dismissed with evidence. No unresolved candidates or source coverage gaps remain. There were no existing inline threads or independently confirmed existing P0/P1 comments, and no additional user-provided focus points.
Critical checkpoint conclusions:
- Goal and proof: the PR adds M1 type qualification plus candidate predicates/root projection. The ARRAY IS NOT NULL case violates the required superset contract; the new tests do not cover it.
- Scope and clarity: production changes are focused on connector translation/planning, three session switches, optional descriptors and native find options; tests, fixtures and documentation match that scope.
- Concurrency and locks: no new worker/shared mutable state or lock ordering; translator state is invocation-local and the existing client-pool synchronization remains.
- Lifecycle and initialization: BSON filter/options are owned and destroyed through Defer on return paths; cursor/collection/client cleanup and borrowed-value copying remain intact. No new static initialization dependency found.
- Configuration: the three switches are dynamically read from statement session snapshots and declare forwarding/result-affecting behavior; forwarding and statement override tests are present.
- Compatibility: stored schemas, SPI surface and block layout are unchanged. Absent optional fields preserve old behavior; older BEs can ignore candidates/projection because Doris retains every residual predicate.
- Thrift/protobuf: new binary fields 23/24 are optional. TMongoScanDesc is absent on the four supported release branches; current release refs were checked and no ID collision/reuse was found.
- Parallel paths: applyFilter consumes nothing; scan-property results do not consume conjuncts. The scan path excludes CAST-containing conjuncts before conversion. Views/time-series retain their logical namespace and views keep predicates local.
- Conditional checks: strict mismatch/cast/view gates, safe root paths, scalar type conversion and term/depth/byte limits have concrete purposes. The missing array-domain restriction is the reported defect.
- Test coverage: FE translation/planning/forwarding tests, BE exact-conversion/options/lifecycle tests and enabled/disabled SQL comparisons provide broad coverage; nullable-element ARRAY IS NOT NULL needs a differential case.
- Changed results: all 38 supplied enabled/disabled output pairs match in static comparison; expected results were cross-checked against fixture/conversion logic. Execution and generated-file provenance were not independently verified.
- Observability: EXPLAIN exposes candidate kind, retained residual and projection state; native row/byte/read/decode/filter counters remain connected. The regression includes a 10-to-1 MongoRowsRead check.
- Transactions, persistence and failover: no EditLog, transaction, persistent schema or metadata-write path changes; internal visible-version/delete-bitmap mechanisms are unaffected.
- Writes and atomicity: this is an external read path. Existing per-value/per-row rollback is preserved and nested failure tests qualify it; no additional crash/leak defect verified.
- FE-BE variables: session settings determine optional BSON fields; MongoScanRange deep-copies them into the typed descriptor and BE validates/applies them. Alternate SPI paths retain residuals.
- Performance: translation and BSON assembly occur during planning/open, with bounded candidates, rather than per row. No additional material regression was substantiated; no benchmark ran.
- Other concerns: complete-root projection preserves required nested values and ancestor/type errors. Pruning unused subtrees deliberately reduces decoder input; the changed reader comment explicitly validates returned subtrees, so that observation is not an additional finding.
Validation limits: static inspection only. No builds, tests, live database execution or performance measurements were run. External topology/version behavior and the author's runtime claims remain unverified here.
Review converged after 2 round(s).
| } | ||
| if (expression instanceof ConnectorIsNull isNull && isNull.isNegated()) { | ||
| MongoSchema.Binding binding = binding(isNull.getOperand()); | ||
| return binding == null ? Result.unsupported() : candidate(binding, "$ne", null); |
There was a problem hiding this comment.
[P1] Preserve non-null arrays when translating IS NOT NULL
With mongodb.type_mismatch=null, a root mapped as ARRAY<INT> can contain [1, null]: the BSON SerDes produce a non-null outer array, so Filter(a IS NOT NULL) -> MongoScan must retain that row. This branch also accepts ARRAY/JSONB bindings and emits {a: {$ne: null}}, which MongoDB applies to array elements, excluding arrays that contain null. The retained Doris residual cannot recover the discarded row. Keep these complex mappings local or use an existence-based superset, and add an enabled/disabled case with null array elements.
There was a problem hiding this comment.
Fixed in b6f2141. IS NOT NULL candidates now require a scalar target; ARRAY/JSON and other complex targets remain local. Added a translator UT and enabled/disabled SQL regressions for ARRAY and JSON arrays containing null. Both retain the previously discarded row, and all 40 sorted off/on result pairs match.
25e8953 to
b6f2141
Compare
|
run buildall |
|
/review |
There was a problem hiding this comment.
No additional actionable findings on b6f2141. The previously reported P1 ARRAY/JSON IS NOT NULL issue (thread 4581951532) is fixed: complex targets stay local, and new unit and enabled/disabled regression cases cover arrays containing null. No existing P0/P1 finding remains confirmed applicable to this head.
Review converged after 1 round. Both complete reviewers and the separate risk reviewer returned NO_NEW_VALUABLE_FINDINGS; all 19 changed paths and the affected FE-to-BE flow were covered. Every risk focus was dismissed with evidence, with no unresolved candidate or source coverage gap. No additional user focus was supplied.
Critical checkpoint conclusions:
- Goal and proof: the PR implements M1 type qualification and conservative MongoDB predicate/LIKE candidates plus root projection. Static code paths and added unit/SQL artifacts support the intended behavior, including the prior array fix; execution was not rerun.
- Scope and clarity: changes stay within connector translation/planning, session switches, optional descriptors, native find options, and corresponding tests/fixtures/docs. Shared SPI and stored schemas are unchanged.
- Concurrency and locks: no new worker or shared mutable translator state. Existing pool synchronization and lock order remain; planning and cursor BSON are local to their invocation.
- Lifecycle and initialization: native BSON validation precedes pool acquisition. Defer releases local find documents; existing cursor/collection/client close order and error cleanup remain safe. No new static initialization dependency found.
- Configuration: all three switches default true, are read from statement session snapshots, and declare forwarding/result-affecting behavior. Added tests cover forwarding and statement overrides.
- Compatibility: absent new fields preserve old full scans; older BEs can ignore candidates/projection because every Doris residual remains. No storage-format or function-symbol change.
- Thrift/protobuf: fields 23/24 are optional binary fields. Full touched definitions were compared against master and branch-4.2/4.1/4.0/3.1; the descriptor is absent on those release branches, with no field collision or reused ID.
- Parallel paths: metadata applyFilter consumes nothing; scan properties have no conjunct tracking and retain all residuals. CAST-containing conjuncts are excluded before neutral conversion. Normal/batch requests retain required columns. Views keep local filters and their collation; time-series scans use the logical namespace.
- Conditional checks: strict mismatch/cast/view gates, scalar target and tag restrictions, safe roots, term/depth/byte bounds and conservative unsupported fallback have concrete purposes. Complex IS NOT NULL is now correctly excluded.
- Test coverage: FE tests cover operators, type representations, budgets, residuals, session switches and projection. BE additions cover BSON options/validation, exact numeric/date boundaries, narrowing, ownership and recursive rollback. SQL comparisons include arrays/JSON, nulls, casts, collation, views, time-series, joins/aggregates and LIMIT.
- Changed results: every output section was inspected against fixture/query semantics; all 40 checked-in disabled/enabled pairs are identical. Execution success and generated-file provenance were not independently verified.
- Observability: EXPLAIN exposes candidate classification, residual retention and projection; existing native row/byte/read/decode/filter counters remain. The regression checks a 10-to-1 MongoRowsRead reduction; no performance measurement was run here.
- Transactions/persistence/failover: no EditLog, transaction or persistent metadata writes are changed. Internal partition-visible-version and delete-bitmap invariants are unaffected.
- Data writes/atomicity: this is an external read path. Existing per-value/per-document conversion rollback remains intact and is covered by added artifacts; no additional crash/leak/atomicity defect was verified.
- FE/BE variables: session choices produce the optional typed BSON fields; MongoScanRange deep-copies them into existing range transport, and BE validates/applies them. No alternate transport requiring a matching change was identified.
- Performance: translation/serialization and find-option construction occur during planning/open rather than per row. Bounds and conservative fallback limit candidate work; no material new performance regression was substantiated.
- Other concerns: whole-root projection preserves requested nested values and scalar-ancestor errors. Source pruning intentionally validates returned subtrees; unused source subtrees are no longer fetched or decoded. Count-only projection preserves document cardinality, including views without _id.
Validation limits: static inspection only. No builds, tests, live database queries, style/static-analysis executions or benchmarks were run. External topology/version behavior and the author's runtime claims remain unverified here.
Review converged after 1 round(s).
|
PR approved by anyone and no changes requested. |
|
PR approved by at least one committer and no changes requested. |
Background
The native MongoDB Catalog on master already provides single-cursor reads and first-release BSON SerDes. SQL filters and source projection have remained local, and the remaining M1 type and pushdown contracts need end-to-end qualification.
What problem does this PR solve?
Issue Number: #70251 (D04b and D05)
Related PR: #70327 (native read path), #70456 (M0 acceptance), #70493 (session read policies)
PR Type: Feature
Problem Summary:
Complete the D04b qualification coverage and connect conservative predicate/LIKE candidates and column pruning to the native MongoDB reader. Every SQL predicate remains a Doris residual; the default strict type policy, strict SQL casts and standard views keep filtering local. This PR does not complete the separate D06 or later snapshot/parallel-scan work.
Details
SETorSET_VARwithenable_mongodb_predicate_pushdown,enable_mongodb_like_pushdownandenable_mongodb_projection_pushdown. Predicate candidates require the explicit catalog policymongodb.type_mismatch=null.type_mismatch=errorpolicy preserves local filtering;enable_strict_cast=trueand views also preserve local filtering. Source projection is eligible independently. DOUBLE/NaN comparisons, NOT/!=/NOT IN, mismatch-sensitive IS NULL, functions, casts, nested predicate paths and existing direct VARBINARY operator restrictions stay local. LIKE keeps the complete SQL residual. LIMIT, aggregation, sort, joins, snapshots and parallel scans are unchanged.Testing
All testing below ran locally against MongoDB 8.0.20 with authentication enabled and an ASAN BE:
MongoRowsReadof 10 with candidates disabled and 1 with them enabled.Release note
Native MongoDB scans can prune source fields and apply conservative WHERE/LIKE candidates under the explicit null-on-type-mismatch policy. SQL residuals remain active, and default strict scans and views preserve local filtering. New session switches allow statement-level comparisons and disabling these optimizations.
Check List (For Author)
AI Disclosure