Repository navigation
[feature](mongodb) Add native catalog, reader and BSON SerDe - #70327
Conversation
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
|
/review |
### Background The native MongoDB connector registers its provider through Java ServiceLoader. Service descriptors support hash comments and are checked by the license workflow. ### What problem does this PR solve? Issue Number: apache#70251 Related PR: apache#70327 (introduced the service descriptor) PR Type: Bugfix Problem Summary: The license workflow rejects the new provider descriptor because it has no ASF license header. Add the same hash-comment header used by the other connectors. ### Details - Frequency: every license check of the original descriptor. - Root cause: the new META-INF/services resource contained only the provider name. - Why existing tests missed it: ServiceLoader accepts the unlicensed descriptor; runtime discovery tests do not validate source licensing. - Fix: add the ASF header as hash comments and document why that syntax is valid. - Other paths and branches: the existing connector descriptors already carry headers. ### Testing Skywalking-eyes v0.8.0 passed the PR incremental check with zero invalid files. All 26 MongoDB connector unit tests passed, including ServiceLoader discovery. FE Checkstyle passed with zero violations during Maven validate. ### Release note None. ### Check List (For Author) - Behavior changed: No. - Does this need documentation: No. - Pre-submission status: Ready; local license, unit and style checks pass. ### AI Disclosure - Tool: Codex (GPT-6). - Generated: the descriptor comment and commit message. - Human review: pending maintainer review.
|
run buildall |
|
/review |
|
Codex automated review failed and did not complete. Error: You’ve hit your usage limit. Visit https://chatgpt.com/settings/usage to purchase more credits or try again at Nov 11th, 2026 6:35 AM. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
…versions ### Background Native MongoDB scans decode BSON through Doris DataTypeSerDes. VARBINARY length is metadata in existing readers, while the new BSON path added a length check. ### What problem does this PR solve? Issue Number: apache#70251 Related PR: apache#70327 (introduces the native reader) PR Type: Bugfix Problem Summary: BSON ObjectId, BinData and raw documents can be rejected solely because their payload exceeds the declared VARBINARY length. Restore the existing metadata semantics and preserve all bytes. Expand regression coverage for NULL, missing, boundary and invalid values, and explain the BSON/reader/scan-descriptor contracts. ### Details - **Change map:** 106 added production/comment lines, 169 test/fixture lines, 60 generated regression-result lines and 2 documentation lines; 24 removed lines. - **Frequency:** whenever a valid BSON binary payload exceeds a declared length. - **Root cause:** the BSON-only SerDe length member enforced a restriction absent from other VARBINARY readers. - **Why existing tests missed it:** the previous BSON test expected this rejection; it did not compare the existing VARBINARY length semantics. - **Fix:** remove the length argument/check while retaining BSON tag/subtype checks, exact byte copying, and mismatch rollback. Document borrowed-value lifetimes, reader lifecycle, and every TMongoScanDesc field. - **Other paths and branches:** existing VARBINARY readers retain their behavior; no wire field IDs, types or requiredness change. ### Testing The two focused ObjectId/raw-BSON tests failed before the fix and passed after it. All 137 selected BE tests passed through run-be-ut.sh. All 26 MongoDB FE connector tests passed, with zero Checkstyle violations. The complete ASAN BE build passed. The SQL regression generated expected output and independently compared it against a recreated MongoDB 8.0.20 fixture: one suite passed, zero failed/fatal/skipped. Coverage includes explicit NULL versus absent keys, numeric extrema/nonfinite/ subnormal values, decimal overflow and rounding, datetime boundaries and precision, ObjectId/BinData bytes and subtype mismatches, strings, nested values and document limits. Clang-format 16, clang-tidy and build hygiene passed. ### Release note MongoDB binary decoding preserves full ObjectId, BinData and document bytes regardless of the declared VARBINARY length. ### Check List (For Author) - Behavior changed: Yes (see Details). - Does this need documentation: Yes; connector README updated. - Pre-submission status: Ready; local checks passed, CI pending. ### AI Disclosure - Tool: Codex (GPT-6). - Generated: code, comments, tests, fixtures, documentation and this message. - Human review: pending maintainer review.
|
run buildall |
|
/review |
### Background The native MongoDB Catalog discovers logical namespaces through the Java driver and reads BSON through the native driver and Doris SerDes. Auto schema mode must retain every declared value through a complete typed schema or document fallback. ### What problem does this PR solve? Issue Number: apache#70251 Related PR: apache#70327 (introduces the affected native paths) PR Type: Bugfix Problem Summary: Wide BSON validation can allocate tens of MiB outside query memory accounting. Metadata continuation can reach the wrong mongos, URI routing criteria are lost, and valid unsupported validators can either hide fields or fail auto resolution. Bound traversal scratch, delegate metadata cursor ownership to the pinned driver, preserve full read preferences, and reject unrepresentable validators consistently. ### Details - **Change map:** 131 added production/build lines, 298 test/fixture lines, 51 generated regression-result lines and 17 documentation lines; 146 removed lines. - **Frequency:** wide nested documents; paginated metadata across multiple mongos; URI tag/max-staleness settings; tuple, pattern and conditional-field validators. - **Root cause:** a width-sized untracked worklist, independent raw cursor commands, mode-only read preferences and an incomplete typed validator grammar. - **Why existing tests missed it:** standalone metadata fakes assumed that session identity implied server affinity; routing tests checked only mode names, schema tests missed valid tuple/pattern forms, and traversal had no allocation probe. - **Fix:** use a fixed-depth iterator stack; retain driver-owned BatchCursor connection sources and cleanup; copy complete URI preferences in FE and BE; turn tuple and schema-bearing constructs into the existing whole-collection fallback in auto mode and explicit errors in validator mode. Preserve URI precedence, metadata primary reads, permissions and object kinds. - **Other paths and branches:** recursively check root/nested/array-element schemas; change no existing connector path, Thrift field layout or capability negotiation. ### Testing Focused tests reproduced the original schema/routing failures and a 48 MiB BSON validation allocation. The old live implementation reproduced CursorNotFound while listing 140 collections across two mongos routers. All 31 FE connector tests and 142 selected BE tests passed; the refactored depth test passed a focused rerun. Complete FE and ASAN BE builds passed, with zero FE Checkstyle violations. Clang-format 16, clang-tidy and header hygiene passed for affected files. The MongoDB SQL suite generated expected output and independently compared it: one suite passed, zero failed/fatal/skipped, including fallback values, view-only permissions, a 900,000-container scalar scan and metadata batch boundaries. The fixed two-router implementation completed 20 full 140-name listings. Matching URI tags returned the scalar value; unmatched tags timed out instead of reading an ineligible member. Maximum staleness preservation is covered by both driver unit tests; this is not a failover or soak qualification. ### Release note Preserve MongoDB routing criteria and metadata cursor affinity, fall back completely for unsupported validator schemas, and bound wide BSON validation scratch memory. ### Check List (For Author) - Behavior changed: Yes (see Details). - Does this need documentation: Yes; connector and fixture READMEs updated. - Pre-submission status: Ready; local checks passed, CI pending. ### AI Disclosure - Tool: Codex (GPT-6). - Generated: implementation, comments, tests, fixtures, documentation and this message. - Human review: pending maintainer review.
|
run buildall |
|
/review |
Local pipeline review — ✅ PASSschema: doris-repo-review/v1
status: PASS
pr: apache/doris#70327
commit: 2134c7569c7ae93fa981fc1695ccfed680726235
base: 59c49656240451ed1caab1e69e69bbe0be8c61aa
reviewed_at: 2026-11-06T15:35+08:00
reviewer: Gabriel39
model: gpt-6-astra
effort: xhigh
findings: {blocker: 0, major: 0, minor: 0, nit: 0}
rounds: 1
converged: trueNotes for maintainers
Reviewed locally with the |
|
Codex automated review failed and did not complete. Error: Review recovery stopped: shared time budget or helper timeout exhausted Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
FE UT Coverage ReportIncrement line coverage |
TPC-H: Total hot run time: 28812 ms |
TPC-DS: Total hot run time: 154360 ms |
ClickBench: Total hot run time: 24.09 s |
### Background Spatial and VARBINARY SerDes share column-typed binary operations. Upstream master adds a BSON decoder specifically for ColumnVarbinary, which conflicts with that shared-base extraction. ### What problem does this PR solve? Issue Number: None Related PR: apache#69032 (spatial feature), apache#70327 (upstream BSON decoder) PR Type: Build/CI/Chore Problem Summary: Merge upstream/master at 08231e3 into the spatial branch. Retain the BSON decoder on a concrete VARBINARY SerDe derived from the existing shared base, while spatial SerDes retain the default BSON rejection. Add a focused regression for ObjectId decoding and rejection by nullable and nonnullable Geometry/Geography. ### Details - **Change map:** Manually reconcile two SerDe files and add 33 lines in the existing VARBINARY test file; the remaining changes come from the upstream merge. No new PR paths or spatial capabilities are added. - **Affected:** BE binary/spatial SerDe inheritance and BSON dispatch. No CI or runtime configuration changes. - **Developer and release impact:** Preserve both upstream VARBINARY BSON support and the spatial type boundary; ordinary key plus spatial value support is unchanged. - **License and third-party:** No PR-specific dependency, version or license edits. Local dependency setup is excluded from this commit. - **Rollback:** Revert this merge if necessary; no external state or configuration migration. ### Testing Clang-format 16, build hygiene and changed-line clang-tidy passed. The Core unity compilation containing the changed SerDe passed in the standard clang21/ASAN BE UT build, and clang-tidy analyzed both the production and test translation units with real build commands. The standard selected BE UT run did not reach test execution: local MongoDB dependencies were installed through the repository script, then an outdated Lance dependency blocked compilation. Rebuilding that pinned dependency was blocked by a reproducible local Rust subprocess-output capture failure. No runtime unit-test pass or full CI pass is claimed. ### Release note None ### Check List (For Author) - Behavior changed: No spatial functionality changes; preserve the upstream BSON behavior for VARBINARY only. - Does this need documentation: No - Pre-submission status: Partially verified: full BE UT execution is blocked by local Lance/Rust dependency setup. ### AI Disclosure - Tool: Codex (model identifier unavailable in this task) - Generated: Conflict resolution, regression, verification and commit description - Human review: Pending author review
Master added `16: optional TMongoScanDesc mongodb_params` to TTableFormatFileDesc (apache#70327), so `deltalake_params` moves to field 17. Field 17 is unused on master, branch-4.2, branch-4.1, branch-4.0 and branch-3.1. Generated code accesses the field by name, so no Java or C++ code changes. build.sh keeps master's mongodb plugin entries and adds deltalake to both connector lists.
…oDB M0 (#70456) ### Background The native MongoDB catalog uses the shared external-table discovery cache and a native single-cursor reader. A cold table lookup first enumerates names from the source. The MongoDB roadmap tracks minimal collection, view and time-series reads in M0. ### What problem does this PR solve? Issue Number: #70251 Related PR: #70327 (native reader/catalog implementation), #70249 (dependencies) PR Type: Bugfix Problem Summary: An incorrect MongoDB password or a TLS hostname failure currently reaches SQL as a missing-table error because the shared existence check discards the connector exception. Preserve the typed source error and its retry behavior. Also close M0 acceptance gaps: the 2,081-row fixture fits inside the current default batch size, so existing batch tests do not necessarily issue getMore. ### Details - **Change map:** 4 production additions; 352 test additions/5 deletions; 21 generated result lines; 36 documentation additions/1 deletion. Most of the diff is an opt-in real-driver fault runner, SQL acceptance coverage, and its instructions. - **Frequency:** Deterministic on a cold lookup when connector metadata enumeration fails, reproduced with invalid credentials and TLS hostname validation failure. - **Root cause:** ExternalDatabase treats runtime failures during the names/existence lookup as evidence that the table is absent. - **Why existing tests missed it:** Native SQL coverage lacked invalid-credential recovery. The multi-batch fixture has 2,081 rows while the default batch size is 8,160; getMore and early cursor closure were insufficiently exercised. - **Fix:** Rethrow DorisConnectorException from the shared lookup. Explicit batch-size hints force the existing boundary tests across source batches. Add credential correction, unsupported-mode rejection, view residual semantics, internal-table materialization/differential checks, and an environment-configured fault runner that verifies source cursors and successful reads after errors, timeouts and cancellation. - **Other paths and branches:** The exception fix applies to Connector SPI catalogs through the shared discovery path. No connector-specific branch, configuration default, IDL, or storage format is added. This PR targets master. - **Behavior change:** Authentication/transport failures remain source failures; a successful lookup that finds no matching table retains missing-table behavior. ### Testing Completed locally on Linux x86_64 with MongoDB 8.0.20, Java driver 5.14.0 and C driver 2.5.8: - Complete clean FE and ASAN BE build through build.sh --clean --be --fe. - FE Checkstyle: zero violations, including a separate fe-core check. - New discovery unit test failed before the fix because no exception was thrown. After the fix, all 51 ExternalCatalogTest tests passed; all 31 native MongoDB connector tests also passed. - Selected BSON SerDe/native reader unit tests through run-be-ut.sh: 44 passed. - Both native SQL suites: generated the new expected output with the normal runner, then independently compared both suites: 2 passed, 0 failed/fatal/skipped. The internal-table differential is zero across 2,081 fixture rows. - Final opt-in lifecycle runner passed EOF/LIMIT/decode-error cleanup, injected getMore failure, socket and query deadlines, cancellation after a blocked getMore was observed, concurrent reads, and subsequent recovery. Source open cursors returned to zero after each scenario. The query deadline case uses a socket timeout longer than the injected delay, so a socket timeout cannot mask a missing query deadline. - Trusted-CA TLS reads passed. Untrusted certificates and hostname mismatches were rejected with source errors. Snappy, zlib and zstd multi-batch reads passed; MongoDB client metadata confirmed negotiation for both drivers. SCRAM-SHA-256 and URI-escaped SCRAM-SHA-1 credentials passed. - Python Ruff checks and formatting passed. Full diff/self-review and private-context scan completed. Generated output retains the regression runner's canonical blank-line separators. ### Notes for Reviewers The fault runner requires an otherwise idle, disposable, test-command-enabled MongoDB fixture and explicit environment-injected connection settings; ordinary external regression does not enable failpoints. Its invocation is documented next to the fixture. The SQL suites remain gated by enableMongoDBTest. This is M0 single-cursor acceptance with matching-version FE/BE in the tested environment. Software upgrade/downgrade qualification, SRV validation, multi-architecture validation and FE/BE capability negotiation are excluded from the agreed scope. MongoDB 7, broader source-topology/failover qualification, performance and release soak remain at their later roadmap gates in #70251. ### Release note Preserve connector metadata authentication and transport errors during cold external-table lookups instead of reporting a missing table. ### Check List (For Author) - Behavior changed: Yes (see Details). - Does this need documentation: No user-facing configuration change; fixture and fault-test instructions are included. - Pre-submission status: Ready; local validation above completed, PR CI pending. ### AI Disclosure - Tool: Codex (GPT-6). - Generated: fix, tests, fixture instructions, and this description. - Human review: Pending maintainer review.
### Background The native MongoDB catalog sends read preference and read concern to the BE in each scan descriptor. Today those values come only from catalog properties or the connection URI, while metadata discovery uses its own primary/default policy. ### What problem does this PR solve? Issue Number: #70251 Related PR: #70327 (native reader and catalog read policies) PR Type: Feature Problem Summary: Users cannot choose MongoDB routing and visibility for an individual session or query without changing the shared catalog. Add `mongodb_read_preference` and `mongodb_read_concern`, supporting `SET` and `SET_VAR`, and pin their effective values in each scan descriptor. Both default to `inherit`, preserving existing catalog/URI behavior. Most of the diff adds tests and generated regression expectations. ### Details - **Change map:** 70 production lines changed, 281 test-source lines changed, 62 generated expectation lines, and 38 documentation lines changed. - **Scope:** Data-read policies for native MongoDB scans. Preference supports `inherit`, `primary`, `primaryPreferred`, `secondary`, `secondaryPreferred`, and `nearest`; concern supports `inherit`, `local`, `majority`, and `available`. Values are case-insensitive. Snapshot and linearizable reads remain unsupported. - **Design:** Register validated, forwarded session variables and resolve them through the existing connector session-property snapshot. Reuse the existing scan descriptor and BE collection policy application. Session overrides do not mutate catalog configuration, metadata clients, or transport pool identity. URI tags, maximum staleness, and hedge options remain effective; an incompatible primary override fails during planning without exposing the URI. - **Usage:** `SET mongodb_read_preference='secondaryPreferred'; SET mongodb_read_concern='local';` or `SELECT /*+ SET_VAR(mongodb_read_preference='primary', mongodb_read_concern='majority') */ ...`. Set either variable to `inherit` to use its catalog/URI policy again. - **Default behavior:** `inherit` independently resolves each policy from URI options first, then catalog defaults. Queries without overrides retain their existing behavior. - **Compatibility:** No connector SPI, Thrift, BE code, or storage-format changes. Metadata discovery and collection-identity verification retain their existing policies. Read concern does not establish a snapshot shared by scans. ### Testing All checks below ran locally against MongoDB 8.0.20, with the new FE and the existing ASAN native BE: - Red/green: the new scan-plan tests first failed because overrides were ignored, and the new session tests first failed because the variables were unregistered; both passed after implementation. - `run-fe-ut.sh --run` with `SessionVariableMongoReadPolicyTest`, `VariableMgrExportTest`, `MongoScanPlanTest`, and `MongoCatalogPropertiesTest`: passed. The three new session tests cover defaults, supported/unsupported values, connector snapshots, isolation, and forwarding. - `mvn -f fe/pom.xml -pl :fe-connector-mongodb -am test`: 35 connector tests passed, including URI precedence, independent policy inheritance, incompatible routing criteria, and stable transport configuration identity. - `build.sh --fe -j12`: passed. FE core and MongoDB connector Checkstyle: zero violations. - Generated `test_mongodb_read_policy.out` using the regression runner, checked its results against the synthetic fixture, then independently reran `test_mongodb_native` and `test_mongodb_read_policy`: 2 suites passed, 0 failed/fatal/skipped. Covers SET/SET_VAR, all supported modes/levels, invalid values, failed-hint restoration, URI policy inheritance, and recovery. - Additional replica-set integration: source profiler verified primary/secondary selection and the actual read concern for 8 sequential reads and 32 concurrent reads across 4 isolated sessions. Also verified URI tags with a mode override and policy restoration after a failed hint. This was an additional local acceptance check; the committed SQL suites use the existing standalone fixture. ### Release note Add `mongodb_read_preference` and `mongodb_read_concern` session variables for native MongoDB queries, with statement-level SET_VAR support. Both default to `inherit` to retain the catalog's existing read policy. ### Check List (For Author) - Behavior changed: Yes (see Details; explicit overrides now affect native MongoDB data reads). - Does this need documentation: Yes; the connector README is updated in this PR. - Pre-submission status: Ready. Local checks passed; CI is pending. ### AI Disclosure - Tool: Codex (GPT-6). - Generated: Implementation, tests, connector documentation, and this description. - Human review: Maintainer review pending; the agent completed a full diff self-review and local validation.
The native MongoDB Catalog on master already provides single-cursor reads and first-release BSON SerDes. SQL filters and source projection have remained local, and the remaining M1 type and pushdown contracts need end-to-end qualification. Issue Number: apache#70251 (D04b and D05) Related PR: apache#70327 (native read path), apache#70456 (M0 acceptance) PR Type: Feature Problem Summary: Complete the D04b qualification coverage and connect conservative predicate/LIKE candidates and column pruning to the native MongoDB reader. Every SQL predicate remains a Doris residual; the default strict type policy, strict SQL casts and standard views keep filtering local. This PR does not complete the separate D06 or later snapshot/parallel-scan work. - **Change map:** 449 added production lines, 1,187 test/fixture/generated-result lines, and 47 documentation lines across 19 files. The production changes comprise a connector-local translator and scan planning, three session controls, two optional descriptor fields, and native find options/validation. Most additions qualify types and compare execution modes. - **Scope:** D04b covers Decimal128 extremes, exact integer/floating-point behavior, UTC dates and scales, binary ownership/subtypes, missing/null, recursive arrays/structs, rollback and existing document limits. D05 covers typed scalar-root comparisons, positive IN/BETWEEN, IS NOT NULL, safe AND/OR, necessary-literal LIKE candidates, root projection and time-series Date ranges. - **Design:** Candidates are classified conservatively as SUPERSET; unsupported expressions remain local, and the EXACT category is reserved for proven equivalence. No conjunct is consumed through either SPI path. CASTs are rejected before the neutral converter can erase them. The translator reconciles the schema/expression SPI date-scale and decimal-width representations, including FE's DECIMAL128 name. Native BSON is bounded and validated before client acquisition; views cannot receive a predicate/collation override. Complete root projection retains residual columns and scalar-ancestor errors. - **Usage:** Use `SET` or `SET_VAR` with `enable_mongodb_predicate_pushdown`, `enable_mongodb_like_pushdown` and `enable_mongodb_projection_pushdown`. Predicate candidates require the explicit catalog policy `mongodb.type_mismatch=null`. - **Default behavior:** All three switches default to true. The default `type_mismatch=error` policy preserves local filtering; `enable_strict_cast=true` and views also preserve local filtering. Source projection is eligible independently. DOUBLE/NaN comparisons, NOT/!=/NOT IN, mismatch-sensitive IS NULL, functions, casts, nested predicate paths and existing direct VARBINARY operator restrictions stay local. LIKE keeps the complete SQL residual. LIMIT, aggregation, sort, joins, snapshots and parallel scans are unchanged. - **Compatibility:** The connector SPI and stored schemas are unchanged. Descriptor fields 23/24 are optional; absent fields preserve the prior full scan. The touched descriptor is absent from the four supported release branches, so no field id is reused. Qualification uses matching FE/BE versions. All testing below ran locally against MongoDB 8.0.20 with authentication enabled and an ASAN BE: - 59 MongoDB connector FE tests, 25 connector-session/forwarding tests and 55 BSON/native-reader BE tests passed. - All three MongoDB SQL suites passed: 3 suites, 0 failed, 0 fatal, 0 skipped. This includes the existing native and M0 acceptance suites plus the new M1 suite. - The generated M1 output contains 38 identical sorted result pairs with predicate/LIKE/projection disabled and enabled. It covers precision, NULL/arrays, casts, collations, residual LIMIT, aggregates/joins, views and time-series reads. The completed Profile check reports `MongoRowsRead` of 10 with candidates disabled and 1 with them enabled. - Source profiling independently confirmed actual BSON int64, Decimal128 and Date filters, projection, simple collection collation, and view reads without a collation override. - FE and ASAN BE builds passed; the final MongoDB plugin was subsequently packaged/tested and deployed from its final source. Full FE Checkstyle and the final affected-core-file Checkstyle run passed with zero violations. All affected C/C++ files pass clang-format 16, build hygiene and clang-tidy. - Red/green checks reproduced missing candidate planning, acceptance of malformed BSON descriptors, missing strict-cast gating, command-budget gaps and schema/expression type representation gaps before the corresponding implementations. Native MongoDB scans can prune source fields and apply conservative WHERE/LIKE candidates under the explicit null-on-type-mismatch policy. SQL residuals remain active, and default strict scans and views preserve local filtering. New session switches allow statement-level comparisons and disabling these optimizations. - Behavior changed: Yes (see Details). - Does this need documentation: Yes; the connector and fixture READMEs are updated here. - Pre-submission status: Ready. Local UT, authenticated SQL regression, builds and style/static checks passed; CI is pending. - Tool: Codex (GPT-6). - Generated: Implementation, tests, fixture, documentation and this description. - Human review: Pending maintainer review.
…udit text ### Background SHOW CREATE CATALOG, SHOW CATALOG, the `catalogs()` table function and the audit text of CREATE CATALOG replace a property value with `*XXX` when `DatasourcePrintableMap.isSensitiveKey` matches its key. Until now the keys came from a list in fe-core and from the keys filesystem plugins report at startup. A connector plugin marks its secret properties `sensitive = true`, but that mark only masks the connector's own log lines. A connector tells the engine about its security-relevant properties through `ConnectorProvider#securitySurface()`. ### What problem does this PR solve? Issue Number: None Related PR: apache#69588 (added the connector security surface this extends), apache#45164 (introduced the unmasked ALTER CATALOG audit text), apache#27115 (masked it for the statement apache#45164 replaced), apache#59502 (added `iceberg.jdbc.password`), apache#54523 (added the Paimon REST DLF keys), apache#65730 (branch-4.1's Lance catalog, whose REST secrets the Lance port will declare on this surface), apache#70366 (fixed the shared Unicode SQL rewrite used by this PR's ALTER masking), apache#70327 (added MongoDB credentials to the engine masking floor) PR Type: Bugfix Problem Summary: Catalog secrets that only a connector knows are on no masking list, so SHOW CREATE CATALOG and the other renderings print them in clear: `iceberg.jdbc.password`, `paimon.jdbc.password`, the Paimon REST DLF key pair, `mc.access_key`, and credentials passed through under `trino.` or `adbc.`. Separately, `ALTER CATALOG ... SET PROPERTIES` is written to the audit log unmasked, whatever the key. Each connector now declares its secrets on its security surface, taken from its `sensitive` marks, and the surface can also declare a namespace of secret keys for options passed through unread; Fluss's lake options were hard-coded in fe-core for that reason. The engine masks the union, with fe-core's own list kept as the floor. ALTER CATALOG is masked like CREATE CATALOG. ### Details - **Change map:** 45 files, +1205 −81 over four commits: production code +384 −51, tests +807 −26, the connector README +13 −3, the plugin API version in `fe/fe-connector/pom.xml` (31.0 to 32.0). The first commit adds secret keys to the surface (reviewed alone before). The second does the rest, the third keeps a surface that fails in the connector's own `ServiceLoader` from stopping FE, and the fourth aligns the newly added MongoDB provider and tests ALTER masking with supplementary characters: 1. Key namespaces on the SPI: `withSensitiveKeyPrefix` and `SensitiveKeyPrefix.covers`, which holds the matching rule; a collection form of `withSensitiveKeys` (`ConnectorSecuritySurface`). 2. Engine: `isSensitiveKey` matches registered namespaces, the Fluss rule becomes one of them; `ConnectorPluginManager` registers them and reads every surface before it publishes a provider, skipping a provider whose surface throws. 3. Connectors: every provider declares its secrets; `MetaStoreProviders.sensitivePropertyKeys()` gives the union of a plugin's metastore backends; `IcebergJdbcMetaStoreProvider` reports its keys like its siblings; the keytab paths lose their `sensitive` mark. MongoDB also declares its two annotated secrets, preserving its SSRF endpoint and the engine fallback. 4. ALTER CATALOG audit masking (+16). - **Frequency:** Always. Create or alter a catalog that sets one of the keys above, then run SHOW CREATE CATALOG, SHOW CATALOG `<name>` or `select * from catalogs()`, or read the audit log after an ALTER CATALOG ... SET PROPERTIES. - **Root cause:** `isSensitiveKey` consulted only fe-core's list and the filesystem providers' keys. A connector's `sensitive` marks fed only `toMaskedString`; `MetaStoreProvider.sensitivePropertyKeys()` existed for this, but nothing called it, and the Iceberg JDBC backend did not implement it. For ALTER CATALOG, `AlterCatalogPropertiesCommand` does not implement `NeedAuditEncryption` and `LogicalPlanBuilderForEncryption` has no `visitAlterCatalogProperties`, so `AuditLogHelper` records the original text; the statement it replaced in apache#45164 was masked by apache#27115. - **Why existing tests missed it:** Unit tests: `DatasourcePrintableMapTest` checks fe-core's own keys, the provider surface tests checked only the SSRF and driver_url keys, and `EncryptSQLTest` and `RepositoryAuditEncryptionTest` cover CREATE CATALOG, repositories and resources but not ALTER CATALOG. Regression: `audit/test_external_catalog_credential_masking` creates only Iceberg REST catalogs, whose keys are on fe-core's list, and never alters one. - **Fix:** The engine must not name a connector's properties, so the connectors declare their secrets, from the same `sensitive` marks that mask their logs; jdbc and es declare both spellings their legacy prefix allows, and trino and adbc declare the namespace they pass through, a key there being a secret when its rest names `key`, `secret`, `password`, `token` or `credential` (also `authorization` for ADBC). fe-core keeps the connector keys it lists, so a catalog whose plugin is not loaded masks what it masked before. A provider whose `securitySurface()` throws at startup is skipped, as a filesystem provider is, instead of stopping FE; that includes the `ServiceConfigurationError` a `ServiceLoader` throws when a descriptor names a backend it cannot load, which is how the Hive, Iceberg and Paimon surfaces fail, since they read their metastore backends' keys through one. - **Other paths and branches:** Fixed here: every rendering above and FE's statement logging, which share `isSensitiveKey` and `NeedAuditEncryption`. Not covered, since a key name cannot express them safely: raw Iceberg SDK keys copied through (`token`, `credential`), Paimon options re-keyed from `paimon.*`, Hadoop `fs.*` keys, and secrets inside otherwise unmarked URLs. The ALTER CATALOG gap is on branch-4.0, 4.1 and 4.2 (from apache#45164); branch-3.1 still uses the masked statement. branch-4.1 and 4.2 also print the connector keys above in clear; without the connector SPI their fix differs and is tracked separately. - **Behavior change:** The namespaces also mask non-secrets whose names carry those words, such as `trino.hive.s3.aws-access-key` or a keystore path. Keytab paths, which no release masks, are no longer masked in connector log lines: marked `sensitive`, `hadoop.kerberos.keytab` would be masked in every storage vault and resource. ### Testing The original local runs (macOS) used master `f5c993f2c94` plus this change: - Regression `audit/test_external_catalog_credential_masking`, extended with an Iceberg JDBC catalog: SHOW CREATE CATALOG masks `iceberg.jdbc.password`, and the audit texts of its CREATE CATALOG and of an ALTER CATALOG that sets the password and `s3.secret_key` hold only `*XXX`. Against the local cluster's FE without this change it fails at the first new check, SHOW CREATE CATALOG printing the password; with a FE built from this change it passes. - fe-connector-spi: all 160 tests, with the refreshed surface baseline (+4 lines); the metastore modules: all tests; ten connector modules: 29 classes, 511 tests, including each connector's surface test, which checks real key names. - fe-core: 26 classes, 341 tests, including `ConnectorPluginManagerTest`, `DatasourcePrintableMapTest`, `EncryptSQLTest` and the audit and plugin gate tests; 14 of them again after the last change to registration, and 7 classes, 76 tests, after the third commit. Its two new `ConnectorPluginManagerTest` cases, a plugin directory whose `lib/` holds a jar with a service descriptor naming a missing metastore-style backend (the plugin next to it must still load) and the same descriptor on the classpath, fail without the third commit with that `ServiceConfigurationError` escaping `loadPlugins` and `loadBuiltins`, and pass with it. - Twenty-two mutations, each caught by its test: for example, reverting either half of the ALTER CATALOG fix (`EncryptSQLTest` sees the statement in clear), matching a marker anywhere in the key, an empty metastore union, re-marking a keytab, or dropping `ServiceConfigurationError` from either startup catch (each fails only its own new case). Rebased onto master `93c16ee2ca1` on 2026-11-16. Post-rebase local verification: - `bash run-fe-ut.sh --run 'org.apache.doris.connector.spi.*Test,ConnectorPluginManagerTest,DatasourcePrintableMapTest,EncryptSQLTest'`, with `CUSTOM_MVN` appending `-Dcheckstyle.skip=false`: BUILD SUCCESS across all 90 reactor modules; all 141 selected tests passed across 22 classes, including the inherited supplementary-character audit cases. Checkstyle reported no violations. - `git diff --check upstream-apache/master...HEAD`: passed. `git range-diff` confirms the replayed patches are unchanged apart from adapting the connector API version and its test to 32.0. - Regression tests and a complete BE/FE build were not rerun locally after this rebase; `run buildall` is requested in CI. Follow-up local verification on the rebased branch plus the MongoDB declaration and audit tests (2026-11-16): - `run-fe-ut.sh --run 'org.apache.doris.connector.spi.*Test,org.apache.doris.connector.mongodb.*Test,ConnectorPluginManagerTest,DatasourcePrintableMapTest,EncryptSQLTest'`, with Checkstyle enabled: all 176 selected tests in 28 classes passed (SPI 93, MongoDB 35, fe-core 48), with no skipped tests; the 90-module reactor completed successfully. - The strengthened `MongoProviderTest` fails without the provider declaration (expected the two MongoDB secret keys, received an empty set), and passes with it. A separate targeted revert confirms the assertion catches the missing declaration. - The new ALTER CATALOG assertions cover supplementary characters within a secret, in a preceding SQL comment, and in a retained non-secret value, with two masked properties. `EncryptSQLTest#testEncryption` passes on the candidate and fails at the new ALTER assertion when only `BaseViewInfo.rewriteSql` is restored to its pre-apache#70366 implementation: the audit text has an extra trailing fragment. - Full cluster regression and a standalone FE/BE build were not rerun for this follow-up. ### Notes for Reviewers - `SensitiveKeyPrefix.covers` lives on the SPI so that a connector's test can check its declaration against real key names and the engine applies the same rule; the Fluss floor in fe-core uses it too. - The entries both fe-core and a connector declare are deliberate: the floor is frozen, as the comment in `DatasourcePrintableMap` says. MongoDB was already masked by apache#70327; its added declaration makes the provider follow the same contract as the other connectors. - The connector plugin API advances from 31.0 to 32.0 for the added security-surface methods. Third-party connector plugins built against 31.x must be rebuilt; the in-tree plugins are built with FE. Hudi declares nothing: it owns no catalog properties. - The base includes apache#70366, so ALTER CATALOG masking uses the shared rewrite that converts parser code-point spans to Java string offsets, including supplementary characters. ### Release note SHOW CREATE CATALOG, SHOW CATALOG, the `catalogs()` table function and the audit log now mask `iceberg.jdbc.password`, `paimon.jdbc.password`, `paimon.rest.dlf.access-key-id`, `paimon.rest.dlf.access-key-secret`, `mc.access_key`, and the `trino.*` and `adbc.*` properties whose names carry `key`, `secret`, `password`, `token` or `credential` (`authorization` too for `adbc.*`). ALTER CATALOG ... SET PROPERTIES statements are masked in the audit log like CREATE CATALOG. Third-party connector plugins built against API 31.x must be rebuilt for 32.0. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: No - Pre-submission status: Ready ### AI Disclosure - Tool: Claude Code (Claude Opus 5.5); OpenAI Codex - Generated: Claude Code wrote the original code, tests, and description. Codex rebased the commits, adapted the connector API version and its test, added the MongoDB declaration and ALTER CATALOG supplementary-character tests, and updated this description. - Human review: not stated - Prompts: The author requested a generic connector surface, prefix rules, declarations for every connector, and masking of their union with the engine floor. They removed sensitive marks from keytab paths, added ALTER CATALOG masking, and fixed startup failure from a connector's ServiceLoader. After review, they chose a separate shared Unicode fix, then requested MongoDB alignment and supplementary-character audit coverage here.
…70604) ### Background The native MongoDB Catalog on master already provides single-cursor reads and first-release BSON SerDes. SQL filters and source projection have remained local, and the remaining M1 type and pushdown contracts need end-to-end qualification. ### What problem does this PR solve? Issue Number: #70251 (D04b and D05) Related PR: #70327 (native read path), #70456 (M0 acceptance), #70493 (session read policies) PR Type: Feature Problem Summary: Complete the D04b qualification coverage and connect conservative predicate/LIKE candidates and column pruning to the native MongoDB reader. Every SQL predicate remains a Doris residual; the default strict type policy, strict SQL casts and standard views keep filtering local. This PR does not complete the separate D06 or later snapshot/parallel-scan work. ### Details - **Change map:** 456 added production lines, 1,221 test/fixture/generated-result lines, and 47 documentation lines across 19 files. The production changes comprise a connector-local translator and scan planning, three session controls, two optional descriptor fields, and native find options/validation. Most additions qualify types and compare execution modes. - **Scope:** D04b covers Decimal128 extremes, exact integer/floating-point behavior, UTC dates and scales, binary ownership/subtypes, missing/null, recursive arrays/structs, rollback and existing document limits. D05 covers typed scalar-root comparisons, positive IN/BETWEEN, scalar IS NOT NULL, safe AND/OR, necessary-literal LIKE candidates, root projection and time-series Date ranges. - **Design:** Candidates are classified conservatively as SUPERSET; unsupported expressions remain local, and the EXACT category is reserved for proven equivalence. No conjunct is consumed through either SPI path. CASTs are rejected before the neutral converter can erase them. The translator reconciles the schema/expression SPI date-scale and decimal-width representations, including FE's DECIMAL128 name. Native BSON is bounded and validated before client acquisition; views cannot receive a predicate/collation override. Complete root projection retains residual columns and scalar-ancestor errors. - **Usage:** Use `SET` or `SET_VAR` with `enable_mongodb_predicate_pushdown`, `enable_mongodb_like_pushdown` and `enable_mongodb_projection_pushdown`. Predicate candidates require the explicit catalog policy `mongodb.type_mismatch=null`. - **Default behavior:** All three switches default to true. The default `type_mismatch=error` policy preserves local filtering; `enable_strict_cast=true` and views also preserve local filtering. Source projection is eligible independently. DOUBLE/NaN comparisons, NOT/!=/NOT IN, mismatch-sensitive IS NULL, functions, casts, nested predicate paths and existing direct VARBINARY operator restrictions stay local. LIKE keeps the complete SQL residual. LIMIT, aggregation, sort, joins, snapshots and parallel scans are unchanged. - **Compatibility:** The connector SPI and stored schemas are unchanged. Descriptor fields 23/24 are optional; absent fields preserve the prior full scan. The touched descriptor is absent from the four supported release branches, so no field id is reused. Qualification uses matching FE/BE versions. ### Testing All testing below ran locally against MongoDB 8.0.20 with authentication enabled and an ASAN BE: - 64 MongoDB connector FE tests, 25 connector-session/forwarding tests, 3 session read-policy tests and 55 BSON/native-reader BE tests passed (147 total). - All four MongoDB SQL suites passed: 4 suites, 0 failed, 0 fatal, 0 skipped. This includes the existing native and M0 acceptance suites, the session read-policy suite and the new M1 suite. - The generated M1 output contains 40 identical sorted result pairs with predicate/LIKE/projection disabled and enabled. It covers precision, NULL/arrays, casts, collations, residual LIMIT, aggregates/joins, views and time-series reads. The completed Profile check reports `MongoRowsRead` of 10 with candidates disabled and 1 with them enabled. - Source profiling independently confirmed actual BSON int64, Decimal128 and Date filters, projection, simple collection collation, and view reads without a collation override. - Full FE and ASAN BE builds passed on the integrated master base; the SQL suites use that matching built pair. Full FE Checkstyle and the final affected-core-file Checkstyle run passed with zero violations. All affected C/C++ files pass clang-format 16, build hygiene and clang-tidy. - Red/green checks reproduced missing candidate planning, acceptance of malformed BSON descriptors, missing strict-cast gating, command-budget gaps, schema/expression type representation gaps, and the false negative for ARRAY/JSON IS NOT NULL with null elements before the corresponding implementations. ### Release note Native MongoDB scans can prune source fields and apply conservative WHERE/LIKE candidates under the explicit null-on-type-mismatch policy. SQL residuals remain active, and default strict scans and views preserve local filtering. New session switches allow statement-level comparisons and disabling these optimizations. ### Check List (For Author) - Behavior changed: Yes (see Details). - Does this need documentation: Yes; the connector and fixture READMEs are updated here. - Pre-submission status: Ready. Local UT, authenticated SQL regression, builds and style/static checks passed; CI is pending. ### AI Disclosure - Tool: Codex (GPT-6). - Generated: Implementation, tests, fixture, documentation and this description. - Human review: Pending maintainer review.
…udit text (#70338) ### Background SHOW CREATE CATALOG, SHOW CATALOG, the `catalogs()` table function and the audit text of CREATE CATALOG replace a property value with `*XXX` when `DatasourcePrintableMap.isSensitiveKey` matches its key. Until now the keys came from a list in fe-core and from the keys filesystem plugins report at startup. A connector plugin marks its secret properties `sensitive = true`, but that mark only masks the connector's own log lines. A connector tells the engine about its security-relevant properties through `ConnectorProvider#securitySurface()`. ### What problem does this PR solve? Issue Number: None Related PR: #69588 (added the connector security surface this extends), #45164 (introduced the unmasked ALTER CATALOG audit text), #27115 (masked it for the statement #45164 replaced), #59502 (added `iceberg.jdbc.password`), #54523 (added the Paimon REST DLF keys), #65730 (branch-4.1's Lance catalog, whose REST secrets the Lance port will declare on this surface), #70366 (fixed the shared Unicode SQL rewrite used by this PR's ALTER masking), #70327 (added MongoDB credentials to the engine masking floor) PR Type: Bugfix Problem Summary: Catalog secrets that only a connector knows are on no masking list, so SHOW CREATE CATALOG and the other renderings print them in clear: `iceberg.jdbc.password`, `paimon.jdbc.password`, the Paimon REST DLF key pair, `mc.access_key`, and credentials passed through under `trino.` or `adbc.`. Separately, `ALTER CATALOG ... SET PROPERTIES` is written to the audit log unmasked, whatever the key. Each connector now declares its secrets on its security surface, taken from its `sensitive` marks, and the surface can also declare a namespace of secret keys for options passed through unread; Fluss's lake options were hard-coded in fe-core for that reason. The engine masks the union, with fe-core's own list kept as the floor. ALTER CATALOG is masked like CREATE CATALOG. ### Details - **Change map:** 45 files, +1205 −81 over four commits: production code +384 −51, tests +807 −26, the connector README +13 −3, the plugin API version in `fe/fe-connector/pom.xml` (31.0 to 32.0). The first commit adds secret keys to the surface (reviewed alone before). The second does the rest, the third keeps a surface that fails in the connector's own `ServiceLoader` from stopping FE, and the fourth aligns the newly added MongoDB provider and tests ALTER masking with supplementary characters: 1. Key namespaces on the SPI: `withSensitiveKeyPrefix` and `SensitiveKeyPrefix.covers`, which holds the matching rule; a collection form of `withSensitiveKeys` (`ConnectorSecuritySurface`). 2. Engine: `isSensitiveKey` matches registered namespaces, the Fluss rule becomes one of them; `ConnectorPluginManager` registers them and reads every surface before it publishes a provider, skipping a provider whose surface throws. 3. Connectors: every provider declares its secrets; `MetaStoreProviders.sensitivePropertyKeys()` gives the union of a plugin's metastore backends; `IcebergJdbcMetaStoreProvider` reports its keys like its siblings; the keytab paths lose their `sensitive` mark. MongoDB also declares its two annotated secrets, preserving its SSRF endpoint and the engine fallback. 4. ALTER CATALOG audit masking (+16). - **Frequency:** Always. Create or alter a catalog that sets one of the keys above, then run SHOW CREATE CATALOG, SHOW CATALOG `<name>` or `select * from catalogs()`, or read the audit log after an ALTER CATALOG ... SET PROPERTIES. - **Root cause:** `isSensitiveKey` consulted only fe-core's list and the filesystem providers' keys. A connector's `sensitive` marks fed only `toMaskedString`; `MetaStoreProvider.sensitivePropertyKeys()` existed for this, but nothing called it, and the Iceberg JDBC backend did not implement it. For ALTER CATALOG, `AlterCatalogPropertiesCommand` does not implement `NeedAuditEncryption` and `LogicalPlanBuilderForEncryption` has no `visitAlterCatalogProperties`, so `AuditLogHelper` records the original text; the statement it replaced in #45164 was masked by #27115. - **Why existing tests missed it:** Unit tests: `DatasourcePrintableMapTest` checks fe-core's own keys, the provider surface tests checked only the SSRF and driver_url keys, and `EncryptSQLTest` and `RepositoryAuditEncryptionTest` cover CREATE CATALOG, repositories and resources but not ALTER CATALOG. Regression: `audit/test_external_catalog_credential_masking` creates only Iceberg REST catalogs, whose keys are on fe-core's list, and never alters one. - **Fix:** The engine must not name a connector's properties, so the connectors declare their secrets, from the same `sensitive` marks that mask their logs; jdbc and es declare both spellings their legacy prefix allows, and trino and adbc declare the namespace they pass through, a key there being a secret when its rest names `key`, `secret`, `password`, `token` or `credential` (also `authorization` for ADBC). fe-core keeps the connector keys it lists, so a catalog whose plugin is not loaded masks what it masked before. A provider whose `securitySurface()` throws at startup is skipped, as a filesystem provider is, instead of stopping FE; that includes the `ServiceConfigurationError` a `ServiceLoader` throws when a descriptor names a backend it cannot load, which is how the Hive, Iceberg and Paimon surfaces fail, since they read their metastore backends' keys through one. - **Other paths and branches:** Fixed here: every rendering above and FE's statement logging, which share `isSensitiveKey` and `NeedAuditEncryption`. Not covered, since a key name cannot express them safely: raw Iceberg SDK keys copied through (`token`, `credential`), Paimon options re-keyed from `paimon.*`, Hadoop `fs.*` keys, and secrets inside otherwise unmarked URLs. The ALTER CATALOG gap is on branch-4.0, 4.1 and 4.2 (from #45164); branch-3.1 still uses the masked statement. branch-4.1 and 4.2 also print the connector keys above in clear; without the connector SPI their fix differs and is tracked separately. - **Behavior change:** The namespaces also mask non-secrets whose names carry those words, such as `trino.hive.s3.aws-access-key` or a keystore path. Keytab paths, which no release masks, are no longer masked in connector log lines: marked `sensitive`, `hadoop.kerberos.keytab` would be masked in every storage vault and resource. ### Testing The original local runs (macOS) used master `f5c993f2c94` plus this change: - Regression `audit/test_external_catalog_credential_masking`, extended with an Iceberg JDBC catalog: SHOW CREATE CATALOG masks `iceberg.jdbc.password`, and the audit texts of its CREATE CATALOG and of an ALTER CATALOG that sets the password and `s3.secret_key` hold only `*XXX`. Against the local cluster's FE without this change it fails at the first new check, SHOW CREATE CATALOG printing the password; with a FE built from this change it passes. - fe-connector-spi: all 160 tests, with the refreshed surface baseline (+4 lines); the metastore modules: all tests; ten connector modules: 29 classes, 511 tests, including each connector's surface test, which checks real key names. - fe-core: 26 classes, 341 tests, including `ConnectorPluginManagerTest`, `DatasourcePrintableMapTest`, `EncryptSQLTest` and the audit and plugin gate tests; 14 of them again after the last change to registration, and 7 classes, 76 tests, after the third commit. Its two new `ConnectorPluginManagerTest` cases, a plugin directory whose `lib/` holds a jar with a service descriptor naming a missing metastore-style backend (the plugin next to it must still load) and the same descriptor on the classpath, fail without the third commit with that `ServiceConfigurationError` escaping `loadPlugins` and `loadBuiltins`, and pass with it. - Twenty-two mutations, each caught by its test: for example, reverting either half of the ALTER CATALOG fix (`EncryptSQLTest` sees the statement in clear), matching a marker anywhere in the key, an empty metastore union, re-marking a keytab, or dropping `ServiceConfigurationError` from either startup catch (each fails only its own new case). Rebased onto master `93c16ee2ca1` on 2026-11-16. Post-rebase local verification: - `bash run-fe-ut.sh --run 'org.apache.doris.connector.spi.*Test,ConnectorPluginManagerTest,DatasourcePrintableMapTest,EncryptSQLTest'`, with `CUSTOM_MVN` appending `-Dcheckstyle.skip=false`: BUILD SUCCESS across all 90 reactor modules; all 141 selected tests passed across 22 classes, including the inherited supplementary-character audit cases. Checkstyle reported no violations. - `git diff --check upstream-apache/master...HEAD`: passed. `git range-diff` confirms the replayed patches are unchanged apart from adapting the connector API version and its test to 32.0. - Regression tests and a complete BE/FE build were not rerun locally after this rebase; `run buildall` is requested in CI. Follow-up local verification on the rebased branch plus the MongoDB declaration and audit tests (2026-11-16): - `run-fe-ut.sh --run 'org.apache.doris.connector.spi.*Test,org.apache.doris.connector.mongodb.*Test,ConnectorPluginManagerTest,DatasourcePrintableMapTest,EncryptSQLTest'`, with Checkstyle enabled: all 176 selected tests in 28 classes passed (SPI 93, MongoDB 35, fe-core 48), with no skipped tests; the 90-module reactor completed successfully. - The strengthened `MongoProviderTest` fails without the provider declaration (expected the two MongoDB secret keys, received an empty set), and passes with it. A separate targeted revert confirms the assertion catches the missing declaration. - The new ALTER CATALOG assertions cover supplementary characters within a secret, in a preceding SQL comment, and in a retained non-secret value, with two masked properties. `EncryptSQLTest#testEncryption` passes on the candidate and fails at the new ALTER assertion when only `BaseViewInfo.rewriteSql` is restored to its pre-#70366 implementation: the audit text has an extra trailing fragment. - Full cluster regression and a standalone FE/BE build were not rerun for this follow-up. ### Notes for Reviewers - `SensitiveKeyPrefix.covers` lives on the SPI so that a connector's test can check its declaration against real key names and the engine applies the same rule; the Fluss floor in fe-core uses it too. - The entries both fe-core and a connector declare are deliberate: the floor is frozen, as the comment in `DatasourcePrintableMap` says. MongoDB was already masked by #70327; its added declaration makes the provider follow the same contract as the other connectors. - The connector plugin API advances from 31.0 to 32.0 for the added security-surface methods. Third-party connector plugins built against 31.x must be rebuilt; the in-tree plugins are built with FE. Hudi declares nothing: it owns no catalog properties. - The base includes #70366, so ALTER CATALOG masking uses the shared rewrite that converts parser code-point spans to Java string offsets, including supplementary characters.
Background
The native MongoDB dependency preparation (#70249) provides the FE Java driver and BE C driver, but does not register a usable Catalog. Doris needs a native metadata and scan path that preserves BSON types and reads collection, view, and time-series namespaces directly.
What problem does this PR solve?
Issue Number: #70251
Related PR: #70249 (client dependencies)
PR Type: Feature
Problem Summary:
Implement D01, D03, and D04 of the native MongoDB Catalog roadmap, plus the minimal single-Split descriptor needed to connect them. Users can create a
type=mongodbCatalog and query ordinary collections, standard views, and time-series measurements through a native C++ reader. FE pins explicit, validator, or document schemas; concreteDataTypeSerDeimplementations convert BSON without losing int64 precision, ObjectId bytes, or binary subtype identity.Details
findaccess; internal time-series buckets are hidden. All SQL filters, LIKE/REGEXP, joins, windows, aggregates, sorting, and LIMIT are evaluated in Doris. Predicate/projection pushdown, parallel range planning, common snapshots, and shared-source admission remain separate work items.TMongoScanDesc. BE owns bounded client-pool leases and cursors with RAII and closes them on EOF, early termination, cancellation, and errors. BSON conversions validate physical tags and exact numeric/date ranges; compound columns roll back on failure. The BSON helpers, MongoFileReader lifecycle, and all TMongoScanDesc fields document their ownership, units, scope, and constraints. Metadata pagination uses the pinned Java driver core operation and its server-affine cursor while retaining authorized/name-only listing and namespace kinds. BE inherits complete URI read preferences, including tags and max staleness. BSON validation uses a depth-bounded iterator stack; unsupported validator structures fall back as a whole in auto mode and fail clearly in validator mode. Canonical Extended JSON and raw BSON provide document access. Reader profiles expose read/decode/filter time, rows, and BSON bytes.CREATE CATALOG mongo_native PROPERTIES ("type"="mongodb", "mongodb.uri"="mongodb://mongo.example:27017", "mongodb.databases"="sample"). See the connector README for explicit schemas, credentials, supported mappings, and fixture setup. ObjectId usesVARBINARY(12); useFROM_BINARYfor supported hex/string analysis because direct VARBINARY operators retain their existing engine restrictions. Declared VARBINARY length remains schema metadata; BSON decoding preserves full ObjectId, BinData, and document payloads.autochooses a namespace's explicit mapping, a fully representable ordinary-collection validator, or document mode; it never samples source rows. All mapped fields are nullable and are not declared keys. Mismatches fail by default; optional top-level null substitution retains atomic nested writes. Dates use UTC. Batch size and query timeout use session settings; connection/read defaults belong to the Catalog.Testing
All checks below ran locally:
build.sh --fe) and complete ASAN BE build (build.sh --be).run-fe-ut.sh.run-be-ut.sh: 142 passed across BSON SerDe/reader/real-driver error paths and existing decoded-value/name/VARBINARY coverage. The focused ObjectId/raw-BSON length tests failed before the fix and passed afterward. New coverage verifies bounded BSON validation memory, depth and element limits, and preservation of complete URI read preferences; the depth-boundary test was rerun after its final refactor.CursorNotFoundon the old implementation; the updated implementation completed 20 consecutive full listings. Matching URI read-preference tags read successfully; unmatched tags correctly timed out instead of silently routing elsewhere. Unit tests also verify preservation of URI max staleness.Notes for Reviewers
Start with
MongoSchema,MongoScanPlanProvider,mongodb_reader.cpp, and the BSON SerDe tests. The regression suite covers all three logical object kinds, schema refresh, historical validator violations, source permissions, exact/special BSON values, nested rollback, SQL analysis, batch boundaries, and repeated early cursor closure. The current local integration environment uses MongoDB 8.0.20; MongoDB 7, full replica-set/mongos topology qualification (including failover and live staleness behavior), TLS/authentication matrices, performance, and soak evidence remain in the tracking issue. The two-mongos check above is a focused metadata-cursor and tag-routing test.Release note
Add a native read-only MongoDB Catalog supporting explicit/validator/document schemas, native BSON decoding, and single-cursor reads of collections, standard views, and time-series measurements.
Check List (For Author)
AI Disclosure