Skip to content

[feature](mongodb) Add native catalog, reader and BSON SerDe - #70327

Merged
yiguolei merged 4 commits into
apache:masterfrom
Gabriel39:dev/mongodb-native-catalog
Nov 10, 2026
Merged

yiguolei merged 4 commits into
apache:masterfrom
Gabriel39:dev/mongodb-native-catalog

Conversation

@Gabriel39

@Gabriel39 Gabriel39 commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Background

The native MongoDB dependency preparation (#70249) provides the FE Java driver and BE C driver, but does not register a usable Catalog. Doris needs a native metadata and scan path that preserves BSON types and reads collection, view, and time-series namespaces directly.

What problem does this PR solve?

Issue Number: #70251
Related PR: #70249 (client dependencies)
PR Type: Feature

Problem Summary:

Implement D01, D03, and D04 of the native MongoDB Catalog roadmap, plus the minimal single-Split descriptor needed to connect them. Users can create a type=mongodb Catalog and query ordinary collections, standard views, and time-series measurements through a native C++ reader. FE pins explicit, validator, or document schemas; concrete DataTypeSerDe implementations convert BSON without losing int64 precision, ObjectId bytes, or binary subtype identity.

Details

  • Change map: 3,088 added production/build lines, 2,231 test/fixture lines, 350 generated regression-result lines, and 157 documentation lines (33 removed lines total). FE adds the Connector SPI provider, catalog-owned metadata client, schema resolution, stable handles, plugin packaging, and a single native scan range. BE adds the cursor/pool lifecycle and BSON view/context, with decoding in existing concrete SerDes. Tests include deterministic MongoDB fixtures, FE/BE unit tests, and a SQL regression suite with generated expected results.
  • Scope: read-only single-cursor scans of ordinary collections, standard views, and time-series collections. Views retain their own pipeline/collation and work with view-only find access; internal time-series buckets are hidden. All SQL filters, LIKE/REGEXP, joins, windows, aggregates, sorting, and LIMIT are evaluated in Doris. Predicate/projection pushdown, parallel range planning, common snapshots, and shared-source admission remain separate work items.
  • Design: FE sends stable namespace/configuration/schema values through TMongoScanDesc. BE owns bounded client-pool leases and cursors with RAII and closes them on EOF, early termination, cancellation, and errors. BSON conversions validate physical tags and exact numeric/date ranges; compound columns roll back on failure. The BSON helpers, MongoFileReader lifecycle, and all TMongoScanDesc fields document their ownership, units, scope, and constraints. Metadata pagination uses the pinned Java driver core operation and its server-affine cursor while retaining authorized/name-only listing and namespace kinds. BE inherits complete URI read preferences, including tags and max staleness. BSON validation uses a depth-bounded iterator stack; unsupported validator structures fall back as a whole in auto mode and fail clearly in validator mode. Canonical Extended JSON and raw BSON provide document access. Reader profiles expose read/decode/filter time, rows, and BSON bytes.
  • Usage: CREATE CATALOG mongo_native PROPERTIES ("type"="mongodb", "mongodb.uri"="mongodb://mongo.example:27017", "mongodb.databases"="sample"). See the connector README for explicit schemas, credentials, supported mappings, and fixture setup. ObjectId uses VARBINARY(12); use FROM_BINARY for supported hex/string analysis because direct VARBINARY operators retain their existing engine restrictions. Declared VARBINARY length remains schema metadata; BSON decoding preserves full ObjectId, BinData, and document payloads.
  • Default behavior: auto chooses a namespace's explicit mapping, a fully representable ordinary-collection validator, or document mode; it never samples source rows. All mapped fields are nullable and are not declared keys. Mismatches fail by default; optional top-level null substitution retains atomic nested writes. Dates use UTC. Batch size and query timeout use session settings; connection/read defaults belong to the Catalog.
  • Compatibility: the new Catalog uses the dependencies introduced by [chore](build) Add native MongoDB client dependencies #70249. Existing connectors and their serialization paths retain their behavior. Cross-version capability negotiation is outside this change.

Testing

All checks below ran locally:

  • Complete FE build (build.sh --fe) and complete ASAN BE build (build.sh --be).
  • FE connector unit tests: 31 passed, including schema resolution, metadata/driver cursor cleanup, handle serialization, session policies and native scan descriptors. Related FE core and legacy MongoDB connector tests also passed through run-fe-ut.sh.
  • BE unit tests through run-be-ut.sh: 142 passed across BSON SerDe/reader/real-driver error paths and existing decoded-value/name/VARBINARY coverage. The focused ObjectId/raw-BSON length tests failed before the fix and passed afterward. New coverage verifies bounded BSON validation memory, depth and element limits, and preservation of complete URI read preferences; the depth-boundary test was rerun after its final refactor.
  • Native MongoDB SQL regression against the deployed FE/BE and MongoDB 8.0.20: generated expected output, then independently compared it; final rerun 1 suite passed, 0 failed/fatal/skipped after recreating the source fixtures. Coverage includes explicit BSON null versus missing fields, integer/decimal bounds, float subnormal/nonfinite/signed-zero values, datetime/timestamp boundaries, binary bytes/subtypes, strings, nested values, mismatch policies, and document limits. New fixtures cover unsupported tuple/pattern/dependency validators, a wide BSON document, and metadata pagination beyond 128 namespaces.
  • License headers: the PR incremental scan passed with skywalking-eyes v0.8.0 after adding the ASF header to the MongoDB ServiceLoader descriptor.
  • FE Checkstyle: zero violations. Clang-format 16.0.6 on all 31 affected C++ source/header files, build hygiene, and the repository clang-tidy check passed. Static analysis used compilation commands from the completed BE/UT builds, expanded from unity commands for individual files.
  • Two-mongos integration with 140 collections reproduced CursorNotFound on the old implementation; the updated implementation completed 20 consecutive full listings. Matching URI read-preference tags read successfully; unmatched tags correctly timed out instead of silently routing elsewhere. Unit tests also verify preservation of URI max staleness.
  • Full diff/self-review and private-context audit completed.

Notes for Reviewers

Start with MongoSchema, MongoScanPlanProvider, mongodb_reader.cpp, and the BSON SerDe tests. The regression suite covers all three logical object kinds, schema refresh, historical validator violations, source permissions, exact/special BSON values, nested rollback, SQL analysis, batch boundaries, and repeated early cursor closure. The current local integration environment uses MongoDB 8.0.20; MongoDB 7, full replica-set/mongos topology qualification (including failover and live staleness behavior), TLS/authentication matrices, performance, and soak evidence remain in the tracking issue. The two-mongos check above is a focused metadata-cursor and tag-routing test.

Release note

Add a native read-only MongoDB Catalog supporting explicit/validator/document schemas, native BSON decoding, and single-cursor reads of collections, standard views, and time-series measurements.

Check List (For Author)

  • Behavior changed: Yes (see Details).
  • Does this need documentation: Yes; connector and fixture READMEs are included. Public user-documentation and release qualification remain tracked in [Feature][Tracking] Native MongoDB Catalog #70251.
  • Pre-submission status: Ready for review; local validation listed above, CI pending.

AI Disclosure

  • Tool: Codex (GPT-6).
  • Generated: implementation, tests, documentation, and this description.
  • Human review: pending maintainer review.

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

### Background

The native MongoDB connector registers its provider through Java ServiceLoader.
Service descriptors support hash comments and are checked by the license workflow.

### What problem does this PR solve?

Issue Number: apache#70251
Related PR: apache#70327 (introduced the service descriptor)
PR Type: Bugfix

Problem Summary:

The license workflow rejects the new provider descriptor because it has no ASF
license header. Add the same hash-comment header used by the other connectors.

### Details

- Frequency: every license check of the original descriptor.
- Root cause: the new META-INF/services resource contained only the provider name.
- Why existing tests missed it: ServiceLoader accepts the unlicensed descriptor;
  runtime discovery tests do not validate source licensing.
- Fix: add the ASF header as hash comments and document why that syntax is valid.
- Other paths and branches: the existing connector descriptors already carry headers.

### Testing

Skywalking-eyes v0.8.0 passed the PR incremental check with zero invalid files.
All 26 MongoDB connector unit tests passed, including ServiceLoader discovery.
FE Checkstyle passed with zero violations during Maven validate.

### Release note

None.

### Check List (For Author)

- Behavior changed: No.
- Does this need documentation: No.
- Pre-submission status: Ready; local license, unit and style checks pass.

### AI Disclosure

- Tool: Codex (GPT-6).
- Generated: the descriptor comment and commit message.
- Human review: pending maintainer review.
@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: You’ve hit your usage limit. Visit https://chatgpt.com/settings/usage to purchase more credits or try again at Nov 11th, 2026 6:35 AM.
Workflow run: https://github.com/apache/doris/actions/runs/42166583347

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

…versions

### Background

Native MongoDB scans decode BSON through Doris DataTypeSerDes. VARBINARY length
is metadata in existing readers, while the new BSON path added a length check.

### What problem does this PR solve?

Issue Number: apache#70251
Related PR: apache#70327 (introduces the native reader)
PR Type: Bugfix

Problem Summary:

BSON ObjectId, BinData and raw documents can be rejected solely because their
payload exceeds the declared VARBINARY length. Restore the existing metadata
semantics and preserve all bytes. Expand regression coverage for NULL, missing,
boundary and invalid values, and explain the BSON/reader/scan-descriptor contracts.

### Details

- **Change map:** 106 added production/comment lines, 169 test/fixture lines,
  60 generated regression-result lines and 2 documentation lines; 24 removed lines.
- **Frequency:** whenever a valid BSON binary payload exceeds a declared length.
- **Root cause:** the BSON-only SerDe length member enforced a restriction absent
  from other VARBINARY readers.
- **Why existing tests missed it:** the previous BSON test expected this rejection;
  it did not compare the existing VARBINARY length semantics.
- **Fix:** remove the length argument/check while retaining BSON tag/subtype checks,
  exact byte copying, and mismatch rollback. Document borrowed-value lifetimes,
  reader lifecycle, and every TMongoScanDesc field.
- **Other paths and branches:** existing VARBINARY readers retain their behavior;
  no wire field IDs, types or requiredness change.

### Testing

The two focused ObjectId/raw-BSON tests failed before the fix and passed after it.
All 137 selected BE tests passed through run-be-ut.sh. All 26 MongoDB FE connector
tests passed, with zero Checkstyle violations. The complete ASAN BE build passed.
The SQL regression generated expected output and independently compared it against
a recreated MongoDB 8.0.20 fixture: one suite passed, zero failed/fatal/skipped.
Coverage includes explicit NULL versus absent keys, numeric extrema/nonfinite/
subnormal values, decimal overflow and rounding, datetime boundaries and precision,
ObjectId/BinData bytes and subtype mismatches, strings, nested values and document
limits. Clang-format 16, clang-tidy and build hygiene passed.

### Release note

MongoDB binary decoding preserves full ObjectId, BinData and document bytes
regardless of the declared VARBINARY length.

### Check List (For Author)

- Behavior changed: Yes (see Details).
- Does this need documentation: Yes; connector README updated.
- Pre-submission status: Ready; local checks passed, CI pending.

### AI Disclosure

- Tool: Codex (GPT-6).
- Generated: code, comments, tests, fixtures, documentation and this message.
- Human review: pending maintainer review.
@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

### Background

The native MongoDB Catalog discovers logical namespaces through the Java driver
and reads BSON through the native driver and Doris SerDes. Auto schema mode must
retain every declared value through a complete typed schema or document fallback.

### What problem does this PR solve?

Issue Number: apache#70251
Related PR: apache#70327 (introduces the affected native paths)
PR Type: Bugfix

Problem Summary:

Wide BSON validation can allocate tens of MiB outside query memory accounting.
Metadata continuation can reach the wrong mongos, URI routing criteria are lost,
and valid unsupported validators can either hide fields or fail auto resolution.
Bound traversal scratch, delegate metadata cursor ownership to the pinned driver,
preserve full read preferences, and reject unrepresentable validators consistently.

### Details

- **Change map:** 131 added production/build lines, 298 test/fixture lines,
  51 generated regression-result lines and 17 documentation lines; 146 removed lines.
- **Frequency:** wide nested documents; paginated metadata across multiple mongos;
  URI tag/max-staleness settings; tuple, pattern and conditional-field validators.
- **Root cause:** a width-sized untracked worklist, independent raw cursor commands,
  mode-only read preferences and an incomplete typed validator grammar.
- **Why existing tests missed it:** standalone metadata fakes assumed that session
  identity implied server affinity; routing tests checked only mode names, schema
  tests missed valid tuple/pattern forms, and traversal had no allocation probe.
- **Fix:** use a fixed-depth iterator stack; retain driver-owned BatchCursor
  connection sources and cleanup; copy complete URI preferences in FE and BE;
  turn tuple and schema-bearing constructs into the existing whole-collection
  fallback in auto mode and explicit errors in validator mode. Preserve URI
  precedence, metadata primary reads, permissions and object kinds.
- **Other paths and branches:** recursively check root/nested/array-element schemas;
  change no existing connector path, Thrift field layout or capability negotiation.

### Testing

Focused tests reproduced the original schema/routing failures and a 48 MiB BSON
validation allocation. The old live implementation reproduced CursorNotFound
while listing 140 collections across two mongos routers. All 31 FE connector tests
and 142 selected BE tests passed; the refactored depth test passed a focused rerun.
Complete FE and ASAN BE builds passed, with zero FE Checkstyle violations.
Clang-format 16, clang-tidy and header hygiene passed for affected files.
The MongoDB SQL suite generated expected output and independently compared it:
one suite passed, zero failed/fatal/skipped, including fallback values, view-only
permissions, a 900,000-container scalar scan and metadata batch boundaries.
The fixed two-router implementation completed 20 full 140-name listings. Matching
URI tags returned the scalar value; unmatched tags timed out instead of reading
an ineligible member. Maximum staleness preservation is covered by both driver
unit tests; this is not a failover or soak qualification.

### Release note

Preserve MongoDB routing criteria and metadata cursor affinity, fall back completely
for unsupported validator schemas, and bound wide BSON validation scratch memory.

### Check List (For Author)

- Behavior changed: Yes (see Details).
- Does this need documentation: Yes; connector and fixture READMEs updated.
- Pre-submission status: Ready; local checks passed, CI pending.

### AI Disclosure

- Tool: Codex (GPT-6).
- Generated: implementation, comments, tests, fixtures, documentation and this message.
- Human review: pending maintainer review.
@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@Gabriel39

Copy link
Copy Markdown
Contributor Author

Local pipeline review — ✅ PASS

schema: doris-repo-review/v1
status: PASS
pr: apache/doris#70327
commit: 2134c7569c7ae93fa981fc1695ccfed680726235
base: 59c49656240451ed1caab1e69e69bbe0be8c61aa
reviewed_at: 2026-11-06T15:35+08:00
reviewer: Gabriel39
model: gpt-6-astra
effort: xhigh
findings: {blocker: 0, major: 0, minor: 0, nit: 0}
rounds: 1
converged: true

Notes for maintainers

  • Static source and pinned dependency-contract review; builds, tests, live topology and performance/soak validation were not run as part of this review.

Reviewed locally with the doris-repo-review pipeline. Repository policy may accept this receipt for the matching commit; it is not a human Apache approval.

@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Review recovery stopped: shared time budget or helper timeout exhausted
Workflow run: https://github.com/apache/doris/actions/runs/42179532956

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 100.00% (3/3) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28812 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2134c7569c7ae93fa981fc1695ccfed680726235, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17605	5147	5037	5037
q2	2131	314	257	257
q3	10256	1383	825	825
q4	4683	465	338	338
q5	7437	801	533	533
q6	184	189	159	159
q7	758	790	595	595
q8	9245	1373	1109	1109
q9	5760	4217	4214	4214
q10	6813	1290	1024	1024
q11	423	241	227	227
q12	642	402	283	283
q13	18110	3631	3055	3055
q14	277	270	240	240
q15	q16	727	717	664	664
q17	1212	756	581	581
q18	7041	6195	6137	6137
q19	1096	983	590	590
q20	386	339	230	230
q21	5567	2503	2446	2446
q22	378	315	268	268
Total cold run time: 100731 ms
Total hot run time: 28812 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	6186	5894	6050	5894
q2	718	628	585	585
q3	5662	5862	5276	5276
q4	2074	2140	1553	1553
q5	5795	5885	5632	5632
q6	234	187	144	144
q7	2261	2055	1814	1814
q8	3007	2760	2686	2686
q9	7992	7635	7543	7543
q10	3614	3566	3107	3107
q11	584	405	376	376
q12	670	658	502	502
q13	3275	3603	3015	3015
q14	288	292	262	262
q15	q16	658	679	617	617
q17	7675	7105	6878	6878
q18	12923	12214	12902	12214
q19	900	785	769	769
q20	2152	2128	1886	1886
q21	5556	4614	4688	4614
q22	494	446	415	415
Total cold run time: 72718 ms
Total hot run time: 65782 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 154360 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2134c7569c7ae93fa981fc1695ccfed680726235, data reload: false

query5	4321	529	392	392
query6	431	218	217	217
query7	4820	472	236	236
query8	328	172	155	155
query9	8791	3211	3220	3211
query10	456	339	272	272
query11	6078	3900	3580	3580
query12	149	95	86	86
query13	1252	459	336	336
query14	6161	3959	3661	3661
query14_1	3455	3405	3400	3400
query15	206	214	181	181
query16	952	422	404	404
query17	866	665	548	548
query18	2420	433	313	313
query19	200	174	140	140
query20	87	86	89	86
query21	211	132	111	111
query22	12996	12919	12770	12770
query23	13645	12797	12514	12514
query23_1	12661	12610	12484	12484
query24	7361	847	494	494
query24_1	493	487	496	487
query25	548	460	373	373
query26	1253	280	152	152
query27	2740	464	281	281
query28	4586	2062	2090	2062
query29	1559	608	438	438
query30	295	216	182	182
query31	1018	939	814	814
query32	143	90	96	90
query33	518	313	236	236
query34	917	903	532	532
query35	748	750	652	652
query36	811	829	723	723
query37	129	100	80	80
query38	1762	1714	1687	1687
query39	792	802	747	747
query39_1	719	716	738	716
query40	219	114	111	111
query41	67	63	62	62
query42	103	85	83	83
query43	359	374	309	309
query44	1387	714	712	712
query45	184	179	167	167
query46	875	1010	564	564
query47	2999	2893	2812	2812
query48	298	329	237	237
query49	606	411	298	298
query50	722	291	227	227
query51	10533	10495	10425	10425
query52	83	80	71	71
query53	195	217	158	158
query54	253	206	209	206
query55	78	73	68	68
query56	239	227	258	227
query57	1657	1665	1538	1538
query58	293	272	258	258
query59	2139	2148	2014	2014
query60	275	236	209	209
query61	147	146	145	145
query62	397	339	289	289
query63	197	160	164	160
query64	2755	901	741	741
query65	3307	3268	3266	3266
query66	1752	461	307	307
query67	22626	22638	22510	22510
query68	3318	1038	629	629
query69	382	295	258	258
query70	867	767	775	767
query71	305	243	212	212
query72	2747	2597	2310	2310
query73	560	568	306	306
query74	4619	4524	4370	4370
query75	2434	2325	1940	1940
query76	2349	1144	619	619
query77	373	404	283	283
query78	9836	9759	9247	9247
query79	990	958	551	551
query80	569	463	389	389
query81	523	323	274	274
query82	587	148	115	115
query83	312	227	199	199
query84	309	131	109	109
query85	939	427	370	370
query86	264	241	233	233
query87	1989	1945	1793	1793
query88	3786	2776	2774	2774
query89	334	289	247	247
query90	2012	178	167	167
query91	162	150	124	124
query92	102	94	87	87
query93	1073	1028	595	595
query94	578	314	250	250
query95	586	316	312	312
query96	705	510	244	244
query97	2431	2449	2380	2380
query98	163	160	149	149
query99	762	756	666	666
Total cold run time: 238305 ms
Total hot run time: 154360 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.09 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2134c7569c7ae93fa981fc1695ccfed680726235, data reload: false

query1	0.01	0.01	0.00
query2	0.10	0.04	0.05
query3	0.26	0.10	0.10
query4	1.60	0.10	0.10
query5	0.23	0.24	0.24
query6	1.18	0.68	0.69
query7	0.04	0.01	0.01
query8	0.06	0.04	0.04
query9	0.39	0.34	0.35
query10	0.57	0.57	0.57
query11	0.19	0.14	0.14
query12	0.18	0.15	0.14
query13	0.49	0.49	0.48
query14	1.02	1.01	1.00
query15	0.61	0.60	0.61
query16	0.35	0.33	0.33
query17	1.13	1.12	1.10
query18	0.24	0.18	0.18
query19	2.12	2.03	2.00
query20	0.02	0.01	0.01
query21	15.42	0.20	0.14
query22	4.84	0.05	0.06
query23	16.28	0.31	0.13
query24	1.06	0.24	0.16
query25	0.08	0.05	0.05
query26	0.77	0.22	0.17
query27	0.02	0.03	0.03
query28	3.59	0.91	0.37
query29	12.48	4.17	3.23
query30	0.30	0.12	0.10
query31	2.77	0.56	0.33
query32	3.21	0.59	0.52
query33	3.19	3.20	3.29
query34	15.31	3.96	3.35
query35	3.36	3.36	3.35
query36	0.54	0.46	0.45
query37	0.09	0.07	0.06
query38	0.05	0.04	0.04
query39	0.04	0.04	0.03
query40	0.18	0.15	0.14
query41	0.09	0.04	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 94.54 s
Total hot run time: 24.09 s

@yiguolei
yiguolei merged commit 13f4d29 into apache:master Nov 10, 2026
43 of 44 checks passed
OIiveirra added a commit to OIiveirra/doris that referenced this pull request Nov 10, 2026
### Background

Spatial and VARBINARY SerDes share column-typed binary operations. Upstream master adds a BSON decoder specifically for ColumnVarbinary, which conflicts with that shared-base extraction.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#69032 (spatial feature), apache#70327 (upstream BSON decoder)
PR Type: Build/CI/Chore

Problem Summary: Merge upstream/master at 08231e3 into the spatial branch. Retain the BSON decoder on a concrete VARBINARY SerDe derived from the existing shared base, while spatial SerDes retain the default BSON rejection. Add a focused regression for ObjectId decoding and rejection by nullable and nonnullable Geometry/Geography.

### Details

- **Change map:** Manually reconcile two SerDe files and add 33 lines in the existing VARBINARY test file; the remaining changes come from the upstream merge. No new PR paths or spatial capabilities are added.
- **Affected:** BE binary/spatial SerDe inheritance and BSON dispatch. No CI or runtime configuration changes.
- **Developer and release impact:** Preserve both upstream VARBINARY BSON support and the spatial type boundary; ordinary key plus spatial value support is unchanged.
- **License and third-party:** No PR-specific dependency, version or license edits. Local dependency setup is excluded from this commit.
- **Rollback:** Revert this merge if necessary; no external state or configuration migration.

### Testing

Clang-format 16, build hygiene and changed-line clang-tidy passed. The Core unity compilation containing the changed SerDe passed in the standard clang21/ASAN BE UT build, and clang-tidy analyzed both the production and test translation units with real build commands. The standard selected BE UT run did not reach test execution: local MongoDB dependencies were installed through the repository script, then an outdated Lance dependency blocked compilation. Rebuilding that pinned dependency was blocked by a reproducible local Rust subprocess-output capture failure. No runtime unit-test pass or full CI pass is claimed.

### Release note

None

### Check List (For Author)

- Behavior changed: No spatial functionality changes; preserve the upstream BSON behavior for VARBINARY only.
- Does this need documentation: No
- Pre-submission status: Partially verified: full BE UT execution is blocked by local Lance/Rust dependency setup.

### AI Disclosure

- Tool: Codex (model identifier unavailable in this task)
- Generated: Conflict resolution, regression, verification and commit description
- Human review: Pending author review
xylaaaaa added a commit to xylaaaaa/doris that referenced this pull request Nov 12, 2026
Master added `16: optional TMongoScanDesc mongodb_params` to
TTableFormatFileDesc (apache#70327), so `deltalake_params` moves to field 17.
Field 17 is unused on master, branch-4.2, branch-4.1, branch-4.0 and
branch-3.1. Generated code accesses the field by name, so no Java or C++
code changes. build.sh keeps master's mongodb plugin entries and adds
deltalake to both connector lists.
Gabriel39 added a commit that referenced this pull request Nov 16, 2026
…oDB M0 (#70456)

### Background

The native MongoDB catalog uses the shared external-table discovery
cache and a native single-cursor reader. A cold table lookup first
enumerates names from the source. The MongoDB roadmap tracks minimal
collection, view and time-series reads in M0.

### What problem does this PR solve?

Issue Number: #70251
Related PR: #70327 (native reader/catalog implementation), #70249
(dependencies)
PR Type: Bugfix

Problem Summary:

An incorrect MongoDB password or a TLS hostname failure currently
reaches SQL as a missing-table error because the shared existence check
discards the connector exception. Preserve the typed source error and
its retry behavior. Also close M0 acceptance gaps: the 2,081-row fixture
fits inside the current default batch size, so existing batch tests do
not necessarily issue getMore.

### Details

- **Change map:** 4 production additions; 352 test additions/5
deletions; 21 generated result lines; 36 documentation additions/1
deletion. Most of the diff is an opt-in real-driver fault runner, SQL
acceptance coverage, and its instructions.
- **Frequency:** Deterministic on a cold lookup when connector metadata
enumeration fails, reproduced with invalid credentials and TLS hostname
validation failure.
- **Root cause:** ExternalDatabase treats runtime failures during the
names/existence lookup as evidence that the table is absent.
- **Why existing tests missed it:** Native SQL coverage lacked
invalid-credential recovery. The multi-batch fixture has 2,081 rows
while the default batch size is 8,160; getMore and early cursor closure
were insufficiently exercised.
- **Fix:** Rethrow DorisConnectorException from the shared lookup.
Explicit batch-size hints force the existing boundary tests across
source batches. Add credential correction, unsupported-mode rejection,
view residual semantics, internal-table materialization/differential
checks, and an environment-configured fault runner that verifies source
cursors and successful reads after errors, timeouts and cancellation.
- **Other paths and branches:** The exception fix applies to Connector
SPI catalogs through the shared discovery path. No connector-specific
branch, configuration default, IDL, or storage format is added. This PR
targets master.
- **Behavior change:** Authentication/transport failures remain source
failures; a successful lookup that finds no matching table retains
missing-table behavior.

### Testing

Completed locally on Linux x86_64 with MongoDB 8.0.20, Java driver
5.14.0 and C driver 2.5.8:

- Complete clean FE and ASAN BE build through build.sh --clean --be
--fe.
- FE Checkstyle: zero violations, including a separate fe-core check.
- New discovery unit test failed before the fix because no exception was
thrown. After the fix, all 51 ExternalCatalogTest tests passed; all 31
native MongoDB connector tests also passed.
- Selected BSON SerDe/native reader unit tests through run-be-ut.sh: 44
passed.
- Both native SQL suites: generated the new expected output with the
normal runner, then independently compared both suites: 2 passed, 0
failed/fatal/skipped. The internal-table differential is zero across
2,081 fixture rows.
- Final opt-in lifecycle runner passed EOF/LIMIT/decode-error cleanup,
injected getMore failure, socket and query deadlines, cancellation after
a blocked getMore was observed, concurrent reads, and subsequent
recovery. Source open cursors returned to zero after each scenario. The
query deadline case uses a socket timeout longer than the injected
delay, so a socket timeout cannot mask a missing query deadline.
- Trusted-CA TLS reads passed. Untrusted certificates and hostname
mismatches were rejected with source errors. Snappy, zlib and zstd
multi-batch reads passed; MongoDB client metadata confirmed negotiation
for both drivers. SCRAM-SHA-256 and URI-escaped SCRAM-SHA-1 credentials
passed.
- Python Ruff checks and formatting passed. Full diff/self-review and
private-context scan completed. Generated output retains the regression
runner's canonical blank-line separators.

### Notes for Reviewers

The fault runner requires an otherwise idle, disposable,
test-command-enabled MongoDB fixture and explicit environment-injected
connection settings; ordinary external regression does not enable
failpoints. Its invocation is documented next to the fixture. The SQL
suites remain gated by enableMongoDBTest.

This is M0 single-cursor acceptance with matching-version FE/BE in the
tested environment. Software upgrade/downgrade qualification, SRV
validation, multi-architecture validation and FE/BE capability
negotiation are excluded from the agreed scope. MongoDB 7, broader
source-topology/failover qualification, performance and release soak
remain at their later roadmap gates in #70251.

### Release note

Preserve connector metadata authentication and transport errors during
cold external-table lookups instead of reporting a missing table.

### Check List (For Author)

- Behavior changed: Yes (see Details).
- Does this need documentation: No user-facing configuration change;
fixture and fault-test instructions are included.
- Pre-submission status: Ready; local validation above completed, PR CI
pending.

### AI Disclosure

- Tool: Codex (GPT-6).
- Generated: fix, tests, fixture instructions, and this description.
- Human review: Pending maintainer review.
Gabriel39 added a commit that referenced this pull request Nov 16, 2026
### Background

The native MongoDB catalog sends read preference and read concern to the
BE in each scan descriptor. Today those values come only from catalog
properties or the connection URI, while metadata discovery uses its own
primary/default policy.

### What problem does this PR solve?

Issue Number: #70251
Related PR: #70327 (native reader and catalog read policies)
PR Type: Feature

Problem Summary:

Users cannot choose MongoDB routing and visibility for an individual
session or query without changing the shared catalog. Add
`mongodb_read_preference` and `mongodb_read_concern`, supporting `SET`
and `SET_VAR`, and pin their effective values in each scan descriptor.
Both default to `inherit`, preserving existing catalog/URI behavior.
Most of the diff adds tests and generated regression expectations.

### Details

- **Change map:** 70 production lines changed, 281 test-source lines
changed, 62 generated expectation lines, and 38 documentation lines
changed.
- **Scope:** Data-read policies for native MongoDB scans. Preference
supports `inherit`, `primary`, `primaryPreferred`, `secondary`,
`secondaryPreferred`, and `nearest`; concern supports `inherit`,
`local`, `majority`, and `available`. Values are case-insensitive.
Snapshot and linearizable reads remain unsupported.
- **Design:** Register validated, forwarded session variables and
resolve them through the existing connector session-property snapshot.
Reuse the existing scan descriptor and BE collection policy application.
Session overrides do not mutate catalog configuration, metadata clients,
or transport pool identity. URI tags, maximum staleness, and hedge
options remain effective; an incompatible primary override fails during
planning without exposing the URI.
- **Usage:** `SET mongodb_read_preference='secondaryPreferred'; SET
mongodb_read_concern='local';` or `SELECT /*+
SET_VAR(mongodb_read_preference='primary',
mongodb_read_concern='majority') */ ...`. Set either variable to
`inherit` to use its catalog/URI policy again.
- **Default behavior:** `inherit` independently resolves each policy
from URI options first, then catalog defaults. Queries without overrides
retain their existing behavior.
- **Compatibility:** No connector SPI, Thrift, BE code, or
storage-format changes. Metadata discovery and collection-identity
verification retain their existing policies. Read concern does not
establish a snapshot shared by scans.

### Testing

All checks below ran locally against MongoDB 8.0.20, with the new FE and
the existing ASAN native BE:

- Red/green: the new scan-plan tests first failed because overrides were
ignored, and the new session tests first failed because the variables
were unregistered; both passed after implementation.
- `run-fe-ut.sh --run` with `SessionVariableMongoReadPolicyTest`,
`VariableMgrExportTest`, `MongoScanPlanTest`, and
`MongoCatalogPropertiesTest`: passed. The three new session tests cover
defaults, supported/unsupported values, connector snapshots, isolation,
and forwarding.
- `mvn -f fe/pom.xml -pl :fe-connector-mongodb -am test`: 35 connector
tests passed, including URI precedence, independent policy inheritance,
incompatible routing criteria, and stable transport configuration
identity.
- `build.sh --fe -j12`: passed. FE core and MongoDB connector
Checkstyle: zero violations.
- Generated `test_mongodb_read_policy.out` using the regression runner,
checked its results against the synthetic fixture, then independently
reran `test_mongodb_native` and `test_mongodb_read_policy`: 2 suites
passed, 0 failed/fatal/skipped. Covers SET/SET_VAR, all supported
modes/levels, invalid values, failed-hint restoration, URI policy
inheritance, and recovery.
- Additional replica-set integration: source profiler verified
primary/secondary selection and the actual read concern for 8 sequential
reads and 32 concurrent reads across 4 isolated sessions. Also verified
URI tags with a mode override and policy restoration after a failed
hint. This was an additional local acceptance check; the committed SQL
suites use the existing standalone fixture.

### Release note

Add `mongodb_read_preference` and `mongodb_read_concern` session
variables for native MongoDB queries, with statement-level SET_VAR
support. Both default to `inherit` to retain the catalog's existing read
policy.

### Check List (For Author)

- Behavior changed: Yes (see Details; explicit overrides now affect
native MongoDB data reads).
- Does this need documentation: Yes; the connector README is updated in
this PR.
- Pre-submission status: Ready. Local checks passed; CI is pending.

### AI Disclosure

- Tool: Codex (GPT-6).
- Generated: Implementation, tests, connector documentation, and this
description.
- Human review: Maintainer review pending; the agent completed a full
diff self-review and local validation.
Gabriel39 added a commit to Gabriel39/incubator-doris that referenced this pull request Nov 16, 2026
The native MongoDB Catalog on master already provides single-cursor reads and first-release BSON SerDes. SQL filters and source projection have remained local, and the remaining M1 type and pushdown contracts need end-to-end qualification.

Issue Number: apache#70251 (D04b and D05)
Related PR: apache#70327 (native read path), apache#70456 (M0 acceptance)
PR Type: Feature

Problem Summary:

Complete the D04b qualification coverage and connect conservative predicate/LIKE candidates and column pruning to the native MongoDB reader. Every SQL predicate remains a Doris residual; the default strict type policy, strict SQL casts and standard views keep filtering local. This PR does not complete the separate D06 or later snapshot/parallel-scan work.

- **Change map:** 449 added production lines, 1,187 test/fixture/generated-result lines, and 47 documentation lines across 19 files. The production changes comprise a connector-local translator and scan planning, three session controls, two optional descriptor fields, and native find options/validation. Most additions qualify types and compare execution modes.
- **Scope:** D04b covers Decimal128 extremes, exact integer/floating-point behavior, UTC dates and scales, binary ownership/subtypes, missing/null, recursive arrays/structs, rollback and existing document limits. D05 covers typed scalar-root comparisons, positive IN/BETWEEN, IS NOT NULL, safe AND/OR, necessary-literal LIKE candidates, root projection and time-series Date ranges.
- **Design:** Candidates are classified conservatively as SUPERSET; unsupported expressions remain local, and the EXACT category is reserved for proven equivalence. No conjunct is consumed through either SPI path. CASTs are rejected before the neutral converter can erase them. The translator reconciles the schema/expression SPI date-scale and decimal-width representations, including FE's DECIMAL128 name. Native BSON is bounded and validated before client acquisition; views cannot receive a predicate/collation override. Complete root projection retains residual columns and scalar-ancestor errors.
- **Usage:** Use `SET` or `SET_VAR` with `enable_mongodb_predicate_pushdown`, `enable_mongodb_like_pushdown` and `enable_mongodb_projection_pushdown`. Predicate candidates require the explicit catalog policy `mongodb.type_mismatch=null`.
- **Default behavior:** All three switches default to true. The default `type_mismatch=error` policy preserves local filtering; `enable_strict_cast=true` and views also preserve local filtering. Source projection is eligible independently. DOUBLE/NaN comparisons, NOT/!=/NOT IN, mismatch-sensitive IS NULL, functions, casts, nested predicate paths and existing direct VARBINARY operator restrictions stay local. LIKE keeps the complete SQL residual. LIMIT, aggregation, sort, joins, snapshots and parallel scans are unchanged.
- **Compatibility:** The connector SPI and stored schemas are unchanged. Descriptor fields 23/24 are optional; absent fields preserve the prior full scan. The touched descriptor is absent from the four supported release branches, so no field id is reused. Qualification uses matching FE/BE versions.

All testing below ran locally against MongoDB 8.0.20 with authentication enabled and an ASAN BE:

- 59 MongoDB connector FE tests, 25 connector-session/forwarding tests and 55 BSON/native-reader BE tests passed.
- All three MongoDB SQL suites passed: 3 suites, 0 failed, 0 fatal, 0 skipped. This includes the existing native and M0 acceptance suites plus the new M1 suite.
- The generated M1 output contains 38 identical sorted result pairs with predicate/LIKE/projection disabled and enabled. It covers precision, NULL/arrays, casts, collations, residual LIMIT, aggregates/joins, views and time-series reads. The completed Profile check reports `MongoRowsRead` of 10 with candidates disabled and 1 with them enabled.
- Source profiling independently confirmed actual BSON int64, Decimal128 and Date filters, projection, simple collection collation, and view reads without a collation override.
- FE and ASAN BE builds passed; the final MongoDB plugin was subsequently packaged/tested and deployed from its final source. Full FE Checkstyle and the final affected-core-file Checkstyle run passed with zero violations. All affected C/C++ files pass clang-format 16, build hygiene and clang-tidy.
- Red/green checks reproduced missing candidate planning, acceptance of malformed BSON descriptors, missing strict-cast gating, command-budget gaps and schema/expression type representation gaps before the corresponding implementations.

Native MongoDB scans can prune source fields and apply conservative WHERE/LIKE candidates under the explicit null-on-type-mismatch policy. SQL residuals remain active, and default strict scans and views preserve local filtering. New session switches allow statement-level comparisons and disabling these optimizations.

- Behavior changed: Yes (see Details).
- Does this need documentation: Yes; the connector and fixture READMEs are updated here.
- Pre-submission status: Ready. Local UT, authenticated SQL regression, builds and style/static checks passed; CI is pending.

- Tool: Codex (GPT-6).
- Generated: Implementation, tests, fixture, documentation and this description.
- Human review: Pending maintainer review.
morningman added a commit to morningman/doris that referenced this pull request Nov 16, 2026
…udit text

### Background

SHOW CREATE CATALOG, SHOW CATALOG, the `catalogs()` table function and the audit text of CREATE CATALOG replace a property value with `*XXX` when `DatasourcePrintableMap.isSensitiveKey` matches its key. Until now the keys came from a list in fe-core and from the keys filesystem plugins report at startup. A connector plugin marks its secret properties `sensitive = true`, but that mark only masks the connector's own log lines. A connector tells the engine about its security-relevant properties through `ConnectorProvider#securitySurface()`.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#69588 (added the connector security surface this extends), apache#45164 (introduced the unmasked ALTER CATALOG audit text), apache#27115 (masked it for the statement apache#45164 replaced), apache#59502 (added `iceberg.jdbc.password`), apache#54523 (added the Paimon REST DLF keys), apache#65730 (branch-4.1's Lance catalog, whose REST secrets the Lance port will declare on this surface), apache#70366 (fixed the shared Unicode SQL rewrite used by this PR's ALTER masking), apache#70327 (added MongoDB credentials to the engine masking floor)
PR Type: Bugfix

Problem Summary:

Catalog secrets that only a connector knows are on no masking list, so SHOW CREATE CATALOG and the other renderings print them in clear: `iceberg.jdbc.password`, `paimon.jdbc.password`, the Paimon REST DLF key pair, `mc.access_key`, and credentials passed through under `trino.` or `adbc.`. Separately, `ALTER CATALOG ... SET PROPERTIES` is written to the audit log unmasked, whatever the key. Each connector now declares its secrets on its security surface, taken from its `sensitive` marks, and the surface can also declare a namespace of secret keys for options passed through unread; Fluss's lake options were hard-coded in fe-core for that reason. The engine masks the union, with fe-core's own list kept as the floor. ALTER CATALOG is masked like CREATE CATALOG.

### Details

- **Change map:** 45 files, +1205 −81 over four commits: production code +384 −51, tests +807 −26, the connector README +13 −3, the plugin API version in `fe/fe-connector/pom.xml` (31.0 to 32.0). The first commit adds secret keys to the surface (reviewed alone before). The second does the rest, the third keeps a surface that fails in the connector's own `ServiceLoader` from stopping FE, and the fourth aligns the newly added MongoDB provider and tests ALTER masking with supplementary characters:
  1. Key namespaces on the SPI: `withSensitiveKeyPrefix` and `SensitiveKeyPrefix.covers`, which holds the matching rule; a collection form of `withSensitiveKeys` (`ConnectorSecuritySurface`).
  2. Engine: `isSensitiveKey` matches registered namespaces, the Fluss rule becomes one of them; `ConnectorPluginManager` registers them and reads every surface before it publishes a provider, skipping a provider whose surface throws.
  3. Connectors: every provider declares its secrets; `MetaStoreProviders.sensitivePropertyKeys()` gives the union of a plugin's metastore backends; `IcebergJdbcMetaStoreProvider` reports its keys like its siblings; the keytab paths lose their `sensitive` mark. MongoDB also declares its two annotated secrets, preserving its SSRF endpoint and the engine fallback.
  4. ALTER CATALOG audit masking (+16).
- **Frequency:** Always. Create or alter a catalog that sets one of the keys above, then run SHOW CREATE CATALOG, SHOW CATALOG `<name>` or `select * from catalogs()`, or read the audit log after an ALTER CATALOG ... SET PROPERTIES.
- **Root cause:** `isSensitiveKey` consulted only fe-core's list and the filesystem providers' keys. A connector's `sensitive` marks fed only `toMaskedString`; `MetaStoreProvider.sensitivePropertyKeys()` existed for this, but nothing called it, and the Iceberg JDBC backend did not implement it. For ALTER CATALOG, `AlterCatalogPropertiesCommand` does not implement `NeedAuditEncryption` and `LogicalPlanBuilderForEncryption` has no `visitAlterCatalogProperties`, so `AuditLogHelper` records the original text; the statement it replaced in apache#45164 was masked by apache#27115.
- **Why existing tests missed it:** Unit tests: `DatasourcePrintableMapTest` checks fe-core's own keys, the provider surface tests checked only the SSRF and driver_url keys, and `EncryptSQLTest` and `RepositoryAuditEncryptionTest` cover CREATE CATALOG, repositories and resources but not ALTER CATALOG. Regression: `audit/test_external_catalog_credential_masking` creates only Iceberg REST catalogs, whose keys are on fe-core's list, and never alters one.
- **Fix:** The engine must not name a connector's properties, so the connectors declare their secrets, from the same `sensitive` marks that mask their logs; jdbc and es declare both spellings their legacy prefix allows, and trino and adbc declare the namespace they pass through, a key there being a secret when its rest names `key`, `secret`, `password`, `token` or `credential` (also `authorization` for ADBC). fe-core keeps the connector keys it lists, so a catalog whose plugin is not loaded masks what it masked before. A provider whose `securitySurface()` throws at startup is skipped, as a filesystem provider is, instead of stopping FE; that includes the `ServiceConfigurationError` a `ServiceLoader` throws when a descriptor names a backend it cannot load, which is how the Hive, Iceberg and Paimon surfaces fail, since they read their metastore backends' keys through one.
- **Other paths and branches:** Fixed here: every rendering above and FE's statement logging, which share `isSensitiveKey` and `NeedAuditEncryption`. Not covered, since a key name cannot express them safely: raw Iceberg SDK keys copied through (`token`, `credential`), Paimon options re-keyed from `paimon.*`, Hadoop `fs.*` keys, and secrets inside otherwise unmarked URLs. The ALTER CATALOG gap is on branch-4.0, 4.1 and 4.2 (from apache#45164); branch-3.1 still uses the masked statement. branch-4.1 and 4.2 also print the connector keys above in clear; without the connector SPI their fix differs and is tracked separately.
- **Behavior change:** The namespaces also mask non-secrets whose names carry those words, such as `trino.hive.s3.aws-access-key` or a keystore path. Keytab paths, which no release masks, are no longer masked in connector log lines: marked `sensitive`, `hadoop.kerberos.keytab` would be masked in every storage vault and resource.

### Testing

The original local runs (macOS) used master `f5c993f2c94` plus this change:

- Regression `audit/test_external_catalog_credential_masking`, extended with an Iceberg JDBC catalog: SHOW CREATE CATALOG masks `iceberg.jdbc.password`, and the audit texts of its CREATE CATALOG and of an ALTER CATALOG that sets the password and `s3.secret_key` hold only `*XXX`. Against the local cluster's FE without this change it fails at the first new check, SHOW CREATE CATALOG printing the password; with a FE built from this change it passes.
- fe-connector-spi: all 160 tests, with the refreshed surface baseline (+4 lines); the metastore modules: all tests; ten connector modules: 29 classes, 511 tests, including each connector's surface test, which checks real key names.
- fe-core: 26 classes, 341 tests, including `ConnectorPluginManagerTest`, `DatasourcePrintableMapTest`, `EncryptSQLTest` and the audit and plugin gate tests; 14 of them again after the last change to registration, and 7 classes, 76 tests, after the third commit. Its two new `ConnectorPluginManagerTest` cases, a plugin directory whose `lib/` holds a jar with a service descriptor naming a missing metastore-style backend (the plugin next to it must still load) and the same descriptor on the classpath, fail without the third commit with that `ServiceConfigurationError` escaping `loadPlugins` and `loadBuiltins`, and pass with it.
- Twenty-two mutations, each caught by its test: for example, reverting either half of the ALTER CATALOG fix (`EncryptSQLTest` sees the statement in clear), matching a marker anywhere in the key, an empty metastore union, re-marking a keytab, or dropping `ServiceConfigurationError` from either startup catch (each fails only its own new case).

Rebased onto master `93c16ee2ca1` on 2026-11-16. Post-rebase local verification:

- `bash run-fe-ut.sh --run 'org.apache.doris.connector.spi.*Test,ConnectorPluginManagerTest,DatasourcePrintableMapTest,EncryptSQLTest'`, with `CUSTOM_MVN` appending `-Dcheckstyle.skip=false`: BUILD SUCCESS across all 90 reactor modules; all 141 selected tests passed across 22 classes, including the inherited supplementary-character audit cases. Checkstyle reported no violations.
- `git diff --check upstream-apache/master...HEAD`: passed. `git range-diff` confirms the replayed patches are unchanged apart from adapting the connector API version and its test to 32.0.
- Regression tests and a complete BE/FE build were not rerun locally after this rebase; `run buildall` is requested in CI.

Follow-up local verification on the rebased branch plus the MongoDB declaration and audit tests (2026-11-16):

- `run-fe-ut.sh --run 'org.apache.doris.connector.spi.*Test,org.apache.doris.connector.mongodb.*Test,ConnectorPluginManagerTest,DatasourcePrintableMapTest,EncryptSQLTest'`, with Checkstyle enabled: all 176 selected tests in 28 classes passed (SPI 93, MongoDB 35, fe-core 48), with no skipped tests; the 90-module reactor completed successfully.
- The strengthened `MongoProviderTest` fails without the provider declaration (expected the two MongoDB secret keys, received an empty set), and passes with it. A separate targeted revert confirms the assertion catches the missing declaration.
- The new ALTER CATALOG assertions cover supplementary characters within a secret, in a preceding SQL comment, and in a retained non-secret value, with two masked properties. `EncryptSQLTest#testEncryption` passes on the candidate and fails at the new ALTER assertion when only `BaseViewInfo.rewriteSql` is restored to its pre-apache#70366 implementation: the audit text has an extra trailing fragment.
- Full cluster regression and a standalone FE/BE build were not rerun for this follow-up.

### Notes for Reviewers

- `SensitiveKeyPrefix.covers` lives on the SPI so that a connector's test can check its declaration against real key names and the engine applies the same rule; the Fluss floor in fe-core uses it too.
- The entries both fe-core and a connector declare are deliberate: the floor is frozen, as the comment in `DatasourcePrintableMap` says. MongoDB was already masked by apache#70327; its added declaration makes the provider follow the same contract as the other connectors.
- The connector plugin API advances from 31.0 to 32.0 for the added security-surface methods. Third-party connector plugins built against 31.x must be rebuilt; the in-tree plugins are built with FE. Hudi declares nothing: it owns no catalog properties.
- The base includes apache#70366, so ALTER CATALOG masking uses the shared rewrite that converts parser code-point spans to Java string offsets, including supplementary characters.

### Release note

SHOW CREATE CATALOG, SHOW CATALOG, the `catalogs()` table function and the audit log now mask `iceberg.jdbc.password`, `paimon.jdbc.password`, `paimon.rest.dlf.access-key-id`, `paimon.rest.dlf.access-key-secret`, `mc.access_key`, and the `trino.*` and `adbc.*` properties whose names carry `key`, `secret`, `password`, `token` or `credential` (`authorization` too for `adbc.*`). ALTER CATALOG ... SET PROPERTIES statements are masked in the audit log like CREATE CATALOG. Third-party connector plugins built against API 31.x must be rebuilt for 32.0.

### Check List (For Author)

- Behavior changed: Yes (see Details)
- Does this need documentation: No
- Pre-submission status: Ready

### AI Disclosure

- Tool: Claude Code (Claude Opus 5.5); OpenAI Codex
- Generated: Claude Code wrote the original code, tests, and description. Codex rebased the commits, adapted the connector API version and its test, added the MongoDB declaration and ALTER CATALOG supplementary-character tests, and updated this description.
- Human review: not stated
- Prompts: The author requested a generic connector surface, prefix rules, declarations for every connector, and masking of their union with the engine floor. They removed sensitive marks from keytab paths, added ALTER CATALOG masking, and fixed startup failure from a connector's ServiceLoader. After review, they chose a separate shared Unicode fix, then requested MongoDB alignment and supplementary-character audit coverage here.
Gabriel39 added a commit that referenced this pull request Nov 17, 2026
…70604)

### Background

The native MongoDB Catalog on master already provides single-cursor
reads and first-release BSON SerDes. SQL filters and source projection
have remained local, and the remaining M1 type and pushdown contracts
need end-to-end qualification.

### What problem does this PR solve?

Issue Number: #70251 (D04b and D05)
Related PR: #70327 (native read path), #70456 (M0 acceptance), #70493
(session read policies)
PR Type: Feature

Problem Summary:

Complete the D04b qualification coverage and connect conservative
predicate/LIKE candidates and column pruning to the native MongoDB
reader. Every SQL predicate remains a Doris residual; the default strict
type policy, strict SQL casts and standard views keep filtering local.
This PR does not complete the separate D06 or later
snapshot/parallel-scan work.

### Details

- **Change map:** 456 added production lines, 1,221
test/fixture/generated-result lines, and 47 documentation lines across
19 files. The production changes comprise a connector-local translator
and scan planning, three session controls, two optional descriptor
fields, and native find options/validation. Most additions qualify types
and compare execution modes.
- **Scope:** D04b covers Decimal128 extremes, exact
integer/floating-point behavior, UTC dates and scales, binary
ownership/subtypes, missing/null, recursive arrays/structs, rollback and
existing document limits. D05 covers typed scalar-root comparisons,
positive IN/BETWEEN, scalar IS NOT NULL, safe AND/OR, necessary-literal
LIKE candidates, root projection and time-series Date ranges.
- **Design:** Candidates are classified conservatively as SUPERSET;
unsupported expressions remain local, and the EXACT category is reserved
for proven equivalence. No conjunct is consumed through either SPI path.
CASTs are rejected before the neutral converter can erase them. The
translator reconciles the schema/expression SPI date-scale and
decimal-width representations, including FE's DECIMAL128 name. Native
BSON is bounded and validated before client acquisition; views cannot
receive a predicate/collation override. Complete root projection retains
residual columns and scalar-ancestor errors.
- **Usage:** Use `SET` or `SET_VAR` with
`enable_mongodb_predicate_pushdown`, `enable_mongodb_like_pushdown` and
`enable_mongodb_projection_pushdown`. Predicate candidates require the
explicit catalog policy `mongodb.type_mismatch=null`.
- **Default behavior:** All three switches default to true. The default
`type_mismatch=error` policy preserves local filtering;
`enable_strict_cast=true` and views also preserve local filtering.
Source projection is eligible independently. DOUBLE/NaN comparisons,
NOT/!=/NOT IN, mismatch-sensitive IS NULL, functions, casts, nested
predicate paths and existing direct VARBINARY operator restrictions stay
local. LIKE keeps the complete SQL residual. LIMIT, aggregation, sort,
joins, snapshots and parallel scans are unchanged.
- **Compatibility:** The connector SPI and stored schemas are unchanged.
Descriptor fields 23/24 are optional; absent fields preserve the prior
full scan. The touched descriptor is absent from the four supported
release branches, so no field id is reused. Qualification uses matching
FE/BE versions.

### Testing

All testing below ran locally against MongoDB 8.0.20 with authentication
enabled and an ASAN BE:

- 64 MongoDB connector FE tests, 25 connector-session/forwarding tests,
3 session read-policy tests and 55 BSON/native-reader BE tests passed
(147 total).
- All four MongoDB SQL suites passed: 4 suites, 0 failed, 0 fatal, 0
skipped. This includes the existing native and M0 acceptance suites, the
session read-policy suite and the new M1 suite.
- The generated M1 output contains 40 identical sorted result pairs with
predicate/LIKE/projection disabled and enabled. It covers precision,
NULL/arrays, casts, collations, residual LIMIT, aggregates/joins, views
and time-series reads. The completed Profile check reports
`MongoRowsRead` of 10 with candidates disabled and 1 with them enabled.
- Source profiling independently confirmed actual BSON int64, Decimal128
and Date filters, projection, simple collection collation, and view
reads without a collation override.
- Full FE and ASAN BE builds passed on the integrated master base; the
SQL suites use that matching built pair. Full FE Checkstyle and the
final affected-core-file Checkstyle run passed with zero violations. All
affected C/C++ files pass clang-format 16, build hygiene and clang-tidy.
- Red/green checks reproduced missing candidate planning, acceptance of
malformed BSON descriptors, missing strict-cast gating, command-budget
gaps, schema/expression type representation gaps, and the false negative
for ARRAY/JSON IS NOT NULL with null elements before the corresponding
implementations.

### Release note

Native MongoDB scans can prune source fields and apply conservative
WHERE/LIKE candidates under the explicit null-on-type-mismatch policy.
SQL residuals remain active, and default strict scans and views preserve
local filtering. New session switches allow statement-level comparisons
and disabling these optimizations.

### Check List (For Author)

- Behavior changed: Yes (see Details).
- Does this need documentation: Yes; the connector and fixture READMEs
are updated here.
- Pre-submission status: Ready. Local UT, authenticated SQL regression,
builds and style/static checks passed; CI is pending.

### AI Disclosure

- Tool: Codex (GPT-6).
- Generated: Implementation, tests, fixture, documentation and this
description.
- Human review: Pending maintainer review.
morningman added a commit that referenced this pull request Nov 17, 2026
…udit text (#70338)

### Background

SHOW CREATE CATALOG, SHOW CATALOG, the `catalogs()` table function and
the audit text of CREATE CATALOG replace a property value with `*XXX`
when `DatasourcePrintableMap.isSensitiveKey` matches its key. Until now
the keys came from a list in fe-core and from the keys filesystem
plugins report at startup. A connector plugin marks its secret
properties `sensitive = true`, but that mark only masks the connector's
own log lines. A connector tells the engine about its security-relevant
properties through `ConnectorProvider#securitySurface()`.

### What problem does this PR solve?

Issue Number: None
Related PR: #69588 (added the connector security surface this extends),
#45164 (introduced the unmasked ALTER CATALOG audit text), #27115
(masked it for the statement #45164 replaced), #59502 (added
`iceberg.jdbc.password`), #54523 (added the Paimon REST DLF keys),
#65730 (branch-4.1's Lance catalog, whose REST secrets the Lance port
will declare on this surface), #70366 (fixed the shared Unicode SQL
rewrite used by this PR's ALTER masking), #70327 (added MongoDB
credentials to the engine masking floor)
PR Type: Bugfix

Problem Summary:

Catalog secrets that only a connector knows are on no masking list, so
SHOW CREATE CATALOG and the other renderings print them in clear:
`iceberg.jdbc.password`, `paimon.jdbc.password`, the Paimon REST DLF key
pair, `mc.access_key`, and credentials passed through under `trino.` or
`adbc.`. Separately, `ALTER CATALOG ... SET PROPERTIES` is written to
the audit log unmasked, whatever the key. Each connector now declares
its secrets on its security surface, taken from its `sensitive` marks,
and the surface can also declare a namespace of secret keys for options
passed through unread; Fluss's lake options were hard-coded in fe-core
for that reason. The engine masks the union, with fe-core's own list
kept as the floor. ALTER CATALOG is masked like CREATE CATALOG.

### Details

- **Change map:** 45 files, +1205 −81 over four commits: production code
+384 −51, tests +807 −26, the connector README +13 −3, the plugin API
version in `fe/fe-connector/pom.xml` (31.0 to 32.0). The first commit
adds secret keys to the surface (reviewed alone before). The second does
the rest, the third keeps a surface that fails in the connector's own
`ServiceLoader` from stopping FE, and the fourth aligns the newly added
MongoDB provider and tests ALTER masking with supplementary characters:
1. Key namespaces on the SPI: `withSensitiveKeyPrefix` and
`SensitiveKeyPrefix.covers`, which holds the matching rule; a collection
form of `withSensitiveKeys` (`ConnectorSecuritySurface`).
2. Engine: `isSensitiveKey` matches registered namespaces, the Fluss
rule becomes one of them; `ConnectorPluginManager` registers them and
reads every surface before it publishes a provider, skipping a provider
whose surface throws.
3. Connectors: every provider declares its secrets;
`MetaStoreProviders.sensitivePropertyKeys()` gives the union of a
plugin's metastore backends; `IcebergJdbcMetaStoreProvider` reports its
keys like its siblings; the keytab paths lose their `sensitive` mark.
MongoDB also declares its two annotated secrets, preserving its SSRF
endpoint and the engine fallback.
  4. ALTER CATALOG audit masking (+16).
- **Frequency:** Always. Create or alter a catalog that sets one of the
keys above, then run SHOW CREATE CATALOG, SHOW CATALOG `<name>` or
`select * from catalogs()`, or read the audit log after an ALTER CATALOG
... SET PROPERTIES.
- **Root cause:** `isSensitiveKey` consulted only fe-core's list and the
filesystem providers' keys. A connector's `sensitive` marks fed only
`toMaskedString`; `MetaStoreProvider.sensitivePropertyKeys()` existed
for this, but nothing called it, and the Iceberg JDBC backend did not
implement it. For ALTER CATALOG, `AlterCatalogPropertiesCommand` does
not implement `NeedAuditEncryption` and
`LogicalPlanBuilderForEncryption` has no `visitAlterCatalogProperties`,
so `AuditLogHelper` records the original text; the statement it replaced
in #45164 was masked by #27115.
- **Why existing tests missed it:** Unit tests:
`DatasourcePrintableMapTest` checks fe-core's own keys, the provider
surface tests checked only the SSRF and driver_url keys, and
`EncryptSQLTest` and `RepositoryAuditEncryptionTest` cover CREATE
CATALOG, repositories and resources but not ALTER CATALOG. Regression:
`audit/test_external_catalog_credential_masking` creates only Iceberg
REST catalogs, whose keys are on fe-core's list, and never alters one.
- **Fix:** The engine must not name a connector's properties, so the
connectors declare their secrets, from the same `sensitive` marks that
mask their logs; jdbc and es declare both spellings their legacy prefix
allows, and trino and adbc declare the namespace they pass through, a
key there being a secret when its rest names `key`, `secret`,
`password`, `token` or `credential` (also `authorization` for ADBC).
fe-core keeps the connector keys it lists, so a catalog whose plugin is
not loaded masks what it masked before. A provider whose
`securitySurface()` throws at startup is skipped, as a filesystem
provider is, instead of stopping FE; that includes the
`ServiceConfigurationError` a `ServiceLoader` throws when a descriptor
names a backend it cannot load, which is how the Hive, Iceberg and
Paimon surfaces fail, since they read their metastore backends' keys
through one.
- **Other paths and branches:** Fixed here: every rendering above and
FE's statement logging, which share `isSensitiveKey` and
`NeedAuditEncryption`. Not covered, since a key name cannot express them
safely: raw Iceberg SDK keys copied through (`token`, `credential`),
Paimon options re-keyed from `paimon.*`, Hadoop `fs.*` keys, and secrets
inside otherwise unmarked URLs. The ALTER CATALOG gap is on branch-4.0,
4.1 and 4.2 (from #45164); branch-3.1 still uses the masked statement.
branch-4.1 and 4.2 also print the connector keys above in clear; without
the connector SPI their fix differs and is tracked separately.
- **Behavior change:** The namespaces also mask non-secrets whose names
carry those words, such as `trino.hive.s3.aws-access-key` or a keystore
path. Keytab paths, which no release masks, are no longer masked in
connector log lines: marked `sensitive`, `hadoop.kerberos.keytab` would
be masked in every storage vault and resource.

### Testing

The original local runs (macOS) used master `f5c993f2c94` plus this
change:

- Regression `audit/test_external_catalog_credential_masking`, extended
with an Iceberg JDBC catalog: SHOW CREATE CATALOG masks
`iceberg.jdbc.password`, and the audit texts of its CREATE CATALOG and
of an ALTER CATALOG that sets the password and `s3.secret_key` hold only
`*XXX`. Against the local cluster's FE without this change it fails at
the first new check, SHOW CREATE CATALOG printing the password; with a
FE built from this change it passes.
- fe-connector-spi: all 160 tests, with the refreshed surface baseline
(+4 lines); the metastore modules: all tests; ten connector modules: 29
classes, 511 tests, including each connector's surface test, which
checks real key names.
- fe-core: 26 classes, 341 tests, including
`ConnectorPluginManagerTest`, `DatasourcePrintableMapTest`,
`EncryptSQLTest` and the audit and plugin gate tests; 14 of them again
after the last change to registration, and 7 classes, 76 tests, after
the third commit. Its two new `ConnectorPluginManagerTest` cases, a
plugin directory whose `lib/` holds a jar with a service descriptor
naming a missing metastore-style backend (the plugin next to it must
still load) and the same descriptor on the classpath, fail without the
third commit with that `ServiceConfigurationError` escaping
`loadPlugins` and `loadBuiltins`, and pass with it.
- Twenty-two mutations, each caught by its test: for example, reverting
either half of the ALTER CATALOG fix (`EncryptSQLTest` sees the
statement in clear), matching a marker anywhere in the key, an empty
metastore union, re-marking a keytab, or dropping
`ServiceConfigurationError` from either startup catch (each fails only
its own new case).

Rebased onto master `93c16ee2ca1` on 2026-11-16. Post-rebase local
verification:

- `bash run-fe-ut.sh --run
'org.apache.doris.connector.spi.*Test,ConnectorPluginManagerTest,DatasourcePrintableMapTest,EncryptSQLTest'`,
with `CUSTOM_MVN` appending `-Dcheckstyle.skip=false`: BUILD SUCCESS
across all 90 reactor modules; all 141 selected tests passed across 22
classes, including the inherited supplementary-character audit cases.
Checkstyle reported no violations.
- `git diff --check upstream-apache/master...HEAD`: passed. `git
range-diff` confirms the replayed patches are unchanged apart from
adapting the connector API version and its test to 32.0.
- Regression tests and a complete BE/FE build were not rerun locally
after this rebase; `run buildall` is requested in CI.

Follow-up local verification on the rebased branch plus the MongoDB
declaration and audit tests (2026-11-16):

- `run-fe-ut.sh --run
'org.apache.doris.connector.spi.*Test,org.apache.doris.connector.mongodb.*Test,ConnectorPluginManagerTest,DatasourcePrintableMapTest,EncryptSQLTest'`,
with Checkstyle enabled: all 176 selected tests in 28 classes passed
(SPI 93, MongoDB 35, fe-core 48), with no skipped tests; the 90-module
reactor completed successfully.
- The strengthened `MongoProviderTest` fails without the provider
declaration (expected the two MongoDB secret keys, received an empty
set), and passes with it. A separate targeted revert confirms the
assertion catches the missing declaration.
- The new ALTER CATALOG assertions cover supplementary characters within
a secret, in a preceding SQL comment, and in a retained non-secret
value, with two masked properties. `EncryptSQLTest#testEncryption`
passes on the candidate and fails at the new ALTER assertion when only
`BaseViewInfo.rewriteSql` is restored to its pre-#70366 implementation:
the audit text has an extra trailing fragment.
- Full cluster regression and a standalone FE/BE build were not rerun
for this follow-up.

### Notes for Reviewers

- `SensitiveKeyPrefix.covers` lives on the SPI so that a connector's
test can check its declaration against real key names and the engine
applies the same rule; the Fluss floor in fe-core uses it too.
- The entries both fe-core and a connector declare are deliberate: the
floor is frozen, as the comment in `DatasourcePrintableMap` says.
MongoDB was already masked by #70327; its added declaration makes the
provider follow the same contract as the other connectors.
- The connector plugin API advances from 31.0 to 32.0 for the added
security-surface methods. Third-party connector plugins built against
31.x must be rebuilt; the in-tree plugins are built with FE. Hudi
declares nothing: it owns no catalog properties.
- The base includes #70366, so ALTER CATALOG masking uses the shared
rewrite that converts parser code-point spans to Java string offsets,
including supplementary characters.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants