Repository navigation
[feature](catalog) Add MongoDB session read policies - #70493
Merged
Gabriel39 merged 1 commit intoNov 16, 2026
Merged
Conversation
### Background The native MongoDB catalog sends read preference and read concern to the BE in each scan descriptor. Today those values come only from catalog properties or the connection URI, while metadata discovery uses its own primary/default policy. ### What problem does this PR solve? Issue Number: apache#70251 Related PR: apache#70327 (native reader and catalog read policies) PR Type: Feature Problem Summary: Users cannot choose MongoDB routing and visibility for an individual session or query without changing the shared catalog. Add `mongodb_read_preference` and `mongodb_read_concern`, supporting `SET` and `SET_VAR`, and pin their effective values in each scan descriptor. Both default to `inherit`, preserving existing catalog/URI behavior. Most of the diff adds tests and generated regression expectations. ### Details - **Change map:** 70 production lines changed, 281 test-source lines changed, 62 generated expectation lines, and 38 documentation lines changed. - **Scope:** Data-read policies for native MongoDB scans. Preference supports `inherit`, `primary`, `primaryPreferred`, `secondary`, `secondaryPreferred`, and `nearest`; concern supports `inherit`, `local`, `majority`, and `available`. Values are case-insensitive. Snapshot and linearizable reads remain unsupported. - **Design:** Register validated, forwarded session variables and resolve them through the existing connector session-property snapshot. Reuse the existing scan descriptor and BE collection policy application. Session overrides do not mutate catalog configuration, metadata clients, or transport pool identity. URI tags, maximum staleness, and hedge options remain effective; an incompatible primary override fails during planning without exposing the URI. - **Usage:** `SET mongodb_read_preference='secondaryPreferred'; SET mongodb_read_concern='local';` or `SELECT /*+ SET_VAR(mongodb_read_preference='primary', mongodb_read_concern='majority') */ ...`. Set either variable to `inherit` to use its catalog/URI policy again. - **Default behavior:** `inherit` independently resolves each policy from URI options first, then catalog defaults. Queries without overrides retain their existing behavior. - **Compatibility:** No connector SPI, Thrift, BE code, or storage-format changes. Metadata discovery and collection-identity verification retain their existing policies. Read concern does not establish a snapshot shared by scans. ### Testing All checks below ran locally against MongoDB 8.0.20, with the new FE and the existing ASAN native BE: - Red/green: the new scan-plan tests first failed because overrides were ignored, and the new session tests first failed because the variables were unregistered; both passed after implementation. - `run-fe-ut.sh --run` with `SessionVariableMongoReadPolicyTest`, `VariableMgrExportTest`, `MongoScanPlanTest`, and `MongoCatalogPropertiesTest`: passed. The three new session tests cover defaults, supported/unsupported values, connector snapshots, isolation, and forwarding. - `mvn -f fe/pom.xml -pl :fe-connector-mongodb -am test`: 35 connector tests passed, including URI precedence, independent policy inheritance, incompatible routing criteria, and stable transport configuration identity. - `build.sh --fe -j12`: passed. FE core and MongoDB connector Checkstyle: zero violations. - Generated `test_mongodb_read_policy.out` using the regression runner, checked its results against the synthetic fixture, then independently reran `test_mongodb_native` and `test_mongodb_read_policy`: 2 suites passed, 0 failed/fatal/skipped. Covers SET/SET_VAR, all supported modes/levels, invalid values, failed-hint restoration, URI policy inheritance, and recovery. - Additional replica-set integration: source profiler verified primary/secondary selection and the actual read concern for 8 sequential reads and 32 concurrent reads across 4 isolated sessions. Also verified URI tags with a mode override and policy restoration after a failed hint. This was an additional local acceptance check; the committed SQL suites use the existing standalone fixture. ### Release note Add `mongodb_read_preference` and `mongodb_read_concern` session variables for native MongoDB queries, with statement-level SET_VAR support. Both default to `inherit` to retain the catalog's existing read policy. ### Check List (For Author) - Behavior changed: Yes (see Details; explicit overrides now affect native MongoDB data reads). - Does this need documentation: Yes; the connector README is updated in this PR. - Pre-submission status: Ready. Local checks passed; CI is pending. ### AI Disclosure - Tool: Codex (GPT-6). - Generated: Implementation, tests, connector documentation, and this description. - Human review: Maintainer review pending; the agent completed a full diff self-review and local validation.
Gabriel39
requested review from
924060929,
CalvinKirs and
morrySnow
as code owners
November 12, 2026 03:15
Contributor
Author
|
run buildall |
Contributor
Author
|
/review |
8 of 12 tasks
Contributor
There was a problem hiding this comment.
Reviewed all eight changed paths at de95c7b. No new actionable findings and no existing blocking inline comments. Review converged after two rounds, including independent session, connector, and risk-focused reviews and a final challenge of this frozen payload.
Critical checkpoint conclusions:
- Goal and evidence: SET and SET_VAR policies independently inherit or override catalog/URI defaults and are pinned in Mongo scan descriptors. New FE and connector unit tests and the regression suite cover the intended behavior; test sources were inspected, not executed.
- Scope and reuse: the change is focused and uses the existing session-property snapshot, scan provider, descriptor, and native reader.
- Concurrency and locks: each scan retains its own session map, descriptor and collection policy; pooled clients are leased exclusively. No new shared mutable policy, lock order or work under metadata locks was introduced.
- Lifecycle: range descriptors are deep-copied, cursor/collection cleanup precedes pool return, and the existing failure path closes acquired resources. SET_VAR restoration runs in the executor's finally path, including planning errors. No new static initialization or resource lifetime was introduced.
- Configuration: both variables default to inherit, validate supported values case-insensitively, update without restart, and participate in generic session export and FE forwarding.
- Compatibility: existing Mongo descriptor fields are reused; no connector SPI, function symbol, storage format or new wire field changes. Missing session keys preserve inheritance. Existing native policy consumers accept the canonical values.
- Thrift/protobuf IDs: no gensrc files changed; branch ID comparison is inapplicable.
- Parallel paths: collection, view and time-series scans use the same provider. Prepared external queries replan, forwarded queries carry the variables, and Mongo tables are excluded from SQL result caching. Metadata discovery and collection identity verification retain independent policies.
- Conditional checks and errors: unsupported policies fail rather than fall back. The primary/URI routing-criteria check matches the native consumer's constraints; inherited criteria remain attached to the scan (checked against the pinned C driver and Java driver implementations).
- Test coverage: inspected defaults, supported and rejected values, case handling, URI precedence, primary conflicts, immutable snapshots, transport identity, hints and restoration. The committed standalone fixture cannot establish replica selection or concurrent replica behavior.
- Expected results: the fixture's four rows have seq values 1 through 4, matching count=4 and sum=10. Variable and hint/restoration expectations match the SQL sequence; SHOW output is ordered and aggregate results have one row. Result generation was not independently reproduced.
- Observability: variables are visible through SHOW/session inspection; existing changed-variable audit capture and native errors remain available. No new metric or high-volume log requirement was identified.
- Transactions and persistence: no source transaction, EditLog protocol or failover behavior changes. Existing reflected session-variable serialization covers the added strings; read concern does not create a shared snapshot across scans.
- Data writes and atomicity: this change only affects external reads; no write, delete-bitmap, rowset or Doris visible-version logic changes.
- FE-to-BE propagation: canonical effective preference/concern values reach the existing TMongoScanDesc on each native range; constant folding and OLAP point-query paths do not read Mongo collections.
- Performance and memory: added resolution does bounded local work per scan, without new I/O or row-loop work. No new long-lived allocation or BE memory ownership was introduced.
- Other concerns and user focus: no additional focus was supplied. Both initial risk items were dismissed with concrete upstream/downstream evidence, and no code-level candidate remains unresolved.
Validation limits: static inspection only. No builds, tests, benchmarks, cluster operations or runtime reproduction were performed. Author-reported replica-set profiling and concurrency checks were not independently verified. These execution limits do not leave an uncovered portion of the required static review.
Review converged after 2 round(s).
Contributor
Author
|
run buildall |
924060929
approved these changes
Nov 16, 2026
Gabriel39
added a commit
that referenced
this pull request
Nov 17, 2026
…70604) ### Background The native MongoDB Catalog on master already provides single-cursor reads and first-release BSON SerDes. SQL filters and source projection have remained local, and the remaining M1 type and pushdown contracts need end-to-end qualification. ### What problem does this PR solve? Issue Number: #70251 (D04b and D05) Related PR: #70327 (native read path), #70456 (M0 acceptance), #70493 (session read policies) PR Type: Feature Problem Summary: Complete the D04b qualification coverage and connect conservative predicate/LIKE candidates and column pruning to the native MongoDB reader. Every SQL predicate remains a Doris residual; the default strict type policy, strict SQL casts and standard views keep filtering local. This PR does not complete the separate D06 or later snapshot/parallel-scan work. ### Details - **Change map:** 456 added production lines, 1,221 test/fixture/generated-result lines, and 47 documentation lines across 19 files. The production changes comprise a connector-local translator and scan planning, three session controls, two optional descriptor fields, and native find options/validation. Most additions qualify types and compare execution modes. - **Scope:** D04b covers Decimal128 extremes, exact integer/floating-point behavior, UTC dates and scales, binary ownership/subtypes, missing/null, recursive arrays/structs, rollback and existing document limits. D05 covers typed scalar-root comparisons, positive IN/BETWEEN, scalar IS NOT NULL, safe AND/OR, necessary-literal LIKE candidates, root projection and time-series Date ranges. - **Design:** Candidates are classified conservatively as SUPERSET; unsupported expressions remain local, and the EXACT category is reserved for proven equivalence. No conjunct is consumed through either SPI path. CASTs are rejected before the neutral converter can erase them. The translator reconciles the schema/expression SPI date-scale and decimal-width representations, including FE's DECIMAL128 name. Native BSON is bounded and validated before client acquisition; views cannot receive a predicate/collation override. Complete root projection retains residual columns and scalar-ancestor errors. - **Usage:** Use `SET` or `SET_VAR` with `enable_mongodb_predicate_pushdown`, `enable_mongodb_like_pushdown` and `enable_mongodb_projection_pushdown`. Predicate candidates require the explicit catalog policy `mongodb.type_mismatch=null`. - **Default behavior:** All three switches default to true. The default `type_mismatch=error` policy preserves local filtering; `enable_strict_cast=true` and views also preserve local filtering. Source projection is eligible independently. DOUBLE/NaN comparisons, NOT/!=/NOT IN, mismatch-sensitive IS NULL, functions, casts, nested predicate paths and existing direct VARBINARY operator restrictions stay local. LIKE keeps the complete SQL residual. LIMIT, aggregation, sort, joins, snapshots and parallel scans are unchanged. - **Compatibility:** The connector SPI and stored schemas are unchanged. Descriptor fields 23/24 are optional; absent fields preserve the prior full scan. The touched descriptor is absent from the four supported release branches, so no field id is reused. Qualification uses matching FE/BE versions. ### Testing All testing below ran locally against MongoDB 8.0.20 with authentication enabled and an ASAN BE: - 64 MongoDB connector FE tests, 25 connector-session/forwarding tests, 3 session read-policy tests and 55 BSON/native-reader BE tests passed (147 total). - All four MongoDB SQL suites passed: 4 suites, 0 failed, 0 fatal, 0 skipped. This includes the existing native and M0 acceptance suites, the session read-policy suite and the new M1 suite. - The generated M1 output contains 40 identical sorted result pairs with predicate/LIKE/projection disabled and enabled. It covers precision, NULL/arrays, casts, collations, residual LIMIT, aggregates/joins, views and time-series reads. The completed Profile check reports `MongoRowsRead` of 10 with candidates disabled and 1 with them enabled. - Source profiling independently confirmed actual BSON int64, Decimal128 and Date filters, projection, simple collection collation, and view reads without a collation override. - Full FE and ASAN BE builds passed on the integrated master base; the SQL suites use that matching built pair. Full FE Checkstyle and the final affected-core-file Checkstyle run passed with zero violations. All affected C/C++ files pass clang-format 16, build hygiene and clang-tidy. - Red/green checks reproduced missing candidate planning, acceptance of malformed BSON descriptors, missing strict-cast gating, command-budget gaps, schema/expression type representation gaps, and the false negative for ARRAY/JSON IS NOT NULL with null elements before the corresponding implementations. ### Release note Native MongoDB scans can prune source fields and apply conservative WHERE/LIKE candidates under the explicit null-on-type-mismatch policy. SQL residuals remain active, and default strict scans and views preserve local filtering. New session switches allow statement-level comparisons and disabling these optimizations. ### Check List (For Author) - Behavior changed: Yes (see Details). - Does this need documentation: Yes; the connector and fixture READMEs are updated here. - Pre-submission status: Ready. Local UT, authenticated SQL regression, builds and style/static checks passed; CI is pending. ### AI Disclosure - Tool: Codex (GPT-6). - Generated: Implementation, tests, fixture, documentation and this description. - Human review: Pending maintainer review.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
The native MongoDB catalog sends read preference and read concern to the BE in each scan descriptor. Today those values come only from catalog properties or the connection URI, while metadata discovery uses its own primary/default policy.
What problem does this PR solve?
Issue Number: #70251
Related PR: #70327 (native reader and catalog read policies)
PR Type: Feature
Problem Summary:
Users cannot choose MongoDB routing and visibility for an individual session or query without changing the shared catalog. Add
mongodb_read_preferenceandmongodb_read_concern, supportingSETandSET_VAR, and pin their effective values in each scan descriptor. Both default toinherit, preserving existing catalog/URI behavior. Most of the diff adds tests and generated regression expectations.Details
inherit,primary,primaryPreferred,secondary,secondaryPreferred, andnearest; concern supportsinherit,local,majority, andavailable. Values are case-insensitive. Snapshot and linearizable reads remain unsupported.SET mongodb_read_preference='secondaryPreferred'; SET mongodb_read_concern='local';orSELECT /*+ SET_VAR(mongodb_read_preference='primary', mongodb_read_concern='majority') */ .... Set either variable toinheritto use its catalog/URI policy again.inheritindependently resolves each policy from URI options first, then catalog defaults. Queries without overrides retain their existing behavior.Testing
All checks below ran locally against MongoDB 8.0.20, with the new FE and the existing ASAN native BE:
run-fe-ut.sh --runwithSessionVariableMongoReadPolicyTest,VariableMgrExportTest,MongoScanPlanTest, andMongoCatalogPropertiesTest: passed. The three new session tests cover defaults, supported/unsupported values, connector snapshots, isolation, and forwarding.mvn -f fe/pom.xml -pl :fe-connector-mongodb -am test: 35 connector tests passed, including URI precedence, independent policy inheritance, incompatible routing criteria, and stable transport configuration identity.build.sh --fe -j12: passed. FE core and MongoDB connector Checkstyle: zero violations.test_mongodb_read_policy.outusing the regression runner, checked its results against the synthetic fixture, then independently rerantest_mongodb_nativeandtest_mongodb_read_policy: 2 suites passed, 0 failed/fatal/skipped. Covers SET/SET_VAR, all supported modes/levels, invalid values, failed-hint restoration, URI policy inheritance, and recovery.Release note
Add
mongodb_read_preferenceandmongodb_read_concernsession variables for native MongoDB queries, with statement-level SET_VAR support. Both default toinheritto retain the catalog's existing read policy.Check List (For Author)
AI Disclosure