Skip to content

[feature](catalog) Add MongoDB session read policies - #70493

Merged
Gabriel39 merged 1 commit into
apache:masterfrom
Gabriel39:dev/mongodb-session-read-policy
Nov 16, 2026
Merged

Gabriel39 merged 1 commit into
apache:masterfrom
Gabriel39:dev/mongodb-session-read-policy

Conversation

@Gabriel39

Copy link
Copy Markdown
Contributor

Background

The native MongoDB catalog sends read preference and read concern to the BE in each scan descriptor. Today those values come only from catalog properties or the connection URI, while metadata discovery uses its own primary/default policy.

What problem does this PR solve?

Issue Number: #70251
Related PR: #70327 (native reader and catalog read policies)
PR Type: Feature

Problem Summary:

Users cannot choose MongoDB routing and visibility for an individual session or query without changing the shared catalog. Add mongodb_read_preference and mongodb_read_concern, supporting SET and SET_VAR, and pin their effective values in each scan descriptor. Both default to inherit, preserving existing catalog/URI behavior. Most of the diff adds tests and generated regression expectations.

Details

  • Change map: 70 production lines changed, 281 test-source lines changed, 62 generated expectation lines, and 38 documentation lines changed.
  • Scope: Data-read policies for native MongoDB scans. Preference supports inherit, primary, primaryPreferred, secondary, secondaryPreferred, and nearest; concern supports inherit, local, majority, and available. Values are case-insensitive. Snapshot and linearizable reads remain unsupported.
  • Design: Register validated, forwarded session variables and resolve them through the existing connector session-property snapshot. Reuse the existing scan descriptor and BE collection policy application. Session overrides do not mutate catalog configuration, metadata clients, or transport pool identity. URI tags, maximum staleness, and hedge options remain effective; an incompatible primary override fails during planning without exposing the URI.
  • Usage: SET mongodb_read_preference='secondaryPreferred'; SET mongodb_read_concern='local'; or SELECT /*+ SET_VAR(mongodb_read_preference='primary', mongodb_read_concern='majority') */ .... Set either variable to inherit to use its catalog/URI policy again.
  • Default behavior: inherit independently resolves each policy from URI options first, then catalog defaults. Queries without overrides retain their existing behavior.
  • Compatibility: No connector SPI, Thrift, BE code, or storage-format changes. Metadata discovery and collection-identity verification retain their existing policies. Read concern does not establish a snapshot shared by scans.

Testing

All checks below ran locally against MongoDB 8.0.20, with the new FE and the existing ASAN native BE:

  • Red/green: the new scan-plan tests first failed because overrides were ignored, and the new session tests first failed because the variables were unregistered; both passed after implementation.
  • run-fe-ut.sh --run with SessionVariableMongoReadPolicyTest, VariableMgrExportTest, MongoScanPlanTest, and MongoCatalogPropertiesTest: passed. The three new session tests cover defaults, supported/unsupported values, connector snapshots, isolation, and forwarding.
  • mvn -f fe/pom.xml -pl :fe-connector-mongodb -am test: 35 connector tests passed, including URI precedence, independent policy inheritance, incompatible routing criteria, and stable transport configuration identity.
  • build.sh --fe -j12: passed. FE core and MongoDB connector Checkstyle: zero violations.
  • Generated test_mongodb_read_policy.out using the regression runner, checked its results against the synthetic fixture, then independently reran test_mongodb_native and test_mongodb_read_policy: 2 suites passed, 0 failed/fatal/skipped. Covers SET/SET_VAR, all supported modes/levels, invalid values, failed-hint restoration, URI policy inheritance, and recovery.
  • Additional replica-set integration: source profiler verified primary/secondary selection and the actual read concern for 8 sequential reads and 32 concurrent reads across 4 isolated sessions. Also verified URI tags with a mode override and policy restoration after a failed hint. This was an additional local acceptance check; the committed SQL suites use the existing standalone fixture.

Release note

Add mongodb_read_preference and mongodb_read_concern session variables for native MongoDB queries, with statement-level SET_VAR support. Both default to inherit to retain the catalog's existing read policy.

Check List (For Author)

  • Behavior changed: Yes (see Details; explicit overrides now affect native MongoDB data reads).
  • Does this need documentation: Yes; the connector README is updated in this PR.
  • Pre-submission status: Ready. Local checks passed; CI is pending.

AI Disclosure

  • Tool: Codex (GPT-6).
  • Generated: Implementation, tests, connector documentation, and this description.
  • Human review: Maintainer review pending; the agent completed a full diff self-review and local validation.

### Background

The native MongoDB catalog sends read preference and read concern to the BE in each scan descriptor. Today those values come only from catalog properties or the connection URI, while metadata discovery uses its own primary/default policy.

### What problem does this PR solve?

Issue Number: apache#70251
Related PR: apache#70327 (native reader and catalog read policies)
PR Type: Feature

Problem Summary:

Users cannot choose MongoDB routing and visibility for an individual session or query without changing the shared catalog. Add `mongodb_read_preference` and `mongodb_read_concern`, supporting `SET` and `SET_VAR`, and pin their effective values in each scan descriptor. Both default to `inherit`, preserving existing catalog/URI behavior. Most of the diff adds tests and generated regression expectations.

### Details

- **Change map:** 70 production lines changed, 281 test-source lines changed, 62 generated expectation lines, and 38 documentation lines changed.
- **Scope:** Data-read policies for native MongoDB scans. Preference supports `inherit`, `primary`, `primaryPreferred`, `secondary`, `secondaryPreferred`, and `nearest`; concern supports `inherit`, `local`, `majority`, and `available`. Values are case-insensitive. Snapshot and linearizable reads remain unsupported.
- **Design:** Register validated, forwarded session variables and resolve them through the existing connector session-property snapshot. Reuse the existing scan descriptor and BE collection policy application. Session overrides do not mutate catalog configuration, metadata clients, or transport pool identity. URI tags, maximum staleness, and hedge options remain effective; an incompatible primary override fails during planning without exposing the URI.
- **Usage:** `SET mongodb_read_preference='secondaryPreferred'; SET mongodb_read_concern='local';` or `SELECT /*+ SET_VAR(mongodb_read_preference='primary', mongodb_read_concern='majority') */ ...`. Set either variable to `inherit` to use its catalog/URI policy again.
- **Default behavior:** `inherit` independently resolves each policy from URI options first, then catalog defaults. Queries without overrides retain their existing behavior.
- **Compatibility:** No connector SPI, Thrift, BE code, or storage-format changes. Metadata discovery and collection-identity verification retain their existing policies. Read concern does not establish a snapshot shared by scans.

### Testing

All checks below ran locally against MongoDB 8.0.20, with the new FE and the existing ASAN native BE:

- Red/green: the new scan-plan tests first failed because overrides were ignored, and the new session tests first failed because the variables were unregistered; both passed after implementation.
- `run-fe-ut.sh --run` with `SessionVariableMongoReadPolicyTest`, `VariableMgrExportTest`, `MongoScanPlanTest`, and `MongoCatalogPropertiesTest`: passed. The three new session tests cover defaults, supported/unsupported values, connector snapshots, isolation, and forwarding.
- `mvn -f fe/pom.xml -pl :fe-connector-mongodb -am test`: 35 connector tests passed, including URI precedence, independent policy inheritance, incompatible routing criteria, and stable transport configuration identity.
- `build.sh --fe -j12`: passed. FE core and MongoDB connector Checkstyle: zero violations.
- Generated `test_mongodb_read_policy.out` using the regression runner, checked its results against the synthetic fixture, then independently reran `test_mongodb_native` and `test_mongodb_read_policy`: 2 suites passed, 0 failed/fatal/skipped. Covers SET/SET_VAR, all supported modes/levels, invalid values, failed-hint restoration, URI policy inheritance, and recovery.
- Additional replica-set integration: source profiler verified primary/secondary selection and the actual read concern for 8 sequential reads and 32 concurrent reads across 4 isolated sessions. Also verified URI tags with a mode override and policy restoration after a failed hint. This was an additional local acceptance check; the committed SQL suites use the existing standalone fixture.

### Release note

Add `mongodb_read_preference` and `mongodb_read_concern` session variables for native MongoDB queries, with statement-level SET_VAR support. Both default to `inherit` to retain the catalog's existing read policy.

### Check List (For Author)

- Behavior changed: Yes (see Details; explicit overrides now affect native MongoDB data reads).
- Does this need documentation: Yes; the connector README is updated in this PR.
- Pre-submission status: Ready. Local checks passed; CI is pending.

### AI Disclosure

- Tool: Codex (GPT-6).
- Generated: Implementation, tests, connector documentation, and this description.
- Human review: Maintainer review pending; the agent completed a full diff self-review and local validation.
@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all eight changed paths at de95c7b. No new actionable findings and no existing blocking inline comments. Review converged after two rounds, including independent session, connector, and risk-focused reviews and a final challenge of this frozen payload.

Critical checkpoint conclusions:

  • Goal and evidence: SET and SET_VAR policies independently inherit or override catalog/URI defaults and are pinned in Mongo scan descriptors. New FE and connector unit tests and the regression suite cover the intended behavior; test sources were inspected, not executed.
  • Scope and reuse: the change is focused and uses the existing session-property snapshot, scan provider, descriptor, and native reader.
  • Concurrency and locks: each scan retains its own session map, descriptor and collection policy; pooled clients are leased exclusively. No new shared mutable policy, lock order or work under metadata locks was introduced.
  • Lifecycle: range descriptors are deep-copied, cursor/collection cleanup precedes pool return, and the existing failure path closes acquired resources. SET_VAR restoration runs in the executor's finally path, including planning errors. No new static initialization or resource lifetime was introduced.
  • Configuration: both variables default to inherit, validate supported values case-insensitively, update without restart, and participate in generic session export and FE forwarding.
  • Compatibility: existing Mongo descriptor fields are reused; no connector SPI, function symbol, storage format or new wire field changes. Missing session keys preserve inheritance. Existing native policy consumers accept the canonical values.
  • Thrift/protobuf IDs: no gensrc files changed; branch ID comparison is inapplicable.
  • Parallel paths: collection, view and time-series scans use the same provider. Prepared external queries replan, forwarded queries carry the variables, and Mongo tables are excluded from SQL result caching. Metadata discovery and collection identity verification retain independent policies.
  • Conditional checks and errors: unsupported policies fail rather than fall back. The primary/URI routing-criteria check matches the native consumer's constraints; inherited criteria remain attached to the scan (checked against the pinned C driver and Java driver implementations).
  • Test coverage: inspected defaults, supported and rejected values, case handling, URI precedence, primary conflicts, immutable snapshots, transport identity, hints and restoration. The committed standalone fixture cannot establish replica selection or concurrent replica behavior.
  • Expected results: the fixture's four rows have seq values 1 through 4, matching count=4 and sum=10. Variable and hint/restoration expectations match the SQL sequence; SHOW output is ordered and aggregate results have one row. Result generation was not independently reproduced.
  • Observability: variables are visible through SHOW/session inspection; existing changed-variable audit capture and native errors remain available. No new metric or high-volume log requirement was identified.
  • Transactions and persistence: no source transaction, EditLog protocol or failover behavior changes. Existing reflected session-variable serialization covers the added strings; read concern does not create a shared snapshot across scans.
  • Data writes and atomicity: this change only affects external reads; no write, delete-bitmap, rowset or Doris visible-version logic changes.
  • FE-to-BE propagation: canonical effective preference/concern values reach the existing TMongoScanDesc on each native range; constant folding and OLAP point-query paths do not read Mongo collections.
  • Performance and memory: added resolution does bounded local work per scan, without new I/O or row-loop work. No new long-lived allocation or BE memory ownership was introduced.
  • Other concerns and user focus: no additional focus was supplied. Both initial risk items were dismissed with concrete upstream/downstream evidence, and no code-level candidate remains unresolved.

Validation limits: static inspection only. No builds, tests, benchmarks, cluster operations or runtime reproduction were performed. Author-reported replica-set profiling and concurrency checks were not independently verified. These execution limits do not leave an uncovered portion of the required static review.

Review converged after 2 round(s).

@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39
Gabriel39 merged commit fc29f25 into apache:master Nov 16, 2026
35 checks passed
Gabriel39 added a commit that referenced this pull request Nov 17, 2026
…70604)

### Background

The native MongoDB Catalog on master already provides single-cursor
reads and first-release BSON SerDes. SQL filters and source projection
have remained local, and the remaining M1 type and pushdown contracts
need end-to-end qualification.

### What problem does this PR solve?

Issue Number: #70251 (D04b and D05)
Related PR: #70327 (native read path), #70456 (M0 acceptance), #70493
(session read policies)
PR Type: Feature

Problem Summary:

Complete the D04b qualification coverage and connect conservative
predicate/LIKE candidates and column pruning to the native MongoDB
reader. Every SQL predicate remains a Doris residual; the default strict
type policy, strict SQL casts and standard views keep filtering local.
This PR does not complete the separate D06 or later
snapshot/parallel-scan work.

### Details

- **Change map:** 456 added production lines, 1,221
test/fixture/generated-result lines, and 47 documentation lines across
19 files. The production changes comprise a connector-local translator
and scan planning, three session controls, two optional descriptor
fields, and native find options/validation. Most additions qualify types
and compare execution modes.
- **Scope:** D04b covers Decimal128 extremes, exact
integer/floating-point behavior, UTC dates and scales, binary
ownership/subtypes, missing/null, recursive arrays/structs, rollback and
existing document limits. D05 covers typed scalar-root comparisons,
positive IN/BETWEEN, scalar IS NOT NULL, safe AND/OR, necessary-literal
LIKE candidates, root projection and time-series Date ranges.
- **Design:** Candidates are classified conservatively as SUPERSET;
unsupported expressions remain local, and the EXACT category is reserved
for proven equivalence. No conjunct is consumed through either SPI path.
CASTs are rejected before the neutral converter can erase them. The
translator reconciles the schema/expression SPI date-scale and
decimal-width representations, including FE's DECIMAL128 name. Native
BSON is bounded and validated before client acquisition; views cannot
receive a predicate/collation override. Complete root projection retains
residual columns and scalar-ancestor errors.
- **Usage:** Use `SET` or `SET_VAR` with
`enable_mongodb_predicate_pushdown`, `enable_mongodb_like_pushdown` and
`enable_mongodb_projection_pushdown`. Predicate candidates require the
explicit catalog policy `mongodb.type_mismatch=null`.
- **Default behavior:** All three switches default to true. The default
`type_mismatch=error` policy preserves local filtering;
`enable_strict_cast=true` and views also preserve local filtering.
Source projection is eligible independently. DOUBLE/NaN comparisons,
NOT/!=/NOT IN, mismatch-sensitive IS NULL, functions, casts, nested
predicate paths and existing direct VARBINARY operator restrictions stay
local. LIKE keeps the complete SQL residual. LIMIT, aggregation, sort,
joins, snapshots and parallel scans are unchanged.
- **Compatibility:** The connector SPI and stored schemas are unchanged.
Descriptor fields 23/24 are optional; absent fields preserve the prior
full scan. The touched descriptor is absent from the four supported
release branches, so no field id is reused. Qualification uses matching
FE/BE versions.

### Testing

All testing below ran locally against MongoDB 8.0.20 with authentication
enabled and an ASAN BE:

- 64 MongoDB connector FE tests, 25 connector-session/forwarding tests,
3 session read-policy tests and 55 BSON/native-reader BE tests passed
(147 total).
- All four MongoDB SQL suites passed: 4 suites, 0 failed, 0 fatal, 0
skipped. This includes the existing native and M0 acceptance suites, the
session read-policy suite and the new M1 suite.
- The generated M1 output contains 40 identical sorted result pairs with
predicate/LIKE/projection disabled and enabled. It covers precision,
NULL/arrays, casts, collations, residual LIMIT, aggregates/joins, views
and time-series reads. The completed Profile check reports
`MongoRowsRead` of 10 with candidates disabled and 1 with them enabled.
- Source profiling independently confirmed actual BSON int64, Decimal128
and Date filters, projection, simple collection collation, and view
reads without a collation override.
- Full FE and ASAN BE builds passed on the integrated master base; the
SQL suites use that matching built pair. Full FE Checkstyle and the
final affected-core-file Checkstyle run passed with zero violations. All
affected C/C++ files pass clang-format 16, build hygiene and clang-tidy.
- Red/green checks reproduced missing candidate planning, acceptance of
malformed BSON descriptors, missing strict-cast gating, command-budget
gaps, schema/expression type representation gaps, and the false negative
for ARRAY/JSON IS NOT NULL with null elements before the corresponding
implementations.

### Release note

Native MongoDB scans can prune source fields and apply conservative
WHERE/LIKE candidates under the explicit null-on-type-mismatch policy.
SQL residuals remain active, and default strict scans and views preserve
local filtering. New session switches allow statement-level comparisons
and disabling these optimizations.

### Check List (For Author)

- Behavior changed: Yes (see Details).
- Does this need documentation: Yes; the connector and fixture READMEs
are updated here.
- Pre-submission status: Ready. Local UT, authenticated SQL regression,
builds and style/static checks passed; CI is pending.

### AI Disclosure

- Tool: Codex (GPT-6).
- Generated: Implementation, tests, fixture, documentation and this
description.
- Human review: Pending maintainer review.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants