Skip to content

[fix](fe) Guard timestamptz cast pruning across clock gaps - #70469

Merged
jacktengg merged 2 commits into
apache:masterfrom
jacktengg:DORIS-29990
Nov 16, 2026
Merged

jacktengg merged 2 commits into
apache:masterfrom
jacktengg:DORIS-29990

Conversation

@jacktengg

Copy link
Copy Markdown
Contributor

Issue Number: N/A

Related PR: N/A

Problem Summary: In America/New_York, the local datetime range [2024-03-10 02:30:00, 2024-03-10 03:00:00.100000) does not convert monotonically to TIMESTAMPTZ. The row 02:40:00.900000 converts to 07:00:00.900000 UTC, above the converted upper endpoint 07:00:00.100000 UTC. Cast treated DATETIMEV2 to TIMESTAMPTZ as monotonic, so partition pruning removed a partition that contained a matching row. Route this cast through the existing local-time gap check. Keep the fixed-offset fast path and pruning for ranges outside clock gaps. Add a regression test with one partition, one column, and one row.

Fix missing query results when partition predicates cast DATETIME values to TIMESTAMPTZ across a daylight-saving time gap.

  • Test:
    • Unit Test: 55 tests passed through run-fe-ut.sh for CastMonotonicityTest, TimestampNsMonotonicityTest, FromSecondMonotonicTest, PartitionPrunerTest, and RuntimeFilterPruneClassifierTest
    • Regression test: run-regression-test.sh --run -d datatype_p0/timestamptz -s test_timestamptz_cast_gap_partition_prune; generate the output with -genOut, then verify it with a normal run
    • Manual test: reproduce VEMPTYSET before the fix; verify one returned row and partitions=1/1 after the fix; compare FE and BE endpoint results
    • Regression environment: rebuilt FE and the existing local BE with be_exec_version=18
    • Build and style: build.sh --fe -j8 with Checkstyle enabled; final mvn checkstyle:check -pl fe-core
  • Behavior changed: Yes, retain partitions whose cast input range includes a clock gap to prevent missing matching rows
  • Does this need documentation: No

Background

What problem does this PR solve?

Issue Number:
Related PR:
PR Type:

Problem Summary:

Details

Testing

Notes for Reviewers

Release note

Check List (For Author)

  • Behavior changed:
  • Does this need documentation:
  • Pre-submission status:

AI Disclosure

  • Tool:
  • Generated:
  • Human review:

@jacktengg

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Review setup stopped before context preparation ran; inspect the first failed or cancelled setup step.
Workflow run: https://github.com/apache/doris/actions/runs/42992262147

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

Issue Number: N/A

Related PR: N/A

Problem Summary: In America/New_York, the local datetime range [2024-03-10 02:30:00, 2024-03-10 03:00:00.100000) does not convert monotonically to TIMESTAMPTZ. The row 02:40:00.900000 converts to 07:00:00.900000 UTC, above the converted upper endpoint 07:00:00.100000 UTC. Cast treated DATETIMEV2 to TIMESTAMPTZ as monotonic, so partition pruning removed a partition that contained a matching row. Route this cast through the existing local-time gap check. Keep the fixed-offset fast path and pruning for ranges outside clock gaps. Add a regression test with one partition, one column, and one row.

Fix missing query results when partition predicates cast DATETIME values to TIMESTAMPTZ across a daylight-saving time gap.

- Test:
    - Unit Test: 55 tests passed through run-fe-ut.sh for CastMonotonicityTest, TimestampNsMonotonicityTest, FromSecondMonotonicTest, PartitionPrunerTest, and RuntimeFilterPruneClassifierTest
    - Regression test: run-regression-test.sh --run -d datatype_p0/timestamptz -s test_timestamptz_cast_gap_partition_prune; generate the output with -genOut, then verify it with a normal run
    - Manual test: reproduce VEMPTYSET before the fix; verify one returned row and partitions=1/1 after the fix; compare FE and BE endpoint results
    - Regression environment: rebuilt FE and the existing local BE with be_exec_version=18
    - Build and style: build.sh --fe -j8 with Checkstyle enabled; final mvn checkstyle:check -pl fe-core
- Behavior changed: Yes, retain partitions whose cast input range includes a clock gap to prevent missing matching rows
- Does this need documentation: No
@shuke987

Copy link
Copy Markdown
Contributor

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all four changed files at commit 1a43c58. No new actionable findings were identified, and there are no existing inline findings to carry forward. No additional user review focus was supplied.

The fix routes DATETIMEV2 -> TIMESTAMPTZ through the existing clock-gap check. In the reduced path Filter(CAST(dt AS TIMESTAMPTZ) > constant) -> Scan(dt), partition evaluation now declines endpoint inference when the input range intersects a gap, retaining the matching row. Static and runtime-filter partition pruning share this guard. The TIMESTAMP_NS rounding path and fixed-offset fast path are preserved.

Critical checkpoint conclusions:

  1. Goal and evidence: the implementation addresses the reported missing-row prune; the three added unit cases and one-row regression directly cover it. This is a static assessment, not an executed test result.
  2. Scope and reuse: the production change is small and reuses the existing transition scanner; the other changes are focused tests and expected output.
  3. Concurrency and locks: no shared mutable state, thread creation, or lock changes; session-zone lookup uses the existing thread-local context.
  4. Lifecycle and initialization: no new production resource ownership or static initialization dependency; the unit fixture restores its prior context and the regression restores its timezone in finally.
  5. Configuration: no new settings or dynamic-update requirements.
  6. Compatibility: no persisted format, protocol, registration, or externally consumed function-symbol change; the Java helper rename is consistent with its callers.
  7. Thrift/protobuf: untouched.
  8. Parallel paths: Cast supplies both static and runtime-filter checks; inverse TIMESTAMPTZ casts retain their existing logic, and unsupported legacy/date-only source casts need no extra dispatch branch.
  9. Special conditions: dynamic-zone unbounded ranges conservatively decline inference; fixed offsets keep the fast path. Paired endpoints retain their source type. Zero month/day partition bounds are rejected by current parsers. Scale rounding cannot enter a gap beyond its exact start from a wholly preceding input interval, so raw DATETIMEV2 bounds remain conservative for the added check.
  10. Test coverage: added assertions cover gap intersections, exact boundaries, safe ranges, fall overlap, and unbounded/fixed-offset cases; the regression checks the previously missing row. Existing TIMESTAMP_NS and inverse-cast tests remain in place. No tests were run during this review.
  11. Expected results: the new output agrees with BE conversion: local 02:40:00.900000 maps to 07:00:00.900000 UTC and renders as 03:00:00.900000-04:00. Existing FE endpoint folding differs inside gaps; the new guard prevents those bounds being used for pruning.
  12. Observability: no additional logs or metrics are needed for this local correctness guard.
  13. Transactions, persistence, and failover: no production changes to these paths.
  14. Writes, atomicity, and crash recovery: no production data-write changes.
  15. FE/BE variables: no new transmitted state or send paths.
  16. Performance: fixed offsets remain constant-time; dynamic zones reuse transition scanning with early exit on a gap. No actionable regression was identified.
  17. Further checks: reviewed boundary inclusivity, destination scales, literal subclasses, invalid/min/max bounds, renamed-helper references, and parallel consumers; no unresolved candidate remains.

Coverage limits: static source and test-artifact inspection only. No builds, tests, benchmarks, or runtime checks were executed, and the author's historical test execution/output generation was not independently verified. All changed paths and identified risk focuses were covered; no unresolved code-review coverage gap remains. Review converged after two rounds, including a final challenge of the frozen submission.

Review converged after 2 round(s).

@jacktengg

Copy link
Copy Markdown
Contributor Author

/review

@jacktengg

Copy link
Copy Markdown
Contributor Author

run buildall

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all four changed files at head 1a43c58. No new actionable findings or existing inline blockers were identified. No additional user focus was supplied.

The fix addresses unsafe endpoint inference in the reduced plan Filter(CAST(dt AS TIMESTAMPTZ) > constant) -> Scan(partitioned dt). Cast.isMonotonic now sends DATETIMEV2 ranges through the clock-gap check; Monotonic.inferRange declines inference for unsafe ranges and retains their partitions. Runtime-filter partition classification uses the same guard.

Critical checkpoint conclusions:

  1. Goal and evidence: the production change addresses the reported missing row. Added unit cases and the one-row regression distinguish the old unsafe classification; this conclusion comes from static inspection.
  2. Scope and reuse: the change is focused and generalizes the existing helper and transition scanner.
  3. Concurrency and locking: no new threads, shared mutable state or locks; timezone lookup follows the existing session context.
  4. Lifecycle and initialization: no production ownership or static-initialization changes; tests restore ConnectContext/session timezone.
  5. Configuration: no new settings or dynamic-update obligations.
  6. Compatibility: no storage, wire-format or externally registered function changes; the internal Java helper rename matches its callers.
  7. Thrift/protobuf: unchanged.
  8. Parallel paths: both static and runtime-filter pruning use Cast's classifier. TIMESTAMP_NS rounding, reverse-cast checks and fixed-offset behavior are preserved; CheckCast's supported source types match the dispatch.
  9. Special conditions and errors: dynamic-zone missing bounds conservatively reject inference. Typed range producers supply paired endpoint types; current parsers reject zero month/day partition bounds. Scale reduction from a range wholly before a gap can reach its exact start without crossing an internal fractional reset, so the raw DATETIMEV2 check remains conservative.
  10. Test coverage: added assertions cover intersecting gaps, exact boundaries, adjacent safe ranges, fall overlap and bounded/unbounded timezone behavior. Existing inverse-cast and TIMESTAMP_NS tests remain. The regression checks the previously missing interior row with deterministic output and restores session state.
  11. Expected results: the new row/output is consistent with BE conversion: 02:40:00.900000 maps to 07:00:00.900000 UTC, renders as 03:00:00.900000-04:00, and exceeds the predicate threshold. The test comment's 07:40 value is stale prose without a functional effect. Existing FE/BE fractional behavior inside gaps does not undermine this fix because such ranges now decline inference.
  12. Observability: existing partition-plan inspection is adequate; this local guard needs no additional logs or metrics.
  13. Transactions, persistence and failover: unaffected.
  14. Data writes, atomicity and crash recovery: no production write-path changes.
  15. FE/BE variables: no transmitted state or send-path changes.
  16. Performance and memory: fixed offsets retain their fast path; dynamic-zone transition iteration advances strictly and exits at the first intersecting gap. No actionable allocation or scan-cost regression was substantiated.
  17. Further review: endpoint validity/types, scale reduction, historical/min/max bounds, renamed-helper references and both consumers were rechecked. No unresolved candidate or uncovered changed path remains.

Review converged after two rounds, including a final challenge of the frozen submission. Validation limits: source and test artifacts were inspected only; no build, test, benchmark or runtime validation was executed. The author's reported test runs and output-generation history were not independently verified.

Review converged after 2 round(s).

### Background

The nested-column pruning suite checks a runtime TopN filter on one STRUCT field while a predicate reads another field. The planner generates the filter only when max(estimated rows, 1) times topn_filter_ratio exceeds LIMIT plus OFFSET.

### What problem does this PR solve?

Issue Number: None
Related PR: None
PR Type: Test

Problem Summary:

The suite expects TOPN OPT:1, but ratio 0.5 suppresses the filter when the row count estimate is 1. Even with all four rows counted, the LIMIT 2 query does not use the filter. Use the large session ratio from existing TopN tests and check both limits. Keep all four input rows and the existing result checks.

### Details

- **Failure:** The EXPLAIN check can fail with estimated row count 1 even after ANALYZE WITH SYNC.
- **Cause:** The test depends on the row count available to the planner. The strict threshold rejects 1 * 0.5 > 1 and 4 * 0.5 > 2.
- **Stability evidence:** With an injected row count of 1, ratio 0.5 omitted the filter. Ratio 1000000 enabled it for limits 1 and 2 with the required nested access paths. The revised suite passed five consecutive runs without ANALYZE.

### Testing

All execution checks used the supplied local Doris cluster.

- Controlled row count 1: compared EXPLAIN at ratios 0.5 and 1000000 through mysql.
- ./run-regression-test.sh --run -d nereids_rules_p0/column_pruning -s topn_filter_nested_column_pruning -times 5: passed all five runs, each with two plan checks and 12 result queries.
- ./run-regression-test.sh --run -d datatype_p0/timestamptz -s test_timestamptz_cast_gap_partition_prune: passed.
- git diff --check: passed for the changed file.
- The original suite passed once before the change; the controlled row count reproduced the reported failure condition.

### Release note

None

### Check List (For Author)

- Behavior changed: No
- Does this need documentation: No
- Pre-submission status: Ready

### AI Disclosure

- Tool: Codex
- Generated: Test changes, comments, and this description.
- Human review: Pending.
@jacktengg

Copy link
Copy Markdown
Contributor Author

run buildall

1 similar comment
@jacktengg

Copy link
Copy Markdown
Contributor Author

run buildall

@jacktengg

Copy link
Copy Markdown
Contributor Author

run external

@jacktengg

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Your access token could not be refreshed because your refresh token was already used. Please log out and sign in again.
Workflow run: https://github.com/apache/doris/actions/runs/43726221612

This credential version is excluded from future reviews. Replace its refresh token to restore eligibility. Please trigger /review again; another configured account may be available.

@jacktengg

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all five changed files at head dc4506d. No new actionable findings or existing inline blockers were identified. No additional user review focus was supplied. The review converged after one round with both complete reviewers and the separate conversion-risk reviewer reporting NO_NEW_VALUABLE_FINDINGS.

The reported failure follows the reduced plan Filter(CAST(dt AS TIMESTAMPTZ) > constant) -> Scan(partitioned dt). The new DATETIMEV2 dispatch checks local clock gaps before inferring a range from converted endpoints. Monotonic.inferRange returns no range when the guard rejects an interval, retaining its partition; runtime-filter partition classification consults the same guard.

Critical checkpoint conclusions:

  1. Goal and evidence: the implementation addresses the reported missing row across a spring-forward gap. Added unit cases and the one-row SQL regression directly cover that scenario; this is a static assessment of their coverage.
  2. Scope and reuse: the production change is small and reuses the existing transition scanner. The separate TopN regression adjustment reliably enables filter coverage for LIMIT 1 and LIMIT 2 when estimates are small.
  3. Concurrency and locking: no new threads, shared mutable state, locks or lock ordering; timezone lookup follows the existing session context.
  4. Lifecycle and initialization: no new production ownership or static-initialization dependency. The unit fixture restores ConnectContext, and the new regression restores timezone in finally.
  5. Configuration: no new configuration definitions or dynamic-update obligations. TopN uses an existing session setting within its suite connections.
  6. Compatibility: no persisted format, wire format or SQL-function registration change. The internal helper rename is consistent with its callers.
  7. Thrift/protobuf: unchanged.
  8. Parallel paths: Cast and inherited TryCast share the guard in range inference and runtime-filter classification. TIMESTAMP_NS rounding, inverse-cast checks and the fixed-offset fast path are preserved. CheckCast's supported local source types match the dispatch; inverse-predicate families do not bypass it.
  9. Special conditions and errors: dynamic-zone missing bounds conservatively decline inference. Typed producers supply paired endpoint types, and current RANGE parsers reject zero month/day bounds; implicit MIN is 0000-01-01. Rounding a wholly pre-gap interval can reach the exact integer-second gap start without advancing inside the gap. Non-hour transitions use their actual before/after bounds. No reachable introduced exception or unsafe new precision case was substantiated.
  10. Test coverage: added assertions cover intersecting gaps, exact boundaries, fractional wrap, safe neighboring/fall-back ranges and fixed-offset/unbounded cases. The regression checks the previously missing interior row with ordered output. TopN explain checks cover both limits and nested access paths. Existing inverse-cast/TIMESTAMP_NS tests remain. No tests were run during this review.
  11. Expected results: the new row maps on BE to 07:00:00.900000 UTC, displays as 03:00:00.900000-04:00, and exceeds the predicate threshold. The new saved result is consistent with that conversion. The SQL comment's 07:40 value is a nonfunctional prose typo. Existing TopN expected results remain consistent with its four inserted rows.
  12. Observability: this local classification guard introduces no new distributed behavior requiring logs or metrics; existing partition-plan inspection remains adequate.
  13. Transactions, persistence and failover: no production changes to these paths.
  14. Data writes, atomicity and crash recovery: no production write-path changes.
  15. FE/BE variables: no transmitted state or send-path changes.
  16. Performance and memory: fixed offsets retain constant-time handling; dynamic zones reuse advancing transition iteration with early gap rejection. The larger TopN ratio is test-only. No actionable new allocation or CPU issue was identified.
  17. Further review: rechecked endpoint validity/subtypes, scale reduction, historical/non-hour boundaries, renamed references, nullable/unbounded propagation, both pruning consumers, all changed tests/output and the final finding set. Both risk focuses have concrete dispositions; no unresolved correctness candidate or uncovered changed path remains.

Validation limits: static source and saved test-artifact inspection only. No builds, tests, benchmarks or runtime checks were executed, and the author's reported execution/output-generation history was not independently verified.

Review converged after 1 round(s).

@jacktengg
jacktengg merged commit ec11929 into apache:master Nov 16, 2026
33 of 34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants