Repository navigation
[fix](cache) Invalidate query caches when replaying table changes - #63612
Conversation
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
FE UT Coverage ReportIncrement line coverage |
TPC-H: Total hot run time: 32203 ms |
TPC-DS: Total hot run time: 172962 ms |
FE Regression Coverage ReportIncrement line coverage |
|
run buildall |
FE UT Coverage ReportIncrement line coverage |
TPC-H: Total hot run time: 32080 ms |
FE Regression Coverage ReportIncrement line coverage |
TPC-DS: Total hot run time: 173563 ms |
|
/review |
There was a problem hiding this comment.
Requesting changes for a failover correctness gap in the new table-meta-change journal signal.
Critical checkpoint conclusions:
- Goal/test: The PR targets follower FE cache invalidation after table metadata replay. The approach covers the normal happy path, but no automated regression or failover test is added, and the implementation is not safe across master failover.
- Scope: The change is focused, but it introduces a separate persistent operation for a transient cache side effect, which makes ordering/atomicity part of correctness.
- Concurrency/lifecycle: No new threads; cache invalidation can race with query serving as before, but the larger issue is journal ordering across crash/failover.
- Configuration/compatibility: No new config. A new edit-log opcode is added and read/write dispatch exists.
- Parallel paths: The generic replay signal avoids updating many replay paths, but because it is not part of those metadata logs, any crash before the follow-up signal loses the invalidation.
- Transaction/persistence: Blocking issue found: metadata and invalidation are persisted independently, so master failover between them can replay the metadata without ever replaying the invalidation.
- Tests: No automated test covers follower replay invalidation or the failover window.
- Observability/performance: No blocking observability or performance issue found for the current cache sizes.
User focus: No additional user-provided review focus was supplied.
| return; | ||
| } | ||
| TableMetaChange change = | ||
| TableMetaChange.fromTable(table); |
There was a problem hiding this comment.
This makes the invalidation durable only as a second edit-log record after the real metadata mutation has already been logged by the caller. If the master crashes after the metadata log is committed but before this OP_TABLE_META_CHANGE is committed, the promoted FE replays the schema/property change but never replays this invalidation, so its existing SQL cache can keep serving stale results indefinitely. For example, ALTER TABLE ... RENAME COLUMN logs the rename in the alter path, then reaches notifyTableMetaChange; a crash in this window recreates the original follower-cache bug this PR is fixing. The invalidation needs to be tied to replaying the metadata operation itself, or otherwise made atomic with the metadata journal entry, rather than relying on a best-effort follow-up journal record.
There was a problem hiding this comment.
Added a complementary fix in the latest commit: store OlapTable.getBaseSchemaVersion() in the cache entry and check it on lookup (tablesOrDataChanged). This provides a crash-window-free safety net for OlapTable schema DDLs (rename/add/drop column) — if the schema version changed, the cache entry is invalidated on next access regardless of whether OP_TABLE_META_CHANGE was received.
Now there are two layers:
- Layer 2 (schemaVersion, lazy): no crash window, covers OlapTable schema changes automatically
- Layer 3 (OP_TABLE_META_CHANGE, eager): covers all table types and non-schema DDLs (alter properties, etc.), has a tiny crash window
The crash window in Layer 3 only matters for non-OlapTable or non-schema DDLs — and for those, the existing tablesOrDataChanged checks (visibleVersion, table id, etc.) already catch most cases.
|
run buildall |
…replay (apache#63612) Cherry-pick of apache#63612 to branch-4.1. Two-layer fix: 1. Check OlapTable.baseSchemaVersion on cache lookup (lazy, no crash window) 2. OP_TABLE_META_CHANGE journal broadcast (eager, all table types) Add enable_write_op_table_meta_change FE config (default false) to gate the OP_TABLE_META_CHANGE edit log writing. When false, master still does local cache invalidation but does not broadcast to followers via journal. The schemaVersion check still works regardless of this config.
…replay (apache#63612) Cherry-pick of apache#63612 to branch-4.0. Two-layer fix: 1. Check OlapTable.baseSchemaVersion on cache lookup (lazy, no crash window) 2. OP_TABLE_META_CHANGE journal broadcast (eager, all table types) Add enable_write_op_table_meta_change FE config (default false) to gate the OP_TABLE_META_CHANGE edit log writing. When false, master still does local cache invalidation but does not broadcast to followers via journal. The schemaVersion check still works regardless of this config.
FE Regression Coverage ReportIncrement line coverage |
TPC-H: Total hot run time: 31993 ms |
TPC-DS: Total hot run time: 172625 ms |
There was a problem hiding this comment.
Reviewed all 43 changed files and the relevant mutation/replay, cache publication/lookup, external-catalog and MTMV interactions at head 7310857. One new P2 performance regression is reported inline: unchanged partition-aware cache hits allocate partition-sized temporary storage before their version fast return. No new data-correctness defect or confirmed existing P0/P1 blocker remains. Review converged after two rounds; all three reviewers reported NO_NEW_VALUABLE_FINDINGS in the latest round, and every candidate and initial risk has an evidence-based disposition.
Prior context: catalog-retirement P1 #4480838554 is fixed by named retirement through onCatalogChanged; catalog/MV lock-cycle P1 #4488328455 is fixed by copying dependency names before external resolution; transaction-scan P2 #4489252243 is fixed by retaining lazy dependency validation. Earlier second-journal crash-window/version-skew concerns refer to the superseded implementation. The new inline addresses a distinct per-hit allocation. No additional user focus was supplied.
Critical checkpoint conclusions:
- Goal and proof: existing metadata journals now invalidate local SQL, sorted-partition, MTMV rewrite and exact aggregate caches; shared admission/lookup checks fence stale candidates. Serialized replay and race tests were inspected, with runtime success unverified in this static review.
- Scope and reuse: changes are focused on derived-cache invalidation and dependency validation, with common Env fanout and one shared validator.
- Concurrency and locks: lookup retains ID-sorted table read locks; MV builds and external dependency resolution occur outside MV locks. Admission releases its own metadata locks around cloud version IO. Native VISIBLE publication retains partition-before-table updates without scanning cached contexts under write locks. No additional concrete lock cycle was established.
- Lifecycle and initialization: snapshots retain their revision cells without registering executing contexts. Temporary admission scopes close independently and the executing statement is restored in finally; prepared executions get fresh snapshots. The ID fallback no longer retains a connection. No new static initialization dependency was found.
- Configuration: no new production setting; the regression restores its modified FE configuration in finally.
- Compatibility: no new journal opcode, persisted payload or storage format. Existing journals receive direct replay invalidation; expanded ephemeral BE cache keys produce safe misses against old entries.
- Thrift/protobuf: no definitions changed; field-ID comparison is not applicable.
- Parallel paths: FE/BE admission, live/replay metadata, cloud/native restore, warm/cold external refresh and catalog retirement were traced. The existing cloud source of visible versions is preserved.
- Special conditions: checkpoint skips, cold name-only notifications, legacy internal CREATE payloads, row-TTL rejection and table-wide dependency fallbacks are consistent with their inspected lifecycle.
- Test coverage: inspected metadata/schema replay, renamed/replaced identities, unaffected partitions/tables, COMMITTED versus VISIBLE, late puts, scope cleanup and error containment. The multi-FE regression skips without a follower/observer; it explicitly warms drop/rename cases, while view/truncate cases do not assert their own prior cache admission.
- Expected results: the new output agrees with the SQL projection, insert, view arithmetic and truncate sequence. Output generation provenance cannot be established statically.
- Observability: invalidation/admission failures log warnings with cache/query context; normal hits add no INFO logging. No missing critical metric was identified for these existing cache paths.
- Persistence and failover: fanout follows existing replayed mutations without a second-record crash window; checkpoint replay skips disposable local caches. Original transaction publication semantics are retained.
- Data writes and core invariants: no storage write, delete-bitmap or native-memory implementation changes. Identity/schema/data checks reject stale cache reuse; original metadata mutation locks and restore finish-log/showState ordering remain intact.
- FE/BE passing: captured schema/base-index or partition identity contributes to the existing context MD5 shared by BE fetch/update; no new wire variable is required.
- Performance: the inline identifies the avoidable O(tracked partitions) allocation on unchanged-version hits. Result-row limits do not bound scan-partition dependencies. No benchmark or measured allocation claim is made.
- Further recheck: all changed paths, prior threads, candidate dispositions, risk focuses and the inline anchor were swept after the second round. No unresolved production question or uncovered changed path remains.
Validation limits: source and test inspection only. No builds, tests, benchmarks, runtime reproduction, source edits or submodule initialization were performed. The test-admission and output-provenance limits above remain; they are not unreviewed production scope.
Review converged after 2 round(s).
### Background Partition-aware SQL cache admission and lookup compare metadata and table versions before revalidating individual partition dependencies. Unchanged-version hits can return without building a list of those partitions. ### What problem does this PR solve? Issue Number: None Related PR: apache#63612 PR Type: Bugfix Problem Summary: Splitting the metadata read locks moved construction of the partition-sized ArrayList before the unchanged-version fast return. Every repeated hit could therefore allocate storage proportional to tracked partitions, even for a one-row aggregate. Move the existing local initialization after the metadata and unchanged-version checks while retaining the locking and validation flow. ### Details - Frequency: Every partition-aware unchanged-version SQL cache validation. - Root cause: The local list was initialized before all early returns rather than only on the revalidation path. - Why existing tests missed it: Existing semantic tests validate cache freshness and reuse; they do not measure temporary allocations. - Fix: Declare the invocation-local list before the first read lock and initialize it after both early-return checks inside that lock. Keep the finally unlocks, Cloud IO between metadata lock sections, partition identity/schema recheck and validated version update. - Other paths and branches: Native/Cloud admission and lookup share this helper. No new state, API, cache key, journal, executing-context tracking or protocol field. ### Testing - ./run-fe-ut.sh --run with six focused FE classes: 76 cases, zero failures/errors/skips. Includes 14 native/Cloud version cases and 27 cache/replay cases. - ./build.sh --fe: success, including integrated Checkstyle with zero violations. - git diff --check: passed. - Before/after JDK17 compiler bytecode: the original partition ArrayList allocation precedes the unchanged-version return; the repaired fast return skips that construction. JDK ArrayList(int) bytecode confirms positive capacity constructs Object[]. This is control-flow evidence, not a heap/GC/throughput measurement. - Reused existing semantic coverage; no new implementation-shaped test, benchmark, full FE UT, remote regression or BE/Cloud runtime run for this follow-up. ### Release note Reduce temporary partition storage allocation on unchanged-version SQL cache hits. ### Check List (For Author) - Test: Focused unit tests, FE build/style and compiler artifact inspection above. - Behavior changed: Yes. Unchanged-version validation skips the unnecessary partition-list allocation. - Does this need documentation: No - Pre-submission status: Partially verified: local focused validation passes; new-head remote CI remains separate. ### AI Disclosure - Tool: Codex, gpt-6.1-sol - Generated: Code and commit description. - Human review: Maintainer review pending.
|
run buildall |
|
/review |
Local pipeline review — ✅ PASSschema: doris-repo-review/v1
status: PASS
pr: apache/doris#63612
commit: 77a128ed5c2b710bd19834e5201d136d438d4040
base: b6c67510385391d0dd913589f1b323ec86ba7bd6
reviewed_at: 2026-11-06T17:23+08:00
reviewer: 924060929
model: gpt-6.1-sol
effort: xhigh
findings: {blocker: 0, major: 0, minor: 0, nit: 0}
rounds: 2
converged: trueNotes for maintainers
Reviewed locally with the |
TPC-H: Total hot run time: 28679 ms |
FE UT Coverage ReportIncrement line coverage |
TPC-DS: Total hot run time: 154062 ms |
ClickBench: Total hot run time: 24.19 s |
|
Codex automated review failed and did not complete. Error: Review recovery stopped: shared time budget or helper timeout exhausted Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
### What problem does this PR solve? Related PR: apache#63612 Problem Summary: With a one-entry external database object cache, constructing an MV plan that references two databases evicts a database object. The eviction was reported as a metadata change, invalidating the MV generation and restarting every successful build indefinitely. Separate automatic local-object retirement from explicit metadata invalidation while retaining typed engine cleanup and row-count fencing. Bound rewrite-cache construction to four attempts so sustained metadata changes fall back through the callers' existing optional-optimization exception handling. ### Release note Avoid repeated query planning caused by external database object cache eviction during materialized view rewriting. ### Check List (For Author) - Test: Unit Test - 192 focused FE tests passed with zero failures, errors or skips; FE build and integrated Checkstyle passed. The unfixed 4.1 reproduction had 6 failures among 11 tests. - Behavior changed: Yes - automatic database object eviction no longer reports a semantic metadata change, and unstable optional MV cache construction terminates after four attempts. - Does this need documentation: No
### What problem does this PR solve? Related PR: apache#63612 Problem Summary: With a one-entry external database object cache, constructing an MV plan that references two databases evicts a database object. The eviction was reported as a metadata change, invalidating the MV generation and restarting every successful build indefinitely. Separate automatic local-object retirement from explicit metadata invalidation while retaining typed engine cleanup and row-count fencing. Bound rewrite-cache construction to four attempts so sustained metadata changes fall back through the callers' existing optional-optimization exception handling. ### Release note Avoid repeated query planning caused by external database object cache eviction during materialized view rewriting. ### Check List (For Author) - Test: Unit Test - 192 focused FE tests passed with zero failures, errors or skips; FE build and integrated Checkstyle passed. The unfixed 4.1 reproduction had 6 failures among 11 tests. - Behavior changed: Yes - automatic database object eviction no longer reports a semantic metadata change, and unstable optional MV cache construction terminates after four attempts. - Does this need documentation: No
### What problem does this PR solve? Related PR: apache#63612 Problem Summary: The synchronous-MV cache regression warmed base-table queries before publishing rollups and expected their old SQL cache entries to survive. Rollup publication now notifies table metadata changes and correctly invalidates those entries. Share one check across the rename, insert and drop scenarios to verify cache misses after publication, then rerun each query and verify cache hits. Preserve the later rename, insert and drop assertions. ### Release note None ### Check List (For Author) - Test: 40 focused FE unit tests passed; regression framework and modified Groovy suite compiled; FE build with integrated Checkstyle passed. End-to-end regression was not run locally because this worktree has no BE binary. - Behavior changed: No production behavior change; update regression expectations for metadata invalidation. - Does this need documentation: No
### What problem does this PR solve? Related PR: apache#63612 Problem Summary: The synchronous-MV cache regression warmed base-table queries before publishing rollups and expected their old SQL cache entries to survive. Rollup publication now notifies table metadata changes and correctly invalidates those entries. Share one check across the rename, insert and drop scenarios to verify cache misses after publication, then rerun each query and verify cache hits. Preserve the later rename, insert and drop assertions. ### Release note None ### Check List (For Author) - Test: 40 focused FE unit tests passed; regression framework and modified Groovy suite compiled; FE build with integrated Checkstyle passed. End-to-end regression was not run locally because this worktree has no BE binary. - Behavior changed: No production behavior change; update regression expectations for metadata invalidation. - Does this need documentation: No
What problem does this PR solve?
Problem Summary:
A query cached on a follower or observer can keep returning an old schema after a table change. For example, a heavy
ALTER TABLE ... DROP COLUMNswitches the base index without changing the table's data visible version. The master's ALTER path clears its local SQL cache, while another FE replays the schema change and retains the old cached result.This change invalidates derived query caches while applying existing metadata journals. Ordinary DML publishes table and partition visible versions, which SQL cache admission and lookup validate without scanning the cache under transaction table write locks. It replaces the earlier implementation of this PR that emitted a separate metadata-change journal.
Implementation
Env.onTableChangedis the shared entry point. Every cache invalidation runs in its owntry/catch, logs a failure, and allows the remaining cache invalidations to run. Checkpoint replay skips these local derived caches.The hook runs at the existing mutation boundary: schema-change/rollup completion, table drop/replacement, and truncate. Transaction visibility retains its original partition-before-table version publication; data validation is performed when admitting or looking up a cache entry. Remaining replay handlers notify it after applying their mutation. Renames and replacements notify after applying the name mapping, preserving the original names and table IDs under the existing locks. Batch partition changes notify each affected table once.
Replay coverage includes table/view creation and removal, recovery, column/table/partition/rollup renames, heavy/light schema and index changes, partition changes/replacement, table replacement/engine/properties/distribution changes, administrative version changes, constraints, MTMV changes, branch/tag operations, and database/catalog changes. External refreshes reuse their existing invalidation protocol and retain the row-count cache's final invalidation barrier.
SQL cache publication and lookup share dependency validation: local metadata revisions, table identity/schema/data versions, view definitions, and external catalog properties. Revisions are scoped by table ID/name, database and catalog; each statement retains its own dependency snapshot, with no registry of executing contexts. Publication uses a fresh statement and connector scope, restores the executing statement in finally, and skips cache admission if validation fails. Lookup also rejects a late put that races invalidation. The BE cache key includes the base index/schema identity, and MTMV rewrite invalidation reuses the existing generation barrier.
Partition-version validation constructs temporary partition storage only after the metadata and unchanged-version fast checks.
No journal opcode, extra journal record, persisted payload, or FE/BE protocol field is added. Approximate statistics and transaction-scoped auto-partition routing retain their existing lifecycles.
Release note
Fix stale cached query results on follower and observer FEs after table metadata changes.
Check List (For Author)
build.sh --feand integrated Checkstyle pass. Includes native/Cloud partition-version checks, actual cache lookup, serialized transaction replay, FE/BE publication, metadata revisions and data policy. Before/after compiler and JDK bytecode confirms the unchanged-version branch skips partition-list construction; no heap/GC/throughput benchmark is claimed. Earlier broader and multi-FE validation is historical and was not rerun for this allocation-only follow-up.Earlier local multi-FE validation used the then-patched master FE with an existing 4.1.3 BE. The isolated cluster sets
be_exec_version=10and disables the newer local shuffle planner/local exchange to accommodate that BE. This is not a same-head FE/BE build or a Cloud cluster validation.