Repository navigation
[fix](fe) Let MV dry runs and EXPLAIN REFRESH borrow the session's compute group - #70125
Conversation
…mpute group ### What problem does this PR solve? Issue Number: None Related PR: apache#69749, apache#62606 Problem Summary: In cloud mode, REFRESH MATERIALIZED VIEW ... INCREMENTAL WITH DRY RUN and EXPLAIN REFRESH MATERIALIZED VIEW of an MV that declares no compute_group planned (and the dry run executed) in the compute group the admin user resolves to, not in the group of the issuing session, where a manual REFRESH of the same MV runs. Both statements build an internal admin context with an empty cloud_cluster (MTMVPlanUtil.createMTMVContext), so it resolves admin's default compute group, else the first group admin may use. A manual REFRESH instead hands the session's group to its task (MTMVJobManager#refreshMTMV), which binds it when the MV declares none (MTMVTask#applyComputeGroup). apache#69749 made both statements honour a declared group; an MV that declares none still went to admin's group. Measured on master 856194e with a unit test in cloud mode (groups cg_of_admin and cg_of_session, the session on USE @cg_of_session, an MV that declares no group): the manual REFRESH runs in cg_of_session, the dry run and EXPLAIN REFRESH resolve to cg_of_admin. When the MV's workload_group exists only in the session's group, the dry run fails with "Can not find workload group ... in compute croup cg_of_admin" while the manual REFRESH succeeds; otherwise the delta runs on another group's backends, which the user may have no USAGE on, and can start that group if it is suspended. Results are not affected. The fix: - Move the private lookup of the session's group from MTMVJobManager, unchanged, to MTMVPlanUtil.getSessionComputeGroup(), so a manual REFRESH, a dry run and EXPLAIN REFRESH resolve it with the same code. - RefreshMTMVCommand builds both internal contexts through one helper that reads the session's group while the session is still the thread-local context, then binds the declared group, else the session's group, as MTMVTask#applyComputeGroup does. With neither, the context stays unbound and the existing resolution applies. - As in MTMVTask, the dry run re-checks only a declared group; EXPLAIN REFRESH checks nothing. branch-4.1 and branch-4.2 have neither statement. ### Release note In cloud mode, REFRESH MATERIALIZED VIEW ... WITH DRY RUN and EXPLAIN REFRESH MATERIALIZED VIEW of a materialized view that declares no compute_group run in the compute group of the current session, as a manual REFRESH does, instead of the group the admin user resolves to. ### Check List (For Author) - Test: Unit Test - RefreshMTMVCommandTest: 4 new cases in a mocked cloud mode, one case adjusted; the two cases on the session's group fail on master - 10 related test classes, 335 cases, pass - Behavior changed: Yes. In cloud mode, the dry run and EXPLAIN REFRESH of an MV that declares no compute group use the session's compute group instead of the one admin resolves to. - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
|
/review |
TPC-H: Total hot run time: 28635 ms |
TPC-DS: Total hot run time: 150947 ms |
ClickBench: Total hot run time: 24.08 s |
|
Codex automated review failed and did not complete. Error: Selected model is at capacity. Please try a different model. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
|
/review |
|
Codex automated review failed and did not complete. Error: Selected model is at capacity. Please try a different model. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
|
/review |
|
Codex automated review failed and did not complete. Error: Selected model is at capacity. Please try a different model. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
|
/review |
|
Codex automated review failed and did not complete. Error: Selected model is at capacity. Please try a different model. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
|
/review |
|
Codex automated review failed and did not complete. Error: Selected model is at capacity. Please try a different model. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
|
run cloud_p0 |
|
/review |
|
Codex automated review failed and did not complete. Error: Selected model is at capacity. Please try a different model. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
|
/review |
There was a problem hiding this comment.
Reviewed PR #70125 at exact head fe97758 against base 856194e. No verified inline findings. The change captures the caller's resolved session compute group before installing an internal MTMV context, preserves declared-MV precedence, and keeps manual refresh, dry-run, and EXPLAIN REFRESH aligned. Declared-group revalidation and thread-local cleanup remain consistent with MTMVTask and ExplainCommand. Critical checkpoints: the task goal is met by the inspected control flow and focused unit coverage; the change is small and focused; no new shared concurrency or lock path is introduced; context lifecycle and failure cleanup were traced; no configuration, protocol, persistence, transaction, or data-write format change is involved; tests cover declared, session, absent, dropped, and precedence cases, with existing manual-capture coverage. There were no existing inline threads and no additional user focus. This was a static review only; no builds or tests were executed. Review converged in round 1.
Review converged after 1 round(s).
924060929
left a comment
There was a problem hiding this comment.
In cloud mode, manual MV refresh already prefers the MV's declared compute group and otherwise borrows the session's group. This change applies the same selection to DRY RUN and EXPLAIN REFRESH before installing their internal admin context.
Independent static review of fe97758 against 856194e found no blocking issues in the affected FE paths. I traced context installation/restoration, declared-group validation, backend selection, workload-group lookup, and focused unit coverage. No FE/BE protocol or persistence changes are involved. Recommend APPROVE.
No builds or tests were run. The supplied current-head snapshot reports successful P0, cloud_p0, vault_p0, FE UT, and compilation checks, but FE coverage failed. Required-check rules were not supplied; this recommendation does not establish merge readiness.
Automated FE-focused static review (gpt-6.1-sol, high). No build or tests were run.
|
PR approved by anyone and no changes requested. |
|
PR approved by at least one committer and no changes requested. |
What problem does this PR solve?
Issue Number: None
Related PR: #69749, #62606
Problem Summary:
In short. In cloud mode,
REFRESH MATERIALIZED VIEW ... INCREMENTAL WITH DRY RUNandEXPLAIN REFRESH MATERIALIZED VIEWof an MV that declares nocompute_groupplan (and the dry run executes) in whatever compute group theadminuser resolves to, not in the group of the session that issues them, where a manualREFRESHof the same MV runs. This PR binds both to the session's group, the way a manual REFRESH borrows it.Background. Both statements build an internal context with the admin identity and the MV's session variables (
MTMVPlanUtil.createMTMVContext). Itscloud_clusteris empty, so it resolves a group the way admin would: admin'sdefault_compute_group, else the first group admin may use. A manual REFRESH passes the session's group to its task instead (MTMVJobManager#refreshMTMV), and the task binds it when the MV declares none (MTMVTask#applyComputeGroup). #69749 made both statements honour a declared group; an MV that declares none, which is every MV created before #69749 and any MV created without the property, still goes to admin's group. Found while reviewing #69749 (thread 4401555648).The problem, and what it cost. Measured on master
856194ece0ewith a unit test in cloud mode: groupscg_of_adminandcg_of_session, the session onUSE @cg_of_session, an MV that declares no group.cg_of_sessioncg_of_admincg_of_adminworkload_groupexists only in the session's group (a workload group belongs to one compute group in cloud mode), the dry run fails withCan not find workload group <wg> in compute croup cg_of_admin, while the manual REFRESH succeeds.How this PR fixes it.
MTMVJobManagermoves unchanged toMTMVPlanUtil.getSessionComputeGroup(), so a manual REFRESH, a dry run and EXPLAIN REFRESH resolve it with the same code.RefreshMTMVCommandbuilds both internal contexts through one helper. It reads the session's group first, while the session is still the thread-local context, then binds the declared group, else the session's group, asMTMVTask#applyComputeGroupdoes. With neither, the context stays unbound and the existing resolution applies; outside cloud mode nothing changes.MTMVTask, the dry run re-checks only a declared group, since the MV never chose the session's group. EXPLAIN REFRESH executes nothing and still checks nothing.Results.
USE @cg_of_sessioncg_of_admincg_of_session, as a manual REFRESHcg_declaredUSE @cg_of_sessioncg_declaredbranch-4.1 and branch-4.2 have neither statement.
Release note
In cloud mode,
REFRESH MATERIALIZED VIEW ... WITH DRY RUNandEXPLAIN REFRESH MATERIALIZED VIEWof a materialized view that declares nocompute_grouprun in the compute group of the current session, as a manualREFRESHdoes, instead of the group the admin user resolves to.Check List (For Author)
Test
RefreshMTMVCommandTest(mocked cloud mode) gets 4 cases: without a declaration, the dry run and the EXPLAIN contexts resolve to the session's group; a declared group still wins over it; the dry run does not re-check a borrowed group; and an existing case now runs from a session that resolves no group, where both contexts stay unbound. On master the first two fail (expected: <cg_of_session> but was: <cg_of_admin>). Together with the classes that call the changed code (ExplainRefreshMTMVCommandTest,MTMVJobManagerTest,MTMVPlanUtilTest,MTMVTaskTest,MTMVComputeGroupTest,IvmIncrRefreshMTMVTest,IvmBaselineRebuildTest,ProtocolCapabilityWiringTest,NereidsParserTest), 10 classes and 335 cases pass. Each part of the fix, broken on its own, fails its case: reading the session's group after the internal context is installed, letting the session's group win over a declared one, re-checking the borrowed group, and failing when the session resolves no group. Not run on a cloud cluster: the dry run needs IVM and changelog support in the backends, so a local cloud cluster needs a master backend image; group selection and the workload group lookup both happen in the FE, which the cases exercise with the real code.Behavior changed:
In cloud mode, the dry run and EXPLAIN REFRESH of an MV that declares no compute group use the session's compute group instead of the one admin resolves to.
Does this need documentation?
Check List (For Reviewer who merge this PR)