Repository navigation
[feat](join) support outer join reorder in dphyper - #61146
Merged
Merged
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 28409 ms |
TPC-DS: Total hot run time: 151599 ms |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 27382 ms |
TPC-DS: Total hot run time: 152511 ms |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 27761 ms |
TPC-DS: Total hot run time: 152906 ms |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 26927 ms |
TPC-DS: Total hot run time: 167112 ms |
Contributor
Author
|
run buildall |
Contributor
FE UT Coverage ReportIncrement line coverage |
TPC-H: Total hot run time: 26865 ms |
TPC-DS: Total hot run time: 167837 ms |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 26922 ms |
TPC-DS: Total hot run time: 168082 ms |
Contributor
Author
|
run buildall |
morrySnow
approved these changes
May 7, 2026
zhaorongsheng
pushed a commit
to zhaorongsheng/doris
that referenced
this pull request
Jun 4, 2026
16 tasks
morrySnow
pushed a commit
that referenced
this pull request
Jun 16, 2026
…der to merge consecutive projects (#64409) Related PR: (#61146) After DPHyp join reorder, the ColumnPruning rule may produce consecutive Project nodes in the plan tree. Subsequent optimization rules expect normalized plan shapes and may not handle chains of consecutive projects correctly, leading to plan corruption or incorrect results. This is because the DPHyp reorder path runs a separate rewrite pipeline (pushDownRewrite → columnPrune) on the reordered plan before re-inserting it into the memo. Unlike the main rewrite pipeline which includes MergeProjectable in its standard rule sequence, the DPHyp post-reorder pipeline omitted this cleanup step. Fix: Add MergeProjectable after ColumnPruning in the DPHyp rewrite pipeline within Optimizer.dpHypOptimize(). This ensures that any consecutive Project nodes generated by column pruning are merged into a single project, maintaining a normalized plan shape for downstream rules.
16 tasks
morrySnow
pushed a commit
that referenced
this pull request
Aug 4, 2026
…es on nullable side of outer join (#65682) Related PR: #61146 #### Problem Summary: When enable_dphyp_optimizer=true, DPHyp join reorder can produce wrong results for queries where the nullable side of an outer join contains Alias/Project expressions with functions like COALESCE, IFNULL, or CAST(COALESCE(...)). #### Root Cause: HyperGraph.Builder.addAlias() unconditionally adds all aliases to aliasReplaceMap, including those defined on the nullable side of outer joins. This causes their defining expressions to be "unwrapped" and later reconstructed by PlanReceiver.proposeProject() ABOVE the outer join via finalProjects. The expression then operates on null-extended values, changing semantics — e.g., COALESCE(NULL, 0) = 0 instead of the correct NULL for non-matching outer join rows. #### Fix: Added an isNullableSide context flag that propagates through buildForDPhyper() and addAlias() during graph construction. When processing an outer join: LEFT OUTER JOIN → right child flagged as nullable RIGHT OUTER JOIN → left child flagged as nullable FULL OUTER JOIN → both children flagged as nullable The flag propagates through nested projects and inner joins Aliases on the nullable side are not added to aliasReplaceMap, preserving the original Project boundary below the outer join so that expressions execute before null-extension.
wyxxxcat
pushed a commit
to wyxxxcat/doris
that referenced
this pull request
Aug 17, 2026
…es on nullable side of outer join (apache#65682) Related PR: apache#61146 #### Problem Summary: When enable_dphyp_optimizer=true, DPHyp join reorder can produce wrong results for queries where the nullable side of an outer join contains Alias/Project expressions with functions like COALESCE, IFNULL, or CAST(COALESCE(...)). #### Root Cause: HyperGraph.Builder.addAlias() unconditionally adds all aliases to aliasReplaceMap, including those defined on the nullable side of outer joins. This causes their defining expressions to be "unwrapped" and later reconstructed by PlanReceiver.proposeProject() ABOVE the outer join via finalProjects. The expression then operates on null-extended values, changing semantics — e.g., COALESCE(NULL, 0) = 0 instead of the correct NULL for non-matching outer join rows. #### Fix: Added an isNullableSide context flag that propagates through buildForDPhyper() and addAlias() during graph construction. When processing an outer join: LEFT OUTER JOIN → right child flagged as nullable RIGHT OUTER JOIN → left child flagged as nullable FULL OUTER JOIN → both children flagged as nullable The flag propagates through nested projects and inner joins Aliases on the nullable side are not added to aliasReplaceMap, preserving the original Project boundary below the outer join so that expressions execute before null-extension.
8 tasks done
yiguolei
pushed a commit
that referenced
this pull request
Oct 20, 2026
) (#69308) ### What problem does this PR solve? Issue Number: DORIS-29796 Related PR: #62828 (source of the pick), #61146 (source of the dependency fix), #68211 Problem Summary: backport the CTE empty/cascading pruning fix #62828 to branch-4.2, plus the `ColumnPruning` CTE protection that #62828 implicitly depends on. Without it the three branch-4.2 RQG cases tracked by DORIS-29796 (6540158413 / DORIS-25384, 6540160970 / DORIS-25408, 6540160107 / DORIS-25409) keep failing. #### Why #62828 alone is not enough on branch-4.2 `ColumnPruning` on branch-4.2 only protects `LogicalCTEProducer` with `skipPruneThis` and has no `visitLogicalCTEAnchor`. When the required slots of an anchor do not include the producer outputs (the normal case, because consumers reference the producer slots directly), the producer subtree is pruned with an empty required set and `LogicalProject` materializes the empty projection as the dummy `1 AS 1` projection. The rewrite plan therefore becomes ``` LogicalCteAnchor |--LogicalProject ( projects=[1 AS 1] ) | +--LogicalCteProducer +--... ``` which (a) breaks the `anchor.left() instanceof LogicalCTEProducer` invariant that #62828's `normalizeCtePlan`/`refreshCteContext` relies on (`ClassCastException`), and (b) is an invalid CTE structure: the materialized columns no longer match the slot mapping the consumers were built from, which is what surfaces as `VSlotRef ... have invalid slot id` in case 6540160107. Master has had the protection since #61146 (`visitLogicalCTEAnchor` + `skipPruneThisAndFirstLevelChildren`); this PR ports exactly that hunk. #### Hunk audit of the #62828 pick (source commit `c75eba5488f`) | Source file | Disposition | | --- | --- | | `nereids/jobs/executor/Optimizer.java` | Ported verbatim (`normalizeCtePlan`, `refreshCteContext`, `containsCte`, `generateFilterPushedDownInlinedPlan`, method rename). | | `nereids/rules/rewrite/CTEInliner.java` | Ported verbatim (`inlineByCurrentConsumerCount`, `InlineResult`, zero-consumer anchor removal, recursive anchor trimming). | | `nereids/StatementContext.java` | Not applied: branch-4.2 already has `clearCteEnvironment()` from the multi-mode CTE inline work (#68211), and its variant also clears `consumerIdToLimitRows`. | | `nereids/rules/rewrite/ClearContextStatus.java` | Not applied: 4.2 keeps the explicit per-map clears, which are a superset of what the picked code needs. | | `qe/SessionVariable.java` | Not applied: 4.2 keeps `@VariableMgr.VarAttr(..., alias = "cbo_cte_inline_mode")`; the source commit only dropped the alias. | | `.../CTEInlineTest.java` | Ported verbatim, including the new `refreshCteConsumersAfterNormalizeEliminatesEmptyBranch`. | | `regression-test/suites/query_p0/cte/test_cbo_cte_inline_prune.groovy` | Ported verbatim, path adapted to the 4.2 layout (`suites/nereids_p0/cte/`; `suites/query_p0/cte` does not exist on 4.2). | | `regression-test/data/query_p0/...` baselines | Paths adapted to the 4.2 layout (`data/nereids_p0/repeat/`, `data/nereids_rules_p0/unique_function/`); content unchanged. | The dependency fix is a separate commit, ported from `5de1ed88d4b` (#61146) and limited to the `ColumnPruning` hunk (12 added lines): CTE anchors and producers are no longer column pruned and keep their first level children's full output requirement. No BE changes. #### Isolation matrix (measured: which commit fixes which case) FE was rebuilt with subsets of the two commits (BE is unaffected by both), and the three RQG cases were replayed each time: | build | 6540158413 | 6540160970 | 6540160107 | | --- | --- | --- | --- | | branch-4.2, neither commit | BE `Check failed: thrift_sink.multi_cast_stream_sink.sinks.size() > 0 (0 vs. 0)` -> query fails | same | `VSlotRef ... invalid slot id` | | #62828 only | 0 rows | 0 rows | `ClassCastException: LogicalProject cannot be cast to LogicalCTEProducer` | | ColumnPruning only | same BE CHECK as baseline | same | **0 rows** | | this PR (both) | 0 rows | 0 rows | 0 rows | The three failures therefore come from two independent branch-4.2 defects: the missing empty/zero-consumer CTE pruning (#62828) for 6540158413 / 6540160970, and the missing `ColumnPruning` CTE protection (#61146) for 6540160107. #62828 alone cannot fix the third case, and picking it without the pruning fix would turn that case into a `ClassCastException`. ### Release note None ### Check List (For Author) - Test - [x] FE unit test: `./run-fe-ut.sh --run org.apache.doris.nereids.rules.rewrite.CTEInlineTest` → `Tests run: 2, Failures: 0, Errors: 0` (includes the new `refreshCteConsumersAfterNormalizeEliminatesEmptyBranch`). - [x] FE build: `./build.sh --fe` → `BUILD SUCCESS` on this branch (checkstyle included). - [x] Original RQG cases replayed against a local 4.2 cluster (FE+BE built from this branch, BE alive), using the full DDL/INSERT/session settings from the DORIS-29796 attachment `repro-cte-empty-42.sql` (database renamed to a hyphen-free name so branch-4.2's parser accepts it; no other change): | case | expected | branch-4.2 before | this branch | | --- | --- | --- | --- | | 6540158413 | 0 rows | `size of sources must be greater than 0` | 0 rows, no error | | 6540160970 | 0 rows | `size of sources must be greater than 0` | 0 rows, no error | | 6540160107 | 0 rows | `VSlotRef col_int_undef_signed have invalid slot id: 15` | 0 rows, no error | With only the #62828 commit replayed, cases 6540158413/6540160970 return 0 rows and 6540160107 still fails (`ClassCastException: LogicalProject cannot be cast to LogicalCTEProducer`), which is why the `ColumnPruning` commit is part of this PR. - [x] Candidate 6540166225 from the DORIS-29796 evidence could not be replayed: the evidence attachment only carries its error text, not its SQL, so there is no input to verify locally. - [x] Regression (local 4.2 cluster, JDBC `127.0.0.1:29731`): - `-d nereids_p0/cte` → 9 suites, failed 0 (includes the new `test_cbo_cte_inline_prune`) - `-s test_repeat_output_slot` → pass (baseline updated by the pick) - `-s agg_with_unique_function` → pass (baseline updated by the pick) - `-d nereids_rules_p0/cte` → 3 suites, failed 0 - `-d nereids_rules_p0/cte_limit_pushdown` → 1 suite, failed 0 - `-d nereids_rules_p0/column_pruning` → 9 suites, failed 0 - Introspective analysis - [x] `normalizeCtePlan` iterates to a fixpoint instead of comparing `Plan.equals()`, because `LogicalCTEAnchor`/`LogicalSubQueryAlias` ignore children in `equals()`. - [x] Zero-consumer anchors are removed before memoization; anchors with an empty producer or consumers below `inline_cte_referenced_threshold` are inlined. - [x] CTE environment (`cteIdToProducer`, `cteIdToConsumers`, rewritten caches) is refreshed from the normalized plan so memo construction does not see stale CTE state.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)