Repository navigation
Conversation
| * The checked input must also be the input used by the rewrite. | ||
| */ | ||
| private RelNode prepareCorrelates(RelNode root) { | ||
| final RelNode prepared = root.accept(new RelHomogeneousShuttle() { |
There was a problem hiding this comment.
This shuttle looks important enough to be in its own class with proper javadoc.
There was a problem hiding this comment.
Thanks. I've removed the preparation shuttle. The scalar rules and general rewrite now trim their right inputs locally.
| RelNode right = correlate.getRight(); | ||
| if (!isDeterministic(right)) { | ||
| final RelBuilder trimBuilder = relBuilderFactory().create(right.getCluster(), null); | ||
| final RelNode trimmed = new RelFieldTrimmer(null, trimBuilder).trim(right); |
There was a problem hiding this comment.
Is this the best way to do it?
How about trimming fields in a global pass before you even start the decorrelation?
There was a problem hiding this comment.
Thanks. I tried global trimming before decorrelation, but it removed a key needed by the scalar uniqueness check. I've kept trimming local to the RHS to preserve it.
There was a problem hiding this comment.
This sounds like a bug in the trimmer. Maybe you need to file an issue?
There was a problem hiding this comment.
Sorry, my earlier explanation was incomplete: Using RelFieldTrimmer.trim on the whole plan before decorrelation can remove a key that is unused by the query but needed by the scalar uniqueness check. trimPreservingBindings leaves the Correlate's left input unchanged, so that key is preserved.
However, applying trimPreservingBindings globally can remove the Project below a SINGLE_VALUE aggregate, preventing the scalar rule from eliminating that aggregate. I'd prefer to keep trimming local in this PR. Does that sound good?
There was a problem hiding this comment.
I am not sure I understand your explanation. If the trimmer converts valid queries to valid queries, there is no trimmer bug. Bug you are saying: "a key that is unused by the query but needed by the scalar uniqueness check". This suggests that there are valid queries that the "scalar uniqueness check" cannot correctly analyze.
In principle any analysis should be able to handle any well-formed query; if it cannot, it's a bug in that analysis.
These comments are not really related to your PR, which I haven't yet re-reviewed.
There was a problem hiding this comment.
Thanks for pointing this out. I should have been clearer: the query result is unchanged, but we lose the opportunity to apply this scalar rewrite. Running ordinary RelFieldTrimmer.trim first can remove an unused key and leave the outer rows non-unique. The uniqueness check then skips the rewrite as expected.
| return rel; | ||
| } | ||
| // Trimming can also remove references to the correlation variable. | ||
| final ImmutableBitSet requiredColumns = |
There was a problem hiding this comment.
should this functionality be part of the trimmer?
There was a problem hiding this comment.
Thanks. The cleanup now uses RelFieldTrimmer.trimPreservingBindings, keeping binding inputs intact so references need no remapping. CorrelateProjectExtractor updates dependencies after extraction; default trimming is unchanged.
| .build(); | ||
| if (!isDeterministic(projected)) { | ||
| // Only values and row-producing operations needed by the parameter | ||
| // domain are repeated. Unused volatile projections need not prevent |
There was a problem hiding this comment.
does "volatile" mean "non-deterministic"?
There was a problem hiding this comment.
Yes, I've corrected the wording.
| * Ordinary table scans assume stable contents within a query; unknown | ||
| * operators are not assumed deterministic. | ||
| */ | ||
| private boolean isDeterministic(RelNode rel) { |
There was a problem hiding this comment.
This looks like a very useful function. This should be public someplace so other passes can reuse it.
There was a problem hiding this comment.
But again, as we discussed elsewhere, there are two kinds of determinism: RAND() and NOW(). The latter can actually be duplicated. That problem still hasn't been fixed.
There was a problem hiding this comment.
Thanks, I've exposed it as RelOptUtil.isDeterministic(RelNode). It follows the existing SqlOperator.isDeterministic() declarations: query-stable CURRENT_TIMESTAMP is allowed, while RAND() is rejected.
| || rel instanceof Snapshot | ||
| || rel instanceof Collect | ||
| || rel instanceof Uncollect)) { | ||
| return false; |
There was a problem hiding this comment.
this is a strange way to do it.
I would think that these all return true, and the default is 'false'. It's easier to understand.
There was a problem hiding this comment.
Thanks, changed as suggested.
| !if (use_old_decorr) { | ||
| EnumerableValues(tuples=[[]]) | ||
| EnumerableCalc(expr#0..1=[{inputs}], SAL=[$t0]) | ||
| EnumerableCorrelate(correlation=[$cor0], joinType=[inner], requiredColumns=[{}]) |
There was a problem hiding this comment.
can you explain this change?
There was a problem hiding this comment.
Thanks for reviewing. That plan change was unintended. I've removed the global trimming pass and restored the original empty Values plan.
4904722 to
d20108b
Compare
mihaibudiu
left a comment
There was a problem hiding this comment.
I only left a few superficial comments.
This is a very complicated change. It could benefit from a high-level description of the strategy employed. The way I imagine I may approach this is by rewriting queries to segregate non-deterministic computations only to appear in some specific query operators (e.g., project) to simplify the job of the decorrelator.
|
|
||
|
|
||
| !if (use_old_decorr) { | ||
| # Test case for [CALCITE-7914] Correlation keys must use the values already |
There was a problem hiding this comment.
Typically [CALCITE-7914] is always followed by the issue title.
The description of the test can come next.
There was a problem hiding this comment.
please describe how the results were validated.
There was a problem hiding this comment.
Thanks, I've updated the test comments. I've verified the results in postgresql https://onecompiler.com/postgresql/458u9g9gc
| * deterministic according to their operator declarations. | ||
| * | ||
| * <p>Assumes parameters, correlation bindings, query context and table contents | ||
| * stay fixed within the query. Function calls use |
There was a problem hiding this comment.
what does it mean "they stay fixed"?
There was a problem hiding this comment.
Thanks. I've removed that wording. The definition now says that a deterministic call returns the same result for the same operands within one query execution.
| * | ||
| * <p>Input order must also be stable where it affects the result. This method | ||
| * does not establish that property, for example for an unordered FETCH. | ||
| * It does not trim fields or modify the expression. |
There was a problem hiding this comment.
why would it modify the expression? It is supposed to be a read-only analysis.
There was a problem hiding this comment.
Thanks. The sentence is redundant. I've removed it.
| } | ||
|
|
||
| /** Returns whether a relational expression, its inputs and sub-queries are | ||
| * deterministic according to their operator declarations. |
There was a problem hiding this comment.
well, you need to describe what non-deterministic means, or @link to its definition.
We agreed there are multiple possible definitions.
There was a problem hiding this comment.
Thanks. I've added the definition and a link to SqlOperator.isDeterministic().
| return false; | ||
| } | ||
|
|
||
| /** Visits a supported node's expression fields for read-only analysis. |
There was a problem hiding this comment.
applies a Shuttle to the expression fields of a Rel node (but not to its inputs)?
There was a problem hiding this comment.
Yes. It visits the node's expression fields, but not its relational inputs. The returned expressions are not applied to the node. I've clarified the Javadoc.
| if (other instanceof Correlate | ||
| && ((Correlate) other).getCorrelationId().equals(corrId)) { | ||
| return other; | ||
| /** Applies the visitor within the specified correlation scope. */ |
There was a problem hiding this comment.
is this notion of a "correlation scope" defined?
If not, maybe you can define it here.
There was a problem hiding this comment.
Thanks. I've defined the scope here: the part of root where id refers to a binding outside root. References to nested bindings with the same id are excluded.
|
|
||
| /** | ||
| * A collector of simply correlated row expressions. | ||
| /** Selects outermost deterministic computations over one correlation. |
There was a problem hiding this comment.
what happens to the selected ones? How are they represented?
There was a problem hiding this comment.
Thanks. Each selected expression is stored in a Map<RexNode, Integer>. The integer is its offset among the fields added to the left input. A Project computes those fields. Selected occurrences in the right input are replaced with RexFieldAccess references to the new fields. I've documented this in the Javadoc.
|
|
||
| import java.util.Set; | ||
|
|
||
| /** Conservatively analyzes references to correlation bindings outside a |
There was a problem hiding this comment.
what is the result of the analysis?
There was a problem hiding this comment.
Thanks. The analysis records whether outer bindings may be referenced and which declared fields may still be needed. The class Javadoc now states these results; each method documents its own return value.
| } | ||
|
|
||
| /** Trims the right input, preserving its output fields and the left input. | ||
| * Submits an equivalent Correlate if the plan changes and the trimmed right |
There was a problem hiding this comment.
where does it submit the correlate?
There was a problem hiding this comment.
Thanks. It goes to the rule's planner via RelOptRuleCall.transformTo. I've added that to the Javadoc.
| final Frame frame = dispatcher.invoke(r, isCorVarDefined, parentPropagatesNullValues); | ||
| final Frame frame; | ||
| if (!(r instanceof Correlate) && !r.getVariablesSet().isEmpty()) { | ||
| // Local bindings need their own field remapping. Only the Correlate |
There was a problem hiding this comment.
I don't see how this comment relates to the code.
There was a problem hiding this comment.
The guard skips nodes that define correlation variables, except Correlate. Decorrelation can change the positions of their input fields. References through those variables are not remapped, so they could still use the old positions. Returning null prevents this rewrite. I've made the comment clearer.
d5ddd67 to
d275c16
Compare
Thanks for the review. I've added a strategy overview to the PR description. I tried a small prototype based on your suggestion, covering Filter and INNER JOIN conditions. It let the determinism scan skip those operators' conditions, but added a normalization pass and still needed the shared expression visitor for correlation analysis. I haven't found an overall simplification in this prototype, so I'd lean toward keeping the current approach for this PR. Curious what you think. |
|
Since I am not an expert in the decorrelator implementation I have used an agent to perform a review. I am trying to understand the feedback and post it here. |
|
The agent review complains about the fact that there are some query operators which are also in essence non-deterministic, but which are not functions, e.g., LIMIT without ORDER BY. But I think that should be in the scope of a different issue. |
|
The agent also found the following counterexample: The plan moves the filter into the left input. The cause is Here the PR refuses to decorrelate, because the copied left input now contains RAND(), and keeps the Correlate. But decorrelate() returns the plan that the early rules already changed. So "keep the Correlate" does not mean "keep the original plan". Suggested changes for the non-deterministic part: Fix FilterCorrelateRule: skip non-deterministic conjuncts, as FilterJoinRule does. Add T1 as a test. A RAND() count is random, so the test should check the plan: the RAND() filter must stay above the Correlate. Optional: the PR keeps the whole Correlate when any part of the inner input is non-deterministic. In T4, the RAND() projection could run after the join back, and the rest could still be decorrelated. This is an optimization, not a correctness issue. |
Thanks for catching this. I've updated FilterCorrelateRule to skip non-deterministic conjuncts and added a plan test based on your example. |
|
| * visitor. A supported operator may have no expressions to visit. Exceptions | ||
| * thrown by the visitor are propagated. | ||
| * <p>The shuttle controls visits to expression children and sub-query plans. | ||
| * Rewritten expressions are ignored. An expression may be visited more than once. |
There was a problem hiding this comment.
What does it mean "rewritten expressions are ignored"?
Who ignores them?
There was a problem hiding this comment.
Thanks. This method discards any replacements produced by the shuttle. I've clarified that in the Javadoc.
| * @param visitor Visitor used for read-only analysis | ||
| * @return Whether the operator family is supported; true after visiting its | ||
| * expression fields, false without invoking the visitor | ||
| * @return True if this method can visit expressions for the node type, even |
There was a problem hiding this comment.
"for the node type" -- what node type?
Maybe "kind"?
Moreover, there is no "node", only a "rel".
"type" means many things.
There was a problem hiding this comment.
Thanks. I've used rel and kind in the return description.
| /** Applies the visitor within the scope of an outer correlation binding. | ||
| * | ||
| * <p>The scope is the part of {@code root} where {@code id} refers to the | ||
| * binding defined outside {@code root}. References to nested bindings |
There was a problem hiding this comment.
I never understood this business about nested bindings with the same id. I think these should be illegal plans. The plans are complicated enough as it is. We should use alpha-renaming if there is a confusion about reused names.
There was a problem hiding this comment.
I agree that distinct IDs for distinct bindings would simplify the analysis. This PR preserves the existing IDs and scope handling in findInputRel. CALCITE-7753 also covers nested Correlates reusing an ID. I think alpha-renaming would be a very worthwhile improvement.
| /** Selects outermost deterministic computations over one correlation. | ||
| /** Selects deterministic expressions for evaluation on the outer input. | ||
| * | ||
| * <p>The collector stores each selected {@link RexNode} with its offset among |
There was a problem hiding this comment.
Thanks. collector is a CorrelatedExpressionSelector whose selected method records the expression in expressions and returns it unchanged. I've documented the map's offsets beside its declaration.
| * | ||
| * <p>The collector stores each selected {@link RexNode} with its offset among | ||
| * the fields added to the left input. A Project computes these fields. | ||
| * The replacement visitor replaces their uses on the right with |
There was a problem hiding this comment.
what is the replacement visitor?
There was a problem hiding this comment.
Thanks. I meant replacements, the CorrelatedExpressionSelector used to rewrite the right input. Its selected method replaces each selected expression with a RexFieldAccess pointing to the corresponding field on the left.
| * The replacement visitor replaces their uses on the right with | ||
| * {@link RexFieldAccess} references. | ||
| * | ||
| * <p>Selected expressions go to {@link #selected(RexNode)} without visiting |
There was a problem hiding this comment.
This should probably be part of the JavaDoc of "selected".
There was a problem hiding this comment.
Thanks. Moved to selected. It receives an expression chosen for extraction and returns the expression to use at that position. Its subexpressions are not selected separately.
| * | ||
| * <p>Selected expressions go to {@link #selected(RexNode)} without visiting | ||
| * their operands. Other calls have their operands checked separately. | ||
| * Lambda bodies and sub-query plans are skipped. A complete call containing |
There was a problem hiding this comment.
what is a "complete call"?
There was a problem hiding this comment.
Thanks. I've removed “complete call” and clarified the boundary. The selector searches lambda bodies, but does not extract expressions containing lambda parameter references, following the behavior on main.
| * <p>Records whether outer bindings may be referenced and which declared | ||
| * fields may still be needed. | ||
| * | ||
| * <p>A local binding hides the outer binding with the same id in its scope. |
There was a problem hiding this comment.
see my comment above. But perhaps this is unavoidable if this is the normal behavior of many passes.
There was a problem hiding this comment.
Thanks. This analysis follows the existing binding scopes. With unique binding IDs, the same-ID shadowing checks would no longer be needed.
|
Please also rebase this on main. I will submit also an agent-generated review. |
mihaibudiu
left a comment
There was a problem hiding this comment.
An agent thinks that this PR is sound (and clearly an improvement over main), so I am approving, but I think readability can still be improved.
…determinism checks
…nputs in RelFieldTrimmer
098cfc5 to
44dce06
Compare
Thanks. I've rebased on main. This PR skipped lambda bodies, which conflicted with CALCITE-7933. I removed that restriction and kept main's check for lambda parameter references. |



Jira Link
CALCITE-7914
Changes Proposed
RelDecorrelatorcan recompute a non-deterministic correlation key and lose the subquery result when joining it back. It can also incorrectly share a non-deterministic inner result between outer rows with equal bindings.The fix is to check determinism before extracting expressions or sharing inputs. For the general rewrite, the check covers the rebuilt right input, including any copied outer inputs. Unused expressions can be trimmed locally before retrying the checks. Trimming leaves correlation inputs unchanged.
Both trimming sites use the new
RelFieldTrimmer.trimPreservingBindings, which also preserves the trimmed subtree's output fields.The checks use the new
RelOptUtil.isDeterministicandCorrelationReferencesanalyses. Existing rule patterns and uniqueness checks still apply. If no safe rewrite applies, the Correlate is retained.