Skip to content

fix: share the grouping key type checks between doConvert and canAggregateBeConverted - #7566

Draft
tlfua wants to merge 1 commit into
apache:mainfrom
tlfua:tlfua/fix/6906-collated-agg-guard
Draft

tlfua wants to merge 1 commit into
apache:mainfrom
tlfua:tlfua/fix/6906-collated-agg-guard

Conversation

@tlfua

@tlfua tlfua commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #6906.

Rationale for this change

CometExecRule.canAggregateBeConverted is the check that tagUnsafePartialAggregates uses to decide whether a buffer-producing aggregate needs the COMET_UNSAFE_PARTIAL tag, and its scaladoc says it has to mirror the predicate checks in CometBaseAggregate.doConvert. #4035 added a guard for non-default collated grouping keys to doConvert only, so canAggregateBeConverted reported an aggregate grouped on a collated string as convertible while doConvert rejects it. As the issue explains, this is latent today, but once reachable the tagging pass would replace the specific collation fallback reason with the generic "Partial aggregate disabled" one.

What changes are included in this PR?

  • operators.scala: new CometBaseAggregate.unsupportedGroupingReason, which holds the map-typed and collated grouping key checks with their existing fallback reasons. doConvert calls it, with the same check order and reasons as before.
  • CometExecRule.scala: checkAggregateConversion calls the same helper instead of repeating only the map-type check, and the scaladoc notes which checks are now shared.
  • pr_build_linux.yml / pr_build_macos.yml: list the new suite.

How are these changes tested?

New spark/src/test/spark-4.x/org/apache/comet/rules/CometAggregateCollationSuite.scala (Spark 4.x only, since collated strings do not exist in Spark 3.x). End-to-end SQL cannot reach the gap, because Comet declines the sort that Spark places under a collated grouping key, so the suite plans the query with Comet disabled and calls canAggregateBeConverted on the partial aggregate directly:

  • A UTF8_LCASE grouping key is rejected. This fails without the fix, because a collated attribute serializes through exprToProto.
  • A default UTF8_BINARY grouping key is still accepted, so the rejection comes from the collation guard.

Locally, make format (which compiles main and test sources for the default Spark 4.1 profile) succeeded, and dev/ci/check-suites.py, dev/ci/check-ci-config.py, and dev/ci/check-versioned-source-parity.py passed. Fork CI passed, and both new tests ran in the Spark 4.1 [exec] group: https://github.com/tlfua/datafusion-comet/actions/runs/42431970135

…egateBeConverted

canAggregateBeConverted is meant to mirror CometBaseAggregate.doConvert, but
the non-default collated grouping key guard added to doConvert was never added
to it, so the tagging pass treated such an aggregate as convertible. Move the
map-typed and collated grouping key checks into
CometBaseAggregate.unsupportedGroupingReason and call it from both places.

Closes apache#6906

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c7b481e1-4043-4a63-9e54-6d89b950f731
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

canAggregateBeConverted does not mirror doConvert's collated grouping key guard

1 participant