Description
Part of #5572.
CometStaticInvoke dispatches on an allowlist of (functionName, staticObject) pairs and falls the whole operator back to Spark for anything not in it:
case None =>
withFallbackReason(
expr,
s"Static invoke expression: ${expr.functionName} is not supported")
Nonespark/src/main/scala/org/apache/comet/serde/statics.scala:61-65
StaticInvoke is a codegen-friendly node — doGenCode emits a static method call — and CometStaticInvokeCodegenDispatch (statics.scala:120) already exists and is wired for aesEncrypt, aesDecrypt and StringDecode.decode. Making it the case None default turns a hard fallback into in-pipeline execution for every lowering nobody has gotten to yet.
The compatibility guide already records the symptom. docs/source/user-guide/latest/expressions.md lists encode as "🔜 Lowers to StaticInvoke(encode) (not allowlisted); falls back", and to_binary as "Hex form accelerated; other formats fall back". Both would be covered by the default.
The same applies to Invoke, which has no entry in the serde map at all. Spark4xCometExprShim.sparkVersionSpecificExprToProtoInternal recognizes StructsToJsonEvaluator and ParseUrlEvaluator and returns None for everything else (spark/src/main/spark-4.x/org/apache/comet/shims/Spark4xCometExprShim.scala:94), after which the generic path reports "not supported". Spark 4.x lowers a growing number of RuntimeReplaceable expressions to evaluator-backed Invoke nodes, so the same catch-all is worth having there.
This is safe by construction for the encoder and deserializer trees that make up most StaticInvoke usage in typed Dataset operations: their arguments and return types are ObjectType, which is outside CometBatchKernelCodegen.isSupportedDataType, so canHandle rejects them and they fall back exactly as they do now.
Depends on the closure-serialize guard, since this is the change that starts handing the dispatcher arbitrary trees.
Activity