You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Bug] spark factorial rejects BIGINT and untyped integer literals that Spark accepts #24940
datafusion-spark's factorial declares Signature::exact(vec![DataType::Int32], ...)
(datafusion/spark/src/function/math/factorial.rs). Spark's Factorial declares inputTypes = Seq(IntegerType) but extends ImplicitCastInputTypes, so Spark casts every
integer width to INT before evaluating. Because an untyped integer literal is Int64 in
DataFusion, even the simplest call fails in Spark mode.
Identical in Spark 3.5.8, 4.0.4, 4.1.3 and 4.2.0 (the class only changed how it declares nullIntolerant).
To Reproduce
Spark SQL (verified with pyspark==4.2.0, spark.sql.ansi.enabled both true and false):
SELECT factorial(5);
-- Error during planning: Failed to coerce arguments to satisfy a call to 'factorial' function:-- coercion from Int64 to the signature Exact(Int32) failed.SELECT factorial(5::BIGINT);
-- same error
Expected behavior
factorial accepts every integer width, as Spark does, and SELECT factorial(5) returns 120.
Additional context
The narrow signature was chosen deliberately in #16125 based on the Databricks reference, which
documents the parameter as an INTEGER expression; a running Spark shows the accepted set is wider
because of the implicit cast. The existing test asserts the coercion error for factorial(5::BIGINT) as expected behavior and needs to be replaced.
Out-of-range values are a separate, smaller divergence. DataFusion always fails at the Int32 cast
(this function does not consult datafusion.execution.enable_ansi_mode), while Spark 4.2.0 raises CAST_OVERFLOW under ANSI mode and returns NULL otherwise for factorial(CAST(5000000000 AS BIGINT)).
Surfaced by the audit-datafusion-spark-expression skill.
Activity
fix(spark): accept every integer width in `factorial`, matching Spark…