Repository navigation
feat: run CheckOverflowInTableInsert natively - #7661
FreakingMind wants to merge 4 commits into
Conversation
comphead
left a comment
There was a problem hiding this comment.
The wrapper matches Spark's CheckOverflowInTableInsert at v3.5.8, v4.0.1, v4.1.1 and v4.2.0: only a Cast child is wrapped, only SparkArithmeticException is rewritten, and the shims rebuild the error with castingCauseOverflowErrorInTableInsert so each version gets its own message. One point on is_arithmetic_error after #4653 landed on main.
| /// Whether Spark raises this error as a `SparkArithmeticException`: a native `SparkError`, or an | ||
| /// exception thrown by a child that runs Spark's own code through the JVM codegen dispatcher. | ||
| /// Only the top of a `DataFusionError::External` is checked, because that is where the JNI layer | ||
| /// reads a Spark error from. | ||
| fn is_arithmetic_error(error: &DataFusionError) -> bool { | ||
| let DataFusionError::External(e) = error else { | ||
| return false; | ||
| }; | ||
| if let Some(CometError::JavaException { class, .. }) = e.downcast_ref::<CometError>() { | ||
| return class == "org.apache.spark.SparkArithmeticException"; | ||
| } | ||
| e.downcast_ref::<SparkError>() | ||
| .or_else(|| e.downcast_ref::<SparkErrorWithContext>().map(|e| &e.error)) | ||
| .is_some_and(SparkError::is_arithmetic) | ||
| } |
There was a problem hiding this comment.
#4653 landed on main after this branch was cut. jvm_exception in native/jni-bridge/src/errors.rs now finds the Spark payload anywhere in the source() chain (extract_spark_payload, built on datafusion_comet_common::error_chain). So the claim in this doc comment that the JNI layer only reads the top DataFusionError::External is no longer true.
This function still only inspects the top External. A child error that reaches here wrapped, for example in Context, Shared or ArrowError::ExternalError, is not recognised. I expect it to keep its own class, such as DIVIDE_BY_ZERO, where Spark reports CAST_OVERFLOW_IN_TABLE_INSERT. That is the dispatched ScalaUDF case #4653 fixes. I did not run this.
Suggest walking the chain the same way, taking the first Spark-typed payload in the order extract_spark_payload uses (CometError::JavaException, then SparkErrorWithContext, then SparkError):
fn is_arithmetic_error(error: &DataFusionError) -> bool {
error_chain(error)
.find_map(|e| {
if let Some(CometError::JavaException { class, .. }) = e.downcast_ref::<CometError>() {
Some(class == "org.apache.spark.SparkArithmeticException")
} else if let Some(ctx) = e.downcast_ref::<SparkErrorWithContext>() {
Some(ctx.error.is_arithmetic())
} else {
e.downcast_ref::<SparkError>().map(SparkError::is_arithmetic)
}
})
.unwrap_or(false)
}error_chain is exported from datafusion_comet_common and checked_binary_expr.rs already imports it. A unit test with the child error wrapped in DataFusionError::Context would cover it.
There was a problem hiding this comment.
Thanks, you are right, #4653 landed after I cut the branch. I rebased on main and switched is_arithmetic_error to error_chain, in the same order as extract_spark_payload (bec0a24). The unit tests now also wrap the child error in Context; with the old check those cases fail with CAST_OVERFLOW instead of CAST_OVERFLOW_IN_TABLE_INSERT.
Spark wraps the cast of an INSERT that narrows a numeric column in CheckOverflowInTableInsert, which Comet did not support, so the Project that feeds the write fell back to Spark along with the native write. The native wrapper keeps the cast native and turns an arithmetic error from it into CAST_OVERFLOW_IN_TABLE_INSERT, built on the JVM by Spark's own error factory so each version keeps its message. Closes apache#7656
An INSERT in numeric.sql writes POWER(10, LN(val)) into a decimal column. It used to fall back to Spark, and now runs natively, where pow differs from Math.pow in the last digit (apache#6797).
…erflowInTableInsert Since apache#4653 the JNI layer takes the first Spark error from the source() chain, so a child error wrapped in Context or Shared still reaches Spark with its class. Look for it the same way, so a wrapped arithmetic error becomes CAST_OVERFLOW_IN_TABLE_INSERT too.
1cee3fe to
bec0a24
Compare
Which issue does this PR close?
Closes #7656.
Rationale for this change
When an INSERT narrows a numeric column, Spark wraps the cast in
CheckOverflowInTableInsert. Comet had no serde for it, so the Project that feeds the write fell back to Spark, and with it the native Parquet write.The wrapper only changes the error. Spark rethrows any
SparkArithmeticExceptionfrom the cast asCAST_OVERFLOW_IN_TABLE_INSERT, which names the column instead of the value and has no query context. Passing the wrapper through would giveCAST_OVERFLOWinstead.What changes are included in this PR?
CheckOverflowInTableInsertthat evaluates the cast and replaces an arithmetic error with the newSparkError::CastOverflowInTableInsert. It covers native errors, Comet'sCastOverflowException, and a JavaSparkArithmeticExceptionfrom a child that runs through the JVM codegen dispatcher. Other errors pass through unchanged.Castchild. When the optimizer has rewritten the cast (SPARK-42286), Spark keeps the child's error, so the child is serialized on its own.QueryExecutionErrors.castingCauseOverflowErrorInTableInsert, so each Spark version gets its own message text (3.5 says "insert ... into", 4.x says "assign ... to"). The proto carries the types as Spark SQL type names for this.expressions.md, and the difference below in the compatibility guide.postgreSQL/numeric.sqlnow runs without Comet indev/diffs. One of its INSERTs writesPOWER(10, LN(val))into a decimal column. It used to fall back to Spark, and now runs natively, wherepowdiffers fromMath.powin the last digit (Math functions differ from Spark in the last digits #6797). Regenerated for 3.5.9, 4.0.4, 4.1.3 and 4.2.0; each diff only gains that hunk.There is one difference from Spark, which I documented. When an expression under the cast occurs more than once in the projection, for example
INSERT INTO t SELECT c, c FROM src, Spark's subexpression elimination evaluates it outside the wrapper and reports the original error, such asCAST_OVERFLOW. Comet reportsCAST_OVERFLOW_IN_TABLE_INSERTthere, which is what Spark does withspark.sql.subexpressionElimination.enabled=false. Matching it would need the whole project list in the serde. The query fails either way; only the error class differs.Narrowing INSERTs now stay in Comet by default. The new expression gets the usual
spark.comet.expression.CheckOverflowInTableInsert.enabledswitch.How are these changes tested?
New tests in
CometParquetWriterSuite, each run with the Spark writer and with the native writer:c div 0under the cast, with the exception class, error class, SQLSTATE and message compared with Spark;ConstantFoldingexcluded, which reaches the wrapper through the codegen dispatcher.They are Scala tests rather than SQL file tests because an INSERT is a
statementthere, which is not compared with Spark.All of them fail on main. I ran them on Spark 3.5, 4.0 and 4.1, together with the full
CometParquetWriterSuiteon 4.1. There is also a converter test inSparkErrorConverterSuiteand Rust unit tests for the wrapper.