Skip to content

feat: run CheckOverflowInTableInsert natively - #7661

Open
FreakingMind wants to merge 4 commits into
apache:mainfrom
FreakingMind:check-overflow-table-insert
Open

FreakingMind wants to merge 4 commits into
apache:mainfrom
FreakingMind:check-overflow-table-insert

Conversation

@FreakingMind

@FreakingMind FreakingMind commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #7656.

Rationale for this change

When an INSERT narrows a numeric column, Spark wraps the cast in CheckOverflowInTableInsert. Comet had no serde for it, so the Project that feeds the write fell back to Spark, and with it the native Parquet write.

The wrapper only changes the error. Spark rethrows any SparkArithmeticException from the cast as CAST_OVERFLOW_IN_TABLE_INSERT, which names the column instead of the value and has no query context. Passing the wrapper through would give CAST_OVERFLOW instead.

What changes are included in this PR?

  • A native CheckOverflowInTableInsert that evaluates the cast and replaces an arithmetic error with the new SparkError::CastOverflowInTableInsert. It covers native errors, Comet's CastOverflowException, and a Java SparkArithmeticException from a child that runs through the JVM codegen dispatcher. Other errors pass through unchanged.
  • A serde that wraps only a Cast child. When the optimizer has rewritten the cast (SPARK-42286), Spark keeps the child's error, so the child is serialized on its own.
  • A converter case in both shims that calls QueryExecutionErrors.castingCauseOverflowErrorInTableInsert, so each Spark version gets its own message text (3.5 says "insert ... into", 4.x says "assign ... to"). The proto carries the types as Spark SQL type names for this.
  • A line in expressions.md, and the difference below in the compatibility guide.
  • Spark's postgreSQL/numeric.sql now runs without Comet in dev/diffs. One of its INSERTs writes POWER(10, LN(val)) into a decimal column. It used to fall back to Spark, and now runs natively, where pow differs from Math.pow in the last digit (Math functions differ from Spark in the last digits #6797). Regenerated for 3.5.9, 4.0.4, 4.1.3 and 4.2.0; each diff only gains that hunk.

There is one difference from Spark, which I documented. When an expression under the cast occurs more than once in the projection, for example INSERT INTO t SELECT c, c FROM src, Spark's subexpression elimination evaluates it outside the wrapper and reports the original error, such as CAST_OVERFLOW. Comet reports CAST_OVERFLOW_IN_TABLE_INSERT there, which is what Spark does with spark.sql.subexpressionElimination.enabled=false. Matching it would need the whole project list in the serde. The query fails either way; only the error class differs.

Narrowing INSERTs now stay in Comet by default. The new expression gets the usual spark.comet.expression.CheckOverflowInTableInsert.enabled switch.

How are these changes tested?

New tests in CometParquetWriterSuite, each run with the Spark writer and with the native writer:

  • narrowing INSERTs into int, tinyint, smallint, bigint, decimal and a struct field, compared with the same INSERT run by Spark;
  • overflows, including boundaries, NaN, Infinity and decimal precision, plus c div 0 under the cast, with the exception class, error class, SQLSTATE and message compared with Spark;
  • a non-arithmetic error under the cast, which keeps its class and query context;
  • the SPARK-42286 cases;
  • an overflowing literal with ConstantFolding excluded, which reaches the wrapper through the codegen dispatcher.

They are Scala tests rather than SQL file tests because an INSERT is a statement there, which is not compared with Spark.

All of them fail on main. I ran them on Spark 3.5, 4.0 and 4.1, together with the full CometParquetWriterSuite on 4.1. There is also a converter test in SparkErrorConverterSuite and Rust unit tests for the wrapper.

@comphead comphead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The wrapper matches Spark's CheckOverflowInTableInsert at v3.5.8, v4.0.1, v4.1.1 and v4.2.0: only a Cast child is wrapped, only SparkArithmeticException is rewritten, and the shims rebuild the error with castingCauseOverflowErrorInTableInsert so each version gets its own message. One point on is_arithmetic_error after #4653 landed on main.

Comment on lines +139 to +153
/// Whether Spark raises this error as a `SparkArithmeticException`: a native `SparkError`, or an
/// exception thrown by a child that runs Spark's own code through the JVM codegen dispatcher.
/// Only the top of a `DataFusionError::External` is checked, because that is where the JNI layer
/// reads a Spark error from.
fn is_arithmetic_error(error: &DataFusionError) -> bool {
let DataFusionError::External(e) = error else {
return false;
};
if let Some(CometError::JavaException { class, .. }) = e.downcast_ref::<CometError>() {
return class == "org.apache.spark.SparkArithmeticException";
}
e.downcast_ref::<SparkError>()
.or_else(|| e.downcast_ref::<SparkErrorWithContext>().map(|e| &e.error))
.is_some_and(SparkError::is_arithmetic)
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#4653 landed on main after this branch was cut. jvm_exception in native/jni-bridge/src/errors.rs now finds the Spark payload anywhere in the source() chain (extract_spark_payload, built on datafusion_comet_common::error_chain). So the claim in this doc comment that the JNI layer only reads the top DataFusionError::External is no longer true.

This function still only inspects the top External. A child error that reaches here wrapped, for example in Context, Shared or ArrowError::ExternalError, is not recognised. I expect it to keep its own class, such as DIVIDE_BY_ZERO, where Spark reports CAST_OVERFLOW_IN_TABLE_INSERT. That is the dispatched ScalaUDF case #4653 fixes. I did not run this.

Suggest walking the chain the same way, taking the first Spark-typed payload in the order extract_spark_payload uses (CometError::JavaException, then SparkErrorWithContext, then SparkError):

fn is_arithmetic_error(error: &DataFusionError) -> bool {
    error_chain(error)
        .find_map(|e| {
            if let Some(CometError::JavaException { class, .. }) = e.downcast_ref::<CometError>() {
                Some(class == "org.apache.spark.SparkArithmeticException")
            } else if let Some(ctx) = e.downcast_ref::<SparkErrorWithContext>() {
                Some(ctx.error.is_arithmetic())
            } else {
                e.downcast_ref::<SparkError>().map(SparkError::is_arithmetic)
            }
        })
        .unwrap_or(false)
}

error_chain is exported from datafusion_comet_common and checked_binary_expr.rs already imports it. A unit test with the child error wrapped in DataFusionError::Context would cover it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, you are right, #4653 landed after I cut the branch. I rebased on main and switched is_arithmetic_error to error_chain, in the same order as extract_spark_payload (bec0a24). The unit tests now also wrap the child error in Context; with the old check those cases fail with CAST_OVERFLOW instead of CAST_OVERFLOW_IN_TABLE_INSERT.

Spark wraps the cast of an INSERT that narrows a numeric column in
CheckOverflowInTableInsert, which Comet did not support, so the Project
that feeds the write fell back to Spark along with the native write.

The native wrapper keeps the cast native and turns an arithmetic error
from it into CAST_OVERFLOW_IN_TABLE_INSERT, built on the JVM by Spark's
own error factory so each version keeps its message.

Closes apache#7656
An INSERT in numeric.sql writes POWER(10, LN(val)) into a decimal column. It used to fall back to Spark, and now runs natively, where pow differs from Math.pow in the last digit (apache#6797).
…erflowInTableInsert

Since apache#4653 the JNI layer takes the first Spark error from the source() chain, so a child error wrapped in Context or Shared still reaches Spark with its class. Look for it the same way, so a wrapped arithmetic error becomes CAST_OVERFLOW_IN_TABLE_INSERT too.
@FreakingMind
FreakingMind force-pushed the check-overflow-table-insert branch from 1cee3fe to bec0a24 Compare November 18, 2026 01:58
@github-actions github-actions Bot added enhancement New feature or request area:expressions Expression evaluation labels Nov 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:expressions Expression evaluation enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support CheckOverflowInTableInsert natively

2 participants