Skip to content

Native string to decimal cast raises CAST_INVALID_INPUT instead of NUMERIC_VALUE_OUT_OF_RANGE under ANSI when the scale adjustment is more than 38 #7576

Open

Description

@snmvaughan

Describe the bug

Under ANSI mode, a string whose integral part has up to 38 digits but whose value does not fit the target decimal type raises NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION in Spark. The native cast raises CAST_INVALID_INPUT, which calls the value malformed, when the difference between the target scale and the parsed scale is more than 38. parse_string_to_decimal returns Ok(None) for that case ("scale is too high to fit i128"), and the cast reports Ok(None) as an invalid input under ANSI. Outside ANSI both engines return NULL, so only the error differs.

Input, cast under ANSI Spark 4.1.3 Comet
'1e37' to DECIMAL(10,2) NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION CAST_INVALID_INPUT
'1e30' to DECIMAL(38,10) NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION CAST_INVALID_INPUT
'1e29' to DECIMAL(38,10) NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION CAST_INVALID_INPUT
'1e30' to DECIMAL(10,2) (scale adjustment 32, for comparison) NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION
'6E+38' to DECIMAL(10,2), with spark.sql.legacy.allowNegativeScaleOfDecimal=true NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION CAST_INVALID_INPUT

Steps to reproduce

SET spark.sql.ansi.enabled=true;
CREATE TABLE t (s STRING) USING parquet;
INSERT INTO t VALUES ('1e37');
SELECT CAST(s AS DECIMAL(10, 2)) FROM t;
-- Spark: [NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION] 1E+37 cannot be represented as Decimal(10, 2). ...
-- Comet: [CAST_INVALID_INPUT] The value '1e37' of the type "STRING" cannot be cast to "DECIMAL(10,2)" because it is malformed. ...

The value comes from a table so that the optimizer doesn't fold the cast before Comet sees it.

Expected behavior

The native cast raises NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION for a string that parses but does not fit the target type, as Spark does.

Additional context

Willingness to contribute

I can contribute a fix for this bug independently.

Activity

  1. added
    run-all-spark-profilesRun the Comet test suites against every Spark profile on this pull request, ahead of the merge queue
    and removed
    run-all-spark-profilesRun the Comet test suites against every Spark profile on this pull request, ahead of the merge queue
    on Nov 11, 2026
  2. added
    bugSomething isn't working
    priority:lowMinor issues, test failures, tooling, cosmetic
    and removed on Nov 16, 2026
  3. added this to the 1.3.0 milestone on Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:expressionsExpression evaluationbugSomething isn't workingpriority:lowMinor issues, test failures, tooling, cosmetic

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions