Repository navigation
fix: Fall back to Spark for MakeDecimal with unsupported input type - #2815
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2815 +/- ##
============================================
+ Coverage 56.12% 59.14% +3.02%
- Complexity 976 1471 +495
============================================
Files 119 167 +48
Lines 11743 15238 +3495
Branches 2251 2514 +263
============================================
+ Hits 6591 9013 +2422
- Misses 4012 4942 +930
- Partials 1140 1283 +143 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
| } | ||
| let result_type = DataType::Decimal128(precision, scale); | ||
| ColumnarValue::Array(a) => match a.data_type() { | ||
| DataType::Int64 => { |
There was a problem hiding this comment.
should we also support Int32? 🤔
There was a problem hiding this comment.
for this PR, I just wanted to fall back to avoid crashing. We could choose to add support in the future though
| SQLConf.ANSI_ENABLED.key -> "false", | ||
| SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false", | ||
| CometConf.getExprAllowIncompatConfigKey(classOf[Sum]) -> "true", | ||
| CometConf.COMET_NATIVE_SCAN_IMPL.key -> CometConf.SCAN_NATIVE_ICEBERG_COMPAT, |
There was a problem hiding this comment.
Are all settings needed for this test ? E.g. Sum and Iceberg compat look unrelated ?
| checkSparkAnswerAndFallbackReason(df1, "Unsupported input data type: IntegerType") | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
IMO it would be useful to add a positive test too - with LongType
….scala Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>
|
are we waiting to address any feedback on this PR? |
I think I addressed all of the feedback from @martin-g |
| SQLConf.ADAPTIVE_OPTIMIZER_EXCLUDED_RULES.key -> "org.apache.spark.sql.catalyst.optimizer.ConstantFolding") { | ||
|
|
||
| val df = sql("select * from t1") | ||
| val makeDecimalColumn = createMakeDecimalColumn(df.col("c1").expr, 3, 0) |
There was a problem hiding this comment.
nit: IMO the precision should be increased to at least 6 because 123456 won't fit into decimal(3, 0) and this leads to an overflow and this is not the purpose of this test.
Same for IntegerType above.
comphead
left a comment
There was a problem hiding this comment.
Thanks @andygrove and @martin-g for the review.
I feel this PR is good as it has the consistent issue before the PR and after PR it is fixed, probably remaining feedback we can address in following PRs if you guys okay with that
|
Which issue does this PR close?
Closes #2813
Rationale for this change
What changes are included in this PR?
How are these changes tested?