Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -72,5 +72,6 @@
- Spark 4.0.1 (audited 2026-05-27): adds the `withNewAlwaysEvaluatedInputs` optimizer hook; semantics unchanged.
- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
- Performance (tuned 2026-09-28, PR [#6350](https://github.com/apache/datafusion-comet/pull/6350)): when no branch value and no WHEN after the first can fail, `CaseWhenExpr` evaluates them over the whole batch and takes each row's value with typed merges for primitive, boolean, string and binary results, instead of filtering the batch for each branch and merging the partial results with `MutableArrayData`. A branch is no longer wrapped in a cast to the type it already has, which hid literals and columns from DataFusion's fast paths. 2.8-4x faster with literal results and 2.2-7.8x with column results. When a branch can fail it is about 2% slower, because DataFusion's `CaseExpr` derives the result type of a bare branch for every batch. Benchmark: `benches/conditional.rs`.
- Performance (tuned 2026-11-06, PR [#7538](https://github.com/apache/datafusion-comet/pull/7538)): a struct field counts as unable to fail, so a `CASE WHEN` over struct fields also takes the whole-batch evaluation above, and `AND` and `OR` evaluate a struct field operand for every row instead of masking it. 1.4-1.7x faster end to end over 1M rows of struct fields, 8-11x for the projection alone, and 2x for a filter on struct fields. Benchmark: Spark queries described in the PR.

[Spark Expression Support]: ../../user-guide/latest/expressions.md
36 changes: 31 additions & 5 deletions native/spark-expr/src/conditional_funcs/case_when.rs
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,9 @@
// specific language governing permissions and limitations
// under the License.

use crate::{Cast, EvalMode, IfExpr, NormalizeNaNAndZero, SparkCastOptions, SparkComparison};
use crate::{
Cast, EvalMode, GetStructField, IfExpr, NormalizeNaNAndZero, SparkCastOptions, SparkComparison,
};
use arrow::array::{
downcast_primitive, new_empty_array, Array, ArrayRef, AsArray, BooleanArray, GenericByteArray,
PrimitiveArray,
Expand Down Expand Up @@ -433,10 +435,11 @@ impl PhysicalExpr for CaseWhenExpr {
/// Whether `expr` can be evaluated for rows that Spark would not evaluate it for, at little cost.
///
/// That holds when it can neither fail nor return something different for seeing more rows, and is
/// cheap: a column, a literal, and comparisons, boolean logic, null checks, widening casts and
/// wrapping arithmetic over them. Anything else, including every function and a cast that parses a
/// string, is treated as able to fail. `CASE` uses this to decide whether to evaluate every branch
/// over the whole batch, which pays off only when that costs little more than selecting rows.
/// cheap: a column, a literal, and struct fields, comparisons, boolean logic, null checks, widening
/// casts and wrapping arithmetic over them. Anything else, including every function and a cast that
/// parses a string, is treated as able to fail. `CASE` uses this to decide whether to evaluate
/// every branch over the whole batch, which pays off only when that costs little more than
/// selecting rows.
pub(super) fn is_infallible(expr: &Arc<dyn PhysicalExpr>, input_schema: &Schema) -> bool {
infallible(expr, input_schema, false)
}
Expand Down Expand Up @@ -476,6 +479,8 @@ fn infallible(expr: &Arc<dyn PhysicalExpr>, input_schema: &Schema, parsing: bool
|| expr.is::<IsNotNullExpr>()
|| expr.is::<CaseWhenExpr>()
|| expr.is::<IfExpr>()
// A field of a struct cannot fail
|| expr.is::<GetStructField>()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit, optional: the SQL test gives the same answers on either path, so nothing would catch it if GetStructField dropped out of this list later. Could we add a struct column to the schema in infallible_expressions and assert infallible for a field of it, and for a field of a nested struct?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in d1ea932. infallible_expressions now has a struct column st with a nested struct, and asserts that st.a and st.inner.b are infallible.

};
node_is_infallible
&& expr
Expand Down Expand Up @@ -1309,6 +1314,21 @@ mod tests {
)),
true,
),
Field::new(
"st",
DataType::Struct(
vec![
Field::new("a", DataType::Int64, true),
Field::new(
"inner",
DataType::Struct(vec![Field::new("b", DataType::Utf8, true)].into()),
true,
),
]
.into(),
),
true,
),
]);
let c = |name: &str| col(name, &schema).unwrap();
let cast_in = |mode: EvalMode, e: Arc<dyn PhysicalExpr>, to: DataType| {
Expand Down Expand Up @@ -1348,6 +1368,12 @@ mod tests {
c("i"),
lit(0i64)
))));
// A field of a struct, and a field of a struct in a struct
let field = |e: Arc<dyn PhysicalExpr>, ordinal: usize| -> Arc<dyn PhysicalExpr> {
Arc::new(GetStructField::new(e, ordinal))
};
assert!(infallible(field(c("st"), 0)));
assert!(infallible(field(field(c("st"), 1), 0)));

assert!(infallible(Arc::new(NormalizeNaNAndZero::new(
DataType::Float64,
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
-- Licensed to the Apache Software Foundation (ASF) under one
-- or more contributor license agreements. See the NOTICE file
-- distributed with this work for additional information
-- regarding copyright ownership. The ASF licenses this file
-- to you under the Apache License, Version 2.0 (the
-- "License"); you may not use this file except in compliance
-- with the License. You may obtain a copy of the License at
--
-- http://www.apache.org/licenses/LICENSE-2.0
--
-- Unless required by applicable law or agreed to in writing,
-- software distributed under the License is distributed on an
-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-- KIND, either express or implied. See the License for the
-- specific language governing permissions and limitations
-- under the License.

-- CASE WHEN, IF and COALESCE whose branches read struct fields, and AND, OR and IN whose operands
-- do. A struct field cannot fail, so Comet evaluates these for the whole batch and then picks each
-- row's value, as it does for top-level columns. A field of a NULL struct is NULL. A branch or an
-- operand that divides still runs only for the rows that reach it, which matters with ANSI mode on.

-- ConfigMatrix: spark.sql.ansi.enabled=false,true

statement
CREATE TABLE test_case_struct_field(k int, s struct<
p: boolean, b: boolean, i: int, l: bigint, d: double, dec: decimal(38, 18), dt: date,
ts: timestamp, str: string, bin: binary, n: struct<x: bigint, y: bigint>>) USING parquet

statement
INSERT INTO test_case_struct_field
SELECT k, IF(k = 3, NULL, named_struct(
'p', p, 'b', b, 'i', i, 'l', l, 'd', d, 'dec', dec, 'dt', dt, 'ts', ts, 'str', str,
'bin', bin, 'n', IF(k = 4, NULL, named_struct('x', x, 'y', CAST(y AS BIGINT)))))
FROM VALUES
(0, true, true, 1, 10, 2.5D, 1.000000000000000001, DATE '2024-01-01',
TIMESTAMP '2024-01-01 10:00:00', 'apple', X'01', 7, 2),
(1, false, false, CAST(-2147483648 AS INT), CAST(-9223372036854775808 AS BIGINT),
CAST('NaN' AS DOUBLE), -99999999999999999999.999999999999999999, DATE '1901-01-01',
TIMESTAMP '1901-01-01 00:00:01', '', X'', 9223372036854775807, 0),
(2, true, false, 2147483647, 9223372036854775807, CAST('Infinity' AS DOUBLE),
99999999999999999999.999999999999999999, DATE '9999-12-31', TIMESTAMP '2262-04-11 23:47:16',
'a value that is longer than sixteen bytes', X'00112233445566778899AABBCCDDEEFF0011',
CAST(-9223372036854775808 AS BIGINT), 3),
(3, true, true, 5, 50, 1.0D, 0.0, DATE '2000-01-01', TIMESTAMP '2000-01-01 00:00:00', 'x',
X'02', 1, 1),
(4, NULL, true, NULL, -1, CAST('-Infinity' AS DOUBLE), NULL, NULL, NULL, NULL, NULL, 4, 4),
(5, true, NULL, 0, NULL, -0.0D, 0.000000000000000001, DATE '1970-01-01',
TIMESTAMP '1970-01-01 00:00:00', NULL, X'FFFE', NULL, 0),
(6, false, true, -7, 0, 0.0D, -2.5, NULL, NULL, '日本語', NULL, 0, -1),
(7, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL)
AS t(k, p, b, i, l, d, dec, dt, ts, str, bin, x, y)

query
SELECT k,
CASE WHEN s.p THEN s.b END,
CASE WHEN s.p THEN s.b ELSE NOT s.b END,
CASE WHEN s.p THEN s.i WHEN s.b THEN -1 END,
CASE WHEN s.i > 0 THEN s.l ELSE s.n.x END,
IF(s.b, s.l, s.n.y),
coalesce(s.n.x, s.l, 0)
FROM test_case_struct_field

query
SELECT k,
CASE WHEN s.p THEN s.d END,
CASE WHEN s.p THEN s.d WHEN s.b THEN s.d * 2 ELSE s.d / 2 END,
IF(s.b, s.d, s.d / 2),
coalesce(s.d, 0.0D)
FROM test_case_struct_field

query
SELECT k,
CASE WHEN s.p THEN s.dec END,
IF(s.b, s.dec, NULL),
CASE WHEN s.p THEN s.dt WHEN s.b THEN DATE '2000-01-01' END,
coalesce(s.ts, TIMESTAMP '2000-01-01 00:00:00')
FROM test_case_struct_field

query
SELECT k,
CASE WHEN s.p THEN s.str WHEN s.b THEN 'b' END,
IF(s.p, s.str, ''),
CASE WHEN s.b THEN s.bin END,
coalesce(s.str, 'none')
FROM test_case_struct_field

-- a field of a field, which is NULL when the inner struct is
query
SELECT k, CASE WHEN s.n.x > s.n.y THEN s.n.x ELSE s.n.y END, IF(s.n IS NULL, -1L, s.n.y)
FROM test_case_struct_field

query
SELECT k, s.p AND s.b, s.p OR s.i > 0, s.l IN (s.n.x, s.n.y), s.i IN (1, 2147483647)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit, optional: since lazy_in_list only applies when the value passes is_infallible, this change also means a later IN candidate that can fail now runs only for the rows that haven't matched yet when the value is a struct field. On main today that candidate runs for every row. Could we add something like SELECT k, s.n.x IN (9223372036854775807, s.n.x div s.n.y) FROM test_case_struct_field? Row 1 matches the first candidate and has y = 0, so with ANSI on Spark returns true without dividing. It might be worth a line in the description too.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in d1ea932, with a paragraph in the description. Without this change, the ANSI run of that query fails with DIVIDE_BY_ZERO (checked locally on main). With it, the file passes with ANSI mode off and on.

FROM test_case_struct_field

query
SELECT k FROM test_case_struct_field WHERE s.p AND s.i > 0

-- a division by a zero that the WHEN, the left operand of AND, or an earlier IN candidate excludes.
-- Row 1 matches the first IN candidate, so Spark does not evaluate the second.
query
SELECT k, CASE WHEN s.n.y <> 0 THEN s.n.x div s.n.y ELSE -1 END FROM test_case_struct_field

query
SELECT k, s.n.y <> 0 AND s.n.x div s.n.y > 1 FROM test_case_struct_field

query
SELECT k, s.n.x IN (9223372036854775807, s.n.x div s.n.y) FROM test_case_struct_field