Skip to content

[SPARK-56663][SQL][FOLLOWUP] Fix silent overflow in date_trunc fast path. - #56313

Closed
chenhao-db wants to merge 2 commits into
apache:masterfrom
chenhao-db:fix_trunc_overflow
Closed

chenhao-db wants to merge 2 commits into
apache:masterfrom
chenhao-db:fix_trunc_overflow

Conversation

@chenhao-db

@chenhao-db chenhao-db commented Jun 4, 2026 •

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Fixes the bug introduced by #55610. When local is a very large negative value, Math.floorMod(local, unitMicros) is positive because unitMicros is positive, and the subtraction can overflow to positive values. This produces a inconsistent result from the slow path.

In most common cases, this optimization still takes effect. The performance loss is ignorable.

Why are the changes needed?

Fix incorrect result.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

New unit tests. It passes before #55610, or after this change. But it fails on the current master.

Was this patch authored or co-authored using generative AI tooling?

No.

@chenhao-db

Copy link
Copy Markdown
Contributor Author

@cloud-fan could you take a look? Thanks!

@cloud-fan cloud-fan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 blocking, 1 non-blocking, 0 nits.
Clean, correct fix for the fast-path silent overflow.

Correctness (1)

  • DateTimeUtils.scala:513: same value - Math.floorMod(value, unit) pattern in the sibling MILLISECOND/SECOND branches (lines 540/542) still silently overflows at Long.MinValue — see inline

Verification

Traced the fix end to end: Math.subtractExact equals - for all non-overflowing inputs, and the only inputs it changes are those within one unitMicros of Long.MinValue, whose truncated-down instant is below Long.MinValue and so is unrepresentable by the slow path too. On underflow it throws ArithmeticException, caught by the existing catch (lines 524-525) and falls back to truncToUnit, which itself throws via instantToMicros's multiplyExact for the same input. So the fast path now fails consistently with the slow path rather than returning a wrapped-positive value — no input produces a wrong-but-representable result.

try {
val local = Math.addExact(micros, offsetMicros)
val truncatedLocal = local - Math.floorMod(local, unitMicros)
val truncatedLocal = Math.subtractExact(local, Math.floorMod(local, unitMicros))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nicely targeted fix. Note the same value - Math.floorMod(value, unit) shape lives one match away in the sibling branches: TRUNC_TO_MILLISECOND (line 540) and TRUNC_TO_SECOND (line 542) still use a plain -, so a timestamp at Long.MinValue micros silently wraps to a positive value there (e.g. MIN - floorMod(MIN, 1000) = 9223372036854775616) instead of throwing like the fast path now does. Those branches have no offset arithmetic and no slow-path fallback, so applying Math.subtractExact directly would make them throw ArithmeticException consistently. Worth folding into this PR (with MILLISECOND/SECOND added to the new test), or noting they're intentionally out of scope. Pre-existing, so non-blocking.

@chenhao-db
chenhao-db requested a review from cloud-fan June 4, 2026 03:15

@cloud-fan cloud-fan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 addressed, 0 remaining, 1 new (1 late catch — my miss from last round).
0 blocking, 1 non-blocking, 0 nits.
Prior round's MILLISECOND/SECOND finding fully addressed. Clean, complete, well-tested overflow fix.

Suggestions (1)

  • PR description: the stated mechanism is inverted — "Math.floorMod(local, unitMicros) is also negative" is wrong; floorMod with a positive divisor is always non-negative. The real cause is subtracting the non-negative remainder from a value near Long.MinValue, which underflows. (If it were negative as written, local - negative would move toward zero and could not overflow.) The code is correct — only the description's explanation. Worth fixing as the rationale-of-record.

Verification

Confirmed the now-fixed SECOND/MILLISECOND branches: Math.floorMod(Long.MinValue, unit) is positive (192 for ms, 224192 for sec), so Math.subtractExact underflows -> ArithmeticException, propagating directly (these branches have no slow-path fallback). The new test covers all levels YEAR..MILLISECOND — date levels via daysToMicros's multiplyExact, MINUTE/HOUR/DAY via the truncToUnitFast -> truncToUnit -> instantToMicros fallback — and correctly excludes MICROSECOND (returns micros unchanged). Both interpreted and codegen route through DateTimeUtils.truncTimestamp, so checkExceptionInExpression validates both paths. grep confirms lines 513/540/542 are the only value - floorMod sites; no silent-overflow pattern remains.

@cloud-fan

Copy link
Copy Markdown
Contributor

pyspark failure unrelated, thanks, merging to master/4.x/4.2 (bug fix)

@cloud-fan cloud-fan changed the title [SPARK-56663][FOLLOWUP] Fix silent overflow in date_trunc fast path. [SPARK-56663][SQL][FOLLOWUP] Fix silent overflow in date_trunc fast path. Jun 4, 2026
@cloud-fan cloud-fan closed this in 94d23c3 Jun 4, 2026
cloud-fan pushed a commit that referenced this pull request Jun 4, 2026
…ath.

### What changes were proposed in this pull request?

Fixes the bug introduced by #55610. When `local` is a very large negative value, `Math.floorMod(local, unitMicros)` is positive because `unitMicros` is positive, and the subtraction can overflow to positive values. This produces a inconsistent result from the slow path.

In most common cases, this optimization still takes effect. The performance loss is ignorable.

### Why are the changes needed?

Fix incorrect result.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

New unit tests. It passes before #55610, or after this change. But it fails on the current master.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #56313 from chenhao-db/fix_trunc_overflow.

Authored-by: chenhao-db <chenhao.li@databricks.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
(cherry picked from commit 94d23c3)
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
cloud-fan pushed a commit that referenced this pull request Jun 4, 2026
…ath.

### What changes were proposed in this pull request?

Fixes the bug introduced by #55610. When `local` is a very large negative value, `Math.floorMod(local, unitMicros)` is positive because `unitMicros` is positive, and the subtraction can overflow to positive values. This produces a inconsistent result from the slow path.

In most common cases, this optimization still takes effect. The performance loss is ignorable.

### Why are the changes needed?

Fix incorrect result.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

New unit tests. It passes before #55610, or after this change. But it fails on the current master.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #56313 from chenhao-db/fix_trunc_overflow.

Authored-by: chenhao-db <chenhao.li@databricks.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
(cherry picked from commit 94d23c3)
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
@pan3793

pan3793 commented Jun 9, 2026

Copy link
Copy Markdown
Member

@chenhao-db, the added test is flaky on JDK 25 - likely passes when run this single tests, but likely fails when run the whole suite

[info] - TruncTimestamp of Long.MinValue overflows with ArithmeticException *** FAILED *** (65 milliseconds)
[info]   (unsafe mode) Expected `` but null error message found (ExpressionEvalHelper.scala:219)
[info]   org.scalatest.exceptions.TestFailedException:
[info]   at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
...

https://github.com/apache/spark/actions/runs/27185554437/job/80255424191

This is expected behavior. For "hot throws", the JIT will produce compiled code that does not go through the interpreter via deoptimization to throw an exception but throws a pre-allocated exception object without a stack trace or message.

See #54514 for more details.

meta-codesync Bot pushed a commit to facebookincubator/velox that referenced this pull request Oct 31, 2026
Spark stores TIMESTAMP as int64 microseconds since the epoch, so the range is
[-290308-12-21 19:59:05.224192, 294247-01-10 04:00:54.775807] UTC. Truncating
a value close to the minimum to a coarser unit moves it below that range.
Spark then throws `ArithmeticException("long overflow")`:

    SELECT date_trunc('YEAR', timestamp_micros(-9223372036854775808));
    -- java.lang.ArithmeticException: long overflow

Velox timestamps have a wider range than int64 microseconds. So Spark
`date_trunc` returned a value such as `Timestamp(-9223402780800, 0)` that no
Spark timestamp can hold. It failed later when converted to microseconds
("Could not convert Timestamp(...) to microseconds"), or comparisons, casts
and writes used it silently.

This PR makes Spark `date_trunc` throw "long overflow" (ARITHMETIC_ERROR, a
user error) when the result is outside the int64 microseconds range. Presto
`date_trunc` and the shared `truncateTimestamp` helper are unchanged.

### Spark semantics

Results of `DateTimeUtils.truncTimestamp(Long.MinValue, level, zone)`. These
were produced by calling the released Spark 3.5.5 and 4.2.0 jars and are the
same for UTC, America/Los_Angeles and Asia/Shanghai:

| Unit | Spark 3.5 (to 4.1) | Spark 4.2 | Velox before | Velox after |
|---|---|---|---|---|
| YEAR, QUARTER, MONTH, WEEK | throws | throws | out-of-range value | throws |
| DAY, HOUR, MINUTE | throws | throws | out-of-range value | throws |
| SECOND | 294247-01-10 04:00:54.551616 (wrapped) | throws | out-of-range value | throws |
| MILLISECOND | 294247-01-10 04:00:54.775616 (wrapped) | throws | out-of-range value | throws |
| MICROSECOND | unchanged | unchanged | unchanged | unchanged |

- **YEAR to MINUTE.** Spark 3.5 already throws, because
  [`truncToUnit`/`daysToMicros`][spark35] convert the result through
  `instantToMicros`, which uses `Math.multiplyExact`/`addExact`. For these
  units the PR is a plain bug fix that matches Spark 3.5.
- **SECOND and MILLISECOND.** Spark 3.5 to 4.1 compute
  `micros - floorMod(micros, unit)` without an overflow check, so the result
  wraps around to a timestamp in year 294247. [SPARK-56663] fixed this in Spark
  4.2 as "silent overflow ... incorrect result" ([apache/spark#56313], commit
  [94d23c3]) by switching to `Math.subtractExact` ([4.2 source][spark42]). That
  change added the test [`TruncTimestamp of Long.MinValue overflows with
  ArithmeticException`][spark42test]. Velox didn't match the 3.5 wraparound
  before this PR. It returned a different, unrepresentable value. Matching 3.5
  would mean returning a deliberately wrong timestamp roughly 584,000 years
  later than the input. So this PR throws, which is consistent with how Spark
  3.5 handles every other unit. No config is added, because no valid result
  for these inputs is being changed.
- **Upper bound.** Truncation never moves a value forward, so for inputs in
  range only the lower bound can be crossed. The upper bound only matters for
  Velox timestamps above the Spark range, which Spark can't produce.
- **Affected inputs.** Only inputs within about a year of the minimum
  timestamp are affected (for example, only YEAR, QUARTER and MONTH still
  overflow one day after it). Results for every other input are unchanged.

### Implementation notes

- `Timestamp::minMicros()` and `Timestamp::maxMicros()` define the exact
  int64-microsecond range once. `DateTruncFunction` uses them for the range
  check, and `SecondsToTimestampFunction` uses the same helpers for clamping.
  The helpers avoid `Timestamp::fromMicros(INT64_MIN)`, which overflows
  internally while constructing the lower bound.
- The function throws instead of returning `Status`. A `Status`-returning
  `call()` always produces a non-null result, but `date_trunc` must return NULL
  for an invalid `fmt`. Throwing only happens within about a year of the
  minimum timestamp, so it has no performance impact.
- `VELOX_ARITHMETIC_ERROR` makes this a user error with code ARITHMETIC_ERROR,
  so `TRY` suppresses it.

### Known difference that isn't changed

For MINUTE, the shared `truncateTimestamp` floors in UTC. Spark floors in the
session time zone, which matters for historical offsets that aren't whole
minutes (for example America/Los_Angeles -07:52:58 before 1883; see
SPARK-33404). Near the minimum timestamp this means Velox throws for some
inputs where Spark returns a value. One example is
`timestamp_micros(-9223372036842775808)` (the minimum plus 12 seconds) in
Asia/Shanghai, which has offset +08:05:43 at that date.

This PR doesn't make that worse. It only changes inputs for which Velox
already returned a timestamp outside the int64 microseconds range, and those
inputs were already wrong. Any in-range result stays unchanged. Truncating
MINUTE in local time changes results for ordinary historical timestamps too,
so it belongs in a separate fix. The new tests use the exact minimum, where
Spark throws for every unit and every time zone tested.

### Fuzzer

The signature is unchanged, so the bias fuzzer is not triggered. The Spark
expression fuzzer compares Velox's common and simplified evaluation paths with
each other, not with Spark. It generates timestamps within ±2.14e9 seconds of
the epoch. The new error is a user error raised identically on both paths, so
it cannot cause a mismatch. No skip-list changes are needed.

### Tests

Added cases to `DateTimeFunctionsTest.dateTrunc`:
- The minimum timestamp in UTC, America/Los_Angeles and Asia/Shanghai, for
  every unit.
- The smallest inputs whose SECOND and MILLISECOND truncations are still in
  range.
- The maximum timestamp for every unit. The expected values match Spark 4.2's
  output.
- Inputs just outside the range.

Added `TimestampTest` coverage that converts `Timestamp::minMicros()` and
`Timestamp::maxMicros()` back to `INT64_MIN` and `INT64_MAX`.

[SPARK-56663]: https://issues.apache.org/jira/browse/SPARK-56663
[apache/spark#56313]: apache/spark#56313
[94d23c3]: apache/spark@94d23c3
[spark35]: https://github.com/apache/spark/blob/v3.5.5/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala#L431-L446
[spark42]: https://github.com/apache/spark/blob/v4.2.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala#L533-L553
[spark42test]: https://github.com/apache/spark/blob/v4.2.0/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/DateExpressionsSuite.scala#L856-L880

Differential Revision: D128741219

Pull Request resolved: #19841
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants