Repository navigation
[fix](be) Reject Arrow timestamps outside the DATETIMEV2 and TIMESTAMPTZ ranges #69697
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
morningman
merged 2 commits into
apache:master
from
morningman:b2m-t37f-arrow-timestamp-year-range
Nov 3, 2026
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[P2] Mask NULL ancestors before validating nested timestamp leaves
A valid nullable Arrow STRUCT can have parent validity
[false, true]and a non-nulltimestamp(MICRO)child containing[2005980768000000000, 0]; its logical rows are[NULL, {ts:1970-01-01}]. Arrow specifies independent parent/child validity.DataTypeNullableSerDecopies the parent bitmap, thenDataTypeStructSerDepassesfield(i)unchanged, so the leaf'sIsNull(0)is false and this new check rejects the hidden year-65537 payload. The previous conversion's wrapped year was hidden by the NULL parent, so this batch succeeded. The same regression affects the new TIMESTAMPTZ check and NULL LIST parents with arbitrary nonempty child spans. Normalization leaves ordinary struct/list timestamp arrays unchanged, and stream-load/ADBC callers propagate the error, failing a read whose visible timestamps are valid. Carry ancestor validity or skip/default children of NULL parents before applying either check, and add mixed NULL/non-NULL parent coverage.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agreed on the mechanism. The fix is the NULL-parent masking in #69581 rather than a second copy here: df95b41 (
mask_arrow_children_hidden_by_null_parents) masks the child slots of NULL list, map and struct rows in the shared nested SerDes before any leaf SerDe reads them, so the leaf checks of this PR never see a hidden payload, for every Arrow caller.ArrowParentNullsTestthere covers mixed NULL / non-NULL parents and child offsets.I checked your example on a tree with both PRs: a STRUCT with rows
[NULL, {ts: 0}]whose NULL row carries 2005980768000000000, and a LIST[NULL, [0]]whose NULL row spans that value, both read two rows with row 0 NULL. With the masking taken out, the STRUCT case fails withArrow timestamp 2005980768000000000 of type timestamp[us] is outside the DATETIMEV2 range. This PR will be merged after #69581, so master does not carry the regression in between; the description now says so. When this PR is rebased onto #69581, I will add your STRUCT and LIST cases here as a test.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Follow-up on the STRUCT and LIST cases promised above. This PR was merged on 2026-11-03, before #69581, so the cases went into #69581 instead, which was merged on 2026-11-06 as 0d1ae9e:
DataTypeDateTimeV2SerDeTest.IgnoresArrowTimestampsHiddenByNullParentsinbe/test/core/data_type_serde/data_type_serde_datetime_v2_test.cpp. For both DATETIMEV2 and TIMESTAMPTZ, it hides 65537-01-01 (2005980768000000000) under the NULL row of a LIST, a FIXED_SIZE_LIST, a MAP value and a STRUCT. The NULL row reads as NULL and the visible rows keep their values, while the same value in a visible row still fails with the range error. Returning the child unmasked frommask_arrow_children_hidden_by_null_parentsfails the test, and dropping only the STRUCT mask fails it at the STRUCT case.A correction to my reply above: master did carry the regression between the two merges, from 2026-11-03 to 2026-11-06. In that window, a batch whose NULL list, map or struct row hid an out-of-range timestamp failed with DataQualityError instead of being read. It raised an error and did not return wrong values.