Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 23 additions & 1 deletion be/src/core/data_type_serde/data_type_datetimev2_serde.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -723,7 +723,29 @@ Status DataTypeDateTimeV2SerDe::read_column_from_arrow(IColumn& column,
--seconds;
remainder += divisor;
}
v.from_unixtime(seconds, real_ctz);
// from_unixtime() narrows the cctz year to uint16_t without a check, so an instant
// whose civil year in real_ctz is outside 0000..9999 would read as another value:
// timestamp(us) 2005980768000000000 (65537-01-01) as 0001-01-01, and the years
// 10000..65535 as dates DATETIMEV2 cannot hold. Check the civil year before packing,
// as orc_timestamp_to_datetime() does for ORC.
const auto civil = cctz::convert(
cctz::time_point<cctz::seconds>(cctz::seconds(seconds)), real_ctz);
if (civil.year() < 0 || civil.year() > 9999) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Mask NULL ancestors before validating nested timestamp leaves

A valid nullable Arrow STRUCT can have parent validity [false, true] and a non-null timestamp(MICRO) child containing [2005980768000000000, 0]; its logical rows are [NULL, {ts:1970-01-01}]. Arrow specifies independent parent/child validity. DataTypeNullableSerDe copies the parent bitmap, then DataTypeStructSerDe passes field(i) unchanged, so the leaf's IsNull(0) is false and this new check rejects the hidden year-65537 payload. The previous conversion's wrapped year was hidden by the NULL parent, so this batch succeeded. The same regression affects the new TIMESTAMPTZ check and NULL LIST parents with arbitrary nonempty child spans. Normalization leaves ordinary struct/list timestamp arrays unchanged, and stream-load/ADBC callers propagate the error, failing a read whose visible timestamps are valid. Carry ancestor validity or skip/default children of NULL parents before applying either check, and add mixed NULL/non-NULL parent coverage.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed on the mechanism. The fix is the NULL-parent masking in #69581 rather than a second copy here: df95b41 (mask_arrow_children_hidden_by_null_parents) masks the child slots of NULL list, map and struct rows in the shared nested SerDes before any leaf SerDe reads them, so the leaf checks of this PR never see a hidden payload, for every Arrow caller. ArrowParentNullsTest there covers mixed NULL / non-NULL parents and child offsets.

I checked your example on a tree with both PRs: a STRUCT with rows [NULL, {ts: 0}] whose NULL row carries 2005980768000000000, and a LIST [NULL, [0]] whose NULL row spans that value, both read two rows with row 0 NULL. With the masking taken out, the STRUCT case fails with Arrow timestamp 2005980768000000000 of type timestamp[us] is outside the DATETIMEV2 range. This PR will be merged after #69581, so master does not carry the regression in between; the description now says so. When this PR is rebased onto #69581, I will add your STRUCT and LIST cases here as a test.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up on the STRUCT and LIST cases promised above. This PR was merged on 2026-11-03, before #69581, so the cases went into #69581 instead, which was merged on 2026-11-06 as 0d1ae9e: DataTypeDateTimeV2SerDeTest.IgnoresArrowTimestampsHiddenByNullParents in be/test/core/data_type_serde/data_type_serde_datetime_v2_test.cpp. For both DATETIMEV2 and TIMESTAMPTZ, it hides 65537-01-01 (2005980768000000000) under the NULL row of a LIST, a FIXED_SIZE_LIST, a MAP value and a STRUCT. The NULL row reads as NULL and the visible rows keep their values, while the same value in a visible row still fails with the range error. Returning the child unmasked from mask_arrow_children_hidden_by_null_parents fails the test, and dropping only the STRUCT mask fails it at the STRUCT case.

A correction to my reply above: master did carry the regression between the two merges, from 2026-11-03 to 2026-11-06. In that window, a batch whose NULL list, map or struct row hid an out-of-range timestamp failed with DataQualityError instead of being read. It raised an error and did not return wrong values.

return Status::DataQualityError(
"Arrow timestamp {} of type {} is outside the DATETIMEV2 range", date_value,
type->ToString());
}
v.unchecked_set_time(
static_cast<uint16_t>(civil.year()), static_cast<uint8_t>(civil.month()),
static_cast<uint8_t>(civil.day()), static_cast<uint8_t>(civil.hour()),
static_cast<uint8_t>(civil.minute()), static_cast<uint16_t>(civil.second()), 0);
// cctz counts year 0 as a leap year and Doris does not (is_leap(0) is false), so the
// civil date 0000-02-29 passes the year check but is not a DATETIMEV2 value.
if (!v.is_valid_date()) {
return Status::DataQualityError(
"Arrow timestamp {} of type {} is not a valid DATETIMEV2 value", date_value,
type->ToString());
}
// Get the fractional part.
// add 0 on the right to make it 6 digits. DateTimeV2Value microsecond is 6 digits,
// the scale decides to keep the first few digits, so the valid digits should be kept at the front.
Expand Down
21 changes: 18 additions & 3 deletions be/src/core/data_type_serde/data_type_timestamptz_serde.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -78,8 +78,21 @@ Status decode_timestamp_tz_orc_values(IColumn& nested_column,
Status append_timestamptz_from_utc_epoch_micros(ColumnTimeStampTz::Container& data,
int64_t timestamp_micros) {
static constexpr int64_t MICROS_PER_SECOND = 1000000;
// 0000-01-01 00:00:00 UTC and 9999-12-31 23:59:59.999999 UTC, the TIMESTAMPTZ range.
static constexpr int64_t MIN_TIMESTAMPTZ_MICROS = -62167219200000000LL;
static constexpr int64_t MAX_TIMESTAMPTZ_MICROS = 253402300799999999LL;
static const auto UTC = cctz::utc_time_zone();

// from_unixtime() narrows the cctz year to uint16_t without a check, so an instant past year
// 65535 wraps back into 0000..9999 and would still pass is_valid_date() after packing:
// timestamp(us) 2005980768000000000 (65537-01-01) read as 0001-01-01. The Arrow reader passes
// any int64 here, so check the range before converting; Parquet values arrive inside it.
if (timestamp_micros < MIN_TIMESTAMPTZ_MICROS || timestamp_micros > MAX_TIMESTAMPTZ_MICROS) {
return Status::DataQualityError(
"Decoded TIMESTAMPTZ is outside the Doris 0000-9999 range: micros={}",
timestamp_micros);
}

int64_t epoch_seconds = timestamp_micros / MICROS_PER_SECOND;
int64_t micros_of_second = timestamp_micros % MICROS_PER_SECOND;
if (micros_of_second < 0) {
Expand All @@ -90,10 +103,12 @@ Status append_timestamptz_from_utc_epoch_micros(ColumnTimeStampTz::Container& da
TimestampTzValue timestamp_tz;
timestamp_tz.from_unixtime(epoch_seconds, UTC);
timestamp_tz.set_microsecond(static_cast<uint32_t>(micros_of_second));
// The range check does not make every instant inside it a Doris date: cctz counts year 0 as a
// leap year and Doris does not (is_leap(0) is false), so 0000-02-29 UTC, timestamp(us)
// -62162121600000000, passes the range and packs into an invalid value.
if (!timestamp_tz.is_valid_date()) {
return Status::DataQualityError(
"Decoded TIMESTAMPTZ is outside the Doris 0001-9999 range: micros={}",
timestamp_micros);
return Status::DataQualityError("Decoded TIMESTAMPTZ is not a valid Doris date: micros={}",
timestamp_micros);
}
data.push_back(timestamp_tz);
return Status::OK();
Expand Down
77 changes: 77 additions & 0 deletions be/test/core/data_type_serde/data_type_serde_datetime_v2_test.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -31,10 +31,12 @@

#include "core/assert_cast.h"
#include "core/column/column.h"
#include "core/column/column_array.h"
#include "core/column/column_nullable.h"
#include "core/data_type/common_data_type_serder_test.h"
#include "core/data_type/common_data_type_test.h"
#include "core/data_type/data_type.h"
#include "core/data_type_serde/data_type_array_serde.h"
#include "core/data_type_serde/data_type_datetimev2_serde.h"
#include "core/data_type_serde/data_type_datev2_serde.h"
#include "core/data_type_serde/data_type_nullable_serde.h"
Expand Down Expand Up @@ -589,4 +591,79 @@ TEST_F(DataTypeDateTimeV2SerDeTest, ArrowTimezoneNaiveTimestampIgnoresSessionTim
EXPECT_EQ("1969-12-31 23:59:59.123456", column->get_data()[2].to_string(6));
}

// The civil year of an Arrow timestamp must be checked before it is narrowed into DATETIMEV2:
// unchecked, 65537-01-01 read as 0001-01-01 and 10000-01-01 as a date DATETIMEV2 cannot hold. A
// zoned timestamp is checked in the session time zone, after the shift.
TEST_F(DataTypeDateTimeV2SerDeTest, RejectsArrowTimestampsOutsideDateTimeV2Range) {
const auto read = [](const std::shared_ptr<arrow::DataType>& type, int64_t value,
const cctz::time_zone& timezone, std::string* result) {
arrow::TimestampBuilder builder(type, arrow::default_memory_pool());
EXPECT_TRUE(builder.Append(value).ok());
std::shared_ptr<arrow::Array> array;
EXPECT_TRUE(builder.Finish(&array).ok());
auto column = ColumnDateTimeV2::create();
auto status =
serde_datetime_v2_6->read_column_from_arrow(*column, array.get(), 0, 1, timezone);
if (status.ok()) {
*result = column->get_data()[0].to_string(6);
}
return status;
};
const auto naive = arrow::timestamp(arrow::TimeUnit::MICRO);
const auto utc = cctz::utc_time_zone();
std::string result;
ASSERT_TRUE(read(naive, 253402300799999999, utc, &result).ok());
EXPECT_EQ("9999-12-31 23:59:59.999999", result);
ASSERT_TRUE(read(naive, -62167219200000000, utc, &result).ok());
EXPECT_EQ("0000-01-01 00:00:00.000000", result);
// 10000-01-01, 65537-01-01 (read as 0001-01-01 before the check) and the last microsecond of
// year -1.
for (const int64_t value : {int64_t {253402300800000000}, int64_t {2005980768000000000},
int64_t {-62167219200000001}}) {
SCOPED_TRACE(value);
const auto status = read(naive, value, utc, &result);
EXPECT_FALSE(status.ok());
EXPECT_NE(std::string::npos, status.to_string().find("is outside the DATETIMEV2 range"))
<< status.to_string();
}
// 0000-02-29 is inside the year range, but year 0 is not a leap year in Doris.
const auto leap_day = read(naive, -62162121600000000, utc, &result);
EXPECT_FALSE(leap_day.ok());
EXPECT_NE(std::string::npos, leap_day.to_string().find("is not a valid DATETIMEV2 value"))
<< leap_day.to_string();

// 9999-12-31 15:59:59.999999 UTC is 9999-12-31 23:59:59.999999 at +08:00, and one microsecond
// later is 10000-01-01 there.
cctz::time_zone plus_eight;
ASSERT_TRUE(TimezoneUtils::find_cctz_time_zone("+08:00", plus_eight));
const auto zoned = arrow::timestamp(arrow::TimeUnit::MICRO, "UTC");
ASSERT_TRUE(read(zoned, 253402271999999999, plus_eight, &result).ok());
EXPECT_EQ("9999-12-31 23:59:59.999999", result);
EXPECT_FALSE(read(zoned, 253402272000000000, plus_eight, &result).ok());
}

// A timestamp leaf of ARRAY reaches the same check through the element SerDe.
TEST_F(DataTypeDateTimeV2SerDeTest, RejectsNestedArrowTimestampOutsideDateTimeV2Range) {
arrow::TimestampBuilder values_builder(arrow::timestamp(arrow::TimeUnit::MICRO),
arrow::default_memory_pool());
ASSERT_TRUE(values_builder.Append(2005980768000000000).ok());
std::shared_ptr<arrow::Array> values;
ASSERT_TRUE(values_builder.Finish(&values).ok());
arrow::Int32Builder offsets_builder;
ASSERT_TRUE(offsets_builder.AppendValues({0, 1}).ok());
std::shared_ptr<arrow::Array> offsets;
ASSERT_TRUE(offsets_builder.Finish(&offsets).ok());
auto list = arrow::ListArray::FromArrays(*offsets, *values);
ASSERT_TRUE(list.ok()) << list.status().ToString();

DataTypeArraySerDe serde(std::make_shared<DataTypeNullableSerDe>(serde_datetime_v2_6));
auto column = ColumnArray::create(
ColumnNullable::create(ColumnDateTimeV2::create(), ColumnUInt8::create()));
const auto status = serde.read_column_from_arrow(*column, list.ValueUnsafe().get(), 0, 1,
cctz::utc_time_zone());
EXPECT_FALSE(status.ok());
EXPECT_NE(std::string::npos, status.to_string().find("is outside the DATETIMEV2 range"))
<< status.to_string();
}

} // namespace doris
39 changes: 39 additions & 0 deletions be/test/core/data_type_serde/data_type_serde_timestamptz_test.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -395,4 +395,43 @@ TEST_F(DataTypeTimeStampTzSerDeTest, ArrowTimestampToTimestampTz) {
read_timestamp(arrow::TimeUnit::NANO, -876543211, "Asia/Shanghai",
"1969-12-31 23:59:59.123456+00:00", "1970-01-01 07:59:59.123456+08:00");
}

// An Arrow instant must be range-checked before it is narrowed into TIMESTAMPTZ: unchecked,
// 65537-01-01 UTC wrapped back to 0001-01-01 and passed the validity check after packing.
TEST_F(DataTypeTimeStampTzSerDeTest, RejectsArrowTimestampsOutsideTimestampTzRange) {
const auto read = [](int64_t micros, std::string* result) {
arrow::TimestampBuilder builder(arrow::timestamp(arrow::TimeUnit::MICRO, "UTC"),
arrow::default_memory_pool());
EXPECT_TRUE(builder.Append(micros).ok());
std::shared_ptr<arrow::Array> array;
EXPECT_TRUE(builder.Finish(&array).ok());
auto column = ColumnTimeStampTz::create();
auto status = serde_tz_6->read_column_from_arrow(*column, array.get(), 0, 1,
cctz::utc_time_zone());
if (status.ok()) {
*result = column->get_data()[0].to_string(cctz::utc_time_zone(), 6);
}
return status;
};
std::string result;
ASSERT_TRUE(read(253402300799999999, &result).ok());
EXPECT_EQ("9999-12-31 23:59:59.999999+00:00", result);
ASSERT_TRUE(read(-62167219200000000, &result).ok());
EXPECT_EQ("0000-01-01 00:00:00.000000+00:00", result);
// 10000-01-01, 65537-01-01 (read as 0001-01-01 before the check) and the last microsecond of
// year -1.
for (const int64_t micros : {int64_t {253402300800000000}, int64_t {2005980768000000000},
int64_t {-62167219200000001}}) {
SCOPED_TRACE(micros);
const auto status = read(micros, &result);
EXPECT_FALSE(status.ok());
EXPECT_NE(std::string::npos, status.to_string().find("outside the Doris 0000-9999 range"))
<< status.to_string();
}
// 0000-02-29 UTC is inside the range, but year 0 is not a leap year in Doris.
const auto leap_day = read(-62162121600000000, &result);
EXPECT_FALSE(leap_day.ok());
EXPECT_NE(std::string::npos, leap_day.to_string().find("not a valid Doris date"))
<< leap_day.to_string();
}
} // namespace doris
Loading