Skip to content

[core] Expire TIMESTAMP partitions without time zone with the values-time strategy - #10851

Open
cxzl25 wants to merge 5 commits into
apache:masterfrom
cxzl25:partition-expire-timestamp-values
Open

cxzl25 wants to merge 5 commits into
apache:masterfrom
cxzl25:partition-expire-timestamp-values

Conversation

@cxzl25

@cxzl25 cxzl25 commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Partition expiration with the values-time strategy never expired a partition keyed by a TIMESTAMP column: the value was rendered with Timestamp#toString, whose ISO form (2023-01-01T00:00) the default partition time formatter cannot parse. Render such values as the SQL timestamp string for time extraction, and use the stored partition name when dropping them, as already done for DATE keys.

Tests

Add UT

// Timestamp#toString uses the ISO 'T' separator, which the default
// partition time formatter cannot parse. Render it as the SQL string.
values[index] =
DateTimeUtils.formatTimestamp(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This handles TIMESTAMP_LTZ as if it were a timezone-free TIMESTAMP. The standard TimestampToStringCastRule formats LTZ values with TimeZone.getDefault(), so in a non-UTC timezone this can shift the extracted partition time-and even its date-relative to the logical/stored current-format partition value. Could we use the timezone-aware overload for TIMESTAMP_WITH_LOCAL_TIME_ZONE and add a regression test with a non-UTC default timezone?

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requirement fit: SUPPORTED. Implementation: FINDINGS.

I independently confirmed the already-reported LTZ timezone problem in a real table: with Asia/Shanghai, current partition names, values-time expiration and one-day retention, an instant stored as local partition 2023-01-02 04:00:00 is deleted at local 2023-01-03 02:00:00, when it is only 22 hours old. Both that partition and a legitimately older control partition disappear. The otherwise identical NTZ/Shanghai and LTZ/UTC controls pass. This is a data-retention correctness blocker; please keep the LTZ conversion consistent with the existing timezone-aware cast. See the existing thread: #10851 (comment).

There is also one independent, baseline-verified formatter regression below. Normal JDK 8 core verify passes the 30 existing expiration/extractor cases. The extra head tests have the three failures described here; with both production files restored to merge-base eb4958c, the exact two custom-formatter cases pass. The worktree has been restored to the reviewed head.

// Timestamp#toString uses the ISO 'T' separator, which the default
// partition time formatter cannot parse. Render it as the SQL string.
values[index] =
DateTimeUtils.formatTimestamp(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Preserve existing explicit timestamp formatter inputs

The unconditional SQL rendering changes the input of an existing configured partition.timestamp-formatter. A TIMESTAMP(0) partition with partition.timestamp-pattern=$f0 and formatter yyyy-MM-dd'T'HH:mm previously receives Timestamp.toString() values such as 2023-01-01T00:00, so expiration works. It now receives 2023-01-01 00:00:00; the configured extractor cannot parse it, and the expired partition is silently retained.

I reproduced actual writes, expiration and readback for January 1/January 5 partitions at a January 6 check with two-day retention. Both legacy and current partition-name modes retain the old partition on this head; the identical cases expire it successfully with the two production files restored to the PR merge-base. Please keep explicit formatter inputs compatible, or adapt timestamp extraction to the configured formatter, while retaining the stored names for deletion.

@cxzl25
cxzl25 force-pushed the partition-expire-timestamp-values branch from 47635b1 to f6a0c24 Compare November 2, 2026 13:19

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requirement fit: SUPPORTED. Implementation: FINDINGS.

The current-name LTZ timezone case and explicit NTZ formatter cases from the earlier review are fixed. Normal JDK 8 checks and all 46 expiration/extractor tests pass on this head. Two added real-table legacy-LTZ checks expose a remaining problem: Shanghai retains the expired partition, and New York deletes a fresh partition instead of the selected expired one. The latter is detailed below. Reviewer-only test changes were removed; the original 46 tests passed again.

for (String value : partitionSpec.values()) {
if (array[index] instanceof LocalDate) {
if (array[index] != null && strategy.isNormalizedForTimeExtraction(index)) {
array[index] = value;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Preserve the LTZ partition key when dropping legacy names

The legacy LTZ name is a UTC Timestamp.toString() spelling, but commit.dropPartitions reparses the supplied string as local time. Replacing the normalized value with this name therefore targets a different key. I reproduced actual writes/expiry/readback with TIMESTAMP_LTZ(0), legacy partition names and America/New_York: write instants 2023-01-01T00:00Z and 2023-01-01T05:00Z, then expire at local January 1 22:00 with one-day retention. The returned selection correctly contains only legacy name 2023-01-01T00:00 (the 27-hour-old partition), but readback leaves that old row and removes the 05:00Z row, which is only 22 hours old. The legacy name was reparsed as midnight New York, i.e. 05:00Z. The new normalization activates this deletion path where the old default extractor skipped the ISO values. Please make truncation round-trip the original typed LTZ key (or use a legacy-aware conversion), keeping stored/HMS spelling separate where necessary, and cover a negative-offset timezone with two such partitions. This is a logical deletion of the wrong live partition, not a claim of immediate physical file erasure.

@cxzl25 cxzl25 changed the title [core] Expire TIMESTAMP partitions with the values-time strategy [core] Expire TIMESTAMP partitions without time zone with the values-time strategy Nov 4, 2026

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The scope is now explicitly NTZ and the direct legacy-LTZ reproduction is avoided. Normal JDK 8 verify passes the 47 expiration/extractor cases. I also ran actual-table expiration/readback cases on this head and with the three production files from base bf6c81cbc8 on a separate classpath. They exposed the wrong-partition deletion below.

for (String value : partitionSpec.values()) {
if (array[index] instanceof LocalDate) {
if (array[index] != null && strategy.isNormalizedForTimeExtraction(index)) {
array[index] = value;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Preserve the selected typed partition key when deleting locally. The generated path spelling is not always safe to parse back as a partition spec. With current names, partition.default-name='1970-01-01 00:00:00', an NTZ epoch partition, a NULL partition, and the existing ISO custom formatter, this head selects the epoch partition but deletes the NULL row. The same test against the base deletes epoch and retains NULL. A second real-table case uses (NTZ, LTZ) keys, the default extractor, New York timezone, and an expiration cap of 1. It reports the LTZ 00:00Z partition but deletes the 05:00Z partition because the LTZ ISO string is reparsed in local time. Keep stored/HMS names separate from the original BinaryRow used for local deletion, and cover both readback cases.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rechecked b79af8f: the new binary-key deletion fixes the local path, and 156 expiration/extractor/commit tests pass with normal JDK 8 Maven checks. The metastore path still has the same P1: doBatchExpire delegates names to partitionModification.dropPartitions, which reaches HiveCatalog.dropPartitions -> super.dropPartitions -> FileStoreCommit.dropPartitions and reparses the key.

This reproduces using the PR's own new tests: set metastore.partitioned-table=true in createPartitionedTable. testExpireTimestampPartitionNamedAsDefaultPartition leaves the epoch row and deletes NULL; both testExpireDropsSelectedLocalZonedTimestampPartition naming modes leave 00:00Z and delete 05:00Z. These are actual commit/expiration/readback tests using the existing catalog callback, not just checks of returned partition names.

I also ran the epoch/NULL case with the three expiration production files restored to merge-base d7b521d: it deletes epoch and retains NULL. The identical data assertion fails on this head (only the returned-name assertion is omitted in both runs because base reports ISO and head reports SQL). Please propagate the exact selected typed key through the metastore-backed data deletion path too, while retaining stored names for HMS metadata operations; selecting the right key only in the local branch leaves this regression active.

cxzl25 and others added 4 commits November 6, 2026 16:51
Partition expiration with the values-time strategy never expired a
partition keyed by a TIMESTAMP or TIMESTAMP_LTZ column: the value was
rendered with Timestamp#toString, whose ISO form (2023-01-01T00:00) the
default partition time formatter cannot parse. Render such values as the
SQL timestamp string for time extraction, and use the stored partition
name when dropping them, as already done for DATE keys.
…without time zone

A legacy TIMESTAMP_LTZ partition name is a UTC Timestamp#toString spelling,
while partition specs passed to dropPartitions are parsed in the default time
zone, so expiring such partitions outside UTC could drop a different, live
partition. Reliable LTZ expiration needs dropping by the stored partition
row rather than by a re-parsed string, which is out of scope here.

Normalize only TIMESTAMP without time zone values, keep TIMESTAMP_LTZ on its
previous behavior, and cover that LTZ partitions are neither expired nor
dropped by default in a non-UTC zone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cxzl25
cxzl25 force-pushed the partition-expire-timestamp-values branch from 45a1b8d to b79af8f Compare November 6, 2026 10:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants