Repository navigation
Conversation
| // Timestamp#toString uses the ISO 'T' separator, which the default | ||
| // partition time formatter cannot parse. Render it as the SQL string. | ||
| values[index] = | ||
| DateTimeUtils.formatTimestamp( |
There was a problem hiding this comment.
This handles TIMESTAMP_LTZ as if it were a timezone-free TIMESTAMP. The standard TimestampToStringCastRule formats LTZ values with TimeZone.getDefault(), so in a non-UTC timezone this can shift the extracted partition time-and even its date-relative to the logical/stored current-format partition value. Could we use the timezone-aware overload for TIMESTAMP_WITH_LOCAL_TIME_ZONE and add a regression test with a non-UTC default timezone?
JingsongLi
left a comment
There was a problem hiding this comment.
Requirement fit: SUPPORTED. Implementation: FINDINGS.
I independently confirmed the already-reported LTZ timezone problem in a real table: with Asia/Shanghai, current partition names, values-time expiration and one-day retention, an instant stored as local partition 2023-01-02 04:00:00 is deleted at local 2023-01-03 02:00:00, when it is only 22 hours old. Both that partition and a legitimately older control partition disappear. The otherwise identical NTZ/Shanghai and LTZ/UTC controls pass. This is a data-retention correctness blocker; please keep the LTZ conversion consistent with the existing timezone-aware cast. See the existing thread: #10851 (comment).
There is also one independent, baseline-verified formatter regression below. Normal JDK 8 core verify passes the 30 existing expiration/extractor cases. The extra head tests have the three failures described here; with both production files restored to merge-base eb4958c, the exact two custom-formatter cases pass. The worktree has been restored to the reviewed head.
| // Timestamp#toString uses the ISO 'T' separator, which the default | ||
| // partition time formatter cannot parse. Render it as the SQL string. | ||
| values[index] = | ||
| DateTimeUtils.formatTimestamp( |
There was a problem hiding this comment.
[P2] Preserve existing explicit timestamp formatter inputs
The unconditional SQL rendering changes the input of an existing configured partition.timestamp-formatter. A TIMESTAMP(0) partition with partition.timestamp-pattern=$f0 and formatter yyyy-MM-dd'T'HH:mm previously receives Timestamp.toString() values such as 2023-01-01T00:00, so expiration works. It now receives 2023-01-01 00:00:00; the configured extractor cannot parse it, and the expired partition is silently retained.
I reproduced actual writes, expiration and readback for January 1/January 5 partitions at a January 6 check with two-day retention. Both legacy and current partition-name modes retain the old partition on this head; the identical cases expire it successfully with the two production files restored to the PR merge-base. Please keep explicit formatter inputs compatible, or adapt timestamp extraction to the configured formatter, while retaining the stored names for deletion.
47635b1 to
f6a0c24
Compare
JingsongLi
left a comment
There was a problem hiding this comment.
Requirement fit: SUPPORTED. Implementation: FINDINGS.
The current-name LTZ timezone case and explicit NTZ formatter cases from the earlier review are fixed. Normal JDK 8 checks and all 46 expiration/extractor tests pass on this head. Two added real-table legacy-LTZ checks expose a remaining problem: Shanghai retains the expired partition, and New York deletes a fresh partition instead of the selected expired one. The latter is detailed below. Reviewer-only test changes were removed; the original 46 tests passed again.
| for (String value : partitionSpec.values()) { | ||
| if (array[index] instanceof LocalDate) { | ||
| if (array[index] != null && strategy.isNormalizedForTimeExtraction(index)) { | ||
| array[index] = value; |
There was a problem hiding this comment.
[P1] Preserve the LTZ partition key when dropping legacy names
The legacy LTZ name is a UTC Timestamp.toString() spelling, but commit.dropPartitions reparses the supplied string as local time. Replacing the normalized value with this name therefore targets a different key. I reproduced actual writes/expiry/readback with TIMESTAMP_LTZ(0), legacy partition names and America/New_York: write instants 2023-01-01T00:00Z and 2023-01-01T05:00Z, then expire at local January 1 22:00 with one-day retention. The returned selection correctly contains only legacy name 2023-01-01T00:00 (the 27-hour-old partition), but readback leaves that old row and removes the 05:00Z row, which is only 22 hours old. The legacy name was reparsed as midnight New York, i.e. 05:00Z. The new normalization activates this deletion path where the old default extractor skipped the ISO values. Please make truncation round-trip the original typed LTZ key (or use a legacy-aware conversion), keeping stored/HMS spelling separate where necessary, and cover a negative-offset timezone with two such partitions. This is a logical deletion of the wrong live partition, not a claim of immediate physical file erasure.
JingsongLi
left a comment
There was a problem hiding this comment.
The scope is now explicitly NTZ and the direct legacy-LTZ reproduction is avoided. Normal JDK 8 verify passes the 47 expiration/extractor cases. I also ran actual-table expiration/readback cases on this head and with the three production files from base bf6c81cbc8 on a separate classpath. They exposed the wrong-partition deletion below.
| for (String value : partitionSpec.values()) { | ||
| if (array[index] instanceof LocalDate) { | ||
| if (array[index] != null && strategy.isNormalizedForTimeExtraction(index)) { | ||
| array[index] = value; |
There was a problem hiding this comment.
[P1] Preserve the selected typed partition key when deleting locally. The generated path spelling is not always safe to parse back as a partition spec. With current names, partition.default-name='1970-01-01 00:00:00', an NTZ epoch partition, a NULL partition, and the existing ISO custom formatter, this head selects the epoch partition but deletes the NULL row. The same test against the base deletes epoch and retains NULL. A second real-table case uses (NTZ, LTZ) keys, the default extractor, New York timezone, and an expiration cap of 1. It reports the LTZ 00:00Z partition but deletes the 05:00Z partition because the LTZ ISO string is reparsed in local time. Keep stored/HMS names separate from the original BinaryRow used for local deletion, and cover both readback cases.
There was a problem hiding this comment.
Rechecked b79af8f: the new binary-key deletion fixes the local path, and 156 expiration/extractor/commit tests pass with normal JDK 8 Maven checks. The metastore path still has the same P1: doBatchExpire delegates names to partitionModification.dropPartitions, which reaches HiveCatalog.dropPartitions -> super.dropPartitions -> FileStoreCommit.dropPartitions and reparses the key.
This reproduces using the PR's own new tests: set metastore.partitioned-table=true in createPartitionedTable. testExpireTimestampPartitionNamedAsDefaultPartition leaves the epoch row and deletes NULL; both testExpireDropsSelectedLocalZonedTimestampPartition naming modes leave 00:00Z and delete 05:00Z. These are actual commit/expiration/readback tests using the existing catalog callback, not just checks of returned partition names.
I also ran the epoch/NULL case with the three expiration production files restored to merge-base d7b521d: it deletes epoch and retains NULL. The identical data assertion fails on this head (only the returned-name assertion is omitted in both runs because base reports ISO and head reports SQL). Please propagate the exact selected typed key through the metastore-backed data deletion path too, while retaining stored names for HMS metadata operations; selecting the right key only in the local branch leaves this regression active.
Partition expiration with the values-time strategy never expired a partition keyed by a TIMESTAMP or TIMESTAMP_LTZ column: the value was rendered with Timestamp#toString, whose ISO form (2023-01-01T00:00) the default partition time formatter cannot parse. Render such values as the SQL timestamp string for time extraction, and use the stored partition name when dropping them, as already done for DATE keys.
…without time zone A legacy TIMESTAMP_LTZ partition name is a UTC Timestamp#toString spelling, while partition specs passed to dropPartitions are parsed in the default time zone, so expiring such partitions outside UTC could drop a different, live partition. Reliable LTZ expiration needs dropping by the stored partition row rather than by a re-parsed string, which is out of scope here. Normalize only TIMESTAMP without time zone values, keep TIMESTAMP_LTZ on its previous behavior, and cover that LTZ partitions are neither expired nor dropped by default in a non-UTC zone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
45a1b8d to
b79af8f
Compare
Purpose
Partition expiration with the values-time strategy never expired a partition keyed by a TIMESTAMP column: the value was rendered with Timestamp#toString, whose ISO form (2023-01-01T00:00) the default partition time formatter cannot parse. Render such values as the SQL timestamp string for time extraction, and use the stored partition name when dropping them, as already done for DATE keys.
Tests
Add UT