Repository navigation
[core] Support '::epoch_day' in partition.timestamp-pattern - #11140
JacobSMoller wants to merge 1 commit into
Conversation
…partition fields The value of a DATE partition field is the number of days since 1970-01-01, which no partition.timestamp-formatter can parse, so partition expiration with values-time never drops such a partition. A pattern variable followed by '::epoch_day' (e.g. '$dt::epoch_day $hour:00:00') turns that number into yyyy-MM-dd before formatting. A value that is not a whole number, such as a date read from a partition path, is used as it is. Fields without the marker are unchanged, so an INT partition such as an hour or a yyyyMMdd number keeps its meaning. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
JingsongLi
left a comment
There was a problem hiding this comment.
Requirement fit: PIVOT. Implementation: no code defect found in the exercised extraction/expiration paths.
The ordinary DATE expiration issue in #6439 is already fixed by merged #10464 (4dcd6c4e27, an ancestor of this PR's base): DATE values become LocalDate before extraction, and the existing DATE/hour and real write/expire/read tests work with plain $dt.
There is a distinct useful consumer worth preserving: Flink mark-done reads raw directory specs. I independently verified the production chain InternalRowPartitionComputer(legacy=true) -> PartitionMarkDoneTrigger -> SuccessFileMarkDoneAction using a DATE/hour partition. The generated dt=19723/hour=12 path is discarded with plain $dt $hour:00:00 and produces no _SUCCESS; with $dt::epoch_day $hour:00:00 it is marked done and _SUCCESS is created. Both patterns work for the ISO-path control.
Please align the PR purpose/documentation with this demonstrated remaining scenario and add that public trigger/marker regression to the PR. Also clarify the supported scope: PartitionTimeResolver.parsePattern currently treats ::epoch_day as a literal, so this pattern does not extend to chain-table delta resolution unless that parser is updated and tested.
Validation: 51 core extraction/resolver/expiration tests passed with normal Maven checks and git diff --check. The mark-done check was a standalone probe compiling the reviewed core and production trigger with Flink 1 dependencies, not a full Flink job-suite run.
Purpose
Attempted solution for #6439 which was closed by #6664 but that was reverted again in here but the issue stayed closed even though the functionality never made it in.
Tests
tests added in PartitionTimeExtractorTest.java
Addtional info
We have run it like this in production for a while.
we do something like
SECONDS_TO_DATE_PARTITION and SECONDS_TO_HOUR_PARTITION is used to convert protobuf WKT timestamp to something human readable.
And then we do
e.g. in plain flink sql