Skip to content

[core] Support '::epoch_day' in partition.timestamp-pattern - #11140

Open
JacobSMoller wants to merge 1 commit into
apache:masterfrom
JacobSMoller:date-partition-epoch-day
Open

JacobSMoller wants to merge 1 commit into
apache:masterfrom
JacobSMoller:date-partition-epoch-day

Conversation

@JacobSMoller

Copy link
Copy Markdown

Purpose

Attempted solution for #6439 which was closed by #6664 but that was reverted again in here but the issue stayed closed even though the functionality never made it in.

Tests

tests added in PartitionTimeExtractorTest.java

Addtional info

We have run it like this in production for a while.

we do something like

SELECT
  id, trace_id, timestamp_received, ip, url, user_agent, `method`, body, manual_request_id,
  SECONDS_TO_DATE_PARTITION(timestamp_received.seconds) AS partition_date,  -- DATE
  SECONDS_TO_HOUR_PARTITION(timestamp_received.seconds) AS partition_hour   -- INT
FROM data_stream ds
WHERE ...

SECONDS_TO_DATE_PARTITION and SECONDS_TO_HOUR_PARTITION is used to convert protobuf WKT timestamp to something human readable.

And then we do

"partition.expiration-time": "181 d"
"partition.timestamp-pattern": "$partition_date::epoch_day $partition_hour:00:00"
"partitionKeys":
  - "partition_date"
  - "partition_hour"

e.g. in plain flink sql

-- Flink SQL
CREATE TABLE events (
  id STRING,
  ts TIMESTAMP(3),
  partition_date DATE,
  partition_hour INT
) PARTITIONED BY (partition_date, partition_hour) WITH (
  'partition.expiration-time' = '181 d',
  'partition.timestamp-pattern' = '$partition_date::epoch_day $partition_hour:00:00'
);

…partition fields

The value of a DATE partition field is the number of days since 1970-01-01, which no
partition.timestamp-formatter can parse, so partition expiration with values-time never drops such
a partition. A pattern variable followed by '::epoch_day' (e.g. '$dt::epoch_day $hour:00:00')
turns that number into yyyy-MM-dd before formatting. A value that is not a whole number, such as a
date read from a partition path, is used as it is. Fields without the marker are unchanged, so an
INT partition such as an hour or a yyyyMMdd number keeps its meaning.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requirement fit: PIVOT. Implementation: no code defect found in the exercised extraction/expiration paths.

The ordinary DATE expiration issue in #6439 is already fixed by merged #10464 (4dcd6c4e27, an ancestor of this PR's base): DATE values become LocalDate before extraction, and the existing DATE/hour and real write/expire/read tests work with plain $dt.

There is a distinct useful consumer worth preserving: Flink mark-done reads raw directory specs. I independently verified the production chain InternalRowPartitionComputer(legacy=true) -> PartitionMarkDoneTrigger -> SuccessFileMarkDoneAction using a DATE/hour partition. The generated dt=19723/hour=12 path is discarded with plain $dt $hour:00:00 and produces no _SUCCESS; with $dt::epoch_day $hour:00:00 it is marked done and _SUCCESS is created. Both patterns work for the ISO-path control.

Please align the PR purpose/documentation with this demonstrated remaining scenario and add that public trigger/marker regression to the PR. Also clarify the supported scope: PartitionTimeResolver.parsePattern currently treats ::epoch_day as a literal, so this pattern does not extend to chain-table delta resolution unless that parser is updated and tested.

Validation: 51 core extraction/resolver/expiration tests passed with normal Maven checks and git diff --check. The mark-done check was a standalone probe compiling the reviewed core and production trigger with Flink 1 dependencies, not a full Flink job-suite run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants