Skip to content

[flink] Honor partition_idle_time in Data Evolution compaction - #11197

Open
QuakeWang wants to merge 1 commit into
apache:masterfrom
QuakeWang:flink-de-idle-compaction
Open

QuakeWang wants to merge 1 commit into
apache:masterfrom
QuakeWang:flink-de-idle-compaction

Conversation

@QuakeWang

Copy link
Copy Markdown
Member

Purpose

Flink SQL and CLI parse partition_idle_time. The Data Evolution compaction path drops it, so recently written partitions can still be compacted.

For single-table batch compaction without sorting, read each partition's latest file creation time from the planning snapshot and apply the idle cutoff together with partitions or where.

Tests

Added SQL and CLI regression coverage for partition filtering and writes after job construction. All 44 targeted Flink tests and the documentation build passed.

Flink SQL and CLI parse partition_idle_time. The Data Evolution branch
drops it, so recently written partitions can still be compacted.

Filter eligible partitions by file creation time from the planning
snapshot and apply the requested partition predicate.

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant