You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Bug] The UnawareAppendTableCompactionCoordinator may leave some small files can not be compacted #5130
Hi @Aitozi, I'd like to work on this. Here is my plan:
In batch mode, after packing, AppendCompactCoordinator puts the files left over in a partition into one extra task when there are at least 2 of them, even if they are fewer than compaction.min.file-num. A single file is not compacted, since rewriting it alone does not reduce the file count.
The coordinator already knows the mode from its isStreaming argument, which the Flink compact source, the Flink multi-table compact scan and the Spark compact procedure pass, so no caller or option changes.
Streaming is unchanged: leftover files still wait for COMPACT_AGE, because new files can still arrive there.
In tests, a batch compaction now covers all files, including the leftovers, while a streaming coordinator still keeps them.
Activity
dragonknightk commentedon Nov 16, 2026
Hi @Aitozi, I'd like to work on this. Here is my plan:
AppendCompactCoordinatorputs the files left over in a partition into one extra task when there are at least 2 of them, even if they are fewer thancompaction.min.file-num. A single file is not compacted, since rewriting it alone does not reduce the file count.isStreamingargument, which the Flink compact source, the Flink multi-table compact scan and the Spark compact procedure pass, so no caller or option changes.COMPACT_AGE, because new files can still arrive there.In tests, a batch compaction now covers all files, including the leftovers, while a streaming coordinator still keeps them.