Skip to content

[Bug] The UnawareAppendTableCompactionCoordinator may leave some small files can not be compacted #5130

Closed

Description

@Aitozi

Search before asking

  • I searched in the issues and found nothing similar.

Paimon version

1.0

Compute Engine

all

Minimal reproduce step

.

What doesn't meet your expectations?

when compact in batch, there is no age maintained, we should compact the remaining files

Anything else?

No response

Are you willing to submit a PR?

  • I'm willing to submit a PR!

Activity

  1. dragonknightk commented on Nov 16, 2026

    @dragonknightk
    Contributor

    Hi @Aitozi, I'd like to work on this. Here is my plan:

    • In batch mode, after packing, AppendCompactCoordinator puts the files left over in a partition into one extra task when there are at least 2 of them, even if they are fewer than compaction.min.file-num. A single file is not compacted, since rewriting it alone does not reduce the file count.
    • The coordinator already knows the mode from its isStreaming argument, which the Flink compact source, the Flink multi-table compact scan and the Spark compact procedure pass, so no caller or option changes.
    • Streaming is unchanged: leftover files still wait for COMPACT_AGE, because new files can still arrive there.

    In tests, a batch compaction now covers all files, including the leftovers, while a streaming coordinator still keeps them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions