Skip to content

[core] Delete files of aborted writes with bounded concurrency - #11182

Open
dragonknightk wants to merge 2 commits into
apache:masterfrom
dragonknightk:parallelize-abort-file-cleanup
Open

dragonknightk wants to merge 2 commits into
apache:masterfrom
dragonknightk:parallelize-abort-file-cleanup

Conversation

@dragonknightk

@dragonknightk dragonknightk commented Nov 16, 2026 •

Copy link
Copy Markdown

Purpose

Fixes #11111

FileStoreCommitImpl.abort now deletes the files of an aborted write concurrently, bounded by file-operation.thread-num. Before this change, each delete ran on the caller thread one after another, so abort time grew with the file count times the per-delete latency.

  • Abort first collects the paths with the existing rules, then deletes them as one batch. The set of files owned by the aborted write is unchanged, including the [core] Preserve reused compaction files when aborting commits #10582 rule that keeps files reused by compaction.
  • The deletes run on the shared FileOperationThreadPool, the same pool and option that snapshot expiration ([core] Parallelize snapshot expiration file IO #8241) uses. No option or pool is added. Each delete runs with the caller's context class loader, as the sequential abort did.
  • Every path gets a delete attempt, and the caller blocks until all attempts have finished, including when a delete fails or the caller is interrupted.
  • Each delete still goes through FileIO.deleteQuietly, so an IOException is logged and does not throw. Any other exception is rethrown as is after all attempts, with later failures suppressed into the first in input order.
  • Abort bounds the deletes with its own permits, one per file-operation.thread-num (all paths when it is 0 or below, where the shared pool size is the only bound), and waits for a permit with Semaphore.acquireUninterruptibly before each submission. SemaphoredDelegatingExecutor.execute runs a task without a permit when the caller is interrupted while waiting for one, so with its bound alone an interrupt let more deletes run at once than the limit allows.
  • An interrupt on the caller thread does not cancel the cleanup or raise the bound. Abort keeps waiting for permits and for all attempts, and restores the interrupt flag before returning or throwing.

Tests

  • With file-operation.thread-num = 4 and a FileIO that delays each operation, the deletes of one abort overlap, with 2 to 4 in flight. On the code before this change at most one delete is in flight, so this test fails there. Each delete sees the caller's context class loader, and the pool threads keep their own context class loaders after abort.
  • All files owned by the aborted write are deleted, and a file reused by compaction stays.
  • When one delete fails with an IOException, abort does not throw and still deletes every other file.
  • When the caller is interrupted before abort and file-operation.thread-num is below the shared pool size, abort still deletes every file and the interrupt flag stays set.
  • With file-operation.thread-num = 1, the caller is interrupted twice while it waits for a permit with the first delete held. At most one delete is in flight, every file is deleted and the interrupt flag stays set. Two interrupts are needed because an abort that stops waiting for its own permit on the first interrupt still waits in the bounded executor, which runs a task without a permit only on the second. Such an abort and one that relies on the bounded executor alone both fail this test with two deletes in flight. Without an interrupt, one delete is in flight either way.
  • When the caller is interrupted while abort waits for the deletes, abort still waits for and deletes every file and the interrupt flag stays set.
  • In the same interrupted case with two deletes failing with unchecked exceptions, abort deletes every other file, throws the first failure in input order with its original type, and carries the later one as its only suppressed exception. The interrupt flag stays set.

AI Disclosure

Human-led contribution. Grok Bot, working for dragonknightk, assisted with this change, including the code, tests, and description. All submitted content was reviewed by a human before submission.

(๑•̀ㅂ•́)و✧

### Purpose

Closes issue 11111

`FileStoreCommitImpl.abort` now deletes the files of an aborted write concurrently, bounded by `file-operation.thread-num`. Before this change, each delete ran on the caller thread one after another, so abort time grew with the file count times the per-delete latency.

- Abort first collects the paths with the existing rules, then deletes them as one batch. The set of files owned by the aborted write is unchanged, including the PR 10582 rule that keeps files reused by compaction.
- The deletes run on the shared `FileOperationThreadPool`, the same pool and option that snapshot expiration (PR 8241) uses. No option or pool is added. Each delete runs with the caller's context class loader, as the sequential abort did.
- Every path gets a delete attempt, and the caller blocks until all attempts have finished, including when a delete fails or the caller is interrupted.
- Each delete still goes through `FileIO.deleteQuietly`, so an IOException is logged and does not throw. Any other exception is rethrown as is after all attempts, with later failures suppressed into the first in input order.
- An interrupt on the caller thread does not cancel the cleanup. Abort keeps waiting for all attempts and restores the interrupt flag before returning or throwing.

### Tests

- With `file-operation.thread-num = 4` and a FileIO that delays each operation, the deletes of one abort overlap, with 2 to 4 in flight. On the code before this change at most one delete is in flight, so this test fails there. Each delete sees the caller's context class loader, and the pool threads keep their own context class loaders after abort.
- All files owned by the aborted write are deleted, and a file reused by compaction stays.
- When one delete fails with an IOException, abort does not throw and still deletes every other file.
- When the caller is interrupted before abort and `file-operation.thread-num` is below the shared pool size, abort still deletes every file and the interrupt flag stays set.
- When the caller is interrupted while abort waits for the deletes, abort still waits for and deletes every file and the interrupt flag stays set.
- In the same interrupted case with two deletes failing with unchecked exceptions, abort deletes every other file, throws the first failure in input order with its original type, and carries the later one as its only suppressed exception. The interrupt flag stays set.

## AI Disclosure

Human-led contribution. Grok Bot, working for dragonknightk, assisted with this change, including the code, tests, and description. All submitted content was reviewed by a human before submission.

Signed-off-by: KK <37928252+dragonknightk@users.noreply.github.com>

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requirement fit: SUPPORTED. Implementation: FINDINGS.

Bounded parallel abort cleanup has end-to-end value, and the file-ownership rules remain intact. On this head, all 136 FileStoreCommitTest/TableCommitTest cases passed on JDK 11 with normal Maven checks. An additional deterministic public-abort regression exposes the interruption-during-submission case below: the no-interrupt control stays at one delete, while a single interrupt produces two with the configured limit still one. Both cases verify that every owned file is deleted and the caller's interrupt flag is preserved.

The current Flink 2 CI lookup-changelog test failed; its isolated Flink 2.2.0/JDK 11 rerun passed locally. The normal checkpoint path does not call the changed abort method, so I have not attributed that CI failure to this patch or claimed a green full CI run.

Comment on lines +810 to +811
interrupted |= Thread.interrupted();
executor.execute(delete);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Preserve the delete bound when interrupted while acquiring a permit

Clearing the interrupt flag before execute does not cover an interrupt arriving while SemaphoredDelegatingExecutor.execute is blocked in Semaphore.acquire(). That executor catches the interruption and dispatches the command without a permit. A table limit below the shared pool size is a normal configuration, so cancellation can bypass the advertised maximum concurrent file operations.

I reproduced this through public abort using four real owned file paths, file-operation.thread-num=1, and a shared pool with at least four threads. Holding the first delete, waiting until the caller is blocked in semaphore acquisition, then interrupting it once produces two simultaneous deletes; the identical no-interrupt control produces one. Both finish all deletes and preserve the interrupt flag.

Please make bounded submission continue waiting for a permit under interruption (while remembering and restoring the flag), or use a fixed number of delete workers, and cover interruption during submission. The existing tests interrupt before submission or after every delete has already been submitted.

Signed-off-by: KK <37928252+dragonknightk@users.noreply.github.com>
@dragonknightk

dragonknightk commented Nov 16, 2026 •

Copy link
Copy Markdown
Author

@JingsongLi Thanks, confirmed: an interrupt while the caller is blocked in SemaphoredDelegatingExecutor.execute dispatched a delete without a permit.

Fixed by giving abort its own permits: one per file-operation.thread-num, acquired with acquireUninterruptibly before each submission and released when the delete finishes. The interrupt is remembered and restored after all deletes. I took the first option you suggested, with permits owned by abort, because it keeps one task per file and reports failures in input order as before. SemaphoredDelegatingExecutor is unchanged.

Added testAbortKeepsDeleteBoundWhenInterruptedWhileSubmitting with file-operation.thread-num=1 and 4 real files: it holds the first delete, waits until the caller is blocked on a permit, interrupts it twice (waiting for it to block again after each), and checks that at most one delete is in flight, all files are deleted, and the interrupt flag is preserved. A no-interrupt run of the same test is the control. With the previous version it fails with 2 concurrent deletes.

(๑•̀ㅂ•́)و✧

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Parallelize file cleanup for aborted writes

2 participants