Repository navigation
[core] Delete files of aborted writes with bounded concurrency - #11182
dragonknightk wants to merge 2 commits into
Conversation
### Purpose Closes issue 11111 `FileStoreCommitImpl.abort` now deletes the files of an aborted write concurrently, bounded by `file-operation.thread-num`. Before this change, each delete ran on the caller thread one after another, so abort time grew with the file count times the per-delete latency. - Abort first collects the paths with the existing rules, then deletes them as one batch. The set of files owned by the aborted write is unchanged, including the PR 10582 rule that keeps files reused by compaction. - The deletes run on the shared `FileOperationThreadPool`, the same pool and option that snapshot expiration (PR 8241) uses. No option or pool is added. Each delete runs with the caller's context class loader, as the sequential abort did. - Every path gets a delete attempt, and the caller blocks until all attempts have finished, including when a delete fails or the caller is interrupted. - Each delete still goes through `FileIO.deleteQuietly`, so an IOException is logged and does not throw. Any other exception is rethrown as is after all attempts, with later failures suppressed into the first in input order. - An interrupt on the caller thread does not cancel the cleanup. Abort keeps waiting for all attempts and restores the interrupt flag before returning or throwing. ### Tests - With `file-operation.thread-num = 4` and a FileIO that delays each operation, the deletes of one abort overlap, with 2 to 4 in flight. On the code before this change at most one delete is in flight, so this test fails there. Each delete sees the caller's context class loader, and the pool threads keep their own context class loaders after abort. - All files owned by the aborted write are deleted, and a file reused by compaction stays. - When one delete fails with an IOException, abort does not throw and still deletes every other file. - When the caller is interrupted before abort and `file-operation.thread-num` is below the shared pool size, abort still deletes every file and the interrupt flag stays set. - When the caller is interrupted while abort waits for the deletes, abort still waits for and deletes every file and the interrupt flag stays set. - In the same interrupted case with two deletes failing with unchecked exceptions, abort deletes every other file, throws the first failure in input order with its original type, and carries the later one as its only suppressed exception. The interrupt flag stays set. ## AI Disclosure Human-led contribution. Grok Bot, working for dragonknightk, assisted with this change, including the code, tests, and description. All submitted content was reviewed by a human before submission. Signed-off-by: KK <37928252+dragonknightk@users.noreply.github.com>
JingsongLi
left a comment
There was a problem hiding this comment.
Requirement fit: SUPPORTED. Implementation: FINDINGS.
Bounded parallel abort cleanup has end-to-end value, and the file-ownership rules remain intact. On this head, all 136 FileStoreCommitTest/TableCommitTest cases passed on JDK 11 with normal Maven checks. An additional deterministic public-abort regression exposes the interruption-during-submission case below: the no-interrupt control stays at one delete, while a single interrupt produces two with the configured limit still one. Both cases verify that every owned file is deleted and the caller's interrupt flag is preserved.
The current Flink 2 CI lookup-changelog test failed; its isolated Flink 2.2.0/JDK 11 rerun passed locally. The normal checkpoint path does not call the changed abort method, so I have not attributed that CI failure to this patch or claimed a green full CI run.
| interrupted |= Thread.interrupted(); | ||
| executor.execute(delete); |
There was a problem hiding this comment.
[P2] Preserve the delete bound when interrupted while acquiring a permit
Clearing the interrupt flag before execute does not cover an interrupt arriving while SemaphoredDelegatingExecutor.execute is blocked in Semaphore.acquire(). That executor catches the interruption and dispatches the command without a permit. A table limit below the shared pool size is a normal configuration, so cancellation can bypass the advertised maximum concurrent file operations.
I reproduced this through public abort using four real owned file paths, file-operation.thread-num=1, and a shared pool with at least four threads. Holding the first delete, waiting until the caller is blocked in semaphore acquisition, then interrupting it once produces two simultaneous deletes; the identical no-interrupt control produces one. Both finish all deletes and preserve the interrupt flag.
Please make bounded submission continue waiting for a permit under interruption (while remembering and restoring the flag), or use a fixed number of delete workers, and cover interruption during submission. The existing tests interrupt before submission or after every delete has already been submitted.
Signed-off-by: KK <37928252+dragonknightk@users.noreply.github.com>
|
@JingsongLi Thanks, confirmed: an interrupt while the caller is blocked in Fixed by giving abort its own permits: one per Added (๑•̀ㅂ•́)و✧ |
Purpose
Fixes #11111
FileStoreCommitImpl.abortnow deletes the files of an aborted write concurrently, bounded byfile-operation.thread-num. Before this change, each delete ran on the caller thread one after another, so abort time grew with the file count times the per-delete latency.FileOperationThreadPool, the same pool and option that snapshot expiration ([core] Parallelize snapshot expiration file IO #8241) uses. No option or pool is added. Each delete runs with the caller's context class loader, as the sequential abort did.FileIO.deleteQuietly, so an IOException is logged and does not throw. Any other exception is rethrown as is after all attempts, with later failures suppressed into the first in input order.file-operation.thread-num(all paths when it is 0 or below, where the shared pool size is the only bound), and waits for a permit withSemaphore.acquireUninterruptiblybefore each submission.SemaphoredDelegatingExecutor.executeruns a task without a permit when the caller is interrupted while waiting for one, so with its bound alone an interrupt let more deletes run at once than the limit allows.Tests
file-operation.thread-num = 4and a FileIO that delays each operation, the deletes of one abort overlap, with 2 to 4 in flight. On the code before this change at most one delete is in flight, so this test fails there. Each delete sees the caller's context class loader, and the pool threads keep their own context class loaders after abort.file-operation.thread-numis below the shared pool size, abort still deletes every file and the interrupt flag stays set.file-operation.thread-num = 1, the caller is interrupted twice while it waits for a permit with the first delete held. At most one delete is in flight, every file is deleted and the interrupt flag stays set. Two interrupts are needed because an abort that stops waiting for its own permit on the first interrupt still waits in the bounded executor, which runs a task without a permit only on the second. Such an abort and one that relies on the bounded executor alone both fail this test with two deletes in flight. Without an interrupt, one delete is in flight either way.AI Disclosure
Human-led contribution. Grok Bot, working for dragonknightk, assisted with this change, including the code, tests, and description. All submitted content was reviewed by a human before submission.
(๑•̀ㅂ•́)و✧