Skip to content

feat(transaction): add overwrite action on MergingSnapshotProducer (stacks on #3564, continues #2185) - #3300

Open
jopdorp wants to merge 58 commits into
apache:mainfrom
jopdorp:feat/overwrite-action
Open

jopdorp wants to merge 58 commits into
apache:mainfrom
jopdorp:feat/overwrite-action

Conversation

@jopdorp

@jopdorp jopdorp commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Part of #2186.

What changes are included in this PR?

Stacks on #3564. Continues #2185 by @glitchy.

OverwriteAction adds data files and removes listed data files by path in one snapshot. It builds a MergingIntent and the MergingSnapshotProducer from #3564 does the manifest work, so against that branch this PR is the action, its tests and the wiring.

  • The operation is delete, append or overwrite depending on what the commit does.
  • A delete path that is not live in the table fails the commit.
  • With check_duplicate, an added path that is already live is rejected, unless the same overwrite deletes it.

Are these changes tested?

Four tests in transaction::overwrite::tests: a partial overwrite followed by a delete with the summary counts, a missing delete path is rejected, a path deleted in the same overwrite can be re-added, and an added path that is live and not deleted is rejected. Manifest rewriting, schema and spec evolution, retries and row lineage are covered by the producer's tests in #3564. cargo test -p iceberg --lib, the public API check, fmt and clippy with -D warnings are clean.

AI Disclosure

The restack, the review fixes and their tests were drafted with an AI coding assistant, then reviewed, run and checked against the Java implementation by us. Worth a close look: which operation a commit reports, and the duplicate check exemption for a path the same overwrite deletes.

glitchy and others added 24 commits June 29, 2026 22:21
Adds an OverwriteAction to Transaction for atomic replacement of data files
via an Operation::Overwrite snapshot. add_data_files() stages new files;
delete_data_files() rewrites affected manifests with matching entries marked
ManifestStatus::Deleted (copy-on-write). Manifests are loaded through the
Table::manifest_list_reader abstraction.
The SnapshotProducer precondition check blocked snapshots with only
deleted files and no added files or snapshot properties. A delete-only
Overwrite snapshot is valid per the Iceberg spec — the existing manifests
are rewritten with the target entries marked as ManifestStatus::Deleted.

Relax the check to also allow the case when deleted_data_files is
non-empty, so callers can clear a table without simultaneously adding
new files.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Import Manifest and DataFile at the top instead of using fully-qualified
crate::spec:: paths inline in a signature and a test helper.
# Conflicts:
#	crates/iceberg/public-api.txt
#	crates/iceberg/src/transaction/append.rs
#	crates/iceberg/src/transaction/mod.rs
A manifest rewritten by a second overwrite fell through to add_existing_entry
for entries that were already Deleted, forcing their status back to Existing
and resurrecting the files as live data. Guard on is_alive() first and re-add
prior tombstones as Deleted with their original snapshot id, matching Java's
filterManifestWithDeletedFiles, which only re-adds live entries.
The rewrite used the table's current schema and default partition spec, so
after a schema or partition evolution the Avro header carried the wrong
schema-id and partition-spec-id and readers interpreted the partition values
against the wrong mapping. Take both from the manifest being rewritten, as
Java does via newManifestWriter(reader.spec()).
Both manifest filters dropped a manifest whose entries were all Deleted, which
lets the removed files reappear as live data. This is the same fix FastAppend
took in apache#2545 for apache#2148, applied to the no-deletes fast path and to the rewrite
loop.
The summary counted every file the caller passed to delete_data_files, so a
path that was never committed or was passed twice inflated deleted-data-files
and deleted-records. The rewrite pass now records the entries it marked
Deleted, with the schema and partition spec of the manifest each was recorded
in, and the summary counts those.

Two changes follow from it. The summary is built after the manifests rather
than before, since the counts are only known once the rewrite has run. And the
full-table truncate path is now gated on an operation hook that a partial
overwrite answers false, instead of on the operation being Overwrite; with
truncate off, update_totals needs a saturating subtraction so an overwrite that
deletes more than the previous total does not underflow.
The rewrite hardcoded {location}/metadata and a fresh Uuid::now_v7, so a table
with a custom write.metadata.path got its rewritten manifests in the wrong
prefix, and orphan cleanup could not find them after a failed commit because
the name was not tied to the commit. Reuse the path SnapshotProducer already
builds from metadata_location(), the commit uuid and the manifest counter; the
counter becomes an AtomicU64 because the rewrite runs behind a shared
reference.
rewrite_manifest built its own plain output file, so on an encrypted table the
rewritten manifest went out in plaintext while its key metadata said otherwise.
It now goes through SnapshotProducer::new_manifest_writer, which already
branches on the table's encryption manager; the writer takes the schema and
partition spec as arguments so the rewrite can keep the ones the manifest was
written under. That leaves new_manifest_path with no caller outside this
module, so it becomes private again.
add_deleted_entry set the status but left snapshot_id to the caller, while
add_delete_entry right above it stamps both; the two differed only by tense and
a tombstone written with the wrong snapshot_id misattributes the delete, which
expire_snapshots then acts on. Taking the reviewer's second option: rename
rather than stamp, because carrying an earlier tombstone forward has to keep
its original snapshot_id, and document that the caller owns the field.

Named add_tombstone_entry rather than something closer to main's
add_delete_entry (apache#2367) precisely so the two cannot be confused.
A delete-only overwrite reported Operation::Overwrite, so tools keying off the
summary read it as a replacement. Match Java's BaseOverwriteFiles: DELETE when
there are only deletes, APPEND when there are only adds, OVERWRITE when there
are both. test_overwrite_basic adds without deleting, so its expected operation
becomes APPEND.

Also names deleted data files in the precondition error, which has allowed a
delete-only commit since the guard was relaxed.
is_err() alone passes on any error, so the test would stay green if the commit
started failing earlier for an unrelated reason. Assert the kind and the
message instead.
An overwrite silently ignored a delete path that was not live in any manifest,
so a typo or an already-deleted file committed as a no-op delete. Java's
BaseOverwriteFiles validates the same thing and fails with "Missing required
files to delete"; the rewrite pass already records what it marked Deleted, so
the check is a comparison against the requested set.
Regenerated with make generate-public-api (cargo-public-api 0.51.0); the
snapshot was missing OverwriteAction and its builder methods.
The snapshot summary could only count what the caller asked to remove, so an
operation that skips a path it never had, or is handed the same path twice,
inflated deleted-data-files and deleted-records. SnapshotProduceOperation now
reports the entries it actually marked Deleted, each with the schema and
partition spec of the manifest that recorded it, and the summary counts those.

Two changes follow from it. The manifests are produced before the summary,
since the counts are only known once the operation has run, which means the
added data files are cloned rather than taken. And the full-table truncate that
drops the previous totals is gated on a new operation hook instead of on the
operation being Overwrite: a partial overwrite keeps the totals, and with
truncate off update_totals needs a saturating subtraction so removing more than
the previous total does not underflow.
The manifest name is built inline in new_manifest_writer from the metadata
location, the commit uuid and a counter, so anything else that needs to write a
manifest for the same commit has to reinvent it, and a name invented elsewhere
is neither unique within the commit nor findable by orphan cleanup after a
failed one. Pull the name out into new_manifest_path, and make the counter an
AtomicU64 so it can be bumped behind a shared reference.
new_manifest_writer hardcoded the table's current schema and default partition
spec, so an operation that rewrites an existing manifest could not keep the
schema and spec that manifest's entries were recorded under, and building a
writer by hand instead means losing the encryption manager the method already
branches on: on an encrypted table the manifest would go out in plaintext while
its key metadata said otherwise. The schema and partition spec become arguments,
and the added-files path passes the table's current ones.
…hot summary

Co-authored-by: Marco Kroll <216147982+mkroll-db@users.noreply.github.com>

@laskoviymishka laskoviymishka left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This picks up cleanly from the #2185 round — the things I flagged there have landed: rewritten manifests now stamp the schema and partition-spec they were written with, delete-only manifests survive a later overwrite, the summary reports the files actually marked deleted (and we reject deleting a path the table doesn't have, matching BaseOverwriteFiles), and the rewrite routes through new_manifest_writer so the commit-uuid naming and encryption come along. Test coverage on all of that is solid.

On the two things you flagged: the operation mapping is right and lines up with BaseOverwriteFiles — adds+deletes → Overwrite, deletes-only → Delete, add-only → Append; the only wrinkle is a property-only overwrite falling through to Append, which is the existing #1548 workaround rather than anything new (noted inline). And the add_deleted_entry → add_tombstone_entry rename is fine by me — it's pub(crate), so nothing moves on the published surface; the only thing there is the doc pointing at dead code.

The one thing I'd want fixed before this merges is first_row_id on the rewritten V3 manifests: write_manifest_file() returns None and add_manifests then stamps a fresh id above the row IDs the surviving entries actually carry, so row-ID-range pruning — and Java/PyIceberg readers — can skip real rows. It's latent until MOR lands on top, but I'd rather not bake it in. Details inline.

Two more I'd sort out in the same pass — not correctness, but I wouldn't ship them: the produce-then-summarize reorder means write_added_manifest deep-clones the whole added-file vec on every commit, and that path is shared with FastAppendAction, so plain appends pay for it too; and the new deleted_data_files on SnapshotProducer is dead weight — cloned in only to answer is_empty(), and it reads like it drives the summary when it doesn't. Both trace back to the same reorder, so they're natural to fix together. Smaller stuff (intra-batch dedup, the "fast append" error string) is inline.

Carry the first_row_id through and tidy up the clone/field, and I'm happy to take another pass.

@jopdorp

jopdorp commented Oct 30, 2026

Copy link
Copy Markdown
Contributor Author

@laskoviymishka merged main, so overwrite now sits on the snapshot building blocks from #3474 and the diff no longer touches FastAppend or the producer, apart from making write_added_manifest crate-visible. Along the way we also gave overwrite a retry state like FastAppend has, so a retried commit keeps its snapshot id and never overwrites an earlier attempt's manifest (20693aa). The description drops the #3253 paragraph, and #3586 tracks the unscaled decimal arm. Ready for another look, and for @CTTY on the snapshot side.

@laskoviymishka laskoviymishka left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is close.

Most of round 4 is in, thanks. My request for changes stays on two threads: the manifest reads in the rewrite loop, and matching every path before the first rewrite is written. The clone is a nit. Could you open an issue for a retry test where the base moves between attempts? @CTTY, could you look at overwrite on the #3474 building blocks and the new OverwriteState?

deleted_files.entry(file.file_path()).or_insert(file);
}
if self.check_duplicate {
let added: Vec<DataFile> = self

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could validate_files_not_live take paths instead of &[DataFile]? It only reads file_path(), but with the default check_duplicate this clones every added DataFile on each attempt, even when nothing is deleted. FastAppend passes its slice without a clone, so both call sites could hand over borrowed &strs.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Takes paths now, both callers map to file_path() so nothing is cloned.

continue;
}

let manifest = table.manifest_reader().read(&manifest_file).await?;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this read go through the object cache, like validate_files_not_live does? When the commit also adds files and check_duplicate is on, that pass has already cached these manifests, and this loop fetches each one from storage again, on every retry too.

The concurrent-read TODO from 5a2119b was dropped in cc0c9e491, so HEAD has neither concurrent reads nor the TODO. Could it come back with this?

Suggested change
let manifest = table.manifest_reader().read(&manifest_file).await?;
// TODO: read these manifests concurrently, as `validate_files_not_live` does.
let manifest = table.object_cache().get_manifest(&manifest_file).await?;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reads through the object cache now, same call as validate_files_not_live.

.await
}

async fn cleanup(self, _state: OverwriteState, _table: &Table, _status: CommitStatus) {}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we match every requested path before writing the first rewritten manifest? As written, a missing path alongside matched ones, or a metadata mismatch in a later manifest, returns after earlier rewrites are written, and nothing removes them. Separately, could cleanup carry FastAppend's TODO(#3417)?

Suggested change
async fn cleanup(self, _state: OverwriteState, _table: &Table, _status: CommitStatus) {}
async fn cleanup(self, _state: OverwriteState, _table: &Table, _status: CommitStatus) {
// TODO(#3417): remove this attempt's manifests once terminal cleanup is
// wired up.
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Matching and the metadata check both run before the first rewritten manifest is written, so a missing path or a late mismatch fails with nothing on disk, and the missing-path test asserts that. cleanup carries the TODO.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I held my request for changes on this in round 5. The check now lives in ManifestFilterManager::filter on #3564, so the ordering is a question for that PR and the leftover manifests for #3417, and I'm lifting it here.

@blackmwk blackmwk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @jopdorp for this pr. There are some refactoring of tx framework on going, see #3266. Please hold on this pr after #3266 finished.

Reads go through the object cache so a retry does not refetch manifests, and validate_files_not_live takes paths.
@jopdorp

jopdorp commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

@laskoviymishka all three in with 604c6d1. The rewrite loop reads through the object cache now, every requested path is matched and checked before the first rewritten manifest is written, and validate_files_not_live takes paths so nothing gets cloned. cleanup carries the TODO(#3417) as you suggested. The retry test with a moving base is #3612.

@jopdorp

jopdorp commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

@blackmwk this one already sits on the #3474 producer and the stateful action trait from #3265, so it should be in line with #3266 as it stands. Is there a specific piece of the epic you want landed first? Happy to wait on that, just want to know what to track.

@blackmwk

blackmwk commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

@blackmwk this one already sits on the #3474 producer and the stateful action trait from #3265, so it should be in line with #3266 as it stands. Is there a specific piece of the epic you want landed first? Happy to wait on that, just want to know what to track.

Hi, @jopdorp Stateful transaction trait is only first step. There is another building block to add MergingSnapshotProducer, see #3564. Overwrite tx action should be built on it.

@jopdorp jopdorp changed the title feat(transaction): add overwrite action (continues #2185) feat(transaction): add overwrite action on MergingSnapshotProducer (stacks on #3564, continues #2185) Nov 6, 2026
@jopdorp

jopdorp commented Nov 6, 2026

Copy link
Copy Markdown
Contributor Author

@blackmwk restacked on #3564 with 4aa7e9e. The action only builds a MergingIntent now and the producer does the manifest work, so against that branch the diff is the action, four tests and the wiring.

@laskoviymishka two things differ from what you reviewed, both because they live in the producer now:

  1. Deletes match by path only, like the producer and Java, so the check that record count, size and partition also match is gone.
  2. A missing delete path still fails the commit, but the producer's filter checks it after it wrote its rewritten manifests, not before. They are tracked for the **3. Merging and artifact lifecycle** — introduce MergingSnapshotProducer, manifest filtering, reusable added-manifest production; wire terminal CommitStatus reporting and cleanup #3417 cleanup. I raised it on feat(transaction): Add MergingSnapshotProducer and ManifestFilter #3564.

Re-adding a path deleted in the same overwrite is kept, with its test.

@laskoviymishka laskoviymishka left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The restack onto #3564 is in, thanks. Both deviations are fine with me, answered in their threads; the producer-side points, filter ordering included, are for #3564. Here, could the three regression tests the restack dropped come back?

@blackmwk, does building on MergingSnapshotProducer cover your request for changes?

.find(|f| f.content_type() != DataContentType::Data)
{
return Err(invalid_data!(
"Only data content type is allowed for overwrite, got {:?} for {}",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we bring back the three regression tests the restack dropped? At 604c6d1 they pinned this rejection, a Deletes manifest passing through unchanged, and deleting a path live in two Data manifests. As I read the producer tests on #3564, none of the three is covered there, so nothing at HEAD exercises them.

.await
}

async fn cleanup(self, _state: OverwriteState, _table: &Table, _status: CommitStatus) {}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I held my request for changes on this in round 5. The check now lives in ManifestFilterManager::filter on #3564, so the ordering is a question for that PR and the leftover manifests for #3417, and I'm lifting it here.

@blackmwk

Copy link
Copy Markdown
Contributor

@blackmwk, does building on MergingSnapshotProducer cover your request for changes?

Yes, I think we should get #3564 first?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants