Repository navigation
Conversation
Adds an OverwriteAction to Transaction for atomic replacement of data files via an Operation::Overwrite snapshot. add_data_files() stages new files; delete_data_files() rewrites affected manifests with matching entries marked ManifestStatus::Deleted (copy-on-write). Manifests are loaded through the Table::manifest_list_reader abstraction.
The SnapshotProducer precondition check blocked snapshots with only deleted files and no added files or snapshot properties. A delete-only Overwrite snapshot is valid per the Iceberg spec — the existing manifests are rewritten with the target entries marked as ManifestStatus::Deleted. Relax the check to also allow the case when deleted_data_files is non-empty, so callers can clear a table without simultaneously adding new files. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Import Manifest and DataFile at the top instead of using fully-qualified crate::spec:: paths inline in a signature and a test helper.
# Conflicts: # crates/iceberg/public-api.txt # crates/iceberg/src/transaction/append.rs # crates/iceberg/src/transaction/mod.rs
A manifest rewritten by a second overwrite fell through to add_existing_entry for entries that were already Deleted, forcing their status back to Existing and resurrecting the files as live data. Guard on is_alive() first and re-add prior tombstones as Deleted with their original snapshot id, matching Java's filterManifestWithDeletedFiles, which only re-adds live entries.
The rewrite used the table's current schema and default partition spec, so after a schema or partition evolution the Avro header carried the wrong schema-id and partition-spec-id and readers interpreted the partition values against the wrong mapping. Take both from the manifest being rewritten, as Java does via newManifestWriter(reader.spec()).
Both manifest filters dropped a manifest whose entries were all Deleted, which lets the removed files reappear as live data. This is the same fix FastAppend took in apache#2545 for apache#2148, applied to the no-deletes fast path and to the rewrite loop.
The summary counted every file the caller passed to delete_data_files, so a path that was never committed or was passed twice inflated deleted-data-files and deleted-records. The rewrite pass now records the entries it marked Deleted, with the schema and partition spec of the manifest each was recorded in, and the summary counts those. Two changes follow from it. The summary is built after the manifests rather than before, since the counts are only known once the rewrite has run. And the full-table truncate path is now gated on an operation hook that a partial overwrite answers false, instead of on the operation being Overwrite; with truncate off, update_totals needs a saturating subtraction so an overwrite that deletes more than the previous total does not underflow.
The rewrite hardcoded {location}/metadata and a fresh Uuid::now_v7, so a table
with a custom write.metadata.path got its rewritten manifests in the wrong
prefix, and orphan cleanup could not find them after a failed commit because
the name was not tied to the commit. Reuse the path SnapshotProducer already
builds from metadata_location(), the commit uuid and the manifest counter; the
counter becomes an AtomicU64 because the rewrite runs behind a shared
reference.
rewrite_manifest built its own plain output file, so on an encrypted table the rewritten manifest went out in plaintext while its key metadata said otherwise. It now goes through SnapshotProducer::new_manifest_writer, which already branches on the table's encryption manager; the writer takes the schema and partition spec as arguments so the rewrite can keep the ones the manifest was written under. That leaves new_manifest_path with no caller outside this module, so it becomes private again.
add_deleted_entry set the status but left snapshot_id to the caller, while add_delete_entry right above it stamps both; the two differed only by tense and a tombstone written with the wrong snapshot_id misattributes the delete, which expire_snapshots then acts on. Taking the reviewer's second option: rename rather than stamp, because carrying an earlier tombstone forward has to keep its original snapshot_id, and document that the caller owns the field. Named add_tombstone_entry rather than something closer to main's add_delete_entry (apache#2367) precisely so the two cannot be confused.
A delete-only overwrite reported Operation::Overwrite, so tools keying off the summary read it as a replacement. Match Java's BaseOverwriteFiles: DELETE when there are only deletes, APPEND when there are only adds, OVERWRITE when there are both. test_overwrite_basic adds without deleting, so its expected operation becomes APPEND. Also names deleted data files in the precondition error, which has allowed a delete-only commit since the guard was relaxed.
is_err() alone passes on any error, so the test would stay green if the commit started failing earlier for an unrelated reason. Assert the kind and the message instead.
An overwrite silently ignored a delete path that was not live in any manifest, so a typo or an already-deleted file committed as a no-op delete. Java's BaseOverwriteFiles validates the same thing and fails with "Missing required files to delete"; the rewrite pass already records what it marked Deleted, so the check is a comparison against the requested set.
Regenerated with make generate-public-api (cargo-public-api 0.51.0); the snapshot was missing OverwriteAction and its builder methods.
The snapshot summary could only count what the caller asked to remove, so an operation that skips a path it never had, or is handed the same path twice, inflated deleted-data-files and deleted-records. SnapshotProduceOperation now reports the entries it actually marked Deleted, each with the schema and partition spec of the manifest that recorded it, and the summary counts those. Two changes follow from it. The manifests are produced before the summary, since the counts are only known once the operation has run, which means the added data files are cloned rather than taken. And the full-table truncate that drops the previous totals is gated on a new operation hook instead of on the operation being Overwrite: a partial overwrite keeps the totals, and with truncate off update_totals needs a saturating subtraction so removing more than the previous total does not underflow.
The manifest name is built inline in new_manifest_writer from the metadata location, the commit uuid and a counter, so anything else that needs to write a manifest for the same commit has to reinvent it, and a name invented elsewhere is neither unique within the commit nor findable by orphan cleanup after a failed one. Pull the name out into new_manifest_path, and make the counter an AtomicU64 so it can be bumped behind a shared reference.
new_manifest_writer hardcoded the table's current schema and default partition spec, so an operation that rewrites an existing manifest could not keep the schema and spec that manifest's entries were recorded under, and building a writer by hand instead means losing the encryption manager the method already branches on: on an encrypted table the manifest would go out in plaintext while its key metadata said otherwise. The schema and partition spec become arguments, and the added-files path passes the table's current ones.
…hot summary Co-authored-by: Marco Kroll <216147982+mkroll-db@users.noreply.github.com>
laskoviymishka
left a comment
There was a problem hiding this comment.
This picks up cleanly from the #2185 round — the things I flagged there have landed: rewritten manifests now stamp the schema and partition-spec they were written with, delete-only manifests survive a later overwrite, the summary reports the files actually marked deleted (and we reject deleting a path the table doesn't have, matching BaseOverwriteFiles), and the rewrite routes through new_manifest_writer so the commit-uuid naming and encryption come along. Test coverage on all of that is solid.
On the two things you flagged: the operation mapping is right and lines up with BaseOverwriteFiles — adds+deletes → Overwrite, deletes-only → Delete, add-only → Append; the only wrinkle is a property-only overwrite falling through to Append, which is the existing #1548 workaround rather than anything new (noted inline). And the add_deleted_entry → add_tombstone_entry rename is fine by me — it's pub(crate), so nothing moves on the published surface; the only thing there is the doc pointing at dead code.
The one thing I'd want fixed before this merges is first_row_id on the rewritten V3 manifests: write_manifest_file() returns None and add_manifests then stamps a fresh id above the row IDs the surviving entries actually carry, so row-ID-range pruning — and Java/PyIceberg readers — can skip real rows. It's latent until MOR lands on top, but I'd rather not bake it in. Details inline.
Two more I'd sort out in the same pass — not correctness, but I wouldn't ship them: the produce-then-summarize reorder means write_added_manifest deep-clones the whole added-file vec on every commit, and that path is shared with FastAppendAction, so plain appends pay for it too; and the new deleted_data_files on SnapshotProducer is dead weight — cloned in only to answer is_empty(), and it reads like it drives the summary when it doesn't. Both trace back to the same reorder, so they're natural to fix together. Smaller stuff (intra-batch dedup, the "fast append" error string) is inline.
Carry the first_row_id through and tidy up the clone/field, and I'm happy to take another pass.
Java and PyIceberg derive totals from the files an operation marks deleted, with no truncation path.
# Conflicts: # crates/iceberg/src/transaction/append.rs # crates/iceberg/src/transaction/snapshot.rs
|
@laskoviymishka merged main, so overwrite now sits on the snapshot building blocks from #3474 and the diff no longer touches FastAppend or the producer, apart from making |
laskoviymishka
left a comment
There was a problem hiding this comment.
I think this is close.
Most of round 4 is in, thanks. My request for changes stays on two threads: the manifest reads in the rewrite loop, and matching every path before the first rewrite is written. The clone is a nit. Could you open an issue for a retry test where the base moves between attempts? @CTTY, could you look at overwrite on the #3474 building blocks and the new OverwriteState?
| deleted_files.entry(file.file_path()).or_insert(file); | ||
| } | ||
| if self.check_duplicate { | ||
| let added: Vec<DataFile> = self |
There was a problem hiding this comment.
Could validate_files_not_live take paths instead of &[DataFile]? It only reads file_path(), but with the default check_duplicate this clones every added DataFile on each attempt, even when nothing is deleted. FastAppend passes its slice without a clone, so both call sites could hand over borrowed &strs.
There was a problem hiding this comment.
Takes paths now, both callers map to file_path() so nothing is cloned.
| continue; | ||
| } | ||
|
|
||
| let manifest = table.manifest_reader().read(&manifest_file).await?; |
There was a problem hiding this comment.
Could this read go through the object cache, like validate_files_not_live does? When the commit also adds files and check_duplicate is on, that pass has already cached these manifests, and this loop fetches each one from storage again, on every retry too.
The concurrent-read TODO from 5a2119b was dropped in cc0c9e491, so HEAD has neither concurrent reads nor the TODO. Could it come back with this?
| let manifest = table.manifest_reader().read(&manifest_file).await?; | |
| // TODO: read these manifests concurrently, as `validate_files_not_live` does. | |
| let manifest = table.object_cache().get_manifest(&manifest_file).await?; |
There was a problem hiding this comment.
Reads through the object cache now, same call as validate_files_not_live.
| .await | ||
| } | ||
|
|
||
| async fn cleanup(self, _state: OverwriteState, _table: &Table, _status: CommitStatus) {} |
There was a problem hiding this comment.
Could we match every requested path before writing the first rewritten manifest? As written, a missing path alongside matched ones, or a metadata mismatch in a later manifest, returns after earlier rewrites are written, and nothing removes them. Separately, could cleanup carry FastAppend's TODO(#3417)?
| async fn cleanup(self, _state: OverwriteState, _table: &Table, _status: CommitStatus) {} | |
| async fn cleanup(self, _state: OverwriteState, _table: &Table, _status: CommitStatus) { | |
| // TODO(#3417): remove this attempt's manifests once terminal cleanup is | |
| // wired up. | |
| } |
There was a problem hiding this comment.
Matching and the metadata check both run before the first rewritten manifest is written, so a missing path or a late mismatch fails with nothing on disk, and the missing-path test asserts that. cleanup carries the TODO.
There was a problem hiding this comment.
I held my request for changes on this in round 5. The check now lives in ManifestFilterManager::filter on #3564, so the ordering is a question for that PR and the leftover manifests for #3417, and I'm lifting it here.
Reads go through the object cache so a retry does not refetch manifests, and validate_files_not_live takes paths.
|
@laskoviymishka all three in with 604c6d1. The rewrite loop reads through the object cache now, every requested path is matched and checked before the first rewritten manifest is written, and validate_files_not_live takes paths so nothing gets cloned. cleanup carries the TODO(#3417) as you suggested. The retry test with a moving base is #3612. |
…sion comb better, specify comments
Hi, @jopdorp Stateful transaction trait is only first step. There is another building block to add MergingSnapshotProducer, see #3564. Overwrite tx action should be built on it. |
The producer's ManifestFilter now does the manifest rewrite, path matching and dedupe, so the action only states its intent.
|
@blackmwk restacked on #3564 with 4aa7e9e. The action only builds a @laskoviymishka two things differ from what you reviewed, both because they live in the producer now:
Re-adding a path deleted in the same overwrite is kept, with its test. |
laskoviymishka
left a comment
There was a problem hiding this comment.
The restack onto #3564 is in, thanks. Both deviations are fine with me, answered in their threads; the producer-side points, filter ordering included, are for #3564. Here, could the three regression tests the restack dropped come back?
@blackmwk, does building on MergingSnapshotProducer cover your request for changes?
| .find(|f| f.content_type() != DataContentType::Data) | ||
| { | ||
| return Err(invalid_data!( | ||
| "Only data content type is allowed for overwrite, got {:?} for {}", |
There was a problem hiding this comment.
Could we bring back the three regression tests the restack dropped? At 604c6d1 they pinned this rejection, a Deletes manifest passing through unchanged, and deleting a path live in two Data manifests. As I read the producer tests on #3564, none of the three is covered there, so nothing at HEAD exercises them.
| .await | ||
| } | ||
|
|
||
| async fn cleanup(self, _state: OverwriteState, _table: &Table, _status: CommitStatus) {} |
There was a problem hiding this comment.
I held my request for changes on this in round 5. The check now lives in ManifestFilterManager::filter on #3564, so the ordering is a question for that PR and the leftover manifests for #3417, and I'm lifting it here.
Which issue does this PR close?
Part of #2186.
What changes are included in this PR?
Stacks on #3564. Continues #2185 by @glitchy.
OverwriteActionadds data files and removes listed data files by path in one snapshot. It builds aMergingIntentand theMergingSnapshotProducerfrom #3564 does the manifest work, so against that branch this PR is the action, its tests and the wiring.check_duplicate, an added path that is already live is rejected, unless the same overwrite deletes it.Are these changes tested?
Four tests in
transaction::overwrite::tests: a partial overwrite followed by a delete with the summary counts, a missing delete path is rejected, a path deleted in the same overwrite can be re-added, and an added path that is live and not deleted is rejected. Manifest rewriting, schema and spec evolution, retries and row lineage are covered by the producer's tests in #3564.cargo test -p iceberg --lib, the public API check, fmt and clippy with-D warningsare clean.AI Disclosure
The restack, the review fixes and their tests were drafted with an AI coding assistant, then reviewed, run and checked against the Java implementation by us. Worth a close look: which operation a commit reports, and the duplicate check exemption for a path the same overwrite deletes.