Skip to content

feat(transaction): Add MergingSnapshotProducer and ManifestFilter - #3564

Open
CTTY wants to merge 8 commits into
apache:mainfrom
CTTY:ctty/merging-sp
Open

CTTY wants to merge 8 commits into
apache:mainfrom
CTTY:ctty/merging-sp

Conversation

@CTTY

@CTTY CTTY commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

Which issue does this PR close?

What changes are included in this PR?

  • Add MergingSnapshotProducer as a Transaction state to store deleted files
  • Add ManifestFilterManager to cache the manifests across transaction attempts
  • Fix Transform::to_human_string to widen a value to the declared field type
    before rendering.

Are these changes tested?

Added unit tests

AI Disclosure

Used Claude, but reviewed them with human eyes


/// Produces the resulting manifest set for the current table and writes
/// this attempt's snapshot.
pub(crate) async fn apply(&mut self, table: &Table) -> Result<ActionCommit> {

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like we currently have no logic to validate new deletes or overwrite?

@CTTY CTTY Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can add them later before adding the actions that consume MSP

@CTTY
CTTY marked this pull request as ready for review October 28, 2026 20:45

@laskoviymishka laskoviymishka left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's on the large side, but this is close; I left a couple of comments inline.

Keying each rewrite by ManifestKey rather than path means a retry only redoes the manifests whose inherited values actually moved, and that's the piece the rest of #3417 builds on.

What I'd most like to settle is who decides that a missing delete fails, the action or the filter. I'd like to hear @anoopj's opinion on that thread; a follow-up is fine.

Could the description also mention the to_human_string promotion fix in spec/?

I'll take another look once that thread converges.

Comment thread crates/iceberg/src/spec/transform.rs Resolved
Comment on lines +106 to +108
/// Produces the resulting manifest set for the current table and writes
/// this attempt's snapshot.
pub(crate) async fn apply(&mut self, table: &Table) -> Result<ActionCommit> {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is aligned with the current Java implementation and we are trying to change it for v4. I suggest establishing a separate CommitValidator API. Action execution state can own the validator alongside this producer across retries. This has two advantages:

  1. The validation can be stateful. So retries can reuse the already validated state.
  2. Provides a useful seam for introducing v4 specific implementations of validations.

apache/iceberg#18699 may be a useful reference. This is one of the many upcoming changes we are making for validations.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pointing me to the java side PR! I had some thoughts about this a few weeks ago, and my previous thought was caching validated snapshots directly under the MSP and the seam also makes sense to me.

iiuc, we could still address this #3419 ? I'm leaning toward moving validations to another PR since the current PR is already quite large

@blackmwk blackmwk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @CTTY , just finished first round of review, left some comments.

jopdorp added a commit to jopdorp/iceberg-rust that referenced this pull request Nov 3, 2026


Merges CTTY's branch and drops the producer this branch carried from apache#3046.
@jopdorp

jopdorp commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

@CTTY we put #3585 (RewriteFilesAction, continuing #3046) on top of this branch and the producer carries it as is, remove-only rewrites included. Two things the rewrite tests ran into that I think belong in the producer rather than the action:

  1. A manifest left with only deleted entries is carried forward on every later commit. current_manifests keeps it since fix(transaction): preserve delete-only manifests in FastAppend #2545, and the filter rewrites the source manifest so the FastAppendAction drops delete-only manifests, causing deleted files to reappear #2148 case can't come back, but nothing drops it afterwards. Java keeps a manifest in MergingSnapshotProducer.apply only when it has added or existing files or was written by this commit (MergingSnapshotProducer.java:1057-1062). Could apply do the same after deletes.manifests? We hit this in pyiceberg, where the same carry-forward made every commit slower over time.

  2. MergingIntent has no data sequence number for added files, so the entries always inherit the snapshot's. A compaction needs to keep the number that existing equality deletes apply to. ManifestWriter::add_file(file, seq) is already there for it, plus Java's check that it is not above the new snapshot's. Fine as a follow-up, just flagging that RewriteFiles needs it.

Happy to open either as a PR on top of this once it lands.

Both were in the producer #3585 carried before the restack, in case it saves time: the keep rule at https://github.com/jopdorp/iceberg-rust/blob/682a6150b/crates/iceberg/src/transaction/merging.rs#L301-L313 and the sequence number check at https://github.com/jopdorp/iceberg-rust/blob/682a6150b/crates/iceberg/src/transaction/merging.rs#L259-L270.

@CTTY

CTTY commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator Author

Hi @jopdorp , thanks for the feedback! I've addressed the first point you mentioned. And I'm planning to address the starting-seq-number issue when we are actually wiring actions with MSP and adding validation methods

@jopdorp

jopdorp commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

@CTTY #3300 (overwrite) sits on this branch now as well, it only builds a MergingIntent with fail_missing_delete_paths on. Three things from putting it on top, none blocking for us:

  1. Missing delete paths are checked after the loop in ManifestFilterManager::filter has written its rewritten manifests, and validate_added_files also runs after the filters wrote. On feat(transaction): add overwrite action on MergingSnapshotProducer (stacks on #3564, continues #2185) #3300 @laskoviymishka asked for everything to be matched before the first write. They land in owned_artifacts, so **3. Merging and artifact lifecycle** — introduce MergingSnapshotProducer, manifest filtering, reusable added-manifest production; wire terminal CommitStatus reporting and cleanup #3417 covers them and maybe that is enough.
  2. The MergingIntent doc says the action validates disjoint add/remove sets. Overwrite allows deleting a path and re-adding it in the same commit, which was asked for in feat(transaction): add overwrite action on MergingSnapshotProducer (stacks on #3564, continues #2185) #3300's review, and the producer handles it fine: a deleted entry plus an added one. Should the doc allow that, or do you want actions to reject it?
  3. On v3, should a data file's deletion vectors go when the data file is removed? Overwrite leaves removed_delete_files empty, so they stay live. I have not checked what Java does here.

Happy to send a PR into your branch for 1 if you want it.

@JeonDaehong JeonDaehong left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-ran my check at 8a442c6: removing a.parquet now marks its DV deleted (total-delete-files=0). Naming the DV as well under fail_missing_delete_paths removes it once, and removing only b.parquet or appending leaves a's DV alone. The older delete file cleanup (dropDeleteFilesOlderThan) isn't there yet, which is fine as a follow-up.

@jopdorp

jopdorp commented Nov 10, 2026

Copy link
Copy Markdown
Contributor

@CTTY one thing from my last comment is still open: the MergingIntent doc says add and remove sets are disjoint, but overwrite deletes a path and re-adds it in the same commit (that was asked for in #3300's review), and the producer handles it fine. Should the doc allow that, or should actions reject it?

@jopdorp

jopdorp commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

@blackmwk could you take another look when you get a chance? JeonDaehong approved and the dangling deletion vectors are fixed, and #3300 and #3585 are waiting on this one.

@CTTY CTTY mentioned this pull request Nov 17, 2026
3 tasks

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants