Repository navigation
Conversation
|
|
||
| /// Produces the resulting manifest set for the current table and writes | ||
| /// this attempt's snapshot. | ||
| pub(crate) async fn apply(&mut self, table: &Table) -> Result<ActionCommit> { |
There was a problem hiding this comment.
Seems like we currently have no logic to validate new deletes or overwrite?
There was a problem hiding this comment.
We can add them later before adding the actions that consume MSP
laskoviymishka
left a comment
There was a problem hiding this comment.
It's on the large side, but this is close; I left a couple of comments inline.
Keying each rewrite by ManifestKey rather than path means a retry only redoes the manifests whose inherited values actually moved, and that's the piece the rest of #3417 builds on.
What I'd most like to settle is who decides that a missing delete fails, the action or the filter. I'd like to hear @anoopj's opinion on that thread; a follow-up is fine.
Could the description also mention the to_human_string promotion fix in spec/?
I'll take another look once that thread converges.
| /// Produces the resulting manifest set for the current table and writes | ||
| /// this attempt's snapshot. | ||
| pub(crate) async fn apply(&mut self, table: &Table) -> Result<ActionCommit> { |
There was a problem hiding this comment.
I think this is aligned with the current Java implementation and we are trying to change it for v4. I suggest establishing a separate CommitValidator API. Action execution state can own the validator alongside this producer across retries. This has two advantages:
- The validation can be stateful. So retries can reuse the already validated state.
- Provides a useful seam for introducing v4 specific implementations of validations.
apache/iceberg#18699 may be a useful reference. This is one of the many upcoming changes we are making for validations.
There was a problem hiding this comment.
Thanks for pointing me to the java side PR! I had some thoughts about this a few weeks ago, and my previous thought was caching validated snapshots directly under the MSP and the seam also makes sense to me.
iiuc, we could still address this #3419 ? I'm leaning toward moving validations to another PR since the current PR is already quite large
Merges CTTY's branch and drops the producer this branch carried from apache#3046.
|
@CTTY we put #3585 (
Happy to open either as a PR on top of this once it lands. Both were in the producer #3585 carried before the restack, in case it saves time: the keep rule at https://github.com/jopdorp/iceberg-rust/blob/682a6150b/crates/iceberg/src/transaction/merging.rs#L301-L313 and the sequence number check at https://github.com/jopdorp/iceberg-rust/blob/682a6150b/crates/iceberg/src/transaction/merging.rs#L259-L270. |
|
Hi @jopdorp , thanks for the feedback! I've addressed the first point you mentioned. And I'm planning to address the starting-seq-number issue when we are actually wiring actions with MSP and adding validation methods |
|
@CTTY #3300 (overwrite) sits on this branch now as well, it only builds a
Happy to send a PR into your branch for 1 if you want it. |
JeonDaehong
left a comment
There was a problem hiding this comment.
Re-ran my check at 8a442c6: removing a.parquet now marks its DV deleted (total-delete-files=0). Naming the DV as well under fail_missing_delete_paths removes it once, and removing only b.parquet or appending leaves a's DV alone. The older delete file cleanup (dropDeleteFilesOlderThan) isn't there yet, which is fine as a follow-up.
|
@CTTY one thing from my last comment is still open: the |
Which issue does this PR close?
MergingSnapshotProducer, manifest filtering, reusable added-manifest production; wire terminalCommitStatusreporting and cleanup #3417What changes are included in this PR?
MergingSnapshotProduceras a Transaction state to store deleted filesManifestFilterManagerto cache the manifests across transaction attemptsTransform::to_human_stringto widen a value to the declared field typebefore rendering.
Are these changes tested?
Added unit tests
AI Disclosure
Used Claude, but reviewed them with human eyes