Repository navigation
fix(scan): validate path-based position delete partitions - #3698
iceberg-kitten wants to merge 2 commits into
Conversation
JeonDaehong
left a comment
There was a problem hiding this comment.
Checked this against Java's DeleteFileIndex: findPathDeletes runs validatePartitionMatch on every path-matched delete, and so does findDV, which the DV branch here already mirrors. The delete index tests pass, and dropping the new check fails both rejection tests. LGTM.
| }) | ||
| { | ||
| let delete_data_file = delete.manifest_entry.data_file(); | ||
| if data_file.partition_spec_id != delete.partition_spec_id |
There was a problem hiding this comment.
Same check and the same order as Java (sequence filter first, then validatePartitionMatch), so a stale delete with a mismatched partition is still skipped rather than rejected.
| { | ||
| let delete_data_file = delete.manifest_entry.data_file(); | ||
| if data_file.partition_spec_id != delete.partition_spec_id | ||
| || data_file.partition() != delete_data_file.partition() |
There was a problem hiding this comment.
Struct equality is type-sensitive where Java uses the spec's comparator, but fetch_manifest_and_stream_manifest_entries widens promoted int/float partition tuples (#3483) before entries reach the index, so a source promoted from int to long still compares like-typed here.
| } | ||
|
|
||
| #[test] | ||
| fn test_pos_delete_with_equal_path_bounds_rejects_mismatched_partition() { |
There was a problem hiding this comment.
Removing the check makes this test and test_pos_delete_with_referenced_data_file_rejects_mismatched_partition_spec fail, so both the bounds-derived and the referenced_data_file paths are covered.
Which issue does this PR close?
What
Why
Implementation
DataInvalidwhen a path-matched delete has a different spec ID or partition tuple.AI Disclosure
DataInvalidfor inconsistent delete metadata.