Skip to content

fix(scan): validate path-based position delete partitions - #3698

Open
iceberg-kitten wants to merge 2 commits into
apache:mainfrom
iceberg-kitten:fix/validate-path-position-delete-partitions-20261115
Open

iceberg-kitten wants to merge 2 commits into
apache:mainfrom
iceberg-kitten:fix/validate-path-position-delete-partitions-20261115

Conversation

@iceberg-kitten

Copy link
Copy Markdown

Which issue does this PR close?

  • No issue linked.

What

  • Validate partition metadata for path-indexed position deletes.

Why

  • Iceberg requires both the partition spec and partition values to match for a position delete (spec).
  • Java checks path-matched deletes against both (DeleteFileIndex).

Implementation

  • Return DataInvalid when a path-matched delete has a different spec ID or partition tuple.
  • Preserve path-based matching when the metadata agrees.

AI Disclosure

  • Review focus: the spec does not prescribe an error type; this uses DataInvalid for inconsistent delete metadata.

@JeonDaehong JeonDaehong left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked this against Java's DeleteFileIndex: findPathDeletes runs validatePartitionMatch on every path-matched delete, and so does findDV, which the DV branch here already mirrors. The delete index tests pass, and dropping the new check fails both rejection tests. LGTM.

})
{
let delete_data_file = delete.manifest_entry.data_file();
if data_file.partition_spec_id != delete.partition_spec_id

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same check and the same order as Java (sequence filter first, then validatePartitionMatch), so a stale delete with a mismatched partition is still skipped rather than rejected.

{
let delete_data_file = delete.manifest_entry.data_file();
if data_file.partition_spec_id != delete.partition_spec_id
|| data_file.partition() != delete_data_file.partition()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Struct equality is type-sensitive where Java uses the spec's comparator, but fetch_manifest_and_stream_manifest_entries widens promoted int/float partition tuples (#3483) before entries reach the index, so a source promoted from int to long still compares like-typed here.

}

#[test]
fn test_pos_delete_with_equal_path_bounds_rejects_mismatched_partition() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing the check makes this test and test_pos_delete_with_referenced_data_file_rejects_mismatched_partition_spec fail, so both the bounds-derived and the referenced_data_file paths are covered.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants