Repository navigation
Conversation
JeonDaehong
left a comment
There was a problem hiding this comment.
Checked this against main with a three-column Parquet file (a, b, s), once with field ids and once without. I read it through a schema with the same names and through one where all three are renamed, projecting [1, 2, 3], [2, 1, 3], [3, 1] and [2].
On main, [2, 1, 3] with renamed fields returns rb=1,2 ra=10,20, values swapped, and that happens for the file without field ids too, not only the one with ids. On this branch all 16 cases return values by field id, and the same-name cases are unchanged.
The id-less case goes through the fallback-id path, so a test with a file written without PARQUET:field_id metadata might be worth adding next to test_renamed_reverse_projection_reads_values_by_field_id, since that's where the old name-based comparison was most likely to be relied on.
|
@JeonDaehong Thanks for the suggestion! I've added a test for files without field IDs. |
fm100
left a comment
There was a problem hiding this comment.
We reproduced this with iceberg-rust 0.10.1 while reading equality-delete files through ArrowReader. No column rename was involved.
For two columns with matching types and nullability:
- Table schema:
[a#1, b#2]. - Delete-file schema:
[b#2, a#1], containing[2, 1]. - Requested projection:
[1, 2]. - Expected output:
[a=1, b=2]. - Actual output:
[a=2, b=1].
The field names change to match the projection, but the values remain in file order. When interpreted as equality-delete keys, this can leave the intended deleted row visible and incorrectly hide another live row. We encountered it during a MERGE target scan.
The field-ID check in #3479 looks relevant here too. Could you confirm whether it also covers reordered equality-delete columns without a rename too?
Very much appreciated!
|
@fm100 I ran your case through
Your no-rename case is already fixed on main. #2746 (8b76dbe) added a reorder check by name there, but that change isn't in any 0.10.x release. If the columns were renamed, it's still wrong on main, and this PR fixes that by matching field IDs. |
|
@fm100 Yes, as @JeonDaehong confirmed, reordering equality-delete columns without renaming is already handled correctly on main. This PR additionally fixes the renamed case. |
[Upstream PR apache#3479, open on 2026-10-25.] (cherry picked from commit db7178b)
[Upstream PR apache#3479, open on 2026-10-25.] (cherry picked from commit 8d8e1b6)
[Upstream PR apache#3479, open on 2026-10-25.] (cherry picked from commit db7178b)
[Upstream PR apache#3479, open on 2026-10-25.] (cherry picked from commit 8d8e1b6)
Which issue does this PR close?
What changes are included in this PR?
Reading older Parquet files after column renames can silently return values under the wrong columns when the projection changes their order and the corresponding types and nullability match. For example, projecting
[y#2, x#1]from[a#1=10, b#2=20]can return[10, 20]instead of[20, 10].Require field IDs to match at each position before passing a batch through or updating only its schema. Otherwise, use the existing field-ID-based transformation. This also prevents treating a same-name field with a new ID as the original field.
Are these changes tested?
test_renamed_reverse_projection_reads_values_by_field_idverifies the projected values and schema when reading a local Parquet file.AI Disclosure
Developed with assistance from OpenAI Codex.