Skip to content

fix(arrow): preserve field identity when projecting renamed columns - #3479

Open
lyne7-sc wants to merge 3 commits into
apache:mainfrom
lyne7-sc:fix/renamed-field-reverse-projection
Open

lyne7-sc wants to merge 3 commits into
apache:mainfrom
lyne7-sc:fix/renamed-field-reverse-projection

Conversation

@lyne7-sc

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

What changes are included in this PR?

Reading older Parquet files after column renames can silently return values under the wrong columns when the projection changes their order and the corresponding types and nullability match. For example, projecting [y#2, x#1] from [a#1=10, b#2=20] can return [10, 20] instead of [20, 10].

Require field IDs to match at each position before passing a batch through or updating only its schema. Otherwise, use the existing field-ID-based transformation. This also prevents treating a same-name field with a new ID as the original field.

Are these changes tested?

  • Transformer tests cover reordered renamed columns, cached plan reuse, and same-name fields with different IDs.
  • test_renamed_reverse_projection_reads_values_by_field_id verifies the projected values and schema when reading a local Parquet file.

AI Disclosure

Developed with assistance from OpenAI Codex.

@JeonDaehong JeonDaehong left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked this against main with a three-column Parquet file (a, b, s), once with field ids and once without. I read it through a schema with the same names and through one where all three are renamed, projecting [1, 2, 3], [2, 1, 3], [3, 1] and [2].

On main, [2, 1, 3] with renamed fields returns rb=1,2 ra=10,20, values swapped, and that happens for the file without field ids too, not only the one with ids. On this branch all 16 cases return values by field id, and the same-name cases are unchanged.

The id-less case goes through the fallback-id path, so a test with a file written without PARQUET:field_id metadata might be worth adding next to test_renamed_reverse_projection_reads_values_by_field_id, since that's where the old name-based comparison was most likely to be relied on.

@lyne7-sc

Copy link
Copy Markdown
Contributor Author

@JeonDaehong Thanks for the suggestion! I've added a test for files without field IDs.

@fm100 fm100 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We reproduced this with iceberg-rust 0.10.1 while reading equality-delete files through ArrowReader. No column rename was involved.

For two columns with matching types and nullability:

  • Table schema: [a#1, b#2].
  • Delete-file schema: [b#2, a#1], containing [2, 1].
  • Requested projection: [1, 2].
  • Expected output: [a=1, b=2].
  • Actual output: [a=2, b=1].

The field names change to match the projection, but the values remain in file order. When interpreted as equality-delete keys, this can leave the intended deleted row visible and incorrectly hide another live row. We encountered it during a MERGE target scan.

The field-ID check in #3479 looks relevant here too. Could you confirm whether it also covers reordered equality-delete columns without a rename too?

Very much appreciated!

@JeonDaehong

Copy link
Copy Markdown
Contributor

@fm100 I ran your case through RecordBatchTransformer, which is what the equality delete loader uses (delete_file_loader.rs). The file columns are [b#2=2, a#1=1] and the projection is [1, 2]:

table schema v0.10.1 main (d2679a0) this PR (8d8e1b6)
[a#1, b#2], no rename a=2, b=1 (wrong) a=1, b=2 a=1, b=2
[x#1, y#2], renamed x=2, y=1 (wrong) x=2, y=1 (wrong) x=1, y=2

Your no-rename case is already fixed on main. #2746 (8b76dbe) added a reorder check by name there, but that change isn't in any 0.10.x release. If the columns were renamed, it's still wrong on main, and this PR fixes that by matching field IDs.

@lyne7-sc

Copy link
Copy Markdown
Contributor Author

@fm100 Yes, as @JeonDaehong confirmed, reordering equality-delete columns without renaming is already handled correctly on main. This PR additionally fixes the renamed case.

c-thiel pushed a commit to lakekeeper/iceberg-rust that referenced this pull request Oct 26, 2026
[Upstream PR apache#3479, open on 2026-10-25.]
(cherry picked from commit db7178b)
c-thiel pushed a commit to lakekeeper/iceberg-rust that referenced this pull request Oct 26, 2026
[Upstream PR apache#3479, open on 2026-10-25.]
(cherry picked from commit 8d8e1b6)
JanKaul pushed a commit to lakekeeper/iceberg-rust that referenced this pull request Oct 26, 2026
[Upstream PR apache#3479, open on 2026-10-25.]
(cherry picked from commit db7178b)
JanKaul pushed a commit to lakekeeper/iceberg-rust that referenced this pull request Oct 26, 2026
[Upstream PR apache#3479, open on 2026-10-25.]
(cherry picked from commit 8d8e1b6)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Renamed columns can return swapped values when projection order differs from file order

3 participants