Repository navigation
Open
Labels
Description
Apache Iceberg version
main (development)
Please describe the bug 🐞
Describe the bug
When a table evolves its partition spec by renaming a partition field:
TableMetadata.specs_struct()adopts the latest field name for that partitionfield_id(e.g.1000: p->1000: p_renamed).- However,
table.inspect.partitions(),files(),data_files(),all_files(),all_data_files(), andentries()build partition dictionaries usingfield.namefrom each file's historicalPartitionSpecdirectly ({"p": "a"}). - When PyArrow converts these rows to the schema defined by
specs_struct()(which has field"p_renamed"), the"p"key is ignored and"p_renamed"is filled withNone. All pre-rename partition values are silently replaced withNone({'p_renamed': None}).
Similarly, on format-version 1 tables, removing a partition field (remove_field("p")) and subsequently re-adding it (add_identity("p")) automatically renames the voided field (1000: p_1000) and creates 1001: p. Because inspect keys by historical field.name == "p", historical spec 0 data files written under field_id=1000 are silently misattributed to field_id=1001 ({'p_1000': None, 'p': 'a'}).
To Reproduce
import tempfile
import pyarrow as pa
from pyiceberg.catalog.memory import InMemoryCatalog
from pyiceberg.partitioning import PartitionField, PartitionSpec
from pyiceberg.schema import Schema
from pyiceberg.transforms import IdentityTransform
from pyiceberg.types import LongType, NestedField, StringType
with tempfile.TemporaryDirectory() as tmp:
cat = InMemoryCatalog("local", warehouse=f"file://{tmp}/wh")
cat.create_namespace("ns")
tbl = cat.create_table(
"ns.t_rename",
schema=Schema(NestedField(1, "id", LongType()), NestedField(2, "p", StringType())),
partition_spec=PartitionSpec(PartitionField(2, 1000, IdentityTransform(), "p")),
properties={"format-version": "2"},
)
tbl.append(pa.table({"id": [1, 2], "p": ["a", "b"]}))
tbl.update_spec().rename_field("p", "p_renamed").commit()
tbl.append(pa.table({"id": [3], "p": ["c"]}))
# Historical values 'a' and 'b' silently become None:
print(tbl.inspect.partitions().to_pydict()["partition"])
# Current output: [{'p_renamed': 'c'}, {'p_renamed': None}, {'p_renamed': None}]
# Expected output: [{'p_renamed': 'c'}, {'p_renamed': 'a'}, {'p_renamed': 'b'}]
print([r["partition"] for r in tbl.inspect.files().to_pylist()])
# Current output: [{'p_renamed': 'c'}, {'p_renamed': None}, {'p_renamed': None}]
# Expected output: [{'p_renamed': 'c'}, {'p_renamed': 'a'}, {'p_renamed': 'b'}]Expected behavior
table.inspect tables should map partition field values using the canonical field names from specs_struct() based on field_id, preserving historical partition values across spec renames and correctly separating partition entries per specification layout.
Willingness to contribute
- I can contribute a fix for this bug independently
- I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- I cannot contribute a fix for this bug at this time
Activity