Skip to content

update_spec: removing a partition field and adding one with the same name raises "Cannot add duplicate partition field name" #4306

Closed

Description

@sohailshk

Apache Iceberg version

main (development)

Please describe the bug 🐞

Replacing a partition field while keeping its name, for example to change the number of buckets, fails:

from pyiceberg.catalog.sql import SqlCatalog
from pyiceberg.schema import Schema
from pyiceberg.transforms import BucketTransform
from pyiceberg.types import LongType, NestedField

catalog = SqlCatalog("default", uri="sqlite:///:memory:", warehouse="file:///tmp/warehouse")
catalog.create_namespace("db")
schema = Schema(NestedField(1, "x", LongType()), NestedField(2, "y", LongType()))
table = catalog.create_table("db.events", schema)

with table.update_spec() as update:
    update.add_field("y", BucketTransform(16), "y_shard")

# Change the number of buckets but keep the partition name
with table.update_spec() as update:
    update.remove_field("y_shard")
    update.add_field("y", BucketTransform(32), "y_shard")
ValueError: Cannot add duplicate partition field name: y_shard

UpdateSpec.add_field checks whether the id of the new field is in the set of removed fields, instead of the existing field with that name, so the pending removal is never taken into account. Java's BaseUpdatePartitionSpec.addField checks the existing field, allows the add when that field is being deleted, and renames the deleted field to <name>_<fieldId>.

Willingness to contribute

  • I can contribute a fix for this bug independently

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions