Skip to content

Standardize uuid values on uuid.UUID to fix uuid partition filters - #4370

Open
rahulsmahadev wants to merge 3 commits into
apache:mainfrom
rahulsmahadev:rahul/uuid-partition-filter
Open

rahulsmahadev wants to merge 3 commits into
apache:mainfrom
rahulsmahadev:rahul/uuid-partition-filter

Conversation

@rahulsmahadev

@rahulsmahadev rahulsmahadev commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Rationale for this change

Scan planning gets uuid identity partitions wrong. On a table partitioned by identity(u) on a uuid column:

tbl.scan(row_filter=EqualTo("u", str(first))).plan_files()  # no files; count() is 0
tbl.scan(row_filter=In("u", {str(first), str(second)})).plan_files()  # no files
tbl.scan(row_filter=NotEqualTo("u", str(first))).count()  # counts the excluded partition too

PyIceberg has two in-memory representations of a uuid: the Avro UUIDReader returns a uuid.UUID, so the partition values read from manifests are UUID, while UUIDLiteral and from_bytes(UUIDType) use the 16 bytes. The partition and residual evaluators compare a UUID with bytes and never find them equal, and so does the filter on an identity partition value projected into a data file that does not contain the column.

This change standardizes on uuid.UUID, as Java does (UUIDLiteral is a Literal<UUID>, and Conversions.fromByteBuffer returns a UUID):

  • UUIDLiteral wraps a uuid.UUID. literal(), StringLiteral.to(UUIDType), FixedLiteral.to(UUIDType) and BinaryLiteral.to(UUIDType) build one, and UUIDLiteral.to(FixedType) and to(BinaryType) convert back with .bytes.
  • from_bytes(UUIDType) returns a uuid.UUID, so the metrics evaluators compare the lower and upper bounds as UUID.
  • _to_partition_representation(UUIDType) keeps identity partition values as UUID.
  • PyArrow gets the 16 bytes for uuid filter scalars and IN and NOT IN lists; PyArrow 18 rejects a uuid.UUID there.

Ordering is unchanged: UUID compares by its integer value, which matches the unsigned big-endian byte order, so range predicates and min/max pruning keep working. The conversion in the scan planner from the first version of this PR is gone.

Follow-up to #4360. When a uuid predicate reaches PyArrow against a uuid column stored in a data file, to_arrow() still raises the ArrowNotImplementedError of #2372, and so does count() when it has to read the rows. For = and in filters on matching identity partitions, this replaces silently returning no rows.

Are these changes tested?

Yes. In tests/io/test_pyarrow.py, test_scan_plans_uuid_identity_partitions checks the planned partitions and count() for =, in, != and not in filters on a uuid identity partition, and test_identity_transform_column_projection_uuid filters on a uuid identity partition column of a file that does not contain it. Both fail on main and pass with this change. test_scan_prunes_uuid_files_by_bounds checks that the bounds still prune files, test_uuid_expressions_to_pyarrow checks the Arrow filter values, and test_uuid_identity_partition_record_value_and_path in tests/table/test_partitioning.py checks the partition value and path. The tests that pinned the bytes representation now expect a uuid.UUID (tests/expressions/test_literals.py, tests/test_conversions.py and test_determine_partitions_with_uuid_column). The uuid tests also pass with PyArrow 18.

Are there any user-facing changes?

Yes, worth a note in the changelog. UUIDLiteral now takes a uuid.UUID instead of bytes, and UUIDLiteral.value and from_bytes(UUIDType, ...) return a uuid.UUID; callers that need the encoded value can use .bytes. New uuid identity partition paths use the uuid string, as in Java, instead of base64, and so do the uuid partition bounds in inspect.manifests() and inspect.all_manifests().

This pull request and its description were written by Isaac.

Manifests are read with uuid partition values as uuid.UUID, while UUID literals
hold the 16 bytes, so partition filters and residuals on a uuid identity
partition never matched: EqualTo and In planned no files, and NotEqualTo and
NotIn planned every file. Give partition values the literal representation
before the partition and residual evaluators compare them.

Co-authored-by: Isaac <no-reply@databricks.com>
@rahulsmahadev

Copy link
Copy Markdown
Contributor Author

@dossett could you take a look when you have a chance? It is the uuid partition filter follow-up from #4360: a small change in ManifestGroupPlanner, next to the residual cache from #3666.

@Fokko Fokko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for digging into this! I think the root cause is that we have two in-memory representations of a uuid: the Avro UUIDReader returns uuid.UUID (and #4360 asserts that for partition[0]), while UUIDLiteral and from_bytes(UUIDType) use bytes. Rather than converting at the evaluator call sites, I'd prefer to standardize on uuid.UUID. This is also what Java does: UUIDLiteral is a Literal<UUID> and Conversions.fromByteBuffer returns a UUID.

Some pointers:

Ordering is preserved: UUID compares by its integer value, which matches unsigned big endian byte order, so range predicates and min/max pruning keep working. The visible change is that UUIDLiteral.value becomes a UUID, which is worth a note in the changelog.

This also covers the projected identity values, where Record(field_value) is built from the partition value and compared against the literal. That path has the same mismatch and isn't touched by this PR.

Comment thread pyiceberg/table/__init__.py Outdated
return pa.RecordBatchReader.from_batches(target_schema, batches).cast(target_schema)


def _partition_with_literal_values(partition_type: StructType) -> Callable[[Record], Record]:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we should patch the Record here. It feels like we're working around the actual issue: partition values are uuid.UUID while UUIDLiteral holds bytes. Converting at two call sites in the planner means every other place that evaluates a partition Record still has to know about this. For example, the projected identity values in visitors.py build Record(field_value) from the partition value and run into the same mismatch. It also allocates a new Record for every data file on uuid specs, and adds another spec keyed cache to plan_files. If we make both representations agree (see my comment above), this helper can go away.

rahulsmahadev and others added 2 commits November 16, 2026 14:25
The Avro reader returns uuid.UUID for uuid values, while UUIDLiteral and
from_bytes(UUIDType) used the 16 bytes, so the partition and residual
evaluators, and the projected identity partition values, compared a UUID with
bytes. Standardize on uuid.UUID, as Java does (Literal<UUID>,
Conversions.fromByteBuffer): UUIDLiteral wraps a UUID, the string, fixed and
binary conversions build one, from_bytes(UUIDType) returns one, and identity
partition values stay UUID. PyArrow gets the 16 bytes. This replaces the
partition value conversion in the scan planner.

Co-authored-by: Isaac <no-reply@databricks.com>
rahulsmahadev added a commit to rahulsmahadev/iceberg-python that referenced this pull request Nov 16, 2026
Pass the decoded lower and upper bounds through _storage_value, like the
partition values, so a uuid bound is built from its 16 bytes even when
from_bytes returns a uuid.UUID (proposed in apache#4370).

Co-authored-by: Isaac <no-reply@databricks.com>
@rahulsmahadev rahulsmahadev changed the title Fix filters on a table identity-partitioned on a uuid column Standardize uuid values on uuid.UUID to fix uuid partition filters Nov 16, 2026
@rahulsmahadev

Copy link
Copy Markdown
Contributor Author

@Fokko thanks, that is much cleaner. Done in a4f286a, following your pointers: uuid literals, decoded bounds and identity partition values now use uuid.UUID, and Arrow filter values are converted to bytes, also for IN and NOT IN lists since PyArrow 18 rejects a uuid.UUID there. The planner helper is gone, and a new test covers filtering on an identity partition column that is missing from a data file. I updated the description with the user-visible type and path changes for the changelog. #4358 got a small commit so the two PRs combine in either order, and your comments on #4371 are addressed in bccbed0.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants