Repository navigation
Standardize uuid values on uuid.UUID to fix uuid partition filters - #4370
rahulsmahadev wants to merge 3 commits into
Conversation
Manifests are read with uuid partition values as uuid.UUID, while UUID literals hold the 16 bytes, so partition filters and residuals on a uuid identity partition never matched: EqualTo and In planned no files, and NotEqualTo and NotIn planned every file. Give partition values the literal representation before the partition and residual evaluators compare them. Co-authored-by: Isaac <no-reply@databricks.com>
Fokko
left a comment
There was a problem hiding this comment.
Thanks for digging into this! I think the root cause is that we have two in-memory representations of a uuid: the Avro UUIDReader returns uuid.UUID (and #4360 asserts that for partition[0]), while UUIDLiteral and from_bytes(UUIDType) use bytes. Rather than converting at the evaluator call sites, I'd prefer to standardize on uuid.UUID. This is also what Java does: UUIDLiteral is a Literal<UUID> and Conversions.fromByteBuffer returns a UUID.
Some pointers:
UUIDLiteral: wrap auuid.UUIDinstead ofbytes.ser_modelalready returns aUUID.literal()andStringLiteral.to(UUIDType): stop calling.bytes.FixedLiteral.to(UUIDType)andBinaryLiteral.to(UUIDType): buildUUID(bytes=...), and convert back with.bytesinUUIDLiteral.to(FixedType)/UUIDLiteral.to(BinaryType).from_bytes(UUIDType): returnuuid.UUID(bytes=b), so the metrics evaluators also compare the lower and upper bounds asUUID._to_partition_representation(UUIDType): keep identity values asUUIDinstead of converting them tobytes._convert_scalar: convert aUUIDto.byteswhen building thepa.uuid()scalar for the Arrow filter.
Ordering is preserved: UUID compares by its integer value, which matches unsigned big endian byte order, so range predicates and min/max pruning keep working. The visible change is that UUIDLiteral.value becomes a UUID, which is worth a note in the changelog.
This also covers the projected identity values, where Record(field_value) is built from the partition value and compared against the literal. That path has the same mismatch and isn't touched by this PR.
| return pa.RecordBatchReader.from_batches(target_schema, batches).cast(target_schema) | ||
|
|
||
|
|
||
| def _partition_with_literal_values(partition_type: StructType) -> Callable[[Record], Record]: |
There was a problem hiding this comment.
I don't think we should patch the Record here. It feels like we're working around the actual issue: partition values are uuid.UUID while UUIDLiteral holds bytes. Converting at two call sites in the planner means every other place that evaluates a partition Record still has to know about this. For example, the projected identity values in visitors.py build Record(field_value) from the partition value and run into the same mismatch. It also allocates a new Record for every data file on uuid specs, and adds another spec keyed cache to plan_files. If we make both representations agree (see my comment above), this helper can go away.
The Avro reader returns uuid.UUID for uuid values, while UUIDLiteral and from_bytes(UUIDType) used the 16 bytes, so the partition and residual evaluators, and the projected identity partition values, compared a UUID with bytes. Standardize on uuid.UUID, as Java does (Literal<UUID>, Conversions.fromByteBuffer): UUIDLiteral wraps a UUID, the string, fixed and binary conversions build one, from_bytes(UUIDType) returns one, and identity partition values stay UUID. PyArrow gets the 16 bytes. This replaces the partition value conversion in the scan planner. Co-authored-by: Isaac <no-reply@databricks.com>
Pass the decoded lower and upper bounds through _storage_value, like the partition values, so a uuid bound is built from its 16 bytes even when from_bytes returns a uuid.UUID (proposed in apache#4370). Co-authored-by: Isaac <no-reply@databricks.com>
|
@Fokko thanks, that is much cleaner. Done in a4f286a, following your pointers: uuid literals, decoded bounds and identity partition values now use |
Rationale for this change
Scan planning gets uuid identity partitions wrong. On a table partitioned by
identity(u)on auuidcolumn:PyIceberg has two in-memory representations of a uuid: the Avro
UUIDReaderreturns auuid.UUID, so the partition values read from manifests areUUID, whileUUIDLiteralandfrom_bytes(UUIDType)use the 16 bytes. The partition and residual evaluators compare aUUIDwithbytesand never find them equal, and so does the filter on an identity partition value projected into a data file that does not contain the column.This change standardizes on
uuid.UUID, as Java does (UUIDLiteralis aLiteral<UUID>, andConversions.fromByteBufferreturns aUUID):UUIDLiteralwraps auuid.UUID.literal(),StringLiteral.to(UUIDType),FixedLiteral.to(UUIDType)andBinaryLiteral.to(UUIDType)build one, andUUIDLiteral.to(FixedType)andto(BinaryType)convert back with.bytes.from_bytes(UUIDType)returns auuid.UUID, so the metrics evaluators compare the lower and upper bounds asUUID._to_partition_representation(UUIDType)keeps identity partition values asUUID.INandNOT INlists; PyArrow 18 rejects auuid.UUIDthere.Ordering is unchanged:
UUIDcompares by its integer value, which matches the unsigned big-endian byte order, so range predicates and min/max pruning keep working. The conversion in the scan planner from the first version of this PR is gone.Follow-up to #4360. When a uuid predicate reaches PyArrow against a uuid column stored in a data file,
to_arrow()still raises theArrowNotImplementedErrorof #2372, and so doescount()when it has to read the rows. For=andinfilters on matching identity partitions, this replaces silently returning no rows.Are these changes tested?
Yes. In
tests/io/test_pyarrow.py,test_scan_plans_uuid_identity_partitionschecks the planned partitions andcount()for=,in,!=andnot infilters on a uuid identity partition, andtest_identity_transform_column_projection_uuidfilters on a uuid identity partition column of a file that does not contain it. Both fail on main and pass with this change.test_scan_prunes_uuid_files_by_boundschecks that the bounds still prune files,test_uuid_expressions_to_pyarrowchecks the Arrow filter values, andtest_uuid_identity_partition_record_value_and_pathintests/table/test_partitioning.pychecks the partition value and path. The tests that pinned the bytes representation now expect auuid.UUID(tests/expressions/test_literals.py,tests/test_conversions.pyandtest_determine_partitions_with_uuid_column). The uuid tests also pass with PyArrow 18.Are there any user-facing changes?
Yes, worth a note in the changelog.
UUIDLiteralnow takes auuid.UUIDinstead ofbytes, andUUIDLiteral.valueandfrom_bytes(UUIDType, ...)return auuid.UUID; callers that need the encoded value can use.bytes. New uuid identity partition paths use the uuid string, as in Java, instead of base64, and so do the uuid partition bounds ininspect.manifests()andinspect.all_manifests().This pull request and its description were written by Isaac.