Skip to content

Core: Honor identity sort orders for Arrow table writes - #4449

Open
Anurag-S1ngh wants to merge 2 commits into
apache:mainfrom
Anurag-S1ngh:feat/3848-honor-identity-sort-order
Open

Anurag-S1ngh wants to merge 2 commits into
apache:mainfrom
Anurag-S1ngh:feat/3848-honor-identity-sort-order

Conversation

@Anurag-S1ngh

Copy link
Copy Markdown
Contributor

Rationale for this change

Arrow table writes ignored the table sort order and always wrote sort_order_id=None, so manifests disagreed with the table metadata and readers could not prune on sort order.

Sort materialized Arrow tables when every sort field uses an identity transform, the column exists at the top level, and all fields agree on null placement. Unpartitioned tables are sorted before bin packing and each partition is sorted on its own. The table sort order id is written on each data file.

Unsupported cases (non-identity transforms, nested or missing columns, mixed null placement, streaming RecordBatchReader writes) keep writing unsorted files and emit a warning.

Are these changes tested?

Yes. New tests in tests/io/test_pyarrow.py cover the four direction and null-placement combinations, the unsupported cases, sorted files per partition, and the streaming warning. tests/table and the local write tests pass.

Are there any user-facing changes?

Yes. Tables with an identity sort order now write physically sorted data files that carry the sort order id. Previously written files are unchanged.

Sort materialized Arrow writes that use only identity sort fields with a
consistent null order, and record the table's sort order id on the written
data files. Each partition is sorted independently.

Sort orders that cannot be applied, such as non-identity transforms, nested
or missing sort columns, mixed null ordering, and streaming readers, leave
the files marked unsorted and emit a warning.
write_file expects an Iterator, and mypy rejects a list on Python 3.11.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant