Repository navigation
fix: write each batch's partitions in first-row order for the clustered writer - #81
CaptainAni187 wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
If the input batch already contains the partition order, and we lose it through RecordBatchPartitionSplitter::split, could we fix this in iceberg-rust by preserving first-occurrence order in the splitter’s output?
It already tracks row indexes. Could we sort the existing groups by their first row index before building the output batches, and document that ordering guarantee?
That would add O(P log P) ordering work for P partitions, while avoiding the extra O(N) conversion and temporary allocation for N rows introduced here.
|
@alexanderbianchi good point, that's a nicer place for it. The splitter already has the row ids per group, so sorting the groups by their first row id gets first-occurrence order without converting the partition column again. I can open that on iceberg-rust (with a doc note on the ordering). This repo pins iceberg-rust to a rev though, so @andygrove would you rather keep this as a stopgap until the bump, or drop it and wait for the upstream fix? |
Closes #39
With fanout disabled,
TaskWriterpasses the pieces fromRecordBatchPartitionSplitter::splitstraight toClusteredWriter, but the splitter groups rows in aHashMap, so they come out in arbitrary order. When a partition spans a batch boundary,ClusteredWritercan see it closed and then reopened, and fails with "Cannot write to partition that was previously closed" even though the input is sorted.This went with the
TaskWriterside of the two options in the issue, so no iceberg-rust change is needed. On the clustered path the pieces are now sorted by the first row their partition value appears in the batch. The values come from the batch's_partitioncolumn througharrow_struct_to_literal, the same conversion the splitter uses, so the grouping matches. Fanout and unpartitioned writes are unchanged.This doesn't touch #38: the optimizer can still drop the sort before the writer, so fanout-disabled inserts through SQL need that one too.
Test:
test_task_writer_clustered_partition_spans_batcheswrites 50 sorted partitions of 3 rows in batches of 8, so most batches split a partition. Without the fix it fails every run with the error from the issue.