Repository navigation
feat(python): read manifests and manifest lists as pyarrow RecordBatchReaders - #3652
Conversation
- Drop the unused manifest_length, partition_spec_id, and min_sequence_number arguments from read_manifest_entries_as_arrow. - Share decoding, projection, and stop-after-error between the two readers through a private ProjectingReader. - Restore the original schema wording in the reader docs. - Check for multiple batches instead of the batch size constant in the Python test. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ty records arrow-avro 60 decodes records with no fields (apache/arrow-rs#10771), so the entry reader no longer projects out the empty partition of unpartitioned manifests. It now reads the header from its own reader instead of a separate borrowing one. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
since the arrow 60 pr (#3257) is merged, i pushed a commit to get rid of the extra code for it, and rebased this pr with latest main |
|
Still reviewing, i want to poke at the api boundary a bit more. Here are a couple of small call outs, neither blocking:
EDIT: |
Added docs and a test that make this explicit. |
…er-stream # Conflicts: # crates/iceberg/public-api.txt
kevinjqliu
left a comment
There was a problem hiding this comment.
LGTM
1 nit on function name
| added_snapshot_id: i64, | ||
| first_row_id: Option<u64>, | ||
| ) -> PyResult<Bound<'_, PyAny>> { | ||
| // Inheritance reads only the fields set from the arguments. |
There was a problem hiding this comment.
this feels a bit awkward but i think its the best we can do right now without major refactor
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
I push a commit (67afbbf) to rename the functions to use |
Which issue does this PR close?
What changes are included in this PR?
read_manifest_entries(feat(arrow): read manifest entries into Arrow with arrow-avro #3388) andread_manifest_list(feat(arrow): read manifest lists into Arrow with arrow-avro #3631) haven't been released, so their signatures change without a deprecation.The batch size is 1024 rows. An entry's size grows with the table's column count through the column stats maps. These are the Arrow sizes of 8 MB manifests written by PyIceberg with stats on every long column. The stats compress well, which fits the most entries in 8 MB.
main)Wider tables have larger entries, so fewer fit in an 8 MB manifest. At 1000 columns, a manifest holds fewer than 1024 entries and reads as one batch at either size.
Are these changes tested?
crates/iceberg/src/arrow/manifest.rs.bindings/python/tests/test_manifest.py, which compare the binding's output with PyIceberg's own readers.AI Disclosure
I wrote this PR with help from Claude but understand and support the core approach, implementation, and test coverage.