Skip to content

feat(python): read manifests and manifest lists as pyarrow RecordBatchReaders - #3652

Merged
kevinjqliu merged 10 commits into
apache:mainfrom
mbutrovich:arrow-manifest-reader-stream
Nov 18, 2026
Merged

kevinjqliu merged 10 commits into
apache:mainfrom
mbutrovich:arrow-manifest-reader-stream

Conversation

@mbutrovich

Copy link
Copy Markdown
Collaborator

Which issue does this PR close?

What changes are included in this PR?

  • read_manifest_entries (feat(arrow): read manifest entries into Arrow with arrow-avro #3388) and read_manifest_list (feat(arrow): read manifest lists into Arrow with arrow-avro #3631) haven't been released, so their signatures change without a deprecation.

  • The batch size is 1024 rows. An entry's size grows with the table's column count through the column stats maps. These are the Arrow sizes of 8 MB manifests written by PyIceberg with stats on every long column. The stats compress well, which fits the most entries in 8 MB.

    Table columns Entries in an 8 MB manifest Arrow size per entry One batch (main) Largest batch of 8192 Largest batch of 1024
    12 61,173 1.4 KB 85.2 MB 10.5 MB 1.3 MB
    100 9,935 7.3 KB 71.2 MB 70.0 MB 8.8 MB
    1000 810 86 KB 68.2 MB 68.2 MB (whole manifest) 68.2 MB (whole manifest)

    Wider tables have larger entries, so fewer fit in an 8 MB manifest. At 1000 columns, a manifest holds fewer than 1024 entries and reads as one batch at either size.

Are these changes tested?

  • Unit tests and doctests in crates/iceberg/src/arrow/manifest.rs.
  • Python tests in bindings/python/tests/test_manifest.py, which compare the binding's output with PyIceberg's own readers.

AI Disclosure

I wrote this PR with help from Claude but understand and support the core approach, implementation, and test coverage.

@mbutrovich
mbutrovich marked this pull request as ready for review November 6, 2026 21:32
mbutrovich and others added 4 commits November 10, 2026 11:20
- Drop the unused manifest_length, partition_spec_id, and
  min_sequence_number arguments from read_manifest_entries_as_arrow.
- Share decoding, projection, and stop-after-error between the two
  readers through a private ProjectingReader.
- Restore the original schema wording in the reader docs.
- Check for multiple batches instead of the batch size constant in the
  Python test.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ty records

arrow-avro 60 decodes records with no fields (apache/arrow-rs#10771), so the
entry reader no longer projects out the empty partition of unpartitioned
manifests. It now reads the header from its own reader instead of a separate
borrowing one.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@kevinjqliu

kevinjqliu commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

since the arrow 60 pr (#3257) is merged, i pushed a commit to get rid of the extra code for it, and rebased this pr with latest main

@kevinjqliu

kevinjqliu commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Still reviewing, i want to poke at the api boundary a bit more.

Here are a couple of small call outs, neither blocking:

  1. ArrowReaderBuilder and TableScanBuilder expose with_batch_size, here it's fixed at 1024. wdyt about making it configurable? fine to do in a follow up too

  2. nit: since these take bytes, maybe mention in the docs that callers need to decrypt first. ManifestReader and ManifestListReader handle that on the io path

EDIT:
for 1, we dont need to prematurely optimize this 😄
for 2, its nit, we can do it later

@mbutrovich

Copy link
Copy Markdown
Collaborator Author
  1. nit: since these take bytes, maybe mention in the docs that callers need to decrypt first. ManifestReader and ManifestListReader handle that on the io path

Added docs and a test that make this explicit.

@kevinjqliu kevinjqliu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

1 nit on function name

Comment thread bindings/python/src/manifest.rs Resolved
added_snapshot_id: i64,
first_row_id: Option<u64>,
) -> PyResult<Bound<'_, PyAny>> {
// Inheritance reads only the fields set from the arguments.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this feels a bit awkward but i think its the best we can do right now without major refactor

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@kevinjqliu
kevinjqliu enabled auto-merge November 18, 2026 18:29
@kevinjqliu

Copy link
Copy Markdown
Contributor

I push a commit (67afbbf) to rename the functions to use _as_arrow_reader

@kevinjqliu
kevinjqliu added this pull request to the merge queue Nov 18, 2026
Merged via the queue into apache:main with commit 641f590 Nov 18, 2026
24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Return manifests and manifest lists as pyarrow tables from pyiceberg-core Decide whether the Arrow manifest readers return an iterator of batches

2 participants