Skip to content

feat(scan): carry the data file's split offsets on FileScanTask - #3704

Open
CTTY wants to merge 2 commits into
apache:mainfrom
CTTY:feat/file-scan-task-split-offsets
Open

CTTY wants to merge 2 commits into
apache:mainfrom
CTTY:feat/file-scan-task-split-offsets

Conversation

@CTTY

@CTTY CTTY commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator

Which issue does this PR close?

  • Closes #.

What changes are included in this PR?

An engine that splits a data file into several read tasks must cut on the row-group boundaries the writer recorded as split_offsets (spec field 132), or a split can land inside a row group and one of the two tasks opens the file for nothing. FileScanTask had no way to hand them on, so an engine planning with iceberg-rust had to split arithmetically.

  • FileScanTask::split_offsets: Option<Vec<i64>>, builder default, accessor, serde in both directions; None when the writer recorded none.
  • Populated from the manifest entry's DataFile in ManifestEntryContext:: into_file_scan_task.
  • TableTestFixture::setup_manifest_files_with_split_offsets and a scan test covering recorded, empty and absent offsets; the serde round-trip test carries the field.

Are these changes tested?

  • Added unit tests

AI Disclosure

Used claude

An engine that splits a data file into several read tasks must cut on the
row-group boundaries the writer recorded as `split_offsets` (spec field 132),
or a split can land inside a row group and one of the two tasks opens the
file for nothing. `FileScanTask` had no way to hand them on, so an engine
planning with iceberg-rust had to split arithmetically.

Measured with Spark 3.5.8 planning through a JNI binding of this crate: on a
table of sixty 30 MB single-row-group Parquet files on S3, Spark produced 120
input partitions instead of 60 and the scan stage took twice as long as with
Iceberg Java planning, although the plan itself was two orders of magnitude
faster. With the offsets carried the two planners yield identical splits.

- `FileScanTask::split_offsets: Option<Vec<i64>>`, builder default, accessor,
  serde in both directions; `None` when the writer recorded none.
- Populated from the manifest entry's `DataFile` in `ManifestEntryContext::
  into_file_scan_task`.
- `TableTestFixture::setup_manifest_files_with_split_offsets` and a scan test
  covering recorded, empty and absent offsets; the serde round-trip test
  carries the field.
@CTTY
CTTY marked this pull request as ready for review November 16, 2026 22:15

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant