Repository navigation
refactor(scan) scan plan extraction - #3294
Draft
xanderbailey wants to merge 3 commits into
Draft
xanderbailey wants to merge 3 commits into
xanderbailey wants to merge 3 commits into
Conversation
xanderbailey
commented
Sep 28, 2026
Comment on lines
-400
to
-402
| let Some(plan_context) = self.plan_context.as_ref() else { | ||
| return Ok(Box::pin(futures::stream::empty())); | ||
| }; |
Contributor
Author
There was a problem hiding this comment.
No longer optional
xanderbailey
commented
Sep 28, 2026
| use crate::spec::DataContentType; | ||
| use crate::{Error, ErrorKind, Result}; | ||
|
|
||
| pub(crate) async fn plan_tasks( |
Contributor
Author
There was a problem hiding this comment.
Moved so that we can eventually reuse this between snapshot scans and incremental append scans.
xanderbailey
commented
Sep 28, 2026
Comment on lines
-338
to
+326
| snapshot_schema: schema, | ||
| scan_schema: schema, |
Contributor
Author
There was a problem hiding this comment.
renaming these to make it more generic
xanderbailey
commented
Sep 28, 2026
| .as_ref() | ||
| .get_manifest_list(&self.snapshot, &self.table_metadata) | ||
| .await | ||
| } |
Contributor
Author
There was a problem hiding this comment.
Removed this indirection and loading is now done https://github.com/apache/iceberg-rust/pull/3294/changes#diff-744aa4b2b5886de6437005da5b1c9866b978d01158ad6d712b3a3f30daf8102dR245-R252
xanderbailey
commented
Sep 28, 2026
| } | ||
|
|
||
| #[test] | ||
| fn test_scan_without_any_snapshots_still_validates_projection() { |
Contributor
Author
There was a problem hiding this comment.
A real win from this refactor.
TableScan::plan_files held the whole fan-out: three channels, three spawned tasks, and the two per-entry filter functions. Move that into scan::plan::plan_tasks so mod.rs is left with the scan's public surface and its builder. BoundPredicates moves to context.rs alongside the contexts that carry it, rather than sitting in mod.rs next to the driver that reads it. Pure code motion; no behaviour change.
TableScan held an Option<PlanContext> purely to encode 'this table has no snapshots', which meant build() returned early before projecting or binding the filter, and plan_files() needed a short-circuit ahead of the real plan. Move the Option inward to PlanContext::snapshot: a table with no snapshots lists no manifests and so plans no files, which build_manifest_file_contexts now expresses directly by folding the manifest-list fetch in and iterating Option<Arc<ManifestList>>. This makes the projection and the filter validate on a table with no snapshots, where before they were skipped: select() of an absent column now fails at build() rather than yielding an empty scan. Java and PyIceberg both reject the column, and a scan that silently ignores a typo because the table happens to be empty is a trap. TableScan::snapshot() reads through to plan_context.snapshot, so the snapshot has one home rather than being mirrored on both.
The schema and bound predicate a plan carries are the scan's, not the snapshot's: they are derived from the snapshot's schema today, but a scan over a range of snapshots projects onto one schema that belongs to no single snapshot in the range. Name them for what they are. Scoped to the scan module; ArrowReader's RecordBatchTransformer keeps its own snapshot_schema field name.
xanderbailey
force-pushed
the
xb/scan-plan-extraction
branch
from
October 16, 2026 08:50
6567980 to
1f29386
Compare
Contributor
Author
|
@blackmwk curious to know if you agree with the direction here, I'm very keen to work on incremental append scans |
xanderbailey
marked this pull request as draft
October 28, 2026 17:13
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Simplification of #3108.
Which issue does this PR close?
What changes are included in this PR?
Prep for incremental append scans, which read the manifest lists of several snapshots rather than one.
#3108 added a
ManifestSourcetrait for this. I've dropped it: it only ever had one real impl, and an incremental scan also needs to filter manifest entries (status == ADDED,snapshot_idin range) which the trait has no hook for. Better to shape that in #2997 where there's a second consumer to shape it against.What's left is useful either way:
TableScan::plan_filesintoscan::plan::plan_tasksso an incremental scan can reuse it.PlanContextnon-optional onTableScanby moving theOptioninward toPlanContext::snapshot. That also letsbuild_manifest_file_contextsabsorbget_manifest_list, so there's one planning method rather than one feeding the other.snapshot_schematoscan_schema, since an incremental scan projects onto a schema that belongs to no single snapshot in the range.One behaviour change,
build()used to return early for a table with no snapshots, skipping projection and predicate binding, soSELECT "not_a_column" FROM tsucceeded when the metadata had no snapshot but failed once it had one. Both now fail. Can split that out if preferred.Are these changes tested?
New test:
test_scan_without_any_snapshots_still_validates_projection.AI Disclosure
Claude and I worked on this based on #3108