Skip to content

refactor(scan) scan plan extraction - #3294

Draft
xanderbailey wants to merge 3 commits into
apache:mainfrom
xanderbailey:xb/scan-plan-extraction
Draft

xanderbailey wants to merge 3 commits into
apache:mainfrom
xanderbailey:xb/scan-plan-extraction

Conversation

@xanderbailey

@xanderbailey xanderbailey commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Simplification of #3108.

Which issue does this PR close?

What changes are included in this PR?

Prep for incremental append scans, which read the manifest lists of several snapshots rather than one.

#3108 added a ManifestSource trait for this. I've dropped it: it only ever had one real impl, and an incremental scan also needs to filter manifest entries (status == ADDED, snapshot_id in range) which the trait has no hook for. Better to shape that in #2997 where there's a second consumer to shape it against.

What's left is useful either way:

  1. Move the plan fan-out out of TableScan::plan_files into scan::plan::plan_tasks so an incremental scan can reuse it.
  2. Make PlanContext non-optional on TableScan by moving the Option inward to PlanContext::snapshot. That also lets build_manifest_file_contexts absorb get_manifest_list, so there's one planning method rather than one feeding the other.
  3. Rename snapshot_schema to scan_schema, since an incremental scan projects onto a schema that belongs to no single snapshot in the range.

One behaviour change, build() used to return early for a table with no snapshots, skipping projection and predicate binding, so SELECT "not_a_column" FROM t succeeded when the metadata had no snapshot but failed once it had one. Both now fail. Can split that out if preferred.

Are these changes tested?

New test: test_scan_without_any_snapshots_still_validates_projection.

AI Disclosure

Claude and I worked on this based on #3108

Comment on lines -400 to -402
let Some(plan_context) = self.plan_context.as_ref() else {
return Ok(Box::pin(futures::stream::empty()));
};

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No longer optional

Comment thread crates/iceberg/src/scan/plan.rs Outdated
use crate::spec::DataContentType;
use crate::{Error, ErrorKind, Result};

pub(crate) async fn plan_tasks(

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved so that we can eventually reuse this between snapshot scans and incremental append scans.

Comment on lines -338 to +326
snapshot_schema: schema,
scan_schema: schema,

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

renaming these to make it more generic

.as_ref()
.get_manifest_list(&self.snapshot, &self.table_metadata)
.await
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

}

#[test]
fn test_scan_without_any_snapshots_still_validates_projection() {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A real win from this refactor.

TableScan::plan_files held the whole fan-out: three channels, three
spawned tasks, and the two per-entry filter functions. Move that into
scan::plan::plan_tasks so mod.rs is left with the scan's public surface
and its builder.

BoundPredicates moves to context.rs alongside the contexts that carry
it, rather than sitting in mod.rs next to the driver that reads it.

Pure code motion; no behaviour change.
TableScan held an Option<PlanContext> purely to encode 'this table has
no snapshots', which meant build() returned early before projecting or
binding the filter, and plan_files() needed a short-circuit ahead of the
real plan. Move the Option inward to PlanContext::snapshot: a table with
no snapshots lists no manifests and so plans no files, which
build_manifest_file_contexts now expresses directly by folding the
manifest-list fetch in and iterating Option<Arc<ManifestList>>.

This makes the projection and the filter validate on a table with no
snapshots, where before they were skipped: select() of an absent column
now fails at build() rather than yielding an empty scan. Java and
PyIceberg both reject the column, and a scan that silently ignores a
typo because the table happens to be empty is a trap.

TableScan::snapshot() reads through to plan_context.snapshot, so the
snapshot has one home rather than being mirrored on both.
The schema and bound predicate a plan carries are the scan's, not the
snapshot's: they are derived from the snapshot's schema today, but a scan
over a range of snapshots projects onto one schema that belongs to no
single snapshot in the range. Name them for what they are.

Scoped to the scan module; ArrowReader's RecordBatchTransformer keeps its
own snapshot_schema field name.
@xanderbailey
xanderbailey force-pushed the xb/scan-plan-extraction branch from 6567980 to 1f29386 Compare October 16, 2026 08:50
@xanderbailey

Copy link
Copy Markdown
Contributor Author

@blackmwk curious to know if you agree with the direction here, I'm very keen to work on incremental append scans

@xanderbailey
xanderbailey marked this pull request as draft October 28, 2026 17:13

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant