Skip to content

feat: abstract Iceberg file discovery behind a FileScanTaskProvider trait - #109

Open
gabotechs wants to merge 1 commit into
apache:mainfrom
gabotechs:file-scan-task-provider
Open

gabotechs wants to merge 1 commit into
apache:mainfrom
gabotechs:file-scan-task-provider

Conversation

@gabotechs

@gabotechs gabotechs commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Alternative to #67 for supporting both lazy and eager execution modes.

The idea here is to, rather than using the datafusion::DataSource as the trait that allows switching lazy and eager execution mode, to have a single DataSource implementation and create a new extension point hosted in this project for customizing FileScanTask streaming.

                      IcebergDataSource
        reader, limit, metrics, EXPLAIN prefix, partitioning
                              │
                 Arc<dyn FileScanTaskProvider>
                    ┌─────────┴──────────┐
     LazyFileScanTaskProvider      EagerFileScanTaskProvider
     discovers files on first      file groups planned up front,
     poll, 1 partition             N partitions, memory reserved

This way, enabling eager planning is just a matter of switching the trait FileScanTaskProviderimplementation. This new trait should not only serve as the granular extension point for abstracting lazy VS eager behavior over the same singleIcebergDataSource, but also serve as a public extension point for people to inject their own custom FileScanTask` stream providers.

For example, in DFD, we have the IcebergWorkUnitFeed that can seamlessly stream FileScanTasks either in-memory in singe-node or over the wire through a coordinator->worker connection. We could make IcebergWorkUnitFeed implement FileScanTaskProvider and finally stopping to host our custom DataSource implementations in DFD.

/// (source.file_scan_task_provider() as &dyn Any).downcast_ref::<EagerFileScanTaskProvider>()
/// }
/// ```
pub trait FileScanTaskProvider: Any + Debug + Send + Sync {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alexanderbianchi this could be hour hooking point in DFD for injecting our WorkUnitFeed, so the idea here is to kill two birds with one stone:

  • providing polymorphism that supports both eager and lazy execution modes
  • provide a public hooking point for injecting custom FileScanTask streams, like DFD's WorkUnitFeed

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes that was the idea in #107. Similar API, I like yours returning the partitioning instead of the count more. I don't know I'd considered it also a solution for the eager path (or if I did, I had forgotten 😆)

pub trait FileScanTaskSource: Clone + Debug + Send + Sync + 'static {
    /// Number of local output partitions, not a worker or distributed task count.
    fn partition_count(&self) -> usize;

    /// Opens a partition's stream during execution, without collecting it.
    ///
    /// Discovery can start here or when the stream is polled. Calls on one
    /// instance may coordinate discovery; a repeated claim of the same partition
    /// may be rejected. Implementations must not block waiting for tasks here.
    fn tasks(
        &self,
        partition: usize,
        context: Arc<TaskContext>,
    ) -> Result<FileScanTaskStream>;

    /// Appends task-source details to the data source's explain output.
    fn fmt_as(&self, _f: &mut Formatter<'_>) -> std::fmt::Result {
        Ok(())
    }
}

@gabotechs
gabotechs force-pushed the file-scan-task-provider branch 2 times, most recently from 0e38c7f to 715afa7 Compare November 17, 2026 14:23
@gabotechs
gabotechs force-pushed the file-scan-task-provider branch from 715afa7 to e6583d2 Compare November 17, 2026 14:32
let iceberg_cfg = IcebergDataFusionConfig::from_config_or_default(state.config());
if iceberg_cfg.execution.eager_file_planning {
builder = builder
.try_with_eagerly_planned_file_scan_tasks(state)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we just be passing IcebergDataFusionConfig into the builder instead of if/else in each table provider? For example ArrowOptions I imagine will also need to be passed in, as would most options that IcebergDataFusionConfig eventually has.


pub struct FileScanTaskProviderContext<'a> {
pub task_context: &'a Arc<TaskContext>,
pub table: &'a Table,

@alexanderbianchi alexanderbianchi Nov 17, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like if we also had a way to construct from not a table.

IMO there should be a way to pass in just FileScanTask and construct a data source,

  • FileScanTask is a public API in iceberg rust
  • Comet also passes FileScanTasks directly to DF, bypassing the Iceberg Catalog->Table->manifset planning path.
  • The Table in iceberg rust contains a lot. (FileIO, Cache, runtime)
  • It's needed for reconstruction, although I amm supportive of offering one codec from this library that could be shared between DFD / Ballista.

@alexanderbianchi alexanderbianchi left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a good improvement and should be the foundation of the eager datasource + increase composability of the lazy path. Still some details to hash out that I think can be iterated on safely.

if iceberg_cfg.execution.eager_file_planning {
builder = builder
.try_with_eagerly_planned_file_scan_tasks(state)
.await?;

@alexanderbianchi alexanderbianchi Nov 17, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we just make build() async instead of having an async configuration? IMO more clear, and if anything in the builder impacts the manifest pruning it'll be available in Build() and we won't have to worry about the "state" at the time try_with_eagerly_planned_file_scan_tasks. Alternatively we can have a build_eagerly() or something like it that's async, but always called at the end. For example .with_snapshot_id(...) impacts try_with_eagerly_planned_file_scan_tasks so ordering in the builder matters.

/// Discovers files while executing, in a single partition: the files of a
/// snapshot that may match a predicate.
#[derive(Debug, Clone)]
pub struct LazyFileScanTaskProvider {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we might be able to push this even further

  struct LazyFileScanTaskProvider {
      table: Table,
      snapshot_id: Option<i64>,
      projection: Vec<String>,
      predicate: Option<Predicate>,
  }

and

  struct EagerFileScanTaskProvider {
      groups: Vec<Vec<FileScanTask>>,
  }

Such that we can remove some of the logic from the data source, and it's builder.

   let discovery = Arc::new(LazyFileScanTaskProvider {
       table,
       snapshot_id,
       projection,
       predicate,
   });

   let provider: Arc<dyn FileScanTaskProvider> =
       if iceberg_config.execution.eager_file_planning {
           Arc::new(
               EagerFileScanTaskProvider::plan(discovery, context).await?,
           )
       } else {
           discovery
       };

   let source = IcebergDataSourceBuilder::new(schema, file_io, reader)
       .with_file_scan_task_provider(provider)
       .with_fetch(limit)
       .build()?;

Something like this, isolates the async out of the data source builder and removes the Table from the data source.

@alexanderbianchi

Copy link
Copy Markdown

Playing around with this approach + my comments here #113

snapshot_id: Option<i64>,
predicate: Option<Predicate>,
fetch: Option<usize>,
collected_file_scan_tasks: Option<Vec<Vec<FileScanTask>>>,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is kinda an abstraction leak from the FileScanTaskProvider. We don't need the Table after we have the fileScanTasks and the Table itself is needed only for Lazy.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants