Repository navigation
Conversation
| /// (source.file_scan_task_provider() as &dyn Any).downcast_ref::<EagerFileScanTaskProvider>() | ||
| /// } | ||
| /// ``` | ||
| pub trait FileScanTaskProvider: Any + Debug + Send + Sync { |
There was a problem hiding this comment.
@alexanderbianchi this could be hour hooking point in DFD for injecting our WorkUnitFeed, so the idea here is to kill two birds with one stone:
- providing polymorphism that supports both eager and lazy execution modes
- provide a public hooking point for injecting custom FileScanTask streams, like DFD's WorkUnitFeed
There was a problem hiding this comment.
Yes that was the idea in #107. Similar API, I like yours returning the partitioning instead of the count more. I don't know I'd considered it also a solution for the eager path (or if I did, I had forgotten 😆)
pub trait FileScanTaskSource: Clone + Debug + Send + Sync + 'static {
/// Number of local output partitions, not a worker or distributed task count.
fn partition_count(&self) -> usize;
/// Opens a partition's stream during execution, without collecting it.
///
/// Discovery can start here or when the stream is polled. Calls on one
/// instance may coordinate discovery; a repeated claim of the same partition
/// may be rejected. Implementations must not block waiting for tasks here.
fn tasks(
&self,
partition: usize,
context: Arc<TaskContext>,
) -> Result<FileScanTaskStream>;
/// Appends task-source details to the data source's explain output.
fn fmt_as(&self, _f: &mut Formatter<'_>) -> std::fmt::Result {
Ok(())
}
}0e38c7f to
715afa7
Compare
715afa7 to
e6583d2
Compare
| let iceberg_cfg = IcebergDataFusionConfig::from_config_or_default(state.config()); | ||
| if iceberg_cfg.execution.eager_file_planning { | ||
| builder = builder | ||
| .try_with_eagerly_planned_file_scan_tasks(state) |
There was a problem hiding this comment.
Should we just be passing IcebergDataFusionConfig into the builder instead of if/else in each table provider? For example ArrowOptions I imagine will also need to be passed in, as would most options that IcebergDataFusionConfig eventually has.
|
|
||
| pub struct FileScanTaskProviderContext<'a> { | ||
| pub task_context: &'a Arc<TaskContext>, | ||
| pub table: &'a Table, |
There was a problem hiding this comment.
I'd like if we also had a way to construct from not a table.
IMO there should be a way to pass in just FileScanTask and construct a data source,
FileScanTaskis a public API in iceberg rust- Comet also passes FileScanTasks directly to DF, bypassing the Iceberg Catalog->Table->manifset planning path.
- The Table in iceberg rust contains a lot. (FileIO, Cache, runtime)
- It's needed for reconstruction, although I amm supportive of offering one codec from this library that could be shared between DFD / Ballista.
| if iceberg_cfg.execution.eager_file_planning { | ||
| builder = builder | ||
| .try_with_eagerly_planned_file_scan_tasks(state) | ||
| .await?; |
There was a problem hiding this comment.
Should we just make build() async instead of having an async configuration? IMO more clear, and if anything in the builder impacts the manifest pruning it'll be available in Build() and we won't have to worry about the "state" at the time try_with_eagerly_planned_file_scan_tasks. Alternatively we can have a build_eagerly() or something like it that's async, but always called at the end. For example .with_snapshot_id(...) impacts try_with_eagerly_planned_file_scan_tasks so ordering in the builder matters.
| /// Discovers files while executing, in a single partition: the files of a | ||
| /// snapshot that may match a predicate. | ||
| #[derive(Debug, Clone)] | ||
| pub struct LazyFileScanTaskProvider { |
There was a problem hiding this comment.
I think we might be able to push this even further
struct LazyFileScanTaskProvider {
table: Table,
snapshot_id: Option<i64>,
projection: Vec<String>,
predicate: Option<Predicate>,
}and
struct EagerFileScanTaskProvider {
groups: Vec<Vec<FileScanTask>>,
}Such that we can remove some of the logic from the data source, and it's builder.
let discovery = Arc::new(LazyFileScanTaskProvider {
table,
snapshot_id,
projection,
predicate,
});
let provider: Arc<dyn FileScanTaskProvider> =
if iceberg_config.execution.eager_file_planning {
Arc::new(
EagerFileScanTaskProvider::plan(discovery, context).await?,
)
} else {
discovery
};
let source = IcebergDataSourceBuilder::new(schema, file_io, reader)
.with_file_scan_task_provider(provider)
.with_fetch(limit)
.build()?;Something like this, isolates the async out of the data source builder and removes the Table from the data source.
|
Playing around with this approach + my comments here #113 |
| snapshot_id: Option<i64>, | ||
| predicate: Option<Predicate>, | ||
| fetch: Option<usize>, | ||
| collected_file_scan_tasks: Option<Vec<Vec<FileScanTask>>>, |
There was a problem hiding this comment.
This is kinda an abstraction leak from the FileScanTaskProvider. We don't need the Table after we have the fileScanTasks and the Table itself is needed only for Lazy.
Alternative to #67 for supporting both lazy and eager execution modes.
The idea here is to, rather than using the
datafusion::DataSourceas the trait that allows switching lazy and eager execution mode, to have a singleDataSourceimplementation and create a new extension point hosted in this project for customizingFileScanTaskstreaming.This way, enabling eager planning is just a matter of switching the
traitFileScanTaskProviderimplementation. This new trait should not only serve as the granular extension point for abstracting lazy VS eager behavior over the same singleIcebergDataSource, but also serve as a public extension point for people to inject their own customFileScanTask` stream providers.For example, in DFD, we have the IcebergWorkUnitFeed that can seamlessly stream
FileScanTasks either in-memory in singe-node or over the wire through a coordinator->worker connection. We could makeIcebergWorkUnitFeedimplementFileScanTaskProviderand finally stopping to host our customDataSourceimplementations in DFD.