Repository navigation
Labels
enhancementNew feature or requestNew feature or request
Description
Is your feature request related to a problem or challenge?
DataFusion provides extensibility for functions (UDFs/UDAFs/UDWFs) and expressions (ExprPlanner), but lacks a clean way to customize how SQL table factors (FROM clause elements) are planned into logical plans. This makes it difficult to implement:
Table-valued functions like PostgreSQL'sgenerate_series()or SQL Server'sOPENJSON()TableFunctionactually covers this use case- Special SQL syntax currently unsupported by DataFusion like
PIVOT/UNPIVOT,TABLESAMPLE, orMATCH_RECOGNIZE
Existing workaround (see #17633) has a significant limitation in that it only work at query root: custom relations inside JOINs, CTEs, or subqueries aren't transformed.
Describe the solution you'd like
A RelationPlanner trait that intercepts table factor planning at any nesting level:
pub trait RelationPlanner: Debug + Send + Sync {
fn plan_relation(
&self,
relation: TableFactor,
context: &mut dyn RelationPlannerContext,
) -> Result<RelationPlanning>;
}Key features:
- Works for nested relations (JOINs, CTEs, subqueries)
- Chain multiple planners with
SessionContext::register_relation_planner() - Access SQL utilities via
RelationPlannerContext(expression conversion, identifier normalization) - Return
Planned(plan)to handle, orOriginal(relation)to pass through
Example usage:
ctx.register_relation_planner(Arc::new(MatchRecognizePlanner))?;
ctx.sql("SELECT * FROM stocks MATCH_RECOGNIZE(...);").await?;Describe alternatives you've considered
- PR feat: Add example for implementing
SAMPLEusing extension APIs #17633: CustomQueryPlannerapproach, but only transforms root-level relations
Additional context
Related to:
- Examples of extending SQL syntax #17824 - Examples of extending SQL syntax
- Support data source sampling with TABLESAMPLE #13563 - Support data source sampling with TABLESAMPLE
- feat: Add example for implementing
SAMPLEusing extension APIs #17633 - Attempting TABLESAMPLE via extensions (limited to root relations)
This complements existing extensibility APIs:
ExprPlanner: custom operators/expressionsTypePlanner: custom data typesRelationPlanner(new): custom table factors
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request
Activity
geoffreyclaude commentedon Oct 15, 2025
take
alamb commentedon Oct 15, 2025
You may also be able to use the TableFunction API:
https://docs.rs/datafusion/latest/datafusion/catalog/trait.TableFunctionImpl.html
https://docs.rs/datafusion/latest/datafusion/catalog/struct.TableFunction.html
It may not be the same, but maybe related (I don't think it is documented super well, FWIW)
geoffreyclaude commentedon Oct 16, 2025
Indeed, it's obviously a better solution for implementing table functions. It doesn't help other constructs though, such as PIVOT, MATCH_RECOGNIZE, or TABLESAMPLE. I'll update the issue to be clearer about this.
I could even also see a valid argument to move existing complex functionalities out of the core DataFusion and into a
datafusion-extended-sqlpackage or extension of some sort (UNNEST or table functions for instance,) but that's out of scope of this particular issue!Add relation planner extension support to customize SQL planning (#17843