Skip to content

Relation Planner Extension API #18078

Closed

Description

@geoffreyclaude

Is your feature request related to a problem or challenge?

DataFusion provides extensibility for functions (UDFs/UDAFs/UDWFs) and expressions (ExprPlanner), but lacks a clean way to customize how SQL table factors (FROM clause elements) are planned into logical plans. This makes it difficult to implement:

  • Table-valued functions like PostgreSQL's generate_series() or SQL Server's OPENJSON() TableFunction actually covers this use case
  • Special SQL syntax currently unsupported by DataFusion like PIVOT/UNPIVOT, TABLESAMPLE, or MATCH_RECOGNIZE

Existing workaround (see #17633) has a significant limitation in that it only work at query root: custom relations inside JOINs, CTEs, or subqueries aren't transformed.

Describe the solution you'd like

A RelationPlanner trait that intercepts table factor planning at any nesting level:

pub trait RelationPlanner: Debug + Send + Sync {
    fn plan_relation(
        &self,
        relation: TableFactor,
        context: &mut dyn RelationPlannerContext,
    ) -> Result<RelationPlanning>;
}

Key features:

  • Works for nested relations (JOINs, CTEs, subqueries)
  • Chain multiple planners with SessionContext::register_relation_planner()
  • Access SQL utilities via RelationPlannerContext (expression conversion, identifier normalization)
  • Return Planned(plan) to handle, or Original(relation) to pass through

Example usage:

ctx.register_relation_planner(Arc::new(MatchRecognizePlanner))?;
ctx.sql("SELECT * FROM stocks MATCH_RECOGNIZE(...);").await?;

Describe alternatives you've considered

Additional context

Related to:

This complements existing extensibility APIs:

  • ExprPlanner: custom operators/expressions
  • TypePlanner: custom data types
  • RelationPlanner (new): custom table factors

Activity

  1. geoffreyclaude commented on Oct 15, 2025

    @geoffreyclaude
    ContributorAuthor

    take

  2. alamb commented on Oct 15, 2025

    @alamb
    Contributor

    You may also be able to use the TableFunction API:
    https://docs.rs/datafusion/latest/datafusion/catalog/trait.TableFunctionImpl.html
    https://docs.rs/datafusion/latest/datafusion/catalog/struct.TableFunction.html

    It may not be the same, but maybe related (I don't think it is documented super well, FWIW)

  3. geoffreyclaude commented on Oct 16, 2025

    @geoffreyclaude
    ContributorAuthor

    You may also be able to use the TableFunction API:
    https://docs.rs/datafusion/latest/datafusion/catalog/trait.TableFunctionImpl.html
    https://docs.rs/datafusion/latest/datafusion/catalog/struct.TableFunction.html

    It may not be the same, but maybe related (I don't think it is documented super well, FWIW)

    Indeed, it's obviously a better solution for implementing table functions. It doesn't help other constructs though, such as PIVOT, MATCH_RECOGNIZE, or TABLESAMPLE. I'll update the issue to be clearer about this.

    I could even also see a valid argument to move existing complex functionalities out of the core DataFusion and into a datafusion-extended-sql package or extension of some sort (UNNEST or table functions for instance,) but that's out of scope of this particular issue!

  4. added a commit that references this issue on Dec 9, 2025
    a30cf37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

enhancementNew feature or request

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions