Repository navigation
Description
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
During I try to implement Select (subquery) , I add Select(Box<LogicalPlan>) in Expr. Because Expr derives PartialEq, PartialOrd, but LogicalPlan not, I try to derive PartialEq, PartialOrd for LogicalPlan. Unfortunately, I got an error, because some logical plan contains traits, such as TableScan, which contains the TableProvider trait.
more thoughts: #1228 (comment)
Describe the solution you'd like
In order to achieve better scalability, I think we can extract logical plans in LogicalPlan as independent structs. For example
pub enum LogicalPlan {
...
TableScan {
/// The name of the table
table_name: String,
/// The source of the table
source: Arc<dyn TableProvider>,
/// Optional column indices to use as a projection
projection: Option<Vec<usize>>,
/// The schema description of the output
projected_schema: DFSchemaRef,
/// Optional expressions to be used as filters by the table provider
filters: Vec<Expr>,
/// Optional limit to skip reading
limit: Option<usize>,
},
...
}pub struct TableScanPlan {
/// The name of the table
table_name: String,
/// The source of the table
source: Arc<dyn TableProvider>,
/// Optional column indices to use as a projection
projection: Option<Vec<usize>>,
/// The schema description of the output
projected_schema: DFSchemaRef,
/// Optional expressions to be used as filters by the table provider
filters: Vec<Expr>,
/// Optional limit to skip reading
limit: Option<usize>,
}
pub enum LogicalPlan {
...
TableScan(TableScanPlan),
...
}Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
What changes are planned in further issues/PRs?
- Extract
Projection,Filter,Windowin LogicalPlan as independent struct #1302 - Extract
Aggregate,Sort,Joinin LogicalPlan as independent struct #1303 - Extract
CrossJoin,Repartition,Unionin LogicalPlan as independent struct #1304 - Extract
EmptyRelation,Limit,Valuesin LogicalPlan as independent struct #1305 - Extract
CreateMemoryTable,DropTable,CreateExternalTablein LogicalPlan as independent struct #1306 - Extract
Explain,Analyze,Extensionin LogicalPlan as independent struct #1307
Activity
xudong963 commentedon Nov 3, 2021
Let me know your thoughts ❤️ @houqp @alamb @jimexist @Dandandan, if it's ok, I'll do it after work.
houqp commentedon Nov 3, 2021
@alamb if I recall correctly, the partialOrd for Expr as added to help with unit testing right?
alamb commentedon Nov 3, 2021
Yes, it was added in #1015
The test ordering I was the main usecase but I think there are some other ideas of being able to have Exprs in
BTreeSetsand other ordering based containers.I think extracting
LogicalPlaninto structs would be fine, though it will be a very non trivial change (both for DataFusion and all users)Something that is more of a hack but might be quicker to do would be to use a
NewTypekind of thing:And then implement
PartialOrddirectly on this structure (perhaps by using the point value of the Box directly).So then we would have
Expr::Select(Subquery)or somethinghouqp commentedon Nov 4, 2021
On that topic, how about pushing the
PartialOrdandPartialEqtrait to the trait objects? i.e.LogicalPlan::Extension.nodeandLogicalPlan::TableScan.source.xudong963 commentedon Nov 4, 2021
Good idea!
Long-term thinking, maybe we can profit from extracting
LogicalPlaninto structs. So if all of you agree, I can do it before doing subquery features. cc @houqpalamb commentedon Nov 4, 2021
I like @houqp's idea of marking adding
PartialOrdandPartialEqto traits to allowLogicalPlanto derive them as wellPerhaps something like
(and similarly for any other trait objects in
LogicalPlan)And then
xudong963 commentedon Nov 10, 2021
Because
PartialEqandPartialOrdtraits are not "object safe", so pushing the PartialOrd and PartialEq trait to the trait such asTableProviderwill makeTableProvider"object unsafe".houqp commentedon Nov 11, 2021
Good point @xudong963 , totally forgot about object safety :(
I don't have a strong opinion on this and I am cool with us going either way. I do agree with you that adding the
TableScanPlanstruct makes things more flexible in the long run.xudong963 commentedon Nov 11, 2021
@alamb @Dandandan @jimexist How about your thoughts? If you also agree, I want to try on this weekend and after finishing this, I'll continue to deep into subqueries.
alamb commentedon Nov 11, 2021
I think it is a reasonable thing to try. I suspect we'll have to see what the final code looks like to really know 🤷
xudong963 commentedon Nov 12, 2021
OK. I'll firstly extract
TableScan, then you can what the code looks like, lol 😄liukun4515 commentedon Nov 13, 2021
hi @xudong963 , please at me, if you finish this task and want someone to review it.
Maybe i can begin this task, after your refactoring work.
xudong963 commentedon Nov 15, 2021
I split six sub-issues, newcomers can join us by participating in these issues.
rdettai commentedon Nov 15, 2021
Thanks for your work @xudong963 ! Sorry for getting into the conversation so late but I was in vacations these past two weeks 😉
A few remarks and questions:
PartialEq(andPartialOrd) here? Why don't we just add anfn eq(&self, other: & dyn TableProvider) -> boolmethod onTableProviderand a then implement manually thePartialEqtrait on the logical plan with amatchstatement? The method will surely be a bit verbose, but it would avoid refactoring the entire codebase.PartialEqusing#[derive()], why do we need to extract all variants ofLogicalPlaninto structs? isn't it enough to do it for the variants with trait object fields?I might have missed some parts of the discussion, so please excuse me if my questions are a bit off 😄
xudong963 commentedon Nov 15, 2021
Hi @rdettai, long time no see. Thanks for your feedback.
Now, this is not just a question of subqueries, but a question of which design is better.
During my implementation of subqueries, I encountered some obstacles due to the current pattern.
So I started to think about the disadvantages of this pattern.
First of all, I think the following pattern is more common in Rust, such as https://github.com/rust-lang/rust/blob/ad808d95c4839caedc2be76d0ed059dc920ab4b6/src/librustc/hir/mod.rs#L2752
In addition to the above, it has some other benefits:
It's more clearer and flexiable, long-term thinking, maybe we can profit from extracting LogicalPlan into structs.
Enum variants are not types, so
LogicalPlan::TableScancan't be a function parameter or stored, you need to passLogical Plan. But if each enum variant has its own type,TableScancan be passed and stored.When a match occurs, the following are the match styles corresponding to the above two writing methods
If multiple enums have the same variants, we need to repeat to write the variant with all its fields when using the first pattern.
Sorry, I don't link all the subtasks to an umbrella issue, I'll do it now.
above all, I think this is a meaningful and valuable refactoring, despite major changes to the codebase.
cc @alamb @houqp and welcome everyone to discuss!
houqp commentedon Nov 16, 2021
Thanks @xudong963 for the detailed write up.
This to me is my favorite benefit 👍 This effectively means if a series of function calls needs access to a particular plan node, we only need to match on LogicalPlan once, resulting in more readable and performant code.
rdettai commentedon Nov 16, 2021
Thanks for taking the time to answer my set of questions! Makes sense to me now 🙂.
alamb commentedon Nov 16, 2021
Yes, I think this is the primary benefit of pulling LogicalPlan nodes into their own structs (not that it is required for subqueries -- I think we could probably make that work somehow -- but that this will make the codebase cleaner in the longer term