Skip to content

Fix inconsistencies in type coercion logic #3419

Open

Description

@andygrove

Describe the bug
The TypeCoercion rule incorrectly uses coerce_types (which should probably be renamed to get_binary_op_type because it determines the output type of a binary expression) to determine the types to case the lhs and rhs expressions.

I think we need a new and similar method for determining the common type to case inputs to, for binary expressions.

Maybe we need to introduce signatures for binary ops so this can be consolidated more, similar to how we handle UDF signatrures.

To Reproduce
Steps to reproduce the behavior:

Expected behavior
A clear and concise description of what you expected to happen.

Additional context
Add any other context about the problem here.

Activity

  1. self-assigned this
    on Sep 9, 2022
  2. liukun4515 commented on Sep 9, 2022

    @liukun4515
    Contributor

    Do we need make align the type coercion with arrow?
    The cast system should not be same with arrow, because the datafusion is the SQL system and has some special semantics than the arrow lib
    @andygrove

  3. andygrove commented on Sep 9, 2022

    @andygrove
    MemberAuthor

    I agree that there will be differences. See the conversation at #3407 (comment) for one example where we have to work around a difference and this example is possibly due to a bug in DataFusions logic

  4. changed the title [-]DataFusion coercion logic is different from Arrow's[/-] [+]Fix inconsistencies in type coercion logic[/+] on Sep 9, 2022
  5. andygrove commented on Sep 9, 2022

    @andygrove
    MemberAuthor

    I updated the title and description now that I understand this more.

  6. alamb commented on Oct 3, 2022

    @alamb
    Contributor

    Maybe we could return a struct like

    enum NeededCoercion {
      /// cast inputs to these datatypes
      coerce_inputs: Vec<Datatype>,
      /// the resulting output will be this datatype
      output_type: DataType
    }

    Or something 🤔

  7. removed their assignment
    on Nov 1, 2022
  8. HaoYang670 commented on Nov 10, 2022

    @HaoYang670
    Contributor

    Hi, could someone help to explain what the coerce_types does. Does it coerce the lhs and rhs types, or calculate the return type of the operator?

  9. alamb commented on Nov 10, 2022

    @alamb
    Contributor

    Hi, could someone help to explain what the coerce_types does. Does it coerce the lhs and rhs types, or calculate the return type of the operator?

    I think this is a reasonable description of coercion:

    https://github.com/apache/arrow-datafusion/blob/9c24a79118c0ae0aee480bf039385a06d240b499/datafusion/expr/src/type_coercion.rs#L18-L32

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingoptimizerOptimizer rules

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions