Skip to content

Support computing ScalarValue return types from argument values (e.g. constants) rather than just types #8624

Closed

Description

@alamb

Def one UDF:

my_udf(expr, 'return_type')

Using:

case1:

my_duf(col, 'UInt32')

Return Type is:
DataType::UInt32

case2:

my_duf(col, 'UInt64')

Return Type is:
DataType::UInt64

At present, the above udf cannot be realized through the ReturnTypeFunction. This is because the ReturnTypeFunction provides only the data types of the input parameters.

As we discussed on #7657 (comment) arrow_cast does this but it is handled specially. This ticket tracks adding UDF support for it

Originally posted by @JasonLi-cn in #7657

Activity

  1. alamb commented on Jan 1, 2024

    @alamb
    ContributorAuthor

    I think we can probably work on this now that #8568 is merged

  2. yyy1000 commented on Jan 24, 2024

    @yyy1000
    Contributor

    Since my first PR is close to be merged. I'd like to work on this as a next. 😀

  3. alamb commented on Jan 24, 2024

    @alamb
    ContributorAuthor

    Thanks @yyy1000 -- FYI aware this is likely to be an exercise in careful API design

    Perhaps you could look into extending ScalarFunctionImpl https://github.com/apache/arrow-datafusion/blob/bc0ba6a724aaaf312b770451718cbce696de6640/datafusion/expr/src/udf.rs#L237 in backwards compatible way

    For example:

    trait ScalarUDFImpl { 
    ...
    
        /// What [`DataType`] will be returned by this function, given the types of
        /// the expr arguments
        fn return_type_from_exprs(&self, arg_exprs: &[Expr], schema: &SchemaRef) -> Result<DataType> {
         // provide default implementation that calls `self.return_type()`
         // so that people don't have to implement `return_type_from_exprs` if they dont want to
        }
     ...
    }

    An exellent test of the new API would be to see if you can implement arrow_cast as a ScalarUDF we can put it into datafusion-functions after #8705 is merged

  4. yyy1000 commented on Jan 24, 2024

    @yyy1000
    Contributor

    I created a PR in #8985 that wants to fix this. ❤️

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions