Repository navigation
Closed
Labels
bugSomething isn't workingSomething isn't working
Description
Describe the bug
The array_slice UDF takes 4 parameters.
datafusion/datafusion/functions-array/src/extract.rs
Lines 55 to 61 in 96487ea
| make_udf_function!( | |
| ArraySlice, | |
| array_slice, | |
| array begin end stride, | |
| "returns a slice of the array.", | |
| array_slice_udf | |
| ); |
Which means that args.len() is always 4 in array_slice_inner, even when called with stride = Expr::Null or stride = Expr::Int64(None)
datafusion/datafusion/functions-array/src/extract.rs
Lines 289 to 293 in 96487ea
| let stride = if args_len == 4 { | |
| Some(as_int64_array(&args[3])?) | |
| } else { | |
| None | |
| }; |
To Reproduce
I encountered this while creating the python wrapper for datafusion-python and had to set stride=1 else the function would panic.
Expected behavior
The stride behavior in the UDF should match what's available in the CLI
CREATE TEMPORARY VIEW data3 AS VALUES ([1.0, 2.0, 3.0, 3.0]), ([4.0, 5.0, 3.0]), ([6.0]);
❯ select * from data3;
+----------------------+
| column1 |
+----------------------+
| [1.0, 2.0, 3.0, 3.0] |
| [4.0, 5.0, 3.0] |
| [6.0] |
+----------------------+
> select * from array_slice(data3.column1, -1, 2)
+-----------------------------------------------+
| array_slice(data3.column1,Int64(-1),Int64(2)) |
+-----------------------------------------------+
| [] |
| [] |
| [6.0] |
+-----------------------------------------------+
Additional context
No response
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working
Activity
alamb commentedon May 8, 2024
Thanks @Michael-J-Ward -- what is your ideal outcome? That the UDF function can take three arguments (and default the fourth value to a constant 1)?
alamb commentedon May 8, 2024
Looks like these were last modified in #9788 / #9615 from @jayzhan211
and @erenavsarogullari
https://github.com/apache/datafusion/blame/0d283a49f52d91a58c4e88e3e5e76fe0d59c1260/datafusion/functions-array/src/extract.rs#L314
Michael-J-Ward commentedon May 8, 2024
I would expect that there exists some value for
stride = Expr(likelyExpr::NullorExpr::Int64(None)) such that the behavior of calling the UDF w/ that value matches the behavior of calling the SQL api without itRoughly:
That way
datafusion-pythoncan mapstrideto that value when a user callsf.array_slice(col, literal(2), literal(4), stride=None)EDIT:
We would think
stride = 1should be that.The frustration encountered with #10425 is that we were unable to trigger the code-path that the
SQL APIuses from the UDF.jayzhan211 commentedon May 9, 2024
I agree it is helpful if we support
expr_fn::array_slice(array, begin, end).We can't only provide three args for it now
But I can't think of any good solution to this
What you are expecting is something not supported in rust -- variadic function
I think the issue here should be handled in
datafusion-python@Michael-J-Ward Did you successfully create expression without stride=1 before 38? It would be surprising if it works previously.
jonahgao commentedon May 9, 2024
Is it possible to make the
array_slicefunction accept a vector as its parameter?When making its UDF function, we need to do
alamb commentedon May 9, 2024
Another potential option is to add a second expr fn with a different signature
jayzhan211 commentedon May 10, 2024
The downside of this is we loss the clear argument name and we need to check args length.
I prefer this
example fix to make `stride` optional in `array_slice` UDF
Michael-J-Ward commentedon May 10, 2024
First, I'd like to emphasize that if passing
stride=1to the UDF worked the same as not providing a stride in the SQL api, then this would be just porcelain / a nicety. Reposting the example from: #10425Now to this issue, I agree that I like the explicit arguments versus passing a
Vec, and I would hope the solution doesn't require keeping two separate UDFs.I created a draft with what I would think is the natural rust solution in #10450. I don't know if this violates some datafusion constraint, but all the tests pass.
If that is not possible, then a second proposal would be to amend
array_slice_innerw/ somenullcheck.But again, this is just a nice-to-have if passing
stride=1worked the same as not providing it in the SQL API.Michael-J-Ward commentedon May 10, 2024
I believe we encountered the same issue again with
regex_replace, so I suspect this issue isn't constrained toarray_sliceand instead applies to anyUDFthat previously had an optional argument.Omega359 commentedon May 10, 2024
I recall hitting something like this with the substr/substring function. One would think they would be identical however they were not (since rust doesn't do variadic functions nor does it allow defaults for args ala scala or Kotlin). For the substr it was expanded to two functions that internally used a vec<> and handled the logic of using the optional 3rd argument there.
andygrove commentedon May 10, 2024
I am +1 for this solution suggested by @Michael-J-Ward
jayzhan211 commentedon May 11, 2024
I think there are two issues here, one is the panic issue in
datafusion-cliand another one is the API design forExpr.array_slicepanicThe reason for panic in the second command is due to the bug in
array_slicedatafusion/datafusion/functions-array/src/extract.rs
Lines 420 to 425 in 933b430
We should fix this.
Another issue is that given the current API constraint of
Expr, we can not provide only 3 arguments.There are many different approaches like giving
vec, or introducing another function or your proposalI think this is more a user experience problem, how should we design it is discussable.
Given the design here, we still need to provide 4 args, where you provide
Nonetostrideif you expect stride with 1.TLDR:
The panic issue should be fixed not by redesigning the Expr API, but by fixing the internal array slice code.
Michael-J-Ward commentedon May 11, 2024
Agreed @jayzhan211, these are two separate issues.
The panic issue was filed separately as #10425
andygrove commentedon May 11, 2024
It is more than a user experience issue. The current API is causing test failures in a downstream project (the DataFusion Python bindings) and this cannot be resolved without the API changes being discussed in this issue.
Michael-J-Ward commentedon May 11, 2024
Since the
regexp_*UDFs have the same problem, I suspect thatarray_slicewas just our first encounter w/ the underlying issue: any "inner" UDF implementation that behaves differently based on the number of arguments passed should expose that via its public API.Solutions discussed:
Expose multiple public apis for each variant, as @Omega359 pointed out that inner
substrexposes publicsubstrandsubstring.Change the public APIs to take
Option<Expr>, as I demoed forarray_slicein DRAFT: example fix to makestrideoptional inarray_sliceUDF #10450.Having the public API just take a
Vec<Expr>jayzhan211 commentedon May 11, 2024
I believe the failure occurred because of the bug I mentioned. Can you provide another example where df-python still experiences a panic?
If it is because of the
Exprdesign.Given these 3, I would prefer the second one.
jonahgao commentedon May 12, 2024
I submitted a formal PR #10469 to address this.