Repository navigation
[datafusion-spark]: Refactor hex's signature away from user_defined - #19235
Conversation
| ); | ||
| let binary = Coercion::new_implicit( | ||
| TypeSignatureClass::Native(logical_binary()), | ||
| vec![TypeSignatureClass::Binary], |
There was a problem hiding this comment.
This does not support LargeBinary type.
The old version supported it - https://github.com/apache/datafusion/pull/19235/files#diff-6d740fb90f0b078d335d596dcb8211ae4a32a6df40fcc7aa60ee779b3f800bc6L106
There was a problem hiding this comment.
This seems a bug if logical_binary() doesn't cover LargeBinary; can we fix this at the root cause instead of having a workaround?
There was a problem hiding this comment.
im trying to look at where that may be but cant seem to pinpoint beyond https://github.com/apache/datafusion/blob/main/datafusion/common/src/types/builtin.rs#L77
There was a problem hiding this comment.
My comment was about vec![TypeSignatureClass::Binary],, not logical_binary().
datafusion/datafusion/expr-common/src/signature.rs
Lines 340 to 341 in 2a08013
The comment here mentions (the native) Binary and FixedSizeBinary but nothing about LargeBinary.
There was a problem hiding this comment.
I'm fairly sure it encompasses LargeBinary too, but if we had a test to confirm this it would be nice
There was a problem hiding this comment.
CREATE TABLE t_largebinary AS VALUES
(arrow_cast(arrow_cast('hello', 'Binary'), 'LargeBinary')),
(NULL::LargeBinary),
(arrow_cast(arrow_cast('world', 'Binary'), 'LargeBinary'));CREATE TABLE t_largebinary AS VALUES
(arrow_cast(arrow_cast('hello', 'Binary'), 'LargeBinary')),
(NULL::LargeBinary),
(arrow_cast(arrow_cast('world', 'Binary'), 'LargeBinary'));CREATE TABLE t_largebinary AS VALUES
(arrow_cast(arrow_cast('hello', 'Binary'), 'LargeBinary')),
(NULL::LargeBinary),
(arrow_cast(arrow_cast('world', 'Binary'), 'LargeBinary'));CREATE TABLE t_largebinary AS VALUES
(arrow_cast(arrow_cast('hello', 'Binary'), 'LargeBinary')),
(NULL::LargeBinary),
(arrow_cast(arrow_cast('world', 'Binary'), 'LargeBinary'));
This feature is not implemented: Unsupported SQL type LargeBinary
This feature is not implemented: Unsupported SQL type LargeBinary
This feature is not implemented: Unsupported SQL type LargeBinary
This feature is not implemented: Unsupported SQL type LargeBinary
Getting the above in datafusion cli
There was a problem hiding this comment.
> CREATE TABLE binary_data (id INTEGER, data BYTEA);
0 row(s) fetched.
Elapsed 0.002 seconds.
> INSERT INTO binary_data (id, data) VALUES (1, X'48656C6C6F');
+-------+
| count |
+-------+
| 1 |
+-------+
1 row(s) fetched.
Elapsed 0.017 seconds.
> select * from binary_data;
+----+------------+
| id | data |
+----+------------+
| 1 | 48656c6c6f |
+----+------------+
1 row(s) fetched.
Elapsed 0.003 seconds.
> select arrow_cast(data, 'LargeBinary') from binary_data;
+--------------------------------------------------+
| arrow_cast(binary_data.data,Utf8("LargeBinary")) |
+--------------------------------------------------+
| 48656c6c6f |
+--------------------------------------------------+
1 row(s) fetched.
Elapsed 0.001 seconds.
There was a problem hiding this comment.
my idea was to use explain to determine if LargeBinary comes under native binary using a query in which I cast like - SELECT hex(arrow_cast(arrow_cast('test', 'Binary'), 'LargeBinary')); which did not give anything conclusive.
so I tested out an slt with SELECT hex(arrow_cast('test', 'LargeBinary')) as lar_b; after commenting out
TypeSignature::Exact(vec![DataType::LargeBinary]), ---- which passed without any failure.
to conclude - we can now be sure that LargeBinary is encompassed under the native Binary right?
lmk your thoughts @martin-g @Jefffrey
There was a problem hiding this comment.
OK! Then the comment at
datafusion/datafusion/expr-common/src/signature.rs
Lines 340 to 341 in 2a08013
Please add the SLT test to the suite!
There was a problem hiding this comment.
Actually this signature can just be:
Coercion::new_exact(TypeSignatureClass::Binary);It seems the implementation can already handle FixedSizeBinary's so theres no need to cast those to Binary/LargeBinary type 🤔
| ); | ||
| let binary = Coercion::new_implicit( | ||
| TypeSignatureClass::Native(logical_binary()), | ||
| vec![TypeSignatureClass::Binary], |
There was a problem hiding this comment.
This seems a bug if logical_binary() doesn't cover LargeBinary; can we fix this at the root cause instead of having a workaround?
| Decimal, | ||
| Numeric, | ||
| /// Encompasses both the native Binary as well as arbitrarily sized FixedSizeBinary types | ||
| /// Encompasses Binary, FixedSizeBinary, and LargeBinary types |
There was a problem hiding this comment.
| /// Encompasses Binary, FixedSizeBinary, and LargeBinary types | |
| /// Encompasses both the native Binary/LargeBinary types as well as arbitrarily sized FixedSizeBinary types |
| query T | ||
| SELECT hex(arrow_cast(arrow_cast('!hallucinations!', 'Binary'), 'LargeBinary')); | ||
| ---- | ||
| 2168616C6C7563696E6174696F6E7321 |
There was a problem hiding this comment.
Can remove this as seems redundant with the LargeBinary test at the bottom 🤔
| statement ok | ||
| CREATE TABLE t_largebinary as VALUES (arrow_cast(arrow_cast('hello', 'Binary'), 'LargeBinary')), (NULL), (arrow_cast(arrow_cast('world', 'Binary'), 'LargeBinary')); | ||
|
|
||
| query T | ||
| SELECT hex(column1) FROM t_largebinary; | ||
| ---- | ||
| 68656C6C6F | ||
| NULL | ||
| 776F726C64 |
There was a problem hiding this comment.
| statement ok | |
| CREATE TABLE t_largebinary as VALUES (arrow_cast(arrow_cast('hello', 'Binary'), 'LargeBinary')), (NULL), (arrow_cast(arrow_cast('world', 'Binary'), 'LargeBinary')); | |
| query T | |
| SELECT hex(column1) FROM t_largebinary; | |
| ---- | |
| 68656C6C6F | |
| NULL | |
| 776F726C64 | |
| query T | |
| SELECT hex(column1) FROM VALUES (arrow_cast('hello', 'LargeBinary')), (NULL), (arrow_cast('world', 'LargeBinary')); | |
| ---- | |
| 68656C6C6F | |
| NULL | |
| 776F726C64 |
So we don't need to worry about cleaning up the table at the end. Also avoid need for cast from string -> binary -> largebinary, just convert directly to largebinary
|
Thanks @codetyri0n & @martin-g |
Which issue does this PR close?
Signature::user_definedfor normal functions #12725Rationale for this change
moving
hexaway from a user defined signature as it is out of scope and added a UT for checking correct number of argsAre these changes tested?
existing tests
Are there any user-facing changes?
yes