Skip to content

[datafusion-spark]: Refactor hex's signature away from user_defined - #19235

Merged
Jefffrey merged 5 commits into
apache:mainfrom
codetyri0n:hex
Dec 12, 2025
Merged

Jefffrey merged 5 commits into
apache:mainfrom
codetyri0n:hex

Conversation

@codetyri0n

@codetyri0n codetyri0n commented Dec 9, 2025 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

moving hex away from a user defined signature as it is out of scope and added a UT for checking correct number of args

Are these changes tested?

existing tests

Are there any user-facing changes?

yes

@github-actions github-actions Bot added the spark Changes to datafusion-spark crate label Dec 9, 2025
);
let binary = Coercion::new_implicit(
TypeSignatureClass::Native(logical_binary()),
vec![TypeSignatureClass::Binary],

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems a bug if logical_binary() doesn't cover LargeBinary; can we fix this at the root cause instead of having a workaround?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

im trying to look at where that may be but cant seem to pinpoint beyond https://github.com/apache/datafusion/blob/main/datafusion/common/src/types/builtin.rs#L77

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My comment was about vec![TypeSignatureClass::Binary],, not logical_binary().

/// Encompasses both the native Binary as well as arbitrarily sized FixedSizeBinary types
Binary,

The comment here mentions (the native) Binary and FixedSizeBinary but nothing about LargeBinary.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm fairly sure it encompasses LargeBinary too, but if we had a test to confirm this it would be nice

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CREATE TABLE t_largebinary AS VALUES
(arrow_cast(arrow_cast('hello', 'Binary'), 'LargeBinary')),
(NULL::LargeBinary),
(arrow_cast(arrow_cast('world', 'Binary'), 'LargeBinary'));CREATE TABLE t_largebinary AS VALUES
(arrow_cast(arrow_cast('hello', 'Binary'), 'LargeBinary')),
(NULL::LargeBinary),
(arrow_cast(arrow_cast('world', 'Binary'), 'LargeBinary'));CREATE TABLE t_largebinary AS VALUES
(arrow_cast(arrow_cast('hello', 'Binary'), 'LargeBinary')),
(NULL::LargeBinary),
(arrow_cast(arrow_cast('world', 'Binary'), 'LargeBinary'));CREATE TABLE t_largebinary AS VALUES
(arrow_cast(arrow_cast('hello', 'Binary'), 'LargeBinary')),
(NULL::LargeBinary),
(arrow_cast(arrow_cast('world', 'Binary'), 'LargeBinary'));
This feature is not implemented: Unsupported SQL type LargeBinary
This feature is not implemented: Unsupported SQL type LargeBinary
This feature is not implemented: Unsupported SQL type LargeBinary
This feature is not implemented: Unsupported SQL type LargeBinary

Getting the above in datafusion cli

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

> CREATE TABLE binary_data (id INTEGER, data BYTEA);
0 row(s) fetched.
Elapsed 0.002 seconds.

> INSERT INTO binary_data (id, data) VALUES (1, X'48656C6C6F');
+-------+
| count |
+-------+
| 1     |
+-------+
1 row(s) fetched.
Elapsed 0.017 seconds.

> select * from binary_data;
+----+------------+
| id | data       |
+----+------------+
| 1  | 48656c6c6f |
+----+------------+
1 row(s) fetched.
Elapsed 0.003 seconds.

> select arrow_cast(data, 'LargeBinary') from binary_data;
+--------------------------------------------------+
| arrow_cast(binary_data.data,Utf8("LargeBinary")) |
+--------------------------------------------------+
| 48656c6c6f                                       |
+--------------------------------------------------+
1 row(s) fetched.
Elapsed 0.001 seconds.

@codetyri0n codetyri0n Dec 11, 2025 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

my idea was to use explain to determine if LargeBinary comes under native binary using a query in which I cast like - SELECT hex(arrow_cast(arrow_cast('test', 'Binary'), 'LargeBinary')); which did not give anything conclusive.
so I tested out an slt with SELECT hex(arrow_cast('test', 'LargeBinary')) as lar_b; after commenting out
TypeSignature::Exact(vec![DataType::LargeBinary]), ---- which passed without any failure.

to conclude - we can now be sure that LargeBinary is encompassed under the native Binary right?
lmk your thoughts @martin-g @Jefffrey

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK! Then the comment at

/// Encompasses both the native Binary as well as arbitrarily sized FixedSizeBinary types
Binary,
could be updated to mention LargeBinary too!
Please add the SLT test to the suite!

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually this signature can just be:

Coercion::new_exact(TypeSignatureClass::Binary);

It seems the implementation can already handle FixedSizeBinary's so theres no need to cast those to Binary/LargeBinary type 🤔

);
let binary = Coercion::new_implicit(
TypeSignatureClass::Native(logical_binary()),
vec![TypeSignatureClass::Binary],

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems a bug if logical_binary() doesn't cover LargeBinary; can we fix this at the root cause instead of having a workaround?

@github-actions github-actions Bot added logical-expr Logical plan and expressions sqllogictest SQL Logic Tests (.slt) labels Dec 11, 2025
Comment thread datafusion/expr-common/src/signature.rs Outdated
Decimal,
Numeric,
/// Encompasses both the native Binary as well as arbitrarily sized FixedSizeBinary types
/// Encompasses Binary, FixedSizeBinary, and LargeBinary types

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
/// Encompasses Binary, FixedSizeBinary, and LargeBinary types
/// Encompasses both the native Binary/LargeBinary types as well as arbitrarily sized FixedSizeBinary types

Comment on lines +62 to +65
query T
SELECT hex(arrow_cast(arrow_cast('!hallucinations!', 'Binary'), 'LargeBinary'));
----
2168616C6C7563696E6174696F6E7321

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can remove this as seems redundant with the LargeBinary test at the bottom 🤔

Comment on lines +52 to +60
statement ok
CREATE TABLE t_largebinary as VALUES (arrow_cast(arrow_cast('hello', 'Binary'), 'LargeBinary')), (NULL), (arrow_cast(arrow_cast('world', 'Binary'), 'LargeBinary'));

query T
SELECT hex(column1) FROM t_largebinary;
----
68656C6C6F
NULL
776F726C64

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
statement ok
CREATE TABLE t_largebinary as VALUES (arrow_cast(arrow_cast('hello', 'Binary'), 'LargeBinary')), (NULL), (arrow_cast(arrow_cast('world', 'Binary'), 'LargeBinary'));
query T
SELECT hex(column1) FROM t_largebinary;
----
68656C6C6F
NULL
776F726C64
query T
SELECT hex(column1) FROM VALUES (arrow_cast('hello', 'LargeBinary')), (NULL), (arrow_cast('world', 'LargeBinary'));
----
68656C6C6F
NULL
776F726C64

So we don't need to worry about cleaning up the table at the end. Also avoid need for cast from string -> binary -> largebinary, just convert directly to largebinary

@Jefffrey
Jefffrey added this pull request to the merge queue Dec 12, 2025
Merged via the queue into apache:main with commit 10c933c Dec 12, 2025
72 of 75 checks passed
@Jefffrey

Copy link
Copy Markdown
Contributor

Thanks @codetyri0n & @martin-g

@codetyri0n
codetyri0n deleted the hex branch December 12, 2025 08:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

logical-expr Logical plan and expressions spark Changes to datafusion-spark crate sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants