Skip to content

Simplify Spark sha2 implementation - #19475

Merged
Jefffrey merged 4 commits into
apache:mainfrom
Jefffrey:refactor-spark-sha2
Jan 28, 2026
Merged

Jefffrey merged 4 commits into
apache:mainfrom
Jefffrey:refactor-spark-sha2

Conversation

@Jefffrey

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

Simplify implementation, also remove usage of user_defined.

What changes are included in this PR?

Refactor to be simpler.

Are these changes tested?

Added tests.

Are there any user-facing changes?

No.

@github-actions github-actions Bot added sqllogictest SQL Logic Tests (.slt) functions Changes to functions implementation spark Changes to datafusion-spark crate labels Dec 24, 2025
Comment on lines -49 to +63
signature: Signature::user_defined(Volatility::Immutable),
aliases: vec![],
signature: Signature::coercible(
vec![
Coercion::new_implicit(
TypeSignatureClass::Native(logical_binary()),
vec![TypeSignatureClass::Native(logical_string())],
NativeType::Binary,
),
Coercion::new_implicit(
TypeSignatureClass::Native(logical_int32()),
vec![TypeSignatureClass::Integer],
NativeType::Int32,

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moving away from user_defined; also we cast strings to binary to simplify implementation as we only need raw bytes either way

Comment on lines +121 to +134
let mut digest = sha2::Sha224::default();
digest.update(value);
Some(hex_encode(digest.finalize()))
}
(Some(value), Some(0 | 256)) => {
let mut digest = sha2::Sha256::default();
digest.update(value);
Some(hex_encode(digest.finalize()))
}
(Some(value), Some(384)) => {
let mut digest = sha2::Sha384::default();
digest.update(value);
Some(hex_encode(digest.finalize()))
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We directly use the sha2 crate to do the hashing now; previously we used functions exposed by datafusion-functions but that seemed like unnecessary indirection

Comment thread Cargo.toml
regex = "1.12"
rstest = "0.26.1"
serde_json = "1"
sha2 = "^0.10.9"

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because datafusion-spark now uses sha2 directly, we extract it as a common dependency

@Jefffrey
Jefffrey marked this pull request as ready for review December 24, 2025 08:16
@getChan

getChan commented Dec 29, 2025

Copy link
Copy Markdown
Contributor

@Jefffrey thank you.
Have you tried running any benchmarks? I’m curious whether there’s any performance improvement when calling it from Spark.
ref. apache/datafusion-comet#2986

@Jefffrey

Copy link
Copy Markdown
Contributor Author

@Jefffrey thank you. Have you tried running any benchmarks? I’m curious whether there’s any performance improvement when calling it from Spark. ref. apache/datafusion-comet#2986

Performance wasn't on my mind when I did this refactor; I was more focused on trying to get the code in a more readable state. Adding fast paths for scalars should be trivial from this point on, but I'm not sure if I want to add more to this PR as is. Happy to do so if preferred.

@alamb alamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense to me -- thank you @Jefffrey and @getChan

@Jefffrey
Jefffrey added this pull request to the merge queue Jan 28, 2026
Merged via the queue into apache:main with commit 568f19f Jan 28, 2026
31 checks passed
@Jefffrey
Jefffrey deleted the refactor-spark-sha2 branch January 28, 2026 11:51
de-bgunter pushed a commit to de-bgunter/datafusion that referenced this pull request Mar 24, 2026
## Which issue does this PR close?

<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes apache#123` indicates that this PR will close issue apache#123.
-->

- Part of apache#12725

## Rationale for this change

<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->

Simplify implementation, also remove usage of user_defined.

## What changes are included in this PR?

<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->

Refactor to be simpler.

## Are these changes tested?

<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code

If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->

Added tests.

## Are there any user-facing changes?

<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->

No.

<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

functions Changes to functions implementation spark Changes to datafusion-spark crate sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants