Skip to content

refactor: Refactor spark width bucket signature away from user defined - #19065

Merged
Jefffrey merged 5 commits into
apache:mainfrom
kumarUjjawal:refactor/spark_width_bucket
Dec 9, 2025
Merged

Jefffrey merged 5 commits into
apache:mainfrom
kumarUjjawal:refactor/spark_width_bucket

Conversation

@kumarUjjawal

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

  • As per the goal stated that we should avoid using the user_defined in useful places

What changes are included in this PR?

  • Refactor the user_defined in spark width_bucket

Are these changes tested?

Are there any user-facing changes?

@github-actions github-actions Bot added common Related to common crate spark Changes to datafusion-spark crate labels Dec 3, 2025

@Jefffrey Jefffrey left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for picking this up, left some suggestions

Comment on lines +80 to +135
make_scalar_function(width_bucket_kern, vec![])(&args.args)
let [value, minv, maxv, buckets] = take_function_args(self.name(), &args.args)?;

let arrays = vec![
value.to_array(args.number_rows)?,
minv.to_array(args.number_rows)?,
maxv.to_array(args.number_rows)?,
buckets.to_array(args.number_rows)?,
];

width_bucket_kern(&arrays).map(ColumnarValue::Array)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was there a need for this change?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct me if my understanding is wrong, I only swapped out make_scalar_function so I could call take_function_args up front, once the UDF no longer uses Signature::user_defined, we need the shared take_function_args guard to produce the standard “failed to match any signature” planner error. If we want to keep the helper, I can move the take_function_args call into the closure, but the behavior change was just to hook into the common arity handling.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The signature should now guard that for us; take_function_args would only be used as an ergonomic way of extracting the args required. The fact that it errors if the arg count is wrong would just be an extra layer of defense but not the main defense, as only if something went wrong with the signature related code could that error path ever occur.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, Should I revert this?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That would be preferable

}
}

fn width_bucket_signature(arg: &Coercion, bucket: &Coercion) -> TypeSignature {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd prefer to inline this directly

@kumarUjjawal

Copy link
Copy Markdown
Contributor Author

@Jefffrey The cargo fmt error doen't seem to be from my code changes. Can you check.

@Jefffrey
Jefffrey added this pull request to the merge queue Dec 9, 2025
Merged via the queue into apache:main with commit 4ddee14 Dec 9, 2025
27 checks passed
@Jefffrey

Jefffrey commented Dec 9, 2025

Copy link
Copy Markdown
Contributor

Thanks @kumarUjjawal

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

common Related to common crate spark Changes to datafusion-spark crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants