Repository navigation
Implement CastExtension for extensible casting to/from extension types - #27383
paleolimbot wants to merge 7 commits into
Conversation
|
cc @adriangb, @sdf-jkl, @AdamGS, and @tschwarzinger, who all seemed interested in the previous PR implementing this. |
tschwarzinger
left a comment
There was a problem hiding this comment.
Thanks, this is so cool! I can see that this can then be used by a
CastExtension that is hooked up to a registry.
Two things from my side:
Avoiding Two Functions
Do you think we can avoid having two casting functions? This always leaves the
door open for calling cast_column instead of cast_column_fields and losing
type information.
What about:
- Adding
source_fieldandtarget_field(instead oftarget_type) to
cast_column. This will unfortunately be a breaking change but would avoid
the existence of two functions. - Maybe make
CastExtensionpart ofCastOptions? We already have
FormatOptions::formatter_factoryfor wiring up the formatting extensions and
I think this would be a similar use case.
EDIT: Oh I see now that CastOptions is part of arrow. Ok so this would work only
with an DFCastOptions. I'll still leave the suggestion here as input but this
makes it more unattractive.
End 2 End Prototype
I think having an end to end prototype of a casting registry would be a good
thing to have before committing in on a design. I agree with you that we will
likely have a separate cast registry (possibly with an additional "convenience"
method inside DFExtensionType). From there we could create the CastExtension
and then use it in the planner / cast expr / pyhsical cast expr.
For example, how would we do overrides with the current system? It would allow
defining new Cast semantics for Float64 -> Float64 after all I can see. It
would be interesting how we want to embed this in existing code. For example,
here the code of the physical cast expression. If there is no nested struct
cast, it delegates to arrow kernels which are unaware of any cast extensions.
fn cast_array_by_name(
array: &ArrayRef,
cast_type: &DataType,
cast_options: &CastOptions<'static>,
) -> Result<ArrayRef> {
// If types are already equal, no cast needed
if array.data_type() == cast_type {
return Ok(Arc::clone(array));
}
if datafusion_common::nested_struct::requires_nested_struct_cast(
array.data_type(),
cast_type,
) {
datafusion_common::nested_struct::cast_column(array, cast_type, cast_options)
} else {
if !cast_options.safe {
ensure_temporal_array_timestamp_bounds(array, cast_type)?;
}
Ok(kernels::cast::cast_with_options(
array,
cast_type,
cast_options,
)?)
}
}Curious to hear your thoughts on that!
| None, | ||
| cast_extension, | ||
| )?; | ||
| let (source_key, source_value) = validate_map_entries_field(source_entries)?; |
There was a problem hiding this comment.
Is this only needed because we need the fields for cast_column_fields or should we fix this even without the casting extensions?
There was a problem hiding this comment.
As I understand it this is already validated as part of the map cast validation but conslidating the logic is an invasive change and this additional validation is cheap. I can try harder to consolidate the two pieces of logic if this is important.
| .get("ARROW:extension:name") | ||
| .map(String::as_str) | ||
| == Some("arrow.bool8") | ||
| && source_field.data_type() == &DataType::Int8 |
There was a problem hiding this comment.
Just wondering whether an error is more appropriate here if the extension type has an invalid storage data type (for the tests it's ok anyways)
There was a problem hiding this comment.
I think this would be the role of the CastExtension (it could return true for the "can cast types" but error for a cast it thought was inappropriate).
| field("root", DataType::List(target_item)), | ||
| ); | ||
|
|
||
| assert_boolean_strings(result.as_list::<i32>().values()); |
This seemed like a nice change, particularly since we're introducing a new function, so I went with it. Happy to do whatever here.
I hope so. I went with a new function name to avoid breaking things, but we could deprecate the old function too.
I believe Spark compatibility needs these types of overrides. DataFusion itself overrides some types of casting, too (e.g., structs) if this approach is ever upstreamed into arrow-cast. It's an open question how to design the registry to handle conflicts (last registered wins vs. never dispatch if multiple registrations match vs. require a declaration on register and hash the input and output types).
That's a great point...I'll add it to this PR and we can take it out if it turns out it's something we agree should be done.
If possible I would like to require the type registry to implement CastExtension and also handle those registration. This way it can handle cast to string automatically for extension types that implement a formatter. |
|
I prototyped the plumbing for the registry in paleolimbot#3 . Happy to fold in if it's appropriate. |
Which issue does this PR close?
Rationale for this change
Long ago in #21071 I prototyped cast handling so we can finally get the fabled UUID -> string cast to work. In general, there is no built-in way to teach DataFusion how to handle casts that are not handled by arrow-rs'
cast(). Notably for extension types because they cannot be captured by theDataType, which is the only thingcast()knows about.What changes are included in this PR?
This PR intoduces a
CastExtension, whose job is to override casts to/from an arbitrary type pair or fall back to the existing implementation. Much of the casting in DataFusion has already been consolidated into nested_struct.rs to handle the casting of struct types, which already has plumbing for recursive types (e.g., lists of structs today, but lists of extensions if we implement it there), so that's where this implementation sits.This PR just implements the trait and pipes it through casting in nested_struct.rs. To actually plumb through the cast we need a way to pipe this through to the planner...I can do that here if anybody would like. I also didn't implement a variant of the
CastExtensionthat aggregates other cast extensions...in practice you'd have a registry that dispatches to individually registered cast extensions.I didn't implement this as part of the
DFExtensionTypetrait because casting is an interaction between two types, one or both of which may be an extension type.There are some alternatives: casts that occur purely in SQL can be overridden after 55.1 using an optimizer rule or custom planner that replace
Expr::CastorExpr::TryCastwith function calls or custom physical expressions; however, these are only a subset of casts that occur in DataFusion. Casts inserted as part of the process of combining multiple Parquet files can be worked around with the physical expression adapter, but may other types of casts have no workaround.What is the testing strategy for this PR?
Inline tests directly in the file, including all nested branches
Are there any user-facing changes?
Existing APIs should be unchanged, this adds a new API.