Skip to content

Implement CastExtension for extensible casting to/from extension types - #27383

Open
paleolimbot wants to merge 7 commits into
apache:mainfrom
paleolimbot:codex/cast-extension-nested
Open

paleolimbot wants to merge 7 commits into
apache:mainfrom
paleolimbot:codex/cast-extension-nested

Conversation

@paleolimbot

Copy link
Copy Markdown
Member

Which issue does this PR close?

Rationale for this change

Long ago in #21071 I prototyped cast handling so we can finally get the fabled UUID -> string cast to work. In general, there is no built-in way to teach DataFusion how to handle casts that are not handled by arrow-rs' cast(). Notably for extension types because they cannot be captured by the DataType, which is the only thing cast() knows about.

What changes are included in this PR?

This PR intoduces a CastExtension, whose job is to override casts to/from an arbitrary type pair or fall back to the existing implementation. Much of the casting in DataFusion has already been consolidated into nested_struct.rs to handle the casting of struct types, which already has plumbing for recursive types (e.g., lists of structs today, but lists of extensions if we implement it there), so that's where this implementation sits.

This PR just implements the trait and pipes it through casting in nested_struct.rs. To actually plumb through the cast we need a way to pipe this through to the planner...I can do that here if anybody would like. I also didn't implement a variant of the CastExtension that aggregates other cast extensions...in practice you'd have a registry that dispatches to individually registered cast extensions.

I didn't implement this as part of the DFExtensionType trait because casting is an interaction between two types, one or both of which may be an extension type.

There are some alternatives: casts that occur purely in SQL can be overridden after 55.1 using an optimizer rule or custom planner that replace Expr::Cast or Expr::TryCast with function calls or custom physical expressions; however, these are only a subset of casts that occur in DataFusion. Casts inserted as part of the process of combining multiple Parquet files can be worked around with the physical expression adapter, but may other types of casts have no workaround.

What is the testing strategy for this PR?

Inline tests directly in the file, including all nested branches

Are there any user-facing changes?

Existing APIs should be unchanged, this adds a new API.

@github-actions github-actions Bot added the common Related to common crate label Nov 12, 2026
@paleolimbot
paleolimbot marked this pull request as ready for review November 13, 2026 14:53
@paleolimbot

Copy link
Copy Markdown
Member Author

cc @adriangb, @sdf-jkl, @AdamGS, and @tschwarzinger, who all seemed interested in the previous PR implementing this.

@tschwarzinger tschwarzinger left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this is so cool! I can see that this can then be used by a
CastExtension that is hooked up to a registry.

Two things from my side:

Avoiding Two Functions

Do you think we can avoid having two casting functions? This always leaves the
door open for calling cast_column instead of cast_column_fields and losing
type information.

What about:

  • Adding source_field and target_field (instead of target_type) to
    cast_column. This will unfortunately be a breaking change but would avoid
    the existence of two functions.
  • Maybe make CastExtension part of CastOptions? We already have
    FormatOptions::formatter_factory for wiring up the formatting extensions and
    I think this would be a similar use case.

EDIT: Oh I see now that CastOptions is part of arrow. Ok so this would work only
with an DFCastOptions. I'll still leave the suggestion here as input but this
makes it more unattractive.

End 2 End Prototype

I think having an end to end prototype of a casting registry would be a good
thing to have before committing in on a design. I agree with you that we will
likely have a separate cast registry (possibly with an additional "convenience"
method inside DFExtensionType). From there we could create the CastExtension
and then use it in the planner / cast expr / pyhsical cast expr.

For example, how would we do overrides with the current system? It would allow
defining new Cast semantics for Float64 -> Float64 after all I can see. It
would be interesting how we want to embed this in existing code. For example,
here the code of the physical cast expression. If there is no nested struct
cast, it delegates to arrow kernels which are unaware of any cast extensions.

fn cast_array_by_name(
    array: &ArrayRef,
    cast_type: &DataType,
    cast_options: &CastOptions<'static>,
) -> Result<ArrayRef> {
    // If types are already equal, no cast needed
    if array.data_type() == cast_type {
        return Ok(Arc::clone(array));
    }

    if datafusion_common::nested_struct::requires_nested_struct_cast(
        array.data_type(),
        cast_type,
    ) {
        datafusion_common::nested_struct::cast_column(array, cast_type, cast_options)
    } else {
        if !cast_options.safe {
            ensure_temporal_array_timestamp_bounds(array, cast_type)?;
        }
        Ok(kernels::cast::cast_with_options(
            array,
            cast_type,
            cast_options,
        )?)
    }
}

Curious to hear your thoughts on that!

None,
cast_extension,
)?;
let (source_key, source_value) = validate_map_entries_field(source_entries)?;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this only needed because we need the fields for cast_column_fields or should we fix this even without the casting extensions?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I understand it this is already validated as part of the map cast validation but conslidating the logic is an invasive change and this additional validation is cheap. I can try harder to consolidate the two pieces of logic if this is important.

.get("ARROW:extension:name")
.map(String::as_str)
== Some("arrow.bool8")
&& source_field.data_type() == &DataType::Int8

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just wondering whether an error is more appropriate here if the extension type has an invalid storage data type (for the tests it's ok anyways)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this would be the role of the CastExtension (it could return true for the "can cast types" but error for a cast it thought was inappropriate).

field("root", DataType::List(target_item)),
);

assert_boolean_strings(result.as_list::<i32>().values());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔥

@paleolimbot

paleolimbot commented Nov 17, 2026 •

Copy link
Copy Markdown
Member Author

Ok so this would work only with an DFCastOptions

This seemed like a nice change, particularly since we're introducing a new function, so I went with it. Happy to do whatever here.

Do you think we can avoid having two casting functions?

I hope so. I went with a new function name to avoid breaking things, but we could deprecate the old function too.

For example, how would we do overrides with the current system?

I believe Spark compatibility needs these types of overrides. DataFusion itself overrides some types of casting, too (e.g., structs) if this approach is ever upstreamed into arrow-cast. It's an open question how to design the registry to handle conflicts (last registered wins vs. never dispatch if multiple registrations match vs. require a declaration on register and hash the input and output types).

I think having an end to end prototype of a casting registry would be a good thing to have before committing in on a design.

That's a great point...I'll add it to this PR and we can take it out if it turns out it's something we agree should be done.

possibly with an additional "convenience" method inside DFExtensionType

If possible I would like to require the type registry to implement CastExtension and also handle those registration. This way it can handle cast to string automatically for extension types that implement a formatter.

@paleolimbot

Copy link
Copy Markdown
Member Author

I prototyped the plumbing for the registry in paleolimbot#3 . Happy to fold in if it's appropriate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

common Related to common crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support pluggable cast resolution to support casting to and from extension types

2 participants