Skip to content

Support smaller decimal types through SQL interface #17747

Open

Description

@AdamGS

Part of #17489.

Due to a bug in arrow-rs, the new smaller decimal values couldn't have been used through the SQL interface. Now that arrow 56.2.0 is released (and soon to be updated here), support in datafusion-sql should be implemented + slt tests to verify the behavior of various aggregations which are supported now after #17501.

Activity

  1. AdamGS commented on Sep 23, 2025

    @AdamGS
    ContributorAuthor

    take

  2. AdamGS commented on Oct 14, 2025

    @AdamGS
    ContributorAuthor

    There has been a bunch of work to improve decimal support (by me and others), but the SQL part remains open. My current understanding and thinking here is:

    1. Supporting decimal32/64 through the SQL interface is a breaking change. Even though the current implementation makes it easy to use them together, there are cases where that doesn't currently work (The most obvious one I've run into is the Spark mod function).
    2. Making it work will probably require changes to type coercion and how functions operating on decimals determine their output type.
    3. I know there are people thinking/working on changing DataFusion's type system to a more logical one that abstracts away the arrow type system. This seems like a good fit to other SQL systems, as they usually treat decimal (or numeric as its often called) as effectively a logical type, determining its physical size by the actual data it contains.

    If 3 actually happens, it'll make this much easier. Otherwise - I think we can put it behind some config flag, which is what was done for Utf8View and it was introduced and replaced Utf8 as the default string type.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions