You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Support smaller decimal types through SQL interface #17747
Due to a bug in arrow-rs, the new smaller decimal values couldn't have been used through the SQL interface. Now that arrow 56.2.0 is released (and soon to be updated here), support in datafusion-sql should be implemented + slt tests to verify the behavior of various aggregations which are supported now after #17501.
There has been a bunch of work to improve decimal support (by me and others), but the SQL part remains open. My current understanding and thinking here is:
Supporting decimal32/64 through the SQL interface is a breaking change. Even though the current implementation makes it easy to use them together, there are cases where that doesn't currently work (The most obvious one I've run into is the Spark mod function).
Making it work will probably require changes to type coercion and how functions operating on decimals determine their output type.
I know there are people thinking/working on changing DataFusion's type system to a more logical one that abstracts away the arrow type system. This seems like a good fit to other SQL systems, as they usually treat decimal (or numeric as its often called) as effectively a logical type, determining its physical size by the actual data it contains.
If 3 actually happens, it'll make this much easier. Otherwise - I think we can put it behind some config flag, which is what was done for Utf8View and it was introduced and replaced Utf8 as the default string type.
Activity
AdamGS commentedon Sep 23, 2025
take
AdamGS commentedon Oct 14, 2025
There has been a bunch of work to improve decimal support (by me and others), but the SQL part remains open. My current understanding and thinking here is:
modfunction).If 3 actually happens, it'll make this much easier. Otherwise - I think we can put it behind some config flag, which is what was done for
Utf8Viewand it was introduced and replacedUtf8as the default string type.