Skip to content

Parse scientific notation and mixed integer/decimal IN lists in row filters - #4371

Open
rahulsmahadev wants to merge 2 commits into
apache:mainfrom
rahulsmahadev:rahul/parser-sci-notation-in-lists
Open

rahulsmahadev wants to merge 2 commits into
apache:mainfrom
rahulsmahadev:rahul/parser-sci-notation-in-lists

Conversation

@rahulsmahadev

@rahulsmahadev rahulsmahadev commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Rationale for this change

The row filter parser rejects two kinds of valid numeric literals:

tbl.scan(row_filter="price > 1e-3")
# ParseException: Expected end of text, found 'e'  (at char 9), (line:1, col:10)
tbl.scan(row_filter="quantity in (1, 2.5)")
# ParseException: Expected ')', found '.'  (at char 17), (line:1, col:18)

Decimal literals used pyparsing's common.real(), which has no exponent, so 1e-3 parses as 1 and the rest is left over. A literal set had to be all decimals or all integers, so a list that mixes 1 and 2.5 did not parse either.

This change parses a number with an exponent (1e-3, -1.5E3, 2.5e1) as a DoubleLiteral: in ANSI SQL it is an approximate numeric literal, and Spark, Trino and DuckDB parse it as a double. Numbers without an exponent parse as before (DecimalLiteral or LongLiteral). A literal set can now mix integers, decimals and doubles. Each token is parsed to its own literal type before the existing set deduplication and binding rules apply. To keep long_col = 1e3 working, a DoubleLiteral with a whole value now converts to int and long columns; a fractional value raises, as it does for a DecimalLiteral. Binding to decimal and double columns uses the existing conversions (a double is rounded to the scale of a decimal column). A list that mixes numbers with strings still raises a ParseException (test_in_different_types).

Are these changes tested?

Yes. In tests/expressions/test_parser.py, test_scientific_notation covers exponents on either side of a comparison and in BETWEEN, test_in_mixed_numbers covers mixed IN lists (in both orders) and NOT IN lists, and test_bind_numeric_literals binds exponent forms against decimal, long and double columns, and a mixed integer and double IN list against a long column. In tests/expressions/test_literals.py, test_double_to_integer_and_long_conversion covers the new conversion, and test_invalid_float_conversions no longer expects a DoubleLiteral to reject int and long. The new tests fail on main and pass with this change.

Are there any user-facing changes?

Bug fix: row filter strings accept scientific notation (as doubles) and IN lists that mix numeric literals. A DoubleLiteral with a whole value now converts to int and long. Expressions that parsed before parse to the same result.

This pull request and its description were written by Isaac.

The row filter parser only accepted decimals with a fractional part and
literal sets of a single numeric kind, so `x > 1e-3` and `x in (1, 2.5)`
raised a ParseException.

Co-authored-by: Isaac <no-reply@databricks.com>
@rahulsmahadev

Copy link
Copy Markdown
Contributor Author

@jackylee-ch could you take a look when you have a chance? This small change to the expression parser lets 1e-3 and IN (1, 2.5) parse. You last updated this parser in #3917.

@Fokko

Fokko commented Nov 14, 2026

Copy link
Copy Markdown
Contributor

Thanks for picking this up! I think the scientific notation part should follow ANSI SQL. In ANSI, <mantissa>E<exponent> is an approximate numeric literal (so a double), while only digits with an optional decimal point are exact numeric (decimal). Spark 3+, Trino and DuckDB all parse 1e-3 as a double.

Parsing it as a DecimalLiteral also causes a binding problem: Decimal("1e3") is 1E+3 with scale -3, so dec_col = 1e3 cannot bind to any decimal column, and price > 1e-3 only binds when price has exactly scale 3. The tests don't catch this because Decimal("2E+2") == Decimal("200").

Could we emit a DoubleLiteral for the exponent forms instead? That would need a DoubleLiteral.to(LongType) conversion so long_col = 1e3 keeps working. It would also be great to add a few tests that bind against a schema with decimal, long and double columns.

The mixed integer/decimal IN list change looks good to me and is in line with ANSI.

…er types

A number with an exponent is an approximate numeric literal in ANSI SQL, and
Spark, Trino and DuckDB parse 1e-3 as a double. As a DecimalLiteral, 1e3 has
scale -3 and could not bind to decimal columns. Parse the exponent forms as
DoubleLiteral instead, and convert a whole DoubleLiteral to int and long columns
(a fractional value raises, as for decimals) so that long_col = 1e3 binds.

Co-authored-by: Isaac <no-reply@databricks.com>
@rahulsmahadev

Copy link
Copy Markdown
Contributor Author

Thanks, done in bccbed0: numbers with an exponent now parse as DoubleLiteral and the others as before, and a whole DoubleLiteral converts to long (and int, so int_col = 1e3 keeps working); a fractional one raises like a DecimalLiteral. test_bind_numeric_literals binds them against decimal, long and double columns, so dec_col = 1e3 now binds as 1000.00. I also updated the description.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants