Skip to content

refactor(datafusion): split Convertible implementations by expression - #11507

Draft
danking wants to merge 1 commit into
dk/one-step-expression-converterfrom
dk/one-step-expression-converter-cleanup
Draft

danking wants to merge 1 commit into
dk/one-step-expression-converterfrom
dk/one-step-expression-converter-cleanup

Conversation

@danking

@danking danking commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Replace the built-in conversion enum from #11506 with one struct per expression form. Each struct groups recognition, captured state, and construction and implements the same Convertible interface.

This PR targets the API branch so its diff contains only the implementation organization.

Changes

  • Add individual implementations for binary expressions, columns, literals, casts, boolean and null operations, membership, field access, string functions, list functions, CASE, and scoped array-transform construction.
  • Keep one NodeConversion containing a boxed Convertible; no ConversionPlan or ExpressionConstructor abstraction is introduced.
  • Retain NodeConversion::fixed and the shared fixed-child helper for arity checking.
  • Preserve extension-literal metadata, dictionary-string semantics, cast rules, and the recursive driver's private lambda scope.
  • Move conversion-specific helpers and test imports with their implementation. Public signatures and test bodies are unchanged from the parent PR.

Validation

Source diff, branch ancestry, and DCO sign-off reviewed. Builds, tests, linting, and formatting have not been run, per repository guidance.

Stack

  1. #11506 — One-step Convertible interface, rebased onto current develop.
  2. This cleanup, targeting dk/one-step-expression-converter.

@codspeed

codspeed Bot commented Nov 16, 2026

Copy link
Copy Markdown

Merging this PR will regress 62 benchmarks

2 warnings

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 198 benchmarks with a significant change ran in a different software environment

Their base and head results were measured with a different OS or toolchain, such as compiler, runtime or library versions, so the changes might come from this difference rather than from the code. Open an affected benchmark to see what changed.

See affected benchmarks

⚡ 136 improved benchmarks
❌ 62 regressed benchmarks
✅ 1509 untouched benchmarks
⏩ 304 skipped benchmarks1
🗄️ 611 archived benchmarks run2

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ CPU Simulation filter_run_end[len=65536_run=4_density=0.5] 464.6 µs 2,636 µs -82.38%
❌ CPU Simulation filter_random_by_random[10000] 7.5 µs 31.9 µs -76.48%
❌ CPU Simulation filter_run_end[len=16384_run=16_density=0.9] 46.5 µs 191 µs -75.66%
❌ CPU Simulation filter_run_end[len=16384_run=16_density=0.5] 46.6 µs 191.1 µs -75.63%
❌ CPU Simulation integer_stats_nullable 549.7 µs 1,789.7 µs -69.29%
❌ CPU Simulation filter_run_end[len=16384_run=16_density=0.1] 61 µs 193.1 µs -68.43%
❌ CPU Simulation integer_stats_all_nullable 924.7 µs 2,432.3 µs -61.98%
❌ CPU Simulation filter_run_end[len=4096_run=16_density=0.9] 23.6 µs 57.7 µs -59.15%
❌ CPU Simulation filter_run_end[len=4096_run=16_density=0.1] 23.4 µs 56.4 µs -58.51%
❌ CPU Simulation filter_run_end[len=4096_run=16_density=0.5] 23.5 µs 56.7 µs -58.49%
❌ CPU Simulation test_repeated_scan_filter[int] 1.1 ms 2.1 ms -49.92%
❌ CPU Simulation float_stats_non_null 459.1 µs 893.1 µs -48.6%
❌ CPU Simulation filter_run_end[len=65536_run=256_density=0.5] 36.4 µs 65.6 µs -44.51%
❌ CPU Simulation test_repeated_scan_filter[int_float] 2.3 ms 3.6 ms -36.83%
❌ CPU Simulation integer_stats_all_non_null 757.7 µs 1,184.9 µs -36.05%
❌ CPU Simulation delta_compress_u32 62 µs 96.7 µs -35.86%
❌ CPU Simulation execute_mask_then_value[256] 23.6 µs 36.7 µs -35.57%
❌ CPU Simulation widen_nullable[16384] 232.9 µs 358.6 µs -35.06%
❌ CPU Simulation test_scan_filter_to_arrow_array[int_float] 2.9 ms 4.2 ms -31.44%
❌ CPU Simulation test_scan_filter[int_float] 2.9 ms 4.2 ms -30.77%
... ... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing dk/one-step-expression-converter-cleanup (2d72f0c) with develop (c58d623)3

Open in CodSpeed

Footnotes

  1. 304 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. 611 benchmarks were run, but are now archived. If they were deleted in another branch, consider rebasing to remove them from the report. Instead if they were added back, click here to restore them. ↩

  3. No successful run was found on dk/one-step-expression-converter (dfa887b) during the generation of this report, so develop (c58d623) was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

Signed-off-by: Daniel King <dan@spiraldb.com>
@danking
danking force-pushed the dk/one-step-expression-converter branch from dfa887b to aac7b1e Compare November 16, 2026 20:12
@danking
danking force-pushed the dk/one-step-expression-converter-cleanup branch from 2d72f0c to 2847d23 Compare November 16, 2026 20:12
@danking danking changed the title refactor(datafusion): organize node conversion with Convertible refactor(datafusion): split Convertible implementations by expression Nov 16, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant