Skip to content

Dictionary encode decimals - #11332

Open
robert3005 wants to merge 12 commits into
developfrom
rk/decimal-dict-layout
Open

robert3005 wants to merge 12 commits into
developfrom
rk/decimal-dict-layout

Conversation

@robert3005

@robert3005 robert3005 commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Low-cardinality decimal columns could not be dictionary encoded, either by the compressor or as a dict layout.

The worst case was wide decimals (i128/i256) under the core edition. There, the byte parts scheme runs in v1 mode and leaves them canonical, so they were written uncompressed however few distinct values they had. Without editions, byte parts split them into up to four arrays, and each part needs its own dictionary and codes.

This PR adds a decimal dictionary scheme to the compressor and lets DictStrategy write decimal columns as dict layouts.

Changes

  • DecimalDictScheme (vortex-compressor/src/builtins/dict/decimal.rs) produces Dict(codes, decimal values). It mirrors FloatDictScheme:
    • it skips arrays where more than half of the values are distinct, and otherwise lets sampling decide;
    • it excludes IntDictScheme from its codes and from its values' byte parts.
  • DecimalStats (vortex-compressor/src/stats/decimal.rs) holds the null and value counts, and counts distinct values only when a scheme requests it.
  • DecimalScheme (byte parts) still returns AlwaysUse, unless the distinct count shows at most half of the values are distinct. Then it defers to sampling, so the compressor keeps whichever of byte parts and the dictionary is smaller. The count only exists when DecimalDictScheme is eligible, so without it decimals compress exactly as before.

@robert3005
robert3005 force-pushed the rk/decimal-dict-layout branch from 272bd3f to 2b16f70 Compare November 9, 2026 19:09
@robert3005
robert3005 force-pushed the rk/decimal-dict-layout branch 2 times, most recently from 09205ba to a3481b0 Compare November 11, 2026 09:13
robert3005 added a commit that referenced this pull request Nov 11, 2026
The `vortex-file` `v2::file_stats_reader` tests fail intermittently. For
example, `dynamic_proofs_refresh_before_reads` failed in a full
`vortex-file` lib run while checking #11332.

Each test called `SESSION.clone().with_handle(handle)` on one static
session. Clones of a `VortexSession` share their state, and
`with_handle` stores the runtime handle in that shared state. Tests
running at the same time therefore overwrote each other's handle, and a
test could drive its reads with another test's runtime, including one
that had already shut down. Some readers in these tests were also built
from the static `SESSION` directly, not from the test's own session.

Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
@robert3005
robert3005 force-pushed the rk/decimal-dict-layout branch from 91be5e3 to 7e90929 Compare November 11, 2026 15:01
@robert3005 robert3005 changed the title Dictionary encode low-cardinality decimal columns in DictStrategy Dictionary encode low-cardinality decimals Nov 11, 2026
@robert3005
robert3005 marked this pull request as ready for review November 11, 2026 21:53
@robert3005 robert3005 added the changelog/feature A new feature label Nov 11, 2026
@robert3005 robert3005 added the action/bench-compress Run only the compression benchmark on this PR label Nov 11, 2026
@github-actions github-actions Bot removed the action/bench-compress Run only the compression benchmark on this PR label Nov 11, 2026
@codspeed

codspeed Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

Merging this PR will regress 2 benchmarks

2 warnings

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 1 benchmark is dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚡ 17 improved benchmarks
❌ 2 regressed benchmarks
✅ 1917 untouched benchmarks
⏩ 50 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ CPU Simulation bench_compare_sliced_dict_varbinview[(1000, 10000)] 91.6 µs 108 µs -15.18%
❌ CPU Simulation random_i8[0.01] 101.1 µs 114.9 µs -12.04%
⚡ CPU Simulation encode_primitives[i64, (2000, 512)] 303.8 µs 197.4 µs +53.87%
⚡ CPU Simulation encode_primitives[f32, (2000, 512)] 272.2 µs 197.3 µs +38.01%
⚡ CPU Simulation encode_primitives[u8, (2000, 2)] 164.9 µs 124.9 µs +32.05%
⚡ CPU Simulation encode_primitives[u8, (2000, 32)] 167.8 µs 127.4 µs +31.67%
⚡ CPU Simulation encode_primitives[i64, (2000, 32)] 230.4 µs 177.7 µs +29.66%
⚡ CPU Simulation encode_primitives[i64, (2000, 2)] 222.8 µs 172.5 µs +29.17%
⚡ CPU Simulation encode_primitives[u8, (2000, 512)] 184 µs 143.3 µs +28.43%
⚡ CPU Simulation encode_primitives[f32, (2000, 2)] 206.4 µs 165.3 µs +24.8%
⚡ CPU Simulation encode_primitives[f32, (2000, 32)] 211 µs 170 µs +24.12%
⚡ CPU Simulation encode_varbin[(2000, 512)] 535.6 µs 438.4 µs +22.18%
⚡ CPU Simulation encode_varbinview[(2000, 512)] 544.6 µs 448.2 µs +21.51%
⚡ CPU Simulation dict_compress_string 391.4 µs 336.9 µs +16.18%
⚡ CPU Simulation dict_compress_u32 368.9 µs 319.5 µs +15.48%
⚡ CPU Simulation encode_varbin[(2000, 32)] 406.4 µs 356.1 µs +14.13%
⚡ CPU Simulation encode_varbin[(2000, 2)] 396.6 µs 349.3 µs +13.53%
⚡ CPU Simulation encode_varbinview[(2000, 32)] 440.6 µs 391.5 µs +12.53%
⚡ CPU Simulation encode_varbinview[(2000, 2)] 398.7 µs 356.2 µs +11.94%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing rk/decimal-dict-layout (d2ca683) with develop (25f4349)

Open in CodSpeed

Footnotes

  1. 50 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Benchmarks: Compression 📖

Commits: PR 7e90929e vs base 83e1e0f2


vortex / vortex-file-compressed / ns (1.010x ➖, 0↑ 0↓)
name hot ns (PR / base / %diff) cold ns (PR / base / %diff) hot/cold (PR / base / %diff)
compress time/arade 461782875 / 457793318 / +0.9% 460901416 / 457323216 / +0.8% 1.00 / 1.00 / +0.1%
compress time/bimbo 2365198560 / 2376036856 / -0.5% 2333199038 / 2351705520 / -0.8% 1.01 / 1.01 / +0.3%
compress time/cmsprovider 1063499712 / 1063714152 / -0.0% 1049256703 / 1048176310 / +0.1% 1.01 / 1.01 / -0.1%
compress time/euro2016 170616985 / 170859960 / -0.1% 171775269 / 172946293 / -0.7% 0.99 / 0.99 / +0.5%
compress time/food 209135773 / 208983053 / +0.1% 209800019 / 210174090 / -0.2% 1.00 / 0.99 / +0.3%
compress time/hashtags 362321028 / 360072686 / +0.6% 354842895 / 348881600 / +1.7% 1.02 / 1.03 / -1.1%
compress time/sparse-wide-flat-runs 94520307 / 88774410 / +6.5% 95415383 / 90873972 / +5.0% 0.99 / 0.98 / +1.4%
compress time/sparse-wide-lists-runs 657747262 / 679942618 / -3.3% 656338623 / 679808458 / -3.5% 1.00 / 1.00 / +0.2%
compress time/sparse-wide-lists-shuffled 303781694 / 294832671 / +3.0% 307520527 / 289965754 / +6.1% 0.99 / 1.02 / -2.8%
compress time/sparse-wide-nested-child-nulls 635023611 / 621231174 / +2.2% 623639713 / 611579378 / +2.0% 1.02 / 1.02 / +0.2%
compress time/sparse-wide-nested-runs 637365015 / 633048504 / +0.7% 644693880 / 612747547 / +5.2% 0.99 / 1.03 / -4.3%
compress time/statpopgen 497791314 / 494693622 / +0.6% 499398547 / 493954941 / +1.1% 1.00 / 1.00 / -0.5%
compress time/taxi 207845834 / 205709912 / +1.0% 208286850 / 208416616 / -0.1% 1.00 / 0.99 / +1.1%
compress time/tpc-h l_comment canonical 405098014 / 402464882 / +0.7% 401304010 / 401278343 / +0.0% 1.01 / 1.00 / +0.6%
compress time/tpc-h l_comment chunked 405211802 / 402615760 / +0.6% 404143622 / 403644807 / +0.1% 1.00 / 1.00 / +0.5%
compress time/wide table cols=100 chunks=1 rows=1000 5813659 / 5822596 / -0.2% 8537498 / 7802363 / +9.4% 0.68 / 0.75 / -8.8%
compress time/wide table cols=100 chunks=50 rows=1000 5777344 / 5591925 / +3.3% 8182002 / 8028484 / +1.9% 0.71 / 0.70 / +1.4%
compress time/wide table cols=1000 chunks=1 rows=1000 57530576 / 56131394 / +2.5% 58928365 / 60348677 / -2.4% 0.98 / 0.93 / +5.0%
compress time/wide table cols=1000 chunks=50 rows=1000 56637590 / 55928269 / +1.3% 58553400 / 59040401 / -0.8% 0.97 / 0.95 / +2.1%
compress time/wide table cols=10000 chunks=1 rows=1000 642570358 / 626285140 / +2.6% 632829244 / 624556714 / +1.3% 1.02 / 1.00 / +1.3%
compress time/wide table cols=10000 chunks=50 rows=1000 647181250 / 630556740 / +2.6% 641410923 / 628204662 / +2.1% 1.01 / 1.00 / +0.5%
decompress time/arade 15127946 / 15110863 / +0.1% 16180240 / 15389349 / +5.1% 0.93 / 0.98 / -4.8%
decompress time/bimbo 67391479 / 66569276 / +1.2% 71682893 / 73919784 / -3.0% 0.94 / 0.90 / +4.4%
decompress time/cmsprovider 38487805 / 38682518 / -0.5% 40662513 / 40471821 / +0.5% 0.95 / 0.96 / -1.0%
decompress time/euro2016 12035796 / 12087290 / -0.4% 12744427 / 12677676 / +0.5% 0.94 / 0.95 / -0.9%
decompress time/food 4185777 / 3960589 / +5.7% 4913259 / 6047215 / -18.8% 🟢 0.85 / 0.65 / +30.1%
decompress time/hashtags 42356534 / 42118244 / +0.6% 43768431 / 42856915 / +2.1% 0.97 / 0.98 / -1.5%
decompress time/sparse-wide-flat-runs 13609428 / 13875190 / -1.9% 12641805 / 15251342 / -17.1% 🟢 1.08 / 0.91 / +18.3%
decompress time/sparse-wide-lists-runs 15471004 / 15112954 / +2.4% 15626852 / 15811088 / -1.2% 0.99 / 0.96 / +3.6%
decompress time/sparse-wide-lists-shuffled 57228190 / 57007367 / +0.4% 58482473 / 58133259 / +0.6% 0.98 / 0.98 / -0.2%
decompress time/sparse-wide-nested-child-nulls 17622236 / 16042880 / +9.8% 16700071 / 17039341 / -2.0% 1.06 / 0.94 / +12.1%
decompress time/sparse-wide-nested-runs 15956585 / 15896010 / +0.4% 17280904 / 17221061 / +0.3% 0.92 / 0.92 / +0.0%
decompress time/statpopgen 366793698 / 366494390 / +0.1% 374644841 / 367284277 / +2.0% 0.98 / 1.00 / -1.9%
decompress time/taxi 9848794 / 9948902 / -1.0% 12365005 / 11847691 / +4.4% 0.80 / 0.84 / -5.1%
decompress time/tpc-h l_comment canonical 15376849 / 15322095 / +0.4% 16212676 / 16457844 / -1.5% 0.95 / 0.93 / +1.9%
decompress time/tpc-h l_comment chunked 15562236 / 15218514 / +2.3% 16238109 / 14839119 / +9.4% 0.96 / 1.03 / -6.6%
decompress time/wide table cols=100 chunks=1 rows=1000 1572906 / 1567394 / +0.4% 1636628 / 1646672 / -0.6% 0.96 / 0.95 / +1.0%
decompress time/wide table cols=100 chunks=50 rows=1000 1511505 / 1551388 / -2.6% 1670058 / 1676818 / -0.4% 0.91 / 0.93 / -2.2%
decompress time/wide table cols=1000 chunks=1 rows=1000 13876971 / 14017198 / -1.0% 14529077 / 14463029 / +0.5% 0.96 / 0.97 / -1.5%
decompress time/wide table cols=1000 chunks=50 rows=1000 13999195 / 13819766 / +1.3% 14644413 / 14467690 / +1.2% 0.96 / 0.96 / +0.1%
decompress time/wide table cols=10000 chunks=1 rows=1000 145014402 / 144262528 / +0.5% 146944022 / 145845685 / +0.8% 0.99 / 0.99 / -0.2%
decompress time/wide table cols=10000 chunks=50 rows=1000 144697294 / 143779566 / +0.6% 145940550 / 144724107 / +0.8% 0.99 / 0.99 / -0.2%
vortex / vortex-file-compressed / bytes (1.000x ➖, 0↑ 0↓)
name bytes (PR / base / %diff)
vortex-file-compressed size/arade 142108908 / 142108908 / +0.0%
vortex-file-compressed size/bimbo 425251924 / 425251924 / +0.0%
vortex-file-compressed size/cmsprovider 416464764 / 416464764 / +0.0%
vortex-file-compressed size/euro2016 164045796 / 164045796 / +0.0%
vortex-file-compressed size/food 38951184 / 38951184 / +0.0%
vortex-file-compressed size/hashtags 183731332 / 183731332 / +0.0%
vortex-file-compressed size/sparse-wide-flat-runs 1661720 / 1661720 / +0.0%
vortex-file-compressed size/sparse-wide-lists-runs 10900860 / 10900860 / +0.0%
vortex-file-compressed size/sparse-wide-lists-shuffled 13447164 / 13447164 / +0.0%
vortex-file-compressed size/sparse-wide-nested-child-nulls 6686420 / 6686420 / +0.0%
vortex-file-compressed size/sparse-wide-nested-runs 6295172 / 6295172 / +0.0%
vortex-file-compressed size/statpopgen 201941044 / 201941044 / +0.0%
vortex-file-compressed size/taxi 49105412 / 49105412 / +0.0%
vortex-file-compressed size/tpc-h l_comment canonical 173437112 / 173478160 / -0.0%
vortex-file-compressed size/tpc-h l_comment chunked 173437112 / 173478160 / -0.0%
vortex-file-compressed size/wide table cols=100 chunks=1 rows=1000 934856 / 934856 / +0.0%
vortex-file-compressed size/wide table cols=100 chunks=50 rows=1000 934856 / 934856 / +0.0%
vortex-file-compressed size/wide table cols=1000 chunks=1 rows=1000 9333656 / 9333656 / +0.0%
vortex-file-compressed size/wide table cols=1000 chunks=50 rows=1000 9333656 / 9333656 / +0.0%
vortex-file-compressed size/wide table cols=10000 chunks=1 rows=1000 93357656 / 93357656 / +0.0%
vortex-file-compressed size/wide table cols=10000 chunks=50 rows=1000 93357656 / 93357656 / +0.0%
vortex / parquet / ns (1.000x ➖, 0↑ 0↓)
name hot ns (PR / base / %diff) cold ns (PR / base / %diff) hot/cold (PR / base / %diff)
compress time/arade 1157180026 / 1157445025 / -0.0% 1161727916 / 1163399882 / -0.1% 1.00 / 0.99 / +0.1%
compress time/bimbo 2133530878 / 2148053107 / -0.7% 2137158283 / 2148995854 / -0.6% 1.00 / 1.00 / -0.1%
compress time/cmsprovider 1672843312 / 1672252482 / +0.0% 1665562855 / 1675848149 / -0.6% 1.00 / 1.00 / +0.7%
compress time/euro2016 995042733 / 993711906 / +0.1% 995871970 / 995619648 / +0.0% 1.00 / 1.00 / +0.1%
compress time/food 405292622 / 408934062 / -0.9% 407850022 / 409397211 / -0.4% 0.99 / 1.00 / -0.5%
compress time/hashtags 538610720 / 539172616 / -0.1% 543900313 / 543519606 / +0.1% 0.99 / 0.99 / -0.2%
compress time/sparse-wide-flat-runs 41847842 / 42010498 / -0.4% 42422720 / 41610280 / +2.0% 0.99 / 1.01 / -2.3%
compress time/sparse-wide-lists-runs 80724662 / 81524311 / -1.0% 81497528 / 81609592 / -0.1% 0.99 / 1.00 / -0.8%
compress time/sparse-wide-lists-shuffled 155690937 / 157891582 / -1.4% 155597284 / 154371786 / +0.8% 1.00 / 1.02 / -2.2%
compress time/sparse-wide-nested-child-nulls 54574608 / 54439094 / +0.2% 55513283 / 55792346 / -0.5% 0.98 / 0.98 / +0.8%
compress time/sparse-wide-nested-runs 54992127 / 55198054 / -0.4% 56865124 / 55945677 / +1.6% 0.97 / 0.99 / -2.0%
compress time/statpopgen 590157819 / 589675788 / +0.1% 590438683 / 588613615 / +0.3% 1.00 / 1.00 / -0.2%
compress time/taxi 240709594 / 240304049 / +0.2% 241099703 / 244147790 / -1.2% 1.00 / 0.98 / +1.4%
compress time/tpc-h l_comment canonical 1205086243 / 1196311918 / +0.7% 1208821418 / 1199203488 / +0.8% 1.00 / 1.00 / -0.1%
compress time/tpc-h l_comment chunked 1199759478 / 1205957662 / -0.5% 1204762580 / 1214714644 / -0.8% 1.00 / 0.99 / +0.3%
compress time/wide table cols=100 chunks=1 rows=1000 3769993 / 3855604 / -2.2% 5069729 / 4412979 / +14.9% 🔴 0.74 / 0.87 / -14.9%
compress time/wide table cols=100 chunks=50 rows=1000 3822692 / 3657880 / +4.5% 4815010 / 4363110 / +10.4% 🔴 0.79 / 0.84 / -5.3%
compress time/wide table cols=1000 chunks=1 rows=1000 37599418 / 37162360 / +1.2% 39979135 / 38259821 / +4.5% 0.94 / 0.97 / -3.2%
compress time/wide table cols=1000 chunks=50 rows=1000 37301444 / 37038049 / +0.7% 38557505 / 38741646 / -0.5% 0.97 / 0.96 / +1.2%
compress time/wide table cols=10000 chunks=1 rows=1000 379229964 / 377634634 / +0.4% 390353080 / 389792945 / +0.1% 0.97 / 0.97 / +0.3%
compress time/wide table cols=10000 chunks=50 rows=1000 374864460 / 378069015 / -0.8% 382356586 / 390602137 / -2.1% 0.98 / 0.97 / +1.3%
decompress time/arade 54625676 / 54636202 / -0.0% 54788263 / 55241180 / -0.8% 1.00 / 0.99 / +0.8%
decompress time/bimbo 92135407 / 92089453 / +0.0% 92730751 / 92821259 / -0.1% 0.99 / 0.99 / +0.1%
decompress time/cmsprovider 103541370 / 103650330 / -0.1% 102845838 / 104282053 / -1.4% 1.01 / 0.99 / +1.3%
decompress time/euro2016 159313524 / 159542634 / -0.1% 159976857 / 160356931 / -0.2% 1.00 / 0.99 / +0.1%
decompress time/food 26588740 / 26587064 / +0.0% 26749181 / 27166175 / -1.5% 0.99 / 0.98 / +1.6%
decompress time/hashtags 447339541 / 446194854 / +0.3% 447593045 / 447180897 / +0.1% 1.00 / 1.00 / +0.2%
decompress time/sparse-wide-flat-runs 14866045 / 15076232 / -1.4% 15470255 / 15658045 / -1.2% 0.96 / 0.96 / -0.2%
decompress time/sparse-wide-lists-runs 277020082 / 277393499 / -0.1% 277393046 / 277454555 / -0.0% 1.00 / 1.00 / -0.1%
decompress time/sparse-wide-lists-shuffled 318530022 / 318635506 / -0.0% 319016696 / 319289073 / -0.1% 1.00 / 1.00 / +0.1%
decompress time/sparse-wide-nested-child-nulls 53128204 / 52664303 / +0.9% 53725977 / 53025863 / +1.3% 0.99 / 0.99 / -0.4%
decompress time/sparse-wide-nested-runs 53028770 / 52881100 / +0.3% 53089394 / 53327785 / -0.4% 1.00 / 0.99 / +0.7%
decompress time/statpopgen 1624573526 / 1633084785 / -0.5% 1633671136 / 1640466883 / -0.4% 0.99 / 1.00 / -0.1%
decompress time/taxi 60684591 / 60749436 / -0.1% 61080497 / 60897651 / +0.3% 0.99 / 1.00 / -0.4%
decompress time/tpc-h l_comment canonical 82309328 / 82453171 / -0.2% 82775917 / 83104641 / -0.4% 0.99 / 0.99 / +0.2%
decompress time/tpc-h l_comment chunked 82217112 / 82362416 / -0.2% 82504105 / 83060599 / -0.7% 1.00 / 0.99 / +0.5%
decompress time/wide table cols=100 chunks=1 rows=1000 1935688 / 1964894 / -1.5% 2117479 / 2294562 / -7.7% 0.91 / 0.86 / +6.8%
decompress time/wide table cols=100 chunks=50 rows=1000 1943801 / 1933832 / +0.5% 2144021 / 2105212 / +1.8% 0.91 / 0.92 / -1.3%
decompress time/wide table cols=1000 chunks=1 rows=1000 22796304 / 22477746 / +1.4% 23446693 / 23201239 / +1.1% 0.97 / 0.97 / +0.4%
decompress time/wide table cols=1000 chunks=50 rows=1000 22555205 / 22373172 / +0.8% 22833926 / 23119228 / -1.2% 0.99 / 0.97 / +2.1%
decompress time/wide table cols=10000 chunks=1 rows=1000 233162734 / 232505980 / +0.3% 236690380 / 235671704 / +0.4% 0.99 / 0.99 / -0.1%
decompress time/wide table cols=10000 chunks=50 rows=1000 230725327 / 232915658 / -0.9% 233063002 / 236158029 / -1.3% 0.99 / 0.99 / +0.4%
vortex / parquet / bytes (1.000x ➖, 0↑ 0↓)
name bytes (PR / base / %diff)
parquet size/arade 210020851 / 210020851 / +0.0%
parquet size/bimbo 376187078 / 376187078 / +0.0%
parquet size/cmsprovider 364268159 / 364268159 / +0.0%
parquet size/euro2016 111035158 / 111035158 / +0.0%
parquet size/food 34903783 / 34903783 / +0.0%
parquet size/hashtags 126189203 / 126189203 / +0.0%
parquet size/sparse-wide-flat-runs 1388944 / 1388944 / +0.0%
parquet size/sparse-wide-lists-runs 4327813 / 4327813 / +0.0%
parquet size/sparse-wide-lists-shuffled 5710569 / 5710569 / +0.0%
parquet size/sparse-wide-nested-child-nulls 1266042 / 1266042 / +0.0%
parquet size/sparse-wide-nested-runs 1388944 / 1388944 / +0.0%
parquet size/statpopgen 73385513 / 73385513 / +0.0%
parquet size/taxi 52142534 / 52142534 / +0.0%
parquet size/tpc-h l_comment canonical 153403836 / 153403836 / +0.0%
parquet size/tpc-h l_comment chunked 153403836 / 153403836 / +0.0%
parquet size/wide table cols=100 chunks=1 rows=1000 932404 / 932404 / +0.0%
parquet size/wide table cols=100 chunks=50 rows=1000 932404 / 932404 / +0.0%
parquet size/wide table cols=1000 chunks=1 rows=1000 9324004 / 9324004 / +0.0%
parquet size/wide table cols=1000 chunks=50 rows=1000 9324004 / 9324004 / +0.0%
parquet size/wide table cols=10000 chunks=1 rows=1000 93240004 / 93240004 / +0.0%
parquet size/wide table cols=10000 chunks=50 rows=1000 93240004 / 93240004 / +0.0%
vortex / arrow-ipc / ns (0.995x ➖, 0↑ 1↓)
name hot ns (PR / base / %diff) cold ns (PR / base / %diff) hot/cold (PR / base / %diff)
compress time/arade 94323209 / 95577340 / -1.3% 117848095 / 120569973 / -2.3% 0.80 / 0.79 / +1.0%
compress time/bimbo 597997956 / 605613844 / -1.3% 662371506 / 669643490 / -1.1% 0.90 / 0.90 / -0.2%
compress time/cmsprovider 375669163 / 378891912 / -0.9% 437555491 / 443415550 / -1.3% 0.86 / 0.85 / +0.5%
compress time/euro2016 38048218 / 38293547 / -0.6% 51784991 / 52773187 / -1.9% 0.73 / 0.73 / +1.3%
compress time/food 39638694 / 40101248 / -1.2% 53123556 / 54577150 / -2.7% 0.75 / 0.73 / +1.6%
compress time/hashtags 77629668 / 78073902 / -0.6% 106007695 / 106937592 / -0.9% 0.73 / 0.73 / +0.3%
compress time/sparse-wide-flat-runs 27819022 / 28094070 / -1.0% 36821221 / 37455120 / -1.7% 0.76 / 0.75 / +0.7%
compress time/sparse-wide-lists-runs 44510032 / 45202902 / -1.5% 51556021 / 52123395 / -1.1% 0.86 / 0.87 / -0.4%
compress time/sparse-wide-lists-shuffled 44518668 / 46528697 / -4.3% 49313309 / 50709625 / -2.8% 0.90 / 0.92 / -1.6%
compress time/sparse-wide-nested-child-nulls 40389452 / 40734789 / -0.8% 51792951 / 50724002 / +2.1% 0.78 / 0.80 / -2.9%
compress time/sparse-wide-nested-runs 40562620 / 40877581 / -0.8% 50073726 / 50870572 / -1.6% 0.81 / 0.80 / +0.8%
compress time/statpopgen 101537228 / 101553488 / -0.0% 143794768 / 144239202 / -0.3% 0.71 / 0.70 / +0.3%
compress time/taxi 58342704 / 59072548 / -1.2% 74554475 / 76219706 / -2.2% 0.78 / 0.78 / +1.0%
compress time/tpc-h l_comment canonical 127286004 / 128604607 / -1.0% 163467038 / 164602116 / -0.7% 0.78 / 0.78 / -0.3%
compress time/tpc-h l_comment chunked 128534226 / 128454070 / +0.1% 169951587 / 164134891 / +3.5% 0.76 / 0.78 / -3.4%
compress time/wide table cols=100 chunks=1 rows=1000 193971 / 201825.5 / -3.9% 414071 / 682924 / -39.4% 🟢 0.47 / 0.30 / +58.5%
compress time/wide table cols=100 chunks=50 rows=1000 195572 / 212963.5 / -8.2% 378987 / 647473 / -41.5% 🟢 0.52 / 0.33 / +56.9%
compress time/wide table cols=1000 chunks=1 rows=1000 2315640 / 2378154 / -2.6% 2816510 / 3354068 / -16.0% 🟢 0.82 / 0.71 / +16.0%
compress time/wide table cols=1000 chunks=50 rows=1000 2303108 / 2329366 / -1.1% 2781734 / 3098587 / -10.2% 🟢 0.83 / 0.75 / +10.1%
compress time/wide table cols=10000 chunks=1 rows=1000 24800865 / 25083436 / -1.1% 30919828 / 31540276 / -2.0% 0.80 / 0.80 / +0.9%
compress time/wide table cols=10000 chunks=50 rows=1000 25160263 / 25141529 / +0.1% 31597482 / 31655590 / -0.2% 0.80 / 0.79 / +0.3%
decompress time/arade 216934353 / 210599460 / +3.0% 217001925 / 210868957 / +2.9% 1.00 / 1.00 / +0.1%
decompress time/bimbo 411534164 / 426879820 / -3.6% 409935690 / 427068570 / -4.0% 1.00 / 1.00 / +0.4%
decompress time/cmsprovider 695141896 / 696456459 / -0.2% 695850946 / 697017099 / -0.2% 1.00 / 1.00 / -0.0%
decompress time/euro2016 85503554 / 85812540 / -0.4% 85787657 / 86100710 / -0.4% 1.00 / 1.00 / +0.0%
decompress time/food 67418929 / 65553708 / +2.8% 67519917 / 65736335 / +2.7% 1.00 / 1.00 / +0.1%
decompress time/hashtags 164132602 / 164115471 / +0.0% 164579133 / 164819342 / -0.1% 1.00 / 1.00 / +0.2%
decompress time/sparse-wide-flat-runs 18525600 / 19092552 / -3.0% 19928928 / 20294926 / -1.8% 0.93 / 0.94 / -1.2%
decompress time/sparse-wide-lists-runs 81608852 / 81250072 / +0.4% 81804879 / 81413978 / +0.5% 1.00 / 1.00 / -0.0%
decompress time/sparse-wide-lists-shuffled 81501718 / 81642746 / -0.2% 81555756 / 81798688 / -0.3% 1.00 / 1.00 / +0.1%
decompress time/sparse-wide-nested-child-nulls 32097018 / 27285941 / +17.6% 🔴 29537556 / 29784626 / -0.8% 1.09 / 0.92 / +18.6%
decompress time/sparse-wide-nested-runs 26473694 / 27270122 / -2.9% 27506345 / 27948393 / -1.6% 0.96 / 0.98 / -1.4%
decompress time/statpopgen 156704781 / 156220350 / +0.3% 157185556 / 157295831 / -0.1% 1.00 / 0.99 / +0.4%
decompress time/taxi 50792328 / 51150006 / -0.7% 50965175 / 51361699 / -0.8% 1.00 / 1.00 / +0.1%
decompress time/tpc-h l_comment canonical 188511356 / 189007356 / -0.3% 188871681 / 189306445 / -0.2% 1.00 / 1.00 / -0.0%
decompress time/tpc-h l_comment chunked 188504146 / 189014371 / -0.3% 188689281 / 189347325 / -0.3% 1.00 / 1.00 / +0.1%
decompress time/wide table cols=100 chunks=1 rows=1000 317496.5 / 315909.5 / +0.5% 348323 / 344756 / +1.0% 0.91 / 0.92 / -0.5%
decompress time/wide table cols=100 chunks=50 rows=1000 315929 / 320457 / -1.4% 341814 / 342417 / -0.2% 0.92 / 0.94 / -1.2%
decompress time/wide table cols=1000 chunks=1 rows=1000 3703300 / 3680508 / +0.6% 3806710 / 3828927 / -0.6% 0.97 / 0.96 / +1.2%
decompress time/wide table cols=1000 chunks=50 rows=1000 3672131 / 3769459 / -2.6% 3661813 / 3864562 / -5.2% 1.00 / 0.98 / +2.8%
decompress time/wide table cols=10000 chunks=1 rows=1000 36240491 / 35644852 / +1.7% 36612908 / 35840913 / +2.2% 0.99 / 0.99 / -0.5%
decompress time/wide table cols=10000 chunks=50 rows=1000 37378710 / 36328130 / +2.9% 37728583 / 37122852 / +1.6% 0.99 / 0.98 / +1.2%
vortex / arrow-ipc / bytes (1.000x ➖, 0↑ 0↓)
name bytes (PR / base / %diff)
arrow-ipc size/arade 680639722 / 680639722 / +0.0%
arrow-ipc size/bimbo 3168319202 / 3168319202 / +0.0%
arrow-ipc size/cmsprovider 2394412322 / 2394412322 / +0.0%
arrow-ipc size/euro2016 359300042 / 359300042 / +0.0%
arrow-ipc size/food 252069442 / 252069442 / +0.0%
arrow-ipc size/hashtags 700364562 / 700364562 / +0.0%
arrow-ipc size/sparse-wide-flat-runs 245220370 / 245220370 / +0.0%
arrow-ipc size/sparse-wide-lists-runs 142046306 / 142046306 / +0.0%
arrow-ipc size/sparse-wide-lists-shuffled 142870882 / 142870882 / +0.0%
arrow-ipc size/sparse-wide-nested-child-nulls 251527506 / 251527506 / +0.0%
arrow-ipc size/sparse-wide-nested-runs 251527122 / 251527122 / +0.0%
arrow-ipc size/statpopgen 1094018226 / 1094018226 / +0.0%
arrow-ipc size/taxi 474481242 / 474481242 / +0.0%
arrow-ipc size/tpc-h l_comment canonical 1021072290 / 1021072290 / +0.0%
arrow-ipc size/tpc-h l_comment chunked 1021072290 / 1021072290 / +0.0%
arrow-ipc size/wide table cols=100 chunks=1 rows=1000 1231690 / 1231690 / +0.0%
arrow-ipc size/wide table cols=100 chunks=50 rows=1000 1231690 / 1231690 / +0.0%
arrow-ipc size/wide table cols=1000 chunks=1 rows=1000 12312506 / 12312506 / +0.0%
arrow-ipc size/wide table cols=1000 chunks=50 rows=1000 12312506 / 12312506 / +0.0%
arrow-ipc size/wide table cols=10000 chunks=1 rows=1000 123192506 / 123192506 / +0.0%
arrow-ipc size/wide table cols=10000 chunks=50 rows=1000 123192506 / 123192506 / +0.0%

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Polar Signals Profiling Results

Latest Run

Status Commit Job Attempt Link
🟢 Done 7e90929 compress-bench 2 Explore Profiling Data

Powered by Polar Signals Cloud

@connortsui20 connortsui20 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read at 685d490998. Nothing here was run, so the two compression cases below come from reading the selection code and from arithmetic, not from a measurement.

The two comments that matter most are the mostly-null case in DecimalScheme::expected_compression_ratio and the sample-based estimate in DecimalDictScheme::expected_compression_ratio. The rest are cleanup, docs, and test coverage.

Comment thread vortex-btrblocks/src/schemes/decimal.rs Resolved
Comment thread vortex-btrblocks/src/schemes/decimal.rs Resolved
Comment thread vortex-compressor/src/stats/decimal.rs Resolved
Comment thread vortex-compressor/src/stats/decimal.rs Resolved
Comment thread vortex-file/src/tests.rs Resolved
robert3005 and others added 8 commits November 13, 2026 10:45
DictStrategy only accepted primitive, UTF-8 and binary columns, so decimal
columns always took the fallback path. Add a decimal dictionary encoder that
stores values in the precision's canonical values type, so chunks stored in
different widths share one dictionary, including i128 and i256.

The probe compressor never returns a top-level dictionary for decimals, so
DictStrategy decides for decimals by dictionary encoding the probe sample. It
keeps the dictionary when at most half of the values are distinct and the
compressed codes and values are no larger than the compressed sample.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018SWbEu3keyvDQYHhiRs87n
Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
Make decimal_dict_builder generic over the values type like
primitive_dict_builder, dispatching on the canonical values type in
dict_encoder_in. Pick the code width with codes_ptype_for_max_len like the
bytes builder, and add the reset_clears_dict and codes_fit_dictionary_bounds
tests the other builders have.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018SWbEu3keyvDQYHhiRs87n
Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
Count the dictionary values the value domain allows, including a null entry,
and choose the code width with the shared helper like the bytes and decimal
builders, instead of comparing the largest code inline. The chosen widths
are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018SWbEu3keyvDQYHhiRs87n
Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
Add DecimalDictScheme, which dictionary encodes low-cardinality decimals as a
Dict over decimal values, and DecimalStats, which count the distinct values
when a scheme asks for them. Wide decimals that byte parts leave canonical
under the core edition, or split into one array per part, now get a single
codes array.

DecimalScheme still returns AlwaysUse unless the distinct count shows that at
most half of the values are distinct; then it defers to sampling so the
compressor picks the smaller of byte parts and the dictionary. The
dictionary is registered first and wins ties. The integer schemes' dict-code
exclusions now include it.

With a compressor scheme deciding for decimals, DictStrategy probes them like
every other dtype, so drop its decimal-specific probe.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018SWbEu3keyvDQYHhiRs87n
Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018SWbEu3keyvDQYHhiRs87n
Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
Decimals stored in integer types now take the integer statistics pass over
their buffer, with its bounds, dense and hash distinct counters and runs.
i128 decimals take the same chunked scan without bounds. i256, which is not a
primitive integer, takes a scalar pass that counts each run once. Making the
scan generic over its bounds to cover i256 slowed the integer statistics by
about 15% with distinct counting, so it stays as it is apart from visibility.

DecimalStats now also reports the average run length. Add a decimal_stats
benchmark over each values width, with and without nulls.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018SWbEu3keyvDQYHhiRs87n
Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018SWbEu3keyvDQYHhiRs87n
Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
Compressor selection: an AlwaysUse scheme is now the default for its arrays.
It wins without being estimated when every other candidate skips; otherwise
it is sampled after the other candidates, which it has to beat, and it is
chosen when no candidate has a valid estimate, e.g. when every sample holds
only nulls. DecimalScheme returns plain AlwaysUse again instead of inferring
a competitor from the distinct count.

DecimalDictScheme estimates its ratio from the exact distinct count and runs,
like IntDictScheme, since a sample under-rates a dictionary of thousands of
distinct values. It expects the requested count, and its values keep integer
dictionaries, which shared most significant parts can use. Fix the pull rule
lists in the dict scheme docs.

DecimalStats count runs only with the distinct values, in a plain set, and
the i128 path takes i128.

The primitive and decimal dictionary builders share one core, which reserves
its codes and documents its unsafe pushes. The decimal builder converts
chunks with converted_buffer, stores values at the wider of the canonical
and the stored width, and ends the dictionary at a chunk that does not fit
instead of failing.

The dict layout probe also accepts a decimal whose single same-length child
is a dictionary, as byte parts with one dictionary-encoded part are.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018SWbEu3keyvDQYHhiRs87n
Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
@robert3005
robert3005 force-pushed the rk/decimal-dict-layout branch from 685d490 to 84ae5f1 Compare November 13, 2026 11:07
robert3005 and others added 4 commits November 13, 2026 11:31
`BytesDictBuilder` materialized a byte slice for every row, hashed it, and
compared candidate entries by reading the dictionary's value heap. Values short
enough to be held inline in their 16-byte `BinaryView` - the common shape for the
low-cardinality string columns dictionary encoding targets - need none of that:
the canonical view is the value, so it can be hashed as a single `u128` and
compared against a dictionary entry with one 16-byte comparison, never touching
a data buffer. Longer values still hash and compare their bytes, but reject a
candidate on its length first. Lookups moved from `HashTable::entry` to `find`
plus a cold insert so the hit path stays tight, and the inlined lookup is kept
out of line: inlining it into the per-row loops slowed down the longer values
that share those loops by up to a fifth.

Incoming views may carry junk in the padding after an inlined value, so probes
are canonicalized before being hashed or compared, keeping values that differ
only in their padding on a single dictionary code.

A dictionary can hold at most one entry per row, so `dict_encode_with_constraints`
now bounds the requested dictionary length by the array length. An unconstrained
encode of an 8k chunk emits `u16` codes rather than `u64`. Codes are consecutive
and every entry is referenced, so the dictionary length also determines the
narrowest code type directly, replacing the min/max scan `narrow` ran over the
codes.

Measured with `dict_compress` on a pinned core against the base commit,
interleaving the two binaries (fastest of three runs):

- `encode_varbinview_inlined`: -21% to -28%
- `encode_varbin_inlined`: -7% to -19%
- `encode_varbinview` (16-byte values): -9% to +6%, median 0%
- `encode_varbin` (16-byte values): -6% to +7%, median -2%
- `encode_primitives` (u8, f32, i64): -10% to -20%, all from the code width and narrowing changes

The bench only covered 16-byte words, which are always out of line, so it gained
inlined-word cases for both varbin and varbinview.

Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NUmMBRebyKefXiY1eKnX1W
…chemes

`dict_encode` now returns the narrowest codes that can address the dictionary,
so the `narrow` both schemes ran on the codes afterwards found nothing to do -
but still paid for the min/max scan over every code that `narrow` performs to
discover that. Drop it and hand the codes straight to the child compressor, and
say in `dict_encode_with_constraints` that its codes are already minimal so the
guarantee is discoverable from the API.

The integer and float dict schemes narrow their codes the same way, over codes
their own `dictionary_encode` already sized from the distinct count; that
redundancy predates this change and is left alone.

Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NUmMBRebyKefXiY1eKnX1W
`narrow_codes` chooses the code type from the dictionary length rather than
scanning the codes, which is only correct while every entry is referenced and
codes are handed out consecutively. Nothing covered the width boundaries, where
an off-by-one would quietly change the code type `dict_encode` produces - and
with it the bytes of files written from it, such as the compat fixtures' 255,
256 and 257-entry columns. Check `dict_encode` against `narrow` on either side
of the u8 and u16 boundaries.

Also assert the inline-padding regression test's codes and values with
`assert_arrays_eq!`, so it checks the deduplicated value rather than only the
dictionary's length.

Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NUmMBRebyKefXiY1eKnX1W
Restore BytesDictBuilder's byte-keyed lookups, and drop the code width
boundary tests and the inlined-word benchmark cases with their helper.

Keep sizing dictionary codes from the dictionary length in
dict_encode_with_constraints, the string and binary dict schemes no longer
narrowing those codes again, and the test that inline values differing only in
padding share a code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018SWbEu3keyvDQYHhiRs87n
Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
@robert3005 robert3005 changed the title Dictionary encode low-cardinality decimals Dictionary encode decimals Nov 13, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants