You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Writer: string columns written as per-chunk dicts where Rust uses its dict layout (up to 2x larger) #650
Rust's flats are mostly bare fastlanes.bitpacked codes: its dict layout shares one values pool across chunks. Java writes a per-chunk vortex.dict encoding (each chunk repeating its own values), and nullable ones under vortex.masked. ~+1.3 GB across these columns.
To narrow down: why Java's global dict layout (WriteOptionsglobalDict, default on) does not admit these columns — the admission probe / globalDictMaxRetainedBytes against Rust's DictStrategy.
Layout half fixed in #679 (merged once CI is green): a text column is admitted by its first 8,192 rows (at most half distinct) and starts a new dictionary at 65,535 entries or 1 MiB, as Rust's DictStrategy does; the 32,768-distinct cap is gone. Interop test DictLayoutStringInteropIntegrationTest compares dict-layout counts with vortex-jni.
Still open here:
Nullable text: when no layout is admitted, a nullable Utf8 chunk is still masked(dict(...)); Rust keeps the validity inside the dict (null as one more entry, invalid in the values). This is the bi-generico rows of the table.
Scheme choice inside the values / probe: Java's cascade picks OnPair where Rust's StringDictScheme wins on the same 8,192-row sample, so the probe (probeCompressesToDict) is applied to numbers only; strings use the distinct-count check alone.
Not yet measured on the real clickbench-hits / bi-generico columns, only on synthetic data (pools of 100–150,000 values and a sliding window).
Slice merged in #692: a nullable text column without a null is written without validity (no vortex.masked), as Rust writes an all-valid string array. NullableTextInteropIntegrationTest compares the encoding tree with vortex-jni for the no-null unique shape, and has vortex-jni read every shape (full, filtered, row-range, compact).
What the vortex-jni trees show is still different (9 shapes, 1 matches today):
FSST uncompressed-lengths child is vortex.sequence in Rust, vortex.primitive in Java (dict values of few distinct strings).
Many distinct strings: Rust writes plain vortex.onpair chunks where Java wraps them in a vortex.dict (the string probe, probeCompressesToDict, is applied to numbers only).
Node stats on the FSST child's fastlanes.bitpacked (is_sorted, min/max) differ.
Struct fields and list elements of nullable text keep their previous path; no cross-check yet.
Re-measured on current main: every column in this ticket is at parity
Setup: the same as in #651. Each column was sliced into its own Parquet file and written by Rust (vortex-data0.90.0, vortex-jni's Rust release) and by vortex import with the compressor JFR events on. The corpus files in the table above were built by vortex_data 0.86.1.
column
rows
Rust 0.90
Java (main)
clickbench Title
10M
310,399,832
310,278,294 (0.999x)
clickbench URL
10M
190,268,532
190,121,260 (0.999x)
clickbench Referer
10M
232,531,992
232,431,158 (0.999x)
Generico PrimeraLinea
5M
12,655,648
12,639,650 (0.999x), encoding tree identical
Generico Concatenar 2
5M
13,707,252
13,755,246 (1.003x)
Generico Concatenar 1
5M
13,707,252
13,755,246 (1.003x)
Generico VER ANUNCIO
5M
2,691,700
2,690,054 (0.999x)
Generico Producto
5M
15,348
14,086 (0.92x)
The fixes were a1df400 (a text column starts a new dictionary instead of giving up its dict layout) and 8c577ba (no vortex.masked around a nullable text column without nulls).
What JFR shows
The Winner events at the root of each segment (empty cascadePath) give the shape Java now writes. For Title at 1M rows:
101x fastlanes.bitpacked <- bare dict-layout codes, one per chunk
101x vortex.onpair <- the dict layouts' value pools
2x vortex.dict, 1x vortex.constant
This is Rust's dict-layout shape ("2408x bitpacked + 1629x onpair" in the table above), and no longer the per-chunk dict(bitpacked, onpair(...)) this ticket measured. URL (49 + 49) and Referer (43 + 43) look the same. Text columns skip the cascade probe (admission is Rust's "at most half the first 8192 rows distinct"), so unlike #651 there are no segment = -1 probe events here. The evidence is the root winners plus the identical PrimeraLinea tree.
Every column above is at parity, so this ticket looks fixed. A full-size (100M-row) import has not been checked.
Activity
fix(writer): a text column starts a new dictionary instead of giving …
dfa1 commentedon Nov 6, 2026
Layout half fixed in #679 (merged once CI is green): a text column is admitted by its first 8,192 rows (at most half distinct) and starts a new dictionary at 65,535 entries or 1 MiB, as Rust's
DictStrategydoes; the 32,768-distinct cap is gone. Interop testDictLayoutStringInteropIntegrationTestcompares dict-layout counts with vortex-jni.Still open here:
masked(dict(...)); Rust keeps the validity inside the dict (null as one more entry, invalid in the values). This is thebi-genericorows of the table.StringDictSchemewins on the same 8,192-row sample, so the probe (probeCompressesToDict) is applied to numbers only; strings use the distinct-count check alone.clickbench-hits/bi-genericocolumns, only on synthetic data (pools of 100–150,000 values and a sliding window).fix(writer): a text column starts a new dictionary instead of giving …
fix(writer): a text column starts a new dictionary instead of giving …
fix(writer): a nullable text column without a null is written without…
dfa1 commentedon Nov 7, 2026
Slice merged in #692: a nullable text column without a null is written without validity (no
vortex.masked), as Rust writes an all-valid string array.NullableTextInteropIntegrationTestcompares the encoding tree with vortex-jni for the no-null unique shape, and has vortex-jni read every shape (full, filtered, row-range, compact).What the vortex-jni trees show is still different (9 shapes, 1 matches today):
vortex.masked(<text scheme>, validity). Rust keeps the validity inside the text encoding: avortex.boolchild ofvortex.fsst/vortex.onpair/ varbin, and in a dict the null is one more pool entry. Needs FSST, OnPair, VarBin and Dict to take a nullable carrier, as Writer: nullable columns go through vortex.masked, so the cascade ignores validity cost and picks differently from Rust #604 did for primitives.vortex.sequencein Rust,vortex.primitivein Java (dict values of few distinct strings).vortex.onpairchunks where Java wraps them in avortex.dict(the string probe,probeCompressesToDict, is applied to numbers only).fastlanes.bitpacked(is_sorted, min/max) differ.dfa1 commentedon Nov 8, 2026
Re-measured on current
main: every column in this ticket is at paritySetup: the same as in #651. Each column was sliced into its own Parquet file and written by Rust (
vortex-data0.90.0, vortex-jni's Rust release) and byvortex importwith the compressor JFR events on. The corpus files in the table above were built byvortex_data0.86.1.TitleURLRefererPrimeraLineaConcatenar 2Concatenar 1VER ANUNCIOProductoThe fixes were a1df400 (a text column starts a new dictionary instead of giving up its dict layout) and 8c577ba (no
vortex.maskedaround a nullable text column without nulls).What JFR shows
The
Winnerevents at the root of each segment (emptycascadePath) give the shape Java now writes. ForTitleat 1M rows:This is Rust's dict-layout shape ("2408x bitpacked + 1629x onpair" in the table above), and no longer the per-chunk
dict(bitpacked, onpair(...))this ticket measured.URL(49 + 49) andReferer(43 + 43) look the same. Text columns skip the cascade probe (admission is Rust's "at most half the first 8192 rows distinct"), so unlike #651 there are nosegment = -1probe events here. The evidence is the root winners plus the identicalPrimeraLineatree.Every column above is at parity, so this ticket looks fixed. A full-size (100M-row) import has not been checked.
fix(reader): decode a struct field with its declared nullability (#650)
fix(writer): nullable text keeps its validity in its encoding, as Rus…