Skip to content

Writer: string columns written as per-chunk dicts where Rust uses its dict layout (up to 2x larger) #650

Closed

Description

@dfa1

vortex-java's Parquet import vs the vortex-jni-written file, data bytes:

dataset column Rust Java Rust tree (most common) Java tree (most common)
bi-generico PrimeraLinea (Utf8?) 293.6 MB 585.8 MB (2.00x) 302x bitpacked + 198x fsst 1869x masked(dict(bitpacked, fsst(...)), constant)
bi-generico Concatenar 2 329.1 MB 395.9 MB (1.20x) 879x bitpacked + 279x onpair 3156x masked(dict(bitpacked, onpair(...)), constant)
bi-generico VER ANUNCIO, Concatenar 1, Producto ~290 / 290 / 212 MB 1.18–1.20x same shape same shape
clickbench-hits Title (Utf8) 1,101 MB 1,523 MB (1.38x) 2408x bitpacked + 1629x onpair 1335x dict(bitpacked, onpair(...))
clickbench-hits URL 1,933 MB 2,591 MB (1.34x) 4336x bitpacked + 2036x onpair 1852x onpair
clickbench-hits Referer 1,635 MB 1,836 MB (1.12x) 3243x bitpacked 1788x dict(bitpacked, onpair(...))

Rust's flats are mostly bare fastlanes.bitpacked codes: its dict layout shares one values pool across chunks. Java writes a per-chunk vortex.dict encoding (each chunk repeating its own values), and nullable ones under vortex.masked. ~+1.3 GB across these columns.

To narrow down: why Java's global dict layout (WriteOptions globalDict, default on) does not admit these columns — the admission probe / globalDictMaxRetainedBytes against Rust's DictStrategy.

Activity

  1. dfa1 commented on Nov 6, 2026

    @dfa1
    OwnerAuthor

    Layout half fixed in #679 (merged once CI is green): a text column is admitted by its first 8,192 rows (at most half distinct) and starts a new dictionary at 65,535 entries or 1 MiB, as Rust's DictStrategy does; the 32,768-distinct cap is gone. Interop test DictLayoutStringInteropIntegrationTest compares dict-layout counts with vortex-jni.

    Still open here:

    • Nullable text: when no layout is admitted, a nullable Utf8 chunk is still masked(dict(...)); Rust keeps the validity inside the dict (null as one more entry, invalid in the values). This is the bi-generico rows of the table.
    • Scheme choice inside the values / probe: Java's cascade picks OnPair where Rust's StringDictScheme wins on the same 8,192-row sample, so the probe (probeCompressesToDict) is applied to numbers only; strings use the distinct-count check alone.
    • Not yet measured on the real clickbench-hits / bi-generico columns, only on synthetic data (pools of 100–150,000 values and a sliding window).
  2. dfa1 commented on Nov 7, 2026

    @dfa1
    OwnerAuthor

    Slice merged in #692: a nullable text column without a null is written without validity (no vortex.masked), as Rust writes an all-valid string array. NullableTextInteropIntegrationTest compares the encoding tree with vortex-jni for the no-null unique shape, and has vortex-jni read every shape (full, filtered, row-range, compact).

    What the vortex-jni trees show is still different (9 shapes, 1 matches today):

    • Nullable text with nulls is still vortex.masked(<text scheme>, validity). Rust keeps the validity inside the text encoding: a vortex.bool child of vortex.fsst / vortex.onpair / varbin, and in a dict the null is one more pool entry. Needs FSST, OnPair, VarBin and Dict to take a nullable carrier, as Writer: nullable columns go through vortex.masked, so the cascade ignores validity cost and picks differently from Rust #604 did for primitives.
    • FSST uncompressed-lengths child is vortex.sequence in Rust, vortex.primitive in Java (dict values of few distinct strings).
    • Many distinct strings: Rust writes plain vortex.onpair chunks where Java wraps them in a vortex.dict (the string probe, probeCompressesToDict, is applied to numbers only).
    • Node stats on the FSST child's fastlanes.bitpacked (is_sorted, min/max) differ.
    • Struct fields and list elements of nullable text keep their previous path; no cross-check yet.
  3. dfa1 commented on Nov 8, 2026

    @dfa1
    OwnerAuthor

    Re-measured on current main: every column in this ticket is at parity

    Setup: the same as in #651. Each column was sliced into its own Parquet file and written by Rust (vortex-data 0.90.0, vortex-jni's Rust release) and by vortex import with the compressor JFR events on. The corpus files in the table above were built by vortex_data 0.86.1.

    column rows Rust 0.90 Java (main)
    clickbench Title 10M 310,399,832 310,278,294 (0.999x)
    clickbench URL 10M 190,268,532 190,121,260 (0.999x)
    clickbench Referer 10M 232,531,992 232,431,158 (0.999x)
    Generico PrimeraLinea 5M 12,655,648 12,639,650 (0.999x), encoding tree identical
    Generico Concatenar 2 5M 13,707,252 13,755,246 (1.003x)
    Generico Concatenar 1 5M 13,707,252 13,755,246 (1.003x)
    Generico VER ANUNCIO 5M 2,691,700 2,690,054 (0.999x)
    Generico Producto 5M 15,348 14,086 (0.92x)

    The fixes were a1df400 (a text column starts a new dictionary instead of giving up its dict layout) and 8c577ba (no vortex.masked around a nullable text column without nulls).

    What JFR shows

    The Winner events at the root of each segment (empty cascadePath) give the shape Java now writes. For Title at 1M rows:

    101x fastlanes.bitpacked   <- bare dict-layout codes, one per chunk
    101x vortex.onpair         <- the dict layouts' value pools
      2x vortex.dict, 1x vortex.constant
    

    This is Rust's dict-layout shape ("2408x bitpacked + 1629x onpair" in the table above), and no longer the per-chunk dict(bitpacked, onpair(...)) this ticket measured. URL (49 + 49) and Referer (43 + 43) look the same. Text columns skip the cascade probe (admission is Rust's "at most half the first 8192 rows distinct"), so unlike #651 there are no segment = -1 probe events here. The evidence is the root winners plus the identical PrimeraLinea tree.

    Every column above is at parity, so this ticket looks fixed. A full-size (100M-row) import has not been checked.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions