Skip to content

fix(fsst): train the symbol table as Rust's fsst-rs does (#583) - #698

Merged
dfa1 merged 1 commit into
mainfrom
fix/583-fsst-training
Nov 7, 2026
Merged

dfa1 merged 1 commit into
mainfrom
fix/583-fsst-training

Conversation

@dfa1

@dfa1 dfa1 commented Nov 7, 2026

Copy link
Copy Markdown
Owner

Part of #583 (does not close it).

The trainer drew its own sample and built candidates its own way. It now follows fsst-rs v0.6.0, the version vortex-jni 0.90.0 uses: make_sample (fixed fsst_hash sequence; under 16 KiB the corpus is taken whole, empty rows included), generations 8/38/68/98/128 replaying the lines fsst_hash(i) & 127 selects, compress-count with the first-byte extension, candidates deduped by symbol with gains summed, merges only from codes passing the count floor and never past 8 bytes, none in the last generation, the final prune only for a corpus taken whole, symbols measured by their last non-zero byte, Rust's BinaryHeap for the cut, and finalize's code order. CompressorBuilder#seed is removed.

Result against vortex-jni: the few distinct nullable-text column's whole encoding tree now equals Rust's, FSST offsets as vortex.sequence included (NullableTextInteropIntegrationTest).

Not byte-for-byte, by decision. Equal-gain candidates pop in the order they were counted; Rust pops them in its FxHashMap iteration order, which depends on the CPU's SIMD width, so Rust's own table differs between x86 and arm. About 90% of the symbols and a code stream within 1% are Rust's (FsstTrainingParityIntegrationTest, thresholds 80% / 1.5%; measured on arm64, the ubuntu job checks x86). Recorded in docs/compatibility.md.

Known gap. A nullable column with nulls still trains on its empty placeholder rows, where Rust trains on valid rows only; that needs validity inside the encoding (#650).

🤖 Generated with Claude Code

https://claude.ai/code/session_011fRbYsrGtpKHp9APhU8qDV

The trainer drew its own sample (java.util.Random, always sampled, first-N chunks per
generation) and built candidates its own way. It now follows fsst-rs v0.6.0 (the version
vortex-jni 0.90.0 uses): make_sample with fsst_hash(4637947) (a corpus under 16 KiB is taken
whole, empty rows included), generations 8/38/68/98/128 replaying the lines whose
fsst_hash(i) & 127 is within the fraction, compress-count with the extra count of a match's
first byte, candidates deduped by symbol with their gains summed, merges only from codes that
pass the count floor and never past 8 bytes, none in the last generation, the final prune only
for a corpus taken whole, symbols measured by their last non-zero byte, and Rust's BinaryHeap
(sift for sift) for the cut. The wire order of the codes is finalize's (2-byte without a longer
suffix, 2-byte with one in reverse, 3-8 bytes, 1 byte). CompressorBuilder#seed is gone.

Equal-gain candidates still pop in the order they were counted, where Rust's is its hash map's
iteration order (which depends on the CPU's SIMD width), so a table is not byte for byte Rust's
but shares ~90% of its symbols and a code stream within 1%; FsstTrainingParityIntegrationTest
holds that, and NullableTextInteropIntegrationTest holds a column (few distinct values) whose
whole encoding tree now equals Rust's, FSST offsets as vortex.sequence included.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011fRbYsrGtpKHp9APhU8qDV
@dfa1
dfa1 force-pushed the fix/583-fsst-training branch from 4fcc33c to 39fd18e Compare November 7, 2026 22:07
@dfa1
dfa1 merged commit 32c93cd into main Nov 7, 2026
6 checks passed
@dfa1
dfa1 deleted the fix/583-fsst-training branch November 7, 2026 22:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant