Repository navigation
fix(fsst): train the symbol table as Rust's fsst-rs does (#583) - #698
Merged
Merged
Conversation
The trainer drew its own sample (java.util.Random, always sampled, first-N chunks per generation) and built candidates its own way. It now follows fsst-rs v0.6.0 (the version vortex-jni 0.90.0 uses): make_sample with fsst_hash(4637947) (a corpus under 16 KiB is taken whole, empty rows included), generations 8/38/68/98/128 replaying the lines whose fsst_hash(i) & 127 is within the fraction, compress-count with the extra count of a match's first byte, candidates deduped by symbol with their gains summed, merges only from codes that pass the count floor and never past 8 bytes, none in the last generation, the final prune only for a corpus taken whole, symbols measured by their last non-zero byte, and Rust's BinaryHeap (sift for sift) for the cut. The wire order of the codes is finalize's (2-byte without a longer suffix, 2-byte with one in reverse, 3-8 bytes, 1 byte). CompressorBuilder#seed is gone. Equal-gain candidates still pop in the order they were counted, where Rust's is its hash map's iteration order (which depends on the CPU's SIMD width), so a table is not byte for byte Rust's but shares ~90% of its symbols and a code stream within 1%; FsstTrainingParityIntegrationTest holds that, and NullableTextInteropIntegrationTest holds a column (few distinct values) whose whole encoding tree now equals Rust's, FSST offsets as vortex.sequence included. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011fRbYsrGtpKHp9APhU8qDV
dfa1
force-pushed
the
fix/583-fsst-training
branch
from
November 7, 2026 22:07
4fcc33c to
39fd18e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #583 (does not close it).
The trainer drew its own sample and built candidates its own way. It now follows
fsst-rsv0.6.0, the version vortex-jni 0.90.0 uses:make_sample(fixedfsst_hashsequence; under 16 KiB the corpus is taken whole, empty rows included), generations 8/38/68/98/128 replaying the linesfsst_hash(i) & 127selects, compress-count with the first-byte extension, candidates deduped by symbol with gains summed, merges only from codes passing the count floor and never past 8 bytes, none in the last generation, the final prune only for a corpus taken whole, symbols measured by their last non-zero byte, Rust'sBinaryHeapfor the cut, andfinalize's code order.CompressorBuilder#seedis removed.Result against vortex-jni: the
few distinctnullable-text column's whole encoding tree now equals Rust's, FSST offsets asvortex.sequenceincluded (NullableTextInteropIntegrationTest).Not byte-for-byte, by decision. Equal-gain candidates pop in the order they were counted; Rust pops them in its
FxHashMapiteration order, which depends on the CPU's SIMD width, so Rust's own table differs between x86 and arm. About 90% of the symbols and a code stream within 1% are Rust's (FsstTrainingParityIntegrationTest, thresholds 80% / 1.5%; measured on arm64, the ubuntu job checks x86). Recorded indocs/compatibility.md.Known gap. A nullable column with nulls still trains on its empty placeholder rows, where Rust trains on valid rows only; that needs validity inside the encoding (#650).
🤖 Generated with Claude Code
https://claude.ai/code/session_011fRbYsrGtpKHp9APhU8qDV