Repository navigation
Shrink FSST and OnPair codes when compression leaves large spare capacity - #11434
shu-chen-dot-ca wants to merge 2 commits into
Conversation
|
do we need this for onpair? can you rebase we have added new file size checks |
Merging this PR will improve performance by 10.58%2 warnings
Performance Changes
Tip Curious why performance improved? Comment Comparing Footnotes
|
…city Signed-off-by: Shu Chen <338845574+shu-chen-dot-ca@users.noreply.github.com>
Signed-off-by: Shu Chen <338845574+shu-chen-dot-ca@users.noreply.github.com>
|
Rebased onto the latest Yes, OnPair has the same problem. Checks on the rebased branch: |
9ef884b to
cdfcca4
Compare
|
Thanks for tracking this down! Rather than copying into an exactly sized buffer, could you use the new The change in self.codes.shrink_to_fit();
let codes_bytes = self.codes.freeze();
Generated by Claude Code |
|
Makes sense, no reason to copy when |
Summary
fsst_compressbuilds the codes buffer by appending one batch at a time, andBufferMutgrowth at least doubles the capacity.freezekeeps that allocation, so a compressed FSST array could hold up to about 1.9x itsnbytes(), depending on where the last batch landed relative to the last doubling. OnPair has the same gap:onpair::compress_rowsgrows the codesVecfrom empty, andBuffer::from(Vec)keeps its whole allocation.Changes
FsstSink::finish, when the codes buffer's spare capacity is more than a quarter of its length, copy the codes into an exactly sized buffer (same alignment, the context's allocator) before freezing. Otherwise the buffer is frozen as before.onpair_compress, callshrink_to_fiton the codes under the same condition, before they become aBuffer. The dictionary bytes were already copied into an exactly sized buffer.codes_buffer_has_little_spare_capacityrstestregression to each crate, over 8,192 random hex strings of several lengths. Each one unwraps the uniquely owned codes buffer withtry_into_mutand checks that the spare capacity is at most a quarter of the length. Both fail ondevelopand pass with this change.The issue's reproducer (live heap left behind by compression, divided by
nbytes()), pointed at this branch:developdevelopChecks run:
cargo nextest run -p vortex-fsst -p vortex-onpair --all-features(158 passed, 1 skipped) andcargo test --doc -p vortex-fsst -p vortex-onpair --all-featurescargo clippy -p vortex-fsst -p vortex-onpair --all-targets --all-features -- -D warningscargo +nightly-2026-11-13 fmt -p vortex-fsst -p vortex-onpair -- --checkBenchmarks were not run locally; the extra copy or shrink happens at most once per compression call, on the compressed output.
AI assistance: this change was prepared with help from an AI coding agent, and reviewed and submitted by me.