Repository navigation
Dictionary encode decimals - #11332
robert3005 wants to merge 12 commits into
Conversation
272bd3f to
2b16f70
Compare
09205ba to
a3481b0
Compare
The `vortex-file` `v2::file_stats_reader` tests fail intermittently. For example, `dynamic_proofs_refresh_before_reads` failed in a full `vortex-file` lib run while checking #11332. Each test called `SESSION.clone().with_handle(handle)` on one static session. Clones of a `VortexSession` share their state, and `with_handle` stores the runtime handle in that shared state. Tests running at the same time therefore overwrote each other's handle, and a test could drive its reads with another test's runtime, including one that had already shut down. Some readers in these tests were also built from the static `SESSION` directly, not from the test's own session. Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
91be5e3 to
7e90929
Compare
Merging this PR will regress 2 benchmarks2 warnings
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing Footnotes
|
Benchmarks: Compression 📖Commits: PR vortex / vortex-file-compressed / ns (1.010x ➖, 0↑ 0↓)
vortex / vortex-file-compressed / bytes (1.000x ➖, 0↑ 0↓)
vortex / parquet / ns (1.000x ➖, 0↑ 0↓)
vortex / parquet / bytes (1.000x ➖, 0↑ 0↓)
vortex / arrow-ipc / ns (0.995x ➖, 0↑ 1↓)
vortex / arrow-ipc / bytes (1.000x ➖, 0↑ 0↓)
|
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
connortsui20
left a comment
There was a problem hiding this comment.
Read at 685d490998. Nothing here was run, so the two compression cases below come from reading the selection code and from arithmetic, not from a measurement.
The two comments that matter most are the mostly-null case in DecimalScheme::expected_compression_ratio and the sample-based estimate in DecimalDictScheme::expected_compression_ratio. The rest are cleanup, docs, and test coverage.
DictStrategy only accepted primitive, UTF-8 and binary columns, so decimal columns always took the fallback path. Add a decimal dictionary encoder that stores values in the precision's canonical values type, so chunks stored in different widths share one dictionary, including i128 and i256. The probe compressor never returns a top-level dictionary for decimals, so DictStrategy decides for decimals by dictionary encoding the probe sample. It keeps the dictionary when at most half of the values are distinct and the compressed codes and values are no larger than the compressed sample. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbEu3keyvDQYHhiRs87n Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
Make decimal_dict_builder generic over the values type like primitive_dict_builder, dispatching on the canonical values type in dict_encoder_in. Pick the code width with codes_ptype_for_max_len like the bytes builder, and add the reset_clears_dict and codes_fit_dictionary_bounds tests the other builders have. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbEu3keyvDQYHhiRs87n Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
Count the dictionary values the value domain allows, including a null entry, and choose the code width with the shared helper like the bytes and decimal builders, instead of comparing the largest code inline. The chosen widths are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbEu3keyvDQYHhiRs87n Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
Add DecimalDictScheme, which dictionary encodes low-cardinality decimals as a Dict over decimal values, and DecimalStats, which count the distinct values when a scheme asks for them. Wide decimals that byte parts leave canonical under the core edition, or split into one array per part, now get a single codes array. DecimalScheme still returns AlwaysUse unless the distinct count shows that at most half of the values are distinct; then it defers to sampling so the compressor picks the smaller of byte parts and the dictionary. The dictionary is registered first and wins ties. The integer schemes' dict-code exclusions now include it. With a compressor scheme deciding for decimals, DictStrategy probes them like every other dtype, so drop its decimal-specific probe. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbEu3keyvDQYHhiRs87n Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbEu3keyvDQYHhiRs87n Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
Decimals stored in integer types now take the integer statistics pass over their buffer, with its bounds, dense and hash distinct counters and runs. i128 decimals take the same chunked scan without bounds. i256, which is not a primitive integer, takes a scalar pass that counts each run once. Making the scan generic over its bounds to cover i256 slowed the integer statistics by about 15% with distinct counting, so it stays as it is apart from visibility. DecimalStats now also reports the average run length. Add a decimal_stats benchmark over each values width, with and without nulls. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbEu3keyvDQYHhiRs87n Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbEu3keyvDQYHhiRs87n Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
Compressor selection: an AlwaysUse scheme is now the default for its arrays. It wins without being estimated when every other candidate skips; otherwise it is sampled after the other candidates, which it has to beat, and it is chosen when no candidate has a valid estimate, e.g. when every sample holds only nulls. DecimalScheme returns plain AlwaysUse again instead of inferring a competitor from the distinct count. DecimalDictScheme estimates its ratio from the exact distinct count and runs, like IntDictScheme, since a sample under-rates a dictionary of thousands of distinct values. It expects the requested count, and its values keep integer dictionaries, which shared most significant parts can use. Fix the pull rule lists in the dict scheme docs. DecimalStats count runs only with the distinct values, in a plain set, and the i128 path takes i128. The primitive and decimal dictionary builders share one core, which reserves its codes and documents its unsafe pushes. The decimal builder converts chunks with converted_buffer, stores values at the wider of the canonical and the stored width, and ends the dictionary at a chunk that does not fit instead of failing. The dict layout probe also accepts a decimal whose single same-length child is a dictionary, as byte parts with one dictionary-encoded part are. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbEu3keyvDQYHhiRs87n Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
685d490 to
84ae5f1
Compare
`BytesDictBuilder` materialized a byte slice for every row, hashed it, and compared candidate entries by reading the dictionary's value heap. Values short enough to be held inline in their 16-byte `BinaryView` - the common shape for the low-cardinality string columns dictionary encoding targets - need none of that: the canonical view is the value, so it can be hashed as a single `u128` and compared against a dictionary entry with one 16-byte comparison, never touching a data buffer. Longer values still hash and compare their bytes, but reject a candidate on its length first. Lookups moved from `HashTable::entry` to `find` plus a cold insert so the hit path stays tight, and the inlined lookup is kept out of line: inlining it into the per-row loops slowed down the longer values that share those loops by up to a fifth. Incoming views may carry junk in the padding after an inlined value, so probes are canonicalized before being hashed or compared, keeping values that differ only in their padding on a single dictionary code. A dictionary can hold at most one entry per row, so `dict_encode_with_constraints` now bounds the requested dictionary length by the array length. An unconstrained encode of an 8k chunk emits `u16` codes rather than `u64`. Codes are consecutive and every entry is referenced, so the dictionary length also determines the narrowest code type directly, replacing the min/max scan `narrow` ran over the codes. Measured with `dict_compress` on a pinned core against the base commit, interleaving the two binaries (fastest of three runs): - `encode_varbinview_inlined`: -21% to -28% - `encode_varbin_inlined`: -7% to -19% - `encode_varbinview` (16-byte values): -9% to +6%, median 0% - `encode_varbin` (16-byte values): -6% to +7%, median -2% - `encode_primitives` (u8, f32, i64): -10% to -20%, all from the code width and narrowing changes The bench only covered 16-byte words, which are always out of line, so it gained inlined-word cases for both varbin and varbinview. Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NUmMBRebyKefXiY1eKnX1W
…chemes `dict_encode` now returns the narrowest codes that can address the dictionary, so the `narrow` both schemes ran on the codes afterwards found nothing to do - but still paid for the min/max scan over every code that `narrow` performs to discover that. Drop it and hand the codes straight to the child compressor, and say in `dict_encode_with_constraints` that its codes are already minimal so the guarantee is discoverable from the API. The integer and float dict schemes narrow their codes the same way, over codes their own `dictionary_encode` already sized from the distinct count; that redundancy predates this change and is left alone. Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NUmMBRebyKefXiY1eKnX1W
`narrow_codes` chooses the code type from the dictionary length rather than scanning the codes, which is only correct while every entry is referenced and codes are handed out consecutively. Nothing covered the width boundaries, where an off-by-one would quietly change the code type `dict_encode` produces - and with it the bytes of files written from it, such as the compat fixtures' 255, 256 and 257-entry columns. Check `dict_encode` against `narrow` on either side of the u8 and u16 boundaries. Also assert the inline-padding regression test's codes and values with `assert_arrays_eq!`, so it checks the deduplicated value rather than only the dictionary's length. Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NUmMBRebyKefXiY1eKnX1W
Restore BytesDictBuilder's byte-keyed lookups, and drop the code width boundary tests and the inlined-word benchmark cases with their helper. Keep sizing dictionary codes from the dictionary length in dict_encode_with_constraints, the string and binary dict schemes no longer narrowing those codes again, and the test that inline values differing only in padding share a code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbEu3keyvDQYHhiRs87n Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
Summary
Low-cardinality decimal columns could not be dictionary encoded, either by the compressor or as a dict layout.
The worst case was wide decimals (i128/i256) under the core edition. There, the byte parts scheme runs in v1 mode and leaves them canonical, so they were written uncompressed however few distinct values they had. Without editions, byte parts split them into up to four arrays, and each part needs its own dictionary and codes.
This PR adds a decimal dictionary scheme to the compressor and lets
DictStrategywrite decimal columns as dict layouts.Changes
DecimalDictScheme(vortex-compressor/src/builtins/dict/decimal.rs) producesDict(codes, decimal values). It mirrorsFloatDictScheme:IntDictSchemefrom its codes and from its values' byte parts.DecimalStats(vortex-compressor/src/stats/decimal.rs) holds the null and value counts, and counts distinct values only when a scheme requests it.DecimalScheme(byte parts) still returnsAlwaysUse, unless the distinct count shows at most half of the values are distinct. Then it defers to sampling, so the compressor keeps whichever of byte parts and the dictionary is smaller. The count only exists whenDecimalDictSchemeis eligible, so without it decimals compress exactly as before.