Skip to content

Keep IntDictScheme for chunks DictStrategy rejects - #11331

Merged
robert3005 merged 3 commits into
developfrom
rk/dict-fallback-int-dict
Nov 9, 2026
Merged

robert3005 merged 3 commits into
developfrom
rk/dict-fallback-int-dict

Conversation

@robert3005

@robert3005 robert3005 commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

The default write strategy removed IntDictScheme from the one compressor shared by DictStrategy's codes path and its fallback path. The exclusion is only needed for dictionary codes, so they aren't dictionary-encoded twice. But it also applied to every column DictStrategy rejects, and those could never use integer dictionaries, even nested inside another encoding. This hits decimals hardest: they always take the fallback, so a DecimalByteParts most significant part could never be dict-encoded.

robert3005 and others added 2 commits November 9, 2026 11:36
The default write strategy excluded IntDictScheme from the compressor shared
by DictStrategy's codes and fallback paths. Columns that DictStrategy rejects,
including every decimal column, could therefore never use integer
dictionaries, e.g. for a DecimalByteParts most significant part.

Give the fallback its own pipeline with the full compressor and keep the
IntDictScheme exclusion only for dictionary codes. Add a roundtrip test for a
decimal whose most significant part is dict-encoded inside byte parts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018SWbEu3keyvDQYHhiRs87n
Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
Clones of a VortexSession share state, so both rstest cases registered
LayoutSession on the module's shared static session and whichever case ran
second panicked. The registration also leaked into the other tests that use
that session, depending on run order. Build a fresh session in the test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018SWbEu3keyvDQYHhiRs87n
Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
Comment thread vortex-file/src/strategy.rs Resolved

@connortsui20 connortsui20 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

left a style comment

@connortsui20
connortsui20 marked this pull request as ready for review November 9, 2026 16:14
Split the configured compressor into codes and full compressors in
CompressorConfig::into_compressors, and build each coalesce-then-compress
pipeline with coalesce_then_compress, so build calls it once for dict codes
and once for the fallback.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018SWbEu3keyvDQYHhiRs87n
Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
@robert3005 robert3005 added changelog/fix A bug fix changelog/feature A new feature and removed changelog/fix A bug fix labels Nov 9, 2026
@robert3005
robert3005 merged commit ce2b6d0 into develop Nov 9, 2026
130 of 132 checks passed
@robert3005
robert3005 deleted the rk/dict-fallback-int-dict branch November 9, 2026 17:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants