Repository navigation
Extract from Variants in bulk and push variant_get into the scan - #11505
Draft
robert3005 wants to merge 3 commits into
Draft
robert3005 wants to merge 3 commits into
robert3005 wants to merge 3 commits into
Conversation
- vortex-datafusion converts `variant_get(variant, path, type)` to the Vortex expression, so that the scan extracts the path, and folds filter equalities into the scan projection. - `variant_get` brings chunked, masked and other lazy inputs to the Variant encodings before it falls back to extracting row by row. - A shredded Variant child holds the rest of a path. The rows that the shredded child does not hold are extracted from the core storage on their own and scattered back, so that the core storage is not read when none of them holds a value. - Parquet Variant storage extracts strings and integers from unshredded values in place, reads fixed-width integers with a single load, and resolves field names to ids once per metadata. - Add `decode_unshredded_storage`, which decodes the storage of an unshredded Variant once for many extractions. - Add the `variant_pack` expression, which packs core storage and a shredded child into a Variant. `variant_get` reads a shredded path from its shredded child without reading the core storage. - Add `InferredShredding`, which infers a shredding schema from a chunk of unshredded Variants and shreds later chunks by it. - `CascadingCompressor` can compress the storage of Variant arrays with another compressor. The default BtrBlocks preset uses one that keeps the binary Zstd scheme, since unshredded values are opaque binary blobs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Robert Kruszewski <github@robertk.io>
robert3005
marked this pull request as draft
November 16, 2026 17:51
The in-place extraction turned the whole column down at the first row whose value it could not read as is, such as a JSON null or an integer at a string path. The general kernel then extracted every row again. Such a value now converts with the `Variant` method that the general kernel converts it with, `as_string` or `as_int64`, and a value that does not convert is a null. Objects and arrays convert to neither, so they are null without being decoded. The extraction falls back to the general kernel only for bytes that do not decode. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Robert Kruszewski <github@robertk.io>
The in-place extraction covered strings and 64-bit integers. A struct or a list target went to the general kernel, which builds an Arrow array of the storage and a `Variant` over every row. The extraction is now a tree of builders, one for each node of the target dtype. A struct reads the object at the path once and each of its fields from the object's field of the same name, so it extracts several paths in one pass. A list reads each element of the array at the path. A value that is not an object or an array is a null, as it is for the general kernel. Field names of every struct in the target resolve to dictionary ids once per metadata, as the names of the path do. Structs and lists are covered when their fields and elements are nullable and covered themselves. A struct without fields is left to the general kernel. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Robert Kruszewski <github@robertk.io>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Queries over a Variant column extracted every path row by row, after reading the whole column. This PR makes the extraction happen in the Vortex scan and in bulk, and adds the pieces that the Variant layout in the next PR builds on.
It is the second PR of the stack in #11390, after #11499. The next PRs add an unstable Variant layout and the JSONBench suite.
Changes
Pushdown
variant_get(variant, path, type)to the Vortex expression, so that the scan extracts the path. Filter equalities fold into the scan projection.variant_getbrings chunked, masked and other lazy inputs to the Variant encodings before it falls back to extracting row by row. The canonicalization depth is bounded, so an input that canonicalizes to itself cannot recurse forever.Extraction
Variantmethod as the general kernel (as_stringoras_int64). A value that does not convert is a null. Before, one such value sent the whole column to the general kernel.decode_unshredded_storagedecodes the storage of an unshredded Variant once for many extractions.Building blocks for the Variant layout
variant_packexpression packs core storage and a shredded child into a Variant.variant_getovervariant_packreads a shredded path from the shredded child without reading the core storage.InferredShreddinginfers a shredding schema from a chunk of unshredded Variants and shreds later chunks by it.Compression
CascadingCompressorcan compress the storage of Variant arrays with another compressor. The default BtrBlocks preset uses one that keeps the binary Zstd scheme, because unshredded values are opaque binary blobs that the lightweight schemes cannot compress. Shredded children and all other columns are compressed as before.Tests cover the DataFusion
variant_getconversion and pushdown, in-place extraction from unshredded values (compared against the arrowvariant_getkernel for conversions, structs and lists), the scatter of residual rows,decode_unshredded_storage,variant_packreduction, and inferred shredding.On 1M unshredded Bluesky rows (single thread), extracting one path takes 38-69 ms. A struct of two paths in one pass takes 0.82-0.84 of the time of two separate extractions, and a struct of all five JSONBench paths takes 0.71.
API Changes
vortex_array::expr::variant_packandvariant_pack_with_validity, with theVariantPackscalar function registered in the default session.vortex_array::arrays::variant::decode_unshredded_storage.vortex_parquet_variant::InferredShredding.CascadingCompressor::with_variant_storage_compressor.🤖 Generated with Claude Code