Skip to content

Extract from Variants in bulk and push variant_get into the scan - #11505

Draft
robert3005 wants to merge 3 commits into
developfrom
rk/variant-features
Draft

robert3005 wants to merge 3 commits into
developfrom
rk/variant-features

Conversation

@robert3005

@robert3005 robert3005 commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Queries over a Variant column extracted every path row by row, after reading the whole column. This PR makes the extraction happen in the Vortex scan and in bulk, and adds the pieces that the Variant layout in the next PR builds on.

It is the second PR of the stack in #11390, after #11499. The next PRs add an unstable Variant layout and the JSONBench suite.

Changes

Pushdown

  • vortex-datafusion converts variant_get(variant, path, type) to the Vortex expression, so that the scan extracts the path. Filter equalities fold into the scan projection.
  • variant_get brings chunked, masked and other lazy inputs to the Variant encodings before it falls back to extracting row by row. The canonicalization depth is bounded, so an input that canonicalizes to itself cannot recurse forever.

Extraction

  • A shredded Variant child holds the rest of a path. The rows that the shredded child does not hold are extracted from the core storage on their own and scattered back. When none of them holds a value, the core storage is not read.
  • Parquet Variant storage extracts strings and integers from unshredded values in place, reads fixed-width integers with a single load, and resolves field names to ids once per metadata.
  • A value that the in-place extraction cannot read as is, such as a JSON null or an integer at a string path, converts with the same Variant method as the general kernel (as_string or as_int64). A value that does not convert is a null. Before, one such value sent the whole column to the general kernel.
  • Structs and lists are extracted in place too. A struct reads the object at the path once and each field from it, so one struct target extracts several paths in one pass. A struct or list is covered when its fields or elements are nullable and covered. A struct without fields goes to the general kernel, which cannot build one.
  • decode_unshredded_storage decodes the storage of an unshredded Variant once for many extractions.

Building blocks for the Variant layout

  • The variant_pack expression packs core storage and a shredded child into a Variant. variant_get over variant_pack reads a shredded path from the shredded child without reading the core storage.
  • InferredShredding infers a shredding schema from a chunk of unshredded Variants and shreds later chunks by it.

Compression

  • CascadingCompressor can compress the storage of Variant arrays with another compressor. The default BtrBlocks preset uses one that keeps the binary Zstd scheme, because unshredded values are opaque binary blobs that the lightweight schemes cannot compress. Shredded children and all other columns are compressed as before.

Tests cover the DataFusion variant_get conversion and pushdown, in-place extraction from unshredded values (compared against the arrow variant_get kernel for conversions, structs and lists), the scatter of residual rows, decode_unshredded_storage, variant_pack reduction, and inferred shredding.

On 1M unshredded Bluesky rows (single thread), extracting one path takes 38-69 ms. A struct of two paths in one pass takes 0.82-0.84 of the time of two separate extractions, and a struct of all five JSONBench paths takes 0.71.

API Changes

  • New public vortex_array::expr::variant_pack and variant_pack_with_validity, with the VariantPack scalar function registered in the default session.
  • New public vortex_array::arrays::variant::decode_unshredded_storage.
  • New public vortex_parquet_variant::InferredShredding.
  • New CascadingCompressor::with_variant_storage_compressor.
  • The default BtrBlocks preset now compresses Variant storage with Zstd, so files with Variant columns are written differently.

🤖 Generated with Claude Code

- vortex-datafusion converts `variant_get(variant, path, type)` to the
  Vortex expression, so that the scan extracts the path, and folds
  filter equalities into the scan projection.
- `variant_get` brings chunked, masked and other lazy inputs to the
  Variant encodings before it falls back to extracting row by row.
- A shredded Variant child holds the rest of a path. The rows that the
  shredded child does not hold are extracted from the core storage on
  their own and scattered back, so that the core storage is not read when
  none of them holds a value.
- Parquet Variant storage extracts strings and integers from unshredded
  values in place, reads fixed-width integers with a single load, and
  resolves field names to ids once per metadata.
- Add `decode_unshredded_storage`, which decodes the storage of an
  unshredded Variant once for many extractions.
- Add the `variant_pack` expression, which packs core storage and a
  shredded child into a Variant. `variant_get` reads a shredded path from
  its shredded child without reading the core storage.
- Add `InferredShredding`, which infers a shredding schema from a chunk
  of unshredded Variants and shreds later chunks by it.
- `CascadingCompressor` can compress the storage of Variant arrays with
  another compressor. The default BtrBlocks preset uses one that keeps the
  binary Zstd scheme, since unshredded values are opaque binary blobs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Robert Kruszewski <github@robertk.io>
@robert3005
robert3005 marked this pull request as draft November 16, 2026 17:51
robert3005 and others added 2 commits November 16, 2026 17:55
The in-place extraction turned the whole column down at the first row
whose value it could not read as is, such as a JSON null or an integer at
a string path. The general kernel then extracted every row again.

Such a value now converts with the `Variant` method that the general
kernel converts it with, `as_string` or `as_int64`, and a value that does
not convert is a null. Objects and arrays convert to neither, so they
are null without being decoded. The extraction falls back to the general
kernel only for bytes that do not decode.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Robert Kruszewski <github@robertk.io>
The in-place extraction covered strings and 64-bit integers. A struct or
a list target went to the general kernel, which builds an Arrow array of
the storage and a `Variant` over every row.

The extraction is now a tree of builders, one for each node of the
target dtype. A struct reads the object at the path once and each of its
fields from the object's field of the same name, so it extracts several
paths in one pass. A list reads each element of the array at the path. A
value that is not an object or an array is a null, as it is for the
general kernel. Field names of every struct in the target resolve to
dictionary ids once per metadata, as the names of the path do.

Structs and lists are covered when their fields and elements are
nullable and covered themselves. A struct without fields is left to the
general kernel.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Robert Kruszewski <github@robertk.io>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant