Skip to content

Integrate JSONBench into vortex benchmarking - #11390

Draft
robert3005 wants to merge 5 commits into
developfrom
claude/nice-ramanujan-btk0qh
Draft

robert3005 wants to merge 5 commits into
developfrom
claude/nice-ramanujan-btk0qh

Conversation

@robert3005

@robert3005 robert3005 commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Summary

This PR adds JSONBench to the benchmark suite, together with all the Variant work that it needs to run end to end. It is the top of a stack, and it is kept open to show the complete result. Review and merge the PRs below in sequence:

  1. Add a Variant builder and fix reading chunked Variants #11499 (merged): a Variant builder and a fix for reading chunked Variants.
  2. Extract from Variants in bulk and push variant_get into the scan #11505: extract from Variants in bulk and push variant_get into the scan. This includes in-place extraction of strings, integers, structs and lists from unshredded values.
  3. An unstable Variant layout. It decodes Variant storage once per split.
  4. JSONBench in the benchmark suite.

This PR holds commits 2 to 4. The commits of #11505 come first.

Changes

Extraction and pushdown (#11505)

  • vortex-datafusion converts variant_get(variant, path, type) to the Vortex expression, so that the scan extracts the path.
  • Unshredded Parquet Variant storage extracts strings, integers, structs and lists in place, and converts values as the arrow variant_get kernel does.
  • Paths held by a shredded child are read from the shredded child. Only the rows that it does not hold are read from the core storage.
  • Add variant_pack, decode_unshredded_storage and InferredShredding. The default BtrBlocks preset compresses Variant storage with Zstd.

Variant layout

  • A shredded Variant is stored as a struct of its core storage and its shredded child, so each shredded field is a column of its own, with its own encoding and statistics.
  • The writer records whether the shredded child is complete. Only then is the core storage left unread.
  • A writer can take a VariantShredder, which shreds unshredded Variants by a schema inferred from their first chunk.
  • The layout is in the unstable edition.
  • The flat layout decodes unshredded Variant storage once per split, for every filter conjunct and the projection.

JSONBench

  • vortex-bench downloads the Bluesky NDJSON files and writes Parquet with a Parquet Variant column, optionally shredded (--opt shredding=). Vortex files shred by an inferred schema by default.
  • datafusion-bench registers variant_get(variant, path, type) with the signature of datafusion-variant, which has no release for DataFusion 55 and Arrow 59.
  • bench-orchestrator adds the suite. It rejects DuckDB over Vortex, because a Variant cannot be exported to DuckDB yet.
  • vortex-bench/sql/jsonbench.md documents the results at 1, 5 and 20 million rows.

On 1M unshredded rows (--opt shredding=none, DataFusion), Vortex takes 0.41-0.64 of the Parquet query time:

Query Parquet (ms) Vortex (ms) Ratio
Q0 40.7 25.9 0.64
Q1 81.3 50.2 0.62
Q2 86.7 40.9 0.47
Q3 79.4 32.4 0.41
Q4 80.1 36.2 0.45

API Changes

🤖 Generated with Claude Code

@robert3005 robert3005 added the changelog/feature A new feature label Nov 11, 2026
@robert3005
robert3005 force-pushed the claude/nice-ramanujan-btk0qh branch from dc3863a to 7fa1618 Compare November 16, 2026 17:10
- vortex-datafusion converts `variant_get(variant, path, type)` to the
  Vortex expression, so that the scan extracts the path, and folds
  filter equalities into the scan projection.
- `variant_get` brings chunked, masked and other lazy inputs to the
  Variant encodings before it falls back to extracting row by row.
- A shredded Variant child holds the rest of a path. The rows that the
  shredded child does not hold are extracted from the core storage on
  their own and scattered back, so that the core storage is not read when
  none of them holds a value.
- Parquet Variant storage extracts strings and integers from unshredded
  values in place, reads fixed-width integers with a single load, and
  resolves field names to ids once per metadata.
- Add `decode_unshredded_storage`, which decodes the storage of an
  unshredded Variant once for many extractions.
- Add the `variant_pack` expression, which packs core storage and a
  shredded child into a Variant. `variant_get` reads a shredded path from
  its shredded child without reading the core storage.
- Add `InferredShredding`, which infers a shredding schema from a chunk
  of unshredded Variants and shreds later chunks by it.
- `CascadingCompressor` can compress the storage of Variant arrays with
  another compressor. The default BtrBlocks preset uses one that keeps the
  binary Zstd scheme, since unshredded values are opaque binary blobs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Robert Kruszewski <github@robertk.io>
robert3005 and others added 4 commits November 16, 2026 17:55
The in-place extraction turned the whole column down at the first row
whose value it could not read as is, such as a JSON null or an integer at
a string path. The general kernel then extracted every row again.

Such a value now converts with the `Variant` method that the general
kernel converts it with, `as_string` or `as_int64`, and a value that does
not convert is a null. Objects and arrays convert to neither, so they
are null without being decoded. The extraction falls back to the general
kernel only for bytes that do not decode.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Robert Kruszewski <github@robertk.io>
The in-place extraction covered strings and 64-bit integers. A struct or
a list target went to the general kernel, which builds an Arrow array of
the storage and a `Variant` over every row.

The extraction is now a tree of builders, one for each node of the
target dtype. A struct reads the object at the path once and each of its
fields from the object's field of the same name, so it extracts several
paths in one pass. A list reads each element of the array at the path. A
value that is not an object or an array is a null, as it is for the
general kernel. Field names of every struct in the target resolve to
dictionary ids once per metadata, as the names of the path do.

Structs and lists are covered when their fields and elements are
nullable and covered themselves. A struct without fields is left to the
general kernel.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Robert Kruszewski <github@robertk.io>
A shredded Variant was written as one array per chunk: its metadata, its
binary value, and every shredded field in a single segment. A scan that
extracted one shredded field read and deserialized all of it.

- The Variant layout stores a shredded Variant as a struct of its core
  storage and its shredded child, and hands the struct to the struct
  layout. Each shredded field becomes a column of its own, with its own
  encoding and statistics. The layout plans as an evaluation of
  `variant_pack` over the struct's plan, and reads through the V1 scan
  by rewriting each expression over the stored struct.
- The writer records whether the shredded child is complete, that is
  whether no row left a value for it in the core storage. Only then is
  the core storage left unread.
- A writer can take a `VariantShredder`, which shreds unshredded
  Variants by a schema inferred from their first chunk, so that they are
  written with the Variant layout too.
- The layout is in the unstable edition.
- The flat layout plans a Variant segment scan under a share, and decodes
  unshredded Variant storage once, so that a split decodes the segment
  once for every filter conjunct and the projection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Robert Kruszewski <github@robertk.io>
JSONBench runs ClickHouse's five queries over Bluesky events held in a
single Variant column. DataFusion reads it from Parquet, Vortex, and
Vortex Compact. DuckDB reads it from Parquet and its native format.

- vortex-bench: download the NDJSON source files and write Parquet with
  a Parquet Variant column, optionally shredded (`--opt shredding=`).
  Vortex files shred by an inferred schema by default and use the
  Variant layout through an edition of their own. Add
  `Benchmark::queries_for_engine`, because Variant access has no SQL
  syntax that both engines accept.
- datafusion-bench: register `variant_get(variant, path, type)`. It has
  the signature of datafusion-variant, which has no release for
  DataFusion 55 and Arrow 59.
- bench-orchestrator: add the suite, and reject DuckDB over Vortex,
  which cannot export a Variant to DuckDB yet.
- Document the results at 1, 5 and 20 million rows in
  `vortex-bench/sql/jsonbench.md`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Robert Kruszewski <github@robertk.io>
@robert3005
robert3005 force-pushed the claude/nice-ramanujan-btk0qh branch from 1560ed0 to 7d0ae51 Compare November 16, 2026 20:03

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant