Repository navigation
Integrate JSONBench into vortex benchmarking - #11390
Draft
robert3005 wants to merge 5 commits into
Draft
robert3005 wants to merge 5 commits into
robert3005 wants to merge 5 commits into
Conversation
robert3005
force-pushed
the
claude/nice-ramanujan-btk0qh
branch
from
November 16, 2026 17:10
dc3863a to
7fa1618
Compare
- vortex-datafusion converts `variant_get(variant, path, type)` to the Vortex expression, so that the scan extracts the path, and folds filter equalities into the scan projection. - `variant_get` brings chunked, masked and other lazy inputs to the Variant encodings before it falls back to extracting row by row. - A shredded Variant child holds the rest of a path. The rows that the shredded child does not hold are extracted from the core storage on their own and scattered back, so that the core storage is not read when none of them holds a value. - Parquet Variant storage extracts strings and integers from unshredded values in place, reads fixed-width integers with a single load, and resolves field names to ids once per metadata. - Add `decode_unshredded_storage`, which decodes the storage of an unshredded Variant once for many extractions. - Add the `variant_pack` expression, which packs core storage and a shredded child into a Variant. `variant_get` reads a shredded path from its shredded child without reading the core storage. - Add `InferredShredding`, which infers a shredding schema from a chunk of unshredded Variants and shreds later chunks by it. - `CascadingCompressor` can compress the storage of Variant arrays with another compressor. The default BtrBlocks preset uses one that keeps the binary Zstd scheme, since unshredded values are opaque binary blobs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Robert Kruszewski <github@robertk.io>
robert3005
force-pushed
the
claude/nice-ramanujan-btk0qh
branch
from
November 16, 2026 17:39
7fa1618 to
1560ed0
Compare
The in-place extraction turned the whole column down at the first row whose value it could not read as is, such as a JSON null or an integer at a string path. The general kernel then extracted every row again. Such a value now converts with the `Variant` method that the general kernel converts it with, `as_string` or `as_int64`, and a value that does not convert is a null. Objects and arrays convert to neither, so they are null without being decoded. The extraction falls back to the general kernel only for bytes that do not decode. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Robert Kruszewski <github@robertk.io>
The in-place extraction covered strings and 64-bit integers. A struct or a list target went to the general kernel, which builds an Arrow array of the storage and a `Variant` over every row. The extraction is now a tree of builders, one for each node of the target dtype. A struct reads the object at the path once and each of its fields from the object's field of the same name, so it extracts several paths in one pass. A list reads each element of the array at the path. A value that is not an object or an array is a null, as it is for the general kernel. Field names of every struct in the target resolve to dictionary ids once per metadata, as the names of the path do. Structs and lists are covered when their fields and elements are nullable and covered themselves. A struct without fields is left to the general kernel. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Robert Kruszewski <github@robertk.io>
A shredded Variant was written as one array per chunk: its metadata, its binary value, and every shredded field in a single segment. A scan that extracted one shredded field read and deserialized all of it. - The Variant layout stores a shredded Variant as a struct of its core storage and its shredded child, and hands the struct to the struct layout. Each shredded field becomes a column of its own, with its own encoding and statistics. The layout plans as an evaluation of `variant_pack` over the struct's plan, and reads through the V1 scan by rewriting each expression over the stored struct. - The writer records whether the shredded child is complete, that is whether no row left a value for it in the core storage. Only then is the core storage left unread. - A writer can take a `VariantShredder`, which shreds unshredded Variants by a schema inferred from their first chunk, so that they are written with the Variant layout too. - The layout is in the unstable edition. - The flat layout plans a Variant segment scan under a share, and decodes unshredded Variant storage once, so that a split decodes the segment once for every filter conjunct and the projection. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Robert Kruszewski <github@robertk.io>
JSONBench runs ClickHouse's five queries over Bluesky events held in a single Variant column. DataFusion reads it from Parquet, Vortex, and Vortex Compact. DuckDB reads it from Parquet and its native format. - vortex-bench: download the NDJSON source files and write Parquet with a Parquet Variant column, optionally shredded (`--opt shredding=`). Vortex files shred by an inferred schema by default and use the Variant layout through an edition of their own. Add `Benchmark::queries_for_engine`, because Variant access has no SQL syntax that both engines accept. - datafusion-bench: register `variant_get(variant, path, type)`. It has the signature of datafusion-variant, which has no release for DataFusion 55 and Arrow 59. - bench-orchestrator: add the suite, and reject DuckDB over Vortex, which cannot export a Variant to DuckDB yet. - Document the results at 1, 5 and 20 million rows in `vortex-bench/sql/jsonbench.md`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Robert Kruszewski <github@robertk.io>
robert3005
force-pushed
the
claude/nice-ramanujan-btk0qh
branch
from
November 16, 2026 20:03
1560ed0 to
7d0ae51
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds JSONBench to the benchmark suite, together with all the Variant work that it needs to run end to end. It is the top of a stack, and it is kept open to show the complete result. Review and merge the PRs below in sequence:
variant_getinto the scan. This includes in-place extraction of strings, integers, structs and lists from unshredded values.This PR holds commits 2 to 4. The commits of #11505 come first.
Changes
Extraction and pushdown (#11505)
variant_get(variant, path, type)to the Vortex expression, so that the scan extracts the path.variant_getkernel does.variant_pack,decode_unshredded_storageandInferredShredding. The default BtrBlocks preset compresses Variant storage with Zstd.Variant layout
VariantShredder, which shreds unshredded Variants by a schema inferred from their first chunk.JSONBench
--opt shredding=). Vortex files shred by an inferred schema by default.variant_get(variant, path, type)with the signature of datafusion-variant, which has no release for DataFusion 55 and Arrow 59.vortex-bench/sql/jsonbench.mddocuments the results at 1, 5 and 20 million rows.On 1M unshredded rows (
--opt shredding=none, DataFusion), Vortex takes 0.41-0.64 of the Parquet query time:API Changes
VariantShredderare new, and are in the unstable edition only.🤖 Generated with Claude Code