Skip to content

[Spark] Add reproducible Vortex vs Parquet performance benchmarks and CI reporting #81

Open

Description

@ChristopherWirt

Problem

We should have a maintained Spark performance suite that compares Vortex with Spark's Parquet reader and tracks regressions in the QC fork. Upstream already has useful Spark JMH benchmarks, but they do not provide the engine/format comparisons, automated performance runs, or published history that exist for DataFusion and DuckDB.

This matters because Rust kernel and DataFusion results do not measure Spark's planning, file discovery, Hadoop I/O, JNI/Arrow boundary, task scheduling, or connector pushdown behavior.

What upstream currently covers

Investigated on 4 November 2026 against upstream bc58587293a3.

Integration Existing coverage Gap relevant to this issue
DataFusion / DuckDB End-to-end SQL benchmarks including TPC-H, TPC-DS, ClickBench, and other workloads; Vortex vs Parquet and additional formats. Local NVMe and S3 cases, post-merge runs, larger nightly runs, and opt-in PR comparisons. This is the model to follow for repeatability and reporting.
Spark JMH SparkScanBenchmark: 2 million synthetic rows, 1/16/128 files, count, aggregate, filtered scan, and single-column scan. FooterReadBenchmark: individual footer reads and parallel row-count collection. Uses local[4] and Vortex-only input. No Parquet baseline or standard SQL workload in these benchmarks. No workflow invoking JMH found in upstream CI; no Spark results on the public dashboard.
Iceberg Vortex integration is being developed separately in the Iceberg fork / upstream PR. No maintained Vortex-specific Iceberg performance suite found in the Vortex benchmark catalog or public dashboard. Iceberg-over-Spark would exercise an additional integration path.

Sources:

Proposed initial scope

  1. Extend the existing Spark benchmarks with a Parquet baseline. Reuse the harness and cover narrow/wide projections, full decode and aggregation, selective predicates, and many-small-files overhead. Include supported pushdown predicates and residual-filter cases; record the executed plans so we know which path was measured. Keep footer-only COUNT(*) separate from scans that actually decode values.
  2. Add a reproducible Spark SQL workload. Start with TPC-H-derived queries at SF=10 on one fixed machine, reusing upstream data generation/query definitions where practical and documenting Spark SQL adaptations. Compare the same queries and logical data through Spark+Vortex and Spark+Parquet. Add TPC-DS and larger/distributed runs after this baseline is reliable.
  3. Automate collection and comparison. Provide an on-demand QC workflow, then scheduled runs once noise is understood. Store machine-readable results and compare base/head on equivalent hardware. Start with reporting; introduce regression thresholds only after measuring run-to-run variance. Reuse the existing result schema/reporting where practical, with Spark-specific build/run support.

Measurement requirements

  • Pin Spark, Scala, JDK, Vortex/native build, machine resources, and Spark settings. Record the commit and physical plan, including AQE and pushdown settings. Start with the QC deployment version; track other supported Spark versions separately.
  • Generate data outside timed runs. Match logical rows, ordering, partitioning, and rows per file; record actual file counts and compressed sizes. Explicitly set Parquet Zstd level 3 for generated benchmark data. Any additional production-codec baseline should be separately labelled.
  • Warm up the JVM/native path, repeat measurements, and report median plus spread and per-query results. Define warm/cold cache procedures; distinguish local storage from S3 and single-machine from distributed results.
  • Fully execute queries and validate results against the baseline. Do not use metadata-only counts as evidence of decode throughput.
  • Collect query wall time and useful Spark metrics such as scan/input bytes, task counts, and planning time where available. Keep data conversion/write time separate from read-query timing.

Acceptance criteria

  • A documented command runs comparable Spark+Vortex and Spark+Parquet benchmarks from a clean checkout.
  • Connector cases cover projection, filtering, actual decoding, footer-only operations, and file-count scaling.
  • A TPC-H-derived Spark suite produces validated per-query results at a documented scale.
  • An on-demand QC workflow retains results, environment/configuration metadata, and a base/head comparison.
  • A scheduled baseline provides accessible performance history once repeatability is established.
  • Follow-up scope is recorded for S3, TPC-DS, distributed execution, write throughput, and Iceberg-over-Spark. Iceberg should compare Iceberg+Vortex with Iceberg+Parquet under the same Spark setup when that integration is ready.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions