Skip to content

Research: relate Spark 4.3 Arrow cache serializer to Comet in-memory cache #7517

Open

Description

@andygrove

Summary

Spark 4.3 adds an opt-in Arrow cache format for Dataset caching: org.apache.spark.sql.execution.columnar.ArrowCachedBatchSerializer, selected with the static config spark.sql.cache.serializer (default stays DefaultCachedBatchSerializer). Comet already has an experimental cache of its own (spark.comet.exec.inMemoryCache.enabled, see the In-Memory Cache user guide page) that stores cached data as Arrow and scans it with CometInMemoryTableScan. This issue is to research how the two relate and whether Comet should interoperate with, defer to, or stay independent of Spark's serializer on 4.3+.

This is research only. No code change is proposed yet.

What Spark 4.3.0 ships

Verified against the v4.3.0 tag (docs/sql-arrow-cache-format.md, ArrowCachedBatch.scala, ArrowCachedBatchSerializer.scala):

  • ArrowCachedBatch(numRows, arrowData: Array[Byte], stats: InternalRow) extends SimpleMetricsCachedBatch. arrowData is one schema-less encapsulated Arrow RecordBatch message; the schema is rebuilt from the relation's attributes on read.
  • It accepts both row and columnar input, collects per-column min/max/null-count statistics for batch pruning, and supports none, zstd and lz4 Arrow compression (spark.sql.execution.arrow.compression.codec).
  • Reads can produce ArrowColumnVector-backed ColumnarBatches, with a background prefetch option (spark.sql.execution.arrow.cache.prefetch.enabled).
  • It rejects any schema containing a type Arrow cannot represent. There is no per-relation fallback to the default serializer, since spark.sql.cache.serializer is a single static class for the whole session.

How this overlaps with Comet's cache

Comet's org.apache.spark.sql.comet.execution.arrow.ArrowCachedBatchSerializer (note the same simple name) is installed by CometDriverPlugin as spark.sql.cache.serializer when the cache is enabled. Its CometCachedBatch is also a schema-less encapsulated Arrow IPC RecordBatch message with per-buffer compression and SimpleMetricsCachedBatch stats, so the two formats look close. Differences already visible: Comet delegates unsupported schemas per relation to Spark's default format, scans through CometInMemoryTableScan into native operators, and projects columns by copying only the selected buffers' byte ranges.

Questions to answer

  1. Selection conflict. Both are chosen via the same single static config. What does the plugin do today if a user sets Spark's ArrowCachedBatchSerializer explicitly on 4.3? Should Comet keep overriding, respect the user's choice, or refuse?
  2. Can Comet read Spark's format? If the payloads are layout-compatible (buffer order, compression framing, dictionary and nested-type handling, stats layout), CometInMemoryTableScan could scan relations cached by Spark's serializer natively, with no need to replace the serializer. What would it take, and is the compatibility stable enough to depend on given Spark calls it an internal format?
  3. Can Comet reuse Spark's serializer for writes? Would delegating to Spark's serializer on 4.3+ and keeping only the scan side simplify Comet's cache code (CometCachedBatch, CachedBatchIpc), or lose features we rely on (per-relation fallback, column projection by buffer range, per-batch codec recorded in the message)?
  4. Type coverage. Compare supported types with Comet's writer (intervals, TIME, nanosecond timestamps, variant, nested types). Spark throws on unsupported types where Comet falls back per relation.
  5. Performance. Using the existing cache benchmarks, compare write cost, read cost into native operators, and memory footprint for Spark's Arrow cache + a Comet scan against Comet's own format, on filter-heavy, projection-heavy and wide-table workloads.
  6. Operational. Kryo registration, memory accounting (Spark's serializer holds Array[Byte]; Comet uses ChunkedByteBuffer to pass the 2 GiB array limit), and what changes in the checks that read spark.sql.cache.serializer (CometCacheBuildRule, CometExecRule).

Deliverable

A short write-up (comment on this issue or a doc under the contributor guide) with a recommendation: keep independent, interoperate on read, or converge on Spark's format on 4.3+, plus follow-up issues for whichever path is chosen.

Context

Found while adding the Spark 4.3 build profile (Stage 1 of the new-version bring-up). Nothing in that PR touches the cache.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions