Repository navigation
Description
Summary
Spark 4.3 adds an opt-in Arrow cache format for Dataset caching: org.apache.spark.sql.execution.columnar.ArrowCachedBatchSerializer, selected with the static config spark.sql.cache.serializer (default stays DefaultCachedBatchSerializer). Comet already has an experimental cache of its own (spark.comet.exec.inMemoryCache.enabled, see the In-Memory Cache user guide page) that stores cached data as Arrow and scans it with CometInMemoryTableScan. This issue is to research how the two relate and whether Comet should interoperate with, defer to, or stay independent of Spark's serializer on 4.3+.
This is research only. No code change is proposed yet.
What Spark 4.3.0 ships
Verified against the v4.3.0 tag (docs/sql-arrow-cache-format.md, ArrowCachedBatch.scala, ArrowCachedBatchSerializer.scala):
ArrowCachedBatch(numRows, arrowData: Array[Byte], stats: InternalRow)extendsSimpleMetricsCachedBatch.arrowDatais one schema-less encapsulated Arrow RecordBatch message; the schema is rebuilt from the relation's attributes on read.- It accepts both row and columnar input, collects per-column min/max/null-count statistics for batch pruning, and supports
none,zstdandlz4Arrow compression (spark.sql.execution.arrow.compression.codec). - Reads can produce
ArrowColumnVector-backedColumnarBatches, with a background prefetch option (spark.sql.execution.arrow.cache.prefetch.enabled). - It rejects any schema containing a type Arrow cannot represent. There is no per-relation fallback to the default serializer, since
spark.sql.cache.serializeris a single static class for the whole session.
How this overlaps with Comet's cache
Comet's org.apache.spark.sql.comet.execution.arrow.ArrowCachedBatchSerializer (note the same simple name) is installed by CometDriverPlugin as spark.sql.cache.serializer when the cache is enabled. Its CometCachedBatch is also a schema-less encapsulated Arrow IPC RecordBatch message with per-buffer compression and SimpleMetricsCachedBatch stats, so the two formats look close. Differences already visible: Comet delegates unsupported schemas per relation to Spark's default format, scans through CometInMemoryTableScan into native operators, and projects columns by copying only the selected buffers' byte ranges.
Questions to answer
- Selection conflict. Both are chosen via the same single static config. What does the plugin do today if a user sets Spark's
ArrowCachedBatchSerializerexplicitly on 4.3? Should Comet keep overriding, respect the user's choice, or refuse? - Can Comet read Spark's format? If the payloads are layout-compatible (buffer order, compression framing, dictionary and nested-type handling,
statslayout),CometInMemoryTableScancould scan relations cached by Spark's serializer natively, with no need to replace the serializer. What would it take, and is the compatibility stable enough to depend on given Spark calls it an internal format? - Can Comet reuse Spark's serializer for writes? Would delegating to Spark's serializer on 4.3+ and keeping only the scan side simplify Comet's cache code (
CometCachedBatch,CachedBatchIpc), or lose features we rely on (per-relation fallback, column projection by buffer range, per-batch codec recorded in the message)? - Type coverage. Compare supported types with Comet's writer (intervals, TIME, nanosecond timestamps, variant, nested types). Spark throws on unsupported types where Comet falls back per relation.
- Performance. Using the existing cache benchmarks, compare write cost, read cost into native operators, and memory footprint for Spark's Arrow cache + a Comet scan against Comet's own format, on filter-heavy, projection-heavy and wide-table workloads.
- Operational. Kryo registration, memory accounting (Spark's serializer holds
Array[Byte]; Comet usesChunkedByteBufferto pass the 2 GiB array limit), and what changes in the checks that readspark.sql.cache.serializer(CometCacheBuildRule,CometExecRule).
Deliverable
A short write-up (comment on this issue or a doc under the contributor guide) with a recommendation: keep independent, interoperate on read, or converge on Spark's format on 4.3+, plus follow-up issues for whichever path is chosen.
Context
Found while adding the Spark 4.3 build profile (Stage 1 of the new-version bring-up). Nothing in that PR touches the cache.
Activity