Skip to content

Add an estimate of the file size to the Vortex writer - #11357

Draft
robert3005 wants to merge 4 commits into
developfrom
rk/writer-estimated-size
Draft

robert3005 wants to merge 4 commits into
developfrom
rk/writer-estimated-size

Conversation

@robert3005

Copy link
Copy Markdown
Contributor

Problem

A caller that rolls files by size, such as the Iceberg writer, must know the size of an open Vortex file. The writer gives only bytes_written and buffered_bytes. These do not include:

  • Segments that the layouts emitted, but that wait for earlier segments or for the output. A dictionary layout that writes its values last can hold back every later segment until the end of the file.
  • Data that the writer did not compress yet: the input queues, the blocks that the repartition strategy holds, and the chunks that are being compressed. Early in a file, all the data is in this state.

So the Iceberg writer used the uncompressed size, and closed files at about 1/7.5 of the target size.

Change

estimated_size = bytes_written + buffered_bytes + estimated compressed size of the uncompressed bytes

  • Segments not written yet: the segment sink reserves the bytes of each segment on the BufferedBytesTracker. The reservation goes through the channel with the last buffer of the segment, and the writer drops it after it writes the buffer.
  • Uncompressed bytes: a new SizeScope tree on LayoutWriterContext, scoped to one writer. Each scope has its own byte count and compression ratio.
    • RepartitionStrategy opens a child scope and reserves the blocks that it holds.
    • CompressingStrategy reserves each chunk that it compresses, records (uncompressed, compressed) on the scope, and reserves the output on the buffered bytes before it releases the input.
    • The input queues of Writer and of the JNI writer reserve their batches until the write takes them.
    • The reservations of an emitted block release when the child takes the block, so bytes do not go out of the counts between stages.
  • Ratio: one ratio for the write is not sufficient, because columns compress at very different ratios. A scope uses its own ratio, then the ratio of its nearest ancestor with a sample, then the ratio over the write, then 1.0. The dictionary probe, which compresses the first 8192 rows of a column, seeds the scopes, so a scope has a ratio early in a file. Dictionary codes and values have their own scopes.
  • API:
    • Rust: VortexWriteOptions::size_estimator(), Writer::estimated_size() and BlockingWriter::estimated_size().
    • Java: VortexWriter.estimatedSize() and VortexWriter.compressionRatio().
    • After finish, the estimate is equal to the file size.

Known gaps: the footer, the dictionary values that the encoder holds (at most 1 MiB for each column), and the zone map statistics are not in the estimate.

Results

The Iceberg writer with length() from estimatedSize() (spiraldb/iceberg, branch rk/vortex-estimated-size). TPC-DS SF10 store_sales (28.8M rows), Spark 4.2.1 local[8], 8 partitions:

Target file size Parquet files / median MiB Vortex with estimate Vortex without estimate
16 MB 68 / 15.5 68 / 15.8 484 / 2.1
32 MB 32 / 32.3 38 / 32.4
64 MB 17 / 65.3 23 / 64.5 128 / 8.5
512 MB 8 / 127.9 8 / 134.9

At 32 and 64 MB, Vortex writes one more small file at the end of each partition, because its data is about 5% larger than Parquet. All tables read back the same results as Parquet. The write time at 64 MB did not change (4.0–4.5 s with the estimate, 4.2–4.3 s without).

Testing

  • cargo nextest run for vortex-layout (776), vortex-file (218) and vortex-jni (13) passes. New tests:
    • unit tests for scopes, ratios and reservations;
    • the estimate is scoped to one writer;
    • partway through a write (1/4, 1/2, 3/4), the estimate is within 20% of the final size, also with dictionary columns;
    • the estimate does not drop sharply between samples, and is equal to the file size after finish.
  • ./gradlew :vortex-jni:test passes, with the new testEstimatedSize.
  • cargo clippy --all-targets --all-features -- -D warnings for the three crates, nightly rustfmt and spotless.

Segments reach the file in sequence order. A layout that holds an early
sequence ID, such as a dictionary that writes its values last, can hold
back every later segment until the end of the file. These segments were
in neither bytes_written nor buffered_bytes, so callers that roll files
by size, such as the Iceberg writer, underestimated the file.

The segment sink now reserves the bytes of each segment on the writer's
BufferedBytesTracker. The reservation goes through the channel with the
last buffer of the segment, and the writer drops it after it writes the
buffer. So each byte of a segment is always in bytes_written or in
buffered_bytes, and their sum is an estimate of the file size, with no
new API.

The tracker, writer and Java docs now say what the count includes: the
data that layout strategies hold, which the default strategy holds
compressed, and the compressed segments that are not written yet.

Signed-off-by: Robert Kruszewski <github@robertk.io>
The buffered bytes of a write count only compressed data. The data that
the write accepted but did not compress yet was in no count. Early in a
file, all the data is in this state, so callers that roll files by size
saw a file that was much smaller than it was.

The layout writer context now has a SizeScope, which counts these bytes
in a tree of scopes. Each scope has its own byte count and compression
ratio. One ratio for the write is not sufficient: columns compress at
very different ratios, and a dictionary column holds codes before
compression, which compress at a different ratio than the values.

- The repartition strategy opens a child scope, and reserves the blocks
  that it holds on it. The strategies below it use the same scope.
- The compressing strategy reserves each chunk that it compresses, and
  reserves the compressed output on the buffered bytes tracker before
  it releases the input. It records the ratio of each chunk on the
  scope.
- The dictionary strategy records its probe, which is the first
  compression of a column. It gives the codes and the values their own
  scopes, and seeds the codes scopes with the codes part of the probe.
  So a scope has a ratio before its first full chunk is compressed.

The reservations of an emitted block release when the child takes the
block, so the bytes do not go out of the counts between stages. A scope
without a sample uses the ratio of its nearest ancestor with a sample,
then the ratio over the write, then one.

Signed-off-by: Robert Kruszewski <github@robertk.io>
A caller that rolls files by size must know the size of an open file.
The bytes written and the buffered bytes do not include the data that
the writer did not compress yet, so early in a file they are much too
small.

WriteSizeEstimator holds the writer-scoped counts: the compressed bytes
that the write holds, and the size scopes of the bytes that it did not
compress yet. The estimate of the file size is the bytes written, plus
the buffered bytes, plus the estimated compressed size of the
uncompressed bytes. The estimate does not include the footer. When the
write is complete, the other counts are zero and the estimate is equal
to the file size.

VortexWriteOptions::size_estimator returns the estimator of a write.
Writer and BlockingWriter have estimated_size. The input queue of the
push-based writer counts its arrays as uncompressed until the write
takes them from the queue.

Signed-off-by: Robert Kruszewski <github@robertk.io>
VortexWriter.estimatedSize returns the estimate of the native writer:
the bytes written, plus the compressed bytes that the writer holds,
plus the estimated compressed size of the bytes that it did not
compress yet. After finish, it is equal to the file size.
VortexWriter.compressionRatio returns the ratio over the data that the
writer compressed so far, so that a caller can estimate the compressed
size of data that it holds before it gives the data to the writer.

The in-flight write queue of the JNI writer now counts its batches as
uncompressed until the write task takes them.

Signed-off-by: Robert Kruszewski <github@robertk.io>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant