Repository navigation
Add an estimate of the file size to the Vortex writer - #11357
Draft
robert3005 wants to merge 4 commits into
Draft
robert3005 wants to merge 4 commits into
robert3005 wants to merge 4 commits into
Conversation
Segments reach the file in sequence order. A layout that holds an early sequence ID, such as a dictionary that writes its values last, can hold back every later segment until the end of the file. These segments were in neither bytes_written nor buffered_bytes, so callers that roll files by size, such as the Iceberg writer, underestimated the file. The segment sink now reserves the bytes of each segment on the writer's BufferedBytesTracker. The reservation goes through the channel with the last buffer of the segment, and the writer drops it after it writes the buffer. So each byte of a segment is always in bytes_written or in buffered_bytes, and their sum is an estimate of the file size, with no new API. The tracker, writer and Java docs now say what the count includes: the data that layout strategies hold, which the default strategy holds compressed, and the compressed segments that are not written yet. Signed-off-by: Robert Kruszewski <github@robertk.io>
The buffered bytes of a write count only compressed data. The data that the write accepted but did not compress yet was in no count. Early in a file, all the data is in this state, so callers that roll files by size saw a file that was much smaller than it was. The layout writer context now has a SizeScope, which counts these bytes in a tree of scopes. Each scope has its own byte count and compression ratio. One ratio for the write is not sufficient: columns compress at very different ratios, and a dictionary column holds codes before compression, which compress at a different ratio than the values. - The repartition strategy opens a child scope, and reserves the blocks that it holds on it. The strategies below it use the same scope. - The compressing strategy reserves each chunk that it compresses, and reserves the compressed output on the buffered bytes tracker before it releases the input. It records the ratio of each chunk on the scope. - The dictionary strategy records its probe, which is the first compression of a column. It gives the codes and the values their own scopes, and seeds the codes scopes with the codes part of the probe. So a scope has a ratio before its first full chunk is compressed. The reservations of an emitted block release when the child takes the block, so the bytes do not go out of the counts between stages. A scope without a sample uses the ratio of its nearest ancestor with a sample, then the ratio over the write, then one. Signed-off-by: Robert Kruszewski <github@robertk.io>
A caller that rolls files by size must know the size of an open file. The bytes written and the buffered bytes do not include the data that the writer did not compress yet, so early in a file they are much too small. WriteSizeEstimator holds the writer-scoped counts: the compressed bytes that the write holds, and the size scopes of the bytes that it did not compress yet. The estimate of the file size is the bytes written, plus the buffered bytes, plus the estimated compressed size of the uncompressed bytes. The estimate does not include the footer. When the write is complete, the other counts are zero and the estimate is equal to the file size. VortexWriteOptions::size_estimator returns the estimator of a write. Writer and BlockingWriter have estimated_size. The input queue of the push-based writer counts its arrays as uncompressed until the write takes them from the queue. Signed-off-by: Robert Kruszewski <github@robertk.io>
VortexWriter.estimatedSize returns the estimate of the native writer: the bytes written, plus the compressed bytes that the writer holds, plus the estimated compressed size of the bytes that it did not compress yet. After finish, it is equal to the file size. VortexWriter.compressionRatio returns the ratio over the data that the writer compressed so far, so that a caller can estimate the compressed size of data that it holds before it gives the data to the writer. The in-flight write queue of the JNI writer now counts its batches as uncompressed until the write task takes them. Signed-off-by: Robert Kruszewski <github@robertk.io>
This was referenced Nov 9, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A caller that rolls files by size, such as the Iceberg writer, must know the size of an open Vortex file. The writer gives only
bytes_writtenandbuffered_bytes. These do not include:So the Iceberg writer used the uncompressed size, and closed files at about 1/7.5 of the target size.
Change
estimated_size = bytes_written + buffered_bytes + estimated compressed size of the uncompressed bytesBufferedBytesTracker. The reservation goes through the channel with the last buffer of the segment, and the writer drops it after it writes the buffer.SizeScopetree onLayoutWriterContext, scoped to one writer. Each scope has its own byte count and compression ratio.RepartitionStrategyopens a child scope and reserves the blocks that it holds.CompressingStrategyreserves each chunk that it compresses, records(uncompressed, compressed)on the scope, and reserves the output on the buffered bytes before it releases the input.Writerand of the JNI writer reserve their batches until the write takes them.VortexWriteOptions::size_estimator(),Writer::estimated_size()andBlockingWriter::estimated_size().VortexWriter.estimatedSize()andVortexWriter.compressionRatio().finish, the estimate is equal to the file size.Known gaps: the footer, the dictionary values that the encoder holds (at most 1 MiB for each column), and the zone map statistics are not in the estimate.
Results
The Iceberg writer with
length()fromestimatedSize()(spiraldb/iceberg, branchrk/vortex-estimated-size). TPC-DS SF10store_sales(28.8M rows), Spark 4.2.1local[8], 8 partitions:At 32 and 64 MB, Vortex writes one more small file at the end of each partition, because its data is about 5% larger than Parquet. All tables read back the same results as Parquet. The write time at 64 MB did not change (4.0–4.5 s with the estimate, 4.2–4.3 s without).
Testing
cargo nextest runfor vortex-layout (776), vortex-file (218) and vortex-jni (13) passes. New tests:./gradlew :vortex-jni:testpasses, with the newtestEstimatedSize.cargo clippy --all-targets --all-features -- -D warningsfor the three crates, nightly rustfmt and spotless.