Repository navigation
Conversation
vortex-jni has no compact option (vortex-data/vortex#11374): its writer always takes the session's default strategy, and the options map reaches only the object-store resolution. Python vortex-data at the same Rust version has both (VortexWriteOptions.default() and .compact()), and it writes the Parquet sibling Java imports. So with VORTEX_PYTHON set the script writes the Parquet with both Rust options and compares each Java write with its Rust counterpart (BIGGER, BIGGER_COMPACT), warning when the Python vortex-data and vortex-jni versions differ. The vortex-jni rewrite stays as a reference column only: it writes the batches its scan of Raincloud's file yields, which vary between runs, and its size with them (173.1 MB and 180.6 MB for the same openpowerlifting rewrite). Without VORTEX_PYTHON there is no baseline and nothing is flagged. First finding: openpowerlifting imported with --compact is 3.2% larger than Rust's compact write. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011fRbYvRdQi45rxQgF3QMMP
dfa1
force-pushed
the
build/raincloud-compare-rust-compact
branch
from
November 10, 2026 07:56
98ae0eb to
1ebf312
Compare
dfa1
marked this pull request as draft
November 10, 2026 08:06
Owner
Author
|
Parked until vortex-jni can write compact (vortex-data/vortex#11374). The compact baseline should then come from vortex-jni in-process, not from Python |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
vortex-jni has no compact option (vortex-data/vortex#11374). Its writer always takes the session's default strategy, and the
optionsmap only reaches object-store resolution. Pythonvortex-dataat the same Rust version has bothVortexWriteOptions.default()and.compact().With
VORTEX_PYTHONset to a Python that hasvortex-data==<vortex-jni.version>,RaincloudComparewrites the Parquet with both Rust options and compares each Java write against its Rust counterpart. The flags areBIGGERfor the default write andBIGGER_COMPACTfor compact. The script warns if the two Rust versions differ.jnibecomes a reference column only. The vortex-jni rewrite writes the batches its scan of Raincloud's file yields, and those vary between runs. Its size varies with them: the sameopenpowerliftingrewrite came out at 173.1 MB and at 180.6 MB. WithoutVORTEX_PYTHONthere's no baseline, so nothing is flagged.New finding:
openpowerliftingimported with--compactis 3.2% larger than Rust's compact write. I'll open an issue once it's analyzed.🤖 Generated with Claude Code
https://claude.ai/code/session_011fRbYvRdQi45rxQgF3QMMP