Skip to content

build: RaincloudCompare compares compact writes with Rust's too - #727

Draft
dfa1 wants to merge 1 commit into
mainfrom
build/raincloud-compare-rust-compact
Draft

dfa1 wants to merge 1 commit into
mainfrom
build/raincloud-compare-rust-compact

Conversation

@dfa1

@dfa1 dfa1 commented Nov 10, 2026 •

Copy link
Copy Markdown
Owner

vortex-jni has no compact option (vortex-data/vortex#11374). Its writer always takes the session's default strategy, and the options map only reaches object-store resolution. Python vortex-data at the same Rust version has both VortexWriteOptions.default() and .compact().

With VORTEX_PYTHON set to a Python that has vortex-data==<vortex-jni.version>, RaincloudCompare writes the Parquet with both Rust options and compares each Java write against its Rust counterpart. The flags are BIGGER for the default write and BIGGER_COMPACT for compact. The script warns if the two Rust versions differ.

jni becomes a reference column only. The vortex-jni rewrite writes the batches its scan of Raincloud's file yields, and those vary between runs. Its size varies with them: the same openpowerlifting rewrite came out at 173.1 MB and at 180.6 MB. Without VORTEX_PYTHON there's no baseline, so nothing is flagged.

slug              rust       rust_compact  java       java_compact  java_vs_rust  compact_vs_rust  flag
uci-iris          9468       7132          8744       7232          -7.6%         +1.4%
coig              66646648   44716456      69016252   44789932      +3.6%         +0.2%            BIGGER
mnist             17841708   15754492      18023451   15752268      +1.0%         -0.0%
openpowerlifting  169954388  106515180     169431469  109896845     -0.3%         +3.2%            BIGGER_COMPACT

New finding: openpowerlifting imported with --compact is 3.2% larger than Rust's compact write. I'll open an issue once it's analyzed.

🤖 Generated with Claude Code

https://claude.ai/code/session_011fRbYvRdQi45rxQgF3QMMP

vortex-jni has no compact option (vortex-data/vortex#11374): its writer
always takes the session's default strategy, and the options map reaches only
the object-store resolution. Python vortex-data at the same Rust version has
both (VortexWriteOptions.default() and .compact()), and it writes the Parquet
sibling Java imports. So with VORTEX_PYTHON set the script writes the Parquet
with both Rust options and compares each Java write with its Rust counterpart
(BIGGER, BIGGER_COMPACT), warning when the Python vortex-data and vortex-jni
versions differ.

The vortex-jni rewrite stays as a reference column only: it writes the
batches its scan of Raincloud's file yields, which vary between runs, and its
size with them (173.1 MB and 180.6 MB for the same openpowerlifting rewrite).
Without VORTEX_PYTHON there is no baseline and nothing is flagged.

First finding: openpowerlifting imported with --compact is 3.2% larger than
Rust's compact write.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011fRbYvRdQi45rxQgF3QMMP
@dfa1
dfa1 force-pushed the build/raincloud-compare-rust-compact branch from 98ae0eb to 1ebf312 Compare November 10, 2026 07:56
@dfa1
dfa1 marked this pull request as draft November 10, 2026 08:06
@dfa1 dfa1 added the PARKED do not yet merge label Nov 10, 2026
@dfa1

dfa1 commented Nov 10, 2026

Copy link
Copy Markdown
Owner Author

Parked until vortex-jni can write compact (vortex-data/vortex#11374). The compact baseline should then come from vortex-jni in-process, not from Python vortex-data, so this PR will be reworked against that.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

PARKED do not yet merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant