Repository navigation
Extended Vortex encodings (product_residual) - #8
Draft
MassivePizza wants to merge 21 commits into
Draft
MassivePizza wants to merge 21 commits into
MassivePizza wants to merge 21 commits into
Conversation
MassivePizza
force-pushed
the
vortex-dv-residual
branch
2 times, most recently
from
November 5, 2026 20:53
0c98372 to
d5eda73
Compare
MassivePizza
force-pushed
the
vortex-dv-residual
branch
2 times, most recently
from
November 9, 2026 17:17
85b073c to
cc0914e
Compare
What the plan's rescale did, dividing each decimal column by the trailing zeros all its values have, `DecimalPow10Scheme` does a chunk at a time: a `vortex_ohlc.scaled` decimal of one exponent k for the whole chunk, over DecimalByteParts. Without it dollar_volume's Decimal(38, 12) is wider than an i64 unscaled, so DecimalByteParts cuts it into an integer part and a u64 lower part, and Pow10Scheme only divides the integer part. Every chunk's k turned out to be its column's (parts 2, volume 6, dollar_volume 8, once 10), and after it every chunk fits an i64, so dividing i128s a block at a time would gain nothing. The scheme takes the place of Vortex's decimal scheme in write_options: both always bid, and the first registered wins. It builds the DecimalByteParts itself in the same cascade step, so Pow10Scheme and blocked FoR under it are as deep as under Vortex's scheme (a level of its own would leave the quotients uncompressed at MAX_CASCADE). With one divisor for every row, Scaled reads through its values: filters, slices, takes, masks and nullability casts go through to them, and compares with a constant are rewritten to the quotients. This is the second half of d5c32fe, split from it when the stack was reordered on 2026-11-04 (the first half drops the plan). It was then an encoding of its own, the first to be called `vortex_ohlc.scaled`, with its values at the smaller scale; the numbers below are from that version. Measured before the reorder: Size (2006-08): 334.0 MiB without any rescale (bars-11, its parent here but for the zones' 50 KB) -> 236.3 MiB (bars-13). Against the plan's rescale (bars-12) that is +112 KB (+0.045%, about 48 bytes per chunk and column). Speed against c54c0ab, with the plan's rescale, rounds 1-2 of 3 (round 3 shared the CPU), median ms: scan/dollar_volume 1 partition 12.90 -> 5.18 (-60%), 14: -23% scan/all columns 1 partition 57.50 -> 46.55 (-19%), 14: -13% needles within -7% and +6%, both ways Probably because the scans no longer rescale in DataFusion (a guess; not profiled). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A bar's dollar volume is volume × vwap, with the vwap between its low and high, so dollar_volume - volume × open is small, and 0 in many of a month's bars. Where a table of bars has a decimal dollar_volume and volume, the bars layout stores that residual as dollar_volume_residual (a name no source column has, so its zone map is never taken for dollar_volume's) and puts dollar_volume's own bounds in its zones table. A null volume counts as 0. On reading, a new scalar function, vortex_ohlc.dollar_volume, rebuilds residual + volume × open. It looks through `scaled` to the narrow values, and multiplies 64 × 64 -> 128 bits where they fit: 5 multiply instructions a row instead of 10, by the disassembly. The test table gains a dollar_volume column. Size: bars-15 is 206.7 MiB, -29.7 MiB (-12.6%); dollar_volume 75.6 -> 45.9 MiB. Speed against bd9c937 (best of 2-3 rounds, ms; the machine was busy): - scan/dollar_volume 5.21 -> 9.63 (+85%) at one partition, 9.66 -> 13.02 (+35%) at 14. Without the 64-bit multiplies it was +108% and +48%. - every dv > needle +14% to +23%. - scan/all columns +0-3%, the other queries unchanged. Reading dollar_volume now decodes volume and the open part as well: 28% of scan/dollar_volume's profile, against 15% for the rebuild. Kept for the size. Measured as 89c7bd8, before the stack was reordered on 2026-11-04; its parent there (bd9c937) had the same tree as this one's but for the bench's version. Rewritten on 2026-11-05, when the stack's three dividing encodings became one Scaled: a Scaled decimal's quotients keep its scale, so `decoded` labels them with the smaller one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The residual (89c7bd8) saves 29.7 MiB on a month of minute bars, but reading dollar_volume then decodes volume and the open as well: filters on it were 14-23% slower, a scan of it alone 35-85%. Which matters more depends on the table and how it's queried, so BarsStrategy::with_dollar_volume_residual turns it off, and write_options takes the choice as a parameter. The bench keeps it on. The layout's metadata gains a byte that says which, since the stored table's and the zones' dtypes no longer follow from the source's alone. Files written before can't be read (bars-26, the same size as bars-25). A round trip without the residual stores dollar_volume as it is and reads the source back. Measured as 54ad05e, before the stack was reordered on 2026-11-04; its parent there (89c7bd8) had the same tree as this one's but for the bench's version. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Vortex stores a timestamp as DateTimeParts: days, seconds into the day and subseconds. A minute bar's seconds are all multiples of 60, and a block of one ticker's bars spans most of a trading day, so blocked FoR packed them at 16 bits a row: 131 KB of window_start's 133 KB in a chunk. `GcdScheme` stores a chunk of them divided by the greatest common divisor of its values, as a `vortex_ohlc.scaled` array whose base is the divisor, to the power 1, and takes only DateTimeParts' seconds (the compressor's cascade history is the temporal scheme's child 1). Minute bars' seconds become minutes into the day, which blocked FoR packs at 10 bits. an earlier prototype got there with a copy of DateTimeParts with a fourth part; this keeps Vortex's own. With one divisor for every row, Scaled reads through the quotients: filters, slices, takes, masks and nullability casts go through to them, and a compare with a constant is rewritten to them (`q op ⌊c / d⌋`), so DateTimeParts' compare (days, then seconds) runs on the packed minutes. It was first written as an encoding of its own, `vortex_ohlc.gcd`, before the stack's dividing encodings became one Scaled on 2026-11-05; the numbers below are from that version. Size (2006-08): bars-25 (now bars-27) is 188.9 MiB, -17.8 MiB (-8.6%); window_start 46.9 -> 29.1 MiB, against the earlier prototype's 30.1. Every other column is byte-identical, and every bench query returns the same rows and bytes. Speed against 15b05e4 (best of 3 alternating rounds, 1 / 14 partitions): - the pruned AAPL window needles +2-3%, high > 1000 +2-3%; - all @ 14:30 and all 14:30-14:35 +1% / -2%; - high = 50, dv > 1530000, AAPL & close > 67, scan/all columns -2% to +0%. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…red at Reading a price executed each part through `scaled`, which multiplies the values by 10^k into an i128 buffer whatever their width. The rebuild then narrowed each value to an i64, summed, and widened the price back into an i128 buffer, so a filter on it compared i128s. For high = 50 the i128 buffers alone were 15.7 of its 116 ms of CPU, against 1.1 for upstream's decode. The price function now reads each part through `scaled` to its values at the width they're stored at (dollar_volume's helper, moved to `scaled`), widens each once into the rebuild's type (i64 for prices of at most 18 digits), bringing parts at different scales to the finest of them, and rebuilds into a buffer of that type. A compare then runs on i64s. Speed against 10730a6 (best of 3 alternating rounds, ms, 1 / 14 partitions): - high = 50 20.22 -> 17.46 (-14%) / -12%; low = 50 -14% / -12%; - scan/all columns 45.8 -> 39.4 (-14%) / -13%; scan/decimal columns -17% at both; - high > 1000, close > 400.5, AAPL & close > 67 -6% to -7%; open 400-401 -3%; - AAPL, AAPL @ 14:30, all @ 14:30, dv > 1530000 and scan/dollar_volume within 2%. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Vortex's tracking issue #10701 gives every decimal leaf its precision's one width and makes a narrower one a `Widen` encoding; #10905 and #10995 decode DecimalByteParts and narrow `vortex.decimal` nodes into it. When we take it up, `decoded` must look through `Widen` as it does `Scaled`, or the price and dollar volume rebuilds get their parts copied out wide. (`Scaled`'s decode into i128 buffers, which become invalid for precisions of 18 digits or fewer, has its note from the commit that let it hold decimals.) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The bench's "1 partition" only set DataFusion's target partitions. Vortex's scan keeps scan_concurrency (4) x cores splits of a partition in flight and spawns each onto the runtime's workers (vortex-layout 0.89 repeated_scan.rs:243), so on the bench's runtime, a worker per core, one partition kept ~8 of 14 threads busy, while Parquet's reader keeps a partition on one thread. Vortex's lead over Parquet at one partition was mostly threads, and its ratios couldn't be compared with an earlier prototype's single-threaded harness. One partition now runs on a runtime of its own with one worker; a partition per core keeps a worker per core. At 817e918, vortex/1: high = 50 18.1 -> 76.5 ms, scan/all columns 40.4 -> 237 ms; parquet/1 unchanged (525, 800 ms). The same as setting TOKIO_WORKER_THREADS=1, which older binaries take for the same effect. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e in the df bench A runtime with one worker still runs a query on two threads: block_on polls the plan's operators on the bench's thread while the worker runs the scan's tasks. The upstream Vortex file has a DataFusion projection above its scan, so it kept 1.67 threads busy that way, against ~1.1 for vortex-ohlc's, and the comparison flattered upstream. One partition now runs on a current-thread runtime, as an earlier prototype's harness does: the scan's tasks and the operators above it on one thread. Both files' 1-partition scans now keep 1.03-1.06 threads busy (samply, busy_threads.py). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ndencies vortex-ohlc, datafusion-vortex-ohlc, datafusion-vortex-prune and test-util are declared once in the root's [workspace.dependencies], next to the other path crates, and roaring with the other versions. The crates use them with `.workspace = true` instead of their own paths and version. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every benchmark takes 20 flat samples over 2 s after a 0.5 s warm-up, with no plots. Criterion's minimum of 100 samples took 25 s on a 250 ms needle; a run of 18 benchmarks now takes 64 s, about 3.6 s each against about 8 s. The check before timing streams the query's batches and sums DataFusion's row hashes (`with_hashes`, fixed seed), where it collected every batch (3 GB for a scan of all columns) and hashed each row in the row format with SipHash. The sum still ignores row order, as the readers' partitions need. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The zone pruner turned any error reading a file's zone maps into "no zone
maps" with `.ok()` and scanned the file whole. A failed read isn't cached,
so every query read them again. A bars file whose prices' zones were 1024
rows and its other columns' 65,536 hit this ("has zones of its own"):
nothing was pruned, and nothing failed. Its small needles were only 4-6 ms
slower, single-threaded.
Errors now reach the query: opening the file and reading its zone maps,
walking its layouts, building the pruning predicate (`try_build`; `build`
returned `None` for an error), and casting a zone's bounds to the table's
type. A file without zone maps is still scanned whole. The bars layout's
own zones (`source_zones`, `read_bounds`, `ruled_out`) propagate their
errors the same way, where a failed read or proof dropped its pruning.
A file whose columns have zones of different lengths now fails its
queries; the next commit prunes it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…erent lengths A bars file can keep its prices' zones every 1024 rows while its other columns keep theirs once a chunk (`write_options_zoned`, new), which the pruner refused: it wanted one zone length per file. Each column now keeps its own, and a filter is pruned on the shortest. A column with longer zones has each zone's bounds for every short zone in it, so `ticker = 'T9' AND low < 20` rules out short zones by their low and long ones by their ticker. Every length must be a whole multiple of the shortest. On the bench's file (bars-25 then, bars-27 now) rewritten with 1024-row price zones, `high > 1000` selects a few 1024-row blocks, where before the previous commit the file was scanned whole. Small needles single-threaded now cost 0.2-0.6 ms more than with zones once a chunk, against 4-6 ms more before. Fine zones still don't pay, so the default stays once a chunk. Two costs remain: - Vortex 0.89 builds a split's mask from a roaring selection one row at a time (`dv > 1530000` +45 ms). - The bars layout reads its zone table on every scan. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`vortex_ohlc.dollar_volume` rebuilt `residual + volume × open` from three child expressions and knew nothing else of dollar volumes. It is now `vortex_ohlc.product_residual`, `residual + a × b`, in `product_residual.rs`, with its children named `residual`, `a` and `b`. The bars layout still stores the dollar volume against volume × open. Files are unchanged: they keep layout and encoding IDs, not scalar function IDs, and the bench reads the same rows from bars-25 (now bars-27) on all 23 queries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… two
The layout hard-coded the dollar volume against volume × open, with one
flag byte in its metadata. The writer now takes triples,
`ProductResidual { target, a, b }`, through
`BarsStrategy::with_product_residuals` (and `write_options`), with
`ProductResidual::dollar_volume()` the default. The metadata names each
triple's three columns, the target is stored as `<target>_residual`, and
it keeps its min and max in the layout's zones, as the dollar volume did.
A triple whose columns a table doesn't all have is left out of its file.
One it has but can't store fails the write: a price as the target, a
factor that is a target itself, a target stored twice, a table that
already has `<target>_residual`, or a target whose scale is below a × b's.
Before, a dollar volume at too small a scale was stored as it was without
a word. `a × b` now counts as 0 where either factor is null, not only `a`.
A format change, so the bench's file is bars-28. Measured as bars-26, before
the stack's dividing encodings became one: 32 bytes bigger than bars-25 (the
names), and the same rows on all 23 queries. Its
dollar volume queries time the same, within 3%, except the 1 ms
`AAPL & dv` needle single-threaded at +5% and +7% over two runs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…10^k Six places read a decimal array's unscaled values as i128s with which are valid, each building the validity mask and calling `converted_buffer` itself: the bars layout's prices, the product residual's column and factors, and the decimal scheme's exponent and division. They share `unscaled` in the new `decimal.rs`, and the product residual's two copies of a checked `10^k` share `checked_pow10`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e others They registered vortex-ohlc on a default session themselves; they now take `testing::session()`, as the Scaled, decimal and gcd tests do, which also enables the edition. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`selected_rows` repeated `selected`'s walk of a plan's Vortex scans. Both now map `selections`, each file's selection if it was given one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each `ohlc_price` decodes the parts its price uses, and the stored table's reader evaluated each price of a projection on its own: a whole row decoded the open four times and the body three, each time filtered to the rows read, after the reader had split the four expressions over the struct's fields. A projection that rebuilds more than one price now reads the stored columns it needs as they are, decodes each part its prices use once (`predecoded`: a `Scaled`'s values executed, its dtype kept), and applies the rewritten projection to that. A filter, or a projection of one price, is read as before. Timed once each against d5eda73 on all 23 queries, with the same rows as Parquet: 13.5% faster single-threaded (geometric mean) and 11% on 14 partitions, from 0% (`dollar_volume`, which reads no price) to -22% (`dv > 206000`) on one partition. One samply run over the 23 queries per binary shows where: on `dv > 206000` the prices' decoding goes from 96 ms to 57 ms a query; on the small needles the projection's partitioning, 6% of `AAPL` and 11% of `AAPM (miss)`, all but goes, and so does most of the rebuild. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Gcd scheme cached a chunk's divisor with `.ok()`, so an error computing it skipped the scheme as if the chunk had no common divisor, and its seconds were silently stored undivided. The cache now keeps the error, `compress` returns it, and the estimate fails with it through `failed_estimate`, as Pow10 and blocked FoR do. On success nothing changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The notes in `decoded`, the price rebuild and `product_residual` read as if Vortex enforced one width per decimal precision (vortex#10701) and decoded DecimalByteParts to a `Widen`. 0.91 has the `Widen` array but still accepts any width and decodes them to a plain decimal at their stored width, so nothing needs changing until a release enforces it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MassivePizza
force-pushed
the
vortex-dv-residual
branch
from
November 9, 2026 17:27
cc0914e to
64ee066
Compare
`widened` matched on each part's stored width and widened it with a loop of its own. Vortex's `converted_buffer` does the same widening, with no copy for a part already stored at the rebuild's width, and checks values stored wider than it, which the old cast wrapped. Bringing a part at a coarser scale to the finest one, a multiply by a power of ten, stays here, in place on the converted buffer. Vortex's loop is vectorized for i8, i16 and i32 to i64 in 0.89 (this branch) and 0.93 alike. Timed on the Vortex 0.93 bump further up the stack (dft CLI, medians): high = 50 70.48 -> 70.51 ms, open 400-401 2.321 -> 2.310 ms, with the same output. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.