Skip to content

Extended Vortex encodings (product_residual) - #8

Draft
MassivePizza wants to merge 21 commits into
vortex-pow10from
vortex-dv-residual
Draft

MassivePizza wants to merge 21 commits into
vortex-pow10from
vortex-dv-residual

Conversation

@MassivePizza

Copy link
Copy Markdown

No description provided.

@MassivePizza
MassivePizza added this pull request to stack #4 November 4, 2026 20:09
@MassivePizza MassivePizza changed the title Extended Vortex encodings (dollar-volume residual) Extended Vortex encodings (product_residual) Nov 5, 2026
@MassivePizza
MassivePizza force-pushed the vortex-dv-residual branch 2 times, most recently from 0c98372 to d5eda73 Compare November 5, 2026 20:53
@MassivePizza
MassivePizza force-pushed the vortex-dv-residual branch 2 times, most recently from 85b073c to cc0914e Compare November 9, 2026 17:17
MassivePizza and others added 20 commits November 9, 2026 18:20
What the plan's rescale did, dividing each decimal column by the trailing zeros
all its values have, `DecimalPow10Scheme` does a chunk at a time: a
`vortex_ohlc.scaled` decimal of one exponent k for the whole chunk, over
DecimalByteParts. Without it dollar_volume's Decimal(38, 12) is wider than an
i64 unscaled, so DecimalByteParts cuts it into an integer part and a u64 lower
part, and Pow10Scheme only divides the integer part. Every chunk's k turned out
to be its column's (parts 2, volume 6, dollar_volume 8, once 10), and after it
every chunk fits an i64, so dividing i128s a block at a time would gain nothing.

The scheme takes the place of Vortex's decimal scheme in write_options: both
always bid, and the first registered wins. It builds the DecimalByteParts itself
in the same cascade step, so Pow10Scheme and blocked FoR under it are as deep as
under Vortex's scheme (a level of its own would leave the quotients
uncompressed at MAX_CASCADE). With one divisor for every row, Scaled reads
through its values: filters, slices, takes, masks and nullability casts go
through to them, and compares with a constant are rewritten to the quotients.

This is the second half of d5c32fe, split from it when the stack was reordered
on 2026-11-04 (the first half drops the plan). It was then an encoding of its
own, the first to be called `vortex_ohlc.scaled`, with its values at the smaller
scale; the numbers below are from that version. Measured before the reorder:

Size (2006-08): 334.0 MiB without any rescale (bars-11, its parent here but for
the zones' 50 KB) -> 236.3 MiB (bars-13). Against the plan's rescale (bars-12)
that is +112 KB (+0.045%, about 48 bytes per chunk and column).

Speed against c54c0ab, with the plan's rescale, rounds 1-2 of 3 (round 3 shared
the CPU), median ms:
  scan/dollar_volume       1 partition  12.90 -> 5.18 (-60%), 14: -23%
  scan/all columns         1 partition  57.50 -> 46.55 (-19%), 14: -13%
  needles                  within -7% and +6%, both ways
Probably because the scans no longer rescale in DataFusion (a guess; not profiled).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A bar's dollar volume is volume × vwap, with the vwap between its low and high,
so dollar_volume - volume × open is small, and 0 in many of a month's bars.
Where a table of bars has a decimal dollar_volume and volume, the bars layout
stores that residual as dollar_volume_residual (a name no source column has,
so its zone map is never taken for dollar_volume's) and puts dollar_volume's
own bounds in its zones table. A null volume counts as 0.

On reading, a new scalar function, vortex_ohlc.dollar_volume, rebuilds
residual + volume × open. It looks through `scaled` to the narrow values, and
multiplies 64 × 64 -> 128 bits where they fit: 5 multiply instructions a row
instead of 10, by the disassembly. The test table gains a dollar_volume column.

Size: bars-15 is 206.7 MiB, -29.7 MiB (-12.6%); dollar_volume 75.6 -> 45.9 MiB.

Speed against bd9c937 (best of 2-3 rounds, ms; the machine was busy):
- scan/dollar_volume 5.21 -> 9.63 (+85%) at one partition, 9.66 -> 13.02
  (+35%) at 14. Without the 64-bit multiplies it was +108% and +48%.
- every dv > needle +14% to +23%.
- scan/all columns +0-3%, the other queries unchanged.

Reading dollar_volume now decodes volume and the open part as well: 28% of
scan/dollar_volume's profile, against 15% for the rebuild. Kept for the size.

Measured as 89c7bd8, before the stack was reordered on 2026-11-04; its parent there
(bd9c937) had the same tree as this one's but for the bench's version. Rewritten on
2026-11-05, when the stack's three dividing encodings became one Scaled: a Scaled
decimal's quotients keep its scale, so `decoded` labels them with the smaller one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The residual (89c7bd8) saves 29.7 MiB on a month of minute bars, but reading
dollar_volume then decodes volume and the open as well: filters on it were
14-23% slower, a scan of it alone 35-85%. Which matters more depends on the
table and how it's queried, so BarsStrategy::with_dollar_volume_residual turns
it off, and write_options takes the choice as a parameter. The bench keeps it
on.

The layout's metadata gains a byte that says which, since the stored table's
and the zones' dtypes no longer follow from the source's alone. Files written
before can't be read (bars-26, the same size as bars-25).

A round trip without the residual stores dollar_volume as it is and reads the
source back.

Measured as 54ad05e, before the stack was reordered on 2026-11-04; its parent there
(89c7bd8) had the same tree as this one's but for the bench's version.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Vortex stores a timestamp as DateTimeParts: days, seconds into the day and
subseconds. A minute bar's seconds are all multiples of 60, and a block of one
ticker's bars spans most of a trading day, so blocked FoR packed them at 16 bits
a row: 131 KB of window_start's 133 KB in a chunk.

`GcdScheme` stores a chunk of them divided by the greatest common divisor of
its values, as a `vortex_ohlc.scaled` array whose base is the divisor, to the
power 1, and takes only DateTimeParts' seconds (the compressor's cascade
history is the temporal scheme's child 1). Minute bars' seconds become minutes
into the day, which blocked FoR packs at 10 bits. an earlier prototype got there with
a copy of DateTimeParts with a fourth part; this keeps
Vortex's own. With one divisor for every row, Scaled reads through the
quotients: filters, slices, takes, masks and nullability casts go through to
them, and a compare with a constant is rewritten to them (`q op ⌊c / d⌋`), so
DateTimeParts' compare (days, then seconds) runs on the packed minutes.

It was first written as an encoding of its own, `vortex_ohlc.gcd`, before the
stack's dividing encodings became one Scaled on 2026-11-05; the numbers below
are from that version.

Size (2006-08): bars-25 (now bars-27) is 188.9 MiB, -17.8 MiB (-8.6%);
window_start 46.9 -> 29.1 MiB, against the earlier prototype's 30.1. Every other column is
byte-identical, and every bench query returns the same rows and bytes.

Speed against 15b05e4 (best of 3 alternating rounds, 1 / 14 partitions):
- the pruned AAPL window needles +2-3%, high > 1000 +2-3%;
- all @ 14:30 and all 14:30-14:35 +1% / -2%;
- high = 50, dv > 1530000, AAPL & close > 67, scan/all columns -2% to +0%.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…red at

Reading a price executed each part through `scaled`, which multiplies the
values by 10^k into an i128 buffer whatever their width. The rebuild then
narrowed each value to an i64, summed, and widened the price back into an
i128 buffer, so a filter on it compared i128s. For high = 50 the i128 buffers
alone were 15.7 of its 116 ms of CPU, against 1.1 for upstream's decode.

The price function now reads each part through `scaled` to its values at the
width they're stored at (dollar_volume's helper, moved to `scaled`), widens
each once into the rebuild's type (i64 for prices of at most 18 digits),
bringing parts at different scales to the finest of them, and rebuilds into a
buffer of that type. A compare then runs on i64s.

Speed against 10730a6 (best of 3 alternating rounds, ms, 1 / 14 partitions):
- high = 50 20.22 -> 17.46 (-14%) / -12%; low = 50 -14% / -12%;
- scan/all columns 45.8 -> 39.4 (-14%) / -13%; scan/decimal columns -17% at both;
- high > 1000, close > 400.5, AAPL & close > 67 -6% to -7%; open 400-401 -3%;
- AAPL, AAPL @ 14:30, all @ 14:30, dv > 1530000 and scan/dollar_volume within 2%.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Vortex's tracking issue #10701 gives every decimal leaf its precision's one
width and makes a narrower one a `Widen` encoding; #10905 and #10995 decode
DecimalByteParts and narrow `vortex.decimal` nodes into it. When we take it
up, `decoded` must look through `Widen` as it does `Scaled`, or the price and
dollar volume rebuilds get their parts copied out wide. (`Scaled`'s decode into
i128 buffers, which become invalid for precisions of 18 digits or fewer, has
its note from the commit that let it hold decimals.)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The bench's "1 partition" only set DataFusion's target partitions. Vortex's
scan keeps scan_concurrency (4) x cores splits of a partition in flight and
spawns each onto the runtime's workers (vortex-layout 0.89
repeated_scan.rs:243), so on the bench's runtime, a worker per core, one
partition kept ~8 of 14 threads busy, while Parquet's reader keeps a
partition on one thread. Vortex's lead over Parquet at one partition was
mostly threads, and its ratios couldn't be compared with an earlier prototype's
single-threaded harness.

One partition now runs on a runtime of its own with one worker; a partition
per core keeps a worker per core. At 817e918, vortex/1: high = 50 18.1 -> 76.5
ms, scan/all columns 40.4 -> 237 ms; parquet/1 unchanged (525, 800 ms). The
same as setting TOKIO_WORKER_THREADS=1, which older binaries take for the same
effect.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e in the df bench

A runtime with one worker still runs a query on two threads: block_on polls
the plan's operators on the bench's thread while the worker runs the scan's
tasks. The upstream Vortex file has a DataFusion projection above its scan,
so it kept 1.67 threads busy that way, against ~1.1 for vortex-ohlc's, and
the comparison flattered upstream.

One partition now runs on a current-thread runtime, as an earlier prototype's harness
does: the scan's tasks and the operators above it on one thread. Both files'
1-partition scans now keep 1.03-1.06 threads busy (samply, busy_threads.py).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ndencies

vortex-ohlc, datafusion-vortex-ohlc, datafusion-vortex-prune and test-util
are declared once in the root's [workspace.dependencies], next to the
other path crates, and roaring with the other versions. The crates use
them with `.workspace = true` instead of their own paths and version.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every benchmark takes 20 flat samples over 2 s after a 0.5 s warm-up, with
no plots. Criterion's minimum of 100 samples took 25 s on a 250 ms needle;
a run of 18 benchmarks now takes 64 s, about 3.6 s each against about 8 s.

The check before timing streams the query's batches and sums DataFusion's
row hashes (`with_hashes`, fixed seed), where it collected every batch (3 GB
for a scan of all columns) and hashed each row in the row format with SipHash.
The sum still ignores row order, as the readers' partitions need.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The zone pruner turned any error reading a file's zone maps into "no zone
maps" with `.ok()` and scanned the file whole. A failed read isn't cached,
so every query read them again. A bars file whose prices' zones were 1024
rows and its other columns' 65,536 hit this ("has zones of its own"):
nothing was pruned, and nothing failed. Its small needles were only 4-6 ms
slower, single-threaded.

Errors now reach the query: opening the file and reading its zone maps,
walking its layouts, building the pruning predicate (`try_build`; `build`
returned `None` for an error), and casting a zone's bounds to the table's
type. A file without zone maps is still scanned whole. The bars layout's
own zones (`source_zones`, `read_bounds`, `ruled_out`) propagate their
errors the same way, where a failed read or proof dropped its pruning.

A file whose columns have zones of different lengths now fails its
queries; the next commit prunes it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…erent lengths

A bars file can keep its prices' zones every 1024 rows while its other
columns keep theirs once a chunk (`write_options_zoned`, new), which the
pruner refused: it wanted one zone length per file. Each column now keeps
its own, and a filter is pruned on the shortest. A column with longer
zones has each zone's bounds for every short zone in it, so
`ticker = 'T9' AND low < 20` rules out short zones by their low and long
ones by their ticker. Every length must be a whole multiple of the
shortest.

On the bench's file (bars-25 then, bars-27 now) rewritten with 1024-row
price zones, `high > 1000` selects a few 1024-row blocks, where before the previous
commit the file was scanned whole. Small needles single-threaded now cost
0.2-0.6 ms more than with zones once a chunk, against 4-6 ms more before.
Fine zones still don't pay, so the default stays once a chunk. Two costs
remain:
- Vortex 0.89 builds a split's mask from a roaring selection one row at a
  time (`dv > 1530000` +45 ms).
- The bars layout reads its zone table on every scan.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`vortex_ohlc.dollar_volume` rebuilt `residual + volume × open` from three
child expressions and knew nothing else of dollar volumes. It is now
`vortex_ohlc.product_residual`, `residual + a × b`, in
`product_residual.rs`, with its children named `residual`, `a` and `b`.
The bars layout still stores the dollar volume against volume × open.

Files are unchanged: they keep layout and encoding IDs, not scalar
function IDs, and the bench reads the same rows from bars-25 (now bars-27)
on all 23 queries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… two

The layout hard-coded the dollar volume against volume × open, with one
flag byte in its metadata. The writer now takes triples,
`ProductResidual { target, a, b }`, through
`BarsStrategy::with_product_residuals` (and `write_options`), with
`ProductResidual::dollar_volume()` the default. The metadata names each
triple's three columns, the target is stored as `<target>_residual`, and
it keeps its min and max in the layout's zones, as the dollar volume did.

A triple whose columns a table doesn't all have is left out of its file.
One it has but can't store fails the write: a price as the target, a
factor that is a target itself, a target stored twice, a table that
already has `<target>_residual`, or a target whose scale is below a × b's.
Before, a dollar volume at too small a scale was stored as it was without
a word. `a × b` now counts as 0 where either factor is null, not only `a`.

A format change, so the bench's file is bars-28. Measured as bars-26, before
the stack's dividing encodings became one: 32 bytes bigger than bars-25 (the
names), and the same rows on all 23 queries. Its
dollar volume queries time the same, within 3%, except the 1 ms
`AAPL & dv` needle single-threaded at +5% and +7% over two runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…10^k

Six places read a decimal array's unscaled values as i128s with which are
valid, each building the validity mask and calling `converted_buffer` itself:
the bars layout's prices, the product residual's column and factors, and the
decimal scheme's exponent and division. They share `unscaled` in the new
`decimal.rs`, and the product residual's two copies of a checked `10^k` share
`checked_pow10`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e others

They registered vortex-ohlc on a default session themselves; they now take
`testing::session()`, as the Scaled, decimal and gcd tests do, which also
enables the edition.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`selected_rows` repeated `selected`'s walk of a plan's Vortex scans. Both now
map `selections`, each file's selection if it was given one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each `ohlc_price` decodes the parts its price uses, and the stored table's
reader evaluated each price of a projection on its own: a whole row decoded
the open four times and the body three, each time filtered to the rows read,
after the reader had split the four expressions over the struct's fields.

A projection that rebuilds more than one price now reads the stored columns
it needs as they are, decodes each part its prices use once (`predecoded`:
a `Scaled`'s values executed, its dtype kept), and applies the rewritten
projection to that. A filter, or a projection of one price, is read as
before.

Timed once each against d5eda73 on all 23 queries, with the same rows as
Parquet: 13.5% faster single-threaded (geometric mean) and 11% on 14
partitions, from 0% (`dollar_volume`, which reads no price) to -22%
(`dv > 206000`) on one partition. One samply run over the 23 queries per
binary shows where: on `dv > 206000` the prices' decoding goes from 96 ms
to 57 ms a query; on the small needles the projection's partitioning, 6%
of `AAPL` and 11% of `AAPM (miss)`, all but goes, and so does most of the
rebuild.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Gcd scheme cached a chunk's divisor with `.ok()`, so an error
computing it skipped the scheme as if the chunk had no common divisor,
and its seconds were silently stored undivided. The cache now keeps the
error, `compress` returns it, and the estimate fails with it through
`failed_estimate`, as Pow10 and blocked FoR do. On success nothing
changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The notes in `decoded`, the price rebuild and `product_residual` read as
if Vortex enforced one width per decimal precision (vortex#10701) and
decoded DecimalByteParts to a `Widen`. 0.91 has the `Widen` array but
still accepts any width and decodes them to a plain decimal at their
stored width, so nothing needs changing until a release enforces it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`widened` matched on each part's stored width and widened it with a loop of
its own. Vortex's `converted_buffer` does the same widening, with no copy for
a part already stored at the rebuild's width, and checks values stored wider
than it, which the old cast wrapped. Bringing a part at a coarser scale to the
finest one, a multiply by a power of ten, stays here, in place on the
converted buffer.

Vortex's loop is vectorized for i8, i16 and i32 to i64 in 0.89 (this branch)
and 0.93 alike. Timed on the Vortex 0.93 bump further up the stack (dft CLI,
medians): high = 50 70.48 -> 70.51 ms, open 400-401 2.321 -> 2.310 ms, with
the same output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant