Skip to content

feat(scanner): serve ORDER BY LIMIT from a single-column B-tree - #11124

Open
Ali2Arslan wants to merge 1 commit into
lance-format:mainfrom
Ali2Arslan:feat/scanner-ordered-index-read
Open

Ali2Arslan wants to merge 1 commit into
lance-format:mainfrom
Ali2Arslan:feat/scanner-ordered-index-read

Conversation

@Ali2Arslan

Copy link
Copy Markdown
Contributor

Part of #10250, which lays out the ordered index scan design and the PR sequence. This is O3 there.

Why

ORDER BY <column> LIMIT n sorts every row, even when a B-tree on that column already holds the keys in order. This PR reads the B-tree in key order instead and stops after offset + limit live rows. It builds on #10777, which merged and added OrderedIndexScanExec over the B-tree ordered search from #10346.

What changed

  • create_plan_impl in scanner/plan.rs asks the new Scanner::try_ordered_read for an OrderedIndexScanExec before it plans a filtered read. When it gets one, the plan has no SortExec. The existing take runs on top of the node, and the limit either stays above it or folds into the node's fetch.
  • The scan reads in order only when it orders by one top-level column, has a limit, and has no filter or a filter that the same B-tree answers exactly, on a dataset without stable row IDs. The B-tree must be keyed on that column alone and have one segment that covers every fragment. In every other case, including any error while it opens the index, the plan is the same as before.
  • The read picks a B-tree by its key, IndexMetadata::keyed_field(). Since feat(index): support independent covering fields #10662, a covered B-tree lists its carried columns after its key in IndexMetadata::fields, so the read compares the key the same way OrderedIndexScanExec::try_new does after feat(exec): add OrderedIndexScanExec over B-tree ordered search #10777's covered-segment fix. A single-key B-tree with covering columns serves the read. An overlay on a carried column keeps the ordered read, and an overlay on the key makes the scan sort.
  • The ordered_index mode controls the read, with auto (the default) and off. Rust has OrderedIndexMode::{Off, Auto} and Scanner::ordered_index(mode). Python takes ordered_index="off" on scanner, to_table, to_pandas, to_batches, and ScannerBuilder.ordered_index. Java takes ScanOptions.Builder.orderedIndex(ScanOptions.OrderedIndexMode.OFF). Python passes its string and Java passes toRustString(), and Rust parses both. The parser ignores ASCII case and rejects any other name with an error that lists off, auto. off plans the same sort as a scan without this PR. A later PR in feature: composite B-tree keys and ordered index scans #10250 adds required, which fails instead of sorting.
  • The ordered read skips every index named in ignored_scalar_indices. If no other B-tree on the order column qualifies, the scan sorts, as it does for the other declines. Ignoring an index on another column leaves the plan unchanged. The Rust, Python, and Java docs for the ignore option and for the ordered_index mode say that an ignored index never serves the read.
  • The ordered_limit bench group in scalar_index.rs times auto, off, and a declined query.
  • OrderedIndexScanExec now builds its key field from the column's Arrow field, so a projected key keeps its field metadata.

Scope

  • One index segment over the whole dataset. Merging segments, residual filters, unindexed fragments, and the required mode are later PRs.
  • The scan also sorts for a fragment list other than the dataset's, row prefilters, include_deleted_rows, nested keys, and the legacy file format. A narrower fragment list could make the walk skip most of the index.
  • The scan also sorts on a dataset with stable row IDs. On such a dataset, an update can add the fragment it writes to the index's fragment bitmap without indexing its rows. An update by pylance 13.0.0 does this (bug: an update from 12.x or 13.x makes a 14.x B-tree miss rows on stable-row-id tables #10351). The coverage check then passes, and the ordered read skips the updated rows. A filter on that column already misses them, and the ordered read would spread the miss to every default ORDER BY ... LIMIT. Follow-up: lift this decline once bug: an update from 12.x or 13.x makes a 14.x B-tree miss rows on stable-row-id tables #10351 has a fence.

Blast radius

Every scan that orders by a B-tree column with a limit now reads the index by default, whether or not the B-tree declares covering columns. Rows with equal keys can come back in a different order than the sort returned, so a limit that cuts through a run of equal keys or nulls can return different rows from that run. If the index fails to open, the scan sorts and logs a warning, or a debug message when the error is NotSupported. If an index data file goes missing after the index opens, the scan fails with Not found, as a filter on that column already does. ordered_index="off" (OrderedIndexMode.OFF in Java) restores the sort.

Verification

  • cargo test -p lance --lib ordered: 151 passed, 0 failed, 1 ignored (a full-text-search reuse case that links its issue). 54 of them are this PR's scanner::ordered cases. They run on three fragments with deletions. Every decline case checks that the plan equals the off plan and that the rows match under auto and off.
  • cargo test -p lance --lib scanner passes 1,428 tests with 1 ignored. cargo test -p lance --lib passes 10,827 tests with 9 ignored and none failed.
  • test_ordered_read_serves_a_covered_index_by_its_key declares id as a covering column of the B-tree on ts. The scan plans OrderedIndexScan: index=ts_idx with the same rows as off. After an overlay changes id in one of the first ten rows, it still reads in order and returns the new value. After an overlay on ts, it plans the same sort as off. With the candidate loop comparing IndexMetadata::fields to the order column instead of keyed_field(), only this test fails, because auto plans SortExec: TopK(fetch=10).
  • Three more reverts show that the tests pin each guard. Without the ignore-list check in the candidate loop, only test_ordered_read_skips_ignored_indices::case_1_order_column_index fails, because auto plans OrderedIndexScan: index=ts_idx where off plans SortExec: TopK(fetch=10). With off dropped from the decline, 23 of the 54 cases fail. Those are the cases where auto plans OrderedIndexScan, because off then plans it too. With the parser mapping any unknown name to auto, the 3 rejection cases fail (sorted, the empty string, and auto).
  • On an earlier head of this branch, the stable-row-ID case rebuilt the bug: an update from 12.x or 13.x makes a 14.x B-tree miss rows on stable-row-id tables #10351 state in Rust. It updates six rows into a new fragment, then commits the index with a bitmap that claims that fragment. Without the decline, the case failed on the plan, and the ordered read returned keys [10, 12, 13, 14, 15, 17, 18, 19, 20, 21] where the sort returned [3, 4, 6, 7, 8, 9, 10, 12, 13, 14].
  • On an earlier head, a build with seven guards removed, from the sort skip to the NotSupported fallback, failed 32 tests, each one predicted. The 9 that still passed decline for structural reasons.
  • cargo fmt --all -- --check and cargo clippy --all --tests --benches -- -D warnings are clean. python ci/check_module_size.py passes and adds no exception, with a warning that python/src/dataset.rs grows from 4,786 to 4,791 lines. cargo test -p lance --doc scanner::ordered passes the 2 doctests for OrderedIndexMode and Scanner::ordered_index, and RUSTDOCFLAGS="-D warnings" cargo doc -p lance --no-deps passes.
  • cargo test -p lance --bench scalar_index runs the whole bench in test mode and reports Success for all 8 cases, including the 3 ordered_limit cases.
  • Python: uv run make lint is clean: ruff, pyright with 0 errors, cargo fmt -- --check, and cargo clippy -- -D warnings. pytest python/tests/test_scalar_index.py passes the 4 cases of this PR: the default, ordered_index="off", use_scalar_index=False, and an unknown mode.
  • Java: ./mvnw test -Dtest=ScannerTest,JNITest passes 73 tests, including testOrderedIndex and testParseOrderedIndexMode. testOrderedIndex checks that both modes return the same rows and that only auto reports the pages_loaded counter. testParseOrderedIndexMode sends every Java constant and sorted through the Rust parser. cargo test for java/lance-jni passes 39 tests. ./mvnw spotless:check, plus cargo fmt and cargo clippy --tests -- -D warnings for java/lance-jni, are clean.

Each perf case below ran on an earlier head of this branch, before it merged main at c36ee5445. The cases were not rerun after the merge. Each ran in three interleaved rounds after one warmup, and every timed call returned 100 rows. The declined query adds the filter id >= 0, which the B-tree on ts cannot answer, so both modes sort.

  • The S3 runs used LocalStack 4.0 and a debug pylance build of that head, on 10,000,000 rows in 10 fragments with no stable row IDs. On a dataset where 1% of ts values are null, ORDER BY ts DESC LIMIT 100 returns the same 100 rows under auto and off. There the ordered read reads 811 KB, against 59 MB for the sort. On the bench's dataset shape, auto p50 is 0.212 s against 1.392 s for off, or 0.153 times. The declined query takes 1.957 s under auto against 2.025 s under off, or 0.966 times. Most of the ordered read's time is the take, 77 range reads from the object store after a 10.5 ms index walk that loads one page.
  • The local runs used a release-with-debug build of the ordered_limit bench at that head, on its dataset of 10,000,000 rows in 10 fragments with no stable row IDs. A scratch loop in place of criterion interleaved the ordered and declined queries under both modes. auto p50 is 9.01 ms against 70.1 ms for off, or 0.129 times. The declined query takes 84.9 ms under auto against 88.7 ms under off, or 0.957 times. A second run of the same binary gave ratios of 0.131 and 0.972. Most of the ordered read's time is the take, 98 range reads from the page cache after a 1.1 ms index walk that loads one page.

The plan with this PR:

lance 15.0.0-beta.22, 100,000 rows in 10 fragments, with deletions
>>> ds.scanner(order_by=order_by, limit=10).explain_plan()
ProjectionExec: expr=[ts@0 as ts, payload@2 as payload]
  LanceRead: uri=<tmp>/data/data, projection=[payload], source=stream(_rowaddr)
    OrderedIndexScan: index=ts_idx, order=[ts DESC NULLS LAST], fetch=10

>>> ds.scanner(order_by=order_by, limit=10, ordered_index='off').explain_plan()
SortExec: TopK(fetch=10), expr=[ts@0 DESC NULLS LAST], preserve_partitioning=[false]
  LanceRead: uri=<tmp>/data/data, projection=[ts, payload], num_fragments=10, range_before=None, range_after=None, row_id=false, row_addr=false, full_filter=--, refine_filter=--

>>> ts of the first 10 rows, ordered read: [99990, 99989, 99988, 99987, 99986, 99985, 99984, 99983, 99982, 99981]
>>> equal to ordered_index='off': True
>>> ds.scanner(..., ordered_index='sorted'): ValueError: Invalid user input: unknown ordered_index mode 'sorted', expected one of: off, auto, <source location>

ORDER BY <column> LIMIT n sorts every row even when a B-tree on that
column already holds the keys in order. Read the B-tree in key order
through OrderedIndexScanExec and stop after offset + limit live rows.

The scanner picks a B-tree by its key field, so a single-key B-tree with
covering columns serves the read. It falls back to the sort when the
index does not cover every fragment, when a fragment has an overlay on
the key, on datasets with stable row ids, and for indices named in
with_ignored_scalar_indices.

Add an ordered_index mode (off, auto; default auto) to the Rust scanner,
Python scanner arguments, and Java ScanOptions. Off plans the same sort
as before.

Co-authored-by: Cursor <cursoragent@cursor.com>

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@github-actions github-actions Bot added A-python Python bindings A-java Java bindings + JNI enhancement New feature or request labels Nov 16, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Gate recommendation: approve.

No blocking issues found. The eligibility guards keep the optimization within the existing ordered executor’s contract and preserve the sort fallback for other scans.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Nov 16, 2026
@Ali2Arslan

Copy link
Copy Markdown
Contributor Author

@Xuanwo @wjones127 could one of you review this when you have time? It's O3 in #10250 and builds on #10346 and #10777, which you reviewed. CI is green, it merges cleanly with main, and O4 through O7 are stacked behind it.

@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. and removed K-approved Latest Gatekeeper recommendation permits acceptance. labels Nov 18, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-java Java bindings + JNI A-python Python bindings enhancement New feature or request K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant