Repository navigation
feat(scanner): serve ORDER BY LIMIT from a single-column B-tree - #11124
Open
Ali2Arslan wants to merge 1 commit into
Open
Ali2Arslan wants to merge 1 commit into
Ali2Arslan wants to merge 1 commit into
Conversation
ORDER BY <column> LIMIT n sorts every row even when a B-tree on that column already holds the keys in order. Read the B-tree in key order through OrderedIndexScanExec and stop after offset + limit live rows. The scanner picks a B-tree by its key field, so a single-key B-tree with covering columns serves the read. It falls back to the sort when the index does not cover every fragment, when a fragment has an overlay on the key, on datasets with stable row ids, and for indices named in with_ignored_scalar_indices. Add an ordered_index mode (off, auto; default auto) to the Rust scanner, Python scanner arguments, and Java ScanOptions. Off plans the same sort as before. Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
Author
|
@Xuanwo @wjones127 could one of you review this when you have time? It's O3 in #10250 and builds on #10346 and #10777, which you reviewed. CI is green, it merges cleanly with main, and O4 through O7 are stacked behind it. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #10250, which lays out the ordered index scan design and the PR sequence. This is O3 there.
Why
ORDER BY <column> LIMIT nsorts every row, even when a B-tree on that column already holds the keys in order. This PR reads the B-tree in key order instead and stops afteroffset + limitlive rows. It builds on #10777, which merged and addedOrderedIndexScanExecover the B-tree ordered search from #10346.What changed
create_plan_implinscanner/plan.rsasks the newScanner::try_ordered_readfor anOrderedIndexScanExecbefore it plans a filtered read. When it gets one, the plan has noSortExec. The existing take runs on top of the node, and the limit either stays above it or folds into the node'sfetch.IndexMetadata::keyed_field(). Since feat(index): support independent covering fields #10662, a covered B-tree lists its carried columns after its key inIndexMetadata::fields, so the read compares the key the same wayOrderedIndexScanExec::try_newdoes after feat(exec): add OrderedIndexScanExec over B-tree ordered search #10777's covered-segment fix. A single-key B-tree with covering columns serves the read. An overlay on a carried column keeps the ordered read, and an overlay on the key makes the scan sort.ordered_indexmode controls the read, withauto(the default) andoff. Rust hasOrderedIndexMode::{Off, Auto}andScanner::ordered_index(mode). Python takesordered_index="off"onscanner,to_table,to_pandas,to_batches, andScannerBuilder.ordered_index. Java takesScanOptions.Builder.orderedIndex(ScanOptions.OrderedIndexMode.OFF). Python passes its string and Java passestoRustString(), and Rust parses both. The parser ignores ASCII case and rejects any other name with an error that listsoff, auto.offplans the same sort as a scan without this PR. A later PR in feature: composite B-tree keys and ordered index scans #10250 addsrequired, which fails instead of sorting.ignored_scalar_indices. If no other B-tree on the order column qualifies, the scan sorts, as it does for the other declines. Ignoring an index on another column leaves the plan unchanged. The Rust, Python, and Java docs for the ignore option and for theordered_indexmode say that an ignored index never serves the read.ordered_limitbench group inscalar_index.rstimesauto,off, and a declined query.OrderedIndexScanExecnow builds its key field from the column's Arrow field, so a projected key keeps its field metadata.Scope
requiredmode are later PRs.include_deleted_rows, nested keys, and the legacy file format. A narrower fragment list could make the walk skip most of the index.ORDER BY ... LIMIT. Follow-up: lift this decline once bug: an update from 12.x or 13.x makes a 14.x B-tree miss rows on stable-row-id tables #10351 has a fence.Blast radius
Every scan that orders by a B-tree column with a limit now reads the index by default, whether or not the B-tree declares covering columns. Rows with equal keys can come back in a different order than the sort returned, so a limit that cuts through a run of equal keys or nulls can return different rows from that run. If the index fails to open, the scan sorts and logs a warning, or a debug message when the error is
NotSupported. If an index data file goes missing after the index opens, the scan fails withNot found, as a filter on that column already does.ordered_index="off"(OrderedIndexMode.OFFin Java) restores the sort.Verification
cargo test -p lance --lib ordered: 151 passed, 0 failed, 1 ignored (a full-text-search reuse case that links its issue). 54 of them are this PR'sscanner::orderedcases. They run on three fragments with deletions. Every decline case checks that the plan equals theoffplan and that the rows match underautoandoff.cargo test -p lance --lib scannerpasses 1,428 tests with 1 ignored.cargo test -p lance --libpasses 10,827 tests with 9 ignored and none failed.test_ordered_read_serves_a_covered_index_by_its_keydeclaresidas a covering column of the B-tree onts. The scan plansOrderedIndexScan: index=ts_idxwith the same rows asoff. After an overlay changesidin one of the first ten rows, it still reads in order and returns the new value. After an overlay onts, it plans the same sort asoff. With the candidate loop comparingIndexMetadata::fieldsto the order column instead ofkeyed_field(), only this test fails, becauseautoplansSortExec: TopK(fetch=10).test_ordered_read_skips_ignored_indices::case_1_order_column_indexfails, becauseautoplansOrderedIndexScan: index=ts_idxwhereoffplansSortExec: TopK(fetch=10). Withoffdropped from the decline, 23 of the 54 cases fail. Those are the cases whereautoplansOrderedIndexScan, becauseoffthen plans it too. With the parser mapping any unknown name toauto, the 3 rejection cases fail (sorted, the empty string, andauto).[10, 12, 13, 14, 15, 17, 18, 19, 20, 21]where the sort returned[3, 4, 6, 7, 8, 9, 10, 12, 13, 14].NotSupportedfallback, failed 32 tests, each one predicted. The 9 that still passed decline for structural reasons.cargo fmt --all -- --checkandcargo clippy --all --tests --benches -- -D warningsare clean.python ci/check_module_size.pypasses and adds no exception, with a warning thatpython/src/dataset.rsgrows from 4,786 to 4,791 lines.cargo test -p lance --doc scanner::orderedpasses the 2 doctests forOrderedIndexModeandScanner::ordered_index, andRUSTDOCFLAGS="-D warnings" cargo doc -p lance --no-depspasses.cargo test -p lance --bench scalar_indexruns the whole bench in test mode and reportsSuccessfor all 8 cases, including the 3ordered_limitcases.uv run make lintis clean: ruff, pyright with 0 errors,cargo fmt -- --check, andcargo clippy -- -D warnings.pytest python/tests/test_scalar_index.pypasses the 4 cases of this PR: the default,ordered_index="off",use_scalar_index=False, and an unknown mode../mvnw test -Dtest=ScannerTest,JNITestpasses 73 tests, includingtestOrderedIndexandtestParseOrderedIndexMode.testOrderedIndexchecks that both modes return the same rows and that onlyautoreports thepages_loadedcounter.testParseOrderedIndexModesends every Java constant andsortedthrough the Rust parser.cargo testforjava/lance-jnipasses 39 tests../mvnw spotless:check, pluscargo fmtandcargo clippy --tests -- -D warningsforjava/lance-jni, are clean.Each perf case below ran on an earlier head of this branch, before it merged main at
c36ee5445. The cases were not rerun after the merge. Each ran in three interleaved rounds after one warmup, and every timed call returned 100 rows. The declined query adds the filterid >= 0, which the B-tree ontscannot answer, so both modes sort.tsvalues are null,ORDER BY ts DESC LIMIT 100returns the same 100 rows underautoandoff. There the ordered read reads 811 KB, against 59 MB for the sort. On the bench's dataset shape,autop50 is 0.212 s against 1.392 s foroff, or 0.153 times. The declined query takes 1.957 s underautoagainst 2.025 s underoff, or 0.966 times. Most of the ordered read's time is the take, 77 range reads from the object store after a 10.5 ms index walk that loads one page.release-with-debugbuild of theordered_limitbench at that head, on its dataset of 10,000,000 rows in 10 fragments with no stable row IDs. A scratch loop in place of criterion interleaved the ordered and declined queries under both modes.autop50 is 9.01 ms against 70.1 ms foroff, or 0.129 times. The declined query takes 84.9 ms underautoagainst 88.7 ms underoff, or 0.957 times. A second run of the same binary gave ratios of 0.131 and 0.972. Most of the ordered read's time is the take, 98 range reads from the page cache after a 1.1 ms index walk that loads one page.The plan with this PR: