Repository navigation
[opt](search) Read only the rows the AND and the scan's filters keep in an unscored SEARCH - #70668
Closed
guptashivam04 wants to merge 5 commits into
Closed
guptashivam04 wants to merge 5 commits into
guptashivam04 wants to merge 5 commits into
Conversation
added 5 commits
November 17, 2026 02:47
An OR of many phrases on one field, such as a list of job titles, ran as one phrase query per phrase. Each opened, decoded and intersected its own copy of every word, so a word shared by ten phrases was read ten times, and each read positions on the rows where its own words meet. Inside an AND, a phrase asked for a row it does not match ran ahead to its next real match, checking positions on rows the other clauses had already ruled out. An unscored SEARCH now compiles two or more phrases of one field under an OR (or an OCCUR of SHOULD clauses that one match satisfies) as a phrase set: - every distinct word is opened once for the whole set; - a row is a candidate when it holds every word of at least one phrase, which the words' document lists answer without positions; - positions are read once per word and row, only on candidates, and the first phrase that matches settles the row; - in an AND, the set only checks the rows the other clauses kept, and a word far more common than those rows seeks to each of them instead of reading its whole document list; - a row where the field is NULL stays UNKNOWN, as for each phrase. The set holds a document list per distinct word and per phrase. When those could pass 32 MB on a segment (many common words on a very large segment), the phrases run as separate phrase queries, as before. A scored SEARCH, and a source that batches its reads, keep the phrase queries, so score() does not change. The DSL does not change.
…m keeps An unscored SEARCH evaluated every group under an AND over the whole segment, however few rows the other groups left. In lucene mode (the default) the AND is doc-at-a-time, and each OR group is a buffered union that fills a window of rows from every child, so a phrase group checked positions on all its rows. In standard mode the AND hands its rows only to leaves, and a nested OR built its whole bitmap first. Unscored occur and operator Booleans now list their rows for a set of candidates. The required clauses run first, cheapest scorers first, and each later clause reads only the rows the earlier ones kept. An OR skips the rows an earlier clause already matched. MUST_NOT and NOT read only what is left. Lucene mode stays two-valued, standard mode keeps its NULL rows. On a 2M-row table, a query shaped like account_id:X AND (13 titles on 2 fields) AND (20 skills on 2 fields) AND (location terms and phrases) went from 106 ms to 35 ms, and to 16 ms with the location group second. Rows are the same.
An unscored AND reads its clauses that list their own rows one after another, each only on the rows the ones before kept. They ran in the order the query wrote them, so a broad group written first (skills on long descriptions) was read over every row before a rare one (a location) could narrow it. Weights now give a row estimate from the term dictionary, without reading postings: a term its doc freq, a phrase its rarest word, a phrase set the sum over its phrases, an OR the sum of its clauses and an AND its rarest clause. Occur and operator conjunctions read their row-listing clauses from the smallest estimate up. A CLucene source keeps each term's doc freq for the evaluation, so a word asked about by several phrases or nesting levels is looked up once. On 2M rows, a query of engineer titles AND broad skills AND a Philippines location, written in that order, went from 58 ms to 35 ms. Rows are the same.
…filters kept `account_id = X AND search(...)` evaluated the SEARCH over the whole segment: the scan hides its candidate rows from roots that combine three-valued bitmaps, SEARCH included, and only intersected the account's rows afterwards. A SEARCH that is a whole filtering conjunct answers two-valued, and the scan reads its result only within the candidate rows. The scan now hands those rows to SEARCH, at its root only (the leaves still see none), and an unscored SEARCH reads its tree on them. The rows come from whatever narrowed the scan before: an inverted index on account_id, or zone maps and the short key. A result read this way is partial and stays out of the DSL cache. Scored SEARCH is unchanged. The query above, written as is, on 2M rows with the account at 2.5%: 238 ms -> 23 ms with an index on account_id, 252 ms -> 27 ms with account_id leading the key. Rows are the same.
…ed filter uses A column whose predicates the inverted index fully answered is not read from its data pages unless it is output. Variant subcolumns never took this path: they share their variant's unique id in the scan's output set, so the scan could not tell an output subcolumn from a filter-only one, and read them all. When no column of the variant is output (and the scan has no virtual columns, whose inputs are not in the output set), a subcolumn now follows the same rule as any other column. A SEARCH or MATCH on a long text subcolumn no longer reads that text for every matching row. That rule counts the pushed-down conjuncts on a column by the node each slot sits under, so a conjunct that is the column itself or a cast of it (WHERE CAST(v['n'] AS BOOLEAN)) was not counted, and an index answering another conjunct on the column could skip its read. Such a conjunct now counts as one the index does not answer. This also fixes v['fs'] MATCH 'true' AND CAST(v['fs'] AS BOOLEAN), which returned no rows with the index on and the matching rows with it off. In a ranking query of 17 SEARCH scans, returning only candidate ids, the scans read about 140 MB of work_description per backend each; the query went from 566 ms to 384 ms and its scan CPU from 3.4 s to 1.25 s. Rows are the same.
guptashivam04
requested review from
airborne12,
csun5285,
eldenmoon,
gavinchou,
liaoxin01 and
yiguolei
as code owners
November 17, 2026 07:06
2 tasks done
Contributor
Author
|
Splitting this into smaller PRs as suggested. The first one is #70678 (scan candidates into SEARCH); the rest will follow one or two at a time. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Related PR: #70488 (same candidate-search query, numeric SEARCH leaves)
PR Type: Improvement
Problem Summary:
Candidate-search queries look like this, run without
score():Even though only a small share of the rows can match, each SEARCH did work in proportion to the whole segment:
account_id = Xoutside SEARCH did not narrow it. The scan hides its candidate rows from compound and SEARCH roots, so SEARCH evaluated every account and the scan intersected afterwards.BufferedUnions, which refill a ~4k-row window from every child, so phrase children checked positions on rows the other groups had already ruled out. A nested OR in standard mode built its full three-valued bitmap first.What is changed?
Five commits; each message has the details.
OccurBooleanWeightandOperatorBooleanWeightlist their rows within the candidates an AND hands them: required clauses first, OR children only on rows not yet matched, NOT within the candidates. Lucene mode stays two-valued, standard mode keeps its UNKNOWN rows, andminimum_should_match > 1keeps the scorer path.account_id, key ranges, zone maps, the delete bitmap). Only the root reads within them; a result read this way never enters the DSL cache; scored SEARCH is unchanged.v['fs'] MATCH 'true' AND CAST(v['fs'] AS BOOLEAN)returning no rows with the index on.Numbers
2M candidates, 3 BEs, one account at 2.5% of rows, caches off, median of 3 runs, before = base, after = this PR. Rows and scores returned are byte-identical in every case (19 queries, with the inverted index query cache off and on).
account_idoutsideresume_search, 40 phrases + 30 words, countscore()queries (6 shapes, incl. top-k)Release note
SEARCH without
score()reads only the rows the other conditions of the query can still match: an outer filter such asaccount_id = X, and the other groups of an AND. OR'ed phrases of one field share their word reads, and long variant text answered by the index is no longer read.Check List (For Author)
phrase_set_test: the set against each phrase run alone on 20 random indexes, with and without candidates, seeks, NULL rows on both source kinds, one open per common word for a few candidates.boolean_truth_contract_test: nested AND/OR/NOT within sparse and dense candidates against a scalar three-valued oracle; occur Booleans list exactly the scorer's rows on random trees; rarest-first ordering; collection within candidates.FunctionSearchTest,FunctionSearchNullTest,VCompoundPredScoreTest,BM25SimilarityTest,ListedTermsTest,SniiIndexReader*,SegmentIteratorApplyIndexExprTest,SegmentIteratorCandidatePushdownTest,CountEmitShortcut,InvertedIndexArrayTest,VariantColumnWriterReaderTest,MultiSegmentCollectorTest,DocSetCollectorTest.search/test_search_phrase_sets: rows of OR'ed phrases (phrases sharing words in either order, a word no row holds, NULL rows, text and variant fields, V2 and SNII, lucene and standard modes) against each phrase run alone and against MATCH_PHRASE;score()against the sum of the phrases' scores, at every row and on the top-k path; inside AND and AND NOT.search/test_search_outer_filter_candidates: rows against the full SEARCH filtered afterwards, scores unchanged, the cache never hands a partial result to an unfiltered query; index and sort-key tables.variant_p0/test_variant_index_filter_only_read: selected values (the filtered subcolumn, a sibling, the whole variant, nothing) against the index off and against a read by id; subcolumns also used by expressions no index answers.search/pass, andvariant_p0plusinverted_index_p0(467 suites) fail only the same 6 suites as the base build on the same machine (they depend on local setup: custom analyzer dictionaries, outfile, replicas).