Repository navigation
[opt](search) Start a SEARCH conjunct from the rows the scan's other filters kept - #70678
Open
guptashivam04 wants to merge 1 commit into
Open
guptashivam04 wants to merge 1 commit into
guptashivam04 wants to merge 1 commit into
Conversation
…filters kept `account_id = X AND search(...)` evaluated the SEARCH over the whole segment: the scan hides its candidate rows from roots that combine three-valued bitmaps, SEARCH included, and only intersected the account's rows afterwards. A SEARCH that is a whole filtering conjunct answers two-valued, and the scan reads its result only within the candidate rows. The scan now hands those rows to SEARCH, at its root only (the leaves still see none), and an unscored SEARCH collects its tree on them: a root that lists its rows lists them within the candidates, any other root is read through its scorer within them. The rows come from whatever narrowed the scan before: an inverted index on account_id, or zone maps and the short key. A result read this way is partial and stays out of the DSL cache. Scored SEARCH is unchanged. On 2M rows, with an inverted index on account_id, a ranking query of 16 SEARCH scans for an account with 648 rows (0.03%) went from 1648 ms to 1008 ms (CPU 7.9 s to 5.1 s); for an account at 2.5%, 2002 ms to 1763 ms. Rows and scores are the same.
guptashivam04
requested review from
airborne12,
csun5285,
eldenmoon,
gavinchou,
liaoxin01 and
yiguolei
as code owners
November 17, 2026 09:07
Closed
2 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Related PR: #70668 (split from it)
PR Type: Improvement
Problem Summary:
A SEARCH next to a selective filter outside it, as candidate search writes it:
evaluated the SEARCH over the whole segment, every account included, and only then intersected the account's rows.
SegmentIteratorhides its candidate rows from roots that combine three-valued bitmaps, and SEARCH is one of them, so the rowsaccount_id = 12345had already narrowed the scan to were never seen by SEARCH.What is changed?
IndexQueryContext::search_root_candidates. Its leaves still see none, so how they combine three-valued results does not change. A SEARCH used as a virtual column, or inside a compound predicate, is not affected.collect_truth_setalready handles both sparse and dense candidates).The candidate rows come from whatever narrowed the scan before SEARCH runs: an inverted index on
account_id, zone maps and the short key, the delete bitmap.Numbers
2M rows, 3 BEs, inverted index on
account_id, caches off, median of 3 runs. Rows and scores are byte-identical on 19 SEARCH queries (unscored and scored, lucene and standard mode, top-k), with the inverted index query cache off and on.score()queries (6 shapes, incl. top-k)A SEARCH whose root is an AND of OR groups gains little here, because those groups still evaluate their children over the segment; a follow-up lets nested groups read only the rows they are handed.
Release note
None
Check List (For Author)
search/test_search_outer_filter_candidates: on a table with an inverted index on the account and on one with the account leading the sort key, the rows ofaccount = 7 AND search(...)against the full SEARCH filtered afterwards, in lucene and standard mode, with AND, OR and NOT; scores against the full SEARCH; with the query cache on, the filtered query first, so a partial result would reach the unfiltered one if it were cached.search/andinverted_index_p0suites pass (298), except 3 that fail the same way without this change on the same machine (custom analyzer dictionaries and replica setup missing locally).BooleanTruthContractTest.CollectionWithinCandidatesKeepsTheirMatches: an unscored collection given candidates keeps exactly the matches among them, for a weight that lists its rows and for one read through its scorer.