Repository navigation
branch-4.2: [fix](search) Keep numeric SEARCH leaves on the inverted index #70488 - #70544
Open
github-actions[bot] wants to merge 1 commit into
Open
github-actions[bot] wants to merge 1 commit into
github-actions[bot] wants to merge 1 commit into
Conversation
### Background `search()` only runs through the inverted index; there is no row-by-row path, and `vsearch.cpp` turns any index bypass into "SEARCH cannot fall back to row execution" (added in #66052). For numeric fields (BKD readers), `InvertedIndexIterator::read_from_index` first estimates the hit count and returns `INVERTED_INDEX_BYPASS` when it is over `inverted_index_skip_threshold` percent of the segment (50 by default), so the predicate can be evaluated on rows instead. That bypass is useful for normal predicates and MATCH, not for SEARCH. ### What problem does this PR solve? Issue Number: None Related PR: #64782 (same fix for the pre-refactor code, closed without merge), #68758 (moved the scalar leaf into `ScalarLeafCompiler`, bug kept) PR Type: Bugfix Problem Summary: A SEARCH term on a numeric field fails when the value matches more than half of a segment, for example an account id on a table loaded per account: ``` SELECT count(*) FROM t WHERE search('v.account_id:8069') ERROR: [E-6001]SEARCH cannot fall back to row execution: [E-6004]hit count: 350255, bkd inverted reached limit 50% , segment num rows:500000 ``` `ScalarLeafCompiler::compile` now sets `skip_try` so SEARCH leaves always read the index. MATCH and plain predicates keep the bypass. ### Details - **Frequency:** Any SEARCH term on a numeric column or numeric variant subcolumn whose value covers more than `inverted_index_skip_threshold` percent of a segment. Common for tenant or account ids. On 4.1 it often shows up only on the second run, once the exact hit count comes from the query cache. - **Root cause:** `ScalarLeafCompiler::compile` calls `read_from_index` with `skip_try = false`, so the BKD hit-ratio check applies to SEARCH even though SEARCH cannot evaluate rows. - **Why existing tests missed it:** No SEARCH test runs a numeric term that covers more than half a segment with the default threshold; `test_search_inverted_is_null_pushdown` sets `inverted_index_skip_threshold=0`, which turns the bypass off. - **Fix:** Set `param.skip_try = true` before `read_from_index` in `ScalarLeafCompiler::compile`. This is the only `read_from_index` call on the SEARCH path. - **Other paths and branches:** branch-4.1 has the same bug in `FunctionSearch::build_leaf_query`; a separate PR picks this there. Numeric range clauses (`field:[a TO b]`) are still unsupported in SEARCH; that is not changed here. ### Testing - New regression test `search/test_search_bkd_skip_threshold`: a MOW table where one value covers 70% of the segment, on a variant subcolumn and on a plain INT column, alone, combined with a text term, in standard mode and with ORDER BY ... LIMIT, each run 3 times with the query cache on. Fails before the fix with "SEARCH cannot fall back to row execution", passes after. Ran locally against a cluster built from this branch, and against the same commit without the fix. - `FunctionSearchTest.TestBuildLeafQueryVariantNestedIntUsesDirectIndexReader` now checks that the scalar leaf passes `skip_try = true` to the iterator. Compiled locally with the BE unit-test flags; not run locally (no BE UT build), left to CI. - Manual timing on 2M rows (index query cache and SQL cache off, median of 11): a 20% numeric match 10 ms before vs 12 ms after (noise), the 70% match fails before and takes 16 ms after; text SEARCH and plain filters unchanged. ### Release note Fix `search()` failing with "SEARCH cannot fall back to row execution" when a term on a numeric field matches more than `inverted_index_skip_threshold` percent of a segment. ### Check List (For Author) - Behavior changed: Yes (see Details). SEARCH no longer follows `inverted_index_skip_threshold` for numeric fields; queries that failed now return results. - Does this need documentation: No - Pre-submission status: Partially verified: the BE unit test was compiled but not run locally. --------- Co-authored-by: Shivam Gupta <shivam.gupta@recruitcrm.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-picked from #70488