Skip to content

branch-4.2: [fix](search) Keep numeric SEARCH leaves on the inverted index #70488 - #70544

Open
github-actions[bot] wants to merge 1 commit into
branch-4.2from
auto-pick-70488-branch-4.2
Open

github-actions[bot] wants to merge 1 commit into
branch-4.2from
auto-pick-70488-branch-4.2

Conversation

@github-actions

Copy link
Copy Markdown
Contributor

Cherry-picked from #70488

### Background

`search()` only runs through the inverted index; there is no row-by-row
path, and `vsearch.cpp` turns any index bypass into "SEARCH cannot fall
back to row execution" (added in #66052). For numeric fields (BKD
readers), `InvertedIndexIterator::read_from_index` first estimates the
hit count and returns `INVERTED_INDEX_BYPASS` when it is over
`inverted_index_skip_threshold` percent of the segment (50 by default),
so the predicate can be evaluated on rows instead. That bypass is useful
for normal predicates and MATCH, not for SEARCH.

### What problem does this PR solve?

Issue Number: None
Related PR: #64782 (same fix for the pre-refactor code, closed without
merge), #68758 (moved the scalar leaf into `ScalarLeafCompiler`, bug
kept)
PR Type: Bugfix

Problem Summary:

A SEARCH term on a numeric field fails when the value matches more than
half of a segment, for example an account id on a table loaded per
account:

```
SELECT count(*) FROM t WHERE search('v.account_id:8069')
ERROR: [E-6001]SEARCH cannot fall back to row execution: [E-6004]hit count: 350255, bkd inverted reached limit 50% , segment num rows:500000
```

`ScalarLeafCompiler::compile` now sets `skip_try` so SEARCH leaves
always read the index. MATCH and plain predicates keep the bypass.

### Details

- **Frequency:** Any SEARCH term on a numeric column or numeric variant
subcolumn whose value covers more than `inverted_index_skip_threshold`
percent of a segment. Common for tenant or account ids. On 4.1 it often
shows up only on the second run, once the exact hit count comes from the
query cache.
- **Root cause:** `ScalarLeafCompiler::compile` calls `read_from_index`
with `skip_try = false`, so the BKD hit-ratio check applies to SEARCH
even though SEARCH cannot evaluate rows.
- **Why existing tests missed it:** No SEARCH test runs a numeric term
that covers more than half a segment with the default threshold;
`test_search_inverted_is_null_pushdown` sets
`inverted_index_skip_threshold=0`, which turns the bypass off.
- **Fix:** Set `param.skip_try = true` before `read_from_index` in
`ScalarLeafCompiler::compile`. This is the only `read_from_index` call
on the SEARCH path.
- **Other paths and branches:** branch-4.1 has the same bug in
`FunctionSearch::build_leaf_query`; a separate PR picks this there.
Numeric range clauses (`field:[a TO b]`) are still unsupported in
SEARCH; that is not changed here.

### Testing

- New regression test `search/test_search_bkd_skip_threshold`: a MOW
table where one value covers 70% of the segment, on a variant subcolumn
and on a plain INT column, alone, combined with a text term, in standard
mode and with ORDER BY ... LIMIT, each run 3 times with the query cache
on. Fails before the fix with "SEARCH cannot fall back to row
execution", passes after. Ran locally against a cluster built from this
branch, and against the same commit without the fix.
-
`FunctionSearchTest.TestBuildLeafQueryVariantNestedIntUsesDirectIndexReader`
now checks that the scalar leaf passes `skip_try = true` to the
iterator. Compiled locally with the BE unit-test flags; not run locally
(no BE UT build), left to CI.
- Manual timing on 2M rows (index query cache and SQL cache off, median
of 11): a 20% numeric match 10 ms before vs 12 ms after (noise), the 70%
match fails before and takes 16 ms after; text SEARCH and plain filters
unchanged.

### Release note

Fix `search()` failing with "SEARCH cannot fall back to row execution"
when a term on a numeric field matches more than
`inverted_index_skip_threshold` percent of a segment.

### Check List (For Author)

- Behavior changed: Yes (see Details). SEARCH no longer follows
`inverted_index_skip_threshold` for numeric fields; queries that failed
now return results.
- Does this need documentation: No
- Pre-submission status: Partially verified: the BE unit test was
compiled but not run locally.

---------

Co-authored-by: Shivam Gupta <shivam.gupta@recruitcrm.io>
@github-actions
github-actions Bot requested a review from yiguolei as a code owner November 13, 2026 03:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant