Repository navigation
[BUG] Text search ignores the title: bm25(search_index) has no column weights #1851
Description
Bug Description
The text leg of search_notes scores with bm25(search_index) and no column weights
(repository/sqlite_search_query.py:265). The title is weighted like any body column, so a note whose
title is exactly the query ranks below notes that merely mention the words in their body. Hybrid
inherits the text rank.
In a real project, the note titled "API Documentation" ranks 23rd of 26 for the query
API Documentation in text mode (29th in hybrid). The notes ranked above it are longer progress,
planning and guideline notes that mention "API" and "documentation" in their body.
Steps To Reproduce
Throwaway home (BASIC_MEMORY_CONFIG_DIR pointing at an empty directory) with one project, then:
- Write one note titled
API Documentationwhose body does not repeat the title (e.g. a dozen
endpoint lines). - Write 20 notes titled
Meeting Nwhose body contains the line
Step 1: update the api documentation and link it from the plan. - Write 100 unrelated notes (
Journal N, "Weather was fine…") so the terms keep a real IDF. bm reindex -p <project> --searchbm tool search-notes "API Documentation" --json
Expected Behavior
The exact title match ranks first, or at least near the top.
Actual Behavior
It ranks 21st of 21, last. The order matches BM's own predicate ranked by plain SQL, and a title
weight alone moves the note to first place:
-- BM's predicate for a word query (sqlite_search_query.py:301-318), term from prepare_search_term()
SELECT title FROM search_index
WHERE (search_index.title MATCH 'API* AND Documentation*'
OR search_index.content_stems MATCH 'API* AND Documentation*'
OR search_index.content_snippet MATCH 'API* AND Documentation*')
AND type = 'entity'
ORDER BY bm25(search_index); -- rank 21 of 21
-- ORDER BY bm25(search_index, 1, 5, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1); -- rank 1 of 21The weights follow the column order of search_index: id, title, content_stems, content_snippet, script_ngrams, permalink, …. On the real project above, the same SQL reproduces BM's order exactly
(23 of 26, same scores), and a title weight of 5 moves the note to 1 of 26.
Environment
- OS: macOS 27.0.1
- Python version: 3.13.17
- Basic Memory version: 0.24.1
- Installation method: Homebrew
- Claude Desktop version (if applicable): n/a
Additional Context
Related, but they do not change the per-column weights: #1746 (search rework) and #1748 (fusion
benchmark). Because hybrid fuses the text rank, a fix here also lifts hybrid.
Possible Solution
Pass column weights to bm25() with the title above the body columns, e.g.
bm25(search_index, 1, 5, 1, 1, 1, 1, …). A weight of 5 was enough in both cases above, and 10 or
20 behaved the same. The count must match the table's column count. We only measured SQLite.
Activity
fix(core): weight the title above the body in the sqlite bm25 score
fix(core): score the filtered script path with the weighted bm25