Skip to content

fix: load external fragment reuse index details from the correct base path - #9141

Merged
zhangyue19921010 merged 1 commit into
mainfrom
lu/fix-fri-external-details-base
Sep 11, 2026
Merged

zhangyue19921010 merged 1 commit into
mainfrom
lu/fix-fri-external-details-base

Conversation

@LuQQiu

@LuQQiu LuQQiu commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

The bug

When a fragment reuse index (FRI) entry's details exceed the 204800-byte inline threshold, they are written to an external file (_indices/<uuid>/details.binpb) and the index metadata carries an ExternalFile reference. load_frag_reuse_index_details resolved that reference against the current dataset root only (dataset.indices_dir() + dataset.object_store), ignoring the entry's base_id.

A shallow clone stamps its index metadata with a base_id pointing at the source dataset, and the external details.binpb stays in the source's indices directory: it is never copied into the clone. So on a clone, any path that loads the FRI details (opening the fragment reuse index, compaction, index remapping, conflict resolution) failed with a not-found error, because the loader looked for the file under the clone's own dataset root, which never contained it.

The fix

External FRI details are now resolved through the same base-aware helpers every other index file uses: indice_files_dir(index) for the path and object_store_for_index(index) for the store, both of which honor the entry's base_id and fall back to the current dataset root when there is none. The read range is also computed with checked arithmetic (offset.checked_add(size) with a usize bound), so a corrupt ExternalFile record surfaces as a corrupt-file error instead of overflowing.

Behavior for non-cloned datasets is unchanged: with no base_id, both helpers resolve to exactly the paths used before.

Note: this is independent of the FRI v2 stack and fixes an existing v0 issue on main.

The test

test_shallow_clone_loads_external_frag_reuse_details (in dataset/optimize.rs, next to the existing external-details test): builds an on-disk dataset whose compaction produces FRI details past the inline threshold (verified external), shallow-clones it to a second tempdir (verified the entry carries a base_id), then loads the details on the clone and opens the fragment reuse index through the dataset-level path, asserting the loaded content matches the source. On unpatched main the test fails at the load with a not-found error for details.binpb under the clone's root, confirming it covers the bug.

🤖 Generated with Claude Code

https://claude.ai/code/session_016RynpAxtwGB9Q9CL4JCvR4

… base path

When a fragment reuse index entry's details exceed the inline threshold
they are written to an external details.binpb under the index's
directory. load_frag_reuse_index_details resolved that file against the
current dataset root only, ignoring the entry's base_id. A shallow
clone's FRI entry is base-stamped and its details file stays in the
source dataset, so loading the details on the clone failed with a
not-found error under the clone's own indices dir.

External details now resolve through the base-aware helpers
(indice_files_dir and object_store_for_index), matching how every other
base-stamped index file is read, and the read range uses checked
arithmetic so a corrupt ExternalFile record errors instead of
overflowing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016RynpAxtwGB9Q9CL4JCvR4
@github-actions github-actions Bot added the bug Something isn't working label Sep 11, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Gate recommendation: approve.

The loader now resolves both the external FRI path and object store through the inherited index base_id, matching the established shallow-clone index contract while preserving current-dataset behavior when no base is present. The on-disk regression test covers an external-details entry through both direct loading and the dataset-level open path.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 11, 2026
@zhangyue19921010 zhangyue19921010 self-assigned this Sep 11, 2026

@zhangyue19921010 zhangyue19921010 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch!

@zhangyue19921010
zhangyue19921010 merged commit e89ceb3 into main Sep 11, 2026
49 of 50 checks passed
@zhangyue19921010
zhangyue19921010 deleted the lu/fix-fri-external-details-base branch September 11, 2026 09:22
wjones127 pushed a commit that referenced this pull request Sep 17, 2026
…#9176)

## Problem

`do_commit_new_dataset` restamps every index entry's `base_id` when
committing a shallow clone. For a single-hop clone this is correct: the
source's index files are reachable through the clone's new base path
entry. But for a chained clone (A -> B -> C), an index inherited by B
from A already carries `base_id: Some(a)` pointing at A through B's
`base_paths`. Unconditionally restamping it while committing C points
the entry at B's base instead, where the files do not exist. Queries on
C then fail to open the index. This affects indices of all types and is
latent on main today.

## Fix

Stamp `base_id` only when the entry's `base_id` is `None` (the files
live in the cloned dataset's own base). Entries already carrying
`Some(id)` keep their reference, which stays valid through
`Manifest::shallow_clone`'s base_paths carry-over.

## Tests

- `test_chained_shallow_clone_keeps_index_base`: A -> B -> C with a
scalar index whose files live in A; fails without the fix (C restamped
onto B), passes with it.
- Single-hop control asserting the `Some(0)` stamping and base_paths
contents are unchanged.

## Note

#9162 (tagged FRI shallow-clone support) carries this same fix inside
its larger change; when this PR merges, that branch rebases
mechanically, mirroring the #9141 pattern.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01DgAshYD7wVzPVdjXRuWPPs

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants