Repository navigation
fix: load external fragment reuse index details from the correct base path - #9141
Merged
Merged
Conversation
… base path When a fragment reuse index entry's details exceed the inline threshold they are written to an external details.binpb under the index's directory. load_frag_reuse_index_details resolved that file against the current dataset root only, ignoring the entry's base_id. A shallow clone's FRI entry is base-stamped and its details file stays in the source dataset, so loading the details on the clone failed with a not-found error under the clone's own indices dir. External details now resolve through the base-aware helpers (indice_files_dir and object_store_for_index), matching how every other base-stamped index file is read, and the read range uses checked arithmetic so a corrupt ExternalFile record errors instead of overflowing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016RynpAxtwGB9Q9CL4JCvR4
Contributor
There was a problem hiding this comment.
✅ Gate recommendation: approve.
The loader now resolves both the external FRI path and object store through the inherited index base_id, matching the established shallow-clone index contract while preserving current-dataset behavior when no base is present. The on-disk regression test covers an external-details entry through both direct loading and the dataset-level open path.
wjones127
pushed a commit
that referenced
this pull request
Sep 17, 2026
…#9176) ## Problem `do_commit_new_dataset` restamps every index entry's `base_id` when committing a shallow clone. For a single-hop clone this is correct: the source's index files are reachable through the clone's new base path entry. But for a chained clone (A -> B -> C), an index inherited by B from A already carries `base_id: Some(a)` pointing at A through B's `base_paths`. Unconditionally restamping it while committing C points the entry at B's base instead, where the files do not exist. Queries on C then fail to open the index. This affects indices of all types and is latent on main today. ## Fix Stamp `base_id` only when the entry's `base_id` is `None` (the files live in the cloned dataset's own base). Entries already carrying `Some(id)` keep their reference, which stays valid through `Manifest::shallow_clone`'s base_paths carry-over. ## Tests - `test_chained_shallow_clone_keeps_index_base`: A -> B -> C with a scalar index whose files live in A; fails without the fix (C restamped onto B), passes with it. - Single-hop control asserting the `Some(0)` stamping and base_paths contents are unchanged. ## Note #9162 (tagged FRI shallow-clone support) carries this same fix inside its larger change; when this PR merges, that branch rebases mechanically, mirroring the #9141 pattern. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01DgAshYD7wVzPVdjXRuWPPs Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
When a fragment reuse index (FRI) entry's details exceed the 204800-byte inline threshold, they are written to an external file (
_indices/<uuid>/details.binpb) and the index metadata carries anExternalFilereference.load_frag_reuse_index_detailsresolved that reference against the current dataset root only (dataset.indices_dir()+dataset.object_store), ignoring the entry'sbase_id.A shallow clone stamps its index metadata with a
base_idpointing at the source dataset, and the externaldetails.binpbstays in the source's indices directory: it is never copied into the clone. So on a clone, any path that loads the FRI details (opening the fragment reuse index, compaction, index remapping, conflict resolution) failed with a not-found error, because the loader looked for the file under the clone's own dataset root, which never contained it.The fix
External FRI details are now resolved through the same base-aware helpers every other index file uses:
indice_files_dir(index)for the path andobject_store_for_index(index)for the store, both of which honor the entry'sbase_idand fall back to the current dataset root when there is none. The read range is also computed with checked arithmetic (offset.checked_add(size)with ausizebound), so a corruptExternalFilerecord surfaces as a corrupt-file error instead of overflowing.Behavior for non-cloned datasets is unchanged: with no
base_id, both helpers resolve to exactly the paths used before.Note: this is independent of the FRI v2 stack and fixes an existing v0 issue on main.
The test
test_shallow_clone_loads_external_frag_reuse_details(indataset/optimize.rs, next to the existing external-details test): builds an on-disk dataset whose compaction produces FRI details past the inline threshold (verified external), shallow-clones it to a second tempdir (verified the entry carries abase_id), then loads the details on the clone and opens the fragment reuse index through the dataset-level path, asserting the loaded content matches the source. On unpatched main the test fails at the load with a not-found error fordetails.binpbunder the clone's root, confirming it covers the bug.🤖 Generated with Claude Code
https://claude.ai/code/session_016RynpAxtwGB9Q9CL4JCvR4