Repository navigation
fix(cache): scope RowIdSequence cache by version - #5508
zhangyue19921010 wants to merge 2 commits into
Conversation
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
|
Why your version 3 succeeded since it should be conflicted with version 2. Simply read your test. I think you are assuming we could reproduce a fragement with the same id in a version line, which in my view should never happen |
hi @majin1102 check out and restore to version 1 and trigger another compaction(update this description) |
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
That's a good catch. I wonder if we should keep the max_fragment_id for restoring, or use the approach you pointed, or both. I think fragment id traveling is not a good thing which introduces unnecessary risks. |
For now, in the case of restore, there will indeed be duplicate fragment IDs, row ID sequences, and possible |
I'm referring to this field in manifest. Right now, restore and overwrite can cause this max_fragment_id ID fallback and lead to fragment id traveling. That means fragement id is not globally identical through versions and time. I don't know if this logic was fully thought through initially.
Yeah I agree this is a light fix for the issue we are encountering Would appreciate thoughts from @jackye1995 @wjones127 @westonpace! |
That's a good point. I think we need to redesign that better so we can cache properly. We definitely want to re-use cached objects between dataset versions. But if that's the case we need to be able to rely on |
|
Starting to think about this: #5540 But need some time to think and discuss. Feel free to add ideas there on how we could solve this. |
`RowIdSequenceKey` was keyed on `fragment_id` alone, while the metadata cache is namespaced by dataset URI only. Fragment ids restart at 0 after `WriteMode::Overwrite`, so with a `Session` shared between the writer and the reader, every read after an overwrite was served the previous generation's `RowIdSequence` for the reused id. That silently corrupts stable row ids: row ids are reported for rows that no longer exist, row counts disagree with the manifest, compaction rechunks more ids than the fragments physically hold, and the same id can look live in two fragments at once. The key now also carries the fragment's `row_id_meta`, which identifies where the sequence bytes live. Identical metadata implies identical sequence bytes, so a hit is always a valid hit, while a rewritten fragment misses. Operations that leave row ids alone (deletes, added columns) keep hitting the cache. This matches the sibling `DeletionFileKey`, which already keys on the deletion file's identity rather than the fragment id alone. ## Why not make fragment ids unique instead Discussion #5540 asks for `fragment_id` and `row_id` to be globally unique across a table's history, and #5508 was closed in favour of that over a version-scoped cache key. Seeding the id counter from `max_fragment_id` on overwrite (the restore half shipped as #5554) is *not* a drop-in: `Fragment::id == 0` doubles as the "unassigned" sentinel in `Transaction::fragments_with_ids`, and today's overwrite masks that by starting the counter at 0, so a real fragment 0 is "reassigned" 0. With a high-water-mark counter, a pre-existing fragment passed through `Operation::Overwrite` — a supported public pattern, e.g. `LanceOperation.Overwrite(schema, fragments)` with fragments from `get_fragments()` — is renumbered, orphaning its deletion vector at `_deletions/{old_id}-…`. `test_append_new_columns` catches exactly this. So #5540 needs the sentinel made explicit first, and is left to a follow-up. Content-keying is correct independently of it, and also covers ids reused by paths that never consult a previous manifest: drop a dataset and create a new one at the same URI (#7645), or datasets written before any id-allocation change that already hold duplicate ids in their history. This does not fully close #7645. A dataset recreated at the same URI restarts at version 1, and the version-scoped keys that do not carry the manifest e-tag — `TransactionKey`, `RowIdIndexKey`, `RowAddrMaskKey`, and `IndexMetadataKey` in the index cache — still collide with the previous generation. `ManifestKey` already carries the e-tag; extending that to the rest is a follow-up. Fixes #8075 --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…at#8078) `RowIdSequenceKey` was keyed on `fragment_id` alone, while the metadata cache is namespaced by dataset URI only. Fragment ids restart at 0 after `WriteMode::Overwrite`, so with a `Session` shared between the writer and the reader, every read after an overwrite was served the previous generation's `RowIdSequence` for the reused id. That silently corrupts stable row ids: row ids are reported for rows that no longer exist, row counts disagree with the manifest, compaction rechunks more ids than the fragments physically hold, and the same id can look live in two fragments at once. The key now also carries the fragment's `row_id_meta`, which identifies where the sequence bytes live. Identical metadata implies identical sequence bytes, so a hit is always a valid hit, while a rewritten fragment misses. Operations that leave row ids alone (deletes, added columns) keep hitting the cache. This matches the sibling `DeletionFileKey`, which already keys on the deletion file's identity rather than the fragment id alone. Discussion lance-format#5540 asks for `fragment_id` and `row_id` to be globally unique across a table's history, and lance-format#5508 was closed in favour of that over a version-scoped cache key. Seeding the id counter from `max_fragment_id` on overwrite (the restore half shipped as lance-format#5554) is *not* a drop-in: `Fragment::id == 0` doubles as the "unassigned" sentinel in `Transaction::fragments_with_ids`, and today's overwrite masks that by starting the counter at 0, so a real fragment 0 is "reassigned" 0. With a high-water-mark counter, a pre-existing fragment passed through `Operation::Overwrite` — a supported public pattern, e.g. `LanceOperation.Overwrite(schema, fragments)` with fragments from `get_fragments()` — is renumbered, orphaning its deletion vector at `_deletions/{old_id}-…`. `test_append_new_columns` catches exactly this. So lance-format#5540 needs the sentinel made explicit first, and is left to a follow-up. Content-keying is correct independently of it, and also covers ids reused by paths that never consult a previous manifest: drop a dataset and create a new one at the same URI (lance-format#7645), or datasets written before any id-allocation change that already hold duplicate ids in their history. This does not fully close lance-format#7645. A dataset recreated at the same URI restarts at version 1, and the version-scoped keys that do not carry the manifest e-tag — `TransactionKey`, `RowIdIndexKey`, `RowAddrMaskKey`, and `IndexMetadataKey` in the index cache — still collide with the previous generation. `ManifestKey` already carries the e-tag; extending that to the rest is a follow-up. Fixes lance-format#8075 --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit 4d63a12)
Reproduction
(Simplify the new versions generated by restore and compaction reservation)
Root Cause
Fix
RowIdSequencecache entries by both dataset version and fragment:row_id_sequence/{version}/{fragment_id}.dataset.manifest.versionwhen creating theRowIdSequenceKeyduring sequence loads.