Skip to content

fix(cache): scope RowIdSequence cache by version - #5508

Closed
zhangyue19921010 wants to merge 2 commits into
lance-format:mainfrom
zhangyue19921010:fix-cache-with-stable-rowid
Closed

zhangyue19921010 wants to merge 2 commits into
lance-format:mainfrom
zhangyue19921010:fix-cache-with-stable-rowid

Conversation

@zhangyue19921010

@zhangyue19921010 zhangyue19921010 commented Dec 17, 2025 •

Copy link
Copy Markdown
Collaborator

Reproduction
(Simplify the new versions generated by restore and compaction reservation)

  • Create a dataset with stable Row IDs enabled; current version is 1.
  • Run compaction to produce version 2.
  • Checkout and restore back to version 1, adjust compaction params, run compaction again to produce version 3.
  • Query version 3, then checkout to version 2 and query again.
  • Observed: the query on version 2 fails; disabling caches makes the query succeed.

Root Cause

  • If you temporarily check out and restore a previous version and then perform data operations (not just compaction), fragment IDs may be duplicated across versions. Additionally, since the cache does not cache rowid sequences at the version granularity, this will cause time travel based on different versions to be affected by each other due to the cache.

Fix

  • Scope RowIdSequence cache entries by both dataset version and fragment:
    • Change the cache key to row_id_sequence/{version}/{fragment_id}.
    • Pass dataset.manifest.version when creating the RowIdSequenceKey during sequence loads.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

@github-actions github-actions Bot added the bug Something isn't working label Dec 17, 2025
@zhangyue19921010 zhangyue19921010 changed the title fix(cache): scope RowIdSequence by version fix(cache): scope RowIdSequence cache by version Dec 17, 2025
@codecov

codecov Bot commented Dec 17, 2025

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.82609% with 1 line in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
rust/lance/src/dataset/rowids.rs 50.00% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

@zhangyue19921010
zhangyue19921010 marked this pull request as draft December 17, 2025 13:28
@majin1102

Copy link
Copy Markdown
Contributor

Why your version 3 succeeded since it should be conflicted with version 2.

Simply read your test. I think you are assuming we could reproduce a fragement with the same id in a version line, which in my view should never happen

@zhangyue19921010

Copy link
Copy Markdown
Collaborator Author

Why your version 3 succeeded since it should be conflicted with version 2.

Simply read your test. I think you are assuming we could reproduce a fragement with the same id in a version line, which in my view should never happen

hi @majin1102 check out and restore to version 1 and trigger another compaction(update this description)

@zhangyue19921010
zhangyue19921010 marked this pull request as ready for review December 17, 2025 15:21
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

@majin1102

majin1102 commented Dec 17, 2025 •

Copy link
Copy Markdown
Contributor

Why your version 3 succeeded since it should be conflicted with version 2.
Simply read your test. I think you are assuming we could reproduce a fragement with the same id in a version line, which in my view should never happen

hi @majin1102 check out and restore to version 1 and trigger another compaction(update this description)

That's a good catch.

I wonder if we should keep the max_fragment_id for restoring, or use the approach you pointed, or both. I think fragment id traveling is not a good thing which introduces unnecessary risks.

@zhangyue19921010

Copy link
Copy Markdown
Collaborator Author

Why your version 3 succeeded since it should be conflicted with version 2.
Simply read your test. I think you are assuming we could reproduce a fragement with the same id in a version line, which in my view should never happen

hi @majin1102 check out and restore to version 1 and trigger another compaction(update this description)

That's a good catch.

I wonder if we should keep the max_fragment_id for restoring, or use the approach you pointed, or both. I think fragment id traveling is not a good thing which introduces unnecessary risks.

For now, in the case of restore, there will indeed be duplicate fragment IDs, row ID sequences, and possible max_xxx_xxx in the future. The current PR is a lightweight solution, but for further improvements, we may need to discuss the specific behavior of restore.

@majin1102

majin1102 commented Dec 18, 2025 •

Copy link
Copy Markdown
Contributor

and possible max_xxx_xxx in the future.

pub max_fragment_id: Option<u32>,

I'm referring to this field in manifest.

Right now, restore and overwrite can cause this max_fragment_id ID fallback and lead to fragment id traveling. That means fragement id is not globally identical through versions and time. I don't know if this logic was fully thought through initially.

The current PR is a lightweight solution

Yeah I agree this is a light fix for the issue we are encountering

Would appreciate thoughts from @jackye1995 @wjones127 @westonpace!

@wjones127

Copy link
Copy Markdown
Contributor

Right now, restore and overwrite can cause this max_fragment_id ID fallback and lead to fragment id traveling. That means fragement id is not globally identical through versions and time. I don't know if this logic was fully thought through initially.

That's a good point. I think we need to redesign that better so we can cache properly. We definitely want to re-use cached objects between dataset versions. But if that's the case we need to be able to rely on fragment_id being unique. This seems easy to accomplish with Overwrite (just don't reset the max_fragment_id. But it seems harder with restore. The other hard part is branches: they might also introduce duplicates in terms of fragment_id. I think we need a design discussion around this.

@wjones127 wjones127 self-assigned this Dec 18, 2025
@wjones127

Copy link
Copy Markdown
Contributor

Starting to think about this: #5540

But need some time to think and discuss. Feel free to add ideas there on how we could solve this.

wjones127 added a commit that referenced this pull request Aug 6, 2026
`RowIdSequenceKey` was keyed on `fragment_id` alone, while the metadata
cache is namespaced by dataset URI only. Fragment ids restart at 0 after
`WriteMode::Overwrite`, so with a `Session` shared between the writer
and the reader, every read after an overwrite was served the previous
generation's `RowIdSequence` for the reused id.

That silently corrupts stable row ids: row ids are reported for rows
that no longer exist, row counts disagree with the manifest, compaction
rechunks more ids than the fragments physically hold, and the same id
can look live in two fragments at once.

The key now also carries the fragment's `row_id_meta`, which identifies
where the sequence bytes live. Identical metadata implies identical
sequence bytes, so a hit is always a valid hit, while a rewritten
fragment misses. Operations that leave row ids alone (deletes, added
columns) keep hitting the cache. This matches the sibling
`DeletionFileKey`, which already keys on the deletion file's identity
rather than the fragment id alone.

## Why not make fragment ids unique instead

Discussion #5540 asks for `fragment_id` and `row_id` to be globally
unique across a table's history, and #5508 was closed in favour of that
over a version-scoped cache key. Seeding the id counter from
`max_fragment_id` on overwrite (the restore half shipped as #5554) is
*not* a drop-in: `Fragment::id == 0` doubles as the "unassigned"
sentinel in `Transaction::fragments_with_ids`, and today's overwrite
masks that by starting the counter at 0, so a real fragment 0 is
"reassigned" 0. With a high-water-mark counter, a pre-existing fragment
passed through `Operation::Overwrite` — a supported public pattern, e.g.
`LanceOperation.Overwrite(schema, fragments)` with fragments from
`get_fragments()` — is renumbered, orphaning its deletion vector at
`_deletions/{old_id}-…`. `test_append_new_columns` catches exactly this.

So #5540 needs the sentinel made explicit first, and is left to a
follow-up. Content-keying is correct independently of it, and also
covers ids reused by paths that never consult a previous manifest: drop
a dataset and create a new one at the same URI (#7645), or datasets
written before any id-allocation change that already hold duplicate ids
in their history.

This does not fully close #7645. A dataset recreated at the same URI
restarts at version 1, and the version-scoped keys that do not carry the
manifest e-tag — `TransactionKey`, `RowIdIndexKey`, `RowAddrMaskKey`,
and `IndexMetadataKey` in the index cache — still collide with the
previous generation. `ManifestKey` already carries the e-tag; extending
that to the rest is a follow-up.

Fixes #8075

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
mmatczuk pushed a commit to rerun-io/lance that referenced this pull request Nov 2, 2026
…at#8078)

`RowIdSequenceKey` was keyed on `fragment_id` alone, while the metadata
cache is namespaced by dataset URI only. Fragment ids restart at 0 after
`WriteMode::Overwrite`, so with a `Session` shared between the writer
and the reader, every read after an overwrite was served the previous
generation's `RowIdSequence` for the reused id.

That silently corrupts stable row ids: row ids are reported for rows
that no longer exist, row counts disagree with the manifest, compaction
rechunks more ids than the fragments physically hold, and the same id
can look live in two fragments at once.

The key now also carries the fragment's `row_id_meta`, which identifies
where the sequence bytes live. Identical metadata implies identical
sequence bytes, so a hit is always a valid hit, while a rewritten
fragment misses. Operations that leave row ids alone (deletes, added
columns) keep hitting the cache. This matches the sibling
`DeletionFileKey`, which already keys on the deletion file's identity
rather than the fragment id alone.

Discussion lance-format#5540 asks for `fragment_id` and `row_id` to be globally
unique across a table's history, and lance-format#5508 was closed in favour of that
over a version-scoped cache key. Seeding the id counter from
`max_fragment_id` on overwrite (the restore half shipped as lance-format#5554) is
*not* a drop-in: `Fragment::id == 0` doubles as the "unassigned"
sentinel in `Transaction::fragments_with_ids`, and today's overwrite
masks that by starting the counter at 0, so a real fragment 0 is
"reassigned" 0. With a high-water-mark counter, a pre-existing fragment
passed through `Operation::Overwrite` — a supported public pattern, e.g.
`LanceOperation.Overwrite(schema, fragments)` with fragments from
`get_fragments()` — is renumbered, orphaning its deletion vector at
`_deletions/{old_id}-…`. `test_append_new_columns` catches exactly this.

So lance-format#5540 needs the sentinel made explicit first, and is left to a
follow-up. Content-keying is correct independently of it, and also
covers ids reused by paths that never consult a previous manifest: drop
a dataset and create a new one at the same URI (lance-format#7645), or datasets
written before any id-allocation change that already hold duplicate ids
in their history.

This does not fully close lance-format#7645. A dataset recreated at the same URI
restarts at version 1, and the version-scoped keys that do not carry the
manifest e-tag — `TransactionKey`, `RowIdIndexKey`, `RowAddrMaskKey`,
and `IndexMetadataKey` in the index cache — still collide with the
previous generation. `ManifestKey` already carries the e-tag; extending
that to the rest is a follow-up.

Fixes lance-format#8075

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 4d63a12)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants