Skip to content

perf(ingest): combine source page and closure queries - #778

Draft
beinan wants to merge 1 commit into
lance-format:mainfrom
beinan:codex/annotation-source-page-closure-v2567
Draft

beinan wants to merge 1 commit into
lance-format:mainfrom
beinan:codex/annotation-source-page-closure-v2567

Conversation

@beinan

@beinan beinan commented Nov 5, 2026 •

Copy link
Copy Markdown
Collaborator

The annotation applier currently fetches an accepted bucket page and its source closure with two concurrent RPCs. Return both in one quorum observation so the recovery boundary and page come from the same applied state and the applier submits one metadata request per page.

The applier checks the closure before fetching payloads or advancing its cursor. Existing query variants remain supported; reply admission includes the bounded closure size. This does not change write acknowledgement, staging limits, publication, or log reclamation. Deploy WAL servers before annotation owners; an older server rejects the new query.

Validation covers quorum gating, query bounds and cursor checks, closure plus live suffix after native-node reopen, and existing concurrent publication, recovery and pressure tests. Local validation passed: 1,279 workspace Rust tests, 253 WAL service tests, 220 Python tests, formatting, Clippy, Ruff and Pyright. No end-to-end throughput improvement is claimed; ReadIndex can already batch concurrent queries.

Synthetic mixed ingestion/update/read evaluations passed full-value verification, bounded staging drain, reclamation and post-drain owner-replacement reads. The latest evaluation used fixed offered background rates. It did not establish an ingestion throughput benefit; leader placement and load-start timing also differed. This PR remains a draft.

Part of #378.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant