From a5fb09eab053c842f7444ccdb0aacbdfb767b36d Mon Sep 17 00:00:00 2001 From: Chris Alfano Date: Tue, 13 Oct 2026 20:55:25 -0400 Subject: [PATCH 1/3] docs(specs): hot reload applies only the changed records A full rebuild re-reads every sheet while holding the write lock, and the full import has OOM'd that path before. Apply the commit diff instead, with a full-rebuild fallback for config changes and bulk merges, and require the result to equal a full rebuild. Co-Authored-By: Claude Opus 5.5 --- specs/behaviors/storage.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/specs/behaviors/storage.md b/specs/behaviors/storage.md index 4361f89..03876fc 100644 --- a/specs/behaviors/storage.md +++ b/specs/behaviors/storage.md @@ -333,7 +333,9 @@ A push to the configured `CFP_DATA_BRANCH` from outside the API (typically a mer - **Reconcile + rebuild** — otherwise run the same reconciliation state machine the boot path uses (`fastify.reconcileDataRepo`): - If the branch didn't move (`in-sync`, `pushed-ahead`, `fetch-failed`), skip the rebuild and return 200 noChanges with the outcome. - Otherwise refresh the public store, rebuild the in-memory state, the revocation set and the FTS index from the new tree, then return 200 with the outcome, the old and new commit, and `rebuilt: true`. -- **Atomicity** — the rebuild constructs a fresh `InMemoryState` first; only after that succeeds does it mutate the live Maps in place. **Every** collection on the live state is replaced from the fresh one — the primary entity maps and every secondary index, including the legacy-id, buzz-by-slug, and slug-history indices — so no lookup path can serve pre-reload contents after a reload. The FTS engine exposes a `reload(state)` that drops and re-inserts every FTS5 table. If the rebuild throws partway, the route returns 500 and the operator should restart the pod. +- **Incremental by default** — the rebuild applies only what changed: for each in-memory sheet, the records that differ between the pre-reconcile commit and the new HEAD are unindexed (old version) and re-indexed (new version) in the live state and the FTS index. The result must equal a full rebuild from the new tree, every primary map and secondary index included, so no lookup path can serve pre-reload contents. +- **Full-rebuild fallback** — a full rebuild replaces the incremental one when any `.gitsheets/` config changed between the two commits, or when more than 5,000 records changed (a bulk import merge). The full rebuild constructs a fresh `InMemoryState` first; only after that succeeds does it mutate the live Maps in place, replacing **every** collection from the fresh one. The FTS engine's `reload(state)` drops and re-inserts every FTS5 table. +- **Failure** — if either rebuild throws partway, the route returns 500 and the operator should restart the pod. - **Concurrency** — the fetch runs outside any lock, so a slow network never blocks writes. The ref move, store refresh, rebuild and swap all run inside the repository's write lock (gitsheets `repo.withLock`), the same mutex `transact` holds. A write issued during a reload waits for it and then applies on top of the reloaded state; no write can land between the ref move and the swap, so none is lost. The GitHub Actions workflow that calls this endpoint lives in the `codeforphilly-data` repo (`.github/workflows/notify-deployments.yml`), not in this app repo. It fires on push to `CFP_DATA_BRANCH` and posts `{ branch, commitHash: }` with the secret as a bearer token. From f80e28b66f05fad42683f885a1664afefc2a95a6 Mon Sep 17 00:00:00 2001 From: Chris Alfano Date: Tue, 13 Oct 2026 21:03:55 -0400 Subject: [PATCH 2/3] feat(api): hot reload applies only the records that changed Hot reload re-read every sheet while holding the write lock. Now it diffs each in-memory sheet from the pre-reconcile commit (gitsheets Sheet.diffFrom) and unindexes/indexes just the changed records, with unindex* written as the exact inverse of index*. Falls back to the full rebuild when a sheet config changed or more than 5,000 records moved. The equivalence test checks the incrementally updated state deep-equals a full load across adds, edits, renames and deletes; breaking a single unindex function fails it. The webhook response reports reloadMode. Co-Authored-By: Claude Opus 5.5 --- apps/api/src/routes/internal.ts | 9 +- apps/api/src/store/memory/incremental.ts | 170 +++++++++++++++++ apps/api/src/store/memory/reload.ts | 52 +++++- apps/api/src/store/memory/state.ts | 97 ++++++++++ apps/api/tests/incremental-reload.test.ts | 215 ++++++++++++++++++++++ 5 files changed, 537 insertions(+), 6 deletions(-) create mode 100644 apps/api/src/store/memory/incremental.ts create mode 100644 apps/api/tests/incremental-reload.test.ts diff --git a/apps/api/src/routes/internal.ts b/apps/api/src/routes/internal.ts index b3f016e..83054cf 100644 --- a/apps/api/src/routes/internal.ts +++ b/apps/api/src/routes/internal.ts @@ -34,7 +34,7 @@ import { promisify } from 'node:util'; import type { FastifyInstance } from 'fastify'; import { errorResponse, ok } from '../lib/response.js'; -import { reloadInMemoryStateAndFts } from '../store/memory/reload.js'; +import { reloadInMemoryStateAndFts, type ReloadMode } from '../store/memory/reload.js'; import type { ReconcileOutcome, ReconcileResult } from '../store/reconcile.js'; const exec = promisify(execFile); @@ -197,9 +197,12 @@ export async function internalRoutes(fastify: FastifyInstance): Promise { // ---- Reconcile + rebuild under the repository write lock ---- let result: ReconcileResult; + let reloadMode: ReloadMode | undefined; try { result = await fastify.reconcileDataRepo({ - onMoved: () => reloadInMemoryStateAndFts(fastify), + onMoved: async (moved) => { + reloadMode = await reloadInMemoryStateAndFts(fastify, { fromCommit: moved.oldCommit }); + }, }); } catch (err) { // A throw from the rebuild leaves the in-memory state + FTS index @@ -242,6 +245,7 @@ export async function internalRoutes(fastify: FastifyInstance): Promise { oldCommit: result.oldCommit, newCommit: result.newCommit, conflictBranch: result.conflictBranch, + reloadMode, durationMs, }, 'hot-reload: in-memory state + FTS rebuilt', @@ -251,6 +255,7 @@ export async function internalRoutes(fastify: FastifyInstance): Promise { ok({ noChanges: false, rebuilt: true, + reloadMode, outcome: result.outcome, oldCommit: result.oldCommit, newCommit: result.newCommit, diff --git a/apps/api/src/store/memory/incremental.ts b/apps/api/src/store/memory/incremental.ts new file mode 100644 index 0000000..a756d61 --- /dev/null +++ b/apps/api/src/store/memory/incremental.ts @@ -0,0 +1,170 @@ +/** + * Incremental hot reload — specs/behaviors/storage.md → "Hot reload". + * + * Instead of re-reading every sheet, diff each in-memory sheet between the + * commit the live state reflects and the new HEAD (gitsheets + * `Sheet.diffFrom`), then unindex each changed record's old version and + * index its new one. `unindex*` is the exact inverse of `index*`, so the + * result equals a full load of the new tree. + */ +import { RECORD_PATH_KEY, RECORD_SHEET_KEY, type DiffChange } from 'gitsheets'; + +import type { FtsEngine } from '../fts.js'; +import type { PublicStore } from '../public.js'; +import { + indexBlogPost, + indexHelpWantedInterest, + indexHelpWantedRole, + indexMembership, + indexPerson, + indexPersonEvaluation, + indexProject, + indexProjectBuzz, + indexProjectUpdate, + indexSlugHistory, + indexTag, + indexTagAssignment, + unindexBlogPost, + unindexHelpWantedInterest, + unindexHelpWantedRole, + unindexMembership, + unindexPerson, + unindexPersonEvaluation, + unindexProject, + unindexProjectBuzz, + unindexProjectUpdate, + unindexSlugHistory, + unindexTag, + unindexTagAssignment, + type InMemoryState, +} from './state.js'; + +/** More changed records than this and a full rebuild is cheaper and safer. */ +export const INCREMENTAL_RECORD_LIMIT = 5_000; + +type Sheets = PublicStore; +type RecordOf = Sheets[K] extends { queryAll(): Promise<(infer T)[]> } + ? T + : never; + +interface SheetHandler { + readonly sheet: K; + readonly unindex: (state: InMemoryState, record: RecordOf) => void; + readonly index: (state: InMemoryState, record: RecordOf) => void; + readonly ftsRemove?: (fts: FtsEngine, record: RecordOf) => void; + readonly ftsAdd?: (fts: FtsEngine, record: RecordOf) => void; +} + +function handler(h: SheetHandler): SheetHandler { + return h; +} + +/** Every sheet the loader reads into InMemoryState, in loader order. */ +const HANDLERS = [ + handler({ + sheet: 'projects', + unindex: unindexProject, + index: indexProject, + ftsRemove: (fts, p) => fts.removeProject(p.slug), + ftsAdd: (fts, p) => { + if (!p.deletedAt) fts.upsertProject(p.slug, p.title, p.summary ?? '', p.overview ?? ''); + }, + }), + handler({ + sheet: 'people', + unindex: unindexPerson, + index: indexPerson, + ftsRemove: (fts, p) => fts.removePerson(p.slug), + ftsAdd: (fts, p) => { + if (!p.deletedAt) fts.upsertPerson(p.slug, p.fullName, p.bio ?? ''); + }, + }), + handler({ sheet: 'tags', unindex: unindexTag, index: indexTag }), + handler({ sheet: 'tag-assignments', unindex: unindexTagAssignment, index: indexTagAssignment }), + handler({ sheet: 'project-memberships', unindex: unindexMembership, index: indexMembership }), + handler({ sheet: 'project-updates', unindex: unindexProjectUpdate, index: indexProjectUpdate }), + handler({ sheet: 'project-buzz', unindex: unindexProjectBuzz, index: indexProjectBuzz }), + handler({ sheet: 'blog-posts', unindex: unindexBlogPost, index: indexBlogPost }), + handler({ + sheet: 'help-wanted-roles', + unindex: unindexHelpWantedRole, + index: indexHelpWantedRole, + ftsRemove: (fts, r) => fts.removeHelpWanted(r.id), + ftsAdd: (fts, r) => fts.upsertHelpWanted(r.id, r.title, r.description), + }), + handler({ + sheet: 'help-wanted-interest', + unindex: unindexHelpWantedInterest, + index: indexHelpWantedInterest, + }), + handler({ + sheet: 'slug-history', + unindex: unindexSlugHistory, + index: (state, r) => indexSlugHistory(state, r), + }), + handler({ + sheet: 'person-evaluations', + unindex: unindexPersonEvaluation, + index: indexPersonEvaluation, + }), +] as const; + +interface CollectedChanges { + readonly handler: SheetHandler; + readonly changes: readonly DiffChange[]; +} + +/** + * Collect every changed record between `fromCommit` and each sheet's current + * read snapshot. Returns null when more than {@link INCREMENTAL_RECORD_LIMIT} + * records changed, meaning the caller should do a full rebuild. + */ +export async function collectChanges( + store: PublicStore, + fromCommit: string, +): Promise { + const collected: CollectedChanges[] = []; + let total = 0; + for (const h of HANDLERS) { + const changes: DiffChange[] = []; + for await (const change of store[h.sheet].diffFrom(fromCommit, { records: true })) { + changes.push(change); + if (++total > INCREMENTAL_RECORD_LIMIT) return null; + } + if (changes.length > 0) { + collected.push({ handler: h as unknown as SheetHandler, changes }); + } + } + return collected; +} + +/** + * Apply collected changes to the live state and FTS index. Synchronous, so + * no request handler observes a half-applied diff. + */ +export function applyChanges( + state: InMemoryState, + fts: FtsEngine, + collected: readonly CollectedChanges[], +): number { + let applied = 0; + for (const { handler: h, changes } of collected) { + for (const change of changes) { + const src = change.src as RecordOf | undefined; + const dst = change.dst as RecordOf | undefined; + if (src) { + h.unindex(state, src); + h.ftsRemove?.(fts, src); + } + if (dst) { + // Annotate like `queryAll()` does: gitsheets locates a record's + // current path through these on a later upsert/delete. + Object.assign(dst, { [RECORD_SHEET_KEY]: h.sheet, [RECORD_PATH_KEY]: change.path }); + h.index(state, dst); + h.ftsAdd?.(fts, dst); + } + applied++; + } + } + return applied; +} diff --git a/apps/api/src/store/memory/reload.ts b/apps/api/src/store/memory/reload.ts index 343967e..0ffcae7 100644 --- a/apps/api/src/store/memory/reload.ts +++ b/apps/api/src/store/memory/reload.ts @@ -20,11 +20,36 @@ * Used by `apps/api/src/routes/internal.ts` (the `POST * /api/_internal/reload-data` webhook). */ +import { execFile } from 'node:child_process'; +import { promisify } from 'node:util'; + import type { FastifyInstance } from 'fastify'; import { invalidateFacets } from './facets.js'; +import { applyChanges, collectChanges } from './incremental.js'; import { loadInMemoryState } from './loader.js'; import type { InMemoryState } from './state.js'; +const exec = promisify(execFile); + +export interface ReloadOptions { + /** + * The commit the live state currently reflects. When given, the reload + * applies only the records that changed since, unless a sheet config + * changed or the diff is too large. + */ + readonly fromCommit?: string; +} + +export type ReloadMode = 'incremental' | 'full'; + +/** Whether `.gitsheets/` (the sheet configs) differs between two commits. */ +async function sheetConfigsChanged(repoPath: string, from: string, to: string): Promise { + const tree = async (commit: string): Promise => + (await exec('git', ['rev-parse', `${commit}:.gitsheets`], { cwd: repoPath })).stdout.trim(); + const [a, b] = await Promise.all([tree(from), tree(to)]); + return a !== b; +} + /** * Replace the contents of `target`'s Maps with `source`'s Maps. The * `target` object identity is preserved so services that captured a @@ -42,16 +67,34 @@ function replaceMapContents(target: Map, source: Map): void { } /** - * Build a fresh `InMemoryState` from the public store and swap it into - * the running Fastify instance, then reload the revocation set and the FTS - * index from the same snapshot and invalidate the facet cache. + * Bring the live in-memory state, revocation set and FTS index up to the + * public store's current snapshot, and invalidate the facet cache. + * + * With `fromCommit`, applies only the records that changed since (see + * `incremental.ts`); otherwise, or when configs changed or the diff is + * large, rebuilds everything. Returns which path ran. * * The caller must already have refreshed the public store to the new HEAD * (`fastify.reconcileDataRepo` does, under the repository write lock). */ export async function reloadInMemoryStateAndFts( fastify: FastifyInstance, -): Promise { + opts: ReloadOptions = {}, +): Promise { + const { fromCommit } = opts; + if (fromCommit) { + const repoPath = fastify.config.CFP_DATA_REPO_PATH; + if (!(await sheetConfigsChanged(repoPath, fromCommit, 'HEAD'))) { + const changes = await collectChanges(fastify.store.public, fromCommit); + if (changes) { + applyChanges(fastify.inMemoryState, fastify.fts, changes); + await fastify.revocations.loadFromSheet(fastify.store.public); + invalidateFacets(); + return 'incremental'; + } + } + } + // Step 1: build the fresh in-memory state. If this throws (validator // failure on a new record, etc.), the live state is untouched and the // caller surfaces the error. @@ -74,6 +117,7 @@ export async function reloadInMemoryStateAndFts( // Step 5: drop the cached facet roll-ups so the next request recomputes. invalidateFacets(); + return 'full'; } /** diff --git a/apps/api/src/store/memory/state.ts b/apps/api/src/store/memory/state.ts index 69ce047..dce6464 100644 --- a/apps/api/src/store/memory/state.ts +++ b/apps/api/src/store/memory/state.ts @@ -324,3 +324,100 @@ export function indexPersonEvaluation(state: InMemoryState, record: PersonEvalua } set.add(key); } + +// --------------------------------------------------------------------------- +// Unindexing — the exact inverse of each index* above, so applying a commit +// diff (unindex the old version, index the new one) leaves the state equal to +// a full load of the new tree. Empty secondary Sets are deleted, since a full +// load never creates them. Used by the incremental hot reload. +// --------------------------------------------------------------------------- + +function removeFromSet(map: Map>, key: K, id: string): void { + const set = map.get(key); + if (!set) return; + set.delete(id); + if (set.size === 0) map.delete(key); +} + +/** Delete `key` from `map` only if it still points at `id`. */ +function deleteIfOwned(map: Map, key: K, id: string): void { + if (map.get(key) === id) map.delete(key); +} + +export function unindexSlugHistory(state: InMemoryState, record: SlugHistory): void { + const key = slugHistoryKey(record.entityType, record.oldSlug); + if (state.slugHistory.get(key)?.newSlug === record.newSlug) state.slugHistory.delete(key); +} + +export function unindexProject(state: InMemoryState, project: Project): void { + state.projects.delete(project.id); + state.projectSlugById.delete(project.id); + deleteIfOwned(state.projectIdBySlug, project.slug, project.id); + if (typeof project.legacyId === 'number') { + deleteIfOwned(state.projectIdByLegacyId, project.legacyId, project.id); + } +} + +export function unindexPerson(state: InMemoryState, person: Person): void { + state.people.delete(person.id); + state.personSlugById.delete(person.id); + deleteIfOwned(state.personIdBySlug, person.slug, person.id); +} + +export function unindexTag(state: InMemoryState, tag: Tag): void { + state.tags.delete(tag.id); + deleteIfOwned(state.tagIdByHandle, `${tag.namespace}.${tag.slug}`, tag.id); +} + +export function unindexTagAssignment(state: InMemoryState, ta: TagAssignment): void { + state.tagAssignments.delete(ta.id); + removeFromSet(state.tagAssignmentsByTaggable, ta.taggableId, ta.id); + removeFromSet(state.tagAssignmentsByTag, ta.tagId, ta.id); +} + +export function unindexMembership(state: InMemoryState, m: ProjectMembership): void { + state.projectMemberships.delete(m.id); + removeFromSet(state.membershipsByProject, m.projectId, m.id); + removeFromSet(state.membershipsByPerson, m.personId, m.id); +} + +export function unindexProjectUpdate(state: InMemoryState, update: ProjectUpdate): void { + state.projectUpdates.delete(update.id); + removeFromSet(state.updatesByProject, update.projectId, update.id); + deleteIfOwned(state.updateByProjectAndNumber, `${update.projectId}:${update.number}`, update.id); +} + +export function unindexProjectBuzz(state: InMemoryState, buzz: ProjectBuzz): void { + state.projectBuzz.delete(buzz.id); + removeFromSet(state.buzzByProject, buzz.projectId, buzz.id); + deleteIfOwned(state.buzzByProjectAndSlug, `${buzz.projectId}:${buzz.slug}`, buzz.id); + deleteIfOwned(state.buzzIdBySlug, buzz.slug, buzz.id); +} + +export function unindexBlogPost(state: InMemoryState, post: BlogPost): void { + state.blogPosts.delete(post.id); + deleteIfOwned(state.blogPostIdBySlug, post.slug, post.id); + if (typeof post.legacyId === 'number') { + deleteIfOwned(state.blogPostIdByLegacyId, post.legacyId, post.id); + } +} + +export function unindexHelpWantedRole(state: InMemoryState, role: HelpWantedRole): void { + state.helpWantedRoles.delete(role.id); + removeFromSet(state.helpWantedByProject, role.projectId, role.id); +} + +export function unindexHelpWantedInterest( + state: InMemoryState, + expr: HelpWantedInterestExpression, +): void { + state.helpWantedInterest.delete(expr.id); + removeFromSet(state.interestByRole, expr.roleId, expr.id); + deleteIfOwned(state.interestByRoleAndPerson, `${expr.roleId}:${expr.personId}`, expr.id); +} + +export function unindexPersonEvaluation(state: InMemoryState, record: PersonEvaluation): void { + const key = `${record.personSlug}/${record.evaluator}`; + state.personEvaluations.delete(key); + removeFromSet(state.evaluationsByPerson, record.personSlug, key); +} diff --git a/apps/api/tests/incremental-reload.test.ts b/apps/api/tests/incremental-reload.test.ts new file mode 100644 index 0000000..ee47238 --- /dev/null +++ b/apps/api/tests/incremental-reload.test.ts @@ -0,0 +1,215 @@ +/** + * Incremental hot reload — specs/behaviors/storage.md → "Hot reload". + * + * The guard: after upstream commits that add, modify, rename and delete + * records across the in-memory sheets, the incrementally updated live state + * must deep-equal a full load of the new tree. Any secondary index only the + * loader maintains would show up here as a mismatch. + */ +import { execFile } from 'node:child_process'; +import { mkdtemp, rm } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { promisify } from 'node:util'; + +import { afterEach, beforeEach, describe, expect, it } from 'vitest'; +import type { FastifyInstance } from 'fastify'; +import { openRepo } from 'gitsheets'; + +import { buildApp } from '../src/app.js'; +import { loadInMemoryState } from '../src/store/memory/loader.js'; +import { createPrivateStorageDir } from './helpers/test-full-repo.js'; +import { seedFixtures, seedRawBlob, type SeededFixtures } from './helpers/seed-fixtures.js'; +import { writeSheetConfigs } from './helpers/sheet-configs.js'; + +const exec = promisify(execFile); +const SECRET = 'test-reload-secret-at-least-32-chars-long!!'; +const NOW = '2026-05-20T00:00:00Z'; + +async function git(cwd: string, ...args: string[]): Promise { + return (await exec('git', args, { cwd })).stdout.trim(); +} + +let root: string; +let bare: string; +let local: string; +let privateStore: { path: string; cleanup: () => Promise }; +let app: FastifyInstance | undefined; + +beforeEach(async () => { + root = await mkdtemp(join(tmpdir(), 'cfp-incremental-')); + bare = join(root, 'remote.git'); + local = join(root, 'local.git'); + const seed = join(root, 'seed'); + + await exec('git', ['init', '-b', 'main', seed]); + for (const [k, v] of [ + ['user.email', 'seed@test.local'], + ['user.name', 'seed'], + ['commit.gpgsign', 'false'], + ['core.hooksPath', '/dev/null'], + ]) { + await git(seed, 'config', k as string, v as string); + } + await writeSheetConfigs(seed); + await git(seed, 'add', '.gitsheets'); + await git(seed, 'commit', '-m', 'initial: sheet configs'); + + await exec('git', ['init', '--bare', '-b', 'main', bare]); + await exec('git', ['push', bare, 'main'], { cwd: seed }); + await exec('git', ['clone', '--bare', bare, local]); + await git(local, 'symbolic-ref', 'HEAD', 'refs/heads/main'); + await rm(seed, { recursive: true, force: true }); + + privateStore = await createPrivateStorageDir(); + app = await buildApp({ + serverOptions: { logger: false }, + overrideEnv: { + CFP_DATA_REPO_PATH: local, + CFP_DATA_REMOTE: bare, + CFP_DATA_BRANCH: 'main', + CFP_DATA_RELOAD_SECRET: SECRET, + STORAGE_BACKEND: 'filesystem', + CFP_PRIVATE_STORAGE_PATH: privateStore.path, + CFP_JWT_SIGNING_KEY: 'test-jwt-signing-key-at-least-32-chars!!', + NODE_ENV: 'test', + }, + }); +}); + +afterEach(async () => { + if (app) { + await app.close(); + app = undefined; + } + await privateStore.cleanup(); + await rm(root, { recursive: true, force: true }); +}); + +async function reload(): Promise<{ statusCode: number; reloadMode?: string }> { + const res = await app!.inject({ + method: 'POST', + url: '/api/_internal/reload-data', + headers: { authorization: `Bearer ${SECRET}` }, + payload: { commitHash: await git(bare, 'rev-parse', 'main') }, + }); + const body = res.json<{ data?: { reloadMode?: string } }>(); + return { statusCode: res.statusCode, reloadMode: body.data?.reloadMode }; +} + +async function expectEqualsFullLoad(): Promise { + const full = await loadInMemoryState(app!.store.public); + expect(app!.inMemoryState).toEqual(full); +} + +/** Upstream edits across every in-memory sheet: rename, modify, delete, add. */ +async function editUpstream(f: SeededFixtures): Promise { + const repo = await openRepo({ gitDir: bare }); + await repo.transact( + { message: 'upstream: edits', author: { name: 'test', email: 'test@cfp.test' } }, + async (tx) => { + // Rename + retitle the project. + const projects = tx.sheet('projects'); + const [project] = (await projects.queryAll()) as Array>; + await projects.delete(project!); + await projects.upsert({ ...project, slug: 'squadquest-two', title: 'SquadQuest Two' }); + + // Modify the person; add a second one. + const people = tx.sheet('people'); + const [person] = (await people.queryAll()) as Array>; + await people.upsert({ ...person, bio: 'Changed upstream.' }); + await people.upsert({ + id: '01951a3c-0000-7000-8000-0000000000f1', + slug: 'new-person', + fullName: 'New Person', + accountLevel: 'user', + createdAt: NOW, + updatedAt: NOW, + }); + + // Delete every membership and buzz item. + for (const sheet of ['project-memberships', 'project-buzz'] as const) { + const s = tx.sheet(sheet); + for (const r of (await s.queryAll()) as Array>) await s.delete(r); + } + + // Retitle the help-wanted role; retitle the tag. + const roles = tx.sheet('help-wanted-roles'); + for (const r of (await roles.queryAll()) as Array>) { + await roles.upsert({ ...r, title: 'Retitled role', projectSlug: 'squadquest-two' }); + } + const tags = tx.sheet('tags'); + const [tag] = (await tags.queryAll()) as Array>; + await tags.upsert({ ...tag, title: 'Flutter (retitled)' }); + + // Drop one tag assignment. + const assignments = tx.sheet('tag-assignments'); + const [ta] = (await assignments.queryAll()) as Array>; + await assignments.delete(ta!); + + // Slug history for the rename, and a spam vote. + await tx.sheet('slug-history').upsert({ + id: '01951a3c-0000-7000-8000-0000000000f2', + entityType: 'project', + oldSlug: f.projectSlug, + newSlug: 'squadquest-two', + entityId: f.projectId, + changedAt: NOW, + expiresAt: '2099-01-01T00:00:00Z', + }); + await tx.sheet('person-evaluations').upsert({ + personSlug: 'new-person', + evaluator: 'human-jane-doe', + verdict: 'legit', + flags: [], + evaluatedAt: NOW, + }); + }, + ); +} + +describe('incremental hot reload', () => { + it('matches a full load after upstream adds, edits, renames and deletes', async () => { + const fixtures = await seedFixtures(bare); + const added = await reload(); + expect(added).toEqual({ statusCode: 200, reloadMode: 'incremental' }); + await expectEqualsFullLoad(); + expect(app!.inMemoryState.projects.size).toBe(1); + + await editUpstream(fixtures); + const edited = await reload(); + expect(edited).toEqual({ statusCode: 200, reloadMode: 'incremental' }); + await expectEqualsFullLoad(); + + // Search follows the rename; the old slug is gone from the index. + expect(app!.fts.searchProjects('SquadQuest')).toEqual(['squadquest-two']); + expect(app!.fts.searchPeople('Changed')).toEqual([fixtures.personSlug]); + expect(app!.fts.searchHelpWanted('Retitled')).toEqual([fixtures.helpWantedId]); + + // A record that arrived incrementally can still be written through + // gitsheets: delete locates it by its path annotation. + const project = app!.inMemoryState.projects.get(fixtures.projectId)!; + await app!.store.public.transact( + { message: 'test: delete reloaded project', author: { name: 'test', email: 'test@cfp.test' } }, + async (tx) => { + await tx.projects.delete(project); + }, + ); + expect(await git(local, 'ls-tree', '--name-only', 'HEAD', 'projects/')).toBe(''); + }); + + it('falls back to a full rebuild when a sheet config changes', async () => { + await seedFixtures(bare); + const config = await git(bare, 'show', 'main:.gitsheets/people.toml'); + await seedRawBlob( + bare, + '.gitsheets/people.toml', + Buffer.from(`${config}\n# touched\n`), + 'upstream: touch a sheet config', + ); + + const res = await reload(); + expect(res).toEqual({ statusCode: 200, reloadMode: 'full' }); + await expectEqualsFullLoad(); + }); +}); From 044fe1f85e154fc2743d4499e54c3a0aadc3f949 Mon Sep 17 00:00:00 2001 From: Chris Alfano Date: Tue, 13 Oct 2026 21:04:10 -0400 Subject: [PATCH 3/3] chore(plans): mark incremental-reload done (PR #202) Co-Authored-By: Claude Opus 5.5 --- plans/incremental-reload.md | 28 +++++++++++++++++++++++----- 1 file changed, 23 insertions(+), 5 deletions(-) diff --git a/plans/incremental-reload.md b/plans/incremental-reload.md index 5d33160..c0a7ae3 100644 --- a/plans/incremental-reload.md +++ b/plans/incremental-reload.md @@ -1,8 +1,9 @@ --- -status: planned +status: done depends: [gitsheets-freshness] specs: - specs/behaviors/storage.md +pr: 202 --- # Plan: incremental hot reload via `sheet.diffFrom` @@ -45,14 +46,14 @@ Out: the boot path stays a full load. ## Validation -- [ ] Spec updated first -- [ ] Equivalence test: for a series of upstream commits (add, modify, +- [x] Spec updated first +- [x] Equivalence test: for a series of upstream commits (add, modify, delete, rename in each sheet), state after the incremental apply deep- equals state after a full rebuild -- [ ] Config change under `.gitsheets/` triggers the full-rebuild fallback +- [x] Config change under `.gitsheets/` triggers the full-rebuild fallback - [ ] Sandbox: reload timing and peak heap for a one-record upstream change vs. a full rebuild -- [ ] `type-check` + `lint` + full api suite green +- [x] `type-check` + `lint` + full api suite green ## Risks / unknowns @@ -62,4 +63,21 @@ Out: the boot path stays a full load. ## Notes +- **Unchecked: sandbox timing and peak heap.** Needs a release on the + sandbox. The webhook response and log now carry `reloadMode`, so it can + be read off the first real reload. +- Didn't reuse `StateApply`'s remove ops: several don't mirror `index*` + exactly (e.g. leaving empty Sets a full load never creates). Wrote + `unindex*` in `state.ts` as exact inverses instead, which is what makes a + plain deep-equal against a full load possible. +- `diffFrom` records lack the `gitsheets-sheet` / `gitsheets-path` symbols + that `queryAll()` attaches, and gitsheets' delete/upsert rely on them, so + the apply step adds them. The test deletes an incrementally loaded record + through gitsheets to pin this. +- The fallback threshold (5,000 changed records) is a judgment call. + Full-import merges exceed it by an order of magnitude; routine pushes are + a handful of records. + ## Follow-ups + +- Tracked as: sandbox reload timing/heap after the next release (see Notes).