Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 27 additions & 2 deletions plans/gitsheets-2x-bump.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
---
status: in-progress
status: done
depends: []
specs:
- specs/behaviors/storage.md
issues:
- 150
pr:
pr: 151
---

# Plan: upgrade gitsheets 1.4.1 → 2.x (Rust core)
Expand Down Expand Up @@ -126,3 +126,28 @@ API. Sites:
- [x] type-check + lint clean
- [x] full api suite green (no re-baseline test updates were needed — see below)
- [x] byte-parity on `published` = lossless (only documented re-baselines)

## Notes

- Merged 2026-07-04 as PR #151; the closeout commit was missed at the time
and is recorded retroactively here (2026-10-13). Shipped at `^2.2.0`.
- One test needed an assertion fix, not a re-baseline: a
`not.toContain('null')` false-matched the fixture's own "Nullish Person"
name. Suite was 423/423 under 2.x.
- The null-marshal break was later fixed upstream (gitsheets #241, 2.3.1 —
null/undefined keys are dropped natively), making `stripNullish` removable.
- `gitsheets#184` closed 2026-07-05 via gitsheets PR #239 (first on npm in
2.3.1), the day after this merged — unblocking the workaround retirement.
- Data-repo re-normalization happened organically: as of 2026-10-13 only
two `published` records still carry integer underscores
(`people/chris.toml`, `people/itskatnotcat.toml`, both `githubUserId`).

## Follow-ups

- Deferred to [`gitsheets-freshness`](gitsheets-freshness.md) — retire
`swapPublic`, the attachments `git cat-file` path, and `data-repo-lock`
using gitsheets ≥ 2.3.1 `refresh()` / `readBlobStream()` / `withLock()`
(second half of #150).
- Tracked as: two straggler records with integer underscores in
`codeforphilly-data` `published` — a one-off `git sheet normalize people`
on the data repo, or let them re-normalize on next write.
148 changes: 148 additions & 0 deletions plans/gitsheets-freshness.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,148 @@
---
status: planned
depends: [gitsheets-2x-bump, slug-history-config]
specs:
- specs/behaviors/storage.md
issues:
- 150
---

# Plan: gitsheets 2.7 — retire the stale-Sheet workarounds

## Scope

The second half of #150, now unblocked. Upstream `gitsheets#184` (per-sheet
refresh) shipped via gitsheets PR #239 in **2.3.1** (2.3.0 never reached
npm). Bump `gitsheets` from `^2.2.0` to `^2.7.0`, then use the new APIs to
delete every workaround for the stale-`Sheet` cache, plus the shims and dead
code that 2.3+ made redundant.

In:

- Bump to `^2.7.0` (core-napi 0.1.1 → 0.5.0). Free win: 2.7 reads records
in-core (gitsheets #295), so the markdown `blog-posts` sheet no longer
spawns one process per record. Boot and hot reload get faster.
- **`swapPublic`** → `store.refresh()`.
- **`data-repo-lock`** → `repo.withLock()`, which shares the mutex
`transact` uses. This **fixes a live race**: no write path takes
`dataRepoLock` today, so the hot-reload webhook can race a `transact`,
despite `storage.md` saying otherwise. The rebuild also runs *after* the
lock is released, so a write that lands mid-`loadInMemoryState` can be
clobbered by `swapInPlace`.
- **Raw `git cat-file` attachment streaming** → `sheet.getAttachment()`.
It resolves the blob hash in-core with one spawn to stream, whereas
`readBlobStream` uses 2–3 spawns. Use the blob hash as a strong `ETag`
with immutable caching.
- **Reconcile's own `git push` calls** (`store/reconcile.ts` ~389, ~497) →
`pushDaemon.notifyCommit(newCommit)`. Today a post-rebase push outside the
daemon leaves it stuck in `non-fast-forward`.
- **Hot reload misses revocations**: `InMemoryRevocationStore` loads only
`onReady`. Reload it as part of the rebuild.
- **Webhook `branch` override**: reject a `branch` other than
`CFP_DATA_BRANCH`. `refresh()` reads `HEAD`, so reconciling a ref that
`HEAD` doesn't point to would silently desync.
- Drop `stripNullish` (2.3 drops null/undefined keys natively and
recursively, gitsheets #241) and the `asValidator` / `as PublicStore`
casts (2.3 Standard Schema typing needs no cast, gitsheets #242).
- **Delete `store/sheet-indices.ts`**: 25 `defineIndex` definitions that
nothing outside tests queries. Under auto-refresh, every transact
invalidates index builds, so any future `findByIndex` would rescan after
every write. Uniqueness checks already run against the in-memory maps.
- Write attachments from a Buffer (`setAttachments`), dropping the separate
`writeBlob` step in `routes/people.ts` and the importer.

Out (separate plans):

- Incremental hot reload → [`incremental-reload`](incremental-reload.md).
- Scripts that bypass gitsheets → [`gitsheets-scripts`](gitsheets-scripts.md).
- Reconcile → upstream sync → [`gitsheets-sync`](gitsheets-sync.md).

## Implements

- [behaviors/storage.md](../specs/behaviors/storage.md): the hot-reload
concurrency guarantee ("a webhook can't race a `transact`-driven write")
becomes true, and the "caching limitation" section is retired.

## Approach

1. **Spec first** (own commit). In `storage.md`:
- Rewrite "Direct gitsheets reads after a transact (caching limitation)":
standing sheets auto-refresh after a committing transact, and
out-of-process ref moves (reconcile) need `store.refresh()`.
- Change the hot-reload **Concurrency** bullet: reconcile, rebuild and
swap all run under the repo's transact mutex, and writes queue for the
duration.
- The webhook rejects a non-default `branch`.
- Add an `ETag` to the attachment-serving rule.
2. **Bump**: `npm install gitsheets@^2.7.0 -w apps/api`. Commit the
generated changes first.
3. **Lock**: delete `lib/data-repo-lock.ts` and the `dataRepoLock`
decoration. Boot reconcile and the webhook run inside
`fastify.publicRepo.withLock(...)`. The webhook wraps reconcile, refresh,
rebuild and swap in **one** `withLock`.
- `withLock` is not reentrant. Nothing inside it may call `transact`;
verify reconcile only shells out to git.
- Do the `git fetch` *outside* the lock so a slow network doesn't block
writes, and rely on reconcile's existing CAS `update-ref`.
- Writes queue behind the rebuild. Hot reloads are rare (external pushes
to `published`), and 2.7 makes the load faster.
[`incremental-reload`](incremental-reload.md) shrinks the window
further.
4. **Refresh**: delete `Store.swapPublic`. `reloadInMemoryStateAndFts` runs
`await store.public.refresh()`, then `loadInMemoryState`, then reloads
revocations. Drop the re-open.
5. **Push**: replace reconcile's `git push` with `pushDaemon.notifyCommit`.
At boot, the daemon's startup backlog already pushes commits that are
ahead of the remote.
6. **Attachments**: use `sheet.getAttachment(path, name)` in
`routes/attachments.ts`.
- Map a missing attachment to 404, keep `validateKey`, set
`ETag: "<blobHash>"`, and honor `If-None-Match` with a 304.
- Attach a stream `'error'` handler.
- This also fixes an empty blob being reported as 404.
7. **Shims and dead code**: delete `stripNullish`, `asValidator` and
`sheet-indices.ts`. Pass the Zod schemas directly (`satisfies
ValidatorMap`). Pass Buffers to `setAttachments`.
8. **Tests**: replace the `git show HEAD:…` / `swapPublic` fallbacks
(`account-claim.test.ts`, `store.test.ts`, `moderation.test.ts`, …) with
plain sheet reads or `sheet.get(path)` (gitsheets #296).

## Validation

- [ ] `storage.md` updated in its own commit before the code changes
- [ ] `gitsheets` resolves to ≥ 2.7.0; `@gitsheets/core-napi` loads in the
Alpine (`linux-x64-musl`) image
- [ ] grep clean: `data-repo-lock`, `swapPublic`, `stripNullish`,
`asValidator`, `defineIndex`, `cat-file` in `routes/`, `git push` in
`store/reconcile.ts`
- [ ] New test: a write issued during a hot reload queues behind it and
survives (no clobber)
- [ ] New test: hot reload picks up a revocation added upstream
- [ ] New test: webhook rejects a non-default `branch` with 400
- [ ] Attachments: 200 with `ETag` for an existing key, 304 on
`If-None-Match`, 404 for a missing key, correct empty-blob handling
- [ ] A write of a record with cleared optional fields still produces no
`null` keys on disk; no schema emits a `null` array element (2.3 throws
on those)
- [ ] After a hot-reload rebase, the push daemon status is healthy (not
`non-fast-forward`)
- [ ] `type-check` + `lint` + full api suite green
- [ ] Sandbox: boot to ready and one hot-reload round trip succeed; record
boot and reload timing vs. 2.2

## Risks / unknowns

- **Native addon jump** (core-napi 0.1 → 0.5). Mitigated by the sandbox
check before live.
- **Behavior change**: standing sheets now see new commits. The revocation
sweeper (`auth/middleware.ts`) reads a standing sheet. It gets fresher,
which is intended, but watch its tests.
- **`refresh()` resolves `HEAD`.** Assert that the bare clone's `HEAD` is
`refs/heads/$CFP_DATA_BRANCH` at boot.
- **Writes block for the length of a hot-reload rebuild.** That's
acceptable at current reload frequency; `incremental-reload` is the fix if
it isn't.

## Notes

## Follow-ups
79 changes: 79 additions & 0 deletions plans/gitsheets-scripts.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
---
status: planned
depends: [gitsheets-freshness]
specs:
- specs/behaviors/storage.md
- specs/behaviors/spam-exclusion.md
awaits:
- "JarvusInnovations/gitsheets — ref-scoped read-only sheets (issue TBD) — cleaner than the no-op-transact read trick"
- "JarvusInnovations/gitsheets — public tx.writeFile / orphan commits (issue TBD) — needed for scrub-data's config writes"
---

# Plan: route operator scripts through gitsheets

## Scope

Three operator scripts read and write the data repo with raw git plumbing
and hand-rolled TOML instead of gitsheets. They have bugs that gitsheets
would have prevented.

- **`scripts/prune-spam.ts`** (~170–300, ~420–443) reads records at another
ref with `ls-tree` and `cat-file --batch`, parses TOML with regexes, and
aims commits with `rev-parse`, `update-ref` and `symbolic-ref`.
- **Bug:** it slices utf8 strings by blob byte size, so multibyte LLM
`reasoning` text misparses.
- **Bug:** `hashPathPairs.find` is O(n²) over roughly 54k records.
- **`scripts/import-laddr/importer.ts`** (~664–717) moves `HEAD` and refs by
hand to aim its snapshot commit at `legacy-import`.
- **`scripts/scrub-data.ts`** (~237–675) has its own `toToml`,
`parseFlatToml` and path-template resolver.
- **Bug:** a missing field renders as `''`, so record paths collide.
- **Bug:** it writes `blog-posts` as `.toml` instead of the markdown
format.
- **Bug:** attachments are silently dropped.
- It also builds commits from a working tree and `commit-tree`.

In:

- Reads at a ref: use a no-op `repo.transact({ parent: ref }, tx =>
tx.sheet(...).queryAll())`, which commits nothing, until upstream ships
ref-scoped sheets.
- Writes to a branch: use `transact({ parent, branch })` and drop the manual
ref plumbing.
- Port scrub-data's record writes to sheet upserts, so it gets canonical
TOML, correct paths, the markdown format and attachments. Config writes
and the orphan root commit stay on raw git until upstream exposes them.

## Implements

- [behaviors/storage.md](../specs/behaviors/storage.md): every write to the
data repo goes through gitsheets.
- [behaviors/spam-exclusion.md](../specs/behaviors/spam-exclusion.md):
prune semantics are unchanged.

## Approach

Port one script per commit, prune-spam first because it has the live bugs.
Each port keeps the existing CLI flags. Diff the output against the old
script on a copy of `published` before deleting the old code path.

## Validation

- [ ] prune-spam: identical prune set on a copy of `published` vs. the old
script; a record with multibyte `reasoning` parses correctly; runtime
on 54k records is not quadratic
- [ ] importer: snapshot commit lands on `legacy-import` with no manual
`update-ref` / `symbolic-ref`
- [ ] scrub-data: `blog-posts` written as markdown, attachments preserved,
no path collisions on records with missing optional fields
- [ ] No hand-rolled TOML parse/stringify left in `apps/api/scripts`
- [ ] `type-check` + `lint` + full api suite green

## Risks / unknowns

- The no-op-transact read is a trick, not an API. Swap it out when
ref-scoped sheets land upstream.

## Notes

## Follow-ups
66 changes: 66 additions & 0 deletions plans/gitsheets-sync.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
---
status: planned
depends: [gitsheets-freshness]
specs:
- specs/behaviors/storage.md
awaits:
- "JarvusInnovations/gitsheets — opt-in repo.sync() fetch/replay primitive + repo.isAncestor (issue TBD) — replaces store/reconcile.ts"
---

# Plan: replace hand-rolled reconcile with upstream `repo.sync()`

## Scope

`store/reconcile.ts` implements its own fetch, then fast-forward, replay or
escape-hatch, using `merge-tree --write-tree`, `commit-tree` and a CAS
`update-ref`. `routes/internal.ts` adds its own `rev-parse` and
`merge-base --is-ancestor` pre-check. That is general-purpose sync logic
that every gitsheets consumer with a push daemon and an external writer
needs. gitsheets is push-only today by design.

Upstream ask: an opt-in `repo.sync({ remote, branch, onConflict:
'preserve-branch' })`. It should:

- run under the write lock;
- replay local commits onto the fetched tip;
- on conflict, preserve the local work on a side branch and adopt the
remote;
- auto-refresh standing sheets;
- notify the push daemon;
- return `{ outcome, oldCommit, newCommit }`.

Also add `repo.isAncestor(a, b)`.

In: once that ships, delete `store/reconcile.ts`'s git plumbing and call
`repo.sync()` from boot and the webhook. Keep the `outcome` vocabulary the
spec already defines, and map upstream outcomes onto it.

## Implements

- [behaviors/storage.md](../specs/behaviors/storage.md): the Reconcile and
Hot reload sections. Behavior is unchanged; only the mechanism moves.

## Approach

1. File the upstream issue, seeded with cfp's state machine and its
escape-hatch semantics as the reference design.
2. Once it is released, swap the implementation behind
`fastify.reconcileDataRepo` and keep its signature.

## Validation

- [ ] Upstream issue filed and linked here
- [ ] The existing reconcile test matrix (ff, replay, escape-hatch,
in-sync) passes unchanged against `repo.sync()`
- [ ] No `merge-tree`, `commit-tree`, `update-ref` or `merge-base` left in
`apps/api/src`
- [ ] `type-check` + `lint` + full api suite green

## Risks / unknowns

- Upstream may model conflicts differently. If so, the spec's escape-hatch
semantics win, and we adapt or keep a thin shim.

## Notes

## Follow-ups
Loading
Loading