ops(analytics): Umami Postgres fills in ~25 days — capacity or retention action needed #6375

Summary

Priority: P2, with a hard date. The Umami Postgres volume is projected to fill in
about 25 days, and the Umami Storage Monitor has therefore been red on every scheduled
run — roughly every 15 minutes.

This is a capacity problem, not an observability one. #6348 and #6354 restored the
alarm; the alarm is now telling us something true.

Evidence

From run 31323734220,
step Check capacity and projected headroom:

Umami storage warning: 34351.0/50000.0 MB (68.7% used), 0.603 GiB/day,
projected headroom 25.3 days.
##[warning]Umami Postgres storage needs retention or capacity action before the next threshold.

scripts/check-umami-storage.mjs:34 warns below 30 days of headroom and line 222 exits 1,
so the workflow is red by design and will stay red until the trend changes. The same step
failed the same way at 08:30 UTC, before #6354 merged — the monitor was already telling
the truth whenever it managed to run.

Why it matters beyond the red badge

• At 0.603 GiB/day the volume reaches the 90 % critical threshold in about 11 days and is
  full in about 25.
• A red monitor every 15 minutes is alert fatigue: the next genuine storage regression
  arrives into a channel that has been red for weeks.

Actionable fix

Pick one, or a combination:

1. Tighten Umami retention. The umami-retention Railway service already exists — check
   what horizon it enforces and whether it is keeping up with 0.6 GiB/day.
2. Grow the Railway volume past 50 GB.
3. Reduce ingest volume (sampling, event pruning) if the growth rate is itself the anomaly.

Acceptance criteria

• node scripts/check-umami-storage.mjs exits 0 against production, with projected
  headroom above the 30-day warning threshold.
• The Umami Storage Monitor workflow is green on consecutive scheduled runs.
• The chosen horizon or volume size is written down in the runbook so the next operator
  knows what the number is supposed to be.

Found while reviewing the red workflows on main after the 2026-08-09 merges.