[workflow-health] Activation fails on GitHub API quota exhaustion (daily AIC inspection) around 10:30Z #70777

Summary

Top failing workflows in the 2026-10-21 metrics (20 workflows, each with 1 executed run and 0% success) include several scheduled at ~10:30Z. The ones inspected (glossary-maintainer, feature-grower, ab-testing-advisor) failed on both 10-19 and 10-20 in the activation job.

Error (run 39408690217):
Daily AIC inspection stopped to preserve GitHub API quota: GitHub API quota_exhausted (HTTP 403; resource=core; remaining=0; limit=15000; used=15000; reset=2026-10-20T10:35:13Z) from daily_aic_api_budget.cjs.

Impact

The shared core REST quota is exhausted by the 10:30Z scheduled cluster. Activation fails before the agent starts, so the workflows fail without doing any work. The other failing workflows in the list (daily-doc-healer, daily-security-observability, firewall-escape, smoke-azure-codex and others) were not inspected and may have other causes.

Suggested actions

• Stagger the schedules clustered around 10:30Z.
• Make the daily AIC inspection fail soft (skip with a warning) or wait until retry_not_before instead of failing activation.
• Find the heaviest API consumers in the hour before 10:30Z.

Other

No compile issues were reported. Existing [aw] ... failed issues cover the individual workflows that were not inspected.

Generated by 🏥 Workflow Health Manager - Meta-Orchestrator · copilot · auto · 15.7 AIC · ⌖ 0.719 AIC · ⊞ 12.4K · ◷

• expires on Oct 21, 2026, 8:41 PM UTC-08:00