[reliability] Daily Reliability Review - 2026-10-23

Last 24h, org github, project gh-aw: spans = 17,000 (aggregate query; a second count showed 17,022), errors = 0, logs = 6 (only "cli starting" info entries).
Health: mostly healthy; recurring failures are concentrated in ~10 workflows (about 38 failed-run spans by aggregate). No timeouts/cancellations were identifiable. Runtime outcome for length truncation is inconclusive (see Notes).

Priority	Workflow	Problem	Evidence	Next Action
1	Agent Job Health Monitor	`gh-aw.run.status=failure`, `span.status=error`	6 spans, spans query 24h	Inspect failing run logs
1	Daily Community Attribution Updater	failure	6 spans	Inspect failing run logs
1	Smoke Gemini / The Great Escapi / Package Specification Enforcer / Smoke Copilot - AOAI (Entra)	failure	5 spans each	Check for common cause (engine auth/config)
2	Daily Security Observability Report (2), Daily AIC Consumption Report (1), Agent Container Smoke Test (1), "[Filtered]" (2)	failure	counts as listed	Triage after above
5	All	`gen_ai.response.finish_reasons` null on all 17,022 spans	instrumentation gap	See recommendations

View representative traces

Failed run (Smoke workflows, gh-aw.run.status:failure): https://github.sentry.io/explore/traces/trace/c9eddd10c8e10f217dd901a685801828 (3 spans seen on query, 00:52-00:54 UTC). Full-trace continuity not verified because get_trace_details is not exposed in this MCP build.
Longest latency outlier: "chat agent" span 2,107,075 ms (~35 min), Static Analysis Report, trace 2cbca3775bd7ed8ea4ba914f5f1fcc53. Others: Agent Job Health Monitor 1,841,200 ms (a5be429d1d341b32b6be2247c1b633d8); Copilot Session Insights 1,716,809 ms (ad0a3c0c2c0b7c701533de1a48c3157b).
Timeouts, cancellations, OTLP export failures, finish_reason=length: no evidence available (see Notes).

Investigate the failing runs for Agent Job Health Monitor and Daily Community Attribution Updater first (6 each; highest counts).
Check whether the four 5-failure workflows share a cause (smoke/engine auth).
Fix gen_ai.response.finish_reasons emission: send_otlp_span.cjs only sets it when a stopReason exists (~line 2411); ensure all engines supply it so truncation can be detected.
Review the 35-minute Static Analysis Report agent span for timeout headroom.

View notes

Errors dataset empty; logs dataset has only 6 info entries; neither provides failure evidence.
Attribute names: gh_aw.workflow_name is unknown in Sentry; the emitted gh-aw.workflow.name and gh-aw.run.status are present. span.status is present (error on failures). gen_ai.response.finish_reasons is null on all spans, so length truncation is inconclusive and a confirmed instrumentation gap. release not checked beyond emit-side mapping (service.version resource attribute, send_otlp_span.cjs line ~377).
Many Issue Monster / PR Sous Chef spans have null gh-aw.run.status (370 and 195), likely non-conclusion spans; not analysed.
Cancelled/timed-out statuses did not appear among non-success run statuses (only failure). OTLP export failures cannot be queried directly; no related errors/logs found.
One field-type 400 occurred on the first errors count query (sort by unselected field); resolved by selecting the sort field.
skills/otel-queries/SKILL.md does not exist in the repo; query loop followed from the prompt.

Generated by 🚨 Daily Reliability Review · copilot · auto · ~20 AIC · ⌖ ~13 AIC · ⊞ 8.3K · ◷

expires on Oct 25, 2026, 2:58 PM UTC-08:00

Navigation Menu