Repository navigation
[reliability] Daily Reliability Review - 2026-10-19 #70383
Description
Executive Summary
Last 24h (Sentry org github, project gh-aw, spans with gh-aw.workflow.name): 7,403 spans; 3,813 gh-aw.run.status=success, 111 failure, 3,479 null (non-run spans). span.status: 7,292 ok / 111 error. No timeouts or cancellations observed. errors dataset: 0 events. logs dataset: 6 events. gen_ai.response.finish_reasons:length: 0 matches (no truncation detected, but the attribute was not found populated on any span, so this is inconclusive). Overall: moderate failure concentration in two workflows, plus an instrumentation gap.
Top Reliability Findings
| Priority | Workflow | Problem | Evidence | Next Action |
|---|---|---|---|---|
| 1 | PR Code Quality Reviewer | Failed runs | 40 failure spans (24h) | Inspect failing run logs; check agent/chat error |
| 1 | Contribution Check | Failed runs | 20 failure spans | Inspect failing run logs |
| 1 | Smoke Copilot, Issue Triage Agent | Failed runs | 6 each | Check for shared cause |
| 1 | Daily Storify, Copilot Opt, Issue Monster, The Great Escapi, Smoke Copilot - AOAI (Entra), Smoke Gemini | Failed runs | 5 each | Triage individually |
| 3 | MCP gateway | Error spans on POST /mcp/datadog (8) and POST /mcp/deepwiki (4) |
span.status=error counts | Check MCP server auth/availability |
| 5 | All | release null on all 7,403 spans; no gen_ai.response.finish_reasons values |
queries above | See notes |
Representative Traces
View representative traces
- Failed runs, PR Code Quality Reviewer: https://github.sentry.io/explore/traces/trace/3f2af1f23e451a71c33e0c3a0d902ff8 (multiple error spans with
gh-aw.run.status=failureplus achaterror span at 2026-10-19T20:22:45Z; continuity verified) - Failed runs, Contribution Check: https://github.sentry.io/explore/traces/trace/5e5b570c9732ba9c851ad55898565a6d
- Failed runs, Smoke Copilot: https://github.sentry.io/explore/traces/trace/6e3ea84d5435f02dba91a358b03d5d97
chaterror spans: 16chat, 14chat auto, 2chat agent, 2chat o4-mini-aw, 2chat copilot/gpt-6-sol.
Recommendations
- Investigate PR Code Quality Reviewer and Contribution Check first (60 of 111 failures); the failing-run error cause is not available from spans, so check the run logs.
- Investigate
chaterror spans (agent/model errors) across workflows for a shared cause. - Check datadog and deepwiki MCP endpoints for errors.
- Populate
releaseandgen_ai.response.finish_reasonsso regressions and truncation can be detected.
Notes
View notes
- Timeouts/cancellations:
gh-aw.run.statusonly showed success/failure; no timeout or cancelled values appeared. Status values cannot distinguish timeout from failure, so timeouts are inconclusive. releaseis null on all spans andgen_ai.response.finish_reasonshas no values: confirmed instrumentation gap (emit side maps release to resourceservice.version; the Sentryreleasemapping may be backend-dependent. I did not inspectsend_otlp_span.cjsin this run). Truncation findings are inconclusive.- The workflow name is available as
gh-aw.workflow.name(notgh_aw.workflow_name; that query returned an error). 2 failures have workflow name[Filtered]. - OTLP export failures: no errors found (errors dataset empty); not directly testable from Sentry. The
skills/otel-queries/SKILL.mdfile was not found in the repo, so the query loop was applied from the prompt. - Several
search_eventsqueries returned HTTP 400 on missing sort/field syntax; I fixed them once and did not retry equivalent queries. - Run ID: 39337953751
Generated by 🚨 Daily Reliability Review · copilot · auto · 21.5 AIC · ⌖ 12.5 AIC · ⊞ 8.2K · ◷
- expires on Oct 21, 2026, 2:58 PM UTC-08:00
Activity
github-actions commentedon Oct 20, 2026
This issue is being closed as outdated. A newer issue has been created: #70651
View newer issue
This action was performed automatically by the
Daily Reliability Reviewworkflow.