Repository navigation
[reliability] Daily Reliability Review - 2026-10-22 #71239
Description
Executive Summary
- Window: last 24h, Sentry org
github, projectgh-aw. Datasets: spans have data, logs have data, errors are empty (explicit observability finding: no Sentry error events). - One confirmed operational failure (Issue Triage Agent, run status
failure). Two other traces containspan.status:errorgateway/MCP spans with no workflow identity (inconclusive outcome). - No
gen_ai.response.finish_reasons:lengthspans were found (query returned no results), butfinish_reasonsandreleasewere null on all sampled spans (instrumentation gap, so truncation cannot be ruled out).
Top Reliability Findings
| Priority | Workflow | Problem | Evidence | Next Action |
|---|---|---|---|---|
| 1 | Issue Triage Agent | Run ended gh-aw.run.status:failure; gen_ai.chat spans (chat copilot/gpt-6-sol) and conclusion spans marked error |
6 error spans in trace fe998b6da76396711dc95fd0df72f19e, 14:45-14:52 UTC |
Inspect the run logs for the failure cause; the span lacks a status message field in the queried attributes |
| 2 | unknown (no gh-aw.workflow.name on spans) |
gateway.backend.execute spans error out in bulk |
trace 498902e422a8c3db64d12c671697901b: ~55 error spans, 20:47-20:56 UTC (durations ~3ms); trace 9bdffb6ebe2e6e9e43c993567f774c08: 34 error spans at 13:25 UTC incl. execute_tool work_queue_claim_finish |
Identify the backend error from gateway logs; tag gateway spans with workflow name |
| 3 | unknown | Single error-status spans | traces 4159b25d5d7269fa304c4211f8ce04f1 (22:16), e7d3fb479a1930968a809787e2c43e50 (21:55), 80e25af5986cf2fa4045083a65fcecb0 (18:19), 748bebc14e580537ba62d8e265967192 (14:36) |
Triage after #2; inconclusive |
| 4 | n/a | Instrumentation gap: gen_ai.response.finish_reasons and release null; span.status/run status absent on some spans |
0 of 100 sampled spans (longest by duration) had finish_reasons or release | See recommendations |
Representative Traces
View representative traces
- Failed run: https://github.sentry.io/explore/traces/trace/fe998b6da76396711dc95fd0df72f19e (Issue Triage Agent)
- Gateway errors: https://github.sentry.io/explore/traces/trace/498902e422a8c3db64d12c671697901b
- Gateway/MCP tool error: https://github.sentry.io/explore/traces/trace/9bdffb6ebe2e6e9e43c993567f774c08
- Longest latency (success, not a failure): https://github.sentry.io/explore/traces/trace/a0303865db7534167d6e7567630947ed (The Daily Repository Chronicle,
chat copilot/grok-4.7, max 2,388,316 ms ≈ 39.8 min); also https://github.sentry.io/explore/traces/trace/987c04fbfa65c9fc0126a02aa9387c1a (Static Analysis Report, 2,062,093 ms ≈ 34 min) - Reviewer run: 39868045092
Recommendations
- Investigate the Issue Triage Agent failure (trace
fe998b6d...) via the workflow run logs and confirm whether it recurs. - Add
gh-aw.workflow.name(and error message) togateway.backend.executeandmcp.tool_callspans so error bursts can be attributed. - Emit
service.version/releaseandgen_ai.response.finish_reasons(needsstopReasonin runtime metrics, seeactions/setup/js/send_otlp_span.cjs) so truncation and regression checks become possible. - Consider alerting on the long-running (30-40 min) chat spans if they exceed expected workflow timeouts.
Notes
View notes
skills/otel-queries/SKILL.mddoes not exist in the repo; the prescribed loop was followed from the task instructions.- Attribute
gh_aw.workflow_nameis unknown to Sentry; the emitted names aregh-aw.workflow.nameandgh-aw.run.status. I did not re-inspectsend_otlp_span.cjsin this run. - Of the 100 longest spans sampled: 92 ok/success, 4 error/failure, 4 ok with no run status, 2 with no status. This sample is biased towards long spans. The
span.status:errorquery returned 102 rows (limit 100) so counts are lower bounds. - No timeout, cancelled or OTLP export failure spans were specifically queried by status value beyond
!gh-aw.run.status:success(which mainly returned non-run child spans). Those classes are inconclusive. - Trace details were verified via
search_eventsfiltered bytrace:<id>(noget_trace_detailstool available).
Generated by 🚨 Daily Reliability Review · copilot · auto · ~21 AIC · ⌖ ~13 AIC · ⊞ 8.3K · ◷
- expires on Oct 24, 2026, 2:58 PM UTC-08:00
Activity
github-actions commentedon Oct 23, 2026
This issue is being closed as outdated. A newer issue has been created: #71507
View newer issue
This action was performed automatically by the
Daily Reliability Reviewworkflow.