Repository navigation
[reliability] Daily Reliability Review - 2026-10-21 #70964
Description
Executive Summary
- Sentry (org
github, projectgh-aw) is reachable. Last 24h:spanshas data,logshas data (onlycli startinginfo entries, latest 2026-10-21T00:30Z), anderrorsis empty. - Runtime outcome is derivable from
gh-aw.run.status: 13 distinct failed runs across 12 workflows. A 100-row sample ofgh-aw.run.status:*was allsuccess. The sample is capped at 100 rows, so no overall failure rate is claimed. - No
timed_out,timeoutorcancelledrun status. Nospan.statusofinternal_error,deadline_exceeded,cancelledorunknown_error. Nogen_ai.response.finish_reasons:length. This is confounded because the finish_reasons field returned no rows at all (see Notes). - Only recurring failure: Contribution Check (2 runs).
Top Reliability Findings
| Priority | Workflow | Problem | Evidence | Next Action |
|---|---|---|---|---|
| 1 | Contribution Check | Recurring run failure | gh-aw.run.status:failure, runs 39638883189 and 39604157942 |
Inspect both run logs and fix the common cause |
| 2 | Typist - Go Type Analysis | Run failure | run 39592228678 | Inspect the run log |
| 2 | Smoke Copilot, Smoke Copilot - AOAI (Entra), Smoke Gemini, Smoke Pi, Smoke DeepSeek Harness, Agent Container Smoke Test | Smoke/engine failures across many engines | runs 39510846534, 39520956591, 39522802023, 39521839723, 39522553142, 39522266772 | Check for a shared cause (auth/network/container) |
| 2 | Daily Model Inventory Checker, Daily Community Attribution Updater, The Great Escapi, [Filtered] |
Single failures | runs 39517945363, 39534230107, 39546110649, 39523033703 | Triage individually |
| 3 | Several (chat agent / chat openai/gpt-6-luna spans) | Long agent spans, up to about 2513s | max span.duration 2,512,921 ms (trace ad1f4a784020bbfe03ecc279c319bc81); 2,321,775 ms (trace 0d4551aabfb444f0e5e2fdc2106c74a3) |
Outlier only; compare against a baseline before treating as a regression |
| 4 | n/a | Instrumentation gap | errors dataset empty; gen_ai.response.finish_reasons returned no rows; release not populated |
See Notes |
Representative Traces
View representative traces
- Failure (Contribution Check, run 39638883189): https://github.sentry.io/explore/traces/trace/14a95983498351cff2e42ad5409d7fa8
- Failure (Contribution Check, run 39604157942): https://github.sentry.io/explore/traces/trace/164d50c902114b3215c7f64effe457cd
- Failure (Typist, run 39592228678): https://github.sentry.io/explore/traces/trace/426cc7e5a64083c91ef24c8537392ee7
- Longest span (2513s): https://github.sentry.io/explore/traces/trace/ad1f4a784020bbfe03ecc279c319bc81. The trace was queried by
trace:<id>and returned a continuous multi-span trace from Static Analysis Report. It listsexecute_tool create_issueand a 59s model call with 116 input tokens. - Timeouts, cancellations, OTLP export failures and finish_reasons:length: no matching traces.
Recommendations
- Investigate Contribution Check first. It is the only workflow failing repeatedly.
- Look for a common cause in the smoke tests, which failed across 6 engines and an agent container within about the same window.
- Check why the
errorsdataset is empty and whyreleaseis unset, so failures can be tied to versions. - Check why
gen_ai.response.finish_reasonsis never populated before relying on truncation checks.
Notes
View notes
skills/otel-queries/SKILL.mdwas not present in the checkout. I followed the fixed loop from the task prompt instead. The Sentry CLI has nowhoamiorget_trace_details; I usedfind_organizationsandsearch_eventsfiltered bytrace:<id>.gh_aw.workflow_nameis an unknown attribute in Sentry. Workflow identity is available asgh-aw.workflow.name, which matchessend_otlp_span.cjs.span.statusis present (values seen:ok). It showed no error statuses; outcome is carried bygh-aw.run.status.gen_ai.response.finish_reasonsandreleasereturned no values.send_otlp_span.cjsemits finish reasons only when runtime metrics includestopReason, and emits release as resourceservice.version, which may not map to Sentryrelease. This is an instrumentation gap, so the length-truncation check is inconclusive.- One failure run has workflow name
[Filtered], so scrubbing may hide workflow identity. - The failed-run list is from one query with a 100-row limit and may be incomplete. I did not open the GitHub run logs, so failure causes are unknown.
- Per-workflow failure rates were not computed.
Generated by 🚨 Daily Reliability Review · copilot · auto · 22.1 AIC · ⌖ 12.7 AIC · ⊞ 8.3K · ◷
- expires on Oct 23, 2026, 2:58 PM UTC-08:00
Activity
github-actions commentedon Oct 22, 2026
This issue is being closed as outdated. A newer issue has been created: #71239
View newer issue
This action was performed automatically by the
Daily Reliability Reviewworkflow.