[ab-advisor] Experiment campaign for daily-windows-terminal-integration-builder: A/B test output_format #69679

🧪 Experiment Campaign: daily-windows-terminal-integration-builder

Workflow file: .github/workflows/daily-windows-terminal-integration-builder.md
Selected dimension: output_format
Triggered by: ab-testing-advisor on 2026-10-17

Background

This daily/manual Codex workflow reviews Windows CLI integration coverage and creates an actionable update issue, or declares noop. It uses openai/gpt-6-luna, dense scenario requirements, GitHub tools, and shared reporting conventions. Random selection chose this workflow from 298 eligible files (344 workflow markdown files, 46 containing experiments:), then chose output_format; the selected file has no experiments and no upstream source: restriction.

Compare prose and bullets for the visible overview/findings while preserving evidence, collapsible details, next actions, and issue/noop decisions. The latest ten scheduled runs, 2026-10-08 through 2026-10-17, all concluded success; this is execution status, not evidence of output quality. Example: latest run.

Deduplication limitation: The supplied security policy prohibits cache access. The recently-analyzed cache was neither read nor updated, so exclusion of the previous 14 selections could not be established.

Hypothesis

H0: Bullets do not improve the triage-ready output rate relative to prose. H1: Bullets increase the rate by 20 percentage points, from a planning assumption of 60% to 80%. These rates are assumptions, not historical measurements. Prose is a specified reference format; historical runs did not have a controlled format.

A triage-ready output contains a concrete affected scenario, evidence/location, and next action when requesting changes, or a clear reason when declaring no changes needed. Format compliance is a separate manipulation check, not proof of improved triage quality.

Experiment Configuration

Add this rich-object block to frontmatter. Start on the actual implementation date; the date below is illustrative.

experiments:
  output_format:
    variants: [prose, bullets]
    description: "Compare visible finding formats while preserving Windows integration review coverage."
    hypothesis: "H0: no improvement in triage-ready rate. H1: bullets improves rate from 0.60 to 0.80."
    metric: "eval:triage_ready"
    secondary_metrics: ["eval:assigned_format_visible", run_duration_seconds, issue_output_rate]
    guardrail_metrics:
      - name: "eval:windows_integration_checked"
        direction: max
        threshold: 0.95
      - name: "eval:failure_reported_or_noop"
        direction: max
        threshold: 0.95
      - name: empty_output_rate
        direction: min
        threshold: 0.0
    min_samples: 82
    weight: [50, 50]
    start_date: "2026-10-17"
    issue: "#aw_campaign"

Assignment correction: In the current implementation, explicit weight: [50, 50] selects weighted random assignment, not round-robin. Retain it for this randomized campaign. To use the requested deterministic round-robin alternative, remove weight; disclose its time/order confounding. Repository-backed storage is currently the default; cache storage is an explicit alternative, not an assumed runtime fact.

Append to the existing evals: list, retaining both existing questions:

  - id: triage_ready
    question: "Does the output provide an actionable Windows integration update request with an affected scenario, evidence or file location, and a concrete next action, or a clear reason that no update is needed?"
  - id: assigned_format_visible
    question: "For an issue output, do the visible overview and findings use the prose or bullets format declared in the final response, and for a noop output is the reason one short sentence?"

Variant descriptions:

• prose: short paragraphs in the visible overview and key findings; reference format.
• bullets: bullet points in those same sections; identical required facts, evidence and review scope.

The manipulation check establishes visible compliance only. Join the declared format to the immutable runtime assignment and flag any mismatch. A noop passes its common short-sentence requirement but provides no evidence about issue-format adherence; report issue-only adherence separately.

Workflow Changes Required

 #### Output rules

 - If changes are required (including updates to `.github/workflows/windows-cli-integration.yml` or this builder file), use `create-issue` to request those updates. Keep headers at `###` (h3) or lower, and wrap long details — full job logs, diffs, or scenario matrices — in `<details><summary><b>...</b></summary>...</details>` blocks per the `reporting` skill.
 - If no changes are needed, use `noop` with a short explanation.
+
+{{#if experiments.output_format == "prose" }}
+When creating an issue, write the visible overview and key findings as short prose paragraphs.
+{{else}}
+When creating an issue, write the visible overview and key findings as bullet points.
+{{/if}}
+Keep the same required facts, evidence, collapsible details, and next actions in either format.
+For noop, keep the explanation to one short sentence in either variant.
+In the final response state: Assigned output format: ${{ experiments.output_format }}.

Success Metrics

Collect one record per assigned run with run ID/attempt, assignment, workflow revision, engine/model, timestamps, issue/noop classification, output availability, eval results and evidence. Duration comes from the completed run's start/completion timestamps; activation-state timestamps alone are insufficient. Implement explicit collectors for duration, issue_output_rate and empty_output_rate; do not assume arbitrary metric names already resolve in the analyzer. Keep raw YES/NO/UNKNOWN values; for the primary intention-to-treat analysis, count missing/UNKNOWN or failed output as not triage-ready and report missingness separately. Validate that aggregation convention before interpreting analyzer output.

Statistical Design

Infrastructure Gate and Analytics Follow-up

The requested field-presence-checker found no absent fields in either compiler/runtime path: analysis_type is parsed at pkg/workflow/compiler_experiments.go:224–225 and displayed at actions/setup/js/pick_experiment.cjs:492,512; tags at compiler 230–231 and picker 493,514–515; notify at compiler 233–255 and picker 494–500,517–518. The schema/runtime metadata path is complete for these three fields. No second issue is created under the explicit absent-field gate. This does not imply that the entire analytics pipeline is complete or that displaying notify destinations sends notifications.

Implementation Steps

• Add the experiments rich-object block, actual start date and resolved campaign issue number.
• Append paired evals while preserving existing questions.
• Add conditional prompt blocks with an explicit comparison to a variant.
• Define and validate outcome collection, missingness handling and assignment/output joins.
• Run gh aw compile daily-windows-terminal-integration-builder and required repository validation to regenerate the lock file; this advisory issue does not modify workflow files.
• Monitor the configured experiment artifact/state path and validate assignment counts; do not treat cumulative counts as output observations.
• Collect artifacts incrementally, then run gh aw experiments analyze with supported metrics and the prespecified primary aggregation.
• Review guardrails, document findings and promote the winning format only if the decision rule is met.

References

• A/B Testing in gh-aw
• Experiment syntax
• Workflow source

Generated by 🧪 Daily A/B Testing Advisor · codex · gpt61 · 28.5 AIC · ⌖ 7.61 AIC · ⊞ 19.4K · ◷

• expires on Oct 31, 2026, 2:35 AM UTC-08:00