[harness-experiment-proposal] daily-rendering-scripts-verifier — tool interaction/deterministic_precondition A/B harness experiment #68997

Summary

daily-rendering-scripts-verifier has the highest avg_aic (220.84) of any workflow in the 14-day fleet summary — nearly double the next-highest (go-fan at 126.79). Its own prompt body regenerates ~130 lines of fully static JavaScript test-harness boilerplate via bash heredoc every single run, even when the artifact under test and the parser/render scripts are unchanged from the last clean check. This proposes a tool interaction / deterministic_precondition experiment that adds a cheap, deterministic skip check ahead of the expensive Phase 3-5 audit/harness cycle.

Observation

avg_aic=220.83969 for daily-rendering-scripts-verifier in per-workflow-summary.json is the highest in the current 14-day fleet summary (next highest: go-fan at avg_aic=126.78738), with avg_action_minutes=17 and run_count=1/success_count=1 (no failures — this is a cost signal, not a reliability one). The workflow's own Phase 4 (cat > /tmp/gh-aw/agent-parser-harness.cjs <<EOF ... EOF, ~70 lines) and Phase 5 (cat > /tmp/gh-aw/agent-render-test.cjs <<EOF ... EOF, ~60 lines) instructions require the agent to re-generate byte-identical boilerplate test-driver code on every run, regardless of whether the most recent agent run or the parser/render scripts under test have changed since the last check.

Hypothesis

Mechanism: Gating Phases 3-5 behind a cheap deterministic check (does the cached run_id match, and have no actions/setup/js/*.cjs files changed since the last clean check?) lets the agent skip re-generating and re-running the harness scripts on runs where nothing could plausibly have changed, reducing output-token generation and agent wall-clock time without touching what is actually verified when something has changed.

Expected direction: decrease in grader:execution-duration (ms).

Minimum effect: >= 5000 ms absolute decrease in execution-duration, aggregated across all sampled runs (including runs where no skip occurs, since the skip only fires on a subset of days).

H0: No meaningful difference in execution-duration between control and candidate.
H1: Skipping Phases 3-5 when the cached run_id matches and no .cjs script changed decreases execution-duration without lowering the correct_verification_outcome eval pass rate.

Candidate Mutation

• primary_dimension: tool interaction
• subtype: deterministic_precondition
• control: Unchanged behavior — the agent always proceeds through Phase 3 (audit), Phase 4 (parser harness), and Phase 5 (render tests) on every run, exactly as today.
• candidate: Immediately after Phase 2 (once RUN_DIR/ENGINE are known), the agent checks the cached /tmp/gh-aw/cache-memory/rendering-scripts-verifier/latest.json from the last clean run. If its run_id matches the current run and no actions/setup/js/*.cjs file has changed since that cached check (find ... -newer <cache file>), the agent skips Phases 3-5 entirely, re-saves the cache entry with today's date, and exits without creating a PR. Otherwise it proceeds exactly as control.

This is a single isolated change: it adds a new, candidate-only early-exit path and does not alter any other phase's existing logic.

Experiment

experiments:
  deterministic_precondition_v1:
    variants: [control, candidate]
    description: "Add a deterministic precondition check that skips the expensive audit and test-harness phases (Phases 3-5) when the same run was already verified clean last time and no parser/render script has changed since — instead of always re-running the full audit+harness cycle regardless of whether anything changed."
    hypothesis: "H0: No meaningful difference in execution-duration between control and candidate. H1: Skipping Phases 3-5 when the cached run_id matches and no .cjs script changed decreases execution-duration without lowering the correct_verification_outcome eval pass rate."
    metric: "grader:execution-duration"
    guardrail_metrics:
      - name: "eval:correct_verification_outcome"
        threshold: ">=0.90"
      - name: "grader:tool-failure-count"
        threshold: "<=5"
    min_samples: 20
    analysis_type: mann_whitney
    decision:
      minimum_effect: 5000
      regression_tolerance: 5000
      confidence: 0.95
    tags: ["harness_dimension:tool interaction", "harness_subtype:deterministic_precondition"]

A tool-failure-count: {} entry was also added to the workflow's existing graders: block (alongside the existing execution-duration: {}) to support the second guardrail.

Guardrails

Expected Economics

daily-rendering-scripts-verifier runs on a daily schedule (plus workflow_dispatch). At roughly one run/day split 50/50, reaching min_samples: 20 per arm requires on the order of ~40 days of accumulated runs — comparable to the existing remove_redundant_context_v1 experiment already running on this same workflow (same cadence, same min_samples). Given avg_action_minutes=17 and avg_aic=220.84 today (per-workflow-summary.json), the candidate is expected to reduce execution-duration on the subset of runs where the skip condition fires (unchanged run + unchanged scripts), while runs where real verification work is still needed should show no meaningful difference from control.

Validation

./gh-aw compile daily-rendering-scripts-verifier --strict and ./gh-aw compile daily-rendering-scripts-verifier --strict --validate both exited 0. git diff --stat after compiling showed only .github/workflows/daily-rendering-scripts-verifier.md and its matching .lock.yml changed — no other files. The compiled .lock.yml contains the new deterministic_precondition_v1 experiment spec (GH_AW_EXPERIMENT_SPEC env var and per-step GH_AW_EXPERIMENTS_DETERMINISTIC_PRECONDITION_V1 outputs), confirming the block was accepted and wired into the runtime templating logic. (Several pre-existing "template separator appears mid-line" warnings on lines 217-237 are false positives from literal {{#if true}}/{{#if false}} strings inside the Phase 5 render-test JS test cases, unrelated to this change.)

Interpretation

Applying this patch only starts the experiment; no decision is interpreted here. Once min_samples: 20 is reached per variant, gh aw experiments analyze daily-rendering-scripts-verifier computes the deterministic EXTEND/PROMOTE/REJECT/INCONCLUSIVE decision unchanged — this workflow never recomputes, reinterprets, or overrides that decision, and any eventual PROMOTE still requires a separate, human-reviewed change through the existing daily-experiment-report deterministic decision engine. This workflow never merges anything itself.

Rollback

Revert the commit that applies this patch (git revert <commit-sha>), then run gh aw compile daily-rendering-scripts-verifier --strict --validate to regenerate the .lock.yml without the experiment. No other files depend on this change — the experiment is fully self-contained in this one workflow's .md/.lock.yml pair, and the experiments/dailyrenderingscriptsverifier state branch (if created) can simply be left unused.

Manual Patch (apply by hand)

diff --git a/.github/workflows/daily-rendering-scripts-verifier.md b/.github/workflows/daily-rendering-scripts-verifier.md
index 14666e4..b78f814 100644
--- a/.github/workflows/daily-rendering-scripts-verifier.md
+++ b/.github/workflows/daily-rendering-scripts-verifier.md
@@ -19,6 +19,7 @@ strict: true
 
 graders:
   execution-duration: {}
+  tool-failure-count: {}
 
 evals:
   - id: correct_verification_outcome
@@ -41,6 +42,24 @@ experiments:
       confidence: 0.95
     tags: ["harness_dimension:context assembly", "harness_subtype:remove_redundant_context"]
 
+  deterministic_precondition_v1:
+    variants: [control, candidate]
+    description: "Add a deterministic precondition check that skips the expensive audit and test-harness phases (Phases 3-5) when the same run was already verified clean last time and no parser/render script has changed since — instead of always re-running the full audit+harness cycle regardless of whether anything changed."
+    hypothesis: "H0: No meaningful difference in execution-duration between control and candidate. H1: Skipping Phases 3-5 when the cached run_id matches and no .cjs script changed decreases execution-duration without lowering the correct_verification_outcome eval pass rate."
+    metric: "grader:execution-duration"
+    guardrail_metrics:
+      - name: "eval:correct_verification_outcome"
+        threshold: ">=0.90"
+      - name: "grader:tool-failure-count"
+        threshold: "<=5"
+    min_samples: 20
+    analysis_type: mann_whitney
+    decision:
+      minimum_effect: 5000
+      regression_tolerance: 5000
+      confidence: 0.95
+    tags: ["harness_dimension:tool interaction", "harness_subtype:deterministic_precondition"]
+
 tools:
   cache-memory: true
   bash:
@@ -170,6 +189,21 @@ If `agent-stdio.log` is missing (e.g. the run used the Copilot engine which stor
 
 If no agent output file is found at all, run Phase 3 (the `audit` MCP tool) which will report the correct path in its `overview.logs_path` field.
 
+{{#if experiments.deterministic_precondition_v1 == 'candidate' }}
+## Phase 2.5: Deterministic Skip Check
+
+Before running the expensive audit and harness phases, check whether this exact run was already verified clean and no parser/render script has changed since:
+
+1. If `/tmp/gh-aw/cache-memory/rendering-scripts-verifier/latest.json` does not exist, continue to Phase 3 (no prior check to compare against).
+2. Read that file with the Read tool (bash cannot access `/tmp/` in this environment). If its `run_id` does not equal `RUN_DIR`, or its `status` is not `"ok"`, continue to Phase 3.
+3. Otherwise check whether any parser/render script changed since the cached check:
+   ```bash
+   find ${{ github.workspace }}/actions/setup/js -name "*.cjs" -newer /tmp/gh-aw/cache-memory/rendering-scripts-verifier/latest.json
+   ```
+4. If that command lists any file, continue to Phase 3 (a script changed since the cached check — re-verification is required).
+5. If it lists nothing, the same run was already verified clean against unchanged scripts: skip Phases 3, 4, and 5 entirely, re-save the Phase 6 "No Issues Found" cache entry with today's date, and stop without creating a PR.
+{{#endif}}
+
 ## Phase 3: Audit the Run
 
 Use the agentic-workflows MCP `audit` tool to get the full run report and confirm which agent output file to use:

Application Plan

1. Save the diff above as proposal.patch.
2. git apply proposal.patch
3. gh aw compile daily-rendering-scripts-verifier --strict --validate
4. Open a PR manually if compilation succeeds.

Generated by 🧫 Daily Harness Experiment Proposer · claude · agent · 221.8 AIC · ⊞ 7.4K · ◷

• expires on Oct 22, 2026, 12:48 AM UTC-08:00