Skip to content

test(s16): compare concurrent agent events without scheduler order - #623

Merged
CrazyBoyM merged 1 commit into
shareAI-lab:mainfrom
Henglie:fix/s16-event-order
Nov 15, 2026
Merged

CrazyBoyM merged 1 commit into
shareAI-lab:mainfrom
Henglie:fix/s16-event-order

Conversation

@Henglie

@Henglie Henglie commented Nov 14, 2026

Copy link
Copy Markdown
Contributor

Problem

test_s16_scenario_matches_the_deterministic_runtime compares the full progress list, including the order of concurrent workflow_agent events. agent() appends its event when the runner returns, and pipeline() / parallel() intentionally let agents finish in any order. Python 3.13 and 3.14 interleave those completions differently (and the order varies between runs), while the event set is identical and the runtime behavior is unchanged.

Fixes #622.

Change

  • Compare workflow_agent events as a multiset keyed by phase/label/status, keeping every other field and every non-agent event in exact order.
  • Keep the order assertions that are part of the contract: the Review phase event first, Review -> Verify phase order, verify:* agent events after the Verify phase event, and the workflow log last.
  • No runtime changes.

Verification

  • Python 3.14.9 and 3.13.14: before the change the s16 test failed 8/8; after the change 10/10 repeated runs pass.
  • Python 3.12.13 and 3.11.9 still pass.
  • tests/test_web_scenarios.py: 5 passed, 1 pre-existing failure (test_generated_s16_metadata_extends_s15_without_registry_false_positives, stale versions.json) on all versions.
  • Windows-only verification used a temporary no-op fcntl shim (not committed). Linux/macOS CI was not run locally.
  • The repository python-smoke job currently has the same pre-existing versions.json failure unrelated to this PR (see chore(web): regenerate course data from current chapters #621).

AI assistance was used for implementation and verification; the human submitter reviewed the final diff and evidence.

agent() records its progress event when the runner returns, and pipeline()/parallel() intentionally let agents finish in any order. Python 3.13/3.14 interleave those completions differently (and vary between runs), so compare workflow_agent events as a multiset while still asserting phase order, log placement, and phase tags.
@Henglie

Henglie commented Nov 14, 2026

Copy link
Copy Markdown
Contributor Author

CI 说明:当前两个 Vercel 检查失败是部署账号被 blocked(Account is blocked.),与本次测试改动无关;本地 3.11/3.12/3.13/3.14 的 s16 定向验证与回退证明见 PR 正文。

@CrazyBoyM
CrazyBoyM merged commit 33a5d85 into shareAI-lab:main Nov 15, 2026
0 of 2 checks passed
ruohai0925 pushed a commit to ruohai0925/learn-claude-code that referenced this pull request Nov 17, 2026
…I-lab#623)

agent() records its progress event when the runner returns, and pipeline()/parallel() intentionally let agents finish in any order. Python 3.13/3.14 interleave those completions differently (and vary between runs), so compare workflow_agent events as a multiset while still asserting phase order, log placement, and phase tags.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test: s16 scenario test fails on Python >=3.13 (event order differs)

2 participants