Repository navigation
Add tests that use a real arc so we test its behavior instead of mocking it - #467
Merged
Merged
Conversation
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The installer silently deletes all contents of an existing nonempty destination, risking irreversible loss of the checkout and uncommitted work even if installation later fails.
1 open issue
What changed in this PR
This PR adds integration tests for Updatebot’s Phabricator submission using real arc against a local fake Conduit API, covering behavior that mocked commands cannot exercise.
Changes:
- Add real-arc tests for submissions, revision management, binary files, and oversized diffs.
- Detect unanswered arc prompts, stop retries, and include the error in Bugzilla comments.
- Install arc and Mercurial using Firefox’s image recipe and require the integration tests in CI.
| File | Description |
|---|---|
tests/setup_real_arc.py |
Install arc and Mercurial using Firefox’s image recipe. |
tests/real_arc.py |
Exercise real arc submission and error behavior. |
tests/functionality_all_platforms.py |
Test job outcomes and reporting for unanswered prompts. |
tests/fake_conduit.py |
Provide a local Conduit API for integration tests. |
test.py |
Register the real-arc test suite. |
tasktypes/vendoring.py |
Include unanswered-prompt errors in Bugzilla comments. |
README.md |
Document integration-test setup and prerequisites. |
apis/phabricator.py |
Detect unanswered arc prompts and prevent retries. |
.github/workflows/ci.yml |
Install required tools and enforce real-arc tests. |
Review effort: Balanced
arc diff exits when it needs an answer, because stdin isn't a TTY, and it asks the same question every time. PDF.js's update to 7a4022e2 (bug 2092381) changed binary cmap files that arc doesn't recognise as binary, so arc asked whether to mark them as binary; we retried it four times over nine minutes, then commented only that we couldn't submit to Phabricator. Give up on the first attempt, and quote the question in the comment.
tomrittervg
force-pushed
the
2026-11-13-tjr-real-arc
branch
from
November 13, 2026 14:42
4609edb to
ce1c07c
Compare
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The test helper commits before configuring a Mercurial username, causing every real-arc test to fail during setup on interactive developer runs without an existing author configuration.
1 open issue
1 resolved since last review
Review effort: Balanced
tomrittervg
force-pushed
the
2026-11-13-tjr-real-arc
branch
from
November 13, 2026 15:02
ce1c07c to
e88768e
Compare
…ator. Updatebot's tests mock arc, so they couldn't catch arc's own behaviour: it stopped to ask about PDF.js's non-UTF-8 files (bug 2092381), and marking an added file binary crashed it on PHP 8 (bug 2092399). tests/setup_real_arc.py installs arc and Mercurial the way Firefox's Updatebot Docker image does, reading the recipe from Firefox so it can't drift: the arcanist revision and Mozilla's patches to it, the evolve pin, the Mercurial version, and Firefox's .arcconfig. It only replaces a directory it installed into itself. tests/fake_conduit.py is a fake Conduit API, with responses shaped like Phabricator's, enough for arc diff and arc call-conduit to create, chain and edit revisions and upload files. tests/real_arc.py runs PhabricatorProvider through the real arc in an hg clone with Firefox's .arcconfig: submitting a commit and a stack of two, reviewers, abandoning and commandeering, files that aren't UTF-8 (with or without the image's patch for them), and a diff too large to submit. They're skipped when arc isn't installed, unless UPDATEBOT_REAL_ARC=required. CI installs arc and Mercurial with setup_real_arc.py and requires these tests to run, which also moves every test that uses hg from Ubuntu's Mercurial to the image's version.
tomrittervg
force-pushed
the
2026-11-13-tjr-real-arc
branch
from
November 13, 2026 23:13
e88768e to
cd44c18
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.