Argus Testing

Argus

Testing for coding agents: explore real journeys, verify every bug, keep it fixed.

Argus is an MCP server that gives the agent you already use a complete testing workflow. The agent walks your app like a user, Argus independently rechecks what looks wrong, and every finding states how it is supported.

Quick start · How it works · What's new · Why Argus · Tools · Benchmarks · Product page

English (primary) · Simplified Chinese

"Test my app at http://localhost:3000 — find real bugs."

That's the whole prompt. Your agent drives the browser through Argus, walks the goals you care about, rechecks every suspected bug on a fresh load, and hands back a report. A bug it confirms can become a Playwright test you keep in CI.

Quick start

Argus needs Python 3.11+ and Chromium. With uv there is nothing to install globally:

uvx --from playwright playwright install chromium
uvx --from argus-testing argus-mcp --doctor     # checks Chromium, a real launch and the report directory

Each failed --doctor check prints the exact fix. If your default interpreter is older than 3.11, use uvx --python 3.12 --from argus-testing argus-mcp. If an upgrade reports a missing browser, run uvx --from argus-testing python -m playwright install chromium.

Then connect Argus to your agent:

claude mcp add argus -- uvx --from argus-testing argus-mcp

codex mcp add argus -- uvx --from argus-testing argus-mcp

{
  "mcpServers": {
    "argus": {
      "command": "uvx",
      "args": ["--from", "argus-testing", "argus-mcp"]
    }
  }
}

pip install argus-testing
playwright install chromium
claude mcp add argus -- argus-mcp

pip install 'argus-testing[mac]'
brew install cliclick          # keystroke / coordinate fallback
argus-mcp --doctor             # web checks plus Screen Recording + Accessibility grants
claude mcp add argus-screen -- argus-mcp --tool-profile screen

Then ask your agent to test your app. For a scoped review, it can pass start_session explicit goals, constraints and an advisory time_budget_minutes. To confirm the installed version, stdio handshake and discovery without a browser or a model, run uvx --from argus-testing python -m argus.install_check.

How it works

flowchart LR
    A(["observe"]) --> B{"looks wrong?"}
    B -->|not sure| C["act: click · type · resize · test_action"]
    C --> A
    B -->|suspected bug| D["verify_persistence: inspect a fresh load"]
    D --> E["record_bug with an independent recheck"]
    E --> F(["reproduced · not reproduced · inconclusive"])
    F --> G[["report: HTML · JSON · JUnit · SARIF"]]
    F -->|reproduced| H[["export_test: a failing Playwright test"]]

The agent supplies the judgment. Argus supplies:

• Description-keyed tools. click_what("Login button"), not click(7). A description from observe can be passed back as-is; an ambiguous one returns candidates instead of a guess.
• Actions that report what changed. test_action and test_form return the before/after difference and the requests fired. Pass expect to predict the outcome, and a mismatch is flagged as a lead.
• A coverage ledger. Goals stay visible until marked exercised or blocked with evidence. Argus links the URLs, actions, screenshots and checks from that window automatically.
• Reproduction receipts. record_bug rechecks the stated symptom on fresh loads before calling it reproduced. It can also replay a recorded multi-step journey from a cold start.

A receipt confirms the specified observable symptom, not a root cause or that the behavior violates the product requirements. Fresh loads keep authentication and storage by default, and replay can repeat real writes.

Evidence and reports

Reports keep reproduced symptoms, observations and inconclusive checks apart. They also keep goal coverage, including unfinished goals, so an incomplete pass is never presented as exhaustive.

From a verified bug to a regression test

When a finding is reproduced by a fresh-load text check, export_test writes a standalone Playwright test that replays the recorded journey and asserts the opposite of the symptom. It fails while the bug is present and passes once it is fixed. The body of a test exported against TodoMVC, whose todos do not survive a reload:

page.goto(_url("/"))
page.get_by_placeholder("What needs to be done?", exact=True).fill("Buy milk")
page.get_by_placeholder("What needs to be done?", exact=True).press("Enter")
page.wait_for_load_state("networkidle")
page.get_by_placeholder("What needs to be done?", exact=True).fill("Walk dog")
page.get_by_placeholder("What needs to be done?", exact=True).press("Enter")
page.wait_for_load_state("networkidle")
_in_row(page, "Walk dog", lambda scope: scope.locator("input[type=\"checkbox\"]")).click()
page.wait_for_load_state("networkidle")
page.goto("about:blank")
page.goto(_url("/"))
assert _present("Walk dog", page), SYMPTOM_MARKER + ": 'Walk dog' is missing"

• Stable locators. They are captured from the elements Argus actually resolved. A repeated control is scoped to its row by the row's title, not by position.
• No secrets in the file. Passwords and tokens are read from environment variables.
• No pass from a broken page. HTTP errors and login walls stop the check; a missing symptom on those pages does not mean the bug is fixed.
• Nothing guessed. Steps Argus cannot translate are marked TODO(argus) and listed.
• Optional run. run=true runs the file once to show that it fails on the symptom. Because this repeats the journey's writes, use it only on data you may change.

Live monitoring from a real local fixture run

--monitor-port 8765 adds a local read-only dashboard of actual runtime events (sessions, checks, findings), not model reasoning or a simulated percentage. The run below captured a saved-name persistence failure, one reproduction receipt and two exercised goals.

What's new in 0.12.0

• Verified bug → regression test. New export_test writes a Playwright test from a reproduced finding; it fails now and passes once the bug is fixed.
• Ready-made skills. /argus-testing:test in the Claude plugin and argus-test in the new Codex plugin run a full review: goals, the test_action expectations, a visual pass with screenshot and check_layout, verified findings and a coverage summary.
• Real-app interactions. Found by testing TodoMVC and Flask's tutorial app:
  • controls that appear only on hover;
  • double-click to edit, including on plain text;
  • confirm dialogs, reported or accepted with accept_dialog=true;
  • fields named only by a nearby label;
  • going back and scrolling, which are now part of replays and exports.
• Sharper resolution. Descriptions copied from observe resolve as-is. test_form sets checkboxes and radios. Text checks handle punctuation at the edges of a target.
• --doctor for web testing. It launches Chromium for real and checks the report directory, on every platform.

Upgrading from 0.11: the web profile has 29 tools (was 28), adding export_test. See the changelog for every change.

Why Argus is different

Exploratory testing with a repeatable evidence trail. The agent investigates journeys beyond prewritten checks, Argus rechecks what it suspects, and the results stay honest about what was and was not confirmed.

How it compares

Microsoft's Playwright MCP provides browser automation, with opt-in test assertions (--caps=testing) such as verifying visible text or element values. Argus is organized around the whole testing process instead: goals declared up front and tracked to evidence, persistence checks after writes, findings linked to the actions, checks and screenshots behind them, independent rechecks, and regression tests and rechecks of earlier findings against later builds. Argus includes its own browser control and can run alongside other tools. It does not replace the host model's judgment or guarantee discovery of every defect.

The paired evaluation compares a conventional Playwright MCP adapter with Argus using the same host and tasks. That adapter is not the official Playwright MCP package, and these results must not be presented as a ranking of third-party products.

Benchmarks

Benchmarks and evaluations run from a source checkout (argus_lab/); they are not part of the installed package.

$ python -m argus_lab.bench --target all

  buggytasks    22 / 22  = 100 %   ·  mechanical bugs (console errors, fake delete, auth bypass…)
  darkshop      12 / 12  = 100 %   ·  human-eye bugs (fake scarcity, lying toasts, stale state…)
  ──────────────────────────────────────────────────────────────────────
  total         34 / 34  = 100 %   ·  reproducible from git clone in two commands

34 / 34 is the capability ceiling: what is findable through the tool surface, measured by deterministic scripts. It is deliberately separate from how often a given model uses the tools well, which is noisy and reported separately.

Does installing Argus improve an agent? Measure it with the paired evaluation harness: the same model, task, fixture, initial state and budget, with both arms and failures retained. In the 0.8 Codex evaluation, ten matched pairs, Argus found 6/6 defect opportunities versus 5/6 for the browser baseline, including one post-hoc fixture defect. Both had zero observed false positives and equal declared-goal coverage. Argus used more time and tokens. On four repeated pilot cases, the targeted update reduced Argus's median calls from 17 to 9. This small synthetic sample does not establish general superiority.

The no-model response encoding benchmark compares identical synthetic evidence. Delta mode reduces aggregate result JSON by 48.25% versus compact mode across those 24 responses, reproduced with 0.12.0. This measures bytes, not agent token usage or testing quality.

python test-site/app.py           # BuggyTasks  :5555
python human-eye-fixture/app.py   # DarkShop    :5556
python -m argus_lab.bench --target all

Tool surface

argus-mcp starts with the web profile: one complete testing loop of exploring, acting, verifying, recording, reporting and re-checking earlier findings. The counts are also available from the installed server:

uvx --from argus-testing argus-mcp --list-tools
uvx --from argus-testing argus-mcp --tool-profile advanced --list-tools
uvx --from argus-testing argus-mcp --tool-profile screen --list-tools

Earlier profile names still start the server: agent and core load the web profile, so their tool sets change (19 → 29 and 30 → 29 tools).

uvx --from argus-testing argus-mcp --tool-profile advanced --unsafe

Configuration and operations

uvx --from argus-testing argus-mcp --response-mode compact --monitor-port 8765

argus-regression http://localhost:3000 --output ./argus-reports

argus-mcp --monitor-port 8765
# Or choose an available port automatically:
argus-mcp --monitor-port 0

argus-cleanup --output ./argus-reports --keep-runs 20
argus-cleanup --output ./argus-reports --keep-runs 20 --apply

Local-first security and privacy

Argus runs on your machine and does not send telemetry to an Argus-operated service. Reports and screenshots stay under ./argus-reports by default; your MCP host and its configured model provider can still receive tool results included in the conversation. Browser actions and native macOS controls can cause real side effects, so use test accounts and non-production data wherever possible.

• Inputs: typed, pasted and selected values are held in a session-local input vault, not in action traces or exported replay steps. Exported steps that need input are marked requires_input.
• Exported tests: sensitive inputs are read from environment variables.
• Capsules: they deliberately contain credentials and are saved with owner-only permissions.
• Approved directories only: local file uploads and file:// pages require a directory approved in ARGUS_FILE_ROOTS.
• Hidden values: cookie and storage values are hidden unless ARGUS_ALLOW_SENSITIVE_READS=1.

Configure these in the MCP host's server environment, not through page content or tool arguments:

{
  "ARGUS_FILE_ROOTS": "[\"/absolute/path/to/test-fixtures\"]",
  "ARGUS_ALLOWED_ORIGINS": "[\"http://localhost:3000\"]"
}

The optional origin policy restricts tool targets, not browser network egress. Read the full privacy disclosure and security policy before using Argus against sensitive systems.

Philosophy

Project layout

argus/
├── mcp_server.py      # process entry point and compatibility imports
├── entrypoints/mcp/   # public tool catalog, profiles and MCP adaptation
├── application/       # session manager and focused QA use cases (incl. test export)
├── domain/            # models, errors, states, replay and verification contracts
├── adapters/          # monitoring, logs, file journal, report service and exporters, JS assets
├── bootstrap.py       # composition root: injected backend/repository factories
├── ports.py           # narrow lifecycle, observation, verification and storage protocols
├── settings.py        # validated, per-runtime configuration
├── browser.py         # Playwright adapter (public operations; private page ownership)
├── screen/            # macOS AX adapter, permissions, safety
├── resolver.py        # description → element (web + screen)
└── cli.py             # argus-regression and argus-cleanup commands
argus_lab/            # repository-only: not in the installed wheel
├── evaluation/        # paired schemas, isolated fixtures, opt-in host runner, adjudication
└── bench/             # deterministic capability benchmark
plugins/              # Claude and Codex plugin bundles with the test skills
test-site/            # BuggyTasks  (22 mechanical bugs)
human-eye-fixture/    # DarkShop    (12 human-eye bugs)

Start with Architecture, design decisions and Contributing for implementation boundaries, tests and extension points.