Tests

Tests are how you measure whether an agent succeeded at the experiment task. They are not part of the agent turn: the agent finishes first, then AX runs your test scripts in a completely separate sandbox from the one the agent used. That isolation keeps test scripts, golden answers, and scoring helpers from contaminating the run (the agent never sees them), and keeps test-side mutations from rewriting the agent's working tree during the agent turn. Every experiment needs at least one test.

You can edit the test list later and apply it to the same completed runs (retest) without driving the agent again. For product questions and what to assert, see Experiment Design. For field-level YAML rules, see the tests reference.

How tests are defined

Declare a flat tests: list in the experiment YAML. Each entry is a named bash script. Exit 0 means pass; any non-zero exit means fail.

tests:
  - name: report-exists
    description: Confirms the agent produced the required report artifact.
    script: test -f /workspace/report.json
  - name: under-thirty-tool-calls
    script: |
      [ "$(jq -s '[.[] | select(.kind == "tool_call")] | length' "$AX_TRACE_PATH")" -lt 30 ]

Important properties of that list:

• Separate sandbox. Scripts and test-only path env vars live in the test sandbox, not the agent's. Keep golden answers and answer keys there (or out of the agent workspace entirely); put scoring in tests so they cannot coach or contaminate the run.
• Automatic after each run. When a run finishes, AX executes every test in the list against that variant's evidence.
• One list for every variant. The harness does not scope tests by prompt, product, or agent_mode. If variants expect different outputs, use one test per path and note expected failures in description, or split experiments. See Experiment Design.
• Prefer deterministic checks. When the task has a fixed correct answer, assert files, commands, or product state. Use LLM-as-judge only when you truly need open-ended grading; it is slower, more expensive, and less stable under repetition.

Useful env vars in the test sandbox (full table in the tests reference):

Most tests should check final state: files, commands, endpoints, or product behavior the agent was supposed to produce. Others inspect session data (agent transcript via $AX_TRACE_PATH, stdout, OTEL-related artifacts) to check how the agent got there.

There is no first-class teardown hook after tests today. If setup hits a shared external system (cloud project, shared database), reset it in environment or product setup so parallel variants stay isolated. See the external-systems gotcha in Experiment Design.

Test results

After a run, each test becomes a pass/fail outcome on that variant. View them from the CLI:

ax run view <run_id> --tests
ax experiment query <EXPERIMENT_ID> --metric testPassRate --group-by test

On disk (and in downloaded run packages), latest results live under:

• run.json.tests: latest-wins authority used by results derivation
• tests/<test-name>.json: flat per-test results for the current execution

For SQL over uploaded session-data runs, query test_output (discover tables with SHOW TABLES / DESCRIBE when using legacy scopes). See Data and results and Runs.

A failing test is an evaluation outcome, not a harness crash: the run still completed, and pass rate reflects that assertion.

Retesting

Retest is for the loop where the agent already did the work, but your assertion was too strict, too loose, or missing. Edit the experiment YAML, then re-run the full flat tests: list against the run's existing evidence with ax run rerun --tests. Retest does not drive the agent again. Fresh results merge into run.json and become authoritative for CLI, SQL, and the results UI.

Remote retest

For platform runs from ax experiment run, retest forks each target's live sandbox snapshot on the AX platform:

ax run rerun <run_id> --tests --experiment experiment.yaml
ax run rerun <run_id> --tests --experiment experiment.yaml --variant <variant_id>

The run request must be terminal, with uploaded artifacts and a live sandbox snapshot for every selected run. Pass --variant to limit which variants retest; omit it to retest every run in the request. If a snapshot was garbage-collected or never captured, create new runs and retest those instead.

On completion, the CLI prints a per-run pass/fail table. Open details with ax run view <run_id> --tests, or query test_output for session-data runs. Failing tests exit 0 (evaluation outcomes); submit rejection and infrastructure errors exit non-zero.

Local retest

For local run groups under ./.axp/runs/:

ax run rerun --tests --local --experiment experiment.yaml --run <local_run_group_id>

Local retest stages each completed variant's available evidence, runs the edited tests, and writes updated artifacts back under .axp/runs/<id>/.

What retest can see

Retest runs the full edited tests: list against saved evidence from the original run. It can change tests (and test env vars); it cannot change agent, model, prompt, product, or setup.

• Remote (snapshot): the platform's end-of-run sandbox, including /workspace and session artifacts ($AX_TRACE_PATH, $AX_RUN_CONTEXT_PATH, $AX_AGENT_LOG_PATH when present). Snapshots expire after 14 days idle by default.
• Local (fetched): persisted artifacts staged on disk. Not a full live /workspace or running services, so workspace-heavy tests may fail or report missing evidence.

Keep tests read-only when you care about future snapshot retests: mutations during the first test pass can bake into the snapshot.

Secrets

Remote retest resolves ax://secrets/<slug> from the org store (same as ax experiment run). Local retest uses the same local-first rules as ax experiment run --local. Do not echo secrets from tests; snapshots can retain injected credentials until GC.

Publishing and cancellation

Overlapping remote retests run FIFO; non-overlapping ones can run in parallel. Each target publishes on its own: the latest successful publish wins. Assertion failures still publish; infra failures leave the previous result. Ctrl-C on the CLI only stops polling; the platform retest keeps going. Cancel via the platform API before a target publishes if you need to stop it.

History

CLI, SQL, and the dashboard show the latest test execution (run.json.tests, tests/*.json). Older executions stay under test-executions/ in the run package (ax run download <run_id>). Adding a new test to the YAML and retesting old runs is supported; the new name shows up like any other test.