Retest

Retest lets you fix or add tests after a run has finished, then apply the edited test list to the run's existing evidence. It is for the common loop where the agent did the work, but your assertion was too strict, too loose, or missing.

Retest does not drive the agent again. It re-runs every test in the edited YAML, merges the fresh test results into run.json, and makes the latest test results authoritative for CLI, SQL, and the results UI. The old --tests selector was removed; if you pass it, the CLI rejects the command and points you to the flat tests: list.

Remote retest

Use ax run rerun --tests for platform runs created by ax experiment run. Previously this was spelled ax retest.

ax run rerun <run_request_id> --tests --experiment experiment.yaml

Remote retest executes on the AXP platform by forking each target run's live managed sandbox snapshot. You do not need Docker locally and you do not need to download artifacts first. The target run request must already be terminal, have uploaded artifacts, and every selected run must still have a live sandbox snapshot; if runs are still in flight, submit fails with a clear precondition error. If a run's snapshot was garbage-collected or never captured, submit is rejected with a message telling you to create new runs with ax run and retest those.

Preview deployments do not run the platform admission cron automatically. On previews, a submitted retest can remain queued until an operator forces the admission endpoint with the preview CRON_SECRET; production admits queued retests automatically.

The repository includes tooling/scripts/remote-retest-e2e.sh for release verification of this path. It submits a mock platform run, verifies the finalized run has a sandbox snapshot, retests workspace/port checks from that snapshot, verifies a supplied no-snapshot run request is rejected with the GC'd-or-never-captured message, checks latest-result convergence, and confirms each retest forks from a pristine snapshot. Use RETEST_E2E_ASSERTIONS=1,2,5 or RETEST_E2E_SKIP_ASSERTIONS=3 to select individual assertions.

Common options:

ax run rerun <run_request_id> --tests --experiment experiment.yaml --variant <variant_id>

--variant narrows the retest to exact variant ids. Without it, every run in the request is retested.

On completion, the CLI prints a per-run pass/fail table and next-step commands:

Retest: 01J...
Run:    01H...
Targets: 2 runs

Retest 01J... — completed
RUN_ID  VARIANT  REPEAT  STATUS     EVIDENCE  TESTS            ARTIFACTS  DETAIL
...
Next: ax run view <run_id> --tests
Next: ax results query "SELECT test_name, exit_code FROM tests WHERE run_id = '<run_id>'" --table

Failing tests exit 0 because they are evaluation outcomes. Submit rejection, canceled retests, and retest infrastructure errors exit non-zero with the platform message.

Local retest

Use ax run rerun --tests --local for local run groups under ./.axp/runs/:

ax run rerun --tests --local --experiment experiment.yaml --run <local_run_group_id>

Local retest starts a local sandbox, stages each completed variant's available test evidence, and writes updated artifacts back under .axp/runs/<id>/. It is useful when the run was local, when you want to iterate without platform credentials, or when you have downloaded a run package and want to inspect it locally.

Evidence modes

Retest always re-runs the edited experiment's full flat tests: list. What those tests can observe depends on the evidence mode shown in the retest output. Remote retest accepts only snapshot evidence; local retest reports fetched evidence because it stages local artifacts on disk:

• snapshot evidence uses the platform's captured end-of-run sandbox state plus run artifacts. Tests can inspect the captured /workspace as well as $AXP_TRACE_PATH, $AXP_RUN_CONTEXT_PATH, and $AXP_AGENT_LOG_PATH when present.
• fetched evidence is local-retest-only. It downloads and stages persisted run artifacts, stages the trace when available, and also stages generated run context and agent.log when present. It does not reconstruct the original service processes or full /workspace, so tests that require those may fail or report missing evidence.

Platform-run sandbox snapshots are taken after tests complete. Keep tests read-only when you expect future snapshot-backed retests to reflect the agent's final state rather than test-induced mutations.

Snapshot-mode retests execute the bridge binary captured in the run's snapshot. Retest behavior fixes therefore apply only to runs snapshotted after the fix, and the retest JobSpec wire format must remain backward-compatible with older captured bridges.

Snapshots are garbage-collected after they have been idle for 14 days by default. If a snapshot has been deleted or was never captured, remote retest is rejected for that run with a message explaining that the snapshot was GC'd or never captured and that you should create new runs with ax run.

Secrets

Remote and local retest resolve secrets differently:

• Remote ax run rerun --tests resolves environment_variables entries like axp://secrets/<slug> from the org secret store at submit time, the same as ax experiment run. There are no --env or --env-file flags.
• Local ax run rerun --tests --local resolves secret references from local sources by environment variable name: ./.env, repeated --env-file, host environment, and repeated --env overrides.

The resolved values are injected into test execution. As with normal runs, test stdout/stderr is not scrubbed, so do not echo secret values from tests.

Platform sandbox snapshots preserve filesystem and memory state, including injected secrets and provider/model-proxy credentials, until snapshot GC deletes them. Treat snapshot-backed retest evidence as sensitive.

Concurrency and cancellation

Remote retests serialize per run request. If you submit multiple retests for the same run_request_id, they queue FIFO and run one at a time. Retests for different run requests can proceed independently.

Each retest force re-ingests the refreshed artifacts for its target runs, so the final visible result is deterministic: last submitted wins after the queue drains. This is useful when you submit several YAML revisions while debugging a trace assertion; the last retest becomes the latest authority.

Queued or running retests are cancelable through the platform retest cancellation API. If you stop the CLI with Ctrl-C, it only detaches from polling; the retest continues on the platform.

Test-execution history

Retest preserves history under each run's artifact tree:

• run.json.tests is the latest-wins authority used by results derivation.
• tests/*.json contains the latest flat test results.
• test-executions/<ulid>.json records each execution. The first retest backfills the original test execution if needed; later retests append new records.

Download a run to inspect the history locally:

ax run download <run_id>

Then look under the downloaded run directory for test-executions/. SQL queries and the results UI show only the latest execution's rows; history is retained in artifacts for audit and debugging.

Adding a new test to old evidence

You can add a brand-new test to the experiment YAML and retest old runs. The new test runs against the available evidence, appears in tests/, and becomes visible in ax run view --tests, ax results query, and Results with its own test_name.