Runs
Manage an experiment’s execution from start to completion.
Each run is a single isolated execution of an experiment variant in a dedicated sandbox.
The variant determines the run’s agent, model, prompt, product, and environment configuration. When the run starts, AX Cloud spins up a fresh sandbox, applies the product and environment setup, and starts the agent with the selected model and prompt.
As the agent works, AX Cloud captures available agent activity and workspace changes for evaluation and analysis (see Captured run data).
When the agent finishes its work, AX Cloud snapshots the sandbox and runs the experiment's tests in a separate sandbox created from that snapshot. This is to ensure the agent cannot cheat by previewing the tests before they are run.
Starting a run
Right now you can only start runs with the CLI command ax experiment run, which accepts an experiment YAML path or a registered experiment-id.
Run `ax learn run-experiment`, then help me run my experiment.Without selectors, ax experiment run creates one run for every computed variant in the specified experiment.
AX executes the selected runs in Fiveonefour-managed sandboxes and stores the resulting run data in your organization.
Run specific variants
Selectors can be used to target specific variants to run without editing the experiment. This is useful if you add or change a specific variable (e.g. you add or edit a prompt), after already having run your experiment.
ax experiment run experiment.yaml --model gpt-4o-miniRun multiple times
You can use --repeat to create several independent runs for each variant.
ax experiment run experiment.yaml --repeat 3Run from your GitHub CI
From a GitHub pull request, comment !ax run or use the run checkbox in the Fiveonefour comment. See GitHub CI for more details.
Monitoring runs
Use the experiment's Runs List page in the web app or ax run list to check the status of individual runs and groups of runs:
Use the `ax run list` help me monitor my active, queued, and recently runs. Summarize them by status and highlight runs that did not complete normally.Run status
| Status | Meaning |
|---|---|
waiting-for-ci | A pull request run is waiting for its CI build artifacts. |
queued | The run is waiting for execution capacity. |
running | Setup, the agent, or tests are in progress. |
completed | Execution and testing finished. |
errored | The run did not finish normally. |
canceled | The run was intentionally stopped. |
completed does not mean that every test passed. See Tests for pass/fail outcomes.
Troubleshooting errored runs
You can locate errored runs in the Runs List view in the web app or with ax run list in the CLI, then open the Run Details view or use ax run view to troubleshoot individual runs.
Run `ax learn debug-run`, then help me troubleshoot my errored runs. Identify the likely cause(s) and whether the experiment definition needs to change before running again.When available, both the web app and CLI show the last lifecycle phase AX observed for the run, a machine-readable exit_reason, and an error_message. The phase is absent when AX Cloud has not reported the first sandbox phase, such as when a required CI artifact never arrives. A dispatch failure can show no phase or sandbox, depending on how far it progressed.
| Phase | What is happening | Error values you may see | What to inspect |
|---|---|---|---|
| No phase | AX Cloud has not begun preparing the sandbox. | ci_artifacts_missing, dispatch_failed | The error message and CI configuration |
sandbox | AX Cloud is preparing the sandbox and staging run inputs. | staging_failed, dispatch_failed | The error message and staged-file details |
setup | The product and environment setup scripts are running. | setup_failed | Setup output |
checks | Setup checks are verifying the sandbox before the agent starts. These are separate from tests. | setup_check_failed | Setup-check output |
agent | The agent is working on the prompt. | driver_error, model_unsupported_by_harness, timeout, idle_timeout | The error message, agent logs, last agent activity, and the hit_timeout measurement |
tests | The experiment's tests are evaluating the agent's work. | artifacts_upload_failed may leave tests as the last reported phase | Test output, the publication error, and any available captured evidence |
| Any phase | AX Cloud is waiting for the run to report progress or finish. | run_budget_exceeded, agent_start_timeout, sandbox_timeout, never_started, starved, orphaned | The error message and last recorded activity (see Runs that reach a time limit) |
A run that ends with timeout or idle_timeout still gets its tests. AX Cloud saves the sandbox when the agent stops and runs the tests against that saved work, so the run's test results appear on the run and in ax run view --test. The run stays errored, with its exit reason. Its tests do not count toward the experiment's pass rate, because the run did not finish on its own. If AX Cloud could not save the sandbox, the run has no test results and its error message says why.
idle_timeout means the agent produced no output for limits.max_idle_seconds (10 minutes by default), so AX stopped it and, if it could save the sandbox, ran the tests on the saved work. The run is errored, with exit reason idle_timeout. If the agent normally runs a long command with no output, raise the limit, up to just under 720 seconds (12 minutes). AX Cloud reaps a run that has been silent for 15 minutes, and the stop needs the remaining time to save the workspace and start the tests.
An errored status is terminal. The run cannot resume or be repaired. After identifying the cause, create new runs for the affected variants.
If the experiment definition caused the error, fix the YAML first. If the error was transient, use the same definition again.
Runs that reach a time limit
Cloud runs have a time limit. A run that runs out of time ends errored, and
the exit reason says which clock ran out:
exit_reason | What ran out | What to do |
|---|---|---|
run_budget_exceeded, timeout | The run's own time budget: limits.max_time_seconds, or the plan cap that clamped it. | Raise the limit, or shorten the task. |
agent_start_timeout | The startup deadline, 10 minutes by default. The sandbox produced no heartbeat and no activity at all, so AX reclaimed it before it used any of its time budget. | Run the variant again, and check the product and environment setup if it keeps happening. |
sandbox_timeout | The sandbox ceiling that applies to your organization. A run ends this way when its own budget could not be read, or was clamped to that ceiling. | Fix the experiment YAML if it failed to parse, or lower limits.max_time_seconds so the run fits under the ceiling. |
A run AX reclaims on its own reports never_started when nothing in the sandbox
ever reported, and starved when it started and then stopped reporting.
Where a time budget is what ran out, AX explains which limit was reached instead of leaving you with the raw exit reason.
In the web app, the Run Details view leads with the limit that stopped the run, for example "Timed out at the Basic plan's 10-minute limit", followed by what to do about it. Org admins also get an upgrade action. The exit reason, phase, and error message stay one click away under Details, and the Run time tile carries the budget that applied as a qualifier, for example "limit 10m".
In the CLI, the run's detail cell reads timed out at 600s · Basic plan limit,
the same explanation prints under the failures it accounts for, and a Next:
line gives the one action worth taking:
Next: upgrade at <URL>, then set limits.max_time_seconds up to 1200 and run `ax run rerun <RUN_REQUEST_ID>`When the run's own budget is what ran out, there are two ways forward:
- Raise
limits.max_time_secondsand run again, when the experiment asked for less time than your plan allows. - Upgrade the plan for a longer cap. See Cloud run time limits.
Runs dispatched before this behavior shipped recorded no time budget, so they show a generic "Agent timed out" with no plan detail. Run the experiment again to get the explained version.
Create new runs for errored variants
Find the affected variant ids in the Run Details view or with ax run view, then pass them to ax experiment run --variant. Each selected variant gets a new run id; the errored runs and their evidence remain unchanged.
Run `ax learn debug-run`, then help me create new runs for the affected variants after we identify the cause.Captured run data
AX captures this data so you can understand what the agent did, diagnose failures, compare variants, and verify test outcomes.
| Evidence | What is captured |
|---|---|
| Agent activity | Messages, reasoning when available, tool calls, command output, and agent logs |
| Measurements | Cost, tokens, turns, wall-clock time, and tool-call counts |
| Setup and state | Resolved configuration, setup output and checks, staged-file details, and best-effort workspace changes |
| Tests | Outcome, exit code, duration, stdout, and stderr for each test |
AX Cloud derives queryable platform tables from uploaded artifacts and live session data. See Data and insights for how to query them.
Inside a platform test script, use ax-run-query "$AX_RUN_ID" instead. The test sandbox receives that command and read-only access scoped to the current run only after the evaluated agent exits. See the worked transcript check in Tests.