Runs | Fiveonefour's Documentation

Each run is a single isolated execution of an experiment variant in a dedicated sandbox.

The variant determines the run’s agent, model, prompt, product, and environment configuration. When the run starts, AX Cloud spins up a fresh sandbox, applies the product and environment setup, and starts the agent with the selected model and prompt.

As the agent works, AX Cloud captures available agent activity and workspace changes for evaluation and analysis (see Captured run data).

When the agent finishes its work, AX Cloud snapshots the sandbox and runs the experiment's tests in a separate sandbox created from that snapshot. This is to ensure the agent cannot cheat by previewing the tests before they are run.

Starting a run

Right now you can only start runs with the CLI command ax experiment run, which accepts an experiment YAML path or a registered experiment-id.

Prompt

Run `ax learn run-experiment`, then help me run my experiment.

Without selectors, ax experiment run creates one run for every computed variant in the specified experiment.

Unless you pass --local, AX executes the selected runs in Fiveonefour-managed sandboxes and stores the resulting run data in your organization. See Local runs to use Docker sandboxes on your machine.

Run specific variants

Selectors can be used to target specific variants to run without editing the experiment. This is useful if you add or change a specific variable (e.g. you add or edit a prompt), after already having run your experiment.

ax experiment run experiment.yaml --model gpt-4o-mini

Run multiple times

You can use --repeat to create several independent runs for each variant.

ax experiment run experiment.yaml --repeat 3

Run from your GitHub CI

From a GitHub pull request, comment !ax run or use the run checkbox in the Fiveonefour comment. See GitHub CI for more details.

Monitoring runs

Use the experiment's Runs List page in the web app or ax run list to check the status of individual runs and groups of runs:

Prompt

Use the `ax run list` help me monitor my active, queued, and recently runs. Summarize them by status and highlight runs that did not complete normally.

Run status

Status	Meaning
`waiting-for-ci`	A pull request run is waiting for its CI build artifacts.
`queued`	The run is waiting for execution capacity.
`running`	Setup, the agent, or tests are in progress.
`completed`	Execution and testing finished.
`errored`	The run did not finish normally.
`canceled`	The run was intentionally stopped.

completed does not mean that every test passed. See Tests for pass/fail outcomes.

Troubleshooting errored runs

You can locate errored runs in the Runs List view in the web app or with ax run list in the CLI, then open the Run Details view or use ax run view to troubleshoot individual runs.

Prompt

Run `ax learn debug-run`, then help me troubleshoot my errored runs. Identify the likely cause(s) and whether the experiment definition needs to change before running again.

When available, both the web app and CLI show the last lifecycle phase AX observed for the run, a machine-readable exit_reason, and an error_message. The phase is absent when AX Cloud has not reported the first sandbox phase, such as when a required CI artifact never arrives. A dispatch failure can show no phase or sandbox, depending on how far it progressed.

Phase	What is happening	Error values you may see	What to inspect
No phase	AX Cloud has not begun preparing the sandbox.	`ci_artifacts_missing`, `dispatch_failed`	The error message and CI configuration
`sandbox`	AX Cloud is preparing the sandbox and staging run inputs.	`staging_failed`, `dispatch_failed`	The error message and staged-file details
`setup`	The product and environment setup scripts are running.	`setup_failed`	Setup output
`checks`	Setup checks are verifying the sandbox before the agent starts. These are separate from tests.	`setup_check_failed`	Setup-check output
`agent`	The agent is working on the prompt.	`driver_error`, `model_unsupported_by_harness`, `timeout`	The error message, agent logs, last agent activity, and the `hit_timeout` measurement
`tests`	The experiment's tests are evaluating the agent's work.	`artifacts_upload_failed` may leave `tests` as the last reported phase	Test output, the publication error, and any available captured evidence
Any phase	AX Cloud is waiting for the run to report progress or finish.	`sandbox_timeout`, `orphaned`	The error message and last recorded activity

An agent timeout can still produce test output from partial work.

An errored status is terminal. The run cannot resume or be repaired. After identifying the cause, create new runs for the affected variants.

If the experiment definition caused the error, fix the YAML first. If the error was transient, use the same definition again.

Create new runs for errored variants

Find the affected variant ids in the Run Details view or with ax run view, then pass them to ax experiment run --variant. Each selected variant gets a new run id; the errored runs and their evidence remain unchanged.

Prompt

Run `ax learn debug-run`, then help me create new runs for the affected variants after we identify the cause.

Captured run data

AX captures this data so you can understand what the agent did, diagnose failures, compare variants, and verify test outcomes.

Evidence	What is captured
Agent activity	Messages, reasoning when available, tool calls, command output, and agent logs
Measurements	Cost, tokens, turns, wall-clock time, and tool-call counts
Setup and state	Resolved configuration, setup output and checks, staged-file details, and best-effort workspace changes
Tests	Outcome, exit code, duration, stdout, and stderr for each test

AX Cloud derives queryable platform tables from uploaded artifacts and live session data. See Data and insights for how to query them.

Inside a platform test script, use ax-run-query "$AX_RUN_ID" instead. The test sandbox receives that command and read-only access scoped to the current run only after the evaluated agent exits. See the worked transcript check in Tests.

Local runs

Pass --local to run the same experiment in Docker sandboxes on your machine:

ax experiment run experiment.yaml --local

Detail	Local behavior
Sandboxes	Docker on your machine
Parallelism	Bounded by your machine and `--jobs`
Authentication	Saved login or `AX_API_KEY` for managed model access; `--local-model-keys` uses provider keys instead
Run data	`./.axp/runs/<RUN_REQUEST_ID>/`, with best-effort upload when signed in

Local data remains under ./.axp/runs/. When you are signed in, AX also attempts to upload it. If that upload fails, retry with ax run upload, or inspect the local data with ax run query '<RUN_ID>' sql --tables --local.

Local artifact layout

A local run keeps its durable execution record under ./.axp/runs/<RUN_REQUEST_ID>/. A typical record includes:

.axp/runs/<RUN_REQUEST_ID>/
  group.json                         # request manifest; ended_at marks completion
  index.jsonl                        # one line per variant as it finishes
  experiment.yaml                    # verbatim experiment snapshot
  variants/
    <VARIANT_ID>/                    # adds /<REPEAT_INDEX>/ when repeated
      layout.json                    # artifact layout version
      run.json                       # compact result for this execution
      resolved-variant.yaml          # resolved config, written before the agent
      staging.json                   # staged-file details, when present
      setup.log                      # setup output, on success or failure
      session-data.json              # native-capture lifecycle state, when present
      native/                        # native agent evidence, when present
      agent-events.jsonl             # Agent activity events, when present
      harness-events.jsonl           # run lifecycle events
      agent.log                      # agent process output, when present
      setup-checks/
      tests/
      test-executions/               # retest history, when present
      fs-diff/                       # workspace changes, best effort

resolved-variant.yaml can contain declared secret names, but never resolved secret values. setup.log is written whether setup succeeds or fails. Cost in run.json is fixed-point millionths in cost_usd_micros; divide by 1,000,000 for USD. Capture-specific files are conditional, so a missing native/ or agent-events.jsonl does not by itself mean the run lost evidence.

Interrupted local runs

A finalized group.json has an ended_at value. If it is null, the request stopped before finalization or before every variant summary was written.

The files written before interruption remain useful: the experiment snapshot, resolved variants, setup output, captured agent evidence, completed test results, and any filesystem diff that finished.

Search documentation