Runs

A run is one isolated execution of an experiment variant and the raw data it produces.

Every run is independent and immutable. AXP creates a new sandbox for each run and applies the variant's environment and product setup scripts. No memory or state is shared between runs, so one run cannot depend on files, environment variables, or side effects from another run.

Once the sandbox is ready, AXP gives the agent the variant's prompt and records what happens.

You can review the resulting run data to understand what the agent did, what changed in the workspace, which tests passed, how long it took, and what it cost. AXP uses run data to generate Results when you want to compare across variants, agents, prompts, products, or environments.

Triggering a run

Runs are created through the AXP CLI from an experiment YAML file. By default, AXP resolves the file and creates one run per (variant × repeat) for every variant in the experiment.

It's possible to override this default behavior with targeted selectors that run only a subset of variants — useful when you've changed one part of the configuration and don't want to rerun the whole matrix:

• --variant <id,...> runs exact variant ids.
• --prompt / --agent / --model / --product / --environment run every variant that uses the given value on that axis — so after fixing prompt p0, ax experiment run experiment.yaml --prompt p0 reruns just the variants that use it.
• ax run rerun <run-request-id> reruns failed or canceled variants behind prior platform requests. Find candidates with ax run list --status failed, fix the YAML if needed, then rerun the request. The runs must belong to the same experiment.

Selectors combine as a union and are deduped, so you can ask for several slices at once. With no selector, the full experiment runs.

For the exact flags that control which variants and repeats run, see the experiment run command reference. Local execution uses the same command with --local.

You can also trigger a run from a GitHub pull request by commenting !ax run (or checking the run box in the AXP comment), which runs the repo's linked experiment against the PR's code and reports back in the PR. See GitHub CI.

Remote vs. local execution

You can execute the same experiment remotely through the AXP platform or locally via containers on your own machine. The command you use decides where the sandboxes live and where the run data is stored.

Remote runs (via ax experiment run)

Remote runs execute variants in AXP-managed sandboxes. AXP can run those sandboxes in parallel, which is useful for experiments with many variants or many repeats. Remote runs require you to be signed in (via ax auth login) and you do not need to provide your own model provider API key for supported agents.

ax experiment run polls the request to completion and prints a run request id. To check on a run later — including one you submitted with --detach or detached from with Ctrl-C — pass that id to ax run view <id>, use ax run watch <id> to stream until it finishes, or list recent runs with ax run list.

See ax experiment run for the full command reference.

Local runs (via ax experiment run --local)

Local runs execute variants in Docker containers on your own machine. You can run those containers in parallel, but concurrency is limited by your machine's CPU, memory, and Docker capacity. Local runs do not require you to be signed in. If you are signed in, you can pass --managed-model-access; otherwise, provide your own model provider API key.

See ax experiment run --local for the full command reference.

Inspecting run data

Each run produces the data you use to answer two questions: what happened in this sandbox, and how did this run compare to the others?

• final variant configuration
• setup, agent, command, and test logs
• agent messages and tool calls
• workspace file changes
• status, timing, token usage, and cost

From the terminal, ax run list enumerates your runs (local and remote) one per row, and ax run view <run-id> drills into a single run's status, dimensions, cost, and test results.

Run data from remote runs is stored on the AXP platform. You and other members of your organization can inspect it in the AXP dashboard, query it with ax results query, or download it for local analysis with ax run download.

For local runs, AXP attempts a best-effort upload when you're signed in with ax auth login. If that upload is skipped or fails, the run data stays on your machine under ./.axp/runs/. You can upload it later with ax run upload.

See the Results documentation for more information on how to use run data to generate insights and comparisons.