Runs

A run is one isolated execution of an experiment variant. AX creates a fresh sandbox, applies the variant's product and environment setup, hands the agent the prompt, then scores the result with tests.

Runs are independent and immutable. Nothing is shared between them: one run cannot see files, environment variables, or side effects from another.

• Run request: what one ax experiment run submission creates, across the whole matrix.
• Run: one variant executing once. --repeat <N> gives each variant N runs.
• Run data: everything the run left behind, which you query with ax run query or compare across runs with ax experiment query.

Triggering a run

Runs come from an experiment YAML file. By default AX creates one run per (variant × repeat) for every variant in the experiment.

After changing one part of the file, selectors run a subset instead: exact variant ids, or every variant that uses a given value on an axis. So ax experiment run experiment.yaml --prompt p0 reruns only the variants using prompt p0. Selectors combine as a union and are deduped, so you can ask for several slices at once. See experiment run for the flags.

To retry what already failed, ax run rerun <run-request-id> reruns the failed or canceled variants of a prior platform request. Find candidates with ax run list --status failed, fix the YAML, then rerun.

You can also trigger a run from a GitHub pull request by commenting !ax run (or checking the run box in the AX comment). See GitHub CI.

Remote and local execution

The same experiment runs either way; the command decides where sandboxes live and where data is stored.

Remote runs print a run request id. Use it later with ax run view <id>, stream with ax run watch <id>, or find it again with ax run list. That is also how you pick up a run you submitted with --detach or detached from with Ctrl-C.

Run status

For a run's progress, ax run list reports queued, running, completed, failed, or canceled.

Each variant's run.json records how it went:

A degraded model connection can also look like a bad run. agent.model_transport_errors counts transport failures (websocket connect errors, mid-turn stream disconnects) logged during the run, so a non-zero count means a fail may reflect infrastructure rather than agent quality. Local runs print a warning per affected variant. Detection is best effort: 0 means none detected, not none happened.

Inspecting a run

Start broad and narrow down:

ax run list --status running,failed       # what ran, one row per run
ax run view <run-id>                      # status, dimensions, cost, tests
ax run query <run-id> sql --tables        # what is queryable
ax run download <run-id>                  # pull Parquet down for local SQL

ax run list groups by status, newest first, across every experiment in the org, and combines local runs from ./.axp/runs/ with remote ones. Narrow with --experiment, --agent, --product, --environment, --prompt, --status, or --experiment-version; regroup with --group-by; scope storage with --scope local|remote|both; add --json for machine-readable output.

Remote run data lives on the platform, where you and your org can browse it in the dashboard, query it, or download it. Local run data stays under ./.axp/runs/ unless the best-effort upload succeeds; push it later with ax run upload. See Data and results for summaries, saved insights, and local SQL.

Session data tables

Remote SQL for a session_data_v1 scope exposes exactly three logical tables. Metadata such as experiment, variant, agent, model, prompt, product, environment, version, and status rides every table as a filterable dimension, so there is no separate metadata table.

events is an observations table, not a deduplicated timeline, so the same activity can appear in several source records. Filter its source column (setup, acp, stdout, hooks, otel, transcript, harness, other). Its ts column is nullable because some source records carry no trustworthy time, and payload keeps the source-specific body as JSON-encoded text.

ax run query <RUN_ID> sql --tables events
ax run query <RUN_ID> sql "SELECT ts, source, payload FROM events WHERE source = 'otel' ORDER BY ts"

Rich sql --tables discovery exists only for remote session_data_v1 scopes. Legacy remote scopes and local DuckDB keep their physical tables; discover those with SHOW TABLES and DESCRIBE TABLE <name>.

Artifacts on disk

A local run writes everything under ./.axp/runs/<run-id>/, where the run id is the experiment id plus a ULID that carries the run timestamp:

.axp/runs/<run-id>/
  group.json                        # run manifest, re-stamped with ended_at on finish
  index.jsonl                       # one line per variant as it finishes
  experiment.yaml                   # verbatim copy of the YAML you invoked
  variants/
    <variant-id>/                   # <variant-id>/<repeat-idx>/ when --repeat > 1
      layout.json
      run.json                      # compact result for this variant
      resolved-variant.yaml         # exact resolved config, written before the agent
      staging.json                  # only when the experiment stages files
      agent-events.jsonl            # verbatim ACP frames
      harness-events.jsonl          # runner spans and lifecycle
      agent.log                     # agent stderr, unstructured
      setup.log                     # setup stdout + stderr, success or failure
      setup-checks/<check-name>.json
      tests/<test-name>.json
      fs-diff/000000.json           # workspace changes, best effort

Platform runs upload this same tree, so there is no separate results/ form. Only the variants you selected are written. ax run download fetches the run's Parquet tables to .axp/downloads/<run-id>/ rather than pulling this tree back down.

Four things about it are worth knowing:

• resolved-variant.yaml is written before the agent starts, so it survives errored runs. Declared secret names appear in it; resolved values never do.
• setup.log is written whether setup succeeded or failed, so a non-zero setup script leaves its output here. See Troubleshooting.
• Cost in run.json is fixed-point millionths in cost_usd_micros. Divide by 1,000,000 for USD.
• fs-diff is best effort. If the pre-agent snapshot failed, the harness warns, skips the diff, and continues.

Remote runs also capture an end-of-run sandbox snapshot for retest. That snapshot is platform-managed rather than a file in this tree, and it is garbage-collected after 14 idle days by default.

Event streams

agent-events.jsonl is every ACP frame between harness and agent, written verbatim before parsing. Cost, turns, tokens, tool calls, and the conversation all derive from it, so it is where you check whether a 0 token count means the adapter reported zero or never reported at all. harness-events.jsonl is the runner's own stream: setup, setup checks, test execs, snapshots, and lifecycle transitions.

--flag native runs have no ACP session, so agent-events.jsonl is absent (not empty) and the surfaces derived from it, including the timeline, tool-call and message tables, and $AX_TRACE_PATH, do not exist. Their verbatim record is the raw capture under native/ plus live session_data rows; run.json still carries cost, turns, and tokens.

Because the two produce different result shapes, an experiment id is bound to the capture family its first submit chose, and later submits with the other flag set are rejected. To compare a native arm against an ACP arm, give each one its own experiment id — see an experiment rejects a submit with a different --flag set.

Everything an adapter emits over ACP is captured, so thinking and subagent coverage differs by agent:

If thinking data matters for a Codex variant, use managed model access and a summary-capable model. ax experiment run --local warns at preflight when a Codex variant runs on a BYO key.

In derived query tables, thinking lands in messages (role = 'thought') and in the timeline tables (timeline_kind = 'thought', entry_kind = 'thinking'); subagent attribution lands in parent_tool_call_id, tool_name, and the child_agent_* columns.

Partial runs

A partial run did not finish cleanly: interrupted before finalization, or stopped by a harness failure before every variant summary was written. Treat a finalized group.json (one with ended_at) as the completion marker. A run whose group.json still has ended_at null, or that ax run list flags partial, did not complete.

Partial runs are still worth reading. Whatever got written stays written: group.json and experiment.yaml from run creation, resolved variant YAMLs, setup.log and setup-check results, event streams as far as the agent got, test results for tests that ran, and fs-diff if the snapshots completed.