ax experiment

Platform-backed commands also accept --org <org-id> (target another org for one command), and all but experiment push and experiment run accept --json (machine-readable output); neither flag is repeated in the tables below. Arguments marked required must be supplied; everything else is optional, and flags combine freely unless a row says otherwise.

experiment create

Scaffold a new experiment YAML.

ax experiment create <NAME>

Examples:

# Blank scaffold
ax experiment create my-experiment

# Starter experiment: can agents install your CLI?
ax experiment create my-first-experiment --template cli-install \
  --cli mycli_name \
  --target-description 'a CLI for managing OLAP databases' \
  --install-docs https://example.com/install \
  --install-command 'npm install mycli_name' \
  --smoke-command 'mycli_name --version' \
  --smoke-output-contains 'mycli_name version'

experiment validate

Check an experiment YAML against the schema; exits non-zero on schema errors.

ax experiment validate <EXPERIMENT.yaml>

Schema errors are fatal. Advisory lints print warning[<rule>] on stderr and still exit zero. warning[grading-details-in-prompt] means a prompt names a test or echoes its description; keep scoring in tests and describe a realistic job in the prompt. See Experiment design.

When signed in, validation also prints an advisory AI review of the experiment design to stderr, including checking whether the experiment is designed in a way that lets the agent know it is being evaluated. It never changes the exit code, so a suggestion is not a failure. Skip it with --no-ai or AX_NO_AI_LINT=1; pick a different review model with AX_AI_LINT_MODEL.

experiment run

Run an experiment: submit to the platform (default) or run locally in Docker with --local.

ax experiment run <EXPERIMENT.yaml | EXPERIMENT_ID>

Target a subset of variants with selectors, all repeatable or comma-separated: --variant, --prompt, --agent, --model, --product, --environment. Same flag ORs; different flags AND (--agent cursor --model X runs only variants using agent cursor and model X). Exact --variant ids are additive on top. Each selected variant is included once.

Local-only flags:

Examples:

# Platform run, polling until done
ax experiment run my-experiment.yaml

# Local run with live agent events
ax experiment run my-experiment.yaml --local --watch

# Two prompts only, three repeats each
ax experiment run my-experiment.yaml --prompt baseline,explicit --repeat 3

# A specific registered version
ax experiment run my-experiment --version <fingerprint>

On an attached platform run, if any run contains evidence that the agent is aware that it is being tested, it is classified as "test aware" and a warning prints on stderr. It does not change the exit code. Inspect the run with run view, or list them with run list --agent-test-aware. See runs where the agent noticed it was being tested.

experiment query

Query an experiment's results: flag-based modeled summaries, or SQL via the sql subcommand. Save SQL as insights with --save.

ax experiment query <EXPERIMENT_ID>
ax experiment query <EXPERIMENT_ID> sql --tables

Experimental flag-based modeled queries (metrics with filters and grouping, no SQL) also work; the surface may change:

Both modes leave out runs classified as "test aware" (the agent noticed it was being tested) by default. Only the modeled query reports the count, as a note on stderr (N test-aware runs excluded · pass --include-test-aware to include them); SQL applies the same exclusion silently. Passing --include-test-aware counts them; those runs may not mimic real-world agent behavior. For SQL scoped to one composite run id, use run query; run scope is never filtered that way.

Examples:

# What is queryable
ax experiment query my-experiment sql --tables

# Ad-hoc SQL in experiment scope
ax experiment query my-experiment sql @query.sql --format table

# Save the query as an insight
ax experiment query my-experiment sql @query.sql --save "Failing tests"

# Experimental: pass rate by variant, no SQL
ax experiment query my-experiment --metric testPassRate --group-by variant

# Experimental: cost and tokens per prompt, claude runs only
ax experiment query my-experiment --metric cost,tokens --group-by prompt \
  --filter agent=claude

# Experimental: which test fails most?
ax experiment query my-experiment --metric testPassRate --group-by test

# Experimental: what dimensions and metrics exist?
ax experiment query my-experiment --dimensions

experiment list

List experiments in the active org, most recently updated first.

ax experiment list

experiment view

Show a metadata summary of one registered experiment: ownership, timestamps, run tally, resolved axes, latest version, and resources.

ax experiment view <EXPERIMENT_ID>

Human output lists each resource as title, resource UUID, and URL after Description: (same place as the experiment page). An empty list is a dash. --json adds a resources array (id, title, url, created_at, updated_at). Resources are not in the YAML; push and pull ignore them. get is an alias of view.

experiment resource add

Attach one titled link, or several in one invocation. --title and --url are repeatable and zip in order; counts must match.

ax experiment resource add <EXPERIMENT_ID> --title <TITLE> --url <URL>

Same title or same URL twice is allowed. A non-http URL, or unpaired --title / --url, is an error.

Examples:

# One plan doc
ax experiment resource add <EXPERIMENT_ID> \
  --title "Experiment plan" --url https://docs.google.com/document/d/<DOC_ID>

# Plan and report together
ax experiment resource add <EXPERIMENT_ID> \
  --title "Experiment plan" --url https://docs.google.com/document/d/<DOC_ID> \
  --title "Mongo report" --url https://docs.google.com/presentation/d/<SLIDES_ID>

experiment resource edit

Change a resource's title, URL, or both. Copy <RESOURCE_ID> (a UUID) from ax experiment view. At least one of --title or --url is required.

ax experiment resource edit <EXPERIMENT_ID> <RESOURCE_ID> --title <TITLE>

A missing resource id is an error.

experiment resource delete

Remove a resource. rm is an alias. No confirmation prompt.

ax experiment resource delete <EXPERIMENT_ID> <RESOURCE_ID>

A missing resource id is an error, not a silent success.

experiment push

Register an experiment and its declared file dependencies on the platform without running it. Pushing unchanged YAML keeps the same version id and replaces its stored file manifest.

ax experiment push <EXPERIMENT.yaml>

experiment pull

Download a registered experiment's YAML and stored dependencies as a runnable local package (latest version by default).

ax experiment pull <EXPERIMENT_ID>

Output streams

By default the YAML goes only to the file, never to standard output. The receipt prints to standard output on a terminal and to standard error when output is redirected, so a piped default pull writes nothing to standard output and cannot overwrite the file it just created.

To capture the definition from a script or an agent, ask for it explicitly:

ax experiment pull <EXPERIMENT_ID> --stdout > experiment.yaml

experiment fork

Copy a registered experiment onto the current data pipeline as a new experiment, without creating runs.

ax experiment fork <EXPERIMENT_ID>

The fork registers the source's latest stored definition and dependencies again under the new id, with only id and name rewritten in the YAML, and claims the copy on the current results pipeline. The source experiment is only read: its runs, versions, and results are unchanged. Use it when an experiment's results are held in the retired data pipeline and you want to run the same definition again; ax experiment run <NEW_ID> picks up from there.

Examples:

# Fork with derived id and name, then run the copy
ax experiment fork my-first-experiment
ax experiment run my-first-experiment-v2

# Choose the new identity
ax experiment fork my-first-experiment --new-id my-first-experiment-2026 --name "My first experiment (2026)"

experiment versions

List stored versions of a registered experiment, newest push first.

ax experiment versions <EXPERIMENT_ID>

The VERSION column is the friendly first-created ordinal (v1, v2, ...); re-pushing an older definition can move it to the top without changing its ordinal.

The AGENT TEST AWARENESS column reads - until a run classified as "test aware" (the agent noticed it was being tested) traces back to the experiment content itself, and needs iteration · <n> runs from then on, where n counts those runs. Runs a platform artifact caused leave the column at -, since the definition is not what needs changing. With --json, each row carries the same signal as needs_iteration and test_aware_run_count.

experiment variants

Preview what a YAML resolves to before running it: how many variants the cross product yields, which incompatible agent and model pairs were pruned, and the exact variant ids to use with ax experiment run --variant.

ax experiment variants <EXPERIMENT.yaml>

experiment schema

Print the experiment schema document.

ax experiment schema [--schema-version <n>]