ax experiment

Platform-backed commands also accept --org <org-id> (target another org for one command) and --json (machine-readable output); they are not repeated in the tables below. Arguments marked required must be supplied; everything else is optional, and flags combine freely unless a row says otherwise.

experiment create

Scaffold a new experiment YAML.

ax experiment create <NAME>

Examples:

# Blank scaffold
ax experiment create my-experiment

# Starter experiment: can agents install your CLI?
ax experiment create my-first-experiment --template cli-install \
  --cli mycli_name \
  --target-description 'a CLI for managing OLAP databases' \
  --install-docs https://example.com/install \
  --install-command 'npm install mycli_name' \
  --smoke-command 'mycli_name --version' \
  --smoke-output-contains 'mycli_name version'

experiment validate

Check an experiment YAML against the schema; exits non-zero on schema errors.

ax experiment validate <EXPERIMENT.yaml>

When signed in, validation also prints an advisory AI review of the experiment design to stderr. It never changes the exit code, so a suggestion is not a failure. Skip it with --no-ai or AX_NO_AI_LINT=1; pick a different review model with AX_AI_LINT_MODEL.

experiment run

Run an experiment: submit to the platform (default) or run locally in Docker with --local.

ax experiment run <EXPERIMENT.yaml | EXPERIMENT_ID>

Target a subset of variants with selectors, all repeatable or comma-separated: --variant, --prompt, --agent, --model, --product, --environment. Same flag ORs; different flags AND (--agent cursor --model X runs only variants using agent cursor and model X). Exact --variant ids are additive on top. Each selected variant is included once.

Local-only flags:

Examples:

# Platform run, polling until done
ax experiment run my-experiment.yaml

# Local run with live agent events
ax experiment run my-experiment.yaml --local --watch

# Two prompts only, three repeats each
ax experiment run my-experiment.yaml --prompt baseline,explicit --repeat 3

# A specific registered version
ax experiment run my-experiment --version <fingerprint>

experiment query

Query an experiment's results with SQL scoped to the experiment, and save queries as insights.

ax experiment query <EXPERIMENT_ID>

Experimental flag-based modeled queries (metrics with filters and grouping, no SQL) also work; the surface may change:

For SQL scoped to one composite run id, use run query.

Examples:

# Ad-hoc SQL in experiment scope
ax experiment query my-experiment --sql @query.sql

# Save the query as an insight
ax experiment query my-experiment --sql @query.sql --save "Failing tests"

# Experimental: pass rate by variant, no SQL
ax experiment query my-experiment --metric testPassRate --group-by variant

# Experimental: cost and tokens per prompt, claude runs only
ax experiment query my-experiment --metric cost,tokens --group-by prompt \
  --filter agent=claude

# Experimental: which test fails most?
ax experiment query my-experiment --metric testPassRate --group-by test

# Experimental: what dimensions and metrics exist?
ax experiment query my-experiment --dimensions

experiment list

List experiments in the active org, most recently updated first.

ax experiment list

experiment view

Show a metadata summary of one registered experiment: ownership, timestamps, run tally, resolved axes, and latest version.

ax experiment view <EXPERIMENT_ID>

experiment push

Register an experiment on the platform without running it. Pushing unchanged YAML is idempotent; each push stores a content-addressed version.

ax experiment push <EXPERIMENT.yaml>

experiment pull

Download a registered experiment's YAML (latest version by default).

ax experiment pull <EXPERIMENT_ID>

experiment versions

List stored versions of a registered experiment, newest push first.

ax experiment versions <EXPERIMENT_ID>

The VERSION column is the friendly first-created ordinal (v1, v2, ...); re-pushing an older definition can move it to the top without changing its ordinal.

experiment variants

Preview what a YAML resolves to before running it: how many variants the cross product yields, which incompatible agent and model pairs were pruned, and the exact variant ids to use with ax experiment run --variant.

ax experiment variants <EXPERIMENT.yaml>

experiment schema

Print the experiment schema document.

ax experiment schema [--schema-version <n>]