ax experiment

Platform-backed commands also accept --org <org-id> (target another org for one command), and all but experiment push and experiment run accept --json (machine-readable output); neither flag is repeated in the tables below. Arguments marked required must be supplied; everything else is optional, and flags combine freely unless a row says otherwise.

experiment create

Scaffold a new experiment YAML.

ax experiment create <NAME>

Examples:

# Blank scaffold
ax experiment create my-experiment

# Starter experiment: can agents install your CLI?
ax experiment create my-first-experiment --template cli-install \
  --cli mycli_name \
  --target-description 'a CLI for managing OLAP databases' \
  --install-docs https://example.com/install \
  --install-command 'npm install mycli_name' \
  --smoke-command 'mycli_name --version' \
  --smoke-output-contains 'mycli_name version'

experiment validate

Check an experiment YAML against the schema, and against your plan's cloud-run time limit and the agent-version registry when you are signed in; exits non-zero on any failure.

ax experiment validate <EXPERIMENT.yaml>

Schema errors are fatal. Advisory lints print warning[<rule>] on stderr and still exit zero. warning[grading-details-in-prompt] means a prompt names a test or echoes its description; keep scoring in tests and describe a realistic job in the prompt. See Experiment design.

Validation also parses and lints every script as bash: each setup and teardown, every setup_checks[].script, and every tests[].script, including those inside extensions. A syntax error, such as a missing fi or an unterminated quote, is fatal and names the script and the line within it (environment `py` setup line 3: expected 'fi'). A here-document that never closes is fatal the same way. Bash accepts such a script but reads everything after the heredoc into it, so the commands that follow never run. This happens when the closing marker is missing, misspelled, quoted, indented in a plain << heredoc, or followed by trailing spaces, and no later line matches the marker exactly. When a later line does match, bash closes the heredoc there, and a marker line with trailing spaces is only an advisory warning[S030]. Other findings are advisory and print as warning[<code>] with the same location, for example warning[C004] for a cd whose failure is not checked. To silence an advisory finding, add a # shellcheck disable=SC<NNNN> comment on the line above it in the script. Fatal errors cannot be silenced.

When you are signed in and an organization resolves, validation also checks limits.max_time_seconds against that organization's cloud-run time cap, so an over-cap value fails here instead of at submit. Validation exits non-zero and prints the cap your plan allows, the value to lower limits.max_time_seconds to, and where to upgrade. An organization whose plan has no cap always passes. Signed out, offline, or with the platform unreachable, the schema checks are unchanged and validation prints one advisory line on stderr saying plan limits were not checked: the exit code does not change, so a billing-side outage never fails a validate. AX_NO_PLAN_LIMIT_CHECK=1 skips the check entirely and prints nothing. See Cloud run time limits.

When you are signed in, validation also applies the same agent-version compatibility check as submit. An unregistered exact agent version, or an experiment with no agent and model pairing left to run, fails validation with the platform's message; when no pairing is left, the message lists the agent versions and models that cannot run and a command per model to find compatible agent versions. If the plan-limit check also fails, both errors are printed. Submit runs only supported pairings, verified and on the agent's model menu, the same pairings ax model list and ax agent list show. It skips every other pairing rather than failing, so validation lists the skipped pairings on stderr, grouped by agent and exact version, with an ax agent test command for each pairing that has not been tested yet, a hint naming the version that supports each skipped model when another version does, and a command per model to find compatible agent versions, and still exits zero. This check is deterministic, runs with --no-ai, and never starts a run. Signed out, offline, or with the platform unreachable, validation prints one advisory line saying agent compatibility was not checked and the exit code does not change. If the platform rejects your API key or your organization, validation still passes and prints a hint instead: sign in again with ax auth login, or pass --org with an organization from ax auth orgs. AX_NO_AGENT_VERSION_CHECK=1 skips the check entirely. The agent-model-check- experiment-id prefix is reserved for ax agent test, so validation and submit reject an experiment whose id starts with it.

An agent listed without a version runs one exact version for all its models. If the experiment has a saved pin for that agent, from an earlier submission, the pin is used and never moved; its pairings are checked like any other, and the ones it no longer supports are skipped. Otherwise submit selects the version that supports the most of the agent's requested models, preferring the newest on a tie, and skips that agent's other models. To run a skipped model, pin a version that supports it in a separate agent entry or experiment. Validation reports the same selection.

When signed in, validation also prints an advisory AI review of the experiment design to stderr, including checking whether the experiment is designed in a way that lets the agent know it is being evaluated. It never changes the exit code, so a suggestion is not a failure. Skip it with --no-ai or AX_NO_AI_LINT=1; pick a different review model with AX_AI_LINT_MODEL.

--json prints one object on stdout carrying the path, the lint warnings, the plan-limit result, the agent compatibility result, and any errors. It suppresses the advisory stderr lines and the AI review, since the object already carries what they report. agent_compatibility has checked, a note explaining why it was not checked, agents, one {agent, version} entry per exact agent version submit would select, and skipped, one {agent, version, model, test_command} entry per pairing submit will skip; test_command is the ax agent test command for an untested pairing and null otherwise.

A file that fails validation still prints the object, with valid: false, and exits non-zero; the human-readable error also goes to stderr. Its warnings is empty and plan_limits and agent_compatibility report checked: false, since none of those steps run on an invalid file. Each problem is one errors entry with a kind and a message, plus a location and a line when they apply:

experiment run

Run an experiment: submit it to the platform, which runs every variant in a 514-managed sandbox, and poll until it finishes.

ax experiment run <EXPERIMENT.yaml | EXPERIMENT_ID>

Target a subset of variants with selectors, all repeatable or comma-separated: --variant, --prompt, --agent, --model, --product, --environment. Same flag ORs; different flags AND (--agent cursor --model X runs only variants using agent cursor and model X). Exact --variant ids are additive on top. Each selected variant is included once.

Examples:

# Platform run, polling until done
ax experiment run my-experiment.yaml

# Submit and return immediately; follow it with ax run watch
ax experiment run my-experiment.yaml --detach

# Two prompts only, three repeats each
ax experiment run my-experiment.yaml --prompt baseline,explicit --repeat 3

# A specific registered version
ax experiment run my-experiment --version <fingerprint>

# Check every distinct setup, without running an agent
ax experiment run my-experiment.yaml --setup-only

Setup-only runs

--setup-only checks that an experiment's environments come up before you spend on agents. Each sandbox stages files, runs setup and setup skills, runs setup_checks, then runs teardown. The agent and the tests never run.

Setup depends only on the environment, product, and extension coordinates, so the run groups variants whose setup is identical (same setup scripts, staged files, skills, MCP servers, setup_checks, environment variables, and teardown) and runs one sandbox per group, using the first variant of each group. Variants that stage setup skills also group by whether the agent is Claude, which installs skills through its own discovery path. --all-variants turns the grouping off.

The command exits non-zero when any group fails. A setup-only run creates one run per distinct setup, and that run records the selected variants it covers. Each run incurs the per-run sandbox charge but no model spend. The MCP experiment_run tool exposes both flags as setup_only and all_variants.

To read a run's setup output afterwards, pass its composite run id to ax run view --setup-log, or open the run page, where the Setup log card shows the log with its setup-check results. The card starts expanded for setup-only runs and for runs that ended setup_failed or setup_check_failed.

On an attached platform run, if any run contains evidence that the agent is aware that it is being tested, it is classified as "test aware" and a warning prints on stderr. It does not change the exit code. Inspect the run with run view, or list them with run list --agent-test-aware. See runs where the agent noticed it was being tested.

experiment query

Query an experiment's results: flag-based modeled summaries, or SQL via the sql subcommand. Save SQL as insights with --save.

ax experiment query <EXPERIMENT_ID>
ax experiment query <EXPERIMENT_ID> --table

Experimental flag-based modeled queries (metrics with filters and grouping, no SQL) also work; the surface may change:

Both modes leave out runs classified as "test aware" (the agent noticed it was being tested) by default. Only the modeled query reports the count, as a note on stderr (N test-aware runs excluded · pass --include-test-aware to include them); SQL applies the same exclusion silently. Passing --include-test-aware counts them; those runs may not mimic real-world agent behavior. For SQL scoped to one composite run id, use run query; run scope is never filtered that way.

Examples:

# What is queryable, and one table's columns
ax experiment query my-experiment --table
ax experiment query my-experiment --table normalized_events

# Ad-hoc SQL in experiment scope
ax experiment query my-experiment sql @query.sql --format table

# Save the query as an insight
ax experiment query my-experiment sql @query.sql --save "Failing tests"

# Experimental: pass rate by variant, no SQL
ax experiment query my-experiment --metric testPassRate --group-by variant

# Experimental: cost and tokens per prompt, claude runs only
ax experiment query my-experiment --metric cost,tokens --group-by prompt \
  --filter agent=claude

# Experimental: which test fails most?
ax experiment query my-experiment --metric testPassRate --group-by test

# Experimental: what dimensions and metrics exist?
ax experiment query my-experiment --dimensions

experiment list

List experiments in the active org, most recently updated first.

ax experiment list

When the returned count equals --limit, the command warns on stderr that more matching experiments may exist. Increase --limit to fetch more, or narrow the filters at the maximum of 500.

--json continues to return a bare array on stdout. Warnings appear only on stderr, and successful queries retain exit code 0.

experiment view

Show a metadata summary of one registered experiment: ownership, timestamps, run tally, resolved axes, latest version, and resources.

ax experiment view <EXPERIMENT_ID>

Human output lists each resource as title, resource UUID, and URL after Description: (same place as the experiment page). An empty list is a dash. --json adds a resources array (id, title, url, created_at, updated_at). Resources are not in the YAML; push and pull ignore them. get is an alias of view.

experiment resource add

Attach one titled link, or several in one invocation. --title and --url are repeatable and zip in order; counts must match.

ax experiment resource add <EXPERIMENT_ID> --title <TITLE> --url <URL>

Same title or same URL twice is allowed. A non-http URL, or unpaired --title / --url, is an error.

Examples:

# One plan doc
ax experiment resource add <EXPERIMENT_ID> \
  --title "Experiment plan" --url https://docs.google.com/document/d/<DOC_ID>

# Plan and report together
ax experiment resource add <EXPERIMENT_ID> \
  --title "Experiment plan" --url https://docs.google.com/document/d/<DOC_ID> \
  --title "Mongo report" --url https://docs.google.com/presentation/d/<SLIDES_ID>

experiment resource edit

Change a resource's title, URL, or both. Copy <RESOURCE_ID> (a UUID) from ax experiment view. At least one of --title or --url is required.

ax experiment resource edit <EXPERIMENT_ID> <RESOURCE_ID> --title <TITLE>

A missing resource id is an error.

experiment resource delete

Remove a resource. rm is an alias. No confirmation prompt.

ax experiment resource delete <EXPERIMENT_ID> <RESOURCE_ID>

A missing resource id is an error, not a silent success.

experiment push

Register an experiment and its declared file dependencies on the platform without running it. Pushing unchanged YAML keeps the same version id and replaces its stored file manifest.

ax experiment push <EXPERIMENT.yaml>

experiment pull

Download a registered experiment's YAML and stored dependencies as a runnable local package (latest version by default).

ax experiment pull <EXPERIMENT_ID>

Output streams

By default the YAML goes only to the file, never to standard output. The receipt prints to standard output on a terminal and to standard error when output is redirected, so a piped default pull writes nothing to standard output and cannot overwrite the file it just created.

To capture the definition from a script or an agent, ask for it explicitly:

ax experiment pull <EXPERIMENT_ID> --stdout > experiment.yaml

experiment fork

Copy a registered experiment onto the current data pipeline as a new experiment, without creating runs.

ax experiment fork <EXPERIMENT_ID>

The fork registers the source's latest stored definition and dependencies again under the new id, with only id and name rewritten in the YAML, and claims the copy on the current results pipeline. The source experiment is only read: its runs, versions, and results are unchanged. Use it when an experiment's results are held in the retired data pipeline and you want to run the same definition again; ax experiment run <NEW_ID> picks up from there.

Examples:

# Fork with derived id and name, then run the copy
ax experiment fork my-first-experiment
ax experiment run my-first-experiment-v2

# Choose the new identity
ax experiment fork my-first-experiment --new-id my-first-experiment-2026 --name "My first experiment (2026)"

experiment versions

List stored versions of a registered experiment, newest push first.

ax experiment versions <EXPERIMENT_ID>

The VERSION column is the friendly first-created ordinal (v1, v2, ...); re-pushing an older definition can move it to the top without changing its ordinal.

The AGENT TEST AWARENESS column reads - until a run classified as "test aware" (the agent noticed it was being tested) traces back to the experiment content itself, and needs iteration · <n> runs from then on, where n counts those runs. Runs a platform artifact caused leave the column at -, since the definition is not what needs changing. With --json, each row carries the same signal as needs_iteration and test_aware_run_count.

experiment variants

Preview what a YAML resolves to before running it: how many variants the cross product yields, which incompatible agent and model pairs were pruned, and the exact variant ids to use with ax experiment run --variant.

ax experiment variants <EXPERIMENT.yaml>

experiment schema

Print the experiment schema document.

ax experiment schema [--schema-version <n>]