Experiments

An experiment is the YAML file that says what you want to learn about agents using your product. You declare axes (agents, models, prompts, products, environments), AX multiplies them into variants, and each variant runs in its own isolated sandbox. Tests score the result.

You do not have to fill every axis on day one. Start from a template (ax experiment create --template cli-install), change one thing at a time, and read Experiment Design for how to pick the question, the variants, and the success criteria. For exact field names and validation rules, see the experiment YAML reference.

What you declare

Give any axis a list of values to see how changing that one thing moves agent performance. Advanced cases that go beyond a full cross product live under extensions.

Environments and products

Environments and products are how you seed the world the agent wakes up in. Both take a setup script, and setup can carry its own files, environment variables, and MCP servers.

That makes them the lever for progressive disclosure. Common patterns:

• Tool availability. One environment with your skill, CLI, or MCP server installed; one without. Measure whether agents find and use it.
• Starting world. A blank workspace vs a framework scaffold vs a working prototype. Useful for selection questions such as "what does the agent reach for when taking this to production?"
• Fixtures. Seed data, config, or a running service in setup so the task is realistic without hand-holding in the prompt.

Keep large setup scripts out of the YAML. Stage them with files and call them from setup:

environments:
  - name: with-skill
    setup:
      name: install-skill
      script: bash /workspace/scripts/install-skill.sh
      files:
        - source: ./scripts/install-skill.sh
          dest: scripts/install-skill.sh

Product setup runs before environment setup when both contribute.

Suggested: use description as the research brief

The top-level description is free text, and the schema does not care what you put there. We suggest using it to record why the experiment exists: the research question, what counts as success, and which dimensions you plan to compare. It travels with the experiment, shows up alongside results, and gives an analysis agent something to check the first runs against.

description: |
  Research question: Do agents pick our database when taking a prototype
  to production, and does having docs in the environment change that?
  Measure: pass rate of product-selected (named in the plan conclusion),
  sliced by prompt sophistication and starting framework.

The YAML stays the source of truth for setup, prompts, and axes; description carries the intent behind them. See Experiment Design for how to write questions and bets that survive contact with results.

Variants

A variant is one fully resolved combination of agent, model, prompt, product, and environment: the exact configuration a single run executes.

You do not write variants by hand. They are the cross product of the axes you declared. Two models and two environments give you four variants:

agents: [claude]
models: [claude-opus-4-8, claude-sonnet-4-6]
prompts:
  - id: take-to-prod
    prompt: "Take this prototype to production."
environments:
  - name: blank
    setup: "true"
  - name: next-scaffold
    setup: "npx --yes create-next-app@latest app --yes"

1 agent × 2 models × 1 prompt × 2 environments = 4 variants

claude::anthropic-claude-opus-4-8::take-to-prod::blank
claude::anthropic-claude-opus-4-8::take-to-prod::next-scaffold
claude::anthropic-claude-sonnet-4-6::take-to-prod::blank
claude::anthropic-claude-sonnet-4-6::take-to-prod::next-scaffold

Every value you add multiplies the matrix. A second prompt here makes eight variants. Adding codex to agents and gpt-5.3-codex to models makes twelve, not sixteen, because AX prunes the agent and model pairs that cannot run together (codex with a Claude model) instead of failing. Variant ids are built from those coordinates, so you can select one with --variant. See variant identity.

Preview what your file resolves to before spending:

ax experiment variants experiment.yaml

Each variant runs in its own sandbox with its own /workspace, in parallel by default. Nothing crosses the boundary: a file written by one variant's setup is invisible to every other variant. If your setup touches a shared external system (a cloud project, a shared database), reset it in setup so parallel variants stay independent.

Tests

Tests are bash checks that run after the agent finishes, in a separate sandbox, and turn each run into pass/fail. Every experiment needs at least one. See Tests for how they are defined, what they can observe, and how to re-score old runs.

Limits

limits prevents a single run from looping or overspending. Wall-clock and reported-cost limits abort the agent and mark the run timeout or cost_cap; turn limits are passed to the agent where supported. Set them before you scale a matrix.

Versioning

Change something meaningful (a prompt, model, or test) and AX cuts a new version: v1, v2, v3. Comment edits do not count, and re-running an unchanged experiment reuses the current version.

Runs are versioned per variant, so a new version only needs fresh runs for the variants you actually changed; the rest carry their existing runs forward. ax experiment push prints which ones carry over, and ax experiment versions <id> lists the versions.

Every run remembers the definition it ran against, so later edits never rewrite history. Compare like with like by scoping queries with --experiment-version.

Querying an experiment

After runs finish, summarize from the terminal with ax experiment query:

ax experiment query <EXPERIMENT_ID> --dimensions
ax experiment query <EXPERIMENT_ID> --metric testPassRate --group-by variant
ax experiment query <EXPERIMENT_ID> --sql @query.sql

Add --dry-run to inspect compiled SQL without executing it. To keep a query, pass --save "<heading>"; the CLI stores it as an org-scoped insight you can reopen with ax insight view.