Experiment YAML

Experiment YAML is the file format passed to ax experiment run and ax experiment run --local.

The current parser accepts schema_version: 2 and rejects unknown fields at every level. For the canonical machine-readable schema, use experiment.v2.schema.yaml.

Schema URL

Use the version-pinned schema URL for editor validation:

# yaml-language-server: $schema=https://docs.514.ax/schema/experiment.v2.schema.yaml

Published schema URLs:

ax experiment schema prints the latest supported schema to stdout.

Field index

Minimal example

# yaml-language-server: $schema=https://docs.514.ax/schema/experiment.v2.schema.yaml
schema_version: 2
id: cli-install
name: "CLI install"

agents: [claude]
models: [claude-sonnet-4-6]

prompts:
  - id: install
    prompt: |
      Install the CLI and write its version to /workspace/version.txt.

products:
  - name: cli
    type: CLI
    setup: npm pack

tests:
  - name: version-file-exists
    script: test -f /workspace/version.txt

limits:
  max_turns: 25
  max_time_seconds: 300
  max_cost_usd: 0.50

Top-level experiment

An experiment defines what you want to learn about agents using a product surface. It names the agents, prompts, products, environments, and tests AXP uses to compute and run variants.

Top-level environment_variables, secrets, and files apply to every variant. MCP servers and setup checks are setup-owned fields.

Variant axes

Variant axes are the experiment inputs AXP combines into runnable variants. This lets you define the dimensions you care about once, then compare results by agent, prompt, environment, and product.

variants = agents × models × prompts × agent_mode × environments × products

Only agents and prompts must be specified — every resolved variant carries an agent and a prompt (each can also be supplied by an extension). The rest are optional: omit models and each agent runs its provider-default model; omit agent_mode, environments, or products and that axis is left out of the cross product and of the variant coordinate.

Agents

agents defines which coding agents run the experiment. The model an agent runs comes from the models axis, not the agent.

Accepted shapes:

# String form: one agent.
agents: claude

# List of strings: multiple agents. Valid names: claude, codex, cursor.
agents:
  - claude
  - codex
  - cursor

# Object form is also accepted, but only carries the name.
agents:
  - name: claude

Agent object

Models

models defines which models run, as an independent axis cross-multiplied with agents. Listing agents: [claude] with models: [opus, sonnet] produces both variants without duplicating the agent.

Accepted shapes:

# String form: one model.
models: claude-opus-4-8

# List of strings: multiple models.
models:
  - claude-opus-4-8
  - claude-sonnet-4-6

# Object form: a model with controls.
models:
  - name: gpt-5.3-codex
    effort: high

models is optional. If omitted, each agent runs its provider-default model.

Pruning incompatible pairs

The agents × models cross product keeps only the compatible pairs. A model the catalog knows for a different agent's provider — for example claude × an openai model — is pruned (dropped), not an error, so one agents + models pair of lists expresses every valid combination:

agents: [claude, codex]
models: [claude-opus-4-8, gpt-5.3-codex]
# 2 × 2 = 4 pairs → pruned to the 2 valid ones:
#   ✓ claude + claude-opus-4-8     ✓ codex + gpt-5.3-codex
#   ✗ claude + gpt-5.3-codex       ✗ codex + claude-opus-4-8   (dropped)

Pruning is visible — dropped pairs show up as a smaller resolved matrix in ax experiment variants. If a listed agent or model is compatible with nothing else listed (every pair it appears in is pruned), ax experiment validate hard-errors, since that entry is almost certainly a typo.

Model id format

Prefer the lab-official models.dev id with no owner prefix, such as claude-opus-4-8, gpt-5.3-codex, or composer-2.5. AXP infers the runtime provider from the selected agent and stores a canonical provider/lab/model id behind the scenes.

If a bare id is ambiguous, validation asks for lab; use the model object form, for example { name: claude-opus-4-8, lab: anthropic }. Most model ids do not need this.

Older owner/provider-prefixed and gateway-style ids, such as anthropic/claude-opus-4-8 or anthropic/claude-opus-4.8, are accepted for compatibility and normalize to the same canonical id.

Model object

A model object configures one model and its controls. Optional controls are passed through when the selected agent supports them. To compare two control settings of the same model, list two models entries.

agents: [claude, codex]
models:
  - name: claude-opus-4-8
    effort: high
    context_window_size: 1M
    thinking: true
  - gpt-5

Prompts

prompts defines the tasks agents try to complete. Each resolved variant receives one final prompt.

Accepted shapes:

# String form: one prompt.
prompts: "Build the report."

# List of strings: multiple prompts.
prompts:
  - "Build the report."
  - "Build the report and explain your steps."

# Object form: list of named prompts.
prompts:
  - id: detailed
    prompt: "Build the report and explain your steps."

Prompt object

A prompt object gives a task stable metadata, so Results can filter and group by the prompt that produced each run.

Bare prompt strings get positional ids: p0, p1, and so on.

Agent mode

agent_mode is a top-level model variable. Accepts a single value or list of build / plan. It is cross-multiplied with the other axes just like agents and models to define the variants tested.

• build (default) runs the agent normally: it executes tools, can read and write, and mutates the workspace.
• plan puts the agent in its native read-only / planning mode. This plans, and mutates nothing in the sandbox. The abstract plan value resolves to that agent's planning mode (claude / cursor resolve to plan, codex to read-only). If the selected mode isn't available the variant fails loudly rather than silently running in build.

# Sweep both modes over one agent/model/prompt → 2 variants.
agents: [claude]
models: [claude-opus-4-8]
agent_mode: [build, plan]

agent_mode is optional; defaulting to build mode. When you declare the axis, the mode is emitted into the variant id. This applies to plan and build options. So a [build, plan] sweep produces …::build::my-prompt and …::plan::my-prompt. When the axis is absent nothing is emitted, so experiments that don't use agent_mode keep their original ids (for example claude::anthropic-claude-opus-4-8::my-prompt). Note that authoring agent_mode: build explicitly therefore yields a different id than omitting the axis.

Environments

environments defines the sandbox conditions around the agent, such as installed tools, fixtures, or external integrations. Use environments when you want to compare how agents perform under different surrounding conditions.

Accepted shapes:

# String form: one setup script with a generated environment name.
environments: "pip install -r requirements.txt"

# Object form: one named environment.
environments:
  name: workspace
  setup: "pip install -r requirements.txt"

# List form: multiple named environments.
environments:
  - name: workspace
    setup: "pip install -r requirements.txt"

Environment object

An environment object names one sandbox setup condition. Its setup prepares the sandbox before the agent receives the prompt.

Bare environment strings are shorthand for an environment whose setup is that string.

Products

products defines the agent-facing surface under test. A product can be a CLI, API, MCP server, SDK, docs surface, or other tool the agent uses to complete the prompt.

Accepted shapes:

# String form: one setup script with a generated product name.
products: "npm install -g my-cli"

# Object form: one named product.
products:
  name: my-cli
  type: CLI
  version: "1.2.0"
  setup: "npm install -g my-cli"

# List form: multiple named products.
products:
  - name: my-cli
    type: CLI
    version: "1.2.0"
    setup: "npm install -g my-cli"

Product object

A product object names what you want to compare. Product metadata is recorded so Results can filter and group by product and product version.

Product type

Allowed values: CLI, MCP, API, Skill, SDK, Schema, Docs, Marketing, Agents.md, Other.

Setup

setup prepares the sandbox for an environment or product. Use setup scripts to install tools, create fixtures, start services, or expose resources the agent needs.

Each variant has its own isolated /workspace. A setup script cannot rely on files or side effects created by another variant.

Accepted shapes:

# String form: one setup script.
setup: "npm install"

# List of strings: multiple setup scripts, run in order.
setup:
  - "npm install"
  - "npm test -- --help"

# Object form: one named setup with optional scoped resources.
setup:
  name: install-cli
  script: "npm install"

# List of objects: multiple named setups, run in order.
setup:
  - name: install-cli
    script: "npm install"
  - name: smoke-cli
    script: "npm test -- --help"

# Mixed list: strings and setup objects can be combined.
setup:
  - "npm install"
  - name: smoke-cli
    script: "npm test -- --help"

Setup object

A setup object is the named form of setup. Use it when setup needs its own files, environment variables, MCP servers, or setup checks.

When both a product and environment contribute setup, product setup runs before environment setup.

Setup checks

setup_checks verify that setup produced a usable sandbox before the agent starts. A failed setup check stops that variant before any agent work happens. Setup checks receive resolved environment_variables, AX_VARIANT_ID, and AX_RUN_CONTEXT_PATH; they do not receive trace or agent-log paths because the agent has not run yet.

setup_checks:
  - name: cli-on-path
    script: my-cli --version

Setup check object

MCP servers

mcp_servers exposes MCP servers to the agent for variants that use this setup. Use this field when the product surface or environment includes an MCP tool.

mcp_servers:
  - name: fixture-sentinel
    type: stdio
    command: /workspace/fixture-mcp.py
    args: []
  - name: ax
    type: http
    url: http://localhost:3001/mcp

MCP server object

An MCP server object defines one server and its transport. The required connection fields depend on type.

Transport-specific rules:

• stdio uses command, optional args, and optional env.
• http and sse use url and optional headers.
• Mixing stdio-only and endpoint-only fields is rejected.

MCP stdio env entry

env forwards declared secret values to a stdio MCP process. Each entry is either a bare secret name or an object:

env:
  - GITHUB_TOKEN
  - name: GH_AUTH
    from: GITHUB_TOKEN

MCP HTTP and SSE header object

headers attaches HTTP headers to an http or sse MCP server. Use placeholders when a header needs a declared secret value.

headers:
  - name: Authorization
    value: "Bearer ${SUPABASE_SERVICE_ROLE_KEY}"

MCP env entries and header placeholders reference environment variable names visible to the variant. Values can come from literal environment_variables, axp://secrets/<slug> references, or the deprecated secrets list.

Rules enforced at ax experiment validate time:

• Every stdio env[*].from and every ${NAME} placeholder in a header value must reference a name visible to the variant.
• Bare $NAME is treated as a literal; only ${NAME} is a placeholder.
• HTTP header names are unique per server, case-insensitively.
• command / args / env are only valid for type: stdio; url / headers are only valid for type: http / sse.

Resolved secret values are written into the agent session frame and can appear in run artifacts. Treat artifacts as sensitive whenever an experiment forwards secrets to MCP servers.

Extensions

extensions refine the variant set when the base axes do not express the combinations you need. Use extensions to narrow an axis, swap products or environments for a subtree, or append prompt guidance to a slice of variants.

If an experiment declares any extensions, AXP creates only extension-derived variants.

Extension object

An extension object is one node in the refinement tree. Nested extensions are recursively cross-multiplied with their parents.

prompts:
  - id: analyze
    prompt: "Read /workspace/task.md and write /workspace/report.json."

extensions:
  - id: with-cli
    products:
      - name: cli
        type: CLI
        setup: "curl -fsSL https://clickhouse.com/ | sh"
    prompts: ["Use the ClickHouse CLI for the analysis."]
  - id: without-cli
    products:
      - name: no-cli
        setup: "true"
    prompts: ["Do not use the ClickHouse CLI; use another local method."]

Environment variables

environment_variables injects environment variables into variants at runtime. Use it for non-secret runtime configuration and supported secret references.

environment_variables:
  - name: LOG_LEVEL
    value: debug
  - name: GITHUB_TOKEN
    value: axp://secrets/prod-gh
  - name: DATABASE_URL
    value: axp://secrets/staging-db

Environment variable object

Store secret values once with ax secret set <slug>, then reference them with axp://secrets/<slug>. Both platform and local ax experiment run invocations resolve these references before injecting env vars into the sandbox.

A referenced slug that does not exist in your org fails at preflight.

Reserved names:

• ANTHROPIC_API_KEY
• ANTHROPIC_BASE_URL
• OPENAI_API_KEY
• OPENAI_BASE_URL
• CURSOR_API_KEY
• MODEL
• MAX_TURNS
• IS_SANDBOX
• TRACEPARENT
• any name beginning with AXP_, CLAUDE_CODE_, CODEX_, CURSOR_, or OTEL_

Files

Top-level files stages host files or directories into every variant's /workspace before setup runs.

setup.files is accepted in experiment YAML, but current runs only stage top-level files. Put host file staging at the top level when you need the files delivered during a run.

files:
  - name: my-cli
    source: ../build/mycli
    dest: tools/mycli
  - source: https://example.com/fixtures/data.bin
    sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
    dest: fixtures/data.bin

File entry object

Notes:

• Directory sources copy their contents under dest/.
• File sources land as a single file at dest.
• URL sources must be publicly fetchable and land as a single file at dest; unpack archives in setup if needed.
• Sandboxes are Linux containers; macOS binaries staged from a laptop will not run there.
• Directory walks honor .axpignore, not .gitignore.
• --file NAME=SOURCE binds or overrides the source of a named entry.
• --file SOURCE::DEST stages an ad-hoc entry into every variant.
• Missing or unbound sources abort real runs at preflight. --resolve-variants renders MISSING / UNBOUND annotations without failing.
• Staging failures roll the variant up as status=error / exit_reason=staging_failed; the rest of the run keeps going.
• Platform and local runs both stage top-level files.
• Treat experiment YAML like a script you run: sources are read with your permissions and may point anywhere on the host.

Tests

tests defines how AXP scores each run. It is one flat list of bash checks. Test scripts receive resolved environment_variables, AX_VARIANT_ID, and path env vars for staged run artifacts, so any test can inspect final /workspace state, run commands, read $AX_TRACE_PATH, or branch on $AX_RUN_CONTEXT_PATH.

tests:
  - name: report-exists
    script: test -f /workspace/report.json
  - name: under-thirty-tool-calls
    script: |
      [ "$(jq -s '[.[] | select(.kind == "tool_call")] | length' "$AX_TRACE_PATH")" -lt 30 ]

Test runtime environment

Tests run inside the variant sandbox with /workspace as the working tree. The agent does not see the test scripts or these test-only path variables.

Example:

jq -e '.run_id != null and .agent == "claude"' "$AX_RUN_CONTEXT_PATH"

Test object

Limits

limits sets the run caps for each variant execution.

Limits object

The cost that max_cost_usd caps is the agent's own reported model spend — the running token cost the agent computes as it works. It excludes 514-managed sandbox and other harness infrastructure spend, and it applies the same way whether the run uses your own provider key or 514-managed model access. Because it's the agent's live estimate rather than a reconciled bill, treat the cap as an approximate guardrail.

Secrets deprecated

secrets is a deprecated alias for host environment variable names injected into variants at runtime.

Prefer environment_variables for literal values. Some MCP secret-forwarding fields still reference declared secret names.

secrets:
  - GITHUB_TOKEN

Secret names must match ^[A-Z_][A-Z0-9_]*$.

Validation rules

An experiment is invalid if:

• the YAML contains a field not defined by the schema
• schema_version is not 2
• any required field is missing
• an id that must be kebab-case is not kebab-case
• a model id contains ::
• a declared axis is empty
• duplicate ids or names appear where uniqueness is required
• the resolved variant set is empty
• a resolved variant has an empty prompt
• two resolved variants collide on variant_id
• no tests are defined
• test names are duplicated
• an env var or secret name is invalid or reserved
• a file entry has an invalid name, source, sha256, or dest
• an MCP server mixes transport-specific fields
• an MCP server references a secret not visible to the variant
• any limit is not greater than zero

YAML syntax boundaries

The experiment data model is JSON-compatible even though the authoring file is YAML.

• YAML comments are allowed.
• YAML anchors and aliases are allowed when the resolved value is JSON-compatible.
• Custom YAML tags are unsupported.
• Non-string mapping keys are unsupported.