Experiment YAML | AX Documentation

Experiment YAML is the file format passed to ax experiment run and ax experiment run --local.

The current parser accepts schema_version: 2 and rejects unknown fields at every level. For the canonical machine-readable schema, use experiment.v2.schema.yaml.

Schema URL

Use the version-pinned schema URL for editor validation:

# yaml-language-server: $schema=https://docs.514.ax/schema/experiment.v2.schema.yaml

Published schema URLs:

URL	Use
`https://docs.514.ax/schema/experiment.schema.yaml`	Latest supported schema
`https://docs.514.ax/schema/experiment.v2.schema.yaml`	Version-pinned v2 schema
`https://docs.514.ax/schema/experiment.v1.schema.yaml`	Legacy v1 schema. v1 experiments no longer run; this remains available so older files that pin it still validate in-editor while migrating to v2.

ax experiment schema prints the latest supported schema to stdout.

Field index

Object	Fields
Top-level experiment	`schema_version`, `id`, `name`, `description`, `agents`, `models`, `prompts`, `agent_mode`, `environments`, `products`, `extensions`, `environment_variables`, `files`, `tests`, `limits`
Agent	`name`
Model	`name`, `lab`, `effort`, `context_window_size`, `thinking`, `fast`
Prompt	`id`, `prompt`, `description`, `tags`
Environment	`name`, `setup`, `description`, `tags`, `commit`
Product	`name`, `type`, `setup`, `version`, `commit`, `description`, `tags`
Setup	`name`, `script`, `description`, `tags`, `files`, `environment_variables`, `mcp_servers`, `setup_checks`
Extension	`id`, `description`, `tags`, `agents`, `models`, `prompts`, `agent_mode`, `environments`, `products`, `extensions`
Environment variable	`name`, `value`
File entry	`name`, `source`, `sha256`, `dest`
Test	`name`, `script`
Setup check	`name`, `script`
MCP server	`name`, `type`, `command`, `args`, `url`, `env`, `headers`
MCP stdio env entry	`name`, `from`
MCP HTTP and SSE header	`name`, `value`
Limits	`max_time_seconds`

Minimal example

# yaml-language-server: $schema=https://docs.514.ax/schema/experiment.v2.schema.yaml
schema_version: 2
id: cli-install
name: "CLI install"

agents: [claude]
models: [claude-sonnet-4-6]

prompts:
  - id: install
    prompt: |
      Install the CLI and write its version to /workspace/version.txt.

products:
  - name: cli
    type: CLI
    setup: npm pack

tests:
  - name: version-file-exists
    script: test -f /workspace/version.txt

limits:
  max_time_seconds: 300

Top-level experiment

An experiment defines what you want to learn about agents using a product surface. It names the agents, prompts, products, environments, and tests AX uses to compute and run variants.

Field	Required	Type / shape	Notes
`schema_version`	Yes	Integer	Must be `2`.
`id`	Yes	Kebab-case string	Stable experiment id. Should match the YAML file name without `.yaml`.
`name`	Yes	String	Human-readable experiment name.
`description`	No	String	Optional context used when analyzing outcomes.
`agents`	No	Agent axis	Optional at the top level only if an extension supplies agents. Every resolved variant must have an agent.
`models`	No	Model axis	Optional. Cross-multiplied with `agents`; incompatible pairs are pruned. Absent ⇒ each agent runs its provider-default model.
`prompts`	No	Prompt axis	Optional at the top level only if an extension supplies prompts. Every resolved variant must have a non-empty prompt.
`agent_mode`	No	Agent-mode axis	Optional. `build` (default) or `plan`. Accepts a single value or list. `plan` runs the agent read-only (no mutations). Defaults to `build`.
`environments`	No	Environment axis	Optional.
`products`	No	Product axis	Optional.
`extensions`	No	Extension list	Optional. If present, only extension-derived variants are created.
`environment_variables`	No	Environment variable list	`{name, value}` entries injected into every variant. Values can be literals or `ax://secrets/<slug>` references resolved from your org secret store.
`files`	No	File entry list	Host files staged into every variant before setup runs.
`tests`	Yes	Tests list	Must contain at least one test.
`limits`	Yes	Limits object	Wall-clock timeout.

Top-level environment_variables and files apply to every variant. MCP servers and setup checks are setup-owned fields.

Variant axes

Variant axes are the experiment inputs AX combines into runnable variants. This lets you define the dimensions you care about once, then compare results by agent, prompt, environment, and product.

variants = agents × models × prompts × agent_mode × environments × products

Only agents and prompts must be specified: every resolved variant carries an agent and a prompt (each can also be supplied by an extension). The rest are optional. Omit models and each agent runs its provider-default model; omit agent_mode, environments, or products and that axis is left out of the cross product and of the variant coordinate.

Limits

limits sets the wall-clock timeout for each variant execution.

Limits object

Field	Required	Type / values	Notes
`max_time_seconds`	Yes	Integer greater than `0`	Wall-clock timeout in seconds enforced by the harness.

limits.max_turns and limits.max_cost_usd are removed authoring fields. ax experiment validate and platform submit reject both keys; remove them from new YAML. Only max_time_seconds remains.

Older runtime artifacts may still record a historical max_turns or cost_cap exit reason when a legacy run was stopped on a turn or spend cap; that is compatibility for past runs, not a current authoring field.

Secrets removed

The top-level (and setup) secrets: field is rejected by ax experiment validate and platform submit, including an empty secrets: []. Use environment_variables with literals or ax://secrets/<slug> references. MCP secret-forwarding fields reference declared environment_variables names, not store slugs.

Validation rules

An experiment is invalid if:

the YAML contains a field not defined by the schema
schema_version is not 2
any required field is missing
an id that must be kebab-case is not kebab-case
a model id contains ::
a declared axis is empty
duplicate ids or names appear where uniqueness is required
the resolved variant set is empty
a resolved variant has an empty prompt
two resolved variants collide on variant_id
no tests are defined
test names are duplicated
an env var name is invalid or reserved
a file entry has an invalid name, source, sha256, or dest
an MCP server mixes transport-specific fields
an MCP server references an env var name not visible to the variant
any secrets: key is present (removed; use environment_variables)
any limit is not greater than zero

YAML syntax boundaries

The experiment data model is JSON-compatible even though the authoring file is YAML.

YAML comments are allowed.
YAML anchors and aliases are allowed when the resolved value is JSON-compatible.
Custom YAML tags are unsupported.
Non-string mapping keys are unsupported.

Search documentation