Experiments

Experiments allow you to understand how agents use your products. When you define an experiment you define:

1. the agents that will be evaluated in the experiment
2. the models the agents run — an independent axis, cross-multiplied with agents (incompatible agent/model pairs are pruned; omit it to use each agent's provider-default model)
3. the tasks you want the agents to try to accomplish, in the form of prompts
4. the product(s) or product version(s) that you are trying to test
5. the environments that you want agents to operate in
6. how you will measure the success of the agent, with tests

Each of these is an axis, and you can give any axis a list of values (several agents, models, prompts, products, or environments) to see how changing one aspect affects the agent's performance. When you run the experiment, AXP computes the cross product of those values and runs the task across every resulting variant.

For the exact field names, types, and validation rules, see the experiment YAML reference.

For guidance on choosing the product question, variants, tests, and success criteria, see Experiment Design.

Variants

A variant is one fully resolved combination of an agent, a model, a prompt, a product, and an environment. This forms the exact configuration that a single run executes inside of an isolated sandbox. Variants allow you to compare the performance of different configurations against each other.

You don't write variants by hand. They're the combinations produced by the cross product of your axes described above, so every combination of the values you declared becomes its own variant. Declaring two models and two products, for example, gives you four variants.

Tests

Tests allow you to define how you will measure the success of the agent at performing the task of your experiment. Tests are defined as scripts inside the experiment yaml, and each one you define is automatically executed inside the sandbox after the agent finishes its work. A test passes when its script exits 0 and fails when the script exits with an error, so write each script to inspect the agent's work and exit with an error when the result is wrong. Every test can read $AX_RUN_CONTEXT_PATH for JSON run context such as the run id, variant id, agent, model, prompt, product, environment, tags, and extension id; absent axes and controls are null.

Final-state checks

Most tests should check the resulting state — the files, commands, endpoints, or product behavior the agent was supposed to produce. These are useful to check that the answer or work that the agent completed inside of the sandbox is correct.

Trace checks

Tests can also inspect the agent's trace to check how it completed the task. These are useful to catch a variant that reached the right answer the wrong way, or if the agent took a particularly circuitous path to get there. Use $AX_TRACE_PATH for agent-events.jsonl, $AX_AGENT_LOG_PATH when agent.log exists, and $AX_RUN_CONTEXT_PATH for coordinates you want to branch on with jq.

Every experiment needs at least one test. See the tests reference for scripting details.

Limits

Limits can be set in your experiment YAML file to prevent a single run from looping or spending too much time or money. Limits allow you to specify the maximum number of turns, wall-clock time, and cost for a run. Wall-clock and reported-cost limits abort the agent and mark the run timeout or cost_cap; turn limits are passed to the agent where supported.

Versioning

As you refine an experiment, AXP creates a new experiment version whenever you change something meaningful in its definition, like a prompt, model, or test. Editing comments doesn't count and re-running an unchanged experiment reuses its current version.

Each variant is versioned on its own, so a new version only needs fresh runs for the variants you changed. Variants you didn't change carry over: their earlier runs still appear under the new version, without re-running.

Every run keeps the exact definition it ran against, so later edits never change what a past run was measured against. That's what makes the Experiment version filter in Results reliable for comparing like with like.