Experiment Design

AX experiments work best when they answer a product question and point to a clear next step. This experiment design guide helps you move from observed agent behavior to a product change, docs change, regression test, or follow-up experiment.

Experimental Methodology

1. Write the experiment
   1. Find a starting point. Start from observed agent behavior, product analytics, support feedback, prior runs, or a gut feel about where agents may hit friction. This creates your research questions.
   2. Ask a research question. Turn the starting point into a question that compares surfaces or variants, and say how you will measure the answer. Write both into the experiment YAML description.
   3. Place a bet. Write what you expect before running, concrete enough to be wrong. Put it in the same description. For example: "Sophisticated analytics prompts will raise consideration; once considered, selection will convert at a high rate."
   4. Define the experiment. Author the rest of the YAML: setup, variants, and tests. Also record success criteria and compare dimensions in description.
2. Run the experiment
   1. Smoke test and refine. Run just enough trials to test the experiment itself (ax experiment validate, then a small ax experiment run). Fix ambiguous prompts, brittle tests, broken setup, or variants that do not isolate the thing you meant to study. Do this before you scale.
   2. Scale the experiment. Spend the larger budget across the full variant matrix, stepping up --repeat as signal stabilizes.
3. Analyze results
   1. Explore your data. Query runs and variants against the research question.
   2. Create an insight. Save what you learned so the team can re-run the evidence.

For AX concepts and YAML structure, see Experiments, Runs, Results, and the experiment YAML reference.

1. Write the experiment

Turn friction into an experiment file: a starting point, a research question and bet in description, then setup, variants, and tests.

Find a starting point

You do not need a perfect observation before you start. A useful starting point can be a prior run, analytics, support feedback, a customer question, a product area you suspect is confusing, or a workflow you want agents to complete more reliably.

Examples we will follow through the rest of this page:

• "Agents build analytics apps all the time, but our database rarely shows up in the stack." (call this selection during builds)
• "When agents do pick us, we do not know if they read our docs or chose from memory." (call this docs vs memory)

Ask a research question

Write the research question into the experiment YAML description before you spend budget. Good research questions are derived from your starting point. Prefer report-style questions: comparative, tied to a decision you can act on, and explicit about how you will measure the answer (which test, rate, or session-data signal counts).

Examples:

• How often is our product considered for analytics and billing builds, and how often is it actually selected? Measure with tests over the agent transcript for “considered anywhere” vs “named in the plan’s conclusion.”
• When it is considered, what else is on the shortlist, and why do we win or lose? Measure co-mentioned datastores in the same run, plus reason codes from the plan text.
• Does selection vary by use case, prompt sophistication, or starting framework? Measure selection rate sliced by those axes.
• Do agents cite benchmarks or docs, or is selection from memory? Measure fetches and citations in the transcript, stdout, or OTEL events before the decision.

Put the question and the measurement next to each other in the experiment YAML file’s description field:

description: |
  Research question: How often is our product considered for analytics
  builds, and how often is it actually selected?
  Measure: pass rate of product-considered (named anywhere in the agent
  transcript) and product-selected (named in the plan conclusion with
  decision language).

After the first run, ask whether the results actually teach you about the research question. If not, the test may be measuring the wrong thing, the prompt may be too artificial, or the variants may not isolate the contrast.

Place a bet

Bets turn the research question into a testable hypothesis. Write down what you expect before running the experiment; good bets are concrete enough to be wrong. Put the bet in the same experiment YAML description as the research question so later passes (and your coding agent) can check results against what you expected.

Examples:

• "Sophisticated analytics prompts will raise consideration more than simple ones; once considered, conversion to a real build will stay high."
• "Agents will almost never fetch docs or benchmarks before choosing; selection will look like memory, not research."

description: |
  Research question: How often is our product considered for analytics
  builds, and how often is it actually selected?
  Measure: pass rate of product-considered (named anywhere in the agent
  transcript) and product-selected (named in the plan conclusion with
  decision language).
  Bet: Sophisticated analytics prompts will raise consideration more than
  simple ones; once considered, conversion to a real build will stay high.

Avoid vague bets like "schema matters" or "docs should help." A good bet helps you design the experiment and makes surprises visible when results disagree with expectations.

Ask your coding agent to place a bet too; its expectations can reveal a different theory of the product, and the disagreement is often useful.

Define the experiment

Define three things: the setup (the scenario variants run through), the variants you compare, and the tests that score the result. In AX these map onto the experiment YAML; see Experiments and the experiment YAML reference. By now description should already hold the research question, measure, and bet; optionally add secondary metrics (cost, wall clock) and the dimensions you will compare.

The setup

The setup is the scenario a variant runs through: the task (prompt) plus the world (environments / products and their setup scripts). One environment or product can chain several setup objects in order. Multiple environments or products are different: they become axes, so variants get different worlds.

Keep the shared world stable unless the world itself is what you are studying (for example, blank workspace vs a framework scaffold). If every variant runs through an unrelated world, the results are harder to interpret.

The variants

Variants are the changes you want to measure. You do not write each one out by hand. You declare the variables you want to compare (agents, models, prompts, environments, and products) and AX runs every combination as a variant. Change one variable at a time so the contrast stays clean. For each variant, make clear:

• what changed
• what stayed constant
• which research question it informs
• which test outcomes should move if the bet is right

For selection during builds, vary prompts (use case × simple vs sophisticated) and environments (blank workspace vs common framework scaffolds). Keep the product under test fixed so you measure consideration and selection, not a catalog swap.

For docs vs memory, keep the task fixed and compare what agents fetch and cite: no docs in the environment, docs available but unprompted, or an explicit "check the docs" instruction.

When the product under test is a skill (or similar surface), separate two questions with axes:

1. Availability vs instruction: one prompt where the skill is present but not named; another where the prompt explicitly calls it. First measure whether agents use it; then measure whether use improves performance.
2. Skill on/off: compare a product or environment with the skill present against one without it.

Prefer deterministic functional tests when the task has a fixed correct answer. Use LLM-as-judge only when you truly need open-ended grading; it is slower, more expensive, and less stable under repetition.

The tests

Tests are one flat list. They should measure the task, not coach the agent. Some check the artifact, output, file, command, or product behavior the agent was supposed to produce. Others inspect session data: the agent transcript, stdout, OTEL events, or related run artifacts (ACP runs expose the transcript to tests via $AX_TRACE_PATH when present).

Keep scaffolding in tests, not in prompts. Keep golden answers and answer keys out of the agent workspace; tests run in a separate sandbox. See Tests.

2. Run the experiment

Same commands, different budget: prove the experiment works, then spend.

Smoke test and refine

Before spending a real budget, smoke test the experiment itself, not the agent. Validate, resolve a variant id, then run one cell once (or --mock with no model spend):

ax experiment validate selection.yaml
ax experiment variants selection.yaml
ax experiment run selection.yaml --variant '<variant-id>' --repeat 1
ax experiment run selection.yaml --mock

Use it to check (or ask your coding agent to check):

• Does the setup work?
• Do the tests work?
• Do the results help answer the research question?
• Are variants going through the intended paths?
• Is the agent cheating or exploiting scaffolding?
• Are failures real signal, or broken scaffolding?

Fix ambiguous prompts, brittle tests, broken setup, or leaky scaffolding, then smoke test again. When the setup works, the variants isolate the contrast, and the tests catch real success and failure, you are ready to scale.

Scale the experiment

Scale by dropping the --variant filter to run the full matrix and raising --repeat. Set limits.max_time_seconds before you spend, so a hung variant cannot burn the whole budget. Use a ladder so you spend when the experiment is already trustworthy:

1. Scaffolding pass: one model, one run per variant. Goal: setup, tests, and variants work. Passing tests is optional here.
2. Small multi-rep: about 5 repeats across the models you care about. Confirms the experiment holds under repetition. Ask the cheat / research-question checks again.
3. Shape / significance: about 30 repeats. Use this to read distributions and decide. If you changed the experiment after the n≈5 pass, run a full n≈30 again rather than mixing old and new trials.
4. After a baseline: PR checks can use fewer repeats once you know the distribution. You are checking fit against a known shape, not reshaping the whole experiment.

# scaffolding / small multi-rep / shape
ax experiment run selection.yaml --repeat 1
ax experiment run selection.yaml --repeat 5
ax experiment run selection.yaml --repeat 30

If the outcome you care about is rare (for example, a tool or database is mentioned in only a few percent of runs), n≈30 is often too small. Plan for roughly 100+ repeats when you are hunting a low base-rate event.

3. Analyze results

Start with the primary research question, then drill into variants, runs, tests, and session data. Read individual attempts in Runs and compare variants in Results. AX already records wall clock, tokens, cost, tool calls, and failures for every run; you do not define those as tests unless they are part of how you measure the research question.

Explore your data

Use ax experiment query for experiment-scoped rollups, and ax run view / ax run query when you need one attempt. Ask your coding agent to run these against the research question in description, then open outliers in the agent transcript and session data.

# What dimensions and metrics exist?
ax experiment query selection --dimensions

# Pass rate by variant (the default comparison)
ax experiment query selection --metric testPassRate --group-by variant

# Slice by the axes you varied
ax experiment query selection --metric testPassRate --group-by prompt,environment
ax experiment query selection --metric testPassRate --group-by model
ax experiment query selection --metric testPassRate --group-by test

# Cost and tokens alongside pass rate
ax experiment query selection --metric testPassRate,cost,tokens --group-by prompt

# Filter to one slice, then re-group
ax experiment query selection --metric testPassRate --group-by environment \
  --filter prompt=sophisticated-analytics

# Ad-hoc SQL over the experiment's uploaded data
ax experiment query selection --sql @selection-rate.sql

# One run request, then SQL on a single composite run id
ax run view <run-request-id>
ax run query <run-id> --sql "SELECT test_name, exit_code FROM tests" --table

Useful exploration patterns:

• Compare distributions across variants, not just averages. A variant with the same mean but a tighter spread may be easier to trust.
• Look for interaction effects between the things you varied. For example, docs might help one model more than another, or a CLI example might only help when paired with a specific prompt.
• Ask your coding agent to find outliers in the results, then inspect the agent transcript, stdout, and related session data for those runs. Outliers often reveal the failure mode you actually need to fix.
• After an n≈30 pass, check whether distributions overlap or separate cleanly. If they overlap and the decision still matters, run another 30 to 100 before you commit.

Create an insight

When the exploration answers the research question, save it as an insight so the team can re-run the evidence without living in the YAML. Include the research question, what you compared, how you measured, and the decision. Easiest path: add --save to a query that already shows the result. See ax insight.

ax experiment query selection --metric testPassRate --group-by prompt,environment \
  --save "Selection rate by prompt and environment"
ax insight view selection-rate-by-prompt-and-environment
ax insight share selection-rate-by-prompt-and-environment

Or create from SQL you already wrote:

ax insight create "Selection rate by prompt and environment" \
  --sql @selection-rate.sql \
  --description "RQ: how often is our product considered vs selected?"

Reference: common experiment patterns

• Discovery and install: Where do agents hit bottlenecks when discovering or onboarding onto a product?
• Core user flows: Where do agents succeed or fail at important jobs-to-be-done?
• Product optimization: Which variation of an API, CLI command, tool description, or interface helps agents succeed faster?
• Interface comparison: Do agents perform better with docs, skills, MCP tools, APIs, or another interface?
• Competitive analysis: What alternatives do agents reach for, and what can you learn from their agent experience? When the mention rate is low, plan for larger n (see Scale the experiment).
• Product marketing: Can you generate evidence that agents are more effective with your product?

Reference: common failure modes

Paste these into your coding agent with the experiment YAML (and a smoke-run id when you have one).