Getting Started

This guide walks through creating, validating, running, querying, and saving your first insight from an AX experiment. Run commands from the directory where you want AX to create experiment files.

Install the AX CLI

bash <(curl -fsSL https://dl.514.ax/install.sh) ax

Also available via brew install.

Sign up / sign in

1. Request access: join the waitlist
2. Sign in from the CLI:

ax auth login

For non-interactive setup, append the token to the command above: --token <api-key>.

Optionally, install the AX MCP

Connect the AX MCP server to your agent host. This lets your agent run experiments, monitor remote runs, query results, manage secrets, and link your local CLI. See the MCP tool catalog.

{
  "mcpServers": {
    "AX": {
      "url": "https://app.514.ax/mcp"
    }
  }
}

Create an intro experiment: can agents install your CLI?

ax experiment create --template cli-install creates a starter experiment that measures whether an agent can install and smoke-test a target CLI.

The experiment compares four prompt variants, each with more install guidance.

Ask your agent something like:

I'm a first-time AX user. Create and run an experiment with my CLI using the
`ax` command.

Or run the command directly:

ax experiment create my-first-experiment --template cli-install \
  --cli mycli_name \
  --target-description 'a CLI for managing OLAP databases' \
  --install-docs https://example.com/install \
  --install-command 'npm install mycli_name' \
  --smoke-command 'mycli_name --version' \
  --smoke-output-contains 'mycli_name version'

AX writes my-first-experiment.yaml in the current directory.

At a high level, the generated YAML looks like:

schema_version: 2
id: my-first-experiment
agents: [claude]
models: [anthropic/claude-sonnet-4-6]
prompts:
  - id: baseline
  - id: informed
  - id: documented
  - id: explicit
environments:
  - name: host
tests:
  - name: result-file-valid
  - name: installation-check-passes
limits:
  max_time_seconds: 900

The four prompts entries are the install-guidance variants. The tests entries are hidden from the agent.

Validate the experiment

ax experiment validate my-first-experiment.yaml

ax experiment validate checks the experiment file and exits non-zero on schema errors. When signed in, it may also print an advisory AI review of the experiment design. That review is informational only: it does not change the exit code, so you can run next even if it suggests improvements.

Run the experiment

ax experiment run my-first-experiment.yaml

ax experiment run validates the experiment, submits it to the AX platform, prints the run request id and a platform URL, and polls until the request finishes. Use ax run watch only if you submitted with --detach or detached with Ctrl-C:

ax run watch <run-request-id>

To run only some prompts, or to repeat each selected variant for more signal, add --prompt or --repeat:

ax experiment run my-first-experiment.yaml --prompt baseline --prompt explicit --repeat 3

Or run the experiment locally

Local runs need Docker running on the host. By default they use the same managed model access as platform runs (ax auth login or AX_API_KEY). Pass --local-model-keys only when supplying a host model API key (ANTHROPIC_API_KEY for Claude, OPENAI_API_KEY for Codex, CURSOR_API_KEY for Cursor). See Installation and Troubleshooting if ax experiment run --local exits non-zero immediately.

ax experiment run my-first-experiment.yaml --local --watch

Upload local results with ax run upload <run-id>.

Query results and save an insight

Open the platform link

ax experiment run prints a platform URL. Open it to review the experiment, variants, run statuses, and test outcomes.

Example results:

Query results

Ask your agent something like:

What is the test pass rate for each variant of my-first-experiment?

Or run the query directly:

ax experiment query my-first-experiment --metric testPassRate --group-by variant

Save an insight

Add --save to keep a query as a reusable insight:

ax experiment query my-first-experiment --metric testPassRate --group-by variant \
  --save "Install variant pass rate"

Re-run the saved insight any time:

ax insight view install-variant-pass-rate

Next steps

You have run the starter experiment. Next, learn Experiment Design for choosing the task, variants, tests, and success criteria.