Search documentation

Search the Fumadocs-backed documentation index.

Overview

What AX does and where to start.

AI agents are a new kind of user for your product. AX helps you find out how well they can actually use it.

Unlike eval tools that score model output against datasets, AX runs sandboxed experiments that measure whether agents can operate your real product surfaces: CLIs, SDKs, APIs, web apps, MCP servers, and docs, in real complex environments.

The goal is to use those experiments to observe agents and turn agent behavior into repeatable product evidence.

It is built for product, DevRel, and engineering teams improving how agents use their product, and for teams evaluating competing products before they adopt them.

Use AX to test new features, improvements, and bug fixes before you release them. You can answer questions like:

  • How do agents use your product?
  • Do changes to your product make agents better or worse at completing tasks?
  • Which models or coding agents perform best with your product? Which perform worse?

You can also run AX experiments to evaluate competing products that you are considering adopting, buying, or investing in. You can answer questions like:

  • How does your product perform against an agent inventing a solution from scratch?
  • How does your product perform against your competition?
  • In the contexts your customers work in, do agents suggest your product?

These questions map to common experiment patterns: discovery and install, core user flows, product optimization, interface comparison, competitive analysis, and product marketing. See Experiment Design.

How AX Works

01
Design an experiment
Choose the question, variants, and tests.
02
Run your experiment
Execute each variant in its own sandbox.
03
Analyze results
Compare variants and decide what to change.

Iterate: feed each result into your next experiment.

AX has three main pieces:

  1. Design an experiment: define the task, the variants you compare (different agents, models, environments, or product versions), and the tests that measure success.
  2. Run your experiment: execute each variant in its own sandbox on the AX platform with ax experiment run (local Docker is available with ax experiment run --local), capturing logs, file changes, test results, token usage, and cost.
  3. Analyze results: use ax experiment query to compare runs and variants, then save reusable views with --save and ax insight view.
Experiments
The spec that defines the task, variants, setup, and tests you want an agent to run.
Runs
A completed execution of an experiment, including variant outputs, costs, logs, and artifacts.
$0.57
Avg cost
5m00s
Avg wall clock
321
Tool failures
100%
Test pass rate
Group by: Variant
X-axis: Wall clock
Y-axis: Cost
Cost vs wall clock, grouped by variant
0m00s
13m20s
Data and results
The summaries you generate from runs to compare variants and decide what to change next.

Installation

The fastest way to get started is the official AX CLI installer:

bash <(curl -fsSL https://dl.514.ax/install.sh) axax --help

Platform runs require an account (currently closed alpha). Join the waitlist, then sign in with ax auth login once you have access. See Installation for details.

Supported Coding Agents

AX currently supports Anthropic Claude Code, OpenAI Codex, and Cursor.

Need support for another coding agent? Ask us to add it.

Next steps

  1. Install AX and sign in.
  2. Run your first experiment in Getting Started.
  3. Design your own with Experiment Design.