Search documentation

Search the Fumadocs-backed documentation index.

Test your CLI installation

Run an intro experiment that shows where coding agents get stuck installing your CLI.

This guide uses the cli-install template to find and ship low-hanging fixes where coding agents get stuck installing your CLI. The prompt below walks your coding agent through create, run, analyze, and saving an insight you can act on.

The experiment varies:

  • The prompt used to instruct an agent to install your product
  • Their coding agent and selected model
  • Different environment setups that mimic common configurations on your users' machines

Once you run the experiment, you will have a baseline for how often agents succeed, how many steps they take, and how long installation takes. At the end of this guide, you will mine agent transcripts, tool history, and other raw session data to identify recurring failure patterns, turn them into quick product fixes, and retest to see whether your metrics improve.

The analysis prompts and queries come from hundreds of experiments on the ax CLI and design partners' CLIs.

Setup your environment

At a minimum, you must have the ax CLI installed. We recommend also linking your CLI to your AX Cloud account, but you can run everything locally without signing up if needed.

Already set up? Jump to Create your intro experiment.

Prompt
Use these two commands to install and authenticate the ax CLI:bash <(curl -fsSL https://dl.514.ax/install.sh) axax auth login
This will open a browser for me to authenticate in. Direct me to do so. If you're unable to open a browser for me, then run:ax auth login --no-browserto print the URL and ask me to navigate to that URL to authenticate.
Alternatively ask me to create an API access token at https://app.514.ax/account/api-keys and authenticate the CLI by running:ax auth login --token <<token>>which does not require any browser interaction. Warn me not to enter the raw token in chat with you, as this is a security vulnerability.
If I want to try Fiveonefour locally without an account, verify Docker is running and skip signup. Local runs still need managed model access (sign in or AX_API_KEY) or my own provider key with --local-model-keys.

Create your intro experiment: Can agents install your CLI?

The cli-install template works with the target CLI you choose. Once you specify the CLI, your coding agent configures the experiment with its install docs, install command, smoke command, and expected output.

The template compares four levels of install guidance (basic, informed, documented, explicit) and scores each run with an isolated hidden smoke test that confirms your target CLI is installed and runnable.

Prompt
I'm a first-time ax user. Help me create and run my intro experiment.
Use the `ax` CLI for every step. Coach me through the first-experimentflow, and ask me about MY CLI / product. Do not assume AX itself is thetarget, and do not invent flag values.
1. Run `ax --version`. If it fails, show the install command   (`bash <(curl -fsSL https://dl.514.ax/install.sh) ax` or `brew install 514-labs/tap/ax`), ask me to   confirm, then install and re-check. Do not stop or send me elsewhere.   Signing in is optional. Ask whether I want an AX Cloud run or `--local`.   AX Cloud runs need `ax auth status` to succeed. Local runs need Docker;   managed model access still uses auth or `AX_API_KEY`, otherwise   `--local-model-keys` with my provider key.
2. Figure out what we're testing. Prefer the `cli-install` template.   First ask whether I have my own CLI to test. If I do, ask me for anything   you cannot infer (and never silently invent):   - experiment name (default: my-first-experiment)   - target CLI binary name (`--cli`)   - short product description (`--target-description`)   - install docs URL (`--install-docs`)   - install command (`--install-command`)   - smoke command (`--smoke-command`)   - optional smoke-output text (`--smoke-output-contains`)   - AX Cloud run vs `--local` (and `--local-model-keys` if local without auth)
   If I do not have a CLI of my own and just want to try Fiveonefour, fall   back to testing the `ax` CLI with these exact values:   - `--cli ax`   - `--target-description 'the Fiveonefour CLI for running agent evaluation experiments'`   - `--install-docs https://docs.514.ax/cli/installation`   - `--install-command 'bash <(curl -fsSL https://dl.514.ax/install.sh) ax'`   - `--smoke-command 'ax --version'`   - `--smoke-output-contains 'ax '`
3. Scaffold with `ax experiment create <NAME> --template cli-install` and every   required flag filled in from my answers (or the fallback values above).
4. Validate with `ax experiment validate <NAME>.yaml` and fix any errors.
5. Run with `ax experiment run <NAME>.yaml` for AX Cloud, or   `ax experiment run <NAME>.yaml --local --watch` for local.
6. When the run finishes, give me the AX Cloud URL or local summary. For   local runs, mention `ax run upload <run-id>` if I later want results on   AX Cloud.
7. Analyze with `ax learn analyze-results`. Start from   `ax run list --experiment <NAME>` and pick a mode:   - Debug (failed, errored, or incomplete runs): use `ax run view` /     `ax run query` to find the blocker (auth, Docker/local vs AX Cloud,     model keys, sandbox, YAML, environment). Fix it and re-run until runs     complete. Do not claim product insights yet.   - Readiness (runs completed): say whether we have enough completed runs     across variants (prompt levels, agents/models, environments) for     statistically significant insights. If not, recommend how to get there     (`--repeat`, `--parallel`, or a broader matrix) and re-run first.   - Analyze (analysis-ready only): follow the analyze-results workflow.     Ground every claim in query or transcript evidence and cite run counts.
8. When analysis-ready, answer these questions in order:   - Does more install guidance help? Compare testPassRate across the prompt     variants (baseline, informed, documented, explicit). Call out     differences that hold across repeats vs ones within noise.   - Are results consistent across agents and models, or does one agent or     model drive the trend? Slice testPassRate by model and agent.   - What does success cost? Compare cost, tokens, and wallClockTime by     prompt variant.   - Which test fails most: result-file-valid or installation-check-passes?   - For failed product outcomes (completed runs that fail tests): did the     agent fail to install the target CLI itself, or get stuck on     dependencies around it (package managers, PATH, permissions, network)?     Open transcripts and classify each failure.   - What other failure reasons show up repeatedly? Corroborate the numbers     with the agent's reasoning and list the top 2-3 patterns with run ids.
9. Finish with the single most actionable product fix the data supports.   Then follow `ax learn publish-insight`: save the most useful query with   `--save`, reopen it with `ax insight view`, and ask whether I want to   share it with `ax insight share`.

Understanding the experiment structure

When you create the experiment, the AX CLI writes a YAML file called my-first-experiment.yaml in the current directory.

At a high level, the generated YAML looks like this:

schema_version: 2
id: my-first-experiment
agents: [claude]
models: [anthropic/claude-sonnet-4-6]
prompts:
  - id: baseline
  - id: informed
  - id: documented
  - id: explicit
environments:
  - name: host
tests:
  - name: result-file-valid
  - name: installation-check-passes
limits:
  max_time_seconds: 900

The four prompts entries are the install-guidance variants. The tests entries are hidden from the agent. You can see the full experiment schema in the experiment YAML reference.

Advanced run options

ax experiment run validates the experiment, submits it to AX Cloud, and polls until all runs are complete. If you don't specify any flags in the run command, all possible combinations of prompts, agents and models, as declared in your YAML file, will be computed and queued for execution.

Running specific prompts, agents, and models

You can alternatively select specific prompts, agents, and models to run by adding --prompt, --agent, and --model flags.

ax experiment run my-first-experiment.yaml --prompt baseline --prompt explicit

Running at scale

You should aim to run each variant of your experiment enough times to get a statistically significant result. The more runs you have, the more confident you can be in your results.

AX gives you two levers to get these runs done faster:

  1. Parallelize runs with the --parallel flag:
ax experiment run my-first-experiment.yaml --parallel 4
  1. Repeat each variant with the --repeat flag:
ax experiment run my-first-experiment.yaml --repeat 3

Running locally

To run locally instead (no AX Cloud account required), you need Docker running on the host. By default local runs still use managed model access when you are signed in (ax auth login or AX_API_KEY). Pass --local-model-keys when supplying a host model API key instead (ANTHROPIC_API_KEY for Claude, OPENAI_API_KEY for Codex, CURSOR_API_KEY for Cursor). See Troubleshooting if ax experiment run --local exits non-zero immediately.

ax experiment run my-first-experiment.yaml --local --watch

Upload local results later with ax run upload <run-id> if you sign in.

The create prompt above continues into analysis after the run finishes: debug incomplete runs, confirm the completed set is large enough for statistically significant insights, then answer install-specific questions and save an insight with ax learn publish-insight.