Test your CLI installation

This guide uses the cli-install template to find and ship low-hanging fixes where coding agents get stuck installing your CLI. The prompt below walks your coding agent through create, run, analyze, and saving an insight you can act on.

The experiment varies:

The prompt used to instruct an agent to install your product
Their coding agent and selected model
Different environment setups that mimic common configurations on your users' machines

Once you run the experiment, you will have a baseline for how often agents succeed, how many steps they take, and how long installation takes. At the end of this guide, you will mine agent transcripts, tool history, and other raw session data to identify recurring failure patterns, turn them into quick product fixes, and retest to see whether your metrics improve.

The analysis prompts and queries come from hundreds of experiments on the ax CLI and design partners' CLIs.

Setup your environment

You need the ax CLI installed and linked to your AX Cloud account. Runs execute in Fiveonefour-managed sandboxes, so there is nothing else to set up on your machine.

Already set up? Jump to Create your intro experiment.

Prompt

Use these two commands to install and authenticate the ax CLI:curl -fsSL https://dl.514.ax/install.sh | bash -s -- axax auth login
This will open a browser for me to authenticate in. Direct me to do so. If you're unable to open a browser for me, then run:ax auth login --no-browserto print the URL and ask me to navigate to that URL to authenticate.
Alternatively ask me to create an API access token at https://app.514.ax/account/api-keys and authenticate the CLI by running:ax auth login --token <<token>>which does not require any browser interaction. Warn me not to enter the raw token in chat with you, as this is a security vulnerability.
If I don't have an account yet and want to try ax first, offer to create a temporary account instead:ax auth signup --temp-accountCreating one accepts the Terms of Use, so ask me before you run it. The account lasts 24 hours unless I run `ax auth signup --claim-account` and finish signing up in the browser to keep it.

Create your intro experiment: Can agents install your CLI?

The cli-install template works with the target CLI you choose. Once you specify the CLI, your coding agent configures the experiment with its install docs, install command, smoke command, and expected output.

The template compares four levels of install guidance (basic, informed, documented, explicit) and scores each run with an isolated hidden smoke test that confirms your target CLI is installed and runnable.

Prompt

I'm a first-time ax user. Help me create and run my intro experiment.
Use the `ax` CLI for every step. Coach me through the first-experimentflow, and ask me about MY CLI / product. Do not assume AX itself is thetarget, and do not invent flag values.
1. Run `ax --version`. If it fails, show the install command   (`curl -fsSL https://dl.514.ax/install.sh | bash -s -- ax` or `brew install 514-labs/tap/ax`), ask me to   confirm, then install and re-check. Do not stop or send me elsewhere.   Experiments run on AX Cloud, so `ax auth status` must succeed. If it   does not, help me sign in with `ax auth login` (or `AX_API_KEY`). If I   have no account yet, offer `ax auth signup --temp-account` instead.   Creating one accepts the Terms of Use, so ask me before you run it.
2. Figure out what we're testing. Prefer the `cli-install` template.   First ask whether I have my own CLI to test. If I do, ask me for anything   you cannot infer (and never silently invent):   - experiment name (default: my-first-experiment)   - target CLI binary name (`--cli`)   - short product description (`--target-description`)   - install docs URL (`--install-docs`)   - install command (`--install-command`)   - smoke command (`--smoke-command`)   - optional smoke-output text (`--smoke-output-contains`)
   If I do not have a CLI of my own and just want to try Fiveonefour, fall   back to testing the `ax` CLI with these exact values:   - `--cli ax`   - `--target-description 'the Fiveonefour CLI for running agent evaluation experiments'`   - `--install-docs https://docs.514.ax/cli/installation`   - `--install-command 'curl -fsSL https://dl.514.ax/install.sh | bash -s -- ax'`   - `--smoke-command 'ax --version'`   - `--smoke-output-contains 'ax '`
3. Scaffold with `ax experiment create <NAME> --template cli-install` and every   required flag filled in from my answers (or the fallback values above).
4. Validate with `ax experiment validate <NAME>.yaml` and fix any errors.
5. Run on AX Cloud with `ax experiment run <NAME>.yaml`.
6. When the run finishes, give me the AX Cloud URL.
7. Analyze with `ax learn analyze-results`. Start from   `ax run list --experiment <NAME>` and pick a mode:   - Debug (failed, errored, or incomplete runs): use `ax run view` /     `ax run query` to find the blocker (auth, model access, sandbox,     YAML, environment). Fix it and re-run until runs     complete. Do not claim product insights yet.   - Readiness (runs completed): say whether we have enough completed runs     across variants (prompt levels, agents/models, environments) for     statistically significant insights. If not, recommend how to get there     (`--repeat` or a broader matrix) and re-run first. Do not suggest     `--parallel` (it does not exist).   - Analyze (analysis-ready only): follow the analyze-results workflow.     Ground every claim in query or transcript evidence and cite run counts.
8. When analysis-ready, answer these questions in order:   - Does more install guidance help? Compare testPassRate across the prompt     variants (baseline, informed, documented, explicit). Call out     differences that hold across repeats vs ones within noise.   - Are results consistent across agents and models, or does one agent or     model drive the trend? Slice testPassRate by model and agent.   - What does success cost? Compare cost, tokens, and wallClockTime by     prompt variant.   - How often does `installation-check-passes` fail, and does failure rate     differ by prompt variant?   - For failed product outcomes (completed runs that fail tests): did the     agent fail to install the target CLI itself, or get stuck on     dependencies around it (package managers, PATH, permissions, network)?     Open transcripts and classify each failure.   - What other failure reasons show up repeatedly? Corroborate the numbers     with the agent's reasoning and list the top 2-3 patterns with run ids.
9. Finish with the single most actionable product fix the data supports.   Then follow `ax learn publish-insight`: save the most useful query with   `--save`, reopen it with `ax insight view`, and ask whether I want to   share it with `ax insight share`.

Understanding the experiment structure

When you create the experiment, the AX CLI writes a YAML file called my-first-experiment.yaml in the current directory.

At a high level, the generated YAML looks like this:

schema_version: 2
id: my-first-experiment
agents: [claude]
models:
  - name: claude-sonnet-4-6
    effort: high
prompts:
  - id: baseline
  - id: informed
  - id: documented
  - id: explicit
environments:
  - name: host
tests:
  - name: installation-check-passes
limits:
  max_time_seconds: 300

The four prompts entries are the install-guidance variants. The tests entries are hidden from the agent. You can see the full experiment schema in the experiment YAML reference.

Advanced run options

ax experiment run validates the experiment, submits it to AX Cloud, and polls until all runs are complete. If you don't specify any flags in the run command, all possible combinations of prompts, agents and models, as declared in your YAML file, will be computed and queued for execution.

Running specific prompts, agents, and models

You can alternatively select specific prompts, agents, and models to run by adding --prompt, --agent, and --model flags.

ax experiment run my-first-experiment.yaml --prompt baseline --prompt explicit

Running at scale

You should aim to run each variant of your experiment enough times to get a statistically significant result. The more runs you have, the more confident you can be in your results.

Repeat each variant with the --repeat flag. AX Cloud dispatches the runs in parallel, so there is no separate concurrency flag:

ax experiment run my-first-experiment.yaml --repeat 3

The create prompt above continues into analysis after the run finishes: debug incomplete runs, confirm the completed set is large enough for statistically significant insights, then answer install-specific questions and save an insight with ax learn publish-insight.