Getting started
Install the CLI and run your first AX experiment
Fiveonefour is an experimentation platform for understanding and improving a product's agent experience (AX).
Follow these two steps to get up and running:
- Install Fiveonefour's CLI
- Create and run your first experiment to simulate how coding agents use your product and derive insights on how to improve your AX.
We recommend your proceed with your coding agent of choice, using the prompts provided below. But you can always run the CLI by hand if you prefer.
Fiveonefour is currently available in Research Preview, and you can use the Fiveonefour CLI without signing up for an account - just skip the ax auth commands. You'll need to run your experiments locally, which requires Docker and providing your own LLM provider keys.
If you do create an account, you'll have access to AX Cloud, with managed infrastructure, agents, and data to enable your experimentation quickly, easily, and at scale. You'll start on the Basic plan (free, no card required), which includes $25 in usage credits that reset every month, so you can start running experiments in parallel in the cloud, without any other local setup.
Install the CLI
Install Fiveonefour's ax CLI with the Bash installer or Homebrew.
Use these two commands to install and authenticate the ax CLI:curl -fsSL https://dl.514.ax/install.sh | bash -s -- axax auth login
This will open a browser for me to authenticate in. Direct me to do so. If you're unable to open a browser for me, then run:ax auth login --no-browserto print the URL and ask me to navigate to that URL to authenticate.
Alternatively ask me to create an API access token at https://app.514.ax/account/api-keys and authenticate the CLI by running:ax auth login --token <<token>>which does not require any browser interaction. Warn me not to enter the raw token in chat with you, as this is a security vulnerability.
If I want to try Fiveonefour locally without an account, verify Docker is running and skip signup. Local runs still need managed model access (sign in or AX_API_KEY) or my own provider key with --local-model-keys.Create and run your first experiment
If you're not sure where to start, use the Test my product's CLI installation intro experiment. This will test how effective coding agents can install your product's CLI (or another CLI you choose), and should give you a feel for how Fiveonefour can help you improve your product.
If you have a research question in mind to test already, or want your agent to walk you through exploring other options, just use the "Design my own experiment" prompt instead.
I'm a first-time ax user. Help me create and run my first experiment.
Use the `ax` CLI for every step. Coach me through the first-experimentflow, and ask me about MY CLI / product. Do not assume AX itself is thetarget, and do not invent flag values.
1. Run `ax --version`. If it fails, show the install command (`curl -fsSL https://dl.514.ax/install.sh | bash -s -- ax` or `brew install 514-labs/tap/ax`), ask me to confirm, then install and re-check. Do not stop or send me elsewhere. Signing in is optional. Ask whether I want an AX Cloud run or `--local`. AX Cloud runs need `ax auth status` to succeed. Local runs need Docker; managed model access still uses auth or `AX_API_KEY`, otherwise `--local-model-keys` with my provider key.
2. Prefer the `cli-install` template. First ask whether I have my own CLI to test. If I do, ask me for anything you cannot infer: - experiment name (default: my-first-experiment) - target CLI binary name (`--cli`) - short product description (`--target-description`) - install docs URL (`--install-docs`) - install command (`--install-command`) - smoke command (`--smoke-command`) - optional smoke-output text (`--smoke-output-contains`) - AX Cloud run vs `--local`
If I do not have a CLI of my own, fall back to testing the `ax` CLI with: - `--cli ax` - `--target-description 'the Fiveonefour CLI for running agent evaluation experiments'` - `--install-docs https://docs.514.ax/cli/installation` - `--install-command 'curl -fsSL https://dl.514.ax/install.sh | bash -s -- ax'` - `--smoke-command 'ax --version'` - `--smoke-output-contains 'ax '`
3. Scaffold with `ax experiment create <NAME> --template cli-install` and every required flag filled in from my answers.
4. Validate with `ax experiment validate <NAME>.yaml` and fix any errors.
5. Run with `ax experiment run <NAME>.yaml` for AX Cloud, or `ax experiment run <NAME>.yaml --local --watch` for local.
6. When the run finishes, give me the AX Cloud URL or local summary. For local runs, mention `ax run upload <run-id>` if I later want results on AX Cloud.
7. Analyze with `ax learn analyze-results`. Start from `ax run list --experiment <NAME>` and pick a mode: - Debug (failed, errored, or incomplete runs): use `ax run view` / `ax run query` to find the blocker (auth, Docker/local vs AX Cloud, model keys, sandbox, YAML, environment). Fix it and re-run until runs complete. Do not claim product insights yet. - Readiness (runs completed): say whether we have enough completed runs across variants (prompt levels, agents/models, environments) for statistically significant insights. If not, recommend how to get there (`--repeat`, a broader matrix, or for local runs `--local --jobs N`) and re-run first. Do not suggest `--parallel` (it does not exist). - Analyze (analysis-ready only): follow the analyze-results workflow. Ground every claim in query or transcript evidence and cite run counts.
8. When analysis-ready, answer these questions in order: - Does more install guidance help? Compare testPassRate across the prompt variants (baseline, informed, documented, explicit). Call out differences that hold across repeats vs ones within noise. - Are results consistent across agents and models, or does one agent or model drive the trend? Slice testPassRate by model and agent. - What does success cost? Compare cost, tokens, and wallClockTime by prompt variant. - How often does `installation-check-passes` fail, and does failure rate differ by prompt variant? - For failed product outcomes (completed runs that fail tests): did the agent fail to install the target CLI itself, or get stuck on dependencies around it (package managers, PATH, permissions, network)? Open transcripts and classify each failure. - What other failure reasons show up repeatedly? Corroborate the numbers with the agent's reasoning and list the top 2-3 patterns with run ids.
9. Finish with the single most actionable product fix the data supports. Then follow `ax learn publish-insight`: save the most useful query with `--save`, reopen it with `ax insight view`, and ask whether I want to share it with `ax insight share`.