Getting started
Install the CLI and run your first AX experiment
Fiveonefour is an experimentation platform for understanding and improving a product's agent experience (AX).
Follow these two steps to get up and running:
- Install Fiveonefour's CLI
- Create and run your first experiment to simulate how coding agents use your product and derive insights on how to improve your AX.
We recommend your proceed with your coding agent of choice, using the prompts provided below. But you can always run the CLI by hand if you prefer.
Fiveonefour is currently available in Research Preview. Experiments run on AX Cloud, which provides managed infrastructure, agents, and data, so you need an account to run them. You'll start on the Basic plan (free, no card required), which includes $25 in usage credits that reset every month, so you can start running experiments in parallel in the cloud, without any other setup.
Prefer to work from Slack? Add AX to your Slack workspace and ask @ax about your experiments from any channel it's in.
Not ready to sign up? Run ax auth signup --temp-account to start with a temporary account that lasts 24 hours, then claim it to keep your work.
Install the CLI
Install Fiveonefour's ax CLI with the Bash installer or Homebrew.
Use these two commands to install and authenticate the ax CLI:curl -fsSL https://dl.514.ax/install.sh | bash -s -- axax auth login
This will open a browser for me to authenticate in. Direct me to do so. If you're unable to open a browser for me, then run:ax auth login --no-browserto print the URL and ask me to navigate to that URL to authenticate.
Alternatively ask me to create an API access token at https://app.514.ax/account/api-keys and authenticate the CLI by running:ax auth login --token <<token>>which does not require any browser interaction. Warn me not to enter the raw token in chat with you, as this is a security vulnerability.
If I don't have an account yet and want to try ax first, offer to create a temporary account instead:ax auth signup --temp-accountCreating one accepts the Terms of Use, so ask me before you run it. The account lasts 24 hours unless I run `ax auth signup --claim-account` and finish signing up in the browser to keep it.Create and run your first experiment
If you're not sure where to start, use the Test my product's CLI installation intro experiment. This will test how effective coding agents can install your product's CLI (or another CLI you choose), and should give you a feel for how Fiveonefour can help you improve your product.
If you have a research question in mind to test already, or want your agent to walk you through exploring other options, just use the "Design my own experiment" prompt instead.
I'm a first-time ax user. Help me create and run my first experiment.
Use the `ax` CLI for every step. Coach me through the first-experimentflow, and ask me about MY CLI / product. Do not assume AX itself is thetarget, and do not invent flag values.
1. Run `ax --version`. If it fails, show the install command (`curl -fsSL https://dl.514.ax/install.sh | bash -s -- ax` or `brew install 514-labs/tap/ax`), ask me to confirm, then install and re-check. Do not stop or send me elsewhere. Experiments run on AX Cloud, so `ax auth status` must succeed. If it does not, help me sign in with `ax auth login` (or `AX_API_KEY`). If I have no account yet, offer `ax auth signup --temp-account` instead. Creating one accepts the Terms of Use, so ask me before you run it.
2. Prefer the `cli-install` template. First ask whether I have my own CLI to test. If I do, ask me for anything you cannot infer: - experiment name (default: my-first-experiment) - target CLI binary name (`--cli`) - short product description (`--target-description`) - install docs URL (`--install-docs`) - install command (`--install-command`) - smoke command (`--smoke-command`) - optional smoke-output text (`--smoke-output-contains`)
If I do not have a CLI of my own, fall back to testing the `ax` CLI with: - `--cli ax` - `--target-description 'the Fiveonefour CLI for running agent evaluation experiments'` - `--install-docs https://docs.514.ax/cli/installation` - `--install-command 'curl -fsSL https://dl.514.ax/install.sh | bash -s -- ax'` - `--smoke-command 'ax --version'` - `--smoke-output-contains 'ax '`
3. Scaffold with `ax experiment create <NAME> --template cli-install` and every required flag filled in from my answers.
4. Validate with `ax experiment validate <NAME>.yaml` and fix any errors.
5. Run on AX Cloud with `ax experiment run <NAME>.yaml`.
6. When the run finishes, give me the AX Cloud URL.
7. Analyze with `ax learn analyze-results`. Start from `ax run list --experiment <NAME>` and pick a mode: - Debug (failed, errored, or incomplete runs): use `ax run view` / `ax run query` to find the blocker (auth, model access, sandbox, YAML, environment). Fix it and re-run until runs complete. Do not claim product insights yet. - Readiness (runs completed): say whether we have enough completed runs across variants (prompt levels, agents/models, environments) for statistically significant insights. If not, recommend how to get there (`--repeat` or a broader matrix) and re-run first. Do not suggest `--parallel` (it does not exist). - Analyze (analysis-ready only): follow the analyze-results workflow. Ground every claim in query or transcript evidence and cite run counts.
8. When analysis-ready, answer these questions in order: - Does more install guidance help? Compare testPassRate across the prompt variants (baseline, informed, documented, explicit). Call out differences that hold across repeats vs ones within noise. - Are results consistent across agents and models, or does one agent or model drive the trend? Slice testPassRate by model and agent. - What does success cost? Compare cost, tokens, and wallClockTime by prompt variant. - How often does `installation-check-passes` fail, and does failure rate differ by prompt variant? - For failed product outcomes (completed runs that fail tests): did the agent fail to install the target CLI itself, or get stuck on dependencies around it (package managers, PATH, permissions, network)? Open transcripts and classify each failure. - What other failure reasons show up repeatedly? Corroborate the numbers with the agent's reasoning and list the top 2-3 patterns with run ids.
9. Finish with the single most actionable product fix the data supports. Then follow `ax learn publish-insight`: save the most useful query with `--save`, reopen it with `ax insight view`, and ask whether I want to share it with `ax insight share`.