Troubleshooting

This page covers failures that the current CLI can produce. For command syntax, compare against ax --help and the command-specific --help output.

ax: command not found after install

The installer writes the wrapper to ~/.fiveonefour/bin/ax by default. If that directory is not already on PATH, the installer prints the export to add and, for writable zsh or bash config directories, appends an installer-managed entry.

Open a new shell, reload your shell config, or add the path manually:

export PATH="$HOME/.fiveonefour/bin:$PATH"
ax --help

If you installed with FIVEONEFOUR_HOME=<path>, use <path>/bin instead.

Install download fails

The installer requires curl and downloads the version manifest, release archive (or legacy bare binary), checksum, and optional signature from https://download.514.ax with curl -fsSL. If the manifest download fails, the script prints the failed manifest URL and the underlying curl error.

Check that curl is installed and that your network, proxy, or firewall can reach the download host:

curl -fsSL https://download.514.ax/stable/index.json >/dev/null

Then rerun the installer. If the archive or checksum download fails after the manifest succeeds, retry the install and keep the printed download URL for debugging.

Unsupported OS or architecture

The installer has prebuilt targets for:

• macOS arm64 and x86_64
• Linux arm64/aarch64 and x86_64/amd64

On any other uname -s and uname -m pair, it exits with unsupported platform: <os>/<arch>. Use one of the supported platforms for the hosted installer.

Uninstall leaves a PATH entry

By default, uninstall removes the Fiveonefour wrapper and cached versions but leaves shell startup files untouched. To remove exact installer-managed PATH entries from ~/.zshrc, ~/.bashrc, ~/.bash_profile, and ~/.profile, pass --remove-path:

bash <(curl -fsSL https://download.514.ax/uninstall.sh) --yes --remove-path

The script only removes lines that exactly match the installer-managed ~/.fiveonefour/bin export, plus the adjacent # Fiveonefour's CLI comment when present. Remove custom PATH edits manually.

Uninstall skips the ax binary

The uninstall script only removes ~/.fiveonefour/bin/ax automatically when it looks like the hosted installer wrapper. If that file is a custom binary, symlink, or a Cargo-installed ax, uninstall prints that it was skipped.

Pass --force only when you intentionally want to remove that file anyway:

bash <(curl -fsSL https://download.514.ax/uninstall.sh) --yes --force

Cargo-installed binaries outside ~/.fiveonefour/bin are not removed by the hosted uninstall script.

Non-interactive uninstall fails

Interactive uninstall prompts before removing files. In a non-interactive shell, the script refuses to continue unless you pass --yes:

bash <(curl -fsSL https://download.514.ax/uninstall.sh) --yes

Combine it with other flags when needed, for example --yes --remove-path.

Preview uninstall changes

Use --dry-run to print what uninstall would remove or update without changing files. Dry runs do not require confirmation and print Dry run: no files will be changed.

bash <(curl -fsSL https://download.514.ax/uninstall.sh) --dry-run --remove-path

No command was provided

ax requires a subcommand. Running it with no command exits non-zero and prints usage.

ax --help

Experiment YAML does not validate

ax experiment validate <EXPERIMENT> prints Validated <EXPERIMENT> and a suggested ax experiment run <EXPERIMENT> command on success. On schema failure, it prints one error line and exits non-zero. It may also print non-fatal warning[<rule>] lint messages to stderr while still exiting zero.

Common schema issues include:

• schema_version is not 2.
• id, prompt ids, environment or product names, or test names are not kebab-case.
• name, a prompt's text, or a test script is empty.
• tests has no entries, still uses the removed tests.application / tests.introspection buckets instead of a flat list, or repeats a test name.
• limits.max_time_seconds is zero or invalid.
• The YAML contains an unknown field, including the removed limits.max_turns, limits.max_cost_usd, and secrets keys.
• The agents and models lists leave no pair that can run (no listed agent offers any listed model). An agent or model with no pair while others run is only skipped, with a warning.

max_time_seconds must be a positive integer and is enforced by Fiveonefour as a wall-clock timeout. The former max_turns and max_cost_usd caps were removed from experiment YAML; validation rejects them like any other unknown field.

When you are signed in, validation also fails if limits.max_time_seconds is above your plan's cloud-run cap, naming the cap your plan allows and the value to lower it to. Signed out or with the platform unreachable, it prints one advisory line on stderr and still exits zero, and the same value is caught at submit instead. See Cloud run time limits.

Run validation before starting containers:

ax experiment validate experiment.yaml

If you see warning[polls-for-file-in-setup], check any environment or product setup script that waits for a /workspace/... path. Variants are isolated, so each variant must create or fetch its own setup files.

If you see warning[grading-details-in-prompt], a prompt names a test or echoes its description. Describe a realistic job in the prompt and keep scoring in tests. When you are signed in, the AI review also flags language that tells the agent it is being evaluated.

A run used the wrong variant set

ax experiment run --variant <ID> only accepts ids from the resolved variant matrix (preview it with ax experiment variants experiment.yaml). If a requested variant does not exist, the CLI prints the known variant ids and exits with code 2.

Preview the planned variants without submitting a run:

ax experiment variants experiment.yaml

A submit or query is refused with V1_DATA_PIPELINE_DEPRECATED

Every run records its results in the current session_data_v1 family. An experiment id whose earlier runs landed in the retired V1 family keeps that history, so a new submit, retest, or push against it is refused, and so is any query, export, or share of its results:

V1_DATA_PIPELINE_DEPRECATED: Fork the experiment onto V2 and try again.
Contact support for historical-data recovery or a free V2 upgrade and rerun.

The experiment and its runs stay listed, and ax experiment view, ax run view, the API, and MCP return the same message as a deprecation field on their metadata. To run the definition again, fork it onto the current family and submit the copy:

ax experiment fork my-experiment
ax experiment run my-experiment-v2

See experiment fork. The --flag pipeline-v1 run flag names the retired family and is refused the same way. To recover the historical results themselves, contact support.

An experiment whose earlier runs landed in both families before this rule existed is reported as having "run history in more than one capture family" and cannot accept new runs at all. Fork it, or submit under a fresh experiment id.

A run failed with setup_failed

A variant rolls up as setup_failed when its setup script fails: setup-checks, the agent, and tests never run. setup runs under set -euo pipefail, so it stops at the first failing command, not just when the last line exits non-zero. This is distinct from driver_error, which now means an actual harness/driver fault rather than a mundane setup bug.

The script's stdout + stderr (tail) is included in the ax experiment run failure output, and the full output is saved to the variant's setup.log artifact. Read it with ax run view <RUN_ID> --setup-log, or in the Setup log card on the run page. See run view. A common cause is a missing command, e.g. a package that installs a daemon but not the client the next line calls, so that line exits 127.

Iterate on the setup script with a setup-only run. It writes the same setup.log, runs each distinct setup once, and skips the agent and tests, so there is no model spend:

ax experiment run experiment.yaml --setup-only

Read each setup's log and setup-check results with ax run view <RUN_ID> --setup-log.

A run timed out at the plan limit

A cloud run ends errored with exit_reason: timeout when the agent is still working at the end of its time budget. The budget is the lower of the experiment's limits.max_time_seconds and your plan's cloud-run cap, so a run can stop well before the value the YAML asked for. Signed in, ax experiment validate flags an over-cap limits.max_time_seconds before you submit.

ax run view names which of the two applied. If the plan cap is what stopped the run, upgrade for a longer cap. If the experiment's own limit is what stopped the run, raise it and run again. See Cloud run time limits for the per-plan caps and Runs that reach a time limit for what each surface shows.

--repeat 0 fails

--repeat must be at least 1.

ax experiment run experiment.yaml --repeat 3

A referenced secret can't be resolved

An ax://secrets/<slug> reference resolves from your org's secret store on every run. If the slug doesn't exist in the org, the run fails fast at preflight before any sandbox starts.

Store the value, then re-run:

ax secret set prod-gh        # prompts for the value
ax secret list               # confirm the slug exists (values are never shown)

A name must match ^[A-Z_][A-Z0-9_]*$. Harness-managed names cannot be used, including ANTHROPIC_API_KEY, ANTHROPIC_BASE_URL, OPENAI_API_KEY, OPENAI_BASE_URL, CURSOR_API_KEY, MODEL, MAX_TURNS, IS_SANDBOX, TRACEPARENT, and names starting with AX_, AXP_, CLAUDE_CODE_, CODEX_, CURSOR_, or OTEL_.

Model credentials are missing

ax experiment run uses platform-supplied model access, so it needs no host model key: sign in with ax auth login or set AX_API_KEY. Provider keys such as ANTHROPIC_API_KEY, OPENAI_API_KEY, and CURSOR_API_KEY are harness-managed, so do not list them under environment_variables. To bill model usage to your own provider account, configure an organization provider in Model providers.

ax auth login
ax experiment run experiment.yaml

ax run list shows no runs

ax run list shows platform runs in the active org. If no runs match, it prints an empty-list message. The default hides canceled runs, so pass --status canceled to see those, and check the active org with ax auth status.

Runs are queried in place with ax run query <RUN_ID> sql "<SQL>"; there is no offline copy of results.

ax experiment run says you're not signed in

ax experiment run submits to the platform, so it needs saved credentials. If you see an authentication error, sign in (or set AX_API_KEY in CI):

ax auth login
ax auth status

ax experiment run and secrets

Platform runs support secrets: reference an org secret from environment_variables as ax://secrets/<slug> and dispatch resolves it server-side from your org's secret store. (Earlier releases rejected secret-bearing experiments at submit; that restriction is gone.) Store the value first with ax secret set <slug>; a reference to a slug that doesn't exist in the org fails at submit. The removed top-level secrets: field is rejected; use environment_variables + ax://secrets/<slug> instead.

See Runs and Secrets.

Cancel a platform run

ax experiment run prints a run request id. Cancel queued or in-flight jobs with it:

ax run cancel <run-request-id>

Cancel only specific jobs with --run <run-id> (comma-separated); the default cancels every job in the request.

Not logged in

Platform commands require credentials saved by ax auth login. If credentials are missing, run:

ax auth login
ax auth status

ax auth status round-trips through the platform API and prints the resolved user and active org. Use ax auth orgs to list available orgs, and ax auth orgs --switch org_... to change the active CLI org.

The credentials file lives at ~/.fiveonefour/axp/credentials.toml. AX_API_KEY can override the saved API key.

Platform target is wrong

Auth, run, and query requests go to the Fiveonefour platform by default. If the Fiveonefour team gives you a different platform URL, set AX_PLATFORM_URL:

AX_PLATFORM_URL=https://platform.example.com ax auth status

Active org is wrong

Platform commands use the active CLI org by default. Check it with:

ax auth status
ax auth orgs

Switch the active org with:

ax auth orgs --switch org_...

For a one-off command without changing the active org, pass --org:

ax run list --org org_...

If a run exists in another org that your API key can access, the platform error includes this hint. If the run is not in an accessible org, the error does not reveal where it exists.

Platform query says Not enough privileges

If ax results get, raw SQL via ax results query, or a pasted WITH scoped_runs AS (…) statement fails with Not enough privileges on tables outside the scope's public catalog, prefer the modeled path:

ax experiment query <EXPERIMENT_ID> --metric testPassRate --group-by variant

Modeled ax experiment query compiles and runs server-side. Pasting the CLI's compiled SQL into a reader-scoped SQL surface can hit ClickHouse grants the reader role does not have. For one composite run, use ax run query <RUN_ID> sql … instead of replaying experiment-scoped SQL.

If a table name is rejected, inspect the selected scope instead of guessing a physical name:

ax run query <RUN_ID> sql --tables

Remote scopes expose run_analysis, test_analysis, measurements, events, normalized_events, and tool_interactions; sql "SHOW TABLES" lists exactly these. SQL that names an internal table is refused, and the error names the table to read instead. Older table names from the retired V1 pipeline (runs, tests, tool_calls, agent_events) are not queryable; rewrite the query against test_analysis and events.

Platform query fails on a resource limit

Authenticated SQL has no scan limits, so a query that fails on resources is telling you about real capacity, not a quota. SETTINGS clauses are rejected; the fix is a cheaper query shape:

• Keep aggregation in SQL. Compute per-run flags in a subquery (max(match(payload, ...)) GROUP BY run_id), join them back to measurements, and return grouped counts and rates in one statement. See query limits for a worked example.
• Narrow the scan. Filter by run_id, or run through ax experiment query <EXPERIMENT_ID> sql ... so the experiment scope is applied server-side.
• Do not paginate raw rows out for local processing; one aggregate query beats many raw reads on both speed and cost.

A query timed out error means the execution ceiling elapsed; reduce the number of match() expressions or narrow the scan the same way. A memory or too many concurrent queries error means the analytical store is under pressure from concurrent work; retry shortly. A query read budget exceeded error only occurs on publicly shared insight pages, which keep small scan limits.