Troubleshooting

This page covers failures that the current CLI can produce. For command syntax, compare against ax --help and the command-specific --help output.

ax: command not found after install

The installer writes the wrapper to ~/.fiveonefour/bin/ax by default. If that directory is not already on PATH, the installer prints the export to add and, for writable zsh or bash config directories, appends an installer-managed entry.

Open a new shell, reload your shell config, or add the path manually:

export PATH="$HOME/.fiveonefour/bin:$PATH"
ax --help

If you installed with FIVEONEFOUR_HOME=<path>, use <path>/bin instead.

Install download fails

The installer requires curl and downloads the version manifest, release archive (or legacy bare binary), checksum, and optional signature from https://download.514.ax with curl -fsSL. If the manifest download fails, the script prints the failed manifest URL and the underlying curl error.

Check that curl is installed and that your network, proxy, or firewall can reach the download host:

curl -fsSL https://download.514.ax/stable/index.json >/dev/null

Then rerun the installer. If the archive or checksum download fails after the manifest succeeds, retry the install and keep the printed download URL for debugging.

Unsupported OS or architecture

The installer has prebuilt targets for:

• macOS arm64 and x86_64
• Linux arm64/aarch64 and x86_64/amd64

On any other uname -s and uname -m pair, it exits with unsupported platform: <os>/<arch>. Use one of the supported platforms for the hosted installer.

Uninstall leaves a PATH entry

By default, uninstall removes the Fiveonefour wrapper and cached versions but leaves shell startup files untouched. To remove exact installer-managed PATH entries from ~/.zshrc, ~/.bashrc, ~/.bash_profile, and ~/.profile, pass --remove-path:

bash <(curl -fsSL https://download.514.ax/uninstall.sh) --yes --remove-path

The script only removes lines that exactly match the installer-managed ~/.fiveonefour/bin export, plus the adjacent # Fiveonefour's CLI comment when present. Remove custom PATH edits manually.

Uninstall skips the ax binary

The uninstall script only removes ~/.fiveonefour/bin/ax automatically when it looks like the hosted installer wrapper. If that file is a custom binary, symlink, or a Cargo-installed ax, uninstall prints that it was skipped.

Pass --force only when you intentionally want to remove that file anyway:

bash <(curl -fsSL https://download.514.ax/uninstall.sh) --yes --force

Cargo-installed binaries outside ~/.fiveonefour/bin are not removed by the hosted uninstall script.

Non-interactive uninstall fails

Interactive uninstall prompts before removing files. In a non-interactive shell, the script refuses to continue unless you pass --yes:

bash <(curl -fsSL https://download.514.ax/uninstall.sh) --yes

Combine it with other flags when needed, for example --yes --remove-path.

Preview uninstall changes

Use --dry-run to print what uninstall would remove or update without changing files. Dry runs do not require confirmation and print Dry run: no files will be changed.

bash <(curl -fsSL https://download.514.ax/uninstall.sh) --dry-run --remove-path

No command was provided

ax requires a subcommand. Running it with no command exits non-zero and prints usage.

ax --help

Experiment YAML does not validate

ax experiment validate <EXPERIMENT> prints Validated <EXPERIMENT> and a suggested ax experiment run <EXPERIMENT> command on success. On schema failure, it prints one error line and exits non-zero. It may also print non-fatal warning[<rule>] lint messages to stderr while still exiting zero.

Common schema issues include:

• schema_version is not 2.
• id, prompt ids, environment or product names, or test names are not kebab-case.
• name, a prompt's text, or a test script is empty.
• tests has no entries, still uses the removed tests.application / tests.introspection buckets instead of a flat list, or repeats a test name.
• limits.max_time_seconds is zero or invalid.
• The YAML contains an unknown field, including the removed limits.max_turns, limits.max_cost_usd, and secrets keys.
• The agents and models lists leave some entry with no compatible pair (every pairing it appears in is pruned).

max_time_seconds must be a positive integer and is enforced by Fiveonefour as a wall-clock timeout. The former max_turns and max_cost_usd caps were removed from experiment YAML; validation rejects them like any other unknown field.

Run validation before starting containers:

ax experiment validate experiment.yaml

If you see warning[polls-for-file-in-setup], check any environment or product setup script that waits for a /workspace/... path. Variants are isolated, so each variant must create or fetch its own setup files.

If you see warning[grading-details-in-prompt], a prompt names a test or echoes its description. Describe a realistic job in the prompt and keep scoring in tests. When you are signed in, the AI review also flags language that tells the agent it is being evaluated.

A run used the wrong variant set

ax experiment run --variant <ID> only accepts ids from the resolved variant matrix (preview it with ax experiment variants experiment.yaml). If a requested variant does not exist, the CLI prints the known variant ids and exits with code 2.

Use a local dry run to inspect the planned variants without creating containers or making API calls:

ax experiment run experiment.yaml --local --resolve-variants

A submit or query is refused with V1_DATA_PIPELINE_DEPRECATED

Every run records its results in the current session_data_v1 family. An experiment id whose earlier runs landed in the retired V1 family keeps that history, so a new submit, retest, push, or upload against it is refused, and so is any query, export, or share of its results:

V1_DATA_PIPELINE_DEPRECATED: Fork the experiment onto V2 and try again.
Contact support for historical-data recovery or a free V2 upgrade and rerun.

The experiment and its runs stay listed, and ax experiment view, ax run view, the API, and MCP return the same message as a deprecation field on their metadata. To run the definition again, fork it onto the current family and submit the copy:

ax experiment fork my-experiment
ax experiment run my-experiment-v2

See experiment fork. The --flag pipeline-v1 run flag names the retired family and is refused the same way. To recover the historical results themselves, contact support.

An experiment whose earlier runs landed in both families before this rule existed is reported as having "run history in more than one capture family" and cannot accept new runs at all. Fork it, or submit under a fresh experiment id.

A run failed with setup_failed

A variant rolls up as setup_failed when its setup script exits non-zero: setup-checks, the agent, and tests never run. This is distinct from driver_error, which now means an actual harness/driver fault rather than a mundane setup bug.

The script's stdout + stderr (tail) is included in the ax experiment run failure output, and the full output is saved to the variant's setup.log artifact (visible on the run page, or under .axp/runs/<RUN_REQUEST_ID>/ for a local run). A common cause is a missing command, e.g. a package that installs a daemon but not the client the next line calls, so that line exits 127.

Iterate on the setup script with a local run, which writes the same setup.log:

ax experiment run experiment.yaml --local

--repeat 0 fails

--repeat must be at least 1.

ax experiment run experiment.yaml --repeat 3

A referenced secret can't be resolved

An ax://secrets/<slug> reference resolves from your org's secret store on both run paths. If the slug doesn't exist in the org (and, for local runs, no local source supplies the bound name), the run fails fast at preflight before any container starts.

Store the value, then re-run:

ax secret set prod-gh        # prompts for the value
ax secret list               # confirm the slug exists (values are never shown)

For a local run you can also override the bound name from a local source instead of the store (lowest → highest precedence): auto-loaded ./.env, --env-file, host env, then --env:

echo "GITHUB_TOKEN=..." > .env                          # auto-loaded
ax experiment run experiment.yaml --local --env-file .env.local      # explicit file
GITHUB_TOKEN=... ax experiment run experiment.yaml --local           # host env
ax experiment run experiment.yaml --local --env GITHUB_TOKEN=...     # highest precedence

A name must match ^[A-Z_][A-Z0-9_]*$. Harness-managed names cannot be used, including ANTHROPIC_API_KEY, ANTHROPIC_BASE_URL, OPENAI_API_KEY, OPENAI_BASE_URL, CURSOR_API_KEY, MODEL, MAX_TURNS, IS_SANDBOX, TRACEPARENT, and names starting with AX_, AXP_, CLAUDE_CODE_, CODEX_, CURSOR_, or OTEL_.

Model credentials are missing

ax experiment run and default ax experiment run --local use platform-supplied model access, so they need no host model key: sign in with ax auth login or set AX_API_KEY. Pass --local-model-keys only when you want host provider keys; then each selected agent needs a credential: ANTHROPIC_API_KEY for Claude, OPENAI_API_KEY for Codex, or CURSOR_API_KEY for Cursor. These are harness-managed, so do not list them under environment_variables.

ax auth login
ax experiment run experiment.yaml --local

export ANTHROPIC_API_KEY=...
ax experiment run experiment.yaml --local --local-model-keys

export OPENAI_API_KEY=...
ax experiment run codex-experiment.yaml --local --local-model-keys

A run reports model transport errors

A non-zero agent.model_transport_errors value in run.json means the agent logged a recognized model connection or stream failure. The run may have continued over a fallback connection or completed fewer useful turns, so do not treat its test result as agent quality without checking the evidence. Local runs print a warning when they detect this condition.

Detection is best effort. A value of 0 means no recognized error was detected, not that no transport error occurred.

No runs are listed

ax run list shows local runs from ./.axp/runs/ and, when signed in, remote platform runs. If no local runs exist in the current directory and no remote runs match, it prints an empty-list message.

Run a local experiment from the project directory that should own the results:

ax experiment run experiment.yaml --local
ax run list

ax run list shows no runs

Use ax run list --scope local to check only ./.axp/runs/. If it is empty, run a local experiment first. Inspect a request's artifacts directly under .axp/runs/<RUN_REQUEST_ID>/, or query one execution with ax run query '<RUN_ID>' sql "<SQL>" --local.

Platform runs are queried in place with ax run query <RUN_ID> sql "<SQL>"; there is no offline copy of platform results.

Docker image pull fails

ax experiment run --local pulls axp-base from Docker Hub (514labs/axp-base) on first use. If the pull fails, check Docker is running and Docker Hub is reachable; you can pre-pull manually:

docker pull 514labs/axp-base:latest

Use ax experiment run experiment.yaml --local --resolve-variants when you only want experiment validation without touching containers. (Platform runs with ax experiment run need no local Docker.)

ax experiment run says you're not signed in

ax experiment run submits to the platform, so it needs saved credentials. If you see an authentication error, sign in (or set AX_API_KEY in CI):

ax auth login
ax auth status

To run without platform dispatch, use ax experiment run <path> --local (Docker; default managed model access still needs ax auth login / AX_API_KEY, or pass --local-model-keys with host provider keys).

ax experiment run and secrets

Platform runs support secrets: reference an org secret from environment_variables as ax://secrets/<slug> and dispatch resolves it server-side from your org's secret store. (Earlier releases rejected secret-bearing experiments at submit; that restriction is gone.) Store the value first with ax secret set <slug>; a reference to a slug that doesn't exist in the org fails at submit. The removed top-level secrets: field is rejected; use environment_variables + ax://secrets/<slug> instead.

See Runs and Secrets.

Cancel a platform run

ax experiment run prints a run request id. Cancel queued or in-flight jobs with it:

ax run cancel <run-request-id>

Cancel only specific jobs with --run <run-id> (comma-separated); the default cancels every job in the request.

Not logged in for platform upload

ax run upload requires credentials saved by ax auth login. If credentials are missing, run:

ax auth login
ax auth status

ax auth status round-trips through the platform API and prints the resolved user and active org. Use ax auth orgs to list available orgs, and ax auth orgs --switch org_... to change the active CLI org.

The credentials file lives at ~/.fiveonefour/axp/credentials.toml. AX_API_KEY can override the saved API key.

Upload target is wrong

Auth, upload, and query requests go to the Fiveonefour platform by default. If the Fiveonefour team gives you a different platform URL, set AX_PLATFORM_URL:

AX_PLATFORM_URL=https://platform.example.com ax run upload my-experiment-01HX...

Active org is wrong

Platform commands use the active CLI org by default. Check it with:

ax auth status
ax auth orgs

Switch the active org with:

ax auth orgs --switch org_...

For a one-off upload without changing the active org, pass --org:

ax run upload my-experiment-01HX... --org org_...

If a run exists in another org that your API key can access, the platform error includes this hint. If the run is not in an accessible org, the error does not reveal where it exists.

Run was already uploaded

If the server reports that the run already exists, pass --force. The command retries registration and publication without replacing the run's immutable capture:

ax run upload my-experiment-01HX... --force

Platform query says Not enough privileges

If ax results get, raw SQL via ax results query, or a pasted WITH scoped_runs AS (…) statement fails with Not enough privileges on tables outside the scope's public catalog, prefer the modeled path:

ax experiment query <EXPERIMENT_ID> --metric testPassRate --group-by variant

Modeled ax experiment query compiles and runs server-side. Pasting the CLI's compiled SQL into a reader-scoped SQL surface can hit ClickHouse grants the reader role does not have. For one composite run, use ax run query <RUN_ID> sql … instead of replaying experiment-scoped SQL.

If a table name is rejected, inspect the selected scope instead of guessing a physical name:

ax run query <RUN_ID> sql --tables

Remote scopes expose run_analysis, test_analysis, measurements, events, and the typed sd_current_* projections. Older table names from the retired V1 pipeline (runs, tests, tool_calls, agent_events) are not queryable; rewrite the query against test_analysis and events.

Platform query fails on a resource limit

Authenticated SQL has no scan limits, so a query that fails on resources is telling you about real capacity, not a quota. SETTINGS clauses are rejected; the fix is a cheaper query shape:

• Keep aggregation in SQL. Compute per-run flags in a subquery (max(match(payload, ...)) GROUP BY run_id), join them back to measurements, and return grouped counts and rates in one statement. See query limits for a worked example.
• Narrow the scan. Filter by run_id, or run through ax experiment query <EXPERIMENT_ID> sql ... so the experiment scope is applied server-side.
• Do not paginate raw rows out for local processing; one aggregate query beats many raw reads on both speed and cost.

A query timed out error means the execution ceiling elapsed; reduce the number of match() expressions or narrow the scan the same way. A memory or too many concurrent queries error means the analytical store is under pressure from concurrent work; retry shortly. A query read budget exceeded error only occurs on publicly shared insight pages, which keep small scan limits.

Local query derivation fails

Use ax run query <RUN_ID> sql "<SQL>" --local to derive query files from a raw local run and surface the conversion/query error:

ax run query <RUN_ID> sql "SELECT * FROM runs LIMIT 1" --local

Local queries read a local projection of the run tree (runs, tests, setup_logs, setup_checks, harness_spans, artifacts), not the remote tables. Use SHOW TABLES locally and adapt ClickHouse-specific SQL when needed.

If the run is not local, use ax run list --scope local to find local runs, or query the platform run directly without --local.