Troubleshooting | AX Documentation

This page covers failures that the current CLI can produce. For command syntax, compare against ax --help and the command-specific --help output.

Send a debug bundle

When nothing below explains a run, ax send-debug packages that run, the experiment YAML it used, and host metadata such as CLI version and OS, then uploads it for support.

ax send-debug my-experiment-01HX... --note "failure reproduces on main"
ax send-debug my-experiment-01HX... --dry-run --out debug-bundle.tar.gz
ax send-debug my-experiment-01HX... --no-workspace --no-fs-diff

Omit the run id to use the most recent completed local run. Use --dry-run --out to inspect a bundle before sending it, and --no-workspace / --no-fs-diff when a large workspace makes it unwieldy. Full flags: send-debug.

Bundles include logs verbatim

If a setup script, test, or agent printed a secret, that secret is in the bundle. The CLI warns and prompts before uploading; pass --yes only after you have checked the bundle is safe to send.

`axp: command not found` after install

The installer writes the wrapper to ~/.fiveonefour/bin/ax by default. If that directory is not already on PATH, the installer prints the export to add and, for writable zsh or bash config directories, appends an installer-managed entry.

Open a new shell, reload your shell config, or add the path manually:

export PATH="$HOME/.fiveonefour/bin:$PATH"
ax --help

If you installed with FIVEONEFOUR_HOME=<path>, use <path>/bin instead.

Install download fails

The installer requires curl and downloads the version manifest, release archive (or legacy bare binary), checksum, and optional signature from https://download.514.ax with curl -fsSL. If the manifest download fails, the script prints the failed manifest URL and the underlying curl error.

Check that curl is installed and that your network, proxy, or firewall can reach the download host:

curl -fsSL https://download.514.ax/stable/index.json >/dev/null

Then rerun the installer. If the archive or checksum download fails after the manifest succeeds, retry the install and keep the printed download URL for debugging.

Unsupported OS or architecture

The installer has prebuilt targets for:

macOS arm64 and x86_64
Linux arm64/aarch64 and x86_64/amd64

On any other uname -s and uname -m pair, it exits with unsupported platform: <os>/<arch>. Use one of the supported platforms for the hosted installer.

Uninstall leaves a PATH entry

By default, uninstall removes the AX wrapper and cached versions but leaves shell startup files untouched. To remove exact installer-managed PATH entries from ~/.zshrc, ~/.bashrc, ~/.bash_profile, and ~/.profile, pass --remove-path:

bash <(curl -fsSL https://download.514.ax/uninstall.sh) --yes --remove-path

The script only removes lines that exactly match the installer-managed ~/.fiveonefour/bin export, plus the adjacent # AX CLI comment when present. Remove custom PATH edits manually.

Uninstall skips the `ax` binary

The uninstall script only removes ~/.fiveonefour/bin/ax automatically when it looks like the hosted installer wrapper. If that file is a custom binary, symlink, or a Cargo-installed ax, uninstall prints that it was skipped.

Pass --force only when you intentionally want to remove that file anyway:

bash <(curl -fsSL https://download.514.ax/uninstall.sh) --yes --force

Cargo-installed binaries outside ~/.fiveonefour/bin are not removed by the hosted uninstall script.

Non-interactive uninstall fails

Interactive uninstall prompts before removing files. In a non-interactive shell, the script refuses to continue unless you pass --yes:

bash <(curl -fsSL https://download.514.ax/uninstall.sh) --yes

Combine it with other flags when needed, for example --yes --remove-path.

Preview uninstall changes

Use --dry-run to print what uninstall would remove or update without changing files. Dry runs do not require confirmation and print Dry run: no files will be changed.

bash <(curl -fsSL https://download.514.ax/uninstall.sh) --dry-run --remove-path

No command was provided

ax requires a subcommand. Running it with no command exits non-zero and prints usage.

ax --help

Experiment YAML does not validate

ax experiment validate <EXPERIMENT> prints Validated <EXPERIMENT> and a suggested ax experiment run <EXPERIMENT> command on success. On schema failure, it prints one error line and exits non-zero. It may also print non-fatal warning[<rule>] lint messages to stderr while still exiting zero.

Common schema issues include:

schema_version is not 1.
id, variant IDs, or test names are not kebab-case.
name, flow.prompt, model, variant tags, or test scripts are empty.
matrix.variant has no entries.
tests has no entries.
limits.max_time_seconds is zero or invalid.
The YAML contains an unknown field, including the removed limits.max_turns and limits.max_cost_usd keys.

max_time_seconds must be a positive integer and is enforced by AX as a wall-clock timeout. The former max_turns and max_cost_usd caps were removed from experiment YAML; validation rejects them like any other unknown field.

Run validation before starting containers:

ax experiment validate experiment.yaml

If you see warning[polls-for-file-in-setup], check any variant overrides.setup loop that waits for a /workspace/... path. Variants are isolated, so each variant must create or fetch its own setup files.

A run used the wrong variant set

ax experiment run --variant <ID> only accepts IDs from matrix.variant. If a requested variant does not exist, the CLI prints the known variant IDs and exits with code 2.

Use a local dry run to inspect the planned variants without creating containers or making API calls:

ax experiment run experiment.yaml --local --resolve-variants

An experiment rejects a submit with a different `--flag` set

The first submit of an experiment id fixes which capture family its results are recorded in: --flag native records session_data_v1, and submitting without run flags records legacy. That choice is permanent for the id, so a later submit with the other flag set is rejected:

experiment `my-experiment` is registered as capture family `session_data_v1`,
but this submit uses no run flags, which implies `legacy`.

Resubmitting with the flags that match the registered family always works. To compare the two families against each other, give each arm its own experiment id rather than reusing one:

sed 's/^id: my-experiment$/id: my-experiment-native/' experiment.yaml > native.yaml
ax experiment run native.yaml --flag native

sed 's/^id: my-experiment$/id: my-experiment-production/' experiment.yaml > production.yaml
ax experiment run production.yaml

An experiment whose earlier runs landed in both families before this rule existed is reported as having "run history in more than one capture family" and cannot accept new runs at all. Submit under a fresh experiment id.

A run failed with `setup_failed`

A variant rolls up as setup_failed when its setup script exits non-zero — setup-checks, the agent, and tests never run. This is distinct from driver_error, which now means an actual harness/driver fault rather than a mundane setup bug.

The script's stdout + stderr (tail) is included in the ax experiment run failure output, and the full output is saved to the variant's setup.log artifact (fetch it with ax run download). A common cause is a missing command — e.g. installing docker.io without docker-cli, so a docker --version line exits 127.

Iterate on the setup script with a local run, which writes the same setup.log:

ax experiment run experiment.yaml --local

`--repeat 0` fails

--repeat must be at least 1.

ax experiment run experiment.yaml --repeat 3

A referenced secret can't be resolved

An ax://secrets/<slug> reference resolves from your org's secret store on both run paths. If the slug doesn't exist in the org (and, for local runs, no local source supplies the bound name), the run fails fast at preflight before any container starts.

Store the value, then re-run:

ax secret set prod-gh        # prompts for the value
ax secret list               # confirm the slug exists (values are never shown)

For a local run you can also override the bound name from a local source instead of the store (lowest → highest precedence): auto-loaded ./.env, --env-file, host env, then --env:

echo "GITHUB_TOKEN=..." > .env                          # auto-loaded
ax experiment run experiment.yaml --local --env-file .env.local      # explicit file
GITHUB_TOKEN=... ax experiment run experiment.yaml --local           # host env
ax experiment run experiment.yaml --local --env GITHUB_TOKEN=...     # highest precedence

A name must match ^[A-Z_][A-Z0-9_]*$. Harness-managed names cannot be used, including ANTHROPIC_API_KEY, ANTHROPIC_BASE_URL, OPENAI_API_KEY, OPENAI_BASE_URL, CURSOR_API_KEY, MODEL, MAX_TURNS, IS_SANDBOX, TRACEPARENT, and names starting with AXP_, CLAUDE_CODE_, CODEX_, CURSOR_, or OTEL_.

Model credentials are missing

ax experiment run and default ax experiment run --local use platform-supplied model access, so they need no host model key — sign in with ax auth login or set AX_API_KEY. Pass --local-model-keys only when you want host provider keys; then each selected agent needs a credential — ANTHROPIC_API_KEY for Claude, OPENAI_API_KEY for Codex, or CURSOR_API_KEY for Cursor. These are harness-managed, so do not list them under environment_variables.

ax auth login
ax experiment run experiment.yaml --local

export ANTHROPIC_API_KEY=...
ax experiment run experiment.yaml --local --local-model-keys

export OPENAI_API_KEY=...
ax experiment run codex-experiment.yaml --local --local-model-keys

No runs are listed

ax run list shows local runs from ./.axp/runs/ and, when signed in, remote platform runs. If no local runs exist in the current directory and no remote runs match, it prints an empty-list message.

Run a local experiment from the project directory that should own the results:

ax experiment run experiment.yaml --local
ax run list

`ax run list` shows no runs

Use ax run list --scope local to check only ./.axp/runs/. If it is empty, run a local experiment first. Inspect a run's artifacts directly under .axp/runs/<run-id>/, or query them with ax run query <RUN_ID> sql "<SQL>" --local.

To inspect platform results locally, download the run first:

ax run download my-experiment-01HX...

Docker image pull fails

ax experiment run --local pulls axp-base from Docker Hub (514labs/axp-base) on first use. If the pull fails, check Docker is running and Docker Hub is reachable; you can pre-pull manually:

docker pull 514labs/axp-base:latest

Use ax experiment run experiment.yaml --local --resolve-variants when you only want experiment validation without touching containers. (Platform runs with ax experiment run need no local Docker.)

`ax experiment run` says you're not signed in

ax experiment run submits to the platform, so it needs saved credentials. If you see an authentication error, sign in (or set AX_API_KEY in CI):

ax auth login
ax auth status

To run without platform dispatch, use ax experiment run <path> --local (Docker; default managed model access still needs ax auth login / AX_API_KEY, or pass --local-model-keys with host provider keys).

`ax experiment run` and secrets

Platform runs support secrets: reference an org secret from environment_variables as ax://secrets/<slug> and dispatch resolves it server-side from your org's secret store. (Earlier releases rejected secret-bearing experiments at submit — that restriction is gone.) Store the value first with ax secret set <slug>; a reference to a slug that doesn't exist in the org fails at submit. The removed top-level secrets: field is rejected — use environment_variables + ax://secrets/<slug> instead.

See Runs and Environment and Auth.

Cancel a platform run

ax experiment run prints a run request id. Cancel queued or in-flight jobs with it:

ax run cancel <run-request-id>

Cancel only specific jobs with --run <run-id> (comma-separated); the default cancels every job in the request.

Not logged in for platform upload

ax run upload requires credentials saved by ax auth login. If credentials are missing, run:

ax auth login
ax auth status

ax auth status round-trips through the platform API and prints the resolved user and active org. Use ax auth orgs to list available orgs, and ax auth orgs --switch org_... to change the active CLI org.

The credentials file lives at ~/.fiveonefour/axp/credentials.toml. AX_API_KEY can override the saved API key.

Upload target is wrong

Auth, upload, and download requests go to the AX platform by default. If the AX team gives you a different platform URL, set AX_PLATFORM_URL:

AX_PLATFORM_URL=https://platform.example.com ax run upload my-experiment-01HX...

Active org is wrong

Platform commands use the active CLI org by default. Check it with:

ax auth status
ax auth orgs

Switch the active org with:

ax auth orgs --switch org_...

For a one-off upload or download without changing the active org, pass --org:

ax run upload my-experiment-01HX... --org org_...
ax run download my-experiment-01HX... --org org_...

If a run exists in another org that your API key can access, the platform error includes this hint. If the run is not in an accessible org, the error does not reveal where it exists.

Run was already uploaded

If the server reports that a legacy run already exists, pass --force to overwrite that upload. For a session-data run, the same command retries registration and publication without replacing its immutable generation:

ax run upload my-experiment-01HX... --force

Download destination already exists

ax run download writes to .axp/downloads/<run-id>/ and refuses to replace an existing directory unless you pass --force:

ax run download my-experiment-01HX... --force

Platform query says Not enough privileges

If ax results get, raw SQL via ax results query, or a pasted WITH scoped_runs AS (…) statement fails with Not enough privileges on tables outside the scope's public catalog, prefer the modeled path:

ax experiment query <EXPERIMENT_ID> --metric testPassRate --group-by variant

Modeled ax experiment query compiles and runs server-side. Pasting the CLI's compiled SQL into a reader-scoped SQL surface can hit ClickHouse grants the reader role does not have. For one composite run, use ax run query <RUN_ID> sql … instead of replaying experiment-scoped SQL.

If a table name is rejected, inspect the selected scope instead of guessing a physical name. For a remote session_data_v1 scope:

ax run query <RUN_ID> sql --tables

Remote session_data_v1 scopes expose measurements, events, and test_output. Legacy scopes keep their legacy tables, do not expose those logical views, and reject rich sql --tables; execute SQL directly instead:

ax run query <LEGACY_RUN_ID> sql "SHOW TABLES"
ax run query <LEGACY_RUN_ID> sql "DESCRIBE TABLE <name>"

Local query derivation fails

Use ax run query <RUN_ID> sql "<SQL>" --local to derive query files from a raw local run, or to read downloaded platform Parquet, and surface the conversion/query error:

ax run query <RUN_ID> sql "SELECT * FROM runs LIMIT 1" --local

Local DuckDB uses legacy physical tables even for downloaded data. Remote logical-table parity is not supported, so use SHOW TABLES locally and adapt ClickHouse-specific SQL when needed.

If the run is not local, use ax run list --scope local to find local runs or run ax run download <run-id> before querying locally.

Search documentation