AXP CLI
Canonical command reference for the AXP CLI.
ax is the command-line interface for the AXP agent experimentation platform.
Commands are organized around the nouns you work with most: experiment, run,
results, auth, and secret.
Installing, pinning a version, and uninstalling are done with the install script;
see Installation. You can update with ax update or by
re-running the installer.
ax <COMMAND>
ax --help
ax --versionCommand index
| Command | Use |
|---|---|
experiment list | List registered experiments. |
experiment view | Inspect one registered experiment. |
experiment create | Scaffold experiment YAML. |
experiment validate | Validate experiment YAML. |
experiment schema | Print the schema. |
experiment push | Register YAML without running it. |
experiment pull | Download registered YAML. |
experiment run | Run a path or registered experiment id. |
experiment versions | List stored experiment versions. |
experiment variants | Resolve variants and fingerprints. |
run list | List local and platform runs. |
run view | View a run request or composite run id. |
run watch | Follow a run request until terminal. |
run cancel | Cancel queued or in-flight jobs. |
run rerun | Rerun failed/canceled variants, or re-run tests only. |
run download | Download platform Parquet tables. |
run upload | Upload a local run artifact. |
run clean | Clean orphaned local sandbox containers. |
results view | Summarize an experiment's results. |
results export | Export grouped result rows. |
results share | Print a shareable results URL. |
results dimensions | List result dimensions and values. |
results scatter | Print sampled cost/time points for charting. |
results runs | List runs behind a filtered result set. |
results analyze | Print an analysis prompt for a run. |
results query | Query platform ClickHouse or local DuckDB. |
auth | Log in, switch orgs, and show identity status. |
secret | Manage org-scoped secret values. |
ci upload | Upload PR-trigger artifacts from CI. |
integrations | Link GitHub repos to experiments; manage BYOK LLM providers. |
analyze | Create, read, and list guided trace analyses. |
send-debug | Upload a debug bundle. |
release-notes | Print release notes. |
update | Install the latest stable CLI. |
Experiment commands
ax experiment <COMMAND>experiment list
List experiments in the active org, newest first.
ax experiment list [--search <text>] [--owner <email-or-id>] [--limit <n>] [--json]experiment view
Show a concise metadata summary for one registered experiment.
ax experiment view <EXPERIMENT_ID> [--web] [--json]experiment create
Scaffold a new experiment YAML. The cli-install template replaces the old
first-run generator and accepts the template-specific install flags.
ax experiment create <NAME> [--dir <dir>] [--schema-version <n>]
ax experiment create my-first-experiment \
--template cli-install \
--cli mycli \
--target-description 'a CLI for managing OLAP databases' \
--install-docs https://example.com/install \
--install-command 'npm install mycli' \
--smoke-command 'mycli --version' \
--smoke-output-contains 'mycli version'experiment validate
Validate experiment YAML against the schema. Successful validation prints the next command using the current grammar.
ax experiment validate <EXPERIMENT.yaml> [--no-ai]Successful validation may also run a best-effort AI review of the
experiment's design quality. The review is advisory: it prints to stderr and
never changes the exit code, so stdout remains machine-readable. Signed-in users
(ax auth login or AXP_API_KEY) get a fast 514-managed model review with
prioritized suggestions under an AI review header. Signed-out users get a
one-line note that signing in unlocks AI hints; no network call is made.
Before sending YAML for review, AXP elides large unbroken inline payloads such
as base64 or hex fixture data while keeping surrounding YAML, prompts, and setup
commands visible. If the review cannot run because the platform is unreachable
or the experiment is still too large after elision, validation degrades to a
one-line AI review skipped note and still succeeds. Pass --no-ai or set
AXP_NO_AI_LINT=1 to skip the review entirely, and set AXP_AI_LINT_MODEL to
try a different review model.
experiment schema
Print an experiment schema document.
ax experiment schema [--schema-version <n>]experiment push
Register an experiment definition without running it. A push stores a content-addressed version; pushing unchanged YAML is idempotent.
ax experiment push <EXPERIMENT.yaml> [--org <org-id>]experiment pull
Download registered YAML for editing. By default it pulls the latest version; pass a fingerprint to retrieve an older stored version.
ax experiment pull <EXPERIMENT_ID> [--version <fingerprint>] [--output <path>] [--force]experiment run
Run a local YAML path or a registered experiment id. Platform runs stream status
by default; use --detach for CI. Local execution is selected with --local and
requires a YAML path.
ax experiment run <EXPERIMENT.yaml | EXPERIMENT_ID> [--detach] [-n <n>]
ax experiment run <EXPERIMENT.yaml> --local [--watch] [-j <jobs>]
ax experiment run <EXPERIMENT_ID> --version <fingerprint>Common selectors are repeatable and comma-separated: --variant, --prompt,
--agent, --model, --product, and --environment. The selectors are a
union: each selected variant or axis value is included once. --file and
--file-sha256 stage files into variants before setup. Local-only flags include
--resolve-variants, --watch, --watch-raw, --env-file, --env,
--output-root, --managed-model-access, and --api-base-url.
experiment versions
List stored versions of a registered experiment.
ax experiment versions <EXPERIMENT_ID> [--json]experiment variants
Resolve variant coordinates, ids, and fingerprints with the canonical Rust resolver.
ax experiment variants <EXPERIMENT.yaml> [--json] [--runtime]Run commands
ax run <COMMAND>run list
List local and platform runs, one row per run.
ax run list [--experiment <id-or-url>] [--status queued,waiting-for-ci,running,completed,failed,canceled]
ax run list [--group-by variant|agent|product|environment|prompt] [--json]By default --scope both combines local .axp/runs/ data with remote platform
runs. --group-by also accepts status and experiment.
run view
View a run request roll-up or one composite run id in detail. Composite run ids
contain ::; request ids do not.
ax run view <RUN_REQUEST_ID> [--web] [--tests] [--json]
ax run view <VARIANT_ID::HASH> [--scope local|remote|both] [--tests]For request roll-ups, run view exits non-zero when the run is terminal with
any unsuccessful run simulation, such as a setup, driver, timeout, cost-cap, or
canceled run. A run whose agent completed but whose post-agent tests failed is
still a successful run simulation for CLI exit-code purposes; inspect run
details and results to evaluate task quality. A still-running run exits zero.
run watch
Poll a run request until it reaches a terminal state.
ax run watch <RUN_REQUEST_ID> [--json]run cancel
Cancel all jobs in a run request, or selected child runs.
ax run cancel <RUN_REQUEST_ID> [--run <run-id>[,<run-id>...]]run rerun
Rerun failed/canceled variants from prior platform requests, or re-run tests only without driving the agent again.
ax run rerun <RUN_REQUEST_ID> [--experiment <EXPERIMENT.yaml>]
ax run rerun <RUN_REQUEST_ID> --tests [--experiment <EXPERIMENT.yaml>] [--variant <id>]
ax run rerun --tests --local --experiment <EXPERIMENT.yaml> --run <LOCAL_RUN_ID>Previously this behavior was split between top-level retest commands and rerun
flags; run rerun --tests is the current spelling.
Remote test-only reruns execute against each target run's captured sandbox snapshot; if a snapshot was garbage-collected or never captured, submit is rejected — create new runs with ax experiment run and rerun tests on those. See Retest.
run download
Download platform Parquet tables for local analysis.
ax run download [RUN_ID] [--force]run upload
Upload a local run's artifacts.parquet to the platform.
ax run upload [RUN_ID] [--force]run clean
List or remove orphaned local sandbox containers.
ax run clean [--yes]Results commands
ax results <COMMAND>All modeled result-set commands use the same query flags: --experiment,
--experiment-version, --filter dim=value, --agent, --product,
--environment, --prompt, --group-by, --metric, --json, and --org.
Valid --filter dimensions are agent, product, environment, and prompt.
Valid --group-by values are variant, agent, product, environment, and
prompt.
results dimensions uses the same query flags to narrow each displayed value
list client-side: filtered dimensions show only matching observed values, and a
filter typo leaves that dimension visible with an empty value list.
results view
Print a summary table for an experiment's results.
ax results view --experiment <EXPERIMENT_ID> --group-by variantresults export
Export grouped result rows. The format defaults from .csv, .json, or .tsv.
ax results export out.csv --experiment <EXPERIMENT_ID> [--format csv|json|tsv]results share
Print a shareable results URL.
ax results share --experiment <EXPERIMENT_ID>results dimensions
List valid result dimensions and observed values.
ax results dimensions --experiment <EXPERIMENT_ID>results scatter
Print individual run points for an experiment scatter plot. By default, the CLI
prints a compact table with one row per sampled run. With --json, it prints the
bare point list JSON that charting clients can consume directly.
ax results scatter --experiment <EXPERIMENT_ID> [--json]The sample is deterministic and capped at 2,000 points. When an experiment has
more matching runs, the CLI prints a note to stderr with the shown and total
point counts so you know the table is a sample, not the full population. With
--json, stdout remains a bare point array; any truncation note is still
reported on stderr.
results runs
List the runs behind a filtered result set.
ax results runs --experiment <EXPERIMENT_ID> --filter agent=clauderesults analyze
Print a templated analysis prompt for a run. explore is the default template.
ax results analyze [TEMPLATE] [--run <RUN_ID>]results query
Run read-only SQL. Without --local, the query runs against uploaded platform
ClickHouse data. With --local, it runs DuckDB over a local or downloaded run.
ax results query "SELECT count() FROM runs" --table
ax results query "SHOW TABLES" --local <RUN_ID> --tablePlatform access
auth
ax auth login [--token <token>] [--org <org-id>]
ax auth connect --code <code>
ax auth status [--json]
ax auth orgs [--list|--switch <org-id>] [--json]
ax auth logoutauth whoami is still accepted; auth status is the current spelling.
secret
Manage your organization's write-only secret store. Values are referenced from
experiments as axp://secrets/<slug> and are never printed back to the terminal.
ax secret set <slug> [--description <text>]
ax secret list [--limit <n>] [--page-token <token>] [--json]
ax secret delete <slug>Additional commands
ci upload
Register a CI-built product artifact for the !ax run PR trigger.
ax ci upload <SLOT> <PATH> [--repo <owner/repo>] [--commit <sha>]integrations
Inspect org integrations: GitHub (PR trigger) and BYOK LLM providers (same
surface as Settings → Integrations). Requires ax auth login (or AXP_API_KEY).
ax integrations list [--json]
ax integrations gh list [--json]
ax integrations gh link <experiment-id> <owner/repo>
ax integrations gh link --path <repo-relative-path> <owner/repo>
ax integrations llm-provider list [--json]
ax integrations llm-provider view <provider-id> [--json] [--web]
ax integrations llm-provider create <display-name> --base-url <url> --model <lab/id>... [--priority N]
ax integrations llm-provider update <provider-id> [--display-name <name>] [--base-url <url>] [--model <lab/id>...] [--set-key] [--priority N]
ax integrations llm-provider delete <provider-id> [--yes]
ax integrations llm-provider reorder <provider-id>... [--gateway-priority N]
ax integrations llm-provider test --base-url <url>
ax integrations llm-provider gateway <enable|disable> [--priority N]
ax integrations open [--no-browser]integrations llm-provider
Manage org BYOK LLM providers and the built-in 514.ax AI gateway. Providers are
stack-ranked (lower priority runs first). API keys are write-only — prompted on
a TTY or read from stdin, never accepted as a flag, and never returned by reads.
CLI Bearer/API-key auth requires org admin (session custom permissions such
as org:llm_providers:manage apply in the UI only).
analyze
Create, read, and list persisted guided trace analyses for the active
organization. These commands require ax auth login or ax auth connect;
pass --org <ORG_ID> for a one-off org override.
ax analyze suggest --run <RUN_ID>[,<RUN_ID>] [--experiment-id <EXPERIMENT_ID>]
ax analyze create --input analysis.json
ax analyze get <ANALYSIS_ID>
ax analyze list [--experiment-id <EXPERIMENT_ID>] [--result-id <RESULT_ID>]suggest prepares the guided prompt for a batch of failed/error runs. The
experiment id is inferred from the selected run ids; pass --experiment-id when
you want an explicit guard that rejects runs from another experiment. It returns
the seeded errored-runs template, the ax results query calls the agent
should run, structured_prompt_json, and a create-request template. The
failed-run template excludes completed runs that merely have failing tests.
create reads a request-shaped JSON document. The input can use JSON objects
(structured_prompt, trace_analysis_output, scope.metrics_spec,
scope.filters) or pre-serialized JSON strings (structured_prompt_json,
trace_analysis_output_json, scope.metrics_spec_json, scope.filters_json).
{
"scope": {
"experiment_id": "my-experiment",
"experiment_version": "v1",
"metrics_spec": {},
"filters": {},
"group_by": "variant",
"run_ids": ["run_..."]
},
"structured_prompt": { "templateId": "why-did-this-run-fail" },
"question_template_id": "why-did-this-run-fail",
"trace_analysis_output": {
"scope": { "id": "result_..." },
"question": "Why did this run fail?",
"answer": { "claim": "...", "confidence": "high" },
"evidence": [{ "result": { "id": "result_..." }, "note": "..." }],
"interpretation": {
"likelyCause": "product",
"failureMode": "install discovery failure",
"productSurface": "CLI installation docs",
"rationale": "..."
},
"caveats": [],
"recommendation": {
"actionType": "change-docs",
"action": "...",
"followUpQuestion": "..."
}
}
}| Subcommand | Description |
|---|---|
suggest --experiment-id <EXPERIMENT_ID> --run <RUN_ID> | Prepare the failed-run analysis prompt, query calls, and create-request skeleton. --run may be repeated, comma-separated, or piped on stdin. |
create --input <PATH|-> | Validate and persist caller-supplied trace analysis output. Reads stdin when --input - is used. |
get <ANALYSIS_ID> | Read one persisted trace analysis. |
list | List persisted trace analyses, newest first. |
| Option | Description |
|---|---|
--org <ORG_ID> | Use this org for the command without changing the active CLI org. |
--experiment-fingerprint <FINGERPRINT> | suggest only. Require every run to match this experiment fingerprint. |
--experiment-version <VERSION> | suggest only. Carry this version label into the create scope. |
--limit <LIMIT> | list only. Maximum analyses to return. Defaults to 50; max 500. |
--page-token <TOKEN> | list only. Cursor returned by a previous page. Reserved today. |
--experiment-id <EXPERIMENT_ID> | suggest and list. Guard suggested runs or filter by experiment id. |
--result-id <RESULT_ID> | list only. Filter by persisted scope Result id. |
-h, --help | Print help. |
send-debug
Bundle a run, source experiment, and host metadata for support.
ax send-debug [RUN_ID] [--note <text>] [--out <path>] [--dry-run] [--yes]Use --note to record context in the bundle metadata, --no-workspace to omit
variants/*/workspace/, and --no-fs-diff to omit variants/*/fs-diff/.
--dry-run builds the bundle and prints stats without uploading; --out also
writes the assembled tarball locally.
release-notes
Print release notes as Markdown.
ax release-notes [VERSION] [--raw] [--source-url]update
Install the latest stable CLI, or print the installer command.
ax update
ax update --print