AXP CLI

ax is the command-line interface for the AXP agent experimentation platform. Commands are organized around the nouns you work with most: experiment, run, results, auth, and secret.

Installing, pinning a version, and uninstalling are done with the install script; see Installation. You can update with ax update or by re-running the installer.

ax <COMMAND>
ax --help
ax --version

Command index

Experiment commands

ax experiment <COMMAND>

experiment list

List experiments in the active org, newest first.

ax experiment list [--search <text>] [--owner <email-or-id>] [--limit <n>] [--json]

experiment view

Show a concise metadata summary for one registered experiment.

ax experiment view <EXPERIMENT_ID> [--web] [--json]

experiment create

Scaffold a new experiment YAML. The cli-install template replaces the old first-run generator and accepts the template-specific install flags.

ax experiment create <NAME> [--dir <dir>] [--schema-version <n>]

ax experiment create my-first-experiment \
  --template cli-install \
  --cli mycli \
  --target-description 'a CLI for managing OLAP databases' \
  --install-docs https://example.com/install \
  --install-command 'npm install mycli' \
  --smoke-command 'mycli --version' \
  --smoke-output-contains 'mycli version'

experiment validate

Validate experiment YAML against the schema. Successful validation prints the next command using the current grammar.

ax experiment validate <EXPERIMENT.yaml> [--no-ai]

Successful validation may also run a best-effort AI review of the experiment's design quality. The review is advisory: it prints to stderr and never changes the exit code, so stdout remains machine-readable. Signed-in users (ax auth login or AXP_API_KEY) get a fast 514-managed model review with prioritized suggestions under an AI review header. Signed-out users get a one-line note that signing in unlocks AI hints; no network call is made.

Before sending YAML for review, AXP elides large unbroken inline payloads such as base64 or hex fixture data while keeping surrounding YAML, prompts, and setup commands visible. If the review cannot run because the platform is unreachable or the experiment is still too large after elision, validation degrades to a one-line AI review skipped note and still succeeds. Pass --no-ai or set AXP_NO_AI_LINT=1 to skip the review entirely, and set AXP_AI_LINT_MODEL to try a different review model.

experiment schema

Print an experiment schema document.

ax experiment schema [--schema-version <n>]

experiment push

Register an experiment definition without running it. A push stores a content-addressed version; pushing unchanged YAML is idempotent.

ax experiment push <EXPERIMENT.yaml> [--org <org-id>]

experiment pull

Download registered YAML for editing. By default it pulls the latest version; pass a fingerprint to retrieve an older stored version.

ax experiment pull <EXPERIMENT_ID> [--version <fingerprint>] [--output <path>] [--force]

experiment run

Run a local YAML path or a registered experiment id. Platform runs stream status by default; use --detach for CI. Local execution is selected with --local and requires a YAML path.

ax experiment run <EXPERIMENT.yaml | EXPERIMENT_ID> [--detach] [-n <n>]
ax experiment run <EXPERIMENT.yaml> --local [--watch] [-j <jobs>]
ax experiment run <EXPERIMENT_ID> --version <fingerprint>

Common selectors are repeatable and comma-separated: --variant, --prompt, --agent, --model, --product, and --environment. The selectors are a union: each selected variant or axis value is included once. --file and --file-sha256 stage files into variants before setup. Local-only flags include --resolve-variants, --watch, --watch-raw, --env-file, --env, --output-root, --managed-model-access, and --api-base-url.

experiment versions

List stored versions of a registered experiment.

ax experiment versions <EXPERIMENT_ID> [--json]

experiment variants

Resolve variant coordinates, ids, and fingerprints with the canonical Rust resolver.

ax experiment variants <EXPERIMENT.yaml> [--json] [--runtime]

Run commands

ax run <COMMAND>

run list

List local and platform runs, one row per run.

ax run list [--experiment <id-or-url>] [--status queued,waiting-for-ci,running,completed,failed,canceled]
ax run list [--group-by variant|agent|product|environment|prompt] [--json]

By default --scope both combines local .axp/runs/ data with remote platform runs. --group-by also accepts status and experiment.

run view

View a run request roll-up or one composite run id in detail. Composite run ids contain ::; request ids do not.

ax run view <RUN_REQUEST_ID> [--web] [--tests] [--json]
ax run view <VARIANT_ID::HASH> [--scope local|remote|both] [--tests]

For request roll-ups, run view exits non-zero when the run is terminal with any unsuccessful run simulation, such as a setup, driver, timeout, cost-cap, or canceled run. A run whose agent completed but whose post-agent tests failed is still a successful run simulation for CLI exit-code purposes; inspect run details and results to evaluate task quality. A still-running run exits zero.

run watch

Poll a run request until it reaches a terminal state.

ax run watch <RUN_REQUEST_ID> [--json]

run cancel

Cancel all jobs in a run request, or selected child runs.

ax run cancel <RUN_REQUEST_ID> [--run <run-id>[,<run-id>...]]

run rerun

Rerun failed/canceled variants from prior platform requests, or re-run tests only without driving the agent again.

ax run rerun <RUN_REQUEST_ID> [--experiment <EXPERIMENT.yaml>]
ax run rerun <RUN_REQUEST_ID> --tests [--experiment <EXPERIMENT.yaml>] [--variant <id>]
ax run rerun --tests --local --experiment <EXPERIMENT.yaml> --run <LOCAL_RUN_ID>

Previously this behavior was split between top-level retest commands and rerun flags; run rerun --tests is the current spelling. Remote test-only reruns execute against each target run's captured sandbox snapshot; if a snapshot was garbage-collected or never captured, submit is rejected — create new runs with ax experiment run and rerun tests on those. See Retest.

run download

Download platform Parquet tables for local analysis.

ax run download [RUN_ID] [--force]

run upload

Upload a local run's artifacts.parquet to the platform.

ax run upload [RUN_ID] [--force]

run clean

List or remove orphaned local sandbox containers.

ax run clean [--yes]

Results commands

ax results <COMMAND>

All modeled result-set commands use the same query flags: --experiment, --experiment-version, --filter dim=value, --agent, --product, --environment, --prompt, --group-by, --metric, --json, and --org. Valid --filter dimensions are agent, product, environment, and prompt. Valid --group-by values are variant, agent, product, environment, and prompt.

results dimensions uses the same query flags to narrow each displayed value list client-side: filtered dimensions show only matching observed values, and a filter typo leaves that dimension visible with an empty value list.

results view

Print a summary table for an experiment's results.

ax results view --experiment <EXPERIMENT_ID> --group-by variant

results export

Export grouped result rows. The format defaults from .csv, .json, or .tsv.

ax results export out.csv --experiment <EXPERIMENT_ID> [--format csv|json|tsv]

results share

Print a shareable results URL.

ax results share --experiment <EXPERIMENT_ID>

results dimensions

List valid result dimensions and observed values.

ax results dimensions --experiment <EXPERIMENT_ID>

results scatter

Print individual run points for an experiment scatter plot. By default, the CLI prints a compact table with one row per sampled run. With --json, it prints the bare point list JSON that charting clients can consume directly.

ax results scatter --experiment <EXPERIMENT_ID> [--json]

The sample is deterministic and capped at 2,000 points. When an experiment has more matching runs, the CLI prints a note to stderr with the shown and total point counts so you know the table is a sample, not the full population. With --json, stdout remains a bare point array; any truncation note is still reported on stderr.

results runs

List the runs behind a filtered result set.

ax results runs --experiment <EXPERIMENT_ID> --filter agent=claude

results analyze

Print a templated analysis prompt for a run. explore is the default template.

ax results analyze [TEMPLATE] [--run <RUN_ID>]

results query

Run read-only SQL. Without --local, the query runs against uploaded platform ClickHouse data. With --local, it runs DuckDB over a local or downloaded run.

ax results query "SELECT count() FROM runs" --table
ax results query "SHOW TABLES" --local <RUN_ID> --table

Platform access

auth

ax auth login [--token <token>] [--org <org-id>]
ax auth connect --code <code>
ax auth status [--json]
ax auth orgs [--list|--switch <org-id>] [--json]
ax auth logout

auth whoami is still accepted; auth status is the current spelling.

secret

Manage your organization's write-only secret store. Values are referenced from experiments as axp://secrets/<slug> and are never printed back to the terminal.

ax secret set <slug> [--description <text>]
ax secret list [--limit <n>] [--page-token <token>] [--json]
ax secret delete <slug>

Additional commands

ci upload

Register a CI-built product artifact for the !ax run PR trigger.

ax ci upload <SLOT> <PATH> [--repo <owner/repo>] [--commit <sha>]

integrations

Inspect org integrations: GitHub (PR trigger) and BYOK LLM providers (same surface as Settings → Integrations). Requires ax auth login (or AXP_API_KEY).

ax integrations list [--json]
ax integrations gh list [--json]
ax integrations gh link <experiment-id> <owner/repo>
ax integrations gh link --path <repo-relative-path> <owner/repo>
ax integrations llm-provider list [--json]
ax integrations llm-provider view <provider-id> [--json] [--web]
ax integrations llm-provider create <display-name> --base-url <url> --model <lab/id>... [--priority N]
ax integrations llm-provider update <provider-id> [--display-name <name>] [--base-url <url>] [--model <lab/id>...] [--set-key] [--priority N]
ax integrations llm-provider delete <provider-id> [--yes]
ax integrations llm-provider reorder <provider-id>... [--gateway-priority N]
ax integrations llm-provider test --base-url <url>
ax integrations llm-provider gateway <enable|disable> [--priority N]
ax integrations open [--no-browser]

integrations llm-provider

Manage org BYOK LLM providers and the built-in 514.ax AI gateway. Providers are stack-ranked (lower priority runs first). API keys are write-only — prompted on a TTY or read from stdin, never accepted as a flag, and never returned by reads. CLI Bearer/API-key auth requires org admin (session custom permissions such as org:llm_providers:manage apply in the UI only).

analyze

Create, read, and list persisted guided trace analyses for the active organization. These commands require ax auth login or ax auth connect; pass --org <ORG_ID> for a one-off org override.

ax analyze suggest --run <RUN_ID>[,<RUN_ID>] [--experiment-id <EXPERIMENT_ID>]
ax analyze create --input analysis.json
ax analyze get <ANALYSIS_ID>
ax analyze list [--experiment-id <EXPERIMENT_ID>] [--result-id <RESULT_ID>]

suggest prepares the guided prompt for a batch of failed/error runs. The experiment id is inferred from the selected run ids; pass --experiment-id when you want an explicit guard that rejects runs from another experiment. It returns the seeded errored-runs template, the ax results query calls the agent should run, structured_prompt_json, and a create-request template. The failed-run template excludes completed runs that merely have failing tests.

create reads a request-shaped JSON document. The input can use JSON objects (structured_prompt, trace_analysis_output, scope.metrics_spec, scope.filters) or pre-serialized JSON strings (structured_prompt_json, trace_analysis_output_json, scope.metrics_spec_json, scope.filters_json).

{
  "scope": {
    "experiment_id": "my-experiment",
    "experiment_version": "v1",
    "metrics_spec": {},
    "filters": {},
    "group_by": "variant",
    "run_ids": ["run_..."]
  },
  "structured_prompt": { "templateId": "why-did-this-run-fail" },
  "question_template_id": "why-did-this-run-fail",
  "trace_analysis_output": {
    "scope": { "id": "result_..." },
    "question": "Why did this run fail?",
    "answer": { "claim": "...", "confidence": "high" },
    "evidence": [{ "result": { "id": "result_..." }, "note": "..." }],
    "interpretation": {
      "likelyCause": "product",
      "failureMode": "install discovery failure",
      "productSurface": "CLI installation docs",
      "rationale": "..."
    },
    "caveats": [],
    "recommendation": {
      "actionType": "change-docs",
      "action": "...",
      "followUpQuestion": "..."
    }
  }
}

send-debug

Bundle a run, source experiment, and host metadata for support.

ax send-debug [RUN_ID] [--note <text>] [--out <path>] [--dry-run] [--yes]

Use --note to record context in the bundle metadata, --no-workspace to omit variants/*/workspace/, and --no-fs-diff to omit variants/*/fs-diff/. --dry-run builds the bundle and prints stats without uploading; --out also writes the assembled tarball locally.

release-notes

Print release notes as Markdown.

ax release-notes [VERSION] [--raw] [--source-url]

update

Install the latest stable CLI, or print the installer command.

ax update
ax update --print