Tests
Bash checks that score every run.
tests defines how AX scores each run. It is one flat list of bash checks. Test scripts receive resolved environment_variables, AX_VARIANT_ID, and path env vars for staged run artifacts, so any test can inspect final /workspace state, run commands, read $AX_TRACE_PATH, or branch on $AX_RUN_CONTEXT_PATH.
tests:
- name: report-exists
description: Confirms the agent produced the required report artifact.
script: test -f /workspace/report.json
- name: under-thirty-tool-calls
script: |
[ "$(jq -s '[.[] | select(.kind == "tool_call")] | length' "$AX_TRACE_PATH")" -lt 30 ]Test runtime environment
Tests run inside the variant sandbox with /workspace as the working tree. The agent does not see the test scripts or these test-only path variables.
| Env var | Available to | Value |
|---|---|---|
AX_RUN_CONTEXT_PATH | Setup checks and tests | /tmp/axp/run-context.json, a JSON object with run_id, run_request_id, repeat_idx, variant_id, variant_tag, agent, model, canonical_model_id, effort, context_window_size, thinking, fast, prompt_id, prompt, environment_id, product_id, product_version, tags, and resolved_extend_id. Absent axes/controls are JSON null. |
AX_TRACE_PATH | Tests only | /tmp/axp/agent-events.jsonl, staged when the agent event stream exists. ACP runs and non-native cursor runs produce one; --flag native runs do not (no ACP session), so the variable is never set for them and trace-based tests fail native variants. |
AX_AGENT_LOG_PATH | Tests only | /tmp/axp/agent.log, staged when the agent log exists. |
AX_VARIANT_ID | Setup checks and tests | The derived variant id. Kept for backward compatibility; the same value is also in run context JSON. |
Example:
jq -e '.run_id != null and .agent == "claude"' "$AX_RUN_CONTEXT_PATH"Test object
| Field | Required | Type / values | Notes |
|---|---|---|---|
name | Yes | Kebab-case string | Must be unique. |
description | No | Non-empty string | Explains the test's intent to analysis models. Updating or removing it with ax experiment push keeps the same semantic version and existing runs; ax experiment pull and the mutable description column in ax experiment query return the latest pushed value. Test outcome fields and version-change reports remain unchanged. |
script | Yes | String | Bash script. Streamed over stdin and not shown to the agent. |