Tests
Bash checks that score every run.
tests defines how Fiveonefour scores each run. It is one flat list of bash checks. Test scripts receive resolved environment_variables, run identity, and path env vars for staged run artifacts, so any test can inspect final /workspace state, run commands, query its run, or branch on $AX_RUN_CONTEXT_PATH. See Tests for the complete transcript-query example.
tests:
- name: report-exists
description: Confirms the agent produced the required report artifact.
script: test -f /workspace/report.jsonTest runtime environment
Platform tests run in a separate sandbox forked from the agent's saved /workspace. The agent does not see the test scripts or these test-only variables.
| Env var | Available to | Value |
|---|---|---|
AX_RUN_ID | Platform tests | The current run id. Use ax-run-query "$AX_RUN_ID" for read-only access to this run's derived data. |
AX_RUN_CONTEXT_PATH | Setup checks and tests | /tmp/axp/run-context.json, a JSON object with run_id, run_request_id, repeat_idx, variant_id, variant_tag, agent, model, canonical_model_id, effort, context_window_size, thinking, fast, prompt_id, prompt, environment_id, product_id, product_version, tags, and resolved_extend_id. Absent axes/controls are JSON null. |
AX_AGENT_LOG_PATH | Tests only | /tmp/axp/agent.log, staged when the agent log exists. |
AX_VARIANT_ID | Setup checks and tests | The derived variant id. Kept for backward compatibility; the same value is also in run context JSON. |
For "what the agent said" checks, query events WHERE source IN ('stdout', 'transcript') with the test-only ax-run-query "$AX_RUN_ID" command. Use that same filter for every agent: those sources are different capture paths for conversation, and which one is populated depends on the agent. Do not read $AX_TRACE_PATH or $AXP_TRACE_PATH; current session-data runs do not stage those legacy trace files. Let missing rows and query errors fail the script. Do not use exit 0 when evidence is unavailable.
Example:
jq -e '.run_id != null and .agent == "claude"' "$AX_RUN_CONTEXT_PATH"Test object
| Field | Required | Type / values | Notes |
|---|---|---|---|
name | Yes | Kebab-case string | Must be unique. |
description | No | Non-empty string | Explains the test's intent to analysis models. Updating or removing it with ax experiment push keeps the same semantic version and existing runs; ax experiment pull and the mutable description column in ax experiment query return the latest pushed value. Test outcome fields and version-change reports remain unchanged. |
script | Yes | String | Bash script. Streamed over stdin and not shown to the agent. |