Tests

tests defines how Fiveonefour scores each run. It is one flat list of bash checks. Test scripts receive resolved environment_variables, run identity, and path env vars for staged run artifacts, so any test can inspect final /workspace state, run commands, query its run, or branch on $AX_RUN_CONTEXT_PATH. See Tests for the complete transcript-query example.

tests:
  - name: report-exists
    description: Confirms the agent produced the required report artifact.
    script: test -f /workspace/report.json

Test runtime environment

Platform tests run in a separate sandbox forked from the agent's saved /workspace. The agent does not see the test scripts or these test-only variables.

For "what the agent said" checks, query events WHERE source IN ('stdout', 'transcript') with the test-only ax-run-query "$AX_RUN_ID" command. Use that same filter for every agent: those sources are different capture paths for conversation, and which one is populated depends on the agent. Do not read $AX_TRACE_PATH or $AXP_TRACE_PATH; runs do not stage a trace file. Let missing rows and query errors fail the script. Do not use exit 0 when evidence is unavailable.

For tool-call evidence, query the same events table:

ax-run-query "$AX_RUN_ID" sql "
  SELECT payload
  FROM events
  WHERE kind = 'tool_call'
    AND positionCaseInsensitive(payload, 'SKILL.md') > 0
"

Use sd_current_tool_calls only when you need its typed status and tool-name columns.

Example:

jq -e '.run_id != null and .agent == "claude"' "$AX_RUN_CONTEXT_PATH"

Test object

Test-scoped files

A test's files stage host files into that test's sandbox only, immediately before its script runs, never while the agent's own turn is still in progress. Use it for anything a test must grade against but the agent must not see, such as a reference implementation or an expected-output fixture. Staging that same fixture under top-level files instead puts it in the agent's own workspace, where the agent can read or copy it.

tests:
  - name: matches-reference-output
    description: Grades the agent's report against an expected fixture it never saw.
    script: |
      set -euo pipefail
      python3 /workspace/fixtures/grade.py /workspace/report.json /workspace/fixtures/expected.json
    files:
      - name: grader
        source: ./fixtures/grade.py
        dest: fixtures/grade.py
      - name: expected-output
        source: ./fixtures/expected.json
        dest: fixtures/expected.json

Isolation from the agent depends on how the run executes:

• Hosted runs (ax experiment run) keep test-only files out of the agent's turn. They land on disk only immediately before that test's script runs, whether tests run in the same sandbox after the agent or later in a sandbox forked from the run's saved end-of-agent snapshot (for a retest).
• Local runs (ax experiment run --local) execute tests in the same sandbox the agent used, after the agent phase finishes. Test files wait outside /workspace until immediately before each test's script, then extract into /workspace. Treat that as a convenience, not a guarantee.
• Local retest (ax run rerun --tests --local) runs in a fresh sandbox with no agent phase at all, so test files stage per test exactly as above with nothing to isolate from.

Per-test rules, in every run mode:

• Files stage immediately before their owning test's script, not once for the whole run.
• Two tests may declare the same dest with different source; each test sees only its own.
• The same source staged to the same dest across tests uploads once and is shared; the same source staged to a different dest is a separate upload.
• Test files never appear in the fs-diff artifact, even when a test's own script writes into them.

Staging failures are scoped the same way as top-level files: an unresolvable source aborts the whole run at preflight, and a failure delivering the file into the sandbox rolls the whole variant up as status=error / exit_reason=staging_failed, before the agent runs. Only a failure extracting an already-delivered test file into its owning test's sandbox, immediately before that test's script, fails just that one test; the rest of the run keeps going.

ax experiment validate rejects a test files entry whose dest collides with a top-level or setup files dest, since a local run eventually extracts every scope into the same /workspace. The same dest repeated across different tests is fine. The MCP experiment_run tool cannot run an experiment that declares any files staging, including tests[*].files; use the CLI instead.