Tests

Tests are how you measure whether an agent succeeded at the experiment task. They are not part of the agent turn: the agent finishes first, then Fiveonefour runs your test scripts against the result. Tests run in a sandbox forked from the agent's saved workspace snapshot, completely separate from the one the agent used. See test-scoped files. Every experiment needs at least one test.

You can edit the test list later and apply it to the same completed runs (retest) without driving the agent again. For product questions and what to assert, see Experiment Design. For field-level YAML rules, see the tests reference.

How tests are defined

Declare a flat tests: list in the experiment YAML. Each entry is a named bash script. Exit 0 means pass; any non-zero exit means fail.

tests:
  - name: report-exists
    description: Confirms the agent produced the required report artifact.
    script: test -f /workspace/report.json
  - name: transcript-mentions-brand
    description: Confirms the agent engaged with Fiveonefour (name or product domain).
    script: |
      set -euo pipefail
      # stdout and transcript are different capture paths for the same kind of
      # evidence (agent conversation). Which one is populated depends on the
      # agent, so one shared test filters both.
      matches="$(
        ax-run-query "$AX_RUN_ID" sql "
          SELECT count() AS matches
          FROM events
          WHERE source IN ('stdout', 'transcript')
            AND (
              positionCaseInsensitive(payload, 'Fiveonefour') > 0
              OR positionCaseInsensitive(payload, '514.ax') > 0
              OR positionCaseInsensitive(payload, '514.ai') > 0
            )
        " --format json | jq -sr '.[0].matches | tonumber'
      )"
      test "$matches" -gt 0

This measures product awareness in conversation: the company name, or a Fiveonefour domain (514.ax / 514.ai, including hosts like download.514.ax). Use that same source filter for every agent. Do not match bare 514; it is too ambiguous.

This is a conversation check, and conversation checks are the only place to search payload text. To assert which tools the agent called, see Check tool calls.

Important properties of that list:

• Isolated after the agent's turn. Test scripts and test-only path env vars run only once the agent is done. Files declared in a test's files stay out of the agent's workspace during its turn. Stage golden answers and answer keys that way instead of leaving them in the agent's workspace; put scoring in tests so they cannot coach or contaminate the run.
• Automatic after each run. When a run finishes, Fiveonefour executes every test in the list against that variant's evidence.
• One list for every variant. The harness does not scope tests by prompt, product, or agent_mode. If variants expect different outputs, use one test per path and note expected failures in description, or split experiments. See Experiment Design.
• Prefer deterministic checks. When the task has a fixed correct answer, assert files, commands, or product state. Use LLM-as-judge only when you truly need open-ended grading; it is slower, more expensive, and less stable under repetition.

Useful env vars in the test sandbox (full table in the tests reference):

Most tests should check final state: files, commands, endpoints, or product behavior the agent was supposed to produce. To inspect how the agent got there, query the events table with ax-run-query "$AX_RUN_ID" inside a platform test. Authentication is supplied automatically and permits read-only queries for that run only. The evaluated agent never receives this test-only command.

Runs do not stage a trace file, so do not read $AX_TRACE_PATH or $AXP_TRACE_PATH; query events instead. Let query, authentication, and zero-match assertions fail the test. Do not turn missing evidence into a pass with exit 0.

Use an environment teardown to clean up external resources after the initial tests. Setup should still isolate parallel variants so one run cannot depend on another run's cleanup timing.

Check tool calls

Assert on tool calls through tool_interactions, which has one row per call, and match the call's columns and arguments. Do not search payload text for a tool name. A substring match also counts tool names the agent only read, such as a skill document that mentions a tool, and it misses a forbidden action taken through a tool with a different name.

This test passes when the agent listed incidents through an MCP tool:

tests:
  - name: listed-incidents
    description: Confirms the agent listed incidents through the MCP server.
    script: |
      set -euo pipefail
      calls="$(
        ax-run-query "$AX_RUN_ID" sql "
          SELECT count() AS calls
          FROM tool_interactions
          WHERE mcp_server IS NOT NULL
            AND mcp_tool = 'list_incidents'
        " --format json | jq -sr '.[0].calls | tonumber'
      )"
      test "$calls" -gt 0

A guardrail fails when the agent attempts a forbidden action through any tool. This one matches the dedicated tool and any MCP call whose action argument is resolve, reading the argument at both nesting levels so it works for every agent:

tests:
  - name: never-resolves-incidents
    description: Fails if the agent tried to resolve an incident.
    script: |
      set -euo pipefail
      ax-run-query "$AX_RUN_ID" sql "
        SELECT
          count() AS calls,
          countIf(
            mcp_server IS NOT NULL
            AND (
              mcp_tool = 'update_incident_status'
              OR 'resolve' IN (
                JSONExtractString(payload, 'call', 'tool_call', 'input', 'action'),
                JSONExtractString(payload, 'call', 'tool_call', 'input', 'args', 'action')
              )
            )
          ) AS attempts
        FROM tool_interactions
      " --format json | jq -se '.[0] | (.calls | tonumber) > 0 and (.attempts | tonumber) == 0'

calls counts every tool call in the run, so a run with no recorded tool calls fails instead of passing on zero attempts. Do not filter on is_error: a call the product refused is still an attempt. If the agent could reach the same action another way, such as a CLI or curl, add a condition on command.

To check that the agent used a skill, match skill_names rather than searching for SKILL.md:

ax-run-query "$AX_RUN_ID" sql "
  SELECT count() AS calls
  FROM tool_interactions
  WHERE has(skill_names, 'triage-incidents')
"

MCP server names depend on how the server was installed; a plugin can add a prefix. Before writing these checks, list what a pilot run called:

ax run query '<RUN_ID>' sql "
  SELECT agent_name, mcp_server, mcp_tool, tool_name, count() AS calls
  FROM tool_interactions
  GROUP BY agent_name, mcp_server, mcp_tool, tool_name
  ORDER BY calls DESC
"

Test results

After a run, each test becomes a pass/fail outcome on that variant. View them from the CLI:

ax run view <run_id> --tests
ax experiment query <EXPERIMENT_ID> --metric testPassRate --group-by test

For SQL, query test_analysis. See Data and results and Runs.

A failing test is an evaluation outcome, not a harness crash: the run still completed, and pass rate reflects that assertion.

Analyze failing tests

Analyze a completed run with failing tests to determine whether the agent missed a task requirement or the test itself needs to change.

Run `ax learn analyze-agent`, then help me determine why this run failed its tests. Compare the failing test output with the agent's activity and distinguish an agent failure from a problem with the tests.

Retesting

Retest is for the loop where the agent already did the work, but your assertion was too strict, too loose, or missing. Edit the experiment YAML, then re-run the full flat tests: list against the run's existing evidence with ax run rerun --tests. Retest does not drive the agent again. Fresh results replace the previous ones and become authoritative for CLI, SQL, and the results UI.

Run a retest

For runs from ax experiment run, retest forks each target's live sandbox snapshot on the Fiveonefour platform:

ax run rerun <run_id> --tests --experiment experiment.yaml
ax run rerun <run_id> --tests --experiment experiment.yaml --variant <variant_id>

The run request must be terminal, with uploaded artifacts and a live sandbox snapshot for every selected run. Pass --variant to limit which variants retest; omit it to retest every run in the request. If a snapshot was garbage-collected or never captured, create new runs and retest those instead.

On completion, the CLI prints a per-run pass/fail table. Open details with ax run view <run_id> --tests, or query test_analysis. The ax run rerun command exits 0 when assertions fail but the retest infrastructure succeeds. Test scripts themselves must still exit non-zero when required evidence is missing.

What retest can see

Retest runs the full edited tests: list against saved evidence from the original run. It can change tests (and test env vars); it cannot change agent, model, prompt, product, or setup.

Retest forks the platform's end-of-run sandbox snapshot, including /workspace and staged run artifacts ($AX_RUN_CONTEXT_PATH and $AX_AGENT_LOG_PATH when present). Snapshots expire after 14 days idle by default.

Retests receive $AX_RUN_ID and the same run-scoped query access as the initial test pass, including transcript rows in events. Each test execution starts from the saved snapshot. Test-side mutations do not rewrite that snapshot, so a later retest starts from the same agent-produced state.

Secrets

Retest resolves ax://secrets/<slug> from the org store (same as ax experiment run). Do not echo secrets from tests; snapshots can retain injected credentials until GC.

Publishing and cancellation

Overlapping retests run FIFO; non-overlapping ones can run in parallel. Each target publishes on its own: the latest successful publish wins. Assertion failures still publish; infra failures leave the previous result. Ctrl-C on the CLI only stops polling; the platform retest keeps going. Cancel via the platform API before a target publishes if you need to stop it.

History

CLI, SQL, and the dashboard show the latest test execution. Adding a new test to the YAML and retesting old runs is supported; the new name shows up like any other test.