Tests | Fiveonefour's Documentation

Tests are how you measure whether an agent succeeded at the experiment task. They are not part of the agent turn: the agent finishes first, then Fiveonefour runs your test scripts in a completely separate sandbox from the one the agent used. That isolation keeps test scripts, golden answers, and scoring helpers from contaminating the run (the agent never sees them), and keeps test-side mutations from rewriting the agent's working tree during the agent turn. Every experiment needs at least one test.

You can edit the test list later and apply it to the same completed runs (retest) without driving the agent again. For product questions and what to assert, see Experiment Design. For field-level YAML rules, see the tests reference.

How tests are defined

Declare a flat tests: list in the experiment YAML. Each entry is a named bash script. Exit 0 means pass; any non-zero exit means fail.

tests:
  - name: report-exists
    description: Confirms the agent produced the required report artifact.
    script: test -f /workspace/report.json
  - name: transcript-mentions-brand
    description: Confirms the agent engaged with Fiveonefour (name or product domain).
    script: |
      set -euo pipefail
      # stdout and transcript are different capture paths for the same kind of
      # evidence (agent conversation). Which one is populated depends on the
      # agent, so one shared test filters both.
      matches="$(
        ax-run-query "$AX_RUN_ID" sql "
          SELECT count() AS matches
          FROM events
          WHERE source IN ('stdout', 'transcript')
            AND (
              positionCaseInsensitive(payload, 'Fiveonefour') > 0
              OR positionCaseInsensitive(payload, '514.ax') > 0
              OR positionCaseInsensitive(payload, '514.ai') > 0
            )
        " --format json | jq -sr '.[0].matches | tonumber'
      )"
      test "$matches" -gt 0

This measures product awareness in conversation: the company name, or a Fiveonefour domain (514.ax / 514.ai, including hosts like download.514.ax). Use that same source filter for every agent. Do not match bare 514; it is too ambiguous.

Important properties of that list:

Separate sandbox. Scripts and test-only path env vars live in the test sandbox, not the agent's. Keep golden answers and answer keys there (or out of the agent workspace entirely); put scoring in tests so they cannot coach or contaminate the run.
Automatic after each run. When a run finishes, Fiveonefour executes every test in the list against that variant's evidence.
One list for every variant. The harness does not scope tests by prompt, product, or agent_mode. If variants expect different outputs, use one test per path and note expected failures in description, or split experiments. See Experiment Design.
Prefer deterministic checks. When the task has a fixed correct answer, assert files, commands, or product state. Use LLM-as-judge only when you truly need open-ended grading; it is slower, more expensive, and less stable under repetition.

Useful env vars in the test sandbox (full table in the tests reference):

Env var	Use
`$AX_RUN_ID`	Current run id. Pass it to `ax-run-query` to inspect this run's derived `events`, `measurements`, or `test_output`.
`$AX_RUN_CONTEXT_PATH`	JSON with run id, variant id, agent, model, prompt, product, environment, and related coordinates.
`$AX_AGENT_LOG_PATH`	`agent.log` when present.
`$AX_VARIANT_ID`	Derived variant id (also in run context).

Most tests should check final state: files, commands, endpoints, or product behavior the agent was supposed to produce. To inspect how the agent got there, query the events table with ax-run-query "$AX_RUN_ID" inside a platform test. Authentication is supplied automatically and permits read-only queries for that run only. The evaluated agent never receives this test-only command.

Do not read $AX_TRACE_PATH or $AXP_TRACE_PATH in current experiments. Those variables belong to the legacy pipeline, and session-data runs do not stage a standardized trace file. Let query, authentication, and zero-match assertions fail the test. Do not turn missing evidence into a pass with exit 0.

There is no first-class teardown hook after tests today. If setup hits a shared external system (cloud project, shared database), reset it in environment or product setup so parallel variants stay isolated. See the external-systems gotcha in Experiment Design.

Test results

After a run, each test becomes a pass/fail outcome on that variant. View them from the CLI:

ax run view <run_id> --tests
ax experiment query <EXPERIMENT_ID> --metric testPassRate --group-by test

On disk (and in downloaded run packages), latest results live under:

run.json.tests: latest-wins authority used by results derivation
tests/<test-name>.json: flat per-test results for the current execution

For SQL over uploaded session-data runs, query test_output (discover tables with SHOW TABLES / DESCRIBE when using legacy scopes). See Data and results and Runs.

A failing test is an evaluation outcome, not a harness crash: the run still completed, and pass rate reflects that assertion.

Analyze failing tests

Analyze a completed run with failing tests to determine whether the agent missed a task requirement or the test itself needs to change.

Prompt

Run `ax learn analyze-agent`, then help me determine why this run failed its tests. Compare the failing test output with the agent's activity and distinguish an agent failure from a problem with the tests.

Retesting

Retest is for the loop where the agent already did the work, but your assertion was too strict, too loose, or missing. Edit the experiment YAML, then re-run the full flat tests: list against the run's existing evidence with ax run rerun --tests. Retest does not drive the agent again. Fresh results merge into run.json and become authoritative for CLI, SQL, and the results UI.

Remote retest

For platform runs from ax experiment run, retest forks each target's live sandbox snapshot on the Fiveonefour platform:

ax run rerun <run_id> --tests --experiment experiment.yaml
ax run rerun <run_id> --tests --experiment experiment.yaml --variant <variant_id>

The run request must be terminal, with uploaded artifacts and a live sandbox snapshot for every selected run. Pass --variant to limit which variants retest; omit it to retest every run in the request. If a snapshot was garbage-collected or never captured, create new runs and retest those instead.

On completion, the CLI prints a per-run pass/fail table. Open details with ax run view <run_id> --tests, or query test_output for session-data runs. The ax run rerun command exits 0 when assertions fail but the retest infrastructure succeeds. Test scripts themselves must still exit non-zero when required evidence is missing.

Local retest

For local run groups under ./.axp/runs/:

ax run rerun --tests --local --experiment experiment.yaml --run <local_run_group_id>

Local retest stages each completed variant's available evidence, runs the edited tests, and writes updated artifacts back under .axp/runs/<id>/.

What retest can see

Retest runs the full edited tests: list against saved evidence from the original run. It can change tests (and test env vars); it cannot change agent, model, prompt, product, or setup.

Remote (snapshot): the platform's end-of-run sandbox, including /workspace and staged run artifacts ($AX_RUN_CONTEXT_PATH and $AX_AGENT_LOG_PATH when present). Snapshots expire after 14 days idle by default.
Local (fetched): persisted artifacts staged on disk. Not a full live /workspace or running services, so workspace-heavy tests may fail or report missing evidence.

Remote retests receive $AX_RUN_ID and the same run-scoped query access as the initial test pass, including transcript rows in events. Each remote test execution starts from the saved snapshot. Test-side mutations do not rewrite that snapshot, so a later retest starts from the same agent-produced state.

Secrets

Remote retest resolves ax://secrets/<slug> from the org store (same as ax experiment run). Local retest uses the same local-first rules as ax experiment run --local. Do not echo secrets from tests; snapshots can retain injected credentials until GC.

Publishing and cancellation

Overlapping remote retests run FIFO; non-overlapping ones can run in parallel. Each target publishes on its own: the latest successful publish wins. Assertion failures still publish; infra failures leave the previous result. Ctrl-C on the CLI only stops polling; the platform retest keeps going. Cancel via the platform API before a target publishes if you need to stop it.

History

CLI, SQL, and the dashboard show the latest test execution (run.json.tests, tests/*.json). Older executions stay under test-executions/ in the run package (ax run download <run_id>). Adding a new test to the YAML and retesting old runs is supported; the new name shows up like any other test.

Search documentation