Data, Results and Insights

Individual runs tell you what one agent did once. To answer a product question you need them added up, so Fiveonefour rolls a set of runs into a table: that table is a result.

Three entry points cover almost everything: ax experiment query for rollups across an experiment, ax run query for one run, and ax insight for a query you want to keep and share.

Two ways to query

Both ax experiment query and ax run query accept either style:

• SQL is the primary path: write it yourself, scoped to one experiment or one run.
• Flags are an experimental shorthand for common rollups. You choose metrics, filters, and grouping, and Fiveonefour compiles the SQL.

With SQL

Add the sql subcommand. Start with --tables to see what is queryable, then write the query:

# What tables and columns exist?
ax run query '<RUN_ID>' sql --tables
ax run query '<RUN_ID>' sql --tables events

# Failing tests in one run
ax run query '<RUN_ID>' sql "SELECT test_name, exit_code FROM test_analysis WHERE exit_code != 0" --format table

# Across an experiment, from a file, saved as an insight
ax experiment query <EXPERIMENT_ID> sql @query.sql --save "Failing tests"

Available tables

Run dimensions such as experiment, variant, agent, model, prompt, product, environment, and status are available on these tables, so there is no separate metadata table. --tables also lists typed sd_current_* projections (for example sd_current_tool_calls) for queries that need a narrower, typed shape than events.

events is an observations table, not a deduplicated timeline. The same activity can appear in several source records. Filter its source column (setup, acp, stdout, hooks, otel, transcript, harness, other). Its ts column is nullable because some records have no trustworthy time, and payload keeps the source-specific body as JSON-encoded text.

Query limits

Remote SQL runs with server-enforced limits; SETTINGS clauses in your SQL are rejected. Authenticated queries have no scan limits: they can read as much as the analytical store can serve within a 13 minute execution ceiling, and each query returns at most 10,000 rows and 16 MiB. Publicly shared insight pages run with much smaller scan and time limits.

Result rows are capped but reads are not, so keep aggregation in SQL instead of paginating raw rows out for local processing: one aggregate query is faster, cheaper, and returns the numbers you actually want. The pattern that scales to full experiments is one statement that flags each run in a subquery, then joins every run back in for the denominator:

-- Share of runs whose transcript mentions a term, by agent and mode
SELECT m.agent, m.agent_mode,
  count() AS run_count,
  countIf(t.mentioned = 1) AS matched_count,
  round(matched_count / run_count, 3) AS rate
FROM measurements AS m
LEFT JOIN (
  SELECT run_id, max(match(payload, '(?i)mongodb|mongoose')) AS mentioned
  FROM events
  WHERE kind IN ('message', 'reasoning', 'tool_call')
  GROUP BY run_id
) AS t ON t.run_id = m.run_id
GROUP BY m.agent, m.agent_mode

Run it as one ax experiment query <EXPERIMENT_ID> sql @query.sql call: the experiment scope is applied server-side, runs with no matching events stay in the denominator, and only the grouped rates come back.

With flags (experimental)

Flag-based modeled queries are experimental, so expect the surface to move. Three choices decide what comes back:

• Filters pick which runs count: --filter dim=value, the axis flags (--agent, --prompt, --product, --environment), and --experiment-version.
• Grouping splits those runs, one row per group: --group-by variant|agent|model|product|environment|prompt|test.
• Metrics decide what is measured for each group, one column each: test outcomes (testPassRate, testsPassed, testsFailed), spend (cost, tokens), time (wallClockTime), and tool behavior (toolCalls, toolFailures). See experiment query for the defaults and the full set.

# What can I group and measure on this experiment?
ax experiment query <EXPERIMENT_ID> --dimensions

# Pass rate per product
ax experiment query <EXPERIMENT_ID> --metric testPassRate --group-by product

# Cost and tokens per prompt, claude only
ax experiment query <EXPERIMENT_ID> --metric cost,tokens --group-by prompt --filter agent=claude

# Keep it as an insight
ax experiment query <EXPERIMENT_ID> --metric testPassRate --save "Pass rate by product"

You can group and filter along any dimension the experiment defined (variant, agent, model, prompt, product, environment, experiment version) plus execution-derived fields such as run status and test success.

Each row averages or totals every run in that group, so check RUNS before trusting a comparison: a row backed by 50 runs means more than a row backed by one. --dry-run prints the compiled SQL without running it, and --json gives machine-readable output.

Legacy ax results view, ax results query, and ax results scatter spellings still work (legacy commands); prefer ax experiment query and ax run query.

Test-aware runs

If a run contains evidence that the agent knew it was being tested, or that it changed how it worked because of that, the platform classifies it as "test aware". Those runs measure the observer instead of the product, so experiment statistics leave them out by default: modeled summaries, SQL in experiment scope, scatter points, insights that replay in experiment scope, and the Results view in the web app. Nothing is deleted or hidden. The runs stay in run lists and in single-run views, and the modeled query and the legacy summary and scatter renderers report how many were held back (experiment-scoped SQL applies the same exclusion silently):

# Rows table on stdout, then this note on stderr:
# 6 test-aware runs excluded · pass --include-test-aware to include them
ax experiment query <EXPERIMENT_ID> --metric testPassRate --group-by agent

Add --include-test-aware to count them, in either query mode. Those runs may not mimic real-world agent behavior.

# Modeled summary over every run
ax experiment query <EXPERIMENT_ID> --metric testPassRate --include-test-aware

# Experiment-scoped SQL over every run
ax experiment query <EXPERIMENT_ID> sql @query.sql --include-test-aware

The same opt-in is the include_test_aware field on POST /api/v1/experiments/{id}/query and the experiment_query MCP tool, and --include-test-aware on the legacy ax results view (alias get) and ax results scatter spellings. The web app's Results view applies the exclusion but does not yet report the held-back count or offer an opt-in of its own.

Saving a query with --save keeps the choice you saved it under: an insight saved with --include-test-aware replays over those runs every time, including through a share link. Saving without the flag replays with the exclusion.

Test awareness in SQL

The run_analysis table, one row per run, carries the served verdict in its awareness_level column:

SQL in experiment scope adds awareness_level = '' to reads of run_analysis for you unless you pass --include-test-aware. Queries outside that scope are never narrowed this way: ax run query reads the single run you named, and an org-scoped insight reads every run in the org. Apply the predicate yourself to get the same exclusion:

-- Runs per agent, leaving out runs where the agent noticed it was being tested
SELECT agent, count() AS runs
FROM run_analysis
WHERE awareness_level = ''
GROUP BY agent

Saved insights

An insight is org-scoped saved SQL with a heading and an immutable slug. Create one from a query with --save, or directly:

ax insight create "Cross-experiment cost" --sql @query.sql
ax insight list
ax insight view <slug>
ax insight share <slug>
ax insight edit <slug> --heading "Updated title"
ax insight export <slug> out.csv --format csv
ax insight delete <slug>

Insights re-execute live against your org's uploaded data on view and export. Listing returns metadata only and does not rerun SQL.

ax insight share prints a public capability URL. The first share publishes a frozen copy of the heading, description, query, data family, execution scope, and whether the query includes runs classified as "test aware". That page reruns the frozen query against live data, so rows can change while later edits to the source insight do not. Repeated shares return the same URL, and deleting the insight is the only way to revoke it in this release.

Local data

Use local data when you run an experiment on your machine.

Query local runs

ax experiment run --local always writes the run data to .axp/runs/<RUN_REQUEST_ID>/. Query that data directly from your machine, starting with SHOW TABLES to discover the available local tables (runs, tests, setup_logs, setup_checks, harness_spans, and artifacts, a local projection of the run tree).

# Discover the available local tables
ax run query '<RUN_ID>' sql "SHOW TABLES" --local --format table

# Query the local runs table
ax run query '<RUN_ID>' sql "SELECT * FROM runs LIMIT 10" --local --format table

Local tables differ from the tables available through remote SQL, so start with SHOW TABLES and adapt remote queries rather than reusing them verbatim. See Local runs for the artifact layout.

Upload a local run

Local run data can be uploaded to your org at any time. When authenticated, AX attempts this automatically after the run finishes. Publication is best effort, so the local record remains available when an automatic upload is skipped or fails. If you weren't signed in when the run finished, you can sign in and upload it anytime:

ax auth login
ax run upload <RUN_REQUEST_ID>