Data, Results and Insights

Individual runs tell you what one agent did once. To answer a product question you need them added up, so Fiveonefour rolls a set of runs into a table: that table is a result.

Three entry points cover almost everything: ax experiment query for rollups across an experiment, ax run query for one run, and ax insight for a query you want to keep and share.

Two ways to query

Both ax experiment query and ax run query accept either style:

• SQL is the primary path: write it yourself, scoped to one experiment or one run.
• Flags are an experimental shorthand for common rollups. You choose metrics, filters, and grouping, and Fiveonefour compiles the SQL.

With SQL

Add the sql subcommand. Start with --tables to see what is queryable, then write the query:

# What tables and columns exist?
ax run query '<RUN_ID>' sql --tables
ax run query '<RUN_ID>' sql --tables events

# Failing tests in one run
ax run query '<RUN_ID>' sql "SELECT test_name, exit_code FROM test_analysis WHERE exit_code != 0" --format table

# Across an experiment, from a file, saved as an insight
ax experiment query <EXPERIMENT_ID> sql @query.sql --save "Failing tests"

Available tables

Run dimensions such as experiment, variant, agent, agent_artifact_sha256, model, prompt, product, environment, and status are available on these tables, so there is no separate metadata table. The artifact digest identifies the exact agent bytes provisioned for a run and is empty for runs without retained provisioning evidence. agent includes the agent's version (for example claude-2.1.280) while agent_name does not (claude), so group by agent_name to compare agents across versions.

Table discovery (ax run query <RUN_ID> sql --tables, or ax experiment query <EXPERIMENT_ID> --table) lists the six tables above, and adding a table name describes one table's columns, join keys, and allowed values. SHOW TABLES lists the same six. A table is readable only if discovery lists it, whatever its prefix: SQL that names any other table is refused, and the error names the table above to read instead. The MCP query tools and the REST run and experiment query routes offer the same discovery through a tables field.

events is an observations table, not a deduplicated timeline. The same activity can appear in several source records. Filter its source column (setup, acp, stdout, hooks, otel, transcript, harness, other). Its ts column is nullable because some records have no trustworthy time, and payload keeps the source-specific body as JSON-encoded text.

Normalized events and tool interactions

normalized_events has one row per session event, in the same shape whichever agent produced it, ordered within a run by seq. Its kind is user_message, assistant_message, thinking, tool_call, tool_result, or unknown. On tool_call rows, tool_kind classifies the tool as read, edit, delete, move, search, web_search, execute, think, fetch, switch_mode, or other, and tool_name keeps the agent's own name for it. On tool_result rows, is_error says whether the tool failed. On tool_call rows, skill_names lists the skills the call uses (below), and it is empty on every other kind. When the agent recorded a tool result but not its call, the result follows a tool_call row with tool_kind other and no tool_name or ts. payload holds the complete event as JSON-encoded text, and observation_ids lists the events rows the event came from (each value is an events.event_id in the same run).

tool_interactions has one row per tool call, joined to the results that answer it. It carries the call's tool_kind and tool_name, an is_error that is true when any result failed and null when no result was recorded, and duration_ms, file_paths, command, mcp_server, mcp_tool, and skill_names.

A tool call uses a skill when it invokes the skill (Claude's Skill tool) or reads the skill's instructions file: a file read of a SKILL.md, or a command or script that names one by path or URL. skill_names holds the name of the directory that holds each such SKILL.md, once each, in the order the call names them, so cat .agents/skills/release-notes/SKILL.md lists release-notes. Reading other files in a skill's directory, writing a SKILL.md, and a path whose directory is a shell variable or a wildcard ($s/SKILL.md, */SKILL.md) list none. The same filter finds skill use on every agent:

-- Skills each run used, whatever its agent
SELECT run_id, agent_name, arrayJoin(skill_names) AS skill, count() AS calls
FROM normalized_events
WHERE notEmpty(skill_names)
GROUP BY run_id, agent_name, skill
ORDER BY run_id, skill

Tool call ids are unique only within one agent session, and a run can contain several sessions, so join tool rows on session_native_id as well as tool_call_id. Runs processed before these tables existed have no rows in them. For every column, run ax run query '<RUN_ID>' sql --tables normalized_events or sql --tables tool_interactions.

ts is the time the agent recorded for the event, and it is null when the agent recorded none; AX never fills in a time. Order events by seq, which is always set. duration_ms is null when its call or last result has no time. Which events carry a time depends on the agent:

-- Events of each kind, by agent
SELECT agent, kind, count() AS events
FROM normalized_events
GROUP BY agent, kind
ORDER BY agent, kind

-- Failed shell commands, by agent
SELECT agent, count() AS failed_commands
FROM tool_interactions
WHERE tool_kind = 'execute' AND is_error
GROUP BY agent

Query limits

Remote SQL runs with server-enforced limits; SETTINGS clauses in your SQL are rejected. Authenticated queries have no scan limits: they can read as much as the analytical store can serve within a 13 minute execution ceiling, and each query returns at most 10,000 rows and 16 MiB. Publicly shared insight pages run with much smaller scan and time limits.

Result rows are capped but reads are not, so keep aggregation in SQL instead of paginating raw rows out for local processing: one aggregate query is faster, cheaper, and returns the numbers you actually want. The pattern that scales to full experiments is one statement that flags each run in a subquery, then joins every run back in for the denominator:

-- Share of runs whose transcript mentions a term, by agent and mode
SELECT m.agent, m.agent_mode,
  count() AS run_count,
  countIf(t.mentioned = 1) AS matched_count,
  round(matched_count / run_count, 3) AS rate
FROM measurements AS m
LEFT JOIN (
  SELECT run_id, max(match(payload, '(?i)mongodb|mongoose')) AS mentioned
  FROM events
  WHERE kind IN ('message', 'reasoning', 'tool_call')
  GROUP BY run_id
) AS t ON t.run_id = m.run_id
GROUP BY m.agent, m.agent_mode

Run it as one ax experiment query <EXPERIMENT_ID> sql @query.sql call: the experiment scope is applied server-side, runs with no matching events stay in the denominator, and only the grouped rates come back.

With flags (experimental)

Flag-based modeled queries are experimental, so expect the surface to move. Three choices decide what comes back:

• Filters pick which runs count: --filter dim=value, the axis flags (--agent, --prompt, --product, --environment), and --experiment-version.
• Grouping splits those runs, one row per group: --group-by variant|agent|model|product|environment|prompt|test.
• Metrics decide what is measured for each group, one column each: test outcomes (testPassRate, testsPassed, testsFailed), spend (cost, tokens), time (wallClockTime), and tool behavior (toolCalls, toolFailures). See experiment query for the defaults and the full set.

# What can I group and measure on this experiment?
ax experiment query <EXPERIMENT_ID> --dimensions

# Pass rate per product
ax experiment query <EXPERIMENT_ID> --metric testPassRate --group-by product

# Cost and tokens per prompt, claude only
ax experiment query <EXPERIMENT_ID> --metric cost,tokens --group-by prompt --filter agent=claude

# Keep it as an insight
ax experiment query <EXPERIMENT_ID> --metric testPassRate --save "Pass rate by product"

You can group and filter along any dimension the experiment defined (variant, agent, model, prompt, product, environment, experiment version) plus execution-derived fields such as run status and test success.

Each row averages or totals every run in that group, so check RUNS before trusting a comparison: a row backed by 50 runs means more than a row backed by one. --dry-run prints the compiled SQL without running it, and --json gives machine-readable output.

Legacy ax results view, ax results query, and ax results scatter spellings still work (legacy commands); prefer ax experiment query and ax run query.

Test-aware runs

If a run contains evidence that the agent knew it was being tested, or that it changed how it worked because of that, the platform classifies it as "test aware". Those runs measure the observer instead of the product, so experiment statistics leave them out by default: modeled summaries, SQL in experiment scope, scatter points, insights that replay in experiment scope, and the Results view in the web app. Nothing is deleted or hidden. The runs stay in run lists and in single-run views, and the modeled query and the legacy summary and scatter renderers report how many were held back (experiment-scoped SQL applies the same exclusion silently):

# Rows table on stdout, then this note on stderr:
# 6 test-aware runs excluded · pass --include-test-aware to include them
ax experiment query <EXPERIMENT_ID> --metric testPassRate --group-by agent

Add --include-test-aware to count them, in either query mode. Those runs may not mimic real-world agent behavior.

# Modeled summary over every run
ax experiment query <EXPERIMENT_ID> --metric testPassRate --include-test-aware

# Experiment-scoped SQL over every run
ax experiment query <EXPERIMENT_ID> sql @query.sql --include-test-aware

The same opt-in is the include_test_aware field on POST /api/v1/experiments/{id}/query and the experiment_query MCP tool, and --include-test-aware on the legacy ax results view (alias get) and ax results scatter spellings. The web app's Results view applies the exclusion but does not yet report the held-back count or offer an opt-in of its own.

Saving a query with --save keeps the choice you saved it under: an insight saved with --include-test-aware replays over those runs every time, including through a share link. Saving without the flag replays with the exclusion.

Test awareness in SQL

The run_analysis table, one row per run, carries the served verdict in its awareness_level column:

SQL in experiment scope adds awareness_level = '' to reads of run_analysis for you unless you pass --include-test-aware. Queries outside that scope are never narrowed this way: ax run query reads the single run you named, and an org-scoped insight reads every run in the org. Apply the predicate yourself to get the same exclusion:

-- Runs per agent, leaving out runs where the agent noticed it was being tested
SELECT agent, count() AS runs
FROM run_analysis
WHERE awareness_level = ''
GROUP BY agent

Saved insights

An insight is org-scoped saved SQL with a heading and an immutable slug. Create one from a query with --save, or directly:

ax insight create "Cross-experiment cost" --sql @query.sql
ax insight list
ax insight view <slug>
ax insight share <slug>
ax insight edit <slug> --heading "Updated title"
ax insight export <slug> out.csv --format csv
ax insight delete <slug>

Insights re-execute live against your org's uploaded data on view and export. Listing returns metadata only and does not rerun SQL.

ax insight share prints a public capability URL. The first share publishes a frozen copy of the heading, description, query, data family, execution scope, and whether the query includes runs classified as "test aware". That page reruns the frozen query against live data, so rows can change while later edits to the source insight do not. Repeated shares return the same URL, and deleting the insight is the only way to revoke it in this release.