Evals and datasets MCP tools
Read eval results and manage dataset snapshots through MCP.
All tools require agent_id and use your current organization unless you pass
org_id. Your organization must have agent evals enabled. Dataset tools also
require eval_id, and tools addressing an existing dataset require dataset_id.
| Tool | Use |
|---|---|
eval_list | List evals. |
eval_get | Get an eval. |
eval_trend | Read grader trends. |
eval_results | List eval results. |
eval_result_get | Read a case result. |
eval_result_detail | Read result assertions and events. |
eval_runs | List eval runs. |
eval_run_get | Read an eval run. |
dataset_list | List datasets. |
dataset_get | Get a dataset. |
dataset_cases | List draft cases. |
dataset_case_get | Get a draft case. |
dataset_versions | List published versions. |
dataset_version_get | Get published version metadata. |
dataset_version_cases | Read immutable version cases. |
dataset_reveal | Record a held-out reveal. |
dataset_create | Dataset create. |
dataset_update | Dataset update. |
dataset_case_create | Dataset case create. |
dataset_case_update | Dataset case update. |
dataset_publish | Dataset publish. |
List tools accept page_size (1–200, default 50) and page_token and return
{rows, nextPageToken}. eval_results and dataset_cases accept a split
filter: train, validation or test. Single-record tools return {record}.
Tool results include the same payload as both structured content and readable JSON.
Read published cases
Call dataset_version_cases with a positive integer version. It returns
immutable snapshot rows and contentHash, the server hash of the complete
snapshot. Test rows omit input and expected and carry
redacted_held_out: true until you record a reveal. Older versions without a
snapshot fail with FailedPrecondition.
dataset_version_get instead takes id, the metadata row UUID from
dataset_versions. dataset_case_get takes a draft case row id.
Reveal held-out evidence
Call dataset_reveal with agent_id, eval_id, dataset_id and version
only when you intend to reveal that held-out version. It writes an audit
record for your user. Reads never call it implicitly.
eval_result_get and eval_result_detail require eval_id, dataset_id,
run_id and case_key. Result reads preserve redaction markers. Held-out test detail reads
fail with FailedPrecondition and a message naming RecordReveal until your
user has a reveal for that version. Another user's reveal does not grant access.
Detail responses contain result, assertions, events,
assertionsNextPageToken and eventsNextPageToken. Pass
assertions_page_token and events_page_token independently, with page_size
to bound each list.
Publish a dataset
Create a dataset with dataset_create and values: {"name": "Search cases"}.
Use dataset_case_create with values containing case_key, split, input,
expected and tags. Updates use dataset_update or dataset_case_update
and require expected_updated_at; case updates also require the row id.
Call dataset_publish with values: {"version": 1}. The server snapshots
active cases and computes the content hash. Published versions cannot be
edited or reused. The dataset API reference lists
all writable fields. Use ax datasets pull
to save a local lock file.