ax datasets
List datasets, pull immutable version locks and publish draft cases.
| Command | Use |
|---|---|
datasets list | List an eval's datasets. |
datasets show | Read dataset metadata. |
datasets pull | Save a published version as a local lock file. |
datasets push | Upsert draft cases and publish a version. |
Every command requires --agent <AGENT_ID> and --eval <EVAL_ID> and accepts
--org <ORG_ID>. Agent, eval and dataset ids are UUIDs. Your organization must
have agent evals enabled. Arguments are optional unless marked required.
datasets list
List datasets belonging to an eval.
ax datasets list --agent <AGENT_ID> --eval <EVAL_ID> --json| Argument / flag | What it does |
|---|---|
--limit <N> | Rows per page, default 50, maximum 200. |
--page-token <TOKEN> | Continue from nextPageToken. |
--json | Print {rows, nextPageToken}. Otherwise print one record per row. |
datasets show
Read dataset metadata as {record} JSON.
ax datasets show <DATASET_ID> --agent <AGENT_ID> --eval <EVAL_ID>datasets pull
Save a published snapshot to evals/datasets/<DATASET_ID>.lock.json.
ax datasets pull <DATASET_ID>@<VERSION> --agent <AGENT_ID> --eval <EVAL_ID>| Argument / flag | What it does |
|---|---|
<DATASET_ID>@<VERSION> | Required. Dataset UUID and positive integer version. |
--include-held-out | Record a reveal for your user and this version first, then include test cases. |
Pull reads every snapshot page. It omits test cases by default, even if you
previously revealed them. With --include-held-out, the reveal remains recorded
even if a later page fails. The file is replaced only after all pages succeed.
Draft edits cannot change a published snapshot. Older metadata-only versions
without snapshots fail with FailedPrecondition; publish a new version.
Lock file format
The shared format is defined in
schemas/dataset-lock.schema.json.
Use this format when reading dataset locks from your evaluation code.
{
"datasetId": "38bd4dc8-3e41-4c61-8647-a348e236eaf7",
"version": 3,
"contentHash": "server-provided-content-hash",
"cases": [
{
"caseKey": "search-by-name",
"split": "train",
"input": { "query": "Ada" },
"expected": { "found": true },
"tags": ["search"]
}
]
}input and expected accept any JSON value. tags is an array of strings.
Case keys are unique, trimmed, nonempty and at most 500 UTF-16 code units.
Tags are at most 200 trimmed, nonempty strings. Push rejects integer JSON
values that cannot be represented exactly; encode large identifiers as strings. Splits are train, validation or test.
contentHash is the server's hash of the complete snapshot, including test
cases omitted from the file. It is not a checksum of the local file.
datasets push
Upsert the file's cases into an existing dataset, then publish its version.
ax datasets push <FILE> --agent <AGENT_ID> --eval <EVAL_ID>Use the lock file format above. Set version to a new positive integer;
contentHash is ignored on push. Cases match by caseKey. Cases added after the requested version are rejected before any draft edits.
Existing cases use their latest update stamp to refuse concurrent edits. Cases omitted from the
file remain in the draft and may appear in the published snapshot. Create the
dataset first with the API or dataset_create.
Draft case writes are separate operations. If one fails, earlier draft edits
remain, and the command does not publish. Publication copies all active cases
atomically and returns {record} JSON with the server's content hash. An
already-published version cannot be reused. Pull the new version to save its
server hash locally.