Reviewing behavior observations
Define a behavior, let Fiveonefour classify your runs, and review each observation with your coding agent.
Preview
Behaviors are in preview and available only to organizations that have them enabled. Commands and output may change.
A behavior is something you want to find in agent runs, written as a natural-language definition, such as "the agent chose not to use the skill". Fiveonefour classifies each run against your behaviors and stores what it found as observations: one per matching event, each with a status and a rationale. You, or your own coding agent, then review each observation with an upvote or downvote and a reason, so you know how far to trust the classification.
- Define a behavior
- Classify your runs
- Read the observations
- Read the criteria behind an observation
- Vote with a reason
Every step works from the CLI and from the MCP server. Your coding agent needs nothing else: ax behavior --help describes the same loop, and each read names the call that comes next.
1. Define a behavior
Create a behavior from a name and one definition of what it looks like in a transcript:
ax behavior create --name "Skips the skill" \
--definition "The agent had the skill available and chose not to use it."The definition can also come from a file (--definition @skips-skill.md) or stdin (--definition -). The MCP equivalent is behavior_create.
2. Classify your runs
A new behavior starts in manual mode: runs are not classified with it as they are processed, and manual is the only mode you can set for now. Label the evidence yourself when you see the behavior in a transcript, then list and count those observations across runs with ax behavior query. ax behavior mode <BEHAVIOR_ID> manual sets a behavior back to manual. The MCP tool is behavior_mode.
3. Read the observations
List every behavior's result on one run:
ax behavior run-results <RUN_ID>Or list one version's observations, with the reviews each one already has:
# One run
ax behavior observations <BEHAVIOR_VERSION_ID> --run <RUN_ID>
# Every run of an experiment
ax behavior observations <BEHAVIOR_VERSION_ID> --experiment <EXPERIMENT_ID>Each observation names its behavior_version_id and its anchor: the exact event it was classified on. The ax behavior observations table shows the anchor in its DERIVATION, SEQ, TOOL CALL, and ROW HASH columns, next to RUN; ax behavior run-results names the ax behavior observations call for each behavior with observations. Add --json for the full fields. The MCP tools are behavior_run_results and behavior_observations.
4. Read the criteria behind an observation
Read the version that produced an observation:
ax behavior get <BEHAVIOR_VERSION_ID>A behavior's version prints its definition. A judge version prints the instructions and full prompt every classification call sends. Every version ends with what each status means:
| Status | Meaning |
|---|---|
matched | The event meets the behavior's definition or the judge instructions. Labeled observations carry this status. |
failed_attempt | A clearly linked attempt at the behavior that failed. It counts toward the behavior's rate as matched does. |
unresolved | The evidence is not enough to decide: missing evidence, not a negative result. |
excluded | The evidence shows the event does not meet the judge instructions. |
ax behavior get lists only the statuses that version can carry. The MCP tool is behavior_get.
5. Vote with a reason
Vote up when the observation was classified correctly and down when it was not, naming the observation by the anchor its row lists:
# Read the anchor: the RUN, DERIVATION, SEQ, TOOL CALL, and ROW HASH columns
ax behavior observations <BEHAVIOR_VERSION_ID> --run <RUN_ID>
# Review that observation with those values
ax behavior review <BEHAVIOR_VERSION_ID> down --run <RUN_ID> \
--derivation <DERIVATION> --seq <SEQ> --tool-call <TOOL_CALL> \
--row-hash <ROW_HASH> \
--comment "The agent read the skill, then chose a shell command instead."| Argument / flag | What it does |
|---|---|
up, down, or clear | Required. Your vote, or clear to remove your review. |
--run, --derivation, --seq, --row-hash | Required for a platform observation: its run and anchor, as ax behavior observations lists them. |
--observation | A labeled observation's OBSERVATION_ID, in place of the run and anchor. |
--tool-call | The anchor's tool_call_id; omit it when empty, as for a message. |
--comment | Why the classification was right or wrong, at most 2000 characters. Not allowed with clear. |
--product | Behavior versions only: the product a platform observation was bound to. |
Review a labeled observation, one whose LABELED_BY names a person or an agent, by its ID:
ax behavior review <BEHAVIOR_VERSION_ID> up --observation <OBSERVATION_ID> \
--comment "The span is the commit."A removed label takes no new votes, though you can still clear a review you left on it.
You have one review per observation. A new vote or comment replaces it, and the review is recorded as yours, also when your coding agent casts it with your API key. Every member sees each review with its comment. The MCP tool is behavior_review, and members can also vote on a behavior's observations in the web app, under Behaviors.
An observation whose event no longer resolves exactly, for example after its run's session data is rebuilt, reads as stale. Its reviews stay on it and are never moved to another event.
Label evidence yourself
When you or your coding agent see a behavior in a transcript, label that evidence as an observation of an exact behavior version. Copy event IDs from the run's transcript, then select single events, spans, or both:
# Event IDs are in the EVENT_ID column
ax run view <RUN_ID> --transcript
# One observation of the behavior, from all selected evidence
ax behavior label --behavior <BEHAVIOR_VERSION_ID> --run <RUN_ID> \
--event <EVENT_ID> --span <START_EVENT_ID>:<END_EVENT_ID>| Flag | What it does |
|---|---|
--behavior | Required. The exact version UUID from ax behavior list. A behavior ID or name is refused. |
--run | Required. The run every selected event belongs to. |
--event | One event. Repeat it for events that are not consecutive. |
--span | An inclusive range of events in transcript order. Repeat it, and combine it with --event. |
--agent-model | For a coding agent: the model it is running, when its harness does not report one. |
All selectors together make one observation of one behavior occurrence, with each event counted once. Label separate occurrences with separate commands. The same version, run, and evidence, in any order, returns the existing observation instead of a new one. No classification runs: the label is recorded as yours, or as your coding agent's with its harness and model (unknown when neither reports it). The MCP tool is behavior_label, with event_ids, spans and agent_model.
Remove a wrong label by the observation ID ax behavior label printed:
ax behavior unlabel <OBSERVATION_ID>The observation stops counting everywhere. Its history keeps the label and records who removed it and when, and removing it again changes nothing. Platform observations can't be unlabeled: rate them with ax behavior review instead. The MCP tool is behavior_unlabel.
Labels appear next to the platform's observations in ax behavior observations and ax behavior run-results --json. ax behavior query returns labels only. The OBSERVATION_ID, EVIDENCE and LABELED_BY columns show each observation's ID, its event IDs or spans, and who made it: platform, Olivia Kane · human, or Olivia Kane · agent (claude-code · claude-opus-5-5). With --json, each observation has created_by. Add --platform-only to ax behavior observations or ax behavior run-results to leave labels out. ax behavior query --aggregate counts labels, counting an event once: labels that share an event count once together.
Suggested: let your coding agent review
Give your coding agent a run or experiment and ask it to review the observations. It reads them, follows each one to its criteria, and votes with a reason you can read later:
Review the behavior observations on run <RUN_ID>. For each one, check it
against the criteria it was classified by and vote up or down with a reason.Read the downvotes and their reasons first: they show where a definition is too broad or too narrow. Refine the definition with ax behavior create --behavior <BEHAVIOR_ID>.