Products
The system under test, staged into each variant.
products defines the agent-facing surface under test. A product can be a CLI, API, MCP server, SDK, docs surface, or other tool the agent uses to complete the prompt.
Accepted shapes:
# String form: one setup script with a generated product name.
products: "npm install -g my-cli"
# Object form: one named product.
products:
name: my-cli
type: CLI
version: "1.2.0"
setup: "npm install -g my-cli"
# List form: multiple named products.
products:
- name: my-cli
type: CLI
version: "1.2.0"
setup: "npm install -g my-cli"Product object
A product object names what you want to compare. Product metadata is recorded so Results can filter and group by product and product version.
| Field | Required | Type / values | Notes |
|---|---|---|---|
name | Yes | Kebab-case string | Product coordinate. Recorded for filtering in Results. |
type | No | Product type | Defaults to Other. |
setup | Yes | Setup | Prepares the product before the agent runs. |
version | No | String | Product version. Quote numeric versions, such as "25.3". |
commit | No | String | Source commit of the product under test. |
description | No | String | Human-readable notes. |
tags | No | String list | Free-form labels. |
Product type
Allowed values: CLI, MCP, API, Skill, SDK, Schema, Docs, Marketing, Agents.md, Other.