GitHub CI

Connect a GitHub repository to an AX experiment so that commenting !ax run on a pull request — or checking the run checkbox in the AX comment — runs that experiment against the PR's code and reports the results right in the PR — a rich comment plus a non-blocking status check.

This is the CI/PR trigger: the same run engine as ax run, driven by a PR comment instead of your terminal, so agent-experiment results sit next to code review.

How it works

1. Whenever the PR's code changes, your repo's CI builds the product and uploads it to AX with ax ci upload, keyed to the commit.
2. A developer comments !ax run on the PR, or checks the run box in the AX comment.
3. AX posts instant feedback (a "queued" comment and a pending ax run check), then waits — up to ~20 minutes — for that commit's build to arrive (the rendezvous).
4. AX runs the repo's linked experiment against that build, then edits the comment into a per-variant result and flips the check to pass or fail.

A run triggered from GitHub has no developer laptop to stage local files from, so the PR's software reaches the sandbox as a content-addressed artifact that CI uploads. The experiment consumes it through a source-less files: slot — a files entry with a name and dest but no source. CI fills that slot per commit; everything else about the experiment is unchanged.

Prerequisites

• An AX account with the CLI signed in (ax auth login). See Installation.
• An experiment that tests your product.
• Admin on the GitHub organization or repository, to install the AX GitHub App.

Set up a repository

Setup is one-time per repository. After it, any PR accepts !ax run.

1. Install the AX GitHub App

In the AX platform, open your organization's Settings → GitHub, choose Connect, and install the AX GitHub App on the account and repositories you want. When you return from GitHub, the installation is linked to your active AX org automatically.

The link is owned by your org, so it survives the App being uninstalled and reinstalled. For the !ax run trigger specifically, the App uses read access to repository contents (to read the experiment at the PR's commit), write access to pull-request comments, and write access to status checks — see Permissions and data for the full list and what AX does with it.

2. Declare a build slot in the experiment

Add a source-less files: entry to the experiment — a slot with no source, which CI fills per commit:

files:
  - name: cli # the slot handle you pass to `ax ci upload`
    dest: tools/mycli # where the build lands in each variant's /workspace

Reference tools/mycli from the experiment's setup and prompts exactly as you would any staged file. Declare one slot per product artifact your CI builds.

3. Register and link the experiment

Register the experiment on the platform, then point the repo's PRs at it:

ax experiment push my-experiment.yaml
ax integrations gh connect   # install/connect the GitHub App in the browser (once)
ax integrations gh link my-experiment owner/repo

ax experiment push uploads the experiment definition without running it (the experiment id comes from the YAML — here my-experiment). ax integrations gh connect opens the org integrations settings page for the CSRF-safe App install. ax integrations gh link then points !ax run on owner/repo's PRs at that registered experiment. The experiment must already be registered, so push first; re-linking a repo overwrites its target. Inspect the current links with ax integrations gh list. See ax integrations.

Alternatively, link the repo to an experiment YAML committed in the repo itself instead of a stored experiment — the file is read at each PR's head commit, so the PR under test can change the experiment too:

ax integrations gh link --path .axp/experiment.yaml owner/repo

With a path link there is nothing to ax experiment push; the file doesn't have to exist yet when you create the link (a PR that adds it can be the first to run).

4. Build and upload the product from CI

Add a workflow that, on each pull request, builds the product and runs ax ci upload once per source-less slot. ax ci upload resolves the slot's workspace destination from the linked experiment and authenticates with an org API key stored as an Actions secret named AX_API_KEY.

name: ax
on:
  pull_request:

jobs:
  upload-build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          # Build the PR's head commit — the commit AX keys the artifact to.
          ref: ${{ github.event.pull_request.head.sha }}

      - name: Build the product
        run: cargo build --release # produces ./target/release/mycli

      - name: Install the AX CLI
        run: |
          bash <(curl -fsSL https://dl.514.ax/install.sh) ax
          echo "$HOME/.fiveonefour/bin" >> "$GITHUB_PATH"

      - name: Upload the build for the PR trigger
        run: ax ci upload cli ./target/release/mycli --commit "${{ github.event.pull_request.head.sha }}"
        env:
          AX_API_KEY: ${{ secrets.AX_API_KEY }}

ax ci upload reads the repo from $GITHUB_REPOSITORY automatically (set in GitHub Actions); override it with --repo owner/repo elsewhere.

Generate the AX_API_KEY value from your organization's API-key settings in the platform, and add it as a repository (or organization) Actions secret. (The legacy AXP_API_KEY env var name still works.) ax ci upload re-uploads nothing when your org already has an identical build, so running it on every commit is cheap.

Trigger a run

Trigger a run either way:

• Check the box. The AX comment on the PR carries a task-list checkbox (for example, Run the linked experiment against this PR). Ticking it starts the run; the comment immediately updates into the queued state, so the box resets itself.
• Comment !ax run. The command must be on its own line and exactly !ax run — trailing arguments are not accepted.

AX immediately:

• posts (or updates) its comment to **ax run** ⏳ queued — waiting for the build to upload…, and
• creates a non-blocking ax run status check on the PR's head commit.

It then waits for CI to upload every build slot for that commit, runs the experiment against the build, and edits the comment into a result when the run finishes.

Anyone with write access to the repo can trigger a run. For a !ax run comment, the author must be an OWNER, MEMBER, or COLLABORATOR; bot and agent accounts can trigger it too, so !ax run works from your own automations. The checkbox relies on GitHub's own rule that only users with write access can tick a box in the App's comment — and only the App's own comment is a live trigger surface, so a checkbox copied into any other comment is ignored.

Read the results

When the run finishes, AX edits its comment in place into:

• a headline — Experiment set completed. or Experiment set failed. with a See results → link to the run on the platform;
• a one-line summary — 1/1 variants passed · 5/5 tests · $0.15 · 2.5k tokens; and
• a per-variant table with inline deltas versus the variant's most recent prior run:

| Variant     | Status | Tests | Cost          | Tokens      | Runtime      |
| ----------- | ------ | ----- | ------------- | ----------- | ------------ |
| claude · p0 | ✅     | 5/5   | $0.15 (-0.01) | 2.5k (-0.5k)| 4m 3s (-1m)  |

A variant passes only if it ran cleanly and every test passed; a variant that errored, was canceled, or produced no results shows as a failure row. The ax run check mirrors the same verdict.

The check is non-blocking by default. To make a green !ax run required before merge, mark the ax run check required in the repository's GitHub branch-protection rules — no change on the AX side.

Re-run, and one run per PR

Re-run the experiment by checking the re-run box in the result comment, commenting !ax run again, or clicking Re-run on the ax run check.

Each PR keeps a single run. Starting a new run — or pushing a new commit to the branch — cancels any run still in flight and marks its old comment and check as superseded. Runs always execute against the PR's head commit, so a re-run after a new push evaluates the new code (once CI uploads its build).

Permissions and data

The PR trigger runs on a GitHub App you install — not an OAuth app against your personal account — and it is entirely opt-in and per-repository: only the repositories you explicitly grant are ever accessible, and uninstalling the App (or removing a repository from it) revokes that access immediately.

What AX requests

At install time GitHub asks you to grant the App four repository permissions. AX requests only what the trigger needs, and nothing account-wide:

The App always acts under its own bot identity — never as you or any other user. It gets no write access to your code, and no access at all to issues, Actions, repository secrets, members, or any account-level data.

What GitHub sends AX

Once installed, GitHub notifies AX of a small set of events on the repositories you granted:

• Pull-request comments — to detect a !ax run command or a tick of the run checkbox in the App's own comment; every other comment and edit is ignored.
• Pull requests opened, updated, or closed — to post the one-time intro comment when a PR opens, and to cancel an in-flight run when a new commit is pushed or the PR closes (its artifact is keyed to the superseded commit).
• Re-run clicks on the ax run check.
• Installation changes — install, uninstall, suspend, and repository-selection changes — to keep the App's link to your AX org current.

Every incoming event is cryptographically signature-verified before AX acts on it, and duplicate deliveries are discarded.

What AX reads, writes, and stores

• Reads only the experiment file at the PR's head commit, plus the head commit sha. AX does not clone your repository or read arbitrary source — the one file it fetches is the experiment you linked. Your product's own code reaches a run solely as the build artifact your CI uploads with ax ci upload (authenticated by your AX_API_KEY, not by the App), never pulled by the App.
• Writes only the single AX comment and the ax run status check on the PR. Nothing else in your repository is modified.
• Stores the link between a GitHub installation and the AX org that connected it (with the account name, the repository-selection GitHub reports, and whether GitHub has suspended the App), plus per-PR bookkeeping for the run — the head commit, the run's status, and which comment and check to keep updated. The results the comment reports live with the rest of your run data.

To act on your repositories the App mints short-lived (~1 hour) access tokens scoped to the repositories you granted; those tokens are never written to logs, artifacts, or telemetry.

Troubleshooting

See also

• ax ci upload — the CI-side command reference.
• ax integrations — inspect integrations and link repos.
• Files — source-less slots and how artifacts are staged.
• Runs — what a run captures, and remote vs. local execution.