Docker in Sandbox

Docker is installed in every sandbox and the daemon is running before setup starts, so an experiment uses Docker the way it would on a laptop:

docker run -d -p 8080:80 nginx
curl -sS http://localhost:8080
docker compose up -d
docker build -t my-image .

There is nothing to install, no daemon to start, and no special flags to pass. docker runs as the sandbox user, so sudo is not needed either.

What the sandbox gives you

Containers the agent leaves running are still running when tests execute, so a test can talk to a service the agent started:

tests:
  - name: service-answers-on-published-port
    script: curl -fsS http://localhost:8080/health
  - name: container-still-up
    script: docker ps --filter name=api --filter status=running --quiet | grep -q .

Local runs

ax experiment run --local runs the sandbox as a container on your own machine, and a container cannot host a Docker daemon unless you allow it to. Opt in per run:

AX_SANDBOX_PRIVILEGED=1 ax experiment run experiment.yaml --local

Without it the run proceeds normally, but docker reports that it cannot connect to a daemon. Leave it off unless the experiment needs Docker: it gives agent-authored code a much weaker boundary than the default sandbox container. Remote runs need no equivalent flag.

Cost of images

Each sandbox starts with an empty image store, so the first docker run or docker build in a variant pays for the pull. Account for that in limits.max_time_seconds, and prefer small images (alpine, -slim) for the parts of the task that are not what you are measuring.

Validate a Docker-heavy setup cheaply before spending tokens: ax experiment run <experiment.yaml> --mock runs setup, setup_checks, and tests with a no-op agent. setup runs under set -e, so any non-zero exit aborts the variant.

Complete example

A runnable experiment where the agent containerizes a service and the tests check the running container:

schema_version: 2
id: docker-in-sandbox
name: "Docker in sandbox: agent runs a published service"
agents: [claude]
models: [claude-sonnet-4-6]
prompts:
  - id: serve-in-container
    prompt: |
      Docker is installed and the daemon is running. Start the `nginx:alpine`
      image as a container named `web`, publish it on port 8080 of this
      machine, and confirm it answers with `curl http://localhost:8080`.
      Write the response body to /workspace/published-port.txt.
environments:
  - name: docker-host
    setup:
      - name: prefetch-image
        script: docker pull nginx:alpine
tests:
  - name: published-port-answered
    script: grep -qi "welcome to nginx" /workspace/published-port.txt
  - name: container-is-running
    script: docker ps --filter name=web --filter status=running --quiet | grep -q .
limits:
  max_time_seconds: 1200