Files
coder/scripts/harbor-coder/README.md
T

63 lines
3.4 KiB
Markdown

# harbor-coder
A/B Coder Agents between a branch and `main` using
[Harbor](https://github.com/harbor-framework/harbor) and Terminal-Bench.
- `harbor_coder.environment:CoderEnvironment`: Harbor environment provider
that runs each trial in a Coder workspace (task image plus the `coder`
binary, created from `template/`, exec over `coder ssh`).
- `harbor_coder.agent:CoderAgentsAgent`: Harbor agent that binds a chat to
that workspace, waits for the turn to finish, and records tokens and cost.
- `harbor_coder.bootstrap`: first user, AI provider (dogfood's gateway), chat
model, template push on a fresh coderd.
- `harbor_coder.compare`: pass rate per task and cost for two job directories.
Comment `/coder-agents-eval` on a PR to run `.github/workflows/eval-coder-agents.yaml`:
each arm builds its own coderd (main vs the PR), runs `terminal-bench@2.0`
(74 of 89 tasks: 3 are broken upstream, 9 are 20 to 60 minute ML and
emulation tasks the prompt cannot affect, and 3 security tasks are always
refused by Anthropic's content policy) with Coder Agents on
`claude-sonnet-5-5`, and the comparison is posted on the PR. The baseline is
the commit the PR branched from, not the `main` tip, so later `main` changes
are not attributed to the PR. The system prompt is Go code, so a prompt
change is a branch and the unit of comparison is a build. The verdict calls
a change better or worse only when a paired bootstrap 95% interval over
tasks excludes zero.
Measured on the 77-task set (before the 3 refused tasks were dropped) at
concurrency 5 (chatd's unlicensed cap on
concurrent chats): about $23 and 90 minutes per arm
([run](https://github.com/coder/coder/actions/runs/40423882171)). One
`/coder-agents-eval` runs two arms in parallel, so about $47 of gateway
spend and two 8-core runners for roughly 1.5h. The full 86 tasks on `main`
scored 73% at two attempts, $31.54 and 3.4h for one arm.
Dataset, model, attempts, concurrency, and the Harbor version are the `env`
block at the top of the workflow.
## Local
```sh
# Installs Harbor into .harbor-coder/venv from requirements.lock with
# --require-hashes, so every dependency (litellm included) is pinned by sha256.
./scripts/harbor-coder/install.sh && export PATH="$PWD/.harbor-coder/venv/bin:$PATH"
# dogfood gateway auth; the gateway ignores token scopes, so keep it narrow
export HARBOR_CODER_UPSTREAM_TOKEN=$(coder tokens create --lifetime 8h --scope user:read_personal)
export HARBOR_CODER_MODEL=claude-sonnet-5-5
./scripts/harbor-coder/serve-coderd.sh .harbor-coder/env
set -a; . .harbor-coder/env; set +a; export PATH="$PWD/.harbor-coder:$PATH"
# add the -x list from HARBOR_EXCLUDE in the workflow to match CI
harbor run -d terminal-bench@2.0 -x 'qemu-*' -x build-cython-ext \
--agent harbor_coder.agent:CoderAgentsAgent --env harbor_coder.environment:CoderEnvironment \
-k 2 -n 4 -o jobs --job-name main
```
Stop the first coderd (`kill $(cat .harbor-coder/coderd.pid)`), check out the
other branch, rerun `serve-coderd.sh` and Harbor under another `--job-name`, then
`python -m harbor_coder.compare --baseline jobs/main --candidate jobs/branch`.
`serve-coderd.sh` refuses to start while something answers on its port, and
task images are tagged with the `coder` binary's hash, so the second run cannot
silently reuse the first build.
`--agent oracle` validates tasks without spending tokens. Any registry dataset
works if its image has `bash` and `tar`, runs as root, and does not need
`no-network` or a GPU.