63 lines
3.4 KiB
Markdown
63 lines
3.4 KiB
Markdown
# harbor-coder
|
|
|
|
A/B Coder Agents between a branch and `main` using
|
|
[Harbor](https://github.com/harbor-framework/harbor) and Terminal-Bench.
|
|
|
|
- `harbor_coder.environment:CoderEnvironment`: Harbor environment provider
|
|
that runs each trial in a Coder workspace (task image plus the `coder`
|
|
binary, created from `template/`, exec over `coder ssh`).
|
|
- `harbor_coder.agent:CoderAgentsAgent`: Harbor agent that binds a chat to
|
|
that workspace, waits for the turn to finish, and records tokens and cost.
|
|
- `harbor_coder.bootstrap`: first user, AI provider (dogfood's gateway), chat
|
|
model, template push on a fresh coderd.
|
|
- `harbor_coder.compare`: pass rate per task and cost for two job directories.
|
|
|
|
Comment `/coder-agents-eval` on a PR to run `.github/workflows/eval-coder-agents.yaml`:
|
|
each arm builds its own coderd (main vs the PR), runs `terminal-bench@2.0`
|
|
(74 of 89 tasks: 3 are broken upstream, 9 are 20 to 60 minute ML and
|
|
emulation tasks the prompt cannot affect, and 3 security tasks are always
|
|
refused by Anthropic's content policy) with Coder Agents on
|
|
`claude-sonnet-5-5`, and the comparison is posted on the PR. The baseline is
|
|
the commit the PR branched from, not the `main` tip, so later `main` changes
|
|
are not attributed to the PR. The system prompt is Go code, so a prompt
|
|
change is a branch and the unit of comparison is a build. The verdict calls
|
|
a change better or worse only when a paired bootstrap 95% interval over
|
|
tasks excludes zero.
|
|
|
|
Measured on the 77-task set (before the 3 refused tasks were dropped) at
|
|
concurrency 5 (chatd's unlicensed cap on
|
|
concurrent chats): about $23 and 90 minutes per arm
|
|
([run](https://github.com/coder/coder/actions/runs/40423882171)). One
|
|
`/coder-agents-eval` runs two arms in parallel, so about $47 of gateway
|
|
spend and two 8-core runners for roughly 1.5h. The full 86 tasks on `main`
|
|
scored 73% at two attempts, $31.54 and 3.4h for one arm.
|
|
Dataset, model, attempts, concurrency, and the Harbor version are the `env`
|
|
block at the top of the workflow.
|
|
|
|
## Local
|
|
|
|
```sh
|
|
# Installs Harbor into .harbor-coder/venv from requirements.lock with
|
|
# --require-hashes, so every dependency (litellm included) is pinned by sha256.
|
|
./scripts/harbor-coder/install.sh && export PATH="$PWD/.harbor-coder/venv/bin:$PATH"
|
|
# dogfood gateway auth; the gateway ignores token scopes, so keep it narrow
|
|
export HARBOR_CODER_UPSTREAM_TOKEN=$(coder tokens create --lifetime 8h --scope user:read_personal)
|
|
export HARBOR_CODER_MODEL=claude-sonnet-5-5
|
|
./scripts/harbor-coder/serve-coderd.sh .harbor-coder/env
|
|
set -a; . .harbor-coder/env; set +a; export PATH="$PWD/.harbor-coder:$PATH"
|
|
# add the -x list from HARBOR_EXCLUDE in the workflow to match CI
|
|
harbor run -d terminal-bench@2.0 -x 'qemu-*' -x build-cython-ext \
|
|
--agent harbor_coder.agent:CoderAgentsAgent --env harbor_coder.environment:CoderEnvironment \
|
|
-k 2 -n 4 -o jobs --job-name main
|
|
```
|
|
|
|
Stop the first coderd (`kill $(cat .harbor-coder/coderd.pid)`), check out the
|
|
other branch, rerun `serve-coderd.sh` and Harbor under another `--job-name`, then
|
|
`python -m harbor_coder.compare --baseline jobs/main --candidate jobs/branch`.
|
|
`serve-coderd.sh` refuses to start while something answers on its port, and
|
|
task images are tagged with the `coder` binary's hash, so the second run cannot
|
|
silently reuse the first build.
|
|
`--agent oracle` validates tasks without spending tokens. Any registry dataset
|
|
works if its image has `bash` and `tar`, runs as root, and does not need
|
|
`no-network` or a GPU.
|