Files
coder/coderd/database
Michael Suchacz 3697bb38ce feat!: choose the computer use model per organization (#31300)
> [!NOTE]
> Xum opened this PR for Mike (Coder agent). Mike reviews it before
merge.

Fixes CODAGT-1472.

Computer use now runs on a chat model that each organization picks.
Before, it ran a fixed model on the first enabled AI provider of a
deployment-wide provider type, sorted by name. On dogfood that
provider's key is rejected, so every computer use subagent failed.

## What changed

- **Setting.** A new `computer_use` organization model override, set in
the **Computer use** row of **Organization settings** on the Coder
Agents admin page. Without an override, computer use runs on the
organization's default model. If the chosen model is disabled, deleted,
or unusable for the chat owner, computer use falls back to the default
model, as other overrides do. It also falls back when the chosen model
moves to an AI provider that is not Anthropic or OpenAI, and the row
then says that the saved model is ignored.
- **API.** `PUT
/api/v2/organizations/{organization}/chats/model-overrides/computer_use`
returns 404 while the `chat-virtual-desktop` experiment is off, and 400
unless the model's AI provider type is Anthropic or OpenAI.
- **chatd.** The `computer_use` subagent runs on that model, and the
computer tool flavor follows the model's provider type. OpenAI computer
use always uses the Responses API, so a model that sets
`use_responses_api: false` gets a configuration error. When no usable
model exists, `computer_use` is unavailable, and the reason says whether
a saved computer use model was skipped and why the default model cannot
be used. The default model is read under the chat owner's model ACLs, as
chat turns do.
- **UI.** The row lists only enabled Anthropic and OpenAI models. A
warning icon with the tooltip "Make sure that the selected model
supports computer use." appears when the model in effect (the selected
one, or the default) is not on a curated list of canonical model IDs
known to accept the computer tool (`knownComputerUseModels` in
`knownModels/catalog.ts`, exact match only). It also appears when the
model's AI provider type is OpenAI and its **Use Responses API**
override is Off, because OpenAI computer use needs the Responses API.
The warning does not block saving. I built the list from a live probe on
2026-10-26 that sent each Anthropic and OpenAI model one request with
the computer tool.
- **Docs.** `virtual-desktop.md`, `models.md`, `organizations.md`, and
`architecture.md` describe the new setting, and chatd's
`ARCHITECTURE.md` describes the computer use model routing.

![The Computer use row with an uncurated model selected shows the
warning icon and its
tooltip](https://github.com/user-attachments/assets/3948d936-5d8f-4ff9-a6de-14094c154dc8)

![The Computer use model picker lists only Anthropic and OpenAI
models](https://github.com/user-attachments/assets/913cd4a0-3cdf-4e5d-b2f9-582b9fca999c)

## Breaking change

This removes the deployment-wide computer use provider setting
(`agents_computer_use_provider`), its experimental `GET`/`PUT
/api/experimental/chats/config/computer-use-provider` endpoints (shipped
since v2.35), the Virtual desktop settings card, and its audit field.
Migration 000635 deletes the stored value. After the upgrade, computer
use runs on each organization's default model until an admin picks a
computer use model. Computer use subagents created before the upgrade
keep the model they inherited from their parent chat, so a follow-up
message to one of them can fail; a new computer use subagent uses the
organization's computer use model. Telemetry now reports the
per-organization overrides instead of the provider, and `null` when it
cannot read them.

## Testing

- `make pre-commit` passes. `go test` passes for `./coderd/x/chatd/...`,
`./coderd/database/migrations/`, and the targeted coderd, telemetry,
dbauthz, enterprise audit, and codersdk tests. Vitest and the touched
Storybook stories pass.
- Each new Go test fails when its target behavior is removed from
production code.
- Remote dogfood UAT ran on an isolated Coder instance built from the PR
head, with real Anthropic and OpenAI models:
- 110234e: the row with the experiment on and off, the picker, the
warning icon cases, save and clear, the 400 for other provider types,
which model and computer tool each computer use subagent used
(Anthropic, OpenAI, default, and disabled override), a real desktop
session on an Anthropic and an OpenAI model, the unavailable reasons,
the upgrade through migration 000634, and the removed endpoints.
- 09ce818: an override whose model moved to an unsupported AI provider.
- 7925be9: the alert for a moved and a disabled override, and a computer
use subagent on an Anthropic model.
- 2a01eaf: the row, and computer use subagents on the override and on
the default model.
- 47f5bd1: the warning icon for an OpenAI model with **Use Responses
API** set to Off, On, and Default. With Off, a computer use subagent on
that model fails with a configuration error before any provider request.
With Default, it runs with the computer tool through the Responses API.
- 81d33b4: the unavailable reasons for a disabled override, a cleared
override, and missing credentials, and a computer use subagent on an
Anthropic override. This round found that the docs named the wrong
section of the model form for **Use Responses API**. aeb3d02 fixes that
step, and a re-read of the docs against the recorded UI passed.
2026-10-29 13:00:05 +01:00
..
…