Files
Michael Suchacz 3697bb38ce feat!: choose the computer use model per organization (#31300)
> [!NOTE]
> Xum opened this PR for Mike (Coder agent). Mike reviews it before
merge.

Fixes CODAGT-1472.

Computer use now runs on a chat model that each organization picks.
Before, it ran a fixed model on the first enabled AI provider of a
deployment-wide provider type, sorted by name. On dogfood that
provider's key is rejected, so every computer use subagent failed.

## What changed

- **Setting.** A new `computer_use` organization model override, set in
the **Computer use** row of **Organization settings** on the Coder
Agents admin page. Without an override, computer use runs on the
organization's default model. If the chosen model is disabled, deleted,
or unusable for the chat owner, computer use falls back to the default
model, as other overrides do. It also falls back when the chosen model
moves to an AI provider that is not Anthropic or OpenAI, and the row
then says that the saved model is ignored.
- **API.** `PUT
/api/v2/organizations/{organization}/chats/model-overrides/computer_use`
returns 404 while the `chat-virtual-desktop` experiment is off, and 400
unless the model's AI provider type is Anthropic or OpenAI.
- **chatd.** The `computer_use` subagent runs on that model, and the
computer tool flavor follows the model's provider type. OpenAI computer
use always uses the Responses API, so a model that sets
`use_responses_api: false` gets a configuration error. When no usable
model exists, `computer_use` is unavailable, and the reason says whether
a saved computer use model was skipped and why the default model cannot
be used. The default model is read under the chat owner's model ACLs, as
chat turns do.
- **UI.** The row lists only enabled Anthropic and OpenAI models. A
warning icon with the tooltip "Make sure that the selected model
supports computer use." appears when the model in effect (the selected
one, or the default) is not on a curated list of canonical model IDs
known to accept the computer tool (`knownComputerUseModels` in
`knownModels/catalog.ts`, exact match only). It also appears when the
model's AI provider type is OpenAI and its **Use Responses API**
override is Off, because OpenAI computer use needs the Responses API.
The warning does not block saving. I built the list from a live probe on
2026-10-26 that sent each Anthropic and OpenAI model one request with
the computer tool.
- **Docs.** `virtual-desktop.md`, `models.md`, `organizations.md`, and
`architecture.md` describe the new setting, and chatd's
`ARCHITECTURE.md` describes the computer use model routing.

![The Computer use row with an uncurated model selected shows the
warning icon and its
tooltip](https://github.com/user-attachments/assets/3948d936-5d8f-4ff9-a6de-14094c154dc8)

![The Computer use model picker lists only Anthropic and OpenAI
models](https://github.com/user-attachments/assets/913cd4a0-3cdf-4e5d-b2f9-582b9fca999c)

## Breaking change

This removes the deployment-wide computer use provider setting
(`agents_computer_use_provider`), its experimental `GET`/`PUT
/api/experimental/chats/config/computer-use-provider` endpoints (shipped
since v2.35), the Virtual desktop settings card, and its audit field.
Migration 000635 deletes the stored value. After the upgrade, computer
use runs on each organization's default model until an admin picks a
computer use model. Computer use subagents created before the upgrade
keep the model they inherited from their parent chat, so a follow-up
message to one of them can fail; a new computer use subagent uses the
organization's computer use model. Telemetry now reports the
per-organization overrides instead of the provider, and `null` when it
cannot read them.

## Testing

- `make pre-commit` passes. `go test` passes for `./coderd/x/chatd/...`,
`./coderd/database/migrations/`, and the targeted coderd, telemetry,
dbauthz, enterprise audit, and codersdk tests. Vitest and the touched
Storybook stories pass.
- Each new Go test fails when its target behavior is removed from
production code.
- Remote dogfood UAT ran on an isolated Coder instance built from the PR
head, with real Anthropic and OpenAI models:
- 110234e: the row with the experiment on and off, the picker, the
warning icon cases, save and clear, the 400 for other provider types,
which model and computer tool each computer use subagent used
(Anthropic, OpenAI, default, and disabled override), a real desktop
session on an Anthropic and an OpenAI model, the unavailable reasons,
the upgrade through migration 000634, and the removed endpoints.
- 09ce818: an override whose model moved to an unsupported AI provider.
- 7925be9: the alert for a moved and a disabled override, and a computer
use subagent on an Anthropic model.
- 2a01eaf: the row, and computer use subagents on the override and on
the default model.
- 47f5bd1: the warning icon for an OpenAI model with **Use Responses
API** set to Off, On, and Default. With Off, a computer use subagent on
that model fails with a configuration error before any provider request.
With Default, it runs with the computer tool through the Responses API.
- 81d33b4: the unavailable reasons for a disabled override, a cleared
override, and missing credentials, and a computer use subagent on an
Anthropic override. This round found that the docs named the wrong
section of the model form for **Use Responses API**. aeb3d02 fixes that
step, and a re-read of the docs against the recorded UI passed.
2026-10-29 13:00:05 +01:00

507 lines
15 KiB
Go

package database
import (
"database/sql/driver"
"encoding/json"
"fmt"
"net"
"strings"
"time"
"github.com/google/uuid"
"github.com/lib/pq"
"github.com/sqlc-dev/pqtype"
"golang.org/x/xerrors"
"github.com/coder/coder/v2/coderd/rbac"
"github.com/coder/coder/v2/coderd/rbac/policy"
)
// AuditOAuthConvertState is never stored in the database. It is stored in a cookie
// clientside as a JWT. This type is provided for audit logging purposes.
type AuditOAuthConvertState struct {
CreatedAt time.Time `db:"created_at" json:"created_at"`
// The time at which the state string expires, a merge request times out if the user does not perform it quick enough.
ExpiresAt time.Time `db:"expires_at" json:"expires_at"`
FromLoginType LoginType `db:"from_login_type" json:"from_login_type"`
// The login type the user is converting to. Should be github or oidc.
ToLoginType LoginType `db:"to_login_type" json:"to_login_type"`
UserID uuid.UUID `db:"user_id" json:"user_id"`
}
type HealthSettings struct {
ID uuid.UUID `db:"id" json:"id"`
DismissedHealthchecks []string `db:"dismissed_healthchecks" json:"dismissed_healthchecks"`
}
type NotificationsSettings struct {
ID uuid.UUID `db:"id" json:"id"`
NotifierPaused bool `db:"notifier_paused" json:"notifier_paused"`
}
type PrebuildsSettings struct {
ID uuid.UUID `db:"id" json:"id"`
ReconciliationPaused bool `db:"reconciliation_paused" json:"reconciliation_paused"`
}
type OAuth2ProviderSettings struct {
ID uuid.UUID `db:"id" json:"id"`
DynamicClientRegistrationEnabled bool `db:"dynamic_client_registration_enabled" json:"dynamic_client_registration_enabled"`
}
// ChatInstructionSettings is the auditable shape of the deployment-wide
// chat instruction configuration, stored across the
// agents_chat_system_prompt, agents_chat_include_default_system_prompt and
// agents_chat_plan_mode_instructions site_configs keys. Both the
// system-prompt and plan-mode-instructions endpoints audit this one type;
// each populates only the fields its endpoint can change.
type ChatInstructionSettings struct {
ID uuid.UUID `db:"id" json:"id"`
// Name identifies which setting an audit row concerns (e.g. "System
// prompt"). It is ignored in diffs and set identically on Old and New.
Name string `db:"name" json:"name"`
SystemPrompt string `db:"system_prompt" json:"system_prompt"`
// IncludeDefaultSystemPromptSet records whether the override row
// exists, not only its effective value: writing explicit false over a
// legacy absent row does not move the effective value but changes
// future behavior, so presence must enter the diff.
IncludeDefaultSystemPromptSet bool `db:"include_default_system_prompt_set" json:"include_default_system_prompt_set"`
IncludeDefaultSystemPrompt bool `db:"include_default_system_prompt" json:"include_default_system_prompt"`
PlanModeInstructions string `db:"plan_mode_instructions" json:"plan_mode_instructions"`
}
// ChatOperationalSettings contains deployment-wide chat settings for audit logging.
type ChatOperationalSettings struct {
ID uuid.UUID `db:"id" json:"id"`
ChatRetentionDays string `db:"chat_retention_days" json:"chat_retention_days"`
ChatDebugRetentionDays string `db:"chat_debug_retention_days" json:"chat_debug_retention_days"`
ChatAutoArchiveDays string `db:"chat_auto_archive_days" json:"chat_auto_archive_days"`
WorkspaceTTL string `db:"workspace_ttl" json:"workspace_ttl"`
DebugLoggingAllowUsers string `db:"debug_logging_allow_users" json:"debug_logging_allow_users"`
PersonalModelOverridesEnabled string `db:"personal_model_overrides_enabled" json:"personal_model_overrides_enabled"`
}
// ExperimentRule is the audited form of one runtime experiment rule. The
// rule itself is stored as JSON in site_configs. ID is derived from the
// experiment name (experiments.AuditRecord) so that audit history groups by
// experiment.
type ExperimentRule struct {
ID uuid.UUID `db:"id" json:"id"`
Experiment string `db:"experiment" json:"experiment"`
Mode string `db:"mode" json:"mode"`
Condition string `db:"condition" json:"condition"`
Revision int64 `db:"revision" json:"revision"`
}
type Actions []policy.Action
func (a *Actions) Scan(src interface{}) error {
switch v := src.(type) {
case string:
return json.Unmarshal([]byte(v), &a)
case []byte:
return json.Unmarshal(v, &a)
}
return xerrors.Errorf("unexpected type %T", src)
}
func (a *Actions) Value() (driver.Value, error) {
return json.Marshal(a)
}
// TemplateACL is a map of ids to permissions.
type TemplateACL map[string][]policy.Action
func (t *TemplateACL) Scan(src interface{}) error {
switch v := src.(type) {
case string:
return json.Unmarshal([]byte(v), &t)
case []byte:
return json.Unmarshal(v, &t)
case json.RawMessage:
return json.Unmarshal(v, &t)
}
return xerrors.Errorf("unexpected type %T", src)
}
func (t TemplateACL) Value() (driver.Value, error) {
return json.Marshal(t)
}
type ChatACL map[string]ChatACLEntry
func (c *ChatACL) Scan(src interface{}) error {
switch v := src.(type) {
case string:
return json.Unmarshal([]byte(v), &c)
case []byte:
return json.Unmarshal(v, &c)
case json.RawMessage:
return json.Unmarshal(v, &c)
}
return xerrors.Errorf("unexpected type %T", src)
}
//nolint:revive
func (c ChatACL) RBACACL() map[string][]policy.Action {
rbacACL := make(map[string][]policy.Action, len(c))
for id, entry := range c {
rbacACL[id] = entry.Permissions
}
return rbacACL
}
func (c ChatACL) Value() (driver.Value, error) {
if c == nil {
return json.Marshal(ChatACL{})
}
return json.Marshal(c)
}
type ChatACLEntry struct {
Permissions []policy.Action `json:"permissions"`
}
// AgentMetadataAggregate is the agent_metadata jsonb array the
// GetWorkspaces query aggregates for the include_agent_metadata
// expansion. Elements have WorkspaceAgentMetadatum's JSON shape; each
// carries its workspace_agent_id so multi-agent workspaces can map
// values onto the right agent. The generated row keeps
// json.RawMessage because sqlc overrides cannot target expression
// columns; callers Scan the raw value into this type.
type AgentMetadataAggregate []WorkspaceAgentMetadatum
func (a *AgentMetadataAggregate) Scan(src interface{}) error {
switch v := src.(type) {
case nil:
return nil
case string:
return json.Unmarshal([]byte(v), &a)
case []byte:
return json.Unmarshal(v, &a)
case json.RawMessage:
return json.Unmarshal(v, &a)
}
return xerrors.Errorf("unexpected type %T", src)
}
func (a AgentMetadataAggregate) Value() (driver.Value, error) {
return json.Marshal(a)
}
type WorkspaceACL map[string]WorkspaceACLEntry
func (t *WorkspaceACL) Scan(src interface{}) error {
switch v := src.(type) {
case string:
return json.Unmarshal([]byte(v), &t)
case []byte:
return json.Unmarshal(v, &t)
case json.RawMessage:
return json.Unmarshal(v, &t)
}
return xerrors.Errorf("unexpected type %T", src)
}
//nolint:revive,staticcheck // Receiver name matches the other WorkspaceACL methods in this file.
func (w WorkspaceACL) RBACACL() map[string][]policy.Action {
// Convert WorkspaceACL to a map of string to []policy.Action.
// This is used for RBAC checks.
rbacACL := make(map[string][]policy.Action, len(w))
for id, entry := range w {
rbacACL[id] = entry.Permissions
}
return rbacACL
}
func (t WorkspaceACL) Value() (driver.Value, error) {
return json.Marshal(t)
}
type WorkspaceACLEntry struct {
Permissions []policy.Action `json:"permissions"`
}
// WorkspaceACLDisplayInfo supplements workspace ACLs with the actors'
// display info. Key is string rather than uuid.UUID as this aligns
// with how RBAC represents actor IDs.
type WorkspaceACLDisplayInfo map[string]struct {
Name string `json:"name"`
AvatarURL string `json:"avatar_url"`
}
// WorkspaceACLDisplayInfo is only used to read from the DB.
func (w *WorkspaceACLDisplayInfo) Scan(src interface{}) error {
switch v := src.(type) {
case string:
return json.Unmarshal([]byte(v), w)
case []byte:
return json.Unmarshal(v, w)
case json.RawMessage:
return json.Unmarshal(v, w)
}
return xerrors.Errorf("unexpected type %T", src)
}
type ExternalAuthProvider struct {
ID string `json:"id"`
Optional bool `json:"optional,omitempty"`
}
type StringMap map[string]string
func (m *StringMap) Scan(src interface{}) error {
if src == nil {
return nil
}
switch src := src.(type) {
case []byte:
err := json.Unmarshal(src, m)
if err != nil {
return err
}
default:
return xerrors.Errorf("unsupported Scan, storing driver.Value type %T into type %T", src, m)
}
return nil
}
func (m StringMap) Value() (driver.Value, error) {
return json.Marshal(m)
}
type StringMapOfInt map[string]int64
func (m *StringMapOfInt) Scan(src interface{}) error {
if src == nil {
*m = nil
return nil
}
switch src := src.(type) {
case []byte:
err := json.Unmarshal(src, m)
if err != nil {
return err
}
default:
return xerrors.Errorf("unsupported Scan, storing driver.Value type %T into type %T", src, m)
}
return nil
}
func (m StringMapOfInt) Value() (driver.Value, error) {
return json.Marshal(m)
}
type CustomRolePermissions []CustomRolePermission
func (s *APIKeyScopes) Scan(src any) error {
var arr []string
if err := pq.Array(&arr).Scan(src); err != nil {
return err
}
out := make(APIKeyScopes, len(arr))
for i, v := range arr {
out[i] = APIKeyScope(v)
}
*s = out
return nil
}
func (s APIKeyScopes) Value() (driver.Value, error) {
arr := make([]string, len(s))
for i, v := range s {
arr[i] = string(v)
}
return pq.Array(arr).Value()
}
func (a *CustomRolePermissions) Scan(src interface{}) error {
switch v := src.(type) {
case string:
return json.Unmarshal([]byte(v), &a)
case []byte:
return json.Unmarshal(v, &a)
}
return xerrors.Errorf("unexpected type %T", src)
}
func (a CustomRolePermissions) Value() (driver.Value, error) {
return json.Marshal(a)
}
type CustomRolePermission struct {
Negate bool `json:"negate"`
ResourceType string `json:"resource_type"`
Action policy.Action `json:"action"`
}
func (a CustomRolePermission) String() string {
str := a.ResourceType + "." + string(a.Action)
if a.Negate {
return "-" + str
}
return str
}
// NameOrganizationPair is used as a lookup tuple for custom role rows.
type NameOrganizationPair struct {
Name string `db:"name" json:"name"`
// OrganizationID if unset will assume a null column value
OrganizationID uuid.UUID `db:"organization_id" json:"organization_id"`
}
func (*NameOrganizationPair) Scan(_ interface{}) error {
return xerrors.Errorf("this should never happen, type 'NameOrganizationPair' should only be used as a parameter")
}
// Value returns the tuple **literal**
// To get the literal value to return, you can use the expression syntax in a psql
// shell.
//
// SELECT ('customrole'::text,'ece79dac-926e-44ca-9790-2ff7c5eb6e0c'::uuid);
// To see 'null' option. Using the nil uuid as null to avoid empty string literals for null.
// SELECT ('customrole',00000000-0000-0000-0000-000000000000);
//
// This value is usually used as an array, NameOrganizationPair[]. You can see
// what that literal is as well, with proper quoting.
//
// SELECT ARRAY[('customrole'::text,'ece79dac-926e-44ca-9790-2ff7c5eb6e0c'::uuid)];
func (a NameOrganizationPair) Value() (driver.Value, error) {
// The string values must be escaped in case there are special characters, quotes, etc.
// 'NameOrganizationPair' is a composite value, which has no driver handler
// in the `pq` package.
//
// pq.StringArray formats the single name as `{"<escaped>"}`. Strip
// the outer braces to get the quoted+escaped form that composite
// literal syntax accepts unchanged.
//
// Ideally `appendArrayQuotedBytes` would be exported, and we could call
// it directly.
v, err := (&pq.StringArray{a.Name}).Value()
if err != nil {
return nil, err
}
s, ok := v.(string)
if !ok {
return nil, xerrors.Errorf("unexpected type %T", v)
}
stripCurlyBraces := s[1 : len(s)-1]
return fmt.Sprintf("(%s,%s)", stripCurlyBraces, a.OrganizationID.String()), nil
}
// AgentIDNamePair is used as a result tuple for workspace and agent rows.
type AgentIDNamePair struct {
ID uuid.UUID `db:"id" json:"id"`
Name string `db:"name" json:"name"`
}
func (p *AgentIDNamePair) Scan(src interface{}) error {
var v string
switch a := src.(type) {
case []byte:
v = string(a)
case string:
v = a
default:
return xerrors.Errorf("unexpected type %T", src)
}
parts := strings.Split(strings.Trim(v, "()"), ",")
if len(parts) != 2 {
return xerrors.New("invalid format for AgentIDNamePair")
}
id, err := uuid.Parse(strings.TrimSpace(parts[0]))
if err != nil {
return err
}
p.ID, p.Name = id, strings.TrimSpace(parts[1])
return nil
}
func (p AgentIDNamePair) Value() (driver.Value, error) {
return fmt.Sprintf(`(%s,%s)`, p.ID.String(), p.Name), nil
}
// UserLinkClaims is the returned IDP claims for a given user link.
// These claims are fetched at login time. These are the claims that were
// used for IDP sync.
type UserLinkClaims struct {
IDTokenClaims map[string]interface{} `json:"id_token_claims"`
UserInfoClaims map[string]interface{} `json:"user_info_claims"`
// MergeClaims are computed in Golang. It is the result of merging
// the IDTokenClaims and UserInfoClaims. UserInfoClaims take precedence.
MergedClaims map[string]interface{} `json:"merged_claims"`
}
func (a *UserLinkClaims) Scan(src interface{}) error {
switch v := src.(type) {
case string:
return json.Unmarshal([]byte(v), &a)
case []byte:
return json.Unmarshal(v, &a)
}
return xerrors.Errorf("unexpected type %T", src)
}
func (a UserLinkClaims) Value() (driver.Value, error) {
return json.Marshal(a)
}
func ParseIP(ipStr string) pqtype.Inet {
ip := net.ParseIP(ipStr)
ipNet := net.IPNet{}
if ip != nil {
ipNet = net.IPNet{
IP: ip,
Mask: net.CIDRMask(len(ip)*8, len(ip)*8),
}
}
return pqtype.Inet{
IPNet: ipNet,
Valid: ip != nil,
}
}
// AllowList is a typed wrapper around a list of AllowListTarget entries.
// It implements sql.Scanner and driver.Valuer so it can be stored in and
// loaded from a Postgres text[] column that stores each entry in the
// canonical form "type:id".
type AllowList []rbac.AllowListElement
// Scan implements sql.Scanner. It supports inputs that pq.Array can decode
// into []string, and then converts each element to an AllowListTarget.
func (a *AllowList) Scan(src any) error {
var raw []string
if err := pq.Array(&raw).Scan(src); err != nil {
return err
}
out := make([]rbac.AllowListElement, len(raw))
for i, s := range raw {
e, err := rbac.ParseAllowListEntry(s)
if err != nil {
return err
}
out[i] = e
}
*a = out
return nil
}
// Value implements driver.Valuer by converting the list to []string using the
// canonical "type:id" form and delegating to pq.Array for encoding.
func (a AllowList) Value() (driver.Value, error) {
raw := make([]string, len(a))
for i, t := range a {
raw[i] = t.String()
}
return pq.Array(raw).Value()
}