Files
coder/docs/coder-agents/platform-controls/virtual-desktop.md
T
Michael SuchaczandEthan Dickson 082668fbc1 feat: run computer use on OpenAI-compatible AI providers (#31895)
Computer use only ran on Anthropic and OpenAI AI providers. Chatd
derives the tool flavor from the AI provider type, and `openai-compat`
had no flavor, so every self-hosted model was marked unavailable for
`type=computer_use`, including the model a single-provider deployment
already uses for chat.

Hosted computer tools are not an option on those servers: an
OpenAI-compatible endpoint accepts a `computer_20250124` tool over
`/v1/messages` and a `computer-use-preview` tool over `/v1/responses`
and then drops both, so the model never sees a way to act. Computer use
is now declared as an ordinary function tool for those providers. The
model sends one action per call and every call answers with a fresh
screenshot, which is the shape these models already handle through the
shared action runner. The tool omits `zoom`, which the agent desktop
cannot perform. The desktop clamps a point outside the screen to its
edge, so the tool rejects a coordinate outside the declared screenshot
with an error that states the valid range, and a model that aims off the
screen learns that it missed. Media tool results reach the model because
the pinned fantasy fork promotes them to user-role messages, so no
dependency changed.

Coder can't verify an arbitrary OpenAI-compatible model, so those models
keep the unverified warning icon in the admin UI. The virtual desktop
docs, the Coder Agents index, the chatd architecture doc, and the
organization model settings copy now list OpenAI-compatible providers as
supported, and the docs say what to check before using one.

Three shared behaviors needed fixing to make this usable. A coordinate
of `[0,0]` is the top-left pixel and decoded as an omitted field, so the
desktop rejected those clicks with a missing-coordinate error;
coordinates now decode by key presence, which also keeps an absent
coordinate absent. A `scroll_amount` of zero or less, or a key `repeat`
of zero, was also forwarded as omitted, so the desktop scrolled its
default of three clicks or pressed the key once; the runner now rejects
both, and the function tool declares a minimum of 1 for each. And
because a chat completions model sends one action per call, it plans
every call of a step against one screen, so a failed action now halts
the later calls of that step, as it already did for Anthropic. The
hosted OpenAI flavor still differs here on purpose: it delivers an array
of actions in one call. Halting would strand a mouse press whose release
was planned for a later call, so the function tool offers
`left_click_drag` instead of separate press and release actions, and
rejects any action it does not declare.

Provider types that reuse the chat completions client without being
OpenAI-compatible (Azure, Google, OpenRouter, Vercel, Copilot) stay
unsupported until each is verified.

> [!NOTE]
> `coderd/apidoc/docs.go`, `coderd/apidoc/swagger.json`,
`docs/reference/api/schemas.md` and `site/src/api/typesGenerated.ts` are
hand-edited to match the new enum value, because the environment that
prepared this change had no pnpm to re-run the generators. The same
environment had no node, so `pnpm run typecheck`, biome and prettier did
not run locally; CI is the validator for the `site/` change and the
generated files.

> [!NOTE]
> Prepared by Xum, Mike's coding agent, from the measurement that showed
an OpenAI-compatible server dropping both hosted computer tools.

---------

Co-authored-by: Ethan Dickson <ethanndickson@gmail.com>
2026-11-09 17:26:10 +11:00

3.6 KiB

title
title
Virtual desktop

Note

This feature is experimental. Pin a release before broad rollout and review the release notes before upgrading.

Enable the experiment

coder server --experiments=chat-virtual-desktop

Or set the environment variable:

CODER_EXPERIMENTS=chat-virtual-desktop

What it does

Lets agents drive a graphical desktop inside the workspace through spawn_agent with type=computer_use (screenshots, mouse, keyboard).

Configuration

Virtual desktop requires the portabledesktop module installed in the workspace template.

Choose the computer use model

Computer use runs on a chat model from the chat's organization. Each organization picks its own model as an organization-scoped model override. To change it, you need permission to edit that organization's models.

  1. Go to Admin settings > AI > Coder Agents.
  2. If the Organization settings section shows an organization picker, select the organization.
  3. In the Computer use row, select a model.
  4. Select Save.

The Computer use row appears only while the chat-virtual-desktop experiment is enabled. The list shows the organization's enabled models that computer use supports: models from Anthropic, OpenAI, and OpenAI-compatible AI providers, and Claude models from AWS Bedrock AI providers. To add a model to the list, enable one of these models on the Models page.

When no model is selected, the row shows Use default model and computer use runs on the organization's default model. To return to the default model, select Clear, then select Save. If the selected model is later disabled, deleted, changed to a model that computer use doesn't support, or unusable for the chat owner, computer use falls back to the organization's default model.

Computer use is unavailable in a chat when the chat owner has no usable credentials for the AI provider of the model it runs on. It's also unavailable when it runs on the organization's default model and computer use doesn't support that model. OpenAI computer use requires the Responses API, so it fails on an OpenAI model whose Use Responses API override is set to Off.

An OpenAI-compatible AI provider has no computer use tool of its own, so Coder offers computer use to the model as an ordinary tool and returns a screenshot after every action. Coder can't verify an arbitrary OpenAI-compatible model, so such a model keeps the warning icon.

A warning icon next to the model selector means that Coder hasn't verified the model for computer use, or that the model's AI provider type is OpenAI and its Use Responses API override is set to Off. The icon reflects the selected model, or the organization's default model when no model is selected or the selected model no longer appears in the list. The warning doesn't block saving. Before you save, confirm that the model supports the provider's computer use tool. For an OpenAI-compatible AI provider, confirm instead that the model accepts images and calls tools.

To turn the Responses API back on for a model whose AI provider type is OpenAI:

  1. On the Models page, select the model.
  2. Expand Advanced.
  3. Set Use Responses API to Default or On.
  4. Select Update model.

You can also choose the computer use model through the API at PUT /api/v2/organizations/{organization}/chats/model-overrides/computer_use. The request fails with 404 Not Found while the experiment is disabled, and with 400 Bad Request when computer use doesn't support the model.