## Problem
The concurrent-agent cap (5 root chats and 10 subagent chats per
deployment by default) could be exceeded. Three paths put more chats in
`running` with a fresh lease than the pool allows:
1. **Interrupting a waiting chat.** A user sends a message with
`busy_behavior=interrupt` to a chat that is waiting for capacity. The
chat becomes `interrupting` while unowned, a worker acquires it without
admission, and `FinishInterruption` promotes the queued message into a
running turn under that ownership.
2. **Taking over an orphaned `requires_action` chat.** A replica dies
while its chat is in `requires_action`, its lease goes stale and another
chat takes the slot. A worker acquires the `requires_action` chat
without admission, and a tool result, an interrupt or the action
deadline returns it to `running` under that ownership.
3. **Heartbeat reviving a stale lease.** A replica loses the database
for longer than the 30 second stale threshold and another replica admits
a second chat into the slot. When the first replica comes back, its
heartbeat upsert renews the stale lease unconditionally, and its runner
keeps generating until the turn ends.
## Root cause
`agentCapacityLimiter.Admit` returned `true` without taking the
admission lock or counting for every status other than `running`. The
count query already treats every owned chat with a fresh lease as
holding a slot, so the shortcut let chats acquire a slot uncounted and
later move back to `running` under that ownership. Separately,
`BatchUpsertChatHeartbeats` had no freshness or ownership condition.
## Fix
- **Admission for every runnable status.** `Admit` now counts for
`running`, `interrupting` and `requires_action`, and a refused pool
skips the rest of its candidates in the pass regardless of status. An
owned chat holds its slot until it releases ownership, so every later
transition back to `running` stays within capacity.
- **Fenced heartbeat renewal.** `BatchUpsertChatHeartbeats` is replaced
by `RenewChatHeartbeats`, which only extends rows that are still fresh
and still own their chat, using `statement_timestamp()` read after the
capacity admission lock is granted. The renewal runs under that lock
with a `lock_timeout` (`HeartbeatLockTimeout`) and a per-tick timeout
(`HeartbeatRenewalTimeout`, a third of the stale threshold). Admission
also waits at most 5 seconds for the lock.
- **Stop runners that lost their lease.** `heartbeatOnce` requests
cleanup of every runner whose lease was not renewed, which cancels its
generation. The chat stays owned by the stale lease until an admitted
worker takes it over. A tick that fails or times out cleans up no
runners and retries on the next tick.
- **Instant Stop on a queued chat.** Requiring admission for
`interrupting` would have made Stop on a chat waiting for capacity hang
until a slot frees. When `Interrupt`, or `SendMessage` with
`busy_behavior=interrupt`, runs on a running chat with no owner, the
transition now applies `FinishInterruption` in the same transaction and
cancels orphaned tool calls. An empty queue lands in `waiting`;
otherwise the queue head is promoted into `running`, still unowned, so
it goes through admission.
- **Queued metric.** `CountChatCapacityQueuedByPool` (behind
`coderd_chatd_agents_queued_for_capacity`) now also counts
`interrupting` and `requires_action` chats without a fresh lease, since
they also wait for admission.
## How it was found
A TLA+ model of capacity admission found a counterexample for each path.
Go tests reproduced all three against the previous code (each failed
with `"2" is not less than or equal to "1"` for a root pool of capacity
1), and the model with the fix enabled passes the safety invariant and
the liveness properties. The model itself is not part of this PR.
## Tests
The three reproductions are kept as regression tests in
`coderd/x/chatd/agentcapacity_regression_internal_test.go`:
`TestAdmission_InterruptWithMessageOnWaitingChatStaysWithinCapacity`,
`TestAdmission_RequiresActionTakeoverStaysWithinCapacity` and
`TestRunnerManager_HeartbeatDoesNotReviveStaleLease`. The original
reproductions failed on the base commit. Restoring the admission
shortcut makes the `requires_action` test fail, and restoring the
unconditional heartbeat upsert, or dropping the runner cleanup, makes
the heartbeat test fail. New coverage also includes
`TestAdmission_StopWaitingChatNeedsNoCapacity`,
`TestRenewChatHeartbeats_OnlyRenewsFreshOwnedLeases`,
`TestRunnerManager_FailedHeartbeatTickKeepsRunners`, and chatstate tests
for the unowned interrupt paths. Existing tests that encoded the old
behavior (`requires_action` chats holding no slot, interrupting chats
always acquirable, unconditional heartbeat renewal) were updated.
## Behavior / API change
- `POST /api/v2/chats/{chat}/messages` with `busy_behavior=interrupt` on
a chat that no worker owns (for example one waiting for capacity) now
returns the promoted message with `queued=false` instead of a queued
message, and the chat stays in `running` instead of `interrupting`. If
the chat already had queued messages, the queue head is promoted and the
new message stays queued.
- `POST /api/v2/chats/{chat}/interrupt` on such a chat lands it in
`waiting` immediately (or in `running` with the queue head promoted)
instead of `interrupting`.
- An unowned `interrupting` or `requires_action` chat (for example after
its replica died) now waits for a free slot before its runner starts,
which also delays enforcing its action deadline while the pool is full.
- New internal chat worker settings `HeartbeatRenewalTimeout` and
`HeartbeatLockTimeout`, defaulting to a third of the stale threshold and
half of that.
- Heartbeat ticks now take the capacity admission advisory lock briefly
on each replica.
## Out of scope / follow-ups
- Interrupting a chat still owned by a dead runner (stale lease,
`worker_id` set) still waits for a slot.
- An admission lock timeout should end the acquisition pass instead of
logging a warning per candidate.
- The per-chat `queued_for_capacity` flag in the chat API still reports
only `running` chats.
- Alert on failed heartbeat ticks.
`coderd/x/chatd/ARCHITECTURE.md` gains TODO lines for the affected
sections; the prose is left for the owner.
---
_Generated with [`xum`](https://github.com/coder/xum) • Model:
`anthropic:claude-opus-5-5` • Thinking: `high`_
closes CODAGT-203
## Summary
`list_templates` now returns a ranked shortlist with a recommendation,
so the chat agent can pick the right template the way a colleague would:
prefer what matches the request, what the user already uses, and what
the rest of the organization uses. Instead of teaching the model an enum
protocol in prompts, every result carries a fixed `next_step`
instruction telling the agent what to do.
## How list_templates works
1. **Fetch**: active, non-deprecated templates in the chat's
organization, filtered by the admin template allowlist, authorized as
the chat owner (no system escalation).
2. **Query relevance** (optional `query` argument): each template
receives the highest tier any of its fields matches, and a higher tier
always outranks a lower one regardless of usage:
| Tier | Match |
|------|-------|
| 4 | name or display name equals the query |
| 3 | name or display name starts with the query |
| 2 | name or display name contains the query |
| 1 | description contains the query (checked only when no name field
matched) |
| 0 | no match; the template is excluded |
Matching is case-insensitive and ignores spaces/hyphens/underscores
(`python gpu` matches `python-gpu`).
3. **Usage signals**: a new `GetTemplateRankingSignalsByOwnerID` query
returns, per template, the owner's active and recently-deleted workspace
counts within a 60-day window, the last in-window usage, and the count
of distinct developers with an active workspace (unclaimed prebuilds
excluded).
4. **Affinity score** (computed in Go, per template, from that
template's signals only):
```text
affinity = 10 x (active + 0.5 x deleted) x 0.5^(days_since_last_use /
14)
+ ln(1 + active_developers)
```
`active`/`deleted` are the owner's in-window workspace counts,
`days_since_last_use` is measured from the most recent in-window usage
(the personal term is zero without in-window usage), and
`active_developers` is the org-wide count. Personal usage carries 10x
the weight of org popularity; the confidence floor is the score of two
active developers (`ln 3`) and the required lead over the runner-up is
`ln 3 - ln 2`.
5. **Rank**: query tier first (when a query is present), then affinity
score, then name/ID for determinism. Results paginate 10 per page with
`next_page` present only when more exist.
## Recommendation contract
The result tells the agent what to do next instead of describing
confidence levels:
- `recommended_template_id` is present only when the top template is a
clear winner: the only available template, a decisive query match, or an
affinity score that clears a floor and leads the runner-up by a derived
margin.
- `next_step` is always present and is one of four fixed sentences: use
the recommendation, ask the user to choose, retry a query that matched
nothing, or report that no templates are available.
Per-template items carry raw evidence (`active_developers`,
`your_workspace_count`, `last_used_by_you`) rather than derived labels.
When signals fail to load, the tool logs and degrades to asking the user
unless the query alone is decisive.
Prompts and the `create_workspace`/`read_template` descriptions
reference the field through the `chattool.NextStepField` constant, so
the instruction lives in one place and cannot drift. `create_workspace`
remains idempotent and allowlist-enforced.
## Authorization
The signals query runs with the chat owner's permissions: reading the
owner's own workspaces plus a template-metadata read for the cross-user
popularity count. dbauthz rejects the call if any requested template is
not readable by the owner (covered by allow and deny method tests).
## Docs
Adds `docs/ai-coder/agents/tools/` explaining how agent tool calls work,
with `list_templates` ranking and the `next_step` contract as the first
documented tools.
The agents-access role previously granted chat permissions at user
scope, but chats are org-scoped objects. Rego skips user-level perms
when org_owner is set, making the grants invisible. Handler-level
band-aids used synthetic non-org-scoped objects as a workaround.
- Migrates agents-access from users.rbac_roles (site-level) to
organization_members.roles (org-scoped) via DB migration
- Redefines agents-access as a predefined org-scoped builtin role
alongside organization-admin, organization-auditor, etc., with
Member permissions granting chat create/read/update
- Excludes ResourceChat from OrgMemberPermissions so org membership
alone no longer grants chat access
- Fixes handler Authorize checks to use org-scoped objects with
semantically correct actions (ActionUpdate for message/tool operations)
- Grants org admins the ability to assign agents-access
Closes#24250
Fixes CODAGT-174
Note: this does not update the "Usage" endpoints. Tracked by CODAGT-161.
> 🤖
- Detect `-testify.m` sub-test filtering in `SetupSuite` and skip the `Accounting` check.
> 🤖 This PR was created with the help of Coder Agents, and was reviewed by my human. 🧑💻
The slices package provides type-safe generic replacements for the
old typed sort convenience functions. The codebase already uses
slices.Sort in 43 call sites; this finishes the migration for the
remaining 29.
- sort.Strings(x) -> slices.Sort(x)
- sort.Float64s(x) -> slices.Sort(x)
- sort.StringsAreSorted(x) -> slices.IsSorted(x)
Introduce a three-way workspace sharing setting (none, everyone,
service_accounts) replacing the boolean workspace_sharing_disabled.
In service_accounts mode, only service account-owned workspaces can be
shared while regular members' share permissions are removed. Adds a
new organization-service-account system role with per-org permissions
reconciled alongside the existing organization-member system role.
Related to:
https://linear.app/codercom/issue/PLAT-28/feat-service-accounts-sharing-mode-and-rbac-role
---------
Co-authored-by: Steven Masley <Emyrk@users.noreply.github.com>
Co-authored-by: Kayla はな <mckayla@hey.com>
Fixes all our Go file imports to match the preferred spec that we've _mostly_ been using. For example:
```
import (
"context"
"time"
"github.com/prometheus/client_golang/prometheus"
"golang.org/x/xerrors"
"gopkg.in/natefinch/lumberjack.v2"
"cdr.dev/slog/v3"
"github.com/coder/coder/v2/codersdk/agentsdk"
"github.com/coder/serpent"
)
```
3 groups: standard library, 3rd partly libs, Coder libs.
This PR makes the change across the codebase. The PR in the stack above modifies our formatting to maintain this state of affairs, and is a separate PR so it's possible to review that one in detail.
Upgrades to slog v3 which includes a small, but backward incompatible API change to the acceptible call arguments when logging. This change allows us to verify via compile time type checking that arguments are correct and won't cause a panic, as was possible in slog v1, which this replaces (v2 was tagged but never used in coder/coder).
It also updates dependencies that also use slog and were updated.
I've left the `aibridge` dependency as a commit SHA, under the assumption that the team there (cc @pawbana @dannykopping ) will tag and update the dependency soon and on their own schedule.
Other dependencies, I pushed new tags.
# Add API key allow_list for resource-scoped tokens
This PR adds support for API key allow lists, enabling tokens to be scoped to specific resources. The implementation:
1. Adds a new `allow_list` field to the `CreateTokenRequest` struct, allowing clients to specify resource-specific scopes when creating API tokens
2. Implements `APIAllowListTarget` type to represent resource targets in the format `<type>:<id>` with support for wildcards
3. Adds validation and normalization logic for allow lists to handle wildcards and deduplication
4. Integrates with RBAC by creating an `APIKeyEffectiveScope` that merges API key scopes with allow list restrictions
5. Updates API documentation and TypeScript types to reflect the new functionality
This feature enables creating tokens that are limited to specific resources (like workspaces or templates) by ID, making it possible to create more granular API tokens with limited access.
This is the third PR for moving connection events out of the audit log.
This PR populates `count` on `ConnectionLogResponse` using a separate query, to preemptively mitigate the issue described in #17689. It's structurally identical to a portion of https://github.com/coder/coder/pull/18600, but for the connection log instead of the audit log.
Future PRs:
- Implement a table in the Web UI for viewing connection logs.
- Write a query to delete old events from the audit log, call it from dbpurge.
- Write documentation for the endpoint / feature
Closes#17689
This PR optimizes the audit logs query performance by extracting the
count operation into a separate query and replacing the OR-based
workspace_builds with conditional joins.
## Query changes
* Extracted count query to separate one
* Replaced single `workspace_builds` join with OR conditions with
separate conditional joins
* Added conditional joins
* `wb_build` for workspace_build audit logs (which is a direct lookup)
* `wb_workspace` for workspace create audit logs (via workspace)
Optimized AuditLogsOffset query:
https://explain.dalibo.com/plan/4g1hbedg4a564bg8
New CountAuditLogs query:
https://explain.dalibo.com/plan/ga2fbcecb9efbce3
- Update go.mod to use Go 1.24.1
- Update GitHub Actions setup-go action to use Go 1.24.1
- Fix linting issues with golangci-lint by:
- Updating to golangci-lint v1.57.1 (more compatible with Go 1.24.1)
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <claude@anthropic.com>
Another PR to address https://github.com/coder/coder/issues/15109.
- adds the DisableForeignKeysAndTriggers utility, which simplifies
converting tests from in-mem to postgres
- converts the dbauthz test suite to pass on both the in-mem db and
Postgres
The failure condition being fixed is `w1` and `w2` could belong
to different users, organizations, and templates and still cause a
serializable failure if run concurrently. This is because the old query
did a `seq scan` on the `workspace_builds` table. Since that is the
table being updated, we really want to prevent that.
So before this would fail for any 2 workspaces. Now it only fails if
`w1` and `w2` are owned by the same user and organization.
* chore: create type for unique role names
Using `string` was confusing when something should be combined with
org context, and when not to. Naming this new name, "RoleIdentifier"
Includes db schema and dbauthz layer for upserting custom roles. Unit test in `customroles_test.go` verify against escalating permissions through this feature.
Just moved `rbac.Action` -> `policy.Action`. This is for the stacked PR to not have circular dependencies when doing autogen. Without this, the autogen can produce broken golang code, which prevents the autogen from compiling.
So just avoiding circular dependencies. Doing this in it's own PR to reduce LoC diffs in the primary PR, since this has 0 functional changes.
* Adds UpdateProvisionerDaemonLastSeenAt
* Adds heartbeat to provisioner daemons
* Inserts provisioner daemons to database upon start
* Ensures TagOwner is an empty string and not nil
* Adds COALESCE() in idx_provisioner_daemons_name_owner_key
* chore: add /v2 to import module path
go mod requires semantic versioning with versions greater than 1.x
This was a mechanical update by running:
```
go install github.com/marwan-at-work/mod/cmd/mod@latest
mod upgrade
```
Migrate generated files to import /v2
* Fix gen
* Adds dbmetrics package and wraps database.Store with a Prometheus HistogramVec of timings.
* Adds Wrappers method to database.Store to avoid double-wrapping interfaces
* Fixes test flake in TestLicensesListFake
* chore: Rbac errors should be returned, and not hidden behind 404
SqlErrNoRows was hiding actual errors
* Replace sql.ErrNoRow checks
* Remove sql err no rows check from dbauthz test
* Fix to use dbauthz system user
- rbac: export rbac.Permissions
- dbauthz: move GetDeploymentDAUs, GetTemplateDAUs,
GetTemplateAverageBuildTime from querier.go to system.go
and removes auth checks
- dbauthz: remove AsSystem(), add individual roles for
autostart, provisionerd, add restricted system role for
everything else
The authz library returns a 404 if the authorization fails. If the
context is cancelled, then a 404 message is inaccurate.
Add a unit test to ensure context cancelled errors are raised
properly
feat: Add initial AuthzQuerier implementation
- Adds package database/dbauthz that adds a database.Store implementation where each method goes through AuthZ checks
- Implements all database.Store methods on AuthzQuerier
- Updates and fixes unit tests where required
- Updates coderd initialization to use AuthzQuerier if codersdk.ExperimentAuthzQuerier is enabled