Files
coder/coderd/x/chatd/runner_manager.go
Thomas Kosiewski d11809f095 fix(coderd): enforce chatd concurrent-agent capacity on every path into running (#30243)
## Problem

The concurrent-agent cap (5 root chats and 10 subagent chats per
deployment by default) could be exceeded. Three paths put more chats in
`running` with a fresh lease than the pool allows:

1. **Interrupting a waiting chat.** A user sends a message with
`busy_behavior=interrupt` to a chat that is waiting for capacity. The
chat becomes `interrupting` while unowned, a worker acquires it without
admission, and `FinishInterruption` promotes the queued message into a
running turn under that ownership.
2. **Taking over an orphaned `requires_action` chat.** A replica dies
while its chat is in `requires_action`, its lease goes stale and another
chat takes the slot. A worker acquires the `requires_action` chat
without admission, and a tool result, an interrupt or the action
deadline returns it to `running` under that ownership.
3. **Heartbeat reviving a stale lease.** A replica loses the database
for longer than the 30 second stale threshold and another replica admits
a second chat into the slot. When the first replica comes back, its
heartbeat upsert renews the stale lease unconditionally, and its runner
keeps generating until the turn ends.

## Root cause

`agentCapacityLimiter.Admit` returned `true` without taking the
admission lock or counting for every status other than `running`. The
count query already treats every owned chat with a fresh lease as
holding a slot, so the shortcut let chats acquire a slot uncounted and
later move back to `running` under that ownership. Separately,
`BatchUpsertChatHeartbeats` had no freshness or ownership condition.

## Fix

- **Admission for every runnable status.** `Admit` now counts for
`running`, `interrupting` and `requires_action`, and a refused pool
skips the rest of its candidates in the pass regardless of status. An
owned chat holds its slot until it releases ownership, so every later
transition back to `running` stays within capacity.
- **Fenced heartbeat renewal.** `BatchUpsertChatHeartbeats` is replaced
by `RenewChatHeartbeats`, which only extends rows that are still fresh
and still own their chat, using `statement_timestamp()` read after the
capacity admission lock is granted. The renewal runs under that lock
with a `lock_timeout` (`HeartbeatLockTimeout`) and a per-tick timeout
(`HeartbeatRenewalTimeout`, a third of the stale threshold). Admission
also waits at most 5 seconds for the lock.
- **Stop runners that lost their lease.** `heartbeatOnce` requests
cleanup of every runner whose lease was not renewed, which cancels its
generation. The chat stays owned by the stale lease until an admitted
worker takes it over. A tick that fails or times out cleans up no
runners and retries on the next tick.
- **Instant Stop on a queued chat.** Requiring admission for
`interrupting` would have made Stop on a chat waiting for capacity hang
until a slot frees. When `Interrupt`, or `SendMessage` with
`busy_behavior=interrupt`, runs on a running chat with no owner, the
transition now applies `FinishInterruption` in the same transaction and
cancels orphaned tool calls. An empty queue lands in `waiting`;
otherwise the queue head is promoted into `running`, still unowned, so
it goes through admission.
- **Queued metric.** `CountChatCapacityQueuedByPool` (behind
`coderd_chatd_agents_queued_for_capacity`) now also counts
`interrupting` and `requires_action` chats without a fresh lease, since
they also wait for admission.

## How it was found

A TLA+ model of capacity admission found a counterexample for each path.
Go tests reproduced all three against the previous code (each failed
with `"2" is not less than or equal to "1"` for a root pool of capacity
1), and the model with the fix enabled passes the safety invariant and
the liveness properties. The model itself is not part of this PR.

## Tests

The three reproductions are kept as regression tests in
`coderd/x/chatd/agentcapacity_regression_internal_test.go`:
`TestAdmission_InterruptWithMessageOnWaitingChatStaysWithinCapacity`,
`TestAdmission_RequiresActionTakeoverStaysWithinCapacity` and
`TestRunnerManager_HeartbeatDoesNotReviveStaleLease`. The original
reproductions failed on the base commit. Restoring the admission
shortcut makes the `requires_action` test fail, and restoring the
unconditional heartbeat upsert, or dropping the runner cleanup, makes
the heartbeat test fail. New coverage also includes
`TestAdmission_StopWaitingChatNeedsNoCapacity`,
`TestRenewChatHeartbeats_OnlyRenewsFreshOwnedLeases`,
`TestRunnerManager_FailedHeartbeatTickKeepsRunners`, and chatstate tests
for the unowned interrupt paths. Existing tests that encoded the old
behavior (`requires_action` chats holding no slot, interrupting chats
always acquirable, unconditional heartbeat renewal) were updated.

## Behavior / API change

- `POST /api/v2/chats/{chat}/messages` with `busy_behavior=interrupt` on
a chat that no worker owns (for example one waiting for capacity) now
returns the promoted message with `queued=false` instead of a queued
message, and the chat stays in `running` instead of `interrupting`. If
the chat already had queued messages, the queue head is promoted and the
new message stays queued.
- `POST /api/v2/chats/{chat}/interrupt` on such a chat lands it in
`waiting` immediately (or in `running` with the queue head promoted)
instead of `interrupting`.
- An unowned `interrupting` or `requires_action` chat (for example after
its replica died) now waits for a free slot before its runner starts,
which also delays enforcing its action deadline while the pool is full.
- New internal chat worker settings `HeartbeatRenewalTimeout` and
`HeartbeatLockTimeout`, defaulting to a third of the stale threshold and
half of that.
- Heartbeat ticks now take the capacity admission advisory lock briefly
on each replica.

## Out of scope / follow-ups

- Interrupting a chat still owned by a dead runner (stale lease,
`worker_id` set) still waits for a slot.
- An admission lock timeout should end the acquisition pass instead of
logging a warning per candidate.
- The per-chat `queued_for_capacity` flag in the chat API still reports
only `running` chats.
- Alert on failed heartbeat ticks.

`coderd/x/chatd/ARCHITECTURE.md` gains TODO lines for the affected
sections; the prose is left for the owner.

---
_Generated with [`xum`](https://github.com/coder/xum) • Model:
`anthropic:claude-opus-5-5` • Thinking: `high`_
2026-10-01 16:42:51 +02:00

586 lines
15 KiB
Go

package chatd
import (
"context"
"database/sql"
"encoding/json"
"sync"
"time"
"github.com/google/uuid"
"golang.org/x/xerrors"
"cdr.dev/slog/v3"
"github.com/coder/coder/v2/coderd/database"
coderdpubsub "github.com/coder/coder/v2/coderd/pubsub"
)
const shutdownCleanupTimeout = 5 * time.Second
type runnerKey struct {
ChatID uuid.UUID
RunnerID uuid.UUID
}
type runnerStateUpdate struct {
ChatID uuid.UUID
WorkerID *uuid.UUID
RunnerID *uuid.UUID
SnapshotVersion int64
HistoryVersion int64
QueueVersion int64
GenerationAttempt int64
Status database.ChatStatus
Archived bool
RequiresActionDeadlineAt sql.NullTime
}
type spawnRunnerRequest struct {
ChatID uuid.UUID
WorkerID uuid.UUID
RunnerID uuid.UUID
// TakenOver means the previous owner's heartbeat went stale.
TakenOver bool
}
type runnerRecord struct {
key runnerKey
workerID uuid.UUID
takenOver bool
cancel context.CancelFunc
done <-chan struct{}
stateCh chan runnerStateUpdate
mu sync.Mutex
unsubscribe func()
cleanupStarted bool
}
func (r *runnerRecord) setUnsubscribe(unsubscribe func()) bool {
r.mu.Lock()
if r.cleanupStarted {
r.mu.Unlock()
if unsubscribe != nil {
unsubscribe()
}
return false
}
r.unsubscribe = unsubscribe
r.mu.Unlock()
return true
}
func (r *runnerRecord) startCleanup() {
r.mu.Lock()
if r.cleanupStarted {
r.mu.Unlock()
return
}
r.cleanupStarted = true
unsubscribe := r.unsubscribe
r.unsubscribe = nil
r.mu.Unlock()
if unsubscribe != nil {
unsubscribe()
}
r.cancel()
}
type runnerManager struct {
server *Server
opts chatWorkerOptions
ctx context.Context
closed bool
spawnMu sync.Mutex
mu sync.Mutex
spawnCh chan spawnRunnerRequest
cleanupReqCh chan runnerKey
cleanupDoneCh chan runnerKey
runners map[runnerKey]*runnerRecord
runnersByChat map[uuid.UUID]map[uuid.UUID]*runnerRecord
cleaning map[runnerKey]*runnerRecord
wg sync.WaitGroup
}
func newRunnerManager(ctx context.Context, server *Server, opts chatWorkerOptions) *runnerManager {
return &runnerManager{
server: server,
opts: opts,
ctx: ctx,
spawnCh: make(chan spawnRunnerRequest, opts.RunnerManagerChannelSize),
cleanupReqCh: make(chan runnerKey, opts.RunnerManagerChannelSize),
cleanupDoneCh: make(chan runnerKey, opts.RunnerManagerChannelSize),
runners: make(map[runnerKey]*runnerRecord),
runnersByChat: make(map[uuid.UUID]map[uuid.UUID]*runnerRecord),
cleaning: make(map[runnerKey]*runnerRecord),
}
}
func (m *runnerManager) start() {
m.wg.Go(m.run)
m.wg.Go(m.databaseSyncLoop)
m.wg.Go(m.heartbeatLoop)
m.wg.Go(m.heartbeatCleanupLoop)
}
func (m *runnerManager) wait() {
m.wg.Wait()
}
func (m *runnerManager) idle() bool {
m.mu.Lock()
defer m.mu.Unlock()
return len(m.runners) == 0 && len(m.cleaning) == 0
}
func (m *runnerManager) Spawn(ctx context.Context, req spawnRunnerRequest) error {
m.spawnMu.Lock()
defer m.spawnMu.Unlock()
if m.closed {
return xerrors.New("chatworker: runner manager closed")
}
select {
case m.spawnCh <- req:
return nil
case <-ctx.Done():
return ctx.Err()
case <-m.ctx.Done():
return m.ctx.Err()
}
}
func (m *runnerManager) requestCleanup(ctx context.Context, key runnerKey) {
select {
case m.cleanupReqCh <- key:
case <-ctx.Done():
case <-m.ctx.Done():
}
}
func (m *runnerManager) RouteStateHint(ctx context.Context, state runnerStateUpdate) {
m.mu.Lock()
byRunner := m.runnersByChat[state.ChatID]
targets := make([]*runnerRecord, 0, len(byRunner))
for _, rec := range byRunner {
targets = append(targets, rec)
}
m.mu.Unlock()
for _, rec := range targets {
select {
case rec.stateCh <- state:
case <-rec.done:
// Only this runner exited; keep fanning out to the rest.
continue
case <-ctx.Done():
return
case <-m.ctx.Done():
return
default:
// stateCh is full; drop the hint for this runner.
}
}
}
func (m *runnerManager) run() {
for {
select {
case req := <-m.spawnCh:
m.handleSpawn(req)
case key := <-m.cleanupReqCh:
m.handleCleanupRequest(key)
case key := <-m.cleanupDoneCh:
m.handleCleanupDone(key)
case <-m.ctx.Done():
queued := m.closeAndDrainQueues()
m.cancelAll()
m.releaseOwnershipHints(queued)
return
}
}
}
func (m *runnerManager) handleSpawn(req spawnRunnerRequest) {
key := runnerKey{ChatID: req.ChatID, RunnerID: req.RunnerID}
m.mu.Lock()
if _, ok := m.runners[key]; ok {
// A duplicate spawn for a live runner indicates a logic error
// in the sync loop.
m.opts.Logger.Error(m.ctx, "invalid spawn request: chat runner already spawned", slog.F("key", key))
m.mu.Unlock()
return
}
if _, ok := m.cleaning[key]; ok {
// A duplicate spawn for a live runner indicates a logic error
// in the sync loop.
m.opts.Logger.Error(m.ctx, "invalid spawn request: chat runner in cleanup", slog.F("key", key))
m.mu.Unlock()
return
}
runnerCtx, cancel := context.WithCancel(m.ctx)
done := make(chan struct{})
rec := &runnerRecord{
key: key,
workerID: req.WorkerID,
takenOver: req.TakenOver,
cancel: cancel,
done: done,
stateCh: make(chan runnerStateUpdate, m.opts.StateChannelSize),
}
m.runners[key] = rec
if m.runnersByChat[req.ChatID] == nil {
m.runnersByChat[req.ChatID] = make(map[uuid.UUID]*runnerRecord)
}
m.runnersByChat[req.ChatID][req.RunnerID] = rec
m.mu.Unlock()
r := newRunner(runnerCtx, m, rec, m.opts)
m.wg.Go(func() {
defer close(done)
r.run()
})
}
func (m *runnerManager) closeAndDrainQueues() []runnerKey {
m.spawnMu.Lock()
defer m.spawnMu.Unlock()
m.closed = true
return m.drainQueues()
}
func (m *runnerManager) drainQueues() []runnerKey {
queued := make([]runnerKey, 0)
for {
select {
case req := <-m.spawnCh:
queued = append(queued, runnerKey{ChatID: req.ChatID, RunnerID: req.RunnerID})
case key := <-m.cleanupReqCh:
m.handleCleanupRequest(key)
case key := <-m.cleanupDoneCh:
m.handleCleanupDone(key)
default:
return queued
}
}
}
func (m *runnerManager) handleCleanupRequest(key runnerKey) {
m.mu.Lock()
rec, ok := m.runners[key]
if !ok {
m.mu.Unlock()
return
}
delete(m.runners, key)
if byChat := m.runnersByChat[key.ChatID]; byChat != nil {
delete(byChat, key.RunnerID)
if len(byChat) == 0 {
delete(m.runnersByChat, key.ChatID)
}
}
m.cleaning[key] = rec
m.mu.Unlock()
rec.startCleanup()
m.registerCleanupWaiter(key, rec)
}
func (m *runnerManager) registerCleanupWaiter(key runnerKey, rec *runnerRecord) {
m.wg.Go(func() {
<-rec.done
if m.ctx.Err() != nil {
m.mu.Lock()
delete(m.cleaning, key)
m.mu.Unlock()
return
}
select {
case m.cleanupDoneCh <- key:
case <-m.ctx.Done():
m.mu.Lock()
delete(m.cleaning, key)
m.mu.Unlock()
}
})
}
func (m *runnerManager) handleCleanupDone(key runnerKey) {
m.mu.Lock()
delete(m.cleaning, key)
m.mu.Unlock()
}
func (m *runnerManager) cancelAll() {
type cleanupTarget struct {
key runnerKey
rec *runnerRecord
}
m.mu.Lock()
active := make([]cleanupTarget, 0, len(m.runners))
cleaning := make([]*runnerRecord, 0, len(m.cleaning))
for _, rec := range m.cleaning {
cleaning = append(cleaning, rec)
}
for key, rec := range m.runners {
delete(m.runners, key)
m.cleaning[key] = rec
active = append(active, cleanupTarget{key: key, rec: rec})
}
clear(m.runnersByChat)
m.mu.Unlock()
keys := make([]runnerKey, 0, len(cleaning)+len(active))
for _, rec := range cleaning {
rec.startCleanup()
keys = append(keys, rec.key)
}
for _, target := range active {
target.rec.startCleanup()
m.registerCleanupWaiter(target.key, target.rec)
keys = append(keys, target.key)
}
m.releaseOwnershipHints(keys)
}
func (m *runnerManager) releaseOwnershipHints(keys []runnerKey) {
if len(keys) == 0 {
return
}
ctx, cancel := context.WithTimeout(context.WithoutCancel(m.ctx), shutdownCleanupTimeout)
defer cancel()
chatIDs := make([]uuid.UUID, 0, len(keys))
runnerIDs := make([]uuid.UUID, 0, len(keys))
uniqueChatIDs := make(map[uuid.UUID]struct{}, len(keys))
for _, key := range keys {
chatIDs = append(chatIDs, key.ChatID)
runnerIDs = append(runnerIDs, key.RunnerID)
uniqueChatIDs[key.ChatID] = struct{}{}
}
if _, err := m.opts.Store.BatchDeleteChatHeartbeats(ctx, database.BatchDeleteChatHeartbeatsParams{
ChatIds: chatIDs,
RunnerIds: runnerIDs,
}); err != nil {
m.opts.Logger.Warn(ctx, "chatworker shutdown heartbeat cleanup failed", slogError(err))
}
syncIDs := make([]uuid.UUID, 0, len(uniqueChatIDs))
for id := range uniqueChatIDs {
syncIDs = append(syncIDs, id)
}
chats, err := m.opts.Store.GetChatsByIDsForRunnerSync(ctx, syncIDs)
if err != nil {
m.opts.Logger.Warn(ctx, "chatworker shutdown ownership lookup failed", slogError(err))
}
snapshotByChat := make(map[uuid.UUID]int64, len(chats))
for _, chat := range chats {
snapshotByChat[chat.ID] = chat.SnapshotVersion
}
for _, key := range keys {
payload, err := json.Marshal(coderdpubsub.ChatStateOwnershipMessage{
ChatID: key.ChatID,
SnapshotVersion: snapshotByChat[key.ChatID],
})
if err != nil {
m.opts.Logger.Warn(ctx, "chatworker shutdown ownership marshal failed", slogError(err))
continue
}
if err := m.opts.Pubsub.Publish(coderdpubsub.ChatStateOwnershipChannel, payload); err != nil {
m.opts.Logger.Warn(ctx, "chatworker shutdown ownership publish failed", slogError(err))
}
}
}
func (m *runnerManager) snapshotRunnerKeys() []runnerKey {
m.mu.Lock()
defer m.mu.Unlock()
keys := make([]runnerKey, 0, len(m.runners))
for key := range m.runners {
keys = append(keys, key)
}
return keys
}
func (m *runnerManager) databaseSyncLoop() {
ticker := m.opts.Clock.NewTicker(m.opts.RunnerSyncInterval, "chatworker", "runner-sync")
defer ticker.Stop()
for {
select {
case <-ticker.C:
if err := m.syncOnce(m.ctx); err != nil {
m.opts.Logger.Warn(m.ctx, "chatworker runner sync failed", slogError(err))
}
case <-m.ctx.Done():
return
}
}
}
func (m *runnerManager) syncOnce(ctx context.Context) error {
keys := m.snapshotRunnerKeys()
if len(keys) == 0 {
return nil
}
idsByChat := make(map[uuid.UUID]struct{}, len(keys))
for _, key := range keys {
idsByChat[key.ChatID] = struct{}{}
}
chatIDs := make([]uuid.UUID, 0, len(idsByChat))
for id := range idsByChat {
chatIDs = append(chatIDs, id)
}
chats, err := m.opts.Store.GetChatsByIDsForRunnerSync(ctx, chatIDs)
if err != nil {
return xerrors.Errorf("get chats for runner sync: %w", err)
}
seen := make(map[uuid.UUID]struct{}, len(chats))
for _, chat := range chats {
seen[chat.ID] = struct{}{}
m.RouteStateHint(ctx, stateUpdateFromChat(chat))
}
for _, key := range keys {
if _, ok := seen[key.ChatID]; !ok {
m.requestCleanup(ctx, key)
}
}
return nil
}
func (m *runnerManager) heartbeatLoop() {
ticker := m.opts.Clock.NewTicker(m.opts.HeartbeatInterval, "chatworker", "heartbeat")
defer ticker.Stop()
for {
select {
case <-ticker.C:
if err := m.heartbeatOnce(m.ctx); err != nil {
m.opts.Logger.Warn(m.ctx, "chatworker heartbeat failed", slogError(err))
}
case <-m.ctx.Done():
return
}
}
}
func (m *runnerManager) heartbeatOnce(ctx context.Context) error {
keys := m.snapshotRunnerKeys()
if len(keys) == 0 {
return nil
}
chatIDs := make([]uuid.UUID, 0, len(keys))
runnerIDs := make([]uuid.UUID, 0, len(keys))
for _, key := range keys {
chatIDs = append(chatIDs, key.ChatID)
runnerIDs = append(runnerIDs, key.RunnerID)
}
renewed, err := m.renewHeartbeats(ctx, chatIDs, runnerIDs)
if err != nil {
// A failed or timed-out tick says nothing about individual
// leases, so no runner is cleaned up. The next tick retries.
return xerrors.Errorf("renew chat heartbeats: %w", err)
}
renewedKeys := make(map[runnerKey]struct{}, len(renewed))
for _, row := range renewed {
renewedKeys[runnerKey{ChatID: row.ChatID, RunnerID: row.RunnerID}] = struct{}{}
}
for _, key := range keys {
if _, ok := renewedKeys[key]; ok {
continue
}
// The lease went stale or lost ownership, so its capacity slot
// may belong to another chat. Stop generating; the chat stays
// owned by the stale lease until an admitted worker takes it over.
m.opts.Logger.Warn(ctx, "chatworker heartbeat lease lost, stopping runner",
slog.F("chat_id", key.ChatID), slog.F("runner_id", key.RunnerID))
m.requestCleanup(ctx, key)
}
return nil
}
// renewHeartbeats renews the given leases under the capacity admission
// lock and returns the renewed pairs. The tick is bounded by
// HeartbeatRenewalTimeout and its lock wait by HeartbeatLockTimeout, so a
// stalled lock holder fails the tick instead of blocking it past the
// stale threshold.
func (m *runnerManager) renewHeartbeats(ctx context.Context, chatIDs, runnerIDs []uuid.UUID) ([]database.RenewChatHeartbeatsRow, error) {
ctx, cancel := context.WithTimeout(ctx, m.opts.HeartbeatRenewalTimeout)
defer cancel()
var renewed []database.RenewChatHeartbeatsRow
// Renewal serializes with capacity admission so that a lease an
// admission counted as stale, and whose slot it gave away, cannot be
// renewed afterwards.
err := m.opts.Store.InTx(func(tx database.Store) error {
if err := tx.SetTransactionLockTimeout(ctx, m.opts.HeartbeatLockTimeout.Milliseconds()); err != nil {
return xerrors.Errorf("set lock timeout: %w", err)
}
if err := tx.AcquireLock(ctx, database.LockIDChatCapacityAdmission); err != nil {
return xerrors.Errorf("acquire capacity admission lock: %w", err)
}
var err error
renewed, err = tx.RenewChatHeartbeats(ctx, database.RenewChatHeartbeatsParams{
ChatIds: chatIDs,
RunnerIds: runnerIDs,
StaleSeconds: m.opts.HeartbeatStaleSeconds,
})
return err
}, nil)
if err != nil {
return nil, err
}
return renewed, nil
}
func (m *runnerManager) heartbeatCleanupLoop() {
ticker := m.opts.Clock.NewTicker(m.opts.HeartbeatCleanupInterval, "chatworker", "heartbeat-cleanup")
defer ticker.Stop()
for {
select {
case <-ticker.C:
if err := m.heartbeatCleanupOnce(m.ctx); err != nil {
m.opts.Logger.Warn(m.ctx, "chatworker heartbeat cleanup failed", slogError(err))
}
case <-m.ctx.Done():
return
}
}
}
func (m *runnerManager) heartbeatCleanupOnce(ctx context.Context) error {
_, err := m.opts.Store.DeleteStaleChatHeartbeats(ctx, m.opts.HeartbeatStaleSeconds)
return err
}
func stateUpdateFromChat(chat database.Chat) runnerStateUpdate {
var workerID *uuid.UUID
if chat.WorkerID.Valid {
id := chat.WorkerID.UUID
workerID = &id
}
var runnerID *uuid.UUID
if chat.RunnerID.Valid {
id := chat.RunnerID.UUID
runnerID = &id
}
return runnerStateUpdate{
ChatID: chat.ID,
WorkerID: workerID,
RunnerID: runnerID,
SnapshotVersion: chat.SnapshotVersion,
HistoryVersion: chat.HistoryVersion,
QueueVersion: chat.QueueVersion,
GenerationAttempt: chat.GenerationAttempt,
Status: chat.Status,
Archived: chat.Archived,
RequiresActionDeadlineAt: chat.RequiresActionDeadlineAt,
}
}
func slogError(err error) slog.Field {
return slog.Error(err)
}