Files
coder/coderd/x/chatd/worker_internal_test.go
T
Thomas Kosiewski d11809f095 fix(coderd): enforce chatd concurrent-agent capacity on every path into running (#30243)
## Problem

The concurrent-agent cap (5 root chats and 10 subagent chats per
deployment by default) could be exceeded. Three paths put more chats in
`running` with a fresh lease than the pool allows:

1. **Interrupting a waiting chat.** A user sends a message with
`busy_behavior=interrupt` to a chat that is waiting for capacity. The
chat becomes `interrupting` while unowned, a worker acquires it without
admission, and `FinishInterruption` promotes the queued message into a
running turn under that ownership.
2. **Taking over an orphaned `requires_action` chat.** A replica dies
while its chat is in `requires_action`, its lease goes stale and another
chat takes the slot. A worker acquires the `requires_action` chat
without admission, and a tool result, an interrupt or the action
deadline returns it to `running` under that ownership.
3. **Heartbeat reviving a stale lease.** A replica loses the database
for longer than the 30 second stale threshold and another replica admits
a second chat into the slot. When the first replica comes back, its
heartbeat upsert renews the stale lease unconditionally, and its runner
keeps generating until the turn ends.

## Root cause

`agentCapacityLimiter.Admit` returned `true` without taking the
admission lock or counting for every status other than `running`. The
count query already treats every owned chat with a fresh lease as
holding a slot, so the shortcut let chats acquire a slot uncounted and
later move back to `running` under that ownership. Separately,
`BatchUpsertChatHeartbeats` had no freshness or ownership condition.

## Fix

- **Admission for every runnable status.** `Admit` now counts for
`running`, `interrupting` and `requires_action`, and a refused pool
skips the rest of its candidates in the pass regardless of status. An
owned chat holds its slot until it releases ownership, so every later
transition back to `running` stays within capacity.
- **Fenced heartbeat renewal.** `BatchUpsertChatHeartbeats` is replaced
by `RenewChatHeartbeats`, which only extends rows that are still fresh
and still own their chat, using `statement_timestamp()` read after the
capacity admission lock is granted. The renewal runs under that lock
with a `lock_timeout` (`HeartbeatLockTimeout`) and a per-tick timeout
(`HeartbeatRenewalTimeout`, a third of the stale threshold). Admission
also waits at most 5 seconds for the lock.
- **Stop runners that lost their lease.** `heartbeatOnce` requests
cleanup of every runner whose lease was not renewed, which cancels its
generation. The chat stays owned by the stale lease until an admitted
worker takes it over. A tick that fails or times out cleans up no
runners and retries on the next tick.
- **Instant Stop on a queued chat.** Requiring admission for
`interrupting` would have made Stop on a chat waiting for capacity hang
until a slot frees. When `Interrupt`, or `SendMessage` with
`busy_behavior=interrupt`, runs on a running chat with no owner, the
transition now applies `FinishInterruption` in the same transaction and
cancels orphaned tool calls. An empty queue lands in `waiting`;
otherwise the queue head is promoted into `running`, still unowned, so
it goes through admission.
- **Queued metric.** `CountChatCapacityQueuedByPool` (behind
`coderd_chatd_agents_queued_for_capacity`) now also counts
`interrupting` and `requires_action` chats without a fresh lease, since
they also wait for admission.

## How it was found

A TLA+ model of capacity admission found a counterexample for each path.
Go tests reproduced all three against the previous code (each failed
with `"2" is not less than or equal to "1"` for a root pool of capacity
1), and the model with the fix enabled passes the safety invariant and
the liveness properties. The model itself is not part of this PR.

## Tests

The three reproductions are kept as regression tests in
`coderd/x/chatd/agentcapacity_regression_internal_test.go`:
`TestAdmission_InterruptWithMessageOnWaitingChatStaysWithinCapacity`,
`TestAdmission_RequiresActionTakeoverStaysWithinCapacity` and
`TestRunnerManager_HeartbeatDoesNotReviveStaleLease`. The original
reproductions failed on the base commit. Restoring the admission
shortcut makes the `requires_action` test fail, and restoring the
unconditional heartbeat upsert, or dropping the runner cleanup, makes
the heartbeat test fail. New coverage also includes
`TestAdmission_StopWaitingChatNeedsNoCapacity`,
`TestRenewChatHeartbeats_OnlyRenewsFreshOwnedLeases`,
`TestRunnerManager_FailedHeartbeatTickKeepsRunners`, and chatstate tests
for the unowned interrupt paths. Existing tests that encoded the old
behavior (`requires_action` chats holding no slot, interrupting chats
always acquirable, unconditional heartbeat renewal) were updated.

## Behavior / API change

- `POST /api/v2/chats/{chat}/messages` with `busy_behavior=interrupt` on
a chat that no worker owns (for example one waiting for capacity) now
returns the promoted message with `queued=false` instead of a queued
message, and the chat stays in `running` instead of `interrupting`. If
the chat already had queued messages, the queue head is promoted and the
new message stays queued.
- `POST /api/v2/chats/{chat}/interrupt` on such a chat lands it in
`waiting` immediately (or in `running` with the queue head promoted)
instead of `interrupting`.
- An unowned `interrupting` or `requires_action` chat (for example after
its replica died) now waits for a free slot before its runner starts,
which also delays enforcing its action deadline while the pool is full.
- New internal chat worker settings `HeartbeatRenewalTimeout` and
`HeartbeatLockTimeout`, defaulting to a third of the stale threshold and
half of that.
- Heartbeat ticks now take the capacity admission advisory lock briefly
on each replica.

## Out of scope / follow-ups

- Interrupting a chat still owned by a dead runner (stale lease,
`worker_id` set) still waits for a slot.
- An admission lock timeout should end the acquisition pass instead of
logging a warning per candidate.
- The per-chat `queued_for_capacity` flag in the chat API still reports
only `running` chats.
- Alert on failed heartbeat ticks.

`coderd/x/chatd/ARCHITECTURE.md` gains TODO lines for the affected
sections; the prose is left for the owner.

---
_Generated with [`xum`](https://github.com/coder/xum) • Model:
`anthropic:claude-opus-5-5` • Thinking: `high`_
2026-10-01 16:42:51 +02:00

435 lines
14 KiB
Go

package chatd
import (
"context"
"database/sql"
"errors"
"testing"
"time"
"github.com/google/uuid"
"github.com/stretchr/testify/require"
"github.com/coder/coder/v2/coderd/database"
"github.com/coder/coder/v2/coderd/x/chatd/chatloop"
"github.com/coder/coder/v2/testutil"
"github.com/coder/quartz"
)
func TestWorker_NewRequiresTaskStarterOrMessagePartBuffer(t *testing.T) {
t.Parallel()
f := newWorkerTestFixture(t)
_, err := newChatWorker(newUnstartedServer(t, f.pubsub, f.db), chatWorkerOptions{WorkerID: uuid.New(), Store: f.db, Pubsub: f.pubsub})
require.ErrorContains(t, err, "task starter or message part buffer is required")
}
func TestWorker_NewRequiresServer(t *testing.T) {
t.Parallel()
f := newWorkerTestFixture(t)
_, err := newChatWorker(nil, testOptions(t, f, newRecordingTaskStarter()))
require.ErrorContains(t, err, "server is required")
}
func TestWorker_NewRequiresWorkerID(t *testing.T) {
t.Parallel()
f := newWorkerTestFixture(t)
opts := testOptions(t, f, newRecordingTaskStarter())
opts.WorkerID = uuid.Nil
_, err := newChatWorker(newUnstartedServer(t, f.pubsub, f.db), opts)
require.ErrorContains(t, err, "worker ID is required")
}
func TestWorker_UsesConfiguredWorkerID(t *testing.T) {
t.Parallel()
f := newWorkerTestFixture(t)
starter := newRecordingTaskStarter()
opts := testOptions(t, f, starter)
workerID := opts.WorkerID
worker, err := newChatWorker(newUnstartedServer(t, f.pubsub, f.db), opts)
require.NoError(t, err)
require.Equal(t, workerID, worker.opts.WorkerID)
require.NoError(t, worker.Start(context.Background()))
require.Equal(t, workerID, worker.opts.WorkerID)
require.NoError(t, worker.Close())
}
func TestWorker_AcquiresRunnableChatFromOwnershipHint(t *testing.T) {
t.Parallel()
f := newWorkerTestFixture(t)
chat := f.createRunningChat(t)
starter := newRecordingTaskStarter()
worker := startWorker(t, testOptions(t, f, starter))
call := starter.waitCall(t, taskKindGeneration, chat.ID)
require.Equal(t, worker.opts.WorkerID, call.input.WorkerID)
require.Equal(t, database.ChatStatusRunning, call.input.Status)
require.NotEqual(t, uuid.Nil, call.input.RunnerID)
latest, err := f.db.GetChatByID(testutil.Context(t, testutil.WaitShort), chat.ID)
require.NoError(t, err)
require.Equal(t, worker.opts.WorkerID, latest.WorkerID.UUID)
require.Equal(t, call.input.RunnerID, latest.RunnerID.UUID)
_, err = f.db.GetChatHeartbeat(testutil.Context(t, testutil.WaitShort), database.GetChatHeartbeatParams{
ChatID: chat.ID,
RunnerID: call.input.RunnerID,
})
require.NoError(t, err)
}
func TestWorker_AcquiresRequiresActionChatFromOwnershipHint(t *testing.T) {
t.Parallel()
f := newWorkerTestFixture(t)
chat := f.createRequiresActionChat(t)
starter := newRecordingTaskStarter()
startWorker(t, testOptions(t, f, starter))
call := starter.waitCall(t, taskKindRequiresActionTimeout, chat.ID)
require.Equal(t, database.ChatStatusRequiresAction, call.input.Status)
require.True(t, call.input.RequiresActionDeadlineAt.Valid)
}
func TestWorker_SkipsFreshlyOwnedChat(t *testing.T) {
t.Parallel()
f := newWorkerTestFixture(t)
chat := f.createRunningChat(t)
otherWorker := uuid.New()
otherRunner := uuid.New()
acquireChat(t, f, chat.ID, otherWorker, otherRunner)
starter := newRecordingTaskStarter()
worker := startWorker(t, testOptions(t, f, starter))
worker.Wake()
starter.assertNoCall(t)
latest, err := f.db.GetChatByID(testutil.Context(t, testutil.WaitShort), chat.ID)
require.NoError(t, err)
require.Equal(t, otherWorker, latest.WorkerID.UUID)
require.Equal(t, otherRunner, latest.RunnerID.UUID)
}
func TestWorker_ReacquiresStaleOwnedChat(t *testing.T) {
t.Parallel()
f := newWorkerTestFixture(t)
chat := f.createRunningChat(t)
deadWorker := uuid.New()
deadRunner := uuid.New()
acquireChat(t, f, chat.ID, deadWorker, deadRunner)
makeHeartbeatStale(t, f, chat.ID, deadRunner)
starter := newBlockingTaskStarter(false)
worker := startWorker(t, testOptions(t, f, starter))
call := starter.waitCall(t, taskKindGeneration, chat.ID)
require.Equal(t, worker.opts.WorkerID, call.input.WorkerID)
require.Equal(t, database.ChatStatusRunning, call.input.Status)
require.NotEqual(t, deadRunner, call.input.RunnerID)
latest, err := f.db.GetChatByID(testutil.Context(t, testutil.WaitShort), chat.ID)
require.NoError(t, err)
require.Equal(t, worker.opts.WorkerID, latest.WorkerID.UUID)
require.Equal(t, call.input.RunnerID, latest.RunnerID.UUID)
require.NotEqual(t, deadWorker, latest.WorkerID.UUID)
require.NotEqual(t, deadRunner, latest.RunnerID.UUID)
_, err = f.db.GetChatHeartbeat(testutil.Context(t, testutil.WaitShort), database.GetChatHeartbeatParams{
ChatID: chat.ID,
RunnerID: call.input.RunnerID,
})
require.NoError(t, err)
}
func TestWorker_TwoWorkersRaceSingleOwner(t *testing.T) {
t.Parallel()
f := newWorkerTestFixture(t)
chat := f.createRunningChat(t)
firstStarter := newRecordingTaskStarter()
secondStarter := newRecordingTaskStarter()
first := startWorker(t, testOptions(t, f, firstStarter))
second := startWorker(t, testOptions(t, f, secondStarter))
call := waitAnyTaskCall(t, firstStarter, secondStarter, taskKindGeneration, chat.ID)
require.Contains(t, []uuid.UUID{first.opts.WorkerID, second.opts.WorkerID}, call.input.WorkerID)
firstStarter.assertNoCall(t)
secondStarter.assertNoCall(t)
latest, err := f.db.GetChatByID(testutil.Context(t, testutil.WaitShort), chat.ID)
require.NoError(t, err)
require.True(t, latest.WorkerID.Valid)
require.True(t, latest.RunnerID.Valid)
require.Equal(t, call.input.WorkerID, latest.WorkerID.UUID)
require.Equal(t, call.input.RunnerID, latest.RunnerID.UUID)
}
func TestWorker_AcquisitionBatchSizeLimitsSuccessfulAcquisitions(t *testing.T) {
t.Parallel()
f := newWorkerTestFixture(t)
first := f.createRunningChat(t)
second := f.createRunningChat(t)
third := f.createRunningChat(t)
starter := newRecordingTaskStarter()
opts := testOptions(t, f, starter)
opts.AcquisitionBatchSize = 1
worker := startWorker(t, opts)
want := map[uuid.UUID]bool{first.ID: true, second.ID: true, third.ID: true}
for range 3 {
call := starter.waitCall(t, taskKindGeneration, uuid.Nil)
delete(want, call.input.ChatID)
starter.assertNoCall(t)
worker.Wake()
}
require.Empty(t, want)
}
func TestWorker_DoesNotAcquireIdleOrArchivedChats(t *testing.T) {
t.Parallel()
f := newWorkerTestFixture(t)
waiting := f.createRunningChat(t)
finishTurn(t, f, waiting.ID)
errorChat := f.createRunningChat(t)
forceExecutionStateAndPublish(t, f, errorChat.ID, database.ChatStatusError, false)
archived := f.createRunningChat(t)
forceExecutionStateAndPublish(t, f, archived.ID, database.ChatStatusRunning, true)
starter := newRecordingTaskStarter()
worker := startWorker(t, testOptions(t, f, starter))
worker.Wake()
starter.assertNoCall(t)
}
func TestWorker_HeartbeatLoopRefreshesActiveRunnerHeartbeat(t *testing.T) {
t.Parallel()
f := newWorkerTestFixture(t)
chat := f.createRunningChat(t)
clock := quartz.NewMock(t)
heartbeatTrap := clock.Trap().NewTicker("chatworker", "heartbeat")
defer heartbeatTrap.Close()
starter := newBlockingTaskStarter(false)
opts := testOptions(t, f, starter)
opts.Clock = clock
opts.HeartbeatInterval = time.Minute
startWorker(t, opts)
heartbeatTrap.MustWait(testutil.Context(t, testutil.WaitLong)).MustRelease(testutil.Context(t, testutil.WaitLong))
call := starter.waitCall(t, taskKindGeneration, chat.ID)
// Renewal only extends a lease that is still fresh.
setHeartbeatAge(t, f, chat.ID, call.input.RunnerID, 10*time.Second)
aged, err := f.db.GetChatHeartbeat(testutil.Context(t, testutil.WaitShort), database.GetChatHeartbeatParams{
ChatID: chat.ID,
RunnerID: call.input.RunnerID,
})
require.NoError(t, err)
oldHeartbeat := aged.HeartbeatAt
clock.Advance(time.Minute).MustWait(testutil.Context(t, testutil.WaitLong))
testutil.Eventually(testutil.Context(t, testutil.WaitLong), t, func(ctx context.Context) bool {
heartbeat, err := f.db.GetChatHeartbeat(ctx, database.GetChatHeartbeatParams{
ChatID: chat.ID,
RunnerID: call.input.RunnerID,
})
return err == nil && heartbeat.HeartbeatAt.After(oldHeartbeat)
}, testutil.IntervalFast, "heartbeat should be refreshed")
}
func TestWorker_HeartbeatCleanupDeletesStaleRows(t *testing.T) {
t.Parallel()
f := newWorkerTestFixture(t)
chat := f.createRunningChat(t)
finishTurn(t, f, chat.ID)
runnerID := uuid.New()
require.NoError(t, f.db.UpsertChatHeartbeat(testutil.Context(t, testutil.WaitShort), database.UpsertChatHeartbeatParams{
ChatID: chat.ID,
RunnerID: runnerID,
}))
makeHeartbeatStale(t, f, chat.ID, runnerID)
clock := quartz.NewMock(t)
cleanupTrap := clock.Trap().NewTicker("chatworker", "heartbeat-cleanup")
defer cleanupTrap.Close()
starter := newRecordingTaskStarter()
opts := testOptions(t, f, starter)
opts.Clock = clock
opts.HeartbeatCleanupInterval = time.Minute
startWorker(t, opts)
cleanupTrap.MustWait(testutil.Context(t, testutil.WaitLong)).MustRelease(testutil.Context(t, testutil.WaitLong))
clock.Advance(time.Minute).MustWait(testutil.Context(t, testutil.WaitLong))
testutil.Eventually(testutil.Context(t, testutil.WaitLong), t, func(ctx context.Context) bool {
_, err := f.db.GetChatHeartbeat(ctx, database.GetChatHeartbeatParams{
ChatID: chat.ID,
RunnerID: runnerID,
})
return errors.Is(err, sql.ErrNoRows)
}, testutil.IntervalFast)
}
func TestWorker_CloseDeletesOwnedHeartbeatsAndPublishesOwnershipHints(t *testing.T) {
t.Parallel()
f := newWorkerTestFixture(t)
first := f.createRunningChat(t)
second := f.createRunningChat(t)
starter := newBlockingTaskStarter(false)
pubsub := newRecordingPubsub(f.pubsub)
opts := testOptions(t, f, starter)
opts.Pubsub = pubsub
worker := startWorker(t, opts)
callsByChat := make(map[uuid.UUID]taskCall)
for range 2 {
call := starter.waitCall(t, taskKindGeneration, uuid.Nil)
callsByChat[call.input.ChatID] = call
}
require.Contains(t, callsByChat, first.ID)
require.Contains(t, callsByChat, second.ID)
require.NoError(t, worker.Close())
for _, call := range callsByChat {
_, err := f.db.GetChatHeartbeat(testutil.Context(t, testutil.WaitShort), database.GetChatHeartbeatParams{
ChatID: call.input.ChatID,
RunnerID: call.input.RunnerID,
})
require.ErrorIs(t, err, sql.ErrNoRows)
}
messages := pubsub.ownershipMessages(t)
seen := make(map[uuid.UUID]bool)
for _, msg := range messages {
seen[msg.ChatID] = true
require.NotZero(t, msg.SnapshotVersion)
}
require.True(t, seen[first.ID], "expected ownership hint for first runner")
require.True(t, seen[second.ID], "expected ownership hint for second runner")
}
func TestWorker_CloseIsIdempotentAndDoesNotBlock(t *testing.T) {
t.Parallel()
f := newWorkerTestFixture(t)
chat := f.createRunningChat(t)
starter := newBlockingTaskStarter(false)
worker := startWorker(t, testOptions(t, f, starter))
call := starter.waitCall(t, taskKindGeneration, chat.ID)
closed := make(chan error, 1)
go func() {
if err := worker.Close(); err != nil {
closed <- err
return
}
closed <- worker.Close()
}()
select {
case err := <-closed:
require.NoError(t, err)
case <-time.After(testutil.WaitLong):
t.Fatal("worker close did not return")
}
select {
case <-call.ctx.Done():
case <-time.After(testutil.WaitLong):
t.Fatal("active task was not canceled")
}
}
func waitAnyTaskCall(
t *testing.T,
first *recordingTaskStarter,
second *recordingTaskStarter,
kind taskKind,
chatID uuid.UUID,
) taskCall {
t.Helper()
deadline := time.After(testutil.WaitLong)
for {
select {
case call := <-first.callCh:
if call.kind == kind && call.input.ChatID == chatID {
return call
}
case call := <-second.callCh:
if call.kind == kind && call.input.ChatID == chatID {
return call
}
case <-deadline:
t.Fatal("timed out waiting for either worker to start task")
return taskCall{}
}
}
}
func requireTaskCanceled(t *testing.T, call taskCall) {
t.Helper()
select {
case <-call.ctx.Done():
require.True(t, errors.Is(call.ctx.Err(), context.Canceled))
case <-time.After(testutil.WaitLong):
t.Fatal("task context was not canceled")
}
}
func TestWorkerRunnerTurnSpan(t *testing.T) {
t.Parallel()
tests := []struct {
name string
staleOwner bool
wantAcquisitions int
}{
{name: "Unowned", wantAcquisitions: 1},
{name: "StaleOwner", staleOwner: true},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
ctx := testutil.Context(t, testutil.WaitLong)
f := newWorkerTestFixture(t)
chat := f.createRunningChat(t)
if tt.staleOwner {
deadRunner := uuid.New()
acquireChat(t, f, chat.ID, uuid.New(), deadRunner)
makeHeartbeatStale(t, f, chat.ID, deadRunner)
}
tracer, recorder := newStageTestTracer(t)
server := newUnstartedServer(t, f.pubsub, f.db)
server.stages = tracer
starter := newBlockingTaskStarter(false)
worker, err := newChatWorker(server, testOptions(t, f, starter))
require.NoError(t, err)
require.NoError(t, worker.Start(context.Background()))
call := starter.waitCall(t, taskKindGeneration, chat.ID)
require.NotNil(t, call.input.TurnSpan)
call.input.TurnSpan.Ensure(ctx, call.input.TaskID, chat, chat.CreatedAt)
require.NoError(t, worker.Close())
turns := turnSpansByStart(t, recorder)
require.Len(t, turns, 1)
outcome, _ := spanAttribute(t, turns[0], chatloop.AttrTurnOutcome)
require.Equal(t, chatloop.TurnOutcomeAbandoned, chatloop.TurnOutcome(outcome.AsString()))
require.Len(t, stageSpansByStart(t, recorder, chatloop.StageAcquisition), tt.wantAcquisitions)
})
}
}
func TestWorkerRunnerReleasesTurnOnTaskExit(t *testing.T) {
t.Parallel()
ctx := testutil.Context(t, testutil.WaitLong)
f := newWorkerTestFixture(t)
chat := f.createRunningChat(t)
tracer, recorder := newStageTestTracer(t)
server := newUnstartedServer(t, f.pubsub, f.db)
server.stages = tracer
starter := newBlockingTaskStarter(false)
worker, err := newChatWorker(server, testOptions(t, f, starter))
require.NoError(t, err)
require.NoError(t, worker.Start(context.Background()))
t.Cleanup(func() { _ = worker.Close() })
call := starter.waitCall(t, taskKindGeneration, chat.ID)
_, token := call.input.TurnSpan.Ensure(ctx, call.input.TaskID, chat, chat.CreatedAt)
require.NotZero(t, token)
call.input.TurnSpan.Complete(token)
call.input.TurnSpan.Settle(token)
require.Empty(t, turnSpansByStart(t, recorder))
starter.release(t, 0)
testutil.Eventually(ctx, t, func(context.Context) bool {
return len(turnSpansByStart(t, recorder)) == 1
}, testutil.IntervalFast)
outcome, _ := spanAttribute(t, turnSpansByStart(t, recorder)[0], chatloop.AttrTurnOutcome)
require.Equal(t, chatloop.TurnOutcomeCompleted, chatloop.TurnOutcome(outcome.AsString()))
}