## Problem
The concurrent-agent cap (5 root chats and 10 subagent chats per
deployment by default) could be exceeded. Three paths put more chats in
`running` with a fresh lease than the pool allows:
1. **Interrupting a waiting chat.** A user sends a message with
`busy_behavior=interrupt` to a chat that is waiting for capacity. The
chat becomes `interrupting` while unowned, a worker acquires it without
admission, and `FinishInterruption` promotes the queued message into a
running turn under that ownership.
2. **Taking over an orphaned `requires_action` chat.** A replica dies
while its chat is in `requires_action`, its lease goes stale and another
chat takes the slot. A worker acquires the `requires_action` chat
without admission, and a tool result, an interrupt or the action
deadline returns it to `running` under that ownership.
3. **Heartbeat reviving a stale lease.** A replica loses the database
for longer than the 30 second stale threshold and another replica admits
a second chat into the slot. When the first replica comes back, its
heartbeat upsert renews the stale lease unconditionally, and its runner
keeps generating until the turn ends.
## Root cause
`agentCapacityLimiter.Admit` returned `true` without taking the
admission lock or counting for every status other than `running`. The
count query already treats every owned chat with a fresh lease as
holding a slot, so the shortcut let chats acquire a slot uncounted and
later move back to `running` under that ownership. Separately,
`BatchUpsertChatHeartbeats` had no freshness or ownership condition.
## Fix
- **Admission for every runnable status.** `Admit` now counts for
`running`, `interrupting` and `requires_action`, and a refused pool
skips the rest of its candidates in the pass regardless of status. An
owned chat holds its slot until it releases ownership, so every later
transition back to `running` stays within capacity.
- **Fenced heartbeat renewal.** `BatchUpsertChatHeartbeats` is replaced
by `RenewChatHeartbeats`, which only extends rows that are still fresh
and still own their chat, using `statement_timestamp()` read after the
capacity admission lock is granted. The renewal runs under that lock
with a `lock_timeout` (`HeartbeatLockTimeout`) and a per-tick timeout
(`HeartbeatRenewalTimeout`, a third of the stale threshold). Admission
also waits at most 5 seconds for the lock.
- **Stop runners that lost their lease.** `heartbeatOnce` requests
cleanup of every runner whose lease was not renewed, which cancels its
generation. The chat stays owned by the stale lease until an admitted
worker takes it over. A tick that fails or times out cleans up no
runners and retries on the next tick.
- **Instant Stop on a queued chat.** Requiring admission for
`interrupting` would have made Stop on a chat waiting for capacity hang
until a slot frees. When `Interrupt`, or `SendMessage` with
`busy_behavior=interrupt`, runs on a running chat with no owner, the
transition now applies `FinishInterruption` in the same transaction and
cancels orphaned tool calls. An empty queue lands in `waiting`;
otherwise the queue head is promoted into `running`, still unowned, so
it goes through admission.
- **Queued metric.** `CountChatCapacityQueuedByPool` (behind
`coderd_chatd_agents_queued_for_capacity`) now also counts
`interrupting` and `requires_action` chats without a fresh lease, since
they also wait for admission.
## How it was found
A TLA+ model of capacity admission found a counterexample for each path.
Go tests reproduced all three against the previous code (each failed
with `"2" is not less than or equal to "1"` for a root pool of capacity
1), and the model with the fix enabled passes the safety invariant and
the liveness properties. The model itself is not part of this PR.
## Tests
The three reproductions are kept as regression tests in
`coderd/x/chatd/agentcapacity_regression_internal_test.go`:
`TestAdmission_InterruptWithMessageOnWaitingChatStaysWithinCapacity`,
`TestAdmission_RequiresActionTakeoverStaysWithinCapacity` and
`TestRunnerManager_HeartbeatDoesNotReviveStaleLease`. The original
reproductions failed on the base commit. Restoring the admission
shortcut makes the `requires_action` test fail, and restoring the
unconditional heartbeat upsert, or dropping the runner cleanup, makes
the heartbeat test fail. New coverage also includes
`TestAdmission_StopWaitingChatNeedsNoCapacity`,
`TestRenewChatHeartbeats_OnlyRenewsFreshOwnedLeases`,
`TestRunnerManager_FailedHeartbeatTickKeepsRunners`, and chatstate tests
for the unowned interrupt paths. Existing tests that encoded the old
behavior (`requires_action` chats holding no slot, interrupting chats
always acquirable, unconditional heartbeat renewal) were updated.
## Behavior / API change
- `POST /api/v2/chats/{chat}/messages` with `busy_behavior=interrupt` on
a chat that no worker owns (for example one waiting for capacity) now
returns the promoted message with `queued=false` instead of a queued
message, and the chat stays in `running` instead of `interrupting`. If
the chat already had queued messages, the queue head is promoted and the
new message stays queued.
- `POST /api/v2/chats/{chat}/interrupt` on such a chat lands it in
`waiting` immediately (or in `running` with the queue head promoted)
instead of `interrupting`.
- An unowned `interrupting` or `requires_action` chat (for example after
its replica died) now waits for a free slot before its runner starts,
which also delays enforcing its action deadline while the pool is full.
- New internal chat worker settings `HeartbeatRenewalTimeout` and
`HeartbeatLockTimeout`, defaulting to a third of the stale threshold and
half of that.
- Heartbeat ticks now take the capacity admission advisory lock briefly
on each replica.
## Out of scope / follow-ups
- Interrupting a chat still owned by a dead runner (stale lease,
`worker_id` set) still waits for a slot.
- An admission lock timeout should end the acquisition pass instead of
logging a warning per candidate.
- The per-chat `queued_for_capacity` flag in the chat API still reports
only `running` chats.
- Alert on failed heartbeat ticks.
`coderd/x/chatd/ARCHITECTURE.md` gains TODO lines for the affected
sections; the prose is left for the owner.
---
_Generated with [`xum`](https://github.com/coder/xum) • Model:
`anthropic:claude-opus-5-5` • Thinking: `high`_
435 lines
14 KiB
Go
435 lines
14 KiB
Go
package chatd
|
|
|
|
import (
|
|
"context"
|
|
"database/sql"
|
|
"errors"
|
|
"testing"
|
|
"time"
|
|
|
|
"github.com/google/uuid"
|
|
"github.com/stretchr/testify/require"
|
|
|
|
"github.com/coder/coder/v2/coderd/database"
|
|
"github.com/coder/coder/v2/coderd/x/chatd/chatloop"
|
|
"github.com/coder/coder/v2/testutil"
|
|
"github.com/coder/quartz"
|
|
)
|
|
|
|
func TestWorker_NewRequiresTaskStarterOrMessagePartBuffer(t *testing.T) {
|
|
t.Parallel()
|
|
f := newWorkerTestFixture(t)
|
|
_, err := newChatWorker(newUnstartedServer(t, f.pubsub, f.db), chatWorkerOptions{WorkerID: uuid.New(), Store: f.db, Pubsub: f.pubsub})
|
|
require.ErrorContains(t, err, "task starter or message part buffer is required")
|
|
}
|
|
|
|
func TestWorker_NewRequiresServer(t *testing.T) {
|
|
t.Parallel()
|
|
f := newWorkerTestFixture(t)
|
|
_, err := newChatWorker(nil, testOptions(t, f, newRecordingTaskStarter()))
|
|
require.ErrorContains(t, err, "server is required")
|
|
}
|
|
|
|
func TestWorker_NewRequiresWorkerID(t *testing.T) {
|
|
t.Parallel()
|
|
f := newWorkerTestFixture(t)
|
|
opts := testOptions(t, f, newRecordingTaskStarter())
|
|
opts.WorkerID = uuid.Nil
|
|
_, err := newChatWorker(newUnstartedServer(t, f.pubsub, f.db), opts)
|
|
require.ErrorContains(t, err, "worker ID is required")
|
|
}
|
|
|
|
func TestWorker_UsesConfiguredWorkerID(t *testing.T) {
|
|
t.Parallel()
|
|
f := newWorkerTestFixture(t)
|
|
starter := newRecordingTaskStarter()
|
|
opts := testOptions(t, f, starter)
|
|
workerID := opts.WorkerID
|
|
worker, err := newChatWorker(newUnstartedServer(t, f.pubsub, f.db), opts)
|
|
require.NoError(t, err)
|
|
require.Equal(t, workerID, worker.opts.WorkerID)
|
|
require.NoError(t, worker.Start(context.Background()))
|
|
require.Equal(t, workerID, worker.opts.WorkerID)
|
|
require.NoError(t, worker.Close())
|
|
}
|
|
|
|
func TestWorker_AcquiresRunnableChatFromOwnershipHint(t *testing.T) {
|
|
t.Parallel()
|
|
f := newWorkerTestFixture(t)
|
|
chat := f.createRunningChat(t)
|
|
starter := newRecordingTaskStarter()
|
|
worker := startWorker(t, testOptions(t, f, starter))
|
|
|
|
call := starter.waitCall(t, taskKindGeneration, chat.ID)
|
|
require.Equal(t, worker.opts.WorkerID, call.input.WorkerID)
|
|
require.Equal(t, database.ChatStatusRunning, call.input.Status)
|
|
require.NotEqual(t, uuid.Nil, call.input.RunnerID)
|
|
|
|
latest, err := f.db.GetChatByID(testutil.Context(t, testutil.WaitShort), chat.ID)
|
|
require.NoError(t, err)
|
|
require.Equal(t, worker.opts.WorkerID, latest.WorkerID.UUID)
|
|
require.Equal(t, call.input.RunnerID, latest.RunnerID.UUID)
|
|
_, err = f.db.GetChatHeartbeat(testutil.Context(t, testutil.WaitShort), database.GetChatHeartbeatParams{
|
|
ChatID: chat.ID,
|
|
RunnerID: call.input.RunnerID,
|
|
})
|
|
require.NoError(t, err)
|
|
}
|
|
|
|
func TestWorker_AcquiresRequiresActionChatFromOwnershipHint(t *testing.T) {
|
|
t.Parallel()
|
|
f := newWorkerTestFixture(t)
|
|
chat := f.createRequiresActionChat(t)
|
|
starter := newRecordingTaskStarter()
|
|
startWorker(t, testOptions(t, f, starter))
|
|
|
|
call := starter.waitCall(t, taskKindRequiresActionTimeout, chat.ID)
|
|
require.Equal(t, database.ChatStatusRequiresAction, call.input.Status)
|
|
require.True(t, call.input.RequiresActionDeadlineAt.Valid)
|
|
}
|
|
|
|
func TestWorker_SkipsFreshlyOwnedChat(t *testing.T) {
|
|
t.Parallel()
|
|
f := newWorkerTestFixture(t)
|
|
chat := f.createRunningChat(t)
|
|
otherWorker := uuid.New()
|
|
otherRunner := uuid.New()
|
|
acquireChat(t, f, chat.ID, otherWorker, otherRunner)
|
|
starter := newRecordingTaskStarter()
|
|
worker := startWorker(t, testOptions(t, f, starter))
|
|
worker.Wake()
|
|
|
|
starter.assertNoCall(t)
|
|
latest, err := f.db.GetChatByID(testutil.Context(t, testutil.WaitShort), chat.ID)
|
|
require.NoError(t, err)
|
|
require.Equal(t, otherWorker, latest.WorkerID.UUID)
|
|
require.Equal(t, otherRunner, latest.RunnerID.UUID)
|
|
}
|
|
|
|
func TestWorker_ReacquiresStaleOwnedChat(t *testing.T) {
|
|
t.Parallel()
|
|
f := newWorkerTestFixture(t)
|
|
chat := f.createRunningChat(t)
|
|
deadWorker := uuid.New()
|
|
deadRunner := uuid.New()
|
|
acquireChat(t, f, chat.ID, deadWorker, deadRunner)
|
|
makeHeartbeatStale(t, f, chat.ID, deadRunner)
|
|
starter := newBlockingTaskStarter(false)
|
|
worker := startWorker(t, testOptions(t, f, starter))
|
|
|
|
call := starter.waitCall(t, taskKindGeneration, chat.ID)
|
|
require.Equal(t, worker.opts.WorkerID, call.input.WorkerID)
|
|
require.Equal(t, database.ChatStatusRunning, call.input.Status)
|
|
require.NotEqual(t, deadRunner, call.input.RunnerID)
|
|
|
|
latest, err := f.db.GetChatByID(testutil.Context(t, testutil.WaitShort), chat.ID)
|
|
require.NoError(t, err)
|
|
require.Equal(t, worker.opts.WorkerID, latest.WorkerID.UUID)
|
|
require.Equal(t, call.input.RunnerID, latest.RunnerID.UUID)
|
|
require.NotEqual(t, deadWorker, latest.WorkerID.UUID)
|
|
require.NotEqual(t, deadRunner, latest.RunnerID.UUID)
|
|
_, err = f.db.GetChatHeartbeat(testutil.Context(t, testutil.WaitShort), database.GetChatHeartbeatParams{
|
|
ChatID: chat.ID,
|
|
RunnerID: call.input.RunnerID,
|
|
})
|
|
require.NoError(t, err)
|
|
}
|
|
|
|
func TestWorker_TwoWorkersRaceSingleOwner(t *testing.T) {
|
|
t.Parallel()
|
|
f := newWorkerTestFixture(t)
|
|
chat := f.createRunningChat(t)
|
|
firstStarter := newRecordingTaskStarter()
|
|
secondStarter := newRecordingTaskStarter()
|
|
first := startWorker(t, testOptions(t, f, firstStarter))
|
|
second := startWorker(t, testOptions(t, f, secondStarter))
|
|
|
|
call := waitAnyTaskCall(t, firstStarter, secondStarter, taskKindGeneration, chat.ID)
|
|
require.Contains(t, []uuid.UUID{first.opts.WorkerID, second.opts.WorkerID}, call.input.WorkerID)
|
|
firstStarter.assertNoCall(t)
|
|
secondStarter.assertNoCall(t)
|
|
|
|
latest, err := f.db.GetChatByID(testutil.Context(t, testutil.WaitShort), chat.ID)
|
|
require.NoError(t, err)
|
|
require.True(t, latest.WorkerID.Valid)
|
|
require.True(t, latest.RunnerID.Valid)
|
|
require.Equal(t, call.input.WorkerID, latest.WorkerID.UUID)
|
|
require.Equal(t, call.input.RunnerID, latest.RunnerID.UUID)
|
|
}
|
|
|
|
func TestWorker_AcquisitionBatchSizeLimitsSuccessfulAcquisitions(t *testing.T) {
|
|
t.Parallel()
|
|
f := newWorkerTestFixture(t)
|
|
first := f.createRunningChat(t)
|
|
second := f.createRunningChat(t)
|
|
third := f.createRunningChat(t)
|
|
starter := newRecordingTaskStarter()
|
|
opts := testOptions(t, f, starter)
|
|
opts.AcquisitionBatchSize = 1
|
|
worker := startWorker(t, opts)
|
|
|
|
want := map[uuid.UUID]bool{first.ID: true, second.ID: true, third.ID: true}
|
|
for range 3 {
|
|
call := starter.waitCall(t, taskKindGeneration, uuid.Nil)
|
|
delete(want, call.input.ChatID)
|
|
starter.assertNoCall(t)
|
|
worker.Wake()
|
|
}
|
|
require.Empty(t, want)
|
|
}
|
|
|
|
func TestWorker_DoesNotAcquireIdleOrArchivedChats(t *testing.T) {
|
|
t.Parallel()
|
|
f := newWorkerTestFixture(t)
|
|
waiting := f.createRunningChat(t)
|
|
finishTurn(t, f, waiting.ID)
|
|
errorChat := f.createRunningChat(t)
|
|
forceExecutionStateAndPublish(t, f, errorChat.ID, database.ChatStatusError, false)
|
|
archived := f.createRunningChat(t)
|
|
forceExecutionStateAndPublish(t, f, archived.ID, database.ChatStatusRunning, true)
|
|
starter := newRecordingTaskStarter()
|
|
worker := startWorker(t, testOptions(t, f, starter))
|
|
worker.Wake()
|
|
|
|
starter.assertNoCall(t)
|
|
}
|
|
|
|
func TestWorker_HeartbeatLoopRefreshesActiveRunnerHeartbeat(t *testing.T) {
|
|
t.Parallel()
|
|
f := newWorkerTestFixture(t)
|
|
chat := f.createRunningChat(t)
|
|
clock := quartz.NewMock(t)
|
|
heartbeatTrap := clock.Trap().NewTicker("chatworker", "heartbeat")
|
|
defer heartbeatTrap.Close()
|
|
starter := newBlockingTaskStarter(false)
|
|
opts := testOptions(t, f, starter)
|
|
opts.Clock = clock
|
|
opts.HeartbeatInterval = time.Minute
|
|
startWorker(t, opts)
|
|
heartbeatTrap.MustWait(testutil.Context(t, testutil.WaitLong)).MustRelease(testutil.Context(t, testutil.WaitLong))
|
|
call := starter.waitCall(t, taskKindGeneration, chat.ID)
|
|
// Renewal only extends a lease that is still fresh.
|
|
setHeartbeatAge(t, f, chat.ID, call.input.RunnerID, 10*time.Second)
|
|
aged, err := f.db.GetChatHeartbeat(testutil.Context(t, testutil.WaitShort), database.GetChatHeartbeatParams{
|
|
ChatID: chat.ID,
|
|
RunnerID: call.input.RunnerID,
|
|
})
|
|
require.NoError(t, err)
|
|
oldHeartbeat := aged.HeartbeatAt
|
|
|
|
clock.Advance(time.Minute).MustWait(testutil.Context(t, testutil.WaitLong))
|
|
testutil.Eventually(testutil.Context(t, testutil.WaitLong), t, func(ctx context.Context) bool {
|
|
heartbeat, err := f.db.GetChatHeartbeat(ctx, database.GetChatHeartbeatParams{
|
|
ChatID: chat.ID,
|
|
RunnerID: call.input.RunnerID,
|
|
})
|
|
return err == nil && heartbeat.HeartbeatAt.After(oldHeartbeat)
|
|
}, testutil.IntervalFast, "heartbeat should be refreshed")
|
|
}
|
|
|
|
func TestWorker_HeartbeatCleanupDeletesStaleRows(t *testing.T) {
|
|
t.Parallel()
|
|
f := newWorkerTestFixture(t)
|
|
chat := f.createRunningChat(t)
|
|
finishTurn(t, f, chat.ID)
|
|
runnerID := uuid.New()
|
|
require.NoError(t, f.db.UpsertChatHeartbeat(testutil.Context(t, testutil.WaitShort), database.UpsertChatHeartbeatParams{
|
|
ChatID: chat.ID,
|
|
RunnerID: runnerID,
|
|
}))
|
|
makeHeartbeatStale(t, f, chat.ID, runnerID)
|
|
clock := quartz.NewMock(t)
|
|
cleanupTrap := clock.Trap().NewTicker("chatworker", "heartbeat-cleanup")
|
|
defer cleanupTrap.Close()
|
|
starter := newRecordingTaskStarter()
|
|
opts := testOptions(t, f, starter)
|
|
opts.Clock = clock
|
|
opts.HeartbeatCleanupInterval = time.Minute
|
|
startWorker(t, opts)
|
|
cleanupTrap.MustWait(testutil.Context(t, testutil.WaitLong)).MustRelease(testutil.Context(t, testutil.WaitLong))
|
|
|
|
clock.Advance(time.Minute).MustWait(testutil.Context(t, testutil.WaitLong))
|
|
testutil.Eventually(testutil.Context(t, testutil.WaitLong), t, func(ctx context.Context) bool {
|
|
_, err := f.db.GetChatHeartbeat(ctx, database.GetChatHeartbeatParams{
|
|
ChatID: chat.ID,
|
|
RunnerID: runnerID,
|
|
})
|
|
return errors.Is(err, sql.ErrNoRows)
|
|
}, testutil.IntervalFast)
|
|
}
|
|
|
|
func TestWorker_CloseDeletesOwnedHeartbeatsAndPublishesOwnershipHints(t *testing.T) {
|
|
t.Parallel()
|
|
f := newWorkerTestFixture(t)
|
|
first := f.createRunningChat(t)
|
|
second := f.createRunningChat(t)
|
|
starter := newBlockingTaskStarter(false)
|
|
pubsub := newRecordingPubsub(f.pubsub)
|
|
opts := testOptions(t, f, starter)
|
|
opts.Pubsub = pubsub
|
|
worker := startWorker(t, opts)
|
|
callsByChat := make(map[uuid.UUID]taskCall)
|
|
for range 2 {
|
|
call := starter.waitCall(t, taskKindGeneration, uuid.Nil)
|
|
callsByChat[call.input.ChatID] = call
|
|
}
|
|
require.Contains(t, callsByChat, first.ID)
|
|
require.Contains(t, callsByChat, second.ID)
|
|
|
|
require.NoError(t, worker.Close())
|
|
for _, call := range callsByChat {
|
|
_, err := f.db.GetChatHeartbeat(testutil.Context(t, testutil.WaitShort), database.GetChatHeartbeatParams{
|
|
ChatID: call.input.ChatID,
|
|
RunnerID: call.input.RunnerID,
|
|
})
|
|
require.ErrorIs(t, err, sql.ErrNoRows)
|
|
}
|
|
|
|
messages := pubsub.ownershipMessages(t)
|
|
seen := make(map[uuid.UUID]bool)
|
|
for _, msg := range messages {
|
|
seen[msg.ChatID] = true
|
|
require.NotZero(t, msg.SnapshotVersion)
|
|
}
|
|
require.True(t, seen[first.ID], "expected ownership hint for first runner")
|
|
require.True(t, seen[second.ID], "expected ownership hint for second runner")
|
|
}
|
|
|
|
func TestWorker_CloseIsIdempotentAndDoesNotBlock(t *testing.T) {
|
|
t.Parallel()
|
|
f := newWorkerTestFixture(t)
|
|
chat := f.createRunningChat(t)
|
|
starter := newBlockingTaskStarter(false)
|
|
worker := startWorker(t, testOptions(t, f, starter))
|
|
call := starter.waitCall(t, taskKindGeneration, chat.ID)
|
|
|
|
closed := make(chan error, 1)
|
|
go func() {
|
|
if err := worker.Close(); err != nil {
|
|
closed <- err
|
|
return
|
|
}
|
|
closed <- worker.Close()
|
|
}()
|
|
select {
|
|
case err := <-closed:
|
|
require.NoError(t, err)
|
|
case <-time.After(testutil.WaitLong):
|
|
t.Fatal("worker close did not return")
|
|
}
|
|
select {
|
|
case <-call.ctx.Done():
|
|
case <-time.After(testutil.WaitLong):
|
|
t.Fatal("active task was not canceled")
|
|
}
|
|
}
|
|
|
|
func waitAnyTaskCall(
|
|
t *testing.T,
|
|
first *recordingTaskStarter,
|
|
second *recordingTaskStarter,
|
|
kind taskKind,
|
|
chatID uuid.UUID,
|
|
) taskCall {
|
|
t.Helper()
|
|
deadline := time.After(testutil.WaitLong)
|
|
for {
|
|
select {
|
|
case call := <-first.callCh:
|
|
if call.kind == kind && call.input.ChatID == chatID {
|
|
return call
|
|
}
|
|
case call := <-second.callCh:
|
|
if call.kind == kind && call.input.ChatID == chatID {
|
|
return call
|
|
}
|
|
case <-deadline:
|
|
t.Fatal("timed out waiting for either worker to start task")
|
|
return taskCall{}
|
|
}
|
|
}
|
|
}
|
|
|
|
func requireTaskCanceled(t *testing.T, call taskCall) {
|
|
t.Helper()
|
|
select {
|
|
case <-call.ctx.Done():
|
|
require.True(t, errors.Is(call.ctx.Err(), context.Canceled))
|
|
case <-time.After(testutil.WaitLong):
|
|
t.Fatal("task context was not canceled")
|
|
}
|
|
}
|
|
|
|
func TestWorkerRunnerTurnSpan(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
tests := []struct {
|
|
name string
|
|
staleOwner bool
|
|
wantAcquisitions int
|
|
}{
|
|
{name: "Unowned", wantAcquisitions: 1},
|
|
{name: "StaleOwner", staleOwner: true},
|
|
}
|
|
for _, tt := range tests {
|
|
t.Run(tt.name, func(t *testing.T) {
|
|
t.Parallel()
|
|
ctx := testutil.Context(t, testutil.WaitLong)
|
|
f := newWorkerTestFixture(t)
|
|
chat := f.createRunningChat(t)
|
|
if tt.staleOwner {
|
|
deadRunner := uuid.New()
|
|
acquireChat(t, f, chat.ID, uuid.New(), deadRunner)
|
|
makeHeartbeatStale(t, f, chat.ID, deadRunner)
|
|
}
|
|
tracer, recorder := newStageTestTracer(t)
|
|
server := newUnstartedServer(t, f.pubsub, f.db)
|
|
server.stages = tracer
|
|
starter := newBlockingTaskStarter(false)
|
|
worker, err := newChatWorker(server, testOptions(t, f, starter))
|
|
require.NoError(t, err)
|
|
require.NoError(t, worker.Start(context.Background()))
|
|
|
|
call := starter.waitCall(t, taskKindGeneration, chat.ID)
|
|
require.NotNil(t, call.input.TurnSpan)
|
|
call.input.TurnSpan.Ensure(ctx, call.input.TaskID, chat, chat.CreatedAt)
|
|
require.NoError(t, worker.Close())
|
|
|
|
turns := turnSpansByStart(t, recorder)
|
|
require.Len(t, turns, 1)
|
|
outcome, _ := spanAttribute(t, turns[0], chatloop.AttrTurnOutcome)
|
|
require.Equal(t, chatloop.TurnOutcomeAbandoned, chatloop.TurnOutcome(outcome.AsString()))
|
|
require.Len(t, stageSpansByStart(t, recorder, chatloop.StageAcquisition), tt.wantAcquisitions)
|
|
})
|
|
}
|
|
}
|
|
|
|
func TestWorkerRunnerReleasesTurnOnTaskExit(t *testing.T) {
|
|
t.Parallel()
|
|
ctx := testutil.Context(t, testutil.WaitLong)
|
|
f := newWorkerTestFixture(t)
|
|
chat := f.createRunningChat(t)
|
|
tracer, recorder := newStageTestTracer(t)
|
|
server := newUnstartedServer(t, f.pubsub, f.db)
|
|
server.stages = tracer
|
|
starter := newBlockingTaskStarter(false)
|
|
worker, err := newChatWorker(server, testOptions(t, f, starter))
|
|
require.NoError(t, err)
|
|
require.NoError(t, worker.Start(context.Background()))
|
|
t.Cleanup(func() { _ = worker.Close() })
|
|
|
|
call := starter.waitCall(t, taskKindGeneration, chat.ID)
|
|
_, token := call.input.TurnSpan.Ensure(ctx, call.input.TaskID, chat, chat.CreatedAt)
|
|
require.NotZero(t, token)
|
|
call.input.TurnSpan.Complete(token)
|
|
call.input.TurnSpan.Settle(token)
|
|
require.Empty(t, turnSpansByStart(t, recorder))
|
|
|
|
starter.release(t, 0)
|
|
testutil.Eventually(ctx, t, func(context.Context) bool {
|
|
return len(turnSpansByStart(t, recorder)) == 1
|
|
}, testutil.IntervalFast)
|
|
outcome, _ := spanAttribute(t, turnSpansByStart(t, recorder)[0], chatloop.AttrTurnOutcome)
|
|
require.Equal(t, chatloop.TurnOutcomeCompleted, chatloop.TurnOutcome(outcome.AsString()))
|
|
}
|