## Problem
The concurrent-agent cap (5 root chats and 10 subagent chats per
deployment by default) could be exceeded. Three paths put more chats in
`running` with a fresh lease than the pool allows:
1. **Interrupting a waiting chat.** A user sends a message with
`busy_behavior=interrupt` to a chat that is waiting for capacity. The
chat becomes `interrupting` while unowned, a worker acquires it without
admission, and `FinishInterruption` promotes the queued message into a
running turn under that ownership.
2. **Taking over an orphaned `requires_action` chat.** A replica dies
while its chat is in `requires_action`, its lease goes stale and another
chat takes the slot. A worker acquires the `requires_action` chat
without admission, and a tool result, an interrupt or the action
deadline returns it to `running` under that ownership.
3. **Heartbeat reviving a stale lease.** A replica loses the database
for longer than the 30 second stale threshold and another replica admits
a second chat into the slot. When the first replica comes back, its
heartbeat upsert renews the stale lease unconditionally, and its runner
keeps generating until the turn ends.
## Root cause
`agentCapacityLimiter.Admit` returned `true` without taking the
admission lock or counting for every status other than `running`. The
count query already treats every owned chat with a fresh lease as
holding a slot, so the shortcut let chats acquire a slot uncounted and
later move back to `running` under that ownership. Separately,
`BatchUpsertChatHeartbeats` had no freshness or ownership condition.
## Fix
- **Admission for every runnable status.** `Admit` now counts for
`running`, `interrupting` and `requires_action`, and a refused pool
skips the rest of its candidates in the pass regardless of status. An
owned chat holds its slot until it releases ownership, so every later
transition back to `running` stays within capacity.
- **Fenced heartbeat renewal.** `BatchUpsertChatHeartbeats` is replaced
by `RenewChatHeartbeats`, which only extends rows that are still fresh
and still own their chat, using `statement_timestamp()` read after the
capacity admission lock is granted. The renewal runs under that lock
with a `lock_timeout` (`HeartbeatLockTimeout`) and a per-tick timeout
(`HeartbeatRenewalTimeout`, a third of the stale threshold). Admission
also waits at most 5 seconds for the lock.
- **Stop runners that lost their lease.** `heartbeatOnce` requests
cleanup of every runner whose lease was not renewed, which cancels its
generation. The chat stays owned by the stale lease until an admitted
worker takes it over. A tick that fails or times out cleans up no
runners and retries on the next tick.
- **Instant Stop on a queued chat.** Requiring admission for
`interrupting` would have made Stop on a chat waiting for capacity hang
until a slot frees. When `Interrupt`, or `SendMessage` with
`busy_behavior=interrupt`, runs on a running chat with no owner, the
transition now applies `FinishInterruption` in the same transaction and
cancels orphaned tool calls. An empty queue lands in `waiting`;
otherwise the queue head is promoted into `running`, still unowned, so
it goes through admission.
- **Queued metric.** `CountChatCapacityQueuedByPool` (behind
`coderd_chatd_agents_queued_for_capacity`) now also counts
`interrupting` and `requires_action` chats without a fresh lease, since
they also wait for admission.
## How it was found
A TLA+ model of capacity admission found a counterexample for each path.
Go tests reproduced all three against the previous code (each failed
with `"2" is not less than or equal to "1"` for a root pool of capacity
1), and the model with the fix enabled passes the safety invariant and
the liveness properties. The model itself is not part of this PR.
## Tests
The three reproductions are kept as regression tests in
`coderd/x/chatd/agentcapacity_regression_internal_test.go`:
`TestAdmission_InterruptWithMessageOnWaitingChatStaysWithinCapacity`,
`TestAdmission_RequiresActionTakeoverStaysWithinCapacity` and
`TestRunnerManager_HeartbeatDoesNotReviveStaleLease`. The original
reproductions failed on the base commit. Restoring the admission
shortcut makes the `requires_action` test fail, and restoring the
unconditional heartbeat upsert, or dropping the runner cleanup, makes
the heartbeat test fail. New coverage also includes
`TestAdmission_StopWaitingChatNeedsNoCapacity`,
`TestRenewChatHeartbeats_OnlyRenewsFreshOwnedLeases`,
`TestRunnerManager_FailedHeartbeatTickKeepsRunners`, and chatstate tests
for the unowned interrupt paths. Existing tests that encoded the old
behavior (`requires_action` chats holding no slot, interrupting chats
always acquirable, unconditional heartbeat renewal) were updated.
## Behavior / API change
- `POST /api/v2/chats/{chat}/messages` with `busy_behavior=interrupt` on
a chat that no worker owns (for example one waiting for capacity) now
returns the promoted message with `queued=false` instead of a queued
message, and the chat stays in `running` instead of `interrupting`. If
the chat already had queued messages, the queue head is promoted and the
new message stays queued.
- `POST /api/v2/chats/{chat}/interrupt` on such a chat lands it in
`waiting` immediately (or in `running` with the queue head promoted)
instead of `interrupting`.
- An unowned `interrupting` or `requires_action` chat (for example after
its replica died) now waits for a free slot before its runner starts,
which also delays enforcing its action deadline while the pool is full.
- New internal chat worker settings `HeartbeatRenewalTimeout` and
`HeartbeatLockTimeout`, defaulting to a third of the stale threshold and
half of that.
- Heartbeat ticks now take the capacity admission advisory lock briefly
on each replica.
## Out of scope / follow-ups
- Interrupting a chat still owned by a dead runner (stale lease,
`worker_id` set) still waits for a slot.
- An admission lock timeout should end the acquisition pass instead of
logging a warning per candidate.
- The per-chat `queued_for_capacity` flag in the chat API still reports
only `running` chats.
- Alert on failed heartbeat ticks.
`coderd/x/chatd/ARCHITECTURE.md` gains TODO lines for the affected
sections; the prose is left for the owner.
---
_Generated with [`xum`](https://github.com/coder/xum) • Model:
`anthropic:claude-opus-5-5` • Thinking: `high`_
586 lines
15 KiB
Go
586 lines
15 KiB
Go
package chatd
|
|
|
|
import (
|
|
"context"
|
|
"database/sql"
|
|
"encoding/json"
|
|
"sync"
|
|
"time"
|
|
|
|
"github.com/google/uuid"
|
|
"golang.org/x/xerrors"
|
|
|
|
"cdr.dev/slog/v3"
|
|
"github.com/coder/coder/v2/coderd/database"
|
|
coderdpubsub "github.com/coder/coder/v2/coderd/pubsub"
|
|
)
|
|
|
|
const shutdownCleanupTimeout = 5 * time.Second
|
|
|
|
type runnerKey struct {
|
|
ChatID uuid.UUID
|
|
RunnerID uuid.UUID
|
|
}
|
|
|
|
type runnerStateUpdate struct {
|
|
ChatID uuid.UUID
|
|
WorkerID *uuid.UUID
|
|
RunnerID *uuid.UUID
|
|
SnapshotVersion int64
|
|
HistoryVersion int64
|
|
QueueVersion int64
|
|
GenerationAttempt int64
|
|
Status database.ChatStatus
|
|
Archived bool
|
|
RequiresActionDeadlineAt sql.NullTime
|
|
}
|
|
|
|
type spawnRunnerRequest struct {
|
|
ChatID uuid.UUID
|
|
WorkerID uuid.UUID
|
|
RunnerID uuid.UUID
|
|
// TakenOver means the previous owner's heartbeat went stale.
|
|
TakenOver bool
|
|
}
|
|
|
|
type runnerRecord struct {
|
|
key runnerKey
|
|
workerID uuid.UUID
|
|
takenOver bool
|
|
cancel context.CancelFunc
|
|
done <-chan struct{}
|
|
stateCh chan runnerStateUpdate
|
|
|
|
mu sync.Mutex
|
|
unsubscribe func()
|
|
cleanupStarted bool
|
|
}
|
|
|
|
func (r *runnerRecord) setUnsubscribe(unsubscribe func()) bool {
|
|
r.mu.Lock()
|
|
if r.cleanupStarted {
|
|
r.mu.Unlock()
|
|
if unsubscribe != nil {
|
|
unsubscribe()
|
|
}
|
|
return false
|
|
}
|
|
r.unsubscribe = unsubscribe
|
|
r.mu.Unlock()
|
|
return true
|
|
}
|
|
|
|
func (r *runnerRecord) startCleanup() {
|
|
r.mu.Lock()
|
|
if r.cleanupStarted {
|
|
r.mu.Unlock()
|
|
return
|
|
}
|
|
r.cleanupStarted = true
|
|
unsubscribe := r.unsubscribe
|
|
r.unsubscribe = nil
|
|
r.mu.Unlock()
|
|
if unsubscribe != nil {
|
|
unsubscribe()
|
|
}
|
|
r.cancel()
|
|
}
|
|
|
|
type runnerManager struct {
|
|
server *Server
|
|
opts chatWorkerOptions
|
|
ctx context.Context
|
|
|
|
closed bool
|
|
spawnMu sync.Mutex
|
|
|
|
mu sync.Mutex
|
|
spawnCh chan spawnRunnerRequest
|
|
cleanupReqCh chan runnerKey
|
|
cleanupDoneCh chan runnerKey
|
|
runners map[runnerKey]*runnerRecord
|
|
runnersByChat map[uuid.UUID]map[uuid.UUID]*runnerRecord
|
|
cleaning map[runnerKey]*runnerRecord
|
|
|
|
wg sync.WaitGroup
|
|
}
|
|
|
|
func newRunnerManager(ctx context.Context, server *Server, opts chatWorkerOptions) *runnerManager {
|
|
return &runnerManager{
|
|
server: server,
|
|
opts: opts,
|
|
ctx: ctx,
|
|
spawnCh: make(chan spawnRunnerRequest, opts.RunnerManagerChannelSize),
|
|
cleanupReqCh: make(chan runnerKey, opts.RunnerManagerChannelSize),
|
|
cleanupDoneCh: make(chan runnerKey, opts.RunnerManagerChannelSize),
|
|
runners: make(map[runnerKey]*runnerRecord),
|
|
runnersByChat: make(map[uuid.UUID]map[uuid.UUID]*runnerRecord),
|
|
cleaning: make(map[runnerKey]*runnerRecord),
|
|
}
|
|
}
|
|
|
|
func (m *runnerManager) start() {
|
|
m.wg.Go(m.run)
|
|
m.wg.Go(m.databaseSyncLoop)
|
|
m.wg.Go(m.heartbeatLoop)
|
|
m.wg.Go(m.heartbeatCleanupLoop)
|
|
}
|
|
|
|
func (m *runnerManager) wait() {
|
|
m.wg.Wait()
|
|
}
|
|
|
|
func (m *runnerManager) idle() bool {
|
|
m.mu.Lock()
|
|
defer m.mu.Unlock()
|
|
return len(m.runners) == 0 && len(m.cleaning) == 0
|
|
}
|
|
|
|
func (m *runnerManager) Spawn(ctx context.Context, req spawnRunnerRequest) error {
|
|
m.spawnMu.Lock()
|
|
defer m.spawnMu.Unlock()
|
|
if m.closed {
|
|
return xerrors.New("chatworker: runner manager closed")
|
|
}
|
|
|
|
select {
|
|
case m.spawnCh <- req:
|
|
return nil
|
|
case <-ctx.Done():
|
|
return ctx.Err()
|
|
case <-m.ctx.Done():
|
|
return m.ctx.Err()
|
|
}
|
|
}
|
|
|
|
func (m *runnerManager) requestCleanup(ctx context.Context, key runnerKey) {
|
|
select {
|
|
case m.cleanupReqCh <- key:
|
|
case <-ctx.Done():
|
|
case <-m.ctx.Done():
|
|
}
|
|
}
|
|
|
|
func (m *runnerManager) RouteStateHint(ctx context.Context, state runnerStateUpdate) {
|
|
m.mu.Lock()
|
|
byRunner := m.runnersByChat[state.ChatID]
|
|
targets := make([]*runnerRecord, 0, len(byRunner))
|
|
for _, rec := range byRunner {
|
|
targets = append(targets, rec)
|
|
}
|
|
m.mu.Unlock()
|
|
|
|
for _, rec := range targets {
|
|
select {
|
|
case rec.stateCh <- state:
|
|
case <-rec.done:
|
|
// Only this runner exited; keep fanning out to the rest.
|
|
continue
|
|
case <-ctx.Done():
|
|
return
|
|
case <-m.ctx.Done():
|
|
return
|
|
default:
|
|
// stateCh is full; drop the hint for this runner.
|
|
}
|
|
}
|
|
}
|
|
|
|
func (m *runnerManager) run() {
|
|
for {
|
|
select {
|
|
case req := <-m.spawnCh:
|
|
m.handleSpawn(req)
|
|
case key := <-m.cleanupReqCh:
|
|
m.handleCleanupRequest(key)
|
|
case key := <-m.cleanupDoneCh:
|
|
m.handleCleanupDone(key)
|
|
case <-m.ctx.Done():
|
|
queued := m.closeAndDrainQueues()
|
|
m.cancelAll()
|
|
m.releaseOwnershipHints(queued)
|
|
return
|
|
}
|
|
}
|
|
}
|
|
|
|
func (m *runnerManager) handleSpawn(req spawnRunnerRequest) {
|
|
key := runnerKey{ChatID: req.ChatID, RunnerID: req.RunnerID}
|
|
m.mu.Lock()
|
|
if _, ok := m.runners[key]; ok {
|
|
// A duplicate spawn for a live runner indicates a logic error
|
|
// in the sync loop.
|
|
m.opts.Logger.Error(m.ctx, "invalid spawn request: chat runner already spawned", slog.F("key", key))
|
|
m.mu.Unlock()
|
|
return
|
|
}
|
|
if _, ok := m.cleaning[key]; ok {
|
|
// A duplicate spawn for a live runner indicates a logic error
|
|
// in the sync loop.
|
|
m.opts.Logger.Error(m.ctx, "invalid spawn request: chat runner in cleanup", slog.F("key", key))
|
|
m.mu.Unlock()
|
|
return
|
|
}
|
|
runnerCtx, cancel := context.WithCancel(m.ctx)
|
|
done := make(chan struct{})
|
|
rec := &runnerRecord{
|
|
key: key,
|
|
workerID: req.WorkerID,
|
|
takenOver: req.TakenOver,
|
|
cancel: cancel,
|
|
done: done,
|
|
stateCh: make(chan runnerStateUpdate, m.opts.StateChannelSize),
|
|
}
|
|
m.runners[key] = rec
|
|
if m.runnersByChat[req.ChatID] == nil {
|
|
m.runnersByChat[req.ChatID] = make(map[uuid.UUID]*runnerRecord)
|
|
}
|
|
m.runnersByChat[req.ChatID][req.RunnerID] = rec
|
|
m.mu.Unlock()
|
|
|
|
r := newRunner(runnerCtx, m, rec, m.opts)
|
|
m.wg.Go(func() {
|
|
defer close(done)
|
|
r.run()
|
|
})
|
|
}
|
|
|
|
func (m *runnerManager) closeAndDrainQueues() []runnerKey {
|
|
m.spawnMu.Lock()
|
|
defer m.spawnMu.Unlock()
|
|
|
|
m.closed = true
|
|
return m.drainQueues()
|
|
}
|
|
|
|
func (m *runnerManager) drainQueues() []runnerKey {
|
|
queued := make([]runnerKey, 0)
|
|
for {
|
|
select {
|
|
case req := <-m.spawnCh:
|
|
queued = append(queued, runnerKey{ChatID: req.ChatID, RunnerID: req.RunnerID})
|
|
case key := <-m.cleanupReqCh:
|
|
m.handleCleanupRequest(key)
|
|
case key := <-m.cleanupDoneCh:
|
|
m.handleCleanupDone(key)
|
|
default:
|
|
return queued
|
|
}
|
|
}
|
|
}
|
|
|
|
func (m *runnerManager) handleCleanupRequest(key runnerKey) {
|
|
m.mu.Lock()
|
|
rec, ok := m.runners[key]
|
|
if !ok {
|
|
m.mu.Unlock()
|
|
return
|
|
}
|
|
delete(m.runners, key)
|
|
if byChat := m.runnersByChat[key.ChatID]; byChat != nil {
|
|
delete(byChat, key.RunnerID)
|
|
if len(byChat) == 0 {
|
|
delete(m.runnersByChat, key.ChatID)
|
|
}
|
|
}
|
|
m.cleaning[key] = rec
|
|
m.mu.Unlock()
|
|
|
|
rec.startCleanup()
|
|
m.registerCleanupWaiter(key, rec)
|
|
}
|
|
|
|
func (m *runnerManager) registerCleanupWaiter(key runnerKey, rec *runnerRecord) {
|
|
m.wg.Go(func() {
|
|
<-rec.done
|
|
if m.ctx.Err() != nil {
|
|
m.mu.Lock()
|
|
delete(m.cleaning, key)
|
|
m.mu.Unlock()
|
|
return
|
|
}
|
|
select {
|
|
case m.cleanupDoneCh <- key:
|
|
case <-m.ctx.Done():
|
|
m.mu.Lock()
|
|
delete(m.cleaning, key)
|
|
m.mu.Unlock()
|
|
}
|
|
})
|
|
}
|
|
|
|
func (m *runnerManager) handleCleanupDone(key runnerKey) {
|
|
m.mu.Lock()
|
|
delete(m.cleaning, key)
|
|
m.mu.Unlock()
|
|
}
|
|
|
|
func (m *runnerManager) cancelAll() {
|
|
type cleanupTarget struct {
|
|
key runnerKey
|
|
rec *runnerRecord
|
|
}
|
|
|
|
m.mu.Lock()
|
|
active := make([]cleanupTarget, 0, len(m.runners))
|
|
cleaning := make([]*runnerRecord, 0, len(m.cleaning))
|
|
for _, rec := range m.cleaning {
|
|
cleaning = append(cleaning, rec)
|
|
}
|
|
for key, rec := range m.runners {
|
|
delete(m.runners, key)
|
|
m.cleaning[key] = rec
|
|
active = append(active, cleanupTarget{key: key, rec: rec})
|
|
}
|
|
clear(m.runnersByChat)
|
|
m.mu.Unlock()
|
|
|
|
keys := make([]runnerKey, 0, len(cleaning)+len(active))
|
|
for _, rec := range cleaning {
|
|
rec.startCleanup()
|
|
keys = append(keys, rec.key)
|
|
}
|
|
for _, target := range active {
|
|
target.rec.startCleanup()
|
|
m.registerCleanupWaiter(target.key, target.rec)
|
|
keys = append(keys, target.key)
|
|
}
|
|
m.releaseOwnershipHints(keys)
|
|
}
|
|
|
|
func (m *runnerManager) releaseOwnershipHints(keys []runnerKey) {
|
|
if len(keys) == 0 {
|
|
return
|
|
}
|
|
ctx, cancel := context.WithTimeout(context.WithoutCancel(m.ctx), shutdownCleanupTimeout)
|
|
defer cancel()
|
|
|
|
chatIDs := make([]uuid.UUID, 0, len(keys))
|
|
runnerIDs := make([]uuid.UUID, 0, len(keys))
|
|
uniqueChatIDs := make(map[uuid.UUID]struct{}, len(keys))
|
|
for _, key := range keys {
|
|
chatIDs = append(chatIDs, key.ChatID)
|
|
runnerIDs = append(runnerIDs, key.RunnerID)
|
|
uniqueChatIDs[key.ChatID] = struct{}{}
|
|
}
|
|
if _, err := m.opts.Store.BatchDeleteChatHeartbeats(ctx, database.BatchDeleteChatHeartbeatsParams{
|
|
ChatIds: chatIDs,
|
|
RunnerIds: runnerIDs,
|
|
}); err != nil {
|
|
m.opts.Logger.Warn(ctx, "chatworker shutdown heartbeat cleanup failed", slogError(err))
|
|
}
|
|
|
|
syncIDs := make([]uuid.UUID, 0, len(uniqueChatIDs))
|
|
for id := range uniqueChatIDs {
|
|
syncIDs = append(syncIDs, id)
|
|
}
|
|
chats, err := m.opts.Store.GetChatsByIDsForRunnerSync(ctx, syncIDs)
|
|
if err != nil {
|
|
m.opts.Logger.Warn(ctx, "chatworker shutdown ownership lookup failed", slogError(err))
|
|
}
|
|
snapshotByChat := make(map[uuid.UUID]int64, len(chats))
|
|
for _, chat := range chats {
|
|
snapshotByChat[chat.ID] = chat.SnapshotVersion
|
|
}
|
|
for _, key := range keys {
|
|
payload, err := json.Marshal(coderdpubsub.ChatStateOwnershipMessage{
|
|
ChatID: key.ChatID,
|
|
SnapshotVersion: snapshotByChat[key.ChatID],
|
|
})
|
|
if err != nil {
|
|
m.opts.Logger.Warn(ctx, "chatworker shutdown ownership marshal failed", slogError(err))
|
|
continue
|
|
}
|
|
if err := m.opts.Pubsub.Publish(coderdpubsub.ChatStateOwnershipChannel, payload); err != nil {
|
|
m.opts.Logger.Warn(ctx, "chatworker shutdown ownership publish failed", slogError(err))
|
|
}
|
|
}
|
|
}
|
|
|
|
func (m *runnerManager) snapshotRunnerKeys() []runnerKey {
|
|
m.mu.Lock()
|
|
defer m.mu.Unlock()
|
|
keys := make([]runnerKey, 0, len(m.runners))
|
|
for key := range m.runners {
|
|
keys = append(keys, key)
|
|
}
|
|
return keys
|
|
}
|
|
|
|
func (m *runnerManager) databaseSyncLoop() {
|
|
ticker := m.opts.Clock.NewTicker(m.opts.RunnerSyncInterval, "chatworker", "runner-sync")
|
|
defer ticker.Stop()
|
|
for {
|
|
select {
|
|
case <-ticker.C:
|
|
if err := m.syncOnce(m.ctx); err != nil {
|
|
m.opts.Logger.Warn(m.ctx, "chatworker runner sync failed", slogError(err))
|
|
}
|
|
case <-m.ctx.Done():
|
|
return
|
|
}
|
|
}
|
|
}
|
|
|
|
func (m *runnerManager) syncOnce(ctx context.Context) error {
|
|
keys := m.snapshotRunnerKeys()
|
|
if len(keys) == 0 {
|
|
return nil
|
|
}
|
|
idsByChat := make(map[uuid.UUID]struct{}, len(keys))
|
|
for _, key := range keys {
|
|
idsByChat[key.ChatID] = struct{}{}
|
|
}
|
|
chatIDs := make([]uuid.UUID, 0, len(idsByChat))
|
|
for id := range idsByChat {
|
|
chatIDs = append(chatIDs, id)
|
|
}
|
|
chats, err := m.opts.Store.GetChatsByIDsForRunnerSync(ctx, chatIDs)
|
|
if err != nil {
|
|
return xerrors.Errorf("get chats for runner sync: %w", err)
|
|
}
|
|
seen := make(map[uuid.UUID]struct{}, len(chats))
|
|
for _, chat := range chats {
|
|
seen[chat.ID] = struct{}{}
|
|
m.RouteStateHint(ctx, stateUpdateFromChat(chat))
|
|
}
|
|
for _, key := range keys {
|
|
if _, ok := seen[key.ChatID]; !ok {
|
|
m.requestCleanup(ctx, key)
|
|
}
|
|
}
|
|
return nil
|
|
}
|
|
|
|
func (m *runnerManager) heartbeatLoop() {
|
|
ticker := m.opts.Clock.NewTicker(m.opts.HeartbeatInterval, "chatworker", "heartbeat")
|
|
defer ticker.Stop()
|
|
for {
|
|
select {
|
|
case <-ticker.C:
|
|
if err := m.heartbeatOnce(m.ctx); err != nil {
|
|
m.opts.Logger.Warn(m.ctx, "chatworker heartbeat failed", slogError(err))
|
|
}
|
|
case <-m.ctx.Done():
|
|
return
|
|
}
|
|
}
|
|
}
|
|
|
|
func (m *runnerManager) heartbeatOnce(ctx context.Context) error {
|
|
keys := m.snapshotRunnerKeys()
|
|
if len(keys) == 0 {
|
|
return nil
|
|
}
|
|
chatIDs := make([]uuid.UUID, 0, len(keys))
|
|
runnerIDs := make([]uuid.UUID, 0, len(keys))
|
|
for _, key := range keys {
|
|
chatIDs = append(chatIDs, key.ChatID)
|
|
runnerIDs = append(runnerIDs, key.RunnerID)
|
|
}
|
|
renewed, err := m.renewHeartbeats(ctx, chatIDs, runnerIDs)
|
|
if err != nil {
|
|
// A failed or timed-out tick says nothing about individual
|
|
// leases, so no runner is cleaned up. The next tick retries.
|
|
return xerrors.Errorf("renew chat heartbeats: %w", err)
|
|
}
|
|
renewedKeys := make(map[runnerKey]struct{}, len(renewed))
|
|
for _, row := range renewed {
|
|
renewedKeys[runnerKey{ChatID: row.ChatID, RunnerID: row.RunnerID}] = struct{}{}
|
|
}
|
|
for _, key := range keys {
|
|
if _, ok := renewedKeys[key]; ok {
|
|
continue
|
|
}
|
|
// The lease went stale or lost ownership, so its capacity slot
|
|
// may belong to another chat. Stop generating; the chat stays
|
|
// owned by the stale lease until an admitted worker takes it over.
|
|
m.opts.Logger.Warn(ctx, "chatworker heartbeat lease lost, stopping runner",
|
|
slog.F("chat_id", key.ChatID), slog.F("runner_id", key.RunnerID))
|
|
m.requestCleanup(ctx, key)
|
|
}
|
|
return nil
|
|
}
|
|
|
|
// renewHeartbeats renews the given leases under the capacity admission
|
|
// lock and returns the renewed pairs. The tick is bounded by
|
|
// HeartbeatRenewalTimeout and its lock wait by HeartbeatLockTimeout, so a
|
|
// stalled lock holder fails the tick instead of blocking it past the
|
|
// stale threshold.
|
|
func (m *runnerManager) renewHeartbeats(ctx context.Context, chatIDs, runnerIDs []uuid.UUID) ([]database.RenewChatHeartbeatsRow, error) {
|
|
ctx, cancel := context.WithTimeout(ctx, m.opts.HeartbeatRenewalTimeout)
|
|
defer cancel()
|
|
var renewed []database.RenewChatHeartbeatsRow
|
|
// Renewal serializes with capacity admission so that a lease an
|
|
// admission counted as stale, and whose slot it gave away, cannot be
|
|
// renewed afterwards.
|
|
err := m.opts.Store.InTx(func(tx database.Store) error {
|
|
if err := tx.SetTransactionLockTimeout(ctx, m.opts.HeartbeatLockTimeout.Milliseconds()); err != nil {
|
|
return xerrors.Errorf("set lock timeout: %w", err)
|
|
}
|
|
if err := tx.AcquireLock(ctx, database.LockIDChatCapacityAdmission); err != nil {
|
|
return xerrors.Errorf("acquire capacity admission lock: %w", err)
|
|
}
|
|
var err error
|
|
renewed, err = tx.RenewChatHeartbeats(ctx, database.RenewChatHeartbeatsParams{
|
|
ChatIds: chatIDs,
|
|
RunnerIds: runnerIDs,
|
|
StaleSeconds: m.opts.HeartbeatStaleSeconds,
|
|
})
|
|
return err
|
|
}, nil)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
return renewed, nil
|
|
}
|
|
|
|
func (m *runnerManager) heartbeatCleanupLoop() {
|
|
ticker := m.opts.Clock.NewTicker(m.opts.HeartbeatCleanupInterval, "chatworker", "heartbeat-cleanup")
|
|
defer ticker.Stop()
|
|
for {
|
|
select {
|
|
case <-ticker.C:
|
|
if err := m.heartbeatCleanupOnce(m.ctx); err != nil {
|
|
m.opts.Logger.Warn(m.ctx, "chatworker heartbeat cleanup failed", slogError(err))
|
|
}
|
|
case <-m.ctx.Done():
|
|
return
|
|
}
|
|
}
|
|
}
|
|
|
|
func (m *runnerManager) heartbeatCleanupOnce(ctx context.Context) error {
|
|
_, err := m.opts.Store.DeleteStaleChatHeartbeats(ctx, m.opts.HeartbeatStaleSeconds)
|
|
return err
|
|
}
|
|
|
|
func stateUpdateFromChat(chat database.Chat) runnerStateUpdate {
|
|
var workerID *uuid.UUID
|
|
if chat.WorkerID.Valid {
|
|
id := chat.WorkerID.UUID
|
|
workerID = &id
|
|
}
|
|
var runnerID *uuid.UUID
|
|
if chat.RunnerID.Valid {
|
|
id := chat.RunnerID.UUID
|
|
runnerID = &id
|
|
}
|
|
return runnerStateUpdate{
|
|
ChatID: chat.ID,
|
|
WorkerID: workerID,
|
|
RunnerID: runnerID,
|
|
SnapshotVersion: chat.SnapshotVersion,
|
|
HistoryVersion: chat.HistoryVersion,
|
|
QueueVersion: chat.QueueVersion,
|
|
GenerationAttempt: chat.GenerationAttempt,
|
|
Status: chat.Status,
|
|
Archived: chat.Archived,
|
|
RequiresActionDeadlineAt: chat.RequiresActionDeadlineAt,
|
|
}
|
|
}
|
|
|
|
func slogError(err error) slog.Field {
|
|
return slog.Error(err)
|
|
}
|