Files
coder/coderd/x/chatd/runner.go
T
J. Scott Miller fc0af7fbd8 feat: open a chat_turn span per prompt (#29771)
**TL;DR:** Wraps each prompt in a `chat_turn` root span with every
lifecycle stage parented under it. The span records one outcome per turn
(`completed`, `interrupted`, `error`, `abandoned`), and the new
`coderd_chatd_turn_outcomes_total` counter counts those outcomes. Turns
stay correct when tasks are canceled, replaced, or retried.

Each prompt runs inside a `chat_turn` root span owned by the runner
(`runnerTurnSpan`). The generation step, prepare, MCP connect, stream,
TTFT, provider attempt, commit, thinking, tool call, compaction, and
retry backoff stages are parented under it, activating the stages from
#29770. `chat_turn` spans are standalone roots: the turn runs on a
worker, often on another replica, and no trace context is persisted.

**Lifecycle**

- **Start.** Each generation loop iteration calls `Ensure`, which opens
the turn if none is open and returns a `turnToken`. The span is
backdated to the trigger: the latest of the last user prompt, a dynamic
tool's result message after it, and a pending compaction request. An
`acquisition` stage covers trigger to pickup.
- **Stale trigger / takeover.** A trigger at or before the previous
anchor opens the turn at now with no `acquisition` and counts
`stale_anchor`. A runner that took the chat over from an owner with a
stale heartbeat (`TakenOver`, from acquisition) opens its first turn at
now with no `acquisition`, so the dead owner's run is not reported as
pickup delay.
- **Newer prompt.** `Ensure` on the current turn with a newer trigger
closes it and opens a new one. The closed turn is `abandoned` unless the
task that ran it records an outcome before releasing it.
- **Finish.** After `FinishTurn` or `EnterRequiresAction` commits, the
step calls `Complete`; `Settle` closes the span after the step's own
stage ends, so the finishing step is inside the turn. A chat waiting in
`requires_action` is outside any turn; the generation resumed by its
tool results opens a new one.
- **Emission.** Closing a turn fixes its end time; its span is emitted
once every task that joined it (`Ensure` takes the task ID) has called
`Release`, which `runTask` does after the task returns. A finishing task
whose `FinishTurn` promoted a queued prompt therefore still counts its
turn as `completed` even when the promoted task's `Ensure` runs before
its `Complete`. `End` at runner exit emits whatever is left.
- **Failure.** A step that commits `FinishError`, or fails closed,
invalidates the turn as `error` after the commit. Errors the task runner
retries, including task timeouts, keep the same turn open; an expected,
non-retryable exit records no outcome and leaves the turn open until the
interrupt task, a newer prompt, or runner exit closes it. A canceled
generation task records no outcome and cannot obtain a turn token, so a
replaced task never touches its successor's turn.
- **Interrupt.** The interrupt task marks the open turn `interrupted`
before `FinishInterruption` commits and closes it after. Only a stop (by
the user or a parent agent) is `interrupted`; it ends with unset span
status, since error status is reserved for `error` turns. runner
shutdown, loss of ownership, and message edits close the turn as
`abandoned` with unset status.
- **Promotion.** Every promoting transition (`FinishTurn`,
`FinishInterruption`, a send to an errored chat, `PromoteQueued`)
returns the promoted message's queued time in its result, and one
`recordQueueWait` helper records a standalone `queue_wait` to the
promotion. The promoted turn opens at the promotion time.

`chat_turn` is observed on `coderd_chatd_stage_duration_seconds` only
for completed turns, so the histogram is completed-turn latency; every
turn still emits its span.

`coderd_chatd_turn_outcomes_total{outcome, chat_kind}` counts every
closed turn exactly once (`completed`, `interrupted`, `error`,
`abandoned`), matching the span's `turn_outcome`, under the
`chat-stage-metrics` experiment. Turn time accounting is in draft
#29772.

**Stages and identity**

- `tool_call` is a live stage around each local tool call in `chatloop`;
the tool runs on its context, and it ends errored only when the tool
fails to execute. Calls that never run (inactive, unknown, or rejected
by policy) record no stage.
- `mcp_connect` carries connected and failed server counts and ends
errored only when servers were configured and none connected.
- The compaction stage carries the model that runs the summary.
- `withStageIdentity` puts scope and chat kind on the turn context.
Detached background work runs in `background` scope as its own trace
roots.
- The `advisor` tool's nested model call appears only as its enclosing
`tool_call`.

ARCHITECTURE.md gains a "Lifecycle tracing" section.

<details>
<summary>Review decisions</summary>

- Turn identity stays timestamp-based with targeted fixes (clamped
acquisition, rotation on a newer trigger, invalidate after commit,
fail-closed invalidation, settle closing invalidated turns) plus a
takeover flag; keying turns on message IDs would not detect takeover and
cannot express interrupted/abandoned outcomes.
- Outcome classification: the interrupt task owns `interrupted`, and
cancellation or a fence mismatch alone never records an outcome, so
shutdown, ownership loss, and edits are `abandoned`. Only `error` turns
set error span status. Remaining ordering races between a step and an
interrupt are documented as known limitations in ARCHITECTURE.md.
- Turns are emitted after the tasks that ran them return, not when they
close, so an outcome a replaced task records while unwinding reaches its
own turn; the successor task is not delayed. Chosen over an
after-commit, pre-publish hook on `ChatMachine.Update`, which would fix
only the `FinishTurn` ordering.
- `requires_action` ends the turn rather than pausing it, so `chat_turn`
measures chatd time; a prompt with client tools counts as several turns.
- Promoted turns anchor at promotion time and `queue_wait` is always
standalone, so `chat_turn` never includes time spent queued behind
another turn.
- The former `capacity_wait` stage (#29805) was dropped: it was a
per-worker lower bound, excluded from accounting, and needed its own SQL
and bookkeeping. #29805's remaining helpers are folded into this PR.

</details>

Supersedes #28972, #28973, and #29805.

🤖 This PR was generated by Coder Agents on behalf of @jscottmiller.
2026-09-30 11:53:58 -05:00

369 lines
9.7 KiB
Go

package chatd
import (
"context"
"errors"
"sync"
"github.com/google/uuid"
"golang.org/x/xerrors"
"github.com/coder/coder/v2/coderd/database"
coderdpubsub "github.com/coder/coder/v2/coderd/pubsub"
)
type taskKind string
const (
taskKindGeneration taskKind = "generation"
taskKindInterrupt taskKind = "interrupt"
taskKindRequiresActionTimeout taskKind = "requires_action_timeout"
taskKindAbandon taskKind = "abandon"
)
type taskInstanceID uuid.UUID
type localWorkKey struct {
historyVersion int64
status database.ChatStatus
}
type taskIndexKey struct {
kind taskKind
key localWorkKey
}
type taskRecord struct {
id taskInstanceID
kind taskKind
localKey localWorkKey
cancel context.CancelFunc
done <-chan struct{}
}
type runner struct {
ctx context.Context
mgr *runnerManager
rec *runnerRecord
opts chatWorkerOptions
lastSnapshotVersion int64
hasAcceptedState bool
latestState runnerStateUpdate
activeTaskID taskInstanceID
activeTaskSet bool
tasks map[taskInstanceID]*taskRecord
tasksByIndex map[taskIndexKey]taskInstanceID
localLocks *localLockSet
debugTurn *runnerDebugTurn
turnSpan *runnerTurnSpan
sessionStart sessionStartTracker
stopNudges stopNudgeTracker
experiments turnExperimentDecisions
}
func newRunner(ctx context.Context, mgr *runnerManager, rec *runnerRecord, opts chatWorkerOptions) *runner {
return &runner{
ctx: ctx,
mgr: mgr,
rec: rec,
opts: opts,
tasks: make(map[taskInstanceID]*taskRecord),
tasksByIndex: make(map[taskIndexKey]taskInstanceID),
localLocks: newLocalLockSet(),
debugTurn: newRunnerDebugTurn(ctx, opts.Logger),
turnSpan: newRunnerTurnSpan(mgr.server.stages, rec.takenOver),
}
}
func (r *runner) run() {
if !r.bootstrap() {
return
}
for {
select {
case state := <-r.rec.stateCh:
r.processState(state)
case <-r.ctx.Done():
r.cancelActiveTask()
r.waitForTasks()
r.closeDebugTurn()
r.turnSpan.End(nil)
return
}
}
}
func (r *runner) bootstrap() bool {
channel := coderdpubsub.ChatStateUpdateChannel(r.rec.key.ChatID)
unsubscribe, err := r.opts.Pubsub.SubscribeWithErr(channel, coderdpubsub.HandleChatStateUpdate(
func(ctx context.Context, payload coderdpubsub.ChatStateUpdateMessage, err error) {
if err != nil {
r.opts.Logger.Warn(ctx, "chatworker state update decode failed", slogError(err))
return
}
r.mgr.RouteStateHint(ctx, stateUpdateFromPubsub(r.rec.key.ChatID, payload))
},
))
if err != nil {
r.mgr.requestCleanup(r.ctx, r.rec.key)
return false
}
if !r.rec.setUnsubscribe(unsubscribe) {
return false
}
chat, err := r.opts.Store.GetChatByID(r.ctx, r.rec.key.ChatID)
if err != nil {
r.opts.Logger.Warn(r.ctx, "chatworker runner bootstrap failed", slogError(err))
r.mgr.requestCleanup(r.ctx, r.rec.key)
return false
}
// Apply the database snapshot directly instead of routing it through
// the manager. Routing fans out through stateCh, where a stale hint
// (for example the pre-acquisition chat:update relayed by another
// runner's subscription) could be processed first while
// lastSnapshotVersion is still zero. A stale unowned hint would make
// the runner clean itself up without abandoning the chat, leaving the
// chat owned by a dead runner until its heartbeat goes stale.
// Processing the snapshot here seeds lastSnapshotVersion before the
// run loop drains stateCh, so the dedup in processState drops every
// hint at or below this version regardless of delivery path.
r.processState(stateUpdateFromChat(chat))
return true
}
func stateUpdateFromPubsub(chatID uuid.UUID, payload coderdpubsub.ChatStateUpdateMessage) runnerStateUpdate {
return runnerStateUpdate{
ChatID: chatID,
WorkerID: payload.WorkerID,
RunnerID: payload.RunnerID,
SnapshotVersion: payload.SnapshotVersion,
HistoryVersion: payload.HistoryVersion,
QueueVersion: payload.QueueVersion,
GenerationAttempt: payload.GenerationAttempt,
Status: database.ChatStatus(payload.Status),
Archived: payload.Archived,
}
}
func (r *runner) processState(state runnerStateUpdate) {
if state.SnapshotVersion <= r.lastSnapshotVersion {
return
}
r.removeFinishedTasks()
if !uuidPtrEqual(state.WorkerID, r.rec.workerID) || !uuidPtrEqual(state.RunnerID, r.rec.key.RunnerID) {
r.acceptState(state)
r.mgr.requestCleanup(r.ctx, r.rec.key)
return
}
changed := !r.hasAcceptedState ||
r.latestState.HistoryVersion != state.HistoryVersion ||
r.latestState.Status != state.Status ||
r.latestState.Archived != state.Archived
if !changed {
r.acceptState(state)
return
}
if r.hasAcceptedState && r.activeTaskSet {
r.cancelActiveTask()
}
r.spawnForState(state)
r.acceptState(state)
}
func (r *runner) acceptState(state runnerStateUpdate) {
r.hasAcceptedState = true
r.latestState = state
r.lastSnapshotVersion = state.SnapshotVersion
}
func (r *runner) spawnForState(state runnerStateUpdate) {
if state.Archived {
r.spawnTaskIfNeeded(taskKindAbandon, state)
return
}
switch state.Status {
case database.ChatStatusRunning:
r.spawnTaskIfNeeded(taskKindGeneration, state)
case database.ChatStatusInterrupting:
r.spawnTaskIfNeeded(taskKindInterrupt, state)
case database.ChatStatusRequiresAction:
r.spawnTaskIfNeeded(taskKindRequiresActionTimeout, state)
case database.ChatStatusWaiting, database.ChatStatusError:
r.spawnTaskIfNeeded(taskKindAbandon, state)
default:
r.spawnTaskIfNeeded(taskKindAbandon, state)
}
}
func (r *runner) spawnTaskIfNeeded(kind taskKind, state runnerStateUpdate) {
key := localWorkKey{historyVersion: state.HistoryVersion, status: state.Status}
idx := taskIndexKey{kind: kind, key: key}
if r.activeTaskSet && r.tasksByIndex[idx] == r.activeTaskID {
return
}
id := taskInstanceID(uuid.New())
taskCtx, cancel := context.WithCancel(r.ctx)
done := make(chan struct{})
record := &taskRecord{
id: id,
kind: kind,
localKey: key,
cancel: cancel,
done: done,
}
r.tasks[id] = record
r.tasksByIndex[idx] = id
r.activeTaskID = id
r.activeTaskSet = true
input := chatWorkerTaskStartInput{
TaskID: uuid.UUID(id),
ChatID: r.rec.key.ChatID,
WorkerID: r.rec.workerID,
RunnerID: r.rec.key.RunnerID,
HistoryVersion: state.HistoryVersion,
GenerationAttempt: state.GenerationAttempt,
Status: state.Status,
RequiresActionDeadlineAt: state.RequiresActionDeadlineAt,
DebugTurn: r.debugTurn,
TurnSpan: r.turnSpan,
SessionStart: &r.sessionStart,
StopNudges: &r.stopNudges,
TurnExperiments: &r.experiments,
}
go r.runTask(taskCtx, kind, key, input, done)
}
func (r *runner) runTask(
ctx context.Context,
kind taskKind,
key localWorkKey,
input chatWorkerTaskStartInput,
done chan<- struct{},
) {
defer close(done)
// Runs before close(done), after every stage the task started ended.
defer input.TurnSpan.Release(input.TaskID)
taskInfo := retryWrapperTaskInfo{
ChatID: input.ChatID,
WorkerID: input.WorkerID,
RunnerID: input.RunnerID,
}
err := runTaskWithRetry(ctx, r.opts.retryOptions(), kind, taskInfo, func(ctx context.Context) error {
unlock, ok := r.localLocks.acquire(ctx, key)
if !ok {
return errors.Join(errTaskExpectedExit, xerrors.Errorf("runTask acquire local lock: %w", ctx.Err()))
}
defer unlock()
if ctx.Err() != nil {
return errors.Join(errTaskExpectedExit, xerrors.Errorf("runTask context canceled: %w", ctx.Err()))
}
switch kind {
case taskKindGeneration:
return r.opts.TaskStarter.StartGeneration(ctx, input)
case taskKindInterrupt:
return r.opts.TaskStarter.StartInterrupt(ctx, input)
case taskKindRequiresActionTimeout:
return r.opts.TaskStarter.StartRequiresActionTimeout(ctx, input)
case taskKindAbandon:
return r.opts.TaskStarter.StartAbandon(ctx, input)
default:
return errors.Join(errTaskExpectedExit, xerrors.Errorf("unknown task kind %q", kind))
}
})
if err != nil && ctx.Err() == nil {
r.opts.Logger.Warn(ctx, "chatworker task failed", slogError(err))
}
}
func (r *runner) cancelActiveTask() {
if !r.activeTaskSet {
return
}
id := r.activeTaskID
r.activeTaskSet = false
if record := r.tasks[id]; record != nil {
record.cancel()
}
}
func (r *runner) waitForTasks() {
for _, record := range r.tasks {
<-record.done
}
}
func (r *runner) closeDebugTurn() {
if r.debugTurn == nil {
return
}
ctx, cancel := context.WithTimeout(context.WithoutCancel(r.ctx), debugFinalizeTimeout)
defer cancel()
r.debugTurn.Finalize(ctx)
}
func (r *runner) removeFinishedTasks() {
for id, record := range r.tasks {
select {
case <-record.done:
delete(r.tasks, id)
idx := taskIndexKey{kind: record.kind, key: record.localKey}
if r.tasksByIndex[idx] == id {
delete(r.tasksByIndex, idx)
}
if r.activeTaskSet && r.activeTaskID == id {
r.activeTaskSet = false
}
default:
}
}
}
func uuidPtrEqual(got *uuid.UUID, want uuid.UUID) bool {
return got != nil && *got == want
}
type localLockSet struct {
mu sync.Mutex
locked map[localWorkKey]chan struct{}
}
func newLocalLockSet() *localLockSet {
return &localLockSet{locked: make(map[localWorkKey]chan struct{})}
}
func (l *localLockSet) acquire(ctx context.Context, key localWorkKey) (func(), bool) {
for {
l.mu.Lock()
wait, ok := l.locked[key]
if !ok {
released := make(chan struct{})
l.locked[key] = released
l.mu.Unlock()
return func() {
l.mu.Lock()
if l.locked[key] == released {
delete(l.locked, key)
close(released)
}
l.mu.Unlock()
}, true
}
l.mu.Unlock()
select {
case <-wait:
case <-ctx.Done():
return nil, false
}
}
}