**TL;DR:** Wraps each prompt in a `chat_turn` root span with every lifecycle stage parented under it. The span records one outcome per turn (`completed`, `interrupted`, `error`, `abandoned`), and the new `coderd_chatd_turn_outcomes_total` counter counts those outcomes. Turns stay correct when tasks are canceled, replaced, or retried. Each prompt runs inside a `chat_turn` root span owned by the runner (`runnerTurnSpan`). The generation step, prepare, MCP connect, stream, TTFT, provider attempt, commit, thinking, tool call, compaction, and retry backoff stages are parented under it, activating the stages from #29770. `chat_turn` spans are standalone roots: the turn runs on a worker, often on another replica, and no trace context is persisted. **Lifecycle** - **Start.** Each generation loop iteration calls `Ensure`, which opens the turn if none is open and returns a `turnToken`. The span is backdated to the trigger: the latest of the last user prompt, a dynamic tool's result message after it, and a pending compaction request. An `acquisition` stage covers trigger to pickup. - **Stale trigger / takeover.** A trigger at or before the previous anchor opens the turn at now with no `acquisition` and counts `stale_anchor`. A runner that took the chat over from an owner with a stale heartbeat (`TakenOver`, from acquisition) opens its first turn at now with no `acquisition`, so the dead owner's run is not reported as pickup delay. - **Newer prompt.** `Ensure` on the current turn with a newer trigger closes it and opens a new one. The closed turn is `abandoned` unless the task that ran it records an outcome before releasing it. - **Finish.** After `FinishTurn` or `EnterRequiresAction` commits, the step calls `Complete`; `Settle` closes the span after the step's own stage ends, so the finishing step is inside the turn. A chat waiting in `requires_action` is outside any turn; the generation resumed by its tool results opens a new one. - **Emission.** Closing a turn fixes its end time; its span is emitted once every task that joined it (`Ensure` takes the task ID) has called `Release`, which `runTask` does after the task returns. A finishing task whose `FinishTurn` promoted a queued prompt therefore still counts its turn as `completed` even when the promoted task's `Ensure` runs before its `Complete`. `End` at runner exit emits whatever is left. - **Failure.** A step that commits `FinishError`, or fails closed, invalidates the turn as `error` after the commit. Errors the task runner retries, including task timeouts, keep the same turn open; an expected, non-retryable exit records no outcome and leaves the turn open until the interrupt task, a newer prompt, or runner exit closes it. A canceled generation task records no outcome and cannot obtain a turn token, so a replaced task never touches its successor's turn. - **Interrupt.** The interrupt task marks the open turn `interrupted` before `FinishInterruption` commits and closes it after. Only a stop (by the user or a parent agent) is `interrupted`; it ends with unset span status, since error status is reserved for `error` turns. runner shutdown, loss of ownership, and message edits close the turn as `abandoned` with unset status. - **Promotion.** Every promoting transition (`FinishTurn`, `FinishInterruption`, a send to an errored chat, `PromoteQueued`) returns the promoted message's queued time in its result, and one `recordQueueWait` helper records a standalone `queue_wait` to the promotion. The promoted turn opens at the promotion time. `chat_turn` is observed on `coderd_chatd_stage_duration_seconds` only for completed turns, so the histogram is completed-turn latency; every turn still emits its span. `coderd_chatd_turn_outcomes_total{outcome, chat_kind}` counts every closed turn exactly once (`completed`, `interrupted`, `error`, `abandoned`), matching the span's `turn_outcome`, under the `chat-stage-metrics` experiment. Turn time accounting is in draft #29772. **Stages and identity** - `tool_call` is a live stage around each local tool call in `chatloop`; the tool runs on its context, and it ends errored only when the tool fails to execute. Calls that never run (inactive, unknown, or rejected by policy) record no stage. - `mcp_connect` carries connected and failed server counts and ends errored only when servers were configured and none connected. - The compaction stage carries the model that runs the summary. - `withStageIdentity` puts scope and chat kind on the turn context. Detached background work runs in `background` scope as its own trace roots. - The `advisor` tool's nested model call appears only as its enclosing `tool_call`. ARCHITECTURE.md gains a "Lifecycle tracing" section. <details> <summary>Review decisions</summary> - Turn identity stays timestamp-based with targeted fixes (clamped acquisition, rotation on a newer trigger, invalidate after commit, fail-closed invalidation, settle closing invalidated turns) plus a takeover flag; keying turns on message IDs would not detect takeover and cannot express interrupted/abandoned outcomes. - Outcome classification: the interrupt task owns `interrupted`, and cancellation or a fence mismatch alone never records an outcome, so shutdown, ownership loss, and edits are `abandoned`. Only `error` turns set error span status. Remaining ordering races between a step and an interrupt are documented as known limitations in ARCHITECTURE.md. - Turns are emitted after the tasks that ran them return, not when they close, so an outcome a replaced task records while unwinding reaches its own turn; the successor task is not delayed. Chosen over an after-commit, pre-publish hook on `ChatMachine.Update`, which would fix only the `FinishTurn` ordering. - `requires_action` ends the turn rather than pausing it, so `chat_turn` measures chatd time; a prompt with client tools counts as several turns. - Promoted turns anchor at promotion time and `queue_wait` is always standalone, so `chat_turn` never includes time spent queued behind another turn. - The former `capacity_wait` stage (#29805) was dropped: it was a per-worker lower bound, excluded from accounting, and needed its own SQL and bookkeeping. #29805's remaining helpers are folded into this PR. </details> Supersedes #28972, #28973, and #29805. 🤖 This PR was generated by Coder Agents on behalf of @jscottmiller.
369 lines
9.7 KiB
Go
369 lines
9.7 KiB
Go
package chatd
|
|
|
|
import (
|
|
"context"
|
|
"errors"
|
|
"sync"
|
|
|
|
"github.com/google/uuid"
|
|
"golang.org/x/xerrors"
|
|
|
|
"github.com/coder/coder/v2/coderd/database"
|
|
coderdpubsub "github.com/coder/coder/v2/coderd/pubsub"
|
|
)
|
|
|
|
type taskKind string
|
|
|
|
const (
|
|
taskKindGeneration taskKind = "generation"
|
|
taskKindInterrupt taskKind = "interrupt"
|
|
taskKindRequiresActionTimeout taskKind = "requires_action_timeout"
|
|
taskKindAbandon taskKind = "abandon"
|
|
)
|
|
|
|
type taskInstanceID uuid.UUID
|
|
|
|
type localWorkKey struct {
|
|
historyVersion int64
|
|
status database.ChatStatus
|
|
}
|
|
|
|
type taskIndexKey struct {
|
|
kind taskKind
|
|
key localWorkKey
|
|
}
|
|
|
|
type taskRecord struct {
|
|
id taskInstanceID
|
|
kind taskKind
|
|
localKey localWorkKey
|
|
cancel context.CancelFunc
|
|
done <-chan struct{}
|
|
}
|
|
|
|
type runner struct {
|
|
ctx context.Context
|
|
mgr *runnerManager
|
|
rec *runnerRecord
|
|
opts chatWorkerOptions
|
|
|
|
lastSnapshotVersion int64
|
|
hasAcceptedState bool
|
|
latestState runnerStateUpdate
|
|
|
|
activeTaskID taskInstanceID
|
|
activeTaskSet bool
|
|
tasks map[taskInstanceID]*taskRecord
|
|
tasksByIndex map[taskIndexKey]taskInstanceID
|
|
localLocks *localLockSet
|
|
debugTurn *runnerDebugTurn
|
|
turnSpan *runnerTurnSpan
|
|
sessionStart sessionStartTracker
|
|
stopNudges stopNudgeTracker
|
|
experiments turnExperimentDecisions
|
|
}
|
|
|
|
func newRunner(ctx context.Context, mgr *runnerManager, rec *runnerRecord, opts chatWorkerOptions) *runner {
|
|
return &runner{
|
|
ctx: ctx,
|
|
mgr: mgr,
|
|
rec: rec,
|
|
opts: opts,
|
|
tasks: make(map[taskInstanceID]*taskRecord),
|
|
tasksByIndex: make(map[taskIndexKey]taskInstanceID),
|
|
localLocks: newLocalLockSet(),
|
|
debugTurn: newRunnerDebugTurn(ctx, opts.Logger),
|
|
turnSpan: newRunnerTurnSpan(mgr.server.stages, rec.takenOver),
|
|
}
|
|
}
|
|
|
|
func (r *runner) run() {
|
|
if !r.bootstrap() {
|
|
return
|
|
}
|
|
for {
|
|
select {
|
|
case state := <-r.rec.stateCh:
|
|
r.processState(state)
|
|
case <-r.ctx.Done():
|
|
r.cancelActiveTask()
|
|
r.waitForTasks()
|
|
r.closeDebugTurn()
|
|
r.turnSpan.End(nil)
|
|
return
|
|
}
|
|
}
|
|
}
|
|
|
|
func (r *runner) bootstrap() bool {
|
|
channel := coderdpubsub.ChatStateUpdateChannel(r.rec.key.ChatID)
|
|
unsubscribe, err := r.opts.Pubsub.SubscribeWithErr(channel, coderdpubsub.HandleChatStateUpdate(
|
|
func(ctx context.Context, payload coderdpubsub.ChatStateUpdateMessage, err error) {
|
|
if err != nil {
|
|
r.opts.Logger.Warn(ctx, "chatworker state update decode failed", slogError(err))
|
|
return
|
|
}
|
|
r.mgr.RouteStateHint(ctx, stateUpdateFromPubsub(r.rec.key.ChatID, payload))
|
|
},
|
|
))
|
|
if err != nil {
|
|
r.mgr.requestCleanup(r.ctx, r.rec.key)
|
|
return false
|
|
}
|
|
if !r.rec.setUnsubscribe(unsubscribe) {
|
|
return false
|
|
}
|
|
chat, err := r.opts.Store.GetChatByID(r.ctx, r.rec.key.ChatID)
|
|
if err != nil {
|
|
r.opts.Logger.Warn(r.ctx, "chatworker runner bootstrap failed", slogError(err))
|
|
r.mgr.requestCleanup(r.ctx, r.rec.key)
|
|
return false
|
|
}
|
|
// Apply the database snapshot directly instead of routing it through
|
|
// the manager. Routing fans out through stateCh, where a stale hint
|
|
// (for example the pre-acquisition chat:update relayed by another
|
|
// runner's subscription) could be processed first while
|
|
// lastSnapshotVersion is still zero. A stale unowned hint would make
|
|
// the runner clean itself up without abandoning the chat, leaving the
|
|
// chat owned by a dead runner until its heartbeat goes stale.
|
|
// Processing the snapshot here seeds lastSnapshotVersion before the
|
|
// run loop drains stateCh, so the dedup in processState drops every
|
|
// hint at or below this version regardless of delivery path.
|
|
r.processState(stateUpdateFromChat(chat))
|
|
return true
|
|
}
|
|
|
|
func stateUpdateFromPubsub(chatID uuid.UUID, payload coderdpubsub.ChatStateUpdateMessage) runnerStateUpdate {
|
|
return runnerStateUpdate{
|
|
ChatID: chatID,
|
|
WorkerID: payload.WorkerID,
|
|
RunnerID: payload.RunnerID,
|
|
SnapshotVersion: payload.SnapshotVersion,
|
|
HistoryVersion: payload.HistoryVersion,
|
|
QueueVersion: payload.QueueVersion,
|
|
GenerationAttempt: payload.GenerationAttempt,
|
|
Status: database.ChatStatus(payload.Status),
|
|
Archived: payload.Archived,
|
|
}
|
|
}
|
|
|
|
func (r *runner) processState(state runnerStateUpdate) {
|
|
if state.SnapshotVersion <= r.lastSnapshotVersion {
|
|
return
|
|
}
|
|
|
|
r.removeFinishedTasks()
|
|
|
|
if !uuidPtrEqual(state.WorkerID, r.rec.workerID) || !uuidPtrEqual(state.RunnerID, r.rec.key.RunnerID) {
|
|
r.acceptState(state)
|
|
r.mgr.requestCleanup(r.ctx, r.rec.key)
|
|
return
|
|
}
|
|
|
|
changed := !r.hasAcceptedState ||
|
|
r.latestState.HistoryVersion != state.HistoryVersion ||
|
|
r.latestState.Status != state.Status ||
|
|
r.latestState.Archived != state.Archived
|
|
if !changed {
|
|
r.acceptState(state)
|
|
return
|
|
}
|
|
if r.hasAcceptedState && r.activeTaskSet {
|
|
r.cancelActiveTask()
|
|
}
|
|
|
|
r.spawnForState(state)
|
|
r.acceptState(state)
|
|
}
|
|
|
|
func (r *runner) acceptState(state runnerStateUpdate) {
|
|
r.hasAcceptedState = true
|
|
r.latestState = state
|
|
r.lastSnapshotVersion = state.SnapshotVersion
|
|
}
|
|
|
|
func (r *runner) spawnForState(state runnerStateUpdate) {
|
|
if state.Archived {
|
|
r.spawnTaskIfNeeded(taskKindAbandon, state)
|
|
return
|
|
}
|
|
switch state.Status {
|
|
case database.ChatStatusRunning:
|
|
r.spawnTaskIfNeeded(taskKindGeneration, state)
|
|
case database.ChatStatusInterrupting:
|
|
r.spawnTaskIfNeeded(taskKindInterrupt, state)
|
|
case database.ChatStatusRequiresAction:
|
|
r.spawnTaskIfNeeded(taskKindRequiresActionTimeout, state)
|
|
case database.ChatStatusWaiting, database.ChatStatusError:
|
|
r.spawnTaskIfNeeded(taskKindAbandon, state)
|
|
default:
|
|
r.spawnTaskIfNeeded(taskKindAbandon, state)
|
|
}
|
|
}
|
|
|
|
func (r *runner) spawnTaskIfNeeded(kind taskKind, state runnerStateUpdate) {
|
|
key := localWorkKey{historyVersion: state.HistoryVersion, status: state.Status}
|
|
idx := taskIndexKey{kind: kind, key: key}
|
|
if r.activeTaskSet && r.tasksByIndex[idx] == r.activeTaskID {
|
|
return
|
|
}
|
|
|
|
id := taskInstanceID(uuid.New())
|
|
taskCtx, cancel := context.WithCancel(r.ctx)
|
|
done := make(chan struct{})
|
|
record := &taskRecord{
|
|
id: id,
|
|
kind: kind,
|
|
localKey: key,
|
|
cancel: cancel,
|
|
done: done,
|
|
}
|
|
r.tasks[id] = record
|
|
r.tasksByIndex[idx] = id
|
|
r.activeTaskID = id
|
|
r.activeTaskSet = true
|
|
|
|
input := chatWorkerTaskStartInput{
|
|
TaskID: uuid.UUID(id),
|
|
ChatID: r.rec.key.ChatID,
|
|
WorkerID: r.rec.workerID,
|
|
RunnerID: r.rec.key.RunnerID,
|
|
HistoryVersion: state.HistoryVersion,
|
|
GenerationAttempt: state.GenerationAttempt,
|
|
Status: state.Status,
|
|
RequiresActionDeadlineAt: state.RequiresActionDeadlineAt,
|
|
DebugTurn: r.debugTurn,
|
|
TurnSpan: r.turnSpan,
|
|
SessionStart: &r.sessionStart,
|
|
StopNudges: &r.stopNudges,
|
|
TurnExperiments: &r.experiments,
|
|
}
|
|
go r.runTask(taskCtx, kind, key, input, done)
|
|
}
|
|
|
|
func (r *runner) runTask(
|
|
ctx context.Context,
|
|
kind taskKind,
|
|
key localWorkKey,
|
|
input chatWorkerTaskStartInput,
|
|
done chan<- struct{},
|
|
) {
|
|
defer close(done)
|
|
// Runs before close(done), after every stage the task started ended.
|
|
defer input.TurnSpan.Release(input.TaskID)
|
|
taskInfo := retryWrapperTaskInfo{
|
|
ChatID: input.ChatID,
|
|
WorkerID: input.WorkerID,
|
|
RunnerID: input.RunnerID,
|
|
}
|
|
err := runTaskWithRetry(ctx, r.opts.retryOptions(), kind, taskInfo, func(ctx context.Context) error {
|
|
unlock, ok := r.localLocks.acquire(ctx, key)
|
|
if !ok {
|
|
return errors.Join(errTaskExpectedExit, xerrors.Errorf("runTask acquire local lock: %w", ctx.Err()))
|
|
}
|
|
defer unlock()
|
|
if ctx.Err() != nil {
|
|
return errors.Join(errTaskExpectedExit, xerrors.Errorf("runTask context canceled: %w", ctx.Err()))
|
|
}
|
|
|
|
switch kind {
|
|
case taskKindGeneration:
|
|
return r.opts.TaskStarter.StartGeneration(ctx, input)
|
|
case taskKindInterrupt:
|
|
return r.opts.TaskStarter.StartInterrupt(ctx, input)
|
|
case taskKindRequiresActionTimeout:
|
|
return r.opts.TaskStarter.StartRequiresActionTimeout(ctx, input)
|
|
case taskKindAbandon:
|
|
return r.opts.TaskStarter.StartAbandon(ctx, input)
|
|
default:
|
|
return errors.Join(errTaskExpectedExit, xerrors.Errorf("unknown task kind %q", kind))
|
|
}
|
|
})
|
|
if err != nil && ctx.Err() == nil {
|
|
r.opts.Logger.Warn(ctx, "chatworker task failed", slogError(err))
|
|
}
|
|
}
|
|
|
|
func (r *runner) cancelActiveTask() {
|
|
if !r.activeTaskSet {
|
|
return
|
|
}
|
|
id := r.activeTaskID
|
|
r.activeTaskSet = false
|
|
if record := r.tasks[id]; record != nil {
|
|
record.cancel()
|
|
}
|
|
}
|
|
|
|
func (r *runner) waitForTasks() {
|
|
for _, record := range r.tasks {
|
|
<-record.done
|
|
}
|
|
}
|
|
|
|
func (r *runner) closeDebugTurn() {
|
|
if r.debugTurn == nil {
|
|
return
|
|
}
|
|
ctx, cancel := context.WithTimeout(context.WithoutCancel(r.ctx), debugFinalizeTimeout)
|
|
defer cancel()
|
|
r.debugTurn.Finalize(ctx)
|
|
}
|
|
|
|
func (r *runner) removeFinishedTasks() {
|
|
for id, record := range r.tasks {
|
|
select {
|
|
case <-record.done:
|
|
delete(r.tasks, id)
|
|
idx := taskIndexKey{kind: record.kind, key: record.localKey}
|
|
if r.tasksByIndex[idx] == id {
|
|
delete(r.tasksByIndex, idx)
|
|
}
|
|
if r.activeTaskSet && r.activeTaskID == id {
|
|
r.activeTaskSet = false
|
|
}
|
|
default:
|
|
}
|
|
}
|
|
}
|
|
|
|
func uuidPtrEqual(got *uuid.UUID, want uuid.UUID) bool {
|
|
return got != nil && *got == want
|
|
}
|
|
|
|
type localLockSet struct {
|
|
mu sync.Mutex
|
|
locked map[localWorkKey]chan struct{}
|
|
}
|
|
|
|
func newLocalLockSet() *localLockSet {
|
|
return &localLockSet{locked: make(map[localWorkKey]chan struct{})}
|
|
}
|
|
|
|
func (l *localLockSet) acquire(ctx context.Context, key localWorkKey) (func(), bool) {
|
|
for {
|
|
l.mu.Lock()
|
|
wait, ok := l.locked[key]
|
|
if !ok {
|
|
released := make(chan struct{})
|
|
l.locked[key] = released
|
|
l.mu.Unlock()
|
|
return func() {
|
|
l.mu.Lock()
|
|
if l.locked[key] == released {
|
|
delete(l.locked, key)
|
|
close(released)
|
|
}
|
|
l.mu.Unlock()
|
|
}, true
|
|
}
|
|
l.mu.Unlock()
|
|
|
|
select {
|
|
case <-wait:
|
|
case <-ctx.Done():
|
|
return nil, false
|
|
}
|
|
}
|
|
}
|