Files
coder/coderd/x/chatd/chatdebug/service.go
T
Michael Suchacz ac824cd235 chore(coderd): remove dead code from chatd (#29008)
Removes production code in `coderd/x/chatd` that nothing reaches, the
tests that only existed to exercise it, and option/config fields that
callers filled in but chatd never read. No runtime behaviour changes;
the only edits outside `coderd/x/chatd` drop the now-nonexistent fields
at their three call sites (`coderd/coderd.go`, `coderd/exp_chats.go`,
`coderd/coderdtest`).

Net: 42 files, +147 / -1065.

Verification method for every item: `golang.org/x/tools/cmd/deadcode`
(with and without `-test`) rooted at `cmd/coder`, `enterprise/cmd/coder`
and every package that imports chatd; a `go/packages` pass counting
`TypesInfo.Uses` per declared type, const, var, field and method across
the same roots, split into test and non-test use sites, plus a
read-vs-write split for struct fields; `rg` over the whole repo
including `site/src` for json tag names; then `go build ./...`, `go vet
./coderd/x/chatd/...`, `golangci-lint run ./coderd/x/chatd/... ./coderd/
./coderd/coderdtest/`, and `go test ./coderd/x/chatd/...` (all 20
packages) locally.

<details>
<summary>Evidence per deletion</summary>

| Removed | What it was | Why unused |
|---|---|---|
| `subagent.go` `resolveConfiguredModelOverride`, `parseModelOverride`,
`parsedModelOverride` (~135 lines) +
`TestResolveConfiguredModelOverride_*`, `TestParseModelOverride` |
raw-string model override resolver | Only callers were its two tests;
production resolves overrides through `resolveOrganizationModelOverride`
(`title_override.go`, `compaction_override.go`, `subagent.go`). |
| `chatd.go` `removeSkillIndexMessages`, `isSkillIndexMessage` +
`TestSkillIndexRefreshReplacesStaleAliases` | skill-index refresh
helpers | Zero production references; the test simulated a refresh path
no production code performs. |
| `chatd.go` `chatMessage`, `newChatMessage`, `withCreatedBy`,
`appendMessageFields`, `BuildSingleChatMessageInsertParams` (~90 lines)
| insert-params builder | Exported, but every caller was a `chatd_test`
file (6 sites). Replaced by a 25-line `singleChatMessageInsertParams`
helper in `chatd_helpers_test.go`. |
| `chatd.go` `Server.closeInflightAdmission` | one-line wrapper around
`inflightClosed.Store(true)` | Single call site (`Close`); inlined. |
| `chatd.go` `Server.workerID`, `instructionLookupTimeout`,
`pendingChatAcquireInterval`, `maxChatsPerAcquire`,
`chatHeartbeatInterval` | Server struct fields | Assigned in `New` and
never read anywhere (the values reach the worker through
`chatWorkerOptions`, not these fields). |
| `Config.InstructionLookupTimeout`, `homeInstructionLookupTimeout`,
`coderd.Options.ChatdInstructionLookupTimeout`,
`coderdtest.Options.ChatdInstructionLookupTimeout` | timeout plumbing |
The value only landed in the dead `Server.instructionLookupTimeout`; no
CLI flag sets it and nothing reads it. |
| `PromoteQueuedOptions.CreatedBy` | option field | Set by
`coderd/exp_chats.go` and three tests, never read in chatd (promotion
keeps the queued row's `created_by`). |
| `SubmitToolResultsOptions.DynamicTools` | option field | Set by
`coderd/exp_chats.go` (which re-read `chat.DynamicTools` just to pass
it) and three tests, never read in chatd. |
| `attachments.go` `buildAssistantPartsForPersist` (whole file) |
assistant part builder | Stale copy left by #26270; production uses
`buildAssistantParts` in `message_conversion.go`. Its six tests were
retargeted to the live function (now
`message_conversion_internal_test.go`) and a dead-copy assertion in
`subagent_internal_test.go` was dropped. |
| `chatloop.RunOptions` (94 lines) | old loop options struct | Only two
of 23 fields were ever populated, for one internal call to
`prepareMessagesForRequest`, which now takes `model, logger` directly
and no longer returns the ignored canonical slice.
`GenerateAssistantOptions` / `ExecuteLocalToolsOptions` are the live
types. |
| `chatloop.ErrDynamicToolCall`, `ErrStopAfterTool`, `PendingToolCall`,
`PersistedStep.PendingDynamicToolCalls` | dynamic-tool exit protocol |
Zero references outside their declarations (one stale comment in
`chatdebug/service.go` updated). |
| `chatloop.CompactionOptions.Persist`, `.OnError` | callbacks |
`Persist` had zero references; `OnError` was set only by two tests that
asserted it was never invoked. |
| `chatloop` `stepResult.shouldContinue` + its computation | flag |
Written once, never read. |
| `chatloop.StateWaiting` | metric label const | Zero references. |
| `generation.go` `generationOutcome.InsertedMessages`,
`runnerActionMessage` | outcome field | Built on every commit, never
read. |
| `chatstate` `ExecutionState.IsArchived`, `AllOwnershipStates`,
`OwnershipState`/`StateU`/`StateO`, `TransitionAcquire` | state-machine
leftovers | Zero references (the ownership consts became dead once
`AllOwnershipStates` went). |
| `chatstate` `ExecutionState.QueueNonEmpty` | predicate | Only
referenced by the test asserting its own table; the sibling `IsRunnable`
check in that test is kept. |
| `chatdebug.Attempt.RetryClassification`, `.RetryDelayMs` | json fields
| Never assigned anywhere, `omitempty`, and
`retry_classification`/`retry_delay_ms` do not appear in `site/src` or
`typesGenerated.ts`. |
| `chatdebug.RunContext.Kind`, `StepContext.HistoryTipMessageID`,
`RecorderOptions.Provider`/`.Model`, `stepHandle.opts`/`.usage`/`.err` |
context/handle fields | Written (from `chatd.go`,
`chatloop/compaction.go`, `modelcall.go`, `recorder.go`) and never read.
The DB run row keeps its own `kind` column. |
| `chatprovider.MergeProviderAPIKeys` +
`TestMergeProviderAPIKeys_PreservesProviderRegions` | key merge |
Superseded by `ResolveUserProviderKeys`, whose region test already
covers the same behaviour. |
| `chatprovider` `providerCreationError` | one-line `xerrors.Errorf`
wrapper | Single call site; inlined with identical message. |
| `chatretry.IsRetryable` | classifier wrapper | Zero references. |
| `computeruse.IsTool` + `TestComputerUseTool` | one-line forwarder to
`fantasyopenai.IsComputerUseTool` | No production caller; two test
assertions now call the fantasy function directly, and the dropped test
only asserted the library's own constructor. |
| `chatprompt.ToolResultToPart` | one-line forwarder to
`codersdk.ChatMessageToolResult` | Single call site; inlined. |
| `subagent_catalog.go` `formatSubagentDefinitions` | forwarder with
`nil` overrides | Single call site; inlined. |
| `worker.go` `chatWorker.chatWorkerID` | getter | Only tests called it;
they read `opts.WorkerID` directly now. |
| `chattest.AnthropicErrorResponse`, `OpenAIServerErrorResponse`,
`ChatMessageWithParts` (`messages.go` removed) | test fixtures | Zero
references. |
| `chattest` `AnthropicRequest.Options`, `OpenAIRequest.Options` |
`json:",inline"` map fields | `encoding/json` has no inline tag (the
TODO on each said so); never populated, never read. |

</details>

<details>
<summary>Considered and kept</summary>

- `chatstate.CreateChat` (test-only thin wrapper over `insertChat`,
production uses `CreateChatWithID`): documented package API with 25 test
call sites; not worth the churn.
- `chatprompt.MarshalContent` / `MarshalToolResult`: test-only, but they
deliberately produce legacy-format DB rows so the read path's legacy
handling stays covered.
- `chatstate.AllowedExecutionTransitionsFrom` / `AllowedInputStates` /
`AllowedExecutionTransitionOutputs`, `chatdebug.StepFromContext`,
`WithClock`, `WithStaleThreshold`, `WaitUntilIdleForTest`,
`PublishBuffer.BufferedChannels`, `chattool.NewWorkspaceMCPTool`: test
seams that tests actively use to reach otherwise unexported state.
- `AgentCapacityLimiter` interface (one production implementation):
tests inject fakes.
- `applyHookResultMessages` / `appendHookResultMessages`: a named pair
over one placement enum, both used.
- Nil-receiver guards on `*Metrics`, `*Result`, `*streamSyncPoller`,
`*runnerDebugTurn`: nil is a supported "disabled" value on those paths.
- Tests flagged by heuristics as assertion-light
(`TestNopMetrics_DoesNotPanic`, `TestStepHandleFinish_NilHandle`,
`TestRetry_OnRetryNilDoesNotPanic`, the `chatstate`
synthetic-cancellation tests that assert through helpers): they cover
real nil-safety or delegate assertions to helpers; kept.

</details>

<details>
<summary>Out-of-scope findings (not changed here)</summary>

- `chatdebug.Service.FinalizeStale` has had no production caller since
#26270 removed the background sweep loop; only its tests call it, and
comments in `chatd_debug.go` and `database/querier.go` still describe
the loop. Deleting it would erase the implementation a fix needs, so it
is left for a decision.
- `coderd_chatd_chats` (gauge) and
`coderd_chatd_stream_buffer_dropped_total` (counter) are registered and
documented in `docs/admin/integrations/prometheus.md` but never set or
incremented in production; `Metrics.RecordStreamBufferDropped` is
test-only. Removing them changes the exported metric set and the docs,
so they are left alone.

</details>

## Declined review findings

None yet.

> Xum (AI agent) opened this PR on behalf of @ibetitsmike.
2026-09-05 11:39:10 +02:00

834 lines
27 KiB
Go

package chatdebug
import (
"bytes"
"context"
"database/sql"
"encoding/json"
"errors"
"fmt"
"sync"
"sync/atomic"
"time"
"github.com/google/uuid"
"github.com/sqlc-dev/pqtype"
"golang.org/x/xerrors"
"cdr.dev/slog/v3"
"github.com/coder/coder/v2/coderd/database"
"github.com/coder/coder/v2/coderd/database/dbauthz"
"github.com/coder/coder/v2/coderd/database/pubsub"
"github.com/coder/quartz"
)
// DefaultStaleThreshold is the fallback stale timeout for debug rows
// when no caller-provided value is supplied.
const DefaultStaleThreshold = 5 * time.Minute
// Service persists chat debug rows and fans out lightweight change events.
type Service struct {
db database.Store
log slog.Logger
pubsub pubsub.Pubsub
clock quartz.Clock
alwaysEnable bool
// staleAfterNanos stores the stale threshold as nanoseconds in an
// atomic.Int64 so SetStaleAfter and FinalizeStale can be called
// from concurrent goroutines without a data race.
staleAfterNanos atomic.Int64
// thresholdMu protects thresholdChanged.
thresholdMu sync.Mutex
// thresholdChanged is closed by SetStaleAfter to wake heartbeat
// goroutines so they can re-read the (possibly shorter) interval
// immediately instead of waiting for the old ticker to fire.
thresholdChanged chan struct{}
}
// ServiceOption configures optional Service behavior.
type ServiceOption func(*Service)
// WithStaleThreshold overrides the default stale-row finalization
// threshold. Callers that already have a configurable in-flight chat
// timeout (e.g. chatd's InFlightChatStaleAfter) should pass it here
// so the two sweeps stay in sync.
func WithStaleThreshold(d time.Duration) ServiceOption {
return func(s *Service) {
if d > 0 {
s.staleAfterNanos.Store(d.Nanoseconds())
}
}
}
// WithAlwaysEnable forces debug logging on for every chat regardless
// of the runtime admin and user opt-in settings. This is used for the
// deployment-level serpent flag.
func WithAlwaysEnable(always bool) ServiceOption {
return func(s *Service) {
s.alwaysEnable = always
}
}
// WithClock overrides the default real clock. Tests inject
// quartz.NewMock(t) to control time-dependent behavior such as
// heartbeat tickers and FinalizeStale timestamps.
func WithClock(c quartz.Clock) ServiceOption {
return func(s *Service) {
if c != nil {
s.clock = c
}
}
}
// CreateRunParams contains friendly inputs for creating a debug run.
type CreateRunParams struct {
ChatID uuid.UUID
RootChatID uuid.UUID
ParentChatID uuid.UUID
ModelConfigID uuid.UUID
TriggerMessageID int64
HistoryTipMessageID int64
Kind RunKind
Status Status
Provider string
Model string
Summary any
}
// UpdateRunParams contains inputs for updating a debug run.
// Zero-valued fields are treated as "keep the existing value" by the
// COALESCE-based SQL query. Once a field is set it cannot be cleared
// back to NULL; this is intentional for the write-once-finalize
// lifecycle of debug rows.
type UpdateRunParams struct {
ID uuid.UUID
ChatID uuid.UUID
Status Status
Summary any
FinishedAt time.Time
}
// CreateStepParams contains friendly inputs for creating a debug step.
type CreateStepParams struct {
RunID uuid.UUID
ChatID uuid.UUID
StepNumber int32
Operation Operation
Status Status
HistoryTipMessageID int64
NormalizedRequest any
}
// UpdateStepParams contains optional inputs for updating a debug step.
// Most payload fields are typed as any and serialized through nullJSON
// because their shape varies by provider. The Attempts field uses a
// concrete slice for compile-time safety where the schema is stable.
// Zero-valued fields are treated as "keep the existing value" by the
// COALESCE-based SQL query. Once set, fields cannot be cleared back
// to NULL. This is intentional for the write-once-finalize lifecycle
// of debug rows.
type UpdateStepParams struct {
ID uuid.UUID
ChatID uuid.UUID
Status Status
AssistantMessageID int64
NormalizedResponse any
Usage any
Attempts []Attempt
Error any
Metadata any
FinishedAt time.Time
}
// NewService constructs a chat debug persistence service.
func NewService(db database.Store, log slog.Logger, ps pubsub.Pubsub, opts ...ServiceOption) *Service {
if db == nil {
panic("chatdebug: nil database.Store")
}
s := &Service{
db: db,
log: log,
pubsub: ps,
clock: quartz.NewReal(),
thresholdChanged: make(chan struct{}),
}
s.staleAfterNanos.Store(DefaultStaleThreshold.Nanoseconds())
for _, opt := range opts {
opt(s)
}
return s
}
// SetStaleAfter overrides the in-flight stale threshold used when
// finalizing abandoned debug rows. Zero or negative durations are
// ignored, leaving the current threshold (initial or previously
// overridden) unchanged. Active heartbeat goroutines are woken so
// they can re-read the (possibly shorter) interval immediately.
func (s *Service) SetStaleAfter(staleAfter time.Duration) {
if s == nil || staleAfter <= 0 {
return
}
s.staleAfterNanos.Store(staleAfter.Nanoseconds())
// Wake all heartbeat goroutines by closing the current channel
// and replacing it with a fresh one for the next update.
s.thresholdMu.Lock()
close(s.thresholdChanged)
s.thresholdChanged = make(chan struct{})
s.thresholdMu.Unlock()
}
// thresholdChan returns the current threshold-change notification
// channel. Heartbeat goroutines select on this to detect runtime
// stale-threshold updates.
func (s *Service) thresholdChan() <-chan struct{} {
s.thresholdMu.Lock()
defer s.thresholdMu.Unlock()
return s.thresholdChanged
}
// staleThreshold returns the current stale timeout.
func (s *Service) staleThreshold() time.Duration {
ns := s.staleAfterNanos.Load()
d := time.Duration(ns)
if d <= 0 {
return DefaultStaleThreshold
}
return d
}
// heartbeatInterval returns a safe ticker interval for stream heartbeats.
// It is half the stale threshold so at least one touch lands before the
// stale sweep considers the row abandoned. The result is clamped to a
// minimum of 1 ms to prevent panics from time.NewTicker(0) with
// pathologically small thresholds, while still staying well below any
// practical stale timeout.
func (s *Service) heartbeatInterval() time.Duration {
return max(s.staleThreshold()/2, time.Millisecond)
}
func chatdContext(ctx context.Context) context.Context {
//nolint:gocritic // AsChatd provides narrowly-scoped daemon access for
// chat debug persistence reads and writes.
return dbauthz.AsChatd(ctx)
}
// IsEnabled returns whether debug logging is enabled for the given chat.
func (s *Service) IsEnabled(
ctx context.Context,
chatID uuid.UUID,
ownerID uuid.UUID,
) bool {
if s == nil {
return false
}
if s.alwaysEnable {
return true
}
if s.db == nil {
return false
}
authCtx := chatdContext(ctx)
allowUsers, err := s.db.GetChatDebugLoggingAllowUsers(authCtx)
if err != nil {
if errors.Is(err, sql.ErrNoRows) {
return false
}
s.log.Warn(ctx, "failed to load runtime admin chat debug logging setting",
slog.Error(err),
)
return false
}
if !allowUsers {
return false
}
if ownerID == uuid.Nil {
s.log.Warn(ctx, "missing chat owner for debug logging enablement check",
slog.F("chat_id", chatID),
)
return false
}
enabled, err := s.db.GetUserChatDebugLoggingEnabled(authCtx, ownerID)
if err == nil {
return enabled
}
if errors.Is(err, sql.ErrNoRows) {
return false
}
s.log.Warn(ctx, "failed to load user chat debug logging setting",
slog.Error(err),
slog.F("chat_id", chatID),
slog.F("owner_id", ownerID),
)
return false
}
// CreateRun inserts a new debug run and emits a run update event.
func (s *Service) CreateRun(
ctx context.Context,
params CreateRunParams,
) (database.ChatDebugRun, error) {
now := s.clock.Now()
run, err := s.db.InsertChatDebugRun(chatdContext(ctx),
database.InsertChatDebugRunParams{
ChatID: params.ChatID,
RootChatID: nullUUID(params.RootChatID),
ParentChatID: nullUUID(params.ParentChatID),
ModelConfigID: nullUUID(params.ModelConfigID),
TriggerMessageID: nullInt64(params.TriggerMessageID),
HistoryTipMessageID: nullInt64(params.HistoryTipMessageID),
Kind: string(params.Kind),
Status: string(params.Status),
Provider: nullString(params.Provider),
Model: nullString(params.Model),
Summary: s.nullJSON(ctx, params.Summary),
StartedAt: sql.NullTime{Time: now, Valid: true},
UpdatedAt: sql.NullTime{Time: now, Valid: true},
FinishedAt: sql.NullTime{},
})
if err != nil {
return database.ChatDebugRun{}, err
}
s.publishEvent(ctx, run.ChatID, EventKindRunUpdate, run.ID, uuid.Nil)
return run, nil
}
// UpdateRun updates an existing debug run and emits a run update event.
// When a terminal status is set without an explicit FinishedAt, the
// service auto-fills the timestamp so the row is immediately visible
// to the InsertChatDebugStep atomic guard (finished_at IS NULL).
// UpdateChatDebugRun itself enforces finished_at as write-once: once
// the column is populated, repeated auto-fills or explicit refreshes
// never overwrite the original completion timestamp, so calling this
// more than once on an already-finalized run is idempotent.
func (s *Service) UpdateRun(
ctx context.Context,
params UpdateRunParams,
) (database.ChatDebugRun, error) {
if params.Status.IsTerminal() && params.FinishedAt.IsZero() {
params.FinishedAt = s.clock.Now()
}
run, err := s.db.UpdateChatDebugRun(chatdContext(ctx),
database.UpdateChatDebugRunParams{
RootChatID: uuid.NullUUID{},
ParentChatID: uuid.NullUUID{},
ModelConfigID: uuid.NullUUID{},
TriggerMessageID: sql.NullInt64{},
HistoryTipMessageID: sql.NullInt64{},
Status: nullString(string(params.Status)),
Provider: sql.NullString{},
Model: sql.NullString{},
Summary: s.nullJSON(ctx, params.Summary),
FinishedAt: nullTime(params.FinishedAt),
Now: s.clock.Now(),
ID: params.ID,
ChatID: params.ChatID,
})
if err != nil {
return database.ChatDebugRun{}, err
}
s.publishEvent(ctx, run.ChatID, EventKindRunUpdate, run.ID, uuid.Nil)
return run, nil
}
// errRunFinalized is returned by CreateStep when the parent run has
// already reached a terminal state (finished_at IS NOT NULL). This
// prevents delayed retries from appending in-progress steps to runs
// that FinalizeStale already marked as interrupted.
var errRunFinalized = xerrors.New("parent run is already finalized")
// errRunNotFound is returned by CreateStep when the parent run cannot
// be located (missing run_id or chat_id mismatch). This surfaces
// caller-side data bugs instead of conflating them with the legitimate
// "already finalized" terminal case.
var errRunNotFound = xerrors.New("parent run not found")
// CreateStep inserts a new debug step and emits a step update event.
// It returns errRunFinalized if the parent run has already finished,
// or errRunNotFound if the run_id/chat_id pair does not match an
// existing run. The finalization guard is enforced atomically by the
// INSERT's CTE, which issues an UPDATE on the parent run (taking a
// row lock). This prevents concurrent FinalizeStale from setting
// finished_at between the check and the INSERT.
func (s *Service) CreateStep(
ctx context.Context,
params CreateStepParams,
) (database.ChatDebugStep, error) {
now := s.clock.Now()
insert := database.InsertChatDebugStepParams{
RunID: params.RunID,
StepNumber: params.StepNumber,
Operation: string(params.Operation),
Status: string(params.Status),
HistoryTipMessageID: nullInt64(params.HistoryTipMessageID),
AssistantMessageID: sql.NullInt64{},
NormalizedRequest: s.nullJSON(ctx, params.NormalizedRequest),
NormalizedResponse: pqtype.NullRawMessage{},
Usage: pqtype.NullRawMessage{},
Attempts: pqtype.NullRawMessage{},
Error: pqtype.NullRawMessage{},
Metadata: pqtype.NullRawMessage{},
StartedAt: sql.NullTime{Time: now, Valid: true},
UpdatedAt: sql.NullTime{Time: now, Valid: true},
FinishedAt: sql.NullTime{},
ChatID: params.ChatID,
}
// Cap retry attempts to prevent infinite loops under
// pathological concurrency. Each iteration performs two DB
// round-trips (insert + list), so 10 retries is generous.
const maxCreateStepRetries = 10
for range maxCreateStepRetries {
if err := ctx.Err(); err != nil {
return database.ChatDebugStep{}, err
}
step, err := s.db.InsertChatDebugStep(chatdContext(ctx), insert)
if err == nil {
// The INSERT CTE atomically bumps the parent run's
// updated_at, so no separate touch call is needed.
s.publishEvent(ctx, step.ChatID, EventKindStepUpdate, step.RunID, step.ID)
return step, nil
}
// The INSERT's locked_run CTE filters on id, chat_id, and
// finished_at IS NULL, so sql.ErrNoRows can mean "run not
// found", "chat_id mismatch", or "already finalized." Look
// the run up to disambiguate instead of conflating
// caller-side data bugs with the legitimate terminal case.
if errors.Is(err, sql.ErrNoRows) {
return database.ChatDebugStep{}, s.classifyMissingRun(ctx, params)
}
if !database.IsUniqueViolation(err, database.UniqueIndexChatDebugStepsRunStep) {
return database.ChatDebugStep{}, err
}
steps, listErr := s.db.GetChatDebugStepsByRunID(chatdContext(ctx), params.RunID)
if listErr != nil {
return database.ChatDebugStep{}, listErr
}
nextStepNumber := insert.StepNumber + 1
for _, existing := range steps {
if existing.StepNumber >= nextStepNumber {
nextStepNumber = existing.StepNumber + 1
}
}
insert.StepNumber = nextStepNumber
}
return database.ChatDebugStep{}, xerrors.Errorf(
"chatdebug: failed to create step after %d retries (run %s)",
maxCreateStepRetries, params.RunID,
)
}
// classifyMissingRun disambiguates the sql.ErrNoRows returned by
// InsertChatDebugStep's locked_run CTE. The CTE filters on id,
// chat_id, and finished_at IS NULL, so empty RETURNING rows can mean
// the run is absent, belongs to a different chat, or has already been
// finalized. GetChatDebugRunByID is keyed only by id, which is
// sufficient to tell these cases apart.
func (s *Service) classifyMissingRun(
ctx context.Context,
params CreateStepParams,
) error {
run, err := s.db.GetChatDebugRunByID(chatdContext(ctx), params.RunID)
if errors.Is(err, sql.ErrNoRows) {
return errRunNotFound
}
if err != nil {
return xerrors.Errorf("look up parent run after failed step insert: %w", err)
}
if run.ChatID != params.ChatID {
return errRunNotFound
}
if run.FinishedAt.Valid {
return errRunFinalized
}
// The run matches the caller's (run_id, chat_id) and is still
// open, yet the INSERT returned no rows. This is unexpected
// under write-once-finalize semantics and likely indicates a
// concurrent delete or unrelated defect; surface it instead of
// silently masking it as a terminal case.
return xerrors.Errorf(
"InsertChatDebugStep returned no rows but run is still active (run_id=%s)",
params.RunID,
)
}
// UpdateStep updates an existing debug step and emits a step update event.
// When a terminal status is set without an explicit FinishedAt, the
// service auto-fills the timestamp so the stale sweep does not leave
// terminal rows with finished_at = NULL.
func (s *Service) UpdateStep(
ctx context.Context,
params UpdateStepParams,
) (database.ChatDebugStep, error) {
if params.Status.IsTerminal() && params.FinishedAt.IsZero() {
params.FinishedAt = s.clock.Now()
}
step, err := s.db.UpdateChatDebugStep(chatdContext(ctx),
database.UpdateChatDebugStepParams{
Status: nullString(string(params.Status)),
HistoryTipMessageID: sql.NullInt64{},
AssistantMessageID: nullInt64(params.AssistantMessageID),
NormalizedRequest: pqtype.NullRawMessage{},
NormalizedResponse: s.nullJSON(ctx, params.NormalizedResponse),
Usage: s.nullJSON(ctx, params.Usage),
Attempts: s.nullJSON(ctx, params.Attempts),
Error: s.nullJSON(ctx, params.Error),
Metadata: s.nullJSON(ctx, params.Metadata),
FinishedAt: nullTime(params.FinishedAt),
Now: s.clock.Now(),
ID: params.ID,
ChatID: params.ChatID,
})
if err != nil {
return database.ChatDebugStep{}, err
}
s.publishEvent(ctx, step.ChatID, EventKindStepUpdate, step.RunID, step.ID)
return step, nil
}
// TouchStep bumps the step's and its parent run's updated_at timestamps
// without changing any other fields. This prevents long-running operations
// (e.g. streaming) from being prematurely swept by FinalizeStale, which
// first marks runs stale by chat_debug_runs.updated_at and then cascades
// to steps whose run_id was just finalized.
func (s *Service) TouchStep(
ctx context.Context,
stepID uuid.UUID,
runID uuid.UUID,
chatID uuid.UUID,
) error {
// Atomically bump both the step and its parent run so
// FinalizeStale cannot interleave between the two touches.
return s.db.TouchChatDebugStepAndRun(chatdContext(ctx),
database.TouchChatDebugStepAndRunParams{
Now: s.clock.Now(),
StepID: stepID,
RunID: runID,
ChatID: chatID,
})
}
// TouchRun bumps the run's updated_at timestamp without changing any
// other fields. Runner-owned debug turns use this while no model step is
// active, such as requires-action waits. It is a no-op when the service
// is nil or either ID is uuid.Nil, since debug tracking is optional and
// callers may not have an open run.
func (s *Service) TouchRun(ctx context.Context, runID uuid.UUID, chatID uuid.UUID) error {
if s == nil || runID == uuid.Nil || chatID == uuid.Nil {
return nil
}
return s.db.TouchChatDebugRunUpdatedAt(chatdContext(ctx),
database.TouchChatDebugRunUpdatedAtParams{
Now: s.clock.Now(),
ID: runID,
ChatID: chatID,
})
}
// LaunchRunHeartbeat starts a goroutine that periodically touches an
// open run until done is closed or ctx is canceled. It is a no-op when
// the service is nil, either ID is uuid.Nil, or done is nil, since debug
// tracking is optional and callers may not have an open run.
func (s *Service) LaunchRunHeartbeat(ctx context.Context, runID uuid.UUID, chatID uuid.UUID, done <-chan struct{}) {
if s == nil || runID == uuid.Nil || chatID == uuid.Nil || done == nil {
return
}
go func() {
thresholdCh := s.thresholdChan()
interval := s.heartbeatInterval()
ticker := s.clock.NewTicker(interval, "chatdebug", "run-heartbeat")
defer ticker.Stop()
maybeResetTicker := func() {
if newInterval := s.heartbeatInterval(); newInterval != interval {
interval = newInterval
ticker.Reset(interval, "chatdebug", "run-heartbeat")
}
}
for {
select {
case <-ctx.Done():
return
case <-done:
return
case <-thresholdCh:
thresholdCh = s.thresholdChan()
maybeResetTicker()
case <-ticker.C:
if err := s.TouchRun(ctx, runID, chatID); err != nil {
s.log.Debug(ctx, "run heartbeat touch failed",
slog.Error(err),
slog.F("run_id", runID),
)
}
maybeResetTicker()
}
}
}()
}
// DeleteByChatID deletes debug data for a chat and emits a delete event.
// The startedBefore bound scopes deletion to runs created before that
// instant so that retried cleanup does not remove runs created by a
// replacement turn that raced ahead of the retry window (for example,
// an unarchive that fires between the initial archive-cleanup attempt
// and its retry).
func (s *Service) DeleteByChatID(
ctx context.Context,
chatID uuid.UUID,
startedBefore time.Time,
) (int64, error) {
deleted, err := s.db.DeleteChatDebugDataByChatID(
chatdContext(ctx),
database.DeleteChatDebugDataByChatIDParams{
ChatID: chatID,
StartedBefore: startedBefore,
},
)
if err != nil {
return 0, err
}
s.publishEvent(ctx, chatID, EventKindDelete, uuid.Nil, uuid.Nil)
return deleted, nil
}
// DeleteAfterMessageID deletes debug data newer than the given message.
// The startedBefore bound scopes deletion to runs created before that
// instant so that retried cleanup does not remove runs created by a
// replacement turn that raced ahead of the retry window.
func (s *Service) DeleteAfterMessageID(
ctx context.Context,
chatID uuid.UUID,
messageID int64,
startedBefore time.Time,
) (int64, error) {
deleted, err := s.db.DeleteChatDebugDataAfterMessageID(
chatdContext(ctx),
database.DeleteChatDebugDataAfterMessageIDParams{
ChatID: chatID,
MessageID: messageID,
StartedBefore: startedBefore,
},
)
if err != nil {
return 0, err
}
s.publishEvent(ctx, chatID, EventKindDelete, uuid.Nil, uuid.Nil)
return deleted, nil
}
// FinalizeStale finalizes stale in-flight debug rows and emits a broadcast.
func (s *Service) FinalizeStale(
ctx context.Context,
) (database.FinalizeStaleChatDebugRowsRow, error) {
now := s.clock.Now()
result, err := s.db.FinalizeStaleChatDebugRows(
chatdContext(ctx),
database.FinalizeStaleChatDebugRowsParams{
Now: now,
UpdatedBefore: now.Add(-s.staleThreshold()),
},
)
if err != nil {
return database.FinalizeStaleChatDebugRowsRow{}, err
}
if result.RunsFinalized > 0 || result.StepsFinalized > 0 {
s.publishEvent(ctx, uuid.Nil, EventKindFinalize, uuid.Nil, uuid.Nil)
}
return result, nil
}
// FinalizeRunParams bundles the arguments for FinalizeRun.
type FinalizeRunParams struct {
RunID uuid.UUID
ChatID uuid.UUID
Status Status
SeedSummary map[string]any
// Timeout for the aggregate + update calls. Zero defaults to 5s.
Timeout time.Duration
}
// FinalizeRun aggregates the run summary, updates the run status, and
// cleans up the step counter. It detaches from the parent context's
// cancellation so finalization succeeds even when the request context
// is already done. Errors are returned but are always safe to ignore;
// callers that treat debug instrumentation as best-effort can discard
// them.
func (s *Service) FinalizeRun(ctx context.Context, p FinalizeRunParams) error {
timeout := p.Timeout
if timeout <= 0 {
timeout = 5 * time.Second
}
finalizeCtx, cancel := context.WithTimeout(
context.WithoutCancel(ctx), timeout,
)
defer cancel()
finalSummary := p.SeedSummary
if aggregated, aggErr := s.AggregateRunSummary(
finalizeCtx,
p.RunID,
p.SeedSummary,
); aggErr != nil {
// Non-fatal: proceed with the seed summary.
s.log.Warn(ctx, "failed to aggregate debug run summary",
slog.F("chat_id", p.ChatID),
slog.F("run_id", p.RunID),
slog.Error(aggErr),
)
} else {
finalSummary = aggregated
}
if _, err := s.UpdateRun(finalizeCtx, UpdateRunParams{
ID: p.RunID,
ChatID: p.ChatID,
Status: p.Status,
Summary: finalSummary,
FinishedAt: s.clock.Now(),
}); err != nil {
CleanupStepCounter(p.RunID)
return xerrors.Errorf("update debug run: %w", err)
}
CleanupStepCounter(p.RunID)
return nil
}
// ClassifyError maps a run error to the appropriate debug status.
// nil → StatusCompleted, context.Canceled → StatusInterrupted,
// everything else → StatusError. Callers with additional
// classification rules (e.g. ErrInterrupted)
// should handle those before falling back to this helper.
func ClassifyError(err error) Status {
switch {
case err == nil:
return StatusCompleted
case errors.Is(err, context.Canceled):
return StatusInterrupted
default:
return StatusError
}
}
func nullUUID(id uuid.UUID) uuid.NullUUID {
return uuid.NullUUID{UUID: id, Valid: id != uuid.Nil}
}
func nullInt64(v int64) sql.NullInt64 {
return sql.NullInt64{Int64: v, Valid: v != 0}
}
func nullString(value string) sql.NullString {
return sql.NullString{String: value, Valid: value != ""}
}
func nullTime(value time.Time) sql.NullTime {
return sql.NullTime{Time: value, Valid: !value.IsZero()}
}
// jsonClear is a sentinel value that tells nullJSON to emit a valid
// JSON null (JSONB 'null') instead of SQL NULL. COALESCE treats SQL
// NULL as "keep existing" but replaces with a non-NULL JSONB value,
// so passing jsonClear explicitly overwrites a previously set field.
type jsonClear struct{}
// nullJSON marshals value to a NullRawMessage. When value is nil
// (including typed nils such as `var p *T = nil` whose interface
// representation carries a type but no value) or marshals to JSON
// "null", the result is {Valid: false}. Typed nils fall through the
// `value == nil` guard but produce `[]byte("null")` from
// json.Marshal, which the `bytes.Equal(data, []byte("null"))` check
// catches identically. This is intentional for the write-once-finalize
// pattern: combined with the COALESCE-based UPDATE queries, passing
// nil (typed or untyped) preserves the existing column value. Fields
// accumulate monotonically (request -> response -> usage -> error) and
// never need to be cleared during normal operation. The jsonClear
// sentinel exists for the sole exception (error retry clearing).
func (s *Service) nullJSON(ctx context.Context, value any) pqtype.NullRawMessage {
if value == nil {
return pqtype.NullRawMessage{}
}
// Sentinel: emit a valid JSONB null so COALESCE replaces
// any previously stored value.
if _, ok := value.(jsonClear); ok {
return pqtype.NullRawMessage{
RawMessage: json.RawMessage("null"),
Valid: true,
}
}
data, err := json.Marshal(value)
if err != nil {
s.log.Warn(ctx, "failed to marshal chat debug JSON",
slog.Error(err),
slog.F("value_type", fmt.Sprintf("%T", value)),
)
return pqtype.NullRawMessage{}
}
if bytes.Equal(data, []byte("null")) {
return pqtype.NullRawMessage{}
}
return pqtype.NullRawMessage{RawMessage: data, Valid: true}
}
func (s *Service) publishEvent(
ctx context.Context,
chatID uuid.UUID,
kind EventKind,
runID uuid.UUID,
stepID uuid.UUID,
) {
if s.pubsub == nil {
s.log.Debug(ctx,
"chat debug pubsub unavailable; skipping event",
slog.F("kind", kind),
slog.F("chat_id", chatID),
)
return
}
event := DebugEvent{
Kind: kind,
ChatID: chatID,
RunID: runID,
StepID: stepID,
}
data, err := json.Marshal(event)
if err != nil {
s.log.Warn(ctx, "failed to marshal chat debug event",
slog.Error(err),
slog.F("kind", kind),
slog.F("chat_id", chatID),
)
return
}
channel := PubsubChannel(chatID)
if err := s.pubsub.Publish(channel, data); err != nil {
s.log.Warn(ctx, "failed to publish chat debug event",
slog.Error(err),
slog.F("channel", channel),
slog.F("kind", kind),
slog.F("chat_id", chatID),
)
}
}