Removes production code in `coderd/x/chatd` that nothing reaches, the tests that only existed to exercise it, and option/config fields that callers filled in but chatd never read. No runtime behaviour changes; the only edits outside `coderd/x/chatd` drop the now-nonexistent fields at their three call sites (`coderd/coderd.go`, `coderd/exp_chats.go`, `coderd/coderdtest`). Net: 42 files, +147 / -1065. Verification method for every item: `golang.org/x/tools/cmd/deadcode` (with and without `-test`) rooted at `cmd/coder`, `enterprise/cmd/coder` and every package that imports chatd; a `go/packages` pass counting `TypesInfo.Uses` per declared type, const, var, field and method across the same roots, split into test and non-test use sites, plus a read-vs-write split for struct fields; `rg` over the whole repo including `site/src` for json tag names; then `go build ./...`, `go vet ./coderd/x/chatd/...`, `golangci-lint run ./coderd/x/chatd/... ./coderd/ ./coderd/coderdtest/`, and `go test ./coderd/x/chatd/...` (all 20 packages) locally. <details> <summary>Evidence per deletion</summary> | Removed | What it was | Why unused | |---|---|---| | `subagent.go` `resolveConfiguredModelOverride`, `parseModelOverride`, `parsedModelOverride` (~135 lines) + `TestResolveConfiguredModelOverride_*`, `TestParseModelOverride` | raw-string model override resolver | Only callers were its two tests; production resolves overrides through `resolveOrganizationModelOverride` (`title_override.go`, `compaction_override.go`, `subagent.go`). | | `chatd.go` `removeSkillIndexMessages`, `isSkillIndexMessage` + `TestSkillIndexRefreshReplacesStaleAliases` | skill-index refresh helpers | Zero production references; the test simulated a refresh path no production code performs. | | `chatd.go` `chatMessage`, `newChatMessage`, `withCreatedBy`, `appendMessageFields`, `BuildSingleChatMessageInsertParams` (~90 lines) | insert-params builder | Exported, but every caller was a `chatd_test` file (6 sites). Replaced by a 25-line `singleChatMessageInsertParams` helper in `chatd_helpers_test.go`. | | `chatd.go` `Server.closeInflightAdmission` | one-line wrapper around `inflightClosed.Store(true)` | Single call site (`Close`); inlined. | | `chatd.go` `Server.workerID`, `instructionLookupTimeout`, `pendingChatAcquireInterval`, `maxChatsPerAcquire`, `chatHeartbeatInterval` | Server struct fields | Assigned in `New` and never read anywhere (the values reach the worker through `chatWorkerOptions`, not these fields). | | `Config.InstructionLookupTimeout`, `homeInstructionLookupTimeout`, `coderd.Options.ChatdInstructionLookupTimeout`, `coderdtest.Options.ChatdInstructionLookupTimeout` | timeout plumbing | The value only landed in the dead `Server.instructionLookupTimeout`; no CLI flag sets it and nothing reads it. | | `PromoteQueuedOptions.CreatedBy` | option field | Set by `coderd/exp_chats.go` and three tests, never read in chatd (promotion keeps the queued row's `created_by`). | | `SubmitToolResultsOptions.DynamicTools` | option field | Set by `coderd/exp_chats.go` (which re-read `chat.DynamicTools` just to pass it) and three tests, never read in chatd. | | `attachments.go` `buildAssistantPartsForPersist` (whole file) | assistant part builder | Stale copy left by #26270; production uses `buildAssistantParts` in `message_conversion.go`. Its six tests were retargeted to the live function (now `message_conversion_internal_test.go`) and a dead-copy assertion in `subagent_internal_test.go` was dropped. | | `chatloop.RunOptions` (94 lines) | old loop options struct | Only two of 23 fields were ever populated, for one internal call to `prepareMessagesForRequest`, which now takes `model, logger` directly and no longer returns the ignored canonical slice. `GenerateAssistantOptions` / `ExecuteLocalToolsOptions` are the live types. | | `chatloop.ErrDynamicToolCall`, `ErrStopAfterTool`, `PendingToolCall`, `PersistedStep.PendingDynamicToolCalls` | dynamic-tool exit protocol | Zero references outside their declarations (one stale comment in `chatdebug/service.go` updated). | | `chatloop.CompactionOptions.Persist`, `.OnError` | callbacks | `Persist` had zero references; `OnError` was set only by two tests that asserted it was never invoked. | | `chatloop` `stepResult.shouldContinue` + its computation | flag | Written once, never read. | | `chatloop.StateWaiting` | metric label const | Zero references. | | `generation.go` `generationOutcome.InsertedMessages`, `runnerActionMessage` | outcome field | Built on every commit, never read. | | `chatstate` `ExecutionState.IsArchived`, `AllOwnershipStates`, `OwnershipState`/`StateU`/`StateO`, `TransitionAcquire` | state-machine leftovers | Zero references (the ownership consts became dead once `AllOwnershipStates` went). | | `chatstate` `ExecutionState.QueueNonEmpty` | predicate | Only referenced by the test asserting its own table; the sibling `IsRunnable` check in that test is kept. | | `chatdebug.Attempt.RetryClassification`, `.RetryDelayMs` | json fields | Never assigned anywhere, `omitempty`, and `retry_classification`/`retry_delay_ms` do not appear in `site/src` or `typesGenerated.ts`. | | `chatdebug.RunContext.Kind`, `StepContext.HistoryTipMessageID`, `RecorderOptions.Provider`/`.Model`, `stepHandle.opts`/`.usage`/`.err` | context/handle fields | Written (from `chatd.go`, `chatloop/compaction.go`, `modelcall.go`, `recorder.go`) and never read. The DB run row keeps its own `kind` column. | | `chatprovider.MergeProviderAPIKeys` + `TestMergeProviderAPIKeys_PreservesProviderRegions` | key merge | Superseded by `ResolveUserProviderKeys`, whose region test already covers the same behaviour. | | `chatprovider` `providerCreationError` | one-line `xerrors.Errorf` wrapper | Single call site; inlined with identical message. | | `chatretry.IsRetryable` | classifier wrapper | Zero references. | | `computeruse.IsTool` + `TestComputerUseTool` | one-line forwarder to `fantasyopenai.IsComputerUseTool` | No production caller; two test assertions now call the fantasy function directly, and the dropped test only asserted the library's own constructor. | | `chatprompt.ToolResultToPart` | one-line forwarder to `codersdk.ChatMessageToolResult` | Single call site; inlined. | | `subagent_catalog.go` `formatSubagentDefinitions` | forwarder with `nil` overrides | Single call site; inlined. | | `worker.go` `chatWorker.chatWorkerID` | getter | Only tests called it; they read `opts.WorkerID` directly now. | | `chattest.AnthropicErrorResponse`, `OpenAIServerErrorResponse`, `ChatMessageWithParts` (`messages.go` removed) | test fixtures | Zero references. | | `chattest` `AnthropicRequest.Options`, `OpenAIRequest.Options` | `json:",inline"` map fields | `encoding/json` has no inline tag (the TODO on each said so); never populated, never read. | </details> <details> <summary>Considered and kept</summary> - `chatstate.CreateChat` (test-only thin wrapper over `insertChat`, production uses `CreateChatWithID`): documented package API with 25 test call sites; not worth the churn. - `chatprompt.MarshalContent` / `MarshalToolResult`: test-only, but they deliberately produce legacy-format DB rows so the read path's legacy handling stays covered. - `chatstate.AllowedExecutionTransitionsFrom` / `AllowedInputStates` / `AllowedExecutionTransitionOutputs`, `chatdebug.StepFromContext`, `WithClock`, `WithStaleThreshold`, `WaitUntilIdleForTest`, `PublishBuffer.BufferedChannels`, `chattool.NewWorkspaceMCPTool`: test seams that tests actively use to reach otherwise unexported state. - `AgentCapacityLimiter` interface (one production implementation): tests inject fakes. - `applyHookResultMessages` / `appendHookResultMessages`: a named pair over one placement enum, both used. - Nil-receiver guards on `*Metrics`, `*Result`, `*streamSyncPoller`, `*runnerDebugTurn`: nil is a supported "disabled" value on those paths. - Tests flagged by heuristics as assertion-light (`TestNopMetrics_DoesNotPanic`, `TestStepHandleFinish_NilHandle`, `TestRetry_OnRetryNilDoesNotPanic`, the `chatstate` synthetic-cancellation tests that assert through helpers): they cover real nil-safety or delegate assertions to helpers; kept. </details> <details> <summary>Out-of-scope findings (not changed here)</summary> - `chatdebug.Service.FinalizeStale` has had no production caller since #26270 removed the background sweep loop; only its tests call it, and comments in `chatd_debug.go` and `database/querier.go` still describe the loop. Deleting it would erase the implementation a fix needs, so it is left for a decision. - `coderd_chatd_chats` (gauge) and `coderd_chatd_stream_buffer_dropped_total` (counter) are registered and documented in `docs/admin/integrations/prometheus.md` but never set or incremented in production; `Metrics.RecordStreamBufferDropped` is test-only. Removing them changes the exported metric set and the docs, so they are left alone. </details> ## Declined review findings None yet. > Xum (AI agent) opened this PR on behalf of @ibetitsmike.
834 lines
27 KiB
Go
834 lines
27 KiB
Go
package chatdebug
|
|
|
|
import (
|
|
"bytes"
|
|
"context"
|
|
"database/sql"
|
|
"encoding/json"
|
|
"errors"
|
|
"fmt"
|
|
"sync"
|
|
"sync/atomic"
|
|
"time"
|
|
|
|
"github.com/google/uuid"
|
|
"github.com/sqlc-dev/pqtype"
|
|
"golang.org/x/xerrors"
|
|
|
|
"cdr.dev/slog/v3"
|
|
"github.com/coder/coder/v2/coderd/database"
|
|
"github.com/coder/coder/v2/coderd/database/dbauthz"
|
|
"github.com/coder/coder/v2/coderd/database/pubsub"
|
|
"github.com/coder/quartz"
|
|
)
|
|
|
|
// DefaultStaleThreshold is the fallback stale timeout for debug rows
|
|
// when no caller-provided value is supplied.
|
|
const DefaultStaleThreshold = 5 * time.Minute
|
|
|
|
// Service persists chat debug rows and fans out lightweight change events.
|
|
type Service struct {
|
|
db database.Store
|
|
log slog.Logger
|
|
pubsub pubsub.Pubsub
|
|
clock quartz.Clock
|
|
alwaysEnable bool
|
|
// staleAfterNanos stores the stale threshold as nanoseconds in an
|
|
// atomic.Int64 so SetStaleAfter and FinalizeStale can be called
|
|
// from concurrent goroutines without a data race.
|
|
staleAfterNanos atomic.Int64
|
|
|
|
// thresholdMu protects thresholdChanged.
|
|
thresholdMu sync.Mutex
|
|
// thresholdChanged is closed by SetStaleAfter to wake heartbeat
|
|
// goroutines so they can re-read the (possibly shorter) interval
|
|
// immediately instead of waiting for the old ticker to fire.
|
|
thresholdChanged chan struct{}
|
|
}
|
|
|
|
// ServiceOption configures optional Service behavior.
|
|
type ServiceOption func(*Service)
|
|
|
|
// WithStaleThreshold overrides the default stale-row finalization
|
|
// threshold. Callers that already have a configurable in-flight chat
|
|
// timeout (e.g. chatd's InFlightChatStaleAfter) should pass it here
|
|
// so the two sweeps stay in sync.
|
|
func WithStaleThreshold(d time.Duration) ServiceOption {
|
|
return func(s *Service) {
|
|
if d > 0 {
|
|
s.staleAfterNanos.Store(d.Nanoseconds())
|
|
}
|
|
}
|
|
}
|
|
|
|
// WithAlwaysEnable forces debug logging on for every chat regardless
|
|
// of the runtime admin and user opt-in settings. This is used for the
|
|
// deployment-level serpent flag.
|
|
func WithAlwaysEnable(always bool) ServiceOption {
|
|
return func(s *Service) {
|
|
s.alwaysEnable = always
|
|
}
|
|
}
|
|
|
|
// WithClock overrides the default real clock. Tests inject
|
|
// quartz.NewMock(t) to control time-dependent behavior such as
|
|
// heartbeat tickers and FinalizeStale timestamps.
|
|
func WithClock(c quartz.Clock) ServiceOption {
|
|
return func(s *Service) {
|
|
if c != nil {
|
|
s.clock = c
|
|
}
|
|
}
|
|
}
|
|
|
|
// CreateRunParams contains friendly inputs for creating a debug run.
|
|
type CreateRunParams struct {
|
|
ChatID uuid.UUID
|
|
RootChatID uuid.UUID
|
|
ParentChatID uuid.UUID
|
|
ModelConfigID uuid.UUID
|
|
TriggerMessageID int64
|
|
HistoryTipMessageID int64
|
|
Kind RunKind
|
|
Status Status
|
|
Provider string
|
|
Model string
|
|
Summary any
|
|
}
|
|
|
|
// UpdateRunParams contains inputs for updating a debug run.
|
|
// Zero-valued fields are treated as "keep the existing value" by the
|
|
// COALESCE-based SQL query. Once a field is set it cannot be cleared
|
|
// back to NULL; this is intentional for the write-once-finalize
|
|
// lifecycle of debug rows.
|
|
type UpdateRunParams struct {
|
|
ID uuid.UUID
|
|
ChatID uuid.UUID
|
|
Status Status
|
|
Summary any
|
|
FinishedAt time.Time
|
|
}
|
|
|
|
// CreateStepParams contains friendly inputs for creating a debug step.
|
|
type CreateStepParams struct {
|
|
RunID uuid.UUID
|
|
ChatID uuid.UUID
|
|
StepNumber int32
|
|
Operation Operation
|
|
Status Status
|
|
HistoryTipMessageID int64
|
|
NormalizedRequest any
|
|
}
|
|
|
|
// UpdateStepParams contains optional inputs for updating a debug step.
|
|
// Most payload fields are typed as any and serialized through nullJSON
|
|
// because their shape varies by provider. The Attempts field uses a
|
|
// concrete slice for compile-time safety where the schema is stable.
|
|
// Zero-valued fields are treated as "keep the existing value" by the
|
|
// COALESCE-based SQL query. Once set, fields cannot be cleared back
|
|
// to NULL. This is intentional for the write-once-finalize lifecycle
|
|
// of debug rows.
|
|
type UpdateStepParams struct {
|
|
ID uuid.UUID
|
|
ChatID uuid.UUID
|
|
Status Status
|
|
AssistantMessageID int64
|
|
NormalizedResponse any
|
|
Usage any
|
|
Attempts []Attempt
|
|
Error any
|
|
Metadata any
|
|
FinishedAt time.Time
|
|
}
|
|
|
|
// NewService constructs a chat debug persistence service.
|
|
func NewService(db database.Store, log slog.Logger, ps pubsub.Pubsub, opts ...ServiceOption) *Service {
|
|
if db == nil {
|
|
panic("chatdebug: nil database.Store")
|
|
}
|
|
|
|
s := &Service{
|
|
db: db,
|
|
log: log,
|
|
pubsub: ps,
|
|
clock: quartz.NewReal(),
|
|
thresholdChanged: make(chan struct{}),
|
|
}
|
|
s.staleAfterNanos.Store(DefaultStaleThreshold.Nanoseconds())
|
|
for _, opt := range opts {
|
|
opt(s)
|
|
}
|
|
return s
|
|
}
|
|
|
|
// SetStaleAfter overrides the in-flight stale threshold used when
|
|
// finalizing abandoned debug rows. Zero or negative durations are
|
|
// ignored, leaving the current threshold (initial or previously
|
|
// overridden) unchanged. Active heartbeat goroutines are woken so
|
|
// they can re-read the (possibly shorter) interval immediately.
|
|
func (s *Service) SetStaleAfter(staleAfter time.Duration) {
|
|
if s == nil || staleAfter <= 0 {
|
|
return
|
|
}
|
|
s.staleAfterNanos.Store(staleAfter.Nanoseconds())
|
|
|
|
// Wake all heartbeat goroutines by closing the current channel
|
|
// and replacing it with a fresh one for the next update.
|
|
s.thresholdMu.Lock()
|
|
close(s.thresholdChanged)
|
|
s.thresholdChanged = make(chan struct{})
|
|
s.thresholdMu.Unlock()
|
|
}
|
|
|
|
// thresholdChan returns the current threshold-change notification
|
|
// channel. Heartbeat goroutines select on this to detect runtime
|
|
// stale-threshold updates.
|
|
func (s *Service) thresholdChan() <-chan struct{} {
|
|
s.thresholdMu.Lock()
|
|
defer s.thresholdMu.Unlock()
|
|
return s.thresholdChanged
|
|
}
|
|
|
|
// staleThreshold returns the current stale timeout.
|
|
func (s *Service) staleThreshold() time.Duration {
|
|
ns := s.staleAfterNanos.Load()
|
|
d := time.Duration(ns)
|
|
if d <= 0 {
|
|
return DefaultStaleThreshold
|
|
}
|
|
return d
|
|
}
|
|
|
|
// heartbeatInterval returns a safe ticker interval for stream heartbeats.
|
|
// It is half the stale threshold so at least one touch lands before the
|
|
// stale sweep considers the row abandoned. The result is clamped to a
|
|
// minimum of 1 ms to prevent panics from time.NewTicker(0) with
|
|
// pathologically small thresholds, while still staying well below any
|
|
// practical stale timeout.
|
|
func (s *Service) heartbeatInterval() time.Duration {
|
|
return max(s.staleThreshold()/2, time.Millisecond)
|
|
}
|
|
|
|
func chatdContext(ctx context.Context) context.Context {
|
|
//nolint:gocritic // AsChatd provides narrowly-scoped daemon access for
|
|
// chat debug persistence reads and writes.
|
|
return dbauthz.AsChatd(ctx)
|
|
}
|
|
|
|
// IsEnabled returns whether debug logging is enabled for the given chat.
|
|
func (s *Service) IsEnabled(
|
|
ctx context.Context,
|
|
chatID uuid.UUID,
|
|
ownerID uuid.UUID,
|
|
) bool {
|
|
if s == nil {
|
|
return false
|
|
}
|
|
if s.alwaysEnable {
|
|
return true
|
|
}
|
|
if s.db == nil {
|
|
return false
|
|
}
|
|
|
|
authCtx := chatdContext(ctx)
|
|
|
|
allowUsers, err := s.db.GetChatDebugLoggingAllowUsers(authCtx)
|
|
if err != nil {
|
|
if errors.Is(err, sql.ErrNoRows) {
|
|
return false
|
|
}
|
|
s.log.Warn(ctx, "failed to load runtime admin chat debug logging setting",
|
|
slog.Error(err),
|
|
)
|
|
return false
|
|
}
|
|
if !allowUsers {
|
|
return false
|
|
}
|
|
|
|
if ownerID == uuid.Nil {
|
|
s.log.Warn(ctx, "missing chat owner for debug logging enablement check",
|
|
slog.F("chat_id", chatID),
|
|
)
|
|
return false
|
|
}
|
|
|
|
enabled, err := s.db.GetUserChatDebugLoggingEnabled(authCtx, ownerID)
|
|
if err == nil {
|
|
return enabled
|
|
}
|
|
if errors.Is(err, sql.ErrNoRows) {
|
|
return false
|
|
}
|
|
|
|
s.log.Warn(ctx, "failed to load user chat debug logging setting",
|
|
slog.Error(err),
|
|
slog.F("chat_id", chatID),
|
|
slog.F("owner_id", ownerID),
|
|
)
|
|
return false
|
|
}
|
|
|
|
// CreateRun inserts a new debug run and emits a run update event.
|
|
func (s *Service) CreateRun(
|
|
ctx context.Context,
|
|
params CreateRunParams,
|
|
) (database.ChatDebugRun, error) {
|
|
now := s.clock.Now()
|
|
run, err := s.db.InsertChatDebugRun(chatdContext(ctx),
|
|
database.InsertChatDebugRunParams{
|
|
ChatID: params.ChatID,
|
|
RootChatID: nullUUID(params.RootChatID),
|
|
ParentChatID: nullUUID(params.ParentChatID),
|
|
ModelConfigID: nullUUID(params.ModelConfigID),
|
|
TriggerMessageID: nullInt64(params.TriggerMessageID),
|
|
HistoryTipMessageID: nullInt64(params.HistoryTipMessageID),
|
|
Kind: string(params.Kind),
|
|
Status: string(params.Status),
|
|
Provider: nullString(params.Provider),
|
|
Model: nullString(params.Model),
|
|
Summary: s.nullJSON(ctx, params.Summary),
|
|
StartedAt: sql.NullTime{Time: now, Valid: true},
|
|
UpdatedAt: sql.NullTime{Time: now, Valid: true},
|
|
FinishedAt: sql.NullTime{},
|
|
})
|
|
if err != nil {
|
|
return database.ChatDebugRun{}, err
|
|
}
|
|
|
|
s.publishEvent(ctx, run.ChatID, EventKindRunUpdate, run.ID, uuid.Nil)
|
|
return run, nil
|
|
}
|
|
|
|
// UpdateRun updates an existing debug run and emits a run update event.
|
|
// When a terminal status is set without an explicit FinishedAt, the
|
|
// service auto-fills the timestamp so the row is immediately visible
|
|
// to the InsertChatDebugStep atomic guard (finished_at IS NULL).
|
|
// UpdateChatDebugRun itself enforces finished_at as write-once: once
|
|
// the column is populated, repeated auto-fills or explicit refreshes
|
|
// never overwrite the original completion timestamp, so calling this
|
|
// more than once on an already-finalized run is idempotent.
|
|
func (s *Service) UpdateRun(
|
|
ctx context.Context,
|
|
params UpdateRunParams,
|
|
) (database.ChatDebugRun, error) {
|
|
if params.Status.IsTerminal() && params.FinishedAt.IsZero() {
|
|
params.FinishedAt = s.clock.Now()
|
|
}
|
|
run, err := s.db.UpdateChatDebugRun(chatdContext(ctx),
|
|
database.UpdateChatDebugRunParams{
|
|
RootChatID: uuid.NullUUID{},
|
|
ParentChatID: uuid.NullUUID{},
|
|
ModelConfigID: uuid.NullUUID{},
|
|
TriggerMessageID: sql.NullInt64{},
|
|
HistoryTipMessageID: sql.NullInt64{},
|
|
Status: nullString(string(params.Status)),
|
|
Provider: sql.NullString{},
|
|
Model: sql.NullString{},
|
|
Summary: s.nullJSON(ctx, params.Summary),
|
|
FinishedAt: nullTime(params.FinishedAt),
|
|
Now: s.clock.Now(),
|
|
ID: params.ID,
|
|
ChatID: params.ChatID,
|
|
})
|
|
if err != nil {
|
|
return database.ChatDebugRun{}, err
|
|
}
|
|
|
|
s.publishEvent(ctx, run.ChatID, EventKindRunUpdate, run.ID, uuid.Nil)
|
|
return run, nil
|
|
}
|
|
|
|
// errRunFinalized is returned by CreateStep when the parent run has
|
|
// already reached a terminal state (finished_at IS NOT NULL). This
|
|
// prevents delayed retries from appending in-progress steps to runs
|
|
// that FinalizeStale already marked as interrupted.
|
|
var errRunFinalized = xerrors.New("parent run is already finalized")
|
|
|
|
// errRunNotFound is returned by CreateStep when the parent run cannot
|
|
// be located (missing run_id or chat_id mismatch). This surfaces
|
|
// caller-side data bugs instead of conflating them with the legitimate
|
|
// "already finalized" terminal case.
|
|
var errRunNotFound = xerrors.New("parent run not found")
|
|
|
|
// CreateStep inserts a new debug step and emits a step update event.
|
|
// It returns errRunFinalized if the parent run has already finished,
|
|
// or errRunNotFound if the run_id/chat_id pair does not match an
|
|
// existing run. The finalization guard is enforced atomically by the
|
|
// INSERT's CTE, which issues an UPDATE on the parent run (taking a
|
|
// row lock). This prevents concurrent FinalizeStale from setting
|
|
// finished_at between the check and the INSERT.
|
|
func (s *Service) CreateStep(
|
|
ctx context.Context,
|
|
params CreateStepParams,
|
|
) (database.ChatDebugStep, error) {
|
|
now := s.clock.Now()
|
|
insert := database.InsertChatDebugStepParams{
|
|
RunID: params.RunID,
|
|
StepNumber: params.StepNumber,
|
|
Operation: string(params.Operation),
|
|
Status: string(params.Status),
|
|
HistoryTipMessageID: nullInt64(params.HistoryTipMessageID),
|
|
AssistantMessageID: sql.NullInt64{},
|
|
NormalizedRequest: s.nullJSON(ctx, params.NormalizedRequest),
|
|
NormalizedResponse: pqtype.NullRawMessage{},
|
|
Usage: pqtype.NullRawMessage{},
|
|
Attempts: pqtype.NullRawMessage{},
|
|
Error: pqtype.NullRawMessage{},
|
|
Metadata: pqtype.NullRawMessage{},
|
|
StartedAt: sql.NullTime{Time: now, Valid: true},
|
|
UpdatedAt: sql.NullTime{Time: now, Valid: true},
|
|
FinishedAt: sql.NullTime{},
|
|
ChatID: params.ChatID,
|
|
}
|
|
|
|
// Cap retry attempts to prevent infinite loops under
|
|
// pathological concurrency. Each iteration performs two DB
|
|
// round-trips (insert + list), so 10 retries is generous.
|
|
const maxCreateStepRetries = 10
|
|
|
|
for range maxCreateStepRetries {
|
|
if err := ctx.Err(); err != nil {
|
|
return database.ChatDebugStep{}, err
|
|
}
|
|
|
|
step, err := s.db.InsertChatDebugStep(chatdContext(ctx), insert)
|
|
if err == nil {
|
|
// The INSERT CTE atomically bumps the parent run's
|
|
// updated_at, so no separate touch call is needed.
|
|
s.publishEvent(ctx, step.ChatID, EventKindStepUpdate, step.RunID, step.ID)
|
|
return step, nil
|
|
}
|
|
// The INSERT's locked_run CTE filters on id, chat_id, and
|
|
// finished_at IS NULL, so sql.ErrNoRows can mean "run not
|
|
// found", "chat_id mismatch", or "already finalized." Look
|
|
// the run up to disambiguate instead of conflating
|
|
// caller-side data bugs with the legitimate terminal case.
|
|
if errors.Is(err, sql.ErrNoRows) {
|
|
return database.ChatDebugStep{}, s.classifyMissingRun(ctx, params)
|
|
}
|
|
if !database.IsUniqueViolation(err, database.UniqueIndexChatDebugStepsRunStep) {
|
|
return database.ChatDebugStep{}, err
|
|
}
|
|
|
|
steps, listErr := s.db.GetChatDebugStepsByRunID(chatdContext(ctx), params.RunID)
|
|
if listErr != nil {
|
|
return database.ChatDebugStep{}, listErr
|
|
}
|
|
nextStepNumber := insert.StepNumber + 1
|
|
for _, existing := range steps {
|
|
if existing.StepNumber >= nextStepNumber {
|
|
nextStepNumber = existing.StepNumber + 1
|
|
}
|
|
}
|
|
insert.StepNumber = nextStepNumber
|
|
}
|
|
|
|
return database.ChatDebugStep{}, xerrors.Errorf(
|
|
"chatdebug: failed to create step after %d retries (run %s)",
|
|
maxCreateStepRetries, params.RunID,
|
|
)
|
|
}
|
|
|
|
// classifyMissingRun disambiguates the sql.ErrNoRows returned by
|
|
// InsertChatDebugStep's locked_run CTE. The CTE filters on id,
|
|
// chat_id, and finished_at IS NULL, so empty RETURNING rows can mean
|
|
// the run is absent, belongs to a different chat, or has already been
|
|
// finalized. GetChatDebugRunByID is keyed only by id, which is
|
|
// sufficient to tell these cases apart.
|
|
func (s *Service) classifyMissingRun(
|
|
ctx context.Context,
|
|
params CreateStepParams,
|
|
) error {
|
|
run, err := s.db.GetChatDebugRunByID(chatdContext(ctx), params.RunID)
|
|
if errors.Is(err, sql.ErrNoRows) {
|
|
return errRunNotFound
|
|
}
|
|
if err != nil {
|
|
return xerrors.Errorf("look up parent run after failed step insert: %w", err)
|
|
}
|
|
if run.ChatID != params.ChatID {
|
|
return errRunNotFound
|
|
}
|
|
if run.FinishedAt.Valid {
|
|
return errRunFinalized
|
|
}
|
|
// The run matches the caller's (run_id, chat_id) and is still
|
|
// open, yet the INSERT returned no rows. This is unexpected
|
|
// under write-once-finalize semantics and likely indicates a
|
|
// concurrent delete or unrelated defect; surface it instead of
|
|
// silently masking it as a terminal case.
|
|
return xerrors.Errorf(
|
|
"InsertChatDebugStep returned no rows but run is still active (run_id=%s)",
|
|
params.RunID,
|
|
)
|
|
}
|
|
|
|
// UpdateStep updates an existing debug step and emits a step update event.
|
|
// When a terminal status is set without an explicit FinishedAt, the
|
|
// service auto-fills the timestamp so the stale sweep does not leave
|
|
// terminal rows with finished_at = NULL.
|
|
func (s *Service) UpdateStep(
|
|
ctx context.Context,
|
|
params UpdateStepParams,
|
|
) (database.ChatDebugStep, error) {
|
|
if params.Status.IsTerminal() && params.FinishedAt.IsZero() {
|
|
params.FinishedAt = s.clock.Now()
|
|
}
|
|
step, err := s.db.UpdateChatDebugStep(chatdContext(ctx),
|
|
database.UpdateChatDebugStepParams{
|
|
Status: nullString(string(params.Status)),
|
|
HistoryTipMessageID: sql.NullInt64{},
|
|
AssistantMessageID: nullInt64(params.AssistantMessageID),
|
|
NormalizedRequest: pqtype.NullRawMessage{},
|
|
NormalizedResponse: s.nullJSON(ctx, params.NormalizedResponse),
|
|
Usage: s.nullJSON(ctx, params.Usage),
|
|
Attempts: s.nullJSON(ctx, params.Attempts),
|
|
Error: s.nullJSON(ctx, params.Error),
|
|
Metadata: s.nullJSON(ctx, params.Metadata),
|
|
FinishedAt: nullTime(params.FinishedAt),
|
|
Now: s.clock.Now(),
|
|
ID: params.ID,
|
|
ChatID: params.ChatID,
|
|
})
|
|
if err != nil {
|
|
return database.ChatDebugStep{}, err
|
|
}
|
|
|
|
s.publishEvent(ctx, step.ChatID, EventKindStepUpdate, step.RunID, step.ID)
|
|
return step, nil
|
|
}
|
|
|
|
// TouchStep bumps the step's and its parent run's updated_at timestamps
|
|
// without changing any other fields. This prevents long-running operations
|
|
// (e.g. streaming) from being prematurely swept by FinalizeStale, which
|
|
// first marks runs stale by chat_debug_runs.updated_at and then cascades
|
|
// to steps whose run_id was just finalized.
|
|
func (s *Service) TouchStep(
|
|
ctx context.Context,
|
|
stepID uuid.UUID,
|
|
runID uuid.UUID,
|
|
chatID uuid.UUID,
|
|
) error {
|
|
// Atomically bump both the step and its parent run so
|
|
// FinalizeStale cannot interleave between the two touches.
|
|
return s.db.TouchChatDebugStepAndRun(chatdContext(ctx),
|
|
database.TouchChatDebugStepAndRunParams{
|
|
Now: s.clock.Now(),
|
|
StepID: stepID,
|
|
RunID: runID,
|
|
ChatID: chatID,
|
|
})
|
|
}
|
|
|
|
// TouchRun bumps the run's updated_at timestamp without changing any
|
|
// other fields. Runner-owned debug turns use this while no model step is
|
|
// active, such as requires-action waits. It is a no-op when the service
|
|
// is nil or either ID is uuid.Nil, since debug tracking is optional and
|
|
// callers may not have an open run.
|
|
func (s *Service) TouchRun(ctx context.Context, runID uuid.UUID, chatID uuid.UUID) error {
|
|
if s == nil || runID == uuid.Nil || chatID == uuid.Nil {
|
|
return nil
|
|
}
|
|
return s.db.TouchChatDebugRunUpdatedAt(chatdContext(ctx),
|
|
database.TouchChatDebugRunUpdatedAtParams{
|
|
Now: s.clock.Now(),
|
|
ID: runID,
|
|
ChatID: chatID,
|
|
})
|
|
}
|
|
|
|
// LaunchRunHeartbeat starts a goroutine that periodically touches an
|
|
// open run until done is closed or ctx is canceled. It is a no-op when
|
|
// the service is nil, either ID is uuid.Nil, or done is nil, since debug
|
|
// tracking is optional and callers may not have an open run.
|
|
func (s *Service) LaunchRunHeartbeat(ctx context.Context, runID uuid.UUID, chatID uuid.UUID, done <-chan struct{}) {
|
|
if s == nil || runID == uuid.Nil || chatID == uuid.Nil || done == nil {
|
|
return
|
|
}
|
|
go func() {
|
|
thresholdCh := s.thresholdChan()
|
|
interval := s.heartbeatInterval()
|
|
ticker := s.clock.NewTicker(interval, "chatdebug", "run-heartbeat")
|
|
defer ticker.Stop()
|
|
maybeResetTicker := func() {
|
|
if newInterval := s.heartbeatInterval(); newInterval != interval {
|
|
interval = newInterval
|
|
ticker.Reset(interval, "chatdebug", "run-heartbeat")
|
|
}
|
|
}
|
|
for {
|
|
select {
|
|
case <-ctx.Done():
|
|
return
|
|
case <-done:
|
|
return
|
|
case <-thresholdCh:
|
|
thresholdCh = s.thresholdChan()
|
|
maybeResetTicker()
|
|
case <-ticker.C:
|
|
if err := s.TouchRun(ctx, runID, chatID); err != nil {
|
|
s.log.Debug(ctx, "run heartbeat touch failed",
|
|
slog.Error(err),
|
|
slog.F("run_id", runID),
|
|
)
|
|
}
|
|
maybeResetTicker()
|
|
}
|
|
}
|
|
}()
|
|
}
|
|
|
|
// DeleteByChatID deletes debug data for a chat and emits a delete event.
|
|
// The startedBefore bound scopes deletion to runs created before that
|
|
// instant so that retried cleanup does not remove runs created by a
|
|
// replacement turn that raced ahead of the retry window (for example,
|
|
// an unarchive that fires between the initial archive-cleanup attempt
|
|
// and its retry).
|
|
func (s *Service) DeleteByChatID(
|
|
ctx context.Context,
|
|
chatID uuid.UUID,
|
|
startedBefore time.Time,
|
|
) (int64, error) {
|
|
deleted, err := s.db.DeleteChatDebugDataByChatID(
|
|
chatdContext(ctx),
|
|
database.DeleteChatDebugDataByChatIDParams{
|
|
ChatID: chatID,
|
|
StartedBefore: startedBefore,
|
|
},
|
|
)
|
|
if err != nil {
|
|
return 0, err
|
|
}
|
|
|
|
s.publishEvent(ctx, chatID, EventKindDelete, uuid.Nil, uuid.Nil)
|
|
return deleted, nil
|
|
}
|
|
|
|
// DeleteAfterMessageID deletes debug data newer than the given message.
|
|
// The startedBefore bound scopes deletion to runs created before that
|
|
// instant so that retried cleanup does not remove runs created by a
|
|
// replacement turn that raced ahead of the retry window.
|
|
func (s *Service) DeleteAfterMessageID(
|
|
ctx context.Context,
|
|
chatID uuid.UUID,
|
|
messageID int64,
|
|
startedBefore time.Time,
|
|
) (int64, error) {
|
|
deleted, err := s.db.DeleteChatDebugDataAfterMessageID(
|
|
chatdContext(ctx),
|
|
database.DeleteChatDebugDataAfterMessageIDParams{
|
|
ChatID: chatID,
|
|
MessageID: messageID,
|
|
StartedBefore: startedBefore,
|
|
},
|
|
)
|
|
if err != nil {
|
|
return 0, err
|
|
}
|
|
|
|
s.publishEvent(ctx, chatID, EventKindDelete, uuid.Nil, uuid.Nil)
|
|
return deleted, nil
|
|
}
|
|
|
|
// FinalizeStale finalizes stale in-flight debug rows and emits a broadcast.
|
|
func (s *Service) FinalizeStale(
|
|
ctx context.Context,
|
|
) (database.FinalizeStaleChatDebugRowsRow, error) {
|
|
now := s.clock.Now()
|
|
result, err := s.db.FinalizeStaleChatDebugRows(
|
|
chatdContext(ctx),
|
|
database.FinalizeStaleChatDebugRowsParams{
|
|
Now: now,
|
|
UpdatedBefore: now.Add(-s.staleThreshold()),
|
|
},
|
|
)
|
|
if err != nil {
|
|
return database.FinalizeStaleChatDebugRowsRow{}, err
|
|
}
|
|
|
|
if result.RunsFinalized > 0 || result.StepsFinalized > 0 {
|
|
s.publishEvent(ctx, uuid.Nil, EventKindFinalize, uuid.Nil, uuid.Nil)
|
|
}
|
|
return result, nil
|
|
}
|
|
|
|
// FinalizeRunParams bundles the arguments for FinalizeRun.
|
|
type FinalizeRunParams struct {
|
|
RunID uuid.UUID
|
|
ChatID uuid.UUID
|
|
Status Status
|
|
SeedSummary map[string]any
|
|
// Timeout for the aggregate + update calls. Zero defaults to 5s.
|
|
Timeout time.Duration
|
|
}
|
|
|
|
// FinalizeRun aggregates the run summary, updates the run status, and
|
|
// cleans up the step counter. It detaches from the parent context's
|
|
// cancellation so finalization succeeds even when the request context
|
|
// is already done. Errors are returned but are always safe to ignore;
|
|
// callers that treat debug instrumentation as best-effort can discard
|
|
// them.
|
|
func (s *Service) FinalizeRun(ctx context.Context, p FinalizeRunParams) error {
|
|
timeout := p.Timeout
|
|
if timeout <= 0 {
|
|
timeout = 5 * time.Second
|
|
}
|
|
|
|
finalizeCtx, cancel := context.WithTimeout(
|
|
context.WithoutCancel(ctx), timeout,
|
|
)
|
|
defer cancel()
|
|
|
|
finalSummary := p.SeedSummary
|
|
if aggregated, aggErr := s.AggregateRunSummary(
|
|
finalizeCtx,
|
|
p.RunID,
|
|
p.SeedSummary,
|
|
); aggErr != nil {
|
|
// Non-fatal: proceed with the seed summary.
|
|
s.log.Warn(ctx, "failed to aggregate debug run summary",
|
|
slog.F("chat_id", p.ChatID),
|
|
slog.F("run_id", p.RunID),
|
|
slog.Error(aggErr),
|
|
)
|
|
} else {
|
|
finalSummary = aggregated
|
|
}
|
|
|
|
if _, err := s.UpdateRun(finalizeCtx, UpdateRunParams{
|
|
ID: p.RunID,
|
|
ChatID: p.ChatID,
|
|
Status: p.Status,
|
|
Summary: finalSummary,
|
|
FinishedAt: s.clock.Now(),
|
|
}); err != nil {
|
|
CleanupStepCounter(p.RunID)
|
|
return xerrors.Errorf("update debug run: %w", err)
|
|
}
|
|
CleanupStepCounter(p.RunID)
|
|
return nil
|
|
}
|
|
|
|
// ClassifyError maps a run error to the appropriate debug status.
|
|
// nil → StatusCompleted, context.Canceled → StatusInterrupted,
|
|
// everything else → StatusError. Callers with additional
|
|
// classification rules (e.g. ErrInterrupted)
|
|
// should handle those before falling back to this helper.
|
|
func ClassifyError(err error) Status {
|
|
switch {
|
|
case err == nil:
|
|
return StatusCompleted
|
|
case errors.Is(err, context.Canceled):
|
|
return StatusInterrupted
|
|
default:
|
|
return StatusError
|
|
}
|
|
}
|
|
|
|
func nullUUID(id uuid.UUID) uuid.NullUUID {
|
|
return uuid.NullUUID{UUID: id, Valid: id != uuid.Nil}
|
|
}
|
|
|
|
func nullInt64(v int64) sql.NullInt64 {
|
|
return sql.NullInt64{Int64: v, Valid: v != 0}
|
|
}
|
|
|
|
func nullString(value string) sql.NullString {
|
|
return sql.NullString{String: value, Valid: value != ""}
|
|
}
|
|
|
|
func nullTime(value time.Time) sql.NullTime {
|
|
return sql.NullTime{Time: value, Valid: !value.IsZero()}
|
|
}
|
|
|
|
// jsonClear is a sentinel value that tells nullJSON to emit a valid
|
|
// JSON null (JSONB 'null') instead of SQL NULL. COALESCE treats SQL
|
|
// NULL as "keep existing" but replaces with a non-NULL JSONB value,
|
|
// so passing jsonClear explicitly overwrites a previously set field.
|
|
type jsonClear struct{}
|
|
|
|
// nullJSON marshals value to a NullRawMessage. When value is nil
|
|
// (including typed nils such as `var p *T = nil` whose interface
|
|
// representation carries a type but no value) or marshals to JSON
|
|
// "null", the result is {Valid: false}. Typed nils fall through the
|
|
// `value == nil` guard but produce `[]byte("null")` from
|
|
// json.Marshal, which the `bytes.Equal(data, []byte("null"))` check
|
|
// catches identically. This is intentional for the write-once-finalize
|
|
// pattern: combined with the COALESCE-based UPDATE queries, passing
|
|
// nil (typed or untyped) preserves the existing column value. Fields
|
|
// accumulate monotonically (request -> response -> usage -> error) and
|
|
// never need to be cleared during normal operation. The jsonClear
|
|
// sentinel exists for the sole exception (error retry clearing).
|
|
func (s *Service) nullJSON(ctx context.Context, value any) pqtype.NullRawMessage {
|
|
if value == nil {
|
|
return pqtype.NullRawMessage{}
|
|
}
|
|
// Sentinel: emit a valid JSONB null so COALESCE replaces
|
|
// any previously stored value.
|
|
if _, ok := value.(jsonClear); ok {
|
|
return pqtype.NullRawMessage{
|
|
RawMessage: json.RawMessage("null"),
|
|
Valid: true,
|
|
}
|
|
}
|
|
|
|
data, err := json.Marshal(value)
|
|
if err != nil {
|
|
s.log.Warn(ctx, "failed to marshal chat debug JSON",
|
|
slog.Error(err),
|
|
slog.F("value_type", fmt.Sprintf("%T", value)),
|
|
)
|
|
return pqtype.NullRawMessage{}
|
|
}
|
|
if bytes.Equal(data, []byte("null")) {
|
|
return pqtype.NullRawMessage{}
|
|
}
|
|
|
|
return pqtype.NullRawMessage{RawMessage: data, Valid: true}
|
|
}
|
|
|
|
func (s *Service) publishEvent(
|
|
ctx context.Context,
|
|
chatID uuid.UUID,
|
|
kind EventKind,
|
|
runID uuid.UUID,
|
|
stepID uuid.UUID,
|
|
) {
|
|
if s.pubsub == nil {
|
|
s.log.Debug(ctx,
|
|
"chat debug pubsub unavailable; skipping event",
|
|
slog.F("kind", kind),
|
|
slog.F("chat_id", chatID),
|
|
)
|
|
return
|
|
}
|
|
|
|
event := DebugEvent{
|
|
Kind: kind,
|
|
ChatID: chatID,
|
|
RunID: runID,
|
|
StepID: stepID,
|
|
}
|
|
data, err := json.Marshal(event)
|
|
if err != nil {
|
|
s.log.Warn(ctx, "failed to marshal chat debug event",
|
|
slog.Error(err),
|
|
slog.F("kind", kind),
|
|
slog.F("chat_id", chatID),
|
|
)
|
|
return
|
|
}
|
|
|
|
channel := PubsubChannel(chatID)
|
|
if err := s.pubsub.Publish(channel, data); err != nil {
|
|
s.log.Warn(ctx, "failed to publish chat debug event",
|
|
slog.Error(err),
|
|
slog.F("channel", channel),
|
|
slog.F("kind", kind),
|
|
slog.F("chat_id", chatID),
|
|
)
|
|
}
|
|
}
|