Removes production code in `coderd/x/chatd` that nothing reaches, the tests that only existed to exercise it, and option/config fields that callers filled in but chatd never read. No runtime behaviour changes; the only edits outside `coderd/x/chatd` drop the now-nonexistent fields at their three call sites (`coderd/coderd.go`, `coderd/exp_chats.go`, `coderd/coderdtest`). Net: 42 files, +147 / -1065. Verification method for every item: `golang.org/x/tools/cmd/deadcode` (with and without `-test`) rooted at `cmd/coder`, `enterprise/cmd/coder` and every package that imports chatd; a `go/packages` pass counting `TypesInfo.Uses` per declared type, const, var, field and method across the same roots, split into test and non-test use sites, plus a read-vs-write split for struct fields; `rg` over the whole repo including `site/src` for json tag names; then `go build ./...`, `go vet ./coderd/x/chatd/...`, `golangci-lint run ./coderd/x/chatd/... ./coderd/ ./coderd/coderdtest/`, and `go test ./coderd/x/chatd/...` (all 20 packages) locally. <details> <summary>Evidence per deletion</summary> | Removed | What it was | Why unused | |---|---|---| | `subagent.go` `resolveConfiguredModelOverride`, `parseModelOverride`, `parsedModelOverride` (~135 lines) + `TestResolveConfiguredModelOverride_*`, `TestParseModelOverride` | raw-string model override resolver | Only callers were its two tests; production resolves overrides through `resolveOrganizationModelOverride` (`title_override.go`, `compaction_override.go`, `subagent.go`). | | `chatd.go` `removeSkillIndexMessages`, `isSkillIndexMessage` + `TestSkillIndexRefreshReplacesStaleAliases` | skill-index refresh helpers | Zero production references; the test simulated a refresh path no production code performs. | | `chatd.go` `chatMessage`, `newChatMessage`, `withCreatedBy`, `appendMessageFields`, `BuildSingleChatMessageInsertParams` (~90 lines) | insert-params builder | Exported, but every caller was a `chatd_test` file (6 sites). Replaced by a 25-line `singleChatMessageInsertParams` helper in `chatd_helpers_test.go`. | | `chatd.go` `Server.closeInflightAdmission` | one-line wrapper around `inflightClosed.Store(true)` | Single call site (`Close`); inlined. | | `chatd.go` `Server.workerID`, `instructionLookupTimeout`, `pendingChatAcquireInterval`, `maxChatsPerAcquire`, `chatHeartbeatInterval` | Server struct fields | Assigned in `New` and never read anywhere (the values reach the worker through `chatWorkerOptions`, not these fields). | | `Config.InstructionLookupTimeout`, `homeInstructionLookupTimeout`, `coderd.Options.ChatdInstructionLookupTimeout`, `coderdtest.Options.ChatdInstructionLookupTimeout` | timeout plumbing | The value only landed in the dead `Server.instructionLookupTimeout`; no CLI flag sets it and nothing reads it. | | `PromoteQueuedOptions.CreatedBy` | option field | Set by `coderd/exp_chats.go` and three tests, never read in chatd (promotion keeps the queued row's `created_by`). | | `SubmitToolResultsOptions.DynamicTools` | option field | Set by `coderd/exp_chats.go` (which re-read `chat.DynamicTools` just to pass it) and three tests, never read in chatd. | | `attachments.go` `buildAssistantPartsForPersist` (whole file) | assistant part builder | Stale copy left by #26270; production uses `buildAssistantParts` in `message_conversion.go`. Its six tests were retargeted to the live function (now `message_conversion_internal_test.go`) and a dead-copy assertion in `subagent_internal_test.go` was dropped. | | `chatloop.RunOptions` (94 lines) | old loop options struct | Only two of 23 fields were ever populated, for one internal call to `prepareMessagesForRequest`, which now takes `model, logger` directly and no longer returns the ignored canonical slice. `GenerateAssistantOptions` / `ExecuteLocalToolsOptions` are the live types. | | `chatloop.ErrDynamicToolCall`, `ErrStopAfterTool`, `PendingToolCall`, `PersistedStep.PendingDynamicToolCalls` | dynamic-tool exit protocol | Zero references outside their declarations (one stale comment in `chatdebug/service.go` updated). | | `chatloop.CompactionOptions.Persist`, `.OnError` | callbacks | `Persist` had zero references; `OnError` was set only by two tests that asserted it was never invoked. | | `chatloop` `stepResult.shouldContinue` + its computation | flag | Written once, never read. | | `chatloop.StateWaiting` | metric label const | Zero references. | | `generation.go` `generationOutcome.InsertedMessages`, `runnerActionMessage` | outcome field | Built on every commit, never read. | | `chatstate` `ExecutionState.IsArchived`, `AllOwnershipStates`, `OwnershipState`/`StateU`/`StateO`, `TransitionAcquire` | state-machine leftovers | Zero references (the ownership consts became dead once `AllOwnershipStates` went). | | `chatstate` `ExecutionState.QueueNonEmpty` | predicate | Only referenced by the test asserting its own table; the sibling `IsRunnable` check in that test is kept. | | `chatdebug.Attempt.RetryClassification`, `.RetryDelayMs` | json fields | Never assigned anywhere, `omitempty`, and `retry_classification`/`retry_delay_ms` do not appear in `site/src` or `typesGenerated.ts`. | | `chatdebug.RunContext.Kind`, `StepContext.HistoryTipMessageID`, `RecorderOptions.Provider`/`.Model`, `stepHandle.opts`/`.usage`/`.err` | context/handle fields | Written (from `chatd.go`, `chatloop/compaction.go`, `modelcall.go`, `recorder.go`) and never read. The DB run row keeps its own `kind` column. | | `chatprovider.MergeProviderAPIKeys` + `TestMergeProviderAPIKeys_PreservesProviderRegions` | key merge | Superseded by `ResolveUserProviderKeys`, whose region test already covers the same behaviour. | | `chatprovider` `providerCreationError` | one-line `xerrors.Errorf` wrapper | Single call site; inlined with identical message. | | `chatretry.IsRetryable` | classifier wrapper | Zero references. | | `computeruse.IsTool` + `TestComputerUseTool` | one-line forwarder to `fantasyopenai.IsComputerUseTool` | No production caller; two test assertions now call the fantasy function directly, and the dropped test only asserted the library's own constructor. | | `chatprompt.ToolResultToPart` | one-line forwarder to `codersdk.ChatMessageToolResult` | Single call site; inlined. | | `subagent_catalog.go` `formatSubagentDefinitions` | forwarder with `nil` overrides | Single call site; inlined. | | `worker.go` `chatWorker.chatWorkerID` | getter | Only tests called it; they read `opts.WorkerID` directly now. | | `chattest.AnthropicErrorResponse`, `OpenAIServerErrorResponse`, `ChatMessageWithParts` (`messages.go` removed) | test fixtures | Zero references. | | `chattest` `AnthropicRequest.Options`, `OpenAIRequest.Options` | `json:",inline"` map fields | `encoding/json` has no inline tag (the TODO on each said so); never populated, never read. | </details> <details> <summary>Considered and kept</summary> - `chatstate.CreateChat` (test-only thin wrapper over `insertChat`, production uses `CreateChatWithID`): documented package API with 25 test call sites; not worth the churn. - `chatprompt.MarshalContent` / `MarshalToolResult`: test-only, but they deliberately produce legacy-format DB rows so the read path's legacy handling stays covered. - `chatstate.AllowedExecutionTransitionsFrom` / `AllowedInputStates` / `AllowedExecutionTransitionOutputs`, `chatdebug.StepFromContext`, `WithClock`, `WithStaleThreshold`, `WaitUntilIdleForTest`, `PublishBuffer.BufferedChannels`, `chattool.NewWorkspaceMCPTool`: test seams that tests actively use to reach otherwise unexported state. - `AgentCapacityLimiter` interface (one production implementation): tests inject fakes. - `applyHookResultMessages` / `appendHookResultMessages`: a named pair over one placement enum, both used. - Nil-receiver guards on `*Metrics`, `*Result`, `*streamSyncPoller`, `*runnerDebugTurn`: nil is a supported "disabled" value on those paths. - Tests flagged by heuristics as assertion-light (`TestNopMetrics_DoesNotPanic`, `TestStepHandleFinish_NilHandle`, `TestRetry_OnRetryNilDoesNotPanic`, the `chatstate` synthetic-cancellation tests that assert through helpers): they cover real nil-safety or delegate assertions to helpers; kept. </details> <details> <summary>Out-of-scope findings (not changed here)</summary> - `chatdebug.Service.FinalizeStale` has had no production caller since #26270 removed the background sweep loop; only its tests call it, and comments in `chatd_debug.go` and `database/querier.go` still describe the loop. Deleting it would erase the implementation a fix needs, so it is left for a decision. - `coderd_chatd_chats` (gauge) and `coderd_chatd_stream_buffer_dropped_total` (counter) are registered and documented in `docs/admin/integrations/prometheus.md` but never set or incremented in production; `Metrics.RecordStreamBufferDropped` is test-only. Removing them changes the exported metric set and the docs, so they are left alone. </details> ## Declined review findings None yet. > Xum (AI agent) opened this PR on behalf of @ibetitsmike.
277 lines
9.3 KiB
Go
277 lines
9.3 KiB
Go
package chatloop
|
|
|
|
import (
|
|
"context"
|
|
"errors"
|
|
|
|
"charm.land/fantasy"
|
|
"github.com/prometheus/client_golang/prometheus"
|
|
"github.com/prometheus/client_golang/prometheus/promauto"
|
|
|
|
"github.com/coder/coder/v2/coderd/x/chatd/chaterror"
|
|
)
|
|
|
|
const (
|
|
metricsNamespace = "coderd"
|
|
metricsSubsystem = "chatd"
|
|
|
|
// Label values for Chats.
|
|
StateStreaming = "streaming"
|
|
|
|
// Label values for CompactionTotal.
|
|
CompactionResultSuccess = "success"
|
|
CompactionResultError = "error"
|
|
CompactionResultTimeout = "timeout"
|
|
)
|
|
|
|
// Metrics holds Prometheus metrics for the chatd subsystem.
|
|
type Metrics struct {
|
|
Chats *prometheus.GaugeVec
|
|
MessageCount *prometheus.HistogramVec
|
|
PromptSizeBytes *prometheus.HistogramVec
|
|
ToolResultSizeBytes *prometheus.HistogramVec
|
|
ToolResultTruncatedTotal *prometheus.CounterVec
|
|
ToolErrorsTotal *prometheus.CounterVec
|
|
TTFTSeconds *prometheus.HistogramVec
|
|
CompactionTotal *prometheus.CounterVec
|
|
StepsTotal *prometheus.CounterVec
|
|
StreamRetriesTotal *prometheus.CounterVec
|
|
StreamBufferDroppedTotal prometheus.Counter
|
|
FindToolsCallsTotal prometheus.Counter
|
|
FindToolsEmptyTotal prometheus.Counter
|
|
FindToolsMatchCount prometheus.Histogram
|
|
FindToolsActivationsTotal prometheus.Counter
|
|
}
|
|
|
|
// NewMetrics creates a new Metrics instance registered with the
|
|
// given registerer.
|
|
func NewMetrics(reg prometheus.Registerer) *Metrics {
|
|
factory := promauto.With(reg)
|
|
return &Metrics{
|
|
Chats: factory.NewGaugeVec(prometheus.GaugeOpts{
|
|
Namespace: metricsNamespace,
|
|
Subsystem: metricsSubsystem,
|
|
Name: "chats",
|
|
Help: "Number of chats being processed, by state.",
|
|
}, []string{"state"}),
|
|
MessageCount: factory.NewHistogramVec(prometheus.HistogramOpts{
|
|
Namespace: metricsNamespace,
|
|
Subsystem: metricsSubsystem,
|
|
Name: "message_count",
|
|
Help: "Number of messages in the prompt per LLM request.",
|
|
Buckets: prometheus.ExponentialBuckets(1, 2, 11), // 1, 2, 4, ..., 1024
|
|
}, []string{"provider", "model"}),
|
|
PromptSizeBytes: factory.NewHistogramVec(prometheus.HistogramOpts{
|
|
Namespace: metricsNamespace,
|
|
Subsystem: metricsSubsystem,
|
|
Name: "prompt_size_bytes",
|
|
Help: "Estimated byte size of the prompt per LLM request.",
|
|
Buckets: prometheus.ExponentialBuckets(1024, 4, 10), // 1KB .. 256MB
|
|
}, []string{"provider", "model"}),
|
|
ToolResultSizeBytes: factory.NewHistogramVec(prometheus.HistogramOpts{
|
|
Namespace: metricsNamespace,
|
|
Subsystem: metricsSubsystem,
|
|
Name: "tool_result_size_bytes",
|
|
Help: "Size in bytes of each tool execution result.",
|
|
Buckets: prometheus.ExponentialBuckets(64, 4, 9), // 64B .. 4MB
|
|
}, []string{"provider", "model", "tool_name"}),
|
|
ToolResultTruncatedTotal: factory.NewCounterVec(prometheus.CounterOpts{
|
|
Namespace: metricsNamespace,
|
|
Subsystem: metricsSubsystem,
|
|
Name: "tool_result_truncated_total",
|
|
Help: "Total tool results truncated to fit the model context window.",
|
|
}, []string{"provider", "model", "tool_name"}),
|
|
ToolErrorsTotal: factory.NewCounterVec(prometheus.CounterOpts{
|
|
Namespace: metricsNamespace,
|
|
Subsystem: metricsSubsystem,
|
|
Name: "tool_errors_total",
|
|
Help: "Total tool calls that returned an error result.",
|
|
}, []string{"provider", "model", "tool_name"}),
|
|
TTFTSeconds: factory.NewHistogramVec(prometheus.HistogramOpts{
|
|
Namespace: metricsNamespace,
|
|
Subsystem: metricsSubsystem,
|
|
Name: "ttft_seconds",
|
|
Help: "Time-to-first-token: wall time from LLM request to first streamed chunk.",
|
|
Buckets: []float64{0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60},
|
|
}, []string{"provider", "model"}),
|
|
CompactionTotal: factory.NewCounterVec(prometheus.CounterOpts{
|
|
Namespace: metricsNamespace,
|
|
Subsystem: metricsSubsystem,
|
|
Name: "compaction_total",
|
|
Help: "Total compaction outcomes (only recorded when compaction was triggered or failed).",
|
|
}, []string{"provider", "model", "result"}),
|
|
StepsTotal: factory.NewCounterVec(prometheus.CounterOpts{
|
|
Namespace: metricsNamespace,
|
|
Subsystem: metricsSubsystem,
|
|
Name: "steps_total",
|
|
Help: "Total agentic loop steps across all chats.",
|
|
}, []string{"provider", "model"}),
|
|
StreamRetriesTotal: factory.NewCounterVec(prometheus.CounterOpts{
|
|
Namespace: metricsNamespace,
|
|
Subsystem: metricsSubsystem,
|
|
Name: "stream_retries_total",
|
|
Help: "Total LLM stream retries.",
|
|
}, []string{"provider", "model", "kind"}),
|
|
FindToolsCallsTotal: factory.NewCounter(prometheus.CounterOpts{
|
|
Namespace: metricsNamespace,
|
|
Subsystem: metricsSubsystem,
|
|
Name: "find_tools_calls_total",
|
|
Help: "Total find_tools calls.",
|
|
}),
|
|
FindToolsEmptyTotal: factory.NewCounter(prometheus.CounterOpts{
|
|
Namespace: metricsNamespace,
|
|
Subsystem: metricsSubsystem,
|
|
Name: "find_tools_empty_total",
|
|
Help: "Total find_tools calls with no matches.",
|
|
}),
|
|
FindToolsMatchCount: factory.NewHistogram(prometheus.HistogramOpts{
|
|
Namespace: metricsNamespace,
|
|
Subsystem: metricsSubsystem,
|
|
Name: "find_tools_match_count",
|
|
Help: "Number of matches returned by find_tools calls.",
|
|
Buckets: prometheus.LinearBuckets(0, 2, 11),
|
|
}),
|
|
FindToolsActivationsTotal: factory.NewCounter(prometheus.CounterOpts{
|
|
Namespace: metricsNamespace,
|
|
Subsystem: metricsSubsystem,
|
|
Name: "find_tools_activations_total",
|
|
Help: "Total deferred tool activations returned by find_tools.",
|
|
}),
|
|
StreamBufferDroppedTotal: factory.NewCounter(prometheus.CounterOpts{
|
|
Namespace: metricsNamespace,
|
|
Subsystem: metricsSubsystem,
|
|
Name: "stream_buffer_dropped_total",
|
|
Help: "Number of chat stream buffer events dropped due to the per-chat buffer cap.",
|
|
}),
|
|
}
|
|
}
|
|
|
|
// NopMetrics returns a Metrics instance that discards all data.
|
|
// Useful for tests and when metrics collection is not desired.
|
|
func NopMetrics() *Metrics {
|
|
return NewMetrics(prometheus.NewRegistry())
|
|
}
|
|
|
|
// RecordCompaction classifies and records a compaction attempt.
|
|
// It is a no-op when m is nil.
|
|
func (m *Metrics) RecordCompaction(provider, model string, compacted bool, err error) {
|
|
if m == nil {
|
|
return
|
|
}
|
|
switch {
|
|
case err != nil && errors.Is(err, context.DeadlineExceeded):
|
|
m.CompactionTotal.WithLabelValues(provider, model, CompactionResultTimeout).Inc()
|
|
case err != nil && errors.Is(err, context.Canceled):
|
|
// User interruption, not a compaction failure.
|
|
return
|
|
case err != nil:
|
|
m.CompactionTotal.WithLabelValues(provider, model, CompactionResultError).Inc()
|
|
case compacted:
|
|
m.CompactionTotal.WithLabelValues(provider, model, CompactionResultSuccess).Inc()
|
|
// !compacted && err == nil means threshold not reached -- not
|
|
// recorded.
|
|
}
|
|
}
|
|
|
|
// RecordStreamRetry increments stream_retries_total. The caller
|
|
// must obtain classified via chaterror.Classify (non-empty Kind).
|
|
// No-op when m is nil.
|
|
func (m *Metrics) RecordStreamRetry(provider, model string, classified chaterror.ClassifiedError) {
|
|
if m == nil {
|
|
return
|
|
}
|
|
m.StreamRetriesTotal.WithLabelValues(
|
|
provider,
|
|
model,
|
|
string(classified.Kind),
|
|
).Inc()
|
|
}
|
|
|
|
// RecordToolError increments tool_errors_total for the given
|
|
// tool. No-op when m is nil.
|
|
func (m *Metrics) RecordToolError(provider, model, toolLabel string) {
|
|
if m == nil {
|
|
return
|
|
}
|
|
m.ToolErrorsTotal.WithLabelValues(provider, model, toolLabel).Inc()
|
|
}
|
|
|
|
// RecordToolResultTruncated increments tool_result_truncated_total for
|
|
// the given tool. No-op when m is nil. An empty tool label is
|
|
// normalized to "unknown" to match the other tool metrics.
|
|
func (m *Metrics) RecordToolResultTruncated(provider, model, toolLabel string) {
|
|
if m == nil {
|
|
return
|
|
}
|
|
if toolLabel == "" {
|
|
toolLabel = "unknown"
|
|
}
|
|
m.ToolResultTruncatedTotal.WithLabelValues(provider, model, toolLabel).Inc()
|
|
}
|
|
|
|
// RecordStreamBufferDropped increments stream_buffer_dropped_total
|
|
// once per dropped event. No-op when m is nil.
|
|
func (m *Metrics) RecordStreamBufferDropped() {
|
|
if m == nil {
|
|
return
|
|
}
|
|
m.StreamBufferDroppedTotal.Inc()
|
|
}
|
|
|
|
// EstimatePromptSize returns a cheap byte-size estimate of a
|
|
// fantasy prompt by summing the text content lengths of all
|
|
// message parts. This avoids JSON marshaling overhead.
|
|
func EstimatePromptSize(messages []fantasy.Message) int {
|
|
var size int
|
|
for _, msg := range messages {
|
|
for _, part := range msg.Content {
|
|
size += ContentPartSize(part)
|
|
}
|
|
}
|
|
return size
|
|
}
|
|
|
|
// ContentPartSize returns the byte length of a MessagePart's
|
|
// primary text or data field.
|
|
func ContentPartSize(part fantasy.MessagePart) int {
|
|
switch p := part.(type) {
|
|
case fantasy.TextPart:
|
|
return len(p.Text)
|
|
case fantasy.ReasoningPart:
|
|
return len(p.Text)
|
|
case fantasy.FilePart:
|
|
return len(p.Data)
|
|
case fantasy.ToolCallPart:
|
|
return len(p.Input)
|
|
case fantasy.ToolResultPart:
|
|
return toolResultOutputSize(p.Output)
|
|
default:
|
|
return 0
|
|
}
|
|
}
|
|
|
|
// ToolResultSize returns the byte length of a
|
|
// ToolResultContent's primary text or data field.
|
|
func ToolResultSize(r fantasy.ToolResultContent) int {
|
|
return toolResultOutputSize(r.Result)
|
|
}
|
|
|
|
func toolResultOutputSize(output fantasy.ToolResultOutputContent) int {
|
|
if output == nil {
|
|
return 0
|
|
}
|
|
switch v := output.(type) {
|
|
case fantasy.ToolResultOutputContentText:
|
|
return len(v.Text)
|
|
case fantasy.ToolResultOutputContentError:
|
|
if v.Error != nil {
|
|
return len(v.Error.Error())
|
|
}
|
|
return 0
|
|
case fantasy.ToolResultOutputContentMedia:
|
|
return len(v.Data)
|
|
default:
|
|
return 0
|
|
}
|
|
}
|