Files
coder/coderd/x/chatd/capacity.go
T
J. Scott Miller 7a9442014e feat: add chatd lifecycle stage tracer (#29769)
**TL;DR:** Adds a stage tracer that turns each chat lifecycle stage into
both an OpenTelemetry span and a Prometheus histogram observation, so
traces and metrics always agree. The metrics are only registered with
the `chat-stage-metrics` experiment. Nothing calls the tracer until the
next PRs.

Adds `chatloop.StageTracer`, which emits each chat lifecycle stage as an
OpenTelemetry span and a `coderd_chatd_stage_duration_seconds{stage,
scope, chat_kind}` observation from one `End` call, so trace and metric
durations cannot disagree. The provider-bound stages (`stream`,
`provider_attempt`) are also observed on
`coderd_chatd_model_stage_duration_seconds{stage, provider_type,
chat_kind, model}`, the only family carrying `provider_type` and
`model`, and only for turn-scoped stages. Time to first token per model
stays on the existing `coderd_chatd_ttft_seconds`. Nothing calls the
tracer yet; the next PRs instrument the provider call and the runner.

- The families are registered only when the `chat-stage-metrics`
experiment is enabled (safe and known, so
`--experiments=chat-stage-metrics` or `--experiments=*` turns it on).
Spans are emitted regardless. `NewMetrics` leaves the families off;
`chatd.New` opts in from the experiment, and a test asserts the
experiment decides whether they reach the server's registry.
- `Stage`, `Scope`, `ChatKind`, and `StageAnomaly` are typed string
constants. `scope` separates latency attributable to a prompt (`turn`)
from detached `background` work; `chat_kind` is `root` or `subagent`,
empty without a known chat.
- `StageModel` carries the wire `provider`, `provider_type`, `model`,
and `reasoning_effort`. All are span attributes; only `provider_type`
and `model` are metric labels, on the model family. No family carries an
organization label.
- Observed stages: `chat_turn`, `queue_wait`, `acquisition`,
`mcp_connect`, `stream`, `time_to_first_token`, `provider_attempt`,
`tool_call`, `commit`, `retry_backoff`. `generation_step`, `prepare`,
`thinking`, and `compaction` are span-only. A test keeps the stage sets
and help text consistent.
- API: `Start` / `StartRootAt` for live spans (span timestamps come from
the tracer clock), `Record` for stages reconstructed from timestamps
(the scope comes from the context). `End` is idempotent;
`EndWithoutObservation` ends the span without a sample. A nil tracer,
provider, or metrics discards.
- `coderd_chatd_stage_anomalies_total{reason}` counts observations
dropped, adjusted, or inconsistent (`negative_elapsed`,
`inverted_window`, `missing_timestamp`, `future_start`, `stale_anchor`).
Span-only stages never feed it.
- Buckets are 12 round edges (`0.1 0.25 0.5 1 2.5 5 10 30 60 300 1800
3600`).
- Moves the body of the `for` loop in `StartGeneration` into
`runGenerationStep` so the step has a function boundary for a deferred
stage end. Control flow and error wrapping are unchanged.

<details>
<summary>Series cost (approximate, whole stack)</summary>

A histogram label set is 12 edges + `+Inf` + `_sum` + `_count`: 15
series. Per replica, each `chat_kind` costs about 150 stage series and 4
turn outcome counters; each `(chat_kind, model)` costs 30 (2 stages x
15). Both chat kinds on `M` models: about `308 + 60*M` per replica, plus
background-scope stage series.

Dogfood `main`, 3 replicas: ~3.4k `coderd_*` series per replica (~10.1k
total).

| Models used per replica | Added series (3 replicas) | vs. baseline |
|---:|---:|---:|
| 1 | ~1.1k | +11% |
| 5 | ~1.8k | +18% |
| 10 | ~2.7k | +27% |
| 35 | ~7.2k | +72% |

Most deployments run four or fewer models, so a typical replica adds
roughly 370 to 550 series. The experimental turn accounting in #29772
would add 88 more per replica (11 categories x 4 outcomes x 2 chat
kinds). Series appear lazily and do not grow with organizations.

</details>

<details>
<summary>Review decisions</summary>

- Experiment gate only; no deployment flag or metric levels. If the
experiment graduates, the families become always on like the other chatd
families; a flag is added then only if dogfood shows a cardinality
problem.
- Organization is on neither metrics nor spans, and `effort` is
span-only; `provider_type` (configured type), not the wire `provider`,
is the model-family label.
- `time_to_first_token` is not on the model family; per-model TTFT is
`ttft_seconds`, which observes the same measured window.
- The model family records only turn-scoped stages.
- Nil guards are kept so a nil or zero tracer is always safe to call.
- HTTP attribute keys stay snake_case like the rest of the chatd span
attributes.

</details>

Supersedes #28968 and #28969.

🤖 This PR was generated by Coder Agents on behalf of @jscottmiller.
2026-09-30 11:53:57 -05:00

86 lines
2.9 KiB
Go

package chatd
import (
"context"
"github.com/google/uuid"
"github.com/prometheus/client_golang/prometheus"
"github.com/coder/coder/v2/coderd/database"
"github.com/coder/coder/v2/coderd/x/chatd/chatloop"
)
type capacityMetrics struct {
active *prometheus.GaugeVec
queued *prometheus.GaugeVec
}
func newCapacityMetrics(registerer prometheus.Registerer) *capacityMetrics {
m := &capacityMetrics{
active: prometheus.NewGaugeVec(prometheus.GaugeOpts{
Namespace: "coderd",
Subsystem: "chatd",
Name: "agents_active",
Help: "Deployment-wide number of chats holding a concurrent-agent capacity slot. Every replica reports the same database-derived value; aggregate with max, not sum.",
}, []string{"pool"}),
queued: prometheus.NewGaugeVec(prometheus.GaugeOpts{
Namespace: "coderd",
Subsystem: "chatd",
Name: "agents_queued_for_capacity",
Help: "Deployment-wide number of chats waiting for a concurrent-agent capacity slot. Every replica reports the same database-derived value; aggregate with max, not sum.",
}, []string{"pool"}),
}
registerer.MustRegister(m.active, m.queued)
return m
}
func (w *chatWorker) capacityMetricsLoop(ctx context.Context) {
ticker := w.opts.Clock.NewTicker(w.opts.CapacityMetricsInterval, "chatworker", "capacity-metrics")
defer ticker.Stop()
for {
select {
case <-ticker.C:
case <-ctx.Done():
return
}
w.refreshCapacityMetrics(ctx)
}
}
func (w *chatWorker) refreshCapacityMetrics(ctx context.Context) {
active, err := w.opts.Store.CountChatCapacityActiveByPool(ctx, database.CountChatCapacityActiveByPoolParams{
ExcludeChatID: uuid.Nil,
StaleSeconds: w.opts.HeartbeatStaleSeconds,
})
if err != nil {
if ctx.Err() == nil {
w.opts.Logger.Warn(ctx, "chatworker count active capacity chats failed", slogError(err))
}
return
}
limits, capped := w.opts.AgentCapacityLimiter.Limits()
var queuedRoot, queuedSubagent int64
if capped && (active.ActiveRootCount >= limits.Root || active.ActiveSubagentCount >= limits.Subagent) {
queued, err := w.opts.Store.CountChatCapacityQueuedByPool(ctx, w.opts.HeartbeatStaleSeconds)
if err != nil {
if ctx.Err() == nil {
w.opts.Logger.Warn(ctx, "chatworker count queued capacity chats failed", slogError(err))
}
return
}
if active.ActiveRootCount >= limits.Root {
queuedRoot = queued.QueuedRootCount
}
if active.ActiveSubagentCount >= limits.Subagent {
queuedSubagent = queued.QueuedSubagentCount
}
}
metrics := w.opts.CapacityMetrics
metrics.active.WithLabelValues(string(chatloop.ChatKindRoot)).Set(float64(active.ActiveRootCount))
metrics.active.WithLabelValues(string(chatloop.ChatKindSubagent)).Set(float64(active.ActiveSubagentCount))
metrics.queued.WithLabelValues(string(chatloop.ChatKindRoot)).Set(float64(queuedRoot))
metrics.queued.WithLabelValues(string(chatloop.ChatKindSubagent)).Set(float64(queuedSubagent))
}