Files
coder/coderd/x/chatd/modelcall.go
T
J. Scott Miller e988ddb11d feat: trace chatd provider stages and count only output parts as first token (#29770)
**TL;DR:** Traces the model provider call: a `stream` stage, a nested
`time_to_first_token` stage, and one `provider_attempt` per HTTP round
trip, labelled with provider type and model. Behavior change:
`coderd_chatd_ttft_seconds` now counts only a real output part as the
first token, so error, warning, and finish parts no longer count.

Instruments the provider call.

**Stream and time to first token.** `GenerateAssistant` records a
`stream` stage from stream open to close and a nested
`time_to_first_token` stage. The TTFT window closes once, on the first
streamed output part: any part other than warnings, finish, or error,
including start markers such as `text_start`. A closed window observes
the stage and the pre-existing `coderd_chatd_ttft_seconds` with the same
elapsed time. A failed open, an error part, or a release without an
output part ends the TTFT span with the error (a silence timeout is
classified as such) and observes nothing. The `stream` span ends once,
with the error `GenerateAssistant` returns, so content-filter and
no-output attempts end errored. Failed `stream` windows are observed
like successful ones. The attempt is released before the stream span
ends, so the TTFT window always falls inside its stream. The compaction
summary stream passes a nil tracer, so it records neither.

Without #29771, production emits only `provider_attempt`; #29771 passes
the tracer to `GenerateAssistant`.

**Provider attempts.** The provider HTTP transport is wrapped in
`stageSpanRoundTripper`, which emits one `provider_attempt` stage per
round trip, so retries each get a span. The span closes on response
headers; a status >= 400 marks it errored without changing what the
client sees.

`resolveModelCall` resolves the `StageModel` (wire provider, configured
`provider_type`, model, effective reasoning effort) and threads it to
the transport, so span attributes and labels match the call.
`provider_type` is read from `ai_providers.type`, so types outside the
fantasy set (copilot) keep their configured name. `chatd.New` builds the
tracer with the server clock.

**Behavior change to `coderd_chatd_ttft_seconds`:** only an output part
counts as a first token. Error parts no longer close the window as a
success, and warnings and finish parts are skipped; the help text says
so.

`StageSpan.End` returns the span's elapsed window, which `ttft_seconds`
observes. Callers pass `Stages` and `StageModel` through
`GenerateAssistantOptions`; a nil tracer discards (TTFT still records
`ttft_seconds`). `Server` gains the `stages` tracer and a
`TracerProvider` option, wired from `coderd`.

<details>
<summary>Review decisions</summary>

- Start markers close the TTFT window: they are the first sign the model
is producing output, and waiting for a delta would fold hidden reasoning
time into TTFT.
- Failed `stream` and `provider_attempt` windows are observed: that time
was spent. TTFT is the exception because a first token that never
arrived has no duration.
- `http_host` was dropped from `provider_attempt`; every request goes
through the local AI Gateway transport, so it was constant.
- The advisor tool's nested model call is intentionally not instrumented
as `stream`/TTFT.

</details>

Supersedes #28970 and #28971.

🤖 This PR was generated by Coder Agents on behalf of @jscottmiller.
2026-09-30 11:53:58 -05:00

259 lines
8.9 KiB
Go

package chatd
import (
"context"
"charm.land/fantasy"
"golang.org/x/xerrors"
"cdr.dev/slog/v3"
"github.com/coder/coder/v2/coderd/database"
"github.com/coder/coder/v2/coderd/database/dbauthz"
"github.com/coder/coder/v2/coderd/util/ptr"
"github.com/coder/coder/v2/coderd/x/chatd/chatdebug"
"github.com/coder/coder/v2/coderd/x/chatd/chatloop"
"github.com/coder/coder/v2/coderd/x/chatd/chatprovider"
"github.com/coder/coder/v2/codersdk"
)
const defaultChatMaxOutputTokens = int64(32_000)
// fixedModelCall selects a provider/model pair that has no config row of its
// own (computer use).
type fixedModelCall struct {
providerType string
modelName string
callConfig codersdk.ChatModelCallConfig
}
type modelCallSpec struct {
// purpose labels resolver logs only; it does not affect call behavior.
purpose string
chat database.Chat
explicitConfig *database.ChatModelConfig
fixedModel *fixedModelCall
// requestedEffort overrides the config's default reasoning effort,
// clamped to the config's max.
requestedEffort *string
// maxOutputTokens overrides the config's output-token cap and the
// package default.
maxOutputTokens *int64
// chatdScopedRoute resolves the route with chatd scope. Deployment-wide
// override models must route for user-owned chats regardless of the
// caller's actor.
chatdScopedRoute bool
buildOptions modelBuildOptions
}
func chatRequestedEffort(chat database.Chat) *string {
if !chat.LastReasoningEffort.Valid {
return nil
}
return new(string(chat.LastReasoningEffort.ChatReasoningEffort))
}
// modelCallConfigParseError lets the advisor distinguish malformed options
// from route and client failures when deciding whether to fall back.
type modelCallConfigParseError struct{ err error }
func (e modelCallConfigParseError) Error() string {
return "parse model call config: " + e.err.Error()
}
func (e modelCallConfigParseError) Unwrap() error { return e.err }
type resolvedModelCall struct {
model chatprovider.Model
dbConfig database.ChatModelConfig
callConfig codersdk.ChatModelCallConfig
providerOptions fantasy.ProviderOptions
resolvedProvider string
resolvedModel string
resolvedEffort string
route aiGatewayModelRoute
debugEnabled bool
}
// stageModel uses the configured provider type rather than
// resolvedProvider, which is inferred from the model name for types such
// as copilot.
func (r resolvedModelCall) stageModel() chatloop.StageModel {
stageModel := chatloop.StageModel{
ProviderType: string(r.route.Provider.Type),
Model: r.resolvedModel,
Effort: r.resolvedEffort,
}
if r.model.Valid() {
stageModel.Provider = r.model.Provider()
}
return stageModel
}
// resolveModelCall is the single pipeline from a spec to a ready model
// client plus the call metadata flows need.
func (p *Server) resolveModelCall(ctx context.Context, spec modelCallSpec) (resolvedModelCall, error) {
out := resolvedModelCall{}
var modelName string
var configOptions []byte
switch {
case spec.fixedModel != nil:
modelName = spec.fixedModel.modelName
case spec.explicitConfig != nil:
out.dbConfig = *spec.explicitConfig
modelName = out.dbConfig.Model
configOptions = out.dbConfig.Options
default:
dbConfig, err := p.resolveModelConfig(ctx, spec.chat)
if err != nil {
return resolvedModelCall{}, xerrors.Errorf("resolve model config: %w", err)
}
if !dbConfig.Enabled {
return resolvedModelCall{}, xerrors.Errorf("chat model config %s is disabled", dbConfig.ID)
}
out.dbConfig = dbConfig
modelName = dbConfig.Model
configOptions = dbConfig.Options
}
// clientCallConfig drives client construction; out.callConfig drives
// per-call option derivation and comes from the chat model for computer
// use, whose fixed model has no config of its own.
clientCallConfig, err := parseModelConfigOptions(configOptions)
if err != nil {
return resolvedModelCall{}, modelCallConfigParseError{err: err}
}
if spec.fixedModel != nil {
out.callConfig = spec.fixedModel.callConfig
} else {
out.callConfig = clientCallConfig
}
switch {
case spec.maxOutputTokens != nil:
out.callConfig.MaxOutputTokens = spec.maxOutputTokens
case out.callConfig.MaxOutputTokens == nil:
out.callConfig.MaxOutputTokens = ptr.Ref(defaultChatMaxOutputTokens)
}
routeCtx := ctx
if spec.chatdScopedRoute {
//nolint:gocritic // Deployment-wide override models need chatd-scoped provider reads for user-owned chats.
routeCtx = dbauthz.AsChatd(ctx)
}
if spec.fixedModel != nil {
out.route, err = p.resolveModelRouteForProviderType(routeCtx, spec.chat.OwnerID, spec.fixedModel.providerType)
} else {
out.route, err = p.resolveModelRouteForConfig(routeCtx, spec.chat.OwnerID, out.dbConfig)
}
if err != nil {
return resolvedModelCall{}, err
}
// The resolved identity feeds metadata, logs, and debug labels. The
// client is constructed with the configured model string so gateway
// validation sees the name exactly as configured.
out.resolvedProvider, out.resolvedModel, err = chatprovider.ResolveModelWithProviderHint(
modelName,
out.route.ModelProviderHint,
)
if err != nil {
return resolvedModelCall{}, xerrors.Errorf("resolve model metadata: %w", err)
}
debugSvc := p.debugService()
out.debugEnabled = debugSvc != nil && debugSvc.IsEnabled(ctx, spec.chat.ID, spec.chat.OwnerID)
// Coder-scale effort, before provider mapping can change or drop it.
if effectiveEffort := chatprovider.ResolveReasoningEffort(spec.requestedEffort, out.callConfig.ReasoningEffort); effectiveEffort != nil {
out.resolvedEffort = *effectiveEffort
}
buildOpts := spec.buildOptions
buildOpts.RecordHTTP = out.debugEnabled
buildOpts.StageModel = out.stageModel()
model, err := p.newModel(ctx, modelClientRequest{
Chat: spec.chat,
ModelName: modelName,
UserAgent: chatprovider.UserAgent(),
ExtraHeaders: chatprovider.CoderHeaders(spec.chat),
CallConfig: clientCallConfig,
}, out.route, buildOpts)
if err != nil {
return resolvedModelCall{}, xerrors.Errorf("create model: %w", err)
}
model = p.withThinkingDropBlock(model, out.route.Provider.ID, clientCallConfig, spec.chat.ID)
if out.debugEnabled {
model = model.WithLanguageModel(chatdebug.WrapModel(model.LanguageModel(), debugSvc, chatdebug.RecorderOptions{
ChatID: spec.chat.ID,
OwnerID: spec.chat.OwnerID,
}))
}
out.model = model
// Only the derived provider options see the coerced config: callConfig
// stays as configured so provider substitution (computer use rerouting
// to direct OpenAI) starts from the original value.
out.providerOptions = chatprovider.ProviderOptionsForCall(
out.model,
coerceBedrockReasoningSummary(out.route.Provider.Type, modelName, out.callConfig),
spec.requestedEffort,
)
p.logger.Debug(ctx, "resolved model call",
slog.F("purpose", spec.purpose),
slog.F("chat_id", spec.chat.ID),
slog.F("provider", out.resolvedProvider),
slog.F("model", out.resolvedModel),
slog.F("reasoning_effort", out.resolvedEffort),
slog.F("debug_enabled", out.debugEnabled),
)
return out, nil
}
func (r resolvedModelCall) newCall() fantasy.Call {
return fantasy.Call{
ProviderOptions: r.providerOptions,
MaxOutputTokens: r.callConfig.MaxOutputTokens,
Temperature: r.callConfig.Temperature,
TopP: r.callConfig.TopP,
TopK: r.callConfig.TopK,
PresencePenalty: r.callConfig.PresencePenalty,
FrequencyPenalty: r.callConfig.FrequencyPenalty,
}
}
// compactionSummaryCall follows the resolved call template, except summaries
// must not call tools. Streaming avoids the Anthropic SDK's non-streaming
// duration limit, while the explicit cap prevents adaptive thinking from
// exhausting fantasy's smaller provider default before producing summary text.
// The cap gets 50% headroom because the summary must fit reasoning plus
// summary text in one response; a low configured chat cap could otherwise
// truncate the summary that replaces pruned history. The headroom stays
// modest because a cap configured at a provider's output ceiling overshoots
// it; that residual overshoot is an accepted tradeoff. GenerateCompaction
// bounds the increased cap by the remaining context window at trigger time.
func compactionSummaryCall(resolved resolvedModelCall) fantasy.Call {
call := resolved.newCall()
toolChoiceNone := fantasy.ToolChoiceNone
call.ToolChoice = &toolChoiceNone
if call.MaxOutputTokens != nil {
call.MaxOutputTokens = ptr.Ref(*call.MaxOutputTokens * 3 / 2)
}
return call
}
func (r resolvedModelCall) newObjectCall(schemaName, schemaDescription string, maxOutputTokens int64) fantasy.ObjectCall {
return fantasy.ObjectCall{
SchemaName: schemaName,
SchemaDescription: schemaDescription,
MaxOutputTokens: ptr.Ref(maxOutputTokens),
Temperature: r.callConfig.Temperature,
TopP: r.callConfig.TopP,
TopK: r.callConfig.TopK,
PresencePenalty: r.callConfig.PresencePenalty,
FrequencyPenalty: r.callConfig.FrequencyPenalty,
ProviderOptions: r.providerOptions,
}
}