**TL;DR:** Traces the model provider call: a `stream` stage, a nested `time_to_first_token` stage, and one `provider_attempt` per HTTP round trip, labelled with provider type and model. Behavior change: `coderd_chatd_ttft_seconds` now counts only a real output part as the first token, so error, warning, and finish parts no longer count. Instruments the provider call. **Stream and time to first token.** `GenerateAssistant` records a `stream` stage from stream open to close and a nested `time_to_first_token` stage. The TTFT window closes once, on the first streamed output part: any part other than warnings, finish, or error, including start markers such as `text_start`. A closed window observes the stage and the pre-existing `coderd_chatd_ttft_seconds` with the same elapsed time. A failed open, an error part, or a release without an output part ends the TTFT span with the error (a silence timeout is classified as such) and observes nothing. The `stream` span ends once, with the error `GenerateAssistant` returns, so content-filter and no-output attempts end errored. Failed `stream` windows are observed like successful ones. The attempt is released before the stream span ends, so the TTFT window always falls inside its stream. The compaction summary stream passes a nil tracer, so it records neither. Without #29771, production emits only `provider_attempt`; #29771 passes the tracer to `GenerateAssistant`. **Provider attempts.** The provider HTTP transport is wrapped in `stageSpanRoundTripper`, which emits one `provider_attempt` stage per round trip, so retries each get a span. The span closes on response headers; a status >= 400 marks it errored without changing what the client sees. `resolveModelCall` resolves the `StageModel` (wire provider, configured `provider_type`, model, effective reasoning effort) and threads it to the transport, so span attributes and labels match the call. `provider_type` is read from `ai_providers.type`, so types outside the fantasy set (copilot) keep their configured name. `chatd.New` builds the tracer with the server clock. **Behavior change to `coderd_chatd_ttft_seconds`:** only an output part counts as a first token. Error parts no longer close the window as a success, and warnings and finish parts are skipped; the help text says so. `StageSpan.End` returns the span's elapsed window, which `ttft_seconds` observes. Callers pass `Stages` and `StageModel` through `GenerateAssistantOptions`; a nil tracer discards (TTFT still records `ttft_seconds`). `Server` gains the `stages` tracer and a `TracerProvider` option, wired from `coderd`. <details> <summary>Review decisions</summary> - Start markers close the TTFT window: they are the first sign the model is producing output, and waiting for a delta would fold hidden reasoning time into TTFT. - Failed `stream` and `provider_attempt` windows are observed: that time was spent. TTFT is the exception because a first token that never arrived has no duration. - `http_host` was dropped from `provider_attempt`; every request goes through the local AI Gateway transport, so it was constant. - The advisor tool's nested model call is intentionally not instrumented as `stream`/TTFT. </details> Supersedes #28970 and #28971. 🤖 This PR was generated by Coder Agents on behalf of @jscottmiller.
259 lines
8.9 KiB
Go
259 lines
8.9 KiB
Go
package chatd
|
|
|
|
import (
|
|
"context"
|
|
|
|
"charm.land/fantasy"
|
|
"golang.org/x/xerrors"
|
|
|
|
"cdr.dev/slog/v3"
|
|
"github.com/coder/coder/v2/coderd/database"
|
|
"github.com/coder/coder/v2/coderd/database/dbauthz"
|
|
"github.com/coder/coder/v2/coderd/util/ptr"
|
|
"github.com/coder/coder/v2/coderd/x/chatd/chatdebug"
|
|
"github.com/coder/coder/v2/coderd/x/chatd/chatloop"
|
|
"github.com/coder/coder/v2/coderd/x/chatd/chatprovider"
|
|
"github.com/coder/coder/v2/codersdk"
|
|
)
|
|
|
|
const defaultChatMaxOutputTokens = int64(32_000)
|
|
|
|
// fixedModelCall selects a provider/model pair that has no config row of its
|
|
// own (computer use).
|
|
type fixedModelCall struct {
|
|
providerType string
|
|
modelName string
|
|
callConfig codersdk.ChatModelCallConfig
|
|
}
|
|
|
|
type modelCallSpec struct {
|
|
// purpose labels resolver logs only; it does not affect call behavior.
|
|
purpose string
|
|
chat database.Chat
|
|
explicitConfig *database.ChatModelConfig
|
|
fixedModel *fixedModelCall
|
|
// requestedEffort overrides the config's default reasoning effort,
|
|
// clamped to the config's max.
|
|
requestedEffort *string
|
|
// maxOutputTokens overrides the config's output-token cap and the
|
|
// package default.
|
|
maxOutputTokens *int64
|
|
// chatdScopedRoute resolves the route with chatd scope. Deployment-wide
|
|
// override models must route for user-owned chats regardless of the
|
|
// caller's actor.
|
|
chatdScopedRoute bool
|
|
buildOptions modelBuildOptions
|
|
}
|
|
|
|
func chatRequestedEffort(chat database.Chat) *string {
|
|
if !chat.LastReasoningEffort.Valid {
|
|
return nil
|
|
}
|
|
return new(string(chat.LastReasoningEffort.ChatReasoningEffort))
|
|
}
|
|
|
|
// modelCallConfigParseError lets the advisor distinguish malformed options
|
|
// from route and client failures when deciding whether to fall back.
|
|
type modelCallConfigParseError struct{ err error }
|
|
|
|
func (e modelCallConfigParseError) Error() string {
|
|
return "parse model call config: " + e.err.Error()
|
|
}
|
|
|
|
func (e modelCallConfigParseError) Unwrap() error { return e.err }
|
|
|
|
type resolvedModelCall struct {
|
|
model chatprovider.Model
|
|
dbConfig database.ChatModelConfig
|
|
callConfig codersdk.ChatModelCallConfig
|
|
providerOptions fantasy.ProviderOptions
|
|
resolvedProvider string
|
|
resolvedModel string
|
|
resolvedEffort string
|
|
route aiGatewayModelRoute
|
|
debugEnabled bool
|
|
}
|
|
|
|
// stageModel uses the configured provider type rather than
|
|
// resolvedProvider, which is inferred from the model name for types such
|
|
// as copilot.
|
|
func (r resolvedModelCall) stageModel() chatloop.StageModel {
|
|
stageModel := chatloop.StageModel{
|
|
ProviderType: string(r.route.Provider.Type),
|
|
Model: r.resolvedModel,
|
|
Effort: r.resolvedEffort,
|
|
}
|
|
if r.model.Valid() {
|
|
stageModel.Provider = r.model.Provider()
|
|
}
|
|
return stageModel
|
|
}
|
|
|
|
// resolveModelCall is the single pipeline from a spec to a ready model
|
|
// client plus the call metadata flows need.
|
|
func (p *Server) resolveModelCall(ctx context.Context, spec modelCallSpec) (resolvedModelCall, error) {
|
|
out := resolvedModelCall{}
|
|
|
|
var modelName string
|
|
var configOptions []byte
|
|
switch {
|
|
case spec.fixedModel != nil:
|
|
modelName = spec.fixedModel.modelName
|
|
case spec.explicitConfig != nil:
|
|
out.dbConfig = *spec.explicitConfig
|
|
modelName = out.dbConfig.Model
|
|
configOptions = out.dbConfig.Options
|
|
default:
|
|
dbConfig, err := p.resolveModelConfig(ctx, spec.chat)
|
|
if err != nil {
|
|
return resolvedModelCall{}, xerrors.Errorf("resolve model config: %w", err)
|
|
}
|
|
if !dbConfig.Enabled {
|
|
return resolvedModelCall{}, xerrors.Errorf("chat model config %s is disabled", dbConfig.ID)
|
|
}
|
|
out.dbConfig = dbConfig
|
|
modelName = dbConfig.Model
|
|
configOptions = dbConfig.Options
|
|
}
|
|
|
|
// clientCallConfig drives client construction; out.callConfig drives
|
|
// per-call option derivation and comes from the chat model for computer
|
|
// use, whose fixed model has no config of its own.
|
|
clientCallConfig, err := parseModelConfigOptions(configOptions)
|
|
if err != nil {
|
|
return resolvedModelCall{}, modelCallConfigParseError{err: err}
|
|
}
|
|
if spec.fixedModel != nil {
|
|
out.callConfig = spec.fixedModel.callConfig
|
|
} else {
|
|
out.callConfig = clientCallConfig
|
|
}
|
|
switch {
|
|
case spec.maxOutputTokens != nil:
|
|
out.callConfig.MaxOutputTokens = spec.maxOutputTokens
|
|
case out.callConfig.MaxOutputTokens == nil:
|
|
out.callConfig.MaxOutputTokens = ptr.Ref(defaultChatMaxOutputTokens)
|
|
}
|
|
|
|
routeCtx := ctx
|
|
if spec.chatdScopedRoute {
|
|
//nolint:gocritic // Deployment-wide override models need chatd-scoped provider reads for user-owned chats.
|
|
routeCtx = dbauthz.AsChatd(ctx)
|
|
}
|
|
if spec.fixedModel != nil {
|
|
out.route, err = p.resolveModelRouteForProviderType(routeCtx, spec.chat.OwnerID, spec.fixedModel.providerType)
|
|
} else {
|
|
out.route, err = p.resolveModelRouteForConfig(routeCtx, spec.chat.OwnerID, out.dbConfig)
|
|
}
|
|
if err != nil {
|
|
return resolvedModelCall{}, err
|
|
}
|
|
|
|
// The resolved identity feeds metadata, logs, and debug labels. The
|
|
// client is constructed with the configured model string so gateway
|
|
// validation sees the name exactly as configured.
|
|
out.resolvedProvider, out.resolvedModel, err = chatprovider.ResolveModelWithProviderHint(
|
|
modelName,
|
|
out.route.ModelProviderHint,
|
|
)
|
|
if err != nil {
|
|
return resolvedModelCall{}, xerrors.Errorf("resolve model metadata: %w", err)
|
|
}
|
|
|
|
debugSvc := p.debugService()
|
|
out.debugEnabled = debugSvc != nil && debugSvc.IsEnabled(ctx, spec.chat.ID, spec.chat.OwnerID)
|
|
|
|
// Coder-scale effort, before provider mapping can change or drop it.
|
|
if effectiveEffort := chatprovider.ResolveReasoningEffort(spec.requestedEffort, out.callConfig.ReasoningEffort); effectiveEffort != nil {
|
|
out.resolvedEffort = *effectiveEffort
|
|
}
|
|
|
|
buildOpts := spec.buildOptions
|
|
buildOpts.RecordHTTP = out.debugEnabled
|
|
buildOpts.StageModel = out.stageModel()
|
|
model, err := p.newModel(ctx, modelClientRequest{
|
|
Chat: spec.chat,
|
|
ModelName: modelName,
|
|
UserAgent: chatprovider.UserAgent(),
|
|
ExtraHeaders: chatprovider.CoderHeaders(spec.chat),
|
|
CallConfig: clientCallConfig,
|
|
}, out.route, buildOpts)
|
|
if err != nil {
|
|
return resolvedModelCall{}, xerrors.Errorf("create model: %w", err)
|
|
}
|
|
model = p.withThinkingDropBlock(model, out.route.Provider.ID, clientCallConfig, spec.chat.ID)
|
|
|
|
if out.debugEnabled {
|
|
model = model.WithLanguageModel(chatdebug.WrapModel(model.LanguageModel(), debugSvc, chatdebug.RecorderOptions{
|
|
ChatID: spec.chat.ID,
|
|
OwnerID: spec.chat.OwnerID,
|
|
}))
|
|
}
|
|
out.model = model
|
|
|
|
// Only the derived provider options see the coerced config: callConfig
|
|
// stays as configured so provider substitution (computer use rerouting
|
|
// to direct OpenAI) starts from the original value.
|
|
out.providerOptions = chatprovider.ProviderOptionsForCall(
|
|
out.model,
|
|
coerceBedrockReasoningSummary(out.route.Provider.Type, modelName, out.callConfig),
|
|
spec.requestedEffort,
|
|
)
|
|
|
|
p.logger.Debug(ctx, "resolved model call",
|
|
slog.F("purpose", spec.purpose),
|
|
slog.F("chat_id", spec.chat.ID),
|
|
slog.F("provider", out.resolvedProvider),
|
|
slog.F("model", out.resolvedModel),
|
|
slog.F("reasoning_effort", out.resolvedEffort),
|
|
slog.F("debug_enabled", out.debugEnabled),
|
|
)
|
|
return out, nil
|
|
}
|
|
|
|
func (r resolvedModelCall) newCall() fantasy.Call {
|
|
return fantasy.Call{
|
|
ProviderOptions: r.providerOptions,
|
|
MaxOutputTokens: r.callConfig.MaxOutputTokens,
|
|
Temperature: r.callConfig.Temperature,
|
|
TopP: r.callConfig.TopP,
|
|
TopK: r.callConfig.TopK,
|
|
PresencePenalty: r.callConfig.PresencePenalty,
|
|
FrequencyPenalty: r.callConfig.FrequencyPenalty,
|
|
}
|
|
}
|
|
|
|
// compactionSummaryCall follows the resolved call template, except summaries
|
|
// must not call tools. Streaming avoids the Anthropic SDK's non-streaming
|
|
// duration limit, while the explicit cap prevents adaptive thinking from
|
|
// exhausting fantasy's smaller provider default before producing summary text.
|
|
// The cap gets 50% headroom because the summary must fit reasoning plus
|
|
// summary text in one response; a low configured chat cap could otherwise
|
|
// truncate the summary that replaces pruned history. The headroom stays
|
|
// modest because a cap configured at a provider's output ceiling overshoots
|
|
// it; that residual overshoot is an accepted tradeoff. GenerateCompaction
|
|
// bounds the increased cap by the remaining context window at trigger time.
|
|
func compactionSummaryCall(resolved resolvedModelCall) fantasy.Call {
|
|
call := resolved.newCall()
|
|
toolChoiceNone := fantasy.ToolChoiceNone
|
|
call.ToolChoice = &toolChoiceNone
|
|
if call.MaxOutputTokens != nil {
|
|
call.MaxOutputTokens = ptr.Ref(*call.MaxOutputTokens * 3 / 2)
|
|
}
|
|
return call
|
|
}
|
|
|
|
func (r resolvedModelCall) newObjectCall(schemaName, schemaDescription string, maxOutputTokens int64) fantasy.ObjectCall {
|
|
return fantasy.ObjectCall{
|
|
SchemaName: schemaName,
|
|
SchemaDescription: schemaDescription,
|
|
MaxOutputTokens: ptr.Ref(maxOutputTokens),
|
|
Temperature: r.callConfig.Temperature,
|
|
TopP: r.callConfig.TopP,
|
|
TopK: r.callConfig.TopK,
|
|
PresencePenalty: r.callConfig.PresencePenalty,
|
|
FrequencyPenalty: r.callConfig.FrequencyPenalty,
|
|
ProviderOptions: r.providerOptions,
|
|
}
|
|
}
|