Files
coder/coderd/x/chatd/chatadvisor/runtime.go
T
Michael Suchacz f5092aaa4a feat: make chat limits configurable deployment options (#28927)
Adds deployment options for the chat limits that shape turn behavior and
API acceptance, replacing hardcoded constants in `coderd/x/chatd` and
the chat API. Defaults are unchanged.

| Option | Default | Limit |
|---|---|---|
| `CODER_CHAT_MAX_STEPS_PER_TURN` | 1200 | steps (model responses) in a
chat turn |
| `CODER_CHAT_MAX_GENERATION_RETRIES` | 25 | consecutive retries after a
model generation fails with a transient error |
| `CODER_CHAT_MAX_QUEUED_MESSAGES_PER_CHAT` | 20 | messages that can be
queued in a chat |
| `CODER_CHAT_MAX_ATTACHMENTS_PER_CHAT` | 50 | files linked to a chat;
linking more deletes the earliest uploads |
| `CODER_CHAT_MAX_PROMPT_BYTES` | 131072 | system prompt, plan mode
instructions, and each user's custom prompt |
| `CODER_CHAT_MAX_CONCURRENT_RECORDING_UPLOADS` | 25 | desktop
recordings each server stores at the same time |

`codersdk.DefaultChat*` constants remain the single source of truth.
Values below 1 fail startup validation, and each option's description
ends with "Must be at least 1." `chatd.New` fills any zero field in
`chatd.Config.Limits` with its default once, `LimitsFromConfig` is a
plain conversion, and the code that receives a limit as a parameter (the
`chatstate` queue and file link checks, the turn step check,
`chatretry.Retry`, and `chatadvisor.NewRuntime`) returns an error for a
limit below 1 instead of choosing a default or disabling the limit. The
options are documented in the generated configuration reference and in
the platform controls docs.

`CODER_CHAT_MAX_GENERATION_RETRIES` counts consecutive retries, and the
count resets after each successful step. The default allows 25 retries
(26 attempts) where the old constants allowed 24. The same limit now
applies to each nested advisor model call and to background title,
summary, and turn status label generation, which previously used fixed
`chatretry` budgets.

Chat creation and tool result bodies keep the fixed 256 KiB limit from
main. The prompt-saving endpoints accept bodies up to twice the prompt
limit, for JSON escaping and characters that sanitization removes, but
never less than 256 KiB, so small prompt limits still return the 400
length error. At the default this equals the 256 KiB limit on main.

Four settings from earlier revisions were removed, and the fixed limits
from main are back:

- `CODER_CHAT_MAX_TOOL_OUTPUT_BYTES`: the agent already keeps a 16 KiB
head and a 16 KiB tail of command output, so raising it had no effect
and lowering it cut the tail.
- `CODER_CHAT_MAX_DYNAMIC_TOOLS_PER_CHAT`: tool definitions share the
256 KiB chat creation body limit, so raising the count did little.
- `CODER_CHAT_DEBUG_MAX_BODY_BYTES`: one setting covered two unrelated
debug caps.
- `CODER_CHAT_DEBUG_MAX_TEXT_RUNES`: the name did not say what it
applied to.

Root and subagent concurrency caps stay fixed because they are licensing
behavior, and `MaxChatFileSizeBytes` stays fixed because the web UI
hardcodes the generated constant for client-side checks.
`codersdk.MaxChatFileIDs` is removed; the web UI's "Attachment expired"
tooltip no longer quotes the number because the cap is now per
deployment, and it says that attaching a file beyond the limit
permanently deletes the chat's earliest-uploaded attachments.

## Review record

| Round | Reviewed head | Result | Disposition |
|---|---|---|---|
| Codex 1 and 2 (Sep 2 to 3) | `7f10a1572f8`, `8ee09f8fa3b`,
`197bb78d5b` | 10 findings, then OK on `197bb78d5b` | all threads
resolved |
| Codex 3 (Sep 17) | `e4efac249a5` (pre-rebase) | 4 findings | advisor
retry limit confirmed and fixed in `9e462b7b3d0` (rebased from
`67341e458e8`); ARCHITECTURE.md TODOs added; JSON-escaping body sizing
and streamed-argument rune cap declined as pre-existing design (evidence
in threads) |
| Codex 4 (Sep 18) | `9e462b7b3d0` | 6 findings | recording evicted at
attachment cap 1 confirmed and fixed in `5b61895decb`; `Limits` field
docs and zero-value fallback documented; prompt retro-enforcement,
generated multi-sentence lines, and tool-output truncation metadata
declined as pre-existing design (evidence in threads) |
| Codex 5 (Sep 18) | `5b61895decb` | 1 finding | over-cap batch of
already-linked files accepted after lowering the cap: confirmed and
fixed in `f44063e74e5` |
| mafredri (Sep 23) | `f44063e74e5` | 24 threads, changes requested |
all fixed in `05e3e3d622a`, `c1f78e09e8b`, `709ddb67977`, `d0587a49829`,
`ca4347b24de`, and `1f6c9e2a46f`, then answered and resolved; the
ARCHITECTURE.md text was written at the author's direction and approved
by the author |
| Sibling sweep (Sep 24) | `1f6c9e2a46f` | author-requested audit of all
24 mafredri threads for sibling issues, plus a test refactor | fixed in
`37eaf07d34d` through `1f14ac041b7`: the advisor retry limit,
`chatretry.Retry`, and the turn step check reject values below 1 instead
of disabling retries or the limit; limit tests run against both the
default and configured values set through `chatd.Config` or
`DeploymentValues`; the expired-attachment tooltip is concrete. A
required coder-frontend-review run (8 roles in one agent, CD re-run
once) found a stale story comment, removed. |
| Codex 6 (Sep 27) | `a71383e3a31` | OK, no findings | requested after
the rebase at the author's direction, past the six-pass review budget |

The branch was rebased onto main `4851c7b60c8` on Sep 27; the previous
head was `91b84000adb`, and the earlier SHAs in the review record
predate the rebase. Conflicts in the `chatstate` transitions and
`CreateChat` keep both main's `InitialStatus` and this PR's
`MaxFileLinks`, and main's rewrite of `advisor_internal_test.go` in
#28754 keeps main's version plus this PR's limit tests. Two tests added
on main needed the new retry arguments:
`TestGenerateStructuredTitleWithUsage_FallsBackToTextWhenToolChoiceRejected`
and `TestAdvisorRunStreamsReasoningDeltasSeparatelyFromAdvice`.
Validation on `a71383e3a31`: the pre-commit hook (generation,
golangci-lint, build), the full `coderd/x/chatd/...` suite (20
packages), the chat tests in `codersdk`, `coderd`, and
`enterprise/coderd`, the CLI help tests, race-detector runs of the
`chatd`, `chatstate`, `chatadvisor`, and `chatretry` packages, and `go
build` plus `go vet` on each of the 27 commits. No remote UAT was run.

> Xum acted on behalf of @ibetitsmike for this pull request.
2026-09-28 09:25:01 +00:00

165 lines
5.1 KiB
Go

package chatadvisor
import (
"sync/atomic"
"time"
"charm.land/fantasy"
fantasyopenai "charm.land/fantasy/providers/openai"
"golang.org/x/xerrors"
)
// RuntimeConfig configures a single advisor runtime instance.
type RuntimeConfig struct {
Model fantasy.LanguageModel
// CallTemplate's provider options are cloned for each nested call.
CallTemplate fantasy.Call
MaxUsesPerRun int
MaxOutputTokens int64
StreamSilenceTimeout time.Duration
// MaxRetries is the maximum number of retries after a transient
// provider error within one advisor call. It must be positive.
MaxRetries int
}
// Runtime executes nested, tool-less advisor runs against the configured
// language model.
//
// Each Runtime instance is scoped to a single outer chat run. The
// MaxUsesPerRun counter increments on every successful advisor call and
// is never reset, so callers must construct a fresh Runtime (via
// NewRuntime) for each outer run. There is intentionally no Reset method:
// the per-run quota is a safety bound on a single run, not a rolling
// window.
type Runtime struct {
cfg RuntimeConfig
used atomic.Int64
}
// NewRuntime validates and normalizes advisor runtime configuration.
func NewRuntime(cfg RuntimeConfig) (*Runtime, error) {
if cfg.Model == nil {
return nil, xerrors.New("advisor model is required")
}
if cfg.MaxUsesPerRun <= 0 {
return nil, xerrors.New("advisor max uses per run must be positive")
}
if cfg.MaxOutputTokens <= 0 {
return nil, xerrors.New("advisor max output tokens must be positive")
}
if cfg.MaxRetries < 1 {
return nil, xerrors.New("advisor max retries must be positive")
}
if cfg.CallTemplate.MaxOutputTokens != nil &&
*cfg.CallTemplate.MaxOutputTokens != cfg.MaxOutputTokens {
return nil, xerrors.Errorf(
"advisor call template max output tokens (%d) must match runtime max output tokens (%d)",
*cfg.CallTemplate.MaxOutputTokens,
cfg.MaxOutputTokens,
)
}
normalized := cfg
normalized.CallTemplate.ProviderOptions = cloneProviderOptions(cfg.CallTemplate.ProviderOptions)
maxOutputTokens := cfg.MaxOutputTokens
normalized.CallTemplate.MaxOutputTokens = &maxOutputTokens
return &Runtime{cfg: normalized}, nil
}
// cloneProviderOptions returns a copy of opts with pointer entries for
// known, in-place mutated provider option types replaced by shallow struct
// copies, so a nested advisor call that disables Store does so on its own
// copy rather than the parent run's entry. Value fields such as Metadata
// and Include remain shared; callers that need true deep-copy semantics
// must handle those fields explicitly.
func cloneProviderOptions(opts fantasy.ProviderOptions) fantasy.ProviderOptions {
if opts == nil {
return nil
}
cloned := make(fantasy.ProviderOptions, len(opts))
for key, value := range opts {
switch typed := value.(type) {
case *fantasyopenai.ResponsesProviderOptions:
if typed == nil {
cloned[key] = value
continue
}
copied := *typed
cloned[key] = &copied
default:
cloned[key] = value
}
}
return cloned
}
// resetProviderOptionsForNestedCall forces Store off so ephemeral advisor
// calls leave no stored response behind on the provider. It mutates opts
// in place, so it must be called on a cloned map, never on options shared
// with the parent run.
func resetProviderOptionsForNestedCall(opts fantasy.ProviderOptions) {
for _, value := range opts {
if typed, ok := value.(*fantasyopenai.ResponsesProviderOptions); ok && typed != nil {
storeDisabled := false
typed.Store = &storeDisabled
}
}
}
// RemainingUses reports how many advisor calls are still available for the
// current runtime.
func (rt *Runtime) RemainingUses() int {
if rt == nil || rt.cfg.MaxUsesPerRun <= 0 {
return 0
}
remaining := int64(rt.cfg.MaxUsesPerRun) - rt.used.Load()
if remaining < 0 {
return 0
}
return int(remaining)
}
// MaxOutputTokens reports the resolved output-token cap applied to each
// advisor call. NewRuntime validates that this value is positive and that
// it matches ModelConfig.MaxOutputTokens when both are set, so the
// accessor always returns the value the runtime will actually send.
func (rt *Runtime) MaxOutputTokens() int64 {
if rt == nil {
return 0
}
return rt.cfg.MaxOutputTokens
}
// ProviderOptions reports the resolved provider options applied to each
// advisor call. NewRuntime clones the supplied options so the returned
// map reflects what nested calls will actually receive; callers must not
// mutate the map or its entries.
func (rt *Runtime) ProviderOptions() fantasy.ProviderOptions {
if rt == nil {
return nil
}
return rt.cfg.CallTemplate.ProviderOptions
}
func (rt *Runtime) tryAcquire() bool {
for {
used := rt.used.Load()
if used >= int64(rt.cfg.MaxUsesPerRun) {
return false
}
if rt.used.CompareAndSwap(used, used+1) {
return true
}
}
}
// release returns a previously acquired use to the pool. Callers must
// invoke this at most once per successful tryAcquire when the advisor
// call did not complete successfully, so a transient provider failure
// does not permanently consume quota for the run.
func (rt *Runtime) release() {
rt.used.Add(-1)
}