Adds deployment options for the chat limits that shape turn behavior and API acceptance, replacing hardcoded constants in `coderd/x/chatd` and the chat API. Defaults are unchanged. | Option | Default | Limit | |---|---|---| | `CODER_CHAT_MAX_STEPS_PER_TURN` | 1200 | steps (model responses) in a chat turn | | `CODER_CHAT_MAX_GENERATION_RETRIES` | 25 | consecutive retries after a model generation fails with a transient error | | `CODER_CHAT_MAX_QUEUED_MESSAGES_PER_CHAT` | 20 | messages that can be queued in a chat | | `CODER_CHAT_MAX_ATTACHMENTS_PER_CHAT` | 50 | files linked to a chat; linking more deletes the earliest uploads | | `CODER_CHAT_MAX_PROMPT_BYTES` | 131072 | system prompt, plan mode instructions, and each user's custom prompt | | `CODER_CHAT_MAX_CONCURRENT_RECORDING_UPLOADS` | 25 | desktop recordings each server stores at the same time | `codersdk.DefaultChat*` constants remain the single source of truth. Values below 1 fail startup validation, and each option's description ends with "Must be at least 1." `chatd.New` fills any zero field in `chatd.Config.Limits` with its default once, `LimitsFromConfig` is a plain conversion, and the code that receives a limit as a parameter (the `chatstate` queue and file link checks, the turn step check, `chatretry.Retry`, and `chatadvisor.NewRuntime`) returns an error for a limit below 1 instead of choosing a default or disabling the limit. The options are documented in the generated configuration reference and in the platform controls docs. `CODER_CHAT_MAX_GENERATION_RETRIES` counts consecutive retries, and the count resets after each successful step. The default allows 25 retries (26 attempts) where the old constants allowed 24. The same limit now applies to each nested advisor model call and to background title, summary, and turn status label generation, which previously used fixed `chatretry` budgets. Chat creation and tool result bodies keep the fixed 256 KiB limit from main. The prompt-saving endpoints accept bodies up to twice the prompt limit, for JSON escaping and characters that sanitization removes, but never less than 256 KiB, so small prompt limits still return the 400 length error. At the default this equals the 256 KiB limit on main. Four settings from earlier revisions were removed, and the fixed limits from main are back: - `CODER_CHAT_MAX_TOOL_OUTPUT_BYTES`: the agent already keeps a 16 KiB head and a 16 KiB tail of command output, so raising it had no effect and lowering it cut the tail. - `CODER_CHAT_MAX_DYNAMIC_TOOLS_PER_CHAT`: tool definitions share the 256 KiB chat creation body limit, so raising the count did little. - `CODER_CHAT_DEBUG_MAX_BODY_BYTES`: one setting covered two unrelated debug caps. - `CODER_CHAT_DEBUG_MAX_TEXT_RUNES`: the name did not say what it applied to. Root and subagent concurrency caps stay fixed because they are licensing behavior, and `MaxChatFileSizeBytes` stays fixed because the web UI hardcodes the generated constant for client-side checks. `codersdk.MaxChatFileIDs` is removed; the web UI's "Attachment expired" tooltip no longer quotes the number because the cap is now per deployment, and it says that attaching a file beyond the limit permanently deletes the chat's earliest-uploaded attachments. ## Review record | Round | Reviewed head | Result | Disposition | |---|---|---|---| | Codex 1 and 2 (Sep 2 to 3) | `7f10a1572f8`, `8ee09f8fa3b`, `197bb78d5b` | 10 findings, then OK on `197bb78d5b` | all threads resolved | | Codex 3 (Sep 17) | `e4efac249a5` (pre-rebase) | 4 findings | advisor retry limit confirmed and fixed in `9e462b7b3d0` (rebased from `67341e458e8`); ARCHITECTURE.md TODOs added; JSON-escaping body sizing and streamed-argument rune cap declined as pre-existing design (evidence in threads) | | Codex 4 (Sep 18) | `9e462b7b3d0` | 6 findings | recording evicted at attachment cap 1 confirmed and fixed in `5b61895decb`; `Limits` field docs and zero-value fallback documented; prompt retro-enforcement, generated multi-sentence lines, and tool-output truncation metadata declined as pre-existing design (evidence in threads) | | Codex 5 (Sep 18) | `5b61895decb` | 1 finding | over-cap batch of already-linked files accepted after lowering the cap: confirmed and fixed in `f44063e74e5` | | mafredri (Sep 23) | `f44063e74e5` | 24 threads, changes requested | all fixed in `05e3e3d622a`, `c1f78e09e8b`, `709ddb67977`, `d0587a49829`, `ca4347b24de`, and `1f6c9e2a46f`, then answered and resolved; the ARCHITECTURE.md text was written at the author's direction and approved by the author | | Sibling sweep (Sep 24) | `1f6c9e2a46f` | author-requested audit of all 24 mafredri threads for sibling issues, plus a test refactor | fixed in `37eaf07d34d` through `1f14ac041b7`: the advisor retry limit, `chatretry.Retry`, and the turn step check reject values below 1 instead of disabling retries or the limit; limit tests run against both the default and configured values set through `chatd.Config` or `DeploymentValues`; the expired-attachment tooltip is concrete. A required coder-frontend-review run (8 roles in one agent, CD re-run once) found a stale story comment, removed. | | Codex 6 (Sep 27) | `a71383e3a31` | OK, no findings | requested after the rebase at the author's direction, past the six-pass review budget | The branch was rebased onto main `4851c7b60c8` on Sep 27; the previous head was `91b84000adb`, and the earlier SHAs in the review record predate the rebase. Conflicts in the `chatstate` transitions and `CreateChat` keep both main's `InitialStatus` and this PR's `MaxFileLinks`, and main's rewrite of `advisor_internal_test.go` in #28754 keeps main's version plus this PR's limit tests. Two tests added on main needed the new retry arguments: `TestGenerateStructuredTitleWithUsage_FallsBackToTextWhenToolChoiceRejected` and `TestAdvisorRunStreamsReasoningDeltasSeparatelyFromAdvice`. Validation on `a71383e3a31`: the pre-commit hook (generation, golangci-lint, build), the full `coderd/x/chatd/...` suite (20 packages), the chat tests in `codersdk`, `coderd`, and `enterprise/coderd`, the CLI help tests, race-detector runs of the `chatd`, `chatstate`, `chatadvisor`, and `chatretry` packages, and `go build` plus `go vet` on each of the 27 commits. No remote UAT was run. > Xum acted on behalf of @ibetitsmike for this pull request.
165 lines
5.1 KiB
Go
165 lines
5.1 KiB
Go
package chatadvisor
|
|
|
|
import (
|
|
"sync/atomic"
|
|
"time"
|
|
|
|
"charm.land/fantasy"
|
|
fantasyopenai "charm.land/fantasy/providers/openai"
|
|
"golang.org/x/xerrors"
|
|
)
|
|
|
|
// RuntimeConfig configures a single advisor runtime instance.
|
|
type RuntimeConfig struct {
|
|
Model fantasy.LanguageModel
|
|
// CallTemplate's provider options are cloned for each nested call.
|
|
CallTemplate fantasy.Call
|
|
MaxUsesPerRun int
|
|
MaxOutputTokens int64
|
|
StreamSilenceTimeout time.Duration
|
|
// MaxRetries is the maximum number of retries after a transient
|
|
// provider error within one advisor call. It must be positive.
|
|
MaxRetries int
|
|
}
|
|
|
|
// Runtime executes nested, tool-less advisor runs against the configured
|
|
// language model.
|
|
//
|
|
// Each Runtime instance is scoped to a single outer chat run. The
|
|
// MaxUsesPerRun counter increments on every successful advisor call and
|
|
// is never reset, so callers must construct a fresh Runtime (via
|
|
// NewRuntime) for each outer run. There is intentionally no Reset method:
|
|
// the per-run quota is a safety bound on a single run, not a rolling
|
|
// window.
|
|
type Runtime struct {
|
|
cfg RuntimeConfig
|
|
used atomic.Int64
|
|
}
|
|
|
|
// NewRuntime validates and normalizes advisor runtime configuration.
|
|
func NewRuntime(cfg RuntimeConfig) (*Runtime, error) {
|
|
if cfg.Model == nil {
|
|
return nil, xerrors.New("advisor model is required")
|
|
}
|
|
if cfg.MaxUsesPerRun <= 0 {
|
|
return nil, xerrors.New("advisor max uses per run must be positive")
|
|
}
|
|
if cfg.MaxOutputTokens <= 0 {
|
|
return nil, xerrors.New("advisor max output tokens must be positive")
|
|
}
|
|
if cfg.MaxRetries < 1 {
|
|
return nil, xerrors.New("advisor max retries must be positive")
|
|
}
|
|
if cfg.CallTemplate.MaxOutputTokens != nil &&
|
|
*cfg.CallTemplate.MaxOutputTokens != cfg.MaxOutputTokens {
|
|
return nil, xerrors.Errorf(
|
|
"advisor call template max output tokens (%d) must match runtime max output tokens (%d)",
|
|
*cfg.CallTemplate.MaxOutputTokens,
|
|
cfg.MaxOutputTokens,
|
|
)
|
|
}
|
|
|
|
normalized := cfg
|
|
normalized.CallTemplate.ProviderOptions = cloneProviderOptions(cfg.CallTemplate.ProviderOptions)
|
|
maxOutputTokens := cfg.MaxOutputTokens
|
|
normalized.CallTemplate.MaxOutputTokens = &maxOutputTokens
|
|
|
|
return &Runtime{cfg: normalized}, nil
|
|
}
|
|
|
|
// cloneProviderOptions returns a copy of opts with pointer entries for
|
|
// known, in-place mutated provider option types replaced by shallow struct
|
|
// copies, so a nested advisor call that disables Store does so on its own
|
|
// copy rather than the parent run's entry. Value fields such as Metadata
|
|
// and Include remain shared; callers that need true deep-copy semantics
|
|
// must handle those fields explicitly.
|
|
func cloneProviderOptions(opts fantasy.ProviderOptions) fantasy.ProviderOptions {
|
|
if opts == nil {
|
|
return nil
|
|
}
|
|
cloned := make(fantasy.ProviderOptions, len(opts))
|
|
for key, value := range opts {
|
|
switch typed := value.(type) {
|
|
case *fantasyopenai.ResponsesProviderOptions:
|
|
if typed == nil {
|
|
cloned[key] = value
|
|
continue
|
|
}
|
|
copied := *typed
|
|
cloned[key] = &copied
|
|
default:
|
|
cloned[key] = value
|
|
}
|
|
}
|
|
return cloned
|
|
}
|
|
|
|
// resetProviderOptionsForNestedCall forces Store off so ephemeral advisor
|
|
// calls leave no stored response behind on the provider. It mutates opts
|
|
// in place, so it must be called on a cloned map, never on options shared
|
|
// with the parent run.
|
|
func resetProviderOptionsForNestedCall(opts fantasy.ProviderOptions) {
|
|
for _, value := range opts {
|
|
if typed, ok := value.(*fantasyopenai.ResponsesProviderOptions); ok && typed != nil {
|
|
storeDisabled := false
|
|
typed.Store = &storeDisabled
|
|
}
|
|
}
|
|
}
|
|
|
|
// RemainingUses reports how many advisor calls are still available for the
|
|
// current runtime.
|
|
func (rt *Runtime) RemainingUses() int {
|
|
if rt == nil || rt.cfg.MaxUsesPerRun <= 0 {
|
|
return 0
|
|
}
|
|
|
|
remaining := int64(rt.cfg.MaxUsesPerRun) - rt.used.Load()
|
|
if remaining < 0 {
|
|
return 0
|
|
}
|
|
return int(remaining)
|
|
}
|
|
|
|
// MaxOutputTokens reports the resolved output-token cap applied to each
|
|
// advisor call. NewRuntime validates that this value is positive and that
|
|
// it matches ModelConfig.MaxOutputTokens when both are set, so the
|
|
// accessor always returns the value the runtime will actually send.
|
|
func (rt *Runtime) MaxOutputTokens() int64 {
|
|
if rt == nil {
|
|
return 0
|
|
}
|
|
return rt.cfg.MaxOutputTokens
|
|
}
|
|
|
|
// ProviderOptions reports the resolved provider options applied to each
|
|
// advisor call. NewRuntime clones the supplied options so the returned
|
|
// map reflects what nested calls will actually receive; callers must not
|
|
// mutate the map or its entries.
|
|
func (rt *Runtime) ProviderOptions() fantasy.ProviderOptions {
|
|
if rt == nil {
|
|
return nil
|
|
}
|
|
return rt.cfg.CallTemplate.ProviderOptions
|
|
}
|
|
|
|
func (rt *Runtime) tryAcquire() bool {
|
|
for {
|
|
used := rt.used.Load()
|
|
if used >= int64(rt.cfg.MaxUsesPerRun) {
|
|
return false
|
|
}
|
|
if rt.used.CompareAndSwap(used, used+1) {
|
|
return true
|
|
}
|
|
}
|
|
}
|
|
|
|
// release returns a previously acquired use to the pool. Callers must
|
|
// invoke this at most once per successful tryAcquire when the advisor
|
|
// call did not complete successfully, so a transient provider failure
|
|
// does not permanently consume quota for the run.
|
|
func (rt *Runtime) release() {
|
|
rt.used.Add(-1)
|
|
}
|