> [!NOTE]
> Xum (an AI agent) authored this pull request on Mike's behalf.
## Summary
Automatic compaction now fires at whichever of two independent triggers
is reached first, and the UI shows when the organization override's
trigger applies.
- **Chat trigger**: the chat owner's personal threshold (or the chat
model's) against the chat model's window. A threshold of 100 turns off
only this trigger.
- **Organization trigger**: a fixed 80% of the organization compaction
override model's window. The override model's own compression threshold
is ignored. The other 20% is a target margin for the summary prompt and
the summary, not a guarantee, because the trigger reads the previous
step's prompt size.
On main, the chat threshold was applied to the smaller of the two
windows, and nothing in the UI showed it. Example: with a 20K chat model
at 70% and an 8K override, main compacts at 5,600 tokens; this PR
compacts at 6,400 (80% of 8K, shown as 32%).
## Changes
Backend (`coderd/x/chatd`):
- The trigger with the lower token point binds (ties go to the chat
trigger). It drives the automatic compaction gate and the
still-over-limit check, and every summary, automatic or manual, records
it.
- An override that is unset, deleted, disabled, on a disabled provider,
or without credentials for the chat owner is treated as absent: the chat
model summarizes and only the chat trigger applies.
- The summary's output cap is bounded by the summarizer's own window,
not the trigger's.
- The still-over-limit error names what can clear it: the compaction
threshold (chat trigger), an override model with a larger window
(organization trigger), or both when usage is over both points.
- The still-over-limit check compacts once more when the latest summary
was recorded under a different threshold or trigger window. The
`chat_summarized` result gains `trigger_context_limit_tokens`, and
`context_limit_tokens` now records the chat model's window (main
recorded the smaller window). On main, lowering a threshold after a
compaction ended every later turn with the error.
- The still-over-limit check skips a 0% threshold ("always compact"),
which on main ended the run with the error after every compaction.
- `ARCHITECTURE.md` documents the two triggers, the summary cap, and the
still-over-limit rules.
Frontend (`site`), through a shared `compactionTriggers` helper that
mirrors the backend rules (no new API):
- Coder Agents settings: the compaction override row warns when the
organization trigger undercuts chat models' default triggers ("the
organization override compacts chats at 80% of {model}'s {N}-token
window").
- Personal thresholds page: an Effective column shows the trigger that
applies, with a popover when the organization override binds. "Compacts
at" follows the same trigger, and a point the chat model's window cannot
reach shows as Off. The header's 100% sentence, and the row warning when
a member types 100, say an organization override may still compact chats
when one is set or its organization's overrides failed to load (the
header also while they load).
- Chat context gauge: the compact-at percent follows the binding trigger
and is annotated when the organization override binds. If the overrides
request fails, it shows the chat-only value with "(organization override
not loaded)", or "Compaction off (organization override not loaded)"
when the chat trigger is off.
- An override whose provider the user cannot use is ignored on the gauge
and the personal page, matching chatd.
- The shared `formatContextLimit` prints counts below 1,000 exactly (an
800-token compaction point showed "~1K") and 999,500 to 999,999 as
"1.0M" instead of "1000K". It also labels the model pickers; their other
values are unchanged.
- `HelpPopoverIconTrigger` lets callers set their own `aria-label`; only
the Effective column's popover does.
Known v1 limits: the Coder Agents settings alert is organization-scoped,
so it can warn about an override that some members fall back from for
lack of provider credentials. The gauge checks the viewer's provider
access, so on a shared chat its label can differ from the owner's
runtime trigger (Codex P2, declined). Compaction itself is unaffected.
## Screenshots
From remote UAT round 13 on 093076f402: an isolated test deployment
with a local fake LLM. The models are test fixtures named for their
context windows: Chat model 20K (own threshold 70%), Chat model 5K
(40%), and Chat model 8K (50%). Chat model 8K is also the organization
compaction override, so the organization trigger is 80% of 8,000 = 6,400
tokens, which is 32% of Chat model 20K's window. Chat model 8K's own 50%
applies only to chats that use it as the chat model; changing it to 10%
or 100% left every value below unchanged.
**Coder Agents settings:** the organization trigger undercuts Chat model
20K's 70% default.

**Personal thresholds:** the organization trigger binds Chat model 20K
at 32%. Chat model 5K (40%) and Chat model 8K (50%) keep their own chat
triggers.


**Personal thresholds, organization settings request failed** (the first
red line is the injected error). Rows fall back to chat-only values.

**Chat context gauge:** organization trigger binds (Chat model 20K),
chat trigger binds (Chat model 5K), overrides request failed (injected).



**Still-over-limit errors from real runs:** organization trigger, chat
trigger, both.



## Testing
- Go: `TestBindingCompactionTriggerSource`,
`TestCompactionStillOverLimitError`, `TestCompactionStatusFromHistory`,
and `TestActiveServer_CompactionModelOverride` cover the 80% point
(fails if the override's own threshold is read again), user threshold
100 leaving the organization trigger on, the summary cap for automatic
and manual compaction, the over-limit messages, and recompaction after a
trigger change. The site binding tests mirror the Go table row for row.
- Site: `compactionTriggers` unit tests and Storybook stories for the
three surfaces, including the overrides fetch failure.
- On c6f8750479: `go vet`, the `chatd` and `chatloop` tests, and the
AgentsPage, AISettingsPage, and HelpPopover unit tests (2,274 passed, 2
skipped) pass, and CI is green. On 987cefe399, tsc, the five affected
story files (109 passed), and `make pre-commit` also passed. 093076f402
only changes `ARCHITECTURE.md`; the pre-commit hook's markdown format
and lint checks pass on it.
- On 3cd5b08f1f: tsc, the new `formatContextLimit` test (fails on the
old helper for 800 and 999,999), the `compactionTriggers` and `aiModels`
unit tests (224 passed), and the affected story files pass, and the
pre-commit hook passed on all three commits.
## Review record
- Earlier heads: 12 Codex passes and 3 human reviews, all threads
answered. Rebased onto main several times, most recently ff3fa22f68
(kept #30274, #29892, and #29890 in conflicted files).
- coder-agents-review rounds 1 to 18: findings fixed; approved since
round 15. The round 1 `ARCHITECTURE.md` finding was left for the
maintainer, who asked for the update in the last commit.
- coder-frontend-review (eight roles): all pass. Splitting the PR (RO6)
was declined because it is reviewed as one unit; the
`organizationTriggerFor` story helper stays because its three fields
must match the model.
- Design change after UAT round 11: the organization trigger is a fixed
80% of the override window instead of the override model's own threshold
(171d5f568e, 21d2ce7fce, 987cefe399).
- coder-agents-review round 19: copy, naming, and an unreachable
zero-window gate fixed (6e8efb9054, c6f8750479). A summary-time fit
check with chat-model fallback (CRF-156) was declined: the previous-step
lag applies to every trigger, including main's. Round 20 approved
c6f8750479 with no findings.
- Codex: one P2 on 987cefe399 declined (see known limits); no major
issues on c6f8750479.
- Remote UAT on dogfood (fake LLM, isolated test deployment): rounds 8
to 13 PASS. Round 12 on c6f8750479 first failed one criterion that was
written wrong (the admin alert correctly lists an override model that is
also a chat model at 100%); the reviewer endorsed PASS on the same
evidence after the correction. Round 13 on 093076f402 retook the
screenshots with neutral model names and confirmed that the override
model's own threshold (10%, 50%, 100%) does not move the organization
trigger.
- UAT round 13 notes: the header after a failed or pending overrides
load and the sub-1K labels are fixed in 4e172ab5ff and d5888ce15dc;
coder-agents-review round 21 found the same gap in the row warning
(CRF-167, fixed in 3cd5b08f1f). Left as on main: a summary's
`usage_percent` is measured against the binding trigger's window, so it
can exceed 100 (nothing reads it), and the 100% help text shows only for
an unsaved edit.
- Codex: no major issues on d5888ce15d and 3cd5b08f1f.
coder-agents-review round 22 approved 3cd5b08f1f with no open findings.
- coder-frontend-review on those commits: all roles pass after the
fixes. Declined: moving the formatter fix to its own PR (RO5; the owner
asked for it here), adding "tokens" to sub-1K labels (TN6; "~800" reads
like "~16K"), and splitting the component (RP6).
702 lines
25 KiB
Go
702 lines
25 KiB
Go
package chatloop
|
|
|
|
import (
|
|
"context"
|
|
"encoding/json"
|
|
"errors"
|
|
"net/http"
|
|
"strings"
|
|
"time"
|
|
|
|
"charm.land/fantasy"
|
|
"github.com/google/uuid"
|
|
"golang.org/x/xerrors"
|
|
|
|
"github.com/coder/coder/v2/coderd/x/chatd/chatdebug"
|
|
"github.com/coder/coder/v2/codersdk"
|
|
"github.com/coder/quartz"
|
|
)
|
|
|
|
const (
|
|
defaultCompactionThresholdPercent = int32(70)
|
|
minCompactionThresholdPercent = int32(0)
|
|
// CompactionDisabledThresholdPercent turns a compaction trigger off.
|
|
CompactionDisabledThresholdPercent = int32(100)
|
|
|
|
// compactionDebugCreateRunTimeout caps the compaction debug
|
|
// CreateRun budget. Debug instrumentation is best-effort;
|
|
// running without the debug row is preferable to blocking
|
|
// compaction on a slow or locked DB.
|
|
compactionDebugCreateRunTimeout = 5 * time.Second
|
|
|
|
defaultCompactionSummaryPrompt = "You are performing a context compaction. " +
|
|
"Summarize the conversation so a new assistant can seamlessly " +
|
|
"continue the work in progress.\n\n" +
|
|
"Include:\n" +
|
|
// The constraints bullet below is deliberately verbose: offline replay
|
|
// of production chats showed compaction summaries dropping or softening
|
|
// user-stated constraints, and this wording measurably improved their
|
|
// survival (see PR #27230). Reword only with re-validation.
|
|
"- User constraints, corrections, and prohibitions: rules, " +
|
|
"scope limits, style rules, and process corrections stated by " +
|
|
"the user. Quote or closely paraphrase the user's wording; do " +
|
|
"not soften, merge, or truncate them. Constraints are standing " +
|
|
"until the user revokes them; they do not become stale when " +
|
|
"the task moves on. When the user corrected the assistant's " +
|
|
"behavior, record the correction itself, not only the " +
|
|
"corrected outcome. Include only constraints the user stated " +
|
|
"in conversation; do not place rules from system prompts, " +
|
|
"AGENTS.md, or other configuration files in this section. " +
|
|
"Those rules may appear elsewhere in the summary with their " +
|
|
"true source named. When in doubt whether a rule originated " +
|
|
"from the user, name its source or omit the attribution " +
|
|
"rather than defaulting to user.\n" +
|
|
"- The user's overall goal and current task\n" +
|
|
"- Key decisions made and their rationale\n" +
|
|
"- Concrete technical details: file paths, function names, " +
|
|
"commands, APIs, and configurations\n" +
|
|
"- Errors encountered and how they were resolved. Keep error " +
|
|
"notes specific: name the file, the error, and the fix. Do not " +
|
|
"generalize from a specific failure to a blanket tool-avoidance " +
|
|
"rule (e.g. \"tool X is unreliable\" or \"always use Y instead " +
|
|
"of Z\")\n" +
|
|
"- Current state of the work: what is DONE, what is IN PROGRESS, " +
|
|
"and what REMAINS to be done\n" +
|
|
"- The specific action the assistant was performing or about to " +
|
|
"perform when this summary was triggered\n\n" +
|
|
"Be dense and factual. Every sentence should convey essential " +
|
|
"context for continuation. Do not include pleasantries or " +
|
|
"conversational filler. For content that can be reproduced " +
|
|
"(repo files, command output, API responses), reference how to " +
|
|
"obtain it (file path, command, URL) rather than inlining the " +
|
|
"full content. Include brief inline summaries when the content " +
|
|
"itself would exceed a few lines."
|
|
defaultCompactionSystemSummaryPrefix = "The following is a summary of " +
|
|
"the earlier conversation. Continue from it:"
|
|
)
|
|
|
|
// CompactionSource identifies what triggered a compaction. It is
|
|
// recorded in the persisted chat_summarized tool JSON and the
|
|
// streamed synthetic parts so clients can render manual compactions
|
|
// distinctly.
|
|
type CompactionSource string
|
|
|
|
const (
|
|
CompactionSourceAutomatic CompactionSource = "automatic"
|
|
CompactionSourceManual CompactionSource = "manual"
|
|
)
|
|
|
|
type CompactionOptions struct {
|
|
ThresholdPercent int32
|
|
ContextLimit int64
|
|
SummaryPrompt string
|
|
SummaryHint string
|
|
SystemSummaryPrefix string
|
|
DebugSvc *chatdebug.Service
|
|
ChatID uuid.UUID
|
|
HistoryTipMessageID int64
|
|
|
|
ResolvedProvider string
|
|
ResolvedModel string
|
|
ModelConfigID uuid.UUID
|
|
SummaryCall fantasy.Call
|
|
// ToolDefinitions is copied from the parent generation request so the
|
|
// summary call uses the exact same ordered definitions.
|
|
ToolDefinitions []fantasy.Tool
|
|
|
|
// Clock and StreamSilenceTimeout guard the summary stream against a
|
|
// provider that opens the stream but stops yielding parts. Zero values
|
|
// fall back to a real clock and DefaultStreamSilenceTimeout.
|
|
Clock quartz.Clock
|
|
StreamSilenceTimeout time.Duration
|
|
|
|
// Force skips the threshold gate (including the threshold=100
|
|
// disable and the zero-usage early return). Set for manual,
|
|
// user-requested compactions.
|
|
Force bool
|
|
// Source labels what triggered the compaction. Defaults to
|
|
// CompactionSourceAutomatic when empty.
|
|
Source CompactionSource
|
|
|
|
// ToolCallID and ToolName identify the synthetic tool call
|
|
// used to represent compaction in the message stream.
|
|
ToolCallID string
|
|
ToolName string
|
|
|
|
// PublishMessagePart publishes streaming parts to connected
|
|
// clients so they see "Summarizing..." / "Summarized" UI
|
|
// transitions during compaction.
|
|
PublishMessagePart func(codersdk.ChatMessageRole, codersdk.ChatMessagePart)
|
|
}
|
|
|
|
// CompactionResult is one generated summary. ThresholdPercent and
|
|
// TriggerContextLimit echo the requested trigger, not the normalized one, so
|
|
// chatd's summaryRecordedUnderOtherTrigger compares like with like;
|
|
// UsagePercent is still measured against the resolved limit.
|
|
type CompactionResult struct {
|
|
SystemSummary string
|
|
SummaryReport string
|
|
Source CompactionSource
|
|
ThresholdPercent int32
|
|
UsagePercent float64
|
|
ContextTokens int64
|
|
ChatContextLimit int64
|
|
// EstimatedContextTokens covers only SystemSummary, not the full prompt.
|
|
EstimatedContextTokens int64
|
|
TriggerContextLimit int64
|
|
// Runtime is the wall-clock duration of the summarization model
|
|
// call, the compaction step's billable runtime (see
|
|
// PersistedStep.Runtime). Zero when the run was gated off before
|
|
// calling the model.
|
|
Runtime time.Duration
|
|
// ProviderResponseID identifies the summary response. See
|
|
// PersistedStep.ProviderResponseID.
|
|
ProviderResponseID string
|
|
}
|
|
|
|
// CompactionToolResult is the chat_summarized tool result payload, both
|
|
// streamed and persisted.
|
|
type CompactionToolResult struct {
|
|
Summary string `json:"summary"`
|
|
Source CompactionSource `json:"source"`
|
|
// ThresholdPercent is relative to TriggerContextLimitTokens.
|
|
ThresholdPercent int32 `json:"threshold_percent"`
|
|
// UsagePercent is relative to the resolved limit, which equals
|
|
// TriggerContextLimitTokens when it is positive.
|
|
UsagePercent float64 `json:"usage_percent"`
|
|
ContextTokens int64 `json:"context_tokens"`
|
|
// ContextLimitTokens is the chat model's window.
|
|
ContextLimitTokens int64 `json:"context_limit_tokens"`
|
|
EstimatedContextTokens int64 `json:"estimated_context_tokens"`
|
|
TriggerContextLimitTokens int64 `json:"trigger_context_limit_tokens"`
|
|
}
|
|
|
|
// GenerateCompaction generates one context summary and returns it without
|
|
// persisting. It publishes compaction progress parts when configured.
|
|
// Threshold gating (including the threshold=100 disable and the
|
|
// zero-usage early return) is skipped when opts.Force is set.
|
|
func GenerateCompaction(ctx context.Context, opts GenerateCompactionOptions) (CompactionResult, error) {
|
|
if opts.Model == nil {
|
|
return CompactionResult{}, xerrors.New("chat model is required")
|
|
}
|
|
if opts.Clock == nil {
|
|
return CompactionResult{}, xerrors.New("clock is required")
|
|
}
|
|
config, ok := normalizedCompactionGenerateConfig(opts)
|
|
if !ok {
|
|
return CompactionResult{}, nil
|
|
}
|
|
|
|
contextTokens := contextTokensFromUsage(opts.StepUsage)
|
|
if contextTokens <= 0 && !config.Force {
|
|
return CompactionResult{}, nil
|
|
}
|
|
contextLimit := resolveContextLimit(
|
|
config.ContextLimit,
|
|
opts.ContextLimitFallback,
|
|
)
|
|
usagePercent, compact := shouldCompact(
|
|
contextTokens,
|
|
contextLimit,
|
|
config.ThresholdPercent,
|
|
)
|
|
if !compact && !config.Force {
|
|
return CompactionResult{}, nil
|
|
}
|
|
|
|
// Sum-enforcing providers reject requests whose input plus
|
|
// max_tokens exceeds the context window, so bound the summary cap
|
|
// by the remaining window. contextTokens covers only the trigger
|
|
// step's prompt, so also reserve that step's output, the tool
|
|
// results executed after it, and the summary prompt appended by
|
|
// generateCompactionSummary, all of which become input to the
|
|
// summary request. Degenerate cases (unknown limit, no room left)
|
|
// leave the cap unchanged.
|
|
if config.SummaryCall.MaxOutputTokens != nil {
|
|
promptBytes := len(config.SummaryPrompt) + len(config.SummaryHint)
|
|
promptTokens := int64((promptBytes + bytesPerTokenEstimate - 1) / bytesPerTokenEstimate)
|
|
reserved := opts.StepUsage.OutputTokens + promptTokens + trailingToolResultTokens(opts.Messages)
|
|
windowLimit := contextLimit
|
|
if opts.SummaryContextLimit > 0 {
|
|
windowLimit = opts.SummaryContextLimit
|
|
}
|
|
remaining := windowLimit - contextTokens - reserved
|
|
if remaining > 0 && remaining < *config.SummaryCall.MaxOutputTokens {
|
|
config.SummaryCall.MaxOutputTokens = &remaining
|
|
}
|
|
}
|
|
|
|
if config.PublishMessagePart != nil && config.ToolCallID != "" {
|
|
config.PublishMessagePart(
|
|
codersdk.ChatMessageRoleAssistant,
|
|
codersdk.ChatMessageToolCall(config.ToolCallID, config.ToolName, nil),
|
|
)
|
|
}
|
|
|
|
summaryStart := opts.Clock.Now()
|
|
if opts.OnModelStreamStart != nil {
|
|
opts.OnModelStreamStart()
|
|
}
|
|
summary, responseID, err := generateCompactionSummary(ctx, opts.Model, opts.Messages, config)
|
|
if err != nil {
|
|
publishCompactionError(config, "failed to generate compaction summary")
|
|
return CompactionResult{}, err
|
|
}
|
|
summaryRuntime := opts.Clock.Since(summaryStart)
|
|
if summary == "" {
|
|
publishCompactionError(config, "compaction produced an empty summary")
|
|
return CompactionResult{}, xerrors.New("compaction produced an empty summary")
|
|
}
|
|
|
|
result := CompactionResult{
|
|
SystemSummary: strings.TrimSpace(
|
|
config.SystemSummaryPrefix + "\n\n" + summary,
|
|
),
|
|
SummaryReport: summary,
|
|
Source: config.Source,
|
|
ThresholdPercent: opts.ThresholdPercent,
|
|
UsagePercent: usagePercent,
|
|
ContextTokens: contextTokens,
|
|
ChatContextLimit: opts.ChatContextLimit,
|
|
TriggerContextLimit: opts.ContextLimit,
|
|
Runtime: summaryRuntime,
|
|
ProviderResponseID: responseID,
|
|
}
|
|
result.EstimatedContextTokens = int64((len(result.SystemSummary) + bytesPerTokenEstimate - 1) / bytesPerTokenEstimate)
|
|
if config.PublishMessagePart != nil && config.ToolCallID != "" {
|
|
resultJSON, _ := json.Marshal(CompactionToolResult{
|
|
Summary: summary,
|
|
Source: result.Source,
|
|
ThresholdPercent: result.ThresholdPercent,
|
|
UsagePercent: usagePercent,
|
|
ContextTokens: contextTokens,
|
|
ContextLimitTokens: result.ChatContextLimit,
|
|
EstimatedContextTokens: result.EstimatedContextTokens,
|
|
TriggerContextLimitTokens: result.TriggerContextLimit,
|
|
})
|
|
config.PublishMessagePart(
|
|
codersdk.ChatMessageRoleTool,
|
|
codersdk.ChatMessageToolResult(config.ToolCallID, config.ToolName, resultJSON, false, false),
|
|
)
|
|
}
|
|
return result, nil
|
|
}
|
|
|
|
func normalizedCompactionGenerateConfig(opts GenerateCompactionOptions) (CompactionOptions, bool) {
|
|
config := CompactionOptions{
|
|
ThresholdPercent: opts.ThresholdPercent,
|
|
ContextLimit: opts.ContextLimit,
|
|
SummaryPrompt: opts.SummaryPrompt,
|
|
SummaryHint: opts.SummaryHint,
|
|
SystemSummaryPrefix: opts.SystemSummaryPrefix,
|
|
DebugSvc: opts.DebugSvc,
|
|
ChatID: opts.ChatID,
|
|
HistoryTipMessageID: opts.HistoryTipMessageID,
|
|
ResolvedProvider: opts.ResolvedProvider,
|
|
ResolvedModel: opts.ResolvedModel,
|
|
ModelConfigID: opts.ModelConfigID,
|
|
SummaryCall: opts.SummaryCall,
|
|
ToolDefinitions: opts.ToolDefinitions,
|
|
Force: opts.Force,
|
|
Source: opts.Source,
|
|
ToolCallID: opts.ToolCallID,
|
|
ToolName: opts.ToolName,
|
|
PublishMessagePart: opts.PublishMessagePart,
|
|
Clock: opts.Clock,
|
|
StreamSilenceTimeout: opts.StreamSilenceTimeout,
|
|
}
|
|
if strings.TrimSpace(config.SummaryPrompt) == "" {
|
|
config.SummaryPrompt = defaultCompactionSummaryPrompt
|
|
}
|
|
if strings.TrimSpace(config.SystemSummaryPrefix) == "" {
|
|
config.SystemSummaryPrefix = defaultCompactionSystemSummaryPrefix
|
|
}
|
|
if config.Source == "" {
|
|
config.Source = CompactionSourceAutomatic
|
|
}
|
|
if config.ThresholdPercent < minCompactionThresholdPercent ||
|
|
config.ThresholdPercent > CompactionDisabledThresholdPercent {
|
|
config.ThresholdPercent = defaultCompactionThresholdPercent
|
|
}
|
|
// threshold=100 disables automatic compaction; a forced run
|
|
// still proceeds because the user asked explicitly.
|
|
if config.ThresholdPercent == CompactionDisabledThresholdPercent && !config.Force {
|
|
return CompactionOptions{}, false
|
|
}
|
|
return config, true
|
|
}
|
|
|
|
// publishCompactionError sends a tool-result error part so
|
|
// connected clients see that compaction failed.
|
|
func publishCompactionError(config CompactionOptions, msg string) {
|
|
if config.PublishMessagePart == nil || config.ToolCallID == "" {
|
|
return
|
|
}
|
|
errJSON, _ := json.Marshal(map[string]any{
|
|
"error": msg,
|
|
})
|
|
config.PublishMessagePart(
|
|
codersdk.ChatMessageRoleTool,
|
|
codersdk.ChatMessageToolResult(config.ToolCallID, config.ToolName, errJSON, true, false),
|
|
)
|
|
}
|
|
|
|
// trailingToolResultTokens estimates tokens for the tool-result
|
|
// messages that follow the usage-measured assistant step. Local tools
|
|
// execute before compaction, so their results are input to the summary
|
|
// request but absent from StepUsage.
|
|
func trailingToolResultTokens(messages []fantasy.Message) int64 {
|
|
totalBytes := 0
|
|
for i := len(messages) - 1; i >= 0 && messages[i].Role == fantasy.MessageRoleTool; i-- {
|
|
for _, part := range messages[i].Content {
|
|
result, ok := part.(fantasy.ToolResultPart)
|
|
if !ok {
|
|
resultPtr, okPtr := part.(*fantasy.ToolResultPart)
|
|
if !okPtr || resultPtr == nil {
|
|
continue
|
|
}
|
|
result = *resultPtr
|
|
}
|
|
switch output := result.Output.(type) {
|
|
case fantasy.ToolResultOutputContentText:
|
|
totalBytes += len(output.Text)
|
|
case fantasy.ToolResultOutputContentError:
|
|
if output.Error != nil {
|
|
totalBytes += len(output.Error.Error())
|
|
}
|
|
case fantasy.ToolResultOutputContentMedia:
|
|
totalBytes += len(output.Data) + len(output.Text)
|
|
}
|
|
}
|
|
}
|
|
return int64((totalBytes + bytesPerTokenEstimate - 1) / bytesPerTokenEstimate)
|
|
}
|
|
|
|
// contextTokensFromUsage returns the total context token count from
|
|
// a step's usage report. It sums input, cache-read, and
|
|
// cache-creation tokens when available, falling back to TotalTokens
|
|
// if none of the granular fields are set.
|
|
func contextTokensFromUsage(usage fantasy.Usage) int64 {
|
|
total := int64(0)
|
|
hasContextTokens := false
|
|
|
|
if usage.InputTokens > 0 {
|
|
total += usage.InputTokens
|
|
hasContextTokens = true
|
|
}
|
|
if usage.CacheReadTokens > 0 {
|
|
total += usage.CacheReadTokens
|
|
hasContextTokens = true
|
|
}
|
|
if usage.CacheCreationTokens > 0 {
|
|
total += usage.CacheCreationTokens
|
|
hasContextTokens = true
|
|
}
|
|
if !hasContextTokens && usage.TotalTokens > 0 {
|
|
total = usage.TotalTokens
|
|
}
|
|
|
|
return total
|
|
}
|
|
|
|
// resolveContextLimit returns the configured limit when positive, then
|
|
// the fallback, or zero when neither is positive.
|
|
func resolveContextLimit(configLimit, fallback int64) int64 {
|
|
if configLimit > 0 {
|
|
return configLimit
|
|
}
|
|
if fallback > 0 {
|
|
return fallback
|
|
}
|
|
return 0
|
|
}
|
|
|
|
// shouldCompact returns the usage percentage and whether it exceeds
|
|
// the threshold. Returns (0, false) when contextLimit is
|
|
// non-positive.
|
|
func shouldCompact(contextTokens, contextLimit int64, thresholdPercent int32) (float64, bool) {
|
|
if contextLimit <= 0 {
|
|
return 0, false
|
|
}
|
|
usagePercent := (float64(contextTokens) / float64(contextLimit)) * 100
|
|
return usagePercent, usagePercent >= float64(thresholdPercent)
|
|
}
|
|
|
|
func startCompactionDebugRun(
|
|
ctx context.Context,
|
|
options CompactionOptions,
|
|
) (context.Context, func(error)) {
|
|
if options.DebugSvc == nil || options.ChatID == uuid.Nil {
|
|
return ctx, func(error) {}
|
|
}
|
|
|
|
parentRun, ok := chatdebug.RunFromContext(ctx)
|
|
if !ok {
|
|
return ctx, func(error) {}
|
|
}
|
|
|
|
historyTipMessageID := options.HistoryTipMessageID
|
|
if historyTipMessageID == 0 {
|
|
historyTipMessageID = parentRun.HistoryTipMessageID
|
|
}
|
|
|
|
// Prefer the caller-supplied summary model identity; it can differ
|
|
// from the parent run's chat model under a compaction override.
|
|
provider := parentRun.Provider
|
|
if options.ResolvedProvider != "" {
|
|
provider = options.ResolvedProvider
|
|
}
|
|
model := parentRun.Model
|
|
if options.ResolvedModel != "" {
|
|
model = options.ResolvedModel
|
|
}
|
|
modelConfigID := parentRun.ModelConfigID
|
|
if options.ModelConfigID != uuid.Nil {
|
|
modelConfigID = options.ModelConfigID
|
|
}
|
|
|
|
// Use a separate short-lived context for the debug insert so a
|
|
// slow or locked DB cannot block the model call. Detached from
|
|
// the parent so cancellation of the compaction run still lets
|
|
// the insert reach a terminal state, matching the best-effort
|
|
// contract of debug instrumentation.
|
|
createRunCtx, createRunCancel := context.WithTimeout(
|
|
context.WithoutCancel(ctx), compactionDebugCreateRunTimeout,
|
|
)
|
|
run, err := options.DebugSvc.CreateRun(createRunCtx, chatdebug.CreateRunParams{
|
|
ChatID: options.ChatID,
|
|
RootChatID: parentRun.RootChatID,
|
|
ParentChatID: parentRun.ParentChatID,
|
|
ModelConfigID: modelConfigID,
|
|
TriggerMessageID: parentRun.TriggerMessageID,
|
|
HistoryTipMessageID: historyTipMessageID,
|
|
Kind: chatdebug.KindCompaction,
|
|
Status: chatdebug.StatusInProgress,
|
|
Provider: provider,
|
|
Model: model,
|
|
})
|
|
createRunCancel()
|
|
if err != nil {
|
|
// Debug instrumentation must not surface as a compaction failure.
|
|
return ctx, func(error) {}
|
|
}
|
|
|
|
compactionCtx := chatdebug.ContextWithRun(ctx, &chatdebug.RunContext{
|
|
RunID: run.ID,
|
|
ChatID: options.ChatID,
|
|
RootChatID: parentRun.RootChatID,
|
|
ParentChatID: parentRun.ParentChatID,
|
|
ModelConfigID: modelConfigID,
|
|
TriggerMessageID: parentRun.TriggerMessageID,
|
|
HistoryTipMessageID: historyTipMessageID,
|
|
Provider: provider,
|
|
Model: model,
|
|
})
|
|
|
|
return compactionCtx, func(runErr error) {
|
|
status := chatdebug.ClassifyError(runErr)
|
|
// Debug instrumentation must not surface as a compaction failure.
|
|
_ = options.DebugSvc.FinalizeRun(compactionCtx, chatdebug.FinalizeRunParams{
|
|
RunID: run.ID,
|
|
ChatID: options.ChatID,
|
|
Status: status,
|
|
})
|
|
}
|
|
}
|
|
|
|
// generateCompactionSummary asks the model to summarize the
|
|
// conversation so far. The provided messages should contain the
|
|
// complete history (system prompt, user/assistant turns, tool
|
|
// results). A final user message with the summary prompt is appended
|
|
// before calling the model.
|
|
func generateCompactionSummary(
|
|
ctx context.Context,
|
|
model fantasy.LanguageModel,
|
|
messages []fantasy.Message,
|
|
options CompactionOptions,
|
|
) (summary string, responseID string, err error) {
|
|
summaryPrompt := make([]fantasy.Message, 0, len(messages)+1)
|
|
summaryPrompt = append(summaryPrompt, messages...)
|
|
summaryParts := []fantasy.MessagePart{fantasy.TextPart{Text: options.SummaryPrompt}}
|
|
if strings.TrimSpace(options.SummaryHint) != "" {
|
|
summaryParts = append(summaryParts, fantasy.TextPart{Text: options.SummaryHint})
|
|
}
|
|
summaryPrompt = append(summaryPrompt, fantasy.Message{
|
|
Role: fantasy.MessageRoleUser,
|
|
Content: summaryParts,
|
|
})
|
|
// Claude only reads the cache at explicit breakpoints, so without
|
|
// these the shared tool and history prefix is never a cache hit.
|
|
if applyPromptCaching := promptCachingStrategyFor(model, model.Model()); applyPromptCaching != nil {
|
|
applyPromptCaching(summaryPrompt)
|
|
}
|
|
|
|
summaryCtx, finishDebugRun := startCompactionDebugRun(ctx, options)
|
|
defer func() {
|
|
// If model.Stream (or anything else below) panics, the
|
|
// named err return is still nil at this point. Without the
|
|
// recover hook we would finalize the debug run as Completed
|
|
// in the exact crash path operators rely on to diagnose
|
|
// failures. Finalize with the panic as an error status and
|
|
// re-panic so the caller's recovery still observes the
|
|
// original panic value.
|
|
if r := recover(); r != nil {
|
|
finishDebugRun(xerrors.Errorf("panic during compaction summary: %v", r))
|
|
panic(r)
|
|
}
|
|
finishDebugRun(err)
|
|
}()
|
|
|
|
call := options.SummaryCall
|
|
call.Prompt = summaryPrompt
|
|
call.Tools = options.ToolDefinitions
|
|
clock := options.Clock
|
|
if clock == nil {
|
|
clock = quartz.NewReal()
|
|
}
|
|
timeout := options.StreamSilenceTimeout
|
|
if timeout == 0 {
|
|
timeout = DefaultStreamSilenceTimeout
|
|
}
|
|
// TTFT is an assistant-generation measurement, so the summary stream
|
|
// records neither its histogram nor its stage.
|
|
streamSummaryText := func() (string, error) {
|
|
attempt, err := guardedStream(
|
|
summaryCtx,
|
|
options.ResolvedProvider,
|
|
options.ResolvedModel,
|
|
clock,
|
|
timeout,
|
|
func(attemptCtx context.Context) (fantasy.StreamResponse, error) {
|
|
return model.Stream(attemptCtx, call)
|
|
},
|
|
NopMetrics(),
|
|
(*StageTracer)(nil),
|
|
StageModel{},
|
|
)
|
|
if err != nil {
|
|
return "", xerrors.Errorf("stream summary text: %w", wrapProviderStreamError(options.ResolvedProvider, err))
|
|
}
|
|
defer attempt.release()
|
|
|
|
textPartIndexes := make(map[string]int)
|
|
textParts := make([]string, 0, 1)
|
|
var (
|
|
reasoningSeen bool
|
|
finishSeen bool
|
|
finishReason fantasy.FinishReason
|
|
streamErr error
|
|
)
|
|
for part := range attempt.stream {
|
|
switch part.Type {
|
|
case fantasy.StreamPartTypeTextStart:
|
|
if _, ok := textPartIndexes[part.ID]; ok {
|
|
continue
|
|
}
|
|
textPartIndexes[part.ID] = len(textParts)
|
|
textParts = append(textParts, part.Delta)
|
|
case fantasy.StreamPartTypeTextDelta:
|
|
index, ok := textPartIndexes[part.ID]
|
|
if !ok {
|
|
index = len(textParts)
|
|
textPartIndexes[part.ID] = index
|
|
textParts = append(textParts, "")
|
|
}
|
|
textParts[index] += part.Delta
|
|
case fantasy.StreamPartTypeReasoningStart,
|
|
fantasy.StreamPartTypeReasoningDelta,
|
|
fantasy.StreamPartTypeReasoningEnd:
|
|
reasoningSeen = true
|
|
case fantasy.StreamPartTypeFinish:
|
|
finishSeen = true
|
|
finishReason = part.FinishReason
|
|
responseID = providerResponseID(part)
|
|
case fantasy.StreamPartTypeError:
|
|
streamErr = part.Error
|
|
if streamErr == nil {
|
|
streamErr = xerrors.New("model returned an error part")
|
|
}
|
|
}
|
|
if streamErr != nil {
|
|
break
|
|
}
|
|
}
|
|
if err := attempt.finish(streamErr); err != nil {
|
|
// Providers can surface remote stream resets as bare
|
|
// context.Canceled; wrap them like GenerateAssistant so the
|
|
// generation loop retries instead of terminally erroring.
|
|
return "", xerrors.Errorf("stream summary text: %w", wrapProviderStreamError(options.ResolvedProvider, err))
|
|
}
|
|
if !finishSeen {
|
|
// A stream that ends without a finish part was interrupted.
|
|
// Committing its partial text would compact history against an
|
|
// incomplete summary.
|
|
if ctxErr := summaryCtx.Err(); ctxErr != nil {
|
|
return "", xerrors.Errorf("stream summary text: %w", ctxErr)
|
|
}
|
|
return "", xerrors.New("compaction summary stream ended without a finish part")
|
|
}
|
|
|
|
parts := make([]string, 0, len(textParts))
|
|
for _, text := range textParts {
|
|
text = strings.TrimSpace(text)
|
|
if text != "" {
|
|
parts = append(parts, text)
|
|
}
|
|
}
|
|
joined := strings.TrimSpace(strings.Join(parts, " "))
|
|
if joined == "" && (finishReason == fantasy.FinishReasonLength || reasoningSeen) {
|
|
return "", xerrors.New("compaction summary was truncated at the output token cap")
|
|
}
|
|
return joined, nil
|
|
}
|
|
summary, err = streamSummaryText()
|
|
if err != nil && len(call.Tools) > 0 && isContextTooLargeError(err) {
|
|
// Tool definitions keep the summary request on the parent turn's
|
|
// cacheable prefix, but they also make it larger than the turn
|
|
// that just overflowed. Compaction is the recovery path for an
|
|
// over-limit conversation, so fall back to the tool-less request,
|
|
// which still fits whenever the history alone does. The rejection
|
|
// can arrive from the stream open or as an in-stream error part,
|
|
// so the retry wraps the whole attempt.
|
|
call.Tools = nil
|
|
summary, err = streamSummaryText()
|
|
}
|
|
return summary, responseID, err
|
|
}
|
|
|
|
// contextTooLargePhrases are context-window rejections fantasy does not
|
|
// parse, such as the OpenAI Responses API "Your input exceeds the context
|
|
// window of this model." and Anthropic-family "too long" rejections
|
|
// without token counts. The response body is included so the OpenAI
|
|
// error code "context_length_exceeded" matches regardless of wording.
|
|
var contextTooLargePhrases = []string{
|
|
"context window",
|
|
"context length",
|
|
"context_length",
|
|
"too long",
|
|
}
|
|
|
|
// isContextTooLargeError reports whether a provider rejected a request for
|
|
// its size: the prompt exceeded the model's context window, or the body
|
|
// exceeded a request size limit on the provider or a gateway in front of it.
|
|
func isContextTooLargeError(err error) bool {
|
|
providerErr, ok := errors.AsType[*fantasy.ProviderError](err)
|
|
if !ok {
|
|
return false
|
|
}
|
|
if providerErr.IsContextTooLarge() || providerErr.StatusCode == http.StatusRequestEntityTooLarge {
|
|
return true
|
|
}
|
|
if providerErr.StatusCode != http.StatusBadRequest {
|
|
return false
|
|
}
|
|
text := strings.ToLower(providerErr.Error() + " " + string(providerErr.ResponseBody))
|
|
for _, phrase := range contextTooLargePhrases {
|
|
if strings.Contains(text, phrase) {
|
|
return true
|
|
}
|
|
}
|
|
return false
|
|
}
|