fix(coderd/x/chatd): keep summary tool definitions when the override is the chat model

A compaction override may resolve to the chat model itself (same
provider instance and model), in which case the summary request shares
the chat model's prompt cache and its tool definitions are worth
keeping. Drop them only for an override that resolves elsewhere.

Also leave TODO items in the affected ARCHITECTURE.md sections for the
tool-carrying summary request, the Anthropic cache breakpoints, the
tool-less size retry, and the override behavior.
This commit is contained in:
Michael Suchacz committed 2026-09-16 22:35:32 +00:00
1 parent e7bd72390c
commit 6d8d8d6f71
3 files changed
+25 -4

No files matched your search

+3
View File
@@ -979,6 +979,7 @@ Details that follow from the override:
A usable override that fails at use (route or client construction, provider call failure) fails the generation visibly through the normal error path; there is no silent fallback.
The override model client is constructed inside the compact generation action, not at prepare time, so a broken override cannot fail turns that finish without compacting (including turns over the threshold whose last assistant step already completed).
- Prompt safety: the prompt is built and sanitized for the chat model, so when the override points at a different provider the compaction copy of the prompt is re-sanitized: provider-executed tool history is flattened into plain text parts (keeping its content while dropping the provider-specific wire shape), file parts the compaction model rejects are replaced with text placeholders, and Anthropic provider-tool sanitization is re-run for the compaction provider. The assistant generation prompt is never mutated.
- TODO (#29436): the summary request carries the chat model's tool definitions only when the override resolves to the chat model itself (same provider instance and model); any other override sends the summary without tool definitions.
- Observability: compaction metrics and chat debug runs record the provider and model that actually generated the summary. This includes the "still over limit" terminal error, which is recorded before the override client is built: prepare-time resolution keeps the override's provider/model identity so that error lands on the same metric series as the compact action's own events.
#### Interrupt goroutine
@@ -1016,6 +1017,8 @@ The worker periodically archives old, unused chats.
Compaction reduces the LLM prompt size by summarizing older history into a compressed boundary. It normally runs automatically: while preparing a generation, the worker compares the latest known token usage against the model's compaction threshold, and when the threshold is exceeded it makes a non-streaming LLM call to produce a summary and commits it as a compressed message triplet (a hidden model-only summary boundary, a visible `chat_summarized` tool call, and its tool result). Prompt queries prune history at the newest boundary.
TODO (#29436): the summary request now carries the turn's tool definitions (and, on Anthropic, the same cache-control breakpoints as the turn) so it shares the turn's cached prefix. When the provider rejects that request for its size (context window or HTTP 413), the summary is regenerated once without tool definitions. Describe this here.
Trailing user messages the assistant has not answered yet are not summarized: they are excluded from the summarizer's input and re-committed after the triplet as model-only user rows, so the pruned prompt keeps them verbatim instead of relying on summary fidelity.
Users can also request a compaction on demand via `POST /api/experimental/chats/{chat}/compact` (surfaced in the web UI as the `/compact` slash command). Manual compaction is a durable one-shot request executed through the normal worker loop rather than synchronously in the HTTP handler. This reuses the worker's lock fencing, retry accounting, streamed "Summarizing..." progress parts, metrics, and debug runs, and it survives replica crashes. The flow:
+15 -1
View File
@@ -6283,6 +6283,7 @@ func TestActiveServer_CompactionModelOverride(t *testing.T) {
name string
overrideModel string
effort string
keepsTools bool
assertSummaryRequest func(t *testing.T, req *chattest.AnthropicRequest)
}{
{
@@ -6316,6 +6317,15 @@ func TestActiveServer_CompactionModelOverride(t *testing.T) {
require.Contains(t, string(req.Thinking), `"type":"adaptive"`)
},
},
{
// Only an override resolving to the chat model itself shares
// its prompt cache, so only then do the tool definitions stay.
name: "override resolves to the chat model",
overrideModel: chatModelName,
effort: "low",
keepsTools: true,
assertSummaryRequest: func(*testing.T, *chattest.AnthropicRequest) {},
},
}
for _, tc := range routingCases {
@@ -6331,7 +6341,11 @@ func TestActiveServer_CompactionModelOverride(t *testing.T) {
if !req.Stream {
if strings.Contains(body, "You are performing a context compaction") {
require.Equal(t, tc.overrideModel, req.Model)
require.Empty(t, req.Tools, "an override model shares no prompt cache, so the summary carries no tool definitions")
if tc.keepsTools {
require.NotEmpty(t, req.Tools)
} else {
require.Empty(t, req.Tools, "a different model shares no prompt cache, so the summary carries no tool definitions")
}
tc.assertSummaryRequest(t, req)
return anthropicCompactionResponse(compactionSummary)
}
+7 -3
View File
@@ -1023,9 +1023,13 @@ func (s *taskStarter) generateCompaction(
compactionOpts.ModelConfigID = overrideModel.dbConfig.ID
compactionOpts.SummaryCall = compactionSummaryCall(overrideModel)
// Prompt caches are model-scoped and provider-native tools are
// model-specific, so the chat model's tool definitions buy the
// override request nothing and can get it rejected.
compactionOpts.ToolDefinitions = nil
// model-specific, so unless the override resolves to the chat model
// itself its definitions buy the request nothing and can get it
// rejected.
if !sameCompactionProviderIdentity(prepared.Compaction.ChatModelConfig, overrideModel.dbConfig) ||
overrideModel.resolvedModel != prepared.Compaction.Options.ResolvedModel {
compactionOpts.ToolDefinitions = nil
}
compactionOpts.Messages = sanitizeCompactionPrompt(
ctx,
logger,