Files
coder/coderd/x/chatd/chatloop/compaction_internal_test.go
T
Michael Suchacz c28ae0dc9b feat: trigger compaction independently for the chat and compaction models (#28568)
> [!NOTE]
> Xum (an AI agent) authored this pull request on Mike's behalf.

## Summary

Automatic compaction now fires at whichever of two independent triggers
is reached first, and the UI shows when the organization override's
trigger applies.

- **Chat trigger**: the chat owner's personal threshold (or the chat
model's) against the chat model's window. A threshold of 100 turns off
only this trigger.
- **Organization trigger**: a fixed 80% of the organization compaction
override model's window. The override model's own compression threshold
is ignored. The other 20% is a target margin for the summary prompt and
the summary, not a guarantee, because the trigger reads the previous
step's prompt size.

On main, the chat threshold was applied to the smaller of the two
windows, and nothing in the UI showed it. Example: with a 20K chat model
at 70% and an 8K override, main compacts at 5,600 tokens; this PR
compacts at 6,400 (80% of 8K, shown as 32%).

## Changes

Backend (`coderd/x/chatd`):

- The trigger with the lower token point binds (ties go to the chat
trigger). It drives the automatic compaction gate and the
still-over-limit check, and every summary, automatic or manual, records
it.
- An override that is unset, deleted, disabled, on a disabled provider,
or without credentials for the chat owner is treated as absent: the chat
model summarizes and only the chat trigger applies.
- The summary's output cap is bounded by the summarizer's own window,
not the trigger's.
- The still-over-limit error names what can clear it: the compaction
threshold (chat trigger), an override model with a larger window
(organization trigger), or both when usage is over both points.
- The still-over-limit check compacts once more when the latest summary
was recorded under a different threshold or trigger window. The
`chat_summarized` result gains `trigger_context_limit_tokens`, and
`context_limit_tokens` now records the chat model's window (main
recorded the smaller window). On main, lowering a threshold after a
compaction ended every later turn with the error.
- The still-over-limit check skips a 0% threshold ("always compact"),
which on main ended the run with the error after every compaction.
- `ARCHITECTURE.md` documents the two triggers, the summary cap, and the
still-over-limit rules.

Frontend (`site`), through a shared `compactionTriggers` helper that
mirrors the backend rules (no new API):

- Coder Agents settings: the compaction override row warns when the
organization trigger undercuts chat models' default triggers ("the
organization override compacts chats at 80% of {model}'s {N}-token
window").
- Personal thresholds page: an Effective column shows the trigger that
applies, with a popover when the organization override binds. "Compacts
at" follows the same trigger, and a point the chat model's window cannot
reach shows as Off. The header's 100% sentence, and the row warning when
a member types 100, say an organization override may still compact chats
when one is set or its organization's overrides failed to load (the
header also while they load).
- Chat context gauge: the compact-at percent follows the binding trigger
and is annotated when the organization override binds. If the overrides
request fails, it shows the chat-only value with "(organization override
not loaded)", or "Compaction off (organization override not loaded)"
when the chat trigger is off.
- An override whose provider the user cannot use is ignored on the gauge
and the personal page, matching chatd.
- The shared `formatContextLimit` prints counts below 1,000 exactly (an
800-token compaction point showed "~1K") and 999,500 to 999,999 as
"1.0M" instead of "1000K". It also labels the model pickers; their other
values are unchanged.
- `HelpPopoverIconTrigger` lets callers set their own `aria-label`; only
the Effective column's popover does.

Known v1 limits: the Coder Agents settings alert is organization-scoped,
so it can warn about an override that some members fall back from for
lack of provider credentials. The gauge checks the viewer's provider
access, so on a shared chat its label can differ from the owner's
runtime trigger (Codex P2, declined). Compaction itself is unaffected.

## Screenshots

From remote UAT round 13 on 093076f402: an isolated test deployment
with a local fake LLM. The models are test fixtures named for their
context windows: Chat model 20K (own threshold 70%), Chat model 5K
(40%), and Chat model 8K (50%). Chat model 8K is also the organization
compaction override, so the organization trigger is 80% of 8,000 = 6,400
tokens, which is 32% of Chat model 20K's window. Chat model 8K's own 50%
applies only to chats that use it as the chat model; changing it to 10%
or 100% left every value below unchanged.

**Coder Agents settings:** the organization trigger undercuts Chat model
20K's 70% default.

![Override warning: 80% of Chat model 8K's window undercuts Chat model
20K](https://github.com/user-attachments/assets/188c92e6-faaa-4e9a-a563-567cbf98af85)

**Personal thresholds:** the organization trigger binds Chat model 20K
at 32%. Chat model 5K (40%) and Chat model 8K (50%) keep their own chat
triggers.

![Effective column: 32% from the organization trigger on Chat model
20K](https://github.com/user-attachments/assets/9949be4e-9417-4cfc-a683-f1367a1dd77a)
![Organization override popover: 80% of Chat model 8K's window, about
32% of this model's
window](https://github.com/user-attachments/assets/4bc3e002-0be1-46fd-91f0-aa27503034af)

**Personal thresholds, organization settings request failed** (the first
red line is the injected error). Rows fall back to chat-only values.

![Organization settings load failure above usable chat-only
rows](https://github.com/user-attachments/assets/dcf48546-aa47-4f82-a6eb-85f106b4bf45)

**Chat context gauge:** organization trigger binds (Chat model 20K),
chat trigger binds (Chat model 5K), overrides request failed (injected).

![Gauge: Compacts at 32% (organization
override)](https://github.com/user-attachments/assets/cd5402d9-9849-4254-a7a7-6623c4006a2a)
![Gauge: Compacts at
40%](https://github.com/user-attachments/assets/f372c9e9-72ef-41c8-950c-e77d74965f65)
![Gauge: Compacts at 70% (organization override not
loaded)](https://github.com/user-attachments/assets/ab829746-2ab5-4743-848f-dce4e806de3a)

**Still-over-limit errors from real runs:** organization trigger, chat
trigger, both.

![Organization-only over-limit
error](https://github.com/user-attachments/assets/f210807a-a2cc-47a0-807c-f1404373bb28)
![Chat-only over-limit
error](https://github.com/user-attachments/assets/eecec38e-97de-49b6-8670-b32090103bbb)
![Combined over-limit
error](https://github.com/user-attachments/assets/459a6d08-7215-492d-93f5-96a4f767a09e)

## Testing

- Go: `TestBindingCompactionTriggerSource`,
`TestCompactionStillOverLimitError`, `TestCompactionStatusFromHistory`,
and `TestActiveServer_CompactionModelOverride` cover the 80% point
(fails if the override's own threshold is read again), user threshold
100 leaving the organization trigger on, the summary cap for automatic
and manual compaction, the over-limit messages, and recompaction after a
trigger change. The site binding tests mirror the Go table row for row.
- Site: `compactionTriggers` unit tests and Storybook stories for the
three surfaces, including the overrides fetch failure.
- On c6f8750479: `go vet`, the `chatd` and `chatloop` tests, and the
AgentsPage, AISettingsPage, and HelpPopover unit tests (2,274 passed, 2
skipped) pass, and CI is green. On 987cefe399, tsc, the five affected
story files (109 passed), and `make pre-commit` also passed. 093076f402
only changes `ARCHITECTURE.md`; the pre-commit hook's markdown format
and lint checks pass on it.
- On 3cd5b08f1f: tsc, the new `formatContextLimit` test (fails on the
old helper for 800 and 999,999), the `compactionTriggers` and `aiModels`
unit tests (224 passed), and the affected story files pass, and the
pre-commit hook passed on all three commits.

## Review record

- Earlier heads: 12 Codex passes and 3 human reviews, all threads
answered. Rebased onto main several times, most recently ff3fa22f68
(kept #30274, #29892, and #29890 in conflicted files).
- coder-agents-review rounds 1 to 18: findings fixed; approved since
round 15. The round 1 `ARCHITECTURE.md` finding was left for the
maintainer, who asked for the update in the last commit.
- coder-frontend-review (eight roles): all pass. Splitting the PR (RO6)
was declined because it is reviewed as one unit; the
`organizationTriggerFor` story helper stays because its three fields
must match the model.
- Design change after UAT round 11: the organization trigger is a fixed
80% of the override window instead of the override model's own threshold
(171d5f568e, 21d2ce7fce, 987cefe399).
- coder-agents-review round 19: copy, naming, and an unreachable
zero-window gate fixed (6e8efb9054, c6f8750479). A summary-time fit
check with chat-model fallback (CRF-156) was declined: the previous-step
lag applies to every trigger, including main's. Round 20 approved
c6f8750479 with no findings.
- Codex: one P2 on 987cefe399 declined (see known limits); no major
issues on c6f8750479.
- Remote UAT on dogfood (fake LLM, isolated test deployment): rounds 8
to 13 PASS. Round 12 on c6f8750479 first failed one criterion that was
written wrong (the admin alert correctly lists an override model that is
also a chat model at 100%); the reviewer endorsed PASS on the same
evidence after the correction. Round 13 on 093076f402 retook the
screenshots with neutral model names and confirmed that the override
model's own threshold (10%, 50%, 100%) does not move the organization
trigger.
- UAT round 13 notes: the header after a failed or pending overrides
load and the sub-1K labels are fixed in 4e172ab5ff and d5888ce15dc;
coder-agents-review round 21 found the same gap in the row warning
(CRF-167, fixed in 3cd5b08f1f). Left as on main: a summary's
`usage_percent` is measured against the binding trigger's window, so it
can exceed 100 (nothing reads it), and the 100% help text shows only for
an unsaved edit.
- Codex: no major issues on d5888ce15d and 3cd5b08f1f.
coder-agents-review round 22 approved 3cd5b08f1f with no open findings.
- coder-frontend-review on those commits: all roles pass after the
fixes. Declined: moving the formatter fix to its own PR (RO5; the owner
asked for it here), adding "tokens" to sub-1K labels (TN6; "~800" reads
like "~16K"), and splitting the component (RP6).
2026-10-11 12:10:59 +02:00

1056 lines
36 KiB
Go

package chatloop
import (
"context"
"encoding/json"
"net/http"
"strings"
"testing"
"time"
"charm.land/fantasy"
fantasyanthropic "charm.land/fantasy/providers/anthropic"
fantasyopenai "charm.land/fantasy/providers/openai"
"github.com/google/uuid"
"github.com/sqlc-dev/pqtype"
"github.com/stretchr/testify/require"
"go.uber.org/mock/gomock"
"golang.org/x/xerrors"
"github.com/coder/coder/v2/coderd/database"
"github.com/coder/coder/v2/coderd/database/dbmock"
"github.com/coder/coder/v2/coderd/x/chatd/chatdebug"
"github.com/coder/coder/v2/coderd/x/chatd/chaterror"
"github.com/coder/coder/v2/coderd/x/chatd/chattest"
"github.com/coder/coder/v2/codersdk"
"github.com/coder/coder/v2/testutil"
"github.com/coder/quartz"
)
func compactionStream(parts ...fantasy.StreamPart) fantasy.StreamResponse {
return func(yield func(fantasy.StreamPart) bool) {
for _, part := range parts {
if !yield(part) {
return
}
}
}
}
func summaryTextStream() fantasy.StreamResponse {
return compactionStream(
fantasy.StreamPart{Type: fantasy.StreamPartTypeTextStart, ID: "text"},
fantasy.StreamPart{Type: fantasy.StreamPartTypeTextDelta, ID: "text", Delta: "summary"},
fantasy.StreamPart{Type: fantasy.StreamPartTypeTextEnd, ID: "text"},
fantasy.StreamPart{Type: fantasy.StreamPartTypeFinish, ID: "msg_summary", FinishReason: fantasy.FinishReasonStop},
)
}
func TestStartCompactionDebugRun_DoesNotReportDebugErrors(t *testing.T) {
t.Parallel()
newParentContext := func(chatID uuid.UUID) context.Context {
return chatdebug.ContextWithRun(context.Background(), &chatdebug.RunContext{
RunID: uuid.New(),
ChatID: chatID,
RootChatID: uuid.New(),
ParentChatID: uuid.New(),
ModelConfigID: uuid.New(),
TriggerMessageID: 41,
HistoryTipMessageID: 42,
Provider: "fake-provider",
Model: "fake-model",
})
}
t.Run("CreateRun", func(t *testing.T) {
t.Parallel()
ctrl := gomock.NewController(t)
db := dbmock.NewMockStore(ctrl)
svc := chatdebug.NewService(db, testutil.Logger(t), nil)
chatID := uuid.New()
db.EXPECT().InsertChatDebugRun(
gomock.Any(),
gomock.AssignableToTypeOf(database.InsertChatDebugRunParams{}),
).Return(database.ChatDebugRun{}, xerrors.New("insert compaction debug run"))
ctx := newParentContext(chatID)
compactionCtx, finish := startCompactionDebugRun(ctx, CompactionOptions{
DebugSvc: svc,
ChatID: chatID,
})
require.Same(t, ctx, compactionCtx)
finish(nil)
})
t.Run("FinalizeRunAggregatesSummary", func(t *testing.T) {
t.Parallel()
ctrl := gomock.NewController(t)
db := dbmock.NewMockStore(ctrl)
svc := chatdebug.NewService(db, testutil.Logger(t), nil)
chatID := uuid.New()
runID := uuid.New()
usageJSON, err := json.Marshal(fantasy.Usage{InputTokens: 7, OutputTokens: 3})
require.NoError(t, err)
attemptsJSON, err := json.Marshal([]chatdebug.Attempt{{
Status: "completed",
Method: "POST",
Path: "/v1/messages",
}})
require.NoError(t, err)
db.EXPECT().InsertChatDebugRun(
gomock.Any(),
gomock.AssignableToTypeOf(database.InsertChatDebugRunParams{}),
).Return(database.ChatDebugRun{ //nolint:exhaustruct // Test only needs IDs.
ID: runID,
ChatID: chatID,
}, nil)
db.EXPECT().GetChatDebugStepsByRunID(gomock.Any(), runID).Return([]database.ChatDebugStep{{
ID: uuid.New(),
RunID: runID,
ChatID: chatID,
Status: string(chatdebug.StatusCompleted),
Usage: pqtype.NullRawMessage{RawMessage: usageJSON, Valid: true},
Attempts: attemptsJSON,
}}, nil)
db.EXPECT().UpdateChatDebugRun(
gomock.Any(),
gomock.AssignableToTypeOf(database.UpdateChatDebugRunParams{}),
).DoAndReturn(func(_ context.Context, params database.UpdateChatDebugRunParams) (database.ChatDebugRun, error) {
require.Equal(t, chatID, params.ChatID)
require.Equal(t, runID, params.ID)
require.True(t, params.Summary.Valid)
require.JSONEq(t, `{"endpoint_label":"POST /v1/messages","step_count":1,"total_input_tokens":7,"total_output_tokens":3}`,
string(params.Summary.RawMessage))
return database.ChatDebugRun{ID: runID, ChatID: chatID}, nil
})
ctx := newParentContext(chatID)
compactionCtx, finish := startCompactionDebugRun(ctx, CompactionOptions{
DebugSvc: svc,
ChatID: chatID,
})
require.NotSame(t, ctx, compactionCtx)
finish(nil)
})
t.Run("FinalizeRun", func(t *testing.T) {
t.Parallel()
ctrl := gomock.NewController(t)
db := dbmock.NewMockStore(ctrl)
svc := chatdebug.NewService(db, testutil.Logger(t), nil)
chatID := uuid.New()
runID := uuid.New()
db.EXPECT().InsertChatDebugRun(
gomock.Any(),
gomock.AssignableToTypeOf(database.InsertChatDebugRunParams{}),
).Return(database.ChatDebugRun{ //nolint:exhaustruct // Test only needs IDs.
ID: runID,
ChatID: chatID,
}, nil)
db.EXPECT().GetChatDebugStepsByRunID(gomock.Any(), runID).Return(nil, xerrors.New("aggregate compaction debug run"))
db.EXPECT().UpdateChatDebugRun(
gomock.Any(),
gomock.AssignableToTypeOf(database.UpdateChatDebugRunParams{}),
).Return(database.ChatDebugRun{}, xerrors.New("finalize compaction debug run"))
ctx := newParentContext(chatID)
compactionCtx, finish := startCompactionDebugRun(ctx, CompactionOptions{
DebugSvc: svc,
ChatID: chatID,
})
require.NotSame(t, ctx, compactionCtx)
finish(nil)
})
}
// TestGenerateCompactionSummary_PanicFinalizesAsError verifies that a
// panic originating inside the model call during compaction is
// captured by the deferred debug-run finalizer so the run is recorded
// with StatusError rather than StatusCompleted. Without the recover
// hook the named `err` return is still nil when the defer fires and
// the row silently misclassifies the crash path.
func TestGenerateCompactionSummary_PanicFinalizesAsError(t *testing.T) {
t.Parallel()
ctrl := gomock.NewController(t)
db := dbmock.NewMockStore(ctrl)
svc := chatdebug.NewService(db, testutil.Logger(t), nil)
chatID := uuid.New()
runID := uuid.New()
status := make(chan string, 1)
db.EXPECT().InsertChatDebugRun(
gomock.Any(),
gomock.AssignableToTypeOf(database.InsertChatDebugRunParams{}),
).Return(database.ChatDebugRun{
ID: runID,
ChatID: chatID,
}, nil)
db.EXPECT().GetChatDebugStepsByRunID(gomock.Any(), runID).Return(nil, nil)
db.EXPECT().UpdateChatDebugRun(
gomock.Any(),
gomock.AssignableToTypeOf(database.UpdateChatDebugRunParams{}),
).DoAndReturn(func(_ context.Context, params database.UpdateChatDebugRunParams) (database.ChatDebugRun, error) {
status <- params.Status.String
return database.ChatDebugRun{ID: runID, ChatID: chatID}, nil
})
model := &chattest.FakeModel{
ProviderName: "fake",
StreamFn: func(_ context.Context, _ fantasy.Call) (fantasy.StreamResponse, error) {
panic("compaction model crash")
},
}
parentCtx := chatdebug.ContextWithRun(context.Background(), &chatdebug.RunContext{
RunID: uuid.New(),
ChatID: chatID,
ModelConfigID: uuid.New(),
TriggerMessageID: 1,
HistoryTipMessageID: 2,
Provider: "fake",
Model: "fake-model",
})
require.PanicsWithValue(t, "compaction model crash", func() {
_, _, _ = generateCompactionSummary(parentCtx, model,
[]fantasy.Message{textMessage(fantasy.MessageRoleUser, "hello")},
CompactionOptions{
DebugSvc: svc,
ChatID: chatID,
SummaryPrompt: "summarize",
})
})
select {
case s := <-status:
require.Equal(t, string(chatdebug.StatusError), s,
"panic path must finalize the debug run with StatusError")
case <-time.After(testutil.WaitShort):
t.Fatal("FinalizeRun never reached UpdateChatDebugRun on panic")
}
}
func TestGenerateCompactionSummaryPreservesCallOptions(t *testing.T) {
t.Parallel()
temperature := 0.2
topP := 0.8
topK := int64(40)
presencePenalty := 0.1
frequencyPenalty := 0.3
reasoningEffort := fantasyopenai.ReasoningEffortMedium
providerOptions := fantasy.ProviderOptions{
fantasyopenai.Name: &fantasyopenai.ResponsesProviderOptions{
ReasoningEffort: &reasoningEffort,
},
}
toolDefinitions := []fantasy.Tool{
fantasy.FunctionTool{Name: "read_file", InputSchema: map[string]any{"type": "object"}},
fantasy.ProviderDefinedTool{ID: "web_search", Name: "web_search"},
}
messages := []fantasy.Message{
textMessage(fantasy.MessageRoleSystem, "system prefix"),
textMessage(fantasy.MessageRoleUser, "hello"),
}
originalMessages := append([]fantasy.Message(nil), messages...)
var got fantasy.Call
model := &chattest.FakeModel{
ProviderName: "fake",
ModelName: "fake-model",
StreamFn: func(_ context.Context, call fantasy.Call) (fantasy.StreamResponse, error) {
got = call
return summaryTextStream(), nil
},
}
toolChoice := fantasy.ToolChoiceNone
summary, _, err := generateCompactionSummary(context.Background(), model, messages, CompactionOptions{
SummaryPrompt: "summarize",
SummaryCall: fantasy.Call{
Temperature: &temperature,
TopP: &topP,
TopK: &topK,
PresencePenalty: &presencePenalty,
FrequencyPenalty: &frequencyPenalty,
ProviderOptions: providerOptions,
ToolChoice: &toolChoice,
},
ToolDefinitions: toolDefinitions,
})
require.NoError(t, err)
require.Equal(t, "summary", summary)
require.Equal(t, &temperature, got.Temperature)
require.Equal(t, &topP, got.TopP)
require.Equal(t, &topK, got.TopK)
require.Equal(t, &presencePenalty, got.PresencePenalty)
require.Equal(t, &frequencyPenalty, got.FrequencyPenalty)
require.Equal(t, providerOptions, got.ProviderOptions)
require.Equal(t, toolDefinitions, got.Tools)
require.NotNil(t, got.ToolChoice)
require.Equal(t, fantasy.ToolChoiceNone, *got.ToolChoice)
require.Nil(t, got.MaxOutputTokens)
require.Len(t, got.Prompt, 3)
require.Equal(t, originalMessages, []fantasy.Message(got.Prompt[:2]))
require.Equal(t, fantasy.MessageRoleUser, got.Prompt[2].Role)
require.Equal(t, []fantasy.MessagePart{fantasy.TextPart{Text: "summarize"}}, got.Prompt[2].Content)
require.Equal(t, originalMessages, messages)
}
func TestGenerateCompactionSummaryUsesToolDefinitions(t *testing.T) {
t.Parallel()
toolDefinitions := []fantasy.Tool{
fantasy.FunctionTool{
Name: "read_file",
Description: "Read a file.",
InputSchema: map[string]any{"type": "object"},
},
fantasy.ProviderDefinedTool{ID: "web_search", Name: "web_search"},
}
messages := []fantasy.Message{textMessage(fantasy.MessageRoleUser, "hello")}
originalMessages := append([]fantasy.Message(nil), messages...)
var got fantasy.Call
model := &chattest.FakeModel{
ProviderName: "fake",
ModelName: "fake-model",
StreamFn: func(_ context.Context, call fantasy.Call) (fantasy.StreamResponse, error) {
got = call
return summaryTextStream(), nil
},
}
toolChoice := fantasy.ToolChoiceNone
summary, _, err := generateCompactionSummary(context.Background(), model, messages, CompactionOptions{
SummaryPrompt: "summarize",
SummaryCall: fantasy.Call{ToolChoice: &toolChoice},
ToolDefinitions: toolDefinitions,
})
require.NoError(t, err)
require.Equal(t, "summary", summary)
require.Equal(t, toolDefinitions, got.Tools)
require.NotNil(t, got.ToolChoice)
require.Equal(t, toolChoice, *got.ToolChoice)
require.Len(t, got.Prompt, 2)
require.Equal(t, originalMessages, messages)
}
func TestGenerateCompactionSummaryAppliesAnthropicPromptCaching(t *testing.T) {
t.Parallel()
messages := []fantasy.Message{
textMessage(fantasy.MessageRoleSystem, "system"),
textMessage(fantasy.MessageRoleUser, "hello"),
textMessage(fantasy.MessageRoleAssistant, "hi"),
}
originalMessages := append([]fantasy.Message(nil), messages...)
var got fantasy.Call
model := &chattest.FakeModel{
ProviderName: fantasyanthropic.Name,
ModelName: "claude",
StreamFn: func(_ context.Context, call fantasy.Call) (fantasy.StreamResponse, error) {
got = call
return summaryTextStream(), nil
},
}
_, _, err := generateCompactionSummary(context.Background(), model, messages, CompactionOptions{
SummaryPrompt: "summarize",
})
require.NoError(t, err)
require.Len(t, got.Prompt, 4)
cacheControl := fantasy.ProviderOptions{
fantasyanthropic.Name: &fantasyanthropic.ProviderCacheControlOptions{
CacheControl: fantasyanthropic.CacheControl{Type: "ephemeral"},
},
}
// Breakpoints land on the system message and the final two messages.
require.Equal(t, cacheControl, got.Prompt[0].ProviderOptions)
require.Nil(t, got.Prompt[1].ProviderOptions)
require.Equal(t, cacheControl, got.Prompt[2].ProviderOptions)
require.Equal(t, cacheControl, got.Prompt[3].ProviderOptions)
require.Equal(t, originalMessages, messages)
}
func TestGenerateCompactionSummaryRetriesWithoutToolsWhenContextTooLarge(t *testing.T) {
t.Parallel()
toolDefinitions := []fantasy.Tool{
fantasy.FunctionTool{Name: "read_file", InputSchema: map[string]any{"type": "object"}},
}
toolChoice := fantasy.ToolChoiceNone
var calls []fantasy.Call
model := &chattest.FakeModel{
ProviderName: "fake",
ModelName: "fake-model",
StreamFn: func(_ context.Context, call fantasy.Call) (fantasy.StreamResponse, error) {
calls = append(calls, call)
if len(call.Tools) > 0 {
return nil, &fantasy.ProviderError{
Title: "bad request",
Message: "Your input exceeds the context window of this model. Please adjust your input and try again.",
StatusCode: http.StatusBadRequest,
}
}
return summaryTextStream(), nil
},
}
summary, responseID, err := generateCompactionSummary(context.Background(), model,
[]fantasy.Message{textMessage(fantasy.MessageRoleUser, "hello")},
CompactionOptions{
SummaryPrompt: "summarize",
SummaryCall: fantasy.Call{ToolChoice: &toolChoice},
ToolDefinitions: toolDefinitions,
},
)
require.NoError(t, err)
require.Equal(t, "summary", summary)
require.Equal(t, "msg_summary", responseID)
require.Len(t, calls, 2)
require.Equal(t, toolDefinitions, calls[0].Tools)
require.Nil(t, calls[1].Tools)
require.Equal(t, calls[0].Prompt, calls[1].Prompt)
require.Equal(t, calls[0].ToolChoice, calls[1].ToolChoice)
}
func TestGenerateCompactionSummaryDoesNotRetryOtherFailures(t *testing.T) {
t.Parallel()
contextTooLarge := &fantasy.ProviderError{
Message: "Your input exceeds the context window of this model.",
StatusCode: http.StatusBadRequest,
}
cases := []struct {
name string
toolDefinitions []fantasy.Tool
err error
}{
{
name: "unrelated bad request",
toolDefinitions: []fantasy.Tool{fantasy.FunctionTool{Name: "read_file"}},
err: &fantasy.ProviderError{
Message: "Unsupported parameter: 'temperature' is not supported with this model.",
StatusCode: http.StatusBadRequest,
},
},
{
name: "server error",
toolDefinitions: []fantasy.Tool{fantasy.FunctionTool{Name: "read_file"}},
err: &fantasy.ProviderError{
Message: "context window service unavailable",
StatusCode: http.StatusServiceUnavailable,
},
},
{
name: "non-provider error",
toolDefinitions: []fantasy.Tool{fantasy.FunctionTool{Name: "read_file"}},
err: xerrors.New("context window"),
},
{
name: "no tools to drop",
err: contextTooLarge,
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
calls := 0
model := &chattest.FakeModel{
ProviderName: "fake",
ModelName: "fake-model",
StreamFn: func(_ context.Context, _ fantasy.Call) (fantasy.StreamResponse, error) {
calls++
return nil, tc.err
},
}
_, _, err := generateCompactionSummary(context.Background(), model,
[]fantasy.Message{textMessage(fantasy.MessageRoleUser, "hello")},
CompactionOptions{
SummaryPrompt: "summarize",
ToolDefinitions: tc.toolDefinitions,
},
)
require.ErrorIs(t, err, tc.err)
require.Equal(t, 1, calls)
})
}
}
func TestIsContextTooLargeError(t *testing.T) {
t.Parallel()
cases := []struct {
name string
err error
want bool
}{
{
name: "parsed by fantasy",
err: &fantasy.ProviderError{
StatusCode: http.StatusBadRequest,
ContextTooLargeErr: true,
ContextMaxTokens: 16385,
ContextUsedTokens: 20000,
},
want: true,
},
{
name: "openai responses wording",
err: &fantasy.ProviderError{
StatusCode: http.StatusBadRequest,
Message: "Your input exceeds the context window of this model. Please adjust your input and try again.",
},
want: true,
},
{
name: "anthropic wording",
err: &fantasy.ProviderError{
StatusCode: http.StatusBadRequest,
Message: "prompt is too long: 213462 tokens > 200000 maximum",
},
want: true,
},
{
name: "openai error code in response body",
err: &fantasy.ProviderError{
StatusCode: http.StatusBadRequest,
Message: "Request too large for this model.",
ResponseBody: []byte(`{"error":{"message":"Request too large for this model.","type":"invalid_request_error","code":"context_length_exceeded"}}`),
},
want: true,
},
{
name: "wrapped provider error",
err: xerrors.Errorf("generate summary text: %w", &fantasy.ProviderError{
StatusCode: http.StatusBadRequest,
Message: "This model's maximum context length is 16385 tokens.",
}),
want: true,
},
{
name: "request entity too large",
err: &fantasy.ProviderError{
StatusCode: http.StatusRequestEntityTooLarge,
Message: "Request Entity Too Large",
},
want: true,
},
{
name: "unrelated bad request",
err: &fantasy.ProviderError{
StatusCode: http.StatusBadRequest,
Message: "Invalid schema for function 'read_file'.",
},
want: false,
},
{
name: "matching wording on a non bad-request status",
err: &fantasy.ProviderError{
StatusCode: http.StatusTooManyRequests,
Message: "context window tokens per minute exceeded",
},
want: false,
},
{
name: "non-provider error",
err: xerrors.New("prompt is too long"),
want: false,
},
{
name: "nil",
err: nil,
want: false,
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
require.Equal(t, tc.want, isContextTooLargeError(tc.err))
})
}
}
func TestGenerateCompactionSummary_UsesCallerContext(t *testing.T) {
t.Parallel()
type contextKey string
testCtx := context.WithValue(context.Background(), contextKey("key"), "value")
var (
ctxSeen context.Context
errAtCall error
)
model := &chattest.FakeModel{
ProviderName: "fake",
StreamFn: func(ctx context.Context, _ fantasy.Call) (fantasy.StreamResponse, error) {
ctxSeen = ctx
errAtCall = ctx.Err()
return compactionStream(
fantasy.StreamPart{Type: fantasy.StreamPartTypeTextStart, ID: "text"},
fantasy.StreamPart{Type: fantasy.StreamPartTypeTextDelta, ID: "text", Delta: "summary"},
fantasy.StreamPart{Type: fantasy.StreamPartTypeTextEnd, ID: "text"},
fantasy.StreamPart{Type: fantasy.StreamPartTypeFinish, FinishReason: fantasy.FinishReasonStop},
), nil
},
}
summary, _, err := generateCompactionSummary(testCtx, model,
[]fantasy.Message{textMessage(fantasy.MessageRoleUser, "hello")},
CompactionOptions{SummaryPrompt: "summarize"},
)
require.NoError(t, err)
require.Equal(t, "summary", summary)
// The silence guard wraps the caller context in a cancelable child that
// is released after the summary completes, so assert inheritance rather
// than identity: values propagate, the context is live at call time, and
// no deadline is attached (the guard uses a timer, not a context
// deadline).
require.NoError(t, errAtCall)
_, ok := ctxSeen.Deadline()
require.False(t, ok)
require.Equal(t, "value", ctxSeen.Value(contextKey("key")))
}
func TestGenerateCompactionSummary_Stream(t *testing.T) {
t.Parallel()
t.Run("joins text blocks", func(t *testing.T) {
t.Parallel()
model := &chattest.FakeModel{
ProviderName: "fake",
StreamFn: func(_ context.Context, _ fantasy.Call) (fantasy.StreamResponse, error) {
return compactionStream(
fantasy.StreamPart{Type: fantasy.StreamPartTypeTextStart, ID: "first"},
fantasy.StreamPart{Type: fantasy.StreamPartTypeTextDelta, ID: "first", Delta: " first "},
fantasy.StreamPart{Type: fantasy.StreamPartTypeTextEnd, ID: "first"},
fantasy.StreamPart{Type: fantasy.StreamPartTypeTextStart, ID: "second"},
fantasy.StreamPart{Type: fantasy.StreamPartTypeTextDelta, ID: "second", Delta: " second "},
fantasy.StreamPart{Type: fantasy.StreamPartTypeTextEnd, ID: "second"},
fantasy.StreamPart{Type: fantasy.StreamPartTypeFinish, FinishReason: fantasy.FinishReasonStop},
), nil
},
}
summary, _, err := generateCompactionSummary(context.Background(), model, nil, CompactionOptions{})
require.NoError(t, err)
require.Equal(t, "first second", summary)
})
t.Run("reports output cap truncation", func(t *testing.T) {
t.Parallel()
model := &chattest.FakeModel{
ProviderName: "fake",
StreamFn: func(_ context.Context, _ fantasy.Call) (fantasy.StreamResponse, error) {
return compactionStream(
fantasy.StreamPart{Type: fantasy.StreamPartTypeReasoningStart, ID: "reasoning"},
fantasy.StreamPart{Type: fantasy.StreamPartTypeReasoningDelta, ID: "reasoning", Delta: "thinking"},
fantasy.StreamPart{Type: fantasy.StreamPartTypeReasoningEnd, ID: "reasoning"},
fantasy.StreamPart{Type: fantasy.StreamPartTypeFinish, FinishReason: fantasy.FinishReasonLength},
), nil
},
}
summary, _, err := generateCompactionSummary(context.Background(), model, nil, CompactionOptions{})
require.Empty(t, summary)
require.EqualError(t, err, "compaction summary was truncated at the output token cap")
})
t.Run("rejects stream without finish part", func(t *testing.T) {
t.Parallel()
model := &chattest.FakeModel{
ProviderName: "fake",
StreamFn: func(_ context.Context, _ fantasy.Call) (fantasy.StreamResponse, error) {
return compactionStream(
fantasy.StreamPart{Type: fantasy.StreamPartTypeTextStart, ID: "text"},
fantasy.StreamPart{Type: fantasy.StreamPartTypeTextDelta, ID: "text", Delta: "partial summary"},
), nil
},
}
summary, _, err := generateCompactionSummary(context.Background(), model, nil, CompactionOptions{})
require.Empty(t, summary)
require.EqualError(t, err, "compaction summary stream ended without a finish part")
})
t.Run("classifies canceled stream open as transport reset", func(t *testing.T) {
t.Parallel()
model := &chattest.FakeModel{
ProviderName: "fake",
StreamFn: func(_ context.Context, _ fantasy.Call) (fantasy.StreamResponse, error) {
return nil, context.Canceled
},
}
summary, _, err := generateCompactionSummary(context.Background(), model, nil, CompactionOptions{
ResolvedProvider: "fake",
})
require.Empty(t, summary)
require.ErrorIs(t, err, chaterror.ErrProviderTransportReset)
classified := chaterror.Classify(err)
require.True(t, classified.Retryable)
require.Equal(t, "fake", classified.Provider)
})
t.Run("classifies canceled error part as transport reset", func(t *testing.T) {
t.Parallel()
model := &chattest.FakeModel{
ProviderName: "fake",
StreamFn: func(_ context.Context, _ fantasy.Call) (fantasy.StreamResponse, error) {
return compactionStream(
fantasy.StreamPart{Type: fantasy.StreamPartTypeError, Error: context.Canceled},
), nil
},
}
summary, _, err := generateCompactionSummary(context.Background(), model, nil, CompactionOptions{
ResolvedProvider: "fake",
})
require.Empty(t, summary)
require.ErrorIs(t, err, chaterror.ErrProviderTransportReset)
classified := chaterror.Classify(err)
require.True(t, classified.Retryable)
require.Equal(t, "fake", classified.Provider)
})
t.Run("classifies stream silence", func(t *testing.T) {
t.Parallel()
ctx := testutil.Context(t, testutil.WaitLong)
clock := quartz.NewMock(t)
trap := clock.Trap().AfterFunc(streamSilenceGuardTimerTag)
defer trap.Close()
model := &chattest.FakeModel{
ProviderName: "openai",
StreamFn: func(ctx context.Context, _ fantasy.Call) (fantasy.StreamResponse, error) {
return func(yield func(fantasy.StreamPart) bool) {
<-ctx.Done()
}, nil
},
}
done := make(chan error, 1)
go func() {
_, _, err := generateCompactionSummary(context.Background(), model, nil, CompactionOptions{
ResolvedProvider: "openai",
Clock: clock,
StreamSilenceTimeout: 5 * time.Millisecond,
})
done <- err
}()
trap.MustWait(ctx).MustRelease(ctx)
_, waiter := clock.AdvanceNext()
waiter.MustWait(ctx)
err := <-done
require.Error(t, err)
classified := chaterror.Classify(err)
require.Equal(t, codersdk.ChatErrorKindStreamSilenceTimeout, classified.Kind)
require.Equal(t, "openai", classified.Provider)
require.True(t, classified.Retryable)
})
}
// TestGenerateCompaction_ForceBypassesThresholdGates verifies the
// manual-compaction contract: Force runs the summary even when usage
// is below threshold, when usage is zero, and when threshold=100
// disables automatic compaction; without Force those gates return an
// empty result without calling the model.
func TestGenerateCompaction_ForceBypassesThresholdGates(t *testing.T) {
t.Parallel()
newModel := func(calls *int) *chattest.FakeModel {
return &chattest.FakeModel{
ProviderName: "fake",
ModelName: "fake-model",
StreamFn: func(_ context.Context, _ fantasy.Call) (fantasy.StreamResponse, error) {
*calls++
return compactionStream(
fantasy.StreamPart{Type: fantasy.StreamPartTypeTextStart, ID: "text"},
fantasy.StreamPart{Type: fantasy.StreamPartTypeTextDelta, ID: "text", Delta: "forced summary"},
fantasy.StreamPart{Type: fantasy.StreamPartTypeTextEnd, ID: "text"},
fantasy.StreamPart{Type: fantasy.StreamPartTypeFinish, FinishReason: fantasy.FinishReasonStop},
), nil
},
}
}
messages := []fantasy.Message{textMessage(fantasy.MessageRoleUser, "hello")}
cases := []struct {
name string
opts GenerateCompactionOptions
}{
{
name: "below threshold",
opts: GenerateCompactionOptions{
ThresholdPercent: 70,
ContextLimit: 1000,
StepUsage: fantasy.Usage{InputTokens: 10},
},
},
{
name: "zero usage",
opts: GenerateCompactionOptions{
ThresholdPercent: 70,
ContextLimit: 1000,
},
},
{
name: "threshold disabled",
opts: GenerateCompactionOptions{
ThresholdPercent: 100,
ContextLimit: 1000,
StepUsage: fantasy.Usage{InputTokens: 10},
},
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
// Without Force the gate returns an empty result and
// never calls the model.
calls := 0
opts := tc.opts
opts.Model = newModel(&calls)
opts.Messages = messages
opts.Clock = quartz.NewMock(t)
result, err := GenerateCompaction(context.Background(), opts)
require.NoError(t, err)
require.Empty(t, result.SummaryReport)
require.Zero(t, calls, "gated run must not call the model")
// With Force the summary is generated and labeled manual.
opts.Force = true
opts.Source = CompactionSourceManual
result, err = GenerateCompaction(context.Background(), opts)
require.NoError(t, err)
require.Equal(t, "forced summary", result.SummaryReport)
require.Equal(t, CompactionSourceManual, result.Source)
require.Equal(t, 1, calls, "forced run calls the model once")
})
}
}
// TestGenerateCompaction_ClampsSummaryCapToRemainingWindow pins the
// summary output cap bound: sum-enforcing providers reject requests
// whose input plus max_tokens exceeds the context window, so the cap
// shrinks to the remaining window minus the trigger step's output and
// the appended summary prompt (ceil(bytes/3) tokens), and degenerate
// cases stay unchanged.
func TestGenerateCompaction_ClampsSummaryCapToRemainingWindow(t *testing.T) {
t.Parallel()
cases := []struct {
name string
contextLimit int64
inputTokens int64
outputTokens int64
toolResultBytes int
summaryContextLimit int64
cap int64
wantCap int64
}{
{name: "clamps to remaining window", contextLimit: 100, inputTokens: 75, outputTokens: 10, cap: 64_000, wantCap: 13},
{name: "reserves trailing tool results", contextLimit: 100, inputTokens: 72, toolResultBytes: 30, cap: 64_000, wantCap: 16},
{name: "clamps to a smaller summary window", contextLimit: 200_000, inputTokens: 150_000, summaryContextLimit: 160_000, cap: 64_000, wantCap: 9998},
{name: "uses a larger summary window", contextLimit: 100, inputTokens: 75, outputTokens: 10, summaryContextLimit: 1_000, cap: 64_000, wantCap: 913},
{name: "keeps cap that fits", contextLimit: 200_000, inputTokens: 140_000, outputTokens: 500, cap: 50_000, wantCap: 50_000},
{name: "usage at limit leaves cap unchanged", contextLimit: 100, inputTokens: 100, cap: 64_000, wantCap: 64_000},
{name: "reserves leave no room, cap unchanged", contextLimit: 100, inputTokens: 80, outputTokens: 25, cap: 64_000, wantCap: 64_000},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
var gotCap *int64
model := &chattest.FakeModel{
ProviderName: "fake",
ModelName: "fake-model",
StreamFn: func(_ context.Context, call fantasy.Call) (fantasy.StreamResponse, error) {
gotCap = call.MaxOutputTokens
return compactionStream(
fantasy.StreamPart{Type: fantasy.StreamPartTypeTextStart, ID: "text"},
fantasy.StreamPart{Type: fantasy.StreamPartTypeTextDelta, ID: "text", Delta: "summary"},
fantasy.StreamPart{Type: fantasy.StreamPartTypeTextEnd, ID: "text"},
fantasy.StreamPart{Type: fantasy.StreamPartTypeFinish, FinishReason: fantasy.FinishReasonStop},
), nil
},
}
capTokens := tc.cap
messages := []fantasy.Message{textMessage(fantasy.MessageRoleUser, "hello")}
if tc.toolResultBytes > 0 {
messages = append(messages, fantasy.Message{
Role: fantasy.MessageRoleTool,
Content: []fantasy.MessagePart{fantasy.ToolResultPart{
ToolCallID: "call-1",
Output: fantasy.ToolResultOutputContentText{Text: strings.Repeat("x", tc.toolResultBytes)},
}},
})
}
result, err := GenerateCompaction(context.Background(), GenerateCompactionOptions{
Model: model,
Messages: messages,
Clock: quartz.NewMock(t),
ThresholdPercent: 70,
ContextLimit: tc.contextLimit,
SummaryContextLimit: tc.summaryContextLimit,
SummaryPrompt: "prompt",
StepUsage: fantasy.Usage{InputTokens: tc.inputTokens, OutputTokens: tc.outputTokens},
SummaryCall: fantasy.Call{MaxOutputTokens: &capTokens},
})
require.NoError(t, err)
require.Equal(t, "summary", result.SummaryReport)
require.NotNil(t, gotCap)
require.Equal(t, tc.wantCap, *gotCap)
})
}
}
// TestGenerateCompaction_DefaultSourceAutomatic verifies an unforced
// over-threshold run reports the automatic source by default.
func TestGenerateCompaction_DefaultSourceAutomatic(t *testing.T) {
t.Parallel()
model := &chattest.FakeModel{
ProviderName: "fake",
ModelName: "fake-model",
StreamFn: func(_ context.Context, _ fantasy.Call) (fantasy.StreamResponse, error) {
return compactionStream(
fantasy.StreamPart{Type: fantasy.StreamPartTypeTextStart, ID: "text"},
fantasy.StreamPart{Type: fantasy.StreamPartTypeTextDelta, ID: "text", Delta: "auto summary"},
fantasy.StreamPart{Type: fantasy.StreamPartTypeTextEnd, ID: "text"},
fantasy.StreamPart{Type: fantasy.StreamPartTypeFinish, FinishReason: fantasy.FinishReasonStop},
), nil
},
}
result, err := GenerateCompaction(context.Background(), GenerateCompactionOptions{
Model: model,
Messages: []fantasy.Message{textMessage(fantasy.MessageRoleUser, "hello")},
ThresholdPercent: 70,
ContextLimit: 100,
StepUsage: fantasy.Usage{InputTokens: 90},
Clock: quartz.NewMock(t),
})
require.NoError(t, err)
require.Equal(t, "auto summary", result.SummaryReport)
require.Equal(t, CompactionSourceAutomatic, result.Source)
}
func TestGenerateCompaction_SummaryEstimate(t *testing.T) {
t.Parallel()
for _, prefix := range []string{"P", "Pr", "Pre"} {
t.Run(prefix, func(t *testing.T) {
t.Parallel()
var parts []codersdk.ChatMessagePart
result, err := GenerateCompaction(t.Context(), GenerateCompactionOptions{
Model: &chattest.FakeModel{
StreamFn: func(context.Context, fantasy.Call) (fantasy.StreamResponse, error) {
return compactionStream(
fantasy.StreamPart{Type: fantasy.StreamPartTypeTextStart, ID: "text"},
fantasy.StreamPart{Type: fantasy.StreamPartTypeTextDelta, ID: "text", Delta: "界x"},
fantasy.StreamPart{Type: fantasy.StreamPartTypeTextEnd, ID: "text"},
fantasy.StreamPart{Type: fantasy.StreamPartTypeFinish, FinishReason: fantasy.FinishReasonStop},
), nil
},
},
Messages: []fantasy.Message{textMessage(fantasy.MessageRoleUser, "hello")},
SystemSummaryPrefix: prefix,
Force: true,
ContextLimit: 1000,
StepUsage: fantasy.Usage{InputTokens: 800},
ToolCallID: "summary",
ToolName: "chat_summarized",
PublishMessagePart: func(_ codersdk.ChatMessageRole, part codersdk.ChatMessagePart) {
parts = append(parts, part)
},
Clock: quartz.NewMock(t),
})
require.NoError(t, err)
require.Equal(t, prefix+"\n\n界x", result.SystemSummary)
require.Equal(t, int64(3), result.EstimatedContextTokens)
require.Equal(t, int64(800), result.ContextTokens)
require.Len(t, parts, 2)
require.Equal(t, codersdk.ChatMessagePartTypeToolResult, parts[1].Type)
require.False(t, parts[1].IsError)
var metadata map[string]any
require.NoError(t, json.Unmarshal(parts[1].Result, &metadata))
require.Equal(t, float64(3), metadata["estimated_context_tokens"])
require.Equal(t, float64(1000), metadata["trigger_context_limit_tokens"])
require.Equal(t, float64(0), metadata["context_limit_tokens"])
})
}
}
// TestGenerateCompaction_RecordsRequestedTrigger verifies the result echoes
// the requested trigger, not the normalized one, while UsagePercent uses the
// resolved limit.
func TestGenerateCompaction_RecordsRequestedTrigger(t *testing.T) {
t.Parallel()
var parts []codersdk.ChatMessagePart
result, err := GenerateCompaction(t.Context(), GenerateCompactionOptions{
Model: &chattest.FakeModel{
StreamFn: func(context.Context, fantasy.Call) (fantasy.StreamResponse, error) {
return compactionStream(
fantasy.StreamPart{Type: fantasy.StreamPartTypeTextStart, ID: "text"},
fantasy.StreamPart{Type: fantasy.StreamPartTypeTextDelta, ID: "text", Delta: "summary"},
fantasy.StreamPart{Type: fantasy.StreamPartTypeTextEnd, ID: "text"},
fantasy.StreamPart{Type: fantasy.StreamPartTypeFinish, FinishReason: fantasy.FinishReasonStop},
), nil
},
},
Messages: []fantasy.Message{textMessage(fantasy.MessageRoleUser, "hello")},
Force: true,
ThresholdPercent: 101,
ContextLimit: 0,
ContextLimitFallback: 1000,
ChatContextLimit: 2000,
StepUsage: fantasy.Usage{InputTokens: 800},
ToolCallID: "summary",
ToolName: "chat_summarized",
PublishMessagePart: func(_ codersdk.ChatMessageRole, part codersdk.ChatMessagePart) {
parts = append(parts, part)
},
Clock: quartz.NewMock(t),
})
require.NoError(t, err)
require.Equal(t, int32(101), result.ThresholdPercent)
require.Zero(t, result.TriggerContextLimit)
require.Equal(t, int64(2000), result.ChatContextLimit)
require.InDelta(t, 80, result.UsagePercent, 0.001)
require.Len(t, parts, 2)
var streamed CompactionToolResult
require.NoError(t, json.Unmarshal(parts[1].Result, &streamed))
require.Equal(t, int32(101), streamed.ThresholdPercent)
require.Zero(t, streamed.TriggerContextLimitTokens)
require.Equal(t, int64(2000), streamed.ContextLimitTokens)
}
// TestGenerateCompaction_RequiresClock verifies a nil clock is
// rejected instead of silently falling back to a real clock; tests
// must supply their own.
func TestGenerateCompaction_RequiresClock(t *testing.T) {
t.Parallel()
_, err := GenerateCompaction(context.Background(), GenerateCompactionOptions{
Model: &chattest.FakeModel{ProviderName: "fake", ModelName: "fake-model"},
})
require.ErrorContains(t, err, "clock is required")
}