**TL;DR:** Wraps each prompt in a `chat_turn` root span with every lifecycle stage parented under it. The span records one outcome per turn (`completed`, `interrupted`, `error`, `abandoned`), and the new `coderd_chatd_turn_outcomes_total` counter counts those outcomes. Turns stay correct when tasks are canceled, replaced, or retried. Each prompt runs inside a `chat_turn` root span owned by the runner (`runnerTurnSpan`). The generation step, prepare, MCP connect, stream, TTFT, provider attempt, commit, thinking, tool call, compaction, and retry backoff stages are parented under it, activating the stages from #29770. `chat_turn` spans are standalone roots: the turn runs on a worker, often on another replica, and no trace context is persisted. **Lifecycle** - **Start.** Each generation loop iteration calls `Ensure`, which opens the turn if none is open and returns a `turnToken`. The span is backdated to the trigger: the latest of the last user prompt, a dynamic tool's result message after it, and a pending compaction request. An `acquisition` stage covers trigger to pickup. - **Stale trigger / takeover.** A trigger at or before the previous anchor opens the turn at now with no `acquisition` and counts `stale_anchor`. A runner that took the chat over from an owner with a stale heartbeat (`TakenOver`, from acquisition) opens its first turn at now with no `acquisition`, so the dead owner's run is not reported as pickup delay. - **Newer prompt.** `Ensure` on the current turn with a newer trigger closes it and opens a new one. The closed turn is `abandoned` unless the task that ran it records an outcome before releasing it. - **Finish.** After `FinishTurn` or `EnterRequiresAction` commits, the step calls `Complete`; `Settle` closes the span after the step's own stage ends, so the finishing step is inside the turn. A chat waiting in `requires_action` is outside any turn; the generation resumed by its tool results opens a new one. - **Emission.** Closing a turn fixes its end time; its span is emitted once every task that joined it (`Ensure` takes the task ID) has called `Release`, which `runTask` does after the task returns. A finishing task whose `FinishTurn` promoted a queued prompt therefore still counts its turn as `completed` even when the promoted task's `Ensure` runs before its `Complete`. `End` at runner exit emits whatever is left. - **Failure.** A step that commits `FinishError`, or fails closed, invalidates the turn as `error` after the commit. Errors the task runner retries, including task timeouts, keep the same turn open; an expected, non-retryable exit records no outcome and leaves the turn open until the interrupt task, a newer prompt, or runner exit closes it. A canceled generation task records no outcome and cannot obtain a turn token, so a replaced task never touches its successor's turn. - **Interrupt.** The interrupt task marks the open turn `interrupted` before `FinishInterruption` commits and closes it after. Only a stop (by the user or a parent agent) is `interrupted`; it ends with unset span status, since error status is reserved for `error` turns. runner shutdown, loss of ownership, and message edits close the turn as `abandoned` with unset status. - **Promotion.** Every promoting transition (`FinishTurn`, `FinishInterruption`, a send to an errored chat, `PromoteQueued`) returns the promoted message's queued time in its result, and one `recordQueueWait` helper records a standalone `queue_wait` to the promotion. The promoted turn opens at the promotion time. `chat_turn` is observed on `coderd_chatd_stage_duration_seconds` only for completed turns, so the histogram is completed-turn latency; every turn still emits its span. `coderd_chatd_turn_outcomes_total{outcome, chat_kind}` counts every closed turn exactly once (`completed`, `interrupted`, `error`, `abandoned`), matching the span's `turn_outcome`, under the `chat-stage-metrics` experiment. Turn time accounting is in draft #29772. **Stages and identity** - `tool_call` is a live stage around each local tool call in `chatloop`; the tool runs on its context, and it ends errored only when the tool fails to execute. Calls that never run (inactive, unknown, or rejected by policy) record no stage. - `mcp_connect` carries connected and failed server counts and ends errored only when servers were configured and none connected. - The compaction stage carries the model that runs the summary. - `withStageIdentity` puts scope and chat kind on the turn context. Detached background work runs in `background` scope as its own trace roots. - The `advisor` tool's nested model call appears only as its enclosing `tool_call`. ARCHITECTURE.md gains a "Lifecycle tracing" section. <details> <summary>Review decisions</summary> - Turn identity stays timestamp-based with targeted fixes (clamped acquisition, rotation on a newer trigger, invalidate after commit, fail-closed invalidation, settle closing invalidated turns) plus a takeover flag; keying turns on message IDs would not detect takeover and cannot express interrupted/abandoned outcomes. - Outcome classification: the interrupt task owns `interrupted`, and cancellation or a fence mismatch alone never records an outcome, so shutdown, ownership loss, and edits are `abandoned`. Only `error` turns set error span status. Remaining ordering races between a step and an interrupt are documented as known limitations in ARCHITECTURE.md. - Turns are emitted after the tasks that ran them return, not when they close, so an outcome a replaced task records while unwinding reaches its own turn; the successor task is not delayed. Chosen over an after-commit, pre-publish hook on `ChatMachine.Update`, which would fix only the `FinishTurn` ordering. - `requires_action` ends the turn rather than pausing it, so `chat_turn` measures chatd time; a prompt with client tools counts as several turns. - Promoted turns anchor at promotion time and `queue_wait` is always standalone, so `chat_turn` never includes time spent queued behind another turn. - The former `capacity_wait` stage (#29805) was dropped: it was a per-worker lower bound, excluded from accounting, and needed its own SQL and bookkeeping. #29805's remaining helpers are folded into this PR. </details> Supersedes #28972, #28973, and #29805. 🤖 This PR was generated by Coder Agents on behalf of @jscottmiller.
1702 lines
48 KiB
Go
1702 lines
48 KiB
Go
package chatloop
|
|
|
|
import (
|
|
"bytes"
|
|
"context"
|
|
"encoding/base64"
|
|
"errors"
|
|
"image"
|
|
"image/png"
|
|
"iter"
|
|
"runtime"
|
|
"slices"
|
|
"sync"
|
|
"sync/atomic"
|
|
"testing"
|
|
"time"
|
|
"unicode/utf8"
|
|
|
|
"charm.land/fantasy"
|
|
fantasyanthropic "charm.land/fantasy/providers/anthropic"
|
|
fantasyopenai "charm.land/fantasy/providers/openai"
|
|
"github.com/prometheus/client_golang/prometheus"
|
|
promtestutil "github.com/prometheus/client_golang/prometheus/testutil"
|
|
"github.com/stretchr/testify/assert"
|
|
"github.com/stretchr/testify/require"
|
|
"golang.org/x/xerrors"
|
|
|
|
"cdr.dev/slog/v3"
|
|
"github.com/coder/coder/v2/coderd/x/chatd/chaterror"
|
|
"github.com/coder/coder/v2/coderd/x/chatd/chatsanitize"
|
|
"github.com/coder/coder/v2/coderd/x/chatd/chattest"
|
|
"github.com/coder/coder/v2/codersdk"
|
|
"github.com/coder/coder/v2/testutil"
|
|
"github.com/coder/quartz"
|
|
)
|
|
|
|
func validWebSearchProviderMetadataForTest() fantasy.ProviderMetadata {
|
|
return fantasy.ProviderMetadata{
|
|
fantasyanthropic.Name: &fantasyanthropic.WebSearchResultMetadata{
|
|
Results: []fantasyanthropic.WebSearchResultItem{
|
|
{
|
|
URL: "https://example.com",
|
|
Title: "Example",
|
|
EncryptedContent: "encrypted",
|
|
},
|
|
},
|
|
},
|
|
}
|
|
}
|
|
|
|
func safeToolCallContent(block fantasy.Content) (fantasy.ToolCallContent, bool) {
|
|
var zero fantasy.ToolCallContent
|
|
switch value := block.(type) {
|
|
case fantasy.ToolCallContent:
|
|
return value, true
|
|
case *fantasy.ToolCallContent:
|
|
if value == nil {
|
|
return zero, false
|
|
}
|
|
return *value, true
|
|
default:
|
|
return zero, false
|
|
}
|
|
}
|
|
|
|
func safeToolResultContent(block fantasy.Content) (fantasy.ToolResultContent, bool) {
|
|
var zero fantasy.ToolResultContent
|
|
switch value := block.(type) {
|
|
case fantasy.ToolResultContent:
|
|
return value, true
|
|
case *fantasy.ToolResultContent:
|
|
if value == nil {
|
|
return zero, false
|
|
}
|
|
return *value, true
|
|
default:
|
|
return zero, false
|
|
}
|
|
}
|
|
|
|
func toolCallContentToPart(toolCall fantasy.ToolCallContent) fantasy.ToolCallPart {
|
|
return fantasy.ToolCallPart{
|
|
ToolCallID: toolCall.ToolCallID,
|
|
ToolName: toolCall.ToolName,
|
|
Input: toolCall.Input,
|
|
ProviderExecuted: toolCall.ProviderExecuted,
|
|
ProviderOptions: fantasy.ProviderOptions(toolCall.ProviderMetadata),
|
|
}
|
|
}
|
|
|
|
func toolResultContentToPart(toolResult fantasy.ToolResultContent) fantasy.ToolResultPart {
|
|
return fantasy.ToolResultPart{
|
|
ToolCallID: toolResult.ToolCallID,
|
|
Output: toolResult.Result,
|
|
ProviderExecuted: toolResult.ProviderExecuted,
|
|
ProviderOptions: fantasy.ProviderOptions(toolResult.ProviderMetadata),
|
|
}
|
|
}
|
|
|
|
func TestStreamSilenceGuard_DisarmAndFireRace(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
for range 128 {
|
|
var cancels atomic.Int32
|
|
guard := newStreamSilenceGuard(quartz.NewReal(), time.Hour, func(err error) {
|
|
if errors.Is(err, errStreamSilenceTimeout) {
|
|
cancels.Add(1)
|
|
}
|
|
})
|
|
|
|
start := make(chan struct{})
|
|
var wg sync.WaitGroup
|
|
wg.Add(2)
|
|
|
|
go func() {
|
|
defer wg.Done()
|
|
<-start
|
|
guard.onTimeout()
|
|
}()
|
|
|
|
go func() {
|
|
defer wg.Done()
|
|
<-start
|
|
guard.Disarm()
|
|
}()
|
|
|
|
close(start)
|
|
wg.Wait()
|
|
|
|
guard.onTimeout()
|
|
guard.Disarm()
|
|
|
|
require.LessOrEqual(t, cancels.Load(), int32(1))
|
|
}
|
|
}
|
|
|
|
func TestStreamSilenceGuard_DisarmPreservesPermanentError(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
attemptCtx, cancelAttempt := context.WithCancelCause(context.Background())
|
|
defer cancelAttempt(nil)
|
|
|
|
guard := newStreamSilenceGuard(quartz.NewReal(), time.Hour, cancelAttempt)
|
|
guard.Disarm()
|
|
guard.onTimeout()
|
|
|
|
classified := chaterror.Classify(classifyStreamSilenceTimeout(
|
|
attemptCtx,
|
|
"openai",
|
|
xerrors.New("invalid model"),
|
|
))
|
|
require.Equal(t, codersdk.ChatErrorKindConfig, classified.Kind)
|
|
require.False(t, classified.Retryable)
|
|
require.Nil(t, context.Cause(attemptCtx))
|
|
}
|
|
|
|
func TestGenerateAssistant_ProviderContextSurvivesStreamError(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
model := &chattest.FakeModel{
|
|
ProviderName: "openai",
|
|
ModelName: "test-model",
|
|
StreamFn: func(_ context.Context, _ fantasy.Call) (fantasy.StreamResponse, error) {
|
|
return nil, xerrors.New("upstream returned status 400")
|
|
},
|
|
}
|
|
|
|
_, err := GenerateAssistant(context.Background(), GenerateAssistantOptions{
|
|
Model: model,
|
|
Messages: []fantasy.Message{
|
|
textMessage(fantasy.MessageRoleUser, "hello"),
|
|
},
|
|
})
|
|
require.Error(t, err)
|
|
classified := chaterror.Classify(err)
|
|
require.Equal(t, "openai", classified.Provider)
|
|
require.Equal(t, "OpenAI returned an unexpected error.", classified.Message)
|
|
}
|
|
|
|
func TestGenerateAssistant_ProviderResponseID(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
cases := []struct {
|
|
name string
|
|
finish fantasy.StreamPart
|
|
want string
|
|
}{
|
|
{
|
|
name: "anthropic message ID",
|
|
finish: fantasy.StreamPart{ID: "msg_anthropic"},
|
|
want: "msg_anthropic",
|
|
},
|
|
{
|
|
name: "openai responses metadata",
|
|
finish: fantasy.StreamPart{ProviderMetadata: fantasy.ProviderMetadata{
|
|
fantasyopenai.Name: &fantasyopenai.ResponsesProviderMetadata{ResponseID: "resp_openai"},
|
|
}},
|
|
want: "resp_openai",
|
|
},
|
|
{
|
|
name: "not reported",
|
|
},
|
|
}
|
|
for _, tc := range cases {
|
|
t.Run(tc.name, func(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
finish := tc.finish
|
|
finish.Type = fantasy.StreamPartTypeFinish
|
|
finish.FinishReason = fantasy.FinishReasonStop
|
|
model := &chattest.FakeModel{
|
|
ProviderName: "fake",
|
|
ModelName: "fake-model",
|
|
StreamFn: func(_ context.Context, _ fantasy.Call) (fantasy.StreamResponse, error) {
|
|
return streamFromParts([]fantasy.StreamPart{
|
|
{Type: fantasy.StreamPartTypeTextStart, ID: "text"},
|
|
{Type: fantasy.StreamPartTypeTextDelta, ID: "text", Delta: "hi"},
|
|
{Type: fantasy.StreamPartTypeTextEnd, ID: "text"},
|
|
finish,
|
|
}), nil
|
|
},
|
|
}
|
|
|
|
outcome, err := GenerateAssistant(context.Background(), GenerateAssistantOptions{
|
|
Model: model,
|
|
Messages: []fantasy.Message{textMessage(fantasy.MessageRoleUser, "hello")},
|
|
})
|
|
require.NoError(t, err)
|
|
require.Equal(t, tc.want, outcome.Step.ProviderResponseID)
|
|
})
|
|
}
|
|
}
|
|
|
|
func TestGenerateAssistant_ErrorProviderOverridesTransportLabel(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
// Bedrock routed through aibridge uses the Anthropic transport, so
|
|
// Model.Provider() reports "anthropic". The user-facing error must use
|
|
// the configured provider from ErrorProvider instead.
|
|
model := &chattest.FakeModel{
|
|
ProviderName: "anthropic",
|
|
ModelName: "qwen3-coder-next",
|
|
StreamFn: func(_ context.Context, _ fantasy.Call) (fantasy.StreamResponse, error) {
|
|
return nil, xerrors.New("upstream returned status 400")
|
|
},
|
|
}
|
|
|
|
_, err := GenerateAssistant(context.Background(), GenerateAssistantOptions{
|
|
Model: model,
|
|
ErrorProvider: "bedrock",
|
|
Messages: []fantasy.Message{
|
|
textMessage(fantasy.MessageRoleUser, "hello"),
|
|
},
|
|
})
|
|
require.Error(t, err)
|
|
classified := chaterror.Classify(err)
|
|
require.Equal(t, "bedrock", classified.Provider)
|
|
require.Equal(t, "AWS Bedrock returned an unexpected error.", classified.Message)
|
|
}
|
|
|
|
func TestGenerateAssistant_HTTP2TransportErrorClassifiedAsRetryableTimeout(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
for _, provider := range []string{"anthropic", "openai"} {
|
|
provider := provider
|
|
t.Run(provider, func(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
model := &chattest.FakeModel{
|
|
ProviderName: provider,
|
|
ModelName: "test-model",
|
|
StreamFn: func(_ context.Context, _ fantasy.Call) (fantasy.StreamResponse, error) {
|
|
return nil, xerrors.New("http2: client connection force closed via ClientConn.Close")
|
|
},
|
|
}
|
|
|
|
_, err := GenerateAssistant(context.Background(), GenerateAssistantOptions{
|
|
Model: model,
|
|
})
|
|
require.Error(t, err)
|
|
classified := chaterror.Classify(err)
|
|
require.Equal(t, codersdk.ChatErrorKindTimeout, classified.Kind)
|
|
require.Equal(t, provider, classified.Provider)
|
|
require.True(t, classified.Retryable)
|
|
})
|
|
}
|
|
}
|
|
|
|
func TestGenerateAssistant_StreamSilenceTimeoutRetryClassification(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
t.Run("timeout while opening stream", func(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
ctx := testutil.Context(t, testutil.WaitLong)
|
|
const silenceTimeout = 5 * time.Millisecond
|
|
clock := quartz.NewMock(t)
|
|
trap := clock.Trap().AfterFunc(streamSilenceGuardTimerTag)
|
|
defer trap.Close()
|
|
var calls atomic.Int32
|
|
model := &chattest.FakeModel{
|
|
ProviderName: "openai",
|
|
ModelName: "test-model",
|
|
StreamFn: func(ctx context.Context, _ fantasy.Call) (fantasy.StreamResponse, error) {
|
|
if calls.Add(1) == 1 {
|
|
<-ctx.Done()
|
|
return nil, ctx.Err()
|
|
}
|
|
return streamFromParts([]fantasy.StreamPart{{
|
|
Type: fantasy.StreamPartTypeFinish,
|
|
FinishReason: fantasy.FinishReasonStop,
|
|
}}), nil
|
|
},
|
|
}
|
|
done := make(chan error, 1)
|
|
go func() {
|
|
_, err := GenerateAssistant(context.Background(), GenerateAssistantOptions{
|
|
Model: model,
|
|
Clock: clock,
|
|
StreamSilenceTimeout: silenceTimeout,
|
|
})
|
|
done <- err
|
|
}()
|
|
|
|
trap.MustWait(ctx).MustRelease(ctx)
|
|
_, waiter := clock.AdvanceNext()
|
|
waiter.MustWait(ctx)
|
|
require.Error(t, <-done)
|
|
require.Equal(t, int32(1), calls.Load())
|
|
})
|
|
|
|
t.Run("timeout before first part", func(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
ctx := testutil.Context(t, testutil.WaitLong)
|
|
const silenceTimeout = 5 * time.Millisecond
|
|
clock := quartz.NewMock(t)
|
|
trap := clock.Trap().AfterFunc(streamSilenceGuardTimerTag)
|
|
defer trap.Close()
|
|
var calls atomic.Int32
|
|
model := &chattest.FakeModel{
|
|
ProviderName: "openai",
|
|
ModelName: "test-model",
|
|
StreamFn: func(ctx context.Context, _ fantasy.Call) (fantasy.StreamResponse, error) {
|
|
calls.Add(1)
|
|
return func(yield func(fantasy.StreamPart) bool) {
|
|
<-ctx.Done()
|
|
yield(fantasy.StreamPart{Type: fantasy.StreamPartTypeError, Error: ctx.Err()})
|
|
}, nil
|
|
},
|
|
}
|
|
done := make(chan error, 1)
|
|
go func() {
|
|
_, err := GenerateAssistant(context.Background(), GenerateAssistantOptions{
|
|
Model: model,
|
|
Clock: clock,
|
|
StreamSilenceTimeout: silenceTimeout,
|
|
})
|
|
done <- err
|
|
}()
|
|
|
|
trap.MustWait(ctx).MustRelease(ctx)
|
|
_, waiter := clock.AdvanceNext()
|
|
waiter.MustWait(ctx)
|
|
err := <-done
|
|
require.Error(t, err)
|
|
classified := chaterror.Classify(err)
|
|
require.Equal(t, codersdk.ChatErrorKindStreamSilenceTimeout, classified.Kind)
|
|
require.Equal(t, "openai", classified.Provider)
|
|
require.True(t, classified.Retryable)
|
|
require.Equal(t, int32(1), calls.Load())
|
|
})
|
|
|
|
t.Run("first part disarms timeout", func(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
ctx := testutil.Context(t, testutil.WaitLong)
|
|
const silenceTimeout = 5 * time.Millisecond
|
|
clock := quartz.NewMock(t)
|
|
trap := clock.Trap().AfterFunc(streamSilenceGuardTimerTag)
|
|
defer trap.Close()
|
|
var calls atomic.Int32
|
|
continueStream := make(chan struct{})
|
|
model := &chattest.FakeModel{
|
|
ProviderName: "openai",
|
|
ModelName: "test-model",
|
|
StreamFn: func(ctx context.Context, _ fantasy.Call) (fantasy.StreamResponse, error) {
|
|
calls.Add(1)
|
|
return func(yield func(fantasy.StreamPart) bool) {
|
|
if !yield(fantasy.StreamPart{Type: fantasy.StreamPartTypeTextStart, ID: "text-1"}) {
|
|
return
|
|
}
|
|
select {
|
|
case <-continueStream:
|
|
case <-ctx.Done():
|
|
yield(fantasy.StreamPart{Type: fantasy.StreamPartTypeError, Error: ctx.Err()})
|
|
return
|
|
}
|
|
yield(fantasy.StreamPart{Type: fantasy.StreamPartTypeTextDelta, ID: "text-1", Delta: "done"})
|
|
yield(fantasy.StreamPart{Type: fantasy.StreamPartTypeTextEnd, ID: "text-1"})
|
|
yield(fantasy.StreamPart{Type: fantasy.StreamPartTypeFinish, FinishReason: fantasy.FinishReasonStop})
|
|
}, nil
|
|
},
|
|
}
|
|
done := make(chan error, 1)
|
|
go func() {
|
|
_, err := GenerateAssistant(context.Background(), GenerateAssistantOptions{
|
|
Model: model,
|
|
Clock: clock,
|
|
StreamSilenceTimeout: silenceTimeout,
|
|
})
|
|
done <- err
|
|
}()
|
|
|
|
trap.MustWait(ctx).MustRelease(ctx)
|
|
close(continueStream)
|
|
require.NoError(t, <-done)
|
|
require.Equal(t, int32(1), calls.Load())
|
|
})
|
|
|
|
t.Run("silent stream close after timeout", func(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
ctx := testutil.Context(t, testutil.WaitLong)
|
|
const silenceTimeout = 5 * time.Millisecond
|
|
clock := quartz.NewMock(t)
|
|
trap := clock.Trap().AfterFunc(streamSilenceGuardTimerTag)
|
|
defer trap.Close()
|
|
var calls atomic.Int32
|
|
model := &chattest.FakeModel{
|
|
ProviderName: "openai",
|
|
ModelName: "test-model",
|
|
StreamFn: func(ctx context.Context, _ fantasy.Call) (fantasy.StreamResponse, error) {
|
|
calls.Add(1)
|
|
return func(func(fantasy.StreamPart) bool) {
|
|
<-ctx.Done()
|
|
}, nil
|
|
},
|
|
}
|
|
done := make(chan error, 1)
|
|
go func() {
|
|
_, err := GenerateAssistant(context.Background(), GenerateAssistantOptions{
|
|
Model: model,
|
|
Clock: clock,
|
|
StreamSilenceTimeout: silenceTimeout,
|
|
})
|
|
done <- err
|
|
}()
|
|
|
|
trap.MustWait(ctx).MustRelease(ctx)
|
|
_, waiter := clock.AdvanceNext()
|
|
waiter.MustWait(ctx)
|
|
err := <-done
|
|
require.Error(t, err)
|
|
classified := chaterror.Classify(err)
|
|
require.Equal(t, codersdk.ChatErrorKindStreamSilenceTimeout, classified.Kind)
|
|
require.Equal(t, int32(1), calls.Load())
|
|
})
|
|
}
|
|
|
|
func TestGenerateAssistant_PanicInPublishMessagePartReleasesAttempt(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
attemptReleased := make(chan struct{})
|
|
model := &chattest.FakeModel{
|
|
ProviderName: "openai",
|
|
ModelName: "test-model",
|
|
StreamFn: func(ctx context.Context, _ fantasy.Call) (fantasy.StreamResponse, error) {
|
|
go func() {
|
|
<-ctx.Done()
|
|
close(attemptReleased)
|
|
}()
|
|
return streamFromParts([]fantasy.StreamPart{
|
|
{Type: fantasy.StreamPartTypeTextStart, ID: "text-1"},
|
|
{Type: fantasy.StreamPartTypeTextDelta, ID: "text-1", Delta: "boom"},
|
|
}), nil
|
|
},
|
|
}
|
|
|
|
defer func() {
|
|
r := recover()
|
|
require.NotNil(t, r)
|
|
select {
|
|
case <-attemptReleased:
|
|
case <-time.After(time.Second):
|
|
t.Fatal("attempt context was not released after panic")
|
|
}
|
|
}()
|
|
|
|
_, _ = GenerateAssistant(context.Background(), GenerateAssistantOptions{
|
|
Model: model,
|
|
PublishMessagePart: func(codersdk.ChatMessageRole, codersdk.ChatMessagePart) {
|
|
panic("publish panic")
|
|
},
|
|
})
|
|
|
|
t.Fatal("expected GenerateAssistant to panic")
|
|
}
|
|
|
|
func requireToolResultErrorMessage(
|
|
t *testing.T,
|
|
result fantasy.ToolResultContent,
|
|
expected string,
|
|
) {
|
|
t.Helper()
|
|
|
|
output, ok := result.Result.(fantasy.ToolResultOutputContentError)
|
|
require.Truef(t, ok, "expected error tool result, got %T", result.Result)
|
|
require.Error(t, output.Error)
|
|
require.Equal(t, expected, output.Error.Error())
|
|
}
|
|
|
|
func streamFromParts(parts []fantasy.StreamPart) fantasy.StreamResponse {
|
|
return iter.Seq[fantasy.StreamPart](func(yield func(fantasy.StreamPart) bool) {
|
|
for _, part := range parts {
|
|
if !yield(part) {
|
|
return
|
|
}
|
|
}
|
|
})
|
|
}
|
|
|
|
func textMessage(role fantasy.MessageRole, text string) fantasy.Message {
|
|
return fantasy.Message{
|
|
Role: role,
|
|
Content: []fantasy.MessagePart{
|
|
fantasy.TextPart{Text: text},
|
|
},
|
|
}
|
|
}
|
|
|
|
func requireTextContent(t *testing.T, content []fantasy.Content, text string) fantasy.TextContent {
|
|
t.Helper()
|
|
|
|
for _, block := range content {
|
|
textContent, ok := fantasy.AsContentType[fantasy.TextContent](block)
|
|
if ok && textContent.Text == text {
|
|
return textContent
|
|
}
|
|
}
|
|
t.Fatalf("missing text content %q", text)
|
|
return fantasy.TextContent{}
|
|
}
|
|
|
|
func TestExclusiveToolPolicy_MixedBatchErrors(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
results, violated := applyExclusiveToolPolicy(
|
|
[]fantasy.ToolCallContent{
|
|
{ToolCallID: "advisor-1", ToolName: "advisor", Input: `{}`},
|
|
{ToolCallID: "read-1", ToolName: "read_file", Input: `{"path":"main.go"}`},
|
|
},
|
|
map[string]bool{"advisor": true},
|
|
NopMetrics(),
|
|
"fake",
|
|
"",
|
|
)
|
|
|
|
require.True(t, violated)
|
|
require.Len(t, results, 2)
|
|
require.Equal(t, "advisor-1", results[0].ToolCallID)
|
|
require.Equal(t, "read-1", results[1].ToolCallID)
|
|
requireToolResultErrorMessage(
|
|
t,
|
|
results[0],
|
|
"advisor must be called alone, without other tools in the same batch. If you need more information to feed into the advisor call, execute the other tools first, then retry with only the advisor call.",
|
|
)
|
|
requireToolResultErrorMessage(
|
|
t,
|
|
results[1],
|
|
"this tool was skipped because advisor must run alone in its batch. Retry your tool calls without advisor, or call advisor separately first.",
|
|
)
|
|
}
|
|
|
|
func TestApplyExclusiveToolPolicy_RecordsErrorMetrics(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
reg := prometheus.NewPedanticRegistry()
|
|
m := NewMetrics(reg)
|
|
|
|
_, violated := applyExclusiveToolPolicy(
|
|
[]fantasy.ToolCallContent{
|
|
{ToolCallID: "advisor-1", ToolName: "advisor", Input: `{}`},
|
|
{ToolCallID: "read-1", ToolName: "read_file", Input: `{"path":"main.go"}`},
|
|
},
|
|
map[string]bool{"advisor": true},
|
|
m,
|
|
"fake",
|
|
"claude-test",
|
|
)
|
|
require.True(t, violated)
|
|
|
|
require.Equal(t, 1.0, promtestutil.ToFloat64(
|
|
m.ToolErrorsTotal.WithLabelValues("fake", "claude-test", "advisor"),
|
|
))
|
|
require.Equal(t, 1.0, promtestutil.ToFloat64(
|
|
m.ToolErrorsTotal.WithLabelValues("fake", "claude-test", "read_file"),
|
|
))
|
|
}
|
|
|
|
func TestExclusiveToolPolicy_MultipleExclusive(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
results, violated := applyExclusiveToolPolicy(
|
|
[]fantasy.ToolCallContent{
|
|
{ToolCallID: "advisor-1", ToolName: "advisor", Input: `{}`},
|
|
{ToolCallID: "advisor-2", ToolName: "advisor", Input: `{"mode":"second-opinion"}`},
|
|
},
|
|
map[string]bool{"advisor": true},
|
|
NopMetrics(),
|
|
"fake",
|
|
"",
|
|
)
|
|
|
|
require.True(t, violated)
|
|
require.Len(t, results, 2)
|
|
requireToolResultErrorMessage(
|
|
t,
|
|
results[0],
|
|
"advisor must be called alone, without other tools in the same batch. If you need more information to feed into the advisor call, execute the other tools first, then retry with only the advisor call.",
|
|
)
|
|
requireToolResultErrorMessage(
|
|
t,
|
|
results[1],
|
|
"advisor must be called alone, without other tools in the same batch. If you need more information to feed into the advisor call, execute the other tools first, then retry with only the advisor call.",
|
|
)
|
|
}
|
|
|
|
func TestSanitizeAnthropicProviderToolContent(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
providerCall := func(id, name, input string) fantasy.ToolCallContent {
|
|
return fantasy.ToolCallContent{
|
|
ToolCallID: id,
|
|
ToolName: name,
|
|
Input: input,
|
|
ProviderExecuted: true,
|
|
}
|
|
}
|
|
providerResult := func(id, name string) fantasy.ToolResultContent {
|
|
return fantasy.ToolResultContent{
|
|
ToolCallID: id,
|
|
ToolName: name,
|
|
ProviderExecuted: true,
|
|
ProviderMetadata: validWebSearchProviderMetadataForTest(),
|
|
Result: fantasy.ToolResultOutputContentText{Text: "ok"},
|
|
}
|
|
}
|
|
localCall := func(id, name string) fantasy.ToolCallContent {
|
|
return fantasy.ToolCallContent{
|
|
ToolCallID: id,
|
|
ToolName: name,
|
|
Input: `{}`,
|
|
}
|
|
}
|
|
localResult := func(id, name string) fantasy.ToolResultContent {
|
|
return fantasy.ToolResultContent{
|
|
ToolCallID: id,
|
|
ToolName: name,
|
|
Result: fantasy.ToolResultOutputContentText{Text: "ok"},
|
|
}
|
|
}
|
|
type contentSummary struct {
|
|
providerCalls []string
|
|
providerResults []string
|
|
localCalls []string
|
|
localResults []string
|
|
}
|
|
summarizeContent := func(content []fantasy.Content) contentSummary {
|
|
var summary contentSummary
|
|
for _, block := range content {
|
|
if toolCall, ok := safeToolCallContent(block); ok {
|
|
if toolCall.ProviderExecuted {
|
|
summary.providerCalls = append(summary.providerCalls, toolCall.ToolCallID)
|
|
} else {
|
|
summary.localCalls = append(summary.localCalls, toolCall.ToolCallID)
|
|
}
|
|
continue
|
|
}
|
|
if toolResult, ok := safeToolResultContent(block); ok {
|
|
if toolResult.ProviderExecuted {
|
|
summary.providerResults = append(summary.providerResults, toolResult.ToolCallID)
|
|
} else {
|
|
summary.localResults = append(summary.localResults, toolResult.ToolCallID)
|
|
}
|
|
}
|
|
}
|
|
return summary
|
|
}
|
|
assertProviderHistoryValid := func(t *testing.T, content []fantasy.Content) {
|
|
t.Helper()
|
|
|
|
parts := make([]fantasy.MessagePart, 0)
|
|
for _, block := range content {
|
|
if toolCall, ok := safeToolCallContent(block); ok && toolCall.ProviderExecuted {
|
|
parts = append(parts, toolCallContentToPart(toolCall))
|
|
continue
|
|
}
|
|
if toolResult, ok := safeToolResultContent(block); ok && toolResult.ProviderExecuted {
|
|
parts = append(parts, toolResultContentToPart(toolResult))
|
|
}
|
|
}
|
|
if len(parts) == 0 {
|
|
return
|
|
}
|
|
require.Empty(t, chatsanitize.ValidateAnthropicProviderToolHistory([]fantasy.Message{
|
|
{
|
|
Role: fantasy.MessageRoleAssistant,
|
|
Content: parts,
|
|
},
|
|
}))
|
|
}
|
|
|
|
metadataCall := providerCall("ws-meta", "web_search", `{"query":"coder"}`)
|
|
metadataCall.ProviderMetadata = fantasy.ProviderMetadata{fantasyanthropic.Name: nil}
|
|
metadataResult := providerResult("ws-meta", "web_search")
|
|
metadataResult.ProviderMetadata = fantasy.ProviderMetadata{fantasyanthropic.Name: nil}
|
|
pointerCall := providerCall("ws-pointer", "web_search", `{"query":"coder"}`)
|
|
var nilToolCall *fantasy.ToolCallContent
|
|
|
|
testCases := []struct {
|
|
name string
|
|
provider string
|
|
content []fantasy.Content
|
|
wantSummary contentSummary
|
|
wantRemovedCalls int
|
|
wantRemovedResults int
|
|
wantTexts []string
|
|
validateAnthropic bool
|
|
}{
|
|
{
|
|
name: "orphan provider result textified",
|
|
provider: fantasyanthropic.Name,
|
|
content: []fantasy.Content{
|
|
fantasy.TextContent{Text: "keep"},
|
|
providerResult("ws-1", "web_search"),
|
|
},
|
|
wantRemovedResults: 1,
|
|
wantTexts: []string{"keep", "ok"},
|
|
validateAnthropic: true,
|
|
},
|
|
{
|
|
name: "result before call removes both provider blocks",
|
|
provider: fantasyanthropic.Name,
|
|
content: []fantasy.Content{
|
|
providerResult("ws-1", "web_search"),
|
|
providerCall("ws-1", "web_search", `{"query":"coder"}`),
|
|
},
|
|
wantRemovedCalls: 1,
|
|
wantRemovedResults: 1,
|
|
wantTexts: []string{"ok"},
|
|
validateAnthropic: true,
|
|
},
|
|
{
|
|
name: "valid web search pair preserved",
|
|
provider: fantasyanthropic.Name,
|
|
content: []fantasy.Content{
|
|
providerCall("ws-1", "web_search", `{"query":"coder"}`),
|
|
providerResult("ws-1", "web_search"),
|
|
fantasy.TextContent{Text: "search done"},
|
|
},
|
|
wantSummary: contentSummary{
|
|
providerCalls: []string{"ws-1"},
|
|
providerResults: []string{"ws-1"},
|
|
},
|
|
wantTexts: []string{"search done"},
|
|
validateAnthropic: true,
|
|
},
|
|
{
|
|
name: "invalid JSON provider call drops pair",
|
|
provider: fantasyanthropic.Name,
|
|
content: []fantasy.Content{
|
|
providerCall("ws-1", "web_search", `{`),
|
|
providerResult("ws-1", "web_search"),
|
|
},
|
|
wantRemovedCalls: 1,
|
|
wantRemovedResults: 1,
|
|
wantTexts: []string{"ok"},
|
|
validateAnthropic: true,
|
|
},
|
|
{
|
|
name: "empty ID provider call drops pair",
|
|
provider: fantasyanthropic.Name,
|
|
content: []fantasy.Content{
|
|
providerCall("", "web_search", `{"query":"coder"}`),
|
|
providerResult("", "web_search"),
|
|
},
|
|
wantRemovedCalls: 1,
|
|
wantRemovedResults: 1,
|
|
wantTexts: []string{"ok"},
|
|
validateAnthropic: true,
|
|
},
|
|
{
|
|
name: "empty tool name provider call drops pair",
|
|
provider: fantasyanthropic.Name,
|
|
content: []fantasy.Content{
|
|
providerCall("ws-empty", "", `{"query":"coder"}`),
|
|
providerResult("ws-empty", ""),
|
|
},
|
|
wantRemovedCalls: 1,
|
|
wantRemovedResults: 1,
|
|
wantTexts: []string{"ok"},
|
|
validateAnthropic: true,
|
|
},
|
|
{
|
|
name: "non web search provider pair drops through serializable helper",
|
|
provider: fantasyanthropic.Name,
|
|
content: []fantasy.Content{
|
|
providerCall("code-1", "code_execution", `{"code":"print(1)"}`),
|
|
providerResult("code-1", "code_execution"),
|
|
},
|
|
wantRemovedCalls: 1,
|
|
wantRemovedResults: 1,
|
|
wantTexts: []string{"ok"},
|
|
validateAnthropic: true,
|
|
},
|
|
{
|
|
name: "mismatched provider result tool name drops pair",
|
|
provider: fantasyanthropic.Name,
|
|
content: []fantasy.Content{
|
|
providerCall("ws-mismatch", "web_search", `{"query":"coder"}`),
|
|
providerResult("ws-mismatch", "code_execution"),
|
|
},
|
|
wantRemovedCalls: 1,
|
|
wantRemovedResults: 1,
|
|
wantTexts: []string{"ok"},
|
|
validateAnthropic: true,
|
|
},
|
|
{
|
|
name: "duplicate provider IDs drop all provider content for ID",
|
|
provider: fantasyanthropic.Name,
|
|
content: []fantasy.Content{
|
|
providerCall("dup-1", "web_search", `{"query":"coder"}`),
|
|
providerResult("dup-1", "web_search"),
|
|
providerCall("dup-1", "web_search", `{"query":"coder"}`),
|
|
},
|
|
wantRemovedCalls: 2,
|
|
wantRemovedResults: 1,
|
|
wantTexts: []string{"ok"},
|
|
validateAnthropic: true,
|
|
},
|
|
{
|
|
name: "mismatched provider flags remove only provider side",
|
|
provider: fantasyanthropic.Name,
|
|
content: []fantasy.Content{
|
|
providerCall("mix-1", "web_search", `{"query":"coder"}`),
|
|
localResult("mix-1", "web_search"),
|
|
localCall("mix-2", "read_file"),
|
|
providerResult("mix-2", "web_search"),
|
|
},
|
|
wantSummary: contentSummary{
|
|
localCalls: []string{"mix-2"},
|
|
localResults: []string{"mix-1"},
|
|
},
|
|
wantRemovedCalls: 1,
|
|
wantRemovedResults: 1,
|
|
wantTexts: []string{"ok"},
|
|
validateAnthropic: true,
|
|
},
|
|
{
|
|
name: "malformed provider metadata textifies result",
|
|
provider: fantasyanthropic.Name,
|
|
content: []fantasy.Content{
|
|
metadataCall,
|
|
metadataResult,
|
|
},
|
|
wantRemovedCalls: 1,
|
|
wantRemovedResults: 1,
|
|
wantTexts: []string{"ok"},
|
|
validateAnthropic: true,
|
|
},
|
|
{
|
|
name: "pointer and nil pointer variants are handled safely",
|
|
provider: fantasyanthropic.Name,
|
|
content: []fantasy.Content{
|
|
nilToolCall,
|
|
&pointerCall,
|
|
providerResult("ws-pointer", "web_search"),
|
|
},
|
|
wantSummary: contentSummary{
|
|
providerCalls: []string{"ws-pointer"},
|
|
providerResults: []string{"ws-pointer"},
|
|
},
|
|
validateAnthropic: true,
|
|
},
|
|
{
|
|
name: "local tool content is unchanged",
|
|
provider: fantasyanthropic.Name,
|
|
content: []fantasy.Content{
|
|
localCall("tc-1", "read_file"),
|
|
localResult("tc-1", "read_file"),
|
|
},
|
|
wantSummary: contentSummary{
|
|
localCalls: []string{"tc-1"},
|
|
localResults: []string{"tc-1"},
|
|
},
|
|
validateAnthropic: true,
|
|
},
|
|
{
|
|
name: "non Anthropic provider content is unchanged",
|
|
provider: "fake",
|
|
content: []fantasy.Content{
|
|
providerCall("ws-1", "web_search", `{"query":"coder"}`),
|
|
},
|
|
wantSummary: contentSummary{
|
|
providerCalls: []string{"ws-1"},
|
|
},
|
|
},
|
|
}
|
|
|
|
for _, tc := range testCases {
|
|
t.Run(tc.name, func(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
sanitized, stats := chatsanitize.SanitizeAnthropicProviderToolContent(tc.provider, tc.content)
|
|
require.Equal(t, tc.wantRemovedCalls, stats.RemovedToolCalls)
|
|
require.Equal(t, tc.wantRemovedResults, stats.RemovedToolResults)
|
|
require.Zero(t, stats.DroppedMessages)
|
|
|
|
summary := summarizeContent(sanitized)
|
|
assert.ElementsMatch(t, tc.wantSummary.providerCalls, summary.providerCalls)
|
|
assert.ElementsMatch(t, tc.wantSummary.providerResults, summary.providerResults)
|
|
assert.ElementsMatch(t, tc.wantSummary.localCalls, summary.localCalls)
|
|
assert.ElementsMatch(t, tc.wantSummary.localResults, summary.localResults)
|
|
for _, text := range tc.wantTexts {
|
|
requireTextContent(t, sanitized, text)
|
|
}
|
|
if tc.validateAnthropic {
|
|
assertProviderHistoryValid(t, sanitized)
|
|
}
|
|
})
|
|
}
|
|
}
|
|
|
|
type serialMarkerTool struct{ fantasy.AgentTool }
|
|
|
|
func (serialMarkerTool) SerialToolCalls() bool { return true }
|
|
|
|
type observerMarkerTool struct {
|
|
fantasy.AgentTool
|
|
observed func(names []string)
|
|
}
|
|
|
|
func (t observerMarkerTool) ObserveStepToolCalls(names []string) { t.observed(names) }
|
|
|
|
func TestExecuteToolsNotifiesStepToolCallObservers(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
var mu sync.Mutex
|
|
var observedNames []string
|
|
observedBeforeRun := false
|
|
observer := observerMarkerTool{
|
|
AgentTool: fantasy.NewAgentTool(
|
|
"observer_tool",
|
|
"records sibling calls",
|
|
func(_ context.Context, _ struct{}, _ fantasy.ToolCall) (fantasy.ToolResponse, error) {
|
|
mu.Lock()
|
|
observedBeforeRun = observedNames != nil
|
|
mu.Unlock()
|
|
return fantasy.NewTextResponse("ok"), nil
|
|
},
|
|
),
|
|
observed: func(names []string) {
|
|
mu.Lock()
|
|
defer mu.Unlock()
|
|
observedNames = append([]string{}, names...)
|
|
},
|
|
}
|
|
var uncalledObserved atomic.Bool
|
|
uncalledObserver := observerMarkerTool{
|
|
AgentTool: fantasy.NewAgentTool(
|
|
"uncalled_observer",
|
|
"never called this step",
|
|
func(_ context.Context, _ struct{}, _ fantasy.ToolCall) (fantasy.ToolResponse, error) {
|
|
return fantasy.NewTextResponse("ok"), nil
|
|
},
|
|
),
|
|
observed: func([]string) { uncalledObserved.Store(true) },
|
|
}
|
|
other := fantasy.NewAgentTool(
|
|
"other_tool",
|
|
"plain tool",
|
|
func(_ context.Context, _ struct{}, _ fantasy.ToolCall) (fantasy.ToolResponse, error) {
|
|
return fantasy.NewTextResponse("ok"), nil
|
|
},
|
|
)
|
|
|
|
executeTools(
|
|
context.Background(),
|
|
quartz.NewReal(),
|
|
[]fantasy.AgentTool{observer, uncalledObserver, other},
|
|
nil,
|
|
nil,
|
|
nil,
|
|
[]fantasy.ToolCallContent{
|
|
{ToolCallID: "1", ToolName: "observer_alias", Input: "{}"},
|
|
{ToolCallID: "2", ToolName: "other_tool", Input: "{}"},
|
|
},
|
|
[]fantasy.ToolCallContent{
|
|
{ToolCallID: "1", ToolName: "observer_alias", Input: "{}"},
|
|
{ToolCallID: "2", ToolName: "other_tool", Input: "{}"},
|
|
{ToolCallID: "3", ToolName: "denied_tool", Input: "{}"},
|
|
},
|
|
NewMetrics(prometheus.NewRegistry()),
|
|
slog.Make(),
|
|
"fake", "fake-model",
|
|
map[string]bool{},
|
|
defaultToolResultBytes,
|
|
map[string]string{"observer_alias": "observer_tool"},
|
|
time.Time{},
|
|
nil,
|
|
nil,
|
|
nil,
|
|
StageModel{},
|
|
)
|
|
|
|
require.Equal(t, []string{"observer_tool", "other_tool", "denied_tool"}, observedNames,
|
|
"a called observer sees every observed tool-call name, including calls denied before execution")
|
|
require.True(t, observedBeforeRun, "observers are notified before any tool call executes")
|
|
require.False(t, uncalledObserved.Load(), "tools not called this step are not notified")
|
|
}
|
|
|
|
type resultObserverMarkerTool struct {
|
|
fantasy.AgentTool
|
|
observedResults func(names []string, errored []bool)
|
|
}
|
|
|
|
func (t resultObserverMarkerTool) ObserveStepToolResults(names []string, errored []bool) {
|
|
t.observedResults(names, errored)
|
|
}
|
|
|
|
func TestExecuteToolsNotifiesStepToolResultObservers(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
var mu sync.Mutex
|
|
var gotNames []string
|
|
var gotErrored []bool
|
|
notifications := 0
|
|
observer := resultObserverMarkerTool{
|
|
AgentTool: fantasy.NewAgentTool(
|
|
"observer_tool",
|
|
"records sibling outcomes",
|
|
func(_ context.Context, _ struct{}, _ fantasy.ToolCall) (fantasy.ToolResponse, error) {
|
|
return fantasy.NewTextResponse("ok"), nil
|
|
},
|
|
),
|
|
observedResults: func(names []string, errored []bool) {
|
|
mu.Lock()
|
|
defer mu.Unlock()
|
|
notifications++
|
|
gotNames = append([]string{}, names...)
|
|
gotErrored = append([]bool{}, errored...)
|
|
},
|
|
}
|
|
failing := fantasy.NewAgentTool(
|
|
"failing_tool",
|
|
"returns an error result",
|
|
func(_ context.Context, _ struct{}, _ fantasy.ToolCall) (fantasy.ToolResponse, error) {
|
|
return fantasy.NewTextErrorResponse("remote error"), nil
|
|
},
|
|
)
|
|
|
|
executed := []fantasy.ToolCallContent{
|
|
{ToolCallID: "1", ToolName: "observer_alias", Input: "{}"},
|
|
{ToolCallID: "2", ToolName: "failing_tool", Input: "{}"},
|
|
{ToolCallID: "3", ToolName: "missing_tool", Input: "{}"},
|
|
}
|
|
executeTools(
|
|
context.Background(),
|
|
quartz.NewReal(),
|
|
[]fantasy.AgentTool{observer, failing},
|
|
nil,
|
|
nil,
|
|
nil,
|
|
executed,
|
|
append(slices.Clone(executed), fantasy.ToolCallContent{
|
|
ToolCallID: "4", ToolName: "rejected_tool", Input: "{not json",
|
|
}),
|
|
NewMetrics(prometheus.NewRegistry()),
|
|
slog.Make(),
|
|
"fake", "fake-model",
|
|
map[string]bool{},
|
|
defaultToolResultBytes,
|
|
map[string]string{"observer_alias": "observer_tool"},
|
|
time.Time{},
|
|
nil,
|
|
nil,
|
|
nil,
|
|
StageModel{},
|
|
)
|
|
|
|
require.Equal(t, 1, notifications, "each called observer is notified once per step")
|
|
require.Equal(t, []string{"observer_tool", "failing_tool", "missing_tool", "rejected_tool"}, gotNames,
|
|
"outcomes are reported per call in observed order with aliases resolved")
|
|
require.Equal(t, []bool{false, true, true, true}, gotErrored,
|
|
"error results, unresolvable tools, and observed calls rejected before execution all settle as errored outcomes")
|
|
}
|
|
|
|
type serialResultObserverTool struct {
|
|
fantasy.AgentTool
|
|
observedResults func(names []string, errored []bool)
|
|
}
|
|
|
|
func (serialResultObserverTool) SerialToolCalls() bool { return true }
|
|
|
|
func (t serialResultObserverTool) ObserveStepToolResults(names []string, errored []bool) {
|
|
t.observedResults(names, errored)
|
|
}
|
|
|
|
func TestExecuteToolsReconcilesResultsBeforeSerialCalls(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
var mu sync.Mutex
|
|
var erroredAtNotify []string
|
|
var erroredAtRun []string
|
|
notified := false
|
|
serial := serialResultObserverTool{
|
|
AgentTool: fantasy.NewAgentTool(
|
|
"serial_observer",
|
|
"observes sibling outcomes before running",
|
|
func(_ context.Context, _ struct{}, _ fantasy.ToolCall) (fantasy.ToolResponse, error) {
|
|
mu.Lock()
|
|
erroredAtRun = append([]string{}, erroredAtNotify...)
|
|
mu.Unlock()
|
|
return fantasy.NewTextResponse("ok"), nil
|
|
},
|
|
),
|
|
observedResults: func(names []string, errored []bool) {
|
|
mu.Lock()
|
|
defer mu.Unlock()
|
|
notified = true
|
|
erroredAtNotify = nil
|
|
for i, name := range names {
|
|
if errored[i] {
|
|
erroredAtNotify = append(erroredAtNotify, name)
|
|
}
|
|
}
|
|
},
|
|
}
|
|
failing := fantasy.NewAgentTool(
|
|
"failing_tool",
|
|
"returns an error result",
|
|
func(_ context.Context, _ struct{}, _ fantasy.ToolCall) (fantasy.ToolResponse, error) {
|
|
return fantasy.NewTextErrorResponse("remote error"), nil
|
|
},
|
|
)
|
|
|
|
results := executeTools(
|
|
context.Background(),
|
|
quartz.NewReal(),
|
|
[]fantasy.AgentTool{serial, failing},
|
|
nil,
|
|
nil,
|
|
nil,
|
|
[]fantasy.ToolCallContent{
|
|
{ToolCallID: "1", ToolName: "serial_observer", Input: "{}"},
|
|
{ToolCallID: "2", ToolName: "failing_tool", Input: "{}"},
|
|
},
|
|
nil,
|
|
NewMetrics(prometheus.NewRegistry()),
|
|
slog.Make(),
|
|
"fake", "fake-model",
|
|
map[string]bool{},
|
|
defaultToolResultBytes,
|
|
nil,
|
|
time.Time{},
|
|
nil,
|
|
nil,
|
|
nil,
|
|
StageModel{},
|
|
)
|
|
|
|
require.True(t, notified)
|
|
require.Equal(t, []string{"failing_tool"}, erroredAtRun,
|
|
"a serial tool must see settled sibling outcomes before it executes")
|
|
require.Len(t, results, 2)
|
|
require.Equal(t, "1", results[0].content.ToolCallID, "results keep original call order")
|
|
}
|
|
|
|
func TestExecuteToolsSerialToolCallOrder(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
var mu sync.Mutex
|
|
var events []string
|
|
inFlight := 0
|
|
maxInFlight := 0
|
|
record := func(event string, delta int) {
|
|
mu.Lock()
|
|
defer mu.Unlock()
|
|
inFlight += delta
|
|
if inFlight > maxInFlight {
|
|
maxInFlight = inFlight
|
|
}
|
|
events = append(events, event)
|
|
}
|
|
serial := serialMarkerTool{AgentTool: fantasy.NewAgentTool(
|
|
"serial_tool",
|
|
"records call order",
|
|
func(_ context.Context, _ struct{}, call fantasy.ToolCall) (fantasy.ToolResponse, error) {
|
|
record(call.ID+":start", 1)
|
|
runtime.Gosched()
|
|
record(call.ID+":end", -1)
|
|
return fantasy.NewTextResponse("ok"), nil
|
|
},
|
|
)}
|
|
parallelRan := make(chan struct{})
|
|
parallel := fantasy.NewAgentTool(
|
|
"parallel_tool",
|
|
"plain tool",
|
|
func(_ context.Context, _ struct{}, _ fantasy.ToolCall) (fantasy.ToolResponse, error) {
|
|
close(parallelRan)
|
|
return fantasy.NewTextResponse("ok"), nil
|
|
},
|
|
)
|
|
|
|
calls := []fantasy.ToolCallContent{
|
|
{ToolCallID: "a", ToolName: "serial_tool", Input: "{}"},
|
|
{ToolCallID: "p", ToolName: "parallel_tool", Input: "{}"},
|
|
{ToolCallID: "b", ToolName: "serial_tool", Input: "{}"},
|
|
{ToolCallID: "c", ToolName: "serial_tool", Input: "{}"},
|
|
}
|
|
results := executeTools(
|
|
context.Background(),
|
|
quartz.NewReal(),
|
|
[]fantasy.AgentTool{serial, parallel},
|
|
nil,
|
|
nil,
|
|
nil,
|
|
calls,
|
|
nil,
|
|
NewMetrics(prometheus.NewRegistry()),
|
|
slog.Make(),
|
|
"fake", "fake-model",
|
|
map[string]bool{},
|
|
defaultToolResultBytes,
|
|
nil,
|
|
time.Time{},
|
|
nil,
|
|
nil,
|
|
nil,
|
|
StageModel{},
|
|
)
|
|
|
|
require.Equal(t, []string{"a:start", "a:end", "b:start", "b:end", "c:start", "c:end"}, events,
|
|
"serial tool calls must run one at a time in tool-call order")
|
|
require.Equal(t, 1, maxInFlight)
|
|
select {
|
|
case <-parallelRan:
|
|
default:
|
|
t.Fatal("parallel tool call did not run")
|
|
}
|
|
require.Len(t, results, len(calls))
|
|
for i, tc := range calls {
|
|
require.Equal(t, tc.ToolCallID, results[i].content.ToolCallID, "results keep original call order")
|
|
}
|
|
}
|
|
|
|
type timingEvent struct {
|
|
dispatchIndex int
|
|
at time.Time
|
|
}
|
|
|
|
// liveToolBillingRecorder is a test ToolBillingRecorder that publishes
|
|
// start and completion timestamps as they happen.
|
|
type liveToolBillingRecorder struct {
|
|
started chan timingEvent
|
|
completed chan timingEvent
|
|
}
|
|
|
|
func (r liveToolBillingRecorder) RecordStart(dispatchIndex int, startedAt time.Time) {
|
|
r.started <- timingEvent{dispatchIndex: dispatchIndex, at: startedAt}
|
|
}
|
|
|
|
func (r liveToolBillingRecorder) RecordComplete(dispatchIndex int, completedAt time.Time) {
|
|
r.completed <- timingEvent{dispatchIndex: dispatchIndex, at: completedAt}
|
|
}
|
|
|
|
func TestExecuteToolsReturnsExecutionIntervals(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
ctx := testutil.Context(t, testutil.WaitShort)
|
|
started := make(chan timingEvent, 3)
|
|
completed := make(chan timingEvent, 3)
|
|
|
|
slowGo := make(chan struct{})
|
|
fastGo := make(chan struct{})
|
|
blocking := func(name string, release <-chan struct{}) fantasy.AgentTool {
|
|
return fantasy.NewAgentTool(
|
|
name,
|
|
"waits for the test to release it",
|
|
func(_ context.Context, _ struct{}, _ fantasy.ToolCall) (fantasy.ToolResponse, error) {
|
|
<-release
|
|
return fantasy.NewTextResponse("ok"), nil
|
|
},
|
|
)
|
|
}
|
|
slow := blocking("slow_tool", slowGo)
|
|
fast := blocking("fast_tool", fastGo)
|
|
serial := serialMarkerTool{AgentTool: fantasy.NewAgentTool(
|
|
"serial_tool",
|
|
"runs after concurrent tools",
|
|
func(_ context.Context, _ struct{}, _ fantasy.ToolCall) (fantasy.ToolResponse, error) {
|
|
return fantasy.NewTextResponse("ok"), nil
|
|
},
|
|
)}
|
|
calls := []fantasy.ToolCallContent{
|
|
{ToolCallID: "slow", ToolName: "slow_tool", Input: "{}"},
|
|
{ToolCallID: "fast", ToolName: "fast_tool", Input: "{}"},
|
|
{ToolCallID: "serial", ToolName: "serial_tool", Input: "{}"},
|
|
}
|
|
batchStart := time.Date(2024, time.January, 1, 0, 0, 0, 0, time.UTC)
|
|
resultsCh := make(chan []toolExecutionResult, 1)
|
|
go func() {
|
|
resultsCh <- executeTools(
|
|
ctx,
|
|
quartz.NewReal(),
|
|
[]fantasy.AgentTool{slow, fast, serial},
|
|
nil,
|
|
nil,
|
|
nil,
|
|
calls,
|
|
nil,
|
|
NewMetrics(prometheus.NewRegistry()),
|
|
slog.Make(),
|
|
"fake", "fake-model",
|
|
map[string]bool{},
|
|
defaultToolResultBytes,
|
|
nil,
|
|
batchStart,
|
|
liveToolBillingRecorder{started: started, completed: completed},
|
|
nil,
|
|
nil,
|
|
StageModel{},
|
|
)
|
|
}()
|
|
|
|
startByIndex := make([]time.Time, len(calls))
|
|
for range 2 {
|
|
event := testutil.RequireReceive(ctx, t, started)
|
|
startByIndex[event.dispatchIndex] = event.at
|
|
}
|
|
|
|
close(fastGo)
|
|
fastCompleted := testutil.RequireReceive(ctx, t, completed)
|
|
require.Equal(t, 1, fastCompleted.dispatchIndex)
|
|
|
|
close(slowGo)
|
|
slowCompleted := testutil.RequireReceive(ctx, t, completed)
|
|
require.Equal(t, 0, slowCompleted.dispatchIndex)
|
|
|
|
serialStarted := testutil.RequireReceive(ctx, t, started)
|
|
require.Equal(t, 2, serialStarted.dispatchIndex)
|
|
startByIndex[serialStarted.dispatchIndex] = serialStarted.at
|
|
serialCompleted := testutil.RequireReceive(ctx, t, completed)
|
|
require.Equal(t, 2, serialCompleted.dispatchIndex)
|
|
|
|
endByIndex := []time.Time{slowCompleted.at, fastCompleted.at, serialCompleted.at}
|
|
results := testutil.RequireReceive(ctx, t, resultsCh)
|
|
require.Len(t, results, len(calls))
|
|
for i, call := range calls {
|
|
require.Equal(t, call.ToolCallID, results[i].content.ToolCallID,
|
|
"results keep dispatch order when calls complete out of order")
|
|
require.Equal(t, startByIndex[i], results[i].interval.Start)
|
|
require.Equal(t, endByIndex[i], results[i].interval.End)
|
|
}
|
|
require.Equal(t, batchStart, results[0].interval.Start)
|
|
require.Equal(t, batchStart, results[1].interval.Start)
|
|
require.NotEqual(t, batchStart, results[2].interval.Start)
|
|
require.False(t, results[2].interval.Start.Before(slowCompleted.at),
|
|
"serial execution starts only after concurrent calls settle")
|
|
}
|
|
|
|
func TestExecuteSingleTool_MediaBase64Encoding(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
originalBytes := []byte{0xFF, 0xD8, 0xFF, 0xE0, 0x00, 0x10}
|
|
metrics := NewMetrics(prometheus.NewRegistry())
|
|
logger := slog.Make()
|
|
|
|
t.Run("EncodesRawBytesToBase64", func(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
tool := fantasy.NewAgentTool(
|
|
"screenshot",
|
|
"takes a screenshot",
|
|
func(_ context.Context, _ struct{}, _ fantasy.ToolCall) (fantasy.ToolResponse, error) {
|
|
return fantasy.ToolResponse{
|
|
Type: "image",
|
|
Data: originalBytes,
|
|
MediaType: "image/jpeg",
|
|
}, nil
|
|
},
|
|
)
|
|
|
|
toolMap := map[string]fantasy.AgentTool{
|
|
"screenshot": tool,
|
|
}
|
|
tc := fantasy.ToolCallContent{
|
|
ToolCallID: "call-1",
|
|
ToolName: "screenshot",
|
|
Input: "{}",
|
|
}
|
|
|
|
result, _ := executeSingleTool(
|
|
context.Background(),
|
|
toolMap,
|
|
tc,
|
|
metrics,
|
|
logger,
|
|
"fake", "fake-model",
|
|
map[string]bool{},
|
|
[]string{"screenshot"},
|
|
nil,
|
|
map[string]struct{}{},
|
|
nil,
|
|
defaultToolResultBytes,
|
|
nil,
|
|
nil,
|
|
)
|
|
|
|
media, ok := result.Result.(fantasy.ToolResultOutputContentMedia)
|
|
require.True(t, ok, "expected ToolResultOutputContentMedia")
|
|
require.Equal(t, "image/jpeg", media.MediaType)
|
|
|
|
decoded, err := base64.StdEncoding.DecodeString(media.Data)
|
|
require.NoError(t, err, "Data should be valid base64")
|
|
require.Equal(t, originalBytes, decoded)
|
|
})
|
|
|
|
t.Run("SanitizesInvalidUTF8InContent", func(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
tool := fantasy.NewAgentTool(
|
|
"screenshot",
|
|
"takes a screenshot",
|
|
func(_ context.Context, _ struct{}, _ fantasy.ToolCall) (fantasy.ToolResponse, error) {
|
|
return fantasy.ToolResponse{
|
|
Type: "image",
|
|
Data: originalBytes,
|
|
MediaType: "image/png",
|
|
Content: "hello\xffworld",
|
|
}, nil
|
|
},
|
|
)
|
|
|
|
toolMap := map[string]fantasy.AgentTool{
|
|
"screenshot": tool,
|
|
}
|
|
tc := fantasy.ToolCallContent{
|
|
ToolCallID: "call-2",
|
|
ToolName: "screenshot",
|
|
Input: "{}",
|
|
}
|
|
|
|
result, _ := executeSingleTool(
|
|
context.Background(),
|
|
toolMap,
|
|
tc,
|
|
metrics,
|
|
logger,
|
|
"fake", "fake-model",
|
|
map[string]bool{},
|
|
[]string{"screenshot"},
|
|
nil,
|
|
map[string]struct{}{},
|
|
nil,
|
|
defaultToolResultBytes,
|
|
nil,
|
|
nil,
|
|
)
|
|
|
|
media, ok := result.Result.(fantasy.ToolResultOutputContentMedia)
|
|
require.True(t, ok, "expected ToolResultOutputContentMedia")
|
|
require.True(t, utf8.ValidString(media.Text), "Text should be valid UTF-8")
|
|
require.Contains(t, media.Text, "hello")
|
|
require.Contains(t, media.Text, "world")
|
|
})
|
|
|
|
t.Run("SanitizesInvalidUTF8InTextResult", func(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
tool := fantasy.NewAgentTool(
|
|
"echo",
|
|
"echoes input",
|
|
func(_ context.Context, _ struct{}, _ fantasy.ToolCall) (fantasy.ToolResponse, error) {
|
|
return fantasy.ToolResponse{
|
|
Content: "hello\xffworld",
|
|
}, nil
|
|
},
|
|
)
|
|
|
|
toolMap := map[string]fantasy.AgentTool{
|
|
"echo": tool,
|
|
}
|
|
tc := fantasy.ToolCallContent{
|
|
ToolCallID: "call-3",
|
|
ToolName: "echo",
|
|
Input: "{}",
|
|
}
|
|
|
|
result, _ := executeSingleTool(
|
|
context.Background(),
|
|
toolMap,
|
|
tc,
|
|
metrics,
|
|
logger,
|
|
"fake", "fake-model",
|
|
map[string]bool{},
|
|
[]string{"echo"},
|
|
nil,
|
|
map[string]struct{}{},
|
|
nil,
|
|
defaultToolResultBytes,
|
|
nil,
|
|
nil,
|
|
)
|
|
|
|
textOutput, ok := result.Result.(fantasy.ToolResultOutputContentText)
|
|
require.True(t, ok, "expected ToolResultOutputContentText, got %T", result.Result)
|
|
require.True(t, utf8.ValidString(textOutput.Text), "Text should be valid UTF-8")
|
|
require.Contains(t, textOutput.Text, "hello")
|
|
require.Contains(t, textOutput.Text, "world")
|
|
})
|
|
}
|
|
|
|
func TestExecuteSingleTool_NormalizesMedia(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
var pngData bytes.Buffer
|
|
require.NoError(t, png.Encode(&pngData, image.NewRGBA(image.Rect(0, 0, 1, 1))))
|
|
|
|
for _, tc := range []struct {
|
|
name string
|
|
mediaType string
|
|
data []byte
|
|
// wantMediaType is empty when the media is rejected and only text survives.
|
|
wantMediaType string
|
|
wantText string
|
|
}{
|
|
{name: "OversizedMediaKeepsText", mediaType: "image/png", data: make([]byte, codersdk.MaxChatFileSizeBytes+1), wantText: "Ran Playwright code\n[image/png content omitted"},
|
|
{name: "NonImageBytesDeclaredAsImageKeepsText", mediaType: "image/png", data: []byte("<html>not an image</html>"), wantText: "Ran Playwright code\n[image omitted: payload declared as image/png is text/html"},
|
|
{name: "ImageTypeFollowsBytes", mediaType: "image/jpeg", data: pngData.Bytes(), wantMediaType: "image/png", wantText: "Ran Playwright code"},
|
|
{name: "NonImageMediaPassesThrough", mediaType: "audio/mpeg", data: []byte{1, 2, 3}, wantMediaType: "audio/mpeg", wantText: "Ran Playwright code"},
|
|
} {
|
|
t.Run(tc.name, func(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
tool := fantasy.NewAgentTool("screenshot", "takes a screenshot",
|
|
func(context.Context, struct{}, fantasy.ToolCall) (fantasy.ToolResponse, error) {
|
|
return fantasy.ToolResponse{Type: "media", Data: tc.data, MediaType: tc.mediaType, Content: "Ran Playwright code"}, nil
|
|
},
|
|
)
|
|
result, _ := executeSingleTool(
|
|
context.Background(),
|
|
map[string]fantasy.AgentTool{"screenshot": tool},
|
|
fantasy.ToolCallContent{ToolCallID: "call-1", ToolName: "screenshot", Input: "{}"},
|
|
NewMetrics(prometheus.NewRegistry()),
|
|
slog.Make(),
|
|
"openai", "model",
|
|
map[string]bool{},
|
|
[]string{"screenshot"},
|
|
nil,
|
|
map[string]struct{}{},
|
|
nil,
|
|
defaultToolResultBytes,
|
|
nil,
|
|
nil,
|
|
)
|
|
|
|
switch out := result.Result.(type) {
|
|
case fantasy.ToolResultOutputContentMedia:
|
|
require.Equal(t, tc.wantMediaType, out.MediaType)
|
|
require.Equal(t, tc.wantText, out.Text)
|
|
case fantasy.ToolResultOutputContentText:
|
|
require.Empty(t, tc.wantMediaType, "media rejected: %s", out.Text)
|
|
require.Contains(t, out.Text, tc.wantText)
|
|
default:
|
|
t.Fatalf("unexpected result %T", out)
|
|
}
|
|
})
|
|
}
|
|
}
|
|
|
|
func TestExecuteSingleTool_ResolvesToolNameAlias(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
metrics := NewMetrics(prometheus.NewRegistry())
|
|
logger := slog.Make()
|
|
|
|
var gotName string
|
|
tool := fantasy.NewAgentTool(
|
|
"interrupt_agent",
|
|
"interrupts an agent",
|
|
func(_ context.Context, _ struct{}, call fantasy.ToolCall) (fantasy.ToolResponse, error) {
|
|
gotName = call.Name
|
|
return fantasy.ToolResponse{Content: `{"interrupted":true}`}, nil
|
|
},
|
|
)
|
|
toolMap := map[string]fantasy.AgentTool{"interrupt_agent": tool}
|
|
|
|
// The model emits the deprecated name from old history; only the
|
|
// canonical name is advertised/active.
|
|
tc := fantasy.ToolCallContent{
|
|
ToolCallID: "call-alias",
|
|
ToolName: "close_agent",
|
|
Input: "{}",
|
|
}
|
|
|
|
result, _ := executeSingleTool(
|
|
context.Background(),
|
|
toolMap,
|
|
tc,
|
|
metrics,
|
|
logger,
|
|
"fake", "fake-model",
|
|
map[string]bool{},
|
|
[]string{"interrupt_agent"},
|
|
nil,
|
|
map[string]struct{}{},
|
|
nil,
|
|
defaultToolResultBytes,
|
|
map[string]string{"close_agent": "interrupt_agent"},
|
|
nil,
|
|
)
|
|
|
|
textOutput, ok := result.Result.(fantasy.ToolResultOutputContentText)
|
|
require.True(t, ok, "expected text output, got %T", result.Result)
|
|
require.Contains(t, textOutput.Text, "interrupted")
|
|
// The handler receives the resolved canonical name.
|
|
require.Equal(t, "interrupt_agent", gotName)
|
|
// The persisted result keeps the original alias so existing history
|
|
// renders consistently.
|
|
require.Equal(t, "close_agent", result.ToolName)
|
|
}
|
|
|
|
func TestExecuteSingleTool_UnknownAliasFallsThrough(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
metrics := NewMetrics(prometheus.NewRegistry())
|
|
logger := slog.Make()
|
|
|
|
tc := fantasy.ToolCallContent{
|
|
ToolCallID: "call-missing",
|
|
ToolName: "close_agent",
|
|
Input: "{}",
|
|
}
|
|
|
|
// No alias provided: the deprecated name is neither active nor in the
|
|
// tool map, so it surfaces a clear not-active error and the model can
|
|
// self-correct to the advertised name.
|
|
result, _ := executeSingleTool(
|
|
context.Background(),
|
|
map[string]fantasy.AgentTool{},
|
|
tc,
|
|
metrics,
|
|
logger,
|
|
"fake", "fake-model",
|
|
map[string]bool{},
|
|
[]string{"interrupt_agent"},
|
|
nil,
|
|
map[string]struct{}{},
|
|
nil,
|
|
defaultToolResultBytes,
|
|
nil,
|
|
nil,
|
|
)
|
|
|
|
errOutput, ok := result.Result.(fantasy.ToolResultOutputContentError)
|
|
require.True(t, ok, "expected error output, got %T", result.Result)
|
|
require.Contains(t, errOutput.Error.Error(), "close_agent")
|
|
}
|
|
|
|
func TestExecuteSingleTool_AllowsDeferredDirectCall(t *testing.T) {
|
|
t.Parallel()
|
|
tool := fantasy.NewAgentTool(
|
|
"server__direct",
|
|
"direct",
|
|
func(_ context.Context, _ struct{}, _ fantasy.ToolCall) (fantasy.ToolResponse, error) {
|
|
return fantasy.NewTextResponse("ok"), nil
|
|
},
|
|
)
|
|
result, _ := executeSingleTool(
|
|
context.Background(),
|
|
map[string]fantasy.AgentTool{"server__direct": tool},
|
|
fantasy.ToolCallContent{ToolCallID: "call-direct", ToolName: "server__direct", Input: "{}"},
|
|
NewMetrics(prometheus.NewRegistry()),
|
|
slog.Make(),
|
|
"fake", "fake-model",
|
|
map[string]bool{},
|
|
[]string{"find_tools"},
|
|
map[string]bool{"server__direct": true},
|
|
map[string]struct{}{},
|
|
nil,
|
|
defaultToolResultBytes,
|
|
nil,
|
|
nil,
|
|
)
|
|
text, ok := result.Result.(fantasy.ToolResultOutputContentText)
|
|
require.True(t, ok)
|
|
require.Equal(t, "ok", text.Text)
|
|
}
|