Slice 3 of 4 for runtime experiment rules: evaluate stored rules per user in discovery, the frontend experiments query, and the pilot `mcp-tool-search` gate in chatd. There is still no way to edit rules through the product; that comes in slice 4. With no rules stored and a healthy database, backend decisions match today. This PR deliberately changes three things: 1. **Fail closed on rule reads.** If the rules read fails, `mcp-tool-search` is off, even when it is enabled at startup. 2. **Frontend freshness.** The experiments query now refetches. 3. **Caching.** `/api/v2/experiments` and rendered HTML are sent with `Cache-Control: no-store`. Static assets keep their long-lived caching. ## What's in it - **coderd** builds `api.ExperimentEvaluator` from `experiments.NewDBStore` plus the startup `--experiments` list. - **Discovery:** `GET /api/v2/experiments` and the `<meta property="experiments">` tag both use `EnabledExperiments(ctx, caller)`. - **chatd:** `mcp-tool-search` is now user-scoped. `chatd.Config.ExperimentEvaluator` is required, so `chatd.New` returns an error without it. `prepareGeneration` decides once per turn for the chat owner (`chat.OwnerID`); child chats inherit the owner. - **Ruleguard** (`userScopedExperimentStaticCheck` in `scripts/rules.go`): rejects static `Experiments.Enabled(...)` and `httpmw.RequireExperiment*(...)` checks of user-scoped experiments in server packages. A codersdk test keeps its list equal to `codersdk.ExperimentsUserScoped`. - **Frontend:** the experiments query is keyed by user ID and seeded from the meta tag only when the embedded user matches. It is stale 60 s after page load and refetches on mount, window focus and reconnect. Logout also clears the experiments metadata. The dashboard keeps the last loaded list when a background refetch fails, instead of replacing the page with an error. - **Once per turn:** chatd decides `mcp-tool-search` on a turn's first step and reuses that decision for the turn's later steps, so a rule change applies from the next turn and never withdraws a `find_tools` call the model already made. The decision lives on the chat runner, keyed by the turn's prompt row, like the stop-nudge tracker. A runner handoff in the middle of a turn evaluates again. - **Reference audit:** the only non-test server check of `mcp-tool-search` is in `generation_preparer.go`, and it now goes through the evaluator. ## Testing - **Go:** per-user results and `no-store` on `/api/v2/experiments`; the per-user meta tag and the HTML cache headers in `site`; group and org conditions end to end in enterprise; chatd owner eligibility, on → off → on across turns including a child chat, and a failed rules read. - **Vitest:** user-keyed seeding, staleness from page load after a remount, refetch on focus, logout clearing, and the dashboard surviving a failed refetch. Remote dogfood UAT through Coder Agents passed on a licensed test deployment: - per-user discovery in both the API and the meta tag; - widening a rule, and turning it off, with no restart; - `off` beating a startup default, and `inherit` restoring it; - fail closed on malformed JSON and on an unknown mode, with no condition text in the logs; - the `no-store` headers; - real-model chatd enforcement: `find_tools` offered on turn 1, gone after `off`, back on turn 3. That round tested `b57207d4ee`. The later commits change only tests, and a follow-up round will cover the final head. **For the author:** `coderd/x/chatd/ARCHITECTURE.md`'s list of the runner's local state doesn't mention the per-turn trackers (debug turn, session start, stop nudges, and now this experiment decision). I did not edit that document. **Known behavior:** a broken rule logs one error line per evaluation (per request or turn), with only the experiment, revision, category and position. ## Stack 1. #29869: evaluator library 2. #29898: storage and audit types 3. **This PR:** enforcement and discovery 4. Control plane (experimental API, `coder exp experiment-rules` CLI, docs) > [!NOTE] > This PR was written by an AI agent (Xum). --- _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_ --------- Signed-off-by: Thomas Kosiewski <tk@coder.com>
204 lines
6.6 KiB
Go
204 lines
6.6 KiB
Go
package chatd
|
|
|
|
import (
|
|
"context"
|
|
"encoding/json"
|
|
"testing"
|
|
|
|
"charm.land/fantasy"
|
|
"github.com/google/uuid"
|
|
"github.com/stretchr/testify/require"
|
|
|
|
"github.com/coder/coder/v2/coderd/database"
|
|
"github.com/coder/coder/v2/coderd/database/dbgen"
|
|
"github.com/coder/coder/v2/coderd/database/dbtestutil"
|
|
"github.com/coder/coder/v2/coderd/x/chatd/chathooks"
|
|
"github.com/coder/coder/v2/coderd/x/chatd/chatprompt"
|
|
"github.com/coder/coder/v2/coderd/x/chatd/chatprovider"
|
|
"github.com/coder/coder/v2/coderd/x/chatd/chatstate"
|
|
)
|
|
|
|
func TestPartitionAmbiguousToolCallsGatesOnBuiltins(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
fetch := fetchToolStub()
|
|
ambiguous := fantasy.ToolCallContent{
|
|
ToolCallID: "call_ambiguous",
|
|
ToolName: "fetch",
|
|
Input: `{"URL":"https://example.test","url":"https://other.test"}`,
|
|
}
|
|
clean := fantasy.ToolCallContent{
|
|
ToolCallID: "call_clean",
|
|
ToolName: "fetch",
|
|
Input: `{"url":"https://example.test"}`,
|
|
}
|
|
|
|
t.Run("builtin", func(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
prepared := generationPrepared{
|
|
Tools: []fantasy.AgentTool{fetch},
|
|
BuiltinToolNames: map[string]bool{"fetch": true},
|
|
}
|
|
allowed, allowedIndexes, rejected := partitionAmbiguousToolCalls(prepared, []fantasy.ToolCallContent{ambiguous, clean})
|
|
require.Len(t, rejected, 1)
|
|
require.Equal(t, "call_ambiguous", rejected[0].ToolCallID)
|
|
require.Len(t, allowed, 1)
|
|
require.Equal(t, "call_clean", allowed[0].ToolCallID)
|
|
require.Equal(t, []int{1}, allowedIndexes, "allowed indexes point at positions in the input slice")
|
|
})
|
|
|
|
t.Run("non-builtin", func(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
prepared := generationPrepared{Tools: []fantasy.AgentTool{fetch}}
|
|
allowed, allowedIndexes, rejected := partitionAmbiguousToolCalls(prepared, []fantasy.ToolCallContent{ambiguous, clean})
|
|
require.Empty(t, rejected)
|
|
require.Len(t, allowed, 2)
|
|
require.Equal(t, []int{0, 1}, allowedIndexes)
|
|
})
|
|
|
|
// Execution resolves a deprecated name to its canonical tool, so
|
|
// validation has to resolve it too.
|
|
t.Run("deprecated alias", func(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
var alias, canonical string
|
|
for a, name := range subagentToolNameAliases {
|
|
alias, canonical = a, name
|
|
break
|
|
}
|
|
require.NotEmpty(t, canonical)
|
|
|
|
type input struct {
|
|
ChatID string `json:"chat_id"`
|
|
}
|
|
tool := fantasy.NewAgentTool(canonical, "",
|
|
func(context.Context, input, fantasy.ToolCall) (fantasy.ToolResponse, error) {
|
|
return fantasy.ToolResponse{}, nil
|
|
})
|
|
aliased := fantasy.ToolCallContent{
|
|
ToolCallID: "call_aliased",
|
|
ToolName: alias,
|
|
Input: `{"chat_id":"a","CHAT_ID":"b"}`,
|
|
}
|
|
|
|
prepared := generationPrepared{
|
|
Tools: []fantasy.AgentTool{tool},
|
|
BuiltinToolNames: map[string]bool{canonical: true},
|
|
}
|
|
_, _, rejected := partitionAmbiguousToolCalls(prepared, []fantasy.ToolCallContent{aliased})
|
|
require.Len(t, rejected, 1)
|
|
})
|
|
}
|
|
|
|
func TestValidateOverriddenToolInputs(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
prepared := generationPrepared{
|
|
Tools: []fantasy.AgentTool{fetchToolStub()},
|
|
BuiltinToolNames: map[string]bool{"fetch": true},
|
|
}
|
|
overridden := chathooks.PreToolUseExecutionResult{
|
|
Allowed: []fantasy.ToolCallContent{{
|
|
ToolCallID: "call_overridden",
|
|
ToolName: "fetch",
|
|
Input: `{"URL":"https://other.test"}`,
|
|
}},
|
|
Overrides: map[string]json.RawMessage{
|
|
"call_overridden": json.RawMessage(`{"URL":"https://other.test"}`),
|
|
},
|
|
}
|
|
require.ErrorContains(t, validateOverriddenToolInputs(prepared, overridden),
|
|
`hook input override for tool fetch: input key "URL" differs from schema property "url" only by case`)
|
|
|
|
// The same input is left alone when no consumer replaced it, because
|
|
// the model-authored batch is checked before the dispatch instead.
|
|
untouched := chathooks.PreToolUseExecutionResult{Allowed: overridden.Allowed}
|
|
require.NoError(t, validateOverriddenToolInputs(prepared, untouched))
|
|
}
|
|
|
|
// TestBuiltinToolSchemasDescribeTheirInputs guards the validator's reach: it
|
|
// cannot detect a case-variant key for a builtin that declares no properties.
|
|
func TestBuiltinToolSchemasDescribeTheirInputs(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
db, ps := dbtestutil.NewDB(t)
|
|
ctx := chatdTestContext(t)
|
|
user := dbgen.User(t, db, database.User{})
|
|
org := dbgen.Organization(t, db, database.Organization{})
|
|
dbgen.OrganizationMember(t, db, database.OrganizationMember{
|
|
UserID: user.ID,
|
|
OrganizationID: org.ID,
|
|
})
|
|
provider := dbgen.AIProviderWithOptionalKey(t, db, database.AIProvider{
|
|
Type: database.AIProviderTypeOpenai,
|
|
}, "test-key")
|
|
modelConfig := dbgen.ChatModelConfig(t, db, database.ChatModelConfig{
|
|
Model: "gpt-4o-mini",
|
|
AIProviderID: uuid.NullUUID{UUID: provider.ID, Valid: true},
|
|
OrganizationID: org.ID,
|
|
}, func(p *database.InsertChatModelConfigParams) {
|
|
p.Enabled = true
|
|
})
|
|
created, err := chatstate.CreateChat(ctx, db, ps, chatstate.CreateChatInput{
|
|
OrganizationID: org.ID,
|
|
OwnerID: user.ID,
|
|
LastModelConfigID: modelConfig.ID,
|
|
Title: "builtin tool schemas",
|
|
ClientType: database.ChatClientTypeApi,
|
|
InitialMessages: []chatstate.Message{{
|
|
Role: database.ChatMessageRoleUser,
|
|
Content: mustMarshalText(t, "hello"),
|
|
Visibility: database.ChatMessageVisibilityBoth,
|
|
ModelConfigID: uuid.NullUUID{UUID: modelConfig.ID, Valid: true},
|
|
CreatedBy: uuid.NullUUID{UUID: user.ID, Valid: true},
|
|
ContentVersion: chatprompt.CurrentContentVersion,
|
|
}},
|
|
})
|
|
require.NoError(t, err)
|
|
|
|
server := newInternalTestServer(
|
|
t,
|
|
db,
|
|
ps,
|
|
chatprovider.ProviderAPIKeys{},
|
|
withInternalTestServerTransportFactory(&aibridgeTestFactory{}),
|
|
)
|
|
prepared, err := server.prepareGeneration(ctx, generationPrepareInput{
|
|
Chat: created.Chat,
|
|
Messages: created.InitialMessages,
|
|
TurnExperiments: &turnExperimentDecisions{},
|
|
})
|
|
require.NoError(t, err)
|
|
t.Cleanup(prepared.Cleanup)
|
|
|
|
// These take an empty struct, so they carry no keys to validate.
|
|
noInput := map[string]bool{
|
|
"process_list": true,
|
|
"stop_workspace": true,
|
|
"list_subagent_models": true,
|
|
}
|
|
var unvalidated []string
|
|
require.NotEmpty(t, prepared.BuiltinToolNames)
|
|
for _, tool := range prepared.Tools {
|
|
info := tool.Info()
|
|
if !prepared.BuiltinToolNames[info.Name] || len(info.Parameters) > 0 || noInput[info.Name] {
|
|
continue
|
|
}
|
|
unvalidated = append(unvalidated, info.Name)
|
|
}
|
|
require.Empty(t, unvalidated,
|
|
"these builtin tools declare no schema properties, so their input is not validated")
|
|
}
|
|
|
|
func fetchToolStub() fantasy.AgentTool {
|
|
type input struct {
|
|
URL string `json:"url"`
|
|
}
|
|
return fantasy.NewAgentTool("fetch", "",
|
|
func(context.Context, input, fantasy.ToolCall) (fantasy.ToolResponse, error) {
|
|
return fantasy.ToolResponse{}, nil
|
|
})
|
|
}
|