Slice 3 of 4 for runtime experiment rules: evaluate stored rules per user in discovery, the frontend experiments query, and the pilot `mcp-tool-search` gate in chatd. There is still no way to edit rules through the product; that comes in slice 4. With no rules stored and a healthy database, backend decisions match today. This PR deliberately changes three things: 1. **Fail closed on rule reads.** If the rules read fails, `mcp-tool-search` is off, even when it is enabled at startup. 2. **Frontend freshness.** The experiments query now refetches. 3. **Caching.** `/api/v2/experiments` and rendered HTML are sent with `Cache-Control: no-store`. Static assets keep their long-lived caching. ## What's in it - **coderd** builds `api.ExperimentEvaluator` from `experiments.NewDBStore` plus the startup `--experiments` list. - **Discovery:** `GET /api/v2/experiments` and the `<meta property="experiments">` tag both use `EnabledExperiments(ctx, caller)`. - **chatd:** `mcp-tool-search` is now user-scoped. `chatd.Config.ExperimentEvaluator` is required, so `chatd.New` returns an error without it. `prepareGeneration` decides once per turn for the chat owner (`chat.OwnerID`); child chats inherit the owner. - **Ruleguard** (`userScopedExperimentStaticCheck` in `scripts/rules.go`): rejects static `Experiments.Enabled(...)` and `httpmw.RequireExperiment*(...)` checks of user-scoped experiments in server packages. A codersdk test keeps its list equal to `codersdk.ExperimentsUserScoped`. - **Frontend:** the experiments query is keyed by user ID and seeded from the meta tag only when the embedded user matches. It is stale 60 s after page load and refetches on mount, window focus and reconnect. Logout also clears the experiments metadata. The dashboard keeps the last loaded list when a background refetch fails, instead of replacing the page with an error. - **Once per turn:** chatd decides `mcp-tool-search` on a turn's first step and reuses that decision for the turn's later steps, so a rule change applies from the next turn and never withdraws a `find_tools` call the model already made. The decision lives on the chat runner, keyed by the turn's prompt row, like the stop-nudge tracker. A runner handoff in the middle of a turn evaluates again. - **Reference audit:** the only non-test server check of `mcp-tool-search` is in `generation_preparer.go`, and it now goes through the evaluator. ## Testing - **Go:** per-user results and `no-store` on `/api/v2/experiments`; the per-user meta tag and the HTML cache headers in `site`; group and org conditions end to end in enterprise; chatd owner eligibility, on → off → on across turns including a child chat, and a failed rules read. - **Vitest:** user-keyed seeding, staleness from page load after a remount, refetch on focus, logout clearing, and the dashboard surviving a failed refetch. Remote dogfood UAT through Coder Agents passed on a licensed test deployment: - per-user discovery in both the API and the meta tag; - widening a rule, and turning it off, with no restart; - `off` beating a startup default, and `inherit` restoring it; - fail closed on malformed JSON and on an unknown mode, with no condition text in the logs; - the `no-store` headers; - real-model chatd enforcement: `find_tools` offered on turn 1, gone after `off`, back on turn 3. That round tested `b57207d4ee`. The later commits change only tests, and a follow-up round will cover the final head. **For the author:** `coderd/x/chatd/ARCHITECTURE.md`'s list of the runner's local state doesn't mention the per-turn trackers (debug turn, session start, stop nudges, and now this experiment decision). I did not edit that document. **Known behavior:** a broken rule logs one error line per evaluation (per request or turn), with only the experiment, revision, category and position. ## Stack 1. #29869: evaluator library 2. #29898: storage and audit types 3. **This PR:** enforcement and discovery 4. Control plane (experimental API, `coder exp experiment-rules` CLI, docs) > [!NOTE] > This PR was written by an AI agent (Xum). --- _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_ --------- Signed-off-by: Thomas Kosiewski <tk@coder.com>
76 lines
2.8 KiB
Go
76 lines
2.8 KiB
Go
package coderd_test
|
|
|
|
import (
|
|
"fmt"
|
|
"testing"
|
|
|
|
"github.com/stretchr/testify/require"
|
|
|
|
"github.com/coder/coder/v2/coderd/coderdtest"
|
|
"github.com/coder/coder/v2/coderd/database/dbauthz"
|
|
"github.com/coder/coder/v2/coderd/experiments"
|
|
"github.com/coder/coder/v2/codersdk"
|
|
"github.com/coder/coder/v2/enterprise/coderd/coderdenttest"
|
|
"github.com/coder/coder/v2/enterprise/coderd/license"
|
|
"github.com/coder/coder/v2/testutil"
|
|
)
|
|
|
|
// TestExperimentRuleConditions checks that conditions over group and
|
|
// organization membership decide /api/v2/experiments per user.
|
|
func TestExperimentRuleConditions(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
ownerClient, db, owner := coderdenttest.NewWithDatabase(t, &coderdenttest.Options{
|
|
LicenseOptions: &coderdenttest.LicenseOptions{
|
|
Features: license.Features{
|
|
codersdk.FeatureTemplateRBAC: 1,
|
|
codersdk.FeatureMultipleOrganizations: 1,
|
|
},
|
|
},
|
|
})
|
|
ctx := testutil.Context(t, testutil.WaitLong)
|
|
|
|
defaultOrg, err := ownerClient.Organization(ctx, owner.OrganizationID)
|
|
require.NoError(t, err)
|
|
otherOrg := coderdenttest.CreateOrganization(t, ownerClient, coderdenttest.CreateOrganizationOptions{})
|
|
|
|
groupedClient, grouped := coderdtest.CreateAnotherUser(t, ownerClient, owner.OrganizationID)
|
|
otherOrgClient, _ := coderdtest.CreateAnotherUser(t, ownerClient, otherOrg.ID)
|
|
plainClient, _ := coderdtest.CreateAnotherUser(t, ownerClient, owner.OrganizationID)
|
|
|
|
group, err := ownerClient.CreateGroup(ctx, owner.OrganizationID, codersdk.CreateGroupRequest{Name: "rollout"})
|
|
require.NoError(t, err)
|
|
_, err = ownerClient.PatchGroup(ctx, group.ID, codersdk.PatchGroupRequest{
|
|
AddUsers: []string{grouped.ID.String()},
|
|
})
|
|
require.NoError(t, err)
|
|
|
|
//nolint:gocritic // Tests seed rules directly; there is no rules API yet.
|
|
systemCtx := dbauthz.AsSystemRestricted(ctx)
|
|
for ex, condition := range map[codersdk.Experiment]string{
|
|
codersdk.ExperimentExample: fmt.Sprintf("%q in user.groups", defaultOrg.Name+"/"+group.Name),
|
|
codersdk.ExperimentMCPToolSearch: fmt.Sprintf("%q in user.organizations", otherOrg.Name),
|
|
} {
|
|
_, _, changed, err := experiments.WriteRule(systemCtx, db, owner.UserID, ex, experiments.Rule{
|
|
Mode: experiments.ModeCondition,
|
|
Condition: condition,
|
|
}, 0)
|
|
require.NoError(t, err)
|
|
require.True(t, changed)
|
|
}
|
|
|
|
for _, tc := range []struct {
|
|
name string
|
|
client *codersdk.Client
|
|
want codersdk.Experiments
|
|
}{
|
|
{name: "GroupMember", client: groupedClient, want: codersdk.Experiments{codersdk.ExperimentExample}},
|
|
{name: "OrganizationMember", client: otherOrgClient, want: codersdk.Experiments{codersdk.ExperimentMCPToolSearch}},
|
|
{name: "Neither", client: plainClient, want: codersdk.Experiments{}},
|
|
} {
|
|
got, err := tc.client.Experiments(ctx)
|
|
require.NoError(t, err, tc.name)
|
|
require.Equal(t, tc.want, got, tc.name)
|
|
}
|
|
}
|