Files
coder/enterprise/coderd/experiments_test.go
T
Thomas Kosiewski 9aa7130e01 feat: evaluate experiment rules per user in discovery and chatd (#29964)
Slice 3 of 4 for runtime experiment rules: evaluate stored rules per
user in discovery, the frontend experiments query, and the pilot
`mcp-tool-search` gate in chatd. There is still no way to edit rules
through the product; that comes in slice 4.

With no rules stored and a healthy database, backend decisions match
today. This PR deliberately changes three things:

1. **Fail closed on rule reads.** If the rules read fails,
`mcp-tool-search` is off, even when it is enabled at startup.
2. **Frontend freshness.** The experiments query now refetches.
3. **Caching.** `/api/v2/experiments` and rendered HTML are sent with
`Cache-Control: no-store`. Static assets keep their long-lived caching.

## What's in it

- **coderd** builds `api.ExperimentEvaluator` from
`experiments.NewDBStore` plus the startup `--experiments` list.
- **Discovery:** `GET /api/v2/experiments` and the `<meta
property="experiments">` tag both use `EnabledExperiments(ctx, caller)`.
- **chatd:** `mcp-tool-search` is now user-scoped.
`chatd.Config.ExperimentEvaluator` is required, so `chatd.New` returns
an error without it. `prepareGeneration` decides once per turn for the
chat owner (`chat.OwnerID`); child chats inherit the owner.
- **Ruleguard** (`userScopedExperimentStaticCheck` in
`scripts/rules.go`): rejects static `Experiments.Enabled(...)` and
`httpmw.RequireExperiment*(...)` checks of user-scoped experiments in
server packages. A codersdk test keeps its list equal to
`codersdk.ExperimentsUserScoped`.
- **Frontend:** the experiments query is keyed by user ID and seeded
from the meta tag only when the embedded user matches. It is stale 60 s
after page load and refetches on mount, window focus and reconnect.
Logout also clears the experiments metadata. The dashboard keeps the
last loaded list when a background refetch fails, instead of replacing
the page with an error.
- **Once per turn:** chatd decides `mcp-tool-search` on a turn's first
step and reuses that decision for the turn's later steps, so a rule
change applies from the next turn and never withdraws a `find_tools`
call the model already made. The decision lives on the chat runner,
keyed by the turn's prompt row, like the stop-nudge tracker. A runner
handoff in the middle of a turn evaluates again.
- **Reference audit:** the only non-test server check of
`mcp-tool-search` is in `generation_preparer.go`, and it now goes
through the evaluator.

## Testing

- **Go:** per-user results and `no-store` on `/api/v2/experiments`; the
per-user meta tag and the HTML cache headers in `site`; group and org
conditions end to end in enterprise; chatd owner eligibility, on → off →
on across turns including a child chat, and a failed rules read.
- **Vitest:** user-keyed seeding, staleness from page load after a
remount, refetch on focus, logout clearing, and the dashboard surviving
a failed refetch.

Remote dogfood UAT through Coder Agents passed on a licensed test
deployment:
- per-user discovery in both the API and the meta tag;
- widening a rule, and turning it off, with no restart;
- `off` beating a startup default, and `inherit` restoring it;
- fail closed on malformed JSON and on an unknown mode, with no
condition text in the logs;
- the `no-store` headers;
- real-model chatd enforcement: `find_tools` offered on turn 1, gone
after `off`, back on turn 3.

That round tested `b57207d4ee`. The later commits change only tests, and
a follow-up round will cover the final head.

**For the author:** `coderd/x/chatd/ARCHITECTURE.md`'s list of the
runner's local state doesn't mention the per-turn trackers (debug turn,
session start, stop nudges, and now this experiment decision). I did not
edit that document.

**Known behavior:** a broken rule logs one error line per evaluation
(per request or turn), with only the experiment, revision, category and
position.

## Stack

1. #29869: evaluator library
2. #29898: storage and audit types
3. **This PR:** enforcement and discovery
4. Control plane (experimental API, `coder exp experiment-rules` CLI,
docs)

> [!NOTE]
> This PR was written by an AI agent (Xum).

---
_Generated with [`xum`](https://github.com/coder/xum) • Model:
`anthropic:claude-opus-5-5` • Thinking: `high`_

---------

Signed-off-by: Thomas Kosiewski <tk@coder.com>
2026-09-29 12:23:00 +00:00

76 lines
2.8 KiB
Go

package coderd_test
import (
"fmt"
"testing"
"github.com/stretchr/testify/require"
"github.com/coder/coder/v2/coderd/coderdtest"
"github.com/coder/coder/v2/coderd/database/dbauthz"
"github.com/coder/coder/v2/coderd/experiments"
"github.com/coder/coder/v2/codersdk"
"github.com/coder/coder/v2/enterprise/coderd/coderdenttest"
"github.com/coder/coder/v2/enterprise/coderd/license"
"github.com/coder/coder/v2/testutil"
)
// TestExperimentRuleConditions checks that conditions over group and
// organization membership decide /api/v2/experiments per user.
func TestExperimentRuleConditions(t *testing.T) {
t.Parallel()
ownerClient, db, owner := coderdenttest.NewWithDatabase(t, &coderdenttest.Options{
LicenseOptions: &coderdenttest.LicenseOptions{
Features: license.Features{
codersdk.FeatureTemplateRBAC: 1,
codersdk.FeatureMultipleOrganizations: 1,
},
},
})
ctx := testutil.Context(t, testutil.WaitLong)
defaultOrg, err := ownerClient.Organization(ctx, owner.OrganizationID)
require.NoError(t, err)
otherOrg := coderdenttest.CreateOrganization(t, ownerClient, coderdenttest.CreateOrganizationOptions{})
groupedClient, grouped := coderdtest.CreateAnotherUser(t, ownerClient, owner.OrganizationID)
otherOrgClient, _ := coderdtest.CreateAnotherUser(t, ownerClient, otherOrg.ID)
plainClient, _ := coderdtest.CreateAnotherUser(t, ownerClient, owner.OrganizationID)
group, err := ownerClient.CreateGroup(ctx, owner.OrganizationID, codersdk.CreateGroupRequest{Name: "rollout"})
require.NoError(t, err)
_, err = ownerClient.PatchGroup(ctx, group.ID, codersdk.PatchGroupRequest{
AddUsers: []string{grouped.ID.String()},
})
require.NoError(t, err)
//nolint:gocritic // Tests seed rules directly; there is no rules API yet.
systemCtx := dbauthz.AsSystemRestricted(ctx)
for ex, condition := range map[codersdk.Experiment]string{
codersdk.ExperimentExample: fmt.Sprintf("%q in user.groups", defaultOrg.Name+"/"+group.Name),
codersdk.ExperimentMCPToolSearch: fmt.Sprintf("%q in user.organizations", otherOrg.Name),
} {
_, _, changed, err := experiments.WriteRule(systemCtx, db, owner.UserID, ex, experiments.Rule{
Mode: experiments.ModeCondition,
Condition: condition,
}, 0)
require.NoError(t, err)
require.True(t, changed)
}
for _, tc := range []struct {
name string
client *codersdk.Client
want codersdk.Experiments
}{
{name: "GroupMember", client: groupedClient, want: codersdk.Experiments{codersdk.ExperimentExample}},
{name: "OrganizationMember", client: otherOrgClient, want: codersdk.Experiments{codersdk.ExperimentMCPToolSearch}},
{name: "Neither", client: plainClient, want: codersdk.Experiments{}},
} {
got, err := tc.client.Experiments(ctx)
require.NoError(t, err, tc.name)
require.Equal(t, tc.want, got, tc.name)
}
}