## Why
The Automations RFC controls availability with one user-scoped
experiment. Its text: "The user-scoped `chat-automations` experiment
controls availability. Operators configure defaults with `--experiments`
and can change per-user rules while Coder is running. Each surface
checks the relevant caller or automation's owner, and experiment
evaluation fails closed." The later backend PRs (management routes, the
`manage_automations` tool, `Publish`) need that check first. This PR
registers the experiment and proves the per-user, fail-closed check.
Nothing consumes it yet.
Part of CODAGT-1135
## What changed
- Registers `chat-automations` in `ExperimentsKnown` and
`ExperimentsUserScoped`. An earlier experiment with this name was
removed in #30412, and this PR restores none of its consumers.
- Keeps it out of `ExperimentsSafe`, so `--experiments=*` does not turn
it on. Automations act with their owner's authority, and a test pins the
choice.
- Adds the name to the `userScopedExperimentStaticCheck` ruleguard rule,
so server code cannot decide it from the static list.
`TestExperimentsUserScopedRuleguard` keeps the rule in step with
`ExperimentsUserScoped`.
- Regenerates the swagger files, `docs/reference/api/schemas.md` and
`typesGenerated.ts`.
- Adds tests for the real experiment name. The evaluator tests cover the
startup default, stored `on`, `off` and condition rules for two users,
and an off decision on a rules read error, a malformed rule, an unknown
mode, an attribute read error, a condition error and a nil subject. A
coderd test with a real database checks the same per-user behavior
through `api.ExperimentEvaluator` and `GET /api/v2/experiments`.
## Scope
No routes, tool, UI or chatd change. I added no helper:
`Evaluator.Enabled(ctx, userID, ex)` already decides per user and fails
closed, and API handlers reach it through `api.ExperimentEvaluator`.
chatd has no evaluator handle on main, because #31327 removed the unused
field. The first chatd consumer adds it together with its read.
## Blast radius
With the experiment off, which is the default, behavior on main does not
change. Two things become visible. `GET
/api/experimental/experiments/rules` and `coder exp experiment-rules
list` show a `chat-automations` row, and an administrator can store a
rule for it. A deployment that already passes
`--experiments=chat-automations` sees three small differences: the
startup log reports an opted-in hidden experiment where it reported an
unknown one, the deployment Overview page stops flagging the name as
invalid, and the rules row shows the static default as on. Nothing reads
the experiment yet, so no feature turns on.
## Live check
A local `./scripts/develop.sh` deployment built from this head ran with
no flag, with `--experiments=*` and with
`--experiments=chat-automations`. `GET /api/v2/experiments` lists
`chat-automations` for the owner and for a member only when the startup
list or a rule turns it on, and `--experiments=*` does not turn it on. A
condition rule `user.username == "member"` turned it on for the member
and off for the owner without a restart, and `reset` restored the
startup default.
---
_Generated with [`xum`](https://github.com/coder/xum) • Model:
`anthropic:claude-sonnet-5-5` • Thinking: `xhigh`_
Slice 3 of 4 for runtime experiment rules: evaluate stored rules per
user in discovery, the frontend experiments query, and the pilot
`mcp-tool-search` gate in chatd. There is still no way to edit rules
through the product; that comes in slice 4.
With no rules stored and a healthy database, backend decisions match
today. This PR deliberately changes three things:
1. **Fail closed on rule reads.** If the rules read fails,
`mcp-tool-search` is off, even when it is enabled at startup.
2. **Frontend freshness.** The experiments query now refetches.
3. **Caching.** `/api/v2/experiments` and rendered HTML are sent with
`Cache-Control: no-store`. Static assets keep their long-lived caching.
## What's in it
- **coderd** builds `api.ExperimentEvaluator` from
`experiments.NewDBStore` plus the startup `--experiments` list.
- **Discovery:** `GET /api/v2/experiments` and the `<meta
property="experiments">` tag both use `EnabledExperiments(ctx, caller)`.
- **chatd:** `mcp-tool-search` is now user-scoped.
`chatd.Config.ExperimentEvaluator` is required, so `chatd.New` returns
an error without it. `prepareGeneration` decides once per turn for the
chat owner (`chat.OwnerID`); child chats inherit the owner.
- **Ruleguard** (`userScopedExperimentStaticCheck` in
`scripts/rules.go`): rejects static `Experiments.Enabled(...)` and
`httpmw.RequireExperiment*(...)` checks of user-scoped experiments in
server packages. A codersdk test keeps its list equal to
`codersdk.ExperimentsUserScoped`.
- **Frontend:** the experiments query is keyed by user ID and seeded
from the meta tag only when the embedded user matches. It is stale 60 s
after page load and refetches on mount, window focus and reconnect.
Logout also clears the experiments metadata. The dashboard keeps the
last loaded list when a background refetch fails, instead of replacing
the page with an error.
- **Once per turn:** chatd decides `mcp-tool-search` on a turn's first
step and reuses that decision for the turn's later steps, so a rule
change applies from the next turn and never withdraws a `find_tools`
call the model already made. The decision lives on the chat runner,
keyed by the turn's prompt row, like the stop-nudge tracker. A runner
handoff in the middle of a turn evaluates again.
- **Reference audit:** the only non-test server check of
`mcp-tool-search` is in `generation_preparer.go`, and it now goes
through the evaluator.
## Testing
- **Go:** per-user results and `no-store` on `/api/v2/experiments`; the
per-user meta tag and the HTML cache headers in `site`; group and org
conditions end to end in enterprise; chatd owner eligibility, on → off →
on across turns including a child chat, and a failed rules read.
- **Vitest:** user-keyed seeding, staleness from page load after a
remount, refetch on focus, logout clearing, and the dashboard surviving
a failed refetch.
Remote dogfood UAT through Coder Agents passed on a licensed test
deployment:
- per-user discovery in both the API and the meta tag;
- widening a rule, and turning it off, with no restart;
- `off` beating a startup default, and `inherit` restoring it;
- fail closed on malformed JSON and on an unknown mode, with no
condition text in the logs;
- the `no-store` headers;
- real-model chatd enforcement: `find_tools` offered on turn 1, gone
after `off`, back on turn 3.
That round tested `b57207d4ee`. The later commits change only tests, and
a follow-up round will cover the final head.
**For the author:** `coderd/x/chatd/ARCHITECTURE.md`'s list of the
runner's local state doesn't mention the per-turn trackers (debug turn,
session start, stop nudges, and now this experiment decision). I did not
edit that document.
**Known behavior:** a broken rule logs one error line per evaluation
(per request or turn), with only the experiment, revision, category and
position.
## Stack
1. #29869: evaluator library
2. #29898: storage and audit types
3. **This PR:** enforcement and discovery
4. Control plane (experimental API, `coder exp experiment-rules` CLI,
docs)
> [!NOTE]
> This PR was written by an AI agent (Xum).
---
_Generated with [`xum`](https://github.com/coder/xum) • Model:
`anthropic:claude-opus-5-5` • Thinking: `high`_
---------
Signed-off-by: Thomas Kosiewski <tk@coder.com>
* added new option table type for experiments
* added tests
* fixed go tests
* added go test for new param
* removing query change
* clearing ExperimentsAll
* dont mutate ExperimentsAll
* added new route for safe experiments
* added new route for safe experiments
* added test for new route
* PR feedback
* altered design
* alias children
* chore: add /v2 to import module path
go mod requires semantic versioning with versions greater than 1.x
This was a mechanical update by running:
```
go install github.com/marwan-at-work/mod/cmd/mod@latest
mod upgrade
```
Migrate generated files to import /v2
* Fix gen
- Deprecates the --experimental flag
- Adds a new flag --experiments which supports passing multiple comma-separated values or a wildcard value.
- Exposes a new endpoint /api/v2/experiments that returns the list of enabled experiments.
- Deprecates the field Features.Experimental in favour of this new API.
- Updates apidocgen to support type aliases (shoutout to @mtojek).
- Modifies apitypings to support generating slice types.
- Updates develop.sh to pass additional args after -- to $CODERD_SHIM.