Commit Graph
8 Commits
Author SHA1 Message Date
Thomas Kosiewski 12be685b59 feat: add experimental API and CLI to manage experiment rules (#29971)
Slice 4 of 4 for runtime experiment rules: an experimental management
API and a hidden CLI, so admins can target, widen, or kill a user-scoped
experiment at runtime without a redeploy. This PR also adds a
startup-default note on the Deployment Overview page and a docs section.

## What's in it

- **API** under `/api/experimental/experiments/rules` (behind
`apiKeyMiddleware` and `httpmw.NoStore`):
- `GET` returns, per user-scoped experiment, the startup default and the
stored rule (mode, condition, revision, actor, time). Stored rules for
other experiments follow, marked `ignored`.
  - `PUT /{experiment}` takes `{mode, condition?, expected_revision}`.
- **Write path:**
- It authorizes read and update on `DeploymentConfig` before parsing
anything. A denied request gets 403 and is not audited.
- It validates next. An invalid condition returns 400 with the full CEL
diagnostics, is not audited, and leaves the rule unchanged.
- It then starts the audit and calls `experiments.WriteRule`. A stale
revision returns 409 and a transaction error returns 500; both are
audited with an empty diff.
- An identical write returns 200, creates no audit entry, and keeps the
revision.
- The audit export happens after commit. If it fails, that is logged and
the committed change stands (the existing audit contract).
- **SDK:** `ExperimentalClient.ExperimentRules` and
`ExperimentalClient.PutExperimentRule`.
- **CLI:** hidden `coder exp experiment-rules list|on|off|set|reset`.
- Write commands take `--expected-revision`. Without it they read the
current revision first, which only protects the gap between the two
calls; the help text says so.
  - A 409 prints the current rule and exits non-zero, with no retry.
- `reset` warns when the startup default is on, because reset is not a
kill switch.
- **Deployment Overview:** a note says the Experiments values are
startup defaults that runtime rules can override per user.
- **Docs:** a new "Target experiments at runtime" section in
`docs/reference/feature-stages.md` covers the modes, roles, CEL fields
and limits, reset vs. off, revisions and audit, the kill-switch
guarantee (including the rolling-upgrade requirement), frontend
freshness, fail-closed behavior, and that rules are not an authorization
boundary.

## Testing

- **HTTP:** every mode applies on the next request with no restart;
rejected writes keep the rule; RBAC (members 403 on GET and PUT;
auditors can read but not write; an invalid body from an unauthorized
caller still gets 403); malformed and ignored stored rows.
- **HA:** two coderd instances share one database. A PUT through A is
seen by B's `/api/v2/experiments` and by B's evaluator, which is the one
chatd uses.
- **Audit (enterprise, licensed):** an effective change creates one
entry whose diff carries the condition, and the entry is found before
the diff is checked. No-ops, 403s and 400s create no entry. A 409 and an
injected 500 are audited with an empty diff. A failed export keeps the
change.
- **Redaction:** a sentinel condition appears only in the rules API,
audit entries, and the server log's audit lines (Enterprise's slog audit
backend writes every audit diff there). It is absent from every other
log line, `/deployment/config`, the support bundle, and telemetry. The
test wires the slog audit backend so this is checked as stated.
- **CLI:** write and list, a conflict that is not retried, and the reset
warning.

Remote dogfood UAT through Coder Agents passed on a licensed test
deployment:
- the CLI;
- per-user results in the browser;
- 403 for members, including an invalid body;
- group targeting, including removing the user from the group;
- sentinel redaction across logs, `/deployment/config`, the support
bundle and the audit API (at that head the audit still masked the
condition; it is now tracked);
- the audit page showing the condition as changed but masked (same
note);
- the Overview note;
- the static default with `off` and `reset`;
- chatd enforcement toggled between turns.

That round tested `135aaaa41e`. The later commits are a CLI refactor and
a test addition, and a follow-up round will cover the final head.

## Follow-ups (low, not in this PR)

- A broken rule logs one error line per evaluation. Consider
rate-limiting it per experiment and revision.
- There's no API or CLI path to clear a stored rule for an experiment
that isn't user-scoped; those rules are inert and listed as `ignored`.
- A malformed stored rule is listed with zero `updated_by` and
`updated_at` values instead of omitting them.
- CLI wording: the 409 message, conflict output showing a UUID instead
of a username, a no-op write not saying nothing changed, and the parent
help lacking the `user.*` fields.
- `PutExperimentRuleRequest.mode` is not in the Swagger `required` list.
An omitted mode already returns 400.
- The dashboard's experiments query (from #29964) refreshes on mount,
window focus or reconnect once 60 seconds old, not on a timer, and its
comment says it bounds staleness. Correct the comment, or add a refetch
interval if the UI must pick up rule changes without a reload.

## Stack

1. #29869: evaluator library
2. #29898: storage and audit types
3. #29964: enforcement and discovery
4. **This PR:** control plane

> [!NOTE]
> This PR was written by an AI agent (Xum).

---
_Generated with [`xum`](https://github.com/coder/xum) • Model:
`anthropic:claude-opus-5-5` • Thinking: `high`_

---------

Signed-off-by: Thomas Kosiewski <tk@coder.com>
2026-09-29 12:23:00 +00:00
Thomas Kosiewski 9aa7130e01 feat: evaluate experiment rules per user in discovery and chatd (#29964)
Slice 3 of 4 for runtime experiment rules: evaluate stored rules per
user in discovery, the frontend experiments query, and the pilot
`mcp-tool-search` gate in chatd. There is still no way to edit rules
through the product; that comes in slice 4.

With no rules stored and a healthy database, backend decisions match
today. This PR deliberately changes three things:

1. **Fail closed on rule reads.** If the rules read fails,
`mcp-tool-search` is off, even when it is enabled at startup.
2. **Frontend freshness.** The experiments query now refetches.
3. **Caching.** `/api/v2/experiments` and rendered HTML are sent with
`Cache-Control: no-store`. Static assets keep their long-lived caching.

## What's in it

- **coderd** builds `api.ExperimentEvaluator` from
`experiments.NewDBStore` plus the startup `--experiments` list.
- **Discovery:** `GET /api/v2/experiments` and the `<meta
property="experiments">` tag both use `EnabledExperiments(ctx, caller)`.
- **chatd:** `mcp-tool-search` is now user-scoped.
`chatd.Config.ExperimentEvaluator` is required, so `chatd.New` returns
an error without it. `prepareGeneration` decides once per turn for the
chat owner (`chat.OwnerID`); child chats inherit the owner.
- **Ruleguard** (`userScopedExperimentStaticCheck` in
`scripts/rules.go`): rejects static `Experiments.Enabled(...)` and
`httpmw.RequireExperiment*(...)` checks of user-scoped experiments in
server packages. A codersdk test keeps its list equal to
`codersdk.ExperimentsUserScoped`.
- **Frontend:** the experiments query is keyed by user ID and seeded
from the meta tag only when the embedded user matches. It is stale 60 s
after page load and refetches on mount, window focus and reconnect.
Logout also clears the experiments metadata. The dashboard keeps the
last loaded list when a background refetch fails, instead of replacing
the page with an error.
- **Once per turn:** chatd decides `mcp-tool-search` on a turn's first
step and reuses that decision for the turn's later steps, so a rule
change applies from the next turn and never withdraws a `find_tools`
call the model already made. The decision lives on the chat runner,
keyed by the turn's prompt row, like the stop-nudge tracker. A runner
handoff in the middle of a turn evaluates again.
- **Reference audit:** the only non-test server check of
`mcp-tool-search` is in `generation_preparer.go`, and it now goes
through the evaluator.

## Testing

- **Go:** per-user results and `no-store` on `/api/v2/experiments`; the
per-user meta tag and the HTML cache headers in `site`; group and org
conditions end to end in enterprise; chatd owner eligibility, on → off →
on across turns including a child chat, and a failed rules read.
- **Vitest:** user-keyed seeding, staleness from page load after a
remount, refetch on focus, logout clearing, and the dashboard surviving
a failed refetch.

Remote dogfood UAT through Coder Agents passed on a licensed test
deployment:
- per-user discovery in both the API and the meta tag;
- widening a rule, and turning it off, with no restart;
- `off` beating a startup default, and `inherit` restoring it;
- fail closed on malformed JSON and on an unknown mode, with no
condition text in the logs;
- the `no-store` headers;
- real-model chatd enforcement: `find_tools` offered on turn 1, gone
after `off`, back on turn 3.

That round tested `b57207d4ee`. The later commits change only tests, and
a follow-up round will cover the final head.

**For the author:** `coderd/x/chatd/ARCHITECTURE.md`'s list of the
runner's local state doesn't mention the per-turn trackers (debug turn,
session start, stop nudges, and now this experiment decision). I did not
edit that document.

**Known behavior:** a broken rule logs one error line per evaluation
(per request or turn), with only the experiment, revision, category and
position.

## Stack

1. #29869: evaluator library
2. #29898: storage and audit types
3. **This PR:** enforcement and discovery
4. Control plane (experimental API, `coder exp experiment-rules` CLI,
docs)

> [!NOTE]
> This PR was written by an AI agent (Xum).

---
_Generated with [`xum`](https://github.com/coder/xum) • Model:
`anthropic:claude-opus-5-5` • Thinking: `high`_

---------

Signed-off-by: Thomas Kosiewski <tk@coder.com>
2026-09-29 12:23:00 +00:00
Atif Ali fad69df710 fix: correct SCIM Swagger try it out URLs (#24779) 2026-05-05 02:54:03 +05:00
Cian Johnston d892427b78 fix: do not warn on valid known experiments (#18514)
Fixes https://github.com/coder/coder/issues/18024

* drive-by: renames `handleExperimentsSafe` to
`handleExperimentsAvailable` to better match semantics
* defines list of `codersdk.ExperimentsKnown` and updates
`ReadExperiments` to log on invalid experiments
* typescript-ignores `codersdk.Experiments` so apitypings generates a
valid enum list of possible values of experiment
* updates OverviewPageView to distinguish between known 'hidden'
experiments and unknown 'invalid' experiments
2025-06-24 09:14:41 +01:00
Steven Masley ca38729840 chore: revert dynamic params as a safe experiment (#17510) 2025-04-22 16:21:15 +00:00
Kira Pilot 1656249e07 feat: add all safe experiments to the deployment page (#10276)
* added new option table type for experiments

* added tests

* fixed go tests

* added go test for new param

* removing query change

* clearing ExperimentsAll

* dont mutate ExperimentsAll

* added new route for safe experiments

* added new route for safe experiments

* added test for new route

* PR feedback

* altered design

* alias children
2023-10-17 14:49:19 -04:00
Kyle Carberry 22e781eced chore: add /v2 to import module path (#9072)
* chore: add /v2 to import module path

go mod requires semantic versioning with versions greater than 1.x

This was a mechanical update by running:
```
go install github.com/marwan-at-work/mod/cmd/mod@latest
mod upgrade
```

Migrate generated files to import /v2

* Fix gen
2023-08-18 18:55:43 +00:00
Cian Johnston 56b996532f feat: add --experiments flag to replace --experimental (#5767)
- Deprecates the --experimental flag
- Adds a new flag --experiments which supports passing multiple comma-separated values or a wildcard value.
- Exposes a new endpoint /api/v2/experiments that returns the list of enabled experiments.
- Deprecates the field Features.Experimental in favour of this new API.
- Updates apidocgen to support type aliases (shoutout to @mtojek).
- Modifies apitypings to support generating slice types.
- Updates develop.sh to pass additional args after -- to $CODERD_SHIM.
2023-01-18 19:12:53 +00:00