Files
coder/enterprise/coderd/experimentrules_test.go
Thomas Kosiewski 12be685b59 feat: add experimental API and CLI to manage experiment rules (#29971)
Slice 4 of 4 for runtime experiment rules: an experimental management
API and a hidden CLI, so admins can target, widen, or kill a user-scoped
experiment at runtime without a redeploy. This PR also adds a
startup-default note on the Deployment Overview page and a docs section.

## What's in it

- **API** under `/api/experimental/experiments/rules` (behind
`apiKeyMiddleware` and `httpmw.NoStore`):
- `GET` returns, per user-scoped experiment, the startup default and the
stored rule (mode, condition, revision, actor, time). Stored rules for
other experiments follow, marked `ignored`.
  - `PUT /{experiment}` takes `{mode, condition?, expected_revision}`.
- **Write path:**
- It authorizes read and update on `DeploymentConfig` before parsing
anything. A denied request gets 403 and is not audited.
- It validates next. An invalid condition returns 400 with the full CEL
diagnostics, is not audited, and leaves the rule unchanged.
- It then starts the audit and calls `experiments.WriteRule`. A stale
revision returns 409 and a transaction error returns 500; both are
audited with an empty diff.
- An identical write returns 200, creates no audit entry, and keeps the
revision.
- The audit export happens after commit. If it fails, that is logged and
the committed change stands (the existing audit contract).
- **SDK:** `ExperimentalClient.ExperimentRules` and
`ExperimentalClient.PutExperimentRule`.
- **CLI:** hidden `coder exp experiment-rules list|on|off|set|reset`.
- Write commands take `--expected-revision`. Without it they read the
current revision first, which only protects the gap between the two
calls; the help text says so.
  - A 409 prints the current rule and exits non-zero, with no retry.
- `reset` warns when the startup default is on, because reset is not a
kill switch.
- **Deployment Overview:** a note says the Experiments values are
startup defaults that runtime rules can override per user.
- **Docs:** a new "Target experiments at runtime" section in
`docs/reference/feature-stages.md` covers the modes, roles, CEL fields
and limits, reset vs. off, revisions and audit, the kill-switch
guarantee (including the rolling-upgrade requirement), frontend
freshness, fail-closed behavior, and that rules are not an authorization
boundary.

## Testing

- **HTTP:** every mode applies on the next request with no restart;
rejected writes keep the rule; RBAC (members 403 on GET and PUT;
auditors can read but not write; an invalid body from an unauthorized
caller still gets 403); malformed and ignored stored rows.
- **HA:** two coderd instances share one database. A PUT through A is
seen by B's `/api/v2/experiments` and by B's evaluator, which is the one
chatd uses.
- **Audit (enterprise, licensed):** an effective change creates one
entry whose diff carries the condition, and the entry is found before
the diff is checked. No-ops, 403s and 400s create no entry. A 409 and an
injected 500 are audited with an empty diff. A failed export keeps the
change.
- **Redaction:** a sentinel condition appears only in the rules API,
audit entries, and the server log's audit lines (Enterprise's slog audit
backend writes every audit diff there). It is absent from every other
log line, `/deployment/config`, the support bundle, and telemetry. The
test wires the slog audit backend so this is checked as stated.
- **CLI:** write and list, a conflict that is not retried, and the reset
warning.

Remote dogfood UAT through Coder Agents passed on a licensed test
deployment:
- the CLI;
- per-user results in the browser;
- 403 for members, including an invalid body;
- group targeting, including removing the user from the group;
- sentinel redaction across logs, `/deployment/config`, the support
bundle and the audit API (at that head the audit still masked the
condition; it is now tracked);
- the audit page showing the condition as changed but masked (same
note);
- the Overview note;
- the static default with `off` and `reset`;
- chatd enforcement toggled between turns.

That round tested `135aaaa41e`. The later commits are a CLI refactor and
a test addition, and a follow-up round will cover the final head.

## Follow-ups (low, not in this PR)

- A broken rule logs one error line per evaluation. Consider
rate-limiting it per experiment and revision.
- There's no API or CLI path to clear a stored rule for an experiment
that isn't user-scoped; those rules are inert and listed as `ignored`.
- A malformed stored rule is listed with zero `updated_by` and
`updated_at` values instead of omitting them.
- CLI wording: the 409 message, conflict output showing a UUID instead
of a username, a no-op write not saying nothing changed, and the parent
help lacking the `user.*` fields.
- `PutExperimentRuleRequest.mode` is not in the Swagger `required` list.
An omitted mode already returns 400.
- The dashboard's experiments query (from #29964) refreshes on mount,
window focus or reconnect once 60 seconds old, not on a timer, and its
comment says it bounds staleness. Correct the comment, or add a refetch
interval if the UI must pick up rule changes without a reload.

## Stack

1. #29869: evaluator library
2. #29898: storage and audit types
3. #29964: enforcement and discovery
4. **This PR:** control plane

> [!NOTE]
> This PR was written by an AI agent (Xum).

---
_Generated with [`xum`](https://github.com/coder/xum) • Model:
`anthropic:claude-opus-5-5` • Thinking: `high`_

---------

Signed-off-by: Thomas Kosiewski <tk@coder.com>
2026-09-29 12:23:00 +00:00

332 lines
12 KiB
Go

package coderd_test
import (
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"strings"
"sync/atomic"
"testing"
"github.com/google/uuid"
"github.com/stretchr/testify/require"
"golang.org/x/xerrors"
"cdr.dev/slog/v3"
"github.com/coder/coder/v2/coderd/audit"
"github.com/coder/coder/v2/coderd/coderdtest"
"github.com/coder/coder/v2/coderd/database"
"github.com/coder/coder/v2/coderd/database/dbauthz"
"github.com/coder/coder/v2/coderd/database/dbtestutil"
"github.com/coder/coder/v2/coderd/experiments"
"github.com/coder/coder/v2/codersdk"
entaudit "github.com/coder/coder/v2/enterprise/audit"
"github.com/coder/coder/v2/enterprise/audit/backends"
"github.com/coder/coder/v2/enterprise/coderd/coderdenttest"
"github.com/coder/coder/v2/enterprise/coderd/license"
"github.com/coder/coder/v2/support"
"github.com/coder/coder/v2/testutil"
)
type experimentRulesFixture struct {
db database.Store
owner *codersdk.Client
ownerID uuid.UUID
orgID uuid.UUID
client *codersdk.ExperimentalClient
logs *testutil.FakeSink
failUpsert *atomic.Bool
}
// newExperimentRulesFixture starts a licensed deployment that audits into
// the database, plus extra audit backends built with the server logger,
// and captures server logs.
func newExperimentRulesFixture(t *testing.T, extraBackends ...func(slog.Logger) entaudit.Backend) experimentRulesFixture {
t.Helper()
db, ps := dbtestutil.NewDB(t)
failUpsert := &atomic.Bool{}
logs := testutil.NewFakeSink(t)
logger := logs.Logger()
auditBackends := []entaudit.Backend{backends.NewPostgres(db, true)}
for _, backend := range extraBackends {
auditBackends = append(auditBackends, backend(logger))
}
auditor := entaudit.NewAuditor(db, entaudit.DefaultFilter, auditBackends...)
ownerClient, owner := coderdenttest.New(t, &coderdenttest.Options{
AuditLogging: true,
Options: &coderdtest.Options{
Database: upsertFaultStore{Store: db, fail: failUpsert},
Pubsub: ps,
Auditor: auditor,
Logger: &logger,
},
LicenseOptions: &coderdenttest.LicenseOptions{
Features: license.Features{codersdk.FeatureAuditLog: 1},
},
})
return experimentRulesFixture{
db: db,
owner: ownerClient,
ownerID: owner.UserID,
orgID: owner.OrganizationID,
client: codersdk.NewExperimentalClient(ownerClient),
logs: logs,
failUpsert: failUpsert,
}
}
func (f experimentRulesFixture) auditLogs(ctx context.Context, t *testing.T) []database.AuditLog {
t.Helper()
rows, err := f.db.GetAuditLogsOffset(dbauthz.AsSystemRestricted(ctx), database.GetAuditLogsOffsetParams{
ResourceType: string(database.ResourceTypeExperimentRule),
LimitOpt: 50,
})
require.NoError(t, err)
logs := make([]database.AuditLog, 0, len(rows))
for _, row := range rows {
logs = append(logs, row.AuditLog)
}
return logs
}
// waitAuditLogs waits until n experiment rule audit entries exist. Entries
// are exported after the response is written, so they can trail it.
func (f experimentRulesFixture) waitAuditLogs(ctx context.Context, t *testing.T, n int) []database.AuditLog {
t.Helper()
var logs []database.AuditLog
testutil.Eventually(ctx, t, func(ctx context.Context) bool {
logs = f.auditLogs(ctx, t)
return len(logs) >= n
}, testutil.IntervalFast)
require.Len(t, logs, n)
return logs
}
func (f experimentRulesFixture) put(ctx context.Context, mode codersdk.ExperimentRuleMode, condition string, expected int64) (codersdk.ExperimentRule, error) {
return f.client.PutExperimentRule(ctx, codersdk.ExperimentExample, codersdk.PutExperimentRuleRequest{
Mode: mode, Condition: condition, ExpectedRevision: expected,
})
}
func (f experimentRulesFixture) storedRule(ctx context.Context, t *testing.T) *codersdk.ExperimentRule {
t.Helper()
entries, err := f.client.ExperimentRules(ctx)
require.NoError(t, err)
for _, entry := range entries {
if entry.Experiment == string(codersdk.ExperimentExample) {
return entry.Rule
}
}
t.Fatal("no rules entry for the example experiment")
return nil
}
func requireStatusCode(t *testing.T, err error, status int) {
t.Helper()
var sdkErr *codersdk.Error
require.ErrorAs(t, err, &sdkErr)
require.Equal(t, status, sdkErr.StatusCode(), sdkErr.Error())
}
func TestExperimentRuleAudit(t *testing.T) {
t.Parallel()
t.Run("WritePath", func(t *testing.T) {
t.Parallel()
f := newExperimentRulesFixture(t)
ctx := testutil.Context(t, testutil.WaitLong)
const condition = `user.username == "audited-condition"`
stored, err := f.put(ctx, codersdk.ExperimentRuleModeCondition, condition, 0)
require.NoError(t, err)
// Require the entry before inspecting it, so the diff checks
// cannot pass because nothing was audited.
entry := f.waitAuditLogs(ctx, t, 1)[0]
require.Equal(t, int32(http.StatusOK), entry.StatusCode)
require.Equal(t, database.AuditActionWrite, entry.Action)
require.Equal(t, string(codersdk.ExperimentExample), entry.ResourceTarget)
require.Equal(t, experiments.AuditRecord(codersdk.ExperimentExample, experiments.Rule{}).ID, entry.ResourceID)
require.Equal(t, f.ownerID, entry.UserID)
var diff audit.Map
require.NoError(t, json.Unmarshal(entry.Diff, &diff))
require.Equal(t, audit.OldNew{Old: "", New: condition}, diff["condition"])
require.Equal(t, audit.OldNew{Old: "", New: "condition"}, diff["mode"])
require.Equal(t, audit.OldNew{Old: float64(0), New: float64(1)}, diff["revision"])
// An identical write is a no-op: 200 with the unchanged rule and
// no entry. The conflict below is audited after it, so a stray
// no-op entry would show up in the count.
same, err := f.put(ctx, codersdk.ExperimentRuleModeCondition, condition, 1)
require.NoError(t, err)
require.Equal(t, stored, same)
// Denied and invalid writes are rejected before the audit starts,
// so they add no entry to the count either.
memberClient, _ := coderdtest.CreateAnotherUser(t, f.owner, f.orgID)
_, err = codersdk.NewExperimentalClient(memberClient).PutExperimentRule(ctx, codersdk.ExperimentExample, codersdk.PutExperimentRuleRequest{Mode: codersdk.ExperimentRuleModeOff, ExpectedRevision: 1})
requireStatusCode(t, err, http.StatusForbidden)
_, err = f.put(ctx, codersdk.ExperimentRuleModeCondition, "user.", 1)
requireStatusCode(t, err, http.StatusBadRequest)
_, err = f.put(ctx, codersdk.ExperimentRuleModeOff, "", 0)
requireStatusCode(t, err, http.StatusConflict)
conflict := f.waitAuditLogs(ctx, t, 2)[0]
require.Equal(t, int32(http.StatusConflict), conflict.StatusCode)
require.JSONEq(t, "{}", string(conflict.Diff))
// A database error inside the transaction rolls the write back
// and is audited with its status.
f.failUpsert.Store(true)
_, err = f.put(ctx, codersdk.ExperimentRuleModeOff, "", 1)
requireStatusCode(t, err, http.StatusInternalServerError)
f.failUpsert.Store(false)
failed := f.waitAuditLogs(ctx, t, 3)[0]
require.Equal(t, int32(http.StatusInternalServerError), failed.StatusCode)
require.JSONEq(t, "{}", string(failed.Diff))
require.Equal(t, &stored, f.storedRule(ctx, t))
})
// The audit entry is exported after the rule commits. When the export
// fails, the change stands and the failure is logged.
t.Run("FailedExportKeepsChange", func(t *testing.T) {
t.Parallel()
f := newExperimentRulesFixture(t, func(slog.Logger) entaudit.Backend { return failingAuditBackend{} })
ctx := testutil.Context(t, testutil.WaitLong)
stored, err := f.put(ctx, codersdk.ExperimentRuleModeOn, "", 0)
require.NoError(t, err)
testutil.Eventually(ctx, t, func(context.Context) bool {
return len(f.logs.Entries(func(e slog.SinkEntry) bool {
return e.Level == slog.LevelError && e.Message == "export audit log"
})) > 0
}, testutil.IntervalFast)
require.Equal(t, &stored, f.storedRule(ctx, t))
})
}
// TestExperimentRuleRedaction checks where condition text can appear. The
// rules API, audit entries and the server log's audit lines carry it. No
// other log line, /deployment/config or a support bundle may, even for
// invalid conditions and conditions that fail during evaluation.
func TestExperimentRuleRedaction(t *testing.T) {
t.Parallel()
// Enterprise servers always export audit entries to the server log.
f := newExperimentRulesFixture(t, backends.NewSlog)
memberClient, _ := coderdtest.CreateAnotherUser(t, f.owner, f.orgID)
ctx := testutil.Context(t, testutil.WaitLong)
sentinel := "redaction-sentinel-" + uuid.NewString()
// The author of an invalid condition gets diagnostics quoting it.
_, err := f.put(ctx, codersdk.ExperimentRuleModeCondition, fmt.Sprintf("user.email == %q &&", sentinel), 0)
var sdkErr *codersdk.Error
require.ErrorAs(t, err, &sdkErr)
require.Equal(t, http.StatusBadRequest, sdkErr.StatusCode())
require.Contains(t, sdkErr.Detail, sentinel)
// A valid condition, then one that fails while the member's
// experiments are evaluated.
validCondition := fmt.Sprintf("user.email == %q", sentinel)
failingCondition := fmt.Sprintf("user.groups[7] == %q", sentinel)
_, err = f.put(ctx, codersdk.ExperimentRuleModeCondition, validCondition, 0)
require.NoError(t, err)
_, err = memberClient.Experiments(ctx)
require.NoError(t, err)
_, err = f.put(ctx, codersdk.ExperimentRuleModeCondition, failingCondition, 1)
require.NoError(t, err)
_, err = memberClient.Experiments(ctx)
require.NoError(t, err)
testutil.Eventually(ctx, t, func(context.Context) bool {
return len(f.logs.Entries(func(e slog.SinkEntry) bool {
return e.Message == "experiment condition failed; experiment is off"
})) > 0
}, testutil.IntervalFast)
require.Contains(t, f.storedRule(ctx, t).Condition, sentinel)
// The invalid write is not audited. The newest entry records the
// condition change with both texts.
latest := f.waitAuditLogs(ctx, t, 2)[0]
var diff audit.Map
require.NoError(t, json.Unmarshal(latest.Diff, &diff))
require.Equal(t, audit.OldNew{Old: validCondition, New: failingCondition}, diff["condition"])
res, err := f.owner.Request(ctx, http.MethodGet, "/api/v2/deployment/config", nil)
require.NoError(t, err)
body, err := io.ReadAll(res.Body)
_ = res.Body.Close()
require.NoError(t, err)
require.Equal(t, http.StatusOK, res.StatusCode)
require.NotContains(t, string(body), sentinel)
bundle, err := support.Run(ctx, &support.Deps{Client: f.owner, Log: slog.Make()})
require.NoError(t, err)
bundleJSON, err := json.Marshal(bundle)
require.NoError(t, err)
require.NotContains(t, string(bundleJSON), sentinel)
// Audit lines are exported after the handler returns. Requiring one
// with the condition keeps the check below from passing vacuously.
auditLine := func(e slog.SinkEntry) bool { return e.Message == "audit_log" || e.Message == "export audit log" }
testutil.Eventually(ctx, t, func(context.Context) bool {
for _, entry := range f.logs.Entries(func(e slog.SinkEntry) bool { return e.Message == "audit_log" }) {
if strings.Contains(logText(entry), sentinel) {
return true
}
}
return false
}, testutil.IntervalFast)
for _, entry := range f.logs.Entries(func(e slog.SinkEntry) bool { return !auditLine(e) }) {
require.False(t, strings.Contains(logText(entry), sentinel), "log entry contains condition text: %s", entry.Message)
}
}
// logText renders a log entry's message and field values. Byte slices,
// such as audit diffs, are rendered as text.
func logText(entry slog.SinkEntry) string {
var b strings.Builder
_, _ = b.WriteString(entry.Message)
for _, field := range entry.Fields {
_, _ = b.WriteString(" ")
switch v := field.Value.(type) {
case json.RawMessage:
_, _ = b.Write(v)
case []byte:
_, _ = b.Write(v)
default:
_, _ = fmt.Fprintf(&b, "%+v", v)
}
}
return b.String()
}
// upsertFaultStore fails experiment rule upserts while fail is set.
type upsertFaultStore struct {
database.Store
fail *atomic.Bool
}
func (s upsertFaultStore) InTx(fn func(database.Store) error, opts *database.TxOptions) error {
return s.Store.InTx(func(tx database.Store) error {
return fn(upsertFaultStore{Store: tx, fail: s.fail})
}, opts)
}
func (s upsertFaultStore) UpsertExperimentRule(ctx context.Context, arg database.UpsertExperimentRuleParams) error {
if s.fail.Load() {
return xerrors.New("injected upsert failure")
}
return s.Store.UpsertExperimentRule(ctx, arg)
}
// failingAuditBackend fails every export.
type failingAuditBackend struct{}
func (failingAuditBackend) Decision() entaudit.FilterDecision { return entaudit.FilterDecisionStore }
func (failingAuditBackend) Export(context.Context, database.AuditLog, entaudit.BackendDetails) error {
return xerrors.New("injected audit export failure")
}