Commit Graph
6 Commits
Author SHA1 Message Date
Thomas Kosiewski e307f1d5cf feat: add the user-scoped chat-automations experiment (#32064)
## Why

The Automations RFC controls availability with one user-scoped
experiment. Its text: "The user-scoped `chat-automations` experiment
controls availability. Operators configure defaults with `--experiments`
and can change per-user rules while Coder is running. Each surface
checks the relevant caller or automation's owner, and experiment
evaluation fails closed." The later backend PRs (management routes, the
`manage_automations` tool, `Publish`) need that check first. This PR
registers the experiment and proves the per-user, fail-closed check.
Nothing consumes it yet.

Part of CODAGT-1135

## What changed

- Registers `chat-automations` in `ExperimentsKnown` and
`ExperimentsUserScoped`. An earlier experiment with this name was
removed in #30412, and this PR restores none of its consumers.
- Keeps it out of `ExperimentsSafe`, so `--experiments=*` does not turn
it on. Automations act with their owner's authority, and a test pins the
choice.
- Adds the name to the `userScopedExperimentStaticCheck` ruleguard rule,
so server code cannot decide it from the static list.
`TestExperimentsUserScopedRuleguard` keeps the rule in step with
`ExperimentsUserScoped`.
- Regenerates the swagger files, `docs/reference/api/schemas.md` and
`typesGenerated.ts`.
- Adds tests for the real experiment name. The evaluator tests cover the
startup default, stored `on`, `off` and condition rules for two users,
and an off decision on a rules read error, a malformed rule, an unknown
mode, an attribute read error, a condition error and a nil subject. A
coderd test with a real database checks the same per-user behavior
through `api.ExperimentEvaluator` and `GET /api/v2/experiments`.

## Scope

No routes, tool, UI or chatd change. I added no helper:
`Evaluator.Enabled(ctx, userID, ex)` already decides per user and fails
closed, and API handlers reach it through `api.ExperimentEvaluator`.
chatd has no evaluator handle on main, because #31327 removed the unused
field. The first chatd consumer adds it together with its read.

## Blast radius

With the experiment off, which is the default, behavior on main does not
change. Two things become visible. `GET
/api/experimental/experiments/rules` and `coder exp experiment-rules
list` show a `chat-automations` row, and an administrator can store a
rule for it. A deployment that already passes
`--experiments=chat-automations` sees three small differences: the
startup log reports an opted-in hidden experiment where it reported an
unknown one, the deployment Overview page stops flagging the name as
invalid, and the rules row shows the static default as on. Nothing reads
the experiment yet, so no feature turns on.

## Live check

A local `./scripts/develop.sh` deployment built from this head ran with
no flag, with `--experiments=*` and with
`--experiments=chat-automations`. `GET /api/v2/experiments` lists
`chat-automations` for the owner and for a member only when the startup
list or a rule turns it on, and `--experiments=*` does not turn it on. A
condition rule `user.username == "member"` turned it on for the member
and off for the owner without a restart, and `reset` restored the
startup default.

---
_Generated with [`xum`](https://github.com/coder/xum) • Model:
`anthropic:claude-sonnet-5-5` • Thinking: `xhigh`_
2026-11-11 23:54:05 +01:00
Michael Suchacz 1acacc9c30 feat: graduate MCP tool search from experiments (#31327) 2026-10-29 20:32:56 +01:00
Thomas Kosiewski f923f4a525 chore!: remove the chat automations experiment (#30412)
Removes the experimental Chat Automations feature (`chat-automations` experiment) from main. This reverts the 16-PR stack that added it and adds two forward migrations that undo its schema and credential changes.

> [!WARNING]
> **Upgrade consequences**
>
> - **Automation data is deleted.** Migration 000615 drops the `chat_automations` table and deletes every queued message an automation delivered. Chats, their messages and ordinary queued messages stay. Messages lose their automation provenance (`automation_id`, `input_id`).
> - **Automation-only credentials are revoked.** Migration 000616 deletes API keys, OAuth2 codes and OAuth2 tokens whose only scopes (or allow-list entries) were `chat_automation:*`. Mixed credentials keep their other scopes. Custom roles lose their `chat_automation` permissions. OAuth2 apps that allowed only `chat_automation` scopes get the whitespace allowlist that grants nothing, as in 000595.
> - **A coordinated restart is required.** Stop all old replicas before the new version runs the migrations. Old replicas read the dropped table and columns, so a rolling deploy is not supported.
> - **Remove `--chat-max-automations-per-owner` before upgrading.** The flag, the YAML key `chat.maxAutomationsPerOwner` and the env var `CODER_CHAT_MAX_AUTOMATIONS_PER_OWNER` are removed. A deployment that still sets the flag or the YAML key will not start (`unknown flag` / `unknown option`). The env var is ignored. This includes dogfood. No release or tag contains the flag.
> - **Down migrations restore the schema only.** 000616 down and 000615 down recreate the 000611 schema with empty tables. They do not restore deleted automations, webhook secrets, provenance, queued deliveries, revoked credentials, stripped permissions or the deleted experiment rule.

## What is removed

- The `chat-automations` experiment and its stored runtime rule (`experiment_rule:chat-automations`).
- The `--chat-max-automations-per-owner` deployment option.
- The `chat_automation` RBAC resource, its member and chatd grants, and the `chat_automation:*` API key scopes.
- The automation sqlc queries, dbauthz wrappers and audit table entries.
- The management routes under `/api/experimental/organizations/{organization}/chat-automations`, and the webhook route `POST /api/experimental/chat-automations/{automation}/events` with its rate limiter.
- The chatd management service, webhook publish, schedule loop, the `manage_automations` tool, and the chatstate admission seam and queue promotion guard that only automations used.
- The Agents Automations page, editor, nav entry, chat labels, the Manage automations toggle and the API client.
- The `archived:any` chat search value that the automations "View chats" dialog added.
- `docs/ai-coder/agents/automations.md`, its manifest entry, the glossary entry and the related rows.

Reverted PRs: #30085, #30096, #30112, #30117, #30127, #30134, #30160, #30166, #30167, #30171, #30187, #30207, #30214, #30217, #30219, #30220.

## What stays, and why

- Migrations 000609, 000610 and 000611, their fixture, and the #30297 lock fix with `migration000609_test.go` stay unchanged. Deployments such as dogfood already ran them.
- New migration **000615** reverses the schema changes of 000609 and 000611 against the current schema. It uses the same short-wait lock-and-retry block as 000609, and it rebuilds `chats_expanded` from main's current definition minus the two automation columns. New migration **000616** follows the 000595 Tasks removal for credentials and recreates `api_key_scope` without the five `chat_automation:*` values. It keeps the `chat_project_memory:*` values from 000612.
- The `chat_automation` value stays in the `resource_type` enum, and `codersdk.ResourceTypeChatAutomation` stays. Old audit log rows stay readable, as with `task` after the Tasks removal.
- The 000615 delete fires the existing queue-version trigger, so `queue_version` moves to `snapshot_version` for each affected chat. `snapshot_version` is not bumped: the coordinated restart restarts every stream loop from an empty state, and the site does not read the version fields.
- `coderd/x/chatd/ARCHITECTURE.md` drops the sections the stack added and gets back its pre-stack wording, merged with the later main edits to the same file.

## Notes for reviewers

- #30379 (chat project page) merged while this PR was in progress. It passed `manageAutomationsEnabled` in `AgentCreatePage.test.tsx` and listed an `automations` route in `AgentsPageLayout.test.tsx`. Later main commits also added automation props in `ConversationTimeline.test.tsx` and `ConversationTimelineWorkingBlocks.stories.tsx`. This PR removes all of them.
- Remaining `automation` matches under `coderd`, `codersdk`, `enterprise`, `site/src` and `docs` are intended:
  - migrations 000609 to 000611, their fixture and `migration000609_test.go`;
  - the new 000615 and 000616 migrations and `migration000615_test.go`;
  - the `chat_automation` resource type in `dump.sql`, `models.go`, `codersdk/audit.go`, the generated API docs, `swagger.json` and `typesGenerated.ts`;
  - unrelated uses of the word "automation" (docs, `chatlabels_test.go`, `ratelimit.go`, `autostop.go`, `subagent_catalog.go`, `WorkspaceSharingSection.tsx`, `AgentChatPage.stories.tsx`).

Remote UAT on an isolated dev deployment passed. It upgraded a populated pre-removal database, then checked chats, queued messages, tokens, the 404 routes, the removed page and nav entry, and the experiments list.

---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_

![Agents sidebar after the upgrade, with no Automations entry](https://github.com/user-attachments/assets/bd446035-7fc8-4ca4-a735-3a4f36380738)
2026-10-11 19:57:49 +02:00
McKayla はな 86ebc72236 chore: remove unused autofill-parameters endpoint and experiment (#30525) 2026-10-09 16:42:41 -06:00
Thomas Kosiewski 9316d67440 feat: add chat automations management API behind chat-automations (#30112)
Adds the management API for chat automations under
`/api/experimental/organizations/{organization}/chat-automations`,
behind the new user-scoped `chat-automations` experiment. This is the
first of two PRs for RFC step 3; nothing publishes automation messages
yet.

- **Experiment:** `chat-automations` is in `ExperimentsKnown`,
`ExperimentsUserScoped`, and the ruleguard list in `scripts/rules.go`.
Every route asks `chatd.AutomationsEnabled(ctx, api.ExperimentEvaluator,
callerID)`, which fails closed on a missing subject or evaluator; the
routes stay mounted and return 404 when the experiment is off for the
caller, so a runtime rule for one user applies on the next request
without a restart. There is no `/api/v2` alias.
- **Routes:** list, create (201), get, `PATCH`, delete (204), with
Swagger annotations and audit entries. List returns the automations in
the organization the caller can read (members see their own,
organization admins see all).
- **Service** (`coderd/x/chatd/automations.go`): create validates kind
and target shape, a non-empty prompt, a five-field cron and IANA time
zone, model access, and that an `existing_chat` target is a root chat of
the owner in the same organization. It defaults `when_busy` to `skip`
for schedules and `queue` for webhooks and computes the first schedule
cursor. `PATCH` runs the same per-field validation on each changed
field, rejects `kind` and `target_mode`, bumps `schedule_revision` on
prompt, cron, or time zone changes, and recomputes the cursor on cron or
time zone changes.
- **Owner-only:** only the owner can `PATCH` an automation, even with
organization-admin or site-owner roles (403); admins can list, read, and
delete.
- **Secrets:** a webhook's secret (`coder_automation_` prefix) is
returned once in the create response. Only its SHA-256 is stored, and it
appears in no list, get, audit, or log output.
- **Cap:** new option `--chat-max-automations-per-owner` (default 50)
caps automations per owner across all organizations; creating one more
returns 409. Creates serialize on a per-owner advisory lock.
- **Cron:** `cron.Standard(spec, timeZone)` accepts any standard
five-field expression (the existing `Weekly` rejects day-of-month and
month restrictions). Responses include the next five run times of an
enabled schedule.

PR 3b (stacked on this one) adds disable and re-enable with the queue
cutoff, makes delete run disable first, and adds secret rotation and the
schedule preview route. Until then, the queue promotion guard from
#30096 discards queued rows whose automation was deleted.

Dogfood UAT ran on a real dev instance and passed:
create/list/get/patch/delete via curl, owner-only 403s for an
organization admin and the site owner, the cap (including concurrent
creates and cross-organization counting), a runtime rule turning the
experiment off for one user without a restart, and `/api/v2` 404s.

Part of CODAGT-1209 (PR 3b closes it)
Part of CODAGT-1135
Refs CODAGT-1217

Stack: PR 3 of the Automations for Coder Agents stack, on top of #30096
(which is on top of #30085).

> This PR was written by an AI agent (xum) and reviewed by the agent's
own test audit and UAT. A human has not reviewed it yet.

---
_Generated with [`xum`](https://github.com/coder/xum) • Model:
`anthropic:claude-opus-5-5` • Thinking: `high`_

---------

Signed-off-by: Thomas Kosiewski <tk@coder.com>
2026-10-02 15:54:49 +00:00
Thomas Kosiewski 12be685b59 feat: add experimental API and CLI to manage experiment rules (#29971)
Slice 4 of 4 for runtime experiment rules: an experimental management
API and a hidden CLI, so admins can target, widen, or kill a user-scoped
experiment at runtime without a redeploy. This PR also adds a
startup-default note on the Deployment Overview page and a docs section.

## What's in it

- **API** under `/api/experimental/experiments/rules` (behind
`apiKeyMiddleware` and `httpmw.NoStore`):
- `GET` returns, per user-scoped experiment, the startup default and the
stored rule (mode, condition, revision, actor, time). Stored rules for
other experiments follow, marked `ignored`.
  - `PUT /{experiment}` takes `{mode, condition?, expected_revision}`.
- **Write path:**
- It authorizes read and update on `DeploymentConfig` before parsing
anything. A denied request gets 403 and is not audited.
- It validates next. An invalid condition returns 400 with the full CEL
diagnostics, is not audited, and leaves the rule unchanged.
- It then starts the audit and calls `experiments.WriteRule`. A stale
revision returns 409 and a transaction error returns 500; both are
audited with an empty diff.
- An identical write returns 200, creates no audit entry, and keeps the
revision.
- The audit export happens after commit. If it fails, that is logged and
the committed change stands (the existing audit contract).
- **SDK:** `ExperimentalClient.ExperimentRules` and
`ExperimentalClient.PutExperimentRule`.
- **CLI:** hidden `coder exp experiment-rules list|on|off|set|reset`.
- Write commands take `--expected-revision`. Without it they read the
current revision first, which only protects the gap between the two
calls; the help text says so.
  - A 409 prints the current rule and exits non-zero, with no retry.
- `reset` warns when the startup default is on, because reset is not a
kill switch.
- **Deployment Overview:** a note says the Experiments values are
startup defaults that runtime rules can override per user.
- **Docs:** a new "Target experiments at runtime" section in
`docs/reference/feature-stages.md` covers the modes, roles, CEL fields
and limits, reset vs. off, revisions and audit, the kill-switch
guarantee (including the rolling-upgrade requirement), frontend
freshness, fail-closed behavior, and that rules are not an authorization
boundary.

## Testing

- **HTTP:** every mode applies on the next request with no restart;
rejected writes keep the rule; RBAC (members 403 on GET and PUT;
auditors can read but not write; an invalid body from an unauthorized
caller still gets 403); malformed and ignored stored rows.
- **HA:** two coderd instances share one database. A PUT through A is
seen by B's `/api/v2/experiments` and by B's evaluator, which is the one
chatd uses.
- **Audit (enterprise, licensed):** an effective change creates one
entry whose diff carries the condition, and the entry is found before
the diff is checked. No-ops, 403s and 400s create no entry. A 409 and an
injected 500 are audited with an empty diff. A failed export keeps the
change.
- **Redaction:** a sentinel condition appears only in the rules API,
audit entries, and the server log's audit lines (Enterprise's slog audit
backend writes every audit diff there). It is absent from every other
log line, `/deployment/config`, the support bundle, and telemetry. The
test wires the slog audit backend so this is checked as stated.
- **CLI:** write and list, a conflict that is not retried, and the reset
warning.

Remote dogfood UAT through Coder Agents passed on a licensed test
deployment:
- the CLI;
- per-user results in the browser;
- 403 for members, including an invalid body;
- group targeting, including removing the user from the group;
- sentinel redaction across logs, `/deployment/config`, the support
bundle and the audit API (at that head the audit still masked the
condition; it is now tracked);
- the audit page showing the condition as changed but masked (same
note);
- the Overview note;
- the static default with `off` and `reset`;
- chatd enforcement toggled between turns.

That round tested `135aaaa41e`. The later commits are a CLI refactor and
a test addition, and a follow-up round will cover the final head.

## Follow-ups (low, not in this PR)

- A broken rule logs one error line per evaluation. Consider
rate-limiting it per experiment and revision.
- There's no API or CLI path to clear a stored rule for an experiment
that isn't user-scoped; those rules are inert and listed as `ignored`.
- A malformed stored rule is listed with zero `updated_by` and
`updated_at` values instead of omitting them.
- CLI wording: the 409 message, conflict output showing a UUID instead
of a username, a no-op write not saying nothing changed, and the parent
help lacking the `user.*` fields.
- `PutExperimentRuleRequest.mode` is not in the Swagger `required` list.
An omitted mode already returns 400.
- The dashboard's experiments query (from #29964) refreshes on mount,
window focus or reconnect once 60 seconds old, not on a timer, and its
comment says it bounds staleness. Correct the comment, or add a refetch
interval if the UI must pick up rule changes without a reload.

## Stack

1. #29869: evaluator library
2. #29898: storage and audit types
3. #29964: enforcement and discovery
4. **This PR:** control plane

> [!NOTE]
> This PR was written by an AI agent (Xum).

---
_Generated with [`xum`](https://github.com/coder/xum) • Model:
`anthropic:claude-opus-5-5` • Thinking: `high`_

---------

Signed-off-by: Thomas Kosiewski <tk@coder.com>
2026-09-29 12:23:00 +00:00