Removes the experimental Chat Automations feature (`chat-automations` experiment) from main. This reverts the 16-PR stack that added it and adds two forward migrations that undo its schema and credential changes.
> [!WARNING]
> **Upgrade consequences**
>
> - **Automation data is deleted.** Migration 000615 drops the `chat_automations` table and deletes every queued message an automation delivered. Chats, their messages and ordinary queued messages stay. Messages lose their automation provenance (`automation_id`, `input_id`).
> - **Automation-only credentials are revoked.** Migration 000616 deletes API keys, OAuth2 codes and OAuth2 tokens whose only scopes (or allow-list entries) were `chat_automation:*`. Mixed credentials keep their other scopes. Custom roles lose their `chat_automation` permissions. OAuth2 apps that allowed only `chat_automation` scopes get the whitespace allowlist that grants nothing, as in 000595.
> - **A coordinated restart is required.** Stop all old replicas before the new version runs the migrations. Old replicas read the dropped table and columns, so a rolling deploy is not supported.
> - **Remove `--chat-max-automations-per-owner` before upgrading.** The flag, the YAML key `chat.maxAutomationsPerOwner` and the env var `CODER_CHAT_MAX_AUTOMATIONS_PER_OWNER` are removed. A deployment that still sets the flag or the YAML key will not start (`unknown flag` / `unknown option`). The env var is ignored. This includes dogfood. No release or tag contains the flag.
> - **Down migrations restore the schema only.** 000616 down and 000615 down recreate the 000611 schema with empty tables. They do not restore deleted automations, webhook secrets, provenance, queued deliveries, revoked credentials, stripped permissions or the deleted experiment rule.
## What is removed
- The `chat-automations` experiment and its stored runtime rule (`experiment_rule:chat-automations`).
- The `--chat-max-automations-per-owner` deployment option.
- The `chat_automation` RBAC resource, its member and chatd grants, and the `chat_automation:*` API key scopes.
- The automation sqlc queries, dbauthz wrappers and audit table entries.
- The management routes under `/api/experimental/organizations/{organization}/chat-automations`, and the webhook route `POST /api/experimental/chat-automations/{automation}/events` with its rate limiter.
- The chatd management service, webhook publish, schedule loop, the `manage_automations` tool, and the chatstate admission seam and queue promotion guard that only automations used.
- The Agents Automations page, editor, nav entry, chat labels, the Manage automations toggle and the API client.
- The `archived:any` chat search value that the automations "View chats" dialog added.
- `docs/ai-coder/agents/automations.md`, its manifest entry, the glossary entry and the related rows.
Reverted PRs: #30085, #30096, #30112, #30117, #30127, #30134, #30160, #30166, #30167, #30171, #30187, #30207, #30214, #30217, #30219, #30220.
## What stays, and why
- Migrations 000609, 000610 and 000611, their fixture, and the #30297 lock fix with `migration000609_test.go` stay unchanged. Deployments such as dogfood already ran them.
- New migration **000615** reverses the schema changes of 000609 and 000611 against the current schema. It uses the same short-wait lock-and-retry block as 000609, and it rebuilds `chats_expanded` from main's current definition minus the two automation columns. New migration **000616** follows the 000595 Tasks removal for credentials and recreates `api_key_scope` without the five `chat_automation:*` values. It keeps the `chat_project_memory:*` values from 000612.
- The `chat_automation` value stays in the `resource_type` enum, and `codersdk.ResourceTypeChatAutomation` stays. Old audit log rows stay readable, as with `task` after the Tasks removal.
- The 000615 delete fires the existing queue-version trigger, so `queue_version` moves to `snapshot_version` for each affected chat. `snapshot_version` is not bumped: the coordinated restart restarts every stream loop from an empty state, and the site does not read the version fields.
- `coderd/x/chatd/ARCHITECTURE.md` drops the sections the stack added and gets back its pre-stack wording, merged with the later main edits to the same file.
## Notes for reviewers
- #30379 (chat project page) merged while this PR was in progress. It passed `manageAutomationsEnabled` in `AgentCreatePage.test.tsx` and listed an `automations` route in `AgentsPageLayout.test.tsx`. Later main commits also added automation props in `ConversationTimeline.test.tsx` and `ConversationTimelineWorkingBlocks.stories.tsx`. This PR removes all of them.
- Remaining `automation` matches under `coderd`, `codersdk`, `enterprise`, `site/src` and `docs` are intended:
- migrations 000609 to 000611, their fixture and `migration000609_test.go`;
- the new 000615 and 000616 migrations and `migration000615_test.go`;
- the `chat_automation` resource type in `dump.sql`, `models.go`, `codersdk/audit.go`, the generated API docs, `swagger.json` and `typesGenerated.ts`;
- unrelated uses of the word "automation" (docs, `chatlabels_test.go`, `ratelimit.go`, `autostop.go`, `subagent_catalog.go`, `WorkspaceSharingSection.tsx`, `AgentChatPage.stories.tsx`).
Remote UAT on an isolated dev deployment passed. It upgraded a populated pre-removal database, then checked chats, queued messages, tokens, the 404 routes, the removed page and nav entry, and the experiments list.
---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_

345 lines
9.9 KiB
Go
345 lines
9.9 KiB
Go
package chatd
|
|
|
|
import (
|
|
"context"
|
|
"database/sql"
|
|
"sync"
|
|
"sync/atomic"
|
|
"time"
|
|
|
|
"github.com/google/uuid"
|
|
"github.com/prometheus/client_golang/prometheus"
|
|
"golang.org/x/xerrors"
|
|
|
|
"cdr.dev/slog/v3"
|
|
"github.com/coder/coder/v2/coderd/audit"
|
|
"github.com/coder/coder/v2/coderd/database"
|
|
dbpubsub "github.com/coder/coder/v2/coderd/database/pubsub"
|
|
"github.com/coder/coder/v2/coderd/notifications"
|
|
"github.com/coder/coder/v2/coderd/x/chatd/messagepartbuffer"
|
|
"github.com/coder/quartz"
|
|
)
|
|
|
|
const (
|
|
defaultAcquisitionBatchSize = int32(10)
|
|
defaultCapacityMetricsInterval = 30 * time.Second
|
|
defaultRunnerSyncInterval = 15 * time.Second
|
|
defaultHeartbeatCleanupEvery = 30 * time.Second
|
|
// The archive cutoff is based on UTC start-of-day and only moves
|
|
// once per day, so hourly runs are more than enough to keep up
|
|
// while still catching chats that cross the threshold shortly
|
|
// after midnight.
|
|
defaultArchiveInterval = time.Hour
|
|
defaultArchiveBatchSize = int32(1000)
|
|
defaultStateChannelSize = 64
|
|
defaultTaskRetryInitialBackoff = 100 * time.Millisecond
|
|
defaultTaskRetryMaxBackoff = 5 * time.Second
|
|
)
|
|
|
|
// chatWorkerPubsub is the chat worker pubsub dependency.
|
|
type chatWorkerPubsub interface {
|
|
Publish(event string, message []byte) error
|
|
SubscribeWithErr(event string, listener dbpubsub.ListenerWithErr) (func(), error)
|
|
}
|
|
|
|
// chatWorkerTaskStarter starts runner-owned side-effect tasks.
|
|
type chatWorkerTaskStarter interface {
|
|
StartGeneration(context.Context, chatWorkerTaskStartInput) error
|
|
StartInterrupt(context.Context, chatWorkerTaskStartInput) error
|
|
StartRequiresActionTimeout(context.Context, chatWorkerTaskStartInput) error
|
|
StartAbandon(context.Context, chatWorkerTaskStartInput) error
|
|
}
|
|
|
|
// chatWorkerTaskStartInput describes one runner task invocation.
|
|
type chatWorkerTaskStartInput struct {
|
|
TaskID uuid.UUID
|
|
ChatID uuid.UUID
|
|
// TurnID is a process-local correlation ID minted per generation
|
|
// task run. It groups the run's hook events; it is best-effort only
|
|
// and never persisted.
|
|
TurnID uuid.UUID
|
|
WorkerID uuid.UUID
|
|
RunnerID uuid.UUID
|
|
HistoryVersion int64
|
|
GenerationAttempt int64
|
|
Status database.ChatStatus
|
|
RequiresActionDeadlineAt sql.NullTime
|
|
DebugTurn *runnerDebugTurn
|
|
TurnSpan *runnerTurnSpan
|
|
TurnToken turnToken
|
|
SessionStart *sessionStartTracker
|
|
StopNudges *stopNudgeTracker
|
|
TurnExperiments *turnExperimentDecisions
|
|
}
|
|
|
|
func (i chatWorkerTaskStartInput) hookTurnID() *uuid.UUID {
|
|
if i.TurnID == uuid.Nil {
|
|
return nil
|
|
}
|
|
turnID := i.TurnID
|
|
return &turnID
|
|
}
|
|
|
|
// stopNudgeTracker allows at most one stop-hook nudge continuation per
|
|
// turn. Turns are keyed by the last user prompt's message ID so the
|
|
// claim survives task restarts, which mint fresh process-local turn
|
|
// IDs.
|
|
type stopNudgeTracker struct {
|
|
mu sync.Mutex
|
|
turnKey int64
|
|
claimed bool
|
|
pending bool
|
|
}
|
|
|
|
// stopNudgeKey identifies the current turn by its prompt row. Model
|
|
// visibility user rows are hook context, not prompts.
|
|
func stopNudgeKey(messages []database.ChatMessage) int64 {
|
|
index := lastUserPromptIndex(messages)
|
|
if index == -1 {
|
|
return 0
|
|
}
|
|
return messages[index].ID
|
|
}
|
|
|
|
func (t *stopNudgeTracker) claim(turnKey int64) bool {
|
|
t.mu.Lock()
|
|
defer t.mu.Unlock()
|
|
if t.turnKey != turnKey {
|
|
t.turnKey = turnKey
|
|
t.claimed = false
|
|
}
|
|
if t.claimed {
|
|
return false
|
|
}
|
|
t.claimed = true
|
|
t.pending = true
|
|
return true
|
|
}
|
|
|
|
func (t *stopNudgeTracker) consume(turnKey int64) bool {
|
|
t.mu.Lock()
|
|
defer t.mu.Unlock()
|
|
if t.turnKey != turnKey || !t.pending {
|
|
return false
|
|
}
|
|
t.pending = false
|
|
return true
|
|
}
|
|
|
|
func (t *stopNudgeTracker) cancel(turnKey int64) {
|
|
t.mu.Lock()
|
|
defer t.mu.Unlock()
|
|
if t.turnKey != turnKey || !t.pending {
|
|
return
|
|
}
|
|
t.pending = false
|
|
t.claimed = false
|
|
}
|
|
|
|
func (t *stopNudgeTracker) reset() {
|
|
t.mu.Lock()
|
|
t.turnKey = 0
|
|
t.claimed = false
|
|
t.pending = false
|
|
t.mu.Unlock()
|
|
}
|
|
|
|
// turnExperimentDecisions keeps user-scoped experiment decisions for
|
|
// one turn, so every step of the turn uses the decision made when the
|
|
// turn first prepared and a rule change applies from the next turn.
|
|
// Steps run as separate tasks, so the decisions live on the runner and
|
|
// are keyed by the turn's prompt row like stopNudgeTracker. A runner
|
|
// restart evaluates again.
|
|
type turnExperimentDecisions struct {
|
|
mu sync.Mutex
|
|
turnKey int64
|
|
decided bool
|
|
mcpToolSearch bool
|
|
}
|
|
|
|
// mcpToolSearchEnabled returns the turn's mcp-tool-search decision, calling
|
|
// evaluate only when the turn has not decided yet.
|
|
func (t *turnExperimentDecisions) mcpToolSearchEnabled(turnKey int64, evaluate func() bool) bool {
|
|
if turnKey == 0 {
|
|
// Without a prompt row there is no turn identity to key on, and
|
|
// caching under 0 would pin one decision across such turns.
|
|
return evaluate()
|
|
}
|
|
t.mu.Lock()
|
|
if t.decided && t.turnKey == turnKey {
|
|
enabled := t.mcpToolSearch
|
|
t.mu.Unlock()
|
|
return enabled
|
|
}
|
|
t.mu.Unlock()
|
|
|
|
// Evaluate outside the lock because it reads the database. If
|
|
// another task of the same turn decided meanwhile, its decision wins.
|
|
enabled := evaluate()
|
|
t.mu.Lock()
|
|
defer t.mu.Unlock()
|
|
if t.decided && t.turnKey == turnKey {
|
|
return t.mcpToolSearch
|
|
}
|
|
if t.decided && t.turnKey > turnKey {
|
|
// A canceled task of an older turn finished late. Prompt row IDs
|
|
// increase, so keep the newer turn's decision.
|
|
return enabled
|
|
}
|
|
t.turnKey = turnKey
|
|
t.decided = true
|
|
t.mcpToolSearch = enabled
|
|
return enabled
|
|
}
|
|
|
|
type sessionStartTracker struct {
|
|
mu sync.Mutex
|
|
completed bool
|
|
inFlight chan struct{}
|
|
}
|
|
|
|
func (t *sessionStartTracker) claim(ctx context.Context) (bool, func(bool), error) {
|
|
for {
|
|
t.mu.Lock()
|
|
if t.completed {
|
|
t.mu.Unlock()
|
|
return false, nil, nil
|
|
}
|
|
if t.inFlight == nil {
|
|
t.inFlight = make(chan struct{})
|
|
t.mu.Unlock()
|
|
return true, func(completed bool) {
|
|
t.mu.Lock()
|
|
t.completed = completed
|
|
close(t.inFlight)
|
|
t.inFlight = nil
|
|
t.mu.Unlock()
|
|
}, nil
|
|
}
|
|
inFlight := t.inFlight
|
|
t.mu.Unlock()
|
|
select {
|
|
case <-inFlight:
|
|
case <-ctx.Done():
|
|
return false, nil, ctx.Err()
|
|
}
|
|
}
|
|
}
|
|
|
|
// chatWorkerOptions configures a chatWorker.
|
|
type chatWorkerOptions struct {
|
|
WorkerID uuid.UUID
|
|
|
|
Store database.Store
|
|
Pubsub chatWorkerPubsub
|
|
Logger slog.Logger
|
|
Clock quartz.Clock
|
|
TaskStarter chatWorkerTaskStarter
|
|
MessagePartBuffer *messagepartbuffer.Buffer
|
|
|
|
NotificationsEnqueuer notifications.Enqueuer
|
|
Auditor *atomic.Pointer[audit.Auditor]
|
|
AutoArchiveRecords prometheus.Counter
|
|
|
|
AgentCapacityLimiter AgentCapacityLimiter
|
|
CapacityMetrics *capacityMetrics
|
|
|
|
AcquisitionInterval time.Duration
|
|
CapacityMetricsInterval time.Duration
|
|
AcquisitionBatchSize int32
|
|
ArchiveInterval time.Duration
|
|
ArchiveBatchSize int32
|
|
RunnerSyncInterval time.Duration
|
|
HeartbeatInterval time.Duration
|
|
HeartbeatCleanupInterval time.Duration
|
|
HeartbeatStaleSeconds int32
|
|
// HeartbeatRenewalTimeout bounds one heartbeat renewal tick. It must
|
|
// stay well below the stale threshold so a slow tick fails and
|
|
// retries instead of letting every lease go stale.
|
|
HeartbeatRenewalTimeout time.Duration
|
|
// HeartbeatLockTimeout bounds the renewal's wait for the capacity
|
|
// admission lock.
|
|
HeartbeatLockTimeout time.Duration
|
|
StateChannelSize int
|
|
RunnerManagerChannelSize int
|
|
AcquisitionWakeChannelSize int
|
|
TaskRetryInitialBackoff time.Duration
|
|
TaskRetryMaxBackoff time.Duration
|
|
}
|
|
|
|
func (o chatWorkerOptions) withDefaults() (chatWorkerOptions, error) {
|
|
if o.Store == nil {
|
|
return chatWorkerOptions{}, xerrors.New("chatworker: store is required")
|
|
}
|
|
if o.Pubsub == nil {
|
|
return chatWorkerOptions{}, xerrors.New("chatworker: pubsub is required")
|
|
}
|
|
if o.TaskStarter == nil && o.MessagePartBuffer == nil {
|
|
return chatWorkerOptions{}, xerrors.New("chatworker: task starter or message part buffer is required")
|
|
}
|
|
if o.WorkerID == uuid.Nil {
|
|
return chatWorkerOptions{}, xerrors.New("chatworker: worker ID is required")
|
|
}
|
|
if o.Clock == nil {
|
|
o.Clock = quartz.NewReal()
|
|
}
|
|
if o.AcquisitionInterval <= 0 {
|
|
o.AcquisitionInterval = DefaultPendingChatAcquireInterval
|
|
}
|
|
if o.CapacityMetricsInterval <= 0 {
|
|
o.CapacityMetricsInterval = defaultCapacityMetricsInterval
|
|
}
|
|
if o.AcquisitionBatchSize <= 0 {
|
|
o.AcquisitionBatchSize = defaultAcquisitionBatchSize
|
|
}
|
|
if o.ArchiveInterval <= 0 {
|
|
o.ArchiveInterval = defaultArchiveInterval
|
|
}
|
|
if o.ArchiveBatchSize <= 0 {
|
|
o.ArchiveBatchSize = defaultArchiveBatchSize
|
|
}
|
|
if o.NotificationsEnqueuer == nil {
|
|
o.NotificationsEnqueuer = notifications.NewNoopEnqueuer()
|
|
}
|
|
if o.RunnerSyncInterval <= 0 {
|
|
o.RunnerSyncInterval = defaultRunnerSyncInterval
|
|
}
|
|
if o.HeartbeatInterval <= 0 {
|
|
o.HeartbeatInterval = DefaultChatHeartbeatInterval
|
|
}
|
|
if o.HeartbeatCleanupInterval <= 0 {
|
|
o.HeartbeatCleanupInterval = defaultHeartbeatCleanupEvery
|
|
}
|
|
if o.HeartbeatStaleSeconds <= 0 {
|
|
o.HeartbeatStaleSeconds = int32(DefaultInFlightChatStaleAfter / time.Second)
|
|
}
|
|
if o.HeartbeatRenewalTimeout <= 0 {
|
|
o.HeartbeatRenewalTimeout = time.Duration(o.HeartbeatStaleSeconds) * time.Second / 3
|
|
}
|
|
if o.HeartbeatLockTimeout <= 0 {
|
|
o.HeartbeatLockTimeout = o.HeartbeatRenewalTimeout / 2
|
|
}
|
|
if o.AgentCapacityLimiter == nil {
|
|
o.AgentCapacityLimiter = newAgentCapacityLimiter(nil, o.HeartbeatStaleSeconds)
|
|
}
|
|
if o.StateChannelSize <= 0 {
|
|
o.StateChannelSize = defaultStateChannelSize
|
|
}
|
|
if o.RunnerManagerChannelSize <= 0 {
|
|
o.RunnerManagerChannelSize = defaultStateChannelSize
|
|
}
|
|
if o.AcquisitionWakeChannelSize <= 0 {
|
|
o.AcquisitionWakeChannelSize = 1
|
|
}
|
|
if o.TaskRetryInitialBackoff <= 0 {
|
|
o.TaskRetryInitialBackoff = defaultTaskRetryInitialBackoff
|
|
}
|
|
if o.TaskRetryMaxBackoff <= 0 {
|
|
o.TaskRetryMaxBackoff = defaultTaskRetryMaxBackoff
|
|
}
|
|
if o.TaskRetryMaxBackoff < o.TaskRetryInitialBackoff {
|
|
o.TaskRetryMaxBackoff = o.TaskRetryInitialBackoff
|
|
}
|
|
return o, nil
|
|
}
|