mirror of
https://github.com/coder/coder.git
synced 2026-11-22 19:39:23 +08:00
feea5ab959f59cd2135443ed2a0c19bf7643db8f
5
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
78fa9a8613 |
refactor(coderd/x/chatd)!: retire the close_agent tool alias (#31005)
This PR breaks compatibility with chat histories that contain `close_agent` tool calls, which chats from v2.34 and earlier have. #26673 (v2.35) renamed `close_agent` to `interrupt_agent` and kept a hidden alias, so a model that copied the old name from such a history still interrupted the child. This PR deletes the alias: `subagentToolNameAliases`, `ExecuteLocalToolsOptions.ToolNameAliases` with `(*toolBatch).resolve` and its five call sites in `chatloop`, the alias step in builtin input validation, and the `close_agent` entries in the unbilled tool set, the plan policy, and the chatprompt lifecycle list. Coder Agents is experimental (`coderd/x`), and our rule for experimental persisted data is to drop the legacy shape outright and delete the code that existed only to support it, instead of carrying it forward. What happens to an old chat: - Replay does not change, because the replay path does not look up tool names: the model still sees the historical `close_agent` call and its stored result, and the UI renders them as before. The write path reads a tool name only to parse `execute` commands and to keep structured error results from subagent lifecycle tools. Dropping `close_agent` from that lifecycle list changes nothing, because its calls are now rejected with a plain-text error. - If the model calls `close_agent` again, nothing runs and the child keeps running. The tool result is the error `Tool not active in this turn: close_agent`, and the model can retry with `interrupt_agent`. **Why it is safe** - Deleted alias-only tests: `TestExecuteSingleTool_ResolvesToolNameAlias`, the `AliasNamesClassifyAsCalled` billing row, the alias subtest of `TestPartitionAmbiguousToolCallsGatesOnBuiltins`, the alias loop in `TestUnbilledSubagentToolNamesMatchCatalog`, and the aliased calls that #30939 added to `TestExecuteToolsSerialToolCallOrder` and `TestExecuteLocalTools_ProviderToolRunner`; both tests are back to their #30938 form. - `TestExecuteSingleTool_UnknownAliasFallsThrough` is now `TestExecuteSingleTool_RejectsUnknownTool` and asserts the exact error above. The observer tests call the tool by its real name. - Mutants of the active tool check, step observer notification, serial dispatch, the unbilled set, the plan policy and builtin input validation each fail a remaining test. Only the deleted alias test caught an empty name passed to the tool handler; no chatd tool reads that name (MCP tools use their original name). - `git grep` finds no use of the removed identifiers and no backend code that resolves `close_agent`. The rejection test still uses that name as its input. The frontend keeps its `close_agent` display mapping, which old histories still use. **Review focus** - A rejected `close_agent` call is now billed like any rejected tool call; the rejection is immediate. - `ARCHITECTURE.md`, "Generation goroutine": this PR deletes the sub-bullet that called `close_agent` a deprecated alias of `interrupt_agent`. - The `resolved_tool_name` debug log field is gone. It differed from `tool_name` only for aliased calls, and the alias is gone. **Validation** - `go test -count=1 ./coderd/x/chatd/ ./coderd/x/chatd/chatloop/ ./coderd/x/chatd/chatprompt/`: ok - `go vet` and `golangci-lint run --fix=false ./coderd/x/chatd/...`: clean - `paralleltestctx -custom-funcs="testutil.Context,chatdTestContext" ./coderd/x/chatd/...`: clean - pre-commit hook: passed **Stack** This stack retires chatd compatibility paths for chat data that older Coder versions stored. This PR replaces #31003 and carries its change, adapted to the `toolBatch` code that #30938 and #30939 merged. Mike chose to keep the fallbacks that #31001 and #31002 removed, so those two layers and #31003 left the stack. 1. #30996 stamp chat message fixtures with the current content version 2. #30997 drop tests of the legacy content codec 3. #30998 retire the V0 chat message content decoder 4. #31000 inline the model config provider identity 5. #31005 retire the `close_agent` tool alias (this PR) > Xum, an AI agent, opened this PR for @ibetitsmike and wrote this description. |
||
|
|
e35cbe2a7f |
fix(coderd/x/chatd): stream compaction summaries (#28886)
Chats could get permanently stuck once their context crossed the compaction threshold on adaptive-thinking Claude models: every compaction attempt failed with `compaction produced an empty summary`, and both automatic and manual compaction take the same path, so the chat could never recover. ## Problem `compactionSummaryCall` deliberately set `MaxOutputTokens = nil` because the summary request was non-streaming and the Anthropic SDK rejects non-streaming requests whose `max_tokens` implies a completion longer than ten minutes. But a nil cap makes fantasy's Anthropic provider silently default to 4096 output tokens, and adaptive-thinking Claude models can spend that entire budget on a thinking block, leaving zero summary text. ## Fix - Stream the compaction summary request. Streaming lifts the Anthropic SDK's non-streaming duration restriction, which was the only reason the cap was nil'd. - Keep the resolved output cap (32k default) on the summary call instead of nil'ing it. - When the summary comes back empty with truncation evidence (length finish reason, or reasoning with no text), fail with `compaction summary was truncated at the output token cap` instead of the generic empty-summary error, so the next occurrence is attributable at a glance. Unit tests cover text-block joining and the truncation classification; the compaction e2e fixtures now serve streamed summaries and assert the summary request streams with `max_tokens: 32000`. Dogfood UAT on a deployment built from this branch verified manual compaction with Claude Opus 5 (adaptive thinking, twice, with reload persistence) and an OpenAI regression chat. > 🤖 Opened by [Xum](https://mux.coder.com) on behalf of @ibetitsmike. |
||
|
|
26e4429362 |
test: fold near-identical test bodies into tables across eight packages (#29463)
Folds groups of near-identical test bodies (same parent, same parallel flag, differing only in inputs and expected values) into table-driven tests in eight packages: `coderd/templatebuilder`, `scripts/develop`, `agent/agentfiles`, `coderd/x/chatd/chatloop`, `coderd/x/chatd/chatdebug`, `enterprise/aibridgeproxyd`, `cli`, and `coderd/database/db2sdk`. One commit per package. Every case keeps its previous subtest name, `t.Parallel()` placement is unchanged, and the set of `=== RUN` names is identical except where standalone `TestX_Y` funcs became `TestX/Y` subtests (chatloop: 9 funcs into `TestExecuteLocalTools_BatchRuntime`; chatdebug: 15 funcs into two tables). Members whose bodies differ in more than data stayed as they were (listed per package in the commit messages). In `agent/agentfiles` the four `TestFuzzyReplace_*` funcs were already tables, so the shared request plumbing was hoisted into one helper instead of merging the funcs and losing their per-rule docs. Net -1035 lines across 9 test files; runtime is unchanged (all sub-100ms unit tests). | Package | Diff | Funcs removed | Subtests | RUN lines | |---|---|---|---|---| | coderd/templatebuilder | +237 / -291 | 0 | 31 -> 31 | 33 -> 33 | | scripts/develop | +194 / -246 | 0 | 27 -> 27 | 28 -> 28 | | agent/agentfiles | +117 / -260 | 0 | unchanged | 58 -> 58 | | coderd/x/chatd/chatloop | +224 / -306 | 9 | 3 -> 12 | 14 -> 15 | | coderd/x/chatd/chatdebug | +251 / -537 | 13 | 0 -> 15 | 42 -> 44 | | enterprise/aibridgeproxyd | +287 / -419 | 0 | 31 -> 31 | 32 -> 32 | | cli | +143 / -326 | 0 | 15 -> 15 | 16 -> 16 | | coderd/database/db2sdk | +130 / -233 | 0 | 9 -> 9 | 10 -> 10 | ## Validation - Per package: `go test -count=1 -tags=testsmallbatch -parallel=8 ./<pkg>/ -run <regex> -v` before and after, with the `=== RUN` name sets diffed - `go test -race -count=2 -shuffle=on` on each converted test: pass - `golangci-lint run ./<pkg>/...` with a fresh cache: 0 issues in all eight packages - Repository pre-commit hook (including lint/go) passed on every commit Part of the test-suite speedup series (see #29457); independent of the other PRs. ## Remote validation Remote run on a dogfood workspace (separate compute), chat https://dogfood.cdr.dev/agents/5bbe73d0-866a-4841-9cc3-c5e5b4d97e44, tested head `d7c55d6b0e8` against base `9d973372774`: all eight packages pass on the head and under `-race -count=3 -shuffle=on`; package times are sub-second on both sides (unchanged). `=== RUN` name sets: chatloop 15 to 16 lines with all nine `TestExecuteLocalTools_<X>` leaves surviving as `TestExecuteLocalTools_BatchRuntime/<X>`; chatdebug 42 to 44; `aibridgeproxyd` `TestNew` 32 to 32, byte-identical sorted list. One correction to the text above: seven chatdebug leaves are renamed, not preserved verbatim (`TestRecordingTransport_*UnknownLength*` became `TestRecordingTransport_UnknownLengthClose/<name without "UnknownLength">`), so `-run` filters keyed on those old names need updating. Endorsed verdict: PASS. > Xum (an AI agent) authored this PR on behalf of @ibetitsmike. |
||
|
|
b428c8feb6 |
feat(coderd): bill completed local tool batches in agent runtime (#28360)
**Stack**: #28359 → **#28360 (this PR)** → #28361 → #28362 ## Summary Completed local tool batches now count toward Coder Agent runtime (`chat_messages.runtime_ms`, the source of truth for `hb_agent_runtime_v1`), excluding sub-agent orchestration tools to avoid double counting, and without multiplying runtime for parallel tool calls. Second of a four-PR stack splitting #28211. Stacked on #28359. Refs CODAGT-928 (https://linear.app/codercom/issue/CODAGT-928/track-local-tool-execution-for-agent-runtime). ## Problem Agent runtime previously measured only model invocation wall clock (stream open to fully consumed). Time spent executing local tools between steps, including file operations, terminal commands, workspace provisioning, and MCP tools, was deliberately excluded, which undercounts the product definition of "actively processing a task". Naive inclusion has two hazards: a batch of parallel tool calls would bill N windows for one wall-clock wait, and `wait_agent` would re-bill child agents that already bill their own model and tool time. ## Fix Each completed local tool batch bills one window: the union of the billed tools' execution intervals, persisted as `runtime_ms` on a dedicated usage record appended after the batch's tool-result rows. The record is a tool-role message with `visibility='model'`, so it never reaches the API, SSE, or clients, and prompt replay drops it because its single internal `tool-batch-usage` part converts to no provider content. Its content carries an audit payload (`billed_ms`, `billed_calls`) so the billed window is inspectable after the fact; real tool-result rows never carry batch-level runtime. Concurrent calls all start at batch start, so 5 parallel 10s reads bill 10s, not 50s; serial calls (`SerialToolCalls`) count only from their own launch, so unbilled waits before them do not count. Sub-agent orchestration tools (`spawn_agent`, `wait_agent`, `message_agent`, `interrupt_agent`, `list_agents`, `list_subagent_models`, plus the deprecated `close_agent` alias) never extend the window: every chat, including children, applies the same rules to its own runtime, so a parent's `wait_agent` window would double count. A lone `wait_agent` bills 0 and appends no usage record; `execute` 10s in parallel with `wait_agent` 60s bills 10s. A test pins the unbilled set to the registered sub-agent tool catalog so they cannot drift. `GetTotalChatMessageRuntimeMsInRange` already sums `runtime_ms` role-agnostically, so the usage record is picked up with no schema, query, or cron changes. Client-executed dynamic tools, external agents, parked/idle time, and retry backoff remain unbilled. Interrupted batches still bill only the model window; the rest of the stack adds partial-window billing via the same usage record. `ExecuteLocalToolsOptions` also gains an optional `ToolBillingRecorder` (`RecordStart`/`RecordComplete`) observing per-occurrence dispatch-order execution stamps. This PR wires no recorder; the next PR in the stack connects it to the message part buffer for interrupt billing. **NOTE**: Reported agent runtime (`hb_agent_runtime_v1`) increases from deploy forward. There is no backfill and no feature flag. |
||
|
|
4b7494be72 |
feat: harden chat generation runtime instrumentation for billing (#27451)
Closes CODAGT-835 ## Summary `chat_messages.runtime_ms` becomes the billing source of truth for Coder Agents runtime (summed hourly by #27312), but it was built for debugging: the June refactor (#26270) silently stopped recording tool-step runtime, compaction was never measured, and interrupted turns lost their partial runtime entirely. This PR defines the billable metric, closes the paths that dropped it, and documents the definition where the data lives. ## The billable definition **`runtime_ms` is the wall-clock duration of the model invocation that produced the persisted message content**, measured from just before the provider stream opens until it is fully consumed. What counts: - Assistant generation steps, in top-level and sub-agent chats (sub-agents are ordinary chats on the same generation path). - Compaction summarization calls, persisted on the compaction assistant message (**new**). - Interrupted attempts: the message-part episode's lifetime is persisted on the partial assistant message committed by `FinishInterruption`, so partial generation time survives interruption (**new**; measured via a new `Buffer.EpisodeDuration`, which works even though the generation goroutine and the interrupt task are different tasks). What deliberately does not count (each is documented in code and docs): - **Local tool execution.** Tool wall time includes idle waits, most importantly `wait_agent` polling a sub-agent chat that already bills its own model invocations; billing the batch would double count, and excluding one tool from a concurrent batch's wall time is ill-defined. Pre-refactor instrumentation did include tool time; this makes the exclusion an explicit product definition instead of a silent regression. - **Failed model calls whose output is discarded** (retried attempts, terminal errors, content-filter refusals). They persist no content, so they bill nothing; billing errs toward undercounting. Notably a stream-silence timeout can burn 10 idle minutes before a retry, which should not be billable "active generation". If product later wants failed attempts billed, that needs a place to persist runtime on error turns (`FinishError` inserts no rows today) and is a deliberate follow-up, not instrumentation drift. - **Ancillary calls that produce no chat messages** (title generation, advisor, turn summaries) and all idle/parked time (`requires_action`, queueing). The definition is documented as `COMMENT ON COLUMN chat_messages.runtime_ms` (migration 000551, surfacing as a Go doc comment on `ChatMessage.RuntimeMs`), on `chatloop.PersistedStep.Runtime`, in the chatd architecture doc, and in the Spend Management docs page. ## Index for the hourly scan None needed: `GetTotalChatMessageRuntimeMsInRange` (#27312) filters an hour-wide `created_at` range, which the existing `idx_chat_messages_created_at` b-tree already serves; the residual `runtime_ms IS NOT NULL` filter applies to one hour of rows. A partial index would add permanent write amplification for a query that runs once an hour. > [!NOTE] > Migration 000551 is also claimed by #27312; whichever merges second renumbers via `fix_migration_numbers.sh`. ## Tests - End-to-end: the existing full-server generation test now asserts `RuntimeMs.Valid` on the committed assistant row (it previously read `.Int64` without checking `.Valid`, so it passed on NULL). - Interrupted turn: full task-level test (real DB, mock clock) asserting the partial assistant message persists the attempt's runtime. - Errored stream: asserts a failed invocation yields no step and no runtime. - Tool-using turn: asserts runtime lands on the assistant row only and tool rows stay NULL. - Compaction: asserts the summarization call duration is recorded and lands on the compaction assistant message only. - `messagepartbuffer.EpisodeDuration` unit coverage. Blocks: CODAGT-843 (B3), CODAGT-838 (D8). --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Hugo Dutka <hugo@coder.com> |