mirror of
https://github.com/coder/coder.git
synced 2026-11-22 11:28:40 +08:00
feea5ab959f59cd2135443ed2a0c19bf7643db8f
2
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
0bc2858b6e |
feat: report unpriced AI models to owners (#28419)
Closes [AIGOV-568](https://linear.app/codercom/issue/AIGOV-568/notify-admins-about-unpriced-ai-models). AI Bridge records token usage for a model with no price at a NULL cost, so its spend is neither reported nor enforced against any budget. Until now that only reached admins through an info log and a Prometheus metric. Owners now receive a weekly report listing models used without a price in the past week. The report links to documentation explaining how Coder calculates spend from the default price book and how owners can configure missing prices with `coder exp ai-model-prices`. ## Screenshots <img width="726" height="531" alt="image" src="https://github.com/user-attachments/assets/168ae378-60cb-4c96-8f60-36c6edc0b268" /> <img width="470" height="417" alt="image" src="https://github.com/user-attachments/assets/7dc71a1e-0c57-44f1-86f7-df800603c865" /> ## Design **Derived, not tracked.** The unpriced set is computed at report time from interceptions that recorded token usage, joined against the current `ai_model_prices` table. Requests that produced no token usage are excluded. **No new loop or table.** `reportUnpricedAIModels` is a sibling of `reportFailedWorkspaceBuilds` in the existing report generator, reusing its ticker and `notification_report_generator_logs`. Each report runs in its own transaction under a separate advisory lock, so failures and lock contention do not couple otherwise independent reports. The weekly frequency is enforced by the persisted timestamp rather than by the ticker, which restarts with the process and runs on a different phase in each replica. ## Behaviour | Situation | Outcome | |---|---| | Model used without a price | Listed in the next weekly report | | Price set | Disappears from the next report | | Still unpriced and still in use | Reported again each week | | Model stops being used | Drops out of the report | | Nothing unpriced | No notification; window still advances | | More than 100 unpriced models | Top 100 by usage, with the total reported alongside | | openai-compat models | Excluded, they cannot be priced | | Many replicas | Exactly one report per week | | First ever run | Reports models used in the preceding week immediately | ## Design choices 1. **openai-compat** is excluded because it cannot be priced and would produce permanent, unactionable notifications. 2. **Token volume** orders the list but is not shown. The 100-model cap therefore prioritizes models responsible for the most unpriced usage. 3. **Placement** remains in the generic report generator rather than `aibridgedserver`. This keeps one periodic-report lifecycle at the cost of coupling the notifications package to an AI cost-control query. 4. **No license gate.** The reporter runs even when AI Gateway is disabled. With no recorded token usage the query is empty and no notification is sent. --- Created by Coder Agents on behalf of @evgeniy-scherbina. |
||
|
|
d3f08b1983 |
feat: audit chat system instructions changes (#27668)
Adds an audit record for administrative events on the deployment-wide
chat instruction settings (system prompt, the include-default toggle,
and the plan-mode instructions), per CODAGT-719 and operator decision
D5. Each endpoint records under a stable identity: resource type
`chat_instruction_settings`, a fixed resource ID and a human-readable
target ("System prompt", "Plan mode instructions"), so two changes to
one setting share an ID and history-by-setting works. A real change
exports a Write entry with the old-to-new text visible; a
value-identical PUT still upserts and still returns 204 but records
nothing.
Attempts are recorded, not only transitions. Identity is assigned before
the authorization check, so a denied PUT exports a 403 row with an empty
diff (no request content reaches it), a validation failure exports a 400
row, and a write failure exports a 500 row, each with an empty diff; an
operator can tell "nothing changed" from "something changed and capture
degraded" by the status code.
The write path stays authoritative. The advisory lock and, on plan-mode,
the transaction exist only to serve change-detection; if any of that
machinery fails (lock, begin, commit, rollback), the handler runs main's
idempotent write path directly and derives the response from it, so a
member-visible failure of audit-only infrastructure can never replace
main's successful response. Accepted consequence: when the lock cannot
be taken, two concurrent identical writes can produce two rows instead
of one. That is audit degradation, which is allowed; changing a member's
response is not. Write failures keep the exact response the endpoint
produced before this wiring (transaction error for the system prompt,
which was always transactional; the raw write error for plan mode, which
was not), and the full transaction error is logged so rollback failures
cannot vanish.
<details>
<summary>CODAGT-66 plan entry: S1 (verbatim)</summary>
**S1 `feat: audit chat system instructions changes`** (CODAGT-719; base:
main)
- Struct: `database.ChatSystemPromptSettings{ID uuid.UUID; SystemPrompt
string; IncludeDefaultSystemPrompt bool; PlanModeInstructions string}`
in `coderd/database/types.go` (ticket-sketched shape; one struct, both
endpoints).
- Registration: union entry (diff.go), table.go entry (`id`
ActionIgnore, other three ActionTrack), `AuditActionMap` Write-only;
four request.go cases (`ResourceTarget` "", `ResourceID` from struct,
`ResourceType` new enum value `chat_system_prompt_settings`,
`ResourceRequiresOrgID` false with the "Artificial ID / deployment
singleton" comment convention).
- Migration: `ALTER TYPE resource_type ADD VALUE IF NOT EXISTS
'chat_system_prompt_settings';` comment-only no-op down (000558 shape);
number picked at push per the numbering constraint.
- codersdk: constant + prose `FriendlyString` ("chat system prompt
settings"); `TestAuditDBEnumsCovered` forces both. `coderd/audit.go`
presentation switches: rely on safe defaults (no link, generic
description); no FE changes (filter label falls back to capitalized
value; acceptable per precedent).
- Wiring `putChatSystemPrompt` and `putChatPlanModeInstructions`:
InitRequest with Action Write; artificial `ID: uuid.New()` on `New` only
when a change is detected; no-op suppression by leaving both aReq sides
unset (nil resource IDs skip the log, request.go skip rule); the write
path itself stays byte-identical (upserts still run unconditionally).
- `putChatSystemPrompt` (writes two keys conditionally in one existing
tx): inside that tx, read the pair via `GetChatSystemPromptConfig` for
`Old`, perform the conditional writes exactly as today, then RE-READ the
pair for `New`. The re-read is load-bearing:
`include_default_system_prompt` is computed from the toggle row AND the
prompt, so a prompt-only write can flip the effective value without the
request carrying the pointer. `PlanModeInstructions` stays zero on both
sides.
- `putChatPlanModeInstructions` (no tx exists today): wrap its
read-upsert in `InTx` (behavior-preserving: same single write);
`Old`/`New` populate only `PlanModeInstructions`; the two system-prompt
fields stay zero on both sides; no cross-key reads.
- Change detection compares the populated payload fields only (never the
artificial ID).
- Tests: handler-level coderdtest with `audit.NewMock()` asserting Write
entry on change and NO entry on a value-identical PUT, for both
endpoints (this also exercises `ResourceRequiresOrgID` end to end); the
fallback-flip case (no explicit include-default row, nonempty prompt set
to empty, effective boolean flips: entry emitted with the boolean diff);
diff assertions (old->new prompt text tracked, not secret) in
`enterprise/audit/diff_internal_test.go`; `TestAuditableResources`
passes by construction.
- Bookkeeping at PR open: correct CODAGT-719's no-op premise ("matches
the existing 204-on-unchanged behavior" does not exist on main;
suppression is new, write path unchanged).
- Review focus: Old capture and the New re-read inside the tx (three of
four existing singletons never set Old; do not copy them; and the
computed include-default value makes a naive New construction wrong);
the skip-on-no-op mechanism; prompt text deliberately visible in diffs.
</details>
Note: the plan excerpt above predates operator decision D5 (2026-07-30),
which this PR implements: the resource type is
`chat_instruction_settings` (not `chat_system_prompt_settings`), each
setting carries a stable ID and a display-name target (not a per-write
artificial ID and an empty target), no-op suppression runs through
`InitRequestWithCancel` (not the nil-ID skip), and attempts (denied,
failed, capture-degraded) record rows with real statuses and empty
diffs. Ticket bookkeeping for CODAGT-719 was corrected on Linear at
kickoff: the ticket's "matches the existing 204-on-unchanged behavior"
premise does not exist on main; suppression is new, and the write path
is unchanged.
> 🤖 This PR was created with the help of Coder Agents, and _will be_
reviewed by a human. 🏂🏻
---------
Co-authored-by: Michael Suchacz <203725896+ibetitsmike@users.noreply.github.com>
|