Commit Graph
437 Commits
Author SHA1 Message Date
Michael Suchacz 7c39819665 fix(cli): close coderd on server startup errors (#30280)
`coder server` closes the coderd API only in its graceful shutdown
sequence. Every startup error returned after `newAPI` (creating
aibridged, registering Prometheus metrics, writing the config URL,
creating provisioner daemons, notifying systemd) skipped that close and
leaked the whole coderd instance: tailnet and wireguard-go, gvisor,
metricscache, dbrollup, cryptokeys, chatd, and pubsub queues.
`TestServer/Logging` hits this when test cleanup cancels startup while
aibridged is being created. The test helper ignores the resulting
`context canceled` error, so only the package-level goleak check in
`cli` fails.

The fix closes the API in a deferred `sync.OnceValue` registered right
after `newAPI`, and the shutdown sequence calls the same function. The
guard is needed because the closer is not safe to call twice: the AGPL
API returns "API already closed", and the enterprise closer closes each
component again. The normal shutdown path is unchanged.

## Testing

The new `TestServer/StartupErrorNoLeak` fails startup after coderd is
created by putting a directory at the config URL path.

- Red (test only): `go test ./cli -run
'^TestServer$/^StartupErrorNoLeak$' -count=1` exited 1 in 3 of 3 runs
with `goleak: Errors on successful test run: found unexpected
goroutines` (coderd frames) while the subtest passed.
- Green (with the fix): the same command passed in 3 of 3 runs.
- `TestServer/Logging` alone, 20 runs from one test binary: 20 of 20
passed with no goleak failure, including 3 runs that hit the `create
aibridged` startup error.
- `go test ./cli -count=1` passed in 4 of 4 runs, and `go vet ./cli` is
clean.

Found in the CI of #30272 (`test-go-pg (ubuntu-latest)`).

> Xum, an AI coding agent, implemented, tested, and opened this PR on
behalf of @ibetitsmike.
2026-10-07 13:39:31 +02:00
Callum Styan 15616728d5 refactor: move coderd/x/nats to coderd/nats 2026-10-05 16:52:36 -07:00
Sas Swart 23fb35d185 feat: let the AI Gateway emit the structured interception records (#30030)
coderd emitted the structured interception records as they arrived over
DRPC, so
a record the gateway never sends cannot be exported.

- Add `--ai-gateway-structured-logging-source` (`coderd`, `gateway`,
`both`) and
teach the gateway's `LogRecorder` to emit the same format where the
records
originate. Gateway lines omit `thread_parent_id` and `thread_root_id`,
which
  are resolved by a database lookup only coderd can perform.
- Reaching the serving pool required repairing the pool options: they
were
derived at construction sites and discarded, so the server always built
its
pool with the defaults. They are now derived from the deployment
configuration
in one place and passed to the pool that serves requests. The server
reports
those options, and its ownership of the `key_pool_state` collector, so a
second
  registration cannot silently reappear and panic startup.
- The standalone gateway inherits both structured logging options, since
it is
  now one of the processes that can emit.

This is the keystone of the stack: the flag, the mechanism that carries
it and
the behavior it selects land together, so
`TestAIGatewayStart_InheritedOptions`
and the gateway help output stay truthful at every commit.

<details>
<summary>Why the stack is ordered this way</summary>

The final design is a chain of recorder decorators:

    +---------------------+
    |    AsyncRecorder    |  all records except the interception start
    +---------------------+
    |     LogRecorder     |  emit the structured record
    +---------------------+
    | ValidatingRecorder  |  refuse malformed records
    +---------------------+
    |  caller middleware  |  record policy, e.g. drop content
    +---------------------+
    |    TraceRecorder    |
    +---------------------+
    |    DRPCRecorder     |  acquire client, send to coderd
    +---------------------+

Ordering constraints:

- The record type constants (1) are the vocabulary both emitters and the
  validator use, so they come first.
- Validation (5) rejects unset timestamps, so records must already be
stamped
where they are created (3). Stamping at creation leaves
`WrappedRecorder`
  doing nothing but client acquisition, which is why it is folded into
  `DRPCRecorder` (4) and not before.
- coderd's server-time fallback (2) is what keeps a gateway of an older
version
  from persisting zero times once the gateway owns stamping (3).
- PRs 1 to 5 change no user-visible surface. Each deployment option then
lands
with the code that reads it: the log source with gateway-side emission
(6),
content recording with the filter that drops the records (7). Pool
options are
plumbed to the serving pool in 6, where the first option that needs them
  arrives, rather than as an unused hook beforehand.

</details>

---
🤖 Raised by [Coder Agents](https://coder.com) on behalf of @SasSwart.
2026-09-29 11:48:52 +02:00
Callum Styan 83bef8a12d fix: consolidate pg and nats pubsub metrics generation into one (#26843) 2026-09-22 14:00:44 -07:00
Ehab Younes 9974294ca9 feat: report arbitrary workspace agent session types (#28338)
Report session counts as a dynamic app-name map from the agent, the SSH CLI, and the workspace usage API, so a new IDE flows end to end without a schema, proto, or server change.

- Report through `Stats.session_counts` on Agent API v2.11. Older agents keep reporting through the deprecated count fields.
- Accept arbitrary names from `coder ssh --usage-app` and the workspace usage API. A name the registry does not recognize keeps its identity and is attributed to the `unknown` family.
- Add reconnecting PTY counts to the map without overwriting the counts SSH reported.
- Normalize and merge names before applying the deterministic 64-name cap, ordered by known apps, then count, then name. Overflow sums into `unknown`, and non-positive counts cannot cancel valid activity.
- Strip control characters, treat whitespace-only HTTP app names as absent, and require the exact environment variable prefix when you extract an SSH session type.
- Count overflow entries in `coderd_agentstats_session_counts_overflow_total` and keep raw payloads out of logs.
- Change the usage API's `app_name` to a string, and update the SDK types and API reference.

closes #27410
2026-09-19 00:32:51 +02:00
Spike Curtis aae145bdc9 feat: make NATS pubsub opt-out via no_nats_pubsub experiment (#29283)
Makes the embedded NATS pubsub an opt-out feature instead of opt-in. The
`nats_pubsub` experiment is removed and replaced with a `no_nats_pubsub`
experiment, so NATS pubsub is now the default and can be disabled to
fall back to PostgreSQL pubsub.

NATS clustering requires this replica's routable address, resolved into
`clusterHost` from either `--cluster-host` (`CODER_CLUSTER_HOST`) or the
DERP relay URL. When neither is set (allowed when DERP is disabled on
the primary), NATS cannot form a cluster, so startup logs a big error
and falls back to PostgreSQL pubsub. The NATS cluster CA is only
minted/rotated when NATS clustering is actually active (experiment not
set and a cluster host is present).

<details>
<summary>Implementation notes</summary>

- `codersdk/deployment.go`: removed `ExperimentNATSPubsub`
(`nats_pubsub`), added `ExperimentNoNATSPubsub` (`no_nats_pubsub`,
display name "No NATS Pubsub").
- `cli/server.go`: NATS is used by default unless `no_nats_pubsub` is
enabled; if `clusterHost` is empty, log an error and keep the PostgreSQL
pubsub.
- `coderd/coderd.go`: gate NATS CA minting/rotation on
`!ExperimentNoNATSPubsub && ClusterHost != ""` (i.e. only when NATS
clustering is active).
- `enterprise/coderd/licenses.go`: updated a stale comment.
- `enterprise/coderd/coderd_test.go`: `TestMultiReplica_NATSPubsubPeers`
no longer sets the removed experiment.
- Regenerated `coderd/apidoc/docs.go`, `coderd/apidoc/swagger.json`,
`docs/reference/api/schemas.md`, `site/src/api/typesGenerated.ts`.

</details>

---
*This PR was generated by Coder Agents on behalf of @spikecurtis.*
2026-09-18 12:58:06 +00:00
Michael Suchacz d2e94e299a chore: delete dead functions and skip-only or duplicated tests (#29461) 2026-09-17 18:14:08 +02:00
Bobby Ho 7bb049f24c feat: log the OAuth2 provider state and registered apps at startup (#29267)
PLAT-492: Log OAuth2 provider state at startup

Log whether the OAuth2 provider is enabled and name CODER_OAUTH2_PROVIDER_ENABLE.
When disabled, warn if OAuth2 applications are still registered and tell admins how to re-enable the provider.
Startup continues if the application lookup fails.
Add tests for enabled, disabled, registered-app, and query-error cases.
Manually verified upgrade behavior, route gating, token preservation, dashboard visibility, and all supported flag forms.
2026-09-14 18:31:48 -07:00
Paweł Banaszewski 8472a6884f chore!: remove env-based AI provider configuration (#29053)
> AI tools were used to create this PR

Removes deprecated AI provider configuration through environment
variables, CLI flags, and YAML.
Deletion includes indexed providers and legacy OpenAI, Anthropic, and
Bedrock options under both AI Gateway and AI Bridge branding, eg.
`CODER_AI_GATEWAY_PROVIDER_<N>_*` or `CODER_AI_GATEWAY_ANTHROPIC_KEY`.
Removes the associated startup-seeding code.
Updates tests and documentation.

**Breaking change:** Remove [obsolete AI provider CLI flags and YAML
keys](https://coder.com/docs/ai-coder/ai-gateway/providers#database-management-of-providers)
before upgrading, otherwise Coder will fail to start with an `unknown
flag` or `unknown option` error. Provider environment variables are
ignored and no longer seed the database. Existing database-stored
provider configuration is preserved.
2026-09-09 10:16:24 +02:00
Michael Suchacz d6b67e8e31 feat: enforce SSRF protection for MCP config-directed traffic (#28242)
Routes all MCP config-directed traffic from coderd and chatd through a
shared SSRF-protected HTTP client, now that organization admins, not
only deployment admins, control MCP server URLs (#27942 and its stack
below).

## Summary

- Uses the [`coder/safedial`](https://github.com/coder/safedial)
library: it blocks private and special-purpose destinations at dial time
(validating resolved addresses at connection time so DNS rebinding
cannot bypass the check) and rejects cross-origin redirects.
- Covers the complete traffic surface: OAuth2 discovery, dynamic client
registration, code exchange, token refresh, revocation, and runtime MCP
connections from chatd.
- Deployments that intentionally host internal MCP servers opt in via
the new `--mcp-allowed-private-cidrs`
(`CODER_MCP_ALLOWED_PRIVATE_CIDRS`) option.
- Includes the deployment configuration surface, generated docs, CLI
goldens, TypeScript types, and regression coverage for each traffic
path.

## Merge window

This protection originally lived inside #27942 and was split out to keep
that diff reviewable. Until this PR lands, the stack below ships with
only main's existing discovery IP-range guard
(`CODER_MCP_OAUTH2_DISCOVERY_ALLOWED_IP_RANGES`), while org admins can
already point MCP configs at arbitrary URLs. This PR should merge
promptly after the stack below it.

Top of the MCP org-separation stack (CODAGT-711 -> CODAGT-717 audit ->
CODAGT-712 ACLs -> CODAGT-806 token RBAC -> CODAGT-714 org picker ->
this PR).

> Mux (AI agent) authored this PR on Mike's behalf.

<!-- mux-attribution: model=claude-fable-5 thinking=high -->
2026-08-30 13:09:17 +02:00
Yevhenii Shcherbina 0bc2858b6e feat: report unpriced AI models to owners (#28419)
Closes
[AIGOV-568](https://linear.app/codercom/issue/AIGOV-568/notify-admins-about-unpriced-ai-models).

AI Bridge records token usage for a model with no price at a NULL cost,
so its spend is neither reported nor enforced against any budget. Until
now that only reached admins through an info log and a Prometheus
metric.

Owners now receive a weekly report listing models used without a price
in the past week. The report links to documentation explaining how Coder
calculates spend from the default price book and how owners can
configure missing prices with `coder exp ai-model-prices`.

## Screenshots

<img width="726" height="531" alt="image"
src="https://github.com/user-attachments/assets/168ae378-60cb-4c96-8f60-36c6edc0b268"
/>
<img width="470" height="417" alt="image"
src="https://github.com/user-attachments/assets/7dc71a1e-0c57-44f1-86f7-df800603c865"
/>



## Design

**Derived, not tracked.** The unpriced set is computed at report time
from interceptions that recorded token usage, joined against the current
`ai_model_prices` table. Requests that produced no token usage are
excluded.

**No new loop or table.** `reportUnpricedAIModels` is a sibling of
`reportFailedWorkspaceBuilds` in the existing report generator, reusing
its ticker and `notification_report_generator_logs`. Each report runs in
its own transaction under a separate advisory lock, so failures and lock
contention do not couple otherwise independent reports. The weekly
frequency is enforced by the persisted timestamp rather than by the
ticker, which restarts with the process and runs on a different phase in
each replica.

## Behaviour

| Situation | Outcome |
|---|---|
| Model used without a price | Listed in the next weekly report |
| Price set | Disappears from the next report |
| Still unpriced and still in use | Reported again each week |
| Model stops being used | Drops out of the report |
| Nothing unpriced | No notification; window still advances |
| More than 100 unpriced models | Top 100 by usage, with the total
reported alongside |
| openai-compat models | Excluded, they cannot be priced |
| Many replicas | Exactly one report per week |
| First ever run | Reports models used in the preceding week immediately
|

## Design choices

1. **openai-compat** is excluded because it cannot be priced and would
produce permanent, unactionable notifications.
2. **Token volume** orders the list but is not shown. The 100-model cap
therefore prioritizes models responsible for the most unpriced usage.
3. **Placement** remains in the generic report generator rather than
`aibridgedserver`. This keeps one periodic-report lifecycle at the cost
of coupling the notifications package to an AI cost-control query.
4. **No license gate.** The reporter runs even when AI Gateway is
disabled. With no recorded token usage the query is empty and no
notification is sent.

---

Created by Coder Agents on behalf of @evgeniy-scherbina.
2026-08-27 21:04:13 +00:00
McKayla はな 2a8e390f0c refactor: replace most usage of ptr.Ref with new (#28282) 2026-08-26 11:57:56 -06:00
McKayla はな 5ca0143992 refactor: upgrade golangci-lint to v2.0.2 (#28416) 2026-08-21 23:46:27 -06:00
Cian Johnston 6079c514ee fix: follow-up fixes for conditional VCS requests (#27711)
Follow-ups from #27627 

- Memoizes `Config.Git()` with a mutex so the provider's ETag response
cache survives across calls. Only successful construction is cached;
errors are retried.
- Moves the HTTP client onto `Config.HTTPClient`, wired through
`ConvertConfig`, so `Git()` no longer takes a per-call argument that
would be silently ignored after memoization.
- `newGitHub` and `newGitLab` now return `(Provider, error)`,
eliminating the typed-nil-interface class in `gitprovider.New` rather
than the single instance.
- Gates the 304 branch on a `haveCached` flag instead of a nil body
check.
- Only caches bodies that decode successfully, preventing poisoned
entries.
- Keys the response cache on the full token digest rather than a
truncated prefix.
- Tests added: `TestConfigGitMemoizesProvider`,
`TestConfigGitRetriesOnConstructorError`,
`TestGitLabConstructorErrorReturnsNilInterface`,
`TestResponseCacheStore`,
`TestConditionalRequestReuse/MalformedResponseNotCached`;
`TestConvertYAML/CustomScopesAndEndpoint` now asserts
`Config.HTTPClient` wiring.

Follow-ups tracked in #28139, #28140, #28141, #28142.

> 🤖 Generated by Coder Agents on behalf of @johnstcn.
2026-08-18 09:00:20 +01:00
Asher b5d18bb9c9 feat: add redirect URL override for external auth (#28082) 2026-08-17 14:09:23 -08:00
J. Scott Miller 66b065323b feat: log rate-limited external auth token validation (#26754)
When `ValidateToken` keeps a token because the external auth validation
endpoint was rate-limited (a `403` with rate-limit headers or a `429`),
it returns `valid=true` without provider confirmation. Previously this
happened silently, so operators couldn't tell a provider-confirmed token
from one kept optimistically during a rate limit.

This adds a `Logger` to `externalauth.Config` and emits a `Warn` (with
`provider_id`, `provider_type`, `status_code`, and `reason`) on those
rate-limit branches. It also adds a
`coderd_oauth2_external_requests_rate_limited_total{name, source,
status_code}` counter, incremented in the instrumented round tripper
whenever a provider returns a rate-limited response. The rate-limit
detection is the shared `xhttp.IsRateLimited` (in `coderd/util/xhttp`),
used by both the tripper and `ValidateToken` so the metric and the
validation decision share one definition; no extra wiring is needed
since `ValidateToken` already routes through the instrumented client
with `source="ValidateToken"`.

One deliberate behavioral change rides along: rate-limit detection now
also recognizes the unprefixed `RateLimit-Remaining` header (GitLab, and
the IETF draft rate-limit headers), so a `403` with
`RateLimit-Remaining: 0` is treated as optimistically valid where it was
previously treated as revoked. All other valid/invalid decisions are
unchanged. `TestValidateToken` asserts the warning's fields on the
rate-limited cases and no warning for revocations, `401`, and confirmed
responses; `promoauth` and `xhttp` tests cover the detector and the new
counter.

<details>
<summary>Manual testing</summary>

The signals fire on the external-auth status check (`GET
/api/v2/external-auth/{id}`), which calls `ValidateToken`. To force a
rate-limited response, point a provider's `validate_url` at a mock that
returns the rate-limit shape:

1. Run a mock returning `429` on one path and `403` +
`X-RateLimit-Remaining: 0` on another.
2. Start `coder server` with `--prometheus-enable` and external auth
providers whose `validate_url` point at those mock paths (e.g.
`CODER_EXTERNAL_AUTH_0_VALIDATE_URL=http://127.0.0.1:5599/429`).
3. Create a stored link, either complete the OAuth flow, or insert a row
into `external_auth_links` with a future `oauth_expiry` (token contents
are irrelevant; the mock rejects regardless).
4. `curl` the status endpoint with a session token, then check:
- coderd logs for the `Warn` (`reason=status_code` for `429`,
`reason=rate_limit_headers` for `403`),
- the metrics endpoint for
`coderd_oauth2_external_requests_rate_limited_total{...,status_code="429"|"403"}`.

Notes: `scripts/testidp -429` only rate-limits `/oauth2/userinfo`, not
the `/external-auth-validate/...` path, so it does not exercise this;
use a mock `validate_url`. The default Prometheus port `2112` may
already be taken on dogfood workspaces, set `CODER_PROMETHEUS_ADDRESS`
to a free port.

</details>

🤖 Generated with the help of Coder Agents on behalf of @jscottmiller.
2026-08-10 14:43:16 -05:00
Susana Ferreira c351280a37 feat: add Prometheus metrics for AI Governance cost control (#27490)
## Description

Adds Prometheus metrics for AI budget cost control, emitted by the
aibridged server under the `cost_control` subsystem (full names are
prefixed `coder_ai_gateway_`).

- `blocked_requests_total` (counter) — labels: `group_id`
- `blocked_users` (gauge) — labels: `group_id`
- `unpriced_requests_total` (counter) — labels: `provider`, `model`
- `enforcement_duration_seconds` (histogram) — labels: `outcome`

## Changes

- Add `GetOverBudgetUsersPerGroup` query (plus dbauthz/dbmetrics/dbmock
wiring) to count over-budget users per effective group.
- Add a background collector that refreshes the `blocked_users` gauge on
an interval, started only when Prometheus is enabled.
- Wire `Metrics` through the aibridged server, coderd API,
`cli/server.go`, and the enterprise AI gateway handler; recording is
nil-safe when metrics are unset.

Closes
https://linear.app/codercom/issue/AIGOV-296/add-prometheus-metrics-for-cost-control

> [!NOTE]
> Initially generated by Claude Opus 4.7, modified and reviewed by
@ssncferreira
2026-07-28 09:22:58 +01:00
Paweł Banaszewski 5770085435 fix: add prefix to standalone metrics (#27526)
Adds `coder_ai_gateway_` to standalone Gateway metics to match embedded
case.
2026-07-27 13:02:49 +00:00
Atif Ali 2b2a5c963a Revert "fix(coderd): explain default GitHub app org visibility on login rejection" (#27388) 2026-07-21 16:17:27 +00:00
Atif Ali 48e9bb3391 fix(coderd): explain default GitHub app org visibility on login rejection (#27374)
## Problem

On a fresh deployment with no custom GitHub OAuth app, Coder falls back
to the default Coder-managed GitHub app. That app can only see
organization memberships in organizations where it has been installed.
If `CODER_OAUTH2_GITHUB_ALLOWED_ORGS` is set but the app isn't installed
in the allowed organizations, the membership list comes back empty and
every login, including the first admin login, is rejected with a bare
"You aren't a member of the authorized Github organizations!" with no
hint about the actual cause. This leaves fresh deployments in an
apparently broken state.

## Fix

* Append a remediation hint to the login rejection when the default
provider is configured, pointing at the [app installation
page](<https://github.com/apps/coder/installations/select_target>) and
at configuring a custom GitHub OAuth app.
* Log a startup warning when the default provider is combined with
`CODER_OAUTH2_GITHUB_ALLOWED_ORGS`, listing the allowed orgs and the
install URL.
* Document the installation requirement next to the
`CODER_OAUTH2_GITHUB_ALLOWED_ORGS` step in the GitHub auth docs.

Access-control behavior is unchanged; the org check still rejects logins
as before, it just explains why and how to fix it.

## Testing

* New `TestUserOAuth2Github/NotInAllowedOrganizationDefaultProvider`
asserts the hint appears when `DefaultProviderConfigured` is set; the
existing `NotInAllowedOrganization` subtest asserts it does not leak
into the custom-app path.

Fixes coder/coder#17752
2026-07-21 20:43:10 +05:00
Paweł Banaszewski 3126306598 feat: add --aigateway-proxy-target flag (#27122)
Adds `--aigateway-proxy-target` option to
`deploymentGroupAIGatewayProxy` that defines URL to which intercepted
requests should be forwarded to.
Forward URL used to be hardcoded to `coderAPI.AccessURL` pointing to
embedded Gateway. With addition of standalone AI Gateway this needs to
be configurable.

Renamed `aibridgeproxyd.Server.coderAccessURL` and `coderAccessPort` ->
`gatewayURL` and `gatewayPort` + option to better reflect reality.
2026-07-14 16:40:22 +00:00
Callum Styan ad29777cb2 feat: NATS mTLS pubsub implementation (#26902) 2026-07-13 11:00:02 -07:00
Paweł Banaszewski bab8ce9d41 feat: setup logging, tracing and metrics in standalone AI Gateway (#27068)
Adds logging, tracing and metrics setup to standalone AI Gateway.
Existing options are re-used when possible.
2026-07-09 14:02:18 +00:00
Ethan 8b60d1d877 ci: cache embedded postgres binaries in flake checks (#26986)
flake-go keeps failing every `TestServer` subtest that boots `coder
server` with built-in PostgreSQL ([example
run](https://github.com/coder/coder/actions/runs/28685288282/job/85134336803))
with `no version found matching 13.21.0`, which is embedded-postgres's
error for any non-200 while downloading the Postgres binary archive from
Maven. The archive gets cached under the server's config root, which is
a fresh temp dir in every test, so with `test-count: 35` one flake run
downloads it dozens of times and Maven rate-limits the runner. Using an
external Postgres (like coder/terraform-provider-coderd#370 did) would
defeat the point, since these subtests exist to exercise the built-in
Postgres path.

Instead, `startBuiltinPostgres` now honors `EMBEDDED_PG_CACHE_DIR` as
the archive cache path (test runs only, data/runtime dirs stay
per-test), and flake-go.yaml wires in the existing `embedded-pg-cache`
actions the same way the Windows/macOS lanes in ci.yaml already do. All
iterations then share a single download, and usually zero once the
actions cache is warm. The upload step only saves on `main`.
2026-07-07 11:44:46 +10:00
Itay Dafna d7ad85f7f6 feat: support multiple OIDC redirect URIs (#25408)
This PR adds a new opt-in setting, `CODER_OIDC_REDIRECT_ALLOWED_HOSTS`,
that lets a single Coder deployment complete OIDC login on more than one
hostname. When the allowlist is non-empty, Coder picks the OIDC
`redirect_uri` based on the incoming request's Host header (validated
against the list) instead of always using the static URL derived from
`CODER_ACCESS_URL`. When unset, the (default) behavior is identical to
today.

The motivation is that a single Coder deployment is frequently reachable
via multiple hostnames - for example, an internal hostname for users on
a corporate VPN and a different hostname routed through a zero-trust
gateway for users off-VPN - but OIDC login today only works on whichever
single hostname `CODER_ACCESS_URL` points to, because the `redirect_uri`
sent to the IdP is fixed at server startup. Users who reach the
deployment on any other valid hostname can see the login page but fail
the OIDC callback, since the IdP redirects them back to a hostname they
can't reach (or whose cookies they don't have).
2026-07-05 06:36:33 +02:00
Mathias Fredriksson 047c47495b refactor: drop chat_model_configs provider column (#26877)
The provider type already lives authoritatively in ai_providers.type,
reachable on every active row through ai_provider_id, which the
chat_model_configs_ai_provider_required_when_active CHECK makes
mandatory. The stored provider string was a denormalized copy the system
kept in sync with a startup backfill and no longer needs.

Every surface now derives provider type from the linked ai_providers
row. Telemetry is the one exception: it keeps emitting provider, now
sourced from ai_providers.type via a JOIN, so the BigQuery column and the
Nexus dashboards that read it are unaffected. The experimental HTTP/SDK
response drops provider and makes ai_provider_id required, since those
endpoints return only active configs; consumers resolve provider type
from ai_provider_id and the AI providers listing.

This ships in a single release with no compatibility window: production
reads the table via SELECT *, so a pre-drop binary fails config reads the
moment the column is gone. Operators must scale to zero before upgrading,
and there is no rollback.

Closes CODAGT-599
2026-07-01 15:59:55 +03:00
Danny Kopping ce94d42e19 feat: fetch providers over DRPC (#26650)
Closes [AIGOV-455](https://linear.app/codercom/issue/AIGOV-455/extend-drpc-with-buildproviders).

## Why

The AI Gateway (`aibridged`) is being split into a standalone process that must not touch the database. `coderd` stays the source of truth and seeds the `ai_providers` / `ai_provider_keys` tables from the environment. This PR adds a DRPC call so the gateway fetches provider config from `coderd` instead of reading the DB, for both the embedded and standalone daemons.

## What

- **Proto:** new `ProviderConfigurator` service with a unary `GetAIProviders` RPC, plus `AIProvider` / `AIProviderBedrock` messages. `CurrentMinor` bumped to 1 (additive).
- **Server (`coderd/aibridgedserver`):** `GetAIProviders` runs a read-only `InTx` under `LockIDAIProvidersEnvSeed` so it never returns a mid-seed snapshot, reads providers (incl. disabled) plus keys for enabled ones, and maps to proto under `dbauthz.AsAIBridged`. Unmappable rows are skipped and logged; plaintext keys and Bedrock secrets are never logged.
- **Client:** `DRPCProviderConfiguratorClient` wired into the client union, `dialer.go`, and `CreateInMemoryAIBridgeServer`.
- **cli:** `BuildProvidersFromProto` maps the response through the existing DB-neutral `buildProvider`. A shared `poolRPCReloader` does the fetch/build/replace for both daemons: the embedded daemon reloads on every `ai_providers` change and fails startup if it cannot subscribe; the standalone gateway drives the same reloader once at startup, retrying until success and staying interruptible.
- **Dead code removed:** `BuildProvidersFromConfig`, `ProvidersFromConfig`, `AIProviderFromConfig`, and the DB-read `BuildProviders` path.
2026-06-29 13:34:58 +02:00
Steven Masley ad355aeaa9 feat: add INSECURE oidc email fallback flag for IdP brokers (#26751)
<!-- Authored by Coder Agents on behalf of @Emyrk. -->

Adds an opt-in `CODER_DANGEROUS_OIDC_EMAIL_FALLBACK` flag (alias
`--dangerous-oidc-email-fallback`) for IdP brokers that do not issue a
stable `sub` for the same user across connections.
2026-06-26 11:23:42 -05:00
Steven Masley 84350e4e7c feat: report SCIM configuration on Deployment (#26628)
Adds two nullable booleans to `telemetry.Deployment`:

- `SCIMEnabled`: `true` when `CODER_SCIM_AUTH_HEADER` is set.
- `SCIMUseLegacy`: `true` when `CODER_SCIM_USE_LEGACY` is set.

Both mirror `Deployment.IDPOrgSync`: nullable for backward
compatibility, and report configuration state rather than license
entitlement (#16323).

Lives on `Deployment` rather than `Snapshot` so the existing
`bqDeployment` table on `coder/coder-telemetry-server` gets two columns
instead of a new table.

`SCIMAPIKey` is annotated as a secret and is scrubbed by
`WithoutSecrets` before the config reaches telemetry, so
`DeploymentConfig.SCIMAPIKey` is always empty in production. The
booleans are pre-computed from the pre-scrub `DeploymentValues` in
`cli/server.go` and passed in via `telemetry.Options.SCIMEnabled` /
`SCIMUseLegacy`.

Pairs with
[coder/coder-telemetry-server#43](https://github.com/coder/coder-telemetry-server/pull/43),
which adds the matching `bqDeployment` columns and the manual BigQuery
`ALTER TABLE` step.

---

Generated by Coder Agents on behalf of @Emyrk.
2026-06-25 08:54:37 -05:00
Jon Ayers 6da322d59f feat: add Prometheus metrics to NATS pubsub for parity with PGPubsub (#26441) 2026-06-23 11:59:48 -05:00
Susana Ferreira 970bd73691 feat: add /api/v2/ai-gateway API route aliases (#26475)
## Description

Registers `/api/v2/ai-gateway/*` as the new API path for AI Gateway, replacing `/api/v2/aibridge/*`. Both prefixes share the same route builder (`aiBridgeRoutes`) backed by a single in-memory handler, so existing `/aibridge` endpoints continue to work. New endpoints must be registered on the enterprise API handler under `/api/v2/ai-gateway` only.

Swagger annotations now point to `/api/v2/ai-gateway` paths with a backward-compatibility note referencing `/aibridge`. The legacy `/aibridge` routes are skipped in the swagger documentation test.

## Changes

- Store one raw handler (`aiGatewayHandler`) instead of two prefix-stripped handlers
- Register `/ai-gateway` and `/ai-gateway/proxy` route aliases alongside legacy `/aibridge` routes
- Move `/aibridge/keys` to `/ai-gateway/keys`
- Update in-process transport to use `/api/v2/ai-gateway` prefix
- Update SDK client URLs and proxy forwarding URL
- Swap `@Router` and `@Tags` annotations from `aibridge`/`AI Bridge` to `ai-gateway`/`AI Gateway`
- Rename user-facing error messages from "AI Bridge" to "AI Gateway"
- Define consts for route prefixes (`AIGatewayRootPath`, `AIBridgeRootPath`)
- Update tests and comments to use new paths

Note: the following will be addressed in follow-up PRs:
- Frontend API URLs
- Frontend routes and redirects
- Dogfood main.tf updates
- Hand-written documentation URL updates
- aibridge internal comments and nits
- Scale tests path updates

Refs https://linear.app/coder/issue/AIGOV-230

> Generated with the assistance of Coder Agents (@ssncferreira)
2026-06-23 12:15:10 +01:00
Cian Johnston d5ec26beac chore: replace testing.Testing with flag lookup (#26552)
In our codebase we have an existing convention of using
`flag.Lookup("test.v")` instead of `testing.Testing()`. This avoids
pulling in the entire `testing` package. Another consequence: some of
our custom linters trigger upon import of the `testing` package which
can lead to unexpected linter errors.
2026-06-19 19:59:54 +01:00
Paweł Banaszewski f1ce1013c4 chore: export AI Gateway metrics under new branding + keep old as alias (#26413)
> AI Tools where used in this request.

Registers `coder_aibridged_*` and `coder_aibridgeproxyd_*` metrics under
new prefixes: `coder_ai_gateway_*` and `coder_ai_gateway_proxy_*`.
Old prefix is still exported. Will be removed in later release.

Also updated the `metricsdocgen` static fixture. Added 4
previously-undocumented metrics `key_pool_state`,
`key_pool_state_transitions_total`, `key_pool_exhaustions_total`,
`key_pool_failover_attempts` added the `client` label to the existing
interception, prompt, and token counter samples.

Updated AI Gateway documentation.
2026-06-17 13:10:53 +02:00
Steven Masley 0e45ded0ed feat: deployment flag to auto handle changed oidc providers (#26419)
An opt-out flag exists as an escape hatch

closes https://linear.app/codercom/issue/PLAT-343/automatically-reset-user-link-for-affected-users-when-idp-provider
2026-06-16 13:26:04 -07:00
Steven Masley 1d03e63f4f feat: implement package and cli tool for repairing oidc links (#26418) 2026-06-16 12:46:10 -07:00
Hugo Dutka 62288782fc chore: clean up dbpurge after the chatd refactor (#26344)
Addresses
https://github.com/coder/coder/pull/26109#discussion_r3379072397 and
https://github.com/coder/coder/pull/26109#discussion_r3379093655.
2026-06-16 14:01:31 +02:00
Danny Kopping a1330e3a8c refactor: rename Ai* database identifiers to AI* (AIGOV-369) (#26327)
Adds `ai` to sqlc's `gen.go.initialisms` in `coderd/database/sqlc.yaml`
so the generated DB code follows Go's initialism convention. Adds the
matching `ai` -> `AI` case to the dbgen PascalCase helper
(`scripts/dbgen/main.go`) so the corresponding `dbmem` / mock
identifiers stay in sync. `make gen` regenerates the rest; hand-written
call sites that consume DB-generated identifiers
(`enterprise/audit/table.go`, `coderd/database/modelmethods.go`,
`enterprise/coderd/aigatewaykeys.go`, `coderd/database/dbauthz/*`, etc.)
are updated to match.

Scope is deliberately limited to the database layer:

- `coderd/rbac/*` (resource and scope generators) is untouched —
`ResourceAi*` / `ScopeAi*` constants stay on main's casing.
- `codersdk/*` (Go SDK) is untouched — `codersdk.ResourceAi*` /
`codersdk.APIKeyScopeAi*` constants stay on main's casing, so external
Go SDK consumers see no source-level break.
- `Aibridge*` identifiers (one SQL token `aibridge`, not `ai_bridge`)
are out of scope.

On-the-wire values are unchanged: enum strings, RBAC resource type
strings, API key scope strings, and JSON tags all stay the same. The
HTTP/JSON surface is unaffected.

Refs:
[AIGOV-369](https://linear.app/codercom/issue/AIGOV-369/change-ai-references-in-coderddatabasemodelsgo-to-ai)

🤖 Generated with [Coder Agents](https://coder.com)
2026-06-16 09:01:43 +00:00
Hugo Dutka 4debd23cbb fix: chatd refactor (#26270)
Implements the chatd stabilization RFC.

Combines:
- https://github.com/coder/coder/pull/25908
- https://github.com/coder/coder/pull/25923
- https://github.com/coder/coder/pull/26109
- https://github.com/coder/coder/pull/26110
- https://github.com/coder/coder/pull/26111
- https://github.com/coder/coder/pull/26112
2026-06-12 13:33:12 +02:00
Cian Johnston a4c867f11b fix: backfill legacy Bedrock AI provider rows and stale model config strings (#26155)
Fixes CODAGT-548

Adds two idempotent startup backfills run after `newAPI():

- `BackfillBedrockProviderType`: promotes `ai_providers` rows from
`type=anthropic` with Bedrock settings to `type=bedrock`.
- `BackfillChatModelConfigProviderStrings`: fixes stale
`chat_model_configs.provider = "anthropic"` strings on rows whose linked
provider was just promoted.
- `UpdateAIProvider` query now also writes the `type` column, so the
fix persists on any subsequent PATCH.


> 🤖 Generated by Claude with oversight from a human.
2026-06-11 15:31:31 +01:00
Danny Kopping 78a6ec293e revert: "fix: avoid an errant license warning banner on new deployments that d…" (#26240)
Reverts coder/coder#26239

We cannot disable a feature which was previously enabled; this is a BC
break.
This is also using `AIGatewayRoutingEnabled` which will be removed in
the next release.
2026-06-11 08:07:32 +00:00
Sas Swart d0e9c5eda5 fix: avoid an errant license warning banner on new deployments that d… (#26239)
Problem: CODER_AI_GATEWAY_ENABLED defaulted to true, which both started
the in-memory gateway and enabled the licensed FeatureAIBridge. As a
result, deployments that never configured AI Gateway saw a spurious "AI
Governance add-on is required" warning whenever they had an older
(non-add-on) Premium license, since the feature was enabled-and-entitled
by default.

Fix: Decouple "external AI Gateway API enabled" from "in-memory daemon
running," so the external/licensed surface is off by default while Coder
Agents retain access by default.
2026-06-11 09:17:26 +02:00
Kyle Carberry dab1d3c81e fix(cli): sort external auth env vars by numeric index (#26230) 2026-06-10 14:21:54 -07:00
Cian JohnstonandCopilot Autofix powered by AI a26c46a3bf fix!: validate HostnameSuffix and SSHConfigOptions' (#26154)
- Adds server-side and client-side validation for
CODER_CONFIGSSH_HOSTNAME_SUFFIX and CODER_SSH_CONFIG_OPTIONS.
- **Server-side breaking change:** invalid values for either of these will cause `coderd` to exit with an error.
- Client-side: `coder config-ssh` will exit with an error if it detects invalid config.
- Adds tests for the above

Local smoke-testing: ran `develop.sh --env-file <path to an env file
containing badness>`. Validated that server startup failed as expected.

> 🤖 Generated by Coder Agents with supervision from a human.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-06-10 15:48:02 +01:00
Susana Ferreira 01ec5e4577 feat: add key pool failover metrics to aibridge (#25901)
## Description

This PR adds Prometheus metrics for aibridge's API-key failover, giving visibility into key pool health and failover behavior per provider.

The following metrics are introduced:

- **`key_pool_state`** (gauge): number of keys currently in each state (`valid`, `temporary`, `permanent`) per provider, sampled at scrape time.
- **`key_pool_state_transitions_total`** (counter): key state transitions during failover, labeled by `reason` (`rate_limited`, `unauthorized`, `forbidden`).
- **`key_pool_exhaustions_total`** (counter): times a pool ran out of usable keys, labeled by `outcome` (`rate_limited`, `auth_failed`).
- **`key_pool_failover_attempts`** (histogram): keys attempted before success or exhaustion (per interception for bridged requests, per request for passthrough).

## Changes

- Moves `MarkKeyOnStatus` and key-pool error handling onto `*keypool.Pool`.
- Attaches metrics to each provider's key pool at install time, on construction and on provider reload.
- Adds a scrape-time state collector and a `KeyPools()` accessor on the bridge pool to feed it.
- Tracks per-request key attempts in the bridged and passthrough failover paths.
- Adds test coverage for the new metrics across the keypool unit tests, the bridged intercept failover tests, and the passthrough failover test.

Closes https://github.com/coder/internal/issues/1447
Closes https://linear.app/codercom/issue/AIGOV-198/aibridge-key-failover-observability

> [!NOTE]
> Initially generated by Claude Opus 4.7, modified and reviewed by @ssncferreira
2026-06-09 10:49:47 +01:00
Jon Ayers 167ac7b879 feat: add nats experiment (#25703) 2026-06-03 15:37:19 -05:00
Danny Kopping c8555e2163 fix: deprecate ai provider seeding env config (#25854)
Environment variables used to configure AI Gateway providers are now deprecated, and we need to reflect this as such.
2026-06-01 15:15:47 +02:00
Danny Kopping 12520ee964 feat: add ai provider status and reload freshness metrics (#25770)
Add metrics for `aibridged` and `aibridgeproxyd`'s provider statuses. AI providers can be modified, and possibly misconfigured, at runtime. These metrics help operators understand the state of these provider definitions in case unexpected behaviour is observed.
2026-05-28 14:57:33 +02:00
Danny Kopping 2770bdc9d1 feat: route extra ai_provider_types through OpenAI and Anthropic providers (#25722)
_Disclosure:_ _produced_ _with_ _Claude_ _Opus_ _4\.7_

AI Gateway only supports Anthropic (+Bedrock), OpenAI, and Copilot providers at present. All other types (Vercel, Gemini, etc) will be mapped to OpenAI since they support OpenAI-compatible endpoints.
2026-05-27 16:16:05 +02:00
Danny Kopping 79e007cf30 feat: hot-reload aibridged and aibridgeproxyd providers on DB changes (#25673)
Previously the in-process aibridge daemon and the enterprise aibridgeproxy daemon both snapshotted their provider routing once at boot. Any `ai_providers` or `ai_provider_keys` mutation required a restart for either to pick it up.

Add an `ai_providers_changed` pubsub channel that the CRUD handlers publish on after Create / Update / Delete. Both daemons subscribe:

- **aibridged** rebuilds its `[]aibridge.Provider` snapshot via `BuildProviders` and swaps it into the pool atomically. Inflight requests keep serving against the bridge they already acquired; new acquires build against the new snapshot. Per-provider construction errors stay scoped to the offending row.
- **aibridgeproxyd** rebuilds its routing snapshot from `GetAIProviders` and swaps the host→provider map atomically. The MITM listener picks up new providers without restart.

DB read for aibridgeproxyd uses the existing `AsAIProviderMetadataReader` subject for routing-only access.
2026-05-27 11:58:43 +02:00
Ethan e91bec8574 fix(cli): close aibridge daemon before WebSocket shutdown wait (#25719)
> [!WARNING]
> The investigation and solution in this PR were done with
[Mux](https://mux.coder.com/). I've reviewed the investigation
methodology, evidence and solution, and it all appears sound.

## Summary

PR #25570 (`refactor: move aibridged out of enterprise to AGPL`, merged
2026-05-22) added an in-memory aibridge DRPC server in
`coderd/aibridged.go` that does `api.WebsocketWaitGroup.Add(1)` and only
releases `Done()` when its client session is closed. PR #25575 then
flipped `CODER_AI_GATEWAY_ENABLED` to default to `true`, so every
`cli.Server()` invocation now spins up that goroutine.

In `cli/server.go`, the only call to `aibridgeDaemon.Close()` was a
`defer` scheduled at function return. During graceful shutdown the code
first calls `coderAPICloser.Close()`, which waits on
`api.WebsocketWaitGroup`. That wait sits for the full 10s timeout in
`coderd/coderd.go` (`websocket shutdown timed out after 10 seconds`),
then returns, then the function unwinds, and only then does the deferred
`aibridgeDaemon.Close()` fire and let the goroutine call `Done()`.

The 10s tax was previously latent (aibridged was enterprise-only and
opt-in). After the two May 22 PRs it hit every `cli.Server()` test. On
Linux/macOS CI it just makes the suite slower; on the Depot Windows
runner, the ramdisk reservation leaves only ~17 GiB of headroom and the
~10s shutdown tails of multiple concurrent package binaries overlap into
an OOM, presenting as `test-go-pg (windows-2022)` jobs that die silently
at the ~600s watchdog with an empty `steps` array.

See Slack:
https://codercom.slack.com/archives/C05AE94121Z/p1779807717764189

## Fix

Close `aibridgeDaemon` explicitly during graceful shutdown, **before**
`coderAPICloser.Close()` waits on the WebSocket wait group. This matches
the existing ordered-shutdown pattern used for `tunnel` and
`notificationsManager`. The deferred `aibridgeDaemon.Close()` is
retained as a safety net for early-return paths, and is safe to
double-call because `aibridged.Server.Close()` is already idempotent via
`shutdownOnce` in `coderd/aibridged/aibridged.go`.

## Regression test

`TestServer_AIGatewayShutdownOrdering` boots a real `coder server` with
`--ai-gateway-enabled=true`, cancels its context, and asserts graceful
shutdown finishes in under 8s. With the fix the test runs in ~0.1s;
without the fix it fails deterministically at ~10.0s. The flag is passed
explicitly so the test continues to guard the ordering even if the
deployment default is ever flipped back.

## Evidence this fixes the OOM

On Linux the patched `cli` test package drops from 114 s back to its
pre-regression 30 s wall time at the same single-process peak RSS (~7.6
GiB), and the `websocket shutdown timed out after 10 seconds` log line
disappears from every server-test run. Since the Windows OOM is the sum
of multiple concurrent 10 s shutdown tails overlapping past the runner's
~17 GiB headroom, removing those tails returns the concurrent-RSS budget
to its pre-regression level. The Windows OOM was intermittent (a handful
of hits across many runs since May 22), so a single green `test-go-pg
(windows-2022)` job on this PR is not by itself proof; confirmation will
come from watching Windows runs on `main` over the next several days and
seeing the ~600 s silent-kill fingerprint stop recurring.

Relates to ENG-2771
2026-05-27 17:33:14 +10:00