<!-- Authored with Coder Agents on behalf of @Emyrk -->
Refs
https://linear.app/codercom/issue/CORE-634/upgrade-embedded-postgresql-to-a-non-eol-version.
Precedes #32144.
Refuse to start built-in PostgreSQL when an existing data directory has
a missing or unsupported version, rather than letting embedded-postgres
replace its data.
- An existing `PG_VERSION` other than 13, including 18 or 19, now
returns an error without starting PostgreSQL.
- A missing marker also returns recovery guidance instead of
initializing over existing data.
- Fresh databases and existing PostgreSQL 13 databases keep their
current behavior.
- Development database recovery uses the same guard.
<details><summary>Agent context</summary>
The bundled embedded-postgres v1 library can delete an existing data
directory whose version does not match its PostgreSQL 13 binary. This
change checks the directory before invoking the library, without
changing the binary version or migrating data. Only Coder builds
containing this guard gain the protection; already-released older
binaries remain unchanged.
</details>
---
<sub>Coder Agents on behalf of @Emyrk.</sub>
fix(security): require setup token for first user on tunnels
Prevent unauthorized users from claiming Owner access on public `try.coder.app` tunnels before initial setup.
- Generate a setup token at startup when using a tunnel with no existing users.
- Require the token for first-user creation via the API, web UI, and CLI.
- Block GitHub/OIDC first-user signup until setup is complete.
- Support custom tokens via `--setup-token` / `CODER_SETUP_TOKEN` and non-interactive CLI setup.
- Update development scripts, documentation, and tests.
- Preserve existing behavior for deployments with an access URL or existing users.
Validated with automated tests and 23/23 manual scenarios using a real tunnel.
Refs CORE-784
Removes the deprecated `coder_aibridged_*` and `coder_aibridgeproxyd_*`
Prometheus metric aliases, which were scheduled for removal in v2.37.
The embedded AI Gateway and AI Gateway Proxy now emit only
`coder_ai_gateway_*` and `coder_ai_gateway_proxy_*`.
- Replace `NewMetricAliasRegisterer` with
`prometheus.WrapRegistererWithPrefix` in `cli/server.go` and
`enterprise/cli/aibridgeproxyd.go`.
- Delete `coderd/prometheusmetrics/metricalias.go` and its test (no
remaining callers).
- Drop alias support from the metrics docs scanner.
- Update the AI Gateway monitoring and rebranding migration docs: legacy
names are removed as of v2.40.
**Breaking:** dashboards and alerts that still use the legacy names stop
matching. The monitoring docs show a temporary `metric_relabel_configs`
bridge.
Closes AIGOV-447.
<details>
<summary>Implementation plan</summary>
| File | Change |
|---|---|
| `cli/server.go` | Remove the `TODO(deprecation)` comment; register
embedded gateway metrics with
`prometheus.WrapRegistererWithPrefix(aibridgemetrics.PrometheusMetricPrefix,
...)`. |
| `enterprise/cli/aibridgeproxyd.go` | Same swap with the
`coder_ai_gateway_proxy_` prefix; drop the unused `prometheusmetrics`
import. |
| `coderd/prometheusmetrics/metricalias.go`, `metricalias_test.go` |
Delete; the two call sites above were the only callers. |
| `scripts/metricsdocgen/scanner/prefix.go` | Remove
`NewMetricAliasRegisterer` from `prefixFuncs`. |
| `scripts/metricsdocgen/scanner/scanner_test.go` | Remove the alias
fixture and the legacy-alias assertion; the remaining literal-prefix
case already covers plain `WrapRegistererWithPrefix`. |
| `scripts/metricsdocgen/README.md` | Drop the
`NewMetricAliasRegisterer` mention. |
| `docs/ai-governance/ai-gateway/monitoring.md` | State that legacy
names are no longer emitted as of v2.40; reword the relabel snippet as a
temporary bridge and explain its cost. |
| `docs/ai-governance/ai-gateway/rebranding-migration.md` | Mark metrics
as "No (removed in v2.40)"; rewrite the Metrics section; remove the
now-obsolete "dropping the old names" section. |
Release note: the `!` title applies `release/breaking`, so the change is
listed under breaking changes in the v2.40 release notes.
Verification: `make gen` (no metrics reference changes), `make lint`,
`pnpm run format-docs`, `pnpm run lint-docs`, doc-check and test-audit
self-review, and targeted tests for `enterprise/cli` (AI Gateway /
proxy), `cli` (AI Gateway), `scripts/metricsdocgen`, and
`coderd/prometheusmetrics`.
</details>
Code written by @mollymaefraser, with Coder Agents used as a guide.
Add an opt-in coderd_workspace_last_used_timestamp_seconds gauge that
reports workspaces.last_used_at, the value the dormancy threshold
measures inactivity from. To enable it, set
--prometheus-collect-workspace-stats or
CODER_PROMETHEUS_COLLECT_WORKSPACE_STATS.
A separate WorkspaceStats collector refreshes the gauge once a minute
in the background from a new GetWorkspaceLastUsedForMetrics query. Each
series has a workspace_id label, so a workspace that reuses a deleted
workspace's name starts a new series. Workspaces owned by deleted users
are excluded.
Closes CLIENT-1065
Adds `--ai-gateway-embedded-enabled` /
`CODER_AI_GATEWAY_EMBEDDED_ENABLED` (default `true`) so standalone-only
deployments can stop `coderd` from starting the in-memory AI Gateway.
`CODER_AI_GATEWAY_ENABLED` stays the global switch: the AI Gateway
entitlement, `/api/v2/ai-gateway/serve`, and cost-control metrics still
depend only on it. The embedded daemon now starts only when both options
are `true`.
| `ENABLED` | `EMBEDDED_ENABLED` | Result |
|---|---|---|
| `true` | `true` | Unchanged |
| `true` | `false` | No embedded gateway; standalone gateways keep
working |
| `false` | any | No AI Gateway |
Refs AIGOV-514
## Open questions for reviewers
- **Coder Agents does not work with the embedded gateway disabled.**
`chatd` still starts, but model calls fail with "AI Gateway transport
factory is not configured" because there is no in-process transport.
AIGOV-597 (routing Agents through standalone gateways) was canceled. The
docs and option help state that Agents requires the embedded gateway. Is
that acceptable, or should `chatd` also be turned off in this mode?
- **Option name:** the ticket proposed
`CODER_AI_GATEWAY_EMBEDDED_ENABLED` tentatively. OK to keep?
## Verification
```
$ go test ./cli/ -run 'TestServer_AIGateway' -count=1
ok github.com/coder/coder/v2/cli 0.518s
$ go test ./enterprise/coderd/ -run 'TestAIGatewayServe' -count=1
ok github.com/coder/coder/v2/enterprise/coderd 0.755s
```
`TestServer_AIGatewayEmbedded` fails when the new check is removed.
<details>
<summary>Implementation notes</summary>
- `codersdk/deployment.go`: new option and
`AIBridgeConfig.EmbeddedEnabled` field; updated `--ai-gateway-enabled`
description.
- `cli/server.go`: outer `if Enabled` sets up cost-control metrics and
the blocked-users collector (the standalone `/serve` endpoint uses these
too). Inner `if EmbeddedEnabled` creates the daemon, registers the
in-memory handler, and adds its cleanup.
- `cli/server_test.go`: `TestServer_AIGatewayEmbedded` covers the three
modes by checking for the embedded daemon's startup log.
- `enterprise/coderd/aibridgeserve_test.go`:
`TestAIGatewayServeEmbeddedDisabled` checks the entitlement and `/serve`
still work with embedded disabled.
- `docs/ai-governance/ai-gateway/standalone.md`: new "Disable the
embedded gateway" section.
- Remaining files are generated by `make gen`.
- Found during review: keeping metrics inside the embedded check would
have dropped cost-control metrics for standalone gateways.
</details>
---
Generated with Coder Agents; reviewed, revised, and verified by
@mollymaefraser.
`coder server` closes the coderd API only in its graceful shutdown
sequence. Every startup error returned after `newAPI` (creating
aibridged, registering Prometheus metrics, writing the config URL,
creating provisioner daemons, notifying systemd) skipped that close and
leaked the whole coderd instance: tailnet and wireguard-go, gvisor,
metricscache, dbrollup, cryptokeys, chatd, and pubsub queues.
`TestServer/Logging` hits this when test cleanup cancels startup while
aibridged is being created. The test helper ignores the resulting
`context canceled` error, so only the package-level goleak check in
`cli` fails.
The fix closes the API in a deferred `sync.OnceValue` registered right
after `newAPI`, and the shutdown sequence calls the same function. The
guard is needed because the closer is not safe to call twice: the AGPL
API returns "API already closed", and the enterprise closer closes each
component again. The normal shutdown path is unchanged.
## Testing
The new `TestServer/StartupErrorNoLeak` fails startup after coderd is
created by putting a directory at the config URL path.
- Red (test only): `go test ./cli -run
'^TestServer$/^StartupErrorNoLeak$' -count=1` exited 1 in 3 of 3 runs
with `goleak: Errors on successful test run: found unexpected
goroutines` (coderd frames) while the subtest passed.
- Green (with the fix): the same command passed in 3 of 3 runs.
- `TestServer/Logging` alone, 20 runs from one test binary: 20 of 20
passed with no goleak failure, including 3 runs that hit the `create
aibridged` startup error.
- `go test ./cli -count=1` passed in 4 of 4 runs, and `go vet ./cli` is
clean.
Found in the CI of #30272 (`test-go-pg (ubuntu-latest)`).
> Xum, an AI coding agent, implemented, tested, and opened this PR on
behalf of @ibetitsmike.
coderd emitted the structured interception records as they arrived over
DRPC, so
a record the gateway never sends cannot be exported.
- Add `--ai-gateway-structured-logging-source` (`coderd`, `gateway`,
`both`) and
teach the gateway's `LogRecorder` to emit the same format where the
records
originate. Gateway lines omit `thread_parent_id` and `thread_root_id`,
which
are resolved by a database lookup only coderd can perform.
- Reaching the serving pool required repairing the pool options: they
were
derived at construction sites and discarded, so the server always built
its
pool with the defaults. They are now derived from the deployment
configuration
in one place and passed to the pool that serves requests. The server
reports
those options, and its ownership of the `key_pool_state` collector, so a
second
registration cannot silently reappear and panic startup.
- The standalone gateway inherits both structured logging options, since
it is
now one of the processes that can emit.
This is the keystone of the stack: the flag, the mechanism that carries
it and
the behavior it selects land together, so
`TestAIGatewayStart_InheritedOptions`
and the gateway help output stay truthful at every commit.
<details>
<summary>Why the stack is ordered this way</summary>
The final design is a chain of recorder decorators:
+---------------------+
| AsyncRecorder | all records except the interception start
+---------------------+
| LogRecorder | emit the structured record
+---------------------+
| ValidatingRecorder | refuse malformed records
+---------------------+
| caller middleware | record policy, e.g. drop content
+---------------------+
| TraceRecorder |
+---------------------+
| DRPCRecorder | acquire client, send to coderd
+---------------------+
Ordering constraints:
- The record type constants (1) are the vocabulary both emitters and the
validator use, so they come first.
- Validation (5) rejects unset timestamps, so records must already be
stamped
where they are created (3). Stamping at creation leaves
`WrappedRecorder`
doing nothing but client acquisition, which is why it is folded into
`DRPCRecorder` (4) and not before.
- coderd's server-time fallback (2) is what keeps a gateway of an older
version
from persisting zero times once the gateway owns stamping (3).
- PRs 1 to 5 change no user-visible surface. Each deployment option then
lands
with the code that reads it: the log source with gateway-side emission
(6),
content recording with the filter that drops the records (7). Pool
options are
plumbed to the serving pool in 6, where the first option that needs them
arrives, rather than as an unused hook beforehand.
</details>
---
🤖 Raised by [Coder Agents](https://coder.com) on behalf of @SasSwart.
Report session counts as a dynamic app-name map from the agent, the SSH CLI, and the workspace usage API, so a new IDE flows end to end without a schema, proto, or server change.
- Report through `Stats.session_counts` on Agent API v2.11. Older agents keep reporting through the deprecated count fields.
- Accept arbitrary names from `coder ssh --usage-app` and the workspace usage API. A name the registry does not recognize keeps its identity and is attributed to the `unknown` family.
- Add reconnecting PTY counts to the map without overwriting the counts SSH reported.
- Normalize and merge names before applying the deterministic 64-name cap, ordered by known apps, then count, then name. Overflow sums into `unknown`, and non-positive counts cannot cancel valid activity.
- Strip control characters, treat whitespace-only HTTP app names as absent, and require the exact environment variable prefix when you extract an SSH session type.
- Count overflow entries in `coderd_agentstats_session_counts_overflow_total` and keep raw payloads out of logs.
- Change the usage API's `app_name` to a string, and update the SDK types and API reference.
closes#27410
Makes the embedded NATS pubsub an opt-out feature instead of opt-in. The
`nats_pubsub` experiment is removed and replaced with a `no_nats_pubsub`
experiment, so NATS pubsub is now the default and can be disabled to
fall back to PostgreSQL pubsub.
NATS clustering requires this replica's routable address, resolved into
`clusterHost` from either `--cluster-host` (`CODER_CLUSTER_HOST`) or the
DERP relay URL. When neither is set (allowed when DERP is disabled on
the primary), NATS cannot form a cluster, so startup logs a big error
and falls back to PostgreSQL pubsub. The NATS cluster CA is only
minted/rotated when NATS clustering is actually active (experiment not
set and a cluster host is present).
<details>
<summary>Implementation notes</summary>
- `codersdk/deployment.go`: removed `ExperimentNATSPubsub`
(`nats_pubsub`), added `ExperimentNoNATSPubsub` (`no_nats_pubsub`,
display name "No NATS Pubsub").
- `cli/server.go`: NATS is used by default unless `no_nats_pubsub` is
enabled; if `clusterHost` is empty, log an error and keep the PostgreSQL
pubsub.
- `coderd/coderd.go`: gate NATS CA minting/rotation on
`!ExperimentNoNATSPubsub && ClusterHost != ""` (i.e. only when NATS
clustering is active).
- `enterprise/coderd/licenses.go`: updated a stale comment.
- `enterprise/coderd/coderd_test.go`: `TestMultiReplica_NATSPubsubPeers`
no longer sets the removed experiment.
- Regenerated `coderd/apidoc/docs.go`, `coderd/apidoc/swagger.json`,
`docs/reference/api/schemas.md`, `site/src/api/typesGenerated.ts`.
</details>
---
*This PR was generated by Coder Agents on behalf of @spikecurtis.*
PLAT-492: Log OAuth2 provider state at startup
Log whether the OAuth2 provider is enabled and name CODER_OAUTH2_PROVIDER_ENABLE.
When disabled, warn if OAuth2 applications are still registered and tell admins how to re-enable the provider.
Startup continues if the application lookup fails.
Add tests for enabled, disabled, registered-app, and query-error cases.
Manually verified upgrade behavior, route gating, token preservation, dashboard visibility, and all supported flag forms.
> AI tools were used to create this PR
Removes deprecated AI provider configuration through environment
variables, CLI flags, and YAML.
Deletion includes indexed providers and legacy OpenAI, Anthropic, and
Bedrock options under both AI Gateway and AI Bridge branding, eg.
`CODER_AI_GATEWAY_PROVIDER_<N>_*` or `CODER_AI_GATEWAY_ANTHROPIC_KEY`.
Removes the associated startup-seeding code.
Updates tests and documentation.
**Breaking change:** Remove [obsolete AI provider CLI flags and YAML
keys](https://coder.com/docs/ai-coder/ai-gateway/providers#database-management-of-providers)
before upgrading, otherwise Coder will fail to start with an `unknown
flag` or `unknown option` error. Provider environment variables are
ignored and no longer seed the database. Existing database-stored
provider configuration is preserved.
Routes all MCP config-directed traffic from coderd and chatd through a
shared SSRF-protected HTTP client, now that organization admins, not
only deployment admins, control MCP server URLs (#27942 and its stack
below).
## Summary
- Uses the [`coder/safedial`](https://github.com/coder/safedial)
library: it blocks private and special-purpose destinations at dial time
(validating resolved addresses at connection time so DNS rebinding
cannot bypass the check) and rejects cross-origin redirects.
- Covers the complete traffic surface: OAuth2 discovery, dynamic client
registration, code exchange, token refresh, revocation, and runtime MCP
connections from chatd.
- Deployments that intentionally host internal MCP servers opt in via
the new `--mcp-allowed-private-cidrs`
(`CODER_MCP_ALLOWED_PRIVATE_CIDRS`) option.
- Includes the deployment configuration surface, generated docs, CLI
goldens, TypeScript types, and regression coverage for each traffic
path.
## Merge window
This protection originally lived inside #27942 and was split out to keep
that diff reviewable. Until this PR lands, the stack below ships with
only main's existing discovery IP-range guard
(`CODER_MCP_OAUTH2_DISCOVERY_ALLOWED_IP_RANGES`), while org admins can
already point MCP configs at arbitrary URLs. This PR should merge
promptly after the stack below it.
Top of the MCP org-separation stack (CODAGT-711 -> CODAGT-717 audit ->
CODAGT-712 ACLs -> CODAGT-806 token RBAC -> CODAGT-714 org picker ->
this PR).
> Mux (AI agent) authored this PR on Mike's behalf.
<!-- mux-attribution: model=claude-fable-5 thinking=high -->
Closes
[AIGOV-568](https://linear.app/codercom/issue/AIGOV-568/notify-admins-about-unpriced-ai-models).
AI Bridge records token usage for a model with no price at a NULL cost,
so its spend is neither reported nor enforced against any budget. Until
now that only reached admins through an info log and a Prometheus
metric.
Owners now receive a weekly report listing models used without a price
in the past week. The report links to documentation explaining how Coder
calculates spend from the default price book and how owners can
configure missing prices with `coder exp ai-model-prices`.
## Screenshots
<img width="726" height="531" alt="image"
src="https://github.com/user-attachments/assets/168ae378-60cb-4c96-8f60-36c6edc0b268"
/>
<img width="470" height="417" alt="image"
src="https://github.com/user-attachments/assets/7dc71a1e-0c57-44f1-86f7-df800603c865"
/>
## Design
**Derived, not tracked.** The unpriced set is computed at report time
from interceptions that recorded token usage, joined against the current
`ai_model_prices` table. Requests that produced no token usage are
excluded.
**No new loop or table.** `reportUnpricedAIModels` is a sibling of
`reportFailedWorkspaceBuilds` in the existing report generator, reusing
its ticker and `notification_report_generator_logs`. Each report runs in
its own transaction under a separate advisory lock, so failures and lock
contention do not couple otherwise independent reports. The weekly
frequency is enforced by the persisted timestamp rather than by the
ticker, which restarts with the process and runs on a different phase in
each replica.
## Behaviour
| Situation | Outcome |
|---|---|
| Model used without a price | Listed in the next weekly report |
| Price set | Disappears from the next report |
| Still unpriced and still in use | Reported again each week |
| Model stops being used | Drops out of the report |
| Nothing unpriced | No notification; window still advances |
| More than 100 unpriced models | Top 100 by usage, with the total
reported alongside |
| openai-compat models | Excluded, they cannot be priced |
| Many replicas | Exactly one report per week |
| First ever run | Reports models used in the preceding week immediately
|
## Design choices
1. **openai-compat** is excluded because it cannot be priced and would
produce permanent, unactionable notifications.
2. **Token volume** orders the list but is not shown. The 100-model cap
therefore prioritizes models responsible for the most unpriced usage.
3. **Placement** remains in the generic report generator rather than
`aibridgedserver`. This keeps one periodic-report lifecycle at the cost
of coupling the notifications package to an AI cost-control query.
4. **No license gate.** The reporter runs even when AI Gateway is
disabled. With no recorded token usage the query is empty and no
notification is sent.
---
Created by Coder Agents on behalf of @evgeniy-scherbina.
Follow-ups from #27627
- Memoizes `Config.Git()` with a mutex so the provider's ETag response
cache survives across calls. Only successful construction is cached;
errors are retried.
- Moves the HTTP client onto `Config.HTTPClient`, wired through
`ConvertConfig`, so `Git()` no longer takes a per-call argument that
would be silently ignored after memoization.
- `newGitHub` and `newGitLab` now return `(Provider, error)`,
eliminating the typed-nil-interface class in `gitprovider.New` rather
than the single instance.
- Gates the 304 branch on a `haveCached` flag instead of a nil body
check.
- Only caches bodies that decode successfully, preventing poisoned
entries.
- Keys the response cache on the full token digest rather than a
truncated prefix.
- Tests added: `TestConfigGitMemoizesProvider`,
`TestConfigGitRetriesOnConstructorError`,
`TestGitLabConstructorErrorReturnsNilInterface`,
`TestResponseCacheStore`,
`TestConditionalRequestReuse/MalformedResponseNotCached`;
`TestConvertYAML/CustomScopesAndEndpoint` now asserts
`Config.HTTPClient` wiring.
Follow-ups tracked in #28139, #28140, #28141, #28142.
> 🤖 Generated by Coder Agents on behalf of @johnstcn.
When `ValidateToken` keeps a token because the external auth validation
endpoint was rate-limited (a `403` with rate-limit headers or a `429`),
it returns `valid=true` without provider confirmation. Previously this
happened silently, so operators couldn't tell a provider-confirmed token
from one kept optimistically during a rate limit.
This adds a `Logger` to `externalauth.Config` and emits a `Warn` (with
`provider_id`, `provider_type`, `status_code`, and `reason`) on those
rate-limit branches. It also adds a
`coderd_oauth2_external_requests_rate_limited_total{name, source,
status_code}` counter, incremented in the instrumented round tripper
whenever a provider returns a rate-limited response. The rate-limit
detection is the shared `xhttp.IsRateLimited` (in `coderd/util/xhttp`),
used by both the tripper and `ValidateToken` so the metric and the
validation decision share one definition; no extra wiring is needed
since `ValidateToken` already routes through the instrumented client
with `source="ValidateToken"`.
One deliberate behavioral change rides along: rate-limit detection now
also recognizes the unprefixed `RateLimit-Remaining` header (GitLab, and
the IETF draft rate-limit headers), so a `403` with
`RateLimit-Remaining: 0` is treated as optimistically valid where it was
previously treated as revoked. All other valid/invalid decisions are
unchanged. `TestValidateToken` asserts the warning's fields on the
rate-limited cases and no warning for revocations, `401`, and confirmed
responses; `promoauth` and `xhttp` tests cover the detector and the new
counter.
<details>
<summary>Manual testing</summary>
The signals fire on the external-auth status check (`GET
/api/v2/external-auth/{id}`), which calls `ValidateToken`. To force a
rate-limited response, point a provider's `validate_url` at a mock that
returns the rate-limit shape:
1. Run a mock returning `429` on one path and `403` +
`X-RateLimit-Remaining: 0` on another.
2. Start `coder server` with `--prometheus-enable` and external auth
providers whose `validate_url` point at those mock paths (e.g.
`CODER_EXTERNAL_AUTH_0_VALIDATE_URL=http://127.0.0.1:5599/429`).
3. Create a stored link, either complete the OAuth flow, or insert a row
into `external_auth_links` with a future `oauth_expiry` (token contents
are irrelevant; the mock rejects regardless).
4. `curl` the status endpoint with a session token, then check:
- coderd logs for the `Warn` (`reason=status_code` for `429`,
`reason=rate_limit_headers` for `403`),
- the metrics endpoint for
`coderd_oauth2_external_requests_rate_limited_total{...,status_code="429"|"403"}`.
Notes: `scripts/testidp -429` only rate-limits `/oauth2/userinfo`, not
the `/external-auth-validate/...` path, so it does not exercise this;
use a mock `validate_url`. The default Prometheus port `2112` may
already be taken on dogfood workspaces, set `CODER_PROMETHEUS_ADDRESS`
to a free port.
</details>
🤖 Generated with the help of Coder Agents on behalf of @jscottmiller.
## Description
Adds Prometheus metrics for AI budget cost control, emitted by the
aibridged server under the `cost_control` subsystem (full names are
prefixed `coder_ai_gateway_`).
- `blocked_requests_total` (counter) — labels: `group_id`
- `blocked_users` (gauge) — labels: `group_id`
- `unpriced_requests_total` (counter) — labels: `provider`, `model`
- `enforcement_duration_seconds` (histogram) — labels: `outcome`
## Changes
- Add `GetOverBudgetUsersPerGroup` query (plus dbauthz/dbmetrics/dbmock
wiring) to count over-budget users per effective group.
- Add a background collector that refreshes the `blocked_users` gauge on
an interval, started only when Prometheus is enabled.
- Wire `Metrics` through the aibridged server, coderd API,
`cli/server.go`, and the enterprise AI gateway handler; recording is
nil-safe when metrics are unset.
Closes
https://linear.app/codercom/issue/AIGOV-296/add-prometheus-metrics-for-cost-control
> [!NOTE]
> Initially generated by Claude Opus 4.7, modified and reviewed by
@ssncferreira
## Problem
On a fresh deployment with no custom GitHub OAuth app, Coder falls back
to the default Coder-managed GitHub app. That app can only see
organization memberships in organizations where it has been installed.
If `CODER_OAUTH2_GITHUB_ALLOWED_ORGS` is set but the app isn't installed
in the allowed organizations, the membership list comes back empty and
every login, including the first admin login, is rejected with a bare
"You aren't a member of the authorized Github organizations!" with no
hint about the actual cause. This leaves fresh deployments in an
apparently broken state.
## Fix
* Append a remediation hint to the login rejection when the default
provider is configured, pointing at the [app installation
page](<https://github.com/apps/coder/installations/select_target>) and
at configuring a custom GitHub OAuth app.
* Log a startup warning when the default provider is combined with
`CODER_OAUTH2_GITHUB_ALLOWED_ORGS`, listing the allowed orgs and the
install URL.
* Document the installation requirement next to the
`CODER_OAUTH2_GITHUB_ALLOWED_ORGS` step in the GitHub auth docs.
Access-control behavior is unchanged; the org check still rejects logins
as before, it just explains why and how to fix it.
## Testing
* New `TestUserOAuth2Github/NotInAllowedOrganizationDefaultProvider`
asserts the hint appears when `DefaultProviderConfigured` is set; the
existing `NotInAllowedOrganization` subtest asserts it does not leak
into the custom-app path.
Fixescoder/coder#17752
Adds `--aigateway-proxy-target` option to
`deploymentGroupAIGatewayProxy` that defines URL to which intercepted
requests should be forwarded to.
Forward URL used to be hardcoded to `coderAPI.AccessURL` pointing to
embedded Gateway. With addition of standalone AI Gateway this needs to
be configurable.
Renamed `aibridgeproxyd.Server.coderAccessURL` and `coderAccessPort` ->
`gatewayURL` and `gatewayPort` + option to better reflect reality.
flake-go keeps failing every `TestServer` subtest that boots `coder
server` with built-in PostgreSQL ([example
run](https://github.com/coder/coder/actions/runs/28685288282/job/85134336803))
with `no version found matching 13.21.0`, which is embedded-postgres's
error for any non-200 while downloading the Postgres binary archive from
Maven. The archive gets cached under the server's config root, which is
a fresh temp dir in every test, so with `test-count: 35` one flake run
downloads it dozens of times and Maven rate-limits the runner. Using an
external Postgres (like coder/terraform-provider-coderd#370 did) would
defeat the point, since these subtests exist to exercise the built-in
Postgres path.
Instead, `startBuiltinPostgres` now honors `EMBEDDED_PG_CACHE_DIR` as
the archive cache path (test runs only, data/runtime dirs stay
per-test), and flake-go.yaml wires in the existing `embedded-pg-cache`
actions the same way the Windows/macOS lanes in ci.yaml already do. All
iterations then share a single download, and usually zero once the
actions cache is warm. The upload step only saves on `main`.
This PR adds a new opt-in setting, `CODER_OIDC_REDIRECT_ALLOWED_HOSTS`,
that lets a single Coder deployment complete OIDC login on more than one
hostname. When the allowlist is non-empty, Coder picks the OIDC
`redirect_uri` based on the incoming request's Host header (validated
against the list) instead of always using the static URL derived from
`CODER_ACCESS_URL`. When unset, the (default) behavior is identical to
today.
The motivation is that a single Coder deployment is frequently reachable
via multiple hostnames - for example, an internal hostname for users on
a corporate VPN and a different hostname routed through a zero-trust
gateway for users off-VPN - but OIDC login today only works on whichever
single hostname `CODER_ACCESS_URL` points to, because the `redirect_uri`
sent to the IdP is fixed at server startup. Users who reach the
deployment on any other valid hostname can see the login page but fail
the OIDC callback, since the IdP redirects them back to a hostname they
can't reach (or whose cookies they don't have).
The provider type already lives authoritatively in ai_providers.type,
reachable on every active row through ai_provider_id, which the
chat_model_configs_ai_provider_required_when_active CHECK makes
mandatory. The stored provider string was a denormalized copy the system
kept in sync with a startup backfill and no longer needs.
Every surface now derives provider type from the linked ai_providers
row. Telemetry is the one exception: it keeps emitting provider, now
sourced from ai_providers.type via a JOIN, so the BigQuery column and the
Nexus dashboards that read it are unaffected. The experimental HTTP/SDK
response drops provider and makes ai_provider_id required, since those
endpoints return only active configs; consumers resolve provider type
from ai_provider_id and the AI providers listing.
This ships in a single release with no compatibility window: production
reads the table via SELECT *, so a pre-drop binary fails config reads the
moment the column is gone. Operators must scale to zero before upgrading,
and there is no rollback.
Closes CODAGT-599
Closes [AIGOV-455](https://linear.app/codercom/issue/AIGOV-455/extend-drpc-with-buildproviders).
## Why
The AI Gateway (`aibridged`) is being split into a standalone process that must not touch the database. `coderd` stays the source of truth and seeds the `ai_providers` / `ai_provider_keys` tables from the environment. This PR adds a DRPC call so the gateway fetches provider config from `coderd` instead of reading the DB, for both the embedded and standalone daemons.
## What
- **Proto:** new `ProviderConfigurator` service with a unary `GetAIProviders` RPC, plus `AIProvider` / `AIProviderBedrock` messages. `CurrentMinor` bumped to 1 (additive).
- **Server (`coderd/aibridgedserver`):** `GetAIProviders` runs a read-only `InTx` under `LockIDAIProvidersEnvSeed` so it never returns a mid-seed snapshot, reads providers (incl. disabled) plus keys for enabled ones, and maps to proto under `dbauthz.AsAIBridged`. Unmappable rows are skipped and logged; plaintext keys and Bedrock secrets are never logged.
- **Client:** `DRPCProviderConfiguratorClient` wired into the client union, `dialer.go`, and `CreateInMemoryAIBridgeServer`.
- **cli:** `BuildProvidersFromProto` maps the response through the existing DB-neutral `buildProvider`. A shared `poolRPCReloader` does the fetch/build/replace for both daemons: the embedded daemon reloads on every `ai_providers` change and fails startup if it cannot subscribe; the standalone gateway drives the same reloader once at startup, retrying until success and staying interruptible.
- **Dead code removed:** `BuildProvidersFromConfig`, `ProvidersFromConfig`, `AIProviderFromConfig`, and the DB-read `BuildProviders` path.
<!-- Authored by Coder Agents on behalf of @Emyrk. -->
Adds an opt-in `CODER_DANGEROUS_OIDC_EMAIL_FALLBACK` flag (alias
`--dangerous-oidc-email-fallback`) for IdP brokers that do not issue a
stable `sub` for the same user across connections.
Adds two nullable booleans to `telemetry.Deployment`:
- `SCIMEnabled`: `true` when `CODER_SCIM_AUTH_HEADER` is set.
- `SCIMUseLegacy`: `true` when `CODER_SCIM_USE_LEGACY` is set.
Both mirror `Deployment.IDPOrgSync`: nullable for backward
compatibility, and report configuration state rather than license
entitlement (#16323).
Lives on `Deployment` rather than `Snapshot` so the existing
`bqDeployment` table on `coder/coder-telemetry-server` gets two columns
instead of a new table.
`SCIMAPIKey` is annotated as a secret and is scrubbed by
`WithoutSecrets` before the config reaches telemetry, so
`DeploymentConfig.SCIMAPIKey` is always empty in production. The
booleans are pre-computed from the pre-scrub `DeploymentValues` in
`cli/server.go` and passed in via `telemetry.Options.SCIMEnabled` /
`SCIMUseLegacy`.
Pairs with
[coder/coder-telemetry-server#43](https://github.com/coder/coder-telemetry-server/pull/43),
which adds the matching `bqDeployment` columns and the manual BigQuery
`ALTER TABLE` step.
---
Generated by Coder Agents on behalf of @Emyrk.
## Description
Registers `/api/v2/ai-gateway/*` as the new API path for AI Gateway, replacing `/api/v2/aibridge/*`. Both prefixes share the same route builder (`aiBridgeRoutes`) backed by a single in-memory handler, so existing `/aibridge` endpoints continue to work. New endpoints must be registered on the enterprise API handler under `/api/v2/ai-gateway` only.
Swagger annotations now point to `/api/v2/ai-gateway` paths with a backward-compatibility note referencing `/aibridge`. The legacy `/aibridge` routes are skipped in the swagger documentation test.
## Changes
- Store one raw handler (`aiGatewayHandler`) instead of two prefix-stripped handlers
- Register `/ai-gateway` and `/ai-gateway/proxy` route aliases alongside legacy `/aibridge` routes
- Move `/aibridge/keys` to `/ai-gateway/keys`
- Update in-process transport to use `/api/v2/ai-gateway` prefix
- Update SDK client URLs and proxy forwarding URL
- Swap `@Router` and `@Tags` annotations from `aibridge`/`AI Bridge` to `ai-gateway`/`AI Gateway`
- Rename user-facing error messages from "AI Bridge" to "AI Gateway"
- Define consts for route prefixes (`AIGatewayRootPath`, `AIBridgeRootPath`)
- Update tests and comments to use new paths
Note: the following will be addressed in follow-up PRs:
- Frontend API URLs
- Frontend routes and redirects
- Dogfood main.tf updates
- Hand-written documentation URL updates
- aibridge internal comments and nits
- Scale tests path updates
Refs https://linear.app/coder/issue/AIGOV-230
> Generated with the assistance of Coder Agents (@ssncferreira)
In our codebase we have an existing convention of using
`flag.Lookup("test.v")` instead of `testing.Testing()`. This avoids
pulling in the entire `testing` package. Another consequence: some of
our custom linters trigger upon import of the `testing` package which
can lead to unexpected linter errors.
> AI Tools where used in this request.
Registers `coder_aibridged_*` and `coder_aibridgeproxyd_*` metrics under
new prefixes: `coder_ai_gateway_*` and `coder_ai_gateway_proxy_*`.
Old prefix is still exported. Will be removed in later release.
Also updated the `metricsdocgen` static fixture. Added 4
previously-undocumented metrics `key_pool_state`,
`key_pool_state_transitions_total`, `key_pool_exhaustions_total`,
`key_pool_failover_attempts` added the `client` label to the existing
interception, prompt, and token counter samples.
Updated AI Gateway documentation.
Adds `ai` to sqlc's `gen.go.initialisms` in `coderd/database/sqlc.yaml`
so the generated DB code follows Go's initialism convention. Adds the
matching `ai` -> `AI` case to the dbgen PascalCase helper
(`scripts/dbgen/main.go`) so the corresponding `dbmem` / mock
identifiers stay in sync. `make gen` regenerates the rest; hand-written
call sites that consume DB-generated identifiers
(`enterprise/audit/table.go`, `coderd/database/modelmethods.go`,
`enterprise/coderd/aigatewaykeys.go`, `coderd/database/dbauthz/*`, etc.)
are updated to match.
Scope is deliberately limited to the database layer:
- `coderd/rbac/*` (resource and scope generators) is untouched —
`ResourceAi*` / `ScopeAi*` constants stay on main's casing.
- `codersdk/*` (Go SDK) is untouched — `codersdk.ResourceAi*` /
`codersdk.APIKeyScopeAi*` constants stay on main's casing, so external
Go SDK consumers see no source-level break.
- `Aibridge*` identifiers (one SQL token `aibridge`, not `ai_bridge`)
are out of scope.
On-the-wire values are unchanged: enum strings, RBAC resource type
strings, API key scope strings, and JSON tags all stay the same. The
HTTP/JSON surface is unaffected.
Refs:
[AIGOV-369](https://linear.app/codercom/issue/AIGOV-369/change-ai-references-in-coderddatabasemodelsgo-to-ai)
🤖 Generated with [Coder Agents](https://coder.com)
Fixes CODAGT-548
Adds two idempotent startup backfills run after `newAPI():
- `BackfillBedrockProviderType`: promotes `ai_providers` rows from
`type=anthropic` with Bedrock settings to `type=bedrock`.
- `BackfillChatModelConfigProviderStrings`: fixes stale
`chat_model_configs.provider = "anthropic"` strings on rows whose linked
provider was just promoted.
- `UpdateAIProvider` query now also writes the `type` column, so the
fix persists on any subsequent PATCH.
> 🤖 Generated by Claude with oversight from a human.
Reverts coder/coder#26239
We cannot disable a feature which was previously enabled; this is a BC
break.
This is also using `AIGatewayRoutingEnabled` which will be removed in
the next release.
Problem: CODER_AI_GATEWAY_ENABLED defaulted to true, which both started
the in-memory gateway and enabled the licensed FeatureAIBridge. As a
result, deployments that never configured AI Gateway saw a spurious "AI
Governance add-on is required" warning whenever they had an older
(non-add-on) Premium license, since the feature was enabled-and-entitled
by default.
Fix: Decouple "external AI Gateway API enabled" from "in-memory daemon
running," so the external/licensed surface is off by default while Coder
Agents retain access by default.
- Adds server-side and client-side validation for
CODER_CONFIGSSH_HOSTNAME_SUFFIX and CODER_SSH_CONFIG_OPTIONS.
- **Server-side breaking change:** invalid values for either of these will cause `coderd` to exit with an error.
- Client-side: `coder config-ssh` will exit with an error if it detects invalid config.
- Adds tests for the above
Local smoke-testing: ran `develop.sh --env-file <path to an env file
containing badness>`. Validated that server startup failed as expected.
> 🤖 Generated by Coder Agents with supervision from a human.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
## Description
This PR adds Prometheus metrics for aibridge's API-key failover, giving visibility into key pool health and failover behavior per provider.
The following metrics are introduced:
- **`key_pool_state`** (gauge): number of keys currently in each state (`valid`, `temporary`, `permanent`) per provider, sampled at scrape time.
- **`key_pool_state_transitions_total`** (counter): key state transitions during failover, labeled by `reason` (`rate_limited`, `unauthorized`, `forbidden`).
- **`key_pool_exhaustions_total`** (counter): times a pool ran out of usable keys, labeled by `outcome` (`rate_limited`, `auth_failed`).
- **`key_pool_failover_attempts`** (histogram): keys attempted before success or exhaustion (per interception for bridged requests, per request for passthrough).
## Changes
- Moves `MarkKeyOnStatus` and key-pool error handling onto `*keypool.Pool`.
- Attaches metrics to each provider's key pool at install time, on construction and on provider reload.
- Adds a scrape-time state collector and a `KeyPools()` accessor on the bridge pool to feed it.
- Tracks per-request key attempts in the bridged and passthrough failover paths.
- Adds test coverage for the new metrics across the keypool unit tests, the bridged intercept failover tests, and the passthrough failover test.
Closes https://github.com/coder/internal/issues/1447
Closes https://linear.app/codercom/issue/AIGOV-198/aibridge-key-failover-observability
> [!NOTE]
> Initially generated by Claude Opus 4.7, modified and reviewed by @ssncferreira