Files
coder/docs/admin/integrations/prometheus.md
T
Nick VigilanteandBen Potter 1baff57cb0 fix: resolve docs and code conflicts found in the main docs audit (#32158)
## Summary

Addresses 106 of the 113 conflicts between the docs and the code that
the November audit of `main` found: 102 are fixed and 4 have an interim
docs fix. Every finding was re-checked against the code before it was
fixed. Most fixes are docs-only. Three are small code fixes where the
docs described the intended behavior and the code was wrong, each in its
own commits so it can be split out. No change in this PR alters template
build or agent behavior.

## Changes

Fixes by area (finding counts): templates 28, install 13, integrations
and monitoring 13, auth and security 12, networking 11, Coder Agents 9,
AI governance 7, reference 7, workspaces 6.

Code fixes. Each needs a look from the code owner. If an owner prefers
not to take one, drop its commits and I will change the docs to describe
the current behavior instead.

| Commits | Change | Risk to check |
|---------|--------|---------------|
| `fix: keep built-in listener defaults in packaged coder.env`, and
three follow-up commits | `coder.env` comments out `CODER_HTTP_ADDRESS=`
and the three `CODER_TLS_*` lines, so an untouched package install
listens on `127.0.0.1:3000` as the docs say. Only the
`CODER_HTTP_ADDRESS=` line changed behavior. The TLS options default to
empty, so commenting them out is for consistency. The Lima examples
rewrite the commented line and fail if the line is not found. A test
compares every option that the file sets against the built-in defaults.
| The deb and RPM packages were not built or installed. Every released
package through v2.39.0 ships the empty line, and upgrades keep an
existing file (`config\|noreplace`), so the upgrade pages now say how to
fix it. |
| `fix(cli/clilog): accept log sinks that are all disabled`, and three
follow-up commits | `--log-human=/dev/null` with no other log location
no longer fails with "no loggers provided". It is the documented way to
disable logging. An empty location still fails with that error when no
other sink is set, as before. | Behavior change: with `/dev/null`, the
server and the provisioner daemon can start with no log output. |
| `fix(cli): detect CODER_PROC_PRIO_MGMT by presence`, and a comment
follow-up | Log line only. With the variable set to an empty value, the
agent now logs that process priority management is enabled, which is
what `agentexec` already does. No detection or behavior change. | Low. |

Two option descriptions in `codersdk/deployment.go` are corrected, with
the CLI reference, configuration reference, and golden files
regenerated: the default quiet hours schedule (the max deadline is the
quiet hours window on the next day that the template's autostop
requirement applies) and `--experiments` (`*` opts in to the experiments
marked safe, also in `enterprise/cli/provisionerdaemonstart.go`).

### Code fixes that were reverted after review

Two code fixes from the first version of this PR are reverted in the
branch. The net diff has no change under `agent/` or `coderd/`. The docs
now describe what the code does, and both issues stay open for the
owner.

- **`error` merge strategy for `coder_env` (DOCS-1669).** The fix made
`merge_strategy = "error"` fail whenever a name was set twice, whatever
the resource order. Existing template versions that pair an `error`
entry with a later `replace` or `append` entry would start to fail every
build after `terraform apply`. That is a breaking change that needs its
own PR and release note. The fix and its tests are in `dff8e7c6ce` and
can be cherry-picked. `environment-variables.md` now says the check runs
only when the `error` resource is processed.
- **Nice value before the capability drop (DOCS-1687).** The fix let
`coder agent-exec` apply a negative nice value by calling `setpriority`
before it drops effective capabilities. The current order is deliberate:
with the fix, any local process could run the agent binary with
`--coder-nice=-20` when the binary has the `CAP_SYS_NICE` file
capability. `process-priority.md` now says a nice value below the
agent's own is not applied, and no longer tells admins to grant
`CAP_SYS_NICE`.

## Not included

Seven findings are left out.

- DOCS-1629: covered by #31639.
- DOCS-1703 and DOCS-1724: covered by #31980.
- DOCS-1584, DOCS-1683, DOCS-1685, DOCS-1686: each needs an owner
decision, and no neutral docs wording exists until that decision is
made. The validation comment on each issue names the decision.

Four fixes are interim, so their issues stay open:

- DOCS-1600: the AI Gateway pages now agree that the legacy metric names
are deprecated and "will be removed in a future release". The code still
emits them, and its TODO names v2.37, which has shipped. The owner still
needs to set the removal release.
- DOCS-1689: the page now says **New Template** opens the builder only
for users with the Owner role, because the page gates on read access to
the deployment config. Whether Template Admins should get the builder is
still open.
- DOCS-1669 and DOCS-1687: docs-only for now. Refer to [Code fixes that
were reverted after
review](#code-fixes-that-were-reverted-after-review).

## Review notes

- Claims that describe product intent and need an owner's yes: the
Premium qualifier on per-organization workspace sharing (the API route
has no license check, the dashboard needs multiple organizations),
`organization` port sharing in OSS, the telemetry field list, and the
`1-5` cron range in the prebuilds schedule example.
- Not run end to end: the PostgreSQL TLS procedure on a cluster, the
OpenShift and air-gapped examples, the BSD `date` and `tar` forms on
macOS, the Windows command on the multiple Kubernetes clusters page, and
the keep-alive script in `docs/reference/index.md` (only its comparison
logic was run, against sample values). Nothing was checked on a live
deployment.
- Likely merge conflicts with open PRs: #31990 (`mcp-server.md`,
`feature-stages.md`, `docker-in-workspaces.md`), #31982
(`workspace-management.md`), #31980 (`health-check.md`), #31977
(`manage-provisioner-jobs.md`), #31869 (`airgap.md`), #30486
(`add-envbuilder.md`), and #31326 and #31531 (`codersdk/deployment.go`
and generated files).

## Testing

- `make pre-commit` (gen, fmt, lint, slim build), before the PR opened
and again after the review changes.
- `go test` for `./cli/clilog/...`, `./codersdk/`, `./cli/`
(`TestServer`, `TestAgent`, and golden files), and `./enterprise/cli/`
(provisioner, server, and help golden files). `./agent/agentexec/...`
and `./coderd/provisionerdserver/...` pass after the reverts. Linux
only. CI covers macOS and Windows.
- The `coder.env` test fails against the file on `main` and passes
against the new one. It reads the file with `godotenv`, which accepts
CRLF line endings.
- An empty `CODER_HTTP_ADDRESS=` was confirmed end to end with a full
build of the server (output below).
- The new `sed` and `grep` lines in the Lima examples were run with GNU
tools against the old and the new form of `coder.env`. The Lima VMs were
not started.
- `--log-human=/dev/null` and `CODER_LOGGING_HUMAN=` were confirmed with
the same build (output below).
- `coder agent-exec --coder-nice=-5 -- echo` from a full build, as a
non-root user: `coder-agent: failed to adjust niceness to -5 ...:
permission denied` on stderr, the command output on stdout, exit code 0.
- `postgres-ssl.md` rendered with the GitHub Markdown API: two ordered
lists with three and five items, where `main` has six lists.
- markdownlint, the table formatter, `make lint/emdash`, `make
lint/docs-html`, `make lint/docs-manifest`, `make lint/docs-redirects`,
`make lint/style-claims`, and Vale on the changed lines.
- A doc-check self-review of the docs diff by four independent
reviewers. It found six problems, which are fixed. The accuracy and
coverage lenses ran on every area. The language and structure lenses ran
in full on only part of the diff.

<details>
<summary>Test and terminal output for the code fixes (Linux)</summary>

```console
$ go test ./codersdk/ -run TestPackagedCoderEnvKeepsDefaults -count=1 -v
--- PASS: TestPackagedCoderEnvKeepsDefaults (0.00s)
ok  	github.com/coder/coder/v2/codersdk	0.071s

$ go test ./cli/clilog/ -run TestBuilder -count=1 -v
--- PASS: TestBuilder (0.00s)
    --- PASS: TestBuilder/NoSinks (0.00s)
        --- PASS: TestBuilder/NoSinks/JSONDevNull (0.00s)
        --- PASS: TestBuilder/NoSinks/StackdriverDevNull (0.00s)
        --- PASS: TestBuilder/NoSinks/NoConfiguration (0.00s)
        --- PASS: TestBuilder/NoSinks/HumanDevNull (0.00s)
    --- PASS: TestBuilder/WithHuman (0.00s)
    --- PASS: TestBuilder/WithFilter (0.00s)
    --- PASS: TestBuilder/WithJSON (0.00s)
    --- PASS: TestBuilder/Verbose (0.00s)
    --- PASS: TestBuilder/FromDeploymentValues (0.01s)
        --- PASS: TestBuilder/FromDeploymentValues/Defaults (0.00s)
        --- PASS: TestBuilder/FromDeploymentValues/EmptyHumanEnv (0.00s)
        --- PASS: TestBuilder/FromDeploymentValues/Override (0.00s)
ok  	github.com/coder/coder/v2/cli/clilog	0.171s

$ TMPDIR=$HOME/.cache/agentexec-tmp go test ./agent/agentexec/... ./coderd/provisionerdserver/... -count=1
ok  	github.com/coder/coder/v2/agent/agentexec	4.236s
ok  	github.com/coder/coder/v2/coderd/provisionerdserver	2.404s

# Full build of the server (go build ./cmd/coder). An empty value stops it before it connects to the database.
$ CODER_HTTP_ADDRESS= coder server --postgres-url 'postgres://127.0.0.1:1/x?sslmode=disable'
Encountered an error running "coder server", see "coder server --help" for more information
error: TLS is disabled. Enable with --tls-enable or specify a HTTP address

# With the variable unset, the server passes that check and goes on to the database.
$ coder server --postgres-url 'postgres://127.0.0.1:1/x?sslmode=disable'
2026-11-13 02:19:41.901 [warn]  ping postgres: retrying  error="dial tcp 127.0.0.1:1: connect: connection refused"  try=11
Encountered an error running "coder server", see "coder server --help" for more information
error: connect to postgres: connect to postgres: unable to connect after 11 tries; last error: dial tcp 127.0.0.1:1: connect: connection refused

# An empty log location with no other sink still stops the server.
$ CODER_LOGGING_HUMAN= coder server --postgres-url 'postgres://127.0.0.1:1/x?sslmode=disable'
Encountered an error running "coder server", see "coder server --help" for more information
error: make logger: no loggers provided, use /dev/null to disable logging

# /dev/null passes the logger check. The server prints no log lines and goes on to the database.
$ coder server --log-human=/dev/null --postgres-url 'postgres://127.0.0.1:1/x?sslmode=disable'
Encountered an error running "coder server", see "coder server --help" for more information
error: connect to postgres: connect to postgres: unable to connect after 11 tries; last error: dial tcp 127.0.0.1:1: connect: connection refused
```

</details>

## Linear

Fixes DOCS-1583, DOCS-1585, DOCS-1586, DOCS-1587, DOCS-1588, DOCS-1589,
DOCS-1590, DOCS-1591, DOCS-1592, DOCS-1593, DOCS-1594, DOCS-1595,
DOCS-1596, DOCS-1597, DOCS-1598, DOCS-1599, DOCS-1601, DOCS-1602,
DOCS-1603, DOCS-1604, DOCS-1605, DOCS-1606, DOCS-1607, DOCS-1608,
DOCS-1609, DOCS-1614, DOCS-1618, DOCS-1619, DOCS-1620, DOCS-1624,
DOCS-1635, DOCS-1639, DOCS-1644, DOCS-1651, DOCS-1652, DOCS-1653,
DOCS-1654, DOCS-1655, DOCS-1656, DOCS-1657, DOCS-1658, DOCS-1659,
DOCS-1660, DOCS-1661, DOCS-1662, DOCS-1663, DOCS-1664, DOCS-1665,
DOCS-1666, DOCS-1667, DOCS-1668, DOCS-1670, DOCS-1671, DOCS-1672,
DOCS-1673, DOCS-1674, DOCS-1675, DOCS-1676, DOCS-1677, DOCS-1678,
DOCS-1679, DOCS-1680, DOCS-1681, DOCS-1682, DOCS-1684, DOCS-1688,
DOCS-1690, DOCS-1691, DOCS-1692, DOCS-1693, DOCS-1694, DOCS-1695,
DOCS-1696, DOCS-1697, DOCS-1698, DOCS-1699, DOCS-1700, DOCS-1701,
DOCS-1702, DOCS-1704, DOCS-1705, DOCS-1706, DOCS-1707, DOCS-1708,
DOCS-1709, DOCS-1710, DOCS-1711, DOCS-1712, DOCS-1713, DOCS-1714,
DOCS-1715, DOCS-1716, DOCS-1717, DOCS-1718, DOCS-1719, DOCS-1720,
DOCS-1721, DOCS-1722, DOCS-1723, DOCS-1725, DOCS-1726, DOCS-1727

Part of DOCS-1600, DOCS-1689, DOCS-1669, DOCS-1687

<details>
<summary>Decision log</summary>

### Scope

- One PR for all findings, as agreed with the reviewer. The code fixes
are separate commits so an owner can ask for any of them to move to its
own PR. Nothing in the PR is a breaking change, so the title has no `!`.
- A finding was skipped only when an open PR already covers it, or when
every available wording would assert an undecided product behavior.
- Each validation comment on the Linear issues records the evidence, the
corrections to the original finding, and what was not verified.

### Docs or code

- The default was a docs-only fix that describes what the code does.
- A code fix was chosen when the docs describe a behavior the code
plainly intended and the code change is small and testable: the packaged
env file, the log sink check, and the log line for the environment
variable presence check.
- The `error` merge strategy and the nice value ordering started as code
fixes and went back to docs-only after review. The first breaks existing
template versions. The second weakens a deliberate security ordering.
- The SCIM `userName` finding (DOCS-1609) took the docs path. The
handler is unchanged.

### Wording choices worth a second look

- `feature-stages.md` says `--experiments=*` enables only the
experiments marked safe, and points to the "Get safe experiments" API.
It does not list the other experiments, because the list would drift and
those experiments are hidden on purpose. The `--experiments` flag help
now says the same.
- `agent-metadata.md` replaces the writes-per-second formula with a
description of the batcher (each server replica writes one statement
every 5 seconds, or sooner at 500 entries).
- `extending-templates/index.md` no longer calls `startup_script`
deprecated. The provider documents it as an alias for a `coder_script`.
- `postgres-ssl.md` separates client verification from server
verification, so the server verification steps work alone.
- `rpm.md`, `upgrade.md`, and the 2.29 to 2.34 ESR upgrade page explain
the empty `CODER_HTTP_ADDRESS=` line that older packages shipped,
because upgrades keep the old file. The behavior changed in v2.34.0 with
the serpent upgrade in #24926.
- The keep-alive script in `docs/reference/index.md` reads the current
deadline and skips the request when it is already later. The comparison
is on text, so it applies only when the API returns a UTC timestamp,
which needs the database to use the UTC time zone. Otherwise the script
sets the deadline on every pass, as it did before, and the page says so.
- `external-auth.md` anchors the new GitHub Enterprise regex example.
The older unanchored examples on that page and on `git-providers.md` are
not changed here.

### Self-review findings that were rejected

- "The builder opens for Template Admins": the Templates page needs a
successful deployment config read, which the Template Admin role does
not have.
- "Organization Admins can set AI model prices": the price write
authorizes against a site-level object, which organization-scoped grants
do not match.
- "The provisioner chart sets no default resources":
`helm/libcoder/templates/_coder.yaml` applies 2000m and 4096Mi when
`coder.resources` is empty.

### Automated review, round 1

The `coder-agents-review` app left 45 findings on `8853332`. Fixed: the
two reverts above, the anchored regex, the `coder.env` follow-ups (Lima
examples, upgrade notes, a comment in the file, and a test that covers
every option in the file), the clilog change, the two option
descriptions, and the docs findings (CORS behavior, token lifetime
example, keep-alive script, agent metadata, refresh token errors,
`switch` form type, `dbcrypt` skip list, and a stray line in a YAML
example).

Declined, with the reason on each thread:

- Refactoring the `agentexec` logging and renaming its log field: not
related to a docs conflict.
- Rewording the "process priority management not enabled (linux-only)"
log message: the docs quote the message, and the wording predates this
PR.
- Problems in code and docs that this PR does not change (the
`Capabilities` subtest, the Helm values comment, the AWS provider path
in `airgap.md`, and others in the review's out-of-scope list).

### Automated review, round 2

The app left 14 new findings on `04925bd`, and accepted the three
declined threads on `cli/agent.go`. All 14 are fixed. The main one: the
round 1 change that removed the "no loggers provided" error went too
far, because an empty log location would then start a server with no
logs. The error is back for every case except `/dev/null`.

Reported by the app as out of scope and not changed: the swagger
annotation for `GET /api/v2/experiments/available` declares an array,
but the handler returns an object with a `safe` array. Three Storybook
fixtures keep the old `--experiments` description.

### Automated review, round 3

The app confirmed that all round 2 findings are fixed, and left 10 small
findings on `f55cb6d` (8 P4, 2 Nit). All are fixed: the TLS condition
and the release range in the `coder.env` notes, one place for the
`/dev/null` rule in `clilog`, two test cases, a clearer test failure,
and the CPU example in `agent-metadata.md`.

### Automated review, rounds 4 and 5

Round 4 left one P4 on `8fffb38` (name the `--trace` option in the
logging note), which is fixed. Round 5 on `2ce754e` reports no new
findings and says every earlier finding is fixed or closed. The app did
not post an approval: it leaves its out-of-scope items and the code
fixes to a human reviewer. Two threads on `cli/agent.go` stay open on
purpose for the code owner.

### Doc-check review

The doc-check workflow left 12 comments on `2ce754e` (10 blocking, 2
suggestions). All are fixed in `c888885`: "refer to" or "visit" in place
of "see" for cross-references on the lines this PR added, the **New
template** label as the dashboard shows it, one instruction per step in
the server verification procedure of `postgres-ssl.md`, and two wording
changes in `process-priority.md`. The `CAP_SYS_NICE` sentence keeps a
short reason, which differs from the suggested text.

The `coder-agents-review` app then reviewed that commit (rounds 6 and
7). Round 6 left two P4 findings, fixed in `30b2c79`: the `failed to
adjust niceness` message is described once, in the troubleshooting
entry, with where it appears, and the client verification steps in
`postgres-ssl.md` render as one numbered list. Round 7 reports no new
findings. `2a6035c` then softens one sentence from that fix, because the
agent logs the stderr of devcontainer CLI commands. The app was not run
again on that commit.

### Adjacent gaps found and not fixed here

- `add-envbuilder.md` has the same `zip` command that
`template-from-scratch.md` had.
- `template-from-scratch.md` leaves the reader in `build/` after `cd
$_`, before a step that runs from `template-tour`.

</details>

---

_Generated by Coder Agents on behalf of @nickvigilante._

---------

Co-authored-by: Ben Potter <me@bpmct.net>
2026-11-13 21:34:59 +00:00

332 KiB
Raw Blame History

title
title
Prometheus

Coder exposes many metrics which can be consumed by a Prometheus server, and give insight into the current state of a live Coder deployment.

If you don't have a Prometheus server installed, you can follow the Prometheus Getting started guide.

Enable Prometheus metrics

Coder server exports metrics via the HTTP endpoint, which can be enabled using either the environment variable CODER_PROMETHEUS_ENABLE or the flag --prometheus-enable.

The Prometheus endpoint address is http://localhost:2112/ by default. You can use either the environment variable CODER_PROMETHEUS_ADDRESS or the flag --prometheus-address <network-interface>:<port> to select a different listen address.

If coder server --prometheus-enable is started locally, you can preview the metrics endpoint in your browser or with curl:

$ curl http://localhost:2112/
# HELP coderd_api_active_users_duration_hour The number of users that have been active within the last hour.
# TYPE coderd_api_active_users_duration_hour gauge
coderd_api_active_users_duration_hour 0
...

Kubernetes deployment

The Prometheus endpoint can be enabled in the Helm chart's values.yml by setting CODER_PROMETHEUS_ENABLE=true. Once enabled, the environment variable CODER_PROMETHEUS_ADDRESS will be set by default to 0.0.0.0:2112. A Service Endpoint will not be exposed. If you need to expose the Prometheus port on a Service, (for example, to use a ServiceMonitor), create a separate headless service instead.

apiVersion: v1
kind: Service
metadata:
  name: coder-prom
  namespace: coder
  labels:
    app.kubernetes.io/name: coder-prom
spec:
  clusterIP: None
  ports:
    - name: prom-http
      port: 2112
      protocol: TCP
      targetPort: 2112
  selector:
    app.kubernetes.io/instance: coder
    app.kubernetes.io/name: coder
  type: ClusterIP

Prometheus configuration

To allow Prometheus to scrape the Coder metrics, you will need to create a scrape_config in your prometheus.yml file, or in the Prometheus Helm chart values. The following is an example scrape_config.

scrape_configs:
  - job_name: "coder"
    scheme: "http"
    static_configs:
      # replace with the the IP address of the Coder pod or server
      - targets: ["<ip>:2112"]
        labels:
          apps: "coder"

If you run a standalone AI gateway, each replica exports its own metrics on its own listener. Its Helm chart uses the same 0.0.0.0:2112 default as the coder chart, but sets up no scrape discovery. Refer to AI gateway monitoring for more details.

To use the Kubernetes Prometheus operator to scrape metrics, you will need to create a ServiceMonitor in your Coder deployment namespace. The following is an example ServiceMonitor.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: coder-service-monitor
  namespace: coder
spec:
  endpoints:
    - port: prom-http
      interval: 10s
      scrapeTimeout: 10s
  namespaceSelector:
    matchNames:
    - coder
  selector:
    matchLabels:
      app.kubernetes.io/name: coder-prom

Available metrics

You must first enable coderd_agentstats_* with the flag --prometheus-collect-agent-stats, or the environment variable CODER_PROMETHEUS_COLLECT_AGENT_STATS before they can be retrieved from the deployment. They will always be available from the agent.

coderd_workspace_last_used_timestamp_seconds is also off by default. Refer to Workspace last used time.

The coder_ai_gateway_cost_control_* metrics are exported only by coderd.

Every coderd replica exports the full set of coderd_insights_* series, each computed over that replica's own rolling 5 minute window, so aggregate across replicas with max, not sum.

Name Type Description Labels
agent_boundary_log_proxy_batches_dropped_total counter Total number of boundary log batches dropped before reaching coderd. Reason: buffer_full = the agent's internal buffer is full, meaning boundary is producing logs faster than the agent can forward them to coderd; forward_failed = the agent failed to send the batch to coderd, potentially because coderd is unreachable or the connection was interrupted. reason
agent_boundary_log_proxy_batches_forwarded_total counter Total number of boundary log batches successfully forwarded to coderd. Compare with batches_dropped_total to compute a drop rate.
agent_boundary_log_proxy_logs_dropped_total counter Total number of individual boundary log entries dropped before reaching coderd. Reason: buffer_full = the agent's internal buffer is full; forward_failed = the agent failed to send the batch to coderd; boundary_channel_full = boundary's internal send channel overflowed, meaning boundary is generating logs faster than it can batch and send them; boundary_batch_full = boundary's outgoing batch buffer overflowed after a failed flush, meaning boundary could not write to the agent's socket. reason
agent_scripts_executed_total counter Total number of scripts executed by the Coder agent. Includes cron scheduled scripts. agent_name success template_name username workspace_name
coder_ai_gateway_circuit_breaker_rejects_total counter Total number of requests rejected due to open circuit breaker. endpoint model provider
coder_ai_gateway_circuit_breaker_state gauge Current state of the circuit breaker (0=closed, 0.5=half-open, 1=open). endpoint model provider
coder_ai_gateway_circuit_breaker_trips_total counter Total number of times the circuit breaker transitioned to open state. endpoint model provider
coder_ai_gateway_cost_control_blocked_requests_total counter The number of AI requests blocked because the initiator's budget was exceeded. group_id
coder_ai_gateway_cost_control_blocked_users gauge The number of users currently over their AI budget. group_id
coder_ai_gateway_cost_control_enforcement_duration_seconds histogram The duration of AI budget enforcement checks, in seconds (outcome: allowed, blocked, error). outcome
coder_ai_gateway_cost_control_unpriced_token_usage_records_total counter The number of recorded AI token-usage records for which no price was found for either the requested model or the provider-reported model. provider is the provider instance name, provider_type is its configured type, model is the requested model, and provider_model is the model reported by the provider (empty when not reported). model provider_model provider_type provider
coder_ai_gateway_injected_tool_invocations_total counter The number of times an injected MCP tool was invoked by AI Gateway. model name provider server
coder_ai_gateway_interceptions_duration_seconds histogram The total duration of intercepted requests, in seconds. The majority of this time will be the upstream processing of the request. AI Gateway has no control over upstream processing time, so it's just an illustrative metric. model provider
coder_ai_gateway_interceptions_inflight gauge The number of intercepted requests which are being processed. model provider route
coder_ai_gateway_interceptions_total counter The count of intercepted requests. client initiator_id method model provider route status
coder_ai_gateway_key_pool_exhaustions_total counter The number of times the key pool was exhausted with no usable key (outcome: rate_limited, auth_failed). outcome provider
coder_ai_gateway_key_pool_failover_attempts histogram The number of keys attempted before success or exhaustion, per interception for bridged requests and per request for passthrough requests. provider
coder_ai_gateway_key_pool_state gauge The number of keys currently in each state (state: valid, temporary, permanent). provider state
coder_ai_gateway_key_pool_state_transitions_total counter The number of API key state transitions during failover (reason: rate_limited, unauthorized). provider reason
coder_ai_gateway_non_injected_tool_selections_total counter The number of times an AI model selected a tool to be invoked by the client. model name provider
coder_ai_gateway_passthrough_total counter The count of requests which were not intercepted but passed through to the upstream. method provider route
coder_ai_gateway_prompts_total counter The number of prompts issued by users (initiators). client initiator_id model provider
coder_ai_gateway_provider_info gauge One series per configured AI provider. Value is always 1; the status label (enabled, disabled, error) carries the alertable signal. provider_name provider_type status
coder_ai_gateway_providers_last_reload_success_timestamp_seconds gauge Unix timestamp of the last provider reload that successfully refreshed the providers. A gap against the providers_last_reload_timestamp_seconds gauge means the loop is firing but the refresh function is failing.
coder_ai_gateway_providers_last_reload_timestamp_seconds gauge Unix timestamp of the last provider reload attempt, success or failure.
coder_ai_gateway_proxy_connect_sessions_total counter Total number of CONNECT sessions established. type
coder_ai_gateway_proxy_inflight_mitm_requests gauge Number of MITM requests currently being processed. provider
coder_ai_gateway_proxy_mitm_requests_total counter Total number of MITM requests handled by the proxy. provider
coder_ai_gateway_proxy_mitm_responses_total counter Total number of MITM responses by HTTP status code class. code provider
coder_ai_gateway_proxy_provider_info gauge One series per configured AI provider. Value is always 1; the status label (enabled, disabled, error, proxy_excluded) carries the alertable signal. provider_name provider_type status
coder_ai_gateway_proxy_providers_last_reload_success_timestamp_seconds gauge Unix timestamp of the last provider reload that successfully refreshed the router. A gap against the providers_last_reload_timestamp_seconds gauge means the loop is firing but the refresh function is failing.
coder_ai_gateway_proxy_providers_last_reload_timestamp_seconds gauge Unix timestamp of the last provider reload attempt, success or failure.
coder_ai_gateway_tokens_total counter The number of tokens used by intercepted requests. client initiator_id model provider type
coder_derp_server_accepts_total counter Total DERP connections accepted.
coder_derp_server_average_queue_duration_ms gauge Average queue duration in milliseconds.
coder_derp_server_bytes_received_total counter Total bytes received.
coder_derp_server_bytes_sent_total counter Total bytes sent.
coder_derp_server_clients gauge Total clients (local + remote).
coder_derp_server_clients_local gauge Local clients.
coder_derp_server_clients_remote gauge Remote (mesh) clients.
coder_derp_server_connections gauge Current DERP connections.
coder_derp_server_got_ping_total counter Total pings received.
coder_derp_server_home_connections gauge Current home DERP connections.
coder_derp_server_home_moves_in_total counter Total home moves in.
coder_derp_server_home_moves_out_total counter Total home moves out.
coder_derp_server_packets_dropped_reason_total counter Packets dropped by reason. reason
coder_derp_server_packets_dropped_total counter Total packets dropped.
coder_derp_server_packets_dropped_type_total counter Packets dropped by type. type
coder_derp_server_packets_forwarded_in_total counter Total packets forwarded in from mesh peers.
coder_derp_server_packets_forwarded_out_total counter Total packets forwarded out to mesh peers.
coder_derp_server_packets_received_kind_total counter Packets received by kind. kind
coder_derp_server_packets_received_total counter Total packets received.
coder_derp_server_packets_sent_total counter Total packets sent.
coder_derp_server_peer_gone_disconnected_total counter Total peer gone (disconnected) frames sent.
coder_derp_server_peer_gone_not_here_total counter Total peer gone (not here) frames sent.
coder_derp_server_sent_pong_total counter Total pongs sent.
coder_derp_server_unknown_frames_total counter Total unknown frames received.
coder_derp_server_watchers gauge Current watchers.
coder_pubsub_connected gauge Whether we are connected (1) or not connected (0) to the backend backend
coder_pubsub_current_events gauge The current number of pubsub event channels listened for backend
coder_pubsub_current_subscribers gauge The current number of active pubsub subscribers backend
coder_pubsub_disconnections_total counter Total number of times we disconnected unexpectedly from the backend backend
coder_pubsub_latency_measure_errs_total counter The number of pubsub latency measurement failures backend
coder_pubsub_latency_measures_total counter The number of pubsub latency measurements backend
coder_pubsub_messages_total counter Total number of messages received backend size
coder_pubsub_published_bytes_total counter Total number of bytes successfully published across all publishes backend
coder_pubsub_publishes_total counter Total number of calls to Publish backend success
coder_pubsub_receive_duration_seconds histogram The time taken to receive a message from a pubsub event channel backend
coder_pubsub_receive_latency_seconds gauge The most recent time taken to receive a message from a pubsub event channel. New integrations should prefer the coder_pubsub_receive_duration_seconds histogram. backend
coder_pubsub_received_bytes_total counter Total number of bytes received across all messages backend
coder_pubsub_send_duration_seconds histogram The time taken to send a message into a pubsub event channel backend
coder_pubsub_send_latency_seconds gauge The most recent time taken to send a message into a pubsub event channel. New integrations should prefer the coder_pubsub_send_duration_seconds histogram. backend
coder_pubsub_subscribes_total counter Total number of calls to Subscribe/SubscribeWithErr backend success
coder_servertailnet_agent_unreachable_total counter Number of connection attempts where the workspace agent did not answer in time, by reason: no_node (this server has no node for the agent), peer_lost (the agent's or this server's coordinator connection ended), no_handshake (no WireGuard handshake yet), handshake_stale (last handshake over 180s ago), handshake_ok (last handshake under 180s ago), diagnostics_timeout (peer state not readable within 1s), client_canceled (client left within 5s). reason
coder_servertailnet_await_reachable_seconds histogram Time until a workspace agent answered the first ping when a new connection was opened.
coder_servertailnet_connections_total counter Total number of TCP connections made to workspace agents. network
coder_servertailnet_derp_connects_total counter Number of times the server tailnet connected to the embedded DERP relay. Stays 0 when the embedded relay is disabled.
coder_servertailnet_open_connections gauge Total number of TCP connections currently open to workspace agents. network
coder_workspaceapps_agent_offline_total counter Number of workspace app requests that got the offline page because the agent was not connected. access_method status
coderd_agentapi_metadata_batch_size histogram Total number of metadata entries in each batch, updated before flushes.
coderd_agentapi_metadata_batch_utilization histogram Number of metadata keys per agent in each batch, updated before flushes.
coderd_agentapi_metadata_batches_total counter Total number of metadata batches flushed. reason
coderd_agentapi_metadata_dropped_keys_total counter Total number of metadata keys dropped due to capacity limits.
coderd_agentapi_metadata_flush_duration_seconds histogram Time taken to flush metadata batch to database and pubsub. reason
coderd_agentapi_metadata_flushed_total counter Total number of unique metadatas flushed.
coderd_agentapi_metadata_publish_errors_total counter Total number of metadata batch pubsub publish calls that have resulted in an error.
coderd_agents_apps gauge Agent applications with statuses. agent_name app_name health username workspace_name
coderd_agents_connection_latencies_seconds gauge Agent connection latencies in seconds. agent_name derp_region preferred username workspace_name
coderd_agents_connections gauge Agent connections with statuses. agent_name lifecycle_state status tailnet_node username workspace_name
coderd_agents_first_connection_seconds histogram Duration from agent creation to first connection in seconds. agent_name template_name
coderd_agents_up gauge The number of active agents per workspace. template_name template_version username workspace_name
coderd_agentstats_app_info gauge The current family of each registered session app. Value is always 1. app_name family
coderd_agentstats_connection_count gauge The number of established connections by agent agent_name username workspace_name
coderd_agentstats_connection_median_latency_seconds gauge The median agent connection latency in seconds agent_name username workspace_name
coderd_agentstats_currently_reachable_peers gauge The number of peers (e.g. clients) that are currently reachable over the encrypted network. agent_name connection_type template_name username workspace_name
coderd_agentstats_rx_bytes gauge Agent Rx bytes agent_name username workspace_name
coderd_agentstats_session_count gauge The number of sessions established by app name agent_name app_name username workspace_name
coderd_agentstats_session_count_jetbrains gauge The number of session established by JetBrains agent_name username workspace_name
coderd_agentstats_session_count_reconnecting_pty gauge The number of session established by reconnecting PTY agent_name username workspace_name
coderd_agentstats_session_count_ssh gauge The number of session established by SSH agent_name username workspace_name
coderd_agentstats_session_count_vscode gauge The number of session established by VSCode agent_name username workspace_name
coderd_agentstats_session_counts_overflow_total counter Total number of reported session count entries summed into the overflow app after exceeding the per-report cap.
coderd_agentstats_startup_script_seconds gauge Amount of time taken to run the startup script in seconds. agent_name success template_name username workspace_name
coderd_agentstats_tx_bytes gauge Agent Tx bytes agent_name username workspace_name
coderd_api_active_users_duration_hour gauge The number of users that have been active within the last hour.
coderd_api_concurrent_requests gauge The number of concurrent API requests. method path
coderd_api_concurrent_websockets gauge The total number of concurrent API websockets. path
coderd_api_request_latencies_seconds histogram Latency distribution of requests in seconds. method path
coderd_api_requests_processed_total counter The total number of processed API requests code method path
coderd_api_requests_too_large_total counter The total number of API requests answered 413, by the reason they were rejected. A sustained rate of reason="request_body" on one route is more often a limit set too tight for a legitimate payload than an attempt to exhaust memory. reason="other" counts 413s raised for reasons unrelated to the size of the request body, such as agent log storage overflow or an archive that is too large once expanded. method path reason
coderd_api_total_user_count gauge The total number of registered users, partitioned by status. status
coderd_api_websocket_durations_seconds histogram Websocket duration distribution of requests in seconds. path
coderd_api_websocket_probes_total counter WebSocket liveness probe outcomes by route. Compare rate(...{result="ok"}[1m]) against coderd_api_concurrent_websockets to detect unresponsive WebSocket connections. path result
coderd_api_workspace_latest_build gauge The current number of workspace builds by status for all non-deleted workspaces. status
coderd_authz_authorize_duration_seconds histogram Duration of the 'Authorize' call in seconds. Only counts calls that succeed. allowed
coderd_authz_prepare_authorize_duration_seconds histogram Duration of the 'PrepareAuthorize' call in seconds.
coderd_build_info gauge Describes the current build/version of the Coder server. Value is always 1. revision version
coderd_chat_auto_archive_records_archived_total counter Total number of chats archived by the auto-archive job (counting both roots and cascaded children).
coderd_chat_workspace_upload_total counter Total chat workspace file uploads by HTTP response status. status
coderd_chatd_agents_active gauge Deployment-wide number of chats holding a concurrent-agent capacity slot. Every replica reports the same database-derived value; aggregate with max, not sum. pool
coderd_chatd_agents_queued_for_capacity gauge Deployment-wide number of chats waiting for a concurrent-agent capacity slot. Every replica reports the same database-derived value; aggregate with max, not sum. pool
coderd_chatd_chats gauge Number of chats being processed, by state. state
coderd_chatd_compaction_total counter Total compaction outcomes (only recorded when compaction was triggered or failed). model provider result
coderd_chatd_find_tools_activations_total counter Total deferred tool activations returned by find_tools.
coderd_chatd_find_tools_calls_total counter Total find_tools calls.
coderd_chatd_find_tools_empty_total counter Total find_tools calls with no matches.
coderd_chatd_find_tools_match_count histogram Number of matches returned by find_tools calls.
coderd_chatd_hook_context_size_bytes histogram Lifecycle hook model context response size in bytes. event
coderd_chatd_hook_decisions_total counter Total lifecycle hook permission decisions by event and decision. decision event
coderd_chatd_hook_dispatch_seconds histogram Lifecycle hook dispatch duration in seconds. event
coderd_chatd_hook_dispatches_total counter Total lifecycle hook dispatches by event and result. event result
coderd_chatd_hook_input_overrides_total counter Total lifecycle hook input overrides by event. event
coderd_chatd_message_count histogram Number of messages in the prompt per LLM request. model provider
coderd_chatd_model_stage_duration_seconds histogram Wall time of the stages that are a provider's work on a model: stream (open to close) and provider_attempt (one HTTP round trip, closed on response headers). Time to first token per model is ttft_seconds. Failed stream and provider_attempt windows are observed. Observed only for turn-scoped stages, which are also observed on stage_duration_seconds; background-scoped model stages appear on stage_duration_seconds only. provider_type is the configured AI provider type (for example bedrock), not the wire protocol other chatd metrics report as provider. Registered only with the chat-stage-metrics experiment. chat_kind model provider_type stage
coderd_chatd_prompt_size_bytes histogram Estimated byte size of the prompt per LLM request. model provider
coderd_chatd_response_size_bytes histogram Byte size of streamed model response content (assistant text, reasoning, and tool-call input deltas) per LLM request. model provider
coderd_chatd_stage_anomalies_total counter Chat lifecycle stage observations dropped or adjusted, by reason: negative_elapsed and inverted_window (end before start, clock skew), missing_timestamp (unset start or end), future_start (start ahead of this replica's clock, span started now), stale_anchor (trigger not after the previous turn's anchor, turn started now). Registered only with the chat-stage-metrics experiment. reason
coderd_chatd_stage_duration_seconds histogram Wall time spent in each chat lifecycle stage. Stages overlap, so this is a stage-time profile, not a partition of the turn. scope separates latency attributable to a prompt (stages inside a chat turn, and queue_wait before it) from detached background work; chat_kind is empty without a known chat. Observed: chat_turn, queue_wait, acquisition, mcp_connect, stream, time_to_first_token, provider_attempt, tool_call, commit, retry_backoff; other stages are span-only. Failed windows are observed, except time_to_first_token, which observes only windows closed by an output part, and chat_turn, which observes only completed turns. chat_turn runs from its trigger (the prompt, a submitted client tool result, or a compaction request) to the end of its turn, includes acquisition, and excludes queue_wait and time waiting in requires_action, so a prompt that waits in requires_action is observed once per segment. queue_wait runs from a queued message's creation to its promotion, so it includes time the message spent under edit. Registered only with the chat-stage-metrics experiment. chat_kind scope stage
coderd_chatd_steps_total counter Total agentic loop steps across all chats. model provider
coderd_chatd_stream_retries_total counter Total LLM stream retries. kind model provider
coderd_chatd_tool_errors_total counter Total tool calls that returned an error result. model provider tool_name
coderd_chatd_tool_result_size_bytes histogram Size in bytes of each tool execution result. model provider tool_name
coderd_chatd_tool_result_truncated_total counter Total tool results truncated to fit the model context window. model provider tool_name
coderd_chatd_ttft_seconds histogram Time-to-first-token: wall time from LLM request to the first streamed output part: any part other than warnings, keepalives, finish, or error, including start markers such as text_start. Attempts that end before an output part, by failing or by finishing empty, are not observed. The time_to_first_token stage of stage_duration_seconds measures the same window without a model label and is registered only with the chat-stage-metrics experiment. model provider
coderd_chatd_turn_outcomes_total counter Total closed chat turns by outcome: completed (the turn finished, including a turn that ends when the chat enters requires_action; the submitted tool result opens a new turn), interrupted (an interrupt request from a user, an API client, or a parent agent stopped it), error (a failure stopped it), abandoned (a newer prompt, such as a message edit, or runner exit closed it before it finished; an edit sent while the chat is interrupting can close the stopped turn as abandoned rather than interrupted). Every closed turn is counted exactly once. Registered only with the chat-stage-metrics experiment. chat_kind outcome
coderd_db_query_counts_total counter Total number of queries labeled by HTTP route, method, and query name. method query route
coderd_db_query_latencies_seconds histogram Latency distribution of queries in seconds. query
coderd_db_tx_duration_seconds histogram Duration of transactions in seconds. success tx_id
coderd_db_tx_executions_count counter Total count of transactions executed. 'retries' is expected to be 0 for a successful transaction. retries success tx_id
coderd_dbpurge_chat_search_rows_backfilled_total counter Total number of chat message rows whose search_tsv was backfilled.
coderd_dbpurge_iteration_duration_seconds histogram Duration of each dbpurge iteration in seconds. success
coderd_dbpurge_records_purged_total counter Total number of records purged by type. record_type
coderd_experiments gauge Indicates whether each experiment is enabled (1) or not (0) experiment
coderd_insights_applications_usage_seconds gauge The app usage per template. connection_method is workspace_app for template apps, or ssh, reconnecting_pty, or port_forwarding for built-in apps, which have one series per app name in slug. Overlapping apps each count their own minutes. connection_method display_name organization_name slug template_name
coderd_insights_parameters gauge The parameter usage per template. organization_name parameter_name parameter_type parameter_value template_name
coderd_insights_templates_active_users gauge The number of active users of the template. organization_name template_name
coderd_license_active_users gauge The number of active users.
coderd_license_errors gauge The number of active license errors.
coderd_license_limit_users gauge The user seats limit based on the active Coder license.
coderd_license_user_limit_enabled gauge Returns 1 if the current license enforces the user limit.
coderd_license_warnings gauge The number of active license warnings.
coderd_lifecycle_autobuild_execution_duration_seconds histogram Duration of each autobuild execution.
coderd_notifications_dispatcher_send_seconds histogram The time taken to dispatch notifications. method
coderd_notifications_inflight_dispatches gauge The number of dispatch attempts which are currently in progress. method notification_template_id
coderd_notifications_pending_updates gauge The number of dispatch attempt results waiting to be flushed to the store.
coderd_notifications_queued_seconds histogram The time elapsed between a notification being enqueued in the store and retrieved for dispatching (measures the latency of the notifications system). This should generally be within CODER_NOTIFICATIONS_FETCH_INTERVAL seconds; higher values for a sustained period indicates delayed processing and CODER_NOTIFICATIONS_LEASE_COUNT can be increased to accommodate this. method
coderd_notifications_retry_count counter The count of notification dispatch retry attempts. method notification_template_id
coderd_notifications_synced_updates_total counter The number of dispatch attempt results flushed to the store.
coderd_oauth2_external_requests_rate_limit gauge The total number of allowed requests per interval. name resource
coderd_oauth2_external_requests_rate_limit_next_reset_unix gauge Unix timestamp for when the next interval starts name resource
coderd_oauth2_external_requests_rate_limit_remaining gauge The remaining number of allowed requests in this interval. name resource
coderd_oauth2_external_requests_rate_limit_reset_in_seconds gauge Seconds until the next interval name resource
coderd_oauth2_external_requests_rate_limit_used gauge The number of requests made in this interval. name resource
coderd_oauth2_external_requests_rate_limited_total counter The total number of api calls to external oauth2 providers that returned a rate-limited response (a 429, or a 403 with rate-limit headers). name source status_code
coderd_oauth2_external_requests_total counter The total number of api calls made to external oauth2 providers. 'status_code' will be 0 if the request failed with no response. name source status_code
coderd_open_file_refs_current gauge The count of file references currently open in the file cache. Multiple references can be held for the same file.
coderd_open_file_refs_total counter The total number of file references ever opened in the file cache. The 'hit' label indicates if the file was loaded from the cache. hit
coderd_open_files_current gauge The count of unique files currently open in the file cache.
coderd_open_files_size_bytes_current gauge The current amount of memory of all files currently open in the file cache.
coderd_open_files_size_bytes_total counter The total amount of memory ever opened in the file cache. This number never decrements.
coderd_open_files_total counter The total count of unique files ever opened in the file cache.
coderd_prebuilds_reconciliation_duration_seconds histogram Duration of each prebuilds reconciliation cycle.
coderd_prebuilt_workspace_claim_duration_seconds histogram Time to claim a prebuilt workspace by organization, template, and preset. organization_name preset_name template_name
coderd_prebuilt_workspaces_claimed_total counter Total number of prebuilt workspaces which were claimed by users. Claiming refers to creating a workspace with a preset selected for which eligible prebuilt workspaces are available and one is reassigned to a user. organization_name preset_name template_name
coderd_prebuilt_workspaces_created_total counter Total number of prebuilt workspaces that have been created to meet the desired instance count of each template preset. organization_name preset_name template_name
coderd_prebuilt_workspaces_desired gauge Target number of prebuilt workspaces that should be available for each template preset. organization_name preset_name template_name
coderd_prebuilt_workspaces_eligible gauge Current number of prebuilt workspaces that are eligible to be claimed by users. These are workspaces that have completed their build process with their agent reporting 'ready' status. organization_name preset_name template_name
coderd_prebuilt_workspaces_failed_total counter Total number of prebuilt workspaces that failed to build. organization_name preset_name template_name
coderd_prebuilt_workspaces_metrics_last_updated gauge The unix timestamp when the metrics related to prebuilt workspaces were last updated; these metrics are cached.
coderd_prebuilt_workspaces_preset_hard_limited gauge Indicates whether a given preset has reached the hard failure limit (1 = hard-limited). Metric is omitted otherwise. organization_name preset_name template_name
coderd_prebuilt_workspaces_preset_validation_failed gauge Indicates whether a given preset has validation failures (1 = validation failed). Metric is omitted otherwise. organization_name preset_name template_name
coderd_prebuilt_workspaces_reconciliation_paused gauge Indicates whether prebuilds reconciliation is currently paused (1 = paused, 0 = not paused).
coderd_prebuilt_workspaces_resource_replacements_total counter Total number of prebuilt workspaces whose resource(s) got replaced upon being claimed. In Terraform, drift on immutable attributes results in resource replacement. This represents a worst-case scenario for prebuilt workspaces because the pre-provisioned resource would have been recreated when claiming, thus obviating the point of pre-provisioning. See https://coder.com/docs/admin/templates/extending-templates/prebuilt-workspaces#preventing-resource-replacement organization_name preset_name template_name
coderd_prebuilt_workspaces_running gauge Current number of prebuilt workspaces that are in a running state. These workspaces have started successfully but may not yet be claimable by users (see coderd_prebuilt_workspaces_eligible). organization_name preset_name template_name
coderd_prometheusmetrics_agents_execution_seconds histogram Histogram for duration of agents metrics collection in seconds.
coderd_prometheusmetrics_agentstats_execution_seconds histogram Histogram for duration of agent stats metrics collection in seconds.
coderd_prometheusmetrics_metrics_aggregator_execution_cleanup_seconds histogram Histogram for duration of metrics aggregator cleanup in seconds.
coderd_prometheusmetrics_metrics_aggregator_execution_update_seconds histogram Histogram for duration of metrics aggregator update in seconds.
coderd_prometheusmetrics_metrics_aggregator_store_size gauge The number of metrics stored in the aggregator
coderd_provisioner_job_queue_wait_seconds histogram Time from job creation to acquisition by a provisioner daemon. build_reason job_type provisioner_type transition
coderd_provisionerd_job_timings_seconds histogram The provisioner job time duration in seconds. provisioner status
coderd_provisionerd_jobs_current gauge The number of currently running provisioner jobs. provisioner
coderd_provisionerd_num_daemons gauge The number of provisioner daemons.
coderd_provisionerd_workspace_build_timings_seconds histogram The time taken for a workspace to build. status template_name template_version workspace_transition
coderd_proxyhealth_health_check_duration_seconds histogram Histogram for duration of proxy health collection in seconds.
coderd_proxyhealth_health_check_results gauge This endpoint returns a number to indicate the health status. -3 (unknown), -2 (Unreachable), -1 (Unhealthy), 0 (Unregistered), 1 (Healthy) proxy_id
coderd_template_workspace_build_duration_seconds histogram Duration from workspace build creation to agent ready, by template. is_prebuild organization_name status template_name transition
coderd_usage_events_expired gauge Number of unpublished usage events aged 30 to 60 days. Events older than 60 days are not counted.
coderd_usage_events_pending gauge Number of unpublished usage events within the 30-day publishing window.
coderd_usage_events_pending_oldest_age_seconds gauge Age in seconds of the oldest unpublished usage event within the 30-day publishing window. Zero when no events are pending.
coderd_usage_events_publish_results_total counter Total usage events accepted or rejected after a valid Tallyman response, by result and event type. event_type result
coderd_usage_events_publish_send_errors_total counter Total usage event publish attempts that did not receive a valid Tallyman response. Does not reset after a valid response.
coderd_workspace_builds_enqueued_total counter Total number of workspace build enqueue attempts. build_reason provisioner_type status transition
coderd_workspace_builds_total counter The number of workspaces started, updated, or deleted. status template_name template_version workspace_name workspace_owner workspace_transition
coderd_workspace_creation_duration_seconds histogram Time to create a workspace by organization, template, preset, and type (regular or prebuild). organization_name preset_name template_name type
coderd_workspace_creation_total counter Total regular (non-prebuilt) workspace creations by organization, template, and preset. organization_name preset_name template_name
coderd_workspace_last_used_timestamp_seconds gauge The last time each workspace was used, as a Unix timestamp in seconds. Every replica reports the same database-derived value; aggregate with max, not sum. organization_name template_name workspace_id workspace_name workspace_owner
coderd_workspace_latest_build_status gauge The current workspace statuses by template, transition, and owner for all non-deleted workspaces. status template_name template_version workspace_owner workspace_transition
go_gc_duration_seconds summary A summary of the pause duration of garbage collection cycles.
go_goroutines gauge Number of goroutines that currently exist.
go_info gauge Information about the Go environment. version
go_memstats_alloc_bytes gauge Number of bytes allocated and still in use.
go_memstats_alloc_bytes_total counter Total number of bytes allocated, even if freed.
go_memstats_buck_hash_sys_bytes gauge Number of bytes used by the profiling bucket hash table.
go_memstats_frees_total counter Total number of frees.
go_memstats_gc_sys_bytes gauge Number of bytes used for garbage collection system metadata.
go_memstats_heap_alloc_bytes gauge Number of heap bytes allocated and still in use.
go_memstats_heap_idle_bytes gauge Number of heap bytes waiting to be used.
go_memstats_heap_inuse_bytes gauge Number of heap bytes that are in use.
go_memstats_heap_objects gauge Number of allocated objects.
go_memstats_heap_released_bytes gauge Number of heap bytes released to OS.
go_memstats_heap_sys_bytes gauge Number of heap bytes obtained from system.
go_memstats_last_gc_time_seconds gauge Number of seconds since 1970 of last garbage collection.
go_memstats_lookups_total counter Total number of pointer lookups.
go_memstats_mallocs_total counter Total number of mallocs.
go_memstats_mcache_inuse_bytes gauge Number of bytes in use by mcache structures.
go_memstats_mcache_sys_bytes gauge Number of bytes used for mcache structures obtained from system.
go_memstats_mspan_inuse_bytes gauge Number of bytes in use by mspan structures.
go_memstats_mspan_sys_bytes gauge Number of bytes used for mspan structures obtained from system.
go_memstats_next_gc_bytes gauge Number of heap bytes when next garbage collection will take place.
go_memstats_other_sys_bytes gauge Number of bytes used for other system allocations.
go_memstats_stack_inuse_bytes gauge Number of bytes in use by the stack allocator.
go_memstats_stack_sys_bytes gauge Number of bytes obtained from system for stack allocator.
go_memstats_sys_bytes gauge Number of bytes obtained from system.
go_threads gauge Number of OS threads created.
process_cpu_seconds_total counter Total user and system CPU time spent in seconds.
process_max_fds gauge Maximum number of open file descriptors.
process_open_fds gauge Number of open file descriptors.
process_resident_memory_bytes gauge Resident memory size in bytes.
process_start_time_seconds gauge Start time of the process since unix epoch in seconds.
process_virtual_memory_bytes gauge Virtual memory size in bytes.
process_virtual_memory_max_bytes gauge Maximum amount of virtual memory available in bytes.
promhttp_metric_handler_requests_in_flight gauge Current number of scrapes being served.
promhttp_metric_handler_requests_total counter Total number of scrapes by HTTP status code. code

Application usage by template

coderd_insights_applications_usage_seconds reports app usage per template over the last five minutes. The connection_method label distinguishes template apps from built-in apps:

connection_method Apps slug display_name
workspace_app Apps defined in the template, opened through Coder Slug or port The app's display name
ssh, reconnecting_pty, port_forwarding Built-in apps that connect through the agent, such as desktop IDEs Reported app name Registered name, such as Cursor; Unknown for unknown, Other apps for overflow, and otherwise the app name

A template app can share a slug with a built-in app, such as vscode, so identify an app by connection_method and slug. display_name can change between releases.

Built-in apps have one series per app name, such as vscode, cursor, or goland, and only for apps used in the window. Each app counts its own minutes, so a user with VS Code and Cursor open in the same minute counts in both series. slug="unknown" holds sessions reported without an app name, or with the reserved name overflow.

Per template, every built-in app Coder recognizes gets a series, and so do the 64 busiest other apps. The rest add up under one series, connection_method="ssh" with slug="overflow", which can count a minute more than once. The template insights page in the dashboard uses the same rule, but applies the limit across all selected templates instead of per template.

Upgrade from the family series

This metric changed in three ways:

  • application_name is now display_name.
  • Every series has the new connection_method label.
  • Built-in apps report one series per app instead of one per app family. The family series had slug="" and were reported even at zero.

Selectors with a value for application_name, or with slug="", return nothing. These queries still return data, with different values:

  • A template app query by slug also matches a built-in app with the same name. Add connection_method="workspace_app".
  • A total without a slug selector now includes every built-in app, including port_forwarding, sftp, and unknown.

Update Grafana dashboards, alerting rules, and recording rules:

Old series Closest new series
{slug="", application_name="Visual Studio Code"} {connection_method="ssh", slug="vscode"}, plus its forks such as cursor
{slug="", application_name="JetBrains"} {connection_method="ssh", slug="jetbrains"}, plus its IDEs such as goland
{slug="", application_name="Web Terminal"} {connection_method="reconnecting_pty", slug="reconnecting_pty"}
{slug="", application_name="SSH"} {connection_method="ssh", slug="ssh"}, plus zed
{application_name="<app>"} for a template app {connection_method="workspace_app", display_name="<app>"}

To total a family, sum its apps, for example sum by (template_name, organization_name) (max by (template_name, organization_name, slug, connection_method) (coderd_insights_applications_usage_seconds{connection_method="ssh", slug=~"vscode|cursor|windsurf"})). The sum counts a minute twice if two of the apps were open at once.

In the example, the inner max by keeps one series per app, and the outer sum adds the apps for each template.

Workspace last used time

coderd_workspace_last_used_timestamp_seconds reports when each workspace was last used, as a Unix timestamp in seconds. It's the value the dormancy threshold measures inactivity from.

The metric is off by default. Enable it with the flag --prometheus-collect-workspace-stats or the environment variable CODER_PROMETHEUS_COLLECT_WORKSPACE_STATS.

Series and cost

  • Every workspace has a series, including stopped and unclaimed prebuilt workspaces. Unclaimed prebuilt workspaces have workspace_owner="prebuilds".
  • The workspace_id label identifies the workspace, so a new workspace that reuses a deleted workspace's name gets its own series.
  • Deleting a workspace or its owner removes its series.
  • Every coderd replica reports every series, so aggregate with max, not sum.
  • Each replica adds one active series per workspace to your Prometheus server.

What counts as use

  • Traffic on any connection to the workspace agent counts, such as SSH, an IDE, the web terminal, or port forwarding.
  • Traffic to a workspace app through Coder counts.
  • By default, a running coder ssh or coder port-forward command counts once a minute, even without traffic.

The value is also set to the current time when:

  • The workspace is created.
  • Someone activates a dormant workspace.
  • A template admin resets the inactivity period of the template's workspaces.

Accuracy

With default settings, the value trails real use by up to about 2.5 minutes, plus your scrape interval:

Step Default interval
The workspace agent reports activity 30 seconds
Coder saves activity, with the time of the save, not of the activity 1 minute
The metric refreshes 1 minute

Use the metric for alerts measured in hours or days.

Example

This query returns workspaces that nobody has used for 30 days, excluding unclaimed prebuilt workspaces:

time() - max by (organization_name, template_name, workspace_owner, workspace_name, workspace_id) (coderd_workspace_last_used_timestamp_seconds{workspace_owner!="prebuilds"}) > 30 * 24 * 3600

Note on Prometheus native histogram support

The following metrics support native histograms:

  • coderd_workspace_creation_duration_seconds
  • coderd_prebuilt_workspace_claim_duration_seconds
  • coderd_template_workspace_build_duration_seconds
  • coder_pubsub_send_duration_seconds
  • coder_pubsub_receive_duration_seconds
  • coderd_provisioner_job_queue_wait_seconds
  • coder_ai_gateway_cost_control_enforcement_duration_seconds

Native histograms are an experimental Prometheus feature that removes the need to predefine bucket boundaries and allows higher-resolution buckets that adapt to deployment characteristics. Whether a metric is exposed as classic or native depends entirely on the Prometheus server configuration (see Prometheus docs for details):

  • If native histograms are enabled, Prometheus ingests the high-resolution histogram.
  • If not, it falls back to the predefined buckets.

⚠️ Important: classic and native histograms cannot be aggregated together. If Prometheus is switched from classic to native at a certain point in time, dashboards might need to account for that transition. For this reason, it’s recommended to follow Prometheus’ migration guidelines when moving from classic to native histograms.