## Summary Addresses 106 of the 113 conflicts between the docs and the code that the November audit of `main` found: 102 are fixed and 4 have an interim docs fix. Every finding was re-checked against the code before it was fixed. Most fixes are docs-only. Three are small code fixes where the docs described the intended behavior and the code was wrong, each in its own commits so it can be split out. No change in this PR alters template build or agent behavior. ## Changes Fixes by area (finding counts): templates 28, install 13, integrations and monitoring 13, auth and security 12, networking 11, Coder Agents 9, AI governance 7, reference 7, workspaces 6. Code fixes. Each needs a look from the code owner. If an owner prefers not to take one, drop its commits and I will change the docs to describe the current behavior instead. | Commits | Change | Risk to check | |---------|--------|---------------| | `fix: keep built-in listener defaults in packaged coder.env`, and three follow-up commits | `coder.env` comments out `CODER_HTTP_ADDRESS=` and the three `CODER_TLS_*` lines, so an untouched package install listens on `127.0.0.1:3000` as the docs say. Only the `CODER_HTTP_ADDRESS=` line changed behavior. The TLS options default to empty, so commenting them out is for consistency. The Lima examples rewrite the commented line and fail if the line is not found. A test compares every option that the file sets against the built-in defaults. | The deb and RPM packages were not built or installed. Every released package through v2.39.0 ships the empty line, and upgrades keep an existing file (`config\|noreplace`), so the upgrade pages now say how to fix it. | | `fix(cli/clilog): accept log sinks that are all disabled`, and three follow-up commits | `--log-human=/dev/null` with no other log location no longer fails with "no loggers provided". It is the documented way to disable logging. An empty location still fails with that error when no other sink is set, as before. | Behavior change: with `/dev/null`, the server and the provisioner daemon can start with no log output. | | `fix(cli): detect CODER_PROC_PRIO_MGMT by presence`, and a comment follow-up | Log line only. With the variable set to an empty value, the agent now logs that process priority management is enabled, which is what `agentexec` already does. No detection or behavior change. | Low. | Two option descriptions in `codersdk/deployment.go` are corrected, with the CLI reference, configuration reference, and golden files regenerated: the default quiet hours schedule (the max deadline is the quiet hours window on the next day that the template's autostop requirement applies) and `--experiments` (`*` opts in to the experiments marked safe, also in `enterprise/cli/provisionerdaemonstart.go`). ### Code fixes that were reverted after review Two code fixes from the first version of this PR are reverted in the branch. The net diff has no change under `agent/` or `coderd/`. The docs now describe what the code does, and both issues stay open for the owner. - **`error` merge strategy for `coder_env` (DOCS-1669).** The fix made `merge_strategy = "error"` fail whenever a name was set twice, whatever the resource order. Existing template versions that pair an `error` entry with a later `replace` or `append` entry would start to fail every build after `terraform apply`. That is a breaking change that needs its own PR and release note. The fix and its tests are in `dff8e7c6ce` and can be cherry-picked. `environment-variables.md` now says the check runs only when the `error` resource is processed. - **Nice value before the capability drop (DOCS-1687).** The fix let `coder agent-exec` apply a negative nice value by calling `setpriority` before it drops effective capabilities. The current order is deliberate: with the fix, any local process could run the agent binary with `--coder-nice=-20` when the binary has the `CAP_SYS_NICE` file capability. `process-priority.md` now says a nice value below the agent's own is not applied, and no longer tells admins to grant `CAP_SYS_NICE`. ## Not included Seven findings are left out. - DOCS-1629: covered by #31639. - DOCS-1703 and DOCS-1724: covered by #31980. - DOCS-1584, DOCS-1683, DOCS-1685, DOCS-1686: each needs an owner decision, and no neutral docs wording exists until that decision is made. The validation comment on each issue names the decision. Four fixes are interim, so their issues stay open: - DOCS-1600: the AI Gateway pages now agree that the legacy metric names are deprecated and "will be removed in a future release". The code still emits them, and its TODO names v2.37, which has shipped. The owner still needs to set the removal release. - DOCS-1689: the page now says **New Template** opens the builder only for users with the Owner role, because the page gates on read access to the deployment config. Whether Template Admins should get the builder is still open. - DOCS-1669 and DOCS-1687: docs-only for now. Refer to [Code fixes that were reverted after review](#code-fixes-that-were-reverted-after-review). ## Review notes - Claims that describe product intent and need an owner's yes: the Premium qualifier on per-organization workspace sharing (the API route has no license check, the dashboard needs multiple organizations), `organization` port sharing in OSS, the telemetry field list, and the `1-5` cron range in the prebuilds schedule example. - Not run end to end: the PostgreSQL TLS procedure on a cluster, the OpenShift and air-gapped examples, the BSD `date` and `tar` forms on macOS, the Windows command on the multiple Kubernetes clusters page, and the keep-alive script in `docs/reference/index.md` (only its comparison logic was run, against sample values). Nothing was checked on a live deployment. - Likely merge conflicts with open PRs: #31990 (`mcp-server.md`, `feature-stages.md`, `docker-in-workspaces.md`), #31982 (`workspace-management.md`), #31980 (`health-check.md`), #31977 (`manage-provisioner-jobs.md`), #31869 (`airgap.md`), #30486 (`add-envbuilder.md`), and #31326 and #31531 (`codersdk/deployment.go` and generated files). ## Testing - `make pre-commit` (gen, fmt, lint, slim build), before the PR opened and again after the review changes. - `go test` for `./cli/clilog/...`, `./codersdk/`, `./cli/` (`TestServer`, `TestAgent`, and golden files), and `./enterprise/cli/` (provisioner, server, and help golden files). `./agent/agentexec/...` and `./coderd/provisionerdserver/...` pass after the reverts. Linux only. CI covers macOS and Windows. - The `coder.env` test fails against the file on `main` and passes against the new one. It reads the file with `godotenv`, which accepts CRLF line endings. - An empty `CODER_HTTP_ADDRESS=` was confirmed end to end with a full build of the server (output below). - The new `sed` and `grep` lines in the Lima examples were run with GNU tools against the old and the new form of `coder.env`. The Lima VMs were not started. - `--log-human=/dev/null` and `CODER_LOGGING_HUMAN=` were confirmed with the same build (output below). - `coder agent-exec --coder-nice=-5 -- echo` from a full build, as a non-root user: `coder-agent: failed to adjust niceness to -5 ...: permission denied` on stderr, the command output on stdout, exit code 0. - `postgres-ssl.md` rendered with the GitHub Markdown API: two ordered lists with three and five items, where `main` has six lists. - markdownlint, the table formatter, `make lint/emdash`, `make lint/docs-html`, `make lint/docs-manifest`, `make lint/docs-redirects`, `make lint/style-claims`, and Vale on the changed lines. - A doc-check self-review of the docs diff by four independent reviewers. It found six problems, which are fixed. The accuracy and coverage lenses ran on every area. The language and structure lenses ran in full on only part of the diff. <details> <summary>Test and terminal output for the code fixes (Linux)</summary> ```console $ go test ./codersdk/ -run TestPackagedCoderEnvKeepsDefaults -count=1 -v --- PASS: TestPackagedCoderEnvKeepsDefaults (0.00s) ok github.com/coder/coder/v2/codersdk 0.071s $ go test ./cli/clilog/ -run TestBuilder -count=1 -v --- PASS: TestBuilder (0.00s) --- PASS: TestBuilder/NoSinks (0.00s) --- PASS: TestBuilder/NoSinks/JSONDevNull (0.00s) --- PASS: TestBuilder/NoSinks/StackdriverDevNull (0.00s) --- PASS: TestBuilder/NoSinks/NoConfiguration (0.00s) --- PASS: TestBuilder/NoSinks/HumanDevNull (0.00s) --- PASS: TestBuilder/WithHuman (0.00s) --- PASS: TestBuilder/WithFilter (0.00s) --- PASS: TestBuilder/WithJSON (0.00s) --- PASS: TestBuilder/Verbose (0.00s) --- PASS: TestBuilder/FromDeploymentValues (0.01s) --- PASS: TestBuilder/FromDeploymentValues/Defaults (0.00s) --- PASS: TestBuilder/FromDeploymentValues/EmptyHumanEnv (0.00s) --- PASS: TestBuilder/FromDeploymentValues/Override (0.00s) ok github.com/coder/coder/v2/cli/clilog 0.171s $ TMPDIR=$HOME/.cache/agentexec-tmp go test ./agent/agentexec/... ./coderd/provisionerdserver/... -count=1 ok github.com/coder/coder/v2/agent/agentexec 4.236s ok github.com/coder/coder/v2/coderd/provisionerdserver 2.404s # Full build of the server (go build ./cmd/coder). An empty value stops it before it connects to the database. $ CODER_HTTP_ADDRESS= coder server --postgres-url 'postgres://127.0.0.1:1/x?sslmode=disable' Encountered an error running "coder server", see "coder server --help" for more information error: TLS is disabled. Enable with --tls-enable or specify a HTTP address # With the variable unset, the server passes that check and goes on to the database. $ coder server --postgres-url 'postgres://127.0.0.1:1/x?sslmode=disable' 2026-11-13 02:19:41.901 [warn] ping postgres: retrying error="dial tcp 127.0.0.1:1: connect: connection refused" try=11 Encountered an error running "coder server", see "coder server --help" for more information error: connect to postgres: connect to postgres: unable to connect after 11 tries; last error: dial tcp 127.0.0.1:1: connect: connection refused # An empty log location with no other sink still stops the server. $ CODER_LOGGING_HUMAN= coder server --postgres-url 'postgres://127.0.0.1:1/x?sslmode=disable' Encountered an error running "coder server", see "coder server --help" for more information error: make logger: no loggers provided, use /dev/null to disable logging # /dev/null passes the logger check. The server prints no log lines and goes on to the database. $ coder server --log-human=/dev/null --postgres-url 'postgres://127.0.0.1:1/x?sslmode=disable' Encountered an error running "coder server", see "coder server --help" for more information error: connect to postgres: connect to postgres: unable to connect after 11 tries; last error: dial tcp 127.0.0.1:1: connect: connection refused ``` </details> ## Linear Fixes DOCS-1583, DOCS-1585, DOCS-1586, DOCS-1587, DOCS-1588, DOCS-1589, DOCS-1590, DOCS-1591, DOCS-1592, DOCS-1593, DOCS-1594, DOCS-1595, DOCS-1596, DOCS-1597, DOCS-1598, DOCS-1599, DOCS-1601, DOCS-1602, DOCS-1603, DOCS-1604, DOCS-1605, DOCS-1606, DOCS-1607, DOCS-1608, DOCS-1609, DOCS-1614, DOCS-1618, DOCS-1619, DOCS-1620, DOCS-1624, DOCS-1635, DOCS-1639, DOCS-1644, DOCS-1651, DOCS-1652, DOCS-1653, DOCS-1654, DOCS-1655, DOCS-1656, DOCS-1657, DOCS-1658, DOCS-1659, DOCS-1660, DOCS-1661, DOCS-1662, DOCS-1663, DOCS-1664, DOCS-1665, DOCS-1666, DOCS-1667, DOCS-1668, DOCS-1670, DOCS-1671, DOCS-1672, DOCS-1673, DOCS-1674, DOCS-1675, DOCS-1676, DOCS-1677, DOCS-1678, DOCS-1679, DOCS-1680, DOCS-1681, DOCS-1682, DOCS-1684, DOCS-1688, DOCS-1690, DOCS-1691, DOCS-1692, DOCS-1693, DOCS-1694, DOCS-1695, DOCS-1696, DOCS-1697, DOCS-1698, DOCS-1699, DOCS-1700, DOCS-1701, DOCS-1702, DOCS-1704, DOCS-1705, DOCS-1706, DOCS-1707, DOCS-1708, DOCS-1709, DOCS-1710, DOCS-1711, DOCS-1712, DOCS-1713, DOCS-1714, DOCS-1715, DOCS-1716, DOCS-1717, DOCS-1718, DOCS-1719, DOCS-1720, DOCS-1721, DOCS-1722, DOCS-1723, DOCS-1725, DOCS-1726, DOCS-1727 Part of DOCS-1600, DOCS-1689, DOCS-1669, DOCS-1687 <details> <summary>Decision log</summary> ### Scope - One PR for all findings, as agreed with the reviewer. The code fixes are separate commits so an owner can ask for any of them to move to its own PR. Nothing in the PR is a breaking change, so the title has no `!`. - A finding was skipped only when an open PR already covers it, or when every available wording would assert an undecided product behavior. - Each validation comment on the Linear issues records the evidence, the corrections to the original finding, and what was not verified. ### Docs or code - The default was a docs-only fix that describes what the code does. - A code fix was chosen when the docs describe a behavior the code plainly intended and the code change is small and testable: the packaged env file, the log sink check, and the log line for the environment variable presence check. - The `error` merge strategy and the nice value ordering started as code fixes and went back to docs-only after review. The first breaks existing template versions. The second weakens a deliberate security ordering. - The SCIM `userName` finding (DOCS-1609) took the docs path. The handler is unchanged. ### Wording choices worth a second look - `feature-stages.md` says `--experiments=*` enables only the experiments marked safe, and points to the "Get safe experiments" API. It does not list the other experiments, because the list would drift and those experiments are hidden on purpose. The `--experiments` flag help now says the same. - `agent-metadata.md` replaces the writes-per-second formula with a description of the batcher (each server replica writes one statement every 5 seconds, or sooner at 500 entries). - `extending-templates/index.md` no longer calls `startup_script` deprecated. The provider documents it as an alias for a `coder_script`. - `postgres-ssl.md` separates client verification from server verification, so the server verification steps work alone. - `rpm.md`, `upgrade.md`, and the 2.29 to 2.34 ESR upgrade page explain the empty `CODER_HTTP_ADDRESS=` line that older packages shipped, because upgrades keep the old file. The behavior changed in v2.34.0 with the serpent upgrade in #24926. - The keep-alive script in `docs/reference/index.md` reads the current deadline and skips the request when it is already later. The comparison is on text, so it applies only when the API returns a UTC timestamp, which needs the database to use the UTC time zone. Otherwise the script sets the deadline on every pass, as it did before, and the page says so. - `external-auth.md` anchors the new GitHub Enterprise regex example. The older unanchored examples on that page and on `git-providers.md` are not changed here. ### Self-review findings that were rejected - "The builder opens for Template Admins": the Templates page needs a successful deployment config read, which the Template Admin role does not have. - "Organization Admins can set AI model prices": the price write authorizes against a site-level object, which organization-scoped grants do not match. - "The provisioner chart sets no default resources": `helm/libcoder/templates/_coder.yaml` applies 2000m and 4096Mi when `coder.resources` is empty. ### Automated review, round 1 The `coder-agents-review` app left 45 findings on `8853332`. Fixed: the two reverts above, the anchored regex, the `coder.env` follow-ups (Lima examples, upgrade notes, a comment in the file, and a test that covers every option in the file), the clilog change, the two option descriptions, and the docs findings (CORS behavior, token lifetime example, keep-alive script, agent metadata, refresh token errors, `switch` form type, `dbcrypt` skip list, and a stray line in a YAML example). Declined, with the reason on each thread: - Refactoring the `agentexec` logging and renaming its log field: not related to a docs conflict. - Rewording the "process priority management not enabled (linux-only)" log message: the docs quote the message, and the wording predates this PR. - Problems in code and docs that this PR does not change (the `Capabilities` subtest, the Helm values comment, the AWS provider path in `airgap.md`, and others in the review's out-of-scope list). ### Automated review, round 2 The app left 14 new findings on `04925bd`, and accepted the three declined threads on `cli/agent.go`. All 14 are fixed. The main one: the round 1 change that removed the "no loggers provided" error went too far, because an empty log location would then start a server with no logs. The error is back for every case except `/dev/null`. Reported by the app as out of scope and not changed: the swagger annotation for `GET /api/v2/experiments/available` declares an array, but the handler returns an object with a `safe` array. Three Storybook fixtures keep the old `--experiments` description. ### Automated review, round 3 The app confirmed that all round 2 findings are fixed, and left 10 small findings on `f55cb6d` (8 P4, 2 Nit). All are fixed: the TLS condition and the release range in the `coder.env` notes, one place for the `/dev/null` rule in `clilog`, two test cases, a clearer test failure, and the CPU example in `agent-metadata.md`. ### Automated review, rounds 4 and 5 Round 4 left one P4 on `8fffb38` (name the `--trace` option in the logging note), which is fixed. Round 5 on `2ce754e` reports no new findings and says every earlier finding is fixed or closed. The app did not post an approval: it leaves its out-of-scope items and the code fixes to a human reviewer. Two threads on `cli/agent.go` stay open on purpose for the code owner. ### Doc-check review The doc-check workflow left 12 comments on `2ce754e` (10 blocking, 2 suggestions). All are fixed in `c888885`: "refer to" or "visit" in place of "see" for cross-references on the lines this PR added, the **New template** label as the dashboard shows it, one instruction per step in the server verification procedure of `postgres-ssl.md`, and two wording changes in `process-priority.md`. The `CAP_SYS_NICE` sentence keeps a short reason, which differs from the suggested text. The `coder-agents-review` app then reviewed that commit (rounds 6 and 7). Round 6 left two P4 findings, fixed in `30b2c79`: the `failed to adjust niceness` message is described once, in the troubleshooting entry, with where it appears, and the client verification steps in `postgres-ssl.md` render as one numbered list. Round 7 reports no new findings. `2a6035c` then softens one sentence from that fix, because the agent logs the stderr of devcontainer CLI commands. The app was not run again on that commit. ### Adjacent gaps found and not fixed here - `add-envbuilder.md` has the same `zip` command that `template-from-scratch.md` had. - `template-from-scratch.md` leaves the reader in `build/` after `cd $_`, before a step that runs from `template-tour`. </details> --- _Generated by Coder Agents on behalf of @nickvigilante._ --------- Co-authored-by: Ben Potter <me@bpmct.net>
332 KiB
title
| title |
|---|
| Prometheus |
Coder exposes many metrics which can be consumed by a Prometheus server, and give insight into the current state of a live Coder deployment.
If you don't have a Prometheus server installed, you can follow the Prometheus Getting started guide.
Enable Prometheus metrics
Coder server exports metrics via the HTTP endpoint, which can be enabled using either the environment variable CODER_PROMETHEUS_ENABLE or the flag --prometheus-enable.
The Prometheus endpoint address is http://localhost:2112/ by default.
You can use either the environment variable CODER_PROMETHEUS_ADDRESS or the flag --prometheus-address <network-interface>:<port> to select a different listen address.
If coder server --prometheus-enable is started locally, you can preview the metrics endpoint in your browser or with curl:
$ curl http://localhost:2112/
# HELP coderd_api_active_users_duration_hour The number of users that have been active within the last hour.
# TYPE coderd_api_active_users_duration_hour gauge
coderd_api_active_users_duration_hour 0
...
Kubernetes deployment
The Prometheus endpoint can be enabled in the Helm chart's values.yml by setting CODER_PROMETHEUS_ENABLE=true.
Once enabled, the environment variable CODER_PROMETHEUS_ADDRESS will be set by default to 0.0.0.0:2112.
A Service Endpoint will not be exposed.
If you need to expose the Prometheus port on a Service, (for example, to use a ServiceMonitor), create a separate headless service instead.
apiVersion: v1
kind: Service
metadata:
name: coder-prom
namespace: coder
labels:
app.kubernetes.io/name: coder-prom
spec:
clusterIP: None
ports:
- name: prom-http
port: 2112
protocol: TCP
targetPort: 2112
selector:
app.kubernetes.io/instance: coder
app.kubernetes.io/name: coder
type: ClusterIP
Prometheus configuration
To allow Prometheus to scrape the Coder metrics, you will need to create a scrape_config in your prometheus.yml file, or in the Prometheus Helm chart values.
The following is an example scrape_config.
scrape_configs:
- job_name: "coder"
scheme: "http"
static_configs:
# replace with the the IP address of the Coder pod or server
- targets: ["<ip>:2112"]
labels:
apps: "coder"
If you run a standalone AI gateway, each replica exports its own metrics on its own listener.
Its Helm chart uses the same 0.0.0.0:2112 default as the coder chart, but sets up no scrape discovery.
Refer to AI gateway monitoring for more details.
To use the Kubernetes Prometheus operator to scrape metrics, you will need to create a ServiceMonitor in your Coder deployment namespace.
The following is an example ServiceMonitor.
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: coder-service-monitor
namespace: coder
spec:
endpoints:
- port: prom-http
interval: 10s
scrapeTimeout: 10s
namespaceSelector:
matchNames:
- coder
selector:
matchLabels:
app.kubernetes.io/name: coder-prom
Available metrics
You must first enable coderd_agentstats_* with the flag --prometheus-collect-agent-stats, or the environment variable CODER_PROMETHEUS_COLLECT_AGENT_STATS before they can be retrieved from the deployment.
They will always be available from the agent.
coderd_workspace_last_used_timestamp_seconds is also off by default.
Refer to Workspace last used time.
The coder_ai_gateway_cost_control_* metrics are exported only by coderd.
Every coderd replica exports the full set of coderd_insights_* series, each computed over that replica's own rolling 5 minute window, so aggregate across replicas with max, not sum.
| Name | Type | Description | Labels |
|---|---|---|---|
agent_boundary_log_proxy_batches_dropped_total |
counter | Total number of boundary log batches dropped before reaching coderd. Reason: buffer_full = the agent's internal buffer is full, meaning boundary is producing logs faster than the agent can forward them to coderd; forward_failed = the agent failed to send the batch to coderd, potentially because coderd is unreachable or the connection was interrupted. | reason |
agent_boundary_log_proxy_batches_forwarded_total |
counter | Total number of boundary log batches successfully forwarded to coderd. Compare with batches_dropped_total to compute a drop rate. | |
agent_boundary_log_proxy_logs_dropped_total |
counter | Total number of individual boundary log entries dropped before reaching coderd. Reason: buffer_full = the agent's internal buffer is full; forward_failed = the agent failed to send the batch to coderd; boundary_channel_full = boundary's internal send channel overflowed, meaning boundary is generating logs faster than it can batch and send them; boundary_batch_full = boundary's outgoing batch buffer overflowed after a failed flush, meaning boundary could not write to the agent's socket. | reason |
agent_scripts_executed_total |
counter | Total number of scripts executed by the Coder agent. Includes cron scheduled scripts. | agent_name success template_name username workspace_name |
coder_ai_gateway_circuit_breaker_rejects_total |
counter | Total number of requests rejected due to open circuit breaker. | endpoint model provider |
coder_ai_gateway_circuit_breaker_state |
gauge | Current state of the circuit breaker (0=closed, 0.5=half-open, 1=open). | endpoint model provider |
coder_ai_gateway_circuit_breaker_trips_total |
counter | Total number of times the circuit breaker transitioned to open state. | endpoint model provider |
coder_ai_gateway_cost_control_blocked_requests_total |
counter | The number of AI requests blocked because the initiator's budget was exceeded. | group_id |
coder_ai_gateway_cost_control_blocked_users |
gauge | The number of users currently over their AI budget. | group_id |
coder_ai_gateway_cost_control_enforcement_duration_seconds |
histogram | The duration of AI budget enforcement checks, in seconds (outcome: allowed, blocked, error). | outcome |
coder_ai_gateway_cost_control_unpriced_token_usage_records_total |
counter | The number of recorded AI token-usage records for which no price was found for either the requested model or the provider-reported model. provider is the provider instance name, provider_type is its configured type, model is the requested model, and provider_model is the model reported by the provider (empty when not reported). | model provider_model provider_type provider |
coder_ai_gateway_injected_tool_invocations_total |
counter | The number of times an injected MCP tool was invoked by AI Gateway. | model name provider server |
coder_ai_gateway_interceptions_duration_seconds |
histogram | The total duration of intercepted requests, in seconds. The majority of this time will be the upstream processing of the request. AI Gateway has no control over upstream processing time, so it's just an illustrative metric. | model provider |
coder_ai_gateway_interceptions_inflight |
gauge | The number of intercepted requests which are being processed. | model provider route |
coder_ai_gateway_interceptions_total |
counter | The count of intercepted requests. | client initiator_id method model provider route status |
coder_ai_gateway_key_pool_exhaustions_total |
counter | The number of times the key pool was exhausted with no usable key (outcome: rate_limited, auth_failed). | outcome provider |
coder_ai_gateway_key_pool_failover_attempts |
histogram | The number of keys attempted before success or exhaustion, per interception for bridged requests and per request for passthrough requests. | provider |
coder_ai_gateway_key_pool_state |
gauge | The number of keys currently in each state (state: valid, temporary, permanent). | provider state |
coder_ai_gateway_key_pool_state_transitions_total |
counter | The number of API key state transitions during failover (reason: rate_limited, unauthorized). | provider reason |
coder_ai_gateway_non_injected_tool_selections_total |
counter | The number of times an AI model selected a tool to be invoked by the client. | model name provider |
coder_ai_gateway_passthrough_total |
counter | The count of requests which were not intercepted but passed through to the upstream. | method provider route |
coder_ai_gateway_prompts_total |
counter | The number of prompts issued by users (initiators). | client initiator_id model provider |
coder_ai_gateway_provider_info |
gauge | One series per configured AI provider. Value is always 1; the status label (enabled, disabled, error) carries the alertable signal. | provider_name provider_type status |
coder_ai_gateway_providers_last_reload_success_timestamp_seconds |
gauge | Unix timestamp of the last provider reload that successfully refreshed the providers. A gap against the providers_last_reload_timestamp_seconds gauge means the loop is firing but the refresh function is failing. | |
coder_ai_gateway_providers_last_reload_timestamp_seconds |
gauge | Unix timestamp of the last provider reload attempt, success or failure. | |
coder_ai_gateway_proxy_connect_sessions_total |
counter | Total number of CONNECT sessions established. | type |
coder_ai_gateway_proxy_inflight_mitm_requests |
gauge | Number of MITM requests currently being processed. | provider |
coder_ai_gateway_proxy_mitm_requests_total |
counter | Total number of MITM requests handled by the proxy. | provider |
coder_ai_gateway_proxy_mitm_responses_total |
counter | Total number of MITM responses by HTTP status code class. | code provider |
coder_ai_gateway_proxy_provider_info |
gauge | One series per configured AI provider. Value is always 1; the status label (enabled, disabled, error, proxy_excluded) carries the alertable signal. | provider_name provider_type status |
coder_ai_gateway_proxy_providers_last_reload_success_timestamp_seconds |
gauge | Unix timestamp of the last provider reload that successfully refreshed the router. A gap against the providers_last_reload_timestamp_seconds gauge means the loop is firing but the refresh function is failing. | |
coder_ai_gateway_proxy_providers_last_reload_timestamp_seconds |
gauge | Unix timestamp of the last provider reload attempt, success or failure. | |
coder_ai_gateway_tokens_total |
counter | The number of tokens used by intercepted requests. | client initiator_id model provider type |
coder_derp_server_accepts_total |
counter | Total DERP connections accepted. | |
coder_derp_server_average_queue_duration_ms |
gauge | Average queue duration in milliseconds. | |
coder_derp_server_bytes_received_total |
counter | Total bytes received. | |
coder_derp_server_bytes_sent_total |
counter | Total bytes sent. | |
coder_derp_server_clients |
gauge | Total clients (local + remote). | |
coder_derp_server_clients_local |
gauge | Local clients. | |
coder_derp_server_clients_remote |
gauge | Remote (mesh) clients. | |
coder_derp_server_connections |
gauge | Current DERP connections. | |
coder_derp_server_got_ping_total |
counter | Total pings received. | |
coder_derp_server_home_connections |
gauge | Current home DERP connections. | |
coder_derp_server_home_moves_in_total |
counter | Total home moves in. | |
coder_derp_server_home_moves_out_total |
counter | Total home moves out. | |
coder_derp_server_packets_dropped_reason_total |
counter | Packets dropped by reason. | reason |
coder_derp_server_packets_dropped_total |
counter | Total packets dropped. | |
coder_derp_server_packets_dropped_type_total |
counter | Packets dropped by type. | type |
coder_derp_server_packets_forwarded_in_total |
counter | Total packets forwarded in from mesh peers. | |
coder_derp_server_packets_forwarded_out_total |
counter | Total packets forwarded out to mesh peers. | |
coder_derp_server_packets_received_kind_total |
counter | Packets received by kind. | kind |
coder_derp_server_packets_received_total |
counter | Total packets received. | |
coder_derp_server_packets_sent_total |
counter | Total packets sent. | |
coder_derp_server_peer_gone_disconnected_total |
counter | Total peer gone (disconnected) frames sent. | |
coder_derp_server_peer_gone_not_here_total |
counter | Total peer gone (not here) frames sent. | |
coder_derp_server_sent_pong_total |
counter | Total pongs sent. | |
coder_derp_server_unknown_frames_total |
counter | Total unknown frames received. | |
coder_derp_server_watchers |
gauge | Current watchers. | |
coder_pubsub_connected |
gauge | Whether we are connected (1) or not connected (0) to the backend | backend |
coder_pubsub_current_events |
gauge | The current number of pubsub event channels listened for | backend |
coder_pubsub_current_subscribers |
gauge | The current number of active pubsub subscribers | backend |
coder_pubsub_disconnections_total |
counter | Total number of times we disconnected unexpectedly from the backend | backend |
coder_pubsub_latency_measure_errs_total |
counter | The number of pubsub latency measurement failures | backend |
coder_pubsub_latency_measures_total |
counter | The number of pubsub latency measurements | backend |
coder_pubsub_messages_total |
counter | Total number of messages received | backend size |
coder_pubsub_published_bytes_total |
counter | Total number of bytes successfully published across all publishes | backend |
coder_pubsub_publishes_total |
counter | Total number of calls to Publish | backend success |
coder_pubsub_receive_duration_seconds |
histogram | The time taken to receive a message from a pubsub event channel | backend |
coder_pubsub_receive_latency_seconds |
gauge | The most recent time taken to receive a message from a pubsub event channel. New integrations should prefer the coder_pubsub_receive_duration_seconds histogram. | backend |
coder_pubsub_received_bytes_total |
counter | Total number of bytes received across all messages | backend |
coder_pubsub_send_duration_seconds |
histogram | The time taken to send a message into a pubsub event channel | backend |
coder_pubsub_send_latency_seconds |
gauge | The most recent time taken to send a message into a pubsub event channel. New integrations should prefer the coder_pubsub_send_duration_seconds histogram. | backend |
coder_pubsub_subscribes_total |
counter | Total number of calls to Subscribe/SubscribeWithErr | backend success |
coder_servertailnet_agent_unreachable_total |
counter | Number of connection attempts where the workspace agent did not answer in time, by reason: no_node (this server has no node for the agent), peer_lost (the agent's or this server's coordinator connection ended), no_handshake (no WireGuard handshake yet), handshake_stale (last handshake over 180s ago), handshake_ok (last handshake under 180s ago), diagnostics_timeout (peer state not readable within 1s), client_canceled (client left within 5s). | reason |
coder_servertailnet_await_reachable_seconds |
histogram | Time until a workspace agent answered the first ping when a new connection was opened. | |
coder_servertailnet_connections_total |
counter | Total number of TCP connections made to workspace agents. | network |
coder_servertailnet_derp_connects_total |
counter | Number of times the server tailnet connected to the embedded DERP relay. Stays 0 when the embedded relay is disabled. | |
coder_servertailnet_open_connections |
gauge | Total number of TCP connections currently open to workspace agents. | network |
coder_workspaceapps_agent_offline_total |
counter | Number of workspace app requests that got the offline page because the agent was not connected. | access_method status |
coderd_agentapi_metadata_batch_size |
histogram | Total number of metadata entries in each batch, updated before flushes. | |
coderd_agentapi_metadata_batch_utilization |
histogram | Number of metadata keys per agent in each batch, updated before flushes. | |
coderd_agentapi_metadata_batches_total |
counter | Total number of metadata batches flushed. | reason |
coderd_agentapi_metadata_dropped_keys_total |
counter | Total number of metadata keys dropped due to capacity limits. | |
coderd_agentapi_metadata_flush_duration_seconds |
histogram | Time taken to flush metadata batch to database and pubsub. | reason |
coderd_agentapi_metadata_flushed_total |
counter | Total number of unique metadatas flushed. | |
coderd_agentapi_metadata_publish_errors_total |
counter | Total number of metadata batch pubsub publish calls that have resulted in an error. | |
coderd_agents_apps |
gauge | Agent applications with statuses. | agent_name app_name health username workspace_name |
coderd_agents_connection_latencies_seconds |
gauge | Agent connection latencies in seconds. | agent_name derp_region preferred username workspace_name |
coderd_agents_connections |
gauge | Agent connections with statuses. | agent_name lifecycle_state status tailnet_node username workspace_name |
coderd_agents_first_connection_seconds |
histogram | Duration from agent creation to first connection in seconds. | agent_name template_name |
coderd_agents_up |
gauge | The number of active agents per workspace. | template_name template_version username workspace_name |
coderd_agentstats_app_info |
gauge | The current family of each registered session app. Value is always 1. | app_name family |
coderd_agentstats_connection_count |
gauge | The number of established connections by agent | agent_name username workspace_name |
coderd_agentstats_connection_median_latency_seconds |
gauge | The median agent connection latency in seconds | agent_name username workspace_name |
coderd_agentstats_currently_reachable_peers |
gauge | The number of peers (e.g. clients) that are currently reachable over the encrypted network. | agent_name connection_type template_name username workspace_name |
coderd_agentstats_rx_bytes |
gauge | Agent Rx bytes | agent_name username workspace_name |
coderd_agentstats_session_count |
gauge | The number of sessions established by app name | agent_name app_name username workspace_name |
coderd_agentstats_session_count_jetbrains |
gauge | The number of session established by JetBrains | agent_name username workspace_name |
coderd_agentstats_session_count_reconnecting_pty |
gauge | The number of session established by reconnecting PTY | agent_name username workspace_name |
coderd_agentstats_session_count_ssh |
gauge | The number of session established by SSH | agent_name username workspace_name |
coderd_agentstats_session_count_vscode |
gauge | The number of session established by VSCode | agent_name username workspace_name |
coderd_agentstats_session_counts_overflow_total |
counter | Total number of reported session count entries summed into the overflow app after exceeding the per-report cap. | |
coderd_agentstats_startup_script_seconds |
gauge | Amount of time taken to run the startup script in seconds. | agent_name success template_name username workspace_name |
coderd_agentstats_tx_bytes |
gauge | Agent Tx bytes | agent_name username workspace_name |
coderd_api_active_users_duration_hour |
gauge | The number of users that have been active within the last hour. | |
coderd_api_concurrent_requests |
gauge | The number of concurrent API requests. | method path |
coderd_api_concurrent_websockets |
gauge | The total number of concurrent API websockets. | path |
coderd_api_request_latencies_seconds |
histogram | Latency distribution of requests in seconds. | method path |
coderd_api_requests_processed_total |
counter | The total number of processed API requests | code method path |
coderd_api_requests_too_large_total |
counter | The total number of API requests answered 413, by the reason they were rejected. A sustained rate of reason="request_body" on one route is more often a limit set too tight for a legitimate payload than an attempt to exhaust memory. reason="other" counts 413s raised for reasons unrelated to the size of the request body, such as agent log storage overflow or an archive that is too large once expanded. | method path reason |
coderd_api_total_user_count |
gauge | The total number of registered users, partitioned by status. | status |
coderd_api_websocket_durations_seconds |
histogram | Websocket duration distribution of requests in seconds. | path |
coderd_api_websocket_probes_total |
counter | WebSocket liveness probe outcomes by route. Compare rate(...{result="ok"}[1m]) against coderd_api_concurrent_websockets to detect unresponsive WebSocket connections. | path result |
coderd_api_workspace_latest_build |
gauge | The current number of workspace builds by status for all non-deleted workspaces. | status |
coderd_authz_authorize_duration_seconds |
histogram | Duration of the 'Authorize' call in seconds. Only counts calls that succeed. | allowed |
coderd_authz_prepare_authorize_duration_seconds |
histogram | Duration of the 'PrepareAuthorize' call in seconds. | |
coderd_build_info |
gauge | Describes the current build/version of the Coder server. Value is always 1. | revision version |
coderd_chat_auto_archive_records_archived_total |
counter | Total number of chats archived by the auto-archive job (counting both roots and cascaded children). | |
coderd_chat_workspace_upload_total |
counter | Total chat workspace file uploads by HTTP response status. | status |
coderd_chatd_agents_active |
gauge | Deployment-wide number of chats holding a concurrent-agent capacity slot. Every replica reports the same database-derived value; aggregate with max, not sum. | pool |
coderd_chatd_agents_queued_for_capacity |
gauge | Deployment-wide number of chats waiting for a concurrent-agent capacity slot. Every replica reports the same database-derived value; aggregate with max, not sum. | pool |
coderd_chatd_chats |
gauge | Number of chats being processed, by state. | state |
coderd_chatd_compaction_total |
counter | Total compaction outcomes (only recorded when compaction was triggered or failed). | model provider result |
coderd_chatd_find_tools_activations_total |
counter | Total deferred tool activations returned by find_tools. | |
coderd_chatd_find_tools_calls_total |
counter | Total find_tools calls. | |
coderd_chatd_find_tools_empty_total |
counter | Total find_tools calls with no matches. | |
coderd_chatd_find_tools_match_count |
histogram | Number of matches returned by find_tools calls. | |
coderd_chatd_hook_context_size_bytes |
histogram | Lifecycle hook model context response size in bytes. | event |
coderd_chatd_hook_decisions_total |
counter | Total lifecycle hook permission decisions by event and decision. | decision event |
coderd_chatd_hook_dispatch_seconds |
histogram | Lifecycle hook dispatch duration in seconds. | event |
coderd_chatd_hook_dispatches_total |
counter | Total lifecycle hook dispatches by event and result. | event result |
coderd_chatd_hook_input_overrides_total |
counter | Total lifecycle hook input overrides by event. | event |
coderd_chatd_message_count |
histogram | Number of messages in the prompt per LLM request. | model provider |
coderd_chatd_model_stage_duration_seconds |
histogram | Wall time of the stages that are a provider's work on a model: stream (open to close) and provider_attempt (one HTTP round trip, closed on response headers). Time to first token per model is ttft_seconds. Failed stream and provider_attempt windows are observed. Observed only for turn-scoped stages, which are also observed on stage_duration_seconds; background-scoped model stages appear on stage_duration_seconds only. provider_type is the configured AI provider type (for example bedrock), not the wire protocol other chatd metrics report as provider. Registered only with the chat-stage-metrics experiment. | chat_kind model provider_type stage |
coderd_chatd_prompt_size_bytes |
histogram | Estimated byte size of the prompt per LLM request. | model provider |
coderd_chatd_response_size_bytes |
histogram | Byte size of streamed model response content (assistant text, reasoning, and tool-call input deltas) per LLM request. | model provider |
coderd_chatd_stage_anomalies_total |
counter | Chat lifecycle stage observations dropped or adjusted, by reason: negative_elapsed and inverted_window (end before start, clock skew), missing_timestamp (unset start or end), future_start (start ahead of this replica's clock, span started now), stale_anchor (trigger not after the previous turn's anchor, turn started now). Registered only with the chat-stage-metrics experiment. | reason |
coderd_chatd_stage_duration_seconds |
histogram | Wall time spent in each chat lifecycle stage. Stages overlap, so this is a stage-time profile, not a partition of the turn. scope separates latency attributable to a prompt (stages inside a chat turn, and queue_wait before it) from detached background work; chat_kind is empty without a known chat. Observed: chat_turn, queue_wait, acquisition, mcp_connect, stream, time_to_first_token, provider_attempt, tool_call, commit, retry_backoff; other stages are span-only. Failed windows are observed, except time_to_first_token, which observes only windows closed by an output part, and chat_turn, which observes only completed turns. chat_turn runs from its trigger (the prompt, a submitted client tool result, or a compaction request) to the end of its turn, includes acquisition, and excludes queue_wait and time waiting in requires_action, so a prompt that waits in requires_action is observed once per segment. queue_wait runs from a queued message's creation to its promotion, so it includes time the message spent under edit. Registered only with the chat-stage-metrics experiment. | chat_kind scope stage |
coderd_chatd_steps_total |
counter | Total agentic loop steps across all chats. | model provider |
coderd_chatd_stream_retries_total |
counter | Total LLM stream retries. | kind model provider |
coderd_chatd_tool_errors_total |
counter | Total tool calls that returned an error result. | model provider tool_name |
coderd_chatd_tool_result_size_bytes |
histogram | Size in bytes of each tool execution result. | model provider tool_name |
coderd_chatd_tool_result_truncated_total |
counter | Total tool results truncated to fit the model context window. | model provider tool_name |
coderd_chatd_ttft_seconds |
histogram | Time-to-first-token: wall time from LLM request to the first streamed output part: any part other than warnings, keepalives, finish, or error, including start markers such as text_start. Attempts that end before an output part, by failing or by finishing empty, are not observed. The time_to_first_token stage of stage_duration_seconds measures the same window without a model label and is registered only with the chat-stage-metrics experiment. | model provider |
coderd_chatd_turn_outcomes_total |
counter | Total closed chat turns by outcome: completed (the turn finished, including a turn that ends when the chat enters requires_action; the submitted tool result opens a new turn), interrupted (an interrupt request from a user, an API client, or a parent agent stopped it), error (a failure stopped it), abandoned (a newer prompt, such as a message edit, or runner exit closed it before it finished; an edit sent while the chat is interrupting can close the stopped turn as abandoned rather than interrupted). Every closed turn is counted exactly once. Registered only with the chat-stage-metrics experiment. | chat_kind outcome |
coderd_db_query_counts_total |
counter | Total number of queries labeled by HTTP route, method, and query name. | method query route |
coderd_db_query_latencies_seconds |
histogram | Latency distribution of queries in seconds. | query |
coderd_db_tx_duration_seconds |
histogram | Duration of transactions in seconds. | success tx_id |
coderd_db_tx_executions_count |
counter | Total count of transactions executed. 'retries' is expected to be 0 for a successful transaction. | retries success tx_id |
coderd_dbpurge_chat_search_rows_backfilled_total |
counter | Total number of chat message rows whose search_tsv was backfilled. | |
coderd_dbpurge_iteration_duration_seconds |
histogram | Duration of each dbpurge iteration in seconds. | success |
coderd_dbpurge_records_purged_total |
counter | Total number of records purged by type. | record_type |
coderd_experiments |
gauge | Indicates whether each experiment is enabled (1) or not (0) | experiment |
coderd_insights_applications_usage_seconds |
gauge | The app usage per template. connection_method is workspace_app for template apps, or ssh, reconnecting_pty, or port_forwarding for built-in apps, which have one series per app name in slug. Overlapping apps each count their own minutes. | connection_method display_name organization_name slug template_name |
coderd_insights_parameters |
gauge | The parameter usage per template. | organization_name parameter_name parameter_type parameter_value template_name |
coderd_insights_templates_active_users |
gauge | The number of active users of the template. | organization_name template_name |
coderd_license_active_users |
gauge | The number of active users. | |
coderd_license_errors |
gauge | The number of active license errors. | |
coderd_license_limit_users |
gauge | The user seats limit based on the active Coder license. | |
coderd_license_user_limit_enabled |
gauge | Returns 1 if the current license enforces the user limit. | |
coderd_license_warnings |
gauge | The number of active license warnings. | |
coderd_lifecycle_autobuild_execution_duration_seconds |
histogram | Duration of each autobuild execution. | |
coderd_notifications_dispatcher_send_seconds |
histogram | The time taken to dispatch notifications. | method |
coderd_notifications_inflight_dispatches |
gauge | The number of dispatch attempts which are currently in progress. | method notification_template_id |
coderd_notifications_pending_updates |
gauge | The number of dispatch attempt results waiting to be flushed to the store. | |
coderd_notifications_queued_seconds |
histogram | The time elapsed between a notification being enqueued in the store and retrieved for dispatching (measures the latency of the notifications system). This should generally be within CODER_NOTIFICATIONS_FETCH_INTERVAL seconds; higher values for a sustained period indicates delayed processing and CODER_NOTIFICATIONS_LEASE_COUNT can be increased to accommodate this. | method |
coderd_notifications_retry_count |
counter | The count of notification dispatch retry attempts. | method notification_template_id |
coderd_notifications_synced_updates_total |
counter | The number of dispatch attempt results flushed to the store. | |
coderd_oauth2_external_requests_rate_limit |
gauge | The total number of allowed requests per interval. | name resource |
coderd_oauth2_external_requests_rate_limit_next_reset_unix |
gauge | Unix timestamp for when the next interval starts | name resource |
coderd_oauth2_external_requests_rate_limit_remaining |
gauge | The remaining number of allowed requests in this interval. | name resource |
coderd_oauth2_external_requests_rate_limit_reset_in_seconds |
gauge | Seconds until the next interval | name resource |
coderd_oauth2_external_requests_rate_limit_used |
gauge | The number of requests made in this interval. | name resource |
coderd_oauth2_external_requests_rate_limited_total |
counter | The total number of api calls to external oauth2 providers that returned a rate-limited response (a 429, or a 403 with rate-limit headers). | name source status_code |
coderd_oauth2_external_requests_total |
counter | The total number of api calls made to external oauth2 providers. 'status_code' will be 0 if the request failed with no response. | name source status_code |
coderd_open_file_refs_current |
gauge | The count of file references currently open in the file cache. Multiple references can be held for the same file. | |
coderd_open_file_refs_total |
counter | The total number of file references ever opened in the file cache. The 'hit' label indicates if the file was loaded from the cache. | hit |
coderd_open_files_current |
gauge | The count of unique files currently open in the file cache. | |
coderd_open_files_size_bytes_current |
gauge | The current amount of memory of all files currently open in the file cache. | |
coderd_open_files_size_bytes_total |
counter | The total amount of memory ever opened in the file cache. This number never decrements. | |
coderd_open_files_total |
counter | The total count of unique files ever opened in the file cache. | |
coderd_prebuilds_reconciliation_duration_seconds |
histogram | Duration of each prebuilds reconciliation cycle. | |
coderd_prebuilt_workspace_claim_duration_seconds |
histogram | Time to claim a prebuilt workspace by organization, template, and preset. | organization_name preset_name template_name |
coderd_prebuilt_workspaces_claimed_total |
counter | Total number of prebuilt workspaces which were claimed by users. Claiming refers to creating a workspace with a preset selected for which eligible prebuilt workspaces are available and one is reassigned to a user. | organization_name preset_name template_name |
coderd_prebuilt_workspaces_created_total |
counter | Total number of prebuilt workspaces that have been created to meet the desired instance count of each template preset. | organization_name preset_name template_name |
coderd_prebuilt_workspaces_desired |
gauge | Target number of prebuilt workspaces that should be available for each template preset. | organization_name preset_name template_name |
coderd_prebuilt_workspaces_eligible |
gauge | Current number of prebuilt workspaces that are eligible to be claimed by users. These are workspaces that have completed their build process with their agent reporting 'ready' status. | organization_name preset_name template_name |
coderd_prebuilt_workspaces_failed_total |
counter | Total number of prebuilt workspaces that failed to build. | organization_name preset_name template_name |
coderd_prebuilt_workspaces_metrics_last_updated |
gauge | The unix timestamp when the metrics related to prebuilt workspaces were last updated; these metrics are cached. | |
coderd_prebuilt_workspaces_preset_hard_limited |
gauge | Indicates whether a given preset has reached the hard failure limit (1 = hard-limited). Metric is omitted otherwise. | organization_name preset_name template_name |
coderd_prebuilt_workspaces_preset_validation_failed |
gauge | Indicates whether a given preset has validation failures (1 = validation failed). Metric is omitted otherwise. | organization_name preset_name template_name |
coderd_prebuilt_workspaces_reconciliation_paused |
gauge | Indicates whether prebuilds reconciliation is currently paused (1 = paused, 0 = not paused). | |
coderd_prebuilt_workspaces_resource_replacements_total |
counter | Total number of prebuilt workspaces whose resource(s) got replaced upon being claimed. In Terraform, drift on immutable attributes results in resource replacement. This represents a worst-case scenario for prebuilt workspaces because the pre-provisioned resource would have been recreated when claiming, thus obviating the point of pre-provisioning. See https://coder.com/docs/admin/templates/extending-templates/prebuilt-workspaces#preventing-resource-replacement | organization_name preset_name template_name |
coderd_prebuilt_workspaces_running |
gauge | Current number of prebuilt workspaces that are in a running state. These workspaces have started successfully but may not yet be claimable by users (see coderd_prebuilt_workspaces_eligible). | organization_name preset_name template_name |
coderd_prometheusmetrics_agents_execution_seconds |
histogram | Histogram for duration of agents metrics collection in seconds. | |
coderd_prometheusmetrics_agentstats_execution_seconds |
histogram | Histogram for duration of agent stats metrics collection in seconds. | |
coderd_prometheusmetrics_metrics_aggregator_execution_cleanup_seconds |
histogram | Histogram for duration of metrics aggregator cleanup in seconds. | |
coderd_prometheusmetrics_metrics_aggregator_execution_update_seconds |
histogram | Histogram for duration of metrics aggregator update in seconds. | |
coderd_prometheusmetrics_metrics_aggregator_store_size |
gauge | The number of metrics stored in the aggregator | |
coderd_provisioner_job_queue_wait_seconds |
histogram | Time from job creation to acquisition by a provisioner daemon. | build_reason job_type provisioner_type transition |
coderd_provisionerd_job_timings_seconds |
histogram | The provisioner job time duration in seconds. | provisioner status |
coderd_provisionerd_jobs_current |
gauge | The number of currently running provisioner jobs. | provisioner |
coderd_provisionerd_num_daemons |
gauge | The number of provisioner daemons. | |
coderd_provisionerd_workspace_build_timings_seconds |
histogram | The time taken for a workspace to build. | status template_name template_version workspace_transition |
coderd_proxyhealth_health_check_duration_seconds |
histogram | Histogram for duration of proxy health collection in seconds. | |
coderd_proxyhealth_health_check_results |
gauge | This endpoint returns a number to indicate the health status. -3 (unknown), -2 (Unreachable), -1 (Unhealthy), 0 (Unregistered), 1 (Healthy) | proxy_id |
coderd_template_workspace_build_duration_seconds |
histogram | Duration from workspace build creation to agent ready, by template. | is_prebuild organization_name status template_name transition |
coderd_usage_events_expired |
gauge | Number of unpublished usage events aged 30 to 60 days. Events older than 60 days are not counted. | |
coderd_usage_events_pending |
gauge | Number of unpublished usage events within the 30-day publishing window. | |
coderd_usage_events_pending_oldest_age_seconds |
gauge | Age in seconds of the oldest unpublished usage event within the 30-day publishing window. Zero when no events are pending. | |
coderd_usage_events_publish_results_total |
counter | Total usage events accepted or rejected after a valid Tallyman response, by result and event type. | event_type result |
coderd_usage_events_publish_send_errors_total |
counter | Total usage event publish attempts that did not receive a valid Tallyman response. Does not reset after a valid response. | |
coderd_workspace_builds_enqueued_total |
counter | Total number of workspace build enqueue attempts. | build_reason provisioner_type status transition |
coderd_workspace_builds_total |
counter | The number of workspaces started, updated, or deleted. | status template_name template_version workspace_name workspace_owner workspace_transition |
coderd_workspace_creation_duration_seconds |
histogram | Time to create a workspace by organization, template, preset, and type (regular or prebuild). | organization_name preset_name template_name type |
coderd_workspace_creation_total |
counter | Total regular (non-prebuilt) workspace creations by organization, template, and preset. | organization_name preset_name template_name |
coderd_workspace_last_used_timestamp_seconds |
gauge | The last time each workspace was used, as a Unix timestamp in seconds. Every replica reports the same database-derived value; aggregate with max, not sum. | organization_name template_name workspace_id workspace_name workspace_owner |
coderd_workspace_latest_build_status |
gauge | The current workspace statuses by template, transition, and owner for all non-deleted workspaces. | status template_name template_version workspace_owner workspace_transition |
go_gc_duration_seconds |
summary | A summary of the pause duration of garbage collection cycles. | |
go_goroutines |
gauge | Number of goroutines that currently exist. | |
go_info |
gauge | Information about the Go environment. | version |
go_memstats_alloc_bytes |
gauge | Number of bytes allocated and still in use. | |
go_memstats_alloc_bytes_total |
counter | Total number of bytes allocated, even if freed. | |
go_memstats_buck_hash_sys_bytes |
gauge | Number of bytes used by the profiling bucket hash table. | |
go_memstats_frees_total |
counter | Total number of frees. | |
go_memstats_gc_sys_bytes |
gauge | Number of bytes used for garbage collection system metadata. | |
go_memstats_heap_alloc_bytes |
gauge | Number of heap bytes allocated and still in use. | |
go_memstats_heap_idle_bytes |
gauge | Number of heap bytes waiting to be used. | |
go_memstats_heap_inuse_bytes |
gauge | Number of heap bytes that are in use. | |
go_memstats_heap_objects |
gauge | Number of allocated objects. | |
go_memstats_heap_released_bytes |
gauge | Number of heap bytes released to OS. | |
go_memstats_heap_sys_bytes |
gauge | Number of heap bytes obtained from system. | |
go_memstats_last_gc_time_seconds |
gauge | Number of seconds since 1970 of last garbage collection. | |
go_memstats_lookups_total |
counter | Total number of pointer lookups. | |
go_memstats_mallocs_total |
counter | Total number of mallocs. | |
go_memstats_mcache_inuse_bytes |
gauge | Number of bytes in use by mcache structures. | |
go_memstats_mcache_sys_bytes |
gauge | Number of bytes used for mcache structures obtained from system. | |
go_memstats_mspan_inuse_bytes |
gauge | Number of bytes in use by mspan structures. | |
go_memstats_mspan_sys_bytes |
gauge | Number of bytes used for mspan structures obtained from system. | |
go_memstats_next_gc_bytes |
gauge | Number of heap bytes when next garbage collection will take place. | |
go_memstats_other_sys_bytes |
gauge | Number of bytes used for other system allocations. | |
go_memstats_stack_inuse_bytes |
gauge | Number of bytes in use by the stack allocator. | |
go_memstats_stack_sys_bytes |
gauge | Number of bytes obtained from system for stack allocator. | |
go_memstats_sys_bytes |
gauge | Number of bytes obtained from system. | |
go_threads |
gauge | Number of OS threads created. | |
process_cpu_seconds_total |
counter | Total user and system CPU time spent in seconds. | |
process_max_fds |
gauge | Maximum number of open file descriptors. | |
process_open_fds |
gauge | Number of open file descriptors. | |
process_resident_memory_bytes |
gauge | Resident memory size in bytes. | |
process_start_time_seconds |
gauge | Start time of the process since unix epoch in seconds. | |
process_virtual_memory_bytes |
gauge | Virtual memory size in bytes. | |
process_virtual_memory_max_bytes |
gauge | Maximum amount of virtual memory available in bytes. | |
promhttp_metric_handler_requests_in_flight |
gauge | Current number of scrapes being served. | |
promhttp_metric_handler_requests_total |
counter | Total number of scrapes by HTTP status code. | code |
Application usage by template
coderd_insights_applications_usage_seconds reports app usage per template over the last five minutes.
The connection_method label distinguishes template apps from built-in apps:
connection_method |
Apps | slug |
display_name |
|---|---|---|---|
workspace_app |
Apps defined in the template, opened through Coder | Slug or port | The app's display name |
ssh, reconnecting_pty, port_forwarding |
Built-in apps that connect through the agent, such as desktop IDEs | Reported app name | Registered name, such as Cursor; Unknown for unknown, Other apps for overflow, and otherwise the app name |
A template app can share a slug with a built-in app, such as vscode, so identify an app by connection_method and slug.
display_name can change between releases.
Built-in apps have one series per app name, such as vscode, cursor, or goland, and only for apps used in the window.
Each app counts its own minutes, so a user with VS Code and Cursor open in the same minute counts in both series.
slug="unknown" holds sessions reported without an app name, or with the reserved name overflow.
Per template, every built-in app Coder recognizes gets a series, and so do the 64 busiest other apps.
The rest add up under one series, connection_method="ssh" with slug="overflow", which can count a minute more than once.
The template insights page in the dashboard uses the same rule, but applies the limit across all selected templates instead of per template.
Upgrade from the family series
This metric changed in three ways:
application_nameis nowdisplay_name.- Every series has the new
connection_methodlabel. - Built-in apps report one series per app instead of one per app family. The family series had
slug=""and were reported even at zero.
Selectors with a value for application_name, or with slug="", return nothing.
These queries still return data, with different values:
- A template app query by
slugalso matches a built-in app with the same name. Addconnection_method="workspace_app". - A total without a
slugselector now includes every built-in app, includingport_forwarding,sftp, andunknown.
Update Grafana dashboards, alerting rules, and recording rules:
| Old series | Closest new series |
|---|---|
{slug="", application_name="Visual Studio Code"} |
{connection_method="ssh", slug="vscode"}, plus its forks such as cursor |
{slug="", application_name="JetBrains"} |
{connection_method="ssh", slug="jetbrains"}, plus its IDEs such as goland |
{slug="", application_name="Web Terminal"} |
{connection_method="reconnecting_pty", slug="reconnecting_pty"} |
{slug="", application_name="SSH"} |
{connection_method="ssh", slug="ssh"}, plus zed |
{application_name="<app>"} for a template app |
{connection_method="workspace_app", display_name="<app>"} |
To total a family, sum its apps, for example sum by (template_name, organization_name) (max by (template_name, organization_name, slug, connection_method) (coderd_insights_applications_usage_seconds{connection_method="ssh", slug=~"vscode|cursor|windsurf"})).
The sum counts a minute twice if two of the apps were open at once.
In the example, the inner max by keeps one series per app, and the outer sum adds the apps for each template.
Workspace last used time
coderd_workspace_last_used_timestamp_seconds reports when each workspace was last used, as a Unix timestamp in seconds.
It's the value the dormancy threshold measures inactivity from.
The metric is off by default.
Enable it with the flag --prometheus-collect-workspace-stats or the environment variable CODER_PROMETHEUS_COLLECT_WORKSPACE_STATS.
Series and cost
- Every workspace has a series, including stopped and unclaimed prebuilt workspaces. Unclaimed prebuilt workspaces have
workspace_owner="prebuilds". - The
workspace_idlabel identifies the workspace, so a new workspace that reuses a deleted workspace's name gets its own series. - Deleting a workspace or its owner removes its series.
- Every
coderdreplica reports every series, so aggregate withmax, notsum. - Each replica adds one active series per workspace to your Prometheus server.
What counts as use
- Traffic on any connection to the workspace agent counts, such as SSH, an IDE, the web terminal, or port forwarding.
- Traffic to a workspace app through Coder counts.
- By default, a running
coder sshorcoder port-forwardcommand counts once a minute, even without traffic.
The value is also set to the current time when:
- The workspace is created.
- Someone activates a dormant workspace.
- A template admin resets the inactivity period of the template's workspaces.
Accuracy
With default settings, the value trails real use by up to about 2.5 minutes, plus your scrape interval:
| Step | Default interval |
|---|---|
| The workspace agent reports activity | 30 seconds |
| Coder saves activity, with the time of the save, not of the activity | 1 minute |
| The metric refreshes | 1 minute |
Use the metric for alerts measured in hours or days.
Example
This query returns workspaces that nobody has used for 30 days, excluding unclaimed prebuilt workspaces:
time() - max by (organization_name, template_name, workspace_owner, workspace_name, workspace_id) (coderd_workspace_last_used_timestamp_seconds{workspace_owner!="prebuilds"}) > 30 * 24 * 3600
Note on Prometheus native histogram support
The following metrics support native histograms:
coderd_workspace_creation_duration_secondscoderd_prebuilt_workspace_claim_duration_secondscoderd_template_workspace_build_duration_secondscoder_pubsub_send_duration_secondscoder_pubsub_receive_duration_secondscoderd_provisioner_job_queue_wait_secondscoder_ai_gateway_cost_control_enforcement_duration_seconds
Native histograms are an experimental Prometheus feature that removes the need to predefine bucket boundaries and allows higher-resolution buckets that adapt to deployment characteristics. Whether a metric is exposed as classic or native depends entirely on the Prometheus server configuration (see Prometheus docs for details):
- If native histograms are enabled, Prometheus ingests the high-resolution histogram.
- If not, it falls back to the predefined buckets.
⚠️ Important: classic and native histograms cannot be aggregated together. If Prometheus is switched from classic to native at a certain point in time, dashboards might need to account for that transition. For this reason, it’s recommended to follow Prometheus’ migration guidelines when moving from classic to native histograms.