## Summary Admins who start from the External provisioners docs had no way to find out which Prometheus metrics exist for provisioners, or how to turn them on for standalone provisioner daemons. This PR documents both, and turns the deployment metrics page into a signpost to the monitoring docs for each area. ## Changes - **`docs/install/operate/provisioners/index.md`:** expands the "Prometheus metrics" section. The heading and its `#prometheus-metrics` anchor are unchanged. - It explains which metrics each provisioner daemon reports and which the control plane reports. For built-in provisioners, the control plane also reports the daemon metrics. - Two tables list each metric's purpose and link to the generated [Available metrics](https://coder.com/docs/admin/integrations/prometheus#available-metrics) table for types and labels. - It explains how to read the queue wait metric. Jobs whose tags no provisioner matches don't appear in it until a provisioner acquires them. - It shows how to turn on the daemon's Prometheus endpoint with `--prometheus-enable` / `CODER_PROMETHEUS_ENABLE` and `--prometheus-address` / `CODER_PROMETHEUS_ADDRESS`. It corrects the stated default address from `localhost:2112` to `127.0.0.1:2112`. - It covers the `coder-provisioner` Helm chart: the fixed `0.0.0.0:2112` address, the `prometheus-http` container port, and the missing Service. It adds a Pod discovery example with `coder.podAnnotations` and a `PodMonitor` alternative. - **`docs/install/operate/provisioners/manage-provisioner-jobs.md`:** links from the troubleshooting section to the metrics section. - **`docs/admin/monitoring/metrics.md`:** becomes a signpost page. - It keeps the control plane setup links and the `#setting-up-metrics` anchor. - It adds a table of monitoring pages per area (provisioners, workspace proxies, prebuilt workspaces, AI Gateway, Agent Relay), and notes which areas serve metrics on their own endpoint. - It links to the full metrics list. No new pages, so `docs/manifest.json` is unchanged. ## Testing - `make lint/emdash`, `go run ./scripts/docshtmlcheck`, markdownlint, and `markdown-table-formatter` are clean on the changed files. - Vale reports 0 errors and 0 warnings. Its 4 suggestions are on lines this PR doesn't change. - The pre-commit hook (`pre-commit-light`, including `lint/docs-redirects`, `lint/markdown`, and `lint/style-claims`) passed. - A script resolved every relative link and anchor in the changed files. <details> <summary>Verified against</summary> - `provisionerd/provisionerd.go` (`NewMetrics`): names of the five provisioner daemon metrics. - `provisionerd/runner/runner.go`: where the job and build metrics are recorded. - `cli/server.go:1273-1296`: the control plane registers the daemon metrics only for built-in provisioners, and sets `num_daemons` to the built-in count. `cli/server.go:385-389` registers the provisioner server metrics. - `coderd/provisionerdserver/provisionerdserver.go:1011-1019`: `coderd_provisioner_job_queue_wait_seconds` is recorded only when a provisioner daemon acquires a job. `coderd/coderd.go:2602` and `enterprise/coderd/provisionerdaemons.go:365` pass the same metrics to built-in and external provisioners. - `coderd/jobreaper/detector.go:26-28`: a pending job that nothing acquires is failed after 30 minutes, without a queue wait observation. - `coderd/wsbuilder/metrics.go` and `coderd/wsbuilder/wsbuilder.go:357-373`: `coderd_workspace_builds_enqueued_total`. - `enterprise/cli/provisionerdaemonstart.go:204-218, 371-383`: the flags, env vars, and `127.0.0.1:2112` default. A standalone daemon sets `num_daemons` to 1 and registers the Go, process, and `promhttp` collectors. - `cli/agent.go:603-611`: the metrics handler serves every path, so the default `/metrics` scrape path works. - `helm/provisioner/templates/_coder.tpl:40-41, 81-96`: the fixed address, and the `prometheus-http` port that appears only when `CODER_PROMETHEUS_ENABLE` is `"true"`. The chart has no Service template. - `helm/libcoder/templates/_coder.yaml:24-33`, `helm/libcoder/templates/_helpers.tpl:20-22`, and `helm/provisioner/tests/testdata/labels_annotations.golden`: `coder.podAnnotations` and `coder.podLabels` land on the Pod template, which carries the `app.kubernetes.io/name` and `app.kubernetes.io/instance` labels. - `enterprise/cli/proxyserver.go:243-250`: workspace proxies serve metrics on their own listener. - Tests that pin these claims: `coderd/workspaces_test.go` (queue wait), `helm/provisioner/tests/testdata/default_values.golden` (fixed address), and `enterprise/cli/testdata/coder_provisioner_start_--help.golden` (CLI default). </details> <details> <summary>Decision log</summary> - **Kept the `## Prometheus metrics` heading.** Renaming it gained nothing and would break external links to `#prometheus-metrics`. The AI Gateway and Agent Relay pages that the signpost table links to use the same heading. - **Kept `<a id="setting-up-metrics"></a>` on `metrics.md`.** External links to that anchor keep working, and `scripts/docshtmlcheck` passes with it. - **Dropped the Type and Labels columns.** `scripts/metricsdocgen` fills one marked block in `prometheus.md`, so it can't generate these tables. Types and labels drift the most, so the page now points to the generated table for them. Extending the generator, or adding a check that metrics named in `docs/` exist in `scripts/metricsdocgen/generated_metrics`, would need a separate PR. - **Rewrote the queue wait guidance.** The earlier text said a rising value can mean no provisioner matches a job's tags. The code records the value only when a provisioner acquires a job, so unmatched jobs never reach the metric. The page now links to the existing troubleshooting steps for finding them, instead of restating a command. - **Reworded the signpost intro.** The workspace proxies page explains how to turn on metrics but doesn't list them, so the table describes each linked page as covering monitoring, not listing metrics. - **"Separate endpoint" column.** Checked in code for workspace proxies. Agent Relay has no server code in this repo, so its "Yes" comes from `docs/agent-relay/monitoring/index.md`. </details> <details> <summary>Related gaps not edited</summary> - `docs/admin/integrations/prometheus.md:14` and `docs/admin/networking/workspace-proxies.md:221` still say `localhost:2112`. The server and proxy default is `127.0.0.1:2112` (`codersdk/deployment.go:2658`). - `docs/workspaces/speed-up-templates.md:42-44` implies that control plane metrics include `coderd_provisionerd_job_timings_seconds`. That holds only for built-in provisioners. - `docs/admin/networking/workspace-proxies.md#observability` explains how to turn on the endpoint but doesn't list any proxy metrics. - `docs/admin/networking/` (DERP and tailnet, for example `coder_derp_server_*` and `coder_servertailnet_*`) has no metrics section. - `docs/admin/monitoring/health-check.md` links to the database metrics flag but doesn't list the `coderd_db_*` metrics. - `docs/admin/monitoring/notifications/` doesn't list the `coderd_notifications_*` metrics. - `docs/admin/users/` (`coderd_api_active_users_duration_hour`, license metrics) and `docs/admin/security/audit-logs.md` have no metrics sections. - `docs/templates/extending-templates/prebuilt-workspaces.md#available-metrics` lists metrics without types or labels, and doesn't say how to turn them on. </details> Disclosure: these changes were generated by Coder Agents. Closes DOCS-892 Supersedes #31804, reopened under a human author for attribution. --------- Co-authored-by: Coder Prototypes <294797144+coder-prototypes@users.noreply.github.com>
2.3 KiB
2.3 KiB
title
| title |
|---|
| Deployment metrics |
Coder exposes Prometheus metrics that show the current state of a live Coder deployment. This page helps administrators find the metrics for each part of a deployment and turn them on.
If you don't have a Prometheus server, follow the Prometheus Getting started guide.
Set up metrics
The Prometheus integration guide covers setup for the control plane:
- Enable Prometheus metrics in the control plane.
- Enable the Prometheus endpoint in Helm if you deploy on Kubernetes.
- Configure Prometheus to scrape Coder metrics.
Some components run as separate processes and serve metrics on their own endpoints. Turn on and scrape each of them separately. The pages in the next section explain how.
Metrics by area
Each of these pages covers monitoring for one part of a deployment:
| Area | Page | Separate endpoint |
|---|---|---|
| External provisioners | Provisioner metrics | Yes |
| Workspace proxies | Workspace proxy observability | Yes |
| Prebuilt workspaces | Prebuilt workspace metrics | No |
| AI Gateway | AI Gateway Prometheus metrics | Standalone only |
| Agent Relay | Agent Relay Prometheus metrics | Yes |
All metrics
For every metric Coder exports, with its type, description, and labels, refer to Available metrics.