Files
Nick VigilanteandCoder Prototypes feb6e809c4 docs: add provisioner metrics and a metrics signpost page (#31977)
## Summary

Admins who start from the External provisioners docs had no way to find
out which Prometheus metrics exist for provisioners, or how to turn them
on for standalone provisioner daemons.
This PR documents both, and turns the deployment metrics page into a
signpost to the monitoring docs for each area.

## Changes

- **`docs/install/operate/provisioners/index.md`:** expands the
"Prometheus metrics" section. The heading and its `#prometheus-metrics`
anchor are unchanged.
- It explains which metrics each provisioner daemon reports and which
the control plane reports. For built-in provisioners, the control plane
also reports the daemon metrics.
- Two tables list each metric's purpose and link to the generated
[Available
metrics](https://coder.com/docs/admin/integrations/prometheus#available-metrics)
table for types and labels.
- It explains how to read the queue wait metric. Jobs whose tags no
provisioner matches don't appear in it until a provisioner acquires
them.
- It shows how to turn on the daemon's Prometheus endpoint with
`--prometheus-enable` / `CODER_PROMETHEUS_ENABLE` and
`--prometheus-address` / `CODER_PROMETHEUS_ADDRESS`. It corrects the
stated default address from `localhost:2112` to `127.0.0.1:2112`.
- It covers the `coder-provisioner` Helm chart: the fixed `0.0.0.0:2112`
address, the `prometheus-http` container port, and the missing Service.
It adds a Pod discovery example with `coder.podAnnotations` and a
`PodMonitor` alternative.
- **`docs/install/operate/provisioners/manage-provisioner-jobs.md`:**
links from the troubleshooting section to the metrics section.
- **`docs/admin/monitoring/metrics.md`:** becomes a signpost page.
- It keeps the control plane setup links and the `#setting-up-metrics`
anchor.
- It adds a table of monitoring pages per area (provisioners, workspace
proxies, prebuilt workspaces, AI Gateway, Agent Relay), and notes which
areas serve metrics on their own endpoint.
  - It links to the full metrics list.

No new pages, so `docs/manifest.json` is unchanged.

## Testing

- `make lint/emdash`, `go run ./scripts/docshtmlcheck`, markdownlint,
and `markdown-table-formatter` are clean on the changed files.
- Vale reports 0 errors and 0 warnings. Its 4 suggestions are on lines
this PR doesn't change.
- The pre-commit hook (`pre-commit-light`, including
`lint/docs-redirects`, `lint/markdown`, and `lint/style-claims`) passed.
- A script resolved every relative link and anchor in the changed files.

<details>
<summary>Verified against</summary>

- `provisionerd/provisionerd.go` (`NewMetrics`): names of the five
provisioner daemon metrics.
- `provisionerd/runner/runner.go`: where the job and build metrics are
recorded.
- `cli/server.go:1273-1296`: the control plane registers the daemon
metrics only for built-in provisioners, and sets `num_daemons` to the
built-in count. `cli/server.go:385-389` registers the provisioner server
metrics.
- `coderd/provisionerdserver/provisionerdserver.go:1011-1019`:
`coderd_provisioner_job_queue_wait_seconds` is recorded only when a
provisioner daemon acquires a job. `coderd/coderd.go:2602` and
`enterprise/coderd/provisionerdaemons.go:365` pass the same metrics to
built-in and external provisioners.
- `coderd/jobreaper/detector.go:26-28`: a pending job that nothing
acquires is failed after 30 minutes, without a queue wait observation.
- `coderd/wsbuilder/metrics.go` and
`coderd/wsbuilder/wsbuilder.go:357-373`:
`coderd_workspace_builds_enqueued_total`.
- `enterprise/cli/provisionerdaemonstart.go:204-218, 371-383`: the
flags, env vars, and `127.0.0.1:2112` default. A standalone daemon sets
`num_daemons` to 1 and registers the Go, process, and `promhttp`
collectors.
- `cli/agent.go:603-611`: the metrics handler serves every path, so the
default `/metrics` scrape path works.
- `helm/provisioner/templates/_coder.tpl:40-41, 81-96`: the fixed
address, and the `prometheus-http` port that appears only when
`CODER_PROMETHEUS_ENABLE` is `"true"`. The chart has no Service
template.
- `helm/libcoder/templates/_coder.yaml:24-33`,
`helm/libcoder/templates/_helpers.tpl:20-22`, and
`helm/provisioner/tests/testdata/labels_annotations.golden`:
`coder.podAnnotations` and `coder.podLabels` land on the Pod template,
which carries the `app.kubernetes.io/name` and
`app.kubernetes.io/instance` labels.
- `enterprise/cli/proxyserver.go:243-250`: workspace proxies serve
metrics on their own listener.
- Tests that pin these claims: `coderd/workspaces_test.go` (queue wait),
`helm/provisioner/tests/testdata/default_values.golden` (fixed address),
and `enterprise/cli/testdata/coder_provisioner_start_--help.golden` (CLI
default).

</details>

<details>
<summary>Decision log</summary>

- **Kept the `## Prometheus metrics` heading.** Renaming it gained
nothing and would break external links to `#prometheus-metrics`. The AI
Gateway and Agent Relay pages that the signpost table links to use the
same heading.
- **Kept `<a id="setting-up-metrics"></a>` on `metrics.md`.** External
links to that anchor keep working, and `scripts/docshtmlcheck` passes
with it.
- **Dropped the Type and Labels columns.** `scripts/metricsdocgen` fills
one marked block in `prometheus.md`, so it can't generate these tables.
Types and labels drift the most, so the page now points to the generated
table for them. Extending the generator, or adding a check that metrics
named in `docs/` exist in `scripts/metricsdocgen/generated_metrics`,
would need a separate PR.
- **Rewrote the queue wait guidance.** The earlier text said a rising
value can mean no provisioner matches a job's tags. The code records the
value only when a provisioner acquires a job, so unmatched jobs never
reach the metric. The page now links to the existing troubleshooting
steps for finding them, instead of restating a command.
- **Reworded the signpost intro.** The workspace proxies page explains
how to turn on metrics but doesn't list them, so the table describes
each linked page as covering monitoring, not listing metrics.
- **"Separate endpoint" column.** Checked in code for workspace proxies.
Agent Relay has no server code in this repo, so its "Yes" comes from
`docs/agent-relay/monitoring/index.md`.

</details>

<details>
<summary>Related gaps not edited</summary>

- `docs/admin/integrations/prometheus.md:14` and
`docs/admin/networking/workspace-proxies.md:221` still say
`localhost:2112`. The server and proxy default is `127.0.0.1:2112`
(`codersdk/deployment.go:2658`).
- `docs/workspaces/speed-up-templates.md:42-44` implies that control
plane metrics include `coderd_provisionerd_job_timings_seconds`. That
holds only for built-in provisioners.
- `docs/admin/networking/workspace-proxies.md#observability` explains
how to turn on the endpoint but doesn't list any proxy metrics.
- `docs/admin/networking/` (DERP and tailnet, for example
`coder_derp_server_*` and `coder_servertailnet_*`) has no metrics
section.
- `docs/admin/monitoring/health-check.md` links to the database metrics
flag but doesn't list the `coderd_db_*` metrics.
- `docs/admin/monitoring/notifications/` doesn't list the
`coderd_notifications_*` metrics.
- `docs/admin/users/` (`coderd_api_active_users_duration_hour`, license
metrics) and `docs/admin/security/audit-logs.md` have no metrics
sections.
-
`docs/templates/extending-templates/prebuilt-workspaces.md#available-metrics`
lists metrics without types or labels, and doesn't say how to turn them
on.

</details>

Disclosure: these changes were generated by Coder Agents.

Closes DOCS-892


Supersedes #31804, reopened under a human author for attribution.

---------

Co-authored-by: Coder Prototypes <294797144+coder-prototypes@users.noreply.github.com>
2026-11-19 12:59:31 -05:00
..