Files
Nick VigilanteandBen Potter 1baff57cb0 fix: resolve docs and code conflicts found in the main docs audit (#32158)
## Summary

Addresses 106 of the 113 conflicts between the docs and the code that
the November audit of `main` found: 102 are fixed and 4 have an interim
docs fix. Every finding was re-checked against the code before it was
fixed. Most fixes are docs-only. Three are small code fixes where the
docs described the intended behavior and the code was wrong, each in its
own commits so it can be split out. No change in this PR alters template
build or agent behavior.

## Changes

Fixes by area (finding counts): templates 28, install 13, integrations
and monitoring 13, auth and security 12, networking 11, Coder Agents 9,
AI governance 7, reference 7, workspaces 6.

Code fixes. Each needs a look from the code owner. If an owner prefers
not to take one, drop its commits and I will change the docs to describe
the current behavior instead.

| Commits | Change | Risk to check |
|---------|--------|---------------|
| `fix: keep built-in listener defaults in packaged coder.env`, and
three follow-up commits | `coder.env` comments out `CODER_HTTP_ADDRESS=`
and the three `CODER_TLS_*` lines, so an untouched package install
listens on `127.0.0.1:3000` as the docs say. Only the
`CODER_HTTP_ADDRESS=` line changed behavior. The TLS options default to
empty, so commenting them out is for consistency. The Lima examples
rewrite the commented line and fail if the line is not found. A test
compares every option that the file sets against the built-in defaults.
| The deb and RPM packages were not built or installed. Every released
package through v2.39.0 ships the empty line, and upgrades keep an
existing file (`config\|noreplace`), so the upgrade pages now say how to
fix it. |
| `fix(cli/clilog): accept log sinks that are all disabled`, and three
follow-up commits | `--log-human=/dev/null` with no other log location
no longer fails with "no loggers provided". It is the documented way to
disable logging. An empty location still fails with that error when no
other sink is set, as before. | Behavior change: with `/dev/null`, the
server and the provisioner daemon can start with no log output. |
| `fix(cli): detect CODER_PROC_PRIO_MGMT by presence`, and a comment
follow-up | Log line only. With the variable set to an empty value, the
agent now logs that process priority management is enabled, which is
what `agentexec` already does. No detection or behavior change. | Low. |

Two option descriptions in `codersdk/deployment.go` are corrected, with
the CLI reference, configuration reference, and golden files
regenerated: the default quiet hours schedule (the max deadline is the
quiet hours window on the next day that the template's autostop
requirement applies) and `--experiments` (`*` opts in to the experiments
marked safe, also in `enterprise/cli/provisionerdaemonstart.go`).

### Code fixes that were reverted after review

Two code fixes from the first version of this PR are reverted in the
branch. The net diff has no change under `agent/` or `coderd/`. The docs
now describe what the code does, and both issues stay open for the
owner.

- **`error` merge strategy for `coder_env` (DOCS-1669).** The fix made
`merge_strategy = "error"` fail whenever a name was set twice, whatever
the resource order. Existing template versions that pair an `error`
entry with a later `replace` or `append` entry would start to fail every
build after `terraform apply`. That is a breaking change that needs its
own PR and release note. The fix and its tests are in `dff8e7c6ce` and
can be cherry-picked. `environment-variables.md` now says the check runs
only when the `error` resource is processed.
- **Nice value before the capability drop (DOCS-1687).** The fix let
`coder agent-exec` apply a negative nice value by calling `setpriority`
before it drops effective capabilities. The current order is deliberate:
with the fix, any local process could run the agent binary with
`--coder-nice=-20` when the binary has the `CAP_SYS_NICE` file
capability. `process-priority.md` now says a nice value below the
agent's own is not applied, and no longer tells admins to grant
`CAP_SYS_NICE`.

## Not included

Seven findings are left out.

- DOCS-1629: covered by #31639.
- DOCS-1703 and DOCS-1724: covered by #31980.
- DOCS-1584, DOCS-1683, DOCS-1685, DOCS-1686: each needs an owner
decision, and no neutral docs wording exists until that decision is
made. The validation comment on each issue names the decision.

Four fixes are interim, so their issues stay open:

- DOCS-1600: the AI Gateway pages now agree that the legacy metric names
are deprecated and "will be removed in a future release". The code still
emits them, and its TODO names v2.37, which has shipped. The owner still
needs to set the removal release.
- DOCS-1689: the page now says **New Template** opens the builder only
for users with the Owner role, because the page gates on read access to
the deployment config. Whether Template Admins should get the builder is
still open.
- DOCS-1669 and DOCS-1687: docs-only for now. Refer to [Code fixes that
were reverted after
review](#code-fixes-that-were-reverted-after-review).

## Review notes

- Claims that describe product intent and need an owner's yes: the
Premium qualifier on per-organization workspace sharing (the API route
has no license check, the dashboard needs multiple organizations),
`organization` port sharing in OSS, the telemetry field list, and the
`1-5` cron range in the prebuilds schedule example.
- Not run end to end: the PostgreSQL TLS procedure on a cluster, the
OpenShift and air-gapped examples, the BSD `date` and `tar` forms on
macOS, the Windows command on the multiple Kubernetes clusters page, and
the keep-alive script in `docs/reference/index.md` (only its comparison
logic was run, against sample values). Nothing was checked on a live
deployment.
- Likely merge conflicts with open PRs: #31990 (`mcp-server.md`,
`feature-stages.md`, `docker-in-workspaces.md`), #31982
(`workspace-management.md`), #31980 (`health-check.md`), #31977
(`manage-provisioner-jobs.md`), #31869 (`airgap.md`), #30486
(`add-envbuilder.md`), and #31326 and #31531 (`codersdk/deployment.go`
and generated files).

## Testing

- `make pre-commit` (gen, fmt, lint, slim build), before the PR opened
and again after the review changes.
- `go test` for `./cli/clilog/...`, `./codersdk/`, `./cli/`
(`TestServer`, `TestAgent`, and golden files), and `./enterprise/cli/`
(provisioner, server, and help golden files). `./agent/agentexec/...`
and `./coderd/provisionerdserver/...` pass after the reverts. Linux
only. CI covers macOS and Windows.
- The `coder.env` test fails against the file on `main` and passes
against the new one. It reads the file with `godotenv`, which accepts
CRLF line endings.
- An empty `CODER_HTTP_ADDRESS=` was confirmed end to end with a full
build of the server (output below).
- The new `sed` and `grep` lines in the Lima examples were run with GNU
tools against the old and the new form of `coder.env`. The Lima VMs were
not started.
- `--log-human=/dev/null` and `CODER_LOGGING_HUMAN=` were confirmed with
the same build (output below).
- `coder agent-exec --coder-nice=-5 -- echo` from a full build, as a
non-root user: `coder-agent: failed to adjust niceness to -5 ...:
permission denied` on stderr, the command output on stdout, exit code 0.
- `postgres-ssl.md` rendered with the GitHub Markdown API: two ordered
lists with three and five items, where `main` has six lists.
- markdownlint, the table formatter, `make lint/emdash`, `make
lint/docs-html`, `make lint/docs-manifest`, `make lint/docs-redirects`,
`make lint/style-claims`, and Vale on the changed lines.
- A doc-check self-review of the docs diff by four independent
reviewers. It found six problems, which are fixed. The accuracy and
coverage lenses ran on every area. The language and structure lenses ran
in full on only part of the diff.

<details>
<summary>Test and terminal output for the code fixes (Linux)</summary>

```console
$ go test ./codersdk/ -run TestPackagedCoderEnvKeepsDefaults -count=1 -v
--- PASS: TestPackagedCoderEnvKeepsDefaults (0.00s)
ok  	github.com/coder/coder/v2/codersdk	0.071s

$ go test ./cli/clilog/ -run TestBuilder -count=1 -v
--- PASS: TestBuilder (0.00s)
    --- PASS: TestBuilder/NoSinks (0.00s)
        --- PASS: TestBuilder/NoSinks/JSONDevNull (0.00s)
        --- PASS: TestBuilder/NoSinks/StackdriverDevNull (0.00s)
        --- PASS: TestBuilder/NoSinks/NoConfiguration (0.00s)
        --- PASS: TestBuilder/NoSinks/HumanDevNull (0.00s)
    --- PASS: TestBuilder/WithHuman (0.00s)
    --- PASS: TestBuilder/WithFilter (0.00s)
    --- PASS: TestBuilder/WithJSON (0.00s)
    --- PASS: TestBuilder/Verbose (0.00s)
    --- PASS: TestBuilder/FromDeploymentValues (0.01s)
        --- PASS: TestBuilder/FromDeploymentValues/Defaults (0.00s)
        --- PASS: TestBuilder/FromDeploymentValues/EmptyHumanEnv (0.00s)
        --- PASS: TestBuilder/FromDeploymentValues/Override (0.00s)
ok  	github.com/coder/coder/v2/cli/clilog	0.171s

$ TMPDIR=$HOME/.cache/agentexec-tmp go test ./agent/agentexec/... ./coderd/provisionerdserver/... -count=1
ok  	github.com/coder/coder/v2/agent/agentexec	4.236s
ok  	github.com/coder/coder/v2/coderd/provisionerdserver	2.404s

# Full build of the server (go build ./cmd/coder). An empty value stops it before it connects to the database.
$ CODER_HTTP_ADDRESS= coder server --postgres-url 'postgres://127.0.0.1:1/x?sslmode=disable'
Encountered an error running "coder server", see "coder server --help" for more information
error: TLS is disabled. Enable with --tls-enable or specify a HTTP address

# With the variable unset, the server passes that check and goes on to the database.
$ coder server --postgres-url 'postgres://127.0.0.1:1/x?sslmode=disable'
2026-11-13 02:19:41.901 [warn]  ping postgres: retrying  error="dial tcp 127.0.0.1:1: connect: connection refused"  try=11
Encountered an error running "coder server", see "coder server --help" for more information
error: connect to postgres: connect to postgres: unable to connect after 11 tries; last error: dial tcp 127.0.0.1:1: connect: connection refused

# An empty log location with no other sink still stops the server.
$ CODER_LOGGING_HUMAN= coder server --postgres-url 'postgres://127.0.0.1:1/x?sslmode=disable'
Encountered an error running "coder server", see "coder server --help" for more information
error: make logger: no loggers provided, use /dev/null to disable logging

# /dev/null passes the logger check. The server prints no log lines and goes on to the database.
$ coder server --log-human=/dev/null --postgres-url 'postgres://127.0.0.1:1/x?sslmode=disable'
Encountered an error running "coder server", see "coder server --help" for more information
error: connect to postgres: connect to postgres: unable to connect after 11 tries; last error: dial tcp 127.0.0.1:1: connect: connection refused
```

</details>

## Linear

Fixes DOCS-1583, DOCS-1585, DOCS-1586, DOCS-1587, DOCS-1588, DOCS-1589,
DOCS-1590, DOCS-1591, DOCS-1592, DOCS-1593, DOCS-1594, DOCS-1595,
DOCS-1596, DOCS-1597, DOCS-1598, DOCS-1599, DOCS-1601, DOCS-1602,
DOCS-1603, DOCS-1604, DOCS-1605, DOCS-1606, DOCS-1607, DOCS-1608,
DOCS-1609, DOCS-1614, DOCS-1618, DOCS-1619, DOCS-1620, DOCS-1624,
DOCS-1635, DOCS-1639, DOCS-1644, DOCS-1651, DOCS-1652, DOCS-1653,
DOCS-1654, DOCS-1655, DOCS-1656, DOCS-1657, DOCS-1658, DOCS-1659,
DOCS-1660, DOCS-1661, DOCS-1662, DOCS-1663, DOCS-1664, DOCS-1665,
DOCS-1666, DOCS-1667, DOCS-1668, DOCS-1670, DOCS-1671, DOCS-1672,
DOCS-1673, DOCS-1674, DOCS-1675, DOCS-1676, DOCS-1677, DOCS-1678,
DOCS-1679, DOCS-1680, DOCS-1681, DOCS-1682, DOCS-1684, DOCS-1688,
DOCS-1690, DOCS-1691, DOCS-1692, DOCS-1693, DOCS-1694, DOCS-1695,
DOCS-1696, DOCS-1697, DOCS-1698, DOCS-1699, DOCS-1700, DOCS-1701,
DOCS-1702, DOCS-1704, DOCS-1705, DOCS-1706, DOCS-1707, DOCS-1708,
DOCS-1709, DOCS-1710, DOCS-1711, DOCS-1712, DOCS-1713, DOCS-1714,
DOCS-1715, DOCS-1716, DOCS-1717, DOCS-1718, DOCS-1719, DOCS-1720,
DOCS-1721, DOCS-1722, DOCS-1723, DOCS-1725, DOCS-1726, DOCS-1727

Part of DOCS-1600, DOCS-1689, DOCS-1669, DOCS-1687

<details>
<summary>Decision log</summary>

### Scope

- One PR for all findings, as agreed with the reviewer. The code fixes
are separate commits so an owner can ask for any of them to move to its
own PR. Nothing in the PR is a breaking change, so the title has no `!`.
- A finding was skipped only when an open PR already covers it, or when
every available wording would assert an undecided product behavior.
- Each validation comment on the Linear issues records the evidence, the
corrections to the original finding, and what was not verified.

### Docs or code

- The default was a docs-only fix that describes what the code does.
- A code fix was chosen when the docs describe a behavior the code
plainly intended and the code change is small and testable: the packaged
env file, the log sink check, and the log line for the environment
variable presence check.
- The `error` merge strategy and the nice value ordering started as code
fixes and went back to docs-only after review. The first breaks existing
template versions. The second weakens a deliberate security ordering.
- The SCIM `userName` finding (DOCS-1609) took the docs path. The
handler is unchanged.

### Wording choices worth a second look

- `feature-stages.md` says `--experiments=*` enables only the
experiments marked safe, and points to the "Get safe experiments" API.
It does not list the other experiments, because the list would drift and
those experiments are hidden on purpose. The `--experiments` flag help
now says the same.
- `agent-metadata.md` replaces the writes-per-second formula with a
description of the batcher (each server replica writes one statement
every 5 seconds, or sooner at 500 entries).
- `extending-templates/index.md` no longer calls `startup_script`
deprecated. The provider documents it as an alias for a `coder_script`.
- `postgres-ssl.md` separates client verification from server
verification, so the server verification steps work alone.
- `rpm.md`, `upgrade.md`, and the 2.29 to 2.34 ESR upgrade page explain
the empty `CODER_HTTP_ADDRESS=` line that older packages shipped,
because upgrades keep the old file. The behavior changed in v2.34.0 with
the serpent upgrade in #24926.
- The keep-alive script in `docs/reference/index.md` reads the current
deadline and skips the request when it is already later. The comparison
is on text, so it applies only when the API returns a UTC timestamp,
which needs the database to use the UTC time zone. Otherwise the script
sets the deadline on every pass, as it did before, and the page says so.
- `external-auth.md` anchors the new GitHub Enterprise regex example.
The older unanchored examples on that page and on `git-providers.md` are
not changed here.

### Self-review findings that were rejected

- "The builder opens for Template Admins": the Templates page needs a
successful deployment config read, which the Template Admin role does
not have.
- "Organization Admins can set AI model prices": the price write
authorizes against a site-level object, which organization-scoped grants
do not match.
- "The provisioner chart sets no default resources":
`helm/libcoder/templates/_coder.yaml` applies 2000m and 4096Mi when
`coder.resources` is empty.

### Automated review, round 1

The `coder-agents-review` app left 45 findings on `8853332`. Fixed: the
two reverts above, the anchored regex, the `coder.env` follow-ups (Lima
examples, upgrade notes, a comment in the file, and a test that covers
every option in the file), the clilog change, the two option
descriptions, and the docs findings (CORS behavior, token lifetime
example, keep-alive script, agent metadata, refresh token errors,
`switch` form type, `dbcrypt` skip list, and a stray line in a YAML
example).

Declined, with the reason on each thread:

- Refactoring the `agentexec` logging and renaming its log field: not
related to a docs conflict.
- Rewording the "process priority management not enabled (linux-only)"
log message: the docs quote the message, and the wording predates this
PR.
- Problems in code and docs that this PR does not change (the
`Capabilities` subtest, the Helm values comment, the AWS provider path
in `airgap.md`, and others in the review's out-of-scope list).

### Automated review, round 2

The app left 14 new findings on `04925bd`, and accepted the three
declined threads on `cli/agent.go`. All 14 are fixed. The main one: the
round 1 change that removed the "no loggers provided" error went too
far, because an empty log location would then start a server with no
logs. The error is back for every case except `/dev/null`.

Reported by the app as out of scope and not changed: the swagger
annotation for `GET /api/v2/experiments/available` declares an array,
but the handler returns an object with a `safe` array. Three Storybook
fixtures keep the old `--experiments` description.

### Automated review, round 3

The app confirmed that all round 2 findings are fixed, and left 10 small
findings on `f55cb6d` (8 P4, 2 Nit). All are fixed: the TLS condition
and the release range in the `coder.env` notes, one place for the
`/dev/null` rule in `clilog`, two test cases, a clearer test failure,
and the CPU example in `agent-metadata.md`.

### Automated review, rounds 4 and 5

Round 4 left one P4 on `8fffb38` (name the `--trace` option in the
logging note), which is fixed. Round 5 on `2ce754e` reports no new
findings and says every earlier finding is fixed or closed. The app did
not post an approval: it leaves its out-of-scope items and the code
fixes to a human reviewer. Two threads on `cli/agent.go` stay open on
purpose for the code owner.

### Doc-check review

The doc-check workflow left 12 comments on `2ce754e` (10 blocking, 2
suggestions). All are fixed in `c888885`: "refer to" or "visit" in place
of "see" for cross-references on the lines this PR added, the **New
template** label as the dashboard shows it, one instruction per step in
the server verification procedure of `postgres-ssl.md`, and two wording
changes in `process-priority.md`. The `CAP_SYS_NICE` sentence keeps a
short reason, which differs from the suggested text.

The `coder-agents-review` app then reviewed that commit (rounds 6 and
7). Round 6 left two P4 findings, fixed in `30b2c79`: the `failed to
adjust niceness` message is described once, in the troubleshooting
entry, with where it appears, and the client verification steps in
`postgres-ssl.md` render as one numbered list. Round 7 reports no new
findings. `2a6035c` then softens one sentence from that fix, because the
agent logs the stderr of devcontainer CLI commands. The app was not run
again on that commit.

### Adjacent gaps found and not fixed here

- `add-envbuilder.md` has the same `zip` command that
`template-from-scratch.md` had.
- `template-from-scratch.md` leaves the reader in `build/` after `cd
$_`, before a step that runs from `template-tour`.

</details>

---

_Generated by Coder Agents on behalf of @nickvigilante._

---------

Co-authored-by: Ben Potter <me@bpmct.net>
2026-11-13 21:34:59 +00:00

15 KiB

title
title
Scale Coder

This best practice guide helps you prepare a Coder deployment that you can scale up to a high-scale deployment as use grows, and keep it operating smoothly with a high number of active users and workspaces.

Observability

Observability is one of the most important aspects to a scalable Coder deployment. When you have visibility into performance and usage metrics, you can make informed decisions about what changes you should make.

Monitor your Coder deployment with log output and metrics to identify potential bottlenecks before they negatively affect the end-user experience and measure the effects of modifications you make to your deployment.

  • Log output

    • Capture log output from from control plane instances and external provisioner daemons and store them in a searchable log store like Loki, CloudWatch logs, or other tools.
    • Retain logs for a minimum of thirty days, ideally ninety days. This allows you investigate when anomalous behaviors began.
  • Metrics

    • Capture infrastructure metrics like CPU, memory, open files, and network I/O for all control plane, external provisioner daemon, workspace proxy, and PostgreSQL instances.
    • Capture control plane and external provisioner daemon metrics via Prometheus.

Retain metric time series for at least six months. This allows you to see performance trends relative to user growth.

For a more comprehensive overview, integrate metrics with an observability dashboard like Grafana.

Observability key metrics

Configure alerting based on these metrics to make sure you surface problems before they affect the end-user experience.

  • CPU and Memory Utilization

    • Monitor the utilization as a fraction of the available resources on the instance.

      Utilization will vary with use throughout the course of a day, week, and longer timelines. Monitor trends and pay special attention to the daily and weekly peak utilization. Use long-term trends to plan infrastructure upgrades.

  • Tail latency of Coder API requests

    • High tail latency can indicate the control plane or the PostgreSQL database is underprovisioned for the load.
    • Use the coderd_api_request_latencies_seconds metric.
  • Tail latency of database queries

    • High tail latency can indicate the PostgreSQL database is low in resources.
    • Use the coderd_db_query_latencies_seconds metric.

How to capture control plane metrics with Prometheus

Edit your Helm values.yaml to capture metrics from the control plane and external provisioner daemons with Prometheus:

  1. Enable Prometheus metrics:

    CODER_PROMETHEUS_ENABLE=true
    
  2. Enable database metrics:

    CODER_PROMETHEUS_COLLECT_DB_METRICS=true
    
  3. For a high scale deployment, configure agent stats to avoid large cardinality or disable them:

    • Configure agent stats:

      CODER_PROMETHEUS_AGGREGATE_AGENT_STATS_BY=agent_name
      
    • Disable agent stats:

      CODER_PROMETHEUS_COLLECT_AGENT_STATS=false
      

Control plane

Locality

If increased availability of the Coder API is a concern, deploy at least three control plane instances. Spread the instances across nodes with anti-affinity rules in Kubernetes or in different availability zones of the same geographic region.

Do not deploy in different geographic regions.

Control plane instances need to be able to communicate with one another directly with low latency, under 10 ms. Note that this is for the availability of the Coder API. Workspaces are not fault tolerant unless they are explicitly built that way at the template level.

Deploy control plane instances as geographically close to PostgreSQL as possible. Low-latency communication (under 10 ms) with Postgres is essential for control plane performance.

Scaling

The control plane can be scaled both vertically for bigger instances and horizontally for more instances.

Aim to keep the number of control plane instances relatively small, preferably under ten instances, and opt for vertical scale over horizontal scale after meeting availability requirements.

Coder's validated architectures give specific sizing recommendations for various user scales. These are a useful starting point, but very few deployments will remain stable at a predetermined user level over the long term. We recommend monitoring and adjusting resources as needed.

We don't recommend that you autoscale the control plane instances. Instead, scale the deployment for peak weekly usage.

Although the control plane persists no internal state, it operates as a proxy for end users to their workspaces in two capacities:

  1. As an HTTP proxy when they access workspace applications in their browser via the Coder Dashboard.

  2. As a Designated Encrypted Relay for Packets (DERP) proxy for tunneled connections.

    The tunneled connections come from CLI tools like coder ssh and coder port-forward, and from desktop IDEs.

Stopping a control plane instance will (momentarily) disconnect any users currently connecting through that instance. Adding a new instance is not disruptive, but you should remove instances and perform upgrades during a maintenance window to minimize disruption.

Provisioner daemons

Locality

We recommend that you run one or more provisioner daemon deployments external to the control plane and disable provisioner daemons within your control plane. This allows you to scale them independently of the control plane:

CODER_PROVISIONER_DAEMONS=0

We recommend deploying provisioner daemons within the same cluster as the workspaces they will provision or are hosted in.

  • This gives them a low-latency connection to the APIs they will use to provision workspaces and can speed builds.

  • It allows provisioner daemons to use in-cluster mechanisms (for example Kubernetes service account tokens, AWS Identity and Access Management (IAM) Roles, and others) to authenticate with the infrastructure APIs.

  • If you deploy workspaces in multiple clusters, run multiple provisioner daemon deployments and use template tags to select the correct set of provisioner daemons.

  • Provisioner daemons need to be able to connect to the control plane, but this does not need to be a low-latency connection.

Provisioner daemons make no direct connections to the PostgreSQL database, so there's no need for locality to the Postgres database.

Scaling

Each provisioner daemon instance can handle a single workspace build job at a time. Therefore, the maximum number of simultaneous builds your Coder deployment can handle is equal to the number of provisioner daemon instances within a tagged deployment.

If users experience unacceptably long queues for workspace builds to start, consider increasing the number of provisioner daemon instances in the affected cluster.

You might need to automatically scale the number of provisioner daemon instances throughout the day to meet demand.

If you stop instances with SIGHUP, they will complete their current build job and exit. SIGINT will cancel the current job, which will result in a failed build. Make sure your autoscaler waits long enough for your build jobs to complete before it kills the provisioner daemon process.

If you deploy in Kubernetes, we recommend a single provisioner daemon per pod. On a virtual machine (VM), you can deploy multiple provisioner daemons, making sure each has a unique CODER_CACHE_DIRECTORY value.

Coder's validated architectures give specific sizing recommendations for various user scales. Since the complexity of builds varies significantly depending on the workspace template, consider this a starting point. Monitor queue times and build times and adjust the number and size of your provisioner daemon instances.

PostgreSQL

PostgreSQL is the primary persistence layer for all of Coder's deployment data. We also use LISTEN and NOTIFY to coordinate between different instances of the control plane.

Locality

Control plane instances must have low-latency connections (under 10 ms) to PostgreSQL. If you use multiple PostgreSQL replicas in a clustered config, these must also be low-latency with respect to one another.

Scaling

Prefer scaling PostgreSQL vertically rather than horizontally for best performance. Coder's validated architectures give specific sizing recommendations for various user scales.

Connection pool tuning

The control plane maintains a pool of connections to PostgreSQL. You can tune the pool size with the following settings:

Note

When adjusting these settings, make sure that your PostgreSQL Server has max_connections set appropriately to accommodate all control plane replicas multiplied by the maximum number of open connections. We recommend configuring an additional 20% of connections to account for churn and other clients.

Also note that increasing max_connections will result in potentially higher CPU and RAM usage, so you'll need to monitor accordingly.

  • --postgres-conn-max-open (env: CODER_PG_CONN_MAX_OPEN): Maximum number of open connections. Default: 10.
  • --postgres-conn-max-idle (env: CODER_PG_CONN_MAX_IDLE): Maximum number of idle connections kept in the pool. Default: "auto", which uses max open / 3.

When a connection is returned to the pool and the idle pool is already full, the connection is closed immediately. This can cause connection establishment overhead (churn) when load fluctuates. Monitor these metrics to understand your connection pool behavior:

  • Capacity: go_sql_max_open_connections - go_sql_in_use_connections shows how many connections are available for new requests. If this is 0, control plane performance will start to degrade. This just provides a point-in-time view of the connections, however.

    For a more systematic view, consider running sum by (pod) (increase(go_sql_wait_duration_seconds_total[1m])) to see how long each Coder replica spent waiting on the connection pool (that is, no free connections). sum by (pod) (increase(go_sql_wait_count_total[$__interval])) shows how many connections were waited for.

    If either of these values seem unacceptably high, try tuning the above settings.

  • Churn: sum(rate(go_sql_max_idle_closed_total[$__rate_interval])) shows how many connections are being closed because the idle pool is full.

If you see high churn, consider increasing --postgres-conn-max-idle to keep more connections ready for reuse. If you see capacity consistently near zero, consider increasing --postgres-conn-max-open.

Workspace proxies

Workspace proxies proxy HTTP traffic from end users to workspaces for Coder apps defined in the templates, and HTTP ports opened by the workspace. By default they also include a DERP Proxy.

Locality

We recommend each geographic cluster of workspaces have an associated deployment of workspace proxies. This makes sure that users always have a near-optimal proxy path.

Scaling

Workspace proxy load is determined by the amount of traffic they proxy.

Monitor CPU, memory, and network I/O utilization to decide when to resize the number of proxy instances.

Scale for peak demand and scale down or upgrade during a maintenance window.

We do not recommend autoscaling the workspace proxies because many applications use long-lived connections such as websockets, which would be disrupted by stopping the proxy.

Workspaces

Workspaces represent the vast majority of resources in most Coder deployments. Because they are defined by templates, there is no one-size-fits-all advice for scaling workspaces.

Hard and soft cluster limits

All Infrastructure as a Service (IaaS) clusters have limits to what can be simultaneously provisioned. These could be hard limits, based on the physical size of the cluster, especially in the case of a private cloud, or soft limits, based on configured limits in your public cloud account.

It is important to be aware of these limits and monitor Coder workspace resource utilization against the limits, so that a new influx of users don't encounter failed builds. Monitoring these is outside the scope of Coder, but we recommend that you set up dashboards and alerts for each kind of limited resource.

As you approach soft limits, you can request limit increases to keep growing.

As you approach hard limits, consider deploying to additional cluster(s).

Workspaces per node

Many development workloads are "spiky" in their CPU and memory requirements, for example, they peak during build/test and then lower while editing code. This leads to an opportunity to efficiently use compute resources by packing multiple workspaces onto a single node. This can lead to better experience (more CPU and memory available during brief bursts) and lower cost.

There are a number of things you should consider before you decide how many workspaces you should allow per node:

  • "Noisy neighbor" issues: Users share the node's CPU and memory resources and might be susceptible to a user or process consuming shared resources.

  • If the shared nodes are a provisioned resource, for example, Kubernetes nodes running on VMs in a public cloud, then it can sometimes be a challenge to effectively autoscale down.

    • For example, if half the workspaces are stopped overnight, and there are ten workspaces per node, it's unlikely that all ten workspaces on the node are among the stopped ones.

    • You can mitigate this by lowering the number of workspaces per node, or using autostop policies to stop more workspaces during off-peak hours.

  • If you do overprovision workspaces onto nodes, keep them in a separate node pool and schedule the control plane, PostgreSQL, and workspace proxy components on a different node pool to avoid resource spikes affecting them.

Coder customers have had success with both:

  • One workspace per AWS VM
  • Lots of workspaces on Kubernetes nodes for efficiency

Cost control

  • Use quotas to discourage users from creating many workspaces they don't need simultaneously.

  • Label workspace cloud resources by user, team, organization, or your own labeling conventions to track usage at different granularities.

  • Use autostop requirements to bring off-peak utilization down.

Networking

Set up your network so that most users can get direct, peer-to-peer connections to their workspaces. This drastically reduces the load on the control plane and workspace proxy instances.

Next steps