Files
coder/docs/admin/security/security-best-practices.md
T
Nick VigilanteandBen Potter 1baff57cb0 fix: resolve docs and code conflicts found in the main docs audit (#32158)
## Summary

Addresses 106 of the 113 conflicts between the docs and the code that
the November audit of `main` found: 102 are fixed and 4 have an interim
docs fix. Every finding was re-checked against the code before it was
fixed. Most fixes are docs-only. Three are small code fixes where the
docs described the intended behavior and the code was wrong, each in its
own commits so it can be split out. No change in this PR alters template
build or agent behavior.

## Changes

Fixes by area (finding counts): templates 28, install 13, integrations
and monitoring 13, auth and security 12, networking 11, Coder Agents 9,
AI governance 7, reference 7, workspaces 6.

Code fixes. Each needs a look from the code owner. If an owner prefers
not to take one, drop its commits and I will change the docs to describe
the current behavior instead.

| Commits | Change | Risk to check |
|---------|--------|---------------|
| `fix: keep built-in listener defaults in packaged coder.env`, and
three follow-up commits | `coder.env` comments out `CODER_HTTP_ADDRESS=`
and the three `CODER_TLS_*` lines, so an untouched package install
listens on `127.0.0.1:3000` as the docs say. Only the
`CODER_HTTP_ADDRESS=` line changed behavior. The TLS options default to
empty, so commenting them out is for consistency. The Lima examples
rewrite the commented line and fail if the line is not found. A test
compares every option that the file sets against the built-in defaults.
| The deb and RPM packages were not built or installed. Every released
package through v2.39.0 ships the empty line, and upgrades keep an
existing file (`config\|noreplace`), so the upgrade pages now say how to
fix it. |
| `fix(cli/clilog): accept log sinks that are all disabled`, and three
follow-up commits | `--log-human=/dev/null` with no other log location
no longer fails with "no loggers provided". It is the documented way to
disable logging. An empty location still fails with that error when no
other sink is set, as before. | Behavior change: with `/dev/null`, the
server and the provisioner daemon can start with no log output. |
| `fix(cli): detect CODER_PROC_PRIO_MGMT by presence`, and a comment
follow-up | Log line only. With the variable set to an empty value, the
agent now logs that process priority management is enabled, which is
what `agentexec` already does. No detection or behavior change. | Low. |

Two option descriptions in `codersdk/deployment.go` are corrected, with
the CLI reference, configuration reference, and golden files
regenerated: the default quiet hours schedule (the max deadline is the
quiet hours window on the next day that the template's autostop
requirement applies) and `--experiments` (`*` opts in to the experiments
marked safe, also in `enterprise/cli/provisionerdaemonstart.go`).

### Code fixes that were reverted after review

Two code fixes from the first version of this PR are reverted in the
branch. The net diff has no change under `agent/` or `coderd/`. The docs
now describe what the code does, and both issues stay open for the
owner.

- **`error` merge strategy for `coder_env` (DOCS-1669).** The fix made
`merge_strategy = "error"` fail whenever a name was set twice, whatever
the resource order. Existing template versions that pair an `error`
entry with a later `replace` or `append` entry would start to fail every
build after `terraform apply`. That is a breaking change that needs its
own PR and release note. The fix and its tests are in `dff8e7c6ce` and
can be cherry-picked. `environment-variables.md` now says the check runs
only when the `error` resource is processed.
- **Nice value before the capability drop (DOCS-1687).** The fix let
`coder agent-exec` apply a negative nice value by calling `setpriority`
before it drops effective capabilities. The current order is deliberate:
with the fix, any local process could run the agent binary with
`--coder-nice=-20` when the binary has the `CAP_SYS_NICE` file
capability. `process-priority.md` now says a nice value below the
agent's own is not applied, and no longer tells admins to grant
`CAP_SYS_NICE`.

## Not included

Seven findings are left out.

- DOCS-1629: covered by #31639.
- DOCS-1703 and DOCS-1724: covered by #31980.
- DOCS-1584, DOCS-1683, DOCS-1685, DOCS-1686: each needs an owner
decision, and no neutral docs wording exists until that decision is
made. The validation comment on each issue names the decision.

Four fixes are interim, so their issues stay open:

- DOCS-1600: the AI Gateway pages now agree that the legacy metric names
are deprecated and "will be removed in a future release". The code still
emits them, and its TODO names v2.37, which has shipped. The owner still
needs to set the removal release.
- DOCS-1689: the page now says **New Template** opens the builder only
for users with the Owner role, because the page gates on read access to
the deployment config. Whether Template Admins should get the builder is
still open.
- DOCS-1669 and DOCS-1687: docs-only for now. Refer to [Code fixes that
were reverted after
review](#code-fixes-that-were-reverted-after-review).

## Review notes

- Claims that describe product intent and need an owner's yes: the
Premium qualifier on per-organization workspace sharing (the API route
has no license check, the dashboard needs multiple organizations),
`organization` port sharing in OSS, the telemetry field list, and the
`1-5` cron range in the prebuilds schedule example.
- Not run end to end: the PostgreSQL TLS procedure on a cluster, the
OpenShift and air-gapped examples, the BSD `date` and `tar` forms on
macOS, the Windows command on the multiple Kubernetes clusters page, and
the keep-alive script in `docs/reference/index.md` (only its comparison
logic was run, against sample values). Nothing was checked on a live
deployment.
- Likely merge conflicts with open PRs: #31990 (`mcp-server.md`,
`feature-stages.md`, `docker-in-workspaces.md`), #31982
(`workspace-management.md`), #31980 (`health-check.md`), #31977
(`manage-provisioner-jobs.md`), #31869 (`airgap.md`), #30486
(`add-envbuilder.md`), and #31326 and #31531 (`codersdk/deployment.go`
and generated files).

## Testing

- `make pre-commit` (gen, fmt, lint, slim build), before the PR opened
and again after the review changes.
- `go test` for `./cli/clilog/...`, `./codersdk/`, `./cli/`
(`TestServer`, `TestAgent`, and golden files), and `./enterprise/cli/`
(provisioner, server, and help golden files). `./agent/agentexec/...`
and `./coderd/provisionerdserver/...` pass after the reverts. Linux
only. CI covers macOS and Windows.
- The `coder.env` test fails against the file on `main` and passes
against the new one. It reads the file with `godotenv`, which accepts
CRLF line endings.
- An empty `CODER_HTTP_ADDRESS=` was confirmed end to end with a full
build of the server (output below).
- The new `sed` and `grep` lines in the Lima examples were run with GNU
tools against the old and the new form of `coder.env`. The Lima VMs were
not started.
- `--log-human=/dev/null` and `CODER_LOGGING_HUMAN=` were confirmed with
the same build (output below).
- `coder agent-exec --coder-nice=-5 -- echo` from a full build, as a
non-root user: `coder-agent: failed to adjust niceness to -5 ...:
permission denied` on stderr, the command output on stdout, exit code 0.
- `postgres-ssl.md` rendered with the GitHub Markdown API: two ordered
lists with three and five items, where `main` has six lists.
- markdownlint, the table formatter, `make lint/emdash`, `make
lint/docs-html`, `make lint/docs-manifest`, `make lint/docs-redirects`,
`make lint/style-claims`, and Vale on the changed lines.
- A doc-check self-review of the docs diff by four independent
reviewers. It found six problems, which are fixed. The accuracy and
coverage lenses ran on every area. The language and structure lenses ran
in full on only part of the diff.

<details>
<summary>Test and terminal output for the code fixes (Linux)</summary>

```console
$ go test ./codersdk/ -run TestPackagedCoderEnvKeepsDefaults -count=1 -v
--- PASS: TestPackagedCoderEnvKeepsDefaults (0.00s)
ok  	github.com/coder/coder/v2/codersdk	0.071s

$ go test ./cli/clilog/ -run TestBuilder -count=1 -v
--- PASS: TestBuilder (0.00s)
    --- PASS: TestBuilder/NoSinks (0.00s)
        --- PASS: TestBuilder/NoSinks/JSONDevNull (0.00s)
        --- PASS: TestBuilder/NoSinks/StackdriverDevNull (0.00s)
        --- PASS: TestBuilder/NoSinks/NoConfiguration (0.00s)
        --- PASS: TestBuilder/NoSinks/HumanDevNull (0.00s)
    --- PASS: TestBuilder/WithHuman (0.00s)
    --- PASS: TestBuilder/WithFilter (0.00s)
    --- PASS: TestBuilder/WithJSON (0.00s)
    --- PASS: TestBuilder/Verbose (0.00s)
    --- PASS: TestBuilder/FromDeploymentValues (0.01s)
        --- PASS: TestBuilder/FromDeploymentValues/Defaults (0.00s)
        --- PASS: TestBuilder/FromDeploymentValues/EmptyHumanEnv (0.00s)
        --- PASS: TestBuilder/FromDeploymentValues/Override (0.00s)
ok  	github.com/coder/coder/v2/cli/clilog	0.171s

$ TMPDIR=$HOME/.cache/agentexec-tmp go test ./agent/agentexec/... ./coderd/provisionerdserver/... -count=1
ok  	github.com/coder/coder/v2/agent/agentexec	4.236s
ok  	github.com/coder/coder/v2/coderd/provisionerdserver	2.404s

# Full build of the server (go build ./cmd/coder). An empty value stops it before it connects to the database.
$ CODER_HTTP_ADDRESS= coder server --postgres-url 'postgres://127.0.0.1:1/x?sslmode=disable'
Encountered an error running "coder server", see "coder server --help" for more information
error: TLS is disabled. Enable with --tls-enable or specify a HTTP address

# With the variable unset, the server passes that check and goes on to the database.
$ coder server --postgres-url 'postgres://127.0.0.1:1/x?sslmode=disable'
2026-11-13 02:19:41.901 [warn]  ping postgres: retrying  error="dial tcp 127.0.0.1:1: connect: connection refused"  try=11
Encountered an error running "coder server", see "coder server --help" for more information
error: connect to postgres: connect to postgres: unable to connect after 11 tries; last error: dial tcp 127.0.0.1:1: connect: connection refused

# An empty log location with no other sink still stops the server.
$ CODER_LOGGING_HUMAN= coder server --postgres-url 'postgres://127.0.0.1:1/x?sslmode=disable'
Encountered an error running "coder server", see "coder server --help" for more information
error: make logger: no loggers provided, use /dev/null to disable logging

# /dev/null passes the logger check. The server prints no log lines and goes on to the database.
$ coder server --log-human=/dev/null --postgres-url 'postgres://127.0.0.1:1/x?sslmode=disable'
Encountered an error running "coder server", see "coder server --help" for more information
error: connect to postgres: connect to postgres: unable to connect after 11 tries; last error: dial tcp 127.0.0.1:1: connect: connection refused
```

</details>

## Linear

Fixes DOCS-1583, DOCS-1585, DOCS-1586, DOCS-1587, DOCS-1588, DOCS-1589,
DOCS-1590, DOCS-1591, DOCS-1592, DOCS-1593, DOCS-1594, DOCS-1595,
DOCS-1596, DOCS-1597, DOCS-1598, DOCS-1599, DOCS-1601, DOCS-1602,
DOCS-1603, DOCS-1604, DOCS-1605, DOCS-1606, DOCS-1607, DOCS-1608,
DOCS-1609, DOCS-1614, DOCS-1618, DOCS-1619, DOCS-1620, DOCS-1624,
DOCS-1635, DOCS-1639, DOCS-1644, DOCS-1651, DOCS-1652, DOCS-1653,
DOCS-1654, DOCS-1655, DOCS-1656, DOCS-1657, DOCS-1658, DOCS-1659,
DOCS-1660, DOCS-1661, DOCS-1662, DOCS-1663, DOCS-1664, DOCS-1665,
DOCS-1666, DOCS-1667, DOCS-1668, DOCS-1670, DOCS-1671, DOCS-1672,
DOCS-1673, DOCS-1674, DOCS-1675, DOCS-1676, DOCS-1677, DOCS-1678,
DOCS-1679, DOCS-1680, DOCS-1681, DOCS-1682, DOCS-1684, DOCS-1688,
DOCS-1690, DOCS-1691, DOCS-1692, DOCS-1693, DOCS-1694, DOCS-1695,
DOCS-1696, DOCS-1697, DOCS-1698, DOCS-1699, DOCS-1700, DOCS-1701,
DOCS-1702, DOCS-1704, DOCS-1705, DOCS-1706, DOCS-1707, DOCS-1708,
DOCS-1709, DOCS-1710, DOCS-1711, DOCS-1712, DOCS-1713, DOCS-1714,
DOCS-1715, DOCS-1716, DOCS-1717, DOCS-1718, DOCS-1719, DOCS-1720,
DOCS-1721, DOCS-1722, DOCS-1723, DOCS-1725, DOCS-1726, DOCS-1727

Part of DOCS-1600, DOCS-1689, DOCS-1669, DOCS-1687

<details>
<summary>Decision log</summary>

### Scope

- One PR for all findings, as agreed with the reviewer. The code fixes
are separate commits so an owner can ask for any of them to move to its
own PR. Nothing in the PR is a breaking change, so the title has no `!`.
- A finding was skipped only when an open PR already covers it, or when
every available wording would assert an undecided product behavior.
- Each validation comment on the Linear issues records the evidence, the
corrections to the original finding, and what was not verified.

### Docs or code

- The default was a docs-only fix that describes what the code does.
- A code fix was chosen when the docs describe a behavior the code
plainly intended and the code change is small and testable: the packaged
env file, the log sink check, and the log line for the environment
variable presence check.
- The `error` merge strategy and the nice value ordering started as code
fixes and went back to docs-only after review. The first breaks existing
template versions. The second weakens a deliberate security ordering.
- The SCIM `userName` finding (DOCS-1609) took the docs path. The
handler is unchanged.

### Wording choices worth a second look

- `feature-stages.md` says `--experiments=*` enables only the
experiments marked safe, and points to the "Get safe experiments" API.
It does not list the other experiments, because the list would drift and
those experiments are hidden on purpose. The `--experiments` flag help
now says the same.
- `agent-metadata.md` replaces the writes-per-second formula with a
description of the batcher (each server replica writes one statement
every 5 seconds, or sooner at 500 entries).
- `extending-templates/index.md` no longer calls `startup_script`
deprecated. The provider documents it as an alias for a `coder_script`.
- `postgres-ssl.md` separates client verification from server
verification, so the server verification steps work alone.
- `rpm.md`, `upgrade.md`, and the 2.29 to 2.34 ESR upgrade page explain
the empty `CODER_HTTP_ADDRESS=` line that older packages shipped,
because upgrades keep the old file. The behavior changed in v2.34.0 with
the serpent upgrade in #24926.
- The keep-alive script in `docs/reference/index.md` reads the current
deadline and skips the request when it is already later. The comparison
is on text, so it applies only when the API returns a UTC timestamp,
which needs the database to use the UTC time zone. Otherwise the script
sets the deadline on every pass, as it did before, and the page says so.
- `external-auth.md` anchors the new GitHub Enterprise regex example.
The older unanchored examples on that page and on `git-providers.md` are
not changed here.

### Self-review findings that were rejected

- "The builder opens for Template Admins": the Templates page needs a
successful deployment config read, which the Template Admin role does
not have.
- "Organization Admins can set AI model prices": the price write
authorizes against a site-level object, which organization-scoped grants
do not match.
- "The provisioner chart sets no default resources":
`helm/libcoder/templates/_coder.yaml` applies 2000m and 4096Mi when
`coder.resources` is empty.

### Automated review, round 1

The `coder-agents-review` app left 45 findings on `8853332`. Fixed: the
two reverts above, the anchored regex, the `coder.env` follow-ups (Lima
examples, upgrade notes, a comment in the file, and a test that covers
every option in the file), the clilog change, the two option
descriptions, and the docs findings (CORS behavior, token lifetime
example, keep-alive script, agent metadata, refresh token errors,
`switch` form type, `dbcrypt` skip list, and a stray line in a YAML
example).

Declined, with the reason on each thread:

- Refactoring the `agentexec` logging and renaming its log field: not
related to a docs conflict.
- Rewording the "process priority management not enabled (linux-only)"
log message: the docs quote the message, and the wording predates this
PR.
- Problems in code and docs that this PR does not change (the
`Capabilities` subtest, the Helm values comment, the AWS provider path
in `airgap.md`, and others in the review's out-of-scope list).

### Automated review, round 2

The app left 14 new findings on `04925bd`, and accepted the three
declined threads on `cli/agent.go`. All 14 are fixed. The main one: the
round 1 change that removed the "no loggers provided" error went too
far, because an empty log location would then start a server with no
logs. The error is back for every case except `/dev/null`.

Reported by the app as out of scope and not changed: the swagger
annotation for `GET /api/v2/experiments/available` declares an array,
but the handler returns an object with a `safe` array. Three Storybook
fixtures keep the old `--experiments` description.

### Automated review, round 3

The app confirmed that all round 2 findings are fixed, and left 10 small
findings on `f55cb6d` (8 P4, 2 Nit). All are fixed: the TLS condition
and the release range in the `coder.env` notes, one place for the
`/dev/null` rule in `clilog`, two test cases, a clearer test failure,
and the CPU example in `agent-metadata.md`.

### Automated review, rounds 4 and 5

Round 4 left one P4 on `8fffb38` (name the `--trace` option in the
logging note), which is fixed. Round 5 on `2ce754e` reports no new
findings and says every earlier finding is fixed or closed. The app did
not post an approval: it leaves its out-of-scope items and the code
fixes to a human reviewer. Two threads on `cli/agent.go` stay open on
purpose for the code owner.

### Doc-check review

The doc-check workflow left 12 comments on `2ce754e` (10 blocking, 2
suggestions). All are fixed in `c888885`: "refer to" or "visit" in place
of "see" for cross-references on the lines this PR added, the **New
template** label as the dashboard shows it, one instruction per step in
the server verification procedure of `postgres-ssl.md`, and two wording
changes in `process-priority.md`. The `CAP_SYS_NICE` sentence keeps a
short reason, which differs from the suggested text.

The `coder-agents-review` app then reviewed that commit (rounds 6 and
7). Round 6 left two P4 findings, fixed in `30b2c79`: the `failed to
adjust niceness` message is described once, in the troubleshooting
entry, with where it appears, and the client verification steps in
`postgres-ssl.md` render as one numbered list. Round 7 reports no new
findings. `2a6035c` then softens one sentence from that fix, because the
agent logs the stderr of devcontainer CLI commands. The app was not run
again on that commit.

### Adjacent gaps found and not fixed here

- `add-envbuilder.md` has the same `zip` command that
`template-from-scratch.md` had.
- `template-from-scratch.md` leaves the reader in `build/` after `cd
$_`, before a step that runs from `template-tour`.

</details>

---

_Generated by Coder Agents on behalf of @nickvigilante._

---------

Co-authored-by: Ben Potter <me@bpmct.net>
2026-11-13 21:34:59 +00:00

22 KiB
Raw Blame History

title
title
Security - best practices

December 16, 2024


This best practices guide is separated into parts to help you secure aspects of your Coder deployment.

Each section briefly introduces each threat model, then suggests steps or concepts to help implement security improvements such as authentication and encryption.

As with any security guide, the steps and suggestions outlined in this document are not meant to be exhaustive and do not offer any guarantee.

Control plane

The control plane is the core of a Coder deployment.

If the control plane is compromised in a security incident, it can affect every other part of your deployment. Even a successful read-only attack against the control plane could result in a complete compromise of the Coder deployment if credentials are stolen.

User authentication

Configure OpenID Connect (OIDC) authentication against your organization’s Identity Provider (IdP), such as Okta, to allow single-sign on.

  1. Enable and require two-factor authentication in your identity provider.
  2. Enable IdP Sync to manage users’ roles and groups in Coder.
  3. Use System for Cross-domain Identity Management (SCIM) to automatically suspend users when they leave the organization.

This allows you to manage user credentials according to your company’s central requirements, such as password complexity, 2FA, PassKeys, and others.

Using IdP sync and SCIM means that the central Identity Provider is the source of truth, so that when users change roles or leave, their permissions in Coder are automatically up to date.

Encryption in transit

Place Coder behind a TLS-capable reverse-proxy/load balancer and enable Strict Transport Security so that connections from end users are always encrypted.

Enable TLS on the control plane and encrypt traffic from the reverse-proxy/load balancer to the control plane, so that even if an attacker gains access to your network, they will not be able to snoop on control plane traffic.

Encryption at rest

The control plane persists no state locally. No action is required.

Control plane logs and audit logs

Capture the logging output of all control plane instances and persist them.

Retain all logs for a minimum of thirty days, ideally ninety days. Filter audit logs (which have msg: audit_log) and retain them for a minimum of two years (ideally five years) in a secure system that resists tampering.

If a security incident with Coder does occur, audit logs are invaluable in determining the nature and scope of the impact.

Disable path-based apps

For production deployments, we recommend that you disable path-based apps after you've configured a wildcard access URL.

Path-based apps share the same origin as the Coder API, which can be convenient for trialing Coder, but can expose the deployment to cross-site-scripting (XSS) attacks in production. A malicious workspace could reuse Coder cookies to call the API or interact with other workspaces owned by the same user.

  1. Enable sub-domain apps with a wildcard DNS record (like *.coder.example.com)

  2. Disable path-based apps:

    coder server --disable-path-apps
    # or
    export CODER_DISABLE_PATH_APPS=true
    

By default, Coder mitigates the impact of having path-based apps enabled, but we still recommend disabling it to prevent malicious workspaces accessing other workspaces owned by the same user or performing requests against the Coder API.

If you do keep path-based apps enabled:

  • Path-based apps cannot be shared with other users unless you start coder server with --dangerous-allow-path-app-sharing.
  • Users with the site owner role cannot use their admin privileges to access path-based apps for workspaces unless coder server is started with --dangerous-allow-path-app-site-owner-access.

PostgreSQL

PostgreSQL is the persistent datastore underlying the entire Coder deployment. If the database is compromised, it might leave every other part of your deployment vulnerable.

Coder session tokens and API keys are stored only as SHA-256 hashes of random secrets, so a read-only compromise of the database is unlikely to allow an attacker to log into Coder. However, the database contains the Terraform state for all workspaces, OIDC tokens, and agent tokens, so it is possible that a read-only attack could enable lateral movement to other systems.

A successful attack that modifies database state could be escalated to a full takeover of an owner account in Coder which could lead to a complete compromise of the Coder deployment.

Authentication

  1. Generate a strong, random password for accessing PostgreSQL and store it securely.

  2. Use environment variables to pass the PostgreSQL URL to Coder.

  3. If on Kubernetes, use a Kubernetes secret to set the environment variable.

Encryption in transit

Enable TLS on PostgreSQL and set sslmode=verify-full in your postgres URL on the control plane. This configures the control plane to only establish TLS connections to PostgreSQL and check that the PostgreSQL server’s certificate is valid and matches the expected hostname.

Encryption at rest

Run PostgreSQL on servers with full disk encryption enabled and configured.

Coder supports encrypting some particularly sensitive data including OIDC tokens using an encryption key managed independently of the database, so even a user with full administrative privileges on the PostgreSQL server(s) cannot read the data without the separate key.

If you use this feature:

  1. Generate a random encryption key and store it in a central secrets management system like Vault.

  2. Inject the secret using an environment variable.

    • If you're using Kubernetes, use a Kubernetes secret rather than including the secret directly in the podspec.
  3. Follow your organization's policies about key rotation on a fixed schedule.

Provisioner daemons

Provisioner daemons are deployed with credentials that give them power to make requests to cluster/cloud APIs.

If one of those credentials is compromised, the potential severity of the compromise depends on the permissions granted to the credentials, but will almost certainly include code execution inside the cluster/cloud since the whole purpose of Coder is to deploy workspaces in the cluster/cloud that can run developer code.

In addition, provisioner daemons are given access to parameters entered by end users, which could include sensitive data like credentials for additional systems.

External provisioner daemons

When Coder workspaces are deployed into multiple clusters/clouds, or workspaces are in a different cluster/cloud than the control plane, use external provisioner daemons.

Running provisioner daemons within the same cluster/cloud as the workspaces they provision:

  • Allows you to use infrastructure-provided credentials (see Authentication below) which are typically easier to manage and have shorter lifetimes than credentials issued outside the cloud/cluster.
  • Means that you don’t have to open any ingress ports on the clusters/clouds that host workspaces.
    • The external provisioner daemons dial out to the control plane.
    • Provisioner daemons run in the cluster, so you don’t need to expose cluster/cloud APIs externally.
  • Each cloud/cluster is isolated, so a compromise of a provisioner daemon is limited to a single cluster.

Authentication

  1. Use a scoped key to authenticate the provisioner daemons with Coder.

    These keys can only be used to authenticate provisioner daemons (not other APIs on the control plane).

  2. Store the keys securely and use environment variables to pass them to the provisioner daemon.

  3. If on Kubernetes, use a Kubernetes secret to set the environment variable.

  4. Tag provisioners with identifiers for the specific cluster/cloud.

    This allows your templates to target a specific cluster/cloud such as for geographic proximity to the end user, or for specific features like graphics processing units (GPUs) or managed services.

  5. Scope your keys to organizations and the specific cluster/cloud using the same tags when creating the keys.

    This makes sure that a compromised key will not allow an attacker to gain access to jobs for other clusters or organizations.

Provisioner daemons should have access only to cluster/cloud API credentials for the specific cluster/cloud they are for. This guarantees that compromise of one Provisioner Daemon does not compromise all clusters/clouds.

Deploy the provisioner daemon to the cloud and leverage infrastructure-provided credentials, if available:

Encryption in transit

Enable TLS on the control plane and make sure you use an https:// URL to access the control plane.

See the Encryption in transit subheading of the Templates section for more about encrypting cluster/cloud API calls.

Encryption at rest

Run provisioner daemons only on systems with full disk encryption enabled.

  • Provisioner daemons temporarily persist terraform template files and resource state to disk. Either of these could contain sensitive information, including credentials.

    This temporary state is on disk only while actively building workspaces, but an attacker who compromises physical disks could successfully read this information if not encrypted.

  • Provisioner daemons store cached copies of Terraform provider binaries. These are generally not sensitive in terms of confidentiality, but it is important to maintain their integrity. An attacker that can modify these binaries could inject malicious code.

Workspace proxies

Workspace proxies authenticate end users and then proxy network traffic to workspaces.

Coder takes care to make sure the user credentials processed by workspace proxies are scoped to application access and do not grant full access to the Coder API on behalf of the user. Still, a fully compromised workspace proxy would be in a privileged position to phish unrestricted user credentials.

Workspace proxies have unrestricted access to establish encrypted tunnels to workspaces and can access any port on any running workspace.

Authentication

  1. Securely store the workspace proxy token generated by coder wsproxy create.

  2. Inject the token to the workspace proxy process via an environment variable, rather than via an argument.

  3. If on Kubernetes, use a Kubernetes secret to set the environment variable.

Encryption in transit

Enable TLS on the control plane and make sure you use an https:// URL to access the control plane.

Communication to the proxied workspace applications is always encrypted with Wireguard. No action is required.

Encryption at rest

Workspace proxies persist no state locally. No action is required.

Workspace templates

Coder templates are executed on provisioner daemons and can include arbitrary code via the local-exec provisioner.

Furthermore, Coder templates are designed to provision compute resources in one or more clusters/clouds, and template authors are generally in full control over code and scripts executed by the workspace agent in those compute resources.

This means that template admins have remote code execution privileges for any provisioner daemons in their organization and within any cluster/cloud those provisioner daemons are credentialed to access.

Template admin is a powerful, highly-trusted role that you should not assign lightly. Instead of directly assigning the role to anyone who might need to edit a template, use GitOps to allow users to author and edit templates.

Secrets

Never include credentials or any other secrets directly in templates, including in .tfvars or other files uploaded with the template.

Instead do one of the following:

  • Store secrets in a central secrets manager.

  • Place secrets in TF_VAR_* environment variables.

    • Provide the secrets to the relevant Provisioner Daemons and access them via Terraform variables with sensitive = true.
  • Use Coder parameters to accept secrets from end users at build time.

Coder does not attempt to obscure the contents of template files from users authorized to view and edit templates, so secrets included directly could inadvertently appear on screen while template authors do their work.

Template versions are persisted indefinitely in the PostgreSQL database, so if secrets are inadvertently included, they should be revoked as soon as practical. Pushing a new template version does not expunge them from the database. Contact support if you need assistance expunging any particularly sensitive data.

Encryption in transit

Always use encrypted transport to access any infrastructure APIs. Crucially, this protects confidentiality of the credentials used to access the APIs.

Configuration of this depends on the specific Terraform providers in use and is beyond the scope of this document.

Encryption at rest

While your most privileged secrets should never be included in template files, they might inevitably contain confidential or sensitive data about your operations and/or infrastructure.

  • Make sure that operators who write, review or modify Coder templates are working on laptops/workstations with full disk encryption, or do their work inside a Coder workspace with full disk encryption.
  • Make sure PostgreSQL is encrypted at rest.
  • Make sure any source code repositories that store templates are encrypted at rest and have appropriate access controls.

GitOps

GitOps is the practice of using a Git repository as the source of truth for operational config and reconciling the config in Git with operational systems each time the main (or, archaically, master) branch of the repository is updated.

  1. Store Coder templates in a single Git repository, or a single repository per Coder organization.

    This gives you an easily browsable, auditable history of template changes and who made them. Coder audit logs establish who and when changes happen, but git repositories are particularly handy for analyzing exactly what changes to templates are made.

  2. Use the Coderd Terraform provider to push changes to Coder from a continuous integration and continuous delivery (CI/CD) tool.

  3. Use a Coder user account exclusively for pushing template changes, and give no human users the credentials.

    This guarantees any actions taken by the account correspond exactly to CI/CD actions from the repository and allows you to avoid granting the template admin role widely in your organization.

  4. Use GitHub branch protection, or the equivalent for your source repository to enforce code review of changes to templates.

    Code review increases the chance that someone will catch a potential security bug in your template.

These protections also mitigate the risk of a single trusted insider “going rogue” and acting unilaterally to maliciously modify Coder templates.

Workspaces

The central purpose of Coder is to give end users access to managed compute in clusters/clouds designated by Coder’s operators (like platform or developer experience teams). End users are granted shell access and from there can execute arbitrary commands.

This means that end users have remote code execution privileges within the clusters/clouds that host Coder workspaces.

It is important to limit Coder users to trusted insiders and/or take steps to constrain malicious activity that could be undertaken from a Coder workspace.

Example constraints include:

  • Network policy or segmentation
  • Runtime protections on the workspace host (for example, SELinux)
  • Limiting privileges of the account or role assigned to the workspace such as a service account on Kubernetes, or IAM role on public clouds
  • Monitoring and/or auditing for suspicious activity such as cryptomining or exfiltration

Outbound network access

Identify network assets like production systems or highly confidential datastores and configure the network to limit access from Coder workspaces.

If production systems or confidential data reside in the same cluster/cloud, use separate node pools and network boundaries.

If extraordinary access is required, follow Zero Trust principles:

  • Authenticate the user and the workspace using strong cryptography
  • Apply strict authorization controls
  • Audit access in a tamper resistant secure store

Consider the network assets end users will need to do their job and the level of trust the company has with them. In-house full-time employees have different access than temporary contractors or third-party service providers. Restrict access as appropriate.

A non-exclusive list of network assets to consider:

  • Access to the public Internet
    • If end users will access the workspace over the public Internet, you must allow outbound access to establish the encrypted tunnels.
  • Access to internal corporate networks
    • If end users will access the workspace over the corporate network, you must allow outbound access to establish the encrypted tunnels.
  • Access to staging or production systems
  • Access to confidential data (for example, payment processing data, health records, personally identifiable information)
  • Access to other clusters/clouds

Inbound network access

Coder manages inbound network access to your workspaces via a set of Wireguard encrypted tunnels. These tunnels are established by sending outbound packets, so on stateful firewalls, disable inbound connections to workspaces to make sure inbound connections are handled exclusively by the encrypted tunnels.

DERP

Designated Encrypted Relay for Packets (DERP) is a relay protocol developed by Tailscale.

The control plane and Workspace Proxies include a DERP service by default. Tailcale also runs a set of public DERP servers, globally distributed.

All DERP messages are end-to-end encrypted, so the DERP service only learns the (public) IP addresses of the participants.

If you consider these addresses or the fact that pairs of them communicate over DERP to be sensitive, stick to the Coder-provided DERP services which run on your own infrastructure. If not, feel free to configure Tailscale DERP servers for global coverage.

STUN

Session Traversal Utilities for Network Address Translation (STUN) is an Internet Engineering Task Force (IETF) standard protocol that allows network endpoints behind Network Address Translation (NAT) to learn their public address and port mappings. It is an essential component of Coder’s networking to enable encrypted tunnels to be established without a relay for best performance.

Coder does not ship with a STUN service because it needs to be run directly connected to the network, not behind a reverse proxy or load balancer as Coder usually is.

STUN messages are not encrypted, but do not transmit any tunneled data, they simply query the public address and ports. As such, a STUN service learns the public address and port information such as the address and port on the NAT device of Coder workspaces and the end user's device if STUN is configured.

Unlike DERP, it doesn’t definitively learn about communicating pairs of IPs.

If you consider the public IP and port information to be sensitive, do not use public STUN servers.

You can choose not to configure any STUN servers, in which case most workspace traffic will need to be relayed via DERP. You can choose to deploy your own STUN servers, either on the public Internet, or on your corporate network and configure Coder to use it.

If you do not consider the addresses and ports to be sensitive, we recommend using the default set of STUN servers operated by Google.

Workspace apps

Coder workspace apps are a way to allow users to access web applications running in the workspace via the control plane or Workspace Proxy.

  1. Disable workspace apps on sub-paths of the main Coder domain name.

  2. Use a separate, wildcard domain name for forwarding.

    Because of the default same-origin policy in browsers, serving web apps on the main Coder domain would allow those apps to send API requests to the control plane, authenticated as the logged-in user without their explicit consent.

Port sharing

Coder supports the option to allow users to designate specific network ports on their workspace as shared, which allows others to access those ports via the control plane.

Consider restricting the maximum sharing level for workspaces, located in the template settings for the corresponding template.

Encryption at rest

Deploy Coder workspaces using full disk encryption for all volumes.

This mitigates attempts to recover sensitive data in the workspace by attackers who gain physical access to the disk(s).