Files
coder/docs/admin/networking/high-availability.md
T
Andrew G. Dunn c11d60ad8d docs: note the pubsub cluster port between HA replicas (#30290)
HA replicas connect to each other's embedded NATS pubsub on TCP 6222.
The port is hardcoded
([`coderd/x/nats/pubsub.go`](https://github.com/coder/coder/blob/7f3d35353eddcd8a9e573094e5e2531bd2d73659/coderd/x/nats/pubsub.go#L62))
with no flag or environment variable, and since #29283 NATS pubsub is on
by default. The HA page explains `CODER_CLUSTER_HOST` but doesn't name
the port, so a firewall or default-deny NetworkPolicy between `coderd`
nodes blocks it. This adds the port, and what happens without it, next
to `CODER_CLUSTER_HOST`.

Measured on v2.37.3 (`CODER_EXPERIMENTS=nats_pubsub`, the default on
main) with two replicas on GKE under a default-deny NetworkPolicy, port
8080 open between the replicas in both runs. Through a port-forward to
replica B, `PUT /api/v2/workspaces/{id}/extend` publishes a workspace
update in the request path. A watcher on `/api/v2/workspaces/{id}/watch`
through replica A counts the events. The control sends the same request
through A.

6222 blocked:

```text
label=blocked A=coder-699ccb59f8-695bj B=coder-699ccb59f8-w8glp baseline_data_events=1
control: extend via A -> HTTP 409; data events on A: 1 (expect 1)
test: 5x extend via B -> HTTP 409 409 409 409 409; data events on A: 0
coder-699ccb59f8-695bj :6222 established: 0
coder-699ccb59f8-w8glp :6222 established: 0
```

6222 allowed between the replicas (same pods, no restart):

```text
label=allowed A=coder-699ccb59f8-695bj B=coder-699ccb59f8-w8glp baseline_data_events=1
control: extend via A -> HTTP 409; data events on A: 1 (expect 1)
test: 5x extend via B -> HTTP 409 409 409 409 409; data events on A: 5
coder-699ccb59f8-695bj :6222 established: 4
coder-699ccb59f8-w8glp :6222 established: 4
```

The 409s are expected because the workspace is stopped; the handler
publishes the update before it returns
([`coderd/workspaces.go`](https://github.com/coder/coder/blob/3a248167f6e58462a8a8cddba3b49a8ff72a9f80/coderd/workspaces.go#L1711)
at v2.37.3).

Would it also make sense for the chart to declare 6222 as a container
port, or for the port to be configurable? I left both out of this
change.

AI involvement: I used an AI assistant to draft this change, and
verified it with the runs above against a two-replica deployment.
2026-10-07 19:21:29 -04:00

4.6 KiB

title
title
High availability

High Availability (HA) mode solves for horizontal scalability and automatic failover within a single region. When in HA mode, Coder continues using a single Postgres endpoint. GCP, AWS, and other cloud vendors offer fully-managed HA Postgres services that pair nicely with Coder.

For Coder to operate correctly, coderd instances should have low-latency connections to each other so that they can effectively relay traffic between users and workspaces no matter which coderd instance users or workspaces connect to. We make a best-effort attempt to warn the user when inter-coderd latency is too high, but if requests start dropping, this is one metric to investigate.

We also recommend that you deploy all coderd instances such that they have low-latency connections to Postgres. coderd often makes several database round-trips while processing a single API request, so prioritizing low-latency between coderd and Postgres is more important than low-latency between users and coderd.

Note that this latency requirement applies only to Coder services. Coder will operate correctly even with few seconds of latency on workspace <-> Coder and user <-> Coder connections.

Setup

Coder automatically enters HA mode when multiple instances simultaneously connect to the same Postgres endpoint.

Note

When upgrading HA deployments, database migrations may require special handling to avoid lock contention. See Upgrading Best Practices for recommended procedures.

HA brings one configuration variable to set in each coderd node: CODER_DERP_SERVER_RELAY_URL. The HA nodes use these URLs to communicate with each other. Inter-node communication is only required while using the embedded relay (default). If you're using custom relays, Coder ignores CODER_DERP_SERVER_RELAY_URL since Postgres is the sole rendezvous for the Coder nodes.

CODER_DERP_SERVER_RELAY_URL will never be CODER_ACCESS_URL because CODER_ACCESS_URL is a load balancer to all Coder nodes.

Coder also clusters its inter-node pubsub across HA nodes and needs each replica's routable address to do so. This address comes from CODER_CLUSTER_HOST, falling back to the host in CODER_DERP_SERVER_RELAY_URL when it is unset. Set CODER_CLUSTER_HOST to each node's routable IP address. Setting it is required when you use custom relays, because CODER_DERP_SERVER_RELAY_URL is ignored and no fallback address is available. If a node has neither value set, Coder logs an error and falls back to PostgreSQL for pubsub.

Nodes connect to each other's pubsub on TCP port 6222, which isn't configurable. Allow port 6222 between the CODER_CLUSTER_HOST addresses of your coderd nodes in any firewall or Kubernetes NetworkPolicy that sits between them. If you use the embedded relay, also allow the port in CODER_DERP_SERVER_RELAY_URL. If port 6222 is blocked, Coder doesn't fall back to PostgreSQL, so events published on one node, such as workspace updates, don't reach clients connected to another node.

Here's an example 3-node network configuration setup:

Name CODER_HTTP_ADDRESS CODER_DERP_SERVER_RELAY_URL CODER_CLUSTER_HOST CODER_ACCESS_URL
coder-1 *:80 http://10.0.0.1:80 10.0.0.1 https://coder.big.corp
coder-2 *:80 http://10.0.0.2:80 10.0.0.2 https://coder.big.corp
coder-3 *:80 http://10.0.0.3:80 10.0.0.3 https://coder.big.corp

Kubernetes

If you installed Coder via our Helm Chart, just increase coder.replicaCount in values.yaml.

If you installed Coder into Kubernetes by some other means, insert the relay URL via the environment like so:

env:
  - name: POD_IP
    valueFrom:
      fieldRef:
        fieldPath: status.podIP
  - name: CODER_DERP_SERVER_RELAY_URL
    value: http://$(POD_IP)
  - name: CODER_CLUSTER_HOST
    value: $(POD_IP)

Then, increase the number of pods.

Up next