HA replicas connect to each other's embedded NATS pubsub on TCP 6222. The port is hardcoded ([`coderd/x/nats/pubsub.go`](https://github.com/coder/coder/blob/7f3d35353eddcd8a9e573094e5e2531bd2d73659/coderd/x/nats/pubsub.go#L62)) with no flag or environment variable, and since #29283 NATS pubsub is on by default. The HA page explains `CODER_CLUSTER_HOST` but doesn't name the port, so a firewall or default-deny NetworkPolicy between `coderd` nodes blocks it. This adds the port, and what happens without it, next to `CODER_CLUSTER_HOST`. Measured on v2.37.3 (`CODER_EXPERIMENTS=nats_pubsub`, the default on main) with two replicas on GKE under a default-deny NetworkPolicy, port 8080 open between the replicas in both runs. Through a port-forward to replica B, `PUT /api/v2/workspaces/{id}/extend` publishes a workspace update in the request path. A watcher on `/api/v2/workspaces/{id}/watch` through replica A counts the events. The control sends the same request through A. 6222 blocked: ```text label=blocked A=coder-699ccb59f8-695bj B=coder-699ccb59f8-w8glp baseline_data_events=1 control: extend via A -> HTTP 409; data events on A: 1 (expect 1) test: 5x extend via B -> HTTP 409 409 409 409 409; data events on A: 0 coder-699ccb59f8-695bj :6222 established: 0 coder-699ccb59f8-w8glp :6222 established: 0 ``` 6222 allowed between the replicas (same pods, no restart): ```text label=allowed A=coder-699ccb59f8-695bj B=coder-699ccb59f8-w8glp baseline_data_events=1 control: extend via A -> HTTP 409; data events on A: 1 (expect 1) test: 5x extend via B -> HTTP 409 409 409 409 409; data events on A: 5 coder-699ccb59f8-695bj :6222 established: 4 coder-699ccb59f8-w8glp :6222 established: 4 ``` The 409s are expected because the workspace is stopped; the handler publishes the update before it returns ([`coderd/workspaces.go`](https://github.com/coder/coder/blob/3a248167f6e58462a8a8cddba3b49a8ff72a9f80/coderd/workspaces.go#L1711) at v2.37.3). Would it also make sense for the chart to declare 6222 as a container port, or for the port to be configurable? I left both out of this change. AI involvement: I used an AI assistant to draft this change, and verified it with the runs above against a two-replica deployment.
4.6 KiB
title
| title |
|---|
| High availability |
High Availability (HA) mode solves for horizontal scalability and automatic failover within a single region. When in HA mode, Coder continues using a single Postgres endpoint. GCP, AWS, and other cloud vendors offer fully-managed HA Postgres services that pair nicely with Coder.
For Coder to operate correctly, coderd instances should have low-latency
connections to each other so that they can effectively relay traffic between
users and workspaces no matter which coderd instance users or workspaces
connect to. We make a best-effort attempt to warn the user when inter-coderd
latency is too high, but if requests start dropping, this is one metric to
investigate.
We also recommend that you deploy all coderd instances such that they have
low-latency connections to Postgres. coderd often makes several database
round-trips while processing a single API request, so prioritizing low-latency
between coderd and Postgres is more important than low-latency between users
and coderd.
Note that this latency requirement applies only to Coder services. Coder will operate correctly even with few seconds of latency on workspace <-> Coder and user <-> Coder connections.
Setup
Coder automatically enters HA mode when multiple instances simultaneously connect to the same Postgres endpoint.
Note
When upgrading HA deployments, database migrations may require special handling to avoid lock contention. See Upgrading Best Practices for recommended procedures.
HA brings one configuration variable to set in each coderd node:
CODER_DERP_SERVER_RELAY_URL. The HA nodes use these URLs to communicate with
each other. Inter-node communication is only required while using the embedded
relay (default). If you're using custom relays,
Coder ignores CODER_DERP_SERVER_RELAY_URL since Postgres is the sole
rendezvous for the Coder nodes.
CODER_DERP_SERVER_RELAY_URL will never be CODER_ACCESS_URL because
CODER_ACCESS_URL is a load balancer to all Coder nodes.
Coder also clusters its inter-node pubsub across HA nodes and needs each
replica's routable address to do so. This address comes from
CODER_CLUSTER_HOST, falling back to the host in CODER_DERP_SERVER_RELAY_URL
when it is unset. Set CODER_CLUSTER_HOST to each node's routable IP address.
Setting it is required when you use custom relays,
because CODER_DERP_SERVER_RELAY_URL is ignored and no fallback address is
available. If a node has neither value set, Coder logs an error and falls back
to PostgreSQL for pubsub.
Nodes connect to each other's pubsub on TCP port 6222, which isn't configurable.
Allow port 6222 between the CODER_CLUSTER_HOST addresses of your coderd nodes in any firewall or Kubernetes NetworkPolicy that sits between them.
If you use the embedded relay, also allow the port in CODER_DERP_SERVER_RELAY_URL.
If port 6222 is blocked, Coder doesn't fall back to PostgreSQL, so events published on one node, such as workspace updates, don't reach clients connected to another node.
Here's an example 3-node network configuration setup:
| Name | CODER_HTTP_ADDRESS |
CODER_DERP_SERVER_RELAY_URL |
CODER_CLUSTER_HOST |
CODER_ACCESS_URL |
|---|---|---|---|---|
coder-1 |
*:80 |
http://10.0.0.1:80 |
10.0.0.1 |
https://coder.big.corp |
coder-2 |
*:80 |
http://10.0.0.2:80 |
10.0.0.2 |
https://coder.big.corp |
coder-3 |
*:80 |
http://10.0.0.3:80 |
10.0.0.3 |
https://coder.big.corp |
Kubernetes
If you installed Coder via
our Helm Chart, just
increase coder.replicaCount in values.yaml.
If you installed Coder into Kubernetes by some other means, insert the relay URL via the environment like so:
env:
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: CODER_DERP_SERVER_RELAY_URL
value: http://$(POD_IP)
- name: CODER_CLUSTER_HOST
value: $(POD_IP)
Then, increase the number of pods.