docs: address review feedback on connection tracing guide

This commit is contained in:
Atif Ali authored and GitHub committed 2026-10-27 21:57:42 +05:00
1 parent caf1bb7c85
commit 052d9e840f
3 files changed
+83 -54

No files matched your search

+81 -52
View File
@@ -2,23 +2,26 @@
title: Trace workspace connections
---
This guide is for Coder deployment administrators and platform teams who investigate dropped or failed workspace connections from `coder ssh`, VS Code, JetBrains Toolbox, and the web terminal.
It shows how to collect logs from the user's client, the Coder control plane (`coderd`), and the workspace agent; correlate them with a client session ID; and read the fields that explain why a connection ended.
This guide is for Coder deployment administrators and platform teams who need to correlate workspace connection logs with client session IDs.
It covers `coder ssh`, VS Code, JetBrains Toolbox, and the web terminal.
You'll collect logs from the user's client, `coderd` (the process that runs the control plane), and the workspace agent.
Then you'll match entries by session ID and read the connection and workspace lifecycle fields.
For connection log filters and export formats, refer to [Connection logs](./connection-logs.md).
To diagnose a connection that is still open, refer to [Networking troubleshooting](../networking/troubleshooting.md).
## Before you begin
- Update workspace agents to the same version as your Coder server.
- Update the Coder CLI and workspace agents to the same version as your control plane.
Agents from earlier versions keep working, but they don't record client session IDs, app names, workspace shutdown reasons, or bytes transferred.
- Collect `coderd` logs with a log management tool such as Splunk, Datadog, or Grafana Loki.
JSON output is easiest to query.
JSON output is best for querying by other programs, such as `jq`.
To write JSON logs, set [`CODER_LOGGING_JSON`](../../reference/cli/server/index.md#--log-json) to a file path, such as `/dev/stderr`.
Human-readable logs also go to `/dev/stderr` by default, so set [`CODER_LOGGING_HUMAN`](../../reference/cli/server/index.md#--log-human) to `/dev/null` to keep only JSON there.
- The `connection_log` entries in this guide require a [Premium license](https://coder.com/pricing#compare-plans).
Client and agent logs don't.
- For VS Code, use the Coder Remote extension version 1.17.2 or later.
Earlier versions label output channel lines with only the first 8 characters of the session ID, and versions before 1.17.1 don't report the editor name, such as `cursor`.
Earlier versions label output channel lines with only the first 8 characters of the session ID.
Versions before 1.17.0 don't report the editor name, such as `cursor`.
- For JetBrains Toolbox, use the Coder Toolbox plugin version 0.9.5 or later to send session IDs and to include `coder ssh` logs in **Collect logs**.
Plugin version 0.9.5 requires JetBrains Toolbox version 3.9.0 or later.
@@ -27,7 +30,7 @@ To diagnose a connection that is still open, refer to [Networking troubleshootin
Each supported client creates a client session ID, a 32-character lowercase hexadecimal value.
The client writes the ID to its own logs and sends it with its requests to `coderd` and the workspace agent.
`coderd`, workspace proxies, and the agent add the same ID to their log lines as `client_session_id`.
The agent also records it in the connection log as `ClientSessionID` for SSH and web terminal connections.
The agent also records it in the connection log as `ClientSessionID` for SSH sessions, web terminal connections, and JetBrains Toolbox forwarded-channel connections.
Search each log source for one value to follow a session from the client, through `coderd`, to the agent.
What counts as one session depends on the client:
@@ -49,7 +52,7 @@ Each SSH session or web terminal connection has its own `ConnectionID`, shared b
## Find the session ID
Ask the user for their client logs from the time of the failure, then find the `client_session_id` field.
Ask the user for their client logs from the session, then find the `client_session_id` field.
If the user can't provide them, start from the connection log for their workspace and time window instead, as described in [Correlate a session](#correlate-a-session).
### `coder ssh`
@@ -65,7 +68,8 @@ The default directory depends on the operating system:
File names start with `coder-ssh-` followed by the date and time the session started, for example `coder-ssh-20261013-141502-abcde.log`.
When `coder ssh` runs as an OpenSSH proxy command, the name also ends with the parent process ID.
The CLI keeps the 50 most recent files.
Before it creates a new file, the CLI deletes all but the 50 most recently modified `coder-*.log` files in the directory.
If removal fails, the CLI logs a warning and continues the session.
To write them somewhere else, set [`--log-dir`](../../reference/cli/ssh.md#-l---log-dir) or `CODER_SSH_LOG_DIR`.
Every line in the file includes `client_session_id`.
@@ -78,7 +82,7 @@ To record every debug entry, set `--verbose` or `CODER_VERBOSE=true`.
Ask the user to run **Coder: Create Support Bundle** from the Command Palette.
The bundle includes the extension logs and the `coder ssh` logs under `vscode-logs/`.
For details, refer to [Generate a support bundle](../../support/support-bundle.md#vs-code).
For more details, refer to [Generate a support bundle](../../support/support-bundle.md#vs-code).
The extension starts `coder ssh` with debug logging.
It writes the `coder ssh` logs to the directory in the `coder.proxyLogDirectory` VS Code setting.
@@ -93,10 +97,12 @@ Toolbox starts `coder ssh`.
Toolbox log lines for a session include the full 32-character `client_session_id`.
With Coder Toolbox plugin version 0.9.5 or later, the plugin chooses the directory for the `coder ssh` logs: the SSH log directory in the Coder plugin settings, then `CODER_SSH_LOG_DIR`, then an `ssh-logs` directory in the plugin's data directory for the deployment.
Ask the user to select **Collect logs** from the workspace action menu.
With plugin version 0.9.5 or later, the archive includes the Toolbox logs and a nested Coder support bundle, `coder-support.zip`, with recent `coder ssh` logs under `toolbox-logs/ssh/`.
With plugin version 0.9.5 or later, successful collection includes Toolbox logs and a nested Coder support bundle, `coder-support.zip`.
The nested bundle contains recent `coder ssh` logs under `toolbox-logs/ssh/`.
If Coder bundle collection fails, the plugin attempts to include the SSH logs in `coder-ssh.zip` and collection details in `coder-support-error.txt`.
With earlier plugin versions, **Collect logs** includes only the Toolbox logs.
In that case, `coder ssh` writes to the [`coder ssh` log directory](#coder-ssh) unless an SSH log directory is set in the plugin settings, so ask the user for those files separately and [generate a support bundle with the CLI](../../support/support-bundle.md#generate-with-the-cli).
For details, refer to [Generate a support bundle](../../support/support-bundle.md#jetbrains-toolbox) and [JetBrains Toolbox troubleshooting](../../user-guides/workspace-access/jetbrains/toolbox.md#troubleshooting).
For details, refer to [Generate a support bundle](../../support/support-bundle.md) and [JetBrains Toolbox troubleshooting](../../user-guides/workspace-access/jetbrains/toolbox.md#troubleshooting).
### Web terminal
@@ -105,7 +111,7 @@ If the user didn't keep the console output, find the session in the agent logs o
## Collect server-side logs
Gather `coderd` and agent logs that cover the time of the failure.
Gather `coderd` and agent logs that cover the session's time window.
### `coderd` logs
@@ -114,15 +120,15 @@ For SSH sessions and web terminal connections, the agent reports both events, so
JetBrains Toolbox sessions work differently, as described in [Agent logs](#agent-logs).
The fields that matter most for tracing are in the following table:
| Field | Description |
|----------------------|-------------------------------------------------------------------------------------------------------------------------------------------|
| `ClientSessionID` | The client session ID. Empty for workspace app, port forwarding, and tunnel events, and for connections from older clients or agents. |
| `ConnectionID` | Identifies one SSH session or web terminal connection. For SSH, matches the `id` field in the agent log. |
| `ConnectionStatus` | `connected` or `disconnected`. |
| `ConnectionMethod` | How the connection was made, such as `ssh` or `reconnecting_pty`. |
| `AppNameOrPort` | The app that made an SSH or web terminal connection, such as `vscode` or `cursor`. |
| `DisconnectReason` | Why the connection closed, on `disconnected` entries. Refer to [Read disconnect reasons](#read-disconnect-reasons). |
| `RxBytes`, `TxBytes` | Bytes the agent received and sent over an SSH or web terminal connection. Set only on `disconnected` entries, and empty for older agents. |
| Field | Description |
|----------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------|
| `ClientSessionID` | The client session ID. Empty for workspace app, port forwarding, and tunnel events, and for connections from older clients or agents. |
| `ConnectionID` | Identifies an SSH session, web terminal connection, or tracked JetBrains forwarded channel. For SSH sessions, matches the `id` field in the agent log. |
| `ConnectionStatus` | `connected` or `disconnected`. |
| `ConnectionMethod` | How the connection was made, such as `ssh` or `reconnecting_pty`. |
| `AppNameOrPort` | The app that made an SSH or web terminal connection, such as `vscode` or `cursor`. |
| `DisconnectReason` | Why the connection closed, on `disconnected` entries. Refer to [Read disconnect reasons](#read-disconnect-reasons). |
| `RxBytes`, `TxBytes` | Bytes the agent received and sent over an SSH or web terminal connection. Set only on `disconnected` entries, and empty for older agents. |
For every field and an example entry, refer to [Service logs](./connection-logs.md#service-logs).
@@ -134,7 +140,7 @@ To turn it on, set [`CODER_LOG_FILTER`](../../reference/cli/server/index.md#-l--
### Agent logs
The workspace agent writes its log to `coder-agent.log` in the directory set by `CODER_AGENT_LOG_DIR`.
The agent rotates the file at 5&nbsp;MB and keeps up to 19 older files named `coder-agent-<timestamp>.log` in the same directory.
The agent rotates the file at 5&nbsp;MB and keeps up to 19 rotated files named `coder-agent-<timestamp>.log` in the same directory.
If `CODER_AGENT_LOG_DIR` isn't set, the agent uses the system temporary directory.
On Linux and macOS, that's `$TMPDIR` if it's set, or `/tmp` otherwise.
On Windows, it's the user's temporary directory, usually `%TEMP%`.
@@ -157,9 +163,12 @@ Search the agent log for these messages:
| `ssh session failed` | An SSH session failed with an error. The error text becomes the `disconnect_reason`. |
| `ssh connection complete` | An SSH connection closed. If the agent was shutting down, Coder also includes `disconnect_reason`, `disconnect_expected`, and `disconnect_initiator`. |
| `ssh connection failed` | An SSH connection failed before a session started. |
| `accepted conn`, `reconnecting pty closed` | A web terminal connection opened or closed. The closing entry includes `disconnect_reason`. |
| `accepted conn`, `reconnecting pty closed` | A web terminal connection opened or closed, from the `reconnecting-pty` logger. The closing entry includes `disconnect_reason`. |
| `disconnected from coderd` | The agent lost its connection to `coderd`. The buffered debug entries are written just before this line. |
The `speedtest` logger also writes `accepted conn` for `coder speedtest` connections.
Use the logger name to distinguish those entries from web terminal connections.
JetBrains Toolbox opens many short SSH sessions, so `handling ssh session` lines are frequent for Toolbox users.
For those sessions, the agent doesn't log `ssh session closed` or report them to the connection log.
Instead, it reports the forwarded channel that Toolbox keeps open as one connection, and logs `JetBrains watcher accepted channel` and `JetBrains channel closed` at the debug level.
@@ -176,28 +185,44 @@ With a session ID and the logs, rebuild the timeline of the session:
jq -c 'select(.msg == "connection_log" and .fields.ClientSessionID == "<session-id>")' coderd.log
```
Each matching entry names the workspace, the agent, the connection method and app, and the `ConnectionID`.
Note the time of each `connected` and `disconnected` entry and its `DisconnectReason`.
Each entry names the workspace, agent, connection method, app, and `ConnectionID`.
1. Note the time of each `connected` and `disconnected` entry and its `DisconnectReason`.
1. Search the `coderd` logs for `client_session_id` set to the same value.
Matching warn-level entries show requests that failed with a `5xx` status.
To see `4xx` failures, turn on debug logging.
Warn-level entries show `5xx` responses.
Debug logs also include `4xx` responses, as described in [`coderd` logs](#coderd-logs).
1. Search the agent log for the session ID:
On Linux and macOS:
```sh
grep '<session-id>' <agent-log-dir>/coder-agent*.log
```
On Windows workspaces, use PowerShell:
```ps1
Select-String -Path '<agent-log-dir>\coder-agent*.log' -Pattern '<session-id>' -SimpleMatch
```
The matching lines show each SSH or web terminal session the client opened and the reason it closed.
1. Line up the client log with the server-side entries by time.
The last client entries before a disconnect usually show the error the client hit.
If the client log has the session ID but there are no `connection_log` entries for it, either the connection didn't reach the agent or the agent is older than your Coder server.
Check the `coderd` logs for the session ID, and filter the connection log with `method:tunnel` for the same workspace and time to find denied tunnel requests.
Matching timestamps and session IDs connect the client activity to the reported connection events.
If you start without a session ID, filter the connection log by `workspace_owner`, `workspace_id`, and time, then take `ClientSessionID` from the matching service log entries.
A client session ID doesn't guarantee a corresponding `connection_log` entry.
Connections that don't reach the agent and earlier agents that don't record session IDs can leave no matching entries.
Reports can also be lost if the agent's reporting buffer fills or a report to the control plane fails.
The agent logs `connection report buffer limit reached, dropping report` or `failed to report connection to server` for those reporting failures.
If you start without a session ID, filter the connection log by `workspace_owner` or `workspace_id`.
Use `connected_after` and `connected_before` with RFC3339 timestamps to limit connection start times.
Both time boundaries are inclusive.
Then take `ClientSessionID` from the corresponding `connection_log` service entries.
## Read disconnect reasons
@@ -206,36 +231,40 @@ The two can differ after a workspace stop or delete: the agent logs `server_shut
`coderd` records `workspace_stopped` only when a stop or delete build that was requested before the connection closed is running or has succeeded.
Otherwise, the connection log keeps `server_shutdown`.
| `disconnect_reason` | Meaning |
|---------------------|----------------------------------------------------------------------------------------------------------------------------------------------------|
| `graceful` | The connection ended cleanly, for example when the SSH command exited with status `0`. |
| `network_error` | A web terminal connection's transport failed, for example with an EOF, a read or write error, or a timeout. SSH sessions don't record this reason. |
| `server_shutdown` | The agent closed the connection because the agent was shutting down. |
| `workspace_stopped` | A workspace stop or delete closed the connection. Only the connection log and the `coder ssh` log record this reason. |
| `unknown` | The agent couldn't determine why the connection closed. Only the agent log shows `unknown`; in the connection log, `DisconnectReason` is empty. |
| `normal close` | A JetBrains Toolbox forwarded channel closed while the agent was running. Only the connection log records this reason. |
| `disconnect_reason` | Meaning |
|---------------------|------------------------------------------------------------------------------------------------------------------------------------------------------|
| `graceful` | The connection ended cleanly, for example when the SSH command exited with status `0`. |
| `network_error` | A web terminal connection returned a setup or attachment error, or the agent lost its connection to `coderd`. SSH sessions don't record this reason. |
| `server_shutdown` | The agent closed the connection because the agent was shutting down. |
| `workspace_stopped` | A workspace stop or delete closed the connection. Only the connection log and the `coder ssh` log record this reason. |
| `unknown` | The agent couldn't determine why the connection closed. Only the agent log shows `unknown`; in the connection log, `DisconnectReason` is empty. |
| `normal close` | A JetBrains Toolbox forwarded channel closed while the agent was running. Only the connection log records this reason. |
For web terminal connections, `network_error` includes transport errors and errors creating the terminal command.
The agent also records it on `disconnected from coderd`, with `connect_type=server_to_agent`.
For SSH sessions, the reason can also be a specific message, such as `process exited with error status: 1` when the command exited with a nonzero status, or the raw error text when the session failed.
SSH sessions don't record `network_error`, so search the agent log for `ssh session failed` to find SSH transport failures.
SSH sessions don't record `network_error`.
If the agent was shutting down when an SSH session ended, the reason is `server_shutdown`, and the agent log keeps the specific message in `disconnect_detail`.
In the agent log, `disconnect_expected` is `true` for `graceful`, `server_shutdown`, and `workspace_stopped`, and `false` for `network_error`, `unknown`, and error messages.
Filter the agent log on `disconnect_expected=false` to find the disconnects worth investigating.
In the agent log, `disconnect_expected` is `true` for `graceful` and `server_shutdown`.
It's `false` for `network_error`, `unknown`, and specific error messages, including nonzero remote-command exit statuses.
A value of `false` can describe an ordinary command failure, not a network failure.
`connection_log` entries don't include `disconnect_expected`.
When a workspace stop or delete ends a `coder ssh` session, the `coder ssh` log has a `workspace is shutting down` line with `disconnect_reason` set to `workspace_stopped`, `build_id`, `build_reason`, and `transition`.
The agent log doesn't include these build fields.
`build_reason` shows what stopped the workspace:
| `build_reason` | The workspace stopped because |
|----------------|--------------------------------------------------------------------------------------|
| `initiator` | A user or API client requested the build. |
| `autostop` | The workspace reached its autostop schedule. |
| `dormancy` | The workspace became dormant from inactivity. |
| `autodelete` | Coder deleted the dormant workspace automatically, based on the template's schedule. |
| `build_reason` | The workspace stopped because |
|----------------|--------------------------------------------------------------------------------------------------------------------------|
| `initiator` | A user or API client requested the build. |
| `autostop` | The workspace reached its autostop deadline, its owner was suspended, or a failed start or stop build triggered cleanup. |
| `dormancy` | The workspace became dormant from inactivity. |
| `autodelete` | Coder deleted the dormant workspace automatically, based on the template's schedule. |
## Next steps
## Learn more
- [Connection logs](./connection-logs.md): filter, export, and retain connection events.
- [Generate a support bundle](../../support/support-bundle.md): collect client, agent, and deployment diagnostics in one archive.
- [Networking troubleshooting](../networking/troubleshooting.md): diagnose direct and relayed connections with `coder ping`.
- [Connection logs](./connection-logs.md): filter, export, and retain connection events
- [Generate a support bundle](../../support/support-bundle.md): collect client, agent, and deployment diagnostics in one archive
- [Networking troubleshooting](../networking/troubleshooting.md): diagnose direct and relayed connections with `coder ping`
+1 -1
View File
@@ -1003,7 +1003,7 @@
},
{
"title": "Trace workspace connections",
"description": "Correlate client, coderd, and agent logs to find why an SSH, IDE, or web terminal connection failed.",
"description": "Correlate client, coderd, and agent logs by session ID for SSH, IDE, and web terminal connections.",
"path": "./admin/monitoring/trace-connections.md"
},
{
+1 -1
View File
@@ -17,6 +17,6 @@ children:
path: ./admin/monitoring/connection-logs.md
state: [premium]
- title: Trace workspace connections
description: Correlate client, coderd, and agent logs to find why an SSH, IDE, or web terminal connection failed.
description: Correlate client, coderd, and agent logs by session ID for SSH, IDE, and web terminal connections.
path: ./admin/monitoring/trace-connections.md
- include: admin/monitoring/notifications.yml