docs: address review feedback on connection tracing guide
This commit is contained in:
1 parent
caf1bb7c85
commit
052d9e840f
3 files changed
+83
-54
No files matched your search
@@ -2,23 +2,26 @@
|
|||||||
title: Trace workspace connections
|
title: Trace workspace connections
|
||||||
---
|
---
|
||||||
|
|
||||||
This guide is for Coder deployment administrators and platform teams who investigate dropped or failed workspace connections from `coder ssh`, VS Code, JetBrains Toolbox, and the web terminal.
|
This guide is for Coder deployment administrators and platform teams who need to correlate workspace connection logs with client session IDs.
|
||||||
It shows how to collect logs from the user's client, the Coder control plane (`coderd`), and the workspace agent; correlate them with a client session ID; and read the fields that explain why a connection ended.
|
It covers `coder ssh`, VS Code, JetBrains Toolbox, and the web terminal.
|
||||||
|
You'll collect logs from the user's client, `coderd` (the process that runs the control plane), and the workspace agent.
|
||||||
|
Then you'll match entries by session ID and read the connection and workspace lifecycle fields.
|
||||||
For connection log filters and export formats, refer to [Connection logs](./connection-logs.md).
|
For connection log filters and export formats, refer to [Connection logs](./connection-logs.md).
|
||||||
To diagnose a connection that is still open, refer to [Networking troubleshooting](../networking/troubleshooting.md).
|
To diagnose a connection that is still open, refer to [Networking troubleshooting](../networking/troubleshooting.md).
|
||||||
|
|
||||||
## Before you begin
|
## Before you begin
|
||||||
|
|
||||||
- Update workspace agents to the same version as your Coder server.
|
- Update the Coder CLI and workspace agents to the same version as your control plane.
|
||||||
Agents from earlier versions keep working, but they don't record client session IDs, app names, workspace shutdown reasons, or bytes transferred.
|
Agents from earlier versions keep working, but they don't record client session IDs, app names, workspace shutdown reasons, or bytes transferred.
|
||||||
- Collect `coderd` logs with a log management tool such as Splunk, Datadog, or Grafana Loki.
|
- Collect `coderd` logs with a log management tool such as Splunk, Datadog, or Grafana Loki.
|
||||||
JSON output is easiest to query.
|
JSON output is best for querying by other programs, such as `jq`.
|
||||||
To write JSON logs, set [`CODER_LOGGING_JSON`](../../reference/cli/server/index.md#--log-json) to a file path, such as `/dev/stderr`.
|
To write JSON logs, set [`CODER_LOGGING_JSON`](../../reference/cli/server/index.md#--log-json) to a file path, such as `/dev/stderr`.
|
||||||
Human-readable logs also go to `/dev/stderr` by default, so set [`CODER_LOGGING_HUMAN`](../../reference/cli/server/index.md#--log-human) to `/dev/null` to keep only JSON there.
|
Human-readable logs also go to `/dev/stderr` by default, so set [`CODER_LOGGING_HUMAN`](../../reference/cli/server/index.md#--log-human) to `/dev/null` to keep only JSON there.
|
||||||
- The `connection_log` entries in this guide require a [Premium license](https://coder.com/pricing#compare-plans).
|
- The `connection_log` entries in this guide require a [Premium license](https://coder.com/pricing#compare-plans).
|
||||||
Client and agent logs don't.
|
Client and agent logs don't.
|
||||||
- For VS Code, use the Coder Remote extension version 1.17.2 or later.
|
- For VS Code, use the Coder Remote extension version 1.17.2 or later.
|
||||||
Earlier versions label output channel lines with only the first 8 characters of the session ID, and versions before 1.17.1 don't report the editor name, such as `cursor`.
|
Earlier versions label output channel lines with only the first 8 characters of the session ID.
|
||||||
|
Versions before 1.17.0 don't report the editor name, such as `cursor`.
|
||||||
- For JetBrains Toolbox, use the Coder Toolbox plugin version 0.9.5 or later to send session IDs and to include `coder ssh` logs in **Collect logs**.
|
- For JetBrains Toolbox, use the Coder Toolbox plugin version 0.9.5 or later to send session IDs and to include `coder ssh` logs in **Collect logs**.
|
||||||
Plugin version 0.9.5 requires JetBrains Toolbox version 3.9.0 or later.
|
Plugin version 0.9.5 requires JetBrains Toolbox version 3.9.0 or later.
|
||||||
|
|
||||||
@@ -27,7 +30,7 @@ To diagnose a connection that is still open, refer to [Networking troubleshootin
|
|||||||
Each supported client creates a client session ID, a 32-character lowercase hexadecimal value.
|
Each supported client creates a client session ID, a 32-character lowercase hexadecimal value.
|
||||||
The client writes the ID to its own logs and sends it with its requests to `coderd` and the workspace agent.
|
The client writes the ID to its own logs and sends it with its requests to `coderd` and the workspace agent.
|
||||||
`coderd`, workspace proxies, and the agent add the same ID to their log lines as `client_session_id`.
|
`coderd`, workspace proxies, and the agent add the same ID to their log lines as `client_session_id`.
|
||||||
The agent also records it in the connection log as `ClientSessionID` for SSH and web terminal connections.
|
The agent also records it in the connection log as `ClientSessionID` for SSH sessions, web terminal connections, and JetBrains Toolbox forwarded-channel connections.
|
||||||
Search each log source for one value to follow a session from the client, through `coderd`, to the agent.
|
Search each log source for one value to follow a session from the client, through `coderd`, to the agent.
|
||||||
|
|
||||||
What counts as one session depends on the client:
|
What counts as one session depends on the client:
|
||||||
@@ -49,7 +52,7 @@ Each SSH session or web terminal connection has its own `ConnectionID`, shared b
|
|||||||
|
|
||||||
## Find the session ID
|
## Find the session ID
|
||||||
|
|
||||||
Ask the user for their client logs from the time of the failure, then find the `client_session_id` field.
|
Ask the user for their client logs from the session, then find the `client_session_id` field.
|
||||||
If the user can't provide them, start from the connection log for their workspace and time window instead, as described in [Correlate a session](#correlate-a-session).
|
If the user can't provide them, start from the connection log for their workspace and time window instead, as described in [Correlate a session](#correlate-a-session).
|
||||||
|
|
||||||
### `coder ssh`
|
### `coder ssh`
|
||||||
@@ -65,7 +68,8 @@ The default directory depends on the operating system:
|
|||||||
|
|
||||||
File names start with `coder-ssh-` followed by the date and time the session started, for example `coder-ssh-20261013-141502-abcde.log`.
|
File names start with `coder-ssh-` followed by the date and time the session started, for example `coder-ssh-20261013-141502-abcde.log`.
|
||||||
When `coder ssh` runs as an OpenSSH proxy command, the name also ends with the parent process ID.
|
When `coder ssh` runs as an OpenSSH proxy command, the name also ends with the parent process ID.
|
||||||
The CLI keeps the 50 most recent files.
|
Before it creates a new file, the CLI deletes all but the 50 most recently modified `coder-*.log` files in the directory.
|
||||||
|
If removal fails, the CLI logs a warning and continues the session.
|
||||||
To write them somewhere else, set [`--log-dir`](../../reference/cli/ssh.md#-l---log-dir) or `CODER_SSH_LOG_DIR`.
|
To write them somewhere else, set [`--log-dir`](../../reference/cli/ssh.md#-l---log-dir) or `CODER_SSH_LOG_DIR`.
|
||||||
|
|
||||||
Every line in the file includes `client_session_id`.
|
Every line in the file includes `client_session_id`.
|
||||||
@@ -78,7 +82,7 @@ To record every debug entry, set `--verbose` or `CODER_VERBOSE=true`.
|
|||||||
|
|
||||||
Ask the user to run **Coder: Create Support Bundle** from the Command Palette.
|
Ask the user to run **Coder: Create Support Bundle** from the Command Palette.
|
||||||
The bundle includes the extension logs and the `coder ssh` logs under `vscode-logs/`.
|
The bundle includes the extension logs and the `coder ssh` logs under `vscode-logs/`.
|
||||||
For details, refer to [Generate a support bundle](../../support/support-bundle.md#vs-code).
|
For more details, refer to [Generate a support bundle](../../support/support-bundle.md#vs-code).
|
||||||
|
|
||||||
The extension starts `coder ssh` with debug logging.
|
The extension starts `coder ssh` with debug logging.
|
||||||
It writes the `coder ssh` logs to the directory in the `coder.proxyLogDirectory` VS Code setting.
|
It writes the `coder ssh` logs to the directory in the `coder.proxyLogDirectory` VS Code setting.
|
||||||
@@ -93,10 +97,12 @@ Toolbox starts `coder ssh`.
|
|||||||
Toolbox log lines for a session include the full 32-character `client_session_id`.
|
Toolbox log lines for a session include the full 32-character `client_session_id`.
|
||||||
With Coder Toolbox plugin version 0.9.5 or later, the plugin chooses the directory for the `coder ssh` logs: the SSH log directory in the Coder plugin settings, then `CODER_SSH_LOG_DIR`, then an `ssh-logs` directory in the plugin's data directory for the deployment.
|
With Coder Toolbox plugin version 0.9.5 or later, the plugin chooses the directory for the `coder ssh` logs: the SSH log directory in the Coder plugin settings, then `CODER_SSH_LOG_DIR`, then an `ssh-logs` directory in the plugin's data directory for the deployment.
|
||||||
Ask the user to select **Collect logs** from the workspace action menu.
|
Ask the user to select **Collect logs** from the workspace action menu.
|
||||||
With plugin version 0.9.5 or later, the archive includes the Toolbox logs and a nested Coder support bundle, `coder-support.zip`, with recent `coder ssh` logs under `toolbox-logs/ssh/`.
|
With plugin version 0.9.5 or later, successful collection includes Toolbox logs and a nested Coder support bundle, `coder-support.zip`.
|
||||||
|
The nested bundle contains recent `coder ssh` logs under `toolbox-logs/ssh/`.
|
||||||
|
If Coder bundle collection fails, the plugin attempts to include the SSH logs in `coder-ssh.zip` and collection details in `coder-support-error.txt`.
|
||||||
With earlier plugin versions, **Collect logs** includes only the Toolbox logs.
|
With earlier plugin versions, **Collect logs** includes only the Toolbox logs.
|
||||||
In that case, `coder ssh` writes to the [`coder ssh` log directory](#coder-ssh) unless an SSH log directory is set in the plugin settings, so ask the user for those files separately and [generate a support bundle with the CLI](../../support/support-bundle.md#generate-with-the-cli).
|
In that case, `coder ssh` writes to the [`coder ssh` log directory](#coder-ssh) unless an SSH log directory is set in the plugin settings, so ask the user for those files separately and [generate a support bundle with the CLI](../../support/support-bundle.md#generate-with-the-cli).
|
||||||
For details, refer to [Generate a support bundle](../../support/support-bundle.md#jetbrains-toolbox) and [JetBrains Toolbox troubleshooting](../../user-guides/workspace-access/jetbrains/toolbox.md#troubleshooting).
|
For details, refer to [Generate a support bundle](../../support/support-bundle.md) and [JetBrains Toolbox troubleshooting](../../user-guides/workspace-access/jetbrains/toolbox.md#troubleshooting).
|
||||||
|
|
||||||
### Web terminal
|
### Web terminal
|
||||||
|
|
||||||
@@ -105,7 +111,7 @@ If the user didn't keep the console output, find the session in the agent logs o
|
|||||||
|
|
||||||
## Collect server-side logs
|
## Collect server-side logs
|
||||||
|
|
||||||
Gather `coderd` and agent logs that cover the time of the failure.
|
Gather `coderd` and agent logs that cover the session's time window.
|
||||||
|
|
||||||
### `coderd` logs
|
### `coderd` logs
|
||||||
|
|
||||||
@@ -114,15 +120,15 @@ For SSH sessions and web terminal connections, the agent reports both events, so
|
|||||||
JetBrains Toolbox sessions work differently, as described in [Agent logs](#agent-logs).
|
JetBrains Toolbox sessions work differently, as described in [Agent logs](#agent-logs).
|
||||||
The fields that matter most for tracing are in the following table:
|
The fields that matter most for tracing are in the following table:
|
||||||
|
|
||||||
| Field | Description |
|
| Field | Description |
|
||||||
|----------------------|-------------------------------------------------------------------------------------------------------------------------------------------|
|
|----------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||||
| `ClientSessionID` | The client session ID. Empty for workspace app, port forwarding, and tunnel events, and for connections from older clients or agents. |
|
| `ClientSessionID` | The client session ID. Empty for workspace app, port forwarding, and tunnel events, and for connections from older clients or agents. |
|
||||||
| `ConnectionID` | Identifies one SSH session or web terminal connection. For SSH, matches the `id` field in the agent log. |
|
| `ConnectionID` | Identifies an SSH session, web terminal connection, or tracked JetBrains forwarded channel. For SSH sessions, matches the `id` field in the agent log. |
|
||||||
| `ConnectionStatus` | `connected` or `disconnected`. |
|
| `ConnectionStatus` | `connected` or `disconnected`. |
|
||||||
| `ConnectionMethod` | How the connection was made, such as `ssh` or `reconnecting_pty`. |
|
| `ConnectionMethod` | How the connection was made, such as `ssh` or `reconnecting_pty`. |
|
||||||
| `AppNameOrPort` | The app that made an SSH or web terminal connection, such as `vscode` or `cursor`. |
|
| `AppNameOrPort` | The app that made an SSH or web terminal connection, such as `vscode` or `cursor`. |
|
||||||
| `DisconnectReason` | Why the connection closed, on `disconnected` entries. Refer to [Read disconnect reasons](#read-disconnect-reasons). |
|
| `DisconnectReason` | Why the connection closed, on `disconnected` entries. Refer to [Read disconnect reasons](#read-disconnect-reasons). |
|
||||||
| `RxBytes`, `TxBytes` | Bytes the agent received and sent over an SSH or web terminal connection. Set only on `disconnected` entries, and empty for older agents. |
|
| `RxBytes`, `TxBytes` | Bytes the agent received and sent over an SSH or web terminal connection. Set only on `disconnected` entries, and empty for older agents. |
|
||||||
|
|
||||||
For every field and an example entry, refer to [Service logs](./connection-logs.md#service-logs).
|
For every field and an example entry, refer to [Service logs](./connection-logs.md#service-logs).
|
||||||
|
|
||||||
@@ -134,7 +140,7 @@ To turn it on, set [`CODER_LOG_FILTER`](../../reference/cli/server/index.md#-l--
|
|||||||
### Agent logs
|
### Agent logs
|
||||||
|
|
||||||
The workspace agent writes its log to `coder-agent.log` in the directory set by `CODER_AGENT_LOG_DIR`.
|
The workspace agent writes its log to `coder-agent.log` in the directory set by `CODER_AGENT_LOG_DIR`.
|
||||||
The agent rotates the file at 5 MB and keeps up to 19 older files named `coder-agent-<timestamp>.log` in the same directory.
|
The agent rotates the file at 5 MB and keeps up to 19 rotated files named `coder-agent-<timestamp>.log` in the same directory.
|
||||||
If `CODER_AGENT_LOG_DIR` isn't set, the agent uses the system temporary directory.
|
If `CODER_AGENT_LOG_DIR` isn't set, the agent uses the system temporary directory.
|
||||||
On Linux and macOS, that's `$TMPDIR` if it's set, or `/tmp` otherwise.
|
On Linux and macOS, that's `$TMPDIR` if it's set, or `/tmp` otherwise.
|
||||||
On Windows, it's the user's temporary directory, usually `%TEMP%`.
|
On Windows, it's the user's temporary directory, usually `%TEMP%`.
|
||||||
@@ -157,9 +163,12 @@ Search the agent log for these messages:
|
|||||||
| `ssh session failed` | An SSH session failed with an error. The error text becomes the `disconnect_reason`. |
|
| `ssh session failed` | An SSH session failed with an error. The error text becomes the `disconnect_reason`. |
|
||||||
| `ssh connection complete` | An SSH connection closed. If the agent was shutting down, Coder also includes `disconnect_reason`, `disconnect_expected`, and `disconnect_initiator`. |
|
| `ssh connection complete` | An SSH connection closed. If the agent was shutting down, Coder also includes `disconnect_reason`, `disconnect_expected`, and `disconnect_initiator`. |
|
||||||
| `ssh connection failed` | An SSH connection failed before a session started. |
|
| `ssh connection failed` | An SSH connection failed before a session started. |
|
||||||
| `accepted conn`, `reconnecting pty closed` | A web terminal connection opened or closed. The closing entry includes `disconnect_reason`. |
|
| `accepted conn`, `reconnecting pty closed` | A web terminal connection opened or closed, from the `reconnecting-pty` logger. The closing entry includes `disconnect_reason`. |
|
||||||
| `disconnected from coderd` | The agent lost its connection to `coderd`. The buffered debug entries are written just before this line. |
|
| `disconnected from coderd` | The agent lost its connection to `coderd`. The buffered debug entries are written just before this line. |
|
||||||
|
|
||||||
|
The `speedtest` logger also writes `accepted conn` for `coder speedtest` connections.
|
||||||
|
Use the logger name to distinguish those entries from web terminal connections.
|
||||||
|
|
||||||
JetBrains Toolbox opens many short SSH sessions, so `handling ssh session` lines are frequent for Toolbox users.
|
JetBrains Toolbox opens many short SSH sessions, so `handling ssh session` lines are frequent for Toolbox users.
|
||||||
For those sessions, the agent doesn't log `ssh session closed` or report them to the connection log.
|
For those sessions, the agent doesn't log `ssh session closed` or report them to the connection log.
|
||||||
Instead, it reports the forwarded channel that Toolbox keeps open as one connection, and logs `JetBrains watcher accepted channel` and `JetBrains channel closed` at the debug level.
|
Instead, it reports the forwarded channel that Toolbox keeps open as one connection, and logs `JetBrains watcher accepted channel` and `JetBrains channel closed` at the debug level.
|
||||||
@@ -176,28 +185,44 @@ With a session ID and the logs, rebuild the timeline of the session:
|
|||||||
jq -c 'select(.msg == "connection_log" and .fields.ClientSessionID == "<session-id>")' coderd.log
|
jq -c 'select(.msg == "connection_log" and .fields.ClientSessionID == "<session-id>")' coderd.log
|
||||||
```
|
```
|
||||||
|
|
||||||
Each matching entry names the workspace, the agent, the connection method and app, and the `ConnectionID`.
|
Each entry names the workspace, agent, connection method, app, and `ConnectionID`.
|
||||||
Note the time of each `connected` and `disconnected` entry and its `DisconnectReason`.
|
|
||||||
|
1. Note the time of each `connected` and `disconnected` entry and its `DisconnectReason`.
|
||||||
|
|
||||||
1. Search the `coderd` logs for `client_session_id` set to the same value.
|
1. Search the `coderd` logs for `client_session_id` set to the same value.
|
||||||
Matching warn-level entries show requests that failed with a `5xx` status.
|
|
||||||
To see `4xx` failures, turn on debug logging.
|
Warn-level entries show `5xx` responses.
|
||||||
|
Debug logs also include `4xx` responses, as described in [`coderd` logs](#coderd-logs).
|
||||||
|
|
||||||
1. Search the agent log for the session ID:
|
1. Search the agent log for the session ID:
|
||||||
|
|
||||||
|
On Linux and macOS:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
grep '<session-id>' <agent-log-dir>/coder-agent*.log
|
grep '<session-id>' <agent-log-dir>/coder-agent*.log
|
||||||
```
|
```
|
||||||
|
|
||||||
|
On Windows workspaces, use PowerShell:
|
||||||
|
|
||||||
|
```ps1
|
||||||
|
Select-String -Path '<agent-log-dir>\coder-agent*.log' -Pattern '<session-id>' -SimpleMatch
|
||||||
|
```
|
||||||
|
|
||||||
The matching lines show each SSH or web terminal session the client opened and the reason it closed.
|
The matching lines show each SSH or web terminal session the client opened and the reason it closed.
|
||||||
|
|
||||||
1. Line up the client log with the server-side entries by time.
|
1. Line up the client log with the server-side entries by time.
|
||||||
The last client entries before a disconnect usually show the error the client hit.
|
|
||||||
|
|
||||||
If the client log has the session ID but there are no `connection_log` entries for it, either the connection didn't reach the agent or the agent is older than your Coder server.
|
Matching timestamps and session IDs connect the client activity to the reported connection events.
|
||||||
Check the `coderd` logs for the session ID, and filter the connection log with `method:tunnel` for the same workspace and time to find denied tunnel requests.
|
|
||||||
|
|
||||||
If you start without a session ID, filter the connection log by `workspace_owner`, `workspace_id`, and time, then take `ClientSessionID` from the matching service log entries.
|
A client session ID doesn't guarantee a corresponding `connection_log` entry.
|
||||||
|
Connections that don't reach the agent and earlier agents that don't record session IDs can leave no matching entries.
|
||||||
|
Reports can also be lost if the agent's reporting buffer fills or a report to the control plane fails.
|
||||||
|
The agent logs `connection report buffer limit reached, dropping report` or `failed to report connection to server` for those reporting failures.
|
||||||
|
|
||||||
|
If you start without a session ID, filter the connection log by `workspace_owner` or `workspace_id`.
|
||||||
|
Use `connected_after` and `connected_before` with RFC3339 timestamps to limit connection start times.
|
||||||
|
Both time boundaries are inclusive.
|
||||||
|
Then take `ClientSessionID` from the corresponding `connection_log` service entries.
|
||||||
|
|
||||||
## Read disconnect reasons
|
## Read disconnect reasons
|
||||||
|
|
||||||
@@ -206,36 +231,40 @@ The two can differ after a workspace stop or delete: the agent logs `server_shut
|
|||||||
`coderd` records `workspace_stopped` only when a stop or delete build that was requested before the connection closed is running or has succeeded.
|
`coderd` records `workspace_stopped` only when a stop or delete build that was requested before the connection closed is running or has succeeded.
|
||||||
Otherwise, the connection log keeps `server_shutdown`.
|
Otherwise, the connection log keeps `server_shutdown`.
|
||||||
|
|
||||||
| `disconnect_reason` | Meaning |
|
| `disconnect_reason` | Meaning |
|
||||||
|---------------------|----------------------------------------------------------------------------------------------------------------------------------------------------|
|
|---------------------|------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||||
| `graceful` | The connection ended cleanly, for example when the SSH command exited with status `0`. |
|
| `graceful` | The connection ended cleanly, for example when the SSH command exited with status `0`. |
|
||||||
| `network_error` | A web terminal connection's transport failed, for example with an EOF, a read or write error, or a timeout. SSH sessions don't record this reason. |
|
| `network_error` | A web terminal connection returned a setup or attachment error, or the agent lost its connection to `coderd`. SSH sessions don't record this reason. |
|
||||||
| `server_shutdown` | The agent closed the connection because the agent was shutting down. |
|
| `server_shutdown` | The agent closed the connection because the agent was shutting down. |
|
||||||
| `workspace_stopped` | A workspace stop or delete closed the connection. Only the connection log and the `coder ssh` log record this reason. |
|
| `workspace_stopped` | A workspace stop or delete closed the connection. Only the connection log and the `coder ssh` log record this reason. |
|
||||||
| `unknown` | The agent couldn't determine why the connection closed. Only the agent log shows `unknown`; in the connection log, `DisconnectReason` is empty. |
|
| `unknown` | The agent couldn't determine why the connection closed. Only the agent log shows `unknown`; in the connection log, `DisconnectReason` is empty. |
|
||||||
| `normal close` | A JetBrains Toolbox forwarded channel closed while the agent was running. Only the connection log records this reason. |
|
| `normal close` | A JetBrains Toolbox forwarded channel closed while the agent was running. Only the connection log records this reason. |
|
||||||
|
|
||||||
|
For web terminal connections, `network_error` includes transport errors and errors creating the terminal command.
|
||||||
|
The agent also records it on `disconnected from coderd`, with `connect_type=server_to_agent`.
|
||||||
|
|
||||||
For SSH sessions, the reason can also be a specific message, such as `process exited with error status: 1` when the command exited with a nonzero status, or the raw error text when the session failed.
|
For SSH sessions, the reason can also be a specific message, such as `process exited with error status: 1` when the command exited with a nonzero status, or the raw error text when the session failed.
|
||||||
SSH sessions don't record `network_error`, so search the agent log for `ssh session failed` to find SSH transport failures.
|
SSH sessions don't record `network_error`.
|
||||||
If the agent was shutting down when an SSH session ended, the reason is `server_shutdown`, and the agent log keeps the specific message in `disconnect_detail`.
|
If the agent was shutting down when an SSH session ended, the reason is `server_shutdown`, and the agent log keeps the specific message in `disconnect_detail`.
|
||||||
|
|
||||||
In the agent log, `disconnect_expected` is `true` for `graceful`, `server_shutdown`, and `workspace_stopped`, and `false` for `network_error`, `unknown`, and error messages.
|
In the agent log, `disconnect_expected` is `true` for `graceful` and `server_shutdown`.
|
||||||
Filter the agent log on `disconnect_expected=false` to find the disconnects worth investigating.
|
It's `false` for `network_error`, `unknown`, and specific error messages, including nonzero remote-command exit statuses.
|
||||||
|
A value of `false` can describe an ordinary command failure, not a network failure.
|
||||||
`connection_log` entries don't include `disconnect_expected`.
|
`connection_log` entries don't include `disconnect_expected`.
|
||||||
|
|
||||||
When a workspace stop or delete ends a `coder ssh` session, the `coder ssh` log has a `workspace is shutting down` line with `disconnect_reason` set to `workspace_stopped`, `build_id`, `build_reason`, and `transition`.
|
When a workspace stop or delete ends a `coder ssh` session, the `coder ssh` log has a `workspace is shutting down` line with `disconnect_reason` set to `workspace_stopped`, `build_id`, `build_reason`, and `transition`.
|
||||||
The agent log doesn't include these build fields.
|
The agent log doesn't include these build fields.
|
||||||
`build_reason` shows what stopped the workspace:
|
`build_reason` shows what stopped the workspace:
|
||||||
|
|
||||||
| `build_reason` | The workspace stopped because |
|
| `build_reason` | The workspace stopped because |
|
||||||
|----------------|--------------------------------------------------------------------------------------|
|
|----------------|--------------------------------------------------------------------------------------------------------------------------|
|
||||||
| `initiator` | A user or API client requested the build. |
|
| `initiator` | A user or API client requested the build. |
|
||||||
| `autostop` | The workspace reached its autostop schedule. |
|
| `autostop` | The workspace reached its autostop deadline, its owner was suspended, or a failed start or stop build triggered cleanup. |
|
||||||
| `dormancy` | The workspace became dormant from inactivity. |
|
| `dormancy` | The workspace became dormant from inactivity. |
|
||||||
| `autodelete` | Coder deleted the dormant workspace automatically, based on the template's schedule. |
|
| `autodelete` | Coder deleted the dormant workspace automatically, based on the template's schedule. |
|
||||||
|
|
||||||
## Next steps
|
## Learn more
|
||||||
|
|
||||||
- [Connection logs](./connection-logs.md): filter, export, and retain connection events.
|
- [Connection logs](./connection-logs.md): filter, export, and retain connection events
|
||||||
- [Generate a support bundle](../../support/support-bundle.md): collect client, agent, and deployment diagnostics in one archive.
|
- [Generate a support bundle](../../support/support-bundle.md): collect client, agent, and deployment diagnostics in one archive
|
||||||
- [Networking troubleshooting](../networking/troubleshooting.md): diagnose direct and relayed connections with `coder ping`.
|
- [Networking troubleshooting](../networking/troubleshooting.md): diagnose direct and relayed connections with `coder ping`
|
||||||
Generated
+1
-1
@@ -1003,7 +1003,7 @@
|
|||||||
},
|
},
|
||||||
{
|
{
|
||||||
"title": "Trace workspace connections",
|
"title": "Trace workspace connections",
|
||||||
"description": "Correlate client, coderd, and agent logs to find why an SSH, IDE, or web terminal connection failed.",
|
"description": "Correlate client, coderd, and agent logs by session ID for SSH, IDE, and web terminal connections.",
|
||||||
"path": "./admin/monitoring/trace-connections.md"
|
"path": "./admin/monitoring/trace-connections.md"
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
|
|||||||
@@ -17,6 +17,6 @@ children:
|
|||||||
path: ./admin/monitoring/connection-logs.md
|
path: ./admin/monitoring/connection-logs.md
|
||||||
state: [premium]
|
state: [premium]
|
||||||
- title: Trace workspace connections
|
- title: Trace workspace connections
|
||||||
description: Correlate client, coderd, and agent logs to find why an SSH, IDE, or web terminal connection failed.
|
description: Correlate client, coderd, and agent logs by session ID for SSH, IDE, and web terminal connections.
|
||||||
path: ./admin/monitoring/trace-connections.md
|
path: ./admin/monitoring/trace-connections.md
|
||||||
- include: admin/monitoring/notifications.yml
|
- include: admin/monitoring/notifications.yml
|
||||||
Reference in new issue
Block a user