Files
coder/coderd/tracing/httpmw.go
T
Andrew Aquino 3aeb7f85d9 feat: correlate coder CLI workspace sessions with client_session_id (#28313)
## Summary

Implements DEVEX-664: the coder CLI commands that open a workspace
connection now generate a client session ID and attach it to their
requests, logs, and existing network telemetry, per the connection-log
RFC.

Commands opt in explicitly with a `client_session_id` annotation. The
opted-in set is `coder ssh`, `coder start`, `coder stop`, and `coder
update`. Long-running daemon commands (`coder server`, `agent`,
`provisionerd`, and so on) deliberately stay out, so they never carry a
meaningless session ID in their logs, request baggage, or telemetry.

- **Generate / resolve**: a 16-byte lowercase-hex session ID is
generated per opted-in invocation. When `CODER_TRACE_SESSION_ID` is set
(so Toolbox and the VS Code plugin can correlate the CLI they spawn),
that value is used verbatim; a warning is logged if it is not the
canonical 32-char lowercase hex form, since coderd/agent middleware drop
non-canonical values.
- **Requests**: the ID is attached to every request as
`client_session_id` W3C baggage via a transport wrapper, independent of
whether tracing is enabled. The already-merged coderd tracing middleware
(#27671) reads it.
- **Coordinate websocket**: the ID is also set explicitly as baggage on
the tailnet coordinate handshake in `workspacesdk.DialAgent`, so
coderd's `/api/v2/workspaceagents/{id}/coordinate` endpoint logs it (it
does not rely on the CLI transport for that dial).
- **Logs**: the ID is added to the command's log context.
- **Existing telemetry**: the ID is added to tailnet `TelemetryEvent`
(`client_session_id`, new field 20; proto minor version bumped) and
plumbed from the CLI through `workspacesdk.DialAgent`.

Scope is the coder CLI client only (RFC glossary + requirement 5), and
within the CLI only the opted-in workspace-connecting commands above.
This PR is independent of #28039 (the agent middleware that consumes the
baggage); it depends only on `tracing.SessionIDBaggageKey`, which is
already on `main` via #27671.

## Out of scope

RFC requirements 7-16 (connection-state change logging,
`connection_logs` columns, log buffering, routing/disconnect-reason
logs, coordination-protocol plumbing, `agentssh`/`reconnectingpty`
command logging, support-bundle log capture) and requirements 4.2/4.3
(VS Code telemetry, Prometheus stats) are separate tickets. IDE plugins,
Toolbox, and the daemon commands are also out of scope.

## Testing

- `go test ./coderd/tracing/... ./cli/... ./tailnet/...
./codersdk/workspacesdk/...` (unit tests for the generator, the
env/generate resolver, per-command opt-in, the baggage transport
wrapper, the coordinate-dial baggage helper, and the telemetry field).
- `go build ./...`, `go vet`, and `golangci-lint` pass on the touched
packages.
- `make tailnet/proto/tailnet.pb.go` regenerated the proto; the proto
minor version was bumped and `postStartup` now connects at the current
agent API version so `TestWorkspaceAgent_Startup/OK` tracks it.

> Note: the pre-commit `enterprise/cli/.gen-golden` step could not run
in this environment (no Postgres: `open postgres connection: EOF`).
`cli/testdata/.gen-golden` passed; this change adds no serpent flags, so
there is no CLI help/golden drift.

<details>
<summary>Implementation plan</summary>

### Scope mapping (RFC -> DEVEX-664)

In scope (the CLI workspace-connecting subset of RFC phase 2):
- Req 1: generate a 16-byte session ID as 32-char lowercase hex.
- Req 2: add the session ID to every log the command makes for the
session.
- Req 3: attach the session ID to every API request via
`client_session_id` baggage, including the coordinate websocket
handshake. (The query-param fallback is browser-only; not applicable to
the CLI.)
- Req 4.1: attach the session ID to CLI network telemetry.
- Req 5 / 5.3: support `CODER_TRACE_SESSION_ID`; when set, use it
instead of generating.

### Resolved decisions

- Scope: opt-in per command via a `client_session_id` annotation;
opted-in set is `coder ssh`, `coder start`, `coder stop`, `coder
update`. Daemon commands are excluded.
- `CODER_TRACE_SESSION_ID`: when set, use the value verbatim; warn if it
is not canonical 32-char lowercase hex. When unset, generate a fresh ID.
- Existing telemetry = CLI network telemetry (tailnet `TelemetryEvent`);
requires a new proto field + plumbing and a proto minor version bump.
- Coordinate endpoint: set the session ID as baggage on the coordinate
handshake explicitly in `DialAgent` (header, not query param) so it is
logged server-side without relying on the CLI transport.
- Shared helper: add `tracing.NewSessionID()` and export the validator
as `tracing.ValidSessionID` for a single source of truth.

### Implementation

1. `coderd/tracing`: add `NewSessionID()` (16 random bytes,
hex-encoded); export `ValidSessionID`.
2. `cli`: resolve the ID from `CODER_TRACE_SESSION_ID` or generate;
attach `client_session_id` baggage to every request via
`wrapTransportWithSessionIDHeader`, gated on the
`annotationClientSessionID` opt-in; add it to the command log context.
3. `tailnet` + `codersdk/workspacesdk`: add `client_session_id` to
`TelemetryEvent`, an `Options.ClientSessionID`/`Conn.clientSessionID`
set in `newTelemetryEvent`, a `DialAgentOptions.ClientSessionID` plumbed
from the opted-in commands, and the coordinate-handshake baggage in
`DialAgent`.

</details>

---

_This PR was created by Coder Agents on behalf of @aqandrew._
2026-09-02 19:31:01 +00:00

237 lines
8.4 KiB
Go

package tracing
import (
"context"
cryptorand "crypto/rand"
"encoding/hex"
"fmt"
"net/http"
"net/url"
"github.com/go-chi/chi/v5"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/baggage"
"go.opentelemetry.io/otel/propagation"
semconv "go.opentelemetry.io/otel/semconv/v1.14.0"
"go.opentelemetry.io/otel/semconv/v1.14.0/httpconv"
"go.opentelemetry.io/otel/semconv/v1.14.0/netconv"
"go.opentelemetry.io/otel/trace"
"go.opentelemetry.io/otel/trace/noop"
"cdr.dev/slog/v3"
"github.com/coder/coder/v2/coderd/httpmw/patternmatcher"
)
// SessionIDBaggageKey is the W3C baggage key clients use to propagate their
// per-session correlation ID. The value is a 16-byte identifier encoded as a
// 32-character hexadecimal string.
const SessionIDBaggageKey = "client_session_id"
// DefaultRoutePatterns are the route patterns coderd and wsproxy trace. The
// middleware runs high in the stack so it can capture the entire request, but
// only acts on these patterns.
var DefaultRoutePatterns = []string{
"/api",
"/api/**",
"/@*/*/apps/**",
"/%40*/*/apps/**",
"/external-auth/*/callback",
}
// Middleware adds tracing to http routes. It only acts on requests matching
// routePatterns, and labels emitted spans with serverName. A nil tracerProvider
// disables span creation, leaving the middleware to enrich the log context with
// client_session_id only (as the agent uses it).
func Middleware(
tracerProvider trace.TracerProvider,
routePatterns []string,
serverName string,
) func(http.Handler) http.Handler {
re := patternmatcher.RoutePatterns(routePatterns).MustCompile()
if tracerProvider == nil {
tracerProvider = noop.NewTracerProvider()
}
tracer := tracerProvider.Tracer(TracerName)
return func(next http.Handler) http.Handler {
return http.HandlerFunc(func(rw http.ResponseWriter, r *http.Request) {
if !re.MatchString(r.URL.Path) {
next.ServeHTTP(rw, r)
return
}
// Read the client_session_id from the request and add it to the log
// context. This is done even when tracing is disabled so that logs can
// always be correlated by client_session_id.
sessionID := sessionIDFromRequest(r)
if sessionID != "" {
r = r.WithContext(slog.With(r.Context(), slog.F("client_session_id", sessionID)))
}
// Start span with default span name. Span name will be updated to
// "method route" format once request finishes. The initial name
// excludes the query string because span names are exported to
// tracing backends at span start and some endpoints accept
// bearer credentials as query parameters.
r, span := StartHTTPSpan(tracer, rw, r, fmt.Sprintf("%s %s", r.Method, r.URL.Path))
defer span.End()
if sessionID != "" {
span.SetAttributes(attribute.String("client_session_id", sessionID))
}
sw, ok := rw.(*StatusWriter)
if !ok {
panic(fmt.Sprintf("ResponseWriter not a *tracing.StatusWriter; got %T", rw))
}
// pass the span through the request context and serve the request to the next middleware
next.ServeHTTP(sw, r)
// capture response data
EndHTTPSpan(r, sw.Status, span, serverName)
})
}
}
// sessionIDFromRequest extracts and validates the client_session_id from the
// request. It prefers the W3C baggage member and falls back to the
// client_session_id query parameter. The query parameter fallback exists
// because browser WebSocket clients (such as the web terminal PTY) cannot set
// arbitrary baggage headers. It returns an empty string when neither source
// provides a valid value.
func sessionIDFromRequest(r *http.Request) string {
if id := sessionIDFromHeaders(r.Header); id != "" {
return id
}
return sessionIDFromQueryString(r.URL.Query())
}
// sessionIDFromHeaders extracts and validates the client_session_id baggage member
// from the request headers. It returns an empty string when the member is
// absent or malformed. Extraction uses an explicit baggage propagator so it
// does not depend on the globally configured text map propagator.
func sessionIDFromHeaders(h http.Header) string {
ctx := propagation.Baggage{}.Extract(context.Background(), propagation.HeaderCarrier(h))
id := baggage.FromContext(ctx).Member(SessionIDBaggageKey).Value()
if !ValidSessionID(id) {
return ""
}
return id
}
// sessionIDFromQueryString extracts and validates the client_session_id query
// parameter. It returns an empty string when the parameter is absent or
// malformed. It mirrors sessionIDFromHeaders for the query-parameter fallback
// used by browser WebSocket clients (such as the web terminal PTY) that cannot
// set arbitrary baggage headers.
func sessionIDFromQueryString(q url.Values) string {
id := q.Get(SessionIDBaggageKey)
if !ValidSessionID(id) {
return ""
}
return id
}
// NewSessionID generates a new client session ID: 16 random bytes encoded as a
// 32-character lowercase hexadecimal string, to match OpenTelemetry's identifiers.
func NewSessionID() (string, error) {
var b [16]byte
if _, err := cryptorand.Read(b[:]); err != nil {
return "", err
}
return hex.EncodeToString(b[:]), nil
}
// ValidSessionID reports whether s is a 32-character lowercase hexadecimal
// string (a 16-byte value).
// Only lowercase is accepted so that case-sensitive searches correlate
// reliably. Validating also guards against logging arbitrary client-controlled
// baggage values.
func ValidSessionID(s string) bool {
b, err := hex.DecodeString(s)
if err != nil || len(b) != 16 {
return false
}
// hex.DecodeString also accepts upper-case, so require the canonical
// lowercase encoding.
return hex.EncodeToString(b) == s
}
// StartHTTPSpan starts a span, propagating inbound trace context and writing
// X-Trace-ID/X-Span-ID response headers. The caller must end the span.
func StartHTTPSpan(tracer trace.Tracer, rw http.ResponseWriter, r *http.Request, name string) (*http.Request, trace.Span) {
propagator := otel.GetTextMapPropagator()
ctx := propagator.Extract(r.Context(), propagation.HeaderCarrier(r.Header))
ctx, span := tracer.Start(ctx, name)
r = r.WithContext(ctx)
if span.SpanContext().HasTraceID() && span.SpanContext().HasSpanID() {
// Technically these values are included in the Traceparent header, but
// they are easier to read for humans this way.
rw.Header().Set("X-Trace-ID", span.SpanContext().TraceID().String())
rw.Header().Set("X-Span-ID", span.SpanContext().SpanID().String())
propagator.Inject(ctx, propagation.HeaderCarrier(rw.Header()))
}
return r, span
}
// EndHTTPSpan captures request and response data after the handler is done.
// serverName labels the span's server request attributes.
func EndHTTPSpan(r *http.Request, status int, span trace.Span, serverName string) {
// set the resource name as we get it only once the handler is executed
route := chi.RouteContext(r.Context()).RoutePattern()
span.SetName(fmt.Sprintf("%s %s", r.Method, route))
span.SetAttributes(netconv.Transport("tcp"))
span.SetAttributes(httpconv.ServerRequest(serverName, r)...)
span.SetAttributes(semconv.HTTPRouteKey.String(route))
// 0 status means one has not yet been sent in which case net/http library will write StatusOK
if status == 0 {
status = http.StatusOK
}
span.SetAttributes(semconv.HTTPStatusCodeKey.Int(status))
span.SetStatus(httpconv.ServerStatus(status))
// finally end span
span.End()
}
type tracerNameKey struct{}
// SetTracerName sets the tracer name that will be used by all spans created
// from the context.
func SetTracerName(ctx context.Context, tracerName string) context.Context {
return context.WithValue(ctx, tracerNameKey{}, tracerName)
}
// GetTracerName returns the tracer name from the context, or TracerName if none
// is set.
func GetTracerName(ctx context.Context) string {
if tracerName, ok := ctx.Value(tracerNameKey{}).(string); ok {
return tracerName
}
return TracerName
}
// StartSpan calls StartSpanWithName with the name set to the caller's function
// name.
func StartSpan(ctx context.Context, opts ...trace.SpanStartOption) (context.Context, trace.Span) {
return StartSpanWithName(ctx, FuncNameSkip(1), opts...)
}
// StartSpanWithName starts a new span with the given name from the context. If
// a tracer name was set on the context (or one of its parents), it will be used
// as the tracer name instead of the default TracerName.
func StartSpanWithName(ctx context.Context, name string, opts ...trace.SpanStartOption) (context.Context, trace.Span) {
tracerName := GetTracerName(ctx)
return trace.SpanFromContext(ctx).
TracerProvider().
Tracer(tracerName).
Start(ctx, name, opts...)
}