## Summary Implements DEVEX-664: the coder CLI commands that open a workspace connection now generate a client session ID and attach it to their requests, logs, and existing network telemetry, per the connection-log RFC. Commands opt in explicitly with a `client_session_id` annotation. The opted-in set is `coder ssh`, `coder start`, `coder stop`, and `coder update`. Long-running daemon commands (`coder server`, `agent`, `provisionerd`, and so on) deliberately stay out, so they never carry a meaningless session ID in their logs, request baggage, or telemetry. - **Generate / resolve**: a 16-byte lowercase-hex session ID is generated per opted-in invocation. When `CODER_TRACE_SESSION_ID` is set (so Toolbox and the VS Code plugin can correlate the CLI they spawn), that value is used verbatim; a warning is logged if it is not the canonical 32-char lowercase hex form, since coderd/agent middleware drop non-canonical values. - **Requests**: the ID is attached to every request as `client_session_id` W3C baggage via a transport wrapper, independent of whether tracing is enabled. The already-merged coderd tracing middleware (#27671) reads it. - **Coordinate websocket**: the ID is also set explicitly as baggage on the tailnet coordinate handshake in `workspacesdk.DialAgent`, so coderd's `/api/v2/workspaceagents/{id}/coordinate` endpoint logs it (it does not rely on the CLI transport for that dial). - **Logs**: the ID is added to the command's log context. - **Existing telemetry**: the ID is added to tailnet `TelemetryEvent` (`client_session_id`, new field 20; proto minor version bumped) and plumbed from the CLI through `workspacesdk.DialAgent`. Scope is the coder CLI client only (RFC glossary + requirement 5), and within the CLI only the opted-in workspace-connecting commands above. This PR is independent of #28039 (the agent middleware that consumes the baggage); it depends only on `tracing.SessionIDBaggageKey`, which is already on `main` via #27671. ## Out of scope RFC requirements 7-16 (connection-state change logging, `connection_logs` columns, log buffering, routing/disconnect-reason logs, coordination-protocol plumbing, `agentssh`/`reconnectingpty` command logging, support-bundle log capture) and requirements 4.2/4.3 (VS Code telemetry, Prometheus stats) are separate tickets. IDE plugins, Toolbox, and the daemon commands are also out of scope. ## Testing - `go test ./coderd/tracing/... ./cli/... ./tailnet/... ./codersdk/workspacesdk/...` (unit tests for the generator, the env/generate resolver, per-command opt-in, the baggage transport wrapper, the coordinate-dial baggage helper, and the telemetry field). - `go build ./...`, `go vet`, and `golangci-lint` pass on the touched packages. - `make tailnet/proto/tailnet.pb.go` regenerated the proto; the proto minor version was bumped and `postStartup` now connects at the current agent API version so `TestWorkspaceAgent_Startup/OK` tracks it. > Note: the pre-commit `enterprise/cli/.gen-golden` step could not run in this environment (no Postgres: `open postgres connection: EOF`). `cli/testdata/.gen-golden` passed; this change adds no serpent flags, so there is no CLI help/golden drift. <details> <summary>Implementation plan</summary> ### Scope mapping (RFC -> DEVEX-664) In scope (the CLI workspace-connecting subset of RFC phase 2): - Req 1: generate a 16-byte session ID as 32-char lowercase hex. - Req 2: add the session ID to every log the command makes for the session. - Req 3: attach the session ID to every API request via `client_session_id` baggage, including the coordinate websocket handshake. (The query-param fallback is browser-only; not applicable to the CLI.) - Req 4.1: attach the session ID to CLI network telemetry. - Req 5 / 5.3: support `CODER_TRACE_SESSION_ID`; when set, use it instead of generating. ### Resolved decisions - Scope: opt-in per command via a `client_session_id` annotation; opted-in set is `coder ssh`, `coder start`, `coder stop`, `coder update`. Daemon commands are excluded. - `CODER_TRACE_SESSION_ID`: when set, use the value verbatim; warn if it is not canonical 32-char lowercase hex. When unset, generate a fresh ID. - Existing telemetry = CLI network telemetry (tailnet `TelemetryEvent`); requires a new proto field + plumbing and a proto minor version bump. - Coordinate endpoint: set the session ID as baggage on the coordinate handshake explicitly in `DialAgent` (header, not query param) so it is logged server-side without relying on the CLI transport. - Shared helper: add `tracing.NewSessionID()` and export the validator as `tracing.ValidSessionID` for a single source of truth. ### Implementation 1. `coderd/tracing`: add `NewSessionID()` (16 random bytes, hex-encoded); export `ValidSessionID`. 2. `cli`: resolve the ID from `CODER_TRACE_SESSION_ID` or generate; attach `client_session_id` baggage to every request via `wrapTransportWithSessionIDHeader`, gated on the `annotationClientSessionID` opt-in; add it to the command log context. 3. `tailnet` + `codersdk/workspacesdk`: add `client_session_id` to `TelemetryEvent`, an `Options.ClientSessionID`/`Conn.clientSessionID` set in `newTelemetryEvent`, a `DialAgentOptions.ClientSessionID` plumbed from the opted-in commands, and the coordinate-handshake baggage in `DialAgent`. </details> --- _This PR was created by Coder Agents on behalf of @aqandrew._
237 lines
8.4 KiB
Go
237 lines
8.4 KiB
Go
package tracing
|
|
|
|
import (
|
|
"context"
|
|
cryptorand "crypto/rand"
|
|
"encoding/hex"
|
|
"fmt"
|
|
"net/http"
|
|
"net/url"
|
|
|
|
"github.com/go-chi/chi/v5"
|
|
"go.opentelemetry.io/otel"
|
|
"go.opentelemetry.io/otel/attribute"
|
|
"go.opentelemetry.io/otel/baggage"
|
|
"go.opentelemetry.io/otel/propagation"
|
|
semconv "go.opentelemetry.io/otel/semconv/v1.14.0"
|
|
"go.opentelemetry.io/otel/semconv/v1.14.0/httpconv"
|
|
"go.opentelemetry.io/otel/semconv/v1.14.0/netconv"
|
|
"go.opentelemetry.io/otel/trace"
|
|
"go.opentelemetry.io/otel/trace/noop"
|
|
|
|
"cdr.dev/slog/v3"
|
|
"github.com/coder/coder/v2/coderd/httpmw/patternmatcher"
|
|
)
|
|
|
|
// SessionIDBaggageKey is the W3C baggage key clients use to propagate their
|
|
// per-session correlation ID. The value is a 16-byte identifier encoded as a
|
|
// 32-character hexadecimal string.
|
|
const SessionIDBaggageKey = "client_session_id"
|
|
|
|
// DefaultRoutePatterns are the route patterns coderd and wsproxy trace. The
|
|
// middleware runs high in the stack so it can capture the entire request, but
|
|
// only acts on these patterns.
|
|
var DefaultRoutePatterns = []string{
|
|
"/api",
|
|
"/api/**",
|
|
"/@*/*/apps/**",
|
|
"/%40*/*/apps/**",
|
|
"/external-auth/*/callback",
|
|
}
|
|
|
|
// Middleware adds tracing to http routes. It only acts on requests matching
|
|
// routePatterns, and labels emitted spans with serverName. A nil tracerProvider
|
|
// disables span creation, leaving the middleware to enrich the log context with
|
|
// client_session_id only (as the agent uses it).
|
|
func Middleware(
|
|
tracerProvider trace.TracerProvider,
|
|
routePatterns []string,
|
|
serverName string,
|
|
) func(http.Handler) http.Handler {
|
|
re := patternmatcher.RoutePatterns(routePatterns).MustCompile()
|
|
|
|
if tracerProvider == nil {
|
|
tracerProvider = noop.NewTracerProvider()
|
|
}
|
|
tracer := tracerProvider.Tracer(TracerName)
|
|
|
|
return func(next http.Handler) http.Handler {
|
|
return http.HandlerFunc(func(rw http.ResponseWriter, r *http.Request) {
|
|
if !re.MatchString(r.URL.Path) {
|
|
next.ServeHTTP(rw, r)
|
|
return
|
|
}
|
|
|
|
// Read the client_session_id from the request and add it to the log
|
|
// context. This is done even when tracing is disabled so that logs can
|
|
// always be correlated by client_session_id.
|
|
sessionID := sessionIDFromRequest(r)
|
|
if sessionID != "" {
|
|
r = r.WithContext(slog.With(r.Context(), slog.F("client_session_id", sessionID)))
|
|
}
|
|
|
|
// Start span with default span name. Span name will be updated to
|
|
// "method route" format once request finishes. The initial name
|
|
// excludes the query string because span names are exported to
|
|
// tracing backends at span start and some endpoints accept
|
|
// bearer credentials as query parameters.
|
|
r, span := StartHTTPSpan(tracer, rw, r, fmt.Sprintf("%s %s", r.Method, r.URL.Path))
|
|
defer span.End()
|
|
|
|
if sessionID != "" {
|
|
span.SetAttributes(attribute.String("client_session_id", sessionID))
|
|
}
|
|
|
|
sw, ok := rw.(*StatusWriter)
|
|
if !ok {
|
|
panic(fmt.Sprintf("ResponseWriter not a *tracing.StatusWriter; got %T", rw))
|
|
}
|
|
|
|
// pass the span through the request context and serve the request to the next middleware
|
|
next.ServeHTTP(sw, r)
|
|
// capture response data
|
|
EndHTTPSpan(r, sw.Status, span, serverName)
|
|
})
|
|
}
|
|
}
|
|
|
|
// sessionIDFromRequest extracts and validates the client_session_id from the
|
|
// request. It prefers the W3C baggage member and falls back to the
|
|
// client_session_id query parameter. The query parameter fallback exists
|
|
// because browser WebSocket clients (such as the web terminal PTY) cannot set
|
|
// arbitrary baggage headers. It returns an empty string when neither source
|
|
// provides a valid value.
|
|
func sessionIDFromRequest(r *http.Request) string {
|
|
if id := sessionIDFromHeaders(r.Header); id != "" {
|
|
return id
|
|
}
|
|
return sessionIDFromQueryString(r.URL.Query())
|
|
}
|
|
|
|
// sessionIDFromHeaders extracts and validates the client_session_id baggage member
|
|
// from the request headers. It returns an empty string when the member is
|
|
// absent or malformed. Extraction uses an explicit baggage propagator so it
|
|
// does not depend on the globally configured text map propagator.
|
|
func sessionIDFromHeaders(h http.Header) string {
|
|
ctx := propagation.Baggage{}.Extract(context.Background(), propagation.HeaderCarrier(h))
|
|
id := baggage.FromContext(ctx).Member(SessionIDBaggageKey).Value()
|
|
if !ValidSessionID(id) {
|
|
return ""
|
|
}
|
|
return id
|
|
}
|
|
|
|
// sessionIDFromQueryString extracts and validates the client_session_id query
|
|
// parameter. It returns an empty string when the parameter is absent or
|
|
// malformed. It mirrors sessionIDFromHeaders for the query-parameter fallback
|
|
// used by browser WebSocket clients (such as the web terminal PTY) that cannot
|
|
// set arbitrary baggage headers.
|
|
func sessionIDFromQueryString(q url.Values) string {
|
|
id := q.Get(SessionIDBaggageKey)
|
|
if !ValidSessionID(id) {
|
|
return ""
|
|
}
|
|
return id
|
|
}
|
|
|
|
// NewSessionID generates a new client session ID: 16 random bytes encoded as a
|
|
// 32-character lowercase hexadecimal string, to match OpenTelemetry's identifiers.
|
|
func NewSessionID() (string, error) {
|
|
var b [16]byte
|
|
if _, err := cryptorand.Read(b[:]); err != nil {
|
|
return "", err
|
|
}
|
|
return hex.EncodeToString(b[:]), nil
|
|
}
|
|
|
|
// ValidSessionID reports whether s is a 32-character lowercase hexadecimal
|
|
// string (a 16-byte value).
|
|
// Only lowercase is accepted so that case-sensitive searches correlate
|
|
// reliably. Validating also guards against logging arbitrary client-controlled
|
|
// baggage values.
|
|
func ValidSessionID(s string) bool {
|
|
b, err := hex.DecodeString(s)
|
|
if err != nil || len(b) != 16 {
|
|
return false
|
|
}
|
|
// hex.DecodeString also accepts upper-case, so require the canonical
|
|
// lowercase encoding.
|
|
return hex.EncodeToString(b) == s
|
|
}
|
|
|
|
// StartHTTPSpan starts a span, propagating inbound trace context and writing
|
|
// X-Trace-ID/X-Span-ID response headers. The caller must end the span.
|
|
func StartHTTPSpan(tracer trace.Tracer, rw http.ResponseWriter, r *http.Request, name string) (*http.Request, trace.Span) {
|
|
propagator := otel.GetTextMapPropagator()
|
|
ctx := propagator.Extract(r.Context(), propagation.HeaderCarrier(r.Header))
|
|
|
|
ctx, span := tracer.Start(ctx, name)
|
|
r = r.WithContext(ctx)
|
|
|
|
if span.SpanContext().HasTraceID() && span.SpanContext().HasSpanID() {
|
|
// Technically these values are included in the Traceparent header, but
|
|
// they are easier to read for humans this way.
|
|
rw.Header().Set("X-Trace-ID", span.SpanContext().TraceID().String())
|
|
rw.Header().Set("X-Span-ID", span.SpanContext().SpanID().String())
|
|
propagator.Inject(ctx, propagation.HeaderCarrier(rw.Header()))
|
|
}
|
|
|
|
return r, span
|
|
}
|
|
|
|
// EndHTTPSpan captures request and response data after the handler is done.
|
|
// serverName labels the span's server request attributes.
|
|
func EndHTTPSpan(r *http.Request, status int, span trace.Span, serverName string) {
|
|
// set the resource name as we get it only once the handler is executed
|
|
route := chi.RouteContext(r.Context()).RoutePattern()
|
|
span.SetName(fmt.Sprintf("%s %s", r.Method, route))
|
|
span.SetAttributes(netconv.Transport("tcp"))
|
|
span.SetAttributes(httpconv.ServerRequest(serverName, r)...)
|
|
span.SetAttributes(semconv.HTTPRouteKey.String(route))
|
|
|
|
// 0 status means one has not yet been sent in which case net/http library will write StatusOK
|
|
if status == 0 {
|
|
status = http.StatusOK
|
|
}
|
|
span.SetAttributes(semconv.HTTPStatusCodeKey.Int(status))
|
|
span.SetStatus(httpconv.ServerStatus(status))
|
|
|
|
// finally end span
|
|
span.End()
|
|
}
|
|
|
|
type tracerNameKey struct{}
|
|
|
|
// SetTracerName sets the tracer name that will be used by all spans created
|
|
// from the context.
|
|
func SetTracerName(ctx context.Context, tracerName string) context.Context {
|
|
return context.WithValue(ctx, tracerNameKey{}, tracerName)
|
|
}
|
|
|
|
// GetTracerName returns the tracer name from the context, or TracerName if none
|
|
// is set.
|
|
func GetTracerName(ctx context.Context) string {
|
|
if tracerName, ok := ctx.Value(tracerNameKey{}).(string); ok {
|
|
return tracerName
|
|
}
|
|
|
|
return TracerName
|
|
}
|
|
|
|
// StartSpan calls StartSpanWithName with the name set to the caller's function
|
|
// name.
|
|
func StartSpan(ctx context.Context, opts ...trace.SpanStartOption) (context.Context, trace.Span) {
|
|
return StartSpanWithName(ctx, FuncNameSkip(1), opts...)
|
|
}
|
|
|
|
// StartSpanWithName starts a new span with the given name from the context. If
|
|
// a tracer name was set on the context (or one of its parents), it will be used
|
|
// as the tracer name instead of the default TracerName.
|
|
func StartSpanWithName(ctx context.Context, name string, opts ...trace.SpanStartOption) (context.Context, trace.Span) {
|
|
tracerName := GetTracerName(ctx)
|
|
return trace.SpanFromContext(ctx).
|
|
TracerProvider().
|
|
Tracer(tracerName).
|
|
Start(ctx, name, opts...)
|
|
}
|