Fixes the `TestServer_Metrics/Forwarding` flake reported in [PLAT-609](https://linear.app/codercom/issue/PLAT-609/flake-testserver-metricsforwarding) and, in addition, other defects similar to it. ## Testing defects ### The Original `TestServer_Metrics/Forwarding` waited for `batches_dropped_total` and then asserted on `logs_dropped_total`. The forwarder updates those two counters separately, on adjacent lines, so a read between them sees the first updated and the second not. CI caught exactly that window: expected 2, actual 0 at `proxy_test.go:780`. Each counter is atomic on its own, so the race detector reports nothing. ### The Additional The fallacy of the original is a _de facto_ assumption that sequential updates are atomic as a group. The defect arose from not taking into account that they are atomic individually. The test asserted before the work was completed. A repo-wide sweep for related metrics written together found similar mistakes in other tests: - Live races: all three phases of `TestServer_Metrics/Forwarding` and `TestServer_Metrics/BufferFull` in `agent/boundarylogproxy`, and `TestMetadataBatcher_DropsWhenFull`, which waited for the update channel to empty although the final flush runs after that. - Correct only because of the current implementation's ordering: `TestMetrics_TokenUseCount` in `aibridge`, four tests in `coderd/agentapi/metadatabatcher`, and `TestAgentStatsSessionCounts` in `coderd/prometheusmetrics`. These pass today but would start failing if the code under test reordered its work. ## Fixes All tests the sweep found were fixed. Each test now waits for every value it asserts before asserting it. Where a value is already non-zero from earlier work, or legitimately zero, the wait is for the final value rather than for a change, and several tests use `require.EventuallyWithT` so the wait and the assertion are one step. Two Forwarding phases gain assertions on values they already waited for: `batches_dropped_total == 1` in phase 1, and the reporter's recorded request count in phase 2. Two phases of `TestAgentStatsSessionCounts` assert that the counts did not change. No wait on an unchanged value can show that the collection cycle ran, so they keep waiting on the execution histogram, and a comment at that wait names the dependency. Each fix was checked by temporarily blocking or reordering the code under test between the relevant writes: the old test failed on every run and the new one passed. ## Documentation As a guard against recurrence of defects of this kind, we add the following documentation targeted mostly at agents. The material in `TESTING.md` provides guidance about how to write such a test correctly in the first place. That in `AGENT_FAILURES.md` describes how to recognize the symptoms of this class of test failure, should they arise. - `.claude/docs/TESTING.md`: new section "Wait for Full Completion before Asserting", covering waits on one value with assertions on another, Prometheus metrics in particular, and assertions that nothing changed. - `.claude/docs/AGENT_FAILURES.md`: a catalog entry for the symptom. - `coderd/coderdtest/promhelp`: doc comments noting that each helper call is a separate gather. Generated with Claude Code under the direction of @eric-hughes-coder.
94 lines
3.3 KiB
Go
94 lines
3.3 KiB
Go
package promhelp
|
|
|
|
import (
|
|
"context"
|
|
"io"
|
|
"maps"
|
|
"net/http"
|
|
"net/http/httptest"
|
|
"strings"
|
|
"testing"
|
|
|
|
"github.com/prometheus/client_golang/prometheus"
|
|
"github.com/prometheus/client_golang/prometheus/promhttp"
|
|
ptestutil "github.com/prometheus/client_golang/prometheus/testutil"
|
|
io_prometheus_client "github.com/prometheus/client_model/go"
|
|
"github.com/stretchr/testify/require"
|
|
)
|
|
|
|
// RegistryDump returns the http page for a given registry's metrics.
|
|
// Very useful for visual debugging.
|
|
func RegistryDump(reg *prometheus.Registry) string {
|
|
h := promhttp.HandlerFor(reg, promhttp.HandlerOpts{})
|
|
rec := httptest.NewRecorder()
|
|
req, _ := http.NewRequestWithContext(context.Background(), http.MethodGet, "/", nil)
|
|
h.ServeHTTP(rec, req)
|
|
resp := rec.Result()
|
|
data, _ := io.ReadAll(resp.Body)
|
|
_ = resp.Body.Close()
|
|
return string(data)
|
|
}
|
|
|
|
// Compare can be used to compare a registry to some prometheus formatted
|
|
// text. If any values differ, an error is returned.
|
|
// If metric names are passed in, only those metrics will be compared.
|
|
// Usage: `Compare(reg, RegistryDump(reg))`
|
|
func Compare(reg prometheus.Gatherer, compare string, metricNames ...string) error {
|
|
return ptestutil.GatherAndCompare(reg, strings.NewReader(compare), metricNames...)
|
|
}
|
|
|
|
// HistogramValue returns the value of a histogram metric with the given name
|
|
// and labels. Each call gathers separately; see [MetricValue].
|
|
func HistogramValue(t testing.TB, reg prometheus.Gatherer, metricName string, labels prometheus.Labels) *io_prometheus_client.Histogram {
|
|
t.Helper()
|
|
|
|
labeled := MetricValue(t, reg, metricName, labels)
|
|
require.NotNilf(t, labeled, "metric %q with labels %v not found", metricName, labels)
|
|
return labeled.GetHistogram()
|
|
}
|
|
|
|
// GaugeValue returns the value of a gauge metric with the given name and
|
|
// labels. Each call gathers separately; see [MetricValue].
|
|
func GaugeValue(t testing.TB, reg prometheus.Gatherer, metricName string, labels prometheus.Labels) int {
|
|
t.Helper()
|
|
|
|
labeled := MetricValue(t, reg, metricName, labels)
|
|
require.NotNilf(t, labeled, "metric %q with labels %v not found", metricName, labels)
|
|
return int(labeled.GetGauge().GetValue())
|
|
}
|
|
|
|
// CounterValue returns the value of a counter metric with the given name and
|
|
// labels. Each call gathers separately; see [MetricValue].
|
|
func CounterValue(t testing.TB, reg prometheus.Gatherer, metricName string, labels prometheus.Labels) int {
|
|
t.Helper()
|
|
|
|
labeled := MetricValue(t, reg, metricName, labels)
|
|
require.NotNilf(t, labeled, "metric %q with labels %v not found", metricName, labels)
|
|
return int(labeled.GetCounter().GetValue())
|
|
}
|
|
|
|
// MetricValue returns the metric with the given name and labels, or nil if
|
|
// it is not found. Each call gathers `reg` separately, as does each call to
|
|
// testutil.ToFloat64; see the package documentation.
|
|
func MetricValue(t testing.TB, reg prometheus.Gatherer, metricName string, labels prometheus.Labels) *io_prometheus_client.Metric {
|
|
t.Helper()
|
|
|
|
metrics, err := reg.Gather()
|
|
require.NoError(t, err)
|
|
|
|
for _, m := range metrics {
|
|
if m.GetName() == metricName {
|
|
for _, labeled := range m.GetMetric() {
|
|
mLabels := make(prometheus.Labels)
|
|
for _, v := range labeled.GetLabel() {
|
|
mLabels[v.GetName()] = v.GetValue()
|
|
}
|
|
if maps.Equal(mLabels, labels) {
|
|
return labeled
|
|
}
|
|
}
|
|
}
|
|
}
|
|
return nil
|
|
}
|