Files
eric-hughes-coder bb8912dc46 fix: wait for full completion before asserting in tests (#30885)
Fixes the `TestServer_Metrics/Forwarding` flake reported in
[PLAT-609](https://linear.app/codercom/issue/PLAT-609/flake-testserver-metricsforwarding)
and, in addition, other defects similar to it.

## Testing defects

### The Original

`TestServer_Metrics/Forwarding` waited for `batches_dropped_total` and
then asserted on `logs_dropped_total`. The forwarder updates those two
counters separately, on adjacent lines, so a read between them sees the
first updated and the second not. CI caught exactly that window:
expected 2, actual 0 at `proxy_test.go:780`. Each counter is atomic on
its own, so the race detector reports nothing.

### The Additional

The fallacy of the original is a _de facto_ assumption that sequential
updates are atomic as a group. The defect arose from not taking into
account that they are atomic individually. The test asserted before the
work was completed. A repo-wide sweep for related metrics written
together found similar mistakes in other tests:

- Live races: all three phases of `TestServer_Metrics/Forwarding` and
`TestServer_Metrics/BufferFull` in `agent/boundarylogproxy`, and
`TestMetadataBatcher_DropsWhenFull`, which waited for the update channel
to empty although the final flush runs after that.
- Correct only because of the current implementation's ordering:
`TestMetrics_TokenUseCount` in `aibridge`, four tests in
`coderd/agentapi/metadatabatcher`, and `TestAgentStatsSessionCounts` in
`coderd/prometheusmetrics`. These pass today but would start failing if
the code under test reordered its work.

## Fixes

All tests the sweep found were fixed. Each test now waits for every
value it asserts before asserting it. Where a value is already non-zero
from earlier work, or legitimately zero, the wait is for the final value
rather than for a change, and several tests use
`require.EventuallyWithT` so the wait and the assertion are one step.
Two Forwarding phases gain assertions on values they already waited for:
`batches_dropped_total == 1` in phase 1, and the reporter's recorded
request count in phase 2.

Two phases of `TestAgentStatsSessionCounts` assert that the counts did
not change. No wait on an unchanged value can show that the collection
cycle ran, so they keep waiting on the execution histogram, and a
comment at that wait names the dependency.

Each fix was checked by temporarily blocking or reordering the code
under test between the relevant writes: the old test failed on every run
and the new one passed.

## Documentation

As a guard against recurrence of defects of this kind, we add the
following documentation targeted mostly at agents. The material in
`TESTING.md` provides guidance about how to write such a test correctly
in the first place. That in `AGENT_FAILURES.md` describes how to
recognize the symptoms of this class of test failure, should they arise.

- `.claude/docs/TESTING.md`: new section "Wait for Full Completion
before Asserting", covering waits on one value with assertions on
another, Prometheus metrics in particular, and assertions that nothing
changed.
- `.claude/docs/AGENT_FAILURES.md`: a catalog entry for the symptom.
- `coderd/coderdtest/promhelp`: doc comments noting that each helper
call is a separate gather.

Generated with Claude Code under the direction of @eric-hughes-coder.
2026-10-28 09:17:10 -06:00

94 lines
3.3 KiB
Go

package promhelp
import (
"context"
"io"
"maps"
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
ptestutil "github.com/prometheus/client_golang/prometheus/testutil"
io_prometheus_client "github.com/prometheus/client_model/go"
"github.com/stretchr/testify/require"
)
// RegistryDump returns the http page for a given registry's metrics.
// Very useful for visual debugging.
func RegistryDump(reg *prometheus.Registry) string {
h := promhttp.HandlerFor(reg, promhttp.HandlerOpts{})
rec := httptest.NewRecorder()
req, _ := http.NewRequestWithContext(context.Background(), http.MethodGet, "/", nil)
h.ServeHTTP(rec, req)
resp := rec.Result()
data, _ := io.ReadAll(resp.Body)
_ = resp.Body.Close()
return string(data)
}
// Compare can be used to compare a registry to some prometheus formatted
// text. If any values differ, an error is returned.
// If metric names are passed in, only those metrics will be compared.
// Usage: `Compare(reg, RegistryDump(reg))`
func Compare(reg prometheus.Gatherer, compare string, metricNames ...string) error {
return ptestutil.GatherAndCompare(reg, strings.NewReader(compare), metricNames...)
}
// HistogramValue returns the value of a histogram metric with the given name
// and labels. Each call gathers separately; see [MetricValue].
func HistogramValue(t testing.TB, reg prometheus.Gatherer, metricName string, labels prometheus.Labels) *io_prometheus_client.Histogram {
t.Helper()
labeled := MetricValue(t, reg, metricName, labels)
require.NotNilf(t, labeled, "metric %q with labels %v not found", metricName, labels)
return labeled.GetHistogram()
}
// GaugeValue returns the value of a gauge metric with the given name and
// labels. Each call gathers separately; see [MetricValue].
func GaugeValue(t testing.TB, reg prometheus.Gatherer, metricName string, labels prometheus.Labels) int {
t.Helper()
labeled := MetricValue(t, reg, metricName, labels)
require.NotNilf(t, labeled, "metric %q with labels %v not found", metricName, labels)
return int(labeled.GetGauge().GetValue())
}
// CounterValue returns the value of a counter metric with the given name and
// labels. Each call gathers separately; see [MetricValue].
func CounterValue(t testing.TB, reg prometheus.Gatherer, metricName string, labels prometheus.Labels) int {
t.Helper()
labeled := MetricValue(t, reg, metricName, labels)
require.NotNilf(t, labeled, "metric %q with labels %v not found", metricName, labels)
return int(labeled.GetCounter().GetValue())
}
// MetricValue returns the metric with the given name and labels, or nil if
// it is not found. Each call gathers `reg` separately, as does each call to
// testutil.ToFloat64; see the package documentation.
func MetricValue(t testing.TB, reg prometheus.Gatherer, metricName string, labels prometheus.Labels) *io_prometheus_client.Metric {
t.Helper()
metrics, err := reg.Gather()
require.NoError(t, err)
for _, m := range metrics {
if m.GetName() == metricName {
for _, labeled := range m.GetMetric() {
mLabels := make(prometheus.Labels)
for _, v := range labeled.GetLabel() {
mLabels[v.GetName()] = v.GetValue()
}
if maps.Equal(mLabels, labels) {
return labeled
}
}
}
}
return nil
}