Fixes the `TestServer_Metrics/Forwarding` flake reported in [PLAT-609](https://linear.app/codercom/issue/PLAT-609/flake-testserver-metricsforwarding) and, in addition, other defects similar to it. ## Testing defects ### The Original `TestServer_Metrics/Forwarding` waited for `batches_dropped_total` and then asserted on `logs_dropped_total`. The forwarder updates those two counters separately, on adjacent lines, so a read between them sees the first updated and the second not. CI caught exactly that window: expected 2, actual 0 at `proxy_test.go:780`. Each counter is atomic on its own, so the race detector reports nothing. ### The Additional The fallacy of the original is a _de facto_ assumption that sequential updates are atomic as a group. The defect arose from not taking into account that they are atomic individually. The test asserted before the work was completed. A repo-wide sweep for related metrics written together found similar mistakes in other tests: - Live races: all three phases of `TestServer_Metrics/Forwarding` and `TestServer_Metrics/BufferFull` in `agent/boundarylogproxy`, and `TestMetadataBatcher_DropsWhenFull`, which waited for the update channel to empty although the final flush runs after that. - Correct only because of the current implementation's ordering: `TestMetrics_TokenUseCount` in `aibridge`, four tests in `coderd/agentapi/metadatabatcher`, and `TestAgentStatsSessionCounts` in `coderd/prometheusmetrics`. These pass today but would start failing if the code under test reordered its work. ## Fixes All tests the sweep found were fixed. Each test now waits for every value it asserts before asserting it. Where a value is already non-zero from earlier work, or legitimately zero, the wait is for the final value rather than for a change, and several tests use `require.EventuallyWithT` so the wait and the assertion are one step. Two Forwarding phases gain assertions on values they already waited for: `batches_dropped_total == 1` in phase 1, and the reporter's recorded request count in phase 2. Two phases of `TestAgentStatsSessionCounts` assert that the counts did not change. No wait on an unchanged value can show that the collection cycle ran, so they keep waiting on the execution histogram, and a comment at that wait names the dependency. Each fix was checked by temporarily blocking or reordering the code under test between the relevant writes: the old test failed on every run and the new one passed. ## Documentation As a guard against recurrence of defects of this kind, we add the following documentation targeted mostly at agents. The material in `TESTING.md` provides guidance about how to write such a test correctly in the first place. That in `AGENT_FAILURES.md` describes how to recognize the symptoms of this class of test failure, should they arise. - `.claude/docs/TESTING.md`: new section "Wait for Full Completion before Asserting", covering waits on one value with assertions on another, Prometheus metrics in particular, and assertions that nothing changed. - `.claude/docs/AGENT_FAILURES.md`: a catalog entry for the symptom. - `coderd/coderdtest/promhelp`: doc comments noting that each helper call is a separate gather. Generated with Claude Code under the direction of @eric-hughes-coder.
10 lines
446 B
Go
10 lines
446 B
Go
// Package promhelp provides helper functions for asserting Prometheus
|
|
// metric values in unit tests.
|
|
//
|
|
// Each value-reading helper gathers the registry on every call, so values
|
|
// read by separate calls are separate reads and need not agree with each
|
|
// other. A test that waits on one metric and then asserts another can fail
|
|
// intermittently. See "Wait for Full Completion before Asserting" in
|
|
// .claude/docs/TESTING.md.
|
|
package promhelp
|