Files
eric-hughes-coder bb8912dc46 fix: wait for full completion before asserting in tests (#30885)
Fixes the `TestServer_Metrics/Forwarding` flake reported in
[PLAT-609](https://linear.app/codercom/issue/PLAT-609/flake-testserver-metricsforwarding)
and, in addition, other defects similar to it.

## Testing defects

### The Original

`TestServer_Metrics/Forwarding` waited for `batches_dropped_total` and
then asserted on `logs_dropped_total`. The forwarder updates those two
counters separately, on adjacent lines, so a read between them sees the
first updated and the second not. CI caught exactly that window:
expected 2, actual 0 at `proxy_test.go:780`. Each counter is atomic on
its own, so the race detector reports nothing.

### The Additional

The fallacy of the original is a _de facto_ assumption that sequential
updates are atomic as a group. The defect arose from not taking into
account that they are atomic individually. The test asserted before the
work was completed. A repo-wide sweep for related metrics written
together found similar mistakes in other tests:

- Live races: all three phases of `TestServer_Metrics/Forwarding` and
`TestServer_Metrics/BufferFull` in `agent/boundarylogproxy`, and
`TestMetadataBatcher_DropsWhenFull`, which waited for the update channel
to empty although the final flush runs after that.
- Correct only because of the current implementation's ordering:
`TestMetrics_TokenUseCount` in `aibridge`, four tests in
`coderd/agentapi/metadatabatcher`, and `TestAgentStatsSessionCounts` in
`coderd/prometheusmetrics`. These pass today but would start failing if
the code under test reordered its work.

## Fixes

All tests the sweep found were fixed. Each test now waits for every
value it asserts before asserting it. Where a value is already non-zero
from earlier work, or legitimately zero, the wait is for the final value
rather than for a change, and several tests use
`require.EventuallyWithT` so the wait and the assertion are one step.
Two Forwarding phases gain assertions on values they already waited for:
`batches_dropped_total == 1` in phase 1, and the reporter's recorded
request count in phase 2.

Two phases of `TestAgentStatsSessionCounts` assert that the counts did
not change. No wait on an unchanged value can show that the collection
cycle ran, so they keep waiting on the execution histogram, and a
comment at that wait names the dependency.

Each fix was checked by temporarily blocking or reordering the code
under test between the relevant writes: the old test failed on every run
and the new one passed.

## Documentation

As a guard against recurrence of defects of this kind, we add the
following documentation targeted mostly at agents. The material in
`TESTING.md` provides guidance about how to write such a test correctly
in the first place. That in `AGENT_FAILURES.md` describes how to
recognize the symptoms of this class of test failure, should they arise.

- `.claude/docs/TESTING.md`: new section "Wait for Full Completion
before Asserting", covering waits on one value with assertions on
another, Prometheus metrics in particular, and assertions that nothing
changed.
- `.claude/docs/AGENT_FAILURES.md`: a catalog entry for the symptom.
- `coderd/coderdtest/promhelp`: doc comments noting that each helper
call is a separate gather.

Generated with Claude Code under the direction of @eric-hughes-coder.
2026-10-28 09:17:10 -06:00
..