Closes CODAGT-878
When proxy.golang.org is having a bad day, a module download can stall
for minutes before the proxy resets the stream with `INTERNAL_ERROR`.
`go` has no read timeout of its own, so `retry.sh` only gets to retry
once the proxy gives up. In [this `test-e2e`
run](https://github.com/coder/coder/actions/runs/38567440844/job/115863803256)
the second attempt sat there for 10 minutes, and the job hit its 20
minute timeout during Playwright, even though the third attempt passed
straight away.
To fix, `retry.sh` now takes a `--timeout`, and kills any attempt that
runs longer than that. Killing `go mod download` is safe, and the next
attempt picks up where the last one left off, as finished zips are
already in the module cache. The timer is in bash as we can't rely on
GNU `timeout` on the macOS runners, and a step `timeout-minutes` would
kill the whole retry loop (it isn't supported in composite actions
anyway).
I've gone with 120s per attempt. Across the last 32 runs on main, a
healthy download took at most 41s on any platform, with macOS the
slowest at 26 to 41s. During the incident on 2026-10-14, attempts that
failed on their own took between 66s and 10 minutes, and three passed
slowly at 113s, 133s and 151s. Those last two would now get killed, but
that only costs a retry, which just needs whatever's left. Three
attempts at 120s plus backoff comes to about 6 minutes, which still fits
in `test-e2e`, whose slowest green run takes about 11 of its 20 minutes.
Separately, `make fmt` runs a bare `go mod tidy`, which needs modules
that `go mod download` doesn't fetch (test dependencies of
dependencies). So it was downloading 166 modules without any retries,
and [failing
here](https://github.com/coder/coder/actions/runs/38624596116/job/116064172735).
The `fmt` job now runs `go mod tidy` through `retry.sh` instead of `go
mod download`.
Add bounded Windows PostgreSQL listener, connection, process, memory and
disk diagnostics alongside the existing server logs. Retain timestamped
samples and sanitized failures as an artifact on unsuccessful runs. This
gathers evidence for PLAT-307 without claiming its cause or a fix.
A concrete runner preserves make's exit status and the caller's
module-path environment, and cleans up only its owned sampler. Windows
concurrency remains 8 packages and 16 tests per package; the sampler's
500 ms connection deadline, PostgreSQL settings, retries and production
telemetry are unchanged.
Focused checks cover a strict native Windows sample, connection and
memory boundaries, sampling limits, cleanup, make exits and environment
preservation. Full local pre-commit, action validation and focused
checks pass, including LF and CRLF harness copies. Current-head Windows
native smoke, reduced harness and Go tests pass. All applicable CI
checks pass, including the required check.
Related:
[PLAT-307](https://linear.app/codercom/issue/PLAT-307/flake-testtelemetry-multiple-variants-postgres-connection-refused).
<details>
<summary>Implementation approach</summary>
Run an isolated, bounded sampler around make. Select diagnostic fields
without credentials, command lines or raw error messages. Validate a
completed native sample before Go tests, including cached runs, and keep
that sample in the failure-artifact directory.
</details>
Generated with Coder Agents.
## Description
Add exponential backoff retries to all `go install` and `go mod
download` commands across CI workflows and actions.
## Why
Fixes
[coder/internal#1276](https://github.com/coder/internal/issues/1276) -
CI fails when `sum.golang.org` returns 500 errors during Go module
verification. This is an infrastructure-level flake that can't be
controlled.
## Changes
- Created `.github/scripts/retry.sh` - reusable retry helper with
exponential backoff (2s, 4s, 8s delays, max 3 attempts), using
`scripts/lib.sh` helpers
- Wrapped all `go install` and `go mod download` commands with retry in:
- `.github/actions/setup-go/action.yaml`
- `.github/actions/setup-sqlc/action.yaml`
- `.github/actions/setup-go-tools/action.yaml`
- `.github/workflows/ci.yaml`
- `.github/workflows/release.yaml`
- `.github/workflows/security.yaml`
- Added GNU tools setup (bash 4+, GNU getopt, make 4+) for macOS in
`test-go-pg` job, since `retry.sh` uses `lib.sh` which requires these
tools