Files
coder/.github/scripts
Ethan 7298a29885 ci(.github): time out stalled Go module downloads (#30889)
Closes CODAGT-878

When proxy.golang.org is having a bad day, a module download can stall
for minutes before the proxy resets the stream with `INTERNAL_ERROR`.
`go` has no read timeout of its own, so `retry.sh` only gets to retry
once the proxy gives up. In [this `test-e2e`
run](https://github.com/coder/coder/actions/runs/38567440844/job/115863803256)
the second attempt sat there for 10 minutes, and the job hit its 20
minute timeout during Playwright, even though the third attempt passed
straight away.

To fix, `retry.sh` now takes a `--timeout`, and kills any attempt that
runs longer than that. Killing `go mod download` is safe, and the next
attempt picks up where the last one left off, as finished zips are
already in the module cache. The timer is in bash as we can't rely on
GNU `timeout` on the macOS runners, and a step `timeout-minutes` would
kill the whole retry loop (it isn't supported in composite actions
anyway).

I've gone with 120s per attempt. Across the last 32 runs on main, a
healthy download took at most 41s on any platform, with macOS the
slowest at 26 to 41s. During the incident on 2026-10-14, attempts that
failed on their own took between 66s and 10 minutes, and three passed
slowly at 113s, 133s and 151s. Those last two would now get killed, but
that only costs a retry, which just needs whatever's left. Three
attempts at 120s plus backoff comes to about 6 minutes, which still fits
in `test-e2e`, whose slowest green run takes about 11 of its 20 minutes.

Separately, `make fmt` runs a bare `go mod tidy`, which needs modules
that `go mod download` doesn't fetch (test dependencies of
dependencies). So it was downloading 166 modules without any retries,
and [failing
here](https://github.com/coder/coder/actions/runs/38624596116/job/116064172735).
The `fmt` job now runs `go mod tidy` through `retry.sh` instead of `go
mod download`.
2026-10-19 13:03:10 +11:00
..