5 Commits
Author SHA1 Message Date
Nick Vigilante 9bafd0a525 ci(.github): skip a doc-check run when the PR head already has a finished review (#32198) 2026-11-13 17:31:21 -05:00
Nick VigilanteandBen Potter dc55633b47 ci(.github): start a new doc-check chat per run and stop it when the run ends (#32185)
## Summary

doc-check reused one chat for every push to a PR, and a review kept
running in Coder after its job stopped waiting for it.
This PR starts a new chat for each run, waits up to 5 hours for it, and
stops the review and its workspace when the run ends for any reason.

On dogfood at 2026-11-13 17:28 UTC, 58 doc-check review chats and 45 of
their sub-agents had status `running`, while 5 workflow runs were in
progress and 1 was queued.
34 of the 58 chats were created before that day, and the oldest on
2026-09-29.
The count covers only chats shared with me, so it is a lower bound.

This combines the workflow half of #32061 with the stop script from
#31720, and changes no skill file.

## Changes

- **New chat per run.**
`force-new-chat` replaces the one chat per PR, so a re-run no longer
carries every earlier turn on the PR.
The prompt gets a `## Previous run` section with the `doc-check-trace`
of the latest finished review.
- **One workspace per PR.**
The job finds `doc-check-pr-<N>`, wakes and starts it, and passes its ID
to the chat.
  On the first run the chat creates the workspace under that name.
- **A started review finishes; pushes are debounced.**
The review job's concurrency group is keyed on the PR URL with
`cancel-in-progress: false`. GitHub keeps one pending run per group and
replaces it on each push, so the next review covers the latest head.
A `debounce` job sleeps 5 minutes on `synchronize`, in its own group
with `cancel-in-progress: true`, so a burst of pushes starts one review.
Other triggers do not wait.
Merging the PR still cancels a running review (the `delete-workspace`
job keeps `cancel-in-progress: true`), then deletes the workspace.
- **Every run stops its review.**
`.github/scripts/doc-check-stop-review.sh` runs before the review and in
a final `always()` step.
It drops queued messages, interrupts the chat and its sub-agents, and
stops the workspace.
- **5-hour wait.**
`wait-timeout-seconds` goes from 2400 to 18000, and the job timeout from
50 to 330 minutes.
The job now stays in its concurrency group for the whole review, so the
next run for the PR waits for it.
- **Previous trace lookup.**
`.github/scripts/doc-check-previous-trace.sh` reads the 5 newest review
chats for the PR and prints the first trace with an `outcome:` line.
It reads more than one chat because a cancelled run's chat has no trace.
  A failed lookup costs only that context and never fails the job.
- The `GITHUB_OUTPUT` delimiter for the prompt is random, because the
trace is model output.

### Behavior changes to check

| Before | After | Why |
|--------|-------|-----|
| A review past the wait ended the job with a green notice and kept
running (#32097) | The last step stops the review, and the job fails |
Nothing watches a review after the job ends |
| A review kept running after its PR merged | The merge cancels the run
| The result has no reader |
| The `doc-check` label forced one review, on the `labeled` event | The
label forces a review on every push until it is removed | The push run
would otherwise skip on paths with no docs surface |
| Runs for one PR queued | A started review finishes; pushes during it
collapse into one pending run for the latest head, after a 5-minute
debounce | Cancelling threw away prefill and starved busy PRs of a
verdict |

## Testing

Static checks, all passing: `actionlint`, `zizmor`, `shellcheck
--external-sources`, and `shfmt -d`.

Script tests, run locally:

- `doc-check-stop-review.test.sh`: 48 cases against a fake API.
- `doc-check-previous-trace.test.sh`: 22 cases, including CRLF in the
trace text and in the PR URL.
- Mutation check on the trace script: 25 one-line breaks, and the suite
failed on 24.
  The survivor was a redundant trim, which I removed.
- The trace extraction ran against 12 real finished doc-check chats on
dogfood and returned a trace for all 12 (750 to 3,500 characters).

End to end on dogfood, on this PR, 2026-11-13 UTC:

| Run | Trigger | Result |
|-----|---------|--------|
| [1](https://github.com/coder/coder/actions/runs/43474029515) |
dispatch, 17:33 | No workspace and no earlier chat. New chat created the
workspace `doc-check-pr-32185` and had 2 running sub-agents at 17:48:23.
|
| [2](https://github.com/coder/coder/actions/runs/43476429715) |
dispatch, 17:48:25 | Cancelled run 1 at 17:48:27. Run 1's last step
interrupted the chat and both sub-agents and stopped the workspace by
17:48:51. Run 2 then started the same workspace, opened a second chat in
it, finished clean at 17:52:54, and stopped the workspace. |
| [3](https://github.com/coder/coder/actions/runs/43477511866) |
dispatch, 17:55:08 | Cancelled by run 4 while the workspace was
starting. Its last step stopped the workspace mid-start. No chat was
created. |
| [4](https://github.com/coder/coder/actions/runs/43477572348) |
`ready_for_review`, 17:55:31 | Waited for run 3 to end, then opened a
third chat in the same workspace. That chat's first message carries run
2's trace under `## Previous run`. |

Not covered:

- The 5-hour timeout path has not run live.
The fake-API tests cover the script, and the summary text is a few lines
of shell.
- The merge path (`delete-workspace` cancelling a review) has not run
live.
- No CI job runs the two `.test.sh` suites.
The workflow runs on `ubuntu-latest` only, and the scripts were not run
on macOS or Windows.

## Follow-ups, not in this PR

- Skip a run when the head SHA or the PR diff matches the last reviewed
one.
- Set a status on the head SHA from the trace.
- Cancel a review when its PR closes without merging.

Supersedes #31720 and the workflow half of #32061.

Linear: DOCS-1745

<details>
<summary>Plan and decision log</summary>

This is PR 1 of a 5-part stack from the design doc "Design: reliable,
low-cost doc-check on Solstice".

1. Workflow base: new chat per run, previous trace, workspace ID, 5-hour
wait, cancel-in-progress, and the stop script. **This PR.**
2. Skip in Actions: same head SHA, same PR diff, and a delta since the
last reviewed SHA with no docs surface.
3. Visible outcome: a status on the head SHA from the trace, for clean,
findings, error, and skipped.
4. Packet and lint in Actions: PR data, diffs, open threads, and
deterministic check results, written once.
5. Skill: scope to wrong, missing, and broken, tiers, a sub-agent cap,
and one long wait. Gated by an eval.

Decisions in this PR:

- **Why this first.**
A re-run's first prompt was 1.01 times the previous run's last prompt,
so each push re-sent the whole history of the PR.
A new chat per run removes that, and stopping stale reviews removes load
that no job is watching.
- **5 hours, not 25 or 40 minutes.**
Reviews on the local model took a median of 61 minutes when clean and
247 minutes with findings (2026-11-04 to 11-11).
  A shorter wait would stop most reviews with findings before they post.
With `cancel-in-progress`, a new push still ends the review of the old
head at once.
- **A timeout fails the job.**
  This reverses #32097, where a timeout was a notice.
  A green job with no review reads as a clean result.
- **The stop runs twice.**
The first step catches a review left by a run whose last step did not
finish, for example a runner that died.
- **The trace lookup is a script with tests, not inline YAML.**
It parses model output, and it has to handle chats with no trace, a
trace with no outcome, and CRLF.
- **The trace is passed as a record, not as instructions.**
  The prompt says so, and asks for a review of the whole PR.
A delta review needs the skip logic and the ledger from later PRs in the
stack.
- **No skill change.**
  The skill changes are gated by an eval that does not exist yet.

</details>

Follow-up commit `64a09655a1` (finish-then-rerun and debounce) generated
by Coder Agents on behalf of @bpmct.

_Generated by Coder Agents on behalf of @nickvigilante._

---------

Co-authored-by: Ben Potter <me@bpmct.net>
2026-11-13 21:14:35 +00:00
Ethan 7298a29885 ci(.github): time out stalled Go module downloads (#30889)
Closes CODAGT-878

When proxy.golang.org is having a bad day, a module download can stall
for minutes before the proxy resets the stream with `INTERNAL_ERROR`.
`go` has no read timeout of its own, so `retry.sh` only gets to retry
once the proxy gives up. In [this `test-e2e`
run](https://github.com/coder/coder/actions/runs/38567440844/job/115863803256)
the second attempt sat there for 10 minutes, and the job hit its 20
minute timeout during Playwright, even though the third attempt passed
straight away.

To fix, `retry.sh` now takes a `--timeout`, and kills any attempt that
runs longer than that. Killing `go mod download` is safe, and the next
attempt picks up where the last one left off, as finished zips are
already in the module cache. The timer is in bash as we can't rely on
GNU `timeout` on the macOS runners, and a step `timeout-minutes` would
kill the whole retry loop (it isn't supported in composite actions
anyway).

I've gone with 120s per attempt. Across the last 32 runs on main, a
healthy download took at most 41s on any platform, with macOS the
slowest at 26 to 41s. During the incident on 2026-10-14, attempts that
failed on their own took between 66s and 10 minutes, and three passed
slowly at 113s, 133s and 151s. Those last two would now get killed, but
that only costs a retry, which just needs whatever's left. Three
attempts at 120s plus backoff comes to about 6 minutes, which still fits
in `test-e2e`, whose slowest green run takes about 11 of its 20 minutes.

Separately, `make fmt` runs a bare `go mod tidy`, which needs modules
that `go mod download` doesn't fetch (test dependencies of
dependencies). So it was downloading 166 modules without any retries,
and [failing
here](https://github.com/coder/coder/actions/runs/38624596116/job/116064172735).
The `fmt` job now runs `go mod tidy` through `retry.sh` instead of `go
mod download`.
2026-10-19 13:03:10 +11:00
dylanhuff-at-coder 056c3e4f55 ci(.github): collect Windows PostgreSQL diagnostics (#29362)
Add bounded Windows PostgreSQL listener, connection, process, memory and
disk diagnostics alongside the existing server logs. Retain timestamped
samples and sanitized failures as an artifact on unsuccessful runs. This
gathers evidence for PLAT-307 without claiming its cause or a fix.

A concrete runner preserves make's exit status and the caller's
module-path environment, and cleans up only its owned sampler. Windows
concurrency remains 8 packages and 16 tests per package; the sampler's
500 ms connection deadline, PostgreSQL settings, retries and production
telemetry are unchanged.

Focused checks cover a strict native Windows sample, connection and
memory boundaries, sampling limits, cleanup, make exits and environment
preservation. Full local pre-commit, action validation and focused
checks pass, including LF and CRLF harness copies. Current-head Windows
native smoke, reduced harness and Go tests pass. All applicable CI
checks pass, including the required check.

Related:
[PLAT-307](https://linear.app/codercom/issue/PLAT-307/flake-testtelemetry-multiple-variants-postgres-connection-refused).

<details>
<summary>Implementation approach</summary>

Run an isolated, bounded sampler around make. Select diagnostic fields
without credentials, command lines or raw error messages. Validate a
completed native sample before Go tests, including cached runs, and keep
that sample in the failure-artifact directory.

</details>

Generated with Coder Agents.
2026-09-22 10:14:02 -07:00
Kacper Sawicki 9a417df940 ci: add retry logic for Go module operations (#21609)
## Description

Add exponential backoff retries to all `go install` and `go mod
download` commands across CI workflows and actions.

## Why

Fixes
[coder/internal#1276](https://github.com/coder/internal/issues/1276) -
CI fails when `sum.golang.org` returns 500 errors during Go module
verification. This is an infrastructure-level flake that can't be
controlled.

## Changes

- Created `.github/scripts/retry.sh` - reusable retry helper with
exponential backoff (2s, 4s, 8s delays, max 3 attempts), using
`scripts/lib.sh` helpers
- Wrapped all `go install` and `go mod download` commands with retry in:
  - `.github/actions/setup-go/action.yaml`
  - `.github/actions/setup-sqlc/action.yaml`
  - `.github/actions/setup-go-tools/action.yaml`
  - `.github/workflows/ci.yaml`
  - `.github/workflows/release.yaml`
  - `.github/workflows/security.yaml`
- Added GNU tools setup (bash 4+, GNU getopt, make 4+) for macOS in
`test-go-pg` job, since `retry.sh` uses `lib.sh` which requires these
tools
2026-01-29 16:05:49 +01:00