Files
Nick Vigilante c95b908705 ci: make docs redirects relative and block removed routes (#30805)
## Summary

Moves `docs/redirects.json` to relative routes and makes
`lint/docs-redirects` fail when a PR removes a live docs route without a
redirect.
coder.com and docs-engine build each version's redirects from that
branch's file, so this file becomes the only place contributors add docs
redirects.

Part of DOCS-1290; tracked in DOCS-1296.

## Changes

- New rule format: `{"source": "path/to/page", "destination":
"path/to/another-page"}`, relative to the docs root of the branch's
version.
Trailing `/*` is the only wildcard, `/` is the version's docs root, one
`#fragment` is allowed on exact destinations, and external destinations
are limited to `https://github.com/coder/` and `https://coder.com/`.
The old `/docs/...` form is rejected with the converted rule in the
message.
- Removed-route check now blocks.
Every route in the base manifest, the live page list for the target
branch (`coder.com/beta-docs/<version>/llms.txt`), and the base file's
sources must be a page or resolve through a rule.
The failure names the old file and prints a paste-ready rule, suggesting
a renamed sibling, a page with the same last segment, or `/`.
  An unreadable live list exits 2 in CI; `-offline` skips it locally.
Links under `/docs` count as well, so the check keeps working if the
list URL redirects there after the move to `/docs`.
- `docs/redirects.json` on `main`: 304 rules.
It carries the coder.com docs rules that 404 under `/docs/@main` today,
plus the 144 `@main`-only rules coder.com holds, with three sibling sets
condensed into wildcards (`networking/*`, `admin/notifications/*`,
`guides/*`).
- README, content guidelines, and agent guidance describe the new
format.

## Testing

- `go test ./scripts/docsredirectscheck/...` (also `-race`),
`golangci-lint`, and the pre-commit hook.
- `go run ./scripts/docsredirectscheck -base origin/main -branch main`
online, after merging current `main`: 304 rules OK, 497 routes resolve.
- Renaming `admin/integrations/istio.md` without a rule fails with
`{"source": "admin/integrations/istio", "destination":
"admin/integrations/istio-mesh"}`; adding that rule passes.
- coder.com's `generateRedirects` with this file and `main`'s manifest:
0 dropped; 336 of 336 sampled old URLs resolve to the expected page with
Next's matcher.
- `curl` on every destination under `coder.com/docs/@main`: all 200.
- Live baseline before merge: 185 of 336 sampled `/docs/@main` URLs
return 404.
- The check runs on Linux in `ci.yaml`'s `lint` job; the package's unit
tests also run on macOS and Windows in `test-go-pg`.

## Merge order

Merge after the coder.com generator (coder/coder.com,
`vigilante/docs-redirects-relative`) is deployed, otherwise coder.com
drops these rules with warnings until it is.
The docs-engine change (coder/docs-engine,
`vigilante/version-redirects`) can land before or after.
Then merge the `release/2.38` backport.

<details>
<summary>Contract shared by coder/coder, coder.com, and
docs-engine</summary>

# Docs redirects contract (coder/coder owns the data; coder.com and
docs-engine consume it)

## Ownership

- Every redirect for a moved, renamed, or removed docs page lives in
`coder/coder` `docs/redirects.json`, one file per branch (`main`,
`release/X.Y`).
- A branch's file applies only to that branch's version of the docs.
`main`'s file applies to `@main`; `release/X.Y`'s file applies to vX.Y,
and to unversioned `/docs` while X.Y is mainline.
- URL-scheme rules (`/docs/v1`, `/docs/v2/...`, `/docs/coder-oss/...`,
`/docs/coder/...`, `/docs/code-server/...`, classic `/@v<line>` forms)
are not content moves. They stay in the consumer that owns the URL
scheme, as code or a static engine-owned file.
- Nobody adds content redirects in coder.com or docs-engine. Existing
ones there are removed only after the matching coder/coder rules are
verified live for that version (later phase, not this change).

## File format

`docs/redirects.json` is a JSON array. Each entry is an object with
exactly these fields:

| Field | Required | Meaning |

|---------------|----------|-------------------------------------------------------------------------|
| `source` | yes | A route relative to the docs root of this version |
| `destination` | yes | A route relative to the docs root, `/` for the
docs root, or an allowed `https://` URL |
| `permanent` | no | Boolean, default `true`. `true` answers 308,
`false` answers 307 |

Unknown fields are an error.

### Routes

- A route is the URL path below `/docs` with no version, no leading
slash, no trailing slash, and no `.md`:
`admin/templates/extending-templates`.
- A route is one or more segments joined by `/`. Each segment matches
`[A-Za-z0-9._-]+`. Nothing else is allowed (no `@`, `%`, `:`, `(`, `?`,
`#`, spaces, empty segments, `.` or `..` segments).
- Routes are derived from `docs/manifest.json` the way the sites do
today: strip a leading `./`, then strip `README.md`, `index.md`, `.md`
in that order as plain text, trim slashes. The empty result is the docs
root.

### Wildcards

- A `source` may end in `/*`: `ai-coder/ai-bridge/*`. It matches the
prefix itself and every route beneath it.
- A `destination` may end in `/*` only when the source does. The part
the source's `*` matched (possibly empty) replaces the destination's
`*`; an empty match means the destination prefix itself
(`ai-coder/ai-gateway/*` with an empty match is `ai-coder/ai-gateway`).
- A wildcard source with a non-wildcard destination sends everything
under the prefix to that one page.
- No other pattern syntax exists. Consumers translate `*` into their own
matcher. For Next.js that is `:path([A-Za-z0-9._-]+)*`, never `(.*)`,
`.+`, or a bare `:name*`.

### Destinations

- `/` is the docs root of the same version (`/docs`, `/docs/@main`,
`/beta-docs/v2.37`, and so on). Pages with no suitable home redirect to
`/`.
- An internal destination always stays in the same version as the
request.
- A non-wildcard internal destination may end in one `#fragment`, where
the fragment matches `[A-Za-z0-9._-]+`:
`user-guides/workspace-access#web-ides`. The route part must still be a
live route. A wildcard destination and `/` never carry a fragment.
- An external destination is an `https://` URL whose host and path start
with an allowed prefix: `https://github.com/coder/` or
`https://coder.com/`. It contains no `:` after the scheme and no
parameter. A `*` is not allowed in an external destination.

### Order and precedence

- Rules apply in file order; the first match wins.
- A page always wins over a redirect: a rule only fires for a request
that is not a live page of that version. The checker rejects any rule
whose source matches a live route, so consumers that check redirects
before pages (Next.js static redirects) behave the same as those that
check on a miss.
- Matching ignores one trailing slash on the request.
- Consumers follow at most 3 chained hops and refuse loops. The checker
rejects chains so a valid file never needs this.

### Legacy entries

- During migration, consumers also accept a `source` or `destination`
that starts with `/docs/` (or equals `/docs`) and treat it as the route
after `/docs/` (or `/` for `/docs`). A legacy `:name*` or `:name(.*)`
trailing parameter is treated as `/*`. The coder/coder checker accepts
only the new form.
- A missing file and an empty array both mean no rules. A file that is
not a JSON array is an error.

## CI check (coder/coder `scripts/docsredirectscheck`)

Structural checks as today, adapted to the new form, plus a blocking
removed-route check:

1. **Inventory.** The set of routes that must keep working is the union
of:
   - the base branch's `docs/manifest.json` routes,
- every route in the live production page list for the PR's target
branch, read from `https://coder.com/beta-docs/<segment>/llms.txt` where
the segment is `@main` for `main` and `vX.Y` for `release/X.Y`, keeping
only links under that segment,
- every `source` in the base branch's `docs/redirects.json` (exact
sources as-is; wildcard sources contribute the prefix).
2. **Resolution.** Each inventory route must be a live route in the head
manifest, or be resolved by the head rules (with wildcard substitution)
to a live head route, `/`, or an allowed external URL.
3. **Failure message.** Name the route, the file it came from on the
base branch when known, and a ready-to-paste rule, for example
`{"source": "path/to/page", "destination": "path/to/another-page"}`,
pointing at `/` when no candidate is obvious.
4. **Live list failures.** A network error, non-200, or a list with no
routes is an infrastructure failure (exit 2) in CI, not a pass. Locally,
`-offline` skips the live list and says so.
5. **Known engine difference.** beta-docs currently serves dotted
segments with hyphens (`esr-2.24-2.29-upgrade` is served as
`esr-2-24-2-29-upgrade`). Until docs-engine fixes it, a live route that
equals a head route with `.` replaced by `-` in each segment counts as
present.

## Consumers

- **coder.com classic** (`scripts/generate-docs-redirects.ts`): reads
`main` (scoped to `/docs/@main/...`) and the mainline release branch
(unversioned `/docs/...`) as today, in the new format, and emits Next.js
redirects. Version coverage beyond those two is out of scope.
- **docs-engine beta**: each version's publish reads
`docs/redirects.json` from the commit it builds, validates it, and
stores the compiled rules with that version's archive. The router
applies a version's own rules on a miss inside that version. Versions
published before this change keep today's behavior.

</details>

---

🤖 Built with AI assistance.
2026-11-05 14:04:17 -05:00

355 lines
11 KiB
Go

package main
import (
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"regexp"
"slices"
"strings"
"time"
"golang.org/x/xerrors"
)
const (
// liveListURLFormat is the production page list of one docs version. The
// segment is "@main" for main and "vX.Y" for release/X.Y.
liveListURLFormat = "https://coder.com/beta-docs/%s/llms.txt"
// userAgent identifies the check to the site. Its CDN answers 403 to the
// default Go user agent in some cases.
userAgent = "coder-docsredirectscheck (+https://github.com/coder/coder)"
fetchAttempts = 4
fetchTimeout = 30 * time.Second
fetchMaxBytes = 32 << 20
maxRedirectHops = 3
)
// fetcher reads a URL, so tests never touch the network.
type fetcher interface {
// Get returns the HTTP status and body of url.
Get(ctx context.Context, url string) (status int, body []byte, err error)
}
type httpFetcher struct {
client *http.Client
}
func newHTTPFetcher() httpFetcher {
return httpFetcher{client: &http.Client{Timeout: fetchTimeout}}
}
func (f httpFetcher) Get(ctx context.Context, url string) (int, []byte, error) {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return 0, nil, err
}
req.Header.Set("User-Agent", userAgent)
resp, err := f.client.Do(req)
if err != nil {
return 0, nil, err
}
defer resp.Body.Close()
body, err := io.ReadAll(io.LimitReader(resp.Body, fetchMaxBytes))
if err != nil {
return resp.StatusCode, nil, err
}
return resp.StatusCode, body, nil
}
// releaseBranchRe matches the release branches that have a docs version of their
// own.
var releaseBranchRe = regexp.MustCompile(`^release/(\d+\.\d+)$`)
// versionSegment returns the docs version segment the live page list of a
// target branch lives under, or false when the branch has no published docs.
func versionSegment(branch string) (string, bool) {
if branch == "main" {
return "@main", true
}
if m := releaseBranchRe.FindStringSubmatch(branch); m != nil {
return "v" + m[1], true
}
return "", false
}
// linkRe finds the target of a Markdown link.
var linkRe = regexp.MustCompile(`\]\(([^)\s]+)\)`)
// otherVersionRe matches the first path segment of a link into some version of
// the docs, which an unversioned link never has.
var otherVersionRe = regexp.MustCompile(`^(@[^/]+|v\d+(\.\d+)*)$`)
// liveListSiteBases are the base paths the docs site serves its versions
// under, in order of preference. The list URL may redirect from /beta-docs to
// /docs once the docs move there, and its links then use /docs.
var liveListSiteBases = []string{"/beta-docs", "/docs"}
// parseLiveList returns the routes of the links in an llms.txt body that point
// into the given version. The docs root is the empty route.
//
// The version that is served at a site's unversioned base path links its own
// pages without a version segment, so when no link carries the segment, the
// unversioned links count instead.
func parseLiveList(body []byte, segment string) []string {
var hrefs []string
for _, m := range linkRe.FindAllSubmatch(body, -1) {
href := strings.TrimPrefix(string(m[1]), "https://coder.com")
if i := strings.IndexAny(href, "?#"); i >= 0 {
href = href[:i]
}
hrefs = append(hrefs, href)
}
for _, base := range liveListSiteBases {
if routes := routesUnder(hrefs, base+"/"+segment); len(routes) > 0 {
return routes
}
}
for _, base := range liveListSiteBases {
unversioned := slices.DeleteFunc(routesUnder(hrefs, base), func(route string) bool {
first, _, _ := strings.Cut(route, "/")
return otherVersionRe.MatchString(first)
})
if len(unversioned) > 0 {
return unversioned
}
}
return nil
}
// routesUnder returns the routes of the hrefs at or below base, without
// duplicates and in order of appearance.
func routesUnder(hrefs []string, base string) []string {
var routes []string
seen := map[string]bool{}
for _, href := range hrefs {
if href != base && !strings.HasPrefix(href, base+"/") {
continue
}
route := convertPathToRoute(strings.TrimPrefix(href, base))
if route != "" && checkRoute(route) != "" {
continue
}
if !seen[route] {
seen[route] = true
routes = append(routes, route)
}
}
return routes
}
// errNotPublished means the site has no page list for a version, which is the
// case for a new release branch before its docs are published.
var errNotPublished = xerrors.New("docs version is not published")
// sleepContext waits for d, or returns early with the context's error when ctx
// is done first.
func sleepContext(ctx context.Context, d time.Duration) error {
t := time.NewTimer(d)
defer t.Stop()
select {
case <-ctx.Done():
return ctx.Err()
case <-t.C:
return nil
}
}
// fetchLiveRoutes reads the live page list of a version, retrying transient
// failures a bounded number of times. An empty list is an error. A 404 is not
// retried and returns an error that wraps errNotPublished.
func fetchLiveRoutes(ctx context.Context, f fetcher, url, segment string, wait func(context.Context, time.Duration) error) ([]string, error) {
var lastErr error
for attempt := 0; attempt < fetchAttempts; attempt++ {
if attempt > 0 {
if err := wait(ctx, time.Duration(1<<(attempt-1))*time.Second); err != nil {
return nil, xerrors.Errorf("cannot read %s: %w", url, err)
}
}
status, body, err := f.Get(ctx, url)
switch {
case err != nil:
lastErr = err
case status == http.StatusNotFound:
return nil, xerrors.Errorf("%s answered HTTP status 404: %w", url, errNotPublished)
case status != http.StatusOK:
lastErr = xerrors.Errorf("HTTP status %d", status)
default:
routes := parseLiveList(body, segment)
if len(routes) == 0 {
return nil, xerrors.Errorf("%s lists no pages of docs version %s", url, segment)
}
return routes, nil
}
}
return nil, xerrors.Errorf("cannot read %s after %d attempts: %w", url, fetchAttempts, lastErr)
}
// inventory is the set of routes that must keep working, each with where it
// came from.
type inventory map[string][]string
func (inv inventory) add(route, origin string) {
if route == "" {
// The docs root always exists.
return
}
if !slices.Contains(inv[route], origin) {
inv[route] = append(inv[route], origin)
}
}
// baseRuleSources returns the route each rule of a base branch's
// docs/redirects.json answers for: exact sources as written, and the prefix of
// a wildcard. It is lenient on purpose, because the base file may predate the
// current format. It reads the old absolute form, and skips what it cannot
// read.
func baseRuleSources(data []byte) []string {
var entries []struct {
Source string `json:"source"`
}
if err := json.Unmarshal(data, &entries); err != nil {
return nil
}
var routes []string
for _, e := range entries {
route := strings.TrimSuffix(convertLegacy(e.Source), "/*")
if checkRoute(route) == "" {
routes = append(routes, route)
}
}
return routes
}
// resolver decides whether a route still works in the head branch.
type resolver struct {
live routeSet
// hyphenated holds the live routes with "." replaced by "-".
hyphenated map[string]bool
rules []rule
// added holds the routes that are live in head and were not live in the
// base branch, which are the likely new homes of a renamed page. It is
// empty when there is no base manifest to compare with.
added []string
}
// newResolver builds a resolver for the head branch. baseLive is the base
// branch's live routes, or nil when they are not known.
func newResolver(live, baseLive routeSet, rules []rule) resolver {
r := resolver{live: live, hyphenated: live.hyphenated(), rules: rules}
if baseLive != nil {
for route := range live {
if route != "" && !baseLive.has(route) {
r.added = append(r.added, route)
}
}
slices.Sort(r.added)
}
return r
}
// isLive reports whether a route is a page of the head branch. beta-docs
// serves dotted segments with hyphens ("esr-2.24-2.29-upgrade" is served as
// "esr-2-24-2-29-upgrade"), so a route that equals a head route with "." replaced
// by "-" counts as present until the docs engine is fixed.
func (r resolver) isLive(route string) bool {
return r.live.has(route) || r.hyphenated[route]
}
// resolves reports whether a request for route ends on a page: it is a live
// route, or the head rules send it, within maxRedirectHops, to a live route,
// the docs root, or an allowed external URL. Every redirect counts as a hop,
// whatever its destination.
func (r resolver) resolves(route string) bool {
route = normalizeRoute(route)
for hops := 0; ; hops++ {
if r.isLive(route) {
return true
}
i := slices.IndexFunc(r.rules, func(rl rule) bool { return rl.matches(route) })
if i < 0 || !r.rules[i].dst.valid || hops == maxRedirectHops {
return false
}
kind, next := r.rules[i].target(route)
if kind != destRoute {
return true
}
route = next
}
}
// suggest picks a destination for a rule that restores route. In order of
// preference it is the one live page with the same last segment, the one route
// added by this change in the same parent section (a rename such as
// "admin/integrations/istio" to "admin/integrations/istio-mesh"), or the docs
// root when no page is an obvious home.
func (r resolver) suggest(route string) string {
last := func(s string) string {
s = strings.ReplaceAll(s, ".", "-")
return s[strings.LastIndex(s, "/")+1:]
}
var found []string
for candidate := range r.live {
if candidate != "" && last(candidate) == last(route) {
found = append(found, candidate)
}
}
if len(found) == 1 {
return found[0]
}
parent := func(s string) string {
i := strings.LastIndex(s, "/")
if i < 0 {
return ""
}
return s[:i]
}
found = found[:0]
for _, candidate := range r.added {
if parent(candidate) == parent(route) {
found = append(found, candidate)
}
}
if len(found) == 1 {
return found[0]
}
return "/"
}
// missingRoute is an inventory route that no longer works.
type missingRoute struct {
route string
origins []string
suggestion string
}
func (m missingRoute) rule() string {
return ruleJSON(m.route, m.suggestion, nil)
}
func (m missingRoute) message() string {
return fmt.Sprintf(
"docs route %q no longer resolves (from %s). Add a rule to docs/redirects.json, for example: %s",
m.route, strings.Join(m.origins, ", "), m.rule(),
)
}
// missingRoutes returns the inventory routes the head branch no longer serves
// or redirects, sorted by route.
func missingRoutes(inv inventory, r resolver) []missingRoute {
var missing []missingRoute
for route, origins := range inv {
if r.resolves(route) {
continue
}
missing = append(missing, missingRoute{route: route, origins: origins, suggestion: r.suggest(route)})
}
slices.SortFunc(missing, func(a, b missingRoute) int { return strings.Compare(a.route, b.route) })
return missing
}