Files
Nick Vigilante c95b908705 ci: make docs redirects relative and block removed routes (#30805)
## Summary

Moves `docs/redirects.json` to relative routes and makes
`lint/docs-redirects` fail when a PR removes a live docs route without a
redirect.
coder.com and docs-engine build each version's redirects from that
branch's file, so this file becomes the only place contributors add docs
redirects.

Part of DOCS-1290; tracked in DOCS-1296.

## Changes

- New rule format: `{"source": "path/to/page", "destination":
"path/to/another-page"}`, relative to the docs root of the branch's
version.
Trailing `/*` is the only wildcard, `/` is the version's docs root, one
`#fragment` is allowed on exact destinations, and external destinations
are limited to `https://github.com/coder/` and `https://coder.com/`.
The old `/docs/...` form is rejected with the converted rule in the
message.
- Removed-route check now blocks.
Every route in the base manifest, the live page list for the target
branch (`coder.com/beta-docs/<version>/llms.txt`), and the base file's
sources must be a page or resolve through a rule.
The failure names the old file and prints a paste-ready rule, suggesting
a renamed sibling, a page with the same last segment, or `/`.
  An unreadable live list exits 2 in CI; `-offline` skips it locally.
Links under `/docs` count as well, so the check keeps working if the
list URL redirects there after the move to `/docs`.
- `docs/redirects.json` on `main`: 304 rules.
It carries the coder.com docs rules that 404 under `/docs/@main` today,
plus the 144 `@main`-only rules coder.com holds, with three sibling sets
condensed into wildcards (`networking/*`, `admin/notifications/*`,
`guides/*`).
- README, content guidelines, and agent guidance describe the new
format.

## Testing

- `go test ./scripts/docsredirectscheck/...` (also `-race`),
`golangci-lint`, and the pre-commit hook.
- `go run ./scripts/docsredirectscheck -base origin/main -branch main`
online, after merging current `main`: 304 rules OK, 497 routes resolve.
- Renaming `admin/integrations/istio.md` without a rule fails with
`{"source": "admin/integrations/istio", "destination":
"admin/integrations/istio-mesh"}`; adding that rule passes.
- coder.com's `generateRedirects` with this file and `main`'s manifest:
0 dropped; 336 of 336 sampled old URLs resolve to the expected page with
Next's matcher.
- `curl` on every destination under `coder.com/docs/@main`: all 200.
- Live baseline before merge: 185 of 336 sampled `/docs/@main` URLs
return 404.
- The check runs on Linux in `ci.yaml`'s `lint` job; the package's unit
tests also run on macOS and Windows in `test-go-pg`.

## Merge order

Merge after the coder.com generator (coder/coder.com,
`vigilante/docs-redirects-relative`) is deployed, otherwise coder.com
drops these rules with warnings until it is.
The docs-engine change (coder/docs-engine,
`vigilante/version-redirects`) can land before or after.
Then merge the `release/2.38` backport.

<details>
<summary>Contract shared by coder/coder, coder.com, and
docs-engine</summary>

# Docs redirects contract (coder/coder owns the data; coder.com and
docs-engine consume it)

## Ownership

- Every redirect for a moved, renamed, or removed docs page lives in
`coder/coder` `docs/redirects.json`, one file per branch (`main`,
`release/X.Y`).
- A branch's file applies only to that branch's version of the docs.
`main`'s file applies to `@main`; `release/X.Y`'s file applies to vX.Y,
and to unversioned `/docs` while X.Y is mainline.
- URL-scheme rules (`/docs/v1`, `/docs/v2/...`, `/docs/coder-oss/...`,
`/docs/coder/...`, `/docs/code-server/...`, classic `/@v<line>` forms)
are not content moves. They stay in the consumer that owns the URL
scheme, as code or a static engine-owned file.
- Nobody adds content redirects in coder.com or docs-engine. Existing
ones there are removed only after the matching coder/coder rules are
verified live for that version (later phase, not this change).

## File format

`docs/redirects.json` is a JSON array. Each entry is an object with
exactly these fields:

| Field | Required | Meaning |

|---------------|----------|-------------------------------------------------------------------------|
| `source` | yes | A route relative to the docs root of this version |
| `destination` | yes | A route relative to the docs root, `/` for the
docs root, or an allowed `https://` URL |
| `permanent` | no | Boolean, default `true`. `true` answers 308,
`false` answers 307 |

Unknown fields are an error.

### Routes

- A route is the URL path below `/docs` with no version, no leading
slash, no trailing slash, and no `.md`:
`admin/templates/extending-templates`.
- A route is one or more segments joined by `/`. Each segment matches
`[A-Za-z0-9._-]+`. Nothing else is allowed (no `@`, `%`, `:`, `(`, `?`,
`#`, spaces, empty segments, `.` or `..` segments).
- Routes are derived from `docs/manifest.json` the way the sites do
today: strip a leading `./`, then strip `README.md`, `index.md`, `.md`
in that order as plain text, trim slashes. The empty result is the docs
root.

### Wildcards

- A `source` may end in `/*`: `ai-coder/ai-bridge/*`. It matches the
prefix itself and every route beneath it.
- A `destination` may end in `/*` only when the source does. The part
the source's `*` matched (possibly empty) replaces the destination's
`*`; an empty match means the destination prefix itself
(`ai-coder/ai-gateway/*` with an empty match is `ai-coder/ai-gateway`).
- A wildcard source with a non-wildcard destination sends everything
under the prefix to that one page.
- No other pattern syntax exists. Consumers translate `*` into their own
matcher. For Next.js that is `:path([A-Za-z0-9._-]+)*`, never `(.*)`,
`.+`, or a bare `:name*`.

### Destinations

- `/` is the docs root of the same version (`/docs`, `/docs/@main`,
`/beta-docs/v2.37`, and so on). Pages with no suitable home redirect to
`/`.
- An internal destination always stays in the same version as the
request.
- A non-wildcard internal destination may end in one `#fragment`, where
the fragment matches `[A-Za-z0-9._-]+`:
`user-guides/workspace-access#web-ides`. The route part must still be a
live route. A wildcard destination and `/` never carry a fragment.
- An external destination is an `https://` URL whose host and path start
with an allowed prefix: `https://github.com/coder/` or
`https://coder.com/`. It contains no `:` after the scheme and no
parameter. A `*` is not allowed in an external destination.

### Order and precedence

- Rules apply in file order; the first match wins.
- A page always wins over a redirect: a rule only fires for a request
that is not a live page of that version. The checker rejects any rule
whose source matches a live route, so consumers that check redirects
before pages (Next.js static redirects) behave the same as those that
check on a miss.
- Matching ignores one trailing slash on the request.
- Consumers follow at most 3 chained hops and refuse loops. The checker
rejects chains so a valid file never needs this.

### Legacy entries

- During migration, consumers also accept a `source` or `destination`
that starts with `/docs/` (or equals `/docs`) and treat it as the route
after `/docs/` (or `/` for `/docs`). A legacy `:name*` or `:name(.*)`
trailing parameter is treated as `/*`. The coder/coder checker accepts
only the new form.
- A missing file and an empty array both mean no rules. A file that is
not a JSON array is an error.

## CI check (coder/coder `scripts/docsredirectscheck`)

Structural checks as today, adapted to the new form, plus a blocking
removed-route check:

1. **Inventory.** The set of routes that must keep working is the union
of:
   - the base branch's `docs/manifest.json` routes,
- every route in the live production page list for the PR's target
branch, read from `https://coder.com/beta-docs/<segment>/llms.txt` where
the segment is `@main` for `main` and `vX.Y` for `release/X.Y`, keeping
only links under that segment,
- every `source` in the base branch's `docs/redirects.json` (exact
sources as-is; wildcard sources contribute the prefix).
2. **Resolution.** Each inventory route must be a live route in the head
manifest, or be resolved by the head rules (with wildcard substitution)
to a live head route, `/`, or an allowed external URL.
3. **Failure message.** Name the route, the file it came from on the
base branch when known, and a ready-to-paste rule, for example
`{"source": "path/to/page", "destination": "path/to/another-page"}`,
pointing at `/` when no candidate is obvious.
4. **Live list failures.** A network error, non-200, or a list with no
routes is an infrastructure failure (exit 2) in CI, not a pass. Locally,
`-offline` skips the live list and says so.
5. **Known engine difference.** beta-docs currently serves dotted
segments with hyphens (`esr-2.24-2.29-upgrade` is served as
`esr-2-24-2-29-upgrade`). Until docs-engine fixes it, a live route that
equals a head route with `.` replaced by `-` in each segment counts as
present.

## Consumers

- **coder.com classic** (`scripts/generate-docs-redirects.ts`): reads
`main` (scoped to `/docs/@main/...`) and the mainline release branch
(unversioned `/docs/...`) as today, in the new format, and emits Next.js
redirects. Version coverage beyond those two is out of scope.
- **docs-engine beta**: each version's publish reads
`docs/redirects.json` from the commit it builds, validates it, and
stores the compiled rules with that version's archive. The router
applies a version's own rules on a miss inside that version. Versions
published before this change keep today's behavior.

</details>

---

🤖 Built with AI assistance.
2026-11-05 14:04:17 -05:00

376 lines
14 KiB
Go

// Command docsredirectscheck validates docs/redirects.json, the list of
// redirects the documentation websites apply for pages that have moved or been
// removed.
//
// Each rule is {"source", "destination", "permanent"?}. Sources and internal
// destinations are routes relative to the docs root, such as
// "admin/old-page". A source may end in "/*", which matches the prefix and
// everything beneath it, and a destination may then end in "/*" to carry the
// match across. The destination "/" is the docs root, and an external
// destination is an https URL under github.com/coder/ or coder.com/.
//
// It fails when:
//
// - the file is not a JSON array of well-formed rules, or uses the old
// absolute "/docs/..." form;
// - a source or destination is outside the supported syntax;
// - an internal destination is not a page in docs/manifest.json;
// - a source is a live docs page, which would hide a page readers can reach;
// - two rules share a source, a rule can never apply, or rules form a chain
// or a loop;
// - a route that worked on the base branch or on the live site no longer
// resolves, as a page or through a rule.
//
// The removed-route check compares against three sources: the base branch's
// docs/manifest.json, the live page list for the target branch, and the base
// branch's redirect sources. The live page list is read over the network, and
// failing to read it is an infrastructure failure, not a pass. It is skipped
// when the change leaves docs/manifest.json and docs/redirects.json as they are
// on the base branch, because the change cannot affect the result then, and
// when the target branch's docs version is not published yet. Pass -offline to
// skip it locally.
//
// A missing docs/redirects.json passes. Everything it checks about routes is
// derived from docs/manifest.json the same way the websites derive it.
//
// Usage:
//
// docsredirectscheck [-manifest path] [-redirects path] [-base rev] [-branch name] [-offline]
//
// Exit status is 0 when the file is valid or absent, 1 when it has problems,
// and 2 when the check itself could not run.
package main
import (
"bytes"
"context"
"errors"
"flag"
"fmt"
"io"
"io/fs"
"os"
"os/exec"
"os/signal"
"path/filepath"
"strings"
"time"
"golang.org/x/xerrors"
)
// Repository-relative paths compared across revisions. They are fixed,
// independent of the -manifest and -redirects flags, because git addresses
// files by their place in the repository.
const (
manifestRepoPath = "docs/manifest.json"
redirectsRepoPath = "docs/redirects.json"
)
type options struct {
manifestPath string
redirectsPath string
// base is the git revision the change under review started from. Empty
// means derive it from the environment.
base string
// branch is the branch the change targets, which selects the live page
// list. Empty means derive it from the environment.
branch string
// offline skips the live page list.
offline bool
git gitReader
fetch fetcher
// wait pauses between attempts to read the live page list. It returns
// early with an error when the context is done.
wait func(context.Context, time.Duration) error
getenv func(string) string
stdout io.Writer
stderr io.Writer
}
// gitReader is the slice of git the removed-route check needs, so tests can supply
// a fake repository.
type gitReader interface {
// HasRev reports whether rev resolves to a commit.
HasRev(rev string) bool
// Show returns the contents of path at rev. The error wraps
// fs.ErrNotExist when rev has no such file.
Show(rev, path string) ([]byte, error)
}
type execGit struct{}
// validRev rejects a revision git could read as an option, so a value passed
// with -base cannot add flags to the git commands below. The commands run
// without a shell and path is always one of the fixed repository paths.
func validRev(rev string) bool {
return rev != "" && !strings.HasPrefix(rev, "-")
}
func (execGit) HasRev(rev string) bool {
if !validRev(rev) {
return false
}
return exec.Command("git", "rev-parse", "--verify", "--quiet", rev+"^{commit}").Run() == nil //nolint:gosec // rev is checked by validRev and no shell is involved.
}
func (execGit) Show(rev, path string) ([]byte, error) {
if !validRev(rev) {
return nil, xerrors.Errorf("invalid revision %q", rev)
}
// ls-tree tells a missing file apart from other failures by its output
// rather than by git's localized error text. --full-tree makes path
// relative to the repository root, as it is for git show.
listed, err := exec.Command("git", "ls-tree", "--full-tree", "--name-only", rev, "--", path).Output() //nolint:gosec // rev is checked by validRev and no shell is involved.
if err != nil {
return nil, xerrors.Errorf("git ls-tree %s: %w", rev, err)
}
if len(bytes.TrimSpace(listed)) == 0 {
return nil, xerrors.Errorf("%s does not exist at %s: %w", path, rev, fs.ErrNotExist)
}
return exec.Command("git", "show", rev+":"+path).Output() //nolint:gosec // rev is checked by validRev and no shell is involved.
}
func main() {
o := options{
git: execGit{},
fetch: newHTTPFetcher(),
wait: sleepContext,
getenv: os.Getenv,
stdout: os.Stdout,
stderr: os.Stderr,
}
flag.StringVar(&o.manifestPath, "manifest", manifestRepoPath, "path to the docs manifest")
flag.StringVar(&o.redirectsPath, "redirects", redirectsRepoPath, "path to the redirects file")
flag.StringVar(&o.base, "base", "", "git revision the change started from, for the removed-route check (default: the pull request base in GitHub Actions)")
flag.StringVar(&o.branch, "branch", "", "branch the change targets, which selects the live page list (default: the pull request base or the pushed branch in GitHub Actions)")
flag.BoolVar(&o.offline, "offline", false, "skip the live page list, which needs network access")
flag.Parse()
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
code := run(ctx, o)
stop()
os.Exit(code)
}
func run(ctx context.Context, o options) int {
manifestData, err := os.ReadFile(o.manifestPath)
if err != nil {
_, _ = fmt.Fprintf(o.stderr, "cannot read manifest %s: %v\n", o.manifestPath, err)
return 2
}
m, err := parseManifest(manifestData)
if err != nil {
_, _ = fmt.Fprintf(o.stderr, "invalid manifest %s: %v\n", o.manifestPath, err)
return 2
}
live := liveRoutes(m)
var (
rules []rule
code int
)
redirectsData, err := os.ReadFile(o.redirectsPath)
switch {
case errors.Is(err, fs.ErrNotExist):
_, _ = fmt.Fprintf(o.stdout, "%s not found; nothing to validate\n", o.redirectsPath)
case err != nil:
_, _ = fmt.Fprintf(o.stderr, "cannot read %s: %v\n", o.redirectsPath, err)
return 2
default:
var problems []problem
rules, problems = checkRedirects(redirectsData, live)
if len(problems) > 0 {
for _, p := range problems {
_, _ = fmt.Fprintln(o.stderr, formatProblem(o.redirectsPath, p))
}
_, _ = fmt.Fprintf(o.stderr, "%s: %d %s found\n", o.redirectsPath, len(problems), plural(len(problems), "problem", "problems"))
code = 1
} else {
_, _ = fmt.Fprintf(o.stdout, "%s: %d %s OK\n", o.redirectsPath, len(rules), plural(len(rules), "rule", "rules"))
}
}
switch checkRemovedRoutes(ctx, o, docsFiles{manifest: manifestData, redirects: redirectsData}, live, rules) {
case 2:
return 2
case 1:
code = 1
}
return code
}
func plural(n int, one, many string) string {
if n == 1 {
return one
}
return many
}
// annotation formats a GitHub Actions workflow command. The message must stay
// on one line, so the characters the runner treats as delimiters are escaped.
func annotation(level, file, title, message string) string {
escapeData := strings.NewReplacer("%", "%25", "\r", "%0D", "\n", "%0A")
escapeProperty := strings.NewReplacer("%", "%25", "\r", "%0D", "\n", "%0A", ":", "%3A", ",", "%2C")
return fmt.Sprintf("::%s file=%s,title=%s::%s", level, escapeProperty.Replace(file), escapeProperty.Replace(title), escapeData.Replace(message))
}
// targetBranch returns the branch the change targets: -branch, the pull
// request base, or the pushed branch.
func targetBranch(o options) string {
for _, b := range []string{o.branch, o.getenv("GITHUB_BASE_REF"), o.getenv("GITHUB_REF_NAME")} {
if b != "" {
return b
}
}
return ""
}
// docsFiles holds the contents of docs/manifest.json and docs/redirects.json
// at one revision. A missing redirects file is nil.
type docsFiles struct {
manifest []byte
redirects []byte
}
// equal reports whether two revisions have the same docs files, ignoring line
// endings so a CRLF checkout compares equal to the LF blob in git.
func (d docsFiles) equal(other docsFiles) bool {
normalize := func(b []byte) []byte { return bytes.ReplaceAll(b, []byte("\r\n"), []byte("\n")) }
return bytes.Equal(normalize(d.manifest), normalize(other.manifest)) &&
bytes.Equal(normalize(d.redirects), normalize(other.redirects))
}
// checkRemovedRoutes fails when a route that worked before the change no
// longer resolves. Routes that must keep working are the base branch's
// manifest routes, the live page list of the target branch, and the base
// branch's redirect sources. Each part that cannot be gathered is skipped with
// a note, except a live page list that cannot be read, which is an
// infrastructure failure (exit status 2) and never a pass.
//
// The live page list is not read when the change leaves the docs files as they
// are on the base branch: the result would be the same as on the base branch,
// so a coder.com outage does not block changes that do not touch the docs.
func checkRemovedRoutes(ctx context.Context, o options, head docsFiles, live routeSet, rules []rule) int {
note := func(format string, args ...any) {
_, _ = fmt.Fprintf(o.stdout, "removed-route check: "+format+"\n", args...)
}
inv := inventory{}
var (
baseLive routeSet
unchanged bool
)
base := o.base
if base == "" && o.getenv("GITHUB_BASE_REF") != "" {
// On a pull request, actions/checkout checks out the merge commit, whose
// first parent is exactly the base the change merges into.
base = "HEAD^1"
}
switch {
case base == "":
note("base branch not compared: not a pull request run (pass -base <rev> to enable)")
case !o.git.HasRev(base):
note("base branch not compared: revision %q is not available (shallow checkout?)", base)
default:
var baseFiles *docsFiles
baseLive, baseFiles = addBaseRoutes(o, base, inv, note)
unchanged = baseFiles != nil && head.equal(*baseFiles)
}
switch {
case o.offline:
note("live page list skipped (-offline)")
case unchanged:
note("live page list skipped: %s and %s are unchanged from %s", manifestRepoPath, redirectsRepoPath, base)
default:
branch := targetBranch(o)
segment, ok := versionSegment(branch)
if !ok {
if branch == "" {
note("live page list skipped: cannot tell the target branch (pass -branch main or -branch release/X.Y)")
} else {
note("live page list skipped: branch %q has no published docs version", branch)
}
break
}
url := fmt.Sprintf(liveListURLFormat, segment)
routes, err := fetchLiveRoutes(ctx, o.fetch, url, segment, o.wait)
if errors.Is(err, errNotPublished) {
note("live page list skipped: %s answered 404, so docs version %s is not published yet", url, segment)
break
}
if err != nil {
_, _ = fmt.Fprintf(o.stderr, "cannot read the live page list, so the removed-route check could not run: %v\n", err)
_, _ = fmt.Fprintln(o.stderr, "Re-run the job. Use -offline to skip the live page list when running locally.")
return 2
}
for _, route := range routes {
inv.add(route, "the live page list "+url)
}
}
missing := missingRoutes(inv, newResolver(live, baseLive, rules))
if len(missing) == 0 {
if len(inv) > 0 {
note("%d %s", len(inv), plural(len(inv), "route still resolves", "routes still resolve"))
}
return 0
}
actions := o.getenv("GITHUB_ACTIONS") == "true"
redirectsFile := filepath.ToSlash(redirectsRepoPath)
for _, m := range missing {
msg := m.message()
_, _ = fmt.Fprintln(o.stderr, msg)
if actions {
_, _ = fmt.Fprintln(o.stdout, annotation("error", redirectsFile, "Docs route no longer resolves", msg))
}
}
_, _ = fmt.Fprintf(o.stderr, "%d docs %s. To keep the old URLs working, paste these rules at the top of %s:\n", len(missing), plural(len(missing), "route no longer resolves", "routes no longer resolve"), redirectsFile)
for _, m := range missing {
_, _ = fmt.Fprintf(o.stderr, " %s,\n", m.rule())
}
_, _ = fmt.Fprintln(o.stderr, `Point a rule at "/" or another page when its page was removed on purpose.`)
return 1
}
// addBaseRoutes adds the base branch's manifest routes and redirect sources to
// the inventory. It returns the base branch's live routes, or nil when the
// base manifest could not be read, and the base branch's docs files, or nil
// when either could not be read.
func addBaseRoutes(o options, base string, inv inventory, note func(string, ...any)) (routeSet, *docsFiles) {
manifestData, err := o.git.Show(base, manifestRepoPath)
if err != nil {
note("base manifest not compared: cannot read %s at %s: %v", manifestRepoPath, base, err)
return nil, nil
}
baseManifest, err := parseManifest(manifestData)
if err != nil {
note("base manifest not compared: %s at %s is not a valid manifest: %v", manifestRepoPath, base, err)
return nil, nil
}
baseLive := liveRoutes(baseManifest)
for route, file := range baseLive {
origin := manifestRepoPath + " on the base branch"
if file != "" {
origin = file + " on the base branch"
}
inv.add(route, origin)
}
files := docsFiles{manifest: manifestData}
redirectsData, err := o.git.Show(base, redirectsRepoPath)
switch {
case errors.Is(err, fs.ErrNotExist):
// A missing redirects file at the base means no rules yet.
case err != nil:
note("base redirect sources not compared: cannot read %s at %s: %v", redirectsRepoPath, base, err)
return baseLive, nil
default:
files.redirects = redirectsData
for _, route := range baseRuleSources(redirectsData) {
inv.add(route, "a rule in "+redirectsRepoPath+" on the base branch")
}
}
return baseLive, &files
}