## Summary
Moves `docs/redirects.json` to relative routes and makes
`lint/docs-redirects` fail when a PR removes a live docs route without a
redirect.
coder.com and docs-engine build each version's redirects from that
branch's file, so this file becomes the only place contributors add docs
redirects.
Part of DOCS-1290; tracked in DOCS-1296.
## Changes
- New rule format: `{"source": "path/to/page", "destination":
"path/to/another-page"}`, relative to the docs root of the branch's
version.
Trailing `/*` is the only wildcard, `/` is the version's docs root, one
`#fragment` is allowed on exact destinations, and external destinations
are limited to `https://github.com/coder/` and `https://coder.com/`.
The old `/docs/...` form is rejected with the converted rule in the
message.
- Removed-route check now blocks.
Every route in the base manifest, the live page list for the target
branch (`coder.com/beta-docs/<version>/llms.txt`), and the base file's
sources must be a page or resolve through a rule.
The failure names the old file and prints a paste-ready rule, suggesting
a renamed sibling, a page with the same last segment, or `/`.
An unreadable live list exits 2 in CI; `-offline` skips it locally.
Links under `/docs` count as well, so the check keeps working if the
list URL redirects there after the move to `/docs`.
- `docs/redirects.json` on `main`: 304 rules.
It carries the coder.com docs rules that 404 under `/docs/@main` today,
plus the 144 `@main`-only rules coder.com holds, with three sibling sets
condensed into wildcards (`networking/*`, `admin/notifications/*`,
`guides/*`).
- README, content guidelines, and agent guidance describe the new
format.
## Testing
- `go test ./scripts/docsredirectscheck/...` (also `-race`),
`golangci-lint`, and the pre-commit hook.
- `go run ./scripts/docsredirectscheck -base origin/main -branch main`
online, after merging current `main`: 304 rules OK, 497 routes resolve.
- Renaming `admin/integrations/istio.md` without a rule fails with
`{"source": "admin/integrations/istio", "destination":
"admin/integrations/istio-mesh"}`; adding that rule passes.
- coder.com's `generateRedirects` with this file and `main`'s manifest:
0 dropped; 336 of 336 sampled old URLs resolve to the expected page with
Next's matcher.
- `curl` on every destination under `coder.com/docs/@main`: all 200.
- Live baseline before merge: 185 of 336 sampled `/docs/@main` URLs
return 404.
- The check runs on Linux in `ci.yaml`'s `lint` job; the package's unit
tests also run on macOS and Windows in `test-go-pg`.
## Merge order
Merge after the coder.com generator (coder/coder.com,
`vigilante/docs-redirects-relative`) is deployed, otherwise coder.com
drops these rules with warnings until it is.
The docs-engine change (coder/docs-engine,
`vigilante/version-redirects`) can land before or after.
Then merge the `release/2.38` backport.
<details>
<summary>Contract shared by coder/coder, coder.com, and
docs-engine</summary>
# Docs redirects contract (coder/coder owns the data; coder.com and
docs-engine consume it)
## Ownership
- Every redirect for a moved, renamed, or removed docs page lives in
`coder/coder` `docs/redirects.json`, one file per branch (`main`,
`release/X.Y`).
- A branch's file applies only to that branch's version of the docs.
`main`'s file applies to `@main`; `release/X.Y`'s file applies to vX.Y,
and to unversioned `/docs` while X.Y is mainline.
- URL-scheme rules (`/docs/v1`, `/docs/v2/...`, `/docs/coder-oss/...`,
`/docs/coder/...`, `/docs/code-server/...`, classic `/@v<line>` forms)
are not content moves. They stay in the consumer that owns the URL
scheme, as code or a static engine-owned file.
- Nobody adds content redirects in coder.com or docs-engine. Existing
ones there are removed only after the matching coder/coder rules are
verified live for that version (later phase, not this change).
## File format
`docs/redirects.json` is a JSON array. Each entry is an object with
exactly these fields:
| Field | Required | Meaning |
|---------------|----------|-------------------------------------------------------------------------|
| `source` | yes | A route relative to the docs root of this version |
| `destination` | yes | A route relative to the docs root, `/` for the
docs root, or an allowed `https://` URL |
| `permanent` | no | Boolean, default `true`. `true` answers 308,
`false` answers 307 |
Unknown fields are an error.
### Routes
- A route is the URL path below `/docs` with no version, no leading
slash, no trailing slash, and no `.md`:
`admin/templates/extending-templates`.
- A route is one or more segments joined by `/`. Each segment matches
`[A-Za-z0-9._-]+`. Nothing else is allowed (no `@`, `%`, `:`, `(`, `?`,
`#`, spaces, empty segments, `.` or `..` segments).
- Routes are derived from `docs/manifest.json` the way the sites do
today: strip a leading `./`, then strip `README.md`, `index.md`, `.md`
in that order as plain text, trim slashes. The empty result is the docs
root.
### Wildcards
- A `source` may end in `/*`: `ai-coder/ai-bridge/*`. It matches the
prefix itself and every route beneath it.
- A `destination` may end in `/*` only when the source does. The part
the source's `*` matched (possibly empty) replaces the destination's
`*`; an empty match means the destination prefix itself
(`ai-coder/ai-gateway/*` with an empty match is `ai-coder/ai-gateway`).
- A wildcard source with a non-wildcard destination sends everything
under the prefix to that one page.
- No other pattern syntax exists. Consumers translate `*` into their own
matcher. For Next.js that is `:path([A-Za-z0-9._-]+)*`, never `(.*)`,
`.+`, or a bare `:name*`.
### Destinations
- `/` is the docs root of the same version (`/docs`, `/docs/@main`,
`/beta-docs/v2.37`, and so on). Pages with no suitable home redirect to
`/`.
- An internal destination always stays in the same version as the
request.
- A non-wildcard internal destination may end in one `#fragment`, where
the fragment matches `[A-Za-z0-9._-]+`:
`user-guides/workspace-access#web-ides`. The route part must still be a
live route. A wildcard destination and `/` never carry a fragment.
- An external destination is an `https://` URL whose host and path start
with an allowed prefix: `https://github.com/coder/` or
`https://coder.com/`. It contains no `:` after the scheme and no
parameter. A `*` is not allowed in an external destination.
### Order and precedence
- Rules apply in file order; the first match wins.
- A page always wins over a redirect: a rule only fires for a request
that is not a live page of that version. The checker rejects any rule
whose source matches a live route, so consumers that check redirects
before pages (Next.js static redirects) behave the same as those that
check on a miss.
- Matching ignores one trailing slash on the request.
- Consumers follow at most 3 chained hops and refuse loops. The checker
rejects chains so a valid file never needs this.
### Legacy entries
- During migration, consumers also accept a `source` or `destination`
that starts with `/docs/` (or equals `/docs`) and treat it as the route
after `/docs/` (or `/` for `/docs`). A legacy `:name*` or `:name(.*)`
trailing parameter is treated as `/*`. The coder/coder checker accepts
only the new form.
- A missing file and an empty array both mean no rules. A file that is
not a JSON array is an error.
## CI check (coder/coder `scripts/docsredirectscheck`)
Structural checks as today, adapted to the new form, plus a blocking
removed-route check:
1. **Inventory.** The set of routes that must keep working is the union
of:
- the base branch's `docs/manifest.json` routes,
- every route in the live production page list for the PR's target
branch, read from `https://coder.com/beta-docs/<segment>/llms.txt` where
the segment is `@main` for `main` and `vX.Y` for `release/X.Y`, keeping
only links under that segment,
- every `source` in the base branch's `docs/redirects.json` (exact
sources as-is; wildcard sources contribute the prefix).
2. **Resolution.** Each inventory route must be a live route in the head
manifest, or be resolved by the head rules (with wildcard substitution)
to a live head route, `/`, or an allowed external URL.
3. **Failure message.** Name the route, the file it came from on the
base branch when known, and a ready-to-paste rule, for example
`{"source": "path/to/page", "destination": "path/to/another-page"}`,
pointing at `/` when no candidate is obvious.
4. **Live list failures.** A network error, non-200, or a list with no
routes is an infrastructure failure (exit 2) in CI, not a pass. Locally,
`-offline` skips the live list and says so.
5. **Known engine difference.** beta-docs currently serves dotted
segments with hyphens (`esr-2.24-2.29-upgrade` is served as
`esr-2-24-2-29-upgrade`). Until docs-engine fixes it, a live route that
equals a head route with `.` replaced by `-` in each segment counts as
present.
## Consumers
- **coder.com classic** (`scripts/generate-docs-redirects.ts`): reads
`main` (scoped to `/docs/@main/...`) and the mainline release branch
(unversioned `/docs/...`) as today, in the new format, and emits Next.js
redirects. Version coverage beyond those two is out of scope.
- **docs-engine beta**: each version's publish reads
`docs/redirects.json` from the commit it builds, validates it, and
stores the compiled rules with that version's archive. The router
applies a version's own rules on a miss inside that version. Versions
published before this change keep today's behavior.
</details>
---
🤖 Built with AI assistance.
355 lines
11 KiB
Go
355 lines
11 KiB
Go
package main
|
|
|
|
import (
|
|
"context"
|
|
"encoding/json"
|
|
"fmt"
|
|
"io"
|
|
"net/http"
|
|
"regexp"
|
|
"slices"
|
|
"strings"
|
|
"time"
|
|
|
|
"golang.org/x/xerrors"
|
|
)
|
|
|
|
const (
|
|
// liveListURLFormat is the production page list of one docs version. The
|
|
// segment is "@main" for main and "vX.Y" for release/X.Y.
|
|
liveListURLFormat = "https://coder.com/beta-docs/%s/llms.txt"
|
|
|
|
// userAgent identifies the check to the site. Its CDN answers 403 to the
|
|
// default Go user agent in some cases.
|
|
userAgent = "coder-docsredirectscheck (+https://github.com/coder/coder)"
|
|
|
|
fetchAttempts = 4
|
|
fetchTimeout = 30 * time.Second
|
|
fetchMaxBytes = 32 << 20
|
|
maxRedirectHops = 3
|
|
)
|
|
|
|
// fetcher reads a URL, so tests never touch the network.
|
|
type fetcher interface {
|
|
// Get returns the HTTP status and body of url.
|
|
Get(ctx context.Context, url string) (status int, body []byte, err error)
|
|
}
|
|
|
|
type httpFetcher struct {
|
|
client *http.Client
|
|
}
|
|
|
|
func newHTTPFetcher() httpFetcher {
|
|
return httpFetcher{client: &http.Client{Timeout: fetchTimeout}}
|
|
}
|
|
|
|
func (f httpFetcher) Get(ctx context.Context, url string) (int, []byte, error) {
|
|
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
|
if err != nil {
|
|
return 0, nil, err
|
|
}
|
|
req.Header.Set("User-Agent", userAgent)
|
|
resp, err := f.client.Do(req)
|
|
if err != nil {
|
|
return 0, nil, err
|
|
}
|
|
defer resp.Body.Close()
|
|
body, err := io.ReadAll(io.LimitReader(resp.Body, fetchMaxBytes))
|
|
if err != nil {
|
|
return resp.StatusCode, nil, err
|
|
}
|
|
return resp.StatusCode, body, nil
|
|
}
|
|
|
|
// releaseBranchRe matches the release branches that have a docs version of their
|
|
// own.
|
|
var releaseBranchRe = regexp.MustCompile(`^release/(\d+\.\d+)$`)
|
|
|
|
// versionSegment returns the docs version segment the live page list of a
|
|
// target branch lives under, or false when the branch has no published docs.
|
|
func versionSegment(branch string) (string, bool) {
|
|
if branch == "main" {
|
|
return "@main", true
|
|
}
|
|
if m := releaseBranchRe.FindStringSubmatch(branch); m != nil {
|
|
return "v" + m[1], true
|
|
}
|
|
return "", false
|
|
}
|
|
|
|
// linkRe finds the target of a Markdown link.
|
|
var linkRe = regexp.MustCompile(`\]\(([^)\s]+)\)`)
|
|
|
|
// otherVersionRe matches the first path segment of a link into some version of
|
|
// the docs, which an unversioned link never has.
|
|
var otherVersionRe = regexp.MustCompile(`^(@[^/]+|v\d+(\.\d+)*)$`)
|
|
|
|
// liveListSiteBases are the base paths the docs site serves its versions
|
|
// under, in order of preference. The list URL may redirect from /beta-docs to
|
|
// /docs once the docs move there, and its links then use /docs.
|
|
var liveListSiteBases = []string{"/beta-docs", "/docs"}
|
|
|
|
// parseLiveList returns the routes of the links in an llms.txt body that point
|
|
// into the given version. The docs root is the empty route.
|
|
//
|
|
// The version that is served at a site's unversioned base path links its own
|
|
// pages without a version segment, so when no link carries the segment, the
|
|
// unversioned links count instead.
|
|
func parseLiveList(body []byte, segment string) []string {
|
|
var hrefs []string
|
|
for _, m := range linkRe.FindAllSubmatch(body, -1) {
|
|
href := strings.TrimPrefix(string(m[1]), "https://coder.com")
|
|
if i := strings.IndexAny(href, "?#"); i >= 0 {
|
|
href = href[:i]
|
|
}
|
|
hrefs = append(hrefs, href)
|
|
}
|
|
for _, base := range liveListSiteBases {
|
|
if routes := routesUnder(hrefs, base+"/"+segment); len(routes) > 0 {
|
|
return routes
|
|
}
|
|
}
|
|
for _, base := range liveListSiteBases {
|
|
unversioned := slices.DeleteFunc(routesUnder(hrefs, base), func(route string) bool {
|
|
first, _, _ := strings.Cut(route, "/")
|
|
return otherVersionRe.MatchString(first)
|
|
})
|
|
if len(unversioned) > 0 {
|
|
return unversioned
|
|
}
|
|
}
|
|
return nil
|
|
}
|
|
|
|
// routesUnder returns the routes of the hrefs at or below base, without
|
|
// duplicates and in order of appearance.
|
|
func routesUnder(hrefs []string, base string) []string {
|
|
var routes []string
|
|
seen := map[string]bool{}
|
|
for _, href := range hrefs {
|
|
if href != base && !strings.HasPrefix(href, base+"/") {
|
|
continue
|
|
}
|
|
route := convertPathToRoute(strings.TrimPrefix(href, base))
|
|
if route != "" && checkRoute(route) != "" {
|
|
continue
|
|
}
|
|
if !seen[route] {
|
|
seen[route] = true
|
|
routes = append(routes, route)
|
|
}
|
|
}
|
|
return routes
|
|
}
|
|
|
|
// errNotPublished means the site has no page list for a version, which is the
|
|
// case for a new release branch before its docs are published.
|
|
var errNotPublished = xerrors.New("docs version is not published")
|
|
|
|
// sleepContext waits for d, or returns early with the context's error when ctx
|
|
// is done first.
|
|
func sleepContext(ctx context.Context, d time.Duration) error {
|
|
t := time.NewTimer(d)
|
|
defer t.Stop()
|
|
select {
|
|
case <-ctx.Done():
|
|
return ctx.Err()
|
|
case <-t.C:
|
|
return nil
|
|
}
|
|
}
|
|
|
|
// fetchLiveRoutes reads the live page list of a version, retrying transient
|
|
// failures a bounded number of times. An empty list is an error. A 404 is not
|
|
// retried and returns an error that wraps errNotPublished.
|
|
func fetchLiveRoutes(ctx context.Context, f fetcher, url, segment string, wait func(context.Context, time.Duration) error) ([]string, error) {
|
|
var lastErr error
|
|
for attempt := 0; attempt < fetchAttempts; attempt++ {
|
|
if attempt > 0 {
|
|
if err := wait(ctx, time.Duration(1<<(attempt-1))*time.Second); err != nil {
|
|
return nil, xerrors.Errorf("cannot read %s: %w", url, err)
|
|
}
|
|
}
|
|
status, body, err := f.Get(ctx, url)
|
|
switch {
|
|
case err != nil:
|
|
lastErr = err
|
|
case status == http.StatusNotFound:
|
|
return nil, xerrors.Errorf("%s answered HTTP status 404: %w", url, errNotPublished)
|
|
case status != http.StatusOK:
|
|
lastErr = xerrors.Errorf("HTTP status %d", status)
|
|
default:
|
|
routes := parseLiveList(body, segment)
|
|
if len(routes) == 0 {
|
|
return nil, xerrors.Errorf("%s lists no pages of docs version %s", url, segment)
|
|
}
|
|
return routes, nil
|
|
}
|
|
}
|
|
return nil, xerrors.Errorf("cannot read %s after %d attempts: %w", url, fetchAttempts, lastErr)
|
|
}
|
|
|
|
// inventory is the set of routes that must keep working, each with where it
|
|
// came from.
|
|
type inventory map[string][]string
|
|
|
|
func (inv inventory) add(route, origin string) {
|
|
if route == "" {
|
|
// The docs root always exists.
|
|
return
|
|
}
|
|
if !slices.Contains(inv[route], origin) {
|
|
inv[route] = append(inv[route], origin)
|
|
}
|
|
}
|
|
|
|
// baseRuleSources returns the route each rule of a base branch's
|
|
// docs/redirects.json answers for: exact sources as written, and the prefix of
|
|
// a wildcard. It is lenient on purpose, because the base file may predate the
|
|
// current format. It reads the old absolute form, and skips what it cannot
|
|
// read.
|
|
func baseRuleSources(data []byte) []string {
|
|
var entries []struct {
|
|
Source string `json:"source"`
|
|
}
|
|
if err := json.Unmarshal(data, &entries); err != nil {
|
|
return nil
|
|
}
|
|
var routes []string
|
|
for _, e := range entries {
|
|
route := strings.TrimSuffix(convertLegacy(e.Source), "/*")
|
|
if checkRoute(route) == "" {
|
|
routes = append(routes, route)
|
|
}
|
|
}
|
|
return routes
|
|
}
|
|
|
|
// resolver decides whether a route still works in the head branch.
|
|
type resolver struct {
|
|
live routeSet
|
|
// hyphenated holds the live routes with "." replaced by "-".
|
|
hyphenated map[string]bool
|
|
rules []rule
|
|
// added holds the routes that are live in head and were not live in the
|
|
// base branch, which are the likely new homes of a renamed page. It is
|
|
// empty when there is no base manifest to compare with.
|
|
added []string
|
|
}
|
|
|
|
// newResolver builds a resolver for the head branch. baseLive is the base
|
|
// branch's live routes, or nil when they are not known.
|
|
func newResolver(live, baseLive routeSet, rules []rule) resolver {
|
|
r := resolver{live: live, hyphenated: live.hyphenated(), rules: rules}
|
|
if baseLive != nil {
|
|
for route := range live {
|
|
if route != "" && !baseLive.has(route) {
|
|
r.added = append(r.added, route)
|
|
}
|
|
}
|
|
slices.Sort(r.added)
|
|
}
|
|
return r
|
|
}
|
|
|
|
// isLive reports whether a route is a page of the head branch. beta-docs
|
|
// serves dotted segments with hyphens ("esr-2.24-2.29-upgrade" is served as
|
|
// "esr-2-24-2-29-upgrade"), so a route that equals a head route with "." replaced
|
|
// by "-" counts as present until the docs engine is fixed.
|
|
func (r resolver) isLive(route string) bool {
|
|
return r.live.has(route) || r.hyphenated[route]
|
|
}
|
|
|
|
// resolves reports whether a request for route ends on a page: it is a live
|
|
// route, or the head rules send it, within maxRedirectHops, to a live route,
|
|
// the docs root, or an allowed external URL. Every redirect counts as a hop,
|
|
// whatever its destination.
|
|
func (r resolver) resolves(route string) bool {
|
|
route = normalizeRoute(route)
|
|
for hops := 0; ; hops++ {
|
|
if r.isLive(route) {
|
|
return true
|
|
}
|
|
i := slices.IndexFunc(r.rules, func(rl rule) bool { return rl.matches(route) })
|
|
if i < 0 || !r.rules[i].dst.valid || hops == maxRedirectHops {
|
|
return false
|
|
}
|
|
kind, next := r.rules[i].target(route)
|
|
if kind != destRoute {
|
|
return true
|
|
}
|
|
route = next
|
|
}
|
|
}
|
|
|
|
// suggest picks a destination for a rule that restores route. In order of
|
|
// preference it is the one live page with the same last segment, the one route
|
|
// added by this change in the same parent section (a rename such as
|
|
// "admin/integrations/istio" to "admin/integrations/istio-mesh"), or the docs
|
|
// root when no page is an obvious home.
|
|
func (r resolver) suggest(route string) string {
|
|
last := func(s string) string {
|
|
s = strings.ReplaceAll(s, ".", "-")
|
|
return s[strings.LastIndex(s, "/")+1:]
|
|
}
|
|
var found []string
|
|
for candidate := range r.live {
|
|
if candidate != "" && last(candidate) == last(route) {
|
|
found = append(found, candidate)
|
|
}
|
|
}
|
|
if len(found) == 1 {
|
|
return found[0]
|
|
}
|
|
|
|
parent := func(s string) string {
|
|
i := strings.LastIndex(s, "/")
|
|
if i < 0 {
|
|
return ""
|
|
}
|
|
return s[:i]
|
|
}
|
|
found = found[:0]
|
|
for _, candidate := range r.added {
|
|
if parent(candidate) == parent(route) {
|
|
found = append(found, candidate)
|
|
}
|
|
}
|
|
if len(found) == 1 {
|
|
return found[0]
|
|
}
|
|
return "/"
|
|
}
|
|
|
|
// missingRoute is an inventory route that no longer works.
|
|
type missingRoute struct {
|
|
route string
|
|
origins []string
|
|
suggestion string
|
|
}
|
|
|
|
func (m missingRoute) rule() string {
|
|
return ruleJSON(m.route, m.suggestion, nil)
|
|
}
|
|
|
|
func (m missingRoute) message() string {
|
|
return fmt.Sprintf(
|
|
"docs route %q no longer resolves (from %s). Add a rule to docs/redirects.json, for example: %s",
|
|
m.route, strings.Join(m.origins, ", "), m.rule(),
|
|
)
|
|
}
|
|
|
|
// missingRoutes returns the inventory routes the head branch no longer serves
|
|
// or redirects, sorted by route.
|
|
func missingRoutes(inv inventory, r resolver) []missingRoute {
|
|
var missing []missingRoute
|
|
for route, origins := range inv {
|
|
if r.resolves(route) {
|
|
continue
|
|
}
|
|
missing = append(missing, missingRoute{route: route, origins: origins, suggestion: r.suggest(route)})
|
|
}
|
|
slices.SortFunc(missing, func(a, b missingRoute) int { return strings.Compare(a.route, b.route) })
|
|
return missing
|
|
}
|