4 Commits
Author SHA1 Message Date
Nick Vigilante c95b908705 ci: make docs redirects relative and block removed routes (#30805)
## Summary

Moves `docs/redirects.json` to relative routes and makes
`lint/docs-redirects` fail when a PR removes a live docs route without a
redirect.
coder.com and docs-engine build each version's redirects from that
branch's file, so this file becomes the only place contributors add docs
redirects.

Part of DOCS-1290; tracked in DOCS-1296.

## Changes

- New rule format: `{"source": "path/to/page", "destination":
"path/to/another-page"}`, relative to the docs root of the branch's
version.
Trailing `/*` is the only wildcard, `/` is the version's docs root, one
`#fragment` is allowed on exact destinations, and external destinations
are limited to `https://github.com/coder/` and `https://coder.com/`.
The old `/docs/...` form is rejected with the converted rule in the
message.
- Removed-route check now blocks.
Every route in the base manifest, the live page list for the target
branch (`coder.com/beta-docs/<version>/llms.txt`), and the base file's
sources must be a page or resolve through a rule.
The failure names the old file and prints a paste-ready rule, suggesting
a renamed sibling, a page with the same last segment, or `/`.
  An unreadable live list exits 2 in CI; `-offline` skips it locally.
Links under `/docs` count as well, so the check keeps working if the
list URL redirects there after the move to `/docs`.
- `docs/redirects.json` on `main`: 304 rules.
It carries the coder.com docs rules that 404 under `/docs/@main` today,
plus the 144 `@main`-only rules coder.com holds, with three sibling sets
condensed into wildcards (`networking/*`, `admin/notifications/*`,
`guides/*`).
- README, content guidelines, and agent guidance describe the new
format.

## Testing

- `go test ./scripts/docsredirectscheck/...` (also `-race`),
`golangci-lint`, and the pre-commit hook.
- `go run ./scripts/docsredirectscheck -base origin/main -branch main`
online, after merging current `main`: 304 rules OK, 497 routes resolve.
- Renaming `admin/integrations/istio.md` without a rule fails with
`{"source": "admin/integrations/istio", "destination":
"admin/integrations/istio-mesh"}`; adding that rule passes.
- coder.com's `generateRedirects` with this file and `main`'s manifest:
0 dropped; 336 of 336 sampled old URLs resolve to the expected page with
Next's matcher.
- `curl` on every destination under `coder.com/docs/@main`: all 200.
- Live baseline before merge: 185 of 336 sampled `/docs/@main` URLs
return 404.
- The check runs on Linux in `ci.yaml`'s `lint` job; the package's unit
tests also run on macOS and Windows in `test-go-pg`.

## Merge order

Merge after the coder.com generator (coder/coder.com,
`vigilante/docs-redirects-relative`) is deployed, otherwise coder.com
drops these rules with warnings until it is.
The docs-engine change (coder/docs-engine,
`vigilante/version-redirects`) can land before or after.
Then merge the `release/2.38` backport.

<details>
<summary>Contract shared by coder/coder, coder.com, and
docs-engine</summary>

# Docs redirects contract (coder/coder owns the data; coder.com and
docs-engine consume it)

## Ownership

- Every redirect for a moved, renamed, or removed docs page lives in
`coder/coder` `docs/redirects.json`, one file per branch (`main`,
`release/X.Y`).
- A branch's file applies only to that branch's version of the docs.
`main`'s file applies to `@main`; `release/X.Y`'s file applies to vX.Y,
and to unversioned `/docs` while X.Y is mainline.
- URL-scheme rules (`/docs/v1`, `/docs/v2/...`, `/docs/coder-oss/...`,
`/docs/coder/...`, `/docs/code-server/...`, classic `/@v<line>` forms)
are not content moves. They stay in the consumer that owns the URL
scheme, as code or a static engine-owned file.
- Nobody adds content redirects in coder.com or docs-engine. Existing
ones there are removed only after the matching coder/coder rules are
verified live for that version (later phase, not this change).

## File format

`docs/redirects.json` is a JSON array. Each entry is an object with
exactly these fields:

| Field | Required | Meaning |

|---------------|----------|-------------------------------------------------------------------------|
| `source` | yes | A route relative to the docs root of this version |
| `destination` | yes | A route relative to the docs root, `/` for the
docs root, or an allowed `https://` URL |
| `permanent` | no | Boolean, default `true`. `true` answers 308,
`false` answers 307 |

Unknown fields are an error.

### Routes

- A route is the URL path below `/docs` with no version, no leading
slash, no trailing slash, and no `.md`:
`admin/templates/extending-templates`.
- A route is one or more segments joined by `/`. Each segment matches
`[A-Za-z0-9._-]+`. Nothing else is allowed (no `@`, `%`, `:`, `(`, `?`,
`#`, spaces, empty segments, `.` or `..` segments).
- Routes are derived from `docs/manifest.json` the way the sites do
today: strip a leading `./`, then strip `README.md`, `index.md`, `.md`
in that order as plain text, trim slashes. The empty result is the docs
root.

### Wildcards

- A `source` may end in `/*`: `ai-coder/ai-bridge/*`. It matches the
prefix itself and every route beneath it.
- A `destination` may end in `/*` only when the source does. The part
the source's `*` matched (possibly empty) replaces the destination's
`*`; an empty match means the destination prefix itself
(`ai-coder/ai-gateway/*` with an empty match is `ai-coder/ai-gateway`).
- A wildcard source with a non-wildcard destination sends everything
under the prefix to that one page.
- No other pattern syntax exists. Consumers translate `*` into their own
matcher. For Next.js that is `:path([A-Za-z0-9._-]+)*`, never `(.*)`,
`.+`, or a bare `:name*`.

### Destinations

- `/` is the docs root of the same version (`/docs`, `/docs/@main`,
`/beta-docs/v2.37`, and so on). Pages with no suitable home redirect to
`/`.
- An internal destination always stays in the same version as the
request.
- A non-wildcard internal destination may end in one `#fragment`, where
the fragment matches `[A-Za-z0-9._-]+`:
`user-guides/workspace-access#web-ides`. The route part must still be a
live route. A wildcard destination and `/` never carry a fragment.
- An external destination is an `https://` URL whose host and path start
with an allowed prefix: `https://github.com/coder/` or
`https://coder.com/`. It contains no `:` after the scheme and no
parameter. A `*` is not allowed in an external destination.

### Order and precedence

- Rules apply in file order; the first match wins.
- A page always wins over a redirect: a rule only fires for a request
that is not a live page of that version. The checker rejects any rule
whose source matches a live route, so consumers that check redirects
before pages (Next.js static redirects) behave the same as those that
check on a miss.
- Matching ignores one trailing slash on the request.
- Consumers follow at most 3 chained hops and refuse loops. The checker
rejects chains so a valid file never needs this.

### Legacy entries

- During migration, consumers also accept a `source` or `destination`
that starts with `/docs/` (or equals `/docs`) and treat it as the route
after `/docs/` (or `/` for `/docs`). A legacy `:name*` or `:name(.*)`
trailing parameter is treated as `/*`. The coder/coder checker accepts
only the new form.
- A missing file and an empty array both mean no rules. A file that is
not a JSON array is an error.

## CI check (coder/coder `scripts/docsredirectscheck`)

Structural checks as today, adapted to the new form, plus a blocking
removed-route check:

1. **Inventory.** The set of routes that must keep working is the union
of:
   - the base branch's `docs/manifest.json` routes,
- every route in the live production page list for the PR's target
branch, read from `https://coder.com/beta-docs/<segment>/llms.txt` where
the segment is `@main` for `main` and `vX.Y` for `release/X.Y`, keeping
only links under that segment,
- every `source` in the base branch's `docs/redirects.json` (exact
sources as-is; wildcard sources contribute the prefix).
2. **Resolution.** Each inventory route must be a live route in the head
manifest, or be resolved by the head rules (with wildcard substitution)
to a live head route, `/`, or an allowed external URL.
3. **Failure message.** Name the route, the file it came from on the
base branch when known, and a ready-to-paste rule, for example
`{"source": "path/to/page", "destination": "path/to/another-page"}`,
pointing at `/` when no candidate is obvious.
4. **Live list failures.** A network error, non-200, or a list with no
routes is an infrastructure failure (exit 2) in CI, not a pass. Locally,
`-offline` skips the live list and says so.
5. **Known engine difference.** beta-docs currently serves dotted
segments with hyphens (`esr-2.24-2.29-upgrade` is served as
`esr-2-24-2-29-upgrade`). Until docs-engine fixes it, a live route that
equals a head route with `.` replaced by `-` in each segment counts as
present.

## Consumers

- **coder.com classic** (`scripts/generate-docs-redirects.ts`): reads
`main` (scoped to `/docs/@main/...`) and the mainline release branch
(unversioned `/docs/...`) as today, in the new format, and emits Next.js
redirects. Version coverage beyond those two is out of scope.
- **docs-engine beta**: each version's publish reads
`docs/redirects.json` from the commit it builds, validates it, and
stores the compiled rules with that version's archive. The router
applies a version's own rules on a miss inside that version. Versions
published before this change keep today's behavior.

</details>

---

🤖 Built with AI assistance.
2026-11-05 14:04:17 -05:00
Jon Ayers 63b5c4d64a fix: drop the /docs/about live-route assertion from TestRepositoryManifest 2026-11-04 21:37:07 -06:00
Nick Vigilante 35b3052ea8 ci(scripts/docsredirectscheck): reject redirects an earlier rule swallows (#30383)
## Summary

Redirects apply in file order and the first match wins, so rules have to
run from most specific to least.
Nothing enforced that, so a wildcard placed above a narrower rule inside
its subtree silently swallowed it.

## Problem

Given these two rules in that order, the second never applies:

```json
[
  { "source": "/docs/retired/:section(.*)", "destination": "/docs/reference/:section" },
  { "source": "/docs/retired/foo",          "destination": "/docs/about" }
]
```

A reader asking for `/docs/retired/foo` is sent to `/docs/reference/foo`
instead of `/docs/about`.
If `/docs/reference/foo` happens to exist, every existing check passes:
the destination resolves, there is no duplicate source, no chain, and no
loop.
The rule is simply dead, and the only symptom is readers landing on the
wrong page.

`docs/redirects.json` is empty today, so this adds the guard before the
file gets its first wildcard rather than after.

## Fix

Compare each rule against the ones before it and fail when an earlier
source matches every URL a later source would.

Coverage is decided on path boundaries, not string prefixes, so
`/docs/retired/:path*` does not cover `/docs/retired-too/page`.
Two cases stay legal on purpose:

- An exact rule above a wildcard on the same prefix, which claims the
bare prefix and leaves the subtree to the wildcard.
- A regex wildcard above an exact rule on its bare prefix, since
`/:name(.*)` never matches the prefix itself.

Rules that share a source are still reported as duplicates, not as
unreachable, so one mistake produces one message.

## Changes

- `rules.go`: add `pattern.covers` and `checkOrder`, called from
`checkAcrossRules`.
- `README.md`: document the ordering requirement and the new failure.
- `main_test.go`: 11 cases covering each pattern pairing, both
orderings, the path-boundary case, reporting once against the first
coverer, and the duplicate interaction.

## Testing

- `go test ./scripts/docsredirectscheck/` passes;
`TestCheckRedirectsOrder` runs all 11 subtests.
- `make lint/docs-redirects` passes against the committed
`docs/redirects.json`.
- `gofmt -l` and `go vet` are clean on the package.
- Ran the binary against the example above: it fails with
`unreachable: rule [0] (source "/docs/retired/:section(.*)") comes first
and matches every URL this rule would; order rules from most specific to
least`,
  exits 1, and passes with exit 0 once the two rules are swapped.

<details>
<summary>Why this check, and what it is a prerequisite for</summary>

`docs/redirects.json` is validated in CI and triggers a docs rebuild,
but no site reads it yet.
The redirects actually served live in two other hand-maintained files:
`coder.com/redirects.json` (297 rules, 173 under `/docs`) and
`docs-engine`'s `apps/router/redirects.json` (221 rules).

The plan is to make `docs/redirects.json` the single authored source and
have docs-engine compile it per version, so `apps/router/redirects.json`
becomes generated output.
Classifying the router's 221 rules showed all of them are same-version
moves once the version prefix is stripped, and that expanding them into
this file's narrow grammar yields 279 authored rules.

That expansion is what makes this check necessary now.
Collapsing many exact rules into wildcards is an engine optimization,
and the authored file will carry both wildcards and exact rules in the
same subtree, which is exactly the shape that can silently swallow a
rule.
Adding the guard while the file is still empty means the migration
cannot introduce an unreachable rule without CI saying so.

</details>

---

Resolves DOCS-1188.

Opened by Coder Agents on behalf of @nickvigilante.

<details>
<summary>CI note: one unrelated flake</summary>

`test-go-pg (ubuntu-latest)` failed on `coderd/x/chatd
TestChatTurnStopsAtStepLimit/Default`.
This PR changes three files under `scripts/docsredirectscheck` and
nothing in `coderd/`, so the failure is unrelated to the diff.
`lint`, `gen`, `fmt`, `check-build`, `test-go-race-pg`, `test-go-pg-17`,
`test-e2e`, and `test-go-pg` on macOS and Windows all pass.

The first run of this PR also had `changes`, `cla`, and `release-labels`
cancelled rather than failed, all three at exactly 15m03s, which skipped
the Go jobs entirely. A `--failed` rerun produced the results above.

</details>
2026-10-08 12:26:20 -04:00
Nick Vigilante bb58916e6c ci: validate docs/redirects.json (#30209)
Adds `docsredirectscheck`, a lint for `docs/redirects.json`, the list of
redirects the documentation website applies for pages that have moved or
been removed. It runs as `make lint/docs-redirects`, part of `make lint`
and `make lint-light`, so a bad redirect is caught in the pull request
that adds it. The file does not exist yet, and the check passes while it
is absent, so this can merge before or after the file does.

## What it checks

It fails on:

- a file that is not a JSON array of well-formed rules (`source`,
`destination`, optional boolean `permanent`, no unknown fields);
- syntax outside the supported subset: a source under `/docs/`, written
as an exact path, a trailing `/:name*`, or a trailing `/:name(.*)`, with
no version segment such as `/@main`;
- an internal destination that is not a page in `docs/manifest.json`, or
a pattern destination that points into a section with no pages or uses a
parameter the source does not define;
- a source that matches a live page, which would hide it;
- duplicate sources, chains (a destination that is another rule's
source), and loops.

It warns, and never fails, when a pull request removes routes from
`docs/manifest.json` while `docs/redirects.json` is unchanged and does
not already cover them. Pages are sometimes removed on purpose. The
comparison is by route rather than by file, so a rename that keeps its
URL (for example `admin/foo/index.md` to `admin/foo.md`) does not warn.
In GitHub Actions the warning is an annotation on the manifest. The base
is the first parent of the PR merge commit, the same approach
`scripts/check_emdash.sh` uses, and the warning is skipped with a note
when it is not available.

Routes are derived from `docs/manifest.json` the way the docs website
derives them, including its plain-suffix stripping of `README.md`,
`index.md`, and `.md`, so the check and the site agree about which pages
exist. `scripts/docsredirectscheck/README.md` documents the rule format
and the limits.

## Testing

- 128 Go tests pass under Go 1.26.5: route derivation, every failing and
passing rule shape, the absent-file pass, exit codes, and the warning
logic through a fake git reader.
- I checked that the tests can fail by breaking the code 16 ways (for
example dropping the `index.md` suffix, disabling the shadow check,
reporting loops twice). All 16 were caught. That run found one gap, a
missing test for "redirects file changed", which is now covered.
- Ran against the repository's real `docs/manifest.json`: passes with no
redirects file, and a file with one of each failure class reports every
problem with the rule index, the source, and how to fix it.
- Ran the warning against real git history: it warns for an older base,
stays silent at `HEAD`, and skips with a note for an unknown revision.
- Lint: `golangci-lint` v2.0.2 built with Go 1.26.5 against the
repository config reports 0 issues. `paralleltestctx`, `intxcheck`,
`typos` 1.46.1, `gofumpt` and `gci` (through `make fmt/go`), and `go
vet` pass.

CI: `lint` (which runs this check), `fmt`, `title`, and the Go and
end-to-end test jobs pass. The `gen` job, and so `required`, fails on
this branch for a reason this PR does not touch: `main` is failing it
too. `docs/manifest/generated/cli.json` has not been regenerated for the
`coder sharing` commands that landed in the latest commit on `main`. It
should clear once `main` is fixed and merged into this branch.

Not run locally: the full `make lint` and `make pre-commit`. The local
hooks are opt-in and were off. The annotation output has been exercised
by unit tests and a local run with `GITHUB_ACTIONS=true`, but not yet on
a real pull request merge commit; this PR does not change the manifest,
so it will not exercise it either. No workflow change is needed, because
the `lint` job already runs `make lint` for any change that is not
offlinedocs-only.

---
🤖 Built with AI assistance.
2026-10-02 14:27:44 -04:00