Files
coder/scripts/docgenenv/sources.go
T
Nick Vigilante f1ed84ffab docs: author the sidebar in per-section YAML compiled to manifest.json (#30165)
## Summary

The docs sidebar is now written as YAML files under `docs/manifest/`,
one route and its direct children per file, and a new tool,
`scripts/docsmanifestgen`, compiles them into `docs/manifest.json`.
Moving two top-level sections is now a 2-line change in
`docs/manifest/index.yml` instead of a diff of hundreds of lines in a
2,600-line JSON file.
The compiled `docs/manifest.json` is byte-identical to `main`, so
coder.com, docs-preview, deploy-docs, and offlinedocs need no changes.

Refs
[DOCS-1131](https://linear.app/codercom/issue/DOCS-1131/author-the-docs-sidebar-in-per-section-yaml-compiled-to)

## Changes

- **Sources:** `docs/manifest/index.yml` holds `versions` and the
section files in sidebar order.
Each section file (`about.yml`, `admin.yml`, and so on) holds one
top-level route.
- **One level per file:** a file holds one route and its direct
children.
A child with children of its own, or with `children_from`, lives in its
own file in a directory named after the parent file, and the parent
lists it as `- include: admin/users.yml`.
For example, `get-started.yml` includes
`get-started/customize-your-template.yml`, which lists that page's
children.
  The 8 section files split into 53 source files.
- **`scripts/docsmanifestgen`:** `build` compiles the sources into
`docs/manifest.json`, `fmt` formats them, and `check` validates them.
- Keys must be known (a typo like `titel:` fails with a line number) and
appear in a fixed order in both the YAML and the JSON.
- Every `path` must exist, and every file under `docs/manifest/` must be
referenced from `index.yml`, an `include`, or a `children_from`.
- A grandchild in the same file fails with a message to move the child
into its own file, and an `include` entry can't set other keys or name a
file that is already used.
- A comment on the same line as a `title` or `description` is rejected,
because an unquoted ` #` would silently cut off the value.
- **Generated fragments:** `clidocgen` and `apidocgen` no longer rewrite
the manifest.
They write the Command Line and REST API children to
`docs/manifest/generated/*.json`, which the sources pull in with
`children_from`.
  The tool accepts fragments as YAML or JSON.
- The hand-written REST API descriptions and states moved into the YAML,
attached to generated routes by title.
A YAML entry that matches no generated route, or disagrees with one,
fails the build instead of being dropped silently.
- **Shared code:** loading, validation, and fragment merging live in
`scripts/docgenenv/sources.go`, so the generators and the tool share one
loader.
`docgenenv.Route` gains YAML tags and the source-only `ChildrenFrom` and
`Include` fields (`json:"-"`).
`LoadManifest` is removed because nothing reads the compiled JSON
anymore.
- **Makefile:**
- `docs/manifest.json` is built from the sources and fragments, which
replaces the `_gen/manifest-staging.json` handoff.
- Each fragment has its own rule, and the fragments are added to the
`.PRECIOUS`, `GEN_FILES`, and `gen/mark-fresh` lists.
  - `DOCS_MANIFEST_SOURCES` finds the source files recursively.
- New targets: `fmt/docs-manifest` (part of `fmt`), `lint/docs-manifest`
(part of `lint`), and `gen/docs-manifest`, the quick path after a
sidebar-only edit that skips regenerating the CLI and API fragments.
- **`.gitattributes`:** marks `docs/manifest.json` and
`docs/manifest/generated/*` as generated so GitHub collapses them in
diffs.
- **Guidance:** the content guidelines, the style guide,
`DOCS_STYLE_GUIDE.md`, the `write-docs` and `doc-check` skills,
`docs/README.md`, and `docs/tutorials/example-guide.md` now point to the
YAML sources instead of `docs/manifest.json`.

## Testing

- Split today's `docs/manifest.json` into the sources, rebuilt it, and
confirmed the output is byte-identical to `main`.
After splitting the sources to one level per file, the rebuilt
`docs/manifest.json` and a fresh `clidocgen` CLI fragment were still
byte-identical, and `LoadManifestSources` still finds the REST API and
Command Line routes through their includes.
After rebasing onto `main`, both commits rebuild `main`'s
`docs/manifest.json` byte for byte.
- Hand-edited `docs/manifest.json` and a YAML source, then rebuilt: the
YAML change persisted and the JSON edit was overwritten.
CI's `make -B gen` plus `check_unstaged.sh` fails on a committed hand
edit to the JSON.
- Deleted both fragments and ran `make docs/manifest.json` in a fresh
worktree, which regenerated them through swag and `clidocgen`.
`docs/manifest.json`, the fragments, and every generated CLI and API
page had no diff.
- Swapped User Guides and Administration in `index.yml`: 2 changed lines
in the source, 166 generated lines in `docs/manifest.json`.
  Swapping back restored the file exactly.
- `go test` for `scripts/docgenenv`, `scripts/docsmanifestgen`,
`scripts/clidocgen`, and `scripts/apidocgen/postprocess`.
The new tests cover merging, includes, every validation error,
idempotent formatting that keeps comments, and a check that the key
order matches the struct field order.
- `golangci-lint` on the changed packages, and the full `make
pre-commit` hook (gen, fmt, lint including `lint/docs-manifest`, and the
slim build).

## Notes

- Open PRs that edit `docs/manifest.json` will need to redo their
sidebar changes in the YAML sources after this merges.
- #30161 (merged) stopped escaping `&`, `<`, and `>` in
`docs/manifest.json`.
This PR carries that over: all manifest JSON escaping is in
`docgenenv.MarshalJSON`, which sets `enc.SetEscapeHTML(false)`, and
`Manifest.Marshal` from #30161 is gone because the generators no longer
write the manifest.
- #30129 (merged) flattened `admin/external-auth/index.md` and
`ai-coder/agents/tools/index.md` to single files, and the YAML sources
use the new paths.
- The CLI index and REST API page front matter still update only on a
full `make gen` when their sidebar metadata changes, as it did before.
That gap and the overall cost of `make gen` are tracked in
[DOCS-1141](https://linear.app/codercom/issue/DOCS-1141/reduce-the-compute-cost-of-make-gen).

<details>
<summary>Decision log</summary>

### Goal

Make the sidebar easy to reorder and hard to break, with strict key
order and YAML sources, while keeping `docs/manifest.json` as the file
downstream systems read.

### Jsonnet vs. a custom tool

Jsonnet was evaluated first and rejected, based on tests against
go-jsonnet v0.22.0:

- Jsonnet always sorts object keys alphabetically, and there's no option
to keep them in order (`preserve_order` isn't a parameter of
`manifestJsonEx` or `objectFieldsEx`).
Passing today's manifest through Jsonnet and then biome changed 2,114 of
2,478 lines.
- Unknown keys pass through silently, so a `titel:` typo ends up in the
output.
- The sources would be `.jsonnet` files, not YAML.
Reading YAML takes `std.parseYaml(importstr ...)`, and the keys still
come out sorted.
- Restoring key order would need a Go step after Jsonnet anyway, and
that step is most of this tool.

Decoding YAML into Go structs with yaml.v3 (already a dependency) gives
key order from the struct, rejects unknown keys, keeps comments, and
reproduced the current manifest byte for byte.

### Design choices

- **Ordered by `index.yml`:** reordering sections touches only the
index.
- **One route and its direct children per file:** every file stays short
and flat, so a subsection can be found, reviewed, and moved as one file.
Files nest by parent file name rather than by docs path, so the
directory tree matches the sidebar tree.
Include paths are relative to `docs/manifest/`, like `children_from`, so
every reference to a file is spelled the same way.
A child with `children_from` also gets its own file, so the rule has no
exceptions.
- **Generated subtrees in their own fragments:** the generators can't
safely rewrite hand-edited YAML, so they own separate files.
Hand-written metadata for generated routes lives in the YAML, attached
by title, so descriptions never live in a generated file.
- **Key order enforced in both sources and output:** `fmt` rewrites the
sources, `check` fails on unformatted sources, and the output order
comes from the `docgenenv.Route` field order.
- **`&` left unescaped:** the output matches `main` after #30161, which
stopped escaping `&` downstream.
- **`gen/docs-manifest` separate from `fmt`:** keeping manifest assembly
out of `fmt` means `make -j gen fmt` never has two jobs writing
`docs/manifest.json` at once.
- **YAML pitfalls:** unquoted `no` and `123` stay strings because the
fields are typed as strings, an unquoted `: ` is a parse error, and the
silent ` #` truncation is rejected by the comment check.

### Out of scope

- Replacing `&` with "and" in titles (a style question).
- Reducing the cost of `make gen` (DOCS-1141).

</details>

---

_This PR was generated by Coder Agents on behalf of @nickvigilante._
2026-09-30 21:27:14 +00:00

426 lines
14 KiB
Go

package docgenenv
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"io"
"os"
"path/filepath"
"slices"
"strings"
"golang.org/x/xerrors"
"gopkg.in/yaml.v3"
"github.com/coder/coder/v2/scripts/atomicwrite"
)
const (
// ManifestSourcesDir is the directory, relative to the docs directory,
// that holds the hand-edited YAML sidebar sources compiled into
// manifest.json.
ManifestSourcesDir = "manifest"
// ManifestIndexFile is the sources file that lists the manifest versions
// and the top-level section files in sidebar order.
ManifestIndexFile = "index.yml"
)
// ManifestIndex is the decoded ManifestIndexFile. Its field order is the key
// order enforced in the source file.
type ManifestIndex struct {
Versions []string `yaml:"versions"`
Sections []string `yaml:"sections"`
}
// LoadManifestSources reads the sidebar sources in dir: the index plus one
// top-level route per section file, with every include resolved. Routes that
// set children_from are returned unresolved, so their Children hold only the
// curated metadata overlays. The doc generators use this to read curated
// metadata without depending on the generated fragments they are about to
// rewrite.
func LoadManifestSources(dir string) (*Manifest, error) {
m, _, err := loadSources(dir)
return m, err
}
// BuildManifest reads the sidebar sources in dir, splices every children_from
// fragment into its route, and validates the result. When docsDir is not
// empty, every route path must name an existing file under it. It fails on any
// file in dir that nothing references, so a section can't silently drop out
// of the sidebar.
func BuildManifest(dir, docsDir string) (*Manifest, error) {
m, used, err := loadSources(dir)
if err != nil {
return nil, err
}
var errs []error
m.Routes = resolveRoutes(dir, m.Routes, nil, used, &errs)
if docsDir != "" {
checkPaths(docsDir, m.Routes, nil, &errs)
}
errs = append(errs, unreferencedFiles(dir, used)...)
if len(errs) > 0 {
return nil, errors.Join(errs...)
}
return m, nil
}
// WriteRouteFragment writes routes to the generated fragment at path, as YAML
// when the extension is .yml or .yaml and as JSON otherwise.
func WriteRouteFragment(path string, routes []Route) error {
var (
b []byte
err error
)
if isYAMLFile(path) {
b, err = MarshalYAML(routes)
} else {
b, err = MarshalJSON(routes)
}
if err != nil {
return xerrors.Errorf("marshal fragment %q: %w", path, err)
}
if err := atomicwrite.File(path, b); err != nil {
return xerrors.Errorf("write fragment %q: %w", path, err)
}
return nil
}
// MarshalYAML renders v as YAML with two-space indentation.
func MarshalYAML(v any) ([]byte, error) {
var buf bytes.Buffer
enc := yaml.NewEncoder(&buf)
enc.SetIndent(2)
if err := enc.Encode(v); err != nil {
return nil, err
}
if err := enc.Close(); err != nil {
return nil, err
}
return buf.Bytes(), nil
}
func loadSources(dir string) (*Manifest, map[string]bool, error) {
used := map[string]bool{ManifestIndexFile: true}
var idx ManifestIndex
if err := DecodeSourceFile(filepath.Join(dir, ManifestIndexFile), &idx); err != nil {
return nil, nil, err
}
var errs []error
if len(idx.Versions) == 0 {
errs = append(errs, xerrors.Errorf("%s: versions must list at least one version", ManifestIndexFile))
}
if len(idx.Sections) == 0 {
errs = append(errs, xerrors.Errorf("%s: sections must list at least one section file", ManifestIndexFile))
}
m := &Manifest{Versions: idx.Versions}
for _, section := range idx.Sections {
name, err := sourceRel(section)
if err != nil {
errs = append(errs, xerrors.Errorf("%s: section %q: %w", ManifestIndexFile, section, err))
continue
}
if !isYAMLFile(name) {
errs = append(errs, xerrors.Errorf("%s: section %q must be a .yml file", ManifestIndexFile, section))
continue
}
if used[name] {
errs = append(errs, xerrors.Errorf("%s: section %q is listed more than once", ManifestIndexFile, section))
continue
}
used[name] = true
if r, ok := loadRouteFile(dir, name, nil, used, &errs); ok {
m.Routes = append(m.Routes, r)
}
}
if len(errs) > 0 {
return nil, nil, errors.Join(errs...)
}
return m, used, nil
}
// loadRouteFile decodes the route in the source file name and replaces each
// include entry among its children with the route loaded from the named file.
// A source file holds one route and one level of children, so a child with
// children or children_from of its own must live in its own file.
func loadRouteFile(dir, name string, crumb []string, used map[string]bool, errs *[]error) (Route, bool) {
var r Route
if err := DecodeSourceFile(filepath.Join(dir, name), &r); err != nil {
*errs = append(*errs, err)
return Route{}, false
}
crumb = append(slices.Clip(crumb), r.Title)
where := name + ": " + breadcrumb(crumb)
checkSourceRoute(where, r, errs)
if r.Include != "" {
*errs = append(*errs, xerrors.Errorf("%s: include is allowed only on a child entry", where))
}
if r.ChildrenFrom != "" {
checkOverlays(where, r, errs)
return r, true
}
for i, c := range r.Children {
if c.Include == "" {
cw := where + " > " + c.Title
checkSourceRoute(cw, c, errs)
if len(c.Children) > 0 || c.ChildrenFrom != "" {
*errs = append(*errs, xerrors.Errorf("%s: a source file holds one level of children; move this route into its own file and include it", cw))
}
continue
}
iw := fmt.Sprintf("%s > include %q", where, c.Include)
if c.Title != "" || c.Description != "" || c.Path != "" || c.IconPath != "" || len(c.State) > 0 || c.ChildrenFrom != "" || len(c.Children) > 0 {
*errs = append(*errs, xerrors.Errorf("%s: an include entry may set only include; put the route's keys in the included file", iw))
continue
}
inc, err := sourceRel(c.Include)
switch {
case err != nil:
*errs = append(*errs, xerrors.Errorf("%s: %w", iw, err))
continue
case !isYAMLFile(inc):
*errs = append(*errs, xerrors.Errorf("%s: must be a .yml file", iw))
continue
case used[inc]:
*errs = append(*errs, xerrors.Errorf("%s: file is already listed in %s or included elsewhere", iw, ManifestIndexFile))
continue
}
used[inc] = true
if child, ok := loadRouteFile(dir, inc, crumb, used, errs); ok {
r.Children[i] = child
}
}
return r, true
}
// checkSourceRoute checks the keys every hand-edited route needs.
func checkSourceRoute(where string, r Route, errs *[]error) {
if r.Title == "" {
*errs = append(*errs, xerrors.Errorf("%s: route is missing a title", where))
}
if r.Path == "" {
*errs = append(*errs, xerrors.Errorf("%s: route is missing a path", where))
}
}
// checkOverlays checks a children_from route before its fragment is resolved.
func checkOverlays(where string, r Route, errs *[]error) {
if _, err := sourceRel(r.ChildrenFrom); err != nil {
*errs = append(*errs, xerrors.Errorf("%s: children_from %q: %w", where, r.ChildrenFrom, err))
}
// With children_from, children are metadata overlays matched by title
// onto the generated routes, so they can't carry structure of their own.
for _, o := range r.Children {
ow := where + " > " + o.Title
switch {
case o.Title == "":
*errs = append(*errs, xerrors.Errorf("%s: overlay is missing a title", where))
case o.Path != "" || len(o.Children) > 0 || o.ChildrenFrom != "" || o.Include != "":
*errs = append(*errs, xerrors.Errorf("%s: overlays under children_from may set only title, description, icon_path, and state", ow))
}
}
}
// resolveRoutes returns routes with every children_from fragment spliced in
// and its overlays applied.
func resolveRoutes(dir string, routes []Route, crumb []string, used map[string]bool, errs *[]error) []Route {
out := slices.Clone(routes)
for i := range out {
r := &out[i]
c := append(slices.Clip(crumb), r.Title)
if r.ChildrenFrom == "" {
r.Children = resolveRoutes(dir, r.Children, c, used, errs)
continue
}
name, _ := sourceRel(r.ChildrenFrom) // Validated by loadSources.
used[name] = true
var generated []Route
if err := DecodeSourceFile(filepath.Join(dir, name), &generated); err != nil {
*errs = append(*errs, xerrors.Errorf("%s: %w", breadcrumb(c), err))
continue
}
validateFragment(name, generated, nil, errs)
r.Children = applyOverlays(name, generated, r.Children, breadcrumb(c), errs)
r.ChildrenFrom = ""
}
return out
}
func validateFragment(file string, routes []Route, crumb []string, errs *[]error) {
for _, r := range routes {
c := append(slices.Clip(crumb), r.Title)
switch {
case r.Title == "":
*errs = append(*errs, xerrors.Errorf("%s: %s: route is missing a title", file, breadcrumb(c)))
case r.Path == "":
*errs = append(*errs, xerrors.Errorf("%s: %s: route is missing a path", file, breadcrumb(c)))
case r.ChildrenFrom != "" || r.Include != "":
*errs = append(*errs, xerrors.Errorf("%s: %s: fragments can't set children_from or include", file, breadcrumb(c)))
}
validateFragment(file, r.Children, c, errs)
}
}
// applyOverlays fills empty metadata on the generated routes from the curated
// overlays with the same title. A field set on both sides must match, so the
// manifest and the generated page front matter can't disagree. An overlay
// that matches no generated route is an error rather than silently dropped
// metadata.
func applyOverlays(file string, generated, overlays []Route, where string, errs *[]error) []Route {
byTitle := make(map[string]Route, len(overlays))
for _, o := range overlays {
if _, dup := byTitle[o.Title]; dup {
*errs = append(*errs, xerrors.Errorf("%s: overlay %q is listed more than once", where, o.Title))
}
byTitle[o.Title] = o
}
matched := make(map[string]bool, len(overlays))
out := slices.Clone(generated)
for i := range out {
g := &out[i]
o, ok := byTitle[g.Title]
if !ok {
continue
}
matched[g.Title] = true
ow := where + " > " + g.Title
fill := func(field string, dst *string, src string) {
switch {
case src == "":
case *dst == "":
*dst = src
case *dst != src:
*errs = append(*errs, xerrors.Errorf("%s: overlay %s %q conflicts with generated %q in %s", ow, field, src, *dst, file))
}
}
fill("description", &g.Description, o.Description)
fill("icon_path", &g.IconPath, o.IconPath)
switch {
case len(o.State) == 0:
case len(g.State) == 0:
g.State = o.State
case !slices.Equal(g.State, o.State):
*errs = append(*errs, xerrors.Errorf("%s: overlay state %q conflicts with generated %q in %s", ow, o.State, g.State, file))
}
}
for _, o := range overlays {
if !matched[o.Title] {
*errs = append(*errs, xerrors.Errorf("%s: overlay %q matches no route in %s; rename or remove it", where, o.Title, file))
}
}
return out
}
func checkPaths(docsDir string, routes []Route, crumb []string, errs *[]error) {
for _, r := range routes {
c := append(slices.Clip(crumb), r.Title)
info, err := os.Stat(filepath.Join(docsDir, filepath.FromSlash(r.Path)))
if err != nil || !info.Mode().IsRegular() {
*errs = append(*errs, xerrors.Errorf("%s: path %q does not exist under %s", breadcrumb(c), r.Path, docsDir))
}
checkPaths(docsDir, r.Children, c, errs)
}
}
func unreferencedFiles(dir string, used map[string]bool) []error {
var errs []error
err := filepath.WalkDir(dir, func(path string, d os.DirEntry, err error) error {
if err != nil {
return err
}
if d.IsDir() || strings.HasPrefix(d.Name(), ".") {
return nil
}
rel, err := filepath.Rel(dir, path)
if err != nil {
return err
}
if rel = filepath.ToSlash(rel); !used[rel] {
errs = append(errs, xerrors.Errorf("%s: not listed in %s sections, any include, or any children_from", rel, ManifestIndexFile))
}
return nil
})
if err != nil {
errs = append(errs, xerrors.Errorf("walk %q: %w", dir, err))
}
return errs
}
// DecodeSourceFile strictly decodes one sidebar source or fragment into v:
// YAML for .yml and .yaml files and JSON otherwise. Unknown keys, extra
// documents, and trailing data are errors. In YAML, a comment on the same line
// as a title or description is rejected, because an unquoted " #" there
// silently truncates the value.
func DecodeSourceFile(path string, v any) error {
b, err := os.ReadFile(path)
if err != nil {
return xerrors.Errorf("read %q: %w", path, err)
}
name := filepath.Base(path)
if !isYAMLFile(path) {
dec := json.NewDecoder(bytes.NewReader(b))
dec.DisallowUnknownFields()
if err := dec.Decode(v); err != nil {
return xerrors.Errorf("%s: %w", name, err)
}
if _, err := dec.Token(); !errors.Is(err, io.EOF) {
return xerrors.Errorf("%s: unexpected data after the top-level value", name)
}
return nil
}
dec := yaml.NewDecoder(bytes.NewReader(b))
dec.KnownFields(true)
if err := dec.Decode(v); err != nil {
if errors.Is(err, io.EOF) {
return xerrors.Errorf("%s: file is empty", name)
}
return xerrors.Errorf("%s: %w", name, err)
}
var extra yaml.Node
if err := dec.Decode(&extra); !errors.Is(err, io.EOF) {
return xerrors.Errorf("%s: file must contain a single YAML document", name)
}
var doc yaml.Node
if err := yaml.Unmarshal(b, &doc); err != nil {
return xerrors.Errorf("%s: %w", name, err)
}
var errs []error
checkProseComments(name, &doc, &errs)
return errors.Join(errs...)
}
func checkProseComments(file string, n *yaml.Node, errs *[]error) {
if n.Kind == yaml.MappingNode {
for i := 0; i+1 < len(n.Content); i += 2 {
k, v := n.Content[i], n.Content[i+1]
if (k.Value == "title" || k.Value == "description") && v.Kind == yaml.ScalarNode && (k.LineComment != "" || v.LineComment != "") {
*errs = append(*errs, xerrors.Errorf("%s: line %d: %s has a trailing comment; if the value contains \" #\", quote it", file, v.Line, k.Value))
}
}
}
for _, c := range n.Content {
checkProseComments(file, c, errs)
}
}
// sourceRel validates a sources-relative file name and returns it in slash
// form.
func sourceRel(name string) (string, error) {
clean := filepath.Clean(filepath.FromSlash(name))
if name == "" || !filepath.IsLocal(clean) {
return "", xerrors.New("must be a relative path inside the sources directory")
}
return filepath.ToSlash(clean), nil
}
func isYAMLFile(path string) bool {
ext := filepath.Ext(path)
return ext == ".yml" || ext == ".yaml"
}
func breadcrumb(crumb []string) string {
return strings.Join(crumb, " > ")
}