## Summary
Rebuilds the doc-check skill around one question: if this PR merged now,
would the docs on `main` explain the change correctly and clearly?
On #31159 the current skill stayed silent on real defects, because "a PR
that documents itself needs nothing more" ended the review before
anything checked the docs the PR added.
## Changes
- **doc-check skill:** a change model built from the code, eight focused
personas with a verifier, findings that must cite evidence or get
dropped, explicit blocking classes, a changed-lines rule for style,
design-friction notes, and a run trace on every run (`output.md`,
`personas/`).
- **Rules stay in `docs/.style/`:** each rule page lists its personas in
`review-personas` front matter, so the skill finds rules without
depending on file paths.
- **Style guide:** new rules for leading with the reader's surface and
for time-relative words, a clearer Latin abbreviations rule with Do and
Don't examples, and `as` in the one-meaning table. Coverage counts go
from 88 to 90.
- **Conflicting guidance fixed:** `.claude/docs/DOCS_STYLE_GUIDE.md` and
the old skill no longer recommend semicolons or "see".
- **write-docs and AGENTS.md:** authors run doc-check on their branch
before opening or updating a PR.
- **Workflow prompt:** asks for the full review, reads the skill from
the base branch, and runs persona sub-agents only on the configured
model.
Out of scope, tracked separately: commit statuses (DOCS-1407), the
replay harness (DOCS-923), `draft: true` (DOCS-1412), stable rule IDs
(DOCS-1413), and "native" wording (DOCS-1409).
## Testing
- `make lint/markdown`, `lint/emdash`, `lint/style-claims`,
`lint/docs-html`, `lint/docs-manifest`, `lint/actions` (actionlint and
zizmor), and Vale on the changed `docs/.style` files all pass.
- Read-only evaluation runs of the new skill, with nothing posted:
| Known issue on #31159 | Solstice-1 Local, v1 | Sonnet 5.5, v1 |
Solstice-1 Local, v2 |
|---|---|---|---|
| Personal and organization downloads lose the manual-only line | Filed
as pre-existing | Blocking | Blocking |
| New section contradicts the Project skills section | Missed | Missed |
Blocking |
| Section leads with the API field, not the UI switch | Blocking |
Blocking | Blocking |
| Quoted `"true"` is silently ignored | Filed as pre-existing |
Suggestion | Missed |
| Quoted key is stored unchanged, so the file can disagree | Blocking |
Blocking | Missed |
| Claim about other tools can't be checked | Suggestion | Suggestion |
Suggestion |
| PR description claim the code doesn't support | Trace | Trace | Trace
|
| False positives | 0 | 0 | 0 |
On #31164, a docs-metadata PR with nothing wrong, v2 reported a clean
result with no findings.
v2 fixes the v1 misses: the Continuity Editor reads the whole page
first, and gaps on paths the change didn't edit count as caused by the
change.
The coverage persona's example comes from #31159, so the v2 rerun on
that PR overstates the gain. The replay set in DOCS-923 is the real
test.
## Risks
- Eight personas on Solstice may exceed the workflow's 40-minute wait.
Watch the first runs after merge.
- Reviews now post inline comments as well as the summary comment, on
every PR that reaches the review.
<details>
<summary>Plan and decision log</summary>
# doc-check redesign: decisions so far
Status: design interview in progress. Nothing implemented.
Tracking: DOCS-918 (parent), DOCS-1407 (commit status), DOCS-925
(outcomes), DOCS-923 (replay), DOCS-926 (harvester), DOCS-1378 (blocking
slop), DOCS-597 (renderer class drift).
## First principle
If this PR merged now, would the docs on main correctly and clearly
explain this exact change?
The requirement runs both ways: every docs claim is grounded in the
code, and every behavior that materially changes how a customer uses the
product has docs.
"The PR documents itself" is never a reason to stop reviewing.
## Scope and audience
- One skill answers both "are docs needed" and "are the added docs
good".
- Every PR gets the same depth, including docs-only PRs.
- Findings are written for the PR author, and phrased so an authoring
agent can apply them.
## Priority and blocking
Order: accuracy, journey completeness, consistency, audience fit,
structure, concision, word choice and mechanics.
Blocking findings:
- Claims the code contradicts.
- Required docs that are missing for behavior the PR adds or changes.
- Rendering breaks and typos (deterministic checks first, LLM as
backup).
- Consistency and audience findings only when all three hold: the
finding cites a rule ID or a contradicting line, the verifier confirmed
it, and the PR added or edited the text.
Never blocking:
- Pure judgment calls without a citable rule. To make something
blockable, add a rule.
- Style issues on lines the PR did not touch.
- Design-friction notes (uncapped, own category, each tied to a docs
sentence, re-evaluated every round).
- Gaps that predate the PR: filed to Linear through the DOCS-926
harvester.
- Tutorial or lab needs: filed to TRAIN or DOCS through the harvester.
## Evidence and hallucination handling
- Every finding cites evidence: code location, other page and line, or
style rule ID. No evidence, no finding.
- Verifier persona re-checks every finding on every run.
- The summary lists checks the reviewer could not complete.
- `/doc-check wrong <finding-id> <reason>` logs a false finding to
Linear and adds the PR to the replay set.
## Output
- One PR review per round with as many inline comments as needed,
suggestion blocks for local fixes.
- Re-reviews comment only on changed lines and resolve fixed threads. A
clean round posts nothing new.
- Hidden machine-readable findings block (ID, file, line, rule ID,
blocking, fix) for agent fix loops.
- Every run leaves a trace in the commit status and job summary, even
with zero findings.
## Commit status and override (DOCS-1407)
- `docs` status on every commit, outcomes `finding`, `clean`, `error`,
`skipped`.
- An approving review from a `@coder/docs` member overrides to success.
- The override carries forward while later commits pass.
- A later failing commit re-requests review from the approver, then
`@coder/docs` after one business day. The engineer can still merge.
- Rollout: report-only for a few weeks, then required for accuracy and
missing docs once blocking false positives are at or under 5%, then for
rule-anchored consistency and audience.
## Personas
One orchestrator chat with sub-agents, each loading only its rule slice.
IDs match the job; display names are editorial titles.
| ID | Name | Job |
|---|---|---|
| `accuracy` | Fact Checker | Every docs claim is supported by the code
|
| `coverage` | Cartographer | Undocumented behavior, and docs for
nonexistent behavior |
| `learner` | Newcomer | Reads only post-merge docs, reports what it
learned, compared against a code-derived change summary |
| `consistency` | Continuity Editor | Terms and facts agree across and
within pages |
| `audience` | Reader Advocate | UI before API, reader task before
internals, page type fits |
| `structure` | Architect | Headings, procedures, prerequisites,
placement |
| `language` | Copy Editor | Word choice, sentence shape, mechanics
beyond Vale |
| `verifier` | Skeptic | Re-checks every finding's evidence |
Language and consistency can skip when the PR changes no prose.
The persona list grows as needed.
## Rules
- `docs/.style/` is the single human-readable rule source. Fix
conflicting text in `.claude/docs/DOCS_STYLE_GUIDE.md` and the doc-check
skill first (semicolons, unit spacing, "see").
- Each style-guide page lists `review-personas` in front matter. Each
rule heading carries a stable ID anchor. The bot resolves rules by ID
and front matter, never by path.
- Calibration answers that differ from the guide become style-guide PRs
before a persona relies on them, and the calibration sentences become
persona test cases.
## Ethos captured so far
- Test for implementation detail: keep it only if it changes what the
reader sees, types, or must do.
- UI label leads; the API field appears once as "In the API, ...";
frontmatter keys only where the reader types them.
- "Did I learn the thing this PR changes?" is the core reading test.
- Always block: typos, markup that breaks rendering, wrong claims,
unsupported compliance claims, multiple actions in one step (STE), "See
[link]" with non-descriptive link text, missing prerequisites,
inconsistent terms, user-facing implementation narrative, missing docs
for a flag that changes a user task.
- Suggest: "simply" (rewrite the step), passive voice (make the actor
the subject), long multi-condition sentences, semicolons, "now" (replace
with the condition that makes the action possible), `as` or `since` for
cause (use "because"), niche flags without guide coverage.
- Latin: never block. On a line that changes for another reason, suggest
"for example" in place of "e.g.". The accepted forms are `(e.g., ...)`
and `(for example, ...)`.
## Architecture
- `coder/coder` holds the Markdown: skill, personas, rules.
- Private `coder/doc-check` holds everything else as a hosted GitHub
App: orchestration, status, override, re-request, slash commands,
verdict parsing, replay harness, persona-to-model map.
- Authoring agents run the same skill locally before opening or updating
a PR (required in `AGENTS.md` and `write-docs`).
- Review chats are shared with `@coder/docs`, not the whole
organization.
## Models
- Live PRs: Solstice-1 with cloud fallback. Record whether fallback
happened on each run and why.
- Replay and calibration: Solstice-1 Local (no cloud fallback) so data
stays clean.
- Persona-to-model map in the bot repo. A persona moves to a different
local model only when replay scores show it is better for that persona.
- Opus only through `/doc-check model:opus` and in the manual replay.
## Replay set (DOCS-923)
- Run manually, never per commit.
- Contents: the 48-PR benchmark from #29194 if recoverable, #31159,
#31164, the large Vale PRs, confirmed `/doc-check wrong` reports, and
calibration cases.
- Reports Solstice and Opus scores side by side per persona.
## Calibration set 2 decisions
- Open PRs that would change a claim in this PR are scanned and named in
the summary. They never block.
- External claims (other products) never block. Suggest narrowing to
what was checked, linking vendor docs, or adding a product test that
fails if the claim breaks.
- New rule: when a docs claim depends on product behavior, recommend a
test in the product that fails if the claim stops being true.
- New page rule: block a page that is not in `docs/manifest/` unless its
front matter has `draft: true`, and block a page that is in the manifest
while its front matter has `draft: true`. No page on main uses `draft:
true` and the manifest tooling doesn't read it today, so the convention
needs defining first.
- Admin-only settings: block when they're documented only under
`docs/user-guides/`, and require that some page covers them.
- Stale screenshots: block when the image contradicts a step on the
page, otherwise harvest. Long term, generated screenshots replace hand
captures (P-DOCS-174, DOCS-1408).
- Inclusive wording: replace "non-native readers" in the style guide
with wording about readability and translation.
## Open
- Calibration set 2: items Nick didn't answer keep my bracketed guess.
- How `coder-agents-review-bot` is hosted, to model the doc-check
service on it.
- How to detect cloud fallback per chat (per-message model config or
provider metadata).
- Commit status versus check run.
## Overnight decisions (made without Nick; override any of these)
Each entry: the decision, why, and how to undo it.
### Linear
1. **DOCS-1409** "native" wording: separate issue in the Docs style
guide project, Medium, 2 points. Not fixed in the PR, because Nick asked
for a separate issue. The Latin rewrite drops one "non-native readers"
sentence only because that paragraph was rewritten anyway.
2. **DOCS-1412** `draft: true`: sub-issue of DOCS-918, Medium, 3 points.
Scoped as a deterministic check in `docsmanifestgen`, not an LLM rule,
because blocking rules should be deterministic.
3. **DOCS-1411** Pixel screenshots: in Automated screenshot capture
(P-DOCS-174), Medium, 5 points, related to DOCS-45 and DOCS-1408. Pixel
replaces the Playwright proof of concept for anything a Storybook story
renders; Playwright stays for flows that need a live deployment. The
project summary is unchanged.
4. **DOCS-1410** for this PR: sub-issue of DOCS-918, High, 5 points, In
Progress.
5. **DOCS-1413** stable rule IDs: deferred out of this PR, because IDs
for 90 rules plus a `styleclaims` check is its own reviewable change.
### PR scope
6. One PR for skills plus style guide, because the persona front matter
and the skill depend on each other. Out of scope: the hosted bot, commit
statuses, replay harness, slash commands, sharing chats with
`@coder/docs`.
7. Personas are files in `.claude/skills/doc-check/personas/` with the
agreed names.
8. `review-personas` front matter on `content-guidelines.md` and eight
style-guide pages, documented in `docs/.style/README.md`. Single-line
list form so a plain `grep` finds it. `editor-setup.md` and the READMEs
get none, because they hold no rules.
9. Mapping: content guidelines to accuracy, coverage, audience,
structure. Audience and scope to audience, learner, structure. Word
choice to language, consistency, accuracy. Procedural writing and
formatting to structure, language. Capitalization and punctuation to
language, structure. Accessibility and inclusion to language, structure,
audience. Voice and tone, and numbers, to language.
10. CI reads the skill from the base branch, so a PR can't rewrite its
own review. Author self-check reads it from the branch.
11. Sub-agents run only when the workflow passes a model config ID
(`vars.DOC_CHECK_MODEL_CONFIG_ID`), so a run never mixes models.
Otherwise personas run sequentially.
12. Posting: one `COMMENT` review with inline comments, plus the
existing sticky summary comment and marker, because the Slack notice and
re-review logic depend on the marker. Findings for agents go in a
collapsed YAML block in the summary.
13. The footer tells authors to reply in the thread when a finding is
wrong. `/doc-check wrong` isn't advertised, because it doesn't exist
yet.
14. The run trace is a YAML block at the end of the final chat message,
so DOCS-1407 can parse it into a commit status later.
15. Deterministic checks run first. A failing check on a changed line
blocks.
16. The 40-minute wait timeout is unchanged. Eight personas on Solstice
may run longer, so watch for timeouts after merge. Raising it is a
one-line change.
### Style guide content
17. Latin abbreviations: `(e.g., ...)` and `(for example, ...)` are both
accepted. The comma form is a Don't example but never blocks. Reviewers
suggest the English form only when the line changes for another reason.
`etc.` stays accepted in prose, because Ben's position was to accept
Latin abbreviations. Nick's personal "tables only" preference is not
encoded.
18. `as` row added to Choose words that have one meaning,
documentation-only, because `as` has too many correct uses for a Vale
word list.
19. New rule: Describe conditions, not moments in time (voice and tone).
`now` stays allowed for progress inside a sequenced tutorial, because
word choice already uses "Now that you added a language".
20. New rule: Lead with the surface the reader uses (audience and
scope), with the #31159 sentence as the Don't example. Tutorials that
mirror UI and CLI paths keep both.
21. Coverage counts updated from 88 to 90 rules. `make
lint/style-claims` verifies them.
22. Content guidelines: one paragraph under Programmatic content asking
for a product test when no test pins a documented behavior, which backs
the "recommend a test" rule.
23. `.claude/docs/DOCS_STYLE_GUIDE.md`: removed the semicolon
recommendation, "See" as a navigation verb (kept "See which
applications", which describes the UI), and a bare `150ms` threshold
(now in code font, matching the content guidelines).
24. write-docs: whole-path tracing, checking the PR description against
the code, naming the persona, leading with the reader's surface, running
doc-check before the PR, two anti-patterns, two checklist items, and a
fix for the stale "Premium pages carry the title suffix" item.
25. AGENTS.md: one workflow bullet requiring the doc-check self-check
before PRs that touch `docs/` or user-facing behavior.
### Evaluation
26. Eval set: #31159 (known defects) and #31164 (known clean).
Solstice-1 Local for tuned runs, Sonnet 5.5 as the comparison. PR heads
checked out read-only, nothing posted.
27. Changes after the first eval: both models missed the contradiction
between the new section and an older section of the same page, and
Solstice filed the download gap and the silently ignored quoted value as
`adjacent`. The Continuity Editor now reads the whole page first, "the
change caused it" includes paths the change didn't edit, and only class
4 findings need a style rule to block. Risk: the coverage example
mentions downloads, which comes from #31159, so the v2 rerun on #31159
overstates the improvement. The replay set (DOCS-923) is the real test.
</details>
> Generated by Coder Agents on behalf of @nickvigilante.