Commit Graph
14 Commits
Author SHA1 Message Date
Jeremy RuppelandAndrew Aquino 52142e6e94 fix(coderd/templatebuilder): strip prerequisite markers from delivered README (#28953)
## Problem

Template builder base READMEs delimit their prerequisites section with
HTML comment markers (`<!-- prerequisites:start -->` / `<!--
prerequisites:end -->`). `ExtractPrerequisites` uses these to populate
the builder's Prerequisites step, which works correctly.

However, `Compose` embeds the **full** base README verbatim into the
generated template's tarball. Because `react-markdown` renders raw HTML
comments as literal text, the markers were visible when the delivered
README is viewed via `MemoizedMarkdown`.

<img width="876" height="586" alt="Screenshot 2026-09-03 at 3 28 08 PM"
src="https://github.com/user-attachments/assets/70353ee2-1ff6-4e70-9b5d-7ac0521b945f"
/>

## Fix

Strip the prerequisite markers from the README before writing it into
the tarball (Option 2), keeping the change contained to the template
builder and leaving global markdown rendering untouched.

- `prerequisites.go`: new `StripPrerequisiteMarkers` removes each marker
line (plus an optional trailing blank line so no gap is left) while
preserving the content between them.
- `compose.go`: new `composedReadme` helper applies the strip in both
`Compose` return paths; returns `nil` for an empty README, preserving
the existing "omit README.md when empty" behavior.
- `ExtractPrerequisites` still reads the markers from the source
`BaseReadme`, so the builder's Prerequisites step is unaffected.

## Testing

- `TestStripPrerequisiteMarkers` covers both markers, no markers, single
markers, missing trailing blank line, and end-of-file cases.
- Extended the compose test to assert the delivered README contains no
marker text.
- `go test ./coderd/templatebuilder/`, `go vet`, and `golangci-lint run
./coderd/templatebuilder/...` all pass.

<details>
<summary>Approach considered</summary>

Two options were on the table:

1. Skip HTML comments in the shared `MemoizedMarkdown` component (fixes
any stray comment globally, but changes shared rendering behavior
app-wide).
2. Strip the markers from the README before inserting it into the
template zipball (targeted; markers have no purpose in the delivered
README since prerequisites are already extracted separately).

Option 2 was chosen for being contained and easily unit-tested.
</details>

---
_This PR was created by Coder Agents on behalf of @jeremyruppel._

---------

Co-authored-by: Andrew Aquino <dawneraq@gmail.com>
2026-09-04 19:14:10 +00:00
Jeremy Ruppel 7e0f62c204 feat(coderd/templatebuilder): add base agent metadata and per-module agent targeting (#28794) 2026-09-01 13:10:55 -04:00
Jeremy RuppelandCoder Agents a93b2258cf refactor(coderd/templatebuilder): parse rendered HCL with hclparse and support multiple agents (#28787)
Part of [DEVEX-556](https://linear.app/codercom/issue/DEVEX-556).

## Summary

Replaces the Template Builder's regex-based extraction of information
from base-template Terraform with an `hclsyntax` parse of the rendered
output, and removes the single-`coder_agent` assumption from the compose
core.

## What changed

- `render.go`: parse rendered Terraform via `hclsyntax.ParseConfig`
instead of regex.
- Added `ExtractedAgent{Name, Reference}` and
`ExtractAgentResourceNames`, which enumerates every `coder_agent` in
declaration order and reads `count`/`for_each` from each block body.
This is robust to nested blocks (`env {}`, `metadata {}`), fixing a case
the old `[^}]*` regex missed.
- Kept singular `ExtractAgentResourceName` as a compatibility wrapper
that still errors unless exactly one agent exists.
- `ExtractModuleNames` now walks parsed `module` blocks rather than
line-anchored regex.
- `compose.go`: `Compose` wires modules to the base's first/default
agent instead of erroring when a base declares multiple agents.
Single-agent output is byte-identical.
- `render_test.go`: added `TestExtractAgentResourceNames`
(multiple-in-order, `count` after a nested block, no agents).

## Scope

Parsing/compose layer only. Per-module agent **selection** (SDK/UI) is
intentionally out of scope and left as follow-up.

## Testing

- `go test ./coderd/templatebuilder/` passes (golden snapshots
unchanged).
- `golangci-lint run ./coderd/templatebuilder/` reports 0 issues.
- `go build ./coderd/...` succeeds. No new dependencies
(`hashicorp/hcl/v2` already vendored).

<details>
<summary>Design context</summary>

- The `.tf.tmpl` files are Go templates and only become valid HCL after
rendering, so parsing happens on rendered output (which the extractors
already operated on).
- `base.json` stays authoritative; no manifest fields are inferred or
removed here. A survey found `os` is mostly inferable from
`coder_agent.os` (except `scratch`, which uses a dynamic
`data.coder_provisioner.me.os`), and that `variables` cannot be safely
inferred because base variables are Go-template inputs, not Terraform
`variable` blocks. These were left out to keep the change minimal.
- Multiple agents are supported by wiring modules to the first agent in
document order; single-agent bases are unaffected because first == only.

</details>

---
🤖 Generated with Coder Agents on behalf of @jeremyruppel.

---------

Co-authored-by: Coder Agents <agents@coder.com>
2026-08-31 16:59:36 -04:00
Jeremy Ruppel c73f34c35d feat: honor the deployment registry mirror in template builder bases (#28727)
## Summary

Threads the deployment's configured module registry
(`CODER_TEMPLATE_BUILDER_REGISTRY_URL`) into template-builder **base**
rendering, so a base that embeds a catalog module (e.g. the quickstart
base's `git-clone`) resolves its `source` through a registry mirror the
same way wizard-composed modules already do. Previously base-embedded
module sources hardcoded `registry.coder.com`, so on a
mirror/private-registry deployment a base-embedded module reached the
public registry while wizard modules used the mirror.

Implements [DOCS-610](https://linear.app/codercom/issue/DOCS-610).

## What changed

* `BaseRenderContext.RegistryBase` (new field), mirroring
`ModuleRenderContext.RegistryBase`. `Compose` threads the deployment
registry through `renderBase` into the base render context, and the four
hardcoded base module sources (`quickstart` git-clone,
`azure-linux`/`gcp-linux`/`gcp-windows` region modules) now use `{{
.RegistryBase }}`. Because the value defaults back to
`registry.coder.com`, rendered sources stay byte-identical; the only
golden change is the removal of the now-stale quickstart "follow-up"
note.
* **Boot-time validation.**
`codersdk.ValidateTemplateBuilderRegistryURL` runs in
`DeploymentValues.Validate()`, gated on the builder being enabled,
matching the chat-hook precedent. A malformed value fails at server
start naming the option, instead of surfacing as a per-request error to
a template author who cannot fix it. It uses `net/url` in authority form
to reject a scheme, path, query, fragment, trailing slash, or
credentials and accept a bare `host[:port]`; it does not canonicalize
the host.
* `Compose` **trusts the validated value**, defaulting an unset one and
defensively rejecting a malformed one for direct callers that bypass
boot.

## Tests

* `codersdk`: validator accept/reject matrix (incl.
credentials-without-echo) and a `DeploymentValues.Validate` boundary
test with a disabled-skips-validation case.
* `templatebuilder`: a base honors the mirror and falls back to the
default when unset, plus a defensive-reject case; existing compose tests
updated to bare-host registry values.

<details><summary>Design decision log</summary>

This PR is a deliberately minimal reimplementation of the
[DOCS-610](https://linear.app/codercom/issue/DOCS-610/template-builder-base-embedded-modules-should-honor-the-deployment)
scope, started fresh from `main` rather than carrying forward an
earlier, larger draft (coder/coder#28480). Key decisions:

* **Scope kept to the ticket.** The ticket's fix is three bullets: add a
registry field to `BaseRenderContext`, thread the deployment
`RegistryURL` through `Compose` -> `renderBase`, and use it in base
module sources. Independent hardening explored in the earlier draft was
dropped as out of scope: `terraform-svchost`-based IDN/punycode/port
canonicalization, a rendered-HCL backstop with a 500 mapping, and HCL
module-source extraction helpers. Telemetry misclassification of mirror
modules is tracked separately
([DOCS-712](https://linear.app/codercom/issue/DOCS-712/telemetry-misclassifies-registry-mirror-modules-as-non-coder))
and is not addressed here.
* **Default lives in the render layer.**
`templatebuilder.DefaultRegistryBase` holds the fallback host;
`codersdk` keeps its own option-default literal. An earlier version
exported the constant from `codersdk`, which only emitted a dead
`typesGenerated.ts` entry that nothing in the frontend consumed, so it
was moved out. The `codersdk` validator is a function, so it generates
no TS type.
* **Validate at the config boundary, fail fast.** The repo convention is
to hard-fail structurally invalid config in
`DeploymentValues.Validate()` (access URL, OAuth lifetimes, chat hook
URL) and to warn only for functional-but-risky settings. A malformed
registry host is structurally invalid, so it fails at boot. This also
puts the error in front of the admin who set it, rather than a template
author hitting a per-request error they cannot fix.
* `net/url`**, not a regex.** The codebase validates identifiers/names
with regexes but host/URL-shaped config with `url.Parse` (chat hook URL,
OAuth redirect URIs, AI provider base URL). This value is host-shaped,
so it follows that convention. `url.Parse("//" + v)` with `u.Host == v`
rejects a scheme/path/query/fragment/trailing-slash in one check.

</details>

> This PR was created with AI assistance (Coder Agents).
2026-08-31 09:32:24 -04:00
Nick Vigilante c78dca5141 feat: add Coder Quickstart base to the template builder (#27247)
## Summary

Adds the **Coder Quickstart** template as a selectable base in the
[template builder](coderd/templatebuilder) guided wizard. It is a
Docker-based starter that lets a user pick languages and optionally
clone a repo.

## What changed

- **New base package** `coderd/templatebuilder/bases/quickstart/`:
- `base.json`: `id: "quickstart"`, `display_name: "Coder Quickstart"`,
`os: "linux"`.
- `main.tf.tmpl`: a Docker workspace with a language selector, a single
language-install script, an optional Git clone, and workspace presets.
Editors are added via the builder's module step rather than baked into
the base.
- `install-languages.sh.tftpl` and `README.md` (with prerequisites
markers so the builder can extract the prerequisites section).
- **Placement**: the bases endpoint returns a plain list sorted by
display name (with ID as a deterministic tiebreak). The wizard's
base-infra select step prioritizes the Quickstart and Docker starters to
the front on the client, mirroring the existing client-side module
prioritization in `ModuleSelectStep` (shared `sortByPriority` helper).
- **Base/module collision guard**: a base's bundled catalog modules are
derived from its rendered Terraform (`ExtractModuleNames`, cached with
`sync.OnceValues` and filtered to catalog IDs). `validateModules` seeds
its seen-set from that derived set, so a wizard-selected module the base
already renders is rejected with a clear error, and the modules endpoint
omits it so the wizard never offers a colliding module. The base's
Terraform is the single source of truth, so there is no separate
manifest list that can drift.

## Testing

- `go test ./coderd/` (template builder handler: bases ordering, the
`baseSpec` table, the modules-endpoint base filter): pass.
- `go test ./coderd/templatebuilder/` (all-bases render/snapshot, the
collision guard, `TestBaseIncludedModules`, the selector/install-script
drift test): pass.
- `gofmt`, `go vet`, `golangci-lint`: clean.
- Frontend `tsc`, `biome`, and `knip`: clean.

## Review decisions

- **Template scope**: trimmed to infrastructure, a language selector,
and an optional Git clone. The IDE selector was removed; editors are
added via the builder's module step.
- **Placement**: base ordering is presentational, so it lives on the
client base-infra select step rather than in the API.
- **git-clone in the base**: kept for the build-time clone affordance.
The base renders its module source verbatim, so it currently pins the
public registry and does not yet honor a deployment's registry mirror;
that fix is tracked in DOCS-610 and delivered as a stacked follow-up PR
(an in-code note documents the current behavior).
- **Container image**: left hardcoded
(`codercom/enterprise-base:ubuntu`). The quickstart base is the
opinionated path; the Docker base covers custom images. Revisiting a
`container_image` variable is tracked in DOCS-611.

## Review updates (@jeremyruppel)

- **Ordering moved to the client**: deleted the server-side
Quickstart-before-Docker grouping (and its position test); the
base-infra select step now prioritizes bases client-side via a shared
`sortByPriority` helper, following the `ModuleSelectStep` precedent. The
API just returns a stable, name-sorted list.
- **Derive included modules from render**: dropped the
`included_modules` manifest field and the `IncludedModules` struct
field; `BaseIncludedModules` now derives from the rendered base (single
source of truth). Removed `TestBaseIncludedModulesMatchRendered` and
added a direct `TestBaseIncludedModules`.
- **Kept the PR scoped to "add quickstart base"**: reverted
`ExtractAgentResourceName`/`ExtractModuleNames` to the pre-`hclsyntax`
regex extraction and removed the option/preset HCL extractor family; the
selector↔install-script drift test now uses a small regex. The broader
templatebuilder HCL migration and the preset-language subset guard are
tracked in DOCS-735.
- Branch rebased onto `origin/main` (no content changes beyond the
above).

---

Linear:
[DOCS-558](https://linear.app/codercom/issue/DOCS-558/add-coder-quickstart-template-to-template-builder)

> This PR was created with AI assistance (Coder Agents).
2026-08-26 20:29:20 -04:00
Jeremy Ruppel 3d966d48b5 fix(coderd/templatebuilder): fix archive bundling for nested static files and counted agents (#26901)
Fixes two template builder bugs that caused AWS EC2 (Linux) template
imports to fail:

1. **Missing directory entries in tar archive**: `BundleTar` wrote
static files with nested paths (e.g.
`cloud-init/cloud-config.yaml.tftpl`) without emitting `TypeDir` entries
for parent directories. The provisioner's archive extractor requires
explicit directory entries and failed with "no such file or directory".

2. **Incorrect agent reference for counted resources**:
`ExtractAgentResourceName` returned `dev` for the AWS Linux base
template, but the agent uses `count =
data.coder_workspace.me.start_count`, so module templates need
`coder_agent.dev[0].id`. The function now detects `count`/`for_each` and
appends `[0]`.

> [!NOTE]
> Generated by Coder Agents (on behalf of @jeremyruppel)
2026-06-30 19:37:30 -04:00
Jeremy Ruppel 6f0b81cfbc feat(coderd/templatebuilder): validation extraction, static file bundling, Windows OS (#26633)
Extract validation helpers to a dedicated file, add static file bundling
for base templates, and add Windows OS support.

**Commit 1: Extract validation to `validate.go`**
Move `validateVariableValue`, `validateStringValue`,
`validateNumberValue`, `validateBoolValue`, `toHCLLiteral`, `hclQuote`,
and `isSimpleJSONValue` from `compose.go` into `validate.go`.
Corresponding tests move to `validate_internal_test.go`. This keeps
`compose.go` focused on the compose pipeline.

**Commit 2: Static file bundling**
Add `StaticFiles` field to `ComposeResult` and a `collectStaticFiles`
helper that walks the base template FS to collect non-template files
(e.g. cloud-init `.tftpl` inputs). `BundleTar` now writes these files
into the output archive in sorted order for deterministic output. This
fixes `aws-linux`, whose cloud-init files were embedded but never
included in the tar.

**Commit 3: Windows OS support**
Add `BaseOSWindows` constant and register `"windows"` in `validBaseOS`
so that base templates with `os="windows"` can be loaded and used for
module compatibility filtering.

> [!NOTE]
> This PR was authored by Coder Agents on behalf of @jeremyruppel.
2026-06-30 18:08:02 -04:00
Jeremy Ruppel 10717572ac feat: show template prerequisites in builder UI (#26523)
Surface base template prerequisites to admins before they create a
template in the Template Builder wizard.

Today, template prerequisites (Docker socket setup, Kubernetes auth, AWS
IAM policies) are only visible in the registry README after import.
Admins hit opaque provisioner errors and have to hunt for docs. This
change extracts the prerequisites from the README and serves them via
the API so the frontend can display them inline.

## How it works

Each base template README uses HTML comment markers (`<!--
prerequisites:start -->` / `<!-- prerequisites:end -->`) to delimit the
prerequisites section. At boot time, the base catalog loader reads the
README, extracts the content between markers via `strings.Index`, and
caches both the full README and the prerequisites string.

The prerequisites are served via a new `prerequisites` field on `GET
/api/v2/templatebuilder/bases`. The full README is included in the
composed template tar bundle and stored as the template version readme.

## Changes

- Add `README.md` with prerequisite markers to
`coderd/templatebuilder/bases/{docker,kubernetes,aws-linux}/`
- New `ExtractPrerequisites()` in `prerequisites.go` using literal
string matching
- `bases.go`: load README at boot, fail loudly if missing, extract
prerequisites
- `compose.go`: include README in `ComposeResult` and tar bundle
- `codersdk`: add `Prerequisites` field to `TemplateBuilderBase`
- Handler: populate prerequisites in bases response, set readme on
template version

<details>
<summary>Implementation notes</summary>

- Prerequisites extraction uses `strings.Index` for exact literal marker
matching; no regex or AST parser needed since we control the markers.
- YAML frontmatter is deliberately retained in the stored README. The
frontend `TemplateDocsPage` already strips it at render time via
`front-matter`.
- The prerequisite markers are HTML comments, invisible in rendered
markdown.
- The `RejectsMissingReadme` test enforces that every base template must
include a README.
- AWS Linux prerequisites span two H2 sections (`## Prerequisites` and
`## Required permissions / policy`), which is why heading-based parsing
was rejected in favor of explicit markers.

*Generated with the assistance of an AI coding agent. Reviewed by
@jeremyruppel.*
</details>

Relates to https://linear.app/codercom/issue/DEVEX-446
2026-06-23 11:36:07 -04:00
Jeremy Ruppel 7ea5d48296 fix: render base variables into templates instead of tfvars (DEVEX-287) (#26436)
Part of the Template Builder wizard PR stack.

## Problem

The kubernetes base template used Terraform `variable` blocks and
`var.*` references for `use_kubeconfig` and `namespace`, but the
composed tar bundle never included a `.tfvars` file. This caused
`terraform plan` to fail with "required template variables need values:
namespace".

## Fix

Base templates now use Go template variables (`{{ .Variables.* }}`) just
like module templates do. Values are validated, HCL-quoted, and rendered
directly into the output HCL.

Also adds explicit "variable is required" validation to both
`mergeBaseVariables` and `mergeModuleVariables`, replacing the previous
reliance on `missingkey=error` at render time for clearer error
messages.

---
> [!NOTE]
> Generated by Coder Agents on behalf of @jeremyruppel
2026-06-23 11:26:38 -04:00
Jeremy Ruppel a30631198d feat: template builder backend fixes (DEVEX-287) (#26432)
Part of the Template Builder wizard PR stack.

## Backend fixes

1. **Registry URL scheme fix**: Default
`CODER_TEMPLATE_BUILDER_REGISTRY_URL` was `https://registry.coder.com`
but Terraform module registry addresses must be scheme-less. Changed to
`registry.coder.com`.

2. **Sensitive variable defaults**: Module `.tf.tmpl` files for
claude-code, aider, amazon-q had sensitive `variable` blocks without
`default`, causing `terraform plan` to fail during template import. Also
fixed the `templatebuildermodulegen` script.

3. **Auto-quote string variables**: The backend now accepts raw string
values from callers and wraps them in HCL quotes automatically.
Previously callers were required to send pre-quoted HCL literals, which
is not a reasonable API contract.

---
> [!NOTE]
> Generated by Coder Agents on behalf of @jeremyruppel
2026-06-23 09:17:14 -04:00
Jeremy Ruppel 87de6dc23e feat: add base template variables to API (#26425) 2026-06-17 12:22:35 -04:00
Jeremy Ruppel de31c7c18e feat: add TemplateBuilderCreateTemplate SDK types and client method (#26360)
Adds `POST /api/v2/templatebuilder/compose/template`, a synchronous
endpoint that composes a template from a base and modules, validates it
via a provisioner import job, and creates the template in a single
request.

The handler composes terraform files, bundles them as a tar, inserts the
file with hash-based dedup, creates a template version with an import
job, waits up to 2 minutes for the job to complete, classifies errors
for known failure modes (network-unreachable registry, DNS failures),
then creates the template on success. Canceled and failed jobs return
appropriate error responses.

Also adds `hclwrite.Format` to composed terraform output for canonical
HCL formatting.

Closes https://linear.app/codercom/issue/DEVEX-279

<details>
<summary>Implementation notes</summary>

- SDK types and client method in `codersdk/templatebuilder.go` with
validation tags matching the standard template creation path
(`template_display_name`, `lt=128`)
- `ClassifyProvisionerError` in `coderd/templatebuilder/errors.go`
detects DNS, connection refused, i/o timeout, and TLS handshake failures
and returns actionable messages
- `waitForProvisionerJob` polls with a ramp-up interval schedule (100ms,
200ms, 500ms, then 1s steady) and accepts an `onUpdate` callback for
future SSE streaming
- Audit logging for both template and template version creation
- TOCTOU name uniqueness: early check for fast feedback, DB unique
constraint catch for the race window (returns 409, not 500)
- Swagger annotations for all error responses (400, 404, 409, 504)

</details>

> 🤖 Generated by Coder Agents
2026-06-15 18:12:55 -04:00
Jeremy Ruppel 4574c7d792 feat: validate module variable keys and values (#26354)
Validates caller-supplied module variable keys and values in the
template builder compose endpoint before template rendering. Previously,
`mergeModuleVariables` accepted any caller-supplied key and value
without validation, allowing unknown keys, computed/sensitive variable
overrides, and malformed HCL literals (including injection payloads) to
pass through to rendered output.

Now `mergeModuleVariables` rejects unknown keys (those not in the
manifest's non-computed, non-sensitive variables) and type-checks
values: strings must be quoted HCL literals without interpolation
markers or unescaped newlines, numbers must be strict numeric literals,
and bools must be exactly `true` or `false`. The literal `null` is
accepted for any type.

Closes https://linear.app/codercom/issue/DEVEX-278

<details>
<summary>Implementation details</summary>

- Changed `mergeModuleVariables` signature from `map[string]string` to
`(map[string]string, error)` to surface validation failures
- Added `validateVariableValue`, `validateStringValue`,
`validateNumberValue`, `validateBoolValue` in `compose.go`
- String validation rejects: unquoted values, HCL interpolation (`${`,
`%{`), unescaped newlines/quotes, trailing backslashes (which would
escape the closing delimiter), and values exceeding 4096 bytes
- Errors wrap the module ID and variable name for clear diagnostics
(e.g. `module "code-server": variable "port": invalid number value`)
- Tests cover key validation, type validation, injection attempts, and
full Compose flow integration

> Generated with the help of [Coder Agents](https://coder.com) by
@jeremyruppel
</details>
2026-06-15 11:34:47 -04:00
Jeremy Ruppel 877f4def4a feat(coderd/templatebuilder): add Compose and BundleTar functions (#26349)
> [!NOTE]
> This PR was authored by Coder Agents on behalf of @jeremyruppel.

Part 2 of DEVEX-277 (POST /api/v2/templatebuilder/compose).

Adds the core composition and bundling logic for the template builder.

`Compose` renders a base template and selected modules into Terraform source files. It validates modules before rendering (rejects duplicates, ConflictsWith violations, unknown IDs, OS incompatibility), then for each module merges manifest defaults with caller-supplied variable overrides and renders the module template.

`mergeModuleVariables` fills in defaults for non-computed, non-sensitive variables from the manifest (with basic JSON type validation via `isSimpleJSONValue`), uses `null` for non-required variables without defaults, and leaves required variables absent so `missingkey=error` catches omissions at render time.

`BundleTar` packages the result into a tar archive with reproducible timestamps. Writes `main.tf` always, `modules.tf` only when modules are present.

Conflict detection is bidirectional so module ordering in the request does not affect validation.
2026-06-15 09:28:39 -04:00