23 Commits
Author SHA1 Message Date
Waleed 293ec1b7e1 improvement(workflow-code): server hardening, flaky test root causes, and agent DX across the source loop (#10287)
Server: dry runs charged to the plan budget and slot, create-from-source answers 429 when slots are full, the Code tab has its own slot lane, staged blobs count against the payer's storage quota (consumed on apply, never renewed by reads), file deletes carry a revision precondition, row limits refuse only growth, joined table transactions keep their timeouts, and blocks/edges/subflows load in a stable created_at,ctid order. DX: group help for the source loop, a primer covering renames, failures and test runs, JSON-only output, run errors mapped back to source file and line, --stop-after/--from-block by name, webhook URLs after deploy, GitHub Actions annotations and step summary, exit 15 for a partial write, table names on every table route, status that agrees with the dry run, and quieter output. Flaky tests fixed at the root (keep-alive races, sleep-based waits). Migration 0469 adds a nullable staged_blob.consumed_at.
2026-11-21 07:05:36 -08:00
Waleed bc8f09ccb1 feat(workflow-code): workflows and workspace as code (#10129)
* test: type the MCP base address the concealment check compares

* fix(workspace-code): check file path collisions on a segment tree, not a map of every prefix

Keeping every folder prefix of every path still held about a million prefix strings for
20,000 deep paths. A tree of folded segments costs one fold per segment and holds each
segment once, so the depth cap is no longer needed.

* refactor: drop the exports of isSuspiciousCharacter and readFolderWritten, used only in their files

* docs(workflow-code): CI needs a personal key; document workspace-key E_APPLY_FORBIDDEN and --only

workflows.source.apply and create deny workspace API keys, so the CI guidance now asks for a
personal API key or an OAuth login. Workspace as code names the kinds a workspace key cannot
write (workflows, skills, secrets, deployments), the E_APPLY_FORBIDDEN refusal, and narrowing
an apply with --only.

* docs(workflow-code): link the code pages from workflows, CLI, getting started, API and MCP

Adds a For coding agents section to Workflows as code (get, edit, check, status, apply, starting
from .sim/AGENTS.md), a prefer-source note on the API build guide, and an MCP section on editing
a whole workflow with getWorkflowSource and applyWorkflowSource.

* docs(self-hosting): document the WORKFLOW_CODE preview flag

Adds a Preview features section with WORKFLOW_CODE and links it from the in-preview callouts on
the workflows and workspace as code pages.

* feat(workflow-code): Code tab header with Preview label, CLI copy and docs; Download as code

The Code tab gets a title row: a Preview badge, a button copying
sim workflows source get <workflowId> --dir ./<workflow-name-slug>, and a link to the docs.
The workflow menu gains Download as code beside Export workflow, saving the draft as the
single TypeScript file the Code tab shows (function code inline). The download revalidates the
cached source by ETag first, so an edit since the tab last read is never missed.

* chore(rules): load the workflow-code rule for the v2 source routes, contracts and staging cleanup

* feat(llms): list workflows as code in llms.txt

* docs(cli): workflows and workspaces as code in the README, help epilogue and source group descriptions

Adds a README section (get, check, status, diff, apply and the coding-agent loop), a source get
example to the root help, and describes both source groups as read and apply, marked (preview)
while the server flag gates them. Regenerating the CLI docs changes nothing: the pages list each
command, not group descriptions or the root epilogue.

* test(workspace-source): a re-run adopting a stopped MCP replace with stored() headers still needs an admin

A workspace key re-running an interrupted replace is refused with E_APPLY_FORBIDDEN, the
owner's re-run finishes it, and a read-only member's get prints the adopted server's
literal headers as stored().

* feat(workflow-code): roll the workflow-code flag out by organization, workspace or user

Every source use case, the Code tab layout and useFeatureFlag now evaluate
workflow-code against the workspace, its organization and the acting user,
so AppConfig orgIds/workspaceIds/userIds allowlists take effect. Adds the
rollout plan to the workflow-code design doc.

* feat(mothership): grant the workflow-code entitlement where the source commands run

computeEntitlements now sends workflow-code to the worker for a workspace or
organization chat whose rollout has the flag on, ahead of the worker's
ENTITLEMENTS enum (the wire accepts any name). The CLI inventory marks every
workflows source and workspaces source command with entitlement: workflow-code
so the worker can hide them once it reads the field.

* docs(api): say how workspace source blobs are staged, and that dryRun previews

The applyWorkspaceSource description pointed MCP readers at Stage Workspace
Source Blobs, a multipart route MCP does not expose; it now names the CLI and
the blob upload route. The MCP write tool no longer claims no write can be
previewed: the source applies take dryRun.

* feat(analytics): record workflow source reads and dry runs, and workspace source applies

Adds workflow_source_read, workflow_source_dry_run and workspace_source_applied
beside workflow_source_applied: same attribution, kinds and counts only. A
refused dry run is now a dry-run event, never an apply.

* docs(workflows): commit written.json and point workspace CI to its own section

* test: read the MCP server URL as nullable in the stopped-replace check

* test(workflow-code): resolve the workspace before the flag in the HEAD 403 check

The flag now needs the workspace's organization, so the read routes resolve
the workspace first; the test gives them one to resolve.

* test(analytics): one workspace_source_applied per apply, refused, partial or committed

Busy, rejected and partial applies each record exactly one event with their
outcome, the committed retry one more, and a dry run none.

* fix(cli): resend a rate-limited GET even when its response body is lost

Cancelling an errored body rejects, so a 429 whose body was lost threw the raw
stream error instead of resending.

* fix(tests): order the workspace source "nothing was written" snapshots by id

The snapshots compared before and after a refused apply selected rows with no
ORDER BY, so their order was whichever index the planner chose: on the
workflow table the sort, unique-name and workspace_id indexes each return a
workspace's rows in a different order. Once autovacuum analyzed the table
between the two reads the plan could change, and the cross-kind test failed
with equal rows in a different order. Forcing the second read off index scans
reproduced the exact failure; ordering by id makes the comparison
plan-independent.

* refactor(source): one vocabulary for the fingerprint of what the server stores

PlannedChange.serverFingerprint is storedFingerprint and KindAdapter.fingerprintNow is
storedFingerprintNow; the workflow adapter's currentFingerprint() is readStoredFingerprint;
the sf1: helper in print-stored is storedSourceFingerprint, beside the kinds' rf1:
storedFingerprint; the workflow apply's local and EvaluatedSource field are storedFingerprint,
as source-commit already named it. The 409 wire field currentFingerprint is unchanged.

* fix(workflow-sdk): the workspace runtime exports defineSecrets and defineCredentials

secrets.d.ts and credentials.d.ts declare them and the cookbook imports them, but the
runtime entry did not export them, so running such a file failed with 'export not found'
instead of SimSdkNotExecutable. codeFile now re-exports core's stub, as tool.d.ts declares.
A test keeps the runtime's exports equal to every value the declarations declare.

* fix(workspace-source): re-scan deletes for new users under the apply lease

The plan's in-use scan ran before the lease, so an apply that saved a workflow
using a resource another apply was about to delete went unseen: the deleting
apply took the lease and deleted anyway. The apply now scans its deletes again
once it holds the lease and refuses (E_APPLY_STALE, nothing written) a delete
that a workflow the plan did not find now uses; a scan timeout still fails
closed.

* fix(cli): take a stale source-directory lock over safely, never through a link

Two commands that found one lock stale could both remove it: the later one's
removal deleted the live lock the first had taken, admitting two writers. A
takeover now claims the stale lock folder it removes (only one command can) and
checks it is still stale first, so a lock that replaced it is left alone.

The lock path also bypassed the source directory's symlink checks: a .sim or
command.lock that is a link was followed. Both are now refused
(SOURCE_SYMLINK) before the lock is created, read or removed.

* fix(cli): workflow commands in a workspace folder hold the workspace directory lock

A single-workflow apply, create or get inside workflows/<key>/ of a workspace
directory rewrites the workspace's written record and base, but took no lock, so
it could interleave with a workspace command (or another folder's) and lose a
record. These commands now hold the workspace directory's command lock for the
whole command, refused like a workspace command (SOURCE_DIR_BUSY, exit 10)
with nothing read or written.

* fix(cli): status recommends commands in its own scope, with --allow-delete for dropped data

The command workspaces source status recommended dropped the scope it was asked
for: status --only tables suggested an apply of every kind, and a status pointed
at a workspace with --workspace suggested one at the profile's. Every
recommended command now carries --only and, when the target is not the
profile's default, --workspace.

It also asked for --allow-delete only for whole-resource deletes, so an update
that drops a column was suggested without it and refused. It now uses the same
destructive count the apply prints.

* fix(cli): an apply that reprints a tool as stored() removes its old code file

When the server keeps a tool's code as stored() (it holds a credential), the
apply reprints tools/<key>.ts without tools/<key>.js, but saving the answer
left the old .js and its digest behind, so the next apply failed with
E_TOOL_CODE. Saving an apply now removes, from disk and the written record,
every text file the directory sent for a reprinted resource that the reprint no
longer holds (workflow folders keep pruning by lock).

* refactor(table): move cell value rules into @sim/workspace-code/table-cells

The per-type coerce/validate/salvage/equality rules, the row loops (required
cells, coercion policy, batch unique scan) and the pure date, currency, ttl and
select helpers move from apps/sim/lib/table into the workspace-code package
(timezone helpers into @sim/utils/timezone). The column-type registry spreads
CELL_RULES into each definition and validation.ts delegates, so there is one
implementation. The workspace source table adapter refuses declared rows with
refusedTableRows, which the CLI can now run offline.

* feat(sim-cli): workspaces source check validates table rows files offline

Each tables/<key>.rows.jsonl (managed or seed) is read against its declaration
and checked with the write's own rules (refusedTableRows): a malformed line, an
undeclared column, a managed key two lines share, an empty required cell, a
unique value two rows share, a value its column's type or options cannot hold.
Each is an E_TABLE_ROWS error at its line.

* fix(workspace-source): credential adoption never takes a credential from a key bound outside source

* test(workspace-code): assert linear growth, not a wall-clock budget, for hostile inputs

The file-grouping test asserted 20,000 deep paths group within 6 s and took 7.2 s
on a CI runner. It now counts the characters case folding reads at depth 100 and
400 (about 4x linear, 15x for the per-prefix fold it guards against), which no
machine load changes. The two compile tests with a 3 s budget (JWT pattern,
numeric precision) now compare the fastest of three runs at n and 4n: reverting
either fix gives about 15x against a bound of 9.

* fix(table): a seed reshape refuses a table that gained rows since the plan saw it empty

* fix(table): a keyed row write inside a reshape runs at the bulk-write statement timeout

* fix(workspace-source): hold the apply lease from before the plan, not only the writes

An apply planned before taking the lease, so another apply could write between
its plan and its writes: the first then wrote half of itself and failed
(E_NAME_TAKEN, E_APPLY_STALE), or deleted a resource a workflow had just
started using, because its in-use check had run before the other apply's save.
A non-dry-run apply now takes the lease first and plans and writes under it, so
a second apply is refused SOURCE_BUSY before it reads anything. This replaces
the post-lease in-use re-scan, which covered only deletes. Dry runs still take
no lease; the per-write lease confirm and stored-fingerprint re-checks stay for
UI edits, which the lease does not order.

* fix(workspace-code): literal-secret findings carry a span but no code frame

E_MCP_LITERAL_SECRET, E_DEPLOY_SECRET, W_TOOL_LITERAL_SECRET and the workflow
W_LITERAL_SECRET framed the line holding the secret, echoing it over v2 and MCP.
They keep their span; the finding cap and the server locator drop the frame too.

* fix(workspace-source): a secret declared without a description records no base entry, so it is never falsely stale

* test(workflow-code): time hostile inputs with Date.now, repeated past the clock's resolution

The package has no Node or DOM types, so performance is not declared. Each
growth timing now repeats the work until it spans 25 ms, so millisecond steps
do not swing the ratio.

* fix(workspace-code): an env reference or helper call is never read as an object

headers: env.TOKEN parsed with no finding into a header named $env, and
defineTable(column.string()) reported the encoding's $call/args fields. Both
now get one finding that names the value and how to write the object.

* feat(workflow-code): mixed-script block names, workflow inputs, env hint target

- E_LOOKALIKE_NAME also refuses a block name with a word mixing Latin with
  Cyrillic, Greek, Armenian or Cherokee letters (Unicode script property, no
  confusables table), naming the code point; W_ when the block is stored.
- startInputs and INPUT_TRIGGER_TYPES (shared with apps/sim) read a workflow's
  Start trigger inputs; W_CHILD_INPUT_UNKNOWN and W_CHILD_NO_START join the
  vocabulary for the CLI's offline child-call check.
- CompileInput.envDeclaredIn points a W_ENV_UNKNOWN hint at secrets.ts.

* feat(sim-cli): workspaces source check covers skills, child inputs and workspace.d.ts

- skill('<key>') is checked against skills/ like every other kind; a key with
  a non-ASCII letter names its code point and script.
- A child workflow call is checked against the child's Start trigger in the
  same directory (W_CHILD_INPUT_UNKNOWN, raised on new blocks; W_CHILD_NO_START).
- A get keeps the server's resource manifest beside workspace.d.ts, and check
  regenerates workspace.d.ts from it and the directory's declarations, typing
  new keys and each workflow's inputs without a request.
- Did-you-mean hints on E_MAT_UNKNOWN_RESOURCE, W_TABLE_UNKNOWN_COLUMN and
  W_ENV_UNKNOWN (which points at secrets.ts); a warning raised to an error no
  longer says it is kept as written.
- check --help lists every kind it checks.

* fix(workspace-code): check the option cap before a linear duplicate scan; pin a duplicate column to itself

A select's repeated-option scan was quadratic and ran before the maxOptions cap:
80k options took 7 s per parse (now ~0.1 s). E_TABLE_DUPLICATE_COLUMN spanned the
whole defineTable argument instead of the second column.

* perf(workspace-code): group a status's changed files in linear time; key server findings by code

localResourceChanges copied the kind's array per file: 40k files took 3.3 s.
SERVER_FINDINGS is now checked against WorkspaceDiagnosticCode.

* perf(workflow-code): print a fill in linear time by advancing an offset

printFill re-spread the remaining parts for every content/separator pair, so a
number-only array printed in quadratic time (40k numbers took 17.5 s). The rest
of the fill is now queued as the same parts from a later offset.

* fix(workflow-code): refuse a tool param or operation option the builder pins

A tool builder's fixed params and the operation option overwrote the method's
pin silently, so the tool stored an operation the method contradicts and printed
back as raw(). Report E_FIXED_PARAM, as the block path does.

* refactor(workflow-code): look a tool's builder up once

toolBuilderPath found the builder, returned its path, and compileToolCall looked
it up again behind an early return that could not fire.

* fix(workspace-code): cap knowledge document findings apart from knowledge.ts's own

Document findings were uncapped and counted against the field reporter's
100-finding budget, so the declaration's own findings became E_LIMIT_EXCEEDED.
The field reporter now counts only what it appends.

* fix(workspace-source): a workflow delete is fingerprinted only once its live triggers are counted

* fix(workflow-code): exclude a retyped block from rename inference by its own id

A retype matched by lock ident (or against a prev block the canvas renamed)
excluded the prev block under the new key, not the one it retyped, so another
new block of the old type could inherit the old id, its run history and its
webhook. Record the retyped prev id and exclude that.

* fix(workspaces): a key claim that loses a concurrent race refuses with 409 or retries, never a raw 23505

claimResourceKey read the key with select-for-update, but an unheld key has no row to lock, so a
concurrent get (allocation) or claim could insert first and the plain insert failed with a unique
violation on either index, surfacing as a 500. The insert now skips on conflict and the claim
re-reads the committed winner: kept, ResourceKeyTakenError, or a retry that replaces a key the
resource was allocated meanwhile. Rebinding deletes and re-inserts so it shares the same path.

* perf(workflow-code): push identity group keys instead of re-spreading the group

Each push copied the whole group, quadratic when many unmatched blocks share a
type and parent.

* fix(workflow-sdk): new on a runtime export throws SimSdkNotExecutable; one stub for both entries

The Proxy target was an arrow function, which has no [[Construct]], so the
construct trap never ran and new t() threw a TypeError. The workspace runtime
re-declared notExecutable and seven identical functions; both entries now share
the stub in not-executable.ts.

* feat(workflow-code): x.error compiles in the handler of an onError edge

The executor returns { error: <message> } as a failed block's output when an
error edge leaves it (BlockExecutor errorOutput, block-executor.ts:738-740,
returned at :868-877). Compile accepted no such path, so the INDEX promise of
${x.error} failed with W_UNKNOWN_OUTPUT_PATH. Now x.error checks once the edges
are known: accepted with an error edge, refused with a hint to wire onError
without one. Block<O> types error in the SDK, INDEX says how the handler reads
it, and the integrations cookbook shows it.

* fix(workflow-code): compile stored() in a loop or parallel's meta.extra

The printer writes a container param holding a detected credential as stored()
under meta.extra, but compileContainer passed no keep callback, so the print
failed to compile. Keep the stored value as a regular block's extra does.

* fix(workflow-sdk): never embed a dotfile from the cookbook

WORKSPACE_COOKBOOK embedded every file on disk under cookbook/workspace, so a
.DS_Store or editor swap file would be written into every user's example
directory. The listing now lives in embedded-files.ts, shared with the drift test.

* docs(knowledge): say why an upload's processing relay is safe on every surface

* perf(workflow-code): read each node's successors once in the cycle check

stronglyConnected called successors on every step of a frame, and the caller
maps the node's whole outgoing list each time, so a block wired to many targets
cost O(deg^2). Read them once when the frame is pushed.

* perf(workflow-code): test a restored inactive value with withStoredValue

staysInactiveWithPrev rebuilt the block's whole activity inputs (every param,
the modes, the ids deploy reads) for each inactive id on each pass. Build them
once per pass and swap the one value in, tracking the pass's restores.

* fix(workflow-code): keep collapsed on inputFormat fields whose id stays inline

Canonicalize drops the editor's collapsed flag from every field, but materialize
restored it only for fields whose id moved to the lock, so a field with a shared
or missing name lost it whenever the list changed.

* feat(sim-cli): no login exits 13 with AUTH_REQUIRED

Not logged in, an OAuth login that expired and cannot refresh, and a 401 from
the server exited 1 with CLI_ERROR, the same as a source with errors. They now
exit 13 (AUTH_REQUIRED_EXIT_CODE) with the code AUTH_REQUIRED (a 401 keeps the
server's code when it sends one). A status still falls back to the local view
without a login, and its note now reads as sentences.

* fix(workflow-code): write a subflow collection holding a $sim object back unwrapped

Canonicalize and compile wrap an object with its own $sim key as a literal node,
but materialize wrote forEachItems and distribution raw into block.data and the
loops and parallels maps, so after any config change the executor iterated the
wrappers. Unwrap literal nodes there, and say so in jsonField's TSDoc.

* fix(sim-cli): source diff filters resolve against --dir; cancellation and 413 keep their exit

- diff ./, an absolute path or a path from the current directory selected
  nothing and printed "No local changes."; filters now resolve against --dir and
  one that selects no file is refused (SOURCE_PATH_UNMATCHED).
- A cancelled request carries REQUEST_CANCELLED, so status no longer reports it
  as an unreachable server; noAnswer() is the one copy of that predicate.
- workflows/workspaces source status and diff map a 413 to exit 7, as get does.

* fix(workspace-source): holding a change for one credential key keeps another key's unreachable refusal

* fix(sim-cli): an apply drops the written.json entries of the resources it deleted

The record kept every deleted resource's files, so status reported the delete
forever (exit 11) and recreating the key made get refuse with SOURCE_LOCAL_EDITS.
An entry is dropped only when its file is gone and the returned base no longer
holds the resource or blob, so a held or failed delete stays retryable.

* fix(workspace-source): a chat auth type source cannot name leaves the deployment out instead of reading as public

* perf(workflow-code): bucket wiring and order dependencies by scope once per print

Each scope's wiring rescanned the members of every enclosing scope and refiltered
their edges by edgeScope, and each scope's declaration order was handed every
dependency in the workflow: O(scopes x blocks). Bucket edges by the scope they
print in, and dependencies by the scope they order, in one pass.

* fix(workspace-source): file creates of one name check every key they may take, and a lost key claim refuses the create

* perf(workflow-code): memoize buildFieldIndex and group pair members in one pass

The index was rebuilt for every block in print and compile and for every tool,
and each canonical pair rescanned every subBlock for its members: O(pairs x
subBlocks) per call. Group the members once, and memoize the read-only index by
block, trigger mode and pinned ids.

* refactor(workflow-code): sort compiled edges with canonicalize's compareEdges

Compile kept its own copy of the comparator, and the CWM edge order is hashed
(canonicalHash), so two copies could drift.

* chore(workspace-source): unexport WorkspaceSourceChange, used only in its own file

* docs(workflow-code): move compileRegularBlock's TSDoc onto it

It sat above BlockShape's own TSDoc, documenting nothing.

* fix(workspace-source): the staging sweep removes objects before their rows, keeping a row whose object survived

* docs(workflow-code): describe LoopSpec as canonicalize reads it

The TSDoc said the spec is the loops map's config and that canonicalize never
fills defaults; loopSpec reads block.data first, never takes the map's count,
and falls back to a for loop of 5.

* refactor(workflow-code): stop exporting ConfigItem

Nothing imports it; it is only printedConfig's item type.

* refactor(workflow-code): read pinned values and the catalog block once

compileMetaParams rebuilt the map of every stored value once per pinned id, and
needsTodo looked the catalog block up twice though collectParams already holds it.

* fix(table): a reshape resolves each column update to its id, so a by-name swap renames both

* fix(workspace-source): a get leaves out a secret named __proto__ instead of printing an unapplyable secrets.ts

secrets.ts printed every stored secret, but the declaration reader refuses a
__proto__ key, so after a get the whole file stopped applying. The secret is
now listed in skipped.json with the reason; MCP header names the reader refuses
already round-trip-fail into skipped.

* perf(sim-cli): a status past one dry run prunes its whole read instead of walking the directory again

When the workflow folders exceed one dry run, status read the directory a
second time: a second walk, read and hash of every file. It now prunes the
unchanged folders from the read it already made.

* perf(sim-cli): workspaces source apply walks the directory once, inside its lock

assertWorkspaceDir walked every kind folder outside the lock before
readWorkspaceDir walked it again; the apply now checks the read it made and
runs the full check only to refuse. orphanedWorkflowFiles groups the answer's
paths by folder once instead of per lock, assertBaseWorkspace reuses
baseWorkspaceOf, and printPlan no longer shadows counts.

* fix(sim-cli): --only reads past an empty segment instead of refusing an unnamed kind

* refactor(workspace-code): un-export four unused types, name the skill locator apart from a line's offset, fix two TSDocs

* feat(workspace-source): an existing column becomes required in the same apply as the managed rows that fill it

* perf(workspace-source): list MCP servers' tools concurrently while planning tool references

* perf(workflows): a batched source read prints its workflows concurrently

* fix(workflow-types): hold a container's edges to the handle they leave from

A loop or parallel's start handle connects only into the container, and any other
handle never does. The shared scope rule ignored the handle, so the editing engine
kept a start edge to a block extracted from the loop (and an end edge to a block
moved into it): edges the canvas never draws and workflow source refuses
(E_EDGE_SCOPE), so such a workflow could not be applied back from its own source.

* fix(editing): refuse an error edge from a block with no error output

The engine accepted an 'error' handle on any block, so a trigger or a loop or
parallel container could store an error edge the canvas has no port for and
source cannot write (E_ERROR_PORT_UNSUPPORTED). It now follows the source
catalog's errorPort rule.

* fix(editing): normalize row ids only on fields declared as row lists

Row-id normalization keyed on the subblock id alone, so any field that shares an
id with a table field (supabase's JSON code 'params', 'metrics', 'headers',
'variables') was coerced to a row list: a JSON object or code text was replaced
by [] and the value lost. It now applies only where the block declares the id as
a row-list type.

* fix(workflow-code): keep the pair member a trigger-mode pair's mode does not select

Deploy reads every member of a trigger-mode pair, so the inactive-param restore
never brought back the member the mode does not select. An unchanged apply of a
block toggled into trigger mode dropped that member's stored value, and editing
the selected member dropped the other one. The pair's value is now its
mode-selected member's, and the other members keep their stored values, as on
the canvas.

* fix(workflow-code): write only the subflow config fields an edit changes

Canonicalize fills a loop's count (5) and the collection and conditions the
subflow map holds when block.data omits them; a changed config wrote all of them
back, so renaming the block a loop condition reads also stored count, collection
and condition fields nobody edited. Fields equal to the stored config keep their
stored spelling, absence included.

* fix(workflow-source): compile a block renamed in source against the block it continues

Compile matched the stored block by key, while materialize keeps a renamed
block's id through its ident (lock-ident). Renaming a block in source therefore
refused its stored() secret (E_STORED_VALUE_MISSING), turned a stored cycle into
E_EDGE_CYCLE, and reset fields source never writes (a note's color) to their
defaults. compileSource re-keys the stored model to the source's renames and
compiles again.

* refactor(sim-cli): name what source helpers do and fix their docs

- mergedGitattributes merges .gitignore too: withMissingLines.
- isMissing reads the code through errorCode, not a cast.
- workflows source check calls workspaceFolderOf directly.
- Exit codes 5, 6 and 7 document workspaces source as well.
- workflows source resources lists MCP servers; any source get writes it.
- Two TSDocs a reflow mangled.

* perf(workflow-persistence): read only live in-workspace graph rows in a batch load

loadWorkflowsFromNormalizedTablesRaw read every block, edge and subflow row of every requested id,
then dropped the ones outside the workspace, and kept archived workflows the read snapshot
discarded afterwards. Each graph query now filters by the live in-workspace ids as a subquery.
Measured on 100 live + 300 archived workflows of 30 blocks: loadWorkflowReadSnapshots median
579ms -> 151ms.

* perf(workflows): one credential lookup for a batch load's block migrations

loadWorkflowsFromNormalizedTables ran each workflow's block migrations serially inside the
REPEATABLE READ snapshot, and the credential-id migration queried once per workflow holding a
legacy account id. The batch now prefetches the credential ids for every workflow's legacy ids in
one query, and only queries again for an id a migration surfaces that the prefetch did not ask
about, so the result is unchanged: N serial round trips become one.

* fix(api): one ETag revalidation path; tagged v2 bodies are revalidated, and cached per request

- ifNoneMatchSatisfied was copied in the internal and v2 JSON builders; definition.ts now owns
  entityTagMatches and notModifiedResponse.
- The v2 builder answered a tagged 200 and its 304 with private, no-store, which forbids keeping
  the body the ETag exists to revalidate. Both now say private, no-cache, as the internal one did.
- The v2 validated-body LRU was keyed by the tag alone and served across principals and inputs,
  though present() reads the parsed request. It is keyed by tag and parsed request now, and
  entityTag documents that it must be a strong validator of the result for every caller.
- The internal builder's HEAD branch re-checked a missing authorize() that definition-time
  validation already refuses; it captures the authorize phase once instead.

* fix(workflow-sdk): codegen uses its layout helpers; INDEX.md lists legacy builders only when there are some

- generate.ts builds block, models and index paths from layout.ts; blockTypesFile
  leaves the barrel; layout.ts reads TOOL_NAMESPACE.
- @when names both fields of an and clause.
- typecheckSdkDirectory spreads SOURCE_COMPILER_OPTIONS.
- The cookbook declares ORDERS_API_KEY, which its tool reads, and says a new
  query string updates an MCP server in place; column.ttl() says one per table.
- Doc fixes: arrayOf, declaredBlockTypes order, core.d.ts names tsc.

* fix(schedules): a written zero is an invalid schedule field, not a missing one

The reason classifier read fields through getSubBlockValue (value || ''), so a numeric 0 counted
as empty. It now checks undefined, null and '' explicitly.

* docs(environment): state the non-empty invariant both env paths share

* chore(scripts): name the pure package an offender is in, not always workflow-code

* refactor(chat-deployments): one identifier-ownership check, and one live-chat read per replace pre-check

* refactor(workflows): the code tab names its directory with the shared slugify

* refactor: keep types no other module imports private

* refactor(workflows): drop the field-by-field source apply presenter

The response schema already strips the result's internal fields, so both routes present the
result as it is.

* refactor(workspace): the workflow-code flag is required of every flags producer

* refactor(utils): one mapWithConcurrency, with documented failure semantics

apps/sim had its own mapWithConcurrency next to @sim/utils/concurrency's, with different failure
semantics: it rejected through Promise.all as soon as one item failed while its other workers kept
starting items nobody awaited. Every caller now uses the @sim/utils one: after a failure no item
starts, and the first failure is thrown once the running items settle, so nothing the batch began
is still running when the caller handles it. mapSettledWithConcurrency (attempt every item, keep
each outcome) moves alongside it. createConcurrencyLimit, MATERIALIZE_CONCURRENCY and
waitWithAbort stay in apps/sim.

* perf(workspace-source): a knowledge plan never downloads a document named in neither the base nor the directory

* docs(workspace-code): CI commits back what apply rewrote, tsc step, Gmail provider

- The scripting.mdx CI job runs tsc after check (which regenerates
  workspace.d.ts), and on merge commits back the files the apply rewrote, with
  [skip ci] and contents: write; a protected branch opens a pull request instead.
- workspace-code.mdx says what check now covers (skills, child inputs, rows,
  workspace.d.ts).
- The cookbook credentials.ts shows Gmail's provider id, google-email (the id
  apps/sim/lib/oauth/oauth.ts registers for Gmail).
- The docs baseline records the changed CI fence.

* docs(cli): regenerate the CLI reference for the check help and exit 13

* test(api): the source catalog's tagged 200 is private, no-cache

* fix(workspace-source): a renamed or name-taken stored() tool reports only that, not a missing stored tool

* fix(workspace-source): a deleted chat's base no longer records its password variable

* fix(table): reshape audit entries name columns, never their ids

* refactor(workflows): drop the never-filled SourcePlan.notes, share the stale message, and read a rename's folder from the loaded workflow

* refactor(source): one managed-row key, one tool-input field list, and no unreachable fallbacks

* docs(source): gated manifest sections, the required-column 400, carried tags and the manifest limit

* chore(source): regenerate the v2 OpenAPI and MCP operations for the documentation changes

* test(workspace-code): bound the option-cap and status-grouping fixes by n vs 4n growth, not wall clock

The ratio helper is exported from @sim/workflow-code/scaling.test-helpers so
other packages' scaling tests share it.

* fix(workflow-code): x.error is a string in a JSON field

The executor's error output is the failure message, so a reference to it in a
JSON body (response, API body) is stored quoted, as compile and print decide
through refValueKind; bare, a message would break the JSON.

* refactor(workspace-code): rowKeyText reads any cell value, so the table row writer shares it

* chore: drop the apply wire type no module reads, and shrink the non-null baseline the batch load cleared

* test(workspace-source): the cookbook apply test holds the Gmail credential the cookbook declares

* fix(workflow-code): keep a stored field the printer left out as inactive

Compile fills an omitted param with its default, then keeps the stored value only
when the field is inactive, judged with the default in place. A standalone
advanced field runs once non-empty, so a stored '' choice (cloudflare's 'Active
only' certificate filter) read as active under the default and every apply
replaced it with 'all'. Inactivity is now judged with the stored value.

* refactor(utils): point the merged mapWithConcurrency callers at @sim/utils

* test(workflow-source): prove canvas, JSON API and source stay in lockstep

A seeded fuzz applies random sequences of real editing-engine operations (add,
delete, move, edit fields, connect, disconnect, insert into and extract from
loops and parallels, rename, enable, retype, trigger and advanced mode, loop
config, variables) to corpus workflows, then asserts the source of the result
applies back as a no-op and one random source edit changes exactly that field.
Failures shrink to a one-minimal edit list. Every registry block type is added
(or, for legacy and single-instance types, edited) with non-default values in
each subblock kind it has. Each divergence fixed on this branch keeps a minimal
regression. The integration suite runs the full stack on Postgres: JSON write,
editing operations, source get and apply, JSON read, and a canvas-built workflow
against the one created from its source.

* chore: drop exports the cell-rule move left unused; move the timezone baseline entry

* chore(audits): count staging's new integration double-cast in the merged ratchet

* test(workflow-source): satisfy the audits in the fidelity suites

* test(workspaces): wait with the shared sleep helper

* fix(cli): recover a command lock abandoned before its owner was recorded

The takeover claim inside the lock refreshed the lock folder's mtime before
the staleness re-check, so a lock with no owner.json never read as stale and
every command returned SOURCE_DIR_BUSY. Judge an unrecorded lock's age from
the sighting taken before the claim, and guard against a replacement lock by
checking the folder is still the one seen (dev/ino) and still stale.

* chore(helm): bump chart to 1.11.22 for the source-staging cleanup cron

* fix(workspace-source): match managed row keys after their column's coercion

planRows compared declared key cells with stored ones before coercing them,
so a number key written as "007" counted the stored row 7 as removed and the
apply refused without --allow-delete, though the keyed write coerces "007"
to 7 and matches it. Coerce the declared key with the key column's cell rule
(the reject policy a keyed write uses) before matching.

* fix(workflow-code): print a JSON code field in meta.extra as its stored text

An advanced JSON code field the serializer still forwards under an operation that
hides it prints under meta.extra, which compile reads verbatim. It printed as an
object literal, so compile stored a native object and an unchanged apply rewrote
the stored text (minified, or not at all equal when it held references). It now
prints as text there; only a builder param takes the literal form.

* fix(workflow-source): keep a block keyed __proto__ in the lock and hints

buildLockFromState, hintsFromSourceMeta and printStoredSource assigned by bracket,
so a block whose name normalizes to __proto__ set the prototype instead of an
entry: it vanished from the lock, and renaming it in source lost its identity
(compile treated it as new and reset its unwritten fields).

* test(workflow-source): regressions from the long fidelity run

A JSON field forwarded under a hiding operation, a block keyed __proto__, and the
by-design collision-key refusal the fuzz now leaves out.

* fix(cli): workspaces source status and diff refuse a directory holding no workspace source

A mistyped --dir, an empty directory, the wrong working directory (--dir
defaults to .) or a workflow folder read as clean: status exited 0 after a
dry run of no kinds, and diff printed "No local changes.". They now refuse
it with the reason and the get to run, as apply and check do.

* fix(cli): status recommends the get that restores a removed kind folder, not an apply

Removing a managed kind's folder or root file (tables/, secrets.ts) stops
managing the kind: an apply leaves its resources alone. status reported its
recorded files as deleted and recommended apply --yes, which wrote nothing, so
the next status said the same thing again (exit 11 forever) while get refused
to write the files back. When every local change is in such a kind, status now
recommends get --only <folders> --force, which restores them, and says how to
delete the resources instead.

* fix(utils): mapWithConcurrency throws the first failure in time

The shared helper threw the failure of the lowest-numbered slot, so a slow
slot-0 error masked an earlier one from another slot. The apps/sim helper it
replaced (Promise.all) threw the first failure, and the TSDoc still promises
that.

* fix(cli): workspace source commands handle a directory no one may write

Every workspace source command takes the directory's lock by creating
.sim/command.lock, so in a read-only checkout status, diff and apply --dry-run
crashed with a raw EACCES stack trace. Commands that only read (status, diff,
a dry run) now run without the lock there, since nothing can rewrite the
directory; get, check and apply are refused as SOURCE_DIR_UNWRITABLE with the
reason.

* fix(cli): status refuses a broken .sim/written.json instead of calling every file created

A merge conflict that leaves markers in .sim/written.json (two branches that
each applied) made status read every file as created and recommend an apply.
It now refuses with SOURCE_WRITTEN_INVALID and names the get that rewrites the
record. A broken .sim/base.json names its way out too (SOURCE_BASE_INVALID).

* docs(cli): status and diff help name exit 7, and the workflow guide exit 13

status and diff exit 7 when the server refuses the source as too large (413),
as exit-codes.ts documents, but their help listed only 0, 11, 12 and 1. The
workflow directory's .sim/AGENTS.md now names exit 13 (not logged in) as the
workspace guide does. CLI docs regenerated.

* fix(executor): splice already-escaped values into JSON strings as written

Escaping every reference landing inside a JSON string literal fixed values that
broke the JSON (quotes, newlines, stray backslashes), but it also re-escaped
values that were already valid string content. A deployed workflow whose env
var holds a key with literal \n escapes, or that escaped a value in a Function
block to work around the old bug, now sent backslash-n instead of a newline.
Only a value that would break the literal is escaped now; one that is already
valid string content splices raw, exactly as before.

* fix(table): write every row's cancellation when one write fails

cancelWorkflowGroupRuns cancels every row's queued jobs, then writes each row's
cancelled status. The shared mapWithConcurrency starts nothing after a failure,
so one failed write left the later rows of the page 'running' with no job behind
them; the apps/sim helper it replaced still wrote them. The writes now all
settle before the first failure is thrown.

* fix(cli): count a workflow changed only on the server as unchanged in workspace status

The status dry run sends every workflow folder whole to detect drift, so the server plans an
update for a folder edited only in Sim. An apply sends that folder by its base and leaves it
unchanged, so the status plan line said 'Applying would: 1 update' for an apply that changes
nothing.

* fix(cli): an interrupted source command releases the directory lock

Ctrl-C during a workspace source command left .sim/command.lock behind. A
later command takes a lock over only when its owner ran on the same host, so
one left in a folder shared with a dev container or another machine stayed
until removed by hand (exit 10 every time). The lock is now removed on
SIGINT/SIGTERM before the signal ends the process, as a staged write's folder
already is.

* test(workspace-source): a key-claim failure trigger left by a killed run no longer breaks the next one

* fix(e2e): expire the workflow source e2e session in UTC

session.expires_at is a timestamp without a zone that the app reads as UTC. now() writes the
database's local time, so on a Postgres west of UTC the one-hour session was already expired
and the canvas-route check got 401.

* fix(cli): get keeps a directory bound to its workspace when its base does not parse

get decided which workspace a directory belongs to only from a base that
parsed cleanly, so one this CLI refuses (a field a later CLI added, a bad
fingerprint) read as no base: get then used the profile's workspace, and
--workspace naming another one was not refused (SOURCE_OTHER_WORKSPACE). It now
reads workspaceId as leniently as apply and status do.

* fix(cli): print nested source apply results as JSON in text output

`sim workflows source apply --output text` printed the modified blocks as
`modified	[object Object]`.

* test(workflow-code): alternate small and large runs in the growth check so machine load slows both sides

* fix(workspace-source): in-use scan reads the live deployment version

A delete of a resource that a deployed workflow's live version still uses,
while its draft no longer does, now warns W_RESOURCE_IN_USE naming the
live version, unless the same apply publishes the draft or undeploys it.

* fix(workspace-source): class a refused apply by the errors that refuse it

The rejected error counted and classed every error, including those of
changes the plan holds back, which refuse nothing. The plan now returns
the refusing errors (plan-level plus non-held changes) and the error
classes from those.

* fix(workspace-source): a key taken after a committed write reports the write

A key claim that found the key held by another live resource after the
write committed threw, dropping the written resource from applied. write()
now returns it with E_NAME_TAKEN; the apply records it as applied, keeps
the base entry unchanged, and stops so no later write resolves the key to
the other resource.

* test(workspace-source): per-process source slot refuses past its cap with 429

* fix(cli): check table row columns when the row holds a non-string reference

Compiled row data stores a non-string reference bare ("n": <score.value>), which JSON.parse
refuses, so check skipped the whole row: an unknown column or select option in any row writing a
number, boolean or object output went unreported. Parse it as a JSON field instead.

* feat(workflow-code): hint api for http.request and fn for javascript.* builders

An agent guessing the HTTP request block as http.request got E_UNKNOWN_BUILDER with no hint:
no edit distance reaches api.

* docs(workflow-code): correct workspace-key, rows, written.json and create claims; fill agent-guide gaps

- credentials bind with a workspace API key; secrets and deployments do not (guide, CI docs, API description)
- table rows are source when a table opts in; list rows.jsonl and the limits
- .sim/written.json is committed, not gitignored; drop stale tsc sizes; exit 9
- a created-not-yet directory has no base: create --dry-run, not status
- guide: api is the HTTP block, W_* print as error in check, self-hosted apiKey, promote setup

* test(workspace-source): a key-claim failure trigger left by a killed run no longer breaks the next one

* test(workflow-code): alternate small and large runs in the growth check so machine load slows both sides

* test(workflow-code): time the block-cap round trip by thread CPU, so machine load cannot fail it

* fix(workflow-code): never print a credential through the response body form

The response(name, { body, status, headers }) form printed its fields'
stored values verbatim, so a response block holding a detected credential
(an Authorization header, a token in the body) printed the secret into the
file while W_LITERAL_SECRET claimed it printed as stored(). Such a block now
keeps its own params, where the field prints as stored(), and compile warns
W_LITERAL_SECRET when source writes one in the form.

* fix(workflow-code): check agent tool calls the way blocks are checked

compileToolCall accepted any builder as a tool (api.tool, condition.tool,
trigger.gmail.tool.poller), stored any param key unchecked, and never warned
about a literal secret, though print writes these tools as raw(tool) and
warns W_LITERAL_SECRET. A tool call now needs a builder with a tool form
(E_UNKNOWN_BUILDER), each param must be one its tool type declares
(E_UNKNOWN_PARAM with a did-you-mean, W_FIELD_NOT_APPLICABLE for another
operation's), and a literal secret warns as on a block.

SYNTAX.md and core.d.ts no longer document agent.tool() or
runWorkflow.tool(): neither is typed, and print writes such tools as
raw(tool).

* fix(workflow-code): refuse stored() and codeFile() in a loop or parallel config

compileConfig dropped any config value that was not a plain value, so
loop.forEach('L', { collection: stored() }) or a codeFile() condition
compiled silently to a loop with no collection or condition. Both are now
E_UNSUPPORTED_SYNTAX with the form to write instead.

* fix(workflow-code): give every compile finding a hint that names its fix

E_TARGET_TRIGGER, E_ERROR_PORT_UNSUPPORTED (a loop or parallel has none:
wire onError from its children), E_UNKNOWN_BRANCH in meta.branches (lists
the branches), W_VALUE_KIND (the literal form the field takes) and
E_UNKNOWN_IDENTIFIER for undefined, NaN and Infinity had no hint. The test
catalog's loop and parallel now have no error port, like the real catalog.

* fix(table): keep batch rows that keep a unique value in the DB check

batchUpdateRows excluded every batch row from each unique column's DB check and
compared the rest by the values it read before taking the value locks. A row the
batch leaves alone could take the value from a concurrent edit after that read,
so both it and the batch's writer stored the same unique value. Each column's DB
check now skips only the batch rows that rewrite that column.

* docs(tables): say add-column refuses a required column rows leave empty

* fix(cli): Ctrl-C during an apply write names what it may have done

An interrupt while the apply request was in flight exited 130 silently, so
nobody learned the write may have landed. The lock's interrupt handler (which
already releases the lock) now prints the same next step a lost answer gets,
then ends with the signal.

* fix(cli): a status without a login recommends sim login

With no login on the profile, status shows the local view (the server is
treated as unreachable) and its note says to log in, but next was
apply --dry-run, which only exits 13. next is now the login, then status again,
for workflows and workspaces alike.

* fix(cli): check offline with no cached catalog names the get that caches it

A fresh clone checked while the server could not be reached failed with the
raw "Could not reach …" and no next step. It is now CATALOG_NOT_CACHED (exit
1) naming sim workspaces source get, as a check with no credentials already
is; no login keeps its AUTH_REQUIRED refusal (exit 13, naming sim login).

* fix(workflow-code): keep meta.retry within the bounds the loader pins it to

Persistence saves a retry policy verbatim but loads it clamped (2-5 tries,
0-5000 ms, whole numbers), and canonicalize kept a well-formed policy raw.
A file writing maxTries: 10 stored 10, read back as 5, and every re-apply
of the unchanged file showed a retry change. Compile now refuses an
out-of-range policy with a hint naming the bounds, and canonicalize pins
the numbers as the loader does, so a printed policy always compiles back
unchanged.

* Revert "fix(workflow-types): hold a container's edges to the handle they leave from"

This reverts commit 62872bd29b.

* chore: regenerate the OpenAPI spec and MCP operations for the workspace-key description

* docs(cli): document exit 13 (AUTH_REQUIRED), exit 9 and the 429 resend policy

Both exited 1 before this branch. State which requests are resent (every GET and the
all-or-nothing source writes), the 6-retry count and the one-minute wait cap, in the
scripting exit-code section, troubleshooting, the CLI reference preamble, the README and the
source agent guides.

* fix(workspace-source): managed table rows guard a delete and a column renamed on the server

An --allow-delete of an undeclared managed table whose rows were written
since the base deleted those unseen edits; it is now E_APPLY_STALE. A rows
edit on a table whose declaration is unchanged while the server renamed or
removed one of its columns wrote the cells by names the table no longer
has, silently dropping them; it is now E_APPLY_STALE.

* fix(workspace-source): a stored() tool or MCP server deleted on the server stays deleted

An unchanged directory whose tool code or MCP header is stored() was
refused (E_APPLY_STALE / E_TOOL_STORED_MISSING / E_MCP_STORED_MISSING)
once the resource was deleted on the server, blocking every other kind.
It now reads as I_SERVER_DELETED like any other deleted resource.

* docs(workflow-sdk): document a child workflow's output shape

runWorkflow's result is the child's first response block output ({ data, status, headers }),
else its last block's output, so a response body field is result.data.field; a deployed parent
runs the child's deployed version. Stated in INDEX.md's How it runs, the runWorkflow TSDoc, the
agents example and the agent guide.

* fix(workflow-code): round-trip stored container edges source cannot draw

Stored workflows hold container edges the canvas never draws: a loop or
parallel start edge to a block outside it (the editing engine keeps one when it
extracts a block, and the executor runs it as the container's exit), an end edge
into the container's own child, and an error edge from a container. Source
printed the first two through p.to()/x.to() and every one through onError, and
compile refused them (E_EDGE_SCOPE, E_ERROR_PORT_UNSUPPORTED), so such a workflow
could not apply its own source back.

The printer now writes a mismatched container edge as edge() with its stored
handle, compiled with W_SUBFLOW_HANDLE_SCOPE (the warning names the exit form),
and compile keeps an error edge from a block with no error port when the stored
workflow holds it (W_ERROR_PORT_STORED). The corpus gains both edge shapes.

* fix(editing): keep a stored error edge the engine no longer adds

Since the engine refuses new error edges from loops and parallels, an edit that
re-sends a container's connections dropped the error edge it already stored.
Error edges the graph held before the batch are kept; only new ones are
refused.

* test(workflow-source): stored mismatched container edges survive every write path

The extract and move-into-loop regressions now assert the engine keeps the
edge and source round-trips it; a stored container error edge survives an
engine edit that re-sends its connections; and an integration check writes a
loop start edge to an outside block through PUT /state, edits through the
engine, and applies the source back unchanged.

* fix(workspace-source): type the in-use scan rows for the query builder

* fix(tables): accept a required column on a table with rows again outside workspace source

facb1c4f4d made every surface refuse adding a required column that existing rows
leave empty, which turned a call third-party integrations make on the v1 and v2
table APIs into a 400. The refusal is now an addTableColumn option only
workspace-source apply sets, so its plan and apply agree while the table APIs,
Copilot and undo keep staging's behaviour. Restores staging's undo hook,
OpenAPI description and contract comment.

* fix: attempt every independent side effect of a fan-out after one fails

The shared mapWithConcurrency starts nothing after a failure, where the apps/sim
helper it replaced kept starting the rest. For fan-outs of independent side
effects that left Slack targets undispatched, tombstones and cancellation
writes unwritten, stale-dispatch cancel events unsent, project file retirements
and chat image sweeps unrun. mapEachWithConcurrency attempts every item, then
throws the first failure; those callers use it.

* chore(e2e): name the workspace source e2e report variable like its siblings

WORKSPACE_SOURCE_REPORT_PATH becomes WORKSPACE_SOURCE_E2E_REPORT_PATH, matching every other
source e2e's <SUITE>_E2E_* variables; the CI http-e2e script passes the new name.

* feat(workflow-code): warn on a condition or router branch no edge leaves

W_BRANCH_UNWIRED names the branch and the wiring that fixes it. A branch the stored workflow
already leaves unwired is not reported, and check fails on it only for a block the lock does not
bind, so existing workflows keep checking clean.

* fix(workflow-code): read a Start block's untouched blank input row as no fields

Every new workflow's Start block is seeded with one unnamed string field, which execution skips,
so `get` on a fresh workflow printed `inputFormat: { '': 'string' }`. A list holding only that
starter field now canonicalizes as empty and prints `inputFormat: {}`; an unchanged apply keeps
the stored field byte for byte, and a blank field beside named ones still prints as stored.

* fix(workflow-source): fingerprint a stored workflow the same for every reader

The sf1 fingerprint hashed the reader's canonical model, which names a
credential the reader may see by key and one they may not see by id, so a
member applying source a teammate read got a false 409. It now hashes every
credential by its stored id. A credential key the reader cannot resolve,
standing where the stored workflow holds a credential they cannot see, keeps
the stored credential whatever the key (W_APPLY_CREDENTIAL_KEPT), so an
unchanged or edited teammate source applies without confirming any key.

* fix(workflow-source): refuse workflow names and descriptions v2 refuses

Source could set an empty or whitespace name, a name over 255 characters or
a description over 50,000, which v2 create and update refuse. The plan now
refuses the fields a write sets with E_WORKFLOW_META_INVALID at the header,
reading the limits from the v2 workflow contract.

* fix(workspace-source): keep unkeyed managed rows out of source, and check required fills at plan

A row the table grid's "add row" makes has an empty key cell, so a get printed it as a line the
apply refuses (E_TABLE_ROWS): get -> apply of an unchanged directory failed. Such a row has nothing
a line can match it by, so it is no longer managed: a get leaves it out of the rows file with
I_TABLE_ROWS_UNKEYED, neither rows fingerprint counts it, and the keyed write never removes it.

Making a column required trusted the declared rows to fill it even when no rows write was planned,
so a row added in the UI made the apply fail mid-reshape. The declared rows now stand for the
table only when a managed rows write is planned, and the rows that write keeps (unkeyed ones) are
checked too.

* fix(workspace-source): compare managed rows with the base under the key it recorded

Changing a managed table's rows key and editing a row in one apply was a false E_APPLY_STALE: the
server's rows were fingerprinted under the new key and compared with a base fingerprinted under
the old one. The staleness check now reads the server under the key the table is still marked by.

* fix(workspace-source): a reader's concealed MCP get applies back unchanged

A get by a member who may not rewrite headers prints every literal header value as stored(); the
plan resolved each stored() to a keyed hash, while a value that is not a credential prints (and
fingerprints) as its text, so a writer applying that directory unchanged always planned a header
update. stored() now resolves to the value as a get prints it.

* fix(workflow-source): answer a racing source rename with SOURCE_NAME_TAKEN

Two source renames onto one name in one folder both passed the name SELECT;
the second hit the unique index and answered 500. The commit now maps that
unique violation to WorkflowSourceNameTakenError, and the apply route answers
it, like the create route, with 409 SOURCE_NAME_TAKEN.

* fix(workspace-source): an unchanged file the server can no longer hand to source is a no-op

A declared file edited past the blob cap in the Files tab, or one an older file moved beside it
now takes the path of by letter case, refused the whole apply of an unchanged directory
(E_FILE_TOO_LARGE / E_FILE_PATH_COLLISION). Unchanged since the get that read it, the declaration
now leaves the file as the server has it (I_SERVER_CHANGED); a directory that changes it is still
refused.

* fix(workflow-source): report a dry run's needsRedeployment for the draft it would write

A dry run answered needsRedeployment for the draft as stored, so an edit of a
draft in step with its deployment said false and a revert said true. A dry run
that would change the draft now compares the draft it would write with the
active deployment; a no-op or refused dry run still reports the stored draft.

* fix(workflow-source): record no refused apply for a plan-only evaluation

A workspace plan or dry run evaluates each workflow planOnly, so planning a
locked workflow recorded workflow_source_applied ok:false although nothing was
applied. A planOnly run records nothing; the commit that follows records its
own outcome.

* fix(workspace-source): a knowledge document replace that stops partway resumes without losing tags

A replace uploaded the replacement under the document's own name, then carried the old one's tags
and deleted it. Stopped after the upload, the next apply deleted the old document without
carrying its tags; stopped and then edited again, the two same-named documents read as a server
change (E_APPLY_STALE) until deduplicated by hand, since nothing told the replacement from a
document uploaded twice in the UI.

The replacement is now uploaded under a name that names the old document, takes its tags, the old
one is deleted, and the replacement is renamed. The next apply finishes a stopped replace whose
replacement holds the declared bytes (tags carried, old deleted, renamed), or removes the
replacement and replaces again; a replacement left before its rename stands for its name. The
duplicate-by-hash rule this replaces is gone: an unmarked duplicate is ambiguous, as any other.

* test(workflow-code): compare n with 8n in the growth check, so allocation noise on a slow runner stays far from the quadratic bound

* fix(workflow-source): never keep a hidden credential over a key the apply binds

A credential key the enclosing workspace apply binds in the same run is not an
unknown key, so it resolves to that credential rather than keeping the stored
one.

* test(workflow-code): read and write input-format rows through their real shape in the API-built round trip

* fix(workflow-code): keep a hidden credential on a block the source renames

compileSource now returns the stored model it compiled against, re-keyed by the
source's renames, and keptCredentials pairs compiled blocks with it, so a member
who cannot see a credential can rename the block holding it.

* fix(sim-cli): offline check passes stored container error edges, cycles and branches on bound blocks

The offline check compiles without the stored workflow, so it refused an
onError on a loop or parallel and a cycle that a get printed and the server
keeps, and told the agent to delete them. compileWorkflow takes storedUnread:
an edge between two lock-bound blocks may be stored, so it reports
W_ERROR_PORT_STORED / W_EDGE_CYCLE saying an apply keeps it only if the stored
workflow holds it, and the error stays for a new block. E_ERROR_PORT_UNSUPPORTED
hints no longer say to remove the edge. W_BRANCH_UNWIRED never fails the check
on a bound block, --strict included, matching its stored exemption.

* fix(serializer): never evaluate a viewer-scoped default on the server

* fix(sim-cli): keep a downloaded workflow's stored inputs typed in workspace.d.ts

* fix(sim-cli): exit 13 when the server refuses a stored login's refresh token

* fix(workflow-code): offline check binds a block renamed in source through its lock ident

storedUnread now takes the lock, and compile counts a source block whose ident is one vacated
lock entry's as bound (lockIdentRenames, materialize's lock-ident rule, shared with
compileSource), so a renamed block's stored container error edge or stored cycle warns offline
as the server accepts it.

* fix(sim-cli): regenerate workspace.d.ts from local workflow inputs on a workflows check

A per-folder `sim workflows source check` in a workspace directory already
compiled every workflow's Start inputs for its child-call findings but left
workspace.d.ts on the stored inputs, so a parent mapping an input added to the
child since the get failed tsc with an excess-property error. It now refreshes
workspace.d.ts the way `sim workspaces source check` does.

* fix(workflows): refuse new edge cycles in the editing engine as edge_cycle skips

The edit engine (Copilot edits, v2 operations) could create an edge that
closes a cycle, which the canvas and realtime refuse and the executor
cannot run. createValidatedEdge now runs the shared wouldCreateCycle check
and reports a refused edge as an edge_cycle skipped item. Edges the
workflow already stored are exempt, so a stored cycle survives unrelated
edits and restated connections. Publishes edge_cycle in the v2 skipped-item
enum and regenerates the OpenAPI document and CLI types.

* fix(workflow-code): declare lock and base as type aliases so they fit the open generated API types

* feat(mothership): sync the workflow-code entitlement into the generated protocol

* test(workflow-source): stamp updatedAt on direct block writes so a migrate-on-read save cannot undo them
2026-11-19 12:51:01 -08:00
Waleed 5bbf4e748b feat(api): failing block on polled runs, deployed input schema, paused-runs list, and applied limits (#9906)
* feat(api): attribute polled run failures, deployed input schema, paused-run list, and applied run limits

- GET /v2/workflows/{id}/runs/{runId} reads the failing block back from the run's
  recorded trace, withheld when the viewer's permission group hides execution data
- GET /v2/workflows/{id} adds inputSchema for the active deployment's API run input
- GET /v2/workflows/runs/paused lists a workspace's runs waiting on human input,
  projecting pause points exactly as the run detail does
- Execute responses report executionTimeoutSeconds; GET /v2/billing/status adds plan limits

* fix(api): respect the paused-run page cap in the CLI and keep waiting pause points under the cap

- CLI generator carries a numeric field's contract maximum; the pager never requests
  a page above it while still paging to the caller's total --limit
- Pause point cap keeps points not yet resumed before resumed history, so a run that
  paused again is never listed without its waiting point (run detail and paused list)

* improvement(api): reuse FailedTraceBlock and sort pause points once when capping

* fix(api): custom-block trace boundary, legacy pause point arrays, and list point selection

- A custom block is an opaque trace boundary for failing-block attribution
- The paused-run list reads object and legacy array pause_points
- The list selects waiting human points before the per-run cap, reads points in
  waves that stop once the page budget is spent, and shares one waiting predicate
- Resume response schemas omit executionTimeoutSeconds
- Drop a non-null assertion from the CLI page cap test

* fix(api): attribute concurrent failures to the stored error's block and never fail a workflow read on its deployment

- Failing-block attribution prefers the span the stored error names, else the
  failure that ended first, else none; the block error prefix has one owner
- The workflow read degrades inputSchema to null when the deployment cannot be
  read or its schema has an unexpected shape
- Integration cases for time-only, archived, and cross-workspace exclusion

* fix(api): consistent trace walk, block-named error attribution, required applied timeout, and ungated input schema

- Failing-block attribution walks spans as hasUnhandledError does, through
  successful blocks, with one shared successful-boundary rule
- An error message headed by the failing block's name is attributed to it even
  when the name reads as a run-level failure
- executionTimeoutSeconds is required (nullable) on Execute workflow responses
- The workflow read takes inputSchema from the active version row, not isDeployed
- Pause cap wording: unresolved points are kept in preference, listed earliest first
- Regenerate the CLI API the previous contract wording left stale

* fix(api): parse resume responses with the resume schema in e2e scripts and regenerate the CLI API on staging

- Stop-after and pinned-execute e2e scripts parse resume responses with
  v2ResumeWorkflowDataSchema, which omits executionTimeoutSeconds
- Regenerate the CLI API against the merged staging contracts

* fix(api): parse the stop-after e2e resume helper's response with the resume schema
2026-11-11 15:52:59 -08:00
Waleed 3d8996f0e6 feat(api): workflow outline, workspace schedules list, clearer v2 errors, and catalog value types (#9905)
* feat(api): add traceId and field paths to v2 errors, name the read-only key remedy

* feat(api): publish declared value types on block catalog fields

* feat(api): add workflow outline and workspace schedules reads to v2

* fix(api): type declared catalog inputs as block params and regenerate the docs manifest

* fix(api): compare member responses without their per-request traceId

* improvement(api): tidy catalog field projection default and integration test comment

* docs(api): say the outline's derived fields come from values state already returns
2026-11-11 14:26:20 -08:00
Waleed ba66dfd6b7 feat(api): agent identity on v2 meta, scoped CLI pairing keys, and v2 workspace rename (#9904)
* feat(api): agent identity on v2 meta, scoped CLI pairing keys, and v2 workspace rename

- GET /api/v2/meta adds keyId, userId, workspaceId and links {openapi, docs, mcp};
  existing fields unchanged. The API catalog shares the same link source.
- CLI pairing login accepts --read-only and --expires-in <days>. The request rides
  the approval URL, is shown on the approval card, is recorded with the approval
  and applied at mint; the unauthenticated poll cannot set either. The poll echoes
  access and expiresAt, and the CLI refuses to store a key that does not match
  what was asked (an older server mints full access). The signed-out login bounce
  now keeps scope, workspace, access and expiry instead of dropping them.
  sim whoami shows the key's expiry and bound workspace.
- POST /api/v2/credentials/connections returns draftId. Opening a connection link
  that is unknown, expired or another account's now explains to sign in as the
  creating account, without saying which case applied.
- PATCH /api/v2/workspaces/{workspaceId} renames a workspace or changes its logo
  through the existing workspaces.identity.update operation, widened to personal
  API keys and OAuth tokens (api:write); Admin still required, workspace keys
  still denied. Workspace update operations move into the operations registry.
- Regenerated OpenAPI, MCP catalog, CLI client and CLI docs.

* fix(cli-auth): pass route context in the new approve and poll tests

* test(mcp): expect the identity fields getMeta now returns

* fix(workspaces): count members before the rename so a failed count never hides a committed update

* improvement(cli-auth): reuse the api-key access schema and tidy handoff validation

* fix(api): publish the workspace PATCH body rules and check CLI-minted keys exactly

- PATCH /v2/workspaces/{id}: the body schema publishes minProperties 1, and
  logoUrl is a pattern (root-relative path or https URL, protocol-relative
  refused) instead of an unpublished refine.
- sim login refuses a minted key whose access or expiry differs from the
  request in either direction, with the expiry checked against the requested
  days within a clock tolerance.
- sim workspaces update --logo-url documents that clearing the logo goes
  through the API or MCP.
2026-11-11 14:26:02 -08:00
Waleed c844be83ac docs(api): guide for building and running workflows with the API (#9901)
* docs(api): add a guide for building and running workflows with the API

* docs(api): point the workflow guide at the Sim skills

* docs(api): pass only the operations array to the CLI in the workflow guide

* docs(api): keep polling while a run is redacting
2026-11-11 14:15:58 -08:00
Waleed 4711852316 feat(api): workflow trigger health and enablement through the v2 API (#9853) 2026-11-10 13:51:22 -08:00
Waleed 117409226f feat(api): manage workspace members through the v2 API (#9852) 2026-11-10 13:32:52 -08:00
Waleed 5836fce6e9 feat(search): manage source permissions through a shared API and CLI (#9845)
* feat(search): manage source permissions through a shared API and CLI

* chore(search): document the shared permission grant format
2026-11-10 11:16:43 -08:00
Waleed 5bffff69cb feat(search): update GitLab permission CSVs through the API (#9841)
* feat(search): update GitLab permission CSVs through the API

* fix(search): align GitLab CSV updates with CLI and API policies

* test(cli): cover confirmation for GitLab permission replacement
2026-11-10 10:47:44 -08:00
Waleed ff98c86570 feat(knowledge): import knowledge base bundles through the API and sim knowledge import (#9327)
* feat(knowledge): import knowledge base bundles through the API and `sim knowledge import`

- v2 knowledge imports resource (create, part URLs, complete, get, cancel) over a new knowledge_import upload purpose
- background runner recreates the base, tags, and documents from an exported bundle; reuses vectors when the embedding model matches and queues re-processing otherwise
- knowledge_base_import job table with a stale-import janitor sweep
- `sim knowledge import <bundle.simkb.zip>` plus `knowledge imports get/cancel`, and an import guide
- shared dispatchBackgroundTask helper for Trigger.dev-or-in-process task dispatch

* fix(knowledge): validate bundles before writing, close import cancel and cleanup gaps, and fix CI

- read every chunk entry before the base is created, reject reversed offsets, and length-check vectors before decoding
- confirm the import is still running before creating the base, and remove it if a cancel wins
- record the import inside upload completion, so a refusal or crash cannot strand the bundle; re-read on a concurrent completion
- serialize progress writes, keep the source character count, cap rebuilt text, and recheck uploads when issuing part URLs
- janitor fails in-process documents a dead worker left processing, and deletes bundles when pruning settled imports
- shrink the unused-exports baseline, count the new janitor passes in its test, and fix the integration test's Blob typing

* fix(knowledge): re-dispatch an import whose first dispatch was lost, and remove a base whose tag setup fails

- the base takes the import's id and a worker claims the import before working, so a second dispatch exits untouched
- a retried completion dispatches again an import no worker has claimed
- a worker that finds its import settled before any claim deletes the unused bundle
- tag-definition setup that fails removes the still-empty base

* fix(knowledge): keep a failed dispatch from stopping a live import, audit each import once, and close review gaps

- a dispatch never settles the import itself; a retry or the stale-import janitor does
- projected audit entries take an idempotency key, recorded through recordAuditOnce, so a retried completion audits the import exactly once
- a cancel that races completion stops the record the completion wrote
- refuse empty rebuilt chunk-only documents; knowledge_base_import.user_id references the user

* fix(knowledge): keep import records through account deletion, and recheck uploads on re-dispatch

- knowledge_base_import.user_id is attribution only, so the janitor's cleanup record outlives its importer's account
- re-dispatching an unclaimed import repeats the knowledge.upload assertion

* fix(knowledge): audit an import as started, and drop the mock-only dispatch test

- the completion records knowledge_base.import_started; the base exists only once the runner creates it
- background-task.test.ts only checked forwarded mock arguments; the delete-async route test and the integration suite exercise the helper's real paths
2026-10-22 19:29:41 -07:00
Waleed 3ba393dac2 fix(file-doc): stop the permission resend loop and restore the 10 MB proxy body limit (#9057) 2026-10-14 17:19:50 -07:00
mzxchandra bca78367ac feat(project-files): add core storage and realtime operations (#9016)
* refactor(files): unify mutation accounting and retention

* feat(project-files): add core storage and realtime operations

* feat(files): unify document read subscriptions and write capabilities

* fix(files): resolve capability type and artifact checks

* fix(docs): hide unreleased Project file references

* fix(files): address review admission and collaboration gaps

* test(cli): load canonical documentation in suite setup

* fix(files): bound seed reads and preserve browser snapshots

* fix(files): keep read provenance private unless requested

* fix(files): classify raw Project downloads as opaque
2026-10-14 16:14:37 -07:00
Waleed 2676575f21 improvement(docs): standardize copy and responsive navigation (#9020)
* docs: standardize copy and page organization

* docs: preserve tab spacing and qualify output limits

* docs: align responsive navigation and preserve readable content

* docs: link the scoped tab component in agent guidance

* docs: preserve procedural details and correct reference examples

* docs: record final review and preserve accessible provider bookmarks

* chore(docs): finish ship review and update audit evidence

* fix(docs): correct review findings and tablet header alignment

* fix(docs): polish API controls and inline code

* fix(docs): announce API picker selections

* fix(docs): expose checked API picker choices

* test(ui): align single-choice picker locators
2026-10-14 12:29:33 -07:00
Waleed 3dafafda08 feat(analytics): attribute requests by client surface and add CLI usage telemetry (#7763)
* feat(analytics): attribute requests by client surface and add CLI usage telemetry

* fix(analytics): tolerate get-only header readers, gate CLI notice on stderr, re-read telemetry state before send
2026-09-10 22:44:07 -07:00
Vikhyath Mondreti ccc6f9ed80 feat(workspace-sync): add portable imports and v2 fork workflows (#7700)
* feat(workspace-sync): add portable imports and v2 fork workflows

* fix(workspace-sync): preserve activity attribution and update boundary tests
2026-09-09 18:03:08 -07:00
Waleed 7e9dd2af8c improvement(docs): simplify navigation and share editor previews (#7606)
* improvement(docs): simplify navigation and share editor previews

* fix(docs): preserve focus in workflow illustrations
2026-09-08 14:20:45 -07:00
Waleed 9b76f6e33a chore(docs): remove i18n and the five translated locales (#7247)
The docs shipped `es`, `fr`, `de`, `ja`, and `zh` alongside `en` through
fumadocs i18n and a lingo.dev translation pipeline. Neither is used any
more, so this removes the locale layer end to end rather than leaving a
vestigial `en` directory behind it.

- Delete the five translated content trees (995 files) and flatten
  `content/docs/en/**` up to `content/docs/**`.
- Delete `lib/i18n.ts`, `i18n.json`, `i18n.lock`, and the `i18n.cache`
  gitignore entry; drop `createI18nMiddleware` from `proxy.ts` and the
  `i18n` option from the source loader.
- Collapse `app/[lang]/**` into `app/**`, merging the locale layout's
  html/body shell into the root layout. The OpenAPI virtual pages moved
  off `baseDir: 'en/api-reference/(generated)'`, which was the `en`
  segment the dir parser used to consume.
- Strip locale branching from the sitemap, `llms.txt`, `llms-full.txt`,
  the `llms.mdx` route, the sidebar's active-path matching, structured
  data, and the hreflang/`alternateLocale` metadata.
- Simplify the search route to English-only: one `english` text-search
  config, unconditional vector search, and no locale-prefix filtering of
  `sourceDocument`. The docs-embeddings job already runs with `--clear`,
  so the now-unprefixed paths are rewritten on the next run.
- Repoint every script, test, and the CI path filter at `content/docs`,
  and de-loop the API-reference sidebar checks over the six locales. The
  execution-guide wire-shape test covered only the removed locales; it is
  retargeted at the English guides, which already satisfy it.

`check:audits` (39 audits), the docs build (666 paths), the docs and
OpenAPI suites, and the netsuite docs-path test all pass.
2026-08-28 18:34:20 -07:00
Waleed c930830310 fix(v2,cli): second audit pass over the v2 API and CLI (#7126)
* fix(chat): resolve a caller-supplied conversation id through its owner

The v2 chat route used the caller-supplied conversationId verbatim, with no
existence, owner, or workspace check, against a store keyed by bare text with
no owner column. A caller who knew another user's conversation id reached that
conversation. Ids now resolve through the same owner-scoped loader the web chat
path uses, and anything unresolvable answers one uniform 404 before any
lifecycle work runs. Omitting the id mints a server-issued conversation.

The contract also accepted any 1-128 character string for a column typed uuid,
so a malformed id raised a driver error and rendered 500 while an unknown but
well-formed id rendered 404 - a shape oracle, and a 500 on ordinary input.

The ownership predicate had no coverage anywhere: the route test mocked the
module and the lifecycle test drove a chain mock that ignores its where clause,
so deleting the owner condition left both suites green. It is now asserted by
composition and by condition count, which is what catches a dropped condition.

Also renames the reply's model identifier away from a term the project's own
copy rules forbid on a user-facing surface.

* fix(v2): conceal workspace absence, and stop archived tables faulting their page

Two reads answered a caller more than they were entitled to know.

A workspace a caller cannot reach at all returned FORBIDDEN while one that does
not exist returned NOT_FOUND, so a workspace-key holder could enumerate which
workspace ids exist by diffing the two. Both now answer the same absence, using
the concealment policy the billing routes already use. A refusal from inside the
workspace - a member whose role is too low - still answers FORBIDDEN, because
that caller already knows the workspace exists.

Separately, archiving a folder cascades onto its tables but leaves each table
pointing at the archived folder row. The archived listing resolved those paths
strictly, so one such row faulted the whole page and no cursor could step past
it - which also made the ids undiscoverable and left restore unreachable for
exactly the tables that need it. The archived scope now resolves leniently to
the root, where a restore would place them, matching the shipped workflows
behavior. Active listings still fault loudly on a dangling folder.

* fix(knowledge): validate upload processing options without stranding live sessions

recipe and lang were accepted as free strings up to their length caps, silently
discarded, and echoed back nowhere, so a typo was unobservable: uploading with a
misspelled recipe returned 200 and quietly used the default. Both are now
validated at the boundary and a bad value answers 400 naming what is accepted.

The accepted recipe set deliberately includes the sentinel every first-party
caller sends today alongside the three real chunker recipes, and the three are
derived from the chunker's own union so removing one there is a compile error
here rather than a silent 400 in production.

The same schema also parses metadata read back off a persisted upload session,
so tightening it would have thrown out of resume and complete for any session
created before this - a 500 on work that could then never finish. The read-back
path now drops a value it no longer recognises instead of rejecting it; the
request boundary stays strict.

Neither field reaches chunking, so nothing here moves chunk boundaries,
embeddings, or search results.

* fix(v2): honour a requested stats window, and answer a claimed graph id with a conflict

Log statistics accepted a start and an end, filtered the totals by them, and
then built the series against wall-clock now. Bucket width was computed over a
span the caller never asked for, and every bucket past the requested end was
structurally empty - so a bounded historical query returned a wrong-width series
with fabricated trailing buckets, under a window label that disagreed with the
request. Each edge now honours the bound it was given and keeps its previous
derivation when omitted, so an unbounded request is unchanged.

Separately, block, edge and subflow ids are global primary keys while the
delete that precedes a state replace is scoped to one workflow. An id owned by
another workflow survived that delete, the insert violated the key, and because
callers pass their own transaction the driver error escaped unclassified as a
server fault. The write now refuses such an id up front with a conflict naming
it, and re-classifies the same violation if one races past the check, since the
lock covers only the workflow being written. The dry run checks the ids a commit
would insert and reports the warnings a commit would report, which is what its
own contract already promised.

* fix(secrets): let a workspace secret change its metadata without resending the value

Restoring redaction cost more than removing it. The only way to flip a secret
back to redacted was to re-send the plaintext, because the write required a
value and omitting it fell into an interactive prompt that cannot run in CI.
A workspace secret can now change its description or visibility on its own; the
stored value is never re-encrypted or rewritten, a write that names no existing
secret answers not-found rather than creating one, and a personal secret still
requires a value because it has no other writable field.

The path parameter was also one shared schema across the write and the delete,
so a single description had to cover both and the delete documented an argument
that could create and replace. Split, mirroring the credentials pair.

The metadata write is a new update against the credentials table, so its scope
is asserted by composition and by condition count: an unscoped update would let
one workspace flip another workspace's identically-named secret out of
redaction, and the cache invalidation would then carry that flag into the other
workspace's runtime catalog.

* fix(v2): say what an error means in terms the caller can act on

A size-limit refusal collapsed every value under a kilobyte to "0 Bytes", so a
28-byte file over a 27-byte ceiling read "is 0 Bytes, above the 0 Bytes limit" -
self-contradictory, and useless for choosing a value that would work.

Errors and field descriptions also told callers to invoke raw HTTP endpoints.
These strings serve the REST reference and the CLI's own help equally, so they
now name the operation and its object rather than a method and a path. A sweep
test walks every v2 schema description and holds the line, with the remaining
offenders in files this change does not own recorded explicitly rather than
left to be rediscovered.

Listing the editors of a built-in skill claimed the skill did not exist, while
reading the same id succeeded - a well-formed request for a real resource is
not malformed, so the list answers an empty roster and only the mutations
refuse.

Bulk folder deletion recorded only the leaf name in its audit trail while the
single delete recorded the full path, leaving two same-named folders under
different parents indistinguishable after the fact.

Bulk chunk enable, disable and delete each treated an unmatched id differently
behind one sentence of documentation. They now follow one rule.

A workspace-scoped list refused with the name of a resource the caller never
addressed, which reads as an empty workspace rather than an unreachable one.

* fix(cli): stop a config value forging a section it was never meant to write

The config file is written by joining names and values into INI lines, and
nothing checked what was in them. A profile name carrying a newline and a
section header wrote a section that merged into a different profile and took
over its endpoint - and the next command sent that profile's stored API key
there. A workspace value could do the same from the other side, since only the
endpoint flag validated its input.

The refusal now lives at the writer, the single place untrusted text enters the
document, with the flag-level checks kept for the better message. Either alone
blocks the forgery; the pair is deliberate.

Rejecting rather than escaping, because the format has no escape syntax and
these files are hand-edited and read by other tools that would not decode one
we invented. The forbidden set covers control characters and the two Unicode
line separators, which the previous guard missed - those parse as an unreadable
line, so the key silently vanished on read and the next write appended a
duplicate while the command reported success.

Login also wrote the key before the settings, so a malformed response from the
deployment could leave a key on disk with no endpoint beside it, and the next
command would send it to the default host. Settings are written first, and the
response is checked before anything touches disk.

Name validation applies only when creating a profile, so a hand-written one
that predates the rule keeps working.

* fix(docs): tell the reader which key a command needs, and stop the ids contradicting the CLI

Around sixty v2 operations refuse a workspace API key, and the CLI's help said
nothing about it - the caller found out from a 403 after the request went out.
The restriction is already stated in the API spec, so the generator now reads it
from there and the command description carries it. The sentinel sentences are
imported from the spec's own constants rather than copied, so a reword cannot
silently unmark every command, and the test pins the count as well as named
operations because a reword confined to one family would otherwise slip past.

The generated reference also rendered an empty default as a sentence pointing at
nothing - "Defaults to ." - for every repeatable filter. Omitted now, while
false and zero still render, which is the trap that shape of check usually
walks into.

The hand-written guides used a workflow-shaped id for workflows that the CLI's
own help says never names one, and five other families were equally wrong. All
of them now match the scheme the CLI declares, consistently per entity across
pages, with the shared ones taken from that help text so the two read as one
voice.

The page documenting every flag was linked from nowhere; both landing links
pointed at the overview instead. And the generator's test file was absent from
the hand-maintained list CI runs, so its guards never executed.

* fix(cli): stop a page-size default capping a destructive filter

Every request field named limit inherited the pager's default of 100, but only
a cursor-paginated command interprets that flag. The two filter-based row
mutations declare no cursor, so the default went onto the wire as a row cap: a
filter matching 250 rows deleted 100, exited 0, and said nothing - while the
confirmation the user had just answered promised every matching row. The flag's
own help offered 0 for everything, which those endpoints reject; the unbounded
form is the field being absent. The pager's default now applies only where the
pager runs, and the tests pin the omission on the request body rather than in
help text.

A cap typed alongside an explicit row list was silently ignored; it is now
refused on the client, where refusing costs nothing to already-installed
versions.

Lists also truncated at a hundred with no signal in any format, and the two
inventory endpoints that do report truncation had that field dropped on the way
out - so a caller reconciling against a clipped list could not tell. One note
now goes to stderr while stdout stays a bare array, and a flag raised on a later
page survives the fold.

Also: a folder whose name contains the separator no longer prints a path that
resolves to a different folder; validation errors name the flag the user typed
instead of the wire field; an unknown subcommand with --help exits non-zero
instead of printing the parent's help; a fractional or negative page size is
refused rather than floored; an empty query filter is refused rather than
silently returning everything; and the two spellings of the missing-workspace
message became one.

* fix(cli): gate destructive table imports and fix follow-mode rendering

A `tables import --mode replace` empties the table before its first batch,
so the only warning was in the describe. It now confirms, and the wording
tells the truth per mode: cancelling a replace leaves a prefix of the new
file with the originals already gone, while an append re-adds its rows if
the file is imported twice. `--yes` skips the gate, and the gate runs
before the file is opened.

Import and export cancellation carried no describe at all; the import one
now confirms, the export one records why it deliberately does not.

Follow-mode output truncated cells to whatever the first row happened to
measure, so a longer status or workflow name arrived clipped with no
signal. Cells now clamp at a shared ceiling and pad to the lock, and the
log columns carry width floors so a short first page cannot pin a column
narrower than its own values.

Interrupting a staged download left the staging directory behind; it is
now removed on SIGINT and SIGTERM before the signal is re-raised.

`--select-output` without `--follow` selected from a response that does
not carry outputs, and said nothing. It is refused client-side, with a
separate message for `--async`. Its describe now names what the path
addresses.

`secrets set` always read a value, even when only metadata flags were
passed. Off a TTY that was an immediate refusal, so a metadata-only edit
exited 1 in CI for a value it was never asked for; on a TTY it stopped to
prompt, and the prompt rejects an empty entry, so there was no way to say
"leave the stored value alone" short of re-typing the secret. The read is
now skipped and the field omitted, which is what lets a metadata-only edit
run unattended. On a TTY, setting only a description no longer prompts.
Passing both spellings of the reveal flag is refused rather than silently
resolved.

Four mandatory hand-authored flags now say so, `billing logs` names its
key-type scope, and the dispatch list declares its columns.

* fix: close the gaps an adversarial review of this branch found

A conflict handler added earlier in this branch was dead code. It read the
Postgres error code off the thrown object, but the driver error arrives
wrapped with the real one on `cause`, so the check returned false on its
first line and the 409 never fired. Its test passed only because it threw a
flat shape production never produces. It now reads through the cause chain
with the shared helpers, compares the constraint name exactly instead of
matching a substring of the SQL, and its test throws the real wrapped error.

Resuming a conversation checked its workflow and its workspace but not its
type, so a conversation created by the web surface could be continued as a
CLI turn. It now refuses through the same uniform 404 as every other
mismatch, which closes the same omission on the web posting path. Minting
one no longer leaves a blank untitled row at the top of the Chat list.

The pre-write check on a minted API key refused fewer characters than the
writer does, so a key the check accepted could still fail at the write —
after the endpoint beside it was already stored, pairing a new endpoint with
the previous key. The two had drifted because the set was spelled three
times; there is now one.

A description claimed a processed count reported only the chunks that
changed. The update returns every row it matched, so re-enabling chunks that
were already enabled counts them all. Two OpenAPI sentences promised no
conflict detection and no persistence warnings in a dry run, both of which
the same branch had just made false. A described window was wrong whenever a
start was supplied without an end.

Listing the editors of a built-in skill answered a read with a modification
refusal on the internal surface. Archived table listings could reach the
strict folder projector again through a third scope value the input type
still allowed. A metadata-only secret write skipped the guard its
personal-scope twin has. The internal document boundary still took the two
processing fields as unbounded strings. Truncation was reported only from
the response envelope, so a clipped file body, row search and workflow-stats
list said nothing.

A staged download stopped watching for signals before it finished removing
its directory, and cleared every listener for the signal rather than its own.
Three tests asserted a contract constant against itself; they now drive
rendered help, real argv, or real render output.

* chore: regenerate the API reference, CLI surface, and CLI docs

The published reference still marked a secret value required and described
the delete parameter as one that also creates, the CLI surface still lacked
the marker that says which operations refuse a workspace key, and the
reference rendered an empty sentence for every repeatable filter whose
default is an empty list.

* test(cli): use the package's own delay helper in the staging poll

The audit bans a hand-rolled setTimeout promise. `sim-cli` does not depend
on the shared utils package, and its own idiom is `node:timers/promises`.

* fix: act on a second review round, and correct two earlier claims

The conflict pre-check read block ids from the wrong side. The writer
inserts each block's own `id` field while the check read the record key,
and the two can diverge because preparation copies a value under its key
without reconciling them. Edges already read the value and subflows are
genuinely keyed by the record key, so only blocks were wrong — collecting
every family from the values, as first suggested, would have broken
subflows instead.

A minted API key carrying leading or trailing whitespace passed the
pre-write check but failed the writer, leaving the new endpoint on disk
beside the previous key. It is refused up front now rather than trimmed: a
key is opaque, so trimming would store a value the server never issued and
turn a loud failure into an unexplained 401 later. The endpoint normalizer
does trim, which is what made a padded `--endpoint` fail only after the
browser flow had already minted a key.

A metadata-only secret write raced with deletion returned 500, because the
follow-up read that only assembles the response body threw an unclassified
error; it now reports the same not-found the non-racing miss already gave.
An unusable output format in the environment silently printed a table
instead of refusing. Two validation messages printed control characters
verbatim. A dry run now reports the preparation warnings its own commit
path returns.

The chat route created a titled conversation and never wrote a message, so
it appeared in the Chat list promising content it did not have. Both sides
of a successful turn are now persisted; a failed turn still writes nothing,
so a question is never stored without its answer.

Two claims of mine were wrong. The earlier commit message said `secrets
set` sent an empty value that overwrote the stored secret — it did not; the
prompt refuses off a TTY and rejects empty on one, so the old behaviour was
a clean refusal. And the delay helper commit said this package's idiom is
`node:timers/promises`; the package carries its own `sleep`, which is the
audit's sanctioned home and has five callers. It uses that now.

Also: a test asserting a deadlock stays unclassified could not fail, since
every candidate rejects it; it now pins a unique violation carrying no
constraint name. Workflow ids spelled with the file prefix are corrected in
the remaining fixtures, leaving the genuine file ids alone.

* fix: close a credential-misdirection path this branch had opened

Making the endpoint normalizer trim handled whitespace around a value but
not a control character inside one, and the URL parser removes those from
anywhere in its input — so a value that reads as one host could resolve to
another, and the profile's key went with it. The flag and environment paths
never touch the config writer, so its guard did not cover this. The
normalizer now refuses the same character set the writer does, which also
keeps the invariant that nothing it blesses can be refused by the write
that stores it. Comparing the parsed URL back against its input was the
alternative and is wrong: the parser rewrites percent-encoding, case,
internationalized hosts and default ports, so legitimate endpoints would be
refused.

The blank-query guard tested for exactly empty, so a whitespace-only value
still reached the wire — as a real zero on a numeric filter, an explicit
false on a boolean one, and as an encoded space the server then rejected.
It now refuses any value that is blank once trimmed, while a body string
keeps its meaning, an explicit zero still sends, and a value with content
around its whitespace is passed through untouched rather than trimmed.

A graph-id conflict reported 409 on the v2 route and fell through the older
persistence wrapper as an unclassified 500. That wrapper now classifies
orchestration failures through the cause chain, which also fixes a
pre-existing case where a workflow archived between authorization and the
locked read reported 500 rather than 404.

Persisting a chat turn claimed its row by id alone, so a conversation
soft-deleted mid-turn still received the messages and was bumped back up
the list. It now requires a live row. A turn whose caller hung up after the
model had already answered persisted nothing, though the work was done and
billed; it now persists and still reports the connection as closed.

An empty workspace id from the login response was read as no workspace at
all. A published description still promised a language-tag standard the
schema does not enforce.

The test asserting that a turn is stored before the final event drained the
whole response first, so it held whichever order the code used. It now
reads the stream incrementally and fails if the write moves after the
event.
2026-08-26 15:05:00 -07:00
Siddharth Ganesan 58aa6379e0 feat(cli): add chat command (#6937)
* feat(cli): add chat command

* fix(cli): harden chat command execution
2026-08-21 12:59:08 -07:00
Waleed ae2147645c fix(cli): resolve findings from a full command-surface audit (#6788)
* fix(cli): resolve findings from a full command-surface audit

Exercised all 147 commands against a live deployment. Fixes the defects
that surfaced, plus the docs and generator drift they exposed.

Transport
- Stop following redirects. A bare domain that 301s to www silently
  converted POST to GET and dropped the body, so reads worked while every
  write failed with a misleading validation error and login returned 405.
  Both the client and the device flow now explain the redirect and name
  the endpoint to configure, rather than carrying credentials off-origin.
- Report a non-JSON response as one instead of printing the HTML page.
- Name the personal-API-key remedy on a workspace-key refusal, reading the
  machine-readable code the API actually sends.
- Drop union-branch noise from validation errors that contradicted itself.
- Show paging progress on stderr for multi-page fetches.

Output
- Clamp record values for table only. text is the format built for pipes,
  and it was truncating signed URLs and tool source mid-value.
- Infer timestamp, duration, bytes and boolean formatting for API-owned
  keys so undeclared commands stop printing raw ISO and float ms. Skips
  user-defined table cells and leaves json/yaml on the raw payload.
- Render a declared-but-absent field as an em dash; billing credits were
  vanishing silently.

Paths, naming and validation
- Percent-encode folder paths per segment and decode them for display, so
  a folder reads and types as the name shown in the app.
- Reject a malformed endpoint where it is set and where it resolves,
  instead of crashing with a URL parse trace.
- Request the detail level logs list's own columns need; its workflow
  column could never populate.
- Rename three commands that described themselves wrongly and align two
  flags with their siblings. Old spellings still work: hidden, warned on
  stderr, and kept out of help and docs.
- Verify whoami against the API, separating a bad key from an unreachable
  endpoint, and report the workspace by name.
- Correct the --yes help text, which advertised skipping a prompt that
  does not exist.

Docs
- Teach the docs generator that a flag required by the runtime is required,
  and that hidden commands are not documented.

* fix(cli): clear the paging progress line when a page fails

Progress is written without a trailing newline so it can be overwritten in
place, and both paging loops cleaned it up only on success. A page that
threw part-way through left `fetched 1200…` on the line the error was then
printed onto, so the two ran together.

* fix(cli): name a working API root when an endpoint redirects

The suggested endpoint was the redirect target's origin, which drops a path
prefix. A self-hosted deployment reached at https://host/sim was told to set
https://www.host — not an API root, so following the advice replaced one
broken endpoint with another.

Derive it by stripping the request's own path from the target instead, so a
prefix survives, and say nothing about --set-endpoint when the target
resolves to the endpoint already configured: a trailing-slash or path
normalization redirect keeps the origin, and naming the value the caller
already has explains nothing. The login poll shared both faults and now
shares the helper.
2026-08-17 15:55:04 -07:00
Waleed 025ea4d2bd fix(docs): serve JSON-LD in the HTML, fix sidebar spacing, and tighten the CLI guides (#6763)
* docs(cli): use -g for the install, and cut the prose that was not pulling weight

`--global` is valid but `-g` is what every comparable CLI documents, and the
long form only came from the package README. Also drops the yarn tab: it read
`yarn global add sim`, which works on Yarn 1 only — Yarn 2 removed global
installs, so that command fails for anyone on a modern Yarn. Adds `npx sim` for
running without installing.

The guides had accumulated design rationale that belongs in code comments rather
than user docs — why the filter grammar is JSON, why the config section naming
is asymmetric, why an unexpected error keeps its stack trace. Surveying how gh,
Vercel, Turborepo, Deno, Bun and Supabase write theirs, none carry that kind of
justification, and callouts are reserved for content whose absence produces a
wrong result rather than for general asides.

So: 1016 lines to 763, and 12 callouts to 3. The three that remain are the
pairing-code check, that `sim logout` does not revoke the key, and the
`--limit 100` default on `batch-delete`/`batch-update`, which silently truncates
a larger match. Troubleshooting drops the entries whose error message already
contained its own fix and keeps the seven whose cause is not obvious.

* fix(docs): render JSON-LD as native script tags so it reaches the HTML

All four structured-data blocks — WebSite, TechArticle, BreadcrumbList,
SoftwareApplication — were rendered with `next/script`, which never emitted a
script tag. Measured on a production build, `/api-reference/getting-started`
contained zero `<script type="application/ld+json">` elements; the payload
existed only in the `__next_s` client-injection queue and the RSC flight data,
so anything reading the served HTML saw no structured data at all. React was
also logging "Encountered a script tag while rendering React component" on every
page.

`next/script` is for loading and executing JavaScript. JSON-LD is data, and
Next's own guidance is a native `<script>` in the component. `serializeJsonLd`
already escapes the `<` character to its unicode form, which is the
sanitization that guidance calls for, so only the element changes.

Same build, after: three valid tags per page with `WebSite` in `<head>`, and the
injection queue gone entirely.

* fix(docs): scope the flush-separator rule to a container's first separator

`[data-separator]:not([data-separator] ~ [data-separator])` was meant to keep the
first sidebar group flush against the top padding, but `~` only reaches siblings,
so it also matched the first separator inside every expanded folder. Under
Self-Hosting, "Install" lost its top margin and crowded the "Architecture" link
above it — 25px of gap where "Configure" and "Operate" below it had 40px.

`:first-child` expresses the intent directly. Only the four sidebar roots open
with a separator; every nested folder starts with a page, so the intended case
still goes flush and nothing else changes.

* fix(docs): move the flush-separator rule onto the separator component

Keeps the styling with the component that owns it, per the repo standard, and
lets the global rule be deleted outright rather than corrected — `global.css`
now only loses a rule in this PR. Tailwind's `first:` variant compiles to the
same `:first-child` selector, so behavior is unchanged: the build emits
`.first\:mt-0:first-child{margin-top:0}` and the prerendered HTML carries the
class on the separator.
2026-08-15 20:39:34 -07:00
Waleed fed891f69d docs(cli): add a CLI docs section generated from the command tree (#6762)
* docs(cli): add a CLI section, generated from the command tree

The `sim` CLI shipped with no coverage in the docs site. Adds a fourth
top-level tab for it, and moves Academy last.

The command reference is generated. `sim` exposes 147 leaf commands across
33 groups, most of them derived at runtime from the v2 route contracts, so a
hand-written reference would be wrong the week after it was written. The
generator walks the command tree `buildProgram()` hands to commander — the
same tree the terminal parses — rather than re-deriving it from the contract,
which would be a second implementation free to describe commands nobody can
invoke. `check:cli-docs` is a zero-arg `check:*` script, so the existing audit
runner picks it up and stale pages fail CI.

Generating against the real tree surfaced a collision it had been hiding:
`bulkUpdateKnowledgeDocuments` and `updateKnowledgeDocument` both derived to
`sim knowledge documents update`. Commander resolves a duplicate to the first
match, so the bulk form shadowed the single-document one and its flags were
unreachable while still appearing in `--help`. The bulk form is now
`batch-update`, matching how `tables rows batch-delete`/`batch-update` already
handle the same REST overload, and the generator fails on any duplicate path
so the next one cannot land silently.

Five hand-written guides cover install, auth, configuration, output formats,
and scripting. Also corrects two commands in the package README that do not
exist as documented (`tables columns <tableId>`, and `--sort score:desc`,
which is JSON).

* docs(cli): document every flag from the contracts, add troubleshooting and a single-page reference

The command reference was structurally complete but said almost nothing: 223 of
377 flags rendered as "Set sort by" because the CLI only ever read flag help
from its own contract overrides, and fell back to restating the flag name.

The prose already existed. The v2 route contracts carry 931 `.describe()` calls
and the OpenAPI specs publish all of them — 327 parameters and 282 body
properties, 100% coverage — but the generated operation table dropped every one,
carrying only a per-operation summary. It now carries the field descriptions,
the path-parameter descriptions, and positional help, so `--help` and the docs
explain a flag the same way the API reference does. Placeholder descriptions are
now zero, and 147/147 commands, 377/377 flags and 130/130 arguments are
documented.

`check:cli-docs` fails on a request field with no `.describe()` rather than
letting it render as documentation that says nothing.

Also in this pass:

- Commands are root-level sidebar entries under a Commands heading rather than
  a folder, and headings are the command's description, so the table of
  contents distinguishes entries at the first word instead of repeating
  "sim knowledge documents …" fourteen times. A guard fails the build if two
  descriptions on a page collide, since they would share an anchor.
- A single-page `Complete reference` carrying all 147 commands, for in-page
  search and for agents fetching `/cli/reference.mdx`. It keys on exact command
  paths because descriptions are only unique within a group.
- A troubleshooting page, with every message copied from the source.
- Table columns are sized by a local component; the flag column was starved
  while descriptions kept most of the row empty.
- The prerelease install channels are dropped from the docs and the package
  README, which is what npm renders.

* fix(docs): match the CLI tab by path segment, and escape backslashes before pipes

`pathname.includes('/cli')` also matches `/integrations/clickup` and
`/integrations/clickhouse`, so both existing integration pages lit the CLI tab
and unlit Documentation. Matching is now per path segment. Anchoring to the
start would not work either — a non-default locale prefixes the path, as in
`/ja/cli` — so the segment is matched wherever it sits.

Table cells now double a backslash before escaping pipes. A value ending in one
turned `a\` + `|` into `a\\|`, which the table parser reads as an escaped
backslash followed by an unescaped pipe, splitting the cell early. Nothing in
the command surface contains a backslash today, so this was latent rather than
visible.

The reference page's global options table is two-column and was being wrapped in
`CommandTable`, which sizes the second column for the `Required` cell of the
three-column tables and crushed the description into 5.5rem. It now matches the
overview page, which leaves that table unsized.
2026-08-15 19:25:34 -07:00