6 Commits
Author SHA1 Message Date
Waleed 57cc81e751 improvement(sim-cli): publish each command's path aliases and renamed paths in the command inventory (#9251)
The worker builds its agent grammar from this inventory and did not know that sim runs files mv, workflows folders ls or kb as their commands, or still runs retired paths such as tables count create and workflows deployment list as the commands they were renamed to, so it refused or misread them. Each command now carries the commander aliases of its path segments (pathAliases, aligned with path) and the paths it was renamed from (renamedFrom).
2026-10-20 12:47:19 -07:00
99ed5fb59c feat(project-files): let Chat read and write Project files (#9034)
* feat(files): restore Project file browser and Mothership integration

* fix(files): resolve CI audits and test types after restacking on staging

- Keep prepareProjectsForAccountDeletion returning the Project IDs to delete.
  The object return added an unused storageCleanupEventIds field; the
  durable cleanup outbox already processes those events, so restore the
  staging contract and its stronger integration assertions.
- Exempt lib/mothership/generated from the relative-import convention.
  contracts:sync copies those wire contracts verbatim and byte-checks them,
  so their sibling imports must stay relative; drop their baseline rows.
- Re-record the files page module-count baseline for the Project browser.
- Type the test fixtures that regressed, including a queue-recovery
  fixture that persisted the invalid completion status 'success'.

* fix(files): gate Project sandbox callbacks like workspace callbacks

Staging now holds resource-delegated Chat principals to content
admission, so a Project callback read with no gate was refused and the
sandbox got a 503. Run Project callbacks under the same workbench
session gate as workspace callbacks (the gate's workspace is optional,
since Projects have none) and share the withheld-result handling.

The Project CLI read suite now runs reads under the tool executor's
registry gate, as Chat does.

* fix(projects): keep Project chat tabs behind the rollout flag

The chat resources API stored a Project-owned tab without checking the
projects flag. Refuse new Project tabs while Projects are off; removing
or reordering stored tabs stays possible so they can be cleaned up.

Also drop the projects flag check in list_user_projects that could never
run (listProjects already refuses while the flag is off) and the content
delivery barrel export left unused by the shared sandbox delivery helper.

* refactor(files): drop the Project files UI from the Chat transport PR

The Project files UI will be built for the project-first redesign, so
this PR keeps only the backend and Chat transport behind the projects
flag. Restore the workspace Files page, file viewer, Chat resource
panel, add-resource and mention entries, and their client query hooks
and owner adapters to staging. The previous tip is tagged
project-files-ui-seed for that UI work.

Kept: the shared ChatContext wire type (Project contexts and file owners
reach the Chat server) and its exhaustive kind registry entry.

Also drop the UI-only browser constants and response types nothing uses
now, restore the files page module-count baseline, and type the
use-chat fixtures that returned with the staging test.

* fix(files): redact Project file text like workspace file text

Project grep and text reads admitted every Project artifact as opaque
bytes, so any file carrying a secret was refused to Chat instead of
having the secret masked, unlike the same file in a workspace.

Share one text extraction core between workspace text reads and private
file observation: parse, redact before a line window can cut a secret,
then slice. Project text reads now go through readProjectFileText, which
admits the source and every rendered input as complete text and returns
text only. The complete artifact read stays private to that use case, so
bytes never leave under a complete label; readProjectFileArtifact keeps
admitting opaque bytes, and visual reads of a secret still refuse.

* fix(sim-cli): keep unpublished groups in the Mothership command inventory

#9057 hid the projects group from --help and `sim cli` search until
launch, and the inventory printer dropped every hidden command, so the
Mothership worker lost the Project file commands it routes --project to.

List a hidden top-level group that is only unpublished, marked
`unpublished`, so the worker can route to it once Sim enables Projects
for a turn; other hidden commands stay out, and --help is unchanged.

* fix(files): address Cubic and Greptile review findings

Workspace regressions while Projects are off:
- Sandbox file copies between workspaces keep the workspace route when
  the lease has not negotiated the file-owner protocol; only pairs that
  name a Project owner need it.
- Chat upload contexts keep their uploads/ path in the workspace owner
  adapter.

Project transport:
- Project CLI requests refuse mutations during benchmark reference
  generation, as workspace requests do.
- Project grep defaults an omitted scope to files.
- Project files with unrecorded or empty exact provenance are readable
  when the caller holds a registry; a missing registry is still refused.

Mothership inventory:
- Unpublished Project commands carry response shapes from the
  launch-ready Project OpenAPI definitions.
- An operation whose delegated services exclude Chat no longer mints a
  Chat principal, so the refused-route list now matches runtime for
  Project file version history.

* fix(files): address the second review round

- Project `files read` returns binary metadata for undecodable files:
  the text route checked `workspaceFileTextFormat` against null, but it
  returns undefined.
- Workspace file resources keep `workspaceId` beside their new workspace
  owner, so current browsers can open and close those tabs in
  organization chats; the resource key is unchanged.
- The Project tab flag gate is covered against the database instead of
  through mocked collaborators.

* fix(files): choose a generated Project document's read by its rendered type

A generated PDF is stored under its source type (text/x-python-pdf), so
the Project text route treated `files read report.pdf` as text instead
of the default visual representation. Share the workspace reader's type
selection as observedMediaType: a generated document is read as the
type its name promises, otherwise as its effective stored type.

* fix(mothership): close review gaps in Project file transport

- Dispatch CLI services before file-owner adapters, so a service that
  carries a file owner gets the services validation instead of a bridge
  error. The unreachable Project-side service check is removed.
- Check caller-chosen names on Project file renames and folder create
  and update with the same secret-provenance gate as create and upload,
  including workbench history for sandbox callbacks.
- Move first-message file panel resolution into resolveFileAttachment
  and replace the mock-only post test with a database-backed check of
  the saved tab.

* fix(mothership): match chat tabs within the chat's workspace

- The chat resource store now keys tabs with the chat's workspace as the
  fallback scope, so an unscoped tab and the same tab addressed by that
  workspace merge instead of duplicating. Upsert, remove and reorder all
  use the same identity.
- Project file renames and folder writes recheck cancellation after the
  name check and never forward a cancelled write.
- Leave the .claude and .cursor import rules to staging, whose source
  conventions check now exempts generated contract copies itself.

* fix(mothership): bound Project name checks and keep equal owners

- Parse Project file renames and folder writes with their route contracts
  under the inline body limit before the name check, so an oversized body
  is refused instead of buffered. Only those routes copy the request.
- Treat an equal file owner as unchanged when merging chat tabs, so a
  repeated upsert stays a no-op.

* fix(project-files): harden owner hints, grep provenance, and tab identity

- Restore info/warn levels for stale or forbidden tab attachments and skip an unresolvable target instead of throwing
- Make Project owner hints best-effort so a failed lookup never fails the turn; document that a worker rollback needs projects off first
- Cache the worker file-owner protocol probe per base URL (lru-cache, 30s, success only)
- Answer an invalid copy body on an unnegotiated lease from the workspace route (v2 400) instead of a 403
- One tab-key function on server and client, keyed with the chat workspace on both sides
- Project grep withholds a file without provenance instead of failing the search or latching the registry, and stops listing once the file budget is full
- Bound copy effects: one copied file opens a tab, bulk and folder copies refresh the destination owner; org copies carry the destination workspace
- Bound list_user_projects capability fan-out
- Refuse unextractable or oversized Project text reads before download, with the text-extraction size message
- Share the create-file input mapper, the resource authoring version, and isProjectsEnabled

* fix(project-files): refuse secret-bearing raw Project downloads without a workbench tracker

* test(project-files): pin refusals and cover live owner-hint, tab-key and log-level paths

* refactor(project-files): move owner routing and admission behind the file-owner adapter

- copilot-request carries one opaque resource grant per request; the Project file
  and file-copy grants are minted next to their adapters, so the shared route layer
  no longer imports Project or copy policy
- One projectFileDelegationScope for the verifier, the Copilot adapter and the
  request grant; admitsCopilotResourceDelegation shared by the adapter and the grant
- Private v2 transports derive route patterns from their contracts
  (v2RoutePattern / isV2ContractRoute / tryMatchV2Route), including the name-write
  lookup, so a renamed route can no longer skip interception silently
- Project upload and write transports share their route prologue, cancellation
  check and failure envelope
- One forwarded-header strip list and one callback context for sandbox callbacks;
  SandboxResourceScope is exported instead of re-derived
- isOwnerScopedFile replaces adapter field comparisons; the adapter field is
  renamed invocationScope so it no longer reads like Principal.resourceScope
- authoringInvocation picks a turn's chat-or-workspace invocation in one place

* refactor(mothership): share chat tab keys, CLI session setup, and file read helpers

- chatResourceKeyFor binds the tab key to one chat's workspace for the store,
  reorder, stream handler and use-chat; resource removal looks keys up in a Set
- isFileResourceType names the file/filefolder check; shared owner refinement in
  the resource contracts
- prepareCliSession gives every CLI invocation its endpoint, workbench session,
  file provenance and effect sink
- Text-extraction refusals, the default-visual media check and MIME
  normalization are shared rather than copied
- Grep page cap and listing note are shared; a corpus states its own scope
  refusal instead of the engine naming Projects
- resource-effects builds an owned file path once and shares the owner refresh

* perf(project-files): skip the folder tree for grep reads and check hint gates before the flag

- Project grep reads ask the artifact read not to load the owner's whole folder
  tree per file; grep never reads file.folderPath
- File-owner hints run the route, mode and admission checks before awaiting the
  projects flag
- Project text reads reuse the workspace text-extraction refusals; Project grep
  uses the shared page cap and states its own scope refusal

* test(mothership): drive stale-tab and owner-hint failures through the real boundary

* test(project-files): pin upload cancellation during provenance and new-chat tab order

* fix(project-files): recheck cancellation before completing a workbench upload

Provenance resolution can outlast the invocation; completion now rechecks both
signals before it registers the file, as the create path does.

* fix(mothership): persist new-chat tabs in attachment order

Attachments resolve concurrently, so tabs were persisted in completion order.
Each authorized tab now keeps its attachment index.

* fix(mothership): store the recovery authoring marker only while Projects are on

Earlier builds parse recovery config strictly and skip recovery on an unknown
key. Only Project authoring reads the marker, so flag-off turns no longer
write it and stay recoverable across a rollback.

* fix(mothership): tag an unlabeled file selection as a bare mention

* test(project-files): give new suites their own reports and drop an unreachable adapter case

* refactor(files): return readSourceBuffer's TSDoc to its function

---------

Co-authored-by: Waleed Latif <waleed@sim.ai>
Co-authored-by: Waleed Latif <walif6@gmail.com>
2026-10-16 16:52:58 -07:00
Waleed 842c5ac6a1 improvement(sim-cli): commands reach the API only through the operations they declare (#8877)
* improvement(sim-cli): commands reach the API only through the operations they declare

Hand-written commands called SimClient by path and declared nothing, so the command inventory could not mark one Mothership-unavailable when it calls a refused route. Every command now gets its client from callsOperations: calls name an operation, route and method come from the operation table, and the client is typed to the declared operations.

* test(sim-cli): assert declared-operation routing at the transport

* test(sim-cli): assert directory requests at the transport
2026-10-09 20:13:46 -07:00
Waleed 722510e61c fix(sim-cli): the command inventory describes the operation each command runs (#8874)
The inventory mapped paths back to operations by re-deriving every route, so an operation whose derived path equals a sibling's (undeployWorkflow derives `workflows deploy`) overwrote it. Generated commands now record the operation they run and the inventory reads it.
2026-10-09 18:50:46 -07:00
Waleed 0c5dbfb91d feat(sim-cli): publish which CLI commands Mothership is refused, from the route admission rule (#8838)
* feat(sim-cli): publish which CLI commands Mothership is refused, from the route admission rule

* fix(sim-cli): inventory raw-route admission so chat is flagged too
2026-10-08 22:59:13 -07:00
f734ba5be2 feat(mothership): v1.0.0 (#8208)
* mothership agent cli: grep --in accepts the world/resource path a match line prints

* workflow lint, block catalog, tables, files, sandbox tools: fixes from exploration run 5

Lint: trigger-category blocks are entry blocks (schedule was an orphan); references inside Function code are checked with the runtime tokenizer; the agent-cli lint and grep read the draft state instead of the sanitized export. Catalog: operation inputs publish sub-block ids only (no canonical-param aliases); trigger-category blocks expose their trigger-mode fields. Tables: dispatch processedCount counts unlimited dispatches; json-language code fields accept objects. Files: restore keeps the folder when it still exists. Sandbox tools: outputTable failures report the files already written and the per-language result shape; the file writer keeps the computed result.

* sim-cli: --output json prints the API data verbatim; the single-key unwrap stays a table-only convenience

* route and connection validation, trigger defaults on add, table folder restore, dispatch listing, group column attach, import block summary; logs query and deps engines; run-tool errors and payload compaction; stale tests aligned

Editing: router routes validated as {id?, title, value} with unknown keys named; malformed connections reported instead of dropped; sub-block value() defaults evaluated on add so a webhook gets its token. Tables: folder restore no longer 500s (lock inside the row transaction), completed dispatches are listed, groups attach to existing output columns. Workflows: import answers with its blocks. Mothership engines: logs query resolves bare paths under output, marks missing paths, drops non-executed runs under --where; deps lists graph predecessors and child return shapes; run-from-block validation errors reach the agent; run payloads compact input.code. CLI generated types regenerated.

* sim-cli: the operations apply confirmation describes the batch as written, not a delete

* sandbox export keeps the code's result; --trigger implies --manual; deps mock skeleton; run tools lift the terminal output and take select; lint all-clear names what it checked; docs chunker keeps code spans

* sim-cli: runs get --select-output takes block names; groups delete says it removes the output columns and their data; a publish that lands on public auth prints a note

* null-safe block.error, strict start-trigger coercion, lint checks reference paths against output schemas; logs stats segments; requiredWhen and model ids in the agent catalog; live tag usage; one row-filter grammar; minted service-account credential ids; webhook delivery URLs on deployment status; mv semantics for folder moves

Executor: <block.error> resolves empty on a block that succeeded so a shared error collector is writable; a number input given a non-numeric string fails the run at start naming the field; a regression test pins that failure traces carry child spans. Lint: a reference whose first path segment is not on the block's effective outputs (or its responseFormat schema) is an unknown-field finding. API: logs stats honours segmentCount and omits empty buckets unless includeEmpty; block detail publishes requiredWhen instead of flattening conditional requirements and always lists model options with hosted marks; knowledge tag usage counts through the document slot; every table rows filter accepts a bare condition; service-account credentials mint their id; deployment status lists each webhook's delivery URL; a folder move into an existing folder moves it inside. OpenAPI and CLI generated types regenerated.

* same-workspace import keeps or warns about workspace bindings; generic required-field lint; folder moves to the root; empty-graph and no-entry lint notes; span durationMs; MCP tools reported on undeploy and listed inactive; paged connector types and workspaces; grep --in refuses unknown selectors; run_function names exported files; run tools surface the failing block's error

Export takes includeWorkspaceBindings for same-workspace round trips and import answers with warnings naming every block whose required binding was stripped. Lint reports a missing required field for every block type (the knowledge base id was skipped), notes an empty graph and a graph with no entry block, and the trace spans carry durationMs. Undeploy answers with the MCP tools it archived and the tools list shows them inactive. connector-types list is paged (25, summary projection, detail=full) and workspaces list defaults to 25. The agent CLI's grep refuses a prefix or unknown --in selector with the accepted forms; a run_function that exported files and wrote a table says both; a run tool whose executor result carries no message uses the failing block's own error.

* table groups default to the deployed version and refuse a dispatch without one; column rename reports unmigrated Table-block filters; lint checks table fields against the live schema; runCount counts every settled run; runs cancel is honest about no-ops; dry-run apply answers with previewBlockIds; conditionResult is the chosen test's boolean with selectedTitle; workflows run --select-output keys blockOutputs by the selector and rejects unknown heads; knowledge search exposes rankScore and rank; the multi-trigger error names workflows state get; the Knowledge block declares cost, tokens, model; media writes refuse to fork a moved folder; grep --in knowledge points at knowledge search

* agent cli grep: the searchable text leads with the resource's name and description

* Catalog hides sunset blocks unless includeSunset; enrichment get returns the group's run state and outputs; knowledge upload prints the document fields; docs chunks titled by page not nav link; logs carry hasHandledErrors and opt-in handledErrorRuns; outputTable receipt keeps stdout

* Sandbox exports decode by the path rule the reader used: a .jpg declared without a format is written as image/jpeg bytes, not as its base64 text under the json format (thumbnails opened as raw text)

* Chat uploads resolve by uploads/<name> for reads, sandbox mounts, and image references (never listings or writes); an unresolved reference image fails the call instead of rendering without it; copilot session-sandbox calls are priced like Function-block sandboxes and report the raw cost beside the billed one; CLI docs regenerated

* grep: resolve --in against listings, fetch only the named resources

A bare --in selector (e.g. --in agent, --in file_v5) materialized every searched
world to find one block: 65 block details plus every workflow state per call. On
dev that took 18-34s per grep and tripped the per-user rate limit. Each world now
has a cheap index (its listing) and a per-resource fetch; a --in search reads the
indexes and fetches only the matches. Whole-world searches are unchanged.

* mothership: model-facing strings name only what exists on the CLI surface

The copilot back-derives fixes from the strings sim hands it, so a retired name
becomes a wrong instruction to the user (dev 2026-09-03: a save_upload mention
became "drag the photo into the files panel"). Every model-facing string audited
today now names the current surface:

- upload notice: a chat upload lives at uploads/<name> and is not a workspace
  file; workflows import takes --workflow (there is no --file); a .zip is mounted
  and unzipped in the sandbox rather than a files unzip path that does not resolve
- table import resolves uploads/<name> directly (includeChatUploads) instead of
  pointing at the retired save_upload tool
- function-execute / generate-image: outputs get, files ls, files restore and
  tables list replace read/grep/glob/restore_resource and Go VFS meta.json paths
- process-contents: browser/terminal pointers no longer name browser_* or a
  terminal tool this surface does not have; docs fallback names docs search
- lint/deps usage strings use the plural workflows group
- integration credential error names credentials list

* docs: every model-searchable page states the current surface

The copilot answers users from these pages (docs search), so a wrong sentence
becomes a wrong instruction. Audit fixes:

- chat: uploads live under uploads/, not the Files panel; on-demand workspace
  reach instead of a per-message snapshot; Chat runs Opus 4.8; connectors and
  deletes/restores as they actually behave; login pages via the shared browser
- cli: --select-output works on sync runs (only --async conflicts); runs get
  takes the same block selectors; secrets set needs --scope; workflows run runs
  the deployment or --manual; sim profiles honours --output; which groups have
  no singular alias; the embedded CLI has no profile, login, or config; @path
  reads the chat sandbox
- model defaults are claude-sonnet-5 (guardrails has none); API examples use
  https://www.sim.ai/api/v2 (apex 301s POSTs into GETs)
- quick reference / shortcuts: only affordances that exist (no Deploy tab, no
  workspace duplicate/export, Mod+B not Mod+E, variables under the ⋯ menu,
  <variable.name> syntax)
- editor read-only rules match round-trip-safety.ts; card links fixed
- sim-cli: runs-get selector error and the Settings label match the product;
  batch delete describes its ids as deleted, not updated; docs/api regenerated

* mothership: prepare_file_edit create + new_file pass input validation

The workspace-file handler has always supported operation=create with a
new_file target, but the generated tool schema (from the copilot catalog)
forbade both, so the watched write could never create a file: a "save this
report as a workspace file" turn died on `/operation must be equal to one of
the allowed values` (dev 2026-09-03). Regenerated from the catalog that now
declares create, new_file, and fileName; regression test on the validator.

* mothership: the embedded CLI answers v2 in-process; grep memoizes both platform catalogs

The embedded CLI and the agent-cli engines were typed v2 clients pointed at the
server's own URL: every tool call was a network round trip through the proxy,
API-key auth, the abuse rate limits, and the proxy body ceiling. A grep over one
block definition cost 8-34s and tripped the per-key limit (dev 2026-09-03).

- sim-cli: ResolvedProfile / EmbeddedCliIdentity take an optional transport; the
  client sends through it instead of fetch. The installed CLI never sets it.
- sim: an in-process transport resolves a v2 path against a generated route table
  (scripts/generate-v2-route-table.ts, check:v2-route-table in check:audits) and
  invokes the route handler directly. The request is marked internal through a
  WeakSet — not a header — so admission still authenticates it but skips the
  pre-auth IP bucket and the per-key rate limits, which exist for callers on the
  wire. Contracts, use cases, presenters, and error envelopes are the ones the
  network path runs. Anything outside the v2 table falls through to fetch.
- grep engine: blocks and tools (5,000 built-ins at 100 a page) are memoized per
  workspace; an exact block or tool id (--in file_v5) resolves from that corpus
  without listing any workspace world; name fragments still index every world
  and fetch only the matches.

* mothership: take the embedded CLI off the in-process transport until the dev hang is understood

Every cli_grep on dev timed out at 60s from the first deploy of 99aad281a7. The
transport, marker, route table, and grep memo stay; the embedded identity goes
back to the HTTP path while the hang is diagnosed from sim's logs.

* grep: share in-flight builds, bound nested requests process-wide, cache file text by version

The first dev turn on the in-process transport fired eight world-wide greps at
once. Each rebuilt every world concurrently on the one process serving the chat
— 24 block-catalog listings, 150 tool-catalog pages, 130 file reads in sixty
seconds — until all eight timed out. Over the wire the same fan-out had been
spread across tasks and throttled by the network.

- concurrent greps now await the build already in flight for a (world, workspace)
- nested requests are bounded to eight for the whole process, not per world
- platform corpora (blocks, tools) are kept an hour, not ten minutes
- a file's text is cached by id + updatedAt + size, so repeat greps re-read only
  files that changed

* grep: an exact block id answers before the tool catalog is built

* grep: honour -A and -B context flags

The engine read only -C; -A and -B were ignored silently, so `-A 40` returned
the bare match line and the model concluded context flags were unreliable
(dev 2026-09-03). A bad value is refused like -C's.

* sandbox pricing: normalize COST_MULTIPLIER at the boundary

Under skipValidation the value arrives as the raw string it was deployed with;
the sandbox lease's finite check rejected it, so on dev every run_code failed
as "Boot sandbox" (Sandbox pricing multiplier must be a finite nonnegative
number). getCostMultiplier now goes through envNumber, as the env module's own
rule says numeric overrides must.

* sandbox pricing: cost-multiplier imports env relatively — next.config.ts loads env-flags outside alias resolution

* mothership: pack a workspace inventory with every chat request

The agent spent nine tool rounds and ~20K tokens learning what exists in the
workspace before an orientation task could start. The chat request now carries
a compact inventory (contracts ChatRequest.inventory): workflows, tables,
knowledge bases, files, skills, custom tools, MCP servers, credentials, and
secret names — names and ids, one page per world, capped worlds named in
`truncated`. It is read through the same use cases the v2 listings run, under
the caller's session principal, so authorization is unchanged; a world that
fails to list is left empty and logged rather than failing the turn. The worker
renders it once per turn as a request-local message.

* grep: the tool catalog is one use-case call when the engine has the caller's principal

The tools world paged 5,000 built-in tools through the route stack at 100 a
page — fifty round trips, each re-resolving the gate and walking the registry —
which was the whole cold cost of a world-wide grep. The embedded bridge now
resolves the delegation key to its principal exactly as the v2 surface does and
hands it to the engines; with it, the tools world reads the catalog through
listCatalogTools in one call — same authorization, same projection. Without a
principal the client path stays.

* mothership: background tasks — subscriptions, wake turn, task pill (21 §6)

Sim side of copilot background tasks. copilot_task_subscriptions
(migration 0315) records a worker task's watch on a workflow execution;
the logging session's completion attempt posts every subscribed run's
outcome to the worker. POST /api/mothership/tasks/subscribe and
POST /api/mothership/wake (internal key): the wake runs the inbox's
headless lifecycle under the task id as message id, announces itself on
the chat status channel so an open chat reconnects, persists the user
message with origin 'task', and resolves the pill in the arming turn.

Client: a 'task' content block end to end (run handler, turn-model node,
both serializer directions, persisted normalizer — which also flattened
plan blocks to text on reload —, display block, segment, TaskPill) and a
muted TaskNotificationRow for task-origin messages. The stream
validator's run kinds now derive from the generated contract; the
generated mothership-stream-v1 gains task_armed / task_delivered.

* mothership tasks: resolve the pill by task id (jsonb text spacing broke the exact-key match)

* task pill: icons from @sim/emcn (lucide-react is not an app dependency; the Docker build failed)

* mothership: checkpoint Sim interaction and workbench boundary repairs

Capture the accumulated Mothership control, recovery, sandbox session, embedded CLI, and standalone diagnostic repairs. Companion worker checkpoint: c8245400. Validation and limitations are recorded in the Mothership revamp ledger through iteration 38; the latest expanded Sim diagnostic selection passed 1231 tests. The new table-mount audit is excluded and will follow separately.

* mothership: authorize table snapshot mounts and honor folder references

Move Mothership table snapshot resolution, provenance checks, and bounded storage reads into a registered application operation. Reuse the table VFS resolver for folder paths, recheck current actor access before materialization, and project safe mount errors. Preserve CSV transport, budgets, default paths, and workflow/public API behavior. Validation: 517 related tests, app/auth type checks, API and agent-CLI boundary checks.

* mothership: stream workbench downloads with atomic publication

* mothership: stream uploads and imports from workbench snapshots

* mothership: bind upload requests to the snapshot lifetime

* fix(mothership): preserve and bound embedded CLI output

* fix(mothership): preserve file provenance through host CLI reads

* test(mothership): guard current CLI catalog entry points

* fix(mothership): make search current, isolated, and explicit about coverage

* fix(mothership): bind file mounts to one canonical version

* feat(mothership): persist private upload source classification

* fix(mothership): preserve classification across workbench file copies

Record encrypted source evidence against host-streamed bytes and physical workbench identity. Bind completed upload streams through the authorized application operation, sealing pending provenance with the completion lease so recovery preserves the first result.

* fix(mothership): retain source evidence for saved CLI output

Classify settled stdout through the existing workspace-file helper and persist evidence against the streamed workbench bytes. Preserve completed command outcomes on publication failure and verify copied output across fresh invocations and pending sibling tools.

* test(mothership): exercise CSV composition through the Sim runtime

* test(mothership): connect CLI workflow runs to report publication

* fix(mothership): preserve raw CLI data with scoped presentation

* fix(mothership): expose canonical workflow identity for watches

* fix(mothership): preserve watches across transcript replay

* fix(mothership): convert code execution timeouts once

* fix(mothership): preserve child failures across transcript recovery

* test(mothership): distinguish replay from executable handoff

* test(mothership): verify persisted run diagnostics through CLI

* test(mothership): verify workflow execution log readback

* test(mothership): verify controller execution and resume recovery

* fix(mothership): acknowledge text received before stream retries

* fix(mothership): reconcile response text across reconnects

* test(mothership): verify automatic relay interruption recovery

* fix(mothership): unify replayable child lifecycle handling

* test(mothership): verify child recovery after worker exit

* fix(mothership): distinguish tool history from executable handoffs

* fix(mothership): acknowledge received activity between stream legs

* test(mothership): recover missing initial relay frames

* test(mothership): cover overlapping relay attachments

* fix(mothership): observe results owned by overlapping controllers

* fix(mothership): preserve long CLI command budgets

* test(mothership): verify table pipeline against real execution

* fix(mothership): recover one-shot response delivery

* fix(mothership): align one-shot MCP discovery guidance

* fix(mothership): make inbox attachments readable

* fix(mothership): scope upload names to the requesting chat

* fix(mothership): preserve worker history when forking chats

* fix(mothership): cancel only response-owned workflow executions

* fix(mothership): serialize workflow pickup with run Stop

* fix(mothership): track browser workflow execution through Stop

* fix(mothership): require settlement for Stop before chat attachment

* fix(mothership): preserve canonical history when stopping a response

* fix(mothership): handle abort failures while Stop history persists

* fix(mothership): retain queued Stop dependencies through retries

* fix(mothership): recover outgoing handoffs through the durable queue

* fix(mothership): bind queued retries to the active Stop operation

* test(mothership): exercise physical workflow edits and oracle execution

* test(mothership): verify extracted workflow invocation through saved logs

* test(mothership): verify build-only consent through physical run history

* fix(mothership): retain terminal failures in saved chats

* fix(mothership): retain active runs through worker connection loss

* fix(mothership): persist Stop before worker delivery

* fix(mothership): recover interactive streams after Sim process loss

* fix(mothership): durably admit chat turns before worker dispatch

* fix(mothership): recover lost tool handlers with fenced execution leases

* fix(mothership): finalize steered answers by turn identity

* fix(mothership): separate committed tool outcomes from stream publication

* fix(mothership): serialize saved resource panel changes

* fix(mothership): persist resource effects once before publication

* fix(mothership): preserve complete saved resource addresses

* fix(mothership): preserve saved table views in chat context

* fix(mothership): follow the current embedded table view

* fix(mothership): preserve exact table selection identities

* fix(mothership): attach usable file folder references

* fix(mothership): render saved file folder resources

* Revert "fix(mothership): render saved file folder resources"

This reverts commit d7854344686e06314ad7db3856347e472fda7b32.

* fix(mothership): align attachment context with available tools

* fix(mothership): authorize MCP calls as the chat subject

* fix(mothership): keep workbench policy out of the public CLI

* fix(mothership): preserve composed code export outcomes

* fix(mothership): retain completed files after export failure

* test(mothership): verify hosted workbenches and task compositions

* ci: restore staging jobs and keep Mothership acceptance local

* feat(mothership): receive private Sim controls over outbound transport

* fix(mothership): bound stream reconnection attempts

* fix(mothership): export chat sandbox files through workspace storage

* fix(cli): isolate embedded output from host process logging

* fix(mothership): preserve file intents and workflow result contracts

* perf(mothership): trace CLI and tool persistence boundaries

* fix(mothership): preserve resource effects and execution recovery

* fix(mothership): reconcile staging contracts and consolidate migrations

* fix(mothership): share and verify billing callback contracts

* fix(mothership): preserve catalog curation and embedded CLI access

* fix(mothership): mount code secrets explicitly

* fix(mothership): own sandbox profile at the code tool boundary

* fix(workflows): validate field values before full-state writes

* fix(chat): collapse main tool groups into action summaries

* fix(mothership): restore workspace API key dispatch

* fix(chat): label workflow dry runs as validation

* feat(chat): expose model, reasoning effort and Astra Fast controls

* fix(chat): align composer model and effort controls

* Group Mothership tools by activity and show scoped resource names

* Report requested Mothership model without legacy telemetry defaults

* Restore Mothership Slack bot connection from stored secrets

* Preserve execution events for deployed Copilot workflow runs

* Expose active workflow API details to Mothership

* Restore desktop parity and preserve interrupted chat progress

* Keep Mothership resource panels and activity groups in sync

* Show active tools before completed activity summaries

* Fix shared agent tool execution and discovery boundaries

* Port Assistant to shared Mothership runtime and repair resource recovery

* Refresh CLI docs and desktop validation after staging rebase

* Restore staging image inventory commands

* Fail migrations correctly and resolve empty direct database URLs

* Match chat attachment guidance to supported file capabilities

* fix(ci): retain the latest published desktop prereleases

* fix(mothership): read file contents through one adaptive command

* fix(chat): simplify pending tool activity label

* feat(chat): inspect scratch files and preserve inline images

Resolve workspace, upload, and sandbox references through authorized file readers. Preserve explicit Markdown images privately with each chat turn without creating workspace files, and keep visual bytes out of bounded UI tool-status events.

Keep sandbox CLI credentials scoped to the active callback so file provenance checks cover data entering the workbench.

* fix(chat): keep tool failures in expanded history

* fix(db): avoid repeated dev index builds and search backfills

* fix(chat): preview inline images with shared lightbox

* feat(mothership): operate across organization workspaces

* feat(mothership): manage settings and workflow file inputs

* fix(mothership): preserve stopped chat admission and queued corrections

* fix(mothership): show sequential activity intent and concrete tools

* fix(mothership): expose canonical model hints in internal discovery

* fix(mothership): unify scoped CLI and organization resource panels

* feat(assistant): connect and use personal organization integrations

* feat(chat): unify Home modes and search resource tabs

* feat(chat): tailor Home controls and starters to the selected mode

* fix(mothership): preserve file upload and preview resource context

* fix(mothership): stabilize Home search panels and add optional Fast Search

* fix(mothership): preserve staging authorization and resumed billing

* fix(db): make development schema pushes noninteractive (#7906)

* fix(db): resolve dev schema push column ambiguity

* refactor(db): generalize push rename handling

* docs(db): describe schema push reconciliation steps

* feat(search): unify Home search levels and composer

* fix(db): retire file size bridge before forced schema push

* fix(ci): restore test isolation and memory file scope checks

* fix(ci): keep metadata imports pure and narrow component graphs

* fix(mothership): refresh client settings after scoped mutations

* fix(db): repair partial member sync status on existing databases

* fix(home): reuse the shared loading fallback

* fix(mothership): remove eager catalogs and bulky admission saves

* fix(mothership): keep reads out of resource panels

* test(search): assert lazy integration catalog in headless chat

* fix(search): label balanced search level Auto

* fix(mothership): provide authorized context for chat titles

* fix(search): restore standalone search and refine chat mode controls

* fix(search): align mode selector with the standard search icon

* fix(mothership): resolve integration discovery service identities

* fix(desktop): reuse Sim login for authenticated browser previews

* fix(chat): preserve configured origin in deployment URLs

* fix(search): recover GitHub installation setup and verify ownership

* fix(search): retain sync failures and resume reconnected accounts

* fix(chat): restore thinking indicator after tools finish

* fix(mothership): expose and validate tool attachment identities

* feat(mothership): inspect configured workflow tool bindings

* feat(mothership): support dynamic Sim Chat tool modes

* fix(mothership): continue Sim Chat block conversations

* feat(mothership): create organization workspaces through CLI

* perf(desktop): observe browser state in action calls

* fix(mothership): repair background wakes and simplify watch activity

* fix(mothership): clarify tool permissions and folder context

* fix: reconcile staging rebase contracts and migrations

* fix(db): reconcile staging search migrations in dev

* fix(chat): preserve live turns and recover interrupted streams

* fix(workflows): scope custom block schemas during authoring

* fix: reconcile staging search and file contracts after rebase

* style(db): format rebased migration snapshots

* fix(slack): deliver Sim Chat text and tool progress reliably

* test(slack): verify Agent and Sim Chat thinking parity

* fix(slack): bound each long-answer append request

* fix(slack): continue long streams across message limits

* fix(slack): recover explicitly rejected size overflows

* fix(mothership): simplify saved results display label

* fix(slack): clean up long-response continuations

* feat(mothership): simplify Build picker to Astra effort levels

* fix(mothership): hide mode selection after the first message

* feat(search): add feature-flagged live provider retrieval

* fix(search): scope repository retrieval and add Coda MCP OAuth

* fix(search): restore account hooks client directive

* Fix GitHub connected-account tools in Search Assistant

* Add organization-scoped live search and admin-managed GitLab

* Add date-aware live enterprise search and bounded listings

* Polish live Search results, citations, and effort controls

* fix(mothership): preserve hosted service billing through cutover

Record hosted integration, sandbox and media spend through a trusted durable
service outbox independent of tool output, cancellation and worker delivery.
Preserve service billing under model BYOK and restore title admission.

Use workspace BYOK settings and fresh credentials, retain execute transport,
and restore the locked actor-authority check for billing-sensitive membership
removal. Add migrations and local HTTP/PostgreSQL billing acceptance coverage.

Validation: 1,453 billing tests passed with 30 environment-gated skips, app
and dev infrastructure types passed, and local SQL charge proofs passed.

* fix(search): restore live citations and streamed source panel

* fix(search): replace live results tab with cited sources

* feat(search): simplify federated sources and add scoped secrets

* fix(search): make service account setup directly accessible

* feat(mothership): add dev-gated Plan mode across chat surfaces

* fix(search): preserve account connection links in Slack

* fix(search): remove indexing language from live search

* fix(search): keep admin setup out of member integrations

* fix(search): route Search MCP through live retrieval

* fix(plan): normalize the runtime feature flag to a boolean

* ci(dev): deploy Trigger tasks independently of app images

* feat(mothership): set organization Generic Secrets in chat

* fix(search): verify federated connectors and align setup docs

* fix(search): remove stale Slack indexing copy

* docs(search): document Generic Secrets and clarify source setup

* fix(mothership): allow authorized organization code secrets in Build and Plan

* fix(search): restore live integration connection chips

* fix(search): include Gmail in live connection inventory

* fix(search): expand composer around image attachments

* fix(mothership): restore saved Plan chat history

* fix(mothership): address review findings and CI regressions

* test(deploy): align promotion expectation with Trigger CLI

* improvement(desktop): make saved password autofill field-aware

* fix(mothership): preserve trace filters and shared app workers

* fix(desktop): dismiss credential picker when browser hides

* feat(mothership): simplify model controls and add native desktop files

* fix(desktop): preserve password picker focus and lifetime

* chore(mothership): align model and deployment test expectations

* fix(mothership): resolve model and Plan controls from AppConfig

* fix(desktop): contain native imports and preserve UTF-8 pages

* fix(credentials): preserve unrelated setup controls

* feat(browser): file transfer, popups, PDFs, click gestures, and dialog answers for the browser agent

- Upload workspace, chat-upload, or granted local files into a page's
  file input (DOM.setFileInputFiles on the input a ref, label, or drop
  zone resolves to), staged privately and bound to the claimed tool
  call; save completed downloads into workspace files. New
  /api/desktop/tool/file route with application use cases.
- Same-session popups adopt Chromium's WebContents so window.opener
  works (sign-in and connect flows); a page that closes itself leaves
  cleanly and agent work returns to its opener.
- PDFs render: enable Electron's internal PDF plugin and let its
  packaged viewer resources past the network guard.
- Clicks take button, clickCount, and modifiers; agent right-clicks
  never open the native menu. Actions answer confirm/alert dialogs via
  dialog: {accept}. press_key gains F1-F12, Insert, and repeat; scroll
  and hover accept post-action observation.

* chore(chat): provide runtime flags in composer tests

* fix(browser): isolate actions and bound file transfers

* fix(browser): retain upload input identity through dispatch

* fix(mothership): validate complete billing protocol headers

* fix(browser): preserve upload dispatch before acknowledgement

* fix(browser): distinguish unconfirmed upload outcomes

---------

Co-authored-by: Vikhyath Mondreti <vikhyathvikku@gmail.com>
Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai>
Co-authored-by: Waleed Latif <walif6@gmail.com>
2026-09-23 16:41:27 -07:00