Files
sim/scripts/generate-docs.ts
Waleed bf575fe3fa feat(credentials): saved credentials and credential-only block versions, with keyless AWS/Databricks OIDC (#9763)
* feat(credentials): saved AWS and Databricks credentials with keyless OIDC

Adds an optional saved credential to every AWS block and the Databricks
block, alongside the existing inline key fields, plus Sim's own
workload-identity (OIDC) issuer so a credential can assume a customer's
AWS role or federate into a Databricks service principal without any
stored secret.

Existing workflows are unchanged: the inline fields stay, show, and
validate exactly as before until someone selects a saved credential.

- Issuer at /oidc/workload (discovery + JWKS), signing key from
  WORKLOAD_IDENTITY_SIGNING_KEY; 404 and keyless option off when unset
- Credential-scoped subjects, HMAC-bound to the preparing user and
  workspace, so a member cannot reuse another credential's subject
- aws-service-account: access keys, keys + AssumeRole (external ID), or
  keyless AssumeRoleWithWebIdentity; resolved server-side per call
- databricks-service-account: PAT, service principal M2M secret, or
  keyless RFC 8693 token federation
- AWS families accept the executor-authorized credential through trusted
  operation context; clients add the session token only for that pair
- S3 get_object signs x-amz-security-token for temporary credentials;
  inline signatures are byte-identical to before
- Tool credential binding honors optional (required: false) policies

* fix(credentials): close review findings on saved AWS/Databricks credentials

- Models can no longer inject a credential selector (or the token, session
  token, or host derived from one) into a hybrid tool call, which would
  have swapped an agent tool's inline keys for another workspace credential
- Databricks connections own the workspace host (authoritative `domain`)
- Drop the advanced credential-ID input: with a single picker, "is a
  credential selected" can no longer disagree with the active canonical mode
- OIDC subject proofs are bound to their provider
- Databricks token and identity fetches refuse redirects
- Correct the AWS module doc on presigned URLs carrying a session token

* chore(credentials): satisfy test-pattern, unused-export, and test-type audits

* fix(credentials): keyless reconnect, availability, and shared credential plumbing

- Reconnecting a keyless AWS role or Databricks principal keeps its trusted
  subject by default instead of issuing a new one that breaks the
  customer's trust policy
- The connect dialogs offer the keyless mode only where the deployment can
  sign tokens; the prepare endpoint reports availability instead of failing
- Register AWS (all 19 block types) and Databricks as stored-credential
  integrations, so org settings, the v2 credentials API, and Copilot can
  create and see them
- Optional STS region for GovCloud and China access keys
- One subject binder and config parser shared by both providers (neutral
  copy), one modal core for submit/errors/trust display
- Generic `authorizedServiceAccountCredential` operation context instead of
  an AWS-specific field; Databricks resolver checks credential type and
  honors cancellation
- Model-blocked params derive from each tool's authoritative params

* test(credentials): type the handler mock in the AWS credential seam test

* feat(credentials): credential-only block versions instead of hybrid inline keys

Existing AWS and Databricks blocks are left exactly as on staging, only
marked legacy and hidden from the toolbar; placed instances keep running
and show an upgrade badge. Each gets a credential-only successor
(s3_v2 ... databricks_v2, textract_v3) built by spreading the legacy block
and swapping its secret fields for a required saved-credential picker.

- withSavedCredential derives the successor's fields, inputs, and params
  builder; S3 and Textract split their builders so legacy key checks stay
- tools declare credentialSuppliedParams so serializer validation and the
  canonical-id check stop asking credential-only blocks for inline keys
- CloudWatch gets saved-credential log group/stream selectors that sign
  inside the credential's STS session
- docs generator resolves multi-level spreads and same-file param builders
- the AWS/Databricks providers are derived from the catalog again, so the
  hand-maintained token-credential mapping is reverted

* docs(credentials): document credential-only blocks by their credential, not inline keys

The generator omits a tool's credential-supplied params on pages for
blocks that sign in with a saved credential, as it already omits
accessToken. Intros now point at the AWS/Databricks credential guides.
Regenerates OpenAPI, CLI API, and CLI docs for the new selector keys.

* fix(blocks): never replace trigger fields that share a secret's id

A webhook trigger can register with a field named like the block's
secret (PagerDuty, Fathom, EmailBison, GitLab). withSavedCredential now
leaves trigger-mode fields, and their dependencies, exactly as they are.

* feat(credentials): generic saved-token plumbing for credential-only blocks

Wave 0 of moving inline-secret blocks onto saved credentials.

- tokenCredentialOAuth(provider, credentialParams) declares which tool
  params a saved token fills; credentialSuppliedParams is derived from it
- the executor fills those params after authorization, overriding any
  caller value, and fails closed when the credential lacks a field
- hosted keys are never injected when a saved credential supplies the key
- v2 tool execute no longer demands credential-supplied params when a
  credentialId is passed (fixes AWS/Databricks direct calls)
- withSavedCredential accepts an optional/conditional requirement
- check-block-registry verifies every credential-only successor's tools
  authorize the picker's provider and actually fill what they declare
- redact secretAccessKey/awsSecretAccessKey keys in logs
- serializer contract test covers every operation of each successor

* feat(credentials): Tavily saved credential and credential-only tavily_v2

Reference migration for wave 1. A generic tokenCheckValidator verifies a
token with one documented authenticated request (no redirects); Tavily
uses GET /usage. Legacy tavily is unchanged apart from its legacy marker.

* fix(credentials): close wave-0 audit findings

- agent tool-input no longer synthesizes inline secret fields a block's
  saved-credential picker supplies; one shared rule
  (credentialSuppliedParamsFor) now drives tool-input, serializer
  validation, and the canonical-id check
- the picker inherits the replaced secret's condition and requirement, so
  sts_v2's unsigned calls (web identity, SAML) need no credential
- withSavedCredential replaces both halves of a canonical secret pair and
  refuses options that would silently do nothing
- Databricks fills its inline apiKey/host through credentialParams, so its
  tools are byte-identical to staging apart from the policy line
- AWS operation families record their tool ids; a test and an executor
  guard ensure every AWS-credential tool resolves the credential
- the registry contract check fails on unpinned host params, optional
  credentials over required params, malformed pickers, and kind mismatch
- hosted-key admission (v2) and pre-resolution validation honor a selected
  saved credential; credential errors name fields in plain words

* feat(credentials): wave 1 — saved token credentials for 48 single-token blocks

Each block keeps its legacy definition (only hideFromToolbar + sunset) and
gains a credential-only <type>_v2. Every service gets a token-paste
provider with a one-request validator against a documented, read-only,
credit-free endpoint, and its tools fill the inline key from the saved
credential via tokenCredentialOAuth.

Blocks: affinity, agentmail, agentphone, ahrefs, apify, apollo, brex,
brightdata, browser_use, buffer, circleback, clerk, cloudflare,
crunchbase, daytona, elevenlabs, enrich, gamma, hex, huggingface,
incidentio, instantly, langsmith, launchdarkly, linq, loops, luma,
mailchimp, otter, persona, pinecone, profound, quartr, rb2b, rippling,
rocketlane, semrush, sendgrid, similarweb, sixtyfour, smartlead,
sportmonks, square, tailscale, trigger_dev, uptimerobot, youcom, youtube.

The registry contract check now probes each request tool's real URL
builder to find params that steer the host, and requires the saved
credential to fill them.

* fix(credentials): close wave-1 audit findings

- Pinecone and Semrush leave wave 1: Pinecone's per-call indexHost would
  let a shared key reach any host, and Semrush has no credit-free check
  that rejects a wrong key. Both are back to staging exactly.
- tokenCheckValidator: forbiddenMeansScoped accepts a valid key that lacks
  the check endpoint's scope (Apollo, Brex, Bright Data, Apify, Rippling,
  Persona); acceptsBody proves keys whose rejection arrives in a 200 body
  (Buffer GraphQL)
- the host-steering probe lives in scripts/saved-credential-contract.ts:
  typed placeholders for required params only, id/host shapes, bare and
  scheme probes, and a broad destination-name fallback where the URL
  cannot be built; covered by a fixture test
- S3 refuses a bucket or region that would move the signed request's host
- spread-inheriting docs keep their base block's outputs (restores the
  Smartlead, Trigger.dev, and Instantly output tables)
- incident.io's credential is optional for alert events, which use their
  own alert source token
- withSavedCredential refuses replaced secrets that show or require
  differently; copy, intro, and hint fixes from the audit

* fix(credentials): close wave-1 re-audit findings

- host probe also defeats unparsed suffix checks and finds hosts behind
  optional switches, probing every placeholder shape; broader destination
  names where a URL cannot be built; the contract check fails when a
  successor's tool is missing from the executable registry
- the optional-credential rule evaluates a conditional picker per
  operation; incident.io hides its picker on alert events instead
- withSavedCredential takes an explicit picker condition and refuses one
  that depends on a replaced secret at any nesting depth
- a scoped 403 counts only when it is the vendor's JSON response, never a
  firewall or CDN page
- S3 names a bad bucket or region in its error
- pinecone/semrush catalog dates restored to staging

* feat(credentials): wave 1b — Mem0, Zep, and Discord saved credentials

Their legacy params builders throw without an inline secret, so each is
extracted into a module function with a requireKey flag: the legacy block
keeps identical validation and messages, the credential-only v2 skips only
the key check. discord_v2 hides its picker for webhook posts, which never
use the bot token. BlockToolParams is now a shared type for extracted
builders (S3, Textract, Discord).

* feat(credentials): wave 1b — v3 credential-only versions and Flint

Blocks that already had a v2 get a credential-only v3 that spreads it:
cursor, dub, extend, fireflies, intercom, pulse, quiver, reducto. The v2
becomes legacy and every older version's replacedBy points at v3. Extend,
Pulse, and Reducto's v2 builders trimmed the key inline, so each is
extracted into a builder that takes the key as an argument; the legacy v2
passes the trimmed key exactly as before.

Flint gets flint_v2 with a dedicated validator: its API has no read
endpoint, so an unknown task id answering 404 proves the key.

tokenCheckValidator gains rejectsKey for vendors that answer a bad key
with a server error (Fireflies returns 500 auth_failed), so the user is
told the key is wrong rather than the vendor down.

* fix(credentials): close wave-1b audit findings and CI boundary failures

- token checks for the 59 new providers live in one module
  (validators/token-checks.ts) and their tool policies in
  tools/shared/saved-credentials.ts, so the tool-registry boundary stays
  within its baselines instead of adding ~60 modules to every graph
- the host-steering probe builds URLs through the shared transport
  (new prepareToolRequestUrl) and also flags a host validated against a
  vendor allowlist when two accepted values reach different tenants
- the AWS routing test reads typed metadata through getToolMetadata
- Flint accepts only its documented JSON 'Task not found' 404; Intercom no
  longer treats a 403 (suspended workspace, plan without API) as valid;
  Fireflies keeps its error text in logs when a 5xx isn't auth_failed
- Cursor help text, placeholder wording, Pulse intro, sorted secret fields

* fix(credentials): address pre-ship verification findings

- Refuse a keyless AWS role whose trust policy doesn't pin the credential's
  subject, so no other workspace can assume it
- Treat a malformed workload-identity signing key as not configured
- Brex: reject an unknown token's scope-style 403; the token-check factory
  now runs rejectsKey before accepting a scoped 403
- Flint: probe with a docs-shaped task id and a size-capped body read
- Drop the versioned BlockMeta entries that doubled templates; alias bare
  block types in the app icon map so template icon clusters resolve
- Widen the host-probe fallback names and rotate digits in alternates
- Copy: picker placeholder casing, Databricks v2 description, Pulse and
  Tailscale skills, AWS/Persona/Rippling intros, AWS credential docs
- Tests: AWS executor signing, AWS/Databricks credential setup, legacy
  params builders, serializer coverage for blocks without operations

* refactor(credentials): one saved-credential table and shared helpers

- withSavedCredential reads the replaced fields from SAVED_CREDENTIAL_OAUTH,
  so blocks and tools share one table instead of two hand-kept copies
- Reuse isTriggerModeSubBlock, getDependsOnFields, and a shared
  isSavedCredentialPicker; build the replacedBy map once in the registry check
- newWorkloadIdentitySubject owns the subject format; the AWS trust-pin probe
  runs alongside verification
- The issuer parses a bad signing key once instead of per request
- Token checks share the accepted result and the capped JSON reader

* chore(credentials): cleanup pass

- Keyless setup query gets its own key root, so broad credential
  invalidations can't refetch an open setup and mint a new subject
- Named staleTime constant for the setup query
- Correct stale comments (relying parties, Databricks param names)
- Tighten tests: assert the refusal reason, make the BYOK guard reachable,
  drop a duplicate and a test that could not fail

* fix(credentials): address review round 1

- A scoped 403 whose body can't be read is provider_unavailable, never accepted
- Provider error snippets read through the bounded reader
- Props interfaces for KeylessTrustFields and useWorkloadIdentitySetup; TSDoc on issuer responses

* fix(credentials): never cache the issuer's not-configured 404
2026-11-10 08:25:14 -08:00

5720 lines
209 KiB
TypeScript
Executable File

#!/usr/bin/env bun
import fs from 'fs'
import path from 'path'
import { fileURLToPath, pathToFileURL } from 'url'
import { isVersionedType, stripVersionSuffix } from '@sim/utils/string'
import ts from '@typescript/typescript6'
import { glob } from 'glob'
import remarkGfm from 'remark-gfm'
import remarkParse from 'remark-parse'
import { unified } from 'unified'
import { visit } from 'unist-util-visit'
import type { BlockCategory } from '../apps/sim/blocks/types'
import { IntegrationType } from '../apps/sim/blocks/types'
import type { OAuthServiceConfig } from '../apps/sim/lib/oauth/types'
import type { ToolOutputProperty } from '../apps/sim/tools/types'
import { formatGeneratedSource } from './format-generated-source'
/**
* Cache for resolved const definitions from types files.
* Key: "toolPrefix:constName" (e.g., "calcom:SCHEDULE_DATA_OUTPUT_PROPERTIES")
* Value: The resolved properties object
*/
const constResolutionCache = new Map<string, Record<string, any>>()
const __filename = fileURLToPath(import.meta.url)
const __dirname = path.dirname(__filename)
const rootDir = path.resolve(__dirname, '..')
const BLOCKS_PATH = path.join(rootDir, 'apps/sim/blocks/blocks')
export const DOCS_OUTPUT_PATH = path.join(rootDir, 'apps/docs/content/docs/integrations')
export const DOCS_ORIGIN = 'https://docs.sim.ai/'
/**
* The docs URL a block gets when it declares no `docsLink` — one generated page
* per service, named for the block's base type. Exported so the catalog checker
* validates the same contract this generator emits rather than a second copy of
* it that can silently drift.
*/
export function defaultIntegrationDocsUrl(blockType: string): string {
return `${DOCS_ORIGIN}integrations/${stripVersionSuffix(blockType)}`
}
const ICONS_PATH = path.join(rootDir, 'apps/sim/components/icons.tsx')
const DOCS_ICONS_PATH = path.join(rootDir, 'apps/docs/components/icons.tsx')
const INTEGRATIONS_DATA_PATH = path.join(rootDir, 'apps/sim/lib/integrations')
const INTEGRATIONS_CATALOG_PATH = path.join(rootDir, 'packages/deployment-config/src')
const LANDING_INTEGRATIONS_DATA_PATH = path.join(
rootDir,
'apps/sim/app/(landing)/integrations/data'
)
const TRIGGERS_PATH = path.join(rootDir, 'apps/sim/triggers')
const sourceFileCache = new Map<string, string>()
const sourceGlobCache = new Map<string, Promise<string[]>>()
const blockConfigCache = new Map<string, ReturnType<typeof extractAllBlockConfigs>>()
interface SourceObjectDeclaration {
name: string
start: number
end: number
content: string
satisfies: boolean
blockConfig: boolean
}
const sourceObjectCache = new Map<string, SourceObjectDeclaration[]>()
/** Type assertions and satisfies checks do not change an initializer's runtime value. */
function unwrapExpression(expression: ts.Expression): {
expression: ts.Expression
satisfies: boolean
} {
let current = expression
let satisfies = false
while (
ts.isAsExpression(current) ||
ts.isSatisfiesExpression(current) ||
ts.isParenthesizedExpression(current) ||
ts.isTypeAssertionExpression(current)
) {
satisfies ||= ts.isSatisfiesExpression(current)
current = current.expression
}
return { expression: current, satisfies }
}
/** Reads exported object initializers without executing integration modules. */
function sourceObjectDeclarations(content: string): SourceObjectDeclaration[] {
const cached = sourceObjectCache.get(content)
if (cached) return cached
const source = ts.createSourceFile('integration.ts', content, ts.ScriptTarget.Latest, true)
const declarations: SourceObjectDeclaration[] = []
for (const statement of source.statements) {
if (
!ts.isVariableStatement(statement) ||
!statement.modifiers?.some((modifier) => modifier.kind === ts.SyntaxKind.ExportKeyword)
)
continue
for (const declaration of statement.declarationList.declarations) {
if (!ts.isIdentifier(declaration.name) || !declaration.initializer) continue
const { expression: initializer, satisfies } = unwrapExpression(declaration.initializer)
if (!ts.isObjectLiteralExpression(initializer)) continue
const start = initializer.getStart(source)
const end = initializer.getEnd()
declarations.push({
name: declaration.name.text,
start,
end,
content: content.slice(start, end),
satisfies,
blockConfig:
/^BlockConfig\b/.test(declaration.type?.getText(source) ?? '') ||
/\bsatisfies\s+BlockConfig\b/.test(content.slice(end, declaration.getEnd())),
})
}
}
sourceObjectCache.set(content, declarations)
return declarations
}
function blockDeclarations(content: string): SourceObjectDeclaration[] {
return sourceObjectDeclarations(content).filter(
(declaration) => declaration.name.endsWith('Block') && declaration.blockConfig
)
}
function readSourceFile(filePath: string): string {
const cached = sourceFileCache.get(filePath)
if (cached !== undefined) return cached
const source = fs.readFileSync(filePath, 'utf-8')
sourceFileCache.set(filePath, source)
return source
}
async function sourceGlob(pattern: string): Promise<string[]> {
let pending = sourceGlobCache.get(pattern)
if (!pending) {
pending = glob(pattern)
sourceGlobCache.set(pattern, pending)
}
return [...(await pending)]
}
function blockConfigsForFile(filePath: string): ReturnType<typeof extractAllBlockConfigs> {
const cached = blockConfigCache.get(filePath)
if (cached) return cached
const configs = extractAllBlockConfigs(readSourceFile(filePath))
blockConfigCache.set(filePath, configs)
return configs
}
// Integration triggers are merged into the same per-service page as the service's
// actions (one block per integration: actions + an optional Trigger).
const TRIGGER_DOCS_OUTPUT_PATH = DOCS_OUTPUT_PATH
const integrationNavigation = JSON.parse(
fs.readFileSync(path.join(rootDir, 'apps/docs/content/integration-navigation.json'), 'utf-8')
) as {
guides: Record<string, { integration?: string; title: string }>
redirects: Record<string, string>
}
/**
* Hand-written integration pages in DOCS_OUTPUT_PATH that the generator must
* never clobber. Hand-authored credential guides are registered in the shared
* docs navigation file — these pages carry no `MANUAL-CONTENT` markers and no
* backing block, so the stale-doc cleanup deletes any that go unregistered.
*/
const HANDWRITTEN_INTEGRATION_DOCS = new Set([
'index',
'a2a',
...Object.keys(integrationNavigation.guides),
])
/**
* Native Sim resource blocks (category 'blocks') that still get a generated
* integration page. The writer's filter, the stale-doc cleanup, and the icon
* map must all honor this set: cleanup would otherwise delete what the writer
* emits (losing manual content), and an icon map that omits these types leaves
* their pages rendering the two-letter text fallback instead of the icon.
*/
const NATIVE_RESOURCE_BLOCK_TYPES = new Set([
'memory',
'knowledge',
'table',
'enrichment',
'logs',
'deployments',
])
/** Trigger doc pages that are hand-written and must never be overwritten. */
const HANDWRITTEN_TRIGGER_DOCS = new Set([
'index',
'start',
'schedule',
'webhook',
'rss',
'table',
'sim',
])
/** Omits hand-written providers and Slack's superseded legacy webhook trigger. */
const SKIP_TRIGGER_PROVIDERS = new Set(['generic', 'rss', 'table', 'sim', 'slack'])
/**
* Maps trigger provider names (from TriggerConfig.provider) to their
* corresponding block type when the two differ. Used to resolve icon
* colours from the block registry.
*/
const PROVIDER_TO_BLOCK_TYPE: Record<string, string> = {
'microsoft-teams': 'microsoft_teams',
'google-calendar': 'google_calendar',
'google-drive': 'google_drive',
'google-sheets': 'google_sheets',
jsm: 'jira_service_management',
slack_app: 'slack',
}
/** Human-readable display names for trigger providers. */
const TRIGGER_PROVIDER_DISPLAY_NAMES: Record<string, string> = {
airtable: 'Airtable',
ashby: 'Ashby',
attio: 'Attio',
calcom: 'Cal.com',
calendly: 'Calendly',
circleback: 'Circleback',
confluence: 'Confluence',
fathom: 'Fathom',
fireflies: 'Fireflies',
github: 'GitHub',
gmail: 'Gmail',
gong: 'Gong',
'google-calendar': 'Google Calendar',
'google-drive': 'Google Drive',
'google-sheets': 'Google Sheets',
google_forms: 'Google Forms',
grain: 'Grain',
greenhouse: 'Greenhouse',
hubspot: 'HubSpot',
imap: 'IMAP',
intercom: 'Intercom',
jira: 'Jira',
lemlist: 'Lemlist',
linear: 'Linear',
'microsoft-teams': 'Microsoft Teams',
notion: 'Notion',
outlook: 'Outlook',
resend: 'Resend',
salesforce: 'Salesforce',
servicenow: 'ServiceNow',
slack: 'Slack',
stripe: 'Stripe',
telegram: 'Telegram',
tiktok: 'TikTok',
twilio_voice: 'Twilio Voice',
typeform: 'Typeform',
vercel: 'Vercel',
webflow: 'Webflow',
whatsapp: 'WhatsApp',
zoom: 'Zoom',
}
if (!fs.existsSync(DOCS_OUTPUT_PATH)) {
fs.mkdirSync(DOCS_OUTPUT_PATH, { recursive: true })
}
const docsComponentsDir = path.dirname(DOCS_ICONS_PATH)
if (!fs.existsSync(docsComponentsDir)) {
fs.mkdirSync(docsComponentsDir, { recursive: true })
}
/** Runtime set of valid `IntegrationType` values, derived from the canonical enum. */
const INTEGRATION_CATEGORY_VALUES: ReadonlySet<IntegrationType> = new Set(
Object.values(IntegrationType)
)
interface BlockOperation {
label: string
id: string
group?: string
}
/**
* Defensive shape for blocks parsed out of source files. Fields stay loose
* (`string`) so the AST-style extractor can populate them progressively; the
* canonical taxonomy is enforced at the JSON-write boundary inside
* `writeIntegrationsJson`.
*/
interface BlockConfig {
type: string
name: string
description: string
longDescription?: string
category: string
integrationType?: string
bgColor?: string
outputs?: Record<string, any>
tools?: {
access?: string[]
}
operations?: BlockOperation[]
/**
* Param names the block itself supplies — via a `subBlocks` field (id or
* `canonicalParamId`) or via its `tools.config.params` mapper.
*
* `null` means the block's `subBlocks` array could not be read, so which params it supplies
* is UNKNOWN and the hidden-param filter is skipped for it. Never conflate that with `[]`,
* which asserts the block supplies nothing and strips every hidden param from its page.
*/
userSettableParamIds?: string[] | null
docsLink?: string
[key: string]: any
}
/**
* True when a block's source text marks it as an unreleased `preview: true`
* block. THE single preview gate for this script — every surface it emits
* (docs .mdx, integrations.json, icon mapping) must consult this, because a
* missed gate publishes an unreleased block to docs.sim.ai, the catalog, the
* sitemap, and OG images. Mirrors the `hideFromToolbar` source-text checks.
*/
function isPreviewSource(blockContent: string): boolean {
return /preview\s*:\s*true/.test(blockContent)
}
/**
* Blank out `//` and block comments so source-text property probes match real
* code only. Without this, prose that quotes a property — e.g. slack.ts's
* "At v2 GA this becomes `hideFromToolbar: true`" — reads as the property
* itself and silently drops the block from every generated surface.
*
* Comment bodies are replaced with spaces rather than removed so byte offsets
* stay aligned with the original content. Deliberately not applied to
* {@link isPreviewSource}: that gate is fail-closed on purpose, and a
* false positive there only over-hides an unreleased block.
*/
function stripSourceComments(content: string): string {
return content
.replace(/\/\*[\s\S]*?\*\//g, (m) => m.replace(/[^\n]/g, ' '))
.replace(/(^|[^:])\/\/[^\n]*/g, (m, prefix) => prefix + ' '.repeat(m.length - prefix.length))
}
/**
* Find the position after the matching close delimiter for an opening delimiter.
* Assumes `content[openPos]` is the opening char (e.g. `{` or `[`).
* Returns the index one past the matching close char, or -1 if unbalanced.
*/
function findMatchingClose(
content: string,
openPos: number,
openChar = '{',
closeChar = '}'
): number {
let count = 1
let pos = openPos + 1
while (pos < content.length && count > 0) {
if (content[pos] === openChar) count++
else if (content[pos] === closeChar) count--
pos++
}
return count === 0 ? pos : -1
}
interface TriggerInfo {
id: string
name: string
description: string
}
interface TriggerConfigField {
id: string
title: string
type: string
required: boolean
description?: string
placeholder?: string
}
/** The subset of `SubBlockConfig` the generated configuration table reads. */
interface RegistrySubBlock {
id?: string
type?: string
title?: string
description?: string
placeholder?: unknown
/** `true`, or a condition object making the field required only for some configurations. */
required?: unknown
hidden?: boolean
readOnly?: boolean
}
interface RegistryTrigger {
id?: string
name?: string
provider?: string
description?: string
polling?: boolean
deprecated?: boolean
subBlocks?: RegistrySubBlock[]
outputs?: Record<string, any>
}
/** Present for the operator, not part of a trigger's configuration surface. */
const TRIGGER_UI_ONLY_IDS = new Set([
'webhookUrlDisplay',
'triggerInstructions',
'selectedTriggerId',
])
/**
* Loads the evaluated trigger registry.
*
* Imported by absolute path so Bun resolves the `@/` aliases against `apps/sim`'s tsconfig
* rather than this script's.
*/
async function loadTriggerRegistry(): Promise<Record<string, RegistryTrigger>> {
const module = await import(path.join(rootDir, 'apps/sim/triggers/registry.ts'))
return module.TRIGGER_REGISTRY as Record<string, RegistryTrigger>
}
interface ToolMetadataParam {
type?: string
required?: boolean
description?: string
visibility?: string
}
interface ToolMetadataOAuth {
provider?: string
requiredScopes?: string[]
credentialKind?: 'oauth' | 'service-account'
}
interface ToolMetadataEntry {
name?: string
description?: string
params?: Record<string, ToolMetadataParam>
oauth?: ToolMetadataOAuth
}
/** Client-safe tool metadata, keyed by tool id and kept in sync with the registry by CI. */
let toolMetadata: Record<string, ToolMetadataEntry> | null = null
async function loadToolMetadata(): Promise<Record<string, ToolMetadataEntry>> {
if (toolMetadata) return toolMetadata
const module = await import(path.join(rootDir, 'apps/sim/tools/generated/tool-metadata.ts'))
toolMetadata = module.default as Record<string, ToolMetadataEntry>
return toolMetadata
}
/** Evaluated tool output schemas, keyed by tool id and kept in sync with the registry by CI. */
let toolOutputs: Record<string, Record<string, ToolOutputProperty>> | null = null
async function loadToolOutputs(): Promise<Record<string, Record<string, ToolOutputProperty>>> {
if (toolOutputs) return toolOutputs
const module = await import(path.join(rootDir, 'apps/sim/tools/generated/tool-outputs.ts'))
toolOutputs = module.default as Record<string, Record<string, ToolOutputProperty>>
return toolOutputs
}
/** Human-facing tool names, keyed by tool id. Kept in sync with the registry by CI. */
let toolDisplayNames: Map<string, string> | null = null
async function loadToolDisplayNames(): Promise<Map<string, string>> {
if (toolDisplayNames) return toolDisplayNames
const metadata = await loadToolMetadata()
toolDisplayNames = new Map(
Object.entries(metadata).flatMap(([id, entry]) =>
entry?.name ? [[id, entry.name] as const] : []
)
)
return toolDisplayNames
}
interface TriggerFullInfo {
id: string
name: string
description: string
provider: string
polling: boolean
outputs: Record<string, any>
configFields: TriggerConfigField[]
}
interface OperationInfo {
name: string
description: string
}
interface IntegrationEntry {
type: string
slug: string
name: string
description: string
longDescription: string
bgColor: string
iconName: string
docsUrl: string
operations: OperationInfo[]
operationCount: number
triggers: TriggerInfo[]
triggerCount: number
authType: 'oauth' | 'api-key' | 'none'
oauthServiceId?: string
category: BlockCategory
integrationType: IntegrationType
tags?: string[]
landingContent?: Record<string, unknown>
}
/** A block icon component together with the module it must be imported from. */
interface IconRef {
name: string
source: string
}
/**
* Check mode (`--check`): render every generated artifact in memory and compare
* it against the committed file instead of writing, so CI can fail on docs
* drift the same way `check:tool-metadata` fails on stale tool metadata. Check
* mode performs no filesystem mutations.
*
* The pipeline writes some pages twice per run — the block pass writes the base
* page, then the trigger pass reads it back and appends/merges the Triggers
* section — so check mode keeps an in-memory overlay of everything "written"
* this run (`emittedByPath`), readers consult the overlay before disk
* (`readGeneratedFile`), and staleness is judged once at the end against each
* artifact's FINAL content. Comparing at emit time would flag the intermediate
* block-pass content of every trigger-owning page as a false positive.
*
* Known limitation: `updateMetaJson` derives the sidebar from the mdx files on
* disk, so in check mode a brand-new block's missing page is reported directly
* while the corresponding meta.json entry is not — regenerating fixes both.
*/
let CHECK_ONLY = false
const staleArtifacts: string[] = []
const emittedByPath = new Map<string, string>()
/**
* Deletion candidates recorded by cleanup in check mode. Judged at the end of
* the run, not at cleanup time: generate mode deletes a non-canonical page and
* lets the trigger pass recreate it in the same run, so a candidate that was
* re-emitted this run is that delete-then-recreate dance — content drift (if
* any) is already covered by the overlay comparison — while a candidate nothing
* re-emitted is a genuinely stale page regeneration would remove.
*/
const wouldDeletePaths: string[] = []
/** Writes a generated artifact, or in check mode records its final content for the end-of-run comparison. */
function emitGeneratedFile(filePath: string, content: string): void {
if (CHECK_ONLY) {
emittedByPath.set(filePath, content)
return
}
fs.writeFileSync(filePath, content)
}
/** Reads a generated artifact as the pipeline would see it mid-run: overlay first in check mode, then disk. */
function readGeneratedFile(filePath: string): string | null {
const emitted = emittedByPath.get(filePath)
if (emitted !== undefined) return emitted
return fs.existsSync(filePath) ? fs.readFileSync(filePath, 'utf-8') : null
}
/** Compares every overlay entry against the committed file; returns repo-relative stale paths. */
function collectStaleEmissions(): string[] {
const stale: string[] = []
for (const [filePath, content] of emittedByPath) {
const committed = fs.existsSync(filePath) ? fs.readFileSync(filePath, 'utf-8') : null
if (committed !== content) stale.push(path.relative(rootDir, filePath))
}
return stale
}
/**
* Copy the icons.tsx file from the main sim app to the docs app
* This ensures icons are rendered consistently across both apps
*/
function copyIconsFile(): void {
try {
if (!CHECK_ONLY) console.log('Copying icons from sim app to docs app...')
if (!fs.existsSync(ICONS_PATH)) {
console.error(`Source icons file not found: ${ICONS_PATH}`)
return
}
const iconsContent = readSourceFile(ICONS_PATH)
emitGeneratedFile(DOCS_ICONS_PATH, iconsContent)
if (!CHECK_ONLY) console.log('✓ Icons successfully copied to docs app')
} catch (error) {
console.error('Error copying icons file:', error)
}
}
/**
* Some trigger providers have no block of their own (`slack_app`, `twilio`) yet
* still get a generated page keyed by the provider id. Seed those provider ids
* from the trigger definitions' own `icon` so their pages render the brand mark
* instead of the two-letter fallback. Never overwrites a block-derived entry —
* the block is the canonical icon source when one exists.
*/
async function addTriggerProviderIcons(
iconMappings: readonly Record<string, IconRef>[]
): Promise<void> {
const triggerFiles = (await sourceGlob(`${TRIGGERS_PATH}/**/*.ts`)).filter(
(f) => !f.includes('.test.')
)
const previewOnly = await collectPreviewOnlyTriggerIds()
for (const file of triggerFiles) {
const fileContent = readSourceFile(file)
const source = stripSourceComments(fileContent)
// Pair each trigger's `id` with the `provider` that follows it in the same
// config, so files holding several trigger configs attribute each provider
// (and its icon) to the right trigger.
const configRegex =
/\bid\s*:\s*['"]([^'"]+)['"][\s\S]{0,600}?\bprovider\s*:\s*['"]([^'"]+)['"]/g
for (const match of source.matchAll(configRegex)) {
const [, triggerId, provider] = match
if (iconMappings.every((iconMapping) => iconMapping[provider])) continue
// Preview-only triggers get no page, so they need no provider icon.
if (previewOnly.has(triggerId)) continue
const iconName = extractIconNameFromContent(source.slice(match.index))
if (!iconName) continue
const iconRef = { name: iconName, source: resolveIconSource(fileContent, iconName) }
for (const iconMapping of iconMappings) {
if (!iconMapping[provider]) iconMapping[provider] = iconRef
}
}
}
}
/**
* Generate icon mapping from block definitions.
* Docs need hidden historical version keys so older integration pages and
* versioned docs links still render icons, while landing only needs visible blocks.
*/
export async function generateIconMappings(): Promise<{
docs: Record<string, IconRef>
visible: Record<string, IconRef>
coreBlockTypes: string[]
}> {
try {
console.log('Generating icon mapping from block definitions...')
// OCI credential setup has no product block from which docs can derive its family icon.
const docs: Record<string, IconRef> = {
oci: { name: 'OracleIcon', source: '@/components/icons' },
}
const visible: Record<string, IconRef> = {}
const coreBlockTypes = new Set<string>()
const blockFiles = (await sourceGlob(`${BLOCKS_PATH}/*.ts`)).sort()
for (const blockFile of blockFiles) {
const fileContent = readSourceFile(blockFile)
// For icon mapping, we need ALL blocks including hidden ones
// because V2 blocks inherit icons from legacy blocks via spread
// First, extract the primary icon from the file (usually the legacy block's icon)
const primaryIcon = extractIconNameFromContent(fileContent)
for (const declaration of blockDeclarations(fileContent)) {
const blockName = declaration.name.replace(/Block$/, '')
const startIndex = declaration.start
const endIndex = declaration.end
if (endIndex !== -1) {
const blockContent = fileContent.substring(startIndex, endIndex)
// Check hideFromToolbar - skip hidden blocks for docs but NOT for icon mapping
const hideFromToolbar = /hideFromToolbar\s*:\s*true/.test(
stripSourceComments(blockContent)
)
// Unreleased preview blocks never reach any public surface, icon map included.
if (isPreviewSource(blockContent)) {
continue
}
const blockType =
extractStringPropertyFromContent(blockContent, 'type') || blockName.toLowerCase()
const iconName = extractIconNameFromContent(blockContent) || primaryIcon
if (!blockType || !iconName) {
continue
}
const category = extractStringPropertyFromContent(blockContent, 'category') || 'misc'
const iconRef = {
name: iconName,
source: resolveIconSource(fileContent, iconName),
}
/** Core reference previews share the registry's glyphs without entering the integration catalog. */
const inheritedCategory = extractInheritedBlockCategory(blockContent, fileContent)
if (inheritedCategory === 'blocks' || inheritedCategory === 'triggers') {
docs[blockType] = iconRef
coreBlockTypes.add(blockType)
}
if (
blockType.includes('_trigger') ||
blockType.includes('_webhook') ||
blockType.includes('rss')
) {
continue
}
// Exclude first-party `blocks`-category primitives (except the native
// resource blocks that still get a generated docs page) and
// core/plumbing types. Keying the exception off
// `NATIVE_RESOURCE_BLOCK_TYPES` — the same set the docs writer uses —
// keeps the icon map from drifting behind the pages that consume it.
const baseType = stripVersionSuffix(blockType)
if (
(category === 'blocks' &&
!NATIVE_RESOURCE_BLOCK_TYPES.has(baseType) &&
!HANDWRITTEN_INTEGRATION_DOCS.has(baseType)) ||
ICON_MAP_EXCLUDED_TYPES.has(blockType)
) {
continue
}
const isVersionedBlockType = isVersionedType(blockType)
/**
* A sunset block keeps its docs page — `docsLink` is baked into every
* placed instance — so it still needs an icon there, exactly like a
* hidden versioned block. Without this it renders as a text tile.
*/
const isSunsetBlockType = /sunset\s*:\s*\{/.test(stripSourceComments(blockContent))
if (!hideFromToolbar) {
docs[blockType] = iconRef
visible[blockType] = iconRef
} else if (isVersionedBlockType || isSunsetBlockType) {
docs[blockType] = iconRef
}
}
}
}
await addTriggerProviderIcons([docs, visible])
console.log(
`✓ Generated icon mappings for ${Object.keys(docs).length} docs blocks and ` +
`${Object.keys(visible).length} visible blocks`
)
return { docs, visible, coreBlockTypes: [...coreBlockTypes].sort() }
} catch (error) {
console.error('Error generating icon mapping:', error)
return { docs: {}, visible: {}, coreBlockTypes: [] }
}
}
/**
* Sort strings to match Biome's organizeImports order:
* case-insensitive character-by-character, uppercase before lowercase as tiebreaker.
*/
function biomeSortCompare(a: string, b: string): number {
const minLen = Math.min(a.length, b.length)
for (let i = 0; i < minLen; i++) {
const al = a[i].toLowerCase()
const bl = b[i].toLowerCase()
if (al !== bl) return al < bl ? -1 : 1
if (a[i] !== b[i]) return a[i] < b[i] ? -1 : 1
}
return a.length - b.length
}
/**
* Adds a bare-name alias for each versioned block type (`github_v2` → `github`)
* so lookups by base type resolve: trigger provider names on the docs side, and
* template icon clusters (`otherBlockTypes` are base types) in the app.
*/
function withBareNameAliases(iconMapping: Record<string, IconRef>): Record<string, IconRef> {
const withAliases: Record<string, IconRef> = { ...iconMapping }
for (const [blockType, iconRef] of Object.entries(iconMapping)) {
const baseType = stripVersionSuffix(blockType)
if (baseType !== blockType && !withAliases[baseType]) {
withAliases[baseType] = iconRef
}
}
return withAliases
}
function writeIconMapping(iconMapping: Record<string, IconRef>, coreBlockTypes: string[]): void {
try {
const iconMappingPath = path.join(rootDir, 'apps/docs/components/ui/icon-mapping.ts')
const withAliases = withBareNameAliases(iconMapping)
const imports = renderIconImports(Object.values(withAliases))
const coreTypeEntries = coreBlockTypes.map((type) => ` '${type}',`).join('\n')
// Generate mapping with direct references (no dynamic access for tree shaking)
const mappingEntries = Object.entries(withAliases)
.sort(([a], [b]) => compareCatalogNames(a, b))
.map(([blockType, iconRef]) => ` ${formatIconMapKey(blockType)}: ${iconRef.name},`)
.join('\n')
const content = `// Auto-generated file - do not edit manually
// Generated by scripts/generate-docs.ts
// Maps block types to their icon component references
import type { ComponentType, SVGProps } from 'react'
${imports}
type IconComponent = ComponentType<SVGProps<SVGSVGElement>>
export const blockTypeToIconMap: Record<string, IconComponent> = {
${mappingEntries}
}
export const coreBlockTypes = new Set([
${coreTypeEntries}
])
`
emitGeneratedFile(iconMappingPath, content)
if (!CHECK_ONLY) console.log('✓ Icon mapping file written to docs app')
} catch (error) {
console.error('Error writing icon mapping:', error)
}
}
/**
* Raised when a block's `subBlocks` array is present but cannot be read. Distinguishes
* a parse failure from a block that genuinely exposes no fields — both used to surface
* as an empty array, and the empty array silently strips documented rows.
*/
class SubBlockParseError extends Error {
override name = 'SubBlockParseError'
}
/** Blocks already warned about. The same block is re-parsed by the page pass, the icon pass and
* each spread-base recursion, so without this the same warning prints several times. */
const subBlockParseWarnings = new Set<string>()
/**
* Collects the param names a block exposes to the user through its own `subBlocks`.
*
* A subBlock's `id` is the param it writes, unless it declares `canonicalParamId`,
* which is how a differently-named field maps onto a tool param. A tool param marked
* `visibility: 'hidden'` is not an LLM-settable tool argument, but when the block
* declares a matching field the value is still typed by the user (e.g. Mailchimp's
* `apiKey`) and must stay documented. Params with no matching field are genuinely
* server-derived (Jira's `cloudId`, Salesforce's `idToken`) and stay filtered out.
*
* Brace matching runs on a blanked copy so braces inside string literals and comments
* cannot skew it; only depth-1 properties of each subBlock are read, so `id` fields on
* nested `options`/`condition` objects are never mistaken for the subBlock's own id.
*
* Returns `null` for UNKNOWN — an array whose elements are all spreads of fields arrays this
* scanner cannot follow (`...NotionBlock.subBlocks`, `...getTrigger('x').subBlocks`). `[]` is
* reserved for a block that genuinely exposes no fields, because `[]` strips every hidden param
* from the page. Throws {@link SubBlockParseError} when the array is there but unreadable.
*/
export function extractUserSettableParamIds(
blockContent: string,
blockName = 'block'
): string[] | null {
const scannable = blankStringsAndComments(blockContent)
if (scannable === null) return null
const keyMatch = /\bsubBlocks\s*:/.exec(scannable)
if (!keyMatch) return []
const afterKey = keyMatch.index + keyMatch[0].length
const literalMatch = /^\s*\[/.exec(scannable.slice(afterKey))
if (!literalMatch) {
throw new SubBlockParseError(
`${blockName}: subBlocks is built by an expression rather than an array literal, so the fields it contributes cannot be read`
)
}
const arrayStart = afterKey + literalMatch[0].length - 1
const arrayEnd = findMatchingClose(scannable, arrayStart, '[', ']')
if (arrayEnd === -1) {
throw new SubBlockParseError(
`${blockName}: found a subBlocks array but could not locate its closing bracket`
)
}
const ids = new Set<string>()
let elementsWithoutIds = 0
/**
* Text of the array's own elements with every object literal, call argument and nested
* bracket elided, so each remaining comma-separated segment is one element's head. Used to
* tell an element that names an existing fields array from one that hides its fields behind
* a helper call.
*/
let elementHeads = ''
let nesting = 0
let i = arrayStart + 1
while (i < arrayEnd - 1) {
const char = scannable[i]
if (nesting === 0 && char === '{') {
const objectEnd = findMatchingClose(scannable, i)
if (objectEnd === -1) break
let depth = 0
let topLevel = ''
const sourceIndices: number[] = []
for (let k = i; k < objectEnd; k++) {
const inner = scannable[k]
if (inner === '{' || inner === '[') {
depth++
continue
}
if (inner === '}' || inner === ']') {
depth--
continue
}
if (depth === 1) {
topLevel += inner
sourceIndices.push(k)
}
}
/**
* Matching runs on the blanked characters, so an `id:` sitting inside a string value or a
* `//` comment cannot be mistaken for the subBlock's own id. Blanking keeps a string's
* quotes and its length, so the matched literal's value is read back character by character
* from the original content at the indices the blanked copy matched at.
*/
const readLiteral = (match: RegExpExecArray): string => {
const valueStart = match.index + match[0].length - 1 - match[1].length
let value = ''
for (let offset = 0; offset < match[1].length; offset++) {
value += blockContent[sourceIndices[valueStart + offset]]
}
return value
}
const idMatch = /\bid\s*:\s*['"]([^'"]+)['"]/.exec(topLevel)
if (idMatch) ids.add(readLiteral(idMatch))
const canonicalMatch = /\bcanonicalParamId\s*:\s*['"]([^'"]+)['"]/.exec(topLevel)
if (canonicalMatch) ids.add(readLiteral(canonicalMatch))
/**
* An object that spreads an existing subBlock to override one property
* (`{ ...sb, required: true }`) legitimately carries no id of its own — the id comes from
* the spread source. Only an object with neither an id nor a spread means the scan failed.
*/
if (!idMatch && !canonicalMatch && !topLevel.includes('...')) elementsWithoutIds++
i = objectEnd
continue
}
if (char === '(' || char === '[') {
nesting++
i++
continue
}
if (char === ')' || char === ']') {
nesting--
i++
continue
}
if (nesting === 0) elementHeads += char
i++
}
/**
* Any id at all means the array was read and the page keeps a populated Input table. An
* opaque element alongside real ids can only omit extra rows — the long-standing limitation
* that a spread contributes ids this scanner never sees — and is not this guard's business.
* The guard exists solely to stop an empty result, because empty is what strips every hidden
* param from the page.
*/
if (ids.size > 0) return [...ids]
if (elementsWithoutIds > 0) {
throw new SubBlockParseError(
`${blockName}: subBlocks array holds object literals but no id was extracted`
)
}
/**
* Zero ids is a legitimate answer only when every element names an existing fields array
* (`...NotionBlock.subBlocks`, `...getTrigger('x').subBlocks`, `...Base.subBlocks.filter(…)`),
* because those fields reach the page through the spread base instead. An element that is a
* bare helper call (`...getSlackV2ActionSubBlocks()`) hides whatever fields the helper builds,
* and used to yield a silent empty array indistinguishable from a spread-only block.
*/
const segments = elementHeads
.split(',')
.map((segment) => segment.trim())
.filter((segment) => segment.length > 0)
const opaque = segments.filter((segment) => !segment.includes('.subBlocks'))
if (opaque.length > 0) {
throw new SubBlockParseError(
`${blockName}: subBlocks array yielded no ids and element${
opaque.length > 1 ? 's' : ''
} ${opaque.map((segment) => `\`${segment}\``).join(', ')} do not name a fields array`
)
}
/**
* Every element named a fields array this scanner cannot follow, so the block's fields are
* UNKNOWN, not empty. Returning `[]` here would assert the block supplies nothing and strip
* every hidden param from its tools' Input tables with no warning — the silent false-drop the
* `null` state exists to prevent. Only a genuinely empty array (`subBlocks: []`) reaches the
* `[]` below.
*/
if (segments.length > 0) return null
return []
}
/**
* Locates the bodies of every `tools.config.params` mapper in `scannable`.
*
* Returns `[start, end)` index pairs into `scannable` (a length-preserving blanked copy, so
* the same indices address the original content).
*
* In production this only ever runs on a single block's slice, which holds at most one
* `subBlocks:` key — the loop over every `tools` object is defensive rather than required, and
* the multi-block file it was once justified by (Textract's v1 and v2) is split before it gets
* here. The tests do pass whole files, so the loop is exercised on wider input than production
* ever supplies.
*
* Handles `params: (params) => { ... }`, the concise `params: (params) => ({ ... })` form, the
* `async` and generic-annotated variants, and method shorthand. Candidates are tried in order
* rather than only the first, because a decoy key that is not a mapper at all
* (`params: (GitHubBlock.tools?.config as any)?.params`) would otherwise mask the real one.
*/
function findMapperBodyRanges(scannable: string): [number, number][] {
const ranges: [number, number][] = []
const toolsRegex = /\btools\s*:\s*\{/g
let toolsMatch: RegExpExecArray | null
while ((toolsMatch = toolsRegex.exec(scannable)) !== null) {
const toolsEnd = findMatchingClose(scannable, toolsMatch.index + toolsMatch[0].length - 1)
if (toolsEnd === -1) continue
toolsRegex.lastIndex = toolsEnd
const toolsRegion = scannable.slice(toolsMatch.index, toolsEnd)
const configMatch = /\bconfig\s*:\s*\{/.exec(toolsRegion)
if (!configMatch) continue
const configStart = toolsMatch.index + configMatch.index + configMatch[0].length - 1
const configEnd = findMatchingClose(scannable, configStart)
if (configEnd === -1) continue
const configRegion = scannable.slice(configStart, configEnd)
const paramsRegex = /\bparams\s*(?::\s*(?:async\s*)?(?:<[^<>]*>\s*)?)?\(/g
let paramsMatch: RegExpExecArray | null
while ((paramsMatch = paramsRegex.exec(configRegion)) !== null) {
const argsStart = configStart + paramsMatch.index + paramsMatch[0].length - 1
const argsEnd = findMatchingClose(scannable, argsStart, '(', ')')
if (argsEnd === -1) continue
const afterArgs = scannable.slice(argsEnd, configEnd)
// An expression body that delegates to a builder: `(params) => buildParams(params)`.
const callMatch = /^\s*=>\s*[A-Za-z_$][\w$]*\s*\(/.exec(afterArgs)
if (callMatch) {
const callOpen = argsEnd + callMatch[0].length - 1
const callEnd = findMatchingClose(scannable, callOpen, '(', ')')
if (callEnd === -1) continue
ranges.push([argsEnd + callMatch[0].search(/[A-Za-z_$]/), callEnd])
break
}
const bodyMatch = /^\s*(?::[^=({]*)?(?:=>\s*)?([({])/.exec(afterArgs)
if (!bodyMatch) continue
const open = bodyMatch[1] as '(' | '{'
const bodyStart = argsEnd + bodyMatch[0].length - 1
const bodyEnd = findMatchingClose(scannable, bodyStart, open, open === '(' ? ')' : '}')
if (bodyEnd === -1) continue
ranges.push([bodyStart, bodyEnd])
break
}
}
return ranges
}
/**
* Adds the shorthand property names of every object literal in `body` to `into`.
*
* `{ file }` names the `file` param exactly as `{ file: value }` does, but carries no colon,
* so the key scan below cannot see it — a mapper written in the idiomatic shorthand form used
* to drop the param from the docs silently, which is the one failure mode this whole filter
* exists to prevent.
*
* Only the depth-1 comma segments of a brace-matched region are read, and a segment that
* opens a call or an index is marked so it can no longer look like a bare identifier. That
* keeps argument lists (`fn(a, b, c)`), calls (`{ doWork() }`) and nested values
* (`{ a: { b: 1 }, file }`) from contributing names, while `{ ...rest, file }` still yields
* `file` because `...rest` is not an identifier on its own.
*/
function collectShorthandPropertyNames(body: string, into: Set<string>): void {
for (let i = 0; i < body.length; i++) {
if (body[i] !== '{') continue
const objectEnd = findMatchingClose(body, i)
if (objectEnd === -1) continue
const segments: string[] = []
let current = ''
let depth = 0
for (let k = i; k < objectEnd; k++) {
const char = body[k]
if (char === '{' || char === '[' || char === '(') {
depth++
if (depth === 2) current += '#'
continue
}
if (char === '}' || char === ']' || char === ')') {
depth--
continue
}
if (depth !== 1) continue
if (char === ',') {
segments.push(current)
current = ''
continue
}
current += char
}
segments.push(current)
for (const segment of segments) {
const name = segment.trim()
if (/^[A-Za-z_$][\w$]*$/.test(name)) into.add(name)
}
}
}
/**
* Collects the tool-param names a block's own `tools.config.params` mapper writes.
*
* A block can supply a hidden tool param without ever declaring a subBlock of that name:
* Cal.com assembles `result.attendee` from `attendeeName`/`attendeeEmail`/`attendeeTimeZone`,
* JSM renames its `assetWorkspaceId` field to `workspaceId`, and Textract renames its
* `document` field to `file`. All are user-driven and must stay documented, so this scan
* catches both shapes — `<anyIdentifier>.<param> = …` assignments (the accumulator is named
* `result` in one block and `parameters` in another, so the name is never assumed) and
* `<param>:` keys of the objects the mapper returns.
*
* The rule is deliberately biased toward keeping. Object keys are collected without proving
* they are top-level params, so a key on a nested object (Cal.com's `attendee.name`) can keep
* a same-named hidden param that the mapper never actually supplies. That false keep costs a
* reader one hard-to-set row; a false drop hides a required input — the exact failure this
* filter exists to prevent, and one it has already caused. When the two are in tension, keep.
*
* Scanning runs on the blanked copy, so a commented-out or string-embedded mapper cannot
* contribute names.
*/
export function extractMapperWrittenParamIds(blockContent: string, fileContent?: string): string[] {
const scannable = blankStringsAndComments(blockContent)
if (scannable === null) return []
const ids = new Set<string>()
for (const [start, end] of findMapperBodyRanges(scannable)) {
collectWrittenParamIds(blockContent, scannable, start, end, ids)
if (fileContent) collectDelegatedParamIds(scannable.slice(start, end), fileContent, ids)
}
return [...ids]
}
/**
* A mapper that delegates to a builder declared in the same file
* (`params: (params) => s3ToolParams(params)`) writes whatever that builder
* writes, so the builder's body is scanned too. One level deep, which is as far
* as the delegation pattern goes.
*/
function collectDelegatedParamIds(mapperBody: string, fileContent: string, ids: Set<string>) {
const scannableFile = blankStringsAndComments(fileContent)
if (scannableFile === null) return
const calleeRegex = /(?:^|[^.\w$])([A-Za-z_$][\w$]*)\s*\(/g
const callees = new Set<string>()
let match: RegExpExecArray | null
while ((match = calleeRegex.exec(mapperBody)) !== null) callees.add(match[1])
for (const callee of callees) {
const declaration = new RegExp(`\\bfunction\\s+${callee}\\s*\\(`).exec(scannableFile)
if (!declaration) continue
const argsEnd = findMatchingClose(
scannableFile,
declaration.index + declaration[0].length - 1,
'(',
')'
)
if (argsEnd === -1) continue
const bodyStart = scannableFile.indexOf('{', argsEnd)
if (bodyStart === -1) continue
const bodyEnd = findMatchingClose(scannableFile, bodyStart)
if (bodyEnd === -1) continue
collectWrittenParamIds(fileContent, scannableFile, bodyStart, bodyEnd, ids)
}
}
/** Adds the param names one mapper body writes, read from its blanked copy. */
function collectWrittenParamIds(
original: string,
scannable: string,
start: number,
end: number,
ids: Set<string>
) {
const body = scannable.slice(start, end)
const assignmentRegex = /(?:^|[^.\w$])[A-Za-z_$][\w$]*\.([A-Za-z_$][\w$]*)\s*=(?!=)/g
let match: RegExpExecArray | null
while ((match = assignmentRegex.exec(body)) !== null) ids.add(match[1])
const keyRegex = /(?:^|[^?.\w$])([A-Za-z_$][\w$]*)\s*:/g
while ((match = keyRegex.exec(body)) !== null) ids.add(match[1])
/**
* Quoted keys survive blanking only as their delimiters, so the name is read back from
* the original content at the same index — `blankStringsAndComments` is length-preserving.
*/
const quotedKeyRegex = /(['"])[^'"\n]*\1\s*:/g
while ((match = quotedKeyRegex.exec(body)) !== null) {
const keyStart = start + match.index
const text = original.slice(keyStart, keyStart + match[0].length)
const name = /(['"])([A-Za-z_$][\w$]*)\1/.exec(text)?.[2]
if (name) ids.add(name)
}
/**
* A computed write (`result['file'] = …`) supplies a param exactly as `result.file = …`
* does. Like the quoted keys above, the name only survives blanking as its delimiters and
* is read back from the original content at the matched index.
*/
const bracketAssignmentRegex =
/(?:^|[^.\w$])[A-Za-z_$][\w$]*\s*\[\s*(['"])[^'"\n]*\1\s*\]\s*=(?!=)/g
while ((match = bracketAssignmentRegex.exec(body)) !== null) {
const matchStart = start + match.index
const text = original.slice(matchStart, matchStart + match[0].length)
const name = /\[\s*(['"])([A-Za-z_$][\w$]*)\1\s*\]/.exec(text)?.[2]
if (name) ids.add(name)
}
collectShorthandPropertyNames(body, ids)
}
/** What {@link extractBlockSuppliedParamIds} could and could not read off a block. */
export interface BlockSuppliedParams {
/**
* Every param the block supplies, or `null` when what its `subBlocks` array contributes is
* unknown — either the array could not be read (`parseError` set) or it holds only spreads of
* fields arrays this scanner cannot follow (`parseError` null).
*
* `null` is UNKNOWN and is deliberately distinct from `[]`: an empty array asserts the block
* supplies nothing, which strips every hidden param from the page, while `null` says the scan
* failed and the filter must be skipped entirely for this block.
*/
ids: string[] | null
/**
* The params the block's `tools.config.params` mapper writes. Collected even when the
* `subBlocks` scan failed, so a spread-inheriting block does not lose its mapper's renames
* along with its own fields.
*/
mapperIds: string[]
/** Why the `subBlocks` scan failed, when it did. `null` on success. */
parseError: string | null
}
/**
* Every param name the block itself supplies — via a user-facing `subBlocks` field or via
* its `tools.config.params` mapper. A hidden tool param in this set stays in the public
* Input table; the rest are genuinely resolver-derived and stay filtered out.
*
* An unreadable `subBlocks` array is reported as `ids: null` rather than thrown, because the
* only safe response to "the fields are unknown" is to stop filtering, never to filter against
* an empty set. The mapper scan runs first so its result survives that failure.
*/
export function extractBlockSuppliedParamIds(
blockContent: string,
blockName = 'block',
fileContent?: string
): BlockSuppliedParams {
/**
* A source the blanking scanner cannot get through — it ends inside an unterminated string,
* template literal or comment — makes both scans below report "nothing found" for the same
* reason. It is caught here so it is reported as a parse failure. Left to the branches below
* it would be indistinguishable from a spread-only `subBlocks` array whose block has no
* mapper, and the block's renames would be dropped without a word.
*/
if (blankStringsAndComments(blockContent) === null) {
return {
ids: null,
mapperIds: [],
parseError: `${blockName}: source ends inside an unterminated string, template literal or comment, so neither its subBlocks array nor its params mapper could be read`,
}
}
const mapperIds = extractMapperWrittenParamIds(blockContent, fileContent)
try {
const settableIds = extractUserSettableParamIds(blockContent, blockName)
if (settableIds === null) return { ids: null, mapperIds, parseError: null }
return { ids: [...new Set([...settableIds, ...mapperIds])], mapperIds, parseError: null }
} catch (error) {
if (!(error instanceof SubBlockParseError)) throw error
return { ids: null, mapperIds, parseError: error.message }
}
}
/**
* Extract operation options from the subBlock with id: 'operation' (if present).
* Preserves labels, operation identifiers and optional groups from the operation selector.
* Identifiers construct tool IDs as `{blockType}_{id}`.
* Parses the subBlocks array using brace/bracket counting to safely traverse
* the nested structure without eval or a full AST parser.
*/
function extractOperationsFromContent(blockContent: string): BlockOperation[] {
const subBlocksMatch = /subBlocks\s*:\s*\[/.exec(blockContent)
if (!subBlocksMatch) return []
const arrayStart = subBlocksMatch.index + subBlocksMatch[0].length - 1
const arrayEnd = findMatchingClose(blockContent, arrayStart, '[', ']')
if (arrayEnd === -1) return []
const subBlocksContent = blockContent.substring(arrayStart + 1, arrayEnd - 1)
// Iterate over top-level objects in the subBlocks array, looking for id: 'operation'
let i = 0
while (i < subBlocksContent.length) {
if (subBlocksContent[i] === '{') {
const j = findMatchingClose(subBlocksContent, i)
if (j === -1) break
const objContent = subBlocksContent.substring(i, j)
if (/\bid\s*:\s*['"]operation['"]/.test(objContent)) {
const optionsMatch = /options\s*:\s*\[/.exec(objContent)
if (!optionsMatch) return []
const optArrayStart = optionsMatch.index + optionsMatch[0].length - 1
const optArrayEnd = findMatchingClose(objContent, optArrayStart, '[', ']')
if (optArrayEnd === -1) return []
const optionsContent = objContent.substring(optArrayStart + 1, optArrayEnd - 1)
const pairs: BlockOperation[] = []
const optionObjectRegex = /\{[^{}]*\}/g
let m
while ((m = optionObjectRegex.exec(optionsContent)) !== null) {
const optObj = m[0]
const labelMatch = /label\s*:\s*['"]([^'"]+)['"]/.exec(optObj)
const idMatch = /\bid\s*:\s*['"]([^'"]+)['"]/.exec(optObj)
if (labelMatch) {
const group = extractStringPropertyFromContent(optObj, 'group')
pairs.push({
label: labelMatch[1],
id: idMatch ? idMatch[1] : '',
...(group ? { group } : {}),
})
}
}
return pairs
}
i = j
} else {
i++
}
}
return []
}
/**
* Extract a mapping from operation id → tool id by scanning switch/case/return
* patterns in a block file. Handles both simple returns and ternary returns
* (for ternaries, takes the last quoted tool-like string, which is typically
* the default/list variant). Also picks up named helper functions referenced
* from tools.config.tool (e.g. selectGmailToolId).
*/
function extractSwitchCaseToolMapping(fileContent: string): Map<string, string> {
const mapping = new Map<string, string>()
const caseRegex = /\bcase\s+['"]([^'"]+)['"]\s*:/g
let caseMatch: RegExpExecArray | null
while ((caseMatch = caseRegex.exec(fileContent)) !== null) {
const opId = caseMatch[1]
if (mapping.has(opId)) continue
const searchStart = caseMatch.index + caseMatch[0].length
const searchEnd = Math.min(searchStart + 300, fileContent.length)
const segment = fileContent.substring(searchStart, searchEnd)
const returnIdx = segment.search(/\breturn\b/)
if (returnIdx === -1) continue
const afterReturn = segment.substring(returnIdx + 'return'.length)
// Limit scope to before the next case/default to avoid capturing sibling cases
const nextCaseIdx = afterReturn.search(/\bcase\b|\bdefault\b/)
const returnScope = nextCaseIdx > 0 ? afterReturn.substring(0, nextCaseIdx) : afterReturn
const toolMatches = [...returnScope.matchAll(/['"]([a-z][a-z0-9_]+)['"]/g)]
// Take the last tool-like string (underscore = tool ID pattern); for ternaries this
// is the fallback/list variant
const toolId = toolMatches
.map((m) => m[1])
.filter((id) => id.includes('_'))
.pop()
if (toolId) {
mapping.set(opId, toolId)
}
}
return mapping
}
/**
* Scan all tool files under apps/sim/tools/ and build a map from tool ID to description.
* Used to enrich operation entries with descriptions.
*/
interface ToolMaps {
desc: Map<string, string>
name: Map<string, string>
}
async function buildToolDescriptionMap(): Promise<ToolMaps> {
const toolsDir = path.join(rootDir, 'apps/sim/tools')
const desc = new Map<string, string>()
const name = new Map<string, string>()
try {
const toolFiles = await sourceGlob(`${toolsDir}/**/*.ts`)
for (const file of toolFiles) {
const basename = path.basename(file)
if (basename === 'index.ts' || basename === 'types.ts' || basename.includes('.test.'))
continue
const content = readSourceFile(file)
// Find every `id: 'tool_id'` occurrence in the file. For each, search
// the next ~600 characters for `name:` and `description:` fields, cutting
// off at the first `params:` block within that window. This handles both
// the simple inline pattern (id → description → params in one object) and
// the two-step pattern (base object holds params, ToolConfig export holds
// id + description after the base object).
const idRegex = /\bid\s*:\s*['"]([^'"]+)['"]/g
let idMatch: RegExpExecArray | null
while ((idMatch = idRegex.exec(content)) !== null) {
const toolId = idMatch[1]
if (desc.has(toolId)) continue
const windowStart = idMatch.index
const windowEnd = Math.min(windowStart + 600, content.length)
const window = content.substring(windowStart, windowEnd)
// Stop before any params block so we don't pick up param-level values
const paramsOffset = window.search(/\bparams\s*:\s*\{/)
const searchWindow = paramsOffset > 0 ? window.substring(0, paramsOffset) : window
// Match against the actual opening quote so apostrophes inside a
// double-quoted description (e.g. "Find someone's email") are preserved
// rather than being treated as the closing quote and truncating the value.
const descMatch = searchWindow.match(
/\bdescription\s*:\s*(?:'([^']{5,})'|"([^"]{5,})"|`([^`]{5,})`)/
)
const nameMatch = searchWindow.match(/\bname\s*:\s*(?:'([^']+)'|"([^"]+)"|`([^`]+)`)/)
if (descMatch) desc.set(toolId, descMatch[1] ?? descMatch[2] ?? descMatch[3] ?? '')
if (nameMatch) name.set(toolId, nameMatch[1] ?? nameMatch[2] ?? nameMatch[3] ?? '')
}
}
} catch {
// Non-fatal: descriptions will be empty strings
}
return { desc, name }
}
/**
* Detect the authentication type from block content.
* Returns 'oauth' if the block uses oauth-input credentials,
* 'api-key' if it uses a plain API key field, or 'none' otherwise.
*/
function extractAuthType(blockContent: string): 'oauth' | 'api-key' | 'none' {
if (/authMode\s*:\s*AuthMode\.OAuth\b/.test(blockContent)) return 'oauth'
if (/authMode\s*:\s*AuthMode\.(?:ApiKey|BotToken)\b/.test(blockContent)) return 'api-key'
// Fall back to credential subBlock heuristics for blocks without authMode.
if (/type\s*:\s*['"]oauth-input['"]/.test(blockContent)) return 'oauth'
if (/\bid\s*:\s*['"](?:apiKey|api_key|accessToken)['"]/.test(blockContent)) return 'api-key'
return 'none'
}
/**
* The catalog and every generated mapping are sorted with an explicit `en-US` collation.
* `localeCompare` with no locale uses the runtime default, which varies with `LANG` and the
* ICU build: `tr-TR`, `lt-LT`, `cs-CZ` and `et-EE` each reorder the real integration names, so
* a contributor on one of those locales would regenerate a different artifact and fail CI with
* no obvious cause.
*/
function compareCatalogNames(a: string, b: string): number {
return a.localeCompare(b, 'en-US')
}
/**
* Characters after which a `/` begins a regex literal rather than a division.
*
* `'\n'` is deliberate and load-bearing: a line-leading `/` is treated as opening a regex.
* A newline is recorded as the previous significant character rather than skipped, so this
* entry — not the `(` before it — is what decides a wrapped `value.match(` newline `/re/`.
* Prettier and Biome both emit a binary `/` at end-of-line, never at the start of the next
* one, so in this repo's formatted sources every line-leading `/` really is a regex, without
* exception — `blocks/table.ts` and `blocks/table_v2.ts` both wrap a `.match(` argument this
* way, and removing `'\n'` makes them lex as division and silently mis-scan. It is a
* deliberate trade: a hand-wrapped `b` newline `/ c / d` would be blanked as a regex body,
* which no formatted file in this repo produces.
*/
const REGEX_ALLOWED_AFTER = new Set([
'(',
',',
'=',
':',
'[',
'!',
'&',
'|',
'?',
'{',
'}',
';',
'+',
'-',
'*',
'/',
'%',
'~',
'^',
'<',
'>',
'\n',
])
/**
* Keywords after which a `/` begins a regex literal rather than a division. In every one of
* these positions an operand is expected, so `return /x/`, `typeof /x/` or `case /x/` opens a
* regex — a check on the previous character alone reads the keyword's last letter as an
* identifier and lexes the `/` as division.
*/
const REGEX_START_KEYWORDS = new Set([
'return',
'typeof',
'case',
'in',
'of',
'new',
'delete',
'void',
'instanceof',
'do',
'else',
'yield',
'await',
])
/**
* Whether the word immediately before `index` is a {@link REGEX_START_KEYWORDS} keyword.
* A property access (`counts.in / 2`) is excluded, since there the word is an identifier
* and the `/` really is a division.
*/
function precededByRegexStartKeyword(content: string, index: number): boolean {
let j = index - 1
while (j >= 0 && /\s/.test(content[j])) j--
const wordEnd = j + 1
while (j >= 0 && /[A-Za-z0-9_$]/.test(content[j])) j--
if (!REGEX_START_KEYWORDS.has(content.slice(j + 1, wordEnd))) return false
return content[j] !== '.' && content[j] !== '#'
}
/** Index just past a `//` comment opening at `start`. */
function scanLineComment(content: string, start: number): number {
const newline = content.indexOf('\n', start)
return newline === -1 ? content.length : newline
}
/** Index just past the block comment opening at `start`, or null when it never closes. */
function scanBlockComment(content: string, start: number): number | null {
const close = content.indexOf('*/', start + 2)
return close === -1 ? null : close + 2
}
/** Index just past a regex literal (and its flags) opening at `start`, or null when unterminated. */
function scanRegexLiteral(content: string, start: number): number | null {
let j = start + 1
let inClass = false
let closed = false
while (j < content.length) {
const c = content[j]
if (c === '\\') {
j += 2
continue
}
if (c === '\n') break
if (c === '[') inClass = true
else if (c === ']') inClass = false
else if (c === '/' && !inClass) {
closed = true
break
}
j++
}
if (!closed) return null
j++
while (j < content.length && /[a-z]/.test(content[j])) j++
return j
}
/** Index OF the closing quote of the string opening at `start`, or null when unterminated. */
function scanQuoted(content: string, start: number): number | null {
const quote = content[start]
let j = start + 1
while (j < content.length) {
if (content[j] === '\\') {
j += 2
continue
}
if (content[j] === '\n') break
if (content[j] === quote) return j
j++
}
return null
}
/** Index OF the closing backtick of the template literal opening at `start`, or null. */
function scanTemplateLiteral(content: string, start: number): number | null {
let j = start + 1
while (j < content.length) {
const c = content[j]
if (c === '\\') {
j += 2
continue
}
if (c === '`') return j
if (c === '$' && content[j + 1] === '{') {
const end = scanTemplateExpression(content, j + 2)
if (end === null) return null
j = end
continue
}
j++
}
return null
}
/**
* Index just past the `}` that closes the `${` expression starting at `start`, or null.
*
* The expression is lexed with the same primitives as top-level code, so a brace inside a
* nested string, comment, regex or template never counts toward the depth — a plain counter
* loses the closing backtick of `` `${ f("}") }` `` and reports the whole block unreadable.
*/
function scanTemplateExpression(content: string, start: number): number | null {
let j = start
let depth = 0
let prevSignificant = ''
while (j < content.length) {
const c = content[j]
if (c === '/' && content[j + 1] === '/') {
j = scanLineComment(content, j)
continue
}
if (c === '/' && content[j + 1] === '*') {
const end = scanBlockComment(content, j)
if (end === null) return null
j = end
continue
}
if (c === '/' && startsRegexLiteral(content, j, prevSignificant)) {
const end = scanRegexLiteral(content, j)
if (end === null) return null
prevSignificant = ')'
j = end
continue
}
if (c === "'" || c === '"') {
const close = scanQuoted(content, j)
if (close === null) return null
prevSignificant = c
j = close + 1
continue
}
if (c === '`') {
const close = scanTemplateLiteral(content, j)
if (close === null) return null
prevSignificant = '`'
j = close + 1
continue
}
if (c === '{') depth++
else if (c === '}') {
if (depth === 0) return j + 1
depth--
}
if (!/\s/.test(c)) prevSignificant = c
else if (c === '\n') prevSignificant = '\n'
j++
}
return null
}
/**
* Whether the word immediately before `index` ends in a postfix `++` or `--`.
* {@link REGEX_ALLOWED_AFTER} holds `'+'` and `'-'` for the binary operators, but a postfix
* increment produces a value, so the `/` in `i++ / a` is a division. Only the two-character
* form is matched — a single `+`/`-` stays an operator position.
*/
function precededByPostfixUpdate(content: string, index: number): boolean {
let j = index - 1
while (j >= 0 && /\s/.test(content[j])) j--
const c = content[j]
return (c === '+' || c === '-') && content[j - 1] === c
}
/** Whether the `/` at `index` opens a regex literal rather than a division. */
function startsRegexLiteral(content: string, index: number, prevSignificant: string): boolean {
if (
(prevSignificant === '+' || prevSignificant === '-') &&
precededByPostfixUpdate(content, index)
)
return false
return (
prevSignificant === '' ||
REGEX_ALLOWED_AFTER.has(prevSignificant) ||
precededByRegexStartKeyword(content, index)
)
}
/**
* Blank out string literals, template literals, comments and regex literals so a structural
* scan sees only code punctuation. Length and newlines are preserved, which the `readLiteral`
* index-mapping call sites depend on.
*
* Quoted strings keep their delimiters so callers can still see where one began; comments and
* regex literals are blanked whole, because their final character is arbitrary source text —
* commented-out code ending in `[`, or a character class like `/[}]/`, otherwise leaves an
* unbalanced bracket that derails every downstream scan.
*
* Returns `null` when the scan ends inside an unterminated construct, which means the input
* was not what we assumed and no structural conclusion drawn from it can be trusted.
*/
function blankStringsAndComments(content: string): string | null {
const out = content.split('')
const blank = (start: number, end: number) => {
for (let k = start; k < end && k < out.length; k++) if (out[k] !== '\n') out[k] = ' '
}
let i = 0
let prevSignificant = ''
while (i < content.length) {
const char = content[i]
if (char === '/' && content[i + 1] === '/') {
const end = scanLineComment(content, i)
blank(i, end)
i = end
continue
}
if (char === '/' && content[i + 1] === '*') {
const end = scanBlockComment(content, i)
if (end === null) return null
blank(i, end)
i = end
continue
}
if (char === '/' && startsRegexLiteral(content, i, prevSignificant)) {
const end = scanRegexLiteral(content, i)
if (end === null) return null
blank(i, end)
prevSignificant = ')'
i = end
continue
}
if (char === "'" || char === '"') {
const close = scanQuoted(content, i)
if (close === null) return null
blank(i + 1, close)
prevSignificant = char
i = close + 1
continue
}
if (char === '`') {
const close = scanTemplateLiteral(content, i)
if (close === null) return null
blank(i + 1, close)
prevSignificant = '`'
i = close + 1
continue
}
if (!/\s/.test(char)) prevSignificant = char
else if (char === '\n') prevSignificant = '\n'
i++
}
return out.join('')
}
/**
* Extract the source of the block's `oauth-input` credential subBlock object
* literal, so its fields are never confused with same-named fields on other
* subBlocks (e.g. a file selector's `serviceId`). Brace matching runs on a
* blanked copy of the content so string literals and comments containing
* braces cannot skew it.
*/
function extractOAuthSubBlock(blockContent: string): string | undefined {
const typeMatch = /type\s*:\s*['"]oauth-input['"]/.exec(blockContent)
if (!typeMatch) return undefined
const scannable = blankStringsAndComments(blockContent)
if (scannable === null) return undefined
let depth = 0
let objectStart = -1
for (let i = typeMatch.index; i >= 0; i--) {
const char = scannable[i]
if (char === '}') depth++
else if (char === '{') {
if (depth === 0) {
objectStart = i
break
}
depth--
}
}
if (objectStart === -1) return undefined
const objectEnd = findMatchingClose(scannable, objectStart)
if (objectEnd === -1) return undefined
return blockContent.substring(objectStart, objectEnd)
}
/** Extract the OAuth service id from the block's `oauth-input` credential subBlock. */
function extractOAuthServiceId(blockContent: string): string | undefined {
const subBlock = extractOAuthSubBlock(blockContent)
return subBlock ? /serviceId\s*:\s*['"]([^'"]+)['"]/.exec(subBlock)?.[1] : undefined
}
/**
* Whether the credential depends on the block's `environmentUrl`, which is what
* switches a Dataverse credential to the environment-bound grant (see
* `resolveMicrosoftDataverseCredentialPolicy`).
*/
function isEnvironmentBoundCredential(blockContent: string): boolean {
const subBlock = extractOAuthSubBlock(blockContent)
return subBlock ? /dependsOn\s*:\s*\[[^\]]*['"]environmentUrl['"]/.test(subBlock) : false
}
/**
* Extract the list of trigger IDs from the block's `triggers.available` array.
* Handles blocks that declare `triggers: { enabled: true, available: [...] }`.
*/
function extractTriggersAvailable(blockContent: string, fileContent?: string): string[] {
const triggersMatch = /\btriggers\s*:\s*\{/.exec(blockContent)
if (!triggersMatch) return []
const start = triggersMatch.index + triggersMatch[0].length - 1
const trigEnd = findMatchingClose(blockContent, start)
if (trigEnd === -1) return []
const triggersContent = blockContent.substring(start, trigEnd)
if (!/enabled\s*:\s*true/.test(triggersContent)) return []
const availableMatch = /available\s*:\s*\[/.exec(triggersContent)
if (!availableMatch) return []
const arrayStart = availableMatch.index + availableMatch[0].length - 1
const arrayEnd = findMatchingClose(triggersContent, arrayStart, '[', ']')
if (arrayEnd === -1) return []
const arrayContent = triggersContent.substring(arrayStart + 1, arrayEnd - 1)
// Blocks like emailbison declare `available: [...LOCAL_TRIGGER_IDS]`;
// resolve same-file const spreads to their literal entries so those
// triggers are not silently dropped from the generated data.
let resolvedContent = arrayContent
const constSource = fileContent ?? blockContent
const spreadRegex = /\.\.\.(\w+)/g
let spreadMatch: RegExpExecArray | null
while ((spreadMatch = spreadRegex.exec(arrayContent)) !== null) {
const constMatch = new RegExp(`const\\s+${spreadMatch[1]}\\s*=\\s*\\[`).exec(constSource)
if (!constMatch) continue
const constStart = constMatch.index + constMatch[0].length - 1
const constEnd = findMatchingClose(constSource, constStart, '[', ']')
if (constEnd === -1) continue
resolvedContent += constSource.substring(constStart + 1, constEnd - 1)
}
const ids: string[] = []
const idRegex = /['"]([^'"]+)['"]/g
let m
while ((m = idRegex.exec(resolvedContent)) !== null) {
ids.push(m[1])
}
return ids
}
/**
* Scan all trigger definition files and build a registry mapping trigger IDs
* to their human-readable name and description.
*/
async function buildTriggerRegistry(): Promise<Map<string, TriggerInfo>> {
const registry = new Map<string, TriggerInfo>()
const SKIP = new Set(['index.ts', 'registry.ts', 'types.ts', 'constants.ts', 'utils.ts'])
const triggerFiles = (await sourceGlob(`${TRIGGERS_PATH}/**/*.ts`)).filter(
(f) => !SKIP.has(path.basename(f)) && !f.includes('.test.')
)
for (const file of triggerFiles) {
try {
const content = readSourceFile(file)
// A file may export multiple TriggerConfig objects (e.g. v1 + v2 in
// the same file). Extract all exported configs by splitting on the
// export boundaries and parsing each one independently.
const exportRegex = /export\s+const\s+\w+\s*:\s*TriggerConfig\s*=\s*\{/g
let exportMatch
const exportStarts: number[] = []
while ((exportMatch = exportRegex.exec(content)) !== null) {
exportStarts.push(exportMatch.index)
}
// If no typed exports found, fall back to simple regex on whole file
const segments =
exportStarts.length > 0
? exportStarts.map((start, i) => content.substring(start, exportStarts[i + 1]))
: [content]
const evaluatedTriggers = await loadTriggerRegistry()
for (const segment of segments) {
const idMatch = /\bid\s*:\s*['"]([^'"]+)['"]/.exec(segment)
const nameMatch = /\bname\s*:\s*['"]([^'"]+)['"]/.exec(segment)
// Deprecated triggers stay registered for existing workflows but are
// excluded from generated documentation.
if (/\bdeprecated\s*:\s*true/.test(segment)) continue
if (idMatch && nameMatch) {
const trigger = evaluatedTriggers[idMatch[1]]
registry.set(idMatch[1], {
id: idMatch[1],
name: trigger?.name ?? nameMatch[1],
description:
trigger?.description ??
extractStringPropertyFromContent(segment, 'description', true) ??
'',
})
}
}
const factoryExportRegex =
/export\s+const\s+\w+(?:\s*:\s*TriggerConfig)?\s*=\s*\w+\s*\(\s*['"]([^'"]+)['"]/g
let factoryExportMatch: RegExpExecArray | null
while ((factoryExportMatch = factoryExportRegex.exec(content)) !== null) {
const trigger = evaluatedTriggers[factoryExportMatch[1]]
if (!trigger?.id || !trigger.name || trigger.deprecated || registry.has(trigger.id)) {
continue
}
registry.set(trigger.id, {
id: trigger.id,
name: trigger.name,
description: trigger.description ?? '',
})
}
} catch {
// skip unreadable files silently
}
}
console.log(`✓ Loaded ${registry.size} trigger definitions`)
return registry
}
/**
* Write the icon mapping TypeScript file for the shared integrations data
* directory (`apps/sim/lib/integrations`). Mirrors `writeIconMapping` (the
* docs-app variant) but targets the sim app so it imports from
* `@/components/icons`.
*/
function writeIntegrationsIconMapping(rawIconMapping: Record<string, IconRef>): void {
try {
const iconMapping = withBareNameAliases(rawIconMapping)
if (!fs.existsSync(INTEGRATIONS_DATA_PATH)) {
fs.mkdirSync(INTEGRATIONS_DATA_PATH, { recursive: true })
}
const iconMappingPath = path.join(INTEGRATIONS_DATA_PATH, 'icon-mapping.ts')
const imports = renderIconImports(Object.values(iconMapping))
const mappingEntries = Object.entries(iconMapping)
.sort(([a], [b]) => compareCatalogNames(a, b))
.map(([blockType, iconRef]) => ` ${formatIconMapKey(blockType)}: ${iconRef.name},`)
.join('\n')
const content = `// Auto-generated file - do not edit manually
// Generated by scripts/generate-docs.ts
// Maps block types to their icon component references for the integrations page
import type { ComponentType, SVGProps } from 'react'
${imports}
type IconComponent = ComponentType<SVGProps<SVGSVGElement>>
export const blockTypeToIconMap: Record<string, IconComponent> = {
${mappingEntries}
}
`
emitGeneratedFile(iconMappingPath, content)
if (!CHECK_ONLY) console.log('✓ Integration icon mapping written')
} catch (error) {
console.error('Error writing integration icon mapping:', error)
}
}
/**
* Collect all integration entries from block definitions and write integrations.json
* to the shared integrations data directory (`apps/sim/lib/integrations`).
* Applies the same visibility filters as the docs generation pipeline.
*/
async function writeIntegrationsJson(iconMapping: Record<string, IconRef>): Promise<void> {
try {
if (!fs.existsSync(INTEGRATIONS_DATA_PATH)) {
fs.mkdirSync(INTEGRATIONS_DATA_PATH, { recursive: true })
}
const triggerRegistry = await buildTriggerRegistry()
const { desc: toolDescMap, name: toolNameMap } = await buildToolDescriptionMap()
const toolMetadataById = await loadToolMetadata()
// Hand-authored, integration-specific landing content (install walkthrough,
// privacy blurb), keyed by slug. Imported as pure data — its only import is
// type-only and erased at runtime — and baked into the entries below so the
// landing page reads a single source instead of augmenting at render time.
const landingContentModule = await import(
pathToFileURL(path.join(LANDING_INTEGRATIONS_DATA_PATH, 'landing-content.ts')).href
)
const landingContentMap = (landingContentModule.INTEGRATION_LANDING_CONTENT ?? {}) as Record<
string,
Record<string, unknown>
>
const integrations: IntegrationEntry[] = []
const seenBaseTypes = new Set<string>()
const blockFiles = (await sourceGlob(`${BLOCKS_PATH}/*.ts`)).sort()
for (const blockFile of blockFiles) {
const fileContent = readSourceFile(blockFile)
const switchCaseMap = extractSwitchCaseToolMapping(fileContent)
const configs = blockConfigsForFile(blockFile)
for (const config of configs) {
const blockType = config.type
// Canonical integrations filter: only third-party tool blocks visible in the toolbar.
// `isIntegrationBlock` is the single source of truth for "is integration".
if (!isIntegrationBlock(config)) continue
// Every tools-category block MUST declare an `integrationType` from the canonical
// 16-value enum (apps/sim/blocks/types.ts). Fail loudly so the catalog never
// ships a tool without a category bucket.
if (!config.integrationType) {
throw new Error(
`Block "${blockType}" has \`category: 'tools'\` but is missing required \`integrationType\`. ` +
`Add one of the IntegrationType values from apps/sim/blocks/types.ts.`
)
}
if (!INTEGRATION_CATEGORY_VALUES.has(config.integrationType as IntegrationType)) {
throw new Error(
`Block "${blockType}" has unrecognised \`integrationType: "${config.integrationType}"\`. ` +
`Use one of: ${[...INTEGRATION_CATEGORY_VALUES].join(', ')}.`
)
}
const integrationType = config.integrationType as IntegrationType
const baseType = stripVersionSuffix(blockType)
if (seenBaseTypes.has(baseType)) continue
seenBaseTypes.add(baseType)
const iconName = (config as any).iconName || iconMapping[blockType]?.name || ''
const rawOps: { label: string; id: string }[] = (config as any).operations || []
// Enrich each operation with a description from the tool registry.
// Lookup order:
// 1. Derive toolId as `{baseType}_{operationId}` and check directly.
// 2. Check switch/case mapping parsed from tools.config.tool (handles
// cases where op IDs differ from tool IDs, e.g. get_carts → list_carts,
// or send_gmail → gmail_send).
// 3. Find the tool in tools.access whose name exactly matches the label.
const toolsAccess: string[] = (config as any).tools?.access || []
const operations: OperationInfo[] = rawOps.map(({ label, id }) => {
const toolId = `${baseType}_${id}`
let opDesc = toolDescMap.get(toolId) || toolDescMap.get(id) || ''
if (!opDesc) {
const switchMappedId = switchCaseMap.get(id)
if (switchMappedId) {
opDesc = toolDescMap.get(switchMappedId) || ''
if (!opDesc) {
for (const tId of toolsAccess) {
if (tId === switchMappedId || tId.startsWith(`${switchMappedId}_v`)) {
opDesc = toolDescMap.get(tId) || ''
if (opDesc) break
}
}
}
}
}
if (!opDesc && toolsAccess.length > 0) {
for (const tId of toolsAccess) {
if (
[label, `${config.name} ${label}`].some(
(name) => toolNameMap.get(tId)?.toLowerCase() === name.toLowerCase()
)
) {
opDesc = toolDescMap.get(tId) || ''
if (opDesc) break
}
}
}
return { name: label, description: opDesc }
})
// A block with no operation dropdown (single-tool blocks, provider pickers)
// exposes its tools directly, so list them from the client-safe tool metadata.
if (operations.length === 0) {
for (const toolId of toolsAccess) {
const tool = toolMetadataById[toolId]
if (!tool?.name) continue
operations.push({ name: tool.name, description: tool.description ?? '' })
}
}
const triggerIds: string[] = (config as any).triggerIds || []
const triggers: TriggerInfo[] = triggerIds
.map((id) => triggerRegistry.get(id))
.filter((t): t is TriggerInfo => t !== undefined)
const docsUrl = (config as any).docsLink || defaultIntegrationDocsUrl(baseType)
const slug = config.name
.toLowerCase()
.replace(/[^a-z0-9]+/g, '-')
.replace(/^-|-$/g, '')
const authType = extractAuthType(fileContent)
const oauthServiceId = authType === 'oauth' ? extractOAuthServiceId(fileContent) : undefined
// OAuth integrations resolve their connect UI through the service id
// (see `resolveOAuthServiceForIntegration`), so fail loudly rather than
// shipping a catalog entry that silently falls back to the API-key path.
if (authType === 'oauth' && !oauthServiceId) {
throw new Error(
`Block "${blockType}" is an OAuth integration but no \`serviceId\` could be ` +
`extracted from its \`oauth-input\` subBlock.`
)
}
integrations.push({
type: blockType,
slug,
name: config.name,
description: config.description,
longDescription: config.longDescription || '',
bgColor: config.bgColor || '#6B7280',
iconName,
docsUrl,
operations,
operationCount: operations.length,
triggers,
triggerCount: triggers.length,
authType,
...(oauthServiceId ? { oauthServiceId } : {}),
category: 'tools',
integrationType,
...(config.tags ? { tags: config.tags } : {}),
...(landingContentMap[slug] ? { landingContent: landingContentMap[slug] } : {}),
})
}
}
integrations.sort((a, b) => compareCatalogNames(a.name, b.name))
const metadataPath = path.join(INTEGRATIONS_CATALOG_PATH, 'integration-metadata.ts')
const metadata = integrations.map(
({ type, slug, name, authType, oauthServiceId, bgColor, integrationType }) => ({
type,
slug,
name,
authType,
...(oauthServiceId ? { oauthServiceId } : {}),
bgColor,
integrationType,
})
)
emitGeneratedFile(
metadataPath,
formatGeneratedSource(
`/**
* Generated by scripts/generate-docs.ts from the public integration catalog.
* Identity and authentication metadata without descriptions, operations, or icons.
*/
export interface IntegrationMetadata {
type: string
slug: string
name: string
authType: 'oauth' | 'api-key' | 'none'
oauthServiceId?: string
bgColor: string
integrationType: string
}
export const INTEGRATION_METADATA: readonly IntegrationMetadata[] = JSON.parse(${JSON.stringify(JSON.stringify(metadata))})
`,
metadataPath,
rootDir
)
)
const totalsPath = path.join(INTEGRATIONS_CATALOG_PATH, 'integration-totals.ts')
const totals = {
integrations: integrations.length,
actions: integrations.reduce((sum, integration) => sum + integration.operationCount, 0),
triggers: integrations.reduce((sum, integration) => sum + integration.triggerCount, 0),
integrationsWithTriggers: integrations.filter((integration) => integration.triggerCount > 0)
.length,
}
emitGeneratedFile(
totalsPath,
formatGeneratedSource(
`/**
* Generated by scripts/generate-docs.ts from the public integration catalog.
* Catalog-wide counts that every public integration claim is derived from, kept
* apart from the catalog itself so marketing surfaces import the counts, not the list.
*/
export const INTEGRATION_TOTALS = ${JSON.stringify(totals)} as const
`,
totalsPath,
rootDir
)
)
const jsonPath = path.join(INTEGRATIONS_CATALOG_PATH, 'integrations.json')
// `JSON.stringify` always expands every array across multiple lines, but Biome's
// JSON formatter inlines short arrays of primitive strings. Pre-collapse those
// arrays here so the emitted file is already in Biome's canonical shape and
// `bun run check` does not churn it on every commit.
const serialize = (value: unknown) =>
JSON.stringify(value, null, 2).replace(
/\[\n(\s+"[^"\n]*"(?:,\n\s+"[^"\n]*")*)\n\s+\]/g,
(_match, inner) => {
const items = (inner as string).split(',\n').map((s: string) => s.trim())
return `[${items.join(', ')}]`
}
)
// Each entry keeps its own `updatedAt` until that entry's content changes, so one
// integration's edit never re-dates every other page's sitemap/JSON-LD freshness.
// The catalog-wide `updatedAt` is re-stamped on any real change (it dates the hub).
const previous = fs.existsSync(jsonPath)
? (JSON.parse(fs.readFileSync(jsonPath, 'utf-8')) as {
integrations?: Array<Record<string, unknown>>
})
: null
const today = new Date().toISOString().slice(0, 10)
const previousBySlug = new Map(
(previous?.integrations ?? []).map((entry) => [entry.slug, entry] as const)
)
const datedIntegrations = integrations.map((integration) => {
const prior = previousBySlug.get(integration.slug)
if (!prior) return { ...integration, updatedAt: today }
const { updatedAt: priorUpdatedAt, ...priorContent } = prior
const unchanged =
typeof priorUpdatedAt === 'string' && serialize(priorContent) === serialize(integration)
return { ...integration, updatedAt: unchanged ? priorUpdatedAt : today }
})
if (
previous?.integrations &&
serialize(previous.integrations) === serialize(datedIntegrations)
) {
console.log(`✓ Integration data unchanged: ${integrations.length} integrations → ${jsonPath}`)
return
}
if (CHECK_ONLY) {
staleArtifacts.push(path.relative(rootDir, jsonPath))
return
}
fs.writeFileSync(
jsonPath,
`${serialize({ updatedAt: today, integrations: datedIntegrations })}\n`
)
console.log(`✓ Integration data written: ${integrations.length} integrations → ${jsonPath}`)
} catch (error) {
// Surface taxonomy violations (missing/invalid `integrationType`) loudly —
// they are programmer errors that must fail the generator, not be logged
// and silently swallowed.
console.error('Error writing integrations JSON:', error)
throw error
}
}
/**
* Extract ALL block configs from a file, filtering out hidden blocks
*/
export function extractAllBlockConfigs(fileContent: string): BlockConfig[] {
const configs: BlockConfig[] = []
// First, extract the primary icon from the file (for V2 blocks that inherit via spread)
const primaryIcon = extractIconNameFromContent(fileContent)
for (const declaration of blockDeclarations(fileContent)) {
const blockName = declaration.name.replace(/Block$/, '')
const startIndex = declaration.start
const endIndex = declaration.end
if (endIndex !== -1) {
const blockContent = fileContent.substring(startIndex, endIndex)
const hideFromToolbar = /hideFromToolbar\s*:\s*true/.test(stripSourceComments(blockContent))
if (hideFromToolbar) {
console.log(`Skipping ${blockName}Block - hideFromToolbar is true`)
continue
}
// Unreleased preview blocks stay out of every generated surface: docs
// .mdx pages, integrations.json (landing + workspace catalog + sitemap +
// OG images), and the icon mapping.
if (isPreviewSource(blockContent)) {
console.log(`Skipping ${blockName}Block - preview is true`)
continue
}
const config = extractBlockConfigFromContent(blockContent, blockName, fileContent)
if (config) {
// For V2 blocks that don't have an explicit icon, use the primary icon from the file
if (!config.iconName && primaryIcon) {
;(config as any).iconName = primaryIcon
}
configs.push(config)
}
}
}
return configs
}
/**
* Extract the name of the spread base block (e.g., "GitHubBlock" from "...GitHubBlock")
*/
function extractSpreadBase(blockContent: string): string | null {
const spreadMatch = blockContent.match(/^\s*\.\.\.(\w+Block)\s*,/m)
return spreadMatch ? spreadMatch[1] : null
}
/**
* Operations a block inherits by spreading a same-file block's fields array
* (`subBlocks: [...NotionBlock.subBlocks, ...]`) rather than the whole config.
* Only a plain spread counts: a transformed one (`...XBlock.subBlocks.map(...)`) may rewrite the
* operations, so it falls through to the block's own `tools.access`.
*/
function extractSpreadSubBlocksOperations(
blockContent: string,
fileContent: string | undefined
): { label: string; id: string }[] {
if (!fileContent) return []
const declarations = blockDeclarations(fileContent)
for (const match of blockContent.matchAll(/\.\.\.(\w+Block)\.subBlocks\s*[,\]]/g)) {
const declaration = declarations.find((candidate) => candidate.name === match[1])
const operations = declaration ? extractOperationsFromContent(declaration.content) : []
if (operations.length > 0) return operations
}
return []
}
/** Resolves a block's category through local spread ancestry without loading its registry. */
export function extractInheritedBlockCategory(
blockContent: string,
fileContent: string
): string | null {
const visited = new Set<string>()
let current = blockContent
while (true) {
const category = extractStringPropertyFromContent(current, 'category', true)
if (category) return category
const base = extractSpreadBase(current)
if (!base || visited.has(base)) return null
visited.add(base)
const declaration = blockDeclarations(fileContent).find((candidate) => candidate.name === base)
if (!declaration) return null
current = declaration.content
}
}
/**
* Extract block config from a specific block's content
* If the block uses spread inheritance (e.g., ...GitHubBlock), attempts to resolve
* missing properties from the base block in the file content.
*/
function extractBlockConfigFromContent(
blockContent: string,
blockName: string,
fileContent?: string
): BlockConfig | null {
try {
const spreadBase = extractSpreadBase(blockContent)
let baseConfig: BlockConfig | null = null
if (spreadBase && fileContent) {
const declaration = blockDeclarations(fileContent).find(
(candidate) => candidate.name === spreadBase
)
if (declaration) {
baseConfig = extractBlockConfigFromContent(
declaration.content,
spreadBase.replace(/Block$/, ''),
fileContent
)
}
}
// Extract properties from this block, using topLevelOnly=true for main properties
const blockType =
extractStringPropertyFromContent(blockContent, 'type', true) || blockName.toLowerCase()
const name =
extractStringPropertyFromContent(blockContent, 'name', true) ||
baseConfig?.name ||
`${blockName} Block`
const description =
extractStringPropertyFromContent(blockContent, 'description', true) ||
baseConfig?.description ||
''
const longDescription =
extractStringPropertyFromContent(blockContent, 'longDescription', true) ||
baseConfig?.longDescription ||
''
const category =
extractStringPropertyFromContent(blockContent, 'category', true) || baseConfig?.category || ''
const bgColor =
extractStringPropertyFromContent(blockContent, 'bgColor', true) ||
baseConfig?.bgColor ||
'#F5F5F5'
const iconName = extractIconNameFromContent(blockContent) || (baseConfig as any)?.iconName || ''
const ownOutputs = extractOutputsFromContent(blockContent)
// A spread-inheriting block with no outputs of its own (`{ ...XBlock, type }`) keeps the base's.
const inheritsOutputs =
/\boutputs\s*:\s*\{\s*\.\.\.\w+Block\.outputs\b/.test(blockContent) ||
(spreadBase !== null && !/\boutputs\s*:/.test(blockContent))
const omittedOutputs = extractOmittedOutputs(blockContent, baseConfig?.outputs)
const outputs = omittedOutputs
? { ...omittedOutputs, ...ownOutputs }
: inheritsOutputs
? { ...baseConfig?.outputs, ...ownOutputs }
: ownOutputs
const toolsAccess = extractToolsAccessFromContent(blockContent)
/** Versioned mapped spreads retain their base tools alongside explicit additions. */
let finalToolsAccess = toolsAccess
if (baseConfig?.tools?.access) {
// Check if there's a map operation on base tools
// Pattern: access: (SomeBlock.tools?.access || []).map((toolId) => `${toolId}_v2`)
const mapMatch = blockContent.match(
/access\s*:\s*(?:\[\s*\.\.\.)?\(\s*\w+Block\.tools\?\.access\s*\|\|\s*\[\]\s*\)\.map\s*\(\s*\(\s*\w+\s*\)\s*=>\s*`\$\{\s*\w+\s*\}_v(\d+)`\s*\)/
)
if (mapMatch) {
const versionSuffix = `_v${mapMatch[1]}`
finalToolsAccess = [
...baseConfig.tools.access.map((tool) => `${tool}${versionSuffix}`),
...toolsAccess,
]
}
const replacement = blockContent.match(
/access\s*:\s*\w+Block\.tools\.access\.map\s*\(\s*\(\s*(\w+)\s*\)\s*=>\s*\1\s*===\s*['"]([^'"]+)['"]\s*\?\s*['"]([^'"]+)['"]\s*:\s*\1\s*\)/
)
if (replacement) {
finalToolsAccess = baseConfig.tools.access.map((tool) =>
tool === replacement[2] ? replacement[3] : tool
)
}
}
const ownOperations = extractOperationsFromContent(blockContent)
const baseOperations = baseConfig?.operations ?? []
const operations =
ownOperations.length > 0
? ownOperations
: baseOperations.length > 0
? baseOperations
: extractSpreadSubBlocksOperations(blockContent, fileContent)
const triggerIds = extractTriggersAvailable(blockContent, fileContent)
const supplied = extractBlockSuppliedParamIds(blockContent, blockName, fileContent)
/**
* `null` on the base means the base's own scan failed, which is not the same as a block
* with no base at all (`undefined`) — the fields the base contributes are unknown, so
* anything inheriting them is unknown too.
*/
const baseParamIds: string[] | null | undefined = (baseConfig as any)?.userSettableParamIds
const baseSettableParamIds: string[] = baseParamIds ?? []
/**
* `null` means UNKNOWN and disables the hidden-param filter for this block, restoring the
* pre-filter behaviour of documenting every param. An unreadable `subBlocks` array must
* never be collapsed into `[]`, because `[]` asserts the block supplies nothing and strips
* every hidden param from the page, and it must never abort the run either — one block the
* scanner cannot read used to brick the generator for the entire repository.
*/
let userSettableParamIds: string[] | null
if (supplied.parseError !== null) {
/**
* A block that spreads a base still documents the base's fields, so the filter can stay
* on and lose at most the fields this block adds on top of the base — plus its mapper's
* renames, which are read even when the `subBlocks` scan fails. With no base to fall back
* on there is nothing to filter against, so the filter is switched off entirely.
*/
const fallback =
baseSettableParamIds.length > 0
? [...new Set([...baseSettableParamIds, ...supplied.mapperIds])]
: null
if (!subBlockParseWarnings.has(supplied.parseError)) {
subBlockParseWarnings.add(supplied.parseError)
console.warn(
`⚠ ${supplied.parseError}; ${
fallback
? "documenting the spread base's fields instead"
: 'documenting every param of its tools instead of filtering'
}`
)
}
userSettableParamIds = fallback
} else if (baseParamIds === null) {
userSettableParamIds = null
} else if (supplied.ids === null) {
/**
* With `parseError` null, the only remaining cause is a `subBlocks` array holding just
* spreads of fields arrays this scanner cannot follow — a source the scanner could not
* get through at all is reported as a `parseError` by `extractBlockSuppliedParamIds` and
* handled above. A config-level spread base still contributes its readable fields, so the
* filter stays on against those plus the mapper's renames; with no base there is nothing
* to filter against and the filter is switched off. No warning: the array itself parsed
* fine, and every field it names is documented through the spread source's own page.
*/
userSettableParamIds =
baseSettableParamIds.length > 0
? [...new Set([...baseSettableParamIds, ...supplied.mapperIds])]
: null
} else {
userSettableParamIds = [...new Set([...supplied.ids, ...baseSettableParamIds])]
}
const docsLink =
extractStringPropertyFromContent(blockContent, 'docsLink', true) ||
baseConfig?.docsLink ||
defaultIntegrationDocsUrl(blockType)
const integrationType =
extractEnumPropertyFromContent(blockContent, 'integrationType') ||
baseConfig?.integrationType ||
null
// Tags live on the block's `<BlockName>BlockMeta` export. For spread-inheriting
// blocks (e.g. `ConfluenceV2Block` extending `ConfluenceBlock`), also try the
// spread base's tags (its own meta, or its base's) so V2+ variants inherit them.
const tags =
(fileContent ? extractTagsFromBlockMeta(fileContent, blockName) : null) ||
baseConfig?.tags ||
null
return {
type: blockType,
name,
description,
longDescription,
category,
bgColor,
iconName,
outputs,
tools: {
access: finalToolsAccess.length > 0 ? finalToolsAccess : baseConfig?.tools?.access || [],
},
operations,
userSettableParamIds,
triggerIds: triggerIds.length > 0 ? triggerIds : (baseConfig as any)?.triggerIds || [],
docsLink,
...(integrationType ? { integrationType } : {}),
...(tags ? { tags } : {}),
}
} catch (error) {
console.error(`Error extracting block configuration for ${blockName}:`, error)
return null
}
}
/**
* The single predicate that decides whether an extracted block config belongs
* in the integration surfaces emitted by this script — the integrations
* catalog (`integrations.json`) and the per-tool `/tools/*.mdx` docs. A block
* qualifies only when it is a third-party integration (`category: 'tools'`)
* that is currently surfaced in the toolbar (`hideFromToolbar` not set). Under
* the versioning upgrade paradigm only the latest version is visible, so this
* also naturally selects the canonical version. Recategorizing a block to
* `'blocks'` or `'triggers'` removes it from all integration surfaces.
*/
function isIntegrationBlock(config: {
category?: string
hideFromToolbar?: boolean
preview?: boolean
}): boolean {
return config.category === 'tools' && !config.hideFromToolbar && !config.preview
}
/**
* Block types that never belong in the integrations icon map regardless of
* category — core primitives, triggers, and webhook/feed plumbing.
*/
const ICON_MAP_EXCLUDED_TYPES = new Set([
'evaluator',
'number',
'webhook',
'schedule',
'mcp',
'generic_webhook',
'rss',
])
/**
* Extract a string property from block content.
* For top-level properties like 'description', only looks in the portion before nested objects
* to avoid matching properties inside nested structures like outputs.
*/
function extractStringPropertyFromContent(
content: string,
propName: string,
topLevelOnly = false
): string | null {
let searchContent = content
// For top-level properties, only search before nested objects like outputs, tools, inputs, subBlocks
if (topLevelOnly) {
const nestedObjectPatterns = [
/\boutputs\s*:\s*\{/,
/\btools\s*:\s*\{/,
/\binputs\s*:\s*\{/,
/\bsubBlocks\s*:\s*\[/,
/\btriggers\s*:\s*\{/,
]
let cutoffIndex = content.length
for (const pattern of nestedObjectPatterns) {
const match = content.match(pattern)
if (match && match.index !== undefined && match.index < cutoffIndex) {
cutoffIndex = match.index
}
}
searchContent = content.substring(0, cutoffIndex)
}
const singleQuoteMatch = searchContent.match(new RegExp(`${propName}\\s*:\\s*'([^']*)'`, 'm'))
if (singleQuoteMatch) return singleQuoteMatch[1]
const doubleQuoteMatch = searchContent.match(new RegExp(`${propName}\\s*:\\s*"([^"]*)"`, 'm'))
if (doubleQuoteMatch) return doubleQuoteMatch[1]
const templateMatch = searchContent.match(new RegExp(`${propName}\\s*:\\s*\`([^\`]+)\``, 's'))
if (templateMatch) {
let templateContent = templateMatch[1]
templateContent = templateContent.replace(/\$\{[^}]+\}/g, '')
templateContent = templateContent.replace(/\s+/g, ' ').trim()
return templateContent
}
return null
}
/**
* Extract an enum property value from block content. Maps an `IntegrationType`
* enum key (e.g. `Communication`) to its slug value (e.g. `'communication'`).
* Mirrors `apps/sim/blocks/types.ts → IntegrationType` — keep in sync.
*/
function extractEnumPropertyFromContent(content: string, propName: string): string | null {
const match = content.match(new RegExp(`${propName}\\s*:\\s*IntegrationType\\.(\\w+)`))
if (!match) return null
const enumKey = match[1]
const ENUM_MAP: Record<string, string> = {
AI: 'ai',
Analytics: 'analytics',
Commerce: 'commerce',
Communication: 'communication',
Databases: 'databases',
DevOps: 'devops',
Documents: 'documents',
Email: 'email',
HR: 'hr',
Marketing: 'marketing',
Observability: 'observability',
Productivity: 'productivity',
Sales: 'sales',
Search: 'search',
Security: 'security',
Support: 'support',
}
return ENUM_MAP[enumKey] || enumKey.toLowerCase()
}
/**
* Extract a string array property from block content.
* Matches patterns like `tags: ['api', 'oauth', 'webhooks']`
*/
function extractArrayPropertyFromContent(content: string, propName: string): string[] | null {
const match = content.match(new RegExp(`${propName}\\s*:\\s*\\[([^\\]]+)\\]`))
if (!match) return null
const items = match[1].match(/'([^']+)'|"([^"]+)"/g)
if (!items) return null
return items.map((item) => item.replace(/['"]/g, ''))
}
/**
* Extract `tags` from a `<BlockName>BlockMeta` literal in the source file.
* Looks for `export const <BlockName>BlockMeta = { ... tags: [...] ... }`
* at file scope and scans only the body of that literal. Returns null when
* no matching meta export exists or it contains no `tags` array.
*
* During the in-progress migration to per-block meta, some blocks declare
* `tags` on `BlockConfig` and others on `*BlockMeta`. The caller should
* try this extractor first and fall back to the `BlockConfig` extractor.
*/
function extractTagsFromBlockMeta(fileContent: string, blockName: string): string[] | null {
const headerRegex = new RegExp(`export\\s+const\\s+${blockName}BlockMeta\\s*(?::[^=]+)?=\\s*\\{`)
const metaHeaderMatch = fileContent.match(headerRegex)
if (!metaHeaderMatch || metaHeaderMatch.index === undefined) return null
const openBracePos = fileContent.indexOf('{', metaHeaderMatch.index)
if (openBracePos === -1) return null
const closeBracePos = findMatchingClose(fileContent, openBracePos)
if (closeBracePos === -1) return null
const metaBody = fileContent.substring(openBracePos + 1, closeBracePos)
return extractArrayPropertyFromContent(metaBody, 'tags')
}
/**
* Extract the component identifier assigned to a block's `icon` property.
* Most block icons are named `<Service>Icon`, but some reference a generic emcn
* icon (e.g. `icon: Library`) or use a lowercase identifier (`icon: xIcon`), so
* this matches any identifier rather than requiring the `Icon` suffix. Bare JS
* literals are excluded so `icon: undefined` never resolves to a component.
*/
function extractIconNameFromContent(content: string): string | null {
const iconMatch = content.match(
/(?:^|[\s{,])icon\s*:\s*(?!(?:undefined|null|true|false)\b)([A-Za-z_$][\w$]*)/
)
return iconMatch ? iconMatch[1] : null
}
/** Module an icon component is imported from, when the block file declares it. */
const DEFAULT_ICON_SOURCE = '@/components/icons'
/**
* Resolve the module a block's icon identifier is imported from by scanning the
* block file's import statements. Falls back to `@/components/icons`, which is
* where the overwhelming majority of block icons live and which both the sim
* app and the docs app (via the copied `icons.tsx`) resolve.
*/
function resolveIconSource(fileContent: string, iconName: string): string {
const importRegex = /import\s*(?:type\s*)?\{([^}]*)\}\s*from\s*['"]([^'"]+)['"]/g
let match: RegExpExecArray | null
while ((match = importRegex.exec(fileContent)) !== null) {
const named = match[1].split(',').map((entry) =>
entry
.trim()
.split(/\s+as\s+/)[0]
.trim()
)
if (named.includes(iconName)) return match[2]
}
return DEFAULT_ICON_SOURCE
}
/** Biome's configured `formatter.lineWidth` for this repo. */
const BIOME_LINE_WIDTH = 100
/**
* Quote an icon-map key that is not a bare JS identifier. Trigger providers use
* hyphenated ids (`google-drive`, `microsoft-teams`) that would otherwise emit
* as a subtraction expression and break the generated module.
*/
function formatIconMapKey(key: string): string {
return /^[A-Za-z_$][\w$]*$/.test(key) ? key : `'${key}'`
}
/**
* Render grouped `import { ... } from '...'` statements for every icon
* referenced by a mapping, so icons sourced outside `@/components/icons`
* (e.g. `@sim/emcn/icons`) resolve in the generated file. Output is emitted
* pre-formatted — package specifiers before `@/` aliases, specifiers sorted,
* and collapsed to one line when it fits — because these files are written
* verbatim and never passed through Biome.
*/
function renderIconImports(iconRefs: IconRef[]): string {
const bySource = new Map<string, Set<string>>()
for (const ref of iconRefs) {
let names = bySource.get(ref.source)
if (!names) {
names = new Set()
bySource.set(ref.source, names)
}
names.add(ref.name)
}
const isAlias = (source: string) => source.startsWith('@/') || source.startsWith('.')
return [...bySource.entries()]
.sort(([a], [b]) => Number(isAlias(a)) - Number(isAlias(b)) || biomeSortCompare(a, b))
.map(([source, names]) => {
const sorted = [...names].sort(biomeSortCompare)
const singleLine = `import { ${sorted.join(', ')} } from '${source}'`
if (singleLine.length <= BIOME_LINE_WIDTH) return singleLine
const specifiers = sorted.map((name) => ` ${name},`).join('\n')
return `import {\n${specifiers}\n} from '${source}'`
})
.join('\n')
}
function extractOutputsFromContent(content: string): Record<string, any> {
const outputsStart = content.search(/outputs\s*:\s*{/)
if (outputsStart === -1) return {}
const openBracePos = content.indexOf('{', outputsStart)
if (openBracePos === -1) return {}
const pos = findMatchingClose(content, openBracePos)
if (pos === -1) return {}
const outputsContent = content.substring(openBracePos + 1, pos - 1).trim()
const outputs: Record<string, any> = {}
const fieldRegex = /(\w+)\s*:\s*{/g
let match
const fieldPositions: Array<{ name: string; start: number }> = []
while ((match = fieldRegex.exec(outputsContent)) !== null) {
fieldPositions.push({
name: match[1],
start: match.index + match[0].length - 1,
})
}
fieldPositions.forEach((field) => {
const endPos = findMatchingClose(outputsContent, field.start)
if (endPos !== -1) {
const fieldContent = outputsContent.substring(field.start + 1, endPos - 1).trim()
const typeMatch = fieldContent.match(/type\s*:\s*['"](.*?)['"]/)
const description = extractDescription(fieldContent)
if (typeMatch) {
outputs[field.name] = {
type: typeMatch[1],
description: description || `${field.name} output from the block`,
}
}
}
})
return outputs
}
/** Resolves a version's explicit removal of inherited output fields. */
function extractOmittedOutputs<T>(
content: string,
inherited: Record<string, T> | undefined
): Record<string, T> | null {
const omission = content.match(
/\boutputs\s*:\s*(?:\{\s*\.\.\.\s*)?omit\(\s*\w+\.outputs!?\s*,\s*\[([^\]]*)\]\s*\)/
)
if (!omission || !inherited) return null
const outputs = { ...inherited }
for (const [, key] of omission[1].matchAll(/['"]([^'"]+)['"]/g)) delete outputs[key]
return outputs
}
function extractToolsAccessFromContent(content: string): string[] {
const toolsMatch = /\btools\s*:\s*\{/.exec(content)
if (!toolsMatch) return []
const toolsStart = toolsMatch.index + toolsMatch[0].lastIndexOf('{')
const toolsEnd = findMatchingClose(content, toolsStart)
if (toolsEnd === -1) return []
const toolsContent = content.substring(toolsStart, toolsEnd)
const accessMatch = /access\s*:\s*\[/.exec(toolsContent)
if (!accessMatch) return []
const start = accessMatch.index + accessMatch[0].lastIndexOf('[')
const end = findMatchingClose(toolsContent, start, '[', ']')
if (end === -1) return []
const source = ts.createSourceFile(
'tool-access.ts',
`const access = ${toolsContent.slice(start, end)}`,
ts.ScriptTarget.Latest,
true
)
const statement = source.statements[0]
if (!statement || !ts.isVariableStatement(statement)) return []
const array = statement.declarationList.declarations[0]?.initializer
if (!array || !ts.isArrayLiteralExpression(array)) return []
return array.elements.flatMap((element) => (ts.isStringLiteral(element) ? [element.text] : []))
}
/**
* Get the tool prefix (service name) from a tool name.
* e.g., "calcom_list_schedules" -> "calcom"
*/
function getToolPrefixFromName(toolName: string): string {
const parts = toolName.split('_')
for (let i = parts.length - 1; i >= 1; i--) {
const possiblePrefix = parts.slice(0, i).join('_')
const toolDirPath = path.join(rootDir, `apps/sim/tools/${possiblePrefix}`)
if (fs.existsSync(toolDirPath) && fs.statSync(toolDirPath).isDirectory()) {
return possiblePrefix
}
}
return parts[0]
}
/**
* Resolve a const reference from a types file.
* Handles nested const references recursively.
*
* @param constName - The const name to resolve (e.g., "SCHEDULE_DATA_OUTPUT_PROPERTIES")
* @param toolPrefix - The tool prefix/service name (e.g., "calcom")
* @param depth - Recursion depth to prevent infinite loops
* @returns Resolved properties object or null if not found
*/
function resolveConstReference(
constName: string,
toolPrefix: string,
depth = 0
): Record<string, any> | null {
if (depth > 10) {
console.warn(`Max recursion depth reached resolving const: ${constName}`)
return null
}
const cacheKey = `${toolPrefix}:${constName}`
if (constResolutionCache.has(cacheKey)) {
return constResolutionCache.get(cacheKey)!
}
const constRegex = new RegExp(`export\\s+const\\s+${constName}\\s*(?::\\s*[^=]+)?\\s*=\\s*\\{`)
let typesContent = ''
let constMatch: RegExpExecArray | null = null
for (const fileName of ['types.ts', 'outputs.ts']) {
const sourcePath = path.join(rootDir, `apps/sim/tools/${toolPrefix}/${fileName}`)
if (!fs.existsSync(sourcePath)) continue
const content = readSourceFile(sourcePath)
const match = constRegex.exec(content)
if (match) {
typesContent = content
constMatch = match
break
}
}
if (!constMatch) {
return null
}
const startIndex = constMatch.index + constMatch[0].length - 1
const endIndex = findMatchingClose(typesContent, startIndex)
if (endIndex === -1) {
return null
}
const constContent = typesContent.substring(startIndex + 1, endIndex - 1).trim()
// Check if this const defines a complete output field (has type property)
// like EVENT_TYPE_OUTPUT = { type: 'object', description: '...', properties: {...} }
const typeMatch = constContent.match(/^\s*type\s*:\s*['"]([^'"]+)['"]/)
if (typeMatch) {
// This is a complete output definition - use parseConstFieldContent
const result = parseConstFieldContent(constContent, toolPrefix, typesContent, depth + 1)
if (result) {
constResolutionCache.set(cacheKey, result)
}
return result
}
// Otherwise, this is a properties object - use parseConstProperties
const properties = parseConstProperties(constContent, toolPrefix, typesContent, depth + 1)
constResolutionCache.set(cacheKey, properties)
return properties
}
/**
* Parse properties from a const definition, resolving nested const references.
*/
export function parseConstProperties(
content: string,
toolPrefix: string,
typesContent: string,
depth: number
): Record<string, any> {
const properties: Record<string, any> = {}
// First, handle spread operators (e.g., "...COMMENT_OUTPUT_PROPERTIES,")
const spreadRegex = /\.\.\.([A-Z][A-Z_0-9]+)\s*(?:,|$)/g
let spreadMatch
while ((spreadMatch = spreadRegex.exec(content)) !== null) {
const constName = spreadMatch[1]
const beforeMatch = content.substring(0, spreadMatch.index)
const openBraces = (beforeMatch.match(/\{/g) || []).length
const closeBraces = (beforeMatch.match(/\}/g) || []).length
if (openBraces !== closeBraces) {
continue
}
const resolvedConst = resolveConstFromTypesContent(constName, typesContent, toolPrefix, depth)
if (resolvedConst && typeof resolvedConst === 'object') {
Object.assign(properties, resolvedConst)
}
}
const propRegex = /(\w+)\s*:\s*(?:\{|([A-Z][A-Z_0-9]+)(?:\s*,|\s*$))/g
let match
while ((match = propRegex.exec(content)) !== null) {
const propName = match[1]
const constRef = match[2]
const beforeMatch = content.substring(0, match.index)
const openBraces = (beforeMatch.match(/\{/g) || []).length
const closeBraces = (beforeMatch.match(/\}/g) || []).length
if (openBraces !== closeBraces) {
continue // Skip - this is a nested property
}
// For 'properties' or 'type', check if it's an output field definition vs a keyword
// Output field definitions have 'type:' inside (e.g., { type: 'string', description: '...' })
if ((propName === 'properties' || propName === 'type') && !constRef) {
const startPos = match.index + match[0].length - 1
const endPos = findMatchingClose(content, startPos)
if (endPos !== -1) {
const propContent = content.substring(startPos + 1, endPos - 1).trim()
// If it starts with 'type:', it's an output field definition - process it
if (propContent.match(/^\s*type\s*:/)) {
const parsedProp = parseConstFieldContent(
propContent,
toolPrefix,
typesContent,
depth,
propName
)
if (parsedProp) {
properties[propName] = parsedProp
}
}
// Otherwise, it's a keyword usage (nested properties block or type specifier) - skip it
}
continue
}
if (constRef) {
// This property references a const (e.g., "attendees: ATTENDEES_OUTPUT")
const resolvedConst = resolveConstFromTypesContent(constRef, typesContent, toolPrefix, depth)
if (resolvedConst) {
properties[propName] = resolvedConst
}
} else {
const startPos = match.index + match[0].length - 1
const endPos = findMatchingClose(content, startPos)
if (endPos !== -1) {
const propContent = content.substring(startPos + 1, endPos - 1).trim()
const parsedProp = parseConstFieldContent(
propContent,
toolPrefix,
typesContent,
depth,
propName
)
if (parsedProp) {
properties[propName] = parsedProp
}
}
}
}
return properties
}
/**
* Resolve a const from the types content (for nested references within the same file).
*/
function resolveConstFromTypesContent(
constName: string,
typesContent: string,
toolPrefix: string,
depth: number
): Record<string, any> | null {
if (depth > 10) return null
const cacheKey = `${toolPrefix}:${constName}`
if (constResolutionCache.has(cacheKey)) {
return constResolutionCache.get(cacheKey)!
}
const constRegex = new RegExp(
`\\b(?:export\\s+)?const\\s+${constName}\\s*(?::\\s*[^=]+)?\\s*=\\s*\\{`,
'g'
)
const constMatch = constRegex.exec(typesContent)
if (!constMatch) {
return null
}
const startIndex = constMatch.index + constMatch[0].length - 1
const endIndex = findMatchingClose(typesContent, startIndex)
if (endIndex === -1) return null
const constContent = typesContent.substring(startIndex + 1, endIndex - 1).trim()
// Check if this const defines a complete output field (has type property)
const typeMatch = constContent.match(/^\s*type\s*:\s*['"]([^'"]+)['"]/)
if (typeMatch) {
const result = parseConstFieldContent(constContent, toolPrefix, typesContent, depth)
if (result) {
constResolutionCache.set(cacheKey, result)
}
return result
}
const properties = parseConstProperties(constContent, toolPrefix, typesContent, depth + 1)
constResolutionCache.set(cacheKey, properties)
return properties
}
/**
* Parse a field content from a const, resolving nested const references.
*/
/**
* Extract description from field content, handling quoted strings properly.
* Handles single quotes, double quotes, and backticks, preserving internal quotes.
*/
function extractDescription(fieldContent: string): string | null {
// Walk through all `description:` matches and return the first one at depth 0.
// This prevents accidentally picking up `description:` keys inside nested child objects.
const descRegex = /description\s*:\s*('([^']*)'|"([^"]*)"|`([^`]*)`)/g
let m: RegExpExecArray | null
while ((m = descRegex.exec(fieldContent)) !== null) {
if (isAtDepthZero(fieldContent, m.index)) {
return m[2] ?? m[3] ?? m[4] ?? null
}
}
return null
}
function parseConstFieldContent(
fieldContent: string,
toolPrefix: string,
typesContent: string,
depth: number,
propertyName?: string
): any {
const typeMatch = fieldContent.match(/type\s*:\s*['"]([^'"]+)['"]/)
const description = extractDescription(fieldContent)
if (!typeMatch) return null
const fieldType = typeMatch[1]
const result: any = {
type: fieldType,
description: description || '',
}
if (fieldType === 'object' || fieldType === 'json') {
const propsConstMatch = findTopLevelMatch(fieldContent, PROPERTIES_CONST_PATTERN)
if (propsConstMatch) {
const resolvedProps = resolveConstFromTypesContent(
propsConstMatch[1],
typesContent,
toolPrefix,
depth + 1
)
if (resolvedProps) {
result.properties = resolvedProps
}
} else {
const propertiesStart = findSchemaKeyword(
fieldContent,
propertyName,
PROPERTIES_INLINE_PATTERN
)
if (propertiesStart !== -1) {
const braceStart = fieldContent.indexOf('{', propertiesStart)
const braceEnd = findMatchingClose(fieldContent, braceStart)
if (braceEnd !== -1) {
const propertiesContent = fieldContent.substring(braceStart + 1, braceEnd - 1).trim()
result.properties = parseConstProperties(
propertiesContent,
toolPrefix,
typesContent,
depth + 1
)
}
}
}
}
const itemsConstMatch = matchSchemaKeyword(fieldContent, propertyName, ITEMS_CONST_PATTERN)
if (itemsConstMatch) {
const resolvedItems = resolveConstFromTypesContent(
itemsConstMatch[1],
typesContent,
toolPrefix,
depth + 1
)
if (resolvedItems) {
result.items = resolvedItems
}
} else {
const itemsStart = findSchemaKeyword(fieldContent, propertyName, ITEMS_INLINE_PATTERN)
if (itemsStart !== -1) {
const braceStart = fieldContent.indexOf('{', itemsStart)
const braceEnd = findMatchingClose(fieldContent, braceStart)
if (braceEnd !== -1) {
const itemsContent = fieldContent.substring(braceStart + 1, braceEnd - 1).trim()
const itemsType = itemsContent.match(/type\s*:\s*['"]([^'"]+)['"]/)
const itemsDesc = extractDescription(itemsContent)
result.items = {
type: itemsType ? itemsType[1] : 'object',
description: itemsDesc || '',
}
// Check for properties in items - either inline or const reference
const itemsPropsConstMatch = findTopLevelMatch(itemsContent, PROPERTIES_CONST_PATTERN)
if (itemsPropsConstMatch) {
const resolvedProps = resolveConstFromTypesContent(
itemsPropsConstMatch[1],
typesContent,
toolPrefix,
depth + 1
)
if (resolvedProps) {
result.items.properties = resolvedProps
}
} else {
const itemsPropsStart = itemsContent.search(/properties\s*:\s*\{/)
if (itemsPropsStart !== -1) {
const propsBraceStart = itemsContent.indexOf('{', itemsPropsStart)
let propsBraceCount = 1
let propsBraceEnd = propsBraceStart + 1
while (propsBraceEnd < itemsContent.length && propsBraceCount > 0) {
if (itemsContent[propsBraceEnd] === '{') propsBraceCount++
else if (itemsContent[propsBraceEnd] === '}') propsBraceCount--
propsBraceEnd++
}
if (propsBraceCount === 0) {
const itemsPropsContent = itemsContent
.substring(propsBraceStart + 1, propsBraceEnd - 1)
.trim()
result.items.properties = parseConstProperties(
itemsPropsContent,
toolPrefix,
typesContent,
depth + 1
)
}
}
}
}
}
}
return result
}
/**
* Extract outputs from a tool content block by trying:
* 1. Const reference (e.g., `outputs: GIT_REF_OUTPUT_PROPERTIES,`)
* 2. Inline object (e.g., `outputs: { id: { type: 'string', ... } }`), with `...sharedConst`
* spreads inlined by `expandSpreads` when the caller can resolve them
*/
function extractOutputsFromToolContent(
content: string,
toolPrefix: string,
expandSpreads: (body: string) => string = (body) => body
): Record<string, any> {
const constMatch = content.match(/(?<![a-zA-Z_])outputs\s*:\s*([A-Z][A-Z_0-9]+)\s*(?:,|\}|$)/)
if (constMatch) {
const resolved = resolveConstReference(constMatch[1], toolPrefix)
if (resolved && typeof resolved === 'object') {
return resolved
}
}
const outputsStart = content.search(/(?<![a-zA-Z_])outputs\s*:\s*{/)
if (outputsStart !== -1) {
const openBracePos = content.indexOf('{', outputsStart)
if (openBracePos !== -1) {
const closePos = findMatchingClose(content, openBracePos)
if (closePos !== -1) {
const outputsContent = content.substring(openBracePos + 1, closePos - 1).trim()
return parseToolOutputsField(expandSpreads(outputsContent), toolPrefix)
}
}
}
return {}
}
/**
* Resolves the module a tool delegates its config to, for tools built by a
* factory instead of an inline object literal:
*
* export const embeddingsOpenAITool = createEmbeddingTool({ id, provider, ... })
*
* The `params` block lives in the factory's module, so a file-local search finds
* nothing and the docs page renders an empty Input table. This follows the
* factory's import the way {@link extractSpreadBase} follows a same-file spread.
* Returns null when the tool declares its own config, which is the common case.
*/
function resolveFactorySource(fileContent: string, toolFilePath: string, rootDir: string): string {
const factoryCall = fileContent.match(/=\s*(create\w+)\s*\(\s*\{/)
if (!factoryCall) return ''
const factoryName = factoryCall[1]
const importMatch = fileContent.match(
new RegExp(`import\\s*\\{[^}]*\\b${factoryName}\\b[^}]*\\}\\s*from\\s*['"]([^'"]+)['"]`)
)
if (!importMatch) return ''
const specifier = importMatch[1]
const resolved = specifier.startsWith('@/')
? path.join(rootDir, 'apps/sim', specifier.slice(2))
: path.resolve(path.dirname(toolFilePath), specifier)
for (const candidate of [`${resolved}.ts`, path.join(resolved, 'index.ts')]) {
if (fs.existsSync(candidate)) return readSourceFile(candidate)
}
return ''
}
/**
* Reads the module a symbol is imported from, so a spread of a shared const
* declared in a sibling module can be followed. Returns an empty string when
* the symbol is not imported or the module cannot be located on disk.
*/
function readImportedModuleSource(
fileContent: string,
symbol: string,
toolFilePath: string,
rootDir: string
): string {
const importMatch = fileContent.match(
new RegExp(`import\\s*(?:type\\s*)?\\{[^}]*\\b${symbol}\\b[^}]*\\}\\s*from\\s*['"]([^'"]+)['"]`)
)
if (!importMatch) return ''
const specifier = importMatch[1]
const resolved = specifier.startsWith('@/')
? path.join(rootDir, 'apps/sim', specifier.slice(2))
: specifier.startsWith('.')
? path.resolve(path.dirname(toolFilePath), specifier)
: ''
if (!resolved) return ''
for (const candidate of [`${resolved}.ts`, path.join(resolved, 'index.ts')]) {
if (fs.existsSync(candidate)) return readSourceFile(candidate)
}
return ''
}
/**
* Inlines `...sharedConst` spreads inside a `params:` or `outputs:` object body.
*
* Tools increasingly hoist their repeated auth/paging/output declarations into a
* sibling `params.ts`. This generator reads tool *source* rather than importing
* it, so an unresolved spread silently drops every one of those rows from the
* published table. Follows same-file declarations first, then the module the
* symbol is imported from, and recurses so a shared const may itself spread.
*/
function expandSpreadConsts(
objectBody: string,
fileContent: string,
toolFilePath: string,
rootDir: string,
seen: Set<string> = new Set()
): string {
return objectBody.replace(/\.\.\.(\w+)\s*,?/g, (whole, symbol: string) => {
if (seen.has(symbol)) return ''
const declRegex = new RegExp(`(?:export\\s+)?const\\s+${symbol}(?=[^a-zA-Z0-9_])[^=]*=\\s*\\{`)
for (const source of [
fileContent,
readImportedModuleSource(fileContent, symbol, toolFilePath, rootDir),
]) {
if (!source) continue
const declMatch = source.match(declRegex)
if (!declMatch || declMatch.index === undefined) continue
const open = declMatch.index + declMatch[0].length - 1
const close = findMatchingClose(source, open)
if (close === -1) continue
const body = source.substring(open + 1, close - 1)
return `${expandSpreadConsts(body, source, toolFilePath, rootDir, new Set([...seen, symbol]))},`
}
return whole
})
}
function extractObjectPropertyBody(content: string, propertyName: string): string | null {
const propertyMatch = content.match(new RegExp(`\\b${propertyName}\\s*:\\s*{`))
if (!propertyMatch || propertyMatch.index === undefined) return null
const open = propertyMatch.index + propertyMatch[0].length - 1
const close = findMatchingClose(content, open)
return close === -1 ? null : content.substring(open + 1, close - 1)
}
export function extractToolInfo(
toolName: string,
fileContent: string,
factorySource = '',
toolFilePath = '',
rootDir = '',
userSettableParamIdSet: ReadonlySet<string> | null = null
): {
description: string
params: Array<{ name: string; type: string; required: boolean; description: string }>
outputs: Record<string, ToolOutputProperty>
} | null {
try {
const declarations = sourceObjectDeclarations(fileContent)
const declaration = declarations.find(
(candidate) => extractStringPropertyFromContent(candidate.content, 'id', true) === toolName
)
const omittedBase = declaration?.content.match(/\boutputs\s*:\s*omit\(\s*(\w+)\.outputs!?\s*,/)
const baseDeclaration =
omittedBase && declarations.find((candidate) => candidate.name === omittedBase[1])
const baseId =
baseDeclaration && extractStringPropertyFromContent(baseDeclaration.content, 'id', true)
if (declaration && baseId && baseId !== toolName) {
const baseInfo = extractToolInfo(
baseId,
baseDeclaration.content,
factorySource,
toolFilePath,
rootDir,
userSettableParamIdSet
)
const outputs = extractOmittedOutputs(declaration.content, baseInfo?.outputs)
if (baseInfo && outputs) {
return {
...baseInfo,
description:
extractStringPropertyFromContent(declaration.content, 'description', true) ||
baseInfo.description,
outputs,
}
}
}
// First, try to find the specific tool definition by its ID
// Look for: id: 'toolName' or id: "toolName"
const toolIdRegex = new RegExp(`id:\\s*['"]${toolName}['"]`)
const toolIdMatch = fileContent.match(toolIdRegex)
let toolContent = fileContent
if (toolIdMatch && toolIdMatch.index !== undefined) {
// Find the tool definition block that contains this ID
// Search backwards for 'export const' or start of object
const beforeId = fileContent.substring(0, toolIdMatch.index)
const exportMatch = beforeId.match(/export\s+const\s+\w+[^=]*=\s*\{[\s\S]*$/)
if (exportMatch && exportMatch.index !== undefined) {
const startIndex = exportMatch.index + exportMatch[0].length - 1
const endIndex = findMatchingClose(fileContent, startIndex)
if (endIndex !== -1) {
toolContent = fileContent.substring(startIndex, endIndex)
}
}
}
// Prefer the params block scoped to this specific tool so that files
// defining multiple tools (e.g. file_compress + file_decompress in
// compress.ts) don't all inherit the first tool's params. Fall back to the
// full file for tools that inherit params via spread from a base object.
const paramsBody =
extractObjectPropertyBody(toolContent, 'params') ??
extractObjectPropertyBody(fileContent, 'params') ??
extractObjectPropertyBody(factorySource, 'params')
// Description should come from the specific tool block if found
// Only search before nested objects (params, outputs, request, etc.) to avoid matching
// descriptions inside outputs or params
let descriptionSearchContent = toolContent
const nestedObjectPatterns = [
/\bparams\s*:\s*[{]/,
/\boutputs\s*:\s*\{/,
/\brequest\s*:\s*\{/,
/\boauth\s*:\s*\{/,
/\btransformResponse\s*:/,
]
let cutoffIndex = toolContent.length
for (const pattern of nestedObjectPatterns) {
const match = toolContent.match(pattern)
if (match && match.index !== undefined && match.index < cutoffIndex) {
cutoffIndex = match.index
}
}
descriptionSearchContent = toolContent.substring(0, cutoffIndex)
// Match against the actual opening quote so apostrophes inside a double-quoted
// description (e.g. "Find someone's email") are not treated as the closing quote.
const descriptionRegex = /description\s*:\s*(?:'([^']*)'|"([^"]*)"|`([^`]*)`)/
let descriptionMatch = descriptionSearchContent.match(descriptionRegex)
// If description isn't found as a literal (might be inherited like description: baseTool.description),
// try to find the referenced tool's description
if (!descriptionMatch) {
const inheritedDescMatch = descriptionSearchContent.match(
/description\s*:\s*(\w+)Tool\.description/
)
if (inheritedDescMatch) {
const baseTool = inheritedDescMatch[1]
const baseToolDescRegex = new RegExp(
`export\\s+const\\s+${baseTool}Tool[^{]*\\{[\\s\\S]*?description\\s*:\\s*(?:'([^']+)'|"([^"]+)"|\`([^\`]+)\`)`,
'i'
)
const baseToolMatch = fileContent.match(baseToolDescRegex)
if (baseToolMatch) {
descriptionMatch = baseToolMatch
}
}
}
const description = descriptionMatch
? (descriptionMatch[1] ??
descriptionMatch[2] ??
descriptionMatch[3] ??
'No description available')
: 'No description available'
const params: Array<{ name: string; type: string; required: boolean; description: string }> = []
if (paramsBody !== null) {
const paramsContent = expandSpreadConsts(paramsBody, fileContent, toolFilePath, rootDir)
const paramBlocksRegex = /(\w+)\s*:\s*{/g
let paramMatch
const paramPositions: Array<{ name: string; start: number; content: string }> = []
/**
* Checks if a position in the string is inside a quoted string.
* This prevents matching patterns like "Example: {" inside description strings.
*/
const isInsideString = (content: string, position: number): boolean => {
let inSingleQuote = false
let inDoubleQuote = false
let inBacktick = false
for (let i = 0; i < position; i++) {
const char = content[i]
const prevChar = i > 0 ? content[i - 1] : ''
if (prevChar === '\\') continue
if (char === "'" && !inDoubleQuote && !inBacktick) {
inSingleQuote = !inSingleQuote
} else if (char === '"' && !inSingleQuote && !inBacktick) {
inDoubleQuote = !inDoubleQuote
} else if (char === '`' && !inSingleQuote && !inDoubleQuote) {
inBacktick = !inBacktick
}
}
return inSingleQuote || inDoubleQuote || inBacktick
}
while ((paramMatch = paramBlocksRegex.exec(paramsContent)) !== null) {
const paramName = paramMatch[1]
const startPos = paramMatch.index + paramMatch[0].length - 1
// Skip matches that are inside string literals (e.g., "Example: {" in descriptions)
if (isInsideString(paramsContent, paramMatch.index)) {
continue
}
const endPos = findMatchingClose(paramsContent, startPos)
if (endPos !== -1) {
const paramBlock = paramsContent.substring(startPos + 1, endPos - 1).trim()
paramPositions.push({ name: paramName, start: startPos, content: paramBlock })
// Resume scanning after this param's block so nested descriptors
// (e.g. an array param's `items: {...}`) are not parsed as params.
paramBlocksRegex.lastIndex = endPos
}
}
for (const param of paramPositions) {
const paramName = param.name
const paramBlock = param.content
if (paramName === 'accessToken' || paramName === 'params' || paramName === 'tools') {
continue
}
/**
* `visibility: 'hidden'` means the param is not an LLM-settable tool argument, so
* it must not appear in the public Input table — emitting it tells integrators they
* can override a value they cannot reach, and several such params are credential-
* shaped (idToken, instanceUrl, apiToken). The exception is a param the owning block
* still exposes as a field the user types: Mailchimp's `apiKey` is hidden on every
* tool because the block injects it, yet the block's own `apiKey` subBlock is the
* only place the requirement is documented. Keep those; drop the rest.
*/
if (
userSettableParamIdSet !== null &&
/visibility\s*:\s*['"]hidden['"]/.test(paramBlock) &&
!userSettableParamIdSet.has(paramName)
) {
continue
}
const typeMatch = paramBlock.match(/type\s*:\s*['"]([^'"]+)['"]/)
const requiredMatch = paramBlock.match(/required\s*:\s*(true|false)/)
let descriptionMatch = paramBlock.match(/description\s*:\s*'(.*?)'(?=\s*[,}])/s)
if (!descriptionMatch) {
descriptionMatch = paramBlock.match(/description\s*:\s*"(.*?)"(?=\s*[,}])/s)
}
if (!descriptionMatch) {
descriptionMatch = paramBlock.match(/description\s*:\s*`([^`]+)`/s)
}
if (!descriptionMatch) {
descriptionMatch = paramBlock.match(
/description\s*:\s*['"]([^'"]*(?:\n[^'"]*)*?)['"](?=\s*[,}])/s
)
}
params.push({
name: paramName,
type: typeMatch ? typeMatch[1] : 'string',
required: requiredMatch ? requiredMatch[1] === 'true' : false,
description: descriptionMatch ? descriptionMatch[1] : '',
})
}
}
const toolPrefix = getToolPrefixFromName(toolName)
const expandOutputSpreads = (body: string) =>
expandSpreadConsts(body, fileContent, toolFilePath, rootDir)
let outputs = extractOutputsFromToolContent(toolContent, toolPrefix, expandOutputSpreads)
// If no outputs found, check for spread inheritance (e.g., "...extendParserTool")
// toolContent may be narrowed past the spread line, so reconstruct the full block
if (Object.keys(outputs).length === 0) {
let fullToolBlock = toolContent
if (toolIdMatch && toolIdMatch.index !== undefined) {
const beforeId = fileContent.substring(0, toolIdMatch.index)
const exportRegex = /export\s+const\s+\w+[^=]*=\s*\{/g
let lastExportMatch: RegExpExecArray | null = null
let m: RegExpExecArray | null = null
while ((m = exportRegex.exec(beforeId)) !== null) {
lastExportMatch = m
}
if (lastExportMatch && lastExportMatch.index !== undefined) {
const bracePos = lastExportMatch.index + lastExportMatch[0].length - 1
const ep = findMatchingClose(fileContent, bracePos)
if (ep !== -1) {
fullToolBlock = fileContent.substring(bracePos, ep)
}
}
}
const spreadMatch = fullToolBlock.match(/\.\.\.(\w+(?:Tool|Base)\w*)/)
if (spreadMatch) {
const baseVarName = spreadMatch[1]
const baseToolRegex = new RegExp(
`export\\s+const\\s+${baseVarName}(?=[^a-zA-Z0-9_]|$)[^=]*=\\s*\\{`
)
const baseToolMatch = fileContent.match(baseToolRegex)
if (baseToolMatch && baseToolMatch.index !== undefined) {
const baseStart = baseToolMatch.index + baseToolMatch[0].length - 1
const endIdx = findMatchingClose(fileContent, baseStart)
if (endIdx !== -1) {
const baseToolContent = fileContent.substring(baseStart, endIdx)
outputs = extractOutputsFromToolContent(
baseToolContent,
toolPrefix,
expandOutputSpreads
)
}
}
}
}
return {
description,
params,
outputs,
}
} catch (error) {
console.error(`Error extracting info for tool ${toolName}:`, error)
return null
}
}
function formatOutputStructure(outputs: Record<string, any>, indentLevel = 0): string {
let result = ''
for (const [key, output] of Object.entries(outputs)) {
let type = 'unknown'
let description = ''
if (typeof output === 'object' && output !== null) {
if (output.type) {
type = output.type
}
if (output.description) {
description = output.description
}
}
const escapedDescription = escapeMdxCell(description)
// Build prefix based on indent level - each level adds 2 spaces before the arrow
let prefix = ''
if (indentLevel > 0) {
const spaces = ' '.repeat(indentLevel)
prefix = `${spaces}↳ `
}
if (typeof output === 'object' && output !== null && output.type === 'array') {
result += `| ${prefix}\`${key}\` | ${type} | ${escapedDescription} |\n`
if (output.items?.properties) {
const arrayItemsResult = formatOutputStructure(output.items.properties, indentLevel + 1)
result += arrayItemsResult
}
} else if (
typeof output === 'object' &&
output !== null &&
output.properties &&
(output.type === 'object' || output.type === 'json')
) {
result += `| ${prefix}\`${key}\` | ${type} | ${escapedDescription} |\n`
const nestedResult = formatOutputStructure(output.properties, indentLevel + 1)
result += nestedResult
} else {
result += `| ${prefix}\`${key}\` | ${type} | ${escapedDescription} |\n`
}
}
return result
}
function parseToolOutputsField(outputsContent: string, toolPrefix?: string): Record<string, any> {
const outputs: Record<string, any> = {}
// First, handle top-level const references
// Patterns: "data: BOOKING_DATA_OUTPUT_PROPERTIES" or "pagination: PAGINATION_OUTPUT"
if (toolPrefix) {
const constRefRegex = /(\w+)\s*:\s*([A-Z][A-Z_0-9]+)\s*(?:,|$)/g
let constMatch
while ((constMatch = constRefRegex.exec(outputsContent)) !== null) {
const propName = constMatch[1]
const constName = constMatch[2]
const beforeMatch = outputsContent.substring(0, constMatch.index)
const openBraces = (beforeMatch.match(/\{/g) || []).length
const closeBraces = (beforeMatch.match(/\}/g) || []).length
if (openBraces !== closeBraces) {
continue
}
const resolvedConst = resolveConstReference(constName, toolPrefix)
if (resolvedConst) {
outputs[propName] = resolvedConst
}
}
// Pattern 2: Property access on const (e.g., "status: BOOKING_DATA_OUTPUT_PROPERTIES.status,")
const propAccessRegex = /(\w+)\s*:\s*([A-Z][A-Z_0-9]+)\.(\w+)\s*(?:,|$)/g
let propAccessMatch
while ((propAccessMatch = propAccessRegex.exec(outputsContent)) !== null) {
const propName = propAccessMatch[1]
const constName = propAccessMatch[2]
const accessedProp = propAccessMatch[3]
if (outputs[propName]) {
continue
}
const beforeMatch = outputsContent.substring(0, propAccessMatch.index)
const openBraces = (beforeMatch.match(/\{/g) || []).length
const closeBraces = (beforeMatch.match(/\}/g) || []).length
if (openBraces !== closeBraces) {
continue
}
const resolvedConst = resolveConstReference(constName, toolPrefix)
if (resolvedConst?.[accessedProp]) {
outputs[propName] = resolvedConst[accessedProp]
}
}
// Pattern 3: Spread operator (e.g., "...COMMENT_OUTPUT_PROPERTIES,")
const spreadRegex = /\.\.\.([A-Z][A-Z_0-9]+)\s*(?:,|$)/g
let spreadMatch
while ((spreadMatch = spreadRegex.exec(outputsContent)) !== null) {
const constName = spreadMatch[1]
const beforeMatch = outputsContent.substring(0, spreadMatch.index)
const openBraces = (beforeMatch.match(/\{/g) || []).length
const closeBraces = (beforeMatch.match(/\}/g) || []).length
if (openBraces !== closeBraces) {
continue
}
const resolvedConst = resolveConstReference(constName, toolPrefix)
if (resolvedConst && typeof resolvedConst === 'object') {
Object.assign(outputs, resolvedConst)
}
}
}
const braces: Array<{ type: 'open' | 'close'; pos: number; level: number }> = []
for (let i = 0; i < outputsContent.length; i++) {
if (outputsContent[i] === '{') {
braces.push({ type: 'open', pos: i, level: 0 })
} else if (outputsContent[i] === '}') {
braces.push({ type: 'close', pos: i, level: 0 })
}
}
let currentLevel = 0
for (const brace of braces) {
if (brace.type === 'open') {
brace.level = currentLevel
currentLevel++
} else {
currentLevel--
brace.level = currentLevel
}
}
const fieldStartRegex = /(\w+)\s*:\s*{/g
let match
const fieldPositions: Array<{ name: string; start: number; end: number; level: number }> = []
while ((match = fieldStartRegex.exec(outputsContent)) !== null) {
const fieldName = match[1]
const bracePos = match.index + match[0].length - 1
if (outputs[fieldName]) {
continue
}
const openBrace = braces.find((b) => b.type === 'open' && b.pos === bracePos)
if (openBrace) {
const endPos = findMatchingClose(outputsContent, bracePos)
if (endPos !== -1) {
fieldPositions.push({
name: fieldName,
start: bracePos,
end: endPos,
level: openBrace.level,
})
}
}
}
const topLevelFields = fieldPositions.filter((f) => f.level === 0)
topLevelFields.forEach((field) => {
const fieldContent = outputsContent.substring(field.start + 1, field.end - 1).trim()
const parsedField = parseFieldContent(fieldContent, toolPrefix)
if (parsedField) {
outputs[field.name] = parsedField
}
})
return outputs
}
/**
* Returns true if the regex match at `matchIndex` within `content` is at brace depth 0.
* Used to distinguish top-level keys from keys nested inside child objects.
*/
function isAtDepthZero(content: string, matchIndex: number): boolean {
let depth = 0
for (let i = 0; i < matchIndex; i++) {
if (content[i] === '{') depth++
else if (content[i] === '}') depth--
}
return depth === 0
}
function findTopLevelMatch(content: string, pattern: RegExp): RegExpExecArray | null {
const flags = pattern.flags.includes('g') ? pattern.flags : `${pattern.flags}g`
const regex = new RegExp(pattern.source, flags)
let match: RegExpExecArray | null
while ((match = regex.exec(content)) !== null) {
if (isAtDepthZero(content, match.index)) {
return match
}
}
return null
}
const PROPERTIES_CONST_PATTERN = /properties\s*:\s*([A-Z][A-Z_0-9]+)/
const PROPERTIES_INLINE_PATTERN = /properties\s*:\s*{/
const ITEMS_CONST_PATTERN = /items\s*:\s*([A-Z][A-Z_0-9]+)/
const ITEMS_INLINE_PATTERN = /items\s*:\s*{/
function matchSchemaKeyword(
content: string,
propertyName: string | undefined,
pattern: RegExp
): RegExpExecArray | null {
return propertyName === 'items' ? findTopLevelMatch(content, pattern) : content.match(pattern)
}
function findSchemaKeyword(
content: string,
propertyName: string | undefined,
pattern: RegExp
): number {
return propertyName === 'items'
? (findTopLevelMatch(content, pattern)?.index ?? -1)
: content.search(pattern)
}
function parseFieldContent(fieldContent: string, toolPrefix?: string, propertyName?: string): any {
// Only match `type:` that is at the top level of fieldContent (depth 0).
// Child objects like `title: { type: 'string', ... }` also contain `type:` but at depth 1.
const typeRegex = /type\s*:\s*['"]([^'"]+)['"]/g
let typeMatch: RegExpExecArray | null = null
let m: RegExpExecArray | null
while ((m = typeRegex.exec(fieldContent)) !== null) {
if (isAtDepthZero(fieldContent, m.index)) {
typeMatch = m
break
}
}
const description = extractDescription(fieldContent)
// Check for spread operator at the start of field content (e.g., ...SUBSCRIPTION_OUTPUT)
// This pattern is used when a field spreads a complete output definition and optionally overrides properties
const spreadMatch = fieldContent.match(/^\s*\.\.\.([A-Z][A-Z_0-9]+)\s*,/)
if (spreadMatch && toolPrefix && !typeMatch) {
const constName = spreadMatch[1]
const resolvedConst = resolveConstReference(constName, toolPrefix)
if (resolvedConst && typeof resolvedConst === 'object') {
// Start with the resolved const and override with inline properties
const result: any = { ...resolvedConst }
if (description) {
result.description = description
}
return result
}
}
if (!typeMatch) {
// No top-level `type` key — check if the content contains named child fields that each
// have their own `type` property. This is the "implicit object" pattern used in trigger
// outputs (e.g., Cal.com's `payload`, Linear's `data`).
const properties = parsePropertiesContent(fieldContent, toolPrefix)
if (Object.keys(properties).length > 0) {
return {
type: 'object',
description: description || '',
properties,
}
}
return null
}
const fieldType = typeMatch[1]
const result: any = {
type: fieldType,
description: description || '',
}
if (fieldType === 'object' || fieldType === 'json') {
// Check for const reference first (e.g., properties: SCHEDULE_DATA_OUTPUT_PROPERTIES)
const propsConstMatch = matchSchemaKeyword(fieldContent, propertyName, PROPERTIES_CONST_PATTERN)
if (propsConstMatch && toolPrefix) {
const resolvedProps = resolveConstReference(propsConstMatch[1], toolPrefix)
if (resolvedProps) {
result.properties = resolvedProps
}
} else {
const propertiesStart = findSchemaKeyword(
fieldContent,
propertyName,
PROPERTIES_INLINE_PATTERN
)
if (propertiesStart !== -1) {
const braceStart = fieldContent.indexOf('{', propertiesStart)
const braceEnd = findMatchingClose(fieldContent, braceStart)
if (braceEnd !== -1) {
const propertiesContent = fieldContent.substring(braceStart + 1, braceEnd - 1).trim()
result.properties = parsePropertiesContent(propertiesContent, toolPrefix)
}
}
}
}
// Check for items const reference (e.g., items: ATTENDEES_OUTPUT)
const itemsConstMatch = matchSchemaKeyword(fieldContent, propertyName, ITEMS_CONST_PATTERN)
if (itemsConstMatch && toolPrefix) {
const resolvedItems = resolveConstReference(itemsConstMatch[1], toolPrefix)
if (resolvedItems) {
result.items = resolvedItems
}
} else {
const itemsStart = findSchemaKeyword(fieldContent, propertyName, ITEMS_INLINE_PATTERN)
if (itemsStart !== -1) {
const braceStart = fieldContent.indexOf('{', itemsStart)
const braceEnd = findMatchingClose(fieldContent, braceStart)
if (braceEnd !== -1) {
const itemsContent = fieldContent.substring(braceStart + 1, braceEnd - 1).trim()
const itemsType = itemsContent.match(/type\s*:\s*['"]([^'"]+)['"]/)
// Check for inline properties FIRST (properties: {), then const reference
const propertiesInlineStart = itemsContent.search(/properties\s*:\s*{/)
// Only match const reference if it's at the TOP level (before any {)
const itemsPropsConstMatch =
propertiesInlineStart === -1
? itemsContent.match(/properties\s*:\s*([A-Z][A-Z_0-9]+)/)
: null
const searchContent =
propertiesInlineStart >= 0
? itemsContent.substring(0, propertiesInlineStart)
: itemsContent
const itemsDesc = extractDescription(searchContent)
result.items = {
type: itemsType ? itemsType[1] : 'object',
description: itemsDesc || '',
}
if (itemsPropsConstMatch && toolPrefix) {
const resolvedProps = resolveConstReference(itemsPropsConstMatch[1], toolPrefix)
if (resolvedProps) {
result.items.properties = resolvedProps
}
} else if (propertiesInlineStart !== -1) {
const itemsPropertiesRegex = /properties\s*:\s*{/
const itemsPropsStart = itemsContent.search(itemsPropertiesRegex)
if (itemsPropsStart !== -1) {
const propsBraceStart = itemsContent.indexOf('{', itemsPropsStart)
let propsBraceCount = 1
let propsBraceEnd = propsBraceStart + 1
while (propsBraceEnd < itemsContent.length && propsBraceCount > 0) {
if (itemsContent[propsBraceEnd] === '{') propsBraceCount++
else if (itemsContent[propsBraceEnd] === '}') propsBraceCount--
propsBraceEnd++
}
if (propsBraceCount === 0) {
const itemsPropsContent = itemsContent
.substring(propsBraceStart + 1, propsBraceEnd - 1)
.trim()
result.items.properties = parsePropertiesContent(itemsPropsContent, toolPrefix)
}
}
}
}
}
}
return result
}
export function parsePropertiesContent(
propertiesContent: string,
toolPrefix?: string
): Record<string, any> {
const properties: Record<string, any> = {}
// First, handle const references at the property level
// Patterns: "attendees: ATTENDEES_OUTPUT" or "id: BOOKING_DATA_OUTPUT_PROPERTIES.id"
if (toolPrefix) {
// Pattern 1: Direct const reference (e.g., "eventType: EVENT_TYPE_OUTPUT,")
const constRefRegex = /(\w+)\s*:\s*([A-Z][A-Z_0-9]+)\s*(?:,|$)/g
let constMatch
while ((constMatch = constRefRegex.exec(propertiesContent)) !== null) {
const propName = constMatch[1]
const constName = constMatch[2]
if (propName === 'properties' || propName === 'type') {
continue
}
const beforeMatch = propertiesContent.substring(0, constMatch.index)
const openBraces = (beforeMatch.match(/\{/g) || []).length
const closeBraces = (beforeMatch.match(/\}/g) || []).length
if (openBraces !== closeBraces) {
continue
}
const resolvedConst = resolveConstReference(constName, toolPrefix)
if (resolvedConst) {
properties[propName] = resolvedConst
}
}
// Pattern 2: Property access on const (e.g., "id: BOOKING_DATA_OUTPUT_PROPERTIES.id,")
const propAccessRegex = /(\w+)\s*:\s*([A-Z][A-Z_0-9]+)\.(\w+)\s*(?:,|$)/g
let propAccessMatch
while ((propAccessMatch = propAccessRegex.exec(propertiesContent)) !== null) {
const propName = propAccessMatch[1]
const constName = propAccessMatch[2]
const accessedProp = propAccessMatch[3]
if (propName === 'properties' || propName === 'type') {
continue
}
if (properties[propName]) {
continue
}
const beforeMatch = propertiesContent.substring(0, propAccessMatch.index)
const openBraces = (beforeMatch.match(/\{/g) || []).length
const closeBraces = (beforeMatch.match(/\}/g) || []).length
if (openBraces !== closeBraces) {
continue
}
const resolvedConst = resolveConstReference(constName, toolPrefix)
if (resolvedConst?.[accessedProp]) {
properties[propName] = resolvedConst[accessedProp]
}
}
// Pattern 3: Spread operator (e.g., "...COMMENT_OUTPUT_PROPERTIES,")
const spreadRegex = /\.\.\.([A-Z][A-Z_0-9]+)\s*(?:,|$)/g
let spreadMatch
while ((spreadMatch = spreadRegex.exec(propertiesContent)) !== null) {
const constName = spreadMatch[1]
const beforeMatch = propertiesContent.substring(0, spreadMatch.index)
const openBraces = (beforeMatch.match(/\{/g) || []).length
const closeBraces = (beforeMatch.match(/\}/g) || []).length
if (openBraces !== closeBraces) {
continue
}
const resolvedConst = resolveConstReference(constName, toolPrefix)
if (resolvedConst && typeof resolvedConst === 'object') {
Object.assign(properties, resolvedConst)
}
}
}
const propStartRegex = /(\w+)\s*:\s*{/g
let match
const propPositions: Array<{ name: string; start: number; content: string }> = []
while ((match = propStartRegex.exec(propertiesContent)) !== null) {
const propName = match[1]
if (propName === 'properties') {
continue
}
if (properties[propName]) {
continue
}
// Check if this match is at depth 0 (not inside nested braces)
// Only process top-level properties, skip nested ones
const beforeMatch = propertiesContent.substring(0, match.index)
const openBraces = (beforeMatch.match(/{/g) || []).length
const closeBraces = (beforeMatch.match(/}/g) || []).length
if (openBraces !== closeBraces) {
continue // Skip - this is a nested property
}
const startPos = match.index + match[0].length - 1
const endPos = findMatchingClose(propertiesContent, startPos)
if (endPos !== -1) {
const propContent = propertiesContent.substring(startPos + 1, endPos - 1).trim()
const hasDescription = /description\s*:\s*/.test(propContent)
const hasProperties = /properties\s*:\s*[{A-Z]/.test(propContent)
const hasItems = /items\s*:\s*[{A-Z]/.test(propContent)
const isTypeOnly =
!hasDescription &&
!hasProperties &&
!hasItems &&
/^type\s*:\s*['"].*?['"]\s*,?\s*$/.test(propContent)
if (!isTypeOnly) {
propPositions.push({
name: propName,
start: startPos,
content: propContent,
})
}
}
}
propPositions.forEach((prop) => {
const parsedProp = parseFieldContent(prop.content, toolPrefix, prop.name)
if (parsedProp) {
properties[prop.name] = parsedProp
}
})
return properties
}
/** Only the matching tool declaration determines whether its outputs come from a factory. */
export function isFactoryToolDeclaration(toolName: string, content: string): boolean {
const source = ts.createSourceFile('tool.ts', content, ts.ScriptTarget.Latest, true)
for (const statement of source.statements) {
if (!ts.isVariableStatement(statement)) continue
for (const declaration of statement.declarationList.declarations) {
if (!declaration.initializer) continue
const { expression } = unwrapExpression(declaration.initializer)
if (!ts.isCallExpression(expression)) continue
const config = expression.arguments[0]
if (!config || !ts.isObjectLiteralExpression(config)) continue
if (extractStringPropertyFromContent(config.getText(source), 'id', true) === toolName)
return true
}
}
return false
}
/** Wrapped tool declarations use the canonical evaluated output metadata. */
function hasWrappedToolBase(toolName: string, content: string): boolean {
const declarations = sourceObjectDeclarations(content)
const declaration = declarations.find(
(candidate) => extractStringPropertyFromContent(candidate.content, 'id', true) === toolName
)
if (!declaration) return false
const baseName = declaration.content.match(/^\s*\.\.\.(\w+)\s*,/m)?.[1]
const base = declarations.find((candidate) => candidate.name === baseName)
return base?.satisfies === true
}
export async function getToolInfo(
toolName: string,
userSettableParamIds: readonly string[] | null = null,
/** The block signs in with a saved credential, so params that credential supplies are not inputs. */
credentialOnly = false
): Promise<{
description: string
params: Array<{ name: string; type: string; required: boolean; description: string }>
outputs: Record<string, any>
} | null> {
const userSettableParamIdSet =
userSettableParamIds === null ? null : new Set(userSettableParamIds)
try {
const metadata = (await loadToolMetadata())[toolName]
const generatedOutputs = (await loadToolOutputs())[toolName]
const parts = toolName.split('_')
let toolPrefix = ''
let toolSuffix = ''
for (let i = parts.length - 1; i >= 1; i--) {
const possiblePrefix = parts.slice(0, i).join('_')
const possibleSuffix = parts.slice(i).join('_')
const toolDirPath = path.join(rootDir, `apps/sim/tools/${possiblePrefix}`)
if (fs.existsSync(toolDirPath) && fs.statSync(toolDirPath).isDirectory()) {
toolPrefix = possiblePrefix
toolSuffix = possibleSuffix
break
}
}
if (!toolPrefix) {
toolPrefix = parts[0]
toolSuffix = parts.slice(1).join('_')
}
const isVersionedTool = isVersionedType(toolSuffix)
const strippedToolSuffix = stripVersionSuffix(toolSuffix)
const possibleLocations: Array<{ path: string; priority: 'exact' | 'fallback' }> = []
// For versioned tools, prioritize the exact versioned file first
// This handles cases like google_sheets where V2 is in a separate file (read_v2.ts)
if (isVersionedTool) {
possibleLocations.push({
path: path.join(rootDir, `apps/sim/tools/${toolPrefix}/${toolSuffix}.ts`),
priority: 'exact',
})
// Second priority: stripped file that contains both V1 and V2 (e.g., pr.ts for github)
possibleLocations.push({
path: path.join(rootDir, `apps/sim/tools/${toolPrefix}/${strippedToolSuffix}.ts`),
priority: 'fallback',
})
} else {
possibleLocations.push({
path: path.join(rootDir, `apps/sim/tools/${toolPrefix}/${toolSuffix}.ts`),
priority: 'exact',
})
}
const camelCaseSuffix = strippedToolSuffix
.split('_')
.map((part, i) => (i === 0 ? part : part.charAt(0).toUpperCase() + part.slice(1)))
.join('')
possibleLocations.push({
path: path.join(rootDir, `apps/sim/tools/${toolPrefix}/${camelCaseSuffix}.ts`),
priority: 'fallback',
})
possibleLocations.push({
path: path.join(rootDir, `apps/sim/tools/${toolPrefix}/index.ts`),
priority: 'fallback',
})
let toolFileContent = ''
let foundFile = ''
let foundExactId = false
for (const location of possibleLocations) {
if (fs.existsSync(location.path)) {
const content = readSourceFile(location.path)
const toolIdRegex = new RegExp(`id:\\s*['"]${toolName}['"]`)
if (toolIdRegex.test(content)) {
toolFileContent = content
foundFile = location.path
foundExactId = true
break
}
// For fallback locations, store the content in case we don't find an exact match
if (location.priority === 'fallback' && !toolFileContent) {
toolFileContent = content
foundFile = location.path
}
}
}
// The named-file candidates above miss tools defined inside a sibling tool's
// file (e.g. file_decompress lives in compress.ts). Before accepting an
// arbitrary fallback file, scan the whole tool-prefix directory for the file
// that declares this exact tool ID.
if (!foundExactId) {
const prefixDir = path.join(rootDir, `apps/sim/tools/${toolPrefix}`)
if (fs.existsSync(prefixDir)) {
const dirFiles = await sourceGlob(`${prefixDir}/**/*.ts`)
const toolIdRegex = new RegExp(`id:\\s*['"]${toolName}['"]`)
for (const dirFile of dirFiles) {
if (dirFile.endsWith('.test.ts')) continue
const content = readSourceFile(dirFile)
if (toolIdRegex.test(content)) {
toolFileContent = content
foundFile = dirFile
foundExactId = true
break
}
}
}
}
// If we didn't find a file with the exact ID, use the first available file
if (!toolFileContent) {
for (const location of possibleLocations) {
if (fs.existsSync(location.path)) {
toolFileContent = readSourceFile(location.path)
foundFile = location.path
break
}
}
}
if (!toolFileContent && !metadata) {
console.warn(`Could not find definition for tool: ${toolName}`)
return null
}
const sourceInfo = toolFileContent
? extractToolInfo(
toolName,
toolFileContent,
resolveFactorySource(toolFileContent, foundFile, rootDir),
foundFile,
rootDir,
userSettableParamIdSet
)
: null
if (!metadata) return sourceInfo
/**
* The same hidden-param rule `extractToolInfo` applies to source-parsed params, applied to the
* metadata-derived ones. `tool-metadata.ts` carries `visibility` for every param, so this is the
* authoritative form of the check; the source-side filter remains for tools with no metadata
* entry. A null set means the block's subBlocks could not be read, so nothing is filtered.
*/
const credentialSupplied = new Set(
credentialOnly ? (metadata.oauth?.credentialSuppliedParams ?? []) : []
)
const params = Object.entries(metadata.params ?? {})
.filter(([name]) => name !== 'accessToken' && !credentialSupplied.has(name))
.filter(
([name, param]) =>
userSettableParamIdSet === null ||
param.visibility !== 'hidden' ||
userSettableParamIdSet.has(name)
)
.map(([name, param]) => ({
name,
type: typeof param.type === 'string' ? param.type : 'string',
required: param.required === true,
description: typeof param.description === 'string' ? param.description : '',
}))
return {
description: metadata.description ?? sourceInfo?.description ?? 'No description available',
params,
outputs:
isFactoryToolDeclaration(toolName, toolFileContent) ||
toolPrefix === 'buffer' ||
toolPrefix === 'cloudflare' ||
toolPrefix === 'sailpoint' ||
toolPrefix === 'otter' ||
toolPrefix === 'plane' ||
toolName === 'file_edit' ||
hasWrappedToolBase(toolName, toolFileContent)
? (generatedOutputs ?? sourceInfo?.outputs ?? {})
: (sourceInfo?.outputs ?? generatedOutputs ?? {}),
}
} catch (error) {
console.error(`Error getting info for tool ${toolName}:`, error)
return null
}
}
function extractManualContent(existingContent: string): Record<string, string> {
const manualSections: Record<string, string> = {}
const manualContentRegex =
/\{\/\*\s*MANUAL-CONTENT-START:(\w+)\s*\*\/\}([\s\S]*?)\{\/\*\s*MANUAL-CONTENT-END\s*\*\/\}/g
let match
while ((match = manualContentRegex.exec(existingContent)) !== null) {
const sectionName = match[1]
const content = match[2].trim()
manualSections[sectionName] = content
}
return manualSections
}
function mergeWithManualContent(
generatedMarkdown: string,
existingContent: string | null,
manualSections: Record<string, string>
): string {
if (!existingContent || Object.keys(manualSections).length === 0) {
return generatedMarkdown
}
let mergedContent = generatedMarkdown
Object.entries(manualSections).forEach(([sectionName, content]) => {
const insertionPoints: Record<string, { regex: RegExp }> = {
intro: {
regex: /^---\n[\s\S]*?\n---\n(?:\nimport [^\n]+\n)*/,
},
usage: {
regex: /## Use the block \[#usage-instructions\]/,
},
outputs: {
regex: /## Outputs/,
},
notes: {
regex: /## Notes/,
},
}
const insertionPoint = insertionPoints[sectionName]
const wrapped = `{/* MANUAL-CONTENT-START:${sectionName} */}\n${content}\n{/* MANUAL-CONTENT-END */}`
const match = insertionPoint ? mergedContent.match(insertionPoint.regex) : null
if (match && match.index !== undefined) {
const insertPosition = match.index + match[0].length
mergedContent = `${mergedContent.slice(0, insertPosition)}\n\n${wrapped}\n${mergedContent.slice(insertPosition)}`
} else {
// Never drop manual content: when the anchor is missing (e.g. a `notes`
// section with no generated "## Notes" heading), append at the end.
console.log(`No insertion anchor for manual section "${sectionName}" — appending at end`)
mergedContent = `${mergedContent.replace(/\s*$/, '')}\n\n${wrapped}\n`
}
})
return mergedContent
}
async function generateBlockDoc(blockPath: string) {
try {
const blockFileName = path.basename(blockPath, '.ts')
if (blockFileName.endsWith('.test')) {
return
}
const fileContent = readSourceFile(blockPath)
// Extract ALL block configs from the file (already filters out hideFromToolbar: true)
const blockConfigs = blockConfigsForFile(blockPath)
if (blockConfigs.length === 0) {
console.warn(`Skipping ${blockFileName} - no valid block configs found`)
return
}
for (const blockConfig of blockConfigs) {
if (!blockConfig.type) {
continue
}
if (
blockConfig.type.includes('_trigger') ||
blockConfig.type.includes('_webhook') ||
blockConfig.type.includes('rss')
) {
console.log(`Skipping ${blockConfig.type} - contains '_trigger'`)
continue
}
if (
(blockConfig.category === 'blocks' &&
!NATIVE_RESOURCE_BLOCK_TYPES.has(stripVersionSuffix(blockConfig.type))) ||
blockConfig.type === 'sim_workspace_event' ||
blockConfig.type === 'evaluator' ||
blockConfig.type === 'number' ||
blockConfig.type === 'webhook' ||
blockConfig.type === 'schedule' ||
blockConfig.type === 'mcp' ||
blockConfig.type === 'generic_webhook' ||
blockConfig.type === 'rss'
) {
continue
}
// Use stripped type for file name (removes _v2, _v3 suffixes for cleaner URLs)
const displayType = stripVersionSuffix(blockConfig.type)
const outputFilePath = path.join(DOCS_OUTPUT_PATH, `${displayType}.mdx`)
const existingContent = readGeneratedFile(outputFilePath)
const manualSections = existingContent ? extractManualContent(existingContent) : {}
const oauthServiceId =
extractAuthType(fileContent) === 'oauth' ? extractOAuthServiceId(fileContent) : undefined
const oauthCredential = oauthServiceId
? { serviceId: oauthServiceId, environmentBound: isEnvironmentBoundCredential(fileContent) }
: undefined
const markdown = await generateMarkdownForBlock(blockConfig, oauthCredential)
let finalContent = markdown
if (Object.keys(manualSections).length > 0) {
finalContent = mergeWithManualContent(markdown, existingContent, manualSections)
}
emitGeneratedFile(outputFilePath, finalContent)
if (!CHECK_ONLY) {
const logType =
displayType !== blockConfig.type
? `${displayType} (from ${blockConfig.type})`
: displayType
console.log(`✓ Generated docs for ${logType}`)
}
}
} catch (error) {
console.error(`Error processing ${blockPath}:`, error)
}
}
async function generateMarkdownForBlock(
blockConfig: BlockConfig,
oauthCredential?: OAuthCredentialSource
): Promise<string> {
const {
type,
name,
description,
longDescription,
bgColor,
outputs = {},
tools = { access: [] },
userSettableParamIds = null,
} = blockConfig
let toolsSection = ''
let hasReferenceDetails = false
const inputSchemas = new Map<string, string>()
if (tools.access?.length) {
toolsSection = '## Actions\n\n'
const displayNames = await loadToolDisplayNames()
const actionNames = tools.access.map(
(tool) => displayNames.get(tool) ?? displayNames.get(stripVersionSuffix(tool)) ?? tool
)
const operationGroups = new Map(
(blockConfig.operations ?? []).flatMap((operation) =>
operation.group
? [[`${stripVersionSuffix(type)}_${operation.id}`, operation.group] as const]
: []
)
)
const orderedGroups = tools.access.map((tool) => operationGroups.get(stripVersionSuffix(tool)))
const groupRuns = orderedGroups.filter((group, index) => group !== orderedGroups[index - 1])
const groupActions =
new Set(actionNames).size < actionNames.length &&
orderedGroups.every((group) => group !== undefined) &&
new Set(orderedGroups).size === groupRuns.length
const actionHeading = groupActions ? '####' : '###'
const fieldHeading = groupActions ? '#####' : '####'
let activeGroup: string | undefined
for (const tool of tools.access) {
const group = groupActions ? operationGroups.get(stripVersionSuffix(tool)) : undefined
if (group && group !== activeGroup) {
toolsSection += `### ${group} [#actions-${tool}]\n\n`
activeGroup = group
}
// Prefer the tool's own name ("A2A Send Message") over its id — the id is an
// implementation detail, and these headings are what the page's table of
// contents shows. Falls back to the id for a tool missing from the metadata.
const heading = displayNames.get(tool) ?? displayNames.get(stripVersionSuffix(tool))
toolsSection += `${actionHeading} ${heading ?? stripVersionSuffix(tool)}\n\n`
console.log(`Getting info for tool: ${tool}`)
const toolInfo = await getToolInfo(tool, userSettableParamIds, Boolean(oauthCredential))
if (toolInfo) {
if (toolInfo.description && toolInfo.description !== 'No description available') {
const escapedToolDescription = escapeMdxProse(toolInfo.description)
toolsSection += `${escapedToolDescription}\n\n`
}
let fields = ''
fields += `${fieldHeading} Input\n\n`
fields += '| Parameter | Type | Required | Description |\n'
fields += '| --------- | ---- | -------- | ----------- |\n'
if (toolInfo.params.length > 0) {
for (const param of toolInfo.params) {
const schema = param.description.match(
/^(.*?)\n\n(```graphql\n(?:input|enum) (\w+)[\s\S]*\n```)$/s
)
let description = schema?.[1] ?? param.description
if (schema) {
inputSchemas.set(schema[3], schema[2])
description = `${escapeMdxCell(description)} <a href="#${schema[3].toLowerCase()}">Schema</a>`
} else {
description = description ? escapeMdxCell(description) : ''
}
fields += `| \`${param.name}\` | ${param.type} | ${param.required ? 'Yes' : 'No'} | ${description} |\n`
}
}
fields += `\n${fieldHeading} Output\n\n`
if (Object.keys(toolInfo.outputs).length > 0) {
fields += '| Parameter | Type | Description |\n'
fields += '| --------- | ---- | ----------- |\n'
fields += formatOutputStructure(toolInfo.outputs)
} else if (
Object.hasOwn(await loadToolOutputs(), tool) &&
Object.keys((await loadToolOutputs())[tool]).length === 0
) {
fields += 'No output fields.\n'
} else if (Object.keys(outputs).length > 0) {
fields += '| Parameter | Type | Description |\n'
fields += '| --------- | ---- | ----------- |\n'
for (const [key, output] of Object.entries(outputs)) {
let type = 'string'
let description = ''
if (typeof output === 'string') {
type = output
} else if (typeof output === 'object' && output !== null) {
if ('type' in output && typeof output.type === 'string') {
type = output.type
}
if ('description' in output && typeof output.description === 'string') {
description = output.description
}
}
const escapedDescription = escapeMdxCell(description)
fields += `| \`${key}\` | ${type} | ${escapedDescription} |\n`
}
} else {
fields += 'No output fields are documented for this action.\n'
}
hasReferenceDetails = true
toolsSection += `<ReferenceDetails>\n\n${fields}\n</ReferenceDetails>\n`
}
toolsSection += '\n'
}
}
if (inputSchemas.size) {
toolsSection += '## JSON input schemas [#json-input-schemas]\n\n'
toolsSection +=
'These GraphQL type definitions describe the JSON inputs. `!` marks required values, and brackets mark arrays. Fields with a provider default can be omitted.\n\n'
for (const [name, definition] of inputSchemas) {
toolsSection += `### ${name}\n\n${definition}\n\n`
}
}
let usageInstructions = ''
if (longDescription) {
usageInstructions = `## Use the block [#usage-instructions]\n\n${longDescription}\n\n`
}
const oauthScopesSection = oauthCredential
? await buildOAuthScopesSection(oauthCredential, tools.access ?? [])
: ''
const connectionGuides = Object.entries(integrationNavigation.guides)
.filter(([, guide]) => guide.integration === stripVersionSuffix(type))
.map(([slug, guide]) => `[${guide.title}](/integrations/${slug})`)
const connectionSection = connectionGuides.length
? `Connection guides: ${connectionGuides.join(' · ')}.\n\n`
: ''
return `---
title: ${name}
description: ${description}
integration:
type: ${JSON.stringify(type)}
color: ${JSON.stringify(bgColor || '#F5F5F5')}
---
${hasReferenceDetails ? 'import { ReferenceDetails } from "@/components/ui/reference-details"\n' : ''}
${connectionSection}${oauthScopesSection}${usageInstructions}${toolsSection}
`
}
const OAUTH_SCOPES_HEADING = '## OAuth scopes [#oauth-scopes]'
const DATAVERSE_ENVIRONMENT_SCOPE = '<environment-url>/.default'
const DATAVERSE_SAMPLE_ENVIRONMENT = 'https://yourorg.crm.dynamics.com'
const SELF_HOSTING_OAUTH_GUIDE = '/platform/self-hosting/integrations-oauth'
const CONNECTOR_PROVIDERS_PATH = path.join(rootDir, 'apps/sim/lib/auth/connectors/providers.ts')
const OAUTH_CALLBACK_ROUTES_PATH = path.join(rootDir, 'apps/sim/app/api/auth/oauth2/callback')
/** The block's OAuth credential subBlock, as the scopes section needs it. */
interface OAuthCredentialSource {
serviceId: string
environmentBound: boolean
}
interface OAuthDocsSources {
getMicrosoftDataverseOAuthScopes: (environmentUrl: string) => string[]
getServiceConfigByServiceId: (serviceId: string) => OAuthServiceConfig | null
getScopeDescription: (scope: string, providerId?: string) => string
isScopeSatisfiedBy: (required: string, granted: ReadonlySet<string>) => boolean
getSlackApprovalGatedScopes: (enabled: boolean) => readonly string[]
getOAuthClientCapabilityFields: (serviceId: string) => readonly string[] | null
salesforceLoginHosts: Readonly<Record<string, string>>
}
let oauthDocsSources: OAuthDocsSources | null = null
/**
* Loads the runtime OAuth registries the scopes section documents, so the page
* lists exactly what the app requests rather than a hand-kept copy.
*/
async function loadOAuthDocsSources(): Promise<OAuthDocsSources> {
if (oauthDocsSources) return oauthDocsSources
const [utils, oauth, salesforce, dataverse, capabilities] = await Promise.all([
import(path.join(rootDir, 'apps/sim/lib/oauth/utils.ts')),
import(path.join(rootDir, 'apps/sim/lib/oauth/oauth.ts')),
import(path.join(rootDir, 'apps/sim/lib/oauth/salesforce.ts')),
import(path.join(rootDir, 'apps/sim/lib/oauth/microsoft-dataverse.ts')),
import(path.join(rootDir, 'packages/deployment-config/src/env-capabilities.ts')),
])
oauthDocsSources = {
getMicrosoftDataverseOAuthScopes: dataverse.getMicrosoftDataverseOAuthScopes,
getServiceConfigByServiceId: utils.getServiceConfigByServiceId,
getScopeDescription: utils.getScopeDescription,
isScopeSatisfiedBy: utils.isScopeSatisfiedBy,
getSlackApprovalGatedScopes: oauth.getSlackApprovalGatedScopes,
getOAuthClientCapabilityFields: capabilities.getOAuthClientCapabilityFields,
salesforceLoginHosts: salesforce.SALESFORCE_LOGIN_HOSTS,
}
return oauthDocsSources
}
/**
* The path a provider returns to after consent, or null when the service does
* not use an OAuth 2.0 redirect (Trello authorizes with an API key). Better
* Auth connectors declare their redirect literally (Salesforce builds one per
* login host); Instagram, Shopify, and QuickBooks own a route at that path.
*/
function oauthCallbackPath(providerId: string, sources: OAuthDocsSources): string | null {
const callbackPath = `/api/auth/oauth2/callback/${providerId}`
const registered =
readSourceFile(CONNECTOR_PROVIDERS_PATH).includes(`${callbackPath}\``) ||
providerId in sources.salesforceLoginHosts ||
fs.existsSync(path.join(OAUTH_CALLBACK_ROUTES_PATH, providerId, 'route.ts'))
return registered ? callbackPath : null
}
/**
* The grant an environment-bound Dataverse credential requests, with the
* environment's resource scope shown as a placeholder.
*/
function environmentBoundDataverseScopes(sources: OAuthDocsSources): string[] {
const sampleOrigin = new URL(
sources.getMicrosoftDataverseOAuthScopes(DATAVERSE_SAMPLE_ENVIRONMENT)[3]
).origin
return sources
.getMicrosoftDataverseOAuthScopes(DATAVERSE_SAMPLE_ENVIRONMENT)
.map((scope) => (scope.startsWith(sampleOrigin) ? DATAVERSE_ENVIRONMENT_SCOPE : scope))
}
function scopeTable(
scopes: readonly string[],
describe: (scope: string) => string,
header = 'Description'
): string {
const rows = scopes.map((scope) => `| \`${scope}\` | ${escapeMdxCell(describe(scope))} |`)
return `| Scope | ${header} |\n| ----- | ${'-'.repeat(header.length)} |\n${rows.join('\n')}\n\n`
}
/**
* Builds the "## OAuth Scopes" section for an OAuth integration: the redirect
* URI and environment variables a self-hoster registers, every scope the
* connect flow requests, scopes requested only behind a deployment flag, and
* scopes that actions restricted to a bot or service-account credential need
* beyond the OAuth grant.
*/
async function buildOAuthScopesSection(
{ serviceId: oauthServiceId, environmentBound }: OAuthCredentialSource,
toolIds: readonly string[]
): Promise<string> {
const sources = await loadOAuthDocsSources()
const service = sources.getServiceConfigByServiceId(oauthServiceId)
if (!service || service.authType === 'service_account') return ''
const dataverse = service.providerId === 'microsoft-dataverse'
const describe = (scope: string) => {
if (scope === DATAVERSE_ENVIRONMENT_SCOPE) {
return 'Access your Dataverse environment through the permissions granted to the app registration'
}
const description = sources.getScopeDescription(scope, service.providerId)
return description === scope ? '' : description
}
// Resolved with the flag on so the page does not depend on the generating environment.
const gatedScopes =
service.providerId === 'slack' ? sources.getSlackApprovalGatedScopes(true) : []
const gated = new Set(gatedScopes)
const requested =
dataverse && environmentBound
? environmentBoundDataverseScopes(sources)
: service.scopes.filter((scope) => !gated.has(scope))
const granted = new Set(requested)
let section = `${OAUTH_SCOPES_HEADING}\n\n`
const article = /^[aeiou]/i.test(service.name) ? 'an' : 'a'
section += requested.length
? `Sim requests these scopes when someone connects ${article} ${service.name} account. `
: `Sim requests no OAuth scopes for ${service.name}. `
section += `For a self-hosted deployment, register your own app with the provider using the settings below. See [Integrations and OAuth](${SELF_HOSTING_OAUTH_GUIDE}) for setup.\n\n`
const providerIds = [service.providerId, ...(service.additionalProviderIds ?? [])]
const redirects = providerIds.flatMap((providerId) => {
const callbackPath = oauthCallbackPath(providerId, sources)
if (!callbackPath) return []
const label = providerIds.length > 1 ? service.providerIdLabels?.[providerId] : undefined
const uri = `\`<NEXT_PUBLIC_APP_URL>${callbackPath}\``
return [label ? `${escapeMdxCell(label)}: ${uri}` : uri]
})
const envFields = sources.getOAuthClientCapabilityFields(oauthServiceId)
const settings: string[] = []
if (redirects.length) settings.push(`| Redirect URI | ${redirects.join('<br />')} |`)
if (envFields?.length) {
settings.push(
`| Environment variables | ${envFields.map((field) => `\`${field}\``).join(', ')} |`
)
} else if (service.clientConfiguration) {
settings.push(
`| Client credentials | Entered in Sim when connecting an account, not set as environment variables |`
)
}
if (settings.length) {
section += `| Setting | Value |\n| ------- | ----- |\n${settings.join('\n')}\n\n`
}
if (requested.length) section += scopeTable(requested, describe)
if (dataverse) {
if (environmentBound) {
section += `Each connection is bound to one environment. \`<environment-url>\` is that environment's Web API origin, so the scope looks like \`https://yourorg.api.crm.dynamics.com/.default\`. `
}
section +=
'In Microsoft Entra, add the **Dynamics CRM** API with the delegated `user_impersonation` permission to the app registration.\n\n'
}
if (gatedScopes.length) {
section +=
'Requested only when `SLACK_EXTENDED_SCOPES` and `NEXT_PUBLIC_SLACK_EXTENDED_SCOPES` are enabled. Enable them only after Slack approves the app for these scopes:\n\n'
section += scopeTable(gatedScopes, describe)
}
const metadata = await loadToolMetadata()
const displayNames = await loadToolDisplayNames()
const botOnlyScopes = new Map<string, string[]>()
for (const toolId of toolIds) {
const oauth = metadata[toolId]?.oauth
if (oauth?.credentialKind !== 'service-account') continue
for (const scope of oauth.requiredScopes ?? []) {
if (granted.has(scope) || sources.isScopeSatisfiedBy(scope, granted)) continue
const actions = botOnlyScopes.get(scope) ?? []
const name = displayNames.get(toolId) ?? stripVersionSuffix(toolId)
if (!actions.includes(name)) actions.push(name)
botOnlyScopes.set(scope, actions)
}
}
if (botOnlyScopes.size) {
section +=
'Some actions run only on a bot or service-account credential, never the OAuth connection. That credential needs these additional scopes:\n\n'
section += scopeTable(
[...botOnlyScopes.keys()].sort(compareCatalogNames),
(scope) => botOnlyScopes.get(scope)?.join(', ') ?? '',
'Actions'
)
}
return section
}
/**
* Compute the canonical set of stripped block types that should have a
* `docs/tools/*.mdx` file — namely every visible `category: 'tools'` block
* (matching the writer filter at the top of this script). Any existing MDX
* not in this set is stale and gets cleaned up.
*
* Uses `extractAllBlockConfigs` so spread-inherited fields (e.g. a V2 that
* spreads `...GmailBlock` and inherits `category: 'tools'`) are resolved the
* same way the writer resolves them. `stripVersionSuffix` ensures V1 and V2
* map to the same doc filename — alphabetical glob order means the newest
* version naturally wins for both generation and cleanup.
*/
async function getCanonicalToolDocNames(): Promise<Set<string>> {
const validToolDocs = new Set<string>()
const blockFiles = (await sourceGlob(`${BLOCKS_PATH}/*.ts`)).sort()
for (const blockFile of blockFiles) {
const configs = blockConfigsForFile(blockFile)
for (const config of configs) {
// Match the writer filter: integration blocks, the documented
// native-resource blocks (category 'blocks'), and trigger-only service
// blocks (category 'triggers') whose pages the trigger pass writes.
const stripped = config.type ? stripVersionSuffix(config.type) : ''
const isDocumentedResource = NATIVE_RESOURCE_BLOCK_TYPES.has(stripped)
const isTriggerService =
config.category === 'triggers' &&
!config.hideFromToolbar &&
stripped !== 'sim_workspace_event'
if (!isIntegrationBlock(config) && !isDocumentedResource && !isTriggerService) continue
validToolDocs.add(stripped)
}
}
return validToolDocs
}
/**
* Remove any `docs/tools/*.mdx` that no longer corresponds to a visible
* `category: 'tools'` block — covers both hidden blocks and blocks that
* have been re-categorized to `'blocks'` / `'triggers'`. Keeps the
* tools/ docs directory in lockstep with the canonical block registry.
*/
function cleanupStaleToolDocs(validToolDocs: Set<string>): void {
console.log('Cleaning up stale tool docs...')
const existingDocs = fs
.readdirSync(DOCS_OUTPUT_PATH)
.filter((file: string) => file.endsWith('.mdx'))
let removedCount = 0
let keptForManualContent = 0
for (const docFile of existingDocs) {
const blockType = path.basename(docFile, '.mdx')
if (HANDWRITTEN_INTEGRATION_DOCS.has(blockType)) continue
if (validToolDocs.has(blockType)) continue
const docPath = path.join(DOCS_OUTPUT_PATH, docFile)
// Deleting a page that a later writer re-emits destroys its hand-written
// MANUAL-CONTENT blocks: the writer merges against the file on disk, and a
// deleted file reads as "no manual content". Whenever the two filters
// disagree, keep the prose and let the mismatch be fixed deliberately.
// Gate on what `extractManualContent` can actually recover — a stray or
// unterminated start marker preserves nothing, so it must not pin the page.
const manualSections = extractManualContent(fs.readFileSync(docPath, 'utf-8'))
if (Object.values(manualSections).some((section) => section.length > 0)) {
console.warn(
`⚠ Keeping ${blockType}.mdx: considered stale but holds MANUAL-CONTENT. ` +
`Add it to a doc-emitting set or delete it by hand once the content is migrated.`
)
keptForManualContent++
continue
}
if (CHECK_ONLY) {
wouldDeletePaths.push(docPath)
continue
}
fs.unlinkSync(docPath)
console.log(`✓ Removed stale tool doc: ${blockType}.mdx`)
removedCount++
}
if (keptForManualContent > 0) {
console.log(`⚠ Kept ${keptForManualContent} stale-looking doc(s) holding manual content`)
}
if (removedCount > 0) {
console.log(`✓ Cleaned up ${removedCount} stale tool doc files`)
} else {
console.log('✓ No stale tool docs to clean up')
}
}
// Trigger Documentation Generation
/**
* Format a trigger provider name for display, falling back to Title Case.
*/
function formatTriggerProviderName(provider: string): string {
if (TRIGGER_PROVIDER_DISPLAY_NAMES[provider]) {
return TRIGGER_PROVIDER_DISPLAY_NAMES[provider]
}
return provider.replace(/[-_]/g, ' ').replace(/\b\w/g, (c) => c.toUpperCase())
}
/**
* Escapes MDX-hostile characters in text emitted as a paragraph rather than a table cell.
*
* MDX reads `{` as an expression and `<` as the start of a JSX tag, so a tool description
* like `The copy is named "<original> - copy"` fails the docs build outright with
* "Expected a closing tag for `<original>`". Unlike {@link escapeMdxCell} this leaves
* pipes, parens and brackets alone — those are legal in prose, and escaping them would
* mangle markdown links.
*/
function escapeMdxProse(text: string): string {
return text
.replace(/\{/g, '\\{')
.replace(/\}/g, '\\}')
.replace(/</g, '&lt;')
.replace(/>/g, '&gt;')
}
const markdownCellParser = unified().use(remarkParse).use(remarkGfm)
/** Escape literal reference text without inserting backslashes into GFM autolinks. */
export function escapeMdxCell(text: string): string {
const escaped = text
.replace(/\|/g, '\\|')
.replace(/\{/g, '\\{')
.replace(/\}/g, '\\}')
.replace(/\(/g, '\\(')
.replace(/\)/g, '\\)')
.replace(/\[/g, '\\[')
.replace(/\]/g, '\\]')
.replace(/</g, '&lt;')
.replace(/>/g, '&gt;')
if (!/(?:https?:\/\/|www\.)/i.test(escaped) || !/\\[()[\]]/.test(escaped)) {
return escaped
}
const autolinkRanges: Array<{ start: number; end: number }> = []
visit(markdownCellParser.parse(escaped), 'link', ({ position }) => {
const start = position?.start.offset
const end = position?.end.offset
if (start !== undefined && end !== undefined) autolinkRanges.push({ start, end })
})
return escaped.replace(/\\[()[\]]/g, (escapedCharacter: string, offset: number) =>
autolinkRanges.some(({ start, end }) => offset >= start && offset < end)
? escapedCharacter.slice(1)
: escapedCharacter
)
}
/** Keys a `TriggerOutput` reserves for itself; everything else is a nested property. */
const TRIGGER_OUTPUT_RESERVED_KEYS = new Set([
'type',
'description',
'condition',
'properties',
'items',
])
/**
* Convert a trigger's registry `outputs` into the `{ type, description, properties }`
* shape the docs renderer walks.
*
* A `TriggerOutput` expresses a nested object by simply omitting `type` and holding its
* children as sibling keys (`issue: { id: {...}, title: {...} }`), while the renderer only
* descends into `properties` on a node typed `object`/`json`. Handing it the raw registry
* value would print every nested group as a single `unknown` row with its children dropped.
*
* Classification is unambiguous: a node with a string `type` is a leaf, anything else is a
* group. No registry group carries a string `description` of its own, so a nested property
* genuinely named `description` is preserved rather than mistaken for the group's own text.
*/
function normalizeTriggerOutputs(raw: Record<string, any>): Record<string, any> {
const normalized: Record<string, any> = {}
for (const [key, value] of Object.entries(raw)) {
if (key === 'condition') continue
if (typeof value !== 'object' || value === null) continue
normalized[key] = normalizeTriggerOutputNode(value)
}
return normalized
}
function normalizeTriggerOutputNode(node: Record<string, any>): Record<string, any> {
const type = typeof node.type === 'string' ? node.type : undefined
const description = typeof node.description === 'string' ? node.description : ''
const inlineChildren: Record<string, any> = {}
for (const [key, value] of Object.entries(node)) {
const isOwnDescription = key === 'description' && typeof node.description === 'string'
if (isOwnDescription || (TRIGGER_OUTPUT_RESERVED_KEYS.has(key) && key !== 'description')) {
continue
}
if (typeof value !== 'object' || value === null) continue
inlineChildren[key] = value
}
if (type === undefined) {
return { type: 'object', description, properties: normalizeTriggerOutputs(inlineChildren) }
}
const result: Record<string, any> = { type, description }
if (node.items && typeof node.items === 'object') {
const items = node.items as Record<string, any>
result.items =
items.properties && typeof items.properties === 'object'
? { ...items, properties: normalizeTriggerOutputs(items.properties) }
: items
}
const declaredProperties =
node.properties && typeof node.properties === 'object' ? node.properties : undefined
if (declaredProperties || Object.keys(inlineChildren).length > 0) {
result.properties = normalizeTriggerOutputs({ ...declaredProperties, ...inlineChildren })
}
return result
}
/**
* A trigger's user-facing configuration fields, in declaration order.
*
* Read from the evaluated registry rather than parsed out of source. Static parsing silently
* dropped every field whose builder assembled its array imperatively or took a description as
* a parameter — that is how all ten Jira triggers lost `webhookSecret` and `jqlFilter` — and
* a regex that recovered them also invented `fieldFilters` on the nine triggers that never
* declare it. Wrong config is worse than absent config, so this evaluates the real objects.
*/
function triggerConfigFields(trigger: RegistryTrigger | undefined): TriggerConfigField[] {
const fields: TriggerConfigField[] = []
for (const subBlock of trigger?.subBlocks ?? []) {
if (!subBlock.id || subBlock.hidden === true || TRIGGER_UI_ONLY_IDS.has(subBlock.id)) continue
if (subBlock.type === 'text' || subBlock.readOnly === true) continue
fields.push({
id: subBlock.id,
title: subBlock.title ?? subBlock.id,
type: subBlock.type ?? 'short-input',
// Only an unconditional `true` counts. `required` is also allowed to be a
// condition object (`{ field, value }`), which makes the field required for
// some configurations and not others — "No" is the honest summary there.
required: subBlock.required === true,
placeholder: typeof subBlock.placeholder === 'string' ? subBlock.placeholder : undefined,
// Falling back to the title keeps oauth-input and other description-less
// fields from rendering an empty cell.
description: subBlock.description ?? subBlock.title,
})
}
return fields
}
/**
* Build the full trigger registry: id → TriggerFullInfo.
* Parses every trigger source file for config fields and output schemas.
*/
async function buildFullTriggerRegistry(): Promise<Map<string, TriggerFullInfo>> {
const registry = new Map<string, TriggerFullInfo>()
const SKIP = new Set(['index.ts', 'registry.ts', 'types.ts', 'constants.ts', 'utils.ts'])
const triggerFiles = (await sourceGlob(`${TRIGGERS_PATH}/**/*.ts`)).filter(
(f) => !SKIP.has(path.basename(f)) && !f.includes('.test.')
)
const registryTriggers = await loadTriggerRegistry()
for (const file of triggerFiles) {
try {
const content = readSourceFile(file)
const exportRegex = /export\s+const\s+\w+\s*:\s*TriggerConfig\s*=\s*\{/g
let exportMatch: RegExpExecArray | null
const exportStarts: number[] = []
while ((exportMatch = exportRegex.exec(content)) !== null) {
exportStarts.push(exportMatch.index)
}
const segments =
exportStarts.length > 0
? exportStarts.map((start, i) => content.substring(start, exportStarts[i + 1]))
: [content]
for (const segment of segments) {
const idMatch = /\bid\s*:\s*['"]([^'"]+)['"]/.exec(segment)
const nameMatch = /\bname\s*:\s*['"]([^'"]+)['"]/.exec(segment)
const providerMatch = /\bprovider\s*:\s*['"]([^'"]+)['"]/.exec(segment)
if (!idMatch || !nameMatch || !providerMatch) continue
// Deprecated triggers stay registered for existing workflows but are
// excluded from generated documentation.
if (/\bdeprecated\s*:\s*true/.test(segment)) continue
const polling = /\bpolling\s*:\s*true/.test(segment)
const registryTrigger = registryTriggers[idMatch[1]]
registry.set(idMatch[1], {
id: idMatch[1],
name: nameMatch[1],
description:
registryTrigger?.description ??
extractStringPropertyFromContent(segment, 'description', true) ??
'',
provider: providerMatch[1],
polling,
outputs: normalizeTriggerOutputs(registryTrigger?.outputs ?? {}),
configFields: triggerConfigFields(registryTrigger),
})
}
const factoryExportRegex =
/export\s+const\s+\w+(?:\s*:\s*TriggerConfig)?\s*=\s*\w+\s*\(\s*['"]([^'"]+)['"]/g
let factoryExportMatch: RegExpExecArray | null
while ((factoryExportMatch = factoryExportRegex.exec(content)) !== null) {
const trigger = registryTriggers[factoryExportMatch[1]]
if (
!trigger?.id ||
!trigger.name ||
!trigger.provider ||
trigger.deprecated ||
registry.has(trigger.id)
) {
continue
}
registry.set(trigger.id, {
id: trigger.id,
name: trigger.name,
description: trigger.description ?? '',
provider: trigger.provider,
polling: trigger.polling === true,
outputs: normalizeTriggerOutputs(trigger.outputs ?? {}),
configFields: triggerConfigFields(trigger),
})
}
} catch {
// skip unreadable files silently
}
}
console.log(`✓ Loaded full config for ${registry.size} triggers`)
return registry
}
/**
* Return the numeric version suffix of a trigger ID (e.g. `_v2` → 2, none → 1).
* Used to prefer the latest version when the same trigger name has v1 and v2 variants.
*/
function triggerVersionOrdinal(id: string): number {
const m = /_v(\d+)$/.exec(id)
return m ? Number.parseInt(m[1], 10) : 1
}
/**
* Group triggers by provider; triggers within each group are sorted alphabetically.
* When multiple triggers share the same display name (e.g. v1 + v2 of the same event),
* only the highest-version variant is kept so docs don't show duplicate sections.
*/
function groupTriggersByProvider(
registry: Map<string, TriggerFullInfo>
): Map<string, TriggerFullInfo[]> {
const groups = new Map<string, TriggerFullInfo[]>()
for (const trigger of registry.values()) {
const bucket = groups.get(trigger.provider) ?? []
bucket.push(trigger)
groups.set(trigger.provider, bucket)
}
for (const [provider, triggers] of groups) {
// Deduplicate by name: keep the highest-versioned trigger for each display name
const byName = new Map<string, TriggerFullInfo>()
for (const trigger of triggers) {
const existing = byName.get(trigger.name)
if (!existing || triggerVersionOrdinal(trigger.id) > triggerVersionOrdinal(existing.id)) {
byName.set(trigger.name, trigger)
}
}
groups.set(
provider,
[...byName.values()].sort((a, b) => compareCatalogNames(a.name, b.name))
)
}
return groups
}
/**
* Map subBlock UI type identifiers to semantic data types for documentation.
* Users care about the data type (string/boolean/number), not the UI widget.
*/
const SUBBLOCK_TYPE_TO_SEMANTIC: Record<string, string> = {
'short-input': 'string',
'long-input': 'string',
dropdown: 'string',
switch: 'boolean',
slider: 'number',
'oauth-input': 'string',
code: 'string',
'file-upload': 'string',
'model-fallback-list': 'json',
text: 'string',
}
function toSemanticType(uiType: string): string {
return SUBBLOCK_TYPE_TO_SEMANTIC[uiType] ?? uiType
}
/**
* Build the "## Triggers" section for an integration page. A trigger is a block
* that starts a workflow, so this is appended to the service's actions page (or
* used as the body of a trigger-only service page).
*/
function buildTriggersSection(triggers: TriggerFullInfo[]): string {
const allPolling = triggers.every((t) => t.polling)
const mixedTypes = triggers.some((t) => t.polling) && triggers.some((t) => !t.polling)
let typeNote = ''
if (allPolling) {
typeNote = '\nThese triggers poll for new data on a schedule.\n'
} else if (mixedTypes) {
typeNote = '\nSome triggers poll on a schedule; others receive webhooks from the service.\n'
}
let triggersSection = ''
for (const trigger of triggers) {
let configSection = ''
if (trigger.configFields.length > 0) {
configSection = '#### Configuration\n\n'
configSection += '| Parameter | Type | Required | Description |\n'
configSection += '| --------- | ---- | -------- | ----------- |\n'
for (const field of trigger.configFields) {
const type = toSemanticType(field.type)
const desc = escapeMdxCell(field.description ?? field.placeholder ?? '')
configSection += `| \`${field.id}\` | ${type} | ${field.required ? 'Yes' : 'No'} | ${desc} |\n`
}
configSection += '\n'
}
let outputSection = ''
if (Object.keys(trigger.outputs).length > 0) {
outputSection = '#### Output\n\n'
outputSection += '| Parameter | Type | Description |\n'
outputSection += '| --------- | ---- | ----------- |\n'
outputSection += formatOutputStructure(trigger.outputs)
outputSection += '\n'
}
const fields = configSection + outputSection
triggersSection += `### ${trigger.name}\n\n`
const escapedTriggerDescription = escapeMdxProse(trigger.description)
triggersSection += `${escapedTriggerDescription}\n\n`
triggersSection +=
fields.length > 0
? `<ReferenceDetails label="Configuration and outputs">\n\n${fields}</ReferenceDetails>\n\n`
: fields
}
return `## Triggers
A trigger starts a workflow when a supported event occurs in this service.
${typeNote}
${triggersSection}`
}
function withReferenceDetailsImport(markdown: string): string {
if (!markdown.includes('<ReferenceDetails') || markdown.includes('import { ReferenceDetails }')) {
return markdown
}
return markdown.replace(
/^(---\n[\s\S]*?\n---\n)/,
'$1\nimport { ReferenceDetails } from "@/components/ui/reference-details"\n'
)
}
/** Standalone page for a trigger-only service (no actions block). */
function generateTriggerProviderDoc(
provider: string,
triggers: TriggerFullInfo[],
blockType: string,
providerColor: string
): string {
const providerName = formatTriggerProviderName(provider)
return `---
title: ${providerName}
description: Start workflows from ${providerName} events
integration:
type: ${JSON.stringify(blockType)}
color: ${JSON.stringify(providerColor)}
---
${buildTriggersSection(triggers)}`
}
/**
* Build a map of block-type → bgColor from all block definitions.
* Used to pick provider colors for integration headers on trigger pages.
*/
async function buildProviderColorMap(): Promise<Map<string, string>> {
const colorMap = new Map<string, string>()
const blockFiles = (await sourceGlob(`${BLOCKS_PATH}/*.ts`)).sort()
for (const blockFile of blockFiles) {
const configs = blockConfigsForFile(blockFile)
for (const config of configs) {
if (config.bgColor && config.type) {
const baseType = stripVersionSuffix(config.type)
if (!colorMap.has(baseType)) colorMap.set(baseType, config.bgColor)
}
}
}
return colorMap
}
/**
* Generate one MDX file per trigger provider and update the sidebar meta.json.
* Hand-written docs (HANDWRITTEN_TRIGGER_DOCS) are never touched.
*/
/**
* Trigger ids that every hosting block gates behind `preview: true`.
*
* Blocks declare the triggers they expose via `triggers.available`. A trigger
* listed only by preview blocks inherits their gate, while triggers no block
* claims are left alone because standalone webhook providers are legitimately
* unlisted and must keep their pages.
*/
async function collectPreviewOnlyTriggerIds(): Promise<Set<string>> {
const listedByReleased = new Set<string>()
const listedByPreview = new Set<string>()
const blockFiles = (await sourceGlob(`${BLOCKS_PATH}/*.ts`)).sort()
for (const blockFile of blockFiles) {
const fileContent = readSourceFile(blockFile)
for (const declaration of blockDeclarations(fileContent)) {
const startIndex = declaration.start
const endIndex = declaration.end
if (endIndex === -1) continue
const blockContent = fileContent.substring(startIndex, endIndex)
const available = extractArrayPropertyFromContent(blockContent, 'available')
if (!available?.length) continue
const target = isPreviewSource(blockContent) ? listedByPreview : listedByReleased
for (const triggerId of available) target.add(triggerId)
}
}
return new Set([...listedByPreview].filter((id) => !listedByReleased.has(id)))
}
async function generateAllTriggerDocs(): Promise<void> {
try {
console.log('Generating trigger documentation...')
if (!fs.existsSync(TRIGGER_DOCS_OUTPUT_PATH)) {
fs.mkdirSync(TRIGGER_DOCS_OUTPUT_PATH, { recursive: true })
}
const fullRegistry = await buildFullTriggerRegistry()
// A trigger reachable only through a preview-gated block is as unreleased
// as that block — documenting it publishes an unshipped surface (and, when
// its provider has no released block, mints a whole page for it).
const previewOnly = await collectPreviewOnlyTriggerIds()
for (const triggerId of previewOnly) {
if (fullRegistry.delete(triggerId)) {
console.log(`Skipping trigger ${triggerId} — only hosted by a preview-gated block`)
}
}
const grouped = groupTriggersByProvider(fullRegistry)
const colorMap = await buildProviderColorMap()
const generatedProviders: string[] = []
for (const [provider, triggers] of grouped) {
if (SKIP_TRIGGER_PROVIDERS.has(provider)) {
console.log(`Skipping trigger provider: ${provider}`)
continue
}
// The trigger lives on the same per-service integration page as the
// service's actions (provider ≠ block type for a few services).
const blockType = PROVIDER_TO_BLOCK_TYPE[provider] ?? provider
const outputFilePath = path.join(DOCS_OUTPUT_PATH, `${blockType}.mdx`)
const baseName = path.basename(outputFilePath, '.mdx')
if (HANDWRITTEN_INTEGRATION_DOCS.has(baseName) || HANDWRITTEN_TRIGGER_DOCS.has(baseName)) {
console.log(`Skipping ${provider} — hand-written page`)
continue
}
const existing = readGeneratedFile(outputFilePath)
if (existing?.includes('\n## Actions')) {
if (!existing.includes('\n## Triggers')) {
const triggersSection = buildTriggersSection(triggers)
const insertionIndexes = [
existing.indexOf('\n## JSON input schemas [#json-input-schemas]\n'),
existing.indexOf('\n{/* MANUAL-CONTENT-START:notes */}'),
].filter((index) => index >= 0)
const insertionIndex = insertionIndexes.length
? Math.min(...insertionIndexes)
: existing.length
emitGeneratedFile(
outputFilePath,
withReferenceDetailsImport(
`${existing.slice(0, insertionIndex)}\n${triggersSection}${existing.slice(insertionIndex)}`
)
)
}
} else {
// Trigger-only service (no actions block) — (re)write the standalone page,
// preserving manual content from the previous run. Cleanup spares these
// pages (category 'triggers' blocks are in the canonical set).
const providerColor = colorMap.get(blockType) ?? '#6B7280'
const markdown = generateTriggerProviderDoc(provider, triggers, blockType, providerColor)
const rawSections = existing ? extractManualContent(existing) : {}
const manualSections = Object.fromEntries(
Object.entries(rawSections).filter(([, v]) => v.length > 0)
)
const finalContent =
Object.keys(manualSections).length > 0
? mergeWithManualContent(markdown, existing, manualSections)
: markdown
emitGeneratedFile(outputFilePath, withReferenceDetailsImport(finalContent))
}
generatedProviders.push(blockType)
if (!CHECK_ONLY) {
console.log(
`✓ Triggers for ${formatTriggerProviderName(provider)} (${triggers.length} trigger${triggers.length === 1 ? '' : 's'})`
)
}
}
console.log(`✓ Trigger sections merged into ${generatedProviders.length} integration pages`)
} catch (error) {
console.error('Error generating trigger documentation:', error)
}
}
async function generateAllBlockDocs() {
try {
const blockFiles = (await sourceGlob(`${BLOCKS_PATH}/*.ts`)).sort()
copyIconsFile()
const {
docs: docsIconMapping,
visible: visibleIconMapping,
coreBlockTypes,
} = await generateIconMappings()
writeIconMapping(docsIconMapping, coreBlockTypes)
await writeIntegrationsJson(visibleIconMapping)
writeIntegrationsIconMapping(visibleIconMapping)
// Compute the canonical set of tool docs and clean up anything stale —
// covers hidden blocks AND blocks re-categorized away from `'tools'`.
const validToolDocs = await getCanonicalToolDocNames()
cleanupStaleToolDocs(validToolDocs)
for (const blockFile of blockFiles) {
await generateBlockDoc(blockFile)
}
// Merge trigger sections into the per-service pages (and write trigger-only pages)
await generateAllTriggerDocs()
// Write the integrations meta after both passes so trigger-only pages are included
updateMetaJson()
return true
} catch (error) {
console.error('Error generating documentation:', error)
return false
}
}
function updateMetaJson() {
const metaJsonPath = path.join(DOCS_OUTPUT_PATH, 'meta.json')
const blockFiles = fs
.readdirSync(DOCS_OUTPUT_PATH)
.filter((file: string) => file.endsWith('.mdx'))
.map((file: string) => path.basename(file, '.mdx'))
/** Fumadocs uses an unlisted index as the folder link; listing it creates a duplicate child. */
const items = blockFiles
.filter((file: string) => file !== 'index' && !(file in integrationNavigation.redirects))
.sort()
const metaJson = {
pages: items,
}
emitGeneratedFile(metaJsonPath, `${JSON.stringify(metaJson, null, 2)}\n`)
if (!CHECK_ONLY) console.log(`Updated meta.json with ${items.length} entries`)
}
if (import.meta.main) {
CHECK_ONLY = process.argv.includes('--check')
const metadataProcess = Bun.spawn(
['bun', path.join(__dirname, 'sync-tool-metadata.ts'), ...(CHECK_ONLY ? ['--check'] : [])],
{ cwd: rootDir, stdout: 'inherit', stderr: 'inherit' }
)
const metadataExitCode = await metadataProcess.exited
if (metadataExitCode !== 0) process.exit(metadataExitCode)
console.log(
CHECK_ONLY
? 'Checking generated documentation freshness...'
: 'Starting documentation generator...'
)
generateAllBlockDocs()
.then((success) => {
if (!success) {
console.error('Documentation generation failed')
process.exit(1)
}
if (CHECK_ONLY) {
const genuinelyDeleted = wouldDeletePaths
.filter((docPath) => !emittedByPath.has(docPath))
.map(
(docPath) =>
`${path.relative(rootDir, docPath)} (stale page — regeneration would delete it)`
)
const stale = [...collectStaleEmissions(), ...staleArtifacts, ...genuinelyDeleted]
if (stale.length > 0) {
console.error(
`Generated integration docs are stale:\n- ${stale.join('\n- ')}\n` +
'Run `bun run scripts/generate-docs.ts` and commit the result.'
)
process.exit(1)
}
console.log('✓ Generated integration docs are in sync')
process.exit(0)
}
console.log('Documentation generation completed successfully')
process.exit(0)
})
.catch((error) => {
console.error('Fatal error:', error)
process.exit(1)
})
}