feat(artifact): add run-scoped on-demand attachment reads

This commit is contained in:
huanghuoguoguo committed 2026-11-16 21:24:36 +08:00
1 parent b01af050ff
commit 33dd118872
67 files changed
+2173 -178

No files matched your search

+15 -2
View File
@@ -11,16 +11,27 @@ Only authenticated accounts and API keys are accepted. Each operation checks the
| `artifact.create` | `POST /api/v1/artifacts` (multipart `file`, optional `ttl_seconds`) | `resource.manage` |
| `artifact.get` | `GET /api/v1/artifacts/{id}` | `resource.view` |
| `artifact.read` | `GET /api/v1/artifacts/{id}/content?max_bytes=10485760` | `resource.view` |
| `artifact.read_content` | `GET /api/v1/artifacts/{id}/read` | `resource.view` |
| `artifact.retain` | `POST /api/v1/artifacts/{id}/retain` | `resource.manage` |
| `artifact.release` | `POST /api/v1/artifacts/{id}/release` | `resource.manage` |
| `artifact.delete` | `DELETE /api/v1/artifacts/{id}` | `resource.manage` |
HTTP and MCP execute the same ApplicationAPI operations and Artifact service. HTTP downloads use attachment disposition, `nosniff`, `private, no-store` and a digest ETag. The API provides complete bounded reads, not byte ranges or partial text extraction: an object larger than `max_bytes` is rejected. Uploads and reads are limited to 10 MiB. Field validation diagnostics retain their technical wording; owned business errors use the Workspace message language.
HTTP and MCP execute the same ApplicationAPI operations and Artifact service. HTTP downloads use attachment disposition, `nosniff`, `private, no-store` and a digest ETag. Raw downloads return the complete bounded object: an object larger than `max_bytes` is rejected. Structured content reads support the text and PDF ranges described below. Uploads and source reads are limited to 10 MiB. Field validation diagnostics retain their technical wording; owned business errors use the Workspace message language.
MCP tools are `create_artifact`, `get_artifact`, `read_artifact`, `retain_artifact`, `release_artifact` and `delete_artifact`. Creation accepts strict base64 with a pre-decode encoded-length limit; reads return base64 plus metadata. These are file transfer APIs. Structured image/PDF model consumption, user-attachment routing and Runner file conversion are later stages of #3051.
MCP tools are `create_artifact`, `get_artifact`, `read_artifact`, `read_artifact_content`, `retain_artifact`, `release_artifact` and `delete_artifact`. Creation accepts strict base64 with a pre-decode encoded-length limit; raw reads return base64 plus metadata. Runner file conversion and fixed-version Skill resources remain later stages of #3051.
The existing file upload and Runner `file_id` APIs retain their contracts. No existing storage key or run handle is accepted as an Artifact ID. The separate `lbctl` client can continue using its existing routes; it has no Artifact commands in this phase.
`artifact.read_content` accepts `mode`, `offset`, `max_chars`, `page_start` and `page_end` and returns structured content elements. MCP `read_artifact_content` maps them to native text/image/embedded-PDF blocks without placing encoded bytes into a text result. Both use the same authorized reader as execution attachment reads.
## Agent attachment reads
Inbound platform inputs render text markers and attachment metadata only, independently of model vision support. `InputMedia` and the Host query registry retain original components; `prepare_input_files` also registers media found only in event contents before replacing those contents with markers. Runner descriptors contain no inline bytes, Host paths or source URLs. No eager download is introduced. The existing raw `read_input_attachment` API and new structured `read_input_attachment_content` operation share the frozen `input_attachments` run grant, plugin installation identity and live run checks.
The first authorized platform attachment read creates a run-owned Artifact through the shared service. Only the Host's current-input registry may resolve an `att-N` ref; knowing another asset ID is not a grant. Run-owned assets do not become account-owned or publicly accessible. Agent debug accepts at most 20 distinct `artifact_ids`: it checks metadata with the original RequestContext before execution and stores the authorized set plus context in Host-only query fields. Synthetic event actors cannot select the asset owner. The separate Workspace management assistant still accepts text/approval turns only.
Structured reads use `auto`, `text`, `native` or `pages`, zero-based text `offset`, `max_chars` (8192 default, 16384 maximum), and one-based PDF `page_start`/`page_end` (at most ten pages). Results identify truncation and continuation offsets/pages. Auto mode returns structured images and UTF-8 text, and extracts PDF text. PDF text explicitly excludes images, layout and OCR. Native mode returns structured image/PDF content for a supporting provider; unsupported models fail visibly. Pages mode currently fails explicitly rather than implying that page rendering occurred. Raw bytes and structured content are returned only after source authorization; metadata and retained history do not contain base64 payloads.
## Lifecycle
Creation produces a temporary asset with a default 24-hour lifetime; callers may choose 1 second through 7 days. Retain creates a persistent caller-owned pin. Retain changes retention only, never authorization. Release removes the caller's pin and restores a 24-hour temporary grace period; repeated release does not keep extending the lifetime. Delete refuses an asset protected by a pin. Expired assets cannot be read or revived through retain.
@@ -33,4 +44,6 @@ Errors: missing, deleted and unauthorized asset IDs return 404 without disclosin
## Acceptance
Native PDF uses LiteLLM capability metadata or the administrator's explicit `pdf_input` model ability for a provider-confirmed alias. Unknown capability without that declaration fails explicitly. Content reads charge source materialization once per attachment, and separately cap cumulative returned content at the run read budget; the legacy raw-byte API continues to charge each full read.
Exercise authenticated HTTP and MCP against real persistence: byte-for-byte upload/download and digest, principal and Workspace isolation, limited permissions, expiry, retain/release/delete, malformed and excessive inputs, cancellation, GC concurrency and cleanup failure. PostgreSQL/MinIO acceptance includes restricted-role RLS and a second Core/restart reading persistent assets without shared cache roots. See [cross-component testing](../CROSS_COMPONENT_TESTING.md). A successful base64 transfer does not establish model understanding of an image or PDF.
+22 -2
View File
@@ -12,6 +12,7 @@ Use the normal Workspace HTTP authentication. Reads require `resource.view`; cre
| Upload | `POST /api/v1/artifacts`, multipart `file`, optional `ttl_seconds` |
| Metadata | `GET /api/v1/artifacts/{id}` |
| Download | `GET /api/v1/artifacts/{id}/content`, optional `max_bytes` |
| Read structured content | `GET /api/v1/artifacts/{id}/read`, optional `mode`, `offset`, `max_chars`, `page_start`, `page_end` |
| Retain | `POST /api/v1/artifacts/{id}/retain` |
| Release | `POST /api/v1/artifacts/{id}/release` |
| Delete | `DELETE /api/v1/artifacts/{id}` |
@@ -26,6 +27,25 @@ Deletion blocks subsequent access; maintenance reclaims the underlying bytes. An
## MCP
The built-in `/mcp` server exposes `create_artifact`, `get_artifact`, `read_artifact`, `retain_artifact`, `release_artifact` and `delete_artifact` with the same ownership and permission checks. Creation takes `filename`, `content_base64`, optional `mime_type` and `ttl_seconds`. Reading takes `artifact_id` and optional `max_bytes`, returning base64 plus metadata.
Native PDF input requires trusted model capability information. For an alias whose provider has confirmed PDF support but which the capability catalog does not recognize, an administrator can declare `pdf_input` in the model's existing `abilities` field. Undeclared, unknown capability rejects native PDF input; text mode remains available.
These tools transfer files. They do not extract text, render PDF pages or establish that a model understands an image. The initial API is not available through plugin Workspace grants, plugin typed methods, default model tools or `lbctl`. Existing upload and run `file_id` interfaces continue to use their existing contracts.
The built-in `/mcp` server exposes `create_artifact`, `get_artifact`, `read_artifact`, `read_artifact_content`, `retain_artifact`, `release_artifact` and `delete_artifact` with the same ownership and permission checks. Creation takes `filename`, `content_base64`, optional `mime_type` and `ttl_seconds`. Raw reading takes `artifact_id` and optional `max_bytes`, returning base64 plus metadata. `read_artifact_content` takes the structured-read parameters from the HTTP `/read` endpoint and returns native MCP text, image or embedded-PDF content rather than base64 inside ordinary text. The Workspace operation is `artifact.read_content`.
`read_artifact` transfers the original bytes. Account-owned management operations remain unavailable through plugin Workspace grants, plugin typed methods, default model tools or `lbctl`. Existing upload and run `file_id` interfaces continue to use their existing contracts.
## Agent reads
Runner inputs initially contain attachment references, names, MIME types and sizes. Image/file bytes and source URLs stay in the Host registry; they are not inserted into the initial model message. Current-message and quoted-message attachments retain separate references. The Agent requests content only when needed, under the current run's frozen attachment grant. References from another run do not grant access.
`read_input_attachment_content` returns structured content and supports these modes:
| Mode | Behavior |
| --- | --- |
| `auto` | Read UTF-8 text, return a structured image, or extract PDF text. |
| `text` | Read a bounded character range; PDF results identify the selected pages and exclude visual content. |
| `native` | Return image/PDF content for a model that supports that input; unsupported models report a visible error. |
| `pages` | Page rendering is currently unsupported and returns an explicit error. |
Text uses zero-based `offset` and `max_chars` (default 8192, maximum 16384). Results include `truncated` and `next_offset`; continue from that position rather than treating a partial result as the full file. PDF text uses one-based `page_start` and optional `page_end`, at most ten pages per read, and reports the next page. It does not perform OCR or preserve images/layout. Other binary formats fail explicitly; the original remains downloadable.
Agent debug API requests may include `artifact_ids`, up to 20 distinct IDs from the caller's uploads. The Host checks the authenticated caller's ownership before starting the run and gives that run references only. Supplying an event `actor` does not change ownership. The separate Workspace management assistant does not accept attachment inputs in this phase. The existing raw-byte `read_input_attachment` API remains available to authorized Runner code.
+5 -3
View File
@@ -190,15 +190,15 @@ Each item returned by `get_available_tools()` contains `name`, `description`, `p
### Input attachments
`ctx.input.attachments` describes each image, voice and file of the current message in order of appearance, as metadata only: `ref` (such as `att-1`), `type`, `source` (`message` for the current message, `quote` for a quoted message), `name`, `mime_type`, `size`, the platform `id`, and a directly downloadable `url`. Attachments do not carry file content; the `content` field is deprecated and always empty in current versions.
`ctx.input.attachments` describes each image, voice and file of the current message in order of appearance, as metadata only: `ref` (such as `att-1`), `type`, `source` (`message` for the current message, `quote` for a quoted message), `name`, `mime_type`, `size`, and the platform `id`. The Host keeps source URLs private. Attachments do not carry file content; the `content` field is deprecated and always empty in current versions.
`ctx.input.text` and `ctx.input.contents` place a marker at the position of each media, such as `[attachment att-1: image photo.jpg]`. A quoted message is wrapped in `[quoted message from <sender>]` and `[/quoted message]`. Images still follow their marker as `image_base64` or `image_url` so models can see them.
`ctx.input.text` and `ctx.input.contents` place a marker at the position of each media, such as `[attachment att-1: image photo.jpg]`. A quoted message is wrapped in `[quoted message from <sender>]` and `[/quoted message]`. Images and files are not inlined into the initial model input. Read them explicitly when needed.
| Method | Returns | Description |
| --- | --- | --- |
| `await ctx.read_input_attachment(attachment)` | `InputAttachmentData \| None` | Reads attachment bytes by `ref` and returns `data`, `name`, `mime_type` and `size`. The argument may also be a `ref` string. Returns `None` when LangBot does not offer this API |
Each attachment is limited to 10 MiB and each run may read up to 50 MiB in total. LangBot neither caches the bytes nor writes them to logs. `ctx.context.available_apis.input_attachments` reports whether the API is available. Follow-up messages steered into the current run carry attachment metadata without a `ref`.
Each attachment is limited to 10 MiB and each run may read up to 50 MiB in total. The content-reading API materializes the source as a temporary run-owned Artifact on first use; inline bytes are not written to conversation history or tool events. `ctx.context.available_apis.input_attachments` reports whether the API is available. Follow-up messages steered into the current run carry attachment metadata without a `ref`.
The method requires `langbot-plugin` 0.7.23 or later. To support older LangBot or SDK versions, check that the method exists and fall back to the previous logic:
@@ -210,6 +210,8 @@ if data is None:
... # Older LangBot: use attachment.content, attachment.url or ctx.input.contents
```
`await ctx.read_input_attachment_content(ref, mode="auto", offset=0, max_chars=8192, page_start=1, page_end=None)` returns bounded text or structured media with the same frozen run grant. See [Artifact reading](/en/develop/artifacts) for modes and limits. This new method requires coordinated Core, SDK and Runtime versions; upgrade them and the LocalAgent plugin together.
### Box and attachments
The Runner owns enablement, acquisition, and reuse policy. Use the [LangBot Box APIs](/en/plugin/dev/apis/common#box-sandbox) for resources and these context methods for the current run.
+2
View File
@@ -10,6 +10,8 @@ Every operation has a stable ID, such as `pipeline.list` or `knowledge.base.retr
Artifact operations are currently reserved for authenticated accounts and API keys. They are excluded from plugin grants, wildcard permissions and typed Workspace methods. See the [Artifact file API](/en/develop/artifacts).
This includes `artifact.read_content`, which provides bounded text and structured image/PDF reads. Runner code instead uses the run-bound `read_input_attachment_content` execution API under its existing attachment grant.
## Calling an operation
From any component, use `self.plugin.workspace`. Each operation has a typed method named after its ID:
+23 -1
View File
@@ -12,6 +12,7 @@ Artifact はファイルの不透明な参照とメタデータを提供しま
| アップロード | `POST /api/v1/artifacts`、multipart `file`、任意の `ttl_seconds` |
| メタデータ | `GET /api/v1/artifacts/{id}` |
| ダウンロード | `GET /api/v1/artifacts/{id}/content`、任意の `max_bytes` |
| 構造化読み取り | `GET /api/v1/artifacts/{id}/read`、任意の `mode`、`offset`、`max_chars`、`page_start`、`page_end` |
| 永続保持 | `POST /api/v1/artifacts/{id}/retain` |
| 保持の解放 | `POST /api/v1/artifacts/{id}/release` |
| 削除 | `DELETE /api/v1/artifacts/{id}` |
@@ -26,6 +27,27 @@ Artifact はファイルの不透明な参照とメタデータを提供しま
## MCP
ネイティブPDF入力には信頼できるモデル能力情報が必要です。提供元がPDF対応を確認済みでも能力カタログにない別名の場合、管理者はモデルの既存の `abilities` に `pdf_input` を指定できます。宣言がなく能力が不明な場合、ネイティブPDFは明示的に拒否されますが、テキストモードは利用できます。
組み込み `/mcp` サーバーは、同じ所有者と権限のチェックを適用する `create_artifact`、`get_artifact`、`read_artifact`、`retain_artifact`、`release_artifact`、`delete_artifact` を提供します。作成には `filename`、`content_base64`、任意の `mime_type` と `ttl_seconds` を渡します。読み取りには `artifact_id` と任意の `max_bytes` を渡し、base64 とメタデータを受け取ります。
これらはファイル転送ツールです。テキスト抽出、PDF ページ描画、画像のモデル理解は行いません。初版ではプラグインの Workspace 権限、型付きメソッド、既定のモデルツール、`lbctl` からは利用できません。既存のアップロード API と run の `file_id` API は従来の契約を維持します。
`read_artifact` は元のバイト列を転送します。アカウント所有の管理操作は、引き続きプラグインの Workspace 権限、型付きメソッド、既定のモデルツール、`lbctl` からは利用できません。既存のアップロード API と run の `file_id` API は従来の契約を維持します。
`read_artifact_content` は HTTP `/read` と同じパラメーターを受け付け、通常のテキストに base64 を入れず、MCP のテキスト、画像、埋め込み PDF を返します。Workspace 操作は `artifact.read_content` です。
## Agent による必要時の読み取り
Runner の初期入力には添付参照、名前、MIME 型、サイズのみが含まれます。画像やファイルのバイト列と取得元 URL は Host の実行内レジストリに保持され、最初のモデルメッセージには入りません。現在のメッセージと引用メッセージの添付は別々の参照を持ちます。必要な内容だけを取得し、Host は現在の run に固定された添付権限を確認します。他の run の参照はアクセスを許可しません。
`read_input_attachment_content` は構造化された内容を返します。
| モード | 動作 |
| --- | --- |
| `auto` | UTF-8 テキスト、構造化画像、または PDF 抽出テキストを返します。 |
| `text` | 文字範囲を制限して読み取ります。PDF は対象ページを示し、画像やレイアウトを含みません。 |
| `native` | 対応モデルへ画像や PDF を渡します。非対応モデルでは明示的に失敗します。 |
| `pages` | ページ描画は現在未対応で、明示的なエラーになります。 |
テキストは 0 始まりの `offset` と `max_chars`(既定 8192、最大 16384)を使用します。結果の `truncated` と `next_offset` を確認して続きを読み取ってください。PDF テキストは 1 始まりの `page_start` と任意の `page_end` を使用し、1 回最大 10 ページで、次のページ位置も返します。OCR、画像、ページレイアウトは対象外です。他のバイナリ形式は明示的に拒否され、元のファイルはダウンロードできます。
Agent デバッグ API は、自分がアップロードした異なる ID を最大 20 件まで `artifact_ids` に指定できます。Host が認証済み主体の所有権を確認し、その run に参照だけを渡します。イベントの `actor` で所有者を変更することはできません。独立した Workspace 管理アシスタントの添付入力はこの段階では未対応です。既存の生バイト API `read_input_attachment` は権限のある Runner コード向けに維持されます。
+5 -3
View File
@@ -190,15 +190,15 @@ event_type = ctx.event.event_type
### 入力添付ファイル
`ctx.input.attachments` は、現在のメッセージに含まれる画像・音声・ファイルを出現順に 1 件ずつメタデータとして表します:`ref`(例:`att-1`)、`type`、`source`(現在のメッセージは `message`、引用されたメッセージは `quote`)、`name`、`mime_type`、`size`、プラットフォームの `id`、直接ダウンロードできる `url`。添付ファイルは内容を含みません。`content` フィールドは非推奨で、現行バージョンでは常に空です。
`ctx.input.attachments` は、現在のメッセージに含まれる画像・音声・ファイルを出現順に 1 件ずつメタデータとして表します:`ref`(例:`att-1`)、`type`、`source`(現在のメッセージは `message`、引用されたメッセージは `quote`)、`name`、`mime_type`、`size`、プラットフォームの `id`。元URLはHostだけが保持します。添付ファイルは内容を含みません。`content` フィールドは非推奨で、現行バージョンでは常に空です。
`ctx.input.text` と `ctx.input.contents` には、各メディアの位置に `[attachment att-1: image photo.jpg]` のようなマーカーが入ります。引用されたメッセージは `[quoted message from <送信者>]` と `[/quoted message]` で囲まれます。画像は引き続きマーカーの直後に `image_base64` または `image_url` として置かれ、モデルが直接参照できます。
`ctx.input.text` と `ctx.input.contents` には、各メディアの位置に `[attachment att-1: image photo.jpg]` のようなマーカーが入ります。引用されたメッセージは `[quoted message from <送信者>]` と `[/quoted message]` で囲まれます。画像やファイルは初期モデル入力には含まれず、必要なときに明示的に読み取ります。
| メソッド | 戻り値 | 説明 |
| --- | --- | --- |
| `await ctx.read_input_attachment(attachment)` | `InputAttachmentData \| None` | `ref` で添付ファイルのバイト列を読み取り、`data`、`name`、`mime_type`、`size` を返します。引数には `ref` 文字列も指定できます。LangBot がこの API を提供しない場合は `None` を返します |
添付ファイル 1 件あたりの上限は 10 MiB、1 回の実行で読み取れる合計は 50 MiB です。LangBot は読み取った内容をキャッシュせず、ログにも書き込みません。この API が使えるかどうかは `ctx.context.available_apis.input_attachments` で確認できます。steering で現在の実行に取り込まれた後続メッセージの添付ファイルはメタデータのみで、`ref` を持ちません。
添付ファイル 1 件あたりの上限は 10 MiB、1 回の実行で読み取れる合計は 50 MiB です。内容読み取りAPIは初回に元データをrun専用の一時Artifactとして保存します。バイナリは会話履歴やツールイベントに保存しません。この API が使えるかどうかは `ctx.context.available_apis.input_attachments` で確認できます。steering で現在の実行に取り込まれた後続メッセージの添付ファイルはメタデータのみで、`ref` を持ちません。
このメソッドには `langbot-plugin` 0.7.23 以降が必要です。古い LangBot や SDK にも対応する場合は、メソッドの有無を確認してから従来の処理にフォールバックします:
@@ -210,6 +210,8 @@ if data is None:
... # 古い LangBot:attachment.content、attachment.url、ctx.input.contents を使用
```
`await ctx.read_input_attachment_content(ref, mode="auto", offset=0, max_chars=8192, page_start=1, page_end=None)` は同じrun権限で、制限付きテキストまたは構造化メディアを返します。モードと制限は [Artifactの読み取り](/ja/develop/artifacts) を参照してください。新メソッドには対応するCore、SDK、Runtimeが必要です。LocalAgentと一緒に更新してください。
### Box と添付ファイル
Box の有効化、取得、再利用方針は Runner が決めます。リソースの取得には [LangBot API](/ja/plugin/dev/apis/common#box-サンドボックス)、実行中の操作には以下を使います。
+2
View File
@@ -10,6 +10,8 @@ description: プラグインから Agent、パイプライン、ボット、モ
Artifact 操作は現在、認証済みアカウントと API キー専用です。プラグイン権限、ワイルドカード、型付き Workspace メソッドには含まれません。[Artifact ファイル API](/ja/develop/artifacts) を参照してください。
有界テキストと構造化画像/PDF を読む `artifact.read_content` も同じ境界に従います。Runner は既存の添付権限に制限された、run に結び付く `read_input_attachment_content` 実行 API を使用します。
## 操作を呼び出す
どのコンポーネントからも `self.plugin.workspace` を使います。各操作には ID に対応した型付きメソッドがあります。
+22 -2
View File
@@ -12,6 +12,7 @@ Artifact 为文件提供不透明引用和元信息。账户或 API 密钥只能
| 上传 | `POST /api/v1/artifacts`,multipart `file`,可选 `ttl_seconds` |
| 元信息 | `GET /api/v1/artifacts/{id}` |
| 下载 | `GET /api/v1/artifacts/{id}/content`,可选 `max_bytes` |
| 读取结构化内容 | `GET /api/v1/artifacts/{id}/read`,可选 `mode`、`offset`、`max_chars`、`page_start`、`page_end` |
| 持久保留 | `POST /api/v1/artifacts/{id}/retain` |
| 释放保留 | `POST /api/v1/artifacts/{id}/release` |
| 删除 | `DELETE /api/v1/artifacts/{id}` |
@@ -26,6 +27,25 @@ Artifact 为文件提供不透明引用和元信息。账户或 API 密钥只能
## MCP
内置 `/mcp` 服务提供 `create_artifact`、`get_artifact`、`read_artifact`、`retain_artifact`、`release_artifact` 和 `delete_artifact`,使用相同的归属及权限检查。创建参数为 `filename`、`content_base64`,以及可选的 `mime_type` 和 `ttl_seconds`。读取参数为 `artifact_id` 及可选的 `max_bytes`,返回 base64 和元信息。
原生 PDF 需要可信的模型能力信息;对于已由供应商确认支持、但能力目录尚未识别的模型别名,可由管理员在模型的现有 `abilities` 字段中声明 `pdf_input`。未声明且能力未知时会明确拒绝原生 PDF,仍可使用文本模式。
这些工具用于文件传输,不做文本提取、PDF 页面渲染,也不代表模型已理解图片。初版暂不开放插件工作区授权、插件类型化方法、默认模型工具或 `lbctl` 命令。原有上传接口和运行内 `file_id` 接口继续使用原契约。
内置 `/mcp` 服务提供 `create_artifact`、`get_artifact`、`read_artifact`、`read_artifact_content`、`retain_artifact`、`release_artifact` 和 `delete_artifact`,使用相同的归属及权限检查。创建参数为 `filename`、`content_base64`,以及可选的 `mime_type` 和 `ttl_seconds`。原始读取参数为 `artifact_id` 及可选的 `max_bytes`,返回 base64 和元信息。`read_artifact_content` 沿用 HTTP `/read` 的读取参数,返回原生 MCP 文本、图片或嵌入 PDF 内容,不把 base64 放入普通文本。对应 Workspace 操作为 `artifact.read_content`。
`read_artifact` 传输原始字节。账户资产管理操作仍不开放插件工作区授权、插件类型化方法、默认模型工具或 `lbctl` 命令。原有上传接口和运行内 `file_id` 接口继续使用原契约。
## Agent 按需读取
Runner 初始输入只包含附件引用、名称、MIME 和大小。图片、文件字节和来源 URL 保留在 Host 的运行内注册表中,不直接进入首轮模型消息。当前消息和引用消息中的附件各有独立引用。Agent 需要内容时才调用读取工具,Host 继续检查当前 run 的冻结附件权限;另一个 run 的引用不能授权访问。
`read_input_attachment_content` 返回结构化内容,支持以下模式:
| 模式 | 行为 |
| --- | --- |
| `auto` | 读取 UTF-8 文本、返回结构化图片,或提取 PDF 文本。 |
| `text` | 有界读取字符范围;PDF 结果标明所选页码,明确不包含视觉内容。 |
| `native` | 将图片或 PDF 作为原生输入交给支持的模型;不支持时明确报错。 |
| `pages` | 本阶段尚不支持页面渲染,返回明确错误。 |
文本使用从零开始的 `offset` 和 `max_chars`(默认 8192,最大 16384)。结果带 `truncated` 和 `next_offset`,应从该位置继续读取,不能将截断内容当作全文。PDF 文本使用从一开始的 `page_start` 和可选 `page_end`,每次最多十页,并返回下一页位置;不做 OCR,也不保留图片和页面布局。其他二进制格式明确拒绝,原始文件仍可下载。
Agent 调试 API 的请求可带 `artifact_ids`,最多 20 个不同的自有上传资产 ID。Host 在启动 run 前按真实认证主体检查归属,再给该 run 引用。请求中的事件 `actor` 不会改变资产归属。独立的 Workspace 管理助手本阶段仍不接受附件输入。已有原始字节接口 `read_input_attachment` 继续供有权限的 Runner 代码使用。
+5 -3
View File
@@ -192,15 +192,15 @@ event_type = ctx.event.event_type
### 输入附件
`ctx.input.attachments` 按出现顺序逐一描述当前消息中的图片、语音和文件,只包含元数据:`ref`(如 `att-1`)、`type`、`source`(`message` 为当前消息,`quote` 为被引用的消息)、`name`、`mime_type`、`size`、平台 `id` 和可直接下载的 `url`。附件不携带文件内容,`content` 字段已弃用,当前版本始终为空。
`ctx.input.attachments` 按出现顺序逐一描述当前消息中的图片、语音和文件,只包含元数据:`ref`(如 `att-1`)、`type`、`source`(`message` 为当前消息,`quote` 为被引用的消息)、`name`、`mime_type`、`size`、平台 `id`。来源 URL 由 Host 私有保存。附件不携带文件内容,`content` 字段已弃用,当前版本始终为空。
`ctx.input.text` 和 `ctx.input.contents` 会在每个媒体的原位置插入标记,例如 `[attachment att-1: image photo.jpg]`;被引用的消息包裹在 `[quoted message from <发送者>]` 与 `[/quoted message]` 之间。图片仍以 `image_base64` 或 `image_url` 紧跟在标记之后,供模型直接查看。
`ctx.input.text` 和 `ctx.input.contents` 会在每个媒体的原位置插入标记,例如 `[attachment att-1: image photo.jpg]`;被引用的消息包裹在 `[quoted message from <发送者>]` 与 `[/quoted message]` 之间。图片和文件不会直接进入初始模型输入,需要时再显式读取。
| 方法 | 返回值 | 说明 |
| --- | --- | --- |
| `await ctx.read_input_attachment(attachment)` | `InputAttachmentData \| None` | 按 `ref` 读取附件内容,返回 `data`、`name`、`mime_type` 和 `size`;参数也可以是 `ref` 字符串。LangBot 不提供该接口时返回 `None` |
单个附件最大 10 MiB,每次运行累计最多读取 50 MiB;LangBot 不缓存读取结果,也不把内容写入日志。`ctx.context.available_apis.input_attachments` 表示该接口是否可用。通过 steering 转入当前运行的后续消息,其附件只有元数据,没有 `ref`。
单个附件最大 10 MiB,每次运行累计最多读取 50 MiB;内容读取 API 在首次使用时将原件存为当前 run 的临时 Artifact;二进制不写入对话历史或工具事件。`ctx.context.available_apis.input_attachments` 表示该接口是否可用。通过 steering 转入当前运行的后续消息,其附件只有元数据,没有 `ref`。
该方法需要 `langbot-plugin` 0.7.23 或更高版本。需要兼容旧版 LangBot 或旧版 SDK 时,先判断方法是否存在,再回退到原有逻辑:
@@ -212,6 +212,8 @@ if data is None:
... # 旧版 LangBot:使用 attachment.content、attachment.url 或 ctx.input.contents
```
`await ctx.read_input_attachment_content(ref, mode="auto", offset=0, max_chars=8192, page_start=1, page_end=None)` 使用同一冻结 run 权限,返回有界文本或结构化媒体。模式和限额见 [Artifact 读取](/zh/develop/artifacts)。该新方法要求匹配的 Core、SDK 和 Runtime,应与 LocalAgent 插件一同升级。
### Box 与附件
Box 是否启用、何时创建以及如何复用,由运行器决定。获取状态和申请 Box 使用 [LangBot API](/zh/plugin/dev/apis/common#box-沙箱);当前运行的绑定和附件操作使用下列方法。
+2
View File
@@ -10,6 +10,8 @@ description: 在插件中读取和管理 Agent、流水线、机器人、模型
Artifact 操作目前仅接受已认证账户和 API 密钥,不属于插件可申请的操作、通配授权或工作区类型化方法。参见 [Artifact 文件接口](/zh/develop/artifacts)。
新增 `artifact.read_content` 也遵循这一边界,提供有界文本和结构化图片/PDF 读取。Runner 代码使用运行绑定的 `read_input_attachment_content` 执行 API,继续受原有附件权限约束。
## 调用操作
在任意组件中通过 `self.plugin.workspace` 调用。每个操作都有一个按 ID 命名、带类型标注的方法:
@@ -34,6 +34,7 @@ from pkg.model_calling import (
model_call_error_from_exception,
normalize_llm_call_result,
serialize_tool_result_content,
split_tool_content,
)
from .types import (
@@ -314,16 +315,19 @@ class LangBotToolExecutor:
)
try:
result = await self.local_tools[prepared.request.name](**prepared.parameters)
outcome = self.finalize(prepared, result=result, error=None)
if is_traced_queue:
await results_queue.put(
RunnerResult.tool_call_completed(
run_id,
call_id,
prepared.request.name,
result=result if isinstance(result, dict) else {"value": result},
result=outcome.event_result
if outcome.event_result is not None
else (result if isinstance(result, dict) else {"value": result}),
)
)
return self.finalize(prepared, result=result, error=None)
return outcome
except Exception as exc:
if is_traced_queue:
await results_queue.put(
@@ -360,6 +364,24 @@ class LangBotToolExecutor:
terminate = error is None and _tool_result_terminates(result)
model_result = _strip_tool_runtime_hints(result) if error is None else result
if error is None:
structured = split_tool_content(model_result)
if structured is not None:
text, media = structured
message = build_tool_call_message(
prepared.request.id,
prepared.request.name,
text,
max_result_chars=self.max_result_chars,
)
return ToolExecutionOutcome(
request=prepared.request,
parameters=prepared.parameters,
result=result,
event_result={"type": "langbot_tool_content", "summary": message.content},
message=message,
model_only_messages=[Message(role="user", content=media)],
terminate=terminate,
)
references = extract_tool_result_references(model_result)
if references["file_refs"]:
message, event_result = build_tool_reference_message(
@@ -444,6 +466,12 @@ class LangBotContextHooks(AgentLoopHooks):
async def prepare_model_turn(self, messages: list[Message]) -> list[Message]:
usage_anchor = self._usage_anchor
self._usage_anchor = None
if any(
isinstance(message.content, list)
and any(item.type.startswith(("image", "file")) for item in message.content)
for message in messages
):
usage_anchor = None
assembly = await ContextCompactor(
self.budget,
summarizer=self.summarizer,
@@ -58,6 +58,8 @@ class AgentLoop:
self.tool_executor = tool_executor
self.model_ids = list(model_ids)
self.messages = [message.model_copy(deep=True) for message in messages]
self._model_only_messages: list[Message] = []
self._prepared_model_messages = list(self.messages)
self.tools = list(tools)
self.streaming = streaming
self.max_tool_iterations = max_tool_iterations
@@ -96,14 +98,30 @@ class AgentLoop:
return index, outcome
async def _prepare_model_turn(self) -> None:
messages = await self.hooks.prepare_model_turn(self.messages)
self.messages = [message.model_copy(deep=True) for message in messages]
messages = await self.hooks.prepare_model_turn(self.messages + self._model_only_messages)
self._accept_prepared_messages(messages)
def _accept_prepared_messages(self, messages: list[Message]) -> None:
durable_messages = list(messages)
for media in self._model_only_messages:
try:
durable_messages.remove(media)
except ValueError as exc:
raise ModelCallError(
"Attachment content was removed by context budgeting. Use a smaller attachment or text/page range.",
retryable=False,
) from exc
self._prepared_model_messages = [message.model_copy(deep=True) for message in messages]
self.messages = [message.model_copy(deep=True) for message in durable_messages]
async def _recover_context_overflow(self, error: ModelCallError) -> bool:
messages = await self.hooks.recover_context_overflow(self.messages, error)
messages = await self.hooks.recover_context_overflow(self._prepared_model_messages, error)
if messages is None:
return False
self.messages = [message.model_copy(deep=True) for message in messages]
try:
self._accept_prepared_messages(messages)
except ModelCallError:
return False
return True
def _should_execute_serially(self) -> bool:
@@ -167,6 +185,7 @@ class AgentLoop:
if outcome is None or outcome.message is None:
continue
self.messages.append(outcome.message)
self._model_only_messages.extend(outcome.model_only_messages)
tool_results.append(outcome.message)
yield AgentLoopEvent.message_start(outcome.message)
yield AgentLoopEvent.message_end(outcome.message)
@@ -266,7 +285,7 @@ class AgentLoop:
async with aclosing(
self.model_adapter.stream_turn(
model_ids=turn_model_ids,
messages=self.messages,
messages=self._prepared_model_messages,
tools=self.tools,
visible_prefix=visible_content_prefix,
)
@@ -392,13 +411,13 @@ class AgentLoop:
if committed_model_id is None:
model_turn = await self.model_adapter.invoke_turn(
model_ids=candidate_model_ids,
messages=self.messages,
messages=self._prepared_model_messages,
tools=self.tools,
)
else:
model_turn = await self.model_adapter.invoke_committed_turn(
committed_model_id=committed_model_id,
messages=self.messages,
messages=self._prepared_model_messages,
tools=self.tools,
)
break
@@ -111,6 +111,7 @@ class ToolExecutionOutcome:
event_result: typing.Any = None
error: str | None = None
message: Message | None = None
model_only_messages: list[Message] = field(default_factory=list)
terminate: bool = False
@property
+28 -1
View File
@@ -13,7 +13,7 @@ import uuid
from contextlib import aclosing
from dataclasses import dataclass
from langbot_plugin.api.entities.builtin.provider.message import Message, MessageChunk
from langbot_plugin.api.entities.builtin.provider.message import ContentElement, Message, MessageChunk
from langbot_plugin.api.entities.builtin.resource.tool import LLMTool
from langbot_plugin.api.proxies.runner import RunnerAPIProxy
@@ -866,6 +866,33 @@ def serialize_tool_result_content(result: typing.Any, *, is_error: bool = False)
return json.dumps(result, ensure_ascii=False, default=str)
def split_tool_content(result: typing.Any) -> tuple[list[ContentElement], list[ContentElement]] | None:
"""Keep explicit multimodal tool output out of ordinary tool text and events."""
if not isinstance(result, (list, tuple)) or not result:
return None
media_types = {"image_url", "image_base64", "file_url", "file_base64"}
if not any(_model_or_mapping_get(item, "type") in media_types for item in result):
return None
text, media = [], []
for raw in result:
element = raw.model_copy(deep=True) if isinstance(raw, ContentElement) else ContentElement.model_validate(raw)
if element.type == "text":
text.append(element)
elif element.type in media_types:
payload = getattr(element, element.type)
if not payload or (element.type == "image_url" and not payload.url):
raise ValueError("Tool content is missing its media payload")
media.append(element)
text.append(
ContentElement.from_text(
f"[{element.type}: {element.file_name or 'content'} attached for model inspection]"
)
)
else:
raise ValueError(f"Unsupported tool content type: {element.type}")
return text, media
def _serialize_text_content_elements(result: typing.Any) -> str | None:
"""Flatten MCP-style text-only ContentElement lists for model tool messages."""
if not isinstance(result, (list, tuple)) or not result:
+32 -6
View File
@@ -5,6 +5,7 @@ from __future__ import annotations
from dataclasses import dataclass
from typing import Any
from langbot_plugin.api.agent_tools.external_tools import AgentRunExternalTools
from langbot_plugin.api.entities.builtin.provider.message import Message
from langbot_plugin.api.entities.builtin.resource.tool import LLMTool
from langbot_plugin.api.entities.builtin.runner import RunnerContext
@@ -61,6 +62,12 @@ class AgentRunAssembler:
raise NoAuthorizedModelError(NO_AUTHORIZED_MODEL_MESSAGE)
allowed_tools = self._allowed_tool_names()
input_tools = AgentRunExternalTools(self.api, self.ctx)
input_tool = next(
(tool for tool in input_tools.mcp_tools() if tool["name"] == "langbot_read_input_attachment"), None
)
if input_tool is not None:
allowed_tools.discard(input_tool["name"])
max_tool_iterations = get_max_tool_iterations(self.ctx.config)
max_tool_result_chars = get_max_tool_result_chars(self.ctx.config)
@@ -78,6 +85,23 @@ class AgentRunAssembler:
tools = await self._build_tools(allowed_tools)
tools.extend(workspace_manager.tools)
all_allowed_tools = allowed_tools | workspace_manager._tool_names
local_tools = {}
if input_tool is not None:
async def read_input(**arguments):
return await input_tools.call_tool(input_tool["name"], arguments)
tools.append(
LLMTool(
name=input_tool["name"],
human_desc=input_tool["description"],
description=input_tool["description"],
parameters=input_tool["inputSchema"],
func=read_input,
)
)
all_allowed_tools.add(input_tool["name"])
local_tools[input_tool["name"]] = read_input
token_counter = HostContextTokenCounter(self.api, model_ids[0], tools, reasoning_level=reasoning_level)
@@ -90,16 +114,18 @@ class AgentRunAssembler:
workspace_manager=workspace_manager,
).assemble()
tool_executor = LangBotToolExecutor(
self.api,
all_allowed_tools,
max_result_chars=max_tool_result_chars,
workspace_manager=workspace_manager,
)
tool_executor.local_tools.update(local_tools)
return AgentRunAssembly(
model_ids=model_ids,
messages=context_assembly.messages,
tools=tools,
tool_executor=LangBotToolExecutor(
self.api,
all_allowed_tools,
max_result_chars=max_tool_result_chars,
workspace_manager=workspace_manager,
),
tool_executor=tool_executor,
hooks=LangBotContextHooks(
context_budget,
summarizer=summarizer,
@@ -3,10 +3,17 @@
from __future__ import annotations
import asyncio
import json
from unittest.mock import AsyncMock
import pytest
from langbot_plugin.api.entities.builtin.provider.message import FunctionCall, Message, MessageChunk, ToolCall
from langbot_plugin.api.entities.builtin.provider.message import (
ContentElement,
FunctionCall,
Message,
MessageChunk,
ToolCall,
)
from langbot_plugin.api.entities.builtin.runner.errors import AgentAPIError, AgentAPIException
from langbot_plugin.api.entities.builtin.runner.steering import SteeringPullResult
from langbot_plugin.api.proxies.runner import PermissionDeniedError
@@ -28,10 +35,199 @@ from pkg.context_pipeline import ContextBudget
from pkg.model_calling import ModelCallError, StreamingModelCaller
@pytest.mark.asyncio
@pytest.mark.parametrize("streaming", [False, True])
async def test_multimodal_tool_content_reaches_model_without_entering_history(streaming):
payload = "data:image/png;base64,cHJpdmF0ZS1pbWFnZS1ieXRlcw=="
class FakeAPI:
def __init__(self):
self.calls = []
self.call_tool = AsyncMock(
return_value=[
{"type": "text", "text": '{"artifact_id":"art_test","mime_type":"image/png"}'},
{"type": "image_base64", "image_base64": payload},
]
)
def response(self, messages):
self.calls.append([message.model_copy(deep=True) for message in messages])
if len(self.calls) == 1:
return Message(
role="assistant",
content="Reading",
tool_calls=[
ToolCall(
id="read-1",
type="function",
function=FunctionCall(name="read_input_attachment_content", arguments="{}"),
)
],
)
media = messages[-1]
assert media.role == "user" and media.content[0].image_base64 == payload
assert messages[-2].role == "tool" and payload not in messages[-2].content
return Message(role="assistant", content="The image has been inspected")
async def invoke_llm(self, *args, **kwargs):
return self.response(kwargs["messages"])
def invoke_llm_stream(self, *args, **kwargs):
response = self.response(kwargs["messages"])
async def stream():
yield MessageChunk(
role="assistant", content=response.content, tool_calls=response.tool_calls, is_final=True
)
return stream()
api = FakeAPI()
loop = AgentLoop(
model_adapter=LangBotModelAdapter(api),
tool_executor=LangBotToolExecutor(api, {"read_input_attachment_content"}),
model_ids=["model-1"],
messages=[Message(role="user", content="Inspect my image")],
tools=[],
streaming=streaming,
max_tool_iterations=2,
)
events = [event async for event in loop.run()]
assert len(api.calls) == 2
assert events[-1].type == AgentLoopEventType.AGENT_END
assert payload not in json.dumps([message.model_dump() for message in loop.messages])
for event in events:
if event.message is not None:
assert payload not in event.message.model_dump_json()
assert payload not in json.dumps(event.result)
async def _collect_events(loop: AgentLoop):
return [event async for event in loop.run()]
@pytest.mark.asyncio
@pytest.mark.parametrize("streaming", [False, True])
async def test_attachment_tokens_participate_in_budget_and_cannot_be_silently_dropped(streaming):
payload = "data:image/png;base64,cHJpdmF0ZS1idWRnZXQ="
class MediaCounter:
def __init__(self):
self.saw_media = False
async def count(self, messages):
media = sum(
1
for message in messages
if isinstance(message.content, list)
for item in message.content
if item.type == "image_base64"
)
self.saw_media |= bool(media)
return 80 * media + 2 * len(messages)
class NeverModel:
async def invoke_turn(self, **kwargs):
raise AssertionError("A model must not run after its unread image was removed to fit the budget")
async def stream_turn(self, **kwargs):
raise AssertionError("A model must not run after its unread image was removed to fit the budget")
yield
counter = MediaCounter()
loop = AgentLoop(
model_adapter=NeverModel(),
tool_executor=LangBotToolExecutor(AsyncMock(), set()),
model_ids=["model-1"],
messages=[Message(role="user", content="Inspect this image")],
tools=[],
streaming=streaming,
max_tool_iterations=2,
hooks=LangBotContextHooks(
ContextBudget(window_tokens=40, reserve_tokens=0, keep_recent_tokens=0, summary_tokens=0),
token_counter=counter,
),
)
loop._model_only_messages = [Message(role="user", content=[ContentElement.from_image_base64(payload)])]
events = await _collect_events(loop)
assert counter.saw_media
assert events[-1].type == AgentLoopEventType.RUN_FAILED
assert "Attachment content was removed by context budgeting" in events[-1].error
assert payload not in json.dumps([message.model_dump() for message in loop.messages])
@pytest.mark.asyncio
@pytest.mark.parametrize("streaming", [False, True])
async def test_overflow_recovery_receives_media_and_cannot_retry_after_discarding_it(streaming):
payload = "data:image/png;base64,cHJpdmF0ZS1vdmVyZmxvdw=="
class OverflowModel:
calls = 0
async def invoke_turn(self, *, messages, **kwargs):
self.calls += 1
assert messages[-1].content[0].image_base64 == payload
raise ModelCallError("context length exceeded", retryable=True)
async def stream_turn(self, **kwargs):
await self.invoke_turn(**kwargs)
yield
class DroppingRecovery(AgentLoopHooks):
saw_media = False
async def recover_context_overflow(self, messages, error):
self.saw_media = messages[-1].content[0].image_base64 == payload
return [Message(role="user", content="retry without the image")]
model, hooks = OverflowModel(), DroppingRecovery()
loop = AgentLoop(
model_adapter=model,
tool_executor=LangBotToolExecutor(AsyncMock(), set()),
model_ids=["model-1"],
messages=[Message(role="user", content="Inspect this image")],
tools=[],
streaming=streaming,
max_tool_iterations=2,
hooks=hooks,
)
loop._model_only_messages = [Message(role="user", content=[ContentElement.from_image_base64(payload)])]
events = await _collect_events(loop)
assert hooks.saw_media and model.calls == 1
assert events[-1].type == AgentLoopEventType.RUN_FAILED
assert "context length exceeded" in events[-1].error
assert payload not in json.dumps([message.model_dump() for message in loop.messages])
@pytest.mark.asyncio
async def test_budget_cannot_keep_one_copy_of_a_twice_read_image_and_restore_the_discarded_copy():
media = Message(role="user", content=[ContentElement.from_image_base64("data:image/png;base64,c2FtZQ==")])
class DropDuplicate(AgentLoopHooks):
async def prepare_model_turn(self, messages):
assert messages[-2] == messages[-1]
return messages[:-1]
class NeverModel:
async def invoke_turn(self, **kwargs):
raise AssertionError("Only one of two media copies was budgeted")
loop = AgentLoop(
model_adapter=NeverModel(),
tool_executor=LangBotToolExecutor(AsyncMock(), set()),
model_ids=["model-1"],
messages=[Message(role="user", content="Inspect the two reads")],
tools=[],
streaming=False,
max_tool_iterations=2,
hooks=DropDuplicate(),
)
loop._model_only_messages = [media, media.model_copy(deep=True)]
events = await _collect_events(loop)
assert events[-1].type == AgentLoopEventType.RUN_FAILED
assert "Attachment content was removed by context budgeting" in events[-1].error
@pytest.mark.asyncio
async def test_steering_puller_accepts_sdk_result_models():
"""Host steering_pull returns SDK DTOs, not plain dicts."""
@@ -1,17 +1,69 @@
from __future__ import annotations
import asyncio
import json
from types import SimpleNamespace
from unittest.mock import AsyncMock
import pytest
from langbot_plugin.api.entities.builtin.runner.input import InputAttachment
from langbot_plugin.api.entities.builtin.runner.resources import AgentResources, ModelResource, ToolResource
from pkg.agent_core import LangBotContextHooks, LangBotToolExecutor
from pkg.agent_core import LangBotContextHooks, LangBotToolExecutor, ToolCallRequest
from pkg.messages import build_platform_tools_system_message
from pkg.run_assembly import AgentRunAssembler, NoAuthorizedModelError
from tests.test_runner import FakeRunnerAPIProxy, make_context
@pytest.mark.asyncio
@pytest.mark.parametrize("granted,has_attachment", [(False, True), (True, False), (True, True)])
async def test_input_reader_is_discovered_from_the_run_grant_and_traces_no_binary(granted, has_attachment):
api = FakeRunnerAPIProxy(models=[ModelResource(model_id="model-primary")])
ctx = make_context(
config={"model": {"primary": "model-primary"}, "box-enabled": False},
resources=AgentResources(models=[ModelResource(model_id="model-primary")]),
)
ctx.context.available_apis.input_attachments = granted
ctx.input.attachments = [InputAttachment(ref="att-1", type="image")] if has_attachment else []
payload = "data:image/png;base64,cHJpdmF0ZQ=="
api.read_input_attachment_content = AsyncMock(
return_value=[
{"type": "text", "text": '{"ref":"att-1"}'},
{"type": "image_base64", "image_base64": payload},
]
)
api._results = asyncio.Queue()
api._run_id = "input-run"
assembly = await AgentRunAssembler(api, ctx).assemble()
names = {tool.name for tool in assembly.tools}
assert ("langbot_read_input_attachment" in names) is (granted and has_attachment)
if not (granted and has_attachment):
api.read_input_attachment_content.assert_not_awaited()
return
prepared = assembly.tool_executor.prepare(
ToolCallRequest.from_raw(
{
"id": "call-read",
"function": {"name": "langbot_read_input_attachment", "arguments": '{"ref":"att-1"}'},
}
)
)
outcome = await assembly.tool_executor.execute(prepared)
assert outcome.error is None and outcome.model_only_messages[0].content[0].image_base64 == payload
api.read_input_attachment_content.assert_awaited_once_with(
ref="att-1",
mode="auto",
offset=0,
max_chars=8192,
page_start=1,
page_end=None,
)
assert payload not in outcome.message.content
started, completed = api._results.get_nowait(), api._results.get_nowait()
assert started.type.value == "tool.call.started" and completed.type.value == "tool.call.completed"
assert payload not in json.dumps(completed.data)
@pytest.mark.asyncio
async def test_assembler_builds_authorized_loop_inputs() -> None:
api = FakeRunnerAPIProxy(
+1 -1
View File
@@ -51,7 +51,7 @@ dependencies = [
"python-socks>=2.7.1", # dingtalk missing dependency
"pip>=26.2.0",
"uv>=0.13.4",
"pypdf>=6.0.0",
"pypdf>=6.21.2",
"python-docx>=1.1.0",
"chardet>=5.2.0",
"xxhash>=3.5.0",
@@ -6,6 +6,7 @@ import base64
import copy
import json
import typing
from urllib.parse import quote
import pydantic
@@ -39,6 +40,11 @@ class ImportBoxArgs(EmptyArgs):
class ReadInputAttachmentArgs(EmptyArgs):
ref: str = pydantic.Field(min_length=1)
mode: typing.Literal["auto", "text", "native", "pages"] = "auto"
offset: int = pydantic.Field(default=0, ge=0)
max_chars: int = pydantic.Field(default=8192, ge=1, le=16384)
page_start: int = pydantic.Field(default=1, ge=1)
page_end: int | None = pydantic.Field(default=None, ge=1)
class ReplyFilesArgs(EmptyArgs):
@@ -252,6 +258,88 @@ class AgentRunExternalTools:
}
structured = _dump_jsonable(result)
content_types = {"text", "image_base64", "file_base64", "image_url", "file_url"}
if (
isinstance(structured, list)
and structured
and all(
isinstance(item, dict) and item.get("type") in content_types
for item in structured
)
):
contents = []
try:
for item in structured:
kind = item["type"]
if kind == "text":
if not isinstance(item.get("text"), str):
raise ValueError("Missing tool text content")
contents.append({"type": "text", "text": item["text"]})
elif kind in {"image_url", "file_url"}:
uri = (
item["image_url"]["url"]
if kind == "image_url"
else item["file_url"]
)
if not isinstance(uri, str) or not uri:
raise ValueError("Missing tool media URL")
contents.append(
{
"type": "resource_link",
"uri": uri,
"name": item.get("file_name") or "image",
}
)
else:
encoded = item[kind]
if not isinstance(encoded, str) or not encoded:
raise ValueError("Missing tool media payload")
mime_type = (
"image/png"
if kind == "image_base64"
else "application/octet-stream"
)
if encoded.startswith("data:"):
header, separator, encoded = encoded.partition(",")
if not separator or not header.endswith(";base64"):
raise ValueError("Invalid tool media data URL")
mime_type = header[5:-7]
if not mime_type or (
kind == "image_base64"
and not mime_type.startswith("image/")
):
raise ValueError("Invalid tool media MIME type")
base64.b64decode(encoded, validate=True)
if kind == "image_base64":
contents.append(
{
"type": "image",
"data": encoded,
"mimeType": mime_type,
}
)
else:
contents.append(
{
"type": "resource",
"resource": {
"uri": "langbot-artifact:///"
+ quote(
item.get("file_name") or "file", safe=""
),
"mimeType": mime_type,
"blob": encoded,
},
}
)
return {"content": contents}
except (KeyError, TypeError, ValueError) as exc:
return {
"isError": True,
"content": [
{"type": "text", "text": f"Invalid tool media content: {exc}"}
],
}
if not isinstance(structured, dict):
structured = {"result": structured}
return {
@@ -503,18 +591,11 @@ class AgentRunExternalTools:
@agent_tool(
name="langbot_read_input_attachment",
description="Read one current input attachment by ref, such as att-1 from [attachment att-1: ...] in the message. Returns name, mime_type, size and base64 content.",
description="Read a current input attachment by ref. Auto returns bounded text or a structured image; PDF defaults to text. Use native for PDF visual content with a supporting model. Text offsets are Unicode characters; use next_offset to continue. At most ten PDF pages per call. Page rendering is unavailable.",
args_model=ReadInputAttachmentArgs,
)
async def read_input_attachment(self, args: ReadInputAttachmentArgs):
data = await self.api.read_input_attachment(args.ref)
return {
"ref": data.ref,
"name": data.name,
"mime_type": data.mime_type,
"size": data.size,
"content_base64": base64.b64encode(data.data).decode("ascii"),
}
return await self.api.read_input_attachment_content(**args.model_dump())
@agent_tool(
name="langbot_export_box_files",
@@ -190,6 +190,10 @@ class RunnerContext(pydantic.BaseModel):
return None
return await self.api.read_input_attachment(ref)
async def read_input_attachment_content(self, ref: str, **options) -> list[dict]:
"""Read bounded text or structured media using this run's input grant."""
return await self.api.read_input_attachment_content(ref, **options)
async def import_box_attachments(
self, attachment_ids: list[str] | None = None
) -> list["BoxFile"]:
@@ -41,7 +41,7 @@ class InputAttachment(pydantic.BaseModel):
such as url, base64 or message_chain."""
url: str | None = None
"""External URL when the attachment is backed by a URL."""
"""Legacy external URL; current Hosts keep source locators private."""
path: str | None = None
"""Sandbox/workspace path when Host has staged the attachment as a file."""
@@ -133,6 +133,32 @@ class LangBotAPIProxy:
)
return [BoxFile.model_validate(item) for item in result.get("items", [])]
@run_scoped
async def read_input_attachment_content(
self,
ref: str,
*,
mode: str = "auto",
offset: int = 0,
max_chars: int = 8192,
page_start: int = 1,
page_end: int | None = None,
):
"""Read bounded model content from this run's input attachment."""
result = await self.plugin_runtime_handler.call_action(
PluginToRuntimeAction.READ_INPUT_ATTACHMENT_CONTENT,
{
"ref": ref,
"mode": mode,
"offset": offset,
"max_chars": max_chars,
"page_start": page_start,
"page_end": page_end,
},
180,
)
return result["contents"]
@run_scoped
async def read_input_attachment(self, ref: str):
"""Read the bytes of one current-event attachment by ref."""
@@ -281,6 +281,36 @@ class RunnerContextAPIMixin:
data=data,
)
async def read_input_attachment_content(
self,
ref: str,
*,
mode: str = "auto",
offset: int = 0,
max_chars: int = 8192,
page_start: int = 1,
page_end: int | None = None,
) -> list[dict[str, Any]]:
"""Read bounded model content using the current run's attachment grant."""
self._require_context_api("input_attachments")
response = await self._api.plugin_runtime_handler.call_action(
PluginToRuntimeAction.READ_INPUT_ATTACHMENT_CONTENT,
{
"run_id": self.run_id,
"ref": ref,
"mode": mode,
"offset": offset,
"max_chars": max_chars,
"page_start": page_start,
"page_end": page_end,
},
self._bounded_timeout(default=120.0),
cancel_peer_on_cancel=True,
)
return self._expect_key(
response, "contents", PluginToRuntimeAction.READ_INPUT_ATTACHMENT_CONTENT
)
async def steering_ack(self, delivery_ids: list[str]) -> SteeringAckResult:
"""Acknowledge claimed steering inputs after applying them."""
self._require_context_api("steering_ack")
@@ -8,6 +8,30 @@ from langbot_plugin.api.entities.builtin.runner.result import RunnerResult
from langbot_plugin.api.proxies.runner.api import RunnerAPIProxy
def _tool_trace_result(value):
"""Keep structured media out of retained tool events without changing the tool result."""
if hasattr(value, "model_dump"):
value = value.model_dump(mode="json")
if isinstance(value, list):
return [_tool_trace_result(item) for item in value]
if not isinstance(value, dict):
return (
"[binary content redacted]"
if isinstance(value, str)
and value.startswith("data:")
and ";base64," in value
else value
)
output = {}
for key, item in value.items():
if key in {"base64", "image_base64", "file_base64"} and item:
output[key] = None
output["content_redacted"] = True
else:
output[key] = _tool_trace_result(item)
return output
class _TracedRunAPI:
"""Delegate authorized APIs and stream tool delivery attempts and outcomes."""
@@ -48,7 +72,9 @@ class _TracedRunAPI:
self._run_id,
call_id,
tool_name,
result=result,
result=_tool_trace_result(result)
if isinstance(result, dict)
else {"value": _tool_trace_result(result)},
)
)
return result
@@ -41,6 +41,7 @@ class PluginToRuntimeAction(ActionType):
BIND_BOX = "bind_box"
IMPORT_BOX_ATTACHMENTS = "import_box_attachments"
READ_INPUT_ATTACHMENT = "read_input_attachment"
READ_INPUT_ATTACHMENT_CONTENT = "read_input_attachment_content"
EXPORT_BOX_FILES = "export_box_files"
GET_BOTS = "get_bots"
@@ -169,6 +169,7 @@ class PluginConnectionHandler(handler.Handler):
in {
PluginToRuntimeAction.CALL_TOOL,
PluginToRuntimeAction.REQUEST_INTERACTION,
PluginToRuntimeAction.READ_INPUT_ATTACHMENT_CONTENT,
}
and payload.get("run_id") is not None
)
@@ -922,6 +923,16 @@ class PluginConnectionHandler(handler.Handler):
LONG_RUNNING_OPERATION_TIMEOUT,
)
@self.action(PluginToRuntimeAction.READ_INPUT_ATTACHMENT_CONTENT)
async def read_input_attachment_content(
data: dict[str, Any],
) -> handler.ActionResponse:
return await forward_agent_action(
PluginToRuntimeAction.READ_INPUT_ATTACHMENT_CONTENT,
data,
LONG_RUNNING_OPERATION_TIMEOUT,
)
@self.action(PluginToRuntimeAction.EXPORT_BOX_FILES)
async def export_box_files(data: dict[str, Any]) -> handler.ActionResponse:
return await forward_agent_action(
@@ -23,6 +23,7 @@ from langbot_plugin.api.entities.builtin.runner.runtime import AgentRuntimeConte
from langbot_plugin.api.entities.builtin.runner.trigger import AgentTrigger
from langbot_plugin.api.proxies.runner import RunnerAPIProxy
from langbot_plugin.api.proxies.runner.common import PermissionDeniedError
from langbot_plugin.api.entities.builtin.runner.errors import AgentAPIException
from langbot_plugin.entities.io.actions.enums import PluginToRuntimeAction
PAYLOAD = b"%PDF-1.7 test"
@@ -117,3 +118,47 @@ def test_older_host_context_without_the_flag_disables_the_api():
access = ContextAccess.model_validate({"available_apis": {"box": True}})
assert access.available_apis.input_attachments is False
@pytest.mark.anyio
async def test_content_proxy_preserves_structured_output_and_run_scope():
contents = [
{"type": "text", "text": "metadata"},
{"type": "image_base64", "image_base64": "data:image/png;base64,aGk="},
]
handler = _handler()
handler.call_action.return_value = {"contents": contents}
ctx = _context(enabled=True)
ctx._api = RunnerAPIProxy(ctx=ctx, plugin_runtime_handler=handler)
assert (
await ctx.read_input_attachment_content(
"att-1", mode="native", page_start=2, page_end=3
)
== contents
)
action, data, timeout = handler.call_action.call_args.args
assert action == PluginToRuntimeAction.READ_INPUT_ATTACHMENT_CONTENT
assert data == {
"run_id": "run-1",
"ref": "att-1",
"mode": "native",
"offset": 0,
"max_chars": 8192,
"page_start": 2,
"page_end": 3,
}
assert timeout > 0
assert handler.call_action.call_args.kwargs["cancel_peer_on_cancel"] is True
@pytest.mark.anyio
async def test_content_proxy_rejects_missing_capability_and_malformed_response():
handler = _handler()
proxy = RunnerAPIProxy(ctx=_context(enabled=False), plugin_runtime_handler=handler)
with pytest.raises(PermissionDeniedError):
await proxy.read_input_attachment_content("att-1")
handler.call_action.assert_not_called()
proxy = RunnerAPIProxy(ctx=_context(enabled=True), plugin_runtime_handler=handler)
with pytest.raises(AgentAPIException) as failure:
await proxy.read_input_attachment_content("att-1")
assert failure.value.error.code == "host.malformed_response"
@@ -0,0 +1,33 @@
"""Model content returns intact while retained tool traces contain no inline bytes."""
import asyncio
import json
from types import SimpleNamespace
from unittest.mock import AsyncMock
import pytest
from langbot_plugin.api.proxies.runner.tracing import _TracedRunAPI
@pytest.mark.anyio
async def test_tool_content_list_is_valid_trace_output_and_binary_is_redacted():
payload = "data:image/png;base64,cHJpdmF0ZS1ieXRlcw=="
result = [
{"type": "text", "text": "image metadata"},
{"type": "image_base64", "image_base64": payload},
]
queue = asyncio.Queue()
api = _TracedRunAPI(
SimpleNamespace(call_tool=AsyncMock(return_value=result)), "run-1", queue
)
returned = await api.call_tool("read", {"ref": "att-1"})
assert returned is result and returned[1]["image_base64"] == payload
started, completed = queue.get_nowait(), queue.get_nowait()
assert (
started.type.value == "tool.call.started"
and completed.type.value == "tool.call.completed"
)
assert completed.data["result"]["value"][0]["text"] == "image metadata"
assert completed.data["result"]["value"][1]["content_redacted"] is True
assert payload not in json.dumps(completed.data)
+53 -16
View File
@@ -218,19 +218,13 @@ async def test_external_box_tools_use_authorized_run_api(
@pytest.mark.asyncio
async def test_external_input_attachment_tool_reads_by_ref():
from langbot_plugin.api.entities.builtin.runner.input import (
InputAttachment,
InputAttachmentData,
)
from langbot_plugin.api.entities.builtin.runner.input import InputAttachment
ctx = _authorized_ctx()
ctx.input.attachments = [InputAttachment(ref="att-1", type="file")]
api = MagicMock()
api.read_input_attachment = AsyncMock(
return_value=InputAttachmentData(
ref="att-1", name="a.txt", mime_type="text/plain", data=b"hi"
)
)
contents = [{"type": "text", "text": '{"ref":"att-1","text":"hi"}'}]
api.read_input_attachment_content = AsyncMock(return_value=contents)
tools = AgentRunExternalTools(api, ctx)
assert "langbot_read_input_attachment" not in {
tool["name"] for tool in tools.mcp_tools()
@@ -239,14 +233,57 @@ async def test_external_input_attachment_tool_reads_by_ref():
assert "langbot_read_input_attachment" in {
tool["name"] for tool in tools.mcp_tools()
}
assert await tools.call_tool("langbot_read_input_attachment", {"ref": "att-1"}) == {
"ref": "att-1",
"name": "a.txt",
"mime_type": "text/plain",
"size": 2,
"content_base64": "aGk=",
assert (
await tools.call_tool("langbot_read_input_attachment", {"ref": "att-1"})
== contents
)
api.read_input_attachment_content.assert_awaited_once_with(
ref="att-1",
mode="auto",
offset=0,
max_chars=8192,
page_start=1,
page_end=None,
)
@pytest.mark.asyncio
async def test_external_input_content_uses_native_mcp_image_and_pdf_blocks():
from langbot_plugin.api.entities.builtin.runner.input import InputAttachment
ctx = _authorized_ctx()
ctx.context.available_apis.input_attachments = True
ctx.input.attachments = [InputAttachment(ref="att-1", type="file")]
api = MagicMock()
api.read_input_attachment_content = AsyncMock(
return_value=[
{"type": "text", "text": '{"ref":"att-1"}'},
{"type": "image_base64", "image_base64": "data:image/png;base64,aW1hZ2U="},
{
"type": "file_base64",
"file_base64": "data:application/pdf;base64,cGRm",
"file_name": "report.pdf",
},
]
)
result = await AgentRunExternalTools(api, ctx).call_mcp_tool(
"langbot_read_input_attachment", {"ref": "att-1"}
)
assert result == {
"content": [
{"type": "text", "text": '{"ref":"att-1"}'},
{"type": "image", "data": "aW1hZ2U=", "mimeType": "image/png"},
{
"type": "resource",
"resource": {
"uri": "langbot-artifact:///report.pdf",
"mimeType": "application/pdf",
"blob": "cGRm",
},
},
]
}
api.read_input_attachment.assert_awaited_once_with("att-1")
assert all("base64" not in block.get("text", "") for block in result["content"])
def test_external_box_file_reply_requires_reply_permission():
@@ -4,6 +4,7 @@ import asyncio
import logging
import time
from types import SimpleNamespace
from unittest.mock import AsyncMock
import pytest
@@ -169,6 +170,37 @@ def _handler(debug_plugin=False, action_context=None, *, runtime_profile=None):
return handler, manager, control_handler
async def test_input_content_forwarding_uses_trusted_identity_scope_and_cancellation():
binding = _action_context()
handler, manager, control = _handler(action_context=binding)
manager.plugins = [FakePluginContainer(runtime_handler=handler)]
contents = [{"type": "text", "text": "bounded content"}]
control.call_action = AsyncMock(return_value={"contents": contents})
async with ProtocolSession(handler) as session:
result = await session.request(
PluginToRuntimeAction.READ_INPUT_ATTACHMENT_CONTENT.value,
{
"run_id": "run-1",
"ref": "att-1",
"mode": "text",
"caller_plugin_identity": "forged/identity",
"workspace_uuid": "forged-workspace",
},
action_context=None,
)
assert result["code"] == 0 and result["data"] == {"contents": contents}
action, data = control.call_action.call_args.args
assert action == PluginToRuntimeAction.READ_INPUT_ATTACHMENT_CONTENT
assert data == {
"run_id": "run-1",
"ref": "att-1",
"mode": "text",
"caller_plugin_identity": "tester/demo",
}
assert control.call_action.call_args.kwargs["action_context"] == binding
assert control.call_action.call_args.kwargs["cancel_peer_on_cancel"] is True
def test_reserved_decode_capacity_is_consumed_by_first_message_only():
handler, _manager, _control = _handler()
handler._certified_shared_stdio = True
+12 -5
View File
@@ -349,15 +349,22 @@ controlled request-layer measurements and their scope.
`create_artifact(filename, content_base64, mime_type, ttl_seconds)` uploads at most
10 MiB and returns an opaque ID and metadata. Use `get_artifact(artifact_id)` to
inspect it, and `read_artifact(artifact_id, max_bytes)` only when bytes are needed.
The read result is base64 for file transfer; it does not extract text, render PDF
pages or prove model understanding. Prefer the authenticated HTTP download route
for user downloads. See [Artifact API](../../../docs/en/develop/artifacts.mdx).
inspect it. Prefer `read_artifact_content(artifact_id, mode, offset, max_chars,
page_start, page_end)` for model consumption: it returns bounded text or native
MCP image/PDF content. Continue using the returned text/page offsets when a result
is truncated. PDF text excludes visual content; `native` needs a supporting model,
and `pages` currently reports unsupported. `read_artifact(artifact_id, max_bytes)`
returns base64 for raw file transfer. Use authenticated HTTP downloads for users.
See [Artifact API](../../../docs/en/develop/artifacts.mdx) for limits and modes.
Reads require `resource.view`; writes require `resource.manage`. The authenticated
account or API key must own the asset in the current Workspace. An ID is not an
access grant. Plugin/run references are not accepted by this initial API, and
Artifact operations are not added to default model tool sets or plugin grants.
Account-owned Artifact operations are not added to default model tool sets or
plugin grants. Runner attachment tools use their existing run-scoped grant and
Host-only input references; an account upload can be passed to Agent debug with
`artifact_ids` after ownership validation. The management assistant does not
accept attachments in this phase.
Temporary files expire after 24 hours by default (`ttl_seconds`: 1 to 604800).
`retain_artifact` adds a persistent pin without broadening access.
+1 -19
View File
@@ -132,7 +132,6 @@ def render_input(
chain: typing.Iterable[platform_message.MessageComponent] | None,
*,
include_quotes: bool,
keep_images: bool = True,
include_other_text: bool = False,
) -> RenderedInput:
"""Render a message chain into model contents and ordered media.
@@ -140,8 +139,6 @@ def render_input(
Args:
chain: Platform message chain of the current message.
include_quotes: Render quoted messages and expose their media.
keep_images: Forward image payloads to the model; otherwise only the
marker describes the image.
include_other_text: Render non-media components without a dedicated
projection (such as mentions) with their string form.
"""
@@ -160,24 +157,9 @@ def render_input(
item = InputMedia(ref=f'att-{len(media) + 1}', component=component, origin=origin)
media.append(item)
add_text(item.marker(), plain=False)
payload = component.base64 or None
url = getattr(component, 'url', None) or None
if isinstance(component, platform_message.Image):
if not keep_images:
if origin == ORIGIN_MESSAGE:
plain_parts.append('[Image]')
elif payload:
contents.append(provider_message.ContentElement.from_image_base64(payload))
elif url:
contents.append(provider_message.ContentElement.from_image_url(url))
return
if isinstance(component, platform_message.Voice):
name = 'voice.silk' if payload else 'voice'
else:
name = component.name
if payload:
contents.append(provider_message.ContentElement.from_file_base64(payload, name))
elif url:
contents.append(provider_message.ContentElement.from_file_url(url, name))
def render(components: typing.Iterable[platform_message.MessageComponent], origin: str) -> None:
for component in components:
@@ -3,6 +3,7 @@
from __future__ import annotations
from pydantic import Field
from typing import Literal
from ..authz import Permission
from ..context import PrincipalType
@@ -25,6 +26,14 @@ class ArtifactRead(ArtifactID):
max_bytes: int = Field(default=MAX_ARTIFACT_BYTES, ge=1, le=MAX_ARTIFACT_BYTES)
class ArtifactContent(ArtifactID):
mode: Literal['auto', 'text', 'native', 'pages'] = 'auto'
offset: int = Field(default=0, ge=0)
max_chars: int = Field(default=8192, ge=1, le=16384)
page_start: int = Field(default=1, ge=1)
page_end: int | None = Field(default=None, ge=1)
async def create(ap, context, payload):
return await ap.artifact_service.create(context, **payload.model_dump())
@@ -37,6 +46,10 @@ async def read(ap, context, payload):
return await ap.artifact_service.read(context, payload.artifact_id, max_bytes=payload.max_bytes)
async def read_content(ap, context, payload):
return await ap.artifact_service.read_content(context, **payload.model_dump())
async def retain(ap, context, payload):
return await ap.artifact_service.retain(context, payload.artifact_id)
@@ -56,6 +69,7 @@ def operations():
Operation('artifact.create', manage, create, ArtifactCreate, dict, True, principals=principals),
Operation('artifact.get', view, get, ArtifactID, dict, principals=principals),
Operation('artifact.read', view, read, ArtifactRead, dict, principals=principals),
Operation('artifact.read_content', view, read_content, ArtifactContent, list[dict], principals=principals),
Operation('artifact.retain', manage, retain, ArtifactID, dict, True, principals=principals),
Operation('artifact.release', manage, release, ArtifactID, dict, True, principals=principals),
Operation('artifact.delete', manage, delete, ArtifactID, dict, True, principals=principals),
@@ -529,6 +529,20 @@ EXECUTION_OPERATIONS = (
),
'run',
),
ExecutionOperation(
'read_input_attachment_content',
_input(
'ExecutionReadInputContentInput',
ref=(Text, ...),
mode=(Literal['auto', 'text', 'native', 'pages'], 'auto'),
offset=(Nonnegative, 0),
max_chars=(Annotated[int, Field(ge=1, le=16384)], 8192),
page_start=(Positive, 1),
page_end=(Positive | None, None),
),
_output('ExecutionInputContentOutput', contents=(list[Object], ...)),
'run',
),
ExecutionOperation(
'export_box_files', _input('ExecutionExportBoxInput', paths=(list[str] | None, None)), ItemsOutput, 'run', True
),
@@ -149,6 +149,14 @@ OPERATION_DOCS: dict[str, OperationDoc] = {
risk='read',
exposure='none',
),
'artifact.read_content': OperationDoc(
domain='artifact',
label={'en_US': 'Read Artifact content', 'zh_Hans': '读取资产内容'},
summary='Read bounded text or structured media from an owned asset.',
description='Returns text slices, images or native PDF input. PDF page rendering is unavailable.',
risk='read',
exposure='none',
),
'artifact.retain': OperationDoc(
domain='artifact',
label={'en_US': 'Retain Artifact', 'zh_Hans': '保留资产'},
@@ -122,6 +122,7 @@ CONTEXT_ACTIONS = frozenset(
PluginAction.BIND_BOX,
PluginAction.IMPORT_BOX_ATTACHMENTS,
PluginAction.READ_INPUT_ATTACHMENT,
PluginAction.READ_INPUT_ATTACHMENT_CONTENT,
PluginAction.EXPORT_BOX_FILES,
PluginAction.GET_PROMPT,
PluginAction.HISTORY_PAGE,
@@ -23,7 +23,9 @@ class ArtifactsRouterGroup(group.RouterGroup):
except ApplicationAPIException as exc:
return self.http_status(exc.error.status_code, exc.error.code, exc.error.message)
if not download:
return self.success(data=result)
response = self.success(data=result)
response.headers['Cache-Control'] = 'private, no-store'
return response
metadata = result['metadata']
return quart.Response(
result['content'],
@@ -93,6 +95,22 @@ class ArtifactsRouterGroup(group.RouterGroup):
download=True,
)
@self.route(
'/<artifact_id>/read',
methods=['GET'],
auth_type=group.AuthType.USER_TOKEN_OR_API_KEY,
permission=Permission.RESOURCE_VIEW,
)
async def read_content(artifact_id: str, request_context: RequestContext):
payload = {'artifact_id': artifact_id, 'mode': quart.request.args.get('mode', 'auto')}
for key in ('offset', 'max_chars', 'page_start', 'page_end'):
if key in quart.request.args:
try:
payload[key] = int(quart.request.args[key])
except ValueError:
return self.http_status(400, 'invalid_input', f'Invalid {key}')
return await invoke('artifact.read_content', request_context, payload)
@self.route(
'/<artifact_id>/retain',
methods=['POST'],
+31 -1
View File
@@ -186,6 +186,29 @@ class AgentService:
if not runner_id:
raise ValueError('Agent has no configured runner')
artifact_ids = payload.get('artifact_ids', [])
if (
not isinstance(artifact_ids, list)
or len(artifact_ids) > 20
or any(not isinstance(value, str) or not value for value in artifact_ids)
or len(set(artifact_ids)) != len(artifact_ids)
):
raise ValueError('artifact_ids must contain at most 20 distinct Artifact IDs')
artifacts = []
for artifact_id in artifact_ids:
# Check the authenticated caller, never the synthetic event actor.
metadata = await self.ap.artifact_service.get(context, artifact_id)
artifacts.append(
{
'ref': artifact_id,
'type': 'image' if metadata['mime_type'].startswith('image/') else 'file',
'source': 'artifact',
'name': metadata['name'],
'mime_type': metadata['mime_type'],
'size': metadata['size'],
}
)
conversation_id = str(payload.get('conversation_id') or f'debug:{agent_uuid}').strip()
if not conversation_id or len(conversation_id) > 256:
raise ValueError('Invalid debug conversation_id')
@@ -238,8 +261,12 @@ class AgentService:
'text': text or event_type,
'contents': [
{'type': 'text', 'text': text or event_type},
*(
{'type': 'text', 'text': f'[attachment {item["ref"]}: {item["type"]} {item["name"]}]'}
for item in artifacts
),
],
'attachments': [],
'attachments': artifacts,
}
),
delivery=DeliveryContext(
@@ -324,6 +351,9 @@ class AgentService:
descriptor = await self.ap.runner_registry.get(execution_context, runner_id)
prompt_config = extract_prompt_config(descriptor, runner_config, [])
execution_query = build_execution_query(event, [], execution_context)
if artifact_ids:
object.__setattr__(execution_query, '_input_artifact_context', context)
object.__setattr__(execution_query, '_input_artifact_ids', frozenset(artifact_ids))
execution_query.prompt = provider_prompt.Prompt(
name='default',
messages=[provider_message.Message(**message) for message in copy.deepcopy(prompt_config)],
+48
View File
@@ -22,6 +22,7 @@ import base64
import binascii
import typing
import functools
from mcp import types as mcp_types
from mcp.server.fastmcp import FastMCP
from mcp.server.fastmcp.exceptions import ToolError
@@ -151,6 +152,53 @@ class LangBotMCPServer:
}
)
@mcp.tool(
description='Read bounded text, a structured image or native PDF from an owned Artifact. PDF auto/text excludes visual content; native needs model file support. Page rendering is unavailable.'
)
async def read_artifact_content(
artifact_id: str,
mode: str = 'auto',
offset: int = 0,
max_chars: int = 8192,
page_start: int = 1,
page_end: int | None = None,
) -> mcp_types.CallToolResult:
context = _authorized(Permission.RESOURCE_VIEW)
elements = await artifact_operation(
'artifact.read_content',
context,
{
'artifact_id': artifact_id,
'mode': mode,
'offset': offset,
'max_chars': max_chars,
'page_start': page_start,
'page_end': page_end,
},
)
blocks = []
for element in elements:
if element['type'] == 'text':
blocks.append(mcp_types.TextContent(type='text', text=element['text']))
else:
data_url = element.get('image_base64') or element['file_base64']
header, blob = data_url.split(';base64,', 1)
mime = header[5:]
if element['type'] == 'image_base64':
blocks.append(mcp_types.ImageContent(type='image', data=blob, mimeType=mime))
else:
blocks.append(
mcp_types.EmbeddedResource(
type='resource',
resource=mcp_types.BlobResourceContents(
uri=f'artifact://{artifact_id}',
blob=blob,
mimeType=mime,
),
)
)
return mcp_types.CallToolResult(content=blocks)
@mcp.tool(
description='Add a persistent pin to a caller-owned Artifact without widening its access permissions.'
)
+33 -6
View File
@@ -40,11 +40,9 @@ def prepare_input_files(query, agent_input):
event envelope is metadata, never permission to read the Host filesystem.
"""
from langbot_plugin.api.entities.builtin.runner.input import InputAttachment
from langbot_plugin.api.entities.builtin.provider.message import ContentElement
from ..agent.runner import input_media
# Pipeline queries pin the media they rendered; only bare event envelopes
# may describe media in contents alone.
envelope_only = input_media.remembered_media(query) is None
media = input_media.media_by_ref(query)
components = {ref: item.component for ref, item in media.items()}
# Envelope attachments registered by an earlier preparation of this query.
@@ -52,13 +50,29 @@ def prepare_input_files(query, agent_input):
components.update(previous if isinstance(previous, dict) else {})
if not agent_input.attachments:
agent_input.attachments = [InputAttachment.model_validate(item.to_attachment()) for item in media.values()]
if not agent_input.attachments and envelope_only:
agent_input.attachments = _attachments_from_contents(agent_input.contents)
def source_key(item):
return item.type, item.url, item.content, item.name
by_source = {source_key(item): item for item in agent_input.attachments}
content_attachments = {}
for index, content in enumerate(agent_input.contents):
described = _attachments_from_contents([content])
if not described:
continue
candidate = described[0]
key = source_key(candidate)
attachment = by_source.get(key)
if attachment is None:
attachment = by_source[key] = candidate
agent_input.attachments.append(attachment)
content_attachments[index] = attachment
for attachment in agent_input.attachments:
attachment.path = None
if attachment.ref and attachment.ref in components:
attachment.content = None
attachment.url = None
continue
kind = (attachment.type or 'file').lower()
cls = {'image': pm.Image, 'voice': pm.Voice}.get(kind, pm.File)
@@ -76,9 +90,22 @@ def prepare_input_files(query, agent_input):
index += 1
ref = f'att-{index}'
components[ref] = cls(**fields)
attachment.ref, attachment.content = ref, None
attachment.ref, attachment.content, attachment.url = ref, None, None
object.__setattr__(query, '_input_components', components)
normalized = []
for index, content in enumerate(agent_input.contents):
if content.type not in {'image_url', 'image_base64', 'file_url', 'file_base64'}:
normalized.append(content)
continue
attachment = content_attachments.get(index)
if attachment is not None:
label = f'{attachment.type} {attachment.name}' if attachment.name else attachment.type
normalized.append(ContentElement.from_text(f'[attachment {attachment.ref}: {label}]'))
else:
normalized.append(ContentElement.from_text('[attachment unavailable]'))
agent_input.contents = normalized
def _attachments_from_contents(contents):
"""Describe media that an event envelope carried only in its contents."""
+8 -1
View File
@@ -128,5 +128,12 @@
"Artifact is still in use": "Artifact is still in use",
"Artifact exceeds the read size limit": "Artifact exceeds the read size limit",
"Artifact content failed integrity verification": "Artifact content failed integrity verification",
"Artifact exceeds the size limit.": "Artifact exceeds the size limit."
"Artifact exceeds the size limit.": "Artifact exceeds the size limit.",
"This file type cannot be read by the model. Download the original file instead.": "This file type cannot be read by the model. Download the original file instead.",
"PDF page rendering is unavailable. Use text extraction or a model with native PDF support.": "PDF page rendering is unavailable. Use text extraction or a model with native PDF support.",
"The PDF could not be read. It may be malformed or password-protected.": "The PDF could not be read. It may be malformed or password-protected.",
"PDF text extraction excludes images and page layout. Use native mode with a supported model for visual content.": "PDF text extraction excludes images and page layout. Use native mode with a supported model for visual content.",
"PDF text extraction exceeds the size limit.": "PDF text extraction exceeds the size limit.",
"The selected model does not support native PDF input; read the PDF as text instead.": "The selected model does not support native PDF input; read the PDF as text instead.",
"Input attachment content budget for this run is exhausted": "Input attachment content budget for this run is exhausted"
}
+8 -1
View File
@@ -128,5 +128,12 @@
"Artifact is still in use": "El artefacto sigue en uso",
"Artifact exceeds the read size limit": "El artefacto supera el límite de tamaño de lectura",
"Artifact content failed integrity verification": "Falló la verificación de integridad del contenido del artefacto",
"Artifact exceeds the size limit.": "El artefacto supera el límite de tamaño."
"Artifact exceeds the size limit.": "El artefacto supera el límite de tamaño.",
"This file type cannot be read by the model. Download the original file instead.": "El modelo no puede leer este tipo de archivo. Descargue el original.",
"PDF page rendering is unavailable. Use text extraction or a model with native PDF support.": "La representación de páginas PDF no está disponible. Extraiga texto o use un modelo compatible con PDF nativo.",
"The PDF could not be read. It may be malformed or password-protected.": "No se pudo leer el PDF. Puede estar dañado o protegido por contraseña.",
"PDF text extraction excludes images and page layout. Use native mode with a supported model for visual content.": "La extracción de texto PDF no incluye imágenes ni diseño. Use el modo nativo con un modelo compatible para el contenido visual.",
"PDF text extraction exceeds the size limit.": "La extracción de texto PDF supera el límite de tamaño.",
"The selected model does not support native PDF input; read the PDF as text instead.": "El modelo seleccionado no admite PDF nativo; lea el PDF como texto.",
"Input attachment content budget for this run is exhausted": "Se agotó el límite de lectura de contenido adjunto de esta ejecución"
}
+8 -1
View File
@@ -128,5 +128,12 @@
"Artifact is still in use": "アーティファクトは使用中です",
"Artifact exceeds the read size limit": "アーティファクトが読み取りサイズの上限を超えています",
"Artifact content failed integrity verification": "アーティファクトの内容の整合性検証に失敗しました",
"Artifact exceeds the size limit.": "アーティファクトがサイズの上限を超えています。"
"Artifact exceeds the size limit.": "アーティファクトがサイズの上限を超えています。",
"This file type cannot be read by the model. Download the original file instead.": "モデルはこの形式を読み取れません。元のファイルをダウンロードしてください。",
"PDF page rendering is unavailable. Use text extraction or a model with native PDF support.": "PDFページ描画は利用できません。テキスト抽出またはPDF対応モデルを使用してください。",
"The PDF could not be read. It may be malformed or password-protected.": "PDFを読み取れません。破損またはパスワード保護の可能性があります。",
"PDF text extraction excludes images and page layout. Use native mode with a supported model for visual content.": "PDFテキスト抽出には画像やレイアウトは含まれません。視覚情報には対応モデルのnativeモードを使用してください。",
"PDF text extraction exceeds the size limit.": "PDFテキスト抽出がサイズ制限を超えました。",
"The selected model does not support native PDF input; read the PDF as text instead.": "選択したモデルはPDF入力に対応していません。PDFをテキストとして読み取ってください。",
"Input attachment content budget for this run is exhausted": "この実行の添付内容読み取り上限に達しました"
}
+8 -1
View File
@@ -128,5 +128,12 @@
"Artifact is still in use": "Артефакт всё ещё используется",
"Artifact exceeds the read size limit": "Артефакт превышает ограничение размера чтения",
"Artifact content failed integrity verification": "Проверка целостности содержимого артефакта не пройдена",
"Artifact exceeds the size limit.": "Артефакт превышает ограничение размера."
"Artifact exceeds the size limit.": "Артефакт превышает ограничение размера.",
"This file type cannot be read by the model. Download the original file instead.": "Модель не может прочитать этот тип файла. Скачайте оригинал.",
"PDF page rendering is unavailable. Use text extraction or a model with native PDF support.": "Отрисовка страниц PDF недоступна. Используйте извлечение текста или модель с поддержкой PDF.",
"The PDF could not be read. It may be malformed or password-protected.": "Не удалось прочитать PDF. Возможно, файл повреждён или защищён паролем.",
"PDF text extraction excludes images and page layout. Use native mode with a supported model for visual content.": "Извлечение текста PDF не включает изображения и макет. Для визуального содержимого используйте native с совместимой моделью.",
"PDF text extraction exceeds the size limit.": "Извлечение текста PDF превышает ограничение размера.",
"The selected model does not support native PDF input; read the PDF as text instead.": "Выбранная модель не поддерживает PDF; прочитайте PDF как текст.",
"Input attachment content budget for this run is exhausted": "Лимит чтения содержимого вложений этого запуска исчерпан"
}
+8 -1
View File
@@ -128,5 +128,12 @@
"Artifact is still in use": "อาร์ติแฟกต์ยังถูกใช้งานอยู่",
"Artifact exceeds the read size limit": "อาร์ติแฟกต์มีขนาดเกินขีดจำกัดการอ่าน",
"Artifact content failed integrity verification": "การตรวจสอบความสมบูรณ์ของเนื้อหาอาร์ติแฟกต์ล้มเหลว",
"Artifact exceeds the size limit.": "อาร์ติแฟกต์มีขนาดเกินขีดจำกัด"
"Artifact exceeds the size limit.": "อาร์ติแฟกต์มีขนาดเกินขีดจำกัด",
"This file type cannot be read by the model. Download the original file instead.": "โมเดลอ่านไฟล์ชนิดนี้ไม่ได้ โปรดดาวน์โหลดไฟล์ต้นฉบับ",
"PDF page rendering is unavailable. Use text extraction or a model with native PDF support.": "ยังไม่รองรับการเรนเดอร์หน้า PDF โปรดแยกข้อความหรือใช้โมเดลที่รองรับ PDF โดยตรง",
"The PDF could not be read. It may be malformed or password-protected.": "อ่าน PDF ไม่ได้ ไฟล์อาจเสียหายหรือมีรหัสผ่าน",
"PDF text extraction excludes images and page layout. Use native mode with a supported model for visual content.": "การแยกข้อความ PDF ไม่รวมรูปภาพและเค้าโครง โปรดใช้โหมด native กับโมเดลที่รองรับสำหรับเนื้อหาภาพ",
"PDF text extraction exceeds the size limit.": "การแยกข้อความ PDF เกินขีดจำกัดขนาด",
"The selected model does not support native PDF input; read the PDF as text instead.": "โมเดลที่เลือกไม่รองรับ PDF โดยตรง โปรดอ่าน PDF เป็นข้อความ",
"Input attachment content budget for this run is exhausted": "งบการอ่านเนื้อหาไฟล์แนบของการทำงานนี้หมดแล้ว"
}
+8 -1
View File
@@ -128,5 +128,12 @@
"Artifact is still in use": "Tài nguyên vẫn đang được sử dụng",
"Artifact exceeds the read size limit": "Tài nguyên vượt quá giới hạn kích thước đọc",
"Artifact content failed integrity verification": "Xác minh tính toàn vẹn nội dung tài nguyên không thành công",
"Artifact exceeds the size limit.": "Tài nguyên vượt quá giới hạn kích thước."
"Artifact exceeds the size limit.": "Tài nguyên vượt quá giới hạn kích thước.",
"This file type cannot be read by the model. Download the original file instead.": "Mô hình không đọc được loại tệp này. Hãy tải tệp gốc xuống.",
"PDF page rendering is unavailable. Use text extraction or a model with native PDF support.": "Chưa hỗ trợ kết xuất trang PDF. Hãy trích xuất văn bản hoặc dùng mô hình hỗ trợ PDF gốc.",
"The PDF could not be read. It may be malformed or password-protected.": "Không đọc được PDF. Tệp có thể bị hỏng hoặc được bảo vệ bằng mật khẩu.",
"PDF text extraction excludes images and page layout. Use native mode with a supported model for visual content.": "Trích xuất văn bản PDF không bao gồm hình ảnh và bố cục. Hãy dùng chế độ native với mô hình phù hợp cho nội dung trực quan.",
"PDF text extraction exceeds the size limit.": "Trích xuất văn bản PDF vượt giới hạn kích thước.",
"The selected model does not support native PDF input; read the PDF as text instead.": "Mô hình đã chọn không hỗ trợ PDF gốc; hãy đọc PDF dưới dạng văn bản.",
"Input attachment content budget for this run is exhausted": "Đã hết hạn mức đọc nội dung đính kèm của lần chạy này"
}
+8 -1
View File
@@ -128,5 +128,12 @@
"Artifact is still in use": "资产仍在使用中",
"Artifact exceeds the read size limit": "资产超过读取大小限制",
"Artifact content failed integrity verification": "资产内容完整性校验失败",
"Artifact exceeds the size limit.": "资产超过大小限制。"
"Artifact exceeds the size limit.": "资产超过大小限制。",
"This file type cannot be read by the model. Download the original file instead.": "模型无法读取此文件类型,请下载原文件。",
"PDF page rendering is unavailable. Use text extraction or a model with native PDF support.": "暂不支持 PDF 页面渲染,请提取文本或使用支持原生 PDF 输入的模型。",
"The PDF could not be read. It may be malformed or password-protected.": "无法读取 PDF,文件可能损坏或受密码保护。",
"PDF text extraction excludes images and page layout. Use native mode with a supported model for visual content.": "PDF 文本提取不包含图片和页面布局。需要视觉内容时,请使用受支持模型的原生模式。",
"PDF text extraction exceeds the size limit.": "PDF 文本提取超过大小限制。",
"The selected model does not support native PDF input; read the PDF as text instead.": "所选模型不支持原生 PDF 输入,请改为读取 PDF 文本。",
"Input attachment content budget for this run is exhausted": "本次运行的附件内容读取预算已用尽"
}
+8 -1
View File
@@ -128,5 +128,12 @@
"Artifact is still in use": "資產仍在使用中",
"Artifact exceeds the read size limit": "資產超過讀取大小限制",
"Artifact content failed integrity verification": "資產內容完整性驗證失敗",
"Artifact exceeds the size limit.": "資產超過大小限制。"
"Artifact exceeds the size limit.": "資產超過大小限制。",
"This file type cannot be read by the model. Download the original file instead.": "模型無法讀取此檔案類型,請下載原始檔案。",
"PDF page rendering is unavailable. Use text extraction or a model with native PDF support.": "暫不支援 PDF 頁面渲染,請擷取文字或使用支援原生 PDF 輸入的模型。",
"The PDF could not be read. It may be malformed or password-protected.": "無法讀取 PDF,檔案可能損壞或受密碼保護。",
"PDF text extraction excludes images and page layout. Use native mode with a supported model for visual content.": "PDF 文字擷取不包含圖片和頁面配置。需要視覺內容時,請使用受支援模型的原生模式。",
"PDF text extraction exceeds the size limit.": "PDF 文字擷取超過大小限制。",
"The selected model does not support native PDF input; read the PDF as text instead.": "所選模型不支援原生 PDF 輸入,請改為讀取 PDF 文字。",
"Input attachment content budget for this run is exhausted": "本次執行的附件內容讀取額度已用盡"
}
@@ -346,7 +346,6 @@ class PreProcessor(stage.PipelineStage):
rendered = input_media.render_input(
query.message_chain,
include_quotes=bool(quote_msg),
keep_images=keep_image_inputs,
)
# Runner attachments and Box imports address exactly the media rendered here.
input_media.remember_media(query, rendered.media)
@@ -1,35 +1,33 @@
"""Run-scoped access to the bytes of current-event input attachments."""
"""Run-authorized input reads, backed by Artifact storage on first model read."""
from __future__ import annotations
import asyncio
import base64
from dataclasses import replace
import mimetypes
from typing import Any
from langbot_plugin.entities.io.actions.enums import PluginToRuntimeAction as Action
from langbot_plugin.runtime.io.handler import ActionResponse
from ..agent.runner import input_media
from ..api.context import PrincipalContext, PrincipalType
from ..box.runner import input_files
from ..i18n import t
from ..storage.artifact_reader import read_artifact_content
from .agent_run_support import _validate_agent_run_session
def register(h):
@h.action(Action.READ_INPUT_ATTACHMENT)
async def read_input_attachment(data: dict[str, Any]) -> ActionResponse:
"""Resolve one attachment of the run's own input by ref.
Bytes are resolved on demand and never cached or logged; each read is
bounded per attachment and cumulatively per run.
"""
async def handle(data, *, render):
from .handler import _resolve_action_query
run_id = data.get('run_id')
ref = data.get('ref')
run_id, ref = data.get('run_id'), data.get('ref')
if not run_id:
return ActionResponse.error('run_id is required')
if not isinstance(ref, str) or not ref:
return ActionResponse.error('ref is required')
action_context, _ = await h._require_plugin_action_context()
session, error = await _validate_agent_run_session(
run_id,
@@ -43,33 +41,97 @@ def register(h):
query = _resolve_action_query(data, session, h.ap, action_context)
if query is None:
return ActionResponse.error('The current execution context has expired')
component = input_files(query).get(ref)
if component is None:
return ActionResponse.error(f'Unknown input attachment reference: {ref}')
already_read = getattr(query, '_input_attachment_read_bytes', 0)
remaining = input_media.MAX_RUN_READ_BYTES - already_read
if remaining <= 0:
return ActionResponse.error('Input attachment read budget for this run is exhausted')
resolved = await input_media.read_component_bytes(
component,
max_bytes=min(input_media.MAX_ATTACHMENT_BYTES, remaining),
adapter=getattr(query, 'adapter', None),
)
if resolved is None:
return ActionResponse.error(
f'Input attachment {ref} cannot be read: no payload is available or it exceeds the size limit'
lock = getattr(query, '_input_attachment_lock', None)
if lock is None:
lock = asyncio.Lock()
object.__setattr__(query, '_input_attachment_lock', lock)
async with lock:
already_read = getattr(query, '_input_attachment_read_bytes', 0)
remaining = input_media.MAX_RUN_READ_BYTES - already_read
materialized = getattr(query, '_input_materialized_refs', set())
charge_source = not render or ref not in materialized
if charge_source and remaining <= 0:
return ActionResponse.error('Input attachment read budget for this run is exhausted')
returned_bytes = getattr(query, '_input_content_read_bytes', 0)
if render and returned_bytes >= input_media.MAX_RUN_READ_BYTES:
return ActionResponse.error(t('Input attachment content budget for this run is exhausted'))
limit = (
min(input_media.MAX_ATTACHMENT_BYTES, remaining) if charge_source else input_media.MAX_ATTACHMENT_BYTES
)
payload, mime_type = resolved
object.__setattr__(query, '_input_attachment_read_bytes', already_read + len(payload))
h.ap.logger.debug(f'Read input attachment {ref} for run {run_id}: {len(payload)} bytes')
return ActionResponse.success(
{
'ref': ref,
'name': getattr(component, 'name', None) or None,
'mime_type': mime_type,
'size': len(payload),
'content': base64.b64encode(payload).decode('ascii'),
}
)
uploaded_ids = getattr(query, '_input_artifact_ids', frozenset())
cached = getattr(query, '_input_artifact_refs', {})
run_context = getattr(query, '_execution_context', None)
if run_context is not None:
run_context = replace(
run_context, trigger_principal=PrincipalContext(PrincipalType.PROCESSOR, run_id=run_id)
)
if ref in uploaded_ids:
original_context = query._input_artifact_context
result = await h.ap.artifact_service.read(original_context, ref, max_bytes=limit)
payload, metadata = result['content'], result['metadata']
elif ref in cached:
result = await h.ap.artifact_service.read_run_input(run_context, cached[ref], max_bytes=limit)
payload, metadata = result['content'], result['metadata']
else:
component = input_files(query).get(ref)
if component is None:
return ActionResponse.error(f'Unknown input attachment reference: {ref}')
resolved = await input_media.read_component_bytes(
component,
max_bytes=limit,
adapter=getattr(query, 'adapter', None),
)
if resolved is None:
return ActionResponse.error(
f'Input attachment {ref} cannot be read: no payload is available or it exceeds the size limit'
)
payload, mime_type = resolved
name = getattr(component, 'name', None) or None
metadata = {'ref': ref, 'name': name, 'mime_type': mime_type, 'size': len(payload)}
if render:
name = (name or ref).replace('\\', '/').rsplit('/', 1)[-1]
mime_type = mime_type.split(';', 1)[0].strip()
if mime_type == 'application/octet-stream':
mime_type = mimetypes.guess_type(name)[0] or mime_type
metadata = await h.ap.artifact_service.create_run_input(run_context, name, payload, mime_type)
cached[ref] = metadata['id']
object.__setattr__(query, '_input_artifact_refs', cached)
if charge_source:
object.__setattr__(query, '_input_attachment_read_bytes', already_read + len(payload))
if render:
materialized.add(ref)
object.__setattr__(query, '_input_materialized_refs', materialized)
contents = await read_artifact_content(
payload,
{**metadata, 'ref': ref},
**{
key: data[key]
for key in ('mode', 'offset', 'max_chars', 'page_start', 'page_end')
if key in data
},
)
# Text continuation pays for returned content, not the entire cached source again.
output_size = sum(len(item['text'].encode('utf-8')) for item in contents if item['type'] == 'text')
if any(item['type'] != 'text' for item in contents):
output_size += len(payload)
if output_size > input_media.MAX_RUN_READ_BYTES - returned_bytes:
return ActionResponse.error(t('Input attachment content budget for this run is exhausted'))
object.__setattr__(query, '_input_content_read_bytes', returned_bytes + output_size)
return ActionResponse.success({'contents': contents})
return ActionResponse.success(
{
'ref': ref,
'name': metadata['name'],
'mime_type': metadata['mime_type'],
'size': len(payload),
'content': base64.b64encode(payload).decode('ascii'),
}
)
@h.action(Action.READ_INPUT_ATTACHMENT)
async def read_input_attachment(data: dict[str, Any]) -> ActionResponse:
return await handle(data, render=False)
@h.action(Action.READ_INPUT_ATTACHMENT_CONTENT)
async def read_input_attachment_content(data: dict[str, Any]) -> ActionResponse:
return await handle(data, render=True)
@@ -364,6 +364,10 @@ class LiteLLMRequester(requester.ProviderAPIRequester):
def _uncached_litellm_bool_helper(self, helper_name, model_name):
helper = getattr(litellm, helper_name, None)
if helper_name == 'supports_pdf_input' and not callable(helper):
from litellm.utils import supports_pdf_input
helper = supports_pdf_input
if not callable(helper):
return False
@@ -880,6 +884,8 @@ class LiteLLMRequester(requester.ProviderAPIRequester):
messages: typing.List[provider_message.Message],
reasoning_family: str = '',
include_reasoning_context: bool = True,
model_name: str | None = None,
pdf_input_declared: bool = False,
) -> list[dict]:
"""Convert LangBot messages to LiteLLM/OpenAI format."""
req_messages = []
@@ -955,13 +961,30 @@ class LiteLLMRequester(requester.ProviderAPIRequester):
part.pop('image_base64', None)
else:
continue
# OpenAI-compatible chat models reject non-image file parts
# (audio/document base64 or url). These originate from Voice /
# File attachments — including ones replayed from conversation
# history — and the agent already accesses their bytes via the
# sandbox. Drop them from the model payload to avoid
# "Invalid user message ... invalid content type=file_base64".
# Explicit PDF input must succeed or fail visibly; keep other legacy file handling.
if isinstance(part, dict) and part.get('type') in ('file_base64', 'file_url'):
file_data = part.get('file_base64') or part.get('file_url')
if isinstance(file_data, str) and file_data.startswith('data:application/pdf;base64,'):
supported = pdf_input_declared or (
model_name is not None
and self._safe_litellm_bool_helper('supports_pdf_input', model_name)
)
if not supported:
raise errors.RequesterError(
t(
'The selected model does not support native PDF input; read the PDF as text instead.'
),
code='unsupported_pdf_input',
)
converted_parts.append(
{
'type': 'file',
'file': {
'file_data': file_data,
'filename': part.get('file_name') or 'document.pdf',
},
}
)
continue
converted_parts.append(part)
msg_dict['content'] = converted_parts
@@ -1500,13 +1523,25 @@ class LiteLLMRequester(requester.ProviderAPIRequester):
model = scoped_model
reasoning_family = self._reasoning_family(model.model_entity.name, model)
reasoning_level = self._reasoning_level(model)
message_key = ('messages', id(messages), reasoning_family, reasoning_level != 'disabled')
pdf_input_declared = 'pdf_input' in (model.model_entity.abilities or [])
message_key = (
'messages',
id(messages),
reasoning_family,
reasoning_level != 'disabled',
model.model_entity.name,
pdf_input_declared,
)
if cache_preparation and message_key in state.prepared:
req_messages = copy.deepcopy(state.prepared[message_key])
state.metrics['preparation_cache_hits'] = state.metrics.get('preparation_cache_hits', 0) + 1
else:
req_messages = self._convert_messages(
messages, reasoning_family=reasoning_family, include_reasoning_context=reasoning_level != 'disabled'
messages,
reasoning_family=reasoning_family,
include_reasoning_context=reasoning_level != 'disabled',
model_name=model.model_entity.name,
pdf_input_declared=pdf_input_declared,
)
if cache_preparation:
state.prepared[message_key] = copy.deepcopy(req_messages)
@@ -1610,6 +1645,8 @@ class LiteLLMRequester(requester.ProviderAPIRequester):
messages,
reasoning_family=self._reasoning_family(model.model_entity.name, model),
include_reasoning_context=reasoning_config['level'] != 'disabled',
model_name=model.model_entity.name,
pdf_input_declared='pdf_input' in (model.model_entity.abilities or []),
)
chat_provider = 'openrouter' if self._uses_openrouter_protocol() else self._get_custom_llm_provider()
self._adapt_messages_for_provider(req_messages, chat_provider)
+19 -1
View File
@@ -4,6 +4,7 @@ from ....i18n import t
import contextvars
import dataclasses
import base64
import enum
import json
import math
@@ -11,7 +12,7 @@ import re
import time
import typing
import ipaddress
from urllib.parse import urlparse
from urllib.parse import unquote, urlparse
from contextlib import AsyncExitStack, asynccontextmanager
from datetime import timedelta
import traceback
@@ -60,6 +61,7 @@ MCP_RESOURCE_CACHE_MAX_ENTRIES = 32
MCP_RESOURCE_PREVIEW_MAX_BYTES = 64 * 1024
MCP_RESOURCE_AGENT_READ_MAX_BYTES = 64 * 1024
MCP_RESOURCE_AGENT_READ_MAX_TOKENS = 12000
MCP_TOOL_PDF_MAX_BYTES = 10 * 1024 * 1024
MCP_RESOURCE_CONTEXT_MAX_TOKENS = 8000
MCP_RESOURCE_CONTEXT_MAX_BYTES = 96 * 1024
MCP_RESOURCE_TRACE_QUERY_KEY = '_mcp_resource_reads'
@@ -1219,6 +1221,22 @@ class RuntimeMCPSession:
}
return [provider_message.ContentElement.from_text(f'{json.dumps(header, ensure_ascii=False)}\n{text}')]
if isinstance(resource, mcp_types.BlobResourceContents):
if (resource.mimeType or '').split(';', 1)[0].strip().lower() == 'application/pdf':
if len(resource.blob) > 4 * ((MCP_TOOL_PDF_MAX_BYTES + 2) // 3):
raise ValueError(t('Artifact exceeds the read size limit'))
try:
value = base64.b64decode(resource.blob, validate=True)
except ValueError as exc:
raise ValueError('MCP tool returned invalid PDF base64') from exc
if len(value) > MCP_TOOL_PDF_MAX_BYTES:
raise ValueError(t('Artifact exceeds the read size limit'))
filename = unquote(urlparse(str(resource.uri)).path).rsplit('/', 1)[-1] or 'document.pdf'
return [
provider_message.ContentElement.from_file_base64(
'data:application/pdf;base64,' + resource.blob,
filename,
)
]
return [
provider_message.ContentElement.from_text(
json.dumps(
+198
View File
@@ -0,0 +1,198 @@
"""Bounded model-facing reads of already-authorized Artifact bytes."""
from __future__ import annotations
import asyncio
import base64
import io
import json
import re
from ..i18n import t
from .artifacts import ArtifactError, MAX_ARTIFACT_BYTES
MAX_TEXT_CHARS = 16384
MAX_PDF_PAGES = 10
MAX_PDF_EXTRACTED_CHARS = 1024 * 1024
MAX_PDF_STREAM_BYTES = 2 * 1024 * 1024
def _unsupported(message: str):
return ArtifactError('artifact_read_unsupported', t(message), 415)
def _text_result(metadata: dict, mode: str, **result) -> list[dict]:
public = {
key: metadata[key] for key in ('id', 'ref', 'name', 'mime_type', 'size', 'digest', 'source') if key in metadata
}
return [{'type': 'text', 'text': json.dumps({'artifact': public, 'mode': mode, **result}, ensure_ascii=False)}]
def _slice(text: str, offset: int, max_chars: int) -> dict:
if offset > len(text):
raise ValueError('Artifact offset exceeds the text length')
end = min(len(text), offset + max_chars)
return {
'offset': offset,
'end_offset': end,
'total_chars': len(text),
'truncated': end < len(text),
'next_offset': end if end < len(text) else None,
'text': text[offset:end],
}
def _extract_pdf(content: bytes, page_start: int, page_end: int | None) -> tuple[str, dict]:
from pypdf import PdfReader, apply_configuration
from pypdf.errors import PdfReadError
# pypdf's ContextVar scopes decoding and parsing budgets to this read, including nested forms.
with apply_configuration(
maximum_declared_stream_length=MAX_PDF_STREAM_BYTES,
array_based_stream_maximum_output_length=MAX_PDF_STREAM_BYTES,
array_maximum_element_count=10000,
dictionary_maximum_element_count=10000,
zlib_maximum_output_length=MAX_PDF_STREAM_BYTES,
lzw_maximum_output_length=MAX_PDF_STREAM_BYTES,
run_length_maximum_output_length=MAX_PDF_STREAM_BYTES,
brotli_maximum_output_length=MAX_PDF_STREAM_BYTES,
jbig2_maximum_output_length=MAX_PDF_STREAM_BYTES,
image_maximum_buffer_size=MAX_PDF_STREAM_BYTES,
stream_decoding_work_maximum_length=4 * MAX_PDF_STREAM_BYTES,
stream_parsing_work_maximum_count=100000,
stream_filters_maximum_length=8,
flate_maximum_columns=10000,
flate_maximum_row_length=65536,
page_tree_maximum_entries=1000,
page_tree_maximum_depth=32,
xform_maximum_invocations_per_extraction=100,
jbig2dec_binary=None,
disable_legacy_handling=True,
):
reader = PdfReader(io.BytesIO(content), strict=True)
if reader.is_encrypted:
raise PdfReadError('Encrypted PDF')
page_count = len(reader.pages)
if page_start > page_count:
raise ArtifactError('invalid_input', 'PDF page_start exceeds the page count', 400)
last_page = min(page_start + MAX_PDF_PAGES - 1, page_count) if page_end is None else page_end
if last_page > page_count:
raise ArtifactError('invalid_input', 'PDF page_end exceeds the page count', 400)
parts = []
chars = 0
for index in range(page_start - 1, last_page):
page = reader.pages[index]
text = page.extract_text() or ''
chars += len(text)
if chars > MAX_PDF_EXTRACTED_CHARS:
raise ArtifactError('artifact_too_large', t('PDF text extraction exceeds the size limit.'), 413)
parts.append(f'[Page {index + 1}]\n{text}')
return '\n\n'.join(parts), {
'page_start': page_start,
'page_end': last_page,
'page_count': page_count,
'next_page': last_page + 1 if last_page < page_count else None,
'text_only': True,
'notice': t(
'PDF text extraction excludes images and page layout. Use native mode with a supported model for visual content.'
),
}
def _pdf_text(content: bytes, page_start: int, page_end: int | None) -> tuple[str, dict]:
from pypdf.errors import LimitReachedError, PdfReadError
try:
return _extract_pdf(content, page_start, page_end)
except (LimitReachedError, MemoryError) as exc:
raise ArtifactError('artifact_too_large', t('PDF text extraction exceeds the size limit.'), 413) from exc
except (PdfReadError, NotImplementedError, KeyError, TypeError, ValueError, IndexError, RecursionError) as exc:
raise ArtifactError(
'artifact_pdf_unreadable',
t('The PDF could not be read. It may be malformed or password-protected.'),
422,
) from exc
def _image_mime(content: bytes) -> str | None:
if content.startswith(b'\x89PNG\r\n\x1a\n'):
return 'image/png'
if content.startswith(b'\xff\xd8\xff'):
return 'image/jpeg'
if content.startswith((b'GIF87a', b'GIF89a')):
return 'image/gif'
if content.startswith(b'RIFF') and content[8:12] == b'WEBP':
return 'image/webp'
return None
async def read_artifact_content(
content: bytes,
metadata: dict,
*,
mode: str = 'auto',
offset: int = 0,
max_chars: int = 8192,
page_start: int = 1,
page_end: int | None = None,
) -> list[dict]:
"""Read text ranges or return structured image/native-PDF input after authorization."""
if not isinstance(content, bytes) or len(content) > MAX_ARTIFACT_BYTES:
raise ValueError('Artifact content must be bytes within the 10 MiB limit')
if not isinstance(mode, str) or mode not in {'auto', 'text', 'native', 'pages'}:
raise ValueError('Artifact mode must be auto, text, native or pages')
if type(offset) is not int or offset < 0:
raise ValueError('Artifact offset must be a non-negative character index')
if type(max_chars) is not int or not 1 <= max_chars <= MAX_TEXT_CHARS:
raise ValueError('Artifact max_chars must be between 1 and 16384')
if type(page_start) is not int or page_start < 1:
raise ValueError('PDF page_start must be a positive page number')
if page_end is not None and (
type(page_end) is not int or page_end < page_start or page_end - page_start >= MAX_PDF_PAGES
):
raise ValueError('PDF page range must contain between 1 and 10 pages')
mime_type = metadata.get('mime_type') or 'application/octet-stream'
is_pdf = content.lstrip(b'\r\n\t ').startswith(b'%PDF-')
if mode == 'pages':
raise _unsupported('PDF page rendering is unavailable. Use text extraction or a model with native PDF support.')
image_mime = _image_mime(content)
if image_mime and mode in {'auto', 'native'}:
if offset or page_start != 1 or page_end is not None:
raise ValueError('Image reads do not accept text offsets or PDF page ranges')
return _text_result(metadata, 'image') + [
{
'type': 'image_base64',
'image_base64': f'data:{image_mime};base64,' + base64.b64encode(content).decode('ascii'),
}
]
if is_pdf or mime_type == 'application/pdf':
if not is_pdf:
raise ArtifactError(
'artifact_pdf_unreadable',
t('The PDF could not be read. It may be malformed or password-protected.'),
422,
)
if mode == 'native':
if offset or page_start != 1 or page_end is not None:
raise ValueError('Native PDF mode reads the complete document, without text offsets or page ranges')
return _text_result(metadata, 'native') + [
{
'type': 'file_base64',
'file_name': metadata.get('name') or 'document.pdf',
'file_base64': 'data:application/pdf;base64,' + base64.b64encode(content).decode('ascii'),
}
]
text, pages = await asyncio.to_thread(_pdf_text, content, page_start, page_end)
return _text_result(metadata, 'text', **pages, **_slice(text, offset, max_chars))
if mode == 'native' or image_mime:
raise _unsupported('This file type cannot be read by the model. Download the original file instead.')
if page_start != 1 or page_end is not None:
raise ValueError('PDF page ranges cannot be used with a text file')
try:
text = content.decode('utf-8')
except UnicodeDecodeError as exc:
raise _unsupported('This file type cannot be read by the model. Download the original file instead.') from exc
if re.search(r'[\x00-\x08\x0b\x0c\x0e-\x1f\x7f]', text):
raise _unsupported('This file type cannot be read by the model. Download the original file instead.')
return _text_result(metadata, 'text', **_slice(text, offset, max_chars))
+31 -2
View File
@@ -88,7 +88,7 @@ class ArtifactService:
'mime_type': row['mime_type'],
'size': row['size'],
'digest': row['digest'],
'source': 'upload',
'source': 'input_attachment' if row['owner_type'] == 'run' else 'upload',
'lifecycle': 'persistent' if row['expires_at'] is None else 'temporary',
'created_at': timestamp(row['created_at']),
'expires_at': timestamp(row['expires_at']),
@@ -103,6 +103,26 @@ class ArtifactService:
ttl_seconds: int = DEFAULT_TTL_SECONDS,
) -> dict:
owner = await self._authorize(context, write=True)
return await self._create_owned(context, owner, filename, content, mime_type, ttl_seconds)
@staticmethod
def _run_owner(context):
principal = context.trigger_principal if isinstance(context, ExecutionContext) else None
if principal is None or principal.principal_type != PrincipalType.PROCESSOR or not principal.run_id:
raise PermissionError('Input assets require a trusted active run context')
return 'run', principal.run_id
async def create_run_input(self, context, filename, content, mime_type):
owner = self._run_owner(context)
await self.content._validate(context)
return await self._create_owned(context, owner, filename, content, mime_type, DEFAULT_TTL_SECONDS)
async def read_run_input(self, context, artifact_id, *, max_bytes=MAX_ARTIFACT_BYTES):
owner = self._run_owner(context)
await self.content._validate(context)
return await self._read_owned(context, owner, artifact_id, max_bytes)
async def _create_owned(self, context, owner, filename, content, mime_type, ttl_seconds):
if (
not isinstance(filename, str)
or not filename
@@ -151,9 +171,12 @@ class ArtifactService:
return self._public(await self._row(uow, context, artifact_id, owner))
async def read(self, context, artifact_id: str, max_bytes: int = MAX_ARTIFACT_BYTES) -> dict:
owner = await self._authorize(context)
return await self._read_owned(context, owner, artifact_id, max_bytes)
async def _read_owned(self, context, owner, artifact_id, max_bytes):
from .content import ContentIntegrityError
owner = await self._authorize(context)
if type(max_bytes) is not int or not 1 <= max_bytes <= MAX_ARTIFACT_BYTES:
raise ValueError('Artifact read limit must be between 1 and 10485760 bytes')
async with self.ap.persistence_mgr.tenant_uow(context.workspace_uuid) as uow:
@@ -172,6 +195,12 @@ class ArtifactService:
raise ArtifactError('artifact_integrity_failed', 'Artifact content failed integrity verification', 422)
return {'metadata': self._public(row), 'content': value}
async def read_content(self, context, artifact_id, **options):
from .artifact_reader import read_artifact_content
result = await self.read(context, artifact_id)
return await read_artifact_content(result['content'], result['metadata'], **options)
async def _has_references(self, uow, context, artifact_id):
return (
await uow.execute(
@@ -343,6 +343,7 @@ async def _capture_sdk_payloads() -> dict[str, dict[str, typing.Any]]:
await try_call(runner.history_page())
await try_call(runner.history_search('query'))
await try_call(runner.read_input_attachment('att-1'))
await try_call(runner.read_input_attachment_content('att-1'))
await try_call(runner.event_get('evt-1'))
await try_call(runner.event_page())
await try_call(runner.steering_pull())
@@ -394,8 +395,8 @@ async def _capture_sdk_payloads() -> dict[str, dict[str, typing.Any]]:
def test_action_audit_and_catalog_completeness(test_host):
"""Audit that all business actions map to ExecutionAPI and transport controls stay internal."""
operations = test_host.api.operations()
assert len(operations) == 80
assert len(test_host.api.catalog()) == 80
assert len(operations) == 81
assert len(test_host.api.catalog()) == 81
execution_action_names = {op.action for op in operations}
handler_actions = set(test_host.handler.actions.keys())
@@ -0,0 +1,63 @@
"""Authorized HTTP and native MCP Artifact content, over real persistence."""
# ruff: noqa: F811
import base64
import io
import json
from mcp import types
import pytest
from quart.datastructures import FileStorage
from langbot.pkg.api.mcp.context import bind_request_context, reset_request_context
from langbot.pkg.api.mcp.server import LangBotMCPServer
from tests.integration.storage.test_artifacts import (
artifact_replicas, # noqa: F401 -- pytest fixture
persistence_case, # noqa: F401 -- fixture dependency
replicas, # noqa: F401
_data,
)
pytestmark = [pytest.mark.integration, pytest.mark.asyncio]
async def upload(case, name, value, mime):
return await _data(
await case.a.client.post(
'/api/v1/artifacts',
headers=case.headers,
files={
'file': FileStorage(io.BytesIO(value), filename=name, content_type=mime),
},
)
)
async def test_http_text_ranges_and_cross_core_mcp_image(artifact_replicas):
case = artifact_replicas
text = '甲乙😀丁戊'
metadata = await upload(case, 'note.txt', text.encode(), 'text/plain')
path = '/api/v1/artifacts/' + metadata['id'] + '/read'
response = await case.b.client.get(path + '?offset=1&max_chars=2', headers=case.headers)
assert response.headers['Cache-Control'] == 'private, no-store'
first = await _data(response)
result = json.loads(first[0]['text'])
assert result['text'] == '乙😀' and result['next_offset'] == 3 and result['truncated'] is True
assert (await case.b.client.get(path)).status_code == 401
assert (await case.b.client.get(path + '?offset=-1', headers=case.headers)).status_code == 400
assert (await case.b.client.get(path + '?mode=pages', headers=case.headers)).status_code == 415
png = base64.b64decode(
'iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAwMCAO+jWZkAAAAASUVORK5CYII='
)
image = await upload(case, 'pixel.png', png, 'image/png')
server = LangBotMCPServer(case.b.app)
token = bind_request_context(case.context)
try:
response = await server.mcp.call_tool('read_artifact_content', {'artifact_id': image['id']})
finally:
reset_request_context(token)
blocks = response.content if isinstance(response, types.CallToolResult) else response[0]
assert blocks[0].type == 'text'
assert blocks[1].type == 'image' and blocks[1].mimeType == 'image/png'
assert base64.b64decode(blocks[1].data) == png
assert base64.b64encode(png).decode() not in blocks[0].text
@@ -0,0 +1,75 @@
"""Run input assets cannot become account assets or cross the run boundary."""
# ruff: noqa: F811
from dataclasses import replace
import uuid
import pytest
from langbot.pkg.api.context import PrincipalContext, PrincipalType
from langbot.pkg.storage.artifacts import ArtifactError
from tests.integration.storage.test_artifacts import (
_expire,
artifact_replicas, # noqa: F401 -- pytest fixture
persistence_case, # noqa: F401 -- fixture dependency
replicas, # noqa: F401 -- fixture dependency
)
pytestmark = [pytest.mark.integration, pytest.mark.asyncio]
_BODY = b'Only this authorized run can read this input.'
def _run_context(case, run_id=None):
return replace(
case.a.context,
trigger_principal=PrincipalContext(PrincipalType.PROCESSOR, run_id=run_id or str(uuid.uuid4())),
)
async def test_run_input_can_be_read_on_another_core_but_not_by_account_or_other_run(artifact_replicas):
case = artifact_replicas
context = _run_context(case)
metadata = await case.a.app.artifact_service.create_run_input(context, 'input.txt', _BODY, 'text/plain')
assert metadata['source'] == 'input_attachment' and metadata['lifecycle'] == 'temporary'
downloaded = await case.b.app.artifact_service.read_run_input(context, metadata['id'])
assert downloaded['content'] == _BODY and downloaded['metadata'] == metadata
with pytest.raises(FileNotFoundError):
await case.b.app.artifact_service.read_run_input(_run_context(case), metadata['id'])
for method, suffix in (('get', ''), ('get', '/content'), ('post', '/retain'), ('delete', '')):
response = await getattr(case.a.client, method)(
'/api/v1/artifacts/' + metadata['id'] + suffix,
headers=case.headers,
)
assert response.status_code == 404
private = await case.a.app.artifact_service.create(case.context, 'private.txt', _BODY, 'text/plain')
with pytest.raises(FileNotFoundError):
await case.a.app.artifact_service.read_run_input(context, private['id'])
async def test_run_input_expiry_and_collection_use_the_same_lifecycle(artifact_replicas):
case = artifact_replicas
context = _run_context(case)
metadata = await case.a.app.artifact_service.create_run_input(context, 'expired.txt', _BODY, 'text/plain')
await _expire(case.a, context, metadata['id'])
with pytest.raises(ArtifactError) as failure:
await case.b.app.artifact_service.read_run_input(context, metadata['id'])
assert failure.value.error_code == 'artifact_expired' and failure.value.status_code == 410
system = replace(case.a.context, trigger_principal=PrincipalContext(PrincipalType.SYSTEM))
assert await case.b.app.artifact_service.collect_expired(system) == 1
with pytest.raises(FileNotFoundError):
await case.a.app.artifact_service.read_run_input(context, metadata['id'])
async def test_run_input_internal_methods_require_a_run_execution_context(artifact_replicas):
case = artifact_replicas
for context in (
case.context,
case.a.context,
replace(case.a.context, trigger_principal=PrincipalContext(PrincipalType.SYSTEM)),
replace(case.a.context, trigger_principal=PrincipalContext(PrincipalType.PROCESSOR)),
):
with pytest.raises(PermissionError):
await case.a.app.artifact_service.create_run_input(context, 'input.txt', _BODY, 'text/plain')
with pytest.raises(PermissionError):
await case.a.app.artifact_service.read_run_input(context, 'art_' + '0' * 32)
@@ -0,0 +1,204 @@
"""The installed input action uses active sessions and real Artifact persistence."""
# ruff: noqa: F811
import asyncio
import base64
from dataclasses import replace
import json
from types import SimpleNamespace
import uuid
import pytest
import sqlalchemy as sa
from langbot_plugin.api.entities.builtin.platform.message import File, MessageChain
from langbot_plugin.api.entities.builtin.runner.resources import AgentResources
from langbot_plugin.entities.io.actions.enums import PluginToRuntimeAction as Action
from langbot.pkg.agent.runner import input_media
from langbot.pkg.agent.runner.session_registry import AgentRunSessionRegistry
from langbot.pkg.api.context import PrincipalContext, PrincipalType
from langbot.pkg.entity.persistence.artifact import Artifact
from langbot.pkg.plugin import agent_run_support, input_attachment_actions
from tests.integration.storage.test_artifacts import (
artifact_replicas, # noqa: F401
persistence_case, # noqa: F401
replicas, # noqa: F401
)
pytestmark = [pytest.mark.integration, pytest.mark.asyncio]
_BODY = 'A猫🙂Z'.encode()
@pytest.fixture
async def input_actions(artifact_replicas, monkeypatch):
case = artifact_replicas
registry = AgentRunSessionRegistry()
monkeypatch.setattr(agent_run_support, 'get_session_registry', lambda: registry)
run_id = str(uuid.uuid4())
query = SimpleNamespace(
message_chain=MessageChain([File(name='sample.txt', base64=base64.b64encode(_BODY).decode())]),
_execution_context=case.a.context,
)
async def register(run):
await registry.register(
run_id=run,
runner_id='plugin:test/runner/default',
query_id=None,
plugin_identity='test/runner',
resources=AgentResources().model_dump(mode='json'),
available_apis={'input_attachments': True},
execution_query=query,
workspace_id=case.context.workspace_uuid,
)
await register(run_id)
class Handler:
ap = case.a.app
def __init__(self):
self.actions = {}
def action(self, name):
def attach(function):
self.actions[name] = function
return function
return attach
async def _require_plugin_action_context(self):
return case.a.context, None
handler = Handler()
input_attachment_actions.register(handler)
return SimpleNamespace(
case=case,
query=query,
run_id=run_id,
registry=registry,
register=register,
render=handler.actions[Action.READ_INPUT_ATTACHMENT_CONTENT],
raw=handler.actions[Action.READ_INPUT_ATTACHMENT],
arguments={'run_id': run_id, 'ref': 'att-1', 'caller_plugin_identity': 'test/runner'},
)
async def _count(case):
async with case.a.app.persistence_mgr.tenant_uow(case.context.workspace_uuid) as uow:
return (await uow.execute(sa.select(sa.func.count()).select_from(Artifact))).scalar_one()
async def test_first_content_read_creates_one_run_asset_and_reuses_it(input_actions):
x = input_actions
response = await x.render({**x.arguments, 'offset': 1, 'max_chars': 2})
assert response.code == 0
result = json.loads(response.data['contents'][0]['text'])
assert result['text'] == '猫🙂' and result['next_offset'] == 3
assert result['artifact']['source'] == 'input_attachment'
asset_id = x.query._input_artifact_refs['att-1']
assert await _count(x.case) == 1
raw = await x.raw(x.arguments)
assert raw.code == 0 and base64.b64decode(raw.data['content']) == _BODY
assert await _count(x.case) == 1
assert x.query._input_attachment_read_bytes == 2 * len(_BODY)
other_run = str(uuid.uuid4())
await x.register(other_run)
with pytest.raises(FileNotFoundError):
await x.render({**x.arguments, 'run_id': other_run})
assert x.query._input_artifact_refs['att-1'] == asset_id
assert (await x.case.a.client.get('/api/v1/artifacts/' + asset_id, headers=x.case.headers)).status_code == 404
async def test_uploaded_debug_ids_keep_the_original_owner_context(input_actions):
x = input_actions
metadata = await x.case.a.app.artifact_service.create(x.case.context, 'uploaded.txt', _BODY, 'text/plain')
x.query._input_artifact_ids = frozenset({metadata['id']})
x.query._input_artifact_context = x.case.context
request = {**x.arguments, 'ref': metadata['id']}
response = await x.render(request)
result = json.loads(response.data['contents'][0]['text'])
assert result['text'] == _BODY.decode() and result['artifact']['id'] == metadata['id']
assert await _count(x.case) == 1
x.query._input_artifact_context = replace(
x.case.context,
principal=PrincipalContext(PrincipalType.ACCOUNT, account_uuid=str(uuid.uuid4())),
)
with pytest.raises(FileNotFoundError):
await x.render(request)
assert await _count(x.case) == 1
async def test_session_identity_expiry_and_budget_precede_content_access(input_actions):
x = input_actions
refused = await x.render({**x.arguments, 'caller_plugin_identity': 'other/plugin'})
assert refused.code != 0 and await _count(x.case) == 0
x.query._input_attachment_read_bytes = input_media.MAX_RUN_READ_BYTES - len(_BODY)
first, second = await asyncio.gather(x.raw(x.arguments), x.raw(x.arguments))
assert sorted([first.code == 0, second.code == 0]) == [False, True]
assert x.query._input_attachment_read_bytes == input_media.MAX_RUN_READ_BYTES
assert await _count(x.case) == 0
await x.registry.unregister(x.run_id)
expired = await x.render(x.arguments)
assert expired.code != 0 and 'expired' in expired.message
async def test_cancellation_before_payload_resolution_leaves_no_asset_and_releases_lock(input_actions, monkeypatch):
x = input_actions
started = asyncio.Event()
original = input_media.read_component_bytes
async def stalled(*args, **kwargs):
started.set()
await asyncio.Event().wait()
monkeypatch.setattr(input_media, 'read_component_bytes', stalled)
task = asyncio.create_task(x.render(x.arguments))
await asyncio.wait_for(started.wait(), 5)
task.cancel()
with pytest.raises(asyncio.CancelledError):
await task
assert await _count(x.case) == 0
assert getattr(x.query, '_input_attachment_read_bytes', 0) == 0
monkeypatch.setattr(input_media, 'read_component_bytes', original)
response = await asyncio.wait_for(x.render(x.arguments), 5)
assert response.code == 0 and await _count(x.case) == 1
async def test_text_continuation_charges_materialization_once_and_bounds_repeated_output(input_actions, monkeypatch):
x = input_actions
text = 'BEGIN:' + '0123456789' * 818 + ':END!!'
assert len(text.encode()) == 8192
monkeypatch.setattr(input_media, 'MAX_RUN_READ_BYTES', 16 * 1024)
x.query.message_chain = MessageChain(
[
File(name='continued.txt', base64=base64.b64encode(text.encode()).decode()),
]
)
offset = 0
chunks = []
for _ in range(10):
response = await x.render({**x.arguments, 'offset': offset, 'max_chars': 900})
assert response.code == 0, response.message
result = json.loads(response.data['contents'][0]['text'])
assert result['offset'] == offset
chunks.append(result['text'])
if result['next_offset'] is None:
assert result['truncated'] is False
break
assert result['truncated'] is True
assert result['next_offset'] == offset + len(result['text'])
offset = result['next_offset']
assert len(chunks) == 10
assert ''.join(chunks) == text
assert await _count(x.case) == 1
# Cached source bytes may be reused, but repeated model output remains bounded.
for _ in range(32):
response = await x.render({**x.arguments, 'offset': 0, 'max_chars': 900})
if response.code != 0:
assert 'budget' in response.message.lower()
break
else:
pytest.fail('Repeated content output bypassed the run budget')
+3 -3
View File
@@ -39,8 +39,8 @@ def test_markers_place_each_media_and_attachments_are_metadata_only():
)
assert rendered.text == 'compare [attachment att-1: image] with [attachment att-2: file report.pdf]'
assert [item.type for item in rendered.contents] == ['text', 'text', 'image_base64', 'text', 'text', 'file_base64']
assert rendered.plain_text == 'compare with '
assert [item.type for item in rendered.contents] == ['text', 'text', 'text', 'text']
assert rendered.plain_text == 'compare [Image] with '
assert rendered.attachments() == [
{
'ref': 'att-1',
@@ -91,7 +91,6 @@ def test_empty_quote_and_text_only_models_render_without_payloads():
platform_message.Image(url=IMAGE_URL, base64=IMAGE_BASE64),
),
include_quotes=True,
keep_images=False,
)
assert _texts(rendered.contents) == [('text', '[attachment att-1: image]')]
@@ -112,6 +111,7 @@ def test_event_first_input_hands_payloads_to_the_run_registry_only():
assert agent_input.text == 'look[attachment att-1: image]'
assert agent_input.attachments[0].content == IMAGE_BASE64
assert all(item.type == 'text' for item in agent_input.contents)
# Event runs execute against a text-only chain; the envelope payload is registered by ref.
query = SimpleNamespace(message_chain=_chain(platform_message.Plain(text=agent_input.text)))
@@ -314,7 +314,7 @@ def test_context_builder_renders_message_chain_without_preprocessing():
input_data = QueryEntryAdapter._build_input(query)
assert [content.type for content in input_data.contents] == ['text', 'image_base64']
assert [content.type for content in input_data.contents] == ['text']
assert input_data.text == '[attachment att-1: image]'
assert len(input_data.attachments) == 1
assert input_data.attachments[0].type == 'image'
@@ -802,7 +802,9 @@ async def test_cancel_request_preserves_one_terminal_event(clean_agent_state, re
('run.cancelled', {'reason': 'stopped', 'cleanup_confirmed': True}, 'cancelled'),
],
)
async def test_terminal_is_persisted_before_runner_stream_cleanup(clean_agent_state, monkeypatch, result_type, data, status):
async def test_terminal_is_persisted_before_runner_stream_cleanup(
clean_agent_state, monkeypatch, result_type, data, status
):
"""Transport cleanup must not hold a validated terminal run open."""
cleanup_started = asyncio.Event()
release_cleanup = asyncio.Event()
@@ -2115,8 +2117,8 @@ class TestQueryEntryAdapterHostCapabilities:
assert len(event_logs) >= 1
# First event should be the incoming message.received
assert event_logs[0]['event_type'] == 'message.received'
assert event_logs[0]['input_json']['contents'][1]['image_base64'] is None
assert event_logs[0]['input_json']['contents'][1]['content_redacted'] is True
assert event_logs[0]['input_json']['contents'][1]['type'] == 'text'
assert event_logs[0]['input_json']['contents'][1]['text'] == '[attachment att-2: image]'
assert 'aGVsbG8=' not in str(event_logs[0]['input_json'])
assert 'Pinned documentation' not in str(event_logs[0]['input_json'])
@@ -2133,7 +2135,8 @@ class TestQueryEntryAdapterHostCapabilities:
assert 'user' in roles
assert 'assistant' in roles
user_item = next(t for t in transcripts if t['role'] == 'user')
assert user_item['content_json']['content'][1]['image_base64'] is None
assert user_item['content_json']['content'][1]['type'] == 'text'
assert user_item['content_json']['content'][1]['text'] == '[attachment att-2: image]'
assert user_item['attachment_refs'][0]['content'] is None
assert 'aGVsbG8=' not in str(user_item)
assert 'Pinned documentation' not in str(user_item)
@@ -159,6 +159,75 @@ class TestAgentServiceMetadata:
class TestAgentServiceDebug:
async def test_debug_artifacts_are_authorized_without_reading_bytes(self):
app = _make_app()
artifact_id = 'art_' + 'a' * 32
app.artifact_service = SimpleNamespace(
get=AsyncMock(
return_value={
'id': artifact_id,
'name': 'report.pdf',
'mime_type': 'application/pdf',
'size': 50,
}
),
read=AsyncMock(),
)
context = SimpleNamespace(
instance_uuid='instance-test',
workspace_uuid=WORKSPACE_UUID,
placement_generation=1,
principal=SimpleNamespace(account_uuid='account-test'),
entitlement_revision=0,
)
async def run_runner(event, binding, adapter_context):
query = adapter_context['_query']
assert query._input_artifact_context is context
assert query._input_artifact_ids == frozenset({artifact_id})
assert event.input.attachments[0].ref == artifact_id
assert event.input.attachments[0].content is None
assert event.input.attachments[0].url is None
assert all(item.type == 'text' for item in event.input.contents)
yield SimpleNamespace(role='assistant', content='read later', all_content=None)
app.agent_run_orchestrator = SimpleNamespace(run=Mock(side_effect=run_runner))
service = AgentService(app)
service.get_agent = AsyncMock(
return_value={
'uuid': 'agent-1',
'kind': AGENT_KIND_AGENT,
'supported_event_patterns': ['*'],
'config': _agent_row().config,
}
)
await service.debug_agent(
context,
'agent-1',
{
'text': 'read report',
'artifact_ids': [artifact_id],
'actor': {'actor_type': 'user', 'actor_id': 'spoofed-owner'},
},
)
app.artifact_service.get.assert_awaited_once_with(context, artifact_id)
app.artifact_service.read.assert_not_awaited()
@pytest.mark.parametrize('artifact_ids', ['art_bad', ['same', 'same'], [None], ['x'] * 21])
async def test_debug_rejects_invalid_artifact_ids(self, artifact_ids):
app = _make_app()
service = AgentService(app)
service.get_agent = AsyncMock(
return_value={
'uuid': 'agent-1',
'kind': AGENT_KIND_AGENT,
'supported_event_patterns': ['*'],
'config': _agent_row().config,
}
)
with pytest.raises(ValueError, match='artifact_ids'):
await service.debug_agent(SimpleNamespace(), 'agent-1', {'artifact_ids': artifact_ids})
@pytest.mark.parametrize('streaming', [False, True])
@pytest.mark.parametrize(
'result_type',
+32 -2
View File
@@ -154,8 +154,11 @@ def test_event_attachments_preserved_without_eager_io_or_host_path_access():
assert input_files(q)['att-1'].base64 == 'YQ=='
assert not input_files(q)['att-1'].path
image = AgentInput(contents=[ContentElement.from_image_url('https://example.invalid/image.png')])
prepare_input_files(SimpleNamespace(message_chain=MessageChain([])), image)
assert image.attachments[0].url == 'https://example.invalid/image.png'
image_query = SimpleNamespace(message_chain=MessageChain([]))
prepare_input_files(image_query, image)
assert image.attachments[0].url is None
assert image.contents[0].text == '[attachment att-1: image]'
assert input_files(image_query)['att-1'].url == 'https://example.invalid/image.png'
def test_platform_media_keep_rendered_refs_and_never_carry_bytes():
@@ -178,6 +181,8 @@ def test_platform_media_keep_rendered_refs_and_never_carry_bytes():
('att-2', 'file', None),
]
assert input_files(query) == {'att-1': image, 'att-2': report}
assert all(item.type == 'text' for item in value.contents)
assert all(item.url is None and item.path is None for item in value.attachments)
def test_repeated_prepare_keeps_envelope_registrations():
@@ -200,6 +205,31 @@ def test_repeated_prepare_keeps_envelope_registrations():
assert all(not item.base64 for item in input_files(other_query).values())
def test_contents_only_files_keep_distinct_sources_behind_markers():
from langbot_plugin.api.entities.builtin.runner.input import AgentInput
from langbot_plugin.api.entities.builtin.provider.message import ContentElement
from langbot.pkg.box.runner import input_files, prepare_input_files
query = SimpleNamespace(message_chain=MessageChain([]))
value = AgentInput(
contents=[
ContentElement.from_file_url('https://files.example/one', 'report.pdf'),
ContentElement.from_file_url('https://files.example/two', 'report.pdf'),
]
)
prepare_input_files(query, value)
assert [item.ref for item in value.attachments] == ['att-1', 'att-2']
assert [item.text for item in value.contents] == [
'[attachment att-1: file report.pdf]',
'[attachment att-2: file report.pdf]',
]
assert [item.url for item in input_files(query).values()] == [
'https://files.example/one',
'https://files.example/two',
]
assert all(item.url is None and item.content is None for item in value.attachments)
@pytest.mark.asyncio
async def test_image_export_accepts_data_url_and_enforces_total_bytes():
api = service()
@@ -309,15 +309,9 @@ class TestPreProcessorImageSegment:
from langbot.pkg.agent.runner.query_entry_adapter import QueryEntryAdapter
from langbot_plugin.api.entities.builtin.provider.message import ContentElement
expected_image = (
ContentElement.from_image_base64('data:image/png;base64,abc123')
if image_source == 'base64'
else ContentElement.from_image_url('https://cdn.example/image.png')
)
assert result.new_query.user_message.content == [
ContentElement.from_text('look at this'),
ContentElement.from_text('[attachment att-1: image]'),
expected_image,
]
attachments = QueryEntryAdapter._build_input(result.new_query).attachments
assert len(attachments) == 1
@@ -1,14 +1,14 @@
"""Unit tests for LiteLLMRequester._convert_messages.
Focus: the content-part normalization that (a) converts image_base64 parts to
the OpenAI image_url shape and (b) drops non-image file parts (file_base64 /
file_url) which OpenAI-compatible chat models reject. The latter is essential
for Voice/File attachments — including ones replayed from conversation history —
since the agent consumes their bytes via the sandbox, not the model payload.
Image parts use OpenAI's image_url shape. Explicit PDF input checks native
model support and uses the file shape; unsupported legacy file parts retain
their existing handling.
"""
import langbot_plugin.api.entities.builtin.provider.message as provider_message
import pytest
from langbot.pkg.provider.modelmgr import errors
from langbot.pkg.provider.modelmgr.requesters.litellmchat import LiteLLMRequester
@@ -17,6 +17,60 @@ def _make_requester() -> LiteLLMRequester:
return LiteLLMRequester.__new__(LiteLLMRequester)
def test_explicit_pdf_input_uses_supported_file_payload(monkeypatch):
req = _make_requester()
probes = []
def capability(helper, model):
probes.append((helper, model))
return True
monkeypatch.setattr(req, '_safe_litellm_bool_helper', capability)
pdf = 'data:application/pdf;base64,JVBERi0xLjc='
message = provider_message.Message(
role='user',
content=[
provider_message.ContentElement.from_file_base64(pdf, 'report.pdf'),
],
)
result = req._convert_messages([message], model_name='pdf-model')
assert result[0]['content'] == [{'type': 'file', 'file': {'file_data': pdf, 'filename': 'report.pdf'}}]
assert probes == [('supports_pdf_input', 'pdf-model')]
assert message.content[0].type == 'file_base64'
def test_explicit_pdf_input_fails_instead_of_being_discarded(monkeypatch):
req = _make_requester()
monkeypatch.setattr(req, '_safe_litellm_bool_helper', lambda *_args: False)
message = provider_message.Message(
role='user',
content=[
provider_message.ContentElement.from_file_base64('data:application/pdf;base64,JVBERg==', 'report.pdf'),
],
)
with pytest.raises(errors.RequesterError) as failure:
req._convert_messages([message], model_name='text-model')
assert failure.value.code == 'unsupported_pdf_input'
def test_explicit_pdf_ability_supports_verified_gateway_alias_without_guessing_metadata(monkeypatch):
req = _make_requester()
def unexpected_metadata_probe(*_args):
raise AssertionError('An explicit PDF declaration must not depend on alias metadata')
monkeypatch.setattr(req, '_safe_litellm_bool_helper', unexpected_metadata_probe)
pdf = 'data:application/pdf;base64,JVBERi0xLjc='
message = provider_message.Message(
role='user',
content=[
provider_message.ContentElement.from_file_base64(pdf, 'report.pdf'),
],
)
converted = req._convert_messages([message], model_name='verified-gateway-alias', pdf_input_declared=True)
assert converted[0]['content'] == [{'type': 'file', 'file': {'file_data': pdf, 'filename': 'report.pdf'}}]
def test_convert_messages_drops_file_base64_part():
req = _make_requester()
msg = provider_message.Message(
@@ -17,6 +17,7 @@ from langbot.pkg.provider.tools.loaders.mcp import (
MCP_RESOURCE_CONTEXT_QUERY_KEY,
MCP_RESOURCE_TRACE_QUERY_KEY,
MCP_TOOL_CALL_TIMEOUT_DEFAULT_SECONDS,
MCP_TOOL_PDF_MAX_BYTES,
MCP_TOOL_LIST_RESOURCES,
MCP_TOOL_READ_RESOURCE,
MCPLoader,
@@ -84,6 +85,67 @@ def _connected_session(
return session
@pytest.mark.asyncio
async def test_explicit_tool_pdf_resource_remains_structured_for_model_consumption():
session = _connected_session()
pdf = b'%PDF-1.7 explicit tool result'
session.session.call_tool = AsyncMock(
return_value=mcp_types.CallToolResult(
content=[
mcp_types.EmbeddedResource(
type='resource',
resource=mcp_types.BlobResourceContents(
uri='langbot-artifact:///sample%20report.pdf',
mimeType='application/pdf',
blob=base64.b64encode(pdf).decode(),
),
),
]
)
)
result = await session.invoke_mcp_tool('read_artifact_content', {'artifact_id': 'art_current', 'mode': 'native'})
assert len(result) == 1 and result[0].type == 'file_base64'
assert result[0].file_name == 'sample report.pdf'
assert result[0].file_base64 == 'data:application/pdf;base64,' + base64.b64encode(pdf).decode()
@pytest.mark.asyncio
@pytest.mark.parametrize(
'blob,error',
[
('not valid base64', 'invalid PDF base64'),
('x' * (4 * ((MCP_TOOL_PDF_MAX_BYTES + 2) // 3) + 1), 'read size limit'),
(base64.b64encode(b'x' * (MCP_TOOL_PDF_MAX_BYTES + 1)).decode(), 'read size limit'),
],
)
async def test_explicit_tool_pdf_resource_rejects_invalid_or_oversized_payload(blob, error):
session = _connected_session()
resource = mcp_types.BlobResourceContents.model_construct(
uri='langbot-artifact:///invalid.pdf',
mimeType='application/pdf',
blob=blob,
)
block = mcp_types.EmbeddedResource.model_construct(type='resource', resource=resource)
with pytest.raises(ValueError, match=error):
session._content_to_provider_elements(block, source_tool='read_artifact_content')
def test_non_pdf_tool_resource_keeps_binary_omitted_behavior():
session = _connected_session()
result = session._content_to_provider_elements(
mcp_types.EmbeddedResource(
type='resource',
resource=mcp_types.BlobResourceContents(
uri='langbot-artifact:///archive.zip',
mimeType='application/zip',
blob='aGk=',
),
),
source_tool='download_archive',
)
assert len(result) == 1 and '"binary_omitted": true' in result[0].text
def _query(variables: dict | None = None, context: ExecutionContext = TEST_EXECUTION_CONTEXT) -> SimpleNamespace:
return SimpleNamespace(
instance_uuid=context.instance_uuid,
@@ -0,0 +1,130 @@
"""Model-facing ranges, media inputs and real PDF decoding budgets."""
import base64
import io
import json
import pytest
from pypdf import PdfWriter, get_configuration
from pypdf.generic import DecodedStreamObject, DictionaryObject, NameObject
from langbot.pkg.storage.artifact_reader import MAX_PDF_STREAM_BYTES, read_artifact_content
from langbot.pkg.storage.artifacts import ArtifactError
pytestmark = pytest.mark.asyncio
_META = {'id': 'art_test', 'name': 'reference.txt', 'mime_type': 'text/plain', 'content_ref': 'private-key'}
_PNG = base64.b64decode('iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAwMCAO+jWZkAAAAASUVORK5CYII=')
def _pdf(*texts, encrypted=False, compressed_payload=None):
writer = PdfWriter()
for text in texts:
page = writer.add_blank_page(width=612, height=792)
font = DictionaryObject(
{
NameObject('/Type'): NameObject('/Font'),
NameObject('/Subtype'): NameObject('/Type1'),
NameObject('/BaseFont'): NameObject('/Helvetica'),
}
)
page[NameObject('/Resources')] = DictionaryObject(
{
NameObject('/Font'): DictionaryObject({NameObject('/F1'): writer._add_object(font)}),
}
)
stream = DecodedStreamObject()
stream.set_data(
compressed_payload
if compressed_payload is not None
else (f'BT /F1 12 Tf 72 720 Td ({text}) Tj ET'.encode('ascii'))
)
page[NameObject('/Contents')] = writer._add_object(
stream.flate_encode() if compressed_payload is not None else stream
)
if encrypted:
writer.encrypt('password')
output = io.BytesIO()
writer.write(output)
return output.getvalue()
def _text(elements):
assert elements[0]['type'] == 'text'
return json.loads(elements[0]['text'])
async def test_text_ranges_use_characters_and_preserve_exact_resume_position():
result = _text(await read_artifact_content('a猫🙂z'.encode(), _META, offset=1, max_chars=2))
assert result['text'] == '猫🙂'
assert (result['offset'], result['end_offset'], result['total_chars']) == (1, 3, 4)
assert result['truncated'] is True and result['next_offset'] == 3
assert 'content_ref' not in result['artifact']
remainder = _text(await read_artifact_content('a猫🙂z'.encode(), _META, offset=3))
assert remainder['text'] == 'z' and remainder['next_offset'] is None
assert remainder['truncated'] is False
async def test_images_and_native_pdf_remain_structured_instead_of_textual_base64():
image = await read_artifact_content(_PNG, {'name': 'pixel.png', 'mime_type': 'image/png'})
assert _text(image)['mode'] == 'image'
assert len(image) == 2 and image[1]['type'] == 'image_base64'
assert base64.b64decode(image[1]['image_base64'].split(',', 1)[1], validate=True) == _PNG
assert 'base64' not in image[0]['text']
pdf = _pdf('Native file payload')
native = await read_artifact_content(pdf, {'name': None}, mode='native')
assert _text(native)['mode'] == 'native'
assert native[1]['type'] == 'file_base64' and native[1]['file_name'] == 'document.pdf'
assert base64.b64decode(native[1]['file_base64'].split(',', 1)[1], validate=True) == pdf
async def test_pdf_selected_pages_and_text_limits_report_continuation_and_visual_boundary():
content = _pdf('First page original', 'Second page original', 'Third page original')
result = _text(await read_artifact_content(content, {'mime_type': 'application/pdf'}, page_start=2, page_end=2))
assert 'Second page original' in result['text']
assert 'First page original' not in result['text'] and 'Third page original' not in result['text']
assert (result['page_start'], result['page_end'], result['page_count'], result['next_page']) == (2, 2, 3, 3)
assert result['text_only'] is True and 'excludes images' in result['notice']
limited = _text(await read_artifact_content(content, {}, max_chars=4, page_end=1))
assert limited['text'] == '[Pag' and limited['truncated'] is True and limited['next_offset'] == 4
async def test_compressed_pdf_is_rejected_before_unbounded_expansion_and_configuration_is_restored():
content = _pdf('', compressed_payload=b' ' * (MAX_PDF_STREAM_BYTES + 1))
assert len(content) < 10000
original = get_configuration()
with pytest.raises(ArtifactError) as failure:
await read_artifact_content(content, {})
assert failure.value.error_code == 'artifact_too_large' and failure.value.status_code == 413
assert get_configuration() is original
assert 'ordinary PDF' in _text(await read_artifact_content(_pdf('ordinary PDF'), {}))['text']
async def test_unreadable_pdf_and_unsupported_formats_fail_explicitly():
for content in (b'%PDF-broken', _pdf('Protected', encrypted=True)):
with pytest.raises(ArtifactError) as failure:
await read_artifact_content(content, {'mime_type': 'application/pdf'})
assert failure.value.error_code == 'artifact_pdf_unreadable' and failure.value.status_code == 422
for content, options in ((b'\x00binary', {}), (_pdf('Text'), {'mode': 'pages'})):
with pytest.raises(ArtifactError) as failure:
await read_artifact_content(content, {}, **options)
assert failure.value.error_code == 'artifact_read_unsupported' and failure.value.status_code == 415
@pytest.mark.parametrize(
'options',
[
{'mode': []},
{'offset': True},
{'offset': -1},
{'max_chars': True},
{'max_chars': 16385},
{'page_start': False},
{'page_start': 0},
{'page_end': True},
{'page_end': 11},
],
)
async def test_read_arguments_reject_invalid_ranges_and_boolean_numbers(options):
with pytest.raises(ValueError):
await read_artifact_content(b'text', _META, **options)
Generated
+1 -1
View File
@@ -2271,7 +2271,7 @@ requires-dist = [
{ name = "pylibseekdb", marker = "(platform_machine == 'arm64' and sys_platform == 'darwin' and extra == 'seekdb') or (sys_platform == 'linux' and extra == 'seekdb')", specifier = "==1.4.0" },
{ name = "pymilvus", marker = "extra == 'milvus'", specifier = ">=2.6.4" },
{ name = "pynacl", specifier = ">=1.5.0" },
{ name = "pypdf", specifier = ">=6.0.0" },
{ name = "pypdf", specifier = ">=6.21.2" },
{ name = "pyseekdb", marker = "extra == 'seekdb'", specifier = "==1.4.0.post1" },
{ name = "python-docx", specifier = ">=1.1.0" },
{ name = "python-multipart", specifier = ">=0.0.32" },