mirror of
https://github.com/langbot-app/LangBot.git
synced 2026-11-22 21:48:20 +08:00
feat(artifact): add run-scoped on-demand attachment reads
This commit is contained in:
1 parent
b01af050ff
commit
33dd118872
67 files changed
+2173
-178
No files matched your search
@@ -11,16 +11,27 @@ Only authenticated accounts and API keys are accepted. Each operation checks the
|
||||
| `artifact.create` | `POST /api/v1/artifacts` (multipart `file`, optional `ttl_seconds`) | `resource.manage` |
|
||||
| `artifact.get` | `GET /api/v1/artifacts/{id}` | `resource.view` |
|
||||
| `artifact.read` | `GET /api/v1/artifacts/{id}/content?max_bytes=10485760` | `resource.view` |
|
||||
| `artifact.read_content` | `GET /api/v1/artifacts/{id}/read` | `resource.view` |
|
||||
| `artifact.retain` | `POST /api/v1/artifacts/{id}/retain` | `resource.manage` |
|
||||
| `artifact.release` | `POST /api/v1/artifacts/{id}/release` | `resource.manage` |
|
||||
| `artifact.delete` | `DELETE /api/v1/artifacts/{id}` | `resource.manage` |
|
||||
|
||||
HTTP and MCP execute the same ApplicationAPI operations and Artifact service. HTTP downloads use attachment disposition, `nosniff`, `private, no-store` and a digest ETag. The API provides complete bounded reads, not byte ranges or partial text extraction: an object larger than `max_bytes` is rejected. Uploads and reads are limited to 10 MiB. Field validation diagnostics retain their technical wording; owned business errors use the Workspace message language.
|
||||
HTTP and MCP execute the same ApplicationAPI operations and Artifact service. HTTP downloads use attachment disposition, `nosniff`, `private, no-store` and a digest ETag. Raw downloads return the complete bounded object: an object larger than `max_bytes` is rejected. Structured content reads support the text and PDF ranges described below. Uploads and source reads are limited to 10 MiB. Field validation diagnostics retain their technical wording; owned business errors use the Workspace message language.
|
||||
|
||||
MCP tools are `create_artifact`, `get_artifact`, `read_artifact`, `retain_artifact`, `release_artifact` and `delete_artifact`. Creation accepts strict base64 with a pre-decode encoded-length limit; reads return base64 plus metadata. These are file transfer APIs. Structured image/PDF model consumption, user-attachment routing and Runner file conversion are later stages of #3051.
|
||||
MCP tools are `create_artifact`, `get_artifact`, `read_artifact`, `read_artifact_content`, `retain_artifact`, `release_artifact` and `delete_artifact`. Creation accepts strict base64 with a pre-decode encoded-length limit; raw reads return base64 plus metadata. Runner file conversion and fixed-version Skill resources remain later stages of #3051.
|
||||
|
||||
The existing file upload and Runner `file_id` APIs retain their contracts. No existing storage key or run handle is accepted as an Artifact ID. The separate `lbctl` client can continue using its existing routes; it has no Artifact commands in this phase.
|
||||
|
||||
`artifact.read_content` accepts `mode`, `offset`, `max_chars`, `page_start` and `page_end` and returns structured content elements. MCP `read_artifact_content` maps them to native text/image/embedded-PDF blocks without placing encoded bytes into a text result. Both use the same authorized reader as execution attachment reads.
|
||||
|
||||
## Agent attachment reads
|
||||
|
||||
Inbound platform inputs render text markers and attachment metadata only, independently of model vision support. `InputMedia` and the Host query registry retain original components; `prepare_input_files` also registers media found only in event contents before replacing those contents with markers. Runner descriptors contain no inline bytes, Host paths or source URLs. No eager download is introduced. The existing raw `read_input_attachment` API and new structured `read_input_attachment_content` operation share the frozen `input_attachments` run grant, plugin installation identity and live run checks.
|
||||
|
||||
The first authorized platform attachment read creates a run-owned Artifact through the shared service. Only the Host's current-input registry may resolve an `att-N` ref; knowing another asset ID is not a grant. Run-owned assets do not become account-owned or publicly accessible. Agent debug accepts at most 20 distinct `artifact_ids`: it checks metadata with the original RequestContext before execution and stores the authorized set plus context in Host-only query fields. Synthetic event actors cannot select the asset owner. The separate Workspace management assistant still accepts text/approval turns only.
|
||||
|
||||
Structured reads use `auto`, `text`, `native` or `pages`, zero-based text `offset`, `max_chars` (8192 default, 16384 maximum), and one-based PDF `page_start`/`page_end` (at most ten pages). Results identify truncation and continuation offsets/pages. Auto mode returns structured images and UTF-8 text, and extracts PDF text. PDF text explicitly excludes images, layout and OCR. Native mode returns structured image/PDF content for a supporting provider; unsupported models fail visibly. Pages mode currently fails explicitly rather than implying that page rendering occurred. Raw bytes and structured content are returned only after source authorization; metadata and retained history do not contain base64 payloads.
|
||||
|
||||
## Lifecycle
|
||||
|
||||
Creation produces a temporary asset with a default 24-hour lifetime; callers may choose 1 second through 7 days. Retain creates a persistent caller-owned pin. Retain changes retention only, never authorization. Release removes the caller's pin and restores a 24-hour temporary grace period; repeated release does not keep extending the lifetime. Delete refuses an asset protected by a pin. Expired assets cannot be read or revived through retain.
|
||||
@@ -33,4 +44,6 @@ Errors: missing, deleted and unauthorized asset IDs return 404 without disclosin
|
||||
|
||||
## Acceptance
|
||||
|
||||
Native PDF uses LiteLLM capability metadata or the administrator's explicit `pdf_input` model ability for a provider-confirmed alias. Unknown capability without that declaration fails explicitly. Content reads charge source materialization once per attachment, and separately cap cumulative returned content at the run read budget; the legacy raw-byte API continues to charge each full read.
|
||||
|
||||
Exercise authenticated HTTP and MCP against real persistence: byte-for-byte upload/download and digest, principal and Workspace isolation, limited permissions, expiry, retain/release/delete, malformed and excessive inputs, cancellation, GC concurrency and cleanup failure. PostgreSQL/MinIO acceptance includes restricted-role RLS and a second Core/restart reading persistent assets without shared cache roots. See [cross-component testing](../CROSS_COMPONENT_TESTING.md). A successful base64 transfer does not establish model understanding of an image or PDF.
|
||||
@@ -12,6 +12,7 @@ Use the normal Workspace HTTP authentication. Reads require `resource.view`; cre
|
||||
| Upload | `POST /api/v1/artifacts`, multipart `file`, optional `ttl_seconds` |
|
||||
| Metadata | `GET /api/v1/artifacts/{id}` |
|
||||
| Download | `GET /api/v1/artifacts/{id}/content`, optional `max_bytes` |
|
||||
| Read structured content | `GET /api/v1/artifacts/{id}/read`, optional `mode`, `offset`, `max_chars`, `page_start`, `page_end` |
|
||||
| Retain | `POST /api/v1/artifacts/{id}/retain` |
|
||||
| Release | `POST /api/v1/artifacts/{id}/release` |
|
||||
| Delete | `DELETE /api/v1/artifacts/{id}` |
|
||||
@@ -26,6 +27,25 @@ Deletion blocks subsequent access; maintenance reclaims the underlying bytes. An
|
||||
|
||||
## MCP
|
||||
|
||||
The built-in `/mcp` server exposes `create_artifact`, `get_artifact`, `read_artifact`, `retain_artifact`, `release_artifact` and `delete_artifact` with the same ownership and permission checks. Creation takes `filename`, `content_base64`, optional `mime_type` and `ttl_seconds`. Reading takes `artifact_id` and optional `max_bytes`, returning base64 plus metadata.
|
||||
Native PDF input requires trusted model capability information. For an alias whose provider has confirmed PDF support but which the capability catalog does not recognize, an administrator can declare `pdf_input` in the model's existing `abilities` field. Undeclared, unknown capability rejects native PDF input; text mode remains available.
|
||||
|
||||
These tools transfer files. They do not extract text, render PDF pages or establish that a model understands an image. The initial API is not available through plugin Workspace grants, plugin typed methods, default model tools or `lbctl`. Existing upload and run `file_id` interfaces continue to use their existing contracts.
|
||||
The built-in `/mcp` server exposes `create_artifact`, `get_artifact`, `read_artifact`, `read_artifact_content`, `retain_artifact`, `release_artifact` and `delete_artifact` with the same ownership and permission checks. Creation takes `filename`, `content_base64`, optional `mime_type` and `ttl_seconds`. Raw reading takes `artifact_id` and optional `max_bytes`, returning base64 plus metadata. `read_artifact_content` takes the structured-read parameters from the HTTP `/read` endpoint and returns native MCP text, image or embedded-PDF content rather than base64 inside ordinary text. The Workspace operation is `artifact.read_content`.
|
||||
|
||||
`read_artifact` transfers the original bytes. Account-owned management operations remain unavailable through plugin Workspace grants, plugin typed methods, default model tools or `lbctl`. Existing upload and run `file_id` interfaces continue to use their existing contracts.
|
||||
|
||||
## Agent reads
|
||||
|
||||
Runner inputs initially contain attachment references, names, MIME types and sizes. Image/file bytes and source URLs stay in the Host registry; they are not inserted into the initial model message. Current-message and quoted-message attachments retain separate references. The Agent requests content only when needed, under the current run's frozen attachment grant. References from another run do not grant access.
|
||||
|
||||
`read_input_attachment_content` returns structured content and supports these modes:
|
||||
|
||||
| Mode | Behavior |
|
||||
| --- | --- |
|
||||
| `auto` | Read UTF-8 text, return a structured image, or extract PDF text. |
|
||||
| `text` | Read a bounded character range; PDF results identify the selected pages and exclude visual content. |
|
||||
| `native` | Return image/PDF content for a model that supports that input; unsupported models report a visible error. |
|
||||
| `pages` | Page rendering is currently unsupported and returns an explicit error. |
|
||||
|
||||
Text uses zero-based `offset` and `max_chars` (default 8192, maximum 16384). Results include `truncated` and `next_offset`; continue from that position rather than treating a partial result as the full file. PDF text uses one-based `page_start` and optional `page_end`, at most ten pages per read, and reports the next page. It does not perform OCR or preserve images/layout. Other binary formats fail explicitly; the original remains downloadable.
|
||||
|
||||
Agent debug API requests may include `artifact_ids`, up to 20 distinct IDs from the caller's uploads. The Host checks the authenticated caller's ownership before starting the run and gives that run references only. Supplying an event `actor` does not change ownership. The separate Workspace management assistant does not accept attachment inputs in this phase. The existing raw-byte `read_input_attachment` API remains available to authorized Runner code.
|
||||
@@ -190,15 +190,15 @@ Each item returned by `get_available_tools()` contains `name`, `description`, `p
|
||||
|
||||
### Input attachments
|
||||
|
||||
`ctx.input.attachments` describes each image, voice and file of the current message in order of appearance, as metadata only: `ref` (such as `att-1`), `type`, `source` (`message` for the current message, `quote` for a quoted message), `name`, `mime_type`, `size`, the platform `id`, and a directly downloadable `url`. Attachments do not carry file content; the `content` field is deprecated and always empty in current versions.
|
||||
`ctx.input.attachments` describes each image, voice and file of the current message in order of appearance, as metadata only: `ref` (such as `att-1`), `type`, `source` (`message` for the current message, `quote` for a quoted message), `name`, `mime_type`, `size`, and the platform `id`. The Host keeps source URLs private. Attachments do not carry file content; the `content` field is deprecated and always empty in current versions.
|
||||
|
||||
`ctx.input.text` and `ctx.input.contents` place a marker at the position of each media, such as `[attachment att-1: image photo.jpg]`. A quoted message is wrapped in `[quoted message from <sender>]` and `[/quoted message]`. Images still follow their marker as `image_base64` or `image_url` so models can see them.
|
||||
`ctx.input.text` and `ctx.input.contents` place a marker at the position of each media, such as `[attachment att-1: image photo.jpg]`. A quoted message is wrapped in `[quoted message from <sender>]` and `[/quoted message]`. Images and files are not inlined into the initial model input. Read them explicitly when needed.
|
||||
|
||||
| Method | Returns | Description |
|
||||
| --- | --- | --- |
|
||||
| `await ctx.read_input_attachment(attachment)` | `InputAttachmentData \| None` | Reads attachment bytes by `ref` and returns `data`, `name`, `mime_type` and `size`. The argument may also be a `ref` string. Returns `None` when LangBot does not offer this API |
|
||||
|
||||
Each attachment is limited to 10 MiB and each run may read up to 50 MiB in total. LangBot neither caches the bytes nor writes them to logs. `ctx.context.available_apis.input_attachments` reports whether the API is available. Follow-up messages steered into the current run carry attachment metadata without a `ref`.
|
||||
Each attachment is limited to 10 MiB and each run may read up to 50 MiB in total. The content-reading API materializes the source as a temporary run-owned Artifact on first use; inline bytes are not written to conversation history or tool events. `ctx.context.available_apis.input_attachments` reports whether the API is available. Follow-up messages steered into the current run carry attachment metadata without a `ref`.
|
||||
|
||||
The method requires `langbot-plugin` 0.7.23 or later. To support older LangBot or SDK versions, check that the method exists and fall back to the previous logic:
|
||||
|
||||
@@ -210,6 +210,8 @@ if data is None:
|
||||
... # Older LangBot: use attachment.content, attachment.url or ctx.input.contents
|
||||
```
|
||||
|
||||
`await ctx.read_input_attachment_content(ref, mode="auto", offset=0, max_chars=8192, page_start=1, page_end=None)` returns bounded text or structured media with the same frozen run grant. See [Artifact reading](/en/develop/artifacts) for modes and limits. This new method requires coordinated Core, SDK and Runtime versions; upgrade them and the LocalAgent plugin together.
|
||||
|
||||
### Box and attachments
|
||||
|
||||
The Runner owns enablement, acquisition, and reuse policy. Use the [LangBot Box APIs](/en/plugin/dev/apis/common#box-sandbox) for resources and these context methods for the current run.
|
||||
|
||||
@@ -10,6 +10,8 @@ Every operation has a stable ID, such as `pipeline.list` or `knowledge.base.retr
|
||||
|
||||
Artifact operations are currently reserved for authenticated accounts and API keys. They are excluded from plugin grants, wildcard permissions and typed Workspace methods. See the [Artifact file API](/en/develop/artifacts).
|
||||
|
||||
This includes `artifact.read_content`, which provides bounded text and structured image/PDF reads. Runner code instead uses the run-bound `read_input_attachment_content` execution API under its existing attachment grant.
|
||||
|
||||
## Calling an operation
|
||||
|
||||
From any component, use `self.plugin.workspace`. Each operation has a typed method named after its ID:
|
||||
|
||||
@@ -12,6 +12,7 @@ Artifact はファイルの不透明な参照とメタデータを提供しま
|
||||
| アップロード | `POST /api/v1/artifacts`、multipart `file`、任意の `ttl_seconds` |
|
||||
| メタデータ | `GET /api/v1/artifacts/{id}` |
|
||||
| ダウンロード | `GET /api/v1/artifacts/{id}/content`、任意の `max_bytes` |
|
||||
| 構造化読み取り | `GET /api/v1/artifacts/{id}/read`、任意の `mode`、`offset`、`max_chars`、`page_start`、`page_end` |
|
||||
| 永続保持 | `POST /api/v1/artifacts/{id}/retain` |
|
||||
| 保持の解放 | `POST /api/v1/artifacts/{id}/release` |
|
||||
| 削除 | `DELETE /api/v1/artifacts/{id}` |
|
||||
@@ -26,6 +27,27 @@ Artifact はファイルの不透明な参照とメタデータを提供しま
|
||||
|
||||
## MCP
|
||||
|
||||
ネイティブPDF入力には信頼できるモデル能力情報が必要です。提供元がPDF対応を確認済みでも能力カタログにない別名の場合、管理者はモデルの既存の `abilities` に `pdf_input` を指定できます。宣言がなく能力が不明な場合、ネイティブPDFは明示的に拒否されますが、テキストモードは利用できます。
|
||||
|
||||
組み込み `/mcp` サーバーは、同じ所有者と権限のチェックを適用する `create_artifact`、`get_artifact`、`read_artifact`、`retain_artifact`、`release_artifact`、`delete_artifact` を提供します。作成には `filename`、`content_base64`、任意の `mime_type` と `ttl_seconds` を渡します。読み取りには `artifact_id` と任意の `max_bytes` を渡し、base64 とメタデータを受け取ります。
|
||||
|
||||
これらはファイル転送ツールです。テキスト抽出、PDF ページ描画、画像のモデル理解は行いません。初版ではプラグインの Workspace 権限、型付きメソッド、既定のモデルツール、`lbctl` からは利用できません。既存のアップロード API と run の `file_id` API は従来の契約を維持します。
|
||||
`read_artifact` は元のバイト列を転送します。アカウント所有の管理操作は、引き続きプラグインの Workspace 権限、型付きメソッド、既定のモデルツール、`lbctl` からは利用できません。既存のアップロード API と run の `file_id` API は従来の契約を維持します。
|
||||
|
||||
`read_artifact_content` は HTTP `/read` と同じパラメーターを受け付け、通常のテキストに base64 を入れず、MCP のテキスト、画像、埋め込み PDF を返します。Workspace 操作は `artifact.read_content` です。
|
||||
|
||||
## Agent による必要時の読み取り
|
||||
|
||||
Runner の初期入力には添付参照、名前、MIME 型、サイズのみが含まれます。画像やファイルのバイト列と取得元 URL は Host の実行内レジストリに保持され、最初のモデルメッセージには入りません。現在のメッセージと引用メッセージの添付は別々の参照を持ちます。必要な内容だけを取得し、Host は現在の run に固定された添付権限を確認します。他の run の参照はアクセスを許可しません。
|
||||
|
||||
`read_input_attachment_content` は構造化された内容を返します。
|
||||
|
||||
| モード | 動作 |
|
||||
| --- | --- |
|
||||
| `auto` | UTF-8 テキスト、構造化画像、または PDF 抽出テキストを返します。 |
|
||||
| `text` | 文字範囲を制限して読み取ります。PDF は対象ページを示し、画像やレイアウトを含みません。 |
|
||||
| `native` | 対応モデルへ画像や PDF を渡します。非対応モデルでは明示的に失敗します。 |
|
||||
| `pages` | ページ描画は現在未対応で、明示的なエラーになります。 |
|
||||
|
||||
テキストは 0 始まりの `offset` と `max_chars`(既定 8192、最大 16384)を使用します。結果の `truncated` と `next_offset` を確認して続きを読み取ってください。PDF テキストは 1 始まりの `page_start` と任意の `page_end` を使用し、1 回最大 10 ページで、次のページ位置も返します。OCR、画像、ページレイアウトは対象外です。他のバイナリ形式は明示的に拒否され、元のファイルはダウンロードできます。
|
||||
|
||||
Agent デバッグ API は、自分がアップロードした異なる ID を最大 20 件まで `artifact_ids` に指定できます。Host が認証済み主体の所有権を確認し、その run に参照だけを渡します。イベントの `actor` で所有者を変更することはできません。独立した Workspace 管理アシスタントの添付入力はこの段階では未対応です。既存の生バイト API `read_input_attachment` は権限のある Runner コード向けに維持されます。
|
||||
@@ -190,15 +190,15 @@ event_type = ctx.event.event_type
|
||||
|
||||
### 入力添付ファイル
|
||||
|
||||
`ctx.input.attachments` は、現在のメッセージに含まれる画像・音声・ファイルを出現順に 1 件ずつメタデータとして表します:`ref`(例:`att-1`)、`type`、`source`(現在のメッセージは `message`、引用されたメッセージは `quote`)、`name`、`mime_type`、`size`、プラットフォームの `id`、直接ダウンロードできる `url`。添付ファイルは内容を含みません。`content` フィールドは非推奨で、現行バージョンでは常に空です。
|
||||
`ctx.input.attachments` は、現在のメッセージに含まれる画像・音声・ファイルを出現順に 1 件ずつメタデータとして表します:`ref`(例:`att-1`)、`type`、`source`(現在のメッセージは `message`、引用されたメッセージは `quote`)、`name`、`mime_type`、`size`、プラットフォームの `id`。元URLはHostだけが保持します。添付ファイルは内容を含みません。`content` フィールドは非推奨で、現行バージョンでは常に空です。
|
||||
|
||||
`ctx.input.text` と `ctx.input.contents` には、各メディアの位置に `[attachment att-1: image photo.jpg]` のようなマーカーが入ります。引用されたメッセージは `[quoted message from <送信者>]` と `[/quoted message]` で囲まれます。画像は引き続きマーカーの直後に `image_base64` または `image_url` として置かれ、モデルが直接参照できます。
|
||||
`ctx.input.text` と `ctx.input.contents` には、各メディアの位置に `[attachment att-1: image photo.jpg]` のようなマーカーが入ります。引用されたメッセージは `[quoted message from <送信者>]` と `[/quoted message]` で囲まれます。画像やファイルは初期モデル入力には含まれず、必要なときに明示的に読み取ります。
|
||||
|
||||
| メソッド | 戻り値 | 説明 |
|
||||
| --- | --- | --- |
|
||||
| `await ctx.read_input_attachment(attachment)` | `InputAttachmentData \| None` | `ref` で添付ファイルのバイト列を読み取り、`data`、`name`、`mime_type`、`size` を返します。引数には `ref` 文字列も指定できます。LangBot がこの API を提供しない場合は `None` を返します |
|
||||
|
||||
添付ファイル 1 件あたりの上限は 10 MiB、1 回の実行で読み取れる合計は 50 MiB です。LangBot は読み取った内容をキャッシュせず、ログにも書き込みません。この API が使えるかどうかは `ctx.context.available_apis.input_attachments` で確認できます。steering で現在の実行に取り込まれた後続メッセージの添付ファイルはメタデータのみで、`ref` を持ちません。
|
||||
添付ファイル 1 件あたりの上限は 10 MiB、1 回の実行で読み取れる合計は 50 MiB です。内容読み取りAPIは初回に元データをrun専用の一時Artifactとして保存します。バイナリは会話履歴やツールイベントに保存しません。この API が使えるかどうかは `ctx.context.available_apis.input_attachments` で確認できます。steering で現在の実行に取り込まれた後続メッセージの添付ファイルはメタデータのみで、`ref` を持ちません。
|
||||
|
||||
このメソッドには `langbot-plugin` 0.7.23 以降が必要です。古い LangBot や SDK にも対応する場合は、メソッドの有無を確認してから従来の処理にフォールバックします:
|
||||
|
||||
@@ -210,6 +210,8 @@ if data is None:
|
||||
... # 古い LangBot:attachment.content、attachment.url、ctx.input.contents を使用
|
||||
```
|
||||
|
||||
`await ctx.read_input_attachment_content(ref, mode="auto", offset=0, max_chars=8192, page_start=1, page_end=None)` は同じrun権限で、制限付きテキストまたは構造化メディアを返します。モードと制限は [Artifactの読み取り](/ja/develop/artifacts) を参照してください。新メソッドには対応するCore、SDK、Runtimeが必要です。LocalAgentと一緒に更新してください。
|
||||
|
||||
### Box と添付ファイル
|
||||
|
||||
Box の有効化、取得、再利用方針は Runner が決めます。リソースの取得には [LangBot API](/ja/plugin/dev/apis/common#box-サンドボックス)、実行中の操作には以下を使います。
|
||||
|
||||
@@ -10,6 +10,8 @@ description: プラグインから Agent、パイプライン、ボット、モ
|
||||
|
||||
Artifact 操作は現在、認証済みアカウントと API キー専用です。プラグイン権限、ワイルドカード、型付き Workspace メソッドには含まれません。[Artifact ファイル API](/ja/develop/artifacts) を参照してください。
|
||||
|
||||
有界テキストと構造化画像/PDF を読む `artifact.read_content` も同じ境界に従います。Runner は既存の添付権限に制限された、run に結び付く `read_input_attachment_content` 実行 API を使用します。
|
||||
|
||||
## 操作を呼び出す
|
||||
|
||||
どのコンポーネントからも `self.plugin.workspace` を使います。各操作には ID に対応した型付きメソッドがあります。
|
||||
|
||||
@@ -12,6 +12,7 @@ Artifact 为文件提供不透明引用和元信息。账户或 API 密钥只能
|
||||
| 上传 | `POST /api/v1/artifacts`,multipart `file`,可选 `ttl_seconds` |
|
||||
| 元信息 | `GET /api/v1/artifacts/{id}` |
|
||||
| 下载 | `GET /api/v1/artifacts/{id}/content`,可选 `max_bytes` |
|
||||
| 读取结构化内容 | `GET /api/v1/artifacts/{id}/read`,可选 `mode`、`offset`、`max_chars`、`page_start`、`page_end` |
|
||||
| 持久保留 | `POST /api/v1/artifacts/{id}/retain` |
|
||||
| 释放保留 | `POST /api/v1/artifacts/{id}/release` |
|
||||
| 删除 | `DELETE /api/v1/artifacts/{id}` |
|
||||
@@ -26,6 +27,25 @@ Artifact 为文件提供不透明引用和元信息。账户或 API 密钥只能
|
||||
|
||||
## MCP
|
||||
|
||||
内置 `/mcp` 服务提供 `create_artifact`、`get_artifact`、`read_artifact`、`retain_artifact`、`release_artifact` 和 `delete_artifact`,使用相同的归属及权限检查。创建参数为 `filename`、`content_base64`,以及可选的 `mime_type` 和 `ttl_seconds`。读取参数为 `artifact_id` 及可选的 `max_bytes`,返回 base64 和元信息。
|
||||
原生 PDF 需要可信的模型能力信息;对于已由供应商确认支持、但能力目录尚未识别的模型别名,可由管理员在模型的现有 `abilities` 字段中声明 `pdf_input`。未声明且能力未知时会明确拒绝原生 PDF,仍可使用文本模式。
|
||||
|
||||
这些工具用于文件传输,不做文本提取、PDF 页面渲染,也不代表模型已理解图片。初版暂不开放插件工作区授权、插件类型化方法、默认模型工具或 `lbctl` 命令。原有上传接口和运行内 `file_id` 接口继续使用原契约。
|
||||
内置 `/mcp` 服务提供 `create_artifact`、`get_artifact`、`read_artifact`、`read_artifact_content`、`retain_artifact`、`release_artifact` 和 `delete_artifact`,使用相同的归属及权限检查。创建参数为 `filename`、`content_base64`,以及可选的 `mime_type` 和 `ttl_seconds`。原始读取参数为 `artifact_id` 及可选的 `max_bytes`,返回 base64 和元信息。`read_artifact_content` 沿用 HTTP `/read` 的读取参数,返回原生 MCP 文本、图片或嵌入 PDF 内容,不把 base64 放入普通文本。对应 Workspace 操作为 `artifact.read_content`。
|
||||
|
||||
`read_artifact` 传输原始字节。账户资产管理操作仍不开放插件工作区授权、插件类型化方法、默认模型工具或 `lbctl` 命令。原有上传接口和运行内 `file_id` 接口继续使用原契约。
|
||||
|
||||
## Agent 按需读取
|
||||
|
||||
Runner 初始输入只包含附件引用、名称、MIME 和大小。图片、文件字节和来源 URL 保留在 Host 的运行内注册表中,不直接进入首轮模型消息。当前消息和引用消息中的附件各有独立引用。Agent 需要内容时才调用读取工具,Host 继续检查当前 run 的冻结附件权限;另一个 run 的引用不能授权访问。
|
||||
|
||||
`read_input_attachment_content` 返回结构化内容,支持以下模式:
|
||||
|
||||
| 模式 | 行为 |
|
||||
| --- | --- |
|
||||
| `auto` | 读取 UTF-8 文本、返回结构化图片,或提取 PDF 文本。 |
|
||||
| `text` | 有界读取字符范围;PDF 结果标明所选页码,明确不包含视觉内容。 |
|
||||
| `native` | 将图片或 PDF 作为原生输入交给支持的模型;不支持时明确报错。 |
|
||||
| `pages` | 本阶段尚不支持页面渲染,返回明确错误。 |
|
||||
|
||||
文本使用从零开始的 `offset` 和 `max_chars`(默认 8192,最大 16384)。结果带 `truncated` 和 `next_offset`,应从该位置继续读取,不能将截断内容当作全文。PDF 文本使用从一开始的 `page_start` 和可选 `page_end`,每次最多十页,并返回下一页位置;不做 OCR,也不保留图片和页面布局。其他二进制格式明确拒绝,原始文件仍可下载。
|
||||
|
||||
Agent 调试 API 的请求可带 `artifact_ids`,最多 20 个不同的自有上传资产 ID。Host 在启动 run 前按真实认证主体检查归属,再给该 run 引用。请求中的事件 `actor` 不会改变资产归属。独立的 Workspace 管理助手本阶段仍不接受附件输入。已有原始字节接口 `read_input_attachment` 继续供有权限的 Runner 代码使用。
|
||||
@@ -192,15 +192,15 @@ event_type = ctx.event.event_type
|
||||
|
||||
### 输入附件
|
||||
|
||||
`ctx.input.attachments` 按出现顺序逐一描述当前消息中的图片、语音和文件,只包含元数据:`ref`(如 `att-1`)、`type`、`source`(`message` 为当前消息,`quote` 为被引用的消息)、`name`、`mime_type`、`size`、平台 `id` 和可直接下载的 `url`。附件不携带文件内容,`content` 字段已弃用,当前版本始终为空。
|
||||
`ctx.input.attachments` 按出现顺序逐一描述当前消息中的图片、语音和文件,只包含元数据:`ref`(如 `att-1`)、`type`、`source`(`message` 为当前消息,`quote` 为被引用的消息)、`name`、`mime_type`、`size`、平台 `id`。来源 URL 由 Host 私有保存。附件不携带文件内容,`content` 字段已弃用,当前版本始终为空。
|
||||
|
||||
`ctx.input.text` 和 `ctx.input.contents` 会在每个媒体的原位置插入标记,例如 `[attachment att-1: image photo.jpg]`;被引用的消息包裹在 `[quoted message from <发送者>]` 与 `[/quoted message]` 之间。图片仍以 `image_base64` 或 `image_url` 紧跟在标记之后,供模型直接查看。
|
||||
`ctx.input.text` 和 `ctx.input.contents` 会在每个媒体的原位置插入标记,例如 `[attachment att-1: image photo.jpg]`;被引用的消息包裹在 `[quoted message from <发送者>]` 与 `[/quoted message]` 之间。图片和文件不会直接进入初始模型输入,需要时再显式读取。
|
||||
|
||||
| 方法 | 返回值 | 说明 |
|
||||
| --- | --- | --- |
|
||||
| `await ctx.read_input_attachment(attachment)` | `InputAttachmentData \| None` | 按 `ref` 读取附件内容,返回 `data`、`name`、`mime_type` 和 `size`;参数也可以是 `ref` 字符串。LangBot 不提供该接口时返回 `None` |
|
||||
|
||||
单个附件最大 10 MiB,每次运行累计最多读取 50 MiB;LangBot 不缓存读取结果,也不把内容写入日志。`ctx.context.available_apis.input_attachments` 表示该接口是否可用。通过 steering 转入当前运行的后续消息,其附件只有元数据,没有 `ref`。
|
||||
单个附件最大 10 MiB,每次运行累计最多读取 50 MiB;内容读取 API 在首次使用时将原件存为当前 run 的临时 Artifact;二进制不写入对话历史或工具事件。`ctx.context.available_apis.input_attachments` 表示该接口是否可用。通过 steering 转入当前运行的后续消息,其附件只有元数据,没有 `ref`。
|
||||
|
||||
该方法需要 `langbot-plugin` 0.7.23 或更高版本。需要兼容旧版 LangBot 或旧版 SDK 时,先判断方法是否存在,再回退到原有逻辑:
|
||||
|
||||
@@ -212,6 +212,8 @@ if data is None:
|
||||
... # 旧版 LangBot:使用 attachment.content、attachment.url 或 ctx.input.contents
|
||||
```
|
||||
|
||||
`await ctx.read_input_attachment_content(ref, mode="auto", offset=0, max_chars=8192, page_start=1, page_end=None)` 使用同一冻结 run 权限,返回有界文本或结构化媒体。模式和限额见 [Artifact 读取](/zh/develop/artifacts)。该新方法要求匹配的 Core、SDK 和 Runtime,应与 LocalAgent 插件一同升级。
|
||||
|
||||
### Box 与附件
|
||||
|
||||
Box 是否启用、何时创建以及如何复用,由运行器决定。获取状态和申请 Box 使用 [LangBot API](/zh/plugin/dev/apis/common#box-沙箱);当前运行的绑定和附件操作使用下列方法。
|
||||
|
||||
@@ -10,6 +10,8 @@ description: 在插件中读取和管理 Agent、流水线、机器人、模型
|
||||
|
||||
Artifact 操作目前仅接受已认证账户和 API 密钥,不属于插件可申请的操作、通配授权或工作区类型化方法。参见 [Artifact 文件接口](/zh/develop/artifacts)。
|
||||
|
||||
新增 `artifact.read_content` 也遵循这一边界,提供有界文本和结构化图片/PDF 读取。Runner 代码使用运行绑定的 `read_input_attachment_content` 执行 API,继续受原有附件权限约束。
|
||||
|
||||
## 调用操作
|
||||
|
||||
在任意组件中通过 `self.plugin.workspace` 调用。每个操作都有一个按 ID 命名、带类型标注的方法:
|
||||
|
||||
@@ -34,6 +34,7 @@ from pkg.model_calling import (
|
||||
model_call_error_from_exception,
|
||||
normalize_llm_call_result,
|
||||
serialize_tool_result_content,
|
||||
split_tool_content,
|
||||
)
|
||||
|
||||
from .types import (
|
||||
@@ -314,16 +315,19 @@ class LangBotToolExecutor:
|
||||
)
|
||||
try:
|
||||
result = await self.local_tools[prepared.request.name](**prepared.parameters)
|
||||
outcome = self.finalize(prepared, result=result, error=None)
|
||||
if is_traced_queue:
|
||||
await results_queue.put(
|
||||
RunnerResult.tool_call_completed(
|
||||
run_id,
|
||||
call_id,
|
||||
prepared.request.name,
|
||||
result=result if isinstance(result, dict) else {"value": result},
|
||||
result=outcome.event_result
|
||||
if outcome.event_result is not None
|
||||
else (result if isinstance(result, dict) else {"value": result}),
|
||||
)
|
||||
)
|
||||
return self.finalize(prepared, result=result, error=None)
|
||||
return outcome
|
||||
except Exception as exc:
|
||||
if is_traced_queue:
|
||||
await results_queue.put(
|
||||
@@ -360,6 +364,24 @@ class LangBotToolExecutor:
|
||||
terminate = error is None and _tool_result_terminates(result)
|
||||
model_result = _strip_tool_runtime_hints(result) if error is None else result
|
||||
if error is None:
|
||||
structured = split_tool_content(model_result)
|
||||
if structured is not None:
|
||||
text, media = structured
|
||||
message = build_tool_call_message(
|
||||
prepared.request.id,
|
||||
prepared.request.name,
|
||||
text,
|
||||
max_result_chars=self.max_result_chars,
|
||||
)
|
||||
return ToolExecutionOutcome(
|
||||
request=prepared.request,
|
||||
parameters=prepared.parameters,
|
||||
result=result,
|
||||
event_result={"type": "langbot_tool_content", "summary": message.content},
|
||||
message=message,
|
||||
model_only_messages=[Message(role="user", content=media)],
|
||||
terminate=terminate,
|
||||
)
|
||||
references = extract_tool_result_references(model_result)
|
||||
if references["file_refs"]:
|
||||
message, event_result = build_tool_reference_message(
|
||||
@@ -444,6 +466,12 @@ class LangBotContextHooks(AgentLoopHooks):
|
||||
async def prepare_model_turn(self, messages: list[Message]) -> list[Message]:
|
||||
usage_anchor = self._usage_anchor
|
||||
self._usage_anchor = None
|
||||
if any(
|
||||
isinstance(message.content, list)
|
||||
and any(item.type.startswith(("image", "file")) for item in message.content)
|
||||
for message in messages
|
||||
):
|
||||
usage_anchor = None
|
||||
assembly = await ContextCompactor(
|
||||
self.budget,
|
||||
summarizer=self.summarizer,
|
||||
|
||||
@@ -58,6 +58,8 @@ class AgentLoop:
|
||||
self.tool_executor = tool_executor
|
||||
self.model_ids = list(model_ids)
|
||||
self.messages = [message.model_copy(deep=True) for message in messages]
|
||||
self._model_only_messages: list[Message] = []
|
||||
self._prepared_model_messages = list(self.messages)
|
||||
self.tools = list(tools)
|
||||
self.streaming = streaming
|
||||
self.max_tool_iterations = max_tool_iterations
|
||||
@@ -96,14 +98,30 @@ class AgentLoop:
|
||||
return index, outcome
|
||||
|
||||
async def _prepare_model_turn(self) -> None:
|
||||
messages = await self.hooks.prepare_model_turn(self.messages)
|
||||
self.messages = [message.model_copy(deep=True) for message in messages]
|
||||
messages = await self.hooks.prepare_model_turn(self.messages + self._model_only_messages)
|
||||
self._accept_prepared_messages(messages)
|
||||
|
||||
def _accept_prepared_messages(self, messages: list[Message]) -> None:
|
||||
durable_messages = list(messages)
|
||||
for media in self._model_only_messages:
|
||||
try:
|
||||
durable_messages.remove(media)
|
||||
except ValueError as exc:
|
||||
raise ModelCallError(
|
||||
"Attachment content was removed by context budgeting. Use a smaller attachment or text/page range.",
|
||||
retryable=False,
|
||||
) from exc
|
||||
self._prepared_model_messages = [message.model_copy(deep=True) for message in messages]
|
||||
self.messages = [message.model_copy(deep=True) for message in durable_messages]
|
||||
|
||||
async def _recover_context_overflow(self, error: ModelCallError) -> bool:
|
||||
messages = await self.hooks.recover_context_overflow(self.messages, error)
|
||||
messages = await self.hooks.recover_context_overflow(self._prepared_model_messages, error)
|
||||
if messages is None:
|
||||
return False
|
||||
self.messages = [message.model_copy(deep=True) for message in messages]
|
||||
try:
|
||||
self._accept_prepared_messages(messages)
|
||||
except ModelCallError:
|
||||
return False
|
||||
return True
|
||||
|
||||
def _should_execute_serially(self) -> bool:
|
||||
@@ -167,6 +185,7 @@ class AgentLoop:
|
||||
if outcome is None or outcome.message is None:
|
||||
continue
|
||||
self.messages.append(outcome.message)
|
||||
self._model_only_messages.extend(outcome.model_only_messages)
|
||||
tool_results.append(outcome.message)
|
||||
yield AgentLoopEvent.message_start(outcome.message)
|
||||
yield AgentLoopEvent.message_end(outcome.message)
|
||||
@@ -266,7 +285,7 @@ class AgentLoop:
|
||||
async with aclosing(
|
||||
self.model_adapter.stream_turn(
|
||||
model_ids=turn_model_ids,
|
||||
messages=self.messages,
|
||||
messages=self._prepared_model_messages,
|
||||
tools=self.tools,
|
||||
visible_prefix=visible_content_prefix,
|
||||
)
|
||||
@@ -392,13 +411,13 @@ class AgentLoop:
|
||||
if committed_model_id is None:
|
||||
model_turn = await self.model_adapter.invoke_turn(
|
||||
model_ids=candidate_model_ids,
|
||||
messages=self.messages,
|
||||
messages=self._prepared_model_messages,
|
||||
tools=self.tools,
|
||||
)
|
||||
else:
|
||||
model_turn = await self.model_adapter.invoke_committed_turn(
|
||||
committed_model_id=committed_model_id,
|
||||
messages=self.messages,
|
||||
messages=self._prepared_model_messages,
|
||||
tools=self.tools,
|
||||
)
|
||||
break
|
||||
|
||||
@@ -111,6 +111,7 @@ class ToolExecutionOutcome:
|
||||
event_result: typing.Any = None
|
||||
error: str | None = None
|
||||
message: Message | None = None
|
||||
model_only_messages: list[Message] = field(default_factory=list)
|
||||
terminate: bool = False
|
||||
|
||||
@property
|
||||
|
||||
@@ -13,7 +13,7 @@ import uuid
|
||||
from contextlib import aclosing
|
||||
from dataclasses import dataclass
|
||||
|
||||
from langbot_plugin.api.entities.builtin.provider.message import Message, MessageChunk
|
||||
from langbot_plugin.api.entities.builtin.provider.message import ContentElement, Message, MessageChunk
|
||||
from langbot_plugin.api.entities.builtin.resource.tool import LLMTool
|
||||
from langbot_plugin.api.proxies.runner import RunnerAPIProxy
|
||||
|
||||
@@ -866,6 +866,33 @@ def serialize_tool_result_content(result: typing.Any, *, is_error: bool = False)
|
||||
return json.dumps(result, ensure_ascii=False, default=str)
|
||||
|
||||
|
||||
def split_tool_content(result: typing.Any) -> tuple[list[ContentElement], list[ContentElement]] | None:
|
||||
"""Keep explicit multimodal tool output out of ordinary tool text and events."""
|
||||
if not isinstance(result, (list, tuple)) or not result:
|
||||
return None
|
||||
media_types = {"image_url", "image_base64", "file_url", "file_base64"}
|
||||
if not any(_model_or_mapping_get(item, "type") in media_types for item in result):
|
||||
return None
|
||||
text, media = [], []
|
||||
for raw in result:
|
||||
element = raw.model_copy(deep=True) if isinstance(raw, ContentElement) else ContentElement.model_validate(raw)
|
||||
if element.type == "text":
|
||||
text.append(element)
|
||||
elif element.type in media_types:
|
||||
payload = getattr(element, element.type)
|
||||
if not payload or (element.type == "image_url" and not payload.url):
|
||||
raise ValueError("Tool content is missing its media payload")
|
||||
media.append(element)
|
||||
text.append(
|
||||
ContentElement.from_text(
|
||||
f"[{element.type}: {element.file_name or 'content'} attached for model inspection]"
|
||||
)
|
||||
)
|
||||
else:
|
||||
raise ValueError(f"Unsupported tool content type: {element.type}")
|
||||
return text, media
|
||||
|
||||
|
||||
def _serialize_text_content_elements(result: typing.Any) -> str | None:
|
||||
"""Flatten MCP-style text-only ContentElement lists for model tool messages."""
|
||||
if not isinstance(result, (list, tuple)) or not result:
|
||||
|
||||
@@ -5,6 +5,7 @@ from __future__ import annotations
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
from langbot_plugin.api.agent_tools.external_tools import AgentRunExternalTools
|
||||
from langbot_plugin.api.entities.builtin.provider.message import Message
|
||||
from langbot_plugin.api.entities.builtin.resource.tool import LLMTool
|
||||
from langbot_plugin.api.entities.builtin.runner import RunnerContext
|
||||
@@ -61,6 +62,12 @@ class AgentRunAssembler:
|
||||
raise NoAuthorizedModelError(NO_AUTHORIZED_MODEL_MESSAGE)
|
||||
|
||||
allowed_tools = self._allowed_tool_names()
|
||||
input_tools = AgentRunExternalTools(self.api, self.ctx)
|
||||
input_tool = next(
|
||||
(tool for tool in input_tools.mcp_tools() if tool["name"] == "langbot_read_input_attachment"), None
|
||||
)
|
||||
if input_tool is not None:
|
||||
allowed_tools.discard(input_tool["name"])
|
||||
|
||||
max_tool_iterations = get_max_tool_iterations(self.ctx.config)
|
||||
max_tool_result_chars = get_max_tool_result_chars(self.ctx.config)
|
||||
@@ -78,6 +85,23 @@ class AgentRunAssembler:
|
||||
tools = await self._build_tools(allowed_tools)
|
||||
tools.extend(workspace_manager.tools)
|
||||
all_allowed_tools = allowed_tools | workspace_manager._tool_names
|
||||
local_tools = {}
|
||||
if input_tool is not None:
|
||||
|
||||
async def read_input(**arguments):
|
||||
return await input_tools.call_tool(input_tool["name"], arguments)
|
||||
|
||||
tools.append(
|
||||
LLMTool(
|
||||
name=input_tool["name"],
|
||||
human_desc=input_tool["description"],
|
||||
description=input_tool["description"],
|
||||
parameters=input_tool["inputSchema"],
|
||||
func=read_input,
|
||||
)
|
||||
)
|
||||
all_allowed_tools.add(input_tool["name"])
|
||||
local_tools[input_tool["name"]] = read_input
|
||||
|
||||
token_counter = HostContextTokenCounter(self.api, model_ids[0], tools, reasoning_level=reasoning_level)
|
||||
|
||||
@@ -90,16 +114,18 @@ class AgentRunAssembler:
|
||||
workspace_manager=workspace_manager,
|
||||
).assemble()
|
||||
|
||||
tool_executor = LangBotToolExecutor(
|
||||
self.api,
|
||||
all_allowed_tools,
|
||||
max_result_chars=max_tool_result_chars,
|
||||
workspace_manager=workspace_manager,
|
||||
)
|
||||
tool_executor.local_tools.update(local_tools)
|
||||
return AgentRunAssembly(
|
||||
model_ids=model_ids,
|
||||
messages=context_assembly.messages,
|
||||
tools=tools,
|
||||
tool_executor=LangBotToolExecutor(
|
||||
self.api,
|
||||
all_allowed_tools,
|
||||
max_result_chars=max_tool_result_chars,
|
||||
workspace_manager=workspace_manager,
|
||||
),
|
||||
tool_executor=tool_executor,
|
||||
hooks=LangBotContextHooks(
|
||||
context_budget,
|
||||
summarizer=summarizer,
|
||||
|
||||
@@ -3,10 +3,17 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
from unittest.mock import AsyncMock
|
||||
|
||||
import pytest
|
||||
from langbot_plugin.api.entities.builtin.provider.message import FunctionCall, Message, MessageChunk, ToolCall
|
||||
from langbot_plugin.api.entities.builtin.provider.message import (
|
||||
ContentElement,
|
||||
FunctionCall,
|
||||
Message,
|
||||
MessageChunk,
|
||||
ToolCall,
|
||||
)
|
||||
from langbot_plugin.api.entities.builtin.runner.errors import AgentAPIError, AgentAPIException
|
||||
from langbot_plugin.api.entities.builtin.runner.steering import SteeringPullResult
|
||||
from langbot_plugin.api.proxies.runner import PermissionDeniedError
|
||||
@@ -28,10 +35,199 @@ from pkg.context_pipeline import ContextBudget
|
||||
from pkg.model_calling import ModelCallError, StreamingModelCaller
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("streaming", [False, True])
|
||||
async def test_multimodal_tool_content_reaches_model_without_entering_history(streaming):
|
||||
payload = "data:image/png;base64,cHJpdmF0ZS1pbWFnZS1ieXRlcw=="
|
||||
|
||||
class FakeAPI:
|
||||
def __init__(self):
|
||||
self.calls = []
|
||||
self.call_tool = AsyncMock(
|
||||
return_value=[
|
||||
{"type": "text", "text": '{"artifact_id":"art_test","mime_type":"image/png"}'},
|
||||
{"type": "image_base64", "image_base64": payload},
|
||||
]
|
||||
)
|
||||
|
||||
def response(self, messages):
|
||||
self.calls.append([message.model_copy(deep=True) for message in messages])
|
||||
if len(self.calls) == 1:
|
||||
return Message(
|
||||
role="assistant",
|
||||
content="Reading",
|
||||
tool_calls=[
|
||||
ToolCall(
|
||||
id="read-1",
|
||||
type="function",
|
||||
function=FunctionCall(name="read_input_attachment_content", arguments="{}"),
|
||||
)
|
||||
],
|
||||
)
|
||||
media = messages[-1]
|
||||
assert media.role == "user" and media.content[0].image_base64 == payload
|
||||
assert messages[-2].role == "tool" and payload not in messages[-2].content
|
||||
return Message(role="assistant", content="The image has been inspected")
|
||||
|
||||
async def invoke_llm(self, *args, **kwargs):
|
||||
return self.response(kwargs["messages"])
|
||||
|
||||
def invoke_llm_stream(self, *args, **kwargs):
|
||||
response = self.response(kwargs["messages"])
|
||||
|
||||
async def stream():
|
||||
yield MessageChunk(
|
||||
role="assistant", content=response.content, tool_calls=response.tool_calls, is_final=True
|
||||
)
|
||||
|
||||
return stream()
|
||||
|
||||
api = FakeAPI()
|
||||
loop = AgentLoop(
|
||||
model_adapter=LangBotModelAdapter(api),
|
||||
tool_executor=LangBotToolExecutor(api, {"read_input_attachment_content"}),
|
||||
model_ids=["model-1"],
|
||||
messages=[Message(role="user", content="Inspect my image")],
|
||||
tools=[],
|
||||
streaming=streaming,
|
||||
max_tool_iterations=2,
|
||||
)
|
||||
events = [event async for event in loop.run()]
|
||||
assert len(api.calls) == 2
|
||||
assert events[-1].type == AgentLoopEventType.AGENT_END
|
||||
assert payload not in json.dumps([message.model_dump() for message in loop.messages])
|
||||
for event in events:
|
||||
if event.message is not None:
|
||||
assert payload not in event.message.model_dump_json()
|
||||
assert payload not in json.dumps(event.result)
|
||||
|
||||
|
||||
async def _collect_events(loop: AgentLoop):
|
||||
return [event async for event in loop.run()]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("streaming", [False, True])
|
||||
async def test_attachment_tokens_participate_in_budget_and_cannot_be_silently_dropped(streaming):
|
||||
payload = "data:image/png;base64,cHJpdmF0ZS1idWRnZXQ="
|
||||
|
||||
class MediaCounter:
|
||||
def __init__(self):
|
||||
self.saw_media = False
|
||||
|
||||
async def count(self, messages):
|
||||
media = sum(
|
||||
1
|
||||
for message in messages
|
||||
if isinstance(message.content, list)
|
||||
for item in message.content
|
||||
if item.type == "image_base64"
|
||||
)
|
||||
self.saw_media |= bool(media)
|
||||
return 80 * media + 2 * len(messages)
|
||||
|
||||
class NeverModel:
|
||||
async def invoke_turn(self, **kwargs):
|
||||
raise AssertionError("A model must not run after its unread image was removed to fit the budget")
|
||||
|
||||
async def stream_turn(self, **kwargs):
|
||||
raise AssertionError("A model must not run after its unread image was removed to fit the budget")
|
||||
yield
|
||||
|
||||
counter = MediaCounter()
|
||||
loop = AgentLoop(
|
||||
model_adapter=NeverModel(),
|
||||
tool_executor=LangBotToolExecutor(AsyncMock(), set()),
|
||||
model_ids=["model-1"],
|
||||
messages=[Message(role="user", content="Inspect this image")],
|
||||
tools=[],
|
||||
streaming=streaming,
|
||||
max_tool_iterations=2,
|
||||
hooks=LangBotContextHooks(
|
||||
ContextBudget(window_tokens=40, reserve_tokens=0, keep_recent_tokens=0, summary_tokens=0),
|
||||
token_counter=counter,
|
||||
),
|
||||
)
|
||||
loop._model_only_messages = [Message(role="user", content=[ContentElement.from_image_base64(payload)])]
|
||||
events = await _collect_events(loop)
|
||||
assert counter.saw_media
|
||||
assert events[-1].type == AgentLoopEventType.RUN_FAILED
|
||||
assert "Attachment content was removed by context budgeting" in events[-1].error
|
||||
assert payload not in json.dumps([message.model_dump() for message in loop.messages])
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("streaming", [False, True])
|
||||
async def test_overflow_recovery_receives_media_and_cannot_retry_after_discarding_it(streaming):
|
||||
payload = "data:image/png;base64,cHJpdmF0ZS1vdmVyZmxvdw=="
|
||||
|
||||
class OverflowModel:
|
||||
calls = 0
|
||||
|
||||
async def invoke_turn(self, *, messages, **kwargs):
|
||||
self.calls += 1
|
||||
assert messages[-1].content[0].image_base64 == payload
|
||||
raise ModelCallError("context length exceeded", retryable=True)
|
||||
|
||||
async def stream_turn(self, **kwargs):
|
||||
await self.invoke_turn(**kwargs)
|
||||
yield
|
||||
|
||||
class DroppingRecovery(AgentLoopHooks):
|
||||
saw_media = False
|
||||
|
||||
async def recover_context_overflow(self, messages, error):
|
||||
self.saw_media = messages[-1].content[0].image_base64 == payload
|
||||
return [Message(role="user", content="retry without the image")]
|
||||
|
||||
model, hooks = OverflowModel(), DroppingRecovery()
|
||||
loop = AgentLoop(
|
||||
model_adapter=model,
|
||||
tool_executor=LangBotToolExecutor(AsyncMock(), set()),
|
||||
model_ids=["model-1"],
|
||||
messages=[Message(role="user", content="Inspect this image")],
|
||||
tools=[],
|
||||
streaming=streaming,
|
||||
max_tool_iterations=2,
|
||||
hooks=hooks,
|
||||
)
|
||||
loop._model_only_messages = [Message(role="user", content=[ContentElement.from_image_base64(payload)])]
|
||||
events = await _collect_events(loop)
|
||||
assert hooks.saw_media and model.calls == 1
|
||||
assert events[-1].type == AgentLoopEventType.RUN_FAILED
|
||||
assert "context length exceeded" in events[-1].error
|
||||
assert payload not in json.dumps([message.model_dump() for message in loop.messages])
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_budget_cannot_keep_one_copy_of_a_twice_read_image_and_restore_the_discarded_copy():
|
||||
media = Message(role="user", content=[ContentElement.from_image_base64("data:image/png;base64,c2FtZQ==")])
|
||||
|
||||
class DropDuplicate(AgentLoopHooks):
|
||||
async def prepare_model_turn(self, messages):
|
||||
assert messages[-2] == messages[-1]
|
||||
return messages[:-1]
|
||||
|
||||
class NeverModel:
|
||||
async def invoke_turn(self, **kwargs):
|
||||
raise AssertionError("Only one of two media copies was budgeted")
|
||||
|
||||
loop = AgentLoop(
|
||||
model_adapter=NeverModel(),
|
||||
tool_executor=LangBotToolExecutor(AsyncMock(), set()),
|
||||
model_ids=["model-1"],
|
||||
messages=[Message(role="user", content="Inspect the two reads")],
|
||||
tools=[],
|
||||
streaming=False,
|
||||
max_tool_iterations=2,
|
||||
hooks=DropDuplicate(),
|
||||
)
|
||||
loop._model_only_messages = [media, media.model_copy(deep=True)]
|
||||
events = await _collect_events(loop)
|
||||
assert events[-1].type == AgentLoopEventType.RUN_FAILED
|
||||
assert "Attachment content was removed by context budgeting" in events[-1].error
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_steering_puller_accepts_sdk_result_models():
|
||||
"""Host steering_pull returns SDK DTOs, not plain dicts."""
|
||||
|
||||
@@ -1,17 +1,69 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import AsyncMock
|
||||
|
||||
import pytest
|
||||
from langbot_plugin.api.entities.builtin.runner.input import InputAttachment
|
||||
from langbot_plugin.api.entities.builtin.runner.resources import AgentResources, ModelResource, ToolResource
|
||||
|
||||
from pkg.agent_core import LangBotContextHooks, LangBotToolExecutor
|
||||
from pkg.agent_core import LangBotContextHooks, LangBotToolExecutor, ToolCallRequest
|
||||
from pkg.messages import build_platform_tools_system_message
|
||||
from pkg.run_assembly import AgentRunAssembler, NoAuthorizedModelError
|
||||
from tests.test_runner import FakeRunnerAPIProxy, make_context
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("granted,has_attachment", [(False, True), (True, False), (True, True)])
|
||||
async def test_input_reader_is_discovered_from_the_run_grant_and_traces_no_binary(granted, has_attachment):
|
||||
api = FakeRunnerAPIProxy(models=[ModelResource(model_id="model-primary")])
|
||||
ctx = make_context(
|
||||
config={"model": {"primary": "model-primary"}, "box-enabled": False},
|
||||
resources=AgentResources(models=[ModelResource(model_id="model-primary")]),
|
||||
)
|
||||
ctx.context.available_apis.input_attachments = granted
|
||||
ctx.input.attachments = [InputAttachment(ref="att-1", type="image")] if has_attachment else []
|
||||
payload = "data:image/png;base64,cHJpdmF0ZQ=="
|
||||
api.read_input_attachment_content = AsyncMock(
|
||||
return_value=[
|
||||
{"type": "text", "text": '{"ref":"att-1"}'},
|
||||
{"type": "image_base64", "image_base64": payload},
|
||||
]
|
||||
)
|
||||
api._results = asyncio.Queue()
|
||||
api._run_id = "input-run"
|
||||
assembly = await AgentRunAssembler(api, ctx).assemble()
|
||||
names = {tool.name for tool in assembly.tools}
|
||||
assert ("langbot_read_input_attachment" in names) is (granted and has_attachment)
|
||||
if not (granted and has_attachment):
|
||||
api.read_input_attachment_content.assert_not_awaited()
|
||||
return
|
||||
prepared = assembly.tool_executor.prepare(
|
||||
ToolCallRequest.from_raw(
|
||||
{
|
||||
"id": "call-read",
|
||||
"function": {"name": "langbot_read_input_attachment", "arguments": '{"ref":"att-1"}'},
|
||||
}
|
||||
)
|
||||
)
|
||||
outcome = await assembly.tool_executor.execute(prepared)
|
||||
assert outcome.error is None and outcome.model_only_messages[0].content[0].image_base64 == payload
|
||||
api.read_input_attachment_content.assert_awaited_once_with(
|
||||
ref="att-1",
|
||||
mode="auto",
|
||||
offset=0,
|
||||
max_chars=8192,
|
||||
page_start=1,
|
||||
page_end=None,
|
||||
)
|
||||
assert payload not in outcome.message.content
|
||||
started, completed = api._results.get_nowait(), api._results.get_nowait()
|
||||
assert started.type.value == "tool.call.started" and completed.type.value == "tool.call.completed"
|
||||
assert payload not in json.dumps(completed.data)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_assembler_builds_authorized_loop_inputs() -> None:
|
||||
api = FakeRunnerAPIProxy(
|
||||
|
||||
+1
-1
@@ -51,7 +51,7 @@ dependencies = [
|
||||
"python-socks>=2.7.1", # dingtalk missing dependency
|
||||
"pip>=26.2.0",
|
||||
"uv>=0.13.4",
|
||||
"pypdf>=6.0.0",
|
||||
"pypdf>=6.21.2",
|
||||
"python-docx>=1.1.0",
|
||||
"chardet>=5.2.0",
|
||||
"xxhash>=3.5.0",
|
||||
|
||||
@@ -6,6 +6,7 @@ import base64
|
||||
import copy
|
||||
import json
|
||||
import typing
|
||||
from urllib.parse import quote
|
||||
|
||||
import pydantic
|
||||
|
||||
@@ -39,6 +40,11 @@ class ImportBoxArgs(EmptyArgs):
|
||||
|
||||
class ReadInputAttachmentArgs(EmptyArgs):
|
||||
ref: str = pydantic.Field(min_length=1)
|
||||
mode: typing.Literal["auto", "text", "native", "pages"] = "auto"
|
||||
offset: int = pydantic.Field(default=0, ge=0)
|
||||
max_chars: int = pydantic.Field(default=8192, ge=1, le=16384)
|
||||
page_start: int = pydantic.Field(default=1, ge=1)
|
||||
page_end: int | None = pydantic.Field(default=None, ge=1)
|
||||
|
||||
|
||||
class ReplyFilesArgs(EmptyArgs):
|
||||
@@ -252,6 +258,88 @@ class AgentRunExternalTools:
|
||||
}
|
||||
|
||||
structured = _dump_jsonable(result)
|
||||
content_types = {"text", "image_base64", "file_base64", "image_url", "file_url"}
|
||||
if (
|
||||
isinstance(structured, list)
|
||||
and structured
|
||||
and all(
|
||||
isinstance(item, dict) and item.get("type") in content_types
|
||||
for item in structured
|
||||
)
|
||||
):
|
||||
contents = []
|
||||
try:
|
||||
for item in structured:
|
||||
kind = item["type"]
|
||||
if kind == "text":
|
||||
if not isinstance(item.get("text"), str):
|
||||
raise ValueError("Missing tool text content")
|
||||
contents.append({"type": "text", "text": item["text"]})
|
||||
elif kind in {"image_url", "file_url"}:
|
||||
uri = (
|
||||
item["image_url"]["url"]
|
||||
if kind == "image_url"
|
||||
else item["file_url"]
|
||||
)
|
||||
if not isinstance(uri, str) or not uri:
|
||||
raise ValueError("Missing tool media URL")
|
||||
contents.append(
|
||||
{
|
||||
"type": "resource_link",
|
||||
"uri": uri,
|
||||
"name": item.get("file_name") or "image",
|
||||
}
|
||||
)
|
||||
else:
|
||||
encoded = item[kind]
|
||||
if not isinstance(encoded, str) or not encoded:
|
||||
raise ValueError("Missing tool media payload")
|
||||
mime_type = (
|
||||
"image/png"
|
||||
if kind == "image_base64"
|
||||
else "application/octet-stream"
|
||||
)
|
||||
if encoded.startswith("data:"):
|
||||
header, separator, encoded = encoded.partition(",")
|
||||
if not separator or not header.endswith(";base64"):
|
||||
raise ValueError("Invalid tool media data URL")
|
||||
mime_type = header[5:-7]
|
||||
if not mime_type or (
|
||||
kind == "image_base64"
|
||||
and not mime_type.startswith("image/")
|
||||
):
|
||||
raise ValueError("Invalid tool media MIME type")
|
||||
base64.b64decode(encoded, validate=True)
|
||||
if kind == "image_base64":
|
||||
contents.append(
|
||||
{
|
||||
"type": "image",
|
||||
"data": encoded,
|
||||
"mimeType": mime_type,
|
||||
}
|
||||
)
|
||||
else:
|
||||
contents.append(
|
||||
{
|
||||
"type": "resource",
|
||||
"resource": {
|
||||
"uri": "langbot-artifact:///"
|
||||
+ quote(
|
||||
item.get("file_name") or "file", safe=""
|
||||
),
|
||||
"mimeType": mime_type,
|
||||
"blob": encoded,
|
||||
},
|
||||
}
|
||||
)
|
||||
return {"content": contents}
|
||||
except (KeyError, TypeError, ValueError) as exc:
|
||||
return {
|
||||
"isError": True,
|
||||
"content": [
|
||||
{"type": "text", "text": f"Invalid tool media content: {exc}"}
|
||||
],
|
||||
}
|
||||
if not isinstance(structured, dict):
|
||||
structured = {"result": structured}
|
||||
return {
|
||||
@@ -503,18 +591,11 @@ class AgentRunExternalTools:
|
||||
|
||||
@agent_tool(
|
||||
name="langbot_read_input_attachment",
|
||||
description="Read one current input attachment by ref, such as att-1 from [attachment att-1: ...] in the message. Returns name, mime_type, size and base64 content.",
|
||||
description="Read a current input attachment by ref. Auto returns bounded text or a structured image; PDF defaults to text. Use native for PDF visual content with a supporting model. Text offsets are Unicode characters; use next_offset to continue. At most ten PDF pages per call. Page rendering is unavailable.",
|
||||
args_model=ReadInputAttachmentArgs,
|
||||
)
|
||||
async def read_input_attachment(self, args: ReadInputAttachmentArgs):
|
||||
data = await self.api.read_input_attachment(args.ref)
|
||||
return {
|
||||
"ref": data.ref,
|
||||
"name": data.name,
|
||||
"mime_type": data.mime_type,
|
||||
"size": data.size,
|
||||
"content_base64": base64.b64encode(data.data).decode("ascii"),
|
||||
}
|
||||
return await self.api.read_input_attachment_content(**args.model_dump())
|
||||
|
||||
@agent_tool(
|
||||
name="langbot_export_box_files",
|
||||
|
||||
@@ -190,6 +190,10 @@ class RunnerContext(pydantic.BaseModel):
|
||||
return None
|
||||
return await self.api.read_input_attachment(ref)
|
||||
|
||||
async def read_input_attachment_content(self, ref: str, **options) -> list[dict]:
|
||||
"""Read bounded text or structured media using this run's input grant."""
|
||||
return await self.api.read_input_attachment_content(ref, **options)
|
||||
|
||||
async def import_box_attachments(
|
||||
self, attachment_ids: list[str] | None = None
|
||||
) -> list["BoxFile"]:
|
||||
|
||||
@@ -41,7 +41,7 @@ class InputAttachment(pydantic.BaseModel):
|
||||
such as url, base64 or message_chain."""
|
||||
|
||||
url: str | None = None
|
||||
"""External URL when the attachment is backed by a URL."""
|
||||
"""Legacy external URL; current Hosts keep source locators private."""
|
||||
|
||||
path: str | None = None
|
||||
"""Sandbox/workspace path when Host has staged the attachment as a file."""
|
||||
|
||||
@@ -133,6 +133,32 @@ class LangBotAPIProxy:
|
||||
)
|
||||
return [BoxFile.model_validate(item) for item in result.get("items", [])]
|
||||
|
||||
@run_scoped
|
||||
async def read_input_attachment_content(
|
||||
self,
|
||||
ref: str,
|
||||
*,
|
||||
mode: str = "auto",
|
||||
offset: int = 0,
|
||||
max_chars: int = 8192,
|
||||
page_start: int = 1,
|
||||
page_end: int | None = None,
|
||||
):
|
||||
"""Read bounded model content from this run's input attachment."""
|
||||
result = await self.plugin_runtime_handler.call_action(
|
||||
PluginToRuntimeAction.READ_INPUT_ATTACHMENT_CONTENT,
|
||||
{
|
||||
"ref": ref,
|
||||
"mode": mode,
|
||||
"offset": offset,
|
||||
"max_chars": max_chars,
|
||||
"page_start": page_start,
|
||||
"page_end": page_end,
|
||||
},
|
||||
180,
|
||||
)
|
||||
return result["contents"]
|
||||
|
||||
@run_scoped
|
||||
async def read_input_attachment(self, ref: str):
|
||||
"""Read the bytes of one current-event attachment by ref."""
|
||||
|
||||
@@ -281,6 +281,36 @@ class RunnerContextAPIMixin:
|
||||
data=data,
|
||||
)
|
||||
|
||||
async def read_input_attachment_content(
|
||||
self,
|
||||
ref: str,
|
||||
*,
|
||||
mode: str = "auto",
|
||||
offset: int = 0,
|
||||
max_chars: int = 8192,
|
||||
page_start: int = 1,
|
||||
page_end: int | None = None,
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Read bounded model content using the current run's attachment grant."""
|
||||
self._require_context_api("input_attachments")
|
||||
response = await self._api.plugin_runtime_handler.call_action(
|
||||
PluginToRuntimeAction.READ_INPUT_ATTACHMENT_CONTENT,
|
||||
{
|
||||
"run_id": self.run_id,
|
||||
"ref": ref,
|
||||
"mode": mode,
|
||||
"offset": offset,
|
||||
"max_chars": max_chars,
|
||||
"page_start": page_start,
|
||||
"page_end": page_end,
|
||||
},
|
||||
self._bounded_timeout(default=120.0),
|
||||
cancel_peer_on_cancel=True,
|
||||
)
|
||||
return self._expect_key(
|
||||
response, "contents", PluginToRuntimeAction.READ_INPUT_ATTACHMENT_CONTENT
|
||||
)
|
||||
|
||||
async def steering_ack(self, delivery_ids: list[str]) -> SteeringAckResult:
|
||||
"""Acknowledge claimed steering inputs after applying them."""
|
||||
self._require_context_api("steering_ack")
|
||||
|
||||
@@ -8,6 +8,30 @@ from langbot_plugin.api.entities.builtin.runner.result import RunnerResult
|
||||
from langbot_plugin.api.proxies.runner.api import RunnerAPIProxy
|
||||
|
||||
|
||||
def _tool_trace_result(value):
|
||||
"""Keep structured media out of retained tool events without changing the tool result."""
|
||||
if hasattr(value, "model_dump"):
|
||||
value = value.model_dump(mode="json")
|
||||
if isinstance(value, list):
|
||||
return [_tool_trace_result(item) for item in value]
|
||||
if not isinstance(value, dict):
|
||||
return (
|
||||
"[binary content redacted]"
|
||||
if isinstance(value, str)
|
||||
and value.startswith("data:")
|
||||
and ";base64," in value
|
||||
else value
|
||||
)
|
||||
output = {}
|
||||
for key, item in value.items():
|
||||
if key in {"base64", "image_base64", "file_base64"} and item:
|
||||
output[key] = None
|
||||
output["content_redacted"] = True
|
||||
else:
|
||||
output[key] = _tool_trace_result(item)
|
||||
return output
|
||||
|
||||
|
||||
class _TracedRunAPI:
|
||||
"""Delegate authorized APIs and stream tool delivery attempts and outcomes."""
|
||||
|
||||
@@ -48,7 +72,9 @@ class _TracedRunAPI:
|
||||
self._run_id,
|
||||
call_id,
|
||||
tool_name,
|
||||
result=result,
|
||||
result=_tool_trace_result(result)
|
||||
if isinstance(result, dict)
|
||||
else {"value": _tool_trace_result(result)},
|
||||
)
|
||||
)
|
||||
return result
|
||||
|
||||
@@ -41,6 +41,7 @@ class PluginToRuntimeAction(ActionType):
|
||||
BIND_BOX = "bind_box"
|
||||
IMPORT_BOX_ATTACHMENTS = "import_box_attachments"
|
||||
READ_INPUT_ATTACHMENT = "read_input_attachment"
|
||||
READ_INPUT_ATTACHMENT_CONTENT = "read_input_attachment_content"
|
||||
EXPORT_BOX_FILES = "export_box_files"
|
||||
|
||||
GET_BOTS = "get_bots"
|
||||
|
||||
@@ -169,6 +169,7 @@ class PluginConnectionHandler(handler.Handler):
|
||||
in {
|
||||
PluginToRuntimeAction.CALL_TOOL,
|
||||
PluginToRuntimeAction.REQUEST_INTERACTION,
|
||||
PluginToRuntimeAction.READ_INPUT_ATTACHMENT_CONTENT,
|
||||
}
|
||||
and payload.get("run_id") is not None
|
||||
)
|
||||
@@ -922,6 +923,16 @@ class PluginConnectionHandler(handler.Handler):
|
||||
LONG_RUNNING_OPERATION_TIMEOUT,
|
||||
)
|
||||
|
||||
@self.action(PluginToRuntimeAction.READ_INPUT_ATTACHMENT_CONTENT)
|
||||
async def read_input_attachment_content(
|
||||
data: dict[str, Any],
|
||||
) -> handler.ActionResponse:
|
||||
return await forward_agent_action(
|
||||
PluginToRuntimeAction.READ_INPUT_ATTACHMENT_CONTENT,
|
||||
data,
|
||||
LONG_RUNNING_OPERATION_TIMEOUT,
|
||||
)
|
||||
|
||||
@self.action(PluginToRuntimeAction.EXPORT_BOX_FILES)
|
||||
async def export_box_files(data: dict[str, Any]) -> handler.ActionResponse:
|
||||
return await forward_agent_action(
|
||||
|
||||
@@ -23,6 +23,7 @@ from langbot_plugin.api.entities.builtin.runner.runtime import AgentRuntimeConte
|
||||
from langbot_plugin.api.entities.builtin.runner.trigger import AgentTrigger
|
||||
from langbot_plugin.api.proxies.runner import RunnerAPIProxy
|
||||
from langbot_plugin.api.proxies.runner.common import PermissionDeniedError
|
||||
from langbot_plugin.api.entities.builtin.runner.errors import AgentAPIException
|
||||
from langbot_plugin.entities.io.actions.enums import PluginToRuntimeAction
|
||||
|
||||
PAYLOAD = b"%PDF-1.7 test"
|
||||
@@ -117,3 +118,47 @@ def test_older_host_context_without_the_flag_disables_the_api():
|
||||
access = ContextAccess.model_validate({"available_apis": {"box": True}})
|
||||
|
||||
assert access.available_apis.input_attachments is False
|
||||
|
||||
|
||||
@pytest.mark.anyio
|
||||
async def test_content_proxy_preserves_structured_output_and_run_scope():
|
||||
contents = [
|
||||
{"type": "text", "text": "metadata"},
|
||||
{"type": "image_base64", "image_base64": "data:image/png;base64,aGk="},
|
||||
]
|
||||
handler = _handler()
|
||||
handler.call_action.return_value = {"contents": contents}
|
||||
ctx = _context(enabled=True)
|
||||
ctx._api = RunnerAPIProxy(ctx=ctx, plugin_runtime_handler=handler)
|
||||
assert (
|
||||
await ctx.read_input_attachment_content(
|
||||
"att-1", mode="native", page_start=2, page_end=3
|
||||
)
|
||||
== contents
|
||||
)
|
||||
action, data, timeout = handler.call_action.call_args.args
|
||||
assert action == PluginToRuntimeAction.READ_INPUT_ATTACHMENT_CONTENT
|
||||
assert data == {
|
||||
"run_id": "run-1",
|
||||
"ref": "att-1",
|
||||
"mode": "native",
|
||||
"offset": 0,
|
||||
"max_chars": 8192,
|
||||
"page_start": 2,
|
||||
"page_end": 3,
|
||||
}
|
||||
assert timeout > 0
|
||||
assert handler.call_action.call_args.kwargs["cancel_peer_on_cancel"] is True
|
||||
|
||||
|
||||
@pytest.mark.anyio
|
||||
async def test_content_proxy_rejects_missing_capability_and_malformed_response():
|
||||
handler = _handler()
|
||||
proxy = RunnerAPIProxy(ctx=_context(enabled=False), plugin_runtime_handler=handler)
|
||||
with pytest.raises(PermissionDeniedError):
|
||||
await proxy.read_input_attachment_content("att-1")
|
||||
handler.call_action.assert_not_called()
|
||||
proxy = RunnerAPIProxy(ctx=_context(enabled=True), plugin_runtime_handler=handler)
|
||||
with pytest.raises(AgentAPIException) as failure:
|
||||
await proxy.read_input_attachment_content("att-1")
|
||||
assert failure.value.error.code == "host.malformed_response"
|
||||
@@ -0,0 +1,33 @@
|
||||
"""Model content returns intact while retained tool traces contain no inline bytes."""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import AsyncMock
|
||||
|
||||
import pytest
|
||||
|
||||
from langbot_plugin.api.proxies.runner.tracing import _TracedRunAPI
|
||||
|
||||
|
||||
@pytest.mark.anyio
|
||||
async def test_tool_content_list_is_valid_trace_output_and_binary_is_redacted():
|
||||
payload = "data:image/png;base64,cHJpdmF0ZS1ieXRlcw=="
|
||||
result = [
|
||||
{"type": "text", "text": "image metadata"},
|
||||
{"type": "image_base64", "image_base64": payload},
|
||||
]
|
||||
queue = asyncio.Queue()
|
||||
api = _TracedRunAPI(
|
||||
SimpleNamespace(call_tool=AsyncMock(return_value=result)), "run-1", queue
|
||||
)
|
||||
returned = await api.call_tool("read", {"ref": "att-1"})
|
||||
assert returned is result and returned[1]["image_base64"] == payload
|
||||
started, completed = queue.get_nowait(), queue.get_nowait()
|
||||
assert (
|
||||
started.type.value == "tool.call.started"
|
||||
and completed.type.value == "tool.call.completed"
|
||||
)
|
||||
assert completed.data["result"]["value"][0]["text"] == "image metadata"
|
||||
assert completed.data["result"]["value"][1]["content_redacted"] is True
|
||||
assert payload not in json.dumps(completed.data)
|
||||
@@ -218,19 +218,13 @@ async def test_external_box_tools_use_authorized_run_api(
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_external_input_attachment_tool_reads_by_ref():
|
||||
from langbot_plugin.api.entities.builtin.runner.input import (
|
||||
InputAttachment,
|
||||
InputAttachmentData,
|
||||
)
|
||||
from langbot_plugin.api.entities.builtin.runner.input import InputAttachment
|
||||
|
||||
ctx = _authorized_ctx()
|
||||
ctx.input.attachments = [InputAttachment(ref="att-1", type="file")]
|
||||
api = MagicMock()
|
||||
api.read_input_attachment = AsyncMock(
|
||||
return_value=InputAttachmentData(
|
||||
ref="att-1", name="a.txt", mime_type="text/plain", data=b"hi"
|
||||
)
|
||||
)
|
||||
contents = [{"type": "text", "text": '{"ref":"att-1","text":"hi"}'}]
|
||||
api.read_input_attachment_content = AsyncMock(return_value=contents)
|
||||
tools = AgentRunExternalTools(api, ctx)
|
||||
assert "langbot_read_input_attachment" not in {
|
||||
tool["name"] for tool in tools.mcp_tools()
|
||||
@@ -239,14 +233,57 @@ async def test_external_input_attachment_tool_reads_by_ref():
|
||||
assert "langbot_read_input_attachment" in {
|
||||
tool["name"] for tool in tools.mcp_tools()
|
||||
}
|
||||
assert await tools.call_tool("langbot_read_input_attachment", {"ref": "att-1"}) == {
|
||||
"ref": "att-1",
|
||||
"name": "a.txt",
|
||||
"mime_type": "text/plain",
|
||||
"size": 2,
|
||||
"content_base64": "aGk=",
|
||||
assert (
|
||||
await tools.call_tool("langbot_read_input_attachment", {"ref": "att-1"})
|
||||
== contents
|
||||
)
|
||||
api.read_input_attachment_content.assert_awaited_once_with(
|
||||
ref="att-1",
|
||||
mode="auto",
|
||||
offset=0,
|
||||
max_chars=8192,
|
||||
page_start=1,
|
||||
page_end=None,
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_external_input_content_uses_native_mcp_image_and_pdf_blocks():
|
||||
from langbot_plugin.api.entities.builtin.runner.input import InputAttachment
|
||||
|
||||
ctx = _authorized_ctx()
|
||||
ctx.context.available_apis.input_attachments = True
|
||||
ctx.input.attachments = [InputAttachment(ref="att-1", type="file")]
|
||||
api = MagicMock()
|
||||
api.read_input_attachment_content = AsyncMock(
|
||||
return_value=[
|
||||
{"type": "text", "text": '{"ref":"att-1"}'},
|
||||
{"type": "image_base64", "image_base64": "data:image/png;base64,aW1hZ2U="},
|
||||
{
|
||||
"type": "file_base64",
|
||||
"file_base64": "data:application/pdf;base64,cGRm",
|
||||
"file_name": "report.pdf",
|
||||
},
|
||||
]
|
||||
)
|
||||
result = await AgentRunExternalTools(api, ctx).call_mcp_tool(
|
||||
"langbot_read_input_attachment", {"ref": "att-1"}
|
||||
)
|
||||
assert result == {
|
||||
"content": [
|
||||
{"type": "text", "text": '{"ref":"att-1"}'},
|
||||
{"type": "image", "data": "aW1hZ2U=", "mimeType": "image/png"},
|
||||
{
|
||||
"type": "resource",
|
||||
"resource": {
|
||||
"uri": "langbot-artifact:///report.pdf",
|
||||
"mimeType": "application/pdf",
|
||||
"blob": "cGRm",
|
||||
},
|
||||
},
|
||||
]
|
||||
}
|
||||
api.read_input_attachment.assert_awaited_once_with("att-1")
|
||||
assert all("base64" not in block.get("text", "") for block in result["content"])
|
||||
|
||||
|
||||
def test_external_box_file_reply_requires_reply_permission():
|
||||
|
||||
@@ -4,6 +4,7 @@ import asyncio
|
||||
import logging
|
||||
import time
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import AsyncMock
|
||||
|
||||
import pytest
|
||||
|
||||
@@ -169,6 +170,37 @@ def _handler(debug_plugin=False, action_context=None, *, runtime_profile=None):
|
||||
return handler, manager, control_handler
|
||||
|
||||
|
||||
async def test_input_content_forwarding_uses_trusted_identity_scope_and_cancellation():
|
||||
binding = _action_context()
|
||||
handler, manager, control = _handler(action_context=binding)
|
||||
manager.plugins = [FakePluginContainer(runtime_handler=handler)]
|
||||
contents = [{"type": "text", "text": "bounded content"}]
|
||||
control.call_action = AsyncMock(return_value={"contents": contents})
|
||||
async with ProtocolSession(handler) as session:
|
||||
result = await session.request(
|
||||
PluginToRuntimeAction.READ_INPUT_ATTACHMENT_CONTENT.value,
|
||||
{
|
||||
"run_id": "run-1",
|
||||
"ref": "att-1",
|
||||
"mode": "text",
|
||||
"caller_plugin_identity": "forged/identity",
|
||||
"workspace_uuid": "forged-workspace",
|
||||
},
|
||||
action_context=None,
|
||||
)
|
||||
assert result["code"] == 0 and result["data"] == {"contents": contents}
|
||||
action, data = control.call_action.call_args.args
|
||||
assert action == PluginToRuntimeAction.READ_INPUT_ATTACHMENT_CONTENT
|
||||
assert data == {
|
||||
"run_id": "run-1",
|
||||
"ref": "att-1",
|
||||
"mode": "text",
|
||||
"caller_plugin_identity": "tester/demo",
|
||||
}
|
||||
assert control.call_action.call_args.kwargs["action_context"] == binding
|
||||
assert control.call_action.call_args.kwargs["cancel_peer_on_cancel"] is True
|
||||
|
||||
|
||||
def test_reserved_decode_capacity_is_consumed_by_first_message_only():
|
||||
handler, _manager, _control = _handler()
|
||||
handler._certified_shared_stdio = True
|
||||
|
||||
@@ -349,15 +349,22 @@ controlled request-layer measurements and their scope.
|
||||
|
||||
`create_artifact(filename, content_base64, mime_type, ttl_seconds)` uploads at most
|
||||
10 MiB and returns an opaque ID and metadata. Use `get_artifact(artifact_id)` to
|
||||
inspect it, and `read_artifact(artifact_id, max_bytes)` only when bytes are needed.
|
||||
The read result is base64 for file transfer; it does not extract text, render PDF
|
||||
pages or prove model understanding. Prefer the authenticated HTTP download route
|
||||
for user downloads. See [Artifact API](../../../docs/en/develop/artifacts.mdx).
|
||||
inspect it. Prefer `read_artifact_content(artifact_id, mode, offset, max_chars,
|
||||
page_start, page_end)` for model consumption: it returns bounded text or native
|
||||
MCP image/PDF content. Continue using the returned text/page offsets when a result
|
||||
is truncated. PDF text excludes visual content; `native` needs a supporting model,
|
||||
and `pages` currently reports unsupported. `read_artifact(artifact_id, max_bytes)`
|
||||
returns base64 for raw file transfer. Use authenticated HTTP downloads for users.
|
||||
See [Artifact API](../../../docs/en/develop/artifacts.mdx) for limits and modes.
|
||||
|
||||
Reads require `resource.view`; writes require `resource.manage`. The authenticated
|
||||
account or API key must own the asset in the current Workspace. An ID is not an
|
||||
access grant. Plugin/run references are not accepted by this initial API, and
|
||||
Artifact operations are not added to default model tool sets or plugin grants.
|
||||
Account-owned Artifact operations are not added to default model tool sets or
|
||||
plugin grants. Runner attachment tools use their existing run-scoped grant and
|
||||
Host-only input references; an account upload can be passed to Agent debug with
|
||||
`artifact_ids` after ownership validation. The management assistant does not
|
||||
accept attachments in this phase.
|
||||
|
||||
Temporary files expire after 24 hours by default (`ttl_seconds`: 1 to 604800).
|
||||
`retain_artifact` adds a persistent pin without broadening access.
|
||||
|
||||
@@ -132,7 +132,6 @@ def render_input(
|
||||
chain: typing.Iterable[platform_message.MessageComponent] | None,
|
||||
*,
|
||||
include_quotes: bool,
|
||||
keep_images: bool = True,
|
||||
include_other_text: bool = False,
|
||||
) -> RenderedInput:
|
||||
"""Render a message chain into model contents and ordered media.
|
||||
@@ -140,8 +139,6 @@ def render_input(
|
||||
Args:
|
||||
chain: Platform message chain of the current message.
|
||||
include_quotes: Render quoted messages and expose their media.
|
||||
keep_images: Forward image payloads to the model; otherwise only the
|
||||
marker describes the image.
|
||||
include_other_text: Render non-media components without a dedicated
|
||||
projection (such as mentions) with their string form.
|
||||
"""
|
||||
@@ -160,24 +157,9 @@ def render_input(
|
||||
item = InputMedia(ref=f'att-{len(media) + 1}', component=component, origin=origin)
|
||||
media.append(item)
|
||||
add_text(item.marker(), plain=False)
|
||||
payload = component.base64 or None
|
||||
url = getattr(component, 'url', None) or None
|
||||
if isinstance(component, platform_message.Image):
|
||||
if not keep_images:
|
||||
if origin == ORIGIN_MESSAGE:
|
||||
plain_parts.append('[Image]')
|
||||
elif payload:
|
||||
contents.append(provider_message.ContentElement.from_image_base64(payload))
|
||||
elif url:
|
||||
contents.append(provider_message.ContentElement.from_image_url(url))
|
||||
return
|
||||
if isinstance(component, platform_message.Voice):
|
||||
name = 'voice.silk' if payload else 'voice'
|
||||
else:
|
||||
name = component.name
|
||||
if payload:
|
||||
contents.append(provider_message.ContentElement.from_file_base64(payload, name))
|
||||
elif url:
|
||||
contents.append(provider_message.ContentElement.from_file_url(url, name))
|
||||
|
||||
def render(components: typing.Iterable[platform_message.MessageComponent], origin: str) -> None:
|
||||
for component in components:
|
||||
|
||||
@@ -3,6 +3,7 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from pydantic import Field
|
||||
from typing import Literal
|
||||
|
||||
from ..authz import Permission
|
||||
from ..context import PrincipalType
|
||||
@@ -25,6 +26,14 @@ class ArtifactRead(ArtifactID):
|
||||
max_bytes: int = Field(default=MAX_ARTIFACT_BYTES, ge=1, le=MAX_ARTIFACT_BYTES)
|
||||
|
||||
|
||||
class ArtifactContent(ArtifactID):
|
||||
mode: Literal['auto', 'text', 'native', 'pages'] = 'auto'
|
||||
offset: int = Field(default=0, ge=0)
|
||||
max_chars: int = Field(default=8192, ge=1, le=16384)
|
||||
page_start: int = Field(default=1, ge=1)
|
||||
page_end: int | None = Field(default=None, ge=1)
|
||||
|
||||
|
||||
async def create(ap, context, payload):
|
||||
return await ap.artifact_service.create(context, **payload.model_dump())
|
||||
|
||||
@@ -37,6 +46,10 @@ async def read(ap, context, payload):
|
||||
return await ap.artifact_service.read(context, payload.artifact_id, max_bytes=payload.max_bytes)
|
||||
|
||||
|
||||
async def read_content(ap, context, payload):
|
||||
return await ap.artifact_service.read_content(context, **payload.model_dump())
|
||||
|
||||
|
||||
async def retain(ap, context, payload):
|
||||
return await ap.artifact_service.retain(context, payload.artifact_id)
|
||||
|
||||
@@ -56,6 +69,7 @@ def operations():
|
||||
Operation('artifact.create', manage, create, ArtifactCreate, dict, True, principals=principals),
|
||||
Operation('artifact.get', view, get, ArtifactID, dict, principals=principals),
|
||||
Operation('artifact.read', view, read, ArtifactRead, dict, principals=principals),
|
||||
Operation('artifact.read_content', view, read_content, ArtifactContent, list[dict], principals=principals),
|
||||
Operation('artifact.retain', manage, retain, ArtifactID, dict, True, principals=principals),
|
||||
Operation('artifact.release', manage, release, ArtifactID, dict, True, principals=principals),
|
||||
Operation('artifact.delete', manage, delete, ArtifactID, dict, True, principals=principals),
|
||||
|
||||
@@ -529,6 +529,20 @@ EXECUTION_OPERATIONS = (
|
||||
),
|
||||
'run',
|
||||
),
|
||||
ExecutionOperation(
|
||||
'read_input_attachment_content',
|
||||
_input(
|
||||
'ExecutionReadInputContentInput',
|
||||
ref=(Text, ...),
|
||||
mode=(Literal['auto', 'text', 'native', 'pages'], 'auto'),
|
||||
offset=(Nonnegative, 0),
|
||||
max_chars=(Annotated[int, Field(ge=1, le=16384)], 8192),
|
||||
page_start=(Positive, 1),
|
||||
page_end=(Positive | None, None),
|
||||
),
|
||||
_output('ExecutionInputContentOutput', contents=(list[Object], ...)),
|
||||
'run',
|
||||
),
|
||||
ExecutionOperation(
|
||||
'export_box_files', _input('ExecutionExportBoxInput', paths=(list[str] | None, None)), ItemsOutput, 'run', True
|
||||
),
|
||||
|
||||
@@ -149,6 +149,14 @@ OPERATION_DOCS: dict[str, OperationDoc] = {
|
||||
risk='read',
|
||||
exposure='none',
|
||||
),
|
||||
'artifact.read_content': OperationDoc(
|
||||
domain='artifact',
|
||||
label={'en_US': 'Read Artifact content', 'zh_Hans': '读取资产内容'},
|
||||
summary='Read bounded text or structured media from an owned asset.',
|
||||
description='Returns text slices, images or native PDF input. PDF page rendering is unavailable.',
|
||||
risk='read',
|
||||
exposure='none',
|
||||
),
|
||||
'artifact.retain': OperationDoc(
|
||||
domain='artifact',
|
||||
label={'en_US': 'Retain Artifact', 'zh_Hans': '保留资产'},
|
||||
|
||||
@@ -122,6 +122,7 @@ CONTEXT_ACTIONS = frozenset(
|
||||
PluginAction.BIND_BOX,
|
||||
PluginAction.IMPORT_BOX_ATTACHMENTS,
|
||||
PluginAction.READ_INPUT_ATTACHMENT,
|
||||
PluginAction.READ_INPUT_ATTACHMENT_CONTENT,
|
||||
PluginAction.EXPORT_BOX_FILES,
|
||||
PluginAction.GET_PROMPT,
|
||||
PluginAction.HISTORY_PAGE,
|
||||
|
||||
@@ -23,7 +23,9 @@ class ArtifactsRouterGroup(group.RouterGroup):
|
||||
except ApplicationAPIException as exc:
|
||||
return self.http_status(exc.error.status_code, exc.error.code, exc.error.message)
|
||||
if not download:
|
||||
return self.success(data=result)
|
||||
response = self.success(data=result)
|
||||
response.headers['Cache-Control'] = 'private, no-store'
|
||||
return response
|
||||
metadata = result['metadata']
|
||||
return quart.Response(
|
||||
result['content'],
|
||||
@@ -93,6 +95,22 @@ class ArtifactsRouterGroup(group.RouterGroup):
|
||||
download=True,
|
||||
)
|
||||
|
||||
@self.route(
|
||||
'/<artifact_id>/read',
|
||||
methods=['GET'],
|
||||
auth_type=group.AuthType.USER_TOKEN_OR_API_KEY,
|
||||
permission=Permission.RESOURCE_VIEW,
|
||||
)
|
||||
async def read_content(artifact_id: str, request_context: RequestContext):
|
||||
payload = {'artifact_id': artifact_id, 'mode': quart.request.args.get('mode', 'auto')}
|
||||
for key in ('offset', 'max_chars', 'page_start', 'page_end'):
|
||||
if key in quart.request.args:
|
||||
try:
|
||||
payload[key] = int(quart.request.args[key])
|
||||
except ValueError:
|
||||
return self.http_status(400, 'invalid_input', f'Invalid {key}')
|
||||
return await invoke('artifact.read_content', request_context, payload)
|
||||
|
||||
@self.route(
|
||||
'/<artifact_id>/retain',
|
||||
methods=['POST'],
|
||||
|
||||
@@ -186,6 +186,29 @@ class AgentService:
|
||||
if not runner_id:
|
||||
raise ValueError('Agent has no configured runner')
|
||||
|
||||
artifact_ids = payload.get('artifact_ids', [])
|
||||
if (
|
||||
not isinstance(artifact_ids, list)
|
||||
or len(artifact_ids) > 20
|
||||
or any(not isinstance(value, str) or not value for value in artifact_ids)
|
||||
or len(set(artifact_ids)) != len(artifact_ids)
|
||||
):
|
||||
raise ValueError('artifact_ids must contain at most 20 distinct Artifact IDs')
|
||||
artifacts = []
|
||||
for artifact_id in artifact_ids:
|
||||
# Check the authenticated caller, never the synthetic event actor.
|
||||
metadata = await self.ap.artifact_service.get(context, artifact_id)
|
||||
artifacts.append(
|
||||
{
|
||||
'ref': artifact_id,
|
||||
'type': 'image' if metadata['mime_type'].startswith('image/') else 'file',
|
||||
'source': 'artifact',
|
||||
'name': metadata['name'],
|
||||
'mime_type': metadata['mime_type'],
|
||||
'size': metadata['size'],
|
||||
}
|
||||
)
|
||||
|
||||
conversation_id = str(payload.get('conversation_id') or f'debug:{agent_uuid}').strip()
|
||||
if not conversation_id or len(conversation_id) > 256:
|
||||
raise ValueError('Invalid debug conversation_id')
|
||||
@@ -238,8 +261,12 @@ class AgentService:
|
||||
'text': text or event_type,
|
||||
'contents': [
|
||||
{'type': 'text', 'text': text or event_type},
|
||||
*(
|
||||
{'type': 'text', 'text': f'[attachment {item["ref"]}: {item["type"]} {item["name"]}]'}
|
||||
for item in artifacts
|
||||
),
|
||||
],
|
||||
'attachments': [],
|
||||
'attachments': artifacts,
|
||||
}
|
||||
),
|
||||
delivery=DeliveryContext(
|
||||
@@ -324,6 +351,9 @@ class AgentService:
|
||||
descriptor = await self.ap.runner_registry.get(execution_context, runner_id)
|
||||
prompt_config = extract_prompt_config(descriptor, runner_config, [])
|
||||
execution_query = build_execution_query(event, [], execution_context)
|
||||
if artifact_ids:
|
||||
object.__setattr__(execution_query, '_input_artifact_context', context)
|
||||
object.__setattr__(execution_query, '_input_artifact_ids', frozenset(artifact_ids))
|
||||
execution_query.prompt = provider_prompt.Prompt(
|
||||
name='default',
|
||||
messages=[provider_message.Message(**message) for message in copy.deepcopy(prompt_config)],
|
||||
|
||||
@@ -22,6 +22,7 @@ import base64
|
||||
import binascii
|
||||
import typing
|
||||
import functools
|
||||
from mcp import types as mcp_types
|
||||
|
||||
from mcp.server.fastmcp import FastMCP
|
||||
from mcp.server.fastmcp.exceptions import ToolError
|
||||
@@ -151,6 +152,53 @@ class LangBotMCPServer:
|
||||
}
|
||||
)
|
||||
|
||||
@mcp.tool(
|
||||
description='Read bounded text, a structured image or native PDF from an owned Artifact. PDF auto/text excludes visual content; native needs model file support. Page rendering is unavailable.'
|
||||
)
|
||||
async def read_artifact_content(
|
||||
artifact_id: str,
|
||||
mode: str = 'auto',
|
||||
offset: int = 0,
|
||||
max_chars: int = 8192,
|
||||
page_start: int = 1,
|
||||
page_end: int | None = None,
|
||||
) -> mcp_types.CallToolResult:
|
||||
context = _authorized(Permission.RESOURCE_VIEW)
|
||||
elements = await artifact_operation(
|
||||
'artifact.read_content',
|
||||
context,
|
||||
{
|
||||
'artifact_id': artifact_id,
|
||||
'mode': mode,
|
||||
'offset': offset,
|
||||
'max_chars': max_chars,
|
||||
'page_start': page_start,
|
||||
'page_end': page_end,
|
||||
},
|
||||
)
|
||||
blocks = []
|
||||
for element in elements:
|
||||
if element['type'] == 'text':
|
||||
blocks.append(mcp_types.TextContent(type='text', text=element['text']))
|
||||
else:
|
||||
data_url = element.get('image_base64') or element['file_base64']
|
||||
header, blob = data_url.split(';base64,', 1)
|
||||
mime = header[5:]
|
||||
if element['type'] == 'image_base64':
|
||||
blocks.append(mcp_types.ImageContent(type='image', data=blob, mimeType=mime))
|
||||
else:
|
||||
blocks.append(
|
||||
mcp_types.EmbeddedResource(
|
||||
type='resource',
|
||||
resource=mcp_types.BlobResourceContents(
|
||||
uri=f'artifact://{artifact_id}',
|
||||
blob=blob,
|
||||
mimeType=mime,
|
||||
),
|
||||
)
|
||||
)
|
||||
return mcp_types.CallToolResult(content=blocks)
|
||||
|
||||
@mcp.tool(
|
||||
description='Add a persistent pin to a caller-owned Artifact without widening its access permissions.'
|
||||
)
|
||||
|
||||
@@ -40,11 +40,9 @@ def prepare_input_files(query, agent_input):
|
||||
event envelope is metadata, never permission to read the Host filesystem.
|
||||
"""
|
||||
from langbot_plugin.api.entities.builtin.runner.input import InputAttachment
|
||||
from langbot_plugin.api.entities.builtin.provider.message import ContentElement
|
||||
from ..agent.runner import input_media
|
||||
|
||||
# Pipeline queries pin the media they rendered; only bare event envelopes
|
||||
# may describe media in contents alone.
|
||||
envelope_only = input_media.remembered_media(query) is None
|
||||
media = input_media.media_by_ref(query)
|
||||
components = {ref: item.component for ref, item in media.items()}
|
||||
# Envelope attachments registered by an earlier preparation of this query.
|
||||
@@ -52,13 +50,29 @@ def prepare_input_files(query, agent_input):
|
||||
components.update(previous if isinstance(previous, dict) else {})
|
||||
if not agent_input.attachments:
|
||||
agent_input.attachments = [InputAttachment.model_validate(item.to_attachment()) for item in media.values()]
|
||||
if not agent_input.attachments and envelope_only:
|
||||
agent_input.attachments = _attachments_from_contents(agent_input.contents)
|
||||
|
||||
def source_key(item):
|
||||
return item.type, item.url, item.content, item.name
|
||||
|
||||
by_source = {source_key(item): item for item in agent_input.attachments}
|
||||
content_attachments = {}
|
||||
for index, content in enumerate(agent_input.contents):
|
||||
described = _attachments_from_contents([content])
|
||||
if not described:
|
||||
continue
|
||||
candidate = described[0]
|
||||
key = source_key(candidate)
|
||||
attachment = by_source.get(key)
|
||||
if attachment is None:
|
||||
attachment = by_source[key] = candidate
|
||||
agent_input.attachments.append(attachment)
|
||||
content_attachments[index] = attachment
|
||||
|
||||
for attachment in agent_input.attachments:
|
||||
attachment.path = None
|
||||
if attachment.ref and attachment.ref in components:
|
||||
attachment.content = None
|
||||
attachment.url = None
|
||||
continue
|
||||
kind = (attachment.type or 'file').lower()
|
||||
cls = {'image': pm.Image, 'voice': pm.Voice}.get(kind, pm.File)
|
||||
@@ -76,9 +90,22 @@ def prepare_input_files(query, agent_input):
|
||||
index += 1
|
||||
ref = f'att-{index}'
|
||||
components[ref] = cls(**fields)
|
||||
attachment.ref, attachment.content = ref, None
|
||||
attachment.ref, attachment.content, attachment.url = ref, None, None
|
||||
object.__setattr__(query, '_input_components', components)
|
||||
|
||||
normalized = []
|
||||
for index, content in enumerate(agent_input.contents):
|
||||
if content.type not in {'image_url', 'image_base64', 'file_url', 'file_base64'}:
|
||||
normalized.append(content)
|
||||
continue
|
||||
attachment = content_attachments.get(index)
|
||||
if attachment is not None:
|
||||
label = f'{attachment.type} {attachment.name}' if attachment.name else attachment.type
|
||||
normalized.append(ContentElement.from_text(f'[attachment {attachment.ref}: {label}]'))
|
||||
else:
|
||||
normalized.append(ContentElement.from_text('[attachment unavailable]'))
|
||||
agent_input.contents = normalized
|
||||
|
||||
|
||||
def _attachments_from_contents(contents):
|
||||
"""Describe media that an event envelope carried only in its contents."""
|
||||
|
||||
@@ -128,5 +128,12 @@
|
||||
"Artifact is still in use": "Artifact is still in use",
|
||||
"Artifact exceeds the read size limit": "Artifact exceeds the read size limit",
|
||||
"Artifact content failed integrity verification": "Artifact content failed integrity verification",
|
||||
"Artifact exceeds the size limit.": "Artifact exceeds the size limit."
|
||||
"Artifact exceeds the size limit.": "Artifact exceeds the size limit.",
|
||||
"This file type cannot be read by the model. Download the original file instead.": "This file type cannot be read by the model. Download the original file instead.",
|
||||
"PDF page rendering is unavailable. Use text extraction or a model with native PDF support.": "PDF page rendering is unavailable. Use text extraction or a model with native PDF support.",
|
||||
"The PDF could not be read. It may be malformed or password-protected.": "The PDF could not be read. It may be malformed or password-protected.",
|
||||
"PDF text extraction excludes images and page layout. Use native mode with a supported model for visual content.": "PDF text extraction excludes images and page layout. Use native mode with a supported model for visual content.",
|
||||
"PDF text extraction exceeds the size limit.": "PDF text extraction exceeds the size limit.",
|
||||
"The selected model does not support native PDF input; read the PDF as text instead.": "The selected model does not support native PDF input; read the PDF as text instead.",
|
||||
"Input attachment content budget for this run is exhausted": "Input attachment content budget for this run is exhausted"
|
||||
}
|
||||
@@ -128,5 +128,12 @@
|
||||
"Artifact is still in use": "El artefacto sigue en uso",
|
||||
"Artifact exceeds the read size limit": "El artefacto supera el límite de tamaño de lectura",
|
||||
"Artifact content failed integrity verification": "Falló la verificación de integridad del contenido del artefacto",
|
||||
"Artifact exceeds the size limit.": "El artefacto supera el límite de tamaño."
|
||||
"Artifact exceeds the size limit.": "El artefacto supera el límite de tamaño.",
|
||||
"This file type cannot be read by the model. Download the original file instead.": "El modelo no puede leer este tipo de archivo. Descargue el original.",
|
||||
"PDF page rendering is unavailable. Use text extraction or a model with native PDF support.": "La representación de páginas PDF no está disponible. Extraiga texto o use un modelo compatible con PDF nativo.",
|
||||
"The PDF could not be read. It may be malformed or password-protected.": "No se pudo leer el PDF. Puede estar dañado o protegido por contraseña.",
|
||||
"PDF text extraction excludes images and page layout. Use native mode with a supported model for visual content.": "La extracción de texto PDF no incluye imágenes ni diseño. Use el modo nativo con un modelo compatible para el contenido visual.",
|
||||
"PDF text extraction exceeds the size limit.": "La extracción de texto PDF supera el límite de tamaño.",
|
||||
"The selected model does not support native PDF input; read the PDF as text instead.": "El modelo seleccionado no admite PDF nativo; lea el PDF como texto.",
|
||||
"Input attachment content budget for this run is exhausted": "Se agotó el límite de lectura de contenido adjunto de esta ejecución"
|
||||
}
|
||||
@@ -128,5 +128,12 @@
|
||||
"Artifact is still in use": "アーティファクトは使用中です",
|
||||
"Artifact exceeds the read size limit": "アーティファクトが読み取りサイズの上限を超えています",
|
||||
"Artifact content failed integrity verification": "アーティファクトの内容の整合性検証に失敗しました",
|
||||
"Artifact exceeds the size limit.": "アーティファクトがサイズの上限を超えています。"
|
||||
"Artifact exceeds the size limit.": "アーティファクトがサイズの上限を超えています。",
|
||||
"This file type cannot be read by the model. Download the original file instead.": "モデルはこの形式を読み取れません。元のファイルをダウンロードしてください。",
|
||||
"PDF page rendering is unavailable. Use text extraction or a model with native PDF support.": "PDFページ描画は利用できません。テキスト抽出またはPDF対応モデルを使用してください。",
|
||||
"The PDF could not be read. It may be malformed or password-protected.": "PDFを読み取れません。破損またはパスワード保護の可能性があります。",
|
||||
"PDF text extraction excludes images and page layout. Use native mode with a supported model for visual content.": "PDFテキスト抽出には画像やレイアウトは含まれません。視覚情報には対応モデルのnativeモードを使用してください。",
|
||||
"PDF text extraction exceeds the size limit.": "PDFテキスト抽出がサイズ制限を超えました。",
|
||||
"The selected model does not support native PDF input; read the PDF as text instead.": "選択したモデルはPDF入力に対応していません。PDFをテキストとして読み取ってください。",
|
||||
"Input attachment content budget for this run is exhausted": "この実行の添付内容読み取り上限に達しました"
|
||||
}
|
||||
@@ -128,5 +128,12 @@
|
||||
"Artifact is still in use": "Артефакт всё ещё используется",
|
||||
"Artifact exceeds the read size limit": "Артефакт превышает ограничение размера чтения",
|
||||
"Artifact content failed integrity verification": "Проверка целостности содержимого артефакта не пройдена",
|
||||
"Artifact exceeds the size limit.": "Артефакт превышает ограничение размера."
|
||||
"Artifact exceeds the size limit.": "Артефакт превышает ограничение размера.",
|
||||
"This file type cannot be read by the model. Download the original file instead.": "Модель не может прочитать этот тип файла. Скачайте оригинал.",
|
||||
"PDF page rendering is unavailable. Use text extraction or a model with native PDF support.": "Отрисовка страниц PDF недоступна. Используйте извлечение текста или модель с поддержкой PDF.",
|
||||
"The PDF could not be read. It may be malformed or password-protected.": "Не удалось прочитать PDF. Возможно, файл повреждён или защищён паролем.",
|
||||
"PDF text extraction excludes images and page layout. Use native mode with a supported model for visual content.": "Извлечение текста PDF не включает изображения и макет. Для визуального содержимого используйте native с совместимой моделью.",
|
||||
"PDF text extraction exceeds the size limit.": "Извлечение текста PDF превышает ограничение размера.",
|
||||
"The selected model does not support native PDF input; read the PDF as text instead.": "Выбранная модель не поддерживает PDF; прочитайте PDF как текст.",
|
||||
"Input attachment content budget for this run is exhausted": "Лимит чтения содержимого вложений этого запуска исчерпан"
|
||||
}
|
||||
@@ -128,5 +128,12 @@
|
||||
"Artifact is still in use": "อาร์ติแฟกต์ยังถูกใช้งานอยู่",
|
||||
"Artifact exceeds the read size limit": "อาร์ติแฟกต์มีขนาดเกินขีดจำกัดการอ่าน",
|
||||
"Artifact content failed integrity verification": "การตรวจสอบความสมบูรณ์ของเนื้อหาอาร์ติแฟกต์ล้มเหลว",
|
||||
"Artifact exceeds the size limit.": "อาร์ติแฟกต์มีขนาดเกินขีดจำกัด"
|
||||
"Artifact exceeds the size limit.": "อาร์ติแฟกต์มีขนาดเกินขีดจำกัด",
|
||||
"This file type cannot be read by the model. Download the original file instead.": "โมเดลอ่านไฟล์ชนิดนี้ไม่ได้ โปรดดาวน์โหลดไฟล์ต้นฉบับ",
|
||||
"PDF page rendering is unavailable. Use text extraction or a model with native PDF support.": "ยังไม่รองรับการเรนเดอร์หน้า PDF โปรดแยกข้อความหรือใช้โมเดลที่รองรับ PDF โดยตรง",
|
||||
"The PDF could not be read. It may be malformed or password-protected.": "อ่าน PDF ไม่ได้ ไฟล์อาจเสียหายหรือมีรหัสผ่าน",
|
||||
"PDF text extraction excludes images and page layout. Use native mode with a supported model for visual content.": "การแยกข้อความ PDF ไม่รวมรูปภาพและเค้าโครง โปรดใช้โหมด native กับโมเดลที่รองรับสำหรับเนื้อหาภาพ",
|
||||
"PDF text extraction exceeds the size limit.": "การแยกข้อความ PDF เกินขีดจำกัดขนาด",
|
||||
"The selected model does not support native PDF input; read the PDF as text instead.": "โมเดลที่เลือกไม่รองรับ PDF โดยตรง โปรดอ่าน PDF เป็นข้อความ",
|
||||
"Input attachment content budget for this run is exhausted": "งบการอ่านเนื้อหาไฟล์แนบของการทำงานนี้หมดแล้ว"
|
||||
}
|
||||
@@ -128,5 +128,12 @@
|
||||
"Artifact is still in use": "Tài nguyên vẫn đang được sử dụng",
|
||||
"Artifact exceeds the read size limit": "Tài nguyên vượt quá giới hạn kích thước đọc",
|
||||
"Artifact content failed integrity verification": "Xác minh tính toàn vẹn nội dung tài nguyên không thành công",
|
||||
"Artifact exceeds the size limit.": "Tài nguyên vượt quá giới hạn kích thước."
|
||||
"Artifact exceeds the size limit.": "Tài nguyên vượt quá giới hạn kích thước.",
|
||||
"This file type cannot be read by the model. Download the original file instead.": "Mô hình không đọc được loại tệp này. Hãy tải tệp gốc xuống.",
|
||||
"PDF page rendering is unavailable. Use text extraction or a model with native PDF support.": "Chưa hỗ trợ kết xuất trang PDF. Hãy trích xuất văn bản hoặc dùng mô hình hỗ trợ PDF gốc.",
|
||||
"The PDF could not be read. It may be malformed or password-protected.": "Không đọc được PDF. Tệp có thể bị hỏng hoặc được bảo vệ bằng mật khẩu.",
|
||||
"PDF text extraction excludes images and page layout. Use native mode with a supported model for visual content.": "Trích xuất văn bản PDF không bao gồm hình ảnh và bố cục. Hãy dùng chế độ native với mô hình phù hợp cho nội dung trực quan.",
|
||||
"PDF text extraction exceeds the size limit.": "Trích xuất văn bản PDF vượt giới hạn kích thước.",
|
||||
"The selected model does not support native PDF input; read the PDF as text instead.": "Mô hình đã chọn không hỗ trợ PDF gốc; hãy đọc PDF dưới dạng văn bản.",
|
||||
"Input attachment content budget for this run is exhausted": "Đã hết hạn mức đọc nội dung đính kèm của lần chạy này"
|
||||
}
|
||||
@@ -128,5 +128,12 @@
|
||||
"Artifact is still in use": "资产仍在使用中",
|
||||
"Artifact exceeds the read size limit": "资产超过读取大小限制",
|
||||
"Artifact content failed integrity verification": "资产内容完整性校验失败",
|
||||
"Artifact exceeds the size limit.": "资产超过大小限制。"
|
||||
"Artifact exceeds the size limit.": "资产超过大小限制。",
|
||||
"This file type cannot be read by the model. Download the original file instead.": "模型无法读取此文件类型,请下载原文件。",
|
||||
"PDF page rendering is unavailable. Use text extraction or a model with native PDF support.": "暂不支持 PDF 页面渲染,请提取文本或使用支持原生 PDF 输入的模型。",
|
||||
"The PDF could not be read. It may be malformed or password-protected.": "无法读取 PDF,文件可能损坏或受密码保护。",
|
||||
"PDF text extraction excludes images and page layout. Use native mode with a supported model for visual content.": "PDF 文本提取不包含图片和页面布局。需要视觉内容时,请使用受支持模型的原生模式。",
|
||||
"PDF text extraction exceeds the size limit.": "PDF 文本提取超过大小限制。",
|
||||
"The selected model does not support native PDF input; read the PDF as text instead.": "所选模型不支持原生 PDF 输入,请改为读取 PDF 文本。",
|
||||
"Input attachment content budget for this run is exhausted": "本次运行的附件内容读取预算已用尽"
|
||||
}
|
||||
@@ -128,5 +128,12 @@
|
||||
"Artifact is still in use": "資產仍在使用中",
|
||||
"Artifact exceeds the read size limit": "資產超過讀取大小限制",
|
||||
"Artifact content failed integrity verification": "資產內容完整性驗證失敗",
|
||||
"Artifact exceeds the size limit.": "資產超過大小限制。"
|
||||
"Artifact exceeds the size limit.": "資產超過大小限制。",
|
||||
"This file type cannot be read by the model. Download the original file instead.": "模型無法讀取此檔案類型,請下載原始檔案。",
|
||||
"PDF page rendering is unavailable. Use text extraction or a model with native PDF support.": "暫不支援 PDF 頁面渲染,請擷取文字或使用支援原生 PDF 輸入的模型。",
|
||||
"The PDF could not be read. It may be malformed or password-protected.": "無法讀取 PDF,檔案可能損壞或受密碼保護。",
|
||||
"PDF text extraction excludes images and page layout. Use native mode with a supported model for visual content.": "PDF 文字擷取不包含圖片和頁面配置。需要視覺內容時,請使用受支援模型的原生模式。",
|
||||
"PDF text extraction exceeds the size limit.": "PDF 文字擷取超過大小限制。",
|
||||
"The selected model does not support native PDF input; read the PDF as text instead.": "所選模型不支援原生 PDF 輸入,請改為讀取 PDF 文字。",
|
||||
"Input attachment content budget for this run is exhausted": "本次執行的附件內容讀取額度已用盡"
|
||||
}
|
||||
@@ -346,7 +346,6 @@ class PreProcessor(stage.PipelineStage):
|
||||
rendered = input_media.render_input(
|
||||
query.message_chain,
|
||||
include_quotes=bool(quote_msg),
|
||||
keep_images=keep_image_inputs,
|
||||
)
|
||||
# Runner attachments and Box imports address exactly the media rendered here.
|
||||
input_media.remember_media(query, rendered.media)
|
||||
|
||||
@@ -1,35 +1,33 @@
|
||||
"""Run-scoped access to the bytes of current-event input attachments."""
|
||||
"""Run-authorized input reads, backed by Artifact storage on first model read."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import base64
|
||||
from dataclasses import replace
|
||||
import mimetypes
|
||||
from typing import Any
|
||||
|
||||
from langbot_plugin.entities.io.actions.enums import PluginToRuntimeAction as Action
|
||||
from langbot_plugin.runtime.io.handler import ActionResponse
|
||||
|
||||
from ..agent.runner import input_media
|
||||
from ..api.context import PrincipalContext, PrincipalType
|
||||
from ..box.runner import input_files
|
||||
from ..i18n import t
|
||||
from ..storage.artifact_reader import read_artifact_content
|
||||
from .agent_run_support import _validate_agent_run_session
|
||||
|
||||
|
||||
def register(h):
|
||||
@h.action(Action.READ_INPUT_ATTACHMENT)
|
||||
async def read_input_attachment(data: dict[str, Any]) -> ActionResponse:
|
||||
"""Resolve one attachment of the run's own input by ref.
|
||||
|
||||
Bytes are resolved on demand and never cached or logged; each read is
|
||||
bounded per attachment and cumulatively per run.
|
||||
"""
|
||||
async def handle(data, *, render):
|
||||
from .handler import _resolve_action_query
|
||||
|
||||
run_id = data.get('run_id')
|
||||
ref = data.get('ref')
|
||||
run_id, ref = data.get('run_id'), data.get('ref')
|
||||
if not run_id:
|
||||
return ActionResponse.error('run_id is required')
|
||||
if not isinstance(ref, str) or not ref:
|
||||
return ActionResponse.error('ref is required')
|
||||
|
||||
action_context, _ = await h._require_plugin_action_context()
|
||||
session, error = await _validate_agent_run_session(
|
||||
run_id,
|
||||
@@ -43,33 +41,97 @@ def register(h):
|
||||
query = _resolve_action_query(data, session, h.ap, action_context)
|
||||
if query is None:
|
||||
return ActionResponse.error('The current execution context has expired')
|
||||
|
||||
component = input_files(query).get(ref)
|
||||
if component is None:
|
||||
return ActionResponse.error(f'Unknown input attachment reference: {ref}')
|
||||
|
||||
already_read = getattr(query, '_input_attachment_read_bytes', 0)
|
||||
remaining = input_media.MAX_RUN_READ_BYTES - already_read
|
||||
if remaining <= 0:
|
||||
return ActionResponse.error('Input attachment read budget for this run is exhausted')
|
||||
resolved = await input_media.read_component_bytes(
|
||||
component,
|
||||
max_bytes=min(input_media.MAX_ATTACHMENT_BYTES, remaining),
|
||||
adapter=getattr(query, 'adapter', None),
|
||||
)
|
||||
if resolved is None:
|
||||
return ActionResponse.error(
|
||||
f'Input attachment {ref} cannot be read: no payload is available or it exceeds the size limit'
|
||||
lock = getattr(query, '_input_attachment_lock', None)
|
||||
if lock is None:
|
||||
lock = asyncio.Lock()
|
||||
object.__setattr__(query, '_input_attachment_lock', lock)
|
||||
async with lock:
|
||||
already_read = getattr(query, '_input_attachment_read_bytes', 0)
|
||||
remaining = input_media.MAX_RUN_READ_BYTES - already_read
|
||||
materialized = getattr(query, '_input_materialized_refs', set())
|
||||
charge_source = not render or ref not in materialized
|
||||
if charge_source and remaining <= 0:
|
||||
return ActionResponse.error('Input attachment read budget for this run is exhausted')
|
||||
returned_bytes = getattr(query, '_input_content_read_bytes', 0)
|
||||
if render and returned_bytes >= input_media.MAX_RUN_READ_BYTES:
|
||||
return ActionResponse.error(t('Input attachment content budget for this run is exhausted'))
|
||||
limit = (
|
||||
min(input_media.MAX_ATTACHMENT_BYTES, remaining) if charge_source else input_media.MAX_ATTACHMENT_BYTES
|
||||
)
|
||||
payload, mime_type = resolved
|
||||
object.__setattr__(query, '_input_attachment_read_bytes', already_read + len(payload))
|
||||
h.ap.logger.debug(f'Read input attachment {ref} for run {run_id}: {len(payload)} bytes')
|
||||
return ActionResponse.success(
|
||||
{
|
||||
'ref': ref,
|
||||
'name': getattr(component, 'name', None) or None,
|
||||
'mime_type': mime_type,
|
||||
'size': len(payload),
|
||||
'content': base64.b64encode(payload).decode('ascii'),
|
||||
}
|
||||
)
|
||||
uploaded_ids = getattr(query, '_input_artifact_ids', frozenset())
|
||||
cached = getattr(query, '_input_artifact_refs', {})
|
||||
run_context = getattr(query, '_execution_context', None)
|
||||
if run_context is not None:
|
||||
run_context = replace(
|
||||
run_context, trigger_principal=PrincipalContext(PrincipalType.PROCESSOR, run_id=run_id)
|
||||
)
|
||||
if ref in uploaded_ids:
|
||||
original_context = query._input_artifact_context
|
||||
result = await h.ap.artifact_service.read(original_context, ref, max_bytes=limit)
|
||||
payload, metadata = result['content'], result['metadata']
|
||||
elif ref in cached:
|
||||
result = await h.ap.artifact_service.read_run_input(run_context, cached[ref], max_bytes=limit)
|
||||
payload, metadata = result['content'], result['metadata']
|
||||
else:
|
||||
component = input_files(query).get(ref)
|
||||
if component is None:
|
||||
return ActionResponse.error(f'Unknown input attachment reference: {ref}')
|
||||
resolved = await input_media.read_component_bytes(
|
||||
component,
|
||||
max_bytes=limit,
|
||||
adapter=getattr(query, 'adapter', None),
|
||||
)
|
||||
if resolved is None:
|
||||
return ActionResponse.error(
|
||||
f'Input attachment {ref} cannot be read: no payload is available or it exceeds the size limit'
|
||||
)
|
||||
payload, mime_type = resolved
|
||||
name = getattr(component, 'name', None) or None
|
||||
metadata = {'ref': ref, 'name': name, 'mime_type': mime_type, 'size': len(payload)}
|
||||
if render:
|
||||
name = (name or ref).replace('\\', '/').rsplit('/', 1)[-1]
|
||||
mime_type = mime_type.split(';', 1)[0].strip()
|
||||
if mime_type == 'application/octet-stream':
|
||||
mime_type = mimetypes.guess_type(name)[0] or mime_type
|
||||
metadata = await h.ap.artifact_service.create_run_input(run_context, name, payload, mime_type)
|
||||
cached[ref] = metadata['id']
|
||||
object.__setattr__(query, '_input_artifact_refs', cached)
|
||||
if charge_source:
|
||||
object.__setattr__(query, '_input_attachment_read_bytes', already_read + len(payload))
|
||||
if render:
|
||||
materialized.add(ref)
|
||||
object.__setattr__(query, '_input_materialized_refs', materialized)
|
||||
contents = await read_artifact_content(
|
||||
payload,
|
||||
{**metadata, 'ref': ref},
|
||||
**{
|
||||
key: data[key]
|
||||
for key in ('mode', 'offset', 'max_chars', 'page_start', 'page_end')
|
||||
if key in data
|
||||
},
|
||||
)
|
||||
# Text continuation pays for returned content, not the entire cached source again.
|
||||
output_size = sum(len(item['text'].encode('utf-8')) for item in contents if item['type'] == 'text')
|
||||
if any(item['type'] != 'text' for item in contents):
|
||||
output_size += len(payload)
|
||||
if output_size > input_media.MAX_RUN_READ_BYTES - returned_bytes:
|
||||
return ActionResponse.error(t('Input attachment content budget for this run is exhausted'))
|
||||
object.__setattr__(query, '_input_content_read_bytes', returned_bytes + output_size)
|
||||
return ActionResponse.success({'contents': contents})
|
||||
return ActionResponse.success(
|
||||
{
|
||||
'ref': ref,
|
||||
'name': metadata['name'],
|
||||
'mime_type': metadata['mime_type'],
|
||||
'size': len(payload),
|
||||
'content': base64.b64encode(payload).decode('ascii'),
|
||||
}
|
||||
)
|
||||
|
||||
@h.action(Action.READ_INPUT_ATTACHMENT)
|
||||
async def read_input_attachment(data: dict[str, Any]) -> ActionResponse:
|
||||
return await handle(data, render=False)
|
||||
|
||||
@h.action(Action.READ_INPUT_ATTACHMENT_CONTENT)
|
||||
async def read_input_attachment_content(data: dict[str, Any]) -> ActionResponse:
|
||||
return await handle(data, render=True)
|
||||
@@ -364,6 +364,10 @@ class LiteLLMRequester(requester.ProviderAPIRequester):
|
||||
|
||||
def _uncached_litellm_bool_helper(self, helper_name, model_name):
|
||||
helper = getattr(litellm, helper_name, None)
|
||||
if helper_name == 'supports_pdf_input' and not callable(helper):
|
||||
from litellm.utils import supports_pdf_input
|
||||
|
||||
helper = supports_pdf_input
|
||||
if not callable(helper):
|
||||
return False
|
||||
|
||||
@@ -880,6 +884,8 @@ class LiteLLMRequester(requester.ProviderAPIRequester):
|
||||
messages: typing.List[provider_message.Message],
|
||||
reasoning_family: str = '',
|
||||
include_reasoning_context: bool = True,
|
||||
model_name: str | None = None,
|
||||
pdf_input_declared: bool = False,
|
||||
) -> list[dict]:
|
||||
"""Convert LangBot messages to LiteLLM/OpenAI format."""
|
||||
req_messages = []
|
||||
@@ -955,13 +961,30 @@ class LiteLLMRequester(requester.ProviderAPIRequester):
|
||||
part.pop('image_base64', None)
|
||||
else:
|
||||
continue
|
||||
# OpenAI-compatible chat models reject non-image file parts
|
||||
# (audio/document base64 or url). These originate from Voice /
|
||||
# File attachments — including ones replayed from conversation
|
||||
# history — and the agent already accesses their bytes via the
|
||||
# sandbox. Drop them from the model payload to avoid
|
||||
# "Invalid user message ... invalid content type=file_base64".
|
||||
# Explicit PDF input must succeed or fail visibly; keep other legacy file handling.
|
||||
if isinstance(part, dict) and part.get('type') in ('file_base64', 'file_url'):
|
||||
file_data = part.get('file_base64') or part.get('file_url')
|
||||
if isinstance(file_data, str) and file_data.startswith('data:application/pdf;base64,'):
|
||||
supported = pdf_input_declared or (
|
||||
model_name is not None
|
||||
and self._safe_litellm_bool_helper('supports_pdf_input', model_name)
|
||||
)
|
||||
if not supported:
|
||||
raise errors.RequesterError(
|
||||
t(
|
||||
'The selected model does not support native PDF input; read the PDF as text instead.'
|
||||
),
|
||||
code='unsupported_pdf_input',
|
||||
)
|
||||
converted_parts.append(
|
||||
{
|
||||
'type': 'file',
|
||||
'file': {
|
||||
'file_data': file_data,
|
||||
'filename': part.get('file_name') or 'document.pdf',
|
||||
},
|
||||
}
|
||||
)
|
||||
continue
|
||||
converted_parts.append(part)
|
||||
msg_dict['content'] = converted_parts
|
||||
@@ -1500,13 +1523,25 @@ class LiteLLMRequester(requester.ProviderAPIRequester):
|
||||
model = scoped_model
|
||||
reasoning_family = self._reasoning_family(model.model_entity.name, model)
|
||||
reasoning_level = self._reasoning_level(model)
|
||||
message_key = ('messages', id(messages), reasoning_family, reasoning_level != 'disabled')
|
||||
pdf_input_declared = 'pdf_input' in (model.model_entity.abilities or [])
|
||||
message_key = (
|
||||
'messages',
|
||||
id(messages),
|
||||
reasoning_family,
|
||||
reasoning_level != 'disabled',
|
||||
model.model_entity.name,
|
||||
pdf_input_declared,
|
||||
)
|
||||
if cache_preparation and message_key in state.prepared:
|
||||
req_messages = copy.deepcopy(state.prepared[message_key])
|
||||
state.metrics['preparation_cache_hits'] = state.metrics.get('preparation_cache_hits', 0) + 1
|
||||
else:
|
||||
req_messages = self._convert_messages(
|
||||
messages, reasoning_family=reasoning_family, include_reasoning_context=reasoning_level != 'disabled'
|
||||
messages,
|
||||
reasoning_family=reasoning_family,
|
||||
include_reasoning_context=reasoning_level != 'disabled',
|
||||
model_name=model.model_entity.name,
|
||||
pdf_input_declared=pdf_input_declared,
|
||||
)
|
||||
if cache_preparation:
|
||||
state.prepared[message_key] = copy.deepcopy(req_messages)
|
||||
@@ -1610,6 +1645,8 @@ class LiteLLMRequester(requester.ProviderAPIRequester):
|
||||
messages,
|
||||
reasoning_family=self._reasoning_family(model.model_entity.name, model),
|
||||
include_reasoning_context=reasoning_config['level'] != 'disabled',
|
||||
model_name=model.model_entity.name,
|
||||
pdf_input_declared='pdf_input' in (model.model_entity.abilities or []),
|
||||
)
|
||||
chat_provider = 'openrouter' if self._uses_openrouter_protocol() else self._get_custom_llm_provider()
|
||||
self._adapt_messages_for_provider(req_messages, chat_provider)
|
||||
|
||||
@@ -4,6 +4,7 @@ from ....i18n import t
|
||||
|
||||
import contextvars
|
||||
import dataclasses
|
||||
import base64
|
||||
import enum
|
||||
import json
|
||||
import math
|
||||
@@ -11,7 +12,7 @@ import re
|
||||
import time
|
||||
import typing
|
||||
import ipaddress
|
||||
from urllib.parse import urlparse
|
||||
from urllib.parse import unquote, urlparse
|
||||
from contextlib import AsyncExitStack, asynccontextmanager
|
||||
from datetime import timedelta
|
||||
import traceback
|
||||
@@ -60,6 +61,7 @@ MCP_RESOURCE_CACHE_MAX_ENTRIES = 32
|
||||
MCP_RESOURCE_PREVIEW_MAX_BYTES = 64 * 1024
|
||||
MCP_RESOURCE_AGENT_READ_MAX_BYTES = 64 * 1024
|
||||
MCP_RESOURCE_AGENT_READ_MAX_TOKENS = 12000
|
||||
MCP_TOOL_PDF_MAX_BYTES = 10 * 1024 * 1024
|
||||
MCP_RESOURCE_CONTEXT_MAX_TOKENS = 8000
|
||||
MCP_RESOURCE_CONTEXT_MAX_BYTES = 96 * 1024
|
||||
MCP_RESOURCE_TRACE_QUERY_KEY = '_mcp_resource_reads'
|
||||
@@ -1219,6 +1221,22 @@ class RuntimeMCPSession:
|
||||
}
|
||||
return [provider_message.ContentElement.from_text(f'{json.dumps(header, ensure_ascii=False)}\n{text}')]
|
||||
if isinstance(resource, mcp_types.BlobResourceContents):
|
||||
if (resource.mimeType or '').split(';', 1)[0].strip().lower() == 'application/pdf':
|
||||
if len(resource.blob) > 4 * ((MCP_TOOL_PDF_MAX_BYTES + 2) // 3):
|
||||
raise ValueError(t('Artifact exceeds the read size limit'))
|
||||
try:
|
||||
value = base64.b64decode(resource.blob, validate=True)
|
||||
except ValueError as exc:
|
||||
raise ValueError('MCP tool returned invalid PDF base64') from exc
|
||||
if len(value) > MCP_TOOL_PDF_MAX_BYTES:
|
||||
raise ValueError(t('Artifact exceeds the read size limit'))
|
||||
filename = unquote(urlparse(str(resource.uri)).path).rsplit('/', 1)[-1] or 'document.pdf'
|
||||
return [
|
||||
provider_message.ContentElement.from_file_base64(
|
||||
'data:application/pdf;base64,' + resource.blob,
|
||||
filename,
|
||||
)
|
||||
]
|
||||
return [
|
||||
provider_message.ContentElement.from_text(
|
||||
json.dumps(
|
||||
|
||||
@@ -0,0 +1,198 @@
|
||||
"""Bounded model-facing reads of already-authorized Artifact bytes."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import base64
|
||||
import io
|
||||
import json
|
||||
import re
|
||||
|
||||
from ..i18n import t
|
||||
from .artifacts import ArtifactError, MAX_ARTIFACT_BYTES
|
||||
|
||||
|
||||
MAX_TEXT_CHARS = 16384
|
||||
MAX_PDF_PAGES = 10
|
||||
MAX_PDF_EXTRACTED_CHARS = 1024 * 1024
|
||||
MAX_PDF_STREAM_BYTES = 2 * 1024 * 1024
|
||||
|
||||
|
||||
def _unsupported(message: str):
|
||||
return ArtifactError('artifact_read_unsupported', t(message), 415)
|
||||
|
||||
|
||||
def _text_result(metadata: dict, mode: str, **result) -> list[dict]:
|
||||
public = {
|
||||
key: metadata[key] for key in ('id', 'ref', 'name', 'mime_type', 'size', 'digest', 'source') if key in metadata
|
||||
}
|
||||
return [{'type': 'text', 'text': json.dumps({'artifact': public, 'mode': mode, **result}, ensure_ascii=False)}]
|
||||
|
||||
|
||||
def _slice(text: str, offset: int, max_chars: int) -> dict:
|
||||
if offset > len(text):
|
||||
raise ValueError('Artifact offset exceeds the text length')
|
||||
end = min(len(text), offset + max_chars)
|
||||
return {
|
||||
'offset': offset,
|
||||
'end_offset': end,
|
||||
'total_chars': len(text),
|
||||
'truncated': end < len(text),
|
||||
'next_offset': end if end < len(text) else None,
|
||||
'text': text[offset:end],
|
||||
}
|
||||
|
||||
|
||||
def _extract_pdf(content: bytes, page_start: int, page_end: int | None) -> tuple[str, dict]:
|
||||
from pypdf import PdfReader, apply_configuration
|
||||
from pypdf.errors import PdfReadError
|
||||
|
||||
# pypdf's ContextVar scopes decoding and parsing budgets to this read, including nested forms.
|
||||
with apply_configuration(
|
||||
maximum_declared_stream_length=MAX_PDF_STREAM_BYTES,
|
||||
array_based_stream_maximum_output_length=MAX_PDF_STREAM_BYTES,
|
||||
array_maximum_element_count=10000,
|
||||
dictionary_maximum_element_count=10000,
|
||||
zlib_maximum_output_length=MAX_PDF_STREAM_BYTES,
|
||||
lzw_maximum_output_length=MAX_PDF_STREAM_BYTES,
|
||||
run_length_maximum_output_length=MAX_PDF_STREAM_BYTES,
|
||||
brotli_maximum_output_length=MAX_PDF_STREAM_BYTES,
|
||||
jbig2_maximum_output_length=MAX_PDF_STREAM_BYTES,
|
||||
image_maximum_buffer_size=MAX_PDF_STREAM_BYTES,
|
||||
stream_decoding_work_maximum_length=4 * MAX_PDF_STREAM_BYTES,
|
||||
stream_parsing_work_maximum_count=100000,
|
||||
stream_filters_maximum_length=8,
|
||||
flate_maximum_columns=10000,
|
||||
flate_maximum_row_length=65536,
|
||||
page_tree_maximum_entries=1000,
|
||||
page_tree_maximum_depth=32,
|
||||
xform_maximum_invocations_per_extraction=100,
|
||||
jbig2dec_binary=None,
|
||||
disable_legacy_handling=True,
|
||||
):
|
||||
reader = PdfReader(io.BytesIO(content), strict=True)
|
||||
if reader.is_encrypted:
|
||||
raise PdfReadError('Encrypted PDF')
|
||||
page_count = len(reader.pages)
|
||||
if page_start > page_count:
|
||||
raise ArtifactError('invalid_input', 'PDF page_start exceeds the page count', 400)
|
||||
last_page = min(page_start + MAX_PDF_PAGES - 1, page_count) if page_end is None else page_end
|
||||
if last_page > page_count:
|
||||
raise ArtifactError('invalid_input', 'PDF page_end exceeds the page count', 400)
|
||||
parts = []
|
||||
chars = 0
|
||||
for index in range(page_start - 1, last_page):
|
||||
page = reader.pages[index]
|
||||
text = page.extract_text() or ''
|
||||
chars += len(text)
|
||||
if chars > MAX_PDF_EXTRACTED_CHARS:
|
||||
raise ArtifactError('artifact_too_large', t('PDF text extraction exceeds the size limit.'), 413)
|
||||
parts.append(f'[Page {index + 1}]\n{text}')
|
||||
return '\n\n'.join(parts), {
|
||||
'page_start': page_start,
|
||||
'page_end': last_page,
|
||||
'page_count': page_count,
|
||||
'next_page': last_page + 1 if last_page < page_count else None,
|
||||
'text_only': True,
|
||||
'notice': t(
|
||||
'PDF text extraction excludes images and page layout. Use native mode with a supported model for visual content.'
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def _pdf_text(content: bytes, page_start: int, page_end: int | None) -> tuple[str, dict]:
|
||||
from pypdf.errors import LimitReachedError, PdfReadError
|
||||
|
||||
try:
|
||||
return _extract_pdf(content, page_start, page_end)
|
||||
except (LimitReachedError, MemoryError) as exc:
|
||||
raise ArtifactError('artifact_too_large', t('PDF text extraction exceeds the size limit.'), 413) from exc
|
||||
except (PdfReadError, NotImplementedError, KeyError, TypeError, ValueError, IndexError, RecursionError) as exc:
|
||||
raise ArtifactError(
|
||||
'artifact_pdf_unreadable',
|
||||
t('The PDF could not be read. It may be malformed or password-protected.'),
|
||||
422,
|
||||
) from exc
|
||||
|
||||
|
||||
def _image_mime(content: bytes) -> str | None:
|
||||
if content.startswith(b'\x89PNG\r\n\x1a\n'):
|
||||
return 'image/png'
|
||||
if content.startswith(b'\xff\xd8\xff'):
|
||||
return 'image/jpeg'
|
||||
if content.startswith((b'GIF87a', b'GIF89a')):
|
||||
return 'image/gif'
|
||||
if content.startswith(b'RIFF') and content[8:12] == b'WEBP':
|
||||
return 'image/webp'
|
||||
return None
|
||||
|
||||
|
||||
async def read_artifact_content(
|
||||
content: bytes,
|
||||
metadata: dict,
|
||||
*,
|
||||
mode: str = 'auto',
|
||||
offset: int = 0,
|
||||
max_chars: int = 8192,
|
||||
page_start: int = 1,
|
||||
page_end: int | None = None,
|
||||
) -> list[dict]:
|
||||
"""Read text ranges or return structured image/native-PDF input after authorization."""
|
||||
if not isinstance(content, bytes) or len(content) > MAX_ARTIFACT_BYTES:
|
||||
raise ValueError('Artifact content must be bytes within the 10 MiB limit')
|
||||
if not isinstance(mode, str) or mode not in {'auto', 'text', 'native', 'pages'}:
|
||||
raise ValueError('Artifact mode must be auto, text, native or pages')
|
||||
if type(offset) is not int or offset < 0:
|
||||
raise ValueError('Artifact offset must be a non-negative character index')
|
||||
if type(max_chars) is not int or not 1 <= max_chars <= MAX_TEXT_CHARS:
|
||||
raise ValueError('Artifact max_chars must be between 1 and 16384')
|
||||
if type(page_start) is not int or page_start < 1:
|
||||
raise ValueError('PDF page_start must be a positive page number')
|
||||
if page_end is not None and (
|
||||
type(page_end) is not int or page_end < page_start or page_end - page_start >= MAX_PDF_PAGES
|
||||
):
|
||||
raise ValueError('PDF page range must contain between 1 and 10 pages')
|
||||
mime_type = metadata.get('mime_type') or 'application/octet-stream'
|
||||
is_pdf = content.lstrip(b'\r\n\t ').startswith(b'%PDF-')
|
||||
if mode == 'pages':
|
||||
raise _unsupported('PDF page rendering is unavailable. Use text extraction or a model with native PDF support.')
|
||||
image_mime = _image_mime(content)
|
||||
if image_mime and mode in {'auto', 'native'}:
|
||||
if offset or page_start != 1 or page_end is not None:
|
||||
raise ValueError('Image reads do not accept text offsets or PDF page ranges')
|
||||
return _text_result(metadata, 'image') + [
|
||||
{
|
||||
'type': 'image_base64',
|
||||
'image_base64': f'data:{image_mime};base64,' + base64.b64encode(content).decode('ascii'),
|
||||
}
|
||||
]
|
||||
if is_pdf or mime_type == 'application/pdf':
|
||||
if not is_pdf:
|
||||
raise ArtifactError(
|
||||
'artifact_pdf_unreadable',
|
||||
t('The PDF could not be read. It may be malformed or password-protected.'),
|
||||
422,
|
||||
)
|
||||
if mode == 'native':
|
||||
if offset or page_start != 1 or page_end is not None:
|
||||
raise ValueError('Native PDF mode reads the complete document, without text offsets or page ranges')
|
||||
return _text_result(metadata, 'native') + [
|
||||
{
|
||||
'type': 'file_base64',
|
||||
'file_name': metadata.get('name') or 'document.pdf',
|
||||
'file_base64': 'data:application/pdf;base64,' + base64.b64encode(content).decode('ascii'),
|
||||
}
|
||||
]
|
||||
text, pages = await asyncio.to_thread(_pdf_text, content, page_start, page_end)
|
||||
return _text_result(metadata, 'text', **pages, **_slice(text, offset, max_chars))
|
||||
if mode == 'native' or image_mime:
|
||||
raise _unsupported('This file type cannot be read by the model. Download the original file instead.')
|
||||
if page_start != 1 or page_end is not None:
|
||||
raise ValueError('PDF page ranges cannot be used with a text file')
|
||||
try:
|
||||
text = content.decode('utf-8')
|
||||
except UnicodeDecodeError as exc:
|
||||
raise _unsupported('This file type cannot be read by the model. Download the original file instead.') from exc
|
||||
if re.search(r'[\x00-\x08\x0b\x0c\x0e-\x1f\x7f]', text):
|
||||
raise _unsupported('This file type cannot be read by the model. Download the original file instead.')
|
||||
return _text_result(metadata, 'text', **_slice(text, offset, max_chars))
|
||||
@@ -88,7 +88,7 @@ class ArtifactService:
|
||||
'mime_type': row['mime_type'],
|
||||
'size': row['size'],
|
||||
'digest': row['digest'],
|
||||
'source': 'upload',
|
||||
'source': 'input_attachment' if row['owner_type'] == 'run' else 'upload',
|
||||
'lifecycle': 'persistent' if row['expires_at'] is None else 'temporary',
|
||||
'created_at': timestamp(row['created_at']),
|
||||
'expires_at': timestamp(row['expires_at']),
|
||||
@@ -103,6 +103,26 @@ class ArtifactService:
|
||||
ttl_seconds: int = DEFAULT_TTL_SECONDS,
|
||||
) -> dict:
|
||||
owner = await self._authorize(context, write=True)
|
||||
return await self._create_owned(context, owner, filename, content, mime_type, ttl_seconds)
|
||||
|
||||
@staticmethod
|
||||
def _run_owner(context):
|
||||
principal = context.trigger_principal if isinstance(context, ExecutionContext) else None
|
||||
if principal is None or principal.principal_type != PrincipalType.PROCESSOR or not principal.run_id:
|
||||
raise PermissionError('Input assets require a trusted active run context')
|
||||
return 'run', principal.run_id
|
||||
|
||||
async def create_run_input(self, context, filename, content, mime_type):
|
||||
owner = self._run_owner(context)
|
||||
await self.content._validate(context)
|
||||
return await self._create_owned(context, owner, filename, content, mime_type, DEFAULT_TTL_SECONDS)
|
||||
|
||||
async def read_run_input(self, context, artifact_id, *, max_bytes=MAX_ARTIFACT_BYTES):
|
||||
owner = self._run_owner(context)
|
||||
await self.content._validate(context)
|
||||
return await self._read_owned(context, owner, artifact_id, max_bytes)
|
||||
|
||||
async def _create_owned(self, context, owner, filename, content, mime_type, ttl_seconds):
|
||||
if (
|
||||
not isinstance(filename, str)
|
||||
or not filename
|
||||
@@ -151,9 +171,12 @@ class ArtifactService:
|
||||
return self._public(await self._row(uow, context, artifact_id, owner))
|
||||
|
||||
async def read(self, context, artifact_id: str, max_bytes: int = MAX_ARTIFACT_BYTES) -> dict:
|
||||
owner = await self._authorize(context)
|
||||
return await self._read_owned(context, owner, artifact_id, max_bytes)
|
||||
|
||||
async def _read_owned(self, context, owner, artifact_id, max_bytes):
|
||||
from .content import ContentIntegrityError
|
||||
|
||||
owner = await self._authorize(context)
|
||||
if type(max_bytes) is not int or not 1 <= max_bytes <= MAX_ARTIFACT_BYTES:
|
||||
raise ValueError('Artifact read limit must be between 1 and 10485760 bytes')
|
||||
async with self.ap.persistence_mgr.tenant_uow(context.workspace_uuid) as uow:
|
||||
@@ -172,6 +195,12 @@ class ArtifactService:
|
||||
raise ArtifactError('artifact_integrity_failed', 'Artifact content failed integrity verification', 422)
|
||||
return {'metadata': self._public(row), 'content': value}
|
||||
|
||||
async def read_content(self, context, artifact_id, **options):
|
||||
from .artifact_reader import read_artifact_content
|
||||
|
||||
result = await self.read(context, artifact_id)
|
||||
return await read_artifact_content(result['content'], result['metadata'], **options)
|
||||
|
||||
async def _has_references(self, uow, context, artifact_id):
|
||||
return (
|
||||
await uow.execute(
|
||||
|
||||
@@ -343,6 +343,7 @@ async def _capture_sdk_payloads() -> dict[str, dict[str, typing.Any]]:
|
||||
await try_call(runner.history_page())
|
||||
await try_call(runner.history_search('query'))
|
||||
await try_call(runner.read_input_attachment('att-1'))
|
||||
await try_call(runner.read_input_attachment_content('att-1'))
|
||||
await try_call(runner.event_get('evt-1'))
|
||||
await try_call(runner.event_page())
|
||||
await try_call(runner.steering_pull())
|
||||
@@ -394,8 +395,8 @@ async def _capture_sdk_payloads() -> dict[str, dict[str, typing.Any]]:
|
||||
def test_action_audit_and_catalog_completeness(test_host):
|
||||
"""Audit that all business actions map to ExecutionAPI and transport controls stay internal."""
|
||||
operations = test_host.api.operations()
|
||||
assert len(operations) == 80
|
||||
assert len(test_host.api.catalog()) == 80
|
||||
assert len(operations) == 81
|
||||
assert len(test_host.api.catalog()) == 81
|
||||
|
||||
execution_action_names = {op.action for op in operations}
|
||||
handler_actions = set(test_host.handler.actions.keys())
|
||||
|
||||
@@ -0,0 +1,63 @@
|
||||
"""Authorized HTTP and native MCP Artifact content, over real persistence."""
|
||||
# ruff: noqa: F811
|
||||
|
||||
import base64
|
||||
import io
|
||||
import json
|
||||
|
||||
from mcp import types
|
||||
import pytest
|
||||
from quart.datastructures import FileStorage
|
||||
|
||||
from langbot.pkg.api.mcp.context import bind_request_context, reset_request_context
|
||||
from langbot.pkg.api.mcp.server import LangBotMCPServer
|
||||
from tests.integration.storage.test_artifacts import (
|
||||
artifact_replicas, # noqa: F401 -- pytest fixture
|
||||
persistence_case, # noqa: F401 -- fixture dependency
|
||||
replicas, # noqa: F401
|
||||
_data,
|
||||
)
|
||||
|
||||
pytestmark = [pytest.mark.integration, pytest.mark.asyncio]
|
||||
|
||||
|
||||
async def upload(case, name, value, mime):
|
||||
return await _data(
|
||||
await case.a.client.post(
|
||||
'/api/v1/artifacts',
|
||||
headers=case.headers,
|
||||
files={
|
||||
'file': FileStorage(io.BytesIO(value), filename=name, content_type=mime),
|
||||
},
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
async def test_http_text_ranges_and_cross_core_mcp_image(artifact_replicas):
|
||||
case = artifact_replicas
|
||||
text = '甲乙😀丁戊'
|
||||
metadata = await upload(case, 'note.txt', text.encode(), 'text/plain')
|
||||
path = '/api/v1/artifacts/' + metadata['id'] + '/read'
|
||||
response = await case.b.client.get(path + '?offset=1&max_chars=2', headers=case.headers)
|
||||
assert response.headers['Cache-Control'] == 'private, no-store'
|
||||
first = await _data(response)
|
||||
result = json.loads(first[0]['text'])
|
||||
assert result['text'] == '乙😀' and result['next_offset'] == 3 and result['truncated'] is True
|
||||
assert (await case.b.client.get(path)).status_code == 401
|
||||
assert (await case.b.client.get(path + '?offset=-1', headers=case.headers)).status_code == 400
|
||||
assert (await case.b.client.get(path + '?mode=pages', headers=case.headers)).status_code == 415
|
||||
png = base64.b64decode(
|
||||
'iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAwMCAO+jWZkAAAAASUVORK5CYII='
|
||||
)
|
||||
image = await upload(case, 'pixel.png', png, 'image/png')
|
||||
server = LangBotMCPServer(case.b.app)
|
||||
token = bind_request_context(case.context)
|
||||
try:
|
||||
response = await server.mcp.call_tool('read_artifact_content', {'artifact_id': image['id']})
|
||||
finally:
|
||||
reset_request_context(token)
|
||||
blocks = response.content if isinstance(response, types.CallToolResult) else response[0]
|
||||
assert blocks[0].type == 'text'
|
||||
assert blocks[1].type == 'image' and blocks[1].mimeType == 'image/png'
|
||||
assert base64.b64decode(blocks[1].data) == png
|
||||
assert base64.b64encode(png).decode() not in blocks[0].text
|
||||
@@ -0,0 +1,75 @@
|
||||
"""Run input assets cannot become account assets or cross the run boundary."""
|
||||
# ruff: noqa: F811
|
||||
|
||||
from dataclasses import replace
|
||||
import uuid
|
||||
|
||||
import pytest
|
||||
|
||||
from langbot.pkg.api.context import PrincipalContext, PrincipalType
|
||||
from langbot.pkg.storage.artifacts import ArtifactError
|
||||
from tests.integration.storage.test_artifacts import (
|
||||
_expire,
|
||||
artifact_replicas, # noqa: F401 -- pytest fixture
|
||||
persistence_case, # noqa: F401 -- fixture dependency
|
||||
replicas, # noqa: F401 -- fixture dependency
|
||||
)
|
||||
|
||||
|
||||
pytestmark = [pytest.mark.integration, pytest.mark.asyncio]
|
||||
_BODY = b'Only this authorized run can read this input.'
|
||||
|
||||
|
||||
def _run_context(case, run_id=None):
|
||||
return replace(
|
||||
case.a.context,
|
||||
trigger_principal=PrincipalContext(PrincipalType.PROCESSOR, run_id=run_id or str(uuid.uuid4())),
|
||||
)
|
||||
|
||||
|
||||
async def test_run_input_can_be_read_on_another_core_but_not_by_account_or_other_run(artifact_replicas):
|
||||
case = artifact_replicas
|
||||
context = _run_context(case)
|
||||
metadata = await case.a.app.artifact_service.create_run_input(context, 'input.txt', _BODY, 'text/plain')
|
||||
assert metadata['source'] == 'input_attachment' and metadata['lifecycle'] == 'temporary'
|
||||
downloaded = await case.b.app.artifact_service.read_run_input(context, metadata['id'])
|
||||
assert downloaded['content'] == _BODY and downloaded['metadata'] == metadata
|
||||
with pytest.raises(FileNotFoundError):
|
||||
await case.b.app.artifact_service.read_run_input(_run_context(case), metadata['id'])
|
||||
for method, suffix in (('get', ''), ('get', '/content'), ('post', '/retain'), ('delete', '')):
|
||||
response = await getattr(case.a.client, method)(
|
||||
'/api/v1/artifacts/' + metadata['id'] + suffix,
|
||||
headers=case.headers,
|
||||
)
|
||||
assert response.status_code == 404
|
||||
private = await case.a.app.artifact_service.create(case.context, 'private.txt', _BODY, 'text/plain')
|
||||
with pytest.raises(FileNotFoundError):
|
||||
await case.a.app.artifact_service.read_run_input(context, private['id'])
|
||||
|
||||
|
||||
async def test_run_input_expiry_and_collection_use_the_same_lifecycle(artifact_replicas):
|
||||
case = artifact_replicas
|
||||
context = _run_context(case)
|
||||
metadata = await case.a.app.artifact_service.create_run_input(context, 'expired.txt', _BODY, 'text/plain')
|
||||
await _expire(case.a, context, metadata['id'])
|
||||
with pytest.raises(ArtifactError) as failure:
|
||||
await case.b.app.artifact_service.read_run_input(context, metadata['id'])
|
||||
assert failure.value.error_code == 'artifact_expired' and failure.value.status_code == 410
|
||||
system = replace(case.a.context, trigger_principal=PrincipalContext(PrincipalType.SYSTEM))
|
||||
assert await case.b.app.artifact_service.collect_expired(system) == 1
|
||||
with pytest.raises(FileNotFoundError):
|
||||
await case.a.app.artifact_service.read_run_input(context, metadata['id'])
|
||||
|
||||
|
||||
async def test_run_input_internal_methods_require_a_run_execution_context(artifact_replicas):
|
||||
case = artifact_replicas
|
||||
for context in (
|
||||
case.context,
|
||||
case.a.context,
|
||||
replace(case.a.context, trigger_principal=PrincipalContext(PrincipalType.SYSTEM)),
|
||||
replace(case.a.context, trigger_principal=PrincipalContext(PrincipalType.PROCESSOR)),
|
||||
):
|
||||
with pytest.raises(PermissionError):
|
||||
await case.a.app.artifact_service.create_run_input(context, 'input.txt', _BODY, 'text/plain')
|
||||
with pytest.raises(PermissionError):
|
||||
await case.a.app.artifact_service.read_run_input(context, 'art_' + '0' * 32)
|
||||
@@ -0,0 +1,204 @@
|
||||
"""The installed input action uses active sessions and real Artifact persistence."""
|
||||
# ruff: noqa: F811
|
||||
|
||||
import asyncio
|
||||
import base64
|
||||
from dataclasses import replace
|
||||
import json
|
||||
from types import SimpleNamespace
|
||||
import uuid
|
||||
|
||||
import pytest
|
||||
import sqlalchemy as sa
|
||||
from langbot_plugin.api.entities.builtin.platform.message import File, MessageChain
|
||||
from langbot_plugin.api.entities.builtin.runner.resources import AgentResources
|
||||
from langbot_plugin.entities.io.actions.enums import PluginToRuntimeAction as Action
|
||||
|
||||
from langbot.pkg.agent.runner import input_media
|
||||
from langbot.pkg.agent.runner.session_registry import AgentRunSessionRegistry
|
||||
from langbot.pkg.api.context import PrincipalContext, PrincipalType
|
||||
from langbot.pkg.entity.persistence.artifact import Artifact
|
||||
from langbot.pkg.plugin import agent_run_support, input_attachment_actions
|
||||
from tests.integration.storage.test_artifacts import (
|
||||
artifact_replicas, # noqa: F401
|
||||
persistence_case, # noqa: F401
|
||||
replicas, # noqa: F401
|
||||
)
|
||||
|
||||
pytestmark = [pytest.mark.integration, pytest.mark.asyncio]
|
||||
_BODY = 'A猫🙂Z'.encode()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
async def input_actions(artifact_replicas, monkeypatch):
|
||||
case = artifact_replicas
|
||||
registry = AgentRunSessionRegistry()
|
||||
monkeypatch.setattr(agent_run_support, 'get_session_registry', lambda: registry)
|
||||
run_id = str(uuid.uuid4())
|
||||
query = SimpleNamespace(
|
||||
message_chain=MessageChain([File(name='sample.txt', base64=base64.b64encode(_BODY).decode())]),
|
||||
_execution_context=case.a.context,
|
||||
)
|
||||
|
||||
async def register(run):
|
||||
await registry.register(
|
||||
run_id=run,
|
||||
runner_id='plugin:test/runner/default',
|
||||
query_id=None,
|
||||
plugin_identity='test/runner',
|
||||
resources=AgentResources().model_dump(mode='json'),
|
||||
available_apis={'input_attachments': True},
|
||||
execution_query=query,
|
||||
workspace_id=case.context.workspace_uuid,
|
||||
)
|
||||
|
||||
await register(run_id)
|
||||
|
||||
class Handler:
|
||||
ap = case.a.app
|
||||
|
||||
def __init__(self):
|
||||
self.actions = {}
|
||||
|
||||
def action(self, name):
|
||||
def attach(function):
|
||||
self.actions[name] = function
|
||||
return function
|
||||
|
||||
return attach
|
||||
|
||||
async def _require_plugin_action_context(self):
|
||||
return case.a.context, None
|
||||
|
||||
handler = Handler()
|
||||
input_attachment_actions.register(handler)
|
||||
return SimpleNamespace(
|
||||
case=case,
|
||||
query=query,
|
||||
run_id=run_id,
|
||||
registry=registry,
|
||||
register=register,
|
||||
render=handler.actions[Action.READ_INPUT_ATTACHMENT_CONTENT],
|
||||
raw=handler.actions[Action.READ_INPUT_ATTACHMENT],
|
||||
arguments={'run_id': run_id, 'ref': 'att-1', 'caller_plugin_identity': 'test/runner'},
|
||||
)
|
||||
|
||||
|
||||
async def _count(case):
|
||||
async with case.a.app.persistence_mgr.tenant_uow(case.context.workspace_uuid) as uow:
|
||||
return (await uow.execute(sa.select(sa.func.count()).select_from(Artifact))).scalar_one()
|
||||
|
||||
|
||||
async def test_first_content_read_creates_one_run_asset_and_reuses_it(input_actions):
|
||||
x = input_actions
|
||||
response = await x.render({**x.arguments, 'offset': 1, 'max_chars': 2})
|
||||
assert response.code == 0
|
||||
result = json.loads(response.data['contents'][0]['text'])
|
||||
assert result['text'] == '猫🙂' and result['next_offset'] == 3
|
||||
assert result['artifact']['source'] == 'input_attachment'
|
||||
asset_id = x.query._input_artifact_refs['att-1']
|
||||
assert await _count(x.case) == 1
|
||||
raw = await x.raw(x.arguments)
|
||||
assert raw.code == 0 and base64.b64decode(raw.data['content']) == _BODY
|
||||
assert await _count(x.case) == 1
|
||||
assert x.query._input_attachment_read_bytes == 2 * len(_BODY)
|
||||
other_run = str(uuid.uuid4())
|
||||
await x.register(other_run)
|
||||
with pytest.raises(FileNotFoundError):
|
||||
await x.render({**x.arguments, 'run_id': other_run})
|
||||
assert x.query._input_artifact_refs['att-1'] == asset_id
|
||||
assert (await x.case.a.client.get('/api/v1/artifacts/' + asset_id, headers=x.case.headers)).status_code == 404
|
||||
|
||||
|
||||
async def test_uploaded_debug_ids_keep_the_original_owner_context(input_actions):
|
||||
x = input_actions
|
||||
metadata = await x.case.a.app.artifact_service.create(x.case.context, 'uploaded.txt', _BODY, 'text/plain')
|
||||
x.query._input_artifact_ids = frozenset({metadata['id']})
|
||||
x.query._input_artifact_context = x.case.context
|
||||
request = {**x.arguments, 'ref': metadata['id']}
|
||||
response = await x.render(request)
|
||||
result = json.loads(response.data['contents'][0]['text'])
|
||||
assert result['text'] == _BODY.decode() and result['artifact']['id'] == metadata['id']
|
||||
assert await _count(x.case) == 1
|
||||
x.query._input_artifact_context = replace(
|
||||
x.case.context,
|
||||
principal=PrincipalContext(PrincipalType.ACCOUNT, account_uuid=str(uuid.uuid4())),
|
||||
)
|
||||
with pytest.raises(FileNotFoundError):
|
||||
await x.render(request)
|
||||
assert await _count(x.case) == 1
|
||||
|
||||
|
||||
async def test_session_identity_expiry_and_budget_precede_content_access(input_actions):
|
||||
x = input_actions
|
||||
refused = await x.render({**x.arguments, 'caller_plugin_identity': 'other/plugin'})
|
||||
assert refused.code != 0 and await _count(x.case) == 0
|
||||
x.query._input_attachment_read_bytes = input_media.MAX_RUN_READ_BYTES - len(_BODY)
|
||||
first, second = await asyncio.gather(x.raw(x.arguments), x.raw(x.arguments))
|
||||
assert sorted([first.code == 0, second.code == 0]) == [False, True]
|
||||
assert x.query._input_attachment_read_bytes == input_media.MAX_RUN_READ_BYTES
|
||||
assert await _count(x.case) == 0
|
||||
await x.registry.unregister(x.run_id)
|
||||
expired = await x.render(x.arguments)
|
||||
assert expired.code != 0 and 'expired' in expired.message
|
||||
|
||||
|
||||
async def test_cancellation_before_payload_resolution_leaves_no_asset_and_releases_lock(input_actions, monkeypatch):
|
||||
x = input_actions
|
||||
started = asyncio.Event()
|
||||
original = input_media.read_component_bytes
|
||||
|
||||
async def stalled(*args, **kwargs):
|
||||
started.set()
|
||||
await asyncio.Event().wait()
|
||||
|
||||
monkeypatch.setattr(input_media, 'read_component_bytes', stalled)
|
||||
task = asyncio.create_task(x.render(x.arguments))
|
||||
await asyncio.wait_for(started.wait(), 5)
|
||||
task.cancel()
|
||||
with pytest.raises(asyncio.CancelledError):
|
||||
await task
|
||||
assert await _count(x.case) == 0
|
||||
assert getattr(x.query, '_input_attachment_read_bytes', 0) == 0
|
||||
monkeypatch.setattr(input_media, 'read_component_bytes', original)
|
||||
response = await asyncio.wait_for(x.render(x.arguments), 5)
|
||||
assert response.code == 0 and await _count(x.case) == 1
|
||||
|
||||
|
||||
async def test_text_continuation_charges_materialization_once_and_bounds_repeated_output(input_actions, monkeypatch):
|
||||
x = input_actions
|
||||
text = 'BEGIN:' + '0123456789' * 818 + ':END!!'
|
||||
assert len(text.encode()) == 8192
|
||||
monkeypatch.setattr(input_media, 'MAX_RUN_READ_BYTES', 16 * 1024)
|
||||
x.query.message_chain = MessageChain(
|
||||
[
|
||||
File(name='continued.txt', base64=base64.b64encode(text.encode()).decode()),
|
||||
]
|
||||
)
|
||||
|
||||
offset = 0
|
||||
chunks = []
|
||||
for _ in range(10):
|
||||
response = await x.render({**x.arguments, 'offset': offset, 'max_chars': 900})
|
||||
assert response.code == 0, response.message
|
||||
result = json.loads(response.data['contents'][0]['text'])
|
||||
assert result['offset'] == offset
|
||||
chunks.append(result['text'])
|
||||
if result['next_offset'] is None:
|
||||
assert result['truncated'] is False
|
||||
break
|
||||
assert result['truncated'] is True
|
||||
assert result['next_offset'] == offset + len(result['text'])
|
||||
offset = result['next_offset']
|
||||
assert len(chunks) == 10
|
||||
assert ''.join(chunks) == text
|
||||
assert await _count(x.case) == 1
|
||||
|
||||
# Cached source bytes may be reused, but repeated model output remains bounded.
|
||||
for _ in range(32):
|
||||
response = await x.render({**x.arguments, 'offset': 0, 'max_chars': 900})
|
||||
if response.code != 0:
|
||||
assert 'budget' in response.message.lower()
|
||||
break
|
||||
else:
|
||||
pytest.fail('Repeated content output bypassed the run budget')
|
||||
@@ -39,8 +39,8 @@ def test_markers_place_each_media_and_attachments_are_metadata_only():
|
||||
)
|
||||
|
||||
assert rendered.text == 'compare [attachment att-1: image] with [attachment att-2: file report.pdf]'
|
||||
assert [item.type for item in rendered.contents] == ['text', 'text', 'image_base64', 'text', 'text', 'file_base64']
|
||||
assert rendered.plain_text == 'compare with '
|
||||
assert [item.type for item in rendered.contents] == ['text', 'text', 'text', 'text']
|
||||
assert rendered.plain_text == 'compare [Image] with '
|
||||
assert rendered.attachments() == [
|
||||
{
|
||||
'ref': 'att-1',
|
||||
@@ -91,7 +91,6 @@ def test_empty_quote_and_text_only_models_render_without_payloads():
|
||||
platform_message.Image(url=IMAGE_URL, base64=IMAGE_BASE64),
|
||||
),
|
||||
include_quotes=True,
|
||||
keep_images=False,
|
||||
)
|
||||
|
||||
assert _texts(rendered.contents) == [('text', '[attachment att-1: image]')]
|
||||
@@ -112,6 +111,7 @@ def test_event_first_input_hands_payloads_to_the_run_registry_only():
|
||||
|
||||
assert agent_input.text == 'look[attachment att-1: image]'
|
||||
assert agent_input.attachments[0].content == IMAGE_BASE64
|
||||
assert all(item.type == 'text' for item in agent_input.contents)
|
||||
|
||||
# Event runs execute against a text-only chain; the envelope payload is registered by ref.
|
||||
query = SimpleNamespace(message_chain=_chain(platform_message.Plain(text=agent_input.text)))
|
||||
|
||||
@@ -314,7 +314,7 @@ def test_context_builder_renders_message_chain_without_preprocessing():
|
||||
|
||||
input_data = QueryEntryAdapter._build_input(query)
|
||||
|
||||
assert [content.type for content in input_data.contents] == ['text', 'image_base64']
|
||||
assert [content.type for content in input_data.contents] == ['text']
|
||||
assert input_data.text == '[attachment att-1: image]'
|
||||
assert len(input_data.attachments) == 1
|
||||
assert input_data.attachments[0].type == 'image'
|
||||
@@ -802,7 +802,9 @@ async def test_cancel_request_preserves_one_terminal_event(clean_agent_state, re
|
||||
('run.cancelled', {'reason': 'stopped', 'cleanup_confirmed': True}, 'cancelled'),
|
||||
],
|
||||
)
|
||||
async def test_terminal_is_persisted_before_runner_stream_cleanup(clean_agent_state, monkeypatch, result_type, data, status):
|
||||
async def test_terminal_is_persisted_before_runner_stream_cleanup(
|
||||
clean_agent_state, monkeypatch, result_type, data, status
|
||||
):
|
||||
"""Transport cleanup must not hold a validated terminal run open."""
|
||||
cleanup_started = asyncio.Event()
|
||||
release_cleanup = asyncio.Event()
|
||||
@@ -2115,8 +2117,8 @@ class TestQueryEntryAdapterHostCapabilities:
|
||||
assert len(event_logs) >= 1
|
||||
# First event should be the incoming message.received
|
||||
assert event_logs[0]['event_type'] == 'message.received'
|
||||
assert event_logs[0]['input_json']['contents'][1]['image_base64'] is None
|
||||
assert event_logs[0]['input_json']['contents'][1]['content_redacted'] is True
|
||||
assert event_logs[0]['input_json']['contents'][1]['type'] == 'text'
|
||||
assert event_logs[0]['input_json']['contents'][1]['text'] == '[attachment att-2: image]'
|
||||
assert 'aGVsbG8=' not in str(event_logs[0]['input_json'])
|
||||
assert 'Pinned documentation' not in str(event_logs[0]['input_json'])
|
||||
|
||||
@@ -2133,7 +2135,8 @@ class TestQueryEntryAdapterHostCapabilities:
|
||||
assert 'user' in roles
|
||||
assert 'assistant' in roles
|
||||
user_item = next(t for t in transcripts if t['role'] == 'user')
|
||||
assert user_item['content_json']['content'][1]['image_base64'] is None
|
||||
assert user_item['content_json']['content'][1]['type'] == 'text'
|
||||
assert user_item['content_json']['content'][1]['text'] == '[attachment att-2: image]'
|
||||
assert user_item['attachment_refs'][0]['content'] is None
|
||||
assert 'aGVsbG8=' not in str(user_item)
|
||||
assert 'Pinned documentation' not in str(user_item)
|
||||
|
||||
@@ -159,6 +159,75 @@ class TestAgentServiceMetadata:
|
||||
|
||||
|
||||
class TestAgentServiceDebug:
|
||||
async def test_debug_artifacts_are_authorized_without_reading_bytes(self):
|
||||
app = _make_app()
|
||||
artifact_id = 'art_' + 'a' * 32
|
||||
app.artifact_service = SimpleNamespace(
|
||||
get=AsyncMock(
|
||||
return_value={
|
||||
'id': artifact_id,
|
||||
'name': 'report.pdf',
|
||||
'mime_type': 'application/pdf',
|
||||
'size': 50,
|
||||
}
|
||||
),
|
||||
read=AsyncMock(),
|
||||
)
|
||||
context = SimpleNamespace(
|
||||
instance_uuid='instance-test',
|
||||
workspace_uuid=WORKSPACE_UUID,
|
||||
placement_generation=1,
|
||||
principal=SimpleNamespace(account_uuid='account-test'),
|
||||
entitlement_revision=0,
|
||||
)
|
||||
|
||||
async def run_runner(event, binding, adapter_context):
|
||||
query = adapter_context['_query']
|
||||
assert query._input_artifact_context is context
|
||||
assert query._input_artifact_ids == frozenset({artifact_id})
|
||||
assert event.input.attachments[0].ref == artifact_id
|
||||
assert event.input.attachments[0].content is None
|
||||
assert event.input.attachments[0].url is None
|
||||
assert all(item.type == 'text' for item in event.input.contents)
|
||||
yield SimpleNamespace(role='assistant', content='read later', all_content=None)
|
||||
|
||||
app.agent_run_orchestrator = SimpleNamespace(run=Mock(side_effect=run_runner))
|
||||
service = AgentService(app)
|
||||
service.get_agent = AsyncMock(
|
||||
return_value={
|
||||
'uuid': 'agent-1',
|
||||
'kind': AGENT_KIND_AGENT,
|
||||
'supported_event_patterns': ['*'],
|
||||
'config': _agent_row().config,
|
||||
}
|
||||
)
|
||||
await service.debug_agent(
|
||||
context,
|
||||
'agent-1',
|
||||
{
|
||||
'text': 'read report',
|
||||
'artifact_ids': [artifact_id],
|
||||
'actor': {'actor_type': 'user', 'actor_id': 'spoofed-owner'},
|
||||
},
|
||||
)
|
||||
app.artifact_service.get.assert_awaited_once_with(context, artifact_id)
|
||||
app.artifact_service.read.assert_not_awaited()
|
||||
|
||||
@pytest.mark.parametrize('artifact_ids', ['art_bad', ['same', 'same'], [None], ['x'] * 21])
|
||||
async def test_debug_rejects_invalid_artifact_ids(self, artifact_ids):
|
||||
app = _make_app()
|
||||
service = AgentService(app)
|
||||
service.get_agent = AsyncMock(
|
||||
return_value={
|
||||
'uuid': 'agent-1',
|
||||
'kind': AGENT_KIND_AGENT,
|
||||
'supported_event_patterns': ['*'],
|
||||
'config': _agent_row().config,
|
||||
}
|
||||
)
|
||||
with pytest.raises(ValueError, match='artifact_ids'):
|
||||
await service.debug_agent(SimpleNamespace(), 'agent-1', {'artifact_ids': artifact_ids})
|
||||
|
||||
@pytest.mark.parametrize('streaming', [False, True])
|
||||
@pytest.mark.parametrize(
|
||||
'result_type',
|
||||
|
||||
@@ -154,8 +154,11 @@ def test_event_attachments_preserved_without_eager_io_or_host_path_access():
|
||||
assert input_files(q)['att-1'].base64 == 'YQ=='
|
||||
assert not input_files(q)['att-1'].path
|
||||
image = AgentInput(contents=[ContentElement.from_image_url('https://example.invalid/image.png')])
|
||||
prepare_input_files(SimpleNamespace(message_chain=MessageChain([])), image)
|
||||
assert image.attachments[0].url == 'https://example.invalid/image.png'
|
||||
image_query = SimpleNamespace(message_chain=MessageChain([]))
|
||||
prepare_input_files(image_query, image)
|
||||
assert image.attachments[0].url is None
|
||||
assert image.contents[0].text == '[attachment att-1: image]'
|
||||
assert input_files(image_query)['att-1'].url == 'https://example.invalid/image.png'
|
||||
|
||||
|
||||
def test_platform_media_keep_rendered_refs_and_never_carry_bytes():
|
||||
@@ -178,6 +181,8 @@ def test_platform_media_keep_rendered_refs_and_never_carry_bytes():
|
||||
('att-2', 'file', None),
|
||||
]
|
||||
assert input_files(query) == {'att-1': image, 'att-2': report}
|
||||
assert all(item.type == 'text' for item in value.contents)
|
||||
assert all(item.url is None and item.path is None for item in value.attachments)
|
||||
|
||||
|
||||
def test_repeated_prepare_keeps_envelope_registrations():
|
||||
@@ -200,6 +205,31 @@ def test_repeated_prepare_keeps_envelope_registrations():
|
||||
assert all(not item.base64 for item in input_files(other_query).values())
|
||||
|
||||
|
||||
def test_contents_only_files_keep_distinct_sources_behind_markers():
|
||||
from langbot_plugin.api.entities.builtin.runner.input import AgentInput
|
||||
from langbot_plugin.api.entities.builtin.provider.message import ContentElement
|
||||
from langbot.pkg.box.runner import input_files, prepare_input_files
|
||||
|
||||
query = SimpleNamespace(message_chain=MessageChain([]))
|
||||
value = AgentInput(
|
||||
contents=[
|
||||
ContentElement.from_file_url('https://files.example/one', 'report.pdf'),
|
||||
ContentElement.from_file_url('https://files.example/two', 'report.pdf'),
|
||||
]
|
||||
)
|
||||
prepare_input_files(query, value)
|
||||
assert [item.ref for item in value.attachments] == ['att-1', 'att-2']
|
||||
assert [item.text for item in value.contents] == [
|
||||
'[attachment att-1: file report.pdf]',
|
||||
'[attachment att-2: file report.pdf]',
|
||||
]
|
||||
assert [item.url for item in input_files(query).values()] == [
|
||||
'https://files.example/one',
|
||||
'https://files.example/two',
|
||||
]
|
||||
assert all(item.url is None and item.content is None for item in value.attachments)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_image_export_accepts_data_url_and_enforces_total_bytes():
|
||||
api = service()
|
||||
|
||||
@@ -309,15 +309,9 @@ class TestPreProcessorImageSegment:
|
||||
from langbot.pkg.agent.runner.query_entry_adapter import QueryEntryAdapter
|
||||
from langbot_plugin.api.entities.builtin.provider.message import ContentElement
|
||||
|
||||
expected_image = (
|
||||
ContentElement.from_image_base64('data:image/png;base64,abc123')
|
||||
if image_source == 'base64'
|
||||
else ContentElement.from_image_url('https://cdn.example/image.png')
|
||||
)
|
||||
assert result.new_query.user_message.content == [
|
||||
ContentElement.from_text('look at this'),
|
||||
ContentElement.from_text('[attachment att-1: image]'),
|
||||
expected_image,
|
||||
]
|
||||
attachments = QueryEntryAdapter._build_input(result.new_query).attachments
|
||||
assert len(attachments) == 1
|
||||
|
||||
@@ -1,14 +1,14 @@
|
||||
"""Unit tests for LiteLLMRequester._convert_messages.
|
||||
|
||||
Focus: the content-part normalization that (a) converts image_base64 parts to
|
||||
the OpenAI image_url shape and (b) drops non-image file parts (file_base64 /
|
||||
file_url) which OpenAI-compatible chat models reject. The latter is essential
|
||||
for Voice/File attachments — including ones replayed from conversation history —
|
||||
since the agent consumes their bytes via the sandbox, not the model payload.
|
||||
Image parts use OpenAI's image_url shape. Explicit PDF input checks native
|
||||
model support and uses the file shape; unsupported legacy file parts retain
|
||||
their existing handling.
|
||||
"""
|
||||
|
||||
import langbot_plugin.api.entities.builtin.provider.message as provider_message
|
||||
import pytest
|
||||
|
||||
from langbot.pkg.provider.modelmgr import errors
|
||||
from langbot.pkg.provider.modelmgr.requesters.litellmchat import LiteLLMRequester
|
||||
|
||||
|
||||
@@ -17,6 +17,60 @@ def _make_requester() -> LiteLLMRequester:
|
||||
return LiteLLMRequester.__new__(LiteLLMRequester)
|
||||
|
||||
|
||||
def test_explicit_pdf_input_uses_supported_file_payload(monkeypatch):
|
||||
req = _make_requester()
|
||||
probes = []
|
||||
|
||||
def capability(helper, model):
|
||||
probes.append((helper, model))
|
||||
return True
|
||||
|
||||
monkeypatch.setattr(req, '_safe_litellm_bool_helper', capability)
|
||||
pdf = 'data:application/pdf;base64,JVBERi0xLjc='
|
||||
message = provider_message.Message(
|
||||
role='user',
|
||||
content=[
|
||||
provider_message.ContentElement.from_file_base64(pdf, 'report.pdf'),
|
||||
],
|
||||
)
|
||||
result = req._convert_messages([message], model_name='pdf-model')
|
||||
assert result[0]['content'] == [{'type': 'file', 'file': {'file_data': pdf, 'filename': 'report.pdf'}}]
|
||||
assert probes == [('supports_pdf_input', 'pdf-model')]
|
||||
assert message.content[0].type == 'file_base64'
|
||||
|
||||
|
||||
def test_explicit_pdf_input_fails_instead_of_being_discarded(monkeypatch):
|
||||
req = _make_requester()
|
||||
monkeypatch.setattr(req, '_safe_litellm_bool_helper', lambda *_args: False)
|
||||
message = provider_message.Message(
|
||||
role='user',
|
||||
content=[
|
||||
provider_message.ContentElement.from_file_base64('data:application/pdf;base64,JVBERg==', 'report.pdf'),
|
||||
],
|
||||
)
|
||||
with pytest.raises(errors.RequesterError) as failure:
|
||||
req._convert_messages([message], model_name='text-model')
|
||||
assert failure.value.code == 'unsupported_pdf_input'
|
||||
|
||||
|
||||
def test_explicit_pdf_ability_supports_verified_gateway_alias_without_guessing_metadata(monkeypatch):
|
||||
req = _make_requester()
|
||||
|
||||
def unexpected_metadata_probe(*_args):
|
||||
raise AssertionError('An explicit PDF declaration must not depend on alias metadata')
|
||||
|
||||
monkeypatch.setattr(req, '_safe_litellm_bool_helper', unexpected_metadata_probe)
|
||||
pdf = 'data:application/pdf;base64,JVBERi0xLjc='
|
||||
message = provider_message.Message(
|
||||
role='user',
|
||||
content=[
|
||||
provider_message.ContentElement.from_file_base64(pdf, 'report.pdf'),
|
||||
],
|
||||
)
|
||||
converted = req._convert_messages([message], model_name='verified-gateway-alias', pdf_input_declared=True)
|
||||
assert converted[0]['content'] == [{'type': 'file', 'file': {'file_data': pdf, 'filename': 'report.pdf'}}]
|
||||
|
||||
|
||||
def test_convert_messages_drops_file_base64_part():
|
||||
req = _make_requester()
|
||||
msg = provider_message.Message(
|
||||
|
||||
@@ -17,6 +17,7 @@ from langbot.pkg.provider.tools.loaders.mcp import (
|
||||
MCP_RESOURCE_CONTEXT_QUERY_KEY,
|
||||
MCP_RESOURCE_TRACE_QUERY_KEY,
|
||||
MCP_TOOL_CALL_TIMEOUT_DEFAULT_SECONDS,
|
||||
MCP_TOOL_PDF_MAX_BYTES,
|
||||
MCP_TOOL_LIST_RESOURCES,
|
||||
MCP_TOOL_READ_RESOURCE,
|
||||
MCPLoader,
|
||||
@@ -84,6 +85,67 @@ def _connected_session(
|
||||
return session
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_explicit_tool_pdf_resource_remains_structured_for_model_consumption():
|
||||
session = _connected_session()
|
||||
pdf = b'%PDF-1.7 explicit tool result'
|
||||
session.session.call_tool = AsyncMock(
|
||||
return_value=mcp_types.CallToolResult(
|
||||
content=[
|
||||
mcp_types.EmbeddedResource(
|
||||
type='resource',
|
||||
resource=mcp_types.BlobResourceContents(
|
||||
uri='langbot-artifact:///sample%20report.pdf',
|
||||
mimeType='application/pdf',
|
||||
blob=base64.b64encode(pdf).decode(),
|
||||
),
|
||||
),
|
||||
]
|
||||
)
|
||||
)
|
||||
result = await session.invoke_mcp_tool('read_artifact_content', {'artifact_id': 'art_current', 'mode': 'native'})
|
||||
assert len(result) == 1 and result[0].type == 'file_base64'
|
||||
assert result[0].file_name == 'sample report.pdf'
|
||||
assert result[0].file_base64 == 'data:application/pdf;base64,' + base64.b64encode(pdf).decode()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
'blob,error',
|
||||
[
|
||||
('not valid base64', 'invalid PDF base64'),
|
||||
('x' * (4 * ((MCP_TOOL_PDF_MAX_BYTES + 2) // 3) + 1), 'read size limit'),
|
||||
(base64.b64encode(b'x' * (MCP_TOOL_PDF_MAX_BYTES + 1)).decode(), 'read size limit'),
|
||||
],
|
||||
)
|
||||
async def test_explicit_tool_pdf_resource_rejects_invalid_or_oversized_payload(blob, error):
|
||||
session = _connected_session()
|
||||
resource = mcp_types.BlobResourceContents.model_construct(
|
||||
uri='langbot-artifact:///invalid.pdf',
|
||||
mimeType='application/pdf',
|
||||
blob=blob,
|
||||
)
|
||||
block = mcp_types.EmbeddedResource.model_construct(type='resource', resource=resource)
|
||||
with pytest.raises(ValueError, match=error):
|
||||
session._content_to_provider_elements(block, source_tool='read_artifact_content')
|
||||
|
||||
|
||||
def test_non_pdf_tool_resource_keeps_binary_omitted_behavior():
|
||||
session = _connected_session()
|
||||
result = session._content_to_provider_elements(
|
||||
mcp_types.EmbeddedResource(
|
||||
type='resource',
|
||||
resource=mcp_types.BlobResourceContents(
|
||||
uri='langbot-artifact:///archive.zip',
|
||||
mimeType='application/zip',
|
||||
blob='aGk=',
|
||||
),
|
||||
),
|
||||
source_tool='download_archive',
|
||||
)
|
||||
assert len(result) == 1 and '"binary_omitted": true' in result[0].text
|
||||
|
||||
|
||||
def _query(variables: dict | None = None, context: ExecutionContext = TEST_EXECUTION_CONTEXT) -> SimpleNamespace:
|
||||
return SimpleNamespace(
|
||||
instance_uuid=context.instance_uuid,
|
||||
|
||||
@@ -0,0 +1,130 @@
|
||||
"""Model-facing ranges, media inputs and real PDF decoding budgets."""
|
||||
|
||||
import base64
|
||||
import io
|
||||
import json
|
||||
|
||||
import pytest
|
||||
from pypdf import PdfWriter, get_configuration
|
||||
from pypdf.generic import DecodedStreamObject, DictionaryObject, NameObject
|
||||
|
||||
from langbot.pkg.storage.artifact_reader import MAX_PDF_STREAM_BYTES, read_artifact_content
|
||||
from langbot.pkg.storage.artifacts import ArtifactError
|
||||
|
||||
|
||||
pytestmark = pytest.mark.asyncio
|
||||
_META = {'id': 'art_test', 'name': 'reference.txt', 'mime_type': 'text/plain', 'content_ref': 'private-key'}
|
||||
_PNG = base64.b64decode('iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAwMCAO+jWZkAAAAASUVORK5CYII=')
|
||||
|
||||
|
||||
def _pdf(*texts, encrypted=False, compressed_payload=None):
|
||||
writer = PdfWriter()
|
||||
for text in texts:
|
||||
page = writer.add_blank_page(width=612, height=792)
|
||||
font = DictionaryObject(
|
||||
{
|
||||
NameObject('/Type'): NameObject('/Font'),
|
||||
NameObject('/Subtype'): NameObject('/Type1'),
|
||||
NameObject('/BaseFont'): NameObject('/Helvetica'),
|
||||
}
|
||||
)
|
||||
page[NameObject('/Resources')] = DictionaryObject(
|
||||
{
|
||||
NameObject('/Font'): DictionaryObject({NameObject('/F1'): writer._add_object(font)}),
|
||||
}
|
||||
)
|
||||
stream = DecodedStreamObject()
|
||||
stream.set_data(
|
||||
compressed_payload
|
||||
if compressed_payload is not None
|
||||
else (f'BT /F1 12 Tf 72 720 Td ({text}) Tj ET'.encode('ascii'))
|
||||
)
|
||||
page[NameObject('/Contents')] = writer._add_object(
|
||||
stream.flate_encode() if compressed_payload is not None else stream
|
||||
)
|
||||
if encrypted:
|
||||
writer.encrypt('password')
|
||||
output = io.BytesIO()
|
||||
writer.write(output)
|
||||
return output.getvalue()
|
||||
|
||||
|
||||
def _text(elements):
|
||||
assert elements[0]['type'] == 'text'
|
||||
return json.loads(elements[0]['text'])
|
||||
|
||||
|
||||
async def test_text_ranges_use_characters_and_preserve_exact_resume_position():
|
||||
result = _text(await read_artifact_content('a猫🙂z'.encode(), _META, offset=1, max_chars=2))
|
||||
assert result['text'] == '猫🙂'
|
||||
assert (result['offset'], result['end_offset'], result['total_chars']) == (1, 3, 4)
|
||||
assert result['truncated'] is True and result['next_offset'] == 3
|
||||
assert 'content_ref' not in result['artifact']
|
||||
remainder = _text(await read_artifact_content('a猫🙂z'.encode(), _META, offset=3))
|
||||
assert remainder['text'] == 'z' and remainder['next_offset'] is None
|
||||
assert remainder['truncated'] is False
|
||||
|
||||
|
||||
async def test_images_and_native_pdf_remain_structured_instead_of_textual_base64():
|
||||
image = await read_artifact_content(_PNG, {'name': 'pixel.png', 'mime_type': 'image/png'})
|
||||
assert _text(image)['mode'] == 'image'
|
||||
assert len(image) == 2 and image[1]['type'] == 'image_base64'
|
||||
assert base64.b64decode(image[1]['image_base64'].split(',', 1)[1], validate=True) == _PNG
|
||||
assert 'base64' not in image[0]['text']
|
||||
pdf = _pdf('Native file payload')
|
||||
native = await read_artifact_content(pdf, {'name': None}, mode='native')
|
||||
assert _text(native)['mode'] == 'native'
|
||||
assert native[1]['type'] == 'file_base64' and native[1]['file_name'] == 'document.pdf'
|
||||
assert base64.b64decode(native[1]['file_base64'].split(',', 1)[1], validate=True) == pdf
|
||||
|
||||
|
||||
async def test_pdf_selected_pages_and_text_limits_report_continuation_and_visual_boundary():
|
||||
content = _pdf('First page original', 'Second page original', 'Third page original')
|
||||
result = _text(await read_artifact_content(content, {'mime_type': 'application/pdf'}, page_start=2, page_end=2))
|
||||
assert 'Second page original' in result['text']
|
||||
assert 'First page original' not in result['text'] and 'Third page original' not in result['text']
|
||||
assert (result['page_start'], result['page_end'], result['page_count'], result['next_page']) == (2, 2, 3, 3)
|
||||
assert result['text_only'] is True and 'excludes images' in result['notice']
|
||||
limited = _text(await read_artifact_content(content, {}, max_chars=4, page_end=1))
|
||||
assert limited['text'] == '[Pag' and limited['truncated'] is True and limited['next_offset'] == 4
|
||||
|
||||
|
||||
async def test_compressed_pdf_is_rejected_before_unbounded_expansion_and_configuration_is_restored():
|
||||
content = _pdf('', compressed_payload=b' ' * (MAX_PDF_STREAM_BYTES + 1))
|
||||
assert len(content) < 10000
|
||||
original = get_configuration()
|
||||
with pytest.raises(ArtifactError) as failure:
|
||||
await read_artifact_content(content, {})
|
||||
assert failure.value.error_code == 'artifact_too_large' and failure.value.status_code == 413
|
||||
assert get_configuration() is original
|
||||
assert 'ordinary PDF' in _text(await read_artifact_content(_pdf('ordinary PDF'), {}))['text']
|
||||
|
||||
|
||||
async def test_unreadable_pdf_and_unsupported_formats_fail_explicitly():
|
||||
for content in (b'%PDF-broken', _pdf('Protected', encrypted=True)):
|
||||
with pytest.raises(ArtifactError) as failure:
|
||||
await read_artifact_content(content, {'mime_type': 'application/pdf'})
|
||||
assert failure.value.error_code == 'artifact_pdf_unreadable' and failure.value.status_code == 422
|
||||
for content, options in ((b'\x00binary', {}), (_pdf('Text'), {'mode': 'pages'})):
|
||||
with pytest.raises(ArtifactError) as failure:
|
||||
await read_artifact_content(content, {}, **options)
|
||||
assert failure.value.error_code == 'artifact_read_unsupported' and failure.value.status_code == 415
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
'options',
|
||||
[
|
||||
{'mode': []},
|
||||
{'offset': True},
|
||||
{'offset': -1},
|
||||
{'max_chars': True},
|
||||
{'max_chars': 16385},
|
||||
{'page_start': False},
|
||||
{'page_start': 0},
|
||||
{'page_end': True},
|
||||
{'page_end': 11},
|
||||
],
|
||||
)
|
||||
async def test_read_arguments_reject_invalid_ranges_and_boolean_numbers(options):
|
||||
with pytest.raises(ValueError):
|
||||
await read_artifact_content(b'text', _META, **options)
|
||||
@@ -2271,7 +2271,7 @@ requires-dist = [
|
||||
{ name = "pylibseekdb", marker = "(platform_machine == 'arm64' and sys_platform == 'darwin' and extra == 'seekdb') or (sys_platform == 'linux' and extra == 'seekdb')", specifier = "==1.4.0" },
|
||||
{ name = "pymilvus", marker = "extra == 'milvus'", specifier = ">=2.6.4" },
|
||||
{ name = "pynacl", specifier = ">=1.5.0" },
|
||||
{ name = "pypdf", specifier = ">=6.0.0" },
|
||||
{ name = "pypdf", specifier = ">=6.21.2" },
|
||||
{ name = "pyseekdb", marker = "extra == 'seekdb'", specifier = "==1.4.0.post1" },
|
||||
{ name = "python-docx", specifier = ">=1.1.0" },
|
||||
{ name = "python-multipart", specifier = ">=0.0.32" },
|
||||
|
||||
Reference in new issue
Block a user