feat(read_file): let capable models view images, video and PDFs
Nightly Build / build (push) Successful in 6m38s

When the resolved model declares an input modality (vision → images,
video, document → PDFs), read_file now hands a binary media file back to
the model as native input instead of failing on non-UTF-8 bytes.

- ToolResult gains a Media { text, media } variant carrying MediaRef
  { host_path, mime }; the tool message keeps only the text note, the
  bytes travel out of band in a new additive chat_llm_tools.media column
  (mirrors preview_old/new).
- read_file sniffs the resolved host file; a recognized medium becomes a
  Media result (with a neutral note), everything else keeps the textual
  path. Capability gating lives in the message builder, so read_file
  never needs the model caps and degrades cleanly on a text-only model.
- MessageBuilder inlines current-turn tool media as a synthetic user
  message right after the tool-result group (media_turn_start boundary,
  so older turns are never re-billed), reusing media.rs primitives via a
  new inline_paths helper that contains against the caller's workspace
  roots. OpenAI forwards the parts verbatim; the Anthropic client now
  also translates the PDF `file` part into a native `document` block
  (image_url → image was already handled).
- read_file's description is annotated per serving model in
  call_llm_round, listing the formats it can open, so the model knows
  reading one shows it the content.

Tests: media sniff (PDF), PDF file-part build, inline_paths containment
+ capability gating, capability hint, read_file media-vs-text, Anthropic
file→document, and the owner-schema-stands-alone check with the new
column.
This commit is contained in:
2026-07-22 11:01:44 +01:00
parent 624f6b0a95
commit cfaa7bace3
12 changed files with 524 additions and 42 deletions
+3 -1
View File
@@ -750,14 +750,16 @@ pub async fn create_owner_tables(pool: &SqlitePool) -> Result<()> {
result_type TEXT NOT NULL DEFAULT 'string' CHECK(result_type IN ('string', 'json')),
preview_old TEXT, -- file-write diff: content before the write
preview_new TEXT, -- file-write diff: content after the write
media TEXT, -- JSON [{host_path,mime}]: media the tool produced, inlined to the model out of band
created_at TEXT NOT NULL DEFAULT (datetime('now'))
)",
)
.execute(pool)
.await?;
// Diff-preview columns are additive — reach an already-created table in place.
// Diff-preview + tool-media columns are additive — reach an already-created table in place.
ensure_column(pool, "chat_llm_tools", "preview_old", "TEXT").await?;
ensure_column(pool, "chat_llm_tools", "preview_new", "TEXT").await?;
ensure_column(pool, "chat_llm_tools", "media", "TEXT").await?;
sqlx::query(
"CREATE INDEX IF NOT EXISTS idx_stack_session ON chat_sessions_stack(session_id)",