feat(read_file): let capable models view images, video and PDFs
Nightly Build / build (push) Successful in 6m38s
Nightly Build / build (push) Successful in 6m38s
When the resolved model declares an input modality (vision → images,
video, document → PDFs), read_file now hands a binary media file back to
the model as native input instead of failing on non-UTF-8 bytes.
- ToolResult gains a Media { text, media } variant carrying MediaRef
{ host_path, mime }; the tool message keeps only the text note, the
bytes travel out of band in a new additive chat_llm_tools.media column
(mirrors preview_old/new).
- read_file sniffs the resolved host file; a recognized medium becomes a
Media result (with a neutral note), everything else keeps the textual
path. Capability gating lives in the message builder, so read_file
never needs the model caps and degrades cleanly on a text-only model.
- MessageBuilder inlines current-turn tool media as a synthetic user
message right after the tool-result group (media_turn_start boundary,
so older turns are never re-billed), reusing media.rs primitives via a
new inline_paths helper that contains against the caller's workspace
roots. OpenAI forwards the parts verbatim; the Anthropic client now
also translates the PDF `file` part into a native `document` block
(image_url → image was already handled).
- read_file's description is annotated per serving model in
call_llm_round, listing the formats it can open, so the model knows
reading one shows it the content.
Tests: media sniff (PDF), PDF file-part build, inline_paths containment
+ capability gating, capability hint, read_file media-vs-text, Anthropic
file→document, and the owner-schema-stands-alone check with the new
column.
This commit is contained in:
@@ -50,6 +50,14 @@ impl ChatSessionHandler {
|
||||
let kind = result.kind();
|
||||
debug!(session_id = self.session_id, tool = %tool_name, tool_call_id, result_len = wire.len(), "tool done");
|
||||
chat_llm_tools::complete(pool, tool_call_id, &wire, kind).await?;
|
||||
// Media the tool produced (e.g. read_file on an image/PDF) rides
|
||||
// out of band in the `media` column; the message builder inlines it
|
||||
// as a synthetic user message for a capable model on the current turn.
|
||||
let media = result.media();
|
||||
if !media.is_empty() {
|
||||
let media_json = serde_json::to_string(media).unwrap_or_else(|_| "[]".to_string());
|
||||
chat_llm_tools::set_media(pool, tool_call_id, &media_json).await?;
|
||||
}
|
||||
// Persist a file-write's diff snapshot so it re-renders after a reload,
|
||||
// and carry it on the event so an auto-allowed write shows the diff live.
|
||||
let (preview_old, preview_new) = match preview {
|
||||
|
||||
Reference in New Issue
Block a user