feat(read_file): let capable models view images, video and PDFs
Nightly Build / build (push) Successful in 6m38s

When the resolved model declares an input modality (vision → images,
video, document → PDFs), read_file now hands a binary media file back to
the model as native input instead of failing on non-UTF-8 bytes.

- ToolResult gains a Media { text, media } variant carrying MediaRef
  { host_path, mime }; the tool message keeps only the text note, the
  bytes travel out of band in a new additive chat_llm_tools.media column
  (mirrors preview_old/new).
- read_file sniffs the resolved host file; a recognized medium becomes a
  Media result (with a neutral note), everything else keeps the textual
  path. Capability gating lives in the message builder, so read_file
  never needs the model caps and degrades cleanly on a text-only model.
- MessageBuilder inlines current-turn tool media as a synthetic user
  message right after the tool-result group (media_turn_start boundary,
  so older turns are never re-billed), reusing media.rs primitives via a
  new inline_paths helper that contains against the caller's workspace
  roots. OpenAI forwards the parts verbatim; the Anthropic client now
  also translates the PDF `file` part into a native `document` block
  (image_url → image was already handled).
- read_file's description is annotated per serving model in
  call_llm_round, listing the formats it can open, so the model knows
  reading one shows it the content.

Tests: media sniff (PDF), PDF file-part build, inline_paths containment
+ capability gating, capability hint, read_file media-vs-text, Anthropic
file→document, and the owner-schema-stands-alone check with the new
column.
This commit is contained in:
2026-07-22 11:01:44 +01:00
parent 624f6b0a95
commit cfaa7bace3
12 changed files with 524 additions and 42 deletions
+47 -1
View File
@@ -277,7 +277,7 @@ mod tests {
use core_api::user_fs::UserFs;
use crate::tools::{ExecutionOutcome, Tool, ToolContext};
use crate::tools::{ExecutionOutcome, Tool, ToolContext, ToolResult};
/// A trivial workspace for the memory-routing tests, which never touch disk.
fn test_fs() -> Arc<UserFs> {
@@ -551,4 +551,50 @@ mod tests {
let _ = std::fs::remove_dir_all(&udir);
let _ = std::fs::remove_dir_all(&sdir);
}
/// A physical `read_file` on a binary image hands the file back as
/// `ToolResult::Media` (host path + sniffed MIME) instead of failing on the
/// non-UTF-8 bytes; a UTF-8 file still reads as line-numbered text.
#[tokio::test]
async fn read_file_returns_media_for_binary_image() {
let (shared, sdir) = store("readmedia-shared").await;
let (user, udir) = store("readmedia-user").await;
let root = std::env::temp_dir().join(format!("skald-readmedia-{}", uuid::Uuid::new_v4()));
let home = root.join("homes").join("u1");
std::fs::create_dir_all(&home).unwrap();
let mut png = b"\x89PNG\r\n\x1a\n".to_vec();
png.extend_from_slice(&[0xAA; 64]);
std::fs::write(home.join("pic.png"), &png).unwrap();
std::fs::write(home.join("note.txt"), "hello\nworld").unwrap();
let fs = Arc::new(UserFs::new(
"u1", home.clone(), "skald-u1", PathBuf::from("/root"), vec![], vec![], None,
));
let ctx = ToolContext { session_id: 1, user_id: "u1".into(), pool: Arc::clone(&user), fs };
let read = ReadFile::new(Arc::clone(&shared));
// image → Media, carrying the resolved host path + MIME.
match read.run_with(&ctx, json!({"path": "~/pic.png"})).wait().await {
ExecutionOutcome::Completed(ToolResult::Media { text, media }) => {
assert!(text.contains("binary media") && text.contains("image/png"), "{text}");
assert_eq!(media.len(), 1);
assert_eq!(media[0].mime, "image/png");
assert!(media[0].host_path.ends_with("pic.png"), "{}", media[0].host_path);
}
other => panic!("expected Media, got {other:?}"),
}
// UTF-8 text → ordinary numbered text.
match read.run_with(&ctx, json!({"path": "~/note.txt"})).wait().await {
ExecutionOutcome::Completed(ToolResult::Text(t)) => {
assert!(t.contains("| hello") && t.contains("| world"), "{t}");
}
other => panic!("expected Text, got {other:?}"),
}
let _ = std::fs::remove_dir_all(&root);
let _ = std::fs::remove_dir_all(&udir);
let _ = std::fs::remove_dir_all(&sdir);
}
}