feat(chat): view context — tell the assistant what you're looking at
Nightly Build / build (push) Successful in 7m51s

An eye next to the paperclip shares what the user has open with their next
message: the page, the folder being browsed, the file open in the viewer and
any highlighted passage (line numbers where a source view exists), plus which
entity a detail page is about. The bag is client-authored {label, value} pairs
in English — the backend only clamps (chars, never bytes), neutralizes the
harness tag and renders one <system-extra> block per message, deduped
consecutively so it appears exactly when the view changed. On by default,
per-device toggle, hover/tap to preview, a chip on every sent message;
docs/view-context.md for users, an updated harness.md clause for the model.
This commit is contained in:
Daniele
2026-08-23 20:53:30 +01:00
parent 488c702517
commit 505f2e95c1
42 changed files with 2096 additions and 122 deletions
+8 -1
View File
@@ -15,7 +15,7 @@ use crate::activation::ActivationSource;
use crate::ids::{ConversationId, FrameId};
use crate::model::ModelInfo;
use crate::projection::{
MediaSource, Projection, ProjectionHooks, ResultLimit, ToolResultDigest,
MediaSource, MessageExtras, Projection, ProjectionHooks, ResultLimit, ToolResultDigest,
};
use crate::store::HistoryStore;
@@ -157,6 +157,13 @@ impl LinearAssembler {
self
}
/// Text appended to each user/agent message (skipped media paths, the view
/// the message was sent from…). One hook, one block — see [`MessageExtras`].
pub fn with_extras(mut self, src: Arc<dyn MessageExtras>) -> Self {
self.hooks.extras = Some(src);
self
}
/// How an over-long tool result is condensed.
pub fn with_digest(mut self, digest: Arc<dyn ToolResultDigest>) -> Self {
self.hooks.digest = Some(digest);
+70 -19
View File
@@ -9,7 +9,8 @@
//!
//! What the host owns: the **content** — the system prompt layers
//! ([`crate::context::SystemContextSource`]), which media a message may inline
//! ([`MediaSource`]) and how an over-long tool result is condensed
//! ([`MediaSource`]), what extra text rides along with a message
//! ([`MessageExtras`]) and how an over-long tool result is condensed
//! ([`ToolResultDigest`]). Everything is optional: with no hooks at all the
//! projection is a complete, correct OpenAI-shaped conversation.
//!
@@ -134,15 +135,36 @@ pub trait MediaSource: Send + Sync {
async fn call_media(&self, _calls: &[StoredCall]) -> Vec<Arc<dyn MediaBlob>> {
Vec::new()
}
/// Text appended to the message for the media that did NOT make it (a path
/// list, so the agent can still reach them with a tool).
}
/// Text appended to a user/agent message — the harness-generated tail a host
/// wants the model to read alongside what the person typed (skipped attachment
/// paths, the view the message was sent from, …).
///
/// **Its own hook, not a `MediaSource` method**, because it must run for every
/// message, media or none: as a media method it was only ever reachable from
/// inside the "this message has blobs" branch, so a message carrying nothing but
/// non-media extras rendered nothing at all.
///
/// The crate does not wrap or frame what comes back — it appends the string
/// verbatim, leading newlines included. Whatever block structure the host wants
/// (`<system-extra>`…) is the host's, which is also why there is exactly **one**
/// call per message: two hooks would mean two blocks.
#[async_trait]
pub trait MessageExtras: Send + Sync {
/// `msg` is the message being projected; `prev` is the previous `User`/`Agent`
/// message of the projected history (`None` for the first one, and after a
/// compaction or a window cut), which lets a host suppress a repeat.
///
/// `skipped` are **positions in the vector `message_media` just returned**
/// for this message, so the host can map them back to whatever it built
/// them from.
fn skipped_text(&self, _msg: &StoredMessage, _skipped: &[usize]) -> Option<String> {
None
}
/// `skipped` are **positions in the vector [`MediaSource::message_media`]
/// returned** for this message — empty when the message has no media at all,
/// so a host must not read it as "nothing was left out of a media message".
async fn appended_text(
&self,
msg: &StoredMessage,
prev: Option<&StoredMessage>,
skipped: &[usize],
) -> Option<String>;
}
/// How an over-long tool result is condensed. The crate decides *when*
@@ -159,6 +181,7 @@ pub trait ToolResultDigest: Send + Sync {
pub struct ProjectionHooks {
pub activation: Option<Arc<dyn ActivationSource>>,
pub media: Option<Arc<dyn MediaSource>>,
pub extras: Option<Arc<dyn MessageExtras>>,
pub digest: Option<Arc<dyn ToolResultDigest>>,
}
@@ -213,10 +236,18 @@ pub async fn project(
window(&mut history, max);
}
// 4. The conversation.
// 4. The conversation. `prev` trails one message behind so `MessageExtras`
// can compare a message with the last thing the person said — carried as a
// running reference rather than an `rposition` per message (same answer,
// linear) and deliberately not put on `HistoryCtx`, which would drag a
// `&[StoredMessage]` lifetime through the whole type for nothing.
let ctx = HistoryCtx::new(&history, cfg, hooks, input).await?;
let mut prev: Option<&StoredMessage> = None;
for (idx, entry) in history.iter().enumerate() {
ctx.project_message(&mut out, idx, entry).await;
ctx.project_message(&mut out, idx, entry, prev).await;
if matches!(entry.role, Role::User | Role::Agent) {
prev = Some(entry);
}
}
// 5. Dynamic tail — the fresh layers, as ONE trailing system message so a
@@ -313,37 +344,57 @@ impl<'a> HistoryCtx<'a> {
})
}
async fn project_message(&self, out: &mut Vec<Value>, idx: usize, entry: &StoredMessage) {
async fn project_message(
&self,
out: &mut Vec<Value>,
idx: usize,
entry: &StoredMessage,
prev: Option<&StoredMessage>,
) {
match entry.role {
// System messages are BUILT (layers 1-2), never replayed from the
// store; a host that stores them gets them back verbatim.
Role::System => out.push(json!({ "role": "system", "content": entry.content })),
Role::User | Role::Agent => self.push_user(out, idx, entry).await,
Role::User | Role::Agent => self.push_user(out, idx, entry, prev).await,
Role::Assistant => self.push_assistant(out, idx, entry).await,
}
}
/// A user/agent message: text plus, for the current turn, inlined media.
async fn push_user(&self, out: &mut Vec<Value>, idx: usize, entry: &StoredMessage) {
/// A user/agent message: text, the host's appended extras, and — for the
/// current turn — inlined media.
async fn push_user(
&self,
out: &mut Vec<Value>,
idx: usize,
entry: &StoredMessage,
prev: Option<&StoredMessage>,
) {
let mut text = entry.content.clone();
let mut parts: Vec<Value> = Vec::new();
let mut skipped: Vec<usize> = Vec::new();
if let Some(src) = &self.hooks.media {
let blobs = src.message_media(entry).await;
if !blobs.is_empty() {
// Older turns keep the textual path: everything is "skipped".
let (inlined, skipped) = if idx >= self.media_turn_start {
let (inlined, left_out) = if idx >= self.media_turn_start {
media::partition(&blobs, &self.model.capabilities, &self.cfg.media).await
} else {
(Vec::new(), (0..blobs.len()).collect())
};
if let Some(extra) = src.skipped_text(entry, &skipped) {
text.push_str(&extra);
}
skipped = left_out;
parts = inlined;
}
}
// Outside the media branch on purpose: extras are not a media feature,
// and a message with none must still get its block.
if let Some(x) = &self.hooks.extras
&& let Some(extra) = x.appended_text(entry, prev, &skipped).await
{
text.push_str(&extra);
}
push_user_chunk(out, text, parts);
}
+119 -3
View File
@@ -10,7 +10,8 @@ use agent_loop::ids::{ConversationId, FrameId, MessageId};
use agent_loop::model::ModelInfo;
use agent_loop::prelude::async_trait;
use agent_loop::projection::{
MediaBlob, MediaSource, Projection, ReasoningEcho, ResultLimit, ToolResultDigest,
MediaBlob, MediaSource, MessageExtras, Projection, ReasoningEcho, ResultLimit,
ToolResultDigest,
};
use agent_loop::store::{
CallOutcome, FrameSpec, HistoryStore, NewCall, NewMessage, NewSummary, StoredCall,
@@ -432,8 +433,34 @@ impl MediaSource for Media {
async fn call_media(&self, _calls: &[StoredCall]) -> Vec<Arc<dyn MediaBlob>> {
vec![Arc::new(Png("tool.png"))]
}
fn skipped_text(&self, _msg: &StoredMessage, skipped: &[usize]) -> Option<String> {
(!skipped.is_empty()).then(|| format!("\n[files: {}]", skipped.len()))
}
/// The appended-text hook, in its own object: a note for the media left out, and
/// — whatever the media — the message's `extra` metadata key, so the tests can
/// tell "there was nothing to inline" from "there was nothing to say".
struct Extras;
#[async_trait]
impl MessageExtras for Extras {
async fn appended_text(
&self,
msg: &StoredMessage,
prev: Option<&StoredMessage>,
skipped: &[usize],
) -> Option<String> {
let mut out = String::new();
if !skipped.is_empty() {
out.push_str(&format!("\n[files: {}]", skipped.len()));
}
let extra = |m: &StoredMessage| {
m.metadata.as_ref().and_then(|v| v["extra"].as_str().map(str::to_string))
};
if let Some(e) = extra(msg)
&& prev.and_then(extra) != Some(e.clone())
{
out.push_str(&format!("\n[extra: {e}]"));
}
(!out.is_empty()).then_some(out)
}
}
@@ -446,6 +473,7 @@ async fn media_is_inlined_for_the_current_turn_and_textual_before_it() {
let msgs = LinearAssembler::new()
.with_media(Arc::new(Media))
.with_extras(Arc::new(Extras))
.build(&store, &input(frame, SystemContext::base("B"), ModelInfo {
capabilities: vec!["vision".into()],
..ModelInfo::default()
@@ -473,6 +501,7 @@ async fn a_model_without_vision_never_receives_bytes() {
let msgs = LinearAssembler::new()
.with_media(Arc::new(Media))
.with_extras(Arc::new(Extras))
.build(&store, &input(frame, SystemContext::base("B"), ModelInfo::default()))
.await
.unwrap();
@@ -493,6 +522,7 @@ async fn tool_produced_media_rides_a_synthetic_user_message_after_the_group() {
let msgs = LinearAssembler::new()
.with_media(Arc::new(Media))
.with_extras(Arc::new(Extras))
.build(&store, &input(frame, SystemContext::base("B"), ModelInfo {
capabilities: vec!["vision".into()],
..ModelInfo::default()
@@ -505,3 +535,89 @@ async fn tool_produced_media_rides_a_synthetic_user_message_after_the_group() {
assert_eq!(last["content"][0]["type"], "image_url");
assert_eq!(msgs[msgs.len() - 2]["role"], "tool", "it follows the result group");
}
// ── Appended extras ──────────────────────────────────────────────────────────
/// The regression this hook exists for: as a `MediaSource` method the appended
/// text was reachable only from inside the "this message has blobs" branch, so a
/// message with something to say and nothing to inline rendered nothing.
#[tokio::test]
async fn extras_reach_a_message_with_no_media_at_all() {
let (store, frame) = store_and_frame("p14").await;
store
.append(frame, NewMessage::user("where am I").with_metadata(json!({ "extra": "files" })))
.await
.unwrap();
// No media hook at all: extras must not depend on one being registered.
let msgs = LinearAssembler::new()
.with_extras(Arc::new(Extras))
.build(&store, &input(frame, SystemContext::base("B"), ModelInfo::default()))
.await
.unwrap();
assert_eq!(msgs[1], json!({ "role": "user", "content": "where am I\n[extra: files]" }));
}
#[tokio::test]
async fn one_appended_chunk_carries_both_halves_media_first() {
let (store, frame) = store_and_frame("p15").await;
store
.append(frame, NewMessage::user("look").with_metadata(json!({ "extra": "files" })))
.await
.unwrap();
// No vision ⇒ the image is skipped, so both halves have something to say.
let msgs = LinearAssembler::new()
.with_media(Arc::new(Media))
.with_extras(Arc::new(Extras))
.build(&store, &input(frame, SystemContext::base("B"), ModelInfo::default()))
.await
.unwrap();
assert_eq!(msgs[1], json!({
"role": "user",
"content": "look\n[files: 1]\n[extra: files]",
}));
}
#[tokio::test]
async fn extras_see_the_previous_user_message_not_the_assistant_turn() {
let (store, frame) = store_and_frame("p16").await;
let meta = |v: &str| json!({ "extra": v });
store.append(frame, NewMessage::user("one").with_metadata(meta("files"))).await.unwrap();
store.append(frame, NewMessage::assistant("ok", None)).await.unwrap();
// Same view as the message before it, across an assistant turn: suppressed.
store.append(frame, NewMessage::user("two").with_metadata(meta("files"))).await.unwrap();
store.append(frame, NewMessage::assistant("ok", None)).await.unwrap();
// Changed view: emitted again.
store.append(frame, NewMessage::user("three").with_metadata(meta("projects"))).await.unwrap();
let msgs = LinearAssembler::new()
.with_extras(Arc::new(Extras))
.build(&store, &input(frame, SystemContext::base("B"), ModelInfo::default()))
.await
.unwrap();
assert_eq!(msgs[1]["content"], "one\n[extra: files]", "prev = None ⇒ emitted");
assert_eq!(msgs[3]["content"], "two", "same as the previous user message ⇒ suppressed");
assert_eq!(msgs[5]["content"], "three\n[extra: projects]", "changed ⇒ emitted");
}
/// The parity contract: with no extras hook the output is what it always was.
#[tokio::test]
async fn no_extras_hook_changes_nothing() {
let (store, frame) = store_and_frame("p17").await;
store
.append(frame, NewMessage::user("look").with_metadata(json!({ "extra": "files" })))
.await
.unwrap();
let msgs = LinearAssembler::new()
.with_media(Arc::new(Media))
.build(&store, &input(frame, SystemContext::base("B"), ModelInfo::default()))
.await
.unwrap();
assert_eq!(msgs[1], json!({ "role": "user", "content": "look" }));
}
+11 -1
View File
@@ -1,7 +1,7 @@
use serde::{Deserialize, Serialize};
use serde_json::Value;
use crate::message_meta::Attachment;
use crate::message_meta::{Attachment, ViewContextItem};
// ── Client → Server ───────────────────────────────────────────────────────────
@@ -11,6 +11,12 @@ pub struct ClientMessage {
/// Files attached to this message (uploaded beforehand via `POST /api/{source}/uploads`).
#[serde(default)]
pub attachments: Vec<Attachment>,
/// What the user had on screen when they sent this, as an ordered list of
/// opaque `{label, value}` pairs in English. Absent for clients that have no
/// view, and absent (not empty) when the user turned the sharing off — the
/// difference is what "not shared" looks like on the wire.
#[serde(default)]
pub view_context: Vec<ViewContextItem>,
}
/// Typed data push from remote clients (iOS app, etc.).
@@ -264,6 +270,10 @@ pub enum ServerEvent {
/// Files attached to the message; lets secondary clients render chips live.
#[serde(default, skip_serializing_if = "Vec::is_empty")]
attachments: Vec<Attachment>,
/// What the sender had on screen; echoed back so every client renders the
/// same chip the sender sees, and so a reload matches the live bubble.
#[serde(default, skip_serializing_if = "Vec::is_empty")]
view_context: Vec<ViewContextItem>,
},
/// Sent to a client right after it (re)connects, reporting whether a turn is
/// currently in flight for its session. Lets a reloaded page restore the
+389 -26
View File
@@ -1,16 +1,24 @@
//! Structured, reusable metadata attached to a `chat_history` row.
//!
//! Persisted as a single JSON column (`chat_history.metadata`) and intentionally
//! generic: today it carries user file **attachments**, but new keys can be added
//! later without a schema change. Two independent readers derive different views
//! from the same source:
//! - the **LLM context** builder appends [`attachments_block`] to the user turn,
//! - the **history UI** renders the structured attachments as chips.
//! generic: today it carries user file **attachments** and the **view context**
//! (what the user was looking at), but new keys can be added later without a
//! schema change. Two independent readers derive different views from the same
//! source:
//! - the **LLM context** builder appends [`attachments_body`] /
//! [`view_context_body`] to the user turn, inside one `<system-extra>` block,
//! - the **history UI** renders the structured metadata as chips.
//!
//! The raw `<system-extra>` text block is therefore never persisted — it is
//! generated on the fly from this metadata. The tag name lives in
//! [`SYSTEM_EXTRA_TAG`] so emission sites and the agent-facing instruction that
//! documents it can never drift apart.
//!
//! The `*_body` functions return **unwrapped** text: a message gets exactly one
//! `<system-extra>` block, so framing belongs to whoever composes it (in this
//! workspace, `SkaldMediaSource`'s `MessageExtras` impl) and never to the pieces.
use std::borrow::Cow;
use serde::{Deserialize, Serialize};
@@ -30,6 +38,24 @@ pub struct Attachment {
pub filesize: Option<u64>,
}
/// One `{label, value}` pair describing a slice of what the user had on screen
/// when the message was sent — the open page, the open folder, the selected text.
///
/// **Both halves are opaque free text written by the client, in English.** The
/// backend never matches on a label, never parses a value, and knows no key
/// names: a new page is a row in the frontend's table and zero lines of Rust.
/// Line numbers, entity names and the like are composed by the client *into the
/// label* (`"Selected text (report.md, lines 12-17)"`) for exactly that reason.
///
/// The list is ordered by the client and rendered in that order — a map would
/// make rendering order an accident of key naming, and order is part of the
/// provider's prefix-cache key.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct ViewContextItem {
pub label: String,
pub value: String,
}
/// Generic metadata bag for a chat message. Extra keys may be added over time;
/// `#[serde(default)]` keeps deserialization tolerant of older/newer shapes.
#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)]
@@ -39,12 +65,19 @@ pub struct MessageMetadata {
/// Present when this user turn was produced by a custom slash command.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub command: Option<CommandRef>,
/// What the user was looking at, as sent by the client and already put
/// through [`sanitize_view_context`] at the ingress. Absent (empty) for every
/// source that has no view — Telegram, cron, background agents.
#[serde(default, skip_serializing_if = "Vec::is_empty")]
pub view_context: Vec<ViewContextItem>,
}
impl MessageMetadata {
/// True when there is nothing worth persisting.
/// True when there is nothing worth persisting. Every field must be listed
/// here: a message carrying *only* view context would otherwise be stored
/// with `metadata = NULL`.
pub fn is_empty(&self) -> bool {
self.attachments.is_empty() && self.command.is_none()
self.attachments.is_empty() && self.command.is_none() && self.view_context.is_empty()
}
}
@@ -74,27 +107,195 @@ pub const SYSTEM_EXTRA_TAG: &str = "system-extra";
///
/// Callers must not add their own leading newlines — this helper owns the
/// framing. An empty `body` still emits the (empty) block; callers that want a
/// no-op on empty input should check themselves (as [`attachments_block`] does).
/// no-op on empty input check themselves — the `*_body` builders return `""`
/// precisely so a composer can test before wrapping.
pub fn system_extra(body: &str) -> String {
format!("\n\n<{TAG}>\n{body}\n</{TAG}>", TAG = SYSTEM_EXTRA_TAG)
}
/// Renders the human-readable block appended to a user turn so the LLM learns
/// which files were attached. Returns an empty string when there are none, so
/// callers can unconditionally concatenate it.
/// Escapes the harness tag so a value can never break out of the block that
/// carries it. Replaces `<` with `&lt;` **only** in the two sequences
/// `<system-extra>` and `</system-extra>` (case-insensitive), leaving every other
/// `<` alone — the body is data the model reads, not markup we own.
///
/// Shared by the web/mobile path and the Telegram plugin so every surface emits
/// an identical format. The wrapping tag is [`SYSTEM_EXTRA_TAG`].
pub fn attachments_block(attachments: &[Attachment]) -> String {
/// This is not a hypothetical: a selected paragraph, or a file written by another
/// member in a shared folder, can contain the closing tag verbatim, and would
/// then continue as if it were the user speaking. Applied to labels, values
/// **and attachment paths** (a file may legitimately be named `<system-extra>`).
pub fn neutralize_harness_tag(s: &str) -> Cow<'_, str> {
let open = format!("<{TAG}>", TAG = SYSTEM_EXTRA_TAG);
let close = format!("</{TAG}>", TAG = SYSTEM_EXTRA_TAG);
// ASCII-only lowercasing: byte-length preserving, so indices into `hay` are
// valid indices into `s` (a Unicode `to_lowercase` is not).
let hay = s.to_ascii_lowercase();
if !hay.contains(&open) && !hay.contains(&close) {
return Cow::Borrowed(s);
}
let mut out = String::with_capacity(s.len() + 8);
let mut i = 0usize;
while i < s.len() {
// `<system-extra>` cannot match at a `</…` position, so "whichever comes
// first" is unambiguous.
let next = match (hay[i..].find(&open), hay[i..].find(&close)) {
(Some(a), Some(b)) if a <= b => Some((a, open.len())),
(Some(_), Some(b)) => Some((b, close.len())),
(Some(a), None) => Some((a, open.len())),
(None, Some(b)) => Some((b, close.len())),
(None, None) => None,
};
match next {
Some((rel, len)) => {
let at = i + rel;
out.push_str(&s[i..at]);
out.push_str("&lt;");
// Keep the rest of the tag verbatim, original casing included.
out.push_str(&s[at + 1..at + len]);
i = at + len;
}
None => {
out.push_str(&s[i..]);
break;
}
}
}
Cow::Owned(out)
}
// ── View-context caps ─────────────────────────────────────────────────────────
//
// A text selection is unbounded by nature: a Cmd+A on a 2 MB file would ride in
// *every* future projection of that message, forever, at cost. So the bag is
// clamped — truncated, never rejected, with an explicit marker so the model
// knows there is more and can read the file with a tool.
/// Maximum number of `{label, value}` pairs kept on one message.
pub const VIEW_CONTEXT_MAX_ITEMS: usize = 12;
/// Maximum length of one label, in `char`s.
pub const VIEW_CONTEXT_MAX_LABEL: usize = 120;
/// Maximum length of one value, in `char`s.
pub const VIEW_CONTEXT_MAX_VALUE: usize = 4_096;
/// Maximum sum of every label + value on one message, in `char`s.
pub const VIEW_CONTEXT_MAX_TOTAL: usize = 16_384;
/// Truncates to `max` **`char`s including the marker**, so the result is always
/// within budget and a second pass leaves it alone (idempotence).
fn clamp_chars(s: &str, max: usize) -> Cow<'_, str> {
let total = s.chars().count();
if total <= max {
return Cow::Borrowed(s);
}
let marker = |kept: usize| format!("… [truncated: {kept} of {total} characters]");
// Two passes: the marker's own length depends on the number it prints, and
// the digit count can shrink once. Either way the result stays ≤ max.
let mut kept = max.saturating_sub(marker(max).chars().count());
kept = max.saturating_sub(marker(kept).chars().count());
let head: String = s.chars().take(kept).collect();
Cow::Owned(format!("{head}{}", marker(kept)))
}
/// Canonicalises an inbound view-context bag: neutralize the tag, clamp each
/// label, clamp each value, clamp the item count, clamp the running total.
///
/// Applied **at the ingress** (so the megabyte is never persisted) and again at
/// render time (old rows, other clients — defence in depth), which is why it is
/// idempotent: sanitizing an already-sanitized bag returns it unchanged.
pub fn sanitize_view_context(items: Vec<ViewContextItem>) -> Vec<ViewContextItem> {
// Below this many chars of budget an item would be nothing but its own
// truncation marker, so it is dropped instead.
const MIN_VALUE_BUDGET: usize = 64;
let mut out: Vec<ViewContextItem> = Vec::with_capacity(items.len().min(VIEW_CONTEXT_MAX_ITEMS));
let mut used = 0usize;
for item in items.into_iter().take(VIEW_CONTEXT_MAX_ITEMS) {
let label = clamp_chars(&neutralize_harness_tag(&item.label), VIEW_CONTEXT_MAX_LABEL).into_owned();
let value = clamp_chars(&neutralize_harness_tag(&item.value), VIEW_CONTEXT_MAX_VALUE).into_owned();
let label_len = label.chars().count();
let value_len = value.chars().count();
if used + label_len + value_len <= VIEW_CONTEXT_MAX_TOTAL {
used += label_len + value_len;
out.push(ViewContextItem { label, value });
continue;
}
// The overflowing item: keep as much of its value as the budget allows,
// then stop — everything after it would be arbitrary anyway.
let budget = VIEW_CONTEXT_MAX_TOTAL.saturating_sub(used + label_len);
if budget >= MIN_VALUE_BUDGET {
let value = clamp_chars(&value, budget).into_owned();
out.push(ViewContextItem { label, value });
}
break;
}
out
}
/// The attachments body — the lines listing attached paths, **without** the
/// `<system-extra>` wrapper: wrapping belongs to whoever composes the block, so
/// attachments and view context can share one.
///
/// Returns an empty string when there are none, so callers can unconditionally
/// concatenate. Shared by the web/mobile path and the Telegram plugin so every
/// surface emits an identical format.
pub fn attachments_body(attachments: &[Attachment]) -> String {
if attachments.is_empty() {
return String::new();
}
let noun = if attachments.len() == 1 { "file" } else { "files" };
let mut body = format!("{} attached {}:", attachments.len(), noun);
for a in attachments {
body.push_str(&format!("\n* {}", a.path));
body.push_str(&format!("\n* {}", neutralize_harness_tag(&a.path)));
}
system_extra(&body)
body
}
/// Constant header introducing the view-context lines.
///
/// **Owned by the backend, not by the client**: it is the temporal clause that
/// stops the model from reading an old block as the current state, and no client
/// may drop it.
const VIEW_CONTEXT_HEADER: &str = "Viewing at the time of this message:";
/// The view-context body — the header plus one line per pair, **without** the
/// `<system-extra>` wrapper (same reason as [`attachments_body`]).
///
/// Empty in, empty out: an empty bag renders the empty string, never an orphan
/// header. A single-line value renders inline (`* {label}: {value}`); a
/// multi-line one goes into a fenced block at column 0, with a fence longer than
/// any backtick run it contains.
pub fn view_context_body(items: &[ViewContextItem]) -> String {
if items.is_empty() {
return String::new();
}
let items = sanitize_view_context(items.to_vec());
if items.is_empty() {
return String::new();
}
let mut body = String::from(VIEW_CONTEXT_HEADER);
for it in &items {
if it.value.contains('\n') {
let fence = "`".repeat(longest_backtick_run(&it.value).max(2) + 1);
body.push_str(&format!("\n* {}:\n{fence}\n{}\n{fence}", it.label, it.value));
} else {
body.push_str(&format!("\n* {}: {}", it.label, it.value));
}
}
body
}
/// Length of the longest run of consecutive backticks in `s` (0 if none).
fn longest_backtick_run(s: &str) -> usize {
let mut best = 0usize;
let mut cur = 0usize;
for c in s.chars() {
if c == '`' {
cur += 1;
best = best.max(cur);
} else {
cur = 0;
}
}
best
}
#[cfg(test)]
@@ -123,12 +324,12 @@ mod tests {
}
#[test]
fn attachments_block_empty_is_empty() {
assert_eq!(attachments_block(&[]), "");
fn attachments_body_empty_is_empty() {
assert_eq!(attachments_body(&[]), "");
}
#[test]
fn attachments_block_lists_paths_inside_tag() {
fn attachments_body_lists_paths_and_pluralises() {
let a = Attachment {
path: "uploads/1/a.png".into(),
name: "a.png".into(),
@@ -141,12 +342,174 @@ mod tests {
mimetype: None,
filesize: None,
};
let out = attachments_block(&[a, b]);
// Pluralised noun, both paths, wrapped in the canonical tag.
assert!(out.contains("2 attached files:"));
assert!(out.contains("* uploads/1/a.png"));
assert!(out.contains("* uploads/1/b.pdf"));
assert!(out.contains(&format!("<{TAG}>", TAG = SYSTEM_EXTRA_TAG)));
assert!(out.contains(&format!("</{TAG}>", TAG = SYSTEM_EXTRA_TAG)));
assert_eq!(
attachments_body(std::slice::from_ref(&a)),
"1 attached file:\n* uploads/1/a.png"
);
assert_eq!(
attachments_body(&[a, b]),
"2 attached files:\n* uploads/1/a.png\n* uploads/1/b.pdf"
);
}
// ── View context ──────────────────────────────────────────────────────────
fn vc(label: &str, value: &str) -> ViewContextItem {
ViewContextItem { label: label.into(), value: value.into() }
}
fn close_tag() -> String {
format!("</{TAG}>", TAG = SYSTEM_EXTRA_TAG)
}
#[test]
fn view_context_body_empty_is_empty() {
assert_eq!(view_context_body(&[]), "");
// A bag that sanitizes down to nothing is empty too — never an orphan header.
assert!(!view_context_body(&[vc("Open page", "Files")]).is_empty());
}
#[test]
fn view_context_body_renders_header_and_single_line_pairs() {
let out = view_context_body(&[
vc("Open page", "File viewer (#file_viewer)"),
vc("Open file", "shared/casa/report.md"),
]);
assert_eq!(
out,
"Viewing at the time of this message:\n\
* Open page: File viewer (#file_viewer)\n\
* Open file: shared/casa/report.md"
);
// No wrapper: composing the block is the caller's job.
assert!(!out.contains(&format!("<{TAG}>", TAG = SYSTEM_EXTRA_TAG)));
}
#[test]
fn view_context_body_fences_multiline_values() {
let out = view_context_body(&[vc("Selected text (lines 12-17)", "one\ntwo")]);
assert!(out.contains("* Selected text (lines 12-17):\n```\none\ntwo\n```"), "{out}");
}
#[test]
fn view_context_body_fence_outgrows_contained_backticks() {
// Four backticks inside ⇒ a five-backtick fence, at column 0.
let out = view_context_body(&[vc("Selected text", "a\n````\nb")]);
assert!(out.contains("\n`````\na\n````\nb\n`````"), "{out}");
assert_eq!(longest_backtick_run("a ``` b `` c"), 3);
assert_eq!(longest_backtick_run("none"), 0);
}
#[test]
fn neutralize_only_touches_the_two_tag_sequences() {
assert!(matches!(neutralize_harness_tag("a < b <div> c"), Cow::Borrowed(_)));
let s = format!("before {} after <{TAG}>", close_tag(), TAG = SYSTEM_EXTRA_TAG);
let out = neutralize_harness_tag(&s);
assert_eq!(out, "before &lt;/system-extra> after &lt;system-extra>");
// Case-insensitive, casing of the rest preserved.
assert_eq!(neutralize_harness_tag("</SYSTEM-EXTRA>"), "&lt;/SYSTEM-EXTRA>");
// Idempotent.
assert_eq!(neutralize_harness_tag(&out), out);
}
#[test]
fn sanitized_rendering_never_carries_a_live_closing_tag() {
let close = close_tag();
let items = sanitize_view_context(vec![
vc(&format!("Selected text {close}"), &format!("evil {close} text")),
]);
let body = view_context_body(&items);
assert!(!body.contains(&close), "{body}");
assert!(body.contains("&lt;/system-extra>"));
// …and the same for an attachment path: a file may be named like the tag.
let a = Attachment {
path: format!("uploads/1/{close}.txt"),
name: "x.txt".into(),
mimetype: None,
filesize: None,
};
let out = attachments_body(&[a]);
assert!(!out.contains(&close), "{out}");
}
#[test]
fn clamp_truncates_per_item_on_char_boundaries_with_a_marker() {
// Accents and emoji: cutting by bytes would split a code point.
let value: String = "é🙂".repeat(4_000);
let items = sanitize_view_context(vec![vc("Selected text", &value)]);
let got = &items[0].value;
assert!(got.chars().count() <= VIEW_CONTEXT_MAX_VALUE);
// The marker reports the real length so the model knows there is more.
assert!(got.contains(&format!("of {} characters]", value.chars().count())), "{got}");
assert!(got.starts_with("é🙂"));
let label: String = "L".repeat(500);
let items = sanitize_view_context(vec![vc(&label, "v")]);
assert!(items[0].label.chars().count() <= VIEW_CONTEXT_MAX_LABEL);
assert!(items[0].label.contains("truncated"));
}
#[test]
fn clamp_caps_the_item_count() {
let many: Vec<_> = (0..40).map(|i| vc(&format!("L{i}"), "v")).collect();
let out = sanitize_view_context(many);
assert_eq!(out.len(), VIEW_CONTEXT_MAX_ITEMS);
// Order preserved: the first N, not an arbitrary N.
assert_eq!(out[0].label, "L0");
assert_eq!(out[VIEW_CONTEXT_MAX_ITEMS - 1].label, format!("L{}", VIEW_CONTEXT_MAX_ITEMS - 1));
}
#[test]
fn clamp_caps_the_running_total() {
let big = "x".repeat(VIEW_CONTEXT_MAX_VALUE);
let items: Vec<_> = (0..8).map(|i| vc(&format!("L{i}"), &big)).collect();
let out = sanitize_view_context(items);
let total: usize = out.iter().map(|i| i.label.chars().count() + i.value.chars().count()).sum();
assert!(total <= VIEW_CONTEXT_MAX_TOTAL, "total {total}");
// Four 4 KiB values fit in 16 KiB; the fifth is what overflows.
assert!(out.len() < 8);
}
#[test]
fn sanitize_is_idempotent() {
let value: String = "é🙂".repeat(4_000);
let close = close_tag();
let mut items: Vec<_> = (0..30)
.map(|i| vc(&format!("{close} L{i}"), &value))
.collect();
items.push(vc("short", "v"));
let once = sanitize_view_context(items);
let twice = sanitize_view_context(once.clone());
assert_eq!(once, twice);
// Rendering re-applies the clamp: same output both ways (defence in depth).
assert_eq!(view_context_body(&once), view_context_body(&twice));
}
#[test]
fn metadata_with_only_view_context_is_not_empty() {
let meta = MessageMetadata {
view_context: vec![vc("Open page", "Files")],
..Default::default()
};
assert!(!meta.is_empty());
assert!(MessageMetadata::default().is_empty());
}
#[test]
fn metadata_round_trips_and_tolerates_older_json() {
let meta = MessageMetadata {
view_context: vec![vc("Open file", "shared/casa/report.md")],
..Default::default()
};
let json = serde_json::to_string(&meta).unwrap();
assert_eq!(json, r#"{"view_context":[{"label":"Open file","value":"shared/casa/report.md"}]}"#);
assert_eq!(serde_json::from_str::<MessageMetadata>(&json).unwrap(), meta);
// A row written before the field existed.
let old = r#"{"attachments":[{"path":"uploads/1/a.png","name":"a.png"}]}"#;
let back: MessageMetadata = serde_json::from_str(old).unwrap();
assert!(back.view_context.is_empty());
assert_eq!(back.attachments.len(), 1);
}
}
@@ -18,13 +18,22 @@
//!
//! Both are re-checked here even though the paths came from trusted code: the
//! container is writable by the agent, so any host-side read must re-verify.
//!
//! The same type also implements `agent_loop::projection::MessageExtras` — the
//! **single** composer of a message's `<system-extra>` block (skipped attachment
//! paths + the view context). One type, one `Arc`, two hooks: the block's first
//! half is a media answer, so splitting them across two objects would mean
//! either two blocks or a handle passed between them.
use std::path::{Path, PathBuf};
use std::sync::Arc;
use agent_loop::projection::{MediaBlob, MediaSource};
use agent_loop::projection::{MediaBlob, MediaSource, MessageExtras};
use agent_loop::store::{StoredCall, StoredMessage};
use core_api::message_meta::{Attachment, MessageMetadata, attachments_block};
use core_api::message_meta::{
Attachment, MessageMetadata, ViewContextItem, attachments_body, sanitize_view_context,
system_extra, view_context_body,
};
use core_api::tool::MediaRef;
use core_api::user_fs::{UPLOADS_SUBDIR, UserFs};
use tracing::debug;
@@ -136,21 +145,32 @@ impl SkaldMediaSource {
Self { fs }
}
/// The attachments a stored message carries, in wire order.
fn attachments(msg: &StoredMessage) -> Vec<Attachment> {
/// The message's metadata bag, or the empty one.
fn meta(msg: &StoredMessage) -> MessageMetadata {
msg.metadata
.as_ref()
.and_then(|v| serde_json::from_value::<MessageMetadata>(v.clone()).ok())
.map(|m| m.attachments)
.unwrap_or_default()
}
/// The attachments a stored message carries, in wire order.
fn attachments(msg: &StoredMessage) -> Vec<Attachment> {
Self::meta(msg).attachments
}
/// The view context a stored message carries, **canonicalized**: the clamp
/// runs at the ingress, but a row written by an older build or another
/// client has not been through it, and it is also what makes the dedupe
/// compare like with like.
fn view_context(msg: &StoredMessage) -> Vec<ViewContextItem> {
sanitize_view_context(Self::meta(msg).view_context)
}
}
#[agent_loop::async_trait]
impl MediaSource for SkaldMediaSource {
async fn message_media(&self, msg: &StoredMessage) -> Vec<Arc<dyn MediaBlob>> {
// Positions matter: `skipped_text` indexes this same list.
// Positions matter: the `MessageExtras` impl below indexes this same list.
attachment_blobs(&self.fs, &Self::attachments(msg))
}
@@ -165,20 +185,58 @@ impl MediaSource for SkaldMediaSource {
ref_blobs(&self.fs, &refs)
}
fn skipped_text(&self, msg: &StoredMessage, skipped: &[usize]) -> Option<String> {
if skipped.is_empty() {
return None;
}
/// The one composer of a message's `<system-extra>` block.
///
/// Registered as the same `Arc` that serves [`MediaSource`], because the two
/// halves need the same knowledge: which attachments were left out is a media
/// answer, and it belongs in the same block as the view context. **One block per
/// message** — attachments first, then the view — because two would read to the
/// model as two unrelated harness interjections.
#[agent_loop::async_trait]
impl MessageExtras for SkaldMediaSource {
async fn appended_text(
&self,
msg: &StoredMessage,
prev: Option<&StoredMessage>,
skipped: &[usize],
) -> Option<String> {
let mut bodies: Vec<String> = Vec::new();
// The media that did not make it: the agent can still read these with a
// tool, so the paths go in as text.
if !skipped.is_empty() {
let attachments = Self::attachments(msg);
let left: Vec<Attachment> = skipped
.iter()
.filter_map(|&i| attachments.get(i).cloned())
.collect();
let body = attachments_body(&left);
if !body.is_empty() {
bodies.push(body);
}
}
let attachments = Self::attachments(msg);
let left: Vec<Attachment> = skipped
.iter()
.filter_map(|&i| attachments.get(i).cloned())
.collect();
if left.is_empty() {
return None;
// What the user was looking at — **unless the previous thing they said
// was sent from the same view**. Consecutive dedupe: in the normal case
// the page does not change between two messages, so this drops nearly
// all of the noise and turns the block into a signal of *change*. Note
// what it deliberately is not: it does not look at attachments (two
// messages from one page with different files still list the files), it
// re-emits on `prev == None` (after a compaction or a window cut the
// model has lost the earlier block), and a message *without* a view
// never suppresses anything — nothing here says "no longer shared", that
// is the header's temporal clause's job.
let view = Self::view_context(msg);
if !view.is_empty() && !prev.is_some_and(|p| Self::view_context(p) == view) {
let body = view_context_body(&view);
if !body.is_empty() {
bodies.push(body);
}
}
// The textual path block: the agent can still read these with a tool.
Some(attachments_block(&left))
(!bodies.is_empty()).then(|| system_extra(&bodies.join("\n\n")))
}
}
@@ -343,4 +401,121 @@ mod tests {
let _ = tokio::fs::remove_dir_all(&tmp).await;
}
// ── The `<system-extra>` composer ─────────────────────────────────────────
use agent_loop::ids::MessageId;
use agent_loop::store::Role;
use core_api::message_meta::ViewContextItem;
fn vc(label: &str, value: &str) -> ViewContextItem {
ViewContextItem { label: label.into(), value: value.into() }
}
/// A stored user message carrying `metadata`, and nothing else that matters.
fn msg(meta: MessageMetadata) -> StoredMessage {
StoredMessage {
id: MessageId(1),
role: Role::User,
content: "hi".into(),
reasoning: None,
synthetic: false,
failed: false,
metadata: Some(serde_json::to_value(meta).unwrap()),
usage: Default::default(),
calls: vec![],
}
}
fn source() -> SkaldMediaSource {
SkaldMediaSource::new(Arc::new(fs_home(Path::new("/nonexistent/homes/u1"))))
}
fn open_tag() -> String {
format!("<{TAG}>", TAG = core_api::message_meta::SYSTEM_EXTRA_TAG)
}
#[tokio::test]
async fn view_context_alone_produces_the_block() {
let m = msg(MessageMetadata {
view_context: vec![vc("Open page", "Files (#files)")],
..Default::default()
});
// `skipped` empty: the message has no media at all.
let out = source().appended_text(&m, None, &[]).await.unwrap();
assert!(out.starts_with("\n\n"), "{out:?}");
assert!(out.contains("Viewing at the time of this message:"));
assert!(out.contains("* Open page: Files (#files)"));
assert_eq!(out.matches(&open_tag()).count(), 1);
}
#[tokio::test]
async fn attachments_and_view_share_one_block_attachments_first() {
let m = msg(MessageMetadata {
attachments: vec![att("uploads/1/a.png")],
view_context: vec![vc("Open page", "Files (#files)")],
..Default::default()
});
let out = source().appended_text(&m, None, &[0]).await.unwrap();
assert_eq!(out.matches(&open_tag()).count(), 1, "exactly one block: {out}");
let at = out.find("1 attached file:").unwrap();
let view = out.find("Viewing at the time").unwrap();
assert!(at < view, "attachments first: {out}");
}
#[tokio::test]
async fn nothing_to_say_appends_nothing() {
assert!(source().appended_text(&msg(MessageMetadata::default()), None, &[]).await.is_none());
}
#[tokio::test]
async fn the_dedupe_is_consecutive_and_structural() {
let bag = vec![vc("Open page", "Files (#files)"), vc("Open folder", "shared/casa")];
let same = msg(MessageMetadata { view_context: bag.clone(), ..Default::default() });
let other = msg(MessageMetadata {
view_context: vec![vc("Open page", "Projects (#projects)")],
..Default::default()
});
let src = source();
// prev = None ⇒ emitted (a compaction or a window cut lands here).
assert!(src.appended_text(&same, None, &[]).await.is_some());
// Identical bag ⇒ suppressed.
assert!(src.appended_text(&same, Some(&same), &[]).await.is_none());
// Different bag ⇒ emitted.
assert!(src.appended_text(&same, Some(&other), &[]).await.is_some());
// A previous message with no view suppresses nothing.
assert!(
src.appended_text(&same, Some(&msg(MessageMetadata::default())), &[])
.await
.is_some()
);
}
#[tokio::test]
async fn the_dedupe_ignores_attachments() {
let bag = vec![vc("Open page", "Files (#files)")];
let prev = msg(MessageMetadata { view_context: bag.clone(), ..Default::default() });
let now = msg(MessageMetadata {
attachments: vec![att("uploads/1/a.png")],
view_context: bag,
..Default::default()
});
let out = source().appended_text(&now, Some(&prev), &[0]).await.unwrap();
assert!(out.contains("1 attached file:"), "{out}");
assert!(!out.contains("Viewing at the time"), "view suppressed, files not: {out}");
}
#[tokio::test]
async fn a_message_with_only_skipped_media_is_byte_identical_to_before() {
let m = msg(MessageMetadata {
attachments: vec![att("uploads/1/a.png"), att("uploads/1/b.pdf")],
..Default::default()
});
let out = source().appended_text(&m, None, &[0, 1]).await.unwrap();
assert_eq!(
out,
"\n\n<system-extra>\n2 attached files:\n* uploads/1/a.png\n* uploads/1/b.pdf\n</system-extra>"
);
}
}
@@ -65,10 +65,16 @@ pub fn skald_projection(
}
}
/// The assembler every Skald turn runs on: the configuration above plus the two
/// The assembler every Skald turn runs on: the configuration above plus the
/// content hooks. `fs` is the caller's filesystem view — without it media is
/// never inlined (nothing can be authorized), which is the right default for a
/// context with no user workspace.
///
/// `SkaldMediaSource` is registered on **two** hooks from one `Arc`: it decides
/// what may be inlined *and* composes the `<system-extra>` block. The block
/// therefore rides on `fs` being present — true on every real path (both live
/// call sites pass `Some`), and a context with no workspace has no view to
/// describe either.
pub fn skald_assembler(
activation: Arc<dyn ActivationSource>,
fs: Option<Arc<UserFs>>,
@@ -85,7 +91,8 @@ pub fn skald_assembler(
.with_activation(activation)
.with_digest(Arc::new(SkaldDigest));
if let Some(fs) = fs {
assembler = assembler.with_media(Arc::new(SkaldMediaSource::new(fs)));
let source = Arc::new(SkaldMediaSource::new(fs));
assembler = assembler.with_media(source.clone()).with_extras(source);
}
assembler
}
@@ -119,12 +119,18 @@ impl EventTranslator {
.as_ref()
.and_then(|v| serde_json::from_value(v.clone()).ok());
let attachments = meta.as_ref().map(|m| m.attachments.clone()).unwrap_or_default();
let view_context = meta.as_ref().map(|m| m.view_context.clone()).unwrap_or_default();
// A custom slash command persists its expanded template (for
// LLM replay) but the bubble shows the typed command.
let echo = meta
.and_then(|m| m.command.map(|c| c.display))
.unwrap_or(content);
self.emit(ServerEvent::UserMessage { message_id: message_id.get(), content: echo, attachments }).await;
self.emit(ServerEvent::UserMessage {
message_id: message_id.get(),
content: echo,
attachments,
view_context,
}).await;
}
LoopEvent::TokenDelta { kind, text } => {