Release 0.2.0 #4

Merged
dguiducci merged 96 commits from main into release 2026-08-17 18:07:20 +01:00
21 changed files with 784 additions and 94 deletions
Showing only changes of commit 6cb4ea0ce8 - Show all commits
+15 -2
View File
@@ -163,12 +163,17 @@ The agent sees **one namespace**, routed on the first path component. The choke
| `shared/{X}/…` | host `{WD}/shared/{X}` (if a member) | `UserFs::host_base_and_tail` | | `shared/{X}/…` | host `{WD}/shared/{X}` (if a member) | `UserFs::host_base_and_tail` |
| `projects/{O}/{S}/…` | host `{WD}/projects/{owner_userid}/{S}` (if a member) | `UserFs::host_base_and_tail` | | `projects/{O}/{S}/…` | host `{WD}/projects/{owner_userid}/{S}` (if a member) | `UserFs::host_base_and_tail` |
| `~/…`, relative | host `{WD}/homes/{userid}` | `UserFs::host_base_and_tail` | | `~/…`, relative | host `{WD}/homes/{userid}` | `UserFs::host_base_and_tail` |
| any other absolute path (`/tmp/…`, `/etc/…`) | the **container's own** filesystem | `resolve_target``container::exec_fs` |
Two views, **one storage**: the fs-tools run **host-side** in the Skald process on `{WD}/homes/{userid}` + `{WD}/shared/{X}`; `execute_cmd` runs **inside the container** (`docker exec -w <container-path> skald-{userid} sh -c …`, via `ExecuteCmd::run_with`) on the same paths bind-mounted (`homes/{userid}``/root`, `shared/{X}``/root/shared/{X}`, read-only when `can_write=0`). A file written in the container appears to the host fs-tools and vice versa. Two views, **one storage**: for the mounted subtree the fs-tools run **host-side** in the Skald process on `{WD}/homes/{userid}` + `{WD}/shared/{X}`; `execute_cmd` runs **inside the container** (`docker exec -w <container-path> skald-{userid} sh -c …`, via `ExecuteCmd::run_with`) on the same paths bind-mounted (`homes/{userid}``/root`, `shared/{X}``/root/shared/{X}`, read-only when `can_write=0`). A file written in the container appears to the host fs-tools and vice versa.
**The security boundary is the container, not the mounted subtree — the mount is the *fast* path, not the only one.** An agent already reaches every corner of its container through `execute_cmd`, which runs there with passwordless `sudo`; fs-tools that stopped at the mounts were not protecting anything, they were offering a poorer view of the same sandbox, and the model answered that by shelling out (the observed failure: `read_file /tmp/cv.txt`*"path escapes your workspace"* → the agent re-read it with `cat`). So `resolve_target` routes a physical path to one of two backings. An **absolute** path is container vocabulary — it is what `execute_cmd` prints — so it is reverse-mapped through `UserFs::container_to_agent` first: landing on a mount takes the host path (**`/root/x` *is* `~/x`**, which the tools used to reject outright, since `PathBuf::join` with an absolute tail silently discards the base and the result then failed the prefix check); landing nowhere means it exists only in the container, and `container::exec_fs` acts there over `docker exec` (paths passed **positionally** as `$1`, so a path containing `$(…)` is data, not syntax). Membership is not bypassed: `/root/shared/{X}` for a non-member still resolves to the same error as `shared/{X}`.
**One implementation per tool, not two.** Every single-file fs-tool already funnels through the same shape — resolve, then run a sync `execute` over one absolute host path — so the container branch is a **shuttle** (`fs::Shuttle`, behind `fs::run_physical`): pull the file out of the container, run the *unchanged* tool on the copy, push it back if the content changed (compared by bytes, not mtime, whose one-second resolution would miss a fast edit). Nothing about a tool's messages, diffs or pure transforms is duplicated. A missing remote file is deliberately **not** pre-created — `write_file` reports "Created" vs "Overwrote" from whether the path existed, and a placeholder would make every creation lie. Three tools opt out of the shuttle because a single file is the wrong unit: `list_files` lists in place via `exec_fs::list` (`find -printf`; `line_count` is omitted, since counting lines would turn a listing into a `docker exec` per file), `read_file` reads container paths as text (a shuttled copy is gone by the time the projection would inline a `MediaRef`, so media stays a mount-only feature), and `grep_files` **refuses** container paths with a pointer to `execute_cmd` + `rg` — its regex flavour, glob, windowing and offset would all have to be re-derived from ripgrep's flags, and a grep that answers *almost* the same is worse than one that says where to go. The viewer follows the same routing through `resolve_view_target` (`GET /api/file` and `show_file_to_user` open container paths; served without an ETag, so the editor stays read-only there).
**The memory roots are signposted inside the container, not merely absent.** `user-memory/`/`shared-memory/` are virtual, so nothing of them existed on disk — and the nothing was worse than it sounds: `cat user-memory/x.md` returned a bare ENOENT (which reads as *the note is missing*, not *wrong door*), while `mkdir -p user-memory && echo … > user-memory/x.md` **succeeded**, writing a real file into the home that no reader ever visits and that the next `ls` then confirms as if it had worked. Each root is therefore a **read-only bind mount** (`{WD}/.memory-signpost/{root}``{container_home}/{root}:ro`, gitignored, rewritten from consts on every `ensure`) holding a README that names the tools. Read-only *as a mount*, not as a mode: the container user has passwordless `sudo`, so a `chmod` would be a suggestion, whereas `:ro` holds — remounting needs `CAP_SYS_ADMIN` (verified: write, `sudo` write, `sudo chmod`, `sudo mount -o remount,rw` and `sudo rm` all fail). A README rather than an empty dir because `Permission denied` is an error, not an instruction — models answer it by reaching for `sudo`; the README puts the correction in the directory the failing command just named. These mounts are deliberately **not** in `UserFs`: they back no agent path and the host-side fs-tools must never resolve into them. They are the **fourth self-heal axis** in `reusable()` (`signposts_mounted`) rather than an `IMAGE_TAG` bump, since the image is unchanged and a bump would make every box rebuild it to fix a mount. The matching half is in `classify_memory`, which now strips the home spellings (`./`, `~/`, `/root/`) before matching the root — without it `~/user-memory/x.md` missed the match, fell through to the disk router, and became exactly the invisible physical file the signpost exists to prevent. **The memory roots are signposted inside the container, not merely absent.** `user-memory/`/`shared-memory/` are virtual, so nothing of them existed on disk — and the nothing was worse than it sounds: `cat user-memory/x.md` returned a bare ENOENT (which reads as *the note is missing*, not *wrong door*), while `mkdir -p user-memory && echo … > user-memory/x.md` **succeeded**, writing a real file into the home that no reader ever visits and that the next `ls` then confirms as if it had worked. Each root is therefore a **read-only bind mount** (`{WD}/.memory-signpost/{root}``{container_home}/{root}:ro`, gitignored, rewritten from consts on every `ensure`) holding a README that names the tools. Read-only *as a mount*, not as a mode: the container user has passwordless `sudo`, so a `chmod` would be a suggestion, whereas `:ro` holds — remounting needs `CAP_SYS_ADMIN` (verified: write, `sudo` write, `sudo chmod`, `sudo mount -o remount,rw` and `sudo rm` all fail). A README rather than an empty dir because `Permission denied` is an error, not an instruction — models answer it by reaching for `sudo`; the README puts the correction in the directory the failing command just named. These mounts are deliberately **not** in `UserFs`: they back no agent path and the host-side fs-tools must never resolve into them. They are the **fourth self-heal axis** in `reusable()` (`signposts_mounted`) rather than an `IMAGE_TAG` bump, since the image is unchanged and a bump would make every box rebuild it to fix a mount. The matching half is in `classify_memory`, which now strips the home spellings (`./`, `~/`, `/root/`) before matching the root — without it `~/user-memory/x.md` missed the match, fell through to the disk router, and became exactly the invisible physical file the signpost exists to prevent.
**Containment** (`resolve_host_path`): every physical fs-tool op canonicalizes the resolved path (following symlinks) and prefix-checks it against its mount base, **fail-closed**. Since the same tree is writable from inside the container, a symlink planted there that points outside the home/shared root is caught here — the host-side tool never escapes the user's workspace. `grep_files` stays disk-only (regex ≠ FTS; memory → `memory_search`) but resolves its root the same way. `execute_cmd`'s `workdir` is an agent path mapped to its container path via `UserFs::to_container`. **Containment** (`resolve_host_path`) is unchanged and still guards **the host branch**: every path that lands on a mount is canonicalized (following symlinks) and prefix-checked against its mount base, **fail-closed**. That check is what it always was — the defence against a symlink planted from inside the container pointing at the **host's** `/etc`, which the host-side tool would otherwise follow off the box. Opening the container branch does not weaken it: that branch never touches the host filesystem, so there is no host to escape from, and the check keeps applying to everything mounted. `grep_files` stays disk-only (regex ≠ FTS; memory → `memory_search`) but resolves its root the same way. `execute_cmd`'s `workdir` is an agent path mapped to its container path via `UserFs::to_container`.
The threading: `UserContext.fs` (built by `container::build_user_fs` at login, snapshotting shared memberships) → `ChatSessionManager``ChatSessionHandler.fs``ToolContext.fs`. **Admin CRUD is wired** (`src/frontend/api/shared_folders.rs``GET/POST /api/shared-folders`, `PATCH/DELETE /api/shared-folders/{id}`, `POST`/`DELETE .../members[/{user_id}]`; UI `shared-folders.js`): a create/describe/delete + per-member `can_write` surface, and each mutation emits `SystemEvent::UserMountsChanged`, on which the lifecycle reconciler runs `Skald::refresh_user_mounts` — rebuilding the affected user's fs + container mounts **in place**, so a membership change lands without a re-login (blueprint §6's "admin CRUD" + "membership refresh without re-login" TODOs, now closed; it still settles at next login/boot if the live remount fails). `execute_cmd` /stop is robust: the command runs under `setsid -w` in its own process-group (leader pid recorded in a container pidfile), and a `KillReaper` drop-guard reaps that group on /stop **or** timeout via a detached `docker exec` that walks `/proc` and kills members by **positive pid** (the container's dash mishandles `kill -<pgid>`); the pidfile is passed positionally (`$1`), and the container's `--init` (tini) reaps the killed processes so no zombies accumulate. **Per-user MCP connectors now run inside this container** (§7) — the container infra enabled it; see the MCP connectors section. The threading: `UserContext.fs` (built by `container::build_user_fs` at login, snapshotting shared memberships) → `ChatSessionManager``ChatSessionHandler.fs``ToolContext.fs`. **Admin CRUD is wired** (`src/frontend/api/shared_folders.rs``GET/POST /api/shared-folders`, `PATCH/DELETE /api/shared-folders/{id}`, `POST`/`DELETE .../members[/{user_id}]`; UI `shared-folders.js`): a create/describe/delete + per-member `can_write` surface, and each mutation emits `SystemEvent::UserMountsChanged`, on which the lifecycle reconciler runs `Skald::refresh_user_mounts` — rebuilding the affected user's fs + container mounts **in place**, so a membership change lands without a re-login (blueprint §6's "admin CRUD" + "membership refresh without re-login" TODOs, now closed; it still settles at next login/boot if the live remount fails). `execute_cmd` /stop is robust: the command runs under `setsid -w` in its own process-group (leader pid recorded in a container pidfile), and a `KillReaper` drop-guard reaps that group on /stop **or** timeout via a detached `docker exec` that walks `/proc` and kills members by **positive pid** (the container's dash mishandles `kill -<pgid>`); the pidfile is passed positionally (`$1`), and the container's `--init` (tini) reaps the killed processes so no zombies accumulate. **Per-user MCP connectors now run inside this container** (§7) — the container infra enabled it; see the MCP connectors section.
@@ -366,6 +371,14 @@ Three consequences worth not re-deriving:
The future automatic pass should trigger off the **resolved model's own context window**, not a hand-tuned `threshold_tokens` that has no idea which model is answering. The future automatic pass should trigger off the **resolved model's own context window**, not a hand-tuned `threshold_tokens` that has no idea which model is answering.
### The system prefix is frozen per conversation
Same economics, other end of the request. `AgentSystemContext::system_context` is called **once per round**, and it reassembled `base` from disk and SQLite every time — so an agent writing `user-memory/index.md` in round 3 made round 4, seconds later and with the cache certainly warm, a full miss. Since `base` is the head of every provider's cache key, that is the most expensive string in the request to touch. `loop_adapters/prefix_cache.rs::PrefixCache` builds it once per `(conversation, agent)` — the agent is in the key because a sub-agent shares its parent's conversation but has a prompt of its own — and holds it on `UserLoopRuntime`, so it outlives the turn.
The refresh rule is the only one that is free: **rebuild once the conversation has been idle longer than a provider's cache could survive** (`PREFIX_TTL`, 20 min). The clock is therefore *idle time of this conversation*, not time since a file changed, and reading restarts it — every `get` is a request about to go out. The asymmetry that sets the constant: below a provider's window you pay misses that buy nothing, above it you only pay freshness.
**Writes are deliberately not reacted to, and there is no bus variant for this.** When the agent itself edits an injected file the content is already in the context — its tool call and result sit two messages downstream — so refreshing would repeat what the model just said. A write from *elsewhere* (the same user's Telegram session, a cron job, another member editing `shared-memory/`) is genuinely invisible until the TTL: that is the case where an immediate rebuild costs the most, since a conversation that would notice is by definition a warm one, and the cheaper freshness path already exists — the agent can `read_file`, and a tool result *appends*, which invalidates nothing. The injection header says so in words. Cross-user invalidation would need a `SystemEventBus` variant plus a subscriber per user (the writer lives in a different `UserContext`); it is future work, and this type's key is the seam for it. Note `base` is frozen **whole**: freezing the memory files while letting `__USER_PROFILE__` move would invalidate just as much. The cost is that an `AGENT.md` edit lands at the next rebuild rather than the next round.
## Approval gate ## Approval gate
The rule engine `ApprovalManager::check` returns `Allow`/`Deny`/`Require` per tool call (default rules seeded on first boot; the catch-all `* require @999999` gates anything not explicitly allowed — e.g. `execute_cmd`, `execute_task`, writes outside whitelisted paths). It is wired to the loop as `loop_adapters/gate.rs::ApprovalGate` (`agent_loop::gate::Gate`). A `Require` registers a `oneshot` in the in-memory `pending` map keyed by `request_id` and emits an approval event over WS. The rule engine `ApprovalManager::check` returns `Allow`/`Deny`/`Require` per tool call (default rules seeded on first boot; the catch-all `* require @999999` gates anything not explicitly allowed — e.g. `execute_cmd`, `execute_task`, writes outside whitelisted paths). It is wired to the loop as `loop_adapters/gate.rs::ApprovalGate` (`agent_loop::gate::Gate`). A `Require` registers a `oneshot` in the in-memory `pending` map keyed by `request_id` and emits an approval event over WS.
+141
View File
@@ -0,0 +1,141 @@
//! Filesystem primitives that act **inside** a user's container, for the paths
//! their bind mounts do not cover (`/tmp`, `/etc`, an installed package's files…).
//!
//! The security boundary is the container, not the bind-mounted subtree: an agent
//! already has unrestricted reach in there through `execute_cmd`, which runs with
//! passwordless `sudo`. Tools that stopped at the mounts were therefore not
//! protecting anything — they offered a poorer view of the same sandbox, and the
//! model routinely worked around them by shelling out. These primitives close
//! that gap so the fs-tools see what the shell sees.
//!
//! What does *not* change is host containment. A path that lands on a mount keeps
//! the host fast path and its canonicalize-and-prefix-check, which is what stops a
//! symlink planted in the container from resolving against the **host's** `/etc`.
//! Nothing here ever touches the host filesystem, so there is no host to escape
//! from on this side.
//!
//! Paths are passed to `sh` **positionally** (`$1`), never interpolated into the
//! script, so a path containing quotes or `$(…)` is data and not shell syntax —
//! the same rule `execute_cmd` already follows for its pidfile.
use std::path::Path;
use std::process::Stdio;
use anyhow::{Context, Result, bail};
use tokio::io::AsyncWriteExt;
/// Runs a shell snippet inside `container` with `args` bound to `$1`, `$2`, …
/// Returns raw stdout — callers that expect text decode it themselves, so a
/// binary `cat` is not mangled on the way through.
async fn sh(container: &str, script: &str, args: &[&str]) -> Result<Vec<u8>> {
let mut argv: Vec<&str> = vec!["exec", container, "sh", "-c", script, "_"];
argv.extend_from_slice(args);
let out = tokio::process::Command::new("docker")
.args(&argv)
.stdin(Stdio::null())
.stdout(Stdio::piped())
.stderr(Stdio::piped())
.output()
.await
.context("failed to spawn `docker` (is the Docker CLI installed?)")?;
if out.status.success() {
Ok(out.stdout)
} else {
let err = String::from_utf8_lossy(&out.stderr);
bail!("{}", err.trim());
}
}
/// True when the snippet exits 0 — for the `test`-style probes, where a non-zero
/// exit is the answer rather than a failure.
async fn sh_ok(container: &str, script: &str, args: &[&str]) -> bool {
sh(container, script, args).await.is_ok()
}
/// Reads a file from inside the container.
pub async fn read(container: &str, path: &Path) -> Result<Vec<u8>> {
let p = path.to_string_lossy();
sh(container, r#"cat -- "$1""#, &[&p])
.await
.with_context(|| format!("Cannot read file: {p}"))
}
/// Writes a file inside the container, creating its parent directories. The
/// bytes travel on stdin rather than inside the script, so content is never
/// shell-parsed and size is bounded by the pipe, not by `ARG_MAX`.
pub async fn write(container: &str, path: &Path, bytes: &[u8]) -> Result<()> {
let p = path.to_string_lossy();
let mut child = tokio::process::Command::new("docker")
.args([
"exec", "-i", container, "sh", "-c",
r#"mkdir -p -- "$(dirname -- "$1")" && cat > "$1""#, "_", &p,
])
.stdin(Stdio::piped())
.stdout(Stdio::piped())
.stderr(Stdio::piped())
.spawn()
.context("failed to spawn `docker`")?;
child
.stdin
.take()
.context("docker exec produced no stdin")?
.write_all(bytes)
.await
.with_context(|| format!("Failed to write: {p}"))?;
let out = child.wait_with_output().await.context("docker exec failed")?;
if !out.status.success() {
bail!("Failed to write {p}: {}", String::from_utf8_lossy(&out.stderr).trim());
}
Ok(())
}
pub async fn exists(container: &str, path: &Path) -> bool {
sh_ok(container, r#"test -e "$1""#, &[&path.to_string_lossy()]).await
}
pub async fn is_dir(container: &str, path: &Path) -> bool {
sh_ok(container, r#"test -d "$1""#, &[&path.to_string_lossy()]).await
}
/// One entry of a container directory listing.
pub struct Entry {
pub name: String,
pub is_dir: bool,
pub size: u64,
}
/// Lists a directory inside the container, `depth` levels deep (1 = immediate
/// children). Emits `type\tsize\tpath` per line via `find`, which is in the image
/// and needs no parsing of `ls`'s locale-dependent output.
pub async fn list(container: &str, path: &Path, depth: usize) -> Result<Vec<Entry>> {
let p = path.to_string_lossy();
let d = depth.max(1).to_string();
let raw = sh(
container,
r#"find "$1" -mindepth 1 -maxdepth "$2" -printf '%y\t%s\t%p\n' 2>/dev/null || true"#,
&[&p, &d],
)
.await
.with_context(|| format!("Cannot list directory: {p}"))?;
let text = String::from_utf8_lossy(&raw);
let prefix = format!("{}/", p.trim_end_matches('/'));
let mut out = Vec::new();
for line in text.lines() {
let mut f = line.splitn(3, '\t');
let (Some(kind), Some(size), Some(full)) = (f.next(), f.next(), f.next()) else {
continue;
};
out.push(Entry {
name: full.strip_prefix(&prefix).unwrap_or(full).to_string(),
is_dir: kind == "d",
size: size.parse().unwrap_or(0),
});
}
out.sort_by(|a, b| b.is_dir.cmp(&a.is_dir).then_with(|| a.name.cmp(&b.name)));
Ok(out)
}
+2
View File
@@ -18,6 +18,8 @@
//! a container can be recreated from the image at any time; boot reconciliation //! a container can be recreated from the image at any time; boot reconciliation
//! relies on that. //! relies on that.
pub mod exec_fs;
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
use std::process::Stdio; use std::process::Stdio;
use std::sync::Arc; use std::sync::Arc;
@@ -28,6 +28,7 @@ use crate::llm::logging::RequestLogTarget;
use crate::loop_adapters::activation::SkaldToolActivator; use crate::loop_adapters::activation::SkaldToolActivator;
use crate::loop_adapters::builtins::{SkaldAskUserTool, SkaldHumanChannel}; use crate::loop_adapters::builtins::{SkaldAskUserTool, SkaldHumanChannel};
use crate::loop_adapters::history::SqliteHistory; use crate::loop_adapters::history::SqliteHistory;
use crate::loop_adapters::prefix_cache::PrefixCache;
use crate::loop_adapters::runtime::LoopConfig; use crate::loop_adapters::runtime::LoopConfig;
use crate::loop_adapters::scope::TurnScope; use crate::loop_adapters::scope::TurnScope;
use crate::loop_adapters::selector::SkaldSelector; use crate::loop_adapters::selector::SkaldSelector;
@@ -51,6 +52,9 @@ pub struct SkaldAgentCatalog {
/// The swappable fs cell, so a §6 remount reaches sub-agents too. /// The swappable fs cell, so a §6 remount reaches sub-agents too.
fs: SharedFs, fs: SharedFs,
config: LoopConfig, config: LoopConfig,
/// Shared with the parent runtime: a child's prefix is keyed by its own
/// agent id, so it never collides with the conversation's root frame.
prefix_cache: Arc<PrefixCache>,
/// The delegate tool, injected post-construction. **Weak** on purpose: the /// The delegate tool, injected post-construction. **Weak** on purpose: the
/// delegate holds the catalog, so an `Arc` here would be a cycle that never /// delegate holds the catalog, so an `Arc` here would be a cycle that never
/// frees (and this graph lives as long as the user). /// frees (and this graph lives as long as the user).
@@ -70,6 +74,7 @@ impl SkaldAgentCatalog {
registry: Arc<ToolRegistry>, registry: Arc<ToolRegistry>,
fs: SharedFs, fs: SharedFs,
config: LoopConfig, config: LoopConfig,
prefix_cache: Arc<PrefixCache>,
) -> Self { ) -> Self {
let core_tools = registry.all_tools(); let core_tools = registry.all_tools();
Self { Self {
@@ -84,6 +89,7 @@ impl SkaldAgentCatalog {
core_tools, core_tools,
fs, fs,
config, config,
prefix_cache,
delegate: RwLock::new(Weak::new()), delegate: RwLock::new(Weak::new()),
} }
} }
@@ -134,6 +140,7 @@ impl AgentCatalog for SkaldAgentCatalog {
// writes the SAME one as its parent. // writes the SAME one as its parent.
scratchpad_sid: scope.scratchpad_sid, scratchpad_sid: scope.scratchpad_sid,
datetime: self.config.datetime.clone(), datetime: self.config.datetime.clone(),
prefix_cache: self.prefix_cache.clone(),
}); });
// The child's def list: parent's base minus root-only minus the // The child's def list: parent's base minus root-only minus the
@@ -21,6 +21,9 @@
//! tool result — the library does the shaping. //! tool result — the library does the shaping.
//! - [`async_task`] — `execute_task mode=async` as a durable cron job, and the //! - [`async_task`] — `execute_task mode=async` as a durable cron job, and the
//! delivery of its result back into the parent conversation (§7.2). //! delivery of its result back into the parent conversation (§7.2).
//! - [`prefix_cache`] — the cacheable half of the system prompt, frozen per
//! conversation so a mid-turn memory write does not invalidate the provider's
//! prompt cache.
//! - [`runtime::UserLoopRuntime`] — the one `LoopManager` per user (D12) these //! - [`runtime::UserLoopRuntime`] — the one `LoopManager` per user (D12) these
//! are all assembled into, plus the per-turn parameters. //! are all assembled into, plus the per-turn parameters.
@@ -33,6 +36,7 @@ pub mod history;
pub mod hooks; pub mod hooks;
pub mod live_input; pub mod live_input;
pub mod media_source; pub mod media_source;
pub mod prefix_cache;
pub mod preview; pub mod preview;
#[cfg(test)] #[cfg(test)]
mod projection_snapshots; mod projection_snapshots;
@@ -0,0 +1,167 @@
//! `PrefixCache` — the system prefix, frozen for as long as a provider's prompt
//! cache could still be holding it.
//!
//! Every provider that caches keys on the longest common *prefix*, and the
//! system prompt is the first thing in it — so rebuilding it changes the whole
//! request. That is what used to happen on every round:
//! [`AgentSystemContext`](super::system::AgentSystemContext) reassembles `base`
//! from disk and SQLite each time it is asked, so an agent writing to
//! `user-memory/index.md` in round 3 turned round 4, seconds later and with the
//! cache certainly warm, into a full miss.
//!
//! So the prefix is built once and kept. The refresh rule is the one that costs
//! nothing: **rebuild only once the conversation has been idle long enough that
//! the provider's cache is gone anyway.** Below that window a rebuild buys
//! freshness at the price of a guaranteed miss; above it, it is free. Hence the
//! clock is *idle time of this conversation*, not time since some file changed
//! — and every call to [`PrefixCache::get`] is a request about to go out, which
//! is why reading restarts the window.
//!
//! **Writes are deliberately not reacted to.** When the agent itself edits an
//! injected file the new content is already in the context — the tool call and
//! its result sit two messages downstream — so refreshing the prefix would only
//! repeat what the model just said. A write from *elsewhere* (the same user's
//! Telegram session, a cron job, another member editing `shared-memory/`) is
//! genuinely invisible until the TTL, and that is the trade taken knowingly: it
//! is precisely the case where an immediate rebuild costs the most, since a
//! conversation that would notice is by definition a warm one. The freshness
//! path already exists and is cheaper — the agent can `read_file`, and a tool
//! result *appends*, which never invalidates anything. The injection header in
//! `system.rs` tells it so.
//!
//! Reacting to another user's write would need a `SystemEventBus` variant and a
//! subscriber per user, since the writer lives in a different `UserContext`.
//! That is future work; the seam for it is this type's key.
use std::collections::HashMap;
use std::sync::Mutex;
use std::time::{Duration, Instant};
use agent_loop::ids::ConversationId;
/// How long a prefix survives without its conversation calling a model.
///
/// The asymmetry that sets it: going *below* a provider's cache window pays
/// misses that buy nothing, while going above only costs freshness we have
/// already decided we do not need. Anthropic's `ephemeral` blocks live 5
/// minutes; OpenAI's automatic prefix cache is fuzzier and can last longer.
pub const PREFIX_TTL: Duration = Duration::from_secs(20 * 60);
/// A conversation plus the agent running in it. Both are needed: a sub-agent
/// shares its parent's conversation but has its own prompt, and therefore its
/// own cache prefix.
type Key = (ConversationId, String);
struct Entry {
base: String,
last_used: Instant,
}
/// One user's frozen prefixes. Lives on `UserLoopRuntime`, so it spans every
/// turn of every conversation that user has open.
pub struct PrefixCache {
ttl: Duration,
entries: Mutex<HashMap<Key, Entry>>,
}
impl PrefixCache {
pub fn new() -> Self {
Self::with_ttl(PREFIX_TTL)
}
/// A cache with a custom idle window — tests, and the knob a config key
/// would turn if one is ever wanted.
pub fn with_ttl(ttl: Duration) -> Self {
Self { ttl, entries: Mutex::new(HashMap::new()) }
}
/// The prefix for this turn, if one was built recently enough. Restarts the
/// idle window on a hit.
pub fn get(&self, key: &Key) -> Option<String> {
let mut entries = self.entries.lock().unwrap();
let entry = entries.get_mut(key)?;
if entry.last_used.elapsed() >= self.ttl {
entries.remove(key);
return None;
}
entry.last_used = Instant::now();
Some(entry.base.clone())
}
/// Stores a freshly built prefix, dropping whatever has gone idle — which is
/// what keeps the map bounded without an eviction policy to remember. It is
/// also what collects the one-shot conversations (system-agent passes,
/// ephemeral turns) that would otherwise each leave an entry behind.
///
/// Two rounds racing on the same key build twice and the last one wins. That
/// is why the build happens *outside* this type: holding the lock across it
/// would serialise every turn of every conversation behind one mutex, to
/// save a duplicated string.
pub fn put(&self, key: Key, base: String) {
let mut entries = self.entries.lock().unwrap();
entries.retain(|_, e| e.last_used.elapsed() < self.ttl);
entries.insert(key, Entry { base, last_used: Instant::now() });
}
}
impl Default for PrefixCache {
fn default() -> Self {
Self::new()
}
}
#[cfg(test)]
mod tests {
use super::*;
fn key(conv: &str, agent: &str) -> Key {
(ConversationId::new(conv), agent.to_string())
}
#[test]
fn a_stored_prefix_is_served_back() {
let cache = PrefixCache::new();
cache.put(key("session:1", "assistant"), "PROMPT".into());
assert_eq!(cache.get(&key("session:1", "assistant")).as_deref(), Some("PROMPT"));
}
#[test]
fn an_idle_prefix_is_a_miss() {
let cache = PrefixCache::with_ttl(Duration::from_millis(20));
cache.put(key("session:1", "assistant"), "PROMPT".into());
std::thread::sleep(Duration::from_millis(40));
assert_eq!(cache.get(&key("session:1", "assistant")), None);
}
/// The whole point of the idle clock: a conversation that keeps talking
/// keeps its prefix, however long it runs.
#[test]
fn using_a_prefix_restarts_the_idle_window() {
let cache = PrefixCache::with_ttl(Duration::from_millis(60));
cache.put(key("session:1", "assistant"), "PROMPT".into());
for _ in 0..4 {
std::thread::sleep(Duration::from_millis(20));
assert!(cache.get(&key("session:1", "assistant")).is_some());
}
}
/// A sub-agent shares the conversation and must not be served its parent's
/// prompt.
#[test]
fn the_agent_is_part_of_the_key() {
let cache = PrefixCache::new();
cache.put(key("session:1", "assistant"), "PARENT".into());
cache.put(key("session:1", "researcher"), "CHILD".into());
assert_eq!(cache.get(&key("session:1", "assistant")).as_deref(), Some("PARENT"));
assert_eq!(cache.get(&key("session:1", "researcher")).as_deref(), Some("CHILD"));
}
#[test]
fn storing_drops_the_entries_that_went_idle() {
let cache = PrefixCache::with_ttl(Duration::from_millis(20));
cache.put(key("session:1", "assistant"), "OLD".into());
std::thread::sleep(Duration::from_millis(40));
cache.put(key("session:2", "assistant"), "NEW".into());
assert_eq!(cache.entries.lock().unwrap().len(), 1);
}
}
@@ -42,6 +42,7 @@ use crate::loop_adapters::gate::ApprovalGate;
use crate::loop_adapters::history::SqliteHistory; use crate::loop_adapters::history::SqliteHistory;
use crate::loop_adapters::hooks::{DtlReanchorHook, SkaldWritePreviewHook}; use crate::loop_adapters::hooks::{DtlReanchorHook, SkaldWritePreviewHook};
use crate::loop_adapters::live_input::PendingLiveInput; use crate::loop_adapters::live_input::PendingLiveInput;
use crate::loop_adapters::prefix_cache::PrefixCache;
use crate::loop_adapters::preview::PreviewContext; use crate::loop_adapters::preview::PreviewContext;
use crate::loop_adapters::projection_cfg::skald_assembler; use crate::loop_adapters::projection_cfg::skald_assembler;
use crate::loop_adapters::scope::TurnScope; use crate::loop_adapters::scope::TurnScope;
@@ -92,6 +93,9 @@ pub struct UserLoopRuntime {
clarification: Arc<ClarificationManager>, clarification: Arc<ClarificationManager>,
tool_discovery: Arc<ToolDiscovery>, tool_discovery: Arc<ToolDiscovery>,
config: LoopConfig, config: LoopConfig,
/// The user's frozen system prefixes, shared with the agent catalog so a
/// sub-agent's own prefix is cached alongside its parent's.
prefix_cache: Arc<PrefixCache>,
} }
/// What a turn contributes on top of the runtime. /// What a turn contributes on top of the runtime.
@@ -154,6 +158,10 @@ impl UserLoopRuntime {
.build()?, .build()?,
); );
// One per user, living as long as this runtime: a conversation's system
// prefix must outlast its turns for the provider's cache to hold.
let prefix_cache = Arc::new(PrefixCache::new());
let catalog = Arc::new(SkaldAgentCatalog::new( let catalog = Arc::new(SkaldAgentCatalog::new(
pool.clone(), pool.clone(),
shared_pool.clone(), shared_pool.clone(),
@@ -165,6 +173,7 @@ impl UserLoopRuntime {
tools.clone(), tools.clone(),
fs.clone(), fs.clone(),
config.clone(), config.clone(),
prefix_cache.clone(),
)); ));
// `mode: "async"` runs as a durable cron job; the manager behind it is // `mode: "async"` runs as a durable cron job; the manager behind it is
// set at wiring time (see `CronExecutor`). // set at wiring time (see `CronExecutor`).
@@ -197,6 +206,7 @@ impl UserLoopRuntime {
clarification, clarification,
tool_discovery, tool_discovery,
config, config,
prefix_cache,
})) }))
} }
@@ -244,6 +254,7 @@ impl UserLoopRuntime {
project_root: scope.project_root.clone(), project_root: scope.project_root.clone(),
scratchpad_sid: scope.scratchpad_sid, scratchpad_sid: scope.scratchpad_sid,
datetime: self.config.datetime.clone(), datetime: self.config.datetime.clone(),
prefix_cache: self.prefix_cache.clone(),
}); });
// The agent's own declarations. Loaded once here and used twice below — // The agent's own declarations. Loaded once here and used twice below —
+78 -46
View File
@@ -16,6 +16,7 @@ use agent_loop::context::{SystemContext, SystemContextSource, TurnInfo};
use sqlx::SqlitePool; use sqlx::SqlitePool;
use crate::config::DatetimeConfig; use crate::config::DatetimeConfig;
use crate::loop_adapters::prefix_cache::PrefixCache;
use crate::mcp::McpProvider; use crate::mcp::McpProvider;
/// Registry of installed skills, relative to Skald's process cwd. Injected /// Registry of installed skills, relative to Skald's process cwd. Injected
@@ -44,18 +45,92 @@ pub struct AgentSystemContext {
/// sub-task (the blackboard is shared by every agent of a session). /// sub-task (the blackboard is shared by every agent of a session).
pub scratchpad_sid: i64, pub scratchpad_sid: i64,
pub datetime: DatetimeConfig, pub datetime: DatetimeConfig,
/// The user's frozen prefixes — `base` is assembled once per conversation
/// and reused while its provider cache could still be warm.
pub prefix_cache: Arc<PrefixCache>,
} }
#[agent_loop::async_trait] #[agent_loop::async_trait]
impl SystemContextSource for AgentSystemContext { impl SystemContextSource for AgentSystemContext {
async fn system_context(&self, _turn: &TurnInfo) -> agent_loop::Result<SystemContext> { async fn system_context(&self, turn: &TurnInfo) -> agent_loop::Result<SystemContext> {
// `base` is the head of every provider's cache key, so reassembling it
// between rounds — which is what an agent editing an injected memory
// file used to cause — invalidates the entire request. It is therefore
// built once per conversation and held; see [`super::prefix_cache`].
let key = (turn.conversation.clone(), self.agent_id.clone());
let static_content = match self.prefix_cache.get(&key) {
Some(base) => base,
None => {
let base = self.build_base().await?;
self.prefix_cache.put(key, base.clone());
base
}
};
// The scratchpad sits before the conversation: shared by every agent of
// the session, and re-read every turn (it changes, so it is its own
// message rather than part of the cached prefix).
let extra_static = self.scratchpad_block().await?.into_iter().collect();
// The fresh layers, in the order the model reads them.
let mut dynamic_tail: Vec<String> = Vec::new();
dynamic_tail.extend(self.extra_dynamic.clone());
dynamic_tail.extend(self.datetime_block());
Ok(SystemContext {
base: static_content,
extra_static,
dynamic_tail,
tail_reminder: self.tail_reminder.clone(),
})
}
}
/// OS description (type + version), computed once.
fn os_description() -> &'static str {
static OS: std::sync::OnceLock<String> = std::sync::OnceLock::new();
OS.get_or_init(|| os_info::get().to_string())
}
/// Formats an instant to hour precision: `Sunday 2026-08-02 17:00 +02:00`.
///
/// Minutes and seconds are dropped by the format string itself, so the
/// truncation always happens in the zone being displayed. The weekday is part
/// of the format on purpose — see [`AgentSystemContext::datetime_block`].
fn render_hour<Tz: chrono::TimeZone>(dt: chrono::DateTime<Tz>) -> String
where
Tz::Offset: std::fmt::Display,
{
dt.format("%A %Y-%m-%d %H:00 %:z").to_string()
}
/// System IANA timezone name, computed once.
fn system_timezone() -> Option<&'static str> {
static TZ: std::sync::OnceLock<Option<String>> = std::sync::OnceLock::new();
TZ.get_or_init(|| iana_time_zone::get_timezone().ok()).as_deref()
}
impl AgentSystemContext {
/// Assembles the cacheable prefix: the agent's prompt, its injected memory,
/// the skills index, the interface extras and every substitution.
///
/// Every layer here is frozen together, because the unit a provider caches
/// is the finished string — freezing the memory files while letting
/// `__USER_PROFILE__` move would invalidate just as much. The cost is that
/// an `AGENT.md` edit is picked up at the next rebuild rather than the next
/// round, which matters only while writing prompts.
async fn build_base(&self) -> agent_loop::Result<String> {
let mut static_content = crate::agents::load_prompt(&self.agent_id)?; let mut static_content = crate::agents::load_prompt(&self.agent_id)?;
let meta = crate::agents::load_meta(&self.agent_id)?; let meta = crate::agents::load_meta(&self.agent_id)?;
if !meta.inject_memory.is_empty() { if !meta.inject_memory.is_empty() {
static_content.push_str( static_content.push_str(
"\n\n---\nThe following memory files have been loaded automatically. \ "\n\n---\nThe following memory files have been loaded automatically. \
You can edit them with `edit_file` or `write_file` using the path shown.\n" You can edit them with `edit_file` or `write_file` using the path shown.\n\
Their contents are a snapshot taken earlier in this conversation. Your own \
edits are already reflected in what you have seen since; but if it matters \
that a file is current — a shared note another member may have changed in \
the meantime — read it again before relying on it.\n"
); );
for mem_path in &meta.inject_memory { for mem_path in &meta.inject_memory {
let (content, display) = self.load_inject_memory(mem_path).await; let (content, display) = self.load_inject_memory(mem_path).await;
@@ -116,52 +191,9 @@ impl SystemContextSource for AgentSystemContext {
} }
} }
static_content = resolve_harness_tag(static_content); Ok(resolve_harness_tag(static_content))
// The scratchpad sits before the conversation: shared by every agent of
// the session, and re-read every turn (it changes, so it is its own
// message rather than part of the cached prefix).
let extra_static = self.scratchpad_block().await?.into_iter().collect();
// The fresh layers, in the order the model reads them.
let mut dynamic_tail: Vec<String> = Vec::new();
dynamic_tail.extend(self.extra_dynamic.clone());
dynamic_tail.extend(self.datetime_block());
Ok(SystemContext {
base: static_content,
extra_static,
dynamic_tail,
tail_reminder: self.tail_reminder.clone(),
})
} }
}
/// OS description (type + version), computed once.
fn os_description() -> &'static str {
static OS: std::sync::OnceLock<String> = std::sync::OnceLock::new();
OS.get_or_init(|| os_info::get().to_string())
}
/// Formats an instant to hour precision: `Sunday 2026-08-02 17:00 +02:00`.
///
/// Minutes and seconds are dropped by the format string itself, so the
/// truncation always happens in the zone being displayed. The weekday is part
/// of the format on purpose — see [`AgentSystemContext::datetime_block`].
fn render_hour<Tz: chrono::TimeZone>(dt: chrono::DateTime<Tz>) -> String
where
Tz::Offset: std::fmt::Display,
{
dt.format("%A %Y-%m-%d %H:00 %:z").to_string()
}
/// System IANA timezone name, computed once.
fn system_timezone() -> Option<&'static str> {
static TZ: std::sync::OnceLock<Option<String>> = std::sync::OnceLock::new();
TZ.get_or_init(|| iana_time_zone::get_timezone().ok()).as_deref()
}
impl AgentSystemContext {
/// The session scratchpad as an XML block, or `None` when empty. /// The session scratchpad as an XML block, or `None` when empty.
async fn scratchpad_block(&self) -> agent_loop::Result<Option<String>> { async fn scratchpad_block(&self) -> agent_loop::Result<Option<String>> {
let notes = crate::db::scratchpad::for_session(&self.pool, self.scratchpad_sid).await?; let notes = crate::db::scratchpad::for_session(&self.pool, self.scratchpad_sid).await?;
@@ -224,6 +224,9 @@ pub async fn project(db: &Db, agent: &AgentFixture, case: &Case) -> Vec<Value> {
project_root: None, project_root: None,
scratchpad_sid: 1, scratchpad_sid: 1,
datetime: datetime(), datetime: datetime(),
// A cache of its own per projection: each case must see a freshly
// assembled prefix, never one another case left behind.
prefix_cache: Arc::new(crate::loop_adapters::prefix_cache::PrefixCache::new()),
}; };
let system = system_source let system = system_source
.system_context(&TurnInfo { .system_context(&TurnInfo {
@@ -82,10 +82,7 @@ impl Tool for AppendFile {
fn run_with<'a>(&'a self, ctx: &ToolContext, args: Value) -> Box<dyn ToolExecution + 'a> { fn run_with<'a>(&'a self, ctx: &ToolContext, args: Value) -> Box<dyn ToolExecution + 'a> {
let path = super::path_arg(&args).unwrap_or_default(); let path = super::path_arg(&args).unwrap_or_default();
let Some(m) = classify_memory(&path) else { let Some(m) = classify_memory(&path) else {
return match super::rewrite_to_host(&ctx.fs, &path, args) { return super::run_physical(self, &ctx.fs, &path, args);
Ok(args) => self.run(args),
Err(e) => super::error_exec(e.to_string()),
};
}; };
let pool = match m.scope { let pool = match m.scope {
MemScope::User => Arc::clone(&ctx.pool), MemScope::User => Arc::clone(&ctx.pool),
+1 -4
View File
@@ -159,10 +159,7 @@ impl Tool for EditFile {
fn run_with<'a>(&'a self, ctx: &ToolContext, args: Value) -> Box<dyn ToolExecution + 'a> { fn run_with<'a>(&'a self, ctx: &ToolContext, args: Value) -> Box<dyn ToolExecution + 'a> {
let path = super::path_arg(&args).unwrap_or_default(); let path = super::path_arg(&args).unwrap_or_default();
let Some(m) = classify_memory(&path) else { let Some(m) = classify_memory(&path) else {
return match super::rewrite_to_host(&ctx.fs, &path, args) { return super::run_physical(self, &ctx.fs, &path, args);
Ok(args) => self.run(args),
Err(e) => super::error_exec(e.to_string()),
};
}; };
let pool = match m.scope { let pool = match m.scope {
MemScope::User => Arc::clone(&ctx.pool), MemScope::User => Arc::clone(&ctx.pool),
+13 -3
View File
@@ -100,9 +100,19 @@ impl Tool for GrepFiles {
user-memory/ or shared-memory/".to_string(), user-memory/ or shared-memory/".to_string(),
); );
} }
match super::rewrite_to_host(&ctx.fs, &path, args) { // Searching a *tree* is the one thing neither the shuttle (one file) nor a
Ok(args) => self.run(args), // faithful `rg` translation can serve: this tool's regex flavour, glob,
Err(e) => super::error_exec(e.to_string()), // windowing and offset would all have to be re-derived from ripgrep's
// flags and output, and a grep that answers *almost* the same is worse
// than one that says where to go.
match super::resolve_target(&ctx.fs, &path) {
Ok(super::FsTarget::Host(host)) => self.run(super::point_at(&path, &host, args)),
Ok(super::FsTarget::Container { .. }) => super::error_exec(format!(
"grep_files only searches your mounted folders (~, shared/, projects/, docs/); \
{path} lives only inside your container. Search it with execute_cmd, e.g. \
`rg -n 'pattern' {path}` (ripgrep is installed)."
)),
Err(e) => super::error_exec(e.to_string()),
} }
} }
@@ -96,10 +96,7 @@ impl Tool for InsertAtLine {
fn run_with<'a>(&'a self, ctx: &ToolContext, args: Value) -> Box<dyn ToolExecution + 'a> { fn run_with<'a>(&'a self, ctx: &ToolContext, args: Value) -> Box<dyn ToolExecution + 'a> {
let path = super::path_arg(&args).unwrap_or_default(); let path = super::path_arg(&args).unwrap_or_default();
let Some(m) = classify_memory(&path) else { let Some(m) = classify_memory(&path) else {
return match super::rewrite_to_host(&ctx.fs, &path, args) { return super::run_physical(self, &ctx.fs, &path, args);
Ok(args) => self.run(args),
Err(e) => super::error_exec(e.to_string()),
};
}; };
let pool = match m.scope { let pool = match m.scope {
MemScope::User => Arc::clone(&ctx.pool), MemScope::User => Arc::clone(&ctx.pool),
+46 -3
View File
@@ -79,10 +79,24 @@ impl Tool for ListFiles {
let path = args["path"].as_str().unwrap_or("").to_string(); let path = args["path"].as_str().unwrap_or("").to_string();
let with_metadata = args["with_metadata"].as_bool().unwrap_or(false); let with_metadata = args["with_metadata"].as_bool().unwrap_or(false);
let Some(m) = classify_memory(&path) else { let Some(m) = classify_memory(&path) else {
return match super::rewrite_to_host(&ctx.fs, &path, args) { // A directory is the one shape the shuttle cannot serve — it moves a
Ok(args) => self.run(args), // single file — so a container-only path is listed in place.
Err(e) => super::error_exec(e.to_string()), let host = match super::resolve_target(&ctx.fs, &path) {
Ok(super::FsTarget::Host(h)) => h,
Ok(super::FsTarget::Container { container, path: dir }) => {
let depth = args["depth"].as_u64().unwrap_or(3) as usize;
let dirs_only = args["dirs_only"].as_bool().unwrap_or(false);
return Box::new(SimpleExecution::new(Box::pin(async move {
let entries =
crate::container::exec_fs::list(&container, &dir, depth).await?;
Ok(ToolResult::Text(render_container_listing(
entries, dirs_only, with_metadata,
)?))
})));
}
Err(e) => return super::error_exec(e.to_string()),
}; };
return self.run(super::point_at(&path, &host, args));
}; };
let pool = match m.scope { let pool = match m.scope {
MemScope::User => Arc::clone(&ctx.pool), MemScope::User => Arc::clone(&ctx.pool),
@@ -139,6 +153,35 @@ impl Tool for ListFiles {
} }
} }
/// Renders a container listing into the same JSON the on-disk walk emits: a bare
/// array of relative paths, or `FileEntry` rows under `with_metadata`.
///
/// `line_count` is always absent here. Counting lines means reading the file, and
/// reading a container file means one `docker exec` each — a listing must not
/// quietly become a full read of the tree.
fn render_container_listing(
entries: Vec<crate::container::exec_fs::Entry>,
dirs_only: bool,
with_metadata: bool,
) -> Result<String> {
let mut rows: Vec<crate::container::exec_fs::Entry> = entries
.into_iter()
.filter(|e| if dirs_only { e.is_dir } else { !e.is_dir })
.filter(|e| !e.name.split('/').any(|c| SKIP_DIRS.contains(&c)))
.collect();
rows.sort_by(|a, b| a.name.cmp(&b.name));
if !with_metadata {
let paths: Vec<String> = rows.into_iter().map(|e| e.name).collect();
return Ok(serde_json::to_string(&paths)?);
}
let entries: Vec<FileEntry> = rows
.into_iter()
.map(|e| FileEntry { path: e.name, line_count: None, size: Some(human_size(e.size)) })
.collect();
Ok(serde_json::to_string(&entries)?)
}
/// A `with_metadata` listing row. Field order (declaration order) is the wire /// A `with_metadata` listing row. Field order (declaration order) is the wire
/// order; `line_count` and `size` are omitted when unavailable. /// order; `line_count` and `size` are omitted when unavailable.
#[derive(serde::Serialize)] #[derive(serde::Serialize)]
+241 -8
View File
@@ -248,28 +248,194 @@ pub(crate) fn resolve_host_path(fs: &UserFs, agent_path: &str) -> Result<PathBuf
/// `GET /api/file` and `GET /api/file/watch`; containment (canonicalize + /// `GET /api/file` and `GET /api/file/watch`; containment (canonicalize +
/// prefix-check, symlink-aware) is handled by [`resolve_host_path`]. /// prefix-check, symlink-aware) is handled by [`resolve_host_path`].
pub fn resolve_view_path(fs: &UserFs, input: &str) -> Result<(PathBuf, String)> { pub fn resolve_view_path(fs: &UserFs, input: &str) -> Result<(PathBuf, String)> {
match resolve_view_target(fs, input)? {
(FsTarget::Host(host), agent) => Ok((host, agent)),
(FsTarget::Container { .. }, agent) => anyhow::bail!(
"{agent} lives only inside your container; this action needs a file in your \
mounted folders (~, shared/, projects/)"
),
}
}
/// The view-surface twin of [`resolve_target`]: normalizes an incoming path to the
/// agent vocabulary and says where it lives, so the viewer can open a
/// container-only path (`/tmp/report.pdf`) the same way the agent reads it.
///
/// A container-only path has no agent-vocabulary spelling — it *is* its own
/// display form, which is also what `show_file_to_user` echoes back.
pub fn resolve_view_target(fs: &UserFs, input: &str) -> Result<(FsTarget, String)> {
if classify_memory(input).is_some() { if classify_memory(input).is_some() {
anyhow::bail!("memory notes can't be opened in the file viewer: {input}"); anyhow::bail!("memory notes can't be opened in the file viewer: {input}");
} }
let raw = Path::new(input);
if raw.is_absolute() && fs.container_to_agent(raw).is_none() {
let path = lexical_normalize(raw);
let display = path.to_string_lossy().into_owned();
return Ok((
FsTarget::Container { container: fs.container_name.clone(), path },
display,
));
}
let agent = fs.to_agent_display(input) let agent = fs.to_agent_display(input)
.ok_or_else(|| anyhow::anyhow!("path is outside your workspace: {input}"))?; .ok_or_else(|| anyhow::anyhow!("path is outside your workspace: {input}"))?;
let host = resolve_host_path(fs, &agent)?; let host = resolve_host_path(fs, &agent)?;
Ok((host, agent)) Ok((FsTarget::Host(host), agent))
} }
/// Rewrites the `path` argument of a physical fs-tool call to the resolved absolute /// Points the `path` argument of a physical fs-tool call at the absolute path the
/// host path, so the on-disk `execute` (which takes absolute paths as-is) acts on /// on-disk `execute` should act on — the caller's host workspace, or the shuttled
/// the caller's per-user workspace rather than the process working directory. /// copy of a container file — instead of the process working directory.
/// ///
/// The caller's agent-visible path is stashed under [`DISPLAY_PATH_KEY`] so `execute` /// The caller's agent-visible path is stashed under [`DISPLAY_PATH_KEY`] so `execute`
/// can show it in its messages — the model must never see the host path. This key is /// can show it in its messages — the model must never see the host path. This key is
/// never persisted: tool args are logged from `call.arguments` *before* `run_with` /// never persisted: tool args are logged from `call.arguments` *before* `run_with`
/// rewrites them, and tool results are plain strings. /// rewrites them, and tool results are plain strings.
pub(crate) fn rewrite_to_host(fs: &UserFs, agent_path: &str, mut args: Value) -> Result<Value> { pub(crate) fn point_at(agent_path: &str, abs: &Path, mut args: Value) -> Value {
let host = resolve_host_path(fs, agent_path)?;
args[DISPLAY_PATH_KEY] = Value::String(agent_path.to_string()); args[DISPLAY_PATH_KEY] = Value::String(agent_path.to_string());
args["path"] = Value::String(host.to_string_lossy().into_owned()); args["path"] = Value::String(abs.to_string_lossy().into_owned());
Ok(args) args
}
// ── Container routing ─────────────────────────────────────────────────────────
//
// The security boundary is the **container**, not the bind-mounted subtree. An
// agent already reaches every corner of its container through `execute_cmd`,
// which runs there with passwordless `sudo`; fs-tools that stopped at the mounts
// were not protecting anything, they were showing a poorer view of the same
// sandbox — and the model routinely answered that by shelling out instead.
//
// So a physical path resolves to one of two backings, and the mount is the *fast*
// one rather than the only one. Host containment is untouched: it is what stops a
// symlink planted in the container from resolving against the **host's** `/etc`,
// and it still guards every path that lands on a mount. The container branch
// never touches the host filesystem, so it has no host to escape from.
/// Where a physical (non-memory) agent path actually lives.
pub enum FsTarget {
/// A bind-mounted path: host and container see the same bytes, so the tool
/// acts on the host directly — no `docker exec`, and full media support.
Host(PathBuf),
/// A container-only path (`/tmp`, `/etc`, a package's files…), reachable
/// solely through the container's own filesystem.
Container { container: String, path: PathBuf },
}
/// Resolves a physical agent path to its backing.
///
/// An absolute path is **container vocabulary** — it is what `execute_cmd` prints
/// and what the agent's shell sees — so it is reverse-mapped first. Landing on a
/// mount takes the host path (`/root/x` *is* `~/x`, which the tools used to
/// reject); landing nowhere means the path exists only inside the container.
pub(crate) fn resolve_target(fs: &UserFs, agent_path: &str) -> Result<FsTarget> {
if Path::new(agent_path).is_absolute() {
return match fs.container_to_agent(Path::new(agent_path)) {
Some(mapped) => Ok(FsTarget::Host(resolve_host_path(fs, &mapped)?)),
None => Ok(FsTarget::Container {
container: fs.container_name.clone(),
path: lexical_normalize(Path::new(agent_path)),
}),
};
}
Ok(FsTarget::Host(resolve_host_path(fs, agent_path)?))
}
/// A container file materialised host-side for the duration of one tool call.
///
/// Every single-file fs-tool funnels through the same shape — resolve, then run a
/// sync `execute` that reads and writes one absolute host path. Rather than give
/// each of them a second implementation, with a second set of messages, diffs and
/// edge cases to keep in step, the file is pulled out of the container, the
/// **unchanged** tool runs on the copy, and the copy goes back if it changed.
///
/// A missing remote file is deliberately not pre-created: `write_file` says
/// "Created" or "Overwrote" based on whether the path existed, and a placeholder
/// would make every creation report the wrong one.
pub(crate) struct Shuttle {
dir: PathBuf,
local: PathBuf,
container: String,
remote: PathBuf,
/// The bytes as pulled, or `None` when the remote file did not exist.
/// Compared by content rather than mtime, whose one-second resolution on some
/// filesystems would miss a fast edit.
before: Option<Vec<u8>>,
}
impl Shuttle {
async fn pull(container: &str, remote: &Path) -> Result<Self> {
let dir = std::env::temp_dir().join(format!("skald-fs-{}", uuid::Uuid::new_v4()));
tokio::fs::create_dir_all(&dir).await
.with_context(|| format!("Failed to create temporary directory: {}", dir.display()))?;
// Keep the basename: tools and media sniffing key off the extension.
let name = remote.file_name().unwrap_or_else(|| std::ffi::OsStr::new("file"));
let local = dir.join(name);
let before = if crate::container::exec_fs::exists(container, remote).await {
let bytes = crate::container::exec_fs::read(container, remote).await?;
tokio::fs::write(&local, &bytes).await
.with_context(|| format!("Failed to stage {}", remote.display()))?;
Some(bytes)
} else {
None
};
Ok(Self {
dir,
local,
container: container.to_string(),
remote: remote.to_path_buf(),
before,
})
}
/// Pushes the copy back when the tool created or changed it, then cleans up.
async fn finish(self) -> Result<()> {
let after = tokio::fs::read(&self.local).await.ok();
let changed = match (&self.before, &after) {
(before, Some(a)) => before.as_ref() != Some(a),
(_, None) => false,
};
let pushed = if changed {
crate::container::exec_fs::write(&self.container, &self.remote, after.as_deref().unwrap_or(&[])).await
} else {
Ok(())
};
let _ = tokio::fs::remove_dir_all(&self.dir).await;
pushed
}
}
/// The single entry point a single-file fs-tool uses for a physical path: resolve
/// the backing, then run the tool's own `execute` against it — directly on the
/// host, or on a shuttled copy for a container-only path.
pub(crate) fn run_physical<'a, T>(
tool: &'a T,
fs: &UserFs,
agent_path: &str,
args: Value,
) -> Box<dyn ToolExecution + 'a>
where
T: crate::tools::Tool + ?Sized,
{
match resolve_target(fs, agent_path) {
Err(e) => error_exec(e.to_string()),
Ok(FsTarget::Host(host)) => tool.run(point_at(agent_path, &host, args)),
Ok(FsTarget::Container { container, path }) => {
let display = agent_path.to_string();
Box::new(SimpleExecution::new(Box::pin(async move {
let shuttle = Shuttle::pull(&container, &path).await?;
let args = point_at(&display, &shuttle.local, args);
// The tool's own error wins over a push failure: the push is
// bookkeeping, the tool's message is what the model must read.
let out = tool.execute_typed(args).await;
let pushed = shuttle.finish().await;
match out {
Ok(v) => pushed.map(|()| v),
Err(e) => Err(e),
}
})))
}
}
} }
/// Private stash key for the agent-visible path, set by [`rewrite_to_host`] alongside /// Private stash key for the agent-visible path, set by [`rewrite_to_host`] alongside
@@ -414,6 +580,73 @@ mod tests {
let _ = std::fs::remove_dir_all(&root); let _ = std::fs::remove_dir_all(&root);
} }
/// Container routing: a container-absolute path that names a **mount** takes
/// the host fast path (`/root/x` *is* `~/x` — it used to be rejected as an
/// escape, because the absolute tail replaced the home base on `join`), while
/// one that names nothing mounted resolves inside the container.
#[test]
fn absolute_paths_route_to_the_mount_or_to_the_container() {
use core_api::user_fs::SharedMount;
let root = std::env::temp_dir().join(format!("skald-fstgt-{}", std::process::id()));
let home = root.join("homes").join("u1");
let shared = root.join("shared").join("family");
let _ = std::fs::remove_dir_all(&root);
std::fs::create_dir_all(&home).unwrap();
std::fs::create_dir_all(&shared).unwrap();
let fs = UserFs::new(
"u1",
home.clone(),
"skald-u1",
PathBuf::from("/root"),
vec![SharedMount {
name: "family".into(),
host: shared.clone(),
container: PathBuf::from("/root/shared/family"),
can_write: true,
}],
vec![],
None,
);
let home_canon = canonicalize_for_policy(&home.to_string_lossy(), Path::new("/"));
let shared_canon = canonicalize_for_policy(&shared.to_string_lossy(), Path::new("/"));
let host = |p: &str| match resolve_target(&fs, p).unwrap() {
FsTarget::Host(h) => h,
FsTarget::Container { path, .. } => panic!("{p} routed to the container as {path:?}"),
};
let container = |p: &str| match resolve_target(&fs, p).unwrap() {
FsTarget::Container { container, path } => (container, path),
FsTarget::Host(h) => panic!("{p} routed to the host as {h:?}"),
};
// The container spelling of the home and of a shared mount reach the same
// host files as the agent vocabulary does.
assert_eq!(host("/root/notes.md"), host("~/notes.md"));
assert!(path_under(&host("/root/notes.md"), &home_canon));
assert_eq!(
host("/root/shared/family/list.md"),
host("shared/family/list.md")
);
assert!(path_under(&host("/root/shared/family/list.md"), &shared_canon));
// Nothing mounted there → the container's own filesystem.
let (name, path) = container("/tmp/cv.txt");
assert_eq!(name, "skald-u1");
assert_eq!(path, PathBuf::from("/tmp/cv.txt"));
assert_eq!(container("/etc/os-release").1, PathBuf::from("/etc/os-release"));
// `..` is collapsed before it can name a parent of anything.
assert_eq!(container("/tmp/../tmp/x").1, PathBuf::from("/tmp/x"));
// A shared folder the user does not belong to stays an error — the
// container spelling must not become a way around membership.
assert!(resolve_target(&fs, "/root/shared/secret/x.md").is_err());
let _ = std::fs::remove_dir_all(&root);
}
#[test] #[test]
fn classify_memory_splits_root_from_key() { fn classify_memory_splits_root_from_key() {
let u = classify_memory("user-memory/notes/x.md").unwrap(); let u = classify_memory("user-memory/notes/x.md").unwrap();
+9 -2
View File
@@ -141,8 +141,15 @@ impl Tool for ReadFile {
let Some(m) = classify_memory(&path) else { let Some(m) = classify_memory(&path) else {
// Physical path: resolve + containment-check up front (so an escape // Physical path: resolve + containment-check up front (so an escape
// fails immediately), then read inside the work future. // fails immediately), then read inside the work future.
let host = match super::resolve_host_path(&ctx.fs, &path) { let host = match super::resolve_target(&ctx.fs, &path) {
Ok(h) => h, Ok(super::FsTarget::Host(h)) => h,
// A container-only path has no host file to sniff or to hand on
// as a `MediaRef` (the shuttled copy is gone by the time the
// projection would inline it), so it is read as text — same
// windowing, same line numbers, via the shared `execute`.
Ok(super::FsTarget::Container { .. }) => {
return super::run_physical(self, &ctx.fs, &path, args);
}
Err(e) => return super::error_exec(e.to_string()), Err(e) => return super::error_exec(e.to_string()),
}; };
let start = args["start_line"].as_u64().map(|n| (n as usize).saturating_sub(1)).unwrap_or(0); let start = args["start_line"].as_u64().map(|n| (n as usize).saturating_sub(1)).unwrap_or(0);
@@ -101,10 +101,7 @@ impl Tool for ReplaceLines {
fn run_with<'a>(&'a self, ctx: &ToolContext, args: Value) -> Box<dyn ToolExecution + 'a> { fn run_with<'a>(&'a self, ctx: &ToolContext, args: Value) -> Box<dyn ToolExecution + 'a> {
let path = super::path_arg(&args).unwrap_or_default(); let path = super::path_arg(&args).unwrap_or_default();
let Some(m) = classify_memory(&path) else { let Some(m) = classify_memory(&path) else {
return match super::rewrite_to_host(&ctx.fs, &path, args) { return super::run_physical(self, &ctx.fs, &path, args);
Ok(args) => self.run(args),
Err(e) => super::error_exec(e.to_string()),
};
}; };
let pool = match m.scope { let pool = match m.scope {
MemScope::User => Arc::clone(&ctx.pool), MemScope::User => Arc::clone(&ctx.pool),
@@ -114,10 +114,7 @@ impl Tool for SearchFile {
fn run_with<'a>(&'a self, ctx: &ToolContext, args: Value) -> Box<dyn ToolExecution + 'a> { fn run_with<'a>(&'a self, ctx: &ToolContext, args: Value) -> Box<dyn ToolExecution + 'a> {
let path = super::path_arg(&args).unwrap_or_default(); let path = super::path_arg(&args).unwrap_or_default();
let Some(m) = classify_memory(&path) else { let Some(m) = classify_memory(&path) else {
return match super::rewrite_to_host(&ctx.fs, &path, args) { return super::run_physical(self, &ctx.fs, &path, args);
Ok(args) => self.run(args),
Err(e) => super::error_exec(e.to_string()),
};
}; };
let pool = match m.scope { let pool = match m.scope {
MemScope::User => Arc::clone(&ctx.pool), MemScope::User => Arc::clone(&ctx.pool),
+1 -4
View File
@@ -65,10 +65,7 @@ impl Tool for WriteFile {
fn run_with<'a>(&'a self, ctx: &ToolContext, args: Value) -> Box<dyn ToolExecution + 'a> { fn run_with<'a>(&'a self, ctx: &ToolContext, args: Value) -> Box<dyn ToolExecution + 'a> {
let path = super::path_arg(&args).unwrap_or_default(); let path = super::path_arg(&args).unwrap_or_default();
let Some(m) = classify_memory(&path) else { let Some(m) = classify_memory(&path) else {
return match super::rewrite_to_host(&ctx.fs, &path, args) { return super::run_physical(self, &ctx.fs, &path, args);
Ok(args) => self.run(args),
Err(e) => super::error_exec(e.to_string()),
};
}; };
let pool = match m.scope { let pool = match m.scope {
MemScope::User => Arc::clone(&ctx.pool), MemScope::User => Arc::clone(&ctx.pool),
+12 -3
View File
@@ -112,12 +112,21 @@ pub fn make_tool(
// Resolve against the caller's workspace snapshot: gives the host path to // Resolve against the caller's workspace snapshot: gives the host path to
// stat and the canonical agent path the viewer will fetch back. // stat and the canonical agent path the viewer will fetch back.
let user_fs = fs.load(); let user_fs = fs.load();
let (abs, display) = fs::resolve_view_path(user_fs.as_ref(), path) let (target, display) = fs::resolve_view_target(user_fs.as_ref(), path)
.map_err(|e| anyhow::anyhow!("show_file_to_user: {e}"))?; .map_err(|e| anyhow::anyhow!("show_file_to_user: {e}"))?;
if !abs.exists() { // A container-only path is statted through the container, the same way
// the viewer will fetch it back.
let (exists, is_dir) = match &target {
fs::FsTarget::Host(abs) => (abs.exists(), abs.is_dir()),
fs::FsTarget::Container { container, path } => (
crate::container::exec_fs::exists(container, path).await,
crate::container::exec_fs::is_dir(container, path).await,
),
};
if !exists {
anyhow::bail!("show_file_to_user: file not found: {display}"); anyhow::bail!("show_file_to_user: file not found: {display}");
} }
if abs.is_dir() { if is_dir {
anyhow::bail!("show_file_to_user: '{display}' is a directory, not a file"); anyhow::bail!("show_file_to_user: '{display}' is a directory, not a file");
} }
+29 -3
View File
@@ -198,12 +198,38 @@ pub async fn get_file(
} }
let user_fs = ctx.fs.load(); let user_fs = ctx.fs.load();
let (abs, agent) = match fs_tools::resolve_view_path(user_fs.as_ref(), &q.path) { let (target, agent) = match fs_tools::resolve_view_target(user_fs.as_ref(), &q.path) {
Ok((abs, agent)) => (abs, agent), Ok(resolved) => resolved,
Err(e) => return (StatusCode::BAD_REQUEST, format!("Invalid path: {e}")).into_response(), Err(e) => return (StatusCode::BAD_REQUEST, format!("Invalid path: {e}")).into_response(),
}; };
let writable = user_fs.can_write_to(&agent); let writable = user_fs.can_write_to(&agent);
// A container-only path (`/tmp/…`) has no host file behind it: the bytes come
// out through the container, so the user sees what the agent read. Served
// read-only — the editor's optimistic locking is an on-disk `mtime`+`len`,
// which has no counterpart here, and without an ETag the frontend keeps the
// file in view mode rather than risking a blind overwrite.
let abs = match target {
fs_tools::FsTarget::Host(abs) => abs,
fs_tools::FsTarget::Container { container, path } => {
return match skald_core::container::exec_fs::read(&container, &path).await {
Ok(bytes) => {
let mut response = bytes.into_response();
response.headers_mut().insert(
header::CONTENT_TYPE,
HeaderValue::from_static(content_type_for(&q.path)),
);
if q.force_download {
set_attachment(&mut response, &basename(&q.path));
}
response
}
Err(_) => (StatusCode::NOT_FOUND, format!("File not found: {}", q.path))
.into_response(),
};
}
};
if q.compile_latex && is_latex(&q.path) { if q.compile_latex && is_latex(&q.path) {
return match state.latex_compiler().compile(&abs).await { return match state.latex_compiler().compile(&abs).await {
Ok(pdf) => { Ok(pdf) => {