diff --git a/CLAUDE.md b/CLAUDE.md index 0cd305a..1901d87 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -69,7 +69,7 @@ Two rules keep the boundary real, and both are enforced by the compiler: | `crates/skald-core/src/chat_hub/` | `ChatHub`: broadcast events to all connected WS clients | | `crates/skald-core/src/chat_event_bus.rs` | Global async bus for cross-session events | | `crates/skald-core/src/agents.rs` | Discovers agents from `agents/*/`, loads meta + system prompt | -| `crates/skald-core/src/tools/` | Built-in tools: `exec`, `restart`, `list_agents`, `fs/*`, `notify`, `ast_outline`, `image_generate`, MCP tools, plugin tools, cron tools | +| `crates/skald-core/src/tools/` | Built-in tools: `exec`, `restart`, `list_agents`, `fs/*` (also route `user-memory/`/`shared-memory/` to `memory_docs` — see DB tables), `notify`, `ast_outline`, `image_generate`, MCP tools, plugin tools, cron tools | | `crates/skald-core/src/tool_catalog.rs` | `ToolCatalog`: unified tool listing façade (wraps ToolRegistry + McpManager) | | `crates/skald-core/src/events.rs` | `ServerEvent` enum streamed over WebSocket to the frontend | | `crates/skald-core/src/db/` | sqlx SQLite — see below | @@ -100,10 +100,14 @@ Two rules keep the boundary real, and both are enforced by the compiler: The schema is split into two buckets (§5.1), and the split is the point: - **`create_registry_tables`** — instance-wide, readable without any user key: `users`, `llm_providers`, `llm_models`, `transcribe_models`, `tts_models`, `image_generate_models`, `plugins`, `approval_rules`, `tool_permission_groups`, `config`, `known_tools`, `llm_requests`. -- **`create_owner_tables`** — one owner's content, **identical schema in every file that has it**: `chat_sessions`, `chat_sessions_stack`, `chat_history`, `chat_llm_tools`, `chat_summaries`, `session_scratchpad`, `session_mcp_grants`, `stack_mcp_grants`, `scheduled_jobs`, `job_runs`, `mcp_servers`, `mcp_events`, `sources`, `secrets`, `projects`, `project_tickets`. +- **`create_owner_tables`** — one owner's content, **identical schema in every file that has it**: `chat_sessions`, `chat_sessions_stack`, `chat_history`, `chat_llm_tools`, `chat_summaries`, `session_scratchpad`, `session_mcp_grants`, `stack_mcp_grants`, `scheduled_jobs`, `job_runs`, `mcp_servers`, `mcp_events`, `sources`, `secrets`, `projects`, `project_tickets`, `llm_request_payloads`, `memory_docs` (+ FTS5 `memory_docs_fts`). Because `memory_docs` is an owner table, one definition backs **private** memory in each `{userid}.db` and **shared** memory in `system.db` (the household owner) — see the memory namespace note below. **No foreign key in the owner bucket may point at a registry table.** SQLite cannot enforce a key across files, not even through `ATTACH`, and sqlx turns on `PRAGMA foreign_keys`: the `CREATE TABLE` succeeds and every `INSERT` fails. `db::tests::owner_tables_stand_alone_with_foreign_keys_on` enforces this by running the owner schema against a database holding nothing else, then inserting a row into each table. Two keys crossed and were fixed: `chat_history.model_db_id` (dropped — write-only, and `llm_requests.model_name` already records the model) and `project_tickets.job_id` (fixed by moving `projects`/`project_tickets` into the owner bucket). +**Memory namespace (blueprint §5).** `memory_docs` (accessor `db/memory_docs.rs` — `get`/`upsert`/`list`/`search`(FTS)/`delete`) backs a virtual note store surfaced through the fs-tools, **not** the disk. Two sibling roots (not the blueprint's nested `memory/{userid}` + `memory/shared`): `user-memory/…` routes to the caller's own pool (`ToolContext::pool`), `shared-memory/…` to the system pool (a singleton captured in `fs::register_all`). `tools/fs/classify_memory()` decides on the raw first path component (a `..` in the tail clamps inside the store, never escapes to disk); `read_file`/`write_file`/`list_files`/`edit_file`/`insert_at_line`/`replace_lines`/`search_file` override `run_with` to route memory paths (each extracting a pure transform shared with its on-disk `execute`) and leave every other path on disk. Approval (seeded in `seed_fs_path_rules`): `user-memory/*` is `@fs_any allow` (private, frictionless); `shared-memory/*` is `@fs_read allow` + `@fs_write require` — reads free, **writes need approval** so the agent can't silently push one person's data into shared memory. `grep_files` stays disk-only (regex-across-tree ≠ FTS); ranked full-text recall over notes is a separate tool, `memory_search` (`tools/fs/memory_search.rs`), over the `memory_docs` FTS index — allowed by a path-less rule (it takes `query`, not `path`). + +**Memory injection into the prompt**: `MessageBuilder::load_inject_memory` routes each `meta.inject_memory` entry — `user-memory/…` → owner pool, `shared-memory/…` → the shared (`system.db`) pool, both via `memory_docs::get`; anything else (`data/…`, `$WD/…`) is a disk read. The shared pool is threaded `ChatSessionManager` → handler → `MessageBuilder`. `main` and `project-coordinator` inject `user-memory/index.md` + `shared-memory/index.md`. + `system.db` currently gets **both** bucket functions, because nothing has migrated to per-user pools yet. That is transitional. `users` (`crates/skald-core/src/db/users.rs`) holds the directory plus auth material. It lives in the system DB, which the box owner can read, so it must never store anything that derives a user's key. `Credentials` is an enum mirroring the table's `CHECK`: an encrypted user carries a **wrapped DEK** (whose AEAD tag *is* the password verifier — hence no `password_hash`); a cleartext user carries an ordinary verifier, or none. `User` is deliberately not `Serialize` and its `Debug` redacts key material — use `User::summary()` for anything leaving the process. `role_id` has no foreign key yet: sqlx enables `PRAGMA foreign_keys`, so referencing the not-yet-existing `roles` table would fail every insert. diff --git a/agents/common/memory.md b/agents/common/memory.md index 57a05d3..3684e40 100644 --- a/agents/common/memory.md +++ b/agents/common/memory.md @@ -1,20 +1,41 @@ # Persistent memory -All memory lives in `data/memory/`. Entry point: `data/memory/index.md` — one line per file with a brief summary. +You have two persistent note stores, kept as Markdown and searchable. **Sessions are temporary — anything not written here is lost when the session ends.** Save proactively. + +- **`user-memory/`** — your **private** memory for this user. Nobody else can read it. Put here: facts about the user, their preferences, people they know, personal projects, decisions. +- **`shared-memory/`** — memory **shared with the whole group**. Every member can read it. Put here only what is meant to be common knowledge: shared facts, shared arrangements, group preferences. **Never** put one person's private information here. Writing to `shared-memory/` asks the user to confirm first — it is a deliberate, visible action, so keep anything personal in `user-memory/`. + +When unsure where something belongs, prefer `user-memory/`. + +## The indexes + +Each store has an `index.md` — one line per note with a brief summary — and **both are injected into your context automatically** at the start of each session (look for them below): + +- `user-memory/index.md` — your private notes. +- `shared-memory/index.md` — the group's shared notes. + +Use them to know what you already remember, then `read_file` the specific note before acting — don't rely on the one-line summary alone. **Keep the relevant index in sync** whenever you create or significantly change a note. Updating `shared-memory/index.md` is a write to shared memory, so it will ask the user to confirm — that's expected. ## When to save Save **immediately** (do not postpone) when: - The user shares a new fact about themselves, a project, a person, or a preference -- A decision is made that may be relevant in future sessions -- You notice an inconsistency with what was previously saved → correct it +- A decision is made that may matter in a future session +- You notice that something you saved before is now wrong → correct it ## When to read -At the start of each session, read `data/memory/index.md` silently. Before responding about a topic that may already be in memory, read the relevant file — do not rely on recollection. +Before responding about a topic that may already be in memory, look it up — do not rely on recollection: -## File format +- The injected `user-memory/index.md` tells you what exists; `read_file` the note it points to. +- `memory_search ""` — full-text search across both stores, ranked by relevance, when you don't know which note holds something. + +## Organising notes + +Use clear, topic-based paths — e.g. `user-memory/people/alice.md`, `user-memory/projects/website.md`, `shared-memory/wifi.md`. Keep one topic per note. + +## Note format ```md # Title @@ -28,8 +49,7 @@ _Updated: YYYY-MM-DD_ ## How to update -1. `read_file` to get the exact current content -2. `edit_file` to modify — always keep the `_Updated: YYYY-MM-DD_` date in sync -3. Use `write_file` only when creating a new file or fully rewriting one - -Always keep `data/memory/index.md` in sync when you create or significantly update a file. +1. `read_file` the note to get its exact current content. +2. `edit_file` to change part of it — keep the `_Updated:_` date in sync. +3. Use `write_file` only to create a new note or fully rewrite one. +4. Keep `user-memory/index.md` in sync when you add or significantly change a note. diff --git a/agents/main/AGENT.md b/agents/main/AGENT.md index cf39e96..4cc90a6 100644 --- a/agents/main/AGENT.md +++ b/agents/main/AGENT.md @@ -2,11 +2,9 @@ You are an extremely powerful general-purpose personal assistant. You help the user with any task — research, writing, planning, analysis, coding, or anything else they bring to you. -Your personality and tone are defined in `data/memory/SOUL.md`. If the file exists, it is automatically injected into your system context — look for it at the end of this prompt. - Think outside the box: you can use tools, write and execute Python scripts on the fly, or even modify your own source code. -The `data/` directory (inside your working directory) is your own space — write there freely; you have permission to create and modify anything under it. **Default to `data/` for everything you produce**: generated files, notes, one-shot scripts, downloads, and persistent memory (e.g. `data/memory/`, `data/notifications.md`). When a path is relative, prefix it with `data/` — a bare filename lands in the project root, which is not where your working files belong. Write **outside** `data/` (the project root, `src/`, `web/`, `agents/`, config, …) only when a specific, well-defined goal genuinely requires it and cannot be accomplished within `data/`. +The `data/` directory (inside your working directory) is your own space — write there freely; you have permission to create and modify anything under it. **Default to `data/` for everything you produce**: generated files, notes, one-shot scripts, downloads. When a path is relative, prefix it with `data/` — a bare filename lands in the project root, which is not where your working files belong. (Persistent **memory** is separate: durable facts go to `user-memory/` or `shared-memory/`, not under `data/` — see the Memory section.) Write **outside** `data/` (the project root, `src/`, `web/`, `agents/`, config, …) only when a specific, well-defined goal genuinely requires it and cannot be accomplished within `data/`. You have access to tools, persistent memory system and sub agents. Use both proactively. Sub agents also help to keep your context windows small and concise. @@ -113,7 +111,7 @@ Configuration tools are hidden by default to keep context small. Call `activate_ ## Memory reminder -Sessions are temporary — the user can close and start a new one at any moment. **Context alone is not enough.** If something is worth remembering, write it to a file in `data/memory/` immediately. If it stays only in context, it is gone forever when the session ends. +Sessions are temporary — the user can close and start a new one at any moment. **Context alone is not enough.** If something is worth remembering, save it to `user-memory/` immediately (or `shared-memory/` if it's meant for the whole group). If it stays only in context, it is gone forever when the session ends. --- diff --git a/agents/main/meta.json b/agents/main/meta.json index d076c95..8eac99b 100644 --- a/agents/main/meta.json +++ b/agents/main/meta.json @@ -1,9 +1,9 @@ { "name": "Main Assistant", - "description": "General-purpose assistant: helps the user with any task using tools, and persists all relevant information in data/memory", + "description": "General-purpose assistant: helps the user with any task using tools, and persists all relevant information in memory", "friendly_description": "Your general-purpose assistant — helps with any task and remembers what matters in memory.", "type": "chat", - "inject_memory": ["data/memory/index.md", "data/memory/SOUL.md"], + "inject_memory": ["user-memory/index.md", "shared-memory/index.md"], "icon": "icon.png", "strength": "average" } diff --git a/agents/project-coordinator/AGENT.md b/agents/project-coordinator/AGENT.md index 312dd50..30805e9 100644 --- a/agents/project-coordinator/AGENT.md +++ b/agents/project-coordinator/AGENT.md @@ -29,7 +29,7 @@ Delegate work to these task specialists via `execute_task` / `execute_subtask`: Your system prompt already contains, without you asking: - The project's **name**, **description**, and **working directory** (the project root — all relative file paths resolve there). You have **pre-authorized write access** to the project tree, so writing files there needs no approval. -- **`data/memory/index.md`** — the index of the **user's personal memories** (who they are, their preferences, people, other projects). It is injected automatically. Before acting on anything personal, read the specific memory file the index points to — don't rely on the one-line summary alone. +- **`user-memory/index.md`** and **`shared-memory/index.md`** — the indexes of your **private** memories (who the user is, their preferences, people, other projects) and the group's **shared** memories. Both are injected automatically. Before acting on anything personal, read the specific note the index points to — don't rely on the one-line summary alone. - **`SKALD.md`** at the project root — this project's **living diary** (see below). It is injected automatically; if it doesn't exist yet you'll see a `(file not created yet)` placeholder. Treat all of this as ground truth. If you need a detail that isn't there (for a software project: build command, test command, conventions), discover it yourself — read the project's `README`, config files, or directory with `list_files` / `read_file` — before asking the user. diff --git a/agents/project-coordinator/meta.json b/agents/project-coordinator/meta.json index 4d9faad..6afa0ff 100644 --- a/agents/project-coordinator/meta.json +++ b/agents/project-coordinator/meta.json @@ -5,6 +5,6 @@ "type": "chat", "scope": "reasoning", "strength": "average", - "inject_memory": ["data/memory/index.md", "$WD/SKALD.md"], + "inject_memory": ["user-memory/index.md", "shared-memory/index.md", "$WD/SKALD.md"], "icon": "icon.png" } diff --git a/crates/skald-core/src/approval/mod.rs b/crates/skald-core/src/approval/mod.rs index cdd39b8..20dc7fa 100644 --- a/crates/skald-core/src/approval/mod.rs +++ b/crates/skald-core/src/approval/mod.rs @@ -29,11 +29,12 @@ //! determines the action; if none matches the tool requires approval //! (default-closed). //! -//! ## Hardcoded exception +//! ## Memory namespace //! -//! File-write tools targeting `memory/` paths bypass the rule engine and are -//! always allowed (this mirrors the original behaviour and can be replaced by -//! an explicit `allow` rule later). +//! The virtual memory roots `user-memory/` and `shared-memory/` (blueprint §5) are +//! allowed by **seeded rules**, not a hardcoded bypass: `seed_fs_path_rules` stamps +//! `@fs_any allow user-memory/*` and `@fs_any allow shared-memory/*`, editable in the +//! File System panel like any other path rule. use std::collections::HashMap; use std::sync::Arc; @@ -327,31 +328,52 @@ impl ApprovalManager { /// Priority `5` places these path-scoped rules *before* the `*` catch-all (999999), /// so an unmatched path still falls through to the default `require`. /// - /// - `memory/*` → **allow** (the LLM manages its own memory; replaces the former - /// hardcoded `is_memory_path` bypass). + /// - `user-memory/*` → **allow** (the caller's private memory namespace, blueprint §5; + /// the LLM manages its own memory). Reads *and* writes: `@fs_any`. + /// - `shared-memory/*` → reads **allow** (`@fs_read`), writes **require** (`@fs_write`): + /// shared memory is visible to everyone, so a write is a deliberate, human-confirmed + /// act — the agent must not silently push one person's information into it. /// - `data/*` → **allow** (scratch/data workspace). /// - `secrets/*` → **deny** (`@fs_any` denies reads *and* writes; a read would leak /// the secret into the LLM context / history / WS stream, and `Deny` is /// non-bypassable). The `/*` pattern also matches the `secrets` dir node itself, so /// recursive `list_files`/`grep_files` rooted at it are covered. + /// - `memory_search` → **allow**, path-less: it searches note *content* (arg `query`, + /// not `path`), so it needs a tool-scoped rule rather than a path pattern. pub async fn seed_fs_path_rules(&self) -> Result<()> { - // (tool_pattern, path_pattern, action, note) - let rules: &[(&str, &str, &str, &str)] = &[ - ("@fs_any", "memory/*", "allow", "auto-allow memory/"), - ("@fs_any", "data/*", "allow", "auto-allow data/"), - ("@fs_any", "secrets/*", "deny", "deny secrets/ access"), + // (tool_pattern, path_pattern, action, note). `path_pattern = None` is a + // tool-scoped rule that matches regardless of args. + let rules: &[(&str, Option<&str>, &str, &str)] = &[ + ("@fs_any", Some("user-memory/*"), "allow", "auto-allow user-memory/"), + ("@fs_read", Some("shared-memory/*"), "allow", "auto-allow read shared-memory/"), + ("@fs_write", Some("shared-memory/*"), "require", "require write shared-memory/"), + ("@fs_any", Some("data/*"), "allow", "auto-allow data/"), + ("@fs_any", Some("secrets/*"), "deny", "deny secrets/ access"), + ("memory_search", None, "allow", "allow memory_search"), ]; let mut seeded = 0; - for (tool_pattern, path_pattern, action, note) in rules { - let exists: i64 = sqlx::query_scalar( - "SELECT COUNT(*) FROM approval_rules - WHERE tool_pattern = ? AND path_pattern = ? AND group_id = 'default'", - ) - .bind(tool_pattern) - .bind(path_pattern) - .fetch_one(self.db.as_ref()) - .await?; + for &(tool_pattern, path_pattern, action, note) in rules { + // A NULL path can't be matched with `=` (NULL comparisons are never true), + // so the existence check branches on it — otherwise the row would re-insert + // on every boot. + let exists: i64 = match path_pattern { + Some(p) => sqlx::query_scalar( + "SELECT COUNT(*) FROM approval_rules + WHERE tool_pattern = ? AND path_pattern = ? AND group_id = 'default'", + ) + .bind(tool_pattern) + .bind(p) + .fetch_one(self.db.as_ref()) + .await?, + None => sqlx::query_scalar( + "SELECT COUNT(*) FROM approval_rules + WHERE tool_pattern = ? AND path_pattern IS NULL AND group_id = 'default'", + ) + .bind(tool_pattern) + .fetch_one(self.db.as_ref()) + .await?, + }; if exists > 0 { continue; } @@ -360,7 +382,7 @@ impl ApprovalManager { VALUES (?, ?, ?, ?, 5, 'default')", ) .bind(tool_pattern) - .bind(path_pattern) + .bind(path_pattern) // Option<&str> → NULL when None .bind(action) .bind(note) .execute(self.db.as_ref()) @@ -383,7 +405,10 @@ impl ApprovalManager { /// - the per-tool write `require` defaults (`note = 'default rule'`, no path) — fs /// gating now lives in the File System panel + the `*` catch-all; /// - the old `data/*` allow rows (`note = 'auto-allow data/ writes'`); - /// - the old `secrets` deny rows (`note = 'deny reading secrets/'`). + /// - the old `secrets` deny rows (`note = 'deny reading secrets/'`); + /// - the old single `memory/*` allow row (`note = 'auto-allow memory/'`) — the memory + /// namespace split into `user-memory/` + `shared-memory/`, so the `memory/*` pattern + /// no longer routes and would otherwise linger as a stale allow on a disk `./memory/`. /// /// Idempotent: a no-op once the legacy rows are gone. Run before `seed_fs_path_rules`. pub async fn migrate_legacy_fs_rules(&self) -> Result<()> { @@ -406,7 +431,20 @@ impl ApprovalManager { .await? .rows_affected(); - let total = n1 + n2 + n3; + let n4 = sqlx::query("DELETE FROM approval_rules WHERE note = 'auto-allow memory/'") + .execute(self.db.as_ref()) + .await? + .rows_affected(); + + // An earlier build seeded `@fs_any allow shared-memory/*`; shared writes now + // require approval, so that blanket allow must go (the new @fs_read/@fs_write + // rows are seeded fresh by `seed_fs_path_rules`). + let n5 = sqlx::query("DELETE FROM approval_rules WHERE note = 'auto-allow shared-memory/'") + .execute(self.db.as_ref()) + .await? + .rows_affected(); + + let total = n1 + n2 + n3 + n4 + n5; if total > 0 { info!("approval_rules: migrated {total} legacy filesystem rules to @fs_* File System panel"); } @@ -419,8 +457,8 @@ impl ApprovalManager { /// /// Evaluation order: /// 1. Rules for `group_id` first, then "default" group as fallback, sorted by priority ASC. - /// First match wins. (`memory/` auto-allow is a seeded `@fs_any allow memory/*` rule, - /// not a hardcoded exception — see `seed_fs_path_rules`.) + /// First match wins. (The `user-memory/` and `shared-memory/` auto-allows are seeded + /// `@fs_any allow …/*` rules, not a hardcoded exception — see `seed_fs_path_rules`.) /// 2. Session bypass: if a matching bypass is active, `Require` → `Allow`. /// `Deny` is never bypassed. /// 3. No match → `Require` (default-closed policy). @@ -1119,21 +1157,31 @@ mod tests { .unwrap(); assert_eq!(legacy, 0, "legacy fs rules should be removed by migration"); - // …and replaced by exactly the three @fs_* token rows. + // …and replaced by exactly the five @fs_* token rows (shared-memory has two: + // read-allow and write-require). let fs_rows: i64 = sqlx::query_scalar( "SELECT COUNT(*) FROM approval_rules WHERE tool_pattern LIKE '@fs%'", ) .fetch_one(db.as_ref()) .await .unwrap(); - assert_eq!(fs_rows, 3, "memory/data/secrets @fs_* rules should be seeded"); + assert_eq!(fs_rows, 5, "user-memory + shared-memory(r/w) + data + secrets @fs_* rules should be seeded"); // Gate decisions through the real check() path. async fn decide(mgr: &ApprovalManager, tool: &str, path: &str) -> GateResult { mgr.check(1, None, "main", "web", tool, &json!({ "path": path }), Some("default")).await } - assert!(matches!(decide(&mgr, "write_file", "memory/notes.md").await, GateResult::Allow)); - assert!(matches!(decide(&mgr, "read_file", "data/x.txt").await, GateResult::Allow)); + // user-memory auto-allows reads and writes; the old `memory/*` no longer matches. + assert!(matches!(decide(&mgr, "write_file", "user-memory/notes.md").await, GateResult::Allow)); + assert!(matches!(decide(&mgr, "list_files", "user-memory").await, GateResult::Allow)); + assert!(matches!(decide(&mgr, "write_file", "memory/notes.md").await, GateResult::Require)); + // shared-memory: reads allowed, writes require approval. + assert!(matches!(decide(&mgr, "read_file", "shared-memory/casa.md").await, GateResult::Allow)); + assert!(matches!(decide(&mgr, "write_file", "shared-memory/casa.md").await, GateResult::Require)); + assert!(matches!(decide(&mgr, "edit_file", "shared-memory/casa.md").await, GateResult::Require)); + assert!(matches!(decide(&mgr, "read_file", "data/x.txt").await, GateResult::Allow)); + // memory_search is allowed by a path-less tool rule (it has `query`, not `path`). + assert!(matches!(decide(&mgr, "memory_search", "ignored").await, GateResult::Allow)); // Improvement over legacy: secrets *writes* are now denied too, not just reads. assert!(matches!(decide(&mgr, "write_file", "secrets/key").await, GateResult::Deny)); assert!(matches!(decide(&mgr, "read_file", "secrets/key").await, GateResult::Deny)); diff --git a/crates/skald-core/src/db/memory_docs.rs b/crates/skald-core/src/db/memory_docs.rs new file mode 100644 index 0000000..662752f --- /dev/null +++ b/crates/skald-core/src/db/memory_docs.rs @@ -0,0 +1,203 @@ +//! Accessor for `memory_docs` — the backing store of the virtual `memory/` +//! namespace (blueprint §5). +//! +//! The **pool is the namespace**: a user pool holds that user's private notes +//! (`memory/{userid}`), the system pool holds shared notes (`memory/shared`). +//! Callers pass a `path` already stripped of the `memory/…` prefix — the file +//! it lands in decides the namespace, the row keeps only the tail. `path` is +//! UNIQUE, so [`upsert`] is the single write path for both create and edit, and +//! the `memory_docs_fts` triggers keep the full-text index in step underneath. + +use anyhow::Result; +use sqlx::SqlitePool; + +#[derive(Debug, Clone, sqlx::FromRow)] +pub struct MemoryDoc { + pub id: i64, + pub path: String, + pub content: String, + pub created_at: String, + pub updated_at: String, +} + +/// One row of a directory-style listing: metadata only, no `content` body. +#[derive(Debug, Clone, sqlx::FromRow)] +pub struct MemoryEntry { + pub path: String, + pub updated_at: String, +} + +/// One full-text hit: the matching note's path and a highlighted excerpt. +#[derive(Debug, Clone, sqlx::FromRow)] +pub struct MemoryHit { + pub path: String, + pub snippet: String, +} + +const SELECT: &str = "SELECT id, path, content, created_at, updated_at FROM memory_docs"; + +/// Fetch one note by its exact path. +pub async fn get(pool: &SqlitePool, path: &str) -> Result> { + let row = sqlx::query_as::<_, MemoryDoc>(sqlx::AssertSqlSafe(format!("{SELECT} WHERE path = ?"))) + .bind(path) + .fetch_optional(pool) + .await?; + Ok(row) +} + +/// Create the note at `path`, or overwrite it if it already exists. `created_at` +/// survives an overwrite; `updated_at` is bumped. Returns the stored row. +pub async fn upsert(pool: &SqlitePool, path: &str, content: &str) -> Result { + sqlx::query( + "INSERT INTO memory_docs (path, content) + VALUES (?, ?) + ON CONFLICT(path) DO UPDATE SET + content = excluded.content, + updated_at = datetime('now')", + ) + .bind(path) + .bind(content) + .execute(pool) + .await?; + + let row = sqlx::query_as::<_, MemoryDoc>(sqlx::AssertSqlSafe(format!("{SELECT} WHERE path = ?"))) + .bind(path) + .fetch_one(pool) + .await?; + Ok(row) +} + +/// List notes whose path starts with `prefix` (pass `""` for all), most recently +/// edited first. Metadata only — the `content` body is not loaded. +pub async fn list(pool: &SqlitePool, prefix: &str) -> Result> { + // Escaped LIKE prefix: a literal `%`/`_` in the caller's path must match as + // itself, not as a wildcard. `\` is the escape character. + let pattern = format!("{}%", escape_like(prefix)); + let rows = sqlx::query_as::<_, MemoryEntry>( + "SELECT path, updated_at FROM memory_docs + WHERE path LIKE ? ESCAPE '\\' + ORDER BY updated_at DESC", + ) + .bind(pattern) + .fetch_all(pool) + .await?; + Ok(rows) +} + +/// Full-text search over note bodies and paths, best match first. `query` is +/// FTS5 MATCH syntax; `snippet` is a short excerpt of the body with the matched +/// terms wrapped in `[` … `]`. +pub async fn search(pool: &SqlitePool, query: &str, limit: i64) -> Result> { + let rows = sqlx::query_as::<_, MemoryHit>( + // `memory_docs_fts` is external-content, so its rowid is `memory_docs.id`; + // join back for the path, and read the excerpt from content column 1. + "SELECT d.path AS path, + snippet(memory_docs_fts, 1, '[', ']', '…', 12) AS snippet + FROM memory_docs_fts + JOIN memory_docs d ON d.id = memory_docs_fts.rowid + WHERE memory_docs_fts MATCH ? + ORDER BY bm25(memory_docs_fts) + LIMIT ?", + ) + .bind(query) + .bind(limit) + .fetch_all(pool) + .await?; + Ok(rows) +} + +/// Delete the note at `path`. Returns whether a row was removed. +pub async fn delete(pool: &SqlitePool, path: &str) -> Result { + let n = sqlx::query("DELETE FROM memory_docs WHERE path = ?") + .bind(path) + .execute(pool) + .await? + .rows_affected(); + Ok(n > 0) +} + +/// Escapes `%`, `_` and `\` so a caller-supplied string is a literal LIKE prefix. +fn escape_like(s: &str) -> String { + let mut out = String::with_capacity(s.len()); + for c in s.chars() { + if matches!(c, '%' | '_' | '\\') { + out.push('\\'); + } + out.push(c); + } + out +} + +#[cfg(test)] +mod tests { + use super::*; + use std::path::PathBuf; + + /// A standalone owner-schema database in a throwaway temp dir. `tag` plus an + /// atomic counter keep parallel tests from colliding on the same file. + /// Returns the pool and the dir so the caller can wipe it (SQLite leaves + /// `-wal`/`-shm` sidecars beside the file). + async fn owner_pool(tag: &str) -> (SqlitePool, PathBuf) { + use std::sync::atomic::{AtomicU64, Ordering}; + static SEQ: AtomicU64 = AtomicU64::new(0); + let n = SEQ.fetch_add(1, Ordering::Relaxed); + let dir = std::env::temp_dir() + .join(format!("skald-memdocs-{}-{tag}-{n}", std::process::id())); + let _ = std::fs::remove_dir_all(&dir); + std::fs::create_dir_all(&dir).unwrap(); + let pool = crate::db::create_user_pool(&dir.join("owner.db"), None).await.unwrap(); + (pool, dir) + } + + #[tokio::test] + async fn upsert_is_create_then_overwrite_and_fts_follows() { + let (pool, dir) = owner_pool("upsert").await; + + // create + let doc = upsert(&pool, "notes/spesa.md", "latte e pane").await.unwrap(); + assert_eq!(doc.path, "notes/spesa.md"); + assert_eq!(doc.content, "latte e pane"); + let first_id = doc.id; + + // get by exact path + assert_eq!(get(&pool, "notes/spesa.md").await.unwrap().unwrap().content, "latte e pane"); + assert!(get(&pool, "notes/altro.md").await.unwrap().is_none()); + + // overwrite: same row, new content, created_at preserved + let doc2 = upsert(&pool, "notes/spesa.md", "latte, pane, uova").await.unwrap(); + assert_eq!(doc2.id, first_id, "upsert must update in place, not insert a new row"); + assert_eq!(doc2.content, "latte, pane, uova"); + assert_eq!(doc2.created_at, doc.created_at, "created_at survives an overwrite"); + assert_eq!(list(&pool, "").await.unwrap().len(), 1, "still one row for that path"); + + // FTS follows the update: the removed token is gone, the new one is found + assert!(search(&pool, "uova", 10).await.unwrap().iter().any(|h| h.path == "notes/spesa.md")); + assert!(search(&pool, "latte", 10).await.unwrap().iter().any(|h| h.path == "notes/spesa.md")); + + pool.close().await; + let _ = std::fs::remove_dir_all(&dir); + } + + #[tokio::test] + async fn list_by_prefix_and_delete_deindexes() { + let (pool, dir) = owner_pool("list").await; + + upsert(&pool, "notes/spesa.md", "latte pane uova").await.unwrap(); + upsert(&pool, "notes/idee.md", "un'idea brillante").await.unwrap(); + upsert(&pool, "diary/2026.md", "oggi e' successo").await.unwrap(); + + let notes = list(&pool, "notes/").await.unwrap(); + assert_eq!(notes.len(), 2, "prefix listing is scoped to the subtree"); + assert!(notes.iter().all(|e| e.path.starts_with("notes/"))); + assert_eq!(list(&pool, "").await.unwrap().len(), 3, "empty prefix lists everything"); + + // delete removes the row and de-indexes it from FTS + assert!(delete(&pool, "notes/spesa.md").await.unwrap()); + assert!(get(&pool, "notes/spesa.md").await.unwrap().is_none()); + assert!(search(&pool, "uova", 10).await.unwrap().is_empty(), "delete must de-index"); + assert!(!delete(&pool, "notes/spesa.md").await.unwrap(), "second delete is a no-op"); + + pool.close().await; + let _ = std::fs::remove_dir_all(&dir); + } +} diff --git a/crates/skald-core/src/db/mod.rs b/crates/skald-core/src/db/mod.rs index 3d59212..87a434e 100644 --- a/crates/skald-core/src/db/mod.rs +++ b/crates/skald-core/src/db/mod.rs @@ -13,6 +13,7 @@ pub mod llm_requests; pub mod llm_request_payloads; pub mod mcp_events; pub mod mcp_servers; +pub mod memory_docs; pub mod plugins; pub mod roles; pub mod scheduled_jobs; @@ -710,6 +711,60 @@ pub async fn create_owner_tables(pool: &SqlitePool) -> Result<()> { .execute(pool) .await?; + // Backing store for the virtual `memory/` namespace (blueprint §5). MD-only + // notes keyed by a path *relative to the namespace root* — the file is the + // namespace, so no `memory/{userid}` / `memory/shared` prefix is stored: + // routing picks the pool, the row keeps only the tail. Because this is an + // owner table, the same schema backs private memory in each `{userid}.db` + // (behind SQLCipher) and shared memory in `system.db` (cleartext, the + // household owner) — §5.1. `path` is UNIQUE, so a write is an upsert. + sqlx::query( + "CREATE TABLE IF NOT EXISTS memory_docs ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + path TEXT NOT NULL UNIQUE, + content TEXT NOT NULL DEFAULT '', + created_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_at TEXT NOT NULL DEFAULT (datetime('now')) + )", + ) + .execute(pool) + .await?; + + // Full-text index over memory notes: the payoff of a SQLite backing over + // opaque file blobs (§5) — a decrypted session can search / RAG its own + // memory. External-content FTS5 keeps no second copy of `content`; the + // triggers below mirror every change from `memory_docs`. FTS5 is compiled + // into the bundled SQLCipher build, so this works inside encrypted user + // files too. + sqlx::query( + "CREATE VIRTUAL TABLE IF NOT EXISTS memory_docs_fts USING fts5( + path, content, + content='memory_docs', + content_rowid='id' + )", + ) + .execute(pool) + .await?; + + for trigger in [ + "CREATE TRIGGER IF NOT EXISTS memory_docs_ai AFTER INSERT ON memory_docs BEGIN + INSERT INTO memory_docs_fts(rowid, path, content) + VALUES (new.id, new.path, new.content); + END", + "CREATE TRIGGER IF NOT EXISTS memory_docs_ad AFTER DELETE ON memory_docs BEGIN + INSERT INTO memory_docs_fts(memory_docs_fts, rowid, path, content) + VALUES ('delete', old.id, old.path, old.content); + END", + "CREATE TRIGGER IF NOT EXISTS memory_docs_au AFTER UPDATE ON memory_docs BEGIN + INSERT INTO memory_docs_fts(memory_docs_fts, rowid, path, content) + VALUES ('delete', old.id, old.path, old.content); + INSERT INTO memory_docs_fts(rowid, path, content) + VALUES (new.id, new.path, new.content); + END", + ] { + sqlx::query(trigger).execute(pool).await?; + } + Ok(()) } @@ -764,6 +819,15 @@ mod tests { one("INSERT INTO projects (id, name, path) VALUES (1, 'p', '/tmp')").await.unwrap(); one("INSERT INTO project_tickets (project_id, title, job_id) VALUES (1, 't', 1)").await.unwrap(); one("INSERT INTO llm_request_payloads (request_id, request_json) VALUES ('r1', '{}')").await.unwrap(); + // Fires the AFTER INSERT trigger into the external-content FTS5 table. + one("INSERT INTO memory_docs (path, content) VALUES ('notes/x.md', 'hello world')").await.unwrap(); + + // ...and the FTS index actually answers a MATCH. + let (hits,): (i64,) = sqlx::query_as( + "SELECT count(*) FROM memory_docs_fts WHERE memory_docs_fts MATCH 'world'", + ) + .fetch_one(&pool).await.unwrap(); + assert_eq!(hits, 1, "memory_docs_fts must index inserted notes"); pool.close().await; let _ = std::fs::remove_dir_all(&dir); diff --git a/crates/skald-core/src/session/handler/message_builder.rs b/crates/skald-core/src/session/handler/message_builder.rs index 95fc810..1001425 100644 --- a/crates/skald-core/src/session/handler/message_builder.rs +++ b/crates/skald-core/src/session/handler/message_builder.rs @@ -34,6 +34,9 @@ fn system_timezone() -> Option<&'static str> { /// without needing the full handler and all its dependencies. pub struct MessageBuilder { pub pool: Arc, + /// The shared (`system.db`) pool, for injecting `shared-memory/` notes. The + /// owner `pool` above backs `user-memory/`. + pub shared_pool: Arc, pub session_id: i64, pub mcp: Arc, pub datetime_config: DatetimeConfig, @@ -93,9 +96,7 @@ impl MessageBuilder { You can edit them with `edit_file` or `write_file` using the path shown.\n" ); for mem_path in &meta.inject_memory { - // Resolve the entry to (absolute path to read, path to show the agent). - let (abs, display) = self.resolve_memory_path(mem_path); - let content = tokio::fs::read_to_string(&abs).await.ok(); + let (content, display) = self.load_inject_memory(mem_path).await; match content { Some(c) => static_content.push_str(&format!( "\n\n{c}\n\n" @@ -405,6 +406,27 @@ impl MessageBuilder { /// when the file lives under it, absolute otherwise** — so when the agent references /// it back via `edit_file`/`write_file`, the loop's working-directory injection /// (which rewrites relative paths against the WD) resolves to the very same file. + /// Loads an `inject_memory` entry, returning `(content, display_path)`. + /// + /// Virtual memory paths are read from SQLite: `user-memory/…` from the owner + /// `pool`, `shared-memory/…` from the `shared_pool` (`system.db`). Everything + /// else (`data/…`, `$WD/…`) is an ordinary disk read. A missing note / file + /// yields `None`, rendered as "(file not created yet)". + async fn load_inject_memory(&self, mem_path: &str) -> (Option, String) { + use crate::tools::fs::{classify_memory, MemScope}; + if let Some(m) = classify_memory(mem_path) { + let pool = match m.scope { + MemScope::User => &self.pool, + MemScope::Shared => &self.shared_pool, + }; + let content = crate::db::memory_docs::get(pool, &m.rel) + .await.ok().flatten().map(|d| d.content); + return (content, mem_path.to_string()); + } + let (abs, display) = self.resolve_memory_path(mem_path); + (tokio::fs::read_to_string(&abs).await.ok(), display) + } + fn resolve_memory_path(&self, mem_path: &str) -> (std::path::PathBuf, String) { let wd = self.working_directory.clone() .unwrap_or_else(|| std::env::current_dir().unwrap_or_default()); diff --git a/crates/skald-core/src/session/handler/messages.rs b/crates/skald-core/src/session/handler/messages.rs index a6cc749..ed5104f 100644 --- a/crates/skald-core/src/session/handler/messages.rs +++ b/crates/skald-core/src/session/handler/messages.rs @@ -28,6 +28,7 @@ impl ChatSessionHandler { .map(|rc| rc.effective_working_dir()); let builder = MessageBuilder { pool: Arc::clone(&self.db), + shared_pool: Arc::clone(&self.shared_pool), session_id: self.scratchpad_sid(), mcp: Arc::clone(&self.mcp), datetime_config: self.datetime_config.clone(), diff --git a/crates/skald-core/src/session/handler/mod.rs b/crates/skald-core/src/session/handler/mod.rs index d664a54..dfdc7d3 100644 --- a/crates/skald-core/src/session/handler/mod.rs +++ b/crates/skald-core/src/session/handler/mod.rs @@ -262,6 +262,9 @@ impl ApprovalDecision { pub struct ChatSessionHandler { pub session_id: i64, pub(super) db: Arc, + /// The shared (`system.db`) pool. Owner-bound work uses `db`; this is only for + /// cross-owner reads, e.g. injecting `shared-memory/` notes into the prompt. + pub(super) shared_pool: Arc, /// The authenticated user who owns this session. Threaded into `ChatOptions` /// so the telemetry metadata row in `system.db` carries `user_id`. pub(super) user_id: String, @@ -332,6 +335,7 @@ impl ChatSessionHandler { pub fn new( session_id: i64, db: Arc, + shared_pool: Arc, user_id: String, llm_manager: Arc, max_history_messages: usize, @@ -357,6 +361,7 @@ impl ChatSessionHandler { Self { session_id, db, + shared_pool, user_id, llm_manager, max_history_messages, diff --git a/crates/skald-core/src/session/manager.rs b/crates/skald-core/src/session/manager.rs index ac4ba32..5570650 100644 --- a/crates/skald-core/src/session/manager.rs +++ b/crates/skald-core/src/session/manager.rs @@ -22,6 +22,9 @@ use super::handler::ChatSessionHandler; pub struct ChatSessionManager { db: Arc, + /// The shared (`system.db`) pool, threaded to each handler for cross-owner + /// reads such as injecting `shared-memory/` notes. + shared_pool: Arc, user_id: String, llm_manager: Arc, max_history_messages: usize, @@ -48,6 +51,7 @@ pub struct ChatSessionManager { impl ChatSessionManager { pub fn new( db: Arc, + shared_pool: Arc, user_id: String, llm_manager: Arc, max_history_messages: usize, @@ -68,6 +72,7 @@ impl ChatSessionManager { ) -> Self { Self { db, + shared_pool, user_id, llm_manager, max_history_messages, @@ -155,6 +160,7 @@ impl ChatSessionManager { let handler = Arc::new(ChatSessionHandler::new( session_id, self.db.clone(), + self.shared_pool.clone(), self.user_id.clone(), Arc::clone(&self.llm_manager), self.max_history_messages, diff --git a/crates/skald-core/src/skald/bundles.rs b/crates/skald-core/src/skald/bundles.rs index 9e70756..f6c6ea4 100644 --- a/crates/skald-core/src/skald/bundles.rs +++ b/crates/skald-core/src/skald/bundles.rs @@ -204,9 +204,9 @@ impl Tools { /// Captures sibling managers (mcp, plugins, cron, secrets) into the tool /// registry. `execute_task` is deliberately NOT registered here — it is injected /// per interactive session by `ChatHub::send_message`. - pub(super) fn build(integrations: &Integrations, tasks: &Tasks, models: &Models) -> Self { + pub(super) fn build(rt: &Runtime, integrations: &Integrations, tasks: &Tasks, models: &Models) -> Self { let mut tool_registry = ToolRegistry::new(); - crate::tools::fs::register_all(&mut tool_registry); + crate::tools::fs::register_all(&mut tool_registry, Arc::clone(&rt.db)); tool_registry.register(crate::tools::ast_outline::AstOutline::new()); tool_registry.register(crate::tools::exec::ExecuteCmd); tool_registry.register(crate::tools::read_notification::ReadNotification); @@ -338,6 +338,7 @@ impl Conversation { let manager = Arc::new(ChatSessionManager::new( Arc::clone(&rt.db), + Arc::clone(&rt.db), // shared pool == system.db (this is the ownerless manager) String::new(), Arc::clone(&models.llm_manager), config.llm.max_history_messages, diff --git a/crates/skald-core/src/skald/mod.rs b/crates/skald-core/src/skald/mod.rs index 4b30dd5..913cef1 100644 --- a/crates/skald-core/src/skald/mod.rs +++ b/crates/skald-core/src/skald/mod.rs @@ -66,7 +66,7 @@ impl Skald { let media = Media::build(&rt, &models).await?; let integrations = Integrations::build(&rt, plugins); let tasks = Tasks::build(&rt, config); - let tools = Tools::build(&integrations, &tasks, &models); + let tools = Tools::build(&rt, &integrations, &tasks, &models); let interaction = Interaction::build(&rt, &tools).await?; let conversation = Conversation::build(&rt, &models, &media, &tools, &integrations, &interaction, config).await?; let infra = Infra::build(); diff --git a/crates/skald-core/src/skald/user_context.rs b/crates/skald-core/src/skald/user_context.rs index bdd208f..d45df51 100644 --- a/crates/skald-core/src/skald/user_context.rs +++ b/crates/skald-core/src/skald/user_context.rs @@ -159,6 +159,7 @@ impl UserContextFactory { let manager = Arc::new(ChatSessionManager::new( Arc::clone(&pool), + Arc::clone(&self.registry_pool), // shared pool = system.db, for shared-memory injection user_id.to_string(), Arc::clone(&self.llm_manager), self.max_history_messages, diff --git a/crates/skald-core/src/tools/fs/edit_file.rs b/crates/skald-core/src/tools/fs/edit_file.rs index 0013631..bb02e4a 100644 --- a/crates/skald-core/src/tools/fs/edit_file.rs +++ b/crates/skald-core/src/tools/fs/edit_file.rs @@ -1,8 +1,55 @@ +use std::sync::Arc; + use anyhow::Result; use serde_json::{Value, json}; +use sqlx::SqlitePool; -use crate::tools::{Tool, ToolDescriptionLength, truncate_label, MAX_LABEL_SHORT}; -use super::{read_to_string, write_string}; +use crate::tools::{ + SimpleExecution, Tool, ToolContext, ToolDescriptionLength, ToolExecution, ToolResult, + truncate_label, MAX_LABEL_SHORT, +}; +use super::{classify_memory, read_to_string, write_string, MemScope}; + +/// Applies the substring edit to `content`, returning the new content. Shared by +/// the on-disk [`EditFile::execute`] and the `memory/` routing in `run_with`; +/// `display` is the path used in error messages. +fn apply_edit(content: &str, args: &Value, display: &str) -> Result { + let old = args["old"].as_str() + .ok_or_else(|| anyhow::anyhow!("Missing required argument: old"))?; + let new = args["new"].as_str() + .ok_or_else(|| anyhow::anyhow!("Missing required argument: new"))?; + let replace_all = args["replace_all"].as_bool().unwrap_or(false); + + let updated = if replace_all { + if !content.contains(old) { + anyhow::bail!( + "Text not found in {display}. \ + Call read_file first and copy the text exactly as shown after the '| ' prefix." + ); + } + content.replace(old, new) + } else { + let exact_count = content.matches(old).count(); + if exact_count > 1 { + anyhow::bail!( + "Text found {exact_count} times in {display}. \ + Include more surrounding context in `old` to make it unique, or set replace_all=true." + ); + } + if exact_count == 1 { + content.replacen(old, new, 1) + } else { + let normalized_old = normalize_ws(old); + let (start, end) = find_normalized(content, &normalized_old) + .ok_or_else(|| anyhow::anyhow!( + "Text not found in {display}. \ + Call read_file first and copy the text exactly as shown after the '| ' prefix." + ))?; + format!("{}{}{}", &content[..start], new, &content[end..]) + } + }; + Ok(updated) +} fn normalize_ws(s: &str) -> String { s.lines() @@ -56,10 +103,13 @@ fn find_normalized(haystack: &str, normalized_needle: &str) -> Option<(usize, us None } -pub struct EditFile; +pub struct EditFile { + /// The `shared-memory` (system) pool; see [`ReadFile`](super::ReadFile). + shared_pool: Arc, +} impl EditFile { - pub fn new() -> Self { Self } + pub fn new(shared_pool: Arc) -> Self { Self { shared_pool } } } impl Tool for EditFile { @@ -102,46 +152,32 @@ impl Tool for EditFile { truncate_label(&format!("edit_file `{path}`"), MAX_LABEL_SHORT) } + /// Routes `user-memory/…` / `shared-memory/…` to the note store; every other + /// path falls through to the on-disk [`execute`](Self::execute). + fn run_with<'a>(&'a self, ctx: &ToolContext, args: Value) -> Box { + let path = super::path_arg(&args).unwrap_or_default(); + let Some(m) = classify_memory(&path) else { return self.run(args); }; + let pool = match m.scope { + MemScope::User => Arc::clone(&ctx.pool), + MemScope::Shared => Arc::clone(&self.shared_pool), + }; + let rel = m.rel; + + Box::new(SimpleExecution::new(Box::pin(async move { + let Some(doc) = crate::db::memory_docs::get(&pool, &rel).await? else { + anyhow::bail!("No note at {path}"); + }; + let updated = apply_edit(&doc.content, &args, &path)?; + crate::db::memory_docs::upsert(&pool, &rel, &updated).await?; + Ok(ToolResult::Text(format!("Edited {path}."))) + }))) + } + fn execute(&self, args: Value) -> Result { let user_path = args["path"].as_str() .ok_or_else(|| anyhow::anyhow!("Missing required argument: path"))?; - let old = args["old"].as_str() - .ok_or_else(|| anyhow::anyhow!("Missing required argument: old"))?; - let new = args["new"].as_str() - .ok_or_else(|| anyhow::anyhow!("Missing required argument: new"))?; - - let replace_all = args["replace_all"].as_bool().unwrap_or(false); let content = read_to_string(user_path)?; - - let updated = if replace_all { - if !content.contains(old) { - anyhow::bail!( - "Text not found in {user_path}. \ - Call read_file first and copy the text exactly as shown after the '| ' prefix." - ); - } - content.replace(old, new) - } else { - let exact_count = content.matches(old).count(); - if exact_count > 1 { - anyhow::bail!( - "Text found {exact_count} times in {user_path}. \ - Include more surrounding context in `old` to make it unique, or set replace_all=true." - ); - } - if exact_count == 1 { - content.replacen(old, new, 1) - } else { - let normalized_old = normalize_ws(old); - let (start, end) = find_normalized(&content, &normalized_old) - .ok_or_else(|| anyhow::anyhow!( - "Text not found in {user_path}. \ - Call read_file first and copy the text exactly as shown after the '| ' prefix." - ))?; - format!("{}{}{}", &content[..start], new, &content[end..]) - } - }; - + let updated = apply_edit(&content, &args, user_path)?; write_string(user_path, &updated)?; Ok(format!("Edited {user_path}.")) } diff --git a/crates/skald-core/src/tools/fs/insert_at_line.rs b/crates/skald-core/src/tools/fs/insert_at_line.rs index 5a3c8cc..c4c5b0a 100644 --- a/crates/skald-core/src/tools/fs/insert_at_line.rs +++ b/crates/skald-core/src/tools/fs/insert_at_line.rs @@ -1,13 +1,49 @@ +use std::sync::Arc; + use anyhow::Result; use serde_json::{Value, json}; +use sqlx::SqlitePool; -use crate::tools::{Tool, ToolDescriptionLength, truncate_label, MAX_LABEL_SHORT, MAX_LABEL_FULL}; -use super::{read_to_string, write_string}; +use crate::tools::{ + SimpleExecution, Tool, ToolContext, ToolDescriptionLength, ToolExecution, ToolResult, + truncate_label, MAX_LABEL_SHORT, MAX_LABEL_FULL, +}; +use super::{classify_memory, read_to_string, write_string, MemScope}; -pub struct InsertAtLine; +pub struct InsertAtLine { + /// The `shared-memory` (system) pool; see [`ReadFile`](super::ReadFile). + shared_pool: Arc, +} impl InsertAtLine { - pub fn new() -> Self { Self } + pub fn new(shared_pool: Arc) -> Self { Self { shared_pool } } +} + +/// Inserts `content` before/after `line` in `text`, returning the new text and a +/// result message. Shared by the on-disk `execute` and the `memory/` routing; +/// `display` is the path used in the message. +fn apply_insert(text: &str, args: &Value, display: &str) -> Result<(String, String)> { + let line_num = args["line"].as_u64() + .ok_or_else(|| anyhow::anyhow!("Missing required argument: line"))? as usize; + let new_text = args["content"].as_str() + .ok_or_else(|| anyhow::anyhow!("Missing required argument: content"))?; + let placement = args["placement"].as_str().unwrap_or("after"); + + anyhow::ensure!(line_num >= 1, "line must be >= 1"); + + let mut lines: Vec<&str> = text.split('\n').collect(); + let idx = (line_num - 1).min(lines.len().saturating_sub(1)); + let insert_idx = if placement == "before" { idx } else { idx + 1 }; + let new_lines: Vec<&str> = new_text.split('\n').collect(); + for (i, l) in new_lines.iter().enumerate() { + lines.insert(insert_idx + i, l); + } + let updated = lines.join("\n"); + let msg = format!( + "Inserted {} line(s) {} line {} in {display}.", + new_lines.len(), placement, line_num + ); + Ok((updated, msg)) } impl Tool for InsertAtLine { @@ -53,32 +89,33 @@ impl Tool for InsertAtLine { } } + /// Routes `user-memory/…` / `shared-memory/…` to the note store; every other + /// path falls through to the on-disk [`execute`](Self::execute). + fn run_with<'a>(&'a self, ctx: &ToolContext, args: Value) -> Box { + let path = super::path_arg(&args).unwrap_or_default(); + let Some(m) = classify_memory(&path) else { return self.run(args); }; + let pool = match m.scope { + MemScope::User => Arc::clone(&ctx.pool), + MemScope::Shared => Arc::clone(&self.shared_pool), + }; + let rel = m.rel; + + Box::new(SimpleExecution::new(Box::pin(async move { + let Some(doc) = crate::db::memory_docs::get(&pool, &rel).await? else { + anyhow::bail!("No note at {path}"); + }; + let (updated, msg) = apply_insert(&doc.content, &args, &path)?; + crate::db::memory_docs::upsert(&pool, &rel, &updated).await?; + Ok(ToolResult::Text(msg)) + }))) + } + fn execute(&self, args: Value) -> Result { let user_path = args["path"].as_str() .ok_or_else(|| anyhow::anyhow!("Missing required argument: path"))?; - let line_num = args["line"].as_u64() - .ok_or_else(|| anyhow::anyhow!("Missing required argument: line"))? as usize; - let new_text = args["content"].as_str() - .ok_or_else(|| anyhow::anyhow!("Missing required argument: content"))?; - let placement = args["placement"].as_str().unwrap_or("after"); - - anyhow::ensure!(line_num >= 1, "line must be >= 1"); - let text = read_to_string(user_path)?; - let mut lines: Vec<&str> = text.split('\n').collect(); - let idx = (line_num - 1).min(lines.len().saturating_sub(1)); - let insert_idx = if placement == "before" { idx } else { idx + 1 }; - let new_lines: Vec<&str> = new_text.split('\n').collect(); - for (i, l) in new_lines.iter().enumerate() { - lines.insert(insert_idx + i, l); - } - let updated = lines.join("\n"); - + let (updated, msg) = apply_insert(&text, &args, user_path)?; write_string(user_path, &updated)?; - - Ok(format!( - "Inserted {} line(s) {} line {} in {user_path}.", - new_lines.len(), placement, line_num - )) + Ok(msg) } } diff --git a/crates/skald-core/src/tools/fs/list_files.rs b/crates/skald-core/src/tools/fs/list_files.rs index ef1453c..60dad7b 100644 --- a/crates/skald-core/src/tools/fs/list_files.rs +++ b/crates/skald-core/src/tools/fs/list_files.rs @@ -1,20 +1,28 @@ use std::path::Path; +use std::sync::Arc; use anyhow::Result; use serde_json::{Value, json}; +use sqlx::SqlitePool; -use crate::tools::{Tool, ToolDescriptionLength, truncate_label, MAX_LABEL_SHORT}; -use super::resolve; +use crate::tools::{ + SimpleExecution, Tool, ToolContext, ToolDescriptionLength, ToolExecution, ToolResult, + truncate_label, MAX_LABEL_SHORT, +}; +use super::{classify_memory, resolve, MemScope}; /// Directories to skip unconditionally when walking. /// `secrets` is skipped so a recursive listing rooted at a parent (e.g. the auto-read /// working directory) never reveals the contents of the secrets store. const SKIP_DIRS: &[&str] = &[".git", "target", "node_modules", ".cache", "secrets"]; -pub struct ListFiles; +pub struct ListFiles { + /// The `shared-memory` (system) pool; see [`ReadFile`](super::ReadFile). + shared_pool: Arc, +} impl ListFiles { - pub fn new() -> Self { Self } + pub fn new(shared_pool: Arc) -> Self { Self { shared_pool } } } impl Tool for ListFiles { @@ -27,7 +35,8 @@ impl Tool for ListFiles { Relative paths are resolved from the project root; absolute paths (starting with /) are used as-is. \ Skips .git, target, node_modules, .cache. \ Returns a JSON array of paths relative to the requested directory. \ - Use depth=1 for immediate contents only, depth=2-3 for moderate exploration." + Use depth=1 for immediate contents only, depth=2-3 for moderate exploration. \ + Listing under user-memory/ (private) or shared-memory/ (shared) lists your memory notes instead of disk." } fn parameters_schema(&self) -> Value { @@ -56,6 +65,32 @@ impl Tool for ListFiles { truncate_label(&format!("list_files `{path}`"), MAX_LABEL_SHORT) } + /// Routes `user-memory/…` / `shared-memory/…` to the note store; every other + /// path falls through to the on-disk [`execute`](Self::execute). Memory is a + /// flat key space, so `depth`/`dirs_only` don't apply — the whole subtree + /// under the prefix is returned, keyed relative to the requested directory. + fn run_with<'a>(&'a self, ctx: &ToolContext, args: Value) -> Box { + let path = args["path"].as_str().unwrap_or("").to_string(); + let Some(m) = classify_memory(&path) else { return self.run(args); }; + let pool = match m.scope { + MemScope::User => Arc::clone(&ctx.pool), + MemScope::Shared => Arc::clone(&self.shared_pool), + }; + let rel = m.rel; + + Box::new(SimpleExecution::new(Box::pin(async move { + // Treat `rel` as a directory prefix: match `rel/…` (or everything at + // the root), then strip it so results are relative to what was asked. + let prefix = if rel.is_empty() || rel.ends_with('/') { rel } else { format!("{rel}/") }; + let entries = crate::db::memory_docs::list(&pool, &prefix).await?; + let mut paths: Vec = entries.into_iter() + .map(|e| e.path.strip_prefix(&prefix).unwrap_or(&e.path).to_string()) + .collect(); + paths.sort(); + Ok(ToolResult::Text(serde_json::to_string(&paths)?)) + }))) + } + fn execute(&self, args: Value) -> Result { let user_path = args["path"].as_str().unwrap_or("."); let max_depth = args["depth"].as_u64().unwrap_or(3) as usize; diff --git a/crates/skald-core/src/tools/fs/memory_search.rs b/crates/skald-core/src/tools/fs/memory_search.rs new file mode 100644 index 0000000..6344ae4 --- /dev/null +++ b/crates/skald-core/src/tools/fs/memory_search.rs @@ -0,0 +1,125 @@ +//! `memory_search` — full-text search over the virtual memory namespace (§5). +//! +//! Unlike the fs-tools, this does **not** route a path: it searches note *content* +//! through the `memory_docs` FTS5 index (`memory_docs::search`, bm25-ranked with a +//! highlighted snippet). `user-memory` is the caller's own pool (`ToolContext::pool`); +//! `shared-memory` is the system pool captured at registration. Kept a distinct tool +//! rather than folding FTS into `grep_files`: grep is regex-per-line over a tree, +//! this is ranked keyword recall — different semantics, so different names. + +use std::sync::Arc; + +use serde_json::{Value, json}; +use sqlx::SqlitePool; + +use crate::db::memory_docs::{self, MemoryHit}; +use crate::tools::{ + SimpleExecution, Tool, ToolContext, ToolDescriptionLength, ToolExecution, ToolResult, + truncate_label, MAX_LABEL_SHORT, +}; + +pub struct MemorySearch { + /// The `shared-memory` (system) pool; see [`ReadFile`](super::ReadFile). + shared_pool: Arc, +} + +impl MemorySearch { + pub fn new(shared_pool: Arc) -> Self { Self { shared_pool } } +} + +/// Turns free text into a robust FTS5 MATCH query: each whitespace token becomes a +/// quoted term (AND-combined), so arbitrary input — colons, dashes, punctuation — +/// can't trip an FTS5 syntax error. Returns `None` when there are no tokens. +fn fts_query(input: &str) -> Option { + let terms: Vec = input + .split_whitespace() + .map(|t| format!("\"{}\"", t.replace('"', "\"\""))) + .collect(); + (!terms.is_empty()).then(|| terms.join(" ")) +} + +fn render_hits(store: &str, hits: &[MemoryHit], out: &mut String) { + for h in hits { + out.push_str(&format!("[{store}] {} — {}\n", h.path, h.snippet)); + } +} + +impl Tool for MemorySearch { + fn name(&self) -> &str { "memory_search" } + fn category(&self) -> crate::tools::ToolCategory { crate::tools::ToolCategory::Introspection } + + fn description(&self) -> &str { + "Full-text search across your memory notes by keyword, ranked by relevance. \ + Searches user-memory/ (private) and shared-memory/ (shared); set scope to narrow it. \ + Returns matching note paths with a short highlighted snippet — open one with read_file. \ + Use this to recall where you wrote something instead of listing and reading notes one by one." + } + + fn parameters_schema(&self) -> Value { + json!({ + "type": "object", + "properties": { + "query": { "type": "string", "description": "Keywords to search for. Plain words; all must appear." }, + "scope": { + "type": "string", + "enum": ["all", "private", "shared"], + "description": "Which store to search: 'private' (user-memory), 'shared' (shared-memory), or 'all' (default)." + }, + "limit": { "type": "integer", "description": "Max results per store (default 10, max 50).", "default": 10 } + }, + "required": ["query"] + }) + } + + fn describe(&self, args: &Value, _length: ToolDescriptionLength) -> String { + let q = args["query"].as_str().unwrap_or("?"); + truncate_label(&format!("memory_search \"{q}\""), MAX_LABEL_SHORT) + } + + fn run_with<'a>(&'a self, ctx: &ToolContext, args: Value) -> Box { + let user_pool = Arc::clone(&ctx.pool); + let shared_pool = Arc::clone(&self.shared_pool); + + Box::new(SimpleExecution::new(Box::pin(async move { + let raw = args["query"].as_str().unwrap_or(""); + let Some(q) = fts_query(raw) else { + anyhow::bail!("memory_search needs a non-empty query"); + }; + let scope = args["scope"].as_str().unwrap_or("all"); + let limit = args["limit"].as_u64().unwrap_or(10).clamp(1, 50) as i64; + + let mut out = String::new(); + let mut total = 0usize; + if scope == "all" || scope == "private" { + let hits = memory_docs::search(&user_pool, &q, limit).await?; + total += hits.len(); + render_hits("user-memory", &hits, &mut out); + } + if scope == "all" || scope == "shared" { + let hits = memory_docs::search(&shared_pool, &q, limit).await?; + total += hits.len(); + render_hits("shared-memory", &hits, &mut out); + } + + if total == 0 { + return Ok(ToolResult::Text(format!("No memory notes match {raw:?}."))); + } + Ok(ToolResult::Text(format!("{total} result(s):\n{out}"))) + }))) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn fts_query_quotes_tokens_and_survives_punctuation() { + assert_eq!(fts_query("spesa settimana").unwrap(), "\"spesa\" \"settimana\""); + // colons / dashes would be FTS5 operators unquoted; quoting makes them literal + assert_eq!(fts_query("budget: 2026-07").unwrap(), "\"budget:\" \"2026-07\""); + // an embedded quote is escaped by doubling + assert_eq!(fts_query("say \"hi\"").unwrap(), "\"say\" \"\"\"hi\"\"\""); + assert!(fts_query(" ").is_none()); + } +} diff --git a/crates/skald-core/src/tools/fs/mod.rs b/crates/skald-core/src/tools/fs/mod.rs index dde0e59..3d3e418 100644 --- a/crates/skald-core/src/tools/fs/mod.rs +++ b/crates/skald-core/src/tools/fs/mod.rs @@ -2,15 +2,18 @@ mod edit_file; mod grep_files; mod insert_at_line; mod list_files; +mod memory_search; mod read_file; mod replace_lines; mod search_file; mod write_file; use std::path::{Component, Path, PathBuf}; +use std::sync::Arc; use anyhow::{Context, Result}; use serde_json::Value; +use sqlx::SqlitePool; use crate::tools::ToolRegistry; @@ -25,11 +28,65 @@ pub use edit_file::EditFile; pub use grep_files::GrepFiles; pub use insert_at_line::InsertAtLine; pub use list_files::ListFiles; +pub use memory_search::MemorySearch; pub use read_file::ReadFile; pub use replace_lines::ReplaceLines; pub use search_file::SearchFile; pub use write_file::WriteFile; +// ── Virtual memory namespace (blueprint §5) ─────────────────────────────────── +// +// Two sibling top-level roots, each backed by the `memory_docs` table in SQLite +// rather than the disk. The fs-tools intercept these prefixes in `run_with` and +// route reads/writes to the `memory_docs` accessor on the right pool, so the LLM +// uses ordinary read/write/list against what looks like two folders. + +/// The current user's **private** memory — routed to `ctx.pool` (`{userid}.db`, +/// behind SQLCipher). +pub const USER_MEMORY_ROOT: &str = "user-memory"; + +/// The instance-wide **shared** memory — routed to the system pool (`system.db`, +/// cleartext, readable by every member). +pub const SHARED_MEMORY_ROOT: &str = "shared-memory"; + +/// Which memory store a path resolves to. +pub(crate) enum MemScope { + /// `user-memory/…` → the caller's own pool (`ToolContext::pool`). + User, + /// `shared-memory/…` → the shared system pool. + Shared, +} + +/// A path that falls inside the virtual memory namespace: the store it belongs to +/// and the note key **relative to that store's root** (the root prefix stripped). +pub(crate) struct MemRef { + pub scope: MemScope, + pub rel: String, +} + +/// Classifies a user-supplied path. Returns `Some` when it lands under one of the +/// virtual memory roots — to be routed to SQLite — and `None` for an ordinary +/// disk path. +/// +/// The **first** component decides the store, taken raw *before* normalization, so +/// a `..` in the tail can never drop the memory root and silently fall back to a +/// disk path. The tail is then normalized (resolving `.`/`..`) and clamped at the +/// store root, so a memory path stays within its store and an absolute path is +/// always disk. +pub(crate) fn classify_memory(user_path: &str) -> Option { + let mut parts = user_path.trim_start_matches("./").splitn(2, ['/', '\\']); + let scope = match parts.next()? { + USER_MEMORY_ROOT => MemScope::User, + SHARED_MEMORY_ROOT => MemScope::Shared, + _ => return None, + }; + // Normalize the tail within the store (empty = the root itself). `..` clamps + // at the root rather than escaping upward. + let tail = parts.next().unwrap_or(""); + let rel = lexical_normalize(Path::new(tail)).to_string_lossy().replace('\\', "/"); + Some(MemRef { scope, rel }) +} + /// Resolve a user-supplied path: /// - starts with `/` → absolute path, used as-is /// - otherwise → relative to the process working directory (project root) @@ -127,13 +184,209 @@ pub(super) fn write_string(user_path: &str, content: &str) -> Result<()> { .with_context(|| format!("Failed to write: {}", abs.display())) } -pub fn register_all(registry: &mut ToolRegistry) { - registry.register(EditFile::new()); - registry.register(GrepFiles::new()); - registry.register(InsertAtLine::new()); - registry.register(ListFiles::new()); - registry.register(ReadFile::new()); - registry.register(ReplaceLines::new()); - registry.register(SearchFile::new()); - registry.register(WriteFile::new()); +/// Registers the filesystem tools. `shared_pool` is the system (`shared-memory`) +/// pool captured once here — a global singleton — and handed to the memory-aware +/// tools; each still resolves the per-user (`user-memory`) pool per call from the +/// `ToolContext`. +pub fn register_all(registry: &mut ToolRegistry, shared_pool: Arc) { + registry.register(EditFile::new(Arc::clone(&shared_pool))); + registry.register(GrepFiles::new()); // not memory-aware yet — see blueprint Prossimi passi + registry.register(InsertAtLine::new(Arc::clone(&shared_pool))); + registry.register(ListFiles::new(Arc::clone(&shared_pool))); + registry.register(ReadFile::new(Arc::clone(&shared_pool))); + registry.register(ReplaceLines::new(Arc::clone(&shared_pool))); + registry.register(SearchFile::new(Arc::clone(&shared_pool))); + registry.register(MemorySearch::new(Arc::clone(&shared_pool))); + registry.register(WriteFile::new(shared_pool)); +} + +#[cfg(test)] +mod tests { + use super::*; + use std::path::PathBuf; + + use serde_json::json; + + use crate::tools::{ExecutionOutcome, Tool, ToolContext}; + + #[test] + fn classify_memory_splits_root_from_key() { + let u = classify_memory("user-memory/notes/x.md").unwrap(); + assert!(matches!(u.scope, MemScope::User)); + assert_eq!(u.rel, "notes/x.md"); + + let s = classify_memory("./shared-memory/casa.md").unwrap(); + assert!(matches!(s.scope, MemScope::Shared)); + assert_eq!(s.rel, "casa.md"); + + // bare roots (with/without trailing slash) resolve to the empty key + assert_eq!(classify_memory("user-memory").unwrap().rel, ""); + assert_eq!(classify_memory("shared-memory/").unwrap().rel, ""); + + // `..` clamps inside the store instead of falling back to a disk path + assert_eq!(classify_memory("user-memory/../secret.md").unwrap().rel, "secret.md"); + + // ordinary, absolute, and look-alike paths are disk (None) + assert!(classify_memory("src/main.rs").is_none()); + assert!(classify_memory("/etc/hosts").is_none()); + assert!(classify_memory("user-memoryish/x").is_none()); + } + + /// A throwaway owner-schema pool (as `Arc`, ready for a `ToolContext`), plus its + /// dir for cleanup. `tag` + a counter keep parallel tests off the same file. + async fn store(tag: &str) -> (Arc, PathBuf) { + use std::sync::atomic::{AtomicU64, Ordering}; + static SEQ: AtomicU64 = AtomicU64::new(0); + let n = SEQ.fetch_add(1, Ordering::Relaxed); + let dir = std::env::temp_dir().join(format!("skald-fsmem-{}-{tag}-{n}", std::process::id())); + let _ = std::fs::remove_dir_all(&dir); + std::fs::create_dir_all(&dir).unwrap(); + let pool = crate::db::create_user_pool(&dir.join("owner.db"), None).await.unwrap(); + (Arc::new(pool), dir) + } + + /// Drives a tool through the context-aware path and returns its text result. + async fn drive(tool: &dyn Tool, ctx: &ToolContext, args: Value) -> Result { + let exec = tool.run_with(ctx, args); + match exec.wait().await { + ExecutionOutcome::Completed(r) => Ok(r.to_wire()), + ExecutionOutcome::Failed(e) => Err(e), + ExecutionOutcome::Cancelled => Err("cancelled".into()), + } + } + + #[tokio::test] + async fn memory_tools_route_and_isolate_user_vs_shared() { + let (user, udir) = store("user").await; + let (shared, sdir) = store("shared").await; + + // The shared pool is captured by the tools; the user pool arrives per call. + let write = WriteFile::new(Arc::clone(&shared)); + let read = ReadFile::new(Arc::clone(&shared)); + let list = ListFiles::new(Arc::clone(&shared)); + let ctx = ToolContext { session_id: 1, pool: Arc::clone(&user) }; + + // Private write lands in the user pool — and never in the shared one. + let out = drive(&write, &ctx, json!({"path":"user-memory/spesa.md","content":"latte\npane"})) + .await.unwrap(); + assert!(out.starts_with("Created user-memory/spesa.md"), "{out}"); + assert!(crate::db::memory_docs::get(&user, "spesa.md").await.unwrap().is_some()); + assert!(crate::db::memory_docs::get(&shared, "spesa.md").await.unwrap().is_none(), + "a user-memory write must not touch the shared store"); + + // Shared write lands in the shared pool — and never in the user one. + drive(&write, &ctx, json!({"path":"shared-memory/casa.md","content":"wifi 1234"})) + .await.unwrap(); + assert!(crate::db::memory_docs::get(&shared, "casa.md").await.unwrap().is_some()); + assert!(crate::db::memory_docs::get(&user, "casa.md").await.unwrap().is_none()); + + // Read back with 1-based line numbers; a missing note errors. + let r = drive(&read, &ctx, json!({"path":"user-memory/spesa.md"})).await.unwrap(); + assert!(r.contains("| latte") && r.contains("| pane"), "{r}"); + assert!(drive(&read, &ctx, json!({"path":"user-memory/nope.md"})).await.is_err()); + + // A second write to the same key overwrites (and says so). + let out = drive(&write, &ctx, json!({"path":"user-memory/spesa.md","content":"latte"})) + .await.unwrap(); + assert!(out.starts_with("Overwrote user-memory/spesa.md"), "{out}"); + + // Listing returns keys relative to the requested directory. + drive(&write, &ctx, json!({"path":"user-memory/notes/idee.md","content":"x"})) + .await.unwrap(); + let l = drive(&list, &ctx, json!({"path":"user-memory"})).await.unwrap(); + assert_eq!(serde_json::from_str::>(&l).unwrap(), + vec!["notes/idee.md".to_string(), "spesa.md".to_string()]); + let l = drive(&list, &ctx, json!({"path":"user-memory/notes"})).await.unwrap(); + assert_eq!(serde_json::from_str::>(&l).unwrap(), + vec!["idee.md".to_string()]); + + let _ = std::fs::remove_dir_all(&udir); + let _ = std::fs::remove_dir_all(&sdir); + } + + #[tokio::test] + async fn memory_edit_insert_replace_search_route_to_the_note() { + let (user, udir) = store("edit-user").await; + let (shared, sdir) = store("edit-shared").await; + + let write = WriteFile::new(Arc::clone(&shared)); + let edit = EditFile::new(Arc::clone(&shared)); + let insert = InsertAtLine::new(Arc::clone(&shared)); + let replace = ReplaceLines::new(Arc::clone(&shared)); + let search = SearchFile::new(Arc::clone(&shared)); + let ctx = ToolContext { session_id: 1, pool: Arc::clone(&user) }; + + async fn note(pool: &SqlitePool, path: &str) -> String { + crate::db::memory_docs::get(pool, path).await.unwrap().unwrap().content + } + + drive(&write, &ctx, json!({"path":"user-memory/todo.md","content":"latte\npane\nuvoa"})) + .await.unwrap(); + + // edit_file: fix the typo, in place + let out = drive(&edit, &ctx, json!({"path":"user-memory/todo.md","old":"uvoa","new":"uova"})) + .await.unwrap(); + assert_eq!(out, "Edited user-memory/todo.md."); + assert_eq!(note(&user, "todo.md").await, "latte\npane\nuova"); + + // insert_at_line: add a line after line 1 + drive(&insert, &ctx, json!({"path":"user-memory/todo.md","line":1,"content":"burro","placement":"after"})) + .await.unwrap(); + assert_eq!(note(&user, "todo.md").await, "latte\nburro\npane\nuova"); + + // replace_lines: collapse lines 2–3 into one + drive(&replace, &ctx, json!({"path":"user-memory/todo.md","from_line":2,"to_line":3,"new":"olio"})) + .await.unwrap(); + assert_eq!(note(&user, "todo.md").await, "latte\nolio\nuova"); + + // search_file: find a line inside the note + let s = drive(&search, &ctx, json!({"path":"user-memory/todo.md","query":"olio"})).await.unwrap(); + assert!(s.contains("match(es) in user-memory/todo.md"), "{s}"); + assert!(s.contains("| olio"), "{s}"); + + // editing a note that doesn't exist errors, not creates + assert!(drive(&edit, &ctx, json!({"path":"user-memory/ghost.md","old":"a","new":"b"})) + .await.is_err()); + + let _ = std::fs::remove_dir_all(&udir); + let _ = std::fs::remove_dir_all(&sdir); + } + + #[tokio::test] + async fn memory_search_scopes_to_user_shared_or_all() { + let (user, udir) = store("search-user").await; + let (shared, sdir) = store("search-shared").await; + + let write = WriteFile::new(Arc::clone(&shared)); + let search = MemorySearch::new(Arc::clone(&shared)); + let ctx = ToolContext { session_id: 1, pool: Arc::clone(&user) }; + + // one note in each store, both mentioning "wifi" + drive(&write, &ctx, json!({"path":"user-memory/rete.md","content":"la mia wifi privata"})) + .await.unwrap(); + drive(&write, &ctx, json!({"path":"shared-memory/casa.md","content":"wifi di casa 1234"})) + .await.unwrap(); + + // scope=private → only the user store + let r = drive(&search, &ctx, json!({"query":"wifi","scope":"private"})).await.unwrap(); + assert!(r.contains("[user-memory] rete.md"), "{r}"); + assert!(!r.contains("shared-memory"), "{r}"); + + // scope=shared → only the shared store + let r = drive(&search, &ctx, json!({"query":"wifi","scope":"shared"})).await.unwrap(); + assert!(r.contains("[shared-memory] casa.md"), "{r}"); + assert!(!r.contains("[user-memory]"), "{r}"); + + // scope=all (default) → both, and the snippet highlights the term + let r = drive(&search, &ctx, json!({"query":"wifi"})).await.unwrap(); + assert!(r.contains("[user-memory] rete.md") && r.contains("[shared-memory] casa.md"), "{r}"); + assert!(r.contains("[wifi]"), "snippet should highlight the match: {r}"); + + // no match → a friendly message, not an error + let r = drive(&search, &ctx, json!({"query":"inesistente"})).await.unwrap(); + assert!(r.starts_with("No memory notes match"), "{r}"); + + let _ = std::fs::remove_dir_all(&udir); + let _ = std::fs::remove_dir_all(&sdir); + } } diff --git a/crates/skald-core/src/tools/fs/read_file.rs b/crates/skald-core/src/tools/fs/read_file.rs index 35b86e0..c9bfbb0 100644 --- a/crates/skald-core/src/tools/fs/read_file.rs +++ b/crates/skald-core/src/tools/fs/read_file.rs @@ -1,13 +1,50 @@ +use std::sync::Arc; + use anyhow::Result; use serde_json::{Value, json}; +use sqlx::SqlitePool; -use crate::tools::{Tool, ToolDescriptionLength, truncate_label, MAX_LABEL_SHORT, MAX_LABEL_FULL}; -use super::read_to_string; +use crate::tools::{ + SimpleExecution, Tool, ToolContext, ToolDescriptionLength, ToolExecution, ToolResult, + truncate_label, MAX_LABEL_SHORT, MAX_LABEL_FULL, +}; +use super::{classify_memory, read_to_string, MemScope}; -pub struct ReadFile; +pub struct ReadFile { + /// The `shared-memory` (system) pool. `user-memory` resolves per call from the + /// `ToolContext`; only the shared store is a global singleton captured here. + shared_pool: Arc, +} impl ReadFile { - pub fn new() -> Self { Self } + pub fn new(shared_pool: Arc) -> Self { Self { shared_pool } } +} + +/// Render `content` with 1-based line numbers, honouring the same +/// `start`/`end_line`/`limit` windowing as the disk path. Shared by the on-disk +/// [`ReadFile::execute`] and the `memory/` routing in [`ReadFile::run_with`]. +fn number_lines(content: &str, start: usize, end_line: Option, limit: Option) -> String { + let lines: Vec<&str> = content.lines().collect(); + let total = lines.len(); + + let end = match (end_line, limit) { + (Some(e), _) => e.min(total), + (None, Some(l)) => (start + l).min(total), + (None, None) => total, + }; + + if start >= total && total > 0 { + return format!("(file has only {total} lines; start_line {} is out of range)", start + 1); + } + + let end = end.max(start); + let width = total.to_string().len().max(3); + lines[start..end] + .iter() + .enumerate() + .map(|(i, line)| format!("{:>width$} | {line}", start + i + 1)) + .collect::>() + .join("\n") } impl Tool for ReadFile { @@ -19,7 +56,8 @@ impl Tool for ReadFile { Use instead of cat/head/tail in the terminal. \ Returns text prefixed as ' N | line'. When calling edit_file, copy the text after '| ' exactly. \ For large files use start_line/end_line to read in chunks — files over ~2000 lines should never be read whole. \ - Use limit to cap output when end_line is unknown." + Use limit to cap output when end_line is unknown. \ + Paths under user-memory/ (private) or shared-memory/ (shared) read a note from your memory instead of disk." } fn parameters_schema(&self) -> Value { @@ -70,37 +108,35 @@ impl Tool for ReadFile { } } + /// Routes `user-memory/…` / `shared-memory/…` to the note store; every other + /// path falls through to the on-disk [`execute`](Self::execute). + fn run_with<'a>(&'a self, ctx: &ToolContext, args: Value) -> Box { + let path = super::path_arg(&args).unwrap_or_default(); + let Some(m) = classify_memory(&path) else { return self.run(args); }; + let pool = match m.scope { + MemScope::User => Arc::clone(&ctx.pool), + MemScope::Shared => Arc::clone(&self.shared_pool), + }; + let rel = m.rel; + let start = args["start_line"].as_u64().map(|n| (n as usize).saturating_sub(1)).unwrap_or(0); + let end_line = args["end_line"].as_u64().map(|n| n as usize); + let limit = args["limit"].as_u64().map(|n| n.min(2000) as usize); + + Box::new(SimpleExecution::new(Box::pin(async move { + let Some(doc) = crate::db::memory_docs::get(&pool, &rel).await? else { + anyhow::bail!("No note at {path}"); + }; + Ok(ToolResult::Text(number_lines(&doc.content, start, end_line, limit))) + }))) + } + fn execute(&self, args: Value) -> Result { let user_path = args["path"].as_str() .ok_or_else(|| anyhow::anyhow!("Missing required argument: path"))?; let content = read_to_string(user_path)?; - let lines: Vec<&str> = content.lines().collect(); - let total = lines.len(); - + let start = args["start_line"].as_u64().map(|n| (n as usize).saturating_sub(1)).unwrap_or(0); + let end_line = args["end_line"].as_u64().map(|n| n as usize); let limit = args["limit"].as_u64().map(|n| n.min(2000) as usize); - let start = args["start_line"].as_u64() - .map(|n| (n as usize).saturating_sub(1)) - .unwrap_or(0); - let end = match (args["end_line"].as_u64(), limit) { - (Some(e), _) => (e as usize).min(total), - (None, Some(l)) => (start + l).min(total), - (None, None) => total, - }; - - if start >= total && total > 0 { - return Ok(format!("(file has only {total} lines; start_line {start_line} is out of range)", - start_line = start + 1)); - } - - let end = end.max(start); - - let width = total.to_string().len().max(3); - let numbered = lines[start..end] - .iter() - .enumerate() - .map(|(i, line)| format!("{:>width$} | {line}", start + i + 1)) - .collect::>() - .join("\n"); - Ok(numbered) + Ok(number_lines(&content, start, end_line, limit)) } } diff --git a/crates/skald-core/src/tools/fs/replace_lines.rs b/crates/skald-core/src/tools/fs/replace_lines.rs index f001949..cdfa09c 100644 --- a/crates/skald-core/src/tools/fs/replace_lines.rs +++ b/crates/skald-core/src/tools/fs/replace_lines.rs @@ -1,13 +1,56 @@ +use std::sync::Arc; + use anyhow::Result; use serde_json::{Value, json}; +use sqlx::SqlitePool; -use crate::tools::{Tool, ToolDescriptionLength, truncate_label, MAX_LABEL_SHORT, MAX_LABEL_FULL}; -use super::{read_to_string, write_string}; +use crate::tools::{ + SimpleExecution, Tool, ToolContext, ToolDescriptionLength, ToolExecution, ToolResult, + truncate_label, MAX_LABEL_SHORT, MAX_LABEL_FULL, +}; +use super::{classify_memory, read_to_string, write_string, MemScope}; -pub struct ReplaceLines; +pub struct ReplaceLines { + /// The `shared-memory` (system) pool; see [`ReadFile`](super::ReadFile). + shared_pool: Arc, +} impl ReplaceLines { - pub fn new() -> Self { Self } + pub fn new(shared_pool: Arc) -> Self { Self { shared_pool } } +} + +/// Replaces the inclusive 1-based line range with `new`, returning the new content +/// and a result message. Shared by the on-disk `execute` and the `memory/` routing; +/// `display` is the path used in the message. +fn apply_replace(content: &str, args: &Value, display: &str) -> Result<(String, String)> { + let from_line = args["from_line"].as_u64() + .ok_or_else(|| anyhow::anyhow!("Missing required argument: from_line"))? as usize; + let to_line = args["to_line"].as_u64() + .ok_or_else(|| anyhow::anyhow!("Missing required argument: to_line"))? as usize; + let new = args["new"].as_str() + .ok_or_else(|| anyhow::anyhow!("Missing required argument: new"))?; + + if from_line == 0 { anyhow::bail!("from_line must be >= 1"); } + if to_line < from_line { anyhow::bail!("to_line must be >= from_line"); } + + let mut lines: Vec<&str> = content.lines().collect(); + let total = lines.len(); + if from_line > total { + anyhow::bail!("from_line {from_line} exceeds file length ({total} lines)"); + } + let to_clamped = to_line.min(total); + let new_lines: Vec<&str> = new.lines().collect(); + lines.splice((from_line - 1)..to_clamped, new_lines); + + let has_trailing = content.ends_with('\n'); + let mut updated = lines.join("\n"); + if has_trailing { updated.push('\n'); } + + let msg = format!( + "Replaced lines {from_line}–{to_clamped} in {display} with {} new lines.", + new.lines().count() + ); + Ok((updated, msg)) } impl Tool for ReplaceLines { @@ -51,38 +94,33 @@ impl Tool for ReplaceLines { } } + /// Routes `user-memory/…` / `shared-memory/…` to the note store; every other + /// path falls through to the on-disk [`execute`](Self::execute). + fn run_with<'a>(&'a self, ctx: &ToolContext, args: Value) -> Box { + let path = super::path_arg(&args).unwrap_or_default(); + let Some(m) = classify_memory(&path) else { return self.run(args); }; + let pool = match m.scope { + MemScope::User => Arc::clone(&ctx.pool), + MemScope::Shared => Arc::clone(&self.shared_pool), + }; + let rel = m.rel; + + Box::new(SimpleExecution::new(Box::pin(async move { + let Some(doc) = crate::db::memory_docs::get(&pool, &rel).await? else { + anyhow::bail!("No note at {path}"); + }; + let (updated, msg) = apply_replace(&doc.content, &args, &path)?; + crate::db::memory_docs::upsert(&pool, &rel, &updated).await?; + Ok(ToolResult::Text(msg)) + }))) + } + fn execute(&self, args: Value) -> Result { let user_path = args["path"].as_str() .ok_or_else(|| anyhow::anyhow!("Missing required argument: path"))?; - let from_line = args["from_line"].as_u64() - .ok_or_else(|| anyhow::anyhow!("Missing required argument: from_line"))? as usize; - let to_line = args["to_line"].as_u64() - .ok_or_else(|| anyhow::anyhow!("Missing required argument: to_line"))? as usize; - let new = args["new"].as_str() - .ok_or_else(|| anyhow::anyhow!("Missing required argument: new"))?; - - if from_line == 0 { anyhow::bail!("from_line must be >= 1"); } - if to_line < from_line { anyhow::bail!("to_line must be >= from_line"); } - let content = read_to_string(user_path)?; - let mut lines: Vec<&str> = content.lines().collect(); - let total = lines.len(); - if from_line > total { - anyhow::bail!("from_line {from_line} exceeds file length ({total} lines)"); - } - let to_clamped = to_line.min(total); - let new_lines: Vec<&str> = new.lines().collect(); - lines.splice((from_line - 1)..to_clamped, new_lines); - - let has_trailing = content.ends_with('\n'); - let mut updated = lines.join("\n"); - if has_trailing { updated.push('\n'); } - + let (updated, msg) = apply_replace(&content, &args, user_path)?; write_string(user_path, &updated)?; - - Ok(format!( - "Replaced lines {from_line}–{to_clamped} in {user_path} with {} new lines.", - new.lines().count() - )) + Ok(msg) } } diff --git a/crates/skald-core/src/tools/fs/search_file.rs b/crates/skald-core/src/tools/fs/search_file.rs index 249500a..684b9b3 100644 --- a/crates/skald-core/src/tools/fs/search_file.rs +++ b/crates/skald-core/src/tools/fs/search_file.rs @@ -1,13 +1,67 @@ +use std::sync::Arc; + use anyhow::Result; use serde_json::{Value, json}; +use sqlx::SqlitePool; -use crate::tools::{Tool, ToolDescriptionLength, truncate_label, MAX_LABEL_SHORT, MAX_LABEL_FULL}; -use super::read_to_string; +use crate::tools::{ + SimpleExecution, Tool, ToolContext, ToolDescriptionLength, ToolExecution, ToolResult, + truncate_label, MAX_LABEL_SHORT, MAX_LABEL_FULL, +}; +use super::{classify_memory, read_to_string, MemScope}; -pub struct SearchFile; +pub struct SearchFile { + /// The `shared-memory` (system) pool; see [`ReadFile`](super::ReadFile). + shared_pool: Arc, +} impl SearchFile { - pub fn new() -> Self { Self } + pub fn new(shared_pool: Arc) -> Self { Self { shared_pool } } +} + +/// Renders the case-insensitive substring search over `text` with context lines. +/// Shared by the on-disk `execute` and the `memory/` routing; `display` is the +/// path shown in the output. +fn render_search(text: &str, args: &Value, display: &str) -> Result { + let query = args["query"].as_str() + .ok_or_else(|| anyhow::anyhow!("Missing required argument: query"))?; + let context = args["context_lines"].as_u64().unwrap_or(3).min(10) as usize; + + let lines: Vec<&str> = text.lines().collect(); + let lower_query = query.to_lowercase(); + let width = lines.len().to_string().len().max(3); + + let matches: Vec = lines.iter().enumerate() + .filter(|(_, l)| l.to_lowercase().contains(&lower_query)) + .map(|(i, _)| i) + .collect(); + + if matches.is_empty() { + return Ok(format!("No matches found for {:?} in {display}.", query)); + } + + let mut chunks: Vec<(usize, usize)> = Vec::new(); + for &m in &matches { + let start = m.saturating_sub(context); + let end = (m + context).min(lines.len() - 1); + if let Some(last) = chunks.last_mut() { + if start <= last.1 + 1 { last.1 = last.1.max(end); continue; } + } + chunks.push((start, end)); + } + + let match_set: std::collections::HashSet = matches.into_iter().collect(); + let mut out = format!("{} match(es) in {display}:\n", match_set.len()); + + for (ci, (start, end)) in chunks.iter().enumerate() { + if ci > 0 { out.push_str(" ···\n"); } + for idx in *start..=*end { + let marker = if match_set.contains(&idx) { ">" } else { " " }; + out.push_str(&format!("{marker}{:>width$} | {}\n", idx + 1, lines[idx])); + } + } + + Ok(out) } impl Tool for SearchFile { @@ -53,48 +107,29 @@ impl Tool for SearchFile { } } + /// Routes `user-memory/…` / `shared-memory/…` to the note store; every other + /// path falls through to the on-disk [`execute`](Self::execute). + fn run_with<'a>(&'a self, ctx: &ToolContext, args: Value) -> Box { + let path = super::path_arg(&args).unwrap_or_default(); + let Some(m) = classify_memory(&path) else { return self.run(args); }; + let pool = match m.scope { + MemScope::User => Arc::clone(&ctx.pool), + MemScope::Shared => Arc::clone(&self.shared_pool), + }; + let rel = m.rel; + + Box::new(SimpleExecution::new(Box::pin(async move { + let Some(doc) = crate::db::memory_docs::get(&pool, &rel).await? else { + anyhow::bail!("No note at {path}"); + }; + Ok(ToolResult::Text(render_search(&doc.content, &args, &path)?)) + }))) + } + fn execute(&self, args: Value) -> Result { let user_path = args["path"].as_str() .ok_or_else(|| anyhow::anyhow!("Missing required argument: path"))?; - let query = args["query"].as_str() - .ok_or_else(|| anyhow::anyhow!("Missing required argument: query"))?; - let context = args["context_lines"].as_u64().unwrap_or(3).min(10) as usize; - let text = read_to_string(user_path)?; - let lines: Vec<&str> = text.lines().collect(); - let lower_query = query.to_lowercase(); - let width = lines.len().to_string().len().max(3); - - let matches: Vec = lines.iter().enumerate() - .filter(|(_, l)| l.to_lowercase().contains(&lower_query)) - .map(|(i, _)| i) - .collect(); - - if matches.is_empty() { - return Ok(format!("No matches found for {:?} in {user_path}.", query)); - } - - let mut chunks: Vec<(usize, usize)> = Vec::new(); - for &m in &matches { - let start = m.saturating_sub(context); - let end = (m + context).min(lines.len() - 1); - if let Some(last) = chunks.last_mut() { - if start <= last.1 + 1 { last.1 = last.1.max(end); continue; } - } - chunks.push((start, end)); - } - - let match_set: std::collections::HashSet = matches.into_iter().collect(); - let mut out = format!("{} match(es) in {user_path}:\n", match_set.len()); - - for (ci, (start, end)) in chunks.iter().enumerate() { - if ci > 0 { out.push_str(" ···\n"); } - for idx in *start..=*end { - let marker = if match_set.contains(&idx) { ">" } else { " " }; - out.push_str(&format!("{marker}{:>width$} | {}\n", idx + 1, lines[idx])); - } - } - - Ok(out) + render_search(&text, &args, user_path) } } diff --git a/crates/skald-core/src/tools/fs/write_file.rs b/crates/skald-core/src/tools/fs/write_file.rs index e88357f..b202f4b 100644 --- a/crates/skald-core/src/tools/fs/write_file.rs +++ b/crates/skald-core/src/tools/fs/write_file.rs @@ -1,13 +1,22 @@ +use std::sync::Arc; + use anyhow::Result; use serde_json::{Value, json}; +use sqlx::SqlitePool; -use crate::tools::{Tool, ToolDescriptionLength, truncate_label, MAX_LABEL_SHORT}; -use super::{resolve, write_string}; +use crate::tools::{ + SimpleExecution, Tool, ToolContext, ToolDescriptionLength, ToolExecution, ToolResult, + truncate_label, MAX_LABEL_SHORT, +}; +use super::{classify_memory, resolve, write_string, MemScope}; -pub struct WriteFile; +pub struct WriteFile { + /// The `shared-memory` (system) pool; see [`ReadFile`](super::ReadFile). + shared_pool: Arc, +} impl WriteFile { - pub fn new() -> Self { Self } + pub fn new(shared_pool: Arc) -> Self { Self { shared_pool } } } impl Tool for WriteFile { @@ -18,7 +27,8 @@ impl Tool for WriteFile { "Create a new file or fully overwrite an existing one. \ Use instead of echo/cat heredoc in the terminal. \ Relative paths are resolved from the project root; absolute paths (starting with /) are used as-is. \ - OVERWRITES the entire file — for targeted edits to an existing file use edit_file instead." + OVERWRITES the entire file — for targeted edits to an existing file use edit_file instead. \ + Write Markdown under user-memory/ (private to you) or shared-memory/ (shared with everyone) to save a durable note in your memory instead of on disk." } fn parameters_schema(&self) -> Value { @@ -48,6 +58,30 @@ impl Tool for WriteFile { truncate_label(&format!("write_file `{path}`"), MAX_LABEL_SHORT) } + /// Routes `user-memory/…` / `shared-memory/…` to the note store; every other + /// path falls through to the on-disk [`execute`](Self::execute). + fn run_with<'a>(&'a self, ctx: &ToolContext, args: Value) -> Box { + let path = super::path_arg(&args).unwrap_or_default(); + let Some(m) = classify_memory(&path) else { return self.run(args); }; + let pool = match m.scope { + MemScope::User => Arc::clone(&ctx.pool), + MemScope::Shared => Arc::clone(&self.shared_pool), + }; + let rel = m.rel; + let content = args["content"].as_str().map(str::to_string); + + Box::new(SimpleExecution::new(Box::pin(async move { + let content = content.ok_or_else(|| anyhow::anyhow!("Missing required argument: content"))?; + if rel.is_empty() { + anyhow::bail!("{path} is a memory root, not a note — write to a path like {path}/notes.md"); + } + let existed = crate::db::memory_docs::get(&pool, &rel).await?.is_some(); + crate::db::memory_docs::upsert(&pool, &rel, &content).await?; + let verb = if existed { "Overwrote" } else { "Created" }; + Ok(ToolResult::Text(format!("{verb} {path} ({} bytes).", content.len()))) + }))) + } + fn execute(&self, args: Value) -> Result { let user_path = args["path"].as_str() .ok_or_else(|| anyhow::anyhow!("Missing required argument: path"))?;