feat(memory): dual-pool memory namespace, FTS search, and prompt injection
Add a virtual memory namespace backed by SQLite, surfaced through the fs-tools, with private (per-user) and shared (system) stores. Storage - `memory_docs` owner table + external-content FTS5 index with sync triggers. - `db/memory_docs.rs` accessor: get / upsert / list / search (bm25+snippet) / delete. Routing (tools/fs) - `classify_memory` splits paths on the raw first component; `..` clamps inside the store, never escaping to disk. - read/write/list/edit/insert/replace/search_file route `user-memory/` to the owner pool and `shared-memory/` to the system pool (a singleton captured in `register_all`); every other path stays on disk. Each tool extracts a pure transform shared between its disk and memory paths. - New `memory_search` tool over the FTS index (scope private/shared/all), with a sanitised FTS5 query. grep_files stays disk-only. Approval - `user-memory/*` allow (read+write); `shared-memory/*` reads allow, writes require approval so the agent can't silently push one person's data into shared memory. `memory_search` allowed via a path-less rule. - migrate away the old `memory/*` and blanket `shared-memory/*` rows. Prompt injection - `MessageBuilder::load_inject_memory` reads `user-memory/` (owner pool) and `shared-memory/` (system pool) inject entries from SQLite; disk paths unchanged. The system pool is threaded ChatSessionManager -> handler -> MessageBuilder. - main and project-coordinator inject `user-memory/index.md` + `shared-memory/index.md`; common/memory.md rewritten for the two stores.
This commit is contained in:
@@ -69,7 +69,7 @@ Two rules keep the boundary real, and both are enforced by the compiler:
|
||||
| `crates/skald-core/src/chat_hub/` | `ChatHub`: broadcast events to all connected WS clients |
|
||||
| `crates/skald-core/src/chat_event_bus.rs` | Global async bus for cross-session events |
|
||||
| `crates/skald-core/src/agents.rs` | Discovers agents from `agents/*/`, loads meta + system prompt |
|
||||
| `crates/skald-core/src/tools/` | Built-in tools: `exec`, `restart`, `list_agents`, `fs/*`, `notify`, `ast_outline`, `image_generate`, MCP tools, plugin tools, cron tools |
|
||||
| `crates/skald-core/src/tools/` | Built-in tools: `exec`, `restart`, `list_agents`, `fs/*` (also route `user-memory/`/`shared-memory/` to `memory_docs` — see DB tables), `notify`, `ast_outline`, `image_generate`, MCP tools, plugin tools, cron tools |
|
||||
| `crates/skald-core/src/tool_catalog.rs` | `ToolCatalog`: unified tool listing façade (wraps ToolRegistry + McpManager) |
|
||||
| `crates/skald-core/src/events.rs` | `ServerEvent` enum streamed over WebSocket to the frontend |
|
||||
| `crates/skald-core/src/db/` | sqlx SQLite — see below |
|
||||
@@ -100,10 +100,14 @@ Two rules keep the boundary real, and both are enforced by the compiler:
|
||||
The schema is split into two buckets (§5.1), and the split is the point:
|
||||
|
||||
- **`create_registry_tables`** — instance-wide, readable without any user key: `users`, `llm_providers`, `llm_models`, `transcribe_models`, `tts_models`, `image_generate_models`, `plugins`, `approval_rules`, `tool_permission_groups`, `config`, `known_tools`, `llm_requests`.
|
||||
- **`create_owner_tables`** — one owner's content, **identical schema in every file that has it**: `chat_sessions`, `chat_sessions_stack`, `chat_history`, `chat_llm_tools`, `chat_summaries`, `session_scratchpad`, `session_mcp_grants`, `stack_mcp_grants`, `scheduled_jobs`, `job_runs`, `mcp_servers`, `mcp_events`, `sources`, `secrets`, `projects`, `project_tickets`.
|
||||
- **`create_owner_tables`** — one owner's content, **identical schema in every file that has it**: `chat_sessions`, `chat_sessions_stack`, `chat_history`, `chat_llm_tools`, `chat_summaries`, `session_scratchpad`, `session_mcp_grants`, `stack_mcp_grants`, `scheduled_jobs`, `job_runs`, `mcp_servers`, `mcp_events`, `sources`, `secrets`, `projects`, `project_tickets`, `llm_request_payloads`, `memory_docs` (+ FTS5 `memory_docs_fts`). Because `memory_docs` is an owner table, one definition backs **private** memory in each `{userid}.db` and **shared** memory in `system.db` (the household owner) — see the memory namespace note below.
|
||||
|
||||
**No foreign key in the owner bucket may point at a registry table.** SQLite cannot enforce a key across files, not even through `ATTACH`, and sqlx turns on `PRAGMA foreign_keys`: the `CREATE TABLE` succeeds and every `INSERT` fails. `db::tests::owner_tables_stand_alone_with_foreign_keys_on` enforces this by running the owner schema against a database holding nothing else, then inserting a row into each table. Two keys crossed and were fixed: `chat_history.model_db_id` (dropped — write-only, and `llm_requests.model_name` already records the model) and `project_tickets.job_id` (fixed by moving `projects`/`project_tickets` into the owner bucket).
|
||||
|
||||
**Memory namespace (blueprint §5).** `memory_docs` (accessor `db/memory_docs.rs` — `get`/`upsert`/`list`/`search`(FTS)/`delete`) backs a virtual note store surfaced through the fs-tools, **not** the disk. Two sibling roots (not the blueprint's nested `memory/{userid}` + `memory/shared`): `user-memory/…` routes to the caller's own pool (`ToolContext::pool`), `shared-memory/…` to the system pool (a singleton captured in `fs::register_all`). `tools/fs/classify_memory()` decides on the raw first path component (a `..` in the tail clamps inside the store, never escapes to disk); `read_file`/`write_file`/`list_files`/`edit_file`/`insert_at_line`/`replace_lines`/`search_file` override `run_with` to route memory paths (each extracting a pure transform shared with its on-disk `execute`) and leave every other path on disk. Approval (seeded in `seed_fs_path_rules`): `user-memory/*` is `@fs_any allow` (private, frictionless); `shared-memory/*` is `@fs_read allow` + `@fs_write require` — reads free, **writes need approval** so the agent can't silently push one person's data into shared memory. `grep_files` stays disk-only (regex-across-tree ≠ FTS); ranked full-text recall over notes is a separate tool, `memory_search` (`tools/fs/memory_search.rs`), over the `memory_docs` FTS index — allowed by a path-less rule (it takes `query`, not `path`).
|
||||
|
||||
**Memory injection into the prompt**: `MessageBuilder::load_inject_memory` routes each `meta.inject_memory` entry — `user-memory/…` → owner pool, `shared-memory/…` → the shared (`system.db`) pool, both via `memory_docs::get`; anything else (`data/…`, `$WD/…`) is a disk read. The shared pool is threaded `ChatSessionManager` → handler → `MessageBuilder`. `main` and `project-coordinator` inject `user-memory/index.md` + `shared-memory/index.md`.
|
||||
|
||||
`system.db` currently gets **both** bucket functions, because nothing has migrated to per-user pools yet. That is transitional.
|
||||
|
||||
`users` (`crates/skald-core/src/db/users.rs`) holds the directory plus auth material. It lives in the system DB, which the box owner can read, so it must never store anything that derives a user's key. `Credentials` is an enum mirroring the table's `CHECK`: an encrypted user carries a **wrapped DEK** (whose AEAD tag *is* the password verifier — hence no `password_hash`); a cleartext user carries an ordinary verifier, or none. `User` is deliberately not `Serialize` and its `Debug` redacts key material — use `User::summary()` for anything leaving the process. `role_id` has no foreign key yet: sqlx enables `PRAGMA foreign_keys`, so referencing the not-yet-existing `roles` table would fail every insert.
|
||||
|
||||
Reference in New Issue
Block a user