tic: run per-user under a system-agent scheduler, with a run log
Nightly Build / build (push) Successful in 6m58s

Reframe TIC from an ownerless global loop into a per-user system agent.
The events it reads live in each user's own encrypted mcp_events, the
connectors that produced them run in that user's container, and the
notifications go to that user's hub — so the previous design (built
against the ownerless Conversation bundle, writing into system.db and
notifying a hub with no subscribers) was inert by construction.

Core changes
- TicManager owns no timer and no user list. It now exposes
  run_for(user_id, pool, sessions, hub): one tick for one user, over
  deps unpacked from that user's UserContext. Removed from the
  Conversation bundle; Skald::tic_manager() is gone.
- New spawn_system_agents in wiring.rs: one instance-wide loop, spawned
  post-construction with a Weak<Skald> (like spawn_user_lifecycle).
  Each pass walks the directory and runs TIC for one user at a time —
  sequential, because a pass is N container round-trips and N LLM calls
  nobody is waiting on. A ConfigKeyUpdated on the interval key cuts the
  current wait short; enabled is re-read per pass.
- A user whose database is still locked is skipped (normal, not an
  error): the pool is the unlock token, so a user who hasn't logged in
  since restart has no readable events and nowhere to record a skip.
- The configured tic.security_group is re-checked per user through
  run_context::reconcile_group_for_user — a restricted member never
  gets a tool set their role wouldn't grant; unconfigured starts from
  role_default_run_context, never None (None = catch-all = wider).
- New system_agent_runs owner table (no user_id column — the file is
  the owner): start/finish split so a crash leaves a visible 'running'
  row, swept to 'failed' by the next start; safe because the scheduler
  is sequential and single-instance. An idle tick writes nothing.
- counting_notify wraps the notify tool so the run log can report
  notifications emitted without the tool knowing it's counted.
- The session's event channel is drained by a spawned task instead of
  a dropped receiver — the translator awaits its sends and would wedge
  at capacity.

EventLog::{Persist,Discard} on McpManager::new
- mcp_events is an owner table and its only reader (TIC) is per-user,
  so an event is something that happened to someone. The per-user
  runtime gets Persist; the ownerless global runtime gets Discard (its
  pool is system.db, rows would be unattributable and unread).

API + UI
- GET /api/system-agents/runs: the caller's own run history, scoped
  through require_context with no admin override (same promise as the
  rest of the private pool).
- web/components/system-agents.js replaces tic-sessions.js. The old
  #tic debug page inferred runs from leftover ephemeral sessions; the
  new #system-agents page (sidebar group 'extensions', visible to
  everyone — the data is the caller's own) reads the real run log.
- i18n: tic.* keys replaced with system_agents.* in en/it/fr.

Docs
- New docs/system-agents.md (user-facing: what TIC does, why it runs
  per person, why a run can be missing). Updated docs/settings.md and
  docs/index.md.
- agents/tic/AGENT.md reframed per-user: events are that person's,
  memory is user-memory/ (private) — never shared-memory/.
- CLAUDE.md records the system-agents design and the EventLog seam.
This commit is contained in:
2026-07-27 11:39:13 +01:00
parent 305bdbdd2b
commit 165af19774
26 changed files with 1151 additions and 499 deletions
+21 -3
View File
@@ -57,7 +57,7 @@ Domain words are allowed only in seed data, preset labels, UI copy and positioni
### Current state
`UserManager` (§11) is now **consumed**. Login exists (`crates/skald-core/src/auth/mod.rs`: `SessionStore``login`/`user_of`/`logout` plus `revoke_user`, the admin-side "drop every session of this user" used by `Skald::revoke_user_runtime`; the deny-by-default middleware is `src/frontend/api/guard.rs`, whose `require_auth` maps token → id and does **not** re-read the row, which is exactly why revocation must be pushed rather than polled; first admin created by `skald-setup`), and the per-user owner-bound runtime is `UserContext` (`crates/skald-core/src/skald/user_context.rs`) — resolved by `Skald::user_context` / the frontend's `require_context`, keyed off `UserManager::pool_of`, and carrying its **own `CancellationToken`** (a child of the instance one) so a single user's cron/hub/MCP loops can be stopped without touching anyone else's. The frontend owner call-sites (WS, sessions, inbox, approval-pending, projects, uploads, run-context, **cron**) route through the per-user pool; dev/stats read `llm_requests` — a *registry* table — from `system.db`, which is correct. The "owner-without-a-user" question resolved to **there isn't one**: every owner content belongs to a logged-in user (the admin included). The global owner-bound bundles (`Conversation`/`Tasks`: the "ownerless" `ChatSessionManager`, `ChatHub`, cron `TaskManager`, `TicManager`) are still constructed but **inert** — their loops never spawn and nothing consumes their accessors; removing them is pending follow-on work (kept for now because `RunContextManager` shares the `Conversation` bundle and *is* used, being registry-backed). See blueprint §19.
`UserManager` (§11) is now **consumed**. Login exists (`crates/skald-core/src/auth/mod.rs`: `SessionStore``login`/`user_of`/`logout` plus `revoke_user`, the admin-side "drop every session of this user" used by `Skald::revoke_user_runtime`; the deny-by-default middleware is `src/frontend/api/guard.rs`, whose `require_auth` maps token → id and does **not** re-read the row, which is exactly why revocation must be pushed rather than polled; first admin created by `skald-setup`), and the per-user owner-bound runtime is `UserContext` (`crates/skald-core/src/skald/user_context.rs`) — resolved by `Skald::user_context` / the frontend's `require_context`, keyed off `UserManager::pool_of`, and carrying its **own `CancellationToken`** (a child of the instance one) so a single user's cron/hub/MCP loops can be stopped without touching anyone else's. The frontend owner call-sites (WS, sessions, inbox, approval-pending, projects, uploads, run-context, **cron**) route through the per-user pool; dev/stats read `llm_requests` — a *registry* table — from `system.db`, which is correct. The "owner-without-a-user" question resolved to **there isn't one**: every owner content belongs to a logged-in user (the admin included). The global owner-bound bundles (`Conversation`/`Tasks`: the "ownerless" `ChatSessionManager`, `ChatHub`, cron `TaskManager`) are still constructed but **inert** — their loops never spawn and nothing consumes their accessors; removing them is pending follow-on work (kept for now because `RunContextManager` shares the `Conversation` bundle and *is* used, being registry-backed). See blueprint §19.
Direction of travel, decided but not yet executed: strip the **power-user surface** (self-rewriting, arbitrary shell, dev-agent suite, ticket system) and move to a **binary-first** layout — the app is built once and run from a compiled binary, not executed from its own source tree.
@@ -107,6 +107,7 @@ Two rules keep the boundary real, and both are enforced by the compiler:
| `crates/skald-core/src/mcp/` | MCP runtimes + the `McpProvider` seam (§7): the shared host **global** runtime and the per-user **container** runtimes, unioned per session as `UserMcpView`. See the MCP connectors section |
| `crates/skald-core/src/plugin/` | Plugin system: discovery, enable/disable, tool registration, per-user access grants + per-user config |
| `crates/skald-core/src/cron/` | Scheduled job runner |
| `crates/skald-core/src/tic/` | `TicManager`: one tick of the TIC system agent for **one** user (`run_for`). No timer of its own — the instance-wide scheduler is `skald::wiring::spawn_system_agents`. See the system-agents section |
| `crates/skald-core/src/compactor.rs` | Context compaction **policy** — when to compact and with which model; the mechanics are `agent_loop::compaction`. Model for the summary call: the instance-wide Settings pick (`compaction_model`, a `PropertyType::LlmModel` config property declared by `compactor::config_set`) wins; else AUTO by `compaction.strength` (config.yml); a missing configured model degrades to the same AUTO path |
| `crates/skald-core/src/approval/` | Approval rules engine |
| `crates/skald-core/src/clarification/` | `ClarificationManager`: background-session question/answer |
@@ -128,7 +129,7 @@ Two rules keep the boundary real, and both are enforced by the compiler:
The schema is split into two buckets (§5.1), and the split is the point:
- **`create_registry_tables`** — instance-wide, readable without any user key: `users`, `roles`, `llm_providers`, `llm_models`, `transcribe_models`, `tts_models`, `image_generate_models`, `plugins`, `plugin_access` + `plugin_user_configs`, `approval_rules`, `tool_permission_groups`, `config`, `known_tools`, `llm_requests`, `mcp_catalog`, `mcp_global_servers` + `mcp_global_access`, `oauth_providers`, `role_capabilities`, `shared_folders` + `shared_folder_members`, `projects` + `project_members`. The MCP tables back the Connectors model (§7/§14/§15 — see its own section); `oauth_providers` (accessor `db/oauth_providers.rs`) holds one row per identity provider (Google…) — endpoints + `client_id`/`client_secret` + `redirect_uri`, admin-owned household secrets (§4/§15b), never a per-user token. The last two pairs are junction-backed membership: `shared_folder_members` (accessor `db/shared_folders.rs`) for the on-disk shared folders (§6), `project_members` (accessor `db/project_members.rs`) for projects (see the Projects section) — both let a member be read-only (`can_write`) and both drive the container mount topology + the fs routing. Their FKs are registry→registry (same file), which is allowed — unlike an owner→registry key.
- **`create_owner_tables`** — one owner's content, **identical schema in every file that has it**: `chat_sessions`, `chat_sessions_stack`, `chat_history`, `chat_llm_tools`, `chat_summaries`, `session_scratchpad`, `session_mcp_grants`, `stack_mcp_grants`, `scheduled_jobs`, `job_runs`, `mcp_user_servers`, `mcp_events`, `sources`, `secrets`, `llm_request_payloads`, `memory_docs` (+ FTS5 `memory_docs_fts`). `mcp_user_servers` (a user's activated per-user connectors) carries `catalog_name` as a **bare `TEXT` snapshot** of `mcp_catalog.name`, never a FK — an owner→registry key would fail every INSERT; for an OAuth connector it also snapshots `oauth_provider` + `deliver_json`, and its `api_key` column holds the refresh token (in the SQLCipher-encrypted file, so no column crypto). Because `memory_docs` is an owner table, one definition backs **private** memory in each `{userid}.db` and **shared** memory in `system.db` (the household owner) — see the memory namespace note below. (`projects`/`project_tickets` were owner tables in the single-user past: projects are shareable now, so `projects` + `project_members` are registry tables and `project_tickets` is gone.)
- **`create_owner_tables`** — one owner's content, **identical schema in every file that has it**: `chat_sessions`, `chat_sessions_stack`, `chat_history`, `chat_llm_tools`, `chat_summaries`, `session_scratchpad`, `session_mcp_grants`, `stack_mcp_grants`, `scheduled_jobs`, `job_runs`, `system_agent_runs`, `mcp_user_servers`, `mcp_events`, `sources`, `secrets`, `llm_request_payloads`, `memory_docs` (+ FTS5 `memory_docs_fts`). `mcp_user_servers` (a user's activated per-user connectors) carries `catalog_name` as a **bare `TEXT` snapshot** of `mcp_catalog.name`, never a FK — an owner→registry key would fail every INSERT; for an OAuth connector it also snapshots `oauth_provider` + `deliver_json`, and its `api_key` column holds the refresh token (in the SQLCipher-encrypted file, so no column crypto). Because `memory_docs` is an owner table, one definition backs **private** memory in each `{userid}.db` and **shared** memory in `system.db` (the household owner) — see the memory namespace note below. (`projects`/`project_tickets` were owner tables in the single-user past: projects are shareable now, so `projects` + `project_members` are registry tables and `project_tickets` is gone.)
Schema is greenfield (no migrations, §0), but a purely **additive** column lands on an existing DB in place: `db::ensure_column` runs `ALTER TABLE … ADD COLUMN` and swallows the "duplicate column" error, a no-op on a fresh DB where the `CREATE TABLE` already has the column. Used for the OAuth columns on `mcp_catalog` / `mcp_user_servers` so a dev box need not be wiped for an additive change (a full recreate is still valid).
@@ -140,7 +141,7 @@ Schema is greenfield (no migrations, §0), but a purely **additive** column land
**Prompt substitutions**: an `AGENT.md` may carry `<!-- KEY -->` placeholders; `agents::resolve_includes` turns each into a `__KEY__` sentinel, replaced at request time. Two are resolved by the system-context source itself (`loop_adapters/system.rs`) from the session owner (`user_id`) + registry (`shared_pool`), so every source (WS, mobile, cron, sub-agents) gets them with no caller plumbing: `__SHARED_FOLDERS__` (the user's shared-folders table) and `__USER_PROFILE__` (the owner's directory profile: `Name`, `Date of birth` with age computed at build time, `Sex`, `Preferred language`, admin `Notes` — unset values render as explicit `unknown` / `not specified`, the `Notes` line is omitted when empty). Any other key comes from the per-call `SendMessageOptions::system_substitutions` map.
`system.db` still gets **both** bucket functions — but no longer because the migration is unstarted. It gets the owner schema because it *is* the owner of **shared** memory (`memory_docs`) plus, for now, the globally-scoped `secrets` and the `mcp_events` lifecycle log (`SecretsStore` and the global `McpManager` are built on the system pool and shared by reference into every `UserContext`; the global runtime's *config* now lives in the registry table `mcp_global_servers`, and per-user connector config in each user's owner `mcp_user_servers`). Every *other* owner table is created there but never written to anymore — the global owner-bound managers that would write them (chat/jobs/etc.) are inert (see "Current state"). Fully dropping `create_owner_tables` from `system.db` is blocked on the §4 scope decision for secrets (plus the residual global `mcp_events` log), not on call-site migration.
`system.db` still gets **both** bucket functions — but no longer because the migration is unstarted. It gets the owner schema because it *is* the owner of **shared** memory (`memory_docs`) plus, for now, the globally-scoped `secrets` (`SecretsStore` is built on the system pool and shared by reference into every `UserContext`; the global runtime's *config* now lives in the registry table `mcp_global_servers`, and per-user connector config in each user's owner `mcp_user_servers`). The global runtime no longer writes `mcp_events` there: notification persistence is an explicit `McpManager::new` argument (`EventLog::{Persist,Discard}`), `Discard` for the ownerless global runtime and `Persist` for each per-user one, because an event belongs to whoever it happened to and its only reader (TIC) is per-user. Every *other* owner table is created there but never written to anymore — the global owner-bound managers that would write them (chat/jobs/etc.) are inert (see "Current state"). Fully dropping `create_owner_tables` from `system.db` is blocked on the §4 scope decision for secrets, not on call-site migration.
`users` (`crates/skald-core/src/db/users.rs`) holds the directory plus auth material. It lives in the system DB, which the box owner can read, so it must never store anything that derives a user's key. `Credentials` is an enum mirroring the table's `CHECK`: an encrypted user carries a **wrapped DEK** (whose AEAD tag *is* the password verifier — hence no `password_hash`); a cleartext user carries an ordinary verifier, or none. `User` is deliberately not `Serialize` and its `Debug` redacts key material — use `User::summary()` for anything leaving the process. `role_id` references `roles(id)` (the `roles` table is now seeded before `users` in `create_registry_tables`). A nullable `locale` column (additive via `ensure_column`) holds the per-user UI language override; role-driven conventions live in the free-form `roles.attrs` JSON — never new columns per attribute — parsed at a **single point** by the typed `db::roles::RoleAttrs` (`ui_mode`, `permission_groups`, `chat_agent`): `ui_mode` (see the frontend section) plus the role's **security-group set** (`roles.permission_group` = the default group, `attrs.permission_groups` = additional allowed groups; `Role::effective_groups()` = the union, `roles::role_allows_group()` gates it with `admin` short-circuiting to all). See the security-group picker in the frontend section. The role's **default entry (chat) agent** is `attrs.chat_agent` — the neutral `chat`-type agent members of the role land on (§0.1: data, not an enum). Resolved by `roles::default_chat_agent_for_user(registry_pool, user_id)` — the single seam behind both the per-user `ChatHub`'s `default_agent` (snapshotted at login in `UserContextFactory::build`, like fs/MCP access, so **every** session-creation path — explicit `provision_session`, lazy WS `get_or_create_session`, notify — honors it) and `provisioning_for_source`'s non-project branch. Falls back to `agents::DEFAULT_CHAT_AGENT` (`"assistant"`, the renamed former `main`) when unset. Seeded: `admin`/`member``assistant`, `children``kid` (Companion). A per-user override is future work, layering on top in the same resolver. The stack **root frame** is created with the session's own `agent_id` (not a literal) — `config.agent_id` (from the frame) drives which prompt runs, so a wrong id there silently runs the wrong agent. The admin-managed **directory profile** lives in three more additive columns — `birthdate` (ISO `YYYY-MM-DD`), `sex` (free text), `notes` (admin-authored) — edited only from the Users admin page (`set_directory_fields`; validation — real non-future date, length caps — lives in the `users_mgmt` API, not the db layer) and rendered into agent prompts by the `__USER_PROFILE__` substitution (see above). They are directory metadata written *by* the admin *about* the user, so the registry is their honest home under the §2 threat model.
@@ -215,6 +216,22 @@ For a per-user connector whose credential is produced by **pairing** (`auth.type
**Deferred:** SSH and other §15 device kinds (would reuse the `login_status` contract), `deliver.as=file`, and non-Google OAuth providers are unimplemented paths that error clearly rather than half-work. No boot seed of catalog presets; the admin populates the catalog from the Marketplace.
## System agents (TIC)
A **system agent** runs on a user's behalf without being asked. TIC — the background event processor — is the only one, and the surface is written so a second needs no new machinery.
**It is per-user, and every part of the design falls out of that.** The events it reads (`mcp_events`) are in the caller's own encrypted database, pushed there by connectors running in the caller's container; the notification it emits goes to the caller's hub; the trace it leaves (`system_agent_runs`) is in that same file. `TicManager` (`crates/skald-core/src/tic/`) therefore owns **no timer and no user list**: it exposes `run_for(user_id, pool, sessions, hub)`, one tick for one user, over deps unpacked from that user's `UserContext`. Building it against the ownerless `Conversation` bundle was exactly what made the pre-multi-user version inert — it wrote sessions into `system.db`, notified a hub with no subscribers, and resolved tool paths against a container that does not exist.
**One scheduler, sequential.** `skald::wiring::spawn_system_agents` is the instance-wide loop — spawned post-construction with a `Weak<Skald>`, like `spawn_user_lifecycle` and for the same reason (it resolves per-user runtimes through `Skald::user_context`). Each pass walks the directory and runs the agent for one user at a time: a pass is N container round-trips and N LLM calls, and nobody is waiting on a background tick, so concurrency would only spike the box every interval. A `ConfigKeyUpdated` on the interval key cuts the current wait short; the enabled flag is re-read per pass.
**A locked user is skipped, and that is the normal case, not an error.** The pool is the unlock token (§9): a user who has not logged in since the last restart has no readable events, no session store — and no place to record the skip, since the only file that could hold it is the one we cannot open. Hence `system_agent_runs` has no `skipped` status: the skip is an INFO log line and nothing else. Their events keep accumulating and the first pass after they log in picks them up.
**The run log is theirs, not the admin's** (`db/system_agent_runs.rs`, owner table, no `user_id` column — the file is the owner). A run summarises what landed in someone's inbox, so `GET /api/system-agents/runs` is scoped through `require_context` with **no admin override**: everyone, admin included, sees their own runs. `stats` is a JSON blob of the agent's own counters (never event contents). The write is split `start`/`finish` (unlike `job_runs`, written once at the end) so a crash leaves a visible `running` row, swept to `failed` by the next `start` for that agent — safe precisely because the scheduler is sequential and single-instance. An **idle tick writes nothing**: a row only exists when there were events, or the log becomes a heartbeat instead of a history.
**The configured security group is not applied verbatim.** `tic.security_group` is an instance-wide admin setting; handing it to a restricted member's run would give their background agent a tool set their role never granted. It goes through `run_context::reconcile_group_for_user` — the same seam a persisted group takes — degrading to the role default when the role disallows it. With nothing configured the run still starts from `role_default_run_context`, never `None`, because `None` means the catch-all group, which is *wider*.
UI: `#system-agents` (`web/components/system-agents.js`, sidebar group `extensions`, visible to everyone — there is nothing to gate when the data is the caller's own). It replaced the old `#tic` "TIC Sessions" debug page, which listed `chat_sessions WHERE source='tic'` and so inferred runs from leftover ephemeral sessions rather than recording them.
## Multimodal attachments
Uploads go through **one centralized seam**`ChatHub::save_upload` (behind `ChatHubApi::save_upload`, backed by `skald_core::uploads::save_to_home`) — so every surface persists identically and no two callers can drift on placement (the class of bug where the agent was handed a path it couldn't reach). The seam writes into the **caller's container home** under `uploads/{session_id}/` (agent path `uploads/{session}/{name}`, the `UPLOADS_SUBDIR` const in `core-api/user_fs.rs`), collision-dedupes the name, and prefers the sniffed magic-byte MIME over the client claim. The **web** handler (`POST /api/{source}/uploads`) buffers each field with a 256 MiB cap then calls the seam; the **Telegram** plugin downloads bytes then calls the same seam via `handle.chat_hub().save_upload("telegram", …)`. Because the file lands in the home (bind-mounted at `/root`), it is reachable by the fs-tools, `execute_cmd`, and the file viewer (`GET /api/file`, per-user via `resolve_view_path`) — there is **no** `/data` static route anymore (removed: it was `require_auth`-only, not ownership-scoped, and also exposed internal server state under `data/`). Attachment metadata travels as structured JSON in `chat_history.metadata` — never as persisted text.
@@ -375,6 +392,7 @@ The role editor (`roles-page.js`) sets the default group + an allowed-groups che
| `plugin-catalog.js` | `<plugin-catalog>` | `#plugin-catalog` — admin status board: one card per plugin (enable toggle + health dot + Configure → `#plugin-detail`) |
| `plugin-detail.js` | `<plugin-detail>` | `#plugin-detail?id=<id>` — one plugin's admin page: instance-config form (`config_schema`) + per-user access checklist (plugin twin of `connector-detail.js`) |
| `plugin-page-host.js` | `<plugin-page-host>` | Host for plugin-contributed pages (`#plugin/<plugin_id>/<page_id>`): dynamic-imports the fragment module, registers its element, mounts it with `plugin-id` |
| `system-agents.js` | `<system-agents-page>` | `#system-agents` — the caller's own run history for the background system agents (TIC): agent, start, status, duration, counters; row → the run's session |
| `shared-folders.js` | `<shared-folders-page>` | `#shared-folders` — admin-only CRUD for on-disk shared folders (§6): create/describe/delete + per-member read-only/read-write grants; description feeds the assistant's `__SHARED_FOLDERS__` context |
| `projects/` | `<projects-page>` | `#projects` — host + list + board; the board is tabbed (**Files** explorer with live watcher + write actions, **Sharing** members), deep-linked `#projects/{id}[/sharing]`. See the Projects section |
| `connector-detail.js` | `<connector-detail-page>` | A connector's own page (`#connector?name=X`): env/secret form + Test, the **OAuth login panel** (sign in → paste code → complete, §15), global enable + per-user access grants |