Daniele
902f47ecd8
Nightly Build / build (push) Successful in 10s
CLAUDE.md had grown to 152 KB (~21k words, ~40k tokens) and is loaded into
every coding-agent session. The cost is not the cache read, it is attention:
the rules that are genuinely invariant were drowning in the mechanics of
subsystems that most tasks never touch.
The split criterion is blast radius, not importance. A rule a change anywhere
could violate stays in CLAUDE.md — the commit rule, the production/schema
constraint, domain neutrality, the event-bus rule, the crate boundaries, and
the module map. The mechanism of one subsystem moves to dev-docs/, opened on
entry to that subsystem via a routing table at the top of CLAUDE.md.
Nothing was rewritten: every section was moved verbatim by line range and
verified line-by-line against the original. The only edits are cross-reference
repairs ("see the DB section" -> a link), the promotion of headings in the
extracted files, and a condensed "Current state" whose full text now lives in
dev-docs/users-auth-and-boot.md.
CLAUDE.md: 152 KB -> 31 KB. Twelve subsystem files plus an index under
dev-docs/, which now carries the same standing rule as docs/ and CHANGELOG.md:
a change to a subsystem updates its dev-doc in the same change.
No CHANGELOG entry: this is documentation for coding agents with no observable
effect on the application.
54 lines
12 KiB
Markdown
54 lines
12 KiB
Markdown
*Skald dev-docs — architectural reference for coding agents. Index: [README.md](README.md) · Entry point: [../CLAUDE.md](../CLAUDE.md)*
|
||
|
||
**Read this when:** you touch MCP, connectors, the marketplace install path, OAuth or device login.
|
||
|
||
---
|
||
|
||
# MCP connectors (blueprint §7/§14/§15)
|
||
|
||
MCP servers are surfaced to users as **"Connectors"** (UI naming; `mcp`/schema stays neutral, §0.1). The old single owner table `mcp_servers`, the agent-facing `register_mcp`/`delete_mcp` tools, and the `mcp` kinds of `list_items`/`toggle_item` are **gone**. Connectors are now admin-curated and user-activated through the Connectors UI/API — never written by the agent, which closes the §14 RCE vector (prompt-injection → agent writes+registers a local script → arbitrary code on the box).
|
||
|
||
**Two runtimes, one view (§7).** A session's MCP tools are the **union** of:
|
||
|
||
- **Global runtime** — shared, stateless connectors (web-search, Tavily…) that run on the **host**, connected at boot from `mcp_global_servers` by `McpManager::initialize`. Filtered per user by `mcp_global_access`.
|
||
- **Per-user runtime** — the connectors a user has activated, run **inside their container**, started at first login from that user's owner `mcp_user_servers` and living until restart (§9; the `docker exec -i` children die via `kill_on_drop` when the `UserContext` drops).
|
||
|
||
`McpProvider` (`mcp/provider.rs`) is the trait the session code talks to, so `all_tool_defs` / `render_mcp_list` / `ActivateTools` never learn which runtime owns a server. `McpManager` implements it directly (used for the inert ownerless bundle, §19); `UserMcpView` implements it as `global ∪ user`, where `accessible_global` is a snapshot of `mcp_global_access` captured when the `UserContext` is built (like fs membership). Both runtimes share `McpManager::connect_all(specs, boot)`; `McpServerSpec` + `global_row_spec`/`user_row_spec` turn a DB row into a connectable spec (a per-user `local_script` spec targets the user's container).
|
||
|
||
**Authorization is a capability on the role, not `if role==admin`** (§0.1/§14 — `db/role_capabilities.rs`): `mcp.register_remote` + `mcp.register_local_from_catalog` are self-service (seeded on every new role by `roles::create` via `seed_defaults`); `mcp.register_local_script` + `mcp.manage_catalog` are admin-only. `admin` holds every capability by construction (short-circuit in `has()`). API handlers gate through `require_cap`.
|
||
|
||
**Tables** (see [database.md](database.md)) — registry: `mcp_catalog` (admin-vetted templates; holds only the *schema* of what an activation must supply, never live creds — plus, for OAuth, `oauth_provider` + `oauth_scopes_json` + `deliver_json`), `mcp_global_servers` + `mcp_global_access`, `oauth_providers` (per-provider client creds), `role_capabilities`. Owner: `mcp_user_servers` (per-user activations; `api_key` encrypted at rest — the refresh token for an OAuth one — `catalog_name`/`oauth_provider`/`deliver_json` bare `TEXT` snapshots).
|
||
|
||
**Endpoints** (`src/frontend/api/mcp.rs`, mounted in `api/mod.rs`) — admin: `/mcp/catalog` (GET/POST/DELETE), `/mcp/global` (list/enable/delete + `/{id}/access` GET/PUT), `/mcp/providers` (GET/POST + DELETE `/{name}` — OAuth provider creds, secret never returned to the browser). User: `/mcp/available`, `/mcp/activate`, `/mcp/activated` (+ DELETE `/{id}` to deactivate), `/mcp/oauth/start` + `/mcp/oauth/complete` (the §15 OAuth login), `/mcp/login/status` + `/mcp/login/reset` (the §15 QR/device login — see below). `connectors.js` (`<connectors-page>`) is the **single** Connectors surface — a row list, one row per connector (there is no separate catalog page): the user view (activate/deactivate + granted globals) always, plus the admin affordances when `role_id === 'admin'` — the **Add connector** dropdown (from the Marketplace, or manually via the `#connectors/new` sub-page), per-row removal from the catalog, and the **Sign-in providers** modal. The Marketplace stays its own page (`marketplace.js`), reached from that dropdown and linking back to `#connectors`. `connector-detail.js` (`<connector-detail-page>`) is a connector's own page and hosts both the OAuth login panel and the QR login panel.
|
||
|
||
**Dependency reconciler (`mcp::install::ensure_installed`).** Copying a local-script connector's files into a container never installed its deps. `ensure_installed` closes that: a **content-hash reconciler** keyed on the connector's *source* files (not a version string) that, when the hash changed, re-copies the files and installs deps inside the container — `npm ci --omit=dev` (node, from `package.json`) and/or `pip install --target .pydeps` (python, from `requirements.txt`, put on the server's `PYTHONPATH` by `user_row_spec`). Runs at activation **and** on every per-user startup path (`UserContext` build, remount) via `mcp::prepare_local_connector`, so a fresh container installs from scratch, an updated connector re-installs, and an unchanged one is a hash-match no-op. Deps are therefore **never vendored** — connectors ship `package.json`/`requirements.txt`, not `node_modules/`. Authoring contract for connectors lives in the marketplace repository — see below.
|
||
|
||
**The host half has no reconciler, so its call sites are the contract.** A `global` connector runs in the Skald process, not a container, and `ensure_installed_host` is not hash-guarded — it leans on `pip`/`npm` being idempotent, which is only safe as long as *every* path that lands new files also calls it. There are two: `global_enable` (the admin saving a connector's config) and, since it was missing, the global branch of `Skald::refresh_connector_after_reinstall`. Without the second, a marketplace **Update** that *adds* a `requirements.txt` copied the file and restarted the server without installing anything — the connector came back exactly as broken, and the only cure was re-saving its config. Note what that asymmetry cost: the per-user branch of the same function had always reinstalled (`prepare_local_connector`), so the bug was invisible on anything `scope: user`.
|
||
|
||
**The verify runs with `.pydeps` on `PYTHONPATH`, and must** (`mcp::verify::verify_env`). Only the *server* launch used to get that path (`global_row_spec` / `user_row_spec`); the verify is a bare `sh -c` inheriting nothing, so a python connector was rejected **by its own verify** for a dependency sitting installed one directory away — and `global_enable` installs *before* it verifies, so the deps were provably there at the moment the check denied them. The failure selected for well-written connectors: declaring no `verify` meant never meeting it. The workdir *is* the connector dir in both targets, so the path is derived, not plumbed, and set with `or_insert` — an explicit `PYTHONPATH` from the form is the author's. One gap left deliberately: `POST /api/mcp/test` (the Test button) shares `run_verify` but **not** `ensure_installed_host`, so testing a python connector that was never enabled on this box still fails on the missing deps. Making a "try it" button write to disk for minutes is the worse trade; enable first.
|
||
|
||
**The connector specs live in the marketplace repository, not here.** The feed and everything that authors for it are a separate repo, checked out at `~/projects/marketplace`, and the authoring contract is **`CONNECTOR_MANIFEST_GUIDE.md` at its root** — same filename this repo used to carry a copy of, which is exactly why the copy is gone: two files with one name drift, and the one next to the connectors is the one an author reads. **If the connector specification ever has to change, that is the file to consult and to edit** — nothing in this repo restates it.
|
||
|
||
**Connector versioning.** `mcp_catalog` carries `version` (INTEGER — the update-comparison key), `version_string` (semver, display) and `version_release_date` (ISO, display), snapshotted from the feed on install. The marketplace list computes `update_available` = feed `version` > installed `version` (strict) and surfaces it as an "Update" button (`marketplace.js`). The integer is the UI signal; the actual re-install trigger is the reconciler's content-hash.
|
||
|
||
## OAuth per-user connectors (blueprint §15 — copy-paste flow)
|
||
|
||
OAuth2 authorization-code + PKCE is wired for per-user connectors (Gmail is the first). The consent is a **human copy-paste**, not a headless action: no callback route into the (NAT'd, hostname-less) box, and no client secret on the public feed.
|
||
|
||
- **Providers, not per-connector URLs.** The client is per-**provider** (one Google app covers Gmail/Calendar/Drive): `oauth_providers` holds `auth_url`/`token_url`/`client_id`/`client_secret`/`redirect_uri`/`extra_params`, admin-entered via the Sign-in-providers modal (Google preset fills all but the two secrets; `redirect_uri` = the static `oauth/show.html` page, `extra_params` = `access_type=offline`+`prompt=consent` so Google returns a refresh token). The manifest only names `auth.provider` + `auth.scopes` + `auth.deliver` — never URLs or secrets (feed is remote data, §14).
|
||
- **Flow** (`mcp/oauth.rs`): `activate` on an OAuth catalog entry persists a **pending** `mcp_user_servers` row (files installed, command wired, no token) and returns `needs_oauth` — it does **not** start the server. `/mcp/oauth/start` builds the consent URL (PKCE S256 + opaque `state`) and stashes the verifier in a RAM-only, TTL'd flow store keyed by `state`; the user approves in a browser, the provider lands the code on `oauth/show.html`, they paste it back. `/mcp/oauth/complete` exchanges code+verifier for a refresh token (`client_secret` sent server-side), stores it in the row's `api_key`, flips to `ready`, and starts the server. PKCE makes an intercepted code worthless; a restart drops in-flight flows (mirrors the RAM-only session model).
|
||
- **Credential delivery = env, nothing on disk.** The manifest's `deliver` (`{as,format,env}`, parsed as `mcp::DeliverSpec`) says how the token reaches the server. `user_row_spec_resolved` assembles the credential (`google_authorized_user` JSON = client creds from the provider + refresh token) and injects it as an env var (`GMAIL_CREDS_JSON`) on the `docker exec` — never a file, coherent with §2 (the tempted admin doesn't read `/proc`). The server reads it via `Credentials.from_authorized_user_info`. Ran both at OAuth-complete and at login-time per-user startup.
|
||
- **Google needs a Web-application client**: a Desktop client rejects an `https://` redirect (loopback only), so the `oauth/show.html` redirect must be registered on a **Web app** OAuth client, and exact-match under Authorized redirect URIs — `redirect_uri_mismatch` otherwise.
|
||
|
||
## QR / interactive device login (blueprint §15 — polling flow)
|
||
|
||
For a per-user connector whose credential is produced by **pairing** (`auth.type: "qr"`; WhatsApp is the first, on Baileys — the slim `skald-runtime` image has no Chromium, so a browser-based client is out), there is no code to paste and the server must **run** to produce the QR. The seam is a generic tool contract, reusable for future device kinds (SSH…):
|
||
|
||
- **`login_status` tool contract.** A connector needing an interactive login exposes one tool, `login_status`, returning JSON `{state, qr?, message}` (state: `connecting|need_scan|ready|logged_out`; `qr` is a data-URL PNG only while `need_scan`). Skald calls it **directly, never the agent**.
|
||
- **Flow.** `activate` on a `qr` entry inserts a **pending** `mcp_user_servers` row and **starts** the server (unlike OAuth, which defers), returning `needs_login`/`login_kind:"qr"`. `/mcp/login/status` ensures the server is running (restarts a pending one), calls `login_status`, and returns its state; on `ready` it flips the row's `auth_state` so `all_startable` picks it up next login. `/mcp/login/reset` calls the connector's `logout` tool to re-arm (link a different device). The `connector-detail.js` QR panel polls `login/status` and renders the QR.
|
||
- **Credential = on-disk session, not a token.** The connector persists its session inside its own dir (e.g. `./auth/`), under the bind-mounted home so it survives a container recreate — the honest §4 gap (admin-root-readable), not `memory_docs`.
|
||
- **Node 18 gotcha**: the container ships Node 18; Baileys uses the Web Crypto global, so the server must `globalThis.crypto ??= require('crypto').webcrypto` or it dies pre-QR with "crypto is not defined".
|
||
|
||
**Deferred:** SSH and other §15 device kinds (would reuse the `login_status` contract), `deliver.as=file`, and non-Google OAuth providers are unimplemented paths that error clearly rather than half-work. No boot seed of catalog presets; the admin populates the catalog from the Marketplace.
|
||
|