feat(mcp): OAuth per-user connectors (§15) — providers, PKCE copy-paste flow, env credential delivery

- oauth_providers registry table (per-provider client creds) + db/oauth_providers.rs
- mcp/oauth.rs: authorization-code + PKCE S256, RAM-only TTL'd flow store, copy-paste consent
- mcp/install.rs + verify.rs: connector file install + manifest verification
- activate persists a pending row (needs_oauth); /mcp/oauth/start + /complete exchange code for refresh token
- credential delivery via env var on docker exec (google_authorized_user JSON), never on disk
- mcp_catalog/mcp_user_servers: additive OAuth columns (ensure_column), catalog_name/oauth_provider/deliver_json bare TEXT snapshots
- frontend: connector-detail.js (OAuth login panel), shared/connector-common.js, connectors.js admin Sign-in providers modal
- API: /mcp/providers (admin OAuth creds), /mcp/oauth/start|complete
- .gitignore: add /homes/ (instance data), /connectors/, /reset.sh; drop stale /secrets/
This commit is contained in:
2026-07-17 21:47:51 +01:00
parent bcd8f7b5c0
commit e6c4e202a4
28 changed files with 3349 additions and 553 deletions
+19 -6
View File
@@ -100,8 +100,10 @@ Two rules keep the boundary real, and both are enforced by the compiler:
The schema is split into two buckets (§5.1), and the split is the point:
- **`create_registry_tables`** — instance-wide, readable without any user key: `users`, `roles`, `llm_providers`, `llm_models`, `transcribe_models`, `tts_models`, `image_generate_models`, `plugins`, `approval_rules`, `tool_permission_groups`, `config`, `known_tools`, `llm_requests`, `mcp_catalog`, `mcp_global_servers` + `mcp_global_access`, `role_capabilities`, `shared_folders` + `shared_folder_members`. The MCP four back the Connectors model (§7/§14/§15 — see its own section). The last two (accessor `db/shared_folders.rs`) are the **membership** of the on-disk shared folders (§6): a junction table so a member can be read-only (`can_write`) and so the container mount topology + the `shared/{X}` fs routing both query it. FK `shared_folder_members.user_id → users(id)` is registry→registry (same file), which is allowed — unlike an owner→registry key.
- **`create_owner_tables`** — one owner's content, **identical schema in every file that has it**: `chat_sessions`, `chat_sessions_stack`, `chat_history`, `chat_llm_tools`, `chat_summaries`, `session_scratchpad`, `session_mcp_grants`, `stack_mcp_grants`, `scheduled_jobs`, `job_runs`, `mcp_user_servers`, `mcp_events`, `sources`, `secrets`, `projects`, `project_tickets`, `llm_request_payloads`, `memory_docs` (+ FTS5 `memory_docs_fts`). `mcp_user_servers` (a user's activated per-user connectors) carries `catalog_name` as a **bare `TEXT` snapshot** of `mcp_catalog.name`, never a FK — an owner→registry key would fail every INSERT. Because `memory_docs` is an owner table, one definition backs **private** memory in each `{userid}.db` and **shared** memory in `system.db` (the household owner) — see the memory namespace note below.
- **`create_registry_tables`** — instance-wide, readable without any user key: `users`, `roles`, `llm_providers`, `llm_models`, `transcribe_models`, `tts_models`, `image_generate_models`, `plugins`, `approval_rules`, `tool_permission_groups`, `config`, `known_tools`, `llm_requests`, `mcp_catalog`, `mcp_global_servers` + `mcp_global_access`, `oauth_providers`, `role_capabilities`, `shared_folders` + `shared_folder_members`. The MCP tables back the Connectors model (§7/§14/§15 — see its own section); `oauth_providers` (accessor `db/oauth_providers.rs`) holds one row per identity provider (Google…) — endpoints + `client_id`/`client_secret` + `redirect_uri`, admin-owned household secrets (§4/§15b), never a per-user token. The last two (accessor `db/shared_folders.rs`) are the **membership** of the on-disk shared folders (§6): a junction table so a member can be read-only (`can_write`) and so the container mount topology + the `shared/{X}` fs routing both query it. FK `shared_folder_members.user_id → users(id)` is registry→registry (same file), which is allowed — unlike an owner→registry key.
- **`create_owner_tables`** — one owner's content, **identical schema in every file that has it**: `chat_sessions`, `chat_sessions_stack`, `chat_history`, `chat_llm_tools`, `chat_summaries`, `session_scratchpad`, `session_mcp_grants`, `stack_mcp_grants`, `scheduled_jobs`, `job_runs`, `mcp_user_servers`, `mcp_events`, `sources`, `secrets`, `projects`, `project_tickets`, `llm_request_payloads`, `memory_docs` (+ FTS5 `memory_docs_fts`). `mcp_user_servers` (a user's activated per-user connectors) carries `catalog_name` as a **bare `TEXT` snapshot** of `mcp_catalog.name`, never a FK — an owner→registry key would fail every INSERT; for an OAuth connector it also snapshots `oauth_provider` + `deliver_json`, and its `api_key` column holds the refresh token (in the SQLCipher-encrypted file, so no column crypto). Because `memory_docs` is an owner table, one definition backs **private** memory in each `{userid}.db` and **shared** memory in `system.db` (the household owner) — see the memory namespace note below.
Schema is greenfield (no migrations, §0), but a purely **additive** column lands on an existing DB in place: `db::ensure_column` runs `ALTER TABLE … ADD COLUMN` and swallows the "duplicate column" error, a no-op on a fresh DB where the `CREATE TABLE` already has the column. Used for the OAuth columns on `mcp_catalog` / `mcp_user_servers` so a dev box need not be wiped for an additive change (a full recreate is still valid).
**No foreign key in the owner bucket may point at a registry table.** SQLite cannot enforce a key across files, not even through `ATTACH`, and sqlx turns on `PRAGMA foreign_keys`: the `CREATE TABLE` succeeds and every `INSERT` fails. `db::tests::owner_tables_stand_alone_with_foreign_keys_on` enforces this by running the owner schema against a database holding nothing else, then inserting a row into each table. Two keys crossed and were fixed: `chat_history.model_db_id` (dropped — write-only, and `llm_requests.model_name` already records the model) and `project_tickets.job_id` (fixed by moving `projects`/`project_tickets` into the owner bucket).
@@ -145,11 +147,20 @@ MCP servers are surfaced to users as **"Connectors"** (UI naming; `mcp`/schema s
**Authorization is a capability on the role, not `if role==admin`** (§0.1/§14 — `db/role_capabilities.rs`): `mcp.register_remote` + `mcp.register_local_from_catalog` are self-service (seeded on every new role by `roles::create` via `seed_defaults`); `mcp.register_local_script` + `mcp.manage_catalog` are admin-only. `admin` holds every capability by construction (short-circuit in `has()`). API handlers gate through `require_cap`.
**Tables** (see DB section) — registry: `mcp_catalog` (admin-vetted templates; holds only the *schema* of what an activation must supply, never live creds), `mcp_global_servers` + `mcp_global_access`, `role_capabilities`. Owner: `mcp_user_servers` (per-user activations; `api_key` encrypted at rest, `catalog_name` a bare `TEXT` snapshot).
**Tables** (see DB section) — registry: `mcp_catalog` (admin-vetted templates; holds only the *schema* of what an activation must supply, never live creds — plus, for OAuth, `oauth_provider` + `oauth_scopes_json` + `deliver_json`), `mcp_global_servers` + `mcp_global_access`, `oauth_providers` (per-provider client creds), `role_capabilities`. Owner: `mcp_user_servers` (per-user activations; `api_key` encrypted at rest — the refresh token for an OAuth one — `catalog_name`/`oauth_provider`/`deliver_json` bare `TEXT` snapshots).
**Endpoints** (`src/frontend/api/mcp.rs`, mounted in `api/mod.rs`) — admin: `/mcp/catalog` (GET/POST/DELETE), `/mcp/global` (list/enable/delete + `/{id}/access` GET/PUT). User: `/mcp/available`, `/mcp/activate`, `/mcp/activated` (+ DELETE `/{id}` to deactivate). `connectors.js` (`<connectors-page>`) renders the user view (activate/deactivate + granted globals) always, and the admin view (catalog + global + per-server access grants) when `role_id === 'admin'`.
**Endpoints** (`src/frontend/api/mcp.rs`, mounted in `api/mod.rs`) — admin: `/mcp/catalog` (GET/POST/DELETE), `/mcp/global` (list/enable/delete + `/{id}/access` GET/PUT), `/mcp/providers` (GET/POST + DELETE `/{name}` — OAuth provider creds, secret never returned to the browser). User: `/mcp/available`, `/mcp/activate`, `/mcp/activated` (+ DELETE `/{id}` to deactivate), `/mcp/oauth/start` + `/mcp/oauth/complete` (the §15 login). `connectors.js` (`<connectors-page>`) renders the user view (activate/deactivate + granted globals) always, plus the admin view (catalog + global + per-server access + a **Sign-in providers** modal) when `role_id === 'admin'`; `connector-detail.js` (`<connector-detail-page>`) is a connector's own page and hosts the OAuth login panel.
**Deferred:** interactive per-user auth (the §15 OAuth-callback / QR / SSH elicitation flow) — only `none`/`api_key` auth kinds are wired. No boot seed of catalog presets yet; the admin populates the catalog from the UI.
### OAuth per-user connectors (blueprint §15 — copy-paste flow)
OAuth2 authorization-code + PKCE is wired for per-user connectors (Gmail is the first). The consent is a **human copy-paste**, not a headless action: no callback route into the (NAT'd, hostname-less) box, and no client secret on the public feed.
- **Providers, not per-connector URLs.** The client is per-**provider** (one Google app covers Gmail/Calendar/Drive): `oauth_providers` holds `auth_url`/`token_url`/`client_id`/`client_secret`/`redirect_uri`/`extra_params`, admin-entered via the Sign-in-providers modal (Google preset fills all but the two secrets; `redirect_uri` = the static `oauth/show.html` page, `extra_params` = `access_type=offline`+`prompt=consent` so Google returns a refresh token). The manifest only names `auth.provider` + `auth.scopes` + `auth.deliver` — never URLs or secrets (feed is remote data, §14).
- **Flow** (`mcp/oauth.rs`): `activate` on an OAuth catalog entry persists a **pending** `mcp_user_servers` row (files installed, command wired, no token) and returns `needs_oauth` — it does **not** start the server. `/mcp/oauth/start` builds the consent URL (PKCE S256 + opaque `state`) and stashes the verifier in a RAM-only, TTL'd flow store keyed by `state`; the user approves in a browser, the provider lands the code on `oauth/show.html`, they paste it back. `/mcp/oauth/complete` exchanges code+verifier for a refresh token (`client_secret` sent server-side), stores it in the row's `api_key`, flips to `ready`, and starts the server. PKCE makes an intercepted code worthless; a restart drops in-flight flows (mirrors the RAM-only session model).
- **Credential delivery = env, nothing on disk.** The manifest's `deliver` (`{as,format,env}`, parsed as `mcp::DeliverSpec`) says how the token reaches the server. `user_row_spec_resolved` assembles the credential (`google_authorized_user` JSON = client creds from the provider + refresh token) and injects it as an env var (`GMAIL_CREDS_JSON`) on the `docker exec` — never a file, coherent with §2 (the tempted admin doesn't read `/proc`). The server reads it via `Credentials.from_authorized_user_info`. Ran both at OAuth-complete and at login-time per-user startup.
- **Google needs a Web-application client**: a Desktop client rejects an `https://` redirect (loopback only), so the `oauth/show.html` redirect must be registered on a **Web app** OAuth client, and exact-match under Authorized redirect URIs — `redirect_uri_mismatch` otherwise.
**Deferred:** the other §15 interactive kinds (QR / SSH via elicitation) — `deliver.as=file` and non-Google providers are unimplemented paths that error clearly rather than half-work. No boot seed of catalog presets; the admin populates the catalog from the Marketplace.
## Sub-agent system
@@ -251,7 +262,9 @@ All extend `LightElement` from `web/lib/base.js` (Lit). `ChatSession` (`web/lib/
| `agent-inbox.js` | `<agent-inbox-page>` | Pending approvals + clarifications from background sessions |
| `approval-rules.js` | `<approval-rules-page>` | Approval rule management |
| `cron-jobs.js` | `<cron-jobs-page>` | Scheduled job management |
| `connectors.js` | `<connectors-page>` | MCP Connectors: user activate/deactivate + granted globals; admin catalog + global-server + per-server access management (§7/§14/§15) |
| `connectors.js` | `<connectors-page>` | MCP Connectors list (one row per connector): user activate/deactivate + granted globals; admin gets a **Sign-in providers** modal (OAuth client creds) + Catalog/Marketplace nav (§7/§14/§15) |
| `connector-detail.js` | `<connector-detail-page>` | A connector's own page (`#connector?name=X`): env/secret form + Test, the **OAuth login panel** (sign in → paste code → complete, §15), global enable + per-user access grants |
| `shared/connector-common.js` | (helpers) | Shared Connectors vocabulary: `statusOf` (incl. `needs_login` for a pending OAuth row), `STATUS_LABEL`, schema normalization, `jf` fetch |
| `llm-providers.js` | `<llm-providers-page>` | LLM provider management |
| `models-hub.js` | `<models-hub-page>` | Models hub landing (LLM / Transcription / Image) |
| `models-llm.js` | `<models-llm-section>` | LLM model CRUD + drag-and-drop priority |