# Skald Connector Authoring Guide **This file is the single source of truth for the connector format.** It lives in the marketplace repo (`connectors.skaldagent.net`) and is referenced — not copied — by the Skald application repo (`~/projects/skald-circle`). Change the format here, and here only. Give this file to any agent that produces new connectors. A connector is a folder served by the marketplace. Skald installs it, verifies every file against a SHA-256 pinned in the index, then either runs it on the host (global connector) or copies it into the user's container and runs it there (per-user connector, blueprint §6/§7). Everything below marked **"the client enforces"** was checked against the implementation in `skald-circle` (`src/frontend/api/marketplace.rs`, `src/frontend/api/mcp.rs`, `crates/skald-core/src/mcp/{mod,install,verify,oauth}.rs`). --- ## 0. Folder layout One folder per connector, named exactly like its `id`, **flat**: ``` connectors/myconn/ ├── fragment.json ← index entry (compiler input, never served as-is) ├── connector.json ← the manifest Skald reads ├── server.py ← the entry file (or index.js…) ├── requirements.txt ← or package.json ├── verify.py ← optional ├── icon_sm.svg └── icon_lg.svg ``` **Keep the folder flat.** The client accepts a relative sub-path (`pkg/server.py`) but `scripts/compile.py` only scans the folder's **top level**: files in a subdirectory are silently left out of `files[]`, never downloaded, and the connector breaks at runtime with no error anywhere. If a tree is genuinely needed, teach the compiler to recurse first. --- ## 1. The three source documents (+ one compiler) You maintain three files by hand. A fourth, `connectors.json`, is **generated**. ### 1a. `connectors/index.json` — the order A flat JSON array of folder ids, in display order: ```json ["gmail", "gcal", "myconn"] ``` This is the compiler's input list. A connector missing from it does not exist. ### 1b. `connectors//fragment.json` — the index entry The connector's entry in the compiled index, **with every field except `files[]`** (the compiler adds that one). ```jsonc { "id": "whatsapp", // unique slug = folder name "name": "WhatsApp", "version": 8, // INTEGER build number — the update key (§7) "version_string": "2.2.0", // semver, display only "version_release_date": "2026-08-23", // ISO date, display only "type": "mcp_local", // mcp_local | mcp_remote (§3) "scope": "user", // user | global (§3) "icon_small": "whatsapp/icon_sm.png", // relative to the FEED ROOT, not the folder "icon_large": "whatsapp/icon_lg.png", "user_description": "Send and read WhatsApp messages from your linked account.", "requires": ["NODE"], // API_KEY | ENV | NODE | PYTHON | OAUTH | DOCKER "tags": ["messaging", "mcp", "local", "whatsapp", "qr"], "folder": "whatsapp", // defaults to id "auth": { "type": "qr" }, // informational here — see the table in §1e "tools": [ … ] // informational here — see §2a } ``` Note the **two icon vocabularies**: `icon_small` / `icon_large` in the index are relative to the feed root (`whatsapp/icon_sm.png`), while `files[]` and `connector.json` name them from inside the folder (`icon_sm.png`). The client reconciles the two, but only records an icon whose file was actually installed — so **icons must be shipped in the folder**, never hot- linked. ### 1c. `connectors//connector.json` — the manifest The richer document, fetched per connector and mapped into Skald's catalog. ```jsonc { "id": "whatsapp", "name": "WhatsApp", "version": 8, // must match fragment.json (§7) "version_string": "2.2.0", "version_release_date": "2026-08-23", "type": "mcp_local", "scope": "user", "requires": ["NODE"], "tags": ["messaging", "mcp", "local", "whatsapp", "qr"], "auth": { "type": "qr" }, // none | api_key | oauth2 | qr | ssh_key (§4) "launch_command": "node index.js", // human-readable; mcp_config is what runs "transport": "stdio", // stdio | streamable-http (may also sit in mcp_config) "mcp_config": { "command": "node", // interpreter (local) … "args": ["index.js"] // … args[0] MUST name the entry file }, "dependencies": ["@whiskeysockets/baileys@7.0.0-rc.14"], // display only (§5) "setup_instructions": ["Scan the QR code with WhatsApp → Linked devices"], "docs": [{ "lang": "en", "description": "Human blurb shown in the UI.", "llm_short_description": "One line the model reads to decide whether to use this connector." }], "env": [], // form fields the user fills (§4b) "verify": { "command": "python3 verify.py", "timeout_secs": 20 }, // optional (§6) "tools": [{ "name": "send_message", "display_name": "Send Message" }], // optional (§2a) "homepage": "https://web.whatsapp.com", "icon_small": "icon_sm.png", // relative to the FOLDER here "icon_large": "icon_lg.png" } ``` **`mcp_config.args[0]` is load-bearing for a local connector:** it is how Skald learns which file to run — an install fails outright without it. At activation Skald rewrites it to the file's path inside the user's container (`/root/.skald/mcp//`), so keep it a plain relative filename (`index.js`, `server.py`). **`transport` may sit either at the top level or inside `mcp_config`** — the client reads `mcp_config.transport` first, then the top-level one, then infers (`remote` → `http`, otherwise `stdio`). Both spellings are in use in this repo; pick one per connector and keep it consistent between the two files. `streamable-http` and `http` both normalise to HTTP; anything unrecognised silently becomes stdio, which for a remote connector means trying to spawn a command that does not exist. **`llm_short_description` is model-facing:** it is the one-liner injected into the LLM's system prompt so the model knows what this connector does. Keep it short and functional ("Weather — current conditions, 16-day forecast, and AQI data for any location"), **not** a list of tools (the model discovers those after `activate_tools`). ### 1d. `connectors/connectors.json` — the compiled index (generated) **Never edit this file by hand.** `scripts/compile.py` reads `index.json`, loads each `fragment.json`, scans the folder's real files, computes their SHA-256 + size, and writes the result: ```bash python3 scripts/compile.py # regenerate python3 scripts/compile.py --verify # fail if the committed index is stale ``` The compiler excludes `fragment.json`, `connectors.json`, `index.json`, `compile.sh`, `compile.py`, `update_hashes.py`, `.DS_Store` and every subdirectory. **Everything else is hashed, including `connector.json`.** ```jsonc "files": [ { "path": "index.js", "sha256": "…", "size": 21258 }, { "path": "package.json", "sha256": "…", "size": 302 }, { "path": "connector.json", "sha256": "…", "size": 620 }, { "path": "icon_sm.png", "sha256": "…", "size": 306 } ] ``` - `files[].path` is relative to the connector folder. - Digests and sizes are computed by the compiler. **Never write them by hand.** - The index is the **signable root**: it is the one document that names a connector's files and their digests. The client refuses any file whose bytes do not match, all-or-nothing — a mismatch leaves nothing on disk. - Do **not** ship `node_modules/` or vendored wheels (§5). ### 1e. Who reads what — the authority table The client hydrates a card from **two** documents, and they are not interchangeable. Getting this wrong is the most common way a correct-looking connector misbehaves. | Field | Read from | Notes | | --- | --- | --- | | `id`, `folder`, `tags` | **index** (`fragment.json`) | `folder` defaults to `id` | | `name` | index, falling back to manifest | | | `user_description` | **index**, falling back to `docs[0].description` | the human blurb | | `icon_small`, `icon_large` | **index** | feed-root-relative; the file must be in `files[]` | | `files[]` | **index** (manifest's is a legacy fallback) | the trust root | | `requires` | manifest, falling back to index | | | `type`, `scope` | manifest, falling back to index | keep them identical | | `version` trio | **manifest wins**, index is the fallback | a mismatch is logged as a desync warning (§7) | | `auth` | **manifest only** | the index's `auth` is *never parsed*; only `requires` is used as a coarse fallback | | `tools[]` | **manifest only** | a `tools[]` that lives only in `fragment.json` does nothing (§2a) | | `env[]`, `verify`, `mcp_config`, `docs`, `dependencies`, `setup_instructions`, `homepage` | **manifest only** | | So: `auth` and `tools[]` in `fragment.json` are documentation for the catalog page and for whoever reads the index — harmless, worth keeping in sync, but **the manifest is what runs**. --- ## 2. Server contract (MCP over stdio) A **local** connector is a program speaking JSON-RPC 2.0 over stdin/stdout. It MUST handle: - `initialize` → `{ protocolVersion: "2024-11-05", capabilities: { tools: {} }, serverInfo }` - `notifications/initialized` → **no response** - `ping` → `{}` - `tools/list` → `{ "tools": [ { name, description, inputSchema, title? } ] }` — an **object**, not a bare array - `tools/call` → `{ content: [ { type: "text", text } ], isError? }` **Any message without an `id` is a notification and must produce no output at all.** Answering one desynchronises a strict client. **stdout is reserved for JSON-RPC only.** Send all logs/diagnostics to **stderr**. Anything a library prints to stdout (a logger, a banner) corrupts the protocol — silence it (Baileys/pino → a silent logger; Python → `print(…, file=sys.stderr)`). Guard stdout writes with a lock if the server has background threads. Report tool failures as a normal result carrying `isError: true` with a readable message — not as a JSON-RPC error, and not as a "successful" result with an `{"status":"error"}` body the model cannot distinguish from data. A **remote** connector is an HTTP MCP endpoint (`mcp_config.url` + `transport: "streamable-http"`); no code runs on the box. ### 2a. Friendly tool names (`tools[]`) — optional Raw MCP tool names are ugly in the chat UI (`search_files`, `send_message`). Two ways to fix that, in order of preference: 1. **`title` in your `tools/list` entries** (preferred for servers we control) — Skald uses it automatically, no manifest change needed. 2. **The manifest's `tools[]` block** — for remote connectors and third-party packages (`npx -y firecrawl-mcp`) whose `tools/list` we cannot edit: ```jsonc "tools": [ { "name": "send_message", "display_name": "Send Message" } ] ``` - `name` — the **raw** tool name exactly as the server returns it from `tools/list`. - `display_name` — the friendly card title (English only; not internationalized). **Resolution order:** `tools[].display_name` → the MCP `title` field → a prettified raw name (`send_message` → "Send Message"). ⚠️ **`tools[]` is read from `connector.json`.** The client parses no `tools` field on the index entry, so a block placed only in `fragment.json` is inert. Put it in the manifest; mirroring it into `fragment.json` is optional and purely documentary. **Icons are per connector, not per tool.** Every tool shows its connector's `icon_small`; there is no per-tool icon. Partial `tools[]` lists are fine — unlisted tools fall through. --- ## 3. Placement & risk vocabulary (what the words mean) | Manifest | Meaning | | --- | --- | | `scope: "user"` | runs **once per user**, inside their container. Personal creds. Stored as `per_user`. | | `scope: "global"` | runs **once for the household**, on the host. Shared, stateless. Admin enables it with a key. | | `type: "mcp_local"` | ships code that will **execute on the box** — installing needs the admin `mcp.register_local_script` capability (RCE-bearing act, §14). Stored as `local_script`. | | `type: "mcp_remote"` | just an HTTP URL; no local code. Stored as `remote`. | Pick the narrowest: a personal messaging/email/calendar connector is `scope: "user"`; a shared search API is `scope: "global"`. Both axes **fail closed** — an unreadable `scope` becomes `per_user`, an unreadable `type` becomes `local_script`, the answer that demands *more* authority. `requires[]` is a human hint rendered as tags on the catalog page: `API_KEY`, `ENV`, `NODE`, `PYTHON`, `OAUTH`, `DOCKER`. The client only inspects it as a fallback when the manifest declares no `auth` (`OAUTH` → oauth, `API_KEY` → api_key). `SECRETS_DIR` is **removed** — the `secrets/` folder is no longer part of the model. --- ## 4. Authentication (`auth.type`) ### 4a. The recognized values | `auth.type` | Flow | Ships | | --- | --- | --- | | `none` | nothing to sign in | — | | `api_key` | user pastes a key/secret into a form | an `env[]` schema (§4b) | | `oauth2` | browser consent → token injected as an env var | `provider` + `scopes` + `deliver` (§4c) | | `qr` | server shows a QR, user scans with a phone | a `login_status` tool (§4d) | | `ssh_key` | reserved for key-based remote access | — | **Anything else normalises to `none`.** `email` declares `auth.type: "password"`, which the client does not recognise and therefore treats as `none`; that connector works only because its credentials arrive through `env[]` like any other form field. Prefer `api_key` or `none` for new connectors and let `env[]` carry the credential. `auth.delivery` (`header` | `query` | `env`) is surfaced to the admin UI. For a remote connector the **URL placeholder is what actually routes the key** (§4b) — the `param` name is not read by the client. ### 4b. `env[]` — the activation form, and what reaches the process Each entry drives one form field, and the collected values are injected into the server process environment: ```jsonc "env": [{ "name": "EMAIL_IMAP_HOST", // ← the ACTUAL environment variable name "label": "IMAP host", "description": "IMAP server hostname (e.g. imap.gmail.com).", "required": true, "secret": false, // true → masked in the form, stored encrypted "example": "imap.gmail.com", "default": "" // non-required fields only }] ``` > **`name` must be exactly the environment variable your server reads.** The process receives > the form's `name` → value map verbatim; there is no renaming layer. `mcp_config.env` is > **not** substituted at runtime — it survives only as a legacy fallback for the form schema, > and if it is ever the sole env source its `{ENV:…}` strings are injected *literally*. Do not > use it to map one name onto another. The server reads each value from `os.environ` / `process.env`. Declare `requires: ["ENV"]` when the connector needs user-supplied config. **Placeholders.** Two tokens, and exactly two places where they are substituted: | Token | Meaning | | --- | --- | | `{ENV:NAME}` | non-sensitive value (host, port, username…) | | `{SECRET:NAME}` | sensitive value (password, API key, token) | | Substituted in | Engine behaviour on an unknown name | | --- | --- | | `mcp_config.url` (remote connectors) | `{SECRET:x}` falls back to the connector's api_key; if that is absent the token is **left in the URL literally**, so a misconfiguration is visible | | `verify.command` (§6) | replaced with the **empty string** | Nowhere else — not in `mcp_config.env`, not in `args`. Legacy `{key}` still resolves to the api_key in URLs; `{env:NAME}` and `{secrets}/…` are dead and are never substituted. Remote example: `"url": "https://mcp.tavily.com/mcp/?tavilyApiKey={SECRET:tavilyApiKey}"` with a matching `env[]` entry named `tavilyApiKey`. ### 4c. `oauth2` — provider consent ```jsonc "auth": { "type": "oauth2", "provider": "google", // slug into the admin's sign-in providers "scopes": ["https://www.googleapis.com/auth/gmail.modify"], "deliver": { "as": "env", "format": "google_authorized_user", "env": "GMAIL_CREDS_JSON" } } ``` The manifest names **only** the provider slug, the scopes, and how the obtained token is delivered — never client ids, client secrets, endpoint URLs or redirect URIs, which are admin-entered and stay off the public feed. Skald runs PKCE + code exchange and injects the credential as the named env var. - `deliver.as`: **only `"env"` is implemented** — `"file"` is rejected at activation with an explicit error rather than half-working. Do not ship it. - `deliver.format`: `google_authorized_user` (the JSON `from_authorized_user_file` reads) or `refresh_token`. - `deliver.env` must **not** also appear in `mcp_config.env` — Skald injects it at runtime. - An OAuth connector activates into a **pending** row; nothing starts until the consent round-trip completes. The admin must have configured the provider first, or activation fails early with a clear message. ### 4d. `qr` / interactive device login For a connector whose credential is produced by **scanning/pairing** (WhatsApp today) there is no code to paste. The contract: > **Expose one extra tool, `login_status`, returning a JSON object** (as the `text` of a normal > text result). Skald calls it directly — never the agent — and a login panel polls it. ```jsonc // login_status result text (a JSON string): { "state": "connecting" | "need_scan" | "ready" | "logged_out", "qr": "data:image/png;base64,…", // present ONLY while state == need_scan "message": "human-readable line" } ``` - Activating a `qr` connector inserts a **pending** row and **starts the server** (so it can produce the QR), then hands off to the login panel. - The panel polls `POST /api/mcp/login/status`; when `state == "ready"` the connector is marked ready and starts automatically on later logins. - Also expose a **`logout`** tool (clears the session, forces a fresh QR) — the panel calls it via `POST /api/mcp/login/reset` to re-link a different phone. - The **credential is the on-disk session**, not a token. Persist it **inside the connector's own directory** (e.g. `./auth/` next to the entry file). That folder lives under the bind- mounted home, so it survives container recreates and connector updates. Never store it under a shared or global path. --- ## 5. Dependencies (node & python) — how they get installed **Do not ship `node_modules/` or vendored wheels.** Declare deps as a standard manifest **file** and Skald installs them where the server will run: - **node:** ship a `package.json` with a `dependencies` map. Skald runs `npm ci --omit=dev --no-audit --no-fund`, falling back to `npm install --omit=dev …` when there is no lockfile. `node_modules/` resolves automatically beside the entry file. - **python:** ship a `requirements.txt`. Skald runs `python3 -m pip install --break-system-packages --target .pydeps -r requirements.txt` and puts `.pydeps` on the server's `PYTHONPATH`. A single install has a **300 s ceiling**; past that it fails rather than hanging a login. This runs at activation **and** on every startup, guarded by a **content hash** of the connector's source files (`.skald-install.lock`): - first activation / a brand-new container → full install, - a connector **update** (any shipped file changed) → re-copy + re-install, - unchanged → skipped in microseconds. So you never write install steps into the manifest — just ship the dep file. The manifest's `dependencies[]` and `setup_instructions[]` are **display metadata** for the admin card; they install nothing. Set `requires: ["NODE"]` / `["PYTHON"]` as a human hint. Pin versions for reproducible installs, and keep the tree lean (containers are slim; prefer a pure-JS/Python library over a native-heavy one — e.g. Baileys instead of a browser). **Host assets are not copied into the container.** Icons (`.svg/.png/.jpg/.webp/.gif/.ico`) and `connector.json` stay on the host, so a server must never expect to read them at runtime. Everything else in the folder — entry file, deps file, helper modules, `verify.py` — is copied. Where the files land: | | Host | Per-user container | | --- | --- | --- | | installed folder | `/connectors//` | `~/.skald/mcp//` (`/root/.skald/mcp//`) | | python deps | `connectors//.pydeps` | `~/.skald/mcp//.pydeps` | --- ## 6. Verify-before-save (optional, strongly recommended) Ship a `verify.py` / `verify.js` (or an inline snippet) and reference it: ```jsonc "verify": { "command": "python3 verify.py", "timeout_secs": 20 } ``` It runs **after** the user fills the form and **before** the activation is persisted, with the collected env/secrets injected and `{ENV:…}` / `{SECRET:…}` substituted into the command. **Output contract** — one JSON object on stdout and nothing else: ```json {"ok": true, "message": "IMAP and SMTP authentication successful", "details": {"imap": "…"}} {"ok": false, "message": "IMAP login failed: INVALID_CREDENTIALS"} ``` Exit 0 on success. If the JSON fails to parse the client falls back to the exit code (0 = ok) and shows stderr. A timeout counts as a failure. **Never print credentials** in `message` or `details`. Rules the client enforces: - **The script must be one of the connector's shipped files.** The client resolves it by matching a **basename from `files[]`** against the command string — so `python3 verify.py` works because `verify.py` is in `files[]`. A script that is not shipped is never downloaded and activation fails; an inline command with no matching basename (firecrawl's `node -e "…"`) is fine and runs as-is. - **Where it runs:** inside the user's container, in the connector's directory, for a `per_user` connector; on the host in `connectors//` for a `global` one. - **Timeout:** declare `timeout_secs` for the record, but the runtime currently applies a fixed **20 s** to every verify. Keep the probe well under that. - Any `auth.type` may use verify — it is not limited to `api_key`. A `qr` connector needs none: its `login_status` is the live check. **Without `verify` there is no test at all.** Activation goes straight to `ready` — including for remote connectors, which get no handshake fallback. The manifest author decides. --- ## 7. Versioning & updates Three fields, in **both** `fragment.json` and `connector.json`, kept identical: | field | type | role | | --- | --- | --- | | `version` | **integer** | monotonic build number, per connector — the machine comparison key | | `version_string` | string (semver) | display only | | `version_release_date` | ISO date `YYYY-MM-DD` | display only | - `version` is a **number, not a string** (`8`, not `"8"` or `"2.2.0"`). **`+1` on every change** to any shipped file **or** to any manifest metadata (description, icons, `version_string`). Never reuse or decrement. - **The integer is the only "is there an update?" signal**, compared strictly (`feed > installed`). `version_string`, icons and `llm_short_description` are never compared, so changing them without bumping the integer is **invisible** — no "update available" badge. This is the classic trap. - **The manifest wins over the index** when the two disagree, and the disagreement is only visible as a log line (`marketplace feed version desync`). If the index carries the *lower* number and the manifest the higher one, the strict comparison can never fire again and the connector silently stops offering updates. Keep them equal — `compile.py` does not check this for you. - **Two propagation paths, do not conflate them:** - *Per-user code + deps* reconcile on a **content hash** of the source files (§5), so new code lands at each user's next login even without a reinstall. - *Catalog metadata* (`llm_short_description` → the model's prompt, icons, friendly name) is **not** in that hash. It lives in the catalog row and is rewritten only by an explicit reinstall/Update, which re-pulls the current feed (never the 300 s browse cache) and restarts enabled global servers and every live user's copy with the fresh description. - So: to ship a new `llm_short_description`, **bump the integer** so the admin sees "update available" and clicks Update. Nothing auto-propagates a description change. --- ## 8. Hard limits the client enforces - **Digest mismatch → refused**, all-or-nothing: nothing is written unless every file verifies. - **8 MiB per file.** A `files[].size` above that is rejected before the download starts. - **Path safety.** A `files[].path` or `mcp_config.args[0]` that is absolute, contains `..`, a backslash or a colon is rejected outright. - **A local connector with no `files[]` cannot be installed** ("refusing to install unverifiable code"), and neither can one whose only file is `connector.json`. - **`mcp_config.args[0]` is mandatory** for `mcp_local`. - The feed is fetched as `GET /connectors.json` plus one `GET //connector.json` per entry, cached 300 s for browsing; installs always refetch. A manifest that fails to load degrades that one card to whatever the index said — it never fails the listing, which is exactly how a broken manifest hides. --- ## 9. Checklist for a new connector 1. Create `connectors//` — flat — with: entry file, `connector.json`, `fragment.json`, deps file (`package.json` / `requirements.txt`), `icon_sm.*`, `icon_lg.*`, optional `verify.*`. 2. Server speaks MCP over stdio (§2); **stdout = JSON-RPC only**; notifications answered with silence; `title` on every tool. 3. `mcp_config.args[0]` names the entry file; `transport` set consistently. 4. Correct `type` + `scope` (§3) and `auth.type` (§4); `auth`, `type`, `scope`, `tags`, `requires` and the version trio **identical** in both documents. 5. `env[].name` = the real environment variable name (§4b). 6. For `qr`: implement `login_status` + `logout`, persist the session under the connector dir (§4d). 7. Deps declared as a file, **not** vendored (§5). 8. `docs[0].llm_short_description` says what the connector *does*, not which tools it has. 9. Add `""` to `connectors/index.json`. 10. Bump `version` (+1), `version_string`, `version_release_date` in **both** documents. 11. Run `python3 scripts/compile.py`, then `python3 scripts/compile.py --verify`. 12. Record the change in `CHANGELOG.md` in the same commit, then commit and deploy.