skald-connectors/CONNECTOR_MANIFEST_GUIDE.md

# Skald Connector Authoring Guide

**This file is the single source of truth for the connector format.** It lives in the
marketplace repo (`connectors.skaldagent.net`) and is referenced — not copied — by the
Skald application repo (`~/projects/skald-circle`). Change the format here, and here only.

Give this file to any agent that produces new connectors.

A connector is a folder served by the marketplace. Skald installs it, verifies every file
against a SHA-256 pinned in the index, then either runs it on the host (global connector)
or copies it into the user's container and runs it there (per-user connector, blueprint
§6/§7).

Everything below marked **"the client enforces"** was checked against the implementation in
`skald-circle` (`src/frontend/api/marketplace.rs`, `src/frontend/api/mcp.rs`,
`crates/skald-core/src/mcp/{mod,install,verify,oauth}.rs`).

---

## 0. Folder layout

One folder per connector, named exactly like its `id`, **flat**:

```
connectors/myconn/
├── fragment.json          ← index entry (compiler input, never served as-is)
├── connector.json         ← the manifest Skald reads
├── server.py              ← the entry file (or index.js…)
├── requirements.txt       ← or package.json
├── verify.py              ← optional
├── icon_sm.svg
└── icon_lg.svg
```

**Keep the folder flat.** The client accepts a relative sub-path (`pkg/server.py`) but
`scripts/compile.py` only scans the folder's **top level**: files in a subdirectory are
silently left out of `files[]`, never downloaded, and the connector breaks at runtime with
no error anywhere. If a tree is genuinely needed, teach the compiler to recurse first.

---

## 1. The three source documents (+ one compiler)

You maintain three files by hand. A fourth, `connectors.json`, is **generated**.

### 1a. `connectors/index.json` — the order

A flat JSON array of folder ids, in display order:

```json
["gmail", "gcal", "myconn"]
```

This is the compiler's input list. A connector missing from it does not exist.

### 1b. `connectors/<id>/fragment.json` — the index entry

The connector's entry in the compiled index, **with every field except `files[]`** (the
compiler adds that one).

```jsonc
{
  "id": "whatsapp",                      // unique slug = folder name
  "name": "WhatsApp",
  "version": 8,                          // INTEGER build number — the update key (§7)
  "version_string": "2.2.0",             // semver, display only
  "version_release_date": "2026-08-23",  // ISO date, display only
  "type": "mcp_local",                   // mcp_local | mcp_remote  (§3)
  "scope": "user",                       // user | global           (§3)
  "icon_small": "whatsapp/icon_sm.png",  // relative to the FEED ROOT, not the folder
  "icon_large": "whatsapp/icon_lg.png",
  "user_description": "Send and read WhatsApp messages from your linked account.",
  "requires": ["NODE"],                  // API_KEY | ENV | NODE | PYTHON | OAUTH | DOCKER
  "tags": ["messaging", "mcp", "local", "whatsapp", "qr"],
  "folder": "whatsapp",                  // defaults to id
  "auth": { "type": "qr" },              // informational here — see the table in §1e
  "tools": [ … ]                         // informational here — see §2a
}
```

Note the **two icon vocabularies**: `icon_small` / `icon_large` in the index are relative to
the feed root (`whatsapp/icon_sm.png`), while `files[]` and `connector.json` name them from
inside the folder (`icon_sm.png`). The client reconciles the two, but only records an icon
whose file was actually installed — so **icons must be shipped in the folder**, never hot-
linked.

### 1c. `connectors/<id>/connector.json` — the manifest

The richer document, fetched per connector and mapped into Skald's catalog.

```jsonc
{
  "id": "whatsapp",
  "name": "WhatsApp",
  "version": 8,                         // must match fragment.json (§7)
  "version_string": "2.2.0",
  "version_release_date": "2026-08-23",
  "type": "mcp_local",
  "scope": "user",
  "requires": ["NODE"],
  "tags": ["messaging", "mcp", "local", "whatsapp", "qr"],
  "auth": { "type": "qr" },             // none | api_key | oauth2 | qr | ssh_key  (§4)
  "launch_command": "node index.js",    // human-readable; mcp_config is what runs
  "transport": "stdio",                 // stdio | streamable-http (may also sit in mcp_config)
  "mcp_config": {
    "command": "node",                  // interpreter (local) …
    "args": ["index.js"]                // … args[0] MUST name the entry file
  },
  "dependencies": ["@whiskeysockets/baileys@7.0.0-rc.14"],  // display only (§5)
  "setup_instructions": ["Scan the QR code with WhatsApp → Linked devices"],
  "docs": [{
    "lang": "en",
    "description": "Human blurb shown in the UI.",
    "llm_short_description": "One line the model reads to decide whether to use this connector."
  }],
  "env": [],                            // form fields the user fills (§4b)
  "verify": { "command": "python3 verify.py", "timeout_secs": 20 },   // optional (§6)
  "tools": [{ "name": "send_message", "display_name": "Send Message" }],  // optional (§2a)
  "homepage": "https://web.whatsapp.com",
  "icon_small": "icon_sm.png",          // relative to the FOLDER here
  "icon_large": "icon_lg.png"
}
```

**`mcp_config.args[0]` is load-bearing for a local connector:** it is how Skald learns which
file to run — an install fails outright without it. At activation Skald rewrites it to the
file's path inside the user's container (`/root/.skald/mcp/<name>/<entry>`), so keep it a
plain relative filename (`index.js`, `server.py`).

**`transport` may sit either at the top level or inside `mcp_config`** — the client reads
`mcp_config.transport` first, then the top-level one, then infers (`remote` → `http`,
otherwise `stdio`). Both spellings are in use in this repo; pick one per connector and keep
it consistent between the two files. `streamable-http` and `http` both normalise to HTTP;
anything unrecognised silently becomes stdio, which for a remote connector means trying to
spawn a command that does not exist.

**`llm_short_description` is model-facing:** it is the one-liner injected into the LLM's
system prompt so the model knows what this connector does. Keep it short and functional
("Weather — current conditions, 16-day forecast, and AQI data for any location"), **not** a
list of tools (the model discovers those after `activate_tools`).

### 1d. `connectors/connectors.json` — the compiled index (generated)

**Never edit this file by hand.** `scripts/compile.py` reads `index.json`, loads each
`fragment.json`, scans the folder's real files, computes their SHA-256 + size, and writes the
result:

```bash
python3 scripts/compile.py           # regenerate
python3 scripts/compile.py --verify  # fail if the committed index is stale
```

The compiler excludes `fragment.json`, `connectors.json`, `index.json`, `compile.sh`,
`compile.py`, `update_hashes.py`, `.DS_Store` and every subdirectory. **Everything else is
hashed, including `connector.json`.**

```jsonc
"files": [
  { "path": "index.js",       "sha256": "…", "size": 21258 },
  { "path": "package.json",   "sha256": "…", "size": 302   },
  { "path": "connector.json", "sha256": "…", "size": 620   },
  { "path": "icon_sm.png",    "sha256": "…", "size": 306   }
]
```

- `files[].path` is relative to the connector folder.
- Digests and sizes are computed by the compiler. **Never write them by hand.**
- The index is the **signable root**: it is the one document that names a connector's files
  and their digests. The client refuses any file whose bytes do not match, all-or-nothing —
  a mismatch leaves nothing on disk.
- Do **not** ship `node_modules/` or vendored wheels (§5).

### 1e. Who reads what — the authority table

The client hydrates a card from **two** documents, and they are not interchangeable. Getting
this wrong is the most common way a correct-looking connector misbehaves.

| Field | Read from | Notes |
| --- | --- | --- |
| `id`, `folder`, `tags` | **index** (`fragment.json`) | `folder` defaults to `id` |
| `name` | index, falling back to manifest | |
| `user_description` | **index**, falling back to `docs[0].description` | the human blurb |
| `icon_small`, `icon_large` | **index** | feed-root-relative; the file must be in `files[]` |
| `files[]` | **index** (manifest's is a legacy fallback) | the trust root |
| `requires` | manifest, falling back to index | |
| `type`, `scope` | manifest, falling back to index | keep them identical |
| `version` trio | **manifest wins**, index is the fallback | a mismatch is logged as a desync warning (§7) |
| `auth` | **manifest only** | the index's `auth` is *never parsed*; only `requires` is used as a coarse fallback |
| `tools[]` | **manifest only** | a `tools[]` that lives only in `fragment.json` does nothing (§2a) |
| `env[]`, `verify`, `mcp_config`, `docs`, `dependencies`, `setup_instructions`, `homepage` | **manifest only** | |

So: `auth` and `tools[]` in `fragment.json` are documentation for the catalog page and for
whoever reads the index — harmless, worth keeping in sync, but **the manifest is what runs**.

---

## 2. Server contract (MCP over stdio)

A **local** connector is a program speaking JSON-RPC 2.0 over stdin/stdout. It MUST handle:

- `initialize` → `{ protocolVersion: "2024-11-05", capabilities: { tools: {} }, serverInfo }`
- `notifications/initialized` → **no response**
- `ping` → `{}`
- `tools/list` → `{ "tools": [ { name, description, inputSchema, title? } ] }` — an
  **object**, not a bare array
- `tools/call` → `{ content: [ { type: "text", text } ], isError? }`

**Any message without an `id` is a notification and must produce no output at all.**
Answering one desynchronises a strict client.

**stdout is reserved for JSON-RPC only.** Send all logs/diagnostics to **stderr**. Anything a
library prints to stdout (a logger, a banner) corrupts the protocol — silence it (Baileys/pino
→ a silent logger; Python → `print(…, file=sys.stderr)`). Guard stdout writes with a lock if
the server has background threads.

Report tool failures as a normal result carrying `isError: true` with a readable message —
not as a JSON-RPC error, and not as a "successful" result with an `{"status":"error"}` body
the model cannot distinguish from data.

A **remote** connector is an HTTP MCP endpoint (`mcp_config.url` + `transport:
"streamable-http"`); no code runs on the box.

### 2a. Friendly tool names (`tools[]`) — optional

Raw MCP tool names are ugly in the chat UI (`search_files`, `send_message`). Two ways to fix
that, in order of preference:

1. **`title` in your `tools/list` entries** (preferred for servers we control) — Skald uses it
   automatically, no manifest change needed.
2. **The manifest's `tools[]` block** — for remote connectors and third-party packages
   (`npx -y firecrawl-mcp`) whose `tools/list` we cannot edit:

```jsonc
"tools": [
  { "name": "send_message", "display_name": "Send Message" }
]
```

- `name` — the **raw** tool name exactly as the server returns it from `tools/list`.
- `display_name` — the friendly card title (English only; not internationalized).

**Resolution order:** `tools[].display_name` → the MCP `title` field → a prettified raw name
(`send_message` → "Send Message").

⚠️ **`tools[]` is read from `connector.json`.** The client parses no `tools` field on the
index entry, so a block placed only in `fragment.json` is inert. Put it in the manifest;
mirroring it into `fragment.json` is optional and purely documentary.

**Icons are per connector, not per tool.** Every tool shows its connector's `icon_small`;
there is no per-tool icon. Partial `tools[]` lists are fine — unlisted tools fall through.

---

## 3. Placement & risk vocabulary (what the words mean)

| Manifest | Meaning |
| --- | --- |
| `scope: "user"` | runs **once per user**, inside their container. Personal creds. Stored as `per_user`. |
| `scope: "global"` | runs **once for the household**, on the host. Shared, stateless. Admin enables it with a key. |
| `type: "mcp_local"` | ships code that will **execute on the box** — installing needs the admin `mcp.register_local_script` capability (RCE-bearing act, §14). Stored as `local_script`. |
| `type: "mcp_remote"` | just an HTTP URL; no local code. Stored as `remote`. |

Pick the narrowest: a personal messaging/email/calendar connector is `scope: "user"`; a shared
search API is `scope: "global"`. Both axes **fail closed** — an unreadable `scope` becomes
`per_user`, an unreadable `type` becomes `local_script`, the answer that demands *more*
authority.

`requires[]` is a human hint rendered as tags on the catalog page: `API_KEY`, `ENV`, `NODE`,
`PYTHON`, `OAUTH`, `DOCKER`. The client only inspects it as a fallback when the manifest
declares no `auth` (`OAUTH` → oauth, `API_KEY` → api_key). `SECRETS_DIR` is **removed** — the
`secrets/` folder is no longer part of the model.

---

## 4. Authentication (`auth.type`)

### 4a. The recognized values

| `auth.type` | Flow | Ships |
| --- | --- | --- |
| `none` | nothing to sign in | — |
| `api_key` | user pastes a key/secret into a form | an `env[]` schema (§4b) |
| `oauth2` | browser consent → token injected as an env var | `provider` + `scopes` + `deliver` (§4c) |
| `qr` | server shows a QR, user scans with a phone | a `login_status` tool (§4d) |
| `ssh_key` | reserved for key-based remote access | — |

**Anything else normalises to `none`.** `email` declares `auth.type: "password"`, which the
client does not recognise and therefore treats as `none`; that connector works only because
its credentials arrive through `env[]` like any other form field. Prefer `api_key` or `none`
for new connectors and let `env[]` carry the credential.

`auth.delivery` (`header` | `query` | `env`) is surfaced to the admin UI. For a remote
connector the **URL placeholder is what actually routes the key** (§4b) — the `param` name is
not read by the client.

### 4b. `env[]` — the activation form, and what reaches the process

Each entry drives one form field, and the collected values are injected into the server
process environment:

```jsonc
"env": [{
  "name": "EMAIL_IMAP_HOST",            // ← the ACTUAL environment variable name
  "label": "IMAP host",
  "description": "IMAP server hostname (e.g. imap.gmail.com).",
  "required": true,
  "secret": false,                      // true → masked in the form, stored encrypted
  "example": "imap.gmail.com",
  "default": ""                         // non-required fields only
}]
```

> **`name` must be exactly the environment variable your server reads.** The process receives
> the form's `name` → value map verbatim; there is no renaming layer. `mcp_config.env` is
> **not** substituted at runtime — it survives only as a legacy fallback for the form schema,
> and if it is ever the sole env source its `{ENV:…}` strings are injected *literally*. Do not
> use it to map one name onto another.

The server reads each value from `os.environ` / `process.env`. Declare `requires: ["ENV"]`
when the connector needs user-supplied config.

**Placeholders.** Two tokens, and exactly two places where they are substituted:

| Token | Meaning |
| --- | --- |
| `{ENV:NAME}` | non-sensitive value (host, port, username…) |
| `{SECRET:NAME}` | sensitive value (password, API key, token) |

| Substituted in | Engine behaviour on an unknown name |
| --- | --- |
| `mcp_config.url` (remote connectors) | `{SECRET:x}` falls back to the connector's api_key; if that is absent the token is **left in the URL literally**, so a misconfiguration is visible |
| `verify.command` (§6) | replaced with the **empty string** |

Nowhere else — not in `mcp_config.env`, not in `args`. Legacy `{key}` still resolves to the
api_key in URLs; `{env:NAME}` and `{secrets}/…` are dead and are never substituted.

Remote example: `"url": "https://mcp.tavily.com/mcp/?tavilyApiKey={SECRET:tavilyApiKey}"`
with a matching `env[]` entry named `tavilyApiKey`.

### 4c. `oauth2` — provider consent

```jsonc
"auth": {
  "type": "oauth2",
  "provider": "google",                 // slug into the admin's sign-in providers
  "scopes": ["https://www.googleapis.com/auth/gmail.modify"],
  "deliver": { "as": "env", "format": "google_authorized_user", "env": "GMAIL_CREDS_JSON" }
}
```

The manifest names **only** the provider slug, the scopes, and how the obtained token is
delivered — never client ids, client secrets, endpoint URLs or redirect URIs, which are
admin-entered and stay off the public feed. Skald runs PKCE + code exchange and injects the
credential as the named env var.

- `deliver.as`: **only `"env"` is implemented** — `"file"` is rejected at activation with an
  explicit error rather than half-working. Do not ship it.
- `deliver.format`: `google_authorized_user` (the JSON `from_authorized_user_file` reads) or
  `refresh_token`.
- `deliver.env` must **not** also appear in `mcp_config.env` — Skald injects it at runtime.
- An OAuth connector activates into a **pending** row; nothing starts until the consent
  round-trip completes. The admin must have configured the provider first, or activation
  fails early with a clear message.

### 4d. `qr` / interactive device login

For a connector whose credential is produced by **scanning/pairing** (WhatsApp today) there is
no code to paste. The contract:

> **Expose one extra tool, `login_status`, returning a JSON object** (as the `text` of a normal
> text result). Skald calls it directly — never the agent — and a login panel polls it.

```jsonc
// login_status result text (a JSON string):
{
  "state": "connecting" | "need_scan" | "ready" | "logged_out",
  "qr": "data:image/png;base64,…",   // present ONLY while state == need_scan
  "message": "human-readable line"
}
```

- Activating a `qr` connector inserts a **pending** row and **starts the server** (so it can
  produce the QR), then hands off to the login panel.
- The panel polls `POST /api/mcp/login/status`; when `state == "ready"` the connector is marked
  ready and starts automatically on later logins.
- Also expose a **`logout`** tool (clears the session, forces a fresh QR) — the panel calls it
  via `POST /api/mcp/login/reset` to re-link a different phone.
- The **credential is the on-disk session**, not a token. Persist it **inside the connector's
  own directory** (e.g. `./auth/` next to the entry file). That folder lives under the bind-
  mounted home, so it survives container recreates and connector updates. Never store it
  under a shared or global path.

---

## 5. Dependencies (node & python) — how they get installed

**Do not ship `node_modules/` or vendored wheels.** Declare deps as a standard manifest **file**
and Skald installs them where the server will run:

- **node:** ship a `package.json` with a `dependencies` map. Skald runs
  `npm ci --omit=dev --no-audit --no-fund`, falling back to `npm install --omit=dev …` when
  there is no lockfile. `node_modules/` resolves automatically beside the entry file.
- **python:** ship a `requirements.txt`. Skald runs
  `python3 -m pip install --break-system-packages --target .pydeps -r requirements.txt` and
  puts `.pydeps` on the server's `PYTHONPATH`.

A single install has a **300 s ceiling**; past that it fails rather than hanging a login.

This runs at activation **and** on every startup, guarded by a **content hash** of the
connector's source files (`.skald-install.lock`):

- first activation / a brand-new container → full install,
- a connector **update** (any shipped file changed) → re-copy + re-install,
- unchanged → skipped in microseconds.

So you never write install steps into the manifest — just ship the dep file. The manifest's
`dependencies[]` and `setup_instructions[]` are **display metadata** for the admin card; they
install nothing. Set `requires: ["NODE"]` / `["PYTHON"]` as a human hint. Pin versions for
reproducible installs, and keep the tree lean (containers are slim; prefer a pure-JS/Python
library over a native-heavy one — e.g. Baileys instead of a browser).

**Host assets are not copied into the container.** Icons (`.svg/.png/.jpg/.webp/.gif/.ico`) and
`connector.json` stay on the host, so a server must never expect to read them at runtime.
Everything else in the folder — entry file, deps file, helper modules, `verify.py` — is copied.

Where the files land:

| | Host | Per-user container |
| --- | --- | --- |
| installed folder | `<workdir>/connectors/<id>/` | `~/.skald/mcp/<name>/` (`/root/.skald/mcp/<name>/`) |
| python deps | `connectors/<id>/.pydeps` | `~/.skald/mcp/<name>/.pydeps` |

---

## 6. Verify-before-save (optional, strongly recommended)

Ship a `verify.py` / `verify.js` (or an inline snippet) and reference it:

```jsonc
"verify": { "command": "python3 verify.py", "timeout_secs": 20 }
```

It runs **after** the user fills the form and **before** the activation is persisted, with the
collected env/secrets injected and `{ENV:…}` / `{SECRET:…}` substituted into the command.

**Output contract** — one JSON object on stdout and nothing else:

```json
{"ok": true,  "message": "IMAP and SMTP authentication successful", "details": {"imap": "…"}}
{"ok": false, "message": "IMAP login failed: INVALID_CREDENTIALS"}
```

Exit 0 on success. If the JSON fails to parse the client falls back to the exit code (0 = ok)
and shows stderr. A timeout counts as a failure. **Never print credentials** in `message` or
`details`.

Rules the client enforces:

- **The script must be one of the connector's shipped files.** The client resolves it by
  matching a **basename from `files[]`** against the command string — so `python3 verify.py`
  works because `verify.py` is in `files[]`. A script that is not shipped is never downloaded
  and activation fails; an inline command with no matching basename (firecrawl's `node -e "…"`)
  is fine and runs as-is.
- **Where it runs:** inside the user's container, in the connector's directory, for a
  `per_user` connector; on the host in `connectors/<id>/` for a `global` one.
- **Timeout:** declare `timeout_secs` for the record, but the runtime currently applies a fixed
  **20 s** to every verify. Keep the probe well under that.
- Any `auth.type` may use verify — it is not limited to `api_key`. A `qr` connector needs none:
  its `login_status` is the live check.

**Without `verify` there is no test at all.** Activation goes straight to `ready` — including
for remote connectors, which get no handshake fallback. The manifest author decides.

---

## 7. Versioning & updates

Three fields, in **both** `fragment.json` and `connector.json`, kept identical:

| field | type | role |
| --- | --- | --- |
| `version` | **integer** | monotonic build number, per connector — the machine comparison key |
| `version_string` | string (semver) | display only |
| `version_release_date` | ISO date `YYYY-MM-DD` | display only |

- `version` is a **number, not a string** (`8`, not `"8"` or `"2.2.0"`). **`+1` on every
  change** to any shipped file **or** to any manifest metadata (description, icons,
  `version_string`). Never reuse or decrement.
- **The integer is the only "is there an update?" signal**, compared strictly
  (`feed > installed`). `version_string`, icons and `llm_short_description` are never
  compared, so changing them without bumping the integer is **invisible** — no "update
  available" badge. This is the classic trap.
- **The manifest wins over the index** when the two disagree, and the disagreement is only
  visible as a log line (`marketplace feed version desync`). If the index carries the *lower*
  number and the manifest the higher one, the strict comparison can never fire again and the
  connector silently stops offering updates. Keep them equal — `compile.py` does not check
  this for you.
- **Two propagation paths, do not conflate them:**
  - *Per-user code + deps* reconcile on a **content hash** of the source files (§5), so new
    code lands at each user's next login even without a reinstall.
  - *Catalog metadata* (`llm_short_description` → the model's prompt, icons, friendly name) is
    **not** in that hash. It lives in the catalog row and is rewritten only by an explicit
    reinstall/Update, which re-pulls the current feed (never the 300 s browse cache) and
    restarts enabled global servers and every live user's copy with the fresh description.
- So: to ship a new `llm_short_description`, **bump the integer** so the admin sees "update
  available" and clicks Update. Nothing auto-propagates a description change.

---

## 8. Hard limits the client enforces

- **Digest mismatch → refused**, all-or-nothing: nothing is written unless every file verifies.
- **8 MiB per file.** A `files[].size` above that is rejected before the download starts.
- **Path safety.** A `files[].path` or `mcp_config.args[0]` that is absolute, contains `..`,
  a backslash or a colon is rejected outright.
- **A local connector with no `files[]` cannot be installed** ("refusing to install
  unverifiable code"), and neither can one whose only file is `connector.json`.
- **`mcp_config.args[0]` is mandatory** for `mcp_local`.
- The feed is fetched as `GET /connectors.json` plus one `GET /<folder>/connector.json` per
  entry, cached 300 s for browsing; installs always refetch. A manifest that fails to load
  degrades that one card to whatever the index said — it never fails the listing, which is
  exactly how a broken manifest hides.

---

## 9. Checklist for a new connector

1. Create `connectors/<id>/` — flat — with: entry file, `connector.json`, `fragment.json`,
   deps file (`package.json` / `requirements.txt`), `icon_sm.*`, `icon_lg.*`, optional
   `verify.*`.
2. Server speaks MCP over stdio (§2); **stdout = JSON-RPC only**; notifications answered with
   silence; `title` on every tool.
3. `mcp_config.args[0]` names the entry file; `transport` set consistently.
4. Correct `type` + `scope` (§3) and `auth.type` (§4); `auth`, `type`, `scope`, `tags`,
   `requires` and the version trio **identical** in both documents.
5. `env[].name` = the real environment variable name (§4b).
6. For `qr`: implement `login_status` + `logout`, persist the session under the connector dir
   (§4d).
7. Deps declared as a file, **not** vendored (§5).
8. `docs[0].llm_short_description` says what the connector *does*, not which tools it has.
9. Add `"<id>"` to `connectors/index.json`.
10. Bump `version` (+1), `version_string`, `version_release_date` in **both** documents.
11. Run `python3 scripts/compile.py`, then `python3 scripts/compile.py --verify`.
12. Record the change in `CHANGELOG.md` in the same commit, then commit and deploy.