skald-connectors/CONNECTOR_MANIFEST_GUIDE.md

Skald Connector Authoring Guide

This file is the single source of truth for the connector format. It lives in the marketplace repo (connectors.skaldagent.net) and is referenced — not copied — by the Skald application repo (~/projects/skald-circle). Change the format here, and here only.

Give this file to any agent that produces new connectors.

A connector is a folder served by the marketplace. Skald installs it, verifies every file against a SHA-256 pinned in the index, then either runs it on the host (global connector) or copies it into the user's container and runs it there (per-user connector, blueprint §6/§7).

Everything below marked "the client enforces" was checked against the implementation in skald-circle (src/frontend/api/marketplace.rs, src/frontend/api/mcp.rs, crates/skald-core/src/mcp/{mod,install,verify,oauth}.rs).

---

0. Folder layout

One folder per connector, named exactly like its id, flat:

connectors/myconn/
├── fragment.json          ← index entry (compiler input, never served as-is)
├── connector.json         ← the manifest Skald reads
├── server.py              ← the entry file (or index.js…)
├── requirements.txt       ← or package.json
├── verify.py              ← optional
├── icon_sm.svg
└── icon_lg.svg

Keep the folder flat. The client accepts a relative sub-path (pkg/server.py) but scripts/compile.py only scans the folder's top level: files in a subdirectory are silently left out of files[], never downloaded, and the connector breaks at runtime with no error anywhere. If a tree is genuinely needed, teach the compiler to recurse first.

---

1. The three source documents (+ one compiler)

You maintain three files by hand. A fourth, connectors.json, is generated.

1a. connectors/index.json — the order

A flat JSON array of folder ids, in display order:

["gmail", "gcal", "myconn"]

This is the compiler's input list. A connector missing from it does not exist.

1b. connectors/<id>/fragment.json — the index entry

The connector's entry in the compiled index, with every field except files[] (the compiler adds that one).

{
  "id": "whatsapp",                      // unique slug = folder name
  "name": "WhatsApp",
  "version": 8,                          // INTEGER build number — the update key (§7)
  "version_string": "2.2.0",             // semver, display only
  "version_release_date": "2026-08-23",  // ISO date, display only
  "type": "mcp_local",                   // mcp_local | mcp_remote  (§3)
  "scope": "user",                       // user | global           (§3)
  "icon_small": "whatsapp/icon_sm.png",  // relative to the FEED ROOT, not the folder
  "icon_large": "whatsapp/icon_lg.png",
  "user_description": "Send and read WhatsApp messages from your linked account.",
  "requires": ["NODE"],                  // API_KEY | ENV | NODE | PYTHON | OAUTH | DOCKER
  "tags": ["messaging", "mcp", "local", "whatsapp", "qr"],
  "folder": "whatsapp",                  // defaults to id
  "auth": { "type": "qr" },              // informational here — see the table in §1e
  "tools": [ … ]                         // informational here — see §2a
}

Note the two icon vocabularies: icon_small / icon_large in the index are relative to the feed root (whatsapp/icon_sm.png), while files[] and connector.json name them from inside the folder (icon_sm.png). The client reconciles the two, but only records an icon whose file was actually installed — so icons must be shipped in the folder, never hot- linked.

1c. connectors/<id>/connector.json — the manifest

The richer document, fetched per connector and mapped into Skald's catalog.

{
  "id": "whatsapp",
  "name": "WhatsApp",
  "version": 8,                         // must match fragment.json (§7)
  "version_string": "2.2.0",
  "version_release_date": "2026-08-23",
  "type": "mcp_local",
  "scope": "user",
  "requires": ["NODE"],
  "tags": ["messaging", "mcp", "local", "whatsapp", "qr"],
  "auth": { "type": "qr" },             // none | api_key | oauth2 | qr | ssh_key  (§4)
  "launch_command": "node index.js",    // human-readable; mcp_config is what runs
  "transport": "stdio",                 // stdio | streamable-http (may also sit in mcp_config)
  "mcp_config": {
    "command": "node",                  // interpreter (local) …
    "args": ["index.js"]                // … args[0] MUST name the entry file
  },
  "dependencies": ["@whiskeysockets/baileys@7.0.0-rc.14"],  // display only (§5)
  "setup_instructions": ["Scan the QR code with WhatsApp → Linked devices"],
  "docs": [{
    "lang": "en",
    "description": "Human blurb shown in the UI.",
    "llm_short_description": "One line the model reads to decide whether to use this connector."
  }],
  "env": [],                            // form fields the user fills (§4b)
  "verify": { "command": "python3 verify.py", "timeout_secs": 20 },   // optional (§6)
  "tools": [{ "name": "send_message", "display_name": "Send Message" }],  // optional (§2a)
  "homepage": "https://web.whatsapp.com",
  "icon_small": "icon_sm.png",          // relative to the FOLDER here
  "icon_large": "icon_lg.png"
}

mcp_config.args[0] is load-bearing for a local connector: it is how Skald learns which file to run — an install fails outright without it. At activation Skald rewrites it to the file's path inside the user's container (/root/.skald/mcp/<name>/<entry>), so keep it a plain relative filename (index.js, server.py).

transport may sit either at the top level or inside mcp_config — the client reads mcp_config.transport first, then the top-level one, then infers (remote → http, otherwise stdio). Both spellings are in use in this repo; pick one per connector and keep it consistent between the two files. streamable-http and http both normalise to HTTP; anything unrecognised silently becomes stdio, which for a remote connector means trying to spawn a command that does not exist.

llm_short_description is model-facing: it is the one-liner injected into the LLM's system prompt so the model knows what this connector does. Keep it short and functional ("Weather — current conditions, 16-day forecast, and AQI data for any location"), not a list of tools (the model discovers those after activate_tools).

1d. connectors/connectors.json — the compiled index (generated)

Never edit this file by hand. scripts/compile.py reads index.json, loads each fragment.json, scans the folder's real files, computes their SHA-256 + size, and writes the result:

python3 scripts/compile.py           # regenerate
python3 scripts/compile.py --verify  # fail if the committed index is stale

The compiler excludes fragment.json, connectors.json, index.json, compile.sh, compile.py, update_hashes.py, .DS_Store and every subdirectory. Everything else is hashed, including connector.json.

"files": [
  { "path": "index.js",       "sha256": "…", "size": 21258 },
  { "path": "package.json",   "sha256": "…", "size": 302   },
  { "path": "connector.json", "sha256": "…", "size": 620   },
  { "path": "icon_sm.png",    "sha256": "…", "size": 306   }
]

• files[].path is relative to the connector folder.
• Digests and sizes are computed by the compiler. Never write them by hand.
• The index is the signable root: it is the one document that names a connector's files and their digests. The client refuses any file whose bytes do not match, all-or-nothing — a mismatch leaves nothing on disk.
• Do not ship node_modules/ or vendored wheels (§5).

1e. Who reads what — the authority table

The client hydrates a card from two documents, and they are not interchangeable. Getting this wrong is the most common way a correct-looking connector misbehaves.

Field	Read from	Notes
id, folder, tags	index (fragment.json)	folder defaults to id
name	index, falling back to manifest
user_description	index, falling back to docs[0].description	the human blurb
icon_small, icon_large	index	feed-root-relative; the file must be in files[]
files[]	index (manifest's is a legacy fallback)	the trust root
requires	manifest, falling back to index
type, scope	manifest, falling back to index	keep them identical
version trio	manifest wins, index is the fallback	a mismatch is logged as a desync warning (§7)
auth	manifest only	the index's auth is never parsed; only requires is used as a coarse fallback
tools[]	manifest only	a tools[] that lives only in fragment.json does nothing (§2a)
env[], verify, mcp_config, docs, dependencies, setup_instructions, homepage	manifest only

So: auth and tools[] in fragment.json are documentation for the catalog page and for whoever reads the index — harmless, worth keeping in sync, but the manifest is what runs.

---

2. Server contract (MCP over stdio)

A local connector is a program speaking JSON-RPC 2.0 over stdin/stdout. It MUST handle:

• initialize → { protocolVersion: "2024-11-05", capabilities: { tools: {} }, serverInfo }
• notifications/initialized → no response
• ping → {}
• tools/list → { "tools": [ { name, description, inputSchema, title? } ] } — an object, not a bare array
• tools/call → { content: [ { type: "text", text } ], isError? }

Any message without an id is a notification and must produce no output at all. Answering one desynchronises a strict client.

stdout is reserved for JSON-RPC only. Send all logs/diagnostics to stderr. Anything a library prints to stdout (a logger, a banner) corrupts the protocol — silence it (Baileys/pino → a silent logger; Python → print(…, file=sys.stderr)). Guard stdout writes with a lock if the server has background threads.

Report tool failures as a normal result carrying isError: true with a readable message — not as a JSON-RPC error, and not as a "successful" result with an {"status":"error"} body the model cannot distinguish from data.

A remote connector is an HTTP MCP endpoint (mcp_config.url + transport: "streamable-http"); no code runs on the box.

2a. Friendly tool names (tools[]) — optional

Raw MCP tool names are ugly in the chat UI (search_files, send_message). Two ways to fix that, in order of preference:

1. title in your tools/list entries (preferred for servers we control) — Skald uses it automatically, no manifest change needed.
2. The manifest's tools[] block — for remote connectors and third-party packages (npx -y firecrawl-mcp) whose tools/list we cannot edit:

"tools": [
  { "name": "send_message", "display_name": "Send Message" }
]

• name — the raw tool name exactly as the server returns it from tools/list.
• display_name — the friendly card title (English only; not internationalized).

Resolution order: tools[].display_name → the MCP title field → a prettified raw name (send_message → "Send Message").

⚠️ tools[] is read from connector.json. The client parses no tools field on the index entry, so a block placed only in fragment.json is inert. Put it in the manifest; mirroring it into fragment.json is optional and purely documentary.

Icons are per connector, not per tool. Every tool shows its connector's icon_small; there is no per-tool icon. Partial tools[] lists are fine — unlisted tools fall through.

---

3. Placement & risk vocabulary (what the words mean)

Manifest	Meaning
scope: "user"	runs once per user, inside their container. Personal creds. Stored as per_user.
scope: "global"	runs once for the household, on the host. Shared, stateless. Admin enables it with a key.
type: "mcp_local"	ships code that will execute on the box — installing needs the admin mcp.register_local_script capability (RCE-bearing act, §14). Stored as local_script.
type: "mcp_remote"	just an HTTP URL; no local code. Stored as remote.

Pick the narrowest: a personal messaging/email/calendar connector is scope: "user"; a shared search API is scope: "global". Both axes fail closed — an unreadable scope becomes per_user, an unreadable type becomes local_script, the answer that demands more authority.

requires[] is a human hint rendered as tags on the catalog page: API_KEY, ENV, NODE, PYTHON, OAUTH, DOCKER. The client only inspects it as a fallback when the manifest declares no auth (OAUTH → oauth, API_KEY → api_key). SECRETS_DIR is removed — the secrets/ folder is no longer part of the model.

---

4. Authentication (auth.type)

4a. The recognized values

auth.type	Flow	Ships
none	nothing to sign in	—
api_key	user pastes a key/secret into a form	an env[] schema (§4b)
oauth2	browser consent → token injected as an env var	provider + scopes + deliver (§4c)
qr	server shows a QR, user scans with a phone	a login_status tool (§4d)
ssh_key	reserved for key-based remote access	—

Anything else normalises to none. email declares auth.type: "password", which the client does not recognise and therefore treats as none; that connector works only because its credentials arrive through env[] like any other form field. Prefer api_key or none for new connectors and let env[] carry the credential.

auth.delivery (header | query | env) is surfaced to the admin UI. For a remote connector the URL placeholder is what actually routes the key (§4b) — the param name is not read by the client.

4b. env[] — the activation form, and what reaches the process

Each entry drives one form field, and the collected values are injected into the server process environment:

"env": [{
  "name": "EMAIL_IMAP_HOST",            // ← the ACTUAL environment variable name
  "label": "IMAP host",
  "description": "IMAP server hostname (e.g. imap.gmail.com).",
  "required": true,
  "secret": false,                      // true → masked in the form, stored encrypted
  "example": "imap.gmail.com",
  "default": ""                         // non-required fields only
}]

name must be exactly the environment variable your server reads. The process receives the form's name → value map verbatim; there is no renaming layer. mcp_config.env is not substituted at runtime — it survives only as a legacy fallback for the form schema, and if it is ever the sole env source its {ENV:…} strings are injected literally. Do not use it to map one name onto another.

The server reads each value from os.environ / process.env. Declare requires: ["ENV"] when the connector needs user-supplied config.

Placeholders. Two tokens, and exactly two places where they are substituted:

Token	Meaning
{ENV:NAME}	non-sensitive value (host, port, username…)
{SECRET:NAME}	sensitive value (password, API key, token)

Substituted in	Engine behaviour on an unknown name
mcp_config.url (remote connectors)	{SECRET:x} falls back to the connector's api_key; if that is absent the token is left in the URL literally, so a misconfiguration is visible
verify.command (§6)	replaced with the empty string

Nowhere else — not in mcp_config.env, not in args. Legacy {key} still resolves to the api_key in URLs; {env:NAME} and {secrets}/… are dead and are never substituted.

Remote example: "url": "https://mcp.tavily.com/mcp/?tavilyApiKey={SECRET:tavilyApiKey}" with a matching env[] entry named tavilyApiKey.

4c. oauth2 — provider consent

"auth": {
  "type": "oauth2",
  "provider": "google",                 // slug into the admin's sign-in providers
  "scopes": ["https://www.googleapis.com/auth/gmail.modify"],
  "deliver": { "as": "env", "format": "google_authorized_user", "env": "GMAIL_CREDS_JSON" }
}

The manifest names only the provider slug, the scopes, and how the obtained token is delivered — never client ids, client secrets, endpoint URLs or redirect URIs, which are admin-entered and stay off the public feed. Skald runs PKCE + code exchange and injects the credential as the named env var.

• deliver.as: only "env" is implemented — "file" is rejected at activation with an explicit error rather than half-working. Do not ship it.
• deliver.format: google_authorized_user (the JSON from_authorized_user_file reads) or refresh_token.
• deliver.env must not also appear in mcp_config.env — Skald injects it at runtime.
• An OAuth connector activates into a pending row; nothing starts until the consent round-trip completes. The admin must have configured the provider first, or activation fails early with a clear message.

4d. qr / interactive device login

For a connector whose credential is produced by scanning/pairing (WhatsApp today) there is no code to paste. The contract:

Expose one extra tool, login_status, returning a JSON object (as the text of a normal text result). Skald calls it directly — never the agent — and a login panel polls it.

// login_status result text (a JSON string):
{
  "state": "connecting" | "need_scan" | "ready" | "logged_out",
  "qr": "data:image/png;base64,…",   // present ONLY while state == need_scan
  "message": "human-readable line"
}

• Activating a qr connector inserts a pending row and starts the server (so it can produce the QR), then hands off to the login panel.
• The panel polls POST /api/mcp/login/status; when state == "ready" the connector is marked ready and starts automatically on later logins.
• Also expose a logout tool (clears the session, forces a fresh QR) — the panel calls it via POST /api/mcp/login/reset to re-link a different phone.
• The credential is the on-disk session, not a token. Persist it inside the connector's own directory (e.g. ./auth/ next to the entry file). That folder lives under the bind- mounted home, so it survives container recreates and connector updates. Never store it under a shared or global path.

---

5. Dependencies (node & python) — how they get installed

Do not ship node_modules/ or vendored wheels. Declare deps as a standard manifest file and Skald installs them where the server will run:

• node: ship a package.json with a dependencies map. Skald runs npm ci --omit=dev --no-audit --no-fund, falling back to npm install --omit=dev … when there is no lockfile. node_modules/ resolves automatically beside the entry file.
• python: ship a requirements.txt. Skald runs python3 -m pip install --break-system-packages --target .pydeps -r requirements.txt and puts .pydeps on the server's PYTHONPATH.

A single install has a 300 s ceiling; past that it fails rather than hanging a login.

This runs at activation and on every startup, guarded by a content hash of the connector's source files (.skald-install.lock):

• first activation / a brand-new container → full install,
• a connector update (any shipped file changed) → re-copy + re-install,
• unchanged → skipped in microseconds.

So you never write install steps into the manifest — just ship the dep file. The manifest's dependencies[] and setup_instructions[] are display metadata for the admin card; they install nothing. Set requires: ["NODE"] / ["PYTHON"] as a human hint. Pin versions for reproducible installs, and keep the tree lean (containers are slim; prefer a pure-JS/Python library over a native-heavy one — e.g. Baileys instead of a browser).

Host assets are not copied into the container. Icons (.svg/.png/.jpg/.webp/.gif/.ico) and connector.json stay on the host, so a server must never expect to read them at runtime. Everything else in the folder — entry file, deps file, helper modules, verify.py — is copied.

Where the files land:

	Host	Per-user container
installed folder	<workdir>/connectors/<id>/	~/.skald/mcp/<name>/ (/root/.skald/mcp/<name>/)
python deps	connectors/<id>/.pydeps	~/.skald/mcp/<name>/.pydeps

---

6. Verify-before-save (optional, strongly recommended)

Ship a verify.py / verify.js (or an inline snippet) and reference it:

"verify": { "command": "python3 verify.py", "timeout_secs": 20 }

It runs after the user fills the form and before the activation is persisted, with the collected env/secrets injected and {ENV:…} / {SECRET:…} substituted into the command.

Output contract — one JSON object on stdout and nothing else:

{"ok": true,  "message": "IMAP and SMTP authentication successful", "details": {"imap": "…"}}
{"ok": false, "message": "IMAP login failed: INVALID_CREDENTIALS"}

Exit 0 on success. If the JSON fails to parse the client falls back to the exit code (0 = ok) and shows stderr. A timeout counts as a failure. Never print credentials in message or details.

Rules the client enforces:

• The script must be one of the connector's shipped files. The client resolves it by matching a basename from files[] against the command string — so python3 verify.py works because verify.py is in files[]. A script that is not shipped is never downloaded and activation fails; an inline command with no matching basename (firecrawl's node -e "…") is fine and runs as-is.
• Where it runs: inside the user's container, in the connector's directory, for a per_user connector; on the host in connectors/<id>/ for a global one.
• Timeout: declare timeout_secs for the record, but the runtime currently applies a fixed 20 s to every verify. Keep the probe well under that.
• Any auth.type may use verify — it is not limited to api_key. A qr connector needs none: its login_status is the live check.

Without verify there is no test at all. Activation goes straight to ready — including for remote connectors, which get no handshake fallback. The manifest author decides.

---

7. Versioning & updates

Three fields, in both fragment.json and connector.json, kept identical:

field	type	role
version	integer	monotonic build number, per connector — the machine comparison key
version_string	string (semver)	display only
version_release_date	ISO date YYYY-MM-DD	display only

• version is a number, not a string (8, not "8" or "2.2.0"). +1 on every change to any shipped file or to any manifest metadata (description, icons, version_string). Never reuse or decrement.
• The integer is the only "is there an update?" signal, compared strictly (feed > installed). version_string, icons and llm_short_description are never compared, so changing them without bumping the integer is invisible — no "update available" badge. This is the classic trap.
• The manifest wins over the index when the two disagree, and the disagreement is only visible as a log line (marketplace feed version desync). If the index carries the lower number and the manifest the higher one, the strict comparison can never fire again and the connector silently stops offering updates. Keep them equal — compile.py does not check this for you.
• Two propagation paths, do not conflate them:
  • Per-user code + deps reconcile on a content hash of the source files (§5), so new code lands at each user's next login even without a reinstall.
  • Catalog metadata (llm_short_description → the model's prompt, icons, friendly name) is not in that hash. It lives in the catalog row and is rewritten only by an explicit reinstall/Update, which re-pulls the current feed (never the 300 s browse cache) and restarts enabled global servers and every live user's copy with the fresh description.
• So: to ship a new llm_short_description, bump the integer so the admin sees "update available" and clicks Update. Nothing auto-propagates a description change.

---

8. Hard limits the client enforces

• Digest mismatch → refused, all-or-nothing: nothing is written unless every file verifies.
• 8 MiB per file. A files[].size above that is rejected before the download starts.
• Path safety. A files[].path or mcp_config.args[0] that is absolute, contains .., a backslash or a colon is rejected outright.
• A local connector with no files[] cannot be installed ("refusing to install unverifiable code"), and neither can one whose only file is connector.json.
• mcp_config.args[0] is mandatory for mcp_local.
• The feed is fetched as GET /connectors.json plus one GET /<folder>/connector.json per entry, cached 300 s for browsing; installs always refetch. A manifest that fails to load degrades that one card to whatever the index said — it never fails the listing, which is exactly how a broken manifest hides.

---

9. Checklist for a new connector

1. Create connectors/<id>/ — flat — with: entry file, connector.json, fragment.json, deps file (package.json / requirements.txt), icon_sm.*, icon_lg.*, optional verify.*.
2. Server speaks MCP over stdio (§2); stdout = JSON-RPC only; notifications answered with silence; title on every tool.
3. mcp_config.args[0] names the entry file; transport set consistently.
4. Correct type + scope (§3) and auth.type (§4); auth, type, scope, tags, requires and the version trio identical in both documents.
5. env[].name = the real environment variable name (§4b).
6. For qr: implement login_status + logout, persist the session under the connector dir (§4d).
7. Deps declared as a file, not vendored (§5).
8. docs[0].llm_short_description says what the connector does, not which tools it has.
9. Add "<id>" to connectors/index.json.
10. Bump version (+1), version_string, version_release_date in both documents.
11. Run python3 scripts/compile.py, then python3 scripts/compile.py --verify.
12. Record the change in CHANGELOG.md in the same commit, then commit and deploy.