dguiducci/skald-connectors
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
skald-connectors/CLAUDE.md
T
Daniele d84ce13dab ssh: fix sudo failing with "password required" (v6 / 1.1.0) 
The connector asked for a sudo password on every privileged call and
failed whenever nobody answered it, which is every unattended run.

- Always probe `sudo -n` first, even for aliases set to sudo="prompt".
  `_sudo_prefix` used to elicit unconditionally, so a host granting this
  user NOPASSWD still opened an Agent Inbox prompt; with no human there
  it hit the client's 300s ELICITATION_DEADLINE, got back `cancel`, and
  surfaced as "sudo password required (user declined or timed out)".
  sudo refuses before running anything when it wants a password, so the
  probe is side-effect free.
- Strip a leading `sudo` from `command` and turn it into sudo=true.
  Agents write `exec(command="sudo systemctl restart x")`: with
  sudo=false that ran a tty-less sudo, with sudo=true it nested
  `sudo -S ... sudo ...` whose inner prompt had no tty either. Handles
  -u/-n/-S/-E/-H/-i/-k/-p/--; an unknown flag leaves the command alone.
  sudo_user now implies sudo=true.
- Run privileged commands as `sh -c '<command>'`, so `&&`, pipes and
  redirections are elevated too instead of only the first word.
- Add SSH_MCP_SUDO_PASSWORD (optional, secret) for unattended runs. It
  is consulted only after `sudo -n` proved a password is needed, so on a
  NOPASSWD host it never lands in the command's own stdin.
- Actionable errors for every sudo failure mode, and a `hint` on a
  nested sudo we could not peel off.

General review of the same server:

- Drain stdout and stderr together and make timeout_sec a real
  wall-clock deadline. Both streams share one SSH channel window, so
  reading stdout to EOF first stalled once a chatty stderr filled it.
  Command stdin is now closed after the optional password.
- Queue messages that arrive while awaiting an elicitation reply instead
  of discarding them, so a concurrent tools/call is not lost.
- Record the client's `elicitation` capability at initialize and fail
  fast when it is absent rather than blocking on a prompt nobody can
  answer.
- Tolerate null/string integer arguments (depth, max_results,
  context_lines, timeout_sec).
- Realign the version across both manifests: fragment.json said 5/1.0.4
  while connector.json said 2/1.0.1, so the feed was permanently ahead
  of the installed version and offered an update forever.

Also lands the pending docs work: CONNECTOR_MANIFEST_GUIDE.md as the
single source of truth, docs/connector.manifest_guide.md retired to a
pointer, CLAUDE.md audited against the repo, compile.py docstring fixed,
and an opencode.json config.
2026-09-03 23:48:00 +01:00

677 lines
33 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
It documents the repo layout, the schemas, and the build/deploy workflow — read it before changing any
manifest. The **authoritative spec** for authoring a connector lives in this repo as
[CONNECTOR_MANIFEST_GUIDE.md](CONNECTOR_MANIFEST_GUIDE.md) (see § The spec).
## What this is
The **Skald Connectors Marketplace** — the catalog of tested connectors for Skald. Each connector is an
adapter that lets a Skald agent talk to an external service (API, email, calendar, search, messaging…).
It is **not an application**: there is no build system, package manager, or test suite. The repo is a set
of JSON manifests, static HTML, icons, and standalone MCP scripts (Python/Node) served as-is over HTTP.
The only tooling is `scripts/compile.py`, which regenerates the index.
- **Remote git**: `https://git.skaldagent.net/dguiducci/skald-connectors.git` (branch `main`)
- **Live site**: `https://connectors.skaldagent.net/`
- **OAuth callback**: `https://connectors.skaldagent.net/oauth/show.html`
`main` is the **release** branch — only production-ready code lands here. Development/alpha versions will
live on separate branches in the future.
### The consuming project
The application that consumes this marketplace — the code that installs, verifies, and runs these
connectors — lives in a **separate repository at `~/projects/skald-circle`** (Rust workspace: `crates/`,
`Cargo.toml`, `blueprint/`, `agents/`). It is the client for everything documented here: it fetches
`connectors.json`, checks each file against its pinned SHA-256, resolves `auth` / `env` / `verify`, and
launches the MCP servers. The parts worth reading when a manifest question comes up:
| Question | File in `~/projects/skald-circle` |
|----------|-----------------------------------|
| How the feed is parsed, hashed, installed | `src/frontend/api/marketplace.rs` |
| Activation, the env form, OAuth/QR handoff | `src/frontend/api/mcp.rs` |
| Server spec, URL placeholders, transports | `crates/skald-core/src/mcp/mod.rs` |
| Folder layout, container copy, deps install | `crates/skald-core/src/mcp/install.rs` |
| Verify runner + `{ENV:}`/`{SECRET:}` engine | `crates/skald-core/src/mcp/verify.rs` |
It **no longer keeps a copy** of the spec: it references
[CONNECTOR_MANIFEST_GUIDE.md](CONNECTOR_MANIFEST_GUIDE.md) in this repo, which is the single source of
truth for the connector format. Edit the format here and nowhere else.
### The spec
The marketplace spec ships **inside this project**: [CONNECTOR_MANIFEST_GUIDE.md](CONNECTOR_MANIFEST_GUIDE.md)
at the repo root is the authoritative, step-by-step specification for producing a correct connector —
the folder layout, the three source documents + the compiler, which document the client reads each field
from, the MCP-over-stdio server contract, friendly tool names, placement & risk vocabulary, the
`auth.type` variants, the `env[]` form and placeholder engines, dependency installation,
verify-before-save, versioning, the limits the client enforces, and the new-connector checklist. Give
this file to any agent that generates new connectors, and update it whenever the connector format
changes — it is the **only** copy, referenced by `~/projects/skald-circle` rather than duplicated there.
[docs/connector.manifest_guide.md](docs/connector.manifest_guide.md) is a retired copy, now a pointer to
the root guide. Do not edit it.
## Architecture
Two-tier manifest model with a single trust root, plus a compile step:
- **`connectors/index.json`** — ordered list of connector ids. The input list for `compile.py`.
- **`connectors/<id>/fragment.json`** — the connector's **index entry without `files[]`** (id, name, type,
scope, icons, `user_description`, requires, tags, version, `tools[]`, auth…).
- **`connectors/<id>/connector.json`** — per-connector **technical manifest** used at activation time
(launch command, transport, `mcp_config`, `auth`, `env`, `verify`, `dependencies`, `docs`). One folder
per connector; the folder name matches the connector `id`.
- **`connectors/connectors.json`** — the **compiled index** and single root of trust. Generated by
`scripts/compile.py` from `index.json` + each `fragment.json` + a physical scan of every connector
folder. It lists every connector with a `files[]` array carrying the `sha256` + `size` of each shipped
file. **Never edit it by hand.** It carries no hash of itself — in the future it may be digitally signed.
- **`connectors/index.html`** — catalog UI. Fetches `/connectors.json` at an absolute path and links to
`/<folder>/`, so it only works when served from the site root (not opened as a `file://`).
- **`connectors/oauth/show.html`** — OAuth callback receiver.
```
connectors/
├── index.json ← ordered list of connector ids (input for compile.py)
├── connectors.json ← COMPILED INDEX (generated by compile.py, do not edit)
├── index.html ← catalog UI (reads connectors.json via fetch)
├── oauth/
│ └── show.html ← OAuth callback receiver
├── gmail/ ← one connector per folder
│ ├── fragment.json ← index fragment (id, name, type, …, WITHOUT files[])
│ ├── connector.json ← technical configuration (mcp_config, auth.deliver, …)
│ ├── gmail_mcp_server.py ← MCP script
│ ├── requirements.txt ← Python dependencies
│ ├── icon_sm.svg ← small icon (~48×48)
│ └── icon_lg.svg ← large icon (~96×96)
├── email/
│ ├── fragment.json
│ ├── connector.json
│ ├── email_mcp_server.py
│ ├── verify.py
│ ├── icon_sm.svg
│ └── icon_lg.svg
└── …
```
## Schema — `connectors.json` (compiled root index)
```json
{
"version": 1,
"connectors": [
{
"id": "gmail",
"name": "Gmail",
"type": "mcp_local",
"scope": "user",
"icon_small": "gmail/icon_sm.svg",
"icon_large": "gmail/icon_lg.svg",
"user_description": "Read, send, and manage Gmail emails via OAuth…",
"requires": ["OAUTH", "PYTHON"],
"tags": ["email", "mcp", "local", "google"],
"auth": {
"type": "oauth2",
"provider": "google",
"scopes": [
"https://www.googleapis.com/auth/gmail.modify",
"https://www.googleapis.com/auth/gmail.labels"
]
},
"folder": "gmail",
"version": 1,
"version_string": "1.0.0",
"version_release_date": "2026-07-19",
"files": [
{"path": "connector.json", "sha256": "…", "size": 1204},
{"path": "gmail_mcp_server.py", "sha256": "…", "size": 46772},
{"path": "icon_lg.svg", "sha256": "…", "size": 254},
{"path": "icon_sm.svg", "sha256": "…", "size": 251},
{"path": "requirements.txt", "sha256": "…", "size": 82}
]
}
]
}
```
### Index fields
| Field | Required | Description |
|-------|----------|-------------|
| `id` | ✅ | Unique identifier (kebab-case) |
| `name` | ✅ | Displayed name |
| `type` | ✅ | `mcp_remote` or `mcp_local` |
| `scope` | ✅ | `global` or `user` |
| `icon_small` | ✅ | Path relative to the marketplace root |
| `icon_large` | ✅ | Path relative to the marketplace root |
| `user_description` | ✅ | Short description for the UI |
| `requires` | ✅ | Array of requirement enums |
| `tags` | ✅ | Array of tags for filtering |
| `folder` | ✅ | Name of the connector folder |
| `version` | ✅ | Per-connector integer, +1 on every file change |
| `version_string` | ✅ | Semver (display only) |
| `version_release_date` | ✅ | ISO 8601 date `YYYY-MM-DD` (display only) |
| `files` | ✅ | Array of `{path, sha256, size}` — added automatically by `compile.py`, NO self-hash |
## Schema — `fragment.json` (per folder)
`fragment.json` contains all the fields of a `connectors.json` entry **except** `files[]`:
| Field | Required | Description |
|-------|----------|-------------|
| `id` | ✅ | Unique identifier (matches the folder name) |
| `name` | ✅ | Displayed name |
| `type` | ✅ | `mcp_remote` or `mcp_local` |
| `scope` | ✅ | `global` or `user` |
| `icon_small` | ✅ | Path relative to the marketplace root |
| `icon_large` | ✅ | Path relative to the marketplace root |
| `user_description` | ✅ | Short description for the UI |
| `requires` | ✅ | Array of requirement enums |
| `tags` | ✅ | Array of tags for filtering |
| `folder` | ✅ | Name of the connector folder (matches `id`) |
| `version` | ✅ | Per-connector integer, +1 on every file change |
| `version_string` | ✅ | Semver (display only) |
| `version_release_date` | ✅ | ISO 8601 date (display only) |
| `tools` | optional | Array of `{name, display_name}` for friendly UI names. **Documentary here** — Skald reads `tools[]` from `connector.json` (see § Friendly tool names) |
| `auth` | optional | Authentication configuration (if other than `"none"`) |
The `files[]` array is added **automatically** by `compile.py` from the files present in the folder — it
must never be written by hand.
## Schema — `connector.json` (per folder)
Technical configuration for connector activation.
```json
{
"id": "gmail",
"name": "Gmail",
"version": 1,
"version_string": "1.0.0",
"version_release_date": "2026-07-19",
"type": "mcp_local",
"scope": "user",
"launch_command": "python3 gmail_mcp_server.py",
"transport": "stdio",
"requires": ["OAUTH", "PYTHON"],
"tags": ["email", "mcp", "local", "google"],
"dependencies": [
"google-api-python-client>=2.150.0",
"google-auth>=2.35.0",
"google-auth-oauthlib>=1.2.0"
],
"setup_instructions": [
"Install dependencies: pip install -r requirements.txt"
],
"docs": [
{
"lang": "en",
"description": "Full description for human users…",
"llm_short_description": "Gmail — read, send, label, and search email. Supports push notifications."
}
],
"auth": {
"type": "oauth2",
"provider": "google",
"scopes": [
"https://www.googleapis.com/auth/gmail.modify",
"https://www.googleapis.com/auth/gmail.labels"
],
"deliver": {
"as": "env",
"format": "google_authorized_user",
"env": "GMAIL_CREDS_JSON"
}
},
"mcp_config": {
"command": "python3",
"args": ["gmail_mcp_server.py"]
},
"homepage": "https://mail.google.com",
"icon_small": "icon_sm.svg",
"icon_large": "icon_lg.svg"
}
```
### `connector.json` fields
| Field | Required | Description |
|-------|----------|-------------|
| `id` | ✅ | Unique identifier (matches the folder name) |
| `name` | ✅ | Displayed name |
| `version` | ✅ | Per-connector integer, +1 on every file change |
| `version_string` | ✅ | Semver (display only) |
| `version_release_date` | ✅ | ISO 8601 date (display only) |
| `type` | ✅ | `mcp_remote` or `mcp_local` |
| `scope` | ✅ | `global` or `user` |
| `requires` | ✅ | Array of requirement enums |
| `tags` | ✅ | Array of tags |
| `auth` | ✅ | Authentication configuration object |
| `docs` | ✅ | Array of multilingual documentation. **`llm_short_description`** is the field that ends up in the LLM's system prompt — it must describe WHAT the connector DOES, not list its tools (the LLM sees them after `activate_tools`). Example: *"Weather — current conditions, 16-day forecast, and AQI data for any location."* |
| `icon_small` | ✅ | Icon filename in the local folder |
| `icon_large` | ✅ | Icon filename in the local folder |
| `launch_command` | `mcp_local` only | Command to start the MCP server |
| `transport` | `mcp_local` only | `stdio` (default) |
| `dependencies` | recommended | Python/Node dependencies (empty array if stdlib only) |
| `env` | if the connector needs user-supplied config | Environment variables the user must provide (schema for the UI) — see § The env field |
| `verify` | recommended | Test-before-save command — see § The verify field |
| `setup_instructions` | recommended | Steps to configure the connector |
| `mcp_config` | ✅ | Configuration for the MCP client (`command`/`args` for `mcp_local`, `url`/`transport` for `mcp_remote`) |
| `homepage` | optional | Service URL |
## Reserved enums
### `type` (connector type)
| Value | Description | Examples |
|-------|-------------|----------|
| `mcp_remote` | Hosted MCP server, reachable via URL | Tavily, Exa |
| `mcp_local` | Script to run locally | Gmail, Google Calendar, WhatsApp |
| `script` | Standalone script (non-MCP) | *(future)* |
### `scope` (configuration scope)
| Value | Description | Examples |
|-------|-------------|----------|
| `global` | A single instance/config for the whole system | Tavily, Weather, Google Trends |
| `user` | Each user has their own instance/authentication | Gmail, WhatsApp, Google Calendar |
### `requires` (prerequisites)
| Value | Description |
|-------|-------------|
| `API_KEY` | Requires an API key to configure |
| `OAUTH` | Requires OAuth authentication (Google, etc.) |
| `DOCKER` | Requires Docker Engine |
| `NODE` | Requires Node.js runtime |
| `PYTHON` | Requires Python 3 |
| `ENV` | Requires environment variables (declared in the manifest `env` field) |
| `SECRETS_DIR` | ❌ **Deprecated** — the `secrets/` folder is removed from the model; connectors must use `ENV`/`SECRET` (see § Placeholder syntax) |
## The `auth` field
Structure describing how the connector handles authentication:
```json
// API key in query string
{"type": "api_key", "delivery": "query", "param": "tavilyApiKey"}
// API key in header
{"type": "api_key", "delivery": "header", "param": "X-API-Key"}
// API key delivered as an environment variable
{"type": "api_key", "delivery": "env", "param": "GOOGLE_MAPS_API_KEY"}
// OAuth2 — provider is ONLY a slug (Skald resolves endpoints + client secrets)
{"type": "oauth2", "provider": "google", "scopes": ["…", "…"]}
// OAuth2 with deliver (Skald injects the authorized_user JSON via env var)
{"type": "oauth2", "provider": "google", "scopes": ["…"],
"deliver": {"as": "env", "format": "google_authorized_user", "env": "GMAIL_CREDS_JSON"}}
// OAuth2 with file-based deliver — ❌ NOT IMPLEMENTED: Skald rejects `as: "file"` at
// activation with an explicit error. Never ship it.
{"type": "oauth2", "provider": "google", "scopes": ["…"],
"deliver": {"as": "file", "format": "google_authorized_user", "path": "{secrets}/gmail_creds.json"}}
// Password / app-password provided via environment variables — ⚠️ `password` is NOT one of
// the values Skald recognizes (`none`/`api_key`/`oauth2`/`qr`/`ssh_key`); it normalizes to
// `none`. `email` works only because its credentials travel through `env[]`. Prefer
// `api_key` or `none` for new connectors.
{"type": "password", "delivery": "env"}
// QR-code pairing at runtime (WhatsApp)
{"type": "qr"}
// No authentication
{"type": "none"}
```
### The `deliver` field (OAuth2 only)
Declares **how** Skald delivers the obtained OAuth credential to the MCP server process.
| Field | Required | Description |
|-------|----------|-------------|
| `as` | ✅ | `"env"` (environment variable). `"file"` parses but is **rejected at activation** — unimplemented |
| `format` | ✅ | Name of the serialization — e.g. `"google_authorized_user"` (Google JSON that `from_authorized_user_file` reads), `"refresh_token"`, `"access_token"` |
| `path` | `as=file` only | Unused while file delivery is unimplemented |
| `env` | `as=env` only | Name of the environment variable into which Skald injects the entire authorized_user JSON. **Must not be declared in `mcp_config.env`** — Skald injects it at runtime. |
The feed NEVER contains `client_id`, `client_secret`, the endpoint URL, or `redirect_uri`. These are
resolved on the Skald side from the `provider` name.
## The `env` field (environment variables)
An **array** declaring the environment variables the user must provide to make the connector work.
**No credential lands on disk nor in `secrets/`** — the host collects the values via a real form
(masking `secret: true` fields, enforcing `required`, using `example` as placeholder) and injects them
into the MCP server process environment at launch. The server reads them from `os.environ`.
```json
"env": [
{
"name": "EMAIL_IMAP_HOST",
"label": "IMAP host",
"description": "IMAP server hostname (e.g. imap.gmail.com)",
"required": true,
"secret": false,
"example": "imap.gmail.com"
},
{
"name": "EMAIL_PASSWORD",
"label": "Password / app password",
"description": "Password or app-password of the provider",
"required": true,
"secret": true,
"default": ""
}
]
```
| Field | Required | Description |
|-------|----------|-------------|
| `name` | ✅ | Name of the environment variable (UPPER_SNAKE_CASE) |
| `label` | ✅ | Short label for the UI |
| `description` | ✅ | Help text |
| `required` | ✅ | If `true`, the host forces the user to provide a value |
| `secret` | recommended | If `true`, sensitive value (masked, not logged) |
| `default` | optional | Value used when not provided (non-required fields only) |
| `example` | optional | Example placeholder for the UI |
The `email` connector is the reference example. `tavily`, `gmaps`, and `linkedin` also declare `env[]`.
## Placeholder syntax (unified)
Every value skald must fill at runtime with user-provided data uses **one of two tokens**:
| Token | Meaning | Example |
|-------|---------|---------|
| `{ENV:NAME}` | Non-sensitive variable (hostname, port, username…) | `{ENV:EMAIL_IMAP_HOST}` |
| `{SECRET:NAME}` | Sensitive variable (password, API key, token) | `{SECRET:EMAIL_PASSWORD}` |
`NAME` is the `name` field declared in the `env[]` array. skald collects the values via a form (masking
`{SECRET:}` fields) and injects them as environment into the MCP server / verify process.
**Substitution happens in exactly two places** — there is no general template engine:
| Where | Engine | Unknown `NAME` resolves to |
|-------|--------|----------------------------|
| `mcp_config.url` (remote connectors) | `skald-core/src/mcp/mod.rs` | `{SECRET:x}` falls back to the connector's api_key; otherwise the token is **left in the URL literally** |
| `verify.command` | `skald-core/src/mcp/verify.rs` | the **empty string** |
⚠️ **`mcp_config.env` is NOT substituted.** What reaches a local server's environment is the form's
`env[]` values keyed by their `name`, verbatim — so `env[].name` must be exactly the variable name the
script reads. The `mcp_config.env` map survives only as a legacy fallback for the form schema (bare key
names); if it is ever the sole env source its `{ENV:…}` strings are injected literally. `email`, `ssh`
and `linkedin` still carry such a map — harmless today, but do not rely on it to rename a variable.
Rules:
- Unrecognized tokens (`{secrets}/…`, `{env:NAME}`) are **deprecated**: skald does not substitute them
and the manifest must be updated. Legacy `{key}` still resolves to the api_key in a URL.
- `{SECRET:<auth.param>}` is reserved for the primary key when `auth.type = "api_key"` (e.g. Tavily:
`?tavilyApiKey={SECRET:tavilyApiKey}`). skald also treats that value as the API key for bearer/header
routing, and stops sending it as a bearer header once it has been spent on the URL.
### Deprecations
| Token | Status | Replacement |
|-------|--------|-------------|
| `{key}` | ❌ deprecated | `{SECRET:<auth.param>}` |
| `{env:NAME}` | ❌ deprecated | `{ENV:NAME}` |
| `{secrets}/…` | ❌ deprecated | Declare the path as `{ENV:…}` (the `secrets/` folder is removed from the model) |
## The `verify` field (test before save)
Declares a shell command that skald runs **after** the user fills in the form and **before** persisting
the activation, to confirm the credentials just entered actually work.
```json
"verify": {
"command": "python3 verify.py",
"timeout_secs": 20
}
```
| Field | Required | Description |
|-------|----------|-------------|
| `command` | ✅ | Shell command. Runs in the same sandbox as the server: container `skald-{userid}` for `mcp_local` user, host for `mcp_remote` global. The declared env/secrets are injected |
| `timeout_secs` | optional | Declared for the record, but **not yet plumbed through**: the runtime applies a fixed 20 s to every verify (`VERIFY_TIMEOUT_SECS` in `src/frontend/api/mcp.rs`). Keep the probe well under that |
### Output convention
The command must print **a single JSON object on stdout** and nothing else:
```json
{"ok": true, "message": "IMAP and SMTP authentication successful", "details": {"imap": "…", "smtp": "…"}}
{"ok": false, "message": "IMAP login failed: INVALID_CREDENTIALS"}
```
| Field | Type | Description |
|-------|------|-------------|
| `ok` | bool | `true` = test passed |
| `message` | string | Message shown to the user (never log secrets inside) |
| `details` | object | Optional, structured details shown in `<pre>` |
Exit code: 0 on success, ≠ 0 on failure (skald uses the exit code as a fallback if the JSON parse
fails). **Never print credentials** in `message`/`details`.
### Where to put the script
If `command` references a file (e.g. `verify.py`), the file must be saved in the connector folder
(`<id>/verify.py`); `compile.py` then picks it up into `files[]` with its SHA-256 and size. skald finds
the script by matching a **basename from `files[]`** against the command string, downloads it, verifies
the hash against the index, and runs it in the connector's own directory — inside the container
`skald-{userid}` (`~/.skald/mcp/<name>/`) for a per-user connector, on the host in `./connectors/<id>/`
for a global one. A `verify` command may also be fully inline (see `firecrawl`, which uses a
`node -e "…"` one-liner) — with no basename match, nothing extra is fetched.
### Without `verify`
If `verify` is absent, skald **runs no test** — activation is direct and the connector goes to
`auth_state='ready'` without verification. For `mcp_remote` there is no handshake fallback: the manifest
author decides whether the test is needed by writing `verify`.
## Friendly tool names
Every MCP tool must expose a friendly name for the Skald UI. Two ways, in order of preference:
1. **Via the MCP script (preferred)** — add `"title": "Friendly Name"` to each tool definition returned by
`tools/list`. Works for all local scripts (Python/Node) that we control.
2. **Via the manifest (fallback)** — add `"tools": [{"name": "…", "display_name": "…"}]` to
**`connector.json`**. Used only for remote connectors or external packages (e.g.
`npx -y firecrawl-mcp`).
**Resolution order** used by Skald:
1. `tools[].display_name` from the manifest
2. `title` from the MCP server's `tools/list`
3. Automatic prettify of the raw name (`send_message` → "Send Message")
⚠️ **Skald reads `tools[]` from `connector.json` only** — the index entry carries no `tools` field in
the client's parser, so a block that lives only in `fragment.json` is inert (that is the state of
`google-trends` today, which is harmless because its script also sets `title`). Mirroring the block into
`fragment.json` is optional and purely documentary.
As of 2026-07-21 all marketplace connectors carry `title` in the script or `tools[]` in the manifest.
## MCP server conventions
The local MCP servers (`connectors/gmail/gmail_mcp_server.py`, `connectors/email/email_mcp_server.py`,
`connectors/wikipedia/`, `connectors/weather/`, `connectors/gmaps/`, …) are hand-rolled JSON-RPC 2.0
servers over stdio — no MCP SDK / FastMCP. Shared conventions, which new local connectors must mirror:
- **stdout is reserved for JSON-RPC**; all logging goes to stderr, and a lock guards stdout writes.
- `handle_request` implements the full handshake: `initialize` → `protocolVersion 2024-11-05` +
`serverInfo`, a silent `notifications/initialized`, `ping` → `{}`, and `tools/list` → `{"tools": TOOLS}`
(an **object**, not a bare array).
- **Notifications are never answered**: any message without an `id` returns `None`.
- Tools are declared as a `TOOLS` manifest list plus a `TOOL_DISPATCH` map.
- Where relevant, a background thread emits push notifications (e.g. `event/new_email`).
Connector-specific notes:
- **Gmail** — Gmail API with OAuth. Skald delivers the credential as `deliver: env/google_authorized_user`
into `GMAIL_CREDS_JSON`; the script also falls back to `GMAIL_CREDS_PATH` or `./secrets/gmail_creds.json`
for standalone use (that `secrets/` path is **deprecated** and gitignored). Push = History API polling.
`verify` is not yet wired — it awaits Phase 2 (OAuth via loopback listener).
- **Email** — generic IMAP+SMTP, **stdlib-only (no dependencies)**, configured entirely from `{ENV:}`/
`{SECRET:}` env vars, works with any provider. Push = IMAP IDLE with a 60s polling fallback; the request
thread and the watcher thread each hold their own IMAP connection (imaplib is not thread-safe). Note:
imaplib does not quote SEARCH arguments, so values with spaces must be wrapped via `_q()`.
- **Tavily** — `mcp_remote`/`global`; API key declared as an `env[]` secret and templated into the URL as
`{SECRET:tavilyApiKey}`.
- **SSH** — `mcp_local`/`user`; stores aliases in `~/.ssh_aliases.json` (auto-managed, 0600). Auth
per-alias: key/agent (default) or elicited password. Sudo always probes `sudo -n` first (free on a
NOPASSWD host, and side-effect free elsewhere since sudo refuses before running the command), then
falls back to `sudo -S` with a password from `SSH_MCP_SUDO_PASSWORD` or MCP elicitation; a leading
`sudo` inside `command` is stripped and turned into `sudo=true`, and under sudo the command runs as
`sh -c '…'` so pipes/redirections are privileged too. Elicitation needs a human in the Agent Inbox
within 300 s (`ELICITATION_DEADLINE` in `skald-core/src/elicitation/mod.rs`) — unattended runs must
rely on NOPASSWD or `SSH_MCP_SUDO_PASSWORD`. No setup-time credentials: `auth.type: "none"` with no
`verify`. All `SSH_MCP_*` env vars are optional with defaults.
## Current connectors
Order as in `connectors/index.json` (18 connectors).
| ID | Name | Type | Scope | Auth | Verify |
|----|------|------|-------|------|--------|
| `gmail` | Gmail | `mcp_local` | `user` | oauth2 (Google) + deliver `env/google_authorized_user` (`GMAIL_CREDS_JSON`) | ⏳ Phase 2 — OAuth via loopback listener |
| `gcal` | Google Calendar | `mcp_local` | `user` | oauth2 (Google) + deliver `env/…` (`GCAL_CREDS_JSON`) | `verify.py` (creds load + API probe) |
| `drive` | Google Drive | `mcp_local` | `user` | oauth2 (Google) + deliver `env/…` (`DRIVE_CREDS_JSON`) | `verify.py` (creds load + Drive API probe) |
| `email` | Email (IMAP/SMTP) | `mcp_local` | `user` | password (env) | `verify.py` (IMAP+SMTP probe) |
| `exa` | Exa | `mcp_remote` | `global` | api_key (`{SECRET:exaApiKey}` in URL — optional, free tier) | `verify.py` (MCP initialize probe) |
| `firecrawl` | Firecrawl | `mcp_local` | `global` | api_key (env) | inline `node -e` scrape probe |
| `http-fetch` | HTTP Fetch | `mcp_local` | `global` | none | — |
| `serpapi-flights` | SerpAPI Flights | `mcp_remote` | `global` | api_key (`{SECRET:serpapiApiKey}` in URL) | `verify.py` (MCP initialize probe) |
| `ssh` | SSH Remote Access | `mcp_local` | `user` | none (auth at runtime, per-alias) | — |
| `tavily` | Tavily | `mcp_remote` | `global` | api_key (`{SECRET:tavilyApiKey}` in URL) | `verify.py` (HTTP probe `/search`) |
| `weather` | Weather (Open-Meteo) | `mcp_local` | `global` | none | — |
| `whatsapp` | WhatsApp | `mcp_local` | `user` | qr | — |
| `wikipedia` | Wikipedia | `mcp_local` | `global` | none | — |
| `context7` | Context7 | `mcp_remote` | `global` | none | `verify.py` (MCP initialize probe) |
| `gmaps` | Google Maps | `mcp_local` | `global` | api_key (env: `GOOGLE_MAPS_API_KEY`) | `verify.py` (Geocoding API probe) |
| `google-trends` | Google Trends | `mcp_local` | `global` | none | `verify.py` (trendspyg import probe) |
| `linkedin` | LinkedIn | `mcp_local` | `user` | api_key (env: `LINKEDIN_LI_AT` session cookie) | `verify.py` |
| `playwright` | Playwright | `mcp_local` | `global` | none | `verify.js` (headless Chromium launch probe) |
A connector without `verify` is activated without any test — see § Without verify.
## Icon conventions
- **Format**: SVG for vector icons (better for retina/zoom), PNG for raster.
- **Name**: `icon_sm.{svg|png}` (small, ~48×48px), `icon_lg.{svg|png}` (large, ~96×96px).
- **Path**: relative to the connector folder in `connector.json`; `{folder}/{filename}` in the index
(e.g. `gmail/icon_sm.svg`).
## File integrity (sha256)
- SHA-256 hashes are generated **automatically** by `scripts/compile.py` from the physical files present
in each folder — never manual, never stale.
- `compile.py` excludes from `files[]`: `fragment.json`, `connectors.json`, `index.json`, `compile.sh`,
`compile.py`, `update_hashes.py`, `.DS_Store`, and the `scripts/`, `__pycache__/`, `.git/` directories.
Everything else in the folder is hashed — **including `connector.json`**.
- `fragment.json` and `index.json` have no hash: they are compilation inputs only.
- The only file signed (in the future) will be `connectors.json`, the index itself.
## Critical invariants
- **Never edit `connectors.json` by hand.** It is generated. Edit `fragment.json` / `index.json` /
the connector files, then run `python3 scripts/compile.py`.
- **Bump `version` (+1) and `version_string` on every file change** to a connector, in both
`fragment.json` and `connector.json`, and keep `version_release_date` current. When the two disagree
**the manifest wins** and Skald only logs a `marketplace feed version desync` warning — and if the
index carries the *lower* number, the strict `feed > installed` comparison can never fire again and
the connector silently stops offering updates. `compile.py` does not check this.
- **Keep `id`, `name`, `type`, `scope`, `tags`, `requires`, and `auth` consistent** between
`fragment.json` and `connector.json`. Only the manifest's `auth` is parsed by Skald; the index's is
documentary, with `requires` as the sole coarse fallback (`OAUTH` → oauth, `API_KEY` → api_key).
- **Keep the connector folder flat.** `compile.py` scans only the folder's top level, so any file in a
subdirectory is silently absent from `files[]` — never downloaded, never installed.
- **Adding a `verify` script means shipping the file in the connector folder** and recompiling — skald
refuses to run a script whose SHA-256 is not pinned in the index.
- **Always run `python3 scripts/compile.py` before deploying**; a stale hash breaks integrity
verification on the client.
- The `secrets/` directory is **deprecated** in the model (credentials flow through `{ENV:}`/`{SECRET:}`).
Gmail still uses it locally pending the OAuth migration; that path is gitignored — never commit
credentials or OAuth tokens.
## Local workflow
1. **Adding a new connector**:
- Create the folder `connectors/<id>/`
- Create `fragment.json` (id, name, type, scope, icons, auth, tools, …)
- Create `connector.json` (technical config: mcp_config, launch_command, env, verify, …)
- Add the MCP script, icons, `verify.py`, `requirements.txt`
- Add the id to `connectors/index.json`
- Run `python3 scripts/compile.py`
2. **Modifying an existing connector**:
- Edit the files in the connector folder, bump `version` / `version_string` / `version_release_date`
- **Do not touch** `connectors.json` — it gets regenerated
- Run `python3 scripts/compile.py`
3. **Before deploying**:
```bash
python3 scripts/compile.py # regenerates connectors.json with fresh SHA-256s
python3 scripts/compile.py --verify # (optional) checks the index is up to date
```
## Commands
Preview the catalog locally (must serve from the `connectors/` root so `/connectors.json` resolves):
```bash
cd connectors && python3 -m http.server 8000 # then open http://localhost:8000/
```
Recompute a single file hash by hand (normally unnecessary — `compile.py` does it):
```bash
python3 -c "import hashlib; print(hashlib.sha256(open('connectors/gmail/gmail_mcp_server.py','rb').read()).hexdigest())"
```
Set up and run the Gmail connector standalone:
```bash
pip install -r connectors/gmail/requirements.txt
python3 connectors/gmail/gmail_mcp_server.py # speaks JSON-RPC on stdin/stdout
```
## Deploy
The remote server:
- **Host**: skald-home-server (192.168.1.100 / 145.40.169.107)
- **User**: dguiducci
- **Path**: `/var/www/connectors.skaldagent.net/` (served by Caddy)
- **Owner**: `caddy:caddy` — sudo required to write in `/var/www/`
Deploy runs [deploy.sh](deploy.sh) on the server (`git pull` in
`/home/dguiducci/repos/skald-connectors/`, then `sudo cp -r connectors/* /var/www/connectors.skaldagent.net/`):
```bash
ssh dguiducci@skald-home-server /home/dguiducci/marketplace_deploy.sh
```
or via MCP SSH:
```bash
mcp__ssh__exec alias=skald-home-server command="/home/dguiducci/marketplace_deploy.sh"
```
Then verify on `https://connectors.skaldagent.net/`.
**Remember to run `python3 scripts/compile.py` and commit before deploying** — the server pulls from git.
## Changelog
Every user-facing change to the marketplace (new connector, connector fix, version bump, icon update,
deploy notes, etc.) must be recorded in **`CHANGELOG.md`** at the repo root. Rules:
- Follow the classic [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) format (version headings,
`Added`/`Changed`/`Fixed`/`Removed` categories, `[Unreleased]` section at the top). Use **date headings**
(`## 2026-08-10`) instead of semver, since each connector has its own version.
- Write in English.
- Add the entry in the same commit/change that modifies the connector files — do not defer it.
Reference in New Issue View Git Blame Copy Permalink