skald-connectors/SKALD.md

# Skald Connectors Marketplace

## What it is

The **marketplace** is the catalog of tested connectors for Skald. Each connector is an adapter that lets Skald interface with an external service (API, email, calendar, search, messaging, etc.).

Two types of connectors:

- **`mcp_remote`** — an already-hosted MCP server, reachable via URL (e.g. Tavily).
- **`mcp_local`** — a Python/Node script to run client-side (e.g. Gmail, Google Calendar).

## Local references

- **[docs/connector.manifest_guide.md](docs/connector.manifest_guide.md)** — Official guide for producing correct connectors (copied from `skald-circle/blueprint/`)

**Remote**: `https://git.skaldagent.net/dguiducci/skald-connectors.git`
**Live**: `https://connectors.skaldagent.net/`
**OAuth callback**: `https://connectors.skaldagent.net/oauth/show.html`

## Deploy

The **`main`** branch is the **release** branch. Only production-ready code ends up here.
Development and alpha versions will live on separate branches in the future.

Quick deploy with MCP SSH:

```bash
mcp__ssh__exec alias=skald-home-server command="/home/dguiducci/marketplace_deploy.sh"
```

Or via classic SSH:

```bash
ssh dguiducci@skald-home-server /home/dguiducci/marketplace_deploy.sh
```

The script on the server does:
1. `git pull` in `/home/dguiducci/repos/skald-connectors/`
2. `cp -r connectors/*` into `/var/www/connectors.skaldagent.net/`

The `/var/www/connectors.skaldagent.net/` directory is owned by `dguiducci`,
so no sudo is needed for the copy.

**Before deploying**, remember to regenerate the index:
```bash
python3 scripts/compile.py
```

## Friendly tool names (2026-07-21)

Every MCP tool must expose a friendly name for the Skald UI. Two ways, in order of preference:

1. **Via the MCP script (preferred)** — add `"title": "Friendly Name"` in the definition of each tool inside `tools/list`. Works for all local scripts (Python/Node) that we control.
2. **Via the manifest (fallback)** — add `"tools": [{"name": "...", "display_name": "..."}]` in `connector.json` **and** in `connectors.json`. Used only for remote connectors or external packages (e.g. `npx -y firecrawl-mcp`).

**Resolution order** (Skald tries them in this order):
1. `tools[].display_name` from the manifest
2. `title` from the MCP server's `tools/list`
3. Automatic prettify of the raw name (`send_message` → "Send Message")

Current status (2026-07-21): all 13 marketplace connectors have `title` in the script or `tools[]` in the manifest.

## Directory structure

```
connectors/
├── index.json              ← ordered list of connector ids (input for compile.py)
├── connectors.json         ← COMPILED INDEX (generated by compile.py, do not edit)
├── compile.py              ← generates connectors.json (launched by scripts/compile.py)
├── index.html              ← Catalog UI (reads connectors.json via fetch)
├── oauth/
│   └── show.html           ← OAuth callback receiver
├── gmail/                   ← one connector per folder
│   ├── fragment.json        ← index fragment (id, name, type, ..., WITHOUT files[])
│   ├── connector.json       ← technical configuration (mcp_config, auth.deliver, ...)
│   ├── gmail_mcp_server.py  ← MCP script
│   ├── gmail_oauth_setup.py ← OAuth setup script
│   ├── requirements.txt     ← Python dependencies
│   ├── icon_sm.svg          ← small icon (48×48)
│   └── icon_lg.svg          ← large icon (e.g. 96×96)
├── email/
│   ├── fragment.json
│   ├── connector.json
│   ├── email_mcp_server.py
│   ├── verify.py
│   ├── requirements.txt
│   ├── icon_sm.svg
│   └── icon_lg.svg
└── ...
```

## Schema — connectors.json (root)

This is the **single root of trust**. It contains `type`, `scope`, and the sha256 of each connector's files.
It has no hash of itself — in the future it may be digitally signed.

```json
{
  "version": 1,
  "connectors": [
    {
      "id": "gmail",
      "name": "Gmail",
      "type": "mcp_local",
      "scope": "user",
      "icon_small": "gmail/icon_sm.svg",
      "icon_large": "gmail/icon_lg.svg",
      "user_description": "Read, send, and manage Gmail emails via OAuth...",
      "requires": ["OAUTH", "PYTHON"],
      "tags": ["email", "mcp", "local", "google"],
      "auth": {
        "type": "oauth2",
        "provider": "google",
        "scopes": [
          "https://www.googleapis.com/auth/gmail.modify",
          "https://www.googleapis.com/auth/gmail.labels"
        ]
      },
      "folder": "gmail",
      "version": 1,
      "version_string": "1.0.0",
      "version_release_date": "2026-07-19",
      "files": [
        {"path": "gmail_mcp_server.py",  "sha256": "a50d4da9621f7a4b092f...", "size": 46772},
        {"path": "gmail_oauth_setup.py", "sha256": "e488acb289c43a3e6d54...", "size": 3627},
        {"path": "icon_lg.svg",          "sha256": "93c8d9c8dae96f0206e5...", "size": 254},
        {"path": "icon_sm.svg",          "sha256": "029d7f5d81de6cf2b17b...", "size": 251},
        {"path": "requirements.txt",     "sha256": "3f659cc5e5f0543f1326...", "size": 82}
      ]
    }
  ]
}
```

### Index fields

| Field | Required | Description |
|-------|----------|-------------|
| `id` | ✅ | Unique identifier (kebab-case) |
| `name` | ✅ | Displayed name |
| `type` | ✅ | `mcp_remote` or `mcp_local` |
| `scope` | ✅ | `global` or `user` |
| `icon_small` | ✅ | Path relative to the marketplace root |
| `icon_large` | ✅ | Path relative to the marketplace root |
| `user_description` | ✅ | Short description for the UI |
| `requires` | ✅ | Array of requirement enums |
| `tags` | ✅ | Array of tags for filtering |
| `folder` | ✅ | Name of the connector folder |
| `version` | ✅ | Per-connector integer, +1 on every file change |
| `version_string` | ✅ | Semver (display only) |
| `version_release_date` | ✅ | ISO 8601 date YYYY-MM-DD (display only) |
| `files` | ✅ | Array of files with sha256 (NO self-hash) |

## Schema — connector.json (per folder)

Technical configuration for connector activation.

```json
{
  "id": "gmail",
  "name": "Gmail",
  "version": 1,
  "version_string": "1.0.0",
  "version_release_date": "2026-07-19",
  "type": "mcp_local",
  "scope": "user",
  "launch_command": "python3 gmail_mcp_server.py",
  "transport": "stdio",
  "requires": ["OAUTH", "PYTHON"],
  "tags": ["email", "mcp", "local", "google"],
  "dependencies": [
    "google-api-python-client>=2.150.0",
    "google-auth>=2.35.0",
    "google-auth-oauthlib>=1.2.0"
  ],
  "setup_instructions": [
    "Install dependencies: pip install -r requirements.txt",
    "Run: python3 gmail_oauth_setup.py (optional, for standalone use — Skald handles OAuth)"
  ],
  "docs": [
    {
      "lang": "en",
      "description": "Full description for human users...",
      "llm_short_description": "Google Calendar — read, create, update, delete, and respond to calendar events. Supports push notifications."
    }
  ],
  "auth": {
    "type": "oauth2",
    "provider": "google",
    "scopes": [
      "https://www.googleapis.com/auth/gmail.modify",
      "https://www.googleapis.com/auth/gmail.labels"
    ],
    "deliver": {
      "as": "env",
      "format": "google_authorized_user",
      "env": "GMAIL_CREDS_JSON"
    }
  },
  "mcp_config": {
    "command": "python3",
    "args": ["gmail_mcp_server.py"]
  },
  "homepage": "https://mail.google.com",
  "icon_small": "icon_sm.svg",
  "icon_large": "icon_lg.svg"
}
```

### connector.json fields

| Field | Required | Description |
|-------|----------|-------------|
| `id` | ✅ | Unique identifier (matches the folder name) |
| `name` | ✅ | Displayed name |
| `version` | ✅ | Per-connector integer, +1 on every file change |
| `version_string` | ✅ | Semver (display only) |
| `version_release_date` | ✅ | ISO 8601 date YYYY-MM-DD (display only) |
| `type` | ✅ | `mcp_remote` or `mcp_local` |
| `scope` | ✅ | `global` or `user` |
| `requires` | ✅ | Array of requirement enums |
| `tags` | ✅ | Array of tags |
| `auth` | ✅ | Authentication configuration object |
| `docs` | ✅ | Array of multilingual documentation. **`llm_short_description`** is the field that ends up in the LLM's system prompt — it must describe WHAT the connector DOES, not list its tools (the LLM sees them after `activate_tools`). Example: *"Weather — current conditions, 16-day forecast, and AQI data for any location."* |
| `icon_small` | ✅ | Icon filename in the local folder |
| `icon_large` | ✅ | Icon filename in the local folder |
| `launch_command` | `mcp_local` only | Command to start the MCP server |
| `transport` | `mcp_local` only | `stdio` (default) |
| `dependencies` | recommended | Python/Node dependencies (empty array if stdlib only) |
| `env` | if `requires` includes `ENV` | Environment variables the user must provide (schema for the UI) — see § The env field |
| `setup_instructions` | recommended | Steps to configure the connector |
| `mcp_config` | `mcp_local` only | Configuration for the MCP client |
| `homepage` | optional | Service URL |

## Reserved enums

### type (connector type)

| Value | Description | Examples |
|-------|-------------|----------|
| `mcp_remote` | Hosted MCP server, reachable via URL | Tavily, Weather |
| `mcp_local` | Script to run locally | Gmail, Google Calendar, WhatsApp |
| `script` | Standalone script (non-MCP) | *(future)* |

### scope (configuration scope)

| Value | Description | Examples |
|-------|-------------|----------|
| `global` | A single instance/config for the whole system | Tavily, Weather, Google Trends |
| `user` | Each user has their own instance/authentication | Gmail, WhatsApp, Google Calendar |

### requires (prerequisites)

| Value | Description |
|-------|-------------|
| `API_KEY` | Requires an API key to configure |
| `OAUTH` | Requires OAuth authentication (Google, etc.) |
| `DOCKER` | Requires Docker Engine |
| `NODE` | Requires Node.js runtime |
| `PYTHON` | Requires Python 3 |
| `SECRETS_DIR` | ❌ **Deprecated** — the `secrets/` folder is removed from the model; connectors must use `ENV`/`SECRET` (see § Placeholder syntax) |
| `ENV` | Requires environment variables (declared in the manifest `env` field) |

## The auth field

Structure describing how the connector handles authentication:

```json
// API key in query string
{"type": "api_key", "delivery": "query",  "param": "tavilyApiKey"}

// API key in header
{"type": "api_key", "delivery": "header", "param": "X-API-Key"}

// OAuth2 — provider is ONLY a slug (Skald resolves endpoints + client secrets)
{"type": "oauth2", "provider": "google", "scopes": ["...", "..."]}

// OAuth2 with deliver (Skald injects the authorized_user JSON via env var)
{"type": "oauth2", "provider": "google", "scopes": ["..."],
 "deliver": {"as": "env", "format": "google_authorized_user", "env": "GMAIL_CREDS_JSON"}}

// OAuth2 with file-based deliver (legacy)
{"type": "oauth2", "provider": "google", "scopes": ["..."],
 "deliver": {"as": "file", "format": "google_authorized_user", "path": "{secrets}/gmail_creds.json"}}

// Password / app-password provided via environment variables
{"type": "password", "delivery": "env"}

// No authentication
{"type": "none"}
```

### The deliver field (OAuth2 only)

Declares **how** Skald delivers the obtained OAuth credential to the MCP server process.

| Field | Required | Description |
|-------|----------|-------------|
| `as` | ✅ | `"file"` (on disk) or `"env"` (environment variable) |
| `format` | ✅ | Name of the serialization — e.g. `"google_authorized_user"` (Google JSON that `from_authorized_user_file` reads), `"refresh_token"`, `"access_token"` |
| `path` | `as=file` only | Path with the `{secrets}` placeholder (Skald expands it to a per-user dir at runtime). MUST match the path in `mcp_config.env`. |
| `env` | `as=env` only | Name of the environment variable into which Skald injects the entire authorized_user JSON. **Must not be declared in `mcp_config.env`** — Skald injects it at runtime. |

The feed NEVER contains: `client_id`, `client_secret`, `endpoint` URL, `redirect_uri`. These are resolved on the Skald side from the `provider` name.

## The env field (environment variables)

Used when `requires` includes `ENV`. It is an **array** declaring the environment
variables the user must provide to make the connector work; **no credential
lands on disk nor in `secrets/`** — the host collects the values, enforces
filling in the required fields, and injects them as environment into the MCP
server process at launch. The server reads them from `os.environ`.

```json
"env": [
  {
    "name": "EMAIL_IMAP_HOST",   // name of the environment variable
    "label": "IMAP host",         // label for the UI
    "description": "IMAP server hostname (e.g. imap.gmail.com)",
    "required": true,             // if true, the host must enforce this field
    "secret": false,              // if true, the UI masks it and treats it as a secret
    "example": "imap.gmail.com"   // placeholder/example (optional)
  },
  {
    "name": "EMAIL_PASSWORD",
    "label": "Password / app password",
    "description": "Password or app-password of the provider",
    "required": true,
    "secret": true,
    "default": ""                 // default value if not required (optional)
  }
]
```

| Field | Required | Description |
|-------|----------|-------------|
| `name` | ✅ | Name of the environment variable (UPPER_SNAKE_CASE) |
| `label` | ✅ | Short label for the UI |
| `description` | ✅ | Help text |
| `required` | ✅ | If `true`, the host forces the user to provide a value |
| `secret` | recommended | If `true`, sensitive value (masked, not logged) |
| `default` | optional | Value used when not provided (non-required fields only) |
| `example` | optional | Example placeholder for the UI |

## Placeholder syntax (unified)

Every value skald must fill at runtime with user-provided data uses **one of
two tokens**, wherever it appears (URL, `mcp_config.env`, `verify.command`):

| Token | Meaning | Example |
|-------|---------|---------|
| `{ENV:NAME}` | Non-sensitive variable (hostname, port, username…) | `{ENV:EMAIL_IMAP_HOST}` |
| `{SECRET:NAME}` | Sensitive variable (password, API key, token) | `{SECRET:EMAIL_PASSWORD}` |

`NAME` is the name declared in the `env[]` array (the `name` field). skald
collects the values via a form (masking `{SECRET:}` fields), injects them as
environment into the MCP server / verify process, and substitutes the tokens
in the manifest.

Rules:
- Unrecognized tokens (e.g. `{secrets}/…`, legacy `{key}`, `{env:NAME}`) are
  **deprecated**: skald does not substitute them and the manifest must be updated.
- `{SECRET:<auth.param>}` is reserved for the primary key when
  `auth.type = "api_key"` (e.g. Tavily: `{SECRET:tavilyApiKey}`). skald also
  treats that value as the API key for bearer/header routing.
- A `{ENV:X}` or `{SECRET:X}` token whose `X` is not in the manifest's `env[]`
  is substituted with an empty string (the host cannot guess it).

### Deprecations

| Token | Status | Replacement |
|-------|--------|-------------|
| `{key}` | ❌ deprecated | `{SECRET:<auth.param>}` |
| `{env:NAME}` | ❌ deprecated | `{ENV:NAME}` |
| `{secrets}/…` | ❌ deprecated | The connector must declare the path as `{ENV:…}` (the `secrets/` folder is removed from the model) |

## The verify field (test before save)

Declares a shell command that skald runs **after** the user has filled in the
form and **before** persisting the activation. It checks that the credentials
just entered actually work.

```json
"verify": {
  "command": "python3 verify.py",
  "timeout_secs": 20
}
```

| Field | Required | Description |
|-------|----------|-------------|
| `command` | ✅ | Shell command. Runs in the same sandbox as the server: container `skald-{userid}` for `mcp_local` user, host for `mcp_remote` global. The declared env/secrets are injected |
| `timeout_secs` | optional | Default 15. skald kills the process at expiry |

### Output convention

The command must print **a single JSON object on stdout** and nothing else:

```json
{"ok": true,  "message": "IMAP and SMTP authentication successful", "details": {"imap": "...", "smtp": "..."}}
{"ok": false, "message": "IMAP login failed: INVALID_CREDENTIALS"}
```

| Field | Type | Description |
|-------|------|-------------|
| `ok` | bool | `true` = test passed |
| `message` | string | Message shown to the user (never log secrets inside) |
| `details` | object | Optional, structured details shown in `<pre>` |

Exit code: 0 on success, ≠ 0 on failure (skald uses the exit code as a
fallback if the JSON parse fails). **Never print credentials** in the
`message`/`details`.

### Where to put the script

If `command` references a file (e.g. `verify.py`), the file must:
1. Be added to the `files[]` array in `connectors.json` (with `sha256` and `size`)
2. Be saved in the connector folder (`<id>/verify.py`)

skald downloads it, verifies its SHA-256 against the index, and makes it
available at the same path as the main server (container for `mcp_local` ones,
`./scripts/<id>/` dir on the host for `mcp_remote` ones).

### Without verify

If `verify` is absent, skald **runs no test** — activation is direct, as
today. The connector goes to `auth_state='ready'` without verification. For
`mcp_remote` there is no handshake fallback: the manifest author decides
whether the test is needed by writing `verify`.

## Icon conventions

- **Format**: SVG for vector icons (better for retina/zoom), PNG for raster
- **Name**: `icon_sm.{svg|png}` (small, ~48×48px), `icon_lg.{svg|png}` (large, ~96×96px)
- **Path**: relative to the connector folder
- **In the index** the path is `{folder}/{filename}` (e.g. `gmail/icon_sm.svg`)

## File integrity (sha256)

- SHA-256 hashes are generated **automatically** by `scripts/compile.py` from
  the physical files present in each folder — never manual, never stale.
- The only file signed (in the future) will be `connectors.json` (the index).
- `fragment.json` and `index.json` have no hash — they are just compilation inputs.

## Local workflow

1. **Adding a new connector**:
   - Create the folder `connectors/<id>/`
   - Create `fragment.json` (id, name, type, scope, icons, auth, tools, ...)
   - Create `connector.json` (technical config: mcp_config, launch_command, ...)
   - Add the MCP script, icons, verify.py, requirements.txt
   - Add the id to `connectors/index.json`
   - Run `python3 scripts/compile.py`

2. **Modifying an existing connector**:
   - Edit the files in the connector folder
   - **Do not touch** `connectors.json` — it gets regenerated
   - Run `python3 scripts/compile.py`

3. **Before deploying**:
   ```bash
   python3 scripts/compile.py           # regenerates connectors.json with fresh SHA-256s
   python3 scripts/compile.py --verify  # (optional) verifies that it is up to date
   ```

4. **Deploy to the server**:
   ```bash
   ssh dguiducci@skald-home-server /home/dguiducci/marketplace_deploy.sh
   ```
   or with MCP SSH:
   ```bash
   # via mcp__ssh__exec alias "skald-home-server"
   /home/dguiducci/marketplace_deploy.sh
   ```
   (The deploy script does `git pull` + `cp -r connectors/* /var/www/connectors.skaldagent.net/`)

5. Verify on `https://connectors.skaldagent.net/`

## Fragment.json

See [docs/connector.manifest_guide.md](docs/connector.manifest_guide.md) for the complete
creation guide.

`fragment.json` contains all the fields of a `connectors.json` entry **except** `files[]`.
These are the required fields:

| Field | Required | Description |
|-------|----------|-------------|
| `id` | ✅ | Unique identifier (matches the folder name) |
| `name` | ✅ | Displayed name |
| `type` | ✅ | `mcp_remote` or `mcp_local` |
| `scope` | ✅ | `global` or `user` |
| `icon_small` | ✅ | Path relative to the marketplace root |
| `icon_large` | ✅ | Path relative to the marketplace root |
| `user_description` | ✅ | Short description for the UI |
| `requires` | ✅ | Array of requirement enums |
| `tags` | ✅ | Array of tags for filtering |
| `folder` | ✅ | Name of the connector folder (matches `id`) |
| `version` | ✅ | Per-connector integer, +1 on every file change |
| `version_string` | ✅ | Semver (display only) |
| `version_release_date` | ✅ | ISO 8601 date (display only) |
| `tools` | ✅ | Array of `{name, display_name}` for friendly UI names |
| `auth` | optional | Authentication configuration (if other than `"none"`) |

Note: the `files[]` array (with SHA-256 and size) is added **automatically**
by `compile.py` by scanning the files present in the folder — it must never be
written by hand.

## Deploy to remote server

The remote server is:

- **Host**: skald-home-server (192.168.1.100 / 145.40.169.107)
- **User**: dguiducci
- **Path**: `/var/www/connectors.skaldagent.net/`
- **Owner**: `caddy:caddy`
- **Sudo**: required to write in `/var/www/`

## Current connectors

| ID | Name | Type | Scope | Auth | Verify |
|----|------|------|-------|------|--------|
| `context7` | Context7 | `mcp_remote` | `global` | none | `verify.py` (MCP initialize probe) |
| `gmaps` | Google Maps | `mcp_local` | `global` | api_key (env: `GOOGLE_MAPS_API_KEY`) | `verify.py` (Geocoding API probe) |
| `exa` | Exa | `mcp_remote` | `global` | api_key (`{SECRET:exaApiKey}` in URL — optional, free tier) | `verify.py` (MCP initialize probe) |
| `tavily` | Tavily | `mcp_remote` | `global` | api_key (`{SECRET:tavilyApiKey}` in URL) | `verify.py` (HTTP probe `/search`) |
| `serpapi-flights` | SerpAPI Flights | `mcp_remote` | `global` | api_key (`{SECRET:serpapiApiKey}` in URL) | `verify.py` (MCP initialize probe) |
| `gmail` | Gmail | `mcp_local` | `user` | oauth2 (Google) + deliver: `env/google_authorized_user` (via `GMAIL_CREDS_JSON`) | ⏳ Phase 2 — OAuth via loopback listener |
| `gcal` | Google Calendar | `mcp_local` | `user` | oauth2 (Google) + deliver: `env/google_authorized_user` (via `GCAL_CREDS_JSON`) | `verify.py` (creds load + API probe) |
| `drive` | Google Drive | `mcp_local` | `user` | oauth2 (Google) + deliver: `env/google_authorized_user` (via `DRIVE_CREDS_JSON`) | `verify.py` (creds load + Drive API probe) |
| `email` | Email (IMAP/SMTP) | `mcp_local` | `user` | password (env) | `verify.py` (IMAP+SMTP probe) |
| `firecrawl` | Firecrawl | `mcp_local` | `global` | api_key (env) | — |
| `http-fetch` | HTTP Fetch | `mcp_local` | `global` | none | — |
| `ssh` | SSH Remote Access | `mcp_local` | `user` | none (auth runtime per-alias) | — |
| `weather` | Weather (Open-Meteo) | `mcp_local` | `global` | none | — |
| `google-trends` | Google Trends | `mcp_local` | `global` | none | `verify.py` (trendspyg import probe) |
| `wikipedia` | Wikipedia | `mcp_local` | `global` | none | — |
| `whatsapp` | WhatsApp | `mcp_local` | `user` | qr | — |
| `playwright` | Playwright | `mcp_local` | `global` | none | `verify.js` (headless Chromium launch probe) |

**verify-before-save status in skald**: `exa`, `drive`, `email`, `tavily`, and `gcal` have a complete `verify`
(script + JSON output); `gmail` awaits Phase 2 (OAuth via loopback listener).
A connector without `verify` is activated without any test — see § Without verify.