skald-connectors/CLAUDE.md

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## What this is

The **Skald Connectors Marketplace** — a catalog of tested connectors (adapters that let Skald agents talk to external services). It is **not an application**: there is no build system, package manager, or test suite. The repo is a set of JSON manifests, static HTML, icons, and standalone Python MCP scripts that are served as-is over HTTP.

`SKALD.md` is the authoritative spec — read it before changing any schema or manifest.

## Architecture

Two-tier manifest model with a single trust root:

- **`connectors/connectors.json`** — the **index** and single root of trust. Lists every connector and, per connector, a `files[]` array with the `sha256` + `size` of each shipped file. It has **no hash of itself**, and the `files[]` arrays **do not include `connector.json`** — both are treated as unsigned manifests. Intended to be digitally signed in the future.
- **`connectors/<id>/connector.json`** — per-connector **technical manifest** for activation (launch command, transport, `mcp_config`, `auth`, `dependencies`, `docs`). One folder per connector; folder name matches the connector `id`.
- **`connectors/index.html`** — catalog UI. Fetches `/connectors.json` at absolute path and links to `/<folder>/`, so it only works when served from the site root (not opened as a `file://`).

Connector taxonomy (see SKALD.md for full enums):
- `type`: `mcp_remote` (hosted MCP server reached by URL, e.g. Tavily) vs `mcp_local` (script run client-side, e.g. Gmail).
- `scope`: `global` (one shared config) vs `user` (per-user auth/instance).
- `requires`: prerequisites like `API_KEY`, `OAUTH`, `PYTHON`, `NODE`, `DOCKER`, `ENV`. (`SECRETS_DIR` is **deprecated** — the `secrets/` folder is gone from the model; connectors must declare their inputs as `ENV`/`SECRET`.)

**Unified placeholder syntax.** Every value skald must fill at runtime uses one of two tokens, anywhere it appears (URL, `mcp_config.env`, `verify.command`):
- `{ENV:NAME}` — a non-sensitive value (host, port, username…), declared in the top-level `env[]` array.
- `{SECRET:NAME}` — a sensitive value (password, API key, token), also declared in `env[]` with `secret: true`.

Legacy tokens `{key}`, `{env:NAME}`, `{secrets}/…` are **deprecated** and skald no longer substitutes them. The api-key of an `auth.type = "api_key"` connector is exposed as `{SECRET:<auth.param>}` (e.g. Tavily: `?tavilyApiKey={SECRET:tavilyApiKey}`). See SKALD.md § Placeholder syntax.

**Config via `env[]` (`env` manifest field):** a connector whose `requires` includes `ENV` declares its required environment variables in a top-level `env` array in `connector.json` — each entry has `name`, `label`, `description`, `required`, `secret`, and optional `default`/`example`. skald renders a real form from this schema (masking `secret: true` fields, marking `required`, using `example` as placeholder) and injects the collected values into the process environment at launch; the server reads `os.environ`. Nothing is written to disk. The `email` connector is the reference example (see SKALD.md § The env field). **`tavily` now also declares `env[]`** (its API key as a `secret` field).

**Verify-before-save (`verify` manifest field).** A connector may declare a `verify` step — a shell command skald runs *after* the user fills the form and *before* persisting the activation, to confirm the credentials actually work:
```json
"verify": { "command": "python3 verify.py", "timeout_secs": 20 }
```
The command runs in the same sandbox as the server (container `skald-{userid}` for `mcp_local` user, host for `mcp_remote` global), with the collected env/secret injected. It must print one JSON object on stdout — `{"ok": bool, "message": string, "details"?: object}` — and exit 0 on success. If `verify` is absent, no test runs and the activation is direct. See SKALD.md § The verify field. `email` and `tavily` ship a `verify.py`; `gmail` will get one in the OAuth phase.

**MCP servers** (`connectors/gmail/gmail_mcp_server.py`, `connectors/email/email_mcp_server.py`) are hand-rolled JSON-RPC 2.0 servers over stdio — no MCP SDK / FastMCP. Shared conventions: **stdout is reserved for JSON-RPC**, all logging goes to stderr, a lock guards stdout writes, a background thread emits `event/new_email` push notifications, and each server exposes the same tool shape (`TOOLS` manifest + `TOOL_DISPATCH` + `handle_request`). Mirror this structure for new local connectors.
- **Gmail** uses the Gmail API with OAuth; today it still reads its token from `./secrets/gmail_creds.json` (override `GMAIL_CREDS_PATH`) generated by `gmail_oauth_setup.py`. The `secrets/` path is **deprecated** — the OAuth phase (skald Fase 2) will migrate Gmail to `{ENV:}`/`{SECRET:}` and a loopback-listener flow. Push = History API polling. `verify` is not yet wired for Gmail.
- **Email** is a generic IMAP+SMTP connector, **stdlib-only (no dependencies)**, configured entirely from `{ENV:}`/`{SECRET:}` env vars (no `secrets/`), works with any provider. Push = IMAP IDLE with a 60s polling fallback; the request thread and the watcher thread each hold their own IMAP connection (imaplib is not thread-safe). Note: imaplib does not quote SEARCH arguments, so values with spaces must be wrapped via `_q()`. Ships a `verify.py` (IMAP+SMTP login probe).
- **Tavily** is `mcp_remote`/`global`; its API key is declared as an `env[]` secret (`tavilyApiKey`) and templated into the URL as `{SECRET:tavilyApiKey}`. Ships a `verify.py` (minimal `/search` POST probe).
- **SSH** is `mcp_local`/`user`; stores aliases in `~/.ssh_aliases.json` (auto-managed, 0600). Auth per-alias: key/agent (default) or elicited password. Sudo via `nopasswd` or elicited password (`sudo -S`). No setup-time credentials: `auth.type: "none"` with no `verify` (activation is direct). All env vars (`SSH_MCP_*`) are optional with defaults. Ships `requirements.txt` (paramiko).

## Critical invariants

- **Editing any shipped connector file (script, icon, requirements, `verify.py`) requires updating its `sha256` AND `size` in `connectors.json`.** A stale hash breaks integrity verification on the client. `connector.json` and `connectors.json` themselves are not hashed, so editing them needs no hash update.
- **Adding a connector** means changes in two places: a new entry in `connectors.json` (with all file hashes) *and* a new `connectors/<id>/connector.json`. Keep `id`, `scope`, `type`, `tags`, and `requires` consistent between the two.
- **A new `verify.py` must be added to `files[]`** (with `sha256`+`size`) exactly like any other shipped file — skald refuses to run an unverified script.
- The `secrets/` directory is **deprecated** in the model (credentials flow through `{ENV:}`/`{SECRET:}`). Gmail still uses it locally pending the OAuth migration; that path is gitignored — never commit credentials or OAuth tokens.

## Changelog

Every user-facing change to the marketplace (new connector, connector fix, version bump, icon update, deploy notes, etc.) must be recorded in **`CHANGELOG.md`** at the repo root. Rules:

- Follow the classic [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) format (version headings, `Added`/`Changed`/`Fixed`/`Removed` categories, `[Unreleased]` section at the top). Use **date headings** (`## 2026-08-10`) instead of semver since each connector has its own version.
- Write in English.
- Add the entry in the same commit/change that modifies the connector files — do not defer it.
- Never put changelog entries in `SKALD.md`; it is spec-only.

## Commands

Recompute a file hash (do this before deploy for every changed file; SKALD.md mentions a `scripts/update_hashes.py` helper that does not exist in the repo yet):
```bash
python3 -c "import hashlib; print(hashlib.sha256(open('connectors/gmail/gmail_mcp_server.py','rb').read()).hexdigest())"
```

Preview the catalog locally (must serve from the `connectors/` root so `/connectors.json` resolves):
```bash
cd connectors && python3 -m http.server 8000   # then open http://localhost:8000/
```

Set up and run the Gmail connector:
```bash
pip install -r connectors/gmail/requirements.txt
python3 connectors/gmail/gmail_oauth_setup.py    # opens browser for OAuth, writes token
python3 connectors/gmail/gmail_mcp_server.py      # speaks JSON-RPC on stdin/stdout
```

Deploy to the production server (Caddy, path `/var/www/connectors.skaldagent.net/`):
```bash
rsync -avz --delete ./connectors/ dguiducci@skald-server:/var/www/connectors.skaldagent.net/
# then fix ownership/permissions on the server:
sudo chown -R caddy:caddy /var/www/connectors.skaldagent.net/
sudo find /var/www/connectors.skaldagent.net/ -type f -exec chmod 644 {} \;
```
Remote git: `https://git.skaldagent.net/dguiducci/skald-connectors.git` (branch `main`). Live site: `https://connectors.skaldagent.net/`.