http-fetch/firecrawl: replace npx launch with a shipped wrapper (v5 / 1.1.0)

Both connectors declared `mcp_config: {command: "npx", args: ["-y", "<pkg>"]}` and shipped no code at all. Skald treats a mcp_local `args[0]` as the *name of the file to run*, not as an argument: install stores `script_path = "<id>/-y"` and global_enable launches `npx /…/connectors/<id>/-y`, a path that does not exist. The server never answered `initialize`, so start_server failed — and since global_enable still returns 200 with an `error` in the body, the UI showed the connector as enabled while `render_mcp_list` (built from the live runtime) left it out of the agent's system context entirely. Ship a two-file wrapper instead: a package.json pinning the upstream package (mcp-fetch-server@1.1.2, firecrawl-mcp@3.23.7) and an index.js importing it for its side effect, so `node index.js` is a real local_script and ensure_installed_host installs the deps beside it. No node_modules, no lockfile. Also: drop the legacy `launch_command` / top-level `transport` / `dependencies` fields; drop firecrawl's inert `mcp_config.env` placeholder (same case as gmaps — tokens are only substituted in the URL); add firecrawl_developer_search to firecrawl's tools[] and API_KEY to its requires; realign manifest and fragment versions to 5 / 1.1.0 (they were 2 vs 4, so no Update badge could ever appear). Verified end-to-end against the shipped files by reproducing Skald's own path: `npm ci || npm install --omit=dev` then `node <abs>/index.js` — initialize, tools/list and a real tools/call, JSON-RPC only on stdout. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-10 14:14:36 +01:00
parent 73d1a51b1f
commit ecb6835fa5
10 changed files with 147 additions and 50 deletions
@@ -1,6 +1,21 @@
 # Skald Connectors Marketplace


+### 2026-08-10 — http-fetch + firecrawl: fix `npx` (v5 / 1.1.0)
+
+Entrambi i connector erano **non funzionanti**: dichiaravano `mcp_config: {command: "npx", args: ["-y", "<package>"]}` e non spedivano nessun file di codice (solo `connector.json` + icone).
+
+- **`npx -y <package>` non è esprimibile in skald.** Per un `type: mcp_local` skald tratta `args[0]` come il **nome del file da eseguire**, non come un argomento: all'install calcola `script_path = "<id>/" + args[0]` e azzera `args_json` (`marketplace.rs::install`), poi `global_enable` (`api/mcp.rs`) lo risolve in path assoluto e lancia `<command> <abs>`. Il comando reale diventava `npx /…/connectors/http-fetch/-y` — un path inesistente, con `-y` e il nome del package spariti. Il processo non rispondeva mai a `initialize`, `start_server` falliva.
+- **Il fallimento era silenzioso**: `global_enable` restituisce comunque HTTP 200 con un campo `error` nel body, quindi la UI mostrava il connector come abilitato mentre il runtime non aveva nessun server. E `render_mcp_list` (`loop_adapters/system.rs`) costruisce la tabella `## MCP servers` da `mcp.tools()`, cioè lo **stato vivo del runtime**, non dal DB → il connector risultava attivato e concesso all'utente ma **assente dal system context**. ⚠️ Questa combinazione (200 + `error` nel body) rende invisibile in UI qualsiasi connector che non parte: vale la pena farla emergere lato skald.
+- **Fix — wrapper di due file** per entrambi: `package.json` che pinna il package upstream (`mcp-fetch-server@1.1.2`, `firecrawl-mcp@3.23.7`) e `index.js` che lo importa per side-effect (il modulo avvia il loop JSON-RPC su stdio all'import). `mcp_config` diventa `{command: "node", args: ["index.js"], transport: "stdio"}`, cioè un vero `local_script`: `ensure_installed_host` fa `npm ci --omit=dev || npm install --omit=dev` nella cartella del connector prima del lancio, esattamente come per whatsapp. Nessun `node_modules` spedito, nessun lockfile (come whatsapp).
+- **Rimossi i campi legacy** `launch_command`, `transport` top-level e `dependencies` (`dependencies` è solo per la card, come già visto su gmaps; `transport` va dentro `mcp_config`).
+- **firecrawl: rimosso `mcp_config.env: {"FIRECRAWL_API_KEY": "{SECRET:FIRECRAWL_API_KEY}"}`** — inerte, stesso caso di gmaps del 2026-08-10: `apply_key_placeholder` sostituisce i token solo nella URL, mai nei valori di `env`. Funziona perché la form admin manda `env` e quel payload sovrascrive `entry.env_json`.
+- **firecrawl**: aggiunto `firecrawl_developer_search` a `tools[]` (27 tool live contro i 26 dichiarati, verificato su 3.23.7); `requires` `["NODE"]` → `["NODE", "API_KEY"]`.
+- **Versioni riallineate** manifest↔fragment a `5` / `1.1.0` / `2026-08-10` per entrambi: erano 2/1.0.1 (manifest) vs 4/1.0.3 (fragment), e skald preferisce il manifest — quindi `installed_version` restava 2 e il badge "Update available" non sarebbe mai comparso.
+- **Requisito host**: sono connector `scope: global`, girano sull'**host** e non nel container. `mcp-fetch-server` vuole Node ≥18, `firecrawl-mcp` vuole Node ≥**22**.
+- Testati end-to-end riproducendo il path di skald (`npm ci || npm install` + `node <abs>/index.js`): `initialize`, `tools/list` e una `tools/call` reale, stdout solo JSON-RPC, stderr pulito ✅
+- Indice rigenerato con compile.py ✅
+
 ### 2026-08-10 — gmaps: fix dipendenze + verify (v6 / 1.1.0)
 - **Aggiunto `requirements.txt` (`googlemaps>=4.10.0`)** — era l'unico connector python senza. Le dipendenze erano dichiarate nel campo `dependencies` del manifest, che skald usa **solo per la card**: `ensure_installed_host` guarda esclusivamente `requirements.txt` / `package.json`. Risultato: `.pydeps` vuoto e log pieno di `No module named 'googlemaps'`, con il server che rispondeva comunque a `tools/list` (→ `connected — 6 tool(s)` su un connector non funzionante).
 - **Cablato il `verify`** (`python3 verify.py`, 20s): `verify.py` era shippato ma il manifest non aveva il blocco `verify`, e skald legge `verify_command` solo da lì. Ora un'abilitazione con dipendenze rotte fallisce visibilmente invece di partire in silenzio.
@@ -304,7 +304,8 @@
      "icon_large": "firecrawl/icon_lg.png",
      "user_description": "Web scraping, crawling, search, deep research, and autonomous browsing \u2014 extract content from any website.",
      "requires": [
-        "NODE"
+        "NODE",
+        "API_KEY"
      ],
      "tags": [
        "scraping",
@@ -319,9 +320,9 @@
        "type": "api_key"
      },
      "folder": "firecrawl",
-      "version": 4,
-      "version_string": "1.0.3",
-      "version_release_date": "2026-07-21",
+      "version": 5,
+      "version_string": "1.1.0",
+      "version_release_date": "2026-08-10",
      "tools": [
        {
          "name": "firecrawl_scrape",
@@ -335,6 +336,10 @@
          "name": "firecrawl_search",
          "display_name": "Web Search"
        },
+        {
+          "name": "firecrawl_developer_search",
+          "display_name": "Developer Search"
+        },
        {
          "name": "firecrawl_search_feedback",
          "display_name": "Search Feedback"
@@ -431,8 +436,8 @@
      "files": [
        {
          "path": "connector.json",
-          "sha256": "6cab1a43e32741eb4f31638841e65b68fb937d2c6313fd18ace38e722bc2eb23",
-          "size": 5008
+          "sha256": "15713684157541a255b3c1eb143a00e715bc833878fd46470bd2d5e396acfb32",
+          "size": 5006
        },
        {
          "path": "icon_lg.png",
@@ -443,6 +448,16 @@
          "path": "icon_sm.png",
          "sha256": "3a4be30d4c16ce93a1d132e063dfdffb8bd27b6c97715c9f3b12414439aaeb01",
          "size": 6858
+        },
+        {
+          "path": "index.js",
+          "sha256": "bbd77881f07ea9fc17fe9545e9ad9bc8fa74b652b77b9d20d42b012e2356d0e9",
+          "size": 817
+        },
+        {
+          "path": "package.json",
+          "sha256": "d5f96ea065a92e12adc41be3babb00bdd8c86de149d7beb8631184530c5fc617",
+          "size": 278
        }
      ]
    },
@@ -470,9 +485,9 @@
        "type": "none"
      },
      "folder": "http-fetch",
-      "version": 4,
-      "version_string": "1.0.3",
-      "version_release_date": "2026-07-21",
+      "version": 5,
+      "version_string": "1.1.0",
+      "version_release_date": "2026-08-10",
      "tools": [
        {
          "name": "fetch_html",
@@ -502,8 +517,8 @@
      "files": [
        {
          "path": "connector.json",
-          "sha256": "8087d8ed8af7c6b628c95ff74f5126023a1ed1c99c2b3fc5c0599f665ffaa6f2",
-          "size": 1862
+          "sha256": "7d9b0881325fe559e0d637dd3a2787c0d3d190b89e5f437db1754ac87f565f2c",
+          "size": 1908
        },
        {
          "path": "icon_lg.svg",
@@ -514,6 +529,16 @@
          "path": "icon_sm.svg",
          "sha256": "fa32e634744d8cba369f6d8ba45746166124f7d3507cefd9a1f96180c787e1a8",
          "size": 380
+        },
+        {
+          "path": "index.js",
+          "sha256": "bc092a323631b8de99bac70a0b0d0486fa77313c4d75d1182b8c426f6d529d28",
+          "size": 703
+        },
+        {
+          "path": "package.json",
+          "sha256": "29a4e0c21b65e7f0a1d69381b48c3fd505520eb9e2e4a4afeb0118135b80348e",
+          "size": 284
        }
      ]
    },
@@ -2,12 +2,10 @@
  "id": "firecrawl",
  "name": "Firecrawl",
  "type": "mcp_local",
-  "launch_command": "npx -y firecrawl-mcp",
-  "transport": "stdio",
  "requires": [
-    "NODE"
+    "NODE",
+    "API_KEY"
  ],
-  "dependencies": [],
  "env": [
    {
      "name": "FIRECRAWL_API_KEY",
@@ -19,9 +17,9 @@
    }
  ],
  "setup_instructions": [
-    "Ensure Node.js is installed.",
+    "Requires Node.js 22+ on the host that runs the connector.",
    "Get an API key from https://www.firecrawl.dev/app/api-keys.",
-    "No pip install needed \u2014 Firecrawl MCP runs via npx."
+    "Dependencies are installed automatically by Skald from package.json."
  ],
  "docs": [
    {
@@ -34,14 +32,11 @@
    "type": "api_key"
  },
  "mcp_config": {
-    "command": "npx",
+    "command": "node",
    "args": [
-      "-y",
-      "firecrawl-mcp"
+      "index.js"
    ],
-    "env": {
-      "FIRECRAWL_API_KEY": "{SECRET:FIRECRAWL_API_KEY}"
-    }
+    "transport": "stdio"
  },
  "verify": {
    "command": "node -e \"const k = process.env.FIRECRAWL_API_KEY; if (!k || k.length < 10) { process.exit(1); } try { fetch('https://api.firecrawl.dev/v1/scrape', {method:'POST', headers:{'Content-Type':'application/json','Authorization':'Bearer '+k}, body:JSON.stringify({url:'https://example.com',formats:['markdown']})}).then(r=>r.json()).then(d=>{if(d.success) {console.log(JSON.stringify({ok:true,message:'Firecrawl API key valid \u2014 scrape probe successful'}))} else {console.log(JSON.stringify({ok:false,message:d.error||'API probe failed'}));process.exit(1)}}).catch(e=>{console.log(JSON.stringify({ok:false,message:e.message}));process.exit(1)}) } catch(e) { console.log(JSON.stringify({ok:false,message:e.message})); process.exit(1) }\"",
@@ -60,9 +55,9 @@
    "browser",
    "research"
  ],
-  "version": 2,
-  "version_string": "1.0.1",
-  "version_release_date": "2026-07-21",
+  "version": 5,
+  "version_string": "1.1.0",
+  "version_release_date": "2026-08-10",
  "tools": [
    {
      "name": "firecrawl_scrape",
@@ -76,6 +71,10 @@
      "name": "firecrawl_search",
      "display_name": "Web Search"
    },
+    {
+      "name": "firecrawl_developer_search",
+      "display_name": "Developer Search"
+    },
    {
      "name": "firecrawl_search_feedback",
      "display_name": "Search Feedback"
@@ -7,7 +7,8 @@
  "icon_large": "firecrawl/icon_lg.png",
  "user_description": "Web scraping, crawling, search, deep research, and autonomous browsing \u2014 extract content from any website.",
  "requires": [
-    "NODE"
+    "NODE",
+    "API_KEY"
  ],
  "tags": [
    "scraping",
@@ -22,9 +23,9 @@
    "type": "api_key"
  },
  "folder": "firecrawl",
-  "version": 4,
-  "version_string": "1.0.3",
-  "version_release_date": "2026-07-21",
+  "version": 5,
+  "version_string": "1.1.0",
+  "version_release_date": "2026-08-10",
  "tools": [
    {
      "name": "firecrawl_scrape",
@@ -38,6 +39,10 @@
      "name": "firecrawl_search",
      "display_name": "Web Search"
    },
+    {
+      "name": "firecrawl_developer_search",
+      "display_name": "Developer Search"
+    },
    {
      "name": "firecrawl_search_feedback",
      "display_name": "Search Feedback"
@@ -0,0 +1,16 @@
+// Skald connector entry point.
+//
+// The MCP server itself is the upstream `firecrawl-mcp` package, pinned in
+// package.json and installed by Skald beside this file (`npm ci --omit=dev`,
+// falling back to `npm install --omit=dev`).
+//
+// This file exists because Skald launches a `mcp_local` connector as
+// `<command> <mcp_config.args[0]>`, where args[0] is rewritten to the absolute
+// path of a *shipped* file — so `npx -y <package>` cannot be expressed: args[0]
+// would be taken as the entry file's name. Importing the package's server module
+// for its side effect (it starts the stdio JSON-RPC loop at import time) is the
+// whole of the wrapper.
+//
+// The API key arrives as FIRECRAWL_API_KEY in the process environment, collected
+// from the manifest's env[] form.
+import "firecrawl-mcp/dist/index.js";
@@ -0,0 +1,13 @@
+{
+  "name": "skald-connector-firecrawl",
+  "version": "1.1.0",
+  "private": true,
+  "type": "module",
+  "description": "Skald connector wrapper around the firecrawl-mcp MCP server.",
+  "dependencies": {
+    "firecrawl-mcp": "3.23.7"
+  },
+  "engines": {
+    "node": ">=22"
+  }
+}
@@ -2,37 +2,35 @@
  "id": "http-fetch",
  "name": "HTTP Fetch",
  "type": "mcp_local",
-  "launch_command": "npx -y mcp-fetch-server",
-  "transport": "stdio",
+  "scope": "global",
  "requires": [
    "NODE"
  ],
-  "dependencies": [],
  "setup_instructions": [
-    "Ensure Node.js is installed.",
-    "No configuration needed \u2014 HTTP Fetch is a free tool with no API key."
+    "Requires Node.js 18+ on the host that runs the connector.",
+    "No configuration needed — HTTP Fetch is free and needs no API key.",
+    "Dependencies are installed automatically by Skald from package.json."
  ],
  "docs": [
    {
      "lang": "en",
-      "description": "Fetch any URL and convert it to HTML, Markdown, plain text, JSON, or YouTube transcripts \u2014 free, no API key required, no rate limits, no cost. Prefer this over Exa or Firecrawl for simple URL fetching. Supports YouTube video transcripts via fetch_youtube_transcript.",
-      "llm_short_description": "HTTP Fetch \u2014 fetch any URL and convert to HTML, Markdown, plain text, JSON, or readable article format (strips ads). Also extracts YouTube transcripts. Free, no API key."
+      "description": "Fetch any URL and convert it to HTML, Markdown, plain text, JSON, or YouTube transcripts — free, no API key required, no rate limits, no cost. Prefer this over Exa or Firecrawl for simple URL fetching. Supports YouTube video transcripts via fetch_youtube_transcript.",
+      "llm_short_description": "HTTP Fetch — fetch any URL and convert to HTML, Markdown, plain text, JSON, or readable article format (strips ads). Also extracts YouTube transcripts. Free, no API key."
    }
  ],
  "auth": {
    "type": "none"
  },
  "mcp_config": {
-    "command": "npx",
+    "command": "node",
    "args": [
-      "-y",
-      "mcp-fetch-server"
-    ]
+      "index.js"
+    ],
+    "transport": "stdio"
  },
-  "homepage": "",
+  "homepage": "https://github.com/zcaceres/fetch-mcp",
  "icon_small": "icon_sm.svg",
  "icon_large": "icon_lg.svg",
-  "scope": "global",
  "tags": [
    "http",
    "fetch",
@@ -42,9 +40,9 @@
    "scraping",
    "youtube"
  ],
-  "version": 2,
-  "version_string": "1.0.1",
-  "version_release_date": "2026-07-21",
+  "version": 5,
+  "version_string": "1.1.0",
+  "version_release_date": "2026-08-10",
  "tools": [
    {
      "name": "fetch_html",
@@ -22,9 +22,9 @@
    "type": "none"
  },
  "folder": "http-fetch",
-  "version": 4,
-  "version_string": "1.0.3",
-  "version_release_date": "2026-07-21",
+  "version": 5,
+  "version_string": "1.1.0",
+  "version_release_date": "2026-08-10",
  "tools": [
    {
      "name": "fetch_html",
@@ -0,0 +1,13 @@
+// Skald connector entry point.
+//
+// The MCP server itself is the upstream `mcp-fetch-server` package, pinned in
+// package.json and installed by Skald beside this file (`npm ci --omit=dev`,
+// falling back to `npm install --omit=dev`).
+//
+// This file exists because Skald launches a `mcp_local` connector as
+// `<command> <mcp_config.args[0]>`, where args[0] is rewritten to the absolute
+// path of a *shipped* file — so `npx -y <package>` cannot be expressed: args[0]
+// would be taken as the entry file's name. Importing the package's server module
+// for its side effect (it starts the stdio JSON-RPC loop at import time) is the
+// whole of the wrapper.
+import "mcp-fetch-server/dist/index.js";
@@ -0,0 +1,13 @@
+{
+  "name": "skald-connector-http-fetch",
+  "version": "1.1.0",
+  "private": true,
+  "type": "module",
+  "description": "Skald connector wrapper around the mcp-fetch-server MCP server.",
+  "dependencies": {
+    "mcp-fetch-server": "1.1.2"
+  },
+  "engines": {
+    "node": ">=18"
+  }
+}