feat(skills): rebuild the skill system for the multi-user model
Nightly Build / build (push) Successful in 8m6s

Per blueprint/skill-project.md: the old single-namespace, hand-maintained
index is gone, replaced by a read-only, two-scope tree whose index is a
runtime function of its content.

- skills/ index generated at runtime (crates/skald-core/src/skills/:
  inventory, install, validate, watch), injected through the new
  <!-- SKILLS_LIST --> placeholder in AGENT.md (agents/common/skills.md);
  meta.json inject_skills flag removed. 11 chat/task agents carry the
  include, the 4 system agents do not.
- Two trees, both read-only in both directions: skills/shared/{id} (the
  group's) and skills/{username}/{id} (one member's own, on the stable
  userid). The root is closed too: UserFs::SkillMounts + RouteError (alias
  probe, plain-denied paths, no home fallback) and a per-user
  .skills-root/{userid} container mount with the two scope mounts nested
  inside, plus the fifth self-heal axis (skills_mounted).
- Agent verbs: skill_register/skill_delete (Config group, global scope
  behind the new skill.manage capability), fetch_repo for public repos,
  list_items(type="skills"); reads are plain read_file on the printed
  path. Seeded @fs_read skills/* allow.
- Freshness: a digest-gated watcher on the two trees emits
  SystemEvent::SkillsChanged, whose subscriber rebuilds the frozen prompt
  prefix via Skald::invalidate_prompt_prefix; in-process writers invalidate
  directly.
- The build ships no skills: the three bundled skills (ics2json,
  mcp-builder, skill-creator) and skills/index.md are removed, skills/ is
  instance data (gitignored, not packaged, no longer pruned by update.sh).
- Docs: skills.md, agents.md, shared-folders.md added; docs/index.md and
  agents/README.md updated.
This commit is contained in:
Daniele
2026-08-08 23:05:35 +01:00
parent 71e1a26b08
commit c27da4e6ab
88 changed files with 4624 additions and 9546 deletions
+10 -2
View File
@@ -53,8 +53,16 @@ node_modules/
# ── macOS ─────────────────────────────────────────────────────────────────────
.DS_Store
# ── Private skills ────────────────────────────────────────────────────────────
skills/.gitignore
# ── Skills (blueprint: skill system) ──────────────────────────────────────────
# The build ships no skills: every one of these directories is instance data,
# filled only by what a member registers. `skills/` is the group-wide tree,
# `skills-users/{userid}/` a member's own, and `.skills-root/{userid}/` the
# read-only mount that carries the signpost plus the two scope mountpoints
# (regenerated at every container `ensure` from the consts in
# crates/skald-core/src/container/mod.rs).
/skills/
/skills-users/
/.skills-root/
# ── Editors & IDEs ────────────────────────────────────────────────────────────
.claude/
+8 -1
View File
@@ -118,6 +118,7 @@ Two rules keep the boundary real, and both are enforced by the compiler:
| `crates/skald-core/src/transcribe/` | Transcription providers |
| `crates/skald-core/src/image_generate/` | Image generation providers |
| `crates/skald-core/src/memory/` | Agent memory tools |
| `crates/skald-core/src/skills/` | The skills index: pure functions over the two read-only trees (enumerate → parse frontmatter → render → digest). No state, no watcher — see the skills paragraphs in Filesystem & containers |
| `src/frontend/mod.rs` | `WebFrontend`: wires router_factory, starts plugins, runs Axum |
| `src/frontend/server.rs` | Axum router, static file serving |
| `src/frontend/api/` | HTTP + WebSocket handlers — `State<Arc<Skald>>` |
@@ -144,7 +145,7 @@ The schema is split into two buckets (§5.1), and the split is the point:
**Memory injection into the prompt**: `AgentSystemContext::load_inject_memory` (`loop_adapters/system.rs`) routes each `meta.inject_memory` entry — `user-memory/…` → owner pool, `shared-memory/…` → the shared (`system.db`) pool, both via `memory_docs::get`; anything else (`data/…`, `$WD/…`) is a disk read. The shared pool is threaded `ChatSessionManager``UserLoopRuntime``AgentSystemContext`. `assistant` and `project-coordinator` inject `user-memory/index.md` + `shared-memory/index.md`.
**Prompt substitutions**: an `AGENT.md` may carry `<!-- KEY -->` placeholders; `agents::resolve_includes` turns each into a `__KEY__` sentinel, replaced at request time. Two are resolved by the system-context source itself (`loop_adapters/system.rs`) from the session owner (`user_id`) + registry (`shared_pool`), so every source (WS, mobile, cron, sub-agents) gets them with no caller plumbing: `__SHARED_FOLDERS__` (the user's shared-folders table) and `__USER_PROFILE__` (the owner's directory profile: `Name`, `Date of birth` with age computed at build time, `Sex`, `Preferred language`, admin `Notes` — unset values render as explicit `unknown` / `not specified`, the `Notes` line is omitted when empty). Any other key comes from the per-call `SendMessageOptions::system_substitutions` map.
**Prompt substitutions**: an `AGENT.md` may carry `<!-- KEY -->` placeholders; `agents::resolve_includes` turns each into a `__KEY__` sentinel, replaced at request time. Several are resolved by the system-context source itself (`loop_adapters/system.rs`) from the session owner (`user_id`) + registry (`shared_pool`) + their `UserFs`, so every source (WS, mobile, cron, sub-agents) gets them with no caller plumbing: `__SKILLS_LIST__` (the generated skills index — see Filesystem & containers), `__SHARED_FOLDERS__` (the user's shared-folders table) and `__USER_PROFILE__` (the owner's directory profile: `Name`, `Date of birth` with age computed at build time, `Sex`, `Preferred language`, admin `Notes` — unset values render as explicit `unknown` / `not specified`, the `Notes` line is omitted when empty). Any other key comes from the per-call `SendMessageOptions::system_substitutions` map.
`system.db` still gets **both** bucket functions — but no longer because the migration is unstarted. It gets the owner schema because it *is* the owner of **shared** memory (`memory_docs`) plus, for now, the globally-scoped `secrets` (`SecretsStore` is built on the system pool and shared by reference into every `UserContext`; the global runtime's *config* now lives in the registry table `mcp_global_servers`, and per-user connector config in each user's owner `mcp_user_servers`). The global runtime no longer writes `mcp_events` there: notification persistence is an explicit `McpManager::new` argument (`EventLog::{Persist,Discard}`), `Discard` for the ownerless global runtime and `Persist` for each per-user one, because an event belongs to whoever it happened to and its only reader (event triage) is per-user. Every *other* owner table is created there but never written to anymore — the global owner-bound managers that would write them (chat/jobs/etc.) are inert (see "Current state"). Fully dropping `create_owner_tables` from `system.db` is blocked on the §4 scope decision for secrets, not on call-site migration.
@@ -162,6 +163,8 @@ The agent sees **one namespace**, routed on the first path component. The choke
| `shared-memory/…` | SQLite `system.db` | `classify_memory``memory_docs` |
| `shared/{X}/…` | host `{WD}/shared/{X}` (if a member) | `UserFs::host_base_and_tail` |
| `projects/{O}/{S}/…` | host `{WD}/projects/{owner_userid}/{S}` (if a member) | `UserFs::host_base_and_tail` |
| `skills/shared/{id}/…` | host `{WD}/skills/{id}`**read-only** | `UserFs::host_base_and_tail` |
| `skills/{username}/{id}/…` | host `{WD}/skills-users/{userid}/{id}`**read-only** | `UserFs::host_base_and_tail` |
| `~/…`, relative | host `{WD}/homes/{userid}` | `UserFs::host_base_and_tail` |
| any other absolute path (`/tmp/…`, `/etc/…`) | the **container's own** filesystem | `resolve_target``container::exec_fs` |
@@ -173,6 +176,10 @@ Two views, **one storage**: for the mounted subtree the fs-tools run **host-side
**The memory roots are signposted inside the container, not merely absent.** `user-memory/`/`shared-memory/` are virtual, so nothing of them existed on disk — and the nothing was worse than it sounds: `cat user-memory/x.md` returned a bare ENOENT (which reads as *the note is missing*, not *wrong door*), while `mkdir -p user-memory && echo … > user-memory/x.md` **succeeded**, writing a real file into the home that no reader ever visits and that the next `ls` then confirms as if it had worked. Each root is therefore a **read-only bind mount** (`{WD}/.memory-signpost/{root}``{container_home}/{root}:ro`, gitignored, rewritten from consts on every `ensure`) holding a README that names the tools. Read-only *as a mount*, not as a mode: the container user has passwordless `sudo`, so a `chmod` would be a suggestion, whereas `:ro` holds — remounting needs `CAP_SYS_ADMIN` (verified: write, `sudo` write, `sudo chmod`, `sudo mount -o remount,rw` and `sudo rm` all fail). A README rather than an empty dir because `Permission denied` is an error, not an instruction — models answer it by reaching for `sudo`; the README puts the correction in the directory the failing command just named. These mounts are deliberately **not** in `UserFs`: they back no agent path and the host-side fs-tools must never resolve into them. They are the **fourth self-heal axis** in `reusable()` (`signposts_mounted`) rather than an `IMAGE_TAG` bump, since the image is unchanged and a bump would make every box rebuild it to fix a mount. The matching half is in `classify_memory`, which now strips the home spellings (`./`, `~/`, `/root/`) before matching the root — without it `~/user-memory/x.md` missed the match, fell through to the disk router, and became exactly the invisible physical file the signpost exists to prevent.
**Skills are a read-only tree with two scopes, and the space *between* them is closed too.** `skills/shared/{id}` is the group's, `skills/{username}/{id}` is one member's own (`core-api`'s `SkillMounts`; agent path on the username like `projects/`, host path on the stable userid). Everything under `skills/` is read-only in **both** directions — `:ro` bind mounts and `can_write_to → false` — because these hold installed artefacts, not working files: a skill body is *read as instruction* by whoever it is visible to, so writing one is a decision that must pass a gate, not a file write — the one door is `skill_register` (with `skill_delete` and `list_items(type="skills")`), called from the chat and gated `require`; a public repo is fetched with `fetch_repo` and then registered. Two traps, both closed together and neither covering the other's half. **Host-side**, the `skills` arm of `can_write_to`/`host_base_and_tail` spans the **whole root**, not the two known scopes: the fallthrough answers `true`/home, so an invented scope segment (`skills/pippo/SKILL.md` — the *likely* guess, not the lucky one) would land in a physical directory under the home that no indexer ever reads. That is the memory-signpost failure exactly. **In-container**, the defect is structural rather than name-dependent: the scope mounts nest inside `container_home`, so `/root/skills` would be a real directory inside the *writable* home mount and `mkdir -p ~/skills/pippo` would succeed. Hence a third mount: `{WD}/.skills-root/{userid}``{container_home}/skills:ro`, holding the signpost README plus the two scope mountpoints. That root is **per-user and materialized whole** (`container::ensure_skills_root`) because Docker refuses to create a mountpoint inside a `:ro` mount — `shared/` and `{username}/` must already exist in the root's own source, and one of those names is the member's — which is also why the three host paths are one `SkillMounts` field rather than three `Option<PathBuf>`. `mounts()` emits them root-first; `skills_mounted` is the **fifth self-heal axis**, for the signposts' reason. A stale scope dir left by a rename is pruned at each `ensure`. The bare-id alias `skills/{id}` (the shortest spelling, so the one a model writes unprompted) resolves in `resolve_skill_alias`**only** when the id is unique across the two trees, failing loudly with both full paths otherwise, since a personal skill silently shadowing a group one is a divergence nobody chose. `UserFs` stays pure: it returns `RouteError::SkillAlias` and skald-core does the probe.
**The skills index is generated, and the sentinel is the knob.** What reaches the model is not a file anyone maintains but a **function of the two trees** (`crates/skald-core/src/skills/`, pure functions in the shape of `LlmCommandManager`): each skill's `SKILL.md` **path** plus its frontmatter `description`, truncated to 200 chars, under an imperative header ("you MUST read its SKILL.md") — the countermeasure to the real failure mode, which is the model *under*-triggering. Printing the full path rather than an id plus a composition rule is what makes a read tool unnecessary: `read_file` on the printed path is one call, and there is no step left for the model to get wrong. Injection is the placeholder `<!-- SKILLS_LIST -->` (normally `<!-- INCLUDE: common/skills.md -->`, a fragment that holds **only** the sentinel), substituted in `AgentSystemContext::build_base` beside `__MCP_LIST__`; `resolve_includes` needs no branch, its generic `<!-- KEY -->``__KEY__` arm already covers it. There is **no `meta.json` flag** — the sentinel *is* the switch, so the four `type: system` agents opt out by not including the fragment (an imperative "read it with read_file" is exactly wrong in an unattended turn, and some of those run with `allow_tools: false`). All eleven `chat`/`task` agents carry the include, sub-agents included: in a delegation the one doing the work is the child. Three rendering rules are load-bearing and each closes a specific failure: a **stable order** (scope, then id) because the index sits inside the provider's cache key; a **deterministic tail cut** at an 8 KB budget, announced by a `[N more skills omitted]` line, because a silently truncated index has the model conclude in good faith that a skill does not exist; and **empty in, empty out** — every word of prose lives inside the render, so an instance with no skills spends zero tokens and leaves no orphan sentence (the MCP list is the counter-example: its prose sits *around* the placeholder, and the empty state once had the model inventing a discovery tool). A colliding id is marked `[name collision]` on **both** lines, never shadowed. A malformed skill is skipped with a `warn!`, never fatal — the index is built while assembling a prompt. Freshness has two doors, one per writer. The in-process tools invalidate directly (`Skald::invalidate_prompt_prefix`, called by `skill_register`/`skill_delete`); a hand edit on the box is caught by the **skills watcher** (`skills/watch.rs`, spawned from `spawn_background`): a recursive `notify` on `{WD}/skills` + `{WD}/skills-users`, debounced ~800 ms, that re-digests each touched tree (`skills::tree_digest` — the (id, description) pairs the index is made of) and emits `SystemEvent::SkillsChanged { scope }` only when the digest moved. The subscriber `spawn_skills_freshness` (next to `spawn_user_lifecycle`, same `Weak` shape) maps the scope and calls the same invalidate accessor. Editing a script leaves the digest byte-identical and announces nothing — which is exactly the §6 rule, so an invisible change costs nobody a cache miss. Two gotchas the code carries comments for: the watcher **canonicalizes `{WD}`** (FSEvents reports real paths, and `/var` is a symlink on macOS), and it creates the two trees if absent (a box before its first user has neither).
**Containment** (`resolve_host_path`) is unchanged and still guards **the host branch**: every path that lands on a mount is canonicalized (following symlinks) and prefix-checked against its mount base, **fail-closed**. That check is what it always was — the defence against a symlink planted from inside the container pointing at the **host's** `/etc`, which the host-side tool would otherwise follow off the box. Opening the container branch does not weaken it: that branch never touches the host filesystem, so there is no host to escape from, and the check keeps applying to everything mounted. `grep_files` stays disk-only (regex ≠ FTS; memory → `memory_search`) but resolves its root the same way. `execute_cmd`'s `workdir` is an agent path mapped to its container path via `UserFs::to_container`.
The threading: `UserContext.fs` (built by `container::build_user_fs` at login, snapshotting shared memberships) → `ChatSessionManager``ChatSessionHandler.fs``ToolContext.fs`. **Admin CRUD is wired** (`src/frontend/api/shared_folders.rs``GET/POST /api/shared-folders`, `PATCH/DELETE /api/shared-folders/{id}`, `POST`/`DELETE .../members[/{user_id}]`; UI `shared-folders.js`): a create/describe/delete + per-member `can_write` surface, and each mutation emits `SystemEvent::UserMountsChanged`, on which the lifecycle reconciler runs `Skald::refresh_user_mounts` — rebuilding the affected user's fs + container mounts **in place**, so a membership change lands without a re-login (blueprint §6's "admin CRUD" + "membership refresh without re-login" TODOs, now closed; it still settles at next login/boot if the live remount fails). `execute_cmd` /stop is robust: the command runs under `setsid -w` in its own process-group (leader pid recorded in a container pidfile), and a `KillReaper` drop-guard reaps that group on /stop **or** timeout via a detached `docker exec` that walks `/proc` and kills members by **positive pid** (the container's dash mishandles `kill -<pgid>`); the pidfile is passed positionally (`$1`), and the container's `--init` (tini) reaps the killed processes so no zombies accumulate. **Per-user MCP connectors now run inside this container** (§7) — the container infra enabled it; see the MCP connectors section.
+2 -2
View File
@@ -124,9 +124,9 @@ systemd service → ExecStart=run.sh
**Problem**: extracting over the install directory only ever adds and overwrites. Anything removed upstream survived every future update — a renamed page under `docs/` kept being mounted read-only into every container for the assistant to read, a deleted command kept being discovered.
**Fix**: after extracting, prune from the directories the tarball owns end to end (`web/`, `commands/`, `skills/`, `docs/`) whatever the already-verified staging copy does not have, then remove the directories left empty. Pruning _after_ the extraction rather than replacing the directory keeps every intermediate state a complete install, and the only files removed are ones the new build has verifiably dropped.
**Fix**: after extracting, prune from the directories the tarball owns end to end (`web/`, `commands/`, `docs/`) whatever the already-verified staging copy does not have, then remove the directories left empty. Pruning _after_ the extraction rather than replacing the directory keeps every intermediate state a complete install, and the only files removed are ones the new build has verifiably dropped.
`agents/` is deliberately excluded: adding an agent is a documented extension point (`agents/<id>/meta.json` + `AGENT.md`), so the directory is not ours alone and pruning it would delete somebody's work — at the price of an upstream-deleted agent lingering. `bin/` is excluded too: two files, both overwritten every time.
`agents/` is deliberately excluded: adding an agent is a documented extension point (`agents/<id>/meta.json` + `AGENT.md`), so the directory is not ours alone and pruning it would delete somebody's work — at the price of an upstream-deleted agent lingering. `skills/` is excluded for a stronger version of the same reason: the build ships no skills, so that directory is pure instance data (every skill in it was registered by a member) and pruning it would delete their work at every update. `bin/` is excluded too: two files, both overwritten every time.
## Bug fix: uninstall.sh could remove containers that are not ours ✅
+32
View File
@@ -1,3 +1,35 @@
# Agents
## Adding a new agent: the skills index is opt-in
An agent sees the installed skills **only** if its `AGENT.md` carries the
`<!-- SKILLS_LIST -->` placeholder, normally through
`<!-- INCLUDE: common/skills.md -->`. There is no `meta.json` flag: the sentinel
*is* the switch, exactly as it is for `<!-- MCP_LIST -->`.
So a new agent starts **without** the index and stays without it until someone
adds the line. That is the deliberate direction of the default: the opposite one
— an agent inheriting the index by forgetfulness — is the worse failure, because
the index is written in the imperative ("you MUST read its SKILL.md") and an
unattended `type: system` agent has its approvals auto-denied and sometimes no
tools at all.
`common/skills.md` is **one line and deliberately holds no prose**, unlike
`common/mcp.md`. Every word — the imperative header, the list, the closing rules
— is produced by the renderer, so that an instance with no skills installed gets
an empty string instead of a header promising a list that isn't there. (That is
not hypothetical: the MCP section keeps its prose in the fragment, and its empty
state once had the model invent a discovery tool to fill the gap.) The fragment
cannot explain itself in place either — `resolve_includes` copies any line that
is not an upper-case sentinel straight into the prompt, so a comment there would
be read by the model.
The rule of thumb: a `chat` or `task` agent gets the include, a `system` agent
does not. Put the line **as low as possible** in the prompt (by convention right
after `common/mcp.md`) — anything above it survives in the provider's cached
prefix when a skill is added or removed. `crates/skald-core/src/agents.rs` has a
test that holds every shipped agent to this.
# Agent icons — style guide
Each agent in the `agents/` directory can have an icon/avatar declared in the `"icon"` field of its `meta.json`. The backend serves the file via `GET /api/agents/{id}/icon`.
+2
View File
@@ -75,6 +75,8 @@ To change what gets notified, edit `data/notifications.md`.
<!-- INCLUDE: common/mcp.md -->
<!-- INCLUDE: common/skills.md -->
## System configuration
Configuration tools are hidden by default to keep context small. Call `activate_tools(["config"])` to load them when you need to manage the instance's setup — plugins, scheduled jobs, secrets — then work normally.
+2
View File
@@ -120,3 +120,5 @@ No other output — the file is the report.
---
<!-- INCLUDE: common/mcp.md -->
<!-- INCLUDE: common/skills.md -->
+2
View File
@@ -64,3 +64,5 @@ _Date: 2026-06-03_
---
<!-- INCLUDE: common/mcp.md -->
<!-- INCLUDE: common/skills.md -->
+1
View File
@@ -0,0 +1 @@
<!-- SKILLS_LIST -->
-1
View File
@@ -13,7 +13,6 @@
}
},
"type": "system",
"inject_skills": false,
"allow_tools": false,
"strength": "high"
}
-1
View File
@@ -13,7 +13,6 @@
}
},
"type": "system",
"inject_skills": false,
"inject_memory": ["user-memory/index.md"],
"icon": "icon.png",
"strength": "low"
+2
View File
@@ -13,3 +13,5 @@ You do NOT delegate to other agents. Do the work yourself.
---
<!-- INCLUDE: common/mcp.md -->
<!-- INCLUDE: common/skills.md -->
+2
View File
@@ -83,6 +83,8 @@ There may be other helpers in the household's team — each good at different th
<!-- INCLUDE: common/mcp.md -->
<!-- INCLUDE: common/skills.md -->
---
## Shared folders
-1
View File
@@ -13,7 +13,6 @@
}
},
"type": "system",
"inject_skills": false,
"inject_memory": ["user-memory/index.md"],
"icon": "icon.png",
"strength": "average"
-1
View File
@@ -13,7 +13,6 @@
}
},
"type": "system",
"inject_skills": false,
"inject_memory": ["shared-memory/index.md"],
"icon": "icon.png",
"strength": "average"
+2
View File
@@ -12,6 +12,8 @@ The user is talking to a single assistant that already knows the project. They s
<!-- INCLUDE: common/mcp.md -->
<!-- INCLUDE: common/skills.md -->
## System configuration
Configuration tools are hidden by default to keep context small. Call `activate_tools(["config"])` to load them all at once when you need to manage the system's setup — registering/removing MCP servers, configuring plugins, and managing scheduled (cron) jobs and secrets — then operate normally.
+2
View File
@@ -116,3 +116,5 @@ If the main agent calls you again on a related topic, check if a relevant scratc
---
<!-- INCLUDE: common/mcp.md -->
<!-- INCLUDE: common/skills.md -->
+2
View File
@@ -8,6 +8,8 @@ You are a staff-level software architect. You receive a change request, study th
<!-- INCLUDE: common/mcp.md -->
<!-- INCLUDE: common/skills.md -->
## Available agents
Delegate work to these task specialists via `execute_task` / `execute_subtask`:
+2
View File
@@ -10,6 +10,8 @@ You work on **any file type** in any project: Rust, Swift, Python, JavaScript/Ty
<!-- INCLUDE: common/mcp.md -->
<!-- INCLUDE: common/skills.md -->
---
## Project context
+2 -1
View File
@@ -26,7 +26,6 @@ Before writing, understand the domain:
- **Web research**: delegate complex multi-step research to `researcher` (e.g. "research best practices for offline-first iOS apps with Core Data + CloudKit sync")
- **Code analysis**: if the project already has existing code or documentation, delegate to `code-explorer` to study it and produce a structured report on the current architecture
- **Proactive MCP use**: if an MCP server could help (Wikipedia for domain background, web fetch for API docs, etc.), call `activate_tools` to activate it and use it — do not wait for instructions
- **Skills**: check `skills/index.md` — there may be reusable Python utilities for your task
### Phase 2 — Structure the Documentation
@@ -125,6 +124,8 @@ Do not wait for permission to use a tool that would clearly help.
<!-- INCLUDE: common/mcp.md -->
<!-- INCLUDE: common/skills.md -->
## Persistent memory
<!-- INCLUDE: common/memory.md -->
+2
View File
@@ -10,6 +10,8 @@ You do **not** implement features yourself except for trivial scaffolding (creat
<!-- INCLUDE: common/mcp.md -->
<!-- INCLUDE: common/skills.md -->
## Available agents
Delegate work to these task specialists via `execute_task` / `execute_subtask`:
+3 -2
View File
@@ -16,7 +16,7 @@
# --output Directory where the .tar.gz will be written
#
# The tarball contains everything needed to run (or uninstall) Skald Circle:
# bin/skald, bin/skald-setup, web/, agents/, commands/, skills/, docs/,
# bin/skald, bin/skald-setup, web/, agents/, commands/, docs/,
# default.config.yaml, providers.yaml, requirements.txt,
# requirements-optional.txt, run.sh, update.sh, uninstall.sh
@@ -93,7 +93,8 @@ chmod 755 "$STAGING/bin/skald" "$STAGING/bin/skald-setup"
cp -r web "$STAGING/web"
cp -r agents "$STAGING/agents"
cp -r commands "$STAGING/commands"
cp -r skills "$STAGING/skills"
# No `skills/`: the build ships no skills (they are instance data, registered by
# members), so the directory is created by the app, never by the tarball.
cp -r docs "$STAGING/docs"
cp default.config.yaml "$STAGING/default.config.yaml"
cp providers.yaml "$STAGING/providers.yaml"
+28
View File
@@ -105,6 +105,21 @@ pub enum SystemEvent {
catalog_name: String,
},
// ── Skills (blueprint skill-project §8) ───────────────────────────────────
/// A skills tree changed on disk **in a way the index feels** — a skill was
/// added, removed or re-described by someone editing files by hand on the
/// box. Emitted by the freshness watcher after its digest gate: a change
/// that leaves the index byte-identical (a script, a reference document)
/// announces nothing, because the frozen system prefix citing that skill
/// has not aged. The in-process writers (`skill_register`/`skill_delete`)
/// never emit this — they invalidate directly.
///
/// Pure reconciliation, the contract this bus already promises: a lost
/// event costs a stale skill index for the prefix TTL, never a wrong one.
SkillsChanged {
scope: SkillScope,
},
// ── Reports (blueprint §13) ───────────────────────────────────────────────
/// A background agent filed a report. Announced by whoever wrote the row,
/// never delivered by it: *who* should hear about a report — the people
@@ -125,6 +140,19 @@ pub enum SystemEvent {
// ── Bus ───────────────────────────────────────────────────────────────────────
/// Which skills tree a [`SystemEvent::SkillsChanged`] is about.
///
/// Distinct from the `"mine" | "global"` vocabulary of the skill tools: this
/// names a *place on disk*, and a change to the group's tree concerns every
/// member's prompt while a change to one member's tree concerns only theirs.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum SkillScope {
/// `{WD}/skills` — the group's tree, in every member's index.
Global,
/// `{WD}/skills-users/{userid}` — one member's own tree.
User(String),
}
pub struct SystemEventBus {
tx: broadcast::Sender<SystemEvent>,
}
+365 -19
View File
@@ -10,6 +10,7 @@
//! | `shared/{X}/…` | host `{WD}/shared/{X}`, mount `{home}/shared/{X}` |
//! | `projects/{O}/{S}`| host `{WD}/projects/{owner_userid}/{S}`, mount `{home}/projects/{O}/{S}` (O = owner username) |
//! | `~/docs/…`, `docs/…` | host `{WD}/docs` (read-only, same for every user), mount `{container_home}/docs` |
//! | `skills/…` | the read-only skills tree — see [`SkillMounts`] |
//! | `~/…`, relative | host `{WD}/homes/{userid}`, mount `{container_home}`|
//!
//! `UserFs` is a **pure value type** with no filesystem access: it carries the
@@ -28,6 +29,15 @@ use std::sync::{Arc, RwLock};
/// root) so the two anchors can never drift.
pub const UPLOADS_SUBDIR: &str = "uploads";
/// The single top-level agent path under which every skill lives. Reserved: a
/// path starting with this segment never falls back to the home, whatever
/// follows it (see [`UserFs::host_base_and_tail`]).
pub const SKILLS_ROOT: &str = "skills";
/// The scope segment of the group-wide skills, `skills/shared/<id>`. The other
/// scope segment is the owner's own username, which is data, not a constant.
pub const SKILLS_SHARED_SCOPE: &str = "shared";
/// One shared folder mounted into a user's container.
#[derive(Debug, Clone)]
pub struct SharedMount {
@@ -60,6 +70,86 @@ pub struct ProjectMount {
pub can_write: bool,
}
/// The skills tree of one user: a single agent root, `skills/`, with two scope
/// subtrees below it — `skills/shared/<id>` (the group's, curated) and
/// `skills/<username>/<id>` (this member's own). The agent path carries the
/// **username** while the host path keys on the stable **userid**, exactly as
/// `projects/{owner_username}/{slug}` already does.
///
/// **Everything here is read-only for the agent, in both directions**: `:ro` bind
/// mounts in the container and [`UserFs::can_write_to`] false host-side. These are
/// not working folders — they hold installed artefacts, and the only door in is the
/// registration tool.
///
/// The three host paths are one field rather than three `Option`s because they
/// cannot exist apart. Docker refuses to create a mountpoint inside a `:ro` bind
/// mount (`mkdirat … read-only file system`, at container create), so the two scope
/// mounts nest inside the root mount only if `shared/` and `<username>/` already
/// exist **in the root mount's own source directory**. That forces the root to be
/// per-user (the username segment differs) and forces it to be materialized
/// together with the scopes it carries.
#[derive(Debug, Clone)]
pub struct SkillMounts {
/// Host dir mounted at `{container_home}/skills` (`{WD}/.skills-root/{userid}`).
/// Holds the signpost README plus the two empty scope mountpoints, and nothing
/// else: its job is to make the space *between* the scopes read-only too, so an
/// invented scope segment fails loudly instead of landing somewhere unread.
pub root_host: PathBuf,
/// Host dir behind `skills/shared/…` (`{WD}/skills`), the same for every user.
pub shared_host: PathBuf,
/// Host dir behind `skills/{own_username}/…` (`{WD}/skills-users/{userid}`).
pub own_host: PathBuf,
/// The owner's username — the agent-visible segment of their own scope.
pub own_username: String,
}
impl SkillMounts {
/// The container path of the root mount, given the home mount point.
pub fn container_root(&self, container_home: &Path) -> PathBuf {
container_home.join(SKILLS_ROOT)
}
/// The container paths of the two scope mounts, which nest inside the root.
pub fn container_scopes(&self, container_home: &Path) -> [PathBuf; 2] {
let root = self.container_root(container_home);
[root.join(SKILLS_SHARED_SCOPE), root.join(&self.own_username)]
}
}
/// Why an agent path does not resolve to a host location.
///
/// This exists because the wrong doors under `skills/` each need to say something
/// different, and a bare `None` could only ever produce one sentence. Saying the
/// right one matters more here than elsewhere: the whole root is read-only, so a
/// model that guesses a scope gets a refusal, and a refusal that does not name the
/// right path is answered with `sudo`.
#[derive(Debug, Clone, PartialEq)]
pub enum RouteError {
/// Not reachable, and this is the message to show the model.
Denied(String),
/// `skills/<id>/<tail>` where `<id>` is neither `shared` nor the owner's
/// username — so it may be the tolerant bare-id alias, the shortest spelling
/// and therefore the one a model produces on its own.
///
/// Resolving it means knowing which of the two trees actually holds `<id>`,
/// i.e. touching the filesystem, which this pure value type must not do. The
/// caller (skald-core's `resolve_host_path`) probes and either resolves it or
/// reports — including the ambiguous case, which fails loudly listing both
/// full paths rather than letting either tree win in silence.
SkillAlias { id: String, tail: String },
}
impl std::fmt::Display for RouteError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
RouteError::Denied(msg) => f.write_str(msg),
RouteError::SkillAlias { id, .. } => {
write!(f, "no skill named `{id}`")
}
}
}
}
/// The filesystem view of one user: their private home plus the shared folders
/// they belong to, and the container those are mounted into.
#[derive(Debug, Clone)]
@@ -79,6 +169,10 @@ pub struct UserFs {
/// every user. `None` when unset (inert placeholders, unit tests that don't
/// touch it) — `docs/…` then resolves like any other unmounted path.
pub docs_host: Option<PathBuf>,
/// The read-only skills tree (see [`SkillMounts`]). `None` for the inert
/// placeholders and unit tests that don't touch it — `skills/…` is then
/// refused outright, never routed to the home.
pub skills: Option<SkillMounts>,
}
impl UserFs {
@@ -99,9 +193,18 @@ impl UserFs {
shared,
projects,
docs_host,
skills: None,
}
}
/// Attach the skills tree. A builder step rather than an eighth constructor
/// argument: only the real per-user build has one, and every inert or test
/// `UserFs` is honestly skill-less.
pub fn with_skills(mut self, skills: SkillMounts) -> Self {
self.skills = Some(skills);
self
}
/// Look up a shared mount by its folder name.
pub fn shared_mount(&self, name: &str) -> Option<&SharedMount> {
self.shared.iter().find(|m| m.name == name)
@@ -116,9 +219,17 @@ impl UserFs {
/// Whether the user may **write** at this agent path: their home → always;
/// a shared-folder or project mount → the membership's `can_write` flag;
/// `docs/…` → never (read-only). A `shared/`/`projects/` mount the user is
/// not a member of → false (fail-closed, same as the read side). Purely
/// lexical: memory paths never reach here (classified earlier).
/// `docs/…` and **anything under `skills/`** → never (read-only). A
/// `shared/`/`projects/` mount the user is not a member of → false
/// (fail-closed, same as the read side). Purely lexical: memory paths never
/// reach here (classified earlier).
///
/// The `skills` arm covers the **whole root**, not the two known scopes, and
/// that width is the point: the fallthrough below answers `true`, so a scope
/// segment the model invented (`skills/pippo/SKILL.md`) would otherwise be
/// writable — and would land in a physical directory under the home that no
/// indexer ever reads. That is the memory-signpost failure exactly, and it is
/// closed here and, for the shell's half, by the root `:ro` mount.
pub fn can_write_to(&self, agent_path: &str) -> bool {
let stripped = strip_home_prefix(agent_path);
let mut parts = stripped.splitn(2, ['/', '\\']);
@@ -136,11 +247,19 @@ impl UserFs {
self.project_mount(owner, slug).map(|m| m.can_write).unwrap_or(false)
}
Some("docs") => false,
// The entire skills root, `self.skills` set or not: the name is
// reserved, so a context without the mounts must refuse rather than
// silently offer a home directory of the same name.
Some(SKILLS_ROOT) => false,
_ => true,
}
}
/// The bind mounts for `docker create`: `(host, container, writable)`, home first.
///
/// Emitted in **destination-depth order**, which the skills tree is the first to
/// actually need: its two scope mounts nest inside its root mount, and the root
/// must be in place before them.
pub fn mounts(&self) -> Vec<(PathBuf, PathBuf, bool)> {
let mut out = vec![(self.home_host.clone(), self.container_home.clone(), true)];
for m in &self.shared {
@@ -152,19 +271,25 @@ impl UserFs {
if let Some(docs) = &self.docs_host {
out.push((docs.clone(), self.container_home.join("docs"), false));
}
if let Some(sk) = &self.skills {
let [shared, own] = sk.container_scopes(&self.container_home);
out.push((sk.root_host.clone(), sk.container_root(&self.container_home), false));
out.push((sk.shared_host.clone(), shared, false));
out.push((sk.own_host.clone(), own, false));
}
out
}
/// The host base a physical agent path resolves against, and the tail relative
/// to it — **without** touching the filesystem. `shared/{X}/…` resolves against
/// the shared mount's host dir (only if the user is a member); everything else
/// resolves against the private home. Returns `None` when the path names a
/// `shared/` folder the user does not belong to. The caller (skald-core) then
/// joins + canonicalizes + prefix-checks against the returned base.
/// the shared mount's host dir (only if the user is a member); `skills/…`
/// against the skills tree; everything else against the private home. The
/// caller (skald-core) then joins + canonicalizes + prefix-checks against the
/// returned base.
///
/// Memory paths (`user-memory/…`, `shared-memory/…`) must be classified and
/// routed to SQLite *before* calling this — they are not physical paths.
pub fn host_base_and_tail<'a>(&self, agent_path: &'a str) -> Option<(PathBuf, String)> {
pub fn host_base_and_tail(&self, agent_path: &str) -> Result<(PathBuf, String), RouteError> {
let stripped = strip_home_prefix(agent_path);
let mut parts = stripped.splitn(2, ['/', '\\']);
match parts.next() {
@@ -173,8 +298,12 @@ impl UserFs {
let mut seg = rest.splitn(2, ['/', '\\']);
let name = seg.next().unwrap_or("");
let tail = seg.next().unwrap_or("");
let mount = self.shared_mount(name)?;
Some((mount.host.clone(), tail.to_string()))
let mount = self.shared_mount(name).ok_or_else(|| {
RouteError::Denied(format!(
"no such shared folder, or you are not a member: {agent_path}"
))
})?;
Ok((mount.host.clone(), tail.to_string()))
}
Some("projects") => {
// Two segments: `projects/{owner_username}/{slug}/{tail…}`.
@@ -183,15 +312,87 @@ impl UserFs {
let owner = seg.next().unwrap_or("");
let slug = seg.next().unwrap_or("");
let tail = seg.next().unwrap_or("");
let mount = self.project_mount(owner, slug)?;
Some((mount.host.clone(), tail.to_string()))
let mount = self.project_mount(owner, slug).ok_or_else(|| {
RouteError::Denied(format!(
"no such project, or you are not a member: {agent_path}"
))
})?;
Ok((mount.host.clone(), tail.to_string()))
}
Some("docs") => {
let host = self.docs_host.clone()?;
let host = self.docs_host.clone().ok_or_else(|| {
RouteError::Denied(format!("docs are not available here: {agent_path}"))
})?;
let tail = parts.next().unwrap_or("");
Some((host, tail.to_string()))
Ok((host, tail.to_string()))
}
_ => Some((self.home_host.clone(), stripped.to_string())),
Some(SKILLS_ROOT) => self.route_skills(agent_path, parts.next().unwrap_or("")),
_ => Ok((self.home_host.clone(), stripped.to_string())),
}
}
/// Routes everything under the reserved `skills/` root. Split out because it is
/// the one branch that must never fall through to the home: `skills/` names a
/// tree the user cannot write to and only partly owns, so the answer to an
/// unrecognised second segment is an error — never a home path that quietly
/// accepts a write nobody will ever read back.
fn route_skills(&self, agent_path: &str, rest: &str) -> Result<(PathBuf, String), RouteError> {
let Some(sk) = &self.skills else {
return Err(RouteError::Denied(format!(
"skills are not available in this context: {agent_path}"
)));
};
let mut seg = rest.splitn(2, ['/', '\\']);
let scope = seg.next().unwrap_or("");
let tail = seg.next().unwrap_or("");
if scope.is_empty() {
// `skills` / `skills/` itself: the root mount, which holds the signpost.
return Ok((sk.root_host.clone(), String::new()));
}
if scope == SKILLS_SHARED_SCOPE {
return Ok((sk.shared_host.clone(), tail.to_string()));
}
if scope == sk.own_username {
return Ok((sk.own_host.clone(), tail.to_string()));
}
Err(RouteError::SkillAlias { id: scope.to_string(), tail: tail.to_string() })
}
/// The two scope trees a bare `skills/<id>` alias may resolve in, as
/// `(agent path of the candidate, host path to probe)`. Pure: the caller checks
/// which of them exist. Ordered shared-then-own only so the ambiguity message
/// reads the same every time — neither wins.
pub fn skill_alias_candidates(&self, id: &str) -> Vec<(String, PathBuf)> {
let Some(sk) = &self.skills else { return Vec::new() };
vec![
(
format!("{SKILLS_ROOT}/{SKILLS_SHARED_SCOPE}/{id}"),
sk.shared_host.join(id),
),
(
format!("{SKILLS_ROOT}/{}/{id}", sk.own_username),
sk.own_host.join(id),
),
]
}
/// The message for a `skills/<seg>/…` that is neither a known scope nor an
/// installed skill id.
///
/// One sentence covers all three wrong doors — an invented scope, a typo'd id,
/// and another member's tree — because `UserFs` knows only its owner's username
/// and cannot tell a stranger's name from nonsense. Naming what *is* reachable,
/// including the fact that other members' skills are not, answers the question
/// behind each of them without pretending to know which one was asked.
pub fn skill_route_hint(&self, id: &str) -> String {
match &self.skills {
Some(sk) => format!(
"no skill named `{id}`. Skills live in `{SKILLS_ROOT}/{SKILLS_SHARED_SCOPE}/<id>/` \
(the group's) and `{SKILLS_ROOT}/{}/<id>/` (yours); other members' skills are \
not accessible, and `{SKILLS_ROOT}/` has no other subfolders.",
sk.own_username
),
None => format!("skills are not available in this context: {SKILLS_ROOT}/{id}"),
}
}
@@ -211,9 +412,12 @@ impl UserFs {
/// Reverse of [`to_container`](Self::to_container) for an already-absolute path:
/// map a **container-absolute** path (`/root/…`, `/root/shared/{X}/…`,
/// `/root/projects/{O}/{S}/…`) back to the agent vocabulary. Shared and project
/// mounts nest *under* `container_home`, so they are matched **first** — otherwise
/// `/root/shared/X` would strip against the home base and mis-route.
/// `/root/projects/{O}/{S}/…`, `/root/skills/…`) back to the agent vocabulary.
/// Shared, project and skill mounts nest *under* `container_home`, so they are
/// matched **first** — otherwise `/root/shared/X` would strip against the home
/// base and come back as `~/shared/X`, a spelling that routes correctly but is
/// not the canonical one the viewer keys on. Within the skills tree the two
/// scopes are matched before the root, which is their prefix.
///
/// Returns `None` when `abs` lies outside every one of this user's container mounts
/// (i.e. it points outside their view) — the caller rejects it fail-closed. Purely
@@ -230,6 +434,18 @@ impl UserFs {
return Some(agent_join(&format!("projects/{}/{}", m.owner_username, m.slug), tail));
}
}
if let Some(sk) = &self.skills {
let [shared, own] = sk.container_scopes(&self.container_home);
if let Ok(tail) = abs.strip_prefix(&shared) {
return Some(agent_join(&format!("{SKILLS_ROOT}/{SKILLS_SHARED_SCOPE}"), tail));
}
if let Ok(tail) = abs.strip_prefix(&own) {
return Some(agent_join(&format!("{SKILLS_ROOT}/{}", sk.own_username), tail));
}
if let Ok(tail) = abs.strip_prefix(sk.container_root(&self.container_home)) {
return Some(agent_join(SKILLS_ROOT, tail));
}
}
abs.strip_prefix(&self.container_home)
.ok()
.map(|tail| agent_join("~", tail))
@@ -252,7 +468,7 @@ impl UserFs {
let cleaned = normalize(Path::new(strip_home_prefix(input)));
let cleaned = cleaned.to_string_lossy().replace('\\', "/");
let root = cleaned.split('/').next().unwrap_or("");
if root == "shared" || root == "projects" {
if root == "shared" || root == "projects" || root == SKILLS_ROOT {
Some(cleaned)
} else if cleaned.is_empty() {
Some("~".to_string())
@@ -326,3 +542,133 @@ fn normalize(p: &Path) -> PathBuf {
}
out
}
#[cfg(test)]
mod tests {
use super::*;
fn fs_with_skills() -> UserFs {
UserFs::new(
"u1",
PathBuf::from("/wd/homes/u1"),
"skald-u1",
PathBuf::from("/root"),
vec![],
vec![],
None,
)
.with_skills(SkillMounts {
root_host: PathBuf::from("/wd/.skills-root/u1"),
shared_host: PathBuf::from("/wd/skills"),
own_host: PathBuf::from("/wd/skills-users/u1"),
own_username: "daniele".into(),
})
}
/// The two scopes route to their own host trees, whichever way the agent spells
/// the home prefix.
#[test]
fn skill_scopes_route_to_their_trees() {
let fs = fs_with_skills();
for spelling in ["skills/shared/ics/SKILL.md", "~/skills/shared/ics/SKILL.md", "./skills/shared/ics/SKILL.md"] {
assert_eq!(
fs.host_base_and_tail(spelling).unwrap(),
(PathBuf::from("/wd/skills"), "ics/SKILL.md".to_string()),
"{spelling}"
);
}
assert_eq!(
fs.host_base_and_tail("skills/daniele/spesa/run.py").unwrap(),
(PathBuf::from("/wd/skills-users/u1"), "spesa/run.py".to_string())
);
// The root itself is the signpost mount, not the home.
assert_eq!(
fs.host_base_and_tail("skills").unwrap(),
(PathBuf::from("/wd/.skills-root/u1"), String::new())
);
}
/// An invented scope segment must never fall back to the home — that fallback is
/// what turns `skills/pippo/SKILL.md` into a real file under `homes/u1/` that no
/// indexer ever reads. It comes back as an alias candidate for the caller to
/// probe, and there is no third answer.
#[test]
fn an_unknown_scope_never_falls_back_to_the_home() {
let fs = fs_with_skills();
match fs.host_base_and_tail("skills/pippo/SKILL.md") {
Err(RouteError::SkillAlias { id, tail }) => {
assert_eq!(id, "pippo");
assert_eq!(tail, "SKILL.md");
}
other => panic!("expected an alias probe, got {other:?}"),
}
// Another member's tree lands in the same branch, and the hint says so.
match fs.host_base_and_tail("skills/serena/x/SKILL.md") {
Err(RouteError::SkillAlias { id, .. }) => {
let hint = fs.skill_route_hint(&id);
assert!(hint.contains("other members' skills are not accessible"), "{hint}");
assert!(hint.contains("skills/daniele/<id>/"), "{hint}");
}
other => panic!("expected an alias probe, got {other:?}"),
}
// Without a skills tree at all the root is still reserved, never the home.
let bare = UserFs::new("u1", PathBuf::from("/wd/homes/u1"), "c", PathBuf::from("/root"), vec![], vec![], None);
assert!(bare.host_base_and_tail("skills/shared/x").is_err());
}
/// The whole root is read-only, including the space between the two scopes and
/// including a context that has no skills tree at all.
#[test]
fn nothing_under_the_skills_root_is_writable() {
let fs = fs_with_skills();
for p in [
"skills",
"skills/README.md",
"skills/shared/ics/SKILL.md",
"skills/daniele/spesa/SKILL.md",
"skills/pippo/SKILL.md",
"~/skills/pippo/SKILL.md",
] {
assert!(!fs.can_write_to(p), "{p} should be read-only");
}
// The home around it is unaffected.
assert!(fs.can_write_to("~/notes.md"));
assert!(fs.can_write_to("skillset/notes.md"));
let bare = UserFs::new("u1", PathBuf::from("/wd/homes/u1"), "c", PathBuf::from("/root"), vec![], vec![], None);
assert!(!bare.can_write_to("skills/anything"));
}
/// The scope mounts nest inside the root mount, so they must be matched first —
/// otherwise the root (their own prefix) claims them, and the home claims all
/// three.
#[test]
fn container_paths_map_back_to_the_scope_that_owns_them() {
let fs = fs_with_skills();
assert_eq!(fs.container_to_agent(Path::new("/root/skills/shared/ics/SKILL.md")).unwrap(), "skills/shared/ics/SKILL.md");
assert_eq!(fs.container_to_agent(Path::new("/root/skills/daniele/spesa")).unwrap(), "skills/daniele/spesa");
assert_eq!(fs.container_to_agent(Path::new("/root/skills/README.md")).unwrap(), "skills/README.md");
assert_eq!(fs.container_to_agent(Path::new("/root/skills")).unwrap(), "skills");
assert_eq!(fs.container_to_agent(Path::new("/root/notes.md")).unwrap(), "~/notes.md");
// And the display form keeps the skills root rather than re-rooting on `~`.
assert_eq!(fs.to_agent_display("skills/shared/ics").unwrap(), "skills/shared/ics");
assert_eq!(fs.to_agent_display("~/skills/shared/ics").unwrap(), "skills/shared/ics");
}
/// Docker cannot create a mountpoint inside a `:ro` mount, so the root has to be
/// mounted before the two scopes that nest in it — and all three read-only.
#[test]
fn skill_mounts_are_read_only_and_root_first() {
let fs = fs_with_skills();
let mounts = fs.mounts();
let skills: Vec<_> = mounts
.iter()
.filter(|(_, container, _)| container.starts_with("/root/skills"))
.collect();
assert_eq!(skills.len(), 3);
assert_eq!(skills[0].1, PathBuf::from("/root/skills"));
assert!(skills.iter().all(|(_, _, writable)| !writable), "{skills:?}");
assert!(skills.iter().any(|(_, c, _)| c == Path::new("/root/skills/shared")));
assert!(skills.iter().any(|(_, c, _)| c == Path::new("/root/skills/daniele")));
}
}
+57 -10
View File
@@ -64,8 +64,6 @@ struct RawMeta {
/// Required: declares the agent's role. A `meta.json` without `type` fails to load.
#[serde(rename = "type")]
agent_type: AgentType,
#[serde(default = "default_true")]
inject_skills: bool,
#[serde(default)]
icon: Option<String>,
#[serde(default = "default_true")]
@@ -113,12 +111,6 @@ pub struct AgentMeta {
/// runnable as a task root; `chat` and `system` are excluded from those paths.
#[serde(rename = "type")]
pub agent_type: AgentType,
/// When true (the default, including when the key is absent), the skills index
/// (`skills/index.md`) is injected into this agent's system prompt so it can
/// discover and use installed skills. Set false for background agents that don't
/// need them (e.g. event triage) to save tokens.
#[serde(default = "default_true")]
pub inject_skills: bool,
/// Path to the agent's icon image file (relative to the agent's directory).
/// Defaults to None if no icon is configured.
#[serde(default)]
@@ -208,7 +200,6 @@ pub fn discover() -> Result<Vec<AgentMeta>> {
client: raw.client,
strength: raw.strength,
agent_type: raw.agent_type,
inject_skills: raw.inject_skills,
icon: raw.icon,
allow_tools: raw.allow_tools,
};
@@ -241,7 +232,6 @@ pub fn load_meta(agent_id: &str) -> Result<AgentMeta> {
client: raw.client,
strength: raw.strength,
agent_type: raw.agent_type,
inject_skills: raw.inject_skills,
icon: raw.icon,
allow_tools: raw.allow_tools,
})
@@ -353,4 +343,61 @@ mod tests {
}
assert!(checked > 0, "no agent meta.json found under {}", root.display());
}
/// The skills index is opt-in through `<!-- SKILLS_LIST -->` (normally the
/// `common/skills.md` include), so the decision "who sees the skills" is now
/// eleven lines in eleven files rather than one default in the code — and a
/// line in a file rots in silence. This is what stops it.
///
/// The rule it holds is the one from the design: whoever **does the work**
/// gets the index, so `chat` and `task` agents both do (in a delegation the
/// worker is the child; an index injected only in the parent would leave it
/// knowing a procedure exists and handing the job to someone who cannot read
/// it). A `system` agent never does: its turns are unattended, its approvals
/// auto-denied, and some run with no tools at all — an imperative "you MUST
/// read its SKILL.md with read_file" would name a tool that isn't there.
///
/// Reads the **repo's** `agents/`, not the cwd one, which under `cargo test`
/// holds the projection fixtures.
#[test]
fn every_agent_that_does_the_work_carries_the_skills_include() {
let root = Path::new(env!("CARGO_MANIFEST_DIR"))
.join("../..")
.join(AGENTS_DIR);
let dir = std::fs::read_dir(&root)
.unwrap_or_else(|e| panic!("cannot read {}: {e}", root.display()));
let mut with = 0;
let mut without = 0;
for entry in dir {
let path = entry.expect("readable dir entry").path();
let Some(id) = path.file_name().and_then(|n| n.to_str()) else { continue };
if !path.is_dir() || id == "common" {
continue;
}
let (meta_path, prompt_path) = (path.join("meta.json"), path.join("AGENT.md"));
if !meta_path.exists() || !prompt_path.exists() {
continue;
}
let raw: RawMeta = serde_json::from_str(
&std::fs::read_to_string(&meta_path).expect("readable meta.json"),
)
.expect("valid meta.json");
let prompt = std::fs::read_to_string(&prompt_path).expect("readable AGENT.md");
let has = prompt.contains("<!-- INCLUDE: common/skills.md -->")
|| prompt.contains("<!-- SKILLS_LIST -->");
match raw.agent_type {
AgentType::System => {
assert!(!has, "system agent `{id}` must not be given the skills index");
without += 1;
}
AgentType::Chat | AgentType::Task => {
assert!(has, "agent `{id}` is missing `<!-- INCLUDE: common/skills.md -->`");
with += 1;
}
}
}
assert!(with > 0 && without > 0, "roster looks wrong: {with} with, {without} without");
}
}
+22 -3
View File
@@ -355,6 +355,9 @@ impl ApprovalManager {
/// is evaluated first: the audit trail must always be writable, and `append_file` is
/// the one write tool that cannot shorten a file.
/// - `data/*` → **allow** (scratch/data workspace).
/// - `skills/*` → reads **allow** (`@fs_read`): the trust decision on a skill is
/// taken at installation, not at each read. There is no write counterpart —
/// the whole tree is read-only in both directions (blueprint §9).
/// - `memory_search` → **allow**, path-less: it searches note *content* (arg `query`,
/// not `path`), so it needs a tool-scoped rule rather than a path pattern.
///
@@ -381,6 +384,15 @@ impl ApprovalManager {
("@fs_read", Some("shared-memory/*"), "allow", "auto-allow read shared-memory/", 5),
("@fs_write", Some("shared-memory/*"), "require", "require write shared-memory/", 5),
("@fs_any", Some("data/*"), "allow", "auto-allow data/", 5),
// The skills tree (blueprint §7.2): reading a skill must never raise a
// card. The trust decision was taken when it was *installed* — the
// `skill_register` card — exactly as a connector is trusted at
// activation and not at each call. Read-only is enforced by the mount
// and by `UserFs::can_write_to`, so there is no write rule to pair
// with this one; today `RunContext::is_read_allowed` would already
// allow it, and this row is what keeps that true if the working
// directory ever narrows (the binary-first direction).
("@fs_read", Some("skills/*"), "allow", "auto-allow read skills/", 5),
// Project folders (`projects/{owner}/{slug}`, blueprint §6): reads + writes
// frictionless, matching the working-project UX. A read-only member's mount
// is `:ro`, so a write physically fails regardless of this allow.
@@ -1285,15 +1297,16 @@ mod tests {
.unwrap();
assert_eq!(legacy, 0, "legacy fs rules should be removed by migration");
// …and replaced by exactly the five @fs_* token rows (shared-memory has two:
// read-allow and write-require; plus user-memory, data, and projects).
// …and replaced by exactly the six @fs_* token rows (shared-memory has two:
// read-allow and write-require; plus user-memory, data, projects, and the
// read-only skills tree).
let fs_rows: i64 = sqlx::query_scalar(
"SELECT COUNT(*) FROM approval_rules WHERE tool_pattern LIKE '@fs%'",
)
.fetch_one(db.as_ref())
.await
.unwrap();
assert_eq!(fs_rows, 5, "user-memory + shared-memory(r/w) + data + projects @fs_* rules should be seeded");
assert_eq!(fs_rows, 6, "user-memory + shared-memory(r/w) + data + projects + skills @fs_* rules should be seeded");
// Gate decisions through the real check() path.
async fn decide(mgr: &ApprovalManager, tool: &str, path: &str) -> GateResult {
@@ -1306,6 +1319,12 @@ mod tests {
// shared-memory: reads allowed, writes require approval.
assert!(matches!(decide(&mgr, "read_file", "shared-memory/casa.md").await, GateResult::Allow));
assert!(matches!(decide(&mgr, "write_file", "shared-memory/casa.md").await, GateResult::Require));
// Reading a skill never raises a card: the trust decision was taken when it
// was installed. A write does not need a rule — the tree is read-only in
// both directions — so it simply falls through to the catch-all.
assert!(matches!(decide(&mgr, "read_file", "skills/shared/ics/SKILL.md").await, GateResult::Allow));
assert!(matches!(decide(&mgr, "list_files", "skills/daniele").await, GateResult::Allow));
assert!(matches!(decide(&mgr, "write_file", "skills/shared/ics/SKILL.md").await, GateResult::Require));
assert!(matches!(decide(&mgr, "edit_file", "shared-memory/casa.md").await, GateResult::Require));
// The shared audit log is the one exception, and only for `append_file` — the
// one write tool that cannot shorten a file. Its lower priority number must
+195 -9
View File
@@ -5,9 +5,10 @@
//! user is created and started at application boot; `execute_cmd` and — later —
//! the user's stateful MCP servers run inside it, against the user's bind-mounted
//! home (`{WD}/homes/{userid}` → `/root`) plus the shared folders they belong to,
//! plus the read-only `{WD}/docs` bundle mounted at `/root/docs` for every user and
//! plus the read-only `{WD}/docs` bundle mounted at `/root/docs` for every user,
//! the read-only memory **signposts** at `/root/{user,shared}-memory` (see
//! [`signpost_mounts`]).
//! [`signpost_mounts`]) and the read-only skills tree at `/root/skills` (see
//! [`ensure_skills_root`]).
//!
//! Docker is a **hard requirement**: [`ContainerManager::check_docker`] fails
//! construction if the daemon is unreachable, and the shell exits at boot.
@@ -28,7 +29,7 @@ use std::time::Duration;
use anyhow::{bail, Context, Result};
use sqlx::SqlitePool;
use core_api::user_fs::{ProjectMount, SharedMount, UserFs};
use core_api::user_fs::{ProjectMount, SharedMount, SkillMounts, UserFs};
use crate::db;
use crate::tools::fs as fs_tools;
@@ -60,6 +61,18 @@ pub const DOCS_DIR: &str = "docs";
/// Subdirectory of the working directory holding the memory **signposts** — see
/// [`signpost_mounts`]. Dot-prefixed: it is internal plumbing, not a user folder.
pub const SIGNPOST_DIR: &str = ".memory-signpost";
/// Subdirectory of the working directory holding the **group's** skills
/// (`{WD}/skills/<id>`), mounted read-only at `{container_home}/skills/shared`.
pub const SKILLS_DIR: &str = "skills";
/// Subdirectory of the working directory holding each member's **own** skills
/// (`{WD}/skills-users/{userid}/<id>`). Outside the home on purpose: a skill is an
/// installed artefact, not a working file, so it must not show up in a home listing
/// nor vanish with a cleanup of one — and keeping the two scopes side by side means
/// the code that manages them handles one shape of path, not two.
pub const SKILLS_USERS_DIR: &str = "skills-users";
/// Subdirectory of the working directory holding each member's skills-root mount —
/// see [`ensure_skills_root`]. Dot-prefixed like [`SIGNPOST_DIR`]: plumbing.
pub const SKILLS_ROOT_DIR: &str = ".skills-root";
/// Home mount point inside the container.
pub const CONTAINER_HOME: &str = "/root";
/// Grace window `docker stop` gives in-container processes (SIGTERM → SIGKILL)
@@ -103,6 +116,11 @@ pub async fn build_user_fs(system: &SqlitePool, user_id: &str) -> Result<UserFs>
let home_host = wd.join(HOMES_DIR).join(user_id);
let container_home = PathBuf::from(CONTAINER_HOME);
// The skills tree needs the owner's **username**, because that is the agent-visible
// segment of their own scope (`skills/{username}/<id>`), while the host path keys on
// the stable userid — the same split `projects/{owner_username}/{slug}` already makes.
let username = db::users::get(system, user_id).await?.map(|u| u.username);
let memberships = db::shared_folders::list_for_user(system, user_id).await?;
let shared = memberships
.into_iter()
@@ -133,7 +151,28 @@ pub async fn build_user_fs(system: &SqlitePool, user_id: &str) -> Result<UserFs>
let docs_host = Some(wd.join(DOCS_DIR));
Ok(UserFs::new(user_id, home_host, container_name(user_id), container_home, shared, projects, docs_host))
let fs = UserFs::new(user_id, home_host, container_name(user_id), container_home, shared, projects, docs_host);
match username {
Some(own_username) => Ok(fs.with_skills(SkillMounts {
root_host: skills_root_host(&wd, user_id),
shared_host: wd.join(SKILLS_DIR),
own_host: wd.join(SKILLS_USERS_DIR).join(user_id),
own_username,
})),
// No directory row: nothing to name the own scope with, so the tree stays
// absent rather than half-built. `skills/…` then refuses outright, which is
// the honest answer — and the only caller that can reach this is one asking
// for a user who does not exist.
None => {
tracing::warn!(user = %user_id, "no user row: building a UserFs without the skills tree");
Ok(fs)
}
}
}
/// The host directory backing a user's skills-**root** mount.
pub fn skills_root_host(wd: &Path, user_id: &str) -> PathBuf {
wd.join(SKILLS_ROOT_DIR).join(user_id)
}
// ── Memory signposts ──────────────────────────────────────────────────────────
@@ -241,6 +280,91 @@ fn ensure_signposts(wd: &Path) -> Result<()> {
Ok(())
}
// ── The skills root ───────────────────────────────────────────────────────────
//
// `skills/` is a read-only tree with two scopes below it — `skills/shared/<id>`
// (the group's) and `skills/{username}/<id>` (the member's own). Mounting only
// those two would leave the space *between* them open, and that gap is where a
// model writes: it invents a scope segment, `mkdir -p ~/skills/pippo` succeeds
// inside the writable home mount, and the folder appears right next to the two
// read-only ones as if it had worked. That is the memory-signpost failure again,
// so the answer is the same — the root itself is a read-only mount.
//
// Its source directory is per-**user** and not one instance-wide dir, for a reason
// Docker decides rather than us: a bind mount cannot create its own mountpoint
// inside a `:ro` mount (`mkdirat … read-only file system`, at container create), so
// `shared/` and `{username}/` must already exist in the root's source — and one of
// those two names is the member's.
//
// The root also carries the README, which makes the sign and the lock the same
// object: they cannot drift apart, because there is only one of them.
/// The signpost text at `skills/README.md`. In English, like everything the agent
/// reads. It explains the *shape* of the tree and where the door is, because with
/// the whole root read-only the first `echo > skills/mine/x/SKILL.md` returns
/// "read-only file system" — an error, not an instruction, and a model answers an
/// error by reaching for `sudo` (which cannot help: `:ro` needs `CAP_SYS_ADMIN` to
/// undo, and the container has none).
const SKILLS_ROOT_SIGNPOST: &str = "\
# Skills
Two subfolders, and they are the only two:
shared/ skills installed for the whole group
<username>/ your own skills (only yours are here other members' are not visible)
Each skill is a folder with a `SKILL.md` inside it, plus whatever scripts and
reference files that file mentions. Read one with `read_file`; run its scripts with
`execute_cmd`, setting `workdir` to the skill's own folder.
**This whole tree is read-only**, including this directory. You cannot create a
skill by writing here, and `sudo` will not change that. A skill is written somewhere
you can write your home, a project and then *installed* from there:
activate_tools([\"config\"]) then
skill_register(scope, path) scope: \"mine\" or \"global\"
Read `docs/skills.md` before writing one; it holds the authoring contract.
Anything a skill needs to write (caches, state, dependencies) goes in your home or
`/tmp`, never next to the skill.
";
/// Creates a user's skills-root mount source and (re)writes its contents: the
/// README plus the two empty directories the scope mounts land on. Unconditional,
/// like [`ensure_signposts`] — a few hundred bytes at every container `ensure`, so
/// an edited text reaches existing installations with no migration step.
///
/// It also **prunes** any other entry: after a rename the previous username would
/// otherwise stay behind as an empty directory and show up in `ls skills/` as a
/// scope that leads nowhere.
fn ensure_skills_root(wd: &Path, user_id: &str, own_username: &str) -> Result<()> {
let root = skills_root_host(wd, user_id);
std::fs::create_dir_all(&root)
.with_context(|| format!("failed to create skills root {}", root.display()))?;
std::fs::write(root.join(SIGNPOST_README), SKILLS_ROOT_SIGNPOST)
.with_context(|| format!("failed to write skills signpost in {}", root.display()))?;
let keep = [core_api::user_fs::SKILLS_SHARED_SCOPE, own_username];
for name in keep {
std::fs::create_dir_all(root.join(name))
.with_context(|| format!("failed to create skills mountpoint {name}"))?;
}
if let Ok(entries) = std::fs::read_dir(&root) {
for entry in entries.flatten() {
let name = entry.file_name();
let name = name.to_string_lossy();
if name == SIGNPOST_README || keep.contains(&name.as_ref()) {
continue;
}
// Only ever an empty leftover mountpoint: the real content lives in the
// trees these directories are mounted *from*, never in here.
let _ = std::fs::remove_dir(entry.path());
}
}
Ok(())
}
/// Owns the container lifecycle: the docker availability check, the runtime image,
/// and per-user create/start/stop/remove. Cheap to clone (holds an `Arc` pool).
#[derive(Clone)]
@@ -327,6 +451,11 @@ impl ContainerManager {
.with_context(|| format!("failed to create host dir {}", host.display()))?;
}
ensure_signposts(&wd)?;
// After the mount dirs, because the two scope mountpoints it creates live
// *inside* the root dir the loop above just made.
if let Some(sk) = &fs.skills {
ensure_skills_root(&wd, user_id, &sk.own_username)?;
}
let name = &fs.container_name;
let want_user = host_uid_gid().map(|(uid, gid)| format!("{uid}:{gid}"));
@@ -334,8 +463,8 @@ impl ContainerManager {
match container_state(name).await {
// Reuse only if it runs as the expected user AND has tini as PID 1;
// otherwise recreate below.
ContainerState::Running if reusable(name, &want_user).await => return Ok(()),
ContainerState::Stopped if reusable(name, &want_user).await => {
ContainerState::Running if reusable(name, &want_user, &fs).await => return Ok(()),
ContainerState::Stopped if reusable(name, &want_user, &fs).await => {
docker(&["start", name]).await.context("docker start failed")?;
return Ok(());
}
@@ -547,14 +676,35 @@ async fn signposts_mounted(name: &str) -> bool {
.all(|(_, container)| dests.iter().any(|d| Path::new(d) == container))
}
/// Whether a container carries all three skills mounts (root + the two scopes).
/// The fifth self-heal axis, and an [`IMAGE_TAG`] bump for the same reason as the
/// signposts: the image is unchanged, so a bump would make every installation
/// rebuild it just to fix a mount. Without this check an existing container keeps a
/// writable `~/skills` — a directory the shell can create folders in that no reader
/// ever visits. Unreadable inspect ⇒ `true`, so a docker hiccup never churns a
/// working container.
async fn skills_mounted(name: &str, fs: &UserFs) -> bool {
let Some(sk) = &fs.skills else { return true };
let Ok(out) = docker(&["inspect", "-f", "{{range .Mounts}}{{println .Destination}}{{end}}", name]).await
else {
return true;
};
let dests: Vec<&str> = out.lines().map(str::trim).collect();
let [shared, own] = sk.container_scopes(&fs.container_home);
[sk.container_root(&fs.container_home), shared, own]
.iter()
.all(|want| dests.iter().any(|d| Path::new(d) == want))
}
/// Whether an existing container can be reused as-is: right `--user` (§6 UID coherence),
/// `--init` (fast, clean `docker stop`), the current image **and** the memory signpost
/// mounts. A mismatch on any of the four recreates it.
async fn reusable(name: &str, want_user: &Option<String>) -> bool {
/// `--init` (fast, clean `docker stop`), the current image, the memory signpost mounts
/// **and** the skills mounts. A mismatch on any of the five recreates it.
async fn reusable(name: &str, want_user: &Option<String>, fs: &UserFs) -> bool {
user_matches(name, want_user).await
&& init_matches(name).await
&& image_matches(name).await
&& signposts_mounted(name).await
&& skills_mounted(name, fs).await
}
/// Gives the container's runtime `uid`/`gid` a passwd + shadow (+ group) entry, so
@@ -610,3 +760,39 @@ async fn docker_ok(args: &[&str]) -> bool {
.map(|s| s.success())
.unwrap_or(false)
}
#[cfg(test)]
mod tests {
use super::*;
/// The root mount's source has to carry the two scope mountpoints, because
/// Docker cannot create them itself inside a `:ro` mount — and it must carry
/// *only* those, or a stale one left by a rename shows up in `ls skills/` as a
/// scope that leads nowhere.
#[test]
fn skills_root_holds_the_signpost_and_exactly_two_mountpoints() {
let wd = std::env::temp_dir().join(format!("skald-skroot-{}", std::process::id()));
let _ = std::fs::remove_dir_all(&wd);
let root = skills_root_host(&wd, "u1");
ensure_skills_root(&wd, "u1", "daniele").unwrap();
assert!(root.join(SIGNPOST_README).is_file());
assert!(root.join("shared").is_dir());
assert!(root.join("daniele").is_dir());
// Idempotent, and a leftover scope directory is pruned on the next pass.
std::fs::create_dir_all(root.join("stale")).unwrap();
ensure_skills_root(&wd, "u1", "daniele").unwrap();
assert!(!root.join("stale").exists(), "a stale mountpoint survived");
let mut names: Vec<String> = std::fs::read_dir(&root)
.unwrap()
.flatten()
.map(|e| e.file_name().to_string_lossy().into_owned())
.collect();
names.sort();
assert_eq!(names, vec!["README.md", "daniele", "shared"]);
let _ = std::fs::remove_dir_all(&wd);
}
}
@@ -36,6 +36,19 @@ pub const MANAGE_SHARED_FOLDERS: &str = "folders.manage";
/// pattern as [`MANAGE_SHARED_FOLDERS`].
pub const MANAGE_PLUGINS: &str = "plugin.manage";
/// Install or delete a skill in the **group's** tree — `skill_register`/
/// `skill_delete` with `scope: "global"` (blueprint §7.3/§9). One's own scope
/// needs no capability: it is the caller's, always.
///
/// Deliberately **not** in [`DEFAULT_USER_CAPABILITIES`], unlike the two
/// self-service MCP ones, and the asymmetry is the point: a global skill is text
/// that enters every member's prompt and is read there as an instruction, so it
/// is closer to curating the catalog than to activating a connector for oneself.
/// `admin` therefore holds it implicitly (via [`has`]) and opening it to another
/// role later is a single [`grant`], no code change — the same shape as
/// [`MANAGE_SHARED_FOLDERS`] and [`MANAGE_PLUGINS`].
pub const MANAGE_SKILLS: &str = "skill.manage";
/// The default capabilities of an ordinary (non-admin) user role.
pub const DEFAULT_USER_CAPABILITIES: &[&str] = &[REGISTER_REMOTE, REGISTER_LOCAL_FROM_CATALOG];
+1
View File
@@ -42,6 +42,7 @@ pub mod secrets;
pub mod service_manager;
pub mod session;
pub mod setup;
pub mod skills;
pub mod system_agents;
pub mod event_triage;
pub mod tool_catalog;
@@ -196,7 +196,7 @@ impl SkaldToolActivator {
tool_prefix: None,
tool_count: self.config_defs.len(),
description: Some(
"Built-in system-configuration tools: connectors, plugins, scheduled jobs, secrets."
"Built-in system-configuration tools: connectors, plugins, scheduled jobs, secrets, installing and deleting skills."
.into(),
),
message: format!("Tools are in context for {} from the next round.", self.scope_label()),
@@ -137,6 +137,11 @@ impl AgentCatalog for SkaldAgentCatalog {
shared_pool: self.shared_pool.clone(),
user_id: self.user_id.clone(),
mcp: self.mcp.clone(),
// A sub-agent sees the same skills its parent does: in a delegation
// the one doing the work is the child, so an index injected only in
// the parent would leave it knowing a procedure exists and handing
// the job to someone who cannot read it.
fs: self.fs.clone(),
project_root: scope.project_root.clone(),
// The scratchpad is the session's blackboard: a sub-agent reads and
// writes the SAME one as its parent.
@@ -123,6 +123,29 @@ impl ApprovalGate {
}
}
/// Builds the review card for a pending `skill_register`: the destination's
/// agent path, the installed body if this replaces one, and the candidate's
/// own `SKILL.md`.
///
/// Resolved through the caller's own `UserFs`, like every other path the gate
/// touches, so a source that lives only in the container (`/tmp/…`) yields
/// `None` here and a spoken refusal from the tool.
async fn skill_registration_preview(
&self,
args: &serde_json::Value,
) -> Option<(String, Option<String>, String)> {
use crate::skills::{Scope, install};
use crate::tools::fs::{FsTarget, resolve_target};
let scope = Scope::parse(args["scope"].as_str()?).ok()?;
let fs = self.fs.as_ref()?.load();
let host = match resolve_target(&fs, args["path"].as_str()?).ok()? {
FsTarget::Host(p) => p,
FsTarget::Container { .. } => return None,
};
install::preview(&fs, scope, &host)
}
/// Emits the approval event for the tool kind: `PendingWrite` (via
/// `LoopEvent::Host`) for file-write tools and `execute_cmd`,
/// `ApprovalRequired` otherwise (port of `emit_approval_event`).
@@ -150,6 +173,31 @@ impl ApprovalGate {
})));
return;
}
} else if name == tn::SKILL_REGISTER {
// The review moment of the whole design (blueprint §9.1): for the
// group's scope this is the *only* time a person reads a text that
// will enter everybody's prompt. So the card carries the candidate's
// `SKILL.md` in full — not its name, not a summary — with a header
// naming the scope, the file list and whether it replaces something;
// on a replacement the installed body goes in as `old_content`, and
// the existing diff renderer turns the card into a review of what
// actually changes. Reusing `pending_write` is what makes that free:
// no new event, no new frontend, exactly as `execute_cmd` below.
if let Some(preview) = self.skill_registration_preview(&call.args).await {
let (path, old_content, new_content) = preview;
events.emit(call.frame, call.parent_frame, LoopEvent::Host(serde_json::json!({
"type": "pending_write",
"request_id": request_id,
"tool_call_id": call.id.get(),
"path": path,
"old_content": old_content,
"new_content": new_content,
})));
return;
}
// Unreadable or invalid source: fall through to the plain card. The
// tool refuses it a moment later with a message that says why, and a
// half-built preview would only make the refusal look like a bug.
} else if name == tn::EXECUTE_CMD {
let cmd = call.args["command"].as_str().unwrap_or("");
events.emit(call.frame, call.parent_frame, LoopEvent::Host(serde_json::json!({
@@ -102,6 +102,25 @@ impl PrefixCache {
entries.retain(|_, e| e.last_used.elapsed() < self.ttl);
entries.insert(key, Entry { base, last_used: Instant::now() });
}
/// Drops every frozen prefix, so the next round of every conversation
/// rebuilds one.
///
/// The single exception to "writes are deliberately not reacted to" above,
/// and it is narrow on purpose. The rule holds for a file the prompt merely
/// *injects*: the agent that edited it already has the new text two messages
/// downstream, and a rebuild would repeat what it just said. It does not hold
/// for the **skills index**, which is not content but a *catalogue*: an admin
/// who installs a skill and immediately asks for it would be told for twenty
/// minutes that it does not exist — the prefix is not stale, it is wrong.
///
/// Coarse by design. A per-key flush would need to know which conversations
/// carry an agent whose prompt includes the index, which is a question about
/// eleven `AGENT.md` files; installing a skill is rare enough that rebuilding
/// a handful of prefixes once is cheaper than keeping that answer correct.
pub fn clear(&self) {
self.entries.lock().unwrap().clear();
}
}
impl Default for PrefixCache {
@@ -224,6 +224,14 @@ impl UserLoopRuntime {
&self.store
}
/// Drops this user's frozen system prefixes — see [`PrefixCache::clear`].
/// Called when the **skills index** they would carry has changed, which is
/// the one case where waiting for the idle window would have the model deny
/// that something exists.
pub fn invalidate_prefixes(&self) {
self.prefix_cache.clear();
}
/// Where this user's LLM traffic is logged: metadata in the registry
/// (attributed to them), payloads in their own encrypted pool.
pub fn log_target(&self) -> RequestLogTarget {
@@ -251,6 +259,7 @@ impl UserLoopRuntime {
shared_pool: self.shared_pool.clone(),
user_id: self.user_id.clone(),
mcp: self.mcp.clone(),
fs: self.fs.clone(),
project_root: scope.project_root.clone(),
scratchpad_sid: scope.scratchpad_sid,
datetime: self.config.datetime.clone(),
+173 -17
View File
@@ -4,7 +4,7 @@
//!
//! | layer | wire position |
//! |---|---|
//! | AGENT.md + `inject_memory` + skills index + `extra_system` + substitutions | `base` — the cacheable prefix |
//! | AGENT.md + `inject_memory` + `extra_system` + substitutions | `base` — the cacheable prefix |
//! | session scratchpad | `extra_static` — a system message before the conversation |
//! | Honcho memory / per-turn overrides, then the date/time block | `dynamic_tail` — joined into the trailing system message |
//! | trailing reminder | `tail_reminder` |
@@ -13,16 +13,13 @@ use std::collections::HashMap;
use std::sync::Arc;
use agent_loop::context::{SystemContext, SystemContextSource, TurnInfo};
use core_api::user_fs::SharedFs;
use sqlx::SqlitePool;
use crate::config::DatetimeConfig;
use crate::loop_adapters::prefix_cache::PrefixCache;
use crate::mcp::McpProvider;
/// Registry of installed skills, relative to Skald's process cwd. Injected
/// into agents that have `inject_skills` enabled (the default).
const SKILLS_INDEX_PATH: &str = "skills/index.md";
/// The static system content of one agent, resolved per turn.
pub struct AgentSystemContext {
pub agent_id: String,
@@ -39,6 +36,12 @@ pub struct AgentSystemContext {
pub shared_pool: Arc<SqlitePool>,
pub user_id: String,
pub mcp: Arc<dyn McpProvider>,
/// The caller's filesystem view — read here for one thing only, the skills
/// index: `UserFs` already *is* the answer to "which skills can this user
/// see", so reading the two trees off it avoids a second source of truth.
/// The swappable cell rather than a snapshot, so a §6 remount is picked up
/// at the next prefix rebuild.
pub fs: SharedFs,
/// Project root for `__PROJECT_ROOT__` expansion in `inject_memory`.
pub project_root: Option<String>,
/// Scratchpad scope: the session's own id, or the parent's for an async
@@ -145,18 +148,6 @@ impl AgentSystemContext {
}
}
// Skills index — injected unless the agent opts out. Skipped silently
// when no skills are installed.
if meta.inject_skills {
let (abs, display) = self.resolve_memory_path(SKILLS_INDEX_PATH);
if let Ok(c) = tokio::fs::read_to_string(&abs).await {
static_content.push_str(&format!(
"\n\n---\nInstalled skills you can use (read the linked `SKILL.md` before running a skill):\n\
\n<skills_index path=\"{display}\">\n{c}\n</skills_index>\n"
));
}
}
if let Some(extra) = &self.extra_static {
static_content.push_str("\n\n---\n");
static_content.push_str(extra);
@@ -165,6 +156,17 @@ impl AgentSystemContext {
if static_content.contains("__MCP_LIST__") {
static_content = static_content.replace("__MCP_LIST__", &self.render_mcp_list());
}
// The sentinel *is* the knob: an agent gets the skills index iff its
// `AGENT.md` carries `<!-- SKILLS_LIST -->` (normally through
// `common/skills.md`). There is no `meta.json` flag — two mechanisms for
// one question is one too many, and the system agents opt out simply by
// not including the fragment.
if static_content.contains("__SKILLS_LIST__") {
static_content = static_content.replace(
"__SKILLS_LIST__",
&crate::skills::render_index(&self.fs.load()),
);
}
if static_content.contains("__SHARED_FOLDERS__") {
static_content = static_content.replace(
"__SHARED_FOLDERS__",
@@ -533,6 +535,160 @@ fn resolve_harness_tag(content: String) -> String {
mod tests {
use super::*;
// ── The skills index, as it reaches (or does not reach) the prompt ───────
//
// These exercise `build_base` rather than the renderer, because the failure
// they exist for is a wiring one: a sentinel with no substitution behind it
// survives **textually** into the system prompt, and `build_base` replaces
// only the keys it knows about.
/// One `agents/<id>/` with the prompt a case needs. Separate from the
/// projection testkit's fixture, which is frozen for the snapshots.
struct PromptFixture {
id: String,
dir: std::path::PathBuf,
}
impl PromptFixture {
fn new(prompt: &str) -> Self {
let id = format!(
"skills-prompt-{}-{:?}",
std::process::id(),
std::thread::current().id()
);
let dir = std::path::Path::new("agents").join(&id);
std::fs::create_dir_all(&dir).unwrap();
std::fs::write(dir.join("AGENT.md"), prompt).unwrap();
std::fs::write(
dir.join("meta.json"),
r#"{"name":"Fixture","description":"skills injection","type":"task"}"#,
)
.unwrap();
Self { id, dir }
}
}
impl Drop for PromptFixture {
fn drop(&mut self) {
let _ = std::fs::remove_dir_all(&self.dir);
}
}
/// A `{WD}` with a group-wide skill in it, plus the matching `UserFs`.
struct SkillsTree {
root: std::path::PathBuf,
fs: core_api::user_fs::UserFs,
}
impl SkillsTree {
fn new(skills: &[(&str, &str)]) -> Self {
use core_api::user_fs::{SkillMounts, UserFs};
let root = std::env::temp_dir().join(format!(
"skald-index-{}-{:?}",
std::process::id(),
std::thread::current().id()
));
let _ = std::fs::remove_dir_all(&root);
let shared = root.join("skills");
let own = root.join("skills-users").join("u1");
std::fs::create_dir_all(&shared).unwrap();
std::fs::create_dir_all(&own).unwrap();
for (id, description) in skills {
let dir = shared.join(id);
std::fs::create_dir_all(&dir).unwrap();
std::fs::write(
dir.join("SKILL.md"),
format!("---\nname: {id}\ndescription: {description}\n---\n\nBody.\n"),
)
.unwrap();
}
let fs = UserFs::new(
"u1",
root.join("homes").join("u1"),
"skald-u1",
std::path::PathBuf::from("/root"),
vec![],
vec![],
None,
)
.with_skills(SkillMounts {
root_host: root.join(".skills-root").join("u1"),
shared_host: shared,
own_host: own,
own_username: "daniele".into(),
});
Self { root, fs }
}
}
impl Drop for SkillsTree {
fn drop(&mut self) {
let _ = std::fs::remove_dir_all(&self.root);
}
}
async fn base_of(agent_id: &str, fs: core_api::user_fs::UserFs) -> String {
let pool = Arc::new(SqlitePool::connect("sqlite::memory:").await.unwrap());
AgentSystemContext {
agent_id: agent_id.to_string(),
extra_static: None,
extra_dynamic: None,
tail_reminder: None,
substitutions: HashMap::new(),
pool: pool.clone(),
shared_pool: pool,
user_id: "u1".into(),
mcp: crate::loop_adapters::testkit::mcp(),
fs: SharedFs::new(fs),
project_root: None,
scratchpad_sid: 1,
datetime: DatetimeConfig { enabled: false, timezone: None },
prefix_cache: Arc::new(crate::loop_adapters::prefix_cache::PrefixCache::new()),
}
.build_base()
.await
.unwrap()
}
/// The sentinel is the knob: a prompt carrying it gets the index, and the
/// sentinel itself never survives into the prompt.
#[tokio::test]
async fn a_prompt_with_the_sentinel_gets_the_index() {
let agent = PromptFixture::new("You are a fixture.\n\n<!-- SKILLS_LIST -->\n");
let tree = SkillsTree::new(&[("ics-import", "Import an iCalendar feed.")]);
let base = base_of(&agent.id, tree.fs.clone()).await;
assert!(base.contains("## Skills (mandatory)"), "{base}");
assert!(base.contains("skills/shared/ics-import/SKILL.md"), "{base}");
assert!(base.contains("Import an iCalendar feed."), "{base}");
assert!(!base.contains("__SKILLS_LIST__"), "sentinel survived: {base}");
}
/// A prompt without the sentinel is byte-identical to what it was before the
/// feature existed — which is how the four `type: system` agents opt out.
#[tokio::test]
async fn a_prompt_without_the_sentinel_is_untouched() {
let agent = PromptFixture::new("You are a fixture.\n");
let tree = SkillsTree::new(&[("ics-import", "Import an iCalendar feed.")]);
let base = base_of(&agent.id, tree.fs.clone()).await;
assert_eq!(base, "You are a fixture.\n");
}
/// Nothing installed ⇒ the sentinel resolves to **nothing**: no header, no
/// orphan sentence promising a list that isn't there. That promise is what
/// once had the model inventing a discovery tool for the MCP section.
#[tokio::test]
async fn with_no_skills_the_sentinel_resolves_to_nothing() {
let agent = PromptFixture::new("Before.\n\n<!-- SKILLS_LIST -->\n\nAfter.\n");
let tree = SkillsTree::new(&[]);
let base = base_of(&agent.id, tree.fs.clone()).await;
assert_eq!(base, "Before.\n\n\n\nAfter.\n");
assert!(!base.contains("__SKILLS_LIST__"), "{base}");
assert!(!base.to_lowercase().contains("skill"), "{base}");
}
#[test]
fn harness_tag_resolves_to_canonical_tag() {
// Every occurrence of the sentinel is replaced with the tag emitted by
+15 -4
View File
@@ -9,8 +9,9 @@
//! builder, the snapshots outlived it.
//!
//! Everything volatile is neutralized here rather than scrubbed afterwards:
//! the datetime block is disabled, the agent opts out of the skills index, and
//! the fixture's own identifiers never reach the wire.
//! the datetime block is disabled, the fixture's prompt carries no
//! `<!-- SKILLS_LIST -->` (so no index is rendered into it), and the fixture's
//! own identifiers never reach the wire.
#![cfg(test)]
@@ -27,7 +28,7 @@ use serde_json::{Value, json};
use sqlx::SqlitePool;
use core_api::message_meta::{Attachment, MessageMetadata};
use core_api::user_fs::UserFs;
use core_api::user_fs::{SharedFs, UserFs};
use crate::config::DatetimeConfig;
use crate::llm::DtlMode;
@@ -85,7 +86,6 @@ impl AgentFixture {
"name": "Parity fixture",
"description": "projection parity",
"type": "task",
"inject_skills": false,
})
.to_string(),
)
@@ -221,6 +221,17 @@ pub async fn project(db: &Db, agent: &AgentFixture, case: &Case) -> Vec<Value> {
shared_pool: db.pool.clone(),
user_id: "u1".into(),
mcp: mcp(),
// Skill-less by construction: the fixture's prompt has no sentinel, so
// nothing here is ever rendered from it.
fs: SharedFs::new(UserFs::new(
"u1",
PathBuf::from("/wd/homes/u1"),
"skald-u1",
PathBuf::from("/root"),
vec![],
vec![],
None,
)),
project_root: None,
scratchpad_sid: 1,
datetime: datetime(),
+2 -3
View File
@@ -17,7 +17,7 @@ pub struct RunContext {
#[serde(default)]
pub allow_fs_writes: Vec<String>,
/// Extra directories/files granted read-only access (beyond the working directory,
/// `docs/`, `skills/`, and everything in `allow_fs_writes`, which is readable too).
/// `docs/`, and everything in `allow_fs_writes`, which is readable too).
#[serde(default)]
pub allow_fs_reads: Vec<String>,
/// Project root (agent path `projects/{owner}/{slug}`) when this is a project
@@ -70,7 +70,7 @@ impl RunContext {
/// True if reading `path` is pre-authorized by this RunContext.
/// Read access is granted (no approval prompt) for: the process working directory
/// itself, its `docs/` and `skills/` subtrees (always-safe baseline), any
/// itself and its `docs/` subtree (always-safe baseline), any
/// `allow_fs_reads` entry, and anything writable (write implies read). All paths
/// are canonicalized first so `..`/symlink escapes cannot widen the grant.
///
@@ -84,7 +84,6 @@ impl RunContext {
let mut roots: Vec<std::path::PathBuf> = vec![
canonicalize_for_policy(".", &wd), // process working directory
canonicalize_for_policy("docs", &wd),
canonicalize_for_policy("skills", &wd),
];
roots.extend(self.allow_fs_reads.iter().map(|e| canonicalize_for_policy(e, &wd)));
roots.extend(self.allow_fs_writes.iter().map(|e| canonicalize_for_policy(e, &wd)));
+27
View File
@@ -196,6 +196,33 @@ impl Skald {
Ok(())
}
/// Rebuilds the frozen system prefix of the conversations a skills change made
/// wrong (blueprint §6).
///
/// The prefix is normally left alone until its conversation has been idle for
/// twenty minutes, which is right for an *injected file* and wrong for the
/// **index**: an admin who installs a skill and then asks the assistant to use
/// it would be told, at length and in good faith, that no such skill exists.
///
/// Called **directly** by the two skill tools, not through the system bus.
/// Whoever writes a skill through a tool is inside this process and can say so;
/// the bus (`SkillsChanged`) is for the other case — someone editing files on
/// the box — where nothing in-process knows. A miss here is not a lost event,
/// it is a user who is simply not logged in and whose next login builds a fresh
/// prefix anyway.
pub async fn invalidate_prompt_prefix(&self, scope: crate::skills::PromptScope) {
for ctx in self.rt_user_contexts().all_live().await {
let concerns = match &scope {
// The group's tree is in everybody's index.
crate::skills::PromptScope::Everyone => true,
crate::skills::PromptScope::User(id) => *id == ctx.user_id,
};
if concerns {
ctx.sessions.loop_runtime().invalidate_prefixes();
}
}
}
/// Refresh every live user's global-connector access set in place — call after an
/// admin enables/deletes a global connector or changes who may use it, so running
/// sessions see it without a restart (the §7 MCP twin of the §6 fs remount). The
+15
View File
@@ -237,6 +237,21 @@ impl Tools {
tool_registry.register(crate::tools::set_secret::SetSecret(Arc::clone(&models.secrets)));
tool_registry.register(crate::tools::list_secrets::ListSecrets(Arc::clone(&models.secrets)));
tool_registry.register(crate::tools::configure_plugin::ConfigurePlugin(Arc::clone(&integrations.plugin_manager)));
// The whole write surface of the read-only skills trees (blueprint §7.3):
// `Config`-category, so neither appears in a request's schema until
// `activate_tools(["config"])` asks. They take the registry to read the
// caller's role (`skill.manage` gates the group's scope) and the cell
// `Skald::new` later fills, through which an installation reaches
// conversations that are already running.
tool_registry.register(crate::tools::skills::SkillRegister::new(
Arc::clone(&rt.db), Arc::clone(&rt.prompt_prefixes)));
tool_registry.register(crate::tools::skills::SkillDelete::new(
Arc::clone(&rt.db), Arc::clone(&rt.prompt_prefixes)));
// The download half of the skills lifecycle (blueprint §7.5): same
// `Config` category, so it too stays out of every request's schema
// until `activate_tools(["config"])`. It needs no state of its own —
// the container it runs git in comes from each caller's `ToolContext`.
tool_registry.register(crate::tools::fetch_repo::FetchRepo);
// Tools contributed by plugins (plugin.md §11), via `Plugin::tools()`.
// The core never names a plugin crate: each one hands over whatever tools
+28 -1
View File
@@ -31,7 +31,7 @@ use bundles::{Conversation, Infra, Integrations, Interaction, Media, Models, Tas
use runtime::Runtime;
use user_context::{UserContextFactory, UserContextRegistry};
pub use user_context::UserContext;
use wiring::{spawn_background, spawn_system_agents, spawn_user_lifecycle, wire};
use wiring::{spawn_background, spawn_skills_freshness, spawn_system_agents, spawn_user_lifecycle, wire};
pub struct Skald {
rt: Runtime,
@@ -125,10 +125,20 @@ impl Skald {
// can only be spawned once the instance exists (blueprint §6).
spawn_user_lifecycle(&skald);
// And the same for the skills seam: the two tools were built with the cell
// during composition; only now is there an instance able to answer it. A
// `Weak`, like the reconciler's — the tools live in the registry `Skald`
// owns, so a strong handle would be a cycle.
skald.rt.prompt_prefixes.install(Arc::new(SkaldPromptPrefixes(Arc::downgrade(&skald))));
// Likewise the system-agent scheduler: it resolves a per-user runtime for
// each user it runs an agent for (blueprint §13).
spawn_system_agents(&skald);
// And the skills-freshness reactor: it reacts to the watcher's
// `SkillsChanged` through `Skald`'s own accessor, same Weak shape (§8.3).
spawn_skills_freshness(&skald);
Ok(skald)
}
@@ -160,3 +170,20 @@ impl Skald {
self.container.clone()
}
}
/// The instance, seen from a skill tool that only wants to say "the index moved".
///
/// A `Weak` rather than a strong `Arc`: the tools holding the other end live in
/// the registry `Skald` itself owns. An instance already on its way down simply
/// stops upgrading, which is the right answer — there is nothing left to keep
/// fresh.
struct SkaldPromptPrefixes(std::sync::Weak<Skald>);
#[async_trait::async_trait]
impl crate::skills::PromptPrefixes for SkaldPromptPrefixes {
async fn invalidate(&self, scope: crate::skills::PromptScope) {
if let Some(skald) = self.0.upgrade() {
skald.invalidate_prompt_prefix(scope).await;
}
}
}
+7
View File
@@ -40,6 +40,12 @@ pub(super) struct Runtime {
pub(super) global_tx: broadcast::Sender<GlobalEvent>,
pub(super) shutdown_token: CancellationToken,
pub(super) supervisor: Arc<TaskSupervisor>,
/// How a skills write reaches conversations already running (blueprint §6).
/// Held here because the two ends are built at different times: the tools
/// that fill it exist during composition, the instance that answers it only
/// afterwards — so `Skald::new` installs the reactor into this cell once it
/// has itself, exactly like the plugin manager's `set_skald`.
pub(super) prompt_prefixes: Arc<crate::skills::PromptPrefixCell>,
}
impl Runtime {
@@ -78,6 +84,7 @@ impl Runtime {
global_tx,
shutdown_token: CancellationToken::new(),
supervisor: TaskSupervisor::new(),
prompt_prefixes: Arc::new(crate::skills::PromptPrefixCell::default()),
}
}
}
+50
View File
@@ -82,6 +82,14 @@ pub(super) fn spawn_background(
}
});
}
// Skills freshness for edits made by hand on the box (blueprint §8.2). The
// in-process writers invalidate directly; this watches the two trees and
// announces `SkillsChanged` only when the digest gate says the index moved.
rt.supervisor.adopt_one(
"skills-watch",
crate::skills::watch::spawn(Arc::clone(&rt.system_bus), rt.shutdown_token.clone()),
);
}
/// Spawns the **user-lifecycle reconciler** — the single subscriber that turns
@@ -174,6 +182,48 @@ pub(super) fn spawn_user_lifecycle(skald: &Arc<super::Skald>) {
});
}
/// Spawns the **skills-freshness reactor** — the subscriber that turns a
/// `SkillsChanged` announcement into a prompt-prefix invalidation (blueprint
/// §8.3).
///
/// Why the bus at all, when the skill tools call `invalidate_prompt_prefix`
/// directly: the watcher exists for a writer *outside* the process (a hand
/// edit on the box), and its consumers live in different places — the per-user
/// loop runtimes here, a UI refresh later. A direct call would make the
/// watcher hold `Skald`, which it deliberately does not. Best-effort by
/// contract, and honestly so: a lost event costs a stale skill index for the
/// twenty minutes of the prefix TTL, never a wrong answer.
pub(super) fn spawn_skills_freshness(skald: &Arc<super::Skald>) {
let weak = Arc::downgrade(skald);
let shutdown = skald.rt.shutdown_token.clone();
let mut rx = skald.rt.system_bus.subscribe();
skald.rt.supervisor.spawn("skills-freshness", async move {
loop {
let event = tokio::select! {
_ = shutdown.cancelled() => break,
event = rx.recv() => match event {
Ok(e) => e,
Err(RecvError::Lagged(n)) => {
warn!(n, "skills-freshness: system_bus lagged; a skill index may be stale until the prefix TTL");
continue;
}
Err(RecvError::Closed) => break,
},
};
let SystemEvent::SkillsChanged { scope } = event else { continue };
let Some(skald) = weak.upgrade() else { break };
let scope = match scope {
core_api::system_bus::SkillScope::Global => crate::skills::PromptScope::Everyone,
core_api::system_bus::SkillScope::User(id) => crate::skills::PromptScope::User(id),
};
skald.invalidate_prompt_prefix(scope).await;
}
info!("skills-freshness: reactor stopped");
});
}
/// Spawns the **system-agent scheduler** — the one instance-wide timer behind
/// every background agent nobody asked for (event triage, the two memory lints).
///
+437
View File
@@ -0,0 +1,437 @@
//! The one door into the two read-only trees (blueprint §7.3/§9).
//!
//! Everything here exists because a skill is an **immutable, validated
//! artefact**: it is copied in whole or not at all, it is never edited in place,
//! and modifying one means registering it again. The tree therefore never holds
//! a half-written skill, which is what lets the index (`super::list`) read it
//! without tolerating intermediate states.
//!
//! Two mechanics carry that promise and neither is optional:
//!
//! - **Staging plus a rename**, never a copy in place, so the indexer cannot
//! observe a directory being filled.
//! - **Three steps on replacement**, because `rename` over a non-empty directory
//! fails: move the old one aside, move the new one in, delete the old. Not
//! atomic in the strict sense — but the uncovered window contains only a state
//! where the id *does not exist*, never one where it exists half-written, and
//! that is the property the indexer actually needs.
use std::path::Path;
use anyhow::{Context, Result, bail};
use core_api::user_fs::UserFs;
use serde::{Deserialize, Serialize};
use super::validate::{ValidSkill, validate_dir};
use super::{SKILL_FILE, Scope};
/// The provenance ticket a fetched skill carries, written next to its files.
///
/// The seam between two tools that deliberately know nothing about each other:
/// `fetch_repo` (blueprint §7.5, session 4) downloads without knowing what it
/// downloaded, `skill_register` installs without knowing where it came from, and
/// this file is what crosses between them. `git clone` leaves nothing traceable,
/// so without it neither "where is this skill from?" nor "has it changed
/// upstream?" has an answer later.
///
/// It is *pinning*, not authenticity — whoever serves the repository serves the
/// commit too — the same honesty the marketplace states about its digests.
pub const SOURCE_FILE: &str = ".source.json";
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct Provenance {
pub url: String,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub sub_path: Option<String>,
/// The commit the files were taken from — the field that makes a later
/// upstream change *detectable*.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub commit: Option<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub fetched_at: Option<String>,
/// Stamped by [`install`], so the ticket answers "since when is this here?"
/// as well as "where from?".
#[serde(default, skip_serializing_if = "Option::is_none")]
pub installed_at: Option<String>,
}
/// Reads the ticket a source folder carries, if it carries one. A malformed one
/// is *ignored*, never fatal: provenance is metadata about the skill, and losing
/// it must not stop an otherwise valid installation.
pub fn read_provenance(dir: &Path) -> Option<Provenance> {
let raw = std::fs::read_to_string(dir.join(SOURCE_FILE)).ok()?;
serde_json::from_str(&raw).ok()
}
/// What an installation did, for the tool's answer to the model.
pub struct Installed {
pub id: String,
/// The agent path of the installed folder (`skills/shared/ics-import`).
pub agent_dir: String,
/// Whether it took the place of a skill with the same id in the same scope.
pub replaced: bool,
}
/// The host directory backing one scope of this user's tree.
pub fn tree_of(fs: &UserFs, scope: Scope) -> Result<&Path> {
let sk = fs.skills.as_ref().ok_or_else(|| {
anyhow::anyhow!("skills are not available in this context")
})?;
Ok(match scope {
Scope::Shared => sk.shared_host.as_path(),
Scope::Own => sk.own_host.as_path(),
})
}
/// The agent-visible scope segment (`shared`, or the caller's username).
pub fn segment_of(fs: &UserFs, scope: Scope) -> Result<&str> {
let sk = fs.skills.as_ref().ok_or_else(|| {
anyhow::anyhow!("skills are not available in this context")
})?;
Ok(match scope {
Scope::Shared => core_api::user_fs::SKILLS_SHARED_SCOPE,
Scope::Own => sk.own_username.as_str(),
})
}
/// Validates `source` and installs it into `scope`, replacing an existing skill
/// of the same id in that scope.
///
/// The source is copied *before* anything at the destination moves, so
/// registering a skill onto itself (`skill_register("global",
/// "skills/daniele/foo")` — the promotion path, which is deliberately the same
/// call rather than an endpoint of its own) needs no special case.
pub fn install(fs: &UserFs, scope: Scope, source: &Path) -> Result<Installed> {
let valid = validate_dir(source)?;
install_validated(fs, scope, source, &valid)
}
fn install_validated(
fs: &UserFs,
scope: Scope,
source: &Path,
valid: &ValidSkill,
) -> Result<Installed> {
let tree = tree_of(fs, scope)?.to_path_buf();
std::fs::create_dir_all(&tree)
.with_context(|| format!("cannot open the skills tree at {}", tree.display()))?;
let target = tree.join(&valid.id);
let replaced = target.exists();
let tag = uuid::Uuid::new_v4().simple().to_string();
// Staging lives **inside the destination tree**: `rename` only works within
// one filesystem, and a dot-directory is skipped by the indexer (see
// `super::collect`), so a crash mid-copy leaves litter, never a skill.
let staging = tree.join(format!(".staging-{tag}"));
let outcome = (|| -> Result<()> {
copy_tree(source, &staging)?;
stamp_provenance(source, &staging);
if replaced {
let parked = tree.join(format!(".old-{tag}"));
std::fs::rename(&target, &parked)
.with_context(|| format!("cannot replace the installed `{}`", valid.id))?;
// From here the id does not exist. If the second rename fails we put
// the old one back rather than leave the scope short of a skill it
// had a moment ago.
if let Err(e) = std::fs::rename(&staging, &target) {
let _ = std::fs::rename(&parked, &target);
return Err(anyhow::Error::new(e)
.context(format!("cannot install `{}`", valid.id)));
}
let _ = std::fs::remove_dir_all(&parked);
} else {
std::fs::rename(&staging, &target)
.with_context(|| format!("cannot install `{}`", valid.id))?;
}
Ok(())
})();
if outcome.is_err() {
let _ = std::fs::remove_dir_all(&staging);
}
outcome?;
Ok(Installed {
id: valid.id.clone(),
agent_dir: format!(
"{}/{}/{}",
core_api::user_fs::SKILLS_ROOT,
segment_of(fs, scope)?,
valid.id
),
replaced,
})
}
/// Carries the provenance ticket across, stamping the install date. Best-effort
/// for the same reason [`read_provenance`] is tolerant: this is a label on the
/// artefact, not part of it.
fn stamp_provenance(source: &Path, staging: &Path) {
let Some(mut p) = read_provenance(source) else { return };
p.installed_at = Some(chrono::Utc::now().to_rfc3339());
if let Ok(json) = serde_json::to_string_pretty(&p) {
let _ = std::fs::write(staging.join(SOURCE_FILE), json);
}
}
/// Copies a validated tree, refusing links again on the way.
///
/// The re-check is not redundant paranoia about the walk in `validate`: the
/// source folder is writable by the caller and by their container, so between
/// the two passes it can change. The cheap answer is to never follow a link at
/// either point.
///
/// Shared with `fetch_repo`, whose staging area is writable by the caller's
/// container for exactly as long and so needs exactly the same guarantee.
pub(crate) fn copy_tree(from: &Path, to: &Path) -> Result<()> {
std::fs::create_dir_all(to)
.with_context(|| format!("cannot create {}", to.display()))?;
for entry in std::fs::read_dir(from)
.with_context(|| format!("cannot read {}", from.display()))?
{
let entry = entry?;
let src = entry.path();
let dst = to.join(entry.file_name());
let meta = std::fs::symlink_metadata(&src)?;
if meta.file_type().is_symlink() {
bail!("`{}` is a symbolic link", entry.file_name().to_string_lossy());
}
if meta.is_dir() {
copy_tree(&src, &dst)?;
} else {
std::fs::copy(&src, &dst)
.with_context(|| format!("cannot copy {}", src.display()))?;
}
}
Ok(())
}
/// Removes an installed skill. No recycle bin, deliberately: the source it was
/// registered from almost always still exists, and a bin would be a second tree
/// to index, mount and explain.
pub fn remove(fs: &UserFs, scope: Scope, id: &str) -> Result<()> {
let tree = tree_of(fs, scope)?;
// The id names a directory *inside* the tree and nothing else — a `/` or a
// `..` here would be a path, and paths are not ids.
if id.is_empty() || id.contains('/') || id.contains('\\') || id.starts_with('.') {
bail!("`{id}` is not a skill id (it is the folder name, e.g. `ics-import`)");
}
let dir = tree.join(id);
if !dir.is_dir() {
bail!(
"no skill `{id}` in {}/{}/",
core_api::user_fs::SKILLS_ROOT,
segment_of(fs, scope)?
);
}
std::fs::remove_dir_all(&dir).with_context(|| format!("cannot remove `{id}`"))?;
Ok(())
}
// ── The approval card ─────────────────────────────────────────────────────────
/// What the human is shown before a registration goes through: `(old, new)` for
/// the diff card the write tools already use.
///
/// This is the review moment blueprint §9.1 calls the point of the whole design
/// — for the group's scope it is the **only** time a person reads a text that
/// will enter everybody's prompt — so `new` is the candidate's `SKILL.md` in
/// full, not a summary of it. When the id already exists, `old` is the installed
/// body, which turns the card into a diff of what actually changes.
///
/// `None` when the source cannot be read or does not validate: the card then
/// falls back to the generic approval event, and the refusal comes from the tool
/// itself with its own message.
pub fn preview(fs: &UserFs, scope: Scope, source: &Path) -> Option<(String, Option<String>, String)> {
let valid = validate_dir(source).ok()?;
let tree = tree_of(fs, scope).ok()?;
let target = tree.join(&valid.id);
let old = std::fs::read_to_string(target.join(SKILL_FILE)).ok();
let agent_dir = format!(
"{}/{}/{}",
core_api::user_fs::SKILLS_ROOT,
segment_of(fs, scope).ok()?,
valid.id
);
let header = card_header(&valid, scope, old.is_some(), &agent_dir);
let body = std::fs::read_to_string(source.join(SKILL_FILE)).ok()?;
Some((
agent_dir,
old.map(|o| format!("{}\n{o}", card_header(&valid, scope, true, ""))),
format!("{header}\n{body}"),
))
}
fn card_header(valid: &ValidSkill, scope: Scope, replacing: bool, _dir: &str) -> String {
let what = if replacing { "REPLACES the installed skill" } else { "new skill" };
let audience = match scope {
Scope::Shared => "the whole group — every member reads it as instructions",
Scope::Own => "you only",
};
let deps = valid.deps().unwrap_or_else(|| "none".into());
format!(
"<!-- {what}: `{}` · visible to {audience}\n \
{} files, {} KB · scripts: {} · dependency manifest: {deps}\n \
files: {} -->\n",
valid.id,
valid.files.len(),
valid.size_bytes.div_ceil(1024),
if valid.has_scripts() { "yes" } else { "no" },
valid
.files
.iter()
.map(|f| f.to_string_lossy().into_owned())
.collect::<Vec<_>>()
.join(", "),
)
}
#[cfg(test)]
mod tests {
use super::*;
use crate::skills::tests_support::Tree;
fn front(name: &str, description: &str) -> String {
format!("---\nname: {name}\ndescription: {description}\n---\n\nBody of {name}.\n")
}
/// The installed folder is named by the frontmatter, and the index picks it
/// up straight away.
#[test]
fn a_draft_folder_installs_under_its_declared_name() {
let t = Tree::new("install-name", "daniele");
let src = t.root.join("homes/u1/draft-2");
std::fs::create_dir_all(&src).unwrap();
std::fs::write(src.join(SKILL_FILE), front("ics-import", "Import an ICS feed.")).unwrap();
let got = install(&t.fs, Scope::Own, &src).unwrap();
assert_eq!(got.id, "ics-import");
assert_eq!(got.agent_dir, "skills/daniele/ics-import");
assert!(!got.replaced);
let listed = crate::skills::list(&t.fs);
assert_eq!(listed.len(), 1);
assert_eq!(listed[0].skill_file(), "skills/daniele/ics-import/SKILL.md");
}
/// Re-registering the same id replaces it, and nothing of the old copy
/// survives — the artefact is whole or absent, never merged.
#[test]
fn re_registering_replaces_without_leaving_the_old_files() {
let t = Tree::new("install-replace", "daniele");
let src = t.root.join("homes/u1/v1");
std::fs::create_dir_all(&src).unwrap();
std::fs::write(src.join(SKILL_FILE), front("x", "First.")).unwrap();
std::fs::write(src.join("old-helper.py"), "1").unwrap();
install(&t.fs, Scope::Own, &src).unwrap();
let src2 = t.root.join("homes/u1/v2");
std::fs::create_dir_all(&src2).unwrap();
std::fs::write(src2.join(SKILL_FILE), front("x", "Second.")).unwrap();
let got = install(&t.fs, Scope::Own, &src2).unwrap();
assert!(got.replaced);
let installed = t.root.join("skills-users/u1/x");
assert!(!installed.join("old-helper.py").exists());
assert_eq!(crate::skills::list(&t.fs)[0].description, "Second.");
// No staging or parked leftovers.
let stray: Vec<_> = std::fs::read_dir(t.root.join("skills-users/u1"))
.unwrap()
.filter_map(|e| e.ok())
.filter(|e| e.file_name().to_string_lossy().starts_with('.'))
.collect();
assert!(stray.is_empty(), "leftovers: {stray:?}");
}
/// Promotion is the same call with a different scope, and its source is the
/// already-installed copy — which the copy-first ordering makes safe.
#[test]
fn promoting_ones_own_skill_to_the_group_is_the_same_call() {
let t = Tree::new("install-promote", "daniele");
let src = t.root.join("homes/u1/draft");
std::fs::create_dir_all(&src).unwrap();
std::fs::write(src.join(SKILL_FILE), front("x", "Mine.")).unwrap();
install(&t.fs, Scope::Own, &src).unwrap();
let mine = t.root.join("skills-users/u1/x");
install(&t.fs, Scope::Shared, &mine).unwrap();
let ids: Vec<(String, String)> = crate::skills::list(&t.fs)
.into_iter()
.map(|s| (s.scope, s.id))
.collect();
assert_eq!(
ids,
vec![("shared".into(), "x".into()), ("daniele".into(), "x".into())]
);
}
/// A refused source leaves the tree exactly as it was — the validation runs
/// before a single byte is copied.
#[test]
fn an_invalid_source_touches_nothing() {
let t = Tree::new("install-invalid", "daniele");
let src = t.root.join("homes/u1/broken");
std::fs::create_dir_all(&src).unwrap();
std::fs::write(src.join("notes.txt"), "no frontmatter here").unwrap();
assert!(install(&t.fs, Scope::Own, &src).is_err());
assert!(crate::skills::list(&t.fs).is_empty());
assert_eq!(std::fs::read_dir(t.root.join("skills-users/u1")).unwrap().count(), 0);
}
#[test]
fn the_provenance_ticket_crosses_into_the_installed_copy() {
let t = Tree::new("install-prov", "daniele");
let src = t.root.join("homes/u1/fetched");
std::fs::create_dir_all(&src).unwrap();
std::fs::write(src.join(SKILL_FILE), front("x", "y")).unwrap();
std::fs::write(
src.join(SOURCE_FILE),
r#"{"url":"https://example.invalid/r","commit":"a1b2c3d"}"#,
)
.unwrap();
install(&t.fs, Scope::Own, &src).unwrap();
let p = read_provenance(&t.root.join("skills-users/u1/x")).unwrap();
assert_eq!(p.commit.as_deref(), Some("a1b2c3d"));
assert!(p.installed_at.is_some(), "install date not stamped");
}
#[test]
fn delete_removes_the_folder_and_refuses_a_path() {
let t = Tree::new("install-delete", "daniele");
let src = t.root.join("homes/u1/d");
std::fs::create_dir_all(&src).unwrap();
std::fs::write(src.join(SKILL_FILE), front("x", "y")).unwrap();
install(&t.fs, Scope::Own, &src).unwrap();
assert!(remove(&t.fs, Scope::Own, "../../etc").is_err());
assert!(remove(&t.fs, Scope::Own, "nope").is_err());
remove(&t.fs, Scope::Own, "x").unwrap();
assert!(crate::skills::list(&t.fs).is_empty());
}
/// The card shows the body that will enter the prompt, and on a replacement
/// it shows the one being replaced — so the human reads a diff, not a name.
#[test]
fn the_card_carries_the_whole_skill_body() {
let t = Tree::new("install-card", "daniele");
let src = t.root.join("homes/u1/c");
std::fs::create_dir_all(&src).unwrap();
std::fs::write(src.join(SKILL_FILE), front("x", "y")).unwrap();
let (path, old, new) = preview(&t.fs, Scope::Shared, &src).unwrap();
assert_eq!(path, "skills/shared/x");
assert!(old.is_none());
assert!(new.contains("Body of x."), "{new}");
assert!(new.contains("every member reads it as instructions"), "{new}");
install(&t.fs, Scope::Shared, &src).unwrap();
let (_, old, _) = preview(&t.fs, Scope::Shared, &src).unwrap();
assert!(old.unwrap().contains("Body of x."));
}
}
+183
View File
@@ -0,0 +1,183 @@
//! The administrative view of the two trees — what `list_items(type="skills")`
//! returns (blueprint §7.7).
//!
//! **Not the index, and the distinction is worth keeping sharp.** The index is
//! for *deciding*: path plus a cut description, the minimum needed to tell
//! whether a skill bears on the request, injected into every prompt and paid for
//! in tokens on every request of every user. This is for *administering*: the
//! full description, size, health and provenance, as JSON, only when asked. One
//! is always there and thin; the other is on demand and complete.
//!
//! Two consequences fall out of that split:
//!
//! - The **description is not truncated here.** Truncate in both places and the
//! full text becomes unreadable anywhere, while this tool exists precisely to
//! be the place it can be read. The real ceiling belongs at registration
//! ([`super::validate::DESCRIPTION_MAX`]), where the author is present and the
//! refusal is useful.
//! - **A broken skill appears here**, with the reason. The index skips it —
//! correctly, since it cannot be trusted to describe itself — but then nothing
//! would say *why* a folder placed on the box never showed up.
use std::path::Path;
use core_api::user_fs::{SKILLS_ROOT, SKILLS_SHARED_SCOPE, UserFs};
use serde_json::{Value, json};
use super::install::read_provenance;
use super::validate::validate_dir;
use super::{DESCRIPTION_LIMIT, SKILL_FILE, parse_front_matter};
/// Every skill folder visible to this user — valid or not — as JSON rows, in the
/// index's order (group's tree first, then their own, each by id).
pub fn report(fs: &UserFs) -> Vec<Value> {
let Some(sk) = &fs.skills else { return Vec::new() };
let mut rows: Vec<(String, String, Value)> = Vec::new();
for (scope, host) in [
(SKILLS_SHARED_SCOPE, sk.shared_host.as_path()),
(sk.own_username.as_str(), sk.own_host.as_path()),
] {
let Ok(entries) = std::fs::read_dir(host) else { continue };
let mut ids: Vec<String> = entries
.flatten()
.filter(|e| e.path().is_dir())
.filter_map(|e| e.file_name().to_str().map(str::to_string))
// Dot-directories are plumbing (a staging leftover), never a skill.
.filter(|id| !id.starts_with('.'))
.collect();
ids.sort();
for id in ids {
let row = inspect(scope, &host.join(&id), &id);
rows.push((id, scope.to_string(), row));
}
}
// A collision is a property of the *pair*, so it can only be marked once
// both trees have been read — the same reason the index marks both lines.
let colliding: std::collections::HashSet<String> = rows
.iter()
.filter(|(id, scope, _)| rows.iter().any(|(o, os, _)| o == id && os != scope))
.map(|(id, _, _)| id.clone())
.collect();
rows.into_iter()
.map(|(id, _, mut row)| {
row["collision"] = Value::Bool(colliding.contains(&id));
row
})
.collect()
}
/// One folder, described as fully as it allows itself to be.
fn inspect(scope: &str, dir: &Path, id: &str) -> Value {
let path = format!("{SKILLS_ROOT}/{scope}/{id}");
let mut row = json!({
"id": id,
"scope": scope,
"path": path,
"valid": false,
"problem": Value::Null,
});
// Validity here means **what the index does**: does its frontmatter parse?
// The structural rules (`validate_dir`) are a stricter set — they gate what
// may be *installed* — so a folder that fails them but parses is still shown
// in the prompt and must be reported as such, with the extra problem named.
let body = match std::fs::read_to_string(dir.join(SKILL_FILE)) {
Ok(b) => b,
Err(e) => {
row["problem"] = json!(format!("no readable {SKILL_FILE}: {e}"));
return row;
}
};
let front = match parse_front_matter(&body) {
Ok(f) => f,
Err(problem) => {
row["problem"] = json!(format!("invalid frontmatter: {problem}"));
return row;
}
};
row["valid"] = Value::Bool(true);
row["description"] = json!(front.description);
row["truncated_in_index"] =
Value::Bool(front.description.chars().count() > DESCRIPTION_LIMIT);
if front.name != id {
row["problem"] = json!(format!(
"the frontmatter `name` is `{}` but the folder is `{id}`; the folder wins",
front.name
));
}
match validate_dir(dir) {
Ok(v) => {
row["files"] = json!(v.files.len());
row["size_bytes"] = json!(v.size_bytes);
row["has_scripts"] = Value::Bool(v.has_scripts());
row["deps"] = v.deps().map(Value::String).unwrap_or(Value::Null);
}
// Reachable only for a folder placed by hand: everything installed
// through `skill_register` passed this very check.
Err(e) => row["problem"] = json!(e.to_string()),
}
if let Some(p) = read_provenance(dir) {
row["source_url"] = json!(p.url);
row["commit"] = p.commit.map(Value::String).unwrap_or(Value::Null);
row["installed_at"] = p.installed_at.map(Value::String).unwrap_or(Value::Null);
}
row
}
#[cfg(test)]
mod tests {
use super::*;
use crate::skills::tests_support::{Tree, valid};
/// The full description survives here even when the index cut it — this is
/// the one place it can be read whole.
#[test]
fn the_description_is_reported_untruncated_and_flagged() {
let long = "d".repeat(DESCRIPTION_LIMIT + 40);
let t = Tree::new("inv-desc", "daniele");
t.write("shared", "x", &valid("x", &long));
let rows = report(&t.fs);
assert_eq!(rows[0]["description"], json!(long));
assert_eq!(rows[0]["truncated_in_index"], json!(true));
assert_eq!(rows[0]["path"], json!("skills/shared/x"));
}
/// A folder the index skipped still shows up, with the reason — otherwise
/// nothing anywhere answers "why did my skill never appear?".
#[test]
fn a_broken_skill_is_reported_with_its_problem() {
let t = Tree::new("inv-broken", "daniele");
t.write("shared", "good", &valid("good", "Works."));
t.write("shared", "broken", "no frontmatter at all\n");
let rows = report(&t.fs);
assert_eq!(rows.len(), 2);
let broken = rows.iter().find(|r| r["id"] == json!("broken")).unwrap();
assert_eq!(broken["valid"], json!(false));
assert!(broken["problem"].as_str().unwrap().contains("frontmatter"));
// …and the index really did skip it, so the two views agree on the facts
// while disagreeing on what they show.
assert_eq!(crate::skills::list(&t.fs).len(), 1);
}
#[test]
fn a_colliding_id_is_marked_on_both_rows() {
let t = Tree::new("inv-collide", "daniele");
t.write("shared", "x", &valid("x", "Group's."));
t.write("mine", "x", &valid("x", "Mine."));
t.write("mine", "y", &valid("y", "Untouched."));
let rows = report(&t.fs);
for r in &rows {
let expect = r["id"] == json!("x");
assert_eq!(r["collision"], json!(expect), "{r}");
}
}
}
+656
View File
@@ -0,0 +1,656 @@
//! The skills index — **the only thing about a skill that reaches the prompt**.
//!
//! A skill is a folder with a `SKILL.md` in it, living in one of the two trees
//! `UserFs` mounts read-only (`skills/shared/<id>` and `skills/<username>/<id>`,
//! see [`core_api::user_fs::SkillMounts`]). Nothing here is a manager in the usual
//! sense: this module is a set of **pure functions over those two paths**, in the
//! shape of `LlmCommandManager` and for the same reason — the list must be a
//! function of the content, never a file someone maintains by hand, or it diverges
//! at the first skill added.
//!
//! What reaches the model is deliberately thin (blueprint §5, progressive
//! disclosure): the **path** of each `SKILL.md` plus a truncated `description`.
//! The body is never injected — the model reads it with `read_file` when the
//! description matches. Carrying the full path rather than an id plus a
//! composition rule is what makes a dedicated read tool unnecessary: it costs a
//! few tokens per line and removes the one step the model can get wrong on its own.
//!
//! Three properties are load-bearing, and each is here because the alternative
//! fails in a specific way:
//!
//! - **A stable order** (scope, then id). The index sits inside the string every
//! provider uses as its cache key, so a non-deterministic order would cost a
//! miss on every rebuild.
//! - **A deterministic cut at the budget**, closed by an explicit omission line.
//! Without the determinism the stable order stops buying a stable cache key;
//! without the line the model reads a truncated list *believing it complete* and
//! concludes in good faith that a skill does not exist.
//! - **A broken skill is skipped, never fatal.** The index is built while
//! assembling a system prompt; one malformed frontmatter must not take a
//! conversation down with it.
use std::path::Path;
use std::sync::{Arc, OnceLock};
use core_api::user_fs::{SKILLS_ROOT, SKILLS_SHARED_SCOPE, UserFs};
use tracing::warn;
pub mod install;
pub mod inventory;
pub mod validate;
pub mod watch;
/// The file that makes a directory a skill.
pub const SKILL_FILE: &str = "SKILL.md";
/// Which of a user's two trees an operation addresses.
///
/// The tools take `"mine"` / `"global"` rather than a username, and that is
/// deliberate (blueprint §4.1): a tool argument naming the caller invites
/// passing somebody *else's* name, which the server would then have to ignore.
/// Human-readable path, stable argument for the machine.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Scope {
/// `skills/{username}/…` — the caller's own.
Own,
/// `skills/shared/…` — the group's, gated by the `skill.manage` capability.
Shared,
}
impl Scope {
pub fn parse(raw: &str) -> anyhow::Result<Self> {
match raw {
"mine" => Ok(Scope::Own),
"global" => Ok(Scope::Shared),
other => anyhow::bail!(
"unknown scope `{other}`: use \"mine\" (your own skills) or \"global\" \
(the whole group's)"
),
}
}
/// The spelling the tools take and the model reads back.
pub fn as_arg(self) -> &'static str {
match self {
Scope::Own => "mine",
Scope::Shared => "global",
}
}
}
// ── Prompt freshness (blueprint §6) ──────────────────────────────────────────
/// Whose system prompts a skills change has made stale.
///
/// Distinct from [`Scope`], which says *where a write went*: a write to the
/// group's tree makes every live member's prompt stale, a write to one member's
/// own tree makes only theirs. Session 5's `SkillsChanged` event carries the
/// same shape, for the same reason.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum PromptScope {
/// The group's tree changed: everyone's index moved.
Everyone,
/// One member's own tree changed.
User(String),
}
/// How a skills write reaches conversations that are already running.
///
/// The index sits inside the frozen system prefix, which is rebuilt only after
/// twenty idle minutes ([`crate::loop_adapters::prefix_cache`]). That is right
/// for a file edited underneath a running conversation and wrong here: an admin
/// who installs a skill and then asks the assistant to use it must not be told
/// for twenty minutes that it does not exist.
///
/// **Not on the system bus.** `SkillsChanged` (session 5) is for a change made
/// *outside* the process, by someone editing files on the box; a write made by a
/// tool is already inside the process and can say so directly, which is both
/// immediate and impossible to lose. The bus stays for the case it was designed
/// for.
#[async_trait::async_trait]
pub trait PromptPrefixes: Send + Sync {
async fn invalidate(&self, scope: PromptScope);
}
/// The cell the skill tools hold, filled once the instance exists.
///
/// The tools are built during composition, before `Skald` does; the reactor they
/// need can therefore only be installed afterwards — the same post-construction
/// shape as the plugin manager's `set_skald` and the user-lifecycle reconciler.
/// An empty cell is a silent no-op rather than an error: the only way to reach
/// it is a `Skald` that failed to finish building, in which case there are no
/// live conversations to keep fresh either.
#[derive(Default)]
pub struct PromptPrefixCell(OnceLock<Arc<dyn PromptPrefixes>>);
impl PromptPrefixCell {
pub fn install(&self, sink: Arc<dyn PromptPrefixes>) {
let _ = self.0.set(sink);
}
pub async fn invalidate(&self, scope: PromptScope) {
if let Some(sink) = self.0.get() {
sink.invalidate(scope).await;
}
}
}
/// How much of a `description` the index carries. The full text lives on disk and
/// is surfaced by the enumeration tool; this cap applies **only** to the index,
/// which is the one place tokens are paid on every request of every user.
///
/// Hermes cuts at 60. Sixty is too few here: the `description` *is* the use
/// condition ("when should I reach for this?"), and cutting mid-sentence removes
/// exactly the part that decides the trigger.
pub const DESCRIPTION_LIMIT: usize = 200;
/// Ceiling on the whole rendered index, in bytes. A badly written skill must not
/// be able to eat the prompt of everybody in the house.
pub const INDEX_BUDGET: usize = 8 * 1024;
/// Room kept aside for the omission line, so appending it can never push the
/// render past [`INDEX_BUDGET`]. A fixed reserve rather than a computed one keeps
/// the cut a pure function of the ordered list.
const OMISSION_RESERVE: usize = 96;
/// One installed skill, as the index sees it: where it is and when to reach for
/// it. The body never enters this type — it is read from disk, by the model.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Skill {
/// The directory name, which **is** the id.
pub id: String,
/// The agent-visible scope segment: `shared`, or the owner's username.
pub scope: String,
/// The frontmatter `description`, verbatim and untruncated.
pub description: String,
}
impl Skill {
/// The skill's folder, in agent vocabulary (`skills/shared/ics-import`).
pub fn agent_dir(&self) -> String {
format!("{SKILLS_ROOT}/{}/{}", self.scope, self.id)
}
/// The path the index prints — the `SKILL.md` itself, ready for `read_file`.
pub fn skill_file(&self) -> String {
format!("{}/{SKILL_FILE}", self.agent_dir())
}
}
/// Every skill visible to this user, in the index's stable order: the group's
/// tree first, then their own, each sorted by id.
///
/// Per-user by construction — the own tree comes from `fs.skills`, which is built
/// for one member — so another member's private skills cannot appear here. A
/// `UserFs` without the skills tree (an inert placeholder, a unit test) simply has
/// none.
pub fn list(fs: &UserFs) -> Vec<Skill> {
let Some(sk) = &fs.skills else { return Vec::new() };
let mut out = Vec::new();
collect(SKILLS_SHARED_SCOPE, &sk.shared_host, &mut out);
collect(&sk.own_username, &sk.own_host, &mut out);
out
}
/// Reads one scope's tree, appending its valid skills in id order. A tree that
/// does not exist yet is not an error: it is the state of every fresh instance.
fn collect(scope: &str, host: &Path, out: &mut Vec<Skill>) {
let Ok(entries) = std::fs::read_dir(host) else { return };
let mut found: Vec<Skill> = Vec::new();
for entry in entries.flatten() {
let path = entry.path();
if !path.is_dir() {
continue;
}
let Some(id) = path.file_name().and_then(|n| n.to_str()) else { continue };
// Dot-directories are plumbing (a staging leftover, an editor's cruft),
// never a skill: an id starting with a dot cannot be registered.
if id.starts_with('.') {
continue;
}
let body = match std::fs::read_to_string(path.join(SKILL_FILE)) {
Ok(b) => b,
Err(e) => {
warn!(scope, skill = id, error = %e, "skill skipped: no readable SKILL.md");
continue;
}
};
let front = match parse_front_matter(&body) {
Ok(f) => f,
Err(problem) => {
warn!(scope, skill = id, problem, "skill skipped: invalid frontmatter");
continue;
}
};
// The id is the directory, always — that is what every path in the index
// is built from. A `name` that disagrees is worth saying out loud (the
// registration tool makes the two agree by construction, so this can only
// be a folder placed by hand) but not worth hiding the skill over.
if front.name != id {
warn!(
scope, skill = id, declared = front.name,
"skill frontmatter `name` differs from its directory; the directory wins"
);
}
found.push(Skill { id: id.to_string(), scope: scope.to_string(), description: front.description });
}
found.sort_by(|a, b| a.id.cmp(&b.id));
out.extend(found);
}
/// The two mandatory frontmatter fields. Unknown keys (`license`,
/// `allowed-tools`, anything a skill written elsewhere carries) are ignored.
#[derive(serde::Deserialize)]
pub(crate) struct FrontMatter {
#[serde(default)]
pub(crate) name: String,
#[serde(default)]
pub(crate) description: String,
}
/// Parses the leading `---` YAML block of a `SKILL.md`. `Err` carries a short
/// reason, which the caller logs — this is the one place a hand-edited skill goes
/// wrong, so the log line has to say which of the three ways it did.
///
/// Shared with [`validate`] on purpose: two frontmatter parsers would agree on
/// the easy cases and diverge on the first odd one, and the pair that must never
/// disagree is exactly *what the registration accepts* and *what the index
/// shows* — a skill installed but invisible is the worst of both.
pub(crate) fn parse_front_matter(body: &str) -> Result<FrontMatter, &'static str> {
let rest = body
.strip_prefix("---\n")
.or_else(|| body.strip_prefix("---\r\n"))
.ok_or("no frontmatter block")?;
let end = rest
.split_inclusive('\n')
.scan(0usize, |at, line| {
let start = *at;
*at += line.len();
Some((start, line))
})
.find(|(_, line)| matches!(line.trim_end(), "---" | "..."))
.map(|(start, _)| start)
.ok_or("unterminated frontmatter block")?;
let front: FrontMatter = serde_yaml::from_str(&rest[..end]).map_err(|_| "not valid YAML")?;
if front.name.trim().is_empty() {
return Err("frontmatter has no `name`");
}
if front.description.trim().is_empty() {
return Err("frontmatter has no `description`");
}
Ok(FrontMatter { name: front.name.trim().to_string(), description: front.description.trim().to_string() })
}
/// The index for one user, ready to replace `__SKILLS_LIST__`.
pub fn render_index(fs: &UserFs) -> String {
render(&list(fs))
}
/// A stable digest of a rendered index. The invalidation rule of blueprint §6 is
/// keyed on **this string changing**, not on a file inside a skill changing:
/// editing a script or a reference document leaves the index byte-identical, so it
/// costs nobody a cache miss. Only adding, removing or re-describing a skill does.
pub fn digest(rendered: &str) -> String {
use sha2::{Digest, Sha256};
format!("{:x}", Sha256::digest(rendered.as_bytes()))
}
/// A digest of one scope **tree's** visible content — the sorted
/// (id, description) pairs, which is everything the index ever prints from it.
///
/// This is the file-watcher's gate (blueprint §8.2), and the rule is §6's, per
/// tree instead of per render: editing a script or a reference document leaves
/// it alone; adding, removing or re-describing a skill moves it. A collision
/// marker needs no hashing of its own — it is a function of the two id sets,
/// and those are hashed. An empty or missing tree digests as [`digest`]`("")`.
pub fn tree_digest(host: &Path) -> String {
use sha2::{Digest, Sha256};
let label = host.file_name().and_then(|n| n.to_str()).unwrap_or("?");
let mut found = Vec::new();
collect(label, host, &mut found);
let mut hasher = Sha256::new();
for s in &found {
hasher.update(s.id.as_bytes());
hasher.update([0]);
hasher.update(s.description.as_bytes());
hasher.update([0]);
}
format!("{:x}", hasher.finalize())
}
/// The imperative preamble. Deliberately pushy — the failure mode of every skill
/// system is the model *under*-triggering, and a neutral "the following skills are
/// available" produces a model that scrolls past them.
const HEADER: &str = "\
## Skills (mandatory)
Before replying, scan the skills below. If a skill matches or is even partially \
relevant to your task, you MUST read its `SKILL.md` with `read_file` and follow \
its instructions. Err on the side of reading it it is always better to have \
context you don't need than to miss critical steps, pitfalls or established \
workflows. Skills encode how a task should be done here, so read one even for a \
task you already know how to do.
<available_skills>
";
/// Closing rules. The `workdir` sentence is here, said once, because there is no
/// launch tool to hide it in: a skill's scripts are run with the general-purpose
/// `execute_cmd`, and a model that runs one from the home gets a bare ENOENT.
const FOOTER: &str = "\
</available_skills>
Only proceed without reading a skill if genuinely none are relevant.
Run a skill's scripts with `execute_cmd`, setting `workdir` to the skill's own \
folder. The whole `skills/` tree is read-only: anything a skill needs to write \
(caches, state, dependencies) goes in your home or `/tmp`.";
/// Renders the index from an ordered skill list — pure, so the budget cut and the
/// collision marking are testable without a filesystem.
///
/// **Empty in, empty out**, and that is a contract rather than an optimisation:
/// every word of prose lives in here, so an instance with no skills spends nothing
/// and leaves no orphan sentence from which the model could infer that something
/// exists. (The MCP list is the counter-example — its prose sits *around* the
/// placeholder, so an empty list left a promise of a table behind, and the model
/// answered by inventing a discovery tool.)
pub fn render(skills: &[Skill]) -> String {
if skills.is_empty() {
return String::new();
}
// An id present in both trees is marked on **both** lines: neither wins in
// silence. The personal one winning would be a quiet divergence from the
// group's set, the group's one winning would ignore the member's own work.
// With full paths in the index the disambiguation is already free — the two
// lines differ — so fail-loud costs only the marker.
let colliding: std::collections::HashSet<&str> = skills
.iter()
.filter(|s| skills.iter().any(|o| o.id == s.id && o.scope != s.scope))
.map(|s| s.id.as_str())
.collect();
let paths: Vec<String> = skills.iter().map(Skill::skill_file).collect();
let width = paths.iter().map(String::len).max().unwrap_or(0);
let rows: Vec<String> = skills
.iter()
.zip(&paths)
.map(|(s, path)| {
let mut desc = truncate(&flatten(&s.description), DESCRIPTION_LIMIT);
if colliding.contains(s.id.as_str()) {
desc.push_str(" [name collision]");
}
format!(" {path:<width$} {desc}\n")
})
.collect();
let mut out = String::with_capacity(HEADER.len() + FOOTER.len() + 128);
out.push_str(HEADER);
let mut room = INDEX_BUDGET
.saturating_sub(HEADER.len() + FOOTER.len() + OMISSION_RESERVE);
let mut omitted = 0;
for (i, row) in rows.iter().enumerate() {
// Stop at the **first** row that does not fit, rather than skipping it and
// trying the next: the cut has to be a suffix of the stable order, or two
// renders of the same set could keep different skills.
if row.len() > room {
omitted = rows.len() - i;
break;
}
room -= row.len();
out.push_str(row);
}
if omitted > 0 {
warn!(omitted, total = rows.len(), "skills index over budget: tail omitted");
out.push_str(&format!(" [{omitted} more skills omitted — index budget reached]\n"));
}
out.push_str(FOOTER);
out
}
/// A description as one line: a multi-line one would break the column layout, and
/// the index is read as a table.
fn flatten(s: &str) -> String {
s.split_whitespace().collect::<Vec<_>>().join(" ")
}
/// Truncates to `limit` **characters** (never bytes — this text is user-authored
/// and routinely accented), marking the cut so the model knows it read a prefix.
fn truncate(s: &str, limit: usize) -> String {
if s.chars().count() <= limit {
return s.to_string();
}
let mut out: String = s.chars().take(limit).collect();
out.push('…');
out
}
/// A temporary `{WD}` with the two scope trees and a `UserFs` over it — shared
/// by the index, install and inventory tests, which all need the same fixture
/// and would otherwise each grow their own slightly different one.
#[cfg(test)]
pub(crate) mod tests_support {
use super::*;
use core_api::user_fs::SkillMounts;
use std::path::PathBuf;
pub(crate) struct Tree {
pub(crate) root: PathBuf,
pub(crate) fs: UserFs,
}
impl Tree {
pub(crate) fn new(tag: &str, username: &str) -> Self {
let root = std::env::temp_dir().join(format!(
"skald-skills-{tag}-{}-{:?}",
std::process::id(),
std::thread::current().id()
));
let _ = std::fs::remove_dir_all(&root);
let shared = root.join("skills");
let own = root.join("skills-users").join("u1");
std::fs::create_dir_all(&shared).unwrap();
std::fs::create_dir_all(&own).unwrap();
let fs = UserFs::new(
"u1",
root.join("homes").join("u1"),
"skald-u1",
PathBuf::from("/root"),
vec![],
vec![],
None,
)
.with_skills(SkillMounts {
root_host: root.join(".skills-root").join("u1"),
shared_host: shared,
own_host: own,
own_username: username.into(),
});
Self { root, fs }
}
/// Drops a skill straight into a scope tree, the way a hand-placed folder
/// on the box arrives — bypassing the registration tool, which these
/// tests are deliberately not exercising.
pub(crate) fn write(&self, scope: &str, id: &str, body: &str) {
let base = match scope {
"shared" => self.root.join("skills"),
_ => self.root.join("skills-users").join("u1"),
};
let dir = base.join(id);
std::fs::create_dir_all(&dir).unwrap();
std::fs::write(dir.join(SKILL_FILE), body).unwrap();
}
}
impl Drop for Tree {
fn drop(&mut self) {
let _ = std::fs::remove_dir_all(&self.root);
}
}
pub(crate) fn valid(name: &str, description: &str) -> String {
format!("---\nname: {name}\ndescription: {description}\n---\n\nThe body.\n")
}
}
#[cfg(test)]
mod tests {
use super::tests_support::{Tree, valid};
use super::*;
use std::path::PathBuf;
fn skill(scope: &str, id: &str, description: &str) -> Skill {
Skill { id: id.into(), scope: scope.into(), description: description.into() }
}
/// Both trees are enumerated, the group's first, each in id order — and the
/// order is the whole reason: it is inside the provider's cache key.
#[test]
fn both_scopes_are_listed_in_a_stable_order() {
let t = Tree::new("order", "daniele");
t.write("shared", "pdf-forms", &valid("pdf-forms", "Fill a PDF form."));
t.write("shared", "ics-import", &valid("ics-import", "Import an ICS feed."));
t.write("mine", "spesa", &valid("spesa", "Reconcile the statement."));
let got: Vec<(String, String)> =
list(&t.fs).into_iter().map(|s| (s.scope, s.id)).collect();
assert_eq!(
got,
vec![
("shared".into(), "ics-import".into()),
("shared".into(), "pdf-forms".into()),
("daniele".into(), "spesa".into()),
]
);
}
/// A skill nobody can parse is skipped; the ones around it are not. The index
/// is built while assembling a prompt, so a broken folder must cost its own
/// line and nothing else.
#[test]
fn a_malformed_skill_is_skipped_not_fatal() {
let t = Tree::new("malformed", "daniele");
t.write("shared", "good", &valid("good", "Works."));
t.write("shared", "no-frontmatter", "Just a body, no YAML at all.\n");
t.write("shared", "unterminated", "---\nname: x\ndescription: y\n");
t.write("shared", "not-yaml", "---\nname: [unclosed\n---\n");
t.write("shared", "no-description", "---\nname: x\n---\n");
std::fs::create_dir_all(t.root.join("skills").join("no-skill-md")).unwrap();
let ids: Vec<String> = list(&t.fs).into_iter().map(|s| s.id).collect();
assert_eq!(ids, vec!["good".to_string()]);
}
/// The directory is the id, whatever the frontmatter says — every path in the
/// index is built from it.
#[test]
fn the_directory_is_the_id() {
let t = Tree::new("id", "daniele");
t.write("shared", "ics-import", &valid("something-else", "Import an ICS feed."));
let got = list(&t.fs);
assert_eq!(got[0].id, "ics-import");
assert_eq!(got[0].skill_file(), "skills/shared/ics-import/SKILL.md");
}
/// The heart of the multi-user half: the own tree is keyed on the userid, so a
/// private skill of one member cannot reach another member's prompt.
#[test]
fn a_private_skill_belongs_to_one_member_only() {
let a = Tree::new("private-a", "anna");
a.write("mine", "budget", &valid("budget", "Anna's own."));
let b = Tree::new("private-b", "bruno");
assert!(render_index(&a.fs).contains("skills/anna/budget/SKILL.md"));
assert_eq!(render_index(&b.fs), "");
}
/// Nothing installed ⇒ nothing rendered. Not "an empty section": the whole
/// prose lives inside the render precisely so that this case costs zero tokens
/// and leaves no sentence the model could read as a promise.
#[test]
fn no_skills_renders_nothing_at_all() {
assert_eq!(render(&[]), "");
let bare = UserFs::new("u1", PathBuf::from("/wd/homes/u1"), "c", PathBuf::from("/root"), vec![], vec![], None);
assert_eq!(render_index(&bare), "");
}
#[test]
fn a_line_carries_the_full_path_and_a_truncated_description() {
let long = "x".repeat(DESCRIPTION_LIMIT + 50);
let out = render(&[
skill("shared", "ics-import", "Download an iCalendar (ICS) feed\nand output JSON."),
skill("daniele", "spesa", &long),
]);
// The path is printed in full: no id-plus-composition-rule for the model
// to get wrong.
assert!(out.contains("skills/shared/ics-import/SKILL.md"), "{out}");
assert!(out.contains("skills/daniele/spesa/SKILL.md"), "{out}");
// A multi-line description becomes one line.
assert!(out.contains("Download an iCalendar (ICS) feed and output JSON."), "{out}");
// …and a long one is cut, visibly.
assert!(out.contains(&format!("{}", "x".repeat(DESCRIPTION_LIMIT))), "{out}");
assert!(!out.contains(&"x".repeat(DESCRIPTION_LIMIT + 1)), "{out}");
// The imperative header and the closing rule are both there.
assert!(out.starts_with("## Skills (mandatory)"), "{out}");
assert!(out.contains("MUST read its `SKILL.md`"), "{out}");
assert!(out.ends_with("goes in your home or `/tmp`."), "{out}");
}
/// The same id in both trees: both lines stay, both are marked. Neither tree
/// shadows the other, here or in the path router.
#[test]
fn a_colliding_id_is_marked_on_both_lines() {
let out = render(&[
skill("shared", "ics-import", "The group's."),
skill("shared", "pdf-forms", "Untouched."),
skill("daniele", "ics-import", "My fork."),
]);
assert_eq!(out.matches("[name collision]").count(), 2, "{out}");
for line in out.lines().filter(|l| l.contains("pdf-forms")) {
assert!(!line.contains("[name collision]"), "{line}");
}
}
/// Over budget the index cuts from the tail of the stable order and says how
/// many it dropped — deterministically, because the cut is part of the cache
/// key, and out loud, because a silently truncated index makes the model
/// conclude in good faith that a skill does not exist.
#[test]
fn over_budget_the_tail_is_cut_deterministically_and_announced() {
let many: Vec<Skill> = (0..400)
.map(|i| skill("shared", &format!("skill-{i:03}"), &"d".repeat(DESCRIPTION_LIMIT)))
.collect();
let out = render(&many);
assert!(out.len() <= INDEX_BUDGET, "budget blown: {} bytes", out.len());
assert!(out.contains("more skills omitted — index budget reached"), "{out}");
// A suffix of the order is what went missing: the first is in, the last is not.
assert!(out.contains("skills/shared/skill-000/SKILL.md"), "{out}");
assert!(!out.contains("skills/shared/skill-399/SKILL.md"), "{out}");
// Same set in, same bytes out — otherwise the stable order buys nothing.
assert_eq!(out, render(&many));
assert_eq!(digest(&out), digest(&render(&many)));
}
/// The digest keys on what the model can see. Editing a script or a reference
/// document leaves it alone; re-describing a skill moves it.
#[test]
fn the_digest_follows_the_index_not_the_files() {
let before = render(&[skill("shared", "ics-import", "Import an ICS feed.")]);
let same = render(&[skill("shared", "ics-import", "Import an ICS feed.")]);
let after = render(&[skill("shared", "ics-import", "Import an ICS feed, then dedupe.")]);
assert_eq!(digest(&before), digest(&same));
assert_ne!(digest(&before), digest(&after));
}
}
+311
View File
@@ -0,0 +1,311 @@
//! What a folder must be before it is allowed to become a skill.
//!
//! **One validation site, because there is one door.** The two trees are
//! read-only in both directions (blueprint §9), so every byte that ever lands in
//! them passes through here — and the index on the other side can therefore
//! assume it is reading well-formed skills instead of tolerating half-written
//! ones. That is the whole argument for the read-only trees: with three write
//! paths (fs-tools, the container shell, an HTTP copy) validation would either
//! live in three places or nowhere.
//!
//! This module is deliberately callable from more than the registration tool:
//! the ZIP upload and the marketplace of blueprint §11.2 are the same check with
//! a different source of bytes, and two validators would diverge at the first
//! edge case.
use std::path::{Path, PathBuf};
use anyhow::{Result, bail};
use super::{SKILL_FILE, parse_front_matter};
/// Longest `name` accepted, matching the `^[a-z0-9][a-z0-9-]{0,63}$` shape: the
/// name becomes a directory name and then a path segment in every prompt.
pub const NAME_MAX: usize = 64;
/// Ceiling on the `description`, applied **here** rather than in the index.
///
/// One limit, at the one place it can fail usefully: the author is present, sees
/// the refusal and can shorten the text. The index's 200-character cut
/// ([`super::DESCRIPTION_LIMIT`]) is a different rule for a different reason —
/// tokens paid on every request of every user — and truncating there is not a
/// rejection, since the full text stays readable through the enumeration tool.
pub const DESCRIPTION_MAX: usize = 1000;
/// Ceiling on how many files one skill may carry.
pub const MAX_FILES: usize = 500;
/// Ceiling on a skill's total size. Generous for instructions plus scripts and
/// reference documents; far below anything that would be a *dataset*, which is
/// not what this tree is for.
pub const MAX_TOTAL_BYTES: u64 = 8 * 1024 * 1024;
/// A source folder that passed every check — the only thing [`super::install`]
/// accepts, so an unvalidated path cannot reach the tree by construction.
#[derive(Debug, Clone)]
pub struct ValidSkill {
/// The id, taken from the frontmatter `name` and **not** from the source
/// folder's name. The working copy may be called `draft-2`; the installed
/// artefact is called what it declares itself to be, and from then on
/// id = directory name by construction.
pub id: String,
pub description: String,
/// Every regular file, relative to the source root, in a stable order —
/// what the approval card lists.
pub files: Vec<PathBuf>,
pub size_bytes: u64,
}
impl ValidSkill {
/// Whether the skill carries anything executable, for the enumeration tool.
pub fn has_scripts(&self) -> bool {
self.files.iter().any(|f| {
matches!(
f.extension().and_then(|e| e.to_str()),
Some("py" | "js" | "mjs" | "cjs" | "ts" | "sh" | "bash")
)
})
}
/// Which ecosystem's dependency manifest it ships, if any. Reported rather
/// than acted on: v1 installs nothing (blueprint §10), and a skill that
/// needs a package says so in its body.
pub fn deps(&self) -> Option<String> {
let has = |n: &str| self.files.iter().any(|f| f == Path::new(n));
match (has("requirements.txt"), has("package.json")) {
(true, true) => Some("python+node".into()),
(true, false) => Some("python".into()),
(false, true) => Some("node".into()),
_ => None,
}
}
}
/// Validates a candidate skill folder on the host filesystem.
///
/// Every refusal names what to fix: this error text is read by a model that will
/// try again, and "invalid skill" would only produce a guess.
pub fn validate_dir(dir: &Path) -> Result<ValidSkill> {
if !dir.is_dir() {
bail!(
"not a folder: {}. A skill is a folder containing a `{SKILL_FILE}`.",
dir.display()
);
}
let skill_md = dir.join(SKILL_FILE);
if !skill_md.is_file() {
bail!(
"no `{SKILL_FILE}` in that folder. A skill is a folder whose `{SKILL_FILE}` \
opens with a YAML frontmatter block declaring `name` and `description`."
);
}
let body = std::fs::read_to_string(&skill_md)
.map_err(|e| anyhow::anyhow!("cannot read {SKILL_FILE}: {e}"))?;
let front = parse_front_matter(&body).map_err(|problem| {
anyhow::anyhow!(
"invalid `{SKILL_FILE}` frontmatter ({problem}). It must start with a `---` line, \
then `name:` and `description:`, then a closing `---`."
)
})?;
check_name(&front.name)?;
if front.description.chars().count() > DESCRIPTION_MAX {
bail!(
"the `description` is {} characters; the limit is {DESCRIPTION_MAX}. It is the \
*use condition* when to reach for this skill not a manual; the body of \
`{SKILL_FILE}` is where the detail goes.",
front.description.chars().count()
);
}
let mut walk = Walk::default();
walk.visit(dir, Path::new(""))?;
Ok(ValidSkill {
id: front.name,
description: front.description,
files: walk.files,
size_bytes: walk.bytes,
})
}
/// The id charset: `^[a-z0-9][a-z0-9-]{0,63}$`.
///
/// Checked by hand rather than by regex because the failure has to *teach* — the
/// caller is a model that will retry, and "does not match a pattern" is not a
/// correction.
fn check_name(name: &str) -> Result<()> {
if name.len() > NAME_MAX {
bail!("the frontmatter `name` is longer than {NAME_MAX} characters: `{name}`");
}
let ok_first = name.chars().next().is_some_and(|c| c.is_ascii_lowercase() || c.is_ascii_digit());
let ok_rest = name.chars().all(|c| c.is_ascii_lowercase() || c.is_ascii_digit() || c == '-');
if !ok_first || !ok_rest {
bail!(
"the frontmatter `name` must be lowercase letters, digits and hyphens, starting \
with a letter or digit (it becomes the folder name and the path the assistant \
reads): `{name}`"
);
}
Ok(())
}
/// Recursive walk that enforces the structural rules while it counts.
#[derive(Default)]
struct Walk {
files: Vec<PathBuf>,
bytes: u64,
}
impl Walk {
fn visit(&mut self, dir: &Path, rel: &Path) -> Result<()> {
let mut entries: Vec<_> = std::fs::read_dir(dir)
.map_err(|e| anyhow::anyhow!("cannot read {}: {e}", dir.display()))?
.filter_map(|e| e.ok())
.collect();
// Stable order: the file list ends up on an approval card, and a card
// that reshuffles between two renders of the same folder reads as a
// different change.
entries.sort_by_key(|e| e.file_name());
for entry in entries {
let path = entry.path();
let name = entry.file_name();
let child_rel = rel.join(&name);
// `symlink_metadata` does NOT follow the link, which is the point:
// a link is refused for what it is, before anything asks where it
// points. A skill is copied into a tree every member reads, and a
// link out of that tree would make the installed artefact a window
// onto something the installation never reviewed.
let meta = std::fs::symlink_metadata(&path)
.map_err(|e| anyhow::anyhow!("cannot stat {}: {e}", child_rel.display()))?;
if meta.file_type().is_symlink() {
bail!(
"`{}` is a symbolic link. A skill must be self-contained — copy the real \
file in instead.",
child_rel.display()
);
}
if meta.is_dir() {
self.visit(&path, &child_rel)?;
continue;
}
if !meta.is_file() {
bail!("`{}` is not a regular file.", child_rel.display());
}
self.bytes += meta.len();
self.files.push(child_rel);
if self.files.len() > MAX_FILES {
bail!("that folder holds more than {MAX_FILES} files — too much for a skill.");
}
if self.bytes > MAX_TOTAL_BYTES {
bail!(
"that folder is over {} MiB — too much for a skill. Skills hold \
instructions, scripts and reference documents; bulk data belongs in your \
home or a project.",
MAX_TOTAL_BYTES / (1024 * 1024)
);
}
}
Ok(())
}
}
#[cfg(test)]
mod tests {
use super::*;
struct Dir(PathBuf);
impl Dir {
fn new(tag: &str) -> Self {
let p = std::env::temp_dir().join(format!(
"skald-skillval-{tag}-{}-{:?}",
std::process::id(),
std::thread::current().id()
));
let _ = std::fs::remove_dir_all(&p);
std::fs::create_dir_all(&p).unwrap();
Dir(p)
}
fn write(&self, rel: &str, body: &str) {
let p = self.0.join(rel);
std::fs::create_dir_all(p.parent().unwrap()).unwrap();
std::fs::write(p, body).unwrap();
}
}
impl Drop for Dir {
fn drop(&mut self) {
let _ = std::fs::remove_dir_all(&self.0);
}
}
fn front(name: &str, description: &str) -> String {
format!("---\nname: {name}\ndescription: {description}\n---\n\nBody.\n")
}
/// The id comes from the frontmatter, never from the folder: the working
/// copy is allowed a scratch name, the artefact is not.
#[test]
fn the_id_comes_from_the_frontmatter_not_the_folder() {
let d = Dir::new("id");
d.write("SKILL.md", &front("ics-import", "Import an ICS feed."));
d.write("scripts/run.py", "print(1)\n");
let v = validate_dir(&d.0).unwrap();
assert_eq!(v.id, "ics-import");
assert_eq!(v.files, vec![PathBuf::from("SKILL.md"), PathBuf::from("scripts/run.py")]);
assert!(v.has_scripts());
assert_eq!(v.deps(), None);
}
#[test]
fn a_folder_without_a_skill_md_is_refused() {
let d = Dir::new("nomd");
d.write("notes.txt", "hello");
let e = validate_dir(&d.0).unwrap_err().to_string();
assert!(e.contains("SKILL.md"), "{e}");
}
#[test]
fn a_name_that_cannot_be_a_folder_is_refused() {
for bad in ["Ics Import", "../escape", "-leading", "UPPER"] {
let d = Dir::new("badname");
d.write("SKILL.md", &front(bad, "x"));
assert!(validate_dir(&d.0).is_err(), "accepted `{bad}`");
}
}
#[test]
fn an_overlong_description_is_refused_here_not_truncated() {
let d = Dir::new("longdesc");
d.write("SKILL.md", &front("x", &"d".repeat(DESCRIPTION_MAX + 1)));
let e = validate_dir(&d.0).unwrap_err().to_string();
assert!(e.contains(&DESCRIPTION_MAX.to_string()), "{e}");
}
/// A symlink is refused for being one, without asking where it points: the
/// installed copy is read as instruction by everyone the scope covers.
#[cfg(unix)]
#[test]
fn a_symlink_anywhere_inside_is_refused() {
let d = Dir::new("symlink");
d.write("SKILL.md", &front("x", "y"));
std::os::unix::fs::symlink("/etc/passwd", d.0.join("secrets.txt")).unwrap();
let e = validate_dir(&d.0).unwrap_err().to_string();
assert!(e.contains("symbolic link"), "{e}");
}
#[test]
fn dependency_manifests_are_reported_not_installed() {
let d = Dir::new("deps");
d.write("SKILL.md", &front("x", "y"));
d.write("requirements.txt", "requests\n");
assert_eq!(validate_dir(&d.0).unwrap().deps().as_deref(), Some("python"));
}
}
+298
View File
@@ -0,0 +1,298 @@
//! The skills freshness watcher (blueprint §8.2): freshness for edits made **by
//! hand on the box**.
//!
//! Every in-process writer — `skill_register`, `skill_delete`, the future UI —
//! already invalidates the prompt prefix directly; this task exists for the one
//! writer that is not in the process: the admin in SSH, a `git pull` of skills.
//! It is deliberately *not* on the correctness path: a missed event costs a
//! stale index for the twenty minutes of the prefix TTL, never a wrong one.
//!
//! The gate is the digest, and it is the whole design. `notify` is noisy — an
//! editor's save is a burst, an install is dozens of events, and every prompt
//! build *reads* every `SKILL.md` — so what reaches the bus is never "the fs
//! moved" but "the rendered index would differ" ([`super::tree_digest`]). A
//! script edit produces events and then silence, which is exactly the property
//! §6 asks for: the frozen prefix citing that skill has not aged.
use std::collections::{HashMap, HashSet};
use std::path::{Path, PathBuf};
use std::sync::Arc;
use std::time::Duration;
use core_api::system_bus::{SkillScope, SystemEvent, SystemEventBus};
use notify::{Config, EventKind, RecommendedWatcher, RecursiveMode, Watcher};
use tokio::sync::mpsc;
use tokio_util::sync::CancellationToken;
use tracing::{info, warn};
use crate::container::{SKILLS_DIR, SKILLS_USERS_DIR};
/// The quiet period after the last fs event before the digests are recomputed.
/// What matters is the settled state of the tree, not any intermediate one.
const DEBOUNCE: Duration = Duration::from_millis(800);
/// Spawns the watcher on the two trees (`{WD}/skills`, `{WD}/skills-users`),
/// emitting `SystemEvent::SkillsChanged` for each scope whose digest moved.
pub fn spawn(bus: Arc<SystemEventBus>, shutdown: CancellationToken) -> tokio::task::JoinHandle<()> {
let wd = std::env::current_dir().unwrap_or_else(|_| PathBuf::from("."));
tokio::spawn(run(bus, shutdown, wd, DEBOUNCE))
}
/// The watcher's body, split from [`spawn`] so the tests can point it at a
/// temporary `{WD}` and shorten the debounce.
async fn run(bus: Arc<SystemEventBus>, shutdown: CancellationToken, wd: PathBuf, debounce: Duration) {
// The fs backend reports **canonical** paths (on macOS `/var` is a symlink
// to `/private/var`, and FSEvents answers with the real one), so the roots
// must be canonical too or `classify` never matches and every event is
// silently dropped.
let wd = match wd.canonicalize() {
Ok(w) => w,
Err(e) => {
warn!(path = %wd.display(), error = %e, "skills-watch: cannot canonicalize the working directory, not started");
return;
}
};
let shared_dir = wd.join(SKILLS_DIR);
let users_dir = wd.join(SKILLS_USERS_DIR);
// Created rather than merely watched: these are instance data dirs that
// `ContainerManager::ensure` would create anyway, and on a box before its
// first user neither exists yet — a watcher that fails to install at boot
// would never notice the first tree appearing.
for dir in [&shared_dir, &users_dir] {
if let Err(e) = std::fs::create_dir_all(dir) {
warn!(path = %dir.display(), error = %e, "skills-watch: cannot create the tree, not started");
return;
}
}
let (tx, mut rx) = mpsc::unbounded_channel::<Vec<PathBuf>>();
let mut watcher = match RecommendedWatcher::new(
move |res: notify::Result<notify::Event>| {
let Ok(event) = res else { return };
// A pure read is never a change — and it matters here: every prompt
// build reads every `SKILL.md`, so IN_ACCESS / CLOSE_NOWRITE would
// re-digest the trees after every single conversation turn.
if matches!(event.kind, EventKind::Access(_)) {
return;
}
if event.paths.is_empty() {
return;
}
let _ = tx.send(event.paths);
},
Config::default(),
) {
Ok(w) => w,
Err(e) => {
warn!(error = %e, "skills-watch: watcher create failed, not started");
return;
}
};
for dir in [&shared_dir, &users_dir] {
if let Err(e) = watcher.watch(dir, RecursiveMode::Recursive) {
warn!(path = %dir.display(), error = %e, "skills-watch: watch install failed, not started");
return;
}
}
info!("skills-watch: watching the two skills trees");
// Baselines, taken before anything can be emitted: only a *change* from
// here on is worth an announcement. `empty` is the digest of a tree with no
// visible skill — a tree first seen in that state (e.g. created empty by
// `ensure` at container setup) alters no index and announces nothing.
let empty = super::tree_digest(&shared_dir.join("__never__"));
let mut shared_digest = super::tree_digest(&shared_dir);
let mut user_digests = digests_by_user(&users_dir);
let mut touched_shared = false;
let mut touched_users: HashSet<String> = HashSet::new();
let mut quiet: Option<std::pin::Pin<Box<tokio::time::Sleep>>> = None;
loop {
tokio::select! {
_ = shutdown.cancelled() => break,
paths = rx.recv() => {
let Some(paths) = paths else { break }; // watcher dropped
for p in &paths {
match classify(p, &shared_dir, &users_dir) {
Some(SkillScope::Global) => touched_shared = true,
Some(SkillScope::User(u)) => { touched_users.insert(u); }
None => {}
}
}
// Restart the quiet period on every event: a save burst or an
// install settles only when the events stop coming.
quiet = Some(Box::pin(tokio::time::sleep(debounce)));
}
// `pending` when disarmed: the arm below fires only once armed.
_ = async { match &mut quiet { Some(s) => s.as_mut().await, None => std::future::pending().await } } => {
quiet = None;
if touched_shared {
touched_shared = false;
let now = super::tree_digest(&shared_dir);
if now != shared_digest {
shared_digest = now;
info!("skills-watch: the group's tree changed, announcing");
bus.send(SystemEvent::SkillsChanged { scope: SkillScope::Global });
}
}
for uid in touched_users.drain() {
let now = super::tree_digest(&users_dir.join(&uid));
let old = user_digests.insert(uid.clone(), now.clone());
let changed = match old {
Some(old) => old != now,
None => now != empty,
};
if changed {
info!(user = %uid, "skills-watch: a member's tree changed, announcing");
bus.send(SystemEvent::SkillsChanged { scope: SkillScope::User(uid) });
}
}
}
}
}
info!("skills-watch: stopped");
}
/// Maps a changed fs path to the scope tree it touches.
///
/// `Path::starts_with` is component-wise, which is exactly what saves the
/// prefix trap here: `{WD}/skills-users/…` does **not** start with
/// `{WD}/skills`. An event on the `skills-users` root itself names no user
/// yet and is ignored — a new member's directory reports itself by its own
/// path.
fn classify(path: &Path, shared_dir: &Path, users_dir: &Path) -> Option<SkillScope> {
if path.starts_with(shared_dir) {
return Some(SkillScope::Global);
}
let rel = path.strip_prefix(users_dir).ok()?;
let uid = rel.components().next()?.as_os_str().to_str()?;
Some(SkillScope::User(uid.to_string()))
}
/// The baseline digests of every member tree present at startup.
fn digests_by_user(users_dir: &Path) -> HashMap<String, String> {
let Ok(entries) = std::fs::read_dir(users_dir) else { return HashMap::new() };
entries
.flatten()
.filter(|e| e.path().is_dir())
.filter_map(|e| e.file_name().into_string().ok())
.map(|uid| {
let d = super::tree_digest(&users_dir.join(&uid));
(uid, d)
})
.collect()
}
#[cfg(test)]
mod tests {
use super::*;
use crate::skills::tests_support::{Tree, valid};
/// The classification trap the whole design leans on: `skills-users` is a
/// **sibling** of `skills`, and a component-wise prefix check must not let
/// a member's edit fall into the group's scope.
#[test]
fn classify_never_confuses_the_two_sibling_trees() {
let wd = Path::new("/wd");
let shared = wd.join(SKILLS_DIR);
let users = wd.join(SKILLS_USERS_DIR);
assert_eq!(
classify(Path::new("/wd/skills/ics-import/SKILL.md"), &shared, &users),
Some(SkillScope::Global)
);
assert_eq!(classify(Path::new("/wd/skills"), &shared, &users), Some(SkillScope::Global));
assert_eq!(
classify(Path::new("/wd/skills-users/u1/spesa/SKILL.md"), &shared, &users),
Some(SkillScope::User("u1".into()))
);
assert_eq!(
classify(Path::new("/wd/skills-users/u1"), &shared, &users),
Some(SkillScope::User("u1".into()))
);
// The users root itself names nobody.
assert_eq!(classify(Path::new("/wd/skills-users"), &shared, &users), None);
// Anything else is not ours at all.
assert_eq!(classify(Path::new("/wd/homes/u1/x"), &shared, &users), None);
}
/// The gate itself, over a real tree: an edit the index cannot see passes
/// in silence; one it can see announces. (The pure half —
/// `the_digest_follows_the_index_not_the_files` — lives in `skills/mod.rs`;
/// this is the watcher half §15 asks for.)
#[test]
fn tree_digest_gates_on_what_the_index_can_see() {
let t = Tree::new("watch-digest", "daniele");
t.write("shared", "ics-import", &valid("ics-import", "Import an ICS feed."));
let dir = t.root.join(SKILLS_DIR);
let before = super::super::tree_digest(&dir);
// A script appears: events fire, the index does not move.
std::fs::write(dir.join("ics-import").join("run.py"), "print('hi')\n").unwrap();
assert_eq!(super::super::tree_digest(&dir), before);
// The body changes under the same description: still invisible.
t.write("shared", "ics-import", &valid("ics-import", "Import an ICS feed."));
assert_eq!(super::super::tree_digest(&dir), before);
// A re-description is what the prefix is made of: the digest moves.
t.write("shared", "ics-import", &valid("ics-import", "Import an ICS feed, then dedupe."));
assert_ne!(super::super::tree_digest(&dir), before);
// An empty or missing tree is the same digest as no tree.
assert_eq!(super::super::tree_digest(&dir.join("__never__")), super::super::digest(""));
}
/// End to end: a hand edit on the box reaches the bus — but only when the
/// index would notice.
#[tokio::test]
async fn a_hand_edit_announces_only_what_the_index_feels() {
let t = Tree::new("watch-e2e", "daniele");
t.write("shared", "ics-import", &valid("ics-import", "Import an ICS feed."));
t.write("mine", "spesa", &valid("spesa", "Reconcile the statement."));
let bus = Arc::new(SystemEventBus::new());
let mut rx = bus.subscribe();
let shutdown = CancellationToken::new();
let task = tokio::spawn(run(
Arc::clone(&bus),
shutdown.clone(),
t.root.clone(),
Duration::from_millis(100),
));
// Give the watcher a moment to install before touching the tree.
tokio::time::sleep(Duration::from_millis(300)).await;
// A script edit is fs activity the index cannot see: no announcement,
// however long we wait.
std::fs::write(t.root.join(SKILLS_DIR).join("ics-import").join("run.py"), "print(1)\n").unwrap();
let quiet = tokio::time::timeout(Duration::from_millis(1500), rx.recv()).await;
assert!(quiet.is_err(), "a script edit announced something: {quiet:?}");
// A re-description of a shared skill announces the group's scope.
t.write("shared", "ics-import", &valid("ics-import", "Import an ICS feed, then dedupe."));
let announced = tokio::time::timeout(Duration::from_secs(10), rx.recv())
.await
.expect("no SkillsChanged within 10s of a description edit")
.expect("bus closed");
assert!(
matches!(announced, SystemEvent::SkillsChanged { scope: SkillScope::Global }),
"expected SkillsChanged(Global), got {announced:?}"
);
// And one of an own skill announces only that member.
t.write("mine", "spesa", &valid("spesa", "Reconcile the statement, monthly."));
let announced = tokio::time::timeout(Duration::from_secs(10), rx.recv())
.await
.expect("no SkillsChanged within 10s of an own-skill edit")
.expect("bus closed");
assert!(
matches!(announced, SystemEvent::SkillsChanged { scope: SkillScope::User(ref u) } if u == "u1"),
"expected SkillsChanged(User(u1)), got {announced:?}"
);
shutdown.cancel();
let _ = tokio::time::timeout(Duration::from_secs(2), task).await;
}
}
+850
View File
@@ -0,0 +1,850 @@
//! `fetch_repo` — downloads a subtree of a **public git repository** into the
//! caller's workspace (blueprint §7.5).
//!
//! It exists for what a `git clone` does **not** do, and the name says so on
//! purpose: it is shallow, it checks out only the requested subtree, it drops
//! `.git`, it refuses symlinks and oversized downloads before a byte reaches the
//! destination, and it leaves a `.source.json` provenance ticket (URL, sub-path,
//! commit SHA, date) next to the files — the one piece of traceability a clone
//! never writes, without which "where did this come from?" and "did it change
//! upstream?" have no answer later.
//!
//! It deliberately **installs nothing**: files land in `destination`, and if
//! what was downloaded is a skill the installation is a separate
//! `skill_register`, with its own approval card over readable files. A
//! download-and-install in one step would move the human's only review moment
//! onto a card showing *a URL*, and a URL is not reviewable.
//!
//! Network egress stays inside the caller's **container** (`docker exec git …`),
//! never in the Skald process — the sandbox's network identity, not the
//! server's. The sanitization and the move into `destination` run host-side on
//! the bind-mounted staging directory, so the two sides never disagree about
//! what landed.
use std::path::{Component, Path, PathBuf};
use std::process::Stdio;
use std::sync::Arc;
use std::time::Duration;
use anyhow::{Context, Result, bail};
use core_api::user_fs::UserFs;
use serde_json::{Value, json};
use crate::skills::install::{Provenance, SOURCE_FILE, copy_tree};
use crate::tools::fs::resolve_host_path;
use crate::tools::{SimpleExecution, Tool, ToolContext, ToolDescriptionLength, ToolExecution, ToolResult};
/// Ceiling on how many files a download may carry. Generous for working
/// material (reference trees, examples, configs); far below a bulk mirror,
/// which this tool is not for.
pub const FETCH_MAX_FILES: usize = 2000;
/// Ceiling on a download's total size (64 MiB).
pub const FETCH_MAX_BYTES: u64 = 64 * 1024 * 1024;
/// Wall-clock bound on the in-container `git` work. Enforced twice: `timeout`
/// inside the script (so a killed client cannot leave a `git` running in the
/// container) and a Rust-side timeout around the `docker` child as backstop.
const CLONE_TIMEOUT: Duration = Duration::from_secs(300);
/// The limits a fetch enforces, as data so a test can shrink them.
#[derive(Debug, Clone, Copy)]
pub(crate) struct Limits {
max_files: usize,
max_bytes: u64,
}
impl Default for Limits {
fn default() -> Self {
Self { max_files: FETCH_MAX_FILES, max_bytes: FETCH_MAX_BYTES }
}
}
/// What a fetch delivered, for the tool's answer to the model.
#[derive(Debug)]
pub(crate) struct Fetched {
files: usize,
bytes: u64,
commit: String,
}
// ── The cloner seam ───────────────────────────────────────────────────────────
/// How a repository gets cloned, as a trait so the tests run the **same script**
/// against a local fixture repo without a Docker daemon.
#[async_trait::async_trait]
pub(crate) trait RepoCloner: Send + Sync {
/// Clones `url` — only `sub_path`, when given — into a fresh `repo/`
/// directory under the staging dir, and returns the checked-out commit SHA.
/// `host_dir` and `container_dir` name the **same** staging directory on the
/// two sides of the home bind mount; implementations use the side they run
/// on.
async fn clone(
&self,
url: &str,
sub_path: Option<&str>,
host_dir: &Path,
container_dir: &Path,
) -> Result<String>;
}
/// The one clone script, run through `sh -c … _ <url> <sub> <dir> <timeout>`
/// with every value **positional**, so a URL or path containing quotes or
/// `$(…)` is data and not shell syntax — the same rule `exec_fs` follows.
///
/// `--filter=blob:none` keeps a big repository's history and untouched blobs
/// off the wire; not every server supports it, so a filtered clone that fails
/// is retried unfiltered rather than reported. `--sparse` plus
/// `sparse-checkout set` materializes only the requested subtree.
const CLONE_SCRIPT: &str = r#"
set -eu
export GIT_TERMINAL_PROMPT=0
TW=""
if [ "$4" != "0" ]; then TW="timeout $4"; fi
mkdir -p -- "$3"
if [ -z "$2" ]; then
$TW git clone --quiet --depth 1 "$1" "$3/repo"
else
if ! $TW git clone --quiet --depth 1 --filter=blob:none --sparse "$1" "$3/repo"; then
rm -rf -- "$3/repo"
$TW git clone --quiet --depth 1 --sparse "$1" "$3/repo"
fi
git -C "$3/repo" sparse-checkout set "$2"
fi
git -C "$3/repo" rev-parse HEAD
"#;
/// Production cloner: runs [`CLONE_SCRIPT`] **inside the caller's container**
/// via `docker exec`, so the egress keeps the sandbox's network identity.
pub(crate) struct ContainerGit {
container: String,
}
#[async_trait::async_trait]
impl RepoCloner for ContainerGit {
async fn clone(
&self,
url: &str,
sub_path: Option<&str>,
_host_dir: &Path,
container_dir: &Path,
) -> Result<String> {
let dir = container_dir.to_string_lossy().into_owned();
let out = run_positional(
"docker",
&[
"exec".into(),
self.container.clone(),
"sh".into(),
"-c".into(),
CLONE_SCRIPT.into(),
"_".into(),
url.into(),
sub_path.unwrap_or("").into(),
dir,
CLONE_TIMEOUT.as_secs().saturating_sub(20).to_string(),
],
CLONE_TIMEOUT,
)
.await?;
parse_sha(&out)
}
}
/// Runs `program argv…` capturing stdout, with `kill_on_drop` plus a hard
/// timeout. On timeout the dropped child is killed — and for the production
/// cloner the script's own `timeout` is what stops the `git` the dead client
/// would otherwise leave behind in the container.
async fn run_positional(program: &str, argv: &[String], timeout: Duration) -> Result<String> {
let mut cmd = tokio::process::Command::new(program);
cmd.args(argv)
.stdin(Stdio::null())
.stdout(Stdio::piped())
.stderr(Stdio::piped())
.kill_on_drop(true);
let child = cmd
.spawn()
.with_context(|| format!("failed to spawn `{program}`"))?;
let out = tokio::time::timeout(timeout, child.wait_with_output())
.await
.map_err(|_| anyhow::anyhow!("timed out after {}s", timeout.as_secs()))?
.with_context(|| format!("`{program}` failed to report"))?;
if !out.status.success() {
bail!("{}", String::from_utf8_lossy(&out.stderr).trim());
}
Ok(String::from_utf8_lossy(&out.stdout).into_owned())
}
/// The clone script's last stdout line is the `rev-parse HEAD` answer.
fn parse_sha(stdout: &str) -> Result<String> {
stdout
.lines()
.rev()
.map(str::trim)
.find(|l| !l.is_empty())
.map(str::to_string)
.ok_or_else(|| anyhow::anyhow!("the clone produced no commit id"))
}
// ── The fetch itself ──────────────────────────────────────────────────────────
/// Drops the staging directory however the fetch ends — mid-copy failures
/// included — so a refused or interrupted download leaves no litter in the
/// caller's home.
struct Staging(PathBuf);
impl Drop for Staging {
fn drop(&mut self) {
let _ = std::fs::remove_dir_all(&self.0);
// Take the `.skald` parent too when nothing else is using it — a
// refused download must leave no litter, and `remove_dir` on a
// non-empty directory is the cheap no-op that says so.
if let Some(parent) = self.0.parent() {
let _ = std::fs::remove_dir(parent);
}
}
}
/// Downloads `url` (only `sub_path`, when given) into the agent path
/// `destination`, sanitizes host-side, and writes the provenance ticket.
pub(crate) async fn fetch(
fs: &UserFs,
url: &str,
sub_path: Option<&str>,
destination: &str,
cloner: &dyn RepoCloner,
limits: Limits,
) -> Result<Fetched> {
// The root spellings converge here, not only in the tool's argument check:
// `Some(".")` reaching the script would take the *sparse* arm and
// `sparse-checkout set .` would materialize the root files and nothing else.
let sub_path = sub_path.and_then(|s| {
let s = s.trim();
if s.is_empty() || s == "." { None } else { Some(s) }
});
// An absolute spelling is container vocabulary: map it back to an agent
// path first, so the writability check below speaks the same language as
// the relative case. Landing nowhere means container-only, and a download
// there could never be reviewed or registered from the host side.
let agent = if Path::new(destination).is_absolute() {
match fs.container_to_agent(Path::new(destination)) {
Some(mapped) => mapped,
None => bail!(
"`{destination}` exists only inside your container. Download somewhere you \
can reach from both sides your home (`~/`), a project or a shared folder."
),
}
} else {
destination.to_string()
};
if !fs.can_write_to(&agent) {
bail!(
"`{destination}` is read-only for you (everything under `skills/` always is — a \
skill is *installed* with skill_register, never downloaded into place). Pick a \
folder in your home, or a project/shared folder you can write to."
);
}
let host_dest = resolve_host_path(fs, &agent)?;
if host_dest.exists() {
if !host_dest.is_dir() {
bail!("`{destination}` already exists and is not a folder. Pick a new path.");
}
if std::fs::read_dir(&host_dest)
.with_context(|| format!("cannot read {}", host_dest.display()))?
.next()
.is_some()
{
bail!(
"`{destination}` already exists and is not empty — fetch_repo never merges \
into files that are already there. Pick a new folder, or empty that one."
);
}
}
// Staging lives inside the home bind mount: the container's `git` writes
// there, and the host-side half then validates and moves from the same
// bytes. `.skald` is transient state, hidden from the agent's listings.
let tag = uuid::Uuid::new_v4().simple().to_string();
let stage_host = fs.home_host.join(".skald").join(format!("fetch-{tag}"));
let stage_container = fs.container_home.join(".skald").join(format!("fetch-{tag}"));
std::fs::create_dir_all(&stage_host)
.with_context(|| format!("cannot stage in {}", stage_host.display()))?;
let _staging = Staging(stage_host.clone());
let commit = cloner
.clone(url, sub_path, &stage_host, &stage_container)
.await?;
// `.git` never crosses into the destination. For a subtree fetch it could
// not travel anyway; for a root fetch this removal is the guarantee, so it
// runs in both cases rather than being the subtree case's good fortune.
let dotgit = stage_host.join("repo").join(".git");
if dotgit.is_dir() {
std::fs::remove_dir_all(&dotgit).context("cannot drop the `.git` history")?;
} else if dotgit.exists() {
std::fs::remove_file(&dotgit).context("cannot drop the `.git` history")?;
}
let extracted = match sub_path {
None => stage_host.join("repo"),
Some(sub) => stage_host.join("repo").join(sub),
};
if !extracted.is_dir() {
match sub_path {
Some(sub) => bail!("the repository has no `{sub}` folder."),
None => bail!("the clone produced no files."),
}
}
let mut found = Sanitized::default();
sanitize(&extracted, Path::new(""), &limits, &mut found)?;
std::fs::create_dir_all(&host_dest)
.with_context(|| format!("cannot create {destination}"))?;
copy_tree(&extracted, &host_dest)?;
let ticket = Provenance {
url: url.to_string(),
sub_path: sub_path.map(str::to_string),
commit: Some(commit.clone()),
fetched_at: Some(chrono::Utc::now().to_rfc3339()),
installed_at: None,
};
let json = serde_json::to_string_pretty(&ticket)?;
std::fs::write(host_dest.join(SOURCE_FILE), json)
.with_context(|| format!("cannot write the {SOURCE_FILE} ticket"))?;
Ok(Fetched { files: found.files, bytes: found.bytes, commit })
}
/// Recursive walk that enforces the download rules while it counts. Every
/// refusal names what to fix — the caller is a model that will retry, and
/// "download rejected" would only produce a guess.
#[derive(Default)]
struct Sanitized {
files: usize,
bytes: u64,
}
fn sanitize(dir: &Path, rel: &Path, limits: &Limits, acc: &mut Sanitized) -> Result<()> {
let mut entries: Vec<_> = std::fs::read_dir(dir)
.map_err(|e| anyhow::anyhow!("cannot read {}: {e}", dir.display()))?
.filter_map(|e| e.ok())
.collect();
entries.sort_by_key(|e| e.file_name());
for entry in entries {
let name = entry.file_name();
let child_rel = rel.join(&name);
// `symlink_metadata` does NOT follow the link, which is the point: a
// link is refused for what it is, before anything asks where it points.
let meta = std::fs::symlink_metadata(entry.path())
.map_err(|e| anyhow::anyhow!("cannot stat {}: {e}", child_rel.display()))?;
if meta.file_type().is_symlink() {
bail!(
"the repository contains a symbolic link (`{}`). fetch_repo does not bring \
links over download the real file instead.",
child_rel.display()
);
}
if meta.is_dir() {
sanitize(&entry.path(), &child_rel, limits, acc)?;
continue;
}
if !meta.is_file() {
bail!("`{}` is not a regular file.", child_rel.display());
}
acc.files += 1;
acc.bytes += meta.len();
if acc.files > limits.max_files {
bail!(
"that subtree holds more than {} files — fetch_repo is for working material, \
not bulk mirrors.",
limits.max_files
);
}
if acc.bytes > limits.max_bytes {
bail!(
"that subtree is over {} MiB — fetch_repo is for working material, not bulk \
data. Clone it by hand with execute_cmd if you really need it all.",
limits.max_bytes / (1024 * 1024)
);
}
}
Ok(())
}
// ── Argument parsing ──────────────────────────────────────────────────────────
/// Public repositories over https only: ssh would need credentials the caller
/// does not have (and should not), and a local path is not a repository fetch.
fn check_url(url: &str) -> Result<()> {
if url.starts_with("https://") || url.starts_with("http://") {
Ok(())
} else {
bail!(
"fetch_repo downloads from public repositories over https: `{url}`. (No ssh or \
local paths no credentials are involved, and none should be.)"
)
}
}
/// Normalizes the `sub_path` argument: `""` / `"."` mean the repository root,
/// anything else must be a relative path that stays inside the repo.
fn check_sub_path(raw: &str) -> Result<Option<String>> {
let s = raw.trim();
if s.is_empty() || s == "." {
return Ok(None);
}
let p = Path::new(s);
if p.is_absolute() {
bail!("`sub_path` is relative to the repository root, not absolute: `{s}`");
}
if p.components().any(|c| matches!(c, Component::ParentDir)) {
bail!("`sub_path` must stay inside the repository: `{s}`");
}
let normalized: PathBuf = p
.components()
.filter_map(|c| match c {
Component::Normal(x) => Some(x),
_ => None,
})
.collect();
if normalized.as_os_str().is_empty() {
return Ok(None);
}
Ok(Some(normalized.to_string_lossy().replace('\\', "/")))
}
// ── The tool ──────────────────────────────────────────────────────────────────
pub struct FetchRepo;
impl Tool for FetchRepo {
fn name(&self) -> &str { crate::tools::tool_names::FETCH_REPO }
fn category(&self) -> crate::tools::ToolCategory { crate::tools::ToolCategory::Config }
fn display_name(&self) -> &str { "Fetch Repository" }
fn description(&self) -> &str {
"Download a subtree of a public git repository (https only) into a folder you can \
write your home, a project or a shared folder. Shallow and sanitized: no `.git` \
history, no symbolic links, size limits apply. It installs NOTHING: the files are \
left at `destination`, plus a `.source.json` ticket recording the URL, sub-path and \
exact commit. If the download is a skill, review the files and then install it with \
`skill_register` never download straight into `skills/`, which is read-only. \
`destination` must not exist yet (or be an empty folder); a path that exists only \
inside the container, such as /tmp, is refused."
}
fn parameters_schema(&self) -> Value {
json!({
"type": "object",
"required": ["url", "destination"],
"properties": {
"url": {
"type": "string",
"description": "The repository's https URL, e.g. \
\"https://github.com/anthropics/skills\"."
},
"sub_path": {
"type": "string",
"description": "Folder inside the repository to download, e.g. \
\"skills/ics-import\". Omit, or pass \"\" or \".\", \
to take the whole repository."
},
"destination": {
"type": "string",
"description": "Agent path of the folder to fill, e.g. \
\"~/downloads/ics-import\". Created if missing; must be \
empty if it already exists."
}
}
})
}
fn describe(&self, args: &Value, _length: ToolDescriptionLength) -> String {
let url = args["url"].as_str().unwrap_or("?");
let dest = args["destination"].as_str().unwrap_or("?");
match args["sub_path"].as_str().filter(|s| !s.is_empty() && *s != ".") {
Some(sub) => format!("download `{sub}` of {url} into `{dest}`"),
None => format!("download {url} into `{dest}`"),
}
}
fn target_path(&self, args: &Value) -> Option<String> {
args["destination"].as_str().map(str::to_string)
}
fn run_with<'a>(&'a self, ctx: &ToolContext, args: Value) -> Box<dyn ToolExecution + 'a> {
let fs = Arc::clone(&ctx.fs);
Box::new(SimpleExecution::new(Box::pin(async move {
let url = args["url"]
.as_str()
.ok_or_else(|| anyhow::anyhow!("missing required argument `url`"))?;
check_url(url)?;
let sub = match args["sub_path"].as_str() {
Some(raw) => check_sub_path(raw)?,
None => None,
};
let destination = args["destination"]
.as_str()
.ok_or_else(|| anyhow::anyhow!("missing required argument `destination`"))?;
let cloner = ContainerGit { container: fs.container_name.clone() };
let done = fetch(&fs, url, sub.as_deref(), destination, &cloner, Limits::default()).await?;
let short = done.commit.chars().take(7).collect::<String>();
Ok(ToolResult::Text(format!(
"Fetched {} files ({:.0} KiB) from {url} @ {short} into {destination}.\n\
A `{SOURCE_FILE}` next to the files records where they came from.\n\
Nothing was installed if this is a skill, review the files and then call \
`skill_register` on `{destination}`.",
done.files,
done.bytes as f64 / 1024.0,
)))
} )))
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::skills::tests_support::Tree;
// ── Fixture: a local git repository, cloned by the same script ────────────
/// The test cloner runs the **same** [`CLONE_SCRIPT`] on the host (the dev
/// machine has git; `$4 = 0` selects the no-`timeout` arm, since macOS
/// lacks the GNU binary). Only the Docker wrapper is faked away.
struct HostGit;
#[async_trait::async_trait]
impl RepoCloner for HostGit {
async fn clone(
&self,
url: &str,
sub_path: Option<&str>,
host_dir: &Path,
_container_dir: &Path,
) -> Result<String> {
let out = run_positional(
"sh",
&[
"-c".into(),
CLONE_SCRIPT.into(),
"_".into(),
url.into(),
sub_path.unwrap_or("").into(),
host_dir.to_string_lossy().into_owned(),
"0".into(),
],
Duration::from_secs(60),
)
.await?;
parse_sha(&out)
}
}
struct Repo(PathBuf);
impl Repo {
/// A fixture repository with two top-level folders and a root file.
fn new(tag: &str) -> Self {
let dir = std::env::temp_dir().join(format!(
"skald-fetchrepo-{tag}-{}-{:?}",
std::process::id(),
std::thread::current().id()
));
let _ = std::fs::remove_dir_all(&dir);
std::fs::create_dir_all(dir.join("alpha")).unwrap();
std::fs::create_dir_all(dir.join("beta/nested")).unwrap();
std::fs::write(dir.join("README.md"), "# fixture\n").unwrap();
std::fs::write(dir.join("alpha/a.txt"), "alpha\n").unwrap();
std::fs::write(dir.join("alpha/second.txt"), "second\n").unwrap();
std::fs::write(dir.join("beta/b.txt"), "beta\n").unwrap();
std::fs::write(dir.join("beta/nested/deep.txt"), "deep\n").unwrap();
let r = Repo(dir);
r.git(&["init", "-q"]);
r.git(&["add", "-A"]);
r.git(&["-c", "user.email=t@t", "-c", "user.name=t", "commit", "-qm", "init"]);
r
}
fn git(&self, args: &[&str]) -> String {
let out = std::process::Command::new("git")
.args(args)
.current_dir(&self.0)
.output()
.unwrap();
assert!(out.status.success(), "git {args:?}: {}", String::from_utf8_lossy(&out.stderr));
String::from_utf8_lossy(&out.stdout).trim().to_string()
}
fn head(&self) -> String {
self.git(&["rev-parse", "HEAD"])
}
fn write(&self, rel: &str, body: &str) {
std::fs::write(self.0.join(rel), body).unwrap();
}
fn commit_all(&self) {
self.git(&["add", "-A"]);
self.git(&["-c", "user.email=t@t", "-c", "user.name=t", "commit", "-qm", "more"]);
}
}
impl Drop for Repo {
fn drop(&mut self) {
let _ = std::fs::remove_dir_all(&self.0);
}
}
async fn fetch_with(fs: &UserFs, url: &str, sub: Option<&str>, dest: &str) -> Result<Fetched> {
fetch(fs, url, sub, dest, &HostGit, Limits::default()).await
}
fn listing(dir: &Path) -> Vec<String> {
let mut out = Vec::new();
let mut stack = vec![dir.to_path_buf()];
while let Some(d) = stack.pop() {
for e in std::fs::read_dir(&d).unwrap().filter_map(|e| e.ok()) {
let p = e.path();
if p.is_dir() {
stack.push(p.clone());
}
out.push(p.strip_prefix(dir).unwrap().to_string_lossy().replace('\\', "/"));
}
}
out.sort();
out
}
// ── The fetch ─────────────────────────────────────────────────────────────
/// Only the requested subtree lands in the destination — the rest of the
/// repository never crosses over.
#[tokio::test]
async fn only_the_requested_subtree_lands_in_the_destination() {
let repo = Repo::new("sub");
let t = Tree::new("sub", "daniele");
fetch_with(&t.fs, &repo.0.to_string_lossy(), Some("alpha"), "~/dl")
.await
.unwrap();
let files = listing(&t.root.join("homes/u1/dl"));
assert!(files.contains(&"a.txt".to_string()), "{files:?}");
assert!(files.contains(&"second.txt".to_string()), "{files:?}");
assert!(!files.iter().any(|f| f.contains("beta") || f.contains("README")), "{files:?}");
}
/// A root fetch gets everything but the `.git` history — the one thing the
/// tool exists to keep out.
#[tokio::test]
async fn a_root_fetch_gets_everything_but_the_git_history() {
let repo = Repo::new("root");
let t = Tree::new("root", "daniele");
for sub in [None, Some(""), Some(".")] {
let dest = format!("~/dl-{}", sub.unwrap_or("bare").replace('.', "dot"));
fetch_with(&t.fs, &repo.0.to_string_lossy(), sub, &dest)
.await
.unwrap_or_else(|e| panic!("sub {sub:?}: {e}"));
let host = t.root.join("homes/u1").join(&dest[2..]);
let files = listing(&host);
assert!(files.iter().any(|f| f == "beta/nested/deep.txt"), "{files:?}");
assert!(!files.iter().any(|f| f.contains(".git")), "{files:?}");
}
}
/// The ticket records the exact commit — the field that makes a later
/// upstream change detectable.
#[tokio::test]
async fn the_provenance_ticket_carries_the_commit() {
let repo = Repo::new("prov");
let t = Tree::new("prov", "daniele");
fetch_with(&t.fs, &repo.0.to_string_lossy(), Some("alpha"), "~/dl")
.await
.unwrap();
let raw = std::fs::read_to_string(t.root.join("homes/u1/dl/.source.json")).unwrap();
let p: Provenance = serde_json::from_str(&raw).unwrap();
assert_eq!(p.url, repo.0.to_string_lossy());
assert_eq!(p.sub_path.as_deref(), Some("alpha"));
assert_eq!(p.commit.as_deref(), Some(repo.head().as_str()));
assert!(p.fetched_at.is_some());
assert!(p.installed_at.is_none(), "install stamps that one");
}
/// A `sub_path` the repository does not have is a speaking refusal, not an
/// empty destination.
#[tokio::test]
async fn a_missing_sub_path_is_refused() {
let repo = Repo::new("nosub");
let t = Tree::new("nosub", "daniele");
let e = fetch_with(&t.fs, &repo.0.to_string_lossy(), Some("gamma"), "~/dl")
.await
.unwrap_err();
assert!(e.to_string().contains("no `gamma` folder"), "{e}");
assert!(!t.root.join("homes/u1/dl").exists(), "nothing landed");
assert!(!t.root.join("homes/u1/.skald").exists(), "no staging litter");
}
/// A symlink is refused for being one, wherever it points — and the
/// destination stays untouched.
#[cfg(unix)]
#[tokio::test]
async fn a_symlink_in_the_repo_is_refused() {
let repo = Repo::new("symlink");
std::os::unix::fs::symlink("/etc/passwd", repo.0.join("alpha/secrets")).unwrap();
repo.commit_all();
let t = Tree::new("symlink", "daniele");
let e = fetch_with(&t.fs, &repo.0.to_string_lossy(), Some("alpha"), "~/dl")
.await
.unwrap_err();
assert!(e.to_string().contains("symbolic link"), "{e}");
assert!(!t.root.join("homes/u1/dl").exists(), "nothing landed");
}
/// Over the file cap the refusal comes **before** a byte is written to the
/// destination (the cap here is the test's, not the shipped one).
#[tokio::test]
async fn over_the_file_cap_is_refused_before_writing() {
let repo = Repo::new("cap");
let t = Tree::new("cap", "daniele");
let limits = Limits { max_files: 2, max_bytes: FETCH_MAX_BYTES };
let e = fetch(&t.fs, &repo.0.to_string_lossy(), None, "~/dl", &HostGit, limits)
.await
.unwrap_err();
assert!(e.to_string().contains("more than 2 files"), "{e}");
assert!(!t.root.join("homes/u1/dl").exists(), "nothing landed");
}
// ── The destination rules ─────────────────────────────────────────────────
/// `skills/` is read-only in both directions: a download is never the way
/// in — `skill_register` is.
#[tokio::test]
async fn the_skills_tree_is_not_a_destination() {
let repo = Repo::new("ro");
let t = Tree::new("ro", "daniele");
let e = fetch_with(&t.fs, &repo.0.to_string_lossy(), None, "skills/shared/x")
.await
.unwrap_err();
assert!(e.to_string().contains("read-only"), "{e}");
}
/// A container-only path is refused with a message that says where to go —
/// the download would be unreachable from the host half.
#[tokio::test]
async fn a_container_only_destination_is_refused() {
let repo = Repo::new("conly");
let t = Tree::new("conly", "daniele");
let e = fetch_with(&t.fs, &repo.0.to_string_lossy(), None, "/tmp/dl")
.await
.unwrap_err();
assert!(e.to_string().contains("only inside your container"), "{e}");
}
/// Existing is fine only while it is empty; a non-empty folder is never
/// merged into.
#[tokio::test]
async fn an_existing_destination_must_be_empty() {
let repo = Repo::new("exists");
let t = Tree::new("exists", "daniele");
let home = t.root.join("homes/u1");
std::fs::create_dir_all(home.join("empty")).unwrap();
std::fs::create_dir_all(home.join("full")).unwrap();
std::fs::write(home.join("full/keep.txt"), "mine\n").unwrap();
std::fs::write(home.join("afile"), "file\n").unwrap();
fetch_with(&t.fs, &repo.0.to_string_lossy(), Some("alpha"), "~/empty")
.await
.unwrap();
let e = fetch_with(&t.fs, &repo.0.to_string_lossy(), None, "~/full")
.await
.unwrap_err();
assert!(e.to_string().contains("not empty"), "{e}");
let e = fetch_with(&t.fs, &repo.0.to_string_lossy(), None, "~/afile")
.await
.unwrap_err();
assert!(e.to_string().contains("not a folder"), "{e}");
}
/// The absolute spelling of a mounted path is the same destination — the
/// container vocabulary maps back before any check runs.
#[tokio::test]
async fn an_absolute_home_spelling_works() {
let repo = Repo::new("abs");
let t = Tree::new("abs", "daniele");
fetch_with(&t.fs, &repo.0.to_string_lossy(), Some("alpha"), "/root/dl")
.await
.unwrap();
assert!(t.root.join("homes/u1/dl/a.txt").exists());
}
// ── Argument parsing ──────────────────────────────────────────────────────
#[test]
fn only_public_https_urls_pass() {
assert!(check_url("https://github.com/x/y").is_ok());
assert!(check_url("http://example.com/r.git").is_ok());
for bad in ["git@github.com:x/y.git", "ssh://git@h/r", "file:///tmp/r", "/tmp/r"] {
assert!(check_url(bad).is_err(), "accepted `{bad}`");
}
}
#[test]
fn the_root_spellings_mean_no_subtree() {
assert_eq!(check_sub_path("").unwrap(), None);
assert_eq!(check_sub_path(".").unwrap(), None);
assert_eq!(check_sub_path("a/b").unwrap(), Some("a/b".to_string()));
assert_eq!(check_sub_path("./a/./b").unwrap(), Some("a/b".to_string()));
assert!(check_sub_path("/etc").is_err());
assert!(check_sub_path("../out").is_err());
assert!(check_sub_path("a/../../out").is_err());
}
// ── The crossing into an installation ─────────────────────────────────────
/// What `fetch_repo` leaves behind is what `skill_register` picks up: the
/// ticket crosses into the installed skill, stamped with the install date.
#[tokio::test]
async fn a_fetched_skill_registers_with_its_provenance() {
let repo = Repo::new("cross");
repo.write("alpha/SKILL.md", &crate::skills::tests_support::valid("alpha-x", "The x."));
repo.commit_all();
let t = Tree::new("cross", "daniele");
fetch_with(&t.fs, &repo.0.to_string_lossy(), Some("alpha"), "~/draft")
.await
.unwrap();
let host = t.root.join("homes/u1/draft");
let done = crate::skills::install::install(&t.fs, crate::skills::Scope::Own, &host).unwrap();
assert_eq!(done.id, "alpha-x");
let p = crate::skills::install::read_provenance(
&t.root.join("skills-users/u1/alpha-x"),
)
.unwrap();
assert_eq!(p.commit.as_deref(), Some(repo.head().as_str()));
assert!(p.installed_at.is_some(), "the install stamped its date");
}
}
+140 -4
View File
@@ -16,7 +16,7 @@ use anyhow::{Context, Result};
use serde_json::Value;
use sqlx::SqlitePool;
use core_api::user_fs::UserFs;
use core_api::user_fs::{RouteError, UserFs};
use crate::tools::{SimpleExecution, ToolExecution, ToolRegistry, ToolResult};
@@ -221,9 +221,11 @@ pub(super) fn write_string(user_path: &str, content: &str) -> Result<()> {
/// from inside the container (`execute_cmd`), a symlink planted there that points
/// outside the home is caught by canonicalizing and prefix-checking against the base.
pub(crate) fn resolve_host_path(fs: &UserFs, agent_path: &str) -> Result<PathBuf> {
let (base, tail) = fs.host_base_and_tail(agent_path).ok_or_else(|| {
anyhow::anyhow!("no such shared folder, or you are not a member: {agent_path}")
})?;
let (base, tail) = match fs.host_base_and_tail(agent_path) {
Ok(pair) => pair,
Err(RouteError::Denied(msg)) => anyhow::bail!(msg),
Err(RouteError::SkillAlias { id, tail }) => resolve_skill_alias(fs, &id, &tail)?,
};
// Canonicalize both sides so the prefix check is symlink-aware.
let base_canon = canonicalize_for_policy(&base.to_string_lossy(), Path::new("/"));
let joined = base.join(&tail);
@@ -234,6 +236,55 @@ pub(crate) fn resolve_host_path(fs: &UserFs, agent_path: &str) -> Result<PathBuf
Ok(canon)
}
/// Resolves the tolerant bare-id alias `skills/<id>/…` — the shortest spelling of a
/// skill path, and therefore the one a model reaches for on its own, both out of
/// habit and because skill bodies written elsewhere cite it that way.
///
/// It resolves **only when the id lives in exactly one** of the two trees. A
/// collision fails loudly, listing both full paths, rather than letting either win:
/// the personal tree winning would mean a silent divergence from the group's set,
/// the group's winning would mean the member's own work is ignored, and neither is
/// something to decide behind the model's back. With the full path printed in the
/// index the disambiguation is free anyway — they are two different lines.
///
/// The root itself is probed last, so the signpost `skills/README.md` reads like any
/// other file rather than being the one path in the tree that fails.
fn resolve_skill_alias(fs: &UserFs, id: &str, tail: &str) -> Result<(PathBuf, String)> {
let found: Vec<(String, PathBuf)> = fs
.skill_alias_candidates(id)
.into_iter()
.filter(|(_, host)| host.is_dir())
.collect();
match found.len() {
1 => {
let (_, host) = found.into_iter().next().expect("len checked");
Ok((host, tail.to_string()))
}
0 => {
// Not a skill id. It may still be something in the root mount — the
// signpost README — before it is nothing at all.
if let Some(sk) = fs.skills.as_ref().filter(|sk| sk.root_host.join(id).exists()) {
return Ok((sk.root_host.clone(), agent_join_str(id, tail)));
}
anyhow::bail!(fs.skill_route_hint(id))
}
_ => {
let paths: Vec<String> = found.into_iter().map(|(agent, _)| agent).collect();
anyhow::bail!(
"`skills/{id}` is ambiguous — that id exists in more than one place. \
Use the full path: {}",
paths.join(" or ")
)
}
}
}
/// Joins a first segment with a possibly-empty tail, for a path relative to a base.
fn agent_join_str(head: &str, tail: &str) -> String {
if tail.is_empty() { head.to_string() } else { format!("{head}/{tail}") }
}
/// Resolve a path arriving from the show-file / file-viewer surface into
/// `(host_abs, agent_display)`, scoped to the caller's workspace.
///
@@ -990,4 +1041,89 @@ mod tests {
let _ = std::fs::remove_dir_all(&udir);
let _ = std::fs::remove_dir_all(&sdir);
}
/// The skills tree end to end, on disk: both scopes resolve, the bare-id alias
/// resolves only when unambiguous, a collision fails loudly naming both paths,
/// an invented scope segment is refused with a hint instead of quietly becoming
/// a file in the home, and containment holds inside a skill exactly as it does
/// in the home.
#[cfg(unix)]
#[test]
fn skills_tree_routes_and_contains() {
use core_api::user_fs::SkillMounts;
let root = std::env::temp_dir().join(format!("skald-skills-{}", std::process::id()));
let home = root.join("homes").join("u1");
let skroot = root.join(".skills-root").join("u1");
let shared = root.join("skills");
let own = root.join("skills-users").join("u1");
let _ = std::fs::remove_dir_all(&root);
for d in [&home, &skroot, &shared, &own] {
std::fs::create_dir_all(d).unwrap();
}
std::fs::write(skroot.join("README.md"), "signpost").unwrap();
std::fs::create_dir_all(shared.join("ics-import")).unwrap();
std::fs::write(shared.join("ics-import").join("SKILL.md"), "shared one").unwrap();
std::fs::create_dir_all(own.join("spesa")).unwrap();
std::fs::write(own.join("spesa").join("SKILL.md"), "mine").unwrap();
let fs = UserFs::new(
"u1",
home.clone(),
"skald-u1",
PathBuf::from("/root"),
vec![],
vec![],
None,
)
.with_skills(SkillMounts {
root_host: skroot.clone(),
shared_host: shared.clone(),
own_host: own.clone(),
own_username: "daniele".into(),
});
let read = |p: &str| std::fs::read_to_string(resolve_host_path(&fs, p).unwrap()).unwrap();
// Both scopes, spelled fully.
assert_eq!(read("skills/shared/ics-import/SKILL.md"), "shared one");
assert_eq!(read("skills/daniele/spesa/SKILL.md"), "mine");
// The container spelling reaches the same files (reverse-mapped by
// `resolve_target`, which is what an absolute path goes through).
let via_container = match resolve_target(&fs, "/root/skills/shared/ics-import/SKILL.md").unwrap() {
FsTarget::Host(h) => h,
FsTarget::Container { path, .. } => panic!("mounted skill routed to the container as {path:?}"),
};
assert_eq!(std::fs::read_to_string(via_container).unwrap(), "shared one");
// The signpost is readable rather than being the one path in the tree that fails.
assert_eq!(read("skills/README.md"), "signpost");
// The bare-id alias: the shortest spelling, resolving because each id is
// unique across the two trees.
assert_eq!(read("skills/ics-import/SKILL.md"), "shared one");
assert_eq!(read("skills/spesa/SKILL.md"), "mine");
// Same id in both trees: neither wins, and the error names both full paths.
std::fs::create_dir_all(own.join("ics-import")).unwrap();
std::fs::write(own.join("ics-import").join("SKILL.md"), "my fork").unwrap();
let err = resolve_host_path(&fs, "skills/ics-import/SKILL.md").unwrap_err().to_string();
assert!(err.contains("skills/shared/ics-import"), "{err}");
assert!(err.contains("skills/daniele/ics-import"), "{err}");
// The full paths still work while the alias is ambiguous.
assert_eq!(read("skills/shared/ics-import/SKILL.md"), "shared one");
assert_eq!(read("skills/daniele/ics-import/SKILL.md"), "my fork");
// An invented scope segment: refused with a hint, and — the part that matters
// — it never becomes a path under the home that no indexer would ever read.
let err = resolve_host_path(&fs, "skills/pippo/SKILL.md").unwrap_err().to_string();
assert!(err.contains("other members' skills are not accessible"), "{err}");
assert!(!home.join("skills").exists(), "the invented scope leaked into the home");
// Containment inside a skill: a symlink planted in one cannot lead out of it.
std::os::unix::fs::symlink(&root, shared.join("ics-import").join("escape")).unwrap();
assert!(resolve_host_path(&fs, "skills/shared/ics-import/escape/homes/u1/x").is_err());
let _ = std::fs::remove_dir_all(&root);
}
}
+19 -7
View File
@@ -48,6 +48,7 @@ impl Tool for ListItems {
`cron` scheduled tasks/cron jobs with id, title, cron expression, agent_id, enabled, kind, last/next run.\n\
`agents` sub-agents available to delegate to (id, name, description, optional `instructions` on how to call the agent well, optional client). Do NOT invoke the `main` agent.\n\
`mcp` MCP servers, which users call \"Connectors\": which ones are already loaded into this session, which are ready for `activate_tools`, which are installed but unusable and why, and which the user could still activate. Read this before assuming a connector is missing.\n\
`skills` installed skills, in both scopes: id, scope, path, the FULL description (the prompt index shows a shortened one), size, whether it is healthy, and where it was fetched from. Use this to answer \"which skills do I have?\" and to find the id before deleting one.\n\
To list stored secret names use `list_secrets` instead."
}
@@ -58,7 +59,7 @@ impl Tool for ListItems {
"properties": {
"type": {
"type": "string",
"enum": ["plugins", "cron", "agents", "mcp"],
"enum": ["plugins", "cron", "agents", "mcp", "skills"],
"description": "Which kind of item to list."
}
}
@@ -70,10 +71,21 @@ impl Tool for ListItems {
format!("list {kind}")
}
/// `mcp` is the one type that needs the caller: which connectors are theirs,
/// which are loaded into *this* session, and what their role may do. The
/// other three are instance-wide and stay on the context-free `execute`.
/// Two types need the caller. `mcp`: which connectors are theirs, which are
/// loaded into *this* session, what their role may do. `skills`: half the
/// tree is that member's own, so the answer is per-user by construction —
/// `ctx.fs` **is** the question "which skills can this user see", already
/// answered, which is why nothing here queries anything. The other three are
/// instance-wide and stay on the context-free `execute`.
fn run_with<'a>(&'a self, ctx: &ToolContext, args: Value) -> Box<dyn ToolExecution + 'a> {
if args["type"].as_str() == Some("skills") {
let fs = ctx.fs.clone();
return Box::new(crate::tools::SimpleExecution::new(Box::pin(async move {
Ok(crate::tools::ToolResult::Json(Value::Array(
crate::skills::inventory::report(&fs),
)))
})));
}
if args["type"].as_str() != Some("mcp") {
return self.run(args);
}
@@ -102,8 +114,8 @@ impl Tool for ListItems {
match kind {
// Reached only through the context-free `execute` (no caller, so no
// report to build) — `run_with` intercepts the real call path.
"mcp" => anyhow::bail!(
"list_items: type `mcp` needs a session context and was called without one"
"mcp" | "skills" => anyhow::bail!(
"list_items: type `{kind}` needs a session context and was called without one"
),
"plugins" => {
let plugins = tokio::task::block_in_place(|| {
@@ -156,7 +168,7 @@ impl Tool for ListItems {
.collect();
Ok(serde_json::to_string_pretty(&arr)?)
}
other => anyhow::bail!("list_items: unknown type `{other}` (expected one of: plugins, cron, agents, mcp)"),
other => anyhow::bail!("list_items: unknown type `{other}` (expected one of: plugins, cron, agents, mcp, skills)"),
}
}
}
+3 -1
View File
@@ -16,7 +16,7 @@ pub fn is_file_write_tool(name: &str) -> bool {
/// Tools that read file contents or directory listings from disk.
/// Used by the approval gate to apply the `RunContext` read fast-path (auto-allow
/// working dir / `docs/` / `skills/` / `allow_fs_reads`). All take a `path` argument.
/// working dir / `docs/` / `allow_fs_reads`). All take a `path` argument.
/// Update this list whenever a new file-read tool is added.
pub const FILE_READ_TOOLS: &[&str] = &[
"read_file",
@@ -36,6 +36,7 @@ pub mod ast_outline;
pub mod configure_plugin;
pub mod cron_jobs;
pub mod exec;
pub mod fetch_repo;
pub mod fs;
pub mod image_generate;
pub mod list_items;
@@ -43,6 +44,7 @@ pub mod mcp_report;
pub mod list_secrets;
pub mod notify;
pub mod set_secret;
pub mod skills;
pub mod read_notification;
pub mod show_file;
pub mod toggle_item;
+311
View File
@@ -0,0 +1,311 @@
//! `skill_register` and `skill_delete` — the whole write surface of the skills
//! trees (blueprint §7.3/§7.4).
//!
//! Both live in the **`Config` category**, so they are absent from the schema of
//! every request until `activate_tools(["config"])` asks for them. The round that
//! costs is a fair price for administration, but the real gain is elsewhere: a
//! prompt injection cannot reach a tool the model has not been shown, so it must
//! first make the model *activate the group* — one more step, and a step that
//! leaves a line in the transcript.
//!
//! Authorization is a **capability on the role**, checked server-side (§14).
//! `scope: "global"` needs `skill.manage`; `scope: "mine"` is always the
//! caller's own. Never inferred from anything the prompt says about who the user
//! is — the same shape as `mcp.register_local_script` versus
//! `mcp.register_remote`.
use std::sync::Arc;
use anyhow::Result;
use serde_json::{Value, json};
use sqlx::SqlitePool;
use crate::skills::{PromptPrefixCell, PromptScope, Scope, install};
use crate::tools::fs::{FsTarget, resolve_target};
use crate::tools::{
SimpleExecution, Tool, ToolContext, ToolDescriptionLength, ToolExecution, ToolResult,
};
/// Everything both tools need: the registry (to read the caller's role) and the
/// seam that tells live conversations their index moved.
struct Deps {
registry: Arc<SqlitePool>,
prefixes: Arc<PromptPrefixCell>,
}
impl Deps {
/// Whether this caller may write to the group's tree.
///
/// A failure to *read* the role is a denial, not a pass: the group's scope is
/// the one that puts text into everybody's prompt, and "the database hiccuped"
/// is not a reason to widen.
async fn may_manage_shared(&self, user_id: &str) -> bool {
let role = match crate::db::users::get(&self.registry, user_id).await {
Ok(Some(u)) => u.role_id,
Ok(None) => return false,
Err(e) => {
tracing::warn!(user = %user_id, error = %e, "skills: cannot read role, denying global scope");
return false;
}
};
crate::db::role_capabilities::has(
&self.registry,
&role,
crate::db::role_capabilities::MANAGE_SKILLS,
)
.await
.unwrap_or(false)
}
async fn authorize(&self, user_id: &str, scope: Scope) -> Result<()> {
if scope == Scope::Shared && !self.may_manage_shared(user_id).await {
anyhow::bail!(
"you are not allowed to change the group's skills. Use scope \"mine\" for a \
skill of your own, or ask an admin to install this one for everybody."
);
}
Ok(())
}
/// Announces that the rendered index has moved, so a conversation that is
/// already warm does not keep quoting the old one for twenty minutes.
async fn invalidate(&self, user_id: &str, scope: Scope) {
let scope = match scope {
Scope::Shared => PromptScope::Everyone,
Scope::Own => PromptScope::User(user_id.to_string()),
};
self.prefixes.invalidate(scope).await;
}
}
/// Parses the `scope` argument shared by both tools.
fn scope_arg(args: &Value) -> Result<Scope> {
let raw = args["scope"]
.as_str()
.ok_or_else(|| anyhow::anyhow!("missing required argument `scope` (\"mine\" or \"global\")"))?;
Scope::parse(raw)
}
/// The `scope` property, identical in both schemas.
fn scope_property() -> Value {
json!({
"type": "string",
"enum": ["mine", "global"],
"description": "\"mine\" — your own skills, visible only to you. \
\"global\" — the group's skills, which every member reads as \
instructions (requires the skill.manage capability)."
})
}
// ── skill_register ────────────────────────────────────────────────────────────
pub struct SkillRegister(Deps);
impl SkillRegister {
pub fn new(registry: Arc<SqlitePool>, prefixes: Arc<PromptPrefixCell>) -> Self {
Self(Deps { registry, prefixes })
}
}
impl Tool for SkillRegister {
fn name(&self) -> &str { crate::tools::tool_names::SKILL_REGISTER }
fn category(&self) -> crate::tools::ToolCategory { crate::tools::ToolCategory::Config }
fn display_name(&self) -> &str { "Install Skill" }
fn description(&self) -> &str {
"Install a skill folder into the read-only skills tree — the only way to add one. \
The folder must live somewhere you can write (your home, a project or a shared \
folder), NOT in the container-only filesystem such as /tmp, and must contain a \
`SKILL.md` opening with a YAML frontmatter block declaring `name` (lowercase \
letters, digits and hyphens) and `description` (when to use the skill, under 1000 \
characters). The installed folder is named after that `name`, not after the source \
folder. Registering an id that already exists in the same scope replaces it that \
is how a skill is updated; a skill is never edited in place. Read `docs/skills.md` \
before writing one."
}
fn parameters_schema(&self) -> Value {
json!({
"type": "object",
"required": ["scope", "path"],
"properties": {
"scope": scope_property(),
"path": {
"type": "string",
"description": "Path of the folder to install, e.g. \"~/drafts/ics-import\". \
It is copied, not moved."
}
}
})
}
fn describe(&self, args: &Value, _length: ToolDescriptionLength) -> String {
let path = args["path"].as_str().unwrap_or("?");
match args["scope"].as_str() {
Some("global") => format!("install `{path}` as a skill for the whole group"),
_ => format!("install `{path}` as one of your skills"),
}
}
fn target_path(&self, args: &Value) -> Option<String> {
args["path"].as_str().map(str::to_string)
}
fn run_with<'a>(&'a self, ctx: &ToolContext, args: Value) -> Box<dyn ToolExecution + 'a> {
let fs = ctx.fs.clone();
let user_id = ctx.user_id.clone();
Box::new(SimpleExecution::new(Box::pin(async move {
let scope = scope_arg(&args)?;
self.0.authorize(&user_id, scope).await?;
let path = args["path"]
.as_str()
.ok_or_else(|| anyhow::anyhow!("missing required argument `path`"))?;
// The copy is host-side, so the source has to be on a mount. `/tmp`
// is the first place a model puts a working folder, and a bare
// ENOENT there reads as "the folder is gone" rather than "wrong side
// of the boundary" — so say which it is.
let host = match resolve_target(&fs, path)? {
FsTarget::Host(p) => p,
FsTarget::Container { .. } => anyhow::bail!(
"`{path}` exists only inside your container, and a skill is installed from \
the host side. Move the folder into your home (e.g. `~/{}`) and register \
that path instead.",
std::path::Path::new(path)
.file_name()
.map(|n| n.to_string_lossy().into_owned())
.unwrap_or_else(|| "my-skill".into())
),
};
let done = install::install(&fs, scope, &host)?;
self.0.invalidate(&user_id, scope).await;
let verb = if done.replaced { "Replaced" } else { "Installed" };
Ok(ToolResult::Text(format!(
"{verb} skill `{}` at {}. It is in the index now — read it back with \
`read_file {}/SKILL.md`.",
done.id, done.agent_dir, done.agent_dir
)))
})))
}
}
// ── skill_delete ──────────────────────────────────────────────────────────────
pub struct SkillDelete(Deps);
impl SkillDelete {
pub fn new(registry: Arc<SqlitePool>, prefixes: Arc<PromptPrefixCell>) -> Self {
Self(Deps { registry, prefixes })
}
}
impl Tool for SkillDelete {
fn name(&self) -> &str { crate::tools::tool_names::SKILL_DELETE }
fn category(&self) -> crate::tools::ToolCategory { crate::tools::ToolCategory::Config }
fn display_name(&self) -> &str { "Delete Skill" }
fn description(&self) -> &str {
"Remove an installed skill. The id is its folder name — use \
`list_items` with type=skills to see the installed ids and scopes. \
There is no recycle bin: the folder is deleted."
}
fn parameters_schema(&self) -> Value {
json!({
"type": "object",
"required": ["scope", "id"],
"properties": {
"scope": scope_property(),
"id": {
"type": "string",
"description": "The skill's id — its folder name, e.g. \"ics-import\"."
}
}
})
}
fn describe(&self, args: &Value, _length: ToolDescriptionLength) -> String {
let id = args["id"].as_str().unwrap_or("?");
match args["scope"].as_str() {
// Said in full on the card: this removes something from every
// member's prompt, not just from the caller's.
Some("global") => format!("delete skill `{id}` for the whole group"),
_ => format!("delete your skill `{id}`"),
}
}
fn run_with<'a>(&'a self, ctx: &ToolContext, args: Value) -> Box<dyn ToolExecution + 'a> {
let fs = ctx.fs.clone();
let user_id = ctx.user_id.clone();
Box::new(SimpleExecution::new(Box::pin(async move {
let scope = scope_arg(&args)?;
self.0.authorize(&user_id, scope).await?;
let id = args["id"]
.as_str()
.ok_or_else(|| anyhow::anyhow!("missing required argument `id`"))?;
install::remove(&fs, scope, id)?;
self.0.invalidate(&user_id, scope).await;
Ok(ToolResult::Text(format!("Deleted skill `{id}` ({}).", scope.as_arg())))
})))
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::tools::ToolRegistry;
fn registry() -> ToolRegistry {
// `connect_lazy` performs no I/O, and neither tool touches the pool
// unless it is asked for the group's scope.
let pool = Arc::new(SqlitePool::connect_lazy("sqlite::memory:").unwrap());
let cell = Arc::new(PromptPrefixCell::default());
let mut r = ToolRegistry::new();
r.register(SkillRegister::new(Arc::clone(&pool), Arc::clone(&cell)));
r.register(SkillDelete::new(pool, cell));
r
}
fn names(defs: &[Value]) -> Vec<String> {
defs.iter()
.filter_map(|d| d["function"]["name"].as_str().map(str::to_string))
.collect()
}
/// The two write verbs are absent from the schema of an ordinary request and
/// arrive only with `activate_tools(["config"])`. That extra round is the
/// price of administration; the gain is that a prompt injection has to make
/// the model activate the group first — a step that shows in the transcript.
#[tokio::test]
async fn the_write_verbs_are_invisible_until_the_config_group_is_activated() {
let r = registry();
assert!(names(&r.openai_definitions_excluding_config()).is_empty());
let mut lazy = names(&r.openai_definitions_config_only());
lazy.sort();
assert_eq!(lazy, vec!["skill_delete".to_string(), "skill_register".to_string()]);
}
/// Enumerating is harmless and is what makes "delete the X skill" possible
/// without guessing an id, so it stays in every request.
#[test]
fn enumeration_is_not_in_the_lazy_group() {
assert_ne!(
crate::tools::ToolCategory::Config,
crate::tools::ToolCategory::Introspection,
);
}
#[test]
fn the_scope_argument_is_the_vocabulary_the_tools_take() {
assert_eq!(Scope::parse("mine").unwrap(), Scope::Own);
assert_eq!(Scope::parse("global").unwrap(), Scope::Shared);
// Not a username: a tool that asked for one would invite passing
// somebody else's, which the server would have to ignore anyway.
assert!(Scope::parse("daniele").is_err());
}
}
@@ -12,3 +12,12 @@ pub const READ_NOTIFICATION: &str = "read_notification";
pub const EXECUTE_CMD: &str = "execute_cmd";
pub const SHOW_FILE_TO_USER: &str = "show_file_to_user";
pub const IMAGE_GENERATE: &str = "image_generate";
/// The one write verb of the read-only skills trees (blueprint §7.3). Named here
/// because the approval gate builds it a review card of its own.
pub const SKILL_REGISTER: &str = "skill_register";
pub const SKILL_DELETE: &str = "skill_delete";
/// Downloads a subtree of a public git repository into the caller's workspace
/// (blueprint §7.5). Deliberately not `git_clone`: it is shallow, drops `.git`,
/// sanitizes, and leaves a `.source.json` provenance ticket — none of which a
/// name borrowed from git would promise.
pub const FETCH_REPO: &str = "fetch_repo";
+81
View File
@@ -0,0 +1,81 @@
# Agents
An **agent** is a role the assistant can play: a name, an icon, a system prompt that shapes its personality and skills, and a set of tools. Every conversation with the assistant is a conversation with **one** agent — the same engine, a different persona.
Agents are defined as plain files in the `agents/` folder on the server (one subfolder per agent: a `meta.json` with the name and description, an `AGENT.md` with the prompt, optionally an icon). The app discovers them at startup and **re-reads the prompt files on every use**, so editing an agent's `AGENT.md` on the server takes effect without restarting anything.
There are three kinds of agent, and the difference is *who starts the conversation*:
| Kind | Who talks to it | Count |
|------|-----------------|-------|
| **Chat** | You, directly | 3 |
| **Task** | The assistant, on your behalf (delegation) | 8 |
| **System** | Nobody — it runs on a schedule, in the background | 4 |
## Which agent are you talking to?
When you start a conversation, which chat agent it lands on is decided by your **role** — not by you picking one from a list:
- Members of most roles get the **Assistant** — the general-purpose agent that helps with anything, remembers what matters in memory, and delegates specialised work (see below).
- Members of the **children's role** get the **Companion** — a warmer, gentler assistant that adapts its tone and vocabulary to the child's age.
- Conversations about a **project** get the **Project Coordinator** — the same agent who runs the project's chat, holding the project's full context.
An admin can change a role's default assistant in the role editor (sidebar → **Roles** → edit a role → **Default assistant**). Leaving it empty means "the Assistant". A role change applies to **new** conversations, from the member's next login — an existing conversation keeps the agent it started with.
If a user asks "who am I talking to?" or "why does my assistant talk differently from theirs?", the answer is: the agent their role defaults to, and it can be changed by the admin — not by the user, and not by asking the assistant (the assistant cannot change its own role).
## The Agents page
Sidebar → **Agents** shows every agent on this instance, in three sections (Chat / Task / System), as cards with the agent's icon, name, and a short description.
Clicking an agent opens its detail page with:
- **The prompt** — the full `AGENT.md` text the agent runs under, rendered as Markdown. This is not secret: it is what the agent is told to be and do, and reading it is a good way to understand why an agent behaves the way it does. (The Assistant's prompt, for example, tells it to keep personal facts in memory, to prefer `user-memory/` notes, and to delegate to task agents when a job needs a specialist.)
- **The models** — which LLMs the agent can run on, and how it picks one (see [How the model is chosen](#how-the-model-is-chosen)).
The System section lists the background agents too — they are invisible in the chat but visible here, with their prompts. For what they *do* and when they run, see [system-agents.md](system-agents.md).
## The task agents: the specialists
Eight agents exist to do specific jobs, and the chat agent calls on them automatically when the job matches — you never talk to them directly. You *trigger* one simply by asking: *"use the researcher to find me…"*, *"get the code explorer to look at…"*, or just describing the task in a way that matches a specialist's job. The assistant recognises the match, delegates, and brings the result back into the conversation.
| Agent | What it is for | Where its output goes |
|-------|----------------|-----------------------|
| **Researcher** | Multi-step web research, with sources | A structured summary in the chat; findings saved to the scratchpad, optionally to `data/research/` |
| **Business Analyst** | Stress-tests a business idea or plan against the evidence you provide; GO / NO-GO / PIVOT verdict | A critique report (path you choose, or the scratchpad) |
| **Code Explorer** | Studies code, investigates bugs, analyses architecture — analysis only, never edits | A structured Markdown report in `data/explorer/` |
| **Spec Writer** | Turns a rough idea into a detailed, unambiguous written specification | A Markdown spec document (never code) |
| **Software Architect** | Plans a code change end-to-end before anything is touched | An implementation plan, possibly delegating the edits to the engineer |
| **Software Engineer** | Writes and edits source files to implement a decided change | The code changes themselves |
| **Tech Lead** | Takes project requirements and builds the whole thing, decomposing the work and orchestrating architect + engineer | The completed implementation |
| **Generalist** | Carries out well-defined hands-on work — file edits, shell commands, batch operations — exactly as instructed | The finished work |
Three things worth knowing about delegation:
- **It is ordinary conversation.** The child agent's work happens in its own context, but what you see is the flow: the assistant calls the specialist, the specialist reports back, the assistant answers you. You can watch it happen in the chat.
- **The specialists have the same rules.** They run with the same tool set, the same security groups and the same approval cards — a specialist wanting to write a file in a shared folder will ask for confirmation exactly as the assistant would. (The Conversation Review agent is the exception by design: it has no tools at all, see [system-agents.md](system-agents.md).)
- **They cannot be summoned from the void.** The assistant decides whether and when to delegate — there is no user-facing list to pick a specialist from, and calling one yourself is not a thing you can do (nor should need to).
## How the model is chosen
Every agent declares a **strength** — how powerful a model it should run on (from *very low* to *very high*). When no model is pinned, the app picks the best available model at or above that strength — so a lightweight background task uses a small model, and the most demanding specialists (Architect, Tech Lead) ask for the strongest.
A specific conversation can **pin** a model instead, per conversation, with the model picker in the chat. Pinning overrides the strength choice for that conversation only.
## Custom agents (admin)
Agents are data, not code — an admin can add a new one by creating a folder on the server:
- `agents/<id>/meta.json` — the name, description, type (`chat`, `task` or `system`), strength, and optionally an icon file.
- `agents/<id>/AGENT.md` — the system prompt (the `name` given in `meta.json` is what users will see; the folder name is the internal id).
No restart and no rebuild: the app discovers the new agent and picks up prompt edits on the next use. An icon (a square image) is served automatically when declared in `meta.json` — optional, but it is what shows on the Agents page and in the chat.
If a user asks for "a different assistant", the honest answer is: there is no UI to create one — an admin can add a custom agent by hand on the server, and this guide tells them how, or the user can be pointed at the role's default-assistant setting instead.
## Notes
- **What the Assistant knows about you.** At the start of a conversation, the chat agent reads a few memory notes automatically: your profile and the private-memory index (`user-memory/`), and the group's shared-memory index. That is *in addition to* whatever you say — it is how the agent "remembers" between conversations. The Project Coordinator additionally reads the project's own `SKALD.md` when one exists, so it arrives already knowing the project. See [memory.md](memory.md).
- **The system agents are invisible on purpose.** They run on a schedule, not in a conversation, and they never talk to you — they notify you when something needs attention, and their work and settings live on the System agents page (`#system-agents`). See [system-agents.md](system-agents.md).
- **Prompts are not secrets.** Nothing on the Agents page is hidden from the user who can see it — if someone asks "what is the assistant told to do?", the answer is "read it on the Agents page".
- **Agents cannot change themselves.** An agent's prompt is a file on the server; the agent cannot edit it, and a user cannot make an agent "become" another agent by asking. New conversations, new agent — the mapping is decided by the role.
+4 -1
View File
@@ -4,19 +4,22 @@ This folder is written for **you, the assistant**, not for the human directly. I
Keep answers grounded in what's actually enabled and configured for this instance — check with the relevant tool (e.g. list installed/enabled plugins) rather than assuming everything described here is turned on. A feature documented here may not be enabled on this particular instance.
This index will grow over time. Right now it covers the interface, memory, projects, background tasks, system agents, access grants, connectors, voice input and plugins; more sections (agents, security groups, shared folders…) will be added later.
This index will grow over time. Right now it covers the interface, agents, memory, projects, shared folders, background tasks, system agents, access grants, connectors, skills, voice input and plugins; more sections (security groups…) will be added later.
## Features
| Document | What it covers |
| --- | --- |
| [memory.md](memory.md) | Private and shared memory: what goes where, the indexes and history log, why some shared facts can't be changed on request |
| [agents.md](agents.md) | Agents: the three kinds (chat, task, system), which one you are talking to and why, the specialist agents the assistant delegates to, how the model is chosen, and adding a custom agent |
| [projects.md](projects.md) | Projects: shared folders with their own assistant chat, a live file explorer, and member sharing |
| [shared-folders.md](shared-folders.md) | Shared folders: admin-managed folders with no chat of their own — who sees them, read vs write access, why the assistant asks before touching them, and when to choose a project instead |
| [system-agents.md](system-agents.md) | Background agents that run on a schedule (event triage, the two memory lints, the nightly conversation review of a supervised account): what they watch, why they only ever report, why a run can be skipped, and their settings |
| [tasks.md](tasks.md) | Background tasks: the strip above the message box, following one live, stopping one, answering the approvals and questions they raise, and how every outcome comes back to the conversation |
| [settings.md](settings.md) | The admin's Config page: interface language, the compaction model picker, debug mode |
| [access.md](access.md) | Who can use which plugin or connector: the open default, removing access per person, and the role switch that keeps children out of it |
| [connectors.md](connectors.md) | Connectors (MCP servers): shared vs per-user, setting one up in the UI, the sign-in and QR-pairing flows, and what to do when one is not working |
| [skills.md](skills.md) | Skills: instruction folders the assistant loads on demand — where they live, how to read and run one, and the contract for writing, installing and downloading one |
| [voice.md](voice.md) | Voice input: configuring a transcription model, and why the microphone button does nothing unless the page is served over HTTPS or localhost |
| [interface.md](interface.md) | The desktop interface: collapsing the sidebar to an icon-only strip to make room for documents |
+90
View File
@@ -0,0 +1,90 @@
# Shared folders
A **shared folder** is a folder on the server that several members of the group can use together — a single place for documents everybody needs, instead of each person keeping their own copy and asking for the latest version by hand.
Examples: a shared recipe collection, the household's documents (bills, contracts, manuals), a folder where members drop files for the whole group to see.
Shared folders are **managed by an admin**: an admin creates them, decides who is a member and what each member may do in them. There is **no owner** — the person who created a folder has no special rights over it; the admin can change or remove anyone, themselves included.
Two things shared folders are *not*, so expectations stay right:
- They have **no chat of their own** — the files are shared, not a conversation. (That is what Projects are for; see [Shared folders vs Projects](#shared-folders-vs-projects) below.)
- They have **no file explorer page** in the web app. Members work with the files through the assistant, and open individual files in the file viewer when the assistant shows them (see [Working with the files](#working-with-the-files)).
## Creating a folder (admin)
1. Open **Shared Folders** in the sidebar (admin only; members do not see this page).
2. Click **New Folder** and fill in:
- **Name** — a short, simple name with no spaces or punctuation: `recipes`, `documents`, `holiday-pics`. The name becomes the folder's path, so it must be a single word (no `/`, `\`, `.` or `..`). It **cannot be changed later** — pick carefully.
- **Description** — what the folder is for, in plain words. This is not decoration: it is what the assistant reads to understand the folder (see [What the assistant knows](#what-the-assistant-knows)). A good description: *"Family documents: bills, contracts, manuals — everyone can read, only Marta writes."*
3. Save. The folder is created on the server immediately.
There is no step 4: a folder starts **empty**, with **no members** — even the admin is not a member until added. Add members next (see below).
## Who can see it: members
A folder is visible only to its members. The admin adds members from the folder's row on the Shared Folders page, choosing each person's access level:
- **Read** — can open and read the files, and ask the assistant to work with them. Cannot create, edit or delete anything.
- **Read & write** — everything Read gives, plus creating, editing and deleting files (directly or through the assistant).
Two things worth knowing about membership:
- **Changes apply immediately.** Adding, removing or changing a member takes effect right away — the other person does not need to log out and back in.
- **Removing access is the only way to take files away** — and it works: a person who is no longer a member can no longer see the folder, its files, or ask the assistant about them.
## Working with the files
There is no file explorer for shared folders — no grid of files, no upload button. The files live on the server, and members reach them through the assistant:
- **Ask the assistant** — "what's in the recipes folder?", "add this note to documents", "send me the manual for the boiler". The assistant knows which folders you belong to, can list their contents, open and search files, and — if you have read & write — create and edit them.
- **Open a file** — when the assistant shows you a file from a shared folder, it opens in the usual file viewer (Markdown rendered, images, PDFs, text), exactly like any other file. You can read it there; editing in the viewer is available if you have read & write access.
A practical consequence: if a member wants a file *from* a shared folder, the assistant is the way to get it — there is no download button on the folder itself. (An admin can of course reach the folder directly on the server, but members should not need to.)
## What the assistant knows
At the start of every conversation, the assistant sees a table of the shared folders you belong to, with four columns:
| Path | Access | Shared with | Description |
|------|--------|-------------|-------------|
| `shared/recipes` | read-write | — | Family recipes, everyone can add |
| `shared/documents` | read-only | Anna, Luca | Bills and contracts |
So the assistant knows the folder exists, **your** access level in it, **who else** can see it, and what the admin wrote in the description — and nothing else. It does not read the files on its own: it looks inside only when you ask, and it may ask you to confirm that something belongs in a shared folder before putting it there (see below).
This is why the description matters: it is the folder's only explanation, and a folder with an empty description is a folder the assistant cannot reason about. If you are an admin and a shared folder has no description, editing it (Shared Folders → the folder → edit) is the most useful thing you can do with it.
## The approval cards
The assistant never changes a shared folder on its own initiative — and even on your request, **reading and writing in a shared folder asks for your confirmation first**, as a small card you answer in the chat (or in the Inbox, or from your phone, depending on where you are talking).
This is deliberate and it applies to *everyone*, the admin included:
- **Reads ask too**, not just writes. A shared folder may contain things other members wrote, and the system does not assume you want the assistant browsing it freely.
- The card shows exactly what the assistant wants to do — open this file, create that one, change this line — with **Approve** and **Deny** buttons. Answering is the whole flow; you do not need to do anything else.
- **If you deny**, the assistant simply does not do it and moves on — nothing is forced.
If this feels like a lot of questions, remember the trade-off is the point: shared folders are the one place where one person's words become *everyone's* files, so every step is a conscious one. (Projects work differently — see below.)
## Shared folders vs Projects
The two features look similar — a shared place for files with per-member access — but they solve different problems:
| | Shared folder | Project |
|---|---|---|
| Who manages it | An admin (no owner; anyone can be removed) | The owner (a member) and read & write members |
| Where it lives in the chat | No chat of its own | Its own conversation with the assistant (`project-{id}`), plus extra tabs |
| Files | No explorer page; work through the assistant | A live file explorer with upload, rename, delete, ZIP download |
| Assistant's access | Every read/write asks for confirmation | Reads and writes are frictionless (only the folder's membership limits them) |
| Typical use | A place to *keep* shared documents | A place to *work together* on something |
When to use which, in one line: if the point is "we have documents here", a shared folder; if the point is "we are working on something here", a project. And the two combine naturally — a project's chat can refer to shared folders, and the assistant can copy files between them if you have the right access.
## Notes
- **The name never changes.** A shared folder cannot be renamed (there is no rename button). To "rename" one, an admin creates a new folder and moves the files into it — which changes the folder's path and requires re-adding members.
- **Deleting a folder does not delete the files.** When an admin deletes a shared folder, only the sharing is removed: the folder stays on the server, and an admin can remove it by hand if that is really intended. Say this plainly if a user believes deleting the folder destroyed its contents.
- **The Shared Folders page is admin-only.** Members never see it, and a member cannot create folders, add members, or change access levels. Direct them to the admin rather than trying to do any of it on their behalf.
- **The assistant can copy files *into* a shared folder** (with your approval) — a good way to publish something from your private space to the group. The reverse works too, if you are a member.
- **A description is not a substitute for access.** Writing "everyone may read this" in the description tells the assistant the intent, but membership is still decided by the admin on the Shared Folders page — the assistant cannot grant access, only tell you who currently has it.
+69
View File
@@ -0,0 +1,69 @@
# Skills
A skill is a **folder of instructions and resources** that you load on demand, when a task calls for it — a procedure written once and followed every time, instead of re-deriving the steps in each conversation. A skill adds no tools and starts no processes: it is knowledge you read, then apply with the tools you already have.
## Where skills live
Skills are installed in one of two read-only trees:
| Path | Whose | Who installs |
| --- | --- | --- |
| `skills/shared/<id>/` | the whole group — every member sees these | an admin |
| `skills/<username>/` | one member's own | that member |
Both trees are **read-only**, everywhere and for everyone. You cannot create or edit files under `skills/` — not with the file tools, not from the shell, not even with `sudo`. A skill is **installed**, never written in place; the only way in is `skill_register` (below). To modify a skill you copy it out, edit the copy, and register it again.
## Using a skill
Your prompt already carries the index of the skills you can see: one line each, with the full path of its `SKILL.md` and a short description of when to use it. When a task matches — even partially — **read the skill before doing the work**: `read_file skills/<scope>/<id>/SKILL.md`, then follow its instructions.
To run a skill's script, use `execute_cmd` with `workdir` set to the skill's folder (e.g. `workdir: "skills/shared/ics-import"`). A skill cannot write next to itself — the tree is read-only — so scripts must write their output, caches and state to your home (`~`) or `/tmp`, never into the skill folder.
To see exactly what is installed, with full descriptions and per-skill health: `list_items` with `type="skills"`.
## Creating a skill — the authoring contract
A skill folder looks like this:
```
<skill-name>/
├── SKILL.md # required
├── scripts/ or *.py, *.js in the root # optional executables
├── references/ # optional documents to read on demand
└── assets/ # optional templates, examples
```
`SKILL.md` opens with a YAML frontmatter block, then the instructions in Markdown:
```markdown
---
name: ics-import
description: Download an iCalendar (ICS) feed and turn it into JSON or a table. Use whenever the user gives you a calendar URL or asks to import, inspect or summarize events from an ICS link.
---
# ICS import
1. Run `python3 scripts/ics2json.py <url>` from this folder...
```
Rules — every one of them is **checked at installation**, and a folder that breaks one is refused with a message naming the problem:
- **`name`**: lowercase letters, digits and hyphens only, at most 64 characters. It becomes the installed folder's name — so the working copy may be called `draft-2`, but what gets installed is named after the frontmatter.
- **`description`**: required, at most 1000 characters. This is the *use condition* — the only thing the model sees when deciding whether the skill is relevant. Write it assertively: say exactly **when** to reach for the skill, and err on the side of triggering too easily rather than too rarely. Save the detail for the body.
- **Write it in English** — the instructions, the frontmatter, the comments. Everything the model reads is English.
- **Self-contained**: no symbolic links; at most 500 files and 8 MiB in total. A skill holds instructions, scripts and reference documents — bulk data belongs in a home or a project.
- Create the folder **somewhere you can write** — your home, a project or a shared folder. **Not in `/tmp`** or anywhere else in the container-only filesystem: installation copies the folder from the host side and cannot see those paths.
- A script that sticks to the Python/Node standard library and the preinstalled command-line tools works always. One that needs PyPI or npm packages **must say so in the body** — those packages are installed by hand (`sudo pip install …`) and are lost when the container is recreated.
## Installing, updating, deleting
All three go through tools, and all three ask a human for approval first — the approval card shows the full `SKILL.md`, the file list and where the skill is going:
- **Install**: `skill_register(scope, path)``scope` is `"mine"` (your own tree) or `"global"` (everyone's; requires the admin capability). The folder is validated, copied in one atomic move, and appears in the prompt index immediately.
- **Update**: register again with the same `name` in the same scope — the old copy is replaced. That is the only way a skill changes.
- **Promote to the group**: register an already-installed skill with `scope: "global"`, e.g. `skill_register("global", "skills/maria/ics-import")`.
- **Delete**: `skill_delete(scope, id)` — the folder is removed, with no recycle bin. Use `list_items` with `type="skills"` to find ids.
## Getting a skill from a public repository
`fetch_repo(url, sub_path, destination)` downloads a subtree of a **public git repository** into a writable folder of yours — shallow, without the `.git` history. It installs nothing: if what you downloaded is a skill, review the files and then call `skill_register` on the destination folder. Every download leaves a `.source.json` ticket in the destination recording the URL, the sub-path and the exact commit it came from, so "where did this come from?" always has an answer.
-75
View File
@@ -1,75 +0,0 @@
---
name: ics2json
description: Download an iCalendar (ICS) feed from a URL and output structured JSON with all events. Use this skill whenever the user needs to read, analyze, or integrate events from a public iCal feed such as Teamup or Google Calendar, especially when no API keys are available. Also use as the base for cron jobs that periodically analyze a calendar feed. If the user mentions an .ics file or calendar feed, use this skill.
---
# ics2json
_Updated: 2026-06-08_
Downloads an iCalendar (ICS) feed from a URL and outputs structured JSON with all events.
## When to use
- Reading and analyzing events from a public iCal feed (Teamup, Google Calendar, etc.)
- Integrating an external calendar without API keys
- As the base for cron jobs that periodically analyze a feed
## Script
`python3 skills/ics2json/ics2json.py <url> [options]`
### Options
| Flag | Description |
|------|-------------|
| `--days N` | Only events starting within the next N days |
| `--all` | Include past events (default: future or ongoing only) |
| `--pretty` | Pretty-print JSON output (default: compact, single-line) |
| `--meta` | Only output calendar metadata (name, description, event count). No events returned. |
### Output JSON format
```json
{
"calendar": "Calendar name",
"description": "Calendar description",
"feed_url": "Feed URL",
"fetched_at": "2026-05-20T12:00:00+01:00",
"total_events": 5,
"events": [
{
"uid": "TU123456",
"title": "Event title",
"description": "Description...",
"location": "Venue",
"where": "Teamup address",
"categories": "kink oriented event ⛓️",
"event_url": "https://teamup.com/...",
"external_url": "https://tickets.example.com",
"start": "2026-05-20T19:00:00+01:00",
"end": "2026-05-20T23:00:00+01:00",
"created": "2026-05-01T10:00:00+00:00",
"last_modified": null,
"stamp": "2026-05-19T13:18:47+00:00",
"attachments": [{"url": "https://...", "type": "image/jpeg", "filename": "flyer.jpg"}]
}
]
}
```
### Examples
```bash
# Download a Teamup feed, future events only (compact output)
python3 skills/ics2json/ics2json.py https://ics.teamup.com/feed/ksmt7zqvai72zisjo4/12645979.ics
# Metadata only (compact)
python3 skills/ics2json/ics2json.py https://ics.teamup.com/feed/ksmt7zqvai72zisjo4/12645979.ics --meta
# Only the next 30 days (compact)
python3 skills/ics2json/ics2json.py https://ics.teamup.com/feed/ksmt7zqvai72zisjo4/12645979.ics --days 30
# All events (including past), pretty-printed for human reading
python3 skills/ics2json/ics2json.py https://ics.teamup.com/feed/ksmt7zqvai72zisjo4/12645979.ics --all --pretty
```
-180
View File
@@ -1,180 +0,0 @@
#!/usr/bin/env python3
"""Download an iCal feed and output events as JSON."""
import argparse
import json
import sys
from datetime import datetime, timedelta, timezone
from typing import Any, Dict, List, Optional
from urllib.request import urlopen
from icalendar import Calendar
def parse_dt(value: Any) -> Optional[str]:
"""Parse an iCal date/datetime value and return ISO 8601 string."""
if value is None:
return None
dt = value.dt
if isinstance(dt, datetime):
if dt.tzinfo is None:
dt = dt.replace(tzinfo=timezone.utc)
return dt.isoformat()
# date-only (all-day events)
return dt.isoformat()
def extract_attachments(event: Any) -> List[Dict[str, str]]:
"""Extract ATTACH properties as a list of {url, type} dicts."""
attachments: List[Dict[str, str]] = []
if "ATTACH" in event:
# May be a single value or a list
items = event["ATTACH"] if isinstance(event["ATTACH"], list) else [event["ATTACH"]]
for item in items:
attach: Dict[str, str] = {"url": str(item)}
params = item.params
if "FMTTYPE" in params:
attach["type"] = params["FMTTYPE"]
if "FILENAME" in params:
attach["filename"] = params["FILENAME"]
attachments.append(attach)
return attachments
def extract_text(component: Any, key: str) -> Optional[str]:
"""Extract a text property, decoded from any encoding."""
raw = component.get(key)
if raw is None:
return None
# vCategory objects have a .cats attribute (list of vText)
if hasattr(raw, "cats"):
return ", ".join(str(item) for item in raw.cats)
# Other list-like properties
if isinstance(raw, list):
return ", ".join(str(item) for item in raw)
return str(raw)
def event_to_dict(event: Any) -> Dict[str, Any]:
"""Convert an iCal VEVENT to a flat dict."""
return {
"uid": extract_text(event, "UID"),
"title": extract_text(event, "SUMMARY"),
"description": extract_text(event, "DESCRIPTION"),
"location": extract_text(event, "LOCATION"),
"where": extract_text(event, "X-TEAMUP-WHERE"),
"categories": extract_text(event, "CATEGORIES"),
"event_url": extract_text(event, "URL"),
"external_url": extract_text(event, "X-TEAMUP-EVENT-URL"),
"start": parse_dt(event.get("DTSTART")),
"end": parse_dt(event.get("DTEND")),
"created": parse_dt(event.get("CREATED")),
"last_modified": parse_dt(event.get("LAST-MODIFIED")),
"stamp": parse_dt(event.get("DTSTAMP")),
"attachments": extract_attachments(event),
}
def download_feed(url: str) -> Calendar:
"""Download and parse an iCal feed."""
with urlopen(url) as response:
raw = response.read()
return Calendar.from_ical(raw)
def _parse_iso(iso: str) -> datetime:
"""Parse an ISO 8601 string, making it UTC-aware."""
dt = datetime.fromisoformat(iso)
if dt.tzinfo is None:
dt = dt.replace(tzinfo=timezone.utc)
return dt
def is_future(event_dict: Dict[str, Any], now: datetime) -> bool:
"""Check if an event is in the future (or ongoing)."""
end = event_dict.get("end")
if end is None:
start = event_dict.get("start")
if start is None:
return True
return _parse_iso(start) >= now
return _parse_iso(end) >= now
def is_within_days(event_dict: Dict[str, Any], days: int, now: datetime) -> bool:
"""Check if an event starts within the next N days."""
start = event_dict.get("start")
if start is None:
return True
dt = _parse_iso(start)
cutoff = now + timedelta(days=days)
return dt >= now and dt <= cutoff
def main() -> None:
parser = argparse.ArgumentParser(description="Download an iCal feed and output events as JSON")
parser.add_argument("url", help="URL of the iCal feed (.ics)")
parser.add_argument("--days", type=int, default=None, help="Only return events starting within the next N days")
parser.add_argument("--all", action="store_true", help="Include past events (default: future only)")
parser.add_argument("--pretty", action="store_true", help="Pretty-print JSON output (default: compact)")
parser.add_argument("--meta", action="store_true", help="Only output calendar metadata (no events)")
args = parser.parse_args()
try:
cal = download_feed(args.url)
except Exception as e:
print(f"Error downloading feed: {e}", file=sys.stderr)
sys.exit(1)
now = datetime.now(timezone.utc)
# Calendar-level metadata
cal_name = extract_text(cal, "X-WR-CALNAME") or extract_text(cal, "SUMMARY") or "Unknown"
cal_desc = extract_text(cal, "X-WR-CALDESC") or extract_text(cal, "DESCRIPTION") or ""
# Count total events (all of them, unfiltered)
total_all = sum(1 for c in cal.walk() if c.name == "VEVENT")
output: Dict[str, Any] = {
"calendar": cal_name,
"description": cal_desc,
"feed_url": args.url,
"fetched_at": now.isoformat(),
"total_events": total_all,
}
if args.meta:
if args.pretty:
print(json.dumps(output, indent=2, ensure_ascii=False))
else:
print(json.dumps(output, ensure_ascii=False))
return
events: List[Dict[str, Any]] = []
for component in cal.walk():
if component.name != "VEVENT":
continue
ev = event_to_dict(component)
# Apply filters
if not args.all and not is_future(ev, now):
continue
if args.days is not None and not is_within_days(ev, args.days, now):
continue
events.append(ev)
# Sort by start date (ascending, None at the end)
events.sort(key=lambda e: e.get("start") or "9999")
output["total_events"] = len(events)
output["events"] = events
if args.pretty:
print(json.dumps(output, indent=2, ensure_ascii=False))
else:
print(json.dumps(output, ensure_ascii=False))
if __name__ == "__main__":
main()
-15
View File
@@ -1,15 +0,0 @@
# Skills Index
Skills are reusable capability packages. Each skill lives in its own directory and contains:
- `SKILL.md` — what the skill does, when to use it, and how to invoke its scripts
- one or more Python scripts that perform the actual work
Read `SKILL.md` before running any script. The file explains inputs, outputs, and usage examples.
## Registered Skills
| ics2json | [SKILL.md](ics2json/SKILL.md) | Download an iCal feed and output events as JSON |
| mcp-builder | [SKILL.md](mcp-builder/SKILL.md) | Complete guide + scripts for building high-quality MCP servers (Python FastMCP / TypeScript SDK) |
| skill-creator | [SKILL.md](skill-creator/SKILL.md) | Framework for creating, validating, evaluating and packaging skills |
-202
View File
@@ -1,202 +0,0 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright 2026 Anthropic, PBC.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-236
View File
@@ -1,236 +0,0 @@
---
name: mcp-builder
description: Guide for creating high-quality MCP (Model Context Protocol) servers that enable LLMs to interact with external services through well-designed tools. Use when building MCP servers to integrate external APIs or services, whether in Python (FastMCP) or Node/TypeScript (MCP SDK).
license: Complete terms in LICENSE.txt
---
# MCP Server Development Guide
## Overview
Create MCP (Model Context Protocol) servers that enable LLMs to interact with external services through well-designed tools. The quality of an MCP server is measured by how well it enables LLMs to accomplish real-world tasks.
---
# Process
## 🚀 High-Level Workflow
Creating a high-quality MCP server involves four main phases:
### Phase 1: Deep Research and Planning
#### 1.1 Understand Modern MCP Design
**API Coverage vs. Workflow Tools:**
Balance comprehensive API endpoint coverage with specialized workflow tools. Workflow tools can be more convenient for specific tasks, while comprehensive coverage gives agents flexibility to compose operations. Performance varies by client—some clients benefit from code execution that combines basic tools, while others work better with higher-level workflows. When uncertain, prioritize comprehensive API coverage.
**Tool Naming and Discoverability:**
Clear, descriptive tool names help agents find the right tools quickly. Use consistent prefixes (e.g., `github_create_issue`, `github_list_repos`) and action-oriented naming.
**Context Management:**
Agents benefit from concise tool descriptions and the ability to filter/paginate results. Design tools that return focused, relevant data. Some clients support code execution which can help agents filter and process data efficiently.
**Actionable Error Messages:**
Error messages should guide agents toward solutions with specific suggestions and next steps.
#### 1.2 Study MCP Protocol Documentation
**Navigate the MCP specification:**
Start with the sitemap to find relevant pages: `https://modelcontextprotocol.io/sitemap.xml`
Then fetch specific pages with `.md` suffix for markdown format (e.g., `https://modelcontextprotocol.io/specification/draft.md`).
Key pages to review:
- Specification overview and architecture
- Transport mechanisms (streamable HTTP, stdio)
- Tool, resource, and prompt definitions
#### 1.3 Study Framework Documentation
**Recommended stack:**
- **Language**: TypeScript (high-quality SDK support and good compatibility in many execution environments e.g. MCPB. Plus AI models are good at generating TypeScript code, benefiting from its broad usage, static typing and good linting tools)
- **Transport**: Streamable HTTP for remote servers, using stateless JSON (simpler to scale and maintain, as opposed to stateful sessions and streaming responses). stdio for local servers.
**Load framework documentation:**
- **MCP Best Practices**: [📋 View Best Practices](./reference/mcp_best_practices.md) - Core guidelines
**For TypeScript (recommended):**
- **TypeScript SDK**: Use WebFetch to load `https://raw.githubusercontent.com/modelcontextprotocol/typescript-sdk/main/README.md`
- [⚡ TypeScript Guide](./reference/node_mcp_server.md) - TypeScript patterns and examples
**For Python:**
- **Python SDK**: Use WebFetch to load `https://raw.githubusercontent.com/modelcontextprotocol/python-sdk/main/README.md`
- [🐍 Python Guide](./reference/python_mcp_server.md) - Python patterns and examples
#### 1.4 Plan Your Implementation
**Understand the API:**
Review the service's API documentation to identify key endpoints, authentication requirements, and data models. Use web search and WebFetch as needed.
**Tool Selection:**
Prioritize comprehensive API coverage. List endpoints to implement, starting with the most common operations.
---
### Phase 2: Implementation
#### 2.1 Set Up Project Structure
See language-specific guides for project setup:
- [⚡ TypeScript Guide](./reference/node_mcp_server.md) - Project structure, package.json, tsconfig.json
- [🐍 Python Guide](./reference/python_mcp_server.md) - Module organization, dependencies
#### 2.2 Implement Core Infrastructure
Create shared utilities:
- API client with authentication
- Error handling helpers
- Response formatting (JSON/Markdown)
- Pagination support
#### 2.3 Implement Tools
For each tool:
**Input Schema:**
- Use Zod (TypeScript) or Pydantic (Python)
- Include constraints and clear descriptions
- Add examples in field descriptions
**Output Schema:**
- Define `outputSchema` where possible for structured data
- Use `structuredContent` in tool responses (TypeScript SDK feature)
- Helps clients understand and process tool outputs
**Tool Description:**
- Concise summary of functionality
- Parameter descriptions
- Return type schema
**Implementation:**
- Async/await for I/O operations
- Proper error handling with actionable messages
- Support pagination where applicable
- Return both text content and structured data when using modern SDKs
**Annotations:**
- `readOnlyHint`: true/false
- `destructiveHint`: true/false
- `idempotentHint`: true/false
- `openWorldHint`: true/false
---
### Phase 3: Review and Test
#### 3.1 Code Quality
Review for:
- No duplicated code (DRY principle)
- Consistent error handling
- Full type coverage
- Clear tool descriptions
#### 3.2 Build and Test
**TypeScript:**
- Run `npm run build` to verify compilation
- Test with MCP Inspector: `npx @modelcontextprotocol/inspector`
**Python:**
- Verify syntax: `python -m py_compile your_server.py`
- Test with MCP Inspector
See language-specific guides for detailed testing approaches and quality checklists.
---
### Phase 4: Create Evaluations
After implementing your MCP server, create comprehensive evaluations to test its effectiveness.
**Load [✅ Evaluation Guide](./reference/evaluation.md) for complete evaluation guidelines.**
#### 4.1 Understand Evaluation Purpose
Use evaluations to test whether LLMs can effectively use your MCP server to answer realistic, complex questions.
#### 4.2 Create 10 Evaluation Questions
To create effective evaluations, follow the process outlined in the evaluation guide:
1. **Tool Inspection**: List available tools and understand their capabilities
2. **Content Exploration**: Use READ-ONLY operations to explore available data
3. **Question Generation**: Create 10 complex, realistic questions
4. **Answer Verification**: Solve each question yourself to verify answers
#### 4.3 Evaluation Requirements
Ensure each question is:
- **Independent**: Not dependent on other questions
- **Read-only**: Only non-destructive operations required
- **Complex**: Requiring multiple tool calls and deep exploration
- **Realistic**: Based on real use cases humans would care about
- **Verifiable**: Single, clear answer that can be verified by string comparison
- **Stable**: Answer won't change over time
#### 4.4 Output Format
Create an XML file with this structure:
```xml
<evaluation>
<qa_pair>
<question>Find discussions about AI model launches with animal codenames. One model needed a specific safety designation that uses the format ASL-X. What number X was being determined for the model named after a spotted wild cat?</question>
<answer>3</answer>
</qa_pair>
<!-- More qa_pairs... -->
</evaluation>
```
---
# Reference Files
## 📚 Documentation Library
Load these resources as needed during development:
### Core MCP Documentation (Load First)
- **MCP Protocol**: Start with sitemap at `https://modelcontextprotocol.io/sitemap.xml`, then fetch specific pages with `.md` suffix
- [📋 MCP Best Practices](./reference/mcp_best_practices.md) - Universal MCP guidelines including:
- Server and tool naming conventions
- Response format guidelines (JSON vs Markdown)
- Pagination best practices
- Transport selection (streamable HTTP vs stdio)
- Security and error handling standards
### SDK Documentation (Load During Phase 1/2)
- **Python SDK**: Fetch from `https://raw.githubusercontent.com/modelcontextprotocol/python-sdk/main/README.md`
- **TypeScript SDK**: Fetch from `https://raw.githubusercontent.com/modelcontextprotocol/typescript-sdk/main/README.md`
### Language-Specific Implementation Guides (Load During Phase 2)
- [🐍 Python Implementation Guide](./reference/python_mcp_server.md) - Complete Python/FastMCP guide with:
- Server initialization patterns
- Pydantic model examples
- Tool registration with `@mcp.tool`
- Complete working examples
- Quality checklist
- [⚡ TypeScript Implementation Guide](./reference/node_mcp_server.md) - Complete TypeScript guide with:
- Project structure
- Zod schema patterns
- Tool registration with `server.registerTool`
- Complete working examples
- Quality checklist
### Evaluation Guide (Load During Phase 4)
- [✅ Evaluation Guide](./reference/evaluation.md) - Complete evaluation creation guide with:
- Question creation guidelines
- Answer verification strategies
- XML format specifications
- Example questions and answers
- Running an evaluation with the provided scripts
-602
View File
@@ -1,602 +0,0 @@
# MCP Server Evaluation Guide
## Overview
This document provides guidance on creating comprehensive evaluations for MCP servers. Evaluations test whether LLMs can effectively use your MCP server to answer realistic, complex questions using only the tools provided.
---
## Quick Reference
### Evaluation Requirements
- Create 10 human-readable questions
- Questions must be READ-ONLY, INDEPENDENT, NON-DESTRUCTIVE
- Each question requires multiple tool calls (potentially dozens)
- Answers must be single, verifiable values
- Answers must be STABLE (won't change over time)
### Output Format
```xml
<evaluation>
<qa_pair>
<question>Your question here</question>
<answer>Single verifiable answer</answer>
</qa_pair>
</evaluation>
```
---
## Purpose of Evaluations
The measure of quality of an MCP server is NOT how well or comprehensively the server implements tools, but how well these implementations (input/output schemas, docstrings/descriptions, functionality) enable LLMs with no other context and access ONLY to the MCP servers to answer realistic and difficult questions.
## Evaluation Overview
Create 10 human-readable questions requiring ONLY READ-ONLY, INDEPENDENT, NON-DESTRUCTIVE, and IDEMPOTENT operations to answer. Each question should be:
- Realistic
- Clear and concise
- Unambiguous
- Complex, requiring potentially dozens of tool calls or steps
- Answerable with a single, verifiable value that you identify in advance
## Question Guidelines
### Core Requirements
1. **Questions MUST be independent**
- Each question should NOT depend on the answer to any other question
- Should not assume prior write operations from processing another question
2. **Questions MUST require ONLY NON-DESTRUCTIVE AND IDEMPOTENT tool use**
- Should not instruct or require modifying state to arrive at the correct answer
3. **Questions must be REALISTIC, CLEAR, CONCISE, and COMPLEX**
- Must require another LLM to use multiple (potentially dozens of) tools or steps to answer
### Complexity and Depth
4. **Questions must require deep exploration**
- Consider multi-hop questions requiring multiple sub-questions and sequential tool calls
- Each step should benefit from information found in previous questions
5. **Questions may require extensive paging**
- May need paging through multiple pages of results
- May require querying old data (1-2 years out-of-date) to find niche information
- The questions must be DIFFICULT
6. **Questions must require deep understanding**
- Rather than surface-level knowledge
- May pose complex ideas as True/False questions requiring evidence
- May use multiple-choice format where LLM must search different hypotheses
7. **Questions must not be solvable with straightforward keyword search**
- Do not include specific keywords from the target content
- Use synonyms, related concepts, or paraphrases
- Require multiple searches, analyzing multiple related items, extracting context, then deriving the answer
### Tool Testing
8. **Questions should stress-test tool return values**
- May elicit tools returning large JSON objects or lists, overwhelming the LLM
- Should require understanding multiple modalities of data:
- IDs and names
- Timestamps and datetimes (months, days, years, seconds)
- File IDs, names, extensions, and mimetypes
- URLs, GIDs, etc.
- Should probe the tool's ability to return all useful forms of data
9. **Questions should MOSTLY reflect real human use cases**
- The kinds of information retrieval tasks that HUMANS assisted by an LLM would care about
10. **Questions may require dozens of tool calls**
- This challenges LLMs with limited context
- Encourages MCP server tools to reduce information returned
11. **Include ambiguous questions**
- May be ambiguous OR require difficult decisions on which tools to call
- Force the LLM to potentially make mistakes or misinterpret
- Ensure that despite AMBIGUITY, there is STILL A SINGLE VERIFIABLE ANSWER
### Stability
12. **Questions must be designed so the answer DOES NOT CHANGE**
- Do not ask questions that rely on "current state" which is dynamic
- For example, do not count:
- Number of reactions to a post
- Number of replies to a thread
- Number of members in a channel
13. **DO NOT let the MCP server RESTRICT the kinds of questions you create**
- Create challenging and complex questions
- Some may not be solvable with the available MCP server tools
- Questions may require specific output formats (datetime vs. epoch time, JSON vs. MARKDOWN)
- Questions may require dozens of tool calls to complete
## Answer Guidelines
### Verification
1. **Answers must be VERIFIABLE via direct string comparison**
- If the answer can be re-written in many formats, clearly specify the output format in the QUESTION
- Examples: "Use YYYY/MM/DD.", "Respond True or False.", "Answer A, B, C, or D and nothing else."
- Answer should be a single VERIFIABLE value such as:
- User ID, user name, display name, first name, last name
- Channel ID, channel name
- Message ID, string
- URL, title
- Numerical quantity
- Timestamp, datetime
- Boolean (for True/False questions)
- Email address, phone number
- File ID, file name, file extension
- Multiple choice answer
- Answers must not require special formatting or complex, structured output
- Answer will be verified using DIRECT STRING COMPARISON
### Readability
2. **Answers should generally prefer HUMAN-READABLE formats**
- Examples: names, first name, last name, datetime, file name, message string, URL, yes/no, true/false, a/b/c/d
- Rather than opaque IDs (though IDs are acceptable)
- The VAST MAJORITY of answers should be human-readable
### Stability
3. **Answers must be STABLE/STATIONARY**
- Look at old content (e.g., conversations that have ended, projects that have launched, questions answered)
- Create QUESTIONS based on "closed" concepts that will always return the same answer
- Questions may ask to consider a fixed time window to insulate from non-stationary answers
- Rely on context UNLIKELY to change
- Example: if finding a paper name, be SPECIFIC enough so answer is not confused with papers published later
4. **Answers must be CLEAR and UNAMBIGUOUS**
- Questions must be designed so there is a single, clear answer
- Answer can be derived from using the MCP server tools
### Diversity
5. **Answers must be DIVERSE**
- Answer should be a single VERIFIABLE value in diverse modalities and formats
- User concept: user ID, user name, display name, first name, last name, email address, phone number
- Channel concept: channel ID, channel name, channel topic
- Message concept: message ID, message string, timestamp, month, day, year
6. **Answers must NOT be complex structures**
- Not a list of values
- Not a complex object
- Not a list of IDs or strings
- Not natural language text
- UNLESS the answer can be straightforwardly verified using DIRECT STRING COMPARISON
- And can be realistically reproduced
- It should be unlikely that an LLM would return the same list in any other order or format
## Evaluation Process
### Step 1: Documentation Inspection
Read the documentation of the target API to understand:
- Available endpoints and functionality
- If ambiguity exists, fetch additional information from the web
- Parallelize this step AS MUCH AS POSSIBLE
- Ensure each subagent is ONLY examining documentation from the file system or on the web
### Step 2: Tool Inspection
List the tools available in the MCP server:
- Inspect the MCP server directly
- Understand input/output schemas, docstrings, and descriptions
- WITHOUT calling the tools themselves at this stage
### Step 3: Developing Understanding
Repeat steps 1 & 2 until you have a good understanding:
- Iterate multiple times
- Think about the kinds of tasks you want to create
- Refine your understanding
- At NO stage should you READ the code of the MCP server implementation itself
- Use your intuition and understanding to create reasonable, realistic, but VERY challenging tasks
### Step 4: Read-Only Content Inspection
After understanding the API and tools, USE the MCP server tools:
- Inspect content using READ-ONLY and NON-DESTRUCTIVE operations ONLY
- Goal: identify specific content (e.g., users, channels, messages, projects, tasks) for creating realistic questions
- Should NOT call any tools that modify state
- Will NOT read the code of the MCP server implementation itself
- Parallelize this step with individual sub-agents pursuing independent explorations
- Ensure each subagent is only performing READ-ONLY, NON-DESTRUCTIVE, and IDEMPOTENT operations
- BE CAREFUL: SOME TOOLS may return LOTS OF DATA which would cause you to run out of CONTEXT
- Make INCREMENTAL, SMALL, AND TARGETED tool calls for exploration
- In all tool call requests, use the `limit` parameter to limit results (<10)
- Use pagination
### Step 5: Task Generation
After inspecting the content, create 10 human-readable questions:
- An LLM should be able to answer these with the MCP server
- Follow all question and answer guidelines above
## Output Format
Each QA pair consists of a question and an answer. The output should be an XML file with this structure:
```xml
<evaluation>
<qa_pair>
<question>Find the project created in Q2 2024 with the highest number of completed tasks. What is the project name?</question>
<answer>Website Redesign</answer>
</qa_pair>
<qa_pair>
<question>Search for issues labeled as "bug" that were closed in March 2024. Which user closed the most issues? Provide their username.</question>
<answer>sarah_dev</answer>
</qa_pair>
<qa_pair>
<question>Look for pull requests that modified files in the /api directory and were merged between January 1 and January 31, 2024. How many different contributors worked on these PRs?</question>
<answer>7</answer>
</qa_pair>
<qa_pair>
<question>Find the repository with the most stars that was created before 2023. What is the repository name?</question>
<answer>data-pipeline</answer>
</qa_pair>
</evaluation>
```
## Evaluation Examples
### Good Questions
**Example 1: Multi-hop question requiring deep exploration (GitHub MCP)**
```xml
<qa_pair>
<question>Find the repository that was archived in Q3 2023 and had previously been the most forked project in the organization. What was the primary programming language used in that repository?</question>
<answer>Python</answer>
</qa_pair>
```
This question is good because:
- Requires multiple searches to find archived repositories
- Needs to identify which had the most forks before archival
- Requires examining repository details for the language
- Answer is a simple, verifiable value
- Based on historical (closed) data that won't change
**Example 2: Requires understanding context without keyword matching (Project Management MCP)**
```xml
<qa_pair>
<question>Locate the initiative focused on improving customer onboarding that was completed in late 2023. The project lead created a retrospective document after completion. What was the lead's role title at that time?</question>
<answer>Product Manager</answer>
</qa_pair>
```
This question is good because:
- Doesn't use specific project name ("initiative focused on improving customer onboarding")
- Requires finding completed projects from specific timeframe
- Needs to identify the project lead and their role
- Requires understanding context from retrospective documents
- Answer is human-readable and stable
- Based on completed work (won't change)
**Example 3: Complex aggregation requiring multiple steps (Issue Tracker MCP)**
```xml
<qa_pair>
<question>Among all bugs reported in January 2024 that were marked as critical priority, which assignee resolved the highest percentage of their assigned bugs within 48 hours? Provide the assignee's username.</question>
<answer>alex_eng</answer>
</qa_pair>
```
This question is good because:
- Requires filtering bugs by date, priority, and status
- Needs to group by assignee and calculate resolution rates
- Requires understanding timestamps to determine 48-hour windows
- Tests pagination (potentially many bugs to process)
- Answer is a single username
- Based on historical data from specific time period
**Example 4: Requires synthesis across multiple data types (CRM MCP)**
```xml
<qa_pair>
<question>Find the account that upgraded from the Starter to Enterprise plan in Q4 2023 and had the highest annual contract value. What industry does this account operate in?</question>
<answer>Healthcare</answer>
</qa_pair>
```
This question is good because:
- Requires understanding subscription tier changes
- Needs to identify upgrade events in specific timeframe
- Requires comparing contract values
- Must access account industry information
- Answer is simple and verifiable
- Based on completed historical transactions
### Poor Questions
**Example 1: Answer changes over time**
```xml
<qa_pair>
<question>How many open issues are currently assigned to the engineering team?</question>
<answer>47</answer>
</qa_pair>
```
This question is poor because:
- The answer will change as issues are created, closed, or reassigned
- Not based on stable/stationary data
- Relies on "current state" which is dynamic
**Example 2: Too easy with keyword search**
```xml
<qa_pair>
<question>Find the pull request with title "Add authentication feature" and tell me who created it.</question>
<answer>developer123</answer>
</qa_pair>
```
This question is poor because:
- Can be solved with a straightforward keyword search for exact title
- Doesn't require deep exploration or understanding
- No synthesis or analysis needed
**Example 3: Ambiguous answer format**
```xml
<qa_pair>
<question>List all the repositories that have Python as their primary language.</question>
<answer>repo1, repo2, repo3, data-pipeline, ml-tools</answer>
</qa_pair>
```
This question is poor because:
- Answer is a list that could be returned in any order
- Difficult to verify with direct string comparison
- LLM might format differently (JSON array, comma-separated, newline-separated)
- Better to ask for a specific aggregate (count) or superlative (most stars)
## Verification Process
After creating evaluations:
1. **Examine the XML file** to understand the schema
2. **Load each task instruction** and in parallel using the MCP server and tools, identify the correct answer by attempting to solve the task YOURSELF
3. **Flag any operations** that require WRITE or DESTRUCTIVE operations
4. **Accumulate all CORRECT answers** and replace any incorrect answers in the document
5. **Remove any `<qa_pair>`** that require WRITE or DESTRUCTIVE operations
Remember to parallelize solving tasks to avoid running out of context, then accumulate all answers and make changes to the file at the end.
## Tips for Creating Quality Evaluations
1. **Think Hard and Plan Ahead** before generating tasks
2. **Parallelize Where Opportunity Arises** to speed up the process and manage context
3. **Focus on Realistic Use Cases** that humans would actually want to accomplish
4. **Create Challenging Questions** that test the limits of the MCP server's capabilities
5. **Ensure Stability** by using historical data and closed concepts
6. **Verify Answers** by solving the questions yourself using the MCP server tools
7. **Iterate and Refine** based on what you learn during the process
---
# Running Evaluations
After creating your evaluation file, you can use the provided evaluation harness to test your MCP server.
## Setup
1. **Install Dependencies**
```bash
pip install -r scripts/requirements.txt
```
Or install manually:
```bash
pip install anthropic mcp
```
2. **Set API Key**
```bash
export ANTHROPIC_API_KEY=your_api_key_here
```
## Evaluation File Format
Evaluation files use XML format with `<qa_pair>` elements:
```xml
<evaluation>
<qa_pair>
<question>Find the project created in Q2 2024 with the highest number of completed tasks. What is the project name?</question>
<answer>Website Redesign</answer>
</qa_pair>
<qa_pair>
<question>Search for issues labeled as "bug" that were closed in March 2024. Which user closed the most issues? Provide their username.</question>
<answer>sarah_dev</answer>
</qa_pair>
</evaluation>
```
## Running Evaluations
The evaluation script (`scripts/evaluation.py`) supports three transport types:
**Important:**
- **stdio transport**: The evaluation script automatically launches and manages the MCP server process for you. Do not run the server manually.
- **sse/http transports**: You must start the MCP server separately before running the evaluation. The script connects to the already-running server at the specified URL.
### 1. Local STDIO Server
For locally-run MCP servers (script launches the server automatically):
```bash
python scripts/evaluation.py \
-t stdio \
-c python \
-a my_mcp_server.py \
evaluation.xml
```
With environment variables:
```bash
python scripts/evaluation.py \
-t stdio \
-c python \
-a my_mcp_server.py \
-e API_KEY=abc123 \
-e DEBUG=true \
evaluation.xml
```
### 2. Server-Sent Events (SSE)
For SSE-based MCP servers (you must start the server first):
```bash
python scripts/evaluation.py \
-t sse \
-u https://example.com/mcp \
-H "Authorization: Bearer token123" \
-H "X-Custom-Header: value" \
evaluation.xml
```
### 3. HTTP (Streamable HTTP)
For HTTP-based MCP servers (you must start the server first):
```bash
python scripts/evaluation.py \
-t http \
-u https://example.com/mcp \
-H "Authorization: Bearer token123" \
evaluation.xml
```
## Command-Line Options
```
usage: evaluation.py [-h] [-t {stdio,sse,http}] [-m MODEL] [-c COMMAND]
[-a ARGS [ARGS ...]] [-e ENV [ENV ...]] [-u URL]
[-H HEADERS [HEADERS ...]] [-o OUTPUT]
eval_file
positional arguments:
eval_file Path to evaluation XML file
optional arguments:
-h, --help Show help message
-t, --transport Transport type: stdio, sse, or http (default: stdio)
-m, --model Claude model to use (default: claude-3-7-sonnet-20250219)
-o, --output Output file for report (default: print to stdout)
stdio options:
-c, --command Command to run MCP server (e.g., python, node)
-a, --args Arguments for the command (e.g., server.py)
-e, --env Environment variables in KEY=VALUE format
sse/http options:
-u, --url MCP server URL
-H, --header HTTP headers in 'Key: Value' format
```
## Output
The evaluation script generates a detailed report including:
- **Summary Statistics**:
- Accuracy (correct/total)
- Average task duration
- Average tool calls per task
- Total tool calls
- **Per-Task Results**:
- Prompt and expected response
- Actual response from the agent
- Whether the answer was correct (✅/❌)
- Duration and tool call details
- Agent's summary of its approach
- Agent's feedback on the tools
### Save Report to File
```bash
python scripts/evaluation.py \
-t stdio \
-c python \
-a my_server.py \
-o evaluation_report.md \
evaluation.xml
```
## Complete Example Workflow
Here's a complete example of creating and running an evaluation:
1. **Create your evaluation file** (`my_evaluation.xml`):
```xml
<evaluation>
<qa_pair>
<question>Find the user who created the most issues in January 2024. What is their username?</question>
<answer>alice_developer</answer>
</qa_pair>
<qa_pair>
<question>Among all pull requests merged in Q1 2024, which repository had the highest number? Provide the repository name.</question>
<answer>backend-api</answer>
</qa_pair>
<qa_pair>
<question>Find the project that was completed in December 2023 and had the longest duration from start to finish. How many days did it take?</question>
<answer>127</answer>
</qa_pair>
</evaluation>
```
2. **Install dependencies**:
```bash
pip install -r scripts/requirements.txt
export ANTHROPIC_API_KEY=your_api_key
```
3. **Run evaluation**:
```bash
python scripts/evaluation.py \
-t stdio \
-c python \
-a github_mcp_server.py \
-e GITHUB_TOKEN=ghp_xxx \
-o github_eval_report.md \
my_evaluation.xml
```
4. **Review the report** in `github_eval_report.md` to:
- See which questions passed/failed
- Read the agent's feedback on your tools
- Identify areas for improvement
- Iterate on your MCP server design
## Troubleshooting
### Connection Errors
If you get connection errors:
- **STDIO**: Verify the command and arguments are correct
- **SSE/HTTP**: Check the URL is accessible and headers are correct
- Ensure any required API keys are set in environment variables or headers
### Low Accuracy
If many evaluations fail:
- Review the agent's feedback for each task
- Check if tool descriptions are clear and comprehensive
- Verify input parameters are well-documented
- Consider whether tools return too much or too little data
- Ensure error messages are actionable
### Timeout Issues
If tasks are timing out:
- Use a more capable model (e.g., `claude-3-7-sonnet-20250219`)
- Check if tools are returning too much data
- Verify pagination is working correctly
- Consider simplifying complex questions
@@ -1,249 +0,0 @@
# MCP Server Best Practices
## Quick Reference
### Server Naming
- **Python**: `{service}_mcp` (e.g., `slack_mcp`)
- **Node/TypeScript**: `{service}-mcp-server` (e.g., `slack-mcp-server`)
### Tool Naming
- Use snake_case with service prefix
- Format: `{service}_{action}_{resource}`
- Example: `slack_send_message`, `github_create_issue`
### Response Formats
- Support both JSON and Markdown formats
- JSON for programmatic processing
- Markdown for human readability
### Pagination
- Always respect `limit` parameter
- Return `has_more`, `next_offset`, `total_count`
- Default to 20-50 items
### Transport
- **Streamable HTTP**: For remote servers, multi-client scenarios
- **stdio**: For local integrations, command-line tools
- Avoid SSE (deprecated in favor of streamable HTTP)
---
## Server Naming Conventions
Follow these standardized naming patterns:
**Python**: Use format `{service}_mcp` (lowercase with underscores)
- Examples: `slack_mcp`, `github_mcp`, `jira_mcp`
**Node/TypeScript**: Use format `{service}-mcp-server` (lowercase with hyphens)
- Examples: `slack-mcp-server`, `github-mcp-server`, `jira-mcp-server`
The name should be general, descriptive of the service being integrated, easy to infer from the task description, and without version numbers.
---
## Tool Naming and Design
### Tool Naming
1. **Use snake_case**: `search_users`, `create_project`, `get_channel_info`
2. **Include service prefix**: Anticipate that your MCP server may be used alongside other MCP servers
- Use `slack_send_message` instead of just `send_message`
- Use `github_create_issue` instead of just `create_issue`
3. **Be action-oriented**: Start with verbs (get, list, search, create, etc.)
4. **Be specific**: Avoid generic names that could conflict with other servers
### Tool Design
- Tool descriptions must narrowly and unambiguously describe functionality
- Descriptions must precisely match actual functionality
- Provide tool annotations (readOnlyHint, destructiveHint, idempotentHint, openWorldHint)
- Keep tool operations focused and atomic
---
## Response Formats
All tools that return data should support multiple formats:
### JSON Format (`response_format="json"`)
- Machine-readable structured data
- Include all available fields and metadata
- Consistent field names and types
- Use for programmatic processing
### Markdown Format (`response_format="markdown"`, typically default)
- Human-readable formatted text
- Use headers, lists, and formatting for clarity
- Convert timestamps to human-readable format
- Show display names with IDs in parentheses
- Omit verbose metadata
---
## Pagination
For tools that list resources:
- **Always respect the `limit` parameter**
- **Implement pagination**: Use `offset` or cursor-based pagination
- **Return pagination metadata**: Include `has_more`, `next_offset`/`next_cursor`, `total_count`
- **Never load all results into memory**: Especially important for large datasets
- **Default to reasonable limits**: 20-50 items is typical
Example pagination response:
```json
{
"total": 150,
"count": 20,
"offset": 0,
"items": [...],
"has_more": true,
"next_offset": 20
}
```
---
## Transport Options
### Streamable HTTP
**Best for**: Remote servers, web services, multi-client scenarios
**Characteristics**:
- Bidirectional communication over HTTP
- Supports multiple simultaneous clients
- Can be deployed as a web service
- Enables server-to-client notifications
**Use when**:
- Serving multiple clients simultaneously
- Deploying as a cloud service
- Integration with web applications
### stdio
**Best for**: Local integrations, command-line tools
**Characteristics**:
- Standard input/output stream communication
- Simple setup, no network configuration needed
- Runs as a subprocess of the client
**Use when**:
- Building tools for local development environments
- Integrating with desktop applications
- Single-user, single-session scenarios
**Note**: stdio servers should NOT log to stdout (use stderr for logging)
### Transport Selection
| Criterion | stdio | Streamable HTTP |
|-----------|-------|-----------------|
| **Deployment** | Local | Remote |
| **Clients** | Single | Multiple |
| **Complexity** | Low | Medium |
| **Real-time** | No | Yes |
---
## Security Best Practices
### Authentication and Authorization
**OAuth 2.1**:
- Use secure OAuth 2.1 with certificates from recognized authorities
- Validate access tokens before processing requests
- Only accept tokens specifically intended for your server
**API Keys**:
- Store API keys in environment variables, never in code
- Validate keys on server startup
- Provide clear error messages when authentication fails
### Input Validation
- Sanitize file paths to prevent directory traversal
- Validate URLs and external identifiers
- Check parameter sizes and ranges
- Prevent command injection in system calls
- Use schema validation (Pydantic/Zod) for all inputs
### Error Handling
- Don't expose internal errors to clients
- Log security-relevant errors server-side
- Provide helpful but not revealing error messages
- Clean up resources after errors
### DNS Rebinding Protection
For streamable HTTP servers running locally:
- Enable DNS rebinding protection
- Validate the `Origin` header on all incoming connections
- Bind to `127.0.0.1` rather than `0.0.0.0`
---
## Tool Annotations
Provide annotations to help clients understand tool behavior:
| Annotation | Type | Default | Description |
|-----------|------|---------|-------------|
| `readOnlyHint` | boolean | false | Tool does not modify its environment |
| `destructiveHint` | boolean | true | Tool may perform destructive updates |
| `idempotentHint` | boolean | false | Repeated calls with same args have no additional effect |
| `openWorldHint` | boolean | true | Tool interacts with external entities |
**Important**: Annotations are hints, not security guarantees. Clients should not make security-critical decisions based solely on annotations.
---
## Error Handling
- Use standard JSON-RPC error codes
- Report tool errors within result objects (not protocol-level errors)
- Provide helpful, specific error messages with suggested next steps
- Don't expose internal implementation details
- Clean up resources properly on errors
Example error handling:
```typescript
try {
const result = performOperation();
return { content: [{ type: "text", text: result }] };
} catch (error) {
return {
isError: true,
content: [{
type: "text",
text: `Error: ${error.message}. Try using filter='active_only' to reduce results.`
}]
};
}
```
---
## Testing Requirements
Comprehensive testing should cover:
- **Functional testing**: Verify correct execution with valid/invalid inputs
- **Integration testing**: Test interaction with external systems
- **Security testing**: Validate auth, input sanitization, rate limiting
- **Performance testing**: Check behavior under load, timeouts
- **Error handling**: Ensure proper error reporting and cleanup
---
## Documentation Requirements
- Provide clear documentation of all tools and capabilities
- Include working examples (at least 3 per major feature)
- Document security considerations
- Specify required permissions and access levels
- Document rate limits and performance characteristics
@@ -1,970 +0,0 @@
# Node/TypeScript MCP Server Implementation Guide
## Overview
This document provides Node/TypeScript-specific best practices and examples for implementing MCP servers using the MCP TypeScript SDK. It covers project structure, server setup, tool registration patterns, input validation with Zod, error handling, and complete working examples.
---
## Quick Reference
### Key Imports
```typescript
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { StreamableHTTPServerTransport } from "@modelcontextprotocol/sdk/server/streamableHttp.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
import express from "express";
import { z } from "zod";
```
### Server Initialization
```typescript
const server = new McpServer({
name: "service-mcp-server",
version: "1.0.0"
});
```
### Tool Registration Pattern
```typescript
server.registerTool(
"tool_name",
{
title: "Tool Display Name",
description: "What the tool does",
inputSchema: { param: z.string() },
outputSchema: { result: z.string() }
},
async ({ param }) => {
const output = { result: `Processed: ${param}` };
return {
content: [{ type: "text", text: JSON.stringify(output) }],
structuredContent: output // Modern pattern for structured data
};
}
);
```
---
## MCP TypeScript SDK
The official MCP TypeScript SDK provides:
- `McpServer` class for server initialization
- `registerTool` method for tool registration
- Zod schema integration for runtime input validation
- Type-safe tool handler implementations
**IMPORTANT - Use Modern APIs Only:**
- **DO use**: `server.registerTool()`, `server.registerResource()`, `server.registerPrompt()`
- **DO NOT use**: Old deprecated APIs such as `server.tool()`, `server.setRequestHandler(ListToolsRequestSchema, ...)`, or manual handler registration
- The `register*` methods provide better type safety, automatic schema handling, and are the recommended approach
See the MCP SDK documentation in the references for complete details.
## Server Naming Convention
Node/TypeScript MCP servers must follow this naming pattern:
- **Format**: `{service}-mcp-server` (lowercase with hyphens)
- **Examples**: `github-mcp-server`, `jira-mcp-server`, `stripe-mcp-server`
The name should be:
- General (not tied to specific features)
- Descriptive of the service/API being integrated
- Easy to infer from the task description
- Without version numbers or dates
## Project Structure
Create the following structure for Node/TypeScript MCP servers:
```
{service}-mcp-server/
├── package.json
├── tsconfig.json
├── README.md
├── src/
│ ├── index.ts # Main entry point with McpServer initialization
│ ├── types.ts # TypeScript type definitions and interfaces
│ ├── tools/ # Tool implementations (one file per domain)
│ ├── services/ # API clients and shared utilities
│ ├── schemas/ # Zod validation schemas
│ └── constants.ts # Shared constants (API_URL, CHARACTER_LIMIT, etc.)
└── dist/ # Built JavaScript files (entry point: dist/index.js)
```
## Tool Implementation
### Tool Naming
Use snake_case for tool names (e.g., "search_users", "create_project", "get_channel_info") with clear, action-oriented names.
**Avoid Naming Conflicts**: Include the service context to prevent overlaps:
- Use "slack_send_message" instead of just "send_message"
- Use "github_create_issue" instead of just "create_issue"
- Use "asana_list_tasks" instead of just "list_tasks"
### Tool Structure
Tools are registered using the `registerTool` method with the following requirements:
- Use Zod schemas for runtime input validation and type safety
- The `description` field must be explicitly provided - JSDoc comments are NOT automatically extracted
- Explicitly provide `title`, `description`, `inputSchema`, and `annotations`
- The `inputSchema` must be a Zod schema object (not a JSON schema)
- Type all parameters and return values explicitly
```typescript
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { z } from "zod";
const server = new McpServer({
name: "example-mcp",
version: "1.0.0"
});
// Zod schema for input validation
const UserSearchInputSchema = z.object({
query: z.string()
.min(2, "Query must be at least 2 characters")
.max(200, "Query must not exceed 200 characters")
.describe("Search string to match against names/emails"),
limit: z.number()
.int()
.min(1)
.max(100)
.default(20)
.describe("Maximum results to return"),
offset: z.number()
.int()
.min(0)
.default(0)
.describe("Number of results to skip for pagination"),
response_format: z.nativeEnum(ResponseFormat)
.default(ResponseFormat.MARKDOWN)
.describe("Output format: 'markdown' for human-readable or 'json' for machine-readable")
}).strict();
// Type definition from Zod schema
type UserSearchInput = z.infer<typeof UserSearchInputSchema>;
server.registerTool(
"example_search_users",
{
title: "Search Example Users",
description: `Search for users in the Example system by name, email, or team.
This tool searches across all user profiles in the Example platform, supporting partial matches and various search filters. It does NOT create or modify users, only searches existing ones.
Args:
- query (string): Search string to match against names/emails
- limit (number): Maximum results to return, between 1-100 (default: 20)
- offset (number): Number of results to skip for pagination (default: 0)
- response_format ('markdown' | 'json'): Output format (default: 'markdown')
Returns:
For JSON format: Structured data with schema:
{
"total": number, // Total number of matches found
"count": number, // Number of results in this response
"offset": number, // Current pagination offset
"users": [
{
"id": string, // User ID (e.g., "U123456789")
"name": string, // Full name (e.g., "John Doe")
"email": string, // Email address
"team": string, // Team name (optional)
"active": boolean // Whether user is active
}
],
"has_more": boolean, // Whether more results are available
"next_offset": number // Offset for next page (if has_more is true)
}
Examples:
- Use when: "Find all marketing team members" -> params with query="team:marketing"
- Use when: "Search for John's account" -> params with query="john"
- Don't use when: You need to create a user (use example_create_user instead)
Error Handling:
- Returns "Error: Rate limit exceeded" if too many requests (429 status)
- Returns "No users found matching '<query>'" if search returns empty`,
inputSchema: UserSearchInputSchema,
annotations: {
readOnlyHint: true,
destructiveHint: false,
idempotentHint: true,
openWorldHint: true
}
},
async (params: UserSearchInput) => {
try {
// Input validation is handled by Zod schema
// Make API request using validated parameters
const data = await makeApiRequest<any>(
"users/search",
"GET",
undefined,
{
q: params.query,
limit: params.limit,
offset: params.offset
}
);
const users = data.users || [];
const total = data.total || 0;
if (!users.length) {
return {
content: [{
type: "text",
text: `No users found matching '${params.query}'`
}]
};
}
// Prepare structured output
const output = {
total,
count: users.length,
offset: params.offset,
users: users.map((user: any) => ({
id: user.id,
name: user.name,
email: user.email,
...(user.team ? { team: user.team } : {}),
active: user.active ?? true
})),
has_more: total > params.offset + users.length,
...(total > params.offset + users.length ? {
next_offset: params.offset + users.length
} : {})
};
// Format text representation based on requested format
let textContent: string;
if (params.response_format === ResponseFormat.MARKDOWN) {
const lines = [`# User Search Results: '${params.query}'`, "",
`Found ${total} users (showing ${users.length})`, ""];
for (const user of users) {
lines.push(`## ${user.name} (${user.id})`);
lines.push(`- **Email**: ${user.email}`);
if (user.team) lines.push(`- **Team**: ${user.team}`);
lines.push("");
}
textContent = lines.join("\n");
} else {
textContent = JSON.stringify(output, null, 2);
}
return {
content: [{ type: "text", text: textContent }],
structuredContent: output // Modern pattern for structured data
};
} catch (error) {
return {
content: [{
type: "text",
text: handleApiError(error)
}]
};
}
}
);
```
## Zod Schemas for Input Validation
Zod provides runtime type validation:
```typescript
import { z } from "zod";
// Basic schema with validation
const CreateUserSchema = z.object({
name: z.string()
.min(1, "Name is required")
.max(100, "Name must not exceed 100 characters"),
email: z.string()
.email("Invalid email format"),
age: z.number()
.int("Age must be a whole number")
.min(0, "Age cannot be negative")
.max(150, "Age cannot be greater than 150")
}).strict(); // Use .strict() to forbid extra fields
// Enums
enum ResponseFormat {
MARKDOWN = "markdown",
JSON = "json"
}
const SearchSchema = z.object({
response_format: z.nativeEnum(ResponseFormat)
.default(ResponseFormat.MARKDOWN)
.describe("Output format")
});
// Optional fields with defaults
const PaginationSchema = z.object({
limit: z.number()
.int()
.min(1)
.max(100)
.default(20)
.describe("Maximum results to return"),
offset: z.number()
.int()
.min(0)
.default(0)
.describe("Number of results to skip")
});
```
## Response Format Options
Support multiple output formats for flexibility:
```typescript
enum ResponseFormat {
MARKDOWN = "markdown",
JSON = "json"
}
const inputSchema = z.object({
query: z.string(),
response_format: z.nativeEnum(ResponseFormat)
.default(ResponseFormat.MARKDOWN)
.describe("Output format: 'markdown' for human-readable or 'json' for machine-readable")
});
```
**Markdown format**:
- Use headers, lists, and formatting for clarity
- Convert timestamps to human-readable format
- Show display names with IDs in parentheses
- Omit verbose metadata
- Group related information logically
**JSON format**:
- Return complete, structured data suitable for programmatic processing
- Include all available fields and metadata
- Use consistent field names and types
## Pagination Implementation
For tools that list resources:
```typescript
const ListSchema = z.object({
limit: z.number().int().min(1).max(100).default(20),
offset: z.number().int().min(0).default(0)
});
async function listItems(params: z.infer<typeof ListSchema>) {
const data = await apiRequest(params.limit, params.offset);
const response = {
total: data.total,
count: data.items.length,
offset: params.offset,
items: data.items,
has_more: data.total > params.offset + data.items.length,
next_offset: data.total > params.offset + data.items.length
? params.offset + data.items.length
: undefined
};
return JSON.stringify(response, null, 2);
}
```
## Character Limits and Truncation
Add a CHARACTER_LIMIT constant to prevent overwhelming responses:
```typescript
// At module level in constants.ts
export const CHARACTER_LIMIT = 25000; // Maximum response size in characters
async function searchTool(params: SearchInput) {
let result = generateResponse(data);
// Check character limit and truncate if needed
if (result.length > CHARACTER_LIMIT) {
const truncatedData = data.slice(0, Math.max(1, data.length / 2));
response.data = truncatedData;
response.truncated = true;
response.truncation_message =
`Response truncated from ${data.length} to ${truncatedData.length} items. ` +
`Use 'offset' parameter or add filters to see more results.`;
result = JSON.stringify(response, null, 2);
}
return result;
}
```
## Error Handling
Provide clear, actionable error messages:
```typescript
import axios, { AxiosError } from "axios";
function handleApiError(error: unknown): string {
if (error instanceof AxiosError) {
if (error.response) {
switch (error.response.status) {
case 404:
return "Error: Resource not found. Please check the ID is correct.";
case 403:
return "Error: Permission denied. You don't have access to this resource.";
case 429:
return "Error: Rate limit exceeded. Please wait before making more requests.";
default:
return `Error: API request failed with status ${error.response.status}`;
}
} else if (error.code === "ECONNABORTED") {
return "Error: Request timed out. Please try again.";
}
}
return `Error: Unexpected error occurred: ${error instanceof Error ? error.message : String(error)}`;
}
```
## Shared Utilities
Extract common functionality into reusable functions:
```typescript
// Shared API request function
async function makeApiRequest<T>(
endpoint: string,
method: "GET" | "POST" | "PUT" | "DELETE" = "GET",
data?: any,
params?: any
): Promise<T> {
try {
const response = await axios({
method,
url: `${API_BASE_URL}/${endpoint}`,
data,
params,
timeout: 30000,
headers: {
"Content-Type": "application/json",
"Accept": "application/json"
}
});
return response.data;
} catch (error) {
throw error;
}
}
```
## Async/Await Best Practices
Always use async/await for network requests and I/O operations:
```typescript
// Good: Async network request
async function fetchData(resourceId: string): Promise<ResourceData> {
const response = await axios.get(`${API_URL}/resource/${resourceId}`);
return response.data;
}
// Bad: Promise chains
function fetchData(resourceId: string): Promise<ResourceData> {
return axios.get(`${API_URL}/resource/${resourceId}`)
.then(response => response.data); // Harder to read and maintain
}
```
## TypeScript Best Practices
1. **Use Strict TypeScript**: Enable strict mode in tsconfig.json
2. **Define Interfaces**: Create clear interface definitions for all data structures
3. **Avoid `any`**: Use proper types or `unknown` instead of `any`
4. **Zod for Runtime Validation**: Use Zod schemas to validate external data
5. **Type Guards**: Create type guard functions for complex type checking
6. **Error Handling**: Always use try-catch with proper error type checking
7. **Null Safety**: Use optional chaining (`?.`) and nullish coalescing (`??`)
```typescript
// Good: Type-safe with Zod and interfaces
interface UserResponse {
id: string;
name: string;
email: string;
team?: string;
active: boolean;
}
const UserSchema = z.object({
id: z.string(),
name: z.string(),
email: z.string().email(),
team: z.string().optional(),
active: z.boolean()
});
type User = z.infer<typeof UserSchema>;
async function getUser(id: string): Promise<User> {
const data = await apiCall(`/users/${id}`);
return UserSchema.parse(data); // Runtime validation
}
// Bad: Using any
async function getUser(id: string): Promise<any> {
return await apiCall(`/users/${id}`); // No type safety
}
```
## Package Configuration
### package.json
```json
{
"name": "{service}-mcp-server",
"version": "1.0.0",
"description": "MCP server for {Service} API integration",
"type": "module",
"main": "dist/index.js",
"scripts": {
"start": "node dist/index.js",
"dev": "tsx watch src/index.ts",
"build": "tsc",
"clean": "rm -rf dist"
},
"engines": {
"node": ">=18"
},
"dependencies": {
"@modelcontextprotocol/sdk": "^1.6.1",
"axios": "^1.7.9",
"zod": "^3.23.8"
},
"devDependencies": {
"@types/node": "^22.10.0",
"tsx": "^4.19.2",
"typescript": "^5.7.2"
}
}
```
### tsconfig.json
```json
{
"compilerOptions": {
"target": "ES2022",
"module": "Node16",
"moduleResolution": "Node16",
"lib": ["ES2022"],
"outDir": "./dist",
"rootDir": "./src",
"strict": true,
"esModuleInterop": true,
"skipLibCheck": true,
"forceConsistentCasingInFileNames": true,
"declaration": true,
"declarationMap": true,
"sourceMap": true,
"allowSyntheticDefaultImports": true
},
"include": ["src/**/*"],
"exclude": ["node_modules", "dist"]
}
```
## Complete Example
```typescript
#!/usr/bin/env node
/**
* MCP Server for Example Service.
*
* This server provides tools to interact with Example API, including user search,
* project management, and data export capabilities.
*/
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
import { z } from "zod";
import axios, { AxiosError } from "axios";
// Constants
const API_BASE_URL = "https://api.example.com/v1";
const CHARACTER_LIMIT = 25000;
// Enums
enum ResponseFormat {
MARKDOWN = "markdown",
JSON = "json"
}
// Zod schemas
const UserSearchInputSchema = z.object({
query: z.string()
.min(2, "Query must be at least 2 characters")
.max(200, "Query must not exceed 200 characters")
.describe("Search string to match against names/emails"),
limit: z.number()
.int()
.min(1)
.max(100)
.default(20)
.describe("Maximum results to return"),
offset: z.number()
.int()
.min(0)
.default(0)
.describe("Number of results to skip for pagination"),
response_format: z.nativeEnum(ResponseFormat)
.default(ResponseFormat.MARKDOWN)
.describe("Output format: 'markdown' for human-readable or 'json' for machine-readable")
}).strict();
type UserSearchInput = z.infer<typeof UserSearchInputSchema>;
// Shared utility functions
async function makeApiRequest<T>(
endpoint: string,
method: "GET" | "POST" | "PUT" | "DELETE" = "GET",
data?: any,
params?: any
): Promise<T> {
try {
const response = await axios({
method,
url: `${API_BASE_URL}/${endpoint}`,
data,
params,
timeout: 30000,
headers: {
"Content-Type": "application/json",
"Accept": "application/json"
}
});
return response.data;
} catch (error) {
throw error;
}
}
function handleApiError(error: unknown): string {
if (error instanceof AxiosError) {
if (error.response) {
switch (error.response.status) {
case 404:
return "Error: Resource not found. Please check the ID is correct.";
case 403:
return "Error: Permission denied. You don't have access to this resource.";
case 429:
return "Error: Rate limit exceeded. Please wait before making more requests.";
default:
return `Error: API request failed with status ${error.response.status}`;
}
} else if (error.code === "ECONNABORTED") {
return "Error: Request timed out. Please try again.";
}
}
return `Error: Unexpected error occurred: ${error instanceof Error ? error.message : String(error)}`;
}
// Create MCP server instance
const server = new McpServer({
name: "example-mcp",
version: "1.0.0"
});
// Register tools
server.registerTool(
"example_search_users",
{
title: "Search Example Users",
description: `[Full description as shown above]`,
inputSchema: UserSearchInputSchema,
annotations: {
readOnlyHint: true,
destructiveHint: false,
idempotentHint: true,
openWorldHint: true
}
},
async (params: UserSearchInput) => {
// Implementation as shown above
}
);
// Main function
// For stdio (local):
async function runStdio() {
if (!process.env.EXAMPLE_API_KEY) {
console.error("ERROR: EXAMPLE_API_KEY environment variable is required");
process.exit(1);
}
const transport = new StdioServerTransport();
await server.connect(transport);
console.error("MCP server running via stdio");
}
// For streamable HTTP (remote):
async function runHTTP() {
if (!process.env.EXAMPLE_API_KEY) {
console.error("ERROR: EXAMPLE_API_KEY environment variable is required");
process.exit(1);
}
const app = express();
app.use(express.json());
app.post('/mcp', async (req, res) => {
const transport = new StreamableHTTPServerTransport({
sessionIdGenerator: undefined,
enableJsonResponse: true
});
res.on('close', () => transport.close());
await server.connect(transport);
await transport.handleRequest(req, res, req.body);
});
const port = parseInt(process.env.PORT || '3000');
app.listen(port, () => {
console.error(`MCP server running on http://localhost:${port}/mcp`);
});
}
// Choose transport based on environment
const transport = process.env.TRANSPORT || 'stdio';
if (transport === 'http') {
runHTTP().catch(error => {
console.error("Server error:", error);
process.exit(1);
});
} else {
runStdio().catch(error => {
console.error("Server error:", error);
process.exit(1);
});
}
```
---
## Advanced MCP Features
### Resource Registration
Expose data as resources for efficient, URI-based access:
```typescript
import { ResourceTemplate } from "@modelcontextprotocol/sdk/types.js";
// Register a resource with URI template
server.registerResource(
{
uri: "file://documents/{name}",
name: "Document Resource",
description: "Access documents by name",
mimeType: "text/plain"
},
async (uri: string) => {
// Extract parameter from URI
const match = uri.match(/^file:\/\/documents\/(.+)$/);
if (!match) {
throw new Error("Invalid URI format");
}
const documentName = match[1];
const content = await loadDocument(documentName);
return {
contents: [{
uri,
mimeType: "text/plain",
text: content
}]
};
}
);
// List available resources dynamically
server.registerResourceList(async () => {
const documents = await getAvailableDocuments();
return {
resources: documents.map(doc => ({
uri: `file://documents/${doc.name}`,
name: doc.name,
mimeType: "text/plain",
description: doc.description
}))
};
});
```
**When to use Resources vs Tools:**
- **Resources**: For data access with simple URI-based parameters
- **Tools**: For complex operations requiring validation and business logic
- **Resources**: When data is relatively static or template-based
- **Tools**: When operations have side effects or complex workflows
### Transport Options
The TypeScript SDK supports two main transport mechanisms:
#### Streamable HTTP (Recommended for Remote Servers)
```typescript
import { StreamableHTTPServerTransport } from "@modelcontextprotocol/sdk/server/streamableHttp.js";
import express from "express";
const app = express();
app.use(express.json());
app.post('/mcp', async (req, res) => {
// Create new transport for each request (stateless, prevents request ID collisions)
const transport = new StreamableHTTPServerTransport({
sessionIdGenerator: undefined,
enableJsonResponse: true
});
res.on('close', () => transport.close());
await server.connect(transport);
await transport.handleRequest(req, res, req.body);
});
app.listen(3000);
```
#### stdio (For Local Integrations)
```typescript
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
const transport = new StdioServerTransport();
await server.connect(transport);
```
**Transport selection:**
- **Streamable HTTP**: Web services, remote access, multiple clients
- **stdio**: Command-line tools, local development, subprocess integration
### Notification Support
Notify clients when server state changes:
```typescript
// Notify when tools list changes
server.notification({
method: "notifications/tools/list_changed"
});
// Notify when resources change
server.notification({
method: "notifications/resources/list_changed"
});
```
Use notifications sparingly - only when server capabilities genuinely change.
---
## Code Best Practices
### Code Composability and Reusability
Your implementation MUST prioritize composability and code reuse:
1. **Extract Common Functionality**:
- Create reusable helper functions for operations used across multiple tools
- Build shared API clients for HTTP requests instead of duplicating code
- Centralize error handling logic in utility functions
- Extract business logic into dedicated functions that can be composed
- Extract shared markdown or JSON field selection & formatting functionality
2. **Avoid Duplication**:
- NEVER copy-paste similar code between tools
- If you find yourself writing similar logic twice, extract it into a function
- Common operations like pagination, filtering, field selection, and formatting should be shared
- Authentication/authorization logic should be centralized
## Building and Running
Always build your TypeScript code before running:
```bash
# Build the project
npm run build
# Run the server
npm start
# Development with auto-reload
npm run dev
```
Always ensure `npm run build` completes successfully before considering the implementation complete.
## Quality Checklist
Before finalizing your Node/TypeScript MCP server implementation, ensure:
### Strategic Design
- [ ] Tools enable complete workflows, not just API endpoint wrappers
- [ ] Tool names reflect natural task subdivisions
- [ ] Response formats optimize for agent context efficiency
- [ ] Human-readable identifiers used where appropriate
- [ ] Error messages guide agents toward correct usage
### Implementation Quality
- [ ] FOCUSED IMPLEMENTATION: Most important and valuable tools implemented
- [ ] All tools registered using `registerTool` with complete configuration
- [ ] All tools include `title`, `description`, `inputSchema`, and `annotations`
- [ ] Annotations correctly set (readOnlyHint, destructiveHint, idempotentHint, openWorldHint)
- [ ] All tools use Zod schemas for runtime input validation with `.strict()` enforcement
- [ ] All Zod schemas have proper constraints and descriptive error messages
- [ ] All tools have comprehensive descriptions with explicit input/output types
- [ ] Descriptions include return value examples and complete schema documentation
- [ ] Error messages are clear, actionable, and educational
### TypeScript Quality
- [ ] TypeScript interfaces are defined for all data structures
- [ ] Strict TypeScript is enabled in tsconfig.json
- [ ] No use of `any` type - use `unknown` or proper types instead
- [ ] All async functions have explicit Promise<T> return types
- [ ] Error handling uses proper type guards (e.g., `axios.isAxiosError`, `z.ZodError`)
### Advanced Features (where applicable)
- [ ] Resources registered for appropriate data endpoints
- [ ] Appropriate transport configured (stdio or streamable HTTP)
- [ ] Notifications implemented for dynamic server capabilities
- [ ] Type-safe with SDK interfaces
### Project Configuration
- [ ] Package.json includes all necessary dependencies
- [ ] Build script produces working JavaScript in dist/ directory
- [ ] Main entry point is properly configured as dist/index.js
- [ ] Server name follows format: `{service}-mcp-server`
- [ ] tsconfig.json properly configured with strict mode
### Code Quality
- [ ] Pagination is properly implemented where applicable
- [ ] Large responses check CHARACTER_LIMIT constant and truncate with clear messages
- [ ] Filtering options are provided for potentially large result sets
- [ ] All network operations handle timeouts and connection errors gracefully
- [ ] Common functionality is extracted into reusable functions
- [ ] Return types are consistent across similar operations
### Testing and Build
- [ ] `npm run build` completes successfully without errors
- [ ] dist/index.js created and executable
- [ ] Server runs: `node dist/index.js --help`
- [ ] All imports resolve correctly
- [ ] Sample tool calls work as expected
@@ -1,719 +0,0 @@
# Python MCP Server Implementation Guide
## Overview
This document provides Python-specific best practices and examples for implementing MCP servers using the MCP Python SDK. It covers server setup, tool registration patterns, input validation with Pydantic, error handling, and complete working examples.
---
## Quick Reference
### Key Imports
```python
from mcp.server.fastmcp import FastMCP
from pydantic import BaseModel, Field, field_validator, ConfigDict
from typing import Optional, List, Dict, Any
from enum import Enum
import httpx
```
### Server Initialization
```python
mcp = FastMCP("service_mcp")
```
### Tool Registration Pattern
```python
@mcp.tool(name="tool_name", annotations={...})
async def tool_function(params: InputModel) -> str:
# Implementation
pass
```
---
## MCP Python SDK and FastMCP
The official MCP Python SDK provides FastMCP, a high-level framework for building MCP servers. It provides:
- Automatic description and inputSchema generation from function signatures and docstrings
- Pydantic model integration for input validation
- Decorator-based tool registration with `@mcp.tool`
**For complete SDK documentation, use WebFetch to load:**
`https://raw.githubusercontent.com/modelcontextprotocol/python-sdk/main/README.md`
## Server Naming Convention
Python MCP servers must follow this naming pattern:
- **Format**: `{service}_mcp` (lowercase with underscores)
- **Examples**: `github_mcp`, `jira_mcp`, `stripe_mcp`
The name should be:
- General (not tied to specific features)
- Descriptive of the service/API being integrated
- Easy to infer from the task description
- Without version numbers or dates
## Tool Implementation
### Tool Naming
Use snake_case for tool names (e.g., "search_users", "create_project", "get_channel_info") with clear, action-oriented names.
**Avoid Naming Conflicts**: Include the service context to prevent overlaps:
- Use "slack_send_message" instead of just "send_message"
- Use "github_create_issue" instead of just "create_issue"
- Use "asana_list_tasks" instead of just "list_tasks"
### Tool Structure with FastMCP
Tools are defined using the `@mcp.tool` decorator with Pydantic models for input validation:
```python
from pydantic import BaseModel, Field, ConfigDict
from mcp.server.fastmcp import FastMCP
# Initialize the MCP server
mcp = FastMCP("example_mcp")
# Define Pydantic model for input validation
class ServiceToolInput(BaseModel):
'''Input model for service tool operation.'''
model_config = ConfigDict(
str_strip_whitespace=True, # Auto-strip whitespace from strings
validate_assignment=True, # Validate on assignment
extra='forbid' # Forbid extra fields
)
param1: str = Field(..., description="First parameter description (e.g., 'user123', 'project-abc')", min_length=1, max_length=100)
param2: Optional[int] = Field(default=None, description="Optional integer parameter with constraints", ge=0, le=1000)
tags: Optional[List[str]] = Field(default_factory=list, description="List of tags to apply", max_items=10)
@mcp.tool(
name="service_tool_name",
annotations={
"title": "Human-Readable Tool Title",
"readOnlyHint": True, # Tool does not modify environment
"destructiveHint": False, # Tool does not perform destructive operations
"idempotentHint": True, # Repeated calls have no additional effect
"openWorldHint": False # Tool does not interact with external entities
}
)
async def service_tool_name(params: ServiceToolInput) -> str:
'''Tool description automatically becomes the 'description' field.
This tool performs a specific operation on the service. It validates all inputs
using the ServiceToolInput Pydantic model before processing.
Args:
params (ServiceToolInput): Validated input parameters containing:
- param1 (str): First parameter description
- param2 (Optional[int]): Optional parameter with default
- tags (Optional[List[str]]): List of tags
Returns:
str: JSON-formatted response containing operation results
'''
# Implementation here
pass
```
## Pydantic v2 Key Features
- Use `model_config` instead of nested `Config` class
- Use `field_validator` instead of deprecated `validator`
- Use `model_dump()` instead of deprecated `dict()`
- Validators require `@classmethod` decorator
- Type hints are required for validator methods
```python
from pydantic import BaseModel, Field, field_validator, ConfigDict
class CreateUserInput(BaseModel):
model_config = ConfigDict(
str_strip_whitespace=True,
validate_assignment=True
)
name: str = Field(..., description="User's full name", min_length=1, max_length=100)
email: str = Field(..., description="User's email address", pattern=r'^[\w\.-]+@[\w\.-]+\.\w+$')
age: int = Field(..., description="User's age", ge=0, le=150)
@field_validator('email')
@classmethod
def validate_email(cls, v: str) -> str:
if not v.strip():
raise ValueError("Email cannot be empty")
return v.lower()
```
## Response Format Options
Support multiple output formats for flexibility:
```python
from enum import Enum
class ResponseFormat(str, Enum):
'''Output format for tool responses.'''
MARKDOWN = "markdown"
JSON = "json"
class UserSearchInput(BaseModel):
query: str = Field(..., description="Search query")
response_format: ResponseFormat = Field(
default=ResponseFormat.MARKDOWN,
description="Output format: 'markdown' for human-readable or 'json' for machine-readable"
)
```
**Markdown format**:
- Use headers, lists, and formatting for clarity
- Convert timestamps to human-readable format (e.g., "2024-01-15 10:30:00 UTC" instead of epoch)
- Show display names with IDs in parentheses (e.g., "@john.doe (U123456)")
- Omit verbose metadata (e.g., show only one profile image URL, not all sizes)
- Group related information logically
**JSON format**:
- Return complete, structured data suitable for programmatic processing
- Include all available fields and metadata
- Use consistent field names and types
## Pagination Implementation
For tools that list resources:
```python
class ListInput(BaseModel):
limit: Optional[int] = Field(default=20, description="Maximum results to return", ge=1, le=100)
offset: Optional[int] = Field(default=0, description="Number of results to skip for pagination", ge=0)
async def list_items(params: ListInput) -> str:
# Make API request with pagination
data = await api_request(limit=params.limit, offset=params.offset)
# Return pagination info
response = {
"total": data["total"],
"count": len(data["items"]),
"offset": params.offset,
"items": data["items"],
"has_more": data["total"] > params.offset + len(data["items"]),
"next_offset": params.offset + len(data["items"]) if data["total"] > params.offset + len(data["items"]) else None
}
return json.dumps(response, indent=2)
```
## Error Handling
Provide clear, actionable error messages:
```python
def _handle_api_error(e: Exception) -> str:
'''Consistent error formatting across all tools.'''
if isinstance(e, httpx.HTTPStatusError):
if e.response.status_code == 404:
return "Error: Resource not found. Please check the ID is correct."
elif e.response.status_code == 403:
return "Error: Permission denied. You don't have access to this resource."
elif e.response.status_code == 429:
return "Error: Rate limit exceeded. Please wait before making more requests."
return f"Error: API request failed with status {e.response.status_code}"
elif isinstance(e, httpx.TimeoutException):
return "Error: Request timed out. Please try again."
return f"Error: Unexpected error occurred: {type(e).__name__}"
```
## Shared Utilities
Extract common functionality into reusable functions:
```python
# Shared API request function
async def _make_api_request(endpoint: str, method: str = "GET", **kwargs) -> dict:
'''Reusable function for all API calls.'''
async with httpx.AsyncClient() as client:
response = await client.request(
method,
f"{API_BASE_URL}/{endpoint}",
timeout=30.0,
**kwargs
)
response.raise_for_status()
return response.json()
```
## Async/Await Best Practices
Always use async/await for network requests and I/O operations:
```python
# Good: Async network request
async def fetch_data(resource_id: str) -> dict:
async with httpx.AsyncClient() as client:
response = await client.get(f"{API_URL}/resource/{resource_id}")
response.raise_for_status()
return response.json()
# Bad: Synchronous request
def fetch_data(resource_id: str) -> dict:
response = requests.get(f"{API_URL}/resource/{resource_id}") # Blocks
return response.json()
```
## Type Hints
Use type hints throughout:
```python
from typing import Optional, List, Dict, Any
async def get_user(user_id: str) -> Dict[str, Any]:
data = await fetch_user(user_id)
return {"id": data["id"], "name": data["name"]}
```
## Tool Docstrings
Every tool must have comprehensive docstrings with explicit type information:
```python
async def search_users(params: UserSearchInput) -> str:
'''
Search for users in the Example system by name, email, or team.
This tool searches across all user profiles in the Example platform,
supporting partial matches and various search filters. It does NOT
create or modify users, only searches existing ones.
Args:
params (UserSearchInput): Validated input parameters containing:
- query (str): Search string to match against names/emails (e.g., "john", "@example.com", "team:marketing")
- limit (Optional[int]): Maximum results to return, between 1-100 (default: 20)
- offset (Optional[int]): Number of results to skip for pagination (default: 0)
Returns:
str: JSON-formatted string containing search results with the following schema:
Success response:
{
"total": int, # Total number of matches found
"count": int, # Number of results in this response
"offset": int, # Current pagination offset
"users": [
{
"id": str, # User ID (e.g., "U123456789")
"name": str, # Full name (e.g., "John Doe")
"email": str, # Email address (e.g., "john@example.com")
"team": str # Team name (e.g., "Marketing") - optional
}
]
}
Error response:
"Error: <error message>" or "No users found matching '<query>'"
Examples:
- Use when: "Find all marketing team members" -> params with query="team:marketing"
- Use when: "Search for John's account" -> params with query="john"
- Don't use when: You need to create a user (use example_create_user instead)
- Don't use when: You have a user ID and need full details (use example_get_user instead)
Error Handling:
- Input validation errors are handled by Pydantic model
- Returns "Error: Rate limit exceeded" if too many requests (429 status)
- Returns "Error: Invalid API authentication" if API key is invalid (401 status)
- Returns formatted list of results or "No users found matching 'query'"
'''
```
## Complete Example
See below for a complete Python MCP server example:
```python
#!/usr/bin/env python3
'''
MCP Server for Example Service.
This server provides tools to interact with Example API, including user search,
project management, and data export capabilities.
'''
from typing import Optional, List, Dict, Any
from enum import Enum
import httpx
from pydantic import BaseModel, Field, field_validator, ConfigDict
from mcp.server.fastmcp import FastMCP
# Initialize the MCP server
mcp = FastMCP("example_mcp")
# Constants
API_BASE_URL = "https://api.example.com/v1"
# Enums
class ResponseFormat(str, Enum):
'''Output format for tool responses.'''
MARKDOWN = "markdown"
JSON = "json"
# Pydantic Models for Input Validation
class UserSearchInput(BaseModel):
'''Input model for user search operations.'''
model_config = ConfigDict(
str_strip_whitespace=True,
validate_assignment=True
)
query: str = Field(..., description="Search string to match against names/emails", min_length=2, max_length=200)
limit: Optional[int] = Field(default=20, description="Maximum results to return", ge=1, le=100)
offset: Optional[int] = Field(default=0, description="Number of results to skip for pagination", ge=0)
response_format: ResponseFormat = Field(default=ResponseFormat.MARKDOWN, description="Output format")
@field_validator('query')
@classmethod
def validate_query(cls, v: str) -> str:
if not v.strip():
raise ValueError("Query cannot be empty or whitespace only")
return v.strip()
# Shared utility functions
async def _make_api_request(endpoint: str, method: str = "GET", **kwargs) -> dict:
'''Reusable function for all API calls.'''
async with httpx.AsyncClient() as client:
response = await client.request(
method,
f"{API_BASE_URL}/{endpoint}",
timeout=30.0,
**kwargs
)
response.raise_for_status()
return response.json()
def _handle_api_error(e: Exception) -> str:
'''Consistent error formatting across all tools.'''
if isinstance(e, httpx.HTTPStatusError):
if e.response.status_code == 404:
return "Error: Resource not found. Please check the ID is correct."
elif e.response.status_code == 403:
return "Error: Permission denied. You don't have access to this resource."
elif e.response.status_code == 429:
return "Error: Rate limit exceeded. Please wait before making more requests."
return f"Error: API request failed with status {e.response.status_code}"
elif isinstance(e, httpx.TimeoutException):
return "Error: Request timed out. Please try again."
return f"Error: Unexpected error occurred: {type(e).__name__}"
# Tool definitions
@mcp.tool(
name="example_search_users",
annotations={
"title": "Search Example Users",
"readOnlyHint": True,
"destructiveHint": False,
"idempotentHint": True,
"openWorldHint": True
}
)
async def example_search_users(params: UserSearchInput) -> str:
'''Search for users in the Example system by name, email, or team.
[Full docstring as shown above]
'''
try:
# Make API request using validated parameters
data = await _make_api_request(
"users/search",
params={
"q": params.query,
"limit": params.limit,
"offset": params.offset
}
)
users = data.get("users", [])
total = data.get("total", 0)
if not users:
return f"No users found matching '{params.query}'"
# Format response based on requested format
if params.response_format == ResponseFormat.MARKDOWN:
lines = [f"# User Search Results: '{params.query}'", ""]
lines.append(f"Found {total} users (showing {len(users)})")
lines.append("")
for user in users:
lines.append(f"## {user['name']} ({user['id']})")
lines.append(f"- **Email**: {user['email']}")
if user.get('team'):
lines.append(f"- **Team**: {user['team']}")
lines.append("")
return "\n".join(lines)
else:
# Machine-readable JSON format
import json
response = {
"total": total,
"count": len(users),
"offset": params.offset,
"users": users
}
return json.dumps(response, indent=2)
except Exception as e:
return _handle_api_error(e)
if __name__ == "__main__":
mcp.run()
```
---
## Advanced FastMCP Features
### Context Parameter Injection
FastMCP can automatically inject a `Context` parameter into tools for advanced capabilities like logging, progress reporting, resource reading, and user interaction:
```python
from mcp.server.fastmcp import FastMCP, Context
mcp = FastMCP("example_mcp")
@mcp.tool()
async def advanced_search(query: str, ctx: Context) -> str:
'''Advanced tool with context access for logging and progress.'''
# Report progress for long operations
await ctx.report_progress(0.25, "Starting search...")
# Log information for debugging
await ctx.log_info("Processing query", {"query": query, "timestamp": datetime.now()})
# Perform search
results = await search_api(query)
await ctx.report_progress(0.75, "Formatting results...")
# Access server configuration
server_name = ctx.fastmcp.name
return format_results(results)
@mcp.tool()
async def interactive_tool(resource_id: str, ctx: Context) -> str:
'''Tool that can request additional input from users.'''
# Request sensitive information when needed
api_key = await ctx.elicit(
prompt="Please provide your API key:",
input_type="password"
)
# Use the provided key
return await api_call(resource_id, api_key)
```
**Context capabilities:**
- `ctx.report_progress(progress, message)` - Report progress for long operations
- `ctx.log_info(message, data)` / `ctx.log_error()` / `ctx.log_debug()` - Logging
- `ctx.elicit(prompt, input_type)` - Request input from users
- `ctx.fastmcp.name` - Access server configuration
- `ctx.read_resource(uri)` - Read MCP resources
### Resource Registration
Expose data as resources for efficient, template-based access:
```python
@mcp.resource("file://documents/{name}")
async def get_document(name: str) -> str:
'''Expose documents as MCP resources.
Resources are useful for static or semi-static data that doesn't
require complex parameters. They use URI templates for flexible access.
'''
document_path = f"./docs/{name}"
with open(document_path, "r") as f:
return f.read()
@mcp.resource("config://settings/{key}")
async def get_setting(key: str, ctx: Context) -> str:
'''Expose configuration as resources with context.'''
settings = await load_settings()
return json.dumps(settings.get(key, {}))
```
**When to use Resources vs Tools:**
- **Resources**: For data access with simple parameters (URI templates)
- **Tools**: For complex operations with validation and business logic
### Structured Output Types
FastMCP supports multiple return types beyond strings:
```python
from typing import TypedDict
from dataclasses import dataclass
from pydantic import BaseModel
# TypedDict for structured returns
class UserData(TypedDict):
id: str
name: str
email: str
@mcp.tool()
async def get_user_typed(user_id: str) -> UserData:
'''Returns structured data - FastMCP handles serialization.'''
return {"id": user_id, "name": "John Doe", "email": "john@example.com"}
# Pydantic models for complex validation
class DetailedUser(BaseModel):
id: str
name: str
email: str
created_at: datetime
metadata: Dict[str, Any]
@mcp.tool()
async def get_user_detailed(user_id: str) -> DetailedUser:
'''Returns Pydantic model - automatically generates schema.'''
user = await fetch_user(user_id)
return DetailedUser(**user)
```
### Lifespan Management
Initialize resources that persist across requests:
```python
from contextlib import asynccontextmanager
@asynccontextmanager
async def app_lifespan():
'''Manage resources that live for the server's lifetime.'''
# Initialize connections, load config, etc.
db = await connect_to_database()
config = load_configuration()
# Make available to all tools
yield {"db": db, "config": config}
# Cleanup on shutdown
await db.close()
mcp = FastMCP("example_mcp", lifespan=app_lifespan)
@mcp.tool()
async def query_data(query: str, ctx: Context) -> str:
'''Access lifespan resources through context.'''
db = ctx.request_context.lifespan_state["db"]
results = await db.query(query)
return format_results(results)
```
### Transport Options
FastMCP supports two main transport mechanisms:
```python
# stdio transport (for local tools) - default
if __name__ == "__main__":
mcp.run()
# Streamable HTTP transport (for remote servers)
if __name__ == "__main__":
mcp.run(transport="streamable_http", port=8000)
```
**Transport selection:**
- **stdio**: Command-line tools, local integrations, subprocess execution
- **Streamable HTTP**: Web services, remote access, multiple clients
---
## Code Best Practices
### Code Composability and Reusability
Your implementation MUST prioritize composability and code reuse:
1. **Extract Common Functionality**:
- Create reusable helper functions for operations used across multiple tools
- Build shared API clients for HTTP requests instead of duplicating code
- Centralize error handling logic in utility functions
- Extract business logic into dedicated functions that can be composed
- Extract shared markdown or JSON field selection & formatting functionality
2. **Avoid Duplication**:
- NEVER copy-paste similar code between tools
- If you find yourself writing similar logic twice, extract it into a function
- Common operations like pagination, filtering, field selection, and formatting should be shared
- Authentication/authorization logic should be centralized
### Python-Specific Best Practices
1. **Use Type Hints**: Always include type annotations for function parameters and return values
2. **Pydantic Models**: Define clear Pydantic models for all input validation
3. **Avoid Manual Validation**: Let Pydantic handle input validation with constraints
4. **Proper Imports**: Group imports (standard library, third-party, local)
5. **Error Handling**: Use specific exception types (httpx.HTTPStatusError, not generic Exception)
6. **Async Context Managers**: Use `async with` for resources that need cleanup
7. **Constants**: Define module-level constants in UPPER_CASE
## Quality Checklist
Before finalizing your Python MCP server implementation, ensure:
### Strategic Design
- [ ] Tools enable complete workflows, not just API endpoint wrappers
- [ ] Tool names reflect natural task subdivisions
- [ ] Response formats optimize for agent context efficiency
- [ ] Human-readable identifiers used where appropriate
- [ ] Error messages guide agents toward correct usage
### Implementation Quality
- [ ] FOCUSED IMPLEMENTATION: Most important and valuable tools implemented
- [ ] All tools have descriptive names and documentation
- [ ] Return types are consistent across similar operations
- [ ] Error handling is implemented for all external calls
- [ ] Server name follows format: `{service}_mcp`
- [ ] All network operations use async/await
- [ ] Common functionality is extracted into reusable functions
- [ ] Error messages are clear, actionable, and educational
- [ ] Outputs are properly validated and formatted
### Tool Configuration
- [ ] All tools implement 'name' and 'annotations' in the decorator
- [ ] Annotations correctly set (readOnlyHint, destructiveHint, idempotentHint, openWorldHint)
- [ ] All tools use Pydantic BaseModel for input validation with Field() definitions
- [ ] All Pydantic Fields have explicit types and descriptions with constraints
- [ ] All tools have comprehensive docstrings with explicit input/output types
- [ ] Docstrings include complete schema structure for dict/JSON returns
- [ ] Pydantic models handle input validation (no manual validation needed)
### Advanced Features (where applicable)
- [ ] Context injection used for logging, progress, or elicitation
- [ ] Resources registered for appropriate data endpoints
- [ ] Lifespan management implemented for persistent connections
- [ ] Structured output types used (TypedDict, Pydantic models)
- [ ] Appropriate transport configured (stdio or streamable HTTP)
### Code Quality
- [ ] File includes proper imports including Pydantic imports
- [ ] Pagination is properly implemented where applicable
- [ ] Filtering options are provided for potentially large result sets
- [ ] All async functions are properly defined with `async def`
- [ ] HTTP client usage follows async patterns with proper context managers
- [ ] Type hints are used throughout the code
- [ ] Constants are defined at module level in UPPER_CASE
### Testing
- [ ] Server runs successfully: `python your_server.py --help`
- [ ] All imports resolve correctly
- [ ] Sample tool calls work as expected
- [ ] Error scenarios handled gracefully
-151
View File
@@ -1,151 +0,0 @@
"""Lightweight connection handling for MCP servers."""
from abc import ABC, abstractmethod
from contextlib import AsyncExitStack
from typing import Any
from mcp import ClientSession, StdioServerParameters
from mcp.client.sse import sse_client
from mcp.client.stdio import stdio_client
from mcp.client.streamable_http import streamablehttp_client
class MCPConnection(ABC):
"""Base class for MCP server connections."""
def __init__(self):
self.session = None
self._stack = None
@abstractmethod
def _create_context(self):
"""Create the connection context based on connection type."""
async def __aenter__(self):
"""Initialize MCP server connection."""
self._stack = AsyncExitStack()
await self._stack.__aenter__()
try:
ctx = self._create_context()
result = await self._stack.enter_async_context(ctx)
if len(result) == 2:
read, write = result
elif len(result) == 3:
read, write, _ = result
else:
raise ValueError(f"Unexpected context result: {result}")
session_ctx = ClientSession(read, write)
self.session = await self._stack.enter_async_context(session_ctx)
await self.session.initialize()
return self
except BaseException:
await self._stack.__aexit__(None, None, None)
raise
async def __aexit__(self, exc_type, exc_val, exc_tb):
"""Clean up MCP server connection resources."""
if self._stack:
await self._stack.__aexit__(exc_type, exc_val, exc_tb)
self.session = None
self._stack = None
async def list_tools(self) -> list[dict[str, Any]]:
"""Retrieve available tools from the MCP server."""
response = await self.session.list_tools()
return [
{
"name": tool.name,
"description": tool.description,
"input_schema": tool.inputSchema,
}
for tool in response.tools
]
async def call_tool(self, tool_name: str, arguments: dict[str, Any]) -> Any:
"""Call a tool on the MCP server with provided arguments."""
result = await self.session.call_tool(tool_name, arguments=arguments)
return result.content
class MCPConnectionStdio(MCPConnection):
"""MCP connection using standard input/output."""
def __init__(self, command: str, args: list[str] = None, env: dict[str, str] = None):
super().__init__()
self.command = command
self.args = args or []
self.env = env
def _create_context(self):
return stdio_client(
StdioServerParameters(command=self.command, args=self.args, env=self.env)
)
class MCPConnectionSSE(MCPConnection):
"""MCP connection using Server-Sent Events."""
def __init__(self, url: str, headers: dict[str, str] = None):
super().__init__()
self.url = url
self.headers = headers or {}
def _create_context(self):
return sse_client(url=self.url, headers=self.headers)
class MCPConnectionHTTP(MCPConnection):
"""MCP connection using Streamable HTTP."""
def __init__(self, url: str, headers: dict[str, str] = None):
super().__init__()
self.url = url
self.headers = headers or {}
def _create_context(self):
return streamablehttp_client(url=self.url, headers=self.headers)
def create_connection(
transport: str,
command: str = None,
args: list[str] = None,
env: dict[str, str] = None,
url: str = None,
headers: dict[str, str] = None,
) -> MCPConnection:
"""Factory function to create the appropriate MCP connection.
Args:
transport: Connection type ("stdio", "sse", or "http")
command: Command to run (stdio only)
args: Command arguments (stdio only)
env: Environment variables (stdio only)
url: Server URL (sse and http only)
headers: HTTP headers (sse and http only)
Returns:
MCPConnection instance
"""
transport = transport.lower()
if transport == "stdio":
if not command:
raise ValueError("Command is required for stdio transport")
return MCPConnectionStdio(command=command, args=args, env=env)
elif transport == "sse":
if not url:
raise ValueError("URL is required for sse transport")
return MCPConnectionSSE(url=url, headers=headers)
elif transport in ["http", "streamable_http", "streamable-http"]:
if not url:
raise ValueError("URL is required for http transport")
return MCPConnectionHTTP(url=url, headers=headers)
else:
raise ValueError(f"Unsupported transport type: {transport}. Use 'stdio', 'sse', or 'http'")
-373
View File
@@ -1,373 +0,0 @@
"""MCP Server Evaluation Harness
This script evaluates MCP servers by running test questions against them using Claude.
"""
import argparse
import asyncio
import json
import re
import sys
import time
import traceback
import xml.etree.ElementTree as ET
from pathlib import Path
from typing import Any
from anthropic import Anthropic
from connections import create_connection
EVALUATION_PROMPT = """You are an AI assistant with access to tools.
When given a task, you MUST:
1. Use the available tools to complete the task
2. Provide summary of each step in your approach, wrapped in <summary> tags
3. Provide feedback on the tools provided, wrapped in <feedback> tags
4. Provide your final response, wrapped in <response> tags
Summary Requirements:
- In your <summary> tags, you must explain:
- The steps you took to complete the task
- Which tools you used, in what order, and why
- The inputs you provided to each tool
- The outputs you received from each tool
- A summary for how you arrived at the response
Feedback Requirements:
- In your <feedback> tags, provide constructive feedback on the tools:
- Comment on tool names: Are they clear and descriptive?
- Comment on input parameters: Are they well-documented? Are required vs optional parameters clear?
- Comment on descriptions: Do they accurately describe what the tool does?
- Comment on any errors encountered during tool usage: Did the tool fail to execute? Did the tool return too many tokens?
- Identify specific areas for improvement and explain WHY they would help
- Be specific and actionable in your suggestions
Response Requirements:
- Your response should be concise and directly address what was asked
- Always wrap your final response in <response> tags
- If you cannot solve the task return <response>NOT_FOUND</response>
- For numeric responses, provide just the number
- For IDs, provide just the ID
- For names or text, provide the exact text requested
- Your response should go last"""
def parse_evaluation_file(file_path: Path) -> list[dict[str, Any]]:
"""Parse XML evaluation file with qa_pair elements."""
try:
tree = ET.parse(file_path)
root = tree.getroot()
evaluations = []
for qa_pair in root.findall(".//qa_pair"):
question_elem = qa_pair.find("question")
answer_elem = qa_pair.find("answer")
if question_elem is not None and answer_elem is not None:
evaluations.append({
"question": (question_elem.text or "").strip(),
"answer": (answer_elem.text or "").strip(),
})
return evaluations
except Exception as e:
print(f"Error parsing evaluation file {file_path}: {e}")
return []
def extract_xml_content(text: str, tag: str) -> str | None:
"""Extract content from XML tags."""
pattern = rf"<{tag}>(.*?)</{tag}>"
matches = re.findall(pattern, text, re.DOTALL)
return matches[-1].strip() if matches else None
async def agent_loop(
client: Anthropic,
model: str,
question: str,
tools: list[dict[str, Any]],
connection: Any,
) -> tuple[str, dict[str, Any]]:
"""Run the agent loop with MCP tools."""
messages = [{"role": "user", "content": question}]
response = await asyncio.to_thread(
client.messages.create,
model=model,
max_tokens=4096,
system=EVALUATION_PROMPT,
messages=messages,
tools=tools,
)
messages.append({"role": "assistant", "content": response.content})
tool_metrics = {}
while response.stop_reason == "tool_use":
tool_use = next(block for block in response.content if block.type == "tool_use")
tool_name = tool_use.name
tool_input = tool_use.input
tool_start_ts = time.time()
try:
tool_result = await connection.call_tool(tool_name, tool_input)
tool_response = json.dumps(tool_result) if isinstance(tool_result, (dict, list)) else str(tool_result)
except Exception as e:
tool_response = f"Error executing tool {tool_name}: {str(e)}\n"
tool_response += traceback.format_exc()
tool_duration = time.time() - tool_start_ts
if tool_name not in tool_metrics:
tool_metrics[tool_name] = {"count": 0, "durations": []}
tool_metrics[tool_name]["count"] += 1
tool_metrics[tool_name]["durations"].append(tool_duration)
messages.append({
"role": "user",
"content": [{
"type": "tool_result",
"tool_use_id": tool_use.id,
"content": tool_response,
}]
})
response = await asyncio.to_thread(
client.messages.create,
model=model,
max_tokens=4096,
system=EVALUATION_PROMPT,
messages=messages,
tools=tools,
)
messages.append({"role": "assistant", "content": response.content})
response_text = next(
(block.text for block in response.content if hasattr(block, "text")),
None,
)
return response_text, tool_metrics
async def evaluate_single_task(
client: Anthropic,
model: str,
qa_pair: dict[str, Any],
tools: list[dict[str, Any]],
connection: Any,
task_index: int,
) -> dict[str, Any]:
"""Evaluate a single QA pair with the given tools."""
start_time = time.time()
print(f"Task {task_index + 1}: Running task with question: {qa_pair['question']}")
response, tool_metrics = await agent_loop(client, model, qa_pair["question"], tools, connection)
response_value = extract_xml_content(response, "response")
summary = extract_xml_content(response, "summary")
feedback = extract_xml_content(response, "feedback")
duration_seconds = time.time() - start_time
return {
"question": qa_pair["question"],
"expected": qa_pair["answer"],
"actual": response_value,
"score": int(response_value == qa_pair["answer"]) if response_value else 0,
"total_duration": duration_seconds,
"tool_calls": tool_metrics,
"num_tool_calls": sum(len(metrics["durations"]) for metrics in tool_metrics.values()),
"summary": summary,
"feedback": feedback,
}
REPORT_HEADER = """
# Evaluation Report
## Summary
- **Accuracy**: {correct}/{total} ({accuracy:.1f}%)
- **Average Task Duration**: {average_duration_s:.2f}s
- **Average Tool Calls per Task**: {average_tool_calls:.2f}
- **Total Tool Calls**: {total_tool_calls}
---
"""
TASK_TEMPLATE = """
### Task {task_num}
**Question**: {question}
**Ground Truth Answer**: `{expected_answer}`
**Actual Answer**: `{actual_answer}`
**Correct**: {correct_indicator}
**Duration**: {total_duration:.2f}s
**Tool Calls**: {tool_calls}
**Summary**
{summary}
**Feedback**
{feedback}
---
"""
async def run_evaluation(
eval_path: Path,
connection: Any,
model: str = "claude-3-7-sonnet-20250219",
) -> str:
"""Run evaluation with MCP server tools."""
print("🚀 Starting Evaluation")
client = Anthropic()
tools = await connection.list_tools()
print(f"📋 Loaded {len(tools)} tools from MCP server")
qa_pairs = parse_evaluation_file(eval_path)
print(f"📋 Loaded {len(qa_pairs)} evaluation tasks")
results = []
for i, qa_pair in enumerate(qa_pairs):
print(f"Processing task {i + 1}/{len(qa_pairs)}")
result = await evaluate_single_task(client, model, qa_pair, tools, connection, i)
results.append(result)
correct = sum(r["score"] for r in results)
accuracy = (correct / len(results)) * 100 if results else 0
average_duration_s = sum(r["total_duration"] for r in results) / len(results) if results else 0
average_tool_calls = sum(r["num_tool_calls"] for r in results) / len(results) if results else 0
total_tool_calls = sum(r["num_tool_calls"] for r in results)
report = REPORT_HEADER.format(
correct=correct,
total=len(results),
accuracy=accuracy,
average_duration_s=average_duration_s,
average_tool_calls=average_tool_calls,
total_tool_calls=total_tool_calls,
)
report += "".join([
TASK_TEMPLATE.format(
task_num=i + 1,
question=qa_pair["question"],
expected_answer=qa_pair["answer"],
actual_answer=result["actual"] or "N/A",
correct_indicator="" if result["score"] else "",
total_duration=result["total_duration"],
tool_calls=json.dumps(result["tool_calls"], indent=2),
summary=result["summary"] or "N/A",
feedback=result["feedback"] or "N/A",
)
for i, (qa_pair, result) in enumerate(zip(qa_pairs, results))
])
return report
def parse_headers(header_list: list[str]) -> dict[str, str]:
"""Parse header strings in format 'Key: Value' into a dictionary."""
headers = {}
if not header_list:
return headers
for header in header_list:
if ":" in header:
key, value = header.split(":", 1)
headers[key.strip()] = value.strip()
else:
print(f"Warning: Ignoring malformed header: {header}")
return headers
def parse_env_vars(env_list: list[str]) -> dict[str, str]:
"""Parse environment variable strings in format 'KEY=VALUE' into a dictionary."""
env = {}
if not env_list:
return env
for env_var in env_list:
if "=" in env_var:
key, value = env_var.split("=", 1)
env[key.strip()] = value.strip()
else:
print(f"Warning: Ignoring malformed environment variable: {env_var}")
return env
async def main():
parser = argparse.ArgumentParser(
description="Evaluate MCP servers using test questions",
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog="""
Examples:
# Evaluate a local stdio MCP server
python evaluation.py -t stdio -c python -a my_server.py eval.xml
# Evaluate an SSE MCP server
python evaluation.py -t sse -u https://example.com/mcp -H "Authorization: Bearer token" eval.xml
# Evaluate an HTTP MCP server with custom model
python evaluation.py -t http -u https://example.com/mcp -m claude-3-5-sonnet-20241022 eval.xml
""",
)
parser.add_argument("eval_file", type=Path, help="Path to evaluation XML file")
parser.add_argument("-t", "--transport", choices=["stdio", "sse", "http"], default="stdio", help="Transport type (default: stdio)")
parser.add_argument("-m", "--model", default="claude-3-7-sonnet-20250219", help="Claude model to use (default: claude-3-7-sonnet-20250219)")
stdio_group = parser.add_argument_group("stdio options")
stdio_group.add_argument("-c", "--command", help="Command to run MCP server (stdio only)")
stdio_group.add_argument("-a", "--args", nargs="+", help="Arguments for the command (stdio only)")
stdio_group.add_argument("-e", "--env", nargs="+", help="Environment variables in KEY=VALUE format (stdio only)")
remote_group = parser.add_argument_group("sse/http options")
remote_group.add_argument("-u", "--url", help="MCP server URL (sse/http only)")
remote_group.add_argument("-H", "--header", nargs="+", dest="headers", help="HTTP headers in 'Key: Value' format (sse/http only)")
parser.add_argument("-o", "--output", type=Path, help="Output file for evaluation report (default: stdout)")
args = parser.parse_args()
if not args.eval_file.exists():
print(f"Error: Evaluation file not found: {args.eval_file}")
sys.exit(1)
headers = parse_headers(args.headers) if args.headers else None
env_vars = parse_env_vars(args.env) if args.env else None
try:
connection = create_connection(
transport=args.transport,
command=args.command,
args=args.args,
env=env_vars,
url=args.url,
headers=headers,
)
except ValueError as e:
print(f"Error: {e}")
sys.exit(1)
print(f"🔗 Connecting to MCP server via {args.transport}...")
async with connection:
print("✅ Connected successfully")
report = await run_evaluation(args.eval_file, connection, args.model)
if args.output:
args.output.write_text(report)
print(f"\n✅ Report saved to {args.output}")
else:
print("\n" + report)
if __name__ == "__main__":
asyncio.run(main())
@@ -1,22 +0,0 @@
<evaluation>
<qa_pair>
<question>Calculate the compound interest on $10,000 invested at 5% annual interest rate, compounded monthly for 3 years. What is the final amount in dollars (rounded to 2 decimal places)?</question>
<answer>11614.72</answer>
</qa_pair>
<qa_pair>
<question>A projectile is launched at a 45-degree angle with an initial velocity of 50 m/s. Calculate the total distance (in meters) it has traveled from the launch point after 2 seconds, assuming g=9.8 m/s². Round to 2 decimal places.</question>
<answer>87.25</answer>
</qa_pair>
<qa_pair>
<question>A sphere has a volume of 500 cubic meters. Calculate its surface area in square meters. Round to 2 decimal places.</question>
<answer>304.65</answer>
</qa_pair>
<qa_pair>
<question>Calculate the population standard deviation of this dataset: [12, 15, 18, 22, 25, 30, 35]. Round to 2 decimal places.</question>
<answer>7.61</answer>
</qa_pair>
<qa_pair>
<question>Calculate the pH of a solution with a hydrogen ion concentration of 3.5 × 10^-5 M. Round to 2 decimal places.</question>
<answer>4.46</answer>
</qa_pair>
</evaluation>
@@ -1,2 +0,0 @@
anthropic>=0.39.0
mcp>=1.1.0
-202
View File
@@ -1,202 +0,0 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright 2026 Anthropic, PBC.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-485
View File
@@ -1,485 +0,0 @@
---
name: skill-creator
description: Create new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.
---
# Skill Creator
A skill for creating new skills and iteratively improving them.
At a high level, the process of creating a skill goes like this:
- Decide what you want the skill to do and roughly how it should do it
- Write a draft of the skill
- Create a few test prompts and run claude-with-access-to-the-skill on them
- Help the user evaluate the results both qualitatively and quantitatively
- While the runs happen in the background, draft some quantitative evals if there aren't any (if there are some, you can either use as is or modify if you feel something needs to change about them). Then explain them to the user (or if they already existed, explain the ones that already exist)
- Use the `eval-viewer/generate_review.py` script to show the user the results for them to look at, and also let them look at the quantitative metrics
- Rewrite the skill based on feedback from the user's evaluation of the results (and also if there are any glaring flaws that become apparent from the quantitative benchmarks)
- Repeat until you're satisfied
- Expand the test set and try again at larger scale
Your job when using this skill is to figure out where the user is in this process and then jump in and help them progress through these stages. So for instance, maybe they're like "I want to make a skill for X". You can help narrow down what they mean, write a draft, write the test cases, figure out how they want to evaluate, run all the prompts, and repeat.
On the other hand, maybe they already have a draft of the skill. In this case you can go straight to the eval/iterate part of the loop.
Of course, you should always be flexible and if the user is like "I don't need to run a bunch of evaluations, just vibe with me", you can do that instead.
Then after the skill is done (but again, the order is flexible), you can also run the skill description improver, which we have a whole separate script for, to optimize the triggering of the skill.
Cool? Cool.
## Communicating with the user
The skill creator is liable to be used by people across a wide range of familiarity with coding jargon. If you haven't heard (and how could you, it's only very recently that it started), there's a trend now where the power of Claude is inspiring plumbers to open up their terminals, parents and grandparents to google "how to install npm". On the other hand, the bulk of users are probably fairly computer-literate.
So please pay attention to context cues to understand how to phrase your communication! In the default case, just to give you some idea:
- "evaluation" and "benchmark" are borderline, but OK
- for "JSON" and "assertion" you want to see serious cues from the user that they know what those things are before using them without explaining them
It's OK to briefly explain terms if you're in doubt, and feel free to clarify terms with a short definition if you're unsure if the user will get it.
---
## Creating a skill
### Capture Intent
Start by understanding the user's intent. The current conversation might already contain a workflow the user wants to capture (e.g., they say "turn this into a skill"). If so, extract answers from the conversation history first — the tools used, the sequence of steps, corrections the user made, input/output formats observed. The user may need to fill the gaps, and should confirm before proceeding to the next step.
1. What should this skill enable Claude to do?
2. When should this skill trigger? (what user phrases/contexts)
3. What's the expected output format?
4. Should we set up test cases to verify the skill works? Skills with objectively verifiable outputs (file transforms, data extraction, code generation, fixed workflow steps) benefit from test cases. Skills with subjective outputs (writing style, art) often don't need them. Suggest the appropriate default based on the skill type, but let the user decide.
### Interview and Research
Proactively ask questions about edge cases, input/output formats, example files, success criteria, and dependencies. Wait to write test prompts until you've got this part ironed out.
Check available MCPs - if useful for research (searching docs, finding similar skills, looking up best practices), research in parallel via subagents if available, otherwise inline. Come prepared with context to reduce burden on the user.
### Write the SKILL.md
Based on the user interview, fill in these components:
- **name**: Skill identifier
- **description**: When to trigger, what it does. This is the primary triggering mechanism - include both what the skill does AND specific contexts for when to use it. All "when to use" info goes here, not in the body. Note: currently Claude has a tendency to "undertrigger" skills -- to not use them when they'd be useful. To combat this, please make the skill descriptions a little bit "pushy". So for instance, instead of "How to build a simple fast dashboard to display internal Anthropic data.", you might write "How to build a simple fast dashboard to display internal Anthropic data. Make sure to use this skill whenever the user mentions dashboards, data visualization, internal metrics, or wants to display any kind of company data, even if they don't explicitly ask for a 'dashboard.'"
- **compatibility**: Required tools, dependencies (optional, rarely needed)
- **the rest of the skill :)**
### Skill Writing Guide
#### Anatomy of a Skill
```
skill-name/
├── SKILL.md (required)
│ ├── YAML frontmatter (name, description required)
│ └── Markdown instructions
└── Bundled Resources (optional)
├── scripts/ - Executable code for deterministic/repetitive tasks
├── references/ - Docs loaded into context as needed
└── assets/ - Files used in output (templates, icons, fonts)
```
#### Progressive Disclosure
Skills use a three-level loading system:
1. **Metadata** (name + description) - Always in context (~100 words)
2. **SKILL.md body** - In context whenever skill triggers (<500 lines ideal)
3. **Bundled resources** - As needed (unlimited, scripts can execute without loading)
These word counts are approximate and you can feel free to go longer if needed.
**Key patterns:**
- Keep SKILL.md under 500 lines; if you're approaching this limit, add an additional layer of hierarchy along with clear pointers about where the model using the skill should go next to follow up.
- Reference files clearly from SKILL.md with guidance on when to read them
- For large reference files (>300 lines), include a table of contents
**Domain organization**: When a skill supports multiple domains/frameworks, organize by variant:
```
cloud-deploy/
├── SKILL.md (workflow + selection)
└── references/
├── aws.md
├── gcp.md
└── azure.md
```
Claude reads only the relevant reference file.
#### Principle of Lack of Surprise
This goes without saying, but skills must not contain malware, exploit code, or any content that could compromise system security. A skill's contents should not surprise the user in their intent if described. Don't go along with requests to create misleading skills or skills designed to facilitate unauthorized access, data exfiltration, or other malicious activities. Things like a "roleplay as an XYZ" are OK though.
#### Writing Patterns
Prefer using the imperative form in instructions.
**Defining output formats** - You can do it like this:
```markdown
## Report structure
ALWAYS use this exact template:
# [Title]
## Executive summary
## Key findings
## Recommendations
```
**Examples pattern** - It's useful to include examples. You can format them like this (but if "Input" and "Output" are in the examples you might want to deviate a little):
```markdown
## Commit message format
**Example 1:**
Input: Added user authentication with JWT tokens
Output: feat(auth): implement JWT-based authentication
```
### Writing Style
Try to explain to the model why things are important in lieu of heavy-handed musty MUSTs. Use theory of mind and try to make the skill general and not super-narrow to specific examples. Start by writing a draft and then look at it with fresh eyes and improve it.
### Test Cases
After writing the skill draft, come up with 2-3 realistic test prompts — the kind of thing a real user would actually say. Share them with the user: [you don't have to use this exact language] "Here are a few test cases I'd like to try. Do these look right, or do you want to add more?" Then run them.
Save test cases to `evals/evals.json`. Don't write assertions yet — just the prompts. You'll draft assertions in the next step while the runs are in progress.
```json
{
"skill_name": "example-skill",
"evals": [
{
"id": 1,
"prompt": "User's task prompt",
"expected_output": "Description of expected result",
"files": []
}
]
}
```
See `references/schemas.md` for the full schema (including the `assertions` field, which you'll add later).
## Running and evaluating test cases
This section is one continuous sequence — don't stop partway through. Do NOT use `/skill-test` or any other testing skill.
Put results in `<skill-name>-workspace/` as a sibling to the skill directory. Within the workspace, organize results by iteration (`iteration-1/`, `iteration-2/`, etc.) and within that, each test case gets a directory (`eval-0/`, `eval-1/`, etc.). Don't create all of this upfront — just create directories as you go.
### Step 1: Spawn all runs (with-skill AND baseline) in the same turn
For each test case, spawn two subagents in the same turn — one with the skill, one without. This is important: don't spawn the with-skill runs first and then come back for baselines later. Launch everything at once so it all finishes around the same time.
**With-skill run:**
```
Execute this task:
- Skill path: <path-to-skill>
- Task: <eval prompt>
- Input files: <eval files if any, or "none">
- Save outputs to: <workspace>/iteration-<N>/eval-<ID>/with_skill/outputs/
- Outputs to save: <what the user cares about — e.g., "the .docx file", "the final CSV">
```
**Baseline run** (same prompt, but the baseline depends on context):
- **Creating a new skill**: no skill at all. Same prompt, no skill path, save to `without_skill/outputs/`.
- **Improving an existing skill**: the old version. Before editing, snapshot the skill (`cp -r <skill-path> <workspace>/skill-snapshot/`), then point the baseline subagent at the snapshot. Save to `old_skill/outputs/`.
Write an `eval_metadata.json` for each test case (assertions can be empty for now). Give each eval a descriptive name based on what it's testing — not just "eval-0". Use this name for the directory too. If this iteration uses new or modified eval prompts, create these files for each new eval directory — don't assume they carry over from previous iterations.
```json
{
"eval_id": 0,
"eval_name": "descriptive-name-here",
"prompt": "The user's task prompt",
"assertions": []
}
```
### Step 2: While runs are in progress, draft assertions
Don't just wait for the runs to finish — you can use this time productively. Draft quantitative assertions for each test case and explain them to the user. If assertions already exist in `evals/evals.json`, review them and explain what they check.
Good assertions are objectively verifiable and have descriptive names — they should read clearly in the benchmark viewer so someone glancing at the results immediately understands what each one checks. Subjective skills (writing style, design quality) are better evaluated qualitatively — don't force assertions onto things that need human judgment.
Update the `eval_metadata.json` files and `evals/evals.json` with the assertions once drafted. Also explain to the user what they'll see in the viewer — both the qualitative outputs and the quantitative benchmark.
### Step 3: As runs complete, capture timing data
When each subagent task completes, you receive a notification containing `total_tokens` and `duration_ms`. Save this data immediately to `timing.json` in the run directory:
```json
{
"total_tokens": 84852,
"duration_ms": 23332,
"total_duration_seconds": 23.3
}
```
This is the only opportunity to capture this data — it comes through the task notification and isn't persisted elsewhere. Process each notification as it arrives rather than trying to batch them.
### Step 4: Grade, aggregate, and launch the viewer
Once all runs are done:
1. **Grade each run** — spawn a grader subagent (or grade inline) that reads `agents/grader.md` and evaluates each assertion against the outputs. Save results to `grading.json` in each run directory. The grading.json expectations array must use the fields `text`, `passed`, and `evidence` (not `name`/`met`/`details` or other variants) — the viewer depends on these exact field names. For assertions that can be checked programmatically, write and run a script rather than eyeballing it — scripts are faster, more reliable, and can be reused across iterations.
2. **Aggregate into benchmark** — run the aggregation script from the skill-creator directory:
```bash
python -m scripts.aggregate_benchmark <workspace>/iteration-N --skill-name <name>
```
This produces `benchmark.json` and `benchmark.md` with pass_rate, time, and tokens for each configuration, with mean ± stddev and the delta. If generating benchmark.json manually, see `references/schemas.md` for the exact schema the viewer expects.
Put each with_skill version before its baseline counterpart.
3. **Do an analyst pass** — read the benchmark data and surface patterns the aggregate stats might hide. See `agents/analyzer.md` (the "Analyzing Benchmark Results" section) for what to look for — things like assertions that always pass regardless of skill (non-discriminating), high-variance evals (possibly flaky), and time/token tradeoffs.
4. **Launch the viewer** with both qualitative outputs and quantitative data:
```bash
nohup python <skill-creator-path>/eval-viewer/generate_review.py \
<workspace>/iteration-N \
--skill-name "my-skill" \
--benchmark <workspace>/iteration-N/benchmark.json \
> /dev/null 2>&1 &
VIEWER_PID=$!
```
For iteration 2+, also pass `--previous-workspace <workspace>/iteration-<N-1>`.
**Cowork / headless environments:** If `webbrowser.open()` is not available or the environment has no display, use `--static <output_path>` to write a standalone HTML file instead of starting a server. Feedback will be downloaded as a `feedback.json` file when the user clicks "Submit All Reviews". After download, copy `feedback.json` into the workspace directory for the next iteration to pick up.
Note: please use generate_review.py to create the viewer; there's no need to write custom HTML.
5. **Tell the user** something like: "I've opened the results in your browser. There are two tabs — 'Outputs' lets you click through each test case and leave feedback, 'Benchmark' shows the quantitative comparison. When you're done, come back here and let me know."
### What the user sees in the viewer
The "Outputs" tab shows one test case at a time:
- **Prompt**: the task that was given
- **Output**: the files the skill produced, rendered inline where possible
- **Previous Output** (iteration 2+): collapsed section showing last iteration's output
- **Formal Grades** (if grading was run): collapsed section showing assertion pass/fail
- **Feedback**: a textbox that auto-saves as they type
- **Previous Feedback** (iteration 2+): their comments from last time, shown below the textbox
The "Benchmark" tab shows the stats summary: pass rates, timing, and token usage for each configuration, with per-eval breakdowns and analyst observations.
Navigation is via prev/next buttons or arrow keys. When done, they click "Submit All Reviews" which saves all feedback to `feedback.json`.
### Step 5: Read the feedback
When the user tells you they're done, read `feedback.json`:
```json
{
"reviews": [
{"run_id": "eval-0-with_skill", "feedback": "the chart is missing axis labels", "timestamp": "..."},
{"run_id": "eval-1-with_skill", "feedback": "", "timestamp": "..."},
{"run_id": "eval-2-with_skill", "feedback": "perfect, love this", "timestamp": "..."}
],
"status": "complete"
}
```
Empty feedback means the user thought it was fine. Focus your improvements on the test cases where the user had specific complaints.
Kill the viewer server when you're done with it:
```bash
kill $VIEWER_PID 2>/dev/null
```
---
## Improving the skill
This is the heart of the loop. You've run the test cases, the user has reviewed the results, and now you need to make the skill better based on their feedback.
### How to think about improvements
1. **Generalize from the feedback.** The big picture thing that's happening here is that we're trying to create skills that can be used a million times (maybe literally, maybe even more who knows) across many different prompts. Here you and the user are iterating on only a few examples over and over again because it helps move faster. The user knows these examples in and out and it's quick for them to assess new outputs. But if the skill you and the user are codeveloping works only for those examples, it's useless. Rather than put in fiddly overfitty changes, or oppressively constrictive MUSTs, if there's some stubborn issue, you might try branching out and using different metaphors, or recommending different patterns of working. It's relatively cheap to try and maybe you'll land on something great.
2. **Keep the prompt lean.** Remove things that aren't pulling their weight. Make sure to read the transcripts, not just the final outputs — if it looks like the skill is making the model waste a bunch of time doing things that are unproductive, you can try getting rid of the parts of the skill that are making it do that and seeing what happens.
3. **Explain the why.** Try hard to explain the **why** behind everything you're asking the model to do. Today's LLMs are *smart*. They have good theory of mind and when given a good harness can go beyond rote instructions and really make things happen. Even if the feedback from the user is terse or frustrated, try to actually understand the task and why the user is writing what they wrote, and what they actually wrote, and then transmit this understanding into the instructions. If you find yourself writing ALWAYS or NEVER in all caps, or using super rigid structures, that's a yellow flag — if possible, reframe and explain the reasoning so that the model understands why the thing you're asking for is important. That's a more humane, powerful, and effective approach.
4. **Look for repeated work across test cases.** Read the transcripts from the test runs and notice if the subagents all independently wrote similar helper scripts or took the same multi-step approach to something. If all 3 test cases resulted in the subagent writing a `create_docx.py` or a `build_chart.py`, that's a strong signal the skill should bundle that script. Write it once, put it in `scripts/`, and tell the skill to use it. This saves every future invocation from reinventing the wheel.
This task is pretty important (we are trying to create billions a year in economic value here!) and your thinking time is not the blocker; take your time and really mull things over. I'd suggest writing a draft revision and then looking at it anew and making improvements. Really do your best to get into the head of the user and understand what they want and need.
### The iteration loop
After improving the skill:
1. Apply your improvements to the skill
2. Rerun all test cases into a new `iteration-<N+1>/` directory, including baseline runs. If you're creating a new skill, the baseline is always `without_skill` (no skill) — that stays the same across iterations. If you're improving an existing skill, use your judgment on what makes sense as the baseline: the original version the user came in with, or the previous iteration.
3. Launch the reviewer with `--previous-workspace` pointing at the previous iteration
4. Wait for the user to review and tell you they're done
5. Read the new feedback, improve again, repeat
Keep going until:
- The user says they're happy
- The feedback is all empty (everything looks good)
- You're not making meaningful progress
---
## Advanced: Blind comparison
For situations where you want a more rigorous comparison between two versions of a skill (e.g., the user asks "is the new version actually better?"), there's a blind comparison system. Read `agents/comparator.md` and `agents/analyzer.md` for the details. The basic idea is: give two outputs to an independent agent without telling it which is which, and let it judge quality. Then analyze why the winner won.
This is optional, requires subagents, and most users won't need it. The human review loop is usually sufficient.
---
## Description Optimization
The description field in SKILL.md frontmatter is the primary mechanism that determines whether Claude invokes a skill. After creating or improving a skill, offer to optimize the description for better triggering accuracy.
### Step 1: Generate trigger eval queries
Create 20 eval queries — a mix of should-trigger and should-not-trigger. Save as JSON:
```json
[
{"query": "the user prompt", "should_trigger": true},
{"query": "another prompt", "should_trigger": false}
]
```
The queries must be realistic and something a Claude Code or Claude.ai user would actually type. Not abstract requests, but requests that are concrete and specific and have a good amount of detail. For instance, file paths, personal context about the user's job or situation, column names and values, company names, URLs. A little bit of backstory. Some might be in lowercase or contain abbreviations or typos or casual speech. Use a mix of different lengths, and focus on edge cases rather than making them clear-cut (the user will get a chance to sign off on them).
Bad: `"Format this data"`, `"Extract text from PDF"`, `"Create a chart"`
Good: `"ok so my boss just sent me this xlsx file (its in my downloads, called something like 'Q4 sales final FINAL v2.xlsx') and she wants me to add a column that shows the profit margin as a percentage. The revenue is in column C and costs are in column D i think"`
For the **should-trigger** queries (8-10), think about coverage. You want different phrasings of the same intent — some formal, some casual. Include cases where the user doesn't explicitly name the skill or file type but clearly needs it. Throw in some uncommon use cases and cases where this skill competes with another but should win.
For the **should-not-trigger** queries (8-10), the most valuable ones are the near-misses — queries that share keywords or concepts with the skill but actually need something different. Think adjacent domains, ambiguous phrasing where a naive keyword match would trigger but shouldn't, and cases where the query touches on something the skill does but in a context where another tool is more appropriate.
The key thing to avoid: don't make should-not-trigger queries obviously irrelevant. "Write a fibonacci function" as a negative test for a PDF skill is too easy — it doesn't test anything. The negative cases should be genuinely tricky.
### Step 2: Review with user
Present the eval set to the user for review using the HTML template:
1. Read the template from `assets/eval_review.html`
2. Replace the placeholders:
- `__EVAL_DATA_PLACEHOLDER__` → the JSON array of eval items (no quotes around it — it's a JS variable assignment)
- `__SKILL_NAME_PLACEHOLDER__` → the skill's name
- `__SKILL_DESCRIPTION_PLACEHOLDER__` → the skill's current description
3. Write to a temp file (e.g., `/tmp/eval_review_<skill-name>.html`) and open it: `open /tmp/eval_review_<skill-name>.html`
4. The user can edit queries, toggle should-trigger, add/remove entries, then click "Export Eval Set"
5. The file downloads to `~/Downloads/eval_set.json` — check the Downloads folder for the most recent version in case there are multiple (e.g., `eval_set (1).json`)
This step matters — bad eval queries lead to bad descriptions.
### Step 3: Run the optimization loop
Tell the user: "This will take some time — I'll run the optimization loop in the background and check on it periodically."
Save the eval set to the workspace, then run in the background:
```bash
python -m scripts.run_loop \
--eval-set <path-to-trigger-eval.json> \
--skill-path <path-to-skill> \
--model <model-id-powering-this-session> \
--max-iterations 5 \
--verbose
```
Use the model ID from your system prompt (the one powering the current session) so the triggering test matches what the user actually experiences.
While it runs, periodically tail the output to give the user updates on which iteration it's on and what the scores look like.
This handles the full optimization loop automatically. It splits the eval set into 60% train and 40% held-out test, evaluates the current description (running each query 3 times to get a reliable trigger rate), then calls Claude to propose improvements based on what failed. It re-evaluates each new description on both train and test, iterating up to 5 times. When it's done, it opens an HTML report in the browser showing the results per iteration and returns JSON with `best_description` — selected by test score rather than train score to avoid overfitting.
### How skill triggering works
Understanding the triggering mechanism helps design better eval queries. Skills appear in Claude's `available_skills` list with their name + description, and Claude decides whether to consult a skill based on that description. The important thing to know is that Claude only consults skills for tasks it can't easily handle on its own — simple, one-step queries like "read this PDF" may not trigger a skill even if the description matches perfectly, because Claude can handle them directly with basic tools. Complex, multi-step, or specialized queries reliably trigger skills when the description matches.
This means your eval queries should be substantive enough that Claude would actually benefit from consulting a skill. Simple queries like "read file X" are poor test cases — they won't trigger skills regardless of description quality.
### Step 4: Apply the result
Take `best_description` from the JSON output and update the skill's SKILL.md frontmatter. Show the user before/after and report the scores.
---
### Package and Present (only if `present_files` tool is available)
Check whether you have access to the `present_files` tool. If you don't, skip this step. If you do, package the skill and present the .skill file to the user:
```bash
python -m scripts.package_skill <path/to/skill-folder>
```
After packaging, direct the user to the resulting `.skill` file path so they can install it.
---
## Claude.ai-specific instructions
In Claude.ai, the core workflow is the same (draft → test → review → improve → repeat), but because Claude.ai doesn't have subagents, some mechanics change. Here's what to adapt:
**Running test cases**: No subagents means no parallel execution. For each test case, read the skill's SKILL.md, then follow its instructions to accomplish the test prompt yourself. Do them one at a time. This is less rigorous than independent subagents (you wrote the skill and you're also running it, so you have full context), but it's a useful sanity check — and the human review step compensates. Skip the baseline runs — just use the skill to complete the task as requested.
**Reviewing results**: If you can't open a browser (e.g., Claude.ai's VM has no display, or you're on a remote server), skip the browser reviewer entirely. Instead, present results directly in the conversation. For each test case, show the prompt and the output. If the output is a file the user needs to see (like a .docx or .xlsx), save it to the filesystem and tell them where it is so they can download and inspect it. Ask for feedback inline: "How does this look? Anything you'd change?"
**Benchmarking**: Skip the quantitative benchmarking — it relies on baseline comparisons which aren't meaningful without subagents. Focus on qualitative feedback from the user.
**The iteration loop**: Same as before — improve the skill, rerun the test cases, ask for feedback — just without the browser reviewer in the middle. You can still organize results into iteration directories on the filesystem if you have one.
**Description optimization**: This section requires the `claude` CLI tool (specifically `claude -p`) which is only available in Claude Code. Skip it if you're on Claude.ai.
**Blind comparison**: Requires subagents. Skip it.
**Packaging**: The `package_skill.py` script works anywhere with Python and a filesystem. On Claude.ai, you can run it and the user can download the resulting `.skill` file.
**Updating an existing skill**: The user might be asking you to update an existing skill, not create a new one. In this case:
- **Preserve the original name.** Note the skill's directory name and `name` frontmatter field -- use them unchanged. E.g., if the installed skill is `research-helper`, output `research-helper.skill` (not `research-helper-v2`).
- **Copy to a writeable location before editing.** The installed skill path may be read-only. Copy to `/tmp/skill-name/`, edit there, and package from the copy.
- **If packaging manually, stage in `/tmp/` first**, then copy to the output directory -- direct writes may fail due to permissions.
---
## Cowork-Specific Instructions
If you're in Cowork, the main things to know are:
- You have subagents, so the main workflow (spawn test cases in parallel, run baselines, grade, etc.) all works. (However, if you run into severe problems with timeouts, it's OK to run the test prompts in series rather than parallel.)
- You don't have a browser or display, so when generating the eval viewer, use `--static <output_path>` to write a standalone HTML file instead of starting a server. Then proffer a link that the user can click to open the HTML in their browser.
- For whatever reason, the Cowork setup seems to disincline Claude from generating the eval viewer after running the tests, so just to reiterate: whether you're in Cowork or in Claude Code, after running tests, you should always generate the eval viewer for the human to look at examples before revising the skill yourself and trying to make corrections, using `generate_review.py` (not writing your own boutique html code). Sorry in advance but I'm gonna go all caps here: GENERATE THE EVAL VIEWER *BEFORE* evaluating inputs yourself. You want to get them in front of the human ASAP!
- Feedback works differently: since there's no running server, the viewer's "Submit All Reviews" button will download `feedback.json` as a file. You can then read it from there (you may have to request access first).
- Packaging works — `package_skill.py` just needs Python and a filesystem.
- Description optimization (`run_loop.py` / `run_eval.py`) should work in Cowork just fine since it uses `claude -p` via subprocess, not a browser, but please save it until you've fully finished making the skill and the user agrees it's in good shape.
- **Updating an existing skill**: The user might be asking you to update an existing skill, not create a new one. Follow the update guidance in the claude.ai section above.
---
## Reference files
The agents/ directory contains instructions for specialized subagents. Read them when you need to spawn the relevant subagent.
- `agents/grader.md` — How to evaluate assertions against outputs
- `agents/comparator.md` — How to do blind A/B comparison between two outputs
- `agents/analyzer.md` — How to analyze why one version beat another
The references/ directory has additional documentation:
- `references/schemas.md` — JSON structures for evals.json, grading.json, etc.
---
Repeating one more time the core loop here for emphasis:
- Figure out what the skill is about
- Draft or edit the skill
- Run claude-with-access-to-the-skill on test prompts
- With the user, evaluate the outputs:
- Create benchmark.json and run `eval-viewer/generate_review.py` to help the user review them
- Run quantitative evals
- Repeat until you and the user are satisfied
- Package the final skill and return it to the user.
Please add steps to your TodoList, if you have such a thing, to make sure you don't forget. If you're in Cowork, please specifically put "Create evals JSON and run `eval-viewer/generate_review.py` so human can review test cases" in your TodoList to make sure it happens.
Good luck!
-274
View File
@@ -1,274 +0,0 @@
# Post-hoc Analyzer Agent
Analyze blind comparison results to understand WHY the winner won and generate improvement suggestions.
## Role
After the blind comparator determines a winner, the Post-hoc Analyzer "unblids" the results by examining the skills and transcripts. The goal is to extract actionable insights: what made the winner better, and how can the loser be improved?
## Inputs
You receive these parameters in your prompt:
- **winner**: "A" or "B" (from blind comparison)
- **winner_skill_path**: Path to the skill that produced the winning output
- **winner_transcript_path**: Path to the execution transcript for the winner
- **loser_skill_path**: Path to the skill that produced the losing output
- **loser_transcript_path**: Path to the execution transcript for the loser
- **comparison_result_path**: Path to the blind comparator's output JSON
- **output_path**: Where to save the analysis results
## Process
### Step 1: Read Comparison Result
1. Read the blind comparator's output at comparison_result_path
2. Note the winning side (A or B), the reasoning, and any scores
3. Understand what the comparator valued in the winning output
### Step 2: Read Both Skills
1. Read the winner skill's SKILL.md and key referenced files
2. Read the loser skill's SKILL.md and key referenced files
3. Identify structural differences:
- Instructions clarity and specificity
- Script/tool usage patterns
- Example coverage
- Edge case handling
### Step 3: Read Both Transcripts
1. Read the winner's transcript
2. Read the loser's transcript
3. Compare execution patterns:
- How closely did each follow their skill's instructions?
- What tools were used differently?
- Where did the loser diverge from optimal behavior?
- Did either encounter errors or make recovery attempts?
### Step 4: Analyze Instruction Following
For each transcript, evaluate:
- Did the agent follow the skill's explicit instructions?
- Did the agent use the skill's provided tools/scripts?
- Were there missed opportunities to leverage skill content?
- Did the agent add unnecessary steps not in the skill?
Score instruction following 1-10 and note specific issues.
### Step 5: Identify Winner Strengths
Determine what made the winner better:
- Clearer instructions that led to better behavior?
- Better scripts/tools that produced better output?
- More comprehensive examples that guided edge cases?
- Better error handling guidance?
Be specific. Quote from skills/transcripts where relevant.
### Step 6: Identify Loser Weaknesses
Determine what held the loser back:
- Ambiguous instructions that led to suboptimal choices?
- Missing tools/scripts that forced workarounds?
- Gaps in edge case coverage?
- Poor error handling that caused failures?
### Step 7: Generate Improvement Suggestions
Based on the analysis, produce actionable suggestions for improving the loser skill:
- Specific instruction changes to make
- Tools/scripts to add or modify
- Examples to include
- Edge cases to address
Prioritize by impact. Focus on changes that would have changed the outcome.
### Step 8: Write Analysis Results
Save structured analysis to `{output_path}`.
## Output Format
Write a JSON file with this structure:
```json
{
"comparison_summary": {
"winner": "A",
"winner_skill": "path/to/winner/skill",
"loser_skill": "path/to/loser/skill",
"comparator_reasoning": "Brief summary of why comparator chose winner"
},
"winner_strengths": [
"Clear step-by-step instructions for handling multi-page documents",
"Included validation script that caught formatting errors",
"Explicit guidance on fallback behavior when OCR fails"
],
"loser_weaknesses": [
"Vague instruction 'process the document appropriately' led to inconsistent behavior",
"No script for validation, agent had to improvise and made errors",
"No guidance on OCR failure, agent gave up instead of trying alternatives"
],
"instruction_following": {
"winner": {
"score": 9,
"issues": [
"Minor: skipped optional logging step"
]
},
"loser": {
"score": 6,
"issues": [
"Did not use the skill's formatting template",
"Invented own approach instead of following step 3",
"Missed the 'always validate output' instruction"
]
}
},
"improvement_suggestions": [
{
"priority": "high",
"category": "instructions",
"suggestion": "Replace 'process the document appropriately' with explicit steps: 1) Extract text, 2) Identify sections, 3) Format per template",
"expected_impact": "Would eliminate ambiguity that caused inconsistent behavior"
},
{
"priority": "high",
"category": "tools",
"suggestion": "Add validate_output.py script similar to winner skill's validation approach",
"expected_impact": "Would catch formatting errors before final output"
},
{
"priority": "medium",
"category": "error_handling",
"suggestion": "Add fallback instructions: 'If OCR fails, try: 1) different resolution, 2) image preprocessing, 3) manual extraction'",
"expected_impact": "Would prevent early failure on difficult documents"
}
],
"transcript_insights": {
"winner_execution_pattern": "Read skill -> Followed 5-step process -> Used validation script -> Fixed 2 issues -> Produced output",
"loser_execution_pattern": "Read skill -> Unclear on approach -> Tried 3 different methods -> No validation -> Output had errors"
}
}
```
## Guidelines
- **Be specific**: Quote from skills and transcripts, don't just say "instructions were unclear"
- **Be actionable**: Suggestions should be concrete changes, not vague advice
- **Focus on skill improvements**: The goal is to improve the losing skill, not critique the agent
- **Prioritize by impact**: Which changes would most likely have changed the outcome?
- **Consider causation**: Did the skill weakness actually cause the worse output, or is it incidental?
- **Stay objective**: Analyze what happened, don't editorialize
- **Think about generalization**: Would this improvement help on other evals too?
## Categories for Suggestions
Use these categories to organize improvement suggestions:
| Category | Description |
|----------|-------------|
| `instructions` | Changes to the skill's prose instructions |
| `tools` | Scripts, templates, or utilities to add/modify |
| `examples` | Example inputs/outputs to include |
| `error_handling` | Guidance for handling failures |
| `structure` | Reorganization of skill content |
| `references` | External docs or resources to add |
## Priority Levels
- **high**: Would likely change the outcome of this comparison
- **medium**: Would improve quality but may not change win/loss
- **low**: Nice to have, marginal improvement
---
# Analyzing Benchmark Results
When analyzing benchmark results, the analyzer's purpose is to **surface patterns and anomalies** across multiple runs, not suggest skill improvements.
## Role
Review all benchmark run results and generate freeform notes that help the user understand skill performance. Focus on patterns that wouldn't be visible from aggregate metrics alone.
## Inputs
You receive these parameters in your prompt:
- **benchmark_data_path**: Path to the in-progress benchmark.json with all run results
- **skill_path**: Path to the skill being benchmarked
- **output_path**: Where to save the notes (as JSON array of strings)
## Process
### Step 1: Read Benchmark Data
1. Read the benchmark.json containing all run results
2. Note the configurations tested (with_skill, without_skill)
3. Understand the run_summary aggregates already calculated
### Step 2: Analyze Per-Assertion Patterns
For each expectation across all runs:
- Does it **always pass** in both configurations? (may not differentiate skill value)
- Does it **always fail** in both configurations? (may be broken or beyond capability)
- Does it **always pass with skill but fail without**? (skill clearly adds value here)
- Does it **always fail with skill but pass without**? (skill may be hurting)
- Is it **highly variable**? (flaky expectation or non-deterministic behavior)
### Step 3: Analyze Cross-Eval Patterns
Look for patterns across evals:
- Are certain eval types consistently harder/easier?
- Do some evals show high variance while others are stable?
- Are there surprising results that contradict expectations?
### Step 4: Analyze Metrics Patterns
Look at time_seconds, tokens, tool_calls:
- Does the skill significantly increase execution time?
- Is there high variance in resource usage?
- Are there outlier runs that skew the aggregates?
### Step 5: Generate Notes
Write freeform observations as a list of strings. Each note should:
- State a specific observation
- Be grounded in the data (not speculation)
- Help the user understand something the aggregate metrics don't show
Examples:
- "Assertion 'Output is a PDF file' passes 100% in both configurations - may not differentiate skill value"
- "Eval 3 shows high variance (50% ± 40%) - run 2 had an unusual failure that may be flaky"
- "Without-skill runs consistently fail on table extraction expectations (0% pass rate)"
- "Skill adds 13s average execution time but improves pass rate by 50%"
- "Token usage is 80% higher with skill, primarily due to script output parsing"
- "All 3 without-skill runs for eval 1 produced empty output"
### Step 6: Write Notes
Save notes to `{output_path}` as a JSON array of strings:
```json
[
"Assertion 'Output is a PDF file' passes 100% in both configurations - may not differentiate skill value",
"Eval 3 shows high variance (50% ± 40%) - run 2 had an unusual failure",
"Without-skill runs consistently fail on table extraction expectations",
"Skill adds 13s average execution time but improves pass rate by 50%"
]
```
## Guidelines
**DO:**
- Report what you observe in the data
- Be specific about which evals, expectations, or runs you're referring to
- Note patterns that aggregate metrics would hide
- Provide context that helps interpret the numbers
**DO NOT:**
- Suggest improvements to the skill (that's for the improvement step, not benchmarking)
- Make subjective quality judgments ("the output was good/bad")
- Speculate about causes without evidence
- Repeat information already in the run_summary aggregates
-202
View File
@@ -1,202 +0,0 @@
# Blind Comparator Agent
Compare two outputs WITHOUT knowing which skill produced them.
## Role
The Blind Comparator judges which output better accomplishes the eval task. You receive two outputs labeled A and B, but you do NOT know which skill produced which. This prevents bias toward a particular skill or approach.
Your judgment is based purely on output quality and task completion.
## Inputs
You receive these parameters in your prompt:
- **output_a_path**: Path to the first output file or directory
- **output_b_path**: Path to the second output file or directory
- **eval_prompt**: The original task/prompt that was executed
- **expectations**: List of expectations to check (optional - may be empty)
## Process
### Step 1: Read Both Outputs
1. Examine output A (file or directory)
2. Examine output B (file or directory)
3. Note the type, structure, and content of each
4. If outputs are directories, examine all relevant files inside
### Step 2: Understand the Task
1. Read the eval_prompt carefully
2. Identify what the task requires:
- What should be produced?
- What qualities matter (accuracy, completeness, format)?
- What would distinguish a good output from a poor one?
### Step 3: Generate Evaluation Rubric
Based on the task, generate a rubric with two dimensions:
**Content Rubric** (what the output contains):
| Criterion | 1 (Poor) | 3 (Acceptable) | 5 (Excellent) |
|-----------|----------|----------------|---------------|
| Correctness | Major errors | Minor errors | Fully correct |
| Completeness | Missing key elements | Mostly complete | All elements present |
| Accuracy | Significant inaccuracies | Minor inaccuracies | Accurate throughout |
**Structure Rubric** (how the output is organized):
| Criterion | 1 (Poor) | 3 (Acceptable) | 5 (Excellent) |
|-----------|----------|----------------|---------------|
| Organization | Disorganized | Reasonably organized | Clear, logical structure |
| Formatting | Inconsistent/broken | Mostly consistent | Professional, polished |
| Usability | Difficult to use | Usable with effort | Easy to use |
Adapt criteria to the specific task. For example:
- PDF form → "Field alignment", "Text readability", "Data placement"
- Document → "Section structure", "Heading hierarchy", "Paragraph flow"
- Data output → "Schema correctness", "Data types", "Completeness"
### Step 4: Evaluate Each Output Against the Rubric
For each output (A and B):
1. **Score each criterion** on the rubric (1-5 scale)
2. **Calculate dimension totals**: Content score, Structure score
3. **Calculate overall score**: Average of dimension scores, scaled to 1-10
### Step 5: Check Assertions (if provided)
If expectations are provided:
1. Check each expectation against output A
2. Check each expectation against output B
3. Count pass rates for each output
4. Use expectation scores as secondary evidence (not the primary decision factor)
### Step 6: Determine the Winner
Compare A and B based on (in priority order):
1. **Primary**: Overall rubric score (content + structure)
2. **Secondary**: Assertion pass rates (if applicable)
3. **Tiebreaker**: If truly equal, declare a TIE
Be decisive - ties should be rare. One output is usually better, even if marginally.
### Step 7: Write Comparison Results
Save results to a JSON file at the path specified (or `comparison.json` if not specified).
## Output Format
Write a JSON file with this structure:
```json
{
"winner": "A",
"reasoning": "Output A provides a complete solution with proper formatting and all required fields. Output B is missing the date field and has formatting inconsistencies.",
"rubric": {
"A": {
"content": {
"correctness": 5,
"completeness": 5,
"accuracy": 4
},
"structure": {
"organization": 4,
"formatting": 5,
"usability": 4
},
"content_score": 4.7,
"structure_score": 4.3,
"overall_score": 9.0
},
"B": {
"content": {
"correctness": 3,
"completeness": 2,
"accuracy": 3
},
"structure": {
"organization": 3,
"formatting": 2,
"usability": 3
},
"content_score": 2.7,
"structure_score": 2.7,
"overall_score": 5.4
}
},
"output_quality": {
"A": {
"score": 9,
"strengths": ["Complete solution", "Well-formatted", "All fields present"],
"weaknesses": ["Minor style inconsistency in header"]
},
"B": {
"score": 5,
"strengths": ["Readable output", "Correct basic structure"],
"weaknesses": ["Missing date field", "Formatting inconsistencies", "Partial data extraction"]
}
},
"expectation_results": {
"A": {
"passed": 4,
"total": 5,
"pass_rate": 0.80,
"details": [
{"text": "Output includes name", "passed": true},
{"text": "Output includes date", "passed": true},
{"text": "Format is PDF", "passed": true},
{"text": "Contains signature", "passed": false},
{"text": "Readable text", "passed": true}
]
},
"B": {
"passed": 3,
"total": 5,
"pass_rate": 0.60,
"details": [
{"text": "Output includes name", "passed": true},
{"text": "Output includes date", "passed": false},
{"text": "Format is PDF", "passed": true},
{"text": "Contains signature", "passed": false},
{"text": "Readable text", "passed": true}
]
}
}
}
```
If no expectations were provided, omit the `expectation_results` field entirely.
## Field Descriptions
- **winner**: "A", "B", or "TIE"
- **reasoning**: Clear explanation of why the winner was chosen (or why it's a tie)
- **rubric**: Structured rubric evaluation for each output
- **content**: Scores for content criteria (correctness, completeness, accuracy)
- **structure**: Scores for structure criteria (organization, formatting, usability)
- **content_score**: Average of content criteria (1-5)
- **structure_score**: Average of structure criteria (1-5)
- **overall_score**: Combined score scaled to 1-10
- **output_quality**: Summary quality assessment
- **score**: 1-10 rating (should match rubric overall_score)
- **strengths**: List of positive aspects
- **weaknesses**: List of issues or shortcomings
- **expectation_results**: (Only if expectations provided)
- **passed**: Number of expectations that passed
- **total**: Total number of expectations
- **pass_rate**: Fraction passed (0.0 to 1.0)
- **details**: Individual expectation results
## Guidelines
- **Stay blind**: DO NOT try to infer which skill produced which output. Judge purely on output quality.
- **Be specific**: Cite specific examples when explaining strengths and weaknesses.
- **Be decisive**: Choose a winner unless outputs are genuinely equivalent.
- **Output quality first**: Assertion scores are secondary to overall task completion.
- **Be objective**: Don't favor outputs based on style preferences; focus on correctness and completeness.
- **Explain your reasoning**: The reasoning field should make it clear why you chose the winner.
- **Handle edge cases**: If both outputs fail, pick the one that fails less badly. If both are excellent, pick the one that's marginally better.
-223
View File
@@ -1,223 +0,0 @@
# Grader Agent
Evaluate expectations against an execution transcript and outputs.
## Role
The Grader reviews a transcript and output files, then determines whether each expectation passes or fails. Provide clear evidence for each judgment.
You have two jobs: grade the outputs, and critique the evals themselves. A passing grade on a weak assertion is worse than useless — it creates false confidence. When you notice an assertion that's trivially satisfied, or an important outcome that no assertion checks, say so.
## Inputs
You receive these parameters in your prompt:
- **expectations**: List of expectations to evaluate (strings)
- **transcript_path**: Path to the execution transcript (markdown file)
- **outputs_dir**: Directory containing output files from execution
## Process
### Step 1: Read the Transcript
1. Read the transcript file completely
2. Note the eval prompt, execution steps, and final result
3. Identify any issues or errors documented
### Step 2: Examine Output Files
1. List files in outputs_dir
2. Read/examine each file relevant to the expectations. If outputs aren't plain text, use the inspection tools provided in your prompt — don't rely solely on what the transcript says the executor produced.
3. Note contents, structure, and quality
### Step 3: Evaluate Each Assertion
For each expectation:
1. **Search for evidence** in the transcript and outputs
2. **Determine verdict**:
- **PASS**: Clear evidence the expectation is true AND the evidence reflects genuine task completion, not just surface-level compliance
- **FAIL**: No evidence, or evidence contradicts the expectation, or the evidence is superficial (e.g., correct filename but empty/wrong content)
3. **Cite the evidence**: Quote the specific text or describe what you found
### Step 4: Extract and Verify Claims
Beyond the predefined expectations, extract implicit claims from the outputs and verify them:
1. **Extract claims** from the transcript and outputs:
- Factual statements ("The form has 12 fields")
- Process claims ("Used pypdf to fill the form")
- Quality claims ("All fields were filled correctly")
2. **Verify each claim**:
- **Factual claims**: Can be checked against the outputs or external sources
- **Process claims**: Can be verified from the transcript
- **Quality claims**: Evaluate whether the claim is justified
3. **Flag unverifiable claims**: Note claims that cannot be verified with available information
This catches issues that predefined expectations might miss.
### Step 5: Read User Notes
If `{outputs_dir}/user_notes.md` exists:
1. Read it and note any uncertainties or issues flagged by the executor
2. Include relevant concerns in the grading output
3. These may reveal problems even when expectations pass
### Step 6: Critique the Evals
After grading, consider whether the evals themselves could be improved. Only surface suggestions when there's a clear gap.
Good suggestions test meaningful outcomes — assertions that are hard to satisfy without actually doing the work correctly. Think about what makes an assertion *discriminating*: it passes when the skill genuinely succeeds and fails when it doesn't.
Suggestions worth raising:
- An assertion that passed but would also pass for a clearly wrong output (e.g., checking filename existence but not file content)
- An important outcome you observed — good or bad — that no assertion covers at all
- An assertion that can't actually be verified from the available outputs
Keep the bar high. The goal is to flag things the eval author would say "good catch" about, not to nitpick every assertion.
### Step 7: Write Grading Results
Save results to `{outputs_dir}/../grading.json` (sibling to outputs_dir).
## Grading Criteria
**PASS when**:
- The transcript or outputs clearly demonstrate the expectation is true
- Specific evidence can be cited
- The evidence reflects genuine substance, not just surface compliance (e.g., a file exists AND contains correct content, not just the right filename)
**FAIL when**:
- No evidence found for the expectation
- Evidence contradicts the expectation
- The expectation cannot be verified from available information
- The evidence is superficial — the assertion is technically satisfied but the underlying task outcome is wrong or incomplete
- The output appears to meet the assertion by coincidence rather than by actually doing the work
**When uncertain**: The burden of proof to pass is on the expectation.
### Step 8: Read Executor Metrics and Timing
1. If `{outputs_dir}/metrics.json` exists, read it and include in grading output
2. If `{outputs_dir}/../timing.json` exists, read it and include timing data
## Output Format
Write a JSON file with this structure:
```json
{
"expectations": [
{
"text": "The output includes the name 'John Smith'",
"passed": true,
"evidence": "Found in transcript Step 3: 'Extracted names: John Smith, Sarah Johnson'"
},
{
"text": "The spreadsheet has a SUM formula in cell B10",
"passed": false,
"evidence": "No spreadsheet was created. The output was a text file."
},
{
"text": "The assistant used the skill's OCR script",
"passed": true,
"evidence": "Transcript Step 2 shows: 'Tool: Bash - python ocr_script.py image.png'"
}
],
"summary": {
"passed": 2,
"failed": 1,
"total": 3,
"pass_rate": 0.67
},
"execution_metrics": {
"tool_calls": {
"Read": 5,
"Write": 2,
"Bash": 8
},
"total_tool_calls": 15,
"total_steps": 6,
"errors_encountered": 0,
"output_chars": 12450,
"transcript_chars": 3200
},
"timing": {
"executor_duration_seconds": 165.0,
"grader_duration_seconds": 26.0,
"total_duration_seconds": 191.0
},
"claims": [
{
"claim": "The form has 12 fillable fields",
"type": "factual",
"verified": true,
"evidence": "Counted 12 fields in field_info.json"
},
{
"claim": "All required fields were populated",
"type": "quality",
"verified": false,
"evidence": "Reference section was left blank despite data being available"
}
],
"user_notes_summary": {
"uncertainties": ["Used 2023 data, may be stale"],
"needs_review": [],
"workarounds": ["Fell back to text overlay for non-fillable fields"]
},
"eval_feedback": {
"suggestions": [
{
"assertion": "The output includes the name 'John Smith'",
"reason": "A hallucinated document that mentions the name would also pass — consider checking it appears as the primary contact with matching phone and email from the input"
},
{
"reason": "No assertion checks whether the extracted phone numbers match the input — I observed incorrect numbers in the output that went uncaught"
}
],
"overall": "Assertions check presence but not correctness. Consider adding content verification."
}
}
```
## Field Descriptions
- **expectations**: Array of graded expectations
- **text**: The original expectation text
- **passed**: Boolean - true if expectation passes
- **evidence**: Specific quote or description supporting the verdict
- **summary**: Aggregate statistics
- **passed**: Count of passed expectations
- **failed**: Count of failed expectations
- **total**: Total expectations evaluated
- **pass_rate**: Fraction passed (0.0 to 1.0)
- **execution_metrics**: Copied from executor's metrics.json (if available)
- **output_chars**: Total character count of output files (proxy for tokens)
- **transcript_chars**: Character count of transcript
- **timing**: Wall clock timing from timing.json (if available)
- **executor_duration_seconds**: Time spent in executor subagent
- **total_duration_seconds**: Total elapsed time for the run
- **claims**: Extracted and verified claims from the output
- **claim**: The statement being verified
- **type**: "factual", "process", or "quality"
- **verified**: Boolean - whether the claim holds
- **evidence**: Supporting or contradicting evidence
- **user_notes_summary**: Issues flagged by the executor
- **uncertainties**: Things the executor wasn't sure about
- **needs_review**: Items requiring human attention
- **workarounds**: Places where the skill didn't work as expected
- **eval_feedback**: Improvement suggestions for the evals (only when warranted)
- **suggestions**: List of concrete suggestions, each with a `reason` and optionally an `assertion` it relates to
- **overall**: Brief assessment — can be "No suggestions, evals look solid" if nothing to flag
## Guidelines
- **Be objective**: Base verdicts on evidence, not assumptions
- **Be specific**: Quote the exact text that supports your verdict
- **Be thorough**: Check both transcript and output files
- **Be consistent**: Apply the same standard to each expectation
- **Explain failures**: Make it clear why evidence was insufficient
- **No partial credit**: Each expectation is pass or fail, not partial
@@ -1,146 +0,0 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Eval Set Review - __SKILL_NAME_PLACEHOLDER__</title>
<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link href="https://fonts.googleapis.com/css2?family=Poppins:wght@500;600&family=Lora:wght@400;500&display=swap" rel="stylesheet">
<style>
* { box-sizing: border-box; margin: 0; padding: 0; }
body { font-family: 'Lora', Georgia, serif; background: #faf9f5; padding: 2rem; color: #141413; }
h1 { font-family: 'Poppins', sans-serif; margin-bottom: 0.5rem; font-size: 1.5rem; }
.description { color: #b0aea5; margin-bottom: 1.5rem; font-style: italic; max-width: 900px; }
.controls { margin-bottom: 1rem; display: flex; gap: 0.5rem; }
.btn { font-family: 'Poppins', sans-serif; padding: 0.5rem 1rem; border: none; border-radius: 6px; cursor: pointer; font-size: 0.875rem; font-weight: 500; }
.btn-add { background: #6a9bcc; color: white; }
.btn-add:hover { background: #5889b8; }
.btn-export { background: #d97757; color: white; }
.btn-export:hover { background: #c4613f; }
table { width: 100%; max-width: 1100px; border-collapse: collapse; background: white; border-radius: 6px; overflow: hidden; box-shadow: 0 1px 3px rgba(0,0,0,0.08); }
th { font-family: 'Poppins', sans-serif; background: #141413; color: #faf9f5; padding: 0.75rem 1rem; text-align: left; font-size: 0.875rem; }
td { padding: 0.75rem 1rem; border-bottom: 1px solid #e8e6dc; vertical-align: top; }
tr:nth-child(even) td { background: #faf9f5; }
tr:hover td { background: #f3f1ea; }
.section-header td { background: #e8e6dc; font-family: 'Poppins', sans-serif; font-weight: 500; font-size: 0.8rem; color: #141413; text-transform: uppercase; letter-spacing: 0.05em; }
.query-input { width: 100%; padding: 0.4rem; border: 1px solid #e8e6dc; border-radius: 4px; font-size: 0.875rem; font-family: 'Lora', Georgia, serif; resize: vertical; min-height: 60px; }
.query-input:focus { outline: none; border-color: #d97757; box-shadow: 0 0 0 2px rgba(217,119,87,0.15); }
.toggle { position: relative; display: inline-block; width: 44px; height: 24px; }
.toggle input { opacity: 0; width: 0; height: 0; }
.toggle .slider { position: absolute; inset: 0; background: #b0aea5; border-radius: 24px; cursor: pointer; transition: 0.2s; }
.toggle .slider::before { content: ""; position: absolute; width: 18px; height: 18px; left: 3px; bottom: 3px; background: white; border-radius: 50%; transition: 0.2s; }
.toggle input:checked + .slider { background: #d97757; }
.toggle input:checked + .slider::before { transform: translateX(20px); }
.btn-delete { background: #c44; color: white; padding: 0.3rem 0.6rem; border: none; border-radius: 4px; cursor: pointer; font-size: 0.75rem; font-family: 'Poppins', sans-serif; }
.btn-delete:hover { background: #a33; }
.summary { margin-top: 1rem; color: #b0aea5; font-size: 0.875rem; }
</style>
</head>
<body>
<h1>Eval Set Review: <span id="skill-name">__SKILL_NAME_PLACEHOLDER__</span></h1>
<p class="description">Current description: <span id="skill-desc">__SKILL_DESCRIPTION_PLACEHOLDER__</span></p>
<div class="controls">
<button class="btn btn-add" onclick="addRow()">+ Add Query</button>
<button class="btn btn-export" onclick="exportEvalSet()">Export Eval Set</button>
</div>
<table>
<thead>
<tr>
<th style="width:65%">Query</th>
<th style="width:18%">Should Trigger</th>
<th style="width:10%">Actions</th>
</tr>
</thead>
<tbody id="eval-body"></tbody>
</table>
<p class="summary" id="summary"></p>
<script>
const EVAL_DATA = __EVAL_DATA_PLACEHOLDER__;
let evalItems = [...EVAL_DATA];
function render() {
const tbody = document.getElementById('eval-body');
tbody.innerHTML = '';
// Sort: should-trigger first, then should-not-trigger
const sorted = evalItems
.map((item, origIdx) => ({ ...item, origIdx }))
.sort((a, b) => (b.should_trigger ? 1 : 0) - (a.should_trigger ? 1 : 0));
let lastGroup = null;
sorted.forEach(item => {
const group = item.should_trigger ? 'trigger' : 'no-trigger';
if (group !== lastGroup) {
const headerRow = document.createElement('tr');
headerRow.className = 'section-header';
headerRow.innerHTML = `<td colspan="3">${item.should_trigger ? 'Should Trigger' : 'Should NOT Trigger'}</td>`;
tbody.appendChild(headerRow);
lastGroup = group;
}
const idx = item.origIdx;
const tr = document.createElement('tr');
tr.innerHTML = `
<td><textarea class="query-input" onchange="updateQuery(${idx}, this.value)">${escapeHtml(item.query)}</textarea></td>
<td>
<label class="toggle">
<input type="checkbox" ${item.should_trigger ? 'checked' : ''} onchange="updateTrigger(${idx}, this.checked)">
<span class="slider"></span>
</label>
<span style="margin-left:8px;font-size:0.8rem;color:#b0aea5">${item.should_trigger ? 'Yes' : 'No'}</span>
</td>
<td><button class="btn-delete" onclick="deleteRow(${idx})">Delete</button></td>
`;
tbody.appendChild(tr);
});
updateSummary();
}
function escapeHtml(text) {
const div = document.createElement('div');
div.textContent = text;
return div.innerHTML;
}
function updateQuery(idx, value) { evalItems[idx].query = value; updateSummary(); }
function updateTrigger(idx, value) { evalItems[idx].should_trigger = value; render(); }
function deleteRow(idx) { evalItems.splice(idx, 1); render(); }
function addRow() {
evalItems.push({ query: '', should_trigger: true });
render();
const inputs = document.querySelectorAll('.query-input');
inputs[inputs.length - 1].focus();
}
function updateSummary() {
const trigger = evalItems.filter(i => i.should_trigger).length;
const noTrigger = evalItems.filter(i => !i.should_trigger).length;
document.getElementById('summary').textContent =
`${evalItems.length} queries total: ${trigger} should trigger, ${noTrigger} should not trigger`;
}
function exportEvalSet() {
const valid = evalItems.filter(i => i.query.trim() !== '');
const data = valid.map(i => ({ query: i.query.trim(), should_trigger: i.should_trigger }));
const blob = new Blob([JSON.stringify(data, null, 2)], { type: 'application/json' });
const url = URL.createObjectURL(blob);
const a = document.createElement('a');
a.href = url;
a.download = 'eval_set.json';
document.body.appendChild(a);
a.click();
document.body.removeChild(a);
URL.revokeObjectURL(url);
}
render();
</script>
</body>
</html>
@@ -1,471 +0,0 @@
#!/usr/bin/env python3
"""Generate and serve a review page for eval results.
Reads the workspace directory, discovers runs (directories with outputs/),
embeds all output data into a self-contained HTML page, and serves it via
a tiny HTTP server. Feedback auto-saves to feedback.json in the workspace.
Usage:
python generate_review.py <workspace-path> [--port PORT] [--skill-name NAME]
python generate_review.py <workspace-path> --previous-feedback /path/to/old/feedback.json
No dependencies beyond the Python stdlib are required.
"""
import argparse
import base64
import json
import mimetypes
import os
import re
import signal
import subprocess
import sys
import time
import webbrowser
from functools import partial
from http.server import HTTPServer, BaseHTTPRequestHandler
from pathlib import Path
# Files to exclude from output listings
METADATA_FILES = {"transcript.md", "user_notes.md", "metrics.json"}
# Extensions we render as inline text
TEXT_EXTENSIONS = {
".txt", ".md", ".json", ".csv", ".py", ".js", ".ts", ".tsx", ".jsx",
".yaml", ".yml", ".xml", ".html", ".css", ".sh", ".rb", ".go", ".rs",
".java", ".c", ".cpp", ".h", ".hpp", ".sql", ".r", ".toml",
}
# Extensions we render as inline images
IMAGE_EXTENSIONS = {".png", ".jpg", ".jpeg", ".gif", ".svg", ".webp"}
# MIME type overrides for common types
MIME_OVERRIDES = {
".svg": "image/svg+xml",
".xlsx": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
".docx": "application/vnd.openxmlformats-officedocument.wordprocessingml.document",
".pptx": "application/vnd.openxmlformats-officedocument.presentationml.presentation",
}
def get_mime_type(path: Path) -> str:
ext = path.suffix.lower()
if ext in MIME_OVERRIDES:
return MIME_OVERRIDES[ext]
mime, _ = mimetypes.guess_type(str(path))
return mime or "application/octet-stream"
def find_runs(workspace: Path) -> list[dict]:
"""Recursively find directories that contain an outputs/ subdirectory."""
runs: list[dict] = []
_find_runs_recursive(workspace, workspace, runs)
runs.sort(key=lambda r: (r.get("eval_id", float("inf")), r["id"]))
return runs
def _find_runs_recursive(root: Path, current: Path, runs: list[dict]) -> None:
if not current.is_dir():
return
outputs_dir = current / "outputs"
if outputs_dir.is_dir():
run = build_run(root, current)
if run:
runs.append(run)
return
skip = {"node_modules", ".git", "__pycache__", "skill", "inputs"}
for child in sorted(current.iterdir()):
if child.is_dir() and child.name not in skip:
_find_runs_recursive(root, child, runs)
def build_run(root: Path, run_dir: Path) -> dict | None:
"""Build a run dict with prompt, outputs, and grading data."""
prompt = ""
eval_id = None
# Try eval_metadata.json
for candidate in [run_dir / "eval_metadata.json", run_dir.parent / "eval_metadata.json"]:
if candidate.exists():
try:
metadata = json.loads(candidate.read_text())
prompt = metadata.get("prompt", "")
eval_id = metadata.get("eval_id")
except (json.JSONDecodeError, OSError):
pass
if prompt:
break
# Fall back to transcript.md
if not prompt:
for candidate in [run_dir / "transcript.md", run_dir / "outputs" / "transcript.md"]:
if candidate.exists():
try:
text = candidate.read_text()
match = re.search(r"## Eval Prompt\n\n([\s\S]*?)(?=\n##|$)", text)
if match:
prompt = match.group(1).strip()
except OSError:
pass
if prompt:
break
if not prompt:
prompt = "(No prompt found)"
run_id = str(run_dir.relative_to(root)).replace("/", "-").replace("\\", "-")
# Collect output files
outputs_dir = run_dir / "outputs"
output_files: list[dict] = []
if outputs_dir.is_dir():
for f in sorted(outputs_dir.iterdir()):
if f.is_file() and f.name not in METADATA_FILES:
output_files.append(embed_file(f))
# Load grading if present
grading = None
for candidate in [run_dir / "grading.json", run_dir.parent / "grading.json"]:
if candidate.exists():
try:
grading = json.loads(candidate.read_text())
except (json.JSONDecodeError, OSError):
pass
if grading:
break
return {
"id": run_id,
"prompt": prompt,
"eval_id": eval_id,
"outputs": output_files,
"grading": grading,
}
def embed_file(path: Path) -> dict:
"""Read a file and return an embedded representation."""
ext = path.suffix.lower()
mime = get_mime_type(path)
if ext in TEXT_EXTENSIONS:
try:
content = path.read_text(errors="replace")
except OSError:
content = "(Error reading file)"
return {
"name": path.name,
"type": "text",
"content": content,
}
elif ext in IMAGE_EXTENSIONS:
try:
raw = path.read_bytes()
b64 = base64.b64encode(raw).decode("ascii")
except OSError:
return {"name": path.name, "type": "error", "content": "(Error reading file)"}
return {
"name": path.name,
"type": "image",
"mime": mime,
"data_uri": f"data:{mime};base64,{b64}",
}
elif ext == ".pdf":
try:
raw = path.read_bytes()
b64 = base64.b64encode(raw).decode("ascii")
except OSError:
return {"name": path.name, "type": "error", "content": "(Error reading file)"}
return {
"name": path.name,
"type": "pdf",
"data_uri": f"data:{mime};base64,{b64}",
}
elif ext == ".xlsx":
try:
raw = path.read_bytes()
b64 = base64.b64encode(raw).decode("ascii")
except OSError:
return {"name": path.name, "type": "error", "content": "(Error reading file)"}
return {
"name": path.name,
"type": "xlsx",
"data_b64": b64,
}
else:
# Binary / unknown — base64 download link
try:
raw = path.read_bytes()
b64 = base64.b64encode(raw).decode("ascii")
except OSError:
return {"name": path.name, "type": "error", "content": "(Error reading file)"}
return {
"name": path.name,
"type": "binary",
"mime": mime,
"data_uri": f"data:{mime};base64,{b64}",
}
def load_previous_iteration(workspace: Path) -> dict[str, dict]:
"""Load previous iteration's feedback and outputs.
Returns a map of run_id -> {"feedback": str, "outputs": list[dict]}.
"""
result: dict[str, dict] = {}
# Load feedback
feedback_map: dict[str, str] = {}
feedback_path = workspace / "feedback.json"
if feedback_path.exists():
try:
data = json.loads(feedback_path.read_text())
feedback_map = {
r["run_id"]: r["feedback"]
for r in data.get("reviews", [])
if r.get("feedback", "").strip()
}
except (json.JSONDecodeError, OSError, KeyError):
pass
# Load runs (to get outputs)
prev_runs = find_runs(workspace)
for run in prev_runs:
result[run["id"]] = {
"feedback": feedback_map.get(run["id"], ""),
"outputs": run.get("outputs", []),
}
# Also add feedback for run_ids that had feedback but no matching run
for run_id, fb in feedback_map.items():
if run_id not in result:
result[run_id] = {"feedback": fb, "outputs": []}
return result
def generate_html(
runs: list[dict],
skill_name: str,
previous: dict[str, dict] | None = None,
benchmark: dict | None = None,
) -> str:
"""Generate the complete standalone HTML page with embedded data."""
template_path = Path(__file__).parent / "viewer.html"
template = template_path.read_text()
# Build previous_feedback and previous_outputs maps for the template
previous_feedback: dict[str, str] = {}
previous_outputs: dict[str, list[dict]] = {}
if previous:
for run_id, data in previous.items():
if data.get("feedback"):
previous_feedback[run_id] = data["feedback"]
if data.get("outputs"):
previous_outputs[run_id] = data["outputs"]
embedded = {
"skill_name": skill_name,
"runs": runs,
"previous_feedback": previous_feedback,
"previous_outputs": previous_outputs,
}
if benchmark:
embedded["benchmark"] = benchmark
data_json = json.dumps(embedded)
return template.replace("/*__EMBEDDED_DATA__*/", f"const EMBEDDED_DATA = {data_json};")
# ---------------------------------------------------------------------------
# HTTP server (stdlib only, zero dependencies)
# ---------------------------------------------------------------------------
def _kill_port(port: int) -> None:
"""Kill any process listening on the given port."""
try:
result = subprocess.run(
["lsof", "-ti", f":{port}"],
capture_output=True, text=True, timeout=5,
)
for pid_str in result.stdout.strip().split("\n"):
if pid_str.strip():
try:
os.kill(int(pid_str.strip()), signal.SIGTERM)
except (ProcessLookupError, ValueError):
pass
if result.stdout.strip():
time.sleep(0.5)
except subprocess.TimeoutExpired:
pass
except FileNotFoundError:
print("Note: lsof not found, cannot check if port is in use", file=sys.stderr)
class ReviewHandler(BaseHTTPRequestHandler):
"""Serves the review HTML and handles feedback saves.
Regenerates the HTML on each page load so that refreshing the browser
picks up new eval outputs without restarting the server.
"""
def __init__(
self,
workspace: Path,
skill_name: str,
feedback_path: Path,
previous: dict[str, dict],
benchmark_path: Path | None,
*args,
**kwargs,
):
self.workspace = workspace
self.skill_name = skill_name
self.feedback_path = feedback_path
self.previous = previous
self.benchmark_path = benchmark_path
super().__init__(*args, **kwargs)
def do_GET(self) -> None:
if self.path == "/" or self.path == "/index.html":
# Regenerate HTML on each request (re-scans workspace for new outputs)
runs = find_runs(self.workspace)
benchmark = None
if self.benchmark_path and self.benchmark_path.exists():
try:
benchmark = json.loads(self.benchmark_path.read_text())
except (json.JSONDecodeError, OSError):
pass
html = generate_html(runs, self.skill_name, self.previous, benchmark)
content = html.encode("utf-8")
self.send_response(200)
self.send_header("Content-Type", "text/html; charset=utf-8")
self.send_header("Content-Length", str(len(content)))
self.end_headers()
self.wfile.write(content)
elif self.path == "/api/feedback":
data = b"{}"
if self.feedback_path.exists():
data = self.feedback_path.read_bytes()
self.send_response(200)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(data)))
self.end_headers()
self.wfile.write(data)
else:
self.send_error(404)
def do_POST(self) -> None:
if self.path == "/api/feedback":
length = int(self.headers.get("Content-Length", 0))
body = self.rfile.read(length)
try:
data = json.loads(body)
if not isinstance(data, dict) or "reviews" not in data:
raise ValueError("Expected JSON object with 'reviews' key")
self.feedback_path.write_text(json.dumps(data, indent=2) + "\n")
resp = b'{"ok":true}'
self.send_response(200)
except (json.JSONDecodeError, OSError, ValueError) as e:
resp = json.dumps({"error": str(e)}).encode()
self.send_response(500)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(resp)))
self.end_headers()
self.wfile.write(resp)
else:
self.send_error(404)
def log_message(self, format: str, *args: object) -> None:
# Suppress request logging to keep terminal clean
pass
def main() -> None:
parser = argparse.ArgumentParser(description="Generate and serve eval review")
parser.add_argument("workspace", type=Path, help="Path to workspace directory")
parser.add_argument("--port", "-p", type=int, default=3117, help="Server port (default: 3117)")
parser.add_argument("--skill-name", "-n", type=str, default=None, help="Skill name for header")
parser.add_argument(
"--previous-workspace", type=Path, default=None,
help="Path to previous iteration's workspace (shows old outputs and feedback as context)",
)
parser.add_argument(
"--benchmark", type=Path, default=None,
help="Path to benchmark.json to show in the Benchmark tab",
)
parser.add_argument(
"--static", "-s", type=Path, default=None,
help="Write standalone HTML to this path instead of starting a server",
)
args = parser.parse_args()
workspace = args.workspace.resolve()
if not workspace.is_dir():
print(f"Error: {workspace} is not a directory", file=sys.stderr)
sys.exit(1)
runs = find_runs(workspace)
if not runs:
print(f"No runs found in {workspace}", file=sys.stderr)
sys.exit(1)
skill_name = args.skill_name or workspace.name.replace("-workspace", "")
feedback_path = workspace / "feedback.json"
previous: dict[str, dict] = {}
if args.previous_workspace:
previous = load_previous_iteration(args.previous_workspace.resolve())
benchmark_path = args.benchmark.resolve() if args.benchmark else None
benchmark = None
if benchmark_path and benchmark_path.exists():
try:
benchmark = json.loads(benchmark_path.read_text())
except (json.JSONDecodeError, OSError):
pass
if args.static:
html = generate_html(runs, skill_name, previous, benchmark)
args.static.parent.mkdir(parents=True, exist_ok=True)
args.static.write_text(html)
print(f"\n Static viewer written to: {args.static}\n")
sys.exit(0)
# Kill any existing process on the target port
port = args.port
_kill_port(port)
handler = partial(ReviewHandler, workspace, skill_name, feedback_path, previous, benchmark_path)
try:
server = HTTPServer(("127.0.0.1", port), handler)
except OSError:
# Port still in use after kill attempt — find a free one
server = HTTPServer(("127.0.0.1", 0), handler)
port = server.server_address[1]
url = f"http://localhost:{port}"
print(f"\n Eval Viewer")
print(f" ─────────────────────────────────")
print(f" URL: {url}")
print(f" Workspace: {workspace}")
print(f" Feedback: {feedback_path}")
if previous:
print(f" Previous: {args.previous_workspace} ({len(previous)} runs)")
if benchmark_path:
print(f" Benchmark: {benchmark_path}")
print(f"\n Press Ctrl+C to stop.\n")
webbrowser.open(url)
try:
server.serve_forever()
except KeyboardInterrupt:
print("\nStopped.")
server.server_close()
if __name__ == "__main__":
main()
File diff suppressed because it is too large Load Diff
-430
View File
@@ -1,430 +0,0 @@
# JSON Schemas
This document defines the JSON schemas used by skill-creator.
---
## evals.json
Defines the evals for a skill. Located at `evals/evals.json` within the skill directory.
```json
{
"skill_name": "example-skill",
"evals": [
{
"id": 1,
"prompt": "User's example prompt",
"expected_output": "Description of expected result",
"files": ["evals/files/sample1.pdf"],
"expectations": [
"The output includes X",
"The skill used script Y"
]
}
]
}
```
**Fields:**
- `skill_name`: Name matching the skill's frontmatter
- `evals[].id`: Unique integer identifier
- `evals[].prompt`: The task to execute
- `evals[].expected_output`: Human-readable description of success
- `evals[].files`: Optional list of input file paths (relative to skill root)
- `evals[].expectations`: List of verifiable statements
---
## history.json
Tracks version progression in Improve mode. Located at workspace root.
```json
{
"started_at": "2026-01-15T10:30:00Z",
"skill_name": "pdf",
"current_best": "v2",
"iterations": [
{
"version": "v0",
"parent": null,
"expectation_pass_rate": 0.65,
"grading_result": "baseline",
"is_current_best": false
},
{
"version": "v1",
"parent": "v0",
"expectation_pass_rate": 0.75,
"grading_result": "won",
"is_current_best": false
},
{
"version": "v2",
"parent": "v1",
"expectation_pass_rate": 0.85,
"grading_result": "won",
"is_current_best": true
}
]
}
```
**Fields:**
- `started_at`: ISO timestamp of when improvement started
- `skill_name`: Name of the skill being improved
- `current_best`: Version identifier of the best performer
- `iterations[].version`: Version identifier (v0, v1, ...)
- `iterations[].parent`: Parent version this was derived from
- `iterations[].expectation_pass_rate`: Pass rate from grading
- `iterations[].grading_result`: "baseline", "won", "lost", or "tie"
- `iterations[].is_current_best`: Whether this is the current best version
---
## grading.json
Output from the grader agent. Located at `<run-dir>/grading.json`.
```json
{
"expectations": [
{
"text": "The output includes the name 'John Smith'",
"passed": true,
"evidence": "Found in transcript Step 3: 'Extracted names: John Smith, Sarah Johnson'"
},
{
"text": "The spreadsheet has a SUM formula in cell B10",
"passed": false,
"evidence": "No spreadsheet was created. The output was a text file."
}
],
"summary": {
"passed": 2,
"failed": 1,
"total": 3,
"pass_rate": 0.67
},
"execution_metrics": {
"tool_calls": {
"Read": 5,
"Write": 2,
"Bash": 8
},
"total_tool_calls": 15,
"total_steps": 6,
"errors_encountered": 0,
"output_chars": 12450,
"transcript_chars": 3200
},
"timing": {
"executor_duration_seconds": 165.0,
"grader_duration_seconds": 26.0,
"total_duration_seconds": 191.0
},
"claims": [
{
"claim": "The form has 12 fillable fields",
"type": "factual",
"verified": true,
"evidence": "Counted 12 fields in field_info.json"
}
],
"user_notes_summary": {
"uncertainties": ["Used 2023 data, may be stale"],
"needs_review": [],
"workarounds": ["Fell back to text overlay for non-fillable fields"]
},
"eval_feedback": {
"suggestions": [
{
"assertion": "The output includes the name 'John Smith'",
"reason": "A hallucinated document that mentions the name would also pass"
}
],
"overall": "Assertions check presence but not correctness."
}
}
```
**Fields:**
- `expectations[]`: Graded expectations with evidence
- `summary`: Aggregate pass/fail counts
- `execution_metrics`: Tool usage and output size (from executor's metrics.json)
- `timing`: Wall clock timing (from timing.json)
- `claims`: Extracted and verified claims from the output
- `user_notes_summary`: Issues flagged by the executor
- `eval_feedback`: (optional) Improvement suggestions for the evals, only present when the grader identifies issues worth raising
---
## metrics.json
Output from the executor agent. Located at `<run-dir>/outputs/metrics.json`.
```json
{
"tool_calls": {
"Read": 5,
"Write": 2,
"Bash": 8,
"Edit": 1,
"Glob": 2,
"Grep": 0
},
"total_tool_calls": 18,
"total_steps": 6,
"files_created": ["filled_form.pdf", "field_values.json"],
"errors_encountered": 0,
"output_chars": 12450,
"transcript_chars": 3200
}
```
**Fields:**
- `tool_calls`: Count per tool type
- `total_tool_calls`: Sum of all tool calls
- `total_steps`: Number of major execution steps
- `files_created`: List of output files created
- `errors_encountered`: Number of errors during execution
- `output_chars`: Total character count of output files
- `transcript_chars`: Character count of transcript
---
## timing.json
Wall clock timing for a run. Located at `<run-dir>/timing.json`.
**How to capture:** When a subagent task completes, the task notification includes `total_tokens` and `duration_ms`. Save these immediately — they are not persisted anywhere else and cannot be recovered after the fact.
```json
{
"total_tokens": 84852,
"duration_ms": 23332,
"total_duration_seconds": 23.3,
"executor_start": "2026-01-15T10:30:00Z",
"executor_end": "2026-01-15T10:32:45Z",
"executor_duration_seconds": 165.0,
"grader_start": "2026-01-15T10:32:46Z",
"grader_end": "2026-01-15T10:33:12Z",
"grader_duration_seconds": 26.0
}
```
---
## benchmark.json
Output from Benchmark mode. Located at `benchmarks/<timestamp>/benchmark.json`.
```json
{
"metadata": {
"skill_name": "pdf",
"skill_path": "/path/to/pdf",
"executor_model": "claude-sonnet-4-20250514",
"analyzer_model": "most-capable-model",
"timestamp": "2026-01-15T10:30:00Z",
"evals_run": [1, 2, 3],
"runs_per_configuration": 3
},
"runs": [
{
"eval_id": 1,
"eval_name": "Ocean",
"configuration": "with_skill",
"run_number": 1,
"result": {
"pass_rate": 0.85,
"passed": 6,
"failed": 1,
"total": 7,
"time_seconds": 42.5,
"tokens": 3800,
"tool_calls": 18,
"errors": 0
},
"expectations": [
{"text": "...", "passed": true, "evidence": "..."}
],
"notes": [
"Used 2023 data, may be stale",
"Fell back to text overlay for non-fillable fields"
]
}
],
"run_summary": {
"with_skill": {
"pass_rate": {"mean": 0.85, "stddev": 0.05, "min": 0.80, "max": 0.90},
"time_seconds": {"mean": 45.0, "stddev": 12.0, "min": 32.0, "max": 58.0},
"tokens": {"mean": 3800, "stddev": 400, "min": 3200, "max": 4100}
},
"without_skill": {
"pass_rate": {"mean": 0.35, "stddev": 0.08, "min": 0.28, "max": 0.45},
"time_seconds": {"mean": 32.0, "stddev": 8.0, "min": 24.0, "max": 42.0},
"tokens": {"mean": 2100, "stddev": 300, "min": 1800, "max": 2500}
},
"delta": {
"pass_rate": "+0.50",
"time_seconds": "+13.0",
"tokens": "+1700"
}
},
"notes": [
"Assertion 'Output is a PDF file' passes 100% in both configurations - may not differentiate skill value",
"Eval 3 shows high variance (50% ± 40%) - may be flaky or model-dependent",
"Without-skill runs consistently fail on table extraction expectations",
"Skill adds 13s average execution time but improves pass rate by 50%"
]
}
```
**Fields:**
- `metadata`: Information about the benchmark run
- `skill_name`: Name of the skill
- `timestamp`: When the benchmark was run
- `evals_run`: List of eval names or IDs
- `runs_per_configuration`: Number of runs per config (e.g. 3)
- `runs[]`: Individual run results
- `eval_id`: Numeric eval identifier
- `eval_name`: Human-readable eval name (used as section header in the viewer)
- `configuration`: Must be `"with_skill"` or `"without_skill"` (the viewer uses this exact string for grouping and color coding)
- `run_number`: Integer run number (1, 2, 3...)
- `result`: Nested object with `pass_rate`, `passed`, `total`, `time_seconds`, `tokens`, `errors`
- `run_summary`: Statistical aggregates per configuration
- `with_skill` / `without_skill`: Each contains `pass_rate`, `time_seconds`, `tokens` objects with `mean` and `stddev` fields
- `delta`: Difference strings like `"+0.50"`, `"+13.0"`, `"+1700"`
- `notes`: Freeform observations from the analyzer
**Important:** The viewer reads these field names exactly. Using `config` instead of `configuration`, or putting `pass_rate` at the top level of a run instead of nested under `result`, will cause the viewer to show empty/zero values. Always reference this schema when generating benchmark.json manually.
---
## comparison.json
Output from blind comparator. Located at `<grading-dir>/comparison-N.json`.
```json
{
"winner": "A",
"reasoning": "Output A provides a complete solution with proper formatting and all required fields. Output B is missing the date field and has formatting inconsistencies.",
"rubric": {
"A": {
"content": {
"correctness": 5,
"completeness": 5,
"accuracy": 4
},
"structure": {
"organization": 4,
"formatting": 5,
"usability": 4
},
"content_score": 4.7,
"structure_score": 4.3,
"overall_score": 9.0
},
"B": {
"content": {
"correctness": 3,
"completeness": 2,
"accuracy": 3
},
"structure": {
"organization": 3,
"formatting": 2,
"usability": 3
},
"content_score": 2.7,
"structure_score": 2.7,
"overall_score": 5.4
}
},
"output_quality": {
"A": {
"score": 9,
"strengths": ["Complete solution", "Well-formatted", "All fields present"],
"weaknesses": ["Minor style inconsistency in header"]
},
"B": {
"score": 5,
"strengths": ["Readable output", "Correct basic structure"],
"weaknesses": ["Missing date field", "Formatting inconsistencies", "Partial data extraction"]
}
},
"expectation_results": {
"A": {
"passed": 4,
"total": 5,
"pass_rate": 0.80,
"details": [
{"text": "Output includes name", "passed": true}
]
},
"B": {
"passed": 3,
"total": 5,
"pass_rate": 0.60,
"details": [
{"text": "Output includes name", "passed": true}
]
}
}
}
```
---
## analysis.json
Output from post-hoc analyzer. Located at `<grading-dir>/analysis.json`.
```json
{
"comparison_summary": {
"winner": "A",
"winner_skill": "path/to/winner/skill",
"loser_skill": "path/to/loser/skill",
"comparator_reasoning": "Brief summary of why comparator chose winner"
},
"winner_strengths": [
"Clear step-by-step instructions for handling multi-page documents",
"Included validation script that caught formatting errors"
],
"loser_weaknesses": [
"Vague instruction 'process the document appropriately' led to inconsistent behavior",
"No script for validation, agent had to improvise"
],
"instruction_following": {
"winner": {
"score": 9,
"issues": ["Minor: skipped optional logging step"]
},
"loser": {
"score": 6,
"issues": [
"Did not use the skill's formatting template",
"Invented own approach instead of following step 3"
]
}
},
"improvement_suggestions": [
{
"priority": "high",
"category": "instructions",
"suggestion": "Replace 'process the document appropriately' with explicit steps",
"expected_impact": "Would eliminate ambiguity that caused inconsistent behavior"
}
],
"transcript_insights": {
"winner_execution_pattern": "Read skill -> Followed 5-step process -> Used validation script",
"loser_execution_pattern": "Read skill -> Unclear on approach -> Tried 3 different methods"
}
}
```
@@ -1,401 +0,0 @@
#!/usr/bin/env python3
"""
Aggregate individual run results into benchmark summary statistics.
Reads grading.json files from run directories and produces:
- run_summary with mean, stddev, min, max for each metric
- delta between with_skill and without_skill configurations
Usage:
python aggregate_benchmark.py <benchmark_dir>
Example:
python aggregate_benchmark.py benchmarks/2026-01-15T10-30-00/
The script supports two directory layouts:
Workspace layout (from skill-creator iterations):
<benchmark_dir>/
eval-N/
with_skill/
run-1/grading.json
run-2/grading.json
without_skill/
run-1/grading.json
run-2/grading.json
Legacy layout (with runs/ subdirectory):
<benchmark_dir>/
runs/
eval-N/
with_skill/
run-1/grading.json
without_skill/
run-1/grading.json
"""
import argparse
import json
import math
import sys
from datetime import datetime, timezone
from pathlib import Path
def calculate_stats(values: list[float]) -> dict:
"""Calculate mean, stddev, min, max for a list of values."""
if not values:
return {"mean": 0.0, "stddev": 0.0, "min": 0.0, "max": 0.0}
n = len(values)
mean = sum(values) / n
if n > 1:
variance = sum((x - mean) ** 2 for x in values) / (n - 1)
stddev = math.sqrt(variance)
else:
stddev = 0.0
return {
"mean": round(mean, 4),
"stddev": round(stddev, 4),
"min": round(min(values), 4),
"max": round(max(values), 4)
}
def load_run_results(benchmark_dir: Path) -> dict:
"""
Load all run results from a benchmark directory.
Returns dict keyed by config name (e.g. "with_skill"/"without_skill",
or "new_skill"/"old_skill"), each containing a list of run results.
"""
# Support both layouts: eval dirs directly under benchmark_dir, or under runs/
runs_dir = benchmark_dir / "runs"
if runs_dir.exists():
search_dir = runs_dir
elif list(benchmark_dir.glob("eval-*")):
search_dir = benchmark_dir
else:
print(f"No eval directories found in {benchmark_dir} or {benchmark_dir / 'runs'}")
return {}
results: dict[str, list] = {}
for eval_idx, eval_dir in enumerate(sorted(search_dir.glob("eval-*"))):
metadata_path = eval_dir / "eval_metadata.json"
if metadata_path.exists():
try:
with open(metadata_path) as mf:
eval_id = json.load(mf).get("eval_id", eval_idx)
except (json.JSONDecodeError, OSError):
eval_id = eval_idx
else:
try:
eval_id = int(eval_dir.name.split("-")[1])
except ValueError:
eval_id = eval_idx
# Discover config directories dynamically rather than hardcoding names
for config_dir in sorted(eval_dir.iterdir()):
if not config_dir.is_dir():
continue
# Skip non-config directories (inputs, outputs, etc.)
if not list(config_dir.glob("run-*")):
continue
config = config_dir.name
if config not in results:
results[config] = []
for run_dir in sorted(config_dir.glob("run-*")):
run_number = int(run_dir.name.split("-")[1])
grading_file = run_dir / "grading.json"
if not grading_file.exists():
print(f"Warning: grading.json not found in {run_dir}")
continue
try:
with open(grading_file) as f:
grading = json.load(f)
except json.JSONDecodeError as e:
print(f"Warning: Invalid JSON in {grading_file}: {e}")
continue
# Extract metrics
result = {
"eval_id": eval_id,
"run_number": run_number,
"pass_rate": grading.get("summary", {}).get("pass_rate", 0.0),
"passed": grading.get("summary", {}).get("passed", 0),
"failed": grading.get("summary", {}).get("failed", 0),
"total": grading.get("summary", {}).get("total", 0),
}
# Extract timing — check grading.json first, then sibling timing.json
timing = grading.get("timing", {})
result["time_seconds"] = timing.get("total_duration_seconds", 0.0)
timing_file = run_dir / "timing.json"
if result["time_seconds"] == 0.0 and timing_file.exists():
try:
with open(timing_file) as tf:
timing_data = json.load(tf)
result["time_seconds"] = timing_data.get("total_duration_seconds", 0.0)
result["tokens"] = timing_data.get("total_tokens", 0)
except json.JSONDecodeError:
pass
# Extract metrics if available
metrics = grading.get("execution_metrics", {})
result["tool_calls"] = metrics.get("total_tool_calls", 0)
if not result.get("tokens"):
result["tokens"] = metrics.get("output_chars", 0)
result["errors"] = metrics.get("errors_encountered", 0)
# Extract expectations — viewer requires fields: text, passed, evidence
raw_expectations = grading.get("expectations", [])
for exp in raw_expectations:
if "text" not in exp or "passed" not in exp:
print(f"Warning: expectation in {grading_file} missing required fields (text, passed, evidence): {exp}")
result["expectations"] = raw_expectations
# Extract notes from user_notes_summary
notes_summary = grading.get("user_notes_summary", {})
notes = []
notes.extend(notes_summary.get("uncertainties", []))
notes.extend(notes_summary.get("needs_review", []))
notes.extend(notes_summary.get("workarounds", []))
result["notes"] = notes
results[config].append(result)
return results
def aggregate_results(results: dict) -> dict:
"""
Aggregate run results into summary statistics.
Returns run_summary with stats for each configuration and delta.
"""
run_summary = {}
configs = list(results.keys())
for config in configs:
runs = results.get(config, [])
if not runs:
run_summary[config] = {
"pass_rate": {"mean": 0.0, "stddev": 0.0, "min": 0.0, "max": 0.0},
"time_seconds": {"mean": 0.0, "stddev": 0.0, "min": 0.0, "max": 0.0},
"tokens": {"mean": 0, "stddev": 0, "min": 0, "max": 0}
}
continue
pass_rates = [r["pass_rate"] for r in runs]
times = [r["time_seconds"] for r in runs]
tokens = [r.get("tokens", 0) for r in runs]
run_summary[config] = {
"pass_rate": calculate_stats(pass_rates),
"time_seconds": calculate_stats(times),
"tokens": calculate_stats(tokens)
}
# Calculate delta between the first two configs (if two exist)
if len(configs) >= 2:
primary = run_summary.get(configs[0], {})
baseline = run_summary.get(configs[1], {})
else:
primary = run_summary.get(configs[0], {}) if configs else {}
baseline = {}
delta_pass_rate = primary.get("pass_rate", {}).get("mean", 0) - baseline.get("pass_rate", {}).get("mean", 0)
delta_time = primary.get("time_seconds", {}).get("mean", 0) - baseline.get("time_seconds", {}).get("mean", 0)
delta_tokens = primary.get("tokens", {}).get("mean", 0) - baseline.get("tokens", {}).get("mean", 0)
run_summary["delta"] = {
"pass_rate": f"{delta_pass_rate:+.2f}",
"time_seconds": f"{delta_time:+.1f}",
"tokens": f"{delta_tokens:+.0f}"
}
return run_summary
def generate_benchmark(benchmark_dir: Path, skill_name: str = "", skill_path: str = "") -> dict:
"""
Generate complete benchmark.json from run results.
"""
results = load_run_results(benchmark_dir)
run_summary = aggregate_results(results)
# Build runs array for benchmark.json
runs = []
for config in results:
for result in results[config]:
runs.append({
"eval_id": result["eval_id"],
"configuration": config,
"run_number": result["run_number"],
"result": {
"pass_rate": result["pass_rate"],
"passed": result["passed"],
"failed": result["failed"],
"total": result["total"],
"time_seconds": result["time_seconds"],
"tokens": result.get("tokens", 0),
"tool_calls": result.get("tool_calls", 0),
"errors": result.get("errors", 0)
},
"expectations": result["expectations"],
"notes": result["notes"]
})
# Determine eval IDs from results
eval_ids = sorted(set(
r["eval_id"]
for config in results.values()
for r in config
))
benchmark = {
"metadata": {
"skill_name": skill_name or "<skill-name>",
"skill_path": skill_path or "<path/to/skill>",
"executor_model": "<model-name>",
"analyzer_model": "<model-name>",
"timestamp": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),
"evals_run": eval_ids,
"runs_per_configuration": 3
},
"runs": runs,
"run_summary": run_summary,
"notes": [] # To be filled by analyzer
}
return benchmark
def generate_markdown(benchmark: dict) -> str:
"""Generate human-readable benchmark.md from benchmark data."""
metadata = benchmark["metadata"]
run_summary = benchmark["run_summary"]
# Determine config names (excluding "delta")
configs = [k for k in run_summary if k != "delta"]
config_a = configs[0] if len(configs) >= 1 else "config_a"
config_b = configs[1] if len(configs) >= 2 else "config_b"
label_a = config_a.replace("_", " ").title()
label_b = config_b.replace("_", " ").title()
lines = [
f"# Skill Benchmark: {metadata['skill_name']}",
"",
f"**Model**: {metadata['executor_model']}",
f"**Date**: {metadata['timestamp']}",
f"**Evals**: {', '.join(map(str, metadata['evals_run']))} ({metadata['runs_per_configuration']} runs each per configuration)",
"",
"## Summary",
"",
f"| Metric | {label_a} | {label_b} | Delta |",
"|--------|------------|---------------|-------|",
]
a_summary = run_summary.get(config_a, {})
b_summary = run_summary.get(config_b, {})
delta = run_summary.get("delta", {})
# Format pass rate
a_pr = a_summary.get("pass_rate", {})
b_pr = b_summary.get("pass_rate", {})
lines.append(f"| Pass Rate | {a_pr.get('mean', 0)*100:.0f}% ± {a_pr.get('stddev', 0)*100:.0f}% | {b_pr.get('mean', 0)*100:.0f}% ± {b_pr.get('stddev', 0)*100:.0f}% | {delta.get('pass_rate', '')} |")
# Format time
a_time = a_summary.get("time_seconds", {})
b_time = b_summary.get("time_seconds", {})
lines.append(f"| Time | {a_time.get('mean', 0):.1f}s ± {a_time.get('stddev', 0):.1f}s | {b_time.get('mean', 0):.1f}s ± {b_time.get('stddev', 0):.1f}s | {delta.get('time_seconds', '')}s |")
# Format tokens
a_tokens = a_summary.get("tokens", {})
b_tokens = b_summary.get("tokens", {})
lines.append(f"| Tokens | {a_tokens.get('mean', 0):.0f} ± {a_tokens.get('stddev', 0):.0f} | {b_tokens.get('mean', 0):.0f} ± {b_tokens.get('stddev', 0):.0f} | {delta.get('tokens', '')} |")
# Notes section
if benchmark.get("notes"):
lines.extend([
"",
"## Notes",
""
])
for note in benchmark["notes"]:
lines.append(f"- {note}")
return "\n".join(lines)
def main():
parser = argparse.ArgumentParser(
description="Aggregate benchmark run results into summary statistics"
)
parser.add_argument(
"benchmark_dir",
type=Path,
help="Path to the benchmark directory"
)
parser.add_argument(
"--skill-name",
default="",
help="Name of the skill being benchmarked"
)
parser.add_argument(
"--skill-path",
default="",
help="Path to the skill being benchmarked"
)
parser.add_argument(
"--output", "-o",
type=Path,
help="Output path for benchmark.json (default: <benchmark_dir>/benchmark.json)"
)
args = parser.parse_args()
if not args.benchmark_dir.exists():
print(f"Directory not found: {args.benchmark_dir}")
sys.exit(1)
# Generate benchmark
benchmark = generate_benchmark(args.benchmark_dir, args.skill_name, args.skill_path)
# Determine output paths
output_json = args.output or (args.benchmark_dir / "benchmark.json")
output_md = output_json.with_suffix(".md")
# Write benchmark.json
with open(output_json, "w") as f:
json.dump(benchmark, f, indent=2)
print(f"Generated: {output_json}")
# Write benchmark.md
markdown = generate_markdown(benchmark)
with open(output_md, "w") as f:
f.write(markdown)
print(f"Generated: {output_md}")
# Print summary
run_summary = benchmark["run_summary"]
configs = [k for k in run_summary if k != "delta"]
delta = run_summary.get("delta", {})
print(f"\nSummary:")
for config in configs:
pr = run_summary[config]["pass_rate"]["mean"]
label = config.replace("_", " ").title()
print(f" {label}: {pr*100:.1f}% pass rate")
print(f" Delta: {delta.get('pass_rate', '')}")
if __name__ == "__main__":
main()
@@ -1,326 +0,0 @@
#!/usr/bin/env python3
"""Generate an HTML report from run_loop.py output.
Takes the JSON output from run_loop.py and generates a visual HTML report
showing each description attempt with check/x for each test case.
Distinguishes between train and test queries.
"""
import argparse
import html
import json
import sys
from pathlib import Path
def generate_html(data: dict, auto_refresh: bool = False, skill_name: str = "") -> str:
"""Generate HTML report from loop output data. If auto_refresh is True, adds a meta refresh tag."""
history = data.get("history", [])
holdout = data.get("holdout", 0)
title_prefix = html.escape(skill_name + " \u2014 ") if skill_name else ""
# Get all unique queries from train and test sets, with should_trigger info
train_queries: list[dict] = []
test_queries: list[dict] = []
if history:
for r in history[0].get("train_results", history[0].get("results", [])):
train_queries.append({"query": r["query"], "should_trigger": r.get("should_trigger", True)})
if history[0].get("test_results"):
for r in history[0].get("test_results", []):
test_queries.append({"query": r["query"], "should_trigger": r.get("should_trigger", True)})
refresh_tag = ' <meta http-equiv="refresh" content="5">\n' if auto_refresh else ""
html_parts = ["""<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
""" + refresh_tag + """ <title>""" + title_prefix + """Skill Description Optimization</title>
<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link href="https://fonts.googleapis.com/css2?family=Poppins:wght@500;600&family=Lora:wght@400;500&display=swap" rel="stylesheet">
<style>
body {
font-family: 'Lora', Georgia, serif;
max-width: 100%;
margin: 0 auto;
padding: 20px;
background: #faf9f5;
color: #141413;
}
h1 { font-family: 'Poppins', sans-serif; color: #141413; }
.explainer {
background: white;
padding: 15px;
border-radius: 6px;
margin-bottom: 20px;
border: 1px solid #e8e6dc;
color: #b0aea5;
font-size: 0.875rem;
line-height: 1.6;
}
.summary {
background: white;
padding: 15px;
border-radius: 6px;
margin-bottom: 20px;
border: 1px solid #e8e6dc;
}
.summary p { margin: 5px 0; }
.best { color: #788c5d; font-weight: bold; }
.table-container {
overflow-x: auto;
width: 100%;
}
table {
border-collapse: collapse;
background: white;
border: 1px solid #e8e6dc;
border-radius: 6px;
font-size: 12px;
min-width: 100%;
}
th, td {
padding: 8px;
text-align: left;
border: 1px solid #e8e6dc;
white-space: normal;
word-wrap: break-word;
}
th {
font-family: 'Poppins', sans-serif;
background: #141413;
color: #faf9f5;
font-weight: 500;
}
th.test-col {
background: #6a9bcc;
}
th.query-col { min-width: 200px; }
td.description {
font-family: monospace;
font-size: 11px;
word-wrap: break-word;
max-width: 400px;
}
td.result {
text-align: center;
font-size: 16px;
min-width: 40px;
}
td.test-result {
background: #f0f6fc;
}
.pass { color: #788c5d; }
.fail { color: #c44; }
.rate {
font-size: 9px;
color: #b0aea5;
display: block;
}
tr:hover { background: #faf9f5; }
.score {
display: inline-block;
padding: 2px 6px;
border-radius: 4px;
font-weight: bold;
font-size: 11px;
}
.score-good { background: #eef2e8; color: #788c5d; }
.score-ok { background: #fef3c7; color: #d97706; }
.score-bad { background: #fceaea; color: #c44; }
.train-label { color: #b0aea5; font-size: 10px; }
.test-label { color: #6a9bcc; font-size: 10px; font-weight: bold; }
.best-row { background: #f5f8f2; }
th.positive-col { border-bottom: 3px solid #788c5d; }
th.negative-col { border-bottom: 3px solid #c44; }
th.test-col.positive-col { border-bottom: 3px solid #788c5d; }
th.test-col.negative-col { border-bottom: 3px solid #c44; }
.legend { font-family: 'Poppins', sans-serif; display: flex; gap: 20px; margin-bottom: 10px; font-size: 13px; align-items: center; }
.legend-item { display: flex; align-items: center; gap: 6px; }
.legend-swatch { width: 16px; height: 16px; border-radius: 3px; display: inline-block; }
.swatch-positive { background: #141413; border-bottom: 3px solid #788c5d; }
.swatch-negative { background: #141413; border-bottom: 3px solid #c44; }
.swatch-test { background: #6a9bcc; }
.swatch-train { background: #141413; }
</style>
</head>
<body>
<h1>""" + title_prefix + """Skill Description Optimization</h1>
<div class="explainer">
<strong>Optimizing your skill's description.</strong> This page updates automatically as Claude tests different versions of your skill's description. Each row is an iteration a new description attempt. The columns show test queries: green checkmarks mean the skill triggered correctly (or correctly didn't trigger), red crosses mean it got it wrong. The "Train" score shows performance on queries used to improve the description; the "Test" score shows performance on held-out queries the optimizer hasn't seen. When it's done, Claude will apply the best-performing description to your skill.
</div>
"""]
# Summary section
best_test_score = data.get('best_test_score')
best_train_score = data.get('best_train_score')
html_parts.append(f"""
<div class="summary">
<p><strong>Original:</strong> {html.escape(data.get('original_description', 'N/A'))}</p>
<p class="best"><strong>Best:</strong> {html.escape(data.get('best_description', 'N/A'))}</p>
<p><strong>Best Score:</strong> {data.get('best_score', 'N/A')} {'(test)' if best_test_score else '(train)'}</p>
<p><strong>Iterations:</strong> {data.get('iterations_run', 0)} | <strong>Train:</strong> {data.get('train_size', '?')} | <strong>Test:</strong> {data.get('test_size', '?')}</p>
</div>
""")
# Legend
html_parts.append("""
<div class="legend">
<span style="font-weight:600">Query columns:</span>
<span class="legend-item"><span class="legend-swatch swatch-positive"></span> Should trigger</span>
<span class="legend-item"><span class="legend-swatch swatch-negative"></span> Should NOT trigger</span>
<span class="legend-item"><span class="legend-swatch swatch-train"></span> Train</span>
<span class="legend-item"><span class="legend-swatch swatch-test"></span> Test</span>
</div>
""")
# Table header
html_parts.append("""
<div class="table-container">
<table>
<thead>
<tr>
<th>Iter</th>
<th>Train</th>
<th>Test</th>
<th class="query-col">Description</th>
""")
# Add column headers for train queries
for qinfo in train_queries:
polarity = "positive-col" if qinfo["should_trigger"] else "negative-col"
html_parts.append(f' <th class="{polarity}">{html.escape(qinfo["query"])}</th>\n')
# Add column headers for test queries (different color)
for qinfo in test_queries:
polarity = "positive-col" if qinfo["should_trigger"] else "negative-col"
html_parts.append(f' <th class="test-col {polarity}">{html.escape(qinfo["query"])}</th>\n')
html_parts.append(""" </tr>
</thead>
<tbody>
""")
# Find best iteration for highlighting
if test_queries:
best_iter = max(history, key=lambda h: h.get("test_passed") or 0).get("iteration")
else:
best_iter = max(history, key=lambda h: h.get("train_passed", h.get("passed", 0))).get("iteration")
# Add rows for each iteration
for h in history:
iteration = h.get("iteration", "?")
train_passed = h.get("train_passed", h.get("passed", 0))
train_total = h.get("train_total", h.get("total", 0))
test_passed = h.get("test_passed")
test_total = h.get("test_total")
description = h.get("description", "")
train_results = h.get("train_results", h.get("results", []))
test_results = h.get("test_results", [])
# Create lookups for results by query
train_by_query = {r["query"]: r for r in train_results}
test_by_query = {r["query"]: r for r in test_results} if test_results else {}
# Compute aggregate correct/total runs across all retries
def aggregate_runs(results: list[dict]) -> tuple[int, int]:
correct = 0
total = 0
for r in results:
runs = r.get("runs", 0)
triggers = r.get("triggers", 0)
total += runs
if r.get("should_trigger", True):
correct += triggers
else:
correct += runs - triggers
return correct, total
train_correct, train_runs = aggregate_runs(train_results)
test_correct, test_runs = aggregate_runs(test_results)
# Determine score classes
def score_class(correct: int, total: int) -> str:
if total > 0:
ratio = correct / total
if ratio >= 0.8:
return "score-good"
elif ratio >= 0.5:
return "score-ok"
return "score-bad"
train_class = score_class(train_correct, train_runs)
test_class = score_class(test_correct, test_runs)
row_class = "best-row" if iteration == best_iter else ""
html_parts.append(f""" <tr class="{row_class}">
<td>{iteration}</td>
<td><span class="score {train_class}">{train_correct}/{train_runs}</span></td>
<td><span class="score {test_class}">{test_correct}/{test_runs}</span></td>
<td class="description">{html.escape(description)}</td>
""")
# Add result for each train query
for qinfo in train_queries:
r = train_by_query.get(qinfo["query"], {})
did_pass = r.get("pass", False)
triggers = r.get("triggers", 0)
runs = r.get("runs", 0)
icon = "" if did_pass else ""
css_class = "pass" if did_pass else "fail"
html_parts.append(f' <td class="result {css_class}">{icon}<span class="rate">{triggers}/{runs}</span></td>\n')
# Add result for each test query (with different background)
for qinfo in test_queries:
r = test_by_query.get(qinfo["query"], {})
did_pass = r.get("pass", False)
triggers = r.get("triggers", 0)
runs = r.get("runs", 0)
icon = "" if did_pass else ""
css_class = "pass" if did_pass else "fail"
html_parts.append(f' <td class="result test-result {css_class}">{icon}<span class="rate">{triggers}/{runs}</span></td>\n')
html_parts.append(" </tr>\n")
html_parts.append(""" </tbody>
</table>
</div>
""")
html_parts.append("""
</body>
</html>
""")
return "".join(html_parts)
def main():
parser = argparse.ArgumentParser(description="Generate HTML report from run_loop output")
parser.add_argument("input", help="Path to JSON output from run_loop.py (or - for stdin)")
parser.add_argument("-o", "--output", default=None, help="Output HTML file (default: stdout)")
parser.add_argument("--skill-name", default="", help="Skill name to include in the report title")
args = parser.parse_args()
if args.input == "-":
data = json.load(sys.stdin)
else:
data = json.loads(Path(args.input).read_text())
html_output = generate_html(data, skill_name=args.skill_name)
if args.output:
Path(args.output).write_text(html_output)
print(f"Report written to {args.output}", file=sys.stderr)
else:
print(html_output)
if __name__ == "__main__":
main()
@@ -1,247 +0,0 @@
#!/usr/bin/env python3
"""Improve a skill description based on eval results.
Takes eval results (from run_eval.py) and generates an improved description
by calling `claude -p` as a subprocess (same auth pattern as run_eval.py
uses the session's Claude Code auth, no separate ANTHROPIC_API_KEY needed).
"""
import argparse
import json
import os
import re
import subprocess
import sys
from pathlib import Path
from scripts.utils import parse_skill_md
def _call_claude(prompt: str, model: str | None, timeout: int = 300) -> str:
"""Run `claude -p` with the prompt on stdin and return the text response.
Prompt goes over stdin (not argv) because it embeds the full SKILL.md
body and can easily exceed comfortable argv length.
"""
cmd = ["claude", "-p", "--output-format", "text"]
if model:
cmd.extend(["--model", model])
# Remove CLAUDECODE env var to allow nesting claude -p inside a
# Claude Code session. The guard is for interactive terminal conflicts;
# programmatic subprocess usage is safe. Same pattern as run_eval.py.
env = {k: v for k, v in os.environ.items() if k != "CLAUDECODE"}
result = subprocess.run(
cmd,
input=prompt,
capture_output=True,
text=True,
env=env,
timeout=timeout,
)
if result.returncode != 0:
raise RuntimeError(
f"claude -p exited {result.returncode}\nstderr: {result.stderr}"
)
return result.stdout
def improve_description(
skill_name: str,
skill_content: str,
current_description: str,
eval_results: dict,
history: list[dict],
model: str,
test_results: dict | None = None,
log_dir: Path | None = None,
iteration: int | None = None,
) -> str:
"""Call Claude to improve the description based on eval results."""
failed_triggers = [
r for r in eval_results["results"]
if r["should_trigger"] and not r["pass"]
]
false_triggers = [
r for r in eval_results["results"]
if not r["should_trigger"] and not r["pass"]
]
# Build scores summary
train_score = f"{eval_results['summary']['passed']}/{eval_results['summary']['total']}"
if test_results:
test_score = f"{test_results['summary']['passed']}/{test_results['summary']['total']}"
scores_summary = f"Train: {train_score}, Test: {test_score}"
else:
scores_summary = f"Train: {train_score}"
prompt = f"""You are optimizing a skill description for a Claude Code skill called "{skill_name}". A "skill" is sort of like a prompt, but with progressive disclosure -- there's a title and description that Claude sees when deciding whether to use the skill, and then if it does use the skill, it reads the .md file which has lots more details and potentially links to other resources in the skill folder like helper files and scripts and additional documentation or examples.
The description appears in Claude's "available_skills" list. When a user sends a query, Claude decides whether to invoke the skill based solely on the title and on this description. Your goal is to write a description that triggers for relevant queries, and doesn't trigger for irrelevant ones.
Here's the current description:
<current_description>
"{current_description}"
</current_description>
Current scores ({scores_summary}):
<scores_summary>
"""
if failed_triggers:
prompt += "FAILED TO TRIGGER (should have triggered but didn't):\n"
for r in failed_triggers:
prompt += f' - "{r["query"]}" (triggered {r["triggers"]}/{r["runs"]} times)\n'
prompt += "\n"
if false_triggers:
prompt += "FALSE TRIGGERS (triggered but shouldn't have):\n"
for r in false_triggers:
prompt += f' - "{r["query"]}" (triggered {r["triggers"]}/{r["runs"]} times)\n'
prompt += "\n"
if history:
prompt += "PREVIOUS ATTEMPTS (do NOT repeat these — try something structurally different):\n\n"
for h in history:
train_s = f"{h.get('train_passed', h.get('passed', 0))}/{h.get('train_total', h.get('total', 0))}"
test_s = f"{h.get('test_passed', '?')}/{h.get('test_total', '?')}" if h.get('test_passed') is not None else None
score_str = f"train={train_s}" + (f", test={test_s}" if test_s else "")
prompt += f'<attempt {score_str}>\n'
prompt += f'Description: "{h["description"]}"\n'
if "results" in h:
prompt += "Train results:\n"
for r in h["results"]:
status = "PASS" if r["pass"] else "FAIL"
prompt += f' [{status}] "{r["query"][:80]}" (triggered {r["triggers"]}/{r["runs"]})\n'
if h.get("note"):
prompt += f'Note: {h["note"]}\n'
prompt += "</attempt>\n\n"
prompt += f"""</scores_summary>
Skill content (for context on what the skill does):
<skill_content>
{skill_content}
</skill_content>
Based on the failures, write a new and improved description that is more likely to trigger correctly. When I say "based on the failures", it's a bit of a tricky line to walk because we don't want to overfit to the specific cases you're seeing. So what I DON'T want you to do is produce an ever-expanding list of specific queries that this skill should or shouldn't trigger for. Instead, try to generalize from the failures to broader categories of user intent and situations where this skill would be useful or not useful. The reason for this is twofold:
1. Avoid overfitting
2. The list might get loooong and it's injected into ALL queries and there might be a lot of skills, so we don't want to blow too much space on any given description.
Concretely, your description should not be more than about 100-200 words, even if that comes at the cost of accuracy. There is a hard limit of 1024 characters descriptions over that will be truncated, so stay comfortably under it.
Here are some tips that we've found to work well in writing these descriptions:
- The skill should be phrased in the imperative -- "Use this skill for" rather than "this skill does"
- The skill description should focus on the user's intent, what they are trying to achieve, vs. the implementation details of how the skill works.
- The description competes with other skills for Claude's attention — make it distinctive and immediately recognizable.
- If you're getting lots of failures after repeated attempts, change things up. Try different sentence structures or wordings.
I'd encourage you to be creative and mix up the style in different iterations since you'll have multiple opportunities to try different approaches and we'll just grab the highest-scoring one at the end.
Please respond with only the new description text in <new_description> tags, nothing else."""
text = _call_claude(prompt, model)
match = re.search(r"<new_description>(.*?)</new_description>", text, re.DOTALL)
description = match.group(1).strip().strip('"') if match else text.strip().strip('"')
transcript: dict = {
"iteration": iteration,
"prompt": prompt,
"response": text,
"parsed_description": description,
"char_count": len(description),
"over_limit": len(description) > 1024,
}
# Safety net: the prompt already states the 1024-char hard limit, but if
# the model blew past it anyway, make one fresh single-turn call that
# quotes the too-long version and asks for a shorter rewrite. (The old
# SDK path did this as a true multi-turn; `claude -p` is one-shot, so we
# inline the prior output into the new prompt instead.)
if len(description) > 1024:
shorten_prompt = (
f"{prompt}\n\n"
f"---\n\n"
f"A previous attempt produced this description, which at "
f"{len(description)} characters is over the 1024-character hard limit:\n\n"
f'"{description}"\n\n'
f"Rewrite it to be under 1024 characters while keeping the most "
f"important trigger words and intent coverage. Respond with only "
f"the new description in <new_description> tags."
)
shorten_text = _call_claude(shorten_prompt, model)
match = re.search(r"<new_description>(.*?)</new_description>", shorten_text, re.DOTALL)
shortened = match.group(1).strip().strip('"') if match else shorten_text.strip().strip('"')
transcript["rewrite_prompt"] = shorten_prompt
transcript["rewrite_response"] = shorten_text
transcript["rewrite_description"] = shortened
transcript["rewrite_char_count"] = len(shortened)
description = shortened
transcript["final_description"] = description
if log_dir:
log_dir.mkdir(parents=True, exist_ok=True)
log_file = log_dir / f"improve_iter_{iteration or 'unknown'}.json"
log_file.write_text(json.dumps(transcript, indent=2))
return description
def main():
parser = argparse.ArgumentParser(description="Improve a skill description based on eval results")
parser.add_argument("--eval-results", required=True, help="Path to eval results JSON (from run_eval.py)")
parser.add_argument("--skill-path", required=True, help="Path to skill directory")
parser.add_argument("--history", default=None, help="Path to history JSON (previous attempts)")
parser.add_argument("--model", required=True, help="Model for improvement")
parser.add_argument("--verbose", action="store_true", help="Print thinking to stderr")
args = parser.parse_args()
skill_path = Path(args.skill_path)
if not (skill_path / "SKILL.md").exists():
print(f"Error: No SKILL.md found at {skill_path}", file=sys.stderr)
sys.exit(1)
eval_results = json.loads(Path(args.eval_results).read_text())
history = []
if args.history:
history = json.loads(Path(args.history).read_text())
name, _, content = parse_skill_md(skill_path)
current_description = eval_results["description"]
if args.verbose:
print(f"Current: {current_description}", file=sys.stderr)
print(f"Score: {eval_results['summary']['passed']}/{eval_results['summary']['total']}", file=sys.stderr)
new_description = improve_description(
skill_name=name,
skill_content=content,
current_description=current_description,
eval_results=eval_results,
history=history,
model=args.model,
)
if args.verbose:
print(f"Improved: {new_description}", file=sys.stderr)
# Output as JSON with both the new description and updated history
output = {
"description": new_description,
"history": history + [{
"description": current_description,
"passed": eval_results["summary"]["passed"],
"failed": eval_results["summary"]["failed"],
"total": eval_results["summary"]["total"],
"results": eval_results["results"],
}],
}
print(json.dumps(output, indent=2))
if __name__ == "__main__":
main()
@@ -1,136 +0,0 @@
#!/usr/bin/env python3
"""
Skill Packager - Creates a distributable .skill file of a skill folder
Usage:
python utils/package_skill.py <path/to/skill-folder> [output-directory]
Example:
python utils/package_skill.py skills/public/my-skill
python utils/package_skill.py skills/public/my-skill ./dist
"""
import fnmatch
import sys
import zipfile
from pathlib import Path
from scripts.quick_validate import validate_skill
# Patterns to exclude when packaging skills.
EXCLUDE_DIRS = {"__pycache__", "node_modules"}
EXCLUDE_GLOBS = {"*.pyc"}
EXCLUDE_FILES = {".DS_Store"}
# Directories excluded only at the skill root (not when nested deeper).
ROOT_EXCLUDE_DIRS = {"evals"}
def should_exclude(rel_path: Path) -> bool:
"""Check if a path should be excluded from packaging."""
parts = rel_path.parts
if any(part in EXCLUDE_DIRS for part in parts):
return True
# rel_path is relative to skill_path.parent, so parts[0] is the skill
# folder name and parts[1] (if present) is the first subdir.
if len(parts) > 1 and parts[1] in ROOT_EXCLUDE_DIRS:
return True
name = rel_path.name
if name in EXCLUDE_FILES:
return True
return any(fnmatch.fnmatch(name, pat) for pat in EXCLUDE_GLOBS)
def package_skill(skill_path, output_dir=None):
"""
Package a skill folder into a .skill file.
Args:
skill_path: Path to the skill folder
output_dir: Optional output directory for the .skill file (defaults to current directory)
Returns:
Path to the created .skill file, or None if error
"""
skill_path = Path(skill_path).resolve()
# Validate skill folder exists
if not skill_path.exists():
print(f"❌ Error: Skill folder not found: {skill_path}")
return None
if not skill_path.is_dir():
print(f"❌ Error: Path is not a directory: {skill_path}")
return None
# Validate SKILL.md exists
skill_md = skill_path / "SKILL.md"
if not skill_md.exists():
print(f"❌ Error: SKILL.md not found in {skill_path}")
return None
# Run validation before packaging
print("🔍 Validating skill...")
valid, message = validate_skill(skill_path)
if not valid:
print(f"❌ Validation failed: {message}")
print(" Please fix the validation errors before packaging.")
return None
print(f"{message}\n")
# Determine output location
skill_name = skill_path.name
if output_dir:
output_path = Path(output_dir).resolve()
output_path.mkdir(parents=True, exist_ok=True)
else:
output_path = Path.cwd()
skill_filename = output_path / f"{skill_name}.skill"
# Create the .skill file (zip format)
try:
with zipfile.ZipFile(skill_filename, 'w', zipfile.ZIP_DEFLATED) as zipf:
# Walk through the skill directory, excluding build artifacts
for file_path in skill_path.rglob('*'):
if not file_path.is_file():
continue
arcname = file_path.relative_to(skill_path.parent)
if should_exclude(arcname):
print(f" Skipped: {arcname}")
continue
zipf.write(file_path, arcname)
print(f" Added: {arcname}")
print(f"\n✅ Successfully packaged skill to: {skill_filename}")
return skill_filename
except Exception as e:
print(f"❌ Error creating .skill file: {e}")
return None
def main():
if len(sys.argv) < 2:
print("Usage: python utils/package_skill.py <path/to/skill-folder> [output-directory]")
print("\nExample:")
print(" python utils/package_skill.py skills/public/my-skill")
print(" python utils/package_skill.py skills/public/my-skill ./dist")
sys.exit(1)
skill_path = sys.argv[1]
output_dir = sys.argv[2] if len(sys.argv) > 2 else None
print(f"📦 Packaging skill: {skill_path}")
if output_dir:
print(f" Output directory: {output_dir}")
print()
result = package_skill(skill_path, output_dir)
if result:
sys.exit(0)
else:
sys.exit(1)
if __name__ == "__main__":
main()
@@ -1,103 +0,0 @@
#!/usr/bin/env python3
"""
Quick validation script for skills - minimal version
"""
import sys
import os
import re
import yaml
from pathlib import Path
def validate_skill(skill_path):
"""Basic validation of a skill"""
skill_path = Path(skill_path)
# Check SKILL.md exists
skill_md = skill_path / 'SKILL.md'
if not skill_md.exists():
return False, "SKILL.md not found"
# Read and validate frontmatter
content = skill_md.read_text()
if not content.startswith('---'):
return False, "No YAML frontmatter found"
# Extract frontmatter
match = re.match(r'^---\n(.*?)\n---', content, re.DOTALL)
if not match:
return False, "Invalid frontmatter format"
frontmatter_text = match.group(1)
# Parse YAML frontmatter
try:
frontmatter = yaml.safe_load(frontmatter_text)
if not isinstance(frontmatter, dict):
return False, "Frontmatter must be a YAML dictionary"
except yaml.YAMLError as e:
return False, f"Invalid YAML in frontmatter: {e}"
# Define allowed properties
ALLOWED_PROPERTIES = {'name', 'description', 'license', 'allowed-tools', 'metadata', 'compatibility'}
# Check for unexpected properties (excluding nested keys under metadata)
unexpected_keys = set(frontmatter.keys()) - ALLOWED_PROPERTIES
if unexpected_keys:
return False, (
f"Unexpected key(s) in SKILL.md frontmatter: {', '.join(sorted(unexpected_keys))}. "
f"Allowed properties are: {', '.join(sorted(ALLOWED_PROPERTIES))}"
)
# Check required fields
if 'name' not in frontmatter:
return False, "Missing 'name' in frontmatter"
if 'description' not in frontmatter:
return False, "Missing 'description' in frontmatter"
# Extract name for validation
name = frontmatter.get('name', '')
if not isinstance(name, str):
return False, f"Name must be a string, got {type(name).__name__}"
name = name.strip()
if name:
# Check naming convention (kebab-case: lowercase with hyphens)
if not re.match(r'^[a-z0-9-]+$', name):
return False, f"Name '{name}' should be kebab-case (lowercase letters, digits, and hyphens only)"
if name.startswith('-') or name.endswith('-') or '--' in name:
return False, f"Name '{name}' cannot start/end with hyphen or contain consecutive hyphens"
# Check name length (max 64 characters per spec)
if len(name) > 64:
return False, f"Name is too long ({len(name)} characters). Maximum is 64 characters."
# Extract and validate description
description = frontmatter.get('description', '')
if not isinstance(description, str):
return False, f"Description must be a string, got {type(description).__name__}"
description = description.strip()
if description:
# Check for angle brackets
if '<' in description or '>' in description:
return False, "Description cannot contain angle brackets (< or >)"
# Check description length (max 1024 characters per spec)
if len(description) > 1024:
return False, f"Description is too long ({len(description)} characters). Maximum is 1024 characters."
# Validate compatibility field if present (optional)
compatibility = frontmatter.get('compatibility', '')
if compatibility:
if not isinstance(compatibility, str):
return False, f"Compatibility must be a string, got {type(compatibility).__name__}"
if len(compatibility) > 500:
return False, f"Compatibility is too long ({len(compatibility)} characters). Maximum is 500 characters."
return True, "Skill is valid!"
if __name__ == "__main__":
if len(sys.argv) != 2:
print("Usage: python quick_validate.py <skill_directory>")
sys.exit(1)
valid, message = validate_skill(sys.argv[1])
print(message)
sys.exit(0 if valid else 1)
-310
View File
@@ -1,310 +0,0 @@
#!/usr/bin/env python3
"""Run trigger evaluation for a skill description.
Tests whether a skill's description causes Claude to trigger (read the skill)
for a set of queries. Outputs results as JSON.
"""
import argparse
import json
import os
import select
import subprocess
import sys
import time
import uuid
from concurrent.futures import ProcessPoolExecutor, as_completed
from pathlib import Path
from scripts.utils import parse_skill_md
def find_project_root() -> Path:
"""Find the project root by walking up from cwd looking for .claude/.
Mimics how Claude Code discovers its project root, so the command file
we create ends up where claude -p will look for it.
"""
current = Path.cwd()
for parent in [current, *current.parents]:
if (parent / ".claude").is_dir():
return parent
return current
def run_single_query(
query: str,
skill_name: str,
skill_description: str,
timeout: int,
project_root: str,
model: str | None = None,
) -> bool:
"""Run a single query and return whether the skill was triggered.
Creates a command file in .claude/commands/ so it appears in Claude's
available_skills list, then runs `claude -p` with the raw query.
Uses --include-partial-messages to detect triggering early from
stream events (content_block_start) rather than waiting for the
full assistant message, which only arrives after tool execution.
"""
unique_id = uuid.uuid4().hex[:8]
clean_name = f"{skill_name}-skill-{unique_id}"
project_commands_dir = Path(project_root) / ".claude" / "commands"
command_file = project_commands_dir / f"{clean_name}.md"
try:
project_commands_dir.mkdir(parents=True, exist_ok=True)
# Use YAML block scalar to avoid breaking on quotes in description
indented_desc = "\n ".join(skill_description.split("\n"))
command_content = (
f"---\n"
f"description: |\n"
f" {indented_desc}\n"
f"---\n\n"
f"# {skill_name}\n\n"
f"This skill handles: {skill_description}\n"
)
command_file.write_text(command_content)
cmd = [
"claude",
"-p", query,
"--output-format", "stream-json",
"--verbose",
"--include-partial-messages",
]
if model:
cmd.extend(["--model", model])
# Remove CLAUDECODE env var to allow nesting claude -p inside a
# Claude Code session. The guard is for interactive terminal conflicts;
# programmatic subprocess usage is safe.
env = {k: v for k, v in os.environ.items() if k != "CLAUDECODE"}
process = subprocess.Popen(
cmd,
stdout=subprocess.PIPE,
stderr=subprocess.DEVNULL,
cwd=project_root,
env=env,
)
triggered = False
start_time = time.time()
buffer = ""
# Track state for stream event detection
pending_tool_name = None
accumulated_json = ""
try:
while time.time() - start_time < timeout:
if process.poll() is not None:
remaining = process.stdout.read()
if remaining:
buffer += remaining.decode("utf-8", errors="replace")
break
ready, _, _ = select.select([process.stdout], [], [], 1.0)
if not ready:
continue
chunk = os.read(process.stdout.fileno(), 8192)
if not chunk:
break
buffer += chunk.decode("utf-8", errors="replace")
while "\n" in buffer:
line, buffer = buffer.split("\n", 1)
line = line.strip()
if not line:
continue
try:
event = json.loads(line)
except json.JSONDecodeError:
continue
# Early detection via stream events
if event.get("type") == "stream_event":
se = event.get("event", {})
se_type = se.get("type", "")
if se_type == "content_block_start":
cb = se.get("content_block", {})
if cb.get("type") == "tool_use":
tool_name = cb.get("name", "")
if tool_name in ("Skill", "Read"):
pending_tool_name = tool_name
accumulated_json = ""
else:
return False
elif se_type == "content_block_delta" and pending_tool_name:
delta = se.get("delta", {})
if delta.get("type") == "input_json_delta":
accumulated_json += delta.get("partial_json", "")
if clean_name in accumulated_json:
return True
elif se_type in ("content_block_stop", "message_stop"):
if pending_tool_name:
return clean_name in accumulated_json
if se_type == "message_stop":
return False
# Fallback: full assistant message
elif event.get("type") == "assistant":
message = event.get("message", {})
for content_item in message.get("content", []):
if content_item.get("type") != "tool_use":
continue
tool_name = content_item.get("name", "")
tool_input = content_item.get("input", {})
if tool_name == "Skill" and clean_name in tool_input.get("skill", ""):
triggered = True
elif tool_name == "Read" and clean_name in tool_input.get("file_path", ""):
triggered = True
return triggered
elif event.get("type") == "result":
return triggered
finally:
# Clean up process on any exit path (return, exception, timeout)
if process.poll() is None:
process.kill()
process.wait()
return triggered
finally:
if command_file.exists():
command_file.unlink()
def run_eval(
eval_set: list[dict],
skill_name: str,
description: str,
num_workers: int,
timeout: int,
project_root: Path,
runs_per_query: int = 1,
trigger_threshold: float = 0.5,
model: str | None = None,
) -> dict:
"""Run the full eval set and return results."""
results = []
with ProcessPoolExecutor(max_workers=num_workers) as executor:
future_to_info = {}
for item in eval_set:
for run_idx in range(runs_per_query):
future = executor.submit(
run_single_query,
item["query"],
skill_name,
description,
timeout,
str(project_root),
model,
)
future_to_info[future] = (item, run_idx)
query_triggers: dict[str, list[bool]] = {}
query_items: dict[str, dict] = {}
for future in as_completed(future_to_info):
item, _ = future_to_info[future]
query = item["query"]
query_items[query] = item
if query not in query_triggers:
query_triggers[query] = []
try:
query_triggers[query].append(future.result())
except Exception as e:
print(f"Warning: query failed: {e}", file=sys.stderr)
query_triggers[query].append(False)
for query, triggers in query_triggers.items():
item = query_items[query]
trigger_rate = sum(triggers) / len(triggers)
should_trigger = item["should_trigger"]
if should_trigger:
did_pass = trigger_rate >= trigger_threshold
else:
did_pass = trigger_rate < trigger_threshold
results.append({
"query": query,
"should_trigger": should_trigger,
"trigger_rate": trigger_rate,
"triggers": sum(triggers),
"runs": len(triggers),
"pass": did_pass,
})
passed = sum(1 for r in results if r["pass"])
total = len(results)
return {
"skill_name": skill_name,
"description": description,
"results": results,
"summary": {
"total": total,
"passed": passed,
"failed": total - passed,
},
}
def main():
parser = argparse.ArgumentParser(description="Run trigger evaluation for a skill description")
parser.add_argument("--eval-set", required=True, help="Path to eval set JSON file")
parser.add_argument("--skill-path", required=True, help="Path to skill directory")
parser.add_argument("--description", default=None, help="Override description to test")
parser.add_argument("--num-workers", type=int, default=10, help="Number of parallel workers")
parser.add_argument("--timeout", type=int, default=30, help="Timeout per query in seconds")
parser.add_argument("--runs-per-query", type=int, default=3, help="Number of runs per query")
parser.add_argument("--trigger-threshold", type=float, default=0.5, help="Trigger rate threshold")
parser.add_argument("--model", default=None, help="Model to use for claude -p (default: user's configured model)")
parser.add_argument("--verbose", action="store_true", help="Print progress to stderr")
args = parser.parse_args()
eval_set = json.loads(Path(args.eval_set).read_text())
skill_path = Path(args.skill_path)
if not (skill_path / "SKILL.md").exists():
print(f"Error: No SKILL.md found at {skill_path}", file=sys.stderr)
sys.exit(1)
name, original_description, content = parse_skill_md(skill_path)
description = args.description or original_description
project_root = find_project_root()
if args.verbose:
print(f"Evaluating: {description}", file=sys.stderr)
output = run_eval(
eval_set=eval_set,
skill_name=name,
description=description,
num_workers=args.num_workers,
timeout=args.timeout,
project_root=project_root,
runs_per_query=args.runs_per_query,
trigger_threshold=args.trigger_threshold,
model=args.model,
)
if args.verbose:
summary = output["summary"]
print(f"Results: {summary['passed']}/{summary['total']} passed", file=sys.stderr)
for r in output["results"]:
status = "PASS" if r["pass"] else "FAIL"
rate_str = f"{r['triggers']}/{r['runs']}"
print(f" [{status}] rate={rate_str} expected={r['should_trigger']}: {r['query'][:70]}", file=sys.stderr)
print(json.dumps(output, indent=2))
if __name__ == "__main__":
main()
-328
View File
@@ -1,328 +0,0 @@
#!/usr/bin/env python3
"""Run the eval + improve loop until all pass or max iterations reached.
Combines run_eval.py and improve_description.py in a loop, tracking history
and returning the best description found. Supports train/test split to prevent
overfitting.
"""
import argparse
import json
import random
import sys
import tempfile
import time
import webbrowser
from pathlib import Path
from scripts.generate_report import generate_html
from scripts.improve_description import improve_description
from scripts.run_eval import find_project_root, run_eval
from scripts.utils import parse_skill_md
def split_eval_set(eval_set: list[dict], holdout: float, seed: int = 42) -> tuple[list[dict], list[dict]]:
"""Split eval set into train and test sets, stratified by should_trigger."""
random.seed(seed)
# Separate by should_trigger
trigger = [e for e in eval_set if e["should_trigger"]]
no_trigger = [e for e in eval_set if not e["should_trigger"]]
# Shuffle each group
random.shuffle(trigger)
random.shuffle(no_trigger)
# Calculate split points
n_trigger_test = max(1, int(len(trigger) * holdout))
n_no_trigger_test = max(1, int(len(no_trigger) * holdout))
# Split
test_set = trigger[:n_trigger_test] + no_trigger[:n_no_trigger_test]
train_set = trigger[n_trigger_test:] + no_trigger[n_no_trigger_test:]
return train_set, test_set
def run_loop(
eval_set: list[dict],
skill_path: Path,
description_override: str | None,
num_workers: int,
timeout: int,
max_iterations: int,
runs_per_query: int,
trigger_threshold: float,
holdout: float,
model: str,
verbose: bool,
live_report_path: Path | None = None,
log_dir: Path | None = None,
) -> dict:
"""Run the eval + improvement loop."""
project_root = find_project_root()
name, original_description, content = parse_skill_md(skill_path)
current_description = description_override or original_description
# Split into train/test if holdout > 0
if holdout > 0:
train_set, test_set = split_eval_set(eval_set, holdout)
if verbose:
print(f"Split: {len(train_set)} train, {len(test_set)} test (holdout={holdout})", file=sys.stderr)
else:
train_set = eval_set
test_set = []
history = []
exit_reason = "unknown"
for iteration in range(1, max_iterations + 1):
if verbose:
print(f"\n{'='*60}", file=sys.stderr)
print(f"Iteration {iteration}/{max_iterations}", file=sys.stderr)
print(f"Description: {current_description}", file=sys.stderr)
print(f"{'='*60}", file=sys.stderr)
# Evaluate train + test together in one batch for parallelism
all_queries = train_set + test_set
t0 = time.time()
all_results = run_eval(
eval_set=all_queries,
skill_name=name,
description=current_description,
num_workers=num_workers,
timeout=timeout,
project_root=project_root,
runs_per_query=runs_per_query,
trigger_threshold=trigger_threshold,
model=model,
)
eval_elapsed = time.time() - t0
# Split results back into train/test by matching queries
train_queries_set = {q["query"] for q in train_set}
train_result_list = [r for r in all_results["results"] if r["query"] in train_queries_set]
test_result_list = [r for r in all_results["results"] if r["query"] not in train_queries_set]
train_passed = sum(1 for r in train_result_list if r["pass"])
train_total = len(train_result_list)
train_summary = {"passed": train_passed, "failed": train_total - train_passed, "total": train_total}
train_results = {"results": train_result_list, "summary": train_summary}
if test_set:
test_passed = sum(1 for r in test_result_list if r["pass"])
test_total = len(test_result_list)
test_summary = {"passed": test_passed, "failed": test_total - test_passed, "total": test_total}
test_results = {"results": test_result_list, "summary": test_summary}
else:
test_results = None
test_summary = None
history.append({
"iteration": iteration,
"description": current_description,
"train_passed": train_summary["passed"],
"train_failed": train_summary["failed"],
"train_total": train_summary["total"],
"train_results": train_results["results"],
"test_passed": test_summary["passed"] if test_summary else None,
"test_failed": test_summary["failed"] if test_summary else None,
"test_total": test_summary["total"] if test_summary else None,
"test_results": test_results["results"] if test_results else None,
# For backward compat with report generator
"passed": train_summary["passed"],
"failed": train_summary["failed"],
"total": train_summary["total"],
"results": train_results["results"],
})
# Write live report if path provided
if live_report_path:
partial_output = {
"original_description": original_description,
"best_description": current_description,
"best_score": "in progress",
"iterations_run": len(history),
"holdout": holdout,
"train_size": len(train_set),
"test_size": len(test_set),
"history": history,
}
live_report_path.write_text(generate_html(partial_output, auto_refresh=True, skill_name=name))
if verbose:
def print_eval_stats(label, results, elapsed):
pos = [r for r in results if r["should_trigger"]]
neg = [r for r in results if not r["should_trigger"]]
tp = sum(r["triggers"] for r in pos)
pos_runs = sum(r["runs"] for r in pos)
fn = pos_runs - tp
fp = sum(r["triggers"] for r in neg)
neg_runs = sum(r["runs"] for r in neg)
tn = neg_runs - fp
total = tp + tn + fp + fn
precision = tp / (tp + fp) if (tp + fp) > 0 else 1.0
recall = tp / (tp + fn) if (tp + fn) > 0 else 1.0
accuracy = (tp + tn) / total if total > 0 else 0.0
print(f"{label}: {tp+tn}/{total} correct, precision={precision:.0%} recall={recall:.0%} accuracy={accuracy:.0%} ({elapsed:.1f}s)", file=sys.stderr)
for r in results:
status = "PASS" if r["pass"] else "FAIL"
rate_str = f"{r['triggers']}/{r['runs']}"
print(f" [{status}] rate={rate_str} expected={r['should_trigger']}: {r['query'][:60]}", file=sys.stderr)
print_eval_stats("Train", train_results["results"], eval_elapsed)
if test_summary:
print_eval_stats("Test ", test_results["results"], 0)
if train_summary["failed"] == 0:
exit_reason = f"all_passed (iteration {iteration})"
if verbose:
print(f"\nAll train queries passed on iteration {iteration}!", file=sys.stderr)
break
if iteration == max_iterations:
exit_reason = f"max_iterations ({max_iterations})"
if verbose:
print(f"\nMax iterations reached ({max_iterations}).", file=sys.stderr)
break
# Improve the description based on train results
if verbose:
print(f"\nImproving description...", file=sys.stderr)
t0 = time.time()
# Strip test scores from history so improvement model can't see them
blinded_history = [
{k: v for k, v in h.items() if not k.startswith("test_")}
for h in history
]
new_description = improve_description(
skill_name=name,
skill_content=content,
current_description=current_description,
eval_results=train_results,
history=blinded_history,
model=model,
log_dir=log_dir,
iteration=iteration,
)
improve_elapsed = time.time() - t0
if verbose:
print(f"Proposed ({improve_elapsed:.1f}s): {new_description}", file=sys.stderr)
current_description = new_description
# Find the best iteration by TEST score (or train if no test set)
if test_set:
best = max(history, key=lambda h: h["test_passed"] or 0)
best_score = f"{best['test_passed']}/{best['test_total']}"
else:
best = max(history, key=lambda h: h["train_passed"])
best_score = f"{best['train_passed']}/{best['train_total']}"
if verbose:
print(f"\nExit reason: {exit_reason}", file=sys.stderr)
print(f"Best score: {best_score} (iteration {best['iteration']})", file=sys.stderr)
return {
"exit_reason": exit_reason,
"original_description": original_description,
"best_description": best["description"],
"best_score": best_score,
"best_train_score": f"{best['train_passed']}/{best['train_total']}",
"best_test_score": f"{best['test_passed']}/{best['test_total']}" if test_set else None,
"final_description": current_description,
"iterations_run": len(history),
"holdout": holdout,
"train_size": len(train_set),
"test_size": len(test_set),
"history": history,
}
def main():
parser = argparse.ArgumentParser(description="Run eval + improve loop")
parser.add_argument("--eval-set", required=True, help="Path to eval set JSON file")
parser.add_argument("--skill-path", required=True, help="Path to skill directory")
parser.add_argument("--description", default=None, help="Override starting description")
parser.add_argument("--num-workers", type=int, default=10, help="Number of parallel workers")
parser.add_argument("--timeout", type=int, default=30, help="Timeout per query in seconds")
parser.add_argument("--max-iterations", type=int, default=5, help="Max improvement iterations")
parser.add_argument("--runs-per-query", type=int, default=3, help="Number of runs per query")
parser.add_argument("--trigger-threshold", type=float, default=0.5, help="Trigger rate threshold")
parser.add_argument("--holdout", type=float, default=0.4, help="Fraction of eval set to hold out for testing (0 to disable)")
parser.add_argument("--model", required=True, help="Model for improvement")
parser.add_argument("--verbose", action="store_true", help="Print progress to stderr")
parser.add_argument("--report", default="auto", help="Generate HTML report at this path (default: 'auto' for temp file, 'none' to disable)")
parser.add_argument("--results-dir", default=None, help="Save all outputs (results.json, report.html, log.txt) to a timestamped subdirectory here")
args = parser.parse_args()
eval_set = json.loads(Path(args.eval_set).read_text())
skill_path = Path(args.skill_path)
if not (skill_path / "SKILL.md").exists():
print(f"Error: No SKILL.md found at {skill_path}", file=sys.stderr)
sys.exit(1)
name, _, _ = parse_skill_md(skill_path)
# Set up live report path
if args.report != "none":
if args.report == "auto":
timestamp = time.strftime("%Y%m%d_%H%M%S")
live_report_path = Path(tempfile.gettempdir()) / f"skill_description_report_{skill_path.name}_{timestamp}.html"
else:
live_report_path = Path(args.report)
# Open the report immediately so the user can watch
live_report_path.write_text("<html><body><h1>Starting optimization loop...</h1><meta http-equiv='refresh' content='5'></body></html>")
webbrowser.open(str(live_report_path))
else:
live_report_path = None
# Determine output directory (create before run_loop so logs can be written)
if args.results_dir:
timestamp = time.strftime("%Y-%m-%d_%H%M%S")
results_dir = Path(args.results_dir) / timestamp
results_dir.mkdir(parents=True, exist_ok=True)
else:
results_dir = None
log_dir = results_dir / "logs" if results_dir else None
output = run_loop(
eval_set=eval_set,
skill_path=skill_path,
description_override=args.description,
num_workers=args.num_workers,
timeout=args.timeout,
max_iterations=args.max_iterations,
runs_per_query=args.runs_per_query,
trigger_threshold=args.trigger_threshold,
holdout=args.holdout,
model=args.model,
verbose=args.verbose,
live_report_path=live_report_path,
log_dir=log_dir,
)
# Save JSON output
json_output = json.dumps(output, indent=2)
print(json_output)
if results_dir:
(results_dir / "results.json").write_text(json_output)
# Write final HTML report (without auto-refresh)
if live_report_path:
live_report_path.write_text(generate_html(output, auto_refresh=False, skill_name=name))
print(f"\nReport: {live_report_path}", file=sys.stderr)
if results_dir and live_report_path:
(results_dir / "report.html").write_text(generate_html(output, auto_refresh=False, skill_name=name))
if results_dir:
print(f"Results saved to: {results_dir}", file=sys.stderr)
if __name__ == "__main__":
main()
-47
View File
@@ -1,47 +0,0 @@
"""Shared utilities for skill-creator scripts."""
from pathlib import Path
def parse_skill_md(skill_path: Path) -> tuple[str, str, str]:
"""Parse a SKILL.md file, returning (name, description, full_content)."""
content = (skill_path / "SKILL.md").read_text()
lines = content.split("\n")
if lines[0].strip() != "---":
raise ValueError("SKILL.md missing frontmatter (no opening ---)")
end_idx = None
for i, line in enumerate(lines[1:], start=1):
if line.strip() == "---":
end_idx = i
break
if end_idx is None:
raise ValueError("SKILL.md missing frontmatter (no closing ---)")
name = ""
description = ""
frontmatter_lines = lines[1:end_idx]
i = 0
while i < len(frontmatter_lines):
line = frontmatter_lines[i]
if line.startswith("name:"):
name = line[len("name:"):].strip().strip('"').strip("'")
elif line.startswith("description:"):
value = line[len("description:"):].strip()
# Handle YAML multiline indicators (>, |, >-, |-)
if value in (">", "|", ">-", "|-"):
continuation_lines: list[str] = []
i += 1
while i < len(frontmatter_lines) and (frontmatter_lines[i].startswith(" ") or frontmatter_lines[i].startswith("\t")):
continuation_lines.append(frontmatter_lines[i].strip())
i += 1
description = " ".join(continuation_lines)
continue
else:
description = value.strip('"').strip("'")
i += 1
return name, description, content
+5 -2
View File
@@ -310,9 +310,12 @@ main() {
# agents/ is deliberately not in this list: adding an agent is a documented
# extension point (agents/<id>/meta.json + AGENT.md), so the directory is not
# ours alone and pruning it would delete somebody's work — at the price of an
# upstream-deleted agent lingering. bin/ is out too: two files, both
# upstream-deleted agent lingering. skills/ is out for a stronger version of the
# same reason: the tarball ships no skills at all, so that directory holds only
# instance data — every skill in it was registered by a member — and pruning it
# would delete their work at every update. bin/ is out too: two files, both
# overwritten every time, nothing to reclaim.
for owned in web commands skills docs; do
for owned in web commands docs; do
[ -d "$STAGING/$owned" ] && [ -d "${INSTALL_DIR}/$owned" ] || continue
( cd "${INSTALL_DIR}/$owned" && find . -type f ) | while IFS= read -r rel; do
rel="${rel#./}"