dguiducci/Skald-Circle
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs overhaul: agent-facing doc bundle, read-only docs mount in containers
Nightly Build / build (push) Successful in 6m38s
Details

Browse Source 
- Strip ~55 stale upstream docs (dev docs never meant for the agents)
- Write new slim index.md as an agent-facing guide to the app's features
- Add new plugin docs: comfyui, elevenlabs, kokoro_tts, orpheus_tts_3b,
  remote_connectivity, whisper_local (replacing old names)
- Add docs_host to UserFs: docs/… and ~/docs/… resolve to {WD}/docs,
  mounted read-only at /root/docs in every user's container
- Instruct assistant/kid/project-coordinator agents to read docs/index.md
  when users ask how the software works
This commit is contained in:
dguiducci
2026-07-22 10:20:21 +01:00
parent 7e9127dc69
commit 9224245f6f
73 changed files with 322 additions and 13342 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/plugins/comfyui.md
+31
View File
@@ -0,0 +1,31 @@
# ComfyUI
- **Plugin id:** `comfyui`
- **Category:** Image generation, local
- **Runs:** talks to a separately-running ComfyUI server over HTTP (default `http://localhost:8188`)
## What it does
Turns a locally running [ComfyUI](https://github.com/comfyanonymous/ComfyUI) installation into one or more image-generation models the assistant can call. Every JSON workflow file dropped into a watched folder becomes its own selectable model — so a user can have separate "photo-realistic", "anime", "upscale" etc. workflows, each showing up independently.
The plugin polls the ComfyUI server every 5 seconds. If it's offline, every model built from it disappears until it comes back; if a workflow file is added, edited, or removed, the model list updates live — no restart needed.
## Requirements
- ComfyUI installed and running somewhere reachable from this machine (its own GPU-heavy install, not something this app manages).
- At least one workflow exported from ComfyUI as a JSON API file, containing a `CLIPTextEncode` node (for text-to-image) and/or an image-loading node (for image-to-image). Advanced workflows can add a `_personal_agent` metadata block inside the JSON to control exactly which node/field receives the prompt, negative prompt, width/height/steps, and input image — without that block the plugin auto-detects the first `CLIPTextEncode` node.
## Enabling & configuring (admin)
1. Plugin catalog → **ComfyUI** → enable, then **Configure**.
2. Fields:
- **`base_url`** (default `http://localhost:8188`) — where ComfyUI's API is listening.
- **`workflows_dir`** (default `data/comfyui/workflows`) — folder to watch for `.json` workflow files. Created automatically if missing.
- **`default_negative`** — an optional negative prompt applied to every workflow that has a negative-prompt node.
3. Put workflow JSON files in the workflows folder. Each becomes a model named after the file (or the `name` given in its `_personal_agent` block) in the **Image generation** section of the Models hub — no separate "add provider" step, since this plugin registers its models directly.
## Notes
- A single generation can take up to 5 minutes before the plugin times out and reports an error.
- Image-to-image is supported when the workflow declares an `input_image_node` in `_personal_agent`.
- If ComfyUI is unreachable, tell the user to start their ComfyUI server — there is nothing to fix in this app's configuration in that case.
docs/plugins/elevenlabs.md
+27
View File
@@ -0,0 +1,27 @@
# ElevenLabs
- **Plugin id:** `elevenlabs`
- **Category:** Cloud text-to-speech + transcription
- **Runs:** calls the ElevenLabs cloud API — needs internet and an ElevenLabs account
## What it does
Adds [ElevenLabs](https://elevenlabs.io) as a provider of both realistic text-to-speech voices and speech-to-text transcription. Unlike the local TTS/transcription plugins, nothing runs on this machine — every request goes out to ElevenLabs' servers and is billed on their account.
Enabling the plugin does not by itself add any voice or transcription model — it only makes "ElevenLabs" available as a provider type. Models are added separately in the Models hub (see below), same as any other cloud LLM/TTS/transcription provider in this app.
## Requirements
- An ElevenLabs account and an API key (from the ElevenLabs dashboard).
## Enabling & configuring (admin)
1. Plugin catalog → **ElevenLabs** → enable. (This plugin has no config form of its own.)
2. Go to the Models hub → **LLM Providers**, add a new provider, choose type **ElevenLabs**, paste the API key (stored as a secret field, not shown again after saving).
3. Go to the Models hub → **Transcription** and/or **TTS**, add a model, pick the ElevenLabs provider just created, and choose a voice/model from the list — fetched live from ElevenLabs, so it always reflects what's actually available on that account.
## Notes
- Usage is billed by ElevenLabs per character (TTS) / per minute (transcription) according to the account's plan.
- Several ElevenLabs voice models understand inline markup for expressiveness: `<break time="0.5s" />` for pauses, `<phoneme alphabet="ipa" ph="...">word</phoneme>` for pronunciation, `(laughs)`/`(sighs)`/`(coughs)` for non-verbal sounds, and ALL CAPS or repeated letters for emphasis. Worth mentioning to a user who wants more expressive speech.
- One provider entry can supply both a TTS model and a transcription model — they don't need to be added twice.
docs/plugins/honcho.md
+22 -197
View File
@@ -1,209 +1,34 @@
# Honcho Memory Plugin
# Honcho Memory
Plugin: `crates/plugin-honcho/src/lib.rs`
HTTP client: `crates/honcho-client/` (separate workspace crate)
- **Plugin id:** `honcho`
- **Category:** Long-term memory (external service)
- **Runs:** talks to a Honcho server (self-hosted or hosted) over HTTP — needs network access to it
---
## What it does
## Purpose
Streams a user's completed chat turns to an external [Honcho](https://honcho.dev) server so it can build long-term memory about them across sessions — facts, preferences, a curated "peer card" — and feeds retrieved context back into future turns automatically. Also adds tools the assistant can call directly: `memory_query`, `honcho_profile`, `honcho_search`, `honcho_context`, `honcho_conclude`.
Streams completed chat turns to a [Honcho](https://honcho.dev) server so that it can extract long-term conclusions about the user (write path), and reads that context back into every LLM turn via the [`Memory`](memory.md) trait (read path).
**This is different from the app's built-in private memory** (`user-memory/…`, stored encrypted in the user's own database). Honcho is an external system and stores conversation content **in cleartext**, so it is **strictly opt-in per user and off by default** — enabling the plugin does nothing on its own until each individual user turns it on for themselves.
---
## Requirements
## Self-hosted Docker package
- A running Honcho server reachable from this machine (local install or hosted), and its base URL.
- Optionally an API key, if that Honcho instance requires one.
A ready-to-run Docker Compose setup is in the [`honcho/`](../honcho/) folder at the project root.
It starts four services: the Honcho API, the deriver background worker, PostgreSQL + pgvector, and Redis.
## Enabling & configuring (admin)
**Quick start:**
1. Plugin catalog → **Honcho Memory** → enable, then **Configure** (or its own admin page, once enabled: sidebar → Honcho).
2. Fields:
- **`base_url`** (default `http://localhost:8000`) — the Honcho server's URL.
- **`api_key`** — optional, only if the server requires auth.
- **`workspace_id`** (default `skald-circle`) — a name identifying this instance inside Honcho. Each user becomes a separate "peer" inside the same workspace. Use a fresh, unique name for a new instance.
3. The admin config page includes a connectivity test.
```sh
cd honcho
cp .env.example .env
# Edit .env — set at least LLM_OPENAI_API_KEY=sk-...
docker compose up -d
# API available at http://localhost:8000
```
## Per-user setup
Full instructions, LLM provider options (OpenAI, OpenRouter, Ollama), and troubleshooting are in [`honcho/README.md`](../honcho/README.md).
Long-term memory is **off for every user until they turn it on themselves**. Once the plugin is enabled and the user has been granted access, they'll see a **"Long-term memory"** page in their sidebar with a single opt-in toggle. If a user asks the assistant to "remember things long-term" or asks why it doesn't remember past conversations, and this plugin is enabled, point them to that page rather than trying to enable it on their behalf.
---
## Notes
## Setup
1. Start the Honcho server (see above).
2. Enable the plugin via the agent or REST API:
```json
PUT /api/plugins/honcho
{
"enabled": true,
"config": {
"base_url": "http://localhost:8000",
"api_key": "",
"workspace_id": "personal-agent"
}
}
```
Or ask the main agent: _"enable the honcho plugin"_.
---
## Configuration
Stored in the `plugins` SQLite table (`config` JSON blob). Managed at runtime — no entry in `config.yml`.
| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `base_url` | string | `http://localhost:8000` | Honcho server URL |
| `api_key` | string | _(empty)_ | API key; leave empty for local/unauthenticated instances |
| `workspace_id` | string | `personal-agent` | Honcho workspace identifier for this agent instance |
---
## Honcho Object Model
```
workspace (workspace_id from config — one per agent instance)
├── peer "user" observe_others=true
├── peer "assistant" observe_me=true
└── session one per local chat_sessions.id
├── message peer_id="user"
├── message peer_id="assistant"
└── …
```
**Workspace and peers** are created (idempotently) each time the plugin starts. If they already exist, the API returns an error which is logged at `WARN`/`DEBUG` and ignored.
**Sessions** are created lazily on the first event for a new `chat_sessions.id`, then cached in memory for the life of the listener task. The Honcho session UUID is stored in the session cache but not persisted to SQLite — restarting the plugin creates new Honcho sessions for subsequent events.
---
## Event Filtering
An event is forwarded only when **all** of the following conditions hold:
| Condition | Reason |
| --- | --- |
| `is_interactive = true` | A real user is in the conversation |
| `is_ephemeral = false` | Not a short-lived automated session (cron, tic) |
| `is_synthetic = false` | Message content was typed by the user, not injected by the system |
| `role` is `User` or `Assistant` | Sub-agent messages (`Agent` role) are skipped |
| `content` is non-empty | Guard against empty strings |
---
## Lifecycle
1. **`start()`** — subscribes to `skald.event_bus`, calls `ensure_workspace_ready` (best-effort), then spawns the listener task.
2. **Listener task** — `tokio::select!` loop on the bus receiver and a `CancellationToken`. On `RecvError::Lagged`, logs a warning and continues (some turns are missed but the task stays alive).
3. **`stop()`** — cancels the token and awaits the task.
4. **`reload()`** — follows the standard plugin pattern: start/stop/restart-on-change.
---
## Error Handling
All Honcho API errors are **fire-and-forget**: logged as `warn!` and never propagated to the session handler or the user. A Honcho outage has zero impact on chat functionality.
`HonchoError::Request`'s `Display` walks the full `source()` chain, so transport
failures surface the real cause in logs (e.g. `Request failed: error sending
request for url (...): Connection reset by peer`) instead of just reqwest's
opaque top-line. This makes host↔container issues (e.g. a stale Docker Desktop
port-forward after a container recreation) diagnosable from the `warn!` alone.
---
## Read Path
`HonchoMemory` implements the [`Memory`](../memory.md) trait. Before each LLM turn,
`query_context` is called automatically by `ChatSessionHandler::handle_message` — for
**all** session types: interactive, cron, and tic.
### Flow
1. Checks `is_available()` — returns `None` immediately if the plugin is stopped.
2. Looks up the Honcho session UUID for the local `session_id` in the shared `session_map`.
3. **If a mapping exists** (interactive session with at least one turn written):
- Calls `client.session_context(workspace_id, honcho_session_id, tokens=2000, search_query=user_msg)`.
- Returns the formatted result on success.
- On error: logs `warn!` and falls through to the peer-context fallback **without** `search_query`
(avoids a second embedding of the same user message — `session_context` already embedded it before failing).
4. **Fallback — `peer_context("user")`** (no mapping, or session_context error):
- Cold start / cron / tic (no `session_map` entry): calls with `search_query=user_msg` for relevance.
- After a `session_context` failure: calls **without** `search_query` to avoid double-embedding.
- Returns global user knowledge derived from all sessions Honcho has observed.
- On error: logs `warn!` and returns `None`.
The formatted context is prepended to `extra_system_context` and injected into the system prompt. Errors are never propagated — they degrade gracefully to `None`.
### Context format
`format_context()` extracts, in priority order:
1. `conclusions[].content` → "Known facts about the user: …"
2. `summary` → "Conversation summary: …"
3. Fallback: pretty-printed raw JSON
The result is wrapped in `--- Honcho memory context --- / --- end ---` markers.
---
## LLM-callable Tools
`HonchoMemory::tools()` returns **five** tools whenever the plugin is active
(`is_available()` true). They give the LLM direct, on-demand access to every
layer of Honcho's API, complementing the automatic pre-turn `query_context`
injection. All operate on the `user` peer and are inherited by sub-agents via
`AgentRunConfig::memory_tools`.
The [official Honcho documentation](https://honcho.dev/docs/v3/documentation/features/chat) recommends exposing these as tools so the agent decides on its own when to read or write memory, rather than only relying on automatic injection.
| Tool | Endpoint | Cost | What it does |
| --- | --- | --- | --- |
| `memory_query` | `POST .../peers/user/chat` | High (LLM synthesis) | Natural-language question → synthesized answer (dialectic reasoning, `reasoning_level=low`) |
| `honcho_search` | `GET .../peers/user/context?search_query=…` | Low | Semantic search over derived facts; returns raw ranked excerpts (with ids when present) |
| `honcho_context` | `GET .../peers/user/context` | Low | Full context snapshot (conclusions + summary), no synthesis; optional focus `query` |
| `honcho_profile` | `GET`/`PUT .../peers/user/card` | Low | Read the peer card, or overwrite it with a list of fact strings (`card`) |
| `honcho_conclude` | `POST .../conclusions` / `DELETE .../conclusions/{id}` | Low | Write a new fact (`conclusion`) or delete one by id (`delete_id`); exactly one required |
**Peer model — all tools operate on the `user` peer as both observer and observed.**
This plugin configures the `user` peer with `observe_me = true`, so the user's
self-knowledge lives in the `observer = user / observed = user` slot. Therefore
`honcho_conclude` writes with `observer_id = observed_id = user`, and `honcho_search`
uses `peer_context` (not the `conclusions/query` endpoint, which requires explicit
observer/observed filters) — the same proven path as the automatic read-path
injection. This differs from setups where the assistant observes the user
(`observer = assistant`); keeping observer = user is what lets the read-path see
facts written by `honcho_conclude`.
**When to use vs. the automatic injection:**
| Mechanism | When it fires | Best for |
| --- | --- | --- |
| `query_context` (auto) | Before every LLM turn | Background context, cold-start facts |
| `memory_query` (tool) | LLM calls it explicitly | On-demand deep reasoning mid-conversation |
| `honcho_search` / `honcho_context` (tools) | LLM calls them explicitly | Cheap raw recall without LLM synthesis |
| `honcho_profile` / `honcho_conclude` (tools) | LLM calls them explicitly | Actively curating long-term memory |
**Implementation note:** `Tool::execute` is synchronous but the Honcho calls are
async. All five tools share the `run_blocking` helper, which uses
`tokio::task::block_in_place` + `Handle::current().block_on(...)` to drive the
future from within the Tokio multi-thread scheduler without spawning a new thread.
---
## Future Work
- **Session persistence** — store the Honcho session UUID in a new `chat_sessions.honcho_session_id` column so the mapping survives a plugin restart.
---
## When to Update This File
- Config fields change
- Honcho object model or peer setup changes
- Filtering rules change
- `query_context` flow changes (session vs peer fallback logic)
- Docker Compose setup in `honcho/` changes significantly
- Public API of `crates/honcho-client/` changes
- Explain the privacy trade-off honestly if a user asks: their messages get stored in cleartext on the Honcho server, outside the encrypted database this app otherwise uses. Some users may not want that.
- Both the "remembering" (write) and "recalling" (read/search tools) sides of this plugin are gated on the same opt-in flag — a user who hasn't opted in gets a clear "not opted in" response instead of a silent no-op.
docs/plugins/index.md
-15
View File
@@ -1,15 +0,0 @@
# Plugin System
Extensible plugin architecture: lifecycle, trait contract, built-in plugins.
## Files
- [../plugins.md](../plugins.md) — Plugin trait, PluginManager, HTTP router integration
- **Built-in Plugins:**
- [honcho.md](honcho.md) — Honcho long-term memory plugin: setup, config, filtering, lifecycle
- [mobile-connector.md](mobile-connector.md) — Mobile app relay bridge, E2E encryption, Inbox sync (v2 protocol)
- [telegram.md](telegram.md) — Telegram bot: setup, pairing, whitelist, HITL approval
- [whisper-local.md](whisper-local.md) — Local STT via whisper.cpp (Metal-accelerated)
- [remote.md](remote.md) — Tailscale mesh remote connectivity
See [../index.md#plugin-system](../index.md#plugin-system) for navigation.
docs/plugins/kokoro_tts.md
+28
View File
@@ -0,0 +1,28 @@
# Kokoro TTS
- **Plugin id:** `kokoro_tts`
- **Category:** Text-to-speech, local
- **Runs:** on this machine, as a Python subprocess this plugin manages — CPU-only, no API key
## What it does
Lightweight, fast local text-to-speech using the Kokoro ONNX model. Runs on CPU (no GPU needed), supports several languages, and needs no account or API key. On first start it downloads its own model files (~310 MB + 27 MB) from GitHub releases and caches them locally — after that it works fully offline.
## Requirements
- `python3` available on the host (the plugin writes and spawns a small embedded Python server itself — nothing to install manually).
- Internet access the first time it starts, to download the model files.
## Enabling & configuring (admin)
1. Plugin catalog → **Kokoro TTS** → enable, then **Configure**.
2. Fields:
- **`voice`** (default `if_sara`) — voice id. Prefix meaning: `a`=American, `b`=British, `i`=Italian, `j`=Japanese, `z`=Chinese; `f`=female, `m`=male. Includes `if_sara`, `im_nicola` (Italian), plus several English voices (`af_*`, `am_*`, `bf_*`, `bm_*`).
- **`lang`** (default `it`) — language code for phonemisation: `it`, `en-us`, `en-gb`, `ja`, `zh`, `es`, `fr`, `hi`, `pt-br`, `ko`.
- **`speed`** (default `1.0`, range `0.5`–`2.0`) — speech speed multiplier.
3. Changing the config restarts the underlying subprocess automatically. No separate "add provider" step — it appears directly in the Models hub's TTS section once running.
## Notes
- Best results when the text is written the way it would be spoken: short sentences, no markdown, no bullet points or symbols, standard orthography (accented letters are fine for Italian).
- Good default choice when a user wants spoken responses but has no GPU and no interest in paying for a cloud voice.
docs/plugins/mobile-connector.md
+26 -275
View File
@@ -1,288 +1,39 @@
# Mobile Connector Plugin (`mobile-connector`)
# Mobile Connector
Bridges Skald's **Inbox** (approvals + clarifications + MCP elicitations) to mobile apps over the
**relay**, implementing the **agent** role of the v2 relay protocol. The plugin is
the namespace owner and the sole authority over authorized devices. Skald is
never exposed on the internet: only this plugin connects out, and only to the
relay.
- **Plugin id:** `mobile-connector`
- **Category:** Mobile companion app bridge
- **Runs:** connects out to a relay server over an end-to-end encrypted WebSocket — needs internet access, but never exposes this machine directly to the internet
- Crate: `crates/plugin-mobile-connector` — the **application** layer (thin).
- Networking: `crates/skald-relay-client` — the **standalone, payload-agnostic**
relay client (WS v2 transport, E2E crypto, anti-replay counters, pairing,
device authorization, SQLite persistence). It depends only on
`skald-relay-common` (never on `core-api`), so it is reusable and unit/
integration-tested in isolation. See [Crate split](#crate-split-skald-relay-client).
- Shared crypto + protobuf: `crates/skald-relay-common` (byte-for-byte interop
with the reference vectors in [relay/test-vectors.md](../relay/test-vectors.md))
- **Protocol documentation** (canonical, in English):
- [relay/index.md](../relay/index.md) — architecture, actors, threat model
- [relay/relay-protocol.md](../relay/relay-protocol.md) — protobuf schema, auth, pairing, live channel, presence
- [relay/framing.md](../relay/framing.md) — E2E plaintext framing (version + compression)
- [relay/payloads.md](../relay/payloads.md) — JSON payload schemas (inbox_request, inbox_update, …)
- [relay/crypto.md](../relay/crypto.md) — crypto contract, key derivation, AEAD, anti-replay
## What it does
---
Bridges the assistant's Inbox — pending approvals, clarification questions, and MCP elicitations — to a companion mobile app on the user's phone, end-to-end encrypted so even the relay server can't read the content. The phone can also show the full web app UI over the same encrypted tunnel, without any port-forwarding or the server being reachable from the internet.
## Crate split: skald-relay-client
Unlike most plugins, per-user access here is **not** the usual grant checklist — it's the device↔user binding itself (see pairing below), so this plugin doesn't show the normal "user access" list in the admin UI.
The plugin is the **application** layer; all networking lives in the standalone
`skald-relay-client` crate. The boundary is **payload-agnostic**: the client
exchanges opaque decrypted `Vec<u8>` payloads keyed by device pubkey and emits
inbound traffic as `RelayEvent`s; it never interprets the JSON. The plugin
consumes `client.events()`, applies the JSON schemas (`payloads.rs`) and the
`InboxApi`, and calls `client.send(dest, bytes, live)`.
## Requirements
Consequences of the split:
- A relay server URL (`wss://…`) to connect through.
- The companion mobile app installed on the user's phone.
- **Authorization policy stays in the plugin.** On `RelayEvent::ClientPaired`,
the client has already derived the `aes_key`, persisted the device as Pending,
and consumed the pairing token. The plugin's event loop decides: if
`require_device_confirmation` it notifies; otherwise it calls
`client.authorize(ed)` and then `broadcast_inbox()`.
- **`client.authorize()` is payload-agnostic** — it does NOT push an Inbox
snapshot. The plugin sends the snapshot after authorizing (both the auto path
and the `RelayAgent::authorize_client` tool path).
- **v2 framing (`compress/decompress_payload`) is transport** — handled inside
the client, so the plugin only ever sees clean JSON.
- **Identity seed** is injected via `SeedSource::Path("data/relay/seed")` (same
relative path as before) so existing identities/devices survive the upgrade.
## Enabling & configuring (admin)
### Module map — `skald-relay-client` (networking)
1. Plugin catalog → **Mobile Connector** → enable, then **Configure**.
2. Fields:
- **`relay_url`** (required) — the relay server's WebSocket URL.
- **`pairing_ttl`** (default `300`, max `600`) — seconds a pairing QR code stays valid.
- **`require_device_confirmation`** (default `true`, recommended) — a newly paired device stays "pending" until an admin explicitly authorizes it; don't turn this off without a good reason.
- **`notify_delay_secs`** (default `20`) — grace period before pushing an approval/question to the phone, so answering on the computer first skips the redundant phone notification. `0` = push immediately. (Elicitations are always pushed immediately regardless of this setting.)
| Module | Role |
|---|---|
| `config.rs` | `RelayClientConfig` + `SeedSource` (`Bytes` / `Path`) |
| `events.rs` | `RelayEvent` (`Connected`/`Disconnected`/`Message`/`ClientPaired`/`ClientRevoked`), broadcast |
| `identity.rs` | Seed load/generate (`0600`, injected path) + derived Ed25519/X25519 keys + `namespace_id` |
| `db.rs` | `relay_clients` table — devices + anti-replay counters (atomic counter helpers, `delete_all`) |
| `pairing.rs` | In-memory single-window pairing sessions (`code → session`) + `QrCodeData` |
| `state.rs` | Networking-only runtime: per-client `aes_key` cache, seal/open, counters, emits events |
| `ws.rs` | Permanent reconnecting agent WebSocket (v2 binary transport). Challenge → `Auth` → role dispatch → forward loop |
| `client.rs` | `RelayClient` — the public façade (`new`/`start`/`shutdown`/`send`/pairing/authorize/revoke/`clear_all`/`events`) |
## Pairing a device (admin-mediated)
### Module map — `plugin-mobile-connector` (application)
This is intentionally **not** self-service, unlike Telegram:
| Module | Role |
|---|---|
| `payloads.rs` | E2E JSON payload schemas (`inbox_update`, `notification`, client responses incl. `inbox_request`). Zlib-compressible per v2 framing.md |
| `app.rs` | `RelayApp`: Inbox dispatch (`broadcast_inbox`/`apply_client_payload`), authorization policy, the `events()` consumer loop |
| `notifier.rs` | `DelayedNotifier`: debounces phone pushes (`notify_delay_secs`) so resolving on the computer suppresses the push. See [Delayed push](#delayed-push) |
| `proxy.rs` | Accepts relay **pipes** of `stream_type = "http-local-proxy"` and reverse-proxies each to `127.0.0.1:<web_port>`. See [HTTP reverse proxy](#http-reverse-proxy-http-local-proxy) |
| `router.rs` | The QR-code HTTP endpoint (`/pairingqrcode`), resolves the current `RelayApp` → `client.lookup_pairing` |
| `agent.rs` | `RelayAgent` control trait (pairing, list, authorize, revoke) |
| `tools.rs` | The three LLM tools, registered in the main crate's `ToolRegistry` |
| `lib.rs` | `MobileConnectorPlugin` (`Plugin` + `RelayAgent`), lifecycle, bus subscriber, `RelayClient`/`RelayApp` wiring |
1. Admin opens **Pair a device** (sidebar, admin-only — `#plugin/mobile-connector/pairing`), which shows a QR code.
2. The user scans it from the mobile app.
3. The new device appears as "pending" on the **Mobile devices** page (`#plugin/mobile-connector/devices`).
4. The admin picks which user account to bind it to and confirms — only then can that device see that user's Inbox.
---
## Notes
## Configuration
Stored in the `plugins` table (JSON, edited via the plugin UI / `configure_plugin`):
```yaml
relay_url: "wss://relay.skaldagent.net/v1/ws" # empty ⇒ plugin idle (no WS)
pairing_ttl: 300 # seconds, max 600
require_device_confirmation: true # manual confirm new devices (recommended)
notify_delay_secs: 20 # debounce before pushing to the phone (0 = immediate)
```
`enabled` (the standard plugin flag) starts/stops the runloop.
`notify_delay_secs` debounces the **phone push** for a new approval/clarification
(see [Delayed push](#delayed-push)). The mobile push is only useful when you're
away from the computer; if you answer at the computer within the window, no phone
notification is sent. Set `0` to push immediately.
---
## Persistence (plugin.md §9)
| Data | Location | Why |
|---|---|---|
| `seed` (32 B) | filesystem `data/relay/seed`, `0600` | the only persistent secret; keys + `namespace_id` are derived at runtime |
| Pairing session | **in-memory** only | transient (≤ TTL); lost on restart ⇒ just re-pair |
| Devices + `send/recv_counter` | DB `relay_clients` | **must** survive restarts |
### Why counters live in the DB
Skald self-restarts by design. If counters reset to 0 on restart:
- `send_counter → 0` reuses an AES-GCM nonce under the same key (breaks
confidentiality + integrity for that device).
- `recv_counter → 0` re-opens the replay window.
So `send_counter` is incremented **and persisted before** sealing/sending
(`db::next_send_counter`, a transaction), and `recv_counter` is persisted only
**after** a valid `open`.
### `aes_key` cache
The per-client AES-256-GCM key is `HKDF(X25519(seed_x_priv, client_x_pub))`. It
is derived once and cached in memory (`HashMap<ed25519_pub, aes_key>` in
`RelayState`), never persisted; on a cache miss it is re-derived from the
client's stored `x25519_pub`. The cache entry is dropped on revoke.
---
## Pairing flow
1. The agent calls `mobile_start_pairing(ttl?)` (gated behind approval).
2. The plugin generates a 32-byte `pairing_token` (CSPRNG), sends
`pairing_start{token, ttl}` to the relay, and registers an in-memory session
keyed by a separate random `code` (latest-wins: any prior active session is
marked *Superseded*). It returns the URL
`/api/plugin/mobile-connector/pairingqrcode?code=<code>`.
3. The copilot renders the URL as an image. The endpoint serves a PNG of the QR
while the session is **Active**, else a placeholder (`QR scaduto` /
`QR già usato`). The QR payload is the normative `QrCodeData` JSON (never on
disk, never in the URL).
4. The client scans, connects as `role:"pairing"`, the relay consumes the token
and forwards `client_paired` to the agent.
5. On `client_paired`: derive + cache `aes_key`, persist the client as
**Pending** (counters 0), mark the session **Consumed**, then apply the
policy:
- `require_device_confirmation = false` ⇒ auto-authorize.
- `require_device_confirmation = true` ⇒ leave Pending; the human authorizes
via the control surface (a `notification` is pushed to existing devices).
`authorize` always reflects the full local set (replacement semantics): adding a
device sends the complete list including it; revoking sends it without.
---
## Message flows
- **Inbox → clients:** the bus subscriber reacts to the six Inbox events
(`approval_requested`, `approval_resolved`, `clarification_requested`,
`clarification_resolved`, `elicitation_requested`, `elicitation_resolved`) and
routes them through the **debouncer** (see
[Delayed push](#delayed-push)) before building an `InboxSnapshot` via
`inbox.list_pending()` and sending a sealed `inbox_update` to every Authorized
client. Each approval
carries a humanised `summary` (from `Tool::describe(Short)`, computed in
`Inbox::list_pending`) for the card/notification plus the raw `arguments`
(untruncated) for the detail dialog — so the user sees the full `execute_cmd`
command, not a truncated label. Each clarification carries its
`suggested_answers`. Each elicitation carries **only** its prompt metadata
(`server_name`, `message`, `field_name`, `sensitive`, `is_confirmation`) — never
a value; the value is supplied by the device in `elicitation_response.content`.
- **Clients → Inbox:** inbound `message` is checked (`from` ∈ Authorized,
nonce direction + counter > `recv_counter`), opened, and dispatched by `kind`:
`approval_response` → `inbox.approve/reject`, `clarification_response` →
`inbox.answer`, `elicitation_response` → `inbox.resolve_elicitation` (its
`content` may be a secret — never logged/persisted in clear), `hello` → persist
`device_info`, `inbox_request` → send a **targeted** `inbox_update` back to
`from` only (see below), `logout` → revoke.
After any response the Inbox is re-snapshotted. `request_id` is mapped
`string ↔ i64` (non-parsing ids are dropped). Inbox ops are idempotent by
`request_id`.
- **Reconnect snapshot (`inbox_request`):** the relay does **not** notify the
agent when a client reconnects, so the client sends `inbox_request` on the
**live channel** (`Message.live=true`) after every `auth_ok` (e.g. when the app
is opened from a push). The agent replies with an `inbox_update` sealed to the
requester only — not a broadcast — so other devices are not needlessly
re-aligned. A pull of stale state is useless, so the live channel is correct:
if the agent is offline, the client gets `PeerOffline` immediately instead of
waiting. Side-effect-free and idempotent (by `request_id`). See
`data/ios-app/v2/relay-protocol.md` §3.1.
---
## HTTP reverse proxy (`http-local-proxy`)
So a remote device can reach Skald's web UI **without** a VPN/Tailscale or an open
port, the plugin reuses the relay **pipe** (relayed E2E byte-stream, see
[relay/pipe.md](../relay/pipe.md)) as a reverse proxy to the local HTTP server.
`proxy.rs` subscribes to `RelayClient::incoming_pipes()` and, for each invite with
`stream_type == "http-local-proxy"`, accepts the pipe and splices it byte-for-byte
to a **fresh** `TcpStream` to `127.0.0.1:<web_port>` (`PluginContext::web_port`).
Per pipe it `split`s the connection into independent send/receive halves and runs
each direction in its own task (full-duplex, so neither blocks the other;
`PipeSender::send` is backpressured by the pipe's ~10 MiB send buffer, and
`recv`/`read` are cancel-safe — see [relay/pipe.md §6.1](../relay/pipe.md#61-full-duplex--client-side-backpressure)).
When either direction ends it cancels a shared token so the other unwinds. Invites
of other `stream_type`s are **ignored** (not rejected) since `incoming_pipes` is a
broadcast shared with possible future consumers.
The native app side (later) opens one pipe per outbound connection and points a
WebView at it; because the tunnel is a transparent TCP splice, HTTP/1.1 keep-alive,
parallel connections, and the chat WebSocket upgrade all work unchanged.
**Security.**
- The destination is **pinned** to `127.0.0.1:<web_port>` — the client cannot pick
host/port, so this is not an open localhost proxy (no SSRF to other local services).
- Access is gated by the relay's pipe auth (`pipe.md §3.1`): only the namespace
agent or an **authorized** client can establish a pipe.
- It exposes the full local web UI remotely — that is the intent; pair/authorize
devices accordingly.
**Relay tuning** (env, `pipe.md §3.3`): a browser opens several connections, so
`RELAY_PIPE_MAX_PER_NS` (default 8) may need raising; an idle chat-WS pipe can be
reaped at `RELAY_PIPE_IDLE_TIMEOUT_SECS` (120 s) — the frontend auto-reconnects.
Teardown: `proxy_one` takes a child of the plugin cancel token, so plugin stop
closes active tunnels; `stop_inner` also `shutdown()`s the relay client.
---
## Delayed push
A phone push is only valuable when the user is *away* from the computer. When
they're at the chat, every approval/clarification would otherwise fire an
instant — and pointless — notification, since they answer on the computer within
seconds. `DelayedNotifier` (`notifier.rs`) debounces this between the bus events
and `broadcast_inbox()`. (Elicitations are not chat-inline, so they are exempt —
see below.)
- **`*_requested`** arms a timer (`notify_delay_secs`, default 20s) keyed by
`(kind, request_id)` — approvals, clarifications, and elicitations use
independent id counters, so the kind is part of the key. If the timer elapses unresolved, the
key is marked *notified* and the Inbox is pushed (`broadcast_inbox`, `live=false`
→ store-and-forward / offline push).
- **Elicitations are the exception**: they live *only* in the Inbox (never
inline in the chat, unlike approvals/clarifications), so there is no
computer-side answer to debounce against. They skip the timer and are pushed
**immediately**, regardless of `notify_delay_secs`.
- **`*_resolved`** before the timer fires ⇒ the timer is cancelled and **nothing
is sent**. If the push already went out, the resolution is broadcast so the
phone clears the item. (Untracked ids fall back to a broadcast for snapshot
freshness.)
This only affects the **phone**: the desktop/web approval UI runs over the
per-session WebSocket (`ApprovalRequired`/`AgentQuestion`) and is never delayed.
Phone-driven responses (`apply_client_payload`) still `broadcast_inbox()`
immediately; the subsequent `*_resolved` bus event is handled idempotently. Armed
timers are cancelled on plugin stop (`cancel_all`). Set `notify_delay_secs: 0`
for the previous instant-push behaviour.
---
## LLM tools (plugin.md §11)
| Tool | Effect | Approval |
|---|---|---|
| `mobile_start_pairing(ttl?)` | Open the pairing window, return the QR URL | **Gated** (a default `require` rule is seeded, like `execute_cmd`/`restart`) |
| `mobile_list_devices()` | List devices (state, platform, device_info, last_seen) | read-only |
| `mobile_revoke_device(pubkey)` | Revoke a device by hex ed25519 pubkey | `Config` category |
These tools are not contributed through the `Plugin` trait (which has no
`tools()` method). They are registered in `Tools::build` (`src/core/skald/bundles.rs`):
the plugin is fetched via `get_plugin_typed::<MobileConnectorPlugin>()`, cast to
`Arc<dyn RelayAgent>`, and bound into the tools via
`plugin_mobile_connector::mobile_tools(agent)` → `ToolRegistry::register_arc`.
`mobile_start_pairing`'s approval gate is the default rule seeded in
`ApprovalManager::seed_defaults` (`src/core/approval/mod.rs`): opening a window
emits a secret (the QR) into chat, so it must be a deliberate human action, not
LLM-triggerable via prompt injection.
---
## HTTP endpoint
`GET /api/plugin/mobile-connector/pairingqrcode?code=<random>` — runtime PNG of
the QR (or placeholder), behind Skald's normal auth. Mounted by `WebFrontend`
via `Plugin::http_router()` (the router closes over the live `RelayState`). The
`code` is a non-enumerable capability; a URL leaked into `chat_history`
self-revokes once the window closes.
- A device stays bound until an admin revokes it from the Mobile devices page.
- If a user asks "why isn't my phone getting notifications", check: the plugin is enabled, their device is bound (not still pending), and — if it's not urgent — that they're not just inside the `notify_delay_secs` grace window.
docs/plugins/orpheus_tts_3b.md
+31
View File
@@ -0,0 +1,31 @@
# Orpheus TTS 3B
- **Plugin id:** `orpheus_tts_3b`
- **Category:** Text-to-speech, local
- **Runs:** on this machine, as a Python subprocess this plugin manages — needs a real GPU for reasonable speed
## What it does
Expressive, high-quality local text-to-speech using the 3-billion-parameter Orpheus model. Heavier than Kokoro TTS, but supports inline emotion tags for much more natural-sounding speech: `<laugh>`, `<chuckle>`, `<sigh>`, `<cough>`, `<sniffle>`, `<groan>`, `<yawn>`, `<gasp>` — placed directly in the text where the effect should occur, e.g. *"He showed up at noon. \<chuckle\> Classic."*
The model is gated on HuggingFace, so it requires a personal access token before it can download.
## Requirements
- A HuggingFace account and access token: create one at <https://huggingface.co/settings/tokens>. It must be stored as a secret named `HUGGINGFACE_TOKEN` — this is **not** a plugin config field. If you (the assistant) are helping set this up, activate the config tools and call `set_secret("HUGGINGFACE_TOKEN", "hf_...")` with the token the user gives you; do not ask them to find a settings page for it. The plugin refuses to start without it.
- GPU VRAM, depending on the quantization level chosen: roughly 7 GB (`none`/fp16), 4 GB (`int8`, the default), or 2.5 GB (`int4`).
- `python3` on the host; the model itself downloads automatically from HuggingFace on first run and is cached locally.
## Enabling & configuring (admin)
1. Get a HuggingFace token and store it as the secret `HUGGINGFACE_TOKEN` (see above).
2. Plugin catalog → **Orpheus TTS 3B** → enable, then **Configure**.
3. Fields:
- **`quantization`** (`none` | `int8` | `int4`, default `int8`) — lower precision uses less VRAM at some quality cost.
- **`voice`** (`tara` | `dan` | `leah` | `zac` | `zoe` | `mia` | `julia` | `leo`, default `tara`).
4. No separate "add provider" step — it appears directly in the Models hub's TTS section once running.
## Notes
- Meaningfully heavier than Kokoro TTS: only worth choosing when the machine has a real GPU and the extra expressiveness (emotion tags) matters.
- If the plugin won't start, the most common cause is a missing or invalid `HUGGINGFACE_TOKEN` secret, or the model being gated and requiring the HuggingFace account to accept its license on the model page first.
docs/plugins/remote.md
-163
View File
@@ -1,163 +0,0 @@
# Remote Connectivity
Exposes the Skald web app on a private mesh network so remote clients (iOS app, NAS, etc.) can connect without port forwarding or internet exposure.
---
## Architecture
```
core-api plugin-tailscale-remote crate
───────────────────────────── ──────────────────────────────────────────────
trait RemoteAccess ←── RemotePlugin (src/lib.rs)
(core_api::remote) TailscaleSystemProvider (src/tailscale_sys.rs) ← default
TailscaleEmbeddedProvider (src/tailscale.rs)
[feature: remote-tailscale]
```
- **`RemoteAccess` trait** (`core_api::remote`): vendor-agnostic interface. The core (`Skald`, WS handler, etc.) only knows this trait.
- **`TailscaleSystemProvider`** (`crates/plugin-tailscale-remote/src/tailscale_sys.rs`): **recommended provider**. Reads the mesh IP via `tailscale ip -4` (requires `tailscaled` running on the host). Binds a standard `tokio::net::TcpListener` — no experimental dependencies.
- **`TailscaleEmbeddedProvider`** (`crates/plugin-tailscale-remote/src/tailscale.rs`): embedded alternative using `tailscale-rs`. Feature-gated: `remote-tailscale`, **off by default** (enable via the root crate's `embedded-tailscale` feature). No system daemon required, but currently pre-1.0 with known DERP/reconnect issues, and it drags the `aws-lc-rs` C crypto build back into the tree (see [Feature Flag](#feature-flag)). Use only when a daemon cannot be installed (e.g. unrooted NAS).
- **`RemotePlugin`** (`crates/plugin-tailscale-remote/src/lib.rs`): wires the provider into the plugin lifecycle. Spawns a second Axum server on the mesh interface using the same router as the local server.
### `RemoteAccess` trait
```rust
pub trait RemoteAccess: Send + Sync {
fn provider_name(&self) -> &str;
async fn device_ip(&self) -> Result<Ipv4Addr>;
fn is_connected(&self) -> bool;
async fn shutdown(&self);
}
```
Stored in `Skald::remote: Arc<RwLock<Option<Arc<dyn RemoteAccess>>>>`. `None` when the plugin is disabled.
### Dual-bind strategy
The plugin (not the `WebServer`) is responsible for the mesh-facing server:
1. `RemotePlugin::start(state)` calls `extract_deps(state)` once, storing three named fields:
- `port: u16` — TCP port to bind on the mesh interface
- `remote_slot: Arc<RwLock<Option<Arc<dyn RemoteAccess>>>>` — slot in `Skald` to register the active provider
- `router_factory: Arc<dyn Fn() -> Router + Send + Sync>` — closure that rebuilds the Axum router
2. The internal helpers (`start_tailscale_sys`, `start_tailscale`) use only those three fields — no `Arc<Skald>` reference after extraction.
3. `start_tailscale_sys` binds `tokio::net::TcpListener::bind((mesh_ip, port))`.
4. `start_tailscale` calls `provider.axum_listener(port)` → `tailscale::axum::Listener`.
5. Both call `router_factory()` to get a fresh router and spawn `axum::serve(listener, router)` guarded by a `CancellationToken`.
`extract_deps` uses `std::sync::OnceLock` — idempotent across `reload()` calls. The values are stable for the lifetime of the process (port and static dir come from config, the remote slot is the same `Arc`).
The local server on `127.0.0.1:PORT` is unaffected.
---
## Configuration
Config is stored in the **`plugins` SQLite table** (not in `config.yml`).
| Key | Type | Default | Description |
|---|---|---|---|
| `provider` | string | `"tailscale_sys"` | `tailscale_sys` (system daemon, recommended) or `tailscale` (embedded, no daemon). |
| `auth_key` | string | — | Tailscale auth key (`tskey-auth-...`). Only required for the `tailscale` embedded provider on first join. |
| `hostname` | string | `"personal-agent"` | Hostname on the tailnet. Only used by the embedded `tailscale` provider. |
| `key_file` | string | `"data/tailscale_keys.json"` | Path for persisting node identity between restarts. Only used by the embedded `tailscale` provider. |
Config is persisted automatically and survives restarts.
---
## Agent Workflow
### Using the system Tailscale daemon (recommended)
Requires Tailscale already installed and logged in on the host machine.
```
1. toggle_item(kind="plugin", id="remote_connectivity", enabled=true)
2. restart
→ On next boot: plugin reads IP from tailscale ip -4, mesh server starts, app reachable at <ts-ip>:3000
```
No auth key needed — the system daemon is already authenticated.
### Using the embedded provider (no daemon)
Requires a binary **built with `--features embedded-tailscale`** (off by default;
see [Feature Flag](#feature-flag)). Without it, selecting `provider="tailscale"`
fails at start with `unknown provider 'tailscale'`.
```
1. configure_plugin "remote_connectivity" {"provider":"tailscale","auth_key":"tskey-auth-...","hostname":"personal-agent"}
2. toggle_item(kind="plugin", id="remote_connectivity", enabled=true)
3. restart
→ On next boot: embedded tailscale connects, mesh server starts, app reachable at <ts-ip>:3000
```
After first setup, the plugin auto-starts on every boot (persisted `enabled=true` in DB).
To check status: `list_items` (type=plugins) → look for `remote_connectivity`, check `running` and `runtime_status.ip`.
---
## Data Streams (iOS → Server)
Remote clients can push typed data over the existing WebSocket connection:
```json
{"type": "data", "stream": "location", "payload": {"lat": 45.1, "lng": 9.2, "accuracy": 10.0}}
```
Handled in `src/frontend/api/ws.rs` → `handle_data_msg()`. Dispatched to:
| Stream | Handler | Notes |
|---|---|---|
| `location` | `state.location_manager.update("remote", ...)` | Stores in-memory; existing `latest()` / `all()` queries work |
| other | `warn!` log | Reserved for future streams (health, etc.) |
---
## Limitations
### `tailscale_sys` (system daemon)
- **Requires `tailscaled` on the host**: if the daemon is not running or the device is not logged in, `start()` fails immediately with a clear error.
- **IP can change**: if the device rejoins the tailnet with a different IP, the plugin must be restarted to pick up the new address. The server binds to a specific IP, not `0.0.0.0`.
### `tailscale` (embedded, tailscale-rs v0.3)
- **DERP-relay only**: direct WireGuard holepunching is not yet implemented (issue #151). Latency is slightly higher than native WireGuard.
- **Known reconnect bugs**: after a node restart, existing connections can hang for ~15 s (issue #11). DERP connectivity can be lost after a control plane reconnect (issue #26).
- **Unaudited cryptography**: do not use for highly sensitive data until an official audit is complete.
- **Breaking changes**: the library is pre-1.0. The `TS_RS_EXPERIMENT=this_is_unstable_software` env var is required at runtime (set by `run.sh`).
- **Auth key expiry**: auth keys expire. Regenerate and call `configure_plugin` with the new value if the plugin fails to connect.
---
## Feature Flag
`tailscale-rs` is compiled only when the `remote-tailscale` feature is active.
It is **off by default**: the crate pulls the pure-Rust `tailscale` library, which
internally forces the `aws-lc-rs` crypto backend (a cmake/NASM C build) back into
the whole workspace, defeating the ring-only static-binary story
(see [build-and-distribution.md](../build-and-distribution.md)). The recommended
`tailscale_sys` provider needs none of this and is always compiled.
```toml
# crates/plugin-tailscale-remote/Cargo.toml
[features]
default = [] # embedded provider OFF by default
remote-tailscale = ["dep:tailscale"]
# root Cargo.toml — opt in from the top-level build:
[features]
embedded-tailscale = ["plugin-tailscale-remote/remote-tailscale"]
```
To build **with** the embedded provider (accepts the aws-lc-rs C build):
```sh
cargo build --features embedded-tailscale
```
The default build already omits it — no `--no-default-features` needed.
docs/plugins/remote_connectivity.md
+32
View File
@@ -0,0 +1,32 @@
# Remote Connectivity (Tailscale)
- **Plugin id:** `remote_connectivity` (package name `plugin-tailscale-remote` — the id differs from the package name)
- **Category:** Remote access
- **Runs:** on this machine; joins a [Tailscale](https://tailscale.com) mesh network
## What it does
Makes the web app reachable from other devices over a Tailscale mesh VPN — e.g. from a phone or laptop away from home — **without** port-forwarding the router or exposing anything to the public internet. Once running, the app is reachable at this device's Tailscale IP from any other device on the same tailnet.
This is a different, simpler style of remote access than the Mobile Connector plugin (which uses an external relay and works even without Tailscale installed anywhere).
## Requirements
Depends on which provider is chosen:
- **`tailscale_sys`** (default, recommended) — the host machine must already have the official Tailscale app/daemon installed and logged in. This plugin just reuses it.
- **`tailscale`** (experimental, embedded) — no separate install needed, but requires a Tailscale auth key generated from the Tailscale admin console.
## Enabling & configuring (admin)
1. Plugin catalog → **Remote Connectivity** → enable, then **Configure**.
2. Fields:
- **`provider`** (`tailscale_sys` | `tailscale`, default `tailscale_sys`) — see requirements above.
- **`auth_key`** — only for the embedded `tailscale` provider; a Tailscale auth key (`tskey-auth-…`), needed on first join.
- **`hostname`** (default `personal-agent`) — only for the embedded provider; the name this node requests on the tailnet.
- **`key_file`** (default `data/tailscale_keys.json`) — only for the embedded provider; where its node identity is persisted between restarts.
## Notes
- `tailscale_sys` is the recommended path for anyone who already uses Tailscale on their network — it's simpler and more reliable than the embedded mode.
- The embedded `tailscale` provider is explicitly marked experimental; prefer `tailscale_sys` unless there's a specific reason (e.g. not wanting a separate Tailscale install on the host) to use it.
docs/plugins/telegram.md
+22 -220
View File
@@ -1,234 +1,36 @@
# Telegram Plugin
# Telegram Bot
A private Telegram bot that forwards messages to the LLM and supports Human-in-the-Loop approvals via inline keyboard buttons.
- **Plugin id:** `telegram`
- **Category:** Messaging channel
- **Runs:** connects out to the Telegram Bot API — needs internet access
---
## What it does
## Setup
Connects a private Telegram bot to this instance, so a user can chat with their assistant directly from Telegram — same assistant, same memory, same tools as the web app. Pending approvals (e.g. "can I run this command?") are shown as Telegram messages with inline buttons the user can tap to approve or deny, right from their phone.
1. Create a bot with [@BotFather](https://t.me/BotFather) and copy the token.
2. Add to `config.yml`:
One bot serves everyone on the instance; each person pairs their **own** Telegram chat to their **own** account — nobody sees anyone else's conversation.
```yaml
plugins:
telegram:
token: "123456789:AABBCC..."
```
## Requirements
3. Restart the app. The bot starts automatically if the token is present.
- A Telegram bot token. Get one by messaging **@BotFather** on Telegram, sending `/newbot`, and following the prompts — it replies with a token that looks like `123456789:AAF...`.
---
## Enabling & configuring (admin)
## Pairing — how to authorize a user
1. Plugin catalog → **Telegram Bot** → enable, then **Configure**.
2. Field:
- **`token`** (required) — the bot token from BotFather. Stored as a secret field, not shown again after saving.
Access control is managed entirely through the file `secrets/telegram_whitelist.json`.
## Per-user pairing (self-service)
### File format
Once the bot is enabled and a user has been granted access to the plugin:
```json
{
"whitelist": [123456789],
"pending_pairings": [
{
"code": "A3KX7P",
"chat_id": 987654321,
"issued_at": "2026-05-19T10:30:00+02:00"
}
]
}
```
1. The user opens Telegram, finds the bot (by the username chosen in BotFather), and sends it any message.
2. The bot replies with a short pairing code.
3. The user goes to their own Plugins page in the web app, finds Telegram, and pastes the code into the **pairing code** field.
- `whitelist` — array of authorized `chat_id` values (integers). Users in this array can send messages to the agent.
- `pending_pairings` — users who have contacted the bot but are not yet authorized. Each entry has a `code` shown to the user in Telegram chat, their `chat_id`, and the `issued_at` timestamp. Entries older than **24 hours** are pruned automatically the next time an unauthorized user contacts the bot, so abandoned codes do not pile up.
That's the whole flow — no admin involvement needed for a normal pairing. (An admin *can* alternatively bind a chat to a user directly using the `telegram_pairing` tool from the assistant, e.g. if a user can't access the web app.)
### Pairing flow
## Notes
1. An unknown user sends any message to the bot.
2. The bot replies with a 6-character pairing code and writes an entry to `pending_pairings` in `secrets/telegram_whitelist.json`.
3. The user communicates the code to you (e.g., through a separate channel).
4. You ask the agent: *"Telegram pairing code A3KX7P — authorize it"*.
5. The agent reads `secrets/telegram_whitelist.json`, finds the entry with `code: "A3KX7P"`, moves the `chat_id` from `pending_pairings` to `whitelist`, and writes the file back.
6. Within 10 seconds the plugin's watchdog detects the file change, logs the event, and **sends a welcome message** to the newly authorized user on Telegram.
### To authorize manually (without asking the agent)
Use `edit_file` or `write_file` to move the `chat_id` from `pending_pairings` to `whitelist` in `secrets/telegram_whitelist.json`.
### To revoke access
Remove the `chat_id` from the `whitelist` array in `secrets/telegram_whitelist.json`. The change takes effect on the user's next message (whitelist is re-read on every message).
---
## Watchdog
The plugin polls `secrets/telegram_whitelist.json` every **10 seconds** for modification-time changes.
When it detects a change:
- Reloads the whitelist.
- Identifies any `chat_id` values newly added to `whitelist`.
- Sends each newly authorized user a welcome message on Telegram.
- Logs the event at INFO level.
This means there is no restart needed after editing the file — authorization takes effect automatically.
---
## Commands
| Command | Effect |
|---|---|---|
| `/new` or `/clear` | Create a new chat session (clears LLM context) |
| `/stop` | Interrupt the agent mid-turn (clears pending approvals and clarifications) |
| `/models` | List available LLM models ordered by priority (numbered `0..N`, index 0 is `auto`) |
| `/model <N\|name\|auto>` | Pin the model for this chat by index, name (substring allowed), or reset to `auto`. State is held in `ChatHub.selected_clients["telegram"]` and broadcast to all clients of the source via `ClientSelected`. Cleared on server restart |
| `/context` | Show last turn's token usage (`↑X tok · ↓Y tok`) |
| `/cost` | Show total spend for this session in USD (sync sub-agents included; async tasks excluded) |
| `/compact` | Force context compaction (bypasses the token threshold) |
| `/resettools` | Remove all activated tool groups (MCP servers + `config`) from the session |
| `/sethome` | Set Telegram as the home source for background notifications |
| `/help` | Show available commands |
| any other `/command` | Unknown — replies with a "Unknown command" notice + the help list, never forwarded to the LLM |
| any text | Forwarded to the LLM agent |
---
## Human-in-the-Loop Approvals
When the LLM triggers a tool that requires user approval (`execute_cmd`, `restart`, write-file tools outside `memory/`):
1. The bot sends a message with the operation details and a content preview.
2. Four inline keyboard buttons appear in two rows:
```text
[✅ Approve] [❌ Reject]
[⏱ 15 min] [🔄 Session]
```
3. Tapping a button resolves the pending approval and execution continues or is cancelled.
4. **⏱ 15 min** — approves and suppresses approval prompts for tools of the same category/MCP server for 15 minutes.
5. **🔄 Session** — approves and suppresses all approval prompts for the rest of the session.
6. The approval message is **deleted** once resolved, whether via Telegram or the web UI.
Bypass buttons call `ApprovalApi::approve_with_bypass` (scope auto-detected from the tool's category or MCP server). See [../approval/index.md](../approval/index.md) for bypass semantics.
---
## Output Formatting
Telegram's HTML parse mode supports only a limited tag set: `<b>` `<i>` `<u>` `<s>` `<code>` `<pre>` `<a>` `<blockquote>`. Structural elements (`<table>`, `<ul>`, `<li>`, `<div>`) are **not supported**.
The plugin injects a compact formatting context into every LLM session (`TELEGRAM_FORMAT_CONTEXT` in `mod.rs`) and a shorter tail reminder (`TELEGRAM_FORMAT_REMINDER`) instructing it to:
- Use Telegram HTML tags only.
- Never use Markdown (`**`, `*`, `` ` ``, `#`, `_`, `|`).
- Replace structured data (tables) with bullet lists (`•`).
| Element | Correct | Wrong |
| --------------- | --------------------- | ------------------ |
| Bold | `<b>text</b>` | `**text**` |
| Italic | `<i>text</i>` | `*text*` |
| Code | `<code>text</code>` | `` `text` `` |
| Code block | `<pre>text</pre>` | ` ```text``` ` |
| Structured data | bullet list `•` | `\| col \| col \|` |
Long responses are automatically split into chunks of ≤ 4000 characters via `send_long()` in `helpers.rs`.
### Markdown sanitizer (post-processing safety net)
Because LLMs occasionally emit Markdown despite instructions, `send_long` applies `sanitize_for_telegram()` on every HTML-mode send **before** chunking. This provides a reliable fallback independent of model compliance:
1. **Markdown tables → bullet lists** — detects `| col | col |` blocks, emits the header row as `<b>header — header</b>` and each data row as `• val — val`.
2. **`**bold**` → `<b>bold</b>`** — converts residual Markdown bold.
3. **`## Header` → `<b>Header</b>`** — converts residual Markdown headers.
### Fallback behavior
If the Telegram API rejects a chunk (e.g., due to malformed HTML), `send_long` retries **without** `ParseMode::Html`. Before retrying it strips all HTML tags (`<…>`) from the chunk using a regex so the user sees plain text rather than raw `<b>…</b>` markup.
---
## Voice (Speech Integration)
If the Speech plugin is configured and running, the Telegram plugin gains two additional capabilities:
### Incoming voice messages (STT)
When the user sends a voice note, the plugin:
1. Downloads the OGG audio from Telegram.
2. Passes it to `SpeechPlugin::transcribe()`.
3. Forwards the resulting text to the LLM as a normal message.
### Outgoing voice replies (TTS)
The LLM has access to a `send_voice_message(text)` tool. When it calls it, the plugin:
1. Passes the text to the active `TextToSpeech` synthesiser (`synthesize()`).
2. **Transcodes the audio to Ogg/Opus** (`to_ogg_opus` in `tools.rs`) — the only format Telegram renders as a playable voice message. The synthesiser's `output_format()` decides the input: `opus`/`ogg` pass through untouched; raw `pcm` (e.g. Gemini TTS) is decoded as 24 kHz/mono/s16le; every other container (mp3, wav, …) is auto-detected. Conversion runs `ffmpeg` over stdin/stdout pipes (no temp files).
3. Sends the resulting Ogg/Opus bytes back to the user as a Telegram voice message.
The LLM is instructed to use voice only for short, conversational replies with no code or complex formatting. The TTS engine's formatting guide (SSML-like tags) is also injected into the system context so the LLM can control pacing and emphasis.
> **Requires `ffmpeg`** on `PATH` for any non-Opus synthesiser. If it is missing, `send_voice_message` returns a clear error (`ffmpeg not available …`) and the LLM falls back to a text reply.
### Requirements
Both `plugins.speech.stt_model` and `plugins.speech.tts_model` must be set in `config.yml`. The Speech plugin must be enabled and running before the Telegram plugin starts.
---
## File & Media Attachments
The Telegram plugin downloads incoming attachments and forwards them to the conversation. The LLM sees them in timeline order — it knows which file was most recently sent without any special indexing.
| Type | Saved to disk | How it reaches the LLM |
| --- | --- | --- |
| Document (PDF, ZIP, …) | `data/uploads/telegram/<chat_id>/<filename>` | Structured `metadata.attachments` (shared with web/mobile) |
| Photo | `data/uploads/telegram/<chat_id>/<file_id>.jpg` | Structured `metadata.attachments` |
| Location | — | `[TELEGRAM SYSTEM INFO]` text (latitude, longitude, accuracy, Google Maps URL) |
**Document and Photo are aligned with the web/mobile attachment model**: `download_and_save`
returns a `core_api::message_meta::Attachment` (project-root-relative path so `/data/…` serves
it), `handle_attachment` puts it in `SendMessageOptions.metadata`, and the message builder
generates the shared `[SYSTEM INFO]` block on the fly. Viewing the `telegram` source from the
copilot therefore shows these as **chips**, not raw text. See
[frontend.md](../frontend.md#attachments) and [database.md](../database.md) (`chat_history.metadata`).
**Location** has no file, so it keeps the legacy `[TELEGRAM SYSTEM INFO]` text path
(`system_info_message`). Captions typed alongside a Document/Photo become the user turn's text.
### Live locations
When the user shares a live location, two things happen:
1. **Initial message** (`message` event) — the LLM is notified via a `[TELEGRAM SYSTEM INFO]` message and the position is written to `skald.location_manager` under the key `"telegram"`.
2. **Subsequent updates** (`edited_message` events) — the position in `location_manager` is updated silently, with no LLM notification. This keeps the store current for any background scripts or tools that read `user_location("telegram")`.
`LocationManager` is in-memory only. On restart, the store starts empty and is repopulated as soon as Telegram delivers the next live location tick (typically within seconds if sharing is still active).
The `uploads/` directory is gitignored.
### Extending attachment types
To add a new type (e.g. sticker, contact):
1. Add a variant to `TelegramAttachment` in `crates/plugin-telegram-bot/src/attachments.rs`.
2. Implement `download_and_save`: return `Ok(Some(Attachment))` for a file-backed type (it flows into `metadata.attachments`) or `Ok(None)` for a file-less one (then add a `system_info_message` arm and handle it in the `None` branch of `handle_attachment`).
3. Detect the message type in `classify_message` in `handlers.rs` and return `IncomingEvent::Attachment(...)`.
---
## Interface Tools
The Telegram plugin can inject custom LLM-callable tools into any session via the `interface_tools` parameter of `SendMessageOptions`. These tools are only visible to the root agent — sub-agents do not inherit them.
To add a Telegram-specific tool, construct an `InterfaceTool` with an OpenAI tool definition and an async handler closure that captures `Arc<Bot>` and `ChatId`, then pass it in the `interface_tools` vec inside `SendMessageOptions`.
`InterfaceTool` and `ToolFuture` are defined in `crates/core-api/src/interface_tool.rs` (re-exported via `crate::chat_hub`). `AgentRunConfig` remains in `src/core/session/handler/interface_tools.rs` (main crate only).
---
## Secrets directory
`secrets/telegram_whitelist.json` is gitignored. The directory is created automatically on first pairing request. Never commit this file.
- Output sent to Telegram is automatically constrained to Telegram-safe HTML formatting (bold, italic, code blocks, links, quotes) — no Markdown, no tables. This is handled automatically; nothing to configure.
- Revoking a user's plugin access immediately stops that person's Telegram chat from working, without needing them to re-pair if access is restored later.
docs/plugins/whisper-local.md
-123
View File
@@ -1,123 +0,0 @@
# WhisperLocal Plugin
Local Speech-to-Text via [whisper.cpp](https://github.com/ggerganov/whisper.cpp), Metal-accelerated on Apple Silicon.
Implemented in pure Rust using the `whisper-rs` crate — no Python involved.
---
## Setup
### 1. Download a GGML model
```sh
mkdir -p models
curl -L -o models/ggml-large-v3.bin \
https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-large-v3.bin
```
Other available sizes (smaller = faster, less accurate):
| Model | Size | Notes |
|---|---|---|
| `ggml-tiny.bin` | ~75 MB | Very fast, lower accuracy |
| `ggml-base.bin` | ~142 MB | Good balance for testing |
| `ggml-small.bin` | ~466 MB | Good accuracy |
| `ggml-medium.bin` | ~1.5 GB | High accuracy |
| `ggml-large-v3.bin` | ~3.1 GB | Best accuracy, recommended |
| `ggml-large-v3-turbo.bin` | ~1.6 GB | large-v3 speed-optimised |
All models: `https://huggingface.co/ggerganov/whisper.cpp`
### 2. Configure `config.yml`
```yaml
plugins:
whisper_local:
model: "models/ggml-large-v3.bin"
language: "it" # BCP-47 code, or "auto" for detection
load_at_startup: false # false (default) = lazy load on first use
idle_timeout_secs: 1200 # unload after 20 min idle; 0 = never unload
```
| Option | Default | Effect |
|---|---|---|
| `model` | — (required) | Path to the GGML `.bin` file |
| `language` | `auto` | BCP-47 code or `auto`. Applied live — runtime changes take effect on the next transcription without a reload |
| `load_at_startup` | `false` | **When** the model first loads: `false` = lazily on the first transcription, `true` = eagerly in `start()` (warm, no first-call latency) |
| `idle_timeout_secs` | `1200` | **When** the model unloads: after this many seconds of inactivity. `0` = never unload (stays resident once loaded) |
The two timing options are orthogonal and cover the whole spectrum:
| `load_at_startup` | `idle_timeout_secs` | Behaviour |
|---|---|---|
| `false` | `1200` | **Default** — load on first use, free ~2 GB after 20 min idle |
| `true` | `0` | Always resident — eager load, never unload (legacy behaviour) |
| `true` | `1200` | Warm at startup, but freed if unused |
| `false` | `0` | Load on first use, then stay resident |
### 3. Build
The first `cargo build` compiles whisper.cpp (a few minutes). Subsequent builds are cached.
---
## How it works
```
Telegram voice message (OGG/Opus)
│
▼ ffmpeg → 16 kHz mono WAV
▼ hound → Vec<f32> PCM samples
▼ whisper.cpp (Metal GPU) → text
│
▼ forwarded to LLM as a normal text message
```
Audio conversion uses the system `ffmpeg` binary (must be installed: `brew install ffmpeg`).
Inference runs on Apple Silicon GPU via Metal. Falls back to CPU if Metal is unavailable.
---
## Memory management (lazy load + idle unload)
The GGML weights are ~2 GB, so the plugin keeps them in memory only while they are
actually useful. The model lives in a shared, droppable cell (`LazyModel`):
- **`start()`** validates the model path and registers a lightweight transcriber, but
does **not** load the weights unless `load_at_startup: true`. The registered handle
holds no strong reference to the weights, so they can be freed at any time.
- **First transcription** triggers `ensure_loaded()`, which loads the weights once
(concurrent first-callers wait on a single load) and records a last-used timestamp.
- A background **eviction task** ticks every 60 s and unloads the model once it has
been idle for `idle_timeout_secs`. Set `idle_timeout_secs: 0` to disable eviction.
- **Unloading** is refcount-safe: an in-flight transcription holds its own handle to
the weights, so memory is reclaimed only after it finishes. The actual free runs on
a blocking thread (whisper.cpp GPU cleanup).
Trade-off: after an unload, the next transcription pays the reload cost (a few
seconds). The OS page cache usually keeps the `.bin` warm, so the reload is mostly
memory copy + Metal allocation rather than disk I/O. Use `load_at_startup: true` /
`idle_timeout_secs: 0` if you prefer zero first-call latency over reclaiming the RAM.
---
## Integration with TranscribeManager
`WhisperLocalPlugin` does **not** expose itself as `Arc<dyn Transcribe>` directly. At `start()` it registers a lightweight `WhisperLocalTranscriber` handle into `skald.transcribe_manager`; at `stop()` it deregisters it. Callers never reference the plugin type — they ask the manager:
```rust
if let Some(t) = skald.transcribe_manager.get().await {
let text = t.transcribe(audio, "ogg").await?;
}
```
See [../plugins.md](../plugins.md) for the `TranscribeManager` API and the `Transcribe` trait.
---
## When to Update This File
- The audio conversion pipeline changes
- Default recommended models change
- Registration/deregistration logic in `start()`/`stop()` changes
- The lazy-load / idle-eviction lifecycle or its config options change
docs/plugins/whisper_local.md
+37
View File
@@ -0,0 +1,37 @@
# Whisper Local
- **Plugin id:** `whisper_local`
- **Category:** Speech-to-text, local
- **Runs:** on this machine, in-process (via `whisper.cpp`, Metal-accelerated on Apple Silicon) — no cloud, no API key
## What it does
Transcribes voice messages entirely on-device using [whisper.cpp](https://github.com/ggerganov/whisper.cpp). Nothing leaves the machine — the right choice when privacy matters more than raw speed, or when there's no budget for a cloud transcription API.
The model (roughly 1–3 GB depending on size) is loaded into memory only when first needed ("lazy" loading) and unloaded again after a configurable idle period to free RAM — unless eager loading is turned on.
## Requirements
- A GGML `.bin` Whisper model file, downloaded manually onto the host filesystem — this plugin does not fetch it automatically. Example:
```
curl -L -o models/ggml-large-v3.bin \
https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-large-v3.bin
```
Smaller/faster models (e.g. `ggml-medium.bin`, `ggml-small.bin`) trade accuracy for speed and size — point at any of them.
- `ffmpeg` installed on the host (used to convert incoming audio to the 16 kHz mono format Whisper needs).
## Enabling & configuring (admin)
1. Download a model file first (see above) and note its path.
2. Plugin catalog → **Whisper Local** → enable, then **Configure**.
3. Fields:
- **`model`** (required) — path to the `.bin` file, e.g. `models/ggml-large-v3.bin`.
- **`language`** — a BCP-47 code (`it`, `en`, …) or `auto` for automatic detection (default `auto`).
- **`load_at_startup`** — load the model into memory as soon as the plugin starts, instead of on first use (default off).
- **`idle_timeout_secs`** — unload the model from memory after this many seconds of inactivity, `0` = never unload (default `1200`, 20 minutes).
## Notes
- The model occupies roughly 1–3 GB of RAM while loaded.
- The very first transcription after an idle unload is slower, since the model has to reload.
- No external service and no per-use cost — the trade-off is local CPU/GPU time and disk space for the model file.