First Version
This commit is contained in:
@@ -0,0 +1,209 @@
|
||||
# Honcho Memory Plugin
|
||||
|
||||
Plugin: `crates/plugin-honcho/src/lib.rs`
|
||||
HTTP client: `crates/honcho-client/` (separate workspace crate)
|
||||
|
||||
---
|
||||
|
||||
## Purpose
|
||||
|
||||
Streams completed chat turns to a [Honcho](https://honcho.dev) server so that it can extract long-term conclusions about the user (write path), and reads that context back into every LLM turn via the [`Memory`](memory.md) trait (read path).
|
||||
|
||||
---
|
||||
|
||||
## Self-hosted Docker package
|
||||
|
||||
A ready-to-run Docker Compose setup is in the [`honcho/`](../honcho/) folder at the project root.
|
||||
It starts four services: the Honcho API, the deriver background worker, PostgreSQL + pgvector, and Redis.
|
||||
|
||||
**Quick start:**
|
||||
|
||||
```sh
|
||||
cd honcho
|
||||
cp .env.example .env
|
||||
# Edit .env — set at least LLM_OPENAI_API_KEY=sk-...
|
||||
docker compose up -d
|
||||
# API available at http://localhost:8000
|
||||
```
|
||||
|
||||
Full instructions, LLM provider options (OpenAI, OpenRouter, Ollama), and troubleshooting are in [`honcho/README.md`](../honcho/README.md).
|
||||
|
||||
---
|
||||
|
||||
## Setup
|
||||
|
||||
1. Start the Honcho server (see above).
|
||||
2. Enable the plugin via the agent or REST API:
|
||||
```json
|
||||
PUT /api/plugins/honcho
|
||||
{
|
||||
"enabled": true,
|
||||
"config": {
|
||||
"base_url": "http://localhost:8000",
|
||||
"api_key": "",
|
||||
"workspace_id": "personal-agent"
|
||||
}
|
||||
}
|
||||
```
|
||||
Or ask the main agent: _"enable the honcho plugin"_.
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
Stored in the `plugins` SQLite table (`config` JSON blob). Managed at runtime — no entry in `config.yml`.
|
||||
|
||||
| Field | Type | Default | Description |
|
||||
| --- | --- | --- | --- |
|
||||
| `base_url` | string | `http://localhost:8000` | Honcho server URL |
|
||||
| `api_key` | string | _(empty)_ | API key; leave empty for local/unauthenticated instances |
|
||||
| `workspace_id` | string | `personal-agent` | Honcho workspace identifier for this agent instance |
|
||||
|
||||
---
|
||||
|
||||
## Honcho Object Model
|
||||
|
||||
```
|
||||
workspace (workspace_id from config — one per agent instance)
|
||||
├── peer "user" observe_others=true
|
||||
├── peer "assistant" observe_me=true
|
||||
└── session one per local chat_sessions.id
|
||||
├── message peer_id="user"
|
||||
├── message peer_id="assistant"
|
||||
└── …
|
||||
```
|
||||
|
||||
**Workspace and peers** are created (idempotently) each time the plugin starts. If they already exist, the API returns an error which is logged at `WARN`/`DEBUG` and ignored.
|
||||
|
||||
**Sessions** are created lazily on the first event for a new `chat_sessions.id`, then cached in memory for the life of the listener task. The Honcho session UUID is stored in the session cache but not persisted to SQLite — restarting the plugin creates new Honcho sessions for subsequent events.
|
||||
|
||||
---
|
||||
|
||||
## Event Filtering
|
||||
|
||||
An event is forwarded only when **all** of the following conditions hold:
|
||||
|
||||
| Condition | Reason |
|
||||
| --- | --- |
|
||||
| `is_interactive = true` | A real user is in the conversation |
|
||||
| `is_ephemeral = false` | Not a short-lived automated session (cron, tic) |
|
||||
| `is_synthetic = false` | Message content was typed by the user, not injected by the system |
|
||||
| `role` is `User` or `Assistant` | Sub-agent messages (`Agent` role) are skipped |
|
||||
| `content` is non-empty | Guard against empty strings |
|
||||
|
||||
---
|
||||
|
||||
## Lifecycle
|
||||
|
||||
1. **`start()`** — subscribes to `skald.event_bus`, calls `ensure_workspace_ready` (best-effort), then spawns the listener task.
|
||||
2. **Listener task** — `tokio::select!` loop on the bus receiver and a `CancellationToken`. On `RecvError::Lagged`, logs a warning and continues (some turns are missed but the task stays alive).
|
||||
3. **`stop()`** — cancels the token and awaits the task.
|
||||
4. **`reload()`** — follows the standard plugin pattern: start/stop/restart-on-change.
|
||||
|
||||
---
|
||||
|
||||
## Error Handling
|
||||
|
||||
All Honcho API errors are **fire-and-forget**: logged as `warn!` and never propagated to the session handler or the user. A Honcho outage has zero impact on chat functionality.
|
||||
|
||||
`HonchoError::Request`'s `Display` walks the full `source()` chain, so transport
|
||||
failures surface the real cause in logs (e.g. `Request failed: error sending
|
||||
request for url (...): Connection reset by peer`) instead of just reqwest's
|
||||
opaque top-line. This makes host↔container issues (e.g. a stale Docker Desktop
|
||||
port-forward after a container recreation) diagnosable from the `warn!` alone.
|
||||
|
||||
---
|
||||
|
||||
## Read Path
|
||||
|
||||
`HonchoMemory` implements the [`Memory`](../memory.md) trait. Before each LLM turn,
|
||||
`query_context` is called automatically by `ChatSessionHandler::handle_message` — for
|
||||
**all** session types: interactive, cron, and tic.
|
||||
|
||||
### Flow
|
||||
|
||||
1. Checks `is_available()` — returns `None` immediately if the plugin is stopped.
|
||||
2. Looks up the Honcho session UUID for the local `session_id` in the shared `session_map`.
|
||||
3. **If a mapping exists** (interactive session with at least one turn written):
|
||||
- Calls `client.session_context(workspace_id, honcho_session_id, tokens=2000, search_query=user_msg)`.
|
||||
- Returns the formatted result on success.
|
||||
- On error: logs `warn!` and falls through to the peer-context fallback **without** `search_query`
|
||||
(avoids a second embedding of the same user message — `session_context` already embedded it before failing).
|
||||
4. **Fallback — `peer_context("user")`** (no mapping, or session_context error):
|
||||
- Cold start / cron / tic (no `session_map` entry): calls with `search_query=user_msg` for relevance.
|
||||
- After a `session_context` failure: calls **without** `search_query` to avoid double-embedding.
|
||||
- Returns global user knowledge derived from all sessions Honcho has observed.
|
||||
- On error: logs `warn!` and returns `None`.
|
||||
|
||||
The formatted context is prepended to `extra_system_context` and injected into the system prompt. Errors are never propagated — they degrade gracefully to `None`.
|
||||
|
||||
### Context format
|
||||
|
||||
`format_context()` extracts, in priority order:
|
||||
|
||||
1. `conclusions[].content` → "Known facts about the user: …"
|
||||
2. `summary` → "Conversation summary: …"
|
||||
3. Fallback: pretty-printed raw JSON
|
||||
|
||||
The result is wrapped in `--- Honcho memory context --- / --- end ---` markers.
|
||||
|
||||
---
|
||||
|
||||
## LLM-callable Tools
|
||||
|
||||
`HonchoMemory::tools()` returns **five** tools whenever the plugin is active
|
||||
(`is_available()` true). They give the LLM direct, on-demand access to every
|
||||
layer of Honcho's API, complementing the automatic pre-turn `query_context`
|
||||
injection. All operate on the `user` peer and are inherited by sub-agents via
|
||||
`AgentRunConfig::memory_tools`.
|
||||
|
||||
The [official Honcho documentation](https://honcho.dev/docs/v3/documentation/features/chat) recommends exposing these as tools so the agent decides on its own when to read or write memory, rather than only relying on automatic injection.
|
||||
|
||||
| Tool | Endpoint | Cost | What it does |
|
||||
| --- | --- | --- | --- |
|
||||
| `memory_query` | `POST .../peers/user/chat` | High (LLM synthesis) | Natural-language question → synthesized answer (dialectic reasoning, `reasoning_level=low`) |
|
||||
| `honcho_search` | `GET .../peers/user/context?search_query=…` | Low | Semantic search over derived facts; returns raw ranked excerpts (with ids when present) |
|
||||
| `honcho_context` | `GET .../peers/user/context` | Low | Full context snapshot (conclusions + summary), no synthesis; optional focus `query` |
|
||||
| `honcho_profile` | `GET`/`PUT .../peers/user/card` | Low | Read the peer card, or overwrite it with a list of fact strings (`card`) |
|
||||
| `honcho_conclude` | `POST .../conclusions` / `DELETE .../conclusions/{id}` | Low | Write a new fact (`conclusion`) or delete one by id (`delete_id`); exactly one required |
|
||||
|
||||
**Peer model — all tools operate on the `user` peer as both observer and observed.**
|
||||
This plugin configures the `user` peer with `observe_me = true`, so the user's
|
||||
self-knowledge lives in the `observer = user / observed = user` slot. Therefore
|
||||
`honcho_conclude` writes with `observer_id = observed_id = user`, and `honcho_search`
|
||||
uses `peer_context` (not the `conclusions/query` endpoint, which requires explicit
|
||||
observer/observed filters) — the same proven path as the automatic read-path
|
||||
injection. This differs from setups where the assistant observes the user
|
||||
(`observer = assistant`); keeping observer = user is what lets the read-path see
|
||||
facts written by `honcho_conclude`.
|
||||
|
||||
**When to use vs. the automatic injection:**
|
||||
|
||||
| Mechanism | When it fires | Best for |
|
||||
| --- | --- | --- |
|
||||
| `query_context` (auto) | Before every LLM turn | Background context, cold-start facts |
|
||||
| `memory_query` (tool) | LLM calls it explicitly | On-demand deep reasoning mid-conversation |
|
||||
| `honcho_search` / `honcho_context` (tools) | LLM calls them explicitly | Cheap raw recall without LLM synthesis |
|
||||
| `honcho_profile` / `honcho_conclude` (tools) | LLM calls them explicitly | Actively curating long-term memory |
|
||||
|
||||
**Implementation note:** `Tool::execute` is synchronous but the Honcho calls are
|
||||
async. All five tools share the `run_blocking` helper, which uses
|
||||
`tokio::task::block_in_place` + `Handle::current().block_on(...)` to drive the
|
||||
future from within the Tokio multi-thread scheduler without spawning a new thread.
|
||||
|
||||
---
|
||||
|
||||
## Future Work
|
||||
|
||||
- **Session persistence** — store the Honcho session UUID in a new `chat_sessions.honcho_session_id` column so the mapping survives a plugin restart.
|
||||
|
||||
---
|
||||
|
||||
## When to Update This File
|
||||
|
||||
- Config fields change
|
||||
- Honcho object model or peer setup changes
|
||||
- Filtering rules change
|
||||
- `query_context` flow changes (session vs peer fallback logic)
|
||||
- Docker Compose setup in `honcho/` changes significantly
|
||||
- Public API of `crates/honcho-client/` changes
|
||||
@@ -0,0 +1,15 @@
|
||||
# Plugin System
|
||||
|
||||
Extensible plugin architecture: lifecycle, trait contract, built-in plugins.
|
||||
|
||||
## Files
|
||||
|
||||
- [../plugins.md](../plugins.md) — Plugin trait, PluginManager, HTTP router integration
|
||||
- **Built-in Plugins:**
|
||||
- [honcho.md](honcho.md) — Honcho long-term memory plugin: setup, config, filtering, lifecycle
|
||||
- [mobile-connector.md](mobile-connector.md) — Mobile app relay bridge, E2E encryption, Inbox sync (v2 protocol)
|
||||
- [telegram.md](telegram.md) — Telegram bot: setup, pairing, whitelist, HITL approval
|
||||
- [whisper-local.md](whisper-local.md) — Local STT via whisper.cpp (Metal-accelerated)
|
||||
- [remote.md](remote.md) — Tailscale mesh remote connectivity
|
||||
|
||||
See [../index.md#plugin-system](../index.md#plugin-system) for navigation.
|
||||
@@ -0,0 +1,288 @@
|
||||
# Mobile Connector Plugin (`mobile-connector`)
|
||||
|
||||
Bridges Skald's **Inbox** (approvals + clarifications + MCP elicitations) to mobile apps over the
|
||||
**relay**, implementing the **agent** role of the v2 relay protocol. The plugin is
|
||||
the namespace owner and the sole authority over authorized devices. Skald is
|
||||
never exposed on the internet: only this plugin connects out, and only to the
|
||||
relay.
|
||||
|
||||
- Crate: `crates/plugin-mobile-connector` — the **application** layer (thin).
|
||||
- Networking: `crates/skald-relay-client` — the **standalone, payload-agnostic**
|
||||
relay client (WS v2 transport, E2E crypto, anti-replay counters, pairing,
|
||||
device authorization, SQLite persistence). It depends only on
|
||||
`skald-relay-common` (never on `core-api`), so it is reusable and unit/
|
||||
integration-tested in isolation. See [Crate split](#crate-split-skald-relay-client).
|
||||
- Shared crypto + protobuf: `crates/skald-relay-common` (byte-for-byte interop
|
||||
with the reference vectors in [relay/test-vectors.md](../relay/test-vectors.md))
|
||||
- **Protocol documentation** (canonical, in English):
|
||||
- [relay/index.md](../relay/index.md) — architecture, actors, threat model
|
||||
- [relay/relay-protocol.md](../relay/relay-protocol.md) — protobuf schema, auth, pairing, live channel, presence
|
||||
- [relay/framing.md](../relay/framing.md) — E2E plaintext framing (version + compression)
|
||||
- [relay/payloads.md](../relay/payloads.md) — JSON payload schemas (inbox_request, inbox_update, …)
|
||||
- [relay/crypto.md](../relay/crypto.md) — crypto contract, key derivation, AEAD, anti-replay
|
||||
|
||||
---
|
||||
|
||||
## Crate split: skald-relay-client
|
||||
|
||||
The plugin is the **application** layer; all networking lives in the standalone
|
||||
`skald-relay-client` crate. The boundary is **payload-agnostic**: the client
|
||||
exchanges opaque decrypted `Vec<u8>` payloads keyed by device pubkey and emits
|
||||
inbound traffic as `RelayEvent`s; it never interprets the JSON. The plugin
|
||||
consumes `client.events()`, applies the JSON schemas (`payloads.rs`) and the
|
||||
`InboxApi`, and calls `client.send(dest, bytes, live)`.
|
||||
|
||||
Consequences of the split:
|
||||
|
||||
- **Authorization policy stays in the plugin.** On `RelayEvent::ClientPaired`,
|
||||
the client has already derived the `aes_key`, persisted the device as Pending,
|
||||
and consumed the pairing token. The plugin's event loop decides: if
|
||||
`require_device_confirmation` it notifies; otherwise it calls
|
||||
`client.authorize(ed)` and then `broadcast_inbox()`.
|
||||
- **`client.authorize()` is payload-agnostic** — it does NOT push an Inbox
|
||||
snapshot. The plugin sends the snapshot after authorizing (both the auto path
|
||||
and the `RelayAgent::authorize_client` tool path).
|
||||
- **v2 framing (`compress/decompress_payload`) is transport** — handled inside
|
||||
the client, so the plugin only ever sees clean JSON.
|
||||
- **Identity seed** is injected via `SeedSource::Path("data/relay/seed")` (same
|
||||
relative path as before) so existing identities/devices survive the upgrade.
|
||||
|
||||
### Module map — `skald-relay-client` (networking)
|
||||
|
||||
| Module | Role |
|
||||
|---|---|
|
||||
| `config.rs` | `RelayClientConfig` + `SeedSource` (`Bytes` / `Path`) |
|
||||
| `events.rs` | `RelayEvent` (`Connected`/`Disconnected`/`Message`/`ClientPaired`/`ClientRevoked`), broadcast |
|
||||
| `identity.rs` | Seed load/generate (`0600`, injected path) + derived Ed25519/X25519 keys + `namespace_id` |
|
||||
| `db.rs` | `relay_clients` table — devices + anti-replay counters (atomic counter helpers, `delete_all`) |
|
||||
| `pairing.rs` | In-memory single-window pairing sessions (`code → session`) + `QrCodeData` |
|
||||
| `state.rs` | Networking-only runtime: per-client `aes_key` cache, seal/open, counters, emits events |
|
||||
| `ws.rs` | Permanent reconnecting agent WebSocket (v2 binary transport). Challenge → `Auth` → role dispatch → forward loop |
|
||||
| `client.rs` | `RelayClient` — the public façade (`new`/`start`/`shutdown`/`send`/pairing/authorize/revoke/`clear_all`/`events`) |
|
||||
|
||||
### Module map — `plugin-mobile-connector` (application)
|
||||
|
||||
| Module | Role |
|
||||
|---|---|
|
||||
| `payloads.rs` | E2E JSON payload schemas (`inbox_update`, `notification`, client responses incl. `inbox_request`). Zlib-compressible per v2 framing.md |
|
||||
| `app.rs` | `RelayApp`: Inbox dispatch (`broadcast_inbox`/`apply_client_payload`), authorization policy, the `events()` consumer loop |
|
||||
| `notifier.rs` | `DelayedNotifier`: debounces phone pushes (`notify_delay_secs`) so resolving on the computer suppresses the push. See [Delayed push](#delayed-push) |
|
||||
| `proxy.rs` | Accepts relay **pipes** of `stream_type = "http-local-proxy"` and reverse-proxies each to `127.0.0.1:<web_port>`. See [HTTP reverse proxy](#http-reverse-proxy-http-local-proxy) |
|
||||
| `router.rs` | The QR-code HTTP endpoint (`/pairingqrcode`), resolves the current `RelayApp` → `client.lookup_pairing` |
|
||||
| `agent.rs` | `RelayAgent` control trait (pairing, list, authorize, revoke) |
|
||||
| `tools.rs` | The three LLM tools, registered in the main crate's `ToolRegistry` |
|
||||
| `lib.rs` | `MobileConnectorPlugin` (`Plugin` + `RelayAgent`), lifecycle, bus subscriber, `RelayClient`/`RelayApp` wiring |
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
Stored in the `plugins` table (JSON, edited via the plugin UI / `configure_plugin`):
|
||||
|
||||
```yaml
|
||||
relay_url: "wss://relay.skaldagent.net/v1/ws" # empty ⇒ plugin idle (no WS)
|
||||
pairing_ttl: 300 # seconds, max 600
|
||||
require_device_confirmation: true # manual confirm new devices (recommended)
|
||||
notify_delay_secs: 20 # debounce before pushing to the phone (0 = immediate)
|
||||
```
|
||||
|
||||
`enabled` (the standard plugin flag) starts/stops the runloop.
|
||||
|
||||
`notify_delay_secs` debounces the **phone push** for a new approval/clarification
|
||||
(see [Delayed push](#delayed-push)). The mobile push is only useful when you're
|
||||
away from the computer; if you answer at the computer within the window, no phone
|
||||
notification is sent. Set `0` to push immediately.
|
||||
|
||||
---
|
||||
|
||||
## Persistence (plugin.md §9)
|
||||
|
||||
| Data | Location | Why |
|
||||
|---|---|---|
|
||||
| `seed` (32 B) | filesystem `data/relay/seed`, `0600` | the only persistent secret; keys + `namespace_id` are derived at runtime |
|
||||
| Pairing session | **in-memory** only | transient (≤ TTL); lost on restart ⇒ just re-pair |
|
||||
| Devices + `send/recv_counter` | DB `relay_clients` | **must** survive restarts |
|
||||
|
||||
### Why counters live in the DB
|
||||
|
||||
Skald self-restarts by design. If counters reset to 0 on restart:
|
||||
- `send_counter → 0` reuses an AES-GCM nonce under the same key (breaks
|
||||
confidentiality + integrity for that device).
|
||||
- `recv_counter → 0` re-opens the replay window.
|
||||
|
||||
So `send_counter` is incremented **and persisted before** sealing/sending
|
||||
(`db::next_send_counter`, a transaction), and `recv_counter` is persisted only
|
||||
**after** a valid `open`.
|
||||
|
||||
### `aes_key` cache
|
||||
|
||||
The per-client AES-256-GCM key is `HKDF(X25519(seed_x_priv, client_x_pub))`. It
|
||||
is derived once and cached in memory (`HashMap<ed25519_pub, aes_key>` in
|
||||
`RelayState`), never persisted; on a cache miss it is re-derived from the
|
||||
client's stored `x25519_pub`. The cache entry is dropped on revoke.
|
||||
|
||||
---
|
||||
|
||||
## Pairing flow
|
||||
|
||||
1. The agent calls `mobile_start_pairing(ttl?)` (gated behind approval).
|
||||
2. The plugin generates a 32-byte `pairing_token` (CSPRNG), sends
|
||||
`pairing_start{token, ttl}` to the relay, and registers an in-memory session
|
||||
keyed by a separate random `code` (latest-wins: any prior active session is
|
||||
marked *Superseded*). It returns the URL
|
||||
`/api/plugin/mobile-connector/pairingqrcode?code=<code>`.
|
||||
3. The copilot renders the URL as an image. The endpoint serves a PNG of the QR
|
||||
while the session is **Active**, else a placeholder (`QR scaduto` /
|
||||
`QR già usato`). The QR payload is the normative `QrCodeData` JSON (never on
|
||||
disk, never in the URL).
|
||||
4. The client scans, connects as `role:"pairing"`, the relay consumes the token
|
||||
and forwards `client_paired` to the agent.
|
||||
5. On `client_paired`: derive + cache `aes_key`, persist the client as
|
||||
**Pending** (counters 0), mark the session **Consumed**, then apply the
|
||||
policy:
|
||||
- `require_device_confirmation = false` ⇒ auto-authorize.
|
||||
- `require_device_confirmation = true` ⇒ leave Pending; the human authorizes
|
||||
via the control surface (a `notification` is pushed to existing devices).
|
||||
|
||||
`authorize` always reflects the full local set (replacement semantics): adding a
|
||||
device sends the complete list including it; revoking sends it without.
|
||||
|
||||
---
|
||||
|
||||
## Message flows
|
||||
|
||||
- **Inbox → clients:** the bus subscriber reacts to the six Inbox events
|
||||
(`approval_requested`, `approval_resolved`, `clarification_requested`,
|
||||
`clarification_resolved`, `elicitation_requested`, `elicitation_resolved`) and
|
||||
routes them through the **debouncer** (see
|
||||
[Delayed push](#delayed-push)) before building an `InboxSnapshot` via
|
||||
`inbox.list_pending()` and sending a sealed `inbox_update` to every Authorized
|
||||
client. Each approval
|
||||
carries a humanised `summary` (from `Tool::describe(Short)`, computed in
|
||||
`Inbox::list_pending`) for the card/notification plus the raw `arguments`
|
||||
(untruncated) for the detail dialog — so the user sees the full `execute_cmd`
|
||||
command, not a truncated label. Each clarification carries its
|
||||
`suggested_answers`. Each elicitation carries **only** its prompt metadata
|
||||
(`server_name`, `message`, `field_name`, `sensitive`, `is_confirmation`) — never
|
||||
a value; the value is supplied by the device in `elicitation_response.content`.
|
||||
- **Clients → Inbox:** inbound `message` is checked (`from` ∈ Authorized,
|
||||
nonce direction + counter > `recv_counter`), opened, and dispatched by `kind`:
|
||||
`approval_response` → `inbox.approve/reject`, `clarification_response` →
|
||||
`inbox.answer`, `elicitation_response` → `inbox.resolve_elicitation` (its
|
||||
`content` may be a secret — never logged/persisted in clear), `hello` → persist
|
||||
`device_info`, `inbox_request` → send a **targeted** `inbox_update` back to
|
||||
`from` only (see below), `logout` → revoke.
|
||||
After any response the Inbox is re-snapshotted. `request_id` is mapped
|
||||
`string ↔ i64` (non-parsing ids are dropped). Inbox ops are idempotent by
|
||||
`request_id`.
|
||||
- **Reconnect snapshot (`inbox_request`):** the relay does **not** notify the
|
||||
agent when a client reconnects, so the client sends `inbox_request` on the
|
||||
**live channel** (`Message.live=true`) after every `auth_ok` (e.g. when the app
|
||||
is opened from a push). The agent replies with an `inbox_update` sealed to the
|
||||
requester only — not a broadcast — so other devices are not needlessly
|
||||
re-aligned. A pull of stale state is useless, so the live channel is correct:
|
||||
if the agent is offline, the client gets `PeerOffline` immediately instead of
|
||||
waiting. Side-effect-free and idempotent (by `request_id`). See
|
||||
`data/ios-app/v2/relay-protocol.md` §3.1.
|
||||
|
||||
---
|
||||
|
||||
## HTTP reverse proxy (`http-local-proxy`)
|
||||
|
||||
So a remote device can reach Skald's web UI **without** a VPN/Tailscale or an open
|
||||
port, the plugin reuses the relay **pipe** (relayed E2E byte-stream, see
|
||||
[relay/pipe.md](../relay/pipe.md)) as a reverse proxy to the local HTTP server.
|
||||
|
||||
`proxy.rs` subscribes to `RelayClient::incoming_pipes()` and, for each invite with
|
||||
`stream_type == "http-local-proxy"`, accepts the pipe and splices it byte-for-byte
|
||||
to a **fresh** `TcpStream` to `127.0.0.1:<web_port>` (`PluginContext::web_port`).
|
||||
Per pipe it `split`s the connection into independent send/receive halves and runs
|
||||
each direction in its own task (full-duplex, so neither blocks the other;
|
||||
`PipeSender::send` is backpressured by the pipe's ~10 MiB send buffer, and
|
||||
`recv`/`read` are cancel-safe — see [relay/pipe.md §6.1](../relay/pipe.md#61-full-duplex--client-side-backpressure)).
|
||||
When either direction ends it cancels a shared token so the other unwinds. Invites
|
||||
of other `stream_type`s are **ignored** (not rejected) since `incoming_pipes` is a
|
||||
broadcast shared with possible future consumers.
|
||||
|
||||
The native app side (later) opens one pipe per outbound connection and points a
|
||||
WebView at it; because the tunnel is a transparent TCP splice, HTTP/1.1 keep-alive,
|
||||
parallel connections, and the chat WebSocket upgrade all work unchanged.
|
||||
|
||||
**Security.**
|
||||
|
||||
- The destination is **pinned** to `127.0.0.1:<web_port>` — the client cannot pick
|
||||
host/port, so this is not an open localhost proxy (no SSRF to other local services).
|
||||
- Access is gated by the relay's pipe auth (`pipe.md §3.1`): only the namespace
|
||||
agent or an **authorized** client can establish a pipe.
|
||||
- It exposes the full local web UI remotely — that is the intent; pair/authorize
|
||||
devices accordingly.
|
||||
|
||||
**Relay tuning** (env, `pipe.md §3.3`): a browser opens several connections, so
|
||||
`RELAY_PIPE_MAX_PER_NS` (default 8) may need raising; an idle chat-WS pipe can be
|
||||
reaped at `RELAY_PIPE_IDLE_TIMEOUT_SECS` (120 s) — the frontend auto-reconnects.
|
||||
|
||||
Teardown: `proxy_one` takes a child of the plugin cancel token, so plugin stop
|
||||
closes active tunnels; `stop_inner` also `shutdown()`s the relay client.
|
||||
|
||||
---
|
||||
|
||||
## Delayed push
|
||||
|
||||
A phone push is only valuable when the user is *away* from the computer. When
|
||||
they're at the chat, every approval/clarification would otherwise fire an
|
||||
instant — and pointless — notification, since they answer on the computer within
|
||||
seconds. `DelayedNotifier` (`notifier.rs`) debounces this between the bus events
|
||||
and `broadcast_inbox()`. (Elicitations are not chat-inline, so they are exempt —
|
||||
see below.)
|
||||
|
||||
- **`*_requested`** arms a timer (`notify_delay_secs`, default 20s) keyed by
|
||||
`(kind, request_id)` — approvals, clarifications, and elicitations use
|
||||
independent id counters, so the kind is part of the key. If the timer elapses unresolved, the
|
||||
key is marked *notified* and the Inbox is pushed (`broadcast_inbox`, `live=false`
|
||||
→ store-and-forward / offline push).
|
||||
- **Elicitations are the exception**: they live *only* in the Inbox (never
|
||||
inline in the chat, unlike approvals/clarifications), so there is no
|
||||
computer-side answer to debounce against. They skip the timer and are pushed
|
||||
**immediately**, regardless of `notify_delay_secs`.
|
||||
- **`*_resolved`** before the timer fires ⇒ the timer is cancelled and **nothing
|
||||
is sent**. If the push already went out, the resolution is broadcast so the
|
||||
phone clears the item. (Untracked ids fall back to a broadcast for snapshot
|
||||
freshness.)
|
||||
|
||||
This only affects the **phone**: the desktop/web approval UI runs over the
|
||||
per-session WebSocket (`ApprovalRequired`/`AgentQuestion`) and is never delayed.
|
||||
Phone-driven responses (`apply_client_payload`) still `broadcast_inbox()`
|
||||
immediately; the subsequent `*_resolved` bus event is handled idempotently. Armed
|
||||
timers are cancelled on plugin stop (`cancel_all`). Set `notify_delay_secs: 0`
|
||||
for the previous instant-push behaviour.
|
||||
|
||||
---
|
||||
|
||||
## LLM tools (plugin.md §11)
|
||||
|
||||
| Tool | Effect | Approval |
|
||||
|---|---|---|
|
||||
| `mobile_start_pairing(ttl?)` | Open the pairing window, return the QR URL | **Gated** (a default `require` rule is seeded, like `execute_cmd`/`restart`) |
|
||||
| `mobile_list_devices()` | List devices (state, platform, device_info, last_seen) | read-only |
|
||||
| `mobile_revoke_device(pubkey)` | Revoke a device by hex ed25519 pubkey | `Config` category |
|
||||
|
||||
These tools are not contributed through the `Plugin` trait (which has no
|
||||
`tools()` method). They are registered in `Tools::build` (`src/core/skald/bundles.rs`):
|
||||
the plugin is fetched via `get_plugin_typed::<MobileConnectorPlugin>()`, cast to
|
||||
`Arc<dyn RelayAgent>`, and bound into the tools via
|
||||
`plugin_mobile_connector::mobile_tools(agent)` → `ToolRegistry::register_arc`.
|
||||
|
||||
`mobile_start_pairing`'s approval gate is the default rule seeded in
|
||||
`ApprovalManager::seed_defaults` (`src/core/approval/mod.rs`): opening a window
|
||||
emits a secret (the QR) into chat, so it must be a deliberate human action, not
|
||||
LLM-triggerable via prompt injection.
|
||||
|
||||
---
|
||||
|
||||
## HTTP endpoint
|
||||
|
||||
`GET /api/plugin/mobile-connector/pairingqrcode?code=<random>` — runtime PNG of
|
||||
the QR (or placeholder), behind Skald's normal auth. Mounted by `WebFrontend`
|
||||
via `Plugin::http_router()` (the router closes over the live `RelayState`). The
|
||||
`code` is a non-enumerable capability; a URL leaked into `chat_history`
|
||||
self-revokes once the window closes.
|
||||
@@ -0,0 +1,163 @@
|
||||
# Remote Connectivity
|
||||
|
||||
Exposes the Skald web app on a private mesh network so remote clients (iOS app, NAS, etc.) can connect without port forwarding or internet exposure.
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
core-api plugin-tailscale-remote crate
|
||||
───────────────────────────── ──────────────────────────────────────────────
|
||||
trait RemoteAccess ←── RemotePlugin (src/lib.rs)
|
||||
(core_api::remote) TailscaleSystemProvider (src/tailscale_sys.rs) ← default
|
||||
TailscaleEmbeddedProvider (src/tailscale.rs)
|
||||
[feature: remote-tailscale]
|
||||
```
|
||||
|
||||
- **`RemoteAccess` trait** (`core_api::remote`): vendor-agnostic interface. The core (`Skald`, WS handler, etc.) only knows this trait.
|
||||
- **`TailscaleSystemProvider`** (`crates/plugin-tailscale-remote/src/tailscale_sys.rs`): **recommended provider**. Reads the mesh IP via `tailscale ip -4` (requires `tailscaled` running on the host). Binds a standard `tokio::net::TcpListener` — no experimental dependencies.
|
||||
- **`TailscaleEmbeddedProvider`** (`crates/plugin-tailscale-remote/src/tailscale.rs`): embedded alternative using `tailscale-rs`. Feature-gated: `remote-tailscale`, **off by default** (enable via the root crate's `embedded-tailscale` feature). No system daemon required, but currently pre-1.0 with known DERP/reconnect issues, and it drags the `aws-lc-rs` C crypto build back into the tree (see [Feature Flag](#feature-flag)). Use only when a daemon cannot be installed (e.g. unrooted NAS).
|
||||
- **`RemotePlugin`** (`crates/plugin-tailscale-remote/src/lib.rs`): wires the provider into the plugin lifecycle. Spawns a second Axum server on the mesh interface using the same router as the local server.
|
||||
|
||||
### `RemoteAccess` trait
|
||||
|
||||
```rust
|
||||
pub trait RemoteAccess: Send + Sync {
|
||||
fn provider_name(&self) -> &str;
|
||||
async fn device_ip(&self) -> Result<Ipv4Addr>;
|
||||
fn is_connected(&self) -> bool;
|
||||
async fn shutdown(&self);
|
||||
}
|
||||
```
|
||||
|
||||
Stored in `Skald::remote: Arc<RwLock<Option<Arc<dyn RemoteAccess>>>>`. `None` when the plugin is disabled.
|
||||
|
||||
### Dual-bind strategy
|
||||
|
||||
The plugin (not the `WebServer`) is responsible for the mesh-facing server:
|
||||
|
||||
1. `RemotePlugin::start(state)` calls `extract_deps(state)` once, storing three named fields:
|
||||
- `port: u16` — TCP port to bind on the mesh interface
|
||||
- `remote_slot: Arc<RwLock<Option<Arc<dyn RemoteAccess>>>>` — slot in `Skald` to register the active provider
|
||||
- `router_factory: Arc<dyn Fn() -> Router + Send + Sync>` — closure that rebuilds the Axum router
|
||||
2. The internal helpers (`start_tailscale_sys`, `start_tailscale`) use only those three fields — no `Arc<Skald>` reference after extraction.
|
||||
3. `start_tailscale_sys` binds `tokio::net::TcpListener::bind((mesh_ip, port))`.
|
||||
4. `start_tailscale` calls `provider.axum_listener(port)` → `tailscale::axum::Listener`.
|
||||
5. Both call `router_factory()` to get a fresh router and spawn `axum::serve(listener, router)` guarded by a `CancellationToken`.
|
||||
|
||||
`extract_deps` uses `std::sync::OnceLock` — idempotent across `reload()` calls. The values are stable for the lifetime of the process (port and static dir come from config, the remote slot is the same `Arc`).
|
||||
|
||||
The local server on `127.0.0.1:PORT` is unaffected.
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
Config is stored in the **`plugins` SQLite table** (not in `config.yml`).
|
||||
|
||||
| Key | Type | Default | Description |
|
||||
|---|---|---|---|
|
||||
| `provider` | string | `"tailscale_sys"` | `tailscale_sys` (system daemon, recommended) or `tailscale` (embedded, no daemon). |
|
||||
| `auth_key` | string | — | Tailscale auth key (`tskey-auth-...`). Only required for the `tailscale` embedded provider on first join. |
|
||||
| `hostname` | string | `"personal-agent"` | Hostname on the tailnet. Only used by the embedded `tailscale` provider. |
|
||||
| `key_file` | string | `"data/tailscale_keys.json"` | Path for persisting node identity between restarts. Only used by the embedded `tailscale` provider. |
|
||||
|
||||
Config is persisted automatically and survives restarts.
|
||||
|
||||
---
|
||||
|
||||
## Agent Workflow
|
||||
|
||||
### Using the system Tailscale daemon (recommended)
|
||||
|
||||
Requires Tailscale already installed and logged in on the host machine.
|
||||
|
||||
```
|
||||
1. toggle_item(kind="plugin", id="remote_connectivity", enabled=true)
|
||||
2. restart
|
||||
→ On next boot: plugin reads IP from tailscale ip -4, mesh server starts, app reachable at <ts-ip>:3000
|
||||
```
|
||||
|
||||
No auth key needed — the system daemon is already authenticated.
|
||||
|
||||
### Using the embedded provider (no daemon)
|
||||
|
||||
Requires a binary **built with `--features embedded-tailscale`** (off by default;
|
||||
see [Feature Flag](#feature-flag)). Without it, selecting `provider="tailscale"`
|
||||
fails at start with `unknown provider 'tailscale'`.
|
||||
|
||||
```
|
||||
1. configure_plugin "remote_connectivity" {"provider":"tailscale","auth_key":"tskey-auth-...","hostname":"personal-agent"}
|
||||
2. toggle_item(kind="plugin", id="remote_connectivity", enabled=true)
|
||||
3. restart
|
||||
→ On next boot: embedded tailscale connects, mesh server starts, app reachable at <ts-ip>:3000
|
||||
```
|
||||
|
||||
After first setup, the plugin auto-starts on every boot (persisted `enabled=true` in DB).
|
||||
|
||||
To check status: `list_items` (type=plugins) → look for `remote_connectivity`, check `running` and `runtime_status.ip`.
|
||||
|
||||
---
|
||||
|
||||
## Data Streams (iOS → Server)
|
||||
|
||||
Remote clients can push typed data over the existing WebSocket connection:
|
||||
|
||||
```json
|
||||
{"type": "data", "stream": "location", "payload": {"lat": 45.1, "lng": 9.2, "accuracy": 10.0}}
|
||||
```
|
||||
|
||||
Handled in `src/frontend/api/ws.rs` → `handle_data_msg()`. Dispatched to:
|
||||
|
||||
| Stream | Handler | Notes |
|
||||
|---|---|---|
|
||||
| `location` | `state.location_manager.update("remote", ...)` | Stores in-memory; existing `latest()` / `all()` queries work |
|
||||
| other | `warn!` log | Reserved for future streams (health, etc.) |
|
||||
|
||||
---
|
||||
|
||||
## Limitations
|
||||
|
||||
### `tailscale_sys` (system daemon)
|
||||
|
||||
- **Requires `tailscaled` on the host**: if the daemon is not running or the device is not logged in, `start()` fails immediately with a clear error.
|
||||
- **IP can change**: if the device rejoins the tailnet with a different IP, the plugin must be restarted to pick up the new address. The server binds to a specific IP, not `0.0.0.0`.
|
||||
|
||||
### `tailscale` (embedded, tailscale-rs v0.3)
|
||||
|
||||
- **DERP-relay only**: direct WireGuard holepunching is not yet implemented (issue #151). Latency is slightly higher than native WireGuard.
|
||||
- **Known reconnect bugs**: after a node restart, existing connections can hang for ~15 s (issue #11). DERP connectivity can be lost after a control plane reconnect (issue #26).
|
||||
- **Unaudited cryptography**: do not use for highly sensitive data until an official audit is complete.
|
||||
- **Breaking changes**: the library is pre-1.0. The `TS_RS_EXPERIMENT=this_is_unstable_software` env var is required at runtime (set by `run.sh`).
|
||||
- **Auth key expiry**: auth keys expire. Regenerate and call `configure_plugin` with the new value if the plugin fails to connect.
|
||||
|
||||
---
|
||||
|
||||
## Feature Flag
|
||||
|
||||
`tailscale-rs` is compiled only when the `remote-tailscale` feature is active.
|
||||
It is **off by default**: the crate pulls the pure-Rust `tailscale` library, which
|
||||
internally forces the `aws-lc-rs` crypto backend (a cmake/NASM C build) back into
|
||||
the whole workspace, defeating the ring-only static-binary story
|
||||
(see [build-and-distribution.md](../build-and-distribution.md)). The recommended
|
||||
`tailscale_sys` provider needs none of this and is always compiled.
|
||||
|
||||
```toml
|
||||
# crates/plugin-tailscale-remote/Cargo.toml
|
||||
[features]
|
||||
default = [] # embedded provider OFF by default
|
||||
remote-tailscale = ["dep:tailscale"]
|
||||
|
||||
# root Cargo.toml — opt in from the top-level build:
|
||||
[features]
|
||||
embedded-tailscale = ["plugin-tailscale-remote/remote-tailscale"]
|
||||
```
|
||||
|
||||
To build **with** the embedded provider (accepts the aws-lc-rs C build):
|
||||
|
||||
```sh
|
||||
cargo build --features embedded-tailscale
|
||||
```
|
||||
|
||||
The default build already omits it — no `--no-default-features` needed.
|
||||
@@ -0,0 +1,234 @@
|
||||
# Telegram Plugin
|
||||
|
||||
A private Telegram bot that forwards messages to the LLM and supports Human-in-the-Loop approvals via inline keyboard buttons.
|
||||
|
||||
---
|
||||
|
||||
## Setup
|
||||
|
||||
1. Create a bot with [@BotFather](https://t.me/BotFather) and copy the token.
|
||||
2. Add to `config.yml`:
|
||||
|
||||
```yaml
|
||||
plugins:
|
||||
telegram:
|
||||
token: "123456789:AABBCC..."
|
||||
```
|
||||
|
||||
3. Restart the app. The bot starts automatically if the token is present.
|
||||
|
||||
---
|
||||
|
||||
## Pairing — how to authorize a user
|
||||
|
||||
Access control is managed entirely through the file `secrets/telegram_whitelist.json`.
|
||||
|
||||
### File format
|
||||
|
||||
```json
|
||||
{
|
||||
"whitelist": [123456789],
|
||||
"pending_pairings": [
|
||||
{
|
||||
"code": "A3KX7P",
|
||||
"chat_id": 987654321,
|
||||
"issued_at": "2026-05-19T10:30:00+02:00"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
- `whitelist` — array of authorized `chat_id` values (integers). Users in this array can send messages to the agent.
|
||||
- `pending_pairings` — users who have contacted the bot but are not yet authorized. Each entry has a `code` shown to the user in Telegram chat, their `chat_id`, and the `issued_at` timestamp. Entries older than **24 hours** are pruned automatically the next time an unauthorized user contacts the bot, so abandoned codes do not pile up.
|
||||
|
||||
### Pairing flow
|
||||
|
||||
1. An unknown user sends any message to the bot.
|
||||
2. The bot replies with a 6-character pairing code and writes an entry to `pending_pairings` in `secrets/telegram_whitelist.json`.
|
||||
3. The user communicates the code to you (e.g., through a separate channel).
|
||||
4. You ask the agent: *"Telegram pairing code A3KX7P — authorize it"*.
|
||||
5. The agent reads `secrets/telegram_whitelist.json`, finds the entry with `code: "A3KX7P"`, moves the `chat_id` from `pending_pairings` to `whitelist`, and writes the file back.
|
||||
6. Within 10 seconds the plugin's watchdog detects the file change, logs the event, and **sends a welcome message** to the newly authorized user on Telegram.
|
||||
|
||||
### To authorize manually (without asking the agent)
|
||||
|
||||
Use `edit_file` or `write_file` to move the `chat_id` from `pending_pairings` to `whitelist` in `secrets/telegram_whitelist.json`.
|
||||
|
||||
### To revoke access
|
||||
|
||||
Remove the `chat_id` from the `whitelist` array in `secrets/telegram_whitelist.json`. The change takes effect on the user's next message (whitelist is re-read on every message).
|
||||
|
||||
---
|
||||
|
||||
## Watchdog
|
||||
|
||||
The plugin polls `secrets/telegram_whitelist.json` every **10 seconds** for modification-time changes.
|
||||
|
||||
When it detects a change:
|
||||
- Reloads the whitelist.
|
||||
- Identifies any `chat_id` values newly added to `whitelist`.
|
||||
- Sends each newly authorized user a welcome message on Telegram.
|
||||
- Logs the event at INFO level.
|
||||
|
||||
This means there is no restart needed after editing the file — authorization takes effect automatically.
|
||||
|
||||
---
|
||||
|
||||
## Commands
|
||||
|
||||
| Command | Effect |
|
||||
|---|---|---|
|
||||
| `/new` or `/clear` | Create a new chat session (clears LLM context) |
|
||||
| `/stop` | Interrupt the agent mid-turn (clears pending approvals and clarifications) |
|
||||
| `/models` | List available LLM models ordered by priority (numbered `0..N`, index 0 is `auto`) |
|
||||
| `/model <N\|name\|auto>` | Pin the model for this chat by index, name (substring allowed), or reset to `auto`. State is held in `ChatHub.selected_clients["telegram"]` and broadcast to all clients of the source via `ClientSelected`. Cleared on server restart |
|
||||
| `/context` | Show last turn's token usage (`↑X tok · ↓Y tok`) |
|
||||
| `/cost` | Show total spend for this session in USD (sync sub-agents included; async tasks excluded) |
|
||||
| `/compact` | Force context compaction (bypasses the token threshold) |
|
||||
| `/resettools` | Remove all activated tool groups (MCP servers + `config`) from the session |
|
||||
| `/sethome` | Set Telegram as the home source for background notifications |
|
||||
| `/help` | Show available commands |
|
||||
| any other `/command` | Unknown — replies with a "Unknown command" notice + the help list, never forwarded to the LLM |
|
||||
| any text | Forwarded to the LLM agent |
|
||||
|
||||
---
|
||||
|
||||
## Human-in-the-Loop Approvals
|
||||
|
||||
When the LLM triggers a tool that requires user approval (`execute_cmd`, `restart`, write-file tools outside `memory/`):
|
||||
|
||||
1. The bot sends a message with the operation details and a content preview.
|
||||
2. Four inline keyboard buttons appear in two rows:
|
||||
|
||||
```text
|
||||
[✅ Approve] [❌ Reject]
|
||||
[⏱ 15 min] [🔄 Session]
|
||||
```
|
||||
|
||||
3. Tapping a button resolves the pending approval and execution continues or is cancelled.
|
||||
4. **⏱ 15 min** — approves and suppresses approval prompts for tools of the same category/MCP server for 15 minutes.
|
||||
5. **🔄 Session** — approves and suppresses all approval prompts for the rest of the session.
|
||||
6. The approval message is **deleted** once resolved, whether via Telegram or the web UI.
|
||||
|
||||
Bypass buttons call `ApprovalApi::approve_with_bypass` (scope auto-detected from the tool's category or MCP server). See [../approval/index.md](../approval/index.md) for bypass semantics.
|
||||
|
||||
---
|
||||
|
||||
## Output Formatting
|
||||
|
||||
Telegram's HTML parse mode supports only a limited tag set: `<b>` `<i>` `<u>` `<s>` `<code>` `<pre>` `<a>` `<blockquote>`. Structural elements (`<table>`, `<ul>`, `<li>`, `<div>`) are **not supported**.
|
||||
|
||||
The plugin injects a compact formatting context into every LLM session (`TELEGRAM_FORMAT_CONTEXT` in `mod.rs`) and a shorter tail reminder (`TELEGRAM_FORMAT_REMINDER`) instructing it to:
|
||||
|
||||
- Use Telegram HTML tags only.
|
||||
- Never use Markdown (`**`, `*`, `` ` ``, `#`, `_`, `|`).
|
||||
- Replace structured data (tables) with bullet lists (`•`).
|
||||
|
||||
| Element | Correct | Wrong |
|
||||
| --------------- | --------------------- | ------------------ |
|
||||
| Bold | `<b>text</b>` | `**text**` |
|
||||
| Italic | `<i>text</i>` | `*text*` |
|
||||
| Code | `<code>text</code>` | `` `text` `` |
|
||||
| Code block | `<pre>text</pre>` | ` ```text``` ` |
|
||||
| Structured data | bullet list `•` | `\| col \| col \|` |
|
||||
|
||||
Long responses are automatically split into chunks of ≤ 4000 characters via `send_long()` in `helpers.rs`.
|
||||
|
||||
### Markdown sanitizer (post-processing safety net)
|
||||
|
||||
Because LLMs occasionally emit Markdown despite instructions, `send_long` applies `sanitize_for_telegram()` on every HTML-mode send **before** chunking. This provides a reliable fallback independent of model compliance:
|
||||
|
||||
1. **Markdown tables → bullet lists** — detects `| col | col |` blocks, emits the header row as `<b>header — header</b>` and each data row as `• val — val`.
|
||||
2. **`**bold**` → `<b>bold</b>`** — converts residual Markdown bold.
|
||||
3. **`## Header` → `<b>Header</b>`** — converts residual Markdown headers.
|
||||
|
||||
### Fallback behavior
|
||||
|
||||
If the Telegram API rejects a chunk (e.g., due to malformed HTML), `send_long` retries **without** `ParseMode::Html`. Before retrying it strips all HTML tags (`<…>`) from the chunk using a regex so the user sees plain text rather than raw `<b>…</b>` markup.
|
||||
|
||||
---
|
||||
|
||||
## Voice (Speech Integration)
|
||||
|
||||
If the Speech plugin is configured and running, the Telegram plugin gains two additional capabilities:
|
||||
|
||||
### Incoming voice messages (STT)
|
||||
|
||||
When the user sends a voice note, the plugin:
|
||||
|
||||
1. Downloads the OGG audio from Telegram.
|
||||
2. Passes it to `SpeechPlugin::transcribe()`.
|
||||
3. Forwards the resulting text to the LLM as a normal message.
|
||||
|
||||
### Outgoing voice replies (TTS)
|
||||
|
||||
The LLM has access to a `send_voice_message(text)` tool. When it calls it, the plugin:
|
||||
|
||||
1. Passes the text to the active `TextToSpeech` synthesiser (`synthesize()`).
|
||||
2. **Transcodes the audio to Ogg/Opus** (`to_ogg_opus` in `tools.rs`) — the only format Telegram renders as a playable voice message. The synthesiser's `output_format()` decides the input: `opus`/`ogg` pass through untouched; raw `pcm` (e.g. Gemini TTS) is decoded as 24 kHz/mono/s16le; every other container (mp3, wav, …) is auto-detected. Conversion runs `ffmpeg` over stdin/stdout pipes (no temp files).
|
||||
3. Sends the resulting Ogg/Opus bytes back to the user as a Telegram voice message.
|
||||
|
||||
The LLM is instructed to use voice only for short, conversational replies with no code or complex formatting. The TTS engine's formatting guide (SSML-like tags) is also injected into the system context so the LLM can control pacing and emphasis.
|
||||
|
||||
> **Requires `ffmpeg`** on `PATH` for any non-Opus synthesiser. If it is missing, `send_voice_message` returns a clear error (`ffmpeg not available …`) and the LLM falls back to a text reply.
|
||||
|
||||
### Requirements
|
||||
|
||||
Both `plugins.speech.stt_model` and `plugins.speech.tts_model` must be set in `config.yml`. The Speech plugin must be enabled and running before the Telegram plugin starts.
|
||||
|
||||
---
|
||||
|
||||
## File & Media Attachments
|
||||
|
||||
The Telegram plugin downloads incoming attachments and forwards them to the conversation. The LLM sees them in timeline order — it knows which file was most recently sent without any special indexing.
|
||||
|
||||
| Type | Saved to disk | How it reaches the LLM |
|
||||
| --- | --- | --- |
|
||||
| Document (PDF, ZIP, …) | `data/uploads/telegram/<chat_id>/<filename>` | Structured `metadata.attachments` (shared with web/mobile) |
|
||||
| Photo | `data/uploads/telegram/<chat_id>/<file_id>.jpg` | Structured `metadata.attachments` |
|
||||
| Location | — | `[TELEGRAM SYSTEM INFO]` text (latitude, longitude, accuracy, Google Maps URL) |
|
||||
|
||||
**Document and Photo are aligned with the web/mobile attachment model**: `download_and_save`
|
||||
returns a `core_api::message_meta::Attachment` (project-root-relative path so `/data/…` serves
|
||||
it), `handle_attachment` puts it in `SendMessageOptions.metadata`, and the message builder
|
||||
generates the shared `[SYSTEM INFO]` block on the fly. Viewing the `telegram` source from the
|
||||
copilot therefore shows these as **chips**, not raw text. See
|
||||
[frontend.md](../frontend.md#attachments) and [database.md](../database.md) (`chat_history.metadata`).
|
||||
|
||||
**Location** has no file, so it keeps the legacy `[TELEGRAM SYSTEM INFO]` text path
|
||||
(`system_info_message`). Captions typed alongside a Document/Photo become the user turn's text.
|
||||
|
||||
### Live locations
|
||||
|
||||
When the user shares a live location, two things happen:
|
||||
|
||||
1. **Initial message** (`message` event) — the LLM is notified via a `[TELEGRAM SYSTEM INFO]` message and the position is written to `skald.location_manager` under the key `"telegram"`.
|
||||
2. **Subsequent updates** (`edited_message` events) — the position in `location_manager` is updated silently, with no LLM notification. This keeps the store current for any background scripts or tools that read `user_location("telegram")`.
|
||||
|
||||
`LocationManager` is in-memory only. On restart, the store starts empty and is repopulated as soon as Telegram delivers the next live location tick (typically within seconds if sharing is still active).
|
||||
|
||||
The `uploads/` directory is gitignored.
|
||||
|
||||
### Extending attachment types
|
||||
|
||||
To add a new type (e.g. sticker, contact):
|
||||
|
||||
1. Add a variant to `TelegramAttachment` in `crates/plugin-telegram-bot/src/attachments.rs`.
|
||||
2. Implement `download_and_save`: return `Ok(Some(Attachment))` for a file-backed type (it flows into `metadata.attachments`) or `Ok(None)` for a file-less one (then add a `system_info_message` arm and handle it in the `None` branch of `handle_attachment`).
|
||||
3. Detect the message type in `classify_message` in `handlers.rs` and return `IncomingEvent::Attachment(...)`.
|
||||
|
||||
---
|
||||
|
||||
## Interface Tools
|
||||
|
||||
The Telegram plugin can inject custom LLM-callable tools into any session via the `interface_tools` parameter of `SendMessageOptions`. These tools are only visible to the root agent — sub-agents do not inherit them.
|
||||
|
||||
To add a Telegram-specific tool, construct an `InterfaceTool` with an OpenAI tool definition and an async handler closure that captures `Arc<Bot>` and `ChatId`, then pass it in the `interface_tools` vec inside `SendMessageOptions`.
|
||||
|
||||
`InterfaceTool` and `ToolFuture` are defined in `crates/core-api/src/interface_tool.rs` (re-exported via `crate::chat_hub`). `AgentRunConfig` remains in `src/core/session/handler/interface_tools.rs` (main crate only).
|
||||
|
||||
---
|
||||
|
||||
## Secrets directory
|
||||
|
||||
`secrets/telegram_whitelist.json` is gitignored. The directory is created automatically on first pairing request. Never commit this file.
|
||||
@@ -0,0 +1,123 @@
|
||||
# WhisperLocal Plugin
|
||||
|
||||
Local Speech-to-Text via [whisper.cpp](https://github.com/ggerganov/whisper.cpp), Metal-accelerated on Apple Silicon.
|
||||
Implemented in pure Rust using the `whisper-rs` crate — no Python involved.
|
||||
|
||||
---
|
||||
|
||||
## Setup
|
||||
|
||||
### 1. Download a GGML model
|
||||
|
||||
```sh
|
||||
mkdir -p models
|
||||
curl -L -o models/ggml-large-v3.bin \
|
||||
https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-large-v3.bin
|
||||
```
|
||||
|
||||
Other available sizes (smaller = faster, less accurate):
|
||||
|
||||
| Model | Size | Notes |
|
||||
|---|---|---|
|
||||
| `ggml-tiny.bin` | ~75 MB | Very fast, lower accuracy |
|
||||
| `ggml-base.bin` | ~142 MB | Good balance for testing |
|
||||
| `ggml-small.bin` | ~466 MB | Good accuracy |
|
||||
| `ggml-medium.bin` | ~1.5 GB | High accuracy |
|
||||
| `ggml-large-v3.bin` | ~3.1 GB | Best accuracy, recommended |
|
||||
| `ggml-large-v3-turbo.bin` | ~1.6 GB | large-v3 speed-optimised |
|
||||
|
||||
All models: `https://huggingface.co/ggerganov/whisper.cpp`
|
||||
|
||||
### 2. Configure `config.yml`
|
||||
|
||||
```yaml
|
||||
plugins:
|
||||
whisper_local:
|
||||
model: "models/ggml-large-v3.bin"
|
||||
language: "it" # BCP-47 code, or "auto" for detection
|
||||
load_at_startup: false # false (default) = lazy load on first use
|
||||
idle_timeout_secs: 1200 # unload after 20 min idle; 0 = never unload
|
||||
```
|
||||
|
||||
| Option | Default | Effect |
|
||||
|---|---|---|
|
||||
| `model` | — (required) | Path to the GGML `.bin` file |
|
||||
| `language` | `auto` | BCP-47 code or `auto`. Applied live — runtime changes take effect on the next transcription without a reload |
|
||||
| `load_at_startup` | `false` | **When** the model first loads: `false` = lazily on the first transcription, `true` = eagerly in `start()` (warm, no first-call latency) |
|
||||
| `idle_timeout_secs` | `1200` | **When** the model unloads: after this many seconds of inactivity. `0` = never unload (stays resident once loaded) |
|
||||
|
||||
The two timing options are orthogonal and cover the whole spectrum:
|
||||
|
||||
| `load_at_startup` | `idle_timeout_secs` | Behaviour |
|
||||
|---|---|---|
|
||||
| `false` | `1200` | **Default** — load on first use, free ~2 GB after 20 min idle |
|
||||
| `true` | `0` | Always resident — eager load, never unload (legacy behaviour) |
|
||||
| `true` | `1200` | Warm at startup, but freed if unused |
|
||||
| `false` | `0` | Load on first use, then stay resident |
|
||||
|
||||
### 3. Build
|
||||
|
||||
The first `cargo build` compiles whisper.cpp (a few minutes). Subsequent builds are cached.
|
||||
|
||||
---
|
||||
|
||||
## How it works
|
||||
|
||||
```
|
||||
Telegram voice message (OGG/Opus)
|
||||
│
|
||||
▼ ffmpeg → 16 kHz mono WAV
|
||||
▼ hound → Vec<f32> PCM samples
|
||||
▼ whisper.cpp (Metal GPU) → text
|
||||
│
|
||||
▼ forwarded to LLM as a normal text message
|
||||
```
|
||||
|
||||
Audio conversion uses the system `ffmpeg` binary (must be installed: `brew install ffmpeg`).
|
||||
Inference runs on Apple Silicon GPU via Metal. Falls back to CPU if Metal is unavailable.
|
||||
|
||||
---
|
||||
|
||||
## Memory management (lazy load + idle unload)
|
||||
|
||||
The GGML weights are ~2 GB, so the plugin keeps them in memory only while they are
|
||||
actually useful. The model lives in a shared, droppable cell (`LazyModel`):
|
||||
|
||||
- **`start()`** validates the model path and registers a lightweight transcriber, but
|
||||
does **not** load the weights unless `load_at_startup: true`. The registered handle
|
||||
holds no strong reference to the weights, so they can be freed at any time.
|
||||
- **First transcription** triggers `ensure_loaded()`, which loads the weights once
|
||||
(concurrent first-callers wait on a single load) and records a last-used timestamp.
|
||||
- A background **eviction task** ticks every 60 s and unloads the model once it has
|
||||
been idle for `idle_timeout_secs`. Set `idle_timeout_secs: 0` to disable eviction.
|
||||
- **Unloading** is refcount-safe: an in-flight transcription holds its own handle to
|
||||
the weights, so memory is reclaimed only after it finishes. The actual free runs on
|
||||
a blocking thread (whisper.cpp GPU cleanup).
|
||||
|
||||
Trade-off: after an unload, the next transcription pays the reload cost (a few
|
||||
seconds). The OS page cache usually keeps the `.bin` warm, so the reload is mostly
|
||||
memory copy + Metal allocation rather than disk I/O. Use `load_at_startup: true` /
|
||||
`idle_timeout_secs: 0` if you prefer zero first-call latency over reclaiming the RAM.
|
||||
|
||||
---
|
||||
|
||||
## Integration with TranscribeManager
|
||||
|
||||
`WhisperLocalPlugin` does **not** expose itself as `Arc<dyn Transcribe>` directly. At `start()` it registers a lightweight `WhisperLocalTranscriber` handle into `skald.transcribe_manager`; at `stop()` it deregisters it. Callers never reference the plugin type — they ask the manager:
|
||||
|
||||
```rust
|
||||
if let Some(t) = skald.transcribe_manager.get().await {
|
||||
let text = t.transcribe(audio, "ogg").await?;
|
||||
}
|
||||
```
|
||||
|
||||
See [../plugins.md](../plugins.md) for the `TranscribeManager` API and the `Transcribe` trait.
|
||||
|
||||
---
|
||||
|
||||
## When to Update This File
|
||||
|
||||
- The audio conversion pipeline changes
|
||||
- Default recommended models change
|
||||
- Registration/deregistration logic in `start()`/`stop()` changes
|
||||
- The lazy-load / idle-eviction lifecycle or its config options change
|
||||
Reference in New Issue
Block a user