Files

T

2026-07-10 15:02:09 +01:00

12 KiB

Raw Blame History

Telegram Plugin

A private Telegram bot that forwards messages to the LLM and supports Human-in-the-Loop approvals via inline keyboard buttons.

Setup

Create a bot with @BotFather and copy the token.
Add to config.yml:

plugins:
  telegram:
    token: "123456789:AABBCC..."

Restart the app. The bot starts automatically if the token is present.

Pairing — how to authorize a user

Access control is managed entirely through the file secrets/telegram_whitelist.json.

File format

{
  "whitelist": [123456789],
  "pending_pairings": [
    {
      "code": "A3KX7P",
      "chat_id": 987654321,
      "issued_at": "2026-05-19T10:30:00+02:00"
    }
  ]
}

whitelist — array of authorized chat_id values (integers). Users in this array can send messages to the agent.
pending_pairings — users who have contacted the bot but are not yet authorized. Each entry has a code shown to the user in Telegram chat, their chat_id, and the issued_at timestamp. Entries older than 24 hours are pruned automatically the next time an unauthorized user contacts the bot, so abandoned codes do not pile up.

Pairing flow

An unknown user sends any message to the bot.
The bot replies with a 6-character pairing code and writes an entry to pending_pairings in secrets/telegram_whitelist.json.
The user communicates the code to you (e.g., through a separate channel).
You ask the agent: "Telegram pairing code A3KX7P — authorize it".
The agent reads secrets/telegram_whitelist.json, finds the entry with code: "A3KX7P", moves the chat_id from pending_pairings to whitelist, and writes the file back.
Within 10 seconds the plugin's watchdog detects the file change, logs the event, and sends a welcome message to the newly authorized user on Telegram.

To authorize manually (without asking the agent)

Use edit_file or write_file to move the chat_id from pending_pairings to whitelist in secrets/telegram_whitelist.json.

To revoke access

Remove the chat_id from the whitelist array in secrets/telegram_whitelist.json. The change takes effect on the user's next message (whitelist is re-read on every message).

Watchdog

The plugin polls secrets/telegram_whitelist.json every 10 seconds for modification-time changes.

When it detects a change:

Reloads the whitelist.
Identifies any chat_id values newly added to whitelist.
Sends each newly authorized user a welcome message on Telegram.
Logs the event at INFO level.

This means there is no restart needed after editing the file — authorization takes effect automatically.

Commands

Command	Effect
`/new` or `/clear`	Create a new chat session (clears LLM context)
`/stop`	Interrupt the agent mid-turn (clears pending approvals and clarifications)
`/models`	List available LLM models ordered by priority (numbered `0..N`, index 0 is `auto`)
`/model <N\|name\|auto>`	Pin the model for this chat by index, name (substring allowed), or reset to `auto`. State is held in `ChatHub.selected_clients["telegram"]` and broadcast to all clients of the source via `ClientSelected`. Cleared on server restart
`/context`	Show last turn's token usage (`↑X tok · ↓Y tok`)
`/cost`	Show total spend for this session in USD (sync sub-agents included; async tasks excluded)
`/compact`	Force context compaction (bypasses the token threshold)
`/resettools`	Remove all activated tool groups (MCP servers + `config`) from the session
`/sethome`	Set Telegram as the home source for background notifications
`/help`	Show available commands
any other `/command`	Unknown — replies with a "Unknown command" notice + the help list, never forwarded to the LLM
any text	Forwarded to the LLM agent

Human-in-the-Loop Approvals

When the LLM triggers a tool that requires user approval (execute_cmd, restart, write-file tools outside memory/):

The bot sends a message with the operation details and a content preview.

Four inline keyboard buttons appear in two rows:

[✅ Approve]  [❌ Reject]
[⏱ 15 min]   [🔄 Session]

Tapping a button resolves the pending approval and execution continues or is cancelled.
⏱ 15 min — approves and suppresses approval prompts for tools of the same category/MCP server for 15 minutes.
🔄 Session — approves and suppresses all approval prompts for the rest of the session.
The approval message is deleted once resolved, whether via Telegram or the web UI.

Bypass buttons call ApprovalApi::approve_with_bypass (scope auto-detected from the tool's category or MCP server). See ../approval/index.md for bypass semantics.

Output Formatting

Telegram's HTML parse mode supports only a limited tag set:    <s> <code> <pre> <a> <blockquote>. Structural elements (<table>, <ul>, <li>, <div>) are not supported.

The plugin injects a compact formatting context into every LLM session (TELEGRAM_FORMAT_CONTEXT in mod.rs) and a shorter tail reminder (TELEGRAM_FORMAT_REMINDER) instructing it to:

Use Telegram HTML tags only.
Never use Markdown (**, *, `, #, _, |).
Replace structured data (tables) with bullet lists (•).

Element	Correct	Wrong
Bold	`<b>text</b>`	`text`
Italic	`<i>text</i>`	`text`
Code	`<code>text</code>`	`text`
Code block	`<pre>text</pre>`	```text```
Structured data	bullet list `•`	`\| col \| col \|`

Long responses are automatically split into chunks of ≤ 4000 characters via send_long() in helpers.rs.

Markdown sanitizer (post-processing safety net)

Because LLMs occasionally emit Markdown despite instructions, send_long applies sanitize_for_telegram() on every HTML-mode send before chunking. This provides a reliable fallback independent of model compliance:

Markdown tables → bullet lists — detects | col | col | blocks, emits the header row as header — header and each data row as • val — val.
**bold** → bold — converts residual Markdown bold.
## Header → Header — converts residual Markdown headers.

Fallback behavior

If the Telegram API rejects a chunk (e.g., due to malformed HTML), send_long retries without ParseMode::Html. Before retrying it strips all HTML tags (<…>) from the chunk using a regex so the user sees plain text rather than raw … markup.

Voice (Speech Integration)

If the Speech plugin is configured and running, the Telegram plugin gains two additional capabilities:

Incoming voice messages (STT)

When the user sends a voice note, the plugin:

Downloads the OGG audio from Telegram.
Passes it to SpeechPlugin::transcribe().
Forwards the resulting text to the LLM as a normal message.

Outgoing voice replies (TTS)

The LLM has access to a send_voice_message(text) tool. When it calls it, the plugin:

Passes the text to the active TextToSpeech synthesiser (synthesize()).
Transcodes the audio to Ogg/Opus (to_ogg_opus in tools.rs) — the only format Telegram renders as a playable voice message. The synthesiser's output_format() decides the input: opus/ogg pass through untouched; raw pcm (e.g. Gemini TTS) is decoded as 24 kHz/mono/s16le; every other container (mp3, wav, …) is auto-detected. Conversion runs ffmpeg over stdin/stdout pipes (no temp files).
Sends the resulting Ogg/Opus bytes back to the user as a Telegram voice message.

The LLM is instructed to use voice only for short, conversational replies with no code or complex formatting. The TTS engine's formatting guide (SSML-like tags) is also injected into the system context so the LLM can control pacing and emphasis.

Requires ffmpeg on PATH for any non-Opus synthesiser. If it is missing, send_voice_message returns a clear error (ffmpeg not available …) and the LLM falls back to a text reply.

Requirements

Both plugins.speech.stt_model and plugins.speech.tts_model must be set in config.yml. The Speech plugin must be enabled and running before the Telegram plugin starts.

File & Media Attachments

The Telegram plugin downloads incoming attachments and forwards them to the conversation. The LLM sees them in timeline order — it knows which file was most recently sent without any special indexing.

Type	Saved to disk	How it reaches the LLM
Document (PDF, ZIP, …)	`data/uploads/telegram/<chat_id>/<filename>`	Structured `metadata.attachments` (shared with web/mobile)
Photo	`data/uploads/telegram/<chat_id>/<file_id>.jpg`	Structured `metadata.attachments`
Location	—	`[TELEGRAM SYSTEM INFO]` text (latitude, longitude, accuracy, Google Maps URL)

Document and Photo are aligned with the web/mobile attachment model: download_and_save returns a core_api::message_meta::Attachment (project-root-relative path so /data/… serves it), handle_attachment puts it in SendMessageOptions.metadata, and the message builder generates the shared [SYSTEM INFO] block on the fly. Viewing the telegram source from the copilot therefore shows these as chips, not raw text. See frontend.md and database.md (chat_history.metadata).

Location has no file, so it keeps the legacy [TELEGRAM SYSTEM INFO] text path (system_info_message). Captions typed alongside a Document/Photo become the user turn's text.

Live locations

When the user shares a live location, two things happen:

Initial message (message event) — the LLM is notified via a [TELEGRAM SYSTEM INFO] message and the position is written to skald.location_manager under the key "telegram".
Subsequent updates (edited_message events) — the position in location_manager is updated silently, with no LLM notification. This keeps the store current for any background scripts or tools that read user_location("telegram").

LocationManager is in-memory only. On restart, the store starts empty and is repopulated as soon as Telegram delivers the next live location tick (typically within seconds if sharing is still active).

The uploads/ directory is gitignored.

Extending attachment types

To add a new type (e.g. sticker, contact):

Add a variant to TelegramAttachment in crates/plugin-telegram-bot/src/attachments.rs.
Implement download_and_save: return Ok(Some(Attachment)) for a file-backed type (it flows into metadata.attachments) or Ok(None) for a file-less one (then add a system_info_message arm and handle it in the None branch of handle_attachment).
Detect the message type in classify_message in handlers.rs and return IncomingEvent::Attachment(...).

Interface Tools

The Telegram plugin can inject custom LLM-callable tools into any session via the interface_tools parameter of SendMessageOptions. These tools are only visible to the root agent — sub-agents do not inherit them.

To add a Telegram-specific tool, construct an InterfaceTool with an OpenAI tool definition and an async handler closure that captures Arc<Bot> and ChatId, then pass it in the interface_tools vec inside SendMessageOptions.

InterfaceTool and ToolFuture are defined in crates/core-api/src/interface_tool.rs (re-exported via crate::chat_hub). AgentRunConfig remains in src/core/session/handler/interface_tools.rs (main crate only).

Secrets directory

secrets/telegram_whitelist.json is gitignored. The directory is created automatically on first pairing request. Never commit this file.

12 KiB Raw Blame History