First Version
This commit is contained in:
@@ -0,0 +1,248 @@
|
||||
# Image Generation
|
||||
|
||||
Framework for generating images from text prompts. Supports DB-backed providers (configured via UI) and plugin-registered providers (ephemeral, registered at runtime).
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
```text
|
||||
crates/core-api/src/image_generate.rs
|
||||
— ImageGenerate trait (provider interface)
|
||||
— ImageGenerateRegistry trait (plugin write-side: register/unregister)
|
||||
|
||||
src/image_generate/
|
||||
mod.rs — record types, re-exports ImageGenerate from core-api
|
||||
manager.rs — ImageGeneratorManager (DB-backed + plugin slots, impls ImageGenerateRegistry)
|
||||
db.rs — CRUD for image_generate_models table
|
||||
openrouter_image.rs — OpenRouterImageGenerator (chat completions + modalities)
|
||||
|
||||
src/tools/
|
||||
image_generate.rs — LLM tools: image_generate_providers_list, image_generate
|
||||
|
||||
src/api/
|
||||
image_generate_models.rs — REST CRUD for image_generate_models
|
||||
images.rs — GET /api/images/:id (serve generated files)
|
||||
```
|
||||
|
||||
Two kinds of providers coexist:
|
||||
|
||||
| Kind | Source | Example |
|
||||
| ---- | ------ | ------- |
|
||||
| **DB-backed** | Rows in `image_generate_models`, built from `llm_providers` credentials | OpenRouter `x-ai/grok-2-vision` |
|
||||
| **Plugin-registered** | Ephemeral — registered at runtime by plugins | future: `StableDiffusionPlugin` |
|
||||
|
||||
Plugin-registered providers take precedence over DB-backed ones in `get()`.
|
||||
|
||||
---
|
||||
|
||||
## Traits (crates/core-api)
|
||||
|
||||
```rust
|
||||
// core_api::image_generate
|
||||
#[async_trait]
|
||||
pub trait ImageGenerate: Send + Sync {
|
||||
fn id(&self) -> &str;
|
||||
fn name(&self) -> &str;
|
||||
async fn generate(&self, prompt: &str) -> Result<Vec<u8>>; // raw PNG bytes
|
||||
}
|
||||
|
||||
/// Write-side used by plugins to register/unregister ephemeral providers.
|
||||
/// Implemented by ImageGeneratorManager in the main crate.
|
||||
#[async_trait]
|
||||
pub trait ImageGenerateRegistry: Send + Sync {
|
||||
async fn register(&self, provider: Arc<dyn ImageGenerate>);
|
||||
async fn unregister(&self, id: &str);
|
||||
}
|
||||
```
|
||||
|
||||
`ImageGenerateRegistry` is also available on `PluginContext` as `ctx.image_generate_registry`, so plugin crates that depend only on `core-api` can register providers without importing anything from the main crate.
|
||||
|
||||
---
|
||||
|
||||
## Manager API
|
||||
|
||||
```rust
|
||||
// Async constructor — loads DB models on startup
|
||||
ImageGeneratorManager::new(pool: Arc<SqlitePool>, data_root: impl Into<PathBuf>)
|
||||
-> Result<Arc<Self>>
|
||||
|
||||
// Plugin registration (ephemeral — called by a plugin's start()/stop())
|
||||
image_generator_manager.register(Arc::new(provider)).await;
|
||||
image_generator_manager.unregister("my_provider_id").await;
|
||||
|
||||
// DB-backed CRUD (called by REST API handlers)
|
||||
image_generator_manager.add_model(record).await // → Result<i64>
|
||||
image_generator_manager.update_model(id, record).await
|
||||
image_generator_manager.delete_model(id).await // soft delete
|
||||
image_generator_manager.get_model(id).await // → Option<ImageGenerateModelRecord>
|
||||
|
||||
// Listings
|
||||
image_generator_manager.list_models_info().await // DB-backed only → Vec<ImageGenerateModelInfo>
|
||||
image_generator_manager.list_all_info().await // plugin + DB → Vec<ImageGenerateModelInfo>
|
||||
image_generator_manager.list().await // lightweight → Vec<ImageGenerateInfo> (for LLM tool)
|
||||
|
||||
// Resolution
|
||||
image_generator_manager.get(id).await // → Option<Arc<dyn ImageGenerate>>
|
||||
|
||||
// Generation (called by image_generate tool via block_in_place)
|
||||
image_generator_manager.generate(provider_id, prompt).await // → Result<(PathBuf, String)> (path, url)
|
||||
|
||||
// Image storage path
|
||||
image_generator_manager.images_dir() // → PathBuf (data/images/)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## LLM Tools
|
||||
|
||||
Two tools are injected per-turn when at least one provider is active (absent otherwise):
|
||||
|
||||
### `image_generate_providers_list`
|
||||
|
||||
Lists all currently active image generation providers.
|
||||
|
||||
```text
|
||||
Parameters: (none)
|
||||
Returns: JSON array of {id: string, name: string}
|
||||
```
|
||||
|
||||
### `image_generate`
|
||||
|
||||
Generates an image synchronously. Blocks the tool round until the image is ready.
|
||||
|
||||
```text
|
||||
Parameters:
|
||||
provider_id string (required) — ID from image_generate_providers_list
|
||||
prompt string (required) — text prompt
|
||||
|
||||
Returns: {"path": "/abs/path/data/images/<id>.png", "url": "/api/images/<id>"}
|
||||
```
|
||||
|
||||
**Typical agent flow:**
|
||||
|
||||
```text
|
||||
1. image_generate_providers_list() → [{id: "grok-imagine", name: "grok-imagine"}]
|
||||
2. image_generate("grok-imagine", "a red sunset") → {"path": "...", "url": "/api/images/abc123"}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Image Storage
|
||||
|
||||
Generated images are written to `data/images/<random_id>.png` (relative to the working directory). The directory is created automatically on first use.
|
||||
|
||||
---
|
||||
|
||||
## REST API
|
||||
|
||||
### Image serving
|
||||
|
||||
```text
|
||||
GET /api/images/:id
|
||||
```
|
||||
|
||||
Serves the generated PNG. Returns `404` if the file does not exist, `400` for invalid IDs. No authentication (local server).
|
||||
|
||||
### Model management
|
||||
|
||||
```text
|
||||
GET /api/image-generate/models — list all active providers (plugin + DB)
|
||||
POST /api/image-generate/models — add a DB-backed model
|
||||
GET /api/image-generate/models/{id} — get a model record
|
||||
PUT /api/image-generate/models/{id} — update a model record
|
||||
DELETE /api/image-generate/models/{id} — soft-delete a model
|
||||
```
|
||||
|
||||
**POST / PUT body:**
|
||||
|
||||
```json
|
||||
{
|
||||
"provider_id": 1,
|
||||
"model_id": "x-ai/grok-2-vision",
|
||||
"name": "grok-imagine",
|
||||
"priority": 100
|
||||
}
|
||||
```
|
||||
|
||||
`name` becomes the `provider_id` used in the `image_generate` LLM tool. If omitted, `model_id` is used.
|
||||
|
||||
---
|
||||
|
||||
## OpenRouter provider
|
||||
|
||||
`OpenRouterImageGenerator` calls the OpenRouter chat completions endpoint with `modalities: ["image"]` (image-only — do **not** use `["image", "text"]`, which is for multimodal models and causes a 404 on image-only models like `grok-imagine-image-quality`). The response image is returned as a base64 data URL at `choices[0].message.images[0].image_url.url`.
|
||||
|
||||
To register an OpenRouter image model:
|
||||
1. Add/use an existing `llm_providers` row with `type = "open_router"` and a valid API key.
|
||||
2. `POST /api/image-generate/models` with that `provider_id` and the desired `model_id`.
|
||||
|
||||
---
|
||||
|
||||
## Plugin Registration
|
||||
|
||||
Plugin crates depend only on `core-api` — no reference to the main crate needed.
|
||||
|
||||
```rust
|
||||
// In crates/plugin-foo/Cargo.toml:
|
||||
// core-api = { path = "../core-api" }
|
||||
|
||||
use core_api::image_generate::ImageGenerate;
|
||||
|
||||
struct MyImageGenerator { /* ... */ }
|
||||
|
||||
#[async_trait]
|
||||
impl ImageGenerate for MyImageGenerator {
|
||||
fn id(&self) -> &str { "my_generator" }
|
||||
fn name(&self) -> &str { "My Generator" }
|
||||
async fn generate(&self, prompt: &str) -> Result<Vec<u8>> {
|
||||
// call external API or local model, return PNG bytes
|
||||
}
|
||||
}
|
||||
|
||||
// In Plugin::reload() when enabled:
|
||||
ctx.image_generate_registry.register(Arc::new(MyImageGenerator { ... })).await;
|
||||
|
||||
// In Plugin::stop() or reload() when disabled:
|
||||
ctx.image_generate_registry.unregister("my_generator").await;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ComfyUI plugin (`crates/plugin-comfyui`)
|
||||
|
||||
Each JSON file in `data/comfyui/workflows/` becomes a separate `ImageGenerate`
|
||||
provider. The plugin monitors ComfyUI health every 5s and unregisters all
|
||||
providers if the server is unreachable.
|
||||
|
||||
Workflow files must be exported from ComfyUI as "API Format". The plugin reads
|
||||
an optional `_personal_agent` key for metadata:
|
||||
|
||||
```json
|
||||
"_personal_agent": {
|
||||
"name": "Realistic Portrait",
|
||||
"description": "Ritratti realistici, formato verticale. Default 768×1024.",
|
||||
"prompt_node": "6",
|
||||
"negative_prompt_node": "7",
|
||||
"prompt_field": "clip_l",
|
||||
"prompt_field_extra": ["clip_g", "t5xxl"],
|
||||
"extra_params": { "width_node": "8", "height_node": "8", "steps_node": "3" }
|
||||
}
|
||||
```
|
||||
|
||||
- `prompt_field` (optional): input field to write the prompt into. Default `"text"` (for `CLIPTextEncode`). Use `"clip_l"` for `CLIPTextEncodeSD3`.
|
||||
- `prompt_field_extra` (optional): additional input fields to copy the same prompt into. For SD3.5: `["clip_g", "t5xxl"]`.
|
||||
- `negative_prompt_field` / `negative_prompt_field_extra`: same for the negative prompt node.
|
||||
|
||||
Provider id: `comfyui-{filename}` (e.g. `realistic-portrait.json` → `comfyui-realistic-portrait`).
|
||||
|
||||
See [../comfyui-workflow-format.md](../comfyui-workflow-format.md) for the complete
|
||||
guide on reading and modifying workflow files.
|
||||
|
||||
---
|
||||
|
||||
## When to Update This File
|
||||
|
||||
- A new concrete `ImageGenerate` implementation is added (e.g. a new provider backend)
|
||||
- Image storage path or REST endpoint changes
|
||||
- LLM tool signatures change
|
||||
@@ -0,0 +1,11 @@
|
||||
# Model Providers
|
||||
|
||||
Trait contracts and implementations for LLM, TTS, transcription, and image generation backends.
|
||||
|
||||
## Files
|
||||
|
||||
- [tts.md](tts.md) — Text-to-Speech: trait, manager, provider catalogue, tts_models DB table
|
||||
- [transcribe.md](transcribe.md) — Speech-to-Text: OpenAI-compatible audio API, transcribe_models DB table
|
||||
- [image.md](image.md) — Image generation: trait, manager, async task system, LLM tools, REST endpoint
|
||||
|
||||
See [../index.md#model-providers](../index.md#model-providers) for navigation. See also [../llm-clients.md](../llm-clients.md) for LLM client trait and selection.
|
||||
@@ -0,0 +1,148 @@
|
||||
# Transcription Providers
|
||||
|
||||
Cloud Speech-to-Text via any OpenAI-compatible audio transcription endpoint.
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
```text
|
||||
crates/core-api/src/transcribe.rs
|
||||
— Transcribe trait (provider interface)
|
||||
— TranscribeProvider trait (resolve active provider)
|
||||
— TranscribeRegistry trait (plugin write-side: register/unregister)
|
||||
— TranscribeModelRecord (DB record type — moved here from main crate)
|
||||
— RemoteTranscribeModelInfo (remote catalog type — moved here from main crate)
|
||||
|
||||
src/transcribe/
|
||||
mod.rs — TranscribeModelInfo (API response type), re-exports from core-api
|
||||
db.rs — SQL layer for transcribe_models table
|
||||
manager.rs — TranscribeManager (DB-aware, owns the table)
|
||||
openai_audio.rs — OpenAiAudioTranscriber: impl Transcribe via HTTP multipart
|
||||
elevenlabs_audio.rs — ElevenLabsTranscriber: impl Transcribe via ElevenLabs Scribe API
|
||||
```
|
||||
|
||||
`TranscribeManager` holds two kinds of providers:
|
||||
|
||||
| Kind | Source | Example |
|
||||
| ---- | ------ | ------- |
|
||||
| **DB-backed** | `transcribe_models` table, built from `llm_providers` credentials | `OpenAiAudioTranscriber` |
|
||||
| **Plugin-registered** | Ephemeral — registered at runtime by plugins | `WhisperLocalTranscriber` |
|
||||
|
||||
`get()` returns the first plugin provider (if any is running), then falls back to the first DB-backed provider ordered by `priority ASC`. Callers never reference a concrete type:
|
||||
|
||||
```rust
|
||||
if let Some(t) = skald.transcribe_manager.get().await {
|
||||
let text = t.transcribe(audio, "ogg").await?;
|
||||
}
|
||||
```
|
||||
|
||||
### Manager API
|
||||
|
||||
```rust
|
||||
// DB-backed CRUD — only TranscribeManager touches transcribe_models
|
||||
transcribe_manager.add_model(record).await?
|
||||
transcribe_manager.update_model(id, record).await?
|
||||
transcribe_manager.delete_model(id).await? // soft-delete
|
||||
transcribe_manager.get_model(id).await // → Option<TranscribeModelRecord>
|
||||
transcribe_manager.list_models_info().await // → Vec<TranscribeModelInfo> (DB-backed only)
|
||||
transcribe_manager.list_all_info().await // → Vec<TranscribeModelInfo> (plugins first, then DB — used by API)
|
||||
|
||||
// Remote model catalog (calls ApiProvider::list_transcribe_models)
|
||||
transcribe_manager.list_provider_models(provider_id).await // → Result<Vec<RemoteTranscribeModelInfo>>
|
||||
|
||||
// Plugin registration (ephemeral — called by WhisperLocalPlugin)
|
||||
transcribe_manager.register(Arc::new(transcriber)).await
|
||||
transcribe_manager.unregister("whisper_local").await
|
||||
```
|
||||
|
||||
`RemoteTranscribeModelInfo` fields: `id`, `name`, `description`, `languages` (BCP-47 codes).
|
||||
|
||||
### REST API
|
||||
|
||||
| Method | Path | Description |
|
||||
| ------ | ---- | ----------- |
|
||||
| `GET` | `/api/transcribe/models` | List all models — plugin-registered first (`from_plugin: true`), then DB-backed |
|
||||
| `POST` | `/api/transcribe/models` | Add a new transcription model |
|
||||
| `GET` | `/api/transcribe/models/{id}` | Get a DB-backed model record |
|
||||
| `PUT` | `/api/transcribe/models/{id}` | Update a DB-backed model |
|
||||
| `DELETE` | `/api/transcribe/models/{id}` | Soft-delete a DB-backed model |
|
||||
| `GET` | `/api/transcribe/providers/{id}/models` | List remote transcription models from a configured provider (`RemoteTranscribeModelInfo[]`) |
|
||||
|
||||
---
|
||||
|
||||
## OpenAiAudioTranscriber
|
||||
|
||||
Implemented in `src/core/transcribe/openai_audio.rs`.
|
||||
|
||||
Calls `POST {base_url}/audio/transcriptions` with a `multipart/form-data` body:
|
||||
|
||||
| Field | Value |
|
||||
| ---------- | ------------------------------------------------ |
|
||||
| `file` | Raw audio bytes with extension-derived MIME type |
|
||||
| `model` | Provider model ID (e.g. `openai/whisper-1`) |
|
||||
| `language` | BCP-47 code (optional — omitted for auto-detect) |
|
||||
|
||||
Accepted formats: `ogg`, `mp3`, `mp4`, `m4a`, `wav`, `webm`, `flac`.
|
||||
No local conversion needed — the provider handles decoding server-side.
|
||||
|
||||
### Supported providers
|
||||
|
||||
| Provider | `base_url` | Notes |
|
||||
| -------- | ---------- | ----- |
|
||||
| OpenRouter | `https://openrouter.ai/api/v1` | Model: `openai/whisper-1`, etc. |
|
||||
| OpenAI | `https://api.openai.com/v1` | Model: `whisper-1` |
|
||||
|
||||
---
|
||||
|
||||
## ElevenLabsTranscriber
|
||||
|
||||
Implemented in `src/core/transcribe/elevenlabs_audio.rs`.
|
||||
|
||||
Calls `POST https://api.elevenlabs.io/v1/speech-to-text` with auth header `xi-api-key` (not Bearer) and a `multipart/form-data` body:
|
||||
|
||||
| Field | Value |
|
||||
| ----- | ----- |
|
||||
| `file` | Raw audio bytes |
|
||||
| `model_id` | ElevenLabs Scribe model (e.g. `scribe_v1`) — stored as `model_id` in the DB record |
|
||||
|
||||
Returns `{ "text": "..." }`. Provider type: `elevenlabs`.
|
||||
|
||||
`ElevenLabsProvider::list_transcribe_models()` calls `GET https://api.elevenlabs.io/v1/models` and filters entries whose `model_id` starts with `scribe` (or `can_do_voice_conversion: true` as a fallback). Returns `RemoteTranscribeModelInfo` with id, name, description, and supported languages.
|
||||
|
||||
Which providers support transcription is declared statically via `ApiProvider::supported_types()` — see [llm-clients.md](llm-clients.md#apiprovider--service-types).
|
||||
|
||||
---
|
||||
|
||||
## DB: transcribe_models table
|
||||
|
||||
```sql
|
||||
CREATE TABLE transcribe_models (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
provider_id INTEGER NOT NULL REFERENCES llm_providers(id),
|
||||
model_id TEXT NOT NULL,
|
||||
name TEXT NOT NULL UNIQUE,
|
||||
language TEXT, -- BCP-47 or NULL for auto-detect
|
||||
priority INTEGER NOT NULL DEFAULT 100,
|
||||
removed_at TEXT,
|
||||
created_at TEXT NOT NULL DEFAULT (datetime('now')),
|
||||
UNIQUE(provider_id, model_id)
|
||||
)
|
||||
```
|
||||
|
||||
`provider_id` references `llm_providers` — the same provider table used by LLM models.
|
||||
Only providers that declare `ServiceType::Transcribe` in `supported_types()` should have rows here.
|
||||
|
||||
---
|
||||
|
||||
## DB insert — soft-delete revival
|
||||
|
||||
`transcribe_models` has two `UNIQUE` constraints: `name` and `(provider_id, model_id)`. `db::insert()` attempts to revive a soft-deleted row before falling back to a plain `INSERT` — same pattern as `tts/db.rs`. See [tts-providers.md](tts-providers.md#db-insert--soft-delete-revival) for the full description.
|
||||
|
||||
---
|
||||
|
||||
## When to Update This File
|
||||
|
||||
- A new concrete `Transcribe` implementation is added
|
||||
- `transcribe_models` schema changes
|
||||
- A provider gains or loses transcription support
|
||||
@@ -0,0 +1,332 @@
|
||||
# Text-to-Speech Providers
|
||||
|
||||
Cloud TTS via OpenAI-compatible or ElevenLabs endpoints, plus plugin-registered local engines.
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
```text
|
||||
crates/core-api/src/tts.rs
|
||||
— TextToSpeech trait (provider interface)
|
||||
— TtsProvider trait (resolve active provider)
|
||||
— TtsRegistry trait (plugin write-side: register/unregister)
|
||||
— TtsModelRecord (DB record type — moved here from main crate)
|
||||
— RemoteTtsModelInfo (remote catalog type — moved here from main crate)
|
||||
|
||||
src/core/tts/
|
||||
mod.rs — TtsModelInfo (API response type), re-exports from core-api
|
||||
db.rs — SQL layer for tts_models table
|
||||
manager.rs — TtsManager (DB-aware, owns the table, impls TtsProvider + TtsRegistry)
|
||||
openai_tts.rs — OpenAiTtsSynthesiser: impl TextToSpeech via OpenAI-compatible HTTP JSON
|
||||
|
||||
crates/plugin-elevenlabs/src/lib.rs
|
||||
ElevenLabsTtsSynthesiser: impl TextToSpeech via ElevenLabs v1 API
|
||||
ElevenLabsTranscriber: impl Transcribe via ElevenLabs Scribe API
|
||||
ElevenLabsProvider: impl ApiProvider (model listing, build_tts, build_transcriber)
|
||||
ElevenLabsPlugin: impl Plugin — registers ElevenLabsProvider on start
|
||||
```
|
||||
|
||||
Two kinds of providers coexist:
|
||||
|
||||
| Kind | Source | Example |
|
||||
| ---- | ------ | ------- |
|
||||
| **DB-backed** | `tts_models` table, built from `llm_providers` credentials | `OpenAiTtsSynthesiser`, `ElevenLabsTtsSynthesiser` |
|
||||
| **Plugin-registered** | Ephemeral — registered at runtime by plugins | `OrpheusTtsPlugin`, `KokoroTtsPlugin` |
|
||||
|
||||
`get()` returns the first plugin provider (if any is running), then the first DB-backed provider ordered by `priority ASC`.
|
||||
|
||||
---
|
||||
|
||||
## Traits (crates/core-api)
|
||||
|
||||
```rust
|
||||
// core_api::tts
|
||||
#[async_trait]
|
||||
pub trait TextToSpeech: Send + Sync {
|
||||
fn id(&self) -> &str;
|
||||
fn name(&self) -> &str;
|
||||
fn description(&self) -> Option<&str>; // default None
|
||||
fn instructions(&self) -> Option<&str>; // default voice style stored in DB
|
||||
async fn synthesize(&self, text: &str, instructions: Option<&str>) -> Result<Vec<u8>>;
|
||||
}
|
||||
|
||||
/// Read-side used by callers to get the active provider.
|
||||
#[async_trait]
|
||||
pub trait TtsProvider: Send + Sync {
|
||||
async fn get(&self) -> Option<Arc<dyn TextToSpeech>>;
|
||||
}
|
||||
|
||||
/// Write-side used by plugins to register/unregister ephemeral providers.
|
||||
#[async_trait]
|
||||
pub trait TtsRegistry: Send + Sync {
|
||||
async fn register(&self, provider: Arc<dyn TextToSpeech>);
|
||||
async fn unregister(&self, id: &str);
|
||||
}
|
||||
```
|
||||
|
||||
### `instructions` semantics
|
||||
|
||||
| Level | Where set | Precedence |
|
||||
|-------|-----------|------------|
|
||||
| **DB-level** | `tts_models.instructions` column | Default for this model config |
|
||||
| **Call-time** | `synthesize(text, Some(override))` | Overrides DB-level for this call |
|
||||
|
||||
This lets the LLM (or a plugin) say "respond in a cheerful tone" on a per-turn basis without changing the model's default configuration.
|
||||
|
||||
---
|
||||
|
||||
## Manager API
|
||||
|
||||
```rust
|
||||
// Async constructor — loads DB models on startup
|
||||
TtsManager::new(pool: Arc<SqlitePool>, registry: Arc<ProviderRegistry>) -> Result<Arc<Self>>
|
||||
|
||||
// Resolution
|
||||
tts_manager.get().await // → Option<Arc<dyn TextToSpeech>> (plugins first, then DB)
|
||||
|
||||
// Plugin registration (ephemeral)
|
||||
tts_manager.register(Arc::new(synthesiser)).await
|
||||
tts_manager.unregister("kokoro_local").await
|
||||
|
||||
// DB-backed CRUD (called by REST API handlers)
|
||||
tts_manager.add_model(record).await // → Result<i64>
|
||||
tts_manager.update_model(id, record).await
|
||||
tts_manager.delete_model(id).await // soft delete
|
||||
tts_manager.get_model(id).await // → Option<TtsModelRecord>
|
||||
|
||||
// Listings
|
||||
tts_manager.list_models_info().await // DB-backed only → Vec<TtsModelInfo>
|
||||
tts_manager.list_all_info().await // plugin + DB → Vec<TtsModelInfo>
|
||||
|
||||
// Remote model catalog (calls ApiProvider::list_tts_models)
|
||||
tts_manager.list_provider_models(provider_id).await // → Result<Vec<RemoteTtsModelInfo>>
|
||||
```
|
||||
|
||||
`RemoteTtsModelInfo` fields: `id`, `name`, `description`, `languages` (BCP-47 codes), `cost_factor: Option<f64>` (relative cost multiplier, e.g. `1.0` = standard), `instructions: Option<String>` (usage guidance for LLM and UI pre-fill).
|
||||
|
||||
---
|
||||
|
||||
## OpenAiTtsSynthesiser
|
||||
|
||||
Implemented in `src/core/tts/openai_tts.rs`.
|
||||
|
||||
Calls `POST {base_url}/audio/speech` with a JSON body:
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| `model` | Provider model ID (e.g. `tts-1`, `tts-1-hd`, `gpt-4o-mini-tts`) |
|
||||
| `input` | Text to synthesise |
|
||||
| `voice` | From `tts_models.voice_id`; `NULL` ⇒ `"alloy"` |
|
||||
| `response_format` | From `tts_models.response_format`; `NULL` ⇒ `"mp3"` |
|
||||
| `instructions` | Optional natural-language style/tone/speed override |
|
||||
|
||||
Returns raw audio bytes in the requested `response_format` (default `mp3`).
|
||||
|
||||
### `voice`
|
||||
|
||||
The `voice` field is taken from the per-model `tts_models.voice_id` column, falling back to `alloy` when unset. Voice names are **provider-specific**: OpenAI uses `alloy`/`echo`/`fable`/`onyx`/`nova`/`shimmer`; Gemini uses `Kore`/`Puck`/`Zephyr`/`Charon`/… An unknown voice name can make the provider error (OpenRouter→Gemini surfaces this as a generic `500`), so set `voice_id` to a value valid for the chosen model.
|
||||
|
||||
### `response_format`
|
||||
|
||||
The audio container/codec is taken from the per-model `tts_models.response_format` column (`mp3`, `opus`, `aac`, `flac`, `wav`, `pcm`). Leaving it empty falls back to `mp3`. Some models reject `mp3` and require a specific value — e.g. `google/gemini-*-tts-*` on OpenRouter returns `400 … only supports response_format="pcm"`. Set the column (UI dropdown in the model form) to the value the model demands.
|
||||
|
||||
> **Note:** `pcm` is raw, headerless audio. Consumers that need a playable container handle the transcode themselves — the Telegram `send_voice_message` tool converts whatever `output_format` reports (mp3/wav/**pcm**/…) to Ogg/Opus via ffmpeg before sending. See [plugins/telegram.md](../plugins/telegram.md).
|
||||
|
||||
### `output_format()`
|
||||
|
||||
`TextToSpeech::output_format()` reports the container/codec of the bytes returned by `synthesize` (`mp3`, `opus`, `wav`, `pcm`, …; default `"mp3"`). `OpenAiTtsSynthesiser` returns its configured `response_format`. Consumers that need a specific container use this to decide whether and how to transcode — e.g. raw `pcm` is headerless and must be described to the decoder, so the hint is essential there.
|
||||
|
||||
### Supported providers
|
||||
|
||||
| Provider | `base_url` | Notes |
|
||||
| -------- | ---------- | ----- |
|
||||
| OpenAI | `https://api.openai.com/v1` | Models: `tts-1`, `tts-1-hd`, `gpt-4o-mini-tts` |
|
||||
| OpenRouter | `https://openrouter.ai/api/v1` | OpenAI-compatible endpoint |
|
||||
|
||||
---
|
||||
|
||||
## ElevenLabsTtsSynthesiser
|
||||
|
||||
Implemented in `crates/plugin-elevenlabs/src/lib.rs` (via `plugin-elevenlabs`).
|
||||
|
||||
Calls `POST https://api.elevenlabs.io/v1/text-to-speech/{voice_id}` with auth header `xi-api-key` (not Bearer).
|
||||
|
||||
| Field in DB record | Meaning |
|
||||
| ------------------ | ------- |
|
||||
| `model_id` | ElevenLabs **generation model** (e.g. `eleven_multilingual_v2`, `eleven_turbo_v2_5`) |
|
||||
| `voice_id` | ElevenLabs **voice ID** (e.g. `21m00Tcm4TlvDq8ikWAM`). Required for ElevenLabs. |
|
||||
| `instructions` | Injected into LLM system prompt; not sent to ElevenLabs API |
|
||||
|
||||
**Legacy fallback:** if `voice_id` is `NULL` (records created before the field split), `model_id` is treated as the voice ID and the generation model defaults to `eleven_multilingual_v2`. This keeps existing records working after the migration.
|
||||
|
||||
Returns raw MP3 bytes. Provider type: `elevenlabs` — requires an `xi-api-key` stored in `llm_providers.api_key`. No `base_url` needed.
|
||||
|
||||
### Remote model catalog
|
||||
|
||||
`ElevenLabsProvider::list_tts_models()` calls `GET https://api.elevenlabs.io/v1/models`, filters entries where `can_do_text_to_speech: true`, and returns `RemoteTtsModelInfo` with:
|
||||
|
||||
- `cost_factor` from the `token_cost_factor` field
|
||||
- `instructions` from `elevenlabs_tts_instructions(model_id)` — per-model usage guidance (supported tags, non-verbal sound syntax, etc.)
|
||||
|
||||
---
|
||||
|
||||
## REST API
|
||||
|
||||
| Method | Path | Description |
|
||||
| ------ | ---- | ----------- |
|
||||
| `GET` | `/api/tts/models` | All models — plugin-registered first (`from_plugin: true`), then DB-backed |
|
||||
| `POST` | `/api/tts/models` | Add a new TTS model |
|
||||
| `GET` | `/api/tts/models/{id}` | Get a DB-backed model record |
|
||||
| `PUT` | `/api/tts/models/{id}` | Update a DB-backed model |
|
||||
| `DELETE` | `/api/tts/models/{id}` | Soft-delete a DB-backed model |
|
||||
| `GET` | `/api/tts/providers/{id}/models` | List remote TTS models from a configured provider (`RemoteTtsModelInfo[]`) |
|
||||
|
||||
The provider models endpoint calls `TtsManager::list_provider_models()` → `ApiProvider::list_tts_models()`. Returns an error if the provider does not support model listing.
|
||||
|
||||
Handled by `src/frontend/api/tts_models.rs`.
|
||||
|
||||
---
|
||||
|
||||
## DB: tts_models table
|
||||
|
||||
```sql
|
||||
CREATE TABLE tts_models (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
provider_id INTEGER NOT NULL REFERENCES llm_providers(id),
|
||||
model_id TEXT NOT NULL, -- generation model (e.g. eleven_multilingual_v2, tts-1-hd)
|
||||
voice_id TEXT, -- speaker voice (required for ElevenLabs; NULL for OpenAI)
|
||||
name TEXT NOT NULL UNIQUE,
|
||||
description TEXT, -- human-readable, shown in UI
|
||||
instructions TEXT, -- default voice style / tone / speed
|
||||
response_format TEXT, -- audio format (mp3/opus/aac/flac/wav/pcm); NULL ⇒ mp3
|
||||
priority INTEGER NOT NULL DEFAULT 100,
|
||||
removed_at TEXT,
|
||||
created_at TEXT NOT NULL DEFAULT (datetime('now')),
|
||||
UNIQUE(provider_id, model_id)
|
||||
)
|
||||
```
|
||||
|
||||
`voice_id` was added in **schema version 2** via `ALTER TABLE tts_models ADD COLUMN voice_id TEXT`. `response_format` was added in **schema version 18** via `ALTER TABLE tts_models ADD COLUMN response_format TEXT`. See [database.md](database.md#migration-pattern).
|
||||
|
||||
---
|
||||
|
||||
## Plugin Registration
|
||||
|
||||
`TtsRegistry` is exposed on `PluginContext` as `ctx.tts_registry`. Plugin crates depend only on `core-api`.
|
||||
|
||||
```rust
|
||||
use core_api::tts::TextToSpeech;
|
||||
|
||||
struct MyTtsSynth { /* ... */ }
|
||||
|
||||
#[async_trait]
|
||||
impl TextToSpeech for MyTtsSynth {
|
||||
fn id(&self) -> &str { "kokoro_local" }
|
||||
fn name(&self) -> &str { "Kokoro Local" }
|
||||
async fn synthesize(&self, text: &str, instructions: Option<&str>) -> Result<Vec<u8>> {
|
||||
// call local engine, return MP3 bytes
|
||||
}
|
||||
}
|
||||
|
||||
// In Plugin::start() when enabled:
|
||||
ctx.tts_registry.register(Arc::new(MyTtsSynth { ... })).await;
|
||||
|
||||
// In Plugin::stop():
|
||||
ctx.tts_registry.unregister("kokoro_local").await;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Kokoro TTS (`plugin-tts-kokoro`)
|
||||
|
||||
Lightweight local TTS using the Kokoro ONNX model (~310 MB model + ~27 MB voices). No PyTorch or GPU required — runs fully on CPU via ONNX Runtime.
|
||||
|
||||
**Crate:** `crates/plugin-tts-kokoro/`
|
||||
**Plugin ID:** `kokoro_tts`
|
||||
|
||||
### How it works
|
||||
|
||||
The Python server (`kokoro_server.py`) is embedded in the crate via `include_str!`. On start the plugin writes it to a temp path and spawns it as a FastAPI subprocess. The server downloads `kokoro-v1.0.onnx` and `voices-v1.0.bin` from GitHub Releases on first run, then exposes `POST /synthesize` returning WAV bytes. The plugin registers itself with `TtsManager` and deregisters on stop.
|
||||
|
||||
### Setup
|
||||
|
||||
```text
|
||||
toggle_item(kind="plugin", id="kokoro_tts", enabled=true)
|
||||
```
|
||||
|
||||
Optional config:
|
||||
|
||||
```json
|
||||
{ "voice": "if_sara", "lang": "it", "speed": 1.0 }
|
||||
```
|
||||
|
||||
### Config
|
||||
|
||||
| Field | Values | Default |
|
||||
| ----- | ------ | ------- |
|
||||
| `voice` | Any Kokoro voice ID (e.g. `if_sara`, `im_nicola`, `af_heart`) | `if_sara` |
|
||||
| `lang` | BCP-47 language code | `it` |
|
||||
| `speed` | Speech rate multiplier | `1.0` |
|
||||
|
||||
Python deps (in `requirements.txt`): `kokoro-onnx`, `soundfile`.
|
||||
|
||||
---
|
||||
|
||||
## Orpheus TTS 3B (`plugin-tts-orpheus-3b`)
|
||||
|
||||
Local, on-device TTS using the Orpheus 3B model. Runs a Python subprocess for inference.
|
||||
|
||||
**Crate:** `crates/plugin-tts-orpheus-3b/`
|
||||
**Plugin ID:** `orpheus_tts_3b`
|
||||
|
||||
**Note:** the FP16 model is large (~6 GB) and uses significant RAM during inference. Prefer `int8` quantization on memory-constrained machines, or use `plugin-tts-kokoro` as a lighter alternative.
|
||||
|
||||
**How it works:** the Python inference server (`orpheus_server.py`) is embedded in the plugin binary via `include_str!`. On start, the plugin writes it to `models/orpheus-3b/orpheus_server.py` and spawns it. The server prints `PORT:<n>` to stdout when ready; the plugin reads that port and registers itself as a `TextToSpeech` provider. On stop, the subprocess is killed.
|
||||
|
||||
**Setup:**
|
||||
|
||||
```text
|
||||
set_secret("HUGGINGFACE_TOKEN", "hf_...")
|
||||
configure_plugin("orpheus_tts_3b", {"quantization": "int8", "voice": "tara"})
|
||||
toggle_item(kind="plugin", id="orpheus_tts_3b", enabled=true)
|
||||
```
|
||||
|
||||
**Config:**
|
||||
|
||||
| Field | Values | Default |
|
||||
| ----- | ------ | ------- |
|
||||
| `quantization` | none / int8 / int4 | int8 |
|
||||
| `voice` | tara / dan / leah / zac / zoe / mia / julia / leo | tara |
|
||||
|
||||
---
|
||||
|
||||
## DB insert — soft-delete revival
|
||||
|
||||
`tts_models` has two `UNIQUE` constraints: `name` and `(provider_id, model_id)`. Soft-deleted rows (where `removed_at IS NOT NULL`) still hold those unique values, which would cause a plain `INSERT` to fail when re-adding a previously deleted model.
|
||||
|
||||
`db::insert()` handles this by first attempting to revive the soft-deleted row: it runs an `UPDATE … RETURNING id` that matches on `removed_at IS NOT NULL AND (provider_id=? AND model_id=? OR name=?)`. If a row is found it is restored with the new values and its `removed_at` is set to `NULL`; only if no match is found does a plain `INSERT` run. The same pattern is applied in `transcribe/db.rs` and `image_generate/db.rs`.
|
||||
|
||||
---
|
||||
|
||||
## Telegram `send_voice_message` tool
|
||||
|
||||
When the Telegram plugin is active and at least one TTS provider is available, the LLM-callable tool `send_voice_message` is injected into every Telegram session. It is absent when no TTS provider is configured.
|
||||
|
||||
| Aspect | Detail |
|
||||
| --- | --- |
|
||||
| Tool name | `send_voice_message` |
|
||||
| Parameter | `text: String` — the text to synthesise |
|
||||
| Provider selection | Highest-priority active provider (`TtsProvider::get()`) |
|
||||
| Transport | `bot.send_voice()` — Telegram voice message |
|
||||
| Instructions | The provider's `instructions()` string is embedded in the tool description so the LLM knows how to format text for that engine |
|
||||
|
||||
The tool resolves the synthesiser at call time (not at registration time), so a TTS provider that becomes available mid-conversation is picked up automatically on the next call.
|
||||
|
||||
---
|
||||
|
||||
## When to Update This File
|
||||
|
||||
- A new concrete `TextToSpeech` implementation is added
|
||||
- `tts_models` schema changes
|
||||
- A provider gains or loses TTS support
|
||||
Reference in New Issue
Block a user