124 lines
4.8 KiB
Markdown
124 lines
4.8 KiB
Markdown
# WhisperLocal Plugin
|
|
|
|
Local Speech-to-Text via [whisper.cpp](https://github.com/ggerganov/whisper.cpp), Metal-accelerated on Apple Silicon.
|
|
Implemented in pure Rust using the `whisper-rs` crate — no Python involved.
|
|
|
|
---
|
|
|
|
## Setup
|
|
|
|
### 1. Download a GGML model
|
|
|
|
```sh
|
|
mkdir -p models
|
|
curl -L -o models/ggml-large-v3.bin \
|
|
https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-large-v3.bin
|
|
```
|
|
|
|
Other available sizes (smaller = faster, less accurate):
|
|
|
|
| Model | Size | Notes |
|
|
|---|---|---|
|
|
| `ggml-tiny.bin` | ~75 MB | Very fast, lower accuracy |
|
|
| `ggml-base.bin` | ~142 MB | Good balance for testing |
|
|
| `ggml-small.bin` | ~466 MB | Good accuracy |
|
|
| `ggml-medium.bin` | ~1.5 GB | High accuracy |
|
|
| `ggml-large-v3.bin` | ~3.1 GB | Best accuracy, recommended |
|
|
| `ggml-large-v3-turbo.bin` | ~1.6 GB | large-v3 speed-optimised |
|
|
|
|
All models: `https://huggingface.co/ggerganov/whisper.cpp`
|
|
|
|
### 2. Configure `config.yml`
|
|
|
|
```yaml
|
|
plugins:
|
|
whisper_local:
|
|
model: "models/ggml-large-v3.bin"
|
|
language: "it" # BCP-47 code, or "auto" for detection
|
|
load_at_startup: false # false (default) = lazy load on first use
|
|
idle_timeout_secs: 1200 # unload after 20 min idle; 0 = never unload
|
|
```
|
|
|
|
| Option | Default | Effect |
|
|
|---|---|---|
|
|
| `model` | — (required) | Path to the GGML `.bin` file |
|
|
| `language` | `auto` | BCP-47 code or `auto`. Applied live — runtime changes take effect on the next transcription without a reload |
|
|
| `load_at_startup` | `false` | **When** the model first loads: `false` = lazily on the first transcription, `true` = eagerly in `start()` (warm, no first-call latency) |
|
|
| `idle_timeout_secs` | `1200` | **When** the model unloads: after this many seconds of inactivity. `0` = never unload (stays resident once loaded) |
|
|
|
|
The two timing options are orthogonal and cover the whole spectrum:
|
|
|
|
| `load_at_startup` | `idle_timeout_secs` | Behaviour |
|
|
|---|---|---|
|
|
| `false` | `1200` | **Default** — load on first use, free ~2 GB after 20 min idle |
|
|
| `true` | `0` | Always resident — eager load, never unload (legacy behaviour) |
|
|
| `true` | `1200` | Warm at startup, but freed if unused |
|
|
| `false` | `0` | Load on first use, then stay resident |
|
|
|
|
### 3. Build
|
|
|
|
The first `cargo build` compiles whisper.cpp (a few minutes). Subsequent builds are cached.
|
|
|
|
---
|
|
|
|
## How it works
|
|
|
|
```
|
|
Telegram voice message (OGG/Opus)
|
|
│
|
|
▼ ffmpeg → 16 kHz mono WAV
|
|
▼ hound → Vec<f32> PCM samples
|
|
▼ whisper.cpp (Metal GPU) → text
|
|
│
|
|
▼ forwarded to LLM as a normal text message
|
|
```
|
|
|
|
Audio conversion uses the system `ffmpeg` binary (must be installed: `brew install ffmpeg`).
|
|
Inference runs on Apple Silicon GPU via Metal. Falls back to CPU if Metal is unavailable.
|
|
|
|
---
|
|
|
|
## Memory management (lazy load + idle unload)
|
|
|
|
The GGML weights are ~2 GB, so the plugin keeps them in memory only while they are
|
|
actually useful. The model lives in a shared, droppable cell (`LazyModel`):
|
|
|
|
- **`start()`** validates the model path and registers a lightweight transcriber, but
|
|
does **not** load the weights unless `load_at_startup: true`. The registered handle
|
|
holds no strong reference to the weights, so they can be freed at any time.
|
|
- **First transcription** triggers `ensure_loaded()`, which loads the weights once
|
|
(concurrent first-callers wait on a single load) and records a last-used timestamp.
|
|
- A background **eviction task** ticks every 60 s and unloads the model once it has
|
|
been idle for `idle_timeout_secs`. Set `idle_timeout_secs: 0` to disable eviction.
|
|
- **Unloading** is refcount-safe: an in-flight transcription holds its own handle to
|
|
the weights, so memory is reclaimed only after it finishes. The actual free runs on
|
|
a blocking thread (whisper.cpp GPU cleanup).
|
|
|
|
Trade-off: after an unload, the next transcription pays the reload cost (a few
|
|
seconds). The OS page cache usually keeps the `.bin` warm, so the reload is mostly
|
|
memory copy + Metal allocation rather than disk I/O. Use `load_at_startup: true` /
|
|
`idle_timeout_secs: 0` if you prefer zero first-call latency over reclaiming the RAM.
|
|
|
|
---
|
|
|
|
## Integration with TranscribeManager
|
|
|
|
`WhisperLocalPlugin` does **not** expose itself as `Arc<dyn Transcribe>` directly. At `start()` it registers a lightweight `WhisperLocalTranscriber` handle into `skald.transcribe_manager`; at `stop()` it deregisters it. Callers never reference the plugin type — they ask the manager:
|
|
|
|
```rust
|
|
if let Some(t) = skald.transcribe_manager.get().await {
|
|
let text = t.transcribe(audio, "ogg").await?;
|
|
}
|
|
```
|
|
|
|
See [../plugins.md](../plugins.md) for the `TranscribeManager` API and the `Transcribe` trait.
|
|
|
|
---
|
|
|
|
## When to Update This File
|
|
|
|
- The audio conversion pipeline changes
|
|
- Default recommended models change
|
|
- Registration/deregistration logic in `start()`/`stop()` changes
|
|
- The lazy-load / idle-eviction lifecycle or its config options change
|