First Version

2026-07-10 15:02:09 +01:00
commit 38494a85a9
562 changed files with 196313 additions and 0 deletions
@@ -0,0 +1,433 @@
+# LLM Clients
+
+## Workspace Location
+
+The `ChatbotClient` trait and all provider implementations live in the standalone crate `crates/llm-client` (no dependencies on the main crate). `src/core/chatbot/mod.rs` is a thin re-export layer. `LoggingChatbotClient` (`src/core/chatbot/logging.rs`) remains in the main crate because it depends on `sqlx`.
+
+---
+
+## ChatbotClient Trait
+
+```rust
+#[async_trait]
+pub trait ChatbotClient: Send + Sync {
+    async fn chat(&self, messages: &[Message], options: &ChatOptions) -> Result<ChatResponse>;
+    async fn chat_with_tools(&self, messages: &[Value], tools: &[Value], options: &ChatOptions) -> Result<LlmTurn>;
+    async fn chat_with_tools_raw(&self, messages: &[Value], tools: &[Value], options: &ChatOptions) -> Result<(LlmTurn, Option<LlmRawMeta>)>;
+}
+```
+
+Only `AnthropicClient` and `OpenAiClient` implement native tool support (`chat_with_tools`). Other clients have a default fallback that strips tool definitions and calls `chat()` instead.
+
+`chat_with_tools_raw` is used by the logging wrapper: it returns the same `LlmTurn` plus raw HTTP request/response metadata (`LlmRawMeta`). `AnthropicClient`, `OpenAiClient`, and `LmStudioClient` override it; all others fall back to calling `chat_with_tools` with no metadata.
+
+`ChatOptions` carries two optional fields — `session_id` and `stack_id` — set by the LLM loop for correlation. Providers ignore them; only `LoggingChatbotClient` reads them.
+
+---
+
+## Transparent Request Logging
+
+`LoggingChatbotClient` (`src/core/chatbot/logging.rs`) is a `ChatbotClient` wrapper that intercepts every `chat_with_tools` call:
+
+1. Calls `inner.chat_with_tools_raw(...)` to capture the HTTP wire data.
+2. Spawns a **fire-and-forget** `tokio::spawn` to insert a row into `llm_requests`.
+3. Returns the `LlmTurn` to the caller unchanged.
+
+The LLM loop is fully unaware — it holds an `Arc<dyn ChatbotClient>` and calls `chat_with_tools` as usual. The wrapper is applied in `LlmManager::build_entry` when `request_log_enabled = true` (set from `config.yml → llm.request_log.enabled`).
+
+What is logged per row: full request body (provider-specific format), request headers (api-key redacted), full response body, response headers, token counts, round-trip duration, session/stack ID.
+
+A background task (boot + every hour) deletes rows older than `retention_days` (default 14).
+
+`LlmTurn` return variants:
+
+- `Message(ChatResponse)` — final text answer
+- `ToolCalls { content, calls, input_tokens, output_tokens, reasoning_content, cost }` — one or more tool calls requested
+
+Both variants carry an optional `reasoning_content: Option<String>`. Populated only by providers that return chain-of-thought (currently DeepSeek thinking mode). Saved to `chat_history.reasoning_content` and echoed back on subsequent turns — see *Reasoning Content / DeepSeek Thinking Mode* below.
+
+Both variants also carry an optional `cost: Option<f64>` — the request price in USD. Populated via the `ChatbotClient::extract_cost(&self, response: &Value)` trait method, whose default reads `usage.cost` from the raw JSON response (OpenRouter and other OpenAI-compatible gateways report it there). Providers that don't bill per-request leave it `None`. `llm_loop` persists it to `chat_history.cost`. Providers with a different response shape can override `extract_cost`.
+
+---
+
+## Provider Registry
+
+Providers are no longer identified by a hard-coded enum. Instead, each provider is a struct implementing the `ApiProvider` trait (`src/core/provider/mod.rs`), registered at startup in `main.rs` via `ProviderRegistry::register_builtin()`. The DB column `llm_providers.type` stores the provider's `type_id` string.
+
+```rust
+// src/provider/mod.rs
+#[async_trait]
+pub trait ApiProvider: Send + Sync {
+    fn type_id(&self) -> &'static str;         // e.g. "open_ai", "anthropic"
+    fn display_name(&self) -> &'static str;
+    fn supported_types(&self) -> &'static [ServiceType];
+
+    // ── Remote model catalogs (default: Ok(None)) ─────────────────────────────
+    async fn list_llm_models(&self, record: &LlmProviderRecord) -> Result<Option<Vec<RemoteLlmModelInfo>>>;
+    async fn llm_model_info(&self, record: &LlmProviderRecord, model_id: &str) -> Result<Option<RemoteLlmModelInfo>>;
+    async fn list_tts_models(&self, record: &LlmProviderRecord) -> Result<Option<Vec<RemoteTtsModelInfo>>>;
+    async fn list_transcribe_models(&self, record: &LlmProviderRecord) -> Result<Option<Vec<RemoteTranscribeModelInfo>>>;
+
+    // ── Factories (default: None) ─────────────────────────────────────────────
+    fn build_llm(&self, record: &LlmProviderRecord, model: &LlmModelRecord) -> Option<Result<BuiltLlmClient>>;
+    fn build_tts(&self, record: &LlmProviderRecord, model: &TtsModelRecord) -> Option<Result<Arc<dyn TextToSpeech>>>;
+    fn build_transcriber(&self, record: &LlmProviderRecord, model: &TranscribeModelRecord) -> Option<Result<Arc<dyn Transcribe>>>;
+    fn build_image_generator(&self, record: &LlmProviderRecord, model: &ImageGenerateModelRecord) -> Option<Result<Arc<dyn ImageGenerate>>>;
+    fn ui_meta(&self) -> ProviderUiMeta;  // served to frontend via GET /api/llm/providers/types
+}
+```
+
+`list_tts_models` and `list_transcribe_models` have a default implementation returning `Ok(None)` — providers that don't support listing do not need to implement them. Only `ElevenLabsProvider` currently overrides both, calling `GET https://api.elevenlabs.io/v1/models` and filtering by capability flag.
+
+`BuiltLlmClient` bundles the constructed `Arc<dyn ChatbotClient>` with a `prompt_cache: bool` flag that controls whether Anthropic KV-cache headers are injected by the session loop.
+
+**`ProviderRegistry`** (`src/core/provider/mod.rs`) holds built-in and plugin providers separately. Plugin providers shadow built-in ones with the same `type_id`. Plugins can call `registry.register_plugin()` / `registry.unregister_plugin()` at any time after startup.
+
+`LlmManager`, `TranscribeManager`, `TtsManager`, and `ImageGeneratorManager` all receive an `Arc<ProviderRegistry>` at construction and use it to build clients and look up `supported_types`.
+
+### Built-in Providers
+
+| `type_id`     | Client struct     | api_key required | Default base_url                   | Prompt cache                   |
+| ------------- | ----------------- | ---------------- | ---------------------------------- | ------------------------------ |
+| `lm_studio`   | `LmStudioClient`  | No               | `http://localhost:1234/v1`         | ❌                             |
+| `ollama`      | `OllamaClient`    | No               | `http://localhost:11434`           | ❌                             |
+| `open_ai`     | `OpenAiClient`    | **Yes**          | `https://api.openai.com/v1`        | ❌                             |
+| `openrouter`  | `OpenAiClient`    | **Yes**          | `https://openrouter.ai/api/v1`     | ✅ `anthropic/*` models only   |
+| `anthropic`   | `AnthropicClient` | **Yes**          | `https://api.anthropic.com`        | ❌ (planned)                   |
+| `deepseek`    | `OpenAiClient`    | **Yes**          | `https://api.deepseek.com/v1`      | ✅ automatic (see below)       |
+| `zai`         | `OpenAiClient`    | **Yes**          | `https://api.z.ai/api/paas/v4`     | ❌                             |
+| `elevenlabs`  | —                 | **Yes**          | `https://api.elevenlabs.io`        | ❌ (TTS + Transcribe only)     |
+
+`openrouter`, `deepseek`, and `zai` reuse `OpenAiClient` with different base URLs. `elevenlabs` does not support LLM chat — `build_llm()` returns `None`.
+
+`zai` (Z.AI / Zhipu AI GLM) has no `GET /models` endpoint, so `list_llm_models()` returns a **curated static catalog** of published GLM models (`glm-5.2`, `glm-5.1`, `glm-5`, `glm-5-turbo`, `glm-4.7`, `glm-4.6`, `glm-4.5`, `glm-4-32b-0414-128k`) with heuristic context lengths — the same UI "model picker" flow as `deepseek`/`openrouter`. Note the base URL ends in `/paas/v4` because `OpenAiClient` appends `/chat/completions`.
+
+---
+
+## Prompt Caching (KV Cache)
+
+When `LlmEntry.prompt_cache = true` (currently set only for `OpenRouter`), the agent enables Anthropic-compatible KV caching on every request:
+
+### What is sent
+
+1. **`anthropic-beta: prompt-caching-2024-07-31` HTTP header** — tells OpenRouter/Anthropic to activate the caching feature.
+
+2. **Static system message tagged for caching** — `build_openai_messages` emits the first system message (AGENT.md + memory files + `extra_system_static` + MCP list) as a content array with `cache_control: {"type": "ephemeral"}` on the single block. This is the KV cache prefix.
+
+3. **Last tool tagged** — the final entry in the `tools` array receives `cache_control: {"type": "ephemeral"}`, caching the entire tool list as part of the prefix.
+
+### Message order and cache stability
+
+The full message array is structured so the stable prefix is as long as possible (see *Context Building* in [session.md](session.md)):
+
+```text
+[static system — cached]  [scratchpad?]  [summary?]  [conversation]  [dynamic tail]  [tail reminder]
+```
+
+The dynamic tail (Honcho memories + date/time) is placed **after** the conversation, so it never shortens the cacheable prefix. Scratchpad is a separate message so a mid-turn write only invalidates that small block, not the large static prefix.
+
+### Cache TTL
+
+Anthropic's `ephemeral` cache has a sliding TTL of ~5 minutes (extended by each hit). A cache hit is reported in the response as `cache_read_input_tokens` in the usage block.
+
+### DeepSeek automatic KV cache
+
+DeepSeek's disk KV cache is **prefix-based and fully automatic** — no explicit markers or special headers are required. Because the static system message is always the first entry in the message array, it becomes the stable cache prefix on every turn.
+
+`prompt_cache = false` for the `DeepSeek` provider: no Anthropic-style `cache_control` blocks are injected (DeepSeek does not understand them). The cache operates transparently. DeepSeek reports cache hits in the response under `usage.prompt_cache_hit_tokens` / `usage.prompt_cache_miss_tokens` (visible in the raw request log).
+
+#### ⚠️ The dynamic tail and cache invalidation
+
+DeepSeek's KV cache compares requests **token by token from position 0**. If any token differs at position N, everything from N onward is recomputed — there is no partial matching inside the sequence.
+
+The dynamic tail (date/time + Honcho memories) is injected as a system message at the end of the message array, after the conversation history. Because it is placed *after* the conversation, it doesn't shorten the cacheable prefix for the system message and tools. However, the **exact timestamp** (`2026-05-28T10:56:34+02:00`) changes every second. This means the stored KV entry from the previous request ends with `[..conversation..][dyn_tail_T1]`, while the new request has `[..conversation..][new_user_msg][dyn_tail_T2]`. The break point occurs right after the last assistant message: everything beyond it must be recomputed.
+
+In practice this means: **without timestamp rounding, only the static system message and tools are effectively cached.** The conversation history accumulates in the cache prefix, but the always-changing timestamp prevents the prefix from extending into the tail message of the stored entry.
+
+**Observed impact (production data):**
+
+| Configuration | `prompt_cache_hit_tokens` | `prompt_cache_miss_tokens` |
+| --- | --- | --- |
+| Exact timestamp (default before fix) | ~6,144 | ~21,583 |
+| `round_minutes: 15` | ~38,272 | ~830 |
+
+With rounding, the timestamp string stays byte-identical for up to 15 minutes, letting the full conversation + tools accumulate in the cache prefix. The remaining ~830 miss tokens represent only the current user message (unavoidably new on every request).
+
+### `llm.datetime` — timestamp injection config
+
+Controlled by `config.yml → llm.datetime`:
+
+```yaml
+llm:
+  datetime:
+    enabled: true        # false = omit date/time from context entirely
+    round_minutes: 15    # round down to nearest N-minute boundary
+                         # e.g. 10:56 → 10:50 with round_minutes: 10
+                         # omit for exact timestamp (hurts KV cache on prefix-based providers)
+```
+
+`round_minutes` is the primary tuning knob for cache efficiency on DeepSeek and any other prefix-based KV cache provider. The trade-off is precision: the LLM sees a timestamp that may be up to `round_minutes` minutes in the past. For most conversational uses this is imperceptible; for time-critical tasks (cron triggers, calendar scheduling) prefer a smaller value or `null`.
+
+The default in `default.config.yaml` is `round_minutes: 15` — a safe value that gives near-100% cache hit rates in typical conversations while keeping the timestamp accurate to within a quarter-hour.
+
+### Future: Anthropic direct
+
+`AnthropicClient` does not yet support `prompt_cache`. The implementation is different: the `system` parameter must be sent as a JSON array of content blocks rather than a plain string. Tracked as a future improvement.
+
+---
+
+---
+
+## Reasoning Content / DeepSeek Thinking Mode
+
+When DeepSeek is configured with `"thinking": {"type": "enabled"}` in `extra_params`, each response includes a `reasoning_content` field alongside the normal `content`. This is the model's chain-of-thought.
+
+**DeepSeek requires that `reasoning_content` be echoed back in the assistant message on subsequent turns.** Omitting it causes a `400 invalid_request_error`.
+
+### How it works
+
+1. `OpenAiClient.chat_with_tools_raw` reads `message.reasoning_content` from the response and propagates it through `LlmTurn`.
+2. `llm_loop` saves it to `chat_history.reasoning_content` alongside the assistant's text content.
+3. `build_openai_messages` includes `reasoning_content` in the reconstructed assistant message whenever the field is non-null.
+
+All other providers always return `reasoning_content: None`; the field is simply absent from their assistant messages in the history.
+
+---
+
+## LlmStrength Enum
+
+Ordered (weakest → strongest): `VeryLow` < `Low` < `Average` < `High` < `VeryHigh`
+
+Used by AUTO selection and `call_agent` to match agents to capable models.
+
+---
+
+## AUTO Selection Algorithm
+
+When `client = "auto"` or no client is specified, `LlmManager::select()` runs four passes in order, returning the first match:
+
+1. Not-Down + strength ≥ required + scope matches
+2. Not-Down + strength ≥ required (scope relaxed)
+3. Any Not-Down model
+4. **Emergency fallback**: strongest model even if Down (logs a `WARN`)
+
+Models are ordered by `priority ASC` in the DB; lower number = tried first.
+
+---
+
+## Health Tracking
+
+| Threshold                    | Status              |
+| ---------------------------- | ------------------- |
+| `consecutive_failures >= 3`  | `Degraded`          |
+| `consecutive_failures >= 5`  | `Down`              |
+| Next success                 | Reset to `Healthy`  |
+
+`mark_failure()` is called by `run_agent_turn` on LLM call errors. `mark_success()` is called on every successful response. Health state is preserved across `reload()` calls (e.g. after adding a new model).
+
+---
+
+## Automatic LLM Failover
+
+When the primary model returns a retriable error (5xx, network error, 429), `run_agent_turn` automatically tries the next available model — up to **3 attempts per round**.
+
+**Retry logic**:
+
+- A fresh `tried_this_round` list is built at the start of every round.
+- On error, `is_retriable_llm_error()` decides whether to try again. Client errors (400/401/403/404/422) are **not** retried — the request itself is invalid.
+- `select_excluding(&tried)` picks the next model, applying the same scope/strength rules as AUTO selection but skipping already-tried ones.
+- If a different model uses different `prompt_cache` settings, messages are rebuilt before the retry.
+- `cur_name`/`cur_llm` persist for the rest of the turn once switched, so subsequent rounds use the new model without re-trying the failed one.
+
+**Events emitted**:
+
+| Event | When | Who reacts |
+| --- | --- | --- |
+| `model_fallback` | Each successful switch | Frontend shows an inline info note |
+| `llm_failed` | All attempts exhausted | Frontend shows error + `_waiting = false`; Telegram sends a message |
+
+Telegram ignores `model_fallback` (silent retry) but sends an error message for `llm_failed`, matching the same behaviour as `Error`.
+
+---
+
+## Valid Scope Values
+
+`basic`, `writing`, `coding`, `reasoning`, `math`, `search`
+
+Defined by convention; any string is accepted by the DB. Agents declare `scope` in `meta.json`; models declare matching scopes in the DB.
+
+---
+
+## Extra Params
+
+Each model can store an optional `extra_params` JSON object. Its top-level keys are **merged into the request body** before every API call, overriding any default key with the same name.
+
+`OpenAiClient` (covers OpenAI-compatible providers) merges extra params via `apply_extra`. `AnthropicClient` merges them too via `apply_extra` (constructed with `with_extra_body`), which additionally enforces the extended-thinking constraints (see *Reasoning* below).
+
+**Example — DeepSeek thinking mode (native DeepSeek provider):**
+
+```json
+{ "thinking": {"type": "enabled"}, "reasoning_effort": "high" }
+```
+
+`reasoning_effort` accepts `"low"`, `"medium"`, or `"high"`. Only supported by DeepSeek reasoning models (e.g. `deepseek-reasoner`); sending these params to non-reasoning models returns a 400.
+
+**Example — DeepSeek reasoning effort on OpenRouter:**
+
+```json
+{ "reasoning": { "effort": "high" } }
+```
+
+Set via the model edit modal in the LLM Models UI, or via `PUT /api/llm/models/{id}` with `extra_params` in the JSON body.
+
+> For the model's **reasoning / thinking** knob, prefer the dedicated `reasoning` field (below) over hand-writing provider-specific keys in `extra_params`.
+
+---
+
+## Reasoning
+
+Reasoning ("thinking") is a first-class, provider-agnostic per-model setting rather than a hand-written `extra_params` blob. Two `ApiProvider` methods own it:
+
+```rust
+fn reasoning_mode(&self, model_id: &str, capabilities: &[String]) -> Option<ReasoningMode>;
+fn reasoning_request(&self, value: &serde_json::Value) -> Option<serde_json::Value>;
+```
+
+- **`reasoning_mode`** describes the *control* (drives the UI). `ReasoningMode` is either:
+  - `ValueSet { values, default }` — discrete choices (e.g. effort levels, or an on/off toggle), rendered as a dropdown.
+  - `Range { min, max, step, default, unit }` — a numeric budget, rendered as a number input.
+  - `None` — the model has no reasoning knob.
+- **`reasoning_request`** translates the *selected value* (stored per model in `llm_models.reasoning`: a JSON string for a `ValueSet`, a JSON number for a `Range`, or `NULL` = off) into the provider-specific request fragment, merged into the request body at `build_llm` time via `providers::extra_with_reasoning`.
+
+| Provider | `reasoning_mode` | Request fragment |
+|---|---|---|
+| `zai` | GLM-5.2+ → effort `ValueSet` (`disabled`/`minimal`…`max`); GLM-5.x/4.5/4.6/4.7 → on/off | `{"thinking":{"type":…}}` (+ `"reasoning_effort"` for GLM-5.2) |
+| `deepseek` | reasoner / v4 → effort `ValueSet` (`disabled`/`low`…`max`) | `{"thinking":{"type":…}}` (+ `"reasoning_effort"`) |
+| `openrouter` | per-model from the catalog `reasoning` object (`supported_efforts` / `supports_max_tokens`), else a fallback effort set | `{"reasoning":{"effort":…}}` or `{"reasoning":{"max_tokens":N}}` |
+| `open_ai` | o-series / gpt-5 → effort `ValueSet` | `{"reasoning_effort":…}` |
+| `anthropic` | thinking-capable models → `Range` (budget_tokens) | `{"thinking":{"type":"enabled","budget_tokens":N}}` |
+
+The descriptor reaches the frontend two ways: attached to each catalog model (`RemoteLlmModelInfo.reasoning`) for the add-from-catalog form, on `LlmModelInfo.reasoning_mode` for the edit form, and via `GET /api/llm/providers/{id}/reasoning-mode?model_id=…` for the manual add form. The UI renders the matching control (dropdown / number input) with an "— off —" option (null value → no reasoning param sent).
+
+**Anthropic** merges the fragment through `AnthropicClient::with_extra_body`; its `apply_extra` also enforces the extended-thinking constraints — `temperature` is dropped and `max_tokens` is bumped above `budget_tokens` when thinking is enabled.
+
+---
+
+## Model Metadata Fields
+
+Each model record now stores additional metadata beyond the core LLM configuration:
+
+| Field | Type | Source | Runtime use |
+|---|---|---|---|
+| `context_length` | `Option<i64>` | Provider catalog sync or manual input | Compaction threshold calculation, `max_tokens` limiting |
+| `max_output_tokens` | `Option<i64>` | Provider catalog sync or manual input | Future: set `max_tokens` on LLM calls (currently `None`) |
+| `knowledge_cutoff` | `Option<String>` | Provider catalog sync or manual input | Future: inject into system prompt |
+| `capabilities` | `Vec<String>` | Provider catalog sync or manual input | Filtering by model feature (vision, function_calling, etc.) |
+
+All fields are optional (`NULL` in the DB). When the provider catalog reports them,
+they are automatically synced to existing DB records by `list_provider_models()`.
+Manual values set via the API or UI take precedence when the provider does not
+report a particular field (the sync uses `COALESCE` — only non-NULL catalog values
+overwrite).
+
+---
+
+## LLM CRUD
+
+All mutations go through `LlmManager` (not direct DB writes) because each operation calls `reload()` to rebuild the in-memory state:
+
+- `add_provider()` / `update_provider()` / `delete_provider()`
+- `add_model()` / `update_model()` / `delete_model()`
+
+Setting `is_default = true` on a model automatically clears the flag on all others.
+
+**Soft delete:** `delete_provider()` and `delete_model()` never issue `DELETE` statements. They set `removed_at = datetime('now')` on the row. Deleting a provider also cascades to all its models and clears the provider's `api_key`. Removed rows are excluded from `load_all_providers()` / `load_all_models()` and therefore from the in-memory state and AUTO selector. The `id` values remain valid as FK references in `chat_history.model_db_id`.
+
+**Model identity is `name`.** `llm_models` has a single unique constraint, `name TEXT UNIQUE` — the alias, which is also the **resolution key** (`LlmManager` holds models in a `HashMap` keyed by name; `resolve(client_name)` / an agent's `client` look up by it). There is deliberately **no** `UNIQUE(provider_id, model_id)`: the same underlying model may be registered several times under one provider with different aliases and reasoning settings (e.g. `glm-4.6` vs `glm-4.6-thinking`). *(Migration v20 dropped the old `(provider_id, model_id)` constraint by rebuilding the table, preserving `id` values — they are FK targets in `llm_requests.model_db_id` / `chat_history.model_db_id`.)*
+
+**Re-adding a removed model (revive):** rows are never hard-deleted, only soft-deleted (`removed_at`), and the `name` uniqueness ignores `removed_at`. So `insert_model()` upserts on `name`: `ON CONFLICT(name) DO UPDATE … removed_at = NULL`. Re-adding a model with the same alias **revives that row** (clearing `removed_at`, overwriting every field including `provider_id`/`model_id`/`reasoning`) instead of failing with a UNIQUE violation.
+
+---
+
+## ApiProvider — Service Types
+
+Each provider declares which service kinds it supports via `ApiProvider::supported_types() -> &'static [ServiceType]`. Hardcoded per implementation — not stored in the DB.
+
+`ServiceType` replaces the old `ModelType` enum (previously in `src/core/llm/providers/mod.rs`); it now lives in `src/core/provider/mod.rs` and is re-exported as `providers::ServiceType` for backwards compatibility.
+
+| Provider (`type_id`) | `supported_types()`                              |
+| -------------------- | ------------------------------------------------ |
+| `openrouter`         | `[Llm, Transcribe, ImageGenerate, Tts]`          |
+| `open_ai`            | `[Llm, Transcribe, Tts]`                         |
+| `anthropic`          | `[Llm]`                                          |
+| `ollama`             | `[Llm]`                                          |
+| `lm_studio`          | `[Llm]`                                          |
+| `deepseek`           | `[Llm]`                                          |
+| `elevenlabs`         | `[Tts, Transcribe]`                              |
+
+`supported_types` is included in the `GET /api/llm/providers` response so the frontend can filter provider dropdowns when adding TTS, transcription, LLM, or image generation models.
+
+`GET /api/llm/providers/types` returns **all** registered provider types (no service-type filter). The frontend filters each picker independently using the `supported_types` array — e.g. the LLM model picker shows only providers where `supported_types.includes('llm')`.
+
+---
+
+## ApiProvider — Remote Model Catalog
+
+`list_llm_models()` and `llm_model_info()` are methods on `ApiProvider`. They both receive the full `LlmProviderRecord` so they can read the `api_key` and `base_url` without constructing a separate credentials struct.
+
+`RemoteLlmModelInfo` fields: `id`, `name`, `context_length`, `max_completion_tokens`,
+`knowledge_cutoff`, `capabilities`, `vision: Option<bool>`, `price_input_per_million`, `price_output_per_million` (USD/M tokens).
+
+`vision` is `Some(true/false)` when the provider reports it explicitly (e.g. OpenRouter `supported_parameters`), `None` when unknown.
+
+| Provider (`type_id`) | `list_llm_models()` | `llm_model_info()` |
+| --- | --- | --- |
+| `openrouter` | `GET /api/v1/models` — sets `vision` from `supported_parameters` | — |
+| `ollama` | `GET /api/tags` | `POST /api/show` |
+| `anthropic` | `None` | `GET /v1/models/{id}` |
+| `deepseek` | `GET /models` | `None` |
+| `lm_studio` | `GET /v1/models` | `None` |
+| `open_ai` | `None` | `None` |
+| `elevenlabs` | `None` (LLM not supported) | `None` |
+
+Provider instances are obtained via `ProviderRegistry::get(type_id)` — no on-demand factory needed.
+
+### Model Catalog Cache
+
+`LlmManager` caches `list_models()` results in memory, keyed by `provider_id`, with a **24-hour TTL**. The cache is discarded on process restart.
+
+### Per-Model Metadata Cache
+
+When `LlmManager::resolve()` is called, it lazily fetches `model_info()` for the resolved model
+if the per-model cache is missing or older than **1 hour**. The `context_length` from the fresh
+metadata is then propagated to the live `LlmEntry` in the model slot so subsequent turns use
+the updated value.
+
+Cache flow:
+
+- Fast path: read lock on `model_meta_cache` → hit + fresh → return immediately.
+- Miss / stale: fetch `model_info()` from the provider → update cache → update `LlmEntry.context_length`.
+- Network failure: the old cached value (or DB value) is preserved — the error is silently ignored.
+
+This ensures the compactor and any future `max_tokens` logic always have a reasonably current
+`context_length` without blocking the first turn of the session.
+
+```text
+LlmManager::list_provider_models(provider_id)
+  → cache hit  (< 24h old) → return cached Vec<RemoteLlmModelInfo>
+  → cache miss / expired   → fetch via ApiProvider, store, return
+```
+
+API endpoint: `GET /api/llm/providers/{id}/models`
+
+Used by the frontend "Add Model" wizard to populate the searchable model picker for OpenRouter, Ollama, and LM Studio providers.
+
+---
+
+## When to Update This File
+
+- A new built-in provider is registered in `main.rs` (add row to the tables above)
+- A new method is added to the `ApiProvider` trait
+- The AUTO selection algorithm changes
+- Health thresholds (`FAILURE_DEGRADED`, `FAILURE_DOWN`) change
+- `ProviderRegistry` plugin API changes (register/unregister)