dguiducci/Skald-Circle
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
Skald-Circle/docs/llm-clients.md
T
dguiducci 38494a85a9 First Version
2026-07-10 15:02:09 +01:00

27 KiB
Raw Permalink Blame History

LLM Clients

Workspace Location

The ChatbotClient trait and all provider implementations live in the standalone crate crates/llm-client (no dependencies on the main crate). src/core/chatbot/mod.rs is a thin re-export layer. LoggingChatbotClient (src/core/chatbot/logging.rs) remains in the main crate because it depends on sqlx.

ChatbotClient Trait

#[async_trait]
pub trait ChatbotClient: Send + Sync {
    async fn chat(&self, messages: &[Message], options: &ChatOptions) -> Result<ChatResponse>;
    async fn chat_with_tools(&self, messages: &[Value], tools: &[Value], options: &ChatOptions) -> Result<LlmTurn>;
    async fn chat_with_tools_raw(&self, messages: &[Value], tools: &[Value], options: &ChatOptions) -> Result<(LlmTurn, Option<LlmRawMeta>)>;
}

Only AnthropicClient and OpenAiClient implement native tool support (chat_with_tools). Other clients have a default fallback that strips tool definitions and calls chat() instead.

chat_with_tools_raw is used by the logging wrapper: it returns the same LlmTurn plus raw HTTP request/response metadata (LlmRawMeta). AnthropicClient, OpenAiClient, and LmStudioClient override it; all others fall back to calling chat_with_tools with no metadata.

ChatOptions carries two optional fields — session_id and stack_id — set by the LLM loop for correlation. Providers ignore them; only LoggingChatbotClient reads them.

Transparent Request Logging

LoggingChatbotClient (src/core/chatbot/logging.rs) is a ChatbotClient wrapper that intercepts every chat_with_tools call:

  1. Calls inner.chat_with_tools_raw(...) to capture the HTTP wire data.
  2. Spawns a fire-and-forget tokio::spawn to insert a row into llm_requests.
  3. Returns the LlmTurn to the caller unchanged.

The LLM loop is fully unaware — it holds an Arc<dyn ChatbotClient> and calls chat_with_tools as usual. The wrapper is applied in LlmManager::build_entry when request_log_enabled = true (set from config.yml → llm.request_log.enabled).

What is logged per row: full request body (provider-specific format), request headers (api-key redacted), full response body, response headers, token counts, round-trip duration, session/stack ID.

A background task (boot + every hour) deletes rows older than retention_days (default 14).

LlmTurn return variants:

  • Message(ChatResponse) — final text answer
  • ToolCalls { content, calls, input_tokens, output_tokens, reasoning_content, cost } — one or more tool calls requested

Both variants carry an optional reasoning_content: Option<String>. Populated only by providers that return chain-of-thought (currently DeepSeek thinking mode). Saved to chat_history.reasoning_content and echoed back on subsequent turns — see Reasoning Content / DeepSeek Thinking Mode below.

Both variants also carry an optional cost: Option<f64> — the request price in USD. Populated via the ChatbotClient::extract_cost(&self, response: &Value) trait method, whose default reads usage.cost from the raw JSON response (OpenRouter and other OpenAI-compatible gateways report it there). Providers that don't bill per-request leave it None. llm_loop persists it to chat_history.cost. Providers with a different response shape can override extract_cost.

Provider Registry

Providers are no longer identified by a hard-coded enum. Instead, each provider is a struct implementing the ApiProvider trait (src/core/provider/mod.rs), registered at startup in main.rs via ProviderRegistry::register_builtin(). The DB column llm_providers.type stores the provider's type_id string.

// src/provider/mod.rs
#[async_trait]
pub trait ApiProvider: Send + Sync {
    fn type_id(&self) -> &'static str;         // e.g. "open_ai", "anthropic"
    fn display_name(&self) -> &'static str;
    fn supported_types(&self) -> &'static [ServiceType];

    // ── Remote model catalogs (default: Ok(None)) ─────────────────────────────
    async fn list_llm_models(&self, record: &LlmProviderRecord) -> Result<Option<Vec<RemoteLlmModelInfo>>>;
    async fn llm_model_info(&self, record: &LlmProviderRecord, model_id: &str) -> Result<Option<RemoteLlmModelInfo>>;
    async fn list_tts_models(&self, record: &LlmProviderRecord) -> Result<Option<Vec<RemoteTtsModelInfo>>>;
    async fn list_transcribe_models(&self, record: &LlmProviderRecord) -> Result<Option<Vec<RemoteTranscribeModelInfo>>>;

    // ── Factories (default: None) ─────────────────────────────────────────────
    fn build_llm(&self, record: &LlmProviderRecord, model: &LlmModelRecord) -> Option<Result<BuiltLlmClient>>;
    fn build_tts(&self, record: &LlmProviderRecord, model: &TtsModelRecord) -> Option<Result<Arc<dyn TextToSpeech>>>;
    fn build_transcriber(&self, record: &LlmProviderRecord, model: &TranscribeModelRecord) -> Option<Result<Arc<dyn Transcribe>>>;
    fn build_image_generator(&self, record: &LlmProviderRecord, model: &ImageGenerateModelRecord) -> Option<Result<Arc<dyn ImageGenerate>>>;
    fn ui_meta(&self) -> ProviderUiMeta;  // served to frontend via GET /api/llm/providers/types
}

list_tts_models and list_transcribe_models have a default implementation returning Ok(None) — providers that don't support listing do not need to implement them. Only ElevenLabsProvider currently overrides both, calling GET https://api.elevenlabs.io/v1/models and filtering by capability flag.

BuiltLlmClient bundles the constructed Arc<dyn ChatbotClient> with a prompt_cache: bool flag that controls whether Anthropic KV-cache headers are injected by the session loop.

ProviderRegistry (src/core/provider/mod.rs) holds built-in and plugin providers separately. Plugin providers shadow built-in ones with the same type_id. Plugins can call registry.register_plugin() / registry.unregister_plugin() at any time after startup.

LlmManager, TranscribeManager, TtsManager, and ImageGeneratorManager all receive an Arc<ProviderRegistry> at construction and use it to build clients and look up supported_types.

Built-in Providers

type_id Client struct api_key required Default base_url Prompt cache
lm_studio LmStudioClient No http://localhost:1234/v1 ❌
ollama OllamaClient No http://localhost:11434 ❌
open_ai OpenAiClient Yes https://api.openai.com/v1 ❌
openrouter OpenAiClient Yes https://openrouter.ai/api/v1 ✅ anthropic/* models only
anthropic AnthropicClient Yes https://api.anthropic.com ❌ (planned)
deepseek OpenAiClient Yes https://api.deepseek.com/v1 ✅ automatic (see below)
zai OpenAiClient Yes https://api.z.ai/api/paas/v4 ❌
elevenlabs — Yes https://api.elevenlabs.io ❌ (TTS + Transcribe only)

openrouter, deepseek, and zai reuse OpenAiClient with different base URLs. elevenlabs does not support LLM chat — build_llm() returns None.

zai (Z.AI / Zhipu AI GLM) has no GET /models endpoint, so list_llm_models() returns a curated static catalog of published GLM models (glm-5.2, glm-5.1, glm-5, glm-5-turbo, glm-4.7, glm-4.6, glm-4.5, glm-4-32b-0414-128k) with heuristic context lengths — the same UI "model picker" flow as deepseek/openrouter. Note the base URL ends in /paas/v4 because OpenAiClient appends /chat/completions.

Prompt Caching (KV Cache)

When LlmEntry.prompt_cache = true (currently set only for OpenRouter), the agent enables Anthropic-compatible KV caching on every request:

What is sent

  1. anthropic-beta: prompt-caching-2024-07-31 HTTP header — tells OpenRouter/Anthropic to activate the caching feature.

  2. Static system message tagged for caching — build_openai_messages emits the first system message (AGENT.md + memory files + extra_system_static + MCP list) as a content array with cache_control: {"type": "ephemeral"} on the single block. This is the KV cache prefix.

  3. Last tool tagged — the final entry in the tools array receives cache_control: {"type": "ephemeral"}, caching the entire tool list as part of the prefix.

Message order and cache stability

The full message array is structured so the stable prefix is as long as possible (see Context Building in session.md):

[static system — cached]  [scratchpad?]  [summary?]  [conversation]  [dynamic tail]  [tail reminder]

The dynamic tail (Honcho memories + date/time) is placed after the conversation, so it never shortens the cacheable prefix. Scratchpad is a separate message so a mid-turn write only invalidates that small block, not the large static prefix.

Cache TTL

Anthropic's ephemeral cache has a sliding TTL of ~5 minutes (extended by each hit). A cache hit is reported in the response as cache_read_input_tokens in the usage block.

DeepSeek automatic KV cache

DeepSeek's disk KV cache is prefix-based and fully automatic — no explicit markers or special headers are required. Because the static system message is always the first entry in the message array, it becomes the stable cache prefix on every turn.

prompt_cache = false for the DeepSeek provider: no Anthropic-style cache_control blocks are injected (DeepSeek does not understand them). The cache operates transparently. DeepSeek reports cache hits in the response under usage.prompt_cache_hit_tokens / usage.prompt_cache_miss_tokens (visible in the raw request log).

⚠️ The dynamic tail and cache invalidation

DeepSeek's KV cache compares requests token by token from position 0. If any token differs at position N, everything from N onward is recomputed — there is no partial matching inside the sequence.

The dynamic tail (date/time + Honcho memories) is injected as a system message at the end of the message array, after the conversation history. Because it is placed after the conversation, it doesn't shorten the cacheable prefix for the system message and tools. However, the exact timestamp (2026-05-28T10:56:34+02:00) changes every second. This means the stored KV entry from the previous request ends with [..conversation..][dyn_tail_T1], while the new request has [..conversation..][new_user_msg][dyn_tail_T2]. The break point occurs right after the last assistant message: everything beyond it must be recomputed.

In practice this means: without timestamp rounding, only the static system message and tools are effectively cached. The conversation history accumulates in the cache prefix, but the always-changing timestamp prevents the prefix from extending into the tail message of the stored entry.

Observed impact (production data):

Configuration prompt_cache_hit_tokens prompt_cache_miss_tokens
Exact timestamp (default before fix) ~6,144 ~21,583
round_minutes: 15 ~38,272 ~830

With rounding, the timestamp string stays byte-identical for up to 15 minutes, letting the full conversation + tools accumulate in the cache prefix. The remaining ~830 miss tokens represent only the current user message (unavoidably new on every request).

llm.datetime — timestamp injection config

Controlled by config.yml → llm.datetime:

llm:
  datetime:
    enabled: true        # false = omit date/time from context entirely
    round_minutes: 15    # round down to nearest N-minute boundary
                         # e.g. 10:56 → 10:50 with round_minutes: 10
                         # omit for exact timestamp (hurts KV cache on prefix-based providers)

round_minutes is the primary tuning knob for cache efficiency on DeepSeek and any other prefix-based KV cache provider. The trade-off is precision: the LLM sees a timestamp that may be up to round_minutes minutes in the past. For most conversational uses this is imperceptible; for time-critical tasks (cron triggers, calendar scheduling) prefer a smaller value or null.

The default in default.config.yaml is round_minutes: 15 — a safe value that gives near-100% cache hit rates in typical conversations while keeping the timestamp accurate to within a quarter-hour.

Future: Anthropic direct

AnthropicClient does not yet support prompt_cache. The implementation is different: the system parameter must be sent as a JSON array of content blocks rather than a plain string. Tracked as a future improvement.

Reasoning Content / DeepSeek Thinking Mode

When DeepSeek is configured with "thinking": {"type": "enabled"} in extra_params, each response includes a reasoning_content field alongside the normal content. This is the model's chain-of-thought.

DeepSeek requires that reasoning_content be echoed back in the assistant message on subsequent turns. Omitting it causes a 400 invalid_request_error.

How it works

  1. OpenAiClient.chat_with_tools_raw reads message.reasoning_content from the response and propagates it through LlmTurn.
  2. llm_loop saves it to chat_history.reasoning_content alongside the assistant's text content.
  3. build_openai_messages includes reasoning_content in the reconstructed assistant message whenever the field is non-null.

All other providers always return reasoning_content: None; the field is simply absent from their assistant messages in the history.

LlmStrength Enum

Ordered (weakest → strongest): VeryLow < Low < Average < High < VeryHigh

Used by AUTO selection and call_agent to match agents to capable models.

AUTO Selection Algorithm

When client = "auto" or no client is specified, LlmManager::select() runs four passes in order, returning the first match:

  1. Not-Down + strength ≥ required + scope matches
  2. Not-Down + strength ≥ required (scope relaxed)
  3. Any Not-Down model
  4. Emergency fallback: strongest model even if Down (logs a WARN)

Models are ordered by priority ASC in the DB; lower number = tried first.

Health Tracking

Threshold Status
consecutive_failures >= 3 Degraded
consecutive_failures >= 5 Down
Next success Reset to Healthy

mark_failure() is called by run_agent_turn on LLM call errors. mark_success() is called on every successful response. Health state is preserved across reload() calls (e.g. after adding a new model).

Automatic LLM Failover

When the primary model returns a retriable error (5xx, network error, 429), run_agent_turn automatically tries the next available model — up to 3 attempts per round.

Retry logic:

  • A fresh tried_this_round list is built at the start of every round.
  • On error, is_retriable_llm_error() decides whether to try again. Client errors (400/401/403/404/422) are not retried — the request itself is invalid.
  • select_excluding(&tried) picks the next model, applying the same scope/strength rules as AUTO selection but skipping already-tried ones.
  • If a different model uses different prompt_cache settings, messages are rebuilt before the retry.
  • cur_name/cur_llm persist for the rest of the turn once switched, so subsequent rounds use the new model without re-trying the failed one.

Events emitted:

Event When Who reacts
model_fallback Each successful switch Frontend shows an inline info note
llm_failed All attempts exhausted Frontend shows error + _waiting = false; Telegram sends a message

Telegram ignores model_fallback (silent retry) but sends an error message for llm_failed, matching the same behaviour as Error.

Valid Scope Values

basic, writing, coding, reasoning, math, search

Defined by convention; any string is accepted by the DB. Agents declare scope in meta.json; models declare matching scopes in the DB.

Extra Params

Each model can store an optional extra_params JSON object. Its top-level keys are merged into the request body before every API call, overriding any default key with the same name.

OpenAiClient (covers OpenAI-compatible providers) merges extra params via apply_extra. AnthropicClient merges them too via apply_extra (constructed with with_extra_body), which additionally enforces the extended-thinking constraints (see Reasoning below).

Example — DeepSeek thinking mode (native DeepSeek provider):

{ "thinking": {"type": "enabled"}, "reasoning_effort": "high" }

reasoning_effort accepts "low", "medium", or "high". Only supported by DeepSeek reasoning models (e.g. deepseek-reasoner); sending these params to non-reasoning models returns a 400.

Example — DeepSeek reasoning effort on OpenRouter:

{ "reasoning": { "effort": "high" } }

Set via the model edit modal in the LLM Models UI, or via PUT /api/llm/models/{id} with extra_params in the JSON body.

For the model's reasoning / thinking knob, prefer the dedicated reasoning field (below) over hand-writing provider-specific keys in extra_params.

Reasoning

Reasoning ("thinking") is a first-class, provider-agnostic per-model setting rather than a hand-written extra_params blob. Two ApiProvider methods own it:

fn reasoning_mode(&self, model_id: &str, capabilities: &[String]) -> Option<ReasoningMode>;
fn reasoning_request(&self, value: &serde_json::Value) -> Option<serde_json::Value>;
  • reasoning_mode describes the control (drives the UI). ReasoningMode is either:
    • ValueSet { values, default } — discrete choices (e.g. effort levels, or an on/off toggle), rendered as a dropdown.
    • Range { min, max, step, default, unit } — a numeric budget, rendered as a number input.
    • None — the model has no reasoning knob.
  • reasoning_request translates the selected value (stored per model in llm_models.reasoning: a JSON string for a ValueSet, a JSON number for a Range, or NULL = off) into the provider-specific request fragment, merged into the request body at build_llm time via providers::extra_with_reasoning.
Provider reasoning_mode Request fragment
zai GLM-5.2+ → effort ValueSet (disabled/minimal…max); GLM-5.x/4.5/4.6/4.7 → on/off {"thinking":{"type":…}} (+ "reasoning_effort" for GLM-5.2)
deepseek reasoner / v4 → effort ValueSet (disabled/low…max) {"thinking":{"type":…}} (+ "reasoning_effort")
openrouter per-model from the catalog reasoning object (supported_efforts / supports_max_tokens), else a fallback effort set {"reasoning":{"effort":…}} or {"reasoning":{"max_tokens":N}}
open_ai o-series / gpt-5 → effort ValueSet {"reasoning_effort":…}
anthropic thinking-capable models → Range (budget_tokens) {"thinking":{"type":"enabled","budget_tokens":N}}

The descriptor reaches the frontend two ways: attached to each catalog model (RemoteLlmModelInfo.reasoning) for the add-from-catalog form, on LlmModelInfo.reasoning_mode for the edit form, and via GET /api/llm/providers/{id}/reasoning-mode?model_id=… for the manual add form. The UI renders the matching control (dropdown / number input) with an "— off —" option (null value → no reasoning param sent).

Anthropic merges the fragment through AnthropicClient::with_extra_body; its apply_extra also enforces the extended-thinking constraints — temperature is dropped and max_tokens is bumped above budget_tokens when thinking is enabled.

Model Metadata Fields

Each model record now stores additional metadata beyond the core LLM configuration:

Field Type Source Runtime use
context_length Option<i64> Provider catalog sync or manual input Compaction threshold calculation, max_tokens limiting
max_output_tokens Option<i64> Provider catalog sync or manual input Future: set max_tokens on LLM calls (currently None)
knowledge_cutoff Option<String> Provider catalog sync or manual input Future: inject into system prompt
capabilities Vec<String> Provider catalog sync or manual input Filtering by model feature (vision, function_calling, etc.)

All fields are optional (NULL in the DB). When the provider catalog reports them, they are automatically synced to existing DB records by list_provider_models(). Manual values set via the API or UI take precedence when the provider does not report a particular field (the sync uses COALESCE — only non-NULL catalog values overwrite).

LLM CRUD

All mutations go through LlmManager (not direct DB writes) because each operation calls reload() to rebuild the in-memory state:

  • add_provider() / update_provider() / delete_provider()
  • add_model() / update_model() / delete_model()

Setting is_default = true on a model automatically clears the flag on all others.

Soft delete: delete_provider() and delete_model() never issue DELETE statements. They set removed_at = datetime('now') on the row. Deleting a provider also cascades to all its models and clears the provider's api_key. Removed rows are excluded from load_all_providers() / load_all_models() and therefore from the in-memory state and AUTO selector. The id values remain valid as FK references in chat_history.model_db_id.

Model identity is name. llm_models has a single unique constraint, name TEXT UNIQUE — the alias, which is also the resolution key (LlmManager holds models in a HashMap keyed by name; resolve(client_name) / an agent's client look up by it). There is deliberately no UNIQUE(provider_id, model_id): the same underlying model may be registered several times under one provider with different aliases and reasoning settings (e.g. glm-4.6 vs glm-4.6-thinking). (Migration v20 dropped the old (provider_id, model_id) constraint by rebuilding the table, preserving id values — they are FK targets in llm_requests.model_db_id / chat_history.model_db_id.)

Re-adding a removed model (revive): rows are never hard-deleted, only soft-deleted (removed_at), and the name uniqueness ignores removed_at. So insert_model() upserts on name: ON CONFLICT(name) DO UPDATE … removed_at = NULL. Re-adding a model with the same alias revives that row (clearing removed_at, overwriting every field including provider_id/model_id/reasoning) instead of failing with a UNIQUE violation.

ApiProvider — Service Types

Each provider declares which service kinds it supports via ApiProvider::supported_types() -> &'static [ServiceType]. Hardcoded per implementation — not stored in the DB.

ServiceType replaces the old ModelType enum (previously in src/core/llm/providers/mod.rs); it now lives in src/core/provider/mod.rs and is re-exported as providers::ServiceType for backwards compatibility.

Provider (type_id) supported_types()
openrouter [Llm, Transcribe, ImageGenerate, Tts]
open_ai [Llm, Transcribe, Tts]
anthropic [Llm]
ollama [Llm]
lm_studio [Llm]
deepseek [Llm]
elevenlabs [Tts, Transcribe]

supported_types is included in the GET /api/llm/providers response so the frontend can filter provider dropdowns when adding TTS, transcription, LLM, or image generation models.

GET /api/llm/providers/types returns all registered provider types (no service-type filter). The frontend filters each picker independently using the supported_types array — e.g. the LLM model picker shows only providers where supported_types.includes('llm').

ApiProvider — Remote Model Catalog

list_llm_models() and llm_model_info() are methods on ApiProvider. They both receive the full LlmProviderRecord so they can read the api_key and base_url without constructing a separate credentials struct.

RemoteLlmModelInfo fields: id, name, context_length, max_completion_tokens, knowledge_cutoff, capabilities, vision: Option<bool>, price_input_per_million, price_output_per_million (USD/M tokens).

vision is Some(true/false) when the provider reports it explicitly (e.g. OpenRouter supported_parameters), None when unknown.

Provider (type_id) list_llm_models() llm_model_info()
openrouter GET /api/v1/models — sets vision from supported_parameters —
ollama GET /api/tags POST /api/show
anthropic None GET /v1/models/{id}
deepseek GET /models None
lm_studio GET /v1/models None
open_ai None None
elevenlabs None (LLM not supported) None

Provider instances are obtained via ProviderRegistry::get(type_id) — no on-demand factory needed.

Model Catalog Cache

LlmManager caches list_models() results in memory, keyed by provider_id, with a 24-hour TTL. The cache is discarded on process restart.

Per-Model Metadata Cache

When LlmManager::resolve() is called, it lazily fetches model_info() for the resolved model if the per-model cache is missing or older than 1 hour. The context_length from the fresh metadata is then propagated to the live LlmEntry in the model slot so subsequent turns use the updated value.

Cache flow:

  • Fast path: read lock on model_meta_cache → hit + fresh → return immediately.
  • Miss / stale: fetch model_info() from the provider → update cache → update LlmEntry.context_length.
  • Network failure: the old cached value (or DB value) is preserved — the error is silently ignored.

This ensures the compactor and any future max_tokens logic always have a reasonably current context_length without blocking the first turn of the session.

LlmManager::list_provider_models(provider_id)
  → cache hit  (< 24h old) → return cached Vec<RemoteLlmModelInfo>
  → cache miss / expired   → fetch via ApiProvider, store, return

API endpoint: GET /api/llm/providers/{id}/models

Used by the frontend "Add Model" wizard to populate the searchable model picker for OpenRouter, Ollama, and LM Studio providers.

When to Update This File

  • A new built-in provider is registered in main.rs (add row to the tables above)
  • A new method is added to the ApiProvider trait
  • The AUTO selection algorithm changes
  • Health thresholds (FAILURE_DEGRADED, FAILURE_DOWN) change
  • ProviderRegistry plugin API changes (register/unregister)
Reference in New Issue View Git Blame Copy Permalink