Skald-Circle/dev-docs/llm-stack.md

Skald dev-docs — architectural reference for coding agents. Index: README.md · Entry point: ../CLAUDE.md

Read this when: you touch LLM clients, providers.yaml, retriability, request logging, token streaming or multimodal attachments.

---

The LLM stack

The client layer (crates/skald-core/src/llm/)

LLM client abstraction (OpenAI-compat, Anthropic, Ollama…). OpenAI-compatible provider types are runtime data, not code: providers/declared.rs loads providers.yaml at boot (see Config in ../CLAUDE.md); only non-OpenAI-compatible or bespoke providers (anthropic, ollama, openai, openrouter) stay native. Retriability (Model::is_retriable, agent-loop) keys on the real HTTP status carried by ModelError { status }, not a substring of the message — a model id/token count containing "404"/"401" cannot mis-classify; 401/403/404/422 don't retry, 400/429/5xx/network do. Request logging is the logging.rs::LoggingModel decorator, attached by the caller's ModelSelector (loop_adapters/selector.rs::SkaldSelector::with_log) — never by LlmManager, which builds one shared client per model and cannot know whose traffic it serves. The decorator's RequestLogTarget carries the owner: metadata → llm_requests in the registry (user_id, the column the UI filters on), payload bodies/headers → llm_request_payloads in that user's own encrypted DB, keyed by request_id; session + frame come from the request's own conversation/frame, so kernel rounds, sub-agent frames and compaction summaries are all attributed with no extra plumbing (ModelRequest::log is unused here)

providers.yaml — two traps in the model metadata

enrich rules stop at the first glob that matches, and the default mode: fill skips a field that already has a value. Both bite when adding a model to an existing family. Ordering: glm-5.3-flash matches glm-5* too, so a rule for it placed after the family rule never runs — the specific glob goes first, always. And fill means "the endpoint listing wins", which for a static: list is not the same as "nothing is set": models.defaults (today only vision) stamps every entry before enrich sees it, so a provider carrying defaults: { vision: false } — Z.AI does — needs mode: override to turn vision on for one model. A fill rule there parses, loads, logs nothing, and leaves the flag off; the only symptom is that images silently keep taking the textual <system-extra> path (see Multimodal attachments) on a model that can read them. vision: true also pushes the vision capability, but only when the rule actually applied.

A reasoning.modes values list is the whole contract with the provider — it is not a superset to trim in the UI. Whatever it lists is what can be sent, so a model that cannot stop thinking (thinking.type accepting only "enabled": GLM-5.3 and GLM-5.3-Flash) simply omits disabled from its rule, rather than inheriting the family's [disabled, enabled] toggle and sending a value the API rejects. With request: { kind: thinking }, any value other than disabled/enabled is emitted as {"thinking":{"type":"enabled"},"reasoning_effort":v}, which is exactly the shape those models want.

The provider API surface — the key is a boolean, never a value

No provider endpoint ever returns a stored api_key, and the trap is that omitting it silently reads as "no key". LlmProviderInfo (list) and ProviderDetail (src/frontend/api/llm.rs, the detail DTO — deliberately not LlmProviderRecord, which does carry the secret) both expose has_api_key: bool instead. That is the whole contract: the UI needs to know whether a key is on file, never what it is, and the browser is where a leaked key would end up in a devtools tab or a screenshot.

It shipped broken in exactly the way this shape invites: list_providers_info never carried api_key (correctly), while the card tested Boolean(p.api_key) — always undefined — so every provider was badged "API key missing" even with a working key. A missing field is falsy, not an error; nothing logs, nothing fails to build. If you add a provider surface, read has_api_key, and if you add a field to either DTO, keep the secret out by construction rather than by remembering to strip it.

The consequence on the write path is load-bearing: since the edit form can no longer prefill the key, an empty api_key in the PUT payload means "keep the stored one" — update_provider re-reads the record and carries the old value over, because a blind UPDATE … SET api_key=NULL would wipe a working provider on any unrelated edit (a renamed description). The i18n placeholder (providers.modal.api_key_ph) already promised this behaviour before the backend implemented it. Side effect to know about: there is no longer a way to clear a key from the form — deleting the provider is the escape hatch.

Token streaming & reasoning display

The chat streams tokens live, as a parallel best-effort side-channel that never alters the turn's authoritative flow: the final Done (or Thinking) event still carries the complete content and the frontend treats it as truth.

• Client seam (core-api::chatbot): ChatbotClient::chat_with_tools_raw_streaming(..., delta_tx: mpsc::Sender<StreamDelta>) — default impl ignores the channel and calls the buffered chat_with_tools_raw, so providers without streaming (Ollama, LM Studio) are untouched. StreamDelta::{Text, Reasoning} splits visible answer from chain-of-thought. Senders use try_send (deltas drop when the channel is full) — streaming must never backpressure the HTTP read.
• SSE implementations (crates/llm-client): OpenAiClient (stream:true + stream_options.include_usage, reasoning_content/reasoning deltas, index-based tool_calls accumulation, usage from the final chunk) and AnthropicClient (stream:true; message_start/content_block_*/message_delta events; thinking_delta → reasoning, input_json_delta → tool input). Both reassemble the same LlmTurn + LlmRawMeta the buffered path returns (the payload log stores a synthesized buffered-shaped body). Failure policy: if the stream dies before any delta the client retries buffered on the same model (providers rejecting stream keep working); a mid-stream failure propagates to the normal model-fallback logic. Framing is shared (llm_client::SseDecoder). Anthropic's buffered path now also parses thinking blocks into reasoning_content (previously discarded).
• Loop wiring: call_llm_round creates the delta channel per attempt and a forwarder task maps deltas to ServerEvent::TokenDelta { kind: content|reasoning, delta } on the turn's event channel (drained before the round's outcome events, so ordering holds); cancellation drops the in-flight future as before. A mid-stream fallback is handled client-side: the frontend clears its pending bubble on model_fallback.
• Reasoning surfacing: reasoning_content rides Done/Thinking events (so buffered providers show it live too) and is projected as reasoning on assistant/thinking history items (build_items); persistence in chat_history.reasoning_content and the echo back into context predate this feature.
• Frontend (chat-session.js + copilot-render.js, shared by desktop copilot and mobile chat-page): token_delta accumulates into a pending assistant bubble (in-place mutation + ~15 Hz flush, blinking caret); done/thinking finalize it in place, error/llm_failed/model_fallback drop it, tool_start/agent_done finalize orphan bubbles (reasoning-only rounds, sub-agent final rounds that emit no Done). The reasoning block is a muted, collapsed-by-default native <details> (renderReasoning, .reasoning-block in copilot-messages.css, i18n key chat.reasoning) — open state survives re-renders, and it renders identically from live events and from history.

Multimodal attachments

Uploads go through one centralized seam — ChatHub::save_upload (behind ChatHubApi::save_upload, backed by skald_core::uploads::save_to_home) — so every surface persists identically and no two callers can drift on placement (the class of bug where the agent was handed a path it couldn't reach). The seam writes into the caller's container home under uploads/{session_id}/ (agent path uploads/{session}/{name}, the UPLOADS_SUBDIR const in core-api/user_fs.rs), collision-dedupes the name, and prefers the sniffed magic-byte MIME over the client claim. The web handler (POST /api/{source}/uploads) buffers each field with a 256 MiB cap then calls the seam; the Telegram plugin downloads bytes then calls the same seam via handle.chat_hub().save_upload("telegram", …). Because the file lands in the home (bind-mounted at /root), it is reachable by the fs-tools, execute_cmd, and the file viewer (GET /api/file, per-user via resolve_view_path) — there is no /data static route anymore (removed: it was require_auth-only, not ownership-scoped, and also exposed internal server state under data/). Attachment metadata travels as structured JSON in chat_history.metadata — never as persisted text.

At context-build time (the crate's projection), attachments of the current turn (the user/agent rows following the last completed assistant reply, including across in-flight tool rounds) are partitioned by agent_loop::projection::media, with loop_adapters/media_source.rs deciding which files may be handed over (§6 containment): when the resolved model's LlmEntry.capabilities include the modality (vision → image_url parts, video → video_url parts), the file is inlined as a base64 data-URL content part — but only if it resolves (through the caller's UserFs, via resolve_host_path) under the home's uploads/ dir, its sniffed MIME is in the allowlist, and it fits the budgets (4 files / 10 MiB image / 32 MiB video / 48 MiB total per turn). Everything else — older turns, other kinds, any failed check — keeps the textual <system-extra> path block (built by core_api::message_meta::attachments_block / system_extra; the tag name is the single SYSTEM_EXTRA_TAG constant), so a non-vision model produces a byte-identical payload to before. OpenAiClient forwards parts verbatim; AnthropicClient translates image_url data URLs to image blocks (video unsupported; Anthropic models get vision by editing the model row's capabilities — no catalog refresh writes them). On LLM fallback mid-round, messages are rebuilt with the replacement model's capabilities.