Daniele
4ea932ef54
Nightly Build / build (push) Successful in 10s
Both are added to the Z.AI static model list with their 1M-token context
and 128K max output. GLM-5.3-Flash is natively multimodal, so it gets the
vision and video capabilities — through an `override` rule, because the
provider's `defaults: { vision: false }` already set the flag and a fill
rule would have skipped it silently.
Neither model can stop thinking (`thinking.type` only accepts "enabled"),
so they get their own reasoning rule with low/high/max and no `disabled`,
placed before the `glm-5*` family rule that would otherwise swallow them.
43 lines
10 KiB
Markdown
43 lines
10 KiB
Markdown
*Skald dev-docs — architectural reference for coding agents. Index: [README.md](README.md) · Entry point: [../CLAUDE.md](../CLAUDE.md)*
|
|
|
|
**Read this when:** you touch LLM clients, `providers.yaml`, retriability, request logging, token streaming or multimodal attachments.
|
|
|
|
---
|
|
|
|
# The LLM stack
|
|
|
|
## The client layer (`crates/skald-core/src/llm/`)
|
|
|
|
LLM client abstraction (OpenAI-compat, Anthropic, Ollama…). OpenAI-compatible provider *types* are runtime data, not code: `providers/declared.rs` loads `providers.yaml` at boot (see Config in [../CLAUDE.md](../CLAUDE.md)); only non-OpenAI-compatible or bespoke providers (anthropic, ollama, openai, openrouter) stay native. **Retriability** (`Model::is_retriable`, `agent-loop`) keys on the real HTTP status carried by `ModelError { status }`, **not** a substring of the message — a model id/token count containing "404"/"401" cannot mis-classify; 401/403/404/422 don't retry, 400/429/5xx/network do. **Request logging** is the `logging.rs::LoggingModel` decorator, attached by the *caller's* `ModelSelector` (`loop_adapters/selector.rs::SkaldSelector::with_log`) — never by `LlmManager`, which builds one shared client per model and cannot know whose traffic it serves. The decorator's `RequestLogTarget` carries the owner: metadata → `llm_requests` in the registry (`user_id`, the column the UI filters on), payload bodies/headers → `llm_request_payloads` in that user's own encrypted DB, keyed by `request_id`; session + frame come from the request's own `conversation`/`frame`, so kernel rounds, sub-agent frames and compaction summaries are all attributed with no extra plumbing (`ModelRequest::log` is unused here)
|
|
|
|
## `providers.yaml` — two traps in the model metadata
|
|
|
|
**`enrich` rules stop at the first glob that matches, and the default `mode: fill` skips a field that already has a value.** Both bite when adding a model to an existing family. Ordering: `glm-5.3-flash` matches `glm-5*` too, so a rule for it placed *after* the family rule never runs — the specific glob goes first, always. And `fill` means "the endpoint listing wins", which for a `static:` list is not the same as "nothing is set": `models.defaults` (today only `vision`) stamps every entry before `enrich` sees it, so a provider carrying `defaults: { vision: false }` — Z.AI does — needs **`mode: override`** to turn vision on for one model. A `fill` rule there parses, loads, logs nothing, and leaves the flag off; the only symptom is that images silently keep taking the textual `<system-extra>` path (see [Multimodal attachments](#multimodal-attachments)) on a model that can read them. `vision: true` also pushes the `vision` capability, but only when the rule actually applied.
|
|
|
|
**A `reasoning.modes` `values` list is the whole contract with the provider — it is not a superset to trim in the UI.** Whatever it lists is what can be sent, so a model that cannot stop thinking (`thinking.type` accepting only `"enabled"`: GLM-5.3 and GLM-5.3-Flash) simply omits `disabled` from its rule, rather than inheriting the family's `[disabled, enabled]` toggle and sending a value the API rejects. With `request: { kind: thinking }`, any value other than `disabled`/`enabled` is emitted as `{"thinking":{"type":"enabled"},"reasoning_effort":v}`, which is exactly the shape those models want.
|
|
|
|
## The provider API surface — the key is a boolean, never a value
|
|
|
|
**No provider endpoint ever returns a stored `api_key`, and the trap is that omitting it silently reads as "no key".** `LlmProviderInfo` (list) and `ProviderDetail` (`src/frontend/api/llm.rs`, the detail DTO — deliberately *not* `LlmProviderRecord`, which does carry the secret) both expose **`has_api_key: bool`** instead. That is the whole contract: the UI needs to know *whether* a key is on file, never what it is, and the browser is where a leaked key would end up in a devtools tab or a screenshot.
|
|
|
|
It shipped broken in exactly the way this shape invites: `list_providers_info` never carried `api_key` (correctly), while the card tested `Boolean(p.api_key)` — always `undefined` — so every provider was badged "API key missing" even with a working key. A missing field is falsy, not an error; nothing logs, nothing fails to build. If you add a provider surface, read `has_api_key`, and if you add a field to either DTO, keep the secret out by construction rather than by remembering to strip it.
|
|
|
|
The consequence on the write path is load-bearing: since the edit form can no longer prefill the key, **an empty `api_key` in the `PUT` payload means "keep the stored one"** — `update_provider` re-reads the record and carries the old value over, because a blind `UPDATE … SET api_key=NULL` would wipe a working provider on any unrelated edit (a renamed description). The i18n placeholder (`providers.modal.api_key_ph`) already promised this behaviour before the backend implemented it. Side effect to know about: there is no longer a way to *clear* a key from the form — deleting the provider is the escape hatch.
|
|
|
|
## Token streaming & reasoning display
|
|
|
|
The chat streams tokens live, as a **parallel best-effort side-channel** that never alters the turn's authoritative flow: the final `Done` (or `Thinking`) event still carries the complete content and the frontend treats it as truth.
|
|
|
|
- **Client seam** (`core-api::chatbot`): `ChatbotClient::chat_with_tools_raw_streaming(..., delta_tx: mpsc::Sender<StreamDelta>)` — default impl ignores the channel and calls the buffered `chat_with_tools_raw`, so providers without streaming (Ollama, LM Studio) are untouched. `StreamDelta::{Text, Reasoning}` splits visible answer from chain-of-thought. Senders use `try_send` (deltas drop when the channel is full) — streaming must never backpressure the HTTP read.
|
|
- **SSE implementations** (`crates/llm-client`): `OpenAiClient` (`stream:true` + `stream_options.include_usage`, `reasoning_content`/`reasoning` deltas, index-based `tool_calls` accumulation, usage from the final chunk) and `AnthropicClient` (`stream:true`; `message_start`/`content_block_*`/`message_delta` events; `thinking_delta` → reasoning, `input_json_delta` → tool input). Both reassemble the **same `LlmTurn` + `LlmRawMeta`** the buffered path returns (the payload log stores a synthesized buffered-shaped body). Failure policy: if the stream dies **before any delta** the client retries buffered on the same model (providers rejecting `stream` keep working); a mid-stream failure propagates to the normal model-fallback logic. Framing is shared (`llm_client::SseDecoder`). Anthropic's **buffered** path now also parses `thinking` blocks into `reasoning_content` (previously discarded).
|
|
- **Loop wiring**: `call_llm_round` creates the delta channel per attempt and a forwarder task maps deltas to `ServerEvent::TokenDelta { kind: content|reasoning, delta }` on the turn's event channel (drained before the round's outcome events, so ordering holds); cancellation drops the in-flight future as before. A mid-stream fallback is handled client-side: the frontend clears its pending bubble on `model_fallback`.
|
|
- **Reasoning surfacing**: `reasoning_content` rides `Done`/`Thinking` events (so buffered providers show it live too) and is projected as `reasoning` on assistant/thinking history items (`build_items`); persistence in `chat_history.reasoning_content` and the echo back into context predate this feature.
|
|
- **Frontend** (`chat-session.js` + `copilot-render.js`, shared by desktop copilot and mobile chat-page): `token_delta` accumulates into a pending assistant bubble (in-place mutation + ~15 Hz flush, blinking caret); `done`/`thinking` finalize it in place, `error`/`llm_failed`/`model_fallback` drop it, `tool_start`/`agent_done` finalize orphan bubbles (reasoning-only rounds, sub-agent final rounds that emit no `Done`). The reasoning block is a muted, collapsed-by-default native `<details>` (`renderReasoning`, `.reasoning-block` in `copilot-messages.css`, i18n key `chat.reasoning`) — open state survives re-renders, and it renders identically from live events and from history.
|
|
|
|
## Multimodal attachments
|
|
|
|
Uploads go through **one centralized seam** — `ChatHub::save_upload` (behind `ChatHubApi::save_upload`, backed by `skald_core::uploads::save_to_home`) — so every surface persists identically and no two callers can drift on placement (the class of bug where the agent was handed a path it couldn't reach). The seam writes into the **caller's container home** under `uploads/{session_id}/` (agent path `uploads/{session}/{name}`, the `UPLOADS_SUBDIR` const in `core-api/user_fs.rs`), collision-dedupes the name, and prefers the sniffed magic-byte MIME over the client claim. The **web** handler (`POST /api/{source}/uploads`) buffers each field with a 256 MiB cap then calls the seam; the **Telegram** plugin downloads bytes then calls the same seam via `handle.chat_hub().save_upload("telegram", …)`. Because the file lands in the home (bind-mounted at `/root`), it is reachable by the fs-tools, `execute_cmd`, and the file viewer (`GET /api/file`, per-user via `resolve_view_path`) — there is **no** `/data` static route anymore (removed: it was `require_auth`-only, not ownership-scoped, and also exposed internal server state under `data/`). Attachment metadata travels as structured JSON in `chat_history.metadata` — never as persisted text.
|
|
|
|
At context-build time (the crate's projection), attachments of the **current turn** (the user/agent rows following the last completed assistant reply, including across in-flight tool rounds) are partitioned by `agent_loop::projection::media`, with `loop_adapters/media_source.rs` deciding **which** files may be handed over (§6 containment): when the resolved model's `LlmEntry.capabilities` include the modality (`vision` → `image_url` parts, `video` → `video_url` parts), the file is inlined as a base64 data-URL content part — but only if it resolves (through the caller's `UserFs`, via `resolve_host_path`) under the home's `uploads/` dir, its sniffed MIME is in the allowlist, and it fits the budgets (4 files / 10 MiB image / 32 MiB video / 48 MiB total per turn). Everything else — older turns, other kinds, any failed check — keeps the textual `<system-extra>` path block (built by `core_api::message_meta::attachments_block` / `system_extra`; the tag name is the single `SYSTEM_EXTRA_TAG` constant), so a non-vision model produces a byte-identical payload to before. `OpenAiClient` forwards parts verbatim; `AnthropicClient` translates `image_url` data URLs to `image` blocks (video unsupported; Anthropic models get `vision` by editing the model row's capabilities — no catalog refresh writes them). On LLM fallback mid-round, messages are rebuilt with the replacement model's capabilities.
|
|
|