Version 0.0.1 #2
@@ -188,6 +188,16 @@ Uploads (`POST /api/{source}/uploads`) are saved per-user under `data/uploads/{u
|
|||||||
|
|
||||||
At context-build time (`MessageBuilder`), attachments of the **current turn** (the user/agent rows following the last completed assistant reply, including across in-flight tool rounds) are partitioned by `session/handler/media.rs`: when the resolved model's `LlmEntry.capabilities` include the modality (`vision` → `image_url` parts, `video` → `video_url` parts), the file is inlined as a base64 data-URL content part — but only if it canonicalizes under `data/uploads/`, its sniffed MIME is in the allowlist, and it fits the budgets (4 files / 10 MiB image / 32 MiB video / 48 MiB total per turn). Everything else — older turns, other kinds, any failed check — keeps the textual `[SYSTEM INFO]` path block, so a non-vision model produces a byte-identical payload to before. `OpenAiClient` forwards parts verbatim; `AnthropicClient` translates `image_url` data URLs to `image` blocks (video unsupported; Anthropic models get `vision` by editing the model row's capabilities — no catalog refresh writes them). On LLM fallback mid-round, messages are rebuilt with the replacement model's capabilities.
|
At context-build time (`MessageBuilder`), attachments of the **current turn** (the user/agent rows following the last completed assistant reply, including across in-flight tool rounds) are partitioned by `session/handler/media.rs`: when the resolved model's `LlmEntry.capabilities` include the modality (`vision` → `image_url` parts, `video` → `video_url` parts), the file is inlined as a base64 data-URL content part — but only if it canonicalizes under `data/uploads/`, its sniffed MIME is in the allowlist, and it fits the budgets (4 files / 10 MiB image / 32 MiB video / 48 MiB total per turn). Everything else — older turns, other kinds, any failed check — keeps the textual `[SYSTEM INFO]` path block, so a non-vision model produces a byte-identical payload to before. `OpenAiClient` forwards parts verbatim; `AnthropicClient` translates `image_url` data URLs to `image` blocks (video unsupported; Anthropic models get `vision` by editing the model row's capabilities — no catalog refresh writes them). On LLM fallback mid-round, messages are rebuilt with the replacement model's capabilities.
|
||||||
|
|
||||||
|
## Token streaming & reasoning display
|
||||||
|
|
||||||
|
The chat streams tokens live, as a **parallel best-effort side-channel** that never alters the turn's authoritative flow: the final `Done` (or `Thinking`) event still carries the complete content and the frontend treats it as truth.
|
||||||
|
|
||||||
|
- **Client seam** (`core-api::chatbot`): `ChatbotClient::chat_with_tools_raw_streaming(..., delta_tx: mpsc::Sender<StreamDelta>)` — default impl ignores the channel and calls the buffered `chat_with_tools_raw`, so providers without streaming (Ollama, LM Studio) are untouched. `StreamDelta::{Text, Reasoning}` splits visible answer from chain-of-thought. Senders use `try_send` (deltas drop when the channel is full) — streaming must never backpressure the HTTP read.
|
||||||
|
- **SSE implementations** (`crates/llm-client`): `OpenAiClient` (`stream:true` + `stream_options.include_usage`, `reasoning_content`/`reasoning` deltas, index-based `tool_calls` accumulation, usage from the final chunk) and `AnthropicClient` (`stream:true`; `message_start`/`content_block_*`/`message_delta` events; `thinking_delta` → reasoning, `input_json_delta` → tool input). Both reassemble the **same `LlmTurn` + `LlmRawMeta`** the buffered path returns (the payload log stores a synthesized buffered-shaped body). Failure policy: if the stream dies **before any delta** the client retries buffered on the same model (providers rejecting `stream` keep working); a mid-stream failure propagates to the normal model-fallback logic. Framing is shared (`llm_client::SseDecoder`). Anthropic's **buffered** path now also parses `thinking` blocks into `reasoning_content` (previously discarded).
|
||||||
|
- **Loop wiring**: `call_llm_round` creates the delta channel per attempt and a forwarder task maps deltas to `ServerEvent::TokenDelta { kind: content|reasoning, delta }` on the turn's event channel (drained before the round's outcome events, so ordering holds); cancellation drops the in-flight future as before. A mid-stream fallback is handled client-side: the frontend clears its pending bubble on `model_fallback`.
|
||||||
|
- **Reasoning surfacing**: `reasoning_content` rides `Done`/`Thinking` events (so buffered providers show it live too) and is projected as `reasoning` on assistant/thinking history items (`build_items`); persistence in `chat_history.reasoning_content` and the echo back into context predate this feature.
|
||||||
|
- **Frontend** (`chat-session.js` + `copilot-render.js`, shared by desktop copilot and mobile chat-page): `token_delta` accumulates into a pending assistant bubble (in-place mutation + ~15 Hz flush, blinking caret); `done`/`thinking` finalize it in place, `error`/`llm_failed`/`model_fallback` drop it, `tool_start`/`agent_done` finalize orphan bubbles (reasoning-only rounds, sub-agent final rounds that emit no `Done`). The reasoning block is a muted, collapsed-by-default native `<details>` (`renderReasoning`, `.reasoning-block` in `copilot-messages.css`, i18n key `chat.reasoning`) — open state survives re-renders, and it renders identically from live events and from history.
|
||||||
|
|
||||||
## Sub-agent system
|
## Sub-agent system
|
||||||
- Synchronous sub-agents (`execute_task` mode=sync / `execute_subtask`) are **not** plain `Tool`s — they are intercepted in `run_agent_turn` before registry dispatch.
|
- Synchronous sub-agents (`execute_task` mode=sync / `execute_subtask`) are **not** plain `Tool`s — they are intercepted in `run_agent_turn` before registry dispatch.
|
||||||
- `dispatch_sub_agent` (in `agent_dispatch.rs`) creates a child `chat_sessions_stack` row and runs `run_agent_turn` **recursively in the same task**, holding the same `processing` lock and sharing the same cancellation token. The child's result string becomes the parent tool call's result (completion lives in one place — the `run_agent_turn` tool-result match); then it terminates the child frame. There is no task-spawn / `WaitingChild` / resume cascade for the sync path.
|
- `dispatch_sub_agent` (in `agent_dispatch.rs`) creates a child `chat_sessions_stack` row and runs `run_agent_turn` **recursively in the same task**, holding the same `processing` lock and sharing the same cancellation token. The child's result string becomes the parent tool call's result (completion lives in one place — the `run_agent_turn` tool-result match); then it terminates the child frame. There is no task-spawn / `WaitingChild` / resume cascade for the sync path.
|
||||||
|
|||||||
Generated
+18
-1
@@ -2189,9 +2189,11 @@ dependencies = [
|
|||||||
"anyhow",
|
"anyhow",
|
||||||
"async-trait",
|
"async-trait",
|
||||||
"core-api",
|
"core-api",
|
||||||
|
"futures-util",
|
||||||
"reqwest 0.13.4",
|
"reqwest 0.13.4",
|
||||||
"serde",
|
"serde",
|
||||||
"serde_json",
|
"serde_json",
|
||||||
|
"tokio",
|
||||||
"tracing",
|
"tracing",
|
||||||
]
|
]
|
||||||
|
|
||||||
@@ -3613,7 +3615,7 @@ dependencies = [
|
|||||||
"url",
|
"url",
|
||||||
"wasm-bindgen",
|
"wasm-bindgen",
|
||||||
"wasm-bindgen-futures",
|
"wasm-bindgen-futures",
|
||||||
"wasm-streams",
|
"wasm-streams 0.4.2",
|
||||||
"web-sys",
|
"web-sys",
|
||||||
"webpki-roots 1.0.7",
|
"webpki-roots 1.0.7",
|
||||||
]
|
]
|
||||||
@@ -3650,12 +3652,14 @@ dependencies = [
|
|||||||
"sync_wrapper",
|
"sync_wrapper",
|
||||||
"tokio",
|
"tokio",
|
||||||
"tokio-rustls",
|
"tokio-rustls",
|
||||||
|
"tokio-util",
|
||||||
"tower",
|
"tower",
|
||||||
"tower-http 0.6.11",
|
"tower-http 0.6.11",
|
||||||
"tower-service",
|
"tower-service",
|
||||||
"url",
|
"url",
|
||||||
"wasm-bindgen",
|
"wasm-bindgen",
|
||||||
"wasm-bindgen-futures",
|
"wasm-bindgen-futures",
|
||||||
|
"wasm-streams 0.5.0",
|
||||||
"web-sys",
|
"web-sys",
|
||||||
]
|
]
|
||||||
|
|
||||||
@@ -6087,6 +6091,19 @@ dependencies = [
|
|||||||
"web-sys",
|
"web-sys",
|
||||||
]
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "wasm-streams"
|
||||||
|
version = "0.5.0"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "9d1ec4f6517c9e11ae630e200b2b65d193279042e28edd4a2cda233e46670bbb"
|
||||||
|
dependencies = [
|
||||||
|
"futures-util",
|
||||||
|
"js-sys",
|
||||||
|
"wasm-bindgen",
|
||||||
|
"wasm-bindgen-futures",
|
||||||
|
"web-sys",
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "wasmparser"
|
name = "wasmparser"
|
||||||
version = "0.244.0"
|
version = "0.244.0"
|
||||||
|
|||||||
@@ -1,5 +1,6 @@
|
|||||||
use async_trait::async_trait;
|
use async_trait::async_trait;
|
||||||
use serde_json::Value;
|
use serde_json::Value;
|
||||||
|
use tokio::sync::mpsc;
|
||||||
|
|
||||||
/// A single message in a conversation.
|
/// A single message in a conversation.
|
||||||
#[derive(Debug, Clone)]
|
#[derive(Debug, Clone)]
|
||||||
@@ -90,6 +91,17 @@ pub struct ToolCall {
|
|||||||
pub arguments: Value,
|
pub arguments: Value,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// An incremental piece of a streaming completion, pushed by providers that
|
||||||
|
/// support SSE streaming. Purely best-effort UI feedback: the final `LlmTurn`
|
||||||
|
/// remains the authoritative result.
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
|
pub enum StreamDelta {
|
||||||
|
/// Visible answer text.
|
||||||
|
Text(String),
|
||||||
|
/// Chain-of-thought / reasoning tokens (thinking models).
|
||||||
|
Reasoning(String),
|
||||||
|
}
|
||||||
|
|
||||||
/// Result of one LLM turn when tools are available.
|
/// Result of one LLM turn when tools are available.
|
||||||
#[derive(Debug)]
|
#[derive(Debug)]
|
||||||
pub enum LlmTurn {
|
pub enum LlmTurn {
|
||||||
@@ -160,4 +172,22 @@ pub trait ChatbotClient: Send + Sync {
|
|||||||
) -> anyhow::Result<(LlmTurn, Option<LlmRawMeta>)> {
|
) -> anyhow::Result<(LlmTurn, Option<LlmRawMeta>)> {
|
||||||
self.chat_with_tools(messages, tools, options).await.map(|t| (t, None))
|
self.chat_with_tools(messages, tools, options).await.map(|t| (t, None))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Like `chat_with_tools_raw`, but the provider may push incremental
|
||||||
|
/// [`StreamDelta`]s into `delta_tx` as tokens arrive (SSE streaming).
|
||||||
|
/// Senders should use `try_send` and drop deltas when the channel is full —
|
||||||
|
/// streaming is best-effort UI feedback and must never backpressure the
|
||||||
|
/// HTTP read. The returned `LlmTurn` is always the complete, authoritative
|
||||||
|
/// result. The default ignores the channel and falls back to the buffered
|
||||||
|
/// call, so providers without streaming behave exactly as before.
|
||||||
|
async fn chat_with_tools_raw_streaming(
|
||||||
|
&self,
|
||||||
|
messages: &[Value],
|
||||||
|
tools: &[Value],
|
||||||
|
options: &ChatOptions,
|
||||||
|
delta_tx: mpsc::Sender<StreamDelta>,
|
||||||
|
) -> anyhow::Result<(LlmTurn, Option<LlmRawMeta>)> {
|
||||||
|
let _ = delta_tx;
|
||||||
|
self.chat_with_tools_raw(messages, tools, options).await
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -34,6 +34,16 @@ pub struct GlobalEvent {
|
|||||||
|
|
||||||
// ── Server → Client ───────────────────────────────────────────────────────────
|
// ── Server → Client ───────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
/// Which token stream a [`ServerEvent::TokenDelta`] belongs to.
|
||||||
|
#[derive(Clone, Copy, Serialize)]
|
||||||
|
#[serde(rename_all = "snake_case")]
|
||||||
|
pub enum TokenDeltaKind {
|
||||||
|
/// Visible answer text.
|
||||||
|
Content,
|
||||||
|
/// Model chain-of-thought (reasoning/thinking tokens).
|
||||||
|
Reasoning,
|
||||||
|
}
|
||||||
|
|
||||||
#[derive(Clone, Serialize)]
|
#[derive(Clone, Serialize)]
|
||||||
#[serde(tag = "type", rename_all = "snake_case")]
|
#[serde(tag = "type", rename_all = "snake_case")]
|
||||||
pub enum ServerEvent {
|
pub enum ServerEvent {
|
||||||
@@ -117,6 +127,10 @@ pub enum ServerEvent {
|
|||||||
content: String,
|
content: String,
|
||||||
input_tokens: Option<u32>,
|
input_tokens: Option<u32>,
|
||||||
output_tokens: Option<u32>,
|
output_tokens: Option<u32>,
|
||||||
|
/// Chain-of-thought, when the model produced any. Lets non-streaming
|
||||||
|
/// providers surface the reasoning block live, not just from history.
|
||||||
|
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||||
|
reasoning_content: Option<String>,
|
||||||
},
|
},
|
||||||
/// A fatal error occurred processing the request.
|
/// A fatal error occurred processing the request.
|
||||||
Error {
|
Error {
|
||||||
@@ -132,6 +146,17 @@ pub enum ServerEvent {
|
|||||||
content: String,
|
content: String,
|
||||||
input_tokens: Option<u32>,
|
input_tokens: Option<u32>,
|
||||||
output_tokens: Option<u32>,
|
output_tokens: Option<u32>,
|
||||||
|
/// Chain-of-thought of this tool-call round, when produced.
|
||||||
|
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||||
|
reasoning_content: Option<String>,
|
||||||
|
},
|
||||||
|
/// One incremental token while the assistant response (or its reasoning)
|
||||||
|
/// is being generated. Best-effort: deltas ride the lossy broadcast bus and
|
||||||
|
/// a lagging client may miss some — the final `Done` is always authoritative
|
||||||
|
/// and carries the complete content.
|
||||||
|
TokenDelta {
|
||||||
|
kind: TokenDeltaKind,
|
||||||
|
delta: String,
|
||||||
},
|
},
|
||||||
/// A write operation requires user approval before executing (shows a diff).
|
/// A write operation requires user approval before executing (shows a diff).
|
||||||
PendingWrite {
|
PendingWrite {
|
||||||
@@ -279,6 +304,7 @@ impl ServerEvent {
|
|||||||
Self::Done { .. } => "done",
|
Self::Done { .. } => "done",
|
||||||
Self::Error { .. } => "error",
|
Self::Error { .. } => "error",
|
||||||
Self::Thinking { .. } => "thinking",
|
Self::Thinking { .. } => "thinking",
|
||||||
|
Self::TokenDelta { .. } => "token_delta",
|
||||||
Self::PendingWrite { .. } => "pending_write",
|
Self::PendingWrite { .. } => "pending_write",
|
||||||
Self::ApprovalRequired { .. } => "approval_required",
|
Self::ApprovalRequired { .. } => "approval_required",
|
||||||
Self::AgentQuestion { .. } => "agent_question",
|
Self::AgentQuestion { .. } => "agent_question",
|
||||||
|
|||||||
@@ -5,9 +5,11 @@ edition = "2024"
|
|||||||
|
|
||||||
[dependencies]
|
[dependencies]
|
||||||
core-api = { path = "../core-api" }
|
core-api = { path = "../core-api" }
|
||||||
reqwest = { version = "0.13", default-features = false, features = ["rustls-no-provider", "charset", "http2", "system-proxy", "json"] }
|
reqwest = { version = "0.13", default-features = false, features = ["rustls-no-provider", "charset", "http2", "system-proxy", "json", "stream"] }
|
||||||
serde = { version = "1", features = ["derive"] }
|
serde = { version = "1", features = ["derive"] }
|
||||||
serde_json = "1"
|
serde_json = "1"
|
||||||
async-trait = "0.1"
|
async-trait = "0.1"
|
||||||
anyhow = "1"
|
anyhow = "1"
|
||||||
tracing = "0.1"
|
tracing = "0.1"
|
||||||
|
tokio = { version = "1", features = ["sync"] }
|
||||||
|
futures-util = "0.3"
|
||||||
|
|||||||
@@ -1,8 +1,12 @@
|
|||||||
|
use std::collections::BTreeMap;
|
||||||
|
|
||||||
use async_trait::async_trait;
|
use async_trait::async_trait;
|
||||||
|
use futures_util::StreamExt;
|
||||||
use serde_json::{Value, json};
|
use serde_json::{Value, json};
|
||||||
|
use tokio::sync::mpsc;
|
||||||
use tracing::{debug, info, trace, warn};
|
use tracing::{debug, info, trace, warn};
|
||||||
|
|
||||||
use crate::{ChatOptions, ChatResponse, ChatbotClient, LlmRawMeta, LlmTurn, Message, Role, ToolCall, headers_to_json, redact_key};
|
use crate::{ChatOptions, ChatResponse, ChatbotClient, LlmRawMeta, LlmTurn, Message, Role, SseDecoder, StreamDelta, ToolCall, headers_to_json, redact_key};
|
||||||
|
|
||||||
const DEFAULT_BASE_URL: &str = "https://api.anthropic.com";
|
const DEFAULT_BASE_URL: &str = "https://api.anthropic.com";
|
||||||
const ANTHROPIC_VERSION: &str = "2023-06-01";
|
const ANTHROPIC_VERSION: &str = "2023-06-01";
|
||||||
@@ -157,6 +161,242 @@ impl AnthropicClient {
|
|||||||
|
|
||||||
out
|
out
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Assembles the `/v1/messages` request body shared by the buffered and the
|
||||||
|
/// streaming path (the caller adds `stream` on top).
|
||||||
|
fn tools_body(&self, system: Option<String>, messages: Vec<Value>, tools: Vec<Value>, options: &ChatOptions) -> Value {
|
||||||
|
let max_tokens = options.max_tokens.unwrap_or(4096);
|
||||||
|
let mut body = json!({
|
||||||
|
"model": options.model,
|
||||||
|
"max_tokens": max_tokens,
|
||||||
|
"messages": messages,
|
||||||
|
"tools": tools,
|
||||||
|
});
|
||||||
|
|
||||||
|
if let Some(sys) = system { body["system"] = sys.into(); }
|
||||||
|
if let Some(t) = options.temperature { body["temperature"] = t.into(); }
|
||||||
|
self.apply_extra(&mut body);
|
||||||
|
body
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Collects ALL system-role messages (main prompt, mid-conversation
|
||||||
|
/// summary, tail_reminder) into a single `system:` string. The Anthropic
|
||||||
|
/// API only accepts a single system parameter.
|
||||||
|
fn merged_system(messages: &[Value]) -> Option<String> {
|
||||||
|
let parts: Vec<&str> = messages
|
||||||
|
.iter()
|
||||||
|
.filter(|m| m["role"].as_str() == Some("system"))
|
||||||
|
.filter_map(|m| m["content"].as_str())
|
||||||
|
.collect();
|
||||||
|
if parts.is_empty() { None } else { Some(parts.join("\n\n---\n\n")) }
|
||||||
|
}
|
||||||
|
|
||||||
|
fn url(&self) -> String {
|
||||||
|
format!("{}/v1/messages", self.base_url.trim_end_matches('/'))
|
||||||
|
}
|
||||||
|
|
||||||
|
fn logged_headers(&self) -> Value {
|
||||||
|
json!({
|
||||||
|
"x-api-key": redact_key(&self.api_key),
|
||||||
|
"anthropic-version": ANTHROPIC_VERSION,
|
||||||
|
"content-type": "application/json",
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn send_request(&self, body: &Value) -> reqwest::Result<reqwest::Response> {
|
||||||
|
self.http
|
||||||
|
.post(self.url())
|
||||||
|
.header("x-api-key", &self.api_key)
|
||||||
|
.header("anthropic-version", ANTHROPIC_VERSION)
|
||||||
|
.header("X-Title", core_api::APP_NAME)
|
||||||
|
.json(body)
|
||||||
|
.send()
|
||||||
|
.await?
|
||||||
|
.error_for_status()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Joined `thinking` blocks of a content array, if any (extended thinking).
|
||||||
|
fn reasoning_of(content_blocks: &[Value]) -> Option<String> {
|
||||||
|
let parts: Vec<&str> = content_blocks
|
||||||
|
.iter()
|
||||||
|
.filter(|b| b["type"].as_str() == Some("thinking"))
|
||||||
|
.filter_map(|b| b["thinking"].as_str())
|
||||||
|
.collect();
|
||||||
|
if parts.is_empty() { None } else { Some(parts.join("\n")) }
|
||||||
|
}
|
||||||
|
|
||||||
|
/// SSE streaming path behind `chat_with_tools_raw_streaming`. Anthropic
|
||||||
|
/// streams typed events (`message_start` / `content_block_*` /
|
||||||
|
/// `message_delta` / `message_stop`); text and thinking deltas are
|
||||||
|
/// forwarded to `delta_tx` best-effort while the blocks are accumulated
|
||||||
|
/// into the same `LlmTurn` the buffered path returns.
|
||||||
|
async fn stream_chat(
|
||||||
|
&self,
|
||||||
|
messages: &[Value],
|
||||||
|
tools: &[Value],
|
||||||
|
options: &ChatOptions,
|
||||||
|
delta_tx: &mpsc::Sender<StreamDelta>,
|
||||||
|
emitted: &mut bool,
|
||||||
|
) -> anyhow::Result<(LlmTurn, Option<LlmRawMeta>)> {
|
||||||
|
let system = Self::merged_system(messages);
|
||||||
|
let anthropic_messages = Self::convert_messages(messages);
|
||||||
|
let anthropic_tools = Self::convert_tools(tools);
|
||||||
|
let mut body = self.tools_body(system, anthropic_messages, anthropic_tools, options);
|
||||||
|
body["stream"] = json!(true);
|
||||||
|
|
||||||
|
debug!(model = %options.model, tools = tools.len(), "anthropic: sending streaming chat_with_tools request");
|
||||||
|
trace!(body = %body, "anthropic: streaming chat_with_tools request body");
|
||||||
|
|
||||||
|
let request_body = body.clone();
|
||||||
|
let request_headers = self.logged_headers();
|
||||||
|
|
||||||
|
let http_resp = self.send_request(&body).await?;
|
||||||
|
let response_headers = headers_to_json(http_resp.headers());
|
||||||
|
|
||||||
|
/// One content block being accumulated by index.
|
||||||
|
#[derive(Default)]
|
||||||
|
struct Block {
|
||||||
|
kind: String, // "text" | "thinking" | "tool_use"
|
||||||
|
buf: String, // text/thinking content or input_json fragments
|
||||||
|
id: String,
|
||||||
|
name: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
let mut blocks: BTreeMap<u64, Block> = BTreeMap::new();
|
||||||
|
let mut stop_reason: Option<String> = None;
|
||||||
|
let mut usage = json!({});
|
||||||
|
let mut sse = SseDecoder::new();
|
||||||
|
let mut byte_stream = http_resp.bytes_stream();
|
||||||
|
|
||||||
|
let mut handle_payload = |payload: &str, emitted: &mut bool| -> anyhow::Result<()> {
|
||||||
|
let Ok(v) = serde_json::from_str::<Value>(payload) else { return Ok(()) };
|
||||||
|
match v["type"].as_str().unwrap_or("") {
|
||||||
|
"message_start" => {
|
||||||
|
if let Some(u) = v["message"]["usage"].as_object() {
|
||||||
|
for (k, val) in u { usage[k.clone()] = val.clone(); }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
"content_block_start" => {
|
||||||
|
let idx = v["index"].as_u64().unwrap_or(0);
|
||||||
|
let cb = &v["content_block"];
|
||||||
|
let block = blocks.entry(idx).or_default();
|
||||||
|
block.kind = cb["type"].as_str().unwrap_or("").to_string();
|
||||||
|
block.id = cb["id"].as_str().unwrap_or("").to_string();
|
||||||
|
block.name = cb["name"].as_str().unwrap_or("").to_string();
|
||||||
|
}
|
||||||
|
"content_block_delta" => {
|
||||||
|
let idx = v["index"].as_u64().unwrap_or(0);
|
||||||
|
let delta = &v["delta"];
|
||||||
|
match delta["type"].as_str().unwrap_or("") {
|
||||||
|
"text_delta" => {
|
||||||
|
if let Some(t) = delta["text"].as_str().filter(|t| !t.is_empty()) {
|
||||||
|
blocks.entry(idx).or_default().buf.push_str(t);
|
||||||
|
*emitted = true;
|
||||||
|
let _ = delta_tx.try_send(StreamDelta::Text(t.to_string()));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
"thinking_delta" => {
|
||||||
|
if let Some(t) = delta["thinking"].as_str().filter(|t| !t.is_empty()) {
|
||||||
|
blocks.entry(idx).or_default().buf.push_str(t);
|
||||||
|
*emitted = true;
|
||||||
|
let _ = delta_tx.try_send(StreamDelta::Reasoning(t.to_string()));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
"input_json_delta" => {
|
||||||
|
if let Some(j) = delta["partial_json"].as_str() {
|
||||||
|
blocks.entry(idx).or_default().buf.push_str(j);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// signature_delta and unknown deltas carry no displayable text.
|
||||||
|
_ => {}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
"message_delta" => {
|
||||||
|
if let Some(sr) = v["delta"]["stop_reason"].as_str() {
|
||||||
|
stop_reason = Some(sr.to_string());
|
||||||
|
}
|
||||||
|
if let Some(u) = v["usage"].as_object() {
|
||||||
|
for (k, val) in u { usage[k.clone()] = val.clone(); }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
"error" => {
|
||||||
|
return Err(anyhow::anyhow!("anthropic: stream error event: {payload}"));
|
||||||
|
}
|
||||||
|
// content_block_stop / message_stop / ping: nothing to accumulate.
|
||||||
|
_ => {}
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
};
|
||||||
|
|
||||||
|
while let Some(chunk) = byte_stream.next().await {
|
||||||
|
let chunk = chunk?;
|
||||||
|
for payload in sse.feed(&chunk) {
|
||||||
|
handle_payload(&payload, emitted)?;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for payload in sse.finish() {
|
||||||
|
handle_payload(&payload, emitted)?;
|
||||||
|
}
|
||||||
|
|
||||||
|
let stop = stop_reason.as_deref().unwrap_or("");
|
||||||
|
let input_tokens = usage["input_tokens"].as_u64().map(|n| n as u32);
|
||||||
|
let output_tokens = usage["output_tokens"].as_u64().map(|n| n as u32);
|
||||||
|
let cache_read_tokens = usage["cache_read_input_tokens"].as_u64().map(|n| n as u32);
|
||||||
|
let cache_creation_tokens = usage["cache_creation_input_tokens"].as_u64().map(|n| n as u32);
|
||||||
|
info!(model = %options.model, ?input_tokens, ?output_tokens, stop_reason = stop, "anthropic: streaming response completed");
|
||||||
|
if stop == "max_tokens" {
|
||||||
|
warn!(model = %options.model, ?output_tokens, "anthropic: response truncated (max_tokens reached)");
|
||||||
|
}
|
||||||
|
|
||||||
|
let text_of = |kind: &str| -> String {
|
||||||
|
blocks.values()
|
||||||
|
.filter(|b| b.kind == kind)
|
||||||
|
.map(|b| b.buf.as_str())
|
||||||
|
.collect::<Vec<_>>()
|
||||||
|
.join("\n")
|
||||||
|
};
|
||||||
|
let reasoning = text_of("thinking");
|
||||||
|
let reasoning_content = if reasoning.is_empty() { None } else { Some(reasoning) };
|
||||||
|
let tool_blocks: Vec<&Block> = blocks.values().filter(|b| b.kind == "tool_use").collect();
|
||||||
|
|
||||||
|
let turn = if !tool_blocks.is_empty() {
|
||||||
|
let calls = tool_blocks
|
||||||
|
.iter()
|
||||||
|
.map(|b| ToolCall {
|
||||||
|
id: b.id.clone(),
|
||||||
|
name: b.name.clone(),
|
||||||
|
arguments: serde_json::from_str(&b.buf).unwrap_or(Value::Object(Default::default())),
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
LlmTurn::ToolCalls { content: text_of("text"), calls, input_tokens, output_tokens, reasoning_content, cache_read_tokens, cache_creation_tokens, cost: None }
|
||||||
|
} else {
|
||||||
|
let truncated = stop == "max_tokens";
|
||||||
|
LlmTurn::Message(ChatResponse {
|
||||||
|
content: text_of("text"), input_tokens, output_tokens, truncated,
|
||||||
|
reasoning_content, cache_read_tokens, cache_creation_tokens, cost: None,
|
||||||
|
})
|
||||||
|
};
|
||||||
|
|
||||||
|
// Buffered-shaped response body for the payload log.
|
||||||
|
let content_log: Vec<Value> = blocks.values().map(|b| match b.kind.as_str() {
|
||||||
|
"tool_use" => json!({"type": "tool_use", "id": b.id, "name": b.name, "input": serde_json::from_str::<Value>(&b.buf).unwrap_or(json!({}))}),
|
||||||
|
"thinking" => json!({"type": "thinking", "thinking": b.buf}),
|
||||||
|
_ => json!({"type": "text", "text": b.buf}),
|
||||||
|
}).collect();
|
||||||
|
let raw_meta = LlmRawMeta {
|
||||||
|
request_headers: Some(request_headers),
|
||||||
|
request_body: Some(request_body),
|
||||||
|
response_headers: Some(response_headers),
|
||||||
|
response_body: Some(json!({
|
||||||
|
"streamed": true,
|
||||||
|
"content": content_log,
|
||||||
|
"stop_reason": stop,
|
||||||
|
"usage": usage,
|
||||||
|
})),
|
||||||
|
};
|
||||||
|
|
||||||
|
Ok((turn, Some(raw_meta)))
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// User content arrives either as a plain string or as an OpenAI-style parts
|
/// User content arrives either as a plain string or as an OpenAI-style parts
|
||||||
@@ -274,7 +514,7 @@ impl ChatbotClient for AnthropicClient {
|
|||||||
|
|
||||||
let content = resp["content"]
|
let content = resp["content"]
|
||||||
.as_array()
|
.as_array()
|
||||||
.and_then(|arr| arr.first())
|
.and_then(|arr| arr.iter().find(|b| b["type"].as_str() == Some("text")))
|
||||||
.and_then(|block| block["text"].as_str())
|
.and_then(|block| block["text"].as_str())
|
||||||
.ok_or_else(|| anyhow::anyhow!("Missing content in Anthropic response"))?
|
.ok_or_else(|| anyhow::anyhow!("Missing content in Anthropic response"))?
|
||||||
.to_string();
|
.to_string();
|
||||||
@@ -304,58 +544,22 @@ impl ChatbotClient for AnthropicClient {
|
|||||||
tools: &[Value],
|
tools: &[Value],
|
||||||
options: &ChatOptions,
|
options: &ChatOptions,
|
||||||
) -> anyhow::Result<(LlmTurn, Option<LlmRawMeta>)> {
|
) -> anyhow::Result<(LlmTurn, Option<LlmRawMeta>)> {
|
||||||
// Collect ALL system-role messages (main prompt, mid-conversation
|
// Mid-conversation system messages (compaction summaries, tail
|
||||||
// summary, tail_reminder) and merge them into a single `system:`
|
// reminders) are merged into the single `system:` parameter — they
|
||||||
// string. The Anthropic API only accepts a single system parameter;
|
// must not be silently dropped.
|
||||||
// mid-conversation system messages generated by build_openai_messages
|
let system = Self::merged_system(messages);
|
||||||
// are intentionally used for injecting compaction summaries and tail
|
|
||||||
// reminders — they must not be silently dropped.
|
|
||||||
let system: Option<String> = {
|
|
||||||
let parts: Vec<&str> = messages
|
|
||||||
.iter()
|
|
||||||
.filter(|m| m["role"].as_str() == Some("system"))
|
|
||||||
.filter_map(|m| m["content"].as_str())
|
|
||||||
.collect();
|
|
||||||
if parts.is_empty() { None } else { Some(parts.join("\n\n---\n\n")) }
|
|
||||||
};
|
|
||||||
|
|
||||||
let anthropic_messages = Self::convert_messages(messages);
|
let anthropic_messages = Self::convert_messages(messages);
|
||||||
let anthropic_tools = Self::convert_tools(tools);
|
let anthropic_tools = Self::convert_tools(tools);
|
||||||
|
let body = self.tools_body(system, anthropic_messages, anthropic_tools, options);
|
||||||
|
|
||||||
let max_tokens = options.max_tokens.unwrap_or(4096);
|
|
||||||
let mut body = json!({
|
|
||||||
"model": options.model,
|
|
||||||
"max_tokens": max_tokens,
|
|
||||||
"messages": anthropic_messages,
|
|
||||||
"tools": anthropic_tools,
|
|
||||||
});
|
|
||||||
|
|
||||||
if let Some(sys) = system { body["system"] = sys.into(); }
|
|
||||||
if let Some(t) = options.temperature { body["temperature"] = t.into(); }
|
|
||||||
self.apply_extra(&mut body);
|
|
||||||
|
|
||||||
let url = format!("{}/v1/messages", self.base_url.trim_end_matches('/'));
|
|
||||||
debug!(model = %options.model, tools = tools.len(), "anthropic: sending chat_with_tools request");
|
debug!(model = %options.model, tools = tools.len(), "anthropic: sending chat_with_tools request");
|
||||||
trace!(body = %body, "anthropic: chat_with_tools request body");
|
trace!(body = %body, "anthropic: chat_with_tools request body");
|
||||||
|
|
||||||
// Capture request metadata for logging.
|
// Capture request metadata for logging.
|
||||||
let request_body = body.clone();
|
let request_body = body.clone();
|
||||||
let request_headers = json!({
|
let request_headers = self.logged_headers();
|
||||||
"x-api-key": redact_key(&self.api_key),
|
|
||||||
"anthropic-version": ANTHROPIC_VERSION,
|
|
||||||
"content-type": "application/json",
|
|
||||||
});
|
|
||||||
|
|
||||||
let http_resp = self
|
let http_resp = self.send_request(&body).await?;
|
||||||
.http
|
|
||||||
.post(&url)
|
|
||||||
.header("x-api-key", &self.api_key)
|
|
||||||
.header("anthropic-version", ANTHROPIC_VERSION)
|
|
||||||
.header("X-Title", core_api::APP_NAME)
|
|
||||||
.json(&body)
|
|
||||||
.send()
|
|
||||||
.await?
|
|
||||||
.error_for_status()?;
|
|
||||||
|
|
||||||
let response_headers = headers_to_json(http_resp.headers());
|
let response_headers = headers_to_json(http_resp.headers());
|
||||||
let resp_text = http_resp.text().await?;
|
let resp_text = http_resp.text().await?;
|
||||||
@@ -383,6 +587,7 @@ impl ChatbotClient for AnthropicClient {
|
|||||||
}
|
}
|
||||||
|
|
||||||
let has_tool_use = content_blocks.iter().any(|b| b["type"].as_str() == Some("tool_use"));
|
let has_tool_use = content_blocks.iter().any(|b| b["type"].as_str() == Some("tool_use"));
|
||||||
|
let reasoning_content = Self::reasoning_of(&content_blocks);
|
||||||
|
|
||||||
// Check content blocks directly: Anthropic sometimes returns stop_reason "end_turn"
|
// Check content blocks directly: Anthropic sometimes returns stop_reason "end_turn"
|
||||||
// even when tool_use blocks are present, so stop_reason alone is not reliable.
|
// even when tool_use blocks are present, so stop_reason alone is not reliable.
|
||||||
@@ -404,7 +609,7 @@ impl ChatbotClient for AnthropicClient {
|
|||||||
})
|
})
|
||||||
.collect();
|
.collect();
|
||||||
|
|
||||||
LlmTurn::ToolCalls { content: text, calls, input_tokens, output_tokens, reasoning_content: None, cache_read_tokens, cache_creation_tokens, cost }
|
LlmTurn::ToolCalls { content: text, calls, input_tokens, output_tokens, reasoning_content, cache_read_tokens, cache_creation_tokens, cost }
|
||||||
} else {
|
} else {
|
||||||
let content = content_blocks
|
let content = content_blocks
|
||||||
.iter()
|
.iter()
|
||||||
@@ -414,17 +619,55 @@ impl ChatbotClient for AnthropicClient {
|
|||||||
.to_string();
|
.to_string();
|
||||||
|
|
||||||
let truncated = stop_reason == "max_tokens";
|
let truncated = stop_reason == "max_tokens";
|
||||||
LlmTurn::Message(ChatResponse { content, input_tokens, output_tokens, truncated, reasoning_content: None, cache_read_tokens, cache_creation_tokens, cost })
|
LlmTurn::Message(ChatResponse { content, input_tokens, output_tokens, truncated, reasoning_content, cache_read_tokens, cache_creation_tokens, cost })
|
||||||
};
|
};
|
||||||
|
|
||||||
Ok((turn, Some(raw_meta)))
|
Ok((turn, Some(raw_meta)))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
async fn chat_with_tools_raw_streaming(
|
||||||
|
&self,
|
||||||
|
messages: &[Value],
|
||||||
|
tools: &[Value],
|
||||||
|
options: &ChatOptions,
|
||||||
|
delta_tx: mpsc::Sender<StreamDelta>,
|
||||||
|
) -> anyhow::Result<(LlmTurn, Option<LlmRawMeta>)> {
|
||||||
|
let mut emitted = false;
|
||||||
|
match self.stream_chat(messages, tools, options, &delta_tx, &mut emitted).await {
|
||||||
|
Ok(ok) => Ok(ok),
|
||||||
|
// Pre-stream failure (nothing shown yet): retry buffered. A
|
||||||
|
// mid-stream failure propagates to the model-fallback logic.
|
||||||
|
Err(e) if !emitted => {
|
||||||
|
debug!(model = %options.model, error = %e, "anthropic: streaming failed before any delta; retrying buffered");
|
||||||
|
self.chat_with_tools_raw(messages, tools, options).await
|
||||||
|
}
|
||||||
|
Err(e) => Err(e),
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
mod tests {
|
mod tests {
|
||||||
use super::*;
|
use super::*;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn reasoning_of_joins_thinking_blocks() {
|
||||||
|
let blocks = vec![
|
||||||
|
json!({"type": "thinking", "thinking": "first"}),
|
||||||
|
json!({"type": "text", "text": "answer"}),
|
||||||
|
json!({"type": "thinking", "thinking": "second"}),
|
||||||
|
];
|
||||||
|
assert_eq!(
|
||||||
|
AnthropicClient::reasoning_of(&blocks),
|
||||||
|
Some("first\nsecond".to_string())
|
||||||
|
);
|
||||||
|
assert_eq!(AnthropicClient::reasoning_of(&[]), None);
|
||||||
|
assert_eq!(
|
||||||
|
AnthropicClient::reasoning_of(&[json!({"type": "text", "text": "a"})]),
|
||||||
|
None
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn user_content_string_passthrough() {
|
fn user_content_string_passthrough() {
|
||||||
let v = convert_user_content(&json!("hello"));
|
let v = convert_user_content(&json!("hello"));
|
||||||
|
|||||||
@@ -6,11 +6,54 @@ pub mod openai;
|
|||||||
// Re-export the trait and all associated types from core-api so existing
|
// Re-export the trait and all associated types from core-api so existing
|
||||||
// callers that import from `llm_client` continue to work unchanged.
|
// callers that import from `llm_client` continue to work unchanged.
|
||||||
pub use core_api::chatbot::{
|
pub use core_api::chatbot::{
|
||||||
ChatOptions, ChatResponse, ChatbotClient, LlmRawMeta, LlmTurn, Message, Role, ToolCall,
|
ChatOptions, ChatResponse, ChatbotClient, LlmRawMeta, LlmTurn, Message, Role, StreamDelta,
|
||||||
|
ToolCall,
|
||||||
};
|
};
|
||||||
|
|
||||||
use serde_json::Value;
|
use serde_json::Value;
|
||||||
|
|
||||||
|
/// Incremental SSE decoder: feed raw response bytes, get back the payload of
|
||||||
|
/// every complete `data:` line seen (`[DONE]` included — callers decide).
|
||||||
|
/// Buffers partial lines across chunks; `event:` lines and comments are
|
||||||
|
/// skipped (both OpenAI and Anthropic put the event type inside the JSON).
|
||||||
|
#[derive(Default)]
|
||||||
|
pub struct SseDecoder {
|
||||||
|
buf: Vec<u8>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl SseDecoder {
|
||||||
|
pub fn new() -> Self {
|
||||||
|
Self::default()
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn feed(&mut self, bytes: &[u8]) -> Vec<String> {
|
||||||
|
self.buf.extend_from_slice(bytes);
|
||||||
|
let mut out = Vec::new();
|
||||||
|
while let Some(pos) = self.buf.iter().position(|&b| b == b'\n') {
|
||||||
|
let line: Vec<u8> = self.buf.drain(..=pos).collect();
|
||||||
|
if let Some(payload) = parse_sse_line(&line) {
|
||||||
|
out.push(payload);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
out
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Flush a trailing line not terminated by `\n` at end-of-stream.
|
||||||
|
pub fn finish(&mut self) -> Vec<String> {
|
||||||
|
let rest = std::mem::take(&mut self.buf);
|
||||||
|
parse_sse_line(&rest).into_iter().collect()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A complete SSE line is valid UTF-8 (a multibyte sequence never contains a
|
||||||
|
/// `\n` byte), but decode lossily anyway — a corrupt line is skipped, not fatal.
|
||||||
|
fn parse_sse_line(line: &[u8]) -> Option<String> {
|
||||||
|
let line = String::from_utf8_lossy(line);
|
||||||
|
let line = line.trim_end_matches('\r').trim();
|
||||||
|
let data = line.strip_prefix("data:")?.trim_start();
|
||||||
|
if data.is_empty() { None } else { Some(data.to_string()) }
|
||||||
|
}
|
||||||
|
|
||||||
/// Converts a reqwest `HeaderMap` into a `serde_json::Value` object.
|
/// Converts a reqwest `HeaderMap` into a `serde_json::Value` object.
|
||||||
pub fn headers_to_json(headers: &reqwest::header::HeaderMap) -> Value {
|
pub fn headers_to_json(headers: &reqwest::header::HeaderMap) -> Value {
|
||||||
let map: serde_json::Map<String, Value> = headers
|
let map: serde_json::Map<String, Value> = headers
|
||||||
@@ -72,3 +115,41 @@ pub fn http_status(err: &anyhow::Error) -> Option<u16> {
|
|||||||
}
|
}
|
||||||
None
|
None
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::SseDecoder;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn sse_decoder_buffers_partial_lines_across_chunks() {
|
||||||
|
let mut dec = SseDecoder::new();
|
||||||
|
// A payload split mid-JSON across two chunks yields one complete line.
|
||||||
|
assert!(dec.feed(br#"data: {"a": 1"#).is_empty());
|
||||||
|
assert_eq!(dec.feed(b"}\r\n").len(), 1);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn sse_decoder_skips_events_comments_and_keeps_done() {
|
||||||
|
let mut dec = SseDecoder::new();
|
||||||
|
let out = dec.feed(b"event: message_start\n: ping\n\ndata: {\"type\":\"ping\"}\ndata: [DONE]\n");
|
||||||
|
assert_eq!(out, vec!["{\"type\":\"ping\"}".to_string(), "[DONE]".to_string()]);
|
||||||
|
assert!(dec.finish().is_empty());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn sse_decoder_finish_flushes_unterminated_tail() {
|
||||||
|
let mut dec = SseDecoder::new();
|
||||||
|
assert!(dec.feed(b"data: tail-without-newline").is_empty());
|
||||||
|
assert_eq!(dec.finish(), vec!["tail-without-newline".to_string()]);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn sse_decoder_handles_multibyte_split() {
|
||||||
|
let mut dec = SseDecoder::new();
|
||||||
|
// "€" is 3 bytes in UTF-8; split across the chunk boundary.
|
||||||
|
let payload = "data: {\"t\":\"€\"}\n".as_bytes();
|
||||||
|
let (a, b) = payload.split_at(12);
|
||||||
|
assert!(dec.feed(a).is_empty());
|
||||||
|
assert_eq!(dec.feed(b), vec!["{\"t\":\"€\"}".to_string()]);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
+229
-41
@@ -1,8 +1,12 @@
|
|||||||
|
use std::collections::BTreeMap;
|
||||||
|
|
||||||
use async_trait::async_trait;
|
use async_trait::async_trait;
|
||||||
|
use futures_util::StreamExt;
|
||||||
use serde_json::{Value, json};
|
use serde_json::{Value, json};
|
||||||
|
use tokio::sync::mpsc;
|
||||||
use tracing::{debug, info, trace, warn};
|
use tracing::{debug, info, trace, warn};
|
||||||
|
|
||||||
use crate::{ChatOptions, ChatResponse, ChatbotClient, LlmRawMeta, LlmTurn, Message, Role, ToolCall, headers_to_json, redact_key};
|
use crate::{ChatOptions, ChatResponse, ChatbotClient, LlmRawMeta, LlmTurn, Message, Role, SseDecoder, StreamDelta, ToolCall, headers_to_json, redact_key};
|
||||||
use core_api::APP_NAME;
|
use core_api::APP_NAME;
|
||||||
|
|
||||||
/// OpenAI ChatGPT client (also compatible with any OpenAI-spec endpoint).
|
/// OpenAI ChatGPT client (also compatible with any OpenAI-spec endpoint).
|
||||||
@@ -45,6 +49,205 @@ impl OpenAiClient {
|
|||||||
fn url(&self) -> String {
|
fn url(&self) -> String {
|
||||||
format!("{}/chat/completions", self.base_url.trim_end_matches('/'))
|
format!("{}/chat/completions", self.base_url.trim_end_matches('/'))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Shared request body for the buffered and the streaming path. Caller adds
|
||||||
|
/// `max_tokens`/`temperature`/`extra_params` afterwards via `finalize_body`.
|
||||||
|
fn base_body(&self, model: &str, messages: &[Value], tools: &[Value]) -> Value {
|
||||||
|
let mut body = json!({
|
||||||
|
"model": model,
|
||||||
|
"messages": messages,
|
||||||
|
});
|
||||||
|
|
||||||
|
if !tools.is_empty() {
|
||||||
|
// When prompt caching is enabled, tag the last tool with cache_control
|
||||||
|
// so the entire tools array is included in the Anthropic KV cache prefix.
|
||||||
|
let tools_value: Value = if self.enable_prompt_cache {
|
||||||
|
let mut tagged = tools.to_vec();
|
||||||
|
if let Some(last) = tagged.last_mut() {
|
||||||
|
last["cache_control"] = json!({"type": "ephemeral"});
|
||||||
|
}
|
||||||
|
tagged.into()
|
||||||
|
} else {
|
||||||
|
tools.into()
|
||||||
|
};
|
||||||
|
body["tools"] = tools_value;
|
||||||
|
body["tool_choice"] = "auto".into();
|
||||||
|
}
|
||||||
|
body
|
||||||
|
}
|
||||||
|
|
||||||
|
fn finalize_body(&self, mut body: Value, options: &ChatOptions) -> Value {
|
||||||
|
if let Some(t) = options.max_tokens { body["max_tokens"] = t.into(); }
|
||||||
|
if let Some(t) = options.temperature { body["temperature"] = t.into(); }
|
||||||
|
self.apply_extra(&mut body);
|
||||||
|
body
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Request metadata for logging (shared by buffered and streaming paths).
|
||||||
|
fn logged_headers(&self) -> Value {
|
||||||
|
let mut logged_headers = json!({
|
||||||
|
"authorization": format!("Bearer {}", redact_key(&self.api_key)),
|
||||||
|
"content-type": "application/json",
|
||||||
|
});
|
||||||
|
if self.enable_prompt_cache {
|
||||||
|
logged_headers["anthropic-beta"] = "prompt-caching-2024-07-31".into();
|
||||||
|
}
|
||||||
|
logged_headers
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn send_request(&self, body: &Value) -> reqwest::Result<reqwest::Response> {
|
||||||
|
let mut req = self.http.post(self.url()).bearer_auth(&self.api_key).header("X-Title", APP_NAME);
|
||||||
|
if self.enable_prompt_cache {
|
||||||
|
req = req.header("anthropic-beta", "prompt-caching-2024-07-31");
|
||||||
|
}
|
||||||
|
req.json(body).send().await
|
||||||
|
}
|
||||||
|
|
||||||
|
/// SSE streaming path behind `chat_with_tools_raw_streaming`. Accumulates
|
||||||
|
/// content/reasoning/tool-call fragments into the same `LlmTurn` the
|
||||||
|
/// buffered path would return, while forwarding text/reasoning deltas to
|
||||||
|
/// `delta_tx` (try_send, best-effort). `emitted` tracks whether any delta
|
||||||
|
/// was pushed, so the caller can distinguish a pre-stream failure (safe to
|
||||||
|
/// retry buffered) from a mid-stream one (partial output already shown).
|
||||||
|
async fn stream_chat(
|
||||||
|
&self,
|
||||||
|
messages: &[Value],
|
||||||
|
tools: &[Value],
|
||||||
|
options: &ChatOptions,
|
||||||
|
delta_tx: &mpsc::Sender<StreamDelta>,
|
||||||
|
emitted: &mut bool,
|
||||||
|
) -> anyhow::Result<(LlmTurn, Option<LlmRawMeta>)> {
|
||||||
|
let mut body = self.base_body(&options.model, messages, tools);
|
||||||
|
body["stream"] = json!(true);
|
||||||
|
body["stream_options"] = json!({ "include_usage": true });
|
||||||
|
let body = self.finalize_body(body, options);
|
||||||
|
|
||||||
|
debug!(model = %options.model, tools = tools.len(), prompt_cache = self.enable_prompt_cache, "openai: sending streaming chat_with_tools request");
|
||||||
|
trace!(body = %body, "openai: streaming chat_with_tools request body");
|
||||||
|
|
||||||
|
let request_body = body.clone();
|
||||||
|
let request_headers = self.logged_headers();
|
||||||
|
|
||||||
|
let http_resp = self.send_request(&body).await?;
|
||||||
|
|
||||||
|
let response_headers = headers_to_json(http_resp.headers());
|
||||||
|
let status = http_resp.status();
|
||||||
|
if !status.is_success() {
|
||||||
|
let resp_text = http_resp.text().await?;
|
||||||
|
return Err(crate::LlmError {
|
||||||
|
status: Some(status.as_u16()),
|
||||||
|
message: format!(
|
||||||
|
"openai: HTTP {status} from {url}\nbody: {resp_text}",
|
||||||
|
url = self.url(),
|
||||||
|
),
|
||||||
|
}.into());
|
||||||
|
}
|
||||||
|
|
||||||
|
let mut content = String::new();
|
||||||
|
let mut reasoning = String::new();
|
||||||
|
// index → (id, name, arguments fragment buffer)
|
||||||
|
let mut tool_calls: BTreeMap<u64, (String, String, String)> = BTreeMap::new();
|
||||||
|
let mut finish_reason: Option<String> = None;
|
||||||
|
let mut usage: Option<Value> = None;
|
||||||
|
let mut sse = SseDecoder::new();
|
||||||
|
let mut byte_stream = http_resp.bytes_stream();
|
||||||
|
|
||||||
|
// One SSE `data:` payload. Fragments update the accumulators; text and
|
||||||
|
// reasoning also go out as deltas. Unparseable chunks are skipped —
|
||||||
|
// the assembled turn stays consistent.
|
||||||
|
let mut handle_payload = |payload: &str, emitted: &mut bool| {
|
||||||
|
if payload == "[DONE]" {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
let Ok(v) = serde_json::from_str::<Value>(payload) else { return };
|
||||||
|
if let Some(u) = v.get("usage").filter(|u| !u.is_null()) {
|
||||||
|
usage = Some(u.clone());
|
||||||
|
}
|
||||||
|
let Some(choice) = v["choices"].as_array().and_then(|a| a.first()) else { return };
|
||||||
|
if let Some(fr) = choice["finish_reason"].as_str() {
|
||||||
|
finish_reason = Some(fr.to_string());
|
||||||
|
}
|
||||||
|
let delta = &choice["delta"];
|
||||||
|
if let Some(t) = delta["content"].as_str().filter(|t| !t.is_empty()) {
|
||||||
|
content.push_str(t);
|
||||||
|
*emitted = true;
|
||||||
|
let _ = delta_tx.try_send(StreamDelta::Text(t.to_string()));
|
||||||
|
}
|
||||||
|
// Same normalization as the buffered path: DeepSeek uses
|
||||||
|
// `reasoning_content`, MiniMax M3 and others `reasoning`.
|
||||||
|
if let Some(t) = delta["reasoning_content"].as_str()
|
||||||
|
.or_else(|| delta["reasoning"].as_str())
|
||||||
|
.filter(|t| !t.is_empty())
|
||||||
|
{
|
||||||
|
reasoning.push_str(t);
|
||||||
|
*emitted = true;
|
||||||
|
let _ = delta_tx.try_send(StreamDelta::Reasoning(t.to_string()));
|
||||||
|
}
|
||||||
|
if let Some(tc_arr) = delta["tool_calls"].as_array() {
|
||||||
|
for tc in tc_arr {
|
||||||
|
let idx = tc["index"].as_u64().unwrap_or(0);
|
||||||
|
let entry = tool_calls.entry(idx).or_default();
|
||||||
|
if let Some(id) = tc["id"].as_str() { entry.0 = id.to_string(); }
|
||||||
|
if let Some(n) = tc["function"]["name"].as_str() { entry.1 = n.to_string(); }
|
||||||
|
if let Some(a) = tc["function"]["arguments"].as_str() { entry.2.push_str(a); }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
while let Some(chunk) = byte_stream.next().await {
|
||||||
|
let chunk = chunk?;
|
||||||
|
for payload in sse.feed(&chunk) {
|
||||||
|
handle_payload(&payload, emitted);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for payload in sse.finish() {
|
||||||
|
handle_payload(&payload, emitted);
|
||||||
|
}
|
||||||
|
|
||||||
|
let finish = finish_reason.as_deref().unwrap_or("stop");
|
||||||
|
let input_tokens = usage.as_ref().and_then(|u| u["prompt_tokens"].as_u64()).map(|n| n as u32);
|
||||||
|
let output_tokens = usage.as_ref().and_then(|u| u["completion_tokens"].as_u64()).map(|n| n as u32);
|
||||||
|
let cache_read_tokens = usage.as_ref()
|
||||||
|
.and_then(|u| u["prompt_tokens_details"]["cached_tokens"].as_u64())
|
||||||
|
.map(|n| n as u32);
|
||||||
|
let cost = usage.as_ref().and_then(|u| u["cost"].as_f64());
|
||||||
|
let reasoning_content = if reasoning.is_empty() { None } else { Some(reasoning) };
|
||||||
|
info!(model = %options.model, ?input_tokens, ?output_tokens, finish_reason = finish, "openai: streaming response completed");
|
||||||
|
if finish == "length" {
|
||||||
|
warn!(model = %options.model, ?output_tokens, "openai: response truncated (max_tokens reached)");
|
||||||
|
}
|
||||||
|
|
||||||
|
let turn = if !tool_calls.is_empty() {
|
||||||
|
let calls = tool_calls
|
||||||
|
.into_values()
|
||||||
|
.map(|(id, name, args)| ToolCall {
|
||||||
|
id,
|
||||||
|
name,
|
||||||
|
arguments: serde_json::from_str(&args).unwrap_or(Value::Object(Default::default())),
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
LlmTurn::ToolCalls { content, calls, input_tokens, output_tokens, reasoning_content, cache_read_tokens, cache_creation_tokens: None, cost }
|
||||||
|
} else {
|
||||||
|
let truncated = finish == "length";
|
||||||
|
LlmTurn::Message(ChatResponse { content, input_tokens, output_tokens, truncated, reasoning_content, cache_read_tokens, cache_creation_tokens: None, cost })
|
||||||
|
};
|
||||||
|
|
||||||
|
// Synthesize a buffered-shaped response body for the payload log, so a
|
||||||
|
// streamed call leaves the same debugging trail as a buffered one.
|
||||||
|
let response_body = json!({
|
||||||
|
"streamed": true,
|
||||||
|
"choices": [{ "finish_reason": finish }],
|
||||||
|
"usage": usage,
|
||||||
|
});
|
||||||
|
let raw_meta = LlmRawMeta {
|
||||||
|
request_headers: Some(request_headers),
|
||||||
|
request_body: Some(request_body),
|
||||||
|
response_headers: Some(response_headers),
|
||||||
|
response_body: Some(response_body),
|
||||||
|
};
|
||||||
|
|
||||||
|
Ok((turn, Some(raw_meta)))
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
#[async_trait]
|
#[async_trait]
|
||||||
@@ -123,53 +326,16 @@ impl ChatbotClient for OpenAiClient {
|
|||||||
tools: &[Value],
|
tools: &[Value],
|
||||||
options: &ChatOptions,
|
options: &ChatOptions,
|
||||||
) -> anyhow::Result<(LlmTurn, Option<LlmRawMeta>)> {
|
) -> anyhow::Result<(LlmTurn, Option<LlmRawMeta>)> {
|
||||||
let mut body = json!({
|
let body = self.finalize_body(self.base_body(&options.model, messages, tools), options);
|
||||||
"model": options.model,
|
|
||||||
"messages": messages,
|
|
||||||
});
|
|
||||||
|
|
||||||
if !tools.is_empty() {
|
|
||||||
// When prompt caching is enabled, tag the last tool with cache_control
|
|
||||||
// so the entire tools array is included in the Anthropic KV cache prefix.
|
|
||||||
let tools_value: Value = if self.enable_prompt_cache {
|
|
||||||
let mut tagged = tools.to_vec();
|
|
||||||
if let Some(last) = tagged.last_mut() {
|
|
||||||
last["cache_control"] = json!({"type": "ephemeral"});
|
|
||||||
}
|
|
||||||
tagged.into()
|
|
||||||
} else {
|
|
||||||
tools.into()
|
|
||||||
};
|
|
||||||
body["tools"] = tools_value;
|
|
||||||
body["tool_choice"] = "auto".into();
|
|
||||||
}
|
|
||||||
|
|
||||||
if let Some(t) = options.max_tokens { body["max_tokens"] = t.into(); }
|
|
||||||
if let Some(t) = options.temperature { body["temperature"] = t.into(); }
|
|
||||||
self.apply_extra(&mut body);
|
|
||||||
|
|
||||||
debug!(model = %options.model, tools = tools.len(), prompt_cache = self.enable_prompt_cache, "openai: sending chat_with_tools request");
|
debug!(model = %options.model, tools = tools.len(), prompt_cache = self.enable_prompt_cache, "openai: sending chat_with_tools request");
|
||||||
trace!(body = %body, "openai: chat_with_tools request body");
|
trace!(body = %body, "openai: chat_with_tools request body");
|
||||||
|
|
||||||
// Capture request metadata for logging.
|
// Capture request metadata for logging.
|
||||||
let mut logged_headers = json!({
|
|
||||||
"authorization": format!("Bearer {}", redact_key(&self.api_key)),
|
|
||||||
"content-type": "application/json",
|
|
||||||
});
|
|
||||||
if self.enable_prompt_cache {
|
|
||||||
logged_headers["anthropic-beta"] = "prompt-caching-2024-07-31".into();
|
|
||||||
}
|
|
||||||
let request_body = body.clone();
|
let request_body = body.clone();
|
||||||
let request_headers = logged_headers;
|
let request_headers = self.logged_headers();
|
||||||
|
|
||||||
let mut req = self.http.post(self.url()).bearer_auth(&self.api_key).header("X-Title", APP_NAME);
|
let http_resp = self.send_request(&body).await?;
|
||||||
if self.enable_prompt_cache {
|
|
||||||
req = req.header("anthropic-beta", "prompt-caching-2024-07-31");
|
|
||||||
}
|
|
||||||
let http_resp = req
|
|
||||||
.json(&body)
|
|
||||||
.send()
|
|
||||||
.await?;
|
|
||||||
|
|
||||||
let response_headers = headers_to_json(http_resp.headers());
|
let response_headers = headers_to_json(http_resp.headers());
|
||||||
let status = http_resp.status();
|
let status = http_resp.status();
|
||||||
@@ -262,4 +428,26 @@ impl ChatbotClient for OpenAiClient {
|
|||||||
|
|
||||||
Ok((turn, Some(raw_meta)))
|
Ok((turn, Some(raw_meta)))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
async fn chat_with_tools_raw_streaming(
|
||||||
|
&self,
|
||||||
|
messages: &[Value],
|
||||||
|
tools: &[Value],
|
||||||
|
options: &ChatOptions,
|
||||||
|
delta_tx: mpsc::Sender<StreamDelta>,
|
||||||
|
) -> anyhow::Result<(LlmTurn, Option<LlmRawMeta>)> {
|
||||||
|
let mut emitted = false;
|
||||||
|
match self.stream_chat(messages, tools, options, &delta_tx, &mut emitted).await {
|
||||||
|
Ok(ok) => Ok(ok),
|
||||||
|
// Nothing was ever streamed: some OpenAI-compatible providers reject
|
||||||
|
// `stream`/`stream_options` outright — retry buffered so they keep
|
||||||
|
// working exactly as before. A mid-stream failure (deltas already
|
||||||
|
// shown) instead propagates to the model-fallback logic.
|
||||||
|
Err(e) if !emitted => {
|
||||||
|
debug!(model = %options.model, error = %e, "openai: streaming failed before any delta; retrying buffered");
|
||||||
|
self.chat_with_tools_raw(messages, tools, options).await
|
||||||
|
}
|
||||||
|
Err(e) => Err(e),
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -2,6 +2,6 @@ pub mod logging;
|
|||||||
|
|
||||||
// Re-export from the independent llm-client crate.
|
// Re-export from the independent llm-client crate.
|
||||||
pub use llm_client::{
|
pub use llm_client::{
|
||||||
ChatOptions, ChatResponse, ChatbotClient, LlmError, LlmRawMeta, LlmTurn, Message, ToolCall,
|
ChatOptions, ChatResponse, ChatbotClient, LlmError, LlmRawMeta, LlmTurn, Message, StreamDelta,
|
||||||
anthropic, http_status, lm_studio, ollama, openai,
|
ToolCall, anthropic, http_status, lm_studio, ollama, openai,
|
||||||
};
|
};
|
||||||
|
|||||||
@@ -1,3 +1,3 @@
|
|||||||
pub use core_api::events::{
|
pub use core_api::events::{
|
||||||
ClientMessage, GlobalEvent, InboundDataMessage, ServerEvent,
|
ClientMessage, GlobalEvent, InboundDataMessage, ServerEvent, TokenDeltaKind,
|
||||||
};
|
};
|
||||||
|
|||||||
@@ -42,13 +42,19 @@ impl<'a> TurnEmitter<'a> {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/// The assistant produced text alongside tool calls (reasoning before acting).
|
/// The assistant produced text alongside tool calls (reasoning before acting).
|
||||||
pub(super) async fn thinking(&self, message_id: i64, content: String, input_tokens: Option<u32>, output_tokens: Option<u32>) {
|
pub(super) async fn thinking(&self, message_id: i64, content: String, input_tokens: Option<u32>, output_tokens: Option<u32>, reasoning_content: Option<String>) {
|
||||||
self.emit(ServerEvent::Thinking { message_id, content, input_tokens, output_tokens }).await;
|
self.emit(ServerEvent::Thinking { message_id, content, input_tokens, output_tokens, reasoning_content }).await;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Clone of the underlying sender, for spawning side-channel tasks that
|
||||||
|
/// emit alongside the turn (e.g. the token-delta forwarder).
|
||||||
|
pub(super) fn sender(&self) -> mpsc::Sender<ServerEvent> {
|
||||||
|
self.tx.clone()
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The assistant response is complete.
|
/// The assistant response is complete.
|
||||||
pub(super) async fn done(&self, message_id: i64, stack_id: i64, content: String, input_tokens: Option<u32>, output_tokens: Option<u32>) {
|
pub(super) async fn done(&self, message_id: i64, stack_id: i64, content: String, input_tokens: Option<u32>, output_tokens: Option<u32>, reasoning_content: Option<String>) {
|
||||||
self.emit(ServerEvent::Done { message_id, stack_id, content, input_tokens, output_tokens }).await;
|
self.emit(ServerEvent::Done { message_id, stack_id, content, input_tokens, output_tokens, reasoning_content }).await;
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The LLM was cut off by the token limit.
|
/// The LLM was cut off by the token limit.
|
||||||
|
|||||||
@@ -9,11 +9,13 @@ use std::collections::HashSet;
|
|||||||
use std::sync::Arc;
|
use std::sync::Arc;
|
||||||
|
|
||||||
use serde_json::Value;
|
use serde_json::Value;
|
||||||
|
use tokio::sync::mpsc;
|
||||||
use tokio_util::sync::CancellationToken;
|
use tokio_util::sync::CancellationToken;
|
||||||
use tracing::{error, warn};
|
use tracing::{error, warn};
|
||||||
|
|
||||||
use crate::chatbot::{ChatOptions, LlmTurn};
|
use crate::chatbot::{ChatOptions, LlmTurn, StreamDelta};
|
||||||
use crate::db::llm_request_payloads;
|
use crate::db::llm_request_payloads;
|
||||||
|
use crate::events::{ServerEvent, TokenDeltaKind};
|
||||||
use crate::llm::{LlmEntry, LlmStrength};
|
use crate::llm::{LlmEntry, LlmStrength};
|
||||||
|
|
||||||
use super::ChatSessionHandler;
|
use super::ChatSessionHandler;
|
||||||
@@ -77,10 +79,19 @@ impl ChatSessionHandler {
|
|||||||
// the fallback reassignment below. On cancel we drop the future
|
// the fallback reassignment below. On cancel we drop the future
|
||||||
// (aborting the request) and return immediately.
|
// (aborting the request) and return immediately.
|
||||||
let client = cur_llm.client.clone();
|
let client = cur_llm.client.clone();
|
||||||
|
// Streaming side-channel: providers that support SSE push deltas here;
|
||||||
|
// the forwarder re-emits them as `TokenDelta` events on the turn bus.
|
||||||
|
// Best-effort — the round's final events remain authoritative.
|
||||||
|
let (delta_tx, delta_rx) = mpsc::channel::<StreamDelta>(256);
|
||||||
|
let forwarder = spawn_delta_forwarder(delta_rx, em.sender());
|
||||||
let call_result = tokio::select! {
|
let call_result = tokio::select! {
|
||||||
_ = token.cancelled() => return RoundLlm::Cancelled,
|
_ = token.cancelled() => return RoundLlm::Cancelled,
|
||||||
r = client.chat_with_tools_raw(messages.as_slice(), defs, &options) => r,
|
r = client.chat_with_tools_raw_streaming(messages.as_slice(), defs, &options, delta_tx) => r,
|
||||||
};
|
};
|
||||||
|
// The client's sender dropped with the completed future: the forwarder
|
||||||
|
// drains any queued deltas and exits, so every `TokenDelta` precedes the
|
||||||
|
// round's outcome events (Thinking / Done) in bus order.
|
||||||
|
forwarder.await.ok();
|
||||||
|
|
||||||
let e = match call_result {
|
let e = match call_result {
|
||||||
Ok((turn, meta)) => {
|
Ok((turn, meta)) => {
|
||||||
@@ -151,6 +162,26 @@ impl ChatSessionHandler {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Forwards streaming deltas from the LLM client onto the turn's event channel
|
||||||
|
/// as `TokenDelta` events. Exits when the client drops its sender (call
|
||||||
|
/// completed or aborted) or when the turn receiver is gone.
|
||||||
|
fn spawn_delta_forwarder(
|
||||||
|
mut rx: mpsc::Receiver<StreamDelta>,
|
||||||
|
tx: mpsc::Sender<ServerEvent>,
|
||||||
|
) -> tokio::task::JoinHandle<()> {
|
||||||
|
tokio::spawn(async move {
|
||||||
|
while let Some(d) = rx.recv().await {
|
||||||
|
let (kind, delta) = match d {
|
||||||
|
StreamDelta::Text(t) => (TokenDeltaKind::Content, t),
|
||||||
|
StreamDelta::Reasoning(t) => (TokenDeltaKind::Reasoning, t),
|
||||||
|
};
|
||||||
|
if tx.send(ServerEvent::TokenDelta { kind, delta }).await.is_err() {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
/// Whether an LLM error is worth retrying on a different model.
|
/// Whether an LLM error is worth retrying on a different model.
|
||||||
///
|
///
|
||||||
/// Classifies on the real HTTP status ([`crate::chatbot::http_status`]), not a
|
/// Classifies on the real HTTP status ([`crate::chatbot::http_status`]), not a
|
||||||
|
|||||||
@@ -152,6 +152,7 @@ impl ChatSessionHandler {
|
|||||||
input_tokens: resp.input_tokens,
|
input_tokens: resp.input_tokens,
|
||||||
output_tokens: resp.output_tokens,
|
output_tokens: resp.output_tokens,
|
||||||
truncated: resp.truncated,
|
truncated: resp.truncated,
|
||||||
|
reasoning_content: resp.reasoning_content,
|
||||||
tool_calls: all_tool_calls,
|
tool_calls: all_tool_calls,
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
@@ -165,7 +166,7 @@ impl ChatSessionHandler {
|
|||||||
chat_history::set_usage(pool, message_id, i, o, 0, cost).await?;
|
chat_history::set_usage(pool, message_id, i, o, 0, cost).await?;
|
||||||
}
|
}
|
||||||
if !assistant_text.trim().is_empty() || input_tokens.is_some() {
|
if !assistant_text.trim().is_empty() || input_tokens.is_some() {
|
||||||
em.thinking(message_id, assistant_text, input_tokens, output_tokens).await;
|
em.thinking(message_id, assistant_text, input_tokens, output_tokens, reasoning_content).await;
|
||||||
}
|
}
|
||||||
|
|
||||||
// A homogeneous batch of ≥2 synchronous sub-agent calls is fanned
|
// A homogeneous batch of ≥2 synchronous sub-agent calls is fanned
|
||||||
|
|||||||
@@ -103,6 +103,8 @@ pub(super) enum TurnOutcome {
|
|||||||
input_tokens: Option<u32>,
|
input_tokens: Option<u32>,
|
||||||
output_tokens: Option<u32>,
|
output_tokens: Option<u32>,
|
||||||
truncated: bool,
|
truncated: bool,
|
||||||
|
/// Chain-of-thought produced by the final round, when any.
|
||||||
|
reasoning_content: Option<String>,
|
||||||
/// All tool calls executed during this turn, across all rounds.
|
/// All tool calls executed during this turn, across all rounds.
|
||||||
tool_calls: Vec<crate::chat_event_bus::ToolCallEvent>,
|
tool_calls: Vec<crate::chat_event_bus::ToolCallEvent>,
|
||||||
},
|
},
|
||||||
@@ -643,7 +645,7 @@ impl ChatSessionHandler {
|
|||||||
let outcome = self.run_agent_turn(stack.id, &config, &token, &tx, pending_input.as_ref()).await?;
|
let outcome = self.run_agent_turn(stack.id, &config, &token, &tx, pending_input.as_ref()).await?;
|
||||||
|
|
||||||
match outcome {
|
match outcome {
|
||||||
TurnOutcome::Final { content, message_id, input_tokens, output_tokens, truncated, tool_calls } => {
|
TurnOutcome::Final { content, message_id, input_tokens, output_tokens, truncated, reasoning_content, tool_calls } => {
|
||||||
// Persist token count so the *next* handle_message call knows
|
// Persist token count so the *next* handle_message call knows
|
||||||
// whether to compact before running the LLM loop.
|
// whether to compact before running the LLM loop.
|
||||||
if let Some(t) = input_tokens {
|
if let Some(t) = input_tokens {
|
||||||
@@ -654,7 +656,7 @@ impl ChatSessionHandler {
|
|||||||
warn!(session_id = self.session_id, ?output_tokens, "response truncated (max_tokens)");
|
warn!(session_id = self.session_id, ?output_tokens, "response truncated (max_tokens)");
|
||||||
em.truncated(output_tokens).await;
|
em.truncated(output_tokens).await;
|
||||||
}
|
}
|
||||||
em.done(message_id, stack.id, content.clone(), input_tokens, output_tokens).await;
|
em.done(message_id, stack.id, content.clone(), input_tokens, output_tokens, reasoning_content).await;
|
||||||
|
|
||||||
// Publish both messages to the event bus now that both are in the DB.
|
// Publish both messages to the event bus now that both are in the DB.
|
||||||
let now = chrono::Utc::now();
|
let now = chrono::Utc::now();
|
||||||
|
|||||||
@@ -128,6 +128,7 @@ impl ChatSessionHandler {
|
|||||||
input_tokens: None,
|
input_tokens: None,
|
||||||
output_tokens: None,
|
output_tokens: None,
|
||||||
truncated: false,
|
truncated: false,
|
||||||
|
reasoning_content: msg.reasoning_content,
|
||||||
tool_calls: Vec::new(),
|
tool_calls: Vec::new(),
|
||||||
};
|
};
|
||||||
break 'seed (outcome, stack);
|
break 'seed (outcome, stack);
|
||||||
@@ -208,13 +209,13 @@ impl ChatSessionHandler {
|
|||||||
|
|
||||||
// current_stack is now the root (depth=0); emit the final event.
|
// current_stack is now the root (depth=0); emit the final event.
|
||||||
match current_outcome {
|
match current_outcome {
|
||||||
TurnOutcome::Final { content, message_id, input_tokens, output_tokens, truncated, .. } => {
|
TurnOutcome::Final { content, message_id, input_tokens, output_tokens, truncated, reasoning_content, .. } => {
|
||||||
info!(session_id = self.session_id, "resume_turn done");
|
info!(session_id = self.session_id, "resume_turn done");
|
||||||
if truncated {
|
if truncated {
|
||||||
warn!(session_id = self.session_id, "response truncated");
|
warn!(session_id = self.session_id, "response truncated");
|
||||||
em.truncated(output_tokens).await;
|
em.truncated(output_tokens).await;
|
||||||
}
|
}
|
||||||
em.done(message_id, current_stack.id, content, input_tokens, output_tokens).await;
|
em.done(message_id, current_stack.id, content, input_tokens, output_tokens, reasoning_content).await;
|
||||||
}
|
}
|
||||||
TurnOutcome::Cancelled => {
|
TurnOutcome::Cancelled => {
|
||||||
info!(session_id = self.session_id, "resume_turn cancelled");
|
info!(session_id = self.session_id, "resume_turn cancelled");
|
||||||
|
|||||||
@@ -633,6 +633,7 @@ fn build_items<'a>(
|
|||||||
"failed": failed,
|
"failed": failed,
|
||||||
"input_tokens": msg.input_tokens,
|
"input_tokens": msg.input_tokens,
|
||||||
"output_tokens": msg.output_tokens,
|
"output_tokens": msg.output_tokens,
|
||||||
|
"reasoning": msg.reasoning_content,
|
||||||
}));
|
}));
|
||||||
} else {
|
} else {
|
||||||
if !msg.content.trim().is_empty() {
|
if !msg.content.trim().is_empty() {
|
||||||
@@ -643,6 +644,7 @@ fn build_items<'a>(
|
|||||||
"failed": failed,
|
"failed": failed,
|
||||||
"input_tokens": msg.input_tokens,
|
"input_tokens": msg.input_tokens,
|
||||||
"output_tokens": msg.output_tokens,
|
"output_tokens": msg.output_tokens,
|
||||||
|
"reasoning": msg.reasoning_content,
|
||||||
}));
|
}));
|
||||||
}
|
}
|
||||||
for tc in &tool_calls {
|
for tc in &tool_calls {
|
||||||
|
|||||||
@@ -194,6 +194,7 @@ async fn handle_socket(mut socket: WebSocket, skald: Arc<Skald>, source: String,
|
|||||||
content: msg,
|
content: msg,
|
||||||
input_tokens: None,
|
input_tokens: None,
|
||||||
output_tokens: None,
|
output_tokens: None,
|
||||||
|
reasoning_content: None,
|
||||||
})).await;
|
})).await;
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
@@ -205,6 +206,7 @@ async fn handle_socket(mut socket: WebSocket, skald: Arc<Skald>, source: String,
|
|||||||
content: dynamic_help(&skald),
|
content: dynamic_help(&skald),
|
||||||
input_tokens: None,
|
input_tokens: None,
|
||||||
output_tokens: None,
|
output_tokens: None,
|
||||||
|
reasoning_content: None,
|
||||||
})).await;
|
})).await;
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
@@ -220,6 +222,7 @@ async fn handle_socket(mut socket: WebSocket, skald: Arc<Skald>, source: String,
|
|||||||
content: format!("↑{input_str} tok · ↓{output_str} tok"),
|
content: format!("↑{input_str} tok · ↓{output_str} tok"),
|
||||||
input_tokens: None,
|
input_tokens: None,
|
||||||
output_tokens: None,
|
output_tokens: None,
|
||||||
|
reasoning_content: None,
|
||||||
})).await;
|
})).await;
|
||||||
}
|
}
|
||||||
Err(e) => {
|
Err(e) => {
|
||||||
@@ -238,6 +241,7 @@ async fn handle_socket(mut socket: WebSocket, skald: Arc<Skald>, source: String,
|
|||||||
content: format!("💰 Costo sessione: ${c:.4}"),
|
content: format!("💰 Costo sessione: ${c:.4}"),
|
||||||
input_tokens: None,
|
input_tokens: None,
|
||||||
output_tokens: None,
|
output_tokens: None,
|
||||||
|
reasoning_content: None,
|
||||||
})).await;
|
})).await;
|
||||||
}
|
}
|
||||||
Ok(None) => {
|
Ok(None) => {
|
||||||
@@ -247,6 +251,7 @@ async fn handle_socket(mut socket: WebSocket, skald: Arc<Skald>, source: String,
|
|||||||
content: "💰 Nessun costo registrato per questa sessione.".to_string(),
|
content: "💰 Nessun costo registrato per questa sessione.".to_string(),
|
||||||
input_tokens: None,
|
input_tokens: None,
|
||||||
output_tokens: None,
|
output_tokens: None,
|
||||||
|
reasoning_content: None,
|
||||||
})).await;
|
})).await;
|
||||||
}
|
}
|
||||||
Err(e) => {
|
Err(e) => {
|
||||||
@@ -265,6 +270,7 @@ async fn handle_socket(mut socket: WebSocket, skald: Arc<Skald>, source: String,
|
|||||||
content: "✅ Contesto compattato.".to_string(),
|
content: "✅ Contesto compattato.".to_string(),
|
||||||
input_tokens: None,
|
input_tokens: None,
|
||||||
output_tokens: None,
|
output_tokens: None,
|
||||||
|
reasoning_content: None,
|
||||||
})).await;
|
})).await;
|
||||||
}
|
}
|
||||||
Ok(false) => {
|
Ok(false) => {
|
||||||
@@ -274,6 +280,7 @@ async fn handle_socket(mut socket: WebSocket, skald: Arc<Skald>, source: String,
|
|||||||
content: "⏩ Compaction skipped (no messages to summarize or compaction disabled).".to_string(),
|
content: "⏩ Compaction skipped (no messages to summarize or compaction disabled).".to_string(),
|
||||||
input_tokens: None,
|
input_tokens: None,
|
||||||
output_tokens: None,
|
output_tokens: None,
|
||||||
|
reasoning_content: None,
|
||||||
})).await;
|
})).await;
|
||||||
}
|
}
|
||||||
Err(e) => {
|
Err(e) => {
|
||||||
@@ -292,6 +299,7 @@ async fn handle_socket(mut socket: WebSocket, skald: Arc<Skald>, source: String,
|
|||||||
content: "✅ Activated tool groups removed from the session.".to_string(),
|
content: "✅ Activated tool groups removed from the session.".to_string(),
|
||||||
input_tokens: None,
|
input_tokens: None,
|
||||||
output_tokens: None,
|
output_tokens: None,
|
||||||
|
reasoning_content: None,
|
||||||
})).await;
|
})).await;
|
||||||
}
|
}
|
||||||
Err(e) => {
|
Err(e) => {
|
||||||
@@ -310,6 +318,7 @@ async fn handle_socket(mut socket: WebSocket, skald: Arc<Skald>, source: String,
|
|||||||
content,
|
content,
|
||||||
input_tokens: None,
|
input_tokens: None,
|
||||||
output_tokens: None,
|
output_tokens: None,
|
||||||
|
reasoning_content: None,
|
||||||
})).await;
|
})).await;
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
@@ -327,6 +336,7 @@ async fn handle_socket(mut socket: WebSocket, skald: Arc<Skald>, source: String,
|
|||||||
content,
|
content,
|
||||||
input_tokens: None,
|
input_tokens: None,
|
||||||
output_tokens: None,
|
output_tokens: None,
|
||||||
|
reasoning_content: None,
|
||||||
})).await;
|
})).await;
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
@@ -358,6 +368,7 @@ async fn handle_socket(mut socket: WebSocket, skald: Arc<Skald>, source: String,
|
|||||||
content: format!("Unknown command: {first}\n\n{}", dynamic_help(&skald)),
|
content: format!("Unknown command: {first}\n\n{}", dynamic_help(&skald)),
|
||||||
input_tokens: None,
|
input_tokens: None,
|
||||||
output_tokens: None,
|
output_tokens: None,
|
||||||
|
reasoning_content: None,
|
||||||
})).await;
|
})).await;
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -507,6 +507,21 @@ export function renderAttachmentChips(host, attachments, { removable = false } =
|
|||||||
</div>`;
|
</div>`;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Collapsible chain-of-thought block: small, muted, collapsed by default so it
|
||||||
|
* never weighs on the UI. A native <details> — Lit keeps the element stable
|
||||||
|
* across re-renders, so a user-expanded block stays open while tokens stream
|
||||||
|
* into it (live) and in past history items alike.
|
||||||
|
*/
|
||||||
|
function renderReasoning(msg) {
|
||||||
|
if (!msg.reasoning) return nothing;
|
||||||
|
return html`
|
||||||
|
<details class="reasoning-block ${msg.streaming ? 'reasoning-block--live' : ''}">
|
||||||
|
<summary>${t('chat.reasoning')}</summary>
|
||||||
|
<div class="reasoning-content">${msg.reasoning}</div>
|
||||||
|
</details>`;
|
||||||
|
}
|
||||||
|
|
||||||
export function renderMsg(host, msg) {
|
export function renderMsg(host, msg) {
|
||||||
try {
|
try {
|
||||||
switch (msg.kind) {
|
switch (msg.kind) {
|
||||||
@@ -516,6 +531,7 @@ export function renderMsg(host, msg) {
|
|||||||
return html`
|
return html`
|
||||||
<div class="copilot-msg assistant copilot-markdown ${msg.failed ? 'copilot-msg--failed' : ''}">
|
<div class="copilot-msg assistant copilot-markdown ${msg.failed ? 'copilot-msg--failed' : ''}">
|
||||||
${msg.failed ? failedBadge() : nothing}
|
${msg.failed ? failedBadge() : nothing}
|
||||||
|
${renderReasoning(msg)}
|
||||||
${unsafeHTML(renderMarkdown(msg.content))}
|
${unsafeHTML(renderMarkdown(msg.content))}
|
||||||
${msg.input_tokens != null ? html`<div class="copilot-token-count">↑${msg.input_tokens.toLocaleString()} tok ↓${msg.output_tokens?.toLocaleString()} tok</div>` : nothing}
|
${msg.input_tokens != null ? html`<div class="copilot-token-count">↑${msg.input_tokens.toLocaleString()} tok ↓${msg.output_tokens?.toLocaleString()} tok</div>` : nothing}
|
||||||
</div>`;
|
</div>`;
|
||||||
@@ -523,8 +539,10 @@ export function renderMsg(host, msg) {
|
|||||||
return html`
|
return html`
|
||||||
<div class="copilot-msg assistant copilot-markdown ${msg.failed ? 'copilot-msg--failed' : ''}">
|
<div class="copilot-msg assistant copilot-markdown ${msg.failed ? 'copilot-msg--failed' : ''}">
|
||||||
${msg.failed ? failedBadge() : nothing}
|
${msg.failed ? failedBadge() : nothing}
|
||||||
|
${renderReasoning(msg)}
|
||||||
${unsafeHTML(renderMarkdown(msg.content))}
|
${unsafeHTML(renderMarkdown(msg.content))}
|
||||||
${msg.input_tokens != null ? html`<div class="copilot-token-count">↑${msg.input_tokens.toLocaleString()} tok ↓${msg.output_tokens?.toLocaleString()} tok</div>` : nothing}
|
${msg.streaming ? html`<span class="stream-caret"></span>` : nothing}
|
||||||
|
${msg.input_tokens != null && !msg.streaming ? html`<div class="copilot-token-count">↑${msg.input_tokens.toLocaleString()} tok ↓${msg.output_tokens?.toLocaleString()} tok</div>` : nothing}
|
||||||
</div>`;
|
</div>`;
|
||||||
case 'error':
|
case 'error':
|
||||||
return html`
|
return html`
|
||||||
|
|||||||
@@ -88,6 +88,75 @@
|
|||||||
letter-spacing: 0.02em;
|
letter-spacing: 0.02em;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* ── Reasoning (chain-of-thought) block ────────────────────────────────────── */
|
||||||
|
/* Small, low-contrast, collapsed by default: visible but never heavy. */
|
||||||
|
|
||||||
|
.reasoning-block {
|
||||||
|
margin: 0 0 0.45rem;
|
||||||
|
font-size: 0.75rem;
|
||||||
|
color: var(--placeholder-color);
|
||||||
|
opacity: 0.75;
|
||||||
|
}
|
||||||
|
|
||||||
|
.reasoning-block > summary {
|
||||||
|
cursor: pointer;
|
||||||
|
user-select: none;
|
||||||
|
font-style: italic;
|
||||||
|
letter-spacing: 0.02em;
|
||||||
|
list-style-position: inside;
|
||||||
|
padding: 0.1rem 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
.reasoning-block > summary:hover {
|
||||||
|
color: var(--msg-assistant-text);
|
||||||
|
}
|
||||||
|
|
||||||
|
.reasoning-block--live > summary {
|
||||||
|
animation: reasoning-pulse 1.6s ease-in-out infinite;
|
||||||
|
}
|
||||||
|
|
||||||
|
.reasoning-content {
|
||||||
|
margin-top: 0.35rem;
|
||||||
|
padding-left: 0.6rem;
|
||||||
|
border-left: 2px solid var(--toolbar-border);
|
||||||
|
white-space: pre-wrap;
|
||||||
|
word-break: break-word;
|
||||||
|
font-family: var(--font-mono);
|
||||||
|
font-size: 0.72rem;
|
||||||
|
line-height: 1.5;
|
||||||
|
max-height: 16rem;
|
||||||
|
overflow-y: auto;
|
||||||
|
}
|
||||||
|
|
||||||
|
@keyframes reasoning-pulse {
|
||||||
|
0%, 100% { opacity: 1; }
|
||||||
|
50% { opacity: 0.45; }
|
||||||
|
}
|
||||||
|
|
||||||
|
@media (prefers-reduced-motion: reduce) {
|
||||||
|
.reasoning-block--live > summary { animation: none; }
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Blinking caret at the end of a streaming assistant bubble. */
|
||||||
|
.stream-caret {
|
||||||
|
display: inline-block;
|
||||||
|
width: 0.5em;
|
||||||
|
height: 1em;
|
||||||
|
margin-left: 0.1em;
|
||||||
|
vertical-align: text-bottom;
|
||||||
|
background: currentColor;
|
||||||
|
opacity: 0.6;
|
||||||
|
animation: stream-caret-blink 1s steps(2, start) infinite;
|
||||||
|
}
|
||||||
|
|
||||||
|
@keyframes stream-caret-blink {
|
||||||
|
to { visibility: hidden; }
|
||||||
|
}
|
||||||
|
|
||||||
|
@media (prefers-reduced-motion: reduce) {
|
||||||
|
.stream-caret { animation: none; }
|
||||||
|
}
|
||||||
|
|
||||||
/* ── Tool call blocks ──────────────────────────────────────────────────────── */
|
/* ── Tool call blocks ──────────────────────────────────────────────────────── */
|
||||||
|
|
||||||
.copilot-tool {
|
.copilot-tool {
|
||||||
|
|||||||
@@ -65,6 +65,7 @@ export default {
|
|||||||
'chat.send': 'Send',
|
'chat.send': 'Send',
|
||||||
'chat.stop': 'Stop',
|
'chat.stop': 'Stop',
|
||||||
'chat.thinking': 'Thinking…',
|
'chat.thinking': 'Thinking…',
|
||||||
|
'chat.reasoning': 'Reasoning…',
|
||||||
'chat.attach': 'Attach files',
|
'chat.attach': 'Attach files',
|
||||||
'chat.new_session': 'New conversation',
|
'chat.new_session': 'New conversation',
|
||||||
'chat.security_group': 'Security group',
|
'chat.security_group': 'Security group',
|
||||||
|
|||||||
@@ -65,6 +65,7 @@ export default {
|
|||||||
'chat.send': 'Envoyer',
|
'chat.send': 'Envoyer',
|
||||||
'chat.stop': 'Arrêter',
|
'chat.stop': 'Arrêter',
|
||||||
'chat.thinking': 'Réflexion…',
|
'chat.thinking': 'Réflexion…',
|
||||||
|
'chat.reasoning': 'Raisonnement…',
|
||||||
'chat.attach': 'Joindre des fichiers',
|
'chat.attach': 'Joindre des fichiers',
|
||||||
'chat.new_session': 'Nouvelle conversation',
|
'chat.new_session': 'Nouvelle conversation',
|
||||||
'chat.security_group': 'Groupe de sécurité',
|
'chat.security_group': 'Groupe de sécurité',
|
||||||
|
|||||||
@@ -65,6 +65,7 @@ export default {
|
|||||||
'chat.send': 'Invia',
|
'chat.send': 'Invia',
|
||||||
'chat.stop': 'Ferma',
|
'chat.stop': 'Ferma',
|
||||||
'chat.thinking': 'Sto pensando…',
|
'chat.thinking': 'Sto pensando…',
|
||||||
|
'chat.reasoning': 'Ragionamento…',
|
||||||
'chat.attach': 'Allega file',
|
'chat.attach': 'Allega file',
|
||||||
'chat.new_session': 'Nuova conversazione',
|
'chat.new_session': 'Nuova conversazione',
|
||||||
'chat.security_group': 'Gruppo di sicurezza',
|
'chat.security_group': 'Gruppo di sicurezza',
|
||||||
|
|||||||
+103
-7
@@ -51,6 +51,7 @@ export class ChatSession extends LightElement {
|
|||||||
// STOP button when reconnecting mid-turn).
|
// STOP button when reconnecting mid-turn).
|
||||||
static _STREAMING_EVENTS = new Set([
|
static _STREAMING_EVENTS = new Set([
|
||||||
'thinking', 'tool_start', 'agent_start', 'pending_write', 'approval_required',
|
'thinking', 'tool_start', 'agent_start', 'pending_write', 'approval_required',
|
||||||
|
'token_delta',
|
||||||
]);
|
]);
|
||||||
|
|
||||||
constructor() {
|
constructor() {
|
||||||
@@ -251,6 +252,7 @@ export class ChatSession extends LightElement {
|
|||||||
this._ws.close();
|
this._ws.close();
|
||||||
this._ws = null;
|
this._ws = null;
|
||||||
}
|
}
|
||||||
|
this._cancelStreamFlush();
|
||||||
this._messages = [];
|
this._messages = [];
|
||||||
this._waiting = false;
|
this._waiting = false;
|
||||||
try {
|
try {
|
||||||
@@ -290,18 +292,62 @@ export class ChatSession extends LightElement {
|
|||||||
});
|
});
|
||||||
break;
|
break;
|
||||||
|
|
||||||
case 'thinking':
|
case 'thinking': {
|
||||||
this._push({ kind: 'thinking', message_id: msg.message_id, content: msg.content,
|
// A tool-call round's text. When the round streamed, its pending bubble
|
||||||
input_tokens: msg.input_tokens, output_tokens: msg.output_tokens });
|
// becomes the thinking item in place. Reasoning comes from the event
|
||||||
|
// (buffered providers) or the streamed accumulation.
|
||||||
|
const last = this._messages[this._messages.length - 1];
|
||||||
|
const streaming = (last?.kind === 'assistant' && last.streaming) ? last : null;
|
||||||
|
const item = { kind: 'thinking', message_id: msg.message_id, content: msg.content,
|
||||||
|
reasoning: msg.reasoning_content ?? streaming?.reasoning ?? null,
|
||||||
|
input_tokens: msg.input_tokens, output_tokens: msg.output_tokens };
|
||||||
|
if (streaming) this._replaceLast(item); else this._push(item);
|
||||||
break;
|
break;
|
||||||
|
}
|
||||||
|
|
||||||
case 'done':
|
case 'token_delta': {
|
||||||
this._waiting = false;
|
// Best-effort live tokens. Accumulate into a pending assistant bubble;
|
||||||
this._push({ kind: 'assistant', content: msg.content,
|
// the final `done` (or `thinking`) event replaces it with authoritative
|
||||||
input_tokens: msg.input_tokens, output_tokens: msg.output_tokens });
|
// content. Mutate in place + throttled flush: deltas can arrive at a
|
||||||
|
// high rate and a full Lit update per token would be wasteful.
|
||||||
|
let last = this._messages[this._messages.length - 1];
|
||||||
|
if (last?.kind !== 'assistant' || !last.streaming) {
|
||||||
|
last = { kind: 'assistant', content: '', reasoning: '', streaming: true };
|
||||||
|
this._messages = [...this._messages, last];
|
||||||
|
this._onMessagePushed(last);
|
||||||
|
}
|
||||||
|
if (msg.kind === 'reasoning') last.reasoning += msg.delta;
|
||||||
|
else last.content += msg.delta;
|
||||||
|
this._scheduleStreamFlush();
|
||||||
break;
|
break;
|
||||||
|
}
|
||||||
|
|
||||||
|
case 'done': {
|
||||||
|
this._waiting = false;
|
||||||
|
const last = this._messages[this._messages.length - 1];
|
||||||
|
if (last?.kind === 'assistant' && last.streaming) {
|
||||||
|
// Finalize the streamed bubble with the authoritative content.
|
||||||
|
this._replaceLast({ kind: 'assistant', content: msg.content,
|
||||||
|
reasoning: msg.reasoning_content ?? last.reasoning ?? null,
|
||||||
|
input_tokens: msg.input_tokens, output_tokens: msg.output_tokens });
|
||||||
|
} else {
|
||||||
|
this._push({ kind: 'assistant', content: msg.content,
|
||||||
|
reasoning: msg.reasoning_content ?? null,
|
||||||
|
input_tokens: msg.input_tokens, output_tokens: msg.output_tokens });
|
||||||
|
}
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
|
||||||
case 'tool_start': {
|
case 'tool_start': {
|
||||||
|
// A round with tool calls but no Thinking event (no usage/text) leaves a
|
||||||
|
// reasoning-only streaming bubble behind: finalize it in place so its
|
||||||
|
// content isn't swallowed by the next round's deltas.
|
||||||
|
const last = this._messages[this._messages.length - 1];
|
||||||
|
if (last?.kind === 'assistant' && last.streaming) {
|
||||||
|
this._replaceLast({ kind: 'thinking', content: last.content,
|
||||||
|
reasoning: last.reasoning || null,
|
||||||
|
input_tokens: null, output_tokens: null });
|
||||||
|
}
|
||||||
// On resume, the server re-emits ToolStart for tools already in history.
|
// On resume, the server re-emits ToolStart for tools already in history.
|
||||||
// Update in place rather than pushing a duplicate card.
|
// Update in place rather than pushing a duplicate card.
|
||||||
const existingIdx = this._messages.findIndex(
|
const existingIdx = this._messages.findIndex(
|
||||||
@@ -399,6 +445,14 @@ export class ChatSession extends LightElement {
|
|||||||
break;
|
break;
|
||||||
|
|
||||||
case 'agent_done': {
|
case 'agent_done': {
|
||||||
|
// A sub-agent's final round emits no Done: its streamed bubble would
|
||||||
|
// stay pending forever — finalize it with the accumulated content.
|
||||||
|
const last = this._messages[this._messages.length - 1];
|
||||||
|
if (last?.kind === 'assistant' && last.streaming) {
|
||||||
|
this._replaceLast({ kind: 'assistant', content: last.content,
|
||||||
|
reasoning: last.reasoning || null,
|
||||||
|
input_tokens: null, output_tokens: null });
|
||||||
|
}
|
||||||
this._updateAgent(msg.stack_id, { done: true });
|
this._updateAgent(msg.stack_id, { done: true });
|
||||||
const agentMsg = this._messages.find(m => m.kind === 'agent' && m.stack_id === msg.stack_id);
|
const agentMsg = this._messages.find(m => m.kind === 'agent' && m.stack_id === msg.stack_id);
|
||||||
if (agentMsg) {
|
if (agentMsg) {
|
||||||
@@ -419,6 +473,7 @@ export class ChatSession extends LightElement {
|
|||||||
|
|
||||||
case 'error':
|
case 'error':
|
||||||
this._waiting = false;
|
this._waiting = false;
|
||||||
|
this._dropStreaming();
|
||||||
this._pushError(msg.message);
|
this._pushError(msg.message);
|
||||||
break;
|
break;
|
||||||
|
|
||||||
@@ -435,6 +490,9 @@ export class ChatSession extends LightElement {
|
|||||||
}
|
}
|
||||||
|
|
||||||
case 'model_fallback':
|
case 'model_fallback':
|
||||||
|
// A fallback mid-stream means the previous attempt's deltas are orphaned:
|
||||||
|
// drop the pending bubble — the replacement model streams a fresh one.
|
||||||
|
this._dropStreaming();
|
||||||
this._push({ kind: 'info', content: `⚡ Model fallback: ${msg.from} → ${msg.to}` });
|
this._push({ kind: 'info', content: `⚡ Model fallback: ${msg.from} → ${msg.to}` });
|
||||||
break;
|
break;
|
||||||
|
|
||||||
@@ -453,6 +511,7 @@ export class ChatSession extends LightElement {
|
|||||||
break;
|
break;
|
||||||
|
|
||||||
case 'new_session':
|
case 'new_session':
|
||||||
|
this._cancelStreamFlush();
|
||||||
this._messages = [];
|
this._messages = [];
|
||||||
this._waiting = false;
|
this._waiting = false;
|
||||||
break;
|
break;
|
||||||
@@ -476,6 +535,7 @@ export class ChatSession extends LightElement {
|
|||||||
|
|
||||||
case 'llm_failed':
|
case 'llm_failed':
|
||||||
this._waiting = false;
|
this._waiting = false;
|
||||||
|
this._dropStreaming();
|
||||||
this._pushError(`LLM unavailable. Tried: ${msg.tried.join(', ')}. ${msg.last_error}`);
|
this._pushError(`LLM unavailable. Tried: ${msg.tried.join(', ')}. ${msg.last_error}`);
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
@@ -487,6 +547,42 @@ export class ChatSession extends LightElement {
|
|||||||
this._onMessagePushed(item);
|
this._onMessagePushed(item);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ── Live token streaming ────────────────────────────────────────────────────
|
||||||
|
// A pending assistant bubble (`streaming: true`) is mutated in place by
|
||||||
|
// `token_delta` events and flushed to Lit at most ~15×/s; turn-ending events
|
||||||
|
// (`done`/`thinking`) finalize it via `_replaceLast`, failures drop it.
|
||||||
|
|
||||||
|
_scheduleStreamFlush() {
|
||||||
|
if (this._streamFlushTimer) return;
|
||||||
|
this._streamFlushTimer = setTimeout(() => {
|
||||||
|
this._streamFlushTimer = null;
|
||||||
|
this._messages = [...this._messages];
|
||||||
|
this._scrollToBottom();
|
||||||
|
}, 66);
|
||||||
|
}
|
||||||
|
|
||||||
|
_cancelStreamFlush() {
|
||||||
|
if (!this._streamFlushTimer) return;
|
||||||
|
clearTimeout(this._streamFlushTimer);
|
||||||
|
this._streamFlushTimer = null;
|
||||||
|
}
|
||||||
|
|
||||||
|
_replaceLast(item) {
|
||||||
|
this._cancelStreamFlush();
|
||||||
|
const updated = [...this._messages];
|
||||||
|
updated[updated.length - 1] = item;
|
||||||
|
this._messages = updated;
|
||||||
|
this._scrollToBottom();
|
||||||
|
}
|
||||||
|
|
||||||
|
_dropStreaming() {
|
||||||
|
this._cancelStreamFlush();
|
||||||
|
const last = this._messages[this._messages.length - 1];
|
||||||
|
if (last?.kind === 'assistant' && last.streaming) {
|
||||||
|
this._messages = this._messages.slice(0, -1);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
_pushError(text) {
|
_pushError(text) {
|
||||||
this._push({ kind: 'error', content: text });
|
this._push({ kind: 'error', content: text });
|
||||||
}
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user