Skald-Circle/docs/compaction.md

Context Compaction

Overview

As a conversation grows, the LLM context window fills up with old messages that are no longer directly relevant. This increases latency and cost, and eventually hits the model's token limit.

Context compaction solves this by periodically summarising the older portion of the history into a dense text block. Only the summary and the most recent raw messages are sent to the LLM on subsequent turns.

The feature is opt-in via config.yml and is disabled by default.

---

Architecture

┌──────────────────────────────────────────────────────────────────┐
│  handle_message()                                                │
│    1. Check last_input_tokens > threshold                        │
│    2. Call ContextCompactor::try_compact()        ←── new        │
│    3. Run normal LLM loop (build_openai_messages injects summary)│
│    4. Store input_tokens in last_input_tokens     ←── new        │
└──────────────────────────────────────────────────────────────────┘
         │
         ▼
┌──────────────────────────────────────────────────────────────────┐
│  ContextCompactor                          src/compactor.rs      │
│  ────────────────────────────────────────────────────────────── │
│  • Stateless service, shared via Arc across all sessions         │
│  • System prompt hard-coded (not an AGENT.md agent)              │
│  • Uses LlmManager::resolve(strength) for AUTO model selection   │
│  • Persists result to chat_summaries DB table                    │
│  • Publishes BusEvent::CompactionDone to ChatEventBus            │
└──────────────────────────────────────────────────────────────────┘
         │
         ▼
┌──────────────────────────────────────────────────────────────────┐
│  DB: chat_summaries                        src/db/chat_summaries │
│  id │ stack_id │ content │ covers_up_to_message_id │ created_at  │
└──────────────────────────────────────────────────────────────────┘
         │ read by
         ▼
┌──────────────────────────────────────────────────────────────────┐
│  build_openai_messages()              src/session/handler/       │
│                                       messages.rs                │
│  if summary exists:                                              │
│    • inject <conversation_summary> after system prompt           │
│    • load only messages with id > covers_up_to_message_id        │
│  else: load all (current behaviour)                              │
│  if compaction disabled: apply max_history_messages as fallback  │
└──────────────────────────────────────────────────────────────────┘
         │ publishes
         ▼
┌──────────────────────────────────────────────────────────────────┐
│  ChatEventBus: broadcast<BusEvent>     src/chat_event_bus.rs     │
│  BusEvent::UserMessage(ChatEvent)                                │
│  BusEvent::AssistantResponse(ChatEvent)                          │
│  BusEvent::CompactionDone(CompactionEvent)   ←── new             │
└──────────────────────────────────────────────────────────────────┘
         │ consumed by
         ├── HonchoPlugin  (UserMessage, AssistantResponse only)
         └── (future consumers: CompactionDone)

---

Trigger Strategy — Opzione C

Compaction is checked at the start of each handle_message call, using the input_tokens value from the previous turn (stored in last_input_tokens: AtomicU32 on ChatSessionHandler).

This means:

• Turn N uses many tokens → last_input_tokens is stored after turn N.
• Turn N+1 starts → compaction is triggered before the LLM runs.
• The user waits for compaction + the new turn, but sees a single response.

No background task, no lock contention, no concurrency hazard.

Skipped cases

Condition	Behaviour
compaction absent from config	Feature disabled entirely
is_ephemeral = true (cron, tic)	Skipped — sessions are short-lived
last_input_tokens == 0 (first turn or no usage data)	Character estimate used as fallback
Fewer messages than keep_recent past the summary boundary	Nothing to summarise, skipped
LLM returns empty summary	Skipped, warning logged

Manual trigger

Compaction can also be triggered manually via ChatSessionHandler::force_compact() or ChatHub::force_compact(source_id). The manual path (force_compact on ContextCompactor) skips the threshold check entirely and uses a character-based token estimate, but still respects the ephemeral guard.

A Telegram /compact command is available as a user-facing interface; see docs/telegram.md.

---

Compaction Flow

try_compact(pool, session_id, stack_id, last_input_tokens, is_ephemeral)
  │
  ├─ guard: is_ephemeral            → Ok(false)
  ├─ resolve effective_tokens:
  │    if last_input_tokens > 0 → use it
  │    else → estimate_tokens_for_stack (sum of chars / 4)
  ├─ guard: effective_tokens < threshold → Ok(false)
  │
  ├─► do_compact(pool, session_id, stack_id, effective_tokens)
  │
  │  (see below)
  │
  └─ Ok(true/false)


force_compact(pool, session_id, stack_id, is_ephemeral)
  │
  ├─ guard: is_ephemeral            → Ok(false)
  ├─ effective_tokens = estimate_tokens_for_stack()   ← no threshold check
  │
  ├─► do_compact(pool, session_id, stack_id, effective_tokens)
  │
  └─ Ok(true/false)


do_compact(pool, session_id, stack_id, effective_tokens)
  │
  ├─ latest_summary = chat_summaries::latest_for_stack()
  ├─ messages = if latest_summary:
  │               for_stack_since(covers_up_to_message_id)
  │             else:
  │               for_stack()
  │
  ├─ guard: messages.len() <= keep_recent → Ok(false)
  │
  ├─ to_summarise = messages[0 .. len - keep_recent]
  ├─ last_covered_id = to_summarise.last().id
  │
  ├─ full_prompt = format_for_summary(
  │     messages      = to_summarise,
  │     prior_summary = latest_summary.content  (if any)
  │   )
  │   → returns a single string (SUMMARIZER_PREAMBLE + transcript + SUMMARY_TEMPLATE)
  │   → if prior summary exists: iterative-update path ("PREVIOUS SUMMARY: …")
  │   → if first compaction: "Create a structured checkpoint summary…"
  │   → transcript uses Hermes-style labels:
  │       [USER]: …
  │       [ASSISTANT]: … [Tool calls: name(args)]
  │       [TOOL RESULT tc_N]: …
  │     with head+tail truncation (6 000+1 500 for messages, 4 000+1 500 for results)
  │
  ├─ (client_name, llm) = LlmManager::resolve(None, None, config.strength)
  ├─ call llm.chat_with_tools([{role: "user", content: full_prompt}], tools=[], options)
  │     options.temperature = 0.3  (faithful, low-creativity)
  │   (Hermes-style: everything in a single user message, no separate system message)
  │
  ├─ summary_id = chat_summaries::save(stack_id, summary_text, last_covered_id)
  ├─ event_bus.compaction_done(CompactionEvent { ... })
  └─ Ok(true)

---

Database Schema

CREATE TABLE chat_summaries (
    id                      INTEGER PRIMARY KEY AUTOINCREMENT,
    stack_id                INTEGER NOT NULL REFERENCES chat_sessions_stack(id),
    content                 TEXT    NOT NULL,
    -- All chat_history rows with id <= this value are covered.
    -- build_openai_messages loads only rows with id > this value.
    covers_up_to_message_id INTEGER NOT NULL,
    created_at              TEXT    NOT NULL DEFAULT (datetime('now'))
);

CREATE INDEX idx_chat_summaries_stack ON chat_summaries (stack_id);

One active summary per stack — multiple rows can exist (each compaction creates a new one), and the most recent is always used. There are no nested summaries: when a second compaction runs, the prior summary body is fed to the LLM under PREVIOUS SUMMARY: in the iterative-update path, and the new row supersedes it.

---

LLM Selection

The compactor uses LlmManager::resolve(None, None, config.strength) — the same AUTO selection used for agents. strength maps to the existing tier system:

Strength	Typical use
very_low / low	Fastest, cheapest local model
average	Recommended for summaries
high / very_high	Overkill for summarisation

The compaction LLM is not a registered agent — its prompt constants are hard-coded in src/core/compactor.rs and are not configurable from agents/ or AGENT.md files:

Constant	Role
pub const SUMMARY_PREFIX	Handoff header prepended to the summary when injected as context. Tells the LLM "reference only — resume from ## Active Task". Also used in messages.rs.
SUMMARIZER_PREAMBLE	Opening of the compaction prompt. Plain wording to avoid content-filter false positives on Azure/OpenAI-compatible providers.
SUMMARY_TEMPLATE	13-section structured template the LLM must fill. Ported from Hermes agent.

---

ChatEventBus Extension

The ChatEventBus was extended from broadcast<ChatEvent> to broadcast<BusEvent>, a new enum:

pub enum BusEvent {
    UserMessage(ChatEvent),
    AssistantResponse(ChatEvent),
    CompactionDone(CompactionEvent),   // new
}

Existing consumers (honcho plugin) were updated to match on BusEvent::UserMessage / BusEvent::AssistantResponse and ignore CompactionDone. Future consumers can subscribe and react to compaction events (e.g. to flush external memory, reset embeddings, etc.).

---

Configuration

# config.yml  (under the llm: section)
llm:
  # ... existing settings ...

  compaction:
    # Trigger compaction when the previous turn used more than this many
    # input tokens. Required.
    threshold_tokens: 30000

    # Number of recent raw messages to keep outside the summary. Default: 6.
    keep_recent: 6

    # Minimum LLM strength for summary generation (AUTO selection).
    # Summaries are simple writing tasks — low or average is sufficient.
    # Omit to use whatever AUTO picks.
    strength: low

Recommended values by use case

Scenario	threshold_tokens	Notes
Local 8k model (LM Studio)	5 000 – 6 000	Compact early and often
Local 32k model	20 000 – 25 000	Leave room for the summary itself
Claude Sonnet (200k)	100 000 – 150 000	Or omit compaction entirely
Claude Haiku (200k)	80 000	Cheaper per call; compact less often

Provider without token usage (e.g. some LM Studio setups)

If the LLM provider does not return input_tokens in the response, the compactor falls back to estimating token usage as total_chars / 4. Set a lower threshold_tokens when relying on this estimate since it underestimates non-ASCII content.

---

Known Limitations

• Sub-agent stacks: compaction applies only to the root session stack (depth = 0). Sub-agent stacks are typically short-lived and do not benefit from compaction.
• Tool results in summary: the serialiser uses a head+tail strategy — message bodies are truncated to 6 000+1 500 chars, tool results to 4 000+1 500 chars, tool-call arguments to 1 200 chars. Very long outputs are preserved at both ends; only the middle is replaced with ...[truncated]....
• Tool result pruning in live context: maybe_hide_tool_result replaces oversized previous-turn results with an informative 1-line summary (e.g. [execute_cmd] ran \cargo build` → exit 0, 47 lines output`) rather than a generic "hidden" placeholder. No LLM call — pure string parsing.
• Frontend visibility: the compaction step is transparent to the user. No Compacting event is sent to the WebSocket. The BusEvent::CompactionDone event is available on the internal bus for future subscribers.
• Cold restart: last_input_tokens is stored in memory (AtomicU32), not in the DB. After a restart, the first turn of a session will not trigger compaction even if the history is already long. The second turn will trigger it correctly once the LLM reports usage.