A single user message triggers multiple LLM calls (one for each tool call step):
Call #1: System(3K) + Skills(5K) + Tools(8K) + History(4K) = 20K input
Call #2: System(3K) + Skills(5K) + Tools(8K) + History(4K) + Result#1(2K) = 22K input
Call #3: System(3K) + Skills(5K) + Tools(8K) + History(4K) + Results(8K) = 28K input
───────────────────────────────────────────────────────────────────────────
Total for ONE message: 70K input tokens (System+Skills+Tools repeated 32K!)
| # | Optimization | Savings | File |
|---|---|---|---|
| 1 | LLM Response Cache (TTL) | Repeated prompts resolved from cache (every provider, for a first call that offers no tools; not for SOP steps) | llm_cache.go |
| 2 | Tool Result Cache | Identical tool calls resolved from cache | cache.go |
| 3 | Chat History Auto-Summary | Token threshold exceeded → LLM-based summary → trim history to 60% of the limit, so the next turns keep its start (SummarizeAndTrim) |
session_summarizer.go |
| 4 | Context Compaction | Inside the tool loop: system prompt + summary + last N messages (ShouldFlush, FlushAndCompact) |
memory_flush.go |
| 5 | Tool Result Truncation | Results over tools.max_result_chars (default 8000) spilled, model gets a preview + data_ref |
output_store.go |
| 6 | Parallel Tool Calls | All tool calls of one model response run concurrently in the worker pool | tool_round.go |
| 7 | One tool list per session | A session's tools keep their order from turn to turn and new ones are added at the end, so the provider's prompt cache holds the tools across turns (Claude reads them at cache breakpoints placed among the tools; OpenAI, DeepSeek, Gemini and Ollama cache a prompt's prefix themselves) | rag_continuity.go |
| 8 | One skill list per session | A session's skills keep their places in the static prompt and new ones are added at the end (keepSessionSkills), so the static prompt holds across turns |
rag_continuity.go |
| 9 | The message's context after the history | What was found for a message (the notes, lessons and personal memory it matches, hook data, db_query's recent queries) and the time follow the person's message, not the system prompt; the person's text goes as the session keeps it. A turn's first call reads the conversation up to the new message from the provider's cache. On Claude it is a system message where the model takes one, with a cache breakpoint on the person's message | message_context.go, agent.go (SessionContexts) |
Gemini context caching (CachedContent) is disabled: the API refuses a per-request system instruction and tool list with cached content, which Skill RAG and Tool RAG need. The full system prompt is sent on every call.
Instead of injecting every skill into the system prompt, we now use RAG (Retrieval-Augmented Generation) to only inject skills relevant to the user's message:
Current: 3-5 skills × ~200 tokens = ~1K tokens (only relevant + core skills)
We stopped sending descriptions for every tool; instead, the RAG engine filters and sends only the tools relevant to the user request:
Current: 10-15 tools × ~180 tokens = ~2.5K tokens (only relevant + core tools)
A tool result longer than tools.max_result_chars (default 8000 characters) is stored in the output store, and the model gets a head preview with a data_ref. It reads more with read_output (head, slice or search), and spreadsheet can write the whole result to a file. db_query results are always stored. Both agent loops do this (spillLargeResult):
Current: file_read → 500 lines → preview of the first 60 lines (at most 4,000 characters) + data_ref
A turn sends everything it has done so far with each call, so a result is paid for once per later call. After each round, both loops cut the results of rounds older than the newest tools.full_result_rounds (default 4) that are over 2,000 characters to their first ~800 characters and a read_output ref (elideOldToolResults, elideOldToolResultsGenai). On a 42-call turn shaped like a pilot's, shortening alone took the last call from 89k conversation tokens to 65k.
When the model asks for several tools in one response, both agent loops submit them all to the worker pool at once and send the results back together in the next call:
Current: git clone + file_read + file_read → 1 call, run concurrently
Whether the model batches its calls is up to the model; Gemini, OpenAI and Claude all support parallel tool calling.
Instead of repeating the entire chat history in each tool call loop iteration, use a sliding window:
Current: 16 history messages per loop = 4K × 3 calls = 12K
Planned: Only last 2-3 messages in the loop = 1K × 3 = 3K
| Optimization | Current | Planned | Savings |
|---|---|---|---|
| Context Cache (disabled) | 48K | 48K | 0 |
| Skills lazy load (done) | 15K | 3K | -12K ✅ |
| Tools lazy load (done) | 24K | 7.5K | -16.5K ✅ |
| Result truncation (done) | ~18K | ~6K | -12K ✅ |
| History sliding | 12K | 3K | -9K |
| TOTAL | ~117K | ~67.5K | ~42% reduction |
Previously, we planned to expose a tool like read_skill_instructions for the LLM to dynamically fetch documentation, but this added tool-call latency (requiring an extra round-trip). Instead, introducing a RAG (Keyword Overlap Engine) proved to be a cleaner, faster solution:
internal/agent/rag.go: Tokenizes the user's query (removing stopwords), computes similarity scores against the names, descriptions, and tags of tools and skills, and selects the most relevant TopK items.memory_search, memory_write, memory_get, message, secret_get and secret_list, the runtime helpers read_output, json_transform, datetime, calc, regex and text_utils, and anything listed in tools.core. The core skill (memory) is always in the system prompt. Dynamic tools are retrieved via RAG only when needed (for instance, web_search and web_fetch when the user asks for a "web search").tools.rag.usage_prior, default 6), and a few of the tools the previous turn of the session used. When a tool the model needs is still missing, it calls search_tools, and the tools found are callable from its next step. Within a session the list only grows: the tools earlier turns were offered keep their places and new ones follow, up to 8 the turn's selection did not pick, so the start of the prompt, which providers cache, stays the same.