Memory Engine Architecture

The Memory Engine is the core component of GoClaw responsible for persistent long-term knowledge, session-based context management, and semantic retrieval. It provides a hybrid search system that blends traditional keyword search with vector-based semantic retrieval.

This page covers the engine's internals. Memory scopes and who may write them, what goes into the prompt, the search settings and the memory lifecycle are described in Memory & RAG.

System Components

1. Multi-Tiered Storage

The memory system uses a combination of structured and unstructured storage:

• Markdown Storage: Used for human-readable long-term bootstrap (Memory.md) and session archives (memory/archive/).
• SQLite DB: The unified persistent store (internal/store, gc_state.sqlite, WAL mode; PostgreSQL, MySQL or ClickHouse with database.driver).
  • memories table: All records (global, personal, agent_notes, agent_config, and the internal wiki and db_cache scopes).
  • memory_chunks table: Indexed text fragments with vector embeddings.
• Graph DB: kg.sqlite in the state directory (or PostgreSQL with memory.graph.backend: postgres), holding graph_entities and graph_relations for graph queries.
• Session Archive: Full transcripts of past conversations stored in memory/archive/.

2. Hybrid Search Engine (RRF)

The engine uses Reciprocal Rank Fusion (RRF) to combine results from two distinct search algorithms:

1. Vector Search (Semantic): Uses AI-generated embeddings (OpenAI, Gemini, or Ollama) to find conceptually similar content. It calculates Cosine Similarity between query and chunk vectors.
2. Keyword Search (BM25-like): A pure Go implementation of keyword-based scoring with IDF (Inverse Document Frequency) and tokenization.

The final score for each chunk is calculated as: RRF Score = 1 / (60 + rank_vector) + 1 / (60 + rank_keyword).

Rank Enhancement Logic

To ensure the most relevant and "fresh" information surfaces:

• Temporal Decay: For chunks older than 14 days, a decay factor is applied: e^(-0.015 * (days - 14)), with a floor of 0.4.
• Access Bonus: Frequently retrieved chunks (hits in memory_search) get a small boost: +0.002 * access_count (capped at 10 accesses).

3. Chunking Strategy

• Smart Delimiters: Instead of fixed character counts, the system splits content by paragraphs (\n\n), then single lines, and finally sentences to preserve semantic integrity.
• Size: Targets 400 tokens per chunk (1600 characters).
• Overlap: 80 tokens are overlapped between adjacent chunks to maintain context.
• Deterministic IDs: Chunk IDs are generated using sha256(source + startLine + content), allowing the system to reuse cached embeddings when unchanged content is re-indexed.

4. Summarization Pipeline (Session Summarizer)

The engine automatically handles context window limitations through the session summarizer (session_summarizer.go), which runs after a turn:

1. Trigger: When the session's history exceeds memory.session_memory.max_tokens (default 40,000) or, with that set to 0, memory.session_memory.messages (default 40). While the memory flush is on, the history kept is also held to half its threshold (sessionTokenLimit): a turn starts with that history, and one that started past the threshold would be flushed at its first call and answered without any of it. The limit is computed over the whole history, so a session however far over it is back within it after one summarize.
2. Summarization: An LLM (typically Gemini) generates a summary of at most 500 characters of the oldest part of the conversation.
3. Compaction: The session history is trimmed to the newest messages within 60% of the limit (historyTrimShare), and the summary is carried into later turns as PREVIOUS CONTEXT. The turns after add to the history at its end until it passes the limit again, so the start of the conversation, from which every provider's prompt cache reads, holds for those turns; trimmed to the limit itself, it moved after every turn, and every turn ran a summarize. Each trim adds a paragraph to the summary; it keeps its newest 4,000 characters (session.MaxSummaryRunes), so it stays within the memory budget of the prompt.
4. Archival: The full text of the trimmed messages is saved to the session archive (memory/archive/) for later retrieval. When every trimmed message was the current user's, the summary is also saved to their personal memory; a shared thread's summary is not, since it is of what everyone asked.
5. Fact extraction: Each person in the trimmed messages is read on their own lines, under their own role (see Memory Formation Policy).

Memory Flush (memory_flush.go) is separate and works inside one request: when the messages of a single agent loop exceed memory.auto_flush_tokens (default 80,000), the older messages are handed to the model to extract the facts worth keeping, which are saved to the agent's daily memory file, and the loop goes on with the system prompt, the current user message and the last 6 messages. The model is shown each turn labelled Person, Assistant or Tool, and keeps what a person said and what a tool returned; the agent's own replies can be wrong and are not facts, so one is left out or, when a person relied on it, saved behind Unverified, the agent's own earlier answer:. Without a model, the flush writes what people said and what tools returned, never the agent's replies.

Both limits count tokens the same way (estimateTextTokens): a digit or a non-Latin letter is one token, other text four characters a token. Counting bytes undercounted Cyrillic (two bytes a letter, about one token) and figures (Gemini spends a token on each digit) by about half.

5. Memory Job Queue & Background Processing

Heavy memory operations are decoupled from the main execution thread via a persistent Job Queue:

1. Serialization: Tasks like fact extraction, belief revision and knowledge graph synchronization are queued as Job records in the state database.
2. MemoryWorker: A dedicated background worker (internal/agent/memory_worker.go) processes these jobs sequentially.
3. Stability: Sequential processing prevents SQLite "database is locked" errors and ensures LLM API rate limits are respected.

6. Auto-Learning & Knowledge Graph

• Fact Extraction: Automatically identifies and saves environment facts and user preferences to agent_notes.
• Knowledge Graph: LLM-based extraction of entities and relations, stored in memory_entities and memory_relations for future causal reasoning.
• Consolidation: The nightly memory_consolidate job (sys-memory-consolidate, 03:00) removes near-duplicate chunks and merges agent_notes records that say the same thing; each merge is recorded in memory_revisions and can be undone. It does not touch Memory.md: the memory_consolidate tool folds old global records and daily files into Memory.md when an admin asks for it.

Implementation Details

• MemoryWorker: The central orchestrator for background memory tasks.
• Job Types: fact_extract, graph_extract, graph_sync, memory_consolidate, belief_revision. auto_learn_error is no longer queued; rows left from an older version are drained as no-ops.
• Hooks: Uses GraphExtractHook to allow the memory package to trigger jobs in the agent package without circular dependencies.
• Indexing: Incremental indexing (IndexFileAsync) is used to avoid system-wide locks during writes.
• Search Optimization: An idMap (O(1) lookup) is used to quickly retrieve specific chunks by ID.
• In-Memory Store: Chunks and their embeddings are kept in memory for high-performance searching. They are persisted in the memory_chunks table and loaded into memory at start-up (warm-up). A memory-index.json left by older versions is deleted at start-up.

Developer Notes

• Embedding Provider: Defined in internal/memory/embedding.go (OpenAI, Gemini, Ollama), chosen by memory.search.provider; none means keyword-only search.
• Memory Retrieval: hybridSearch in internal/memory/store.go, Indexer.Search in internal/memory/indexer.go, and the memory_search tool in internal/agent/tool_memory_search.go. (internal/agent/rag.go selects tools and skills, not memory.)
• Summarization: Located in internal/agent/session_summarizer.go.
• Auto-Learning: Located in internal/agent/auto_learn.go.