# Memory Engine Architecture

The Memory Engine is the core component of GoClaw responsible for persistent long-term knowledge, session-based context management, and semantic retrieval. It provides a hybrid search system that blends traditional keyword search with vector-based semantic retrieval.

> This page covers the engine's internals. Memory scopes and who may write them, what goes into the prompt, the search settings and the memory lifecycle are described in [Memory & RAG](memory.md).

## System Components

### 1. Multi-Tiered Storage
The memory system uses a combination of structured and unstructured storage:
- **Markdown Storage**: Used for human-readable long-term bootstrap (`Memory.md`) and session archives (`memory/archive/`).
- **SQLite DB**: The unified persistent store (`internal/store`, `gc_state.sqlite`, WAL mode; PostgreSQL, MySQL or ClickHouse with `database.driver`).
  - `memories` table: All records (global, personal, agent_notes, agent_config, and the internal `wiki` and `db_cache` scopes).
  - `memory_chunks` table: Indexed text fragments with vector embeddings.
- **Graph DB**: `kg.sqlite` in the state directory (or PostgreSQL with `memory.graph.backend: postgres`), holding `graph_entities` and `graph_relations` for graph queries.
- **Session Archive**: Full transcripts of past conversations stored in `memory/archive/`.

### 2. Hybrid Search Engine (RRF)
The engine uses **Reciprocal Rank Fusion (RRF)** to combine results from two distinct search algorithms:
1. **Vector Search (Semantic)**: Uses AI-generated embeddings (OpenAI, Gemini, or Ollama) to find conceptually similar content. It calculates **Cosine Similarity** between query and chunk vectors.
2. **Keyword Search (BM25-like)**: A pure Go implementation of keyword-based scoring with **IDF (Inverse Document Frequency)** and tokenization.

The final score for each chunk is calculated as:
`RRF Score = 1 / (60 + rank_vector) + 1 / (60 + rank_keyword)`.

#### Rank Enhancement Logic
To ensure the most relevant and "fresh" information surfaces:
- **Temporal Decay**: For chunks older than 14 days, a decay factor is applied: `e^(-0.015 * (days - 14))`, with a floor of 0.4.
- **Access Bonus**: Frequently retrieved chunks (hits in `memory_search`) get a small boost: `+0.002 * access_count` (capped at 10 accesses).

### 3. Chunking Strategy
- **Smart Delimiters**: Instead of fixed character counts, the system splits content by paragraphs (`\n\n`), then single lines, and finally sentences to preserve semantic integrity.
- **Size**: Targets ~400 tokens per chunk (~1600 characters).
- **Overlap**: 80 tokens are overlapped between adjacent chunks to maintain context.
- **Deterministic IDs**: Chunk IDs are generated using `sha256(source + startLine + content)`, allowing the system to reuse cached embeddings when unchanged content is re-indexed.

### 4. Summarization Pipeline (Session Summarizer)
The engine automatically handles context window limitations through the **session summarizer** (`session_summarizer.go`), which runs after a turn:
1. **Trigger**: When the session's history exceeds `memory.session_memory.max_tokens` (default 40,000) or, with that set to 0, `memory.session_memory.messages` (default 40). While the memory flush is on, the history kept is also held to half its threshold (`sessionTokenLimit`): a turn starts with that history, and one that started past the threshold would be flushed at its first call and answered without any of it. The limit is computed over the whole history, so a session however far over it is back within it after one summarize.
2. **Summarization**: An LLM (typically Gemini) generates a summary of at most 500 characters of the oldest part of the conversation.
3. **Compaction**: The session history is trimmed to the newest messages within 60% of the limit (`historyTrimShare`), and the summary is carried into later turns as PREVIOUS CONTEXT. The turns after add to the history at its end until it passes the limit again, so the start of the conversation, from which every provider's prompt cache reads, holds for those turns; trimmed to the limit itself, it moved after every turn, and every turn ran a summarize. Each trim adds a paragraph to the summary; it keeps its newest 4,000 characters (`session.MaxSummaryRunes`), so it stays within the memory budget of the prompt.
4. **Archival**: The full text of the trimmed messages is saved to the session archive (`memory/archive/`) for later retrieval. When every trimmed message was the current user's, the summary is also saved to their personal memory; a shared thread's summary is not, since it is of what everyone asked.
5. **Fact extraction**: Each person in the trimmed messages is read on their own lines, under their own role (see [Memory Formation Policy](memory_policy.md#shared-conversations-each-person-on-their-own-lines)).

**Memory Flush** (`memory_flush.go`) is separate and works inside one request: when the messages of a single agent loop exceed `memory.auto_flush_tokens` (default 80,000), the older messages are handed to the model to extract the facts worth keeping, which are saved to the agent's daily memory file, and the loop goes on with the system prompt, the current user message and the last 6 messages. The model is shown each turn labelled Person, Assistant or Tool, and keeps what a person said and what a tool returned; the agent's own replies can be wrong and are not facts, so one is left out or, when a person relied on it, saved behind `Unverified, the agent's own earlier answer:`. Without a model, the flush writes what people said and what tools returned, never the agent's replies.

Both limits count tokens the same way (`estimateTextTokens`): a digit or a non-Latin letter is one token, other text four characters a token. Counting bytes undercounted Cyrillic (two bytes a letter, about one token) and figures (Gemini spends a token on each digit) by about half.

### 5. Memory Job Queue & Background Processing
Heavy memory operations are decoupled from the main execution thread via a persistent **Job Queue**:
1.  **Serialization**: Tasks like fact extraction, belief revision and knowledge graph synchronization are queued as `Job` records in the state database.
2.  **MemoryWorker**: A dedicated background worker (`internal/agent/memory_worker.go`) processes these jobs sequentially.
3.  **Stability**: Sequential processing prevents SQLite "database is locked" errors and ensures LLM API rate limits are respected.

### 6. Auto-Learning & Knowledge Graph
- **Fact Extraction**: Automatically identifies and saves environment facts and user preferences to `agent_notes`.
- **Knowledge Graph**: LLM-based extraction of entities and relations, stored in `memory_entities` and `memory_relations` for future causal reasoning.
- **Consolidation**: The nightly `memory_consolidate` job (`sys-memory-consolidate`, 03:00) removes near-duplicate chunks and merges `agent_notes` records that say the same thing; each merge is recorded in `memory_revisions` and can be undone. It does not touch `Memory.md`: the `memory_consolidate` tool folds old global records and daily files into `Memory.md` when an admin asks for it.

## Implementation Details

- **MemoryWorker**: The central orchestrator for background memory tasks.
- **Job Types**: `fact_extract`, `graph_extract`, `graph_sync`, `memory_consolidate`, `belief_revision`. `auto_learn_error` is no longer queued; rows left from an older version are drained as no-ops.
- **Hooks**: Uses `GraphExtractHook` to allow the `memory` package to trigger jobs in the `agent` package without circular dependencies.
- **Indexing**: Incremental indexing (`IndexFileAsync`) is used to avoid system-wide locks during writes.
- **Search Optimization**: An `idMap` (O(1) lookup) is used to quickly retrieve specific chunks by ID.
- **In-Memory Store**: Chunks and their embeddings are kept in memory for high-performance searching. They are persisted in the `memory_chunks` table and loaded into memory at start-up (warm-up). A `memory-index.json` left by older versions is deleted at start-up.


## Developer Notes
- **Embedding Provider**: Defined in `internal/memory/embedding.go` (OpenAI, Gemini, Ollama), chosen by `memory.search.provider`; none means keyword-only search.
- **Memory Retrieval**: `hybridSearch` in `internal/memory/store.go`, `Indexer.Search` in `internal/memory/indexer.go`, and the `memory_search` tool in `internal/agent/tool_memory_search.go`. (`internal/agent/rag.go` selects tools and skills, not memory.)
- **Summarization**: Located in `internal/agent/session_summarizer.go`.
- **Auto-Learning**: Located in `internal/agent/auto_learn.go`.