# Tools Engine Architecture The GoClaw Tools Engine is a modular system for managing function-calling (tools), context injection (skills), event handling (hooks), and external extensions (plugins). ## 1. Tool Execution (Function-Calling) Tools are registered via the `Skill` interface in Go. Each tool defines its own JSON schema (parameters) and implementation logic. ### Interface ```go type Skill interface { Definition() llms.Tool // JSON schema for the tool Execute(ctx context.Context, args map[string]any) (string, error) } ``` ### Concurrent Execution via WorkerPool When an LLM requests multiple tool calls in a single response, GoClaw executes them **concurrently** using a bounded **WorkerPool** (default: 8 workers, 64 queue). This prevents goroutine explosion and provides back-pressure under heavy load. Results are collected via channels and returned as a single response to the model. ### Which Tools a Run Is Offered `GetTools(ctx)` builds the list from the registry and the run's `RuntimeScope`; `GetToolsFor` then narrows it per request (section 3). - **Per-agent settings:** an agent's `tools:` block in `agents.list` (shell, git, HTTP, GitLab, SonarQube, observability) becomes `RuntimeScope.Tools` and replaces the matching global setting. An agent's `skills.allow` narrows its tools as well as its skills; core tools stay offered. - **Declared tool sets:** a SOP step, or a subagent whose profile lists tools, gets exactly those tools plus the side-effect-free internal tools (`read_output`, `json_transform`, `datetime`, `calc`, `regex`, `text_utils`), and no RAG selection runs. - **Run cap:** `RuntimeScope.ToolCap` (the tools a subagent run was started with) is applied first, with no exceptions, to the run and every follow-up. - **Role gates:** `db_write` is offered only to a privileged caller when some datasource has `allow_write: true`; `sessions_list` and `sessions_history` only to a human admin; `search_tools` only where tool RAG selects and outside SOP steps. - **Execute:** `Execute()` refuses a tool that was not offered for this request, naming the nearest tools that were. While plan mode is on (`enter_plan_mode` until `exit_plan_mode`), it refuses write tools in the conversation that turned it on (that agent and session); other agents and sessions are not affected. ### Sandboxing & Security - **File System:** All file operations are sandboxed to the workspace directory. Path traversal is strictly blocked. - **Shell:** Command execution is restricted by a prefix-based allowlist (`tools.shell.allow`; empty means no shell at all) and a built-in deny-list of dangerous patterns (e.g., `rm -rf /`, `shutdown`, `ansible`). An allowed command is still refused if it contains a shell metacharacter (`;`, `|`, `&`, `$`, `>`, `=`, …), so commands cannot be chained, piped or redirected. An agent's own `tools.shell` lists replace the global ones. - **Database:** `db_query` runs only `SELECT`, `SHOW`, `DESCRIBE`, `EXPLAIN` and `WITH` statements, and refuses one that hides a write (a data-modifying CTE, `EXPLAIN ANALYZE` of a write, `SELECT … INTO`). `db_write` changes rows only on datasources with `allow_write: true`, for admins and admin-authored SOP/cron runs, at most `tools.max_db_write_rows` rows per call. - **Outbound HTTP:** `SafeHTTPClient` applies `tools.http` (default deny, allow/deny lists, private-address blocking) at dial time. See [Security](security.md#http-default-deny-policy). --- ## 2. Skills System (Context Injection) Skills provide specialized domain knowledge to the LLM via system prompt injection. Unlike tools, they are passive context providers. ### Architecture ``` System prompt = core instruction + IDENTITY.md + SOUL.md + AGENTS.md + instructions + TOOLS.md + WORKSPACE + SKILLS Session context = critical constraints + Memory.md + the person, their role + … Message context = the time + the notes, lessons and personal memory the message matches + … (after the person's message, not in the system prompt) ``` `GetStaticSystemPromptFor` puts the admin's core instruction first. A subagent's profile takes the place of `IDENTITY.md` and `SOUL.md`. `Memory.md` is not in the static prompt: it is part of the session context (`SessionContexts`), whose sections are dropped lowest-priority first when they exceed `memory.context_budget_chars`. See [Memory: Session Context Injection](memory.md#3-session-context-injection). A session's skills, like its tools, keep their places from turn to turn and a turn's new ones are added at the end (`keepSessionSkills` in `rag_continuity.go`), so the static prompt, which providers cache after the tools, holds across turns. The list starts again from the turn's own selection when it would carry more than 4 skills the turn did not pick, more than the skills section holds (`skills.max_in_prompt`, `skills.max_prompt_chars`), or after an hour idle; cron and SOP runs use their own selection. ### Loading & Priority Skills are loaded from four sources, with later sources overriding earlier ones of the same name: 1. **Built-in:** Compiled into the binary. 2. **Plugins:** the skills of the agent's [plugins](#agent-plugins) (the first plugin listed wins between plugins). 3. **Workspace:** `/skills/` (`~/.goclaw/workspace/skills/` for the default agent) 4. **User:** `~/.goclaw/skills/` ### Frontmatter Support Skills can include YAML frontmatter for metadata, tool requirements, and tags. --- ## 3. RAG Engine Integration GoClaw utilizes a multi-layered RAG (Retrieval-Augmented Generation) engine to select relevant tools and skills for each user query (`GetToolsFor`, `FilterSkillsFor` in `rag.go`; `tools.rag`). - **Keyword & Category Matching:** (Layers 1-4: exact name, name parts, categories, IDF-weighted token overlap) Fast, deterministic matching; up to `tools.rag.top_k` tools. - **Semantic channel (optional):** with `tools.rag.semantic: true`, embedding similarity between the request and each description adds a few more entries (`semantic_top_k`, `semantic_min_similarity`). It adds to the keyword hits, never replaces them, and is off by default. - **Pinned tools:** offered whatever the scorer says — the core tools (`memory_search`, `memory_write`, `memory_get`, `message`, `secret_get`, `secret_list`), the internal tools, `tools.core`, the tools the request's own contents call for (an attachment, a URL, a request to edit a SOP), and `subagents` for an agent with subagents. - **Usage prior:** the agent's most-used tools of the last 30 days, read from its traces (`tools.rag.usage_prior`, default 6; 0 turns it off). - **Continuity:** the tools the session's previous turn used, so a follow-up that names no tool keeps them. - **One list per session:** the tools a session's earlier turns were offered stay in their places, and a turn's new ones are added at the end (`keepSessionTools` in `rag_continuity.go`). Every provider caches a prompt by its prefix, and the tools come first in it, so a turn's first call reads the tools from the cache instead of sending them again. When the selection no longer picks more than 8 of the list's tools, or the session has been idle for an hour, the list starts again from the turn's own selection. A SOP step's declared tools and an agent with `tools.rag.enabled: false` are not affected. For the same reason a tool's description does not change with the message: the approved saved queries in `db_query`'s description and a tool's presets are listed in their stored order, and ranked by the message only when the list is too long to show whole; the queries `db_query` ran earlier in the conversation are in the message's context, not in its description. - **`search_tools`:** a pinned tool that finds a permitted tool the selection left out and makes it callable from the next step. A match the caller's role keeps from it (the role gates above, `roleWithheldTools`) is not added; the search says it needs an administrator, and a direct call to it is refused in the same words. - **Tool-error hints:** A failed call to a covered tool comes back, in the same turn, with a hint and the matching approved Saved Queries or Tool Presets (`auto_learn.go`). Nothing is stored: no `agent_notes` row is written and no tool description changes. See [Memory: Auto-Learn](memory.md#7-autonomous-error-learning-auto-learn). --- ## 4. Hooks & Plugins ### Admin Hooks (rules on tool calls) Admins attach rules to the agent's tool calls on the Hooks page; they are saved under `hooks:` in `config.yaml` and run by `hook_rules.go`. `Execute()` runs the before-hooks ahead of the tool cache and the tool, and the after-hooks once the call ends, so both agent loops, SOP and cron runs and subagents are covered. Actions are `block` (before only), `notify` and `webhook`. A deciding hook fails closed: only an explicit `{"decision": "allow"}` lets the call through. A rule limited to `agents` matches the agent a call runs as, by its configured ID; `main` or `default` in the list, in any case, names the default agent, unless an agent is configured under that ID. A SOP run is its owner's when its schedule, the SOPs page, a resume or the background starts it; the webhook route's agent's when a route starts it; the conversation's agent's when `sop_run` starts it there. ### Gateway Events (HookRegistry) GoClaw implements a priority-based event registry for system-wide lifecycle events (`internal/gateway/hooks.go`). Handlers registered by Go code observe these events; they cannot change or stop what happens: - `gateway:start/stop` - `message:received/sending/sent` - `before_prompt_build`, `llm:input`, `llm:output` - `before/after_tool_call` - `session:start/end` ### Extensions (Go / OpenClaw) `PluginAPI` lets a Go plugin compiled into the gateway register hook handlers (`RegisterHook`) and HTTP routes (`RegisterHTTPRoute`). The gateway also discovers extension folders with a `plugin.json` or `openclaw.plugin.json` manifest in `/extensions/` and `/extensions/`, and lists them at `/api/extensions`; it does not load or run code from them. ### Agent Plugins A separate feature: bundles of subagents, instructions, skills, hooks and MCP servers in the Agent Plugins layout VS Code uses (`plugin.json`, `skills//SKILL.md`, `mcp.json`). They are installed on the Plugins page into `/plugins` and turned on per agent under `plugins:` in `config.yaml`. A workspace file of the same name overrides a plugin's part, and a plugin's hooks and MCP servers stay off until an admin approves each one. --- ## 5. Caching Layers 1. **Tool Result Cache:** In-memory LRU cache with TTL and cross-tool invalidation (e.g., `file_write` invalidates `file_read` for the same path). 2. **LLM Response Cache:** a cache in the state database (`llm_cache` table) for model responses that do not contain tool calls. It is on when `cache.enabled` and `cache.llm_cache.enabled` are both true (the default), keeps entries for `cache.llm_cache.ttl` seconds (default 300), and is used by the agent loop for every provider, for a first call that offers no tools. SOP steps bypass it. --- ## 6. Large Output Spill When a tool returns more than `max_result_chars` (default: 8,000 characters), GoClaw **spills** the full content to an in-memory store instead of plain truncation. The LLM receives a compact, type-aware **HEAD preview** and a `ref` ID. ### Flow ```mermaid flowchart LR TOOL["Tool Result
> 8,000 chars"] --> DET["detectContentType()
json / jsonl / html / xml / csv / text"] DET --> STORE["OutputStore.Put(id, full)
TTL: 1h · max: 200 entries"] STORE --> LLM["LLM Receives Preview + ref
Explore full data via read_output(ref)"] ``` Example header and preview returned to the LLM: ```text [LARGE OUTPUT: gitlab | type:json | lines:1847 | chars:58420 | ref:a3f8c2d1] 🛑 TRUNCATED: 2950 of 58420 characters are shown below; 55470 are NOT. ... Preview: [0] {"id":234,"title":"Fix login","state":"merged"} ... To explore full data — call the SEPARATE 'read_output' TOOL ...: read_output(ref: "a3f8c2d1", mode: "search", query: "") ``` The preview is capped at 4,000 characters. When it holds the whole result, the header is followed by a line saying the complete result is shown, and `Result:` in place of `Preview:`. `db_query` results are always stored, even small ones (which are returned whole), so `spreadsheet` can build a file from the `data_ref`; a query with no rows returns a notice instead. Inside a SOP run, a stored result is also written to the run's directory, so a ref handed to a later step outlives the in-memory store. ### Content Type Detection | Type | Detect | HEAD Preview | |---|---|---| | `json` | starts with `{`/`[` + valid JSON | first 5 keys / items | | `jsonl` | every line is `{...}` | first 5 objects | | `html` | `` + text nodes | | `xml` | `?token=`. - **Security scanner** blocks inline `on*` handlers, `javascript:` URIs, untrusted iframes, and `data:text/html`. - **Trusted CDN allowlist** permits scripts/styles from jsDelivr, cdnjs, Google Fonts, Plotly, D3, Bootstrap, etc. - **Per-page token auth** — each page gets a random view token; global gateway auth is also accepted. - **Sandboxed in the browser** — the page is served with a CSP `sandbox` that allows scripts, popups, dialogs and downloads but not `allow-same-origin` or forms, and with `connect-src`, `frame-src` and `form-action` set to `'none'`. The page runs as an origin of its own, so it cannot reach the dashboard API with the cookie of the admin viewing it. The scanner reads the HTML with patterns; the sandbox is what the browser enforces. - **TTL**: `1h`, `6h`, `24h`, `7d` or `30d`; 30 days is both the default and the maximum. At most 100 pages are kept, in memory, so a restart removes them. --- ## 8. Strategic Roadmap Future developments for the tools engine include: - **Code Sandboxing:** Isolated execution of Python/JS snippets using Docker. - **Subprocess Plugins:** Enabling non-Go extensions via a JSON-RPC bridge.