# GoClaw — Complete Official Documentation (ulab.mn/agents/goclaw) > Official Website: https://goclaw.ulab.mn > LLM Index: https://goclaw.ulab.mn/llms.txt > Design notes whose status says the work is not wired or not implemented are listed under "Optional" in llms.txt and left out of this bundle. ================================================================================ # FILE: https://goclaw.ulab.mn/docs/index.md ================================================================================ # GoClaw Documentation GoClaw is a high-performance **Multi-Agent Orchestration Platform and AI Gateway** implemented in Go. It provides a unified interface for multiple AI models, advanced agent routing, and a robust tool ecosystem, optimized for low-resource environments like home servers and NAS devices. ## Key Capabilities - **Agent Router:** Intelligent message routing based on channel, user, or group to specific specialized agents. - **Multi-Agent Orchestration:** Run multiple isolated agents (e.g., Personal Assistant, Coding Expert, Family Bot) with unique personalities, models, and toolsets. - **Subagent Delegation:** Primary agents can spawn temporary subagents for complex multi-step tasks. - **RAG & Long-term Memory:** Integrated Retrieval-Augmented Generation for persistent memory and context-aware interactions. - **AI Gateway:** A unified REST/WebSocket gateway compatible with major AI providers (Gemini, OpenAI, Claude, DeepSeek, Ollama, and any OpenAI-compatible endpoint). - **Built-in Tools:** Shell, files, web, databases, observability and code analysis, compiled into the binary and selected per query by a RAG tool filter. A few wrap programs that must be installed (Chrome for the browser and PDF reports, linters and scanners). - **Model Context Protocol (MCP):** Native support for MCP servers (stdio, SSE, HTTP) to extend agent capabilities dynamically. ## Quick Navigation ### Getting Started - [User Guide](user-guide/index.md) — Where to start: configuration, CLI, automation, observability and the fleet toolkit - [CLI Reference](user-guide/cli-reference.md) — `goclaw setup`, gateway runtime, and every subcommand - [Configuration](user-guide/configuration.md) — Comprehensive configuration guide - [Webhook Routes](user-guide/configuration.md#webhook-routes) — `webhooks:` in `config.yaml`: which agent or SOP an outside system's POST to `/goclaw/hook//` reaches - [Public Ingress](user-guide/ingress.md) — Exposing the gateway through ngrok, Cloudflare, or a reverse proxy - [Gateway Endpoints](api-reference/gateway_endpoints.md) — WebSocket server and REST APIs ### Multi-Agent & Routing - [Agent Architecture](architecture/concepts/agent.md) — How GoClaw manages multiple agents - [Agent Routing](architecture/routing.md) — Channel/role/tenant based agent resolution - [Routing Bindings](user-guide/configuration.md#routing-bindings) — Configuring message-to-agent mapping - [Subagents](architecture/subagents_and_jobs.md) — Spawning and managing task-specific agents - [Autonomy Levels](architecture/autonomy_levels.md) — The guardrails between a chatbot and an AI employee ### Channels - [Overview](user-guide/channels/index.md) — All supported messaging platforms - [Telegram](user-guide/channels/telegram.md) — Long polling by default, so no public URL is needed - [Discord](user-guide/channels/discord.md) — Bot token, guild overrides, threads and embeds; no public URL needed - [Microsoft Teams](user-guide/channels/msteams.md) — Azure Bot registration and Teams packaging - [Facebook Messenger](user-guide/channels/facebook.md) — Webhooks, page tokens, pairing - [Enterprise App API](user-guide/channels/app-api.md) — Your own mobile app / intranet over REST, SSE and WebSocket - [Pairing](user-guide/channels/pairing.md) — Access control and identity linking ### Automation - [Automation Overview](user-guide/automation/index.md) — How scheduled and event-driven work fits together - [Cron Jobs](user-guide/automation/cron-jobs.md) — Scheduled SOP runs and built-in commands - [Standard Operating Procedures (SOPs)](user-guide/automation/sop.md) — Autonomous multi-step workflows - [GitLab Webhook Integration](user-guide/automation/gitlab-webhook.md) — Receiving push/pipeline/MR events from GitLab. Set it up by asking the agent (`gitlab_hook`), with the `goclaw gitlab hook` wizard, or by hand - [SonarQube Quality Gates That Run a SOP](user-guide/automation/sonarqube-hooks.md) — SonarQube's webhook after each analysis runs a SOP when the quality gate fails (or always). Set it up by asking the agent (`sonarqube_hook`) or by hand - [Zabbix Problems That Run a SOP](user-guide/automation/zabbix-alerts.md) — a Zabbix action calls the gateway when a problem starts on the host groups you choose, and runs a SOP. Set it up by asking the agent (`zabbix_alert`) or by hand - [Uptrace Monitors That Run a SOP](user-guide/automation/uptrace-monitors.md) — an Uptrace monitor's webhook channel calls the gateway when an alert opens, and runs a SOP. Set it up by asking the agent (`uptrace_monitor`) or by hand - [Wazuh Alerts That Run a SOP](user-guide/automation/wazuh-alerts.md) — a monitor in the Wazuh indexer calls the gateway with the Wazuh alerts it finds, and runs a SOP. Set it up by asking the agent (`wazuh_alert`) or by hand - [Grafana Alert Rules That Run a SOP](user-guide/automation/grafana-alert-rules.md) — a Grafana alert rule calls the gateway when it fires, and runs a SOP. Set it up by asking the agent (`grafana_alert_rule`) or by hand - [Graylog Alerts That Run a SOP](user-guide/automation/graylog-alerts.md) — Graylog alert webhook: Graylog calls the gateway's hook (`/goclaw/hook/graylog/`) and each alert runs a SOP. Set it up by asking the agent, or by hand on the Webhooks page ### Tools & Memory - [Tools, Skills & MCP](user-guide/tools_and_mcp.md) — Shell, File, Web, MCP, and more - [Tool Offline Testing](user-guide/tool-exec-reference.md) — Direct tool execution without an LLM - [Memory & Context Guide](user-guide/memory_and_context.md) — How the agent remembers and summarizes - [Token Budgeting](user-guide/token-budgeting.md) — Keeping prompt cost predictable - [Memory & RAG Architecture](architecture/memory.md) — Scopes, access, prompt context and the memory lifecycle - [Memory Engine Architecture](architecture/memory_engine.md) — Internal RAG and summarization details - [Tools Engine](architecture/tools_engine.md) — Tool and skill selection, caching, hooks and plugins - [User Identity](architecture/concepts/identity_and_memory.md) — Cross-platform identity and personal memory - [MCP Setup](user-guide/configuration.md#mcp-setup-model-context-protocol) — Connecting to Model Context Protocol servers ### Observability & Monitoring - [Observability Overview](user-guide/observability/index.md) — Multi-source monitoring, incident workflow, config - [SRE Tools](user-guide/observability/sre-tools.md) — Grafana, Graylog (log search, and `allow_write` for alerts that run a SOP), Uptrace, Zabbix, Wazuh - [NetBox IPAM/DCIM](user-guide/observability/netbox.md) — IP, rack, server, VLAN, VM management - [Metrics Store](user-guide/metrics-store.md) — Token and runtime metrics retention - [Fleet Toolkit](user-guide/fleet.md) — Status, logs, reports, upgrades and restarts across many agents over SSH ### Security - [Security Model](architecture/security.md) — Trust boundaries, sandboxing and tool permissions - [Secrets Vault](architecture/secrets.md) — AES-256 encrypted credential storage - [Audit Logging](architecture/audit_logging.md) — What gets recorded and where - [Enterprise Security Matrix](architecture/enterprise-security-matrix.md) — Ports, egress allowlist, OS hardening and InfoSec controls - [SBOM & Supply Chain](architecture/sbom.md) — CycloneDX/SPDX inventories and how to obtain them - [Vulnerability Management & VEX](architecture/vulnerability-management.md) — govulncheck reports and OpenVEX advisories ### API Reference - [Gateway Endpoints](api-reference/gateway_endpoints.md) — REST and WebSocket surface - [OpenAI Compatibility](api-reference/openai_compatibility.md) — Drop-in `/v1/chat/completions` ### Vision - [Vision Overview](vision/index.md) — Where GoClaw is heading - [Level 4: AI Employees](vision/level4_final.md) - [Level 5 Vision](vision/level5.md) - [GoClaw Appliance](vision/goclaw-appliance.md) ## Architecture ```mermaid flowchart TD subgraph CH["Channels"] C1["Telegram"] C2["Discord"] C3["MS Teams"] C4["Messenger"] C5["App API / Web"] end GW["AI Gateway
REST · WebSocket · SSE · Webhooks"] ROUTER["Agent Router
channel / role / tenant → agent"] subgraph ENGINE["Execution Engine"] direction LR MA["Multi-Agent Orchestrator
Isolated contexts · Subagents · SOPs"] TOOLS["Tools & MCP
Built-in tools · RAG-filtered · Security"] end subgraph STORE["Persistence & Memory"] direction LR MEM["Memory & Sessions
SQLite (gc_state) · Embeddings · Replay"] KG["Knowledge Graph
SQLite (kg) · Entity Relations"] end CH --> GW GW --> ROUTER ROUTER --> MA ROUTER --> TOOLS MA <--> TOOLS MA --> STORE ``` ## State Directory Default: `~/.goclaw/` (override with `GOCLAW_STATE_DIR`) ``` ~/.goclaw/ ├── config.yaml # Unified configuration (hot-reloaded) ├── gc_state.sqlite # Sessions, long-term memory, SOPs, cron jobs ├── kg.sqlite # Knowledge graph store ├── workspace/ # IDENTITY.md, USER.md, Memory.md, memory/ ├── agents/ # Isolated agent environments │ └── / │ └── workspace/ # Agent-scoped files and memory ├── memory/personal/ # Per-user private memory (user_.md) ├── credentials/ # AES-256 encrypted secrets and channel credentials ├── skills/ # Global skills ├── plugins/ # Installed plugins (subagents, instructions, skills, hooks, MCP servers) ├── media/ # Uploaded and generated media ├── canvas/ # Diagram and screenshot artifacts ├── cache/ # Transient caches ├── traces/archive/ # Nightly trace dumps (JSONL.gz) └── logs/ # System and agent logs ``` ================================================================================ # FILE: https://goclaw.ulab.mn/docs/user-guide/index.md ================================================================================ # GoClaw User Guide This guide helps you configure, run, and interact with the GoClaw platform. ## Navigation - **[Configuration](configuration.md)**: Comprehensive guide on `config.yaml` options, [LLM providers](configuration.md#llm-providers), the [gateway token](configuration.md#gateway-authentication-token), routing binds, and [webhook routes](configuration.md#webhook-routes). - **[Public Ingress](ingress.md)**: The address outside systems (Teams, GitLab, Graylog, …) reach the gateway at: ngrok, a reverse proxy, or a dedicated webhook port. - **[CLI Reference](cli-reference.md)**: Exhaustive documentation of all available commands (goclaw setup, memory, agents, tools). - **[Channels](channels/index.md)**: Telegram, Discord, Microsoft Teams, Facebook Messenger, the App API, and pairing. - **[Tools, Skills & MCP](tools_and_mcp.md)**: The agent's built-in tools, skills, and MCP servers. Run a tool without an LLM with the [Tool Offline Testing](tool-exec-reference.md) reference. - **[Memory and Context](memory_and_context.md)**: How the agent remembers, summarizes, and keeps personal memory apart. - **[Token Budgeting](token-budgeting.md)**: Token usage analytics and hard token limits per session, agent, and day. - **[Metrics Store](metrics-store.md)**: The `metrics_store` tool for recording and trending computed numbers. - **[Automation](automation/index.md)**: Standard Operating Procedures (SOPs), scheduled tasks (cron), GitLab webhooks, and [Graylog alerts that run a SOP](automation/graylog-alerts.md) (a Graylog → GoClaw webhook). - **[Observability](observability/index.md)**: Direct integrations with Uptrace, Grafana, Graylog, Zabbix, Wazuh and NetBox. - **[Fleet Toolkit](fleet.md)**: Status, logs, reports, upgrades and restarts across many agents, from your own computer. ================================================================================ # FILE: https://goclaw.ulab.mn/docs/user-guide/configuration.md ================================================================================ # GoClaw Configuration Guide This guide covers everything you need to configure the GoClaw Multi-Agent Orchestration Platform. GoClaw uses a powerful YAML-based configuration system that supports hot reloading, environment variables, and secret interpolation. --- ## Configuration Methods GoClaw can be configured using multiple methods: 1. **YAML Configuration File**: The primary configuration source: `config.yaml` in the state directory (`~/.goclaw/config.yaml`), else `config.yaml` in the working directory. 2. **Environment Variables**: Three override the file: `PORT` (`server.port`), `AI_PROVIDER` (`ai.provider`), and `DEBUG=1` or `DEBUG=true` (`debug`). `GOCLAW_STATE_DIR` sets the state directory, which is where `config.yaml` is read from (default `~/.goclaw`); `GOCLAW_HOME` replaces the home directory that `~/.goclaw` is under. `GOCLAW_SECRET_KEY` sets the key the secrets vault is encrypted with. Other settings are not read from the environment, except through a `${env:VAR}` reference in a credential field (see below); the few variables that switch on diagnostics or act as key fallbacks are listed in the [CLI reference](cli-reference.md#environment-variables). 3. **CLI Commands**: Direct manipulation of the YAML file via the CLI. 4. **Setup Wizard**: Interactive onboarding using `goclaw setup`. It opens by itself the first time `goclaw` starts with no `config.yaml`. ### The setup wizard The wizard offers **QuickStart** and **Advanced**. QuickStart asks only what GoClaw cannot guess: 1. Which AI to use, and its API key. The key is checked with the provider straight away (a request for its list of models, which costs nothing): a refused key is asked for again, and one that could not be checked, with no network for example, is kept. For Ollama, QuickStart lists the models the local Ollama has pulled and asks which one to use. 2. What the agent is called and, optionally, what it helps with, in one sentence. This becomes the agent's `IDENTITY.md`. It is asked only while `IDENTITY.md` is still the file GoClaw seeded; after that, the persona is changed on the Web UI's **Instructions** page, where you can edit it as text or describe it in words and press **Draft**. 3. Optionally, a Discord bot token. Everything else takes its default: the provider's model and endpoint, the workspace (`~/.goclaw/workspace`), minimal operating mode, port 8080 on `127.0.0.1`. Advanced asks for each of these (the model, a base URL, Vertex AI, the workspace, channels, the gateway) and also checks the key. When you change provider in either flow, the default agent follows it, unless its `model` names a model of its own. ### YAML Features - **Hot Reload**: Changes to `config.yaml` are detected automatically. Some changes may require a restart, which will be logged. - **References**: `${env:VAR}` (or `${VAR}`) reads an environment variable, `${secret:NAME}` a value from the encrypted vault (`goclaw secret set NAME `), and `${file:/path}` the first line of a file. A bare `${VAR}` that is not set in the environment is also looked up in the vault. References are expanded only in credential fields: the `ai.*.api_key` keys, channel tokens and secrets (including `channels.msteams.app_id` and `tenant_id`, and `channels.facebook.verify_token`), `gateway.auth.token`, `gateway.ingress.authtoken`, `database.dsn`, `memory.graph.dsn` and `password`, datasource `password` and `dsn`, the tokens and passwords of `tools.observability`, `tools.gitlab`/`gitlabs` and `tools.sonarqube`/`sonarqubes` (also in an agent's `tools:`), MCP server `env` and `headers`, `logging.otel.dsn` and `headers`, the URL and secret of an admin hook's webhook, `tools.web_search` keys, voice and email credentials, and a webhook route's `secret`, `token` and `agent`. Anywhere else a `${...}` stays as written. - **$include**: Split large configurations into multiple files using `$include: "path/to/extra.yaml"`, or a list of files: ```yaml channels: $include: - ./channels/discord.yaml - ./channels/msteams.yaml ``` An included file's content is indented to the `$include` key's level; a relative path is relative to the file that includes it. - **CLI commands** resolve `${secret:NAME}` the way the gateway does: `goclaw status`, `sessions`, `memory`, `message` and `tools exec` use the vault's value. --- ## CLI Configuration Management You can get, set, append, remove or delete configuration keys directly from the terminal. | Command | Description | |---|---| | `config set ` | Add or update a key-value pair. | | `config get ` | Retrieve the current value of a key. | | `config append ` | Add a value to a list. | | `config remove ` | Remove a value from a list. | | `config delete ` | Remove a key entirely, including one GoClaw does not recognise (the start-up log reports those as "unknown setting"). | **Examples:** ```bash # Set AI provider and API key goclaw config set ai.provider=gemini goclaw config set ai.gemini.api_key="AIzaSyA..." # Add a skill to the allowed list goclaw config append skills.allow=calculator # View current provider goclaw config get ai.provider ``` --- ## LLM Providers `ai:` says which model provider answers and holds each provider's credentials. Only the block of a provider you use needs a key. ```yaml ai: provider: "gemini" # gemini | openai | claude | deepseek | ollama (default: openai) gemini: api_key: "${secret:GEMINI_API_KEY}" model: "gemini-2.5-flash" # default # use_vertex: true # Vertex AI instead of an API key, with Google # project_id: "my-gcp-project" # Cloud application default credentials # location: "us-central1" # default openai: api_key: "${secret:OPENAI_API_KEY}" model: "gpt-4o" # default # base_url: "https://api.openai.com/v1" # or any OpenAI-compatible endpoint anthropic: # the block for provider "claude" api_key: "${secret:ANTHROPIC_API_KEY}" model: "claude-sonnet-5-5" # default # effort: "medium" # low, medium, high, xhigh, max; empty is the model's default # max_output_tokens: 16000 # default; thinking counts against it deepseek: api_key: "${secret:DEEPSEEK_API_KEY}" model: "deepseek-v4-flash" # default; or deepseek-v4-pro # base_url: "https://api.deepseek.com/v1" # default ollama: base_url: "http://localhost:11434" # default model: "llama3" # default ``` - `ai.provider` is the default agent's provider. Claude's settings are under `anthropic`, but the provider is named `claude`. An unknown name stops the gateway at start-up with the list of valid ones. `gemini-legacy` is still accepted and served as `gemini`. - An agent's `model` in `agents.list` (see [Multi-Agent Setup](#multi-agent-setup)) chooses its provider and model, and for the default agent it wins over `ai.provider`. Write `provider/model` (`ollama/llama3`, `openai/gpt-4o-mini`), a provider name alone (`gemini`: that provider's `model` above), or a model name that starts with `gemini`, `claude`, `deepseek`, `gpt-`, `o1` or `o3`. For other agents any other name is sent to the OpenAI client; for the default agent it is refused and the gateway does not start. Use the `provider/model` form. - `ai.fallback` lists `provider/model` pairs tried in order when an agent's provider fails before the reply used a tool: a rate limit, an outage, a bad key or quota, a network failure. Each needs its credentials under `ai.`. An agent's own `fallback:` in `agents.list` replaces it (`[]` turns it off for that agent). A failure caused by the request itself (too long, malformed), or one after a tool ran, is not handed on. See [Provider Fallback](../architecture/provider_fallback.md). `goclaw doctor` and the dashboard's Doctor page show an `ai_fallback` card per agent: a warning when it has no fallback, or only one on its own provider (one outage then leaves it without replies and fails its SOP steps), an error when no fallback answers the same probe as the AI provider check. - Token limits per session, agent and day are set under `ai.token_budget`, and each SOP run stops at 500,000 tokens unless `sop.max_run_tokens` says otherwise: see [Token Budgeting](token-budgeting.md). --- ## Gateway Authentication Token The gateway does not start without `gateway.auth.token`; it stops with `CRITICAL: Gateway authentication token is missing!`. `goclaw setup` writes a random token when you leave the field blank. ```yaml gateway: auth: token: "${secret:GATEWAY_TOKEN}" # or "${env:GATEWAY_TOKEN}", or the token itself ``` Requests to `/api/*`, the `/gateway` WebSocket, `/v1/*` (OpenAI-compatible API), `/media/`, `/uploads/` and `/canvas/` must carry it, in any of these forms: - an `X-Goclaw-Token: ` header, - an `Authorization: Bearer ` header, - a `?token=` query parameter (WebSocket and SSE clients), - the `X-Goclaw-Token` cookie, which the one-time sign-in link in the start-up log (`Web UI: http://…/?auth_code=…`) sets. Health probes (`/health`, `/ready`), the Web UI's own files, the App API (`/api/app/*`, which checks its own credentials) and the webhook receivers (`/hook/*`, `/goclaw/hook/*`, which check their own secrets) do not take it. When the token is a `${secret:…}` reference whose secret is not in the vault, no token is accepted and every request on the gateway's main port fails with a server misconfiguration error, the exempt routes above included (`/health`, the Web UI's files, the App API, and webhooks that share the port). --- ## Multi-Agent Setup GoClaw allows you to define multiple "Agents," each with its own isolated identity, model, and workspace. ### Defining Agents In `config.yaml`, agents are listed under `agents.list`. ```yaml agents: default: "main" # Fallback agent ID list: - id: "main" name: "Personal Assistant" model: "gemini-2.0-flash" - id: "coding" name: "Code Expert" model: "claude-sonnet-4-5-20250929" skills: allow: ["git", "coding"] tools: shell: allow: ["*"] ``` ### Routing Bindings Bindings map incoming messages from different channels to specific agents. The most specific match wins: a `peer`, then a thread's parent peer, then `tenant_id` with `platform_roles`, then `platform_roles` alone, then `tenant_id`, then `account_id`, then the channel alone. A message no binding matches goes to `agents.default`. ```yaml bindings: - agent_id: "coding" match: channel: "discord" tenant_id: "123456789012345678" # one Discord server (guild ID) - agent_id: "main" match: channel: "discord" # Fallback for all other Discord messages ``` What `tenant_id` and `account_id` hold depends on the channel. On Discord, `tenant_id` is the server (guild) ID and `account_id` is the ID of the user who wrote the message. On Microsoft Teams, `tenant_id` is the team and `account_id` is the bot's app ID; on Telegram `account_id` is the bot's ID, and on Facebook the Page ID. `peer: {kind: "direct" | "group", id: "…"}` matches one conversation by its ID. See [Agent Routing](../architecture/routing.md). --- ## MCP Setup (Model Context Protocol) GoClaw provides native support for MCP, allowing agents to connect to external servers for tools, resources, and prompts. ### Configuring MCP Servers You can define MCP servers in `config.yaml` using either `stdio` (command-based) or `HTTP/SSE`. ```yaml mcp: servers: # Stdio-based server (runs as a subprocess) filesystem: command: "npx" args: ["-y", "@modelcontextprotocol/server-filesystem", "/tmp"] # Remote server via HTTP/SSE remote: url: "https://mcp.example.com" transport: "sse" # auto | streamable | sse headers: Authorization: "Bearer ${secret:MCP_TOKEN}" ``` ### Using MCP via Tools When an MCP server is configured, agents gain access to the `mcp` tool. They can: 1. `connect` to a server. 2. `list_tools` to see what is available. 3. `call_tool` to execute an MCP-hosted tool. --- ## Session Management ### Session Reset Rules Clear sessions automatically, every day at an hour or after a time with no message. `reset` is the rule for every session. `reset_by_type` sets a rule per channel; its key is a channel name as it appears in a session key, such as `telegram`, `discord`, `msteams` or `facebook`. `reset_triggers` are the chat commands that clear a session on demand. Automatic resets reach the sessions of every agent in `agents.list`, each in its own session store. ```yaml session: reset: mode: daily # daily | idle at_hour: 4 # UTC hour, for daily reset_by_type: msteams: mode: idle idle_minutes: 60 # cleared after an hour with no message reset_triggers: ["/new", "/reset"] ``` A reset never cuts into a reply. When a session is still answering a message, a daily reset waits for the reply to be saved and then clears the session. An idle reset leaves it alone, since a session that is answering is not idle, and its idle time starts again. Idle time counts from the last message the running gateway saw: after a restart, a session's idle time starts with its next message. --- ## Tool Safety & Loop Detection GoClaw watches each session's tool calls for loops. Every call gets a loop score from the session's last `history_size` calls: the same tool with the same arguments again, a polling tool called over and over, two tools alternating (A, B, A, B), one tool called many times with different arguments, or one call repeated in between others. The thresholds are compared with that score, not with a count of repeats. - **Warning** (`warning_threshold`, default 10): written to the log only. - **Critical** (`critical_threshold`, default 20): the call runs, and a message telling the model to change its approach and answer is added to the result. - **Circuit breaker** (`global_circuit_breaker_threshold`, default 30): the call is not run. The model gets a message to stop calling tools and answer with what it has. ```yaml tools: loop_detection: enabled: true history_size: 100 warning_threshold: 10 critical_threshold: 20 global_circuit_breaker_threshold: 30 detectors: # all on by default generic_repeat: true # same tool, same arguments known_poll_no_progress: true # a polling tool called again and again ping_pong: true # A, B, A, B same_tool_flood: true # one tool, many different arguments identical_repeat_ratio: true # one call repeated between other calls ``` --- ## Context Pruning To keep LLM context windows efficient, GoClaw can prune old or large tool results. ```yaml context_pruning: mode: "cache-ttl" # off | cache-ttl soft_trim: head_chars: 1000 tail_chars: 1000 ``` --- ## Observability Configure integration with external observability tools like Grafana or Graylog. ```yaml tools: observability: grafana: prod: url: "http://grafana.prod:3000" token: "${secret:GRAFANA_PROD_TOKEN}" ``` --- ## Gateway Ingress & Webhook Isolation To protect the internal Web UI and APIs from accidental public exposure via reverse proxies (like Nginx), you can isolate the webhook receiver on a dedicated port. When `gateway.ingress.port` is specified, GoClaw creates a completely separate webhook-only HTTP server. ```yaml gateway: bind: "loopback" ingress: type: "proxy" port: "8081" # Dedicated listener for webhooks (/hook/*, /goclaw/hook/*, /api/webhook) bind: "lan" # the proxy is on another host; the default, loopback, takes only local connections allow_ips: ["10.0.0.0/8"] ``` This ensures that even if your proxy indiscriminately forwards traffic to `8081`, the Web UI remains safe and hidden on `server.port` (default `8080`). Health probes (`/health`, `/ready`) will remain accessible on both listeners. The dedicated listener binds to `gateway.ingress.bind`, which defaults to loopback (`127.0.0.1`): set `lan` (or an IP) when the proxy or the sender is on another host. With a dedicated port, the webhook routes (`/hook/*`, `/goclaw/hook/*`) are served only there: a hook URL that points at `server.port` gets `404`. See [Public Ingress](ingress.md). ### Rate limiting ```yaml gateway: rate_limit: enabled: true # default per_minute: 60 # per client IP (default 60) trust_proxy: false # take the client IP from X-Forwarded-For ``` The budget covers what can be abused without the gateway token: requests with a missing or wrong token (so guessing the token stays limited), webhooks, and the app API. Requests that carry the gateway token and the Web UI's own static files are not counted, so opening the dashboard — dozens of scripts and API calls — does not use it up. Over-budget requests get `429` with `Retry-After`. Behind a reverse proxy, enable `trust_proxy`, or every client shares the proxy's IP and budget. --- ## Webhook Routes `webhooks:` lists the routes of the webhook receiver at `/goclaw/hook/[/]`: which agent an outside system's POST reaches, the credential it must carry, and whether it starts a SOP. Routes can also be made on the **Webhooks** page or through `POST /api/webhooks` ([Gateway Endpoints](../api-reference/gateway_endpoints.md#webhook-routes-api)); those are stored in the database. A route for the exact key comes before a route of its type without a key; between two routes for the same key, the stored one comes first. ```yaml webhooks: # Graylog alert → the incident-triage SOP, once per alert - type: graylog key: payments-errors secret: "${secret:GRAYLOG_HOOK_PAYMENTS}" agent: main sop_id: incident-triage # GitLab events → a chat turn on agent main - type: gitlab key: myproject secret: "${secret:GITLAB_WEBHOOK_MYPROJECT}" agent: main events: [push, pipeline, merge_request] ``` The first route answers `https:///goclaw/hook/graylog/payments-errors`. | Key | Meaning | |---|---| | `type` | First path segment: `/goclaw/hook/`. `gitlab`, `github`, `graylog` and `zabbix` payloads are read specially; any other name is a generic route. A Graylog notification needs `type: graylog`: under any other type, Graylog's test notification would run like an alert. | | `key` | Second path segment: `/goclaw/hook//`. A route without a key takes every key of its type that has no route of its own. | | `secret` | The shared secret. GitLab sends it as `X-Gitlab-Token`; GitHub signs the body with it (`X-Hub-Signature-256`). Every other type, Graylog included, accepts it as an `X-Token` header, a `?token=` query parameter, or an `X-Signature: sha256=` HMAC of the body. | | `token` | A second credential, compared only with the `?token=` query parameter. | | `agent` | The agent ID that gets the event, or `"*"` for every agent. A chat turn runs on that agent's `model`. An agent the gateway is not running (not in `agents.list`, added since the gateway started, or its model failed to start) does not get it: the delivery is answered `409`, or, on a `"*"` route, goes to the other agents. | | `events` | Event names to take; empty takes every event. Any other event is answered `200 {"status":"filtered"}` and runs nothing. | | `sop_id` | Run this SOP for the event instead of a chat turn. See [Runs started by a webhook](automation/sop.md#runs-started-by-a-webhook). | | `batch_mode` | With `sop_id`: `single` (default) runs the SOP once per event, except that events about an object whose run is in progress wait for it and then run once, together ([Runs started by a webhook](automation/sop.md#runs-started-by-a-webhook)); `bulk` gathers the route's events and runs it once for them. | | `batch_window` | With `batch_mode: bulk`: how many seconds the route gathers events (default 300). A run takes at most 500 events, oldest first; the rest wait for the next run. | A route with neither `secret` nor `token` refuses every delivery with `401`. Keep the secret in the vault (`goclaw secret set GRAYLOG_HOOK_PAYMENTS `) and write a `${secret:…}` reference. The receiver reads `webhooks:` for each delivery, so a change saved to `config.yaml` applies to the next delivery without a restart. A `config.yaml` route is listed on the Webhooks page, but it cannot be paused, edited or deleted there, and it records no hit count or last error: look for `[webhook]` lines in the gateway log instead. A route made on the page, or through `/api/webhooks`, shows both. An admin can also ask the agent in chat, for example "what did the build-events webhook receive today?". Its `webhooks` tool lists the routes with their hit count, last hit and last error, and shows a route's recent deliveries: when each came, its event, and the summary of the payload the receiver handed to the agent or SOP. It never shows a route's secret or token, and it is read-only. Only deliveries that reached an agent or a SOP are there; one refused or filtered out shows only in the route's hit count and last error. Anyone else who asks is told that this needs an administrator. The address the sender calls is the gateway's public address (see [Public Ingress](ingress.md)). Step-by-step guides: [GitLab Webhook Integration](automation/gitlab-webhook.md) and [Graylog Alerts That Run a SOP](automation/graylog-alerts.md). --- ## State Database Configuration GoClaw persists sessions, job queues, SOP definitions, scheduled cron states, agent metrics, and auto-learning logs through GORM. The `database:` block configures the backend driver: ```yaml database: driver: "sqlite" # "sqlite" (default) | "postgres" | "mysql" | "clickhouse" # path: "/var/lib/goclaw/state.sqlite" # SQLite only; leave unset for ~/.goclaw/gc_state.sqlite ``` Leave `path` unset unless you mean to move the database. A relative path is taken inside the state directory, so `path: "sessions.db"` opens a new, empty `~/.goclaw/sessions.db` instead of the existing `gc_state.sqlite`. ### Supported Drivers & DSN Examples | Driver | Description | Connection | |---|---|---| | `sqlite` | Embedded SQLite (WAL mode, zero setup) | `path` (default `~/.goclaw/gc_state.sqlite`) | | `clickhouse` | High-throughput analytics & distributed storage | `dsn: "clickhouse://user:password@localhost:9000/goclaw"` | | `postgres` | Enterprise relational storage | `dsn: "host=localhost user=goclaw password=secret dbname=goclaw sslmode=disable"` | | `mysql` | MySQL / MariaDB relational storage | `dsn: "goclaw:secret@tcp(localhost:3306)/goclaw?charset=utf8mb4&parseTime=True&loc=Local"` | `dsn` is read only by the `postgres`, `mysql` and `clickhouse` drivers, which require it; `sqlite` reads only `path`. ================================================================================ # FILE: https://goclaw.ulab.mn/docs/user-guide/cli-reference.md ================================================================================ # GoClaw CLI Reference The GoClaw gateway runs from a single binary. All commands are executed in the format `goclaw `; `goclaw --help` lists a command's flags. > **Start Gateway:** `goclaw` (without arguments) → HTTP/WS gateway starts (default port 8080). --- ## Quick Start ```bash goclaw setup # Configuration wizard (first time) goclaw quickstart # Short setup wizard: which AI and its key, the agent's name and purpose goclaw # Start gateway (after preparing config) goclaw chat # LLM chat (does not require gateway) goclaw chat --debug # Chat with debug logging ``` --- ## Setup & Configuration ```bash goclaw setup # Interactive wizard goclaw quickstart # Quick wizard (minimal prompts) # Script-friendly (non-interactive) goclaw setup --non-interactive \ --auth-choice gemini-api-key \ --gemini-api-key $GEMINI_KEY \ --gateway-port 8080 \ --gateway-bind lan ``` ### `goclaw setup` flags `--non-interactive` writes a new `config.yaml` from the defaults and these flags, replacing one that is already there. Without `--gateway-token` it generates a token and prints it. | Flag | Description | |---|---| | `--non-interactive` | Non-interactive mode, writes config directly from flags | | `--auth-choice ` | `gemini-api-key` \| `openai-api-key` \| `anthropic-api-key` \| `deepseek-api-key` \| `ollama` \| `custom-api-key` | | `--gemini-api-key ` | Gemini API key | | `--openai-api-key ` | OpenAI API key | | `--anthropic-api-key ` | Anthropic API key | | `--deepseek-api-key ` | DeepSeek API key | | `--custom-base-url ` | Custom provider base URL (also the base URL for `deepseek-api-key` and `ollama`) | | `--custom-model-id ` | Custom model ID | | `--custom-api-key ` | Custom provider API key | | `--model ` | Specify model name (ignored for `anthropic-api-key`) | | `--workspace ` | Agent workspace path (default: `~/.goclaw/workspace`) | | `--gateway-port ` | HTTP/WS port (default: 8080) | | `--gateway-bind loopback\|lan` | Bind network interface (default: `loopback`) | | `--gateway-token ` | Static authentication token | | `--mode local\|remote` | No effect: the config written is always for a local gateway | | `--remote-url ` | No effect: with `--mode remote` the URL is printed, not saved | | `--install-daemon` | No effect: prints a reminder to run `goclaw service install` | | `--skip-skills` | No effect | --- ## Gateway Runtime ```bash goclaw # Start gateway goclaw --debug # Start gateway with debug logging goclaw status # Status of the running gateway goclaw doctor # Run health checks goclaw version # View version information ``` One gateway runs per state directory: a second one started on the same directory exits with `another GoClaw gateway is already running`. `goclaw doctor` checks the config file, the state, workspace and memory directories, the secrets vault, the database and its memory schema, each configured integration (datasources, observability tools, GitLab, SonarQube, MCP servers, enabled channels, mail, and a live call to the AI provider and to each agent's fallback models), whether the gateway is running and, on Linux, whether the installed service starts at boot. A line marked `⚠` is a warning, not an issue: it works today, as an agent with no fallback model does until its provider is down. `status`, `sessions`, `message` and the `memory` commands talk to the running gateway, which they find through `/gateway.lock`, and send `gateway.auth.token` from config.yaml. They do not open the secrets vault: a token written as `${secret:NAME}` is sent as written and refused. `${env:VAR}` works. --- ## Config Management ```bash goclaw config get ai.provider goclaw config set ai.provider=gemini goclaw config set tools.git.allow_write=true goclaw config append skills.allow=my-skill goclaw config remove skills.deny=my-skill goclaw config delete channels.msteams.webhook_port ``` | Command | Description | |---|---| | `config get ` | Read config value | | `config set ` | Set config value | | `config append ` | Append value to list config | | `config remove ` | Remove value from list config | | `config delete ` | Remove the key itself, comments kept. Takes keys GoClaw does not know, so it clears a typo or a setting the start-up log reports as unknown or removed | --- ## Secrets An encrypted vault (AES-256-GCM) for the credentials `config.yaml` refers to with `${secret:NAME}`, so tokens never sit in the config file. ```bash goclaw secret gen APP_KEY_ERP --provider app_api # generate one, store, print once goclaw secret set GEMINI_API_KEY AIza... # store a value you already have goclaw secret get GEMINI_API_KEY # print the decrypted value goclaw secret list # all secrets, values masked goclaw secret list --show-values goclaw secret delete GEMINI_API_KEY ``` | Command | Description | |---|---| | `secret gen ` | Generate a random value, store it, and print it once | | `secret set ` | Store a value you supply — takes both arguments | | `secret get ` | Print the decrypted value | | `secret list` | List names with masked values | | `secret delete ` | Remove a secret | `secret gen` is for the shared secrets GoClaw asks for but does not issue — `channels.app_api.api_key`, `jwt_secret`, webhook tokens. Run it once per calling system to give each its own credential: ```bash goclaw secret gen APP_KEY_DJANGO --provider app_api goclaw secret gen APP_KEY_ERP --provider app_api ``` | Flag | Default | Description | |---|---|---| | `--length` | `32` | Random bytes to generate (minimum 16) | | `--format` | `hex` | Encoding of the stored value: `hex` or `base64` | | `--category` | `api_key` | `api_key`, `token`, `password`, `credential` | | `--provider` | — | Owning provider, e.g. `app_api`, `gitlab` | | `--force` | `false` | Replace an existing secret of the same name | Regenerating over an existing name needs `--force`: it revokes whatever is still presenting the old value, and the only copy of that value is gone. A running gateway picks up vault writes on its own: the secrets page re-reads the file per request. A `${secret:NAME}` already expanded into the live config is refreshed on the next config load, so touch `config.yaml` after adding a secret the config refers to. > Set `GOCLAW_SECRET_KEY` on a production host before writing to the vault. > Without it the passphrase is derived from machine and user identifiers — the > gateway warns about this at startup — and the encryption offers little > protection against someone who already has access to the host. See [Secrets architecture](../architecture/secrets.md) for the interpolation syntax and [App API](channels/app-api.md#one-key-per-calling-system) for issuing one key per caller. ### Password generator `goclaw genpass` (also `gp`, `password`) prints strong random passwords and stores nothing. To generate a value and keep it in the vault, use `secret gen`. ```bash goclaw genpass # profile high: 24 characters, all character classes goclaw genpass --profile ultra -n 5 # five passwords of 32 characters goclaw genpass --profile pin -l 8 # an 8-digit PIN goclaw genpass --no-symbols -l 20 # letters and digits only ``` Profiles: `high` (default, 24), `ultra` (32), `pin` (6 digits), `alpha` (20, no symbols); `-l`/`--length` changes the length. --- ## Chat ```bash goclaw chat # Chat with default agent (does not require gateway) goclaw chat --debug # Chat with debug logging ``` --- ## Message ```bash # Requires a running gateway goclaw message "Hello from CLI" # channel: web goclaw message discord "Deploy production please" goclaw message msteams "Report is ready" ``` The agent answers the text and the command prints the reply; nothing is posted to the channel. With a single argument the channel is `web`: the text goes into the Web UI's admin conversation (`admin-session`) as the admin. With another channel it is answered as a message from a sender named `cli` on that channel, in the session `p2p:cli:`. --- ## Agents ```bash goclaw agents list goclaw agents add analyst goclaw agents bind analyst discord goclaw agents unbind analyst discord goclaw agents delete analyst ``` --- ## Sessions ```bash goclaw sessions list goclaw sessions inspect ``` --- ## Memory The memory commands go through the running gateway and stop with an error when it is not running. `memory check` and `memory migrate` open the database themselves and need no gateway. The commands that delete or rewrite records (`clear`, `clear-user`, `forget`, `notes clear`, `notes purge`, `replay`, `migrate`) first copy the database to `/backups/gc_state_.sqlite`; `dedupe` and `notes delete` do not. | Command | Description | |---|---| | `memory check [--db PATH] [--json]` | Read-only health report: schema migrations, chunk types, oversized chunks, missing embeddings, orphaned and stale chunks. `--db` points it at another database file, a backup for example | | `memory stats` | Record and search-index statistics | | `memory reindex` | Rebuild the search index from all memory sources | | `memory export --file F` / `memory import --file F` | Knowledge catalog (agent notes and agent config) to or from a JSON file | | `memory replay [channel] [--dry-run]` | Restore Memory.md from session archives | | `memory relearn [--agent=ID] [--dry-run] [--force]` | Re-extract agent notes from session archives; `--force` also reprocesses archives that have not changed | | `memory revise [--dry-run]` | Retire learned facts and preferences that a newer one supersedes | | `memory revisions [--run ID] [--record ID] [--limit N]` | List those decisions, newest first (default 50) | | `memory revisions undo ` or `--run ` | Restore a retired record, or everything one run retired. The same record does not replace it again | | `memory backfill-metadata [--batch-size N] [--dry-run]` | Extract topics and entities for records that have none (LLM calls; default 10 records per call) | | `memory migrate [--dry-run]` | Apply pending memory schema migrations; `--dry-run` lists the statements | | `memory prune [--stale=DAYS] [--min-len=N]` | Delete search-index chunks: those never read and indexed more than DAYS ago (default 90), and those shorter than N characters (default 80) indexed before the same cutoff. Memory records are not touched | | `memory dedupe [--ratio=R]` | Delete near-duplicate chunks of the same source, and duplicate agent notes (Jaccard similarity ≥ R, default 0.9) | | `memory forget [--older=AGE]` | Delete global-scope records created before AGE: `90d` (default), `6m`, `1y`, or a date `2026-01-01` | | `memory clear [--yes]` | Delete all memory records, every scope | | `memory clear-user [--yes]` | Delete one user's personal records | | `memory notes list \| grep \| delete \| clear \| purge` | Manage agent notes | ```bash # Statistics and health goclaw memory stats goclaw memory check goclaw memory check --db ~/.goclaw/backups/gc_state_20260405_120000.sqlite --json # Export Knowledge Catalog to JSON goclaw memory export --file catalog.json # Import Knowledge Catalog goclaw memory import --file catalog.json # Rebuild/Reindex memory index goclaw memory reindex # Replay/Restore Memory.md from archive goclaw memory replay discord goclaw memory replay discord --dry-run # Relearn agent notes from session archives goclaw memory relearn # main agent (default) goclaw memory relearn --agent=analyst # specific agent goclaw memory relearn --dry-run # preview what facts would be generated # Retire learned facts/preferences a newer one supersedes (tagged, never deleted) goclaw memory revise --dry-run # preview: each old record, what replaces it, and why goclaw memory revise # apply goclaw memory revisions # what was retired, by what, and why goclaw memory revisions undo # put one back # Cleanup goclaw memory prune # default: stale=90d, min-len=80 goclaw memory prune --stale=60 # chunks never read, indexed more than 60 days ago goclaw memory prune --min-len=120 # also older chunks shorter than 120 characters # Delete duplicate chunks and agent notes (Jaccard similarity) goclaw memory dedupe # default ratio=0.9 goclaw memory dedupe --ratio=0.85 # Delete old global SQLite records goclaw memory forget --older=6m goclaw memory forget --older=1y goclaw memory forget --older=2026-01-01 # Clear all memory goclaw memory clear --yes goclaw memory clear # Clear specific user's personal memory goclaw memory clear-user u_123456 --yes # Manage agent notes goclaw memory notes list goclaw memory notes grep ticket goclaw memory notes delete 100000000000000105 goclaw memory notes clear --yes goclaw memory notes purge '%Table%does NOT exist%' --yes ``` `--dry-run` on `prune`, `dedupe` and `forget` only prints a note and exits: it previews nothing. Look first with `memory check` (stale and oversized chunks) and `memory stats`. **Recommended Workflow/Sequence:** ```bash goclaw memory stats # 1. View current status goclaw memory check # 2. Stale, oversized and orphaned chunks (read-only) goclaw memory notes list # 3. Check agent notes goclaw memory notes grep Table # 4. Find incorrect notes goclaw memory notes purge '%Table%' --yes # 5. Delete incorrect notes goclaw memory prune # 6. Remove noise goclaw memory dedupe # 7. Remove duplicates goclaw memory forget --older=6m # 8. Delete old records goclaw memory reindex # 9. Rebuild index ``` How the agent uses memory: [Memory and Context](memory_and_context.md). The revision trail: [Audit and undo](../architecture/memory_policy.md#audit-and-undo). ### Knowledge graph The graph is kept in `kg.sqlite` in the state directory, or in the file `memory.graph.path` names. These commands open it and the main database directly. | Command | Description | |---|---| | `graph stats` | Entity, relation and topic counts, next to the entity and relation counts in the main database | | `graph check` | Graph reachable, tables readable, counts close to the main database's, one-hop traversal and full-text search working | | `graph migrate [--dry-run]` | Copy memory entities and relations from the main database into the graph. Safe to repeat | | `graph reindex [--dry-run] [--with-llm] [--scopes=global,personal]` | Sync each memory record's entities and relations into the graph; `--with-llm` also queues Gemini extraction for records that have none | | `graph cleanup [--dry-run]` | Delete entities no memory mentions, from the main database and the graph | --- ## Database Snapshots `goclaw db` writes and reads `.gclaw` files: gzip-compressed JSON of the database tables, and optionally workspace files and wiki attachments. These commands open the database directly. ```bash goclaw db export -o backup.gclaw # the core tables goclaw db export -o backup.gclaw --sessions --usage --files --attachments goclaw db export -o ops.gclaw --agent-id=ops # one agent's rows goclaw db inspect backup.gclaw # manifest and row counts goclaw db import -i backup.gclaw --dry-run goclaw db import -i backup.gclaw --replace ``` | Command / flag | Description | |---|---| | `db export` | Write a snapshot (default file `goclaw-export_YYYY_MM_DD.gclaw`, or `-o`/`--output`) | | `--agent-id ` | Only that agent's rows in the tables that have an `agent_id` column; other tables are exported whole | | `--sessions`, `--chunks`, `--executions`, `--usage` | Add conversation history, search-index chunks with embeddings (large), SOP run history, token and voice usage | | `--files`, `--attachments` | Add workspace markdown files; add wiki attachments with their files (large) | | `--no-compress` | Write plain JSON | | `db inspect [--json]` | Print a snapshot's manifest and row counts per table | | `db import -i ` | Import a snapshot. A row whose key already exists is kept unless `--replace`; `--dry-run` writes nothing. What could not be imported is listed, and the command exits non-zero | The [fleet toolkit](fleet.md) fetches these snapshots from remote agents with `make fetch_db`. --- ## Skills ```bash goclaw skills list goclaw skills install github:user/my-skills # every skills/**/SKILL.md in the repository goclaw skills install github:user/my-skills/skills/foo/SKILL.md # one file, from the main branch goclaw skills install https://example.com/skills/foo/SKILL.md # the SKILL.md file itself goclaw skills install ./my-skill/ # local directory or file goclaw skills add github:user/my-skills # choose from a list goclaw skills show git goclaw skills remove my-skill ``` | Command | Description | |---|---| | `skills list` | Loaded skills with their source and status | | `skills check [name]` | Binary dependencies of every skill; with a name, a compatibility report for that skill | | `skills scan [name]` | Security scan of all skills, or of one | | `skills install [--force]` | Install from an `http(s)` URL, `github:user/repo[/path/to/SKILL.md]` or a local path | | `skills add [--force]` | Clone a repository and pick its skills from a list; without a terminal, every compatible skill is installed | | `skills show ` | Skill content and metadata | | `skills remove ` | Remove a user-installed skill | A URL must point at a `SKILL.md` file: what it returns is saved as the skill, so a repository's web page (`https://github.com/user/my-skill`) would be saved as HTML. For a repository use `github:user/my-skill`, which clones it with `git`. Every install, local paths included, runs a compatibility check first and stops on a FAIL verdict; `--force` installs anyway. Every skill an `install` or `add` puts in place is then security-scanned, and one with a critical finding is removed again (the command exits non-zero). A repository with a single `SKILL.md` at its root is installed under the repository's name. Skills are installed under `/skills/`. --- ## Tools — Offline Testing & Debugging `goclaw tools` allows calling registered tools directly from the CLI **without an LLM**. There is no need for the gateway to be running — the configuration and SQLite store will load automatically. ### Subcommands | Command | Description | |---------|-------------| | `tools list` | List all registered tools | | `tools exec [k=v ...]` | Execute a tool directly and print the full output | | `tools bench [k=v ...] [--runs=N]` | Measure tool execution speed | ```bash # List all tools goclaw tools list # Execute tool directly goclaw tools exec datetime goclaw tools exec datetime action=diff from=2026-01-01 to=2026-03-30 goclaw tools exec gitlab action=mrs_list project=myorg/backend goclaw tools exec db_query datasource=main query="SELECT count(*) FROM users" goclaw tools exec memory_search query='deploy error' max_results=10 # Benchmark execution speed goclaw tools bench datetime action=now --runs=50 goclaw tools bench db_query datasource=main query='SELECT 1' --runs=20 ``` Output Example: ``` ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Tool: datetime Args: {"action":"now"} ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2026-03-30T08:45:00+08:00 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Time: 1ms | Chars: 25 | Lines: 1 ``` ### Arg Type Coercion Values are automatically coerced into types. The shell removes one level of quotes before `goclaw` sees an argument, so the examples below are written as you type them: | You type | Tool receives | Description | |-------|---------------|-------------| | `limit=5` | number | Every number is a float64, as in a model's tool call | | `ratio=0.85` | number | Floating-point number | | `active=true` | `bool` | Boolean | | `name=hello` | `string` | Simple text | | `"id='12345'"` | `string` | Quotes that reach `goclaw` keep the value a string, not a number | | `'filter={"k":"v"}'` | `JSON object` | JSON object; quote it, or the shell strips its quotes | | `'tags=["a","b"]'` | `JSON array` | JSON array; quote it, or the shell strips its quotes (zsh also reads `[...]` as a glob) | → **For detailed examples of all tools**, see the [Tool Execution Reference](./tool-exec-reference.md) page. --- ## Teams Integration ```bash goclaw teams setup # Fully interactive configuration wizard # Package from channels.msteams.* in config.yaml goclaw teams package # Or give the values as flags: zip + config goclaw teams package \ --app-id="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" \ --app-password="YOUR_CLIENT_SECRET" \ --tenant-id="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" \ --app-name="GoClaw Bot" \ --icon-source=./logo.png \ --out="./teams-app.zip" # A changed package for an app already in Teams: raise the version goclaw teams package --app-version=1.0.1 ``` Every flag overrides a value from `config.yaml`; only the App ID has to come from one or the other. `--app-id`, `--app-password` and `--tenant-id` given as flags are also written to `channels.msteams.*` in config.yaml (`--app-id` sets `channels.msteams.enabled: true` too) once the package is built, and so are the four publisher flags, under `channels.msteams.app_package`. A package Teams would refuse (an App ID that is not a GUID, a version not of the form `1.0.1`, an image that gives no valid icon) is not written. | Flag | Default | Description | |------|---------|-------------| | `--app-id` | `channels.msteams.app_id` | Microsoft App ID of the Azure Bot (required here or in config.yaml) | | `--app-password` | `channels.msteams.app_password` | Client Secret value | | `--tenant-id` | `channels.msteams.tenant_id` | Directory (tenant) ID, for a single-tenant bot | | `--app-name` | `channels.msteams.bot_name`, else `GoClaw Bot` | Bot name in Teams (cut to 30 characters for `name.short`) | | `--app-version` | `1.0.0` | Manifest `version`; raise it for every changed package of an app already uploaded | | `--developer-name` | `channels.msteams.app_package.developer_name`, else `Ulab LLC` | Publisher shown on the app's page in Teams, at most 32 characters | | `--website-url` | `channels.msteams.app_package.website_url`, else `https://goclaw.ulab.mn/` | Publisher's website (`https://`) | | `--privacy-url` | `channels.msteams.app_package.privacy_url`, else `https://goclaw.ulab.mn/privacy.html` | Privacy policy (`https://`) | | `--terms-of-use-url` | `channels.msteams.app_package.terms_of_use_url`, else `https://goclaw.ulab.mn/terms.html` | Terms of use (`https://`) | | `--out` | `./teams-app.zip` | Zip output path | | `--icon-source` | GoClaw's own icons | PNG, JPEG or GIF file or http(s) URL, resized to the 192×192 color and 32×32 outline icons | | `--ffmpeg` | — | Deprecated and ignored: icons are resized without ffmpeg | > ⚠️ **`--app-id`** must be the **Microsoft App ID** from **Azure Bot → Configuration**. → Read more: [MS Teams Channel Setup](./channels/msteams.md) --- ## GitLab Webhooks ```bash goclaw gitlab hook setup # wizard: GitLab webhook, GoClaw route and vault secret goclaw gitlab hook list goclaw gitlab hook test # default trigger: push_events goclaw gitlab hook test --trigger issues_events goclaw gitlab hook remove # from GitLab and GoClaw goclaw gitlab hook remove --keep-route # from GitLab only goclaw gitlab hook remove --keep-gitlab # from GoClaw only ``` → Read more: [GitLab Webhook Integration](./automation/gitlab-webhook.md#command-reference) --- ## Pairing ```bash goclaw pairing pending [channel] # pending requests (default channel: discord) goclaw pairing list [channel] # approved users (default channel: discord) goclaw pairing approve # role user goclaw pairing approve --role admin goclaw pairing revoke # the u_… ID that approve prints goclaw pairing link # add a channel account to an existing user ``` > While the gateway is running, approve on the Web UI's **Pairing** tab or through the API. > The running gateway does not see a pairing change made with the CLI until it restarts, and > its next write to the pairing files can drop it. See > [Pairing: REST API](./channels/pairing.md#3-rest-api). --- ## Service (OS Daemon) ```bash goclaw service install # Install per-user systemd / launchd service sudo goclaw service install --system # Install system service (runs as your user) goclaw service uninstall # Uninstall service goclaw service start # Start service goclaw service stop # Stop service goclaw service restart # Restart service goclaw service status # Show service status ``` `goclaw service install` writes a per-user service; `sudo goclaw service install --system` writes a system service that still runs as an ordinary user: the one who ran `sudo`, or the one `--run-as ` names. Its data is in that user's `~/.goclaw`. On Linux a per-user service starts at boot only while lingering is on for its user; install turns it on, or prints `sudo loginctl enable-linger ` when it cannot. An installed service keeps the settings of the version that wrote it: to write it again, run `goclaw service uninstall` and then `goclaw service install`. `goclaw doctor` says when an installed service will not start at boot, and how to fix it. `goclaw service start` on a service that already runs says so and exits 0; `goclaw service restart` stops it first and starts the new binary. On macOS both also start a job launchd has loaded but is not running (one that crashed), which `launchctl load` refuses with "Load failed: 5: Input/output error". `goclaw service status` says whether the service runs and where its logs are: the gateway's own log (`~/.goclaw/logs/gateway.log`, or `logging.file.path`), and on macOS what the service printed (`~/Library/Logs/goclaw.stdout.log` and `goclaw.stderr.log`, where a crash at start-up shows). On Linux it gives the `journalctl` commands. **On a Mac, the service can wait for someone to click Allow.** macOS asks at the Mac's screen, and whoever installs the service over SSH or a terminal does not see it: - **Local Network** (macOS 15 and later). A per-user service is a launchd agent, which macOS stops from connecting to the local network (a database, GitLab or Zabbix on the LAN; the call fails with "no route to host") until someone clicks Allow in "goclaw would like to find devices on your local network". There is no command to allow it. A system service is a launchd daemon, which macOS lets through without asking, and it still runs as you, with your `~/.goclaw`: `goclaw service uninstall && sudo goclaw service install --system`. The gateway, started as an agent, makes macOS ask at once, and writes `[macos] WARNING` in its log when it is blocked. - **The firewall**, when it is on. Connections from other machines (the Web UI, webhooks) wait for Allow. Allow goclaw from a terminal: `sudo /usr/libexec/ApplicationFirewall/socketfilterfw --add && sudo /usr/libexec/ApplicationFirewall/socketfilterfw --unblockapp `. macOS sees an unsigned program as new after every update, so both can ask again. `goclaw service install`, `start`, `status` and `goclaw doctor` say which of them applies to this Mac, with the command to run. | | Per-user | `--system` | |---|---|---| | Linux (systemd) | `~/.config/systemd/user/goclaw.service` | `/etc/systemd/system/goclaw.service` | | macOS (launchd) | `~/Library/LaunchAgents/goclaw.plist` | `/Library/LaunchDaemons/goclaw.plist` | On Linux the service logs to the journal; on macOS to `~/Library/Logs/goclaw.stdout.log` and `goclaw.stderr.log`. --- ## Update ```bash goclaw update # the latest release goclaw update --backup # back up database and state files before updating goclaw update --version v0.0.96 # a given release goclaw update --force # install even when that version is already running ``` `goclaw update` reads the latest version from `https://goclaw.ulab.mn/releases/latest.json`, downloads the binary for this platform to a temporary file, checks its SHA-256 against the release's `checksums.json` (and on macOS its code signature, when the binary is signed), keeps the current binary as `.prev`, puts the new one in its place and restarts the installed service. With `--backup`, it first snapshots the SQLite databases (`gc_state.sqlite`, `kg.sqlite`) and state files (`config.yaml`, `credentials/`, `workspace/`, `agents/`, `memory/`, `skills/`, `extensions/`, `plugins/`) to `~/.goclaw/backups/goclaw-backup___.tar.gz` (keeping the 3 most recent update archives). When the binary directory (such as `/usr/local/bin`) or a `--system` service requires root and `goclaw update` was run without `sudo`, it asks for `sudo`; when run as `sudo goclaw update` on a host with a per-user service, it restarts the invoking user's service as that user rather than root's. The [fleet toolkit](fleet.md#upgrade-and-restart) runs it on several hosts. The site keeps only the newest release. When it does not answer, or does not have the version asked for, `goclaw update` downloads from the release bucket, `https://storage.googleapis.com/goclaw-releases/releases/`, which keeps every release from v0.2.2 on; so `--version` can also go back to an earlier one. The binary is checked against the `checksums.json` it came with either way. --- ## Transcribe ```bash goclaw transcribe audio.wav goclaw transcribe audio.wav --language mn ``` --- ## SBOM ```bash goclaw sbom # summary of the modules compiled in goclaw sbom -f cyclonedx -o goclaw.cdx.json # CycloneDX 1.5 JSON goclaw sbom -f spdx -o goclaw.spdx.json # SPDX 2.3 JSON goclaw sbom -f json # JSON summary ``` → Read more: [SBOM & supply chain](../architecture/sbom.md) --- ## Version ```bash goclaw version ``` Output: ``` goclaw v1.2.3 commit : abc1234 built : 2026-04-03T14:53:00Z ``` --- ## Shell Completion ```bash goclaw completion bash > /etc/bash_completion.d/goclaw goclaw completion zsh > "${fpath[1]}/_goclaw" goclaw completion fish > ~/.config/fish/completions/goclaw.fish goclaw completion powershell ``` --- ## Environment Variables | Variable | Description | |---|---| | `GOCLAW_STATE_DIR` | State directory: config.yaml, database, vault, workspace, logs (default `~/.goclaw`) | | `GOCLAW_HOME` | Home directory to look for `.goclaw` in, when `GOCLAW_STATE_DIR` is not set | | `GOCLAW_SECRET_KEY` | Passphrase of the secrets vault (see [Secrets](#secrets)) | | `PORT` | Overrides `server.port` | | `AI_PROVIDER` | Overrides `ai.provider` | | `DEBUG` | `1` or `true`: debug logging in the gateway, as `--debug` | | `GOCLAW_LOG_PROMPTS` | `1` or `true`: write every prompt and response to the log in full. They carry memory and tool results, so credentials and customer data can end up in the log | | `GOCLAW_DEBUG` | `1`: CLI commands print how they look for the running gateway | API keys and tokens are set in `config.yaml`, not read from environment variables of their own. To keep one out of the file, write a reference in its field: `${GEMINI_API_KEY}`, `${env:VAR}`, `${secret:NAME}` or `${file:/path}` (see [Configuration Interpolation](../architecture/secrets.md#2-configuration-interpolation)). Two fallbacks exist: with `ai.gemini.api_key` empty, the Gemini chat client takes `GEMINI_API_KEY` or `GOOGLE_API_KEY` from the environment, and so do the background model calls (the session summarizer, fact extraction, the belief-revision judge, consolidation, `memory_replay`, skill quality checks and the Draft buttons), which ask the default agent's own model when there is no Gemini key in either place; and memory embeddings take `GEMINI_API_KEY` or `OPENAI_API_KEY` when `ai.gemini.api_key` or `ai.openai.api_key` is empty. ================================================================================ # FILE: https://goclaw.ulab.mn/docs/user-guide/ingress.md ================================================================================ # GoClaw Ingress Configuration Guide Ingress is the public entry point for webhooks and channel callbacks. It lets platforms such as Microsoft Teams, Facebook, GitLab, Graylog, and custom webhook senders reach a GoClaw process that is running on a local or private port. ## Request Flow ```mermaid flowchart TD EXT["External Platform
Teams, Facebook, GitLab, Graylog, custom webhook"] --> ING["Ingress Layer
ngrok, nginx, Caddy, HAProxy, or another reverse proxy"] ING --> GW["GoClaw Gateway or Dedicated Webhook Listener
• /hook/msteams, /hook/facebook, /hook/telegram
• /hook/<slug>/msteams, /hook/<slug>/facebook, /hook/<slug>/telegram
• /goclaw/hook/<type>[/<key>]
• /api/webhook"] ``` ## Config Fields ```yaml gateway: ingress: type: "" # "ngrok", "proxy", "nginx", or "" (no tunnel; see Ingress Types) url: "" # The address outside systems reach this gateway at authtoken: "" # Ngrok auth token, only used with type: "ngrok" allow_preset: "" # "microsoft-teams", "none", or empty allow_ips: [] # CIDR/IP allowlist; when set, it replaces the preset port: "" # Optional dedicated webhook port bind: "" # Webhook port's interface: "loopback" (default), "lan", or an explicit IP trust_proxy: false # Trust X-Forwarded-For / X-Real-IP from a known proxy ``` If `gateway.ingress.port` is empty, webhook routes share the main gateway port. If it is set, GoClaw starts a separate webhook server for ingress traffic. This is useful when the WebUI and admin API should stay private while public webhooks are exposed. That server listens on `gateway.ingress.bind`, which defaults to loopback: set `bind: "lan"` (or an IP) when the sender reaches it directly rather than through a proxy on the same host. With a dedicated port, the webhook routes (`/hook/*` and `/goclaw/hook/*`) leave the Web UI port: they are served only on `gateway.ingress.port`, and a hook URL that points at `server.port` gets `404`. Health probes and `/api/webhook` are served on both ports; `/api/webhook` needs the gateway token on either. The Web UI, `/gateway` and the rest of `/api/*` stay on `server.port` only. `url` is the gateway's address as the sender sees it, including a proxy's path prefix if it has one. Features that give this address to another system read it whatever `type` is: the Webhooks page, the agent's `graylog_alert` tool (Graylog calls `/goclaw/hook/graylog/`) and the `goclaw gitlab hook` wizard. On a LAN with no proxy it can be the gateway's own address, for example `http://192.0.2.10:8080`; the GitLab wizard, though, accepts only an `https://` address. ## Ingress Types | Type | Behavior | |---|---| | empty | No tunnel is started. `url`, when set, is still the address the Webhooks page, `graylog_alert` and the GitLab wizard give senders; without it the Webhooks page builds hook URLs from the address you opened it at. This does not limit who can reach the gateway: that is wherever `gateway.bind` (or `gateway.ingress.bind`, with a dedicated `port`) listens, subject to `allow_ips` / `allow_preset` | | `ngrok` | GoClaw starts or uses an ngrok tunnel for webhook traffic | | `proxy` | An external reverse proxy forwards traffic to GoClaw | | `nginx` | Alias-style value for reverse proxy deployments | ## Common Routes | Route | Purpose | |---|---| | `/hook/msteams` | Microsoft Teams channel webhook | | `/hook/facebook` | Facebook channel webhook | | `/hook/telegram` | Telegram channel webhook (only with `channels.telegram.mode: webhook`) | | `/hook//msteams`, `/hook//facebook`, `/hook//telegram` | The same webhooks when `gateway.webhook_secret_slug` is set: the plain `/hook/` path is then not served | | `/goclaw/hook/[/]` | Generic webhook router | | `/api/webhook` | Inbound message API: the agent answers a text as if it came from the named channel (`goclaw message` uses it) | | `/api/send` | Outbound message API for local scripts: main gateway port only, loopback callers only, with the gateway token. Not a route to expose | | `/gateway` | Web chat WebSocket when the main gateway is exposed | ## Ngrok For Testing ```yaml gateway: ingress: type: "ngrok" authtoken: "${NGROK_AUTHTOKEN}" url: "https://my-agent.ngrok.app" port: "8001" allow_preset: "microsoft-teams" ``` The built-in tunnel delivers requests to GoClaw from the local machine (loopback), which the allowlist always lets through, so with `trust_proxy: false` (the default) `allow_preset` does not filter ngrok traffic. The Teams webhook is still protected by its Bot Framework token check (see [Microsoft Teams](channels/msteams.md)). The allowlist filters callers that connect directly or through a reverse proxy. Use a stable ngrok domain when the external platform stores a fixed callback URL. For Microsoft Teams, set the Azure Bot messaging endpoint to: ```text https://my-agent.ngrok.app/hook/msteams ``` or, with a secret slug: ```text https://my-agent.ngrok.app/hook//msteams ``` ## Reverse Proxy With Path Prefixes One domain can serve multiple agents by prefix. The proxy must strip the prefix before forwarding to GoClaw. ```text https://agents.example.com/finance/ -> 127.0.0.1:8001 https://agents.example.com/analyst/ -> 127.0.0.1:8002 ``` Agent config: ```yaml gateway: bind: "loopback" ingress: type: "nginx" url: "https://agents.example.com/finance" port: "8001" bind: "loopback" allow_preset: "microsoft-teams" trust_proxy: true ``` Nginx example: ```nginx location /finance/ { proxy_pass http://127.0.0.1:8001/; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $remote_addr; proxy_set_header X-Forwarded-Proto $scheme; proxy_read_timeout 300s; proxy_send_timeout 300s; proxy_buffering off; } location = /finance { return 301 $scheme://$host/finance/; } ``` The trailing slash in `proxy_pass http://127.0.0.1:8001/;` is important. It strips `/finance/` before forwarding, so `/finance/health` reaches GoClaw as `/health`. ## Reverse Proxy With Subdomains Use a subdomain per agent when DNS and TLS automation are already in place. ```nginx server { server_name finance.agents.example.com; location / { proxy_pass http://127.0.0.1:8001/; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $remote_addr; proxy_read_timeout 300s; proxy_buffering off; } } ``` ```yaml gateway: ingress: type: "nginx" url: "https://finance.agents.example.com" port: "8001" bind: "loopback" trust_proxy: true ``` ## Allowlist And Proxy Trust `allow_preset: "microsoft-teams"` admits the Microsoft 365 address ranges Teams calls from. GoClaw downloads them from `endpoints.office.com` at start-up; if the download fails, the gateway does not start (`Ingress failed: …`), so the host needs outbound HTTPS to that address. `allow_ips` is a list of IPs and CIDR ranges. When it has any entry, it **replaces** the preset: the Teams ranges are not loaded, and only the listed addresses get in. To admit Teams and your own addresses, use one of these: - Leave `allow_ips` empty and keep the preset. Callers on loopback are always allowed, so health checks and local tools still get in. Behind a reverse proxy on the same host, set `trust_proxy: true`: the allowlist then judges the client address the proxy forwards, not the proxy's own loopback address. - List everything yourself in `allow_ips`: the Teams ranges (Microsoft publishes them at `endpoints.office.com`, service area "Skype") and your own proxy or test addresses. ```yaml gateway: ingress: allow_ips: # no allow_preset: these are the only addresses let in - "10.0.1.5" # the reverse proxy - "203.0.113.10/32" # a test machine ``` The allowlist covers every request on the listener it guards. With `gateway.ingress.port` set, that is the dedicated webhook listener. Without it, webhooks share the main port, and the allowlist covers the Web UI and the API there too: a browser on an address that is not listed gets `403`. Set `trust_proxy: true` only when the immediate upstream is a proxy you control, and have that proxy set or append `X-Forwarded-For` with the client address it saw (`proxy_set_header X-Forwarded-For $remote_addr;` in nginx, as the examples above do; `$proxy_add_x_forwarded_for` works too). GoClaw takes the **last** entry, the one your proxy wrote: anything before it is what the caller sent. Otherwise, GoClaw uses the TCP remote address and ignores spoofable forwarding headers. Loopback callers are always allowed for local health checks and reverse proxy deployments. ## Troubleshooting | Symptom | Likely cause | Fix | |---|---|---| | Prefix URL returns 404 | The reverse proxy did not strip the prefix | Use a trailing slash in `proxy_pass` | | WebSocket does not connect | `Upgrade` and `Connection` headers are missing | Forward WebSocket headers | | Teams endpoint validation fails | Wrong path or method filtering in proxy | Forward all methods to GoClaw; let GoClaw validate | | Requests are denied by allowlist | GoClaw sees the proxy IP, not the external IP | Enable `trust_proxy` for a trusted proxy, or add the proxy IP to `allow_ips` (which replaces `allow_preset`) | | Gateway exits at start-up with `Ingress failed` | `allow_preset: microsoft-teams` could not download the Microsoft 365 ranges | Allow outbound HTTPS to `endpoints.office.com`, or list the ranges in `allow_ips` | | Hook URL returns 404 | `gateway.ingress.port` is set and the URL points at `server.port` | Point the sender (or the proxy) at the webhook port | | WebUI is public unintentionally | Webhook and WebUI share the same public listener | Use `gateway.ingress.port` and expose only that port | ## Implementation Files - `internal/config/config.go` - `internal/config/config_loader.go` - `cmd/gateway/main.go` - `cmd/gateway/startup.go` - `internal/gateway/handler_api.go` - `internal/runtimeinfo/runtimeinfo.go` ================================================================================ # FILE: https://goclaw.ulab.mn/docs/user-guide/fleet.md ================================================================================ # Fleet Toolkit (Remote Management via SSH) Whether you run GoClaw on **a single remote machine** (a Mac Mini, Linux VPS, or Raspberry Pi) or across **dozens of hosts**, the Fleet Toolkit lets you inspect, back up, audit, and upgrade them from your own laptop over SSH. Even for a single host, the toolkit gives you one-command workflows that go beyond the on-host CLI: - **`make macos-check`** — Checks whether a headless Mac Mini will sleep, stop at the FileVault/login screen, or fail to auto-start after a power cut, and prints the exact `pmset` command to fix it. - **`make upgrade`** — Prints the plan for every host and the exact command that carries it out. With `CONFIRM=yes` it checks that no scheduled job or active turn is running, downloads a `.gclaw` backup to your laptop first, runs `goclaw update`, verifies the new process answers `/health`, and **automatically rolls back** to the previous binary if the new gateway fails to come up. - **`make report`** — Downloads a database snapshot and recent logs to your laptop and generates a 7-day Markdown health report (`reports/report_.md`) covering request latency, tool failures, SOP runs, and log errors without exposing user chat text. - **`make teams_package`** — Reads your remote agent's Microsoft Teams settings and builds the ready-to-upload Teams app `.zip` on your computer. It is a set of Ansible playbooks behind `make` commands. It reaches each host over SSH with your key and installs nothing on the host: the playbooks run the host's own `goclaw` binary and read the files it already wrote. macOS and Linux hosts can be mixed in one list. ## 1. Prerequisites ### A. On your computer (Operator machine) Choose the instructions for the operating system you work from: #### macOS 1. Install Ansible (`make` and `python3` are included with Xcode Command Line Tools or Homebrew): ```bash brew install ansible ``` 2. Generate an SSH key (if you do not have one) and copy it to the GoClaw host: ```bash ssh-keygen -t ed25519 ssh-copy-id ops@192.0.2.10 ``` #### Linux (Ubuntu / Debian / RHEL / Fedora) 1. Install `ansible`, `make`, `python3`, and `curl`: ```bash # Ubuntu / Debian sudo apt update && sudo apt install -y ansible make python3 curl # RHEL / Fedora / Rocky sudo dnf install -y ansible-core make python3 curl ``` 2. Generate an SSH key and copy it to the GoClaw host: ```bash ssh-keygen -t ed25519 ssh-copy-id ops@192.0.2.10 ``` #### Windows (via WSL2) Ansible requires a POSIX environment and does not run directly in PowerShell or CMD. Run the toolkit inside **WSL2 (Ubuntu)**: 1. Open PowerShell as Administrator and install WSL (if not already installed): ```powershell wsl --install -d Ubuntu ``` 2. Open the **Ubuntu** terminal from the Start menu and install the prerequisites: ```bash sudo apt update && sudo apt install -y ansible make python3 curl ``` 3. Inside the Ubuntu terminal, generate an SSH key and copy it to the GoClaw host: ```bash ssh-keygen -t ed25519 ssh-copy-id ops@192.0.2.10 ``` > [!IMPORTANT] > The toolkit connects with `BatchMode=yes` and never stops to ask for a password. Always test `ssh ops@192.0.2.10` once from your terminal to confirm it logs in without a password prompt before running `make ping`. --- ### B. On each Agent Host (Target machine running GoClaw) The host only needs an existing GoClaw installation, `python3`, `curl`, and SSH enabled for the account that runs GoClaw. #### macOS Agent Host (e.g., Mac Mini) 1. **Enable SSH (Remote Login):** - Open **System Settings → General → Sharing** and turn on **Remote Login** (or run `sudo systemsetup -setremotelogin on` in Terminal). 2. **Ensure `python3` is installed:** - Run `python3 --version` on the Mac. If macOS prompts to install the Command Line Developer Tools, accept it (`xcode-select --install`), or install via Homebrew (`brew install python`). 3. **Prevent sleep and login-screen hangs on a headless Mac:** - Disable sleep and enable automatic restart after power failure: ```bash sudo pmset -a sleep 0 autorestart 1 womp 1 ``` - If GoClaw runs as a user LaunchAgent (`~/Library/LaunchAgents/goclaw.plist`), turn **FileVault OFF** and enable **Automatic Login** (**System Settings → Users & Groups → Automatically log in as…**) so the Mac does not wait at the pre-boot unlock screen after a power cut. Run `make macos-check` from your computer to verify all of these settings. #### Linux Agent Host (e.g., Ubuntu/Debian VPS, Raspberry Pi, RHEL) 1. **Install SSH server, Python 3, and curl:** ```bash # Ubuntu / Debian / Raspberry Pi OS sudo apt update && sudo apt install -y openssh-server python3 curl sudo systemctl enable --now ssh # RHEL / Fedora / Rocky sudo dnf install -y openssh-server python3 curl sudo systemctl enable --now sshd ``` 2. **Service mode (`systemd --user` vs `--system`):** - **Per-user service (`goclaw service install`):** Enable user lingering on the Linux host so the agent starts at boot and stays running after you log out of SSH: ```bash sudo loginctl enable-linger $USER ``` - **System-wide service (`sudo goclaw service install --system`):** Read commands (`make status`, `make fetch_logs`, `make fetch_db`, `make report`) work without root. To run `make upgrade` or `make restart` remotely against a system service, add `SUDO=yes` (which requires passwordless `sudo` for the SSH user), or run `sudo goclaw update` directly on the host. 3. **Binary in `/usr/local/bin`:** `install.sh` puts the binary there, owned by root. Whatever the service mode, replacing it needs root, so `make upgrade` needs `SUDO=yes` and passwordless `sudo` for the SSH user. A binary in `~/bin` or `~/.local/bin` is replaced as the SSH user. --- ## 2. Download Each release publishes the toolkit as `goclaw-fleet-.tar.gz`: ```bash v=$(curl -fsSL https://goclaw.ulab.mn/releases/latest.json | grep -o '"version"[[:space:]]*:[[:space:]]*"[^"]*"' | cut -d'"' -f4) curl -fsSL "https://goclaw.ulab.mn/releases/$v/goclaw-fleet-$v.tar.gz" | tar -xz cd goclaw-fleet ``` *(Optional)* Verify the archive against `checksums.json` before extracting: ```bash curl -fsSLO "https://goclaw.ulab.mn/releases/$v/goclaw-fleet-$v.tar.gz" curl -fsSL "https://goclaw.ulab.mn/releases/$v/checksums.json" \ | python3 -c "import json,sys,hashlib; want=json.load(sys.stdin)['goclaw-fleet-$v.tar.gz']; got=hashlib.sha256(open('goclaw-fleet-$v.tar.gz','rb').read()).hexdigest(); print('checksum ok' if got==want else 'CHECKSUM MISMATCH')" tar -xzf "goclaw-fleet-$v.tar.gz" && cd goclaw-fleet ``` ## 3. Set up `hosts.ini` ```bash cp inventory/hosts.ini.example inventory/hosts.ini $EDITOR inventory/hosts.ini make ping make status ``` `inventory/hosts.ini` has one line per host (even if you only have one): ```ini [goclaw_agents] agent-ops ansible_host=192.0.2.10 ansible_user=ops agent-finance ansible_host=192.0.2.11 ansible_user=finance ``` If your host uses a non-default SSH port or a specific key file, add `ansible_port=2222` or `ansible_ssh_private_key_file=~/.ssh/id_ed25519` on the same line. The toolkit automatically detects on each host where the `goclaw` binary, its state directory, and its logs are, which port it answers on, and how its service runs (`launchd` or `systemd`, per-user or system-wide). The state directory is read from the service unit or plist, defaulting to `~/.goclaw`. Every `goclaw` command the toolkit runs on the host uses that state directory. ## Read commands These change nothing on the agents. | Command | What it does | |---|---| | `make status` | Shows the version and process, whether the gateway answers, and the service state. | | `make version` | Shows the version on every host. | | `make doctor` | Runs the agent's own `goclaw doctor`, which also says whether its service starts at boot. | | `make quiet_check` | Checks the agent is idle: no scheduled job running, and its log still for two minutes. | | `make get-config` | Shows `config.yaml` with credential values masked on the agent before anything is sent. | | `make macos-check` | Reports a macOS agent's sleep, firewall and login settings. | | `make logs` | Shows the end of the gateway log and the service log (launchd file or systemd journal). | | `make fetch_logs` | Downloads the gateway log to `fetched//gateway.log`: the last hour, unless `ARGS` says otherwise (below). | | `make fetch_db` | Downloads a `.gclaw` database snapshot to `fetched//`. | | `make fetch_md` | Downloads the workspace's Markdown files (`Memory.md`, `AGENTS.md` and the other bootstrap files) to `fetched//`. | | `make backup` | Saves a snapshot with SOP runs to `backups//`. | | `make memory_stats` | Shows memory counts and index health; it does not reindex. | | `make report` | Downloads snapshots and logs, then writes `reports/report_.md` (`report__.md` with `LIMIT`). | | `make teams_package` | Reads an agent's Teams settings and builds the Teams app `.zip` on your computer. | Every command takes `LIMIT=` and `ARGS="-e key=value"`. `make help` lists every command, and the `Makefile` documents each one's options above its target. The ones you will reach for most: ```bash make fetch_logs LIMIT=agent-ops ARGS="-e hours=3" make fetch_logs LIMIT=agent-ops ARGS="-e since=2026-06-01T09:00" # the agent's local time, not UTC make fetch_logs LIMIT=agent-ops ARGS="-e commit=" # since that build was deployed make fetch_db LIMIT=agent-ops ARGS="-e fetch_db_sessions=true -e fetch_db_usage=true" make backup LIMIT=agent-ops ARGS="-e backup_sessions=true" ``` `fetch_logs` takes the first of `commit`, `since` and `hours` that is given. `fetch_db` adds tables with `fetch_db_sessions`, `fetch_db_chunks`, `fetch_db_executions`, `fetch_db_usage`, `fetch_db_files` and `fetch_db_attachments`, and `fetch_db_agent_id=` keeps one agent's rows; `backup` takes `backup_sessions`, `backup_chunks`, `backup_usage`, `backup_files` and `backup_attachments`. ## The report `make report` (the last 7 days by default; `DAYS=1` for one) opens with a table across the fleet. Each row gives a host's version, its requests, the share that failed, tool failures, SOP runs that did not complete, failing scheduled jobs, and the panics and errors in its log. Then, for each host: - its requests by channel and status, how long they took, and the tokens used; - the slowest requests and the failed or interrupted ones; - each tool's calls and how often they failed; - SOP runs per SOP: completed, partial, failed, interrupted; - the starts that failed: many of them usually mean a service restarting while a second gateway, started by hand, holds its port; - the most frequent error and warning messages in the log. Requests are named by their trace id, so the agent's **Traces** page finds them. The report never shows what a person wrote. With `LIMIT` (hosts or a group from `hosts.ini`) the report covers those hosts and gets their name, so the day's report across the fleet is kept. ## Upgrade and restart `make upgrade` and `make restart` change the agents. Without `CONFIRM=yes` they only print the plan, and the last line they print is `PLAN ONLY: nothing was upgraded on any host` (or restarted): ```bash make upgrade # the plan: each host's version → target, and what is in the way make upgrade CONFIRM=yes # the upgrade make upgrade CONFIRM=yes SUDO=yes # with sudo on the hosts that need root make upgrade LIMIT=agent-ops CONFIRM=yes make restart LIMIT=agent-ops CONFIRM=yes ``` The plan covers every host at once and ends with a summary: which hosts are ready, which need `SUDO=yes`, which cannot be upgraded from here, and the exact command that upgrades the rest. Run that command. A host needs root when its binary is in a directory the SSH account cannot write (`/usr/local/bin`, where `install.sh` puts it) or its service is system-wide (`goclaw service install --system`). The plan checks whether the account has passwordless `sudo` there. If it has, `SUDO=yes` upgrades the host; if not, run `sudo goclaw update` on that host yourself, since nothing here can answer a password prompt. `SUDO=yes` uses `sudo` only on the hosts that need root, so one command covers a fleet that mixes `~/bin` and `/usr/local/bin` installs, and a `systemd --user` service is still restarted as the SSH account. With `CONFIRM=yes`, while one host in the run needs `SUDO=yes`, has no passwordless `sudo` or could not be reached, no host is upgraded: each fails with its reason, and the summary gives the command for the others. `CONFIRM=true` and other values than `yes` are refused. An upgrade installs the latest release one host at a time; `ARGS="-e target=vX.Y.Z"` installs that version instead. For each host: 1. A host already on the target version is skipped. 2. A busy host (a scheduled job running, or its log written in the last two minutes) is refused unless you add `ARGS="-e force=true"`. 3. A snapshot is saved to `backups//…_before-.gclaw`. 4. `goclaw update` downloads the release, checks its checksum, swaps the binary and restarts the service. An agent that was not running is started, and the plan says so. 5. The playbook waits until a gateway started since step 4 answers `/health`, and checks the binary is the new version. An older gateway still answering means the service did not restart, and counts as a failure. If step 4 or 5 fails and the upgrade replaced the binary, the one that ran before is put back and the service restarted. If the upgrade did not replace it (the download failed), nothing is put back, so an older `goclaw.prev` never downgrades the agent. No further host is touched. A restart, too, refuses a busy host unless you add `ARGS="-e force=true"`, which cuts off what the agent is doing. A system-wide service needs `SUDO=yes`. ## Your data `fetched/`, `backups/` and `reports/` hold the agents' real data. Treat them as you would the agents' own disks. The toolkit's `.gitignore` keeps them, and `inventory/hosts.ini`, out of any repository you keep the directory in. ================================================================================ # FILE: https://goclaw.ulab.mn/docs/user-guide/channels/index.md ================================================================================ # GoClaw Messaging Channels GoClaw supports first-class integrations with multiple enterprise messaging platforms. ## Channels - **[Enterprise App API Integration](app-api.md)**: Mobile App (iOS/Android) and internal systems (ERP/Intranet) integration with trusted identity resolution, isolated memory, and SSE streaming. - **[Flutter → Django → GoClaw Integration Assignment](flutter-django-integration.md)**: A work-assignment guide for embedding the assistant in a Flutter app through a Django proxy — config, backend client, SSE plumbing, acceptance criteria. - **[Discord Integration](discord.md)**: Bot token setup, slash commands, guild/channel overrides, threads, and rich embeds. - **[Microsoft Teams Integration](msteams.md)**: Details Azure Bot registration, Teams package generation, and single/multi-tenant setups. - **[Facebook Messenger Integration](facebook.md)**: Webhook endpoints, page access tokens, and approving the first people who write to the Page. - **[Telegram Integration](telegram.md)**: BotFather setup, long polling vs webhook, group mention handling. Long polling is the default, so no public URL is needed (Discord needs none either). - **[MS Teams Migration Guide](msteams-migration.md)**: Guidelines for migrating bot endpoints securely. - **[User Pairing & Access Control](pairing.md)**: Who may talk to the agents on these channels: pairing codes, approvals, allowlists and revoking access. ================================================================================ # FILE: https://goclaw.ulab.mn/docs/user-guide/channels/telegram.md ================================================================================ # Telegram Channel — Configuration Guide Integrate GoClaw with Telegram so users can talk to the agent in a private chat or in a group. Telegram works with **no public URL**: long polling is the default, so a gateway on `127.0.0.1` or behind NAT can run a live bot. ## Prerequisites - A Telegram account - A bot token from [@BotFather](https://t.me/BotFather) - A public HTTPS endpoint — **only** if you choose webhook mode ## 1. Create the bot 1. Open [@BotFather](https://t.me/BotFather) in Telegram 2. Send `/newbot`, then pick a display name and a username ending in `bot` 3. Copy the token it returns — it looks like `123456789:AAE...` ## 2. Configure GoClaw ```yaml channels: telegram: enabled: true token: "${TELEGRAM_BOT_TOKEN}" mode: "polling" ``` Store the token outside the config file with a secret reference (`${TELEGRAM_BOT_TOKEN}` from the environment, or `${secret:TELEGRAM_BOT_TOKEN}` from the encrypted vault). Restart the gateway. The startup log confirms the bot identity: ``` [telegram] Channel started: @your_bot (mode=polling, require_mention=true) [telegram] Long polling started (timeout=30s) ``` Verify the token independently with: ```bash goclaw doctor ``` ## 3. Talk to it Search your bot's `@username` in Telegram and send a message. The first message from an unknown user goes through the normal pairing flow — see [Pairing](pairing.md) — unless `channels.dm_policy` is `open`. The same applies in groups: an unapproved member who addresses the bot gets the pairing reply in the group. An edited message is handled as a new message, so correcting a question gets it answered again. ## Inbound modes | | `polling` (default) | `webhook` | |---|---|---| | Public URL | not needed | required (HTTPS) | | Setup | token only | token + secret + URL | | Latency | ~instant (30s long poll) | instant | | Best for | dev, NAT, single instance | production, multiple replicas | ### Webhook mode ```yaml channels: telegram: enabled: true token: "${TELEGRAM_BOT_TOKEN}" mode: "webhook" webhook_secret: "${TELEGRAM_WEBHOOK_SECRET}" webhook_url: "https://bot.example.com" ``` `webhook_secret` is **required**. Telegram echoes it back in the `X-Telegram-Bot-Api-Secret-Token` header, and GoClaw compares it in constant time; without it, anyone who discovers the hook path could inject fake messages. The channel refuses to start if it is missing. Telegram accepts 1–256 characters, only `A-Z`, `a-z`, `0-9`, `_` and `-`; with any other character `setWebhook` fails and the channel does not start. When `webhook_url` is set, GoClaw calls `setWebhook` itself on startup and mounts the receiver at `/hook/telegram` (or `/hook//telegram` when `gateway.webhook_secret_slug` is configured). Leave `webhook_url` empty to register the endpoint yourself. The registration discards messages that queued up while the gateway was down, so they are not answered after a restart. Switching back to `polling` is safe: Telegram refuses `getUpdates` while a webhook is registered, so the channel clears the stale registration on startup. ## Group chats Add the bot to a group and, by default, it stays quiet until addressed: - `@your_bot what is the deploy status?` - a reply to one of the bot's own messages Set `require_mention: false` to answer every message in the group — only sensible for a small, dedicated room. All members of a group share one conversation with the agent. > Telegram's privacy mode limits what a bot can see in groups, and it is on by > default. For `require_mention: false`, or if the bot receives nothing, send > `/setprivacy` to BotFather and choose **Disable** (or make the bot a group > admin). Telegram applies the change to a group only after the bot is removed > and added to it again. ## Attachments ### Inbound Documents, photos, voice notes, audio and video are saved to `uploads/` in the workspace (`~/.goclaw/workspace/uploads/`, or under `GOCLAW_STATE_DIR` when it is set) and referenced in the prompt, so the agent's document tools can read them and `voice_input` can transcribe voice notes. The Bot API lets a bot download files of up to 20 MB, so a larger file fails to save and the agent is told so. Filenames are sanitised before they touch disk. ### Outbound Files the agent produces are routed by type: | Condition | Method | Why | |---|---|---| | Raster image within Telegram's photo limits | `sendPhoto` | Shows an inline preview | | Everything else | `sendDocument` | Keeps the original bytes and filename | The Bot API caps `sendPhoto` at 10 MB, a width + height sum of 10000, and a 20:1 ratio. Images outside those bounds — a tall full-page screenshot, a large render — are sent as documents instead of being rejected. Dimensions are read locally before upload; if Telegram still refuses the photo, the file is retried as a document rather than lost. `sendDocument` accepts up to 50 MB; anything larger fails with an explicit error instead of a silent drop. A single file with a short caption (≤1024 characters) carries the text on the file bubble. Longer text, or text accompanying several files, is sent as its own message first. ## Formatting Replies are converted from Markdown to Telegram's HTML parse mode — more forgiving than MarkdownV2, which rejects a whole message over a single unescaped character. If Telegram still rejects the markup, the chunk is resent as plain text rather than dropped. | Markdown | Telegram | |---|---| | `**bold**`, `*italic*`, `~~strike~~` | ``, ``, `` | | `# Heading` | bold line (Telegram has no heading tag) | | `` `code` ``, ```` ```lang ```` | ``, `
` |
| `> quote` | `
` |
| `[text](url)` | `` |
| **tables** | aligned `
` block |

### Tables

Telegram has no table tag and renders messages in a proportional font, so a raw
Markdown table arrives as unreadable pipe soup. Tables are re-rendered as
fixed-width text inside `
`, where the columns line up and the block scrolls
horizontally:

```
Date   Provider  Tokens
─────  ────────  ───────
10-01  gemini    120,450
10-01  openai    18,200
```

Column alignment (`:---`, `:---:`, `---:`) is honoured, and widths are measured
in characters so Cyrillic headers align correctly. Full-width CJK text still
misaligns — one character occupies two monospace cells.

Columns only work while the table fits the screen. A phone shows roughly 40–45
monospace characters, and a table whose cells hold sentences blows well past
that — one observed reply padded out to 141 characters, where the separator rule
alone wrapped over four lines. Tables wider than that are laid out as one record
per row instead, which reads correctly at any width:

```
payments-api
Check: "error rate over the last hour"
Result: ❌ Above the threshold, 4.2% of requests failed
```

The row label is bolded and the column names label their values. Because this
layout is ordinary text rather than a `
` block, inline formatting inside
cells is converted properly; in the aligned form it is stripped, since `
`
cannot carry tags and the markers would otherwise show as literal asterisks.

### Long messages

Messages longer than 4096 **characters** (the Bot API's unit, not bytes) are
split on line boundaries. When a split lands inside a fenced code block, the
fence is closed and reopened across the boundary so both halves keep their
monospace formatting and language hint.

Pieces are sized by their **rendered** length, not by the Markdown they came
from. Telegram measures the value it receives — the HTML, tags included — so
sizing the source at 4096 overshoots on tag-heavy content: a 4090-character
source of ordinary bold-and-prose renders to 4438 characters and is rejected.
Because that rejection falls back to plain text, one oversized piece would
otherwise turn an entire reply into raw Markdown. The same rule applies to file
captions.

## Configuration reference

| Key | Default | Description |
|---|---|---|
| `enabled` | `false` | Turn the channel on |
| `token` | — | BotFather token (required) |
| `mode` | `polling` | `polling` or `webhook` |
| `webhook_secret` | — | Required in webhook mode |
| `webhook_url` | — | Public HTTPS base URL; enables auto-`setWebhook` |
| `require_mention` | `true` | In groups, only answer when addressed |
| `bot_name` | from `getMe` | Display name shown in the dashboard |
| `text_chunk_limit` | `4096` | Outbound chunk size |

## Troubleshooting

**`token check failed: telegram API 401`** — the token is wrong or was revoked.
Re-issue it with `/token` in BotFather.

**Bot is silent in a group** — it is not being addressed, or Telegram's privacy
mode is hiding messages. Mention it explicitly, or disable privacy mode via
BotFather's `/setprivacy`.

**`getUpdates failed (retry in 5s): telegram API 409: Conflict: …`** — two
processes are polling the same token. Only one gateway instance can poll a bot;
use webhook mode to run several.

**Nothing arrives in webhook mode** — check that Telegram can reach your URL and
that `webhook_secret` matches on both sides. `getWebhookInfo` reports the last
delivery error:

```bash
curl "https://api.telegram.org/bot/getWebhookInfo"
```

================================================================================
# FILE: https://goclaw.ulab.mn/docs/user-guide/channels/discord.md
================================================================================

# Discord Bot Setup

> All user, server, channel and role IDs below are synthetic. Replace them with your own IDs.

GoClaw provides full support for Discord, including direct messages, guild channels, slash commands, and file processing.

---

## Quick Setup

### 1. Create a Discord Application and Bot

1. Go to the [Discord Developer Portal](https://discord.com/developers/applications) and click **New Application**.
2. Name your application and agree to the terms.
3. Click **Bot** on the sidebar.
4. Set the bot **Username** and (optionally) an icon.

### 2. Enable Privileged Intents

GoClaw requires specific intents to see and process messages. On the **Bot** page, scroll down to **Privileged Gateway Intents** and enable:

- **Message Content Intent** (Required: to read messages)

### 3. Copy Your Bot Token

On the **Bot** page, click **Reset Token**. Copy and save the token securely. You will need it for the GoClaw configuration.

### 4. Generate an Invite URL

Click **OAuth2** on the sidebar, then **URL Generator**. Enable:

- **Scopes:** `bot`, `applications.commands`
- **Bot Permissions:**
  - View Channels
  - Send Messages
  - Read Message History
  - Embed Links
  - Attach Files
  - Add Reactions (the status reactions on each message the bot works on)
  - Create Public Threads and Send Messages in Threads (for `auto_thread` and for answering inside threads)

Copy the generated URL and open it in your browser to add the bot to your server.

### 5. Enable Developer Mode

In your Discord client: **User Settings** → **Advanced** → toggle on **Developer Mode**. This allows you to right-click on users, roles, or channels to copy their IDs — needed for allowlists and per-guild configuration.

### 6. Configure GoClaw

Edit your `~/.goclaw/config.yaml` to include the Discord section:

```yaml
channels:
  discord:
    enabled: true
    token: "YOUR_BOT_TOKEN"
```

To keep the token out of the file, use a reference: `token: "${DISCORD_BOT_TOKEN}"` reads the environment variable, and `token: "${secret:DISCORD_BOT_TOKEN}"` reads the encrypted vault. GoClaw does not read `DISCORD_BOT_TOKEN` by itself; the channel starts only when `enabled` is true and `token` is set.

### 7. Start and Pair

Start GoClaw and send a DM to your bot. If `dm_policy` is set to `pairing` (the default), it will respond with a pairing code.

Approve the request on the Web UI (**Pairing** tab). The CLI works too, but a running gateway does not see a CLI approval until it restarts — see [Pairing](pairing.md):
```bash
./goclaw pairing approve 
```

---

## Runtime Behavior

### Message Handling
- **Direct Messages (DMs):** The bot responds to all DMs from approved users.
- **Guild Channels:** By default, the bot responds only when **@mentioned** or **replied to**. The mention text is automatically stripped before the message is sent to the agent. Each person has their own conversation with the bot in each channel.
- **Threads:** Each person has their own conversation with the bot in a thread, and a thread follows its channel's settings unless it has an entry of its own (see [Threads](#threads)).

### Slash Commands
GoClaw automatically registers several slash commands:

| Command | Description |
|---------|-------------|
| `/ask ` | Ask the agent a question using a deferred reply (bypasses Discord's 3-second timeout). The question goes into your own conversation with the bot in that channel, the one your messages there are in. |
| `/memory [show\|clear]` | Sends the agent the message "Show me my memory contents. Action: show" (or `clear`) and posts its answer. The command itself clears nothing. |
| `/status` | Replies with a fixed line: "GoClaw is running. Use the web UI for detailed stats." |

Slash commands do not need an @mention. In a server they follow the same guild and channel settings as messages: a channel set to `allow: false`, or a member not on the `users`/`roles` allowlist, gets a private "You cannot use this bot in this channel." and the agent is not asked. In a DM, the DM policy (pairing) applies as it does to messages.

### Text Attachments
Discord file attachments with text-readable MIME types (e.g., `.md`, `.txt`, `.json`, `.py`, `.go`, `.js`, `.ts`, `.sql`, `.csv`, `.yaml`, `.xml`) are automatically downloaded and included in the message context. Only the first 256 KB of a text file is included.

Binary files (images, audio, PDFs) are saved to `workspace/uploads/` and the agent is given the local path to process them.

A text file larger than 10 MB, or a binary file larger than 50 MB, is not read; the agent is told it was attached and could not be read, and the rest of the message is still answered.

### Typing Indicator
A typing indicator is shown while the agent processes the message, refreshing every 8 seconds to ensure visibility.

### Message Chunking
Messages exceeding 2,000 characters are split at line breaks. Code blocks are properly closed and reopened across chunks to maintain formatting.

---

## Access Control

### DM Policy
Controlled via `channels.dm_policy` in `config.yaml`. Despite the name, it applies to guild messages too: an unapproved member who @mentions the bot in a server channel gets the pairing reply in that channel.

| Value | Behavior |
|-------|----------|
| `pairing` (default) | Unknown users receive a one-time pairing code; owner approves via Web UI or CLI |
| `allowlist` | Only user IDs listed in `channels.allow_from`, and users already approved through pairing, get an answer; no codes are issued |
| `open` | Anyone can use the bot without approval |

`allow_from: ["*"]` admits everyone. The guild settings below are checked in addition to this policy, not instead of it.

```yaml
channels:
  dm_policy: "pairing"      # pairing | allowlist | open
  allow_from:
    - "100000000000000103"   # Discord user IDs
```

---

## Guild Channel Restrictions

GoClaw offers a **per-guild** and **per-channel** access control system. This lets you:

- Require or remove the @mention requirement per channel
- Limit access to specific users or roles
- Disable the bot entirely in certain channels

### How Precedence Works

Resolution order (most specific wins):

```
per-channel config  →  per-guild config  →  global default (allow all)
```

1. **Channel-level** (`guilds..channels.`) is checked first.
2. **Guild-level** (`guilds..users` / `guilds..roles`) is the fallback.
3. If neither is configured, all users are allowed.

A channel that is not listed under `channels` follows the guild-level settings, so listing one channel does not keep the bot out of the others.

---

### Per-Guild Configuration

Under `channels.discord.guilds`, add an entry keyed by the Guild (Server) ID:

```yaml
channels:
  discord:
    enabled: true
    token: "YOUR_BOT_TOKEN"
    guilds:
      "100000000000000103":          # Guild ID (right-click server → Copy Server ID)
        require_mention: true        # true = bot must be @mentioned (default for guilds)
        users:                       # Allowlist: only these users can interact
          - "100000000000000101"
          - "100000000000000106"
        roles:                       # Allowlist: only members with these roles can interact
          - "100000000000000111"      # Role ID (right-click role → Copy ID)
        reaction_notifications: "own"  # off | own | all
```

**Guild-level fields:**

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `require_mention` | bool | `true` in guilds | Whether the bot must be @mentioned to respond |
| `users` | []string | — | Discord user IDs allowed to interact. If set, only these users are allowed |
| `roles` | []string | — | Discord role IDs allowed to interact. If set, only members with at least one listed role are allowed |
| `reaction_notifications` | string | `"off"` | Controls emoji reaction forwarding: `off`, `own` (bot's messages only), `all` |
| `channels` | map | — | Per-channel overrides (see below) |

> **Note:** If both `users` and `roles` are empty at the guild level, all users in that guild are permitted (subject to channel-level rules).
>
> A guild entry also accepts `slug`, but GoClaw does not use it.

---

### Per-Channel Configuration

Inside a guild entry, add a `channels` map keyed by Discord channel ID:

```yaml
guilds:
  "100000000000000103":
    channels:
      "100000000000000110":          # Channel ID (right-click channel → Copy ID)
        allow: true                  # true = enabled, false = disabled entirely
        require_mention: false       # override: respond to all messages, no @mention needed
        auto_thread: true            # Create a thread for each message automatically
        users:                       # Channel-level user allowlist
          - "100000000000000101"
        roles:                       # Channel-level role allowlist
          - "100000000000000108"
```

**Per-channel fields:**

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `allow` | *bool | nil (inherit) | `false` disables the bot entirely in this channel regardless of guild config |
| `require_mention` | *bool | nil (inherit guild or global) | `false` lets the bot respond to all messages without @mention; `true` enforces it |
| `auto_thread` | bool | `false` | Automatically creates a Discord thread per message |
| `users` | []string | — | Channel-specific user allowlist (overrides guild allowlist) |
| `roles` | []string | — | Channel-specific role allowlist (overrides guild allowlist) |

A channel entry also accepts `system_prompt`, but GoClaw does not use it: nothing is added to the agent's prompt.

### Threads

Inside a thread, GoClaw uses the thread's own entry under `channels` when it has one, and otherwise its channel's. A channel's `allow`, `require_mention`, `users` and `roles` therefore apply in its threads too: a channel set to `allow: false` is closed in its threads as well, and in a channel with `require_mention: false` its threads need no @mention either.

With `auto_thread: true`, the bot answers each new message in the channel in a thread it creates from that message. The conversation belongs to that thread from the first message, so the agent sees the whole exchange when the person goes on writing in the thread.

---

### Common Patterns

#### Public server — bot responds anywhere it's mentioned

```yaml
channels:
  discord:
    enabled: true
    token: "YOUR_BOT_TOKEN"
    # No guilds config needed: bot responds to @mentions everywhere by default
```

#### Answer every message in one channel, stay out of another

```yaml
channels:
  discord:
    enabled: true
    token: "YOUR_BOT_TOKEN"
    guilds:
      "100000000000000103":
        channels:
          "100000000000000110":
            allow: true
            require_mention: false   # Respond to every message in this channel
          "100000000000000101":
            allow: false             # Explicitly disable bot in this channel
```

Channels not listed here still answer @mentions. GoClaw has no "only this channel" switch: to keep the bot to one channel, list every other channel with `allow: false`, or deny the bot the **View Channels** permission in the other channels in Discord's own channel settings.

#### Allow only specific roles (e.g., "Admin", "Staff")

```yaml
channels:
  discord:
    enabled: true
    token: "YOUR_BOT_TOKEN"
    guilds:
      "100000000000000103":
        roles:
          - "100000000000000107"     # Admin role ID
          - "100000000000000108"     # Staff role ID
```

#### Per-channel role override — different access per channel

```yaml
guilds:
  "100000000000000103":
    roles:
      - "100000000000000107"         # Guild default: Admin only
    channels:
      "100000000000000110":          # This channel
        roles:
          - "100000000000000109"     # Allowed for a different role (e.g., "Support Team")
          - "100000000000000107"
```

#### Auto-thread channel

```yaml
guilds:
  "100000000000000103":
    channels:
      "100000000000000110":
        allow: true
        require_mention: false
        auto_thread: true
```

Follow-ups inside the created threads need an @mention or a reply to the bot. See [Threads](#threads) for how the conversation is split between the channel and the thread.

#### Reaction notifications — bot forwards reactions as messages

```yaml
guilds:
  "100000000000000103":
    reaction_notifications: "own"   # Only reactions on bot's own messages
```

---

## Bot Presence & Activity

Control the bot's online status and activity text:

```yaml
channels:
  discord:
    status: "online"         # online | dnd | idle | invisible
    activity: "tasks"
    activity_type: 3         # 0=Playing, 1=Streaming, 2=Listening, 3=Watching, 4=Custom, 5=Competing
```

Presence is set only when `activity` is set: without it, `status` is ignored too and the bot shows as plain online. An unset `activity_type` is `0` (Playing). Avoid `4` (Custom): Discord shows a custom status's separate state text, which GoClaw does not set.

---

## Streaming Mode

By default, the bot sends a complete reply after processing.

```yaml
channels:
  discord:
    streaming: "partial"     # off (default) | partial
```

In `partial` mode, a `_thinking…_` placeholder message is posted right away, shows the answer as it is written (updated at most every 1.5 seconds), and is replaced by the full reply when the agent finishes. In this mode the status reactions are not shown, and the reply is split at 2,000 characters whatever `text_chunk_limit` and `max_lines_per_message` say.

---

## Advanced Options

```yaml
channels:
  discord:
    enabled: true
    token: "YOUR_BOT_TOKEN"

    text_chunk_limit: 2000       # Max characters per message (Discord hard cap: 2000)
    max_lines_per_message: 17    # Soft line limit; set to avoid collapsed walls of text
    allow_bots: false            # Whether other bot messages trigger responses
    ack_reaction: ""             # Override status emoji (e.g. "✅"). Empty = animated sequence
    ack_reaction_scope: ""       # "off" turns status reactions off; any other value leaves them on
    streaming: "off"             # "off" | "partial"
    status: "online"             # online | dnd | idle | invisible (needs activity)
    activity: "tasks"            # Presence text
    activity_type: 3             # 0=Playing 1=Streaming 2=Listening 3=Watching 5=Competing
```

`ack_reaction_scope` has one effective value, `off`. `group-mentions` and `all` behave like the empty default: reactions are shown on every message the bot works on.

---

## Configuration Reference Summary

```yaml
channels:
  dm_policy: "pairing"          # pairing | allowlist | open
  allow_from:                   # User IDs for allowlist DM policy
    - "100000000000000103"

  discord:
    enabled: true
    token: "YOUR_BOT_TOKEN"
    text_chunk_limit: 2000
    max_lines_per_message: 0    # 0 = unlimited
    streaming: "off"
    allow_bots: false
    status: "online"
    activity: ""
    activity_type: 0
    ack_reaction: ""
    ack_reaction_scope: ""

    guilds:
      "":
        require_mention: true
        reaction_notifications: "off"   # off | own | all
        users:
          - ""
        roles:
          - ""
        channels:
          "":
            allow: true                 # false = disable bot in this channel
            require_mention: false
            auto_thread: false
            users:
              - ""
            roles:
              - ""
```

---

## Troubleshooting

### Bot sees no messages
- Verify **Message Content Intent** is enabled in the Discord Developer Portal.
- Restart the GoClaw gateway after changing intents.

### Guild messages not working
- Ensure the bot has **View Channels** and **Read Message History** permissions in the server.
- Remember: the bot only responds when @mentioned or replied to in guild channels by default.
- If you have a `guilds` config, check that the Guild ID is correct (right-click server icon → Copy Server ID).

### A user is being blocked unexpectedly
- Check if `users` or `roles` is set at the guild level — if so, only listed entries are allowed.
- Channel-level `allow: false` will silently block all messages in that channel.
- Channel-level `users`/`roles` overrides guild-level allowlists; ensure the user/role ID appears at the right level.
- In a thread, the parent channel's settings do not apply; the guild-level ones do (see [Threads](#threads)).
- If the bot answers with a pairing code, the user is not approved yet: `dm_policy` applies in server channels too.

### Pairing not working
- Check that DMs are enabled for your account: **User Settings** → **Privacy** → **Allow DMs from server members**.
- Verify `dm_policy` is set to `pairing` (not `allowlist`).
- List pending requests with `goclaw pairing pending discord` or on the Web UI's **Pairing** tab (they are kept in `/credentials/goclaw-pending.json`).
- If you approved with the CLI while the gateway was running, restart the gateway, or approve on the Web UI instead.

### Finding IDs (Developer Mode required)
| Object | How to copy ID |
|--------|----------------|
| Guild (Server) | Right-click the server icon in the sidebar → **Copy Server ID** |
| Channel | Right-click the channel name → **Copy Channel ID** |
| User | Right-click on a username in chat or member list → **Copy User ID** |
| Role | **Server Settings** → **Roles** → right-click a role → **Copy Role ID** |

================================================================================
# FILE: https://goclaw.ulab.mn/docs/user-guide/channels/msteams.md
================================================================================

# Microsoft Teams Bot Setup

GoClaw can run as a Microsoft Teams bot in direct messages, group chats and team channels (the generated app package declares the `personal`, `groupChat` and `team` scopes). The implementation uses Bot Framework activities, a mounted webhook path, conversation reference persistence, optional mention filtering, file consent for outbound files, and attachment handling for inbound Teams files.

## What You Need

Teams setup has two separate Microsoft resources:

| Resource | Where | Purpose |
|---|---|---|
| App Registration | Microsoft Entra ID -> App registrations | Identity, client ID, tenant ID, client secret |
| Azure Bot | Azure Portal -> Azure Bot | Registers the bot with Microsoft Bot Framework and Teams |

The Teams app manifest `botId` must match the Azure Bot Microsoft App ID. If the Azure Bot was created from an existing app registration, this is usually the app registration Application ID. If Azure created a new Microsoft App ID, use the value shown in Azure Bot -> Configuration.

For single-tenant bots, set `channels.msteams.tenant_id`: GoClaw requests its Bot Framework token from that tenant. For multi-tenant bots, leave it empty; GoClaw then uses the `botframework.com` tenant.

## Runtime Paths

GoClaw serves the Teams webhook only when the Teams channel has started: `enabled: true` with `app_id` and `app_password` set, and the token request made at startup succeeded. When the start fails, the gateway log shows `Teams channel failed: ...`; until the channel starts, the path answers 404.

| Config | Messaging endpoint |
|---|---|
| No `gateway.webhook_secret_slug` | `https:///hook/msteams` |
| With `gateway.webhook_secret_slug: "abc"` | `https:///hook/abc/msteams` |

The public URL should normally come from `gateway.ingress.url`, either through ngrok or through a reverse proxy.

## Minimal Config

```yaml
gateway:
  ingress:
    type: "ngrok"                  # "ngrok", "proxy", "nginx", or empty
    url: "https://example.ngrok.app"
    authtoken: "${NGROK_AUTHTOKEN}"
    allow_preset: "microsoft-teams"

channels:
  dm_policy: "pairing"
  msteams:
    enabled: true
    app_id: "00000000-0000-0000-0000-000000000000"      # Azure Bot Microsoft App ID
    app_password: "${secret:TEAMS_APP_PASSWORD}"
    tenant_id: "11111111-1111-1111-1111-111111111111"   # empty for a multi-tenant bot
    bot_name: "GoClaw"
    require_mention: true
    app_package:                        # shown on the app's page in Teams; see below
      developer_name: "Example LLC"
      website_url: "https://example.com"
      privacy_url: "https://example.com/privacy"
      terms_of_use_url: "https://example.com/terms"
```

`app_id` and `tenant_id` are not secrets, so plain values are simplest; `${secret:NAME}` and `${VAR}` references also work in both. Keep `app_password` (the client secret) in the vault.

Use `gateway.webhook_secret_slug` if you want the public Teams path to be harder to guess:

```yaml
gateway:
  webhook_secret_slug: "team-prod-9f4c"
```

Then set the Azure Bot messaging endpoint to:

```text
https:///hook/team-prod-9f4c/msteams
```

## Setup Flow

1. Create an App Registration in Microsoft Entra ID.
2. Create a client secret and save the secret value.
3. Create an Azure Bot resource and connect it to the app registration.
4. Enable the Microsoft Teams channel on the Azure Bot.
5. Set the Azure Bot messaging endpoint to the GoClaw Teams webhook path.
6. Generate a Teams App Package with `goclaw teams package`.
7. Upload the package in Microsoft Teams (custom app upload must be allowed in the tenant).
8. Start GoClaw and send a test message.

## Generate The Teams App Package

If `config.yaml` already contains `channels.msteams.*`, run:

```bash
goclaw teams package
```

Override values from the command line when needed:

```bash
goclaw teams package \
  --app-id="00000000-0000-0000-0000-000000000000" \
  --tenant-id="11111111-1111-1111-1111-111111111111" \
  --app-name="Ops Assistant" \
  --out="./teams-app.zip"
```

`--app-id`, `--tenant-id` and `--app-password` are also written to `config.yaml` as plain values (`--app-id` also sets `channels.msteams.enabled: true`), but only once the package has been built. The package itself needs only the App ID and the name, so leave `--app-password` out and keep the client secret in the vault. `--app-name` defaults to `channels.msteams.bot_name`, then to `GoClaw Bot`.

The zip holds three files at its root, which is where Teams looks for them:

| File | Content |
|---|---|
| `manifest.json` | Manifest version 1.17. `id` and `bots[0].botId` are both the App ID; `version` is `--app-version` (default `1.0.0`); `name.short` is the bot name cut to 30 characters; `developer` is your organization and its pages (below); `validDomains` is the host of `gateway.ingress.url`, if set. The bot is installable in the `personal`, `groupChat` and `team` scopes |
| `color.png` | 192×192 color icon |
| `outline.png` | 32×32 outline icon: white on a transparent background |

### Publisher and links

Teams shows the manifest's `developer` block on the app's details page: the publisher's name and links to its website, privacy policy and terms of use. Set them to your organization's own:

| Setting | Flag | Manifest field | Rule |
|---|---|---|---|
| `channels.msteams.app_package.developer_name` | `--developer-name` | `developer.name` | At most 32 characters |
| `channels.msteams.app_package.website_url` | `--website-url` | `developer.websiteUrl` | An `https://` URL |
| `channels.msteams.app_package.privacy_url` | `--privacy-url` | `developer.privacyUrl` | An `https://` URL |
| `channels.msteams.app_package.terms_of_use_url` | `--terms-of-use-url` | `developer.termsOfUseUrl` | An `https://` URL |

```bash
goclaw teams package \
  --developer-name="Example LLC" \
  --website-url="https://example.com" \
  --privacy-url="https://example.com/privacy" \
  --terms-of-use-url="https://example.com/terms" \
  --app-version=1.0.1
```

A flag wins over `config.yaml` and is saved there once the package is built, so the next package keeps it. `goclaw teams setup` asks for the four values too. Teams requires all four fields, so one left unset keeps GoClaw's, and the command names the ones it kept:

| Setting | GoClaw's value |
|---|---|
| `developer_name` | `Ulab LLC` |
| `website_url` | `https://goclaw.ulab.mn/` |
| `privacy_url` | `https://goclaw.ulab.mn/privacy.html` |
| `terms_of_use_url` | `https://goclaw.ulab.mn/terms.html` |

An app already in Teams shows the new links after you upload the new package with a higher `--app-version`.

### Icons

Without `--icon-source`, both icons are GoClaw's own (a white speech bubble). To use your logo:

```bash
goclaw teams package --icon-source=/path/to/logo.png
```

The source is a PNG, JPEG or GIF file, or an `http(s)` URL to one. It is fitted into both squares keeping its proportions: on a blue (`#0078D4`) background for `color.png`, and as a white silhouette for `outline.png`. The silhouette comes from the image's transparency; an image without any is cut from its background, taken to be the color of its top-left pixel. When that gives no usable shape (a photo, or an image of one color), the package keeps GoClaw's outline icon and the command says so. A logo on a transparent background gives the best outline. No external tool is needed: GoClaw resizes the image itself (`--ffmpeg` is ignored).

### Checks and versions

Before writing the zip, the command checks what Teams checks on upload, and refuses with an error instead of producing a package Teams would answer with `Bad Request`:

- the App ID is a GUID. An unresolved `${secret:NAME}` reference, or any other value, is refused; give the ID with `--app-id`;
- `--app-version` has the form `MAJOR.MINOR.PATCH`;
- the developer name has at most 32 characters, and the website, privacy and terms links are `https://` URLs;
- the icons are PNG files of exactly 192×192 and 32×32, and the outline has only white and transparent pixels.

Teams installs a changed package over an earlier one with the same App ID only when its version is higher. Raise `--app-version` (`1.0.1`, `1.0.2`, …) every time you upload a package again, for example after changing the name or the icons:

```bash
goclaw teams package --app-version=1.0.1
```

### Interactive setup

For an interactive flow, run:

```bash
goclaw teams setup
```

## Upload To Teams

For yourself, a chat or a team, in the Teams desktop or web client:

1. Select **Apps** > **Manage your apps** > **Upload an app**.
2. Select **Upload a custom app**.
3. Select `teams-app.zip`, then **Add**.
4. Select **Open** for a direct chat with the bot, or choose a team or chat to add it to.

"Upload a custom app" is shown only when the tenant allows it: a Teams administrator turns on **Upload custom apps** under Teams admin center > **Teams apps** > **Setup policies** (the Global policy, or one assigned to you), and under **Teams apps** > **Manage apps** > **Actions** > **Org-wide app settings** > **Custom apps**. A changed policy can take up to 24 hours to apply.

For the whole organization, a Teams administrator uploads the same zip in Teams admin center > **Teams apps** > **Manage apps** > **Actions** > **Upload new app**. Who can then install it follows the tenant's app permission policies.

To check a package before uploading it, import it in the Teams Developer Portal (`https://dev.teams.microsoft.com`, **Apps** > **Import app**) and open its validation: it names the field Teams objects to, where the upload in the client answers only `Bad Request`.

## Message Handling

Inbound Teams activities are handled by `internal/channel/msteams/msteams.go`.

- Outbound, the channel authenticates with the configured app ID and password.
- Inbound, every request must carry a Bot Framework token: signed by Microsoft (RS256), issued by `https://api.botframework.com`, with an audience equal to `channels.msteams.app_id` and, when the token names one, the same service URL as the activity. Any other request gets `401` and the log line `[teams] inbound auth rejected: ...`. Setting the environment variable `GOCLAW_TEAMS_SKIP_JWT=1` turns this check off, for the Bot Framework Emulator or a test with curl; never set it on a reachable gateway.
- Incoming activities are normalized into GoClaw inbound messages.
- Conversation references are saved in the gateway database on every inbound activity, so replies and proactive messages (cron jobs, SOP notices) can reach a conversation after a restart. At startup, references with no inbound activity for 30 days are deleted and the 1,000 most recent are loaded. A send to a conversation without a reference fails with `teams: no conversation reference for `: someone in that conversation has to message the bot first.
- `require_mention` defaults to `false`: the bot answers every message in a group chat or channel. With `require_mention: true` it answers only messages that mention it or reply to one of its own messages.
- Every sender, in a direct message or a group chat, passes the global channel access policy (`channels.dm_policy`: pairing, allowlist, or open). Under `pairing`, an unapproved sender's message in a group chat is answered with the pairing reply in that chat. See [Pairing](pairing.md).
- While the agent works, the bot shows a placeholder message and replaces it with the whole answer when the turn ends; the answer is not updated token by token.
- Processing uses the shared inbound handler and `ProcessStream`, so Teams keeps the same agent, memory, tool, session, and canvas behavior as other channels.

Outbound markdown tables and standalone HTTPS images are sent as Adaptive Cards (`msteams_format.go`, `msteams_adaptive.go`), because Teams does not render markdown tables in bot messages. A card table cannot scroll sideways, so each column's width is estimated from its longest word (about 8px a character plus padding), and a table of more than six columns or wider than about 520px is reshaped first:

- up to three data rows: turned sideways, one row per column with its name on the left;
- more rows: split into as few tables as fit, of evenly many columns (at most six), each repeating the first column. Seven columns with long headers become two tables of four.

Column widths follow the longest word in each column, table cards use the full message width, and at most 50 data rows are shown (a table turned sideways keeps every column). A table too big for one message — Teams refuses one over about 100 KB — is sent as several cards of at most 60 KB each. An agent or SOP needs no change; it writes an ordinary markdown table.

## File Handling

Inbound Teams attachments are handled by `msteams_attachments.go`.

- Files shared with the paperclip and pasted images are downloaded only from Microsoft hosts (Bot Framework, Teams, SharePoint/OneDrive, Azure blob storage). A download URL on any other host is refused and logged as `refusing download from disallowed host`.
- Text files (`.txt`, `.md`, `.json`, `.csv`, `.log`, `.yaml`, source code and similar, or any `text/*` content type) are put into the message the agent reads, up to the first 256 KB. They are not saved.
- Other files (PDFs, Office documents, images) are saved to `uploads/` in the gateway workspace (`~/.goclaw/workspace/uploads/`; at most 50 MB is read), and the agent is told the path.
- Audio attachments are transcribed with the configured speech-to-text provider (`voice.stt_provider`) and the transcript goes into the message; if transcription fails, the file is saved as above.

Outbound files use the Teams file consent flow in `msteams_consent.go`.

- GoClaw sends a file consent card.
- If the user accepts, Teams provides an upload URL.
- GoClaw uploads the file through that URL.

## Ingress Notes

For local testing, ngrok is the fastest option:

```yaml
gateway:
  ingress:
    type: "ngrok"
    authtoken: "${NGROK_AUTHTOKEN}"
    url: "https://my-agent.ngrok.app"
    allow_preset: "microsoft-teams"
```

For production, prefer a reverse proxy with TLS and a dedicated ingress port. See [Ingress Configuration](../ingress.md).

With the built-in ngrok tunnel, requests reach GoClaw from the local tunnel (loopback, which the allowlist always lets through), so with `trust_proxy: false` (the default) `allow_preset` does not filter them; the Teams webhook is still protected by its Bot Framework token check (below). The allowlist filters callers that connect directly, or through a reverse proxy.

Without `gateway.ingress.port`, the webhooks share the main port, and `allow_preset` / `allow_ips` then apply to that whole port: the Web UI and the app API answer `403 Forbidden` to every address outside the list, except loopback. Set `gateway.ingress.port` so the allowlist covers only the webhook listener.

The Azure Bot endpoint should never point at the WebUI-only port unless that port also serves the Teams webhook. If `gateway.ingress.port` is configured, point the proxy or tunnel to that dedicated webhook port.

## Troubleshooting

| Symptom | Check |
|---|---|
| `Bad Request` while uploading a package an earlier GoClaw built without `--icon-source` | Its placeholder icons were 1×1 and `color.png` was not a valid PNG. Build the package again with the current `goclaw teams package` |
| `Bad Request` while uploading the package | Import it in the Teams Developer Portal to see which field fails. If an app with this App ID is already in Teams, build the package again with a higher `--app-version` |
| `invalid BotId` while uploading the package | The manifest `botId` must match Azure Bot -> Configuration -> Microsoft App ID |
| `goclaw teams package` says the App ID is not a GUID or not resolved | `channels.msteams.app_id` holds something other than the Azure Bot's Microsoft App ID, or a `${secret:NAME}` the shell cannot read from the vault. Give the ID with `--app-id` |
| The app's page in Teams links to GoClaw's site instead of yours | Set `channels.msteams.app_package.*` (or the flags) and upload a new package with a higher `--app-version`: see [Publisher and links](#publisher-and-links) |
| No "Upload a custom app" in Teams | Custom app upload is off for you: see [Upload To Teams](#upload-to-teams) |
| Azure Bot test shows 404 | The endpoint path must be `/hook/msteams` or `/hook//msteams`. The path exists only when the Teams channel started: look for `Teams channel failed:` in the gateway log (wrong App ID or client secret, or no `app_password`) |
| Azure Bot test shows 401, log shows `[teams] inbound auth rejected` | `channels.msteams.app_id` must equal the Azure Bot's Microsoft App ID, which is the audience of the token Bot Framework sends. Check the host clock too (5 minutes of skew are allowed), and that the gateway can reach `login.botframework.com`, where Microsoft's signing keys are fetched |
| Web UI or app API answers `403 Forbidden` | `allow_preset` or `allow_ips` without `gateway.ingress.port` covers the whole gateway port. Set `gateway.ingress.port` |
| Group chat messages are ignored | Check `channels.msteams.require_mention` and mention the bot |
| Proactive messages fail with `teams: no conversation reference` | GoClaw holds no reference for that conversation: nobody there has messaged the bot, or not within 30 days (older references are dropped at startup). Have someone send it a message, and confirm the gateway database is writable so the reference persists |
| Attachments are not visible to the agent | Check Teams attachment type, download host, file size, and gateway logs |
| Outbound file transfer fails | Confirm the recipient accepted the Teams file consent card |

## Implementation Files

- `internal/channel/msteams/msteams.go`
- `internal/channel/msteams/msteams_attachments.go`
- `internal/channel/msteams/msteams_consent.go`
- `internal/channel/msteams/msteams_graph.go`
- `internal/cli/cmd_teams.go`
- `internal/cli/teams.go` (the package and its manifest)
- `internal/cli/teams_image.go` (the icons)
- `cmd/gateway/startup.go`

================================================================================
# FILE: https://goclaw.ulab.mn/docs/user-guide/channels/msteams-migration.md
================================================================================

# MS Teams Integration Migration Guide

In recent versions of GoClaw, the Microsoft Teams integration no longer uses the legacy **AzureBot + Local File Server** architecture, in which the agent's files were served from GoClaw's own HTTP server. Files the agent sends now go through Microsoft's **File Consent** flow, and every inbound request is checked as coming from Bot Framework.

If you configured MS Teams with the older instructions, check the steps below: a single-tenant bot needs `tenant_id`, or the Teams channel does not start.

---

## 1. What Has Changed?

*   **File Sharing:** The legacy version hosted files generated by the agent on GoClaw's own HTTP server and sent a direct link. The new version replaces this entirely, using Microsoft's official **File Consent Card** to upload the file to the user's OneDrive once the user approves. Files a user sends to the agent need no card: GoClaw downloads them directly from Microsoft.
*   **Identity:** The sender's name and ID come from the Teams activity itself. Each inbound request must carry a Bot Framework token signed by Microsoft whose audience is your bot's App ID; others are refused with `401`. GoClaw does not call the Microsoft Graph API.
*   **Replies:** GoClaw posts a placeholder message while the agent works and replaces it with the whole answer when the turn ends. An answer too long for one Teams message continues in further messages.

---

## 2. Migration Steps

You do not need to delete or recreate your existing **Azure Bot** or **App Registration**, and no Microsoft Graph API permissions are needed. Earlier versions of this guide asked for `User.Read.All` and `Files.ReadWrite.All`; GoClaw does not use them, so a tenant administrator can remove them.

### Step 1: Verify Tenant ID in `config.yaml`

A single-tenant bot needs its `Tenant ID`. GoClaw requests its Bot Framework token from the tenant in `tenant_id`; with `tenant_id` empty it uses the multi-tenant `botframework.com` endpoint, which does not issue tokens for a single-tenant app, and the Teams channel does not start (`Teams channel failed:` in the gateway log). A multi-tenant bot can leave it empty.

Check if the `tenant_id` value is filled in the `channels.msteams` section of your GoClaw `config.yaml` file.

```yaml
channels:
  msteams:
    enabled: true
    app_id: "YOUR_APP_ID"
    app_password: "YOUR_APP_PASSWORD"
    tenant_id: "YOUR_TENANT_ID" # ← required for a single-tenant bot
    bot_name: "MyGoClawBot"
```

> **Where to find the Tenant ID?**
> Go to Azure Portal -> App Registrations -> Select your App -> Overview page, and copy the **Directory (tenant) ID**.

### Step 2: Restart the Server

Once the configuration is complete, restart the GoClaw server. 

```bash
# Installed with `goclaw service install` (a per-user service, the default):
systemctl --user restart goclaw

# Installed with `sudo goclaw service install --system`:
sudo systemctl restart goclaw

# Any installed service, including launchd on macOS:
goclaw service stop && goclaw service start

# Or if you are running it directly, stop and run it again:
goclaw
```

From now on, when the agent sends you a file (an Excel report, for example), a **"File Consent Card"** asking for permission appears in Teams. Clicking "Accept" uploads the file to your OneDrive; GoClaw posts no further message after the upload. A card left unanswered for 24 hours, or across a gateway restart, has expired: clicking "Accept" then gets a reply that the file is no longer available. Files you send to the agent need no card.

---

## 3. Known Limitations

- **File Consent Cards do not work in group chats or channels.** The Microsoft Bot Framework only renders consent cards inside 1:1 DMs. If a file is sent to a group, the card will appear but users will not be able to click it—hence, this feature is only supported in DMs.
- **SSO token exchange is not implemented.** GoClaw answers a `signin/tokenExchange` invoke with HTTP 200 and a body whose `status` is `412`, so the Teams client falls back to interactive sign-in. Inbound Bot Framework requests are verified; what GoClaw does not validate is the user's SSO token, so Teams SSO cannot be used.

================================================================================
# FILE: https://goclaw.ulab.mn/docs/user-guide/channels/facebook.md
================================================================================

# Facebook Messenger Channel — Configuration Guide

Integrate GoClaw with Facebook Messenger so people can talk to an agent through a Facebook Page.

This guide follows the Meta App Dashboard as it is in 2026, where an app is built around **use cases**. If your dashboard still shows the older "Products → Messenger → Settings" layout, the same settings are there under those names.

## Prerequisites

- A Meta developer account ([developers.facebook.com](https://developers.facebook.com))
- A Facebook Page, and full control of it (the `MESSAGING` and `MODERATE` tasks on the Page)
- A public HTTPS endpoint for GoClaw with a valid certificate — a real domain or a tunnel such as ngrok. Meta does not accept self-signed certificates.

## 1. Create the app

1. Go to [developers.facebook.com/apps](https://developers.facebook.com/apps) → **Create app**.
2. Name the app and give a contact e-mail.
3. Under **Use cases**, choose **Engage with customers on Messenger from Meta**. This adds the Messenger permissions (`pages_messaging`, `pages_manage_metadata`, `pages_show_list`) to the app.
4. Connect a business portfolio if you are asked to, then **Create app**.
5. In the dashboard, open **Use cases** → **Engage with customers on Messenger from Meta** → **Customize**. The page that opens is **Messenger API Settings**; steps 2 and 5 happen there.

## 2. Connect the Page and generate a Page access token

1. In **Messenger API Settings** → **Generate access tokens**, click **Connect** (or **Add or remove Pages**).
2. Log in, select the Page, grant the permissions, and **Save**.
3. Next to the Page, click **Generate** and copy the token. Meta shows it only once.
4. Put it in `page_token` (step 4).

A token generated here does not expire on a timer. It stops working when the person who generated it loses their role on the Page, changes their password, or removes the app — generate a new one then.

## 3. Copy the app secret and choose a verify token

1. **App settings** → **Basic** → **App secret** → **Show**, and copy it into `app_secret`. GoClaw uses it twice: to check the `X-Hub-Signature-256` signature on every webhook delivery, and to add `appsecret_proof` to every Send API call. Without it, anyone who finds the webhook URL can post fake messages, so GoClaw refuses Page comments and Lead Ads submissions until it is set (Messenger messages still pass pairing or the allowlist), and says so once at start-up.
2. Choose any random string as the verify token, for example `my-secret-verify-token-2026`, and put it in `verify_token`. You enter the same string in the dashboard in step 5.

If **App settings** → **Advanced** → **Require App Secret** is on, `app_secret` is required: Meta refuses Send API calls that do not carry `appsecret_proof`.

## 4. Configure `config.yaml`

```yaml
channels:
  facebook:
    enabled: true
    page_id: ""                          # optional Page ID (auto-detected from /me if omitted)
    page_token: "${FB_PAGE_TOKEN}"       # Page access token (step 2)
    verify_token: "${FB_VERIFY_TOKEN}"   # the string you chose (step 3)
    app_secret: "${FB_APP_SECRET}"       # App secret (step 3)
    comments_enabled: false              # set true to receive Page post comments (webhook field: feed)
    auto_reply_comments: false           # set true to post the agent's reply as a public comment reply
```

Keep the values in the environment or the secret vault (`${secret:KEY}`) rather than in the file.

## 5. Start GoClaw and register the webhook

Start the gateway:

```bash
goclaw          # no subcommand — running goclaw is what starts the gateway
```

```
[facebook] Channel started (webhook: /hook/facebook, comments_enabled: false)
```

The callback URL is `https:///hook/facebook`. If `gateway.webhook_secret_slug` is set, it is `https:///hook//facebook` — use the path the log line prints.

In **Messenger API Settings** → **Configure webhooks**:

1. **Callback URL**: the URL above.
2. **Verify token**: the string from step 3.
3. **Verify and save**. Meta calls the URL; the log shows `[facebook] Webhook verified successfully`.
4. Subscribe to the webhook fields you use:
   - **`messages`** — inbound Messenger chat messages, attachments, and quick-reply payloads.
   - **`messaging_postbacks`** — button taps on templates, Get Started, Ice Breakers, and Persistent Menu items.
   - **`messaging_referrals`** — Click-to-Messenger ad referrals and `m.me?ref=...` entry links.
   - **`feed`** (under Page subscriptions) — real-time Page post comments when `comments_enabled: true`.
   - **`leadgen`** (under Page subscriptions) — Facebook Lead Ad form submissions.
   - Echoes of the Page's own messages (`message_echoes`) and comments posted by the Page itself are ignored automatically.

Then, back in **Generate access tokens**, click **Add subscriptions** next to the Page and tick the same fields (`messages`, `messaging_postbacks`, `messaging_referrals`, and optionally `feed` / `leadgen`). **This step is easy to miss and nothing arrives without it**: the webhook settings say what the app wants to hear about, and this says which Page it listens to. The same can be done with the Page token:

```bash
curl -X POST "https://graph.facebook.com/v25.0//subscribed_apps?subscribed_fields=messages,messaging_postbacks,messaging_referrals,feed&access_token="
curl "https://graph.facebook.com/v25.0//subscribed_apps?access_token="   # check
```

> **With ngrok:** run `ngrok http 8080` and use `https://xxxx.ngrok-free.app/hook/facebook` as the callback URL. A free ngrok URL changes on every restart, and the webhook has to be registered again.

## 6. Who can talk to the bot: development mode and App Review

While the app is in **development mode**, only people with a role on the app (administrators, developers, testers — **App roles** → **Roles**) get replies. Everyone else's messages never reach the webhook.

To answer the public:

1. Request **Advanced access** for `pages_messaging` (and `pages_manage_metadata`) under **App Review** → **Permissions and features**, with a screencast of the bot answering.
2. If you use the **`facebook_page`** tool for Page posts, comments, or analytics, also grant these permissions to the Page token:
   - `pages_manage_posts` — publish, schedule, update, and delete Page posts (`create_post`, `update_post`, `delete_post`).
   - `pages_read_engagement` & `pages_manage_engagement` — read comments, post public comment replies, send Comment-to-Messenger Private Replies (`private_reply`), like objects, and hide/delete comments (`moderate_comment`).
   - `read_insights` — read Page and post analytics (`page_insights`, `post_insights`: Views, Reach, Engagements, Reactions).
3. Complete **business verification** if Meta asks for it.
4. Switch the app to **Live**.

## 7. Bind an agent (optional)

```yaml
bindings:
  - agent_id: "support"
    match:
      channel: "facebook"
```

## 8. The `facebook_page` Agent Tool (Posts, Comments, Analytics & Flows)

When `channels.facebook.page_token` is configured, agents can call the `facebook_page` tool:

- **Publishing & Scheduling Posts**: `page_info`, `list_posts` (published or `scheduled: true`), `get_post`, `create_post` (text, link, remote `image_url`, or workspace `image_path`, with optional `scheduled_publish_time`), `update_post`, `delete_post`.
- **Comments & Comment-to-Messenger DM**: `list_comments`, `reply_comment` (public reply under a comment), `private_reply` (sends a direct Messenger DM to the commenter via `recipient.comment_id` within 7 days of their comment), `moderate_comment` (hide, unhide, or delete), `like_object`.
- **Page & Post Insights**: `page_insights` (`page_media_view`, `page_impressions_unique`, `page_post_engagements`, `page_daily_follows_unique`, `page_video_views` by `day`, `week`, or `days_28`) and `post_insights` (`post_media_view`, `post_impressions_unique`, `post_clicks`, `post_reactions_by_type_total`).
- **Interactive Messenger Flows**: `send_quick_replies` (up to 13 quick-reply chips), `send_template` (`button` template or `generic` carousel of up to 10 cards), `get_messenger_profile`, and `set_messenger_profile` (Welcome Greeting, Get Started payload, Ice Breakers FAQ starters, and Persistent Menu).

Administrative mutations (`create_post`, `update_post`, `delete_post`, `moderate_comment`, `page_insights`, `post_insights`, `set_messenger_profile`) require an admin or SOP context so external chat users cannot trigger them via prompt injection.

## How GoClaw talks to Messenger

- Replies use the Send API (`/me/messages`, Graph API v25.0) with `messaging_type: RESPONSE`. Meta allows that within **24 hours** of the person's last message; after that a reply is refused, and GoClaw does not send message tags.
- For Page comments (`comments_enabled: true`), the agent can reply publicly (`reply_comment` or `auto_reply_comments: true`) and send a private DM (`private_reply`) within 7 days of the comment.
- Sender display names (`first_name`, `last_name`) are resolved from the Graph API and cached in memory.
- A reply longer than 2,000 characters is sent as several messages, split between lines or at a space.
- Images, audio, video and files sent to the Page are downloaded (from Facebook's CDN only, up to 50 MB) and handed to the agent.
- Files the agent produces are sent as Messenger attachments: an image, video or audio attachment by the file's type, anything else as a file.
- While the agent works, the person sees the typing indicator; it is renewed every 15 seconds.
- Repeated deliveries of the same message, postback, or comment are processed once.

## Comments, leads and `dm_policy`

`channels.dm_policy` and pairing are for people who write to the Page in Messenger. A comment under a Page post and a Lead Ads submission are events from the Page, not a person asking to talk to the agent, so they do not go through pairing or the allowlist:

| Event | Reaches the agent when | Runs as |
|---|---|---|
| Messenger message, postback, referral | `dm_policy` lets the sender in (see [Pairing](pairing.md)) | the paired user, with their role |
| Page post comment (`feed`) | `app_secret` is set and `comments_enabled: true`, under any `dm_policy` | an ordinary `user`, with no paired identity |
| Lead Ads submission (`leadgen`) | `app_secret` is set and the Page is subscribed to `leadgen`, under any `dm_policy` | an ordinary `user`, with no paired identity |

- A comment or lead never creates a pairing request, and the access gate's text ("Pairing Required" with a code, or "You are not authorized to use this bot.") is never posted as a comment reply.
- A comment runs as an ordinary user even when the commenter's Facebook ID belongs to a paired admin: the comment is public, and anyone can write one.
- With `comments_enabled: false`, comments are not handled at all, whatever `dm_policy` is.
- Without `app_secret`, comments and leads are not handled at all: they skip the access gate, and only the signature proves that Facebook sent them.
- Instructions and hooks limited to `roles: [user]` apply to comments and leads as they do to paired users. To write an instruction for comments or leads alone, give it `channels: [facebook]` and a `message_match` on `Facebook Page Comment` or `Facebook Lead Ad Submission`, the first line of what the agent receives for each.

## Testing

From an account with a role on the app (step 6), send a message to the Page. The log shows `[facebook] Message from : …`.

With the default `channels.dm_policy: pairing`, the first reply to a Messenger message is **Pairing Required** with an 8-character code and the sender's Facebook ID (their page-scoped ID, PSID). Approve the code on the Web UI's **Pairing** tab — see [Pairing](pairing.md) — and send the message again; this time the agent's reply arrives in Messenger.

## Troubleshooting

| Symptom | Cause and fix |
|---|---|
| "The URL couldn't be validated" when saving the webhook | GoClaw is not reachable over HTTPS at that URL, the path is wrong (check the slug), or `verify_token` differs. The log says `Webhook verification failed` when the token is the problem. |
| Webhook verified, but no message ever arrives | The Page is not subscribed: **Add subscriptions** in step 5. Or the sender has no role on an app in development mode (step 6). |
| `Invalid signature` in the log | `app_secret` is not this app's secret. |
| Comments or leads never reach the agent, and the start-up log says `Page events (comments and lead ads) are refused` | `app_secret` is not set (step 3). |
| Send fails with error 190 (`OAuthException`) | The Page token is no longer valid — generate a new one (step 2). |
| Send fails with "API calls from the server require an appsecret_proof argument" | **Require App Secret** is on and `app_secret` is not set. |
| Send fails with error 10 / 2018278 | The reply came more than 24 hours after the person's last message. |
| Send fails with a permissions error (code 200) | The token lacks `pages_messaging`, or the app has no Advanced access and the person has no role on it. |
| Messages stop after the server was down | Meta retries for a while; after about an hour of failures it disables the webhook for the Page and sends a "Webhooks Disabled" alert. Fix the endpoint, then subscribe again (step 5). |

================================================================================
# FILE: https://goclaw.ulab.mn/docs/user-guide/channels/app-api.md
================================================================================

# Enterprise App API Channel — Mobile App & Internal Systems Integration

The **App API Channel (`app_api`)** allows enterprise organizations to embed GoClaw AI agents directly into their **Mobile Applications (iOS, Android, Flutter, React Native)** and **Internal Systems (ERP, Intranet, CRM, Helpdesk)**.

It provides **Trusted Identity Resolution** (eliminating the need for manual 8-character pairing codes), **Per-Employee Memory Isolation** (`user_.md`), **SSE streaming** of replies plus an SSE event stream for agent-initiated messages, and **Role-Based Agent Routing**.

---

## Key Capabilities

1. **No Manual Pairing Challenges:** Employees already authenticate against your company's identity provider (SSO / Keycloak / Active Directory / App Auth). GoClaw trusts authenticated headers or signed JWTs from your backend.
2. **Per-User Memory Isolation:** Each employee has their own private conversation history and personal memory (records of scope `personal` in the state database, keyed by the employee's user ID and shared by every agent). Company-wide RAG and SOPs are shared, but personal notes/preferences remain completely private.
3. **Multi-Agent Routing by Role/Dept:** Route requests automatically (e.g. `role: "engineer"` → DevOps Agent, `role: "accountant"` → Financial Analysis Agent).
4. **Real-time Streaming:** Native Server-Sent Events (`text/event-stream`) for responsive typing animations.
5. **Agent-initiated Messages:** A long-lived event stream (`GET /api/app/events`) delivers what the agent says outside a reply — cron reports, SOP results, finished delegations.

The API carries text only. A file the agent sends to an app user arrives on the event stream as its name (`[Attachments: report.xlsx]`), and the chat response has no file field.

---

## Security Model — read before enabling

The `app_api` routes (`/api/app/*`, `/ws/app`) are **exempt from the gateway dashboard token** (`gateway.auth.token`), because they authenticate their own callers with the app credential below. Three consequences follow:

1. **A credential is mandatory.** With `auth_mode: "api_key"` and no `api_key` — or `auth_mode: "jwt"` and no `jwt_secret` — the channel refuses to start and every request is answered `503`. There is no anonymous mode.
2. **`api_key` mode trusts the `X-User-*` headers.** Whoever holds the key may present themselves as any employee, and each employee's identity selects their private memory file and session. **Do not ship the key inside a mobile app binary** — put it in your own backend and have the app talk to that, or use `auth_mode: "jwt"` for clients you cannot trust with a shared secret.
3. **JWT mode is verified, not decoded.** Tokens must be `HS256`, signed with `jwt_secret`, and carry an `exp` claim; expired tokens, wrong signatures and `alg: none` are rejected. A 60-second clock skew is tolerated. An optional `nbf` claim is honoured.

Requests are rate limited per client IP **and** per user ID (`rate_limit_per_min`, default 60/min each); over-budget callers get `429` with a `Retry-After` header. The client IP is the connecting address, or the last `X-Forwarded-For` entry with `gateway.rate_limit.trust_proxy: true`.

The gateway-wide limit applies on top of that: `gateway.rate_limit.per_minute` (default 60 a minute) per client address, for every request that does not carry the gateway token, so app API requests count against it. It is keyed by the connecting address, or by the last `X-Forwarded-For` entry with `gateway.rate_limit.trust_proxy: true`. Behind a proxy, all callers share the proxy's address, so raise `per_minute` to cover the proxy's peak load. Over budget, it answers `429` with `{"error":"rate limit exceeded","retry_after":60}` and `Retry-After: 60`.

---

## Getting the key

There is nothing to request and nothing to click: `api_key` is a **pre-shared
secret you generate yourself**, exactly like a password you choose for two of
your own systems. GoClaw never issues it, and no wizard creates one — the
channel only compares what a caller presents against what your config holds.

GoClaw can generate and store one for you:

```bash
goclaw secret gen APP_GATEWAY_API_KEY --provider app_api
```

It prints the value once — hand that to the caller — and keeps it in the
encrypted vault. Re-read it later with `goclaw secret get APP_GATEWAY_API_KEY`.
Regenerating over an existing name needs `--force`, because the callers still
presenting the old value will start failing.

If you would rather produce the value yourself, any CSPRNG will do
(`openssl rand -hex 32`) and `goclaw secret set  ` stores it.

Then give it to GoClaw in **one** of two ways. Both are referenced from
`config.yaml` with `${...}`, but they are different mechanisms:

| In `config.yaml` | Where the value lives | When to use it |
|---|---|---|
| `api_key: "${secret:APP_GATEWAY_API_KEY}"` | GoClaw's encrypted vault (AES-256-GCM, machine-specific), written with `goclaw secret set` | Default. The value never appears in the config file or in the process environment. |
| `api_key: "${APP_GATEWAY_API_KEY}"` | An environment variable on the gateway host | Containers and orchestrators that already inject secrets as env vars (Kubernetes, systemd `EnvironmentFile`, Docker secrets). |

Into the vault:

```bash
goclaw secret set APP_GATEWAY_API_KEY "$(openssl rand -hex 32)"
goclaw secret list                      # confirm it is there (values masked)
```

The command is `secret`, singular, and it takes the key **and** the value.

On a production host also set `GOCLAW_SECRET_KEY` before writing anything to the
vault. Without it the vault passphrase is derived from machine and user
identifiers, which the gateway warns about at startup; the encryption then
offers little protection against someone who already has access to the host.

The same string then goes to whatever calls GoClaw — your backend proxy, not a
shipped app. Both forms are resolved when the config is loaded, so a rotated
value takes effect on the next config reload or gateway restart, not the moment
you write it; callers still presenting the old key then start getting `401`.

> `jwt` mode has the same shape: `jwt_secret` is a shared secret —
> `goclaw secret gen APP_JWT_SECRET` works for it too.

### One key per calling system

A single shared key means a leak forces every integration onto a new secret at
once, and an accepted request cannot be attributed to the system that made it.
`api_keys` issues one credential each:

```yaml
channels:
  app_api:
    enabled: true
    auth_mode: "api_key"
    api_keys:
      - name: "django-proxy"
        key: "${secret:APP_KEY_DJANGO}"
      - name: "erp"
        key: "${secret:APP_KEY_ERP}"
      - name: "mobile-bff"
        key: "${secret:APP_KEY_MOBILE}"
        disabled: true          # suspended without losing the entry
```

```bash
goclaw secret gen APP_KEY_DJANGO --provider app_api
goclaw secret gen APP_KEY_ERP    --provider app_api
```

- **`name` is required** and identifies the caller. It appears in the gateway
  log (`[app_api] client=erp user=emp_10482 agent=main`) and as a `client:`
  tag on the trace, so you can see which system called. It is never returned to
  the caller.
- **Revoke one by removing its entry**, or set `disabled: true` to suspend it
  and keep the record. The others keep working.
- **`api_key` and `api_keys` can both be set.** Every one of them is accepted,
  so an existing deployment migrates one caller at a time; requests on the old
  single key are attributed to `client=api_key`.
- Keys are compared in constant time and every candidate is checked even after a
  match, so response timing does not reveal which credential a probe came close
  to — or which callers exist.

---

## Configuration (`config.yaml`)

```yaml
channels:
  app_api:
    enabled: true

    # REQUIRED for auth_mode="api_key": either this single key, or at least one
    # enabled entry under api_keys. See "Getting the key" above.
    # "${secret:NAME}" reads GoClaw's encrypted vault; "${NAME}" reads an
    # environment variable on the gateway host.
    api_key: "${secret:APP_GATEWAY_API_KEY}"

    # Preferred for more than one caller — one credential per system, revocable
    # on its own, and the name is logged with each accepted request.
    # api_keys:
    #   - name: "django-proxy"
    #     key: "${secret:APP_KEY_DJANGO}"

    # auth_mode: "api_key" (default) or "jwt"
    auth_mode: "api_key"

    # REQUIRED for auth_mode="jwt". HMAC secret used to verify HS256 tokens.
    # jwt_secret: "${APP_JWT_SECRET}"

    # Register each authenticated employee as a GoClaw user (at the "user" tier, in
    # ~/.goclaw/credentials/goclaw-users.json) on first contact. Nothing else is
    # created: personal memory is stored as database records when the agent saves
    # something about that person. Default: true
    auto_provision_memory: true

    # Default role if none supplied in headers. Default: "user"
    default_role: "user"

    # CORS allowlist for browser callers (intranet portals). Requests from a
    # browser origin that is not listed are refused with 403. Non-browser
    # clients (mobile apps, servers, curl) send no Origin and are unaffected.
    # An entry is an exact origin, "*.company.com" or "https://*.company.com"
    # (any subdomain and the domain itself; with a scheme, only that scheme),
    # or "*" for any origin.
    allowed_origins:
      - "https://portal.company.internal"
      - "http://localhost:3000"

    # Enforced per client IP and per user ID (default: 60)
    rate_limit_per_min: 60

# Route different employee roles to specialized agents
bindings:
  - agent_id: "achilles"
    match:
      channel: "app_api"
      platform_roles: ["engineer", "devops"]

  - agent_id: "odysseus"
    match:
      channel: "app_api"
```

---

## API Reference

### Which URL to call

`/api/app/chat` and `/api/app/events` are served on GoClaw's **main port** — the
same one the Web UI answers on — not on `gateway.ingress.port`, which carries
only the webhook routes. (`/ws/app` is retired; see below.)

Two settings decide whether a caller can reach it at all:

| Setting | Default | Effect |
|---|---|---|
| `server.port` | `8080` | The port. The `8080` in the examples below is this default, not a fixed value — check the config of the deployment you are calling. |
| `gateway.bind` | `loopback` | With the default, GoClaw listens on `127.0.0.1` only and is unreachable from another host. Use `lan` to listen on all interfaces, or keep loopback and put a reverse proxy in front. |

The gateway prints both at startup: `GoClaw gateway starting on 127.0.0.1:8080`.

`/health` needs no credential, so it settles in one command whether you are
talking to GoClaw at all:

```bash
curl -s http://:/health
# {"status":"ok","uptime_seconds":...}
```

If that returns HTML, an nginx page, or nothing, the address is wrong or
something else is answering on that port — no API key will fix it.

### 1. REST Chat Endpoint (`POST /api/app/chat`)

#### Request Headers:
| Header | Required | Description |
|---|---|---|
| `Authorization` | Yes | `Bearer ` in `api_key` mode, `Bearer ` in `jwt` mode. This is the **app credential, not** `gateway.auth.token` — these routes do not take the dashboard token. `X-API-Key: ` works instead. |
| `Content-Type` | Yes | `application/json` |
| `X-User-ID` | Yes* | Unique employee / user identifier (e.g. `emp_10482`, `usr_abc`) |
| `X-User-Name` | Optional | Display name of the employee (e.g. `Alice`) |
| `X-User-Role` | Optional | Business role, used for agent routing: with `X-User-Dept` it forms the `platform_roles` that `bindings` match (e.g. `engineer`, `manager`). It grants no GoClaw privileges: every app API request runs at GoClaw's `user` tier, whatever this header says. |
| `X-User-Dept` | Optional | Department (e.g. `infrastructure`, `finance`) |
| `X-Tenant-ID` | Optional | Organization or branch ID (e.g. `hq`, `branch_ub`) |

*\* When using `auth_mode: "jwt"`, identity fields come from the verified JWT claims (`sub`, `name`, `email`, `role`, `dept`, `tenant`) and the `X-User-*` headers are ignored. A request with no resolvable user ID is rejected with `401` — there is no anonymous or IP-derived identity.*

#### Error Responses:
| Status | Meaning |
|---|---|
| `404` | **The request never reached this endpoint.** The route is registered whenever the gateway runs, so a 404 is not about the channel or the key — it is the wrong host, the wrong port, or a reverse proxy that does not forward `/api/app/*`. GoClaw's own 404 is the plain text `404 page not found`; an HTML body means something else answered. Check `/health` first. |
| `400` | The body is not valid JSON (`malformed JSON request`), or `message` is empty (`message is required`) |
| `401` | Missing/invalid credential, or no user identity in the request. Note this endpoint does **not** accept `gateway.auth.token` — the dashboard token is a different credential; use `channels.app_api.api_key` or an `api_keys` entry. |
| `403` | Browser `Origin` is not in `allowed_origins`. A plain-text `Forbidden` instead comes from the ingress allowlist (`gateway.ingress.allow_preset` / `allow_ips`), which covers the main port when `gateway.ingress.port` is not set |
| `405` | Wrong method: the chat endpoint takes `POST`, the event stream `GET` |
| `408` | `session busy or request cancelled`: an earlier request on the same session was still running and the caller gave up waiting. Requests on one session run one at a time |
| `429` | Over `rate_limit_per_min` for this IP or user, or over the gateway-wide limit; see `Retry-After` |
| `500` | Processing failed (usually the agent or its model provider); `error` says why |
| `503` | Channel disabled, or enabled with no credential configured. On `/api/app/events` also `app_api channel is not running` (see the event stream notes) |

Errors from these endpoints are JSON: `{"error": "..."}`. The plain-text 404 and ingress 403 come from in front of them.

#### Request Body:
```json
{
  "message": "Can you check the server metrics for today?",
  "stream": true,
  "session_id": "optional-custom-session-id"
}
```

`session_id` is optional and always scoped to the caller: GoClaw trims it, cuts it to 128 characters and builds the session key `app_api:/`, both parts URL-escaped (`app_api:emp_10482/optional-custom-session-id` here). Without one the key is `app_api:`. One employee can therefore hold several conversations, and no session ID reaches another employee's conversation. Requests on the same session run one after another.

---

### 2. Streaming Response (SSE — `stream: true`)

`Content-Type: text/event-stream`

```
data: {"delta":"Certainly","event":"token","session_id":"app_api:emp_10482"}

data: {"delta":", checking","event":"token","session_id":"app_api:emp_10482"}

data: {"event":"tool_start","tool":"uptrace_query","session_id":"app_api:emp_10482"}

data: {"event":"tool_result","tool":"uptrace_query","session_id":"app_api:emp_10482"}

data: {"delta":" CPU usage is at 24%.","event":"token","session_id":"app_api:emp_10482"}

data: {"done":true,"event":"done","session_id":"app_api:emp_10482"}
```

If the agent fails, an error frame arrives, followed by `done` while the client is still connected:

```
data: {"event":"error","error":"...","session_id":"app_api:emp_10482"}
```

---

### 3. Synchronous Response (`stream: false`)

`Content-Type: application/json`

```json
{
  "response": "Certainly, CPU usage is at 24% and all services are healthy.",
  "session_id": "app_api:emp_10482",
  "user_id": "emp_10482",
  "user_name": "Alice",
  "role": "engineer",
  "prompt_tokens": 182,
  "output_tokens": 45,
  "elapsed_ms": 780
}
```

---

### 4. Event Stream Endpoint (`GET /api/app/events`)

`POST /api/app/chat` answers one question and closes. Anything the agent says on
its own — a cron report, an SOP result, a subagent finishing a delegated task —
has no reply to ride along with, so it needs somewhere to go. That is this
endpoint: one long-lived SSE stream per client, held open alongside the per-turn
chat calls.

```
GET /api/app/events
Authorization: Bearer 
X-User-ID: emp_10482
```

Browsers cannot set headers on an `EventSource`, so the credential and identity
may travel as query parameters instead — the same allowance the WebSocket had.
Native clients should prefer headers; query strings end up in proxy and access
logs.

```
https://goclaw.company.internal/api/app/events?token=&user_id=emp_10482
```

Frames are the same `StreamChunk` JSON objects as the chat SSE:

```
data: {"event":"ready","session_id":"app_api:emp_10482"}

: keepalive

data: {"event":"message","delta":"The morning report is ready","session_id":"emp_10482","done":true}
```

Notes:

- The `ready` frame is the handshake: once it arrives, the client is registered
  and proactive messages will be delivered.
- A comment line (`: keepalive`) is sent every 30s so a dead proxy surfaces as a
  disconnect instead of a silent hang. Every SSE client ignores it.
- One employee may hold several streams open (phone, tablet, desktop); each
  receives a copy.
- Cross-origin browser callers must have their origin listed in
  `allowed_origins`.
- If no stream is open for a user, a proactive send reports non-delivery to the
  agent — the message is not queued for later.
- A client too slow to drain its stream loses messages rather than blocking the
  cron run or SOP that produced them.
- The stream needs the `app_api` channel to have started with the gateway. If the
  channel was enabled (or its credential fixed) by a config reload, the stream
  answers `503` `app_api channel is not running` until the gateway restarts;
  `POST /api/app/chat` works without the restart.

Reconnect with the usual `EventSource` backoff; the stream carries no per-turn
state, so reopening it is always safe.

---

### 5. WebSocket Endpoint (`GET /ws/app` — DEPRECATED)

> [!WARNING]
> **Deprecated (HTTP 410 Gone):** The WebSocket endpoint `/ws/app` has been deprecated in favor of on-demand Server-Sent Events (SSE) via `POST /api/app/chat` (`stream: true`). 
> WebSocket keepalive timeouts and socket concurrency locks under heavy LLM processing turns (>30s) previously caused request dropping and agent deadlocks.
> All client integrations (mobile and web) MUST use `POST /api/app/chat` with SSE.

Connecting to `/ws/app` now returns `HTTP 410 Gone` with instructions to migrate.

Migration map:

| Old (`/ws/app`) | New |
|---|---|
| Sending `{"message": "..."}` over the socket | `POST /api/app/chat` with `stream: true` |
| Receiving turn output over the socket | SSE response of that same POST |
| Receiving agent-initiated (proactive) messages | `GET /api/app/events` |

---

## Client Integration Examples

### cURL Example
`8080` here is the default `server.port` — substitute the port the target
deployment actually listens on.

```bash
curl -X POST http://localhost:8080/api/app/chat \
  -H "Authorization: Bearer $APP_GATEWAY_API_KEY" \
  -H "X-User-ID: emp_10482" \
  -H "X-User-Name: Alice" \
  -H "X-User-Role: engineer" \
  -H "Content-Type: application/json" \
  -d '{
    "message": "Hello Agent!",
    "stream": false
  }'
```

### TypeScript / Node.js (backend proxy)

Runs on your server, where the shared key belongs. Your mobile app calls this, not GoClaw directly.

```typescript
async function askGoClaw(employeeId: string, employeeName: string, role: string, message: string) {
  const response = await fetch("https://goclaw.company.internal/api/app/chat", {
    method: "POST",
    headers: {
      "Authorization": `Bearer ${process.env.APP_GATEWAY_API_KEY}`,
      "X-User-ID": employeeId,
      "X-User-Name": employeeName,
      "X-User-Role": role,
      "Content-Type": "application/json"
    },
    body: JSON.stringify({ message, stream: false })
  });
  return await response.json();
}
```

> **Running Flutter + Django?** A step-by-step assignment for connecting this
> channel to a Flutter app through a Django proxy is in
> [flutter-django-integration.md](flutter-django-integration.md).

### Flutter / Dart (SSE Stream)

> The snippet below points at your **own backend proxy**, which adds the
> `Authorization` and `X-User-*` headers server-side. Do not embed
> `APP_GATEWAY_API_KEY` in a shipped app — see [Security Model](#security-model--read-before-enabling).

```dart
import 'dart:convert';
import 'package:http/http.dart' as http;

Stream streamChat(String empId, String empName, String message, String backendSessionToken) async* {
  final request = http.Request('POST', Uri.parse('https://api.company.internal/goclaw/chat'));
  request.headers.addAll({
    // Session token issued by your backend, which holds the GoClaw key.
    'Authorization': 'Bearer $backendSessionToken',
    'X-User-ID': empId,
    'X-User-Name': empName,
    'Content-Type': 'application/json',
  });
  request.body = jsonEncode({'message': message, 'stream': true});

  final response = await request.send();
  await for (final line in response.stream.transform(utf8.decoder).transform(const LineSplitter())) {
    if (line.startsWith('data: ')) {
      final jsonStr = line.substring(6).trim();
      if (jsonStr.isNotEmpty) {
        final data = jsonDecode(jsonStr);
        if (data['delta'] != null) {
          yield data['delta'];
        }
      }
    }
  }
}
```

================================================================================
# FILE: https://goclaw.ulab.mn/docs/user-guide/channels/flutter-django-integration.md
================================================================================

# Work Assignment — Flutter → Django → GoClaw AI Assistant Integration

**Version:** 1.0
**Date:** 2026-08-28
**Scope:** Add an AI assistant to a Flutter mobile app. The Django backend proxies to GoClaw's `app_api` channel.

This guide adds an AI assistant to a Flutter mobile app: the app talks to its own Django backend, and Django proxies each request to GoClaw's `app_api` channel, so the gateway key never ships inside the app.

> This is an implementation guide. The full reference for the GoClaw side of the API is in
> [app-api.md](app-api.md); this document builds on it.

---

## 1. Architecture

```mermaid
flowchart LR
    A["Flutter App
(iOS / Android)"] -->|"HTTPS + the user's
session token"| B["Django Backend"]
    B -->|"HTTPS
Authorization: Bearer APP_KEY
X-User-ID, X-User-Role..."| C["GoClaw Gateway
POST /api/app/chat"]
    C --> D["LLM + Tools + Memory"]
    D --> C
    C -->|"SSE: token / tool_start / done"| B
    B -->|"SSE relay"| A
```

### Why Django has to sit in the middle

This is the key architectural decision; do not skip it:

1. **Never ship `APP_GATEWAY_API_KEY` inside the app.** With `auth_mode: "api_key"`, whoever holds the key can act as **any employee** through the `X-User-ID` header. Unpacking the app binary is enough to get the key.
2. **Only Django knows who the user really is.** Django authenticates the user with its own session/JWT, then adds `X-User-ID` for GoClaw **on the server side**. The app never chooses this value.
3. **Auditing, rate limiting and logging are centralised in Django.**

---

## 2. Prerequisites

| Item | Requirement |
|---|---|
| GoClaw | A running gateway (`goclaw doctor` clean) |
| Django | ≥ 5.0, in **ASGI** mode (uvicorn / daphne). WSGI does not suit SSE.
5.0 is required for `request.auser()` |
| Python | `httpx` (async streaming) |
| Flutter | The `http` or `dio` package |
| Network | A direct connection from Django to GoClaw (same VPC / internal DNS) |

---

## 3. Layer 1 — GoClaw configuration

### 3.1 `config.yaml`

```yaml
channels:
  app_api:
    enabled: true

    # The secret key shared with Django. Required —
    # without it the channel does not start and every request returns 503.
    api_key: "${secret:APP_GATEWAY_API_KEY}"

    auth_mode: "api_key"

    # Registers each employee as a GoClaw user on first contact. Their personal
    # memory is stored as database records when the agent saves something about
    # them.
    auto_provision_memory: true

    default_role: "user"

    # Requests from Django carry no Origin header, so this list has no
    # effect in this architecture. It matters only for direct browser calls.
    allowed_origins: []

    # ⚠ Read section 4.4 — this value matters behind a Django proxy
    rate_limit_per_min: 600
```

Create the key yourself and put it in the vault. GoClaw does not issue keys —
this is a secret of your own choosing, shared between your two systems:

```bash
goclaw secret gen APP_GATEWAY_API_KEY --provider app_api
goclaw secret list                      # the value is shown masked
```

`secret gen` creates a random value, puts it in the vault and prints it once —
set it on the Django side. You can read it again later with
`goclaw secret get `. If you prepared a value yourself, use
`goclaw secret set  `.

When several systems connect (Django, ERP, mobile), give each its own key —
see the `api_keys` list in
[app-api.md — One key per calling system](app-api.md#one-key-per-calling-system).
For details, see [app-api.md — Getting the key](app-api.md#getting-the-key).

### 3.2 Routing each role to a different agent (optional)

```yaml
bindings:
  - agent_id: "devops-agent"
    match:
      channel: "app_api"
      platform_roles: ["engineer", "devops"]

  - agent_id: "main"
    match:
      channel: "app_api"
```

`platform_roles` is built automatically from `X-User-Role` + `X-User-Dept`.

### 3.3 Confirming the configuration works

```bash
# 8080 is the default `server.port` — check the port of your own
# installation. `gateway.bind` defaults to loopback, so reaching it from
# outside needs `bind: lan` or a reverse proxy in front.
# First of all: curl -s http://GOCLAW_HOST:PORT/health
curl -s -X POST http://GOCLAW_HOST:8080/api/app/chat \
  -H "Authorization: Bearer $APP_GATEWAY_API_KEY" \
  -H "X-User-ID: emp_test_1" \
  -H "X-User-Name: Test" \
  -H "Content-Type: application/json" \
  -d '{"message":"Hello","stream":false}' | jq
```

Expected response:

```json
{
  "response": "Hello! ...",
  "session_id": "app_api:emp_test_1",
  "user_id": "emp_test_1",
  "user_name": "Test",
  "role": "user",
  "prompt_tokens": 182,
  "output_tokens": 45,
  "elapsed_ms": 780
}
```

**Acceptance criterion A1:** the curl above returns 200. The log shows `[app_api] Enterprise App & Mobile API channel enabled`.

---

## 4. Layer 2 — Django

### 4.1 Settings

```python
# settings.py
import os

GOCLAW_BASE_URL = os.environ["GOCLAW_BASE_URL"]        # http://goclaw.internal:8080
# GoClaw itself serves plain HTTP; use an https:// URL only for a TLS proxy in front of it.
GOCLAW_API_KEY  = os.environ["GOCLAW_API_KEY"]         # the same as APP_GATEWAY_API_KEY
GOCLAW_TIMEOUT_READ = float(os.environ.get("GOCLAW_TIMEOUT_READ", "180"))
```

Do not write `GOCLAW_API_KEY` into the code; read it from `.env` or a secret manager.

### 4.2 Thread model

Each user must be able to hold several conversations (threads). The GoClaw session key is:

```
app_api:                    ← when no session_id is sent
app_api:/       ← when one is sent
```

Both parts are URL-escaped, and the `session_id` is trimmed and cut to 128
characters. Conversations stored by earlier releases under
`app_api::` are carried over the first time they are used.

```python
# assistant/models.py
import uuid
from django.conf import settings
from django.db import models


class AssistantThread(models.Model):
    id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)
    user = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE,
                             related_name="assistant_threads")
    title = models.CharField(max_length=200, blank=True)
    created_at = models.DateTimeField(auto_now_add=True)
    updated_at = models.DateTimeField(auto_now=True)

    @property
    def goclaw_session_id(self) -> str:
        # The session_id passed to GoClaw. GoClaw keeps the first 128
        # characters (a UUID has 36), so it must be stable and unique within them.
        return str(self.id)


class AssistantMessage(models.Model):
    ROLE_CHOICES = [("user", "user"), ("assistant", "assistant")]

    thread = models.ForeignKey(AssistantThread, on_delete=models.CASCADE,
                              related_name="messages")
    role = models.CharField(max_length=16, choices=ROLE_CHOICES)
    content = models.TextField()
    created_at = models.DateTimeField(auto_now_add=True)

    class Meta:
        ordering = ["created_at"]
```

> **Note:** GoClaw keeps its own conversation history. Storing it again in Django
> is only for showing and searching history in the app's UI. The two histories may
> drift apart — treat GoClaw as the source of truth.

### 4.3 Identity headers

```python
# assistant/goclaw.py
from django.conf import settings


def user_identity_headers(user, *, client_ip: str | None = None) -> dict[str, str]:
    headers = {
        "Authorization": f"Bearer {settings.GOCLAW_API_KEY}",
        "Content-Type": "application/json",

        # A stable ID that is never reused. This value selects the employee's
        # PERSONAL memory file, so it must never change or be reused.
        # Use a UUID field rather than the AutoField pk.
        "X-User-ID": str(user.profile.public_uid),
        "X-User-Name": user.get_full_name() or user.get_username(),
        "X-User-Email": user.email or "",
        "X-User-Role": resolve_role(user),          # "engineer" | "manager" | ...
        "X-User-Dept": resolve_department(user),    # "infrastructure" | ...
        "X-Tenant-ID": resolve_tenant(user),        # branch / organisation
    }
    if client_ip:
        # See 4.4 — this value decides the rate-limit IP bucket
        headers["X-Forwarded-For"] = client_ip
    return {k: v for k, v in headers.items() if v}
```

### 4.4 ⚠ Rate limits — the most common mistake

GoClaw counts app API requests in three buckets:

* **per IP** and **per user_id**, each at `channels.app_api.rate_limit_per_min` (default **60/min**);
* the **gateway-wide** limit, `gateway.rate_limit.per_minute` (default **60/min**) per connecting address. It counts every request that does not carry the gateway token, app API requests included.

Behind a Django proxy every employee arrives from **one IP**: both the per-IP bucket and the
gateway-wide limit are keyed by the connecting address, unless `gateway.rate_limit.trust_proxy`
is `true`. Then both take the last `X-Forwarded-For` entry. So:

* With the defaults → the whole company shares 60 requests/min, then everyone gets `429`.
* The fix:
  1. Django passes the end user's IP in `X-Forwarded-For`.
  2. When Django connects to GoClaw directly and nothing else can reach GoClaw's port, set
     `gateway.rate_limit.trust_proxy: true`, so the limits are kept per end-user IP. (With a
     proxy in between, have it append Django's header as it is; GoClaw reads the last entry.)
  3. Set `rate_limit_per_min` 3–5 times above the real load (600, for example), and raise
     `gateway.rate_limit.per_minute` the same way: without `trust_proxy` they are the only fix.

```yaml
gateway:
  rate_limit:
    per_minute: 600
```

A `429` carries a `Retry-After` header — Django passes it on to the app.

### 4.5 GoClaw client

```python
# assistant/goclaw.py (continued)
import json
from typing import AsyncIterator

import httpx
from django.conf import settings


class GoClawError(Exception):
    def __init__(self, status: int, message: str, retry_after: int | None = None):
        super().__init__(message)
        self.status = status
        self.message = message
        self.retry_after = retry_after


def _timeout() -> httpx.Timeout:
    # The read timeout must be long: the agent may spend 30–120 s running tools
    return httpx.Timeout(connect=5.0, read=settings.GOCLAW_TIMEOUT_READ,
                         write=10.0, pool=5.0)


def _raise_for_status(status: int, body: bytes, retry_after: str | None) -> None:
    try:
        message = json.loads(body).get("error", "unknown error")
    except (ValueError, AttributeError):
        message = body[:200].decode("utf-8", "replace")
    raise GoClawError(status, message,
                      int(retry_after) if retry_after and retry_after.isdigit() else None)


async def chat(user, message: str, *, session_id: str | None = None,
               client_ip: str | None = None) -> dict:
    """A synchronous (stream=False) reply. Suits short questions."""
    payload: dict = {"message": message, "stream": False}
    if session_id:
        payload["session_id"] = session_id

    async with httpx.AsyncClient(timeout=_timeout()) as client:
        resp = await client.post(
            f"{settings.GOCLAW_BASE_URL}/api/app/chat",
            headers=user_identity_headers(user, client_ip=client_ip),
            json=payload,
        )
        if resp.status_code != 200:
            _raise_for_status(resp.status_code, resp.content,
                              resp.headers.get("Retry-After"))
        return resp.json()


async def stream_chat(user, message: str, *, session_id: str | None = None,
                      client_ip: str | None = None) -> AsyncIterator[dict]:
    """Yields each SSE frame as a dict."""
    payload: dict = {"message": message, "stream": True}
    if session_id:
        payload["session_id"] = session_id

    async with httpx.AsyncClient(timeout=_timeout()) as client:
        async with client.stream(
            "POST",
            f"{settings.GOCLAW_BASE_URL}/api/app/chat",
            headers=user_identity_headers(user, client_ip=client_ip),
            json=payload,
        ) as resp:
            if resp.status_code != 200:
                body = await resp.aread()
                _raise_for_status(resp.status_code, body,
                                  resp.headers.get("Retry-After"))

            async for line in resp.aiter_lines():
                if not line.startswith("data: "):
                    continue
                raw = line[6:].strip()
                if not raw:
                    continue
                try:
                    yield json.loads(raw)
                except ValueError:
                    continue
```

### 4.6 Views

```python
# assistant/views.py
import json

from django.http import JsonResponse, StreamingHttpResponse
from django.shortcuts import aget_object_or_404
from django.views.decorators.csrf import csrf_exempt

from .goclaw import GoClawError, chat, stream_chat
from .models import AssistantMessage, AssistantThread


def client_ip(request) -> str:
    """The end user's IP. §4.4 — GoClaw's rate-limit buckets are split by it."""
    xff = request.META.get("HTTP_X_FORWARDED_FOR", "")
    if xff:
        return xff.split(",")[0].strip()
    return request.META.get("REMOTE_ADDR", "")


async def _load(request):
    """Shared checks: method, authentication, thread ownership, message."""
    if request.method != "POST":
        return None, JsonResponse({"error": "method not allowed"}, status=405)

    user = await request.auser()
    if not user.is_authenticated:
        return None, JsonResponse({"error": "unauthenticated"}, status=401)

    try:
        body = json.loads(request.body or b"{}")
    except ValueError:
        return None, JsonResponse({"error": "malformed JSON"}, status=400)

    message = (body.get("message") or "").strip()
    if not message:
        return None, JsonResponse({"error": "message is required"}, status=400)

    # The user=user filter stops anyone writing to another person's thread
    thread = await aget_object_or_404(AssistantThread,
                                      id=body.get("thread_id"), user=user)
    return (user, thread, message), None


# The app signs in with a token, not a session cookie, so CSRF does not apply.
# If you use cookie-based auth, REMOVE csrf_exempt and have the app send
# a CSRF token.
@csrf_exempt
async def chat_sync(request):
    loaded, err = await _load(request)
    if err:
        return err
    user, thread, message = loaded

    await AssistantMessage.objects.acreate(thread=thread, role="user", content=message)

    try:
        result = await chat(user, message,
                            session_id=thread.goclaw_session_id,
                            client_ip=client_ip(request))
    except GoClawError as exc:
        resp = JsonResponse({"error": exc.message}, status=_client_status(exc.status))
        if exc.retry_after:
            resp["Retry-After"] = str(exc.retry_after)
        return resp

    await AssistantMessage.objects.acreate(thread=thread, role="assistant",
                                           content=result["response"])
    return JsonResponse({
        "response": result["response"],
        "thread_id": str(thread.id),
        "elapsed_ms": result.get("elapsed_ms"),
    })


@csrf_exempt
async def chat_stream(request):
    loaded, err = await _load(request)
    if err:
        return err
    user, thread, message = loaded

    await AssistantMessage.objects.acreate(thread=thread, role="user", content=message)

    async def frames():
        parts: list[str] = []
        try:
            async for chunk in stream_chat(user, message,
                                           session_id=thread.goclaw_session_id,
                                           client_ip=client_ip(request)):
                if chunk.get("delta"):
                    parts.append(chunk["delta"])
                # session_id is internal — do not pass it to the app
                chunk.pop("session_id", None)
                yield f"data: {json.dumps(chunk, ensure_ascii=False)}\n\n".encode()
        except GoClawError as exc:
            payload = {"event": "error", "error": exc.message}
            yield f"data: {json.dumps(payload, ensure_ascii=False)}\n\n".encode()
        finally:
            # Keep what was written even if the client disconnected
            if parts:
                await AssistantMessage.objects.acreate(thread=thread, role="assistant",
                                                       content="".join(parts))

    resp = StreamingHttpResponse(frames(), content_type="text/event-stream")
    resp["Cache-Control"] = "no-cache"
    # Turn off nginx buffering — otherwise nothing arrives until the stream ends
    resp["X-Accel-Buffering"] = "no"
    return resp


def _client_status(goclaw_status: int) -> int:
    """Maps GoClaw's status to the status the app receives (see §6)."""
    # A 401 is our configuration error — return 500 so the user is not
    # misled into thinking they are signed out.
    return 500 if goclaw_status == 401 else goclaw_status
```

```python
# assistant/urls.py
from django.urls import path
from . import views

urlpatterns = [
    path("api/assistant/chat", views.chat_sync, name="assistant-chat"),
    path("api/assistant/stream", views.chat_stream, name="assistant-stream"),
]
```

### 4.7 Deployment requirements

| Item | Value | Why |
|---|---|---|
| Server | `uvicorn myproj.asgi:application` | Async views + SSE |
| nginx | `proxy_buffering off;` `proxy_read_timeout 300s;` | The stream is not buffered and long replies are not cut off |
| Django | `ALLOWED_HOSTS`, CSRF | CSRF exemption or token auth for POSTs from the app |
| Healthcheck | `GET {GOCLAW_BASE_URL}/health` | Needs no token, returns `{"status":"ok"}` |

**Acceptance criterion B1:** with `curl -N -X POST .../api/assistant/stream` the
first `data:` line arrives within 3 seconds (proving buffering is off).

---

## 5. Layer 3 — Flutter

### 5.1 Service

The user-facing strings below are English examples; localize them for your app.

```dart
// lib/services/assistant_api.dart
import 'dart:async';
import 'dart:convert';
import 'package:http/http.dart' as http;

class AssistantEvent {
  final String? delta;      // the text fragment of a "token" event
  final String event;       // token | tool_start | tool_result | done | error
  final String? tool;
  final String? error;
  final bool done;

  AssistantEvent({this.delta, required this.event, this.tool, this.error,
                  this.done = false});

  factory AssistantEvent.fromJson(Map j) => AssistantEvent(
        delta: j['delta'] as String?,
        event: (j['event'] as String?) ?? 'token',
        tool: j['tool'] as String?,
        error: j['error'] as String?,
        done: (j['done'] as bool?) ?? false,
      );
}

class AssistantApi {
  AssistantApi({required this.baseUrl, required this.sessionToken});

  final String baseUrl;         // https://api.example.mn
  final String sessionToken;    // the token Django issued

  http.Client? _client;

  /// Streaming mode. Shows the reply in the UI as it arrives.
  Stream stream(String threadId, String message) async* {
    final client = http.Client();
    _client = client;

    try {
      final req = http.Request('POST', Uri.parse('$baseUrl/api/assistant/stream'))
        ..headers.addAll({
          'Authorization': 'Bearer $sessionToken',
          'Content-Type': 'application/json',
          'Accept': 'text/event-stream',
        })
        ..body = jsonEncode({'thread_id': threadId, 'message': message});

      final resp = await client.send(req);

      if (resp.statusCode != 200) {
        final body = await resp.stream.bytesToString();
        yield AssistantEvent(
          event: 'error',
          error: _errorMessage(resp.statusCode, body),
        );
        return;
      }

      // In streaming mode Utf8Decoder joins multi-byte characters correctly
      // across chunks — Cyrillic text does not break.
      final lines = resp.stream
          .transform(utf8.decoder)
          .transform(const LineSplitter());

      await for (final line in lines) {
        if (!line.startsWith('data: ')) continue;
        final raw = line.substring(6).trim();
        if (raw.isEmpty) continue;
        yield AssistantEvent.fromJson(
            jsonDecode(raw) as Map);
      }
    } finally {
      client.close();
      _client = null;
    }
  }

  /// When the user taps "Stop".
  void cancel() {
    _client?.close();
    _client = null;
  }

  String _errorMessage(int status, String body) {
    try {
      final decoded = jsonDecode(body) as Map;
      final msg = decoded['error'] as String?;
      if (msg != null && msg.isNotEmpty) return msg;
    } catch (_) {}
    return switch (status) {
      401 => 'Your session has expired. Please sign in again.',
      429 => 'Too many requests. Please wait a moment and try again.',
      503 => 'The AI assistant is temporarily unavailable.',
      _ => 'Something went wrong ($status).',
    };
  }
}
```

### 5.2 Wiring it into the UI

```dart
// lib/controllers/chat_controller.dart
class ChatController extends ChangeNotifier {
  ChatController(this._api);

  final AssistantApi _api;
  final List messages = [];

  bool isStreaming = false;
  String? activeTool;          // shown as "Searching..." and the like

  StreamSubscription? _sub;

  Future send(String threadId, String text) async {
    messages.add(ChatMessage.user(text));
    final reply = ChatMessage.assistant('');
    messages.add(reply);
    isStreaming = true;
    notifyListeners();

    _sub = _api.stream(threadId, text).listen(
      (ev) {
        if (ev.event == 'token') {
          reply.text += ev.delta ?? '';
        } else if (ev.event == 'tool_start') {
          activeTool = ev.tool;
        } else if (ev.event == 'tool_result') {
          activeTool = null;
        } else if (ev.event == 'error') {
          reply.error = ev.error;
        } else if (ev.event == 'done') {
          isStreaming = false;
        }
        notifyListeners();
      },
      onError: (e) {
        reply.error = 'Network error';
        isStreaming = false;
        notifyListeners();
      },
      onDone: () {
        isStreaming = false;
        activeTool = null;
        notifyListeners();
      },
    );
  }

  void stop() {
    _api.cancel();
    _sub?.cancel();
    isStreaming = false;
    notifyListeners();
  }

  @override
  void dispose() {
    _sub?.cancel();
    super.dispose();
  }
}
```

**UI requirements:**

* When `tool_start` arrives, show a "🔧 Running {tool}…" indicator — the agent may think for 30+ seconds, and the user must not think the app froze.
* A "Stop" button is required (`cancel()`).
* An `error` may arrive without a `done` after it — guard with a timeout.

---

## 6. SSE frame reference

Django passes GoClaw's frames through unchanged (it only removes `session_id`).

| `event` | Fields | Meaning |
|---|---|---|
| `token` | `delta` | The next part of the reply. Append it |
| `tool_start` | `tool` | The agent started calling a tool |
| `tool_result` | `tool` | The tool finished |
| `error` | `error` | An error. A `done` may not follow |
| `done` | `done: true` | The end |

Details: [app-api.md → Streaming Response](app-api.md).

### HTTP status codes (GoClaw → Django)

| Code | Meaning | What Django does |
|---|---|---|
| `400` | `message` empty / malformed JSON | Pass on 400 |
| `401` | Wrong key, or `X-User-ID` missing | **Pass on as 500** — this is a server configuration error, not the user's fault. Raise an alert |
| `403` | Origin not allowed | Should not happen for requests from Django |
| `408` | `session busy or request cancelled`: the request was cancelled while an earlier one on the same thread was still running (requests on one thread run one at a time) | Pass on 408; the app retries once the earlier reply has arrived |
| `429` | Rate limit (`rate_limit_per_min` or the gateway-wide limit) | Pass on 429 + `Retry-After` |
| `500` | The agent or its model provider failed; `error` says why | Pass on 500 with a generic message; log GoClaw's `error` |
| `503` | Channel off / key not configured | 503 + "The AI assistant is temporarily unavailable". Raise an alert |

---

## 7. Security checklist

- [ ] `GOCLAW_API_KEY` is **not** in the Flutter binary (check with `strings`)
- [ ] `X-User-ID` is set by Django **on the server side**; it is **never** taken from the request body
- [ ] `X-User-ID` is stable, and a deleted user's ID is never reused
- [ ] The Django → GoClaw connection uses TLS or is closed within the internal network
- [ ] The GoClaw port (8080) is not open to the internet (`gateway.bind: loopback` + reverse proxy)
- [ ] Every `thread_id` is checked against its owner (`user=request.user` filter) — writing to another person's thread is impossible
- [ ] Error messages do not leak GoClaw's internal paths or session_id
- [ ] Django has a rate limit of its own (one user cannot saturate GoClaw)

---

## 8. Tests and acceptance criteria

| # | Criterion | How to check |
|---|---|---|
| A1 | GoClaw `app_api` is on | §3.3 curl → 200 |
| A2 | A request without the key is refused | curl without Authorization → 401 |
| B1 | SSE is not buffered | `curl -N` → first frame < 3 s |
| B2 | Users are isolated | Two different users each say "remember my name" → nothing mixes between them |
| B3 | Threads are isolated | Two threads of one user → contexts do not mix |
| B4 | `429` is passed on correctly | Set `rate_limit_per_min: 1` and send two requests → the app shows "please wait" |
| B5 | GoClaw is down | Stop GoClaw → the app does not crash and shows a clear message |
| C1 | Stream interrupted | Tap "Stop" midway → the partial reply is kept |
| C2 | Cyrillic text | A long Mongolian reply → no broken characters (UTF-8 chunk boundaries) |
| C3 | Tool indicator | A question that needs a search → `tool_start` is visible in the UI |
| D1 | Long replies | A question that runs 60+ s → no timeout |

---

## 9. Work breakdown

| # | Task | Owner | Estimate | Depends on |
|---|---|---|---|---|
| 1 | Configure GoClaw `app_api`, put the key in the vault | DevOps | 0.5 day | — |
| 2 | `AssistantThread` / `AssistantMessage` models + migration | Backend | 0.5 day | — |
| 3 | `goclaw.py` client (sync + stream) | Backend | 1 day | 1 |
| 4 | Views + URLs + error mapping | Backend | 1 day | 2, 3 |
| 5 | ASGI deployment, nginx buffering settings | DevOps | 0.5 day | 4 |
| 6 | `AssistantApi` Dart service | Mobile | 1 day | 4 |
| 7 | Chat UI + streaming + tool indicator + stop | Mobile | 2 days | 6 |
| 8 | Thread list / history screen | Mobile | 1 day | 7 |
| 9 | The tests in §8 | QA | 1 day | 7 |
| 10 | Security review (§7) | Backend + DevOps | 0.5 day | 9 |

**Total:** ~9 person-days; ~5 working days when done in parallel.

---

## 10. Common problems

| Symptom | Cause | Fix |
|---|---|---|
| Every request returns `503` | `api_key` not set / channel off | Look for `[app_api] channel NOT started` in the GoClaw log |
| Every request returns `401` | Django sends `gateway.auth.token` | `/api/app/*` **does not accept the dashboard token**. Use `channels.app_api.api_key` |
| The whole reply arrives at once | nginx is buffering | `proxy_buffering off;` + `X-Accel-Buffering: no` |
| Everyone gets `429` under load | The bucket for Django's single IP is full | §4.4 — pass `X-Forwarded-For`, raise `rate_limit_per_min` and `gateway.rate_limit.per_minute` |
| Users see each other's information | `X-User-ID` is unstable / duplicated | Move to a stable UUID |
| Long replies are cut off midway | httpx read timeout / nginx `proxy_read_timeout` | Set both to 300 s |
| Mongolian text is garbled | UTF-8 was decoded across byte-chunk boundaries | Use `utf8.decoder` as a **stream transformer** (do not decode each line separately) |

---

## 11. Out of scope

These are **not part of v1**; they are for a later phase:

* **The event stream (`GET /api/app/events`)** — only if the agent needs to push messages on its own initiative (cron reports, SOP results). SSE replies are enough for plain question and answer. `/ws/app` is retired and answers `410`. Note: a proactive message reaches only an **open** stream; if the app is closed it is lost. Push notifications are a separate matter. See [app-api.md — Event Stream Endpoint](app-api.md#4-event-stream-endpoint-get-apiappevents).
* **The `context_meta` field** — defined in `ChatRequest`, but the server **does not read it** yet. Do not build features on it.
* **File attachments** — `/api/media/upload` is a separate flow, for a later phase.
* **`auth_mode: "jwt"`** — only if Flutter must reach GoClaw **directly**. It is not needed in the Django proxy architecture.

---

## Related documents

- [Enterprise App API Channel](app-api.md) — the full API reference for the GoClaw side
- [Configuration](../configuration.md)
- [Pairing & per-user memory](pairing.md)
- [Ingress / reverse proxy](../ingress.md)

================================================================================
# FILE: https://goclaw.ulab.mn/docs/user-guide/channels/pairing.md
================================================================================

# User Pairing & Access Control

Pairing is GoClaw's primary mechanism for granting users access to chat with your agents. It provides a secure way to authorize individuals on third-party platforms like Discord or Microsoft Teams.

## How Pairing Works

When the `pairing` DM policy is active (the default), GoClaw acts as a gatekeeper:

1.  **Challenge:** An unknown user messages the bot.
2.  **Code Issued:** The bot generates an 8-character challenge code (e.g., `ABCD1234`) and sends it to the user.
3.  **Approval:** You (the bot owner) review the request and approve the code.
4.  **Authorized:** The user is added to the approved list and can now interact with the agent.

Despite its name, `channels.dm_policy` applies to every message the bot receives on Discord, Teams, Telegram and Facebook Messenger, not only to direct messages. An unapproved member who @mentions the bot in a Discord server channel or a Telegram group gets the pairing reply in that channel or group. Comments on a Facebook Page's posts and Lead Ads submissions are not messages to the bot and do not go through pairing; see [Facebook: comments, leads and `dm_policy`](facebook.md#comments-leads-and-dm_policy).

| `channels.dm_policy` | Who gets an answer |
|---|---|
| `pairing` (default) | IDs in `channels.allow_from` and approved (paired) users. Anyone else gets a pairing code. |
| `allowlist` | IDs in `channels.allow_from` and users already approved. Anyone else is told "You are not authorized to use this bot." and no code is issued. |
| `open` | Everyone. |

---

## Approving a Request

Once a user has their pairing code, you can approve it using any of the following methods:

### 1. Web UI (Recommended)
Navigate to the **Pairing** tab in your GoClaw dashboard (typically `http://localhost:8080`). Pending requests are listed with their channel, the user's platform ID and their code. Pick a role for the user (`user`, `admin` or `system`) and click **Approve**. The tab can be filtered by channel.

### 2. CLI
Use the `goclaw` binary directly:
```bash
# List pending pairing requests (the channel defaults to discord)
./goclaw pairing pending telegram

# Approve a specific code (--role user | admin | system, default user)
./goclaw pairing approve 
./goclaw pairing approve  --role admin
```

`approve` prints the new user's GoClaw user ID (`u_…`), which `revoke` and `link` take.

A running gateway sees an approval, revocation or link made with the CLI at once: it reads the pairing files again whenever they change.

### 3. REST API
The dashboard uses this API under the hood. It needs the gateway token (`gateway.auth.token`):
```bash
curl -X POST http://localhost:8080/api/pairing \
  -H "X-Goclaw-Token: " \
  -H "Content-Type: application/json" \
  -d '{"action": "approve", "channel": "discord", "code": "ABCD1234", "role": "user"}'
```

`channel` may be left out of an approve; the answer names the account and channel approved. `GET /api/pairing?channel=` (or `all`) lists pending requests and approved users. The other `POST` actions are `{"action": "revoke", "user_id": "u_…"}` and `{"action": "update_role", "user_id": "u_…", "role": "admin"}`. A role other than `user`, `admin` or `system` is refused (`400`).

---

## Pre-Approved Users (Bypassing Pairing)

If you already know the platform-specific IDs for yourself or your team, you can bypass the pairing flow by adding them to the `allow_from` list in `config.yaml`:

```yaml
channels:
  allow_from:
    - "100000000000000103"    # Discord user ID
    - "29:1abc-def-ghi"       # Teams user ID
```

`allow_from: ["*"]` lets everyone in, under either policy.

---

## Revoking Access

To remove a user from the approved list, use the Web UI (**Pairing** tab → approved users → **Revoke**) or the CLI. Revoking removes the user and every channel account linked to them.

```bash
# Revoke access via CLI: takes the GoClaw user ID (u_…), not the platform ID
./goclaw pairing revoke 

# Approved users on a channel (shows platform IDs; the channel defaults to discord)
./goclaw pairing list telegram
```

`pairing list` shows each user's platform ID, not the `u_…` ID. Find the `u_…` ID in the output of `approve`, on the Web UI (shown as "User ID"), or in `goclaw-users.json`. As with approvals, a CLI revoke is not seen by a running gateway until it restarts, so the user keeps access until then: revoke on the Web UI while the gateway runs.

Approved users are stored in `/credentials/goclaw-users.json` and pending requests in `/credentials/goclaw-pending.json` (the state directory is `~/.goclaw` unless `GOCLAW_STATE_DIR` says otherwise).

---

## Security Details

- **Challenge Codes:** Codes are 8 characters long, uppercase, and avoid ambiguous characters (like `0`, `O`, `1`, `I`).
- **Expiration:** Codes expire after **1 hour**.
- **Rate Limiting:** GoClaw keeps at most 5 pending requests in total, across all channels, to prevent spam. The oldest request is dropped when a new one arrives over the limit.
- **Cross-Platform Identity:** Approving a user on Discord does not automatically approve them on Teams. To let the same person in on another channel, link that channel's account to their existing GoClaw user: `./goclaw pairing link u_123456 telegram 123456789` (user ID, channel, platform account ID). The same restart rule applies as for a CLI approval.

================================================================================
# FILE: https://goclaw.ulab.mn/docs/user-guide/tools_and_mcp.md
================================================================================

# Tools & Skills User Guide

GoClaw provides a powerful ecosystem of tools and skills that enable the agent to interact with your system, data, and external services.

## Tools (Function Calling)

Tools are capabilities that the agent can actively execute. GoClaw features a library of **over 100 built-in tools** (ranging from file operations and web automation to database and observability integrations). To prevent context window bloat and keep the LLM focused, GoClaw utilizes a **RAG Tool Filter** to dynamically select and load only the relevant tools for each user query.

### Core Tools

| Category | Tools | Description |
|----------|-------|-------------|
| **File Operations** | `file_read`, `file_write`, `file_edit`, `read_file_lines`, `file_search`, `glob_files` | Read, write, and search files within the workspace. `file_edit` and `apply_patch` are preferred for precise changes. |
| **Shell & System** | `shell_exec`, `shell_exec_multi`, `process`, `clipboard` | Execute commands (one, or several in sequence), manage background processes, and interact with the system clipboard. |
| **Git & GitLab** | `git`, `gitlab`, `gitlab_hook` | Perform git operations and interact with GitLab projects, issues, and merge requests; have a project's events run a SOP ([`gitlab_hook`](automation/gitlab-webhook.md#asking-the-agent-for-a-hook), an admin in a conversation). |
| **Web & Network** | `web_search`, `web_fetch`, `browser`, `http_request` | Search the web, fetch content, automate headless Chrome, and make API requests. All of them go through the [outbound HTTP policy](#outbound-http). |
| **Database** | `db_query`, `mongo_query`, `db_write`, `db_analyze`, `db_enrich` | Query SQL (PostgreSQL, MySQL, ClickHouse) and MongoDB databases (read-only); `db_write` inserts, updates and deletes rows on datasources with `allow_write: true`. `db_analyze` reads a datasource's schema and caches it (the tables first, then one table's columns); `db_enrich` adds business notes — what a column means, what its values stand for — to that cached schema. |
| **Memory** | `memory_search`, `memory_get`, `memory_write`, `archive_search`, `archive_get` | Search and manage long-term agent memory and session archives. |
| **Knowledge** | `wiki_read`, `wiki_write`, `doc_index`, `review_learned`, `notes_distill` | Read and write the organisation wiki shared by all agents, and read the pages copied from [Wiki.js](wikijs.md); index a web page, wiki page or Markdown file in the workspace into searchable memory (admins); review learned facts held back as unverified (admins); clear resolved error notes and merge duplicate notes. |
| **Communication** | `message`, `notify`, `email` | Send messages to Discord/Teams, trigger notifications, and manage emails. |
| **Data & Utilities** | `json_transform`, `regex`, `text_utils`, `calc`, `datetime`, `spreadsheet`, `diff` | Process data, perform math, handle dates, and compare text. |
| **Agent & Sessions** | `agents_list`, `subagents`, `sessions_list`, `sessions_history`, `sessions_spawn`, `webhooks` | Manage multiple agents and conversation sessions; read the [webhook routes](configuration.md#webhook-routes) and what each received (`webhooks`, admins). |
| **SOP & Administration** | `sop_list`, `sop_get`, `sop_save`, `sop_run`, `sop_versions`, `sop_rollback`, `sop_review`, `sop_approve`, `sop_pin_channel`, `cron`, `update_core_instruction`, `capability_request` | Manage [SOPs](automation/sop.md) (versions, rollback, review date, approving a step that waits for approval, the channel a SOP's notices go to) and scheduled jobs; set the agent's core instruction; record a missing tool, view or permission for IT to provide. |
| **Infrastructure & Code** | `grafana_alerts`, `grafana_alert_rule`, `graylog_query`, `graylog_alert`, `uptrace_query`, `uptrace_monitor`, `zabbix_query`, `zabbix_alert`, `wazuh_query`, `wazuh_alert`, `netbox`, `parse_unl`, `sonarqube`, `sonarqube_hook`, `codereview_pull_repo`, `project_blast_radius` | The [observability tools](observability/index.md) and [NetBox](observability/netbox.md); read EVE-NG `.unl` topology files; query SonarQube (`tools.sonarqube`, or several instances under `tools.sonarqubes`); clone a repository, index its code structure and list every function that calls a given one, directly or through others. |
| **Media & Docs** | `image`, `pdf`, `document`, `diagram_render`, `generate_pdf`, `generate_html`, `voice_input`, `voice_output` | Analyze images; extract text from PDF, `.docx` and `.pptx` files; render a Mermaid diagram or raw HTML to a PNG with headless Chrome; generate PDF and HTML reports; transcribe an audio file and turn text into speech. |
| **Output Navigation** | `read_output` | Navigate large tool results (over `tools.max_result_chars`, default 8,000 characters) kept in memory for an hour — search, slice, or paginate without re-running the tool. |
| **Tool Discovery** | `search_tools` | Find a tool the agent may use but that was not selected for this request, and make it callable from the next step. A matching tool the caller's role may not use (an admin-only one, for a user) is never added: the search says it needs an administrator. |

### Configuration & Security

#### Shell Execution
Shell access is controlled via an allowlist in `config.yaml`:

```yaml
tools:
  shell:
    allow: ["git *", "npm *", "ls"]    # Allowed command prefixes
    deny: ["npm publish"]              # Refused even when allow matches
```

- With an empty `allow` (the default), `shell_exec` and `shell_exec_multi` refuse every
  command. `["*"]` allows any command.
- An entry allows every command that starts with it, unless a `deny` entry matches first.
- An allowed command is still refused when it contains a shell metacharacter:
  `;` `&` `|` `$` `>` `<` `` ` `` `\` `!` `(` `)` `{` `}` `[` `]` `=` `#` `^` `%` `@` or a
  line break. Pipes, redirection, chaining and variables are therefore never run, and an
  option is written `--flag value`, not `--flag=value`.
- A built-in deny list applies whatever the config says: destructive file-system commands,
  shutdown and reboot, service management, SSH, system package installs, user management,
  `ansible` and the deployment `make` targets.
- An agent's own `tools.shell.allow` or `deny` (in its `agents.list` entry) replaces the global
  list for that agent; the two are not merged.
- Commands run in the agent's workspace directory.

#### Outbound HTTP

Every tool that calls a URL goes through one outbound policy, `tools.http`: `http_request`,
`web_fetch`, `web_search`, the browser, MCP servers over HTTP/SSE, `gitlab`, `sonarqube`, the
observability tools (Grafana, Graylog, Uptrace, Zabbix, Wazuh, NetBox), and the tools that read a
file from a URL (`pdf`, `document`, `image`, `spreadsheet`).

The policy denies by default: with no `tools.http` settings, every request is refused.

An integration's own host is the exception. `gitlab`, `sonarqube` and the observability tools
may reach the host of the `url` (or Uptrace `dsn`) configured for the instance they call, private
address included, without an `allow` entry: it is the address you configured, not one the model
chose. That host is opened to that tool only — `http_request` and the browser still need it on
`allow` — and `deny` still refuses it. Any other host such a call reaches, a redirect elsewhere
say, is checked as usual. The Integrations page's **Test** runs under the same rule.

```yaml
tools:
  http:
    default_policy: deny       # the default; see below
    allow:
      - "api.search.brave.com" # web_search with Brave
      - "*.example.com"        # example.com and every subdomain
      - "wiki.internal"        # an internal host: its private address is allowed too
      - "10.0.0.17"            # a URL whose host is written as this IP
      - "10.0.0.0/24"          # URLs whose host is an IP in this range
    deny:
      - "169.254.169.254"      # cloud metadata endpoint
```

- `deny` is checked first and always wins.
- When `allow` has entries, only matching hosts are reached and `default_policy` makes no
  difference. When `allow` is empty, `default_policy: deny` refuses every host
  (`domain "…" is blocked: no domains in allow list (default_policy=deny)`), and
  `default_policy: allow` lets any public host through.
- An entry is an exact host name, `*.example.com` (that domain and its subdomains), `*`
  (every host), or an IP address or CIDR range. On `allow`, an IP or CIDR entry matches only a
  URL whose host is written as an IP address; a host name that resolves into the range still
  needs its own entry. On `deny`, it also refuses every host name that resolves into the range
  (`address … is blocked by policy`), even a name on `allow`.
- Private addresses — loopback, `10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`,
  `100.64.0.0/10`, link-local (including `169.254.169.254`) and IPv6 unique-local — are reached
  only through an `allow` entry that names them: the host name, or, for a URL written with an
  IP, that IP or a CIDR range around it. With `default_policy: allow` and an empty `allow`, every
  host that resolves to a private address is refused (`access to private IP … is blocked (SSRF
  protection)`).
- A wildcard entry lets every host name it matches reach private addresses too: `*.example.com`
  admits any name under `example.com` that resolves to a LAN address, and `*` turns the
  private-address check off for every host. Allow internal hosts by their exact name, and keep
  wildcards for public domains.
- The browser and `notify` (sending to a webhook URL) follow the same rules, a host name on
  `allow` included. The requests a page makes (redirects, scripts, images) are checked against
  the policy of the agent using the browser: its own `tools.http` when it has one, else the
  global one.
- An agent's own `tools.http` (in its `agents.list` entry) replaces the global setting for that
  agent, field by field: its `allow` replaces the global `allow` rather than adding to it, and
  the same holds for `deny` and `default_policy`.

```yaml
agents:
  list:
    - id: ops
      tools:
        http:
          allow: ["wiki.internal", "api.github.com"]   # this agent reaches only these
```

The policy is read on every request, so a change applies without a restart. To add a host from
the command line:

```bash
goclaw config append tools.http.allow=api.example.com
```

##### Hosts per run

One agent run (the answer to one message, a SOP step, a cron job) may reach at most
`tools.http.max_hosts_per_run` different hosts, 100 unless set. Every request a tool makes
counts: `http_request`, `web_fetch`, `web_search`, `notify` and the integration tools. A
subagent counts toward the run that started it. Once the run has reached the limit, a request
to a new host is refused (`host "…" refused: this run has already reached 100 different
hosts`), hosts it already reached keep working, and the gateway log says `[http-policy] a run
reached tools.http.max_hosts_per_run`. The browser's page loads are not counted.

An assistant's turn reaches a few hosts, a research turn a few dozen; a run that keeps going
from one host to the next is working through a list of addresses. Raise the limit for an agent
that genuinely needs more, in its own `tools.http`; `0` turns it off.

```yaml
tools:
  http:
    max_hosts_per_run: 100
```

When the gateway is reachable from other machines (`gateway.bind` is `lan`, `all` or a
non-loopback address, or an ingress tunnel is set) and `tools.shell.allow` is `["*"]` or
`tools.http.default_policy` is `allow` with an empty `allow` list, globally or for one agent,
the gateway logs a `[config] WARNING` naming the setting at start-up and on every config change
that alters it. Anyone who can message such an agent can have it run any command or reach any
host for them.

#### Git Operations
Read-only operations are always allowed. Write operations must be explicitly enabled:

```yaml
tools:
  git:
    allow_write: true
```

An agent's own `tools.git.allow_write` (in its `agents.list` entry) overrides the global
setting, in either direction. Even with writes enabled, the `git` tool refuses:

- `push --force` / `push -f`, `reset --hard`, `clean -fd`, and deleting `main` or `master`;
- options that run a program or write a file (`--upload-pack`, `--exec`, `--output`), and the
  global options that change git's configuration or repository (`-c`, `-C`, `--git-dir`,
  `--work-tree`);
- a push to a GitLab host whose `tools.gitlab` / `tools.gitlabs` entry has `allow_write: false`;
- subcommands it does not know.

#### Database Connections
Configure datasources in `config.yaml`:

```yaml
datasources:
  analytics:
    driver: clickhouse
    dsn: "clickhouse://localhost:9000/analytics"
  crm:
    driver: postgres
    dsn: "${secret:CRM_DSN}"
    allow_write: true   # opens this datasource to db_write and metrics_store writes
```

`db_query` runs `SELECT`, `SHOW`, `DESCRIBE`, `EXPLAIN` and `WITH` queries only, and
refuses one that hides a write (a data-modifying `WITH`, `EXPLAIN ANALYZE` of a write,
`SELECT … INTO`). It returns at most `tools.max_db_query_rows` rows (default 10,000).

The database enforces the same rules, so they hold if goclaw's own checks miss
something and when goclaw stops waiting:

| Engine | Time limit (about 29s, under `db_query`'s 30s) | Read-only |
|---|---|---|
| PostgreSQL | `statement_timeout` in the query's transaction; `client_connection_check_interval` (14+) ends a query whose gateway died | A read-only transaction |
| MySQL | `max_execution_time` | `transaction_read_only=1` on the read's connection, put back after the read; a connection that cannot be put back is closed. A read-only transaction alone still runs `CREATE`, `DROP` and other DDL |
| MariaDB | `max_statement_time` | as MySQL |
| ClickHouse | `max_execution_time` (with `timeout_overflow_mode=throw`), where the account lets goclaw set it; otherwise the profile's own | `readonly=1`, where the account allows it: writes, `SET`, and `url()`, `file()`, `remote()`, `dictionary()` or `format()` read as a table are refused. `DESCRIBE TABLE url(…)` still connects out, so goclaw's checks refuse those calls everywhere. Read a dictionary with `dictGet()` |

A DBA's tighter limit on the account is kept: goclaw sets the lower of the
two. That covers PostgreSQL's `statement_timeout` and
`idle_in_transaction_session_timeout` (`ALTER ROLE … SET`), and MySQL's and
MariaDB's session time limit. On PostgreSQL, a pool whose DSN asked for the
simple protocol runs an approved saved query on it, so its values keep the
types they had.

A query the database stops at its time limit comes back with a message telling
the agent to narrow it. A query cannot raise its own limit: `SET`, `set_config`,
MySQL's `SET_VAR` and `MAX_EXECUTION_TIME` hints and ClickHouse's `SETTINGS`
clause are refused before the query is sent.

What the account allows decides what ClickHouse is sent, and goclaw logs
one `[db_limits]` line per datasource saying what that is:

- **`readonly=1` profile.** It can change no setting unless its profile
  marks one `CHANGEABLE_IN_READONLY`; `db_query` sends only those. Without
  that, the time limit is the profile's own `max_execution_time`: set one
  there, or mark it `CHANGEABLE_IN_READONLY`.
- **`readonly=2` account.** It gets only the time limit. It lets a query run
  `url()`, `file()` and `remote()` and change its own time limit, so prefer
  `readonly=1`.
- **`readonly=0` account whose profile fixes `readonly`.** It gets no
  read-only on the server.
- **Bounded `max_execution_time`.** A limit outside the account's bounds is
  brought within them.

Over ClickHouse's HTTP interface, goclaw also sends
`cancel_http_readonly_queries_on_client_close` where the account allows it.
Otherwise a query whose client gave up kept running there.

ClickHouse turns a deadline into a setting of its own, so every query goclaw
sends ClickHouse has none: health checks, `db_write`, `metrics_store` and
org-directory lookups included.

goclaw learns what each server allows when the datasource is first used.
When it cannot, for example because the server is busy, `db_query` makes
do for a minute and then tries again; Postgres still gets
`statement_timeout`. A PostgreSQL DSN that asks for the simple protocol
(`default_query_exec_mode=simple_protocol`, common with PgBouncer) is opened
on the extended protocol without named prepared statements, which works
through PgBouncer.

Every datasource is read-only unless it sets `allow_write: true`. On those,
`db_write` can insert rows, and update or delete them by key (`where` takes
equality, `null` and lists — no operators). Only admins and admin-authored
SOP/cron runs are offered the tool; ordinary users, webhook-triggered runs and
delegated SOP runs are refused. One call changes at most
`tools.max_db_write_rows` rows (default 100): an update or delete that would
change more is rolled back and nothing is written. ClickHouse datasources
accept insert only; MongoDB is not supported.

### GitLab Integration Setup

To use the `gitlab` tool, you need a Personal Access Token (PAT):

1. Go to **User Settings → Access Tokens** in GitLab.
2. Create a token with `api` and `read_repository` scopes.
3. Configure it in `config.yaml` using an environment variable for security:

```yaml
tools:
  gitlab:
    url: https://gitlab.com
    token: "${env:GITLAB_TOKEN}"
    allow_write: false
```

More GitLab instances go under `tools.gitlabs`, by name; the agent picks one with the
tool's `instance` argument, and without it uses `tools.gitlab`:

```yaml
tools:
  gitlabs:
    internal:
      url: https://gitlab.example.com
      token: "${secret:GITLAB_INTERNAL_TOKEN}"
      auth_type: oauth      # "pat" (default) sends PRIVATE-TOKEN; "oauth" sends Authorization: Bearer
      allow_write: true
      description: "Internal services"   # helps the agent choose the instance
```

An agent with its own `tools.gitlab` (in its `agents.list` entry) uses that instance for
every call. The GitLab host needs no entry on `tools.http.allow` (see
[Outbound HTTP](#outbound-http)).

---

## Tool Testing & Benchmarking (CLI)

You can call any registered tool directly from the command line — **without starting the gateway or involving an LLM**. This is useful for testing configurations, debugging tool output, and measuring latency.

### List all tools

```bash
goclaw tools list
```

### Execute a tool directly

```bash
# Basic call
goclaw tools exec datetime action=now

# Multiple args (auto-typed)
goclaw tools exec gitlab action=mrs_list project=group/repo page=2

# SQL query
goclaw tools exec db_query datasource=analytics query="SELECT count(*) FROM users"
```

Arguments use `key=value` syntax with automatic type detection:

| Value | Parsed as |
|---|---|
| `page=2` | number (as in a model's tool call) |
| `active=true` | `bool` |
| `name=hello` | `string` |
| `"id='12345'"` | `string` (quoted, so it is not read as a number) |
| `'filters={"status":"open"}'` | JSON object |

The shell removes one level of quotes before GoClaw sees the argument: wrap a value that must
stay a string in a second pair of quotes, and quote JSON and anything with `*`, `?` or spaces.
The [tool reference](tool-exec-reference.md) lists each tool's arguments.

Output shows the **full result** (never truncated), execution time, char count, and line count.

### Benchmark a tool

```bash
goclaw tools bench datetime action=now --runs=20
goclaw tools bench memory_search query=test --runs=10
```

```
Bench: datetime — 20 runs
  Min:    1ms
  Max:    8ms
  Avg:    3ms
  P95:    6ms
  Errors: 0 / 20
Last output (chars: 25):
2026-03-27T11:50:47+08:00
```

---

## Skills (Context Injection)

Skills are Markdown files that provide specialized knowledge or best practices to the agent. Unlike tools, skills are injected into the agent's system prompt as reference material.

### Managing Skills via CLI

| Command | Description |
|---------|-------------|
| `skills list` | List all available skills |
| `skills install ` | Install a skill from a registry, URL, GitHub ref, or local path |
| `skills show ` | View the content of a specific skill |
| `skills remove ` | Uninstall a user-provided skill |

### Built-in Skills
GoClaw comes with built-in skills, including:
- `coding`: Best practices for file operations and testing.
- `git`: Guidelines for commits and branching.
- `mcp`: How to utilize the Model Context Protocol.
- `observability`: SRE patterns for monitoring and incident response.
- `skill-creator`: A meta-skill to help you write your own skills.

### Custom Skills
You can add your own skills by placing `.md` files (or `/SKILL.md` folders) in:
- Workspace: `~/.goclaw/workspace/skills/` (another agent's own workspace for that agent)
- User-level: `~/.goclaw/skills/`

A skill with the same name as an earlier one replaces it, in this order: built-in, the
agent's plugins, the workspace, the user-level directory.

---

## Model Context Protocol (MCP)

GoClaw supports the Model Context Protocol, allowing the agent to connect to external MCP servers to gain additional tools and resources. Use the `mcp` tool to interact with configured servers.

### Registering a server from the Web UI

The **MCP** tab registers servers without hand-editing `config.yaml`, and — the
part a config file cannot do — proves a definition works before it is saved.

1. **Register server** opens the editor. Pick the kind:
   - **HTTP / SSE** — a URL such as `http://127.0.0.1:3001/mcp/`. Leave transport
     on `auto` unless the server only speaks one of them.
   - **Local command (stdio)** — a command such as `npx` with arguments. It runs
     on the gateway host, as the gateway user.
2. **Test connection** dials the draft *as typed*, before anything is written. A
   success reports the server's name, version, protocol, capabilities and every
   tool it advertises.
3. **Try a tool** appears after a successful test: pick a tool, supply arguments
   as JSON, and run it. This calls the real server, so treat it as you would any
   other live call.
4. **Save** writes the definition to `mcp.servers.` in `config.yaml`. The
   config watcher picks it up, and any cached session for that name is closed so
   the agent uses the new definition immediately — no restart.

Saved servers are listed with a **Test** button, so an operator can check a
server that used to work without opening the editor.

### Credentials

The list and the editor show header and environment values as they are written in
`config.yaml`. A token pasted as plain text is therefore shown to everyone who can
open the dashboard, and it sits unencrypted in the file.

Store the token in the vault and write a reference instead — the reference is what
the editor shows, and the gateway resolves it when it connects:

```
Authorization: Bearer ${secret:MY_MCP_TOKEN}
```

`${env:VAR}` and `${file:/path}` (the file's first line) work the same way. See the
secret reference syntax in `config.yaml.example`.

### In `config.yaml`

A server saved from the Web UI lands under `mcp.servers`; you can also write it by hand:

```yaml
mcp:
  servers:
    docs:
      url: "http://127.0.0.1:3001/mcp/"
      transport: auto                 # auto (Streamable HTTP, then SSE) | streamable | sse
      headers:
        Authorization: "Bearer ${secret:MY_MCP_TOKEN}"
      description: "Internal documentation search"   # helps the agent pick this server
    files:
      command: npx                    # stdio: runs on the gateway host
      args: ["-y", "@modelcontextprotocol/server-filesystem", "/data"]
      env:
        LOG_LEVEL: info
```

The agent reaches them through the `mcp` tool: `list_servers`, `list_tools`, and
`call_tool` with `server`, `tool` and an `arguments` object, plus the resource and prompt
actions (`list_resources`, `read_resource`, `list_prompts`, `get_prompt`, …). From the
command line:

```bash
goclaw tools exec mcp action=call_tool server=docs tool=search 'arguments={"query":"hello"}'
```

### If the test fails

The most common failure is not a broken server. GoClaw denies outbound HTTP to
hosts that are not allowlisted, and private addresses on top of that — which is
exactly where MCP servers usually live:

```
Outbound HTTP refused "127.0.0.1" … add "127.0.0.1" to tools.http.allow
```

Add the host to `tools.http.allow` in config and test again (see
[Outbound HTTP](#outbound-http) for how the list matches). The editor turns the
other frequent failures — a 401, a wrong path, a command that is not installed,
a port with nothing behind it — into the same kind of specific advice.

================================================================================
# FILE: https://goclaw.ulab.mn/docs/user-guide/saved-queries.md
================================================================================

# Saved queries

A saved query is SQL that an admin wrote and approved for one datasource. It has typed parameters, and `db_query` runs it by name. When a saved query covers a question, the agent fills in values and writes no SQL, so it has no table or column name to guess. A report built on one returns the same answer every run.

```text
db_query(datasource: "reports-db", template: "active_orgs",
         params: '{"start_date": "2026-10-01", "end_date": "2026-11-01"}')
```

Questions that no saved query covers still go through ordinary `db_query` SQL.

## How the agent finds them

When at least one saved query is approved, `db_query` is always offered (pinned). Its description lists the approved queries with their datasource, description and parameters, for example:

```text
Saved queries, approved by an admin. When one answers the question, run it by name instead of writing SQL: …
- active_orgs [reports-db]: Organisations with at least min_requests events in a period — params: start_date date required, end_date date required, min_requests int ≥1 =100
```

The list shows at most 20 queries, or about 3000 characters; any further queries are listed by name only (30 names, then a count). It shows only the queries this run can actually run. In a delegated run, that means PostgreSQL datasources with a `delegated_role`. The agent picks one from the list. It does not depend on the question sharing words with the query, so a question in Mongolian finds an English description too.

Give the agent either `query` (its own SQL) or `template`, not both. A call with both is refused.

## Writing or drafting one

Saved queries run on PostgreSQL, MySQL and ClickHouse datasources, not on MongoDB. A saved query enters the list as an unapproved **draft** in three ways:

1. **On the dashboard.** Open **Integrations**, and click **Saved queries** on a datasource's card. The dialog lists the datasource's queries and shows each one's status: draft, approved version, approval that no longer holds, or deprecated. From there you can:
   - write a new query, or a new version of an existing one;
   - click **Mine from traces** to turn repeated successful `db_query` calls from recent traces into drafts;
   - **Test** a version with values (20 rows at most);
   - **Approve** it, which shows the exact SQL, parameters and hash first;
   - revoke, deprecate or delete it.
2. **Directly from the agent (`save_draft`).** In a conversation, the agent can pass `save_draft: ""` and `description: ""` to `db_query` (writing parameters as `{name}` in `query` and passing sample values in `params`, or passing a plain query whose filter literals are extracted as parameters automatically). `db_query` runs the query against the database first; only if it succeeds is it saved as a draft version.
3. **Automatically from repeated runs.** When the same read-only query shape (excluding schema probes such as `SHOW`, `DESCRIBE`, `information_schema` and `LIMIT 1`) succeeds at least 3 times or across at least 2 sessions, GoClaw extracts its date, datetime, string and numeric filter literals into typed `{name}` parameters (with the observed values as defaults) and saves it as a draft. The daily `trace_prune` maintenance job also mines recurring queries before pruning old traces. Deleting a draft permanently vetoes that SQL shape so automatic mining does not recreate it.

**Through the API.** The dialog uses the API below, which is also open to any client with the gateway token.

Make saved queries through the dashboard API, `/api/saved-queries`. That API takes the gateway token.

```json
POST /api/saved-queries
{
  "action": "save",
  "datasource": "reports-db",
  "name": "active_orgs",
  "description": "Organisations with at least min_requests events in a period",
  "wiki_slug": "metrics/active-organisation",
  "body": "SELECT org_id, count(*) AS events FROM events WHERE day >= {start_date} AND day < {end_date} GROUP BY org_id HAVING count(*) >= {min_requests} ORDER BY events DESC",
  "params": [
    {"name": "start_date", "type": "date", "required": true},
    {"name": "end_date", "type": "date", "required": true},
    {"name": "min_requests", "type": "int", "default": 100, "min": 1}
  ],
  "note": "first version"
}
```

- **`name`.** Use 3–64 lower-case letters, digits and `_`, starting with a letter. A name is unique per datasource.
- **`description`.** One line of up to 200 characters, with no control characters. It is what the agent reads, so state what the query answers. Business terms such as "active" belong in the body, not in a parameter the agent has to choose. The headings goclaw writes into tool descriptions are refused here, and so are they in enum values.
- **`wiki_slug`** (optional). Names the org wiki page that defines the terms the query uses.
- **`body`.** One read-only query: `SELECT`, `WITH`, `SHOW`, `DESCRIBE` or `EXPLAIN`. Write each parameter as `{name}`.
  - A `{name}` inside a string literal, a quoted identifier or a comment is left alone.
  - Do not write ClickHouse's `{name:Type}` yourself, or `{ name }` with spaces. The type comes from the parameter's declaration.
  - Do not write the driver's own placeholders either: `$1` on PostgreSQL, `?` on MySQL. PostgreSQL's jsonb `?` operator is fine.
  - Placeholders and comments are found the way the datasource reads them. On MySQL and ClickHouse, `#` starts a comment and a backslash escapes inside quotes, so a `{name}` in a MySQL comment is not a placeholder.
  - Table and column names cannot be parameters.
- **`params`.** Each parameter is either `required` or has a `default`. The types are:

| type | value | PostgreSQL | ClickHouse | MySQL |
|---|---|---|---|---|
| `date` | `YYYY-MM-DD` | `$n::date` | `{p:Date}` | `?` |
| `datetime` | `YYYY-MM-DD HH:MM:SS` (UTC) or RFC 3339 (converted to UTC) | `$n::timestamptz` | `{p:DateTime('UTC')}` | `?`, as UTC text |
| `int` | a whole number; `min`, `max`. Above 2^53, give it as text in `params` | `$n::bigint` | `{p:Int64}` | `?` |
| `number` | a number; `min`, `max`. Text is bound exactly | `$n::numeric` | `{p:Float64}` | `?` |
| `string` | text, at most `max_len` (default 200) | `$n` | `{p:String}` | `?` |
| `enum` | one of `values` | `$n` | `{p:String}` | `?` |
| `bool` | `true` / `false` | `$n::boolean` | `{p:Bool}` | `?` |

Values are always passed to the driver as bound parameters, never spliced into the text. A value that is not of its parameter's type is refused, and the error names what is allowed.

- **PostgreSQL text.** `string` and `enum` are bound without a cast, so the server reads them as the column's own type, as it would a literal. An enum, `uuid` or `citext` column compares as itself. Where PostgreSQL cannot tell the type (`SELECT {q}`, `{q} IS NULL`), the Test run shows the error; write `{q}::text` in the body.
- **Never NULL, never NUL.** No parameter is ever bound as NULL, and text containing a NUL character is refused.
- **ClickHouse limits.**
  - ClickHouse reads a parameter's text in its escaped format, so a backslash or a control character in a ClickHouse `string` or `enum` value is refused.
  - A `date` must fall between 1970-01-01 and 2149-06-06, and a `datetime` between 1970 and 2105.
  - Parameters need ClickHouse 22.8 or newer on the native protocol. An older server fails the Test run.

When you save, the body is checked the same way `db_query` checks SQL. Every placeholder must be declared, and every declared parameter must be used. The query must be read-only, and it must not call anything that reaches past the database, such as `url()`, `pg_read_file()` or `query_to_xml()`.

## Versions and approval

Every save adds a version, and versions are never changed. A new version is a draft, and `db_query` keeps running the approved version until the new one is approved.

```json
POST /api/saved-queries {"action": "test", "datasource": "reports-db", "name": "active_orgs", "version": 2,
                         "values": {"start_date": "2026-10-01", "end_date": "2026-11-01"}}
POST /api/saved-queries {"action": "approve", "datasource": "reports-db", "name": "active_orgs", "version": 2, "hash": ""}
POST /api/saved-queries {"action": "mine", "datasource": "reports-db"}
```

- **`test`.** Runs any version, draft or approved, through the same path as `db_query` and returns at most 20 rows. The query is stopped there, not left to run to the end. No hook or trace sees a test run; the gateway log records it.
- **`approve`.** Takes the hash that the save or the listing (`GET /api/saved-queries?datasource=…`) showed. If the version differs from what you saw, the approval is refused with `409`.
  - The version is checked again against the rules of the running release.
  - The hash covers the version's content and the database the datasource reaches: its driver, host, port and database, without credentials.
  - When `db_query` runs a query, it works out the hash again. A query whose content changed in the database, or whose datasource now reaches another database, does not run. That includes a datasource that was deleted and added again under the same name, and a snapshot imported where the name means another database.
  - The listing marks such a query `approval_holds: false`. To use it on the new database, save it again and approve that version.
- **`revoke`.** Withdraws the approval.
- **`deprecate`.** Stops offering and running a query but keeps its approval. Approving a version again undoes it.
- **`delete`.** Withdraws the query and hides it from the listing. Its versions are kept. If a query of the same name is made again, its versions go on from the next number, so a version that an old trace names is never other SQL.
- **`mine`.** Scans `agent_traces` for repeated successful `db_query` calls on the datasource, parameterizes their filter literals and saves any new shapes as drafts.

No tool can approve a saved query. Only this API can, behind the gateway token. That token is shared by the dashboard and any other client of the API, so the approval records `approved_via: "api"` and not a person.

## What a run looks like

`db_query` resolves the saved query before the admin's hooks run. A hook on `db_query` therefore sees the query's SQL in `query`, and `args_match` works on saved queries as it does on SQL the agent writes. The cache key includes the template, its version and the values, and the trace records the template and the values.

The result starts with `Saved query  v #:` on its first line. That line stays when the result is spilled or has no rows. If a saved query fails, the agent is told that it failed and not to write its own SQL in its place, because the failure is an admin's to fix.

A delegated run (a SOP acting for a person) runs a saved query exactly as it runs other SQL. That means PostgreSQL only, under the datasource's `delegated_role`, reading only the granted views. See [the SOP virtual level](../architecture/sop-virtual-level.md).

A saved query is held to the same limits on the database as other `db_query` SQL: a time limit the server enforces, and read-only where the engine can enforce it. See [Database Connections](tools_and_mcp.md#database-connections). A ClickHouse saved query cannot have a `SETTINGS` clause; one saved before this rule stops running until it is saved again without it.

## Binding to a SOP step

A [SOP step](automation/sop.md#step-configuration-reference) can bind one or more approved saved queries directly in its `queries` array (`[{template, datasource, params, as, max_rows}]`):

- When `queries_shadow` is false (the default), the step executes those saved queries directly through the worker pool and `db_query` with **zero LLM calls** and **zero tokens**.
- Each query publishes two named outputs for downstream steps: `` (the Markdown table, defaulting to the `template` name when `as` is omitted) and `_ref` (the `data_ref` ID in the output store, ready for `spreadsheet(action: "create", data_ref: ...)`).
- Parameter values in `params` may be literals, `{{run.date±Nd}}`, `{{run.time±Nh|m}}` (evaluated against the run's `StartedAt` in the SOP's `timezone`, or the server's local timezone when unset), or `{{steps..outputs.}}`.
- Setting `queries_shadow: true` runs the bound queries in shadow mode alongside the normal LLM step and records the outcome (`version`, `hash`, `rows`, `bytes`, `ms`, `error`) in the step result's `query_runs` without altering the step's output.
- **`sop_only`.** A saved query marked `sop_only: true` (via the **SOP only** checkbox in the editor or `{"action": "sop_only", "sop_only": true}` on `/api/saved-queries`) is hidden from chat tool descriptions, does not pin `db_query` in chat, and runs only inside a SOP step.
- **Used-by tracking and audit log.** `GET /api/saved-queries` and lifecycle responses (`save`, `approve`, `revoke`, `deprecate`, `sop_only`, `delete`) include `used_by`, the list of SOP IDs referencing that query. Approving, revoking, deprecating or deleting a saved query appends a `saved_query_` entry to each referencing SOP's `change_log`.
- **Auto-pausing a schedule on repeated pre-flight refusals.** If a SOP's pre-flight saved-query check fails 3 consecutive times with the same reason and the SOP is linked to a cron schedule, the schedule is automatically paused, a `cron_auto_paused` entry is recorded in the SOP's `change_log`, and a single alert is sent.

## Tool and MCP presets

A preset does for a read-only tool call what a saved query does for SQL: an admin approves the call once, with its arguments as a template, and the agent runs it by name with parameter values. Presets are managed through `POST /api/tool-presets` and the **Presets** dialog on the **Tools**, **MCP** and **Integrations** pages; `db_query` has saved queries instead.

A preset is:

| Field | Meaning |
|---|---|
| `kind` | `tool` for a registered tool, `mcp` for a tool on an MCP server. |
| `server` | For `mcp`: the MCP server's name in `config.yaml`. |
| `target` | The tool the preset calls: the registered tool's name, or the MCP tool's name on that server. |
| `name` | What the agent calls it by. Unique per tool (`tool`) or per server (`mcp`). |
| `description` | One line on what the call answers. |
| `args` | The call's arguments as a JSON object. A string value may be a placeholder, `{name}`. |
| `params` | What the placeholders take: name, type (`date`, `datetime`, `int`, `number`, `string`, `enum`, `bool`), required, default, min/max or values, as for a saved query. |

Rules:

- **Whole-value placeholders.** A placeholder stands for a whole argument value. A string that mixes a placeholder with text (`"project = {p}"`) is refused, so a query-language expression (JQL, UQL, PromQL, Lucene) is never built from a parameter value.
- **Read-only calls only.** A preset is saved and run only for a tool call the SOP effects journal counts as read-only (the same list), and for an MCP tool. For a tool with both reading and writing actions (`gitlab`, `git_workspace`) every value a parameter can take must keep the call read-only: a preset whose `enum` parameter has a value that would make it a write is refused when saved, and a call rendered from any preset is checked again when it runs.
- **Versions and approval, as for saved queries.** Every `save` is a new version; `test` runs one version with values; `approve` takes the version and its hash. The hash covers the kind, server, target, name, description, arguments, parameters and, for an MCP preset, the server's transport and address as configured, so a preset approved against one server does not run against another. `revoke`, `deprecate` and `delete` work as they do for saved queries.
- **Not in a delegated run.** A SOP acting for a person can neither list nor run presets.

Running one: a tool with an approved preset gets two more arguments, `preset` and `preset_params`, and its description lists the presets it has. The arguments the preset fixes are not given; `preset` with raw arguments (`arguments` for `mcp`, or another `action`) is refused.

```text
gitlab(preset: "open_blockers", preset_params: {"project": "platform/api"})
mcp(server: "jira", preset: "my_open_issues", preset_params: {"assignee": "alice"})
```

For `mcp`, `server` is needed only when the same preset name exists on more than one server.

Drafts come from two places, and an admin approves them as any version:

- **The agent.** In a conversation with a person (not a SOP, cron, webhook, subagent or delegated run), the agent can save a read-only call that just worked by adding `save_preset_draft: {"name": "...", "description": "...", "args": {...}, "params": [...]}` to it. `args` is the template with its placeholders; the draft is version 1, unapproved.
- **Mining.** `POST /api/tool-presets {"action": "mine"}` reads the last seven days of successful chat traces (`agent_traces`; the 500 newest, cron, webhook, SOP and subagent sessions left out), groups read-only tool calls and MCP `call_tool` calls by shape, with literal values turned into parameters, and saves a draft for every shape seen in at least two sessions by at least two people. `db_query` calls are left to saved-query mining. An existing or deleted preset of the mined name is never written over.

## Not yet

- There is no per-datasource cap on the rows or bytes a query may read (ClickHouse `max_rows_to_read`, `max_bytes_to_read`). A run stops reading at the row cap, and the database stops the query at its time limit.

## Storage

Saved queries are stored in `saved_queries` and `saved_query_versions`. Tool and MCP presets are stored in `tool_presets` and `tool_preset_versions`. All four tables are included in every `.gclaw` snapshot.

================================================================================
# FILE: https://goclaw.ulab.mn/docs/user-guide/tool-exec-reference.md
================================================================================

# Tool Execution Reference

`goclaw tools exec` calls registered tools directly from the CLI **without using an LLM**.

```bash
goclaw tools list                                   # every registered tool, with its description
goclaw tools exec  [key=value ...]
goclaw tools bench  [key=value ...] --runs=N
```

The arguments are the ones the tool's definition declares, the same ones a model sends in a tool call. `goclaw tools list` names every tool; this page shows working calls for the common ones.

## Prerequisites

CLI tool execution automatically loads the following runtime pieces:

| Component | Description |
|-----------|---------|
| `config.yaml` | Tool configuration: API keys, datasources, shell and HTTP allow lists. The global settings apply; no agent is selected, so an agent's own `tools:` overrides do not |
| Database | Memory, SOPs, cron jobs, metrics and other local state (`gc_state.sqlite`, or the configured `database:`) |
| Workspace | `/workspace` (`~/.goclaw/workspace` by default), where file tools read and write |
| Runtime context | `channel=cli`, `role=admin`, session `cli-offline-test`, timeout 300s per call (10s per run for `bench`) |

Some tools require additional configuration:

```yaml
# config.yaml - configuration required by selected tools
tools:
  shell:
    allow: ["ls", "git *", "go build *"]  # commands shell_exec may run; empty = shell disabled
  http:
    allow: ["gitlab.example.com", "api.example.com"]  # hosts any HTTP-calling tool may reach
  git:
    allow_write: true                     # needed by auto_commit and git write commands
  gitlab:
    url: "https://gitlab.example.com"
    token: "${env:GITLAB_TOKEN}"

datasources:
  reports-db:
    driver: postgres
    dsn: "${env:REPORTS_DB_DSN}"
```

`exec` does not open the secret vault. A `${secret:NAME}` reference in `config.yaml` reaches the tool as that literal text, so a credential kept in the vault fails to authenticate; `${env:VAR}` and `${file:/path}` references are expanded. The `secret_*` tools answer `secret store not initialized`; manage the vault with `goclaw secret list|get|set|delete` instead.

## Rules for every call

**How values are read.** Each argument is `key=value`, split at the first `=`:

| Value | Becomes |
|-------|---------|
| `10`, `2.5` | a number (float64, as in a model's tool call), so `limit=10` works |
| `true`, `false` | a boolean |
| `{...}` or `[...]` that is valid JSON | an object or array |
| `'...'` or `"..."` | a string, with those quotes removed, whatever is inside |
| anything else | a string |

A parameter the tool declares a string is always the text given, whatever it looks like: `mongo_query filter={"status":"open"}` and `gitlab project=12345` reach the tool as text. The table applies to the other parameters.

**Quoting.** The shell removes one level of quotes before goclaw sees the value, so:

- A parameter the tool does not declare a string is typed by the table: to keep a number-like value a string there, quote it twice: `"id='12345'"`. Written as `id='12345'`, it arrives as `id=12345` and becomes a number.
- Quote a JSON value as a whole word: `'filter={"k":"v"}'`. Unquoted, the shell drops the inner double quotes, and zsh treats `[ ]` as a glob.
- Tools that take JSON *as a string* (`mongo_query filter`, `json_transform data`, `http_request body`, `sop_save steps`) declare it a string, so it stays one: `'filter={"status":"active"}'`, or from a file: `"steps=$(cat steps.json)"`.
- Quote globs and anything with `*`, `?`, `[`, `!` or spaces: `pattern='*.go'`, `endpoint='/syscheck/000?summary=yes'`.
- A multi-line value goes in single quotes with real line breaks, or comes from a file: `patch="$(cat fix.patch)"`.

The `Args:` line at the top of the output is the JSON the tool received. Check it when a value does not do what you expect.

**Paths.** File tools resolve relative paths inside the workspace and refuse paths outside it (`access denied: path "/tmp/a.txt" is outside workspace ...`). An absolute path works only when it points inside the workspace.

**One call per process.** Each `exec` runs one tool call in a fresh process, which exits when the call returns. What a tool keeps in memory does not carry over to the next `exec`: the page `browser` opened, background processes, plan mode, generated HTML pages and `read_output` refs. What lives in the running gateway is not there either: connected channels, the session store, subagent runs and the SOP executor. The sections below say which tools this affects.

**Network.** Tools that make HTTP requests (`web_fetch`, `web_search`, `http_request`, `browser`, `gitlab`, the observability tools, `notify` to a webhook, `pdf`/`document`/`image` with a URL) go through `tools.http`. Its default policy is deny: a host not on `tools.http.allow` is refused, and so is a private address unless the allow list names it. The exception is an integration's own host: `gitlab`, `sonarqube` and the observability tools may reach the `url` you configured for the instance without an allow entry; `tools.http.deny` still applies.

Admin hooks (`hooks:` in `config.yaml`) run on these calls as on an agent's.

---

## Date & Time

### `datetime`

Actions: `now`, `add`, `diff`, `format`, `parse`, `unix`, `weekday`.

```bash
goclaw tools exec datetime action=now
goclaw tools exec datetime action=now timezone=America/New_York
goclaw tools exec datetime action=format date=2026-03-27T11:00:00 format='Mon, 02 Jan 2006 15:04'
goclaw tools exec datetime action=format date=2026-03-27T11:00:00Z timezone=Asia/Ulaanbaatar
goclaw tools exec datetime action=diff from=2026-01-01 to=2026-03-27
goclaw tools exec datetime action=add date=2026-03-27 days=3 hours=6
goclaw tools exec datetime action=unix timestamp=1767225600
goclaw tools exec datetime action=weekday date=2026-03-27
```

`format` takes a Go layout (`2006-01-02 15:04:05`). `add` takes `months`, `days`, `hours` and `minutes` (negative to subtract) and starts from now when `date` is omitted.

---

## Math & Text

### `calc`

```bash
goclaw tools exec calc expression='(1234 * 5.67) / 100'
goclaw tools exec calc expression='sqrt(2) + pi'
goclaw tools exec calc expression='2 ** 10'
```

### `text_utils`

Actions: `base64_encode`, `base64_decode`, `url_encode`, `url_decode`, `sha256`, `md5`, `regex_match`, `regex_extract`, `regex_replace`, `word_count`, `char_count`, `line_count`, `json_prettify`, `json_minify`, `uuid_generate`, `upper`, `lower`, `trim`, `reverse`.

```bash
goclaw tools exec text_utils action=upper text=hello
goclaw tools exec text_utils action=word_count text='Hello world foo'
goclaw tools exec text_utils action=sha256 text=hello
goclaw tools exec text_utils action=regex_extract text='Order 1234 shipped, order 5678 pending' pattern='\d{4}'
goclaw tools exec text_utils action=uuid_generate
```

### `regex`

Actions: `find_all`, `find_first`, `replace`, `split`, `test`. Patterns use Go (RE2) syntax.

```bash
goclaw tools exec regex action=find_all pattern='\d{4}' text='Codes: 1234-5678'
goclaw tools exec regex action=replace pattern='(\w+)@example\.com' replacement='$1@example.org' text='alice@example.com, bob@example.com'
goclaw tools exec regex action=test pattern='^v\d+\.\d+\.\d+$' text=v1.4.0
```

### `diff`

Compares two texts (`old_text`/`new_text`) or two workspace files (`old_file`/`new_file`).

```bash
goclaw tools exec diff old_text='hello world' new_text='hello goclaw'
goclaw tools exec diff old_file=docs/a.txt new_file=docs/b.txt
```

---

## File Operations

All paths below are relative to the workspace.

### `file_read` / `read_file_lines` / `file_write`

```bash
goclaw tools exec file_read path=notes/todo.txt
goclaw tools exec read_file_lines path=payments-api/internal/billing/invoice.go start_line=20 end_line=40
goclaw tools exec file_write path=notes/todo.txt content='Hello'
```

### `file_edit`

Replaces one exact, unique string in a file.

```bash
goclaw tools exec file_edit path=notes/todo.txt old_string=Hello new_string='Hello, world'
```

### `file_search`

```bash
goclaw tools exec file_search path=payments-api query=CalculateTotal
goclaw tools exec file_search path=payments-api query='func \w+Total' is_regex=true file_type=go context_lines=1
goclaw tools exec file_search path=payments-api 'queries=["ErrNotFound","MarkPaid"]' include='*.go'
```

### `glob_files`

```bash
goclaw tools exec glob_files pattern='*.go' dir=payments-api
goclaw tools exec glob_files pattern='**/*.md'
```

### `read_many_files`

```bash
goclaw tools exec read_many_files 'paths=["payments-api/go.mod","notes/todo.txt"]'
goclaw tools exec read_many_files pattern='*.go' dir=payments-api max_files=5
```

### `apply_patch`

Takes a unified diff; the file paths come from its `---`/`+++` headers.

```bash
goclaw tools exec apply_patch patch="$(cat fix.patch)"
goclaw tools exec apply_patch patch='--- a/docs/a.txt
+++ b/docs/a.txt
@@ -1 +1 @@
-old line
+new line
'
```

---

## Shell & Process

### `shell_exec` / `shell_exec_multi` (must be allowed by config)

The command runs in the workspace. It must match `tools.shell.allow`, and characters that chain or redirect commands (`; & | $ > < ( ) =` and similar) are refused even when it does.

```bash
goclaw tools exec shell_exec command='ls -la payments-api'
goclaw tools exec shell_exec command='go build ./...'
goclaw tools exec shell_exec_multi 'commands=["git status","git log --oneline -5"]'
```

### `process`

`process` (actions `start`, `list`, `output`, `kill`) keeps its background processes in the memory of the process that started them. From `exec`, `list`, `output` and `kill` never see a process an earlier `exec` started, so use it through an agent.

### `clipboard`

```bash
goclaw tools exec clipboard action=read
goclaw tools exec clipboard action=write text='Copied from goclaw'
```

---

## Git

### `git`

Runs in the workspace, so the workspace must be a git repository. `args` is the git command line. Read commands always work; writes (`add`, `commit`, `push`, ...) need `tools.git.allow_write: true`. `push --force`, `reset --hard`, `clean -fd` and deleting `main`/`master` are refused, as are `-C`, `-c` and `--git-dir`.

```bash
goclaw tools exec git args=status
goclaw tools exec git args='log --oneline -10'
goclaw tools exec git args='diff --stat HEAD~1'
goclaw tools exec git args='branch -a'
```

### `auto_commit`

Needs `tools.git.allow_write: true`. Stages the listed `paths` (otherwise commits what is already staged) and writes a commit message unless you give one.

```bash
goclaw tools exec auto_commit 'paths=["payments-api/internal/billing/invoice.go"]' message='fix: round invoice totals'
goclaw tools exec auto_commit
```

`undo=true` reverts the last auto-commit made in the same process, so it works only within an
agent run, not from a separate `exec`.

---

## GitLab

### `gitlab`

`project` is a path (`group/repo`) or a numeric ID quoted as a string. Lists take `page` (20 per page). `instance` picks one of `tools.gitlabs` (`gitlab/` works too), or `default` for `tools.gitlab`; omit it for `tools.gitlab`, or for the only instance there is. A name that matches no instance is refused with the list of names.

```bash
goclaw tools exec gitlab action=mrs_list project=group/repo
goclaw tools exec gitlab action=mrs_list project=group/repo state=opened page=2
goclaw tools exec gitlab action=mr_get project=group/repo mr_iid=42
goclaw tools exec gitlab action=mr_changes project=group/repo mr_iid=42
goclaw tools exec gitlab action=issues_list project=group/repo labels=bug state=opened
goclaw tools exec gitlab action=issue_get "project='123'" issue_iid=15
goclaw tools exec gitlab action=pipelines_list project=group/repo
goclaw tools exec gitlab action=pipeline_jobs project=group/repo pipeline_id=9001
goclaw tools exec gitlab action=job_log project=group/repo job_id=55012
goclaw tools exec gitlab action=file_get project=group/repo path=README.md ref=main
goclaw tools exec gitlab action=milestones_list project=group/repo
goclaw tools exec gitlab action=mr_label_events project=group/repo mr_iid=42
goclaw tools exec gitlab action=issue_label_events project=group/repo issue_iid=15 instance=staging
```

---

## Database

### `db_query`

Read-only (`SELECT`, `SHOW`, `DESCRIBE`, `EXPLAIN`, `WITH`).

```bash
goclaw tools exec db_query datasource=reports-db query='SELECT count(*) FROM users'
goclaw tools exec db_query datasource=reports-db query='SELECT date, COUNT(*) FROM events GROUP BY date ORDER BY date DESC LIMIT 7'
goclaw tools exec db_query datasource=clickhouse query='SELECT name FROM system.tables LIMIT 10'
```

### `db_write`

Needs `allow_write: true` on the datasource. `rows`, `set` and `where` are JSON (an object or a JSON string both work).

```bash
goclaw tools exec db_write datasource=crm operation=insert table=orders 'rows=[{"customer":"Alice","total":"120.50"}]'
goclaw tools exec db_write datasource=crm operation=update table=orders 'set={"status":"paid"}' 'where={"id":42}' max_rows=1
goclaw tools exec db_write datasource=crm operation=delete table=orders 'where={"id":[41,42]}' dry_run=true
```

### `mongo_query`

`filter` is a JSON string, so force it (see [Rules for every call](#rules-for-every-call)). `limit` defaults to 50, at most 500.

```bash
goclaw tools exec mongo_query datasource=events-db collection=users "filter='{\"active\":true}'" limit=10
goclaw tools exec mongo_query datasource=events-db collection=orders "filter='{\"status\":\"pending\"}'"
```

---

## Web & Network

### `web_search`

Needs `tools.web_search.brave_api_key` (or `serp_api_key`) and `api.search.brave.com` (or `serpapi.com`) on `tools.http.allow`.

```bash
goclaw tools exec web_search query='Go 1.23 release notes'
goclaw tools exec web_search query='output store pattern site:github.com' max_results=10
```

### `web_fetch`

```bash
goclaw tools exec web_fetch url=https://api.github.com/repos/golang/go/releases/latest
goclaw tools exec web_fetch url=https://example.com
```

### `http_request`

`headers` is text, one `Key: Value` per line; `body` is the raw request body.

```bash
goclaw tools exec http_request url=https://api.example.com/v1/items
goclaw tools exec http_request method=POST url=https://api.example.com/v1/items \
  headers='Content-Type: application/json' "body='{\"name\":\"widget\"}'"
goclaw tools exec http_request method=DELETE url=https://api.example.com/v1/items/7 headers='Accept: application/json
X-Request-Id: cli-test'
```

### `browser`

Needs Chrome. Actions: `navigate`, `screenshot`, `content`, `click`, `type`, `eval`, `wait`, `close`. Every action except `navigate` works on the page an earlier call opened, which a new `exec` does not have, so from the CLI only `navigate` is useful:

```bash
goclaw tools exec browser action=navigate url=https://example.com
goclaw tools exec browser action=navigate url=file://report.html   # an HTML file in the workspace
```

---

## Memory

### `memory_search`

```bash
goclaw tools exec memory_search query='deploy process' max_results=5
goclaw tools exec memory_search query='invoice rounding' time_start=2026-09-01 'tags=["billing"]'
```

### `memory_get`

Takes a chunk ID from a `memory_search` result.

```bash
goclaw tools exec memory_get "chunk_id='3f2a9c0e7b1d4a65'"
```

### `memory_consolidate`

```bash
goclaw tools exec memory_consolidate dry_run=true
goclaw tools exec memory_consolidate lookback_days=60
```

`memory_write` files a record under the person who asked, and `exec` acts for no one, so it answers `could not identify user`. Write memory through an agent, the Web UI or the memory API.

---

## Sessions

The session tools work on the running gateway's sessions and subagent runs. From `exec`, `sessions_list` and `sessions_history` answer `session store not available`, and `sessions_spawn` and `sessions_send` answer `subagent registry not initialized`. Runs are recorded in the database, so `session_status` can look one up:

```bash
goclaw tools exec session_status run_id=job-1759887600123456789-1
```

---

## Agents

### `agents_list`

```bash
goclaw tools exec agents_list
```

### `subagents`

Registration actions (`list`, `inspect`, `create`, `update`, `delete`) work from `exec`; `agent_id` is the agent the subagents belong to. `run` and `follow_up` need the gateway (`subagent registry not initialized`); `list_runs` and `status` read the recorded runs.

```bash
goclaw tools exec subagents action=list agent_id=ops
goclaw tools exec subagents action=inspect agent_id=ops name=code-reviewer
goclaw tools exec subagents action=list_runs
```

### `suggest_agent`

```bash
goclaw tools exec suggest_agent task='database performance issue' top_n=3
```

### `context_detect`

```bash
goclaw tools exec context_detect query='invoice discount' dir=payments-api max_files=5
```

---

## Communication

### `message` / `notify` / `discord_mod`

Sending to a channel needs that channel connected, and channels connect only in the running gateway. From `exec`, `message` refuses a cross-channel send from the `cli` session, `notify` to a channel answers `channel "discord" not connected`, and `discord_mod` answers `no Discord session connected`. `notify` can post to a webhook URL from the CLI (the host must be on `tools.http.allow`):

```bash
goclaw tools exec notify channel=webhook target=https://hooks.example.com/goclaw text='DB connection failed'
```

The call shapes, as an agent sends them: `message channel=discord "target='100000000000000102'" text=...` (optionally `file=reports/week-41.pdf`), `notify channel=msteams target= text=...`, and `discord_mod action=timeout "guild_id='...'" "user_id='...'" duration_minutes=60` (at most 40320; `ban`, `kick`, `add_role`, `remove_role` take `role_id` or `reason`). Discord IDs are longer than a float64 holds exactly, so always pass them as forced strings.

### `email`

Uses the `email.smtp` / `email.imap` settings. Actions: `send`, `read`, `search`, `reply`.

```bash
goclaw tools exec email action=send to=alice@example.com subject='Weekly report' body='See the attached file.' 'attachments=["reports/week-41.pdf"]'
goclaw tools exec email action=read count=10
goclaw tools exec email action=search from=bob@example.com since=2026-09-01
```

---

## Code Analysis

These tools read files in the workspace. `ast_declarations_search`, `ast_context_extractor`, `call_graph` and `complexity_analyzer` take one file; `dead_code_finder`, `symbol_usage` and `code_health_report` take a directory (`dir`, default the workspace root).

### `ast_declarations_search`

```bash
goclaw tools exec ast_declarations_search path=payments-api/internal/billing/invoice.go
```

### `ast_context_extractor`

Shows the declaration that contains a line.

```bash
goclaw tools exec ast_context_extractor path=payments-api/internal/billing/invoice.go line=25
```

### `ast_diff_analyzer`

Compares two versions of code passed as text.

```bash
goclaw tools exec ast_diff_analyzer filename_hint=invoice.go \
  old_content="$(git show HEAD~1:internal/billing/invoice.go)" \
  new_content="$(cat internal/billing/invoice.go)"
```

### `call_graph`

```bash
goclaw tools exec call_graph path=payments-api/internal/billing/invoice.go focus=CalculateTotal depth=2
```

### `complexity_analyzer`

```bash
goclaw tools exec complexity_analyzer path=payments-api/internal/billing/invoice.go threshold=10
```

### `dead_code_finder`

```bash
goclaw tools exec dead_code_finder dir=payments-api
goclaw tools exec dead_code_finder dir=payments-api include_exported=true
```

### `dependency_map`

```bash
goclaw tools exec dependency_map package=payments-api/internal/billing
goclaw tools exec dependency_map package=payments-api/internal/billing format=mermaid depth=1
```

### `symbol_usage`

```bash
goclaw tools exec symbol_usage symbol=CalculateTotal dir=payments-api
goclaw tools exec symbol_usage symbol=CalculateTotal dir=payments-api include_tests=false
```

### `repo_map`

Maps the whole workspace; `query` and `focus_files` weight the ranking.

```bash
goclaw tools exec repo_map query='invoice totals' max_tokens=1024
```

### `code_health_report`

```bash
goclaw tools exec code_health_report dir=payments-api
goclaw tools exec code_health_report dir=payments-api format=detailed
```

---

## Linting & Security

The linters run in the workspace and must be installed; `setup_dev_env` shows which are missing and how to install them.

### `run_code_linter`

```bash
goclaw tools exec run_code_linter linter=golangci-lint path=payments-api
goclaw tools exec run_code_linter linter=ruff path=scripts/analyze.py
goclaw tools exec run_code_linter linter=dart-analyze path=mobile-app/lib
```

### `run_sast_scanner`

```bash
goclaw tools exec run_sast_scanner scanner=gosec path=payments-api
goclaw tools exec run_sast_scanner scanner=bandit path=scripts
```

### `setup_dev_env`

```bash
goclaw tools exec setup_dev_env
goclaw tools exec setup_dev_env 'languages=["go","python"]'
```

---

## Observability

Each instance is configured under `tools.observability`; its own host needs no entry on `tools.http.allow`. `instance` can be omitted when only one is configured.

### `grafana_alerts`

```bash
goclaw tools exec grafana_alerts instance=prod action=alerts
goclaw tools exec grafana_alerts instance=prod action=dashboards
goclaw tools exec grafana_alerts instance=prod action=annotate text='Deploy started' 'tags=["deploy","api"]'
```

### `graylog_query`

`range` is in seconds (default 3600).

```bash
goclaw tools exec graylog_query instance=prod action=search query='source:api-gateway' range=3600 limit=50
goclaw tools exec graylog_query instance=prod action=search_absolute query='level:3' from=2026-03-27T00:00:00Z to=2026-03-27T12:00:00Z
goclaw tools exec graylog_query instance=prod action=terms_stats query='level:3' field=source range=86400
goclaw tools exec graylog_query instance=prod action=streams_list
```

### `uptrace_query`

Start with `action=projects` to find the `project_id`. UQL strings use single quotes, so wrap the query in double quotes.

```bash
goclaw tools exec uptrace_query instance=main action=projects
goclaw tools exec uptrace_query instance=main action=query project_id=1 query='perMin(count()) | p99(_dur_ms) | group by service.name' time_gte=now-1h
goclaw tools exec uptrace_query instance=main action=spans project_id=1 query="service.name = 'api-gateway'" limit=20
goclaw tools exec uptrace_query instance=main action=spans project_id=1 trace_id=4bf92f3577b34da6a3ce929d0e0e4736
goclaw tools exec uptrace_query instance=main action=monitors project_id=1
```

### `wazuh_query`

```bash
goclaw tools exec wazuh_query instance=main endpoint=/agents/summary/status
goclaw tools exec wazuh_query instance=main endpoint='/syscheck/000?summary=yes'
```

### `zabbix_query`

`params` is a JSON object.

```bash
goclaw tools exec zabbix_query instance=prod method=host.get 'params={"output":["hostid","host"]}'
goclaw tools exec zabbix_query instance=prod method=problem.get 'params={"output":"extend","recent":true}'
goclaw tools exec zabbix_query instance=prod method=trigger.get 'params={"output":"extend","filter":{"value":"1"}}'
goclaw tools exec zabbix_query instance=prod method=item.get 'params={"hostids":"10084","output":"extend"}'
```

### `analyze_pcap`

Takes a `.pcap`, `.pcapng` or `.cap` file in the workspace. Needs `tshark`; without it only `summary` works (through `tcpdump`). Perspectives: `summary`, `conversations`, `dns`, `http`, `errors`, `latency`, `payload`.

```bash
goclaw tools exec analyze_pcap file=captures/edge-router.pcap
goclaw tools exec analyze_pcap file=captures/edge-router.pcap perspective=http
goclaw tools exec analyze_pcap file=captures/edge-router.pcap perspective=dns filter='port 53'
goclaw tools exec analyze_pcap file=captures/edge-router.pcap perspective=latency filter='host 192.0.2.10'
```

---

## Data & Documents

### `json_transform`

`data` is the input *as a string*, so force JSON input to stay a string. Actions: `jq`, `format`, `minify`, `csv_to_json`, `json_to_csv`, `yaml_to_json`, `json_to_yaml`.

```bash
goclaw tools exec json_transform action=jq "data='{\"a\":{\"b\":42}}'" query=.a.b
goclaw tools exec json_transform action=format "data='$(cat response.json)'"
goclaw tools exec json_transform action=json_to_csv "data='[{\"name\":\"Alice\",\"age\":30},{\"name\":\"Bob\",\"age\":25}]'"
goclaw tools exec json_transform action=yaml_to_json data='name: Alice
age: 30'
```

### `spreadsheet`

Actions: `read`, `sheets`, `read_csv`, `create`. `create` writes `.xlsx` or `.csv` from `headers` and `rows`; the directory of `output_path` must already exist.

```bash
goclaw tools exec spreadsheet action=create output_path=team.xlsx 'headers=["name","role"]' 'rows=[["Alice","ops"],["Bob","dev"]]'
goclaw tools exec spreadsheet action=sheets path=team.xlsx
goclaw tools exec spreadsheet action=read path=team.xlsx sheet=Sheet1 range=A1:B10
goclaw tools exec spreadsheet action=read_csv path=exports/orders.csv
```

`data_ref` (a ref from an agent run's large output) is not available from the CLI; see [Large Output Navigation](#large-output-navigation).

### `diagram_render`

Renders Mermaid (default) or raw HTML (`format=html`) to a PNG with headless Chrome, saved in `/canvas/`.

```bash
goclaw tools exec diagram_render filename=flow.png content='graph TD
  Request --> Gateway
  Gateway --> Agent'
goclaw tools exec diagram_render format=html content='
Weekly status
' filename=status.png width=800 height=200
```

### `document`

Extracts text from `.docx` and `.pptx` files (a workspace path or a URL).

```bash
goclaw tools exec document path=specs/payments-spec.docx
goclaw tools exec document path=slides/quarterly-review.pptx
```

### `pdf`

```bash
goclaw tools exec pdf path=reports/week-41.pdf
goclaw tools exec pdf path=scans/contract.pdf use_ocr=true
```

`use_ocr` uses Gemini Vision, as `image` does.

### `image`

Analyzes an image with Gemini Vision (needs a Gemini API key).

```bash
goclaw tools exec image path=canvas/flow.png
goclaw tools exec image path=screenshots/dashboard.png question='What is the error rate on the main chart?'
```

---

## Report Generation

### `generate_pdf`

Needs Chrome. `action` is `weekly_report` or `custom`; `sections` is an array of `{type, heading, content}` with `type` one of `text`, `table` (CSV), `bar`, `pie` (CSV of label,value). The file goes to `reports/` in the workspace unless `output_path` says otherwise.

```bash
goclaw tools exec generate_pdf action=custom title='Monthly report' 'sections=[{"type":"text","heading":"Summary","content":"Deploys went out on time."},{"type":"table","heading":"Incidents","content":"service,count\napi-gateway,2\ncore-bank,0"}]'
goclaw tools exec generate_pdf action=weekly_report title='Week 41' output_path=reports/week-41.pdf 'sections=[{"type":"bar","heading":"Tickets closed","content":"Mon,4\nTue,7\nWed,5"}]'
```

### `generate_html`

`generate_html` keeps its pages in the memory of the process that serves them. A page created from `exec` disappears when the command exits, so create pages through an agent.

---

## Large Output Navigation

### `read_output`

The agent loops store a large tool result and hand the model a `[LARGE OUTPUT: ... | ref:]` preview, which `read_output` (and `spreadsheet data_ref=`) reads. `exec` prints a tool's whole result and stores nothing, and the store lives in the gateway process, so a ref from an agent run cannot be read from the CLI.

---

## Secrets

`exec` does not open the vault, so `secret_get`, `secret_set`, `secret_delete` and `secret_list` answer `secret store not initialized`. Use the `goclaw secret` command:

```bash
goclaw secret list
goclaw secret get GITLAB_TOKEN
goclaw secret set MY_API_KEY sk-xxx --category api_key
goclaw secret delete OLD_KEY
```

---

## SOP (Standard Operating Procedures)

SOPs are addressed by `id`. `steps` is a JSON array passed as a string; keep it in a file. Each step needs `id`, `name`, `intent` (at least 20 characters), `instruction` (at least 80) and `tools`.

```bash
goclaw tools exec sop_list
goclaw tools exec sop_get id=open-tickets
goclaw tools exec sop_history id=open-tickets limit=5
goclaw tools exec sop_versions id=open-tickets
goclaw tools exec sop_enable id=open-tickets enabled=false
goclaw tools exec sop_save id=open-tickets name='Open tickets' \
  purpose='Report how many support tickets are open, by priority.' "steps='$(cat steps.json)'"
goclaw tools exec sop_delete id=open-tickets
```

With `steps.json`:

```json
[{"id":"s1","name":"Count open tickets","intent":"Count the support tickets that are still open today","instruction":"Run a SELECT on the tickets table of the reports-db datasource that counts rows with status open, grouped by priority. Return the counts as a compact JSON object.","tools":["db_query"],"output_format":"JSON object {priority: count}","success_check":"json"}]
```

`sop_run` needs the gateway's SOP executor: from `exec` it answers `SOP executor not initialized`. Run a SOP from the Web UI, the API or a chat.

---

## Task Tracker

Task IDs are strings like `T001`.

```bash
goclaw tools exec tracker_create_task title='Fix DB connection' priority=high
goclaw tools exec tracker_list_tasks
goclaw tools exec tracker_list_tasks status=todo
goclaw tools exec tracker_get_task id=T001
goclaw tools exec tracker_update_task id=T001 status=done
goclaw tools exec tracker_add_dependency task_id=T002 depends_on=T001
goclaw tools exec tracker_visualize
```

---

## Metrics Store

```bash
goclaw tools exec metrics_store action=record source=gitlab entity_type=mr entity_id='group/repo#42' \
  'metrics={"cycle_time_h":18.5,"review_rounds":2}' 'tags={"team":"payments"}'
goclaw tools exec metrics_store action=query filter_source=gitlab filter_metric=cycle_time_h date_from=7d
goclaw tools exec metrics_store action=trend filter_metric=cycle_time_h weeks=4
goclaw tools exec metrics_store action=annotate source=gitlab entity_id='group/repo#42' notes='Release 1.4.0 deployed'
goclaw tools exec metrics_store action=purge date_from=90d filter_source=custom
```

`annotate` adds notes to the rows already recorded for that source and entity. `purge` deletes rows older than `date_from`.

---

## Cron

Schedules have six fields, seconds first (`0 0 9 * * 1-5` is 09:00 on weekdays), or `@every 30s`. A new job runs a SOP (`sop_id`) or a built-in `notify ...` command; shell commands are refused.

```bash
goclaw tools exec cron action=list
goclaw tools exec cron action=create schedule='0 0 9 * * 1-5' sop_id=open-tickets
goclaw tools exec cron action=create schedule='0 30 8 * * *' command='notify channel=discord target=100000000000000102 text=Standup in 30 minutes'
goclaw tools exec cron action=history id=cron-1
goclaw tools exec cron action=delete id=cron-1
```

A job created or deleted from `exec` is written to the database, and a running gateway picks the change up within 30 seconds. Only the gateway runs jobs: `exec`, and `goclaw chat`, run none themselves.

---

## Planning Mode

```bash
goclaw tools exec write_todos title='Release 1.4' \
  'todos=[{"content":"Fix bug","status":"completed"},{"content":"Write tests","status":"in_progress","priority":"high"},{"content":"Deploy","status":"not_started"}]'
```

`enter_plan_mode` (`reason=...`) and `exit_plan_mode` (`plan=...`) switch a flag held in memory. It lasts only for the `exec` that set it, so they do nothing useful from the CLI.

---

## MCP (Model Context Protocol)

`server` is a name from `mcp.servers` in `config.yaml` (or an MCP URL); `arguments` is a JSON object.

```bash
goclaw tools exec mcp action=list_servers
goclaw tools exec mcp action=list_tools server=my_mcp
goclaw tools exec mcp action=call_tool server=my_mcp tool=search 'arguments={"query":"hello"}'
goclaw tools exec mcp action=list_resources server=my_mcp
goclaw tools exec mcp action=read_resource server=my_mcp uri='resource://docs/readme'
```

---

## Voice

### `voice_input`

Transcribes an audio file (wav, mp3, m4a, webm, ogg, flac and similar, up to 25 MB) with the configured speech-to-text provider. `audio_path` is read inside the workspace, like `file_read`: a path outside it is refused.

```bash
goclaw tools exec voice_input audio_path=recordings/standup.m4a
goclaw tools exec voice_input audio_path=recordings/standup.m4a language=mn
```

---

## Archive

### `archive_search`

Searches the conversation archives written when session history is summarized. With no `query`, it lists them.

```bash
goclaw tools exec archive_search
goclaw tools exec archive_search query='deploy error' max_results=10
goclaw tools exec archive_search query='Q1 report' date=2026-03-14
```

### `archive_get`

Takes a file name from an `archive_search` result.

```bash
goclaw tools exec archive_get filename=session_discord_123_2026-03-14_2313.md
goclaw tools exec archive_get filename=session_discord_123_2026-03-14_2313.md section=summary
```

---

## Benchmark Examples

`bench` runs the call `--runs` times (default 5), each with a 10-second limit, and prints min, max, average and p95 latency.

```bash
goclaw tools bench datetime action=now --runs=50
goclaw tools bench memory_search query=test --runs=10
goclaw tools bench db_query datasource=reports-db query='SELECT 1' --runs=20
goclaw tools bench web_fetch url=https://example.com --runs=5
```

================================================================================
# FILE: https://goclaw.ulab.mn/docs/user-guide/memory_and_context.md
================================================================================

# Memory and Context

This guide explains how the agent remembers information, how it manages conversation history, and how you can control what it learns and retains.

## How the Agent Remembers

The agent uses a multi-tiered memory system to provide relevant context while staying within the limits of its AI model.

### 1. Short-Term Memory (The Current Conversation)
The agent keeps track of your current session's back-and-forth messages. However, as the conversation grows very long, it may exceed the "context window" (the maximum amount of text the AI can process at once).

### 2. Context Pruning and Summarization
To save tokens and stay within limits, the agent automatically "prunes" old messages when the conversation gets too long:
- **Summarization**: The agent uses an AI model to create a concise summary of the older parts of the conversation.
- **Archiving**: The full transcript of the pruned messages is saved to an archive, which the agent can still search if needed.
- **Retention**: The agent always keeps the most recent messages and the summary of previous interactions in its active memory.
- **When the summary fails**: the summary is asked of the background model (the Gemini model of `ai.gemini` where it has a key, else the agent's own model), retried when the provider is briefly overloaded and then asked of the agent's fallback models (`ai.fallback`, or the agent's own `fallback` list). If every one fails, the older messages stay in the conversation and the next turn tries again; they are archived and trimmed without a summary only once the conversation is twice past its limit, or when no model is configured at all.

### 3. Long-Term Memory (RAG)
For information that needs to last across different sessions, the agent uses **Retrieval-Augmented Generation (RAG)**. It stores facts and notes in two ways:
- **Global Memory**: General knowledge or organizational facts that apply to everyone.
- **Personal Memory**: Facts specifically about you (preferences, role, specific tasks).

All long-term memories are stored in GoClaw's state database (**SQLite** by default) for high-performance searching and strict privacy isolation.

The agent uses **Hybrid Search** (combining keyword search and, when `memory.search.provider` is set, AI-powered semantic search) to find the most relevant information from its database and archive files when you ask a question.

## Background Learning (Asynchronous)

GoClaw learns from your interactions without slowing down the conversation:
- **Fact Extraction**: After a session is summarized, a background process automatically identifies and saves new facts about your environment or preferences.
- **Auto-Correction**: If a tool fails (e.g., a wrong database table name), the error comes back with a hint and the matching approved saved queries or tool presets, so the agent corrects the call in the same turn. Nothing is stored from the failure; to make the right approach permanent, an admin saves it as a [saved query or tool preset](saved-queries.md).
- **Job Queue**: These tasks are queued and processed sequentially, ensuring the agent remains responsive and the database stays stable.

## Memory Consolidation

Over time, global memory and daily notes can become cluttered. An administrator (or the agent, in an admin or system run) can run the `memory_consolidate` tool, which:
1. Collects global memory records and daily memory files (`memory/YYYY-MM-DD.md`) older than `memory.consolidate_after_days` (30 by default; the tool's `lookback_days` argument overrides it).
2. Extracts the most important facts using AI.
3. Moves these facts into a long-term "Core Memory" file (`Memory.md`) which serves as the agent's primary bootstrap context.
4. Deletes the processed records and moves the processed files to `memory/archived/`.

`dry_run: true` reports what would be consolidated without changing anything.

Separately, a built-in daily job (`sys-memory-consolidate`, 03:00) merges learned agent notes that say the same thing into one record. Each merge is recorded, and an administrator can undo it with `goclaw memory revisions undo`.

## User-Facing Configuration

You can influence how the agent uses memory through configuration (usually in `config.yaml`):

| Setting | Default | Description |
|---------|---------|-------------|
| `memory.search.provider` | empty | Embedding provider for semantic search: `gemini`, `openai` or `ollama`, using that provider's key or address from `ai:`. Empty means keyword search only. |
| `memory.search.model` | the provider's default | Embedding model. |
| `memory.session_memory.max_tokens` | `40000` | When a conversation grows past this many tokens, its older messages are summarized and trimmed. A digit or a non-Latin (for example Cyrillic) letter counts as one token, other text as four characters a token, about as the model counts them. A conversation keeps at most half of `memory.auto_flush_tokens`, whatever this says: more would put every turn past the flush threshold at its first call. To keep more history, raise both. |
| `memory.session_memory.messages` | `40` | Messages to keep instead, used only when `max_tokens` is `0`. |
| `memory.session_memory.enabled` | `true` | `false` keeps the transcript of a session cleared with `/new` or `/reset` out of memory. |
| `memory.bootstrap_max_chars` | `20000` | Most characters of `Memory.md` put into each prompt; a longer file keeps its beginning and end. An agent's own `bootstrap_max_chars` in `agents.list` overrides it. |
| `memory.auto_flush_tokens` | `80000` | When one request's messages grow past this many tokens, the older ones are distilled into the daily memory file and dropped from the request. `memory.flush.threshold_tokens` sets the same thing when this is left at its default; `memory.flush.enabled: false` turns the flush off. |
| `memory.context_budget_chars` | `32000` | Most characters of memory, notes and lessons put into each prompt; the lowest-priority sections are dropped first. `0` means no limit. |
| `memory.consolidate_after_days` | `30` | Age after which `memory_consolidate` takes records and daily files. |
| `memory.write_access` | `enabled: true`, `allowed_roles: ["system"]` | Who may write global memory: administrators, callers whose role is in `allowed_roles` (`system` is the Web UI administrator) and the users in `allowed_users`. With `enabled: false`, only administrators may. SOP, cron and webhook runs never write global memory. |

`memory.mode` and `memory.search.enabled` are accepted but not read: memory is saved the same way whatever `mode` says, and search is semantic when `memory.search.provider` is set and keyword-only otherwise.

## Privacy and Isolation

The agent respects your privacy through **Memory Isolation**:
- Your **Personal Memory** is only accessible to you and administrators.
- When the agent searches for information, it filters out personal files belonging to other users.
- Each platform (Discord, MS Teams, Facebook, Telegram, the App API, Web) uses a unified identity to ensure your preferences follow you across channels.

## Core Instructions

Administrators can set **Core Instructions** for the agent. These are persistent rules that survive session resets and define the agent's fundamental behavior, persona, and boundaries. Unlike regular memory, these cannot be overridden by standard users during a conversation.

================================================================================
# FILE: https://goclaw.ulab.mn/docs/user-guide/wikijs.md
================================================================================

# Wiki.js as Org Knowledge

If your organization keeps its knowledge in [Wiki.js](https://js.wiki/), GoClaw can read it
from there instead of from its own [org wiki](../architecture/memory.md#org-wiki). A sync job
copies the pages through the Wiki.js API. The agent finds them with `memory_search` and reads
them with `wiki_read`, and SOP and cron runs find them too. Wiki.js stays where the pages are
written: in GoClaw they are read-only copies, refreshed on a schedule.

GoClaw's own org wiki is still there for a site without Wiki.js. Both can be used at once.

Supported: Wiki.js 2.x (tested with 2.5). Wiki.js 3.0 is still in beta and is not supported yet.

## How it works

- One sync of a site makes one `pages.list` GraphQL call. That call returns every page the API
  key may see, with its last update time.
- The sync compares that list with the copies it already holds. It downloads only the pages
  that are new or changed, each from `GET /d//` (the page's source).
- Pages deleted on Wiki.js, or no longer visible to the key, are removed from the copies.
- Drafts (unpublished pages) and private pages are not copied.
- Pages written in the visual editor (HTML) are converted to Markdown. Pages over
  `max_page_bytes` (default 1 MiB) are cut, and `wiki_read` says so.
- The copies are kept in the `wikijs_pages` table. A searchable copy of each page goes into
  memory (scope `wikijs`), indexed separately for each site.
- A sync makes no LLM calls. Only the embedding of new or changed text costs anything, and
  only when embeddings are configured.

Wiki.js 2.x sends no change notifications, so GoClaw polls: every 30 minutes by default. The
**Sync now** button on the Integrations page syncs a site at once.

GoClaw does not use the Wiki.js search. Its default engine matches page titles and
descriptions only, not the text, while GoClaw indexes the text itself, Mongolian included.

## Set up Wiki.js

1. **Administration → API Access**: turn on **Enable API**. Until it is on, every key fails.
2. **Administration → Groups**: create a group for GoClaw, for example `goclaw-reader`.
   - **Permissions**: `read:pages` and `read:source`. Nothing else.
   - **Page Rules**: allow the paths GoClaw should read, and deny the rest, for example HR pages.
3. **Administration → API Access → New API Key**: choose the group from step 2, not
   *Full Access*, and set an expiry.

GoClaw sees exactly what that group sees. It **refuses a key that can do more than read**: a
full-access key, or one whose group can edit (`write:pages`, `manage:pages`) or delete
(`delete:pages`) pages. The Test button and every sync check this with read-only queries.

A page the group may list but whose source it may not read (`read:pages` without `read:source`
on that path) is left out, with a warning on the Integrations page.

## Configure GoClaw

On the **Integrations** page, choose **Add Integration → Wiki.js**. Enter the URL and the API
key; **Store in vault** keeps the key in the secrets vault. Press **Test**: it lists the pages
the key sees and says how many of them would be copied. Then save.

The same definition in `config.yaml`:

```yaml
tools:
  wikijs:
    docs:
      url: "https://wiki.example.com"
      token: "${secret:WIKIJS_DOCS_TOKEN}"
      description: "Handbook and runbooks"
      locales: ["mn", "en"]         # empty: every locale
      include: ["ops", "policy"]    # path prefixes; empty: every path
      exclude: ["ops/archive"]      # wins over include
      sync_interval: "30m"          # a duration, or "off" for Sync now only
      # agents: ["agent-ops"]       # only these agents see the pages; empty: all
      # admin_only: true            # only admins and SOP/cron runs
      # max_page_bytes: 1048576
      # insecure: true              # self-signed certificate
```

Prefixes match on path segments: `ops` covers `ops` and `ops/runbook`, but not `operations`.

The Wiki.js host does not need to be on `tools.http.allow`. As with the other integrations,
GoClaw may reach the configured `url`, while `tools.http.deny` still applies, and a redirect
to another host stays under the HTTP policy. The key is sent only in the `Authorization`
header.

## Who sees the pages

Wiki.js API keys belong to a group, not to a person, so GoClaw cannot apply each person's
Wiki.js rights. Each site definition is one group's view, and GoClaw decides who that view is
shown to:

- By default, every user of every agent, like GoClaw's own org wiki.
- With `agents`, only the agents listed.
- With `admin_only: true`, only admins and SOP and cron runs. Other users and delegated runs
  do not see the pages, in `memory_search` or in `wiki_read`.

For pages only some people should read, make a second group and key with the stricter page
rules, and add it as a second site with `admin_only` or its own `agents`.

## Reading the pages

A copied page's slug is `wikijs///`. Each search result includes it:

```
wiki_read slug="wikijs/docs/mn/ops/runbook"
wiki_read prefix="wikijs/docs/mn/ops"        # list a folder
wiki_read prefix="wikijs"                    # list the sites
wiki_read slug="wikijs/docs/mn/ops/runbook" refresh=true   # fetch it from Wiki.js now
```

`wiki_read` with no slug lists the org wiki pages and then the Wiki.js sites. The page text
ends with a link to the page on Wiki.js. Saved queries and tool presets can link a copied page
in their wiki field, like an org wiki page.

`wiki_write` refuses these slugs: edit the page on Wiki.js, and the next sync copies it.

## Status and troubleshooting

The Wiki.js card on the Integrations page shows how many pages are copied, the last sync and
its warnings. The gateway log has one line per sync:

```
[wikijs] docs: synced — 120 listed, 96 mirrored, 3 fetched, 3 updated, 1 deleted, 0 warning(s) in 1.2s
```

| Message | What to do |
|---|---|
| `API is disabled` | Turn on **Enable API** under Administration → API Access. |
| `API Key is invalid or was revoked` | Create a new key and save it. |
| `this API key administers the site` / `can edit pages` / `can delete or manage pages` | Use a key for a group with `read:pages` and `read:source` only. |
| `listed no pages for this API key; the … mirrored pages are kept` | The key sees nothing any more. The copies are kept on purpose, so that a revoked key does not empty the org knowledge. Fix the key or its group's page rules. |
| `… is listed but its source is not readable` | Give the group `read:source` on that path, or deny the path entirely. |

If you remove a site from `config.yaml` or from the Integrations page, its copies are dropped
as well.

================================================================================
# FILE: https://goclaw.ulab.mn/docs/user-guide/token-budgeting.md
================================================================================

# Token Analytics & Hard Budgeting

GoClaw records the tokens every request uses and can refuse new requests once a token limit is reached. Because GoClaw agents act autonomously and can spawn long-running SOPs or Subagents, keeping a hard limit on token use is how you keep API costs under control.

## 1. Token Analytics (Usage Tracking)

When a request finishes, GoClaw records the token counts the provider (e.g., OpenAI, Anthropic, Gemini) reported for it. Background work that calls a model, such as session summaries, fact extraction, embeddings and the image and PDF tools, is recorded the same way.

Each record goes into the `token_usage` table of the state database, with:
- **`user_id`**: The person: their linked ID when they are paired, else their ID on the channel the request came from; for a SOP or cron run, its principal (`cron-system`, `sop-system`); `system` for background work.
- **`agent_id`**: The agent that answered.
- **`session_id`**: The conversation or job the request belonged to; for a SOP run with no conversation, its principal (see the SOP note below).
- **`channel`**: Where it came from (Discord, MS Teams, Facebook, Telegram, App API, Web, or `internal:…` for background work).
- **`message`**: The request's text.
- **`tool_calls`**: How many tool calls the request made.
- **`provider`** & **`model`**: The provider and model that answered.
- **`prompt_tokens`** & **`output_tokens`**: The token counts.
- **`created_at`**: When it was recorded (Unix time).

GoClaw records tokens, not money: there is no cost column and no model pricing to configure.

### Viewing Analytics
You can track token use directly in the **GoClaw Web UI**:
1. Open **Usage** in the dashboard's navigation (the **Token Usage & Analytics** view).
2. Filter by agent, channel, provider, model, or date range.
3. Review the daily and hourly charts and the per-agent and per-channel tables to find spikes caused by looping agents or expensive tool-use chains.

## 2. Circuit Breaker (Hard Token Budgeting)

To prevent runaway agents from draining your API credits, GoClaw checks token limits before every model call.

### How it Works
Before every model call, GoClaw adds up the tokens already recorded, plus what the request in progress has used so far, and compares them with the limits that are set, in this order:

1. **Per-agent daily limit**: tokens the agent used today, across all its sessions and the SOP runs that run as it (a SOP run counts for its owner, where `default`, `main` (unless an agent is configured under that ID) and an owner that names no agent are the default agent; for the route's agent when a webhook starts it, and for the conversation's agent when `sop_run` does).
2. **Per-session daily limit**: tokens the request's session used today.
3. **Per-session hourly limit**: tokens the session used in the current hour.
4. **Global daily limit**: tokens all agents used today.

Days and hours are UTC. Which session a SOP run counts against depends on how it started: a run started from a chat (`sop_run`) uses that chat's session; a run started by a webhook or in the background with no request uses `sop-system`. Runs started by their cron schedule use `cron-system` (a job that runs for a person, with `subject_ref`, uses that person), and runs started from the Web UI's **Run** button use `sop-system`.

* **Below the limits**: the request proceeds.
* **A limit reached**: the request is refused before its next model call (a request already running stops there), and the person gets a message such as `⛔ Token budget exceeded for this session. Used … / … tokens today. …`, `⛔ Hourly token limit reached for this session. …`, `⛔ Agent "…" has reached its daily token limit. …` or `⛔ Global daily token budget exceeded. …`.
* **Near a limit**: at the warning threshold (80% by default) a warning is written to the log; the request still runs.

### Configuration
Global limits are set under `ai.token_budget`; an agent can have its own daily limit under `agents.list[].token_budget`. A limit of `0` (the default) means no limit.

```yaml
ai:
  token_budget:
    daily_limit: 5000000          # tokens per session per day
    hourly_limit: 500000          # tokens per session per hour
    global_daily_limit: 50000000  # tokens across all agents per day
    warning_threshold_pct: 80     # log a warning at this percentage of a limit (default 80; 0 = no warnings)

agents:
  list:
    - id: analytics
      token_budget:
        daily_limit: 2000000      # tokens this agent may use per day
        warning_threshold: 0.8    # log a warning at this fraction (default: warning_threshold_pct; 0 = no warnings)
```

A request's tokens are recorded when it ends, so the check before each further call adds the tokens the request has used so far. A request stopped at a limit has what it used recorded too. A request can still go past a limit by its last call: the call is checked before it is made, not cut off halfway.

## 3. SOP Run Limit

Every SOP run has a token limit of its own, on by default: **500,000 tokens**, prompt and output, for every step, attempt and retry of the run together. It is checked before every model call of the run.

- **When a run reaches it**, it stops before its next call. It ends `failed` whatever the step's `failure_mode` says: no retry, no next step, no `on_failure_goto`. The rollback still runs (shell commands use no tokens). The SOP's owner is told through `sop.alerts` how many tokens the run used. A last step that passes as the run reaches the limit completes the run.
- **A cron job whose run stopped at the limit is paused**, with `token_limit` as its last error on the **Cron** page, and the owner is told. Left on, it would spend as much again at every tick. Resume it once the SOP is fixed or the limit raised.
- **A run past 200,000 tokens is marked heavy**: the log says so, and the SOP's run history shows its token count highlighted.

The run history on the **SOPs** page shows what each run, and each step, used.

```yaml
sop:
  max_run_tokens: 500000    # stop a run here; 0 = no limit
  heavy_run_tokens: 200000  # mark a run heavy from here; 0 = never
```

Raise `max_run_tokens` when your SOPs really need more and you are ready to pay for it. `0` turns the limit off, which is not recommended: the cron pause then never happens either.

**How big should a run be?** A rough guide. A model call sends the whole conversation of the step: the system prompt, the tool definitions, the step's messages and the tool results so far. A step that works takes a few calls. Large query results stay out of the conversation (`data_ref`), so a well-made step rarely sends more than 30–40k tokens a call.

| | One model call | One step | One run |
|---|---|---|---|
| Normal | up to 30k | up to 60k | up to 200k |
| Heavy, worth a look | 30k–80k | 60k–150k | 200k–500k |
| Something is wrong | over 80k | over 150k | over 500k |

A run of millions of tokens is almost always a step that loops: it calls tools until the 20-call limit (`tools.max_tool_iterations`), sends large tool results back in every call, and is then tried again. Look at the run's trace on the **Traces** page: how many model calls each step made, and how large each one was.

================================================================================
# FILE: https://goclaw.ulab.mn/docs/architecture/index.md
================================================================================

# GoClaw Architecture Documentation

GoClaw is designed as a secure, high-performance cognitive multi-agent gateway. It is compiled into a single binary with zero external runtime dependencies.

## Key Subsystems

- **[Cognitive Memory & RAG](memory.md)**: Memory scopes and access, what goes into the prompt, hybrid search, fact extraction and the memory lifecycle.
- **[Memory Engine Architecture](memory_engine.md)**: The engine's internals: tables, ranking formulas, chunking, the session summarizer and memory flush, and the background job queue.
- **[Tools Engine](tools_engine.md)**: How tools and skills are registered, selected for a request (Tool and Skill RAG), cached and run, and the admin hooks and plugins around them.
- **[Ecosystem & Tech Stack](techstack.md)**: Explains the internal modules, databases, LLM providers, and the event-driven actor model.
- **[Secrets Vault](secrets.md)**: Secure credentials management using local AES-256 vault configurations and reference resolutions.
- **[Autonomy Levels & Security](autonomy_levels.md)**: Highlights the security boundaries of GoClaw, detailing the 5 levels of AI autonomy and the Zero-Trust execution model.
- **[Subagents & the Job Queue](subagents_and_jobs.md)**: How a parent agent delegates scoped tasks to child agents, the spawn policy, and the SQLite-backed job queue, EventBus, and workers behind async execution.

## Agent Runtime

- **[Agent Runtime](concepts/agent.md)**: The agent loop, providers, the system prompt, bootstrap files, the workspace and the canvas.
- **[Event-Driven Runtime](event_driven_actor.md)**: How the agent loop moved from blocking calls to event streams: the bounded tool worker pool, the single-writer WebSocket, cancellation, and one turn per session.
- **[LLM Providers](providers.md)**: The chat and embedding providers and the OpenAI-compatible API.
- **[Gemini API Implementation](gemini_implementation.md)**: The native Gemini loop: request configuration, thinking, sampling, multimodal input, Google Search and tool schema conversion.
- **[Provider Fallback & Cooldown](provider_fallback.md)** and **[Model Failover](concepts/model-failover.md)**: When an agent's provider fails before the reply has used a tool, the request moves to the next provider of `ai.fallback` (or the agent's own list), with cooldowns for providers that keep failing.
- **[LLM Token Optimization](llm_optimization.md)**: Where tokens go in the tool-call loop, and what Skill RAG, Tool RAG and result truncation save.
- **[Sessions](concepts/sessions.md)**: Session keys per channel, storage, lifecycle and how sessions are inspected.
- **[User Identity & Context](user_identity.md)**: How the current user's name and role are resolved and shown to the agent.
- **[Identity & Memory](concepts/identity_and_memory.md)**: How one person is recognised across channels, and how their personal memory is kept.
- **[Memory Formation Policy](memory_policy.md)**: What the agent remembers from a message, and under whose authority.
- **[SOP Virtual Level](sop-virtual-level.md)**: The model of delegated permissions a SOP runs under for a user.

## Channels, Gateway & APIs

- **[Channel Router & Pairing](channel_router.md)**: The `Channel` interface, inbound message metadata, agent routing priority, pairing, and how to add a channel.
- **[Multi-Agent Routing](routing.md)**: Bindings that send a channel, server, team or role to an agent, and identity links.
- **[Protocol](protocol.md)**: The gateway's HTTP and WebSocket protocol and its endpoints, compared with OpenClaw's.
- **[OpenAI-Compatible API](openai_compat.md)**: How `/v1/chat/completions` maps onto an agent run.
- **[Media Pipeline](media.md)**: Attachments, uploads, the media store and canvas files.
- **[Web UI / Control Panel](web-ui.md)**: The dashboard's pages and how it talks to the gateway.
- **[Apps](apps.md)**: The voice and desktop features (built, and disabled) and the mobile and menubar apps (planned, not built).

## Operations

- **[Debug & Observability](observability.md)**: Debug logging, prompt capture, live agent traces, and the SRE tools (Graylog, Uptrace, Zabbix, Grafana, Wazuh).
- **[OpenTelemetry Logging Pipeline](otel_logging.md)**: Where the gateway's logs go: console, file, the Web UI and an OTLP exporter.
- **[Runtime Info & Ingress Metrics](runtime_metrics.md)**: The in-process status the Web UI and the CLI read: gateway, ingress, the access log and the webhook audit log.
- **[CLI & Daemon](cli-daemon.md)**: The `goclaw` commands and how the gateway runs as an OS service.
- **[Distribution & Auto-Update](distribution.md)**: The single binary, Docker image, service templates and updates.
- **[Ecosystem](ecosystem.md)**: The Go plugin SDK, gateway hook events, the skills registry and the cron system.
- **[Minimal Dependency Strategy](minimal-dependency.md)**: Why GoClaw is one binary with no runtime dependencies, which tools need something from the OS, and the rules for contributors.
- **[Testing](testing.md)**: How the test suite is organised.

================================================================================
# FILE: https://goclaw.ulab.mn/docs/architecture/techstack.md
================================================================================

# System Specifications & Technology Stack

GoClaw is a lightweight, high-performance **Autonomous AI Agent Gateway and Multi-Agent Orchestrator** compiled as a single binary. It is engineered to run on consumer hardware, local home servers, and NAS environments.

---

## 1. System Specifications

- **Binary Size:** about 110–155 MB, one self-contained executable (the size depends on the platform).
- **Memory Footprint:** Depends on the workload: browser sessions, document parsing and the memory index use the most.
- **CPU Overhead:** Extremely low idle footprint; spikes during local embedding extraction or concurrent tool runs.
- **External Dependencies:** Zero required runtime runtimes (no Python, Node.js, or Docker required by default, though Docker is supported).
- **Build Requirements:** Requires CGO compilation due to embedded SQLite database and tree-sitter AST parser libraries.

---

## 2. Component Architecture

GoClaw is organized into several logical system boundaries that separate user interactions, agent execution, and system state:

```mermaid
flowchart LR
    CH["Chat Channels
Discord, MS Teams, Web, …"] --> GW["AI Gateway
WebSocket / REST"]
    GW --> RT["Agent Router
Orchestrator"]
    GW --> DB["Session State
SQLite / Postgres / MySQL / ClickHouse"]
    RT --> POOL["Agent Pools
Isolated Soul / Workspace"]
```

- **Gateway Manager:** Manages real-time WebSocket communication with the Web UI and processes incoming REST API calls and webhooks. Serves an OpenAI-compatible endpoint (`/v1/chat/completions`). Enforces API rate limiting and authenticates requests.
- **Agent Orchestrator:** Manages the lifecycle of multiple independent agents, coordinates message routing, and delegates work to background tasks.
- **State Database:** A flexible persistence engine via GORM supporting SQLite (default, WAL mode), ClickHouse, PostgreSQL, and MySQL. It persists chat sessions, memory, SOPs and their runs, cron jobs, agent traces, token usage and metrics. The credentials `config.yaml` refers to are not kept there: they are in the encrypted vault (`credentials/secrets.enc` in the state directory), and pairings in JSON files. A webhook route made on the Webhooks page keeps its secret and token in the database, as typed (a `${secret:NAME}` reference keeps the value in the vault).
- **RAG & Memory Engine:** Handles document chunking, indexing, and hybrid searches (semantic vectors + keyword) to provide agents with contextual long-term memory.
- **Channel Adapters:** Connects the gateway to messaging platforms (Discord, Microsoft Teams, Facebook Messenger, Telegram) and to mobile or web apps through the App API, and handles session pairing.
- **MCP Client:** Connects agents to external MCP servers (stdio or HTTP) and offers their tools alongside the built-in ones.

---

## 3. Supported Database Engines
GoClaw includes built-in driver layers for both internal state persistence (GORM) and external agent query datasources:

- **SQLite:** Default internal state engine (WAL mode) and local agent query storage.
- **ClickHouse:** High-performance columnar analytics database supported both for internal GORM state/metrics and as an external query datasource.
- **PostgreSQL:** Enterprise relational database supported for internal GORM state and agent queries.
- **MySQL / MariaDB:** Relational database support for internal state and agent queries.
- **MongoDB:** NoSQL document database query support for agents.

---

## 4. Built-in Agent Tools

GoClaw's tools are compiled directly into the binary; `goclaw tools list` lists them. Agents dynamically request access to these tools depending on their allowed roles and current tasks:

| Category | Description | Primary Tools |
|---|---|---|
| **File Operations** | Read, write, search, and patch files in sandboxed workspaces. | `file_read`, `file_write`, `apply_patch`, `glob_files` |
| **Code Analysis** | Generate AST structure maps and analyze source code files. | `repo_map`, `ast_declarations_search`, `dependency_map` |
| **Version Control** | Execute local Git actions and review repository states. | `git`, `auto_commit`, `diff` |
| **Collaboration** | Call GitLab APIs to read issues, pipeline status, and MRs. | `gitlab` |
| **Databases** | Securely query PostgreSQL, MySQL, ClickHouse, and MongoDB. | `db_query`, `mongo_query` |
| **Communication** | Route outgoing messages, alerts, emails, and notifications. | `message`, `email`, `notify` |
| **Memory & RAG** | Read and write from scoped long-term memory. | `memory_search`, `memory_write`, `memory_consolidate` |
| **Web & Automation** | SSRF-safe HTTP calls and headless Chrome browser control. | `http_request`, `browser` (actions: navigate, screenshot, content, click, type, …) |
| **Observability** | Search system logs, traces, APM alerts, and inventory data. | `grafana_alerts`, `graylog_query`, `uptrace_query`, `zabbix_query`, `netbox` |
| **Media & Documents** | Speech-to-text and text-to-speech, read PDFs, and analyze images. | `voice_input` (speech-to-text), `voice_output` (text-to-speech), `pdf`, `image` |
| **Scheduling** | Define and manage automated workflows and periodic jobs. | `cron`, `sop_run`, `sop_save`, `sop_list`, … |

---

## 5. Runtime Concurrency & Execution Limits

To safeguard the host server from resource depletion, GoClaw implements bounded concurrency pools and loop detection parameters:

### Concurrency Allocations

| Parameter | Configuration Key | Default Value | Description |
|---|---|:---:|---|
| **Tool Worker Pool** | `tools.worker_pool_size` | `8` | Maximum number of concurrent tool executions across all sessions. |
| **Backpressure Queue** | `tools.worker_queue_size` | `64` | Depth of the queue holding pending tool executions. |
| **Output Writer** | - | `1` | Single-writer pump per WebSocket connection to guarantee message order. |

### Execution Thresholds

| Parameter | Configuration Key | Default Value | Description |
|---|---|:---:|---|
| **Max Tool Iterations** | `tools.max_tool_iterations` | `20` | Maximum number of continuous tool runs permitted in a single chat turn. Prevents infinite loops. |
| **Result Spill** | `tools.max_result_chars` | `8000` | A tool result longer than this is not passed to the model whole: the full result is kept in the output store for 1 hour and the model gets a preview with a `data_ref`, which it reads further with `read_output`. `db_query` results are always stored this way. |
| **Full Result Rounds** | `tools.full_result_rounds` | `4` | Within one turn, the results of this many newest tool rounds are sent to the model whole; an older result over 2,000 characters is cut to its start and the `read_output` ref that reads it whole. `-1` keeps every result whole. |
| **Dynamic Tools (RAG)** | `tools.rag.top_k` | `12` | Maximum number of dynamically matched tools injected into the prompt. |
| **RAG Threshold** | `tools.rag.min_score` | `0.15` | Minimum matching confidence score required to auto-enable a tool for a query. |

---

## 6. Deployment Models

GoClaw is highly flexible and can be deployed in multiple environments:

- **Standalone Binary:** Compile for target OS (macOS, Linux, Windows) and run directly.
- **Containerized Daemon:** Run using `docker-compose.yaml`. It mounts `./config.yaml` at `/app/config.yaml` and a named volume, `goclaw-state`, on the state directory `/home/goclaw/.goclaw` (database, vault, workspace, logs). The mounted `config.yaml` must set `gateway.bind: lan`, or the gateway listens on the container's loopback and the port mapping reaches nothing. Set `GOCLAW_SECRET_KEY` before the vault is first written: without it the vault key is derived from values that are the same in every container of the image, so anyone with the image can derive it.
- **Ansible Automation:** Deployment playbooks are provided under `/ansible` for multi-node setups.
- **Remote Ingress Tunnel:** Supports built-in Ngrok tunneling to expose the local webhook receiver securely without requiring manual firewall rule modifications.

================================================================================
# FILE: https://goclaw.ulab.mn/docs/architecture/routing.md
================================================================================

# Multi-Agent Routing System

How GoClaw decides which agent answers an inbound message: `ResolveAgent` matches the channel, peer, thread, tenant, roles and account against the routing rules in eight priority tiers, then builds the session key.

`internal/routing/routing.go` + `internal/routing/identity_links.go` + `internal/routing/routing_test.go`

## Architecture

```mermaid
flowchart TD
    IN["Inbound Message → RouteInput
• Channel, AccountID, PeerKind (direct/group), PeerID
• ParentPeerKind, ParentPeerID (thread inheritance)
• TenantID, PlatformRoles[]"]

    RES["ResolveAgent(input) — 8-tier priority matching
1. Peer match (exact channel + peer)
2. Parent peer match (thread inherits parent binding)
3. Tenant + Roles (tenant + role intersection)
4. Roles match (channel role/dept without tenant)
5. Tenant match (tenant without role filter)
6. Account match (account ID)
7. Channel match (channel-wide, no peer filter)
8. Default (config.ResolveDefaultAgentID)"]

    KEY["buildSessionKey(agentID, input)
• ResolveIdentity() → cross-channel session sharing
• Multi-agent: agent:channel:peer
• Single-agent: channel:peer (backward compat)
• Output: AgentID + SessionKey + MatchedBy"]

    IN --> RES --> KEY
```

## Feature Comparison

| # | Feature | OpenClaw | GoClaw | Status |
|---|---------|----------|---------|--------|
| R.1 | Multi-agent routing | ✅ | ✅ `ResolveAgent()` — 8-tier priority | Done |
| R.2 | Peer binding (channel+peer) | ✅ | ✅ Exact peer match (kind + ID) | Done |
| R.3 | **Thread inheritance** | — | ✅ ParentPeerID → inherits parent binding | **GoClaw only** |
| R.4 | **Tenant + Role matching** | — | ✅ Tenant + platform role intersection (Discord guild and role IDs; App API) | **GoClaw only** |
| R.5 | Tenant-wide binding | ✅ | ✅ TenantID match (Discord guild, Teams group conversation) | Done |
| R.6 | Account binding | ✅ | ✅ Account ID → specific agent (the bot on Teams, Telegram and Facebook; the sender on Discord) | Done |
| R.7 | Channel-wide binding | ✅ | ✅ All messages on channel → agent | Done |
| R.8 | Default agent fallback | ✅ | ✅ `config.ResolveDefaultAgentID()` | Done |
| R.9 | **Identity-linked sessions** | — | ✅ `ResolveIdentity()` — cross-channel session sharing | **GoClaw only** |
| R.10 | **Multi-agent session keys** | — | ✅ `agent:channel:peer` format | **GoClaw only** |
| R.11 | **Peer kind normalization** | — | ✅ "channel" → "group" alias | **GoClaw only** |
| R.12 | Debug logging | ✅ | ✅ `debug.Log("routing", ...)` | Done |

### What each channel supplies

A binding can only match what the channel puts on the message:

| Channel | Account ID | Peer ID | Tenant ID | Platform roles |
|---------|------------|---------|-----------|----------------|
| Discord | the sender's user ID | `dm:` in a DM; `:` in a guild channel, for messages, reactions and slash commands alike; the parent is the channel itself, or for a thread the channel it belongs to | guild ID | the member's role IDs |
| MS Teams | the bot's app ID | conversation ID (`;messageid=…` removed) | the conversation ID in a group chat or channel; empty in a 1:1 chat | none |
| Telegram | the bot's ID | chat ID | none | none |
| Facebook | the page ID | sender's page-scoped ID | none | none |
| App API | `app_api` | the employee's user ID | from the caller's identity | from the caller's identity |

A Discord guild message's peer ID includes the user, so a `peer` binding with a bare channel ID matches through the parent: every message in that channel and in its threads. Route a whole Discord server with `tenant_id`. Teams never supplies roles, so a Teams binding with `platform_roles` never matches.

### Config Example

```yaml
agents:
  default: support-bot
  list:
    - id: support-bot
      model: gemini/gemini-2.5-flash
    - id: dev-bot
      model: claude/claude-sonnet-4-5-20250929

bindings:
  - match:
      channel: discord
      peer:
        kind: group
        id: "123456789012345678"           # a Discord channel and its threads
    agent_id: dev-bot

  - match:
      channel: discord
      tenant_id: "112233445566778899"      # a Discord server (guild ID)
      platform_roles: ["998877665544332211"]  # Discord role ID
    agent_id: dev-bot

  - match:
      channel: msteams
      tenant_id: "19:0123456789abcdef0123456789abcdef@thread.v2"  # one Teams group chat or channel
    agent_id: dev-bot

  - match:
      channel: discord                     # the rest of Discord
    agent_id: support-bot
```

**Coverage:** every feature in the table: routing with 8 priorities, identity links, thread inheritance.

================================================================================
# FILE: https://goclaw.ulab.mn/docs/architecture/memory.md
================================================================================

# Memory & Retrieval-Augmented Generation (RAG) Architecture

GoClaw features a multi-tiered memory architecture designed to provide agents with both short-term session recall and long-term persistent context. This system is optimized to operate efficiently on local servers or NAS devices while ensuring multi-user security, high performance, and continuous learning.

> This page is the overview: scopes and who may write them, what goes into the prompt, search settings, the memory lifecycle and administration. The engine's internals (tables, ranking formulas, chunk IDs, the session summarizer and memory flush, the background job queue, where the code lives) are in [Memory Engine Architecture](memory_engine.md).

---

## 1. Storage Layers

GoClaw divides memory storage into two main layers: an embedded database for dynamic, queryable records, and Markdown files for structured archives and core identity.

### State Database (Dynamic Records)
By default the dynamic records live in an embedded **SQLite database**, `gc_state.sqlite` in the state directory (`~/.goclaw/`, or `GOCLAW_STATE_DIR`), running in WAL (Write-Ahead Logging) mode. `database.driver` can put the same tables in PostgreSQL, MySQL or ClickHouse instead. The memory tables are:
- **Core Memories** (`memories`): the content, its scope, a classification domain, and a JSON `metadata` column (provenance, `superseded_by`, …).
- **Memory Chunks** (`memory_chunks`): text segments, with their embeddings when an embedding provider is configured, that search runs on.
- **Memory Tags** (`memory_tags`): one row per tag on a memory (`memory_id`, `tag`), such as `critical`, `superseded` or `unverified`.
- **Knowledge Graph:** entities and relations extracted from memories (`memory_entities`, `memory_relations`). A copy for graph queries is kept in a second database: `kg.sqlite` in the state directory, or PostgreSQL with `memory.graph.backend: postgres`.

### Markdown Files (Identity & Archive)
For bootstrap state and historical logging, GoClaw uses standard text files in each agent's workspace:
- `Memory.md`: Long-term curated knowledge base. Injected into the prompt on every turn (see [Session Context Injection](#3-session-context-injection)); it changes only when someone edits it, runs `memory_consolidate`, or rebuilds it with `goclaw memory replay`.
- `IDENTITY.md`: The agent's core character definition, instructions, and behavior limits.
- `memory/archive/*.md`: Archived historical chat sessions, kept for audit and memory replay.

### Org Wiki
The org wiki holds curated pages of organization knowledge (data dictionaries, runbooks, how systems connect), edited by people on the Web UI's **Wiki** page and by the agent with `wiki_write` (human admins and SOP/cron runs only; `wiki_read` is open to all).
- Every save is a new version in `org_wiki_revisions`; the newest version of a slug is the page, and older versions can be restored.
- The current text of each page is also kept as a memory record with scope `wiki`, so `memory_search` (and SOP and cron runs) find it. Wiki pages are not injected into the prompt and are not subject to the agent-notes budget.
- Files attached to a page are stored in the media store (`/media`, listed in `wiki_attachments`) and served at `/media/`.
- **Distill** (the button on the Wiki page, or `POST /api/memory/wiki/distill`) asks the default agent's model to turn the learned facts in `agent_notes` into pages under `learned/`, saved as agent edits for a person to review. It runs only when triggered.
- The API is `/api/memory/wiki` (with `/history`, `/restore`, `/search`, `/attachments` and `/distill`).
- A site that keeps its knowledge in Wiki.js can have it copied in instead (`tools.wikijs`): read-only copies under the slugs `wikijs///`, searchable as memory scope `wikijs`. See [Wiki.js as Org Knowledge](../user-guide/wikijs.md).

---

## 2. Memory Scopes and Access Control

To maintain security and prevent context contamination, memory is divided into distinct scopes with strict access boundaries:

| Scope | Purpose | Storage | Writer | Reader |
|---|---|---|---|---|
| `personal` | User preferences, custom facts, and private configurations. | Database | That user (`memory_write`), fact extraction from their own messages, and the session summary of a conversation that was theirs alone | Only that specific user |
| `agent_notes` | Organization knowledge: learned facts and preferences, host details, schema notes. | Database | Administrators (`memory_write`, `@notes:`); roles or users listed under `memory.write_access` (`memory_write` only); SOP, cron and webhook runs; fact extraction from an administrator's messages | All users & agents |
| `global` | Shared team guidelines, instructions, and standard values. | Database | The roles and users listed under `memory.write_access` while it is enabled (the default); administrators when it is disabled | All users & agents |
| `agent_config` | Core settings and parameters modifying agent behavior. | Database | Administrators and the roles/users listed under `memory.write_access` (`memory_write`); administrators (`@config:`) | System prompt injector |

`memory_write` refuses any other scope. Three more scopes are written by GoClaw itself: `wiki` (the searchable copy of [org wiki](#org-wiki) pages), `wikijs` (copies of [Wiki.js pages](../user-guide/wikijs.md)) and `db_cache` (schema notes from `db_analyze` / `db_enrich`). `Memory.md` is not a scope: it is a file in the agent's workspace, injected on every turn.

### Security Boundaries
- **Autonomous Job Isolation:** Automated runs (Cron Jobs, SOPs, and Webhooks) cannot write to the `personal` scope, and never read a user's personal memory. This prevents automated tasks from corrupting or polluting a human user's private context.
- **Administrative Gates:** Writing `agent_config` or `agent_notes` needs an administrator or a role or user listed under `memory.write_access`; `global`, while `write_access` is enabled, needs a listed role or user. A regular user is refused and pointed to `personal`. Autonomous runs (SOP, cron and webhook) may write `agent_notes` (they carry out admin-authored procedures) and never `global` or `agent_config`: they work on outside input, which must not become a standing instruction.

---

## 3. Session Context Injection

On every turn, GoClaw assembles a context block for the message (`GetSessionContext` in `agent.go`) and sends it with the conversation, in three groups and in this order:

```
Memory group
 1. Critical constraints (agent notes tagged `critical`, always kept)
 2. Memory.md (a file over 20,000 characters keeps its first 14,000 and last 6,000)
 3. Agent configuration (`agent_config` records, verbatim)
 4. Agent notes: up to 5 notes matching the message, or a hint with the note count
 5. Agent notes that name the current user
 6. LEARNED KNOWLEDGE (learned facts and preferences not shown above)
 7. RECENT MEMORY (global records of the last 7 days, and the user's personal records)
 8. PERSONAL MEMORY (the user's 5 most relevant personal records)
 9. PREVIOUS CONTEXT (the summary of trimmed earlier conversation)
System group
10. Configured datasources, GitLab and SonarQube instances, observability instances, MCP servers
11. The newest memory files (up to 15), and context fetched by message hooks
User group
12. USER.md (only when the request has no identified user)
13. The current user's identity block
```

SOP, cron and webhook runs get no personal records (in 7 and 8) and no PREVIOUS CONTEXT. Tool errors are not stored at all: a failing call gets its hint in the same turn (see [Auto-Learn](#7-autonomous-error-learning-auto-learn)).

### Context Bloat Prevention
To keep the context window efficient, each part has its own cap: RECENT MEMORY takes at most **6,000 characters** of global records and 6,000 of personal records, walking from newest to oldest so the most recent context survives. The whole memory group is then held to `memory.context_budget_chars` (default **32,000** characters; `0` means no limit). Over that budget, the lowest-priority sections are dropped first: the note-count hint, then RECENT MEMORY, PREVIOUS CONTEXT and the notes; critical constraints, agent configuration, LEARNED KNOWLEDGE and Memory.md are dropped last.

---

## 4. Multi-User Isolation

In multi-user environments (such as shared Discord servers or MS Teams channels), GoClaw guarantees strict privacy boundaries:
- **Unified Identity:** Personal memory is bound to a unique, cross-platform user ID rather than a transient chat session. This ensures that whether a user chats via Discord or the Web UI, they access the same private memory bank.
- **Strict Query Filtering:** During semantic or keyword searches, search queries are dynamically filtered. The retrieval engine matches incoming request credentials against memory owner IDs, programmatically blocking any cross-user data leakage.

---

## 5. Background Indexing & Hybrid Search

To maintain instant response times during chat, GoClaw processes all memory-heavy tasks—such as chunking, embedding generation, and indexing—asynchronously in the background.

```mermaid
flowchart LR
    IN["Incoming Write"] --> DB["Write to SQLite"]
    DB --> Q["Add to Background Queue"]
    Q --> W["Worker Processes Chunking & Vector Generation"]
```

### Hybrid Search (Vector + Keyword)
GoClaw utilizes a **Hybrid Reciprocal Rank Fusion (RRF)** search algorithm to find relevant context. Rather than relying solely on semantic vector search, it combines:
1. **Keyword Search:** A lightweight, Unicode-aware search engine that calculates term frequency and inverse document frequency (similar to BM25). This ensures exact terminology, serial numbers, and IP addresses are matched accurately.
2. **Vector Search:** Semantic embeddings generated via Gemini, OpenAI, or Ollama (`memory.search.provider` and `memory.search.model`) to capture conceptual meaning. Without a provider, search is keyword-only.
3. **RRF Ranking:** Results are merged using the RRF formula:
   $$\text{Score} = \frac{1}{60 + \text{Rank}_{\text{Vector}}} + \frac{1}{60 + \text{Rank}_{\text{Keyword}}}$$
   The score is scaled to 0–1 (divided by its maximum, 2/61) before it is compared with `memory.search.min_score` (default `0.35`). `memory.search.max_results` (default `6`) caps the results.

### Evidence Floor
Ranking always puts something first, however little matches. So a result must also show evidence that it answers the query: a cosine similarity to the query of at least the floor, or at least half of the query's terms in its text. The floor is `memory.search.min_similarity` when set; otherwise `0.62` for `gemini-embedding-001`, and no floor for other embedding models (their similarities sit on another scale). A negative `min_similarity` turns the check off.

### Scoring Enhancements
- **Temporal Decay:** To keep context fresh, GoClaw applies an exponential decay multiplier to records older than 14 days. Extremely old records receive a minimum decay floor of `0.4`, ensuring highly relevant but old facts are still discoverable.
- **Access Bonus:** Frequently retrieved chunks receive a small score boost based on their retrieval count, promoting active facts to the top of search results.

### Smart Chunking
Documents are parsed and split into target sizes of **400 tokens** (~1,600 characters) with an overlap of **80 tokens** (~320 characters). The chunker splits text at paragraph, newline, and sentence boundaries to preserve the logical flow of information.

---

## 6. Memory Lifecycle & Consolidation

Memory undergoes a continuous lifecycle of summarization, archiving, and consolidation:

```mermaid
flowchart LR
    CHAT["Active Chat"] --> TRIM["Auto-Trimming & Archiving"]
    TRIM --> FACT["Fact Extraction"]
    FACT --> SCOPE["agent_notes / personal"]
    SCOPE --> CONS["Nightly Consolidation"]
```

### Auto-Trimming & Summarization
When a session's message count or token usage exceeds configured thresholds, GoClaw triggers an automatic cleanup:
1. An LLM compiles a concise summary of the conversation.
2. The raw messages are written to an archive file (`memory/archive/*.md`).
3. Key interaction notes and user preferences are extracted and saved to the `personal` memory scope.

### Nightly Consolidation
The system cron job `sys-memory-consolidate` runs every day at 03:00 and queues a `memory_consolidate` job on the memory worker. It:
1. Removes near-duplicate search chunks (paraphrases and translations of the same text within one source).
2. Merges live `agent_notes` facts and preferences that say the same thing into one record. Each merged source is retired as superseded and recorded in `memory_revisions` (`via` consolidation), so a merge can be undone like any other revision. Unverified, superseded and `critical` records are left out.

It does not touch `Memory.md`, which stays curated by hand.

### Folding Old Memory into Memory.md
`Memory.md` grows only on request, through the `memory_consolidate` tool (admins and SOP/cron runs). It takes the global records and the daily memory files (`memory/YYYY-MM-DD.md`) older than `memory.consolidate_after_days` (default 30) or the call's `lookback_days`, has the model extract the key facts, appends them to `Memory.md` under a dated heading, deletes those global records, moves the daily files to `memory/archived/`, and re-indexes. `dry_run: true` lists what it would take.

### Superseded Knowledge (Belief Revision)
The agent tends to re-learn a topic in slightly different words, and each version used to stay live and be injected. When a learned fact or preference (`agent_notes`) is written, older ones on the same subject are retired, never deleted:
1. **Rules** in `writeLearnedRecord` catch the same IP, a `  is …` value, and a few single-valued preferences (response language, report format, timezone, …).
2. **Belief revision** handles everything else. A `belief_revision` job on the memory worker finds up to three older live records of the same kind that share at least 40% of the new record's words. A small call to the background model (the one fact extraction uses: Gemini where `ai.gemini` has a key, otherwise the default agent's own model) then decides, for each, whether the new record fully replaces it. When unsure, it keeps the old record.

A retired record is tagged `superseded`. Its metadata records `superseded_by`, `superseded_at` and `superseded_reason`, and each decision is a row in `memory_revisions` (`goclaw memory revisions` lists them). Undo one with `goclaw memory revisions undo `, everything one run retired with `goclaw memory revisions undo --run `, or through `POST /api/memory/revisions/undo`. The record returns to the prompt, and the undone decision stands as a veto: the same record never replaces it again. Removing the `superseded` tag by hand records no veto. Superseded records leave the prompt and plain `memory_search`; `memory_search` with `include_history: true` returns them, marked. Records from an unconfirmed session (`unverified`) never take part: they can neither retire a record nor be retired.

`goclaw memory revise` applies the same judge to records stored before this existed, oldest first, so a topic learned in several versions collapses onto its newest. Run it with `--dry-run` first: that only reports what would be retired.

### Memory Replay
If the main `Memory.md` file is corrupted or deleted, administrators can trigger a **Memory Replay**. This utility parses archived chat logs, keeps each turn with its speaker, and regenerates the consolidated core memory base using an LLM. It builds the memory from what people said, decided and confirmed; the agent's own replies are context and can be wrong, so a claim only the agent made is left out or kept under an "Unverified" section.

---

## 7. Autonomous Error Learning (Auto-Learn)

When a tool call fails, GoClaw checks the error against known patterns for that tool (`auto_learn.go`). No model is involved.
1. Covered tools: `db_query`, `db_analyze`, `uptrace_query`, `mcp`, `gitlab`, `zabbix_query`, `http_request`, `message` / `notify`.
2. A pattern that matches produces a hint (e.g., "DB [reports-db]: column `created_at` does not exist in table `users`"). An infrastructure or credential error (network failure, timeout, 401, a login page) is only logged: the agent cannot fix it by changing its approach.
3. The hint is returned with the error, in the same turn, together with up to three matching approved [Saved Queries or Tool Presets](../user-guide/saved-queries.md) and the wiki pages they link to. The agent corrects the call from there.
4. Nothing is stored: no `agent_notes` row is written and no tool description changes, so a later turn does not remember the mistake. What the agent should do every time is written down by an admin, as a saved query, a tool preset or a wiki page.

---

## 8. CLI Administration

Administrators can inspect and maintain memory from the terminal. These commands go through the running gateway's API, so start the gateway first. `goclaw memory check` and `goclaw memory migrate` read the database directly, and so does the backup the CLI takes before `clear`, `clear-user`, `forget` and `migrate`.

```bash
# View memory database statistics
goclaw memory stats

# Export the knowledge catalog (agent_notes and agent_config only) to a JSON file
goclaw memory export --file catalog.json

# Import a previously exported knowledge catalog
goclaw memory import --file catalog.json

# Rebuild the semantic vector index from scratch
goclaw memory reindex

# Remove near-duplicate chunks and agent notes
goclaw memory dedupe --ratio 0.85
```

The catalog export is not a backup: for a portable snapshot of the whole database (and, with flags, sessions, workspace files and wiki attachments) use `goclaw db export` (see [Backup & Maintenance](#9-backup--maintenance)).

---

## 9. Backup & Maintenance

With the default SQLite driver, dynamic memory lives in two database files in the state directory, and a disk failure takes it with them. Schedule periodic backups of:
- `gc_state.sqlite`: memories, sessions, SOPs, cron jobs and the rest of the state (or the file `database.path` names);
- `kg.sqlite`: the knowledge graph (or the file `memory.graph.path` names);
- the workspaces (`workspace/` and `agents//workspace/`): `Memory.md`, `IDENTITY.md`, daily memory files and session archives;
- `credentials/`: the secrets vault (`secrets.enc`) and the pairing files, and `config.yaml`. Without them a restore loses every `${secret:...}` credential and every pairing.

```bash
mkdir -p ~/goclaw-backups

# Run a safe online backup of the SQLite databases while the gateway runs
sqlite3 ~/.goclaw/gc_state.sqlite ".backup $HOME/goclaw-backups/gc_state-$(date +%Y%m%d).sqlite"
sqlite3 ~/.goclaw/kg.sqlite ".backup $HOME/goclaw-backups/kg-$(date +%Y%m%d).sqlite"

# Copy the workspaces (add "agents" when you run more than one agent)
tar czf ~/goclaw-backups/workspace-$(date +%Y%m%d).tar.gz -C ~/.goclaw workspace
```

With `database.driver` set to PostgreSQL or MySQL, back up that database with `pg_dump` or `mysqldump` (and the graph's PostgreSQL database, if `memory.graph.backend` is `postgres`).

For a portable snapshot that `goclaw db import` can load on another machine, use `goclaw db export --output backup.gclaw` (add `--sessions`, `--files`, `--attachments` and the other flags for more; see `goclaw db export --help`).

The destructive memory commands (`goclaw memory clear`, `clear-user`, `forget`, …) first save a copy of `gc_state.sqlite` to `/backups/gc_state_.sqlite`. They skip this for other drivers.

================================================================================
# FILE: https://goclaw.ulab.mn/docs/architecture/memory_engine.md
================================================================================

# Memory Engine Architecture

The Memory Engine is the core component of GoClaw responsible for persistent long-term knowledge, session-based context management, and semantic retrieval. It provides a hybrid search system that blends traditional keyword search with vector-based semantic retrieval.

> This page covers the engine's internals. Memory scopes and who may write them, what goes into the prompt, the search settings and the memory lifecycle are described in [Memory & RAG](memory.md).

## System Components

### 1. Multi-Tiered Storage
The memory system uses a combination of structured and unstructured storage:
- **Markdown Storage**: Used for human-readable long-term bootstrap (`Memory.md`) and session archives (`memory/archive/`).
- **SQLite DB**: The unified persistent store (`internal/store`, `gc_state.sqlite`, WAL mode; PostgreSQL, MySQL or ClickHouse with `database.driver`).
  - `memories` table: All records (global, personal, agent_notes, agent_config, and the internal `wiki` and `db_cache` scopes).
  - `memory_chunks` table: Indexed text fragments with vector embeddings.
- **Graph DB**: `kg.sqlite` in the state directory (or PostgreSQL with `memory.graph.backend: postgres`), holding `graph_entities` and `graph_relations` for graph queries.
- **Session Archive**: Full transcripts of past conversations stored in `memory/archive/`.

### 2. Hybrid Search Engine (RRF)
The engine uses **Reciprocal Rank Fusion (RRF)** to combine results from two distinct search algorithms:
1. **Vector Search (Semantic)**: Uses AI-generated embeddings (OpenAI, Gemini, or Ollama) to find conceptually similar content. It calculates **Cosine Similarity** between query and chunk vectors.
2. **Keyword Search (BM25-like)**: A pure Go implementation of keyword-based scoring with **IDF (Inverse Document Frequency)** and tokenization.

The final score for each chunk is calculated as:
`RRF Score = 1 / (60 + rank_vector) + 1 / (60 + rank_keyword)`.

#### Rank Enhancement Logic
To ensure the most relevant and "fresh" information surfaces:
- **Temporal Decay**: For chunks older than 14 days, a decay factor is applied: `e^(-0.015 * (days - 14))`, with a floor of 0.4.
- **Access Bonus**: Frequently retrieved chunks (hits in `memory_search`) get a small boost: `+0.002 * access_count` (capped at 10 accesses).

### 3. Chunking Strategy
- **Smart Delimiters**: Instead of fixed character counts, the system splits content by paragraphs (`\n\n`), then single lines, and finally sentences to preserve semantic integrity.
- **Size**: Targets ~400 tokens per chunk (~1600 characters).
- **Overlap**: 80 tokens are overlapped between adjacent chunks to maintain context.
- **Deterministic IDs**: Chunk IDs are generated using `sha256(source + startLine + content)`, allowing the system to reuse cached embeddings when unchanged content is re-indexed.

### 4. Summarization Pipeline (Session Summarizer)
The engine automatically handles context window limitations through the **session summarizer** (`session_summarizer.go`), which runs after a turn:
1. **Trigger**: When the session's history exceeds `memory.session_memory.max_tokens` (default 40,000) or, with that set to 0, `memory.session_memory.messages` (default 40). While the memory flush is on, the history kept is also held to half its threshold (`sessionTokenLimit`): a turn starts with that history, and one that started past the threshold would be flushed at its first call and answered without any of it. The limit is computed over the whole history, so a session however far over it is back within it after one summarize.
2. **Summarization**: An LLM (typically Gemini) generates a summary of at most 500 characters of the oldest part of the conversation.
3. **Compaction**: The session history is trimmed to the newest messages that fit, and the summary is carried into later turns as PREVIOUS CONTEXT. Each trim adds a paragraph to the summary; it keeps its newest 4,000 characters (`session.MaxSummaryRunes`), so it stays within the memory budget of the prompt.
4. **Archival**: The full text of the trimmed messages is saved to the session archive (`memory/archive/`) for later retrieval. When every trimmed message was the current user's, the summary is also saved to their personal memory; a shared thread's summary is not, since it is of what everyone asked.
5. **Fact extraction**: Each person in the trimmed messages is read on their own lines, under their own role (see [Memory Formation Policy](memory_policy.md#shared-conversations-each-person-on-their-own-lines)).

**Memory Flush** (`memory_flush.go`) is separate and works inside one request: when the messages of a single agent loop exceed `memory.auto_flush_tokens` (default 80,000), the older messages are handed to the model to extract the facts worth keeping, which are saved to the agent's daily memory file, and the loop goes on with the system prompt, the current user message and the last 6 messages. The model is shown each turn labelled Person, Assistant or Tool, and keeps what a person said and what a tool returned; the agent's own replies can be wrong and are not facts, so one is left out or, when a person relied on it, saved behind `Unverified, the agent's own earlier answer:`. Without a model, the flush writes what people said and what tools returned, never the agent's replies.

Both limits count tokens the same way (`estimateTextTokens`): a digit or a non-Latin letter is one token, other text four characters a token. Counting bytes undercounted Cyrillic (two bytes a letter, about one token) and figures (Gemini spends a token on each digit) by about half.

### 5. Memory Job Queue & Background Processing
Heavy memory operations are decoupled from the main execution thread via a persistent **Job Queue**:
1.  **Serialization**: Tasks like fact extraction, belief revision and knowledge graph synchronization are queued as `Job` records in the state database.
2.  **MemoryWorker**: A dedicated background worker (`internal/agent/memory_worker.go`) processes these jobs sequentially.
3.  **Stability**: Sequential processing prevents SQLite "database is locked" errors and ensures LLM API rate limits are respected.

### 6. Auto-Learning & Knowledge Graph
- **Fact Extraction**: Automatically identifies and saves environment facts and user preferences to `agent_notes`.
- **Knowledge Graph**: LLM-based extraction of entities and relations, stored in `memory_entities` and `memory_relations` for future causal reasoning.
- **Consolidation**: The nightly `memory_consolidate` job (`sys-memory-consolidate`, 03:00) removes near-duplicate chunks and merges `agent_notes` records that say the same thing; each merge is recorded in `memory_revisions` and can be undone. It does not touch `Memory.md`: the `memory_consolidate` tool folds old global records and daily files into `Memory.md` when an admin asks for it.

## Implementation Details

- **MemoryWorker**: The central orchestrator for background memory tasks.
- **Job Types**: `fact_extract`, `graph_extract`, `graph_sync`, `memory_consolidate`, `belief_revision`. `auto_learn_error` is no longer queued; rows left from an older version are drained as no-ops.
- **Hooks**: Uses `GraphExtractHook` to allow the `memory` package to trigger jobs in the `agent` package without circular dependencies.
- **Indexing**: Incremental indexing (`IndexFileAsync`) is used to avoid system-wide locks during writes.
- **Search Optimization**: An `idMap` (O(1) lookup) is used to quickly retrieve specific chunks by ID.
- **In-Memory Store**: Chunks and their embeddings are kept in memory for high-performance searching. They are persisted in the `memory_chunks` table and loaded into memory at start-up (warm-up). A `memory-index.json` left by older versions is deleted at start-up.


## Developer Notes
- **Embedding Provider**: Defined in `internal/memory/embedding.go` (OpenAI, Gemini, Ollama), chosen by `memory.search.provider`; none means keyword-only search.
- **Memory Retrieval**: `hybridSearch` in `internal/memory/store.go`, `Indexer.Search` in `internal/memory/indexer.go`, and the `memory_search` tool in `internal/agent/tool_memory_search.go`. (`internal/agent/rag.go` selects tools and skills, not memory.)
- **Summarization**: Located in `internal/agent/session_summarizer.go`.
- **Auto-Learning**: Located in `internal/agent/auto_learn.go`.

================================================================================
# FILE: https://goclaw.ulab.mn/docs/architecture/tools_engine.md
================================================================================

# Tools Engine Architecture

The GoClaw Tools Engine is a modular system for managing function-calling (tools), context injection (skills), event handling (hooks), and external extensions (plugins).

## 1. Tool Execution (Function-Calling)

Tools are registered via the `Skill` interface in Go. Each tool defines its own JSON schema (parameters) and implementation logic.

### Interface
```go
type Skill interface {
    Definition() llms.Tool    // JSON schema for the tool
    Execute(ctx context.Context, args map[string]any) (string, error)
}
```

### Concurrent Execution via WorkerPool
When an LLM requests multiple tool calls in a single response, GoClaw executes them **concurrently** using a bounded **WorkerPool** (default: 8 workers, 64 queue). This prevents goroutine explosion and provides back-pressure under heavy load. Results are collected via channels and returned as a single response to the model.

### Which Tools a Run Is Offered
`GetTools(ctx)` builds the list from the registry and the run's `RuntimeScope`; `GetToolsFor` then narrows it per request (section 3).

- **Per-agent settings:** an agent's `tools:` block in `agents.list` (shell, git, HTTP, GitLab, SonarQube, observability) becomes `RuntimeScope.Tools` and replaces the matching global setting. An agent's `skills.allow` narrows its tools as well as its skills; core tools stay offered.
- **Declared tool sets:** a SOP step, or a subagent whose profile lists tools, gets exactly those tools plus the side-effect-free internal tools (`read_output`, `json_transform`, `datetime`, `calc`, `regex`, `text_utils`), and no RAG selection runs.
- **Run cap:** `RuntimeScope.ToolCap` (the tools a subagent run was started with) is applied first, with no exceptions, to the run and every follow-up.
- **Role gates:** `db_write` is offered only to a privileged caller when some datasource has `allow_write: true`; `sessions_list` and `sessions_history` only to a human admin; `search_tools` only where tool RAG selects and outside SOP steps.
- **Execute:** `Execute()` refuses a tool that was not offered for this request, naming the nearest tools that were. While plan mode is on (`enter_plan_mode` until `exit_plan_mode`), it refuses write tools in the conversation that turned it on (that agent and session); other agents and sessions are not affected.

### Sandboxing & Security
- **File System:** All file operations are sandboxed to the workspace directory. Path traversal is strictly blocked.
- **Shell:** Command execution is restricted by a prefix-based allowlist (`tools.shell.allow`; empty means no shell at all) and a built-in deny-list of dangerous patterns (e.g., `rm -rf /`, `shutdown`, `ansible`). An allowed command is still refused if it contains a shell metacharacter (`;`, `|`, `&`, `$`, `>`, `=`, …), so commands cannot be chained, piped or redirected. An agent's own `tools.shell` lists replace the global ones.
- **Database:** `db_query` runs only `SELECT`, `SHOW`, `DESCRIBE`, `EXPLAIN` and `WITH` statements, and refuses one that hides a write (a data-modifying CTE, `EXPLAIN ANALYZE` of a write, `SELECT … INTO`). `db_write` changes rows only on datasources with `allow_write: true`, for admins and admin-authored SOP/cron runs, at most `tools.max_db_write_rows` rows per call.
- **Outbound HTTP:** `SafeHTTPClient` applies `tools.http` (default deny, allow/deny lists, private-address blocking) at dial time. See [Security](security.md#http-default-deny-policy).

---

## 2. Skills System (Context Injection)

Skills provide specialized domain knowledge to the LLM via system prompt injection. Unlike tools, they are passive context providers.

### Architecture
```
System prompt   = core instruction + IDENTITY.md + SOUL.md + AGENTS.md + instructions
                  + TOOLS.md + WORKSPACE + SKILLS
Session context = critical constraints + Memory.md + notes, lessons and personal memory + …
```

`GetStaticSystemPromptFor` puts the admin's core instruction first. A subagent's profile takes the place of `IDENTITY.md` and `SOUL.md`. `Memory.md` is not in the system prompt: it is part of the session context (`GetSessionContext`), whose sections are dropped lowest-priority first when they exceed `memory.context_budget_chars`.

### Loading & Priority
Skills are loaded from four sources, with later sources overriding earlier ones of the same name:
1. **Built-in:** Compiled into the binary.
2. **Plugins:** the skills of the agent's [plugins](#agent-plugins) (the first plugin listed wins between plugins).
3. **Workspace:** `/skills/` (`~/.goclaw/workspace/skills/` for the default agent)
4. **User:** `~/.goclaw/skills/`

### Frontmatter Support
Skills can include YAML frontmatter for metadata, tool requirements, and tags.

---

## 3. RAG Engine Integration

GoClaw utilizes a multi-layered RAG (Retrieval-Augmented Generation) engine to select relevant tools and skills for each user query (`GetToolsFor`, `FilterSkillsFor` in `rag.go`; `tools.rag`).

- **Keyword & Category Matching:** (Layers 1-4: exact name, name parts, categories, IDF-weighted token overlap) Fast, deterministic matching; up to `tools.rag.top_k` tools.
- **Semantic channel (optional):** with `tools.rag.semantic: true`, embedding similarity between the request and each description adds a few more entries (`semantic_top_k`, `semantic_min_similarity`). It adds to the keyword hits, never replaces them, and is off by default.
- **Pinned tools:** offered whatever the scorer says — the core tools (`memory_search`, `memory_write`, `memory_get`, `message`, `secret_get`, `secret_list`), the internal tools, `tools.core`, the tools the request's own contents call for (an attachment, a URL, a request to edit a SOP), and `subagents` for an agent with subagents.
- **Usage prior:** the agent's most-used tools of the last 30 days, read from its traces (`tools.rag.usage_prior`, default 6; 0 turns it off).
- **Continuity:** the tools the session's previous turn used, so a follow-up that names no tool keeps them.
- **`search_tools`:** a pinned tool that finds a permitted tool the selection left out and makes it callable from the next step. A match the caller's role keeps from it (the role gates above, `roleWithheldTools`) is not added; the search says it needs an administrator, and a direct call to it is refused in the same words.
- **Tool-error hints:** A failed call to a covered tool comes back, in the same turn, with a hint and the matching approved Saved Queries or Tool Presets (`auto_learn.go`). Nothing is stored: no `agent_notes` row is written and no tool description changes. See [Memory: Auto-Learn](memory.md#7-autonomous-error-learning-auto-learn).

---

## 4. Hooks & Plugins

### Admin Hooks (rules on tool calls)
Admins attach rules to the agent's tool calls on the Hooks page; they are saved under `hooks:` in `config.yaml` and run by `hook_rules.go`. `Execute()` runs the before-hooks ahead of the tool cache and the tool, and the after-hooks once the call ends, so both agent loops, SOP and cron runs and subagents are covered. Actions are `block` (before only), `notify` and `webhook`. A deciding hook fails closed: only an explicit `{"decision": "allow"}` lets the call through. A rule limited to `agents` matches the agent a call runs as, by its configured ID; `main` or `default` in the list, in any case, names the default agent, unless an agent is configured under that ID. A SOP run is its owner's when its schedule, the SOPs page, a resume or the background starts it; the webhook route's agent's when a route starts it; the conversation's agent's when `sop_run` starts it there.

### Gateway Events (HookRegistry)
GoClaw implements a priority-based event registry for system-wide lifecycle events (`internal/gateway/hooks.go`). Handlers registered by Go code observe these events; they cannot change or stop what happens:
- `gateway:start/stop`
- `message:received/sending/sent`
- `before_prompt_build`, `llm:input`, `llm:output`
- `before/after_tool_call`
- `session:start/end`

### Extensions (Go / OpenClaw)
`PluginAPI` lets a Go plugin compiled into the gateway register hook handlers (`RegisterHook`) and HTTP routes (`RegisterHTTPRoute`). The gateway also discovers extension folders with a `plugin.json` or `openclaw.plugin.json` manifest in `/extensions/` and `/extensions/`, and lists them at `/api/extensions`; it does not load or run code from them.

### Agent Plugins
A separate feature: bundles of subagents, instructions, skills, hooks and MCP servers in the Agent Plugins layout VS Code uses (`plugin.json`, `skills//SKILL.md`, `mcp.json`). They are installed on the Plugins page into `/plugins` and turned on per agent under `plugins:` in `config.yaml`. A workspace file of the same name overrides a plugin's part, and a plugin's hooks and MCP servers stay off until an admin approves each one.

---

## 5. Caching Layers

1. **Tool Result Cache:** In-memory LRU cache with TTL and cross-tool invalidation (e.g., `file_write` invalidates `file_read` for the same path).
2. **LLM Response Cache:** a cache in the state database (`llm_cache` table) for model responses that do not contain tool calls. It is on when `cache.enabled` and `cache.llm_cache.enabled` are both true (the default), keeps entries for `cache.llm_cache.ttl` seconds (default 300), and is used only by the langchaingo loop (OpenAI, Claude, DeepSeek, Ollama), not by Gemini. SOP steps bypass it.

---

## 6. Large Output Spill

When a tool returns more than `max_result_chars` (default: 8,000 characters), GoClaw **spills** the full content to an in-memory store instead of plain truncation. The LLM receives a compact, type-aware **HEAD preview** and a `ref` ID.

### Flow

```mermaid
flowchart LR
    TOOL["Tool Result
> 8,000 chars"] --> DET["detectContentType()
json / jsonl / html / xml / csv / text"]
    DET --> STORE["OutputStore.Put(id, full)
TTL: 1h · max: 200 entries"]
    STORE --> LLM["LLM Receives Preview + ref
Explore full data via read_output(ref)"]
```

Example header and preview returned to the LLM:

```text
[LARGE OUTPUT: gitlab | type:json | lines:1847 | chars:58420 | ref:a3f8c2d1]
🛑 TRUNCATED: 2950 of 58420 characters are shown below; 55470 are NOT.
...
Preview:
[0] {"id":234,"title":"Fix login","state":"merged"}
...
To explore full data — call the SEPARATE 'read_output' TOOL ...:
read_output(ref: "a3f8c2d1", mode: "search", query: "")
```

The preview is capped at 4,000 characters. When it holds the whole result, the header is followed by a line saying the complete result is shown, and `Result:` in place of `Preview:`. `db_query` results are always stored, even small ones (which are returned whole), so `spreadsheet` can build a file from the `data_ref`; a query with no rows returns a notice instead. Inside a SOP run, a stored result is also written to the run's directory, so a ref handed to a later step outlives the in-memory store.

### Content Type Detection

| Type | Detect | HEAD Preview |
|---|---|---|
| `json` | starts with `{`/`[` + valid JSON | first 5 keys / items |
| `jsonl` | every line is `{...}` | first 5 objects |
| `html` | `` + text nodes |
| `xml` | `?token=`.

- **Security scanner** blocks inline `on*` handlers, `javascript:` URIs, untrusted iframes, and `data:text/html`.
- **Trusted CDN allowlist** permits scripts/styles from jsDelivr, cdnjs, Google Fonts, Plotly, D3, Bootstrap, etc.
- **Per-page token auth** — each page gets a random view token; global gateway auth is also accepted.
- **Sandboxed in the browser** — the page is served with a CSP `sandbox` that allows scripts, popups, dialogs and downloads but not `allow-same-origin` or forms, and with `connect-src`, `frame-src` and `form-action` set to `'none'`. The page runs as an origin of its own, so it cannot reach the dashboard API with the cookie of the admin viewing it. The scanner reads the HTML with patterns; the sandbox is what the browser enforces.
- **TTL**: `1h`, `6h`, `24h`, `7d` or `30d`; 30 days is both the default and the maximum. At most 100 pages are kept, in memory, so a restart removes them.

---

## 8. Strategic Roadmap

Future developments for the tools engine include:
- **Code Sandboxing:** Isolated execution of Python/JS snippets using Docker.
- **Subprocess Plugins:** Enabling non-Go extensions via a JSON-RPC bridge.

================================================================================
# FILE: https://goclaw.ulab.mn/docs/architecture/providers.md
================================================================================

# LLM Providers

GoClaw has **5 native providers** (Gemini, OpenAI, Anthropic, Ollama, DeepSeek), embedding providers and an OpenAI-compatible API, where OpenClaw supports 30+ providers. Each agent is served by its own provider and, when that fails, by the providers of its fallback list (see [Provider Fallback & Cooldown System](#provider-fallback--cooldown-system)).

## Architecture

```mermaid
flowchart TD
    subgraph PROV["GoClaw Provider Architecture"]
        P1["Gemini
Native SDK (genai)"]
        P2["OpenAI
LangChainGo"]
        P3["Anthropic
LangChainGo"]
        P4["Ollama (local)
LangChainGo"]
        P5["DeepSeek
OpenAI-compatible"]

        FB["Per-Agent Selection & Failover
ai.provider or agents.list[].model, then ai.fallback
(or agent fallback) in order, with cooldowns, until a tool has run"]

        EMB["Embedding Providers (3)
OpenAI / Gemini / Ollama"]
        OAI["OpenAI-Compatible API
/v1/chat/completions (streaming + sync)"]

        P1 --> FB
        P2 --> FB
        P3 --> FB
        P4 --> FB
        P5 --> FB
        FB --- EMB
        EMB --- OAI
    end
```

## Chat/Agent Providers

| # | Provider | Library | Status | Notes |
|---|----------|---------|--------|-------|
| 2.1 | **Google Gemini (Studio)** | Native `google.golang.org/genai` SDK | Done | Primary provider — Gemini 2.5/3+ support, `thought_signature`, native function calling, and Thinking Configuration. |
| 2.2 | **Google Vertex AI** | Native `google.golang.org/genai` SDK (Vertex AI backend) | Done | Enterprise-grade deployment via Google Cloud (gcloud ADC support). |
| 2.3 | **OpenAI** | LangChainGo | Done | GPT-4o, GPT-4-turbo, etc. |
| 2.4 | **Anthropic (Claude)** | Native `github.com/anthropics/anthropic-sdk-go` SDK | Done | The shared agent loop with a native adapter (`claude_stream.go`): thinking blocks sent back whole, all of a round's tool results in one message, prompt caching, `ai.anthropic.effort`, and a fallback model on a refusal where the model takes one. Default model `claude-sonnet-5-5`; set another with `ai.anthropic.model`. |
| 2.5 | **Ollama (local)** | LangChainGo | Done | Local models (llama3, mistral, etc.) |
| 2.6 | **Gemini (Legacy)** | `gemini-legacy` | Deprecated | A deprecated name for `gemini`, served by the native genai SDK. The gateway already served `ai.provider: gemini-legacy` this way; `goclaw chat` and `gemini-legacy/` used LangChainGo's Google client, which sent `temperature`, `top_p` and `top_k` on every request. |
| 2.7 | **DeepSeek** | LangChainGo (OpenAI-compatible client) | Done | `deepseek-v4-flash` (default) and `deepseek-v4-pro` via `https://api.deepseek.com/v1`. Shares the OpenAI tool-calling loop. V4 reasoning traces arrive in `reasoning_content` and are currently discarded. |

...

## Chat/Agent Features

### Gemini Thinking Configuration
For Gemini 2.5 and later, GoClaw sets how much the model thinks from `ai.gemini.thinking_level` (`off`, `minimal`, `low`, `medium`, `high`) or `ai.gemini.thinking_budget`. Gemini 3 and later are sent `thinking_level` only, Gemini 2.5 `thinking_budget` only; see [Gemini API Implementation](gemini_implementation.md#thinking-configuration).

### Google Search Grounding
GoClaw supports server-side **Google Search Grounding** for Gemini models. When enabled, the model can query Google Search in real-time to provide up-to-date information, with citations automatically formatted and appended to the response.

## Embedding Providers

GoClaw supports 3 embedding providers for memory search (same providers used in chat):

| # | Provider | Model | Dimensions | Status |
|---|----------|-------|:----------:|--------|
| 2.E1 | OpenAI | `text-embedding-3-small` | 1536 | Done |
| 2.E2 | Gemini | `text-embedding-004` | 768 | Done |
| 2.E3 | Ollama | configurable | varies | Done |

## Provider Fallback & Cooldown System

An agent's provider failing before the reply used a tool hands the request to the next provider of
`ai.fallback` (or the agent's `agents.list[].fallback`); see
[Provider Fallback & Cooldown](provider_fallback.md).

`internal/agent/fallback.go` + `internal/agent/provider_cooldown.go`

| # | Feature | OpenClaw | GoClaw | Status |
|---|---------|----------|---------|--------|
| 2.F1 | Multi-provider fallback chain | ✅ Provider rotation | ✅ `ai.fallback` / `agents.list[].fallback` | Done |
| 2.F2 | Error classification (8 types) | Basic retry | ✅ auth, billing, rate_limit, overloaded, timeout, model_missing, format, unknown | **GoClaw only** |
| 2.F3 | Exponential backoff cooldown | ✅ | ✅ `60s × 5^min(n-1, 3)`, max 1 hour | Done |
| 2.F4 | Probe throttling | — | ✅ Primary provider only, 30s minimum interval | **GoClaw only** |
| 2.F5 | Context overflow detection | ✅ | ✅ `isContextOverflow` — stops fallback chain, returns error | Done |
| 2.F6 | Failure window reset | — | ✅ 10 min no-error → counter reset | **GoClaw only** |
| 2.F7 | Probe state TTL + eviction | — | ✅ 24h TTL, max 256 keys | **GoClaw only** |
| 2.F8 | Per-provider usage stats | — | ✅ `ProviderUsageStats` — lastUsed, errorCount, failureCounts, disabled | **GoClaw only** |
| 2.F9 | Disabled vs Cooldown distinction | — | `IsDisabled` checks `DisabledUntil`, which nothing sets; only the cooldown is ever applied | Unused |
| 2.F10 | Provider health dashboard | ✅ | — | **TODO** |

### Cooldown Backoff Formula

```
cooldown_ms = min(60000 × 5^min(errorCount-1, 3), 3600000)

Error 1:  1 min
Error 2:  5 min
Error 3:  25 min
Error 4+: 60 min (max)
```

### Error Classification (`classifyError`)

| Keyword Pattern | Failure Reason |
|----------------|----------------|
| 401, 403, unauthorized | `auth` |
| 402, billing, quota | `billing` |
| 429, rate limit | `rate_limit` |
| 503, overloaded | `overloaded` |
| timeout, deadline | `timeout` |
| 404, model not found | `model_not_found` |
| 400, invalid | `format` |
| (other) | `unknown` |

## OpenAI-Compatible API

`internal/gateway/openai_compat.go`

External tools (Cursor, Continue.dev, etc.) can use GoClaw as OpenAI API:

| # | Feature | OpenClaw | GoClaw | Status |
|---|---------|----------|---------|--------|
| 2.C1 | `/v1/chat/completions` POST | ✅ | ✅ `HandleChatCompletions` | Done |
| 2.C2 | Streaming (SSE) | ✅ | ✅ `text/event-stream` | Done |
| 2.C3 | Non-streaming | ✅ | ✅ JSON response | Done |
| 2.C4 | Token usage estimation | ✅ | ✅ ~4 chars/token approximation | Done |
| 2.C5 | Subagent command routing | — | ✅ `HandleSubagentCommand` pass-through | **GoClaw only** |
| 2.C6 | Session persistence | ✅ | ✅ Per-IP session (`openai-compat:`) | Done |
| 2.C7 | Model routing | model → provider | Uses default agent | Partial |
| 2.C8 | `/v1/models` list | ✅ | — | **TODO** |
| 2.C9 | Function calling pass-through | ✅ | — | **TODO** |
| 2.C10 | Embeddings API | ✅ | — | **TODO** |

---

## OpenClaw vs GoClaw Provider Comparison

| Aspect | OpenClaw | GoClaw |
|--------|----------|---------|
| **Total providers** | 30+ | 5 (Gemini + OpenAI + Anthropic + DeepSeek + Ollama; `gemini-legacy` is served as Gemini) |
| **Provider framework** | JS plugin system | LangChainGo + native Gemini SDK |
| **Fallback system** | Basic rotation | `ai.fallback` / `agents.list[].fallback`, tried in order before a tool has run |
| **Cooldown tracking** | — | Exponential cooldown per provider, with probes of the agent's own provider |
| **Embedding providers** | Multi-provider | 3 (OpenAI, Gemini, Ollama) |
| **OpenAI-compat API** | ✅ | ✅ + subagent command routing |
| **Local models** | Ollama | Ollama (same) |

**Coverage:** 5 chat providers, 3/3 embedding providers, provider fallback with cooldowns, 7/10 OpenAI-compat features.

================================================================================
# FILE: https://goclaw.ulab.mn/docs/architecture/subagents_and_jobs.md
================================================================================

# Subagents & the Job Queue

GoClaw lets a **parent agent delegate a scoped task to a child agent** ("subagent")
that runs **asynchronously** in its own isolated session. Every run is persisted to an
embedded SQLite `jobs` table, so delegation survives process restarts, is auditable, and
never leaks goroutines.

This document explains what a subagent is, how you trigger one, the guardrails that bound
it, and the runtime machinery (job queue, EventBus, workers) that executes it.

---

## 1. What is a subagent?

Three terms, used the same way in the Web UI, the tools and this document:

- An **Agent** is an AI employee of its own: its configuration, workspace, memory and
  channels.
- A **Subagent** belongs to one agent: a name, a description of when to use it,
  instructions, a narrowed set of tools and optionally its own model. It is a file in that
  agent's workspace, `agents/.agent.md`, made on the Agents page or with the
  `subagents` tool. It runs *inside* its agent, under that agent's permissions.
- A **run** is one task handed either to one of the agent's own subagents or to another
  agent. Every surface names a run the same way: `Requester → Target (Type)`, for example
  `Support → code-reviewer (Subagent)` or `Support → Finance (Agent)`.

A **run** answers a single, well-scoped task. The child runs in a fresh session
(`agent::subagent:`), streams its answer, and reports back to the parent — all
without blocking the parent's own turn.

Use a subagent when the task is a distinct sub-problem that benefits from a different agent's
persona, model, or toolset (e.g. a general assistant delegating a code change to a coding
agent), or when you want fan-out/parallel work. Do **not** use one for a trivial single-tool
call the parent can do itself — a subagent spins up a full agent turn (LLM + tools) and has
real cost.

Key properties:

- **Isolated session** — the child gets its own conversation history; it does not see the
  parent's full context, only the `task` string it is handed.
- **Asynchronous** — with `wait` (the default) the tool waits up to 60 seconds and returns the
  answer; a run that takes longer, or one started with `wait: false`, returns a `run_id`. The
  parent polls, yields, or reads the result the next time it is invoked (see
  [§6](#6-completion-eventbus-yield-and-announcements)). The run goes on after the turn that
  started it ends, until it finishes, is stopped, or reaches its one-hour limit (see
  [§10](#10-limits--gotchas)).
- **RBAC-preserving** — the parent's identity (`GoClawRole`, `UnifiedUserID`, `Channel`) is
  forwarded into the child, so memory scoping and role-gated writes behave correctly inside
  the subagent. A subagent is not a privilege-escalation path.

---

## 2. How to trigger a subagent

There are two surfaces, both backed by the same `SpawnSubagent` entry point. The `sessions_spawn` and `sessions_send` tools start a run and send it a follow-up through the same functions (`SpawnSubagent`, `SendSubagentMessage`).

### 2.1 The `subagents` tool (LLM-facing)

A registered agent tool named **`subagents`** (`internal/agent/tool_subagents.go`) keeps
**registration** (which subagents an agent has) apart from **execution** (running tasks).
Registering never runs anything, and running never changes a registration.

| Action | Parameters | Effect |
|---|---|---|
| `create` | `name`, `description`, `instructions`, `tools`, `model`, `agent_id`, `draft` | Register a subagent. Admins only. Refuses a name that already exists; `draft: true` shows it without saving. |
| `update` | `name`, any of the above | Change a subagent. Admins only. |
| `delete` | `name`, `agent_id` | Remove a subagent. Admins only. |
| `list` | `agent_id` | List an agent's subagents. |
| `inspect` | `name`, `agent_id` | Show one subagent. Its instructions are shown to admins only. |
| `run` | `agent_id`, `task`, `target_type`, `wait` | Start a run (`spawn` still works). `agent_id` may be omitted → auto-routing to an agent (below). |
| `status` | `run_id` | Report one run's state, timing, and error. |
| `list_runs` | — | List this agent's active and recent runs. |
| `follow_up` | `run_id`, `message` | Send a further message to a run's session. |
| `yield` | `run_id` | Return the run's result if finished; otherwise report "still running". |
| `stop` | `run_id` | Cancel a queued/running run. |

For registration, `name` is the subagent and `agent_id` the agent it belongs to (default:
the agent of this conversation). Another agent's subagents can be listed, inspected or
changed by an administrator only. For `run`, `agent_id` is the target: one of the agent's
own subagents, or another agent's ID. If a subagent and a configured agent have the same
name, `target_type` (`"subagent"` or `"agent"`) must say which.

The execution actions only reach runs the calling agent started, and of those only the runs
the caller may touch: the person who started the run, a human admin, or the system acting for
no one (a cron or SOP run). A run that recorded no person belongs to the session that started
it. Any other run answers "not found", so its task and result stay with it. A follow-up acts
with the run's delegation: a caller acting for another person, or whose own grant does not
cover the run's, is refused.

Example (as the model would call it):

```json
{ "action": "run", "agent_id": "code-reviewer", "task": "Review merge request !42 for bugs" }
```

With `wait` (the default) the tool waits up to 60 seconds and returns the answer; a run that
takes longer keeps going while the turn that started it runs (see
[§10](#10-limits--gotchas)). `action` is validated at runtime (not by the
JSON schema), so an unknown action returns an error listing the valid ones.

If `agent_id` is omitted on `run`, GoClaw attempts **embedding-based auto-routing**
(`RouteToAgent`) to pick the best-matching agent for the task; an explicit `agent_id` always
wins. If no confident match is found, the tool asks the caller to pick one via `agents_list`
or `suggest_agent`.

### 2.2 The `/subagents` slash command (human/channel-facing)

The same operations are available as a chat command, parsed by `HandleSubagentCommand`
(`internal/agent/subagent_commands.go`) and wired into every message entry point — chat
channels, the WebSocket UI, the REST API, and the OpenAI-compatible endpoint:

```
/subagents spawn  
/subagents list
/subagents status 
/subagents stop 
```

---

## 3. Spawn policy — the guardrails

Before any run starts, `validateSubagentPolicy` enforces per-parent limits declared in the
parent agent's config, plus a system-wide safety cap. This is what prevents runaway
"explosive chaining" loops.

| Guardrail | Config key (per agent) | Default | Meaning |
|---|---|---|---|
| **Allow-list** | `subagents.allow_agents` | *(empty = allow all)* | Which other configured agents this parent may spawn. `"*"` = any. If set and the target is not listed, the spawn is denied. It does not apply to the agent's own subagents (`agents/.agent.md`). |
| **Max concurrency** | `subagents.max_active` | `5` | Max simultaneously *queued + running* children for this parent, its own subagents included. |
| **Max depth** | `subagents.max_depth` | `1` | How deep the spawn chain may nest. Depth is counted from the number of `:subagent:` segments in the parent session ID, so a child cannot recurse past the limit. |
| **System-wide cap** | *(constant)* | `10` | `maxConcurrentSubagentsV2` — a global semaphore bounding *all* concurrently executing subagents across every parent (prevents goroutine leaks). |

Two details worth knowing: `max_active` is counted **per parent agent** (all of that agent's
sessions share the one budget), and the whole policy block is only applied when the parent's
config is found — if `config.GetAgentConfig(parent)` returns nil, **all checks are skipped**.

A separate switch governs **agent-to-agent messaging** (the "send a follow-up to a run"
path, `follow_up` and `SendSubagentMessage`) when the run's target is **another agent**:
`tools.agent_to_agent.enabled` must be `true` (it defaults to **off**), and
`tools.agent_to_agent.allow` (if non-empty) must list the participating agents. A follow-up to
one of the agent's own subagents does not need it. A follow-up runs the subagent with the same
instructions, tools and model as its first run, and fails if the subagent has been deleted
since. This gate is about *messaging* an existing run, not starting one.

> **Note:** these limits are what stands between a prompt-injected message and a spawn
> storm. Registering, changing and deleting subagents needs an administrator, but starting a
> run does not, so `allow_agents` is the real boundary for runs on other agents — keep it
> tight for agents exposed to untrusted channels. The agent's own subagents are bounded by
> `max_active` and by their narrowed tools.

---

## 4. Execution model

Runs are `SubagentManagerV2`'s (`subagent_v2.go`): each run is a row in the SQLite `jobs`
table, so it survives a restart and can be audited, its end is published on the EventBus and
announced to the parent, and it has its own trace.

`SpawnSubagent()` — the function both the tool and the slash command call — delegates to
`SpawnSubagentV2` and converts the resulting `Job` back into a `SubagentRun` for callers;
`GetSubagentRun`, `ListSubagentRuns` and `StopSubagentRun` read and stop jobs the same way.
`SubagentManager` (`subagent.go`) is what remains of the first, in-memory manager: it holds the
agent registry that a follow-up (`Send`, behind `follow_up` and `sessions_send`) runs on, and
finds the run to follow up in the `jobs` table.

---

## 5. Lifecycle & state machine

A subagent run moves through these states (`SubagentStatus`):

```mermaid
stateDiagram-v2
    direction LR
    [*] --> queued : spawn
    queued --> running : acquire slot
    queued --> failed : stop / cancel
    running --> done : finish
    running --> failed : finish (error) / stop / cancel
```

A stopped or cancelled run ends `failed`, usually with the error `killed` (a stop records
`killed by user` first; the run's own exit then overwrites it), or `cancelled waiting for
execution slot` when it had not started. A run that reaches its one-hour limit (queued time
included) ends the same way. The turn that started a run ending does not stop it. There is no
separate `killed` state.

Under the hood, `SpawnSubagentV2` performs these steps:

1. **Validate** the spawn against the policy in [§3](#3-spawn-policy--the-guardrails).
2. **Insert** a `Job` row (`job_type="subagent"`, `state="queued"`) with the parent/child
   context serialized into `context_json`.
3. **Launch** a bounded goroutine. It acquires a slot from the global semaphore
   (cap `10`); if the context is cancelled while waiting, the job is failed with
   *"cancelled waiting for execution slot"*.
4. **Run** — flips the job to `running`, forwards the parent's RBAC identity, and calls the
   child agent's `ProcessStream`, draining it to a final result (`DrainToResult`).
5. **Persist** the task + answer into the child's session history.
6. **Finish** — marks the job `done` (or `failed` on error or cancel), publishes an
   EventBus event, and, when it is `done`, **announces to the parent** (see next section).

> The `jobs` row for a subagent is a **ledger** (state, result, audit, recovery) — not a
> dispatch queue. Execution is driven directly by the spawning goroutine, not by a background
> poller. (Contrast this with SOP steps and memory jobs, [§7](#7-who-actually-runs-the-jobs).)

---

## 6. Completion: EventBus, yield, and announcements

When a run finishes, the parent finds out through either of two mechanisms:

- **Session injection** — a run that is `done` has its answer appended to the parent's session as
  `"[Support → code-reviewer (Subagent) done (Run: )] …"`, so the parent sees it in
  history on its next turn. It waits for the parent's turn in progress to end (at most 15
  minutes) so neither write drops the other. A failed run is not announced; nothing is posted
  to a channel.
- **EventBus** — the run publishes to the in-process `EventBus` (`internal/agent/eventbus.go`)
  on topic `job.done` or `job.failed`.

The `yield` action (and `YieldSubagentV2`) uses the EventBus to wait efficiently: it
subscribes to `job.done`/`job.failed`, and — because publishing is non-blocking and *drops*
events to a full subscriber channel — also polls the DB every **3 seconds** as a safety net,
plus an overall timeout. This makes waiting correct even under bursty, high-concurrency
fan-out.

---

## 7. Who actually runs the jobs?

The `jobs` table is shared by several kinds of async work, but they are executed by **three
distinct mechanisms**. Keeping them straight is important — they have different concurrency
limits.

| Executor | Handles | Concurrency | Dispatch |
|---|---|---|---|
| **Subagent manager** (`subagent_v2.go`) | `subagent` jobs | Semaphore, cap **10** | In-process goroutine per spawn (no poller) |
| **MemoryWorker** (`memory_worker.go`) | `fact_extract`, `memory_consolidate`, `graph_extract`, `graph_sync`, `belief_revision` (an `auto_learn_error` row left by an older version is drained as a no-op) | **Sequential** (one at a time) | Polls `queued` jobs every **5 s** |
| **Tool WorkerPool** (`worker_pool.go`) | Tool calls (not jobs at all) | `tools.worker_pool_size` (**8**), queue `tools.worker_queue_size` (**64**) | Buffered channel with back-pressure |

Two clarifications that the older version of this doc got wrong:

- **The `8`-worker / `64`-queue pool is the *tool* execution pool**, not the subagent pool.
  Subagent concurrency is the semaphore of `10` plus the per-parent `max_active` limit.
- **Memory background work is deliberately serialized** by the `MemoryWorker` (one job at a
  time) to respect LLM rate limits and to avoid SQLite write contention on the `memories`
  table.

SOP steps also live in the `jobs` table (`job_type="sop_step"`) and use the extra
`suspended` state for human-in-the-loop approval, resuming via the EventBus and the
`parent_job_id` link. SOPs are covered in [Standard Operating Procedures](../user-guide/automation/sop.md).

### The `jobs` table

| Column | Purpose |
|---|---|
| `id` | Unique job ID (`job--`). |
| `parent_job_id` | Links a child job to its parent (used by SOP resume). |
| `job_type` | `subagent` \| `sop_step` \| `fact_extract` \| `memory_consolidate` \| `graph_extract` \| `graph_sync` \| `belief_revision`; `auto_learn_error` only in rows left by an older version. |
| `agent_id` | The agent that executes the job. |
| `task` | The work description handed to the agent. |
| `state` | `queued` \| `running` \| `suspended` \| `done` \| `failed`. |
| `context_json` | Serialized context for resume (parent/child IDs, etc.). |
| `result_json` | Output captured on completion. |
| `error` | Failure reason, if any. |
| `created_at` / `updated_at` | Timestamps. |

---

## 8. Reliability & crash recovery

Because state lives in SQLite, a restart does not lose or double-run work:

- On startup, `InitSubagentV2` calls `RecoverPendingJobs` → `FailStaleJobs`, which marks any
  job left in `running` (interrupted mid-execution by the crash) as `failed` with a clear
  reason. Jobs still `queued` are left intact.
- Panics inside a subagent goroutine are recovered and turn into a `failed` job rather than
  crashing the process.
- Each subagent run records its own trace, so child execution is observable separately from
  the parent in the dashboard.

---

## 9. Configuration example

Enable subagents on a parent agent and allow it to delegate to two workers:

```yaml
agents:
  default: main
  list:
    - id: main
      subagents:
        allow_agents: [coding, research]  # who "main" may spawn ("*" = any)
        max_active: 5                      # concurrent children (default 5)
        max_depth: 1                       # nesting depth (default 1)
    - id: coding
    - id: research

tools:
  # Required to send a follow-up to a run on another agent (not to the agent's own subagents).
  agent_to_agent:
    enabled: true
    allow: [main, coding, research]

  # Tool-execution pool (separate from subagent concurrency).
  worker_pool_size: 8
  worker_queue_size: 64
```

---

## 10. Limits & gotchas

- **Depth is capped** (`max_depth`, default `1`): by default another agent running a task
  cannot itself spawn a run. Raise it deliberately — deep chains multiply cost. An agent's own
  subagents (`.agent.md`) never nest: they are never offered `subagents`, `sessions_spawn` or
  `sessions_send`, whatever their profile lists. The policy is checked only for agents listed
  in `agents.list`.
- **Global cap of 10**: even with generous per-agent `max_active`, no more than 10 subagents
  run at once process-wide.
- **A run may outlast the wait**: `run` waits up to 60 seconds (or not at all with
  `wait: false`) and then returns the `run_id`. Use `status`/`yield` or wait for the
  announcement — do not assume the result is ready synchronously.
- **A run outlasts its turn, up to an hour**: it keeps who it acts for and the scope of the
  turn that started it, but not that turn's cancellation, so a chat channel answering does not
  stop it. A run still going after an hour (queued time included) is cancelled and ends
  `failed` (`killed`).
- **A subagent is a full agent turn**: it consumes tokens and can call tools. Delegate whole
  sub-problems, not one-liners.
- **`allow_agents` is a security boundary**, not just organization: it is the guard against a
  prompt-injected channel message spawning arbitrary agents. Keep it tight on exposed agents.
- **Queued-at-crash jobs are orphaned**: startup recovery fails `running` jobs, but a subagent
  left `queued` at crash time is never re-dispatched (there is no queue poller for subagents).
  In practice the `queued` window is momentary, so this is rare.

================================================================================
# FILE: https://goclaw.ulab.mn/docs/architecture/autonomy_levels.md
================================================================================

# AI Autonomy Levels & Security Philosophy in GoClaw

GoClaw was inspired by projects like OpenClaw, but it diverges significantly in one critical area: **Security & Trust**. 

The fundamental problem with many existing AI Agent frameworks (AutoGPT, OpenClaw, LangChain) is the unchecked execution of arbitrary code, unlimited shell access, and untrusted third-party tool execution. 

GoClaw solves this by treating the agent not as an all-powerful script kiddie, but as an **Enterprise AI Employee (Level 4)** that operates strictly within a **Zero-Trust, Built-In Tool Ecosystem**.

This document outlines the 5 levels of AI Autonomy and how GoClaw safely implements them without compromising the host system.

---

## 🛡️ The GoClaw Security Philosophy

1.  **Built-In Over Bolted-On:** GoClaw ships as a single compiled binary. Its tools (GitLab, ClickHouse, Shell, PDF parsing, Memory Search) are *built into* the binary. They do not install Python packages, Node modules, or external curl scripts; a few call programs that must already be on the host (Chrome for the browser and PDF reports, linters such as golangci-lint or ruff, `pdftotext` when present). The one way to add outside tools is an MCP server: one an admin configures under `mcp.servers` in `config.yaml`, or one a plugin brings and an admin approves. A stdio server is a process the gateway starts, often through `npx` or `uvx`, so it brings that runtime and its packages with it. If a tool isn't compiled into the binary, provided by such a server, or explicitly permitted via the `SKILL.md` / `config.yaml` allowlists, the agent cannot use it.
2.  **Principle of Least Privilege (Sandboxing):** 
    - The `shell_exec` tool operates on a strict **Allowlist** (e.g., `ls`, `grep`, `cat`, `top`). It blocks dangerous operations (pipes, output redirection, executing downloaded binaries).
    - Database tools (`db_query`, `mongo_query`) enforce **Read-Only (SELECT/SHOW)** operations. DDL/DML (DROP, DELETE, UPDATE) are hard-coded to fail. There are two exceptions. `db_write` inserts, updates and deletes by key, only on a datasource marked `allow_write`, only for an admin or a SOP or cron run, and capped in the rows it may touch. `metrics_store` creates and writes its own `agent_metrics` table on a datasource, under the same `allow_write` and caller rules, never for a webhook-triggered or delegated run.
    - File system access is strictly jailed to the configured Workspace directory (`~/.goclaw/workspace`).
3.  **Human-in-the-Loop (HITL):** Critical actions (merging code to production, moving money, mutating infrastructure) must be delegated to a specific SOP step that halts execution and requests explicit human approval via Discord/Teams/Web UI.
4.  **Deterministic Boundaries:** Level 4 SOPs define *exactly* which tools are allowed at which step. An agent tasked with "Reviewing a Pull Request" is physically prevented from using the `shell_exec` tool during that specific step.

---

## 📈 The 5 Levels of AI Autonomy

### 🟢 Level 1: Conversational AI (Chatbots)
**Autonomy:** 0% (Human drives, AI responds)
**Definition:** The AI can only read its training data and the context provided in the immediate chat prompt. It cannot take actions.
**GoClaw Implementation:** 
- Secure by default. Interactive `goclaw chat` CLI mode.
- System prompt enforces the persona (`SOUL.md`).

### 🟡 Level 2: AI Copilot (Context-Aware Assistants)
**Autonomy:** 20% (Human drives, AI suggests and reads context)
**Definition:** The AI can read the environment (local files, current git branch, screen content) to provide highly contextual suggestions, but the human must execute the final action (e.g., hitting `Tab` to complete code, or clicking "Apply").
**GoClaw Implementation:**
- Uses read-only tools: `file_read`, `repo_map`, `memory_search`, `gitlab` (read operations).
- **Security Boundary:** The agent can see the code, but it cannot commit or push without explicit instruction. 

### 🟠 Level 3: Autonomous Agents
**Autonomy:** 60% (Human assigns a goal, AI executes via a ReAct loop)
**Definition:** The AI is given a goal ("Find the bug in login.go and fix it") and is trusted to select its own tools, iterate through errors, and execute changes. 
**GoClaw Implementation:**
- Restricts autonomy using built-in **Safe HTTP Clients** with SSRF protection.
- **Event-Driven Proactivity:** Beyond simple prompt-response, the **Webhook Router** allows agents to react to external triggers (GitLab MRs, Zabbix alerts) autonomously.

### 🔵 Level 4: AI Employee / Digital Workforce (GoClaw's Target)
**Autonomy:** 95% (AI executes long-running processes, learns from humans, and generates professional artifacts)
**Definition:** The AI operates proactively, follows strict SOPs, and most importantly, **learns from every interaction**. It doesn't just chat; it produces work products (PDFs, HTML dashboards) and refines its own knowledge base.
**GoClaw Implementation:**
- **Continuous Learning (Fact Extraction):** Automatically extracts concrete facts (IP changes, config hints, user rules) from conversations and persists them to long-term memory.
- **Conflict-Aware Memory:** Smartly replaces old facts (e.g., outdated IPs) with new ones, ensuring the "Digital Employee" never gets confused by stale data.
- **Actionable Reporting:** Generates professional **PDF reports** (with Mongolian support) and **HTML dashboards** with TTL security, turning raw data into executive-ready artifacts.
- **State-Machine SOPs:** JSON-defined workflows with strict success criteria and automated reflection.
- **Asynchronous Background Processing:** Decouples heavy memory tasks (extraction, consolidation, KG sync) into a persistent **Job Queue** and dedicated **MemoryWorker**, ensuring system responsiveness and stability while operating at scale.
- **Token Analytics:** Built-in dashboard to monitor ROI and prevent cost overruns.

### 🟣 Level 5: Fully Autonomous Entity (AGI)
**Autonomy:** 100% (AI defines its own goals, creates its own tools, manages its own resources)
**Definition:** The AI operates an entire business or system end-to-end without human oversight. It writes its own SOPs and provisions its own infrastructure.
**GoClaw Implementation:**
- **Not implemented.** No part of Level 5 is built. The [Level 5 vision](../vision/level5.md) describes it only as *governed* autonomy: agents propose goals, run experiments and suggest SOPs and tools, and a person or a policy approves each one. Unsupervised self-direction (an agent setting its own goals, provisioning its own resources or rewriting its boundaries without review) stays out of scope. Today agents may not change system boundaries or bypass the persistence layer. The agent changes its own core instruction (`update_core_instruction`) only in a human admin's conversation, at that admin's request; SOP, cron and webhook runs are refused. Humans remain in control of the `config.yaml` and SOP definitions.

---

## 🚀 Conclusion

By compiling trusted tools directly into the binary and replacing probabilistic "ReAct" loops with deterministic SOP workflows, GoClaw provides the autonomy of a **Level 4 AI Employee** with the security guarantees of a traditional enterprise application.

================================================================================
# FILE: https://goclaw.ulab.mn/docs/user-guide/automation/index.md
================================================================================

# GoClaw Automation Guides

GoClaw runs workflows on a schedule, when you start them, or when an outside system calls one of its webhooks (GitLab, Graylog, or anything that can send an HTTP POST).

## Guides

- **[Standard Operating Procedures (SOPs)](sop.md)**: JSON-defined, state-machine based workflows with automated failovers and validations.
- **[Upgrading existing SOPs](sop-migration-prompt.md)**: A prompt that has an agent audit and upgrade the SOPs it owns to the step data-handoff model.
- **[Scheduled Tasks (Cron Jobs)](cron-jobs.md)**: SQLite-backed cron manager using robfig/cron for simple recurring executions.
- **[GitLab Integration](gitlab-webhook.md)**: Handling push, MR, and pipeline alerts from GitLab repositories securely. Set it up by asking the agent, which creates the project webhook in GitLab and the route in GoClaw (`gitlab_hook`), with the `goclaw gitlab hook` wizard, or by hand.
- **[Graylog Alerts That Run a SOP](graylog-alerts.md)**: A Graylog → GoClaw webhook. Graylog watches the logs, and each alert calls the gateway's hook (`/goclaw/hook/graylog/`) and runs a SOP. Set it up by asking the agent, which registers the alert in Graylog and the route in GoClaw, or [by hand](graylog-alerts.md#connecting-graylog-without-the-agent): a Graylog route on the Webhooks page and an HTTP notification in Graylog.
- **[SonarQube Quality Gates That Run a SOP](sonarqube-hooks.md)**: SonarQube calls the gateway after each analysis (`/goclaw/hook/sonarqube/`) and a failed quality gate (or every analysis) runs a SOP. Set it up by asking the agent, which makes the webhook in SonarQube and the route in GoClaw, or by hand.
- **[Zabbix Problems That Run a SOP](zabbix-alerts.md)**: A Zabbix action sends problems on the host groups you choose, at or above a severity, to a webhook media type that calls the gateway (`/goclaw/hook/zabbix/`), and each runs a SOP. Set it up by asking the agent, which makes the media type, a user and group of its own and the action in Zabbix and the route in GoClaw, or by hand.
- **[Uptrace Monitors That Run a SOP](uptrace-monitors.md)**: An Uptrace metric monitor's webhook channel calls the gateway (`/goclaw/hook/uptrace/?token=…`) when an alert opens, and runs a SOP. Set it up by asking the agent, which makes the monitor and its channel in Uptrace and the route in GoClaw, or by hand.
- **[Wazuh Alerts That Run a SOP](wazuh-alerts.md)**: An alerting monitor in the Wazuh indexer searches Wazuh's alerts every few minutes and, through a webhook channel, calls the gateway (`/goclaw/hook/wazuh/`) with what it found, which runs a SOP. Nothing changes on the manager. Set it up by asking the agent, which makes the monitor and its channel in the indexer and the route in GoClaw, or by hand.
- **[Grafana Alert Rules That Run a SOP](grafana-alert-rules.md)**: A Grafana alert rule's webhook contact point calls the gateway (`/goclaw/hook/grafana/`) when the rule fires, and runs a SOP. Set it up by asking the agent, which makes the rule, its contact point and the route in GoClaw, or by hand.

Any other system that can POST JSON reaches an agent the same way, through a webhook route: see [Webhook Routes](../configuration.md#webhook-routes). A route can run a SOP for each event, or gather a window of events into one run.

================================================================================
# FILE: https://goclaw.ulab.mn/docs/user-guide/automation/sop.md
================================================================================

# Standard Operating Procedures (SOPs)

SOPs are autonomous, multi-step workflows that allow an AI agent to perform complex tasks without human intervention. Unlike simple Cron Jobs, SOPs include planning, tool restrictions, and self-reflection to ensure reliability.

## Overview

An SOP consists of multiple **Steps**. Each step defines:
- **Instruction**: What the agent should do.
- **Tools**: Which tools are allowed for this specific step.
- **Success Criteria**: How to verify the step's output (e.g., `contains:OK` or `json`).
- **Failure Action**: What to do if criteria aren't met (`retry`, `abort`, or `escalate`).

## Managing SOPs via WebUI

The GoClaw Admin Dashboard provides a dedicated interface for managing SOPs.

### 1. Viewing SOPs
Navigate to the **SOPs** tab in the sidebar. Each SOP is a card showing its name
and ID, its purpose, the number of steps, its owner, and whether it is **Active**
or **Paused**; badges show the linked cron job and how many steps require
approval. Click a card to open the SOP: its screen has **Run**, pause/enable and
delete buttons, and the tabs **Overview**, **Steps**, **Diagram**, **Channels**,
**History** and **Edit**.

### 2. Creating a New SOP
Click the **New SOP** button. A blank form opens on the **Edit** tab.
- **ID (slug)**: A unique slug (e.g., `daily-report-generator`).
- **Name**: A descriptive name for humans.
- **Purpose**: A brief explanation for the AI to understand why this SOP exists.
- **SOP-level tools**: Tools every step may call in addition to its own (see
  [Which tools a step may call](#which-tools-a-step-may-call)).
- **Shared knowledge**: Markdown injected into every step's prompt (`context`).
- **Owner agent**: The agent the run runs as: its model, its workspace and
  configuration, its core instruction, the hooks limited to it and its daily
  token limit. `default`, `main` or an empty owner is the default agent (unless
  an agent is configured under that ID), and so is an owner that names no
  configured agent. A run a webhook route starts runs as the
  route's agent instead, and one `sop_run` starts in a conversation as that
  conversation's agent, on that agent's model too. A run for an agent the
  gateway is not running (removed from `config.yaml`, added since the gateway
  started, or whose model failed to start) does not start: a webhook route is
  answered `409`, `sop_run` and **Run** are refused, and a scheduled run fails
  before its first step.
- **Cron schedule**: When the SOP should run. Saving with a schedule creates a
  cron job for the SOP and links it, or moves the job it already has to the new
  schedule (see [Cron Jobs](cron-jobs.md)). Schedules have six fields, seconds
  first: `0 0 9 * * *` is daily at 09:00; a five-field `0 9 * * *` is read the
  same. Clearing the field leaves the job as it is: pause or delete it on the
  **Cron** page. A schedule that cannot be applied is reported after the save,
  which still goes through.
- **Enabled**: A paused SOP is skipped by its schedule and refused by webhooks.
- **Workflow steps**: **Add step** adds a step, with the fields described in the
  [Step Configuration Reference](#step-configuration-reference) below, except
  `timeout`, which a save keeps (see below).

**Validate** checks the draft without saving it; **Save** saves it.

### 3. Editing an SOP
Open the SOP and its **Edit** tab. The form has the same fields; the next run
uses what you save.

> **What a save keeps.** The form has no field for a step's `timeout` or the
> SOP's `prerequisites`, `virtual_level` and review dates; a save keeps them as
> they are. `POST /api/sop` keeps any field the request leaves out, at the top
> level and in each step (matched by its `id`); a field it sends, even empty,
> replaces the stored one. A save that changes the SOP's definition archives
> the previous version, which `sop_rollback` restores, and raises the version;
> pausing or resuming it does not.
>
> The agent's `sop_save` tool keeps the owner, timezone, SOP-level tools, shared
> knowledge and prerequisites when an update leaves them out (an empty value
> clears one), always keeps `virtual_level`, and keeps a step's `timeout`, `queries`,
> `queries_shadow` and `mcp_call` when the step sends none. It archives the previous version too.

### 4. Running SOPs Manually
Click **Run** on the SOP's screen to start a run immediately. This is useful for testing new procedures before scheduling them.
The run goes on in the background, as one started with `sop_run`: it ends after
2 hours at most, and it waits for a free slot. Runs started with **Run**,
`sop_run` or a webhook, and runs resumed after a restart, share 4 slots;
scheduled runs do not take one. A paused SOP, or one overdue for review, is not
run: the page says why. Resume it (**Resume** on its screen), or ask the agent
to review it (its `sop_review` tool sets the next review date), first.

### 5. Viewing Execution History
Open the **History** tab to see a detailed log of past runs.
- **Status**: `completed` (every step passed, or a step's `end_run_if` ended
  the run early with nothing failed: the entry says which step and why; see
  [Ending a run early, or skipping a step](#ending-a-run-early-or-skipping-a-step)),
  `partial` (the run reached its last step, or ended early, but at least one
  step failed), `failed`, `escalated` (a step with
  `failure_mode: escalate` ended the run), `interrupted` (a restart cut the run
  off and it was not resumed; see [Time limits and restarts](#time-limits-and-restarts)),
  or `running`. A run is `failed` when a step with `failure_mode: abort` stopped
  it, when it could not start because a prerequisite or virtual level check
  failed, when one step was reached more than 3 times through
  `on_success_goto`/`on_failure_goto` jumps, or when an attempt did not stop
  after running out of time.
- **Steps**: Shows how many steps passed (e.g., `3/4`).
- **Trigger**: `manual` (**Run**, or the agent's `sop_run` tool), `cron`, or
  `webhook:` for a run a webhook started: `webhook:graylog` for a
  [Graylog alert](graylog-alerts.md), `webhook:gitlab` for a
  [GitLab event](gitlab-webhook.md). See [Runs started by a webhook](#runs-started-by-a-webhook).

## Runs started by a webhook

A SOP also runs when an outside system calls the gateway. A webhook route with a
`sop_id` runs that SOP for the event instead of a chat turn: a
[Graylog alert](graylog-alerts.md) works this way, and so can a
[GitLab event](gitlab-webhook.md) or anything else that can POST JSON. Routes are
made on the **Webhooks** page, in [`config.yaml`](../configuration.md#webhook-routes),
or, for a Graylog alert, by the agent.

- **One run per event.** With `batch_mode: bulk` the route gathers its events
  for `batch_window` seconds (default 300) and runs the SOP once for them, at
  most 500 events per run.
- **The runs of one object take turns.** An event about an object whose run is
  still in progress — an issue or merge request and its comments, one alert and
  group-by value — is answered `200`, queued, and waits for that run. When it
  ends, the events that came meanwhile start one run together, every payload
  in the order they came, so a person editing an issue field by field (GitLab
  sends an event per field) starts two runs rather than one per field, and the
  second sees the first one's comment. Queued events survive a restart and run
  once the gateway is back. Events about different objects run side by side, as
  do a GitLab project's pushes, jobs and deployments, which name no object of
  their own. More than 500 events waiting for one object are refused with `503`.
- **The event is data, not instructions.** Every step sees it in a section
  headed *Inbound event* and marked as coming from outside: at most 16 KB of each
  event, and 48 KB for all the events of a bulk run. It is never added to the
  SOP's own context. Whoever can write into the payload (a log line, an MR title)
  writes into that section, so write the steps to treat it as evidence.
- **It runs as `webhook-system`**, on the route's agent. That identity is not
  privileged: whatever tools a step declares, it cannot manage cron jobs or
  SOPs, write system files, or register Graylog alerts. A step that declares
  `db_write` or `metrics_store` can write to a datasource opened with
  `allow_write: true`, as in any other run: only an admin can save a SOP, so the
  step's tool list carries the author's grant. Every step sees the inbound
  event, so declare them only where the write is safe whatever the payload
  says.
- **A restart does not resume it.** A webhook-started run that a restart cuts
  off is closed and reported to the admin (see
  [Time limits and restarts](#time-limits-and-restarts)).
- **The sender is told whether the run started**: `409` when the SOP was
  deleted or disabled or is overdue for review, `503` when too many webhook-started runs
  are already waiting or running. A bulk route answers `200` once the event is
  queued. When its run later cannot start, nothing is sent back: the events are
  dropped when the SOP refuses to run (deleted, disabled, overdue for review) or
  the route was deleted, with the reason noted on the route on the Webhooks page
  and in the gateway log. When too many runs are in progress they stay queued for
  the next round without a note; after any other error they stay queued and the
  error is noted on the route and in the log. A paused route's events wait until it is
  enabled again. The full list is in
  [Gateway Endpoints](../../api-reference/gateway_endpoints.md#webhook-receiver).

## Step Configuration Reference

Each step in the `steps` array has the following fields. These are the **exact
JSON keys**. A key spelled any other way is not an error — it is simply ignored,
and the step runs as if you had never set it. Copy the names from this table
rather than from memory.

| Field | Type | Description |
| :--- | :--- | :--- |
| `id` | string | **Required.** Unique ID for the step. |
| `name` | string | Display name. |
| `intent` | string | **Required, min 20 chars** (optional for pure deterministic query steps). One sentence saying why this step exists. |
| `instruction` | string | **Required, min 80 chars** (optional for pure deterministic query steps). What the agent should do. Can use `{prev_output}` for the result of the step immediately before, or `{{steps..output}}` / `{{steps..file}}` for any earlier step. |
| `output_format` | string | The shape the answer must take. Required in practice for any step that declares `outputs` — the Web UI warns when it does not mention JSON. |
| `constraints` | array | Inline rules the step must obey, injected verbatim into its prompt. |
| `context` | string | Step-level knowledge (markdown) injected into the step prompt. |
| `inputs` | object | Hardcoded parameters, e.g. `{"project_id": "5"}`. |
| `outputs` | array | Optional. Names this step publishes. Declaring them forces the step to answer with a JSON object keyed by exactly these names. |
| `queries` | array | Optional. Deterministic [saved query](../saved-queries.md) bindings (`[{template, datasource, params, as, max_rows}]`). When `queries_shadow` is false, the step runs these queries directly through the worker pool with **zero LLM calls** and publishes `` (the Markdown table) and `_ref` (its `data_ref` ID for `spreadsheet`). Parameter values may be literals, `{{run.date±Nd}}`, `{{run.time±Nh\|m}}` (evaluated in the SOP's `timezone`, defaulting to the server's local timezone), or `{{steps..outputs.}}`. |
| `queries_shadow` | bool | Optional. When `true`, `queries` run in shadow mode (recording version, row count, duration, and errors in `query_runs` on the step result) while the step still executes normally through the LLM. |
| `for_each` | string | Optional. Collection to fan out over — the step runs once per item. Cannot be combined with non-shadow `queries`. |
| `item_var` | string | Optional. Placeholder for the current item, default `item`. |
| `max_items` | int | Optional. Fan-out cap, default 50, ceiling 500. |
| `on_item_failure` | string | Optional. `abort` (default) or `continue`. |
| `mcp_call` | object | Optional, with `for_each`. `{server, tool, arguments, timeout}`: the step calls that MCP tool once per item itself, with **zero LLM calls**, and publishes `results` and `results_ref`. See [Map steps that only call an MCP tool](#map-steps-that-only-call-an-mcp-tool-mcp_call). |
| `uses_outputs` | array | Optional. Earlier results this step consumes — `"s1"` for a whole step, `"s1.staff_list"` for one published field. |
| `tools` | array | Tool names this step may call. **An empty list does not mean "all"** — see [Which tools a step may call](#which-tools-a-step-may-call) below. |
| `success_check` | string | Rule to validate output: `json`, `json_fields:a,b`, `contains:TEXT`, `not_contains:TEXT` (both ignore case), `equals:VALUE`, `min_len:N`, `max_len:N`, `regex:PATTERN`, or `expr:CONDITION`, a comparison of published values (see [Checking published values](#checking-published-values-expr)). Empty means no check. A rule GoClaw does not recognise (a misspelling such as `contain:`), a length that is not a number, or a regex that does not compile is refused when the SOP is saved. A SOP saved before that check runs as it did (an unknown rule, or `min_len` without a number, passes every output), and the gateway logs the rule each time; correct it at the next save. |
| `failure_mode` | string | What to do when retries are exhausted: `retry` (the default: carry on to the next step), `abort` (fail the run and run the rollback), or `escalate` (end the run `escalated`, run the rollback, and send a notice to `sop.alerts` in `config.yaml`, or only to the gateway log when that is not set). **Jumping to another step is a separate field — `on_failure_goto`, not a value here.** |
| `on_failure_goto` | string | Optional. Step ID to jump to when retries are exhausted. Takes precedence over `failure_mode`. |
| `max_retries` | int | How many attempts the step gets in all, the first one included. Default 2 (one retry); `1` means no retry; cap 5. |
| `timeout` | string | Optional. How long one attempt may run before it is cancelled and retried, e.g. `"20m"`; between `1m` and `2h`. Default: `sop.step_timeout` in `config.yaml`, 15 minutes when unset. |
| `on_success_goto` | string | Optional. Jump to a specific `step_id` instead of the next sequential step (Conditional Branching). |
| `end_run_if` | string | Optional. One comparison over published values, e.g. `"{{steps.s3.outputs.rows}} == 0"`, checked after the step passes: when it holds, the run ends there, `completed`, with no model call. See [Ending a run early, or skipping a step](#ending-a-run-early-or-skipping-a-step). |
| `skip_if` | string | Optional. The same kind of comparison over earlier steps' values, checked before the step starts: when it holds, the step is skipped and the run goes on to the next one. |
| `requires_approval` | bool | Optional. After the step answers, the run waits (`suspended` state) until someone approves or rejects the step through the API — see [Human-in-the-Loop](#human-in-the-loop-hitl). Cannot be combined with non-shadow `queries`. |
| `rollback_command` | string | Optional. A shell command that undoes this step, run when a later step ends the run with `abort` or `escalate` (the newest step's first). It runs only if `tools.shell.allow` and the deny lists permit it, and each command is stopped after 60 seconds. |

## Which tools a step may call

A step calls **exactly what the SOP declares for it, and nothing else**. Nothing
is chosen for it at runtime by relevance ranking.

The effective list is built like this:

1. the step's own `tools`, **unioned** with the SOP-level `tools` (declare
   shared tools such as `memory_write` or `notify` once at SOP level instead of
   repeating them on every step);
2. `"*"` in either list stands for **every registered tool**;
3. **nothing declared in either list means the step runs text-only** — it works
   from its inputs and produces an answer, and any tool call it attempts is
   refused;
4. a virtual level's tool grants **narrow** the result; they never widen it.

Six side-effect-free internal tools are available to every step regardless of
what it declares: `calc`, `datetime`, `json_transform`, `read_output`, `regex`,
`text_utils`.

> **A SOP sends only to its pinned targets.** Inside a run, `message` and
> `notify` send to the conversation pinned for that SOP and channel, whatever
> target the model names; with no pin for the channel, the call is refused
> (*`no pinned discord target for SOP …`*) and the step fails. Pin the target on
> the SOP's **Channels** tab, with `POST /api/sop/channels`, or by asking the
> agent in the destination group to run `sop_pin_channel`. `notify` to
> `channel: "webhook"` posts to the URL given and needs no pin.

> **`message` and `notify` are not automatic.** A step that sends must declare
> the sending tool itself. A step scoped to, say, `["db_query"]` that tries to
> `message` is refused at runtime with
> *`tool "message" exists but was not offered for this request`* — the report is
> built and then never leaves the agent. Put `message` (or `notify`) on the step
> whose job is to send, and only on that step: a metrics step that can also post
> will eventually post its raw intermediate JSON to your channel.

> **A send that goes nowhere fails the step** (from 0.0.86). A step passes on
> its own answer, not on whether its tools worked — which is right for a query,
> since "no rows" is a real answer, and wrong for a send. So delivery is checked
> separately: if a step called `message` or `notify` at all and **every** one of
> those calls failed, the step does not pass, whatever it wrote. A step that
> never tried to send is unaffected, and one that failed on `message` but got
> through on `notify` has delivered and still passes.
>
> The verdict is applied once, after the retries, not inside them: a channel
> that has stopped accepting messages will not start because the step ran again,
> so a retry would only repeat the queries and the tokens to reach the same
> rejection.
>
> Every tool error a step hit is recorded on its result as `tool_failures`
> (tool, calls, failures, last error) whether or not the step passed, so a step
> that quietly got nothing back from a query is visible too.

> **A retry does not do again what an earlier attempt did.** A step that fails
> its check is retried to correct its answer, not to repeat its work:
>
> - When an earlier attempt's `message` or `notify` went out, the retry is offered
>   neither, and is told the message was delivered.
> - When an earlier attempt's call that changes something went through, the
>   retry is not offered that tool, and is told what was already done. This
>   covers `memory_write`, a `metrics_store` record, `db_write`, `file_write`,
>   `email` and every other tool that is not read-only (see
>   [Time limits and restarts](#time-limits-and-restarts)).
> - A tool that reads or writes depending on the call stays on the retry, for
>   its reads, and a write through it is refused. These are `http_request`,
>   `metrics_store`, and the Uptrace, Grafana and Zabbix tools.
> - A call that failed did not go through, so the retry may make it again.
> - Two tools stay available whatever they did. `mcp` may only have read, and a
>   retry may need the data again; `db_analyze` writes only its own schema cache.
> - Every attempt starts a new conversation, so the retry is handed what the
>   earlier ones got, under *From earlier attempts*: the `data_ref` of every
>   result they stored (up to 20), which `read_output` reads without fetching
>   the data again, and what each call that went through returned (up to 10,
>   400 characters each), so that its answer can say what was done.

Tool names must exist in the agent's registry — check `/api/tools` for the live
list. **MCP servers are not tools**: reach them through the `mcp` tool (or
`db_query` for a datasource), naming the server in the call.

From 0.0.86 the step's prompt is scoped the same way: the workspace `TOOLS.md`
catalogue is narrowed to the tools that step may call, so it is no longer shown
descriptions of tools it would be refused for using. If your own `TOOLS.md` does
not use the `- **tool_name** — description` form, it is passed through whole
instead of being filtered.

> A step still publishing several messages is not blocked — a step may have more
> than one thing to say — but a non-map step that publishes more than once logs a
> warning, because one report split across posts arrives out of order.

## Passing data between steps

A step does **not** automatically see what earlier steps produced. `{prev_output}`
only ever means the step immediately before, so a step that needs results from
two different places used to have no way to get them.

The following fictional work-assignment SOP uses synthetic staff, jobs and categories:

| Step | Produces |
| :--- | :--- |
| `s1` | The staff rostered on shift today |
| `s2` | Pending jobs, grouped by category |
| `s3` | Assigns `s2`'s example jobs to `s1`'s staff |

By the time `s3` runs, `{prev_output}` holds only `s2`. Declare the dependency
instead:

```json
{
  "id": "s3",
  "name": "Assign work",
  "intent": "Distribute pending category jobs across the staff on shift today",
  "uses_outputs": ["s1", "s2"],
  "instruction": "Use the staff roster from step s1 and the pending job list from step s2. Assign the example jobs, matching each job's category to a staff member rostered for that category.",
  "tools": ["db_query", "file_read"],
  "output_format": "JSON array of {staff_id, job_id, category}"
}
```

The executor injects both outputs under an **Upstream step outputs** heading in
`s3`'s prompt.

> Declaring `uses_outputs` **replaces** the implicit "Previous step output"
> block. A step that asks for one small field should not also be handed the
> whole previous answer — that is the cost it was avoiding. List the previous
> step in `uses_outputs` when you want both.

## Named outputs

A step publishes a single blob of text by default. Declare `outputs` and it
publishes addressable fields instead:

```json
{
  "id": "s1",
  "name": "Shift roster",
  "outputs": ["staff_list", "shift_date"],
  "output_format": "JSON object {staff_list: [{staff_id, name, category}], shift_date: \"YYYY-MM-DD\"}",
  "instruction": "Query the rota for today and return the staff on shift with the category each covers.",
  "tools": ["db_query"]
}
```

The step must then answer with a JSON object carrying exactly those top-level
keys. Markdown fences and a short preamble are tolerated; a missing key or
unparseable JSON **fails the step** and feeds the reflection retry with a
message naming the offending key — so the model is corrected instead of the SOP
quietly continuing with nothing.

Downstream, take one field rather than the whole answer:

```json
{ "id": "s3", "uses_outputs": ["s1.staff_list", "s2.jobs_by_category"] }
```

Referencing a name the producing step does not declare is rejected at save time,
with the real list of names in the error.

### Inline references

For finer control, name a value where you want it to land:

| Reference | Expands to |
| :--- | :--- |
| `{{steps.s1.output}}` | That step's full captured answer, uncapped |
| `{{steps.s1.file}}` | The artifact path of that answer, for `file_read` |
| `{{steps.s1.outputs.staff_list}}` | One published field |
| `{{steps.s1.outputs.staff_list.file}}` | That field's own artifact path |
| `{prev_output}` | The previous step only (unchanged) |

Single braces (`{steps.s1.output}`) work too. A reference that is also declared
in `uses_outputs` is injected once, not twice. A misspelled selector produces an
explicit marker in the prompt rather than passing through as literal text.

## Map steps (one call per item)

A step that handles a whole collection does it in one LLM call: *"assign the
pending jobs across every category"*. That holds until the collection grows.
The model must then fit every category into one prompt and write every
assignment into one answer — which is where long lists get silently truncated,
and where one bad row fails the entire step.

`for_each` turns that into one call per item:

```json
{
  "id": "s3",
  "name": "Assign per category",
  "for_each": "{{steps.s2.outputs.categories}}",
  "item_var": "category",
  "on_item_failure": "continue",
  "uses_outputs": ["s1.staff_list"],
  "instruction": "Fetch the pending jobs for {{category}} with db_query and assign them to staff rostered in {{category}} today. Handle only this category.",
  "outputs": ["assignments", "unassigned_count"],
  "output_format": "JSON object {assignments: [{staff_id, job_id}], unassigned_count: N}",
  "tools": ["db_query"]
}
```

Each iteration's prompt carries **one** category, so its size stays constant
however many categories there are. Every iteration is also told, in its own
header, that it is handling one item of N and must not touch the others.

Map iterations do not automatically receive the previous step's whole answer.
Declare `uses_outputs` or an explicit output placeholder when an iteration
needs additional shared data. Retries keep the same resolved inputs and item
scope, along with feedback from the last attempt.

### The collection

`for_each` is expanded against the run before it is read, so it normally names
an upstream value. A JSON array, one item per line, or a comma-separated line
all parse — but have the producing step publish a **JSON array**, which is the
only unambiguous form. An unresolvable reference fails the step rather than
iterating once over the error text.

`{{item_index}}` (1-based) and `{{item_count}}` are available inside every
iteration alongside your `item_var`.

### Gathering the results

Declared `outputs` are collected across iterations:

- if every iteration returned a **JSON array** for a name, the published value
  is those arrays **concatenated** — one flat list of all assignments;
- otherwise it is the **list of per-iteration values**, in item order.

Original JSON types are preserved: a string ID such as `"123"` stays a string,
and `null` retains its position in the result list.

Failed iterations contribute nothing. The step's own `output` is a JSON document
listing every iteration with its item, status and answer — useful for auditing,
and what lands in the artifact file.

### Failure and limits

| Setting | Behaviour |
| :--- | :--- |
| `on_item_failure: "abort"` (default) | The first failing item fails the step. |
| `on_item_failure: "continue"` | Every item runs; the step passes if at least one did, and the aggregate records which failed. |
| `max_items` | Default 50, ceiling 500. Extra items are not run, rather than failing the step, so one over-broad upstream query cannot become thousands of LLM calls. The step's output says how many were left out (`not_run_items` and a `note`), so the step after it knows the results are incomplete. |

An empty collection is a **pass**, not a failure — there was simply nothing to
do this run.

Cancellation fails the map even if some iterations already succeeded; completed
results remain available for inspection, and no further items are started.

Iterations run **sequentially**. The gain is bounded prompt size and per-item
error isolation, not wall-clock time. `for_each` cannot be combined with
`requires_approval`; approve a summarising step before or after the map instead.

### Map steps that only call an MCP tool (`mcp_call`)

Some map steps do nothing for an item but one MCP tool call whose arguments
come straight from the item: add each ticket of a list to a work plan, create
each row in another system. Run through the model, every item costs an LLM
round trip carrying the whole step prompt. On one pilot agent such a step made
349 calls through about 280 LLM calls of ~45k prompt tokens each, 13.9M tokens
a day, although the model added nothing to any call.

Give such a step `mcp_call` instead of a brief, and the executor makes the
calls itself:

```json
{
  "id": "s3",
  "name": "Add tickets to the plan",
  "for_each": "{{steps.s2.outputs.tickets_ref}}",
  "item_var": "ticket",
  "max_items": 400,
  "on_item_failure": "continue",
  "tools": ["mcp"],
  "mcp_call": {
    "server": "planner",
    "tool": "add_item",
    "arguments": {"ticket_id": "{{ticket.id}}", "owner": "{{ticket.assignee}}", "day": "{{run.date}}"},
    "timeout": "30s"
  }
}
```

- **The rows.** `for_each` may name a `data_ref` (a [saved query step](../saved-queries.md)'s
  `_ref`, or a ref another step published): each row of its table is an
  object of its columns, named as `data_compute` names them (lower case, other
  characters as `_`: `Ticket ID` is `ticket_id`). It may also be a JSON array
  (objects keep their fields and types) or a plain list.
- **The arguments.** A string value may be one whole placeholder:
  `{{ticket}}` (the row), `{{ticket.}}`, `{{item_index}}`,
  `{{item_count}}`, `{{run.date±Nd}}`, `{{run.time±Nh|m}}`,
  `{{steps..outputs.}}` or `{{steps..output}}`. A placeholder
  inside other text (`"ticket {{ticket.id}}"`) is refused when the SOP is
  saved: a row's value goes in as a whole argument, never spliced into a query
  or command the server would read. Table cells are text; the tool's input
  schema turns them into numbers or booleans where it asks for one.
- **The server** is a configured name (`mcp.servers` in `config.yaml`, or an
  approved plugin server of the agent), never a URL, and the step must have
  `mcp` among its tools (or the SOP's), as an LLM step would.
- **What it publishes.** `results` is a table with one line per row: its
  status (`ok`, `failed` or `not run`), the arguments sent, what the server
  answered and the error, each cell cut at 300 characters. `results_ref` is
  the same table as a `data_ref` with every cell whole, for `read_output`,
  `data_compute` or `spreadsheet`. The step's own output is the
  counts and the failed rows, for the step after it. `intent` and
  `instruction` are optional; `outputs` cannot be declared.

Each call is the one the model would have made: `mcp` with `call_tool`,
through the worker pool. The Hooks page's rules see every call with its row's
arguments, every call is recorded in the run's effects journal before it is
made, and each shows in the trace. The step makes **one** attempt:

- more rows than `max_items` (default 50, ceiling 500) fails the step before
  the first call, instead of doing the first 50 of a list that is longer than
  expected;
- `on_item_failure` works as for any map step: `abort` stops at the first
  failed row and marks the rest `not run`, `continue` calls every row and
  passes if one succeeded. A tool error the server answers with is a failed row;
- `mcp_call.timeout` limits one call (default `1m`, between `1s` and `10m`);
  the step's `timeout` limits all of them together;
- a step that has made a call is never run again in the same run: not retried,
  not resumed after a restart, and refused when an `on_failure_goto` leads back
  to it, since every call would be made twice;
- it does not run in a run that acts for a person (a delegated cron or
  virtual-level run): nothing confines an MCP server's calls to what that
  person may see or change.

## Keeping prompts small

Every value that moves between steps costs tokens twice: the producing step
writes it, the consuming step reads it. Three mechanisms bound that, in
increasing order of effectiveness:

0. **Fan out instead of batching.** A `for_each` step's prompt does not grow
   with the collection at all — it is the only option on this list that is
   bounded by construction.
1. **Take the field, not the answer.** `uses_outputs: ["s1.staff_list"]`
   instead of `["s1"]`. Typically halves the handoff, often much more.
2. **Hand over a reference.** Values past the inline limit
   (`sop.inline_output_chars`, 12,000 characters) are cut in the prompt; the
   step is told so and given a `data_ref` and the artifact path, and reads what
   it needs with `read_output` or pages through the file with
   `read_file_lines`.
3. **Do not move the rows at all.** The most effective option, and the one no
   handoff mechanism can substitute for: if a step must emit 5,000 rows, the
   model has to *write* 5,000 rows — slow, expensive, and where long lists get
   silently truncated or invented. Instead have the producing step publish a
   **reference** and let the consumer re-query:

   ```json
   { "id": "s2", "outputs": ["jobs_query", "job_count", "categories"],
     "instruction": "Do NOT return the rows. Return the SQL that selects them, the row count, and the distinct categories." }
   ```

   `s3` then runs `jobs_query` itself with `db_query`. Rows never enter a
   prompt; only the query, a count, and a short category list do.

Rule of thumb: **aggregates and identifiers travel through the prompt; rows do
not.** A step whose output grows with the size of the dataset is a step that
will eventually break.

### Large tool results inside a step

A tool result over `tools.max_result_chars` (default 8,000) — and every
`db_query` result regardless of size — is **spilled** out of the prompt into an
in-memory store, and the step receives a `ref` plus a schema-only preview
instead of the rows. It reads what it needs with `read_output(ref, …)`, or hands
the ref straight to `spreadsheet(data_ref: …)` so the rows never enter a prompt
at all. See [the tools engine](../../architecture/tools_engine.md) for the full
mechanism.

That store evicts after an hour, keeps only the newest 200 entries, and is lost
when the gateway restarts — while a SOP may run for two hours, and a step
waiting on an approval can sit idle for one. A ref produced by an early step and
read by a later one could therefore expire mid-run.

So inside a SOP the spilled result is **also written under the run's own
directory**:

```
/sop_runs///tool-.json
```

`read_output` falls back to that file when the in-memory entry is gone, and puts
it back in memory for the rest of the run. The file belongs to the execution
that produced it — another run of the same SOP cannot resolve the ref — and it
is swept with the rest of that run's artifacts under the same 20-run retention.

One consequence worth knowing: a rehydrated entry carries no pre-generated CSV,
because that file lives in the OS temp directory rather than in the run. A
`spreadsheet(data_ref: …)` call after a rehydration parses the content instead
of streaming the file — slower, same result.

### Artifact files

Every output — and every named output — is written to a file:

```
/sop_runs///.md
/sop_runs///..md
```

Each step's prompt carries an index of the artifacts written so far in that
run, so a step can reach back past `uses_outputs` with `file_read` — without
anyone having to predict a filename.

> The index lists what this step's prompt does **not** already carry (from
> 0.0.86). A value delivered in full through `uses_outputs`, through a
> `{{steps...}}` placeholder, or as the previous step's output is not offered as
> a file, because a step shown a path reads it: on the pilot fleet a reporting
> step re-read its own upstream metrics in 37 of 53 runs, and another spent a
> tool call fetching a 10-character date it had been handed three paragraphs
> earlier. A **truncated** value keeps its entry — that path is the only way to
> the rest of it — and so does every step this one never asked for, which is
> what the index is for.

The execution ID is part of the path, so
concurrent runs of the same SOP never overwrite each other, and the files stay
on disk afterwards for debugging. `sop_history` reports each step's
`artifact_path`. The **20 most recent runs** of each SOP are retained; older
run directories are removed when the SOP next executes.

Automatically injected values longer than **12,000 characters**
(`sop.inline_output_chars`) are cut in the prompt. A note before the value says
how much of it is shown, the line the rest starts at, and where all of it is: a
`data_ref` that `read_output` reads (`slice`, `search` or `sql`), and the
artifact file. **Declare `file_read` on any step that may receive a long list**
— the validator warns when a step consumes upstream output without it — and
`read_file_lines` when it may need to page through one. A step without either
gets the value whole, however long. An explicit `{{steps.s1.output}}` reference
is never cut: naming a value by hand is taken as asking for all of it.

### Validation

Output references are checked when the SOP is saved:

- referencing a step that does not exist → rejected
- referencing a step that runs **later** → rejected
- referencing the step's own output → rejected
- referencing a name the producing step does not publish → rejected
- an output name that is not a plain identifier, or declared twice → rejected
- `for_each` combined with `requires_approval` → rejected
- `item_var` / `max_items` / `on_item_failure` set without `for_each` → rejected
  (the step would have run once, which is not what the author meant)
- `max_items` above the ceiling, or an unknown `on_item_failure` → rejected
- `mcp_call` without `for_each`, with a URL for `server`, without `mcp` in the
  step's or the SOP's tools, with `outputs`, or with a placeholder inside other
  text in its `arguments` → rejected
- an `end_run_if` or `skip_if` that is not one comparison as described in
  [Ending a run early, or skipping a step](#ending-a-run-early-or-skipping-a-step),
  or that reads a step that runs later, a name that step does not publish, or
  (`skip_if`) the step's own outputs → rejected
- a `success_check` of `expr:` with no condition, or one that is not one
  comparison, or that reads a step that runs later or a name that step does
  not publish → rejected

The Web UI also warns when a map step's instruction never mentions its
`item_var` (every iteration would get an identical brief), and when a map step
lists the collection it iterates in `uses_outputs` as well (every iteration
would carry the whole list on top of its own item).

The Web UI additionally warns when a step declares `outputs` but its
`output_format` does not mention JSON.

Failed steps are still addressable by ID; the prompt marks their output as not
having passed its success check. A step that never ran in this execution (a
branch skipped it) yields an explicit "did not run" marker rather than an
invented value.

## SOP v2: Advanced Features

### Conditional Branching (If/Else)
Unlike simple linear scripts, GoClaw SOPs can conditionally skip or loop steps based on verification criteria.
By setting `on_failure_goto: "remediation_step"`, an agent will autonomously detour to fix a failed constraint. If successful, `on_success_goto: "deploy_step"` can jump back to the standard flow. (`failure_mode` only chooses between `retry`, `abort` and `escalate` — the jump target is its own field.)

### Ending a run early, or skipping a step

`on_success_goto` and `on_failure_goto` jump on pass or fail, so a SOP whose
early step finds nothing to do still runs, and pays for, every step after it.
On one pilot agent a daily allocation SOP's third step found no staff for the
slot, and its instructions said to stop there. The run still went on through a
map over zero items, a report and an audit: 766k tokens for a run that
allocated nothing.

Two step fields let the executor make that decision itself, with no model call:

| Field | Checked | When its condition holds |
| :--- | :--- | :--- |
| `end_run_if` | after the step passes | The run ends there. No later step runs. |
| `skip_if` | before the step starts | The step does not run, and the run goes on to the next step. |

```json
{
  "id": "s3",
  "name": "Open requests",
  "outputs": ["requests", "rows"],
  "output_format": "JSON object {requests: [{id, title}], rows: N}",
  "end_run_if": "{{steps.s3.outputs.rows}} == 0"
}
```

```json
{ "id": "s5", "name": "Report", "skip_if": "{{steps.s4.outputs.assignments}} == []" }
```

A condition is **one comparison**:

- Each side is a named output, `{{steps..outputs.}}`, or a value: a
  number (`0`, `-1.5`), a quoted string (`"changed"` or `'changed'`), `true`,
  `false`, or `[]`.
- `==` and `!=` compare numbers when both sides are numbers, and text otherwise
  (the published value trimmed; `true` and `false` ignore case). `<`, `>`, `<=`
  and `>=` compare numbers only.
- `[]` means empty: a blank value, `[]`, `{}`, `null`, or a saved query that
  returned no rows. It goes with `==` or `!=` only.
- There is no `and`/`or` and no arithmetic, and a step's whole answer
  (`{{steps..output}}`) cannot be compared. A decision that needs more
  belongs in the step that publishes the value: have it publish, say,
  `nothing_to_do: true`.

`end_run_if` reads the step's own outputs or an earlier step's; `skip_if` reads
earlier steps only. Both are checked when the SOP is saved.

A run that ends early is `completed` (or `partial` when an earlier step had
failed, with the usual alert) and counts as passed in the SOP's statistics; no
alert is sent for it. Its History entry says which step ended it and what held,
for example `step "s3": end_run_if {{steps.s3.outputs.rows}} == 0 held
(s3.rows = 0)`. So do its trace (the final output and the
`sop_end_reason` metadata), the cron job's last output and `sop_history`. Only
the steps that ran used tokens; a step that reaches the run's token limit and
ends the run completes it rather than failing it. The decision is saved with
the step, so a restart right after it does not go on to the steps it ended.

A skipped step shows in the History as ○ with its reason. It publishes nothing:
a later step that reads it gets the "did not run" marker, and `{prev_output}`
stays the last step that ran. It is not a failure.

A condition that cannot be decided does not hold: the step it reads did not run
or did not pass, or a value that should be a number is not. The run then goes on
as it would without the condition, and the step's record says why.

### Checking published values (`expr:`)

The other `success_check` rules read the step's answer as text, so none of
them can say "fail when these two numbers differ". On one pilot agent a daily
allocation SOP's audit step published 253 assigned against 200 expected and
passed; the report before it had said everyone got their share, and one person
had got none. The audit found the mismatch, and nothing could fail on it.

`expr:` takes one comparison, in the same language as `end_run_if`, and the
step passes only when it holds. The executor decides it from the values the
step published, with no model call:

```json
{
  "id": "s4",
  "name": "Audit assignments",
  "intent": "Count today's assignments against the jobs that were due",
  "instruction": "Count the jobs assigned today and the jobs that were due today with db_query. Publish both counts as found.",
  "tools": ["db_query"],
  "outputs": ["assigned", "expected"],
  "output_format": "JSON object {assigned: N, expected: N}",
  "success_check": "expr:{{steps.s4.outputs.assigned}} == {{steps.s4.outputs.expected}}",
  "failure_mode": "escalate"
}
```

- The step reads its own outputs as `{{steps..outputs.}}`, from
  the attempt being checked, and an earlier step's as usual. Each name it reads
  must be one the step publishes: declare it in `outputs` (or as a query's
  `as`).
- A comparison that does not hold fails the step, and the message names the
  values: `success_check expr:{{steps.s4.outputs.assigned}} ==
  {{steps.s4.outputs.expected}} does not hold (s4.assigned = 253, s4.expected
  = 200)`. The failure takes the usual path: the step's retries, then its
  `on_failure_goto` or `failure_mode`, and the admin alert carries the message.
- A comparison that cannot be decided fails too, and says why: the value is
  not a number, a step it reads did not run or did not pass, the step did not
  publish the name. This is the opposite of `end_run_if`, where a condition
  that cannot be decided lets the run go on: a check that cannot tell is not a
  pass.
- The retry prompt carries the values and tells the model to derive them again
  from the data, not to change them until the check holds. The step's own
  prompt says the same. A check that reads only earlier steps' outputs is not
  retried, since another attempt cannot change them.
- On a map step the check runs on each iteration's own outputs; on a query or
  `mcp_call` step, on the values the step published, with no retry.

Compare two published values, or one with a fixed value
(`expr:{{steps.s4.outputs.unassigned}} == 0`). The check is only as good as
the values: a count the step reads from the database is data, while a number
the model writes from memory is its word. Have the step publish what it
counted, with the tool that counted it.

### Human-in-the-Loop (HITL)
Critical procedures (e.g., executing schema drops or sending broadcast emails) can be safeguarded by setting `requires_approval: true`.
When the agent reaches this step:
1. The step runs once and its answer is held: the step's job enters a
   `suspended` state in the job queue, and the run waits.
2. A notice with the step's output and its job ID goes to `sop.alerts` in
   `config.yaml`, or only to the gateway log when that is not set.
3. A human administrator approves or rejects it. In a chat with the agent, ask
   it to approve (or reject) the SOP step with that job ID: its `sop_approve`
   tool does it, and refuses anyone who is not an administrator, and any SOP,
   cron or webhook run. Or call the API with the gateway token:

   ```bash
   curl -X POST https:///api/sop/approve \
     -H "Authorization: Bearer $GOCLAW_TOKEN" -H "Content-Type: application/json" \
     -d '{"job_id": "", "action": "approve"}'
   ```

   To reject, send `"action": "reject"` and, optionally, a `"reason"`.
4. Approved, the step's `success_check` decides whether it passed, and the run
   goes on from there. Rejected, the step fails and its `on_failure_goto` or
   `failure_mode` applies. A step that nobody approves or rejects within 1 hour
   fails the same way.

The wait for approval lives in the running gateway. A restart ends it, and the
run is then treated like any other run a restart cut off (below).

## Time limits and restarts

**Each attempt of a step has a time limit**, 15 minutes unless `sop.step_timeout`
in `config.yaml` or the step's own `timeout` says otherwise. A whole run is
still capped at 2 hours. An attempt that runs out of time is cancelled and
counts as a failed attempt. What happens next depends on what it had done:

- It had only read: it is retried, with a note to make fewer and smaller tool
  calls, and `failure_mode` decides what happens when the retries run out.
- It had already called something that sends or changes (see below): it is not
  retried, so that is not done twice.
- It did not stop within 10 seconds of being cancelled: it may still be running,
  so the run ends there, whatever `failure_mode` says, and the admin is told.
  The rollback is not run either, since the attempt may still be using what it
  would undo; the alert names the steps whose `rollback_command` is left to a
  person to decide.

**A crash in the SOP executor ends the run, not the gateway.** The run ends
`failed` (the log shows `[sop_v2] PANIC` and the stack): the steps that
finished keep their results, the step it was on says `panic: …`, and the
admin is told through `sop.alerts`. The rollback is not run, since what the run
had done when it crashed is not known; the alert names the steps whose
`rollback_command` is left to a person to decide.

**A run is saved as it goes**: when it starts, and as each step starts and ends.
When the gateway is stopped or restarted in the middle of a run, the next start
finds it and marks it `interrupted`. It then does one of two things.

- **Resume from the step it was on**, when all of these hold:
  - the run was started by a schedule (not by hand or by a webhook), and that schedule still
    exists, is not paused, and still runs as the same person. Several schedules
    can run one SOP for different people; a resume acts only as the one the run
    began for;
  - the SOP has not been edited since the run started (its purpose, steps,
    context, tools, prerequisites, owner and virtual level); the finished steps belong
    to the version the run began with;
  - it started at most `sop.resume_window` ago (10 minutes by default; `"0"`
    never resumes), no newer run of the SOP has started, and the SOP is enabled;
  - the run was not already ending: a step that failed with `failure_mode:
    abort` or `escalate`, a rollback, an attempt that did not stop;
  - the step it was on had made no call that sends or changes something.

  The steps that finished are not run again; their results are carried over.
  The run's history shows where it was interrupted.
- **Otherwise close it and tell the admin** through `sop.alerts`, with the reason
  it was not resumed. Check what the step had done before running the SOP by
  hand: the alert names its calls.

**What counts as a call that sends or changes something**: every tool call that
is not known to be read-only. Read-only are database queries (`db_query`, which
only runs SELECT, SHOW, DESCRIBE and EXPLAIN), Uptrace, Graylog, Wazuh and Grafana
queries, Zabbix `*.get` methods, `metrics_store` `query` and `trend`,
`http_request` GET and HEAD, memory and file
reads, and the date, calculation and text tools. Everything else counts —
`message`, `notify`, `db_write`, `shell_exec`, MCP tools, an Uptrace or Grafana
annotation, and any tool not listed — whether the call finished or not: a
message may or may not have arrived, a row may already be written. Such a call
is recorded before it is made; if the record cannot be written (a locked
database, a full disk), the call is refused and the step sees the error, rather
than make a call that recovery would not know about. A refused call changed
nothing, so it does not stop a timed-out attempt from being retried.

```yaml
sop:
  alerts: { channel: msteams, target: "" }
  step_timeout: 15m
  resume_window: 10m
  max_run_tokens: 500000
  heavy_run_tokens: 200000
  inline_output_chars: 12000
  max_output_tokens: 8192
```

## Token limit of a run

**A run stops at 500,000 tokens** (`sop.max_run_tokens`), its steps, attempts and retries together. It is checked before every model call. A run that reaches it ends `failed`: it does not retry, does not go on to the next step, and its rollback runs. A cron job whose run stopped there is paused (`token_limit` on the Cron page), so it does not spend as much again at the next tick. A run past 200,000 tokens (`sop.heavy_run_tokens`) is marked heavy. A SOP whose early step can find nothing to do should stop there with [`end_run_if`](#ending-a-run-early-or-skipping-a-step) rather than run its later steps to say so. The run history shows what each run and step used. See [Token Budgeting](../token-budgeting.md#3-sop-run-limit) for what a normal run uses, and for raising or turning off the limit.

## How much of a step's result reaches the next step

A step's result goes through several limits on its way to the steps after it.
Each protects the run's token budget, and each says so when it cuts in:

| What | Limit | Setting | When it cuts in |
| :--- | :--- | :--- | :--- |
| A step's answer | 8,192 output tokens a model call (a chat reply on Claude, OpenAI, DeepSeek and Ollama: 4,096). A Gemini agent keeps `ai.gemini.max_output_tokens`, 16,384 by default, when that is higher. | `sop.max_output_tokens`; 0 = the provider's own | The model stops mid-answer. The step is asked again, for a shorter answer that gives counts and the `data_ref` of the rows rather than the rows. If its last attempt is cut too, its output ends with *[This answer stopped at the model's output limit …]*, and a step that publishes `outputs` fails with that reason. |
| An earlier step's output in a step's prompt | 12,000 characters, for a step with `file_read` or `read_file_lines` | `sop.inline_output_chars`; 0 = never cut | The value is cut, with a note before it naming its `data_ref` and file. See [Artifact files](#artifact-files). |
| A saved-query step's output | 12,000 characters; 256K when every step that reads it has `file_read` or `read_file_lines` | `sop.inline_output_chars` | The step fails: *query step output is N characters, exceeding the M-character limit*. |
| A tool result inside a step | 8,000 characters | `tools.max_result_chars` | The step sees a preview and a `ref`. See [Large tool results inside a step](#large-tool-results-inside-a-step). |
| A step's older tool results | Results older than the newest 4 rounds, over 2,000 characters | `tools.full_result_rounds`; -1 = never | Each is cut to its first lines and the `ref` that reads it whole. |
| A step's context | 80,000 tokens | `memory.auto_flush_tokens` | The step's older tool results leave the context; a note lists each with the `ref` that reads it whole. The step's own prompt, with the earlier steps' outputs, stays. |
| `mcp_call` results | 300 characters a cell of `results` | — | `results_ref` holds every cell whole. |
| A map step's items | `max_items`, 50 by default, ceiling 500 | the step's `max_items` | Items past it are not run, and the step's output says how many. |
| A run | 500,000 tokens | `sop.max_run_tokens` | The run stops. See [Token limit of a run](#token-limit-of-a-run). |

Every `data_ref` made in a run is also written under the run's directory, so it
still reads after the in-memory store's hour, or a restart. The run's record in
the database keeps every step's output whole: no column there is limited.

**Every cut is shown.** A run in which a limit cut data says **⚠ data cut** in
the SOP's **History** tab; opened, each step lists its cuts under it (what was
cut, the setting, where the rest is), and the run's result keeps them as the
step's `warnings`. On the **Traces** page the trace has the same tag and a ⚠
span under the step. See [Where a limit cut data](../../architecture/observability.md#where-a-limit-cut-data).

## What happens to SOPs you never touch

> **One exception, from 0.0.85 — read this before upgrading.** A step that
> declares no `tools`, in a SOP that declares none either, used to have its
> tools chosen for it at runtime by relevance ranking. It now runs **text-only**
> and every tool call it makes is refused. If such a step used to query a
> database or send a message, it will stop doing so silently — the run can still
> end `completed`. Give it the tools it needs, or `"*"`, before you upgrade. A step
> that already declares its tools is unaffected. In the same release, `"*"` is
> understood for the first time: a SOP saved through the `@sop:` directive
> writes `tools: ["*"]`, and before 0.0.85 that meant no tools at all.

Otherwise, taking the binary update without migrating anything is safe. The
data-handoff fields are opt-in, and a SOP that declares none of them is executed
the way it always was:

- `{prev_output}` and the "Previous step output" block behave identically.
- **Nothing is ever truncated that the step could not fetch back.** The 12,000
  character cap only applies when the step declares `file_read` or
  `read_file_lines` — otherwise the value is passed through whole, however
  large, exactly as before. A step is never handed a pointer it cannot follow.
- The artifact index is only shown to steps that can read files, so a legacy
  step's prompt gains nothing it cannot act on.
- Validation only rejects the new fields. A SOP without them cannot fail a
  check it passed before, and validation does not run at execution time anyway.
- Cron, the job queue and the scheduler are untouched.

From 0.0.86 one more thing changes for every SOP: a run that reaches its last
step with any step failed ends **`partial`** instead of `completed`, and
`run_stats` counts it under Failed. Nothing about how the run executes changed —
`failure_mode: retry` still carries on to the next step — only what the record
says afterwards. A SOP that was quietly failing a step every run will start
reporting it; that is the point, and the number it reports was always the true
one.

Two things do change for every SOP, migrated or not:

- **Step outputs are written to disk.** Each step's result lands in
  `sop_runs///.md` inside the workspace. The newest
  20 runs per SOP are kept and older ones are swept on the next execution, so
  growth is bounded, but a SOP with large outputs on a frequent schedule will
  use noticeably more disk than before. A failed write is logged and the run
  continues.
- **A step whose LLM call fails outright now ends in a terminal job state.**
  Previously a stream or drain error on the final attempt left the step's row
  in the job queue stuck at `running` forever; it is now marked `failed`.

## Upgrading existing SOPs

SOPs written before the handoff model still run unchanged — and still carry its
failure. [sop-migration-prompt.md](sop-migration-prompt.md) holds a prompt you
hand to an agent so it audits and upgrades the SOPs it owns, with a diagnosis
checklist and the guardrails to keep it from rewriting what already works.

## Best Practices

- **Atomic Steps**: Keep instructions focused. Instead of "Fix the code and deploy", use Step 1: "Find the bug", Step 2: "Write the fix", Step 3: "Run tests", Step 4: "Deploy".
- **Strict Criteria**: Use `json` criteria for steps that should output structured data to ensure follow-up steps can parse it correctly.
- **Declare Your Inputs**: If a step consumes an earlier step's result, list it in `uses_outputs`. Relying on `{prev_output}` alone silently loses everything older than one step.
- **Publish Fields, Not Blobs**: Declare `outputs` on any step whose result is consumed in parts. A downstream step that needs one field should ask for one field.
- **Never Page Rows Through the LLM**: If the row count grows with the data, publish a query or a table name instead of the rows themselves.
- **Fan Out Over Groups**: If a step says "for each category / branch / customer", make it a `for_each` step. One call per group beats one call for all of them, and a failure is isolated to its own item.
- **Least Privilege**: Only allow necessary tools for each step to minimize the risk of unintended actions during autonomous runs.
- **Decouple Frequent Collection from Reporting (Paired SOPs)**: When monitoring systems 24/7, do not attempt to run a 24-hour analysis in a single run or flood channels hourly. Instead, pair an hourly collector (`_hourly`, e.g. `0 0 * * * *`) that records data via `metrics_store` with an evening summary SOP (`_daily_summary`, e.g. `0 0 22 * * *`) that queries the day's metrics and prepares executive digests.

================================================================================
# FILE: https://goclaw.ulab.mn/docs/user-guide/automation/sop-migration-prompt.md
================================================================================

# Upgrading existing SOPs — agent prompt

The step data-handoff model (`uses_outputs`, `outputs`, `for_each`) is opt-in.
Every SOP written before it still runs exactly as it did, which also means every
SOP written before it still has the old failure: a step can only see the result
of the step immediately before it, and a step that handles a whole collection
handles it in one call.

This page holds a prompt you give an agent so it audits and upgrades the SOPs it
owns. Paste it into a chat with the agent.

## Before you start

- **Give it as an administrator.** `sop_save` and `sop_rollback` refuse anyone
  but an admin (or the system), so paste the prompt in a conversation where you
  are one.
- **Every `sop_save` archives the previous version.** `sop_versions ` lists
  them and `sop_rollback  ` restores one, so a bad rewrite is one
  call away from being undone. Say so to the operator if they hesitate.
- **Keep the agent away from SOPs with a virtual level or step time limits.**
  `sop_save` replaces the whole SOP and has no way to keep a `virtual_level`,
  and `sop_get` does not show a step's `timeout`, so the agent's save removes
  both. Name those SOPs in the prompt as ones to skip, and change them by hand
  through `POST /api/sop` (see [Editing an SOP](sop.md#3-editing-an-sop)). A
  save that removed them is undone with `sop_rollback`.
- **Run it against one agent at a time**, and read the report before letting it
  loose on the next.
- **Do not let the agent run the modified SOPs.** Most pilot SOPs write
  somewhere — assign work, send messages, update tickets. Re-running one to
  "check the change" performs those side effects again. The prompt below tells
  the agent to stop and ask.

## The prompt

````text
You are upgrading the Standard Operating Procedures you own so they use the
platform's step data-handoff model. Work through them ONE AT A TIME and finish
with a report.

# Background: what the platform now supports

1. `uses_outputs` — a step declares which earlier results it consumes:
   ["s1"] for a whole step's answer, ["s1.staff_list"] for one published field.
   Without it a step sees ONLY the step immediately before it. Declaring
   uses_outputs replaces that implicit previous-step block.

2. `outputs` — a step declares the values it publishes:
   "outputs": ["staff_list","total_count"]. The step must then answer with a
   JSON object whose top-level keys are exactly those names, and later steps
   read them as {{steps.s1.outputs.staff_list}} or via uses_outputs.

3. `for_each` — a step runs once per item of a collection instead of once for
   all of them:
     "for_each": "{{steps.s2.outputs.categories}}", "item_var": "category"
   and the instruction refers to the current item as {{category}}.
   Each iteration's prompt carries one item, so it stays the same size however
   long the list grows. Declared outputs are gathered across iterations:
   arrays concatenate into one list, scalars become a list.

4. Every step's output — and every named output — is written to
   sop_runs///[.].md, and each step is handed
   the exact paths. Values over 12000 characters are truncated in the prompt
   with a pointer to that file.

# Procedure

1. Call `sop_list`, then `sop_get` each SOP, and take only those whose Owner is
   your agent ID. Leave out the SOPs the operator told you to skip.
2. For each SOP, in order:
   a. `sop_get` the full definition.
   b. `sop_history` for its recent runs — a step that keeps failing, or an
      execution that "completed" while producing nothing, is the strongest
      evidence of the problem you are fixing.
   c. Diagnose it against the checklist below.
   d. If nothing on the checklist applies, say so and move on. DO NOT rewrite
      a SOP that is already correct — churn on a working SOP is a regression,
      not an improvement.
   e. Otherwise apply the smallest set of changes that fixes what you found,
      and `sop_save` that one SOP.
3. Write the report described at the end.

# Diagnosis checklist

Check every step of every SOP against ALL of these — a step can match several,
and C in particular hides behind whatever else the step does. Each line is:
what you see → what you change.

A. FAN-IN — the step's instruction refers to a result from further back than
   the step immediately before it. Look for phrases like "the list from step 1",
   "the employees found earlier", "as identified above", "өмнөх алхамд гарсан"
   ("from the previous step"), "дээрх жагсаалт" ("the list above"), or a
   {prev_output} that is meant to reach back further than one step.
   → Add `uses_outputs` naming those steps. Keep the instruction's wording; it
     is now true instead of aspirational.

B. PARTIAL CONSUMPTION — a later step needs only part of what an earlier step
   produced (its list, its count, its date), but receives the whole answer.
   → Add `outputs` to the PRODUCING step naming the parts, set its
     `output_format` to a JSON object with exactly those keys, and have the
     consuming step take `uses_outputs: ["s1.that_field"]`.
   → This is the main token saving. Prefer it over passing whole answers.

C. COLLECTION IN ONE CALL — a step's instruction says it handles every member
   of a group. Read the instruction for the quantifier, in whatever language it
   is written:
     English  — "for each X", "for every X", "per X", "each X in the list",
                "all X", "one by one"
     Mongolian — "X бүрийн хувьд", "X тус бүрийн хувьд", "X тус бүрээс",
                "X бүрийг", "бүх X-ээр", "X бүр дээр", "нэг бүрчлэн"
                (all: "for each X", "per X", "every X", "one by one")
   This is the finding most often missed. If the instruction quantifies over a
   collection ANYWHERE in its text — even in a subordinate clause after some
   other work — it is a candidate. Check every step for it explicitly before
   moving on.
   → Turn it into a `for_each` step over the collection, with `item_var` named
     after the domain, and rewrite the instruction to handle ONE item.
   → The step that produces the collection must publish it as a named output
     containing a JSON array.
   → SPLIT WHEN THE LIST IS FETCHED INSIDE THE STEP. If one step both obtains
     the collection and then processes every member of it ("read the source
     list from the wiki, then for each site fetch its content"), `for_each`
     cannot reference a list that does not exist yet. Split it in two:
       s1a — obtain the collection, publish it: outputs: ["sites"]
       s1b — for_each: "{{steps.s1a.outputs.sites}}", handling ONE site
     Renumbering steps is allowed here. Update every reference to the old step
     id, and say in the report that you split it.
   → Do NOT also list the iterated collection in `uses_outputs`. `for_each`
     already hands each iteration its own item; adding the collection puts the
     whole list into every iteration's prompt, which is exactly the growth the
     map step exists to remove. The validator warns about this.
   → Set `on_item_failure: "continue"` when one bad group should not stop the
     rest; leave it unset when it should.
   → `for_each` cannot be combined with `requires_approval`. If the step needs
     approval, split it: a map step that prepares, then a separate approval
     step that commits.

D. ROWS THROUGH THE PROMPT — a step returns rows whose count grows with the
   data ("return all pending jobs", "list every unpaid invoice"), and a later
   step consumes them.
   → Change the producing step to publish a REFERENCE instead: the SQL that
     selects the rows, a count, and any small grouping list. The consuming step
     re-runs the query with db_query.
   → Aggregates and identifiers travel through the prompt; rows do not. A step
     whose output grows with the dataset will eventually truncate silently.

E. MISSING TOOLS — a step consumes an earlier step's output but does not
   declare `file_read`.
   → Add `file_read`, and `read_file_lines` when the value may be long.
   → Only ever ADD these two tools. Do not add any other tool, and never remove
     one.

F. ORDERING — a step references a result produced by a step that comes LATER in
   the array.
   → This is rejected at save time. Reorder the steps, or drop the reference —
     whichever matches what the SOP actually needs. Say which you chose.

G. PROBLEMS YOU ARE NOT ALLOWED TO FIX — while reading a step you will see
   faults outside your remit. Do not fix them; do not stay silent about them
   either. Collect them under "found but not fixed" in the report, one line
   each, naming the SOP, the step and the exact symptom.
   The one to look for in every step: the instruction tells the agent to call a
   tool that the step's `tools` array does not declare. At runtime that fails
   with 'tool "" exists but was not offered for this request', so the
   step has never worked
   as written. Compare every tool name mentioned in the instruction against the
   declared list, and report each mismatch. You may still only ADD file_read
   and read_file_lines yourself — adding anything else is an authority change
   and is the operator's decision, not yours.

# Rules you must not break

- Keep every instruction, intent, constraint and context in ITS ORIGINAL
  LANGUAGE. Do not translate anything. If the SOP is written in Mongolian, your
  edits are in Mongolian.
- Do not change what a SOP does. You are changing how data moves between its
  steps, not its purpose, its schedule, or its business rules.
- Do not touch: id, name, purpose, owner, cron, enabled, prerequisites,
  virtual_level, rollback_command, requires_approval, on_success_goto,
  on_failure_goto, end_run_if, skip_if, success_check, max_retries,
  failure_mode.
- You may change: instruction (wording needed to reference data correctly),
  output_format, outputs, uses_outputs, for_each, item_var, max_items,
  on_item_failure, and tools (to ADD file_read / read_file_lines only).
- `sop_save` replaces the whole SOP with what you send. Send `owner`, the
  SOP-level `tools`, `context` and `prerequisites` exactly as `sop_get` showed
  them, and every field of every step, including the ones you did not change.
  Anything you leave out is removed; an owner left out becomes `default`.
- One `sop_save` per SOP. Never batch several SOPs into one call.
- If `sop_save` rejects your change, read the error — it names the exact
  problem — fix it and retry. After three failed attempts on the same SOP,
  leave that SOP UNCHANGED and record it in the report as needing a human.
- Do NOT call `sop_run`. These SOPs assign work, send messages and update
  records; re-running one repeats those effects. If a change should be
  verified by running it, say so in the report and let the operator decide.
- If a SOP is unclear enough that you cannot tell what a step consumes, do not
  guess. Leave it alone and list it under "needs a human".
- Report only what the definition, `sop_history` and the change log actually
  show. Do not infer why a SOP is disabled, what it was superseded by, or what
  anyone intended — if the reason is not written down, say the state you
  observed and that the reason is not recorded. An invented history is worse
  than no history: it gets acted on.

# Report

Finish with a table, one row per SOP:

| SOP | Finding (A–G, or "none") | What you changed | Status |

Then, separately:
- SOPs left unchanged because they were already correct.
- **Found but not fixed** — every finding G, one line each: SOP, step, symptom.
- SOPs that need a human, and the specific question you would ask about each.
- Any SOP where you recommend a test run, and why.
````

## A worked example

This fictional example uses synthetic staff and jobs. The prompt turns this:

```json
{ "id": "s3", "name": "Assign jobs",
  "instruction": "Assign the jobs from step 2 to the staff from step 1, by category." }
```

— where `s3` in fact only ever saw `s2` — into this:

```json
{ "id": "s3", "name": "Assign jobs",
  "uses_outputs": ["s1.staff_list", "s2.jobs_by_category"],
  "instruction": "Assign the jobs from step 2 to the staff from step 1, by category.",
  "outputs": ["assignments", "unassigned_count"],
  "output_format": "JSON object {assignments: [{staff_id, job_id, category}], unassigned_count: N}",
  "tools": ["db_query", "file_read"] }
```

The instruction is untouched — it was always describing the right thing. What
changed is that it is now true.

## After the run

- Read the report before enabling anything on a schedule.
- Spot-check one upgraded SOP with the operator present.
- `sop_versions ` and `sop_rollback  ` undo any single change.

See [sop.md](sop.md) for the full field reference.

================================================================================
# FILE: https://goclaw.ulab.mn/docs/user-guide/automation/cron-jobs.md
================================================================================

# Scheduled Tasks (Cron Jobs)

GoClaw features an integrated scheduler that allows agents to create, manage, and execute automated periodic tasks (Cron Jobs). This enables agents to perform routine health checks, generate daily reports, or pull data feeds at specified intervals.

---

## 1. How It Works

Using the built-in scheduler, agents can register automated tasks. A new job runs a Standard Operating Procedure (SOP) workflow, named by its `sop_id`, or one of two built-in commands:

- `notify channel= target= text=` sends the message to that channel.
- `agent [channel=] [target=] text=` has the agent answer the prompt and posts the answer to that channel.

When `target=` is left out, the job goes to the conversation it was created from: its ID becomes the target and, unless the command names a `channel=`, its channel the channel. A job with a `target=` and no `channel=` goes to Discord. Creating a job sends nothing: a malformed `notify` or `agent` command (no `text=`, or no target) is refused when the job is created. Raw shell commands are refused for new jobs (`raw shell commands are not allowed for new cron jobs`), and so is a job with neither a `sop_id` nor a built-in command. A job with a shell command made before that rule runs only if the command passes the same lists as the shell tool (`tools.shell.allow` / `deny` and the built-in deny list); otherwise it is recorded as not run, with the reason `blocked`. One that runs is stopped after 10 minutes, with everything it started.

```mermaid
flowchart LR
    A["Agent"] --> T["Cron Tool"]
    T --> S["Cron Scheduler"]
    S --> W["SOP Workflow / Built-in Command"]
    S --> DB[("Database
Persisted")]
```

---

## 2. Scheduler Operations

You or the agent can manage scheduled jobs using the `cron` tool:

- **`create`:** Register a new scheduled job.
- **`list`:** Display all scheduled jobs.
- **`delete`:** Remove a scheduled job by its unique ID.
- **`history`:** Show a job's schedule, run count, last run, last error and next run.

Creating and deleting a job take an administrator (or the system acting on its own); anyone else is refused.

The **Cron** page of the Web UI pauses and resumes a job, runs it now, and deletes it. A schedule is changed through `POST /api/cron` with `{"action": "update", "id": "cron-3", "schedule": "0 0 8 * * *"}` and the gateway token, by the agent's `sop_save` with a new `cron_schedule` for the SOP's linked job, or in the SOP's **Cron schedule** field on the SOPs page.

Two maintenance jobs are built in: `sys-memory-consolidate` (daily at 03:00) and `sys-trace-prune` (daily at 02:30, keeps the newest 5,000 traces). Neither the Cron page nor the `cron` tool deletes them (a deleted one would be created again, with its default schedule, at the next start); pause one instead.

---

## 3. Schedule Syntax

GoClaw supports a precise **6-field cron syntax** (which includes seconds for granular scheduling):

```
┌───────── second (0 - 59)
│ ┌─────── minute (0 - 59)
│ │ ┌───── hour (0 - 23)
│ │ │ ┌─── day of month (1 - 31)
│ │ │ │ ┌─ month (1 - 12)
│ │ │ │ │ ┌ day of week (0 - 6, Sunday = 0)
│ │ │ │ │ │
* * * * * *
```

### Scheduling Shortcuts

| Shortcut | Description | Equivalent Cron |
|----------|-------------|-----------------|
| `@every 30s` | Runs every 30 seconds | None: the interval counts from when the job was scheduled (created, or the gateway started), not from the clock |
| `@every 1h` | Runs every hour | None, as above: not at minute 0 |
| `@hourly` | Runs once an hour at minute 0 | `0 0 * * * *` |
| `@daily` | Runs once a day at midnight | `0 0 0 * * *` |

A five-field expression such as `0 9 * * *` is read as firing at second 0 and stored as `0 0 9 * * *`.

---

## 4. Persistence & Reliability

All scheduled tasks are persisted to the local database, ensuring they are automatically restored if the GoClaw gateway restarts. Only the gateway runs them. A job created or deleted from another process (`goclaw tools exec cron`, or the agent in `goclaw chat`) is picked up by the running gateway within 30 seconds.

### Job Configuration Fields

- **ID:** The unique job identifier (e.g., `cron-1`).
- **Schedule:** The 6-field cron or shortcut expression.
- **SOP ID:** The SOP the job runs.
- **Command:** For a job without a SOP, the built-in command it runs (`notify …`, `agent …`; the two maintenance jobs run `memory_consolidate` and `trace_prune`).
- **Concurrency:** `forbid` (the default) skips a run while the previous one is still running; `allow` lets runs overlap.
- **Subject (`subject_ref`):** The person the job runs for. With it, a SOP job's run, or an `agent …` job's turn, runs as that person, at user level, and can never take admin or system actions; an `agent …` turn has a conversation of its own with its target (`p2p:::for:`), on the web channel too. Without it, a SOP job runs as the system (`cron-system`) and an `agent …` job as its creator (see Created By). Leave it empty only for maintenance jobs, and for an `agent …` job that should act with its creator's authority.
- **Created By:** The user identity that set the job up. An `agent …` command job without a subject runs its turn as this user, with their pairing role (an admin's job runs as that admin), except on the `web` channel, where it runs as the system in the admin's chat (`admin-session`), whoever created it. For SOP jobs, and jobs with a subject, it is a record only.
- **Run Count:** The total number of times the job has run since creation.
- **Enabled:** Whether the job is scheduled or paused.
- **Last Run:** Timestamp of the last run.
- **Last Error:** The error of the last run, if it failed, with its category (below).

---

## 5. Error & Failure Tracking

The scheduler automatically classifies execution failures to help administrators diagnose issue patterns:

| Error Category | Trigger |
|----------------|---------|
| `sop_failure` | The job's SOP was not found, or its run ended `failed`, `partial`, `escalated` or `interrupted`. |
| `panic` | The job crashed inside the gateway. |
| `blocked` | A shell-command job was not run: its command is not on the shell allowlist, or is on a deny list. |
| `timeout` | A command job's error mentions a timeout, or a shell-command job ran past 10 minutes. |
| `auth` | A command job was refused: permission or access denied. |
| `not_found` | Something a command job needed was not found. |
| `connection_refused` | A command job's network request failed to connect. |
| `unknown` | Any other command job error. |

---

## 6. Example Interactions

Here is how an agent interacts with the scheduler during a chat session:

**User:** *"Run the disk-check SOP every hour."*

**Agent:** *"I will schedule the disk-check SOP hourly."*
- *Tool Call:* `cron(action="create", schedule="0 0 * * * *", sop_id="disk-check")`
- *Response:*
  ```
  Created job cron-1
    Schedule: 0 0 * * * *
    SOP: disk-check
    Next run: 2026-10-08 15:00:00
  ```

**User:** *"Remind this channel about stand-up at 09:50 on weekdays."*

- *Tool Call:* `cron(action="create", schedule="0 50 9 * * 1-5", command="notify channel=discord text=Stand-up in 10 minutes")`
- *Response:* `Created job cron-2` with its schedule, the command (with this conversation filled in as `target=`) and the next run.

**User:** *"Show all my scheduled jobs."*

**Agent:**
- *Tool Call:* `cron(action="list")`
- *Response:*
  ```
  4 job(s):

  | ID | Schedule | Command | Runs | Last Run | Next Run |
  |---|---|---|---|---|---|
  | cron-1 | 0 0 * * * * |  | 3 | 14:00:00 | 15:00:00 |
  | cron-2 | 0 50 9 * * 1-5 | notify target=123456789 channe... | 0 | — | 09:50:00 |
  | sys-memory-consolidate | 0 0 3 * * * | memory_consolidate | 12 | 03:00:00 | 03:00:00 |
  | sys-trace-prune | 0 30 2 * * * | trace_prune | 12 | 02:30:00 | 02:30:00 |
  ```

  A job that runs a SOP shows no command. The command is cut at 30 characters,
  and a `⚠` after the run count marks a job whose last run failed.

================================================================================
# FILE: https://goclaw.ulab.mn/docs/user-guide/automation/gitlab-webhook.md
================================================================================

# GitLab Webhook Integration

Go-Claw agents can automatically process and respond to GitLab events such as push, pipeline, and merge request triggers.

There are three ways to connect a project:

- **Ask the agent** (`gitlab_hook`): an admin describes it in a conversation, and the agent
  creates the project webhook in GitLab and the route in GoClaw, each event running a SOP. See
  [Asking the agent for a hook](#asking-the-agent-for-a-hook).
- **The `goclaw gitlab hook setup` wizard**: five questions in a terminal configure both sides.
- **By hand**: see [Alternative Manual Setup](#alternative-manual-setup).

---

## Table of Contents

1. [Asking the agent for a hook](#asking-the-agent-for-a-hook)
2. [Prerequisites](#prerequisites)
3. [1-Minute Connection (CLI Wizard)](#1-minute-connection-cli-wizard)
4. [Command Reference](#command-reference)
5. [How It Works](#how-it-works)
6. [Events](#events)
7. [Usage Examples](#usage-examples)
8. [Security](#security)
9. [Troubleshooting](#troubleshooting)
10. [Alternative Manual Setup](#alternative-manual-setup)

---

## Asking the agent for a hook

An admin asks in a conversation, and the agent's `gitlab_hook` tool does the rest: it creates a
webhook in the GitLab project for the events you name, and the route `/goclaw/hook/gitlab/`
in GoClaw. Each of those events then runs a SOP with the event as its input. Removing the hook
removes both.

```mermaid
flowchart LR
    GL["GitLab Project Webhook
Only the events asked for
X-Gitlab-Token = route secret"] -->|"merge request / pipeline / push …"| GW["/goclaw/hook/gitlab/<key>"]
    GW --> SOP["SOP Run on the Agent"]
```

### Before you start

| What | Where |
| :--- | :--- |
| The GitLab instance, with `allow_write: true` | `tools.gitlab` (or `tools.gitlabs.`): the **Integrations** page or `config.yaml` |
| A token with the **Maintainer** (or Owner) role on the project and the `api` scope | the instance's `token` |
| An address GitLab can reach the gateway at | `gateway.ingress.url`, and a listener there |
| A SOP for the events to run | the SOPs page, or the agent's `sop_save` tool |

The GitLab host needs no entry on `tools.http.allow`: the GitLab tools may reach the instance you
configured.

**GitLab and local addresses.** GitLab refuses a webhook to a loopback or private address
(`10.x`, `192.168.x`, a name without dots) unless its administrator allows it: **Admin › Settings ›
Network › Outbound requests › Allow requests to the local network from webhooks and
integrations**, or the host on the allowlist below it. The plan says so when the gateway's
address is such an address; GitLab's refusal is passed on with the same advice.

### What you can say

```
Make merge requests in group/backend run the mr-review SOP.
Pipelines on group/backend should run pipeline-triage. Call the hook "Backend pipelines".
Run release-notes on pushes to release/* in group/app.
```

The agent answers with a plan, and nothing is created until you confirm it in your next message:

```
Plan 3f9a1c20 — nothing is created yet.

GitLab "default" (version 17.5.1-ee), project group/backend, gets a webhook "Backend MR review":
- on merge_request events
- GitLab calls https://bot.example.com/goclaw/hook/gitlab/backend-mr-review, with a new secret only this route knows (X-Gitlab-Token)
GoClaw: each event runs SOP "mr-review" (MR review) as agent main.
```

Events: `merge_request`, `pipeline`, `push`, `tag_push`, `issue`, `note`, `job`, `deployment`,
`release`, `wiki`. `branch_filter` (a wildcard such as `main` or `release/*`) narrows push events.
Confidential issues and comments are never sent.

The plan is refused, with the reason, when the project does not exist for the token, the
token's role is below Maintainer, the project already has a webhook to that route, the route key
is taken, or the instance does not allow writes.

### Listing, testing, removing

- **list**: every hook this gateway registered, the project, its events, whether GitLab still has
  it and whether GitLab has disabled it after failed deliveries, and how many events arrived.
- **test** (GitLab 16.11 or later): GitLab sends a test delivery. GitLab's test is a real-looking
  event, so it is sent for an event the hook does not run on: the route counts it and runs
  nothing. On older GitLab, use **Test** in the project's **Settings › Webhooks**.
- **delete**: planned and confirmed like a create; removes the GitLab webhook and the route.
  Deleting the route on the **Webhooks** page removes the GitLab webhook too.

Only a human admin can do this, in a conversation of their own: never a SOP, cron, webhook or
subagent run, whose inputs are text anyone can write. The `gitlab` tool itself only reads hooks
(`hooks_list`, `hook_get`).

---

## Prerequisites

For the wizard to work, the following **3 items** must be configured in `config.yaml`:

```yaml
# 1. Gateway public HTTPS URL — destination address for GitLab webhooks
gateway:
  ingress:
    type: "proxy"                   # or "nginx" / "ngrok"
    url: "https://bot.example.com"

# 2. GitLab host + PAT (api scope) — used by the wizard to call the GitLab API
tools:
  gitlab:
    url: "https://gitlab.com"       # or your self-hosted instance
    token: "${secret:GITLAB_PAT}"
    allow_write: true               # only for the agent's own writes (comment_add, …); the wizard does not need it

# 3. Default agent (router/main) — receives events
```

Store the PAT in the vault:

```bash
$ goclaw secret set GITLAB_PAT glpat-xxxxxxxxxxxxxxxxxxxx
```

---

## 1-Minute Connection (CLI Wizard)

```bash
$ goclaw gitlab hook setup
```

The wizard will ask 5 questions and perform the **following actions automatically**:

- ✅ Generates a random 16-byte secret (no human input required)
- ✅ Registers the webhook via the GitLab REST API
- ✅ Registers the inbound route in the Go-Claw database
- ✅ Stores the secret in the encrypted vault
- ✅ Automatically rolls back changes if any step fails

### Wizard Flow

```
  ╔════════════════════════════════════════════╗
  ║   GoClaw GitLab Webhook Setup Wizard       ║
  ╚════════════════════════════════════════════╝

  ── Step 1: Gateway Public URL ──
  Gateway public URL [https://bot.example.com]: ↵

  ── Step 2: GitLab Instance ──
  GitLab host [https://gitlab.com]: ↵
  ✓ Using PAT from tools.gitlab.token
  ✓ Connected to GitLab 16.9

  ── Step 3: Project ──
  Project path (e.g. group/repo): group/payments-api
  ✓ Project found: group/payments-api (ID: 42)

  ── Step 4: Events ──
  Select events (space-separated numbers, or 'all'):
    [1] push          [2] tag_push     [3] merge_request
    [4] pipeline      [5] issue        [6] note
    [7] wiki          [8] deployment   [9] release
  Choice [1 3 4]: ↵
  ✓ Selected: push, merge_request, pipeline

  ── Step 5: Target Agent ──
  Which agent should receive events? [main]: ↵

  ── Step 6: Creating Webhook ──
  ⚙ Webhook URL: https://bot.example.com/goclaw/hook/gitlab/group-payments-api
  ⚙ Secret: glsec_a7b3…  (auto-generated)
  ✓ Webhook created in GitLab (hook_id=123)
  ✓ Route registered in GoClaw (db_id=5)
  ✓ Secret stored in vault as GITLAB_WH_GROUP_PAYMENTS_API

  ✅ Setup complete!
```

**Notes:**

- Pressing Enter accepts the default value in brackets.
- `[1 3 4]` corresponds to the numeric options `push, merge_request, pipeline`. You can type `all` to select all events.
- The secret is **not shown in plain text**. `goclaw secret get GITLAB_WH_` prints it later (here `goclaw secret get GITLAB_WH_GROUP_PAYMENTS_API`); `goclaw secret list` shows it masked.

---

## Command Reference

```bash
# Register a new webhook (wizard)
goclaw gitlab hook setup

# List all GitLab webhooks in a table
goclaw gitlab hook list

# Send a test event via GitLab API
goclaw gitlab hook test                   # default: push_events
goclaw gitlab hook test  --trigger issues_events

# Remove
goclaw gitlab hook remove                 # deletes from both sides
goclaw gitlab hook remove  --keep-route   # deletes only from GitLab
goclaw gitlab hook remove  --keep-gitlab  # deletes only from Go-Claw DB
```

### `list` Output Example

```
KEY                 AGENT   EVENTS                       ENABLED  HITS  LAST HIT              ERROR
group-payments-api  main    push,merge_request,pipeline  yes      42    2026-04-15 14:23 UTC  —
backend-api         devops  pipeline                     yes      8     2026-04-15 09:10 UTC  —
frontend-web        main    push                         no       0     —                     —
```

---

## How It Works

```mermaid
flowchart TD
    GL["GitLab Project"] -->|POST| EP["/goclaw/hook/gitlab/<key>
GoClaw gateway"]
    EP --> RT["Webhook Router
1. Verify X-Gitlab-Token
2. Check if event matches filters
3. Add summary to agent session
4. Return 200 OK once the event is taken"]
    RT --> BG["Agent Processes in Background"]
    BG --> SESS["Saved in Session
webhook:<agent>:gitlab:<object>"]
```

The events of one object share a session: `webhook::gitlab:-`
(for example `merge_request-1234`, with the object's ID, not its `!42` number), or
`proj-` for an event about no single object, such as a push. A comment belongs
to what it hangs off: a note on an issue is in `issue-`, a note on a merge request
in `merge_request-`, and a work item's events are in `issue-` too, so an issue's edits
and its comments share one session. A route that runs a SOP logs what started its runs in
`webhook-sop::gitlab::` instead, and the runs of one object take turns
(see [Runs started by a webhook](sop.md#runs-started-by-a-webhook)).

### What the agent is told

An agent driven by a webhook sees a summary, not the raw payload. When the route
is bound to a SOP, each step of its run is shown the JSON as data from outside —
in a section marked untrusted, never as the SOP's own context — up to 16 KB of
each event and 48 KB in all. The summary names the object, so a skill does not
have to go looking for it:

```
🔔 Webhook received: type=gitlab key=group-payments-api event=merge_request
Project: group/payments-api
MR: !42
Title: Add rate limiting to the refresh endpoint
Action: update
State: opened
Draft: yes
Branches: feat/rate-limit → main
SHA: a1b2c3d4
URL: https://gitlab.example.com/group/payments-api/-/merge_requests/42
Labels: Status::Review required, Type::Feature
```

| Event | What the summary carries |
| :--- | :--- |
| `merge_request` | MR iid, title, action, state, draft flag, source → target branch, last commit SHA, URL, labels |
| `issue` | issue iid, title, action, state, labels |
| `note` | which MR or issue the comment hangs off, and its author |
| `pipeline` | pipeline id, status, ref, SHA, and the MR iid when the pipeline belongs to one |
| `push` / `tag_push` | ref, checkout SHA, commit count, who pushed |

The SHA is shortened to 8 characters — the form used in comment markers and in
`git log --oneline`. An event whose `object_kind` is not modelled falls back to
printing `ref` and `message`.

**Route Lookup Precedence:**

1. A route for the exact key comes before a catch-all (a route of the type without a key).
2. Among those, a stored route (registered via the Web UI or the CLI wizard) comes before a
   `config.yaml` `webhooks:` route.

A paused route keeps its deliveries: they get 404, not the catch-all. Every key a route has
(`sop_id`, `batch_mode`, `token`, …) is in [Webhook Routes](../configuration.md#webhook-routes).

**What the sender is told:**

| Status | When |
| :--- | :--- |
| `200` | The event was taken: a SOP run started (or queued, for a bulk route), or the agent will answer it. `{"status":"filtered"}` when the route's event filter leaves it out, `{"status":"duplicate"}` when the route took this delivery before. |
| `401` | The token or signature does not match the route's secret. |
| `403` | The sender's address is not in `gateway.ingress.allow_ips` (see [Security](#security)). |
| `404` | No route for the type and key, or the route is paused. |
| `409` | The route's SOP cannot run: it was deleted or disabled, or it is overdue for review. On a route without a SOP, the route's agent is not running on this gateway: it is not in `agents.list`, was added since the gateway started, or its model failed to start. |
| `500` | The SOP run could not be started; send it again. |
| `413` | The payload is larger than 4 MB. |
| `429` | With `gateway.rate_limit` on, the sender's address made too many requests in a minute (`Retry-After: 60`). |
| `503` | Too many deliveries are in progress (`Retry-After: 30`), another request is taking this same delivery right now, or the delivery's ID could not be recorded (both `Retry-After: 10`). |

**A delivery sent again runs once.** For a week, the route remembers the ID the sender gave
each delivery it took:
- GitLab's `Idempotency-Key` (GitLab 17.4 and later; it stays the same when GitLab retries),
  else its `X-Gitlab-Event-UUID`;
- GitHub's `X-GitHub-Delivery`;
- a Graylog event's ID;
- an `Idempotency-Key` header from any other sender.

A delivery with an ID the route took before is answered `{"status":"duplicate"}` and runs
nothing. A delivery the route did not take (401, 409, 500, 503) runs when it is sent again (a
503 for a delivery another request was taking is a repeat or runs, as that request ended), and
so does one it took and then dropped without running it, as the route shows on the Webhooks
page (no agent came free in time, a bulk batch whose SOP refused to run). On a route to every
agent, a chat delivery runs again only if no agent ran it, and a bulk one does not.

On a route that runs a SOP, a restart does not change that. A delivery counts as taken only once
what it started is on record: its SOP run, or its events on a bulk route's queue. One the gateway
was taking when it stopped is settled when it starts again: a resend runs, unless its SOP run had
begun a step (or its events are queued); on a route for every agent, the resend runs only the
agents whose run had not begun. A bulk batch is marked with its run before the run starts and
stays on the queue until the run begins a step, so a batch whose run had begun is not run a
second time after a restart, and one whose run had not — still waiting for a free SOP slot when
the gateway stopped, or until the wait timed out — runs. A single delivery whose SOP run was
still waiting for a slot was answered `200`, so its sender does not send it again, and it is
lost. A route without a SOP hands the delivery to the agent's turn, which is not on record:
a turn still waiting or running when the gateway stops is lost.

Every refusal a route answers, and an event dropped after it was taken (no worker came free in
time), is shown next to the route on the Webhooks page, except the 503 for a delivery another
request is taking. A 403, 413 or 429 is answered before any route is looked up, a 404 for an
unknown key has no route to show it on, and a route in `config.yaml` records nothing there:
look for those in the gateway log. Deleting a route set up by `goclaw gitlab hook` there
also removes its webhook in the GitLab project, or says why it could not.

---

## Events

CLI event name → GitLab API field → Incoming `X-Gitlab-Event` header. A route's event filter
can use any of the three spellings: they are compared as the CLI name, so `merge request`,
`Merge Request Hook` and `merge_requests_events` all mean `merge_request`.

| CLI Name | GitLab API Field | X-Gitlab-Event Header |
|----------|------------------|-----------------------|
| `push` | `push_events` | `Push Hook` |
| `tag_push` | `tag_push_events` | `Tag Push Hook` |
| `merge_request` | `merge_requests_events` | `Merge Request Hook` |
| `pipeline` | `pipeline_events` | `Pipeline Hook` |
| `issue` | `issues_events` | `Issue Hook` |
| `note` | `note_events` | `Note Hook` |
| `wiki` | `wiki_page_events` | `Wiki Page Hook` |
| `deployment` | `deployment_events` | `Deployment Hook` |
| `release` | `releases_events` | `Release Hook` |

---

## Usage Examples

### 1. Notify on Pipeline Failure

`goclaw gitlab hook setup` → select only the `pipeline` event. Add the following to the agent's system prompt:

> "When a pipeline event is received from the webhook: if `status=failed`, immediately raise an alert, include the assignee's name, and summarize the failed job's logs."

### 2. Code Review on Push Events

`push` + `merge_request` events. The agent, with the `gitlab` tool:
- Reads diffs with the `mr_changes` action
- Submits feedback as comments with the `comment_add` action (needs `allow_write: true` on the GitLab instance)

### 3. Multiple Projects — Single Agent

Run `setup` once for each project. They will be registered in the database as follows:

```
KEY                AGENT    EVENTS
backend-api        devops   pipeline
frontend-web       devops   pipeline
infra-terraform    devops   merge_request,pipeline
```

All events route to the same agent. If you set `agent=*`, they will be broadcast to all agents.

---

## Security

**1. Random secret + vault.** The wizard generates a 16-byte `glsec_*` secret and stores it in the encrypted vault. No plain-text secrets are stored in the configuration file.

**2. HTTPS enforced.** The wizard rejects any URL that does not start with `https://`. By default, GitLab also rejects plain HTTP webhooks.

**3. Event filtering.** By excluding unnecessary events from the database `events` field, you:
- Save tokens
- Prevent agent sessions from cluttering
- Focus strictly on relevant events

**4. GitLab IP allowlist (optional).** Restrict requests by adding GitLab.com's IP range to `gateway.ingress.allow_ips`:

```yaml
gateway:
  ingress:
    allow_ips:
      - "34.74.90.64/28"   # GitLab.com SaaS
      - "10.0.0.0/8"       # On-prem
```

`allow_ips` replaces `allow_preset`: with a list here, a preset such as `microsoft-teams` no
longer applies, so add the addresses of every other sender that reaches the gateway this way.
Requests from the gateway's own machine (loopback) are always let through.

**5. Secret rotation (every 3-6 months):**

```bash
$ goclaw gitlab hook remove group-payments-api   # clean up the old one
$ goclaw gitlab hook setup                        # recreate with a new secret
```

---

## Troubleshooting

Start where GitLab shows the answer: **Settings → Webhooks → Recent events**, then the
request's **Response body**. GoClaw says there why it refused a delivery, and records the
same reason on the route: the **Health** column of **Automation → Webhooks** shows it.

### 401 Unauthorized

The response names the cause:

| Response says | Cause | Fix |
|---|---|---|
| `the X-Gitlab-Token header is missing` | The GitLab webhook has no **Secret token**. | Edit the webhook in GitLab and enter the route's secret as its Secret token. |
| `X-Gitlab-Token does not match this route's secret` | GitLab sends a different secret. | Enter the route's secret in GitLab, or set the route's secret to what GitLab has. |
| `this route has no secret` | The route was saved without one. | Edit the route and enter a secret (or create it again: an empty secret is generated). |
| `…a ${secret:…} reference with no value in the vault` | The route's secret points at a vault key that does not exist. | Add the key with `goclaw secret set`, or enter the secret itself. |

One route answers one secret. A route with no **key** answers every
`/goclaw/hook/gitlab/` URL, so every project that uses it must send that same
secret. To give projects their own secrets, create a route per project with its key.

### 404 Not Found

| Response says | Cause | Fix |
|---|---|---|
| `no active webhook route for gitlab/` | A route matches, but it is paused. | Enable it on the Webhooks page (▶). |
| `no webhook route for gitlab/` | No route has that key. | Check the key in the URL; the gateway log lists the keys that exist. |
| An HTML page, or nothing from GoClaw | The request reached the Web UI port, not the webhook listener. | When `gateway.ingress.port` is set, webhooks are served only on that port. The Webhooks page shows both ports. |

`webhook_secret_slug` only applies to channel webhooks (`/hook//msteams`), not to
`/goclaw/hook/…` routes.

### "no public URL available"

The wizard stops if `gateway.ingress.url` is missing. Add it to the config and try again.

### Hit count remains 0

- Is the route paused? The Webhooks page shows it, and a delivery to it is answered 404.
- Do the route's **Events** include what GitLab sends? A filtered event is answered
  `200 {"status":"filtered"}` and not counted.
- Check **Recent events** in GitLab for the status and response body.

### No events received from GitLab

1. GitLab Settings → Webhooks → Recent events.
2. The status of recent calls:
   - `200` ✓
   - `404` → see 404 above
   - `401` → see 401 above
   - `5xx` → check lines matching `[webhook]` in the GoClaw log

---

## Alternative Manual Setup

If you cannot run the CLI wizard (e.g., headless server without SSH access):

**A. Register the route on the Go-Claw side** — via the WebUI or in config.yaml:

WebUI → **Automation → Webhooks** → **+ New Webhook** (Type: gitlab, Key, Agent, Events). Leave
**Secret** empty to have one generated: after saving, the page shows the URL and the secret to
enter in GitLab, and the secret is not shown again. When you edit a route later, an empty
Secret field keeps the current one.

Or in `config.yaml`:

```yaml
webhooks:
  - type: gitlab
    key: myproject
    secret: "${secret:GITLAB_WEBHOOK_MYPROJECT}"
    agent: main
    events: [push, pipeline, merge_request]
```

The gateway reads the change when it reloads `config.yaml`; no restart is needed. Save the secret using the `goclaw secret set` command.

Every route key — `sop_id` to run a SOP per event, `batch_mode: bulk` and `batch_window` to run one
for a window of events, `token` — is described in [Webhook Routes](../configuration.md#webhook-routes).
Routes for other senders work the same way; for Graylog alerts, see
[Graylog Alerts That Run a SOP](graylog-alerts.md).

**B. Register the webhook on the GitLab side** — from the GitLab UI:

GitLab Project → **Settings → Webhooks**:

| Field | Value |
|--------|------|
| URL | `https://bot.example.com/goclaw/hook/gitlab/myproject` |
| Secret Token | (Must **exactly match** the secret configured on the Go-Claw side) |
| Trigger | Push / Merge / Pipeline (select what you need) |
| Enable SSL verification | ✅ |

> ⚠️ The secret must be **exactly the same** on both sides. A mismatch of even a single character will cause a 401 response for incoming calls from GitLab. The CLI wizard handles this alignment automatically.

---

## Related Documents

- [Configuration](../configuration.md) — all config.yaml options; [Webhook Routes](../configuration.md#webhook-routes) for the `webhooks:` keys
- [Graylog Alerts That Run a SOP](graylog-alerts.md) — the same receiver, called by Graylog alerts
- [Gateway Endpoints](../../api-reference/gateway_endpoints.md#webhook-routes-api) — `/api/webhooks` and the receiver's status codes
- [Ingress](../ingress.md) — Public URL, proxy, and ngrok configuration
- [Tools & MCP](../tools_and_mcp.md) — `gitlab` tool actions
- [SOP Automation](sop.md) — Map events to multi-step workflows

================================================================================
# FILE: https://goclaw.ulab.mn/docs/user-guide/automation/graylog-alerts.md
================================================================================

# Graylog Alerts That Run a SOP

Graylog does the watching: every minute or so it counts the messages that match a query over a
time window. When the count crosses a threshold, Graylog calls the gateway, and the gateway runs
a SOP with the alert and its matching messages as input.

```mermaid
flowchart LR
    DEF["Graylog Event Definition
query, window, threshold"] -->|alert| NOTIF["HTTP Notification"]
    NOTIF -->|POST| GW["/goclaw/hook/graylog/<key>"]
    GW --> SOP["SOP Run on the Agent"]
```

There are two ways to set it up:

- **Ask the agent.** An admin describes the alert in a conversation. The agent's `graylog_alert`
  tool creates the event definition and the notification in Graylog and the webhook route in
  GoClaw, and removes all three together. See [Asking for an alert](#asking-for-an-alert).
- **Connect it yourself.** You make the route on the **Webhooks** page (or in `config.yaml`) and
  the HTTP Notification in Graylog. GoClaw then only receives alerts. See
  [Connecting Graylog without the agent](#connecting-graylog-without-the-agent).

Either way, Graylog must be able to reach the gateway, and the alert needs a SOP to run.

---

## Before you start

| What | Where |
| :--- | :--- |
| Graylog 5.0 or later | Older versions describe alerts differently; the tool refuses them. |
| The Graylog instance, with `allow_write: true` (only when the agent sets alerts up) | `tools.observability.graylog.`: the **Integrations** page, `config.yaml`, or `goclaw config set` |
| An address Graylog can reach the gateway at | `gateway.ingress.url`, e.g. `http://192.0.2.10:8080`, and a listener there: `gateway.bind` or `gateway.ingress.port`/`bind` |
| Graylog let in, if the gateway restricts callers | `gateway.ingress.allow_ips` |
| A SOP for the alerts to run | the SOPs page, or the agent's `sop_save` tool ([Standard Operating Procedures](sop.md)) |

```yaml
gateway:
  bind: lan                           # listen on the network, not only on 127.0.0.1
  ingress:
    url: "http://192.0.2.10:8080"     # as Graylog sees this gateway

tools:
  observability:
    graylog:
      main:
        url: "http://graylog.internal:9000/api"
        token: "${secret:GRAYLOG_TOKEN}"
        allow_write: true             # off by default: the agent cannot create or remove alerts
```

### The Graylog instance

`allow_write` is off by default. The agent can then still search Graylog, and list and test the
alerts it registered, but it cannot create or remove one. Turn it on in one of three ways:

- on the **Integrations** page, edit the Graylog instance and tick **Allow write actions (create
  and remove alerts)**; the instance then shows the *write-enabled* tag;
- in `config.yaml`, as in the example above;
- `goclaw config set tools.observability.graylog.main.allow_write=true`.

The agent's next request uses it; no restart is needed.

The instance is read from the tools of the agent the alerts run as (by default, the agent you are
talking to). An agent with its own `tools.observability` block in its `agents.list` entry uses
that block instead of the global one, as a whole. The Integrations page edits the global block, so
for such an agent put the instance, with `allow_write`, into its own block. With more than one
instance, name the one you mean in your request ("on Graylog main").

The Graylog token needs these rights:

- read streams;
- create, read and delete event definitions;
- create, edit and delete notifications.

If the gateway's address is not yet on Graylog's URL allowlist (System › Configurations), the
tool adds it. That needs the right to change Graylog's configuration. Without it, a Graylog
admin adds the address and you ask the agent again.

### Letting the gateway reach Graylog

The agent calls Graylog's API through the gateway's outbound HTTP policy. The host of the Graylog
instance's own `url` needs no entry on `tools.http.allow`, private address included: the Graylog
tools may reach the instance you configured. Only `tools.http.deny` stops them (the global one,
or the agent's own in its `agents.list` entry), with `domain "graylog.internal" is blocked by
policy` or `address … is blocked by policy`. If `graylog_query` works for the agent, so does
this.

### Letting Graylog reach the gateway

`gateway.ingress.url` only tells Graylog where to call: the agent builds the alert's URL from it.
The gateway must also accept the call at that address.

- **The gateway listens on 127.0.0.1 by default** (`gateway.bind: loopback`), which a Graylog
  server on another host cannot reach. Set `gateway.bind: lan`, or put a reverse proxy on the
  gateway's host that forwards to it and use the proxy's address as `gateway.ingress.url`.
- **With `gateway.ingress.port` set**, webhooks are served only on that port, by a listener of
  their own bound by `gateway.ingress.bind` (127.0.0.1 when empty). Put that port in the URL and
  set `gateway.ingress.bind: lan`. A call to the Web UI port then gets `404 page not found`. This
  is also the way to let Graylog in without putting the Web UI on the network:

  ```yaml
  gateway:
    bind: loopback                  # the Web UI and API stay on 127.0.0.1
    ingress:
      url: "http://192.0.2.10:8081"
      port: "8081"                  # webhooks only
      bind: lan
  ```

- **If `gateway.ingress.allow_ips` or `allow_preset` is set**, any caller outside the list gets
  `403 Forbidden`. Add the address of every Graylog node that sends notifications to
  `allow_ips`. When `allow_ips` is set, it replaces the preset rather than adding to it: on a
  gateway with `allow_preset: microsoft-teams`, adding `allow_ips` drops the Teams ranges. Either
  list those ranges in `allow_ips` too, or have Graylog come in through a reverse proxy on the
  gateway's host: callers on 127.0.0.1 are always let in while `gateway.ingress.trust_proxy` is
  off.

Changes to `gateway.bind` and to the ingress port, bind and allow lists take effect after a
restart. The **Webhook URL Reference** on the Webhooks page shows which port webhooks are served
on. See [Public Ingress](../ingress.md) for proxies and ngrok.

---

## Asking for an alert

Talk to the agent as an admin, and say "Graylog alert":

> Create a Graylog alert that runs the incident-triage SOP when service payments logs more than
> 50 errors in 5 minutes, per host, in the Payments stream.

**Name the feature.** The agent is offered the tools that match what you write, and "Graylog
alert" is the surest way to get it `graylog_alert`. If admins manage alerts often, add the tool to
`tools.core` in `config.yaml`: the tools listed there are offered on every request.

```yaml
tools:
  core: [graylog_alert]
```

The agent checks everything and answers with a plan. Nothing is created yet:

```
Plan 3f9a1c2e — nothing is created yet.

Graylog "main" (version 6.1) gets an alert "Payments errors", priority normal:
- every 1 min, count the messages matching `service:payments AND level:ERROR` in streams
  Payments over the last 5 min, counted per host
- alert when the count > 50; the same alert is not repeated within 10 min
- each alert carries up to 5 matching messages
- Graylog calls http://192.0.2.10:8080/goclaw/hook/graylog/payments-errors, with a new
  secret only this route knows
GoClaw: each alert runs SOP "incident-triage" (Incident triage) as agent main.
```

Answer in your **next message**, for example "yes, create it". The agent then creates it in
Graylog and in GoClaw. It then has Graylog send its test notification, and says whether the test
reached the gateway. A plan cannot be carried out in the same turn it was made, nor from another
conversation. It expires after 30 minutes.

### What you can say

The agent turns your words into these settings; you can name any of them in the request.

| Setting | Default | |
| :--- | :--- | :--- |
| `title` | (required) | what the alert is, e.g. "Payments errors"; it also gives the alert's key (below) |
| `query` | (required) | the Graylog search the alert counts, e.g. `service:payments AND level:ERROR` |
| `threshold` | (required) | alert when the count crosses it |
| `comparator` | `>` | `>`, `>=`, `<`, `<=` or `==` against the threshold |
| `streams` | all streams | stream titles or IDs |
| `group_by` | none | fields counted separately, e.g. `host`: one alert per host that crosses the threshold |
| `window_minutes` | 5 | count over the last N minutes, 1 to 1440 |
| `every_minutes` | 1 | how often Graylog checks, 1 to the window |
| `grace_minutes` | 10 | the same alert, for the same group-by values, is not repeated sooner; 0 to 1440 |
| `backlog` | 5 | matching messages sent with each alert, 0 to 20 |
| `priority` | 2 | 1 low, 2 normal, 3 high |
| `sop_id` | (required) | the SOP each alert runs |
| `agent_id` | the agent you are talking to | the agent the SOP runs as, whose tools the Graylog instance is read from |
| `instance` | the only one | the Graylog instance, needed when there are several |
| `description` | empty | the event definition's description in Graylog |

**The key.** The title names the alert, and its key is the title in lower case with every run of
characters other than Latin letters and digits turned into one dash, at most 40 characters:
"Payments errors" gives `payments-errors`. The key is the end of the route's URL
(`/goclaw/hook/graylog/payments-errors`), and you name the alert by it to test or remove it. A
title with no Latin letters or digits, such as one written only in Cyrillic, gives no key and is
refused: add a Latin word or a number. A title whose key a route already has is refused too: an
alert's, one made on the Webhooks page, or one in `config.yaml`.

---

## What the SOP is given

Each alert starts one SOP run, as the agent the alert names, under the `webhook-system`
identity. Graylog's call is answered as soon as the run has been created, not when it ends. The
run's trigger shows as `webhook:graylog` in the SOP's History.

Every step of the run sees the alert as **data from outside**: in a section marked untrusted,
never as part of the SOP's own instructions. That section shows at most 16 KB per alert. It holds
the alert's title and message, its group-by values, and the matching messages Graylog sent with
it. A log line is text anyone who can make the service log can write: the SOP should treat it as
evidence, not as instructions.

Alerts of one event definition and one group-by value share a log session in GoClaw
(`webhook-sop::graylog::alert--`), which keeps its last 50
alerts.

Graylog's test notification only shows that the alert is connected. The route answers it and
counts it, and no SOP runs. An alert Graylog sends again (a retry, say) is known by its event's
ID for a week: the repeat is answered and runs nothing. That holds across a restart too: an alert
the gateway was taking when it stopped runs when Graylog sends it again, unless its SOP run had
begun a step.

The SOP must be able to run when the alert comes. A SOP that has been deleted or disabled, or
whose review date has passed, refuses every alert with `409`. Disabling the SOP (pausing it on the
SOPs page, or `sop_enable` with `enabled: false`) stops its alerts for a while; enable it again to
resume. A SOP overdue for review runs again once someone reviews it (ask the agent to review the
SOP; its `sop_review` tool sets the next review date). See
[What Graylog is told](#what-graylog-is-told).

---

## Listing, testing, removing

| Ask the agent to… | It… | Needs `allow_write` |
| :--- | :--- | :--- |
| list the Graylog alerts | shows each alert's key, title and query, its Graylog instance and state there, its SOP and agent, whether the route is enabled or paused, how many calls from Graylog the route took, and the last problem, if any | no |
| test alert `` | has Graylog send the test notification again, and says whether it arrived within 10 seconds | no |
| remove alert `` | answers with a plan of what it removes; after you confirm in your next message, it deletes the event definition and the notification in Graylog and the route in GoClaw | yes, both when it plans and when it removes |

The count in the list is the calls the route took, test notifications included: a new alert whose
creation test arrived shows 1. Duplicate and refused calls are not counted.

Without `allow_write`, removing is refused: the agent says to turn it on, or to delete the route
on the Webhooks page and the event definition and notification in Graylog by hand. If the Graylog
instance is no longer configured for the alert's agent at all, removing goes ahead: the route is
removed, and the agent says which event definition and notification to delete in Graylog.

These actions see only the alerts the agent registered. A route you made yourself is not listed,
tested or removed by them.

### The alert's route on the Webhooks page

The alert's route appears on **Automation › Webhooks** as `graylog/`.

- **Delete** removes the Graylog side as well, or says what is left for a Graylog admin to delete.
  Without `allow_write` it changes nothing in Graylog.
- **Pause** stops the alerts: while the route is paused, Graylog's calls are answered `404` and
  run nothing, and the route's **Health** says a delivery arrived while it was paused. Graylog
  keeps alerting; whether it sends a refused call again is up to Graylog.
- **Edit**: do not change the type, the key or the secret. Graylog's notification keeps the
  address and secret it was created with, so after a new key every alert is answered `404` (or
  goes to a Graylog route without a key, if there is one), and after a new secret `401`. A
  changed agent runs the alerts as that agent, and list, test and remove then look for the
  Graylog instance in that agent's tools. A changed SOP applies to the next alert. To change what
  the alert counts, remove it and ask for a new one.

---

## Connecting Graylog without the agent

You can make the route and the Graylog notification yourself: for a Graylog instance without
`allow_write`, for an event definition that already exists, or because you would rather set it up
in Graylog. GoClaw then only receives alerts. It creates and removes nothing in Graylog,
`graylog_alert` does not list, test or remove the route, and deleting the route leaves Graylog as
it is. GoClaw makes no calls to Graylog here, so `tools.observability` is not needed; Graylog
must still reach the gateway (see
[Letting Graylog reach the gateway](#letting-graylog-reach-the-gateway)).

### 1. The route

**On the Webhooks page** (Automation › Webhooks › **New Webhook**):

- **Type**: Graylog.
- **Key**: a name for the alert, e.g. `checkout-errors`. It ends the URL.
- **Agent**: the agent the SOP runs as, e.g. `main`.
- **Secret**: leave it empty to have one generated.
- **Events**: leave empty: Graylog sends only alerts. (Its test notification is answered whatever
  the filter says.)
- **SOP ID**: the SOP each alert runs.

After you save, the page shows **Set up in Graylog** with the URL, the API Key and the API Secret
to enter in Graylog. A generated secret is shown only then: copy it before you close the window.
Check that Graylog can reach that URL: the page builds it from the ngrok tunnel's address, else
from `gateway.ingress.url`, and when neither is set from the address you opened the page at and
the port webhooks are served on (the page then says so).

**Or in `config.yaml`.** Store the secret in the vault first, so the route finds it:

```bash
goclaw secret gen GRAYLOG_HOOK_CHECKOUT     # prints the value; goclaw secret get shows it again
```

```yaml
webhooks:
  - type: graylog
    key: checkout-errors            # URL: /goclaw/hook/graylog/checkout-errors
    secret: "${secret:GRAYLOG_HOOK_CHECKOUT}"
    agent: main                     # an agent ID, or "*" for every agent
    sop_id: incident-triage         # the SOP each alert runs
```

Put the shared value in `secret`, not `token`: `token` is compared only with a `?token=`
parameter, and Graylog 5.1 and later sends the value as a header. A route in `config.yaml` applies
to the next call without a restart. It is listed on the Webhooks page, but it records no hits and
no refusals there: the gateway log has them. The keys are explained under
[Webhook Routes](../configuration.md#webhook-routes).

`batch_mode: bulk` with `batch_window` (seconds, default 300) runs the SOP once for all the alerts
of a window instead of once per alert. Those runs share one log session for the route, and the
per-alert log sessions described under [What the SOP is given](#what-the-sop-is-given) do not
apply.

**Without `sop_id`**, each alert is a chat turn: the agent is given the alert's summary (the
matching messages are counted, not included), and its reply is saved in the session
`webhook::graylog:alert--` and sent nowhere. Give the route a SOP when
the alert should lead to something.

**Use type Graylog, not Custom.** A Custom route accepts Graylog's calls the same way, but loses
what type Graylog does:

| | Graylog | Custom |
| :--- | :--- | :--- |
| Graylog's test notification | answered `{"status":"test"}`, counted, nothing runs | runs like an alert |
| An alert Graylog sends again | known by its event ID, runs once | runs again |
| What the session log and a chat turn read | a summary: alert, description, priority, message, group-by values, time, number of matching messages | up to five top-level text fields of the payload |
| Sessions | one per alert and group-by value | SOP runs: one for the whole route; chat turns: a new one per call |

### 2. The notification in Graylog

In Graylog, open **Alerts › Notifications**, create a notification and choose the type **HTTP
Notification**:

| Field | Value |
| :--- | :--- |
| URL | `/goclaw/hook/graylog/`. With `gateway.ingress.port` set, the URL uses that port. |
| API Key | `X-Token` |
| API Secret | the route's secret |
| The option that sends the API key as a header | on (Graylog 5.1 and later) |
| Basic authentication | leave empty: GoClaw does not read it |

Graylog 5.0 has no header option. Enter `token` as the API Key instead: Graylog then adds
`?token=` to the URL, which the route accepts too.

If Graylog refuses the URL, add it to Graylog's URL allowlist (System › Configurations › URL
Allowlist). Run the test (step 3), save the notification, and add it to the event definition that
should call it (**Alerts › Event Definitions**, its Notifications step). The definition's message
backlog decides how many matching messages each alert carries.

### 3. Test it

Use **Execute Test Notification** in Graylog. The route answers `200 {"status":"test"}` and runs
nothing. On a route made on the Webhooks page, **Hits** goes up by one. A route in `config.yaml`
counts nothing there; the gateway log shows `Graylog test notification reached route
graylog/`.

A `401` says why in its body:

- `no credential sent — send this route's secret as X-Token, ?token=, or an X-Signature HMAC`:
  the API Key is not `X-Token` (or `token` on 5.0); or, on 5.1 and later, it is `X-Token` but the
  option that sends it as a header is off, so Graylog adds `?X-Token=` to the URL, which the route
  does not read; or only Basic authentication is filled in.
- `the X-Token / ?token= value does not match this route's secret`: the API Secret is not the
  route's secret.
- `this route has no secret, …` or `this route's secret is a ${secret:…} reference with no value
  in the vault`: the route itself has no usable secret.

For any other answer, see the next section.

---

## What Graylog is told

A call to the route's URL gets one of these answers. Graylog reports a failed call in its server
log, and **Execute Test Notification** shows the answer directly. On the GoClaw side, a route
made on the Webhooks page or by the agent keeps the reason for its last refusal in its **Health**
column (and in the agent's list of alerts, as the last problem), for example
`2026-10-07 14:03 409: SOP "incident-triage" is overdue for review. Please run sop_review.`
Answers given before the call reaches a route are only in the gateway log, and so is everything
for a route kept in `config.yaml`.

| Answer | Why | Where it shows |
| :--- | :--- | :--- |
| `200 {"status":"ok"}` | Taken: the SOP run was created (queued, for a bulk route), or the chat turn admitted. | **Hits** goes up |
| `301` or `405` | The URL ends with `/` (the key is missing: the page's **Base** button copies the URL without one), or the call is not a POST. | not counted |
| `200 {"status":"test"}` | Graylog's test notification. Nothing runs. | **Hits** goes up |
| `200 {"status":"duplicate"}` | The route took this event (same event ID) before, within the last week. Nothing runs. | not counted |
| `200 {"status":"filtered"}` | The route's **Events** leave this call out. Leave Events empty on a Graylog route. | not counted |
| `401` | The secret is missing or wrong, or the route has none. The body says which (see [Test it](#3-test-it)). After a route's secret was changed on the Webhooks page, Graylog still sends the old one. | Health |
| `403 Forbidden` | The caller's address is not in `gateway.ingress.allow_ips` or `allow_preset`. | gateway log only (`[ingress] denied`) |
| `404` | `no webhook route for graylog/`: a wrong key, or a key changed on the Webhooks page. A plain `404 page not found` is the Web UI port, when `gateway.ingress.port` is set. | gateway log, with the keys that exist |
| `404` | `no active webhook route for graylog/`: the route is paused. | Health |
| `409` | The route's SOP was deleted or disabled, or is overdue for review. Nothing runs until the SOP exists, is enabled and is reviewed. On a route without a SOP, the route's agent is not running on this gateway: restart the gateway after adding the agent to `config.yaml`, or look in its log for `[multi-agent] Failed to create agent`. | Health |
| `413` | The body is larger than 4 MB. Lower the backlog. | gateway log only |
| `429` | Over the gateway's rate limit: `gateway.rate_limit.per_minute` (default 60) calls a minute from one address. `Retry-After: 60`. | gateway log only |
| `500` | The SOP could not be read, or its run could not be started. | Health |
| `503` | Too many in progress: 256 webhook-started SOP runs already waiting or running (for a route without a SOP, 256 chat turns). `Retry-After: 30`. Or another request is taking this same alert right now (`Retry-After: 10`), which a resend settles. Or the alert's event ID could not be recorded (`Retry-After: 10`), so it was not run. | Health (too many in progress, not recorded) |

Whether Graylog sends a refused call again is up to Graylog. A refused call is not remembered, so
it runs if it comes again.

**If the agent's test fails**, look at what it says. Graylog's test is the same call as **Execute
Test Notification**:

- *Graylog's test notification failed*: the call did not get a `2xx`, and the error carries
  Graylog's own message with the answer it got. Find that answer in the table above. A refused
  connection or a timeout is the address or port
  ([Letting Graylog reach the gateway](#letting-graylog-reach-the-gateway)); a URL Graylog will
  not call is its URL allowlist.
- *Graylog sent its test notification and got an answer, but this route counted nothing*:
  something else at that address answered — another gateway or server, or a proxy that answers by
  itself. Check that `gateway.ingress.url` reaches this gateway.

---

## Who can do this

Only a human admin, in a conversation of their own, can ask the agent to create, list, test or
remove alerts. Creating and removing also need `allow_write` on the Graylog instance; listing and
testing do not. The tool is refused in SOP steps, scheduled jobs, webhook-started runs and
subagents.

These rules exist because the agent reads log lines: in search results, and in the alerts
themselves. A crafted log line could otherwise lead an agent to create, change or remove the
automation that reads it. The confirmation in a later message keeps a person's decision between
what the agent proposes and what changes.

---

## Related

- [Standard Operating Procedures](sop.md): writing the SOP an alert runs, and
  [runs started by a webhook](sop.md#runs-started-by-a-webhook)
- [Public Ingress](../ingress.md): `gateway.ingress`, reverse proxies, the caller allow list
- [Configuration › Webhook Routes](../configuration.md#webhook-routes): the `webhooks:` keys
- [SRE Tools › Graylog](../observability/sre-tools.md#graylog): the Graylog instance and
  `graylog_query`, the agent's log search
- [GitLab Webhook Integration](gitlab-webhook.md): the same webhook receiver, for GitLab events

================================================================================
# FILE: https://goclaw.ulab.mn/docs/user-guide/automation/sonarqube-hooks.md
================================================================================

# SonarQube Quality Gates That Run a SOP

SonarQube calls a webhook after each analysis, with the project, the branch or pull request, and
the quality gate's verdict and conditions. Pointed at the gateway, that call runs a SOP: when the
quality gate fails (the default), or after every analysis.

```mermaid
flowchart LR
    CI["CI Runs an Analysis"] --> WH["SonarQube Webhook
project or global"]
    WH -->|"POST (signed)"| GW["/goclaw/hook/sonarqube/<key>"]
    GW --> SOP["SOP Run on the Agent"]
```

There are two ways to set it up:

- **Ask the agent.** An admin describes it in a conversation. The agent's `sonarqube_hook` tool
  creates the webhook in SonarQube and the route in GoClaw, and removes both together. See
  [Asking for a hook](#asking-for-a-hook).
- **Connect it yourself.** Make a `sonarqube` route on the **Webhooks** page, and a webhook in
  SonarQube with the URL and secret the page shows. See
  [Connecting SonarQube without the agent](#connecting-sonarqube-without-the-agent).

---

## Before you start

| What | Where |
| :--- | :--- |
| The SonarQube instance, with `allow_write: true` (only when the agent sets hooks up) | `tools.sonarqube` (or `tools.sonarqubes.`): the **Integrations** page or `config.yaml` |
| A **user** token with **Administer** on the project (global **Administer** for a hook on every project) | the instance's `token`; analysis tokens cannot manage webhooks |
| An address SonarQube can reach the gateway at | `gateway.ingress.url`, and a listener there |
| A SOP for the deliveries to run | the SOPs page, or the agent's `sop_save` tool ([Standard Operating Procedures](sop.md)) |

```yaml
gateway:
  bind: lan
  ingress:
    url: "http://192.0.2.10:8080"     # as SonarQube sees this gateway

tools:
  sonarqube:
    url: "https://sonar.example.com"
    token: "${secret:SONARQUBE_TOKEN}"
    allow_write: true                 # off by default: the agent cannot create or remove hooks
```

The SonarQube host needs no entry on `tools.http.allow`: the SonarQube tools may reach the
instance you configured.

**SonarQube and local addresses.** SonarQube refuses a webhook to its own host and to loopback,
and SonarQube Community Build 26.10 and later refuse private addresses (`10.x`, `192.168.x`) as
well, unless its admin unticks **Enable local webhooks validation**
(**Administration › Configuration › General Settings › Security**, `sonar.validateWebhooks`). The
plan says so when the gateway's address is such an address, and SonarQube's refusal is passed on
with the same advice.

---

## Asking for a hook

```
When the quality gate of backend fails, run the quality-triage SOP.
Run release-check after every analysis of backend. Call it "Backend analyses".
Run gate-report whenever any project's quality gate fails.
```

The agent answers with a plan; nothing is created until you confirm it in your next message:

```
Plan 7c41a9d2 — nothing is created yet.

SonarQube "default" (version 10.6.0.92116), project backend (Backend), gets a webhook "GoClaw: Backend quality gate":
- after every analysis SonarQube calls; the SOP runs when the quality gate fails
- SonarQube calls http://192.0.2.10:8080/goclaw/hook/sonarqube/backend-quality-gate, signing each delivery with a new secret only this route knows
- SonarQube has no test call: the hook is first seen working after the next analysis
GoClaw: each delivery runs SOP "quality-triage" (Quality triage) as agent main.
```

`on: any` runs the SOP after every analysis; `all_projects` makes a global webhook, called for
every project, in place of one project's.

The plan is refused, with the reason, when the project does not exist for the token, the token
lacks Administer, SonarQube already has a webhook to that route or already has its 10 webhooks
there, the route key is taken, the instance has no token, or it does not allow writes.

---

## What the SOP is given

Each delivery is the run's trigger data, in an untrusted section of every step. The agent is told:

```
Project: Backend (backend)
Branch: main (main branch)
Quality gate: Sonar way — ERROR
  ✗ new_coverage: 42.1 (must be at least 80)
Analysed: 2026-10-09T10:00:00+0800
Revision: c739069e
Dashboard: https://sonar.example.com/dashboard?id=backend
```

Only the conditions that failed are listed. A pull request's analysis says `Pull request:` in
place of `Branch:`. One project's branch (or pull request) keeps one session.

The route's events are `failed`, `passed`, `warning` (older SonarQube) and `analysis` (no
quality gate). A hook made for failures filters on `failed`; one made for every analysis has no
filter.

A delivery SonarQube sends again — the same analysis task — is answered and not run again. A
quality gate flipped later by an issue change is its own delivery.

---

## Listing, checking, removing

- **list**: every hook this gateway registered, its project, whether it runs on failures or
  every analysis, and SonarQube's own record of its last delivery.
- **test**: SonarQube has no test call. This reports SonarQube's last delivery (when, and what
  the gateway answered) beside the deliveries the route counted. Before the first analysis it
  says there has been none.
- **delete**: planned and confirmed like a create; removes the SonarQube webhook and the route.
  Deleting the route on the **Webhooks** page removes the SonarQube webhook too, when the
  instance allows writes; otherwise it says which webhook to delete in SonarQube.

Only a human admin can do this, in a conversation of their own: never a SOP, cron, webhook or
subagent run.

---

## Connecting SonarQube without the agent

1. On the **Webhooks** page, create a route of type **SonarQube** with a key (for example the
   project's), the agent, and the SOP. Set **Events** to `failed` to run only on failed quality
   gates. Saving shows the URL and the secret once.
2. In SonarQube, open the project's **Project Settings › Webhooks › Create** (or
   **Administration › Configuration › Webhooks** for every project). **URL**: the route's URL.
   **Secret**: the route's secret (SonarQube 10.6 and later want at least 16 characters).
3. Run an analysis. The route's **Hits** go up, and SonarQube's webhook page shows the delivery.

SonarQube signs each delivery with the secret (`X-Sonar-Webhook-HMAC-SHA256`); a delivery whose
signature does not match is refused with 401, and the reason is shown on the route.

## Related

- [Graylog Alerts That Run a SOP](graylog-alerts.md) — the same pattern for Graylog alerts
- [GitLab Webhook Integration](gitlab-webhook.md) — the same for GitLab project events
- [Standard Operating Procedures](sop.md)

================================================================================
# FILE: https://goclaw.ulab.mn/docs/user-guide/automation/grafana-alert-rules.md
================================================================================

# Grafana Alert Rules That Run a SOP

Grafana evaluates a query on a schedule, and when the value crosses a threshold for long enough
the alert fires. With a webhook contact point that calls the gateway, a firing alert runs a SOP,
with the alert's labels, values and annotations as its input.

```mermaid
flowchart LR
    EVAL["Grafana Evaluates the Rule
every 60 s, for 5 min"] -->|fires| CP["Rule Contact Point
webhook"]
    CP -->|"POST (Bearer)"| GW["/goclaw/hook/grafana/<key>"]
    GW --> SOP["SOP Run on the Agent"]
```

There are two ways to set it up:

- **Ask the agent.** An admin describes it in a conversation. The agent's `grafana_alert_rule`
  tool makes the alert rule, its own contact point and the route in GoClaw, and removes them
  together. See [Asking for an alert rule](#asking-for-an-alert-rule).
- **Connect it yourself.** Make a `grafana` route on the **Webhooks** page, and a webhook contact
  point in Grafana with the URL and secret the page shows; then send an alert rule to it. See
  [Connecting Grafana without the agent](#connecting-grafana-without-the-agent).

---

## Before you start

| What | Where |
| :--- | :--- |
| Grafana 11.0 or later, with `allow_write: true` on the instance (only when the agent makes rules) | `tools.observability.grafana.`: the **Integrations** page or `config.yaml` |
| A service account token with the **Editor** role (some older Grafana versions keep the alerting provisioning API from Editors; then **Admin**) | the instance's `token` |
| A Prometheus or Loki datasource for the rule's query | Grafana's **Connections › Data sources** |
| An address Grafana can reach the gateway at | `gateway.ingress.url`, and a listener there |
| A SOP for the alerts to run | the SOPs page, or the agent's `sop_save` tool ([Standard Operating Procedures](sop.md)) |

```yaml
gateway:
  bind: lan
  ingress:
    url: "http://192.0.2.10:8080"     # as Grafana sees this gateway

tools:
  observability:
    grafana:
      prod:
        url: "https://grafana.example.com"
        token: "${secret:GRAFANA_TOKEN}"
        allow_write: true             # off by default: the agent cannot make or remove rules
```

The Grafana host needs no entry on `tools.http.allow`: the Grafana tools may reach the instance
you configured.

---

## Asking for an alert rule

```
When the API's 5xx rate stays above 5 per second, run the api-errors SOP.
  (sum(rate(http_requests_total{code=~"5.."}[5m])) on Prometheus)
Alert when free disk on the reports host drops below 10 GB for 15 minutes, and run disk-cleanup — also when it recovers.
```

The agent answers with a plan; nothing is created until you confirm it in your next message:

```
Plan 3e9b07c1 — nothing is created yet.

Grafana "prod" (version 12.2.1) gets an alert rule "API 5xx rate":
- every 60s, evaluate `sum(rate(http_requests_total{code=~"5.."}[5m]))` on prometheus datasource Prometheus
- fire when the value > 5 for 5 min; no data counts as OK, a failed evaluation fires a DatasourceError alert
- in folder "GoClaw" (made now), group "GoClaw api-5xx-rate", with its own contact point "GoClaw api-5xx-rate"
- Grafana calls http://192.0.2.10:8080/goclaw/hook/grafana/api-5xx-rate, with a new secret only this route knows; notification policies and other rules are not changed
GoClaw: when the rule fires, SOP "api-errors" (API errors) runs as agent main.
```

What the request can set: the query and its datasource (name or UID), the threshold and
whether the value must be above (`>`, the default) or below (`<`) it, how long it must hold
(`for_minutes`, default 5), how often Grafana evaluates it (`every_seconds`, default 60), and
whether the SOP also runs when the alert resolves (`on_resolved`).

The rule is a new one, in a folder named **GoClaw**, and reaches its own contact point directly
(the rule's **Configure notifications**, Grafana 11 and later). Notification policies, contact
points and every other rule stay as they were. The folder is made when it is missing and left in
place when a rule is removed.

The plan is refused, with the reason, when Grafana is older than 11.0, the datasource is not a
Prometheus or Loki one the token can see, Grafana already has the rule or a contact point of the
same name, the route key is taken, or the instance does not allow writes. A rule on another kind
of query (SQL, Elasticsearch, a classic condition) is made in Grafana and sent to a contact point
by hand, as [below](#connecting-grafana-without-the-agent).

After creating it, the agent has Grafana send a test notification through the new contact point
and says whether it reached the route.

---

## What the SOP is given

Each notification is the run's trigger data, in an untrusted section of every step. The agent is
told:

```
Alert: API 5xx rate — FIRING
Folder: GoClaw
- FIRING {service=api-gateway} values A=7.25, B=1
  Summary: API 5xx rate
  Since: 2026-10-09T02:00:00Z
  Rule: https://grafana.example.com/alerting/grafana/goclaw-api-5xx-rate/view
```

Up to five alerts of the notification are listed, each with its labels (those every alert of the
rule carries are left out) and its values. One rule keeps one session.

The route's events are `firing` and `resolved`. A rule made to run on firing alerts filters on
`firing`; one made with `on_resolved` lets both through.

Grafana repeats a firing alert's notification every repeat interval (4 hours by default). The
repeat is the same delivery and is not run again; the alert firing again after it resolved is a
new one.

---

## Listing, checking, removing

- **list**: every rule this gateway registered, its query, whether it is paused or gone in
  Grafana, and how many alerts the route received.
- **test**: Grafana sends a test notification through the rule's contact point. The route
  answers it, counts it and runs nothing, so the SOP does not run; the agent says whether it
  arrived.
- **delete**: planned and confirmed like a create; removes the rule, its contact point and the
  route. Deleting the route on the **Webhooks** page removes the rule and the contact point too,
  when the instance allows writes; otherwise it says what to delete in Grafana.

Only a human admin can do this, in a conversation of their own: never a SOP, cron, webhook or
subagent run.

---

## Connecting Grafana without the agent

1. On the **Webhooks** page, create a route of type **Grafana** with a key (for example the
   rule's name), the agent and the SOP. Set **Events** to `firing` to run only when the alert
   fires. Saving shows the URL and the secret once.
2. In Grafana, open **Alerting › Contact points › Add contact point**, choose the integration
   **Webhook**, and set **URL** to the route's URL. Under **Optional Webhook settings**, set
   **Authorization Header - Scheme** to `Bearer` and **Authorization Header - Credentials** to the
   route's secret.
3. **Test**: the route answers Grafana's test notification and runs nothing, and its **Hits** go
   up. Save the contact point.
4. Send the alert rule to it: in the rule's **Configure notifications**, choose the contact point,
   or add a notification policy that matches the rule's labels.

From Grafana 12 the contact point can sign each notification instead: put the secret under
**HMAC Signature › Secret** and leave the header as it is (`X-Grafana-Alerting-Signature`). A
notification whose token or signature does not match is refused with 401, and the reason is shown
on the route.

## Related

- [Graylog Alerts That Run a SOP](graylog-alerts.md) — the same pattern for Graylog alerts
- [SonarQube Quality Gates That Run a SOP](sonarqube-hooks.md) — the same for SonarQube analyses
- [SRE Tools](../observability/sre-tools.md#grafana) — the `grafana_alerts` tool and the instance's configuration
- [Standard Operating Procedures](sop.md)

================================================================================
# FILE: https://goclaw.ulab.mn/docs/user-guide/automation/zabbix-alerts.md
================================================================================

# Zabbix Problems That Run a SOP

Zabbix reports a problem when a trigger fires, and an action decides who is told. With a webhook
media type that calls the gateway, a problem on the hosts you choose runs a SOP, with the
problem's host, severity, operational data and tags as its input.

```mermaid
flowchart LR
    TRG["Trigger Fires
host groups, severity, tag"] --> ACT["Action 'GoClaw <key>'
sends a message"]
    ACT --> USR["User goclaw-<key>
its only media"]
    USR --> MED["Webhook Media Type
script"]
    MED -->|"POST (X-Token)"| GW["/goclaw/hook/zabbix/<key>"]
    GW --> SOP["SOP Run on the Agent"]
```

There are two ways to set it up:

- **Ask the agent.** An admin describes it in a conversation. The agent's `zabbix_alert` tool
  makes the Zabbix objects and the route in GoClaw, and removes them together. See
  [Asking for an alert](#asking-for-an-alert).
- **Connect it yourself.** Make a `zabbix` route on the **Webhooks** page, and in Zabbix a webhook
  media type, a user with that media and an action. See
  [Connecting Zabbix without the agent](#connecting-zabbix-without-the-agent).

---

## Before you start

| What | Where |
| :--- | :--- |
| Zabbix 5.0 or later, with `allow_write: true` on the instance (only when the agent makes alerts) | `tools.observability.zabbix.`: the **Integrations** page or `config.yaml` |
| A **Super admin** user's API token (Zabbix 5.4 and later), or its user and password: only a Super admin may make media types, users and user groups | the instance's `token` (or `user` and `password`) |
| An address the Zabbix **server** can reach the gateway at (the server runs the webhook, not the frontend) | `gateway.ingress.url`, and a listener there |
| A SOP for the problems to run | the SOPs page, or the agent's `sop_save` tool ([Standard Operating Procedures](sop.md)) |

```yaml
gateway:
  bind: lan
  ingress:
    url: "http://192.0.2.10:8080"     # as the Zabbix server sees this gateway

tools:
  observability:
    zabbix:
      prod:
        url: "https://zabbix.example.com"
        token: "${secret:ZABBIX_TOKEN}"
        allow_write: true             # off by default: the agent cannot make or remove alerts
```

---

## Asking for an alert

```
When a High or worse problem starts on the Linux servers host group, run the server-triage SOP.
Run db-check for Average and above problems on Databases tagged service=payments, and again when they resolve.
```

The agent answers with a plan; nothing is created until you confirm it in your next message:

```
Plan 5d20e8a4 — nothing is created yet.

Zabbix "prod" (version 7.0.12) gets, all named "GoClaw linux-servers-high":
- an action: problems on host group(s) Linux servers at severity high or above
- a webhook media type that calls http://192.0.2.10:8080/goclaw/hook/zabbix/linux-servers-high with a new secret only this route knows
- a user group with read access to those host groups and no frontend access, and a user goclaw-linux-servers-high in it whose only media is that media type
- no other user, group, media type or action is changed; Zabbix has no test call, so the first matching problem shows it working
GoClaw: each problem runs SOP "server-triage" (Server triage) as agent main.
```

What the request can set: the host groups (one or more), the lowest severity (`not classified`,
`information`, `warning`, `average`, `high` — the default — or `disaster`), an event tag and
optionally its value, and whether the SOP also runs when the problem resolves (`on_resolved`).

Why four objects: a Zabbix action sends messages to users, through their media, and only about
hosts the user may read. So the alert has a user of its own, in a group of its own that may read
the chosen host groups and cannot log in to the frontend, whose only media is the alert's webhook
media type. Existing users, groups, media types and actions stay as they were.

The plan is refused, with the reason, when a host group does not exist, Zabbix already has an
object of one of those names, the route key is taken, the instance does not allow writes, or
Zabbix is older than 5.0. A token that is not a Super admin's is refused by Zabbix when the
first object is made, and nothing is left behind.

The route's secret is in the media type's parameters, where Zabbix's Super admins can read it.

The media type's script works on every version from 5.0: it uses `HttpRequest` (5.4 and later)
or `CurlHttpRequest` (5.0 and 5.2), and each parameter holds a single macro, as Zabbix 7.4 and
later require.

---

## What the SOP is given

Each problem is the run's trigger data, in an untrusted section of every step. The agent is told:

```
Problem: High CPU on web-01 — PROBLEM
Severity: High
Host: web-01 (192.0.2.10)
Operational data: CPU 97 %
Tags: service:api-gateway, env:prod
Started: 2026.10.09 10:00:00
Event: 4711
Trigger URL: https://zabbix.example.com/tr_events.php?triggerid=1&eventid=4711
```

A recovery says `RESOLVED`, with the time it resolved and how long the problem lasted. A
problem and its recovery keep one session.

The route's events are `problem`, `recovery` and `update`. An alert made for problems filters on
`problem`; one made with `on_resolved` lets `recovery` through too.

Zabbix retries a message the gateway did not take (busy, unreachable); each retry carries the
same `Idempotency-Key`, so a problem is run once.

---

## Listing, checking, removing

- **list**: every alert this gateway registered, what it matches, whether its action is still
  there and enabled in Zabbix, Zabbix's last message for it, and how many problems the route
  received.
- **test**: Zabbix has no test call. This reports Zabbix's last message for the alert (sent,
  failed and why, or still being retried) beside what the route received. In Zabbix, the media
  type's **Test** button sends a delivery with no event; the route answers it, counts it and runs
  nothing.
- **delete**: planned and confirmed like a create; removes the action, the user, the user group,
  the media type and the route. Deleting the route on the **Webhooks** page removes the Zabbix
  objects too, when the instance allows writes; otherwise it says what to delete in Zabbix.

Only a human admin can do this, in a conversation of their own: never a SOP, cron, webhook or
subagent run.

---

## Connecting Zabbix without the agent

1. On the **Webhooks** page, create a route of type **Zabbix** with a key, the agent and the SOP.
   Set **Events** to `problem` to run only when a problem starts. Saving shows the URL, the
   secret, and the media type's script and parameters to paste.
2. In Zabbix, **Alerts › Media types › Create media type** (before 6.4: **Administration › Media
   types**): type **Webhook**, the script and parameters from the guide, with `goclaw_url` and
   `token` set to the route's URL and secret, and message templates for **Problem** and **Problem
   recovery** (any text: an action's default message needs them).
3. Give a user that media: a user of its own is best, in a group that may read the host groups.
4. Make a trigger action (**Alerts › Actions › Trigger actions**; before 6.4 **Configuration ›
   Actions**) whose operation sends a message to that user through that media type, and, for
   resolved problems, a recovery operation that does the same.
5. The media type's **Test** button: the route answers it without running the SOP, and its
   **Hits** go up.

A route made by hand with a script of your own still works: the SOP is told the `subject`,
`message`, `host` and `severity` fields it sends, and its `event_type` (else `alert`) is what the
route's **Events** filter on.

## Related

- [Graylog Alerts That Run a SOP](graylog-alerts.md) — the same pattern for Graylog alerts
- [Grafana Alert Rules That Run a SOP](grafana-alert-rules.md) — the same for Grafana
- [SRE Tools](../observability/sre-tools.md#zabbix) — the `zabbix_query` tool and the instance's configuration
- [Standard Operating Procedures](sop.md)

================================================================================
# FILE: https://goclaw.ulab.mn/docs/user-guide/automation/uptrace-monitors.md
================================================================================

# Uptrace Monitors That Run a SOP

An Uptrace metric monitor checks a query every minute, and opens an alert when the value stays
beyond its bounds. With a webhook notification channel that calls the gateway, an alert runs a
SOP, with the alert's name, status and attributes as its input.

```mermaid
flowchart LR
    MON["Uptrace Monitor 'GoClaw: <title>'
query, above/below, N minutes"] -->|"opens an alert"| CH["Webhook Channel 'GoClaw: <title>'
notifies only this monitor"]
    CH -->|POST| GW["/goclaw/hook/uptrace/<key>?token=…"]
    GW --> SOP["SOP Run on the Agent"]
```

There are two ways to set it up:

- **Ask the agent.** An admin describes it in a conversation. The agent's `uptrace_monitor` tool
  makes the monitor and its channel in Uptrace and the route in GoClaw, and removes them
  together. See [Asking for a monitor](#asking-for-a-monitor).
- **Connect it yourself.** Make an `uptrace` route on the **Webhooks** page, and in Uptrace a
  webhook channel with the URL the page shows, which a monitor notifies. See
  [Connecting Uptrace without the agent](#connecting-uptrace-without-the-agent).

---

## Before you start

| What | Where |
| :--- | :--- |
| The Uptrace instance with `url` and a **user** token, and `allow_write: true` (only when the agent makes monitors) | `tools.observability.uptrace.`: the **Integrations** page or `config.yaml` |
| The project the monitor belongs to | the instance's `project_id`, or named in the request |
| An address Uptrace can reach the gateway at | `gateway.ingress.url`, and a listener there |
| A SOP for the alerts to run | the SOPs page, or the agent's `sop_save` tool ([Standard Operating Procedures](sop.md)) |

```yaml
gateway:
  bind: lan
  ingress:
    url: "http://192.0.2.10:8080"     # as Uptrace sees this gateway

tools:
  observability:
    uptrace:
      main:
        url: "https://uptrace.example.com"
        token: "${secret:UPTRACE_USER_TOKEN}"   # a user's token, not a DSN's project token
        project_id: "1"
        allow_write: true                       # off by default: the agent cannot make or remove monitors
```

A DSN's project token sends data; it cannot manage monitors. The user token is in Uptrace's
user profile (2.x), or `auth.users[].auth_token` in Uptrace's own config (1.7).

**Uptrace versions.** Uptrace does not document the API its UI uses to manage monitors, and it
changed twice. The agent finds out which one the server speaks before it plans, and the plan
names it:

| Version | How sure | Notes |
| :--- | :--- | :--- |
| 1.7 | checked against Uptrace's source | queries write `per_min(…)`; saving a channel sends a test message |
| 2.0 | from a client tested against 2.0.2; not seen against a live server | no test call |
| 2.1 and later | from Uptrace's published OpenAPI | queries write `perMin(…)`; a **Test** call; webhooks to private addresses are refused unless the server's `outbound_http.ssrf_protection` allows them |

---

## Asking for a monitor

```
When the checkout service's error rate stays above 10 per minute for 5 minutes, run checkout-triage.
```

The agent looks the metric and query up with `uptrace_query` first, then answers with a plan;
nothing is created until you confirm it in your next message:

```
Plan 9a3c6e10 — nothing is created yet.

Uptrace "main" (API of version 1.7), project 1, gets a metric monitor "GoClaw: Checkout errors":
- metric uptrace_tracing_spans as $m, query `per_min(count($m{_status_code="error"})) as errors | where service_name = "checkout"`, column errors
- fires when errors is above 10 for 5 minute(s) in a row
- it notifies only its own webhook channel "GoClaw: Checkout errors", which calls http://192.0.2.10:8080/goclaw/hook/uptrace/checkout-errors with a new secret only this route knows (in the URL: Uptrace sends no header of its own)
- other monitors and channels are not changed
GoClaw: when an alert opens, SOP "checkout-triage" (Checkout triage) runs as agent main.
```

What the request can set: the metric and its alias in the query (`$m` by default), the query and
the column it checks, `above` and/or `below`, how many minutes in a row (`points`, default 5),
the project, and whether the SOP also runs when the alert resolves (`on_resolved`).

The plan is refused, with the reason, when the project does not exist for the token, the query
does not read the metric's alias or names no column, the bounds are missing or the wrong way
round, Uptrace already has a monitor or channel of that name, the route key is taken, the
instance has no user token or does not allow writes, or the server's version cannot be told.

After creating it, the agent has Uptrace send its test message where the version has one, and
says whether it reached the route.

---

## What the SOP is given

Each alert event is the run's trigger data, in an untrusted section of every step. The agent is
told:

```
Alert: GoClaw: Checkout errors — OPEN
Event: created
Type: metric
Since: 2026-10-09T01:55:00Z
Alert URL: https://uptrace.example.com/alerting/1/alerts/88
```

Uptrace 2.x adds the alert's attributes (`Attributes: service.name=checkout, …`). One alert keeps
one session.

The route's events are `alert` (it opened, or opened again), `resolved` (closed, resolved or
archived), `recurring` (still open; Uptrace repeats it), and Uptrace's own name for anything else.
A monitor made for alerts filters on `alert`; one made with `on_resolved` lets `resolved` through
too.

Uptrace retries a delivery the gateway did not take; each retry is the same alert event, which is
run once.

The route's secret is in the channel's URL, where anyone who can see the channel in Uptrace can
read it.

---

## Listing, checking, removing

- **list**: every monitor this gateway registered, its query, whether it is still in Uptrace, and
  how many alerts the route received.
- **test**: Uptrace sends its test message to the channel (2.1: the test call; 1.7: saving the
  channel again, which Uptrace refuses when the route does not answer). The route answers it and
  runs nothing. 2.0 has no test call.
- **delete**: planned and confirmed like a create; removes the monitor, then its channel, and the
  route. Deleting the route on the **Webhooks** page removes them too, when the instance allows
  writes; otherwise it says what to delete in Uptrace.

Only a human admin can do this, in a conversation of their own: never a SOP, cron, webhook or
subagent run.

---

## Connecting Uptrace without the agent

1. On the **Webhooks** page, create a route of type **Uptrace** with a key, the agent and the
   SOP. Set **Events** to `alert` to run only when an alert opens. Saving shows the URL with the
   secret in it, once.
2. In Uptrace, add a **Webhook** notification channel to the project with that URL. On 2.1 and
   later, untick **match all monitors** and choose the monitor, or every monitor's alerts reach
   the route.
3. In the monitor, notify that channel.
4. Test the channel (2.1: **Test**; 1.7: save it again): the route answers without running the
   SOP, and its **Hits** go up.

## Related

- [Grafana Alert Rules That Run a SOP](grafana-alert-rules.md) — the same pattern for Grafana
- [Zabbix Problems That Run a SOP](zabbix-alerts.md) — the same for Zabbix
- [SRE Tools](../observability/sre-tools.md#uptrace) — the `uptrace_query` tool and the instance's configuration
- [Standard Operating Procedures](sop.md)

================================================================================
# FILE: https://goclaw.ulab.mn/docs/user-guide/automation/wazuh-alerts.md
================================================================================

# Wazuh Alerts That Run a SOP

Wazuh writes every alert it raises to its indexer (OpenSearch), in `wazuh-alerts-4.x-*`. An
alerting monitor in the indexer can search those alerts on a schedule and, through a webhook
notification channel, call the gateway with what it found: each run that finds Wazuh alerts
runs a SOP, with their rule, agent and log as its input.

```mermaid
flowchart LR
    MGR["Wazuh Manager"] -->|alerts| IDX["Indexer
wazuh-alerts-4.x-*"]
    MON["Monitor 'GoClaw: <title>'"] -->|"every N min"| IDX
    MON -->|matches| CH["Channel goclaw-<key>"]
    CH -->|"POST (X-Token)"| GW["/goclaw/hook/wazuh/<key>"]
    GW --> SOP["SOP Run on the Agent"]
```

Nothing changes on the Wazuh manager: no `ossec.conf` integration, no restart.

There are two ways to set it up:

- **Ask the agent.** An admin describes it in a conversation. The agent's `wazuh_alert` tool
  makes the monitor and its channel in the indexer and the route in GoClaw, and removes them
  together. See [Asking for an alert hook](#asking-for-an-alert-hook).
- **Connect it yourself.** Make a `wazuh` route on the **Webhooks** page, and in the Wazuh
  dashboard a webhook channel and a monitor that notifies it. See
  [Connecting Wazuh without the agent](#connecting-wazuh-without-the-agent).

---

## Before you start

| What | Where |
| :--- | :--- |
| Wazuh 4.4 to 4.14 (indexer on OpenSearch 2.4 to 2.19) | 4.3's indexer has no notifications plugin; 5.0 keeps alerts in other indices, which the agent does not search yet |
| The indexer's URL, an indexer user and its password, and `allow_write: true` (only when the agent makes hooks) | `tools.observability.wazuh.`: the **Integrations** page or `config.yaml` |
| An address the indexer can reach the gateway at | `gateway.ingress.url`, and a listener there |
| A SOP for the runs to run | the SOPs page, or the agent's `sop_save` tool ([Standard Operating Procedures](sop.md)) |

```yaml
gateway:
  bind: lan
  ingress:
    url: "http://192.0.2.10:8080"         # as the Wazuh indexer sees this gateway

tools:
  observability:
    wazuh:
      siem:
        url: "https://wazuh.example.com:55000"          # the manager API (wazuh_query)
        user: "wazuh-api"
        password: "${secret:WAZUH_API_PASSWORD}"
        indexer_url: "https://wazuh.example.com:9200"   # the indexer, for wazuh_alert
        indexer_user: "admin"
        indexer_password: "${secret:WAZUH_INDEXER_PASSWORD}"
        insecure: true                                  # Wazuh's own certificates; applies to both
        allow_write: true                               # off by default: the agent cannot make or remove hooks
```

The indexer user needs the `alerting_full_access` and `notifications_full_access` roles and read
access to `wazuh-alerts-*`; Wazuh's `admin` indexer user has all of them. A monitor searches with
the roles of the user that made it.

---

## Asking for an alert hook

```
Run intrusion-triage on Wazuh alerts of level 10 and above from web-01 and db-01 in the authentication_failed group.
When rule 5712 fires anywhere, run ssh-brute-force-response. Check every 2 minutes.
```

The agent answers with a plan; nothing is created until you confirm it in your next message:

```
Plan 41d0c8e2 — nothing is created yet.

Wazuh "siem" indexer (OpenSearch 2.19.3) gets an alerting monitor "GoClaw: SSH brute force":
- every 5 minute(s) it searches wazuh-alerts-4.x-* for Wazuh alerts at rule level 10 or above, in group authentication_failed, from agent web-01 or db-01
- a run that finds some sends the first 20 to its own notification channel goclaw-ssh-brute-force, which calls http://192.0.2.10:8080/goclaw/hook/wazuh/ssh-brute-force with a new secret only this route knows
- the manager's configuration, other monitors and channels are not changed
GoClaw: each run with matches runs SOP "intrusion-triage" (Intrusion triage) as agent main.
```

What the request can set: the lowest rule level (`min_level`, default 10), rule IDs, rule
groups, agent names, and how often the monitor runs (`every_minutes`, default 5).

Each run searches the interval before it, ending a minute early so that an alert the indexer
takes a moment to write is found by the next run; runs do not overlap. A run that finds nothing
calls nothing. **One run with matches is one SOP run**, however many Wazuh alerts it found: a
burst of fifty failed logins is one run, not fifty.

The plan is refused, with the reason, when the indexer lacks the alerting or notifications
plugin (Wazuh 4.3), is a Wazuh 5 indexer, has no `wazuh-alerts-4.x-*` index, already has a
channel or monitor of that name, the route key is taken, the instance has no indexer settings,
or it does not allow writes.

After creating it, the agent has the indexer send the channel's test message and says whether
it reached the route.

---

## What the SOP is given

Each run is the trigger data, in an untrusted section of every step. The agent is told:

```
Monitor: GoClaw: SSH brute force — 34 Wazuh alert(s) between 2026-10-09T02:00:00Z and 2026-10-09T02:05:00Z
- level 10, rule 5712: sshd: brute force trying to get access to the system.
  Agent: web-01 (192.0.2.10)
  Time: 2026-10-09T02:01:00.000+0000
  Groups: syslog, sshd, authentication_failures
  Source IP: 203.0.113.7
  Location: /var/log/auth.log
  Log: Oct  9 02:01:00 web-01 sshd[811]: Failed password for root from 203.0.113.7 port 52211 ssh2
  Alert ID: 1760000000.123
…
```

Up to ten of the run's alerts are told in full, and the count of the rest. One monitor's runs
keep one session.

The indexer does not deliver a run again when the gateway does not answer: the next run sends
the alerts it finds then.

The route's secret is a header of the indexer's notification channel, which anyone who may read
the channel's settings in the indexer can see.

---

## Listing, checking, removing

- **list**: every hook this gateway registered, what it searches for, whether its monitor is
  still in the indexer and enabled, and how many runs the route received.
- **test**: the indexer sends the channel's test message (plain text). The route answers it,
  counts it and runs nothing.
- **delete**: planned and confirmed like a create; removes the monitor, then its channel, and
  the route. Deleting the route on the **Webhooks** page removes them too, when the instance
  allows writes; otherwise it says what to delete in the indexer.

Only a human admin can do this, in a conversation of their own: never a SOP, cron, webhook or
subagent run.

---

## Connecting Wazuh without the agent

1. On the **Webhooks** page, create a route of type **Wazuh** with a key, the agent and the SOP.
   Saving shows the URL, the secret, and the monitor action's message to paste, once.
2. In the Wazuh dashboard, **Explore › Notifications › Channels › Create channel**: type
   **Webhook**, the route's URL, method **POST**, and the headers `Content-Type:
   application/json` and `X-Token` with the route's secret. The indexer sends only the headers it
   is given. **Send test message**: the route answers it and its **Hits** go up.
3. **Explore › Alerting › Monitors › Create monitor**: a per-query monitor on
   `wazuh-alerts-4.x-*`, a trigger such as `ctx.results[0].hits.total.value > 0`, and an action
   that notifies the channel with the message from step 1.

## Related

- [Graylog Alerts That Run a SOP](graylog-alerts.md) — the same pattern for Graylog
- [Zabbix Problems That Run a SOP](zabbix-alerts.md) — the same for Zabbix
- [SRE Tools](../observability/sre-tools.md#wazuh) — the `wazuh_query` tool and the instance's configuration
- [Standard Operating Procedures](sop.md)

================================================================================
# FILE: https://goclaw.ulab.mn/docs/user-guide/observability/index.md
================================================================================

# Observability & Infrastructure Monitoring

GoClaw serves as a unified orchestration hub for infrastructure monitoring and observability. By connecting system agents to your SRE and DevOps monitoring platforms, agents can dynamically query metrics, logs, traces, security events, and hardware inventories to diagnose production incidents.

---

## 1. Core Principles

### Multi-Source Diagnosis
Instead of diagnosing incidents using a single metric, GoClaw agents can query multiple observability platforms in sequence to correlate events. For example, if a database response time spikes:
1. **Grafana Alerts:** Check active alerts on the system.
2. **Graylog Logs:** Search for database error traces.
3. **Zabbix Metrics:** Check host CPU, RAM, and Disk I/O utilization.
4. **Uptrace APM:** Trace request pathways to identify the bottleneck.
5. **Wazuh Security:** Verify if any unauthorized access or security events occurred.
6. **NetBox Asset Inventory:** Retrieve physical host locations, virtual machines, or rack configurations.

### Agent-Friendly Design
All integrated observability tools follow these key principles:
- **Named Instances:** Configure multiple profiles (e.g., `dev`, `staging`, `prod`) for the same tool.
- **Human-Readable Output:** Summarized findings instead of large, raw JSON payloads.
- **Implicit Defaults:** If only a single instance is configured, the agent queries it automatically without needing explicit instructions.
- **Read-Only by Default:** Write actions — a Grafana annotation or alert rule, a NetBox change, registering a Graylog, Zabbix or Wazuh alert or an Uptrace monitor — stay off until the instance sets `allow_write: true`. One exception: `uptrace_query` can always add an Uptrace annotation.
- **Secret Interpolation:** Authentication keys are kept out of plain-text config files using `${secret:KEY}` vault references.

---

## 2. Configuration Setup

All monitoring integrations are configured under the `tools.observability` key in `config.yaml`.

```yaml
tools:
  observability:
    # Grafana Dashboard & Alerts
    grafana:
      prod:
        url: "https://grafana.example.com"
        token: "${secret:GRAFANA_PROD_TOKEN}"
        description: "Production dashboards"

    # Centralized Log Analytics
    graylog:
      prod:
        url: "https://graylog.example.com"
        token: "${secret:GRAYLOG_PROD_TOKEN}"
        # allow_write: true   # lets graylog_alert create and remove alerts; off by default

    # Distributed Tracing (OTel)
    uptrace:
      main:
        dsn: "https://@api.uptrace.dev/"

    # Infrastructure Monitoring
    zabbix:
      prod:
        url: "https://zabbix.example.com/api_jsonrpc.php"
        token: "${secret:ZABBIX_PROD_TOKEN}"

    # SIEM & Security Events
    wazuh:
      siem:
        url: "https://wazuh.example.com:55000"
        user: "wazuh-api"
        password: "${secret:WAZUH_PASSWORD}"
        insecure: true

    # DCIM / IPAM Inventory
    netbox:
      prod:
        url: "https://netbox.example.com"
        token: "${secret:NETBOX_PROD_TOKEN}"
        allow_write: false
```

**Reaching each host.** The observability tools call their hosts through GoClaw's
[outbound HTTP policy](../tools_and_mcp.md#outbound-http), but the host of an instance's own
`url` (or of the Uptrace `dsn`) needs no entry on `tools.http.allow`, private address included:
you configured it, so each tool may reach it. That opens it to that tool only, not to
`http_request` or the browser. `tools.http.deny` still applies: a host on it is refused with
`domain "…" is blocked by policy` or `address … is blocked by policy`, and so is any other host
a call is sent on to, such as a redirect elsewhere, unless it is on `tools.http.allow`.

**Graylog alerts that run a SOP.** Graylog can also start work on its own: an alert in Graylog
calls the gateway's webhook (`/goclaw/hook/graylog/`), and the gateway runs a SOP with the
alert and its matching messages. A human admin can ask the agent to set one up (tool
`graylog_alert`, Graylog 5.0 or later, `allow_write: true` on the instance), or
[connect Graylog by hand](../automation/graylog-alerts.md#connecting-graylog-without-the-agent) on
the Webhooks page. See [Graylog Alerts That Run a SOP](../automation/graylog-alerts.md).

---

## 3. Incident Response Workflow

When tasked with investigating an issue (e.g., *"Our production API is responding slowly. Please check."*), the agent follows a standardized incident workflow:

```mermaid
flowchart LR
    S1["1. Grafana Alerts
List all active system alarms"] --> S2["2. Uptrace APM
Trace slow endpoints & topologies"]
    S2 --> S3["3. Graylog Logs
Locate stack traces & error logs"]
    S3 --> S4["4. Zabbix
Check host resource consumption"]
    S4 --> S5["5. Wazuh
Verify if security alerts exist"]
    S5 --> S6["6. NetBox
Confirm asset location & status"]
```

The agent correlates the collected data and provides a single, comprehensive incident summary with suggested remediation steps.

---

## 4. Offline Tool Testing (CLI)

Operators can verify monitoring connections directly from the CLI without running the main LLM loop. This is useful for testing API credentials and verifying connectivity:

```bash
# List all active Grafana alerts
goclaw tools exec grafana_alerts instance=prod action=alerts

# Search Graylog logs for system errors in the last 15 minutes
goclaw tools exec graylog_query instance=prod query='level:ERROR' range=900 limit=5

# Query Zabbix hosts directly via RPC methods
goclaw tools exec zabbix_query instance=prod method=host.get params='{"output":"extend"}'

# Search for active hardware devices in NetBox
goclaw tools exec netbox action=list resource_type=devices filters='{"status":"active"}'
```

================================================================================
# FILE: https://goclaw.ulab.mn/docs/user-guide/observability/sre-tools.md
================================================================================

# SRE Observability Tools

GoClaw includes SRE tools that call observability APIs directly through Go HTTP clients. They do not require external CLIs.

Configured credentials are read from `tools.observability` in `config.yaml`, or, for an agent with its own `tools.observability` in its `agents.list` entry, from that: each agent's calls go to its own instances. Agents should call these tools directly and should not search memory for tokens.

The tools call their hosts through the [outbound HTTP policy](../tools_and_mcp.md#outbound-http).
An instance's own host needs no entry on `tools.http.allow`, private address included; a host on
`tools.http.deny` (or the agent's own deny list) is refused with `… is blocked by policy`. See
[Configuration Setup](index.md#2-configuration-setup).

## Config Shape

```yaml
tools:
  observability:
    grafana:
      prod:
        url: "https://grafana.example.com"
        token: "${secret:GRAFANA_TOKEN}"
        # user: "viewer"                       # basic auth, used when no token is set
        # password: "${secret:GRAFANA_PASSWORD}"
        # allow_write: true   # lets action=annotate write annotations, and grafana_alert_rule make rules; off by default
        description: "Production dashboards"

    graylog:
      prod:
        url: "https://graylog.example.com/api"
        token: "${secret:GRAYLOG_TOKEN}"
        # insecure: true      # skip TLS verification for a self-signed Graylog
        # allow_write: true   # lets graylog_alert create and remove alerts; off by default

    uptrace:
      main:
        url: "https://api.uptrace.dev"
        token: "${secret:UPTRACE_TOKEN}"
        project_id: "1"       # default for project-scoped actions
        # allow_write: true   # lets uptrace_monitor make monitors; off by default (annotations need no flag)
        # or, instead of url + token + project_id:
        # dsn: "https://@api.uptrace.dev/"

    zabbix:
      prod:
        url: "https://zabbix.example.com/api_jsonrpc.php"
        token: "${secret:ZABBIX_TOKEN}"       # API token (Zabbix 5.4+)
        # user: "api-user"                    # older Zabbix: user.login, used when no token is set
        # password: "${secret:ZABBIX_PASSWORD}"
        # allow_write: true   # lets event.acknowledge update problems, and zabbix_alert make alerts; off by default

    wazuh:
      siem:
        url: "https://wazuh.example.com:55000"
        user: "wazuh-api"
        password: "${secret:WAZUH_PASSWORD}"
        insecure: false
        # indexer_url: "https://wazuh.example.com:9200"   # the indexer, for wazuh_alert
        # indexer_user: "admin"
        # indexer_password: "${secret:WAZUH_INDEXER_PASSWORD}"
        # allow_write: true   # lets wazuh_alert make alert hooks in the indexer; off by default
```

Use `instance=` when more than one instance is configured. If there is exactly one instance, most tools can infer it.

## Grafana

**Tool:** `grafana_alerts`

Despite the name, this tool supports several Grafana API actions.

| Action | Purpose |
|---|---|
| `alerts` | List active firing alerts |
| `dashboards` | Search/list dashboards |
| `annotate` | Create a graph annotation; requires `allow_write: true` |

```bash
goclaw tools exec grafana_alerts instance=prod action=alerts
goclaw tools exec grafana_alerts instance=prod action=dashboards
goclaw tools exec grafana_alerts instance=prod action=annotate text='Deployed v1.2.3' tags='["deploy","api"]'
```

To have an alert run a SOP, an admin asks the agent for an alert rule (`grafana_alert_rule`): see
[Grafana Alert Rules That Run a SOP](../automation/grafana-alert-rules.md).

## Graylog

**Tool:** `graylog_query`

The tool detects Graylog major versions and switches API strategy for v4, v5, v6, and v7.

| Action | Purpose |
|---|---|
| `search` | Search logs with a relative range in seconds |
| `search_absolute` | Search logs with explicit `from` and `to` timestamps |
| `terms_stats` | Count top values for a field |
| `streams_list` | List available streams |
| `version` | Show server version/system info |

```bash
goclaw tools exec graylog_query instance=prod action=search query='level:ERROR' range=3600 limit=50
goclaw tools exec graylog_query instance=prod action=search_absolute query='source:gateway' from=2026-06-20T00:00:00Z to=2026-06-20T12:00:00Z
goclaw tools exec graylog_query instance=prod action=terms_stats query='level:ERROR' field=source range=86400 limit=20
goclaw tools exec graylog_query instance=prod action=streams_list
```

Graylog query examples:

```text
level:ERROR
source:gateway AND level:WARN
(timeout OR connection) AND level:ERROR
message:*database*
response_time:[100 TO 500]
```

### Alerts that run a SOP

**Tool:** `graylog_alert`

A human admin can ask the agent, in a conversation, to register a Graylog alert: for example
*"run the incident-triage SOP when service payments logs more than 50 errors in 5 minutes"*. The
agent creates the event definition and the HTTP notification in Graylog and the webhook route in
GoClaw (`/goclaw/hook/graylog/`). Each alert then runs the SOP. The agent shows a plan
first and acts only when the admin confirms it in a later message.

It needs:

- Graylog 5.0 or later;
- `allow_write: true` on the Graylog instance (without it the agent only searches);
- `gateway.ingress.url`, the address Graylog reaches the gateway at.

The tool is refused in SOP steps, scheduled jobs, webhook-started runs and subagents. The same
route can also be [set up by hand](../automation/graylog-alerts.md#connecting-graylog-without-the-agent)
on the Webhooks page, which does not need `allow_write`. See
[Graylog Alerts That Run a SOP](../automation/graylog-alerts.md).

## Uptrace

**Tool:** `uptrace_query`

Start with `projects`, then pass `project_id` to project-scoped actions. An instance with a
`project_id` in its config (or in its `dsn`) uses it when the call gives none.

| Area | Actions |
|---|---|
| Discovery | `projects` |
| Monitoring | `monitors`, `monitor`, `notifications`, `alerts` |
| Tracing | `spans`, `query`, `timeseries` |
| Performance | `percentiles`, `topology` |
| Dashboards | `dashboards`, `dashboard`, `dashboard_yaml` |
| Attributes | `attributes`, `systems` |
| Metrics | `metrics_timeseries` |
| Write | `annotate` |

```bash
goclaw tools exec uptrace_query instance=main action=projects
goclaw tools exec uptrace_query instance=main action=monitors project_id=1
goclaw tools exec uptrace_query instance=main action=query project_id=1 query='perMin(count()) | where _status = "error" | group by service.name'
goclaw tools exec uptrace_query instance=main action=spans project_id=1 trace_id=abc123defabc
goclaw tools exec uptrace_query instance=main action=topology project_id=1
```

Time parameters are `time_gte` and `time_lte`. They accept RFC3339, `YYYY-MM-DD`, or relative values such as `now-1h`, `now-30m`, `now-2d`, and `now`.

To have an alert run a SOP, an admin asks the agent for a monitor (`uptrace_monitor`): see
[Uptrace Monitors That Run a SOP](../automation/uptrace-monitors.md).

## Zabbix

**Tool:** `zabbix_query`

This tool is a JSON-RPC client. Use the Zabbix API method name and pass `params` as a JSON object. It accepts the read methods (any `.get`, and `apiinfo.version`) and one write, `event.acknowledge`, which needs `allow_write: true` on the instance; any other method is refused.

It works with Zabbix 5.0 and later. It asks the API its version first (`apiinfo.version`, cached for 10 minutes) and sends the credentials the way that version takes them: the `Authorization: Bearer` header from 6.4, the request's `auth` field before. An API `token` needs Zabbix 5.4 or later; on 5.0 and 5.2 set `user` and `password`.

```bash
goclaw tools exec zabbix_query instance=prod method=host.get params='{"output":"extend"}'
goclaw tools exec zabbix_query instance=prod method=problem.get params='{"output":"extend","recent":true}'
goclaw tools exec zabbix_query instance=prod method=trigger.get params='{"output":"extend","filter":{"value":"1"}}'
goclaw tools exec zabbix_query instance=prod method=item.get params='{"hostids":"10084","output":"extend","search":{"key_":"cpu"}}'
```

Common methods:

| Method | Purpose |
|---|---|
| `host.get` | Host inventory and status |
| `problem.get` | Current problems |
| `trigger.get` | Trigger definitions and states |
| `item.get` | Metric items |
| `history.get` | Historical metric values |
| `event.get` | Historical events |
| `alert.get` | Recent alerts |
| `hostgroup.get` | Host groups |
| `event.acknowledge` | Acknowledge, comment on or close a problem; requires `allow_write: true` |

```bash
goclaw tools exec zabbix_query instance=prod method=event.acknowledge params='{"eventids":"12345","action":6,"message":"Investigating"}'
```

To have problems run a SOP, an admin asks the agent for an alert (`zabbix_alert`): see
[Zabbix Problems That Run a SOP](../automation/zabbix-alerts.md).

## Wazuh

**Tool:** `wazuh_query`

The current Wazuh tool accepts an API endpoint and handles JWT authentication internally. It is intentionally generic so callers can use any supported Wazuh endpoint.

```bash
goclaw tools exec wazuh_query instance=siem endpoint=/agents/summary/status
goclaw tools exec wazuh_query instance=siem endpoint=/agents
goclaw tools exec wazuh_query instance=siem endpoint='/syscheck/000?summary=yes'
goclaw tools exec wazuh_query instance=siem endpoint=/vulnerability/000
```

To have Wazuh alerts run a SOP, an admin asks the agent for an alert hook (`wazuh_alert`): see
[Wazuh Alerts That Run a SOP](../automation/wazuh-alerts.md).

## Benchmarking

Use `goclaw tools bench` to measure direct tool latency.

```bash
goclaw tools bench grafana_alerts instance=prod action=alerts --runs=10
goclaw tools bench graylog_query instance=prod action=search query='level:ERROR' --runs=5
goclaw tools bench zabbix_query instance=prod method=host.get params='{"output":"extend"}' --runs=10
```

## Implementation Files

- `internal/agent/tool_grafana.go`
- `internal/agent/tool_graylog.go`
- `internal/agent/tool_uptrace.go`
- `internal/agent/tool_zabbix.go`
- `internal/agent/tool_wazuh.go`
- `internal/config/config_types.go`

================================================================================
# FILE: https://goclaw.ulab.mn/docs/user-guide/observability/netbox.md
================================================================================

# NetBox IPAM And DCIM Integration

GoClaw integrates with NetBox through the `netbox` tool. It can read IPAM/DCIM inventory, allocate addresses, inspect rack layout, add journal entries, and create or update records when write access is enabled.

## Configuration

```yaml
tools:
  observability:
    netbox:
      prod:
        url: "https://netbox.example.com"
        token: "${secret:NETBOX_TOKEN}"
        allow_write: false
        description: "Production IPAM/DCIM"
      dev:
        url: "http://netbox-dev:8080"
        token: "${secret:NETBOX_DEV_TOKEN}"
        allow_write: true
        insecure: true
        description: "Development NetBox"
```

`allow_write: false` is the safe default. Keep production instances read-only unless the agent is explicitly allowed to modify NetBox.

The NetBox host does not need to be on `tools.http.allow`: like the other observability tools, `netbox` may reach its own `url`, while `tools.http.deny` still applies. `insecure: true` skips TLS certificate verification; use it only for a development instance.

## Actions

| Action | Type | Notes |
|---|---|---|
| `list` | Read | Search resources with `filters` |
| `get` | Read | Fetch one resource by `id` |
| `rack-elevation` | Read | Render rack elevation; requires `resource_type=racks` and `id` |
| `available-ips` | Read; Write with `data` | List free IPs in a prefix; requires `id`. With `data`, allocates one, which needs `allow_write: true` |
| `available-prefixes` | Read; Write with `data` | List available child prefixes; requires `id`. With `data`, allocates one, which needs `allow_write: true` |
| `available-vlans` | Read; Write with `data` | List available VLAN IDs in a VLAN group; requires `id`. With `data`, allocates one, which needs `allow_write: true` |
| `create` | Write | Requires `data` and `allow_write: true` |
| `bulk-create` | Write | Requires `bulk_data` and `allow_write: true` |
| `update` | Write | Full replacement; requires `id`, `data`, and `allow_write: true` |
| `patch` | Write | Partial update; requires `id`, `data`, and `allow_write: true` |
| `delete` | Write | Requires `id` and `allow_write: true` |
| `journal-add` | Write | Adds a journal entry; requires `data` and `allow_write: true` |
| `cable-connect` | Write | Creates a cable; requires `data` and `allow_write: true` |

## Resource Types

| Area | Resource types |
|---|---|
| DCIM | `sites`, `regions`, `locations`, `racks`, `rack-types`, `rack-roles`, `rack-reservations`, `devices`, `device-types`, `device-roles`, `platforms`, `manufacturers`, `interfaces`, `cables`, `power-panels`, `power-feeds`, `inventory-items`, `virtual-chassis`, `modules` |
| IPAM | `ip-addresses`, `prefixes`, `ip-ranges`, `vlans`, `vlan-groups`, `vrfs`, `aggregates`, `rirs`, `asns`, `asn-ranges`, `roles`, `services` |
| Virtualization | `virtual-machines`, `clusters`, `cluster-types`, `cluster-groups` |
| Extras | `tags`, `custom-fields`, `webhooks`, `journals` |

The tool also normalizes common aliases such as "server" to `devices` and "vm" to `virtual-machines`.

## Read Examples

```bash
goclaw tools exec netbox action=list resource_type=devices filters='{"status":"active","limit":50}'
goclaw tools exec netbox action=list resource_type=racks filters='{"site":"dc1"}'
goclaw tools exec netbox action=list resource_type=ip-addresses filters='{"address":"10.0.0.50/24"}'
goclaw tools exec netbox action=get resource_type=prefixes id=5
goclaw tools exec netbox action=available-ips resource_type=prefixes id=5
goclaw tools exec netbox action=available-prefixes resource_type=prefixes id=5
goclaw tools exec netbox action=available-vlans resource_type=vlan-groups id=2
goclaw tools exec netbox action=rack-elevation resource_type=racks id=3
```

## Write Examples

These require `allow_write: true` on the selected instance.

```bash
goclaw tools exec netbox action=create resource_type=ip-addresses data='{"address":"10.0.0.55/24","status":"active","description":"prod-web-04 mgmt"}'
goclaw tools exec netbox action=create resource_type=devices data='{"name":"prod-db-04","device_type":5,"device_role":3,"site":1,"status":"staged"}'
goclaw tools exec netbox action=patch resource_type=devices id=42 data='{"status":"offline"}'
goclaw tools exec netbox action=journal-add resource_type=journals data='{"assigned_object_type":"dcim.device","assigned_object_id":42,"comments":"Maintenance completed"}'
goclaw tools exec netbox action=cable-connect resource_type=cables data='{"a_terminations":[{"object_type":"dcim.interface","object_id":1}],"b_terminations":[{"object_type":"dcim.interface","object_id":2}]}'
```

## Filter Rules

Pass `filters` as an object, not as a JSON-encoded string when the LLM calls the tool. CLI examples quote JSON only because the shell needs it.

Useful filters:

| Resource | Filters |
|---|---|
| `devices` | `status`, `site`, `role`, `rack`, `q`, `tenant`, `tag`, `limit`, `offset` |
| `ip-addresses` | `address`, `status`, `device`, `tenant`, `tag`, `limit`, `offset` |
| `prefixes` | `within`, `site`, `vrf`, `tenant`, `tag`, `limit`, `offset` |
| `vlans` | `vid`, `site`, `tenant`, `tag`, `limit`, `offset` |
| `virtual-machines` | `cluster`, `status`, `q`, `tenant`, `tag`, `limit`, `offset` |

## Foreign Key Rules

NetBox foreign keys such as `tenant`, `site`, `rack`, `device_type`, and `device_role` usually require integer IDs in write payloads, not display labels.

```text
Wrong: {"site":"Production"}
Right: {"site":1}
```

Look up IDs first:

```bash
goclaw tools exec netbox action=list resource_type=sites filters='{"q":"Production"}'
goclaw tools exec netbox action=list resource_type=device-types filters='{"q":"PowerEdge R650"}'
```

## Server Registration Workflow

1. Find the site, rack, device type, and role IDs.
2. Find an available management IP.
3. Create the device.
4. Create the IP address.
5. Attach the IP to an interface if needed.
6. Add a journal entry for traceability.

Example:

```bash
goclaw tools exec netbox action=list resource_type=racks filters='{"site":"dc1"}'
goclaw tools exec netbox action=list resource_type=device-types filters='{"q":"PowerEdge R650"}'
goclaw tools exec netbox action=available-ips resource_type=prefixes id=2
goclaw tools exec netbox action=create resource_type=devices data='{"name":"prod-db-04","device_type":8,"device_role":3,"site":1,"rack":3,"status":"staged"}'
goclaw tools exec netbox action=create resource_type=ip-addresses data='{"address":"10.0.0.56/24","status":"active","description":"prod-db-04 mgmt"}'
```

## Implementation Files

- `internal/agent/tool_netbox.go`
- `internal/config/config_types.go`
- `config.yaml.example`

================================================================================
# FILE: https://goclaw.ulab.mn/docs/user-guide/metrics-store.md
================================================================================

# Metrics Store Tool

`metrics_store` is a structured metric memory for computed values such as cycle time, lead time, error rate, MTTR, CPU load, or custom SRE metrics. It stores already-computed numbers so agents can query and trend them later without calling the original API again.

## Backends

The tool supports two storage modes:

| Mode | Config required | Notes |
|---|---|---|
| Built-in | No | Uses GoClaw's own state database (`database:`, SQLite by default) |
| External datasource | Yes | Use `datasource=` to keep metrics in a Postgres or ClickHouse database |

For external datasources, define a normal datasource in `config.yaml` and pass its name to the tool. A datasource is read-only unless it sets `allow_write: true`:

- `record`, `annotate` and `purge` need `allow_write: true`, and an administrator or an admin-authored SOP or cron run. A webhook-triggered run, a user in a conversation, and a SOP run a user started are refused even with `allow_write`, and told to use the built-in store (omit `datasource`).
- `query` and `trend` are open to everyone. They create nothing: on a datasource where no metrics were ever recorded, they say so.
- A run acting on behalf of a person (a delegated SOP run) cannot use an external datasource at all, because metrics_store cannot confine what it reads there.
- On ClickHouse, `annotate` is refused: ClickHouse does not update rows in place.

```yaml
datasources:
  metrics_db:
    driver: postgres
    dsn: "postgres://user:pass@localhost/metrics?sslmode=disable"
    allow_write: true   # lets metrics_store (and db_write) write here
```

The external table is created by the first write. Postgres uses `agent_metrics` with JSONB tags; ClickHouse uses a `MergeTree` table.

## Actions

| Action | Purpose | Required fields |
|---|---|---|
| `record` | Store one or more metric values | `source`, `entity_type`, `entity_id`, `metrics` |
| `query` | Read matching metric rows | Optional filters |
| `trend` | Week-over-week aggregation | Optional filters |
| `annotate` | Add notes to existing records | `source`, `entity_id`, `notes` |
| `purge` | Delete old rows | `date_from` |

Date fields accept absolute `YYYY-MM-DD` dates or relative offsets such as `7d`, `4w`, and `3m`.

## Record Metrics

```bash
goclaw tools exec metrics_store action=record source=gitlab entity_type=mr entity_id=myorg/backend#42 metrics='{"cycle_time_h":18.5,"review_time_h":4.2}' tags='{"sprint":"W13","team":"backend"}' notes='Waited on review'
```

Using an external datasource:

```bash
goclaw tools exec metrics_store datasource=metrics_db action=record source=uptrace entity_type=service entity_id=api-service metrics='{"error_rate":0.023,"p95_ms":410}'
```

## Query Metrics

```bash
goclaw tools exec metrics_store action=query filter_source=gitlab filter_entity_type=mr filter_metric=cycle_time_h date_from=4w limit=100
goclaw tools exec metrics_store datasource=metrics_db action=query filter_source=uptrace date_from=2026-06-01 date_to=2026-06-20
```

`query` returns a compact Markdown table with source, entity type, entity ID, timestamp, metric name, value, and tags (notes are stored but not shown). It returns 50 rows unless `limit` says otherwise (at most 500).

## Trend Metrics

```bash
goclaw tools exec metrics_store action=trend filter_source=gitlab filter_metric=cycle_time_h weeks=8
goclaw tools exec metrics_store datasource=metrics_db action=trend filter_source=uptrace filter_metric=error_rate weeks=4
```

`trend` aggregates by week and returns count, average, min, and max for the selected metric.

## Annotate Metrics

```bash
goclaw tools exec metrics_store action=annotate source=gitlab entity_id=myorg/backend#42 notes='Hotfix; merged faster than the normal release path'
```

## Purge Old Metrics

```bash
goclaw tools exec metrics_store action=purge date_from=6m filter_source=gitlab
```

Without `filter_source`, `purge` deletes all records older than the threshold.

## Example Workflows

GitLab cycle time:

```text
1. Use gitlab mr_label_events or issue_label_events.
2. Calculate time between workflow labels.
3. Store cycle_time_h, review_time_h, and lead_time_h with metrics_store record.
4. Use metrics_store trend for weekly reporting.
```

Uptrace error rate:

```text
1. Use uptrace_query to fetch span or service aggregates.
2. Calculate error_rate or p95_ms.
3. Store metrics by service name.
4. Trend them by week or report them through SOP automation.
```

Weekly SOP sketch. A SOP has no schedule of its own: a [cron job](automation/cron-jobs.md) that names its `sop_id` runs it, here with the schedule `0 0 9 * * 1` (Mondays at 09:00; six fields, seconds first). The steps are abridged; see [SOPs](automation/sop.md) for every step field.

```json
{
  "id": "weekly-metrics-report",
  "name": "Weekly metrics report",
  "steps": [
    {"id": "s1", "name": "Last week", "tools": ["metrics_store"],
     "instruction": "Call metrics_store with action=query and date_from=7d."},
    {"id": "s2", "name": "Trend", "tools": ["metrics_store"],
     "instruction": "Call metrics_store with action=trend and weeks=4."},
    {"id": "s3", "name": "Report", "tools": ["generate_pdf"], "uses_outputs": ["s1", "s2"],
     "instruction": "Write a PDF report from the week's metrics and the trend."},
    {"id": "s4", "name": "Send", "tools": ["message"], "uses_outputs": ["s3"],
     "instruction": "Send the report to the team's Discord channel."}
  ]
}
```

## Paired SOP Architectural Pattern (`_hourly` + `_daily_summary`)

For continuous monitoring, an established best practice is pairing a frequent collection SOP with a daily roll-up summary SOP via `metrics_store`:

```mermaid
flowchart TD
    HOURLY["Hourly SOP: <topic>_hourly (Runs: 0 0 * * * *)
• Polls raw APIs / logs / spans
• Computes numeric values
• metrics_store action=record"]
    STORE[("metrics_store
SQLite, PostgreSQL, or ClickHouse")]
    DAILY["Daily SOP: <topic>_daily_sum (Runs: 0 0 22 * * *)
• metrics_store action=query
• metrics_store action=trend
• Generates executive report
• Sends notification to Teams"]

    HOURLY -->|"saves metrics"| STORE
    STORE -->|"queries 24h data"| DAILY
```

### Why use this pattern?
- **Avoids context-window overflow**: Raw logs and telemetry from an entire day would overwhelm the LLM's prompt window.
- **Resilient & cheap**: If a single hourly run fails or encounters an API glitch, the other 23 hours of data remain intact.
- **No notification fatigue**: Team channels are not spammed every hour; only the daily 22:00 rollup report is posted, while critical threshold alerts can still trigger independently.


## Benchmarking

```bash
goclaw tools exec metrics_store action=query
goclaw tools bench metrics_store action=query filter_source=gitlab --runs=10
```

## Implementation Files

- `internal/agent/tool_metrics.go`
- `internal/store/models.go` (GORM AutoMigrate creates the `agent_metrics` table)
- `config.yaml.example`

================================================================================
# FILE: https://goclaw.ulab.mn/docs/architecture/security.md
================================================================================

# Security + Exec Approval

GoClaw implements deep application-level security across multiple layers: path sandboxing, shell/HTTP allowlists, SSRF protection, PBKDF2 encryption, skill security scanning, SAST analysis, ingress IP allowlisting, and gateway auth.

## Security Architecture

```mermaid
flowchart TD
    subgraph SEC["GoClaw Security Layers"]
        GW["Gateway
Auth + Rate limit
trustProxy · IP allowlist"]
        TOOL["Tool-level
Sandboxing
Path / Shell · HTTP policy"]
        DATA["Data-at-rest
Encryption
AES-256-GCM · PBKDF2 (600K)"]

        NET["Network
SSRF block
Domain WL · Private IP"]
        CODE["Code
SAST scan
Linter · gosec / ruff"]
        CONT["Content
Skill scanner
Rule-based · Prompt inject"]

        GW --> NET
        TOOL --> CODE
        DATA --> CONT
    end
```

## Security Features

| # | Feature | OpenClaw | GoClaw | Status |
|---|---------|----------|---------|--------|
| 7.1 | **Path sandboxing** | Workspace isolation | `resolveFilePath()` — path traversal blocked, all paths resolved within workspace | Done |
| 7.2 | **Shell command allowlist** | Permission system | `tools.shell.allow` / `tools.shell.deny` in config. Empty allow = all blocked. Deny checked first. Entries match by prefix (`git` also matches `gitk`). A command containing a shell metacharacter (`;` `&` `\|` `$` `>` `<` `` ` `` `\` `!` `(` `)` `{` `}` `[` `]` `=` `#` `^` `%` `@` or a newline) is refused even with `allow: ["*"]`. Hardcoded deny patterns, matched anywhere in the command and case-insensitively: destructive commands (`rm -rf /`, `dd if=`, `mkfs.`, `format `), `shutdown`/`reboot`, Ansible, `make deploy`/`install`/`update`/`restart` and similar targets, service management (`systemctl`, `launchctl`, `service`), `ssh`/`scp`, system package installs, user management, Windows registry and firewall changes | Done |
| 7.3 | **DM pairing system** | Device pairing (web) | Pairing code for Discord/Teams DMs; approve on the Web UI's Pairing tab. A CLI approval made while the gateway runs reaches it only after a restart | Done |
| 7.4 | **DM policy modes** | Per-channel config | `dm_policy: pairing` (default), `allowlist`, `open` | Done |
| 7.5 | **Guild channel isolation** | Channel allowlists | Bot only responds to @mentions or direct replies in Discord guilds | Done |
| 7.6 | **Read-only database queries** | — | `db_query` allows only SELECT/SHOW/DESCRIBE/EXPLAIN. INSERT/UPDATE/DELETE blocked. Writes go through `db_write`, only on datasources with `allow_write: true`, only for admins and SOP/cron runs (not webhook-triggered or delegated runs), capped at `tools.max_db_write_rows` rows per call. `metrics_store` writes its own `agent_metrics` table on a datasource only with `allow_write`, for an admin or admin-authored SOP/cron run (not webhook-triggered or delegated runs); its first write creates the table, and its writes are not row-capped | Done |
| 7.7 | **Query timeout + row limits** | — | 30s timeout, which the database enforces too: about 29s on PostgreSQL and MySQL/MariaDB, and on ClickHouse where the account lets goclaw set `max_execution_time` (otherwise the profile's own); max 10,000 rows (`tools.max_db_query_rows`), max 1000 chars per cell | Done |
| 7.7a | **Read-only on the database** | — | `db_query` reads in a read-only PostgreSQL transaction; on MySQL/MariaDB with the connection's session set to `transaction_read_only=1` for the read (a read-only transaction alone still runs DDL); on ClickHouse with `readonly=1` where the account allows it (writes and `SET` refused; `url()`/`file()`/`remote()` stay refused by goclaw's checks, since `DESCRIBE TABLE url(…)` still connects out). `db_query`'s SQL never runs on PostgreSQL's simple protocol, which would run several statements from one query | Done |
| 7.8 | **HTTP response size limit** | — | `http_request` caps response body at 100KB | Done |
| 7.9 | **Tool loop safety** | — | `tools.max_tool_iterations` (default 20) prevents infinite LLM tool loops | Done |
| 7.10 | **Dangerous command hardcodes** | — | Fork bombs, disk wipes, shutdown commands blocked regardless of config | Done |
| 7.11 | **Git write gating** | — | Read ops always allowed, write ops require config flag, dangerous ops always blocked | Done |
| 7.12 | **File edit safety** | — | `file_edit` requires unique string match — no mass replacements | Done |
| 7.13 | **Bootstrap file protection** | Read-only MEMORY.md | For anyone but an admin or a SOP/cron run, `file_write`/`file_edit` block writes to MEMORY.md, SOUL.md, TOOLS.md, AGENTS.md, IDENTITY.md, USER.md, subagent profiles (`*.agent.md`) and instructions (`*.instructions.md`). No file tool writes inside a `.git` directory | Done |
| 7.14 | **Gateway auth (token)** | OAuth / device code | `X-GoClaw-Token` header, `Bearer` auth, health probes exempt | Done |
| 7.15 | **Rate limiting** | — | Per-IP token bucket middleware, configurable `gateway.rate_limit.per_minute` | Done |
| 7.16 | **Secrets encryption** | Credential store | AES-256-GCM vault + **PBKDF2 key derivation** (600K iterations) | Done |
| 7.17 | **Trust proxy** | — | `gateway.ingress.trust_proxy` (and `gateway.rate_limit.trust_proxy`) — only reads `X-Forwarded-For` when trusted, and then its last entry, the one the proxy wrote. Used by the ingress allowlist, the rate limits (gateway and App API) and the audit log. Prevents IP spoofing | Done |
| 7.18 | **HTTP default-deny policy** | — | nginx-style: empty allow list blocks all external HTTP. `default_policy: deny\|allow` | **Done (GoClaw only)** |
| 7.19 | **SSRF protection** | — | DNS-level private IP blocking (0.x, 10.x, 100.64-127.x, 172.16-31.x, 192.168.x, loopback, link-local, ULA IPv6) | **Done (GoClaw only)** |
| 7.20 | **Domain whitelist/blacklist** | — | `tools.http.allow` / `tools.http.deny` with wildcard (`*.mn`), CIDR, exact match | **Done (GoClaw only)** |
| 7.21 | **Internal hostname bypass** | — | Allowlisted hostnames that resolve to private IPs are permitted (e.g. `monitor.example.mn` → 10.0.2.7). A wildcard entry counts: see [Bypass Mechanisms](#bypass-mechanisms) | **Done (GoClaw only)** |
| 7.22 | **Per-agent HTTP scope** | — | Agent-level `tools.http.allow/deny` overrides global policy via `runtimeScope` | **Done (GoClaw only)** |
| 7.22a | **Hosts per run** | — | `tools.http.max_hosts_per_run` (default 100): one agent run, its subagents included, reaches at most that many different hosts through `SafeHTTPClient`; the next new host is refused and logged. A start-up warning names `shell.allow: ["*"]` or an open `default_policy: allow` on a gateway other machines can reach | **Done (GoClaw only)** |
| 7.23 | **Pre-flight URL validation** | — | `ValidateURL()` — domain policy + SSRF check before the browser tools navigate and before `notify` posts to a webhook URL. Other tools' requests are checked when they connect (`SafeHTTPClient`) | **Done (GoClaw only)** |
| 7.24 | **Skill Security Scanner** | — | Rules for prompt injection in skill text and dangerous code patterns. Blocks critical findings on install | **Done (GoClaw only)** |
| 7.25 | **SAST Scanner** | — | `run_sast_scanner` tool — runs gosec (Go) or bandit (Python) on workspace code; the scanner must be installed on the host | **Done (GoClaw only)** |
| 7.26 | **Code Linter** | — | `run_code_linter` tool — runs golangci-lint (Go), ruff (Python) or `dart analyze` (Dart/Flutter); the linter must be installed on the host | **Done (GoClaw only)** |
| 7.27 | **Ingress IP allowlist** | — | Webhook ingress: M365 preset auto-fetch or explicit IPs/CIDRs + access logging (see [below](#ingress-ip-allowlist-ms-teams)) | **Done (GoClaw only)** |
| 7.28 | **Memory write RBAC** | — | `memory_write`: `personal` scope for any identified (paired) user, never from SOP/cron/webhook runs; `agent_notes` and `agent_config` for admins and the roles/users listed under `memory.write_access` (SOP/cron/webhook runs may write `agent_notes` only, never `global` or `agent_config`); `global`: while `memory.write_access` is enabled (the default, with `allowed_roles: [system]`), only the roles and users it lists; with it disabled, admins. `memory_consolidate`: admins and SOP/cron runs. `update_core_instruction`: a human admin only | Done |
| 7.29 | **isHumanAdmin Check** | — | Critical actions (altering identity, persona) require a real human admin — automated SOP/Cron runs are blocked. | **Done (GoClaw only)** |
| 7.30 | **Rollback Safety** | — | SOP rollback commands are validated against the shell allowlist to prevent privilege escalation. | **Done (GoClaw only)** |
| 7.31 | Docker sandboxing | Docker-based isolation | — | **TODO** |


## SSRF Protection Detail (safe_client.go)

GoClaw's `SafeHTTPClient` implements DNS-level SSRF protection at the transport layer:

```mermaid
flowchart LR
    REQ["HTTP Request"] --> DOM["Domain Filter"]
    DOM -->|"Pass"| DNS["DNS Resolve"]
    DNS --> IP["Private IP Check"]
    IP -->|"Public / Allowed"| CONN["Connect"]

    DOM -->|"Deny list / Not in allow /
default_policy=deny + empty allow"| B1["BLOCK"]
    IP -->|"10.x / 172.16-31.x / 192.168.x /
loopback / link-local / IPv6 ULA"| B2["BLOCK"]
```

### Private IP Ranges Blocked
| Range | Description |
|-------|------------|
| `0.0.0.0/8` | "This host" (a connect to `0.0.0.0` reaches loopback) |
| `10.0.0.0/8` | Class A private |
| `100.64.0.0/10` | Carrier-grade NAT |
| `172.16.0.0/12` | Class B private |
| `192.168.0.0/16` | Class C private |
| `127.0.0.0/8`, `::1` | Loopback |
| `169.254.0.0/16`, `fe80::/10` | Link-local |
| `fc00::/7` | IPv6 Unique Local Address |

### Bypass Mechanisms
- Explicit IP/CIDR in `tools.http.allow` list. IP and CIDR entries match only a URL whose host is an IP address: a hostname that resolves into an allowed CIDR is still refused unless the hostname itself is allowed.
- Hostname in allow list that resolves to private IP (e.g. `monitor.example.mn` → `10.0.2.7`)
- A wildcard entry allows private addresses for every hostname it matches: with `*.mn`, any `.mn` name that resolves to a private address is reached, and `*` turns private-address blocking off for every host. Allow internal hosts by their exact name, and keep wildcards for public domains.
- An integration's configured host: `gitlab`, `sonarqube`, the observability tools, Doctor and the Integrations page's Test reach the host of the instance's own `url` (or Uptrace `dsn`) as if it were on allow (`WithConfiguredHost`), for their own requests only. The address is the admin's, not the model's; `http_request` and the browser are not affected, and every other host such a request reaches (a redirect) is checked as usual.
- The browser tools check the URL before navigating (`ValidateURL()`) by the same rules, and every request a page makes against the policy of the agent using the browser.
- Not bypassed: deny list always takes priority. An IP or CIDR on `tools.http.deny` is refused whichever host name resolves to it, a name on the allow list included.
- Keep-alive connections are pooled per HTTP policy (a configured integration host included), so a connection opened under one agent's policy, for an integration, or before a config reload changed the lists, is not reused by a request the current policy would refuse.

## HTTP Default-Deny Policy

```yaml
tools:
  http:
    default_policy: "deny"   # "deny" (default) or "allow"
    allow:
      - "*.googleapis.com"
      - "api.github.com"
      - "monitor.example.mn"           # internal hostname → private IP allowed
      - "10.0.2.0/24"      # explicit CIDR allowlist (URLs with an IP host only)
    deny:
      - "127.0.0.1"
      - "*.internal"
```

### Domain Matching
| Pattern | Matches |
|---------|---------|
| `*` | All domains, private addresses included |
| `*.mn` | `example.mn`, `sub.example.mn`, including names that resolve to private addresses |
| `api.github.com` | Exact match only |
| `10.0.2.0/24` | CIDR range, for URLs whose host is an IP address |
| `192.168.1.1` | Exact IP, for URLs whose host is that IP |

## Skill Security Scanner

| Layer | Rules | Detects |
|-------|:-----:|--------|
| **Prompt Injection** | `prompt-override`, `role-injection`, `jailbreak-attempt`, `shell-danger-in-skill`, `secret-extraction`, `hidden-instruction`, `invisible-unicode`, `data-exfil-url` | LLM manipulation, `curl\|bash`, credential theft, instructions hidden in HTML comments or zero-width characters |
| **Code Patterns** | `dangerous-exec`, `dynamic-code-exec`, `crypto-mining`, `reverse-shell`, `env-harvesting`, `potential-exfiltration`, `obfuscated-hex`, `obfuscated-base64`, `suspicious-network` | Shell injection, mining, exfiltration |

- Scans each skill installed with `goclaw skills install` or `goclaw skills add`, from any source
- Critical findings = **blocked + auto-removed**; warnings are printed and the skill stays installed
- `goclaw skills scan` / `goclaw skills scan `

## Ingress IP Allowlist (MS Teams)

| Feature | Description |
|---------|-------------|
| M365 preset | `gateway.ingress.allow_preset: microsoft-teams` fetches the Microsoft 365 Skype/Teams IPv4 ranges from `endpoints.office.com` at startup. If the fetch fails, the gateway does not start |
| Explicit IPs | Direct IP addresses in `gateway.ingress.allow_ips`. A non-empty `allow_ips` replaces the preset: the two are not combined |
| CIDR ranges | Network ranges (e.g. `52.112.0.0/14`) |
| Scope | Without `gateway.ingress.port` the webhooks share the main port, and the allowlist then applies to the whole gateway, Web UI included. Set `gateway.ingress.port` to keep it to the webhook listener |
| Loopback | Requests from `127.0.0.0/8` and `::1` are always let through (CLI, local health probes) |
| Access logging | Every request logged via `runtimeinfo.RecordIngressAccess()` |
| Denied tracking | `runtimeinfo.RecordIngressDenied()` for monitoring |
| Web UI display | Gateway tab shows access logs table |

---

## Exec Approval (Section 19)

GoClaw's interactive approval covers SOP steps only. A step marked `requires_approval: true` waits for an approval made through the gateway API; everything else is governed by the allow/deny lists.

| # | Feature | OpenClaw | GoClaw | Status |
|---|---------|----------|---------|--------|
| 19.1 | Approval policy | `security` (deny/allowlist/full) | Shell allowlist + HTTP domain policy + per-agent scopes | Done |
| 19.2 | **Control UI approval** | Visual approve/deny in web panel | No dashboard control. A SOP step marked `requires_approval: true` is approved or rejected with `POST /api/sop/approve` (`{"job_id": "…", "action": "approve"}` or `"reject"` with a `"reason"`) and the gateway token. A step still waiting after 1 hour fails. | Partial |
| 19.3 | macOS app approval | Native dialog | — | **TODO** |
| 19.4 | **Channel approval** | `/approve ` | — (no chat or CLI command approves a step) | **TODO** |
| 19.5 | Allowlist persistence | `exec-approvals.json` | `config.yaml` based allow/deny lists with hot-reload support. | Done |
| 19.6 | Safe bins auto-allow | Known safe commands pre-approved | Hardcoded deny patterns + skill bin-check | Partial |
| 19.7 | Skill CLI auto-allow | Skills register their CLIs as safe | — | **TODO** |
| 19.8 | Approval forwarding | Route approval prompts to channels | A step waiting for approval sends a notice to the channel and target set in `sop.alerts` (else only to the log). The notice carries the job ID to pass to the API; the `sop approve ` it suggests is not a command. | **Done (GoClaw only)** |

---

## OpenClaw vs GoClaw Security Comparison

| Aspect | OpenClaw | GoClaw |
|--------|----------|---------|
| **Sandboxing model** | Docker container isolation | Application-level (path, shell, HTTP, IP) |
| **HTTP policy** | No default-deny | **nginx-style default-deny** |
| **SSRF protection** | — | **DNS-level private IP blocking** |
| **Domain filtering** | — | **Wildcard + CIDR + hostname matching** |
| **Secret encryption** | Credential store | **AES-256-GCM + PBKDF2 (600K iterations)** |
| **Skill scanning** | — | **Rule-based security scanner** |
| **Code analysis** | — | **SAST scanner + code linter (gosec, bandit, golangci-lint, ruff)** |
| **Ingress security** | — | **M365 IP preset + CIDR allowlist + access logs** |
| **Memory RBAC** | — | **Admin-only write for core tools** |
| **Interactive approval** | Multi-surface (web, macOS, channel) | SOP step approval through the API; otherwise shell/HTTP allowlists |
| **Per-agent scopes** | — | **Agent-level HTTP policy overrides** |

**GoClaw security philosophy:** Defense in depth at the application layer — no Docker dependency for security. Every tool has its own safety boundary (path traversal, SQL injection, SSRF, shell injection) with configurable allow/deny policies.

**Coverage:** the security features above are in place except Docker sandboxing; exec approval is limited to SOP steps approved through the API.

================================================================================
# FILE: https://goclaw.ulab.mn/docs/architecture/secrets.md
================================================================================

# Secrets Vault & Configuration Interpolation

GoClaw features a secure, built-in **Secrets Vault** and dynamic **Configuration Interpolation** engine. This system ensures that API keys, webhook credentials, database passwords, and other sensitive parameters are encrypted at rest and never stored in plain-text inside configuration files.

---

## 1. Secrets Vault Architecture

The Secrets Vault is designed to provide secure credentials management on local filesystems:

```mermaid
flowchart TD
    CLI["goclaw CLI / API"] --> ENG["AES-256-GCM Engine
• PBKDF2 Key Derivation (600K iterations)
• Salt: random 32 bytes per vault"]
    ENG --> FILE["~/.goclaw/credentials/secrets.enc
(File permissions: 0600)"]
```

- **Encryption at Rest:** Secrets are stored in a single encrypted file, `credentials/secrets.enc` in the state directory (`~/.goclaw`, or `GOCLAW_STATE_DIR` when set), using **AES-256-GCM** encryption.
- **Key Derivation:** The encryption key is derived from a passphrase using **PBKDF2** (SHA-256) with **600,000 iterations** and a random 32-byte salt generated for each vault and stored in the file's header. A vault written by an older version with the fixed legacy salt is still read, and is rewritten in the new format on its next save.
- **Passphrase:** The passphrase is the `GOCLAW_SECRET_KEY` environment variable when it is set. Otherwise it is derived (SHA-256) from `goclaw-secrets`, the OS user name and a machine identifier (`/etc/machine-id` on Linux, `/var/db/SystemKey` on macOS, `COMPUTERNAME` on Windows, else the home directory), and the gateway logs a warning at startup. Those identifiers are readable by anyone with access to the host, so the derived passphrase protects little against a local attacker. It also ties the vault to one user on one machine: a vault copied to another host, or opened by another user (for example through `sudo`), fails with `decrypt failed (wrong passphrase?)`. In production, set `GOCLAW_SECRET_KEY` and keep it outside the state directory (an OS keychain, or a `0600` file the service reads).
- **File Permissions:** The `credentials/` directory is created `0700` and `secrets.enc` is written `0600` (read/write by the owner only), so other users on the host cannot read the file.
- **Masked Listing:** When listing credentials via the CLI or admin dashboard, values are masked to their first and last four characters (e.g., `AIza****xAPw`); a value of 8 characters or fewer is shown as `****`. `goclaw secret list --show-values` and `goclaw secret get` print the plaintext.

---

## 2. Configuration Interpolation

GoClaw supports configuration interpolation inside `config.yaml`. This allows you to reference environment variables, system files, or vault secrets from the fields that hold credentials.

### Supported Reference Patterns

| Pattern Syntax | Resolution Source | Example |
|---|---|---|
| `${secret:KEY}` | Looks up `KEY` inside the encrypted Secrets Vault. | `api_key: "${secret:GEMINI_API_KEY}"` |
| `${env:VAR}` | Fetches the value of the environment variable `VAR`. | `api_key: "${env:OPENAI_API_KEY}"` |
| `${file:/path}` | Reads the first line of the specified file, trimming whitespace. | `token: "${file:/etc/secrets/token}"` |
| `${VAR}` (Bare) | Fallback mode: tries environment first, then queries the Secrets Vault. | `token: "${DISCORD_TOKEN}"` |

### Which Fields Are Expanded

References are expanded only in fields that hold credentials. In any other
field (a port, a URL, a model name, a channel ID) a `${...}` reference stays
literal text. The expanded fields are:

- **AI providers:** `ai.openai.api_key`, `ai.gemini.api_key`, `ai.anthropic.api_key`, `ai.deepseek.api_key`.
- **Gateway:** `gateway.auth.token`, `gateway.ingress.authtoken`, `gateway.ngrok.authtoken`.
- **Databases:** `database.dsn`; `memory.graph.dsn` and `memory.graph.password`. The gateway opens the vault before the state database, so its DSN may name a vault secret.
- **Channels:** `channels.discord.token`; `channels.msteams.app_id`, `app_password`, `tenant_id`, `ngrok.authtoken`; `channels.facebook.page_token`, `app_secret`, `verify_token`; `channels.telegram.token`, `webhook_secret`; `channels.app_api.api_key`, `jwt_secret` and the `key` of each `api_keys` entry.
- **Datasources:** `password` and `dsn` of each entry under `datasources`.
- **Tools:** `tools.web_search.brave_api_key`, `serp_api_key`; the `token` of `tools.gitlab`, `tools.sonarqube` and each instance under `tools.gitlabs` / `tools.sonarqubes`; the credentials of each instance under `tools.observability` (Grafana `token`/`password`, Graylog `token`, Uptrace `token`/`dsn`, Zabbix `token`/`password`, Wazuh `password`, NetBox `token`); and the same GitLab, SonarQube and observability fields in an agent's own `tools` overrides.
- **MCP servers:** every value under `env` and `headers` of each `mcp.servers` entry.
- **Logging:** `logging.otel.dsn` and every value under `logging.otel.headers`.
- **Voice:** `voice.elevenlabs.api_key`; `voice.chimege.tts_token`, `stt_short_token`, `stt_long_token`.
- **Email:** `email.smtp.password`, `email.imap.password`.
- **Hooks:** the `url` and `secret` of each webhook hook under `hooks`.

A webhook route's `secret`, `token` and `agent`, whether set on the Webhooks page or under
`webhooks:` in `config.yaml`, may also be a `${secret:NAME}` reference; it is expanded on each
delivery.

### Configuration Example

Below is an example of a secure `config.yaml` utilizing all interpolation types:

```yaml
server:
  port: "8080"            # not a credential field: written as a plain value

gateway:
  bind: "loopback"
  auth:
    token: "${secret:GATEWAY_ACCESS_TOKEN}"

ai:
  provider: "gemini"
  gemini:
    api_key: "${secret:GEMINI_API_KEY}"
  openai:
    api_key: "${env:OPENAI_API_KEY}"
  anthropic:
    api_key: "${file:/run/secrets/anthropic_key}"

channels:
  discord:
    token: "${DISCORD_BOT_TOKEN}"
```

---

## 3. Managing Secrets via CLI

You can easily manage your vault credentials from the terminal. The CLI supports category tags and provider association to organize tokens.

### Common CLI Operations

- **Generate a Secret:**
  ```bash
  goclaw secret gen  --provider 
  ```
  Produces a cryptographically random value, stores it, and prints it once so
  it can be handed to whatever will present it. Use this for the shared secrets
  GoClaw asks for but does not issue — `channels.app_api.api_key` and its
  `api_keys` entries, `jwt_secret`, webhook tokens — rather than inventing a
  value by hand.
  *Example:*
  ```bash
  goclaw secret gen APP_KEY_ERP --provider app_api
  ```
  Flags: `--length` (bytes, default 32, minimum 16), `--format hex|base64`,
  `--category`, `--provider`, and `--force` to replace an existing name.
  Replacing revokes whatever still presents the old value, so it is not the
  default.

- **Add or Update a Secret:**
  ```bash
  goclaw secret set   --category  --provider 
  ```
  *Example:*
  ```bash
  goclaw secret set GEMINI_API_KEY AIzaSyBEt... --category api_key --provider gemini
  ```

- **Retrieve a Decrypted Secret:**
  ```bash
  goclaw secret get GEMINI_API_KEY
  ```
  Prints the plaintext. The CLI has no role check of its own: anyone who can
  run `goclaw` as the gateway's OS user (the derived passphrase is the same),
  or who has `GOCLAW_SECRET_KEY`, can read every value.

- **List All Secrets (Masked):**
  ```bash
  goclaw secret list
  ```
  `--show-values` prints the values unmasked.

- **Delete a Secret:**
  ```bash
  goclaw secret delete GEMINI_API_KEY
  ```

### Agent Tools

The agent has four vault tools: `secret_set`, `secret_get`, `secret_list` and
`secret_delete`. Each one is refused unless a human admin is the caller; a
non-admin user, and SOP, cron and webhook runs, get an access-denied error.
`secret_list` masks values as the CLI does. `secret_get` returns the plaintext
value to the model, so it then appears in the conversation and its trace and
is sent to the LLM provider. Tools that need a credential do not need it: they
get it from `${secret:NAME}` expansion in their configuration.

The model can also name a vault entry without seeing it: a header it writes for
`http_request`, or for an `mcp` call, may hold `${secret:NAME}` (or `${env:VAR}`,
`${file:/path}`), which is expanded as the request is sent. Only an admin, or a
SOP or cron run an admin set up, may do this; for anyone else, a webhook-started
run included, the call is refused. A credential a regular user's requests need
belongs in the integration's own configuration instead.

---

## 4. Reload behaviour

The vault file is shared with the `goclaw secret` CLI, which writes it from a
separate process. The gateway notices:

- **The secrets API re-reads the vault on every request**, so a secret written
  from a terminal shows up in the Web UI immediately.
- **`${secret:NAME}` in `config.yaml` is re-expanded whenever the config is
  loaded** — at startup and on every hot reload. Storing a secret does not by
  itself re-resolve a reference that was already expanded, so after adding one
  that the config points at, touch `config.yaml` (or restart) to put it into
  effect.

The config watcher accepts timestamp-only changes (`touch`) and continues
watching when an editor saves by replacing the file. Reloads run in order.
If the vault changes again during a read, that newer version remains detectable
on the next reload.

A vault that cannot be decrypted — re-keyed elsewhere, or caught mid-write —
leaves the already-loaded credentials in place and logs a warning rather than
emptying a running gateway.

---

## 5. Troubleshooting

- **A secret written while the gateway is running does not appear.** The secrets
  page re-reads the vault per request, so this should not happen; if it does, the
  gateway logged a reload failure (a re-keyed vault, or one written under a
  different passphrase — see section 1). A `${secret:NAME}` that the config
  already expanded is a different case and is expected: it refreshes on the next
  config load, so touch `config.yaml`.

- **Unresolved Value Warns:** If an interpolation reference (e.g., `${secret:MISSING_KEY}`) cannot be resolved, GoClaw logs a warning (once per reference) and leaves the reference in the field as literal text. Credential checks treat such a value as missing, never as the token: an unresolved `gateway.auth.token` makes the gateway answer every request that needs the token with HTTP 500 (`gateway.auth.token could not be resolved`), while the health probes, the Web UI's own files and the app API still answer, an unresolved `channels.app_api` key or JWT secret refuses every app API caller, and an unresolved webhook route secret refuses every delivery to that route.
- **`decrypt failed (wrong passphrase?)`:** The vault was written under a different passphrase: another machine, another OS user, or a different `GOCLAW_SECRET_KEY` (see section 1). Run the gateway as the user that created the vault, or with the `GOCLAW_SECRET_KEY` it was created with.
- **Permission Denied Errors:** If GoClaw fails to read the secrets database, ensure the running user has ownership of the `~/.goclaw/` directory and that the credentials file has the correct `0600` permissions.

================================================================================
# FILE: https://goclaw.ulab.mn/docs/architecture/audit_logging.md
================================================================================

# Webhook Audit Logging

Webhook audit logging records semantic channel events in memory so operators can inspect recent activity from the Web UI. It is designed for real-time visibility, not long-term forensic storage.

## Data Flow

```mermaid
flowchart TD
    CH["Channels
Discord / Teams / Facebook / Telegram"] --> H["Channel Handler"]
    H --> REC["runtimeinfo.RecordWebhookAudit"]
    REC --> BUF["runtimeinfo.WebhookAuditLogs
In-memory ring buffer (newest 500)"]
    BUF --> API["GET /api/audit"]
```

## Data Structure

```go
type WebhookAuditLog struct {
    Timestamp  time.Time `json:"timestamp"`
    Channel    string    `json:"channel"`
    EventType  string    `json:"event_type"`
    AuthorID   string    `json:"author_id"`
    AuthorName string    `json:"author_name"`
    PeerID     string    `json:"peer_id"`
    PeerKind   string    `json:"peer_kind"`
    Preview    string    `json:"preview"`
    Status     string    `json:"status"`
}
```

`RecordWebhookAudit` sets `Timestamp` when missing, cuts `Preview` to its first 120 bytes followed by `…`, and keeps only the newest 500 records. The buffer is in memory: a restart empties it.

`Status` is `processed`, `error` (handling or answering the message failed), `duplicate` (a repeated Teams delivery that was ignored), `ignored`, or, for a Teams SSO attempt, `unimplemented`.

## Event Types

| Event type | Meaning |
|---|---|
| `chat` | Regular text message |
| `command` | Slash command or command-style message |
| `typing` | Typing indicator, usually ignored |
| `reaction` | Message reaction, usually ignored |
| `comment` | Conversation update or comment-style event |
| `login_attempt` | Teams SSO token exchange (recorded with channel `msteams_sso`) |
| `unknown` | Channel event that did not match a known class |

## API

```text
GET /api/audit?channel=facebook&event=chat&user=USER123&limit=50
```

All query parameters are optional.

| Parameter | Description |
|---|---|
| `channel` | Filter by channel: `discord`, `msteams`, `msteams_sso`, `facebook` or `telegram` |
| `event` | Filter by event type |
| `user` | Filter by author ID |
| `limit` | Maximum rows, 1 to 500 (default 100). A missing, invalid, zero, negative or larger value gives 100 |

Response:

```json
{
  "logs": [
    {
      "timestamp": "2026-06-20T08:45:00Z",
      "channel": "discord",
      "event_type": "chat",
      "author_id": "123456789",
      "author_name": "User",
      "peer_id": "987654321",
      "peer_kind": "direct",
      "preview": "Hello, can you help me?",
      "status": "processed"
    }
  ],
  "total": 1
}
```

## Recording Points

| File | Event source |
|---|---|
| `internal/channel/discord/discord_handler.go` | Discord messages |
| `internal/channel/discord/discord_slash.go` | Discord slash commands |
| `internal/channel/msteams/msteams.go` | Teams activities |
| `internal/channel/msteams/msteams_sso.go` | Teams SSO token exchange attempts (channel `msteams_sso`, event `login_attempt`) |
| `internal/channel/facebook/facebook.go` | Facebook messages |
| `internal/channel/telegram/telegram.go` | Telegram messages |

## Audit Lines in the Gateway Log

A separate trail, `internal/audit`, records destructive or administrative
actions as one line each in the gateway log:

```text
[audit] action=sop.delete actor=… role=… source=… ip=… resource=sop/ success=true
```

It is written for `sop.delete` (from the `sop` tool and the SOP API) and for
`sop.channel.pin` / `sop.channel.unpin` (the SOP channels API). Find them with
`grep '\[audit\]' gateway.log`. The last 500 are also kept in memory and
served by `GET /api/audit?kind=admin`, newest first, filtered by `?action=`,
`?actor=` and `?resource=` (a resource type such as `sop`) and limited by
`?limit=` (default 100); they do not survive a restart.

## Tests

```bash
go test ./internal/runtimeinfo -count=1
```

The runtimeinfo tests cover recording, timestamp defaults, preview truncation, ring buffer overflow, filters, and newest-first ordering.

================================================================================
# FILE: https://goclaw.ulab.mn/docs/architecture/enterprise-security-matrix.md
================================================================================

# Enterprise Security Matrix & InfoSec Controls

This document provides a comprehensive security controls matrix for Enterprise and Pilot Information Security (InfoSec) teams deploying GoClaw in sensitive corporate, financial, government, or telecom environments.

---

## 1. Network Perimeter & Egress Firewall Matrix

GoClaw is designed to operate securely within strictly air-gapped or egress-restricted perimeter networks (e.g. DMZ or isolated worker VLANs).

### Inbound Ports (Ingress)
| Port | Protocol | Interface / Bind | Purpose | Auth Required |
|---|---|---|---|---|
| **8080** (default, `server.port`) | HTTP | `gateway.bind`: `127.0.0.1` by default (`loopback`), `lan` for all interfaces, or a specific IP | Web UI, REST API, WebSocket Gateway | **Yes** (`X-Goclaw-Token` via `AuthMiddleware`) |
| `gateway.ingress.port` (optional, unset by default; e.g. 8081) | HTTP | `gateway.ingress.bind`: `127.0.0.1` by default, or `lan` / a specific IP | Dedicated webhook listener: Teams, Facebook and Telegram channel webhooks, and `/goclaw/hook/...` routes (GitLab, GitHub, Graylog, …). Unset, these share the main port | **Yes** (per channel: signature, Bot Framework token or secret token; per route: HMAC secret or token) |
| **8181** (optional, `tools.html.local_server`) | HTTP | `lan` by default | Serves only `/goclaw/view/` (agent-generated HTML pages) to the local network | **Yes** (per-page view token) |

GoClaw serves plain HTTP on every port; it does not terminate TLS. Put a
reverse proxy or load balancer in front of it for HTTPS.

> [!IMPORTANT]
> All other inbound ports are **CLOSED**. GoClaw exposes no SSH, Telnet, SMB, or unauthenticated RPC endpoints.

### Outbound Ports (Egress Allowlist)
When deploying behind an enterprise egress firewall or proxy, configure the following domain allowlist. **All other outbound traffic can be blocked with a default `DENY ALL` rule.**

| Destination Domain | Port | Protocol | Purpose | Fallback / Alternative |
|---|---|---|---|---|
| `generativelanguage.googleapis.com` | 443 | HTTPS/TLS 1.3 | Google Gemini / GenAI inference | Local Ollama / Private vLLM |
| `api.anthropic.com` | 443 | HTTPS/TLS 1.3 | Anthropic Claude inference | Local Ollama / Private vLLM |
| `api.openai.com` | 443 | HTTPS/TLS 1.3 | OpenAI inference | Local Ollama / Private vLLM |
| `api.deepseek.com` | 443 | HTTPS/TLS 1.3 | DeepSeek inference | Local Ollama / Private vLLM |
| `discord.com`, `gateway.discord.gg`, `cdn.discordapp.com` (optional) | 443 | WSS / HTTPS | Discord channel integration (API, gateway, attachments) | Disable if not used |
| `login.microsoftonline.com`, `*.botframework.com`, `smba.trafficmanager.net`, `graph.microsoft.com` (optional) | 443 | HTTPS | MS Teams: bot token, Bot Framework, replies, Graph | Disable if not used |
| `endpoints.office.com` (optional) | 443 | HTTPS | Microsoft 365 IP list for `gateway.ingress.allow_preset: microsoft-teams`, fetched at startup (the gateway does not start without it) | Use explicit `allow_ips` instead |
| `graph.facebook.com`, `*.fbcdn.net`, `*.fbsbx.com` (optional) | 443 | HTTPS | Facebook Messenger channel (messages; attachment downloads) | Disable if not used |
| `api.telegram.org` (optional) | 443 | HTTPS | Telegram channel | Disable if not used |
| `connect.ngrok-agent.com` (optional) | 443 | TLS | ngrok tunnel when `gateway.ingress.type: ngrok` | Reverse proxy (`type: proxy`) |
| *All other destinations* | * | * | **DENY ALL** | — |

Tools that call other systems (GitLab, Grafana, Uptrace, MCP servers, databases,
`http_request`, web search) need their own destinations opened as well.

---

## 2. Operating System Hardening & Least Privilege

### Rootless Container (Docker)
The official GoClaw Docker container operates with least privilege:
- **Dedicated Non-Root User**: Runs as UID `10001` (`goclaw`).
- **No SUID Binaries**: Root escalation is disabled.
- **Read-Only Root Filesystem**: Compatible with `read_only: true` container flags when the state directory `/home/goclaw/.goclaw` (or `GOCLAW_STATE_DIR`) is a mounted volume — it holds the database (`gc_state.sqlite`), the secrets vault, the workspace and the logs — and `/tmp` is writable (for example a tmpfs), since tools write temporary files there. The bundled `docker-compose.yaml` mounts `config.yaml` at `/app/config.yaml` and a named volume `goclaw-state` on the state directory.

### Linux systemd Hardening
`goclaw service install` writes a per-user unit (`~/.config/systemd/user/goclaw.service`)
by default. `sudo goclaw service install --system` writes a system unit,
`/etc/systemd/system/goclaw.service`, that runs as the user who ran `sudo` (or
the user given with `--run-as `), never as root. The generated unit sets
`Restart=always` and `RestartSec=5`, `Environment=HOME=` and
`GOCLAW_STATE_DIR=/.goclaw` for that user, and reads optional variables
from `/etc/sysconfig/goclaw` (`EnvironmentFile=-`). For the system unit that
file is a place for `GOCLAW_SECRET_KEY`, root-owned and `0600`; a per-user
unit's manager runs as the user, so there the key file must be one the user
can read, or the key is silently not set.

Add sandbox directives as a drop-in rather than by editing the unit, which a
reinstall rewrites. For the system unit, `sudo systemctl edit goclaw` opens
`/etc/systemd/system/goclaw.service.d/override.conf`; for the per-user unit use
`systemctl --user edit goclaw`. Keep `Restart=always`.
The state directory lives in the run-as user's home, so `ProtectHome=read-only`
and `ProtectSystem=strict` need a `ReadWritePaths=` line for it; without one
the gateway cannot write its database, vault or logs.

```ini
# sudo systemctl edit goclaw
[Service]
# ── Sandboxing & Process Hardening ──
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=read-only
# the run-as user's state directory:
ReadWritePaths=/home/goclaw/.goclaw
PrivateTmp=true
PrivateDevices=true
ProtectKernelTunables=true
ProtectControlGroups=true
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
RestrictRealtime=true
CapabilityBoundingSet=
```

Then `sudo systemctl daemon-reload && sudo systemctl restart goclaw`. These
settings also bind the commands the agent runs through `shell_exec`: one that
needs a device or a capability (a live `tcpdump` capture, for example) fails
under them.

---

## 3. AI & Data Privacy Guarantees

| Concern | GoClaw Guarantee | Technical Control |
|---|---|---|
| **Model Training on Customer Data** | **No retention by GoClaw; provider retention is yours to choose** | GoClaw stores no prompt or completion outside the customer network, and adds no telemetry of its own to inference calls. Whether the *provider* retains or trains on the data is a property of the endpoint and contract you configure, not of GoClaw: enterprise tiers (Vertex AI, Anthropic Enterprise, Azure OpenAI) offer zero-retention terms, whereas the standard public endpoints listed in the egress table do not grant them by default. For a zero-retention deployment, point GoClaw at an enterprise tier under your own agreement, or at a local Ollama / private vLLM backend. |
| **Local Vector & Memory Storage** | **On-Premise Persistence** | All episodic memories, session histories, and knowledge graphs reside in local SQLite / PostgreSQL / MySQL databases inside the customer network. |
| **Secrets in LLM Context** | **Vault References** | Credentials in `config.yaml` are written as `${secret:NAME}` references to the GoClaw secrets vault and resolved inside the gateway when the configuration loads (only in credential fields; see [Secrets](secrets.md#which-fields-are-expanded)). The model works with tools, not with the credentials behind them: a tool reads its credential from the loaded configuration. The exception is the vault tool `secret_get`, which only a human admin may call: it returns the plaintext value into the conversation, and so to the LLM provider. |

---

## 4. Tool Sandboxing & Prompt Injection Defense (OWASP LLM Top 10)

GoClaw implements multiple defense-in-depth layers to mitigate prompt injection and unauthorized command execution:
1. **Shell Command Allowlist**: Shell execution (`shell_exec`, `shell_exec_multi`) is governed by `tools.shell.allow` and `tools.shell.deny` in `config.yaml`:

   ```yaml
   tools:
     shell:
       allow: ["git", "date", "whoami"]   # empty or absent = every command blocked
       deny:  ["curl", "wget"]            # evaluated before allow
   ```

   An **empty `allow` list blocks every shell command**, and it is the default: no `allow` key means no shell. The setup wizard's operating mode writes the list: *Minimal* leaves it empty, *Full* writes `allow: ["*"]`, so a deployment set up as Full must replace that with an explicit allow list to narrow it again. `tools.mode` itself only chooses which `AGENTS.md` is seeded into the workspace; it does not change the shell policy. A hard-coded deny list (`rm -rf /`, `mkfs.`, `dd if=`, `format `, `del /s`, fork bombs, `shutdown`, Ansible, `systemctl`, `ssh`, package installs, and similar, matched anywhere in the command) is checked before both lists and cannot be overridden by configuration. Per-agent overrides of the same two keys are honoured where configured.

   Entries match by prefix: `git` also allows `gitk`, and `curl` in `deny` also blocks `curlie`. Even an allowed command is refused when it contains a shell metacharacter (`;` `&` `|` `$` `>` `<` `` ` `` `\` `!` `(` `)` `{` `}` `[` `]` `=` `#` `^` `%` `@` or a newline), so pipes, redirection and chaining never run, and neither does a command with an argument such as `git log --format=%H`.

   > [!WARNING]
   > GoClaw logs a one-line `unknown setting` warning for configuration keys it does not recognise and then continues to boot. A misspelled key is ignored, so its setting falls back to the default: a misspelled `deny` denies nothing beyond the hard-coded list. After editing this section, check the startup log for that warning before treating the restriction as applied.

2. **Path Sandboxing**: File read/write sandboxing is enforced in code, not behind a configuration flag — there is no setting that disables it. `resolveFilePath` (`internal/agent/tools.go`) rejects any path that resolves outside the agent workspace or the primary workspace, including absolute paths and `../..` traversal. System instruction files (`SOUL.md`, `AGENTS.md`, `TOOLS.md`, `IDENTITY.md`, `USER.md`) are additionally gated to privileged callers. Both checks sit in the shared resolver, so every tool that resolves a path inherits them rather than reimplementing them.
3. **Human-in-the-Loop (Approval Gates)**: A SOP step can be marked `requires_approval: true`. The run then waits until a human administrator approves or rejects the step: in a chat, through the agent's `sop_approve` tool (refused to anyone else, and to SOP, cron and webhook runs), or with `POST /api/sop/approve` and the gateway token. There is no Web UI control or CLI command for it. A notice goes to the channel set in `sop.alerts` (else only to the log), and a step not decided within 1 hour fails. Approval gates exist for SOP steps only, not for individual tool calls; for those, use the allow/deny lists above or an admin hook that blocks a call.
4. **Loop & Token Breakers**: Autonomous execution loops are guarded by circuit breakers (`loop_detector.go`) that terminate execution if repetitive tool invocations are detected.

---

## 5. Security Event Auditing & SIEM Integration

GoClaw streams structured telemetry and security audit logs to enterprise SOC / SIEM platforms:
- **OpenTelemetry (OTel)**: Native export of the gateway's logs over OTLP/HTTP (`logging.otel`) to any OTLP collector (Splunk, Datadog, Grafana, Uptrace). Traces and metrics are not exported: agent traces (steps, tool calls, timings) are kept in the `agent_traces` table and shown on the Web UI's Traces page.
- **Webhook Audit Logs**: Inbound webhook requests and verification statuses are recorded in structured format (`internal/gateway/access_log.go`).
- **Syslog / Journald**: Systemd journal logs capture all authentication attempts, pairing token generations, and configuration changes. A config reload logs each changed key with credential values masked. The Web UI authenticates with one shared gateway token, so a change is recorded by key and time but not attributed to a person.

================================================================================
# FILE: https://goclaw.ulab.mn/docs/architecture/sbom.md
================================================================================

# Software Bill of Materials (SBOM) & Supply Chain Security

This document outlines the **Software Bill of Materials (SBOM)** architecture in GoClaw, supported standards, distribution mechanisms, and Information Security (InfoSec) placement controls.

---

## 1. Overview & Objectives

Enterprise and pilot customers (e.g. banking, fintech, government, and telecommunications) require a verifiable Software Bill of Materials (SBOM) to comply with cybersecurity frameworks such as ISO/IEC 27001, PCI-DSS, NIST SP 800-218 (SSDF), and US Executive Order 14028.

GoClaw's SBOM infrastructure satisfies the following objectives:
1. **Continuous Vulnerability Management**: Enables automated dependency auditing via vulnerability scanners such as Grype, Trivy, Snyk, and OWASP Dependency-Track.
2. **Rapid Incident Response (0-Day Triage)**: Instantly determines whether GoClaw utilizes a vulnerable library when critical zero-day vulnerabilities (e.g. Log4j, XZ backdoor, or Go `net/http` DoS vectors) emerge.
3. **Supply Chain Integrity**: Records, for every dependency, the `h1:` module checksum the build actually linked, in the same form `go.sum` stores it — so an auditor can verify the inventory against the module graph with `go mod verify`. The SBOM files themselves are covered by SHA-256 digests in `checksums.json`.
4. **Dependency Inventory for License Review**: Provides the exact module path and version of every linked component, which is the input a license review needs. **The SBOM does not assert a license per dependency** — in SPDX those fields are `NOASSERTION`, and CycloneDX leaves the license field out. See "License data" below.

---

## 2. Information Security (InfoSec) Threat Model & Placement Controls

Exposing software dependency inventories naively can introduce security risks. GoClaw implements a 3-tier placement and access model to mitigate these threats:

```mermaid
flowchart LR
    subgraph SBOM["GoClaw SBOM Architecture"]
        direction LR
        T1["1. Release / Supply Chain
• goclaw.ulab.mn/releases/
• Current release only
• SHA256 in checksums.json
• CycloneDX 1.5 + SPDX 2.3"]
        T2["2. Live Gateway
• /api/admin/sbom
• AuthMiddleware
• Token Protected
• Anti-Recon"]
        T3["3. Offline CLI
• goclaw sbom
• Zero Network
• Binary Built-in
• Pipe to Grype"]
    end
```

### A. Anti-Reconnaissance Control
- **Threat**: Publishing an unauthenticated SBOM endpoint on a public server exposes exact versions of all internal libraries and database drivers. Attackers can crawl this endpoint to identify unpatched or zero-day CVEs before administrators can update.
- **Control**: On the live gateway, SBOM access requires a valid gateway token via **`GET /api/admin/sbom`** (and `/api/sbom`, the same handler), guarded by `AuthMiddleware`.
  - Unauthenticated requests receive `401 Unauthorized`.
  - The endpoint is never exposed publicly without valid gateway credentials.
  - Note that GoClaw has a single gateway token rather than per-role credentials: the `/admin/` path segment is naming, not a second privilege tier. Anyone who can open the dashboard can read the SBOM, so treat the token as the control boundary.

### B. Supply Chain Integrity
- **Threat**: Attackers or intermediaries could tamper with distributed SBOM manifests or distribute mismatched inventories.
- **Control**: Official release SBOMs are generated in the GitHub Actions release workflow from the release's linux/amd64 binary. Their SHA-256 digests are published in `checksums.json`, a JSON object that maps each asset name to its digest. It is not in `sha256sum` format, so `sha256sum -c` cannot read it: compare the digest as in [Release Distribution & Checksum Verification](#3-release-distribution--checksum-verification).
- **Limits**: `checksums.json` is served from the same host as the files it covers, so a match shows that a download is complete and uncorrupted, not who produced it. The SBOMs and `checksums.json` are not signed. The macOS binaries are code-signed when the release has a signing certificate configured; build provenance attestations for the binaries are optional and off unless enabled for the repository.

### C. Air-Gapped & Offline Inspection
- **Threat**: Security teams in air-gapped or restricted networks cannot query remote servers or download external scanning dependencies.
- **Control**: The `goclaw sbom` CLI command extracts the SBOM directly from the Go binary's embedded runtime metadata (`runtime/debug.ReadBuildInfo()`). It requires zero internet access and executes in milliseconds.

### D. License Data — What Is and Is Not Included

GoClaw's own license is declared in both formats: CycloneDX carries it as the
metadata component license, SPDX as `LicenseRef-GoClaw-Binary-License` with a matching
`hasExtractedLicensingInfos` entry.

**Per-dependency licenses are not asserted.** Go's build metadata
(`runtime/debug.ReadBuildInfo`) carries module paths, versions and checksums, but no
license information, and GoClaw does not ship a license database to infer it. In SPDX
every dependency therefore reports `NOASSERTION`, which is the SPDX term for "not
evaluated" — it is not a claim that a component is unlicensed. In CycloneDX a
dependency has no `licenses` field at all.

For license review, use the SBOM as the authoritative component inventory and pair it
with `THIRD_PARTY_NOTICES.md`, or run a dedicated license scanner over the module graph.
An exhaustive per-release license bundle is available from security@ulab.mn.

### E. Module Checksums

Each dependency carries the `h1:` value recorded in `go.sum` — as the `go:mod:h1`
property in CycloneDX, and as an `OTHER` external reference in SPDX. This is a Go
*dirhash* (a hash over the module's file listing), **not** the SHA-256 of a downloadable
artifact, so it is deliberately not published under a `SHA-256` label that no verifier
could reproduce. It is checkable directly against `go.sum` and `go mod verify`.

---

## 3. Supported Formats

| Format | Specification | Primary Use Case | Media Type |
|---|---|---|---|
| **CycloneDX 1.5** | OWASP Standard | Vulnerability scanning (Grype, Trivy, Snyk, Dependency-Track) | `application/vnd.cyclonedx+json` |
| **SPDX 2.3** | ISO/IEC 5962:2021 | Procurement documentation, dependency inventory for legal review | `application/spdx+json` |
| **Summary** | Human-Readable Text / JSON | Operator inspections, terminal review, Web UI dashboard | `application/json` / `text/plain` |

---

## 4. Usage Guide

### 1. Offline Inspection via CLI

```bash
# View human-readable dependency summary
goclaw sbom

# Export CycloneDX 1.5 JSON for vulnerability scanners
goclaw sbom --format cyclonedx --output goclaw-sbom.cdx.json

# Export SPDX 2.3 JSON
goclaw sbom --format spdx --output goclaw-sbom.spdx.json

# Pipe directly into Anchore Grype for instant scanning (zero-file workflow)
goclaw sbom -f cyclonedx | grype

# Scan using Aqua Trivy
goclaw sbom -f cyclonedx -o sbom.json && trivy sbom sbom.json
```

### 2. Fetching via Authenticated Admin API

On a running gateway instance, operators or automated compliance agents can retrieve the SBOM with a bearer token:

```bash
# Download CycloneDX 1.5 JSON
curl -H "X-Goclaw-Token: " \
  "http://localhost:8080/api/admin/sbom?format=cyclonedx" \
  -o goclaw-sbom.cdx.json

# Download SPDX 2.3 JSON
curl -H "X-Goclaw-Token: " \
  "http://localhost:8080/api/admin/sbom?format=spdx" \
  -o goclaw-sbom.spdx.json

# Direct browser download with attachment header
http://localhost:8080/api/admin/sbom?format=cyclonedx&download=1
```

### 3. Release Distribution & Checksum Verification

Only the current release is served at `https://goclaw.ulab.mn/releases//`;
`/releases/latest.json` names it (for example `{"version":"v0.0.96",…}`). A request
for an earlier version's files there returns 404. Every release from v0.2.2 on also
stays at `https://storage.googleapis.com/goclaw-releases/releases//`, the
same files and `checksums.json`, so the commands below work there with that
`BASE_URL`. The files of releases before that are available from security@ulab.mn.

```bash
VERSION=$(curl -fsSL https://goclaw.ulab.mn/releases/latest.json \
  | python3 -c 'import json,sys; print(json.load(sys.stdin)["version"])')
BASE_URL="https://goclaw.ulab.mn/releases/${VERSION}"
SBOM="goclaw-${VERSION}-linux-amd64-sbom.cdx.json"

# Download SBOM and checksums manifest
curl -fsSL "${BASE_URL}/${SBOM}" -o "${SBOM}"
curl -fsSL "${BASE_URL}/checksums.json" -o checksums.json

# Verify cryptographic integrity
EXPECTED_SHA=$(grep -o "\"${SBOM}\"[[:space:]]*:[[:space:]]*\"[^\"]*" checksums.json | grep -o '[^"]*$')
ACTUAL_SHA=$(shasum -a 256 "${SBOM}" | awk '{print $1}')

if [ "$EXPECTED_SHA" = "$ACTUAL_SHA" ]; then
    echo "✅ SBOM Integrity Verified: SHA-256 matches."
else
    echo "❌ Integrity Mismatch!"
    exit 1
fi
```

### 4. Integration with OWASP Dependency-Track

Automate continuous supply-chain monitoring by posting release SBOMs to your internal Dependency-Track server:

```bash
curl -X "POST" "https://dtrack.corp.example.com/api/v1/bom" \
     -H "X-Api-Key: $DTRACK_API_KEY" \
     -H "Content-Type: multipart/form-data" \
     -F "projectName=GoClaw" \
     -F "projectVersion=${VERSION}" \
     -F "autoCreate=true" \
     -F "bom=@goclaw-sbom.cdx.json"
```

---

## 5. Codebase Structure

- `internal/sbom/sbom.go`: Core collector, module dependency model, and validation of Go `h1:` module checksums.
- `internal/sbom/cyclonedx.go`: OWASP CycloneDX 1.5 JSON serializer.
- `internal/sbom/spdx.go`: Linux Foundation SPDX 2.3 JSON serializer.
- `internal/sbom/summary.go`: Component categorizer and human-readable terminal formatter.
- `internal/cli/cmd_sbom.go`: CLI command implementation for `goclaw sbom`.
- `internal/gateway/handler_sbom.go`: Authenticated API handler for `/api/admin/sbom` and `/api/sbom`.
- `web/js/views/system.js` & `web/index.html`: Admin Web UI card for viewing and downloading SBOMs.
- `.github/workflows/release.yml`: Release CI/CD automation generating official CycloneDX and SPDX assets.

================================================================================
# FILE: https://goclaw.ulab.mn/docs/architecture/vulnerability-management.md
================================================================================

# Vulnerability Management & VEX Architecture

This guide describes GoClaw's vulnerability assessment lifecycle, how `govulncheck` is used for call-graph analysis, and how to consume **OpenVEX** statements to eliminate false-positive security alerts.

---

## 1. Vulnerability Assessment Lifecycle

Traditional software security tools rely on naive package version matching. If an application imports a library with a known CVE, the scanner reports a vulnerability even if the vulnerable function is never invoked or compiled into the binary.

GoClaw solves this problem using a 3-pillar supply-chain security architecture:
1. **SBOM (Software Bill of Materials)**: Exhaustive machine-readable dependency inventory (CycloneDX 1.5 and SPDX 2.3).
2. **govulncheck (Call-Graph Reachability Analysis)**: Go's official vulnerability tool that performs static analysis on symbol execution paths.
3. **OpenVEX (Vulnerability Exploitability eXchange)**: Machine-readable declarations explaining why specific reported vulnerabilities do not affect GoClaw.

---

## 2. govulncheck Static Analysis

`govulncheck` analyzes symbol call graphs against the official Go vulnerability database (`vuln.go.dev`).

### Key Differences: Naive Scanner vs. govulncheck
| Feature | Naive Scanner (Trivy / Grype basic) | govulncheck (GoClaw standard) |
|---|---|---|
| **Analysis Level** | Module name & version string only | AST, Symbol, and Call-Graph reachability |
| **False Positive Rate** | **High** (reports CVEs in unused functions) | **Near Zero** (only flags executed symbols) |
| **Runtime Impact** | None | None (runs at CI/build time only) |

### Generating govulncheck Reports
`govulncheck` is automatically executed during the release build pipeline. You can also run it locally:

The release pipeline pins the scanner version rather than tracking `@latest`, so that a
report can be reproduced later from the same inputs. Use the same pin locally (it is
`GOVULNCHECK` in `.github/workflows/release.yml`; v1.1.4, the earlier pin, crashes on Go 1.27):

```bash
# Run human-readable vulnerability inspection
go run golang.org/x/vuln/cmd/govulncheck@v1.8.0 ./...

# Export machine-readable JSON for compliance archives
go run golang.org/x/vuln/cmd/govulncheck@v1.8.0 -json ./... > govulncheck-report.json
```

In its default text mode, `govulncheck` exits `3` when it finds reachable
vulnerabilities and `0` when it does not; any other non-zero exit is a tool or build
failure, and the report it produced should not be trusted. With `-json` it exits `0`
whether or not it finds anything, so read the findings from the report itself, not from
the exit code.

---

## 3. OpenVEX (Vulnerability Exploitability eXchange)

### What is VEX?
When an enterprise security scanner audits GoClaw's SBOM, it may flag transitive library CVEs. Instead of lengthy email exchanges and manual review meetings, GoClaw publishes an official **OpenVEX** file (`goclaw-${VERSION}-vex.json`) alongside each release.

VEX lets compliance scanners filter out non-exploitable findings using the fixed set of
justifications defined by the OpenVEX specification. The definitions below are the
specification's, not GoClaw's — a VEX that reinterprets them is a VEX an auditor can reject:

- `component_not_present`: The vulnerable component is not in the product at all.
- `vulnerable_code_not_present`: The component is present, but the vulnerable code was removed or never shipped.
- `vulnerable_code_not_in_execute_path`: The vulnerable code is present but is never invoked.
- `vulnerable_code_cannot_be_controlled_by_adversary`: The vulnerable code executes, but an attacker cannot influence the input that reaches it.
- `inline_mitigations_already_exist`: The product contains a built-in protection that prevents exploitation, which the attacker cannot subvert and the operator cannot disable.

> [!IMPORTANT]
> A statement is only worth publishing if it survives being checked against the
> `govulncheck` report shipped beside it. Where a vulnerability is reachable and a
> fixed upstream release exists, GoClaw upgrades the dependency instead of writing a
> justification for it.

### Sample OpenVEX Statement (`security/vex.json`)
```json
{
  "@context": "https://openvex.dev/ns/v0.2.0",
  "@id": "https://goclaw.ulab.mn/vex/release-advisory",
  "author": "ULAB.mn Security Team ",
  "timestamp": "2026-09-21T00:00:00Z",
  "version": 2,
  "statements": [
    {
      "vulnerability": {
        "name": "GO-2026-6452",
        "description": "Panic via negative shared-string index in github.com/xuri/excelize"
      },
      "products": [
        "pkg:golang/ulab.mn/agents/goclaw"
      ],
      "status": "not_affected",
      "justification": "inline_mitigations_already_exist",
      "impact_statement": "The vulnerable code is reachable ... the panic is recovered in WorkerPool.executeTask (internal/agent/worker_pool.go), so the residual impact is the failure of the single spreadsheet call that triggered it."
    }
  ]
}
```

---

## 4. How Enterprise InfoSec Teams Consume Release Artifacts

The audit assets of the current release are hosted under
`https://goclaw.ulab.mn/releases/${VERSION}/`, where `VERSION` is the version named in
`https://goclaw.ulab.mn/releases/latest.json` (for example `v0.0.96`). Only the current
release is served there: a request for an earlier version's files returns 404. The
files of earlier releases are available from security@ulab.mn.

1. **SBOM**: `https://goclaw.ulab.mn/releases/${VERSION}/goclaw-${VERSION}-linux-amd64-sbom.cdx.json` (and `…-sbom.spdx.json`)
2. **OpenVEX**: `https://goclaw.ulab.mn/releases/${VERSION}/goclaw-${VERSION}-vex.json`
3. **govulncheck**: `https://goclaw.ulab.mn/releases/${VERSION}/goclaw-${VERSION}-govulncheck.json` (JSON) and `goclaw-${VERSION}-govulncheck.txt` (text summary)
4. **Checksums**: `https://goclaw.ulab.mn/releases/${VERSION}/checksums.json`

### Scanning with Grype + VEX:
```bash
VERSION=$(curl -fsSL https://goclaw.ulab.mn/releases/latest.json \
  | python3 -c 'import json,sys; print(json.load(sys.stdin)["version"])')

# Scan SBOM with VEX suppression in a single command
grype "sbom:goclaw-${VERSION}-linux-amd64-sbom.cdx.json" --vex "goclaw-${VERSION}-vex.json"
```

### Scanning with Trivy + VEX:
```bash
trivy sbom "goclaw-${VERSION}-linux-amd64-sbom.cdx.json" --vex "goclaw-${VERSION}-vex.json"
```
With the VEX advisory supplied, a scanner suppresses the findings the advisory accounts for. It will not necessarily report a clean result: the scanner's own database may carry advisories GoClaw has not yet assessed, and suppression depends on the scanner matching the VEX product identifier to the SBOM. Treat the VEX as evidence to review alongside the `govulncheck` report, not as a guarantee of a green scan.

================================================================================
# FILE: https://goclaw.ulab.mn/docs/api-reference/gateway_endpoints.md
================================================================================

# Gateway API Reference

The GoClaw gateway exposes the Web UI, web chat over Server-Sent Events,
WebSocket chat, REST administration APIs, webhook ingress, media endpoints, and
an OpenAI-compatible chat API.

Routes are registered in `cmd/gateway/startup.go` (`registerAPIRoutes`, which
also mounts the trace routes of `internal/gateway/handler_traces.go`) and in
`cmd/gateway/main.go` (the Web UI's files, `/canvas/` and `/uploads/`), and
implemented under `internal/gateway`.

## Authentication

Every route on this page needs the gateway token, `gateway.auth.token`. The
gateway does not start without one. These are the exceptions:

| Route | How it is checked |
|---|---|
| `/health`, `/healthz`, `/ready`, `/readyz` | Not checked. |
| The Web UI's own files (`/`, its scripts, styles and images) | Not checked; every API call the page makes sends the token. |
| `/goclaw/hook/…` and the channel webhooks (`/hook/…`) | Each checks its sender's own credential. See [Webhook receiver](#webhook-receiver). |
| `/goclaw/view/` | The page's own view token (`?token=` or an `X-View-Token` header), or the gateway token. |
| `/api/app/*`, `/ws/app` | The App API's own credential. See [App API](#app-api). |

Send the token in any of these:

- an `X-Goclaw-Token: ` header;
- an `Authorization: Bearer ` header;
- a `?token=` query parameter, for clients that cannot set a header
  (`EventSource`, a browser WebSocket);
- an `X-Goclaw-Token` cookie. The gateway sets it when a browser opens the
  one-time login link it logs at startup (`Web UI: http://…/?auth_code=…`): the
  code is valid for 5 minutes, and for 30 seconds after its first use. The
  cookie lasts 7 days, and the gateway redirects to the same path without the
  code. The code is not the token. The cookie is taken only from the dashboard
  itself: a request to `/api/`, `/v1/` or `/plugins/` that carries it alone and
  that the browser marks as started by another site (`Sec-Fetch-Site:
  same-site` or `cross-site` — a page on another port of the same host is
  same-site) is answered `403`. Send the token in a header there instead.

A missing or wrong token is answered `401 {"error":"unauthorized"}`. When
`gateway.auth.token` is a `${secret:…}` reference the vault cannot resolve, every
request is answered `500`.

Requests that carry the token are not counted against `gateway.rate_limit`;
neither are health probes and the Web UI's files. When webhooks share the main
port (no `gateway.ingress.port`), `gateway.ingress.allow_ips` / `allow_preset`
apply to every route on it; loopback is always let through.

## Web Chat

### `POST /api/chat`

The Web UI's chat. It runs one message on the default agent as the admin (role
`system`) and streams the reply as Server-Sent Events.

```json
{
  "session_id": "admin-session",
  "content": "Hello",
  "attachments": []
}
```

| Field | Notes |
|---|---|
| `session_id` | Optional; defaults to `admin-session`. |
| `content` | The message (`message` is accepted as an alias). Required unless `attachments` has an item. |
| `attachments` | Files uploaded first with `/api/upload-workspace` or `/api/media/upload`, each as `{media_id, media_url, local_path, filename, mime_type, size}`. |

A message sent while an earlier one in the same session is still being answered
waits for it. If the request ends while it waits, the answer is `409`.

The response is `text/event-stream`. Each frame is `event: ` with a JSON
`data:` line:

| Event | Data |
|---|---|
| `token` | `content` (also `delta`): the next piece of the reply. |
| `tool_start` | `tool`: the tool the agent called. |
| `tool_result` | `tool`: the tool that finished. |
| `canvas` | `title`, `content` (HTML): a canvas artifact. |
| `buffer_reset` | The reply is being retried: discard the text received so far. |
| `error` | `error`: what went wrong. No `done` follows. |
| `done` | `content` (the whole reply), `attachments` (files the agent produced, served under `/canvas/`), `prompt_tokens`, `output_tokens`, `elapsed_ms`, `session_id`. |

A reset trigger (`session.reset_triggers`, by default `/new` and `/reset`)
clears the session, and `@config:`, `@notes:`, `@sop:` directives and
`/subagents` commands are answered without the model; each ends with `done`, or `error` when it fails.
The directives need an administrator, which this endpoint always is. If
the client disconnects, the agent still finishes (for up to 10 minutes) and the
reply is saved to the session.

## WebSocket

### `GET /gateway`

Upgrades to a WebSocket connection, for chat clients other than the Web UI (the
Web UI uses [`POST /api/chat`](#post-apichat)). When `gateway.allowed_origins`
is set, only those origins may connect.

Message envelope:

```json
{
  "id": "msg-123",
  "kind": "inbound",
  "session_id": "admin-session",
  "payload": {
    "channel": "web",
    "peer_id": "web-tab",
    "content": "Hello"
  },
  "timestamp": "2026-06-20T10:00:00Z"
}
```

Supported message kinds:

| Kind | Direction | Purpose |
|---|---|---|
| `inbound` | client -> gateway | User message and optional attachments. |
| `outbound_chunk` | gateway -> client | Streaming token chunk. |
| `outbound` | gateway -> client | Final assistant response. |
| `event` | gateway -> client | Tool lifecycle and buffer-reset events. |
| `canvas` | gateway -> client | Rich canvas artifact update. |
| `error` | gateway -> client | User-facing processing error. |

Payloads:

| Kind | Payload |
|---|---|
| `inbound` | `channel`, `account_id`, `peer_id`, `content`, and optional `attachments`, each `{media_id, media_url, local_path, filename, mime_type, size}`. |
| `outbound_chunk` | `content`: the next piece of the reply. |
| `outbound` | `content`, and `attachments` the agent produced (served under `/canvas/`). |
| `event` | `{"event_type":"tool_start","tool_name","args"}`, `{"event_type":"tool_result","tool_name","result","error","elapsed_ms"}` or `{"event_type":"buffer_reset"}` (discard the text received so far). |
| `canvas` | `title`, `content` (HTML). |
| `error` | `error`, and `ref_id`: the `id` of the message it answers. |

Sessions: on channel `web`, a `session_id` that starts with `admin-` is used as
given; any other value, or none, gives the connection its own session,
`admin-conn-<16 hex digits>`. Web messages run as the admin (role `system`). On
any other channel the `session_id` is used as given, and without one the
session is `p2p::`.

A connection's messages are handled one at a time. Reset triggers, directives
and `/subagents` commands work as in [`POST /api/chat`](#post-apichat). Off the
`web` channel a message runs with the role GoClaw resolves for its `channel` and
`peer_id`, and `@config:`, `@notes:` and `@sop:` need a peer paired as `admin`:
anyone else is told so and nothing is changed (the message and the refusal stay
in the session, as for any directive).

## Core Admin APIs

### `GET /api/stats`

Returns gateway health, provider/model information, session count, connected
channels, runtime ingress status, and version metadata.

### `GET /api/gateway`

Returns detailed runtime status, ingress settings, rate-limit state, and recent
access logs. Supports `?limit=N`.

### `GET /api/doctor`

Runs health checks and returns diagnostic results. With `?stream=1` the results
arrive as Server-Sent Events, one `check` event per check as it finishes and a
final `done` event with `{"total", "ok"}`.

### `GET /api/inspect`

Returns a combined diagnostic view with sessions, memory snippets, database
path, state directory, and server time.

### `GET|POST /api/config`

Reads or mutates active configuration. `GET` returns the whole configuration
keyed as `config.yaml` is (`tools.http.allow`), with secrets masked;
`GET ?key=ai.provider` returns a single key. `POST` supports mutation actions
such as `set`, `append`, and `remove`.

### `GET|POST|DELETE /api/secrets`

Manages the encrypted secret vault. Secret values are masked when listed.

### `GET|POST /api/integrations`

Lists, tests, saves and deletes the external services the tools use: GitLab,
SonarQube, Grafana, Graylog, NetBox, Uptrace, Zabbix, Wazuh and datasources.
`GET` returns each definition with credentials masked (`${secret:…}` references
are shown as written). `POST` takes `{"action", "type", "name", "config"}`:

- `test` probes the definition as the default agent and succeeds only when the
  credentials are accepted. `warnings` names agents that override the enclosing
  block (and so will not see it) or whose HTTP policy blocks the host.
- `save` merges `config` onto the stored definition — a key left out keeps its
  stored value, and the mask keeps a stored secret only while the URL or host is
  unchanged. `create: true` refuses an existing name (`409`);
  `store_secrets: true` moves typed credentials into the vault as
  `integrations...` and writes a `${secret:…}` reference.
- `delete` removes the definition; a datasource named by
  `org_directory.sql.datasource` is refused (`409`).

### `GET|POST /api/pairing`

Lists pending/approved channel pairings and approves, revokes, or updates user
roles. `GET` takes an optional `?channel=`. `POST` takes an `action`:
`{"action":"approve","code","role"}` approves a pairing code,
`{"action":"revoke","user_id"}` removes a user, and
`{"action":"update_role","user_id","role"}` changes a user's role. Approvals made
here take effect in the running gateway at once.

## Sessions

### `GET|POST|DELETE /api/sessions`

- `GET`: Lists persisted sessions, each as `{id, title, message_count, last_seen}`.
  The messages themselves are served by [`GET /api/session`](#get-apisession).
- `POST {"action":"create","id":"admin-name","name":"Label"}`: Creates an admin
  Web UI session. The `id` must start with `admin-`; without one it is
  `admin-`. `name` is optional.
- `POST {"action":"rename","id":"admin-name","name":"New label"}`: Renames a
  session.
- `POST {"action":"clear"}`: Deletes all sessions; the response says how many.
- `DELETE ?id=`: Deletes one session.

Session creation is persisted immediately, including empty sessions, so Web UI
refreshes do not make newly created sessions disappear.

Each of these changes waits until a message being answered in that session is
done, so the reply cannot write the old session back over the change. A request
that ends while it waits is answered `503`; `clear` then reports how many
sessions it had already deleted.

### `GET /api/session`

Returns paginated message history for one session.

Query parameters:

| Parameter | Required | Notes |
|---|---:|---|
| `id` | yes | Session ID. |
| `limit` | no | Defaults to `50`; a value that is not a number from 1 to `500` gives `50`. |
| `offset` | no | Offset from the newest message. |

The response includes `history`, parallel `meta`, `total`, `limit`, `offset`,
and `username`. Web chat attachment previews are stored in `meta[].attachments`.

## Memory and Search

### `GET /api/memory`

Returns memory records grouped for the Web UI.

### `GET|POST /api/memory/core`

Reads or updates an agent's core instruction: `?agent=`, the default
agent's when left out. The ID is used as given: where the default agent has an
ID of its own and no agent is configured as `main`, `?agent=main` reads an
instruction stored under `main`, which no run follows.

### `GET|POST /api/memory/bootstrap-file`

Reads or updates controlled bootstrap files such as `AGENTS.md`, `TOOLS.md`, and
other admin-managed prompt files, in the workspace of `?agent=` (the default
agent's when left out).

### Memory maintenance endpoints

| Endpoint | Purpose |
|---|---|
| `/api/memory/replay` | Replay archived/session memory into the memory engine. |
| `/api/memory/reindex` | Rebuild memory indexes. |
| `/api/memory/stats` | Return memory statistics. |
| `/api/memory/prune` | Prune old or low-value memory. |
| `/api/memory/dedupe` | Deduplicate memory records. |
| `/api/memory/forget` | Delete selected memory. |
| `/api/memory/clear` | Clear memory data. |
| `/api/memory/relearn` | Trigger relearning workflows. |
| `/api/memory/revise` | `POST {"dry_run"}`: weigh each learned fact or preference against older ones on the same subject and retire those it replaces (tagged `superseded`, not deleted). `goclaw memory revise` calls it. |
| `/api/memory/revisions` | `GET ?run=&record=&limit=`: the recorded decisions that one learned record replaces another, newest first (`goclaw memory revisions`). |
| `/api/memory/revisions/undo` | `POST {"record":""}` or `{"run":""}`: restore what a decision retired; an undone decision is not made again (`goclaw memory revisions undo`). |
| `/api/memory/backfill-metadata` | Populate missing memory metadata. |
| `/api/memory/notes` | Read or update agent notes. |
| `/api/memory/export` | Export memory. |
| `/api/memory/import` | Import memory. |
| `/api/memory/graph` | Return knowledge graph data. |

### `GET /api/search`

Searches memory and archived material. Common parameters include `q` and
`source`.

## Agents, Tools, and RAG

| Endpoint | Purpose |
|---|---|
| `GET /api/agents` | List configured agents and bindings. Each agent carries `tool_policy`: its effective tool, shell, HTTP and git policy and the keys it overrides. |
| `GET\|POST\|DELETE /api/skills` | `GET` lists installed and built-in skills. `POST` takes an `action`: `install`, `toggle` (turn a skill on or off), `verify` or `save`. `DELETE ?name=` removes a user-installed skill. |
| `GET\|POST /api/skills/quality` | A skill's quality score: `GET ?name=` returns the cached score or evaluates it; `POST {"name","force":true}` evaluates it again. |
| `GET /api/tools` | List registered tools. |
| `POST /api/rag-test` | `{"query"}`: show which tools and skills Tool RAG and Skill RAG would select for it, with their scores. |
| `GET\|POST /api/mcp` | `GET` lists the configured MCP servers. `POST` takes an `action`: `test` connects a draft definition and lists its tools, `call` runs one tool, `save` writes the definition into `config.yaml`, `delete` removes it. |

## Automation

| Endpoint | Purpose |
|---|---|
| `/api/cron` | `GET` lists cron jobs with their next run. `POST {"action","id"}` with `run`, `enable`, `disable`, `update` (with `"schedule"`) or `delete`; a system job (`sys-…`) cannot be deleted, only disabled. Jobs are created with the agent's `cron` tool; this API has no create. |
| `/api/sop` | Manage SOP definitions. `GET` lists them, or with `?id=` returns one with `cron_schedule`, its cron job's schedule. `POST` saves one: on an existing SOP, a field the body leaves out (top level, or in a step with the same `id`) keeps its stored value, and a changed definition archives the previous version. A `cron_schedule` in the body creates or reschedules the SOP's cron job; one that cannot be applied is returned as `warning`. `DELETE ?id=` removes one. |
| `/api/sop/executions` | List SOP executions. |
| `/api/sop/run` | `POST {"id"}`: start a run of an SOP in the background; the answer has its `execution_id`. A paused SOP, or one overdue for review, is answered `409` with the reason. |
| `/api/sop/approve` | `POST {"job_id","action":"approve"\|"reject","reason"}`: approve or reject an SOP step that is waiting for approval. A step nobody decides fails after 1 hour. |
| `/api/sop/validate` | Validate an SOP definition. |
| `/api/sop/channels` | Pin SOP notification channels. |
| `/api/capability-requests` | `GET` lists capability requests; `?status=` filters them. |
| `/api/capability-requests/resolve` | `POST {"id","status","resolved_by","note"}`, with `status` `resolved` or `rejected`. |

## Webhooks and Plugins

| Endpoint | Purpose |
|---|---|
| `POST /api/webhook` | Generic inbound message endpoint. See [`POST /api/webhook`](#post-apiwebhook). |
| `GET /api/hooks` | Registered gateway hook events (built-in and Go extensions) with their handlers. The Hooks page's rules are `/api/hooks/rules` (see [Other Routes](#other-routes)). |
| `GET\|POST /api/webhooks` | List webhook routes (`GET`); create, update, enable, disable or delete a stored route (`POST` with `action`). See [Webhook routes API](#webhook-routes-api). |
| `GET\|DELETE /api/plugins` | `GET` lists the installed agent plugins (the Plugins page); `DELETE ?name=` turns one off for every agent and removes it. |
| `GET /api/extensions` | Discovered Go/OpenClaw extensions (`extensions/` directories) and which are active. |
| `GET /api/audit` | Query webhook audit logs: `?channel=`, `?event=`, `?user=`, `?limit=` (default `100`, at most `500`). With `?kind=admin`, the administrative trail (SOP deletes, channel pins) instead: `?action=`, `?actor=`, `?resource=`, `?limit=`. |
| `POST /goclaw/hook/[/]` | The webhook receiver outside systems (GitLab, GitHub, Graylog, Zabbix, …) deliver events to. See [Webhook receiver](#webhook-receiver). |

When the Teams or Facebook channel, or the Telegram channel in webhook mode, is
enabled, its webhook is mounted on the webhook listener at `/hook/msteams`,
`/hook/facebook` or `/hook/telegram`, or at `/hook//`
when `gateway.webhook_secret_slug` is set.

### `POST /api/webhook`

Runs one message on the default agent and answers with the whole reply. It needs
the gateway token on every port, a separate `gateway.ingress.port` included:

```json
{"channel": "discord", "account_id": "discord", "peer_id": "u_123456", "content": "Hello"}
```

```json
{"content": "Hi! How can I help?"}
```

The session is `p2p::`, and the message runs with the
role GoClaw resolves for that `channel` and `peer_id`, as for a message from that
channel, or, when `account_id` is `cron:` (as cron agent jobs send it),
with that user's role. With
`"channel": "web"` it runs in `admin-session` as the admin (role `system`).
With `"subject_ref": ""` (a cron `agent …` job with a subject sends
it) the message runs as that person instead, at role `user` with a
delegation, on every channel, `web` included: `account_id`, the peer's pairing
and `memory.write_access` do not set its role, so directives are refused. Its
session is `p2p:::for:`.
`goclaw message  ` posts here. A message sent while an earlier one
in the same session is still being answered waits for it; if the request ends
while it waits, the answer is `409`. Reset triggers, directives and `/subagents`
commands work as in [`POST /api/chat`](#post-apichat), except that off the `web`
channel `@config:`, `@notes:` and `@sop:` need that role to be `admin`.
`goclaw message ` sends as peer `cli`, which is paired with no one, so
a directive goes through it only on `web` (the default). An agent error is
answered `500` with the error text.

### Webhook routes API

`/api/webhooks` needs the gateway token (`X-Goclaw-Token: ` or
`Authorization: Bearer `).

`GET` lists every route: the stored ones, and the `config.yaml` ones with
`"source": "config"` and `"id": 0`, which cannot be changed here. Secrets are
masked; `secret_set` says whether the route has a secret or token at all.
`hit_count`, `last_hit_at` and `last_error` are kept for stored routes only.
`last_error` is the latest refusal or drop (its time, status and why: a
delivery refused, or taken and then dropped unrun); the next delivery that gets
through clears it.

`POST` takes a JSON body with an `action`:

```bash
curl -X POST https:///api/webhooks \
  -H "Authorization: Bearer $GOCLAW_TOKEN" -H "Content-Type: application/json" \
  -d '{"action":"create","type":"graylog","key":"payments-errors","agent":"main","sop_id":"incident-triage"}'
```

```json
{"ok": true, "id": 7, "agent": "main", "generated_secret": "whsec_…"}
```

| Field | Actions | Meaning |
|---|---|---|
| `action` | all | `create`, `update`, `enable`, `disable` or `delete`. |
| `id` | `update`, `enable`, `disable`, `delete` | The stored route's `id` from `GET`. |
| `type` | `create` (required), `update` | First path segment of `/goclaw/hook//`. On `update`, empty keeps the stored type. |
| `key` | `create`, `update` | Second path segment. Empty makes the type's route for any key that has no route of its own. |
| `agent` | `create`, `update` (required) | Agent ID or name, any case, or `*` for every agent. An unknown agent is refused with `400`, naming the agents there are. |
| `secret`, `token` | `create`, `update` | On `create`, leave both out to have a `whsec_…` secret generated: it is returned once, as `generated_secret`, and not shown again. On `update`, an empty value keeps the stored one. |
| `events` | `create`, `update` | Comma-separated event names, e.g. `"push,merge_request"`; empty takes every event. |
| `sop_id`, `batch_mode`, `batch_window` | `create`, `update` | Run a SOP per event, or per batch. On `update`, an empty `batch_mode` or a `batch_window` of `0` keeps the stored value. |

The fields mean what they mean in `config.yaml`: see
[Webhook Routes](../user-guide/configuration.md#webhook-routes).

- `update` writes `type`, `key`, `agent`, `events` and `sop_id` as sent, so send
  the route's current values along with the change: an empty `key`, `events` or
  `sop_id` clears it. It does not change whether the route is paused.
- `disable` pauses a route: its deliveries are answered `404`, and each is noted
  on the route. `enable` resumes it.
- `delete` removes the route and what was set up for it on the sender's side: the
  GitLab project hook `goclaw gitlab hook` registered, or the Graylog event
  definition and notification `graylog_alert` created. The response's `notice`
  says what was removed, or what is left to delete by hand.

### Webhook receiver

`POST /goclaw/hook/[/]` is where outside systems deliver events. It is
served on the webhook listener: `gateway.ingress.port` when that is set, else the
main port. The route is the one for this type and exact key, else the type's route
without a key; between a stored route and a `config.yaml` route for the same key,
the stored one. How each type authenticates (GitLab token, GitHub HMAC, otherwise
`X-Token`, `?token=` or an `X-Signature` HMAC) is in
[Webhook Routes](../user-guide/configuration.md#webhook-routes).

| Status | When |
|---|---|
| `200` `{"status":"ok", …}` | The event was taken: a SOP run started (or the event queued, for a bulk route or behind the run in progress for the same object), or the agent will answer it. `dispatched` counts the agents it reached. |
| `200` `{"status":"filtered"}` | The route's `events` leave this event out. Nothing runs. |
| `200` `{"status":"duplicate"}` | The route took a delivery with this ID before (a retry, a redelivery). Nothing runs again. |
| `200` `{"status":"test"}` | Graylog's test notification, on a `graylog` route. Nothing runs. |
| `401` | The credential is missing or does not match, or the route has none; the body says which. |
| `403` | The sender's IP is not on `gateway.ingress.allow_ips` / `allow_preset` (only when an allowlist is set; loopback is always let through). |
| `404` | No route for the type and key, or the route is paused. |
| `409` | The route's SOP cannot run: it was deleted or disabled, or it is overdue for review. |
| `413` | The body is larger than 4 MB. |
| `429` | The sender's IP is over `gateway.rate_limit`. |
| `500` | The event was not taken, for example the SOP run could not be started. Send it again. |
| `503` | Too many deliveries are in progress, or 500 events already wait for the same object's run (`Retry-After: 30`), or another request is taking this same delivery right now (`Retry-After: 10`); sent again, it is a repeat or it runs. Also when the delivery's ID could not be recorded (`Retry-After: 10`): run unrecorded, a resend could not be told from it. |

A refused delivery to a stored route is noted on the route (`last_error`, and on
the Webhooks page); for a `config.yaml` route, only the gateway log has it
(`[webhook]` lines). The `503` for a delivery another request is taking is only
in the gateway log: sent again, that delivery is a repeat or runs, as the other
request ended.

## Media and Canvas

### `POST /api/media/upload`

Uploads a file, sent as the multipart form field `file`, to the media store
(`/media`) and returns an ID, URL, filename, MIME type, size, and
creation time.

### `GET|HEAD /media/`

Serves a media-store object. Images, audio, video, PDF and plain text are served
inline with their stored content type; anything else (HTML, SVG, XML, scripts,
unknown types) is served as a download (`application/octet-stream`).

### `POST /api/upload-workspace`

Uploads a file, sent as the multipart form field `file`, directly into
`workspace/uploads`. Web chat uses this endpoint so tools can access uploaded
files by workspace-relative path.

The response includes:

```json
{
  "id": "timestamp-filename.png",
  "local_path": "uploads/timestamp-filename.png",
  "filename": "filename.png",
  "url": "/uploads/timestamp-filename.png",
  "mime_type": "image/png",
  "size": 12345
}
```

### `GET /uploads/`

Serves files from `workspace/uploads`. Images, audio, video, PDF, `.txt` and
`.log` files are served inline; any other file, or one without an extension, is
served as a download.

### `GET /api/canvas`

Lists canvas artifacts, each as `{name, size, mod_time}`.

### `GET /canvas/`

Serves generated canvas artifacts from `/canvas`, inline or as a
download by the same rule as `/uploads/`. Artifacts are removed after 48 hours.

### `GET /goclaw/view/`

Serves an HTML page the agent made (`generate_html`). It needs the page's view
token (`?token=` or an `X-View-Token` header) or the gateway token.
`/goclaw/view/` without a slug lists the pages and needs the gateway token.

The page is served sandboxed (`Content-Security-Policy: sandbox allow-scripts
…`, without `allow-same-origin`), so the browser treats it as an origin of its
own: its scripts run, but no dashboard cookie goes with its requests, and it
cannot post forms, open frames, use `fetch`, or read `localStorage` or
`document.cookie`.

## Messaging

### `POST /api/send`

Sends a message to a connected channel, without the agent:

```json
{"channel": "discord", "target": "", "text": "Hello"}
```

All three fields are required. It needs the gateway token and accepts only
requests from a loopback address (`127.0.0.1`, `::1`); any other is answered
`403`. A channel that is not connected is answered `404`, with the connected
channels in `available`.

## Token Metrics

### `GET /api/metrics/tokens`

Returns token usage summaries. Optional filters include `agent_id`, `channel`,
`provider`, `model`, `date_from`, `date_to`, and `limit` (1 to `500`, default
`50`).

## Traces and Logs

| Endpoint | Purpose |
|---|---|
| `GET /api/traces` | Agent traces, newest first. Filters: `session_id`, `agent_id`, `channel`, `status`, `tag`, `sop_id`, `from` and `to` (unix seconds, RFC 3339 or `YYYY-MM-DD`), `min_tokens` (prompt and output together), `before` (unix seconds, for paging) and `limit` (default `50`, at most `200`). `sort=tokens` lists the traces that used the most tokens first, paged by `offset` instead of `before`. A SOP run's trace is filed under the agent that ran it; `sop_id` finds a SOP's runs. A scheduled built-in or shell command, or a schedule whose SOP is gone, has no agent: find its job's runs by `session_id` (`cron:`); `GET /api/traces/` also gives the job as `cron_id`. |
| `GET /api/traces/stats` | Trace counts (total, done, error) and averages of duration, tool calls and tokens. |
| `GET /api/traces/` | One trace with its steps, tool calls and timings. |
| `GET /api/traces/.csv` | The same trace as a CSV download. |
| `POST /api/traces//feedback` | `{"score": 1 or -1, "comment", "user_id"}`: rate a trace. |
| `GET /api/logs` | The gateway log as Server-Sent Events, one line per event. |

## Other Routes

| Endpoint | Purpose |
|---|---|
| `GET /api/sbom`, `GET /api/admin/sbom` | The software bill of materials: `?format=cyclonedx` (default), `spdx` or `summary`; `?download=1` sends it as a file. See [SBOM](../architecture/sbom.md). |
| Web UI only: `/api/agents/profiles`, `/api/agents/profiles/draft`, `/api/agents/instructions`, `/api/agents/instructions/draft`, `/api/agents/persona/draft`, `/api/agents/constraints`, `/api/agents/preview`, `/api/hooks/rules`, `/api/plugins/{inspect,install,agents,approve,inputs,fetch,install-fetched,marketplaces,marketplaces/browse,check-updates,export}`, `/api/memory/wiki` and `/api/memory/wiki/{history,restore,distill,attachments,search}` | The Agents, Instructions, Hooks, Plugins and Wiki pages of the Web UI. Their request shapes follow those pages and are not described here. They need the gateway token like the rest. |

## App API

`POST /api/app/chat` and `GET /api/app/events` serve the Enterprise App API
channel. They check the app's own credential, not the gateway token, and
`/ws/app` answers `410 Gone`. See
[App API](../user-guide/channels/app-api.md).

## OpenAI Compatibility

### `POST /v1/chat/completions`

OpenAI-compatible chat completions endpoint. See
[OpenAI Compatibility](./openai_compatibility.md).

## Health Probes

### `GET /health`, `GET /healthz`, `GET /ready`, `GET /readyz`

Lightweight health probes that return `{"status":"ok","uptime_seconds":…}`.
These endpoints are intentionally unauthenticated. When `gateway.ingress.port`
gives webhooks their own listener, they are served there too.

================================================================================
# FILE: https://goclaw.ulab.mn/docs/api-reference/openai_compatibility.md
================================================================================

# OpenAI Compatibility Guide

The GoClaw Gateway exposes an OpenAI-compatible interface, allowing it to act as a drop-in proxy for tools like Cursor, Continue.dev, or any library that supports the OpenAI API.

## Endpoints

### `/v1/chat/completions`
**Method:** `POST`

Implements the standard OpenAI Chat Completions API.

**Key Features:**
- **Streaming:** Supports `stream: true` using Server-Sent Events (SSE). Besides the usual `chat.completion.chunk` frames, a stream can carry `data: {"buffer_reset":true}` (the reply is being retried: discard the text received so far) and `data: {"error":{"message":"…"}}`, which ends the stream with no finish chunk and no `data: [DONE]`.
- **Non-Streaming:** Returns a standard JSON response.
- **Message content:** Each message's `content` must be a string. Content sent as an array of parts (text and images) is refused with `400`.
- **Model Routing:** All requests go to the **Default Agent** configured in GoClaw. The `model` parameter is echoed back in the response and does not select an agent or a model. `max_tokens` and `temperature` are read and ignored.
- **Subagent Commands:** A last user message that starts with `/subagents` is a GoClaw subagent command, answered without the model: `/subagents spawn  `, `/subagents list`, `/subagents status `, `/subagents stop `.

**Example Request:**
```bash
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your-token" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "How many files are in the current directory?"}]
  }'
```

---

### `/v1/models`
**Method:** `GET`

Not implemented: the gateway answers `404`. Use the `default` model or any string, as all traffic goes to the default agent.

---

## Authentication

The gateway token is always required: the gateway does not start without `gateway.auth.token`. Provide it in one of the following ways:

- **Bearer Token:** `Authorization: Bearer `
- **Custom Header:** `X-GoClaw-Token: `
- **Query Parameter:** `?token=` (for browser clients that cannot set a header, such as `EventSource`)
- **Cookie:** the `X-Goclaw-Token` cookie a browser receives from the Web UI's login link

`?auth_code=` is a different thing: a one-time login code the gateway prints at startup, which
exchanges itself for a session cookie. It is not the token.

## Request Handling

When a request is received via the OpenAI compatibility layer:

1. **Agent Selection:** The Gateway uses the **Default Agent**: `agents.default`, else the agent marked `default: true`, else the first agent in `agents.list`, else `main`.
2. **Conversation:** The last `user` message is the prompt; the `system`, `user` and `assistant` messages before it are the conversation history. GoClaw keeps no history for this endpoint: a session `openai-compat:` is created, but nothing is written to it, so send the whole conversation in `messages` on every request.
3. **Provider Logic:** The request is processed by the default agent's provider. When it fails before the reply used a tool (a rate limit, an outage, a bad key), the next provider of `ai.fallback` serves it (see [Provider Fallback](../architecture/provider_fallback.md)). When every provider fails, the error is returned as an OpenAI-style error (`{"error":{"message":…}}`, status `500` when not streaming).
4. **Token Estimation:** Usage stats (`prompt_tokens`, `completion_tokens`) are estimated based on character count (~4 characters per token).

## Usage in IDEs (Cursor/Continue)

To use GoClaw in your IDE:
1. Set the **Base URL** to `http://your-gateway:8080/v1`.
2. Set the **API Key** to your gateway token.
3. Select any model (GoClaw will route to your default configured agent).

================================================================================
# FILE: https://goclaw.ulab.mn/docs/vision/index.md
================================================================================

# GoClaw Vision & Roadmap

This section details the design principles and the roadmap of the GoClaw platform.

## Documentation

- **[Level 4: AI Employees](level4_final.md)**: The platform philosophy: code does the deterministic work, the model reasons, and code checks the result.
- **[Roadmap to Level 4 Autonomy](level4.md)**: The principles behind GoClaw's path to Level 4, what is in place and what is still open.
- **[GoClaw Appliance](goclaw-appliance.md)**: Hardware vision for autonomous local servers.
- **[Level 5 Vision](level5.md)**: A possible path to Level 5 as governed autonomy: agents propose goals, experiments, SOPs and tools under human review and policy checks. None of it is implemented.

================================================================================
# FILE: https://goclaw.ulab.mn/docs/vision/level4.md
================================================================================

# GoClaw: Roadmap to Level 4 (Autonomous Digital Workforce)

Level 4 is an AI employee: an agent that runs a business procedure from start to finish on its own, inside tools the platform provides, under rules people set ([Autonomy Levels](../architecture/autonomy_levels.md)). This page covers three things: the principles that decide what GoClaw builds toward that goal, what is in place, and what is still open. It changes when a capability lands or changes. The platform philosophy is in [Level 4: AI Employees](level4_final.md).

---

## 🧭 Principles

1. **Procedures, not prompts.** Work that must come out right every time is a SOP. A SOP has steps, each step gets only the tools it needs, and Go code checks each step's answer before the next one starts.
2. **Code does the deterministic work.** Queries, calculations and calls that repeat for every row of a list run as SQL, data tools and MCP map steps. The model reasons about the results; it is not used as a loop.
3. **Zero trust by default.** The agent works through built-in tools and connected MCP servers only. Shell commands are sandboxed and file access stays in the workspace. No tool can write `config.yaml` or the plugins directory, and every webhook authenticates.
4. **People hold the controls.** Approval steps, hook rules on tool calls, roles and admin-only files decide what an agent may do and when it must ask.
5. **Failures are visible.** A step whose check fails is retried or ends the run. Every run records each step's answer. A run that fails, escalates, cannot resume after a restart or waits for approval sends a notice to `sop.alerts`.
6. **One general platform.** A customer's need becomes a general capability, or advice on how to write their SOP. GoClaw does not grow customer-specific code.
7. **Cost has a ceiling.** Token budgets and per-run limits stop runaway runs, and large results stay out of the prompt until the model asks for a part of them.

---

## 🟢 In place

### Procedures (SOPs) — [SOP guide](../user-guide/automation/sop.md)
*   **Step-by-step runs with state:** each step's answer, attempts and tokens are stored, and a step can call only the tools listed for it.
*   **Checks in Go:** `success_check` (`json`, or `expr:` on the values a step publishes) decides whether a step passed. A step's constraints are rules the model is given. A failed check is retried. `failure_mode` then carries on (`retry`), fails the run (`abort`) or hands it to people (`escalate`).
*   **Branching:** `on_failure_goto` jumps to a recovery step. `end_run_if` ends a run early, and `skip_if` skips a step.
*   **Map steps:** `for_each` runs a step once per item. `mcp_call` calls an MCP tool per item with no model call in between.
*   **Human-in-the-loop:** a step with `requires_approval` suspends the run until someone approves or rejects it.
*   **Triggers:** cron schedules and authenticated webhook routes start runs.
*   **Restarts:** steps have time limits, and an effects journal records what a step already did. A run cut by a restart resumes within `sop.resume_window`; past that, a notice goes to `sop.alerts`.
*   **Prerequisites:** a SOP's `prerequisites` (tools, views, permissions) are checked before it runs. `capability_request` lets the agent hand a missing capability to IT ([Tools and MCP](../user-guide/tools_and_mcp.md)).
*   **Undo (opt-in):** a step may name a `rollback_command`. When a run ends with `abort` or `escalate`, the completed steps' commands run newest first.

### Events and channels
*   **Channels:** MS Teams, Discord, Facebook Messenger, Telegram, Signal (one-to-one chats), WebSocket, REST and an OpenAI-compatible API ([Channels](../user-guide/channels/index.md)).
*   **Alerts into investigations:** managed hooks for GitLab, Graylog, Grafana, SonarQube, Zabbix, Uptrace and Wazuh turn an event into a SOP run ([Automation](../user-guide/automation/index.md)).

### Memory and learning — [Memory Engine](../architecture/memory_engine.md)
*   **Facts with a source:** facts come from what people said and what tools returned, never from the agent's own earlier answers. They are stored as memory and as a knowledge graph.
*   **Conflicts and hygiene:** belief revision decides between conflicting facts, and consolidation merges duplicates every night.
*   **Learning from errors:** a failed tool call is diagnosed in the same turn and pointed at approved [Saved Queries](../user-guide/saved-queries.md), tool presets and wiki pages. No error lessons are stored.
*   **Organisation knowledge:** the org wiki, and a [Wiki.js](../user-guide/wikijs.md) mirror.

### Delegation — [Subagents and Jobs](../architecture/subagents_and_jobs.md)
*   **Subagents and background jobs:** a managing agent delegates work under a spawn policy.
*   **Agent-to-agent messages:** off unless `tools.agent_to_agent.enabled` is set.

### Guardrails — [Security](../architecture/security.md)
*   **Loop detection:** repeated calls of the same tool and repeated errors are caught and stopped.
*   **Hook rules:** an admin can block a tool call, require approval for it, send a notice, or ask an outside check, which fails closed.
*   **Read-only data access:** `db_query` checks the SQL, runs it in a read-only transaction under server limits, and confines a delegated run to its role and subject.
*   **Sandbox:** shell deny lists, workspace-only files, SSRF protection for outgoing HTTP, and an encrypted secrets vault.

### Cost and reliability
*   **Token budgets:** daily limits per agent and per session, an hourly limit per session, a global daily limit, and `max_run_tokens` per SOP run ([Token Budgeting](../user-guide/token-budgeting.md)).
*   **Large results:** results over the size limit go to the output store and are read in parts with `read_output` ([LLM Optimization](../architecture/llm_optimization.md)).
*   **Models:** each agent can list fallback models, and an overloaded provider is retried ([Provider Fallback](../architecture/provider_fallback.md)).
*   **Runtime:** a bounded worker pool runs tools, and a panic ends its request, not the gateway. `goclaw doctor` checks the setup ([CLI](../user-guide/cli-reference.md)), and OpenTelemetry traces each run.

---

## 🛠️ Still open

1. **Dynamic negotiation.** Agents do not yet weigh each other's workload or skills to share out work. Messages reach an agent through static bindings, and the model picks a subagent with the `subagents` tool.
2. **Atomic rollback.** A failed run undoes only the steps that name a `rollback_command`. There is no transaction across steps, and nothing reverts files or database changes on its own.

Beyond Level 4, [Level 5 Vision](level5.md) describes agents that propose goals, SOPs and tools for people to review. None of it is implemented.

================================================================================
# FILE: https://goclaw.ulab.mn/docs/vision/level4_final.md
================================================================================

# GoClaw Level 4: The Autonomous Agentic Platform

GoClaw is built to run business procedures (SOPs) reliably in any industry: banking, telecoms, logistics, IT operations. Each procedure has its own steps, data and approvals, so the platform supplies only what every procedure needs. Code checks the work, and the model does the reasoning in between. This page describes that design. What is in place and what is still open is in [Roadmap to Level 4](level4.md).

## 🎯 Platform Philosophy
**Decoupled execution.** A request mixes two kinds of work. Deterministic work, such as fetching rows, adding numbers or calling an API once per item in a list, has one right answer. Reasoning work, such as reading a result, choosing what to do next or writing a reply, does not. GoClaw gives the deterministic work to code:

- SQL through `db_query` and saved queries;
- calculations through `calc` and `json_transform`;
- one MCP call per item through `mcp_call` map steps.

The model reasons about the results instead of repeating a tool call for every row. A SOP author makes the split explicit: a step can run approved saved queries with no model call at all ([SOP guide](../user-guide/automation/sop.md)).

---

## 🏗️ Platform Capabilities

### 1. Capability gaps are handed to people
*   **What it does:** a SOP lists its `prerequisites` (tools, views, permissions), and GoClaw checks them when the SOP is saved and again before each run. When an agent finds that a goal needs something it does not have, `capability_request` records what is missing, so IT or a DBA can provide it ([Tools and MCP](../user-guide/tools_and_mcp.md)).
*   **Why:** the agent knows the limits of what it may do and asks for more, instead of working around them. The infrastructure grows with a person in the loop.

### 2. The tools a call needs come with it
*   **What it does:** a set of helper tools is offered in every context, a SOP step that lists only one tool included: `read_output`, `json_transform`, `datetime`, `calc`, `regex` and `text_utils`. A request that carries an image, a PDF, a document, a voice message or a link is offered the tool that reads it. Approved saved queries and tool presets that match the request are offered with the tool they run on.
*   **Why:** an agent should not fail, or invent an answer, because a helper it needs was left out of its tool list.

### 3. Code checks the work
*   **What it does:** a step's `success_check` is evaluated in Go: valid JSON, required fields, text it must or must not contain, length, a regular expression, or a comparison of the values the step published (`expr:`). A step that fails its check is retried, and then carries on, fails the run or escalates to people, as its `failure_mode` says.
*   **Why:** a model's answer varies from run to run. The check does not, so a run either meets its success criteria or says that it did not.

### 4. Built to run unattended
*   **What it does:**
    *   A bounded worker pool runs every tool call, and a panic ends its request, not the gateway.
    *   A SOP run cut by a restart resumes from its effects journal, or tells `sop.alerts` why it did not.
    *   Token budgets and per-run limits stop a runaway run.
    *   Old traces are pruned every night, and `goclaw doctor` and the health endpoints show whether the setup works.
*   **Why:** an AI employee is only useful if it keeps working for months without someone watching it.

---

## 🔁 Knowledge that stays consistent
*   **Facts with a kind and a source:** each fact the agent keeps has a kind, such as a fact or a preference. It comes from what a person said or a tool returned, never from the agent's own earlier answers.
*   **Conflicts:** when a new fact contradicts an old one, a belief-revision step decides which one holds, and the change is recorded so it can be undone.
*   **Duplicates:** a nightly consolidation removes near-duplicate memory chunks and merges notes that say the same thing ([Memory Engine](../architecture/memory_engine.md)).

================================================================================
# FILE: https://goclaw.ulab.mn/docs/vision/level5.md
================================================================================

# GoClaw Level 5 Vision

This document describes a possible path from GoClaw's current Level 4 model, an autonomous digital workforce gateway, toward a Level 5 enterprise operating layer. It is a roadmap document, not a statement of current implementation.

## Level 4 To Level 5

| Dimension | Level 4 | Level 5 target |
|---|---|---|
| Goal source | Humans assign tasks | Agents infer candidate goals from organizational signals |
| Learning | Learns facts from conversations; corrects a failed tool call in the same turn | Learns from controlled experiments and outcomes |
| Collaboration | Manager-to-worker delegation | Peer negotiation, reputation, and collective decisions |
| Organization model | Channels, sessions, tools, SOPs | Live world model of systems, dependencies, risk, and priorities |
| Safety | Role, tool, auth, and channel controls | Constitutional policy layer with decision provenance |
| Self-improvement | Admin-approved notes, saved queries and SOPs | Proposed SOPs, skill gaps, and tool composition under review |

## Pillar 1: Autonomous Goal Formation

Level 5 should propose goals from observed operational signals instead of waiting only for direct commands.

Candidate inputs:

- Repeated cron failures.
- Repeated Zabbix or Grafana alerts.
- GitLab pipeline patterns.
- Unresolved SOP failures.
- Frequent user requests in the same area.
- Missing documentation discovered during repeated tool use.

Conceptual model:

```go
type StrategicGoal struct {
    ID          string
    Description string
    Priority    int
    Evidence    []string
    GeneratedBy string
    CreatedAt   time.Time
}
```

Goal lifecycle:

```mermaid
stateDiagram-v2
    direction LR
    [*] --> proposed
    proposed --> approved
    approved --> in_progress
    in_progress --> completed
    in_progress --> abandoned
```

Human approval should remain the default. Auto-approval can be considered only for low-risk goals with clear policy boundaries.

## Pillar 2: Hypothesis-Driven Learning

Level 5 should learn by proposing and testing hypotheses in controlled environments.

Experiment lifecycle:

1. Form a hypothesis.
2. Define a safe test environment.
3. Execute the smallest useful test.
4. Measure the outcome.
5. Write verified learning to memory or agent notes.
6. Mark unverified assumptions as such.

Example:

```text
Hypothesis: Increasing GitLab runner cache reuse will reduce deploy duration.
Environment: staging project only.
Metric: median pipeline duration over 10 runs.
Approval: required before changing runner configuration.
```

## Pillar 3: Agent Society

The current system supports subagents and session tools. A Level 5 system would add richer peer coordination.

Future concepts:

- Peer task bidding based on expertise and availability.
- Agent reputation scores based on completed work and reviewed outcomes.
- Collective decision sessions for high-risk actions.
- Cross-agent knowledge transfer with confidence and verification metadata.

Conceptual model:

```go
type AgentReputation struct {
    AgentID        string
    TrustScore     float64
    SuccessRate    float64
    SpecialtyAreas []string
}
```

## Pillar 4: World Model And Causal Reasoning

GoClaw already has observability tools, memory, graph storage, and metrics storage. Level 5 would connect those into a live world model.

World model inputs:

- Service topology from Uptrace.
- Host and trigger state from Zabbix.
- Alerts from Grafana.
- Logs from Graylog.
- Security events from Wazuh.
- Inventory from NetBox.
- Deploy and issue data from GitLab.
- Stored metrics from `metrics_store`.

Conceptual model:

```go
type WorldModel struct {
    Systems            map[string]*SystemNode
    Dependencies       []Dependency
    CurrentState       map[string]StateValue
    HistoricalPatterns []Pattern
    LastUpdated        time.Time
}
```

The goal is not just to answer "what is broken?" but to evaluate "what is likely to happen if we take this action?"

## Pillar 5: Adaptive Communication

Level 5 should adjust communication style to user state, channel, and urgency while staying transparent.

Examples:

| Situation | Preferred behavior |
|---|---|
| Incident response | Short status, concrete action, no long background unless asked |
| New user | Plain language, fewer internal abbreviations |
| Risky action | Explicit risk, approval requirement, rollback plan |
| Repeated failure | State what changed since the previous attempt |

This should be implemented as style guidance and response policy, not as hidden emotional profiling.

## Pillar 6: Self-Architecture And Meta-Learning

GoClaw already has SOPs, tools, auto-learning, and skill registries. Level 5 would add governed self-improvement proposals.

Future capabilities:

- Detect repeated tool sequences and propose SOPs.
- Detect skill gaps and recommend new skills or docs to index.
- Suggest tool compositions, then require review before promotion.
- Track whether generated SOPs actually improve outcomes.

Conceptual model:

```go
type SynthesizedToolProposal struct {
    Name        string
    Components  []string
    Reason      string
    Evidence    []string
    RequiresReview bool
}
```

## Pillar 7: Constitutional Policy And Auditability

As autonomy increases, the policy layer must become more explicit.

Required controls:

- Immutable prohibitions for destructive production actions.
- Approval thresholds by blast radius, data volume, financial risk, and security scope.
- Decision chains for every automated action.
- Prompt-injection boundaries for webhook, email, web, and document inputs.
- Separation between suggestions, approved plans, and executed actions.

Example policy sketch:

```yaml
constitution:
  absolute_prohibitions:
    - "Delete production data without explicit approval"
    - "Expose credentials or secrets"
    - "Escalate beyond configured role permissions"
  approval_required:
    - "Actions affecting more than 500 users"
    - "External API bursts above configured thresholds"
    - "Bulk data movement above configured size limits"
```

## Implementation Roadmap

### Phase A: Foundation

- World model store.
- Goal proposal API and review UI.
- Cross-agent knowledge bus.
- Experiment sandbox with read-only defaults.
- SOP suggestion from repeated patterns.

### Phase B: Intelligence

- Causal inference over observability and deploy signals.
- Counterfactual simulation for operational actions.
- Peer negotiation protocol.
- Adaptive communication policy.
- Skill gap tracking.

### Phase C: Governed Autonomy

- Strategic intent engine.
- Constitutional enforcement middleware.
- Collective decision system.
- Full decision-chain UI.
- Outcome measurement for goals and SOPs.

## Near-Term Practical Steps

1. Store observed operational facts in a world-model schema.
2. Add a goal proposal table and Web UI review flow.
3. Add SOP suggestion from repeated trace patterns.
4. Add policy checks before any high-risk SOP or tool action.
5. Record outcome metrics for completed goals.

## Summary

Level 5 is not just a smarter single agent. It is a governed coordination layer where agents can propose goals, test hypotheses, collaborate, reason over operational state, and improve workflows while remaining auditable and policy-bound.

================================================================================
# FILE: https://goclaw.ulab.mn/docs/vision/goclaw-appliance.md
================================================================================

# GoClaw Appliance — Vision and Long-Term Roadmap Document

> "A box that sits in the user's home or office, runs without internet access, understands Mongolian, and acts as their personal AI employee."

**Status:** Vision / Long-term Roadmap  
**Date:** 2026-03-30

The GoClaw Appliance is a box that sits in the user's home or office, runs without internet access, understands Mongolian, and acts as their personal AI employee. This page is the vision and long-term roadmap for it.

---

## 1. Why Do We Need This?

### The Problem

Alexa, Google Home, Apple HomePod — the world's most famous AI assistants:

```
❌ Do not understand Mongolian
❌ Send all your conversations to Amazon/Google servers
❌ Do not work if the internet is down
❌ Cannot be customized or extended (black box)
❌ Require subscriptions and suffer from cloud lock-in
```

### Our Answer

```
✅ Native support for the Mongolian language
✅ Your voice recordings never leave your premises
✅ Offline-first — works even without internet access
✅ Leverages all GoClaw SOPs, tools, and memory
✅ One-time purchase, no subscription fees
```

---

## 2. Product Description

### GoClaw Box

```mermaid
flowchart TD
    subgraph BOX["GoClaw One — Matte Black / White Cylinder (iPhone height)"]
        LED["LED Ring
12x RGB status indicator"]
        LOGO["GoClaw Logo"]
        MIC["Mic Array
6-mic ReSpeaker (360° wake word)"]
        SPK["Speaker
360° full-range sound"]
        PORTS["Ports
USB-C Power · Gigabit Ethernet"]

        LED --- LOGO --- MIC --- SPK --- PORTS
    end
```

### Target Audience

| Segment | Use Case | Benefits |
|---|---|---|
| **Enterprise** | Office AI assistant in meeting rooms | Fetch GitLab reports and query DBs via voice commands |
| **Family** | Mongolian smart home voice assistant | Weather, schedules, and reminders |
| **Developers** | Local AI dev environment | Perform code reviews and git operations via voice commands |
| **Government** | Classified/secure environments | Air-gapped deployment, zero data leaks |

---

## 3. Technical Architecture

### Hardware Stack

```mermaid
flowchart TD
    subgraph HW["GoClaw One Hardware Stack (BOM: $120-150 · Retail: $299-399)"]
        direction LR
        PROC["PROCESSING
Intel N100 / ARM (mini PC SBC)
RAM: 16GB LPDDR5
Storage: 128GB NVMe (OS + models)"]
        IN["AUDIO INPUT
ReSpeaker 6-mic Array (360° wake word)
Echo cancellation built-in"]
        OUT["AUDIO OUTPUT
HiFiBerry DAC+ / Class D amplifier
2x 2.5 in full-range speaker"]
        CONN["CONNECTIVITY & STATUS
Gigabit Ethernet · WiFi 6 · Bluetooth 5.0
RGB LED ring (12 LEDs — WS2812B)"]
    end
```

### Software Stack

```mermaid
flowchart TD
    L4["LAYER 4: MEMORY & SKILLS
SQLite (local, encrypted) · GoClaw SOP engine · GoClaw skill registry"]
    L3["LAYER 3: INTELLIGENCE
GoClaw binary (single static binary) · Ollama (offline LLM — qwen2.5:7b) · Cloud API fallback (Gemini/Claude)"]
    L2["LAYER 2: VOICE PIPELINE
Wake word: OpenWakeWord 'Hey GoClaw' · STT: Whisper (faster-whisper, small) · TTS: Piper TTS (Mongolian model)"]
    L1["LAYER 1: OS
Custom Linux image (LinuxKit / Buildroot) · Kernel 6.x + network + audio subsystem only · Boot time: < 5 sec (NVMe)"]

    L4 --> L3 --> L2 --> L1
```

### Offline / Online Hybrid Flow

```mermaid
flowchart TD
    SPEAK["User speaks"] --> WAKE["Wake word
100% local"]
    WAKE --> STT["Whisper STT
100% local, ~0.5-1 sec"]
    STT --> NET{"Is there Internet?"}

    NET -->|YES| CLOUD["Gemini / Claude
Fast, high-quality"]
    NET -->|NO| LOCAL["Local Ollama
Slower, offline"]

    CLOUD --> GC["GoClaw
SOP, tools, memory"]
    LOCAL --> GC

    GC --> TTS["Piper TTS
100% local"]
    TTS --> OUT["Speaker speaks response"]
```

---

## 4. Mongolian Language Challenges

This is the largest technical hurdle. Let's list them honestly.

### 4.1 Speech-to-Text (STT)

```
Current Status:
  Whisper large-v3 → Mongolian ~60-70% accuracy
  For smaller models (tiny/small) → even lower
  Technical terms and regional dialects → weak

Proposed Mitigation:
  1. Expand the Common Voice MN dataset (community-driven)
  2. Fine-tune Whisper small on Mongolian data
  3. Inject domain vocabulary bias (GoClaw commands)
```

### 4.2 Text-to-Speech (TTS)

```
Current Status:
  Piper TTS → No Mongolian voice model available
  Google/Azure TTS → Mongolian models exist (cloud-only)
  XTTS-v2 zero-shot → Not optimized for Mongolian

Proposed Mitigation:
  1. Collect 10+ hours of high-quality Mongolian audio/text dataset
  2. Train a Mongolian model on Piper VITS
  3. Interim solution: Google/Azure TTS (online mode)
              basic fallback voice (offline mode)
```

### 4.3 Wake Word

```
"Hey GoClaw" Mongolian pronunciation:
  → OpenWakeWord custom model (~30 min of training data is sufficient)
  → Personal mode — customize with the user's own voice
  → Community wake word — covering multiple accents and speakers
```

---

## 5. Roadmap

### Phase 1 — Proof of Concept (3-6 months)
```
[ ] GoClaw static binary (CGO-free, modernc/sqlite)
[ ] Ollama + GoClaw on an Intel N100 Mini PC
[ ] Whisper tiny/base Mongolian test
[ ] USB mic + speaker prototype
[ ] Detect "hey goclaw" wake word
[ ] LED ring status indicator

Milestone: A desk-friendly demo prototype
```

### Phase 2 — Voice Pipeline (6-12 months)
```
[ ] Collect Mongolian STT dataset (community effort)
[ ] Whisper small → Mongolian fine-tuning
[ ] Mongolian TTS voice model (Piper)
[ ] Custom wake word model
[ ] Integrate hybrid cloud/offline routing into GoClaw
[ ] Extend tool_voice_input.go
```

### Phase 3 — Hardware Design (12-18 months)
```
[ ] PCB design (Intel N100 / CM4 based)
[ ] 3D print enclosure prototype
[ ] Integrate ReSpeaker 6-mic array
[ ] Speaker + amplifier tuning
[ ] Custom Linux OS image (LinuxKit/Buildroot)
[ ] Secure boot + OTA update

Milestone: 10-20 hand-assembled prototypes
```

### Phase 4 — Pilot Program (18-24 months)
```
[ ] 10-50 pilot users (Mongolian companies)
[ ] Privacy audit (certify that all data remains strictly local)
[ ] Feedback → iterate
[ ] Establish pricing model ($299 target)

Milestone: Proven, user-accepted appliance in real environments
```

### Phase 5 — Production and Release (24+ months)
```
[ ] Manufacturing partner
[ ] B2B sales targeting Mongolian organizations
[ ] Open source hardware design (CC BY-SA)

Milestone: Full market release
```

---

## 6. Competitor Comparison

| | Alexa Echo | Google Home | Humane Pin | **GoClaw One** |
|---|---|---|---|---|
| Mongolian language | ❌ | ❌ | ❌ | ✅ |
| Privacy | ❌ Cloud | ❌ Cloud | ⚠️ | ✅ 100% Local |
| Offline | ❌ | ❌ | ❌ | ✅ hybrid |
| Customizable SOP | ⚠️ | ⚠️ | ❌ | ✅ |
| Enterprise | ❌ | ❌ | ❌ | ✅ |
| Open source | ❌ | ❌ | ❌ | ✅ |
| Price | $50-250 | $30-150 | $699+sub | **$299 target** |

---

## 7. Open Issues

```
🔴 HARD — Critical Hurdles:
   → Mongolian TTS dataset (10+ hours of high-quality audio)
   → Mongolian STT accuracy for technical terminology
   → Keeping hardware costs within a $150 BOM while ensuring quality

   Proposed Mitigation:
   → Dialects/accent variation in wake word detection
   → Offline LLM quality (limitations of 7B models)
   → OTA update security

🟢 RESOLVED — Already Available:
   → GoClaw static binary compilation works
   → Ollama fallback (fallback.go is ready)
   → SOP + memory engines are ready
   → Hybrid routing is supported
   → tool_voice_input.go base is ready
```

---

## 8. Open Source Foundations

```
STT:       OpenAI Whisper        (MIT)
TTS:       Piper TTS             (MIT)
Wake word: OpenWakeWord          (Apache 2.0)
LLM:       Ollama + Llama/Qwen   (MIT/Apache)
OS:        Linux + LinuxKit      (GPL/Apache)
Core:      GoClaw               (proprietary; binary-use license)

Hardware design: CC BY-SA — schematics are publicly open
```

---

## 9. One-Line Definition

> **"A Mongolian-language, privacy-first, offline-capable AI assistant — in your room, on your desk, heard by no one else."**

---

*Journey started: 2026-03-30*  
*Recorded as one of the dreams of the GoClaw project.*

================================================================================
# FILE: https://goclaw.ulab.mn/docs/architecture/apps.md
================================================================================

# Apps — Voice, Mobile, Desktop

## Voice & Desktop (Section 14)

Speech works on audio files: the `voice_output` tool (`tool_voice_output.go`) writes speech to a file in the workspace, and the `voice_input` tool (`tool_voice_input.go`) transcribes one, with the provider set by `voice.tts_provider` / `voice.stt_provider` (OpenAI, Gemini or Chimege). The same transcriber, registered in `cmd/gateway/main.go`, turns voice notes that arrive on a channel (Teams audio attachments, for example) into text. What stays disabled, because Windows Defender flagged it, is local microphone recording and speaker playback (`voice.go`) and desktop control (`desktop.go`).

| # | Feature | Status | Notes |
|---|---------|--------|-------|
| 14.1 | TTS (text-to-speech) | Done | `voice_output` tool, writes an audio file. Local playback (`voice.go`) disabled — Windows Defender |
| 14.2 | STT (speech-to-text) | Done | `voice_input` tool and channel voice notes. Local microphone recording (`voice.go`) disabled — Windows Defender |
| 14.3 | Desktop screenshot | Disabled | desktop.go — Windows Defender |
| 14.4 | Mouse/keyboard control | Disabled | desktop.go — Windows Defender |

---

## Mobile Apps — iOS / Android (Section 15)

OpenClaw has full native iOS and Android apps (~140 files each) that act as both companion UIs and remote execution nodes.

| # | Feature | OpenClaw | GoClaw | Status |
|---|---------|----------|-------|--------|
| 15.0 | API for a company's own mobile app | — | `app_api` channel: `POST /api/app/chat` (JSON or SSE), `GET /api/app/events` for agent-initiated messages. See [App API](../user-guide/channels/app-api.md) | Done |
| 15.1 | iOS app | SwiftUI — Screen, Voice, Settings tabs | — | **TODO** |
| 15.2 | Android app | Kotlin/Compose — gateway pairing, voice, node | — | **TODO** |
| 15.3 | Gateway pairing | QR scan or manual host/port/token | — | **TODO** |
| 15.4 | Voice Wake (wake word) | Background listening → trigger agent | — | **TODO** |
| 15.5 | Talk Mode (voice I/O) | STT → agent → TTS, continuous conversation | — | **TODO** |
| 15.6 | Camera node capability | Take photos on request from agent | — | **TODO** |
| 15.7 | Screen capture | Share/record screen for agent | — | **TODO** |
| 15.8 | SMS / Contacts / Calendar | Access on behalf of agent (Android) | — | **TODO** |
| 15.9 | Apple Watch extension | Quick voice/text from wrist | — | **TODO** |
| 15.10 | Share extension (iOS) | Share content to agent from any app | — | **TODO** |
| 15.11 | Foreground service (Android) | Keep node alive for background execution | — | **TODO** |

**GoClaw approach:** A company's own app can already reach agents through the `app_api` channel, usually through its backend ([Flutter + Django guide](../user-guide/channels/flutter-django-integration.md)). For a GoClaw app of its own, consider a cross-platform mobile app (Flutter / Go Mobile) or defer — mobile is the most complex area of OpenClaw.

---

## Desktop App — macOS Menubar (Section 16)

OpenClaw has a full macOS menubar app (~359 files, SwiftUI) that hosts the gateway and provides quick access.

| # | Feature | OpenClaw | GoClaw | Status |
|---|---------|----------|-------|--------|
| 16.1 | Menubar icon + status | NSStatusItem with status indicator | — | **TODO** |
| 16.2 | Hover HUD | Quick chat panel on hover | — | **TODO** |
| 16.3 | Web Chat panel | Embedded web view for Control UI | — | **TODO** |
| 16.4 | Voice Wake overlay | Full-screen overlay for voice | — | **TODO** |
| 16.5 | Push-to-talk | Hold key to speak | — | **TODO** |
| 16.6 | Exec approval prompts | Allow once / Always allow / Deny dialog | — | **TODO** |
| 16.7 | Node pairing approval | Approve new mobile devices | — | **TODO** |
| 16.8 | Gateway process management | Start/stop/restart gateway from menu | — | **TODO** |
| 16.9 | Sparkle auto-update | Signed update via appcast.xml | — | **TODO** |
| 16.10 | Onboarding wizard | First-run guided setup | — | **TODO** |

**GoClaw approach:** Consider systray (Go) for cross-platform tray icon, or Wails/Tauri for a desktop wrapper. GoClaw's strength is running as a headless binary — desktop app may be lower priority.

**Coverage: 0%**

================================================================================
# FILE: https://goclaw.ulab.mn/docs/architecture/channel_router.md
================================================================================

# Channel Router & Pairing Architecture

This document describes the internal architecture of GoClaw's channel system, which enables communication across multiple platforms while maintaining a consistent experience.

---

## 1. The `Channel` Interface

Every messaging platform integrated into GoClaw must implement the `Channel` interface (defined in `internal/channel/channel.go`). This ensures the core gateway can interact with all platforms uniformly.

```go
type Channel interface {
    // Lifecycle
    Start(ctx context.Context) error
    Stop() error

    // Outbound Messaging
    SendMessage(ctx context.Context, peerID, content string) error
    SendFile(ctx context.Context, peerID, text string, files []Attachment) error
    SendEmbed(ctx context.Context, peerID string, embed *Embed) error
    SendTyping(ctx context.Context, peerID string) error
}
```

---

## 2. Inbound Data Normalization (`InboundMeta`)

When a channel receives a message, it must normalize the platform-specific payload into an `InboundMeta` struct. This metadata is used by the router to resolve the correct agent and session.

| Field | Description |
|-------|-------------|
| `ChannelName` | The platform name (e.g., `"discord"`, `"msteams"`, `"facebook"`). |
| `AccountID` | The account the message arrived on: the bot's ID on Teams, Telegram and Facebook. Discord puts the sender's user ID here. |
| `PeerID` | **Session Routing Key.** The ID of the conversation (DM or channel). Discord uses `dm:` for a DM and `:` in a guild channel, so each user has their own session there. |
| `PeerKind` | Either `"direct"` (DM) or `"group"` (guild/team/channel). |
| `AuthorID` | The platform-specific ID of the user who sent the message. |
| `AuthorName` | The display name of the author. |
| `TenantID` | The guild ID in Discord. In MS Teams, the conversation ID of a group chat or channel (with `;messageid=…` removed); empty in a 1:1 chat. Other channels leave it empty. |
| `ParentPeerID` | Used for threaded conversations: on Discord, the channel a thread belongs to. |
| `PlatformRoles` | An array of role IDs the user holds on that platform. Only Discord (the member's role IDs) and the App API set it. |

---

## 3. Agent Routing Priority

The routing engine (`internal/routing/routing.go`) maps an inbound message to an agent based on the following priority:

1.  **Exact Peer Match:** An explicit binding for a specific peer ID (see `PeerID` above).
2.  **Thread Inheritance:** Inheriting the agent bound to the parent channel of a thread.
3.  **Tenant + Platform Role:** Matching a specific tenant and a user's role (e.g., a role ID in a specific Discord guild).
4.  **Platform Role:** A role binding with no tenant, on any tenant of that channel.
5.  **Tenant:** A broad match for an entire tenant (a Discord server, or one Teams group chat or channel).
6.  **AccountID:** Used when running multiple bot identities.
7.  **Channel-wide Fallback:** The agent for that platform (e.g., all Discord messages).
8.  **Default Agent:** `agents.default`, else the agent marked `default: true`, else the first agent in `agents.list`.

See [Multi-Agent Routing](routing.md) for the binding syntax.

---

## 4. The Pairing System Architecture

Access control is enforced via a centralized pairing store. The gatekeeper checks if an incoming `AuthorID` is authorized before allowing the message to reach the agent.

### The Access Gate
Channels do not enforce pairing locally. Instead, the `InboundHandler` (provided during initialization) wraps the request in an access gate. `channels.dm_policy` decides what it does:

| `dm_policy` | Behaviour |
|-------------|-----------|
| `pairing` (default) | A sender listed in `channels.allow_from` (or `"*"` there) or approved in the pairing store gets through. Anyone else gets a challenge code to hand to an admin. |
| `allowlist` | The same check, but an unknown sender is refused ("You are not authorized to use this bot.") and gets no code. |
| `open` | Everyone gets through. |

The App API (`app_api`) skips the gate: its callers are authenticated by the API key or token before the message reaches the handler.

### Data Structures
GoClaw maintains a unified user identity across platforms.

```go
type User struct {
    ID             string              // Unified internal ID: "u_" + creation time in nanoseconds
    GoClawRole     string              // "admin" or "user"
    LinkedAccounts map[string]string   // e.g., {"discord": "1482...", "msteams": "abc..."}
    Meta           map[string]string   // Arbitrary user metadata
}
```

### Storage and TTL
Pairing state is stored in `~/.goclaw/credentials/` as JSON files:
- **`goclaw-pending.json`**: Active pairing requests with a 1-hour TTL.
- **`goclaw-users.json`**: The approved user database.

The running gateway reads these files once and keeps them in memory. A change made with `goclaw pairing` while the gateway runs is not seen until it restarts, and the gateway's next write to the file can drop it. While it runs, approve on the Web UI's Pairing tab or with `POST /api/pairing` (gateway token, body `{"action": "approve", "code": "ABCD1234", "role": "user"}`).

**Session history** is managed separately via the main **SQLite database**, ensuring persistent conversations across restarts.

---

## 5. Implementing a New Channel

To add support for a new platform, follow these steps:

1.  **Define the Channel:** Create a new package in `internal/channel/`.
2.  **Implement `Start()`:** Initialize the platform SDK and start a message listener.
3.  **Normalize Messages:** In your listener, convert inbound events into `InboundMeta` and call the provided `InboundHandler`.
4.  **Implement Outbound Methods:** Map `SendMessage`, `SendFile`, etc., to the platform's API.
5.  **Register the Channel:** In `startChannels` (`cmd/gateway/startup.go`), instantiate the channel, start it, and register its outbound sender with `agent.RegisterChannelSender`. A webhook-based channel is also mounted on the gateway's HTTP server.

### Important Principles
- **Always provide `AuthorName`:** Essential for identity recognition and LLM interaction.
- **Handle Chunking:** Platforms have different message limits. Use the built-in chunking utilities to split long responses.
- **Typing Indicators:** Use a background goroutine to refresh typing indicators during long agent processing.

================================================================================
# FILE: https://goclaw.ulab.mn/docs/architecture/cli-daemon.md
================================================================================

# CLI Commands + Daemon / Service Architecture

GoClaw is one binary: run with no subcommand it is the gateway, and its subcommands either call the REST API of a running gateway or work on their own (`chat`, `setup`, `doctor`, `config`). On Linux and macOS it installs itself as a systemd or launchd service.

## Architecture

```mermaid
flowchart TD
    subgraph BIN["goclaw (single binary, Cobra subcommand CLI, internal/cli)"]
        GW["Gateway Mode (default)
HTTP server + channels"]
        CLI["CLI Commands
Calls REST API of running gateway"]
        DIR["Direct Mode (no gateway needed)
chat, setup, doctor, config"]
        SVC["OS Service & Lock
systemd (Linux) / launchd (macOS)
Lock file: ~/.goclaw/gateway.lock (PID + port + time)"]

        GW --> SVC
        CLI --> SVC
        DIR --> SVC
    end
```

## CLI Commands

OpenClaw has 25+ command groups. GoClaw has one binary with top-level commands and nested subcommands.

| # | Command | Subcommands | Description | Status |
|---|---------|:-----------:|-------------|--------|
| 3.1 | (default) | — | Start gateway server | Done |
| 3.2 | `setup` | — | Interactive setup wizard (provider, API test, channels, summary); `--non-interactive` writes config.yaml from flags | Done |
| 3.3 | `--debug` | — | Debug mode (structured LLM/tool/gateway logging) | Done |
| 3.4 | `status` | — | Query running gateway (`GET /api/stats`) — provider, model, sessions, uptime, channels | Done |
| 3.5 | `pairing` | `approve`, `list`, `pending`, `revoke`, `link` | Pairing management (reads and writes the pairing files directly; a running gateway sees the change after a restart) | Done |
| 3.6 | `config get` | — | Read config value (dot-notation) | Done |
| 3.7 | `config set` | — | Write config value (AST-preserving YAML update) | Done |
| 3.8 | `config append` | — | Append to config list (e.g. `tools.shell.allow`) | Done |
| 3.9 | `config remove` | — | Remove from config list | Done |
| 3.10 | `doctor` | — | Health checks: config, state/workspace/memory dirs, vault, database, memory schema, integrations (live provider call), gateway, service boot (Linux) | Done |
| 3.11 | `message` | `[channel] ` | Run one agent turn through the running gateway (`POST /api/webhook`, peer `cli`, gateway token) and print the reply; nothing is posted to the channel | Done |
| 3.12 | `sessions` | `list`, `inspect` | Session management | Done |
| 3.13 | `agents` | `list`, `add`, `bind`, `unbind`, `delete` | Multi-agent CRUD + config save | Done |
| 3.14 | `chat` | — | Interactive terminal chat (direct LLM, no gateway, streaming) | Done |
| 3.15 | `skills` | `list`, `check`, `scan`, `install`, `add`, `remove`, `show` (see below) | Skill management + security | Done |
| 3.16 | `tools` | `list`, `exec`, `bench` | List, execute, and benchmark registered tools | Done |
| 3.17 | `service install` | — | Install as OS service (systemd/launchd); `--system [--run-as ]` for a system service run by an unprivileged user | Done |
| 3.18 | `service uninstall` | — | Uninstall OS service | Done |
| 3.19 | `service` | `start`, `stop`, `restart`, `status` | Control the installed service | Done |
| 3.20 | `quickstart` | — | Short setup wizard | Done |
| 3.21 | `version` | — | Version, commit and build date | Done |
| 3.22 | `config delete` | — | Remove a key, including keys the schema does not know | Done |
| 3.23 | `memory` | `check`, `stats`, `reindex`, `replay`, `relearn`, `revise`, `revisions`, `prune`, `dedupe`, `forget`, `clear`, `clear-user`, `backfill-metadata`, `notes`, `migrate`, `export`, `import` | Memory maintenance through the running gateway (`/api/memory/*`); `check` and `migrate` open the database directly | Done |
| 3.24 | `graph` | `migrate`, `reindex`, `check`, `stats`, `cleanup` | Knowledge graph (`kg.sqlite`) maintenance | Done |
| 3.25 | `db` | `export`, `import`, `inspect` | `.gclaw` database snapshots | Done |
| 3.26 | `secret` | `gen`, `set`, `get`, `list`, `delete` | Encrypted vault | Done |
| 3.27 | `genpass` | — | Print strong random passwords (aliases `gp`, `password`) | Done |
| 3.28 | `gitlab hook` | `setup`, `list`, `test`, `remove` | GitLab webhook wizard and helpers | Done |
| 3.29 | `teams` | `setup`, `package` | Teams setup wizard and app package (.zip) | Done |
| 3.30 | `transcribe` | — | Transcribe an audio or video file | Done |
| 3.31 | `sbom` | — | SBOM: summary, CycloneDX, SPDX, JSON | Done |
| 3.32 | `update` | — | Download, verify and install a release; restart the service | Done |
| 3.33 | `completion` | `bash`, `zsh`, `fish`, `powershell` | Shell completion scripts | Done |

The user-facing reference for every command is [CLI Reference](../user-guide/cli-reference.md).

### Skills Subcommands

| Subcommand | Description |
|-----------|-------------|
| `list` | List all loaded skills with tags, status, source |
| `check` | Check skill binary dependencies (pure Go detection) |
| `check ` | Compatibility report for one skill |
| `scan` | Security scan all skills (prompt-injection, data-exfiltration and dangerous-code rules) |
| `scan ` | Security scan a specific skill |
| `install ` | Install from URL (HTTP/HTTPS); the response is saved as the SKILL.md |
| `install github:` | Install from GitHub (single file from `main`, or repo clone) |
| `install ` | Install from local path (file or directory) |
| `install ... --force` | Install even when the compatibility check says FAIL |
| `add github:/` | Clone a repository and pick its skills interactively (`--force` as for install) |
| `remove ` | Remove user-installed skill |
| `show ` | Show skill content + metadata |

**Post-install security scan:** After every install, skills are automatically scanned. Critical findings = **blocked + auto-removed**.

### Doctor Checks

| # | Check | Description |
|---|-------|-------------|
| 1 | Config file | `/config.yaml` exists (state dir: `GOCLAW_STATE_DIR`, default `~/.goclaw`) |
| 2 | State directory | `~/.goclaw/` exists |
| 3 | Workspace | `~/.goclaw/workspace/` exists |
| 4 | Memory directory | `~/.goclaw/workspace/memory/` exists |
| 5 | Config and vault | Config loads; the secrets vault opens (with a note when `GOCLAW_SECRET_KEY` is unset); `${secret:…}` references resolved for the checks below |
| 6 | Database | `database.path` (default `~/.goclaw/gc_state.sqlite`) exists or will be created |
| 7 | Memory health | Schema migrations and critical findings, as `goclaw memory check` reports them |
| 8 | Integrations | `doctor.RunSystemChecks`: datasources, Zabbix/Uptrace/Grafana per agent, GitLab, SonarQube, MCP servers, enabled channels, SMTP/IMAP, and a live call to the AI provider |
| 9 | Gateway | Running gateway reachable + PID |
| 10 | Service boot | Linux only: an installed service starts at boot (lingering, `WantedBy` target) |

---

## Daemon / Service Architecture

OpenClaw runs as a background daemon with CLI-to-gateway IPC. GoClaw uses HTTP REST for CLI→Gateway communication.

| # | Feature | OpenClaw | GoClaw | Status |
|---|---------|----------|---------|--------|
| 3b.1 | Foreground server | HTTP/WS server | `gateway.Claim` takes the state directory and listens on its ports before anything starts; `httpSrv.Serve` on the claimed listener | Done |
| 3b.2 | Graceful shutdown | SIGINT/SIGTERM | SIGINT/SIGTERM handler | Done |
| 3b.3 | Lock file (PID) | `gateway..lock` with stale detection | `/gateway.lock` (PID+port+timestamp, stale detection: process alive and `/health` answers). A second gateway on the same state dir exits with "another GoClaw gateway is already running" | Done |
| 3b.4 | CLI → Gateway IPC | WebSocket JSON-RPC | **HTTP REST** (simpler, reuses API routes) | Done |
| 3b.5 | OS service install | launchd / systemd / schtasks | `goclaw service install` for systemd + launchd | Done |
| 3b.6 | OS service uninstall | Platform-specific | `goclaw service uninstall` | Done |
| 3b.7 | SIGUSR1 restart | Hot restart without downtime | — | **TODO** |
| 3b.8 | Bind modes | loopback / lan / tailnet / auto / custom | loopback / lan / all / custom IP | Done |
| 3b.9 | Gateway auth | Token/password + challenge handshake | Token auth (X-GoClaw-Token / Bearer) | Done |
| 3b.10 | Rate limiting | `AuthRateLimiter` | Per-IP token bucket (configurable req/min) | Done |
| 3b.11 | Remote gateway | `OPENCLAW_GATEWAY_URL` | — | **TODO** |
| 3b.12 | Public ingress | External tunnel/proxy | `gateway.ingress` (`type` ngrok or proxy, `url`, `authtoken`, allow list, optional dedicated `port`/`bind`); the deprecated `gateway.ngrok` and `channels.msteams.ngrok` are mapped onto it when `gateway.ingress.type` is unset | Done |

---

## GoClaw CLI → Gateway Flow

```mermaid
flowchart TD
    DAEMON["goclaw (daemon — foreground or OS service)
• Lock file: ~/.goclaw/gateway.lock (PID + port)
• HTTP server (bind: loopback / lan / all / custom)
• Middleware: rate limit → token auth → router
• SIGINT/SIGTERM → graceful stop
• OS service: systemd (Linux) / launchd (macOS)"]

    subgraph REST["Gateway HTTP REST + OpenAI-compat Endpoints"]
        R1["goclaw status → GET /api/stats"]
        R2["goclaw sessions list → GET /api/sessions"]
        R3["goclaw memory … → /api/memory/*"]
        R4["goclaw message → POST /api/webhook (agent turn)"]
        R5["POST /v1/chat/completions → OpenAI-compatible API"]
        R6["GET /health → health probe"]
    end

    subgraph LOCAL["Direct CLI Operations (no gateway required)"]
        L1["goclaw chat → direct LLM chat"]
        L2["goclaw agents / skills / tools → local CRUD & inventory"]
        L3["goclaw config get / set → AST-preserving YAML"]
        L4["goclaw doctor → health checks"]
    end

    DAEMON <-->|"HTTP REST + OpenAI-compat"| REST
```

---

## OpenClaw vs GoClaw CLI Comparison

| Aspect | OpenClaw | GoClaw |
|--------|----------|---------|
| **CLI style** | Command groups (`openclaw gateway run`) | Single binary with subcommands (`goclaw status`) |
| **Commands** | 25+ command groups | Top-level commands + nested config, memory, graph, db, secret, skills, tools, service, pairing, teams, gitlab groups |
| **IPC** | WebSocket JSON-RPC | **HTTP REST** (reuses API routes) |
| **Config management** | CLI config set/get | **get/set/append/remove/delete** with AST-preserving YAML |
| **Skill management** | npm/Clawhub | **list/check/scan/install/add/remove/show** |
| **Install sources** | npm | **3 sources: URL, GitHub, local** |
| **Post-install scan** | — | ✅ Auto security scan, critical → blocked |
| **Tools CLI** | — | `goclaw tools list`, `goclaw tools exec`, `goclaw tools bench` |
| **Doctor** | `openclaw doctor` | `goclaw doctor` (local files, vault, database, integrations, gateway, service boot) |
| **OS service** | launchd/systemd/schtasks | systemd + launchd |
| **Self-update** | `openclaw update` | `goclaw update` (checksum-verified, restarts the service) |

Not implemented: SIGUSR1 hot restart (3b.7) and a remote gateway for the CLI (3b.11; `goclaw setup --mode remote` has no effect).

================================================================================
# FILE: https://goclaw.ulab.mn/docs/architecture/concepts/agent.md
================================================================================

# Agent Runtime

GoClaw runs one or more agents in one process. Each agent is an entry in `agents.list` with its own workspace, model and memory, and channel messages reach an agent through [routing](../routing.md). With no `agents.list`, there is one agent, `main`. Every agent processes messages with an LLM provider and tool calling.

## Architecture

```mermaid
flowchart TD
    MSG["User Message"] --> RUNTIME["Agent Runtime
System Prompt + History + Memory"]
    RUNTIME --> LLM["LLM Provider
Gemini / Claude / OpenAI / DeepSeek"]
    LLM -->|Tool Calls| POOL["WorkerPool
Parallel execution · Bounded concurrency"]
    POOL -->|Tool Results| LLM
    LLM -->|Streamed Tokens| RES["Final Response"]
    RES --> CH["Channel Output"]
```

## Agent Loop

The event-driven agent loop (`ProcessStream`) is shared across all LLM providers:

1. Build the system prompt from workspace files + skills + memory
2. Append session history and the new user message
3. Call the LLM with the tools selected for this request: the core tools, plus the tools Tool RAG picks for the message (`tools.rag.top_k`, default 12) and the agent's most-used tools (`tools.rag.usage_prior`, default 6). The model can add a tool the selection left out with `search_tools`
4. If the LLM returns tool calls:
   - Execute tools **concurrently** via the bounded `WorkerPool`
   - Feed tool results back to the LLM via event channels
   - Repeat (up to `tools.max_tool_iterations`, default 20)
5. If the LLM returns text only, stream `EventTokenChunk` to the client

The Gemini provider uses a custom loop (native `genai` SDK) that natively streams content, but follows the same logical flow.

## LLM Providers

| Provider | Config Key | SDK |
|----------|------------|-----|
| OpenAI | `openai` | LangChainGo |
| Claude (Anthropic) | `claude` | LangChainGo |
| Gemini | `gemini` | google.golang.org/genai (native) |
| Gemini Legacy (deprecated) | `gemini-legacy` | served as `gemini` |
| DeepSeek | `deepseek` | LangChainGo (OpenAI-compatible client) |
| Ollama | `ollama` | LangChainGo |

The provider is chosen at startup from `ai.provider`. An agent's `agents.list[].model` overrides it for that agent, either as a provider name (`claude`) or as `provider/model` (`claude/claude-sonnet-4-5-20250929`). With neither set, the provider is `openai`.

## Model Failover

Every provider makes a model call again when it fails because the provider is overloaded (twice, after about 2 and 6 seconds) or on the network (once, after half a second); quota, billing and key errors are not retried ([Provider Fallback](../provider_fallback.md#before-it-moves-on-the-same-provider-is-asked-again)).

When the provider still fails before the reply has used a tool, the request moves to the next `provider/model` of `ai.fallback`, or of the agent's own `agents.list[].fallback`, which replaces it. A provider that keeps failing is put in cooldown and skipped while others remain. [Model Failover](model-failover.md) explains how.

## System Prompt

The system prompt is assembled in this order (`GetStaticSystemPromptFor` in `agent.go`):

1. **Core instruction** — the admin's core instruction for this agent, when one is set
2. `IDENTITY.md` and `SOUL.md` — Identity, persona, tone, boundaries. A subagent gets its profile here instead
3. `AGENTS.md` — Operating instructions, rules
4. **Instructions** — the admin's `instructions/.instructions.md` that apply to this run and name no tools
5. `TOOLS.md` — Tool usage notes and conventions (narrowed to the step's tools in a SOP step)
6. **Tool priority**; when Tool RAG is on, **finding tools** (`search_tools`); **when you cannot get it**: say which of three it is, in a sentence or two (no tool can do it; a tool exists but needs an administrator, was not offered here or was blocked by a rule; or the tool failed or found nothing), and never present other data as the answer; and **GoClaw documentation**: `goclaw_docs` for how GoClaw is configured and how it works, not for data a webhook or a system sent or holds
7. **Workspace, runtime and workspace files** — Working directory path, OS, host, timezone, and a listing of the workspace
8. **Skills** — Loaded skill markdown (built-in + user + workspace), filtered by Skill RAG
9. **Memory protocol** and **memory routing policy** — Which memory scope to use, and who may teach what
10. **Subagent delegation** — when the agent has `subagents.allow_agents`

A missing file is skipped. `Memory.md`, personal and global memory, agent notes and lessons are not part of this prompt: they go into the session context that follows it.

### Bootstrap File Protection

The following files are **read-only** except to an admin and to cron and SOP runs that act for no one in particular:

- `Memory.md`
- `SOUL.md`
- `TOOLS.md`
- `AGENTS.md`
- `IDENTITY.md`
- `USER.md`
- subagent profiles (`agents/*.agent.md`) and instructions (`instructions/*.instructions.md`)

The `file_write` and `file_edit` tools refuse to write them for other callers.

## Workspace

The default agent's workspace is `~/.goclaw/workspace/`. Every other agent's is `~/.goclaw/agents//workspace/`. `agents.list[].workspace` sets another path for either.

The workspace is the agent's working directory for all file operations. Path traversal outside the workspace is blocked.

### Bootstrap Files

| File | Purpose |
|------|---------|
| `SOUL.md` | Agent persona and boundaries |
| `AGENTS.md` | Operating rules and guidelines |
| `TOOLS.md` | Tool documentation and conventions |
| `Memory.md` | Long-term memory (shared across sessions) |

A new workspace is seeded with these four files from defaults built into the binary. In the default agent's workspace, `TOOLS.md` and `AGENTS.md` are written again from those defaults at every start (`AGENTS.md` from the `tools.mode` profile), so edits to those two files there do not last; `SOUL.md` and `Memory.md` are seeded only when missing. Rules of your own belong in an instruction (`instructions/.instructions.md`, on the Web UI's Instructions page) instead.

### Memory

`Memory.md` is the agent's curated long-term memory; it goes into the session context of every request. Everything the agent learns at run time is stored as records in the database, not as files:

- `memory_write` saves a record in one of four scopes: `personal` (one user's own context, never written or loaded in cron and SOP runs), `agent_notes` (org-wide knowledge: admins, cron and SOP runs, and whoever `memory.write_access` lists), `global` and `agent_config` (admins, and whoever `memory.write_access` lists; never SOP, cron or webhook runs)
- `memory_search` searches memory by meaning and keywords; `memory_get` reads a full entry
- Bootstrap files are **read-only** for non-admins (see above)

## Canvas

There is no canvas tool the model calls directly (`canvas_show` exists in `canvas.go` but is not registered). Files reach the canvas from tools that produce them:
- `browser` (its `screenshot` action) saves the PNG and shows it to Web UI clients as a `canvas` message
- `diagram_render` renders a Mermaid diagram to PNG
- `file_write` with `attach: true` attaches the file it wrote

These files are collected for the request and sent as attachments with the answer, on every channel. Screenshots and diagrams are stored in `~/.goclaw/canvas/`.

================================================================================
# FILE: https://goclaw.ulab.mn/docs/architecture/concepts/identity_and_memory.md
================================================================================

# User Identity & Cross-Platform Memory

This document provides a simple explanation of how the GoClaw system identifies users, recognizes them as the same person across different channels (Discord, MS Teams, Telegram, Facebook), and consolidates their memory records.

## 1. Gateway & Tenant

When you send a message to GoClaw, it first passes through the **Gateway** (the entry point). At this stage, the system verifies which organization and group the message is coming from.
Previously, these were named differently depending on the channel (e.g., `GuildID` for Discord, `TeamID` for MS Teams), but they have now been consolidated into a unified **TenantID**, which a routing binding matches with the key `tenant_id` whatever the channel. What it holds depends on the channel: the guild (server) ID on Discord; on MS Teams, the conversation ID of the group chat or channel (with `;messageid=…` removed), and nothing in a 1:1 chat. Telegram and Facebook set none. See [Multi-Agent Routing](../routing.md).

## 2. Pairing & Unified User

To accurately identify you, the system requires a **Pairing** step.

- **Pairing Challenge:** If you send a message to the bot for the first time, the system will not recognize you and will respond with a verification challenge code (e.g., `ABCD1234`), instructing you to give this code to your administrator to activate your account. This is the default `channels.dm_policy: pairing`. With `allowlist`, an unknown sender is refused and gets no code; with `open`, everyone gets through. Accounts listed in `channels.allow_from` always get through. The App API skips this step: its callers are already authenticated.
- **Unified Identity:** When the admin registers your account using this code, a unique **Unified ID** (e.g., `u_1710408593123456789`: `u_` and the creation time in nanoseconds) is created for you in the pairing store.
- **Linked Accounts:** Multiple channel-specific IDs (e.g., your Discord numerical ID and MS Teams ID) are linked to this single Unified ID. Therefore, whether you write from Discord tomorrow or MS Teams today, the system immediately recognizes you as the same user via your Unified ID.

This identification process is handled by the "AccessGate" layer within the Gateway, which resolves identity before passing the message context to the Agent.

## 3. GoClaw Role

When a user is registered, a specific **GoClawRole** is assigned to them. For example, if you are configured as an "admin" for an organization, you will retain admin permissions whether you interact via Discord or MS Teams.
This is crucial for the bot to determine authorization rules, such as whether a user is allowed to write to the global memory (shared system instructions).

## 4. Cross-Platform Memory

The most significant benefit of this architecture is **Personal Memory** consistency. 

As the agent interacts with you, it saves important facts about you with the `memory_write` tool (`scope: personal`). 
Previously, if you wrote from Discord, the system saved the memory to a file like `memory/user_discord_123.md`. If you wrote from MS Teams the next day, the bot would create a separate file like `memory/user_msteams_abc.md` and treat you as a completely new, unrecognized user.

Now:
- Regardless of the channel, `AccessGate` identifies you and passes your **Unified ID** to the Agent.
- Personal memory is stored as records in the database under your Unified ID (an unpaired user's platform ID). Every agent reads the same records, whichever channel you write from.
- A personal file from older versions, `user_.md` in `~/.goclaw/memory/personal/` (or in an agent's workspace `memory/` folder), is still read, but only while you have no personal records in the database.
- Personal memory is left out of cron, SOP and webhook runs, and those runs cannot write it.
- As a result, even if you switch between different devices or communication apps, the bot retains a consolidated memory of your history and tasks.

## 5. Administrative Tools

Managing these user records is straightforward and can be done in two ways:

1. **Web Dashboard:** Access the application using a web browser (e.g., `http://localhost:8080`) and go to the "Pairing" menu. You will see verification codes for new users. Simply click the "Approve" button to verify their accounts and update their roles.
2. **Command Line (Terminal):** To manage pairing directly from the terminal. The running gateway keeps the pairing files in memory, so a change made here while it runs is not seen until the gateway restarts, and the gateway's next write can drop it. While the gateway runs, approve on the Web UI's Pairing tab or with `POST /api/pairing` (gateway token, body `{"action": "approve", "code": "ABCD1234", "role": "admin"}`).
  - `goclaw pairing list discord` (List registered Discord users)
  - `goclaw pairing pending discord` (View pending authorization codes)
  - `goclaw pairing approve ABCD1234 --role=admin` (Approve a code and assign the admin role)
  - `goclaw pairing link u_123456 discord 9876543210` (Manually link an account from another channel to a Unified ID)
  - `goclaw pairing revoke u_123456` (Revoke authorization for a unified user)
  
With this architecture, the system operates as a personal assistant tailored directly to your context.

================================================================================
# FILE: https://goclaw.ulab.mn/docs/architecture/concepts/model-failover.md
================================================================================

# Model Failover

Every agent is served through `fallbackAgent` (`internal/agent/fallback_agent.go`): its own
provider first, then each `provider/model` of its fallback list in order, when the provider fails
before the reply has used a tool. The list is `ai.fallback`, or the agent's own
`agents.list[].fallback`, which replaces it (`[]` means none). It is read on each request, so a
config reload applies to the next one. [Provider Fallback](../provider_fallback.md) has the
configuration and the full rules; this page explains the moving parts.

Before a request moves on, the same provider is asked again: both agent loops retry an overloaded
call twice and a network failure once, for every provider (`internal/agent/llm_retry.go`).

## How It Works

```mermaid
flowchart TD
    REQ["Request"] --> P0["Agent's Provider"]
    P0 -->|Success| RES["Response"]
    P0 -->|"Provider-side failure
(no tool has run)"| F1["Fallback #1"]
    F1 -->|Success| RES
    F1 -->|"Provider-side failure
(no tool has run)"| F2["Fallback #2"]
    F2 -->|Success| RES
    F2 -->|"Provider-side failure
(no tool has run)"| ERR["Error
Names every provider's failure"]
```

A failure after a tool started is not passed on: the next provider would run the tool again.
Text the failed provider had streamed is discarded (`EventBufferReset`).

## Cooldown System

When a provider fails, it enters a cooldown period. The cooldown duration uses exponential backoff:

| Error Count | Cooldown Duration |
|-------------|-------------------|
| 1st failure | 1 minute |
| 2nd failure | 5 minutes |
| 3rd failure | 25 minutes |
| 4th+ failure | 1 hour (maximum) |

**Formula**: `60s * 5^min(errorCount - 1, 3)`

Every failure reason gets the same cooldown. A provider in cooldown is skipped while others
remain; the last candidate is always tried.

## Counter Reset

Error counters are reset in two scenarios:

1. **On success**: When a provider call succeeds, all error state is cleared immediately
2. **On expiry**: When both `cooldownUntil` and `disabledUntil` have expired, `errorCount` and `failureCounts` are reset to zero

This prevents **infinite escalation** — without reset, stale error counts from hours ago would cause disproportionately long cooldowns on the next failure.

Additionally, if no failures have occurred within the **failure window** (10 minutes) and the previous cooldown has expired, counters are reset before incrementing the new failure.

## Probe Throttling

When the agent's own provider is in cooldown and fallbacks exist, GoClaw **probes** it now and then
to find out whether it has recovered. Probes are throttled to avoid hammering a failing provider:

| Parameter | Value |
|-----------|-------|
| Min probe interval | 30 seconds |
| Probe margin | 2 minutes before cooldown expiry |
| Probe state TTL | 24 hours |
| Max probe keys | 256 |

### Probe Decision Logic

1. Only the agent's **own provider** gets probe attempts (fallback providers in cooldown are skipped)
2. The probe is allowed when `now >= soonestCooldownExpiry - 2 minutes`
3. The probe throttle must be open (at least 30s since last probe for this provider)
4. If both conditions are met, the provider is attempted; otherwise, it's skipped

## Error Classification

Provider errors are classified into structured failure reasons. The last column says whether the
request moves to the next provider:

| Reason | Trigger Patterns | Moves on |
|--------|-----------------|----------|
| `auth` | 401, 403, authentication, unauthorized | yes |
| `billing` | 402, billing, quota exceeded | yes |
| `rate_limit` | 429, rate limit, too many requests | yes |
| `overloaded` | 503, overloaded, capacity | yes |
| `timeout` | timeout, deadline exceeded, ETIMEDOUT | yes |
| `model_not_found` | 404, model not found | yes |
| `format` | 400, invalid, malformed | no |
| `unknown` | Anything else | only a network failure |

## Context Overflow

Context overflow errors (e.g., `context_length_exceeded`) never move to a fallback provider. These
errors indicate the prompt is too large, and switching providers won't help. Neither does a
cancelled request or one over its token budget.

## Attempt Tracking

Each move to another provider is logged with the failures before it, and a fallback whose client
cannot be built (no key) is logged and skipped:

```
[fallback] serving the request with deepseek/deepseek-v4-flash (gemini/gemini-3-flash: 429 Too Many Requests)
[fallback] openai/gpt-4o-mini cannot be used: 
```

When every provider fails, the request fails with `every provider failed: …`, naming each one's
error. `goclaw doctor` shows an `ai_fallback` card per agent.

## Related

- [Provider Fallback](../provider_fallback.md) — Configuration and rules
- [Agent Runtime](agent.md) — How the agent processes messages
- [Configuration](../../user-guide/configuration.md) — Provider configuration

================================================================================
# FILE: https://goclaw.ulab.mn/docs/architecture/concepts/sessions.md
================================================================================

# Sessions

Sessions store the conversation history between users and the agent. Each channel conversation gets its own isolated session.

## Session Keys

Messages from a chat channel (Discord, Teams, Telegram, Facebook) get the key `{channel}:{peerID}`. With more than one agent in `agents.list`, the key starts with the agent the message was routed to: `{agentID}:{channel}:{peerID}`.

| Channel | Example Key | Scope |
|---------|-------------|-------|
| Discord DM | `discord:dm:987654321098765432` | Per user |
| Discord Guild | `discord:123456789012345678:987654321098765432` | Per user in each guild channel (channel ID, then user ID) |
| MS Teams | `msteams:19:0123456789abcdef0123456789abcdef@thread.v2` | Per conversation: one session for everyone in a group chat or channel, one per user in a 1:1 chat |
| Telegram | `telegram:123456789` | Per chat |
| Multi-agent | `agent-ops:msteams:19:0123456789abcdef0123456789abcdef@thread.v2` | As above, per agent |

When `session.identity_links` lists a person's accounts on several channels, all of them use the key of the first account in the list, so they share one session. Each account is written as the channel and the peer ID of its key in the table above, for example `telegram:123456789` or `discord:dm:987654321098765432` (a Discord DM); upper and lower case are not distinguished.

```yaml
session:
  identity_links:
    alice: ["telegram:123456789", "discord:dm:987654321098765432"]
```

Other entry points name their sessions themselves:

| Entry point | Key | Scope |
|-------------|-----|-------|
| Web UI chat (`POST /api/chat`) | the request's `session_id`: `admin-session`, or an `admin-…` conversation the page created | Per named conversation, shared by the browser's tabs |
| WebSocket on channel `web` | `admin-conn-`, or an `admin-…` session ID the client sends | Per connection, or per named conversation |
| WebSocket from other clients | the session ID the client sends, else `p2p::` | Per peer |
| `/api/webhook` (and `goclaw message`) | `p2p::`; channel `web` with the gateway token uses `admin-session`; with `subject_ref` (a delegated cron `agent …` job), `p2p:::for:` | Per peer, and per person a job acts for |
| OpenAI-compatible API | `openai-compat:` | Per client IP |
| App API | `app_api:`, or `app_api:/` (both URL-escaped) | Per employee, and per session the app asks for |
| Webhook (chat) | `webhook:::` | Per object the event is about (merge request, issue, alert); `ts-