# Debug & Observability | # | Feature | Status | Notes | |---|---------|--------|-------| | 10.1 | Debug mode (`--debug` / `config.debug`) | Done | | | 10.2 | LLM request/response logging | Done | Provider, model, message count, elapsed | | 10.2a | Verbatim prompt/response capture | Done | `GOCLAW_LOG_PROMPTS=1` (or `true`), which also turns debug mode on — see below | | 10.2b | Reasoning-model thinking progress | Done | `THINKING n chars` for DeepSeek v4 / o-series and Gemini thinking | | 10.3 | Tool call/result logging | Done | Args, result length, elapsed, errors | | 10.4 | Gateway event logging | Done | Inbound/outbound with session/content info | | 10.5 | Error chain unwrapping | Done | Full `errors.Unwrap` chain in debug mode | | 10.6 | Startup banner | Done | Provider, model, port in box | | 10.7 | SSE log streaming to web UI | Done | Real-time logs in browser | | 10.8 | Live agent traces | Done | A request's trace is on the Traces page from its start — see below | ## Agent traces while they run Every request — a chat turn, a webhook, a cron job, a SOP run, a subagent run — has a trace on the **Traces** page: its model calls, tool calls and retrievals, with their inputs, outputs, tokens and timings, grouped by SOP step. The list shows the tokens each trace used, prompt and output together, marked from 200k (heavy) and 500k (a SOP run's default limit, `sop.max_run_tokens`). It can keep only the traces past a token count (50k, 200k, 500k or 1M) and list the ones that used the most first, so a costly run is found without opening traces one by one. A trace is on the page from the moment its request starts, not only when it ends. This follows how agent tracing systems such as LangSmith work: a run is created when it starts and updated as its children finish. While it runs: - its status is **running** and its elapsed time counts up; - every model call and tool call that has finished is listed; - a call still in progress — a model streaming, a tool working — is shown as **● running**, with how long it has run, and the SOP step it belongs to is marked running too; - the page refreshes itself every few seconds while it is open. Collapsed steps, expanded outputs and the span search stay as they were. The gateway writes a running trace at most every 3 seconds, and only when something in it changed, so a long SOP run is not rewritten for every span. When the request ends, its final write replaces the running one. A call that was still open when the request ended (a model call that failed, a tool left behind by a cancelled attempt) is kept in the trace, marked unfinished. If the gateway stops while a request runs, its trace stays as it was last written. At the next start every trace still marked running is set to **interrupted**, with its last write as its end. ### Where a limit cut data A request whose data a limit cut is tagged **⚠ data cut** in the list. In the trace each cut is a ⚠ span, under the SOP step it happened in, that says what was cut, by which setting, and where the rest is: | Span | What happened | | :--- | :--- | | answer cut at the output limit | The model stopped its answer at its output limit (`sop.max_output_tokens` in a SOP step, `ai.gemini.max_output_tokens` on Gemini, 4,096 for a chat reply elsewhere). The rest was never written. | | earlier output cut in the prompt | Only the first part of an earlier SOP step's output went into this step's prompt (`sop.inline_output_chars`); the step was given the `data_ref` of the rest. | | tool results taken out of the context | The context reached `memory.auto_flush_tokens`, and the request's older tool results left it; the model was given the ref of each. | | items not run past max_items | A map step left out the items past its `max_items`. | | result cells cut | `mcp_call`'s `results` cut long cells; `results_ref` holds them whole. | The same cuts are listed under the step in the SOP's run history, and the log has a `[WARN][data-cut]` line for each. A tool span's output on this page is itself kept to its first 16,000 characters, and says so: that cut is the trace's, not the data's. ## Verbatim prompt capture (LLM testing) `--debug` logs the *shape* of each LLM call — provider, model, message count, token counts, elapsed time — but not the text. To reproduce a bad answer you need the exact bytes that were sent, so there is a separate switch: ```bash GOCLAW_LOG_PROMPTS=1 goclaw ``` (`GOCLAW_LOG_PROMPTS=true` works too.) The dump goes through the debug logger, so this switch turns debug mode on as well. This writes every prompt and every response verbatim, with no truncation, for both agent loops (langchaingo and the native Gemini loop): ``` [DEBUG][LLM] ╔═══ PROMPT ═══════════════════════════════════════════════ ║ [0] role=system chars=8214 ╟────────────────────────────────────────────────────────── You are GoClaw, a helpful agent. ... ║ [1] role=human chars=46 ╟────────────────────────────────────────────────────────── Show the last 7 days' usage as a table ╚══════════════════════════════════════════════════════════ [DEBUG][LLM] ╔═══ RESPONSE ═════════════════════════════════════════════ (no text content) ║ tool_call: memory_search({"query":"usage"}) ╚══════════════════════════════════════════════════════════ ``` Tool calls and tool results appear inline as `[tool_call …]` and `[tool_result …]`, so a multi-iteration turn can be replayed step by step. > **Why this is not part of `--debug`.** A prompt contains the entire system > instruction, the agent's memory, and every tool result from the turn. That > routinely includes credentials, customer data and tens of thousands of > characters. Turning on debug logging to diagnose a timeout should not > silently start writing all of that to disk — opting in has to be deliberate. > Do not leave it enabled in production. ## Reasoning models DeepSeek v4 and the OpenAI o-series stream their chain of thought in a separate `reasoning_content` delta while the content delta stays empty; Gemini models with thinking send it as thought parts. A measured `deepseek-v4-flash` turn produced 61 stream events, 25 of them with no content. GoClaw consumes the reasoning stream separately: empty deltas never reach the channel, and thinking progress is reported on its own line so a long pause is visible rather than looking like a hang. ``` [DEBUG][LLM] THINKING 2000 chars (not shown to the user) [DEBUG][LLM] THINKING 4000 chars (not shown to the user) [DEBUG][LLM] STREAM chunk #1 len=12 ``` The chain of thought itself is not forwarded to the user; only the final answer is delivered. ## Observability Platform Tools (SRE Robot) Native Go tools for SRE/DevOps workflows — all use `net/http` (no external binaries). | # | Tool | Platform | Status | Notes | |---|------|----------|--------|-------| | 10.9 | `graylog_query` | Graylog | Done | Full-text log search, time-range filtering, field extraction | | 10.10 | `uptrace_query` | Uptrace | Done | UQL trace queries, project listing, service topology (dependency graph), compressed span listing (`limit` default 15, at most 50) | | 10.11 | `zabbix_query` | Zabbix | Done | Host metrics, triggers via JSON-RPC API; `event.acknowledge` needs `allow_write` on the instance | | 10.12 | `grafana_alerts` | Grafana | Done | Active alerts and dashboards; `annotate` needs `allow_write` on the instance | | 10.13 | `wazuh_query` | Wazuh | Done | FIM/syscheck events, agent security status | | 10.14 | `netbox` | NetBox | Done | IPAM/DCIM lookups (list, get, rack elevation); create, update, patch, delete and the other write actions need `allow_write` on the instance | | 10.15 | `graylog_alert` | Graylog | Done | Registers a Graylog alert (event definition + HTTP notification) and the webhook route `/goclaw/hook/graylog/`; each alert runs a SOP. Human admin in a conversation only, Graylog 5.0+, `allow_write` on the instance — see [Graylog Alerts That Run a SOP](../user-guide/automation/graylog-alerts.md) | ### Builtin `observability` Skill Guides the LLM through dynamic topology mapping and multi-source incident correlation: ```mermaid flowchart LR S1["1. grafana_alerts
Discover firing alerts"] --> S2["2. uptrace_query
Map service dependencies"] S2 --> S3["3. graylog_query
Search related error logs"] S3 --> S4["4. zabbix_query
Check host CPU/memory/load"] S4 --> S5["5. wazuh_query
Check security events"] S5 --> S6["6. Consolidate
Single cohesive analysis"] ``` ### Config Every tool reads named instances from `tools.observability` in `config.yaml` (an agent's own `tools:` block can override it). With one instance most tools use it without being told; with several, the call names one (`instance=prod`). ```yaml tools: observability: graylog: main: url: "https://graylog.example.com/api" token: "${secret:GRAYLOG_TOKEN}" allow_write: false # true lets graylog_alert register alerts uptrace: main: dsn: "https://@api.uptrace.dev/" zabbix: prod: url: "https://zabbix.example.com/api_jsonrpc.php" token: "${secret:ZABBIX_TOKEN}" grafana: prod: url: "https://grafana.example.com" token: "${secret:GRAFANA_TOKEN}" wazuh: siem: url: "https://wazuh.example.com:55000" user: "wazuh-api" password: "${secret:WAZUH_PASSWORD}" netbox: dcim: url: "https://netbox.example.com" token: "${secret:NETBOX_TOKEN}" allow_write: false # true lets netbox create, update and delete ``` More on each tool's keys and actions is in [SRE Tools](../user-guide/observability/sre-tools.md); the complete set of keys is the config structs in `internal/config/config_types.go`.