Debug & Observability

#	Feature	Status	Notes
10.1	Debug mode (`--debug` / `config.debug`)	Done
10.2	LLM request/response logging	Done	Provider, model, message count, elapsed
10.2a	Verbatim prompt/response capture	Done	`GOCLAW_LOG_PROMPTS=1` (or `true`), which also turns debug mode on — see below
10.2b	Reasoning-model thinking progress	Done	`THINKING n chars` for DeepSeek v4 / o-series and Gemini thinking
10.3	Tool call/result logging	Done	Args, result length, elapsed, errors
10.4	Gateway event logging	Done	Inbound/outbound with session/content info
10.5	Error chain unwrapping	Done	Full `errors.Unwrap` chain in debug mode
10.6	Startup banner	Done	Provider, model, port in box
10.7	SSE log streaming to web UI	Done	Real-time logs in browser
10.8	Live agent traces	Done	A request's trace is on the Traces page from its start — see below

Agent traces while they run

Every request — a chat turn, a webhook, a cron job, a SOP run, a subagent run — has a trace on the Traces page: its model calls, tool calls and retrievals, with their inputs, outputs, tokens and timings, grouped by SOP step.

The list shows the tokens each trace used, prompt and output together, marked from 200k (heavy) and 500k (a SOP run's default limit, sop.max_run_tokens). It can keep only the traces past a token count (50k, 200k, 500k or 1M) and list the ones that used the most first, so a costly run is found without opening traces one by one.

A trace is on the page from the moment its request starts, not only when it ends. This follows how agent tracing systems such as LangSmith work: a run is created when it starts and updated as its children finish. While it runs:

its status is running and its elapsed time counts up;
every model call and tool call that has finished is listed;
a call still in progress — a model streaming, a tool working — is shown as ● running, with how long it has run, and the SOP step it belongs to is marked running too;
the page refreshes itself every few seconds while it is open. Collapsed steps, expanded outputs and the span search stay as they were.

The gateway writes a running trace at most every 3 seconds, and only when something in it changed, so a long SOP run is not rewritten for every span. When the request ends, its final write replaces the running one. A call that was still open when the request ended (a model call that failed, a tool left behind by a cancelled attempt) is kept in the trace, marked unfinished.

If the gateway stops while a request runs, its trace stays as it was last written. At the next start every trace still marked running is set to interrupted, with its last write as its end.

Where a limit cut data

A request whose data a limit cut is tagged ⚠ data cut in the list. In the trace each cut is a ⚠ span, under the SOP step it happened in, that says what was cut, by which setting, and where the rest is:

Span	What happened
answer cut at the output limit	The model stopped its answer at its output limit (`sop.max_output_tokens` in a SOP step, `ai.gemini.max_output_tokens` on Gemini, 4,096 for a chat reply elsewhere). The rest was never written.
earlier output cut in the prompt	Only the first part of an earlier SOP step's output went into this step's prompt (`sop.inline_output_chars`); the step was given the `data_ref` of the rest.
tool results taken out of the context	The context reached `memory.auto_flush_tokens`, and the request's older tool results left it; the model was given the ref of each.
items not run past max_items	A map step left out the items past its `max_items`.
result cells cut	`mcp_call`'s `results` cut long cells; `results_ref` holds them whole.

The same cuts are listed under the step in the SOP's run history, and the log has a [WARN][data-cut] line for each. A tool span's output on this page is itself kept to its first 16,000 characters, and says so: that cut is the trace's, not the data's.

Verbatim prompt capture (LLM testing)

--debug logs the shape of each LLM call — provider, model, message count, token counts, elapsed time — but not the text. To reproduce a bad answer you need the exact bytes that were sent, so there is a separate switch:

GOCLAW_LOG_PROMPTS=1 goclaw

(GOCLAW_LOG_PROMPTS=true works too.) The dump goes through the debug logger, so this switch turns debug mode on as well.

This writes every prompt and every response verbatim, with no truncation, for both agent loops (langchaingo and the native Gemini loop):

[DEBUG][LLM]
╔═══ PROMPT ═══════════════════════════════════════════════
║ [0] role=system chars=8214
╟──────────────────────────────────────────────────────────
You are GoClaw, a helpful agent.
...
║ [1] role=human chars=46
╟──────────────────────────────────────────────────────────
Show the last 7 days' usage as a table
╚══════════════════════════════════════════════════════════

[DEBUG][LLM]
╔═══ RESPONSE ═════════════════════════════════════════════
(no text content)
║ tool_call: memory_search({"query":"usage"})
╚══════════════════════════════════════════════════════════

Tool calls and tool results appear inline as [tool_call …] and [tool_result …], so a multi-iteration turn can be replayed step by step.

Why this is not part of --debug. A prompt contains the entire system instruction, the agent's memory, and every tool result from the turn. That routinely includes credentials, customer data and tens of thousands of characters. Turning on debug logging to diagnose a timeout should not silently start writing all of that to disk — opting in has to be deliberate. Do not leave it enabled in production.

Reasoning models

DeepSeek v4 and the OpenAI o-series stream their chain of thought in a separate reasoning_content delta while the content delta stays empty; Gemini models with thinking send it as thought parts. A measured deepseek-v4-flash turn produced 61 stream events, 25 of them with no content.

GoClaw consumes the reasoning stream separately: empty deltas never reach the channel, and thinking progress is reported on its own line so a long pause is visible rather than looking like a hang.

[DEBUG][LLM]     THINKING 2000 chars (not shown to the user)
[DEBUG][LLM]     THINKING 4000 chars (not shown to the user)
[DEBUG][LLM]     STREAM chunk #1 len=12

The chain of thought itself is not forwarded to the user; only the final answer is delivered.

Observability Platform Tools (SRE Robot)

Native Go tools for SRE/DevOps workflows — all use net/http (no external binaries).

#	Tool	Platform	Status	Notes
10.9	`graylog_query`	Graylog	Done	Full-text log search, time-range filtering, field extraction
10.10	`uptrace_query`	Uptrace	Done	UQL trace queries, project listing, service topology (dependency graph), compressed span listing (`limit` default 15, at most 50)
10.11	`zabbix_query`	Zabbix	Done	Host metrics, triggers via JSON-RPC API; `event.acknowledge` needs `allow_write` on the instance
10.12	`grafana_alerts`	Grafana	Done	Active alerts and dashboards; `annotate` needs `allow_write` on the instance
10.13	`wazuh_query`	Wazuh	Done	FIM/syscheck events, agent security status
10.14	`netbox`	NetBox	Done	IPAM/DCIM lookups (list, get, rack elevation); create, update, patch, delete and the other write actions need `allow_write` on the instance
10.15	`graylog_alert`	Graylog	Done	Registers a Graylog alert (event definition + HTTP notification) and the webhook route `/goclaw/hook/graylog/<key>`; each alert runs a SOP. Human admin in a conversation only, Graylog 5.0+, `allow_write` on the instance — see Graylog Alerts That Run a SOP

Builtin `observability` Skill

Guides the LLM through dynamic topology mapping and multi-source incident correlation:

flowchart LR
    S1["1. grafana_alerts<br/>Discover firing alerts"] --> S2["2. uptrace_query<br/>Map service dependencies"]
    S2 --> S3["3. graylog_query<br/>Search related error logs"]
    S3 --> S4["4. zabbix_query<br/>Check host CPU/memory/load"]
    S4 --> S5["5. wazuh_query<br/>Check security events"]
    S5 --> S6["6. Consolidate<br/>Single cohesive analysis"]

Config

Every tool reads named instances from tools.observability in config.yaml (an agent's own tools: block can override it). With one instance most tools use it without being told; with several, the call names one (instance=prod).

tools:
  observability:
    graylog:
      main:
        url: "https://graylog.example.com/api"
        token: "${secret:GRAYLOG_TOKEN}"
        allow_write: false   # true lets graylog_alert register alerts
    uptrace:
      main:
        dsn: "https://<token>@api.uptrace.dev/<project>"
    zabbix:
      prod:
        url: "https://zabbix.example.com/api_jsonrpc.php"
        token: "${secret:ZABBIX_TOKEN}"
    grafana:
      prod:
        url: "https://grafana.example.com"
        token: "${secret:GRAFANA_TOKEN}"
    wazuh:
      siem:
        url: "https://wazuh.example.com:55000"
        user: "wazuh-api"
        password: "${secret:WAZUH_PASSWORD}"
    netbox:
      dcim:
        url: "https://netbox.example.com"
        token: "${secret:NETBOX_TOKEN}"
        allow_write: false   # true lets netbox create, update and delete

More on each tool's keys and actions is in SRE Tools; the complete set of keys is the config structs in internal/config/config_types.go.