🏠 Home ← Docs Index
Open Beta 📖 Interactive Reader Raw .md llms.txt

SRE Observability Tools

GoClaw includes SRE tools that call observability APIs directly through Go HTTP clients. They do not require external CLIs.

Configured credentials are read from tools.observability in config.yaml, or, for an agent with its own tools.observability in its agents.list entry, from that: each agent's calls go to its own instances. Agents should call these tools directly and should not search memory for tokens.

The tools call their hosts through the outbound HTTP policy. An instance's own host needs no entry on tools.http.allow, private address included; a host on tools.http.deny (or the agent's own deny list) is refused with … is blocked by policy. See Configuration Setup.

Config Shape

tools:
  observability:
    grafana:
      prod:
        url: "https://grafana.example.com"
        token: "${secret:GRAFANA_TOKEN}"
        # user: "viewer"                       # basic auth, used when no token is set
        # password: "${secret:GRAFANA_PASSWORD}"
        # allow_write: true   # lets action=annotate write annotations, and grafana_alert_rule make rules; off by default
        description: "Production dashboards"

    graylog:
      prod:
        url: "https://graylog.example.com/api"
        token: "${secret:GRAYLOG_TOKEN}"
        # insecure: true      # skip TLS verification for a self-signed Graylog
        # allow_write: true   # lets graylog_alert create and remove alerts; off by default

    uptrace:
      main:
        url: "https://api.uptrace.dev"
        token: "${secret:UPTRACE_TOKEN}"
        project_id: "1"       # default for project-scoped actions
        # allow_write: true   # lets uptrace_monitor make monitors; off by default (annotations need no flag)
        # or, instead of url + token + project_id:
        # dsn: "https://<project_token>@api.uptrace.dev/<project_id>"

    zabbix:
      prod:
        url: "https://zabbix.example.com/api_jsonrpc.php"
        token: "${secret:ZABBIX_TOKEN}"       # API token (Zabbix 5.4+)
        # user: "api-user"                    # older Zabbix: user.login, used when no token is set
        # password: "${secret:ZABBIX_PASSWORD}"
        # allow_write: true   # lets event.acknowledge update problems, and zabbix_alert make alerts; off by default

    wazuh:
      siem:
        url: "https://wazuh.example.com:55000"
        user: "wazuh-api"
        password: "${secret:WAZUH_PASSWORD}"
        insecure: false
        # indexer_url: "https://wazuh.example.com:9200"   # the indexer, for wazuh_alert
        # indexer_user: "admin"
        # indexer_password: "${secret:WAZUH_INDEXER_PASSWORD}"
        # allow_write: true   # lets wazuh_alert make alert hooks in the indexer; off by default

Use instance=<name> when more than one instance is configured. If there is exactly one instance, most tools can infer it.

Grafana

Tool: grafana_alerts

Despite the name, this tool supports several Grafana API actions.

Action Purpose
alerts List active firing alerts
dashboards Search/list dashboards
annotate Create a graph annotation; requires allow_write: true 
goclaw tools exec grafana_alerts instance=prod action=alerts
goclaw tools exec grafana_alerts instance=prod action=dashboards
goclaw tools exec grafana_alerts instance=prod action=annotate text='Deployed v1.2.3' tags='["deploy","api"]'

To have an alert run a SOP, an admin asks the agent for an alert rule (grafana_alert_rule): see Grafana Alert Rules That Run a SOP.

Graylog

Tool: graylog_query

The tool detects Graylog major versions and switches API strategy for v4, v5, v6, and v7.

Action Purpose
search Search logs with a relative range in seconds
search_absolute Search logs with explicit from and to timestamps
terms_stats Count top values for a field
streams_list List available streams
version Show server version/system info 
goclaw tools exec graylog_query instance=prod action=search query='level:ERROR' range=3600 limit=50
goclaw tools exec graylog_query instance=prod action=search_absolute query='source:gateway' from=2026-06-20T00:00:00Z to=2026-06-20T12:00:00Z
goclaw tools exec graylog_query instance=prod action=terms_stats query='level:ERROR' field=source range=86400 limit=20
goclaw tools exec graylog_query instance=prod action=streams_list

Graylog query examples:

level:ERROR
source:gateway AND level:WARN
(timeout OR connection) AND level:ERROR
message:*database*
response_time:[100 TO 500]

Alerts that run a SOP

Tool: graylog_alert

A human admin can ask the agent, in a conversation, to register a Graylog alert: for example "run the incident-triage SOP when service payments logs more than 50 errors in 5 minutes". The agent creates the event definition and the HTTP notification in Graylog and the webhook route in GoClaw (/goclaw/hook/graylog/<key>). Each alert then runs the SOP. The agent shows a plan first and acts only when the admin confirms it in a later message.

It needs:

The tool is refused in SOP steps, scheduled jobs, webhook-started runs and subagents. The same route can also be set up by hand on the Webhooks page, which does not need allow_write. See Graylog Alerts That Run a SOP.

Uptrace

Tool: uptrace_query

Start with projects, then pass project_id to project-scoped actions. An instance with a project_id in its config (or in its dsn) uses it when the call gives none.

Area Actions
Discovery projects
Monitoring monitors, monitor, notifications, alerts
Tracing spans, query, timeseries
Performance percentiles, topology
Dashboards dashboards, dashboard, dashboard_yaml
Attributes attributes, systems
Metrics metrics_timeseries
Write annotate 
goclaw tools exec uptrace_query instance=main action=projects
goclaw tools exec uptrace_query instance=main action=monitors project_id=1
goclaw tools exec uptrace_query instance=main action=query project_id=1 query='perMin(count()) | where _status = "error" | group by service.name'
goclaw tools exec uptrace_query instance=main action=spans project_id=1 trace_id=abc123defabc
goclaw tools exec uptrace_query instance=main action=topology project_id=1

Time parameters are time_gte and time_lte. They accept RFC3339, YYYY-MM-DD, or relative values such as now-1h, now-30m, now-2d, and now.

To have an alert run a SOP, an admin asks the agent for a monitor (uptrace_monitor): see Uptrace Monitors That Run a SOP.

Zabbix

Tool: zabbix_query

This tool is a JSON-RPC client. Use the Zabbix API method name and pass params as a JSON object. It accepts the read methods (any <object>.get, and apiinfo.version) and one write, event.acknowledge, which needs allow_write: true on the instance; any other method is refused.

It works with Zabbix 5.0 and later. It asks the API its version first (apiinfo.version, cached for 10 minutes) and sends the credentials the way that version takes them: the Authorization: Bearer header from 6.4, the request's auth field before. An API token needs Zabbix 5.4 or later; on 5.0 and 5.2 set user and password.

goclaw tools exec zabbix_query instance=prod method=host.get params='{"output":"extend"}'
goclaw tools exec zabbix_query instance=prod method=problem.get params='{"output":"extend","recent":true}'
goclaw tools exec zabbix_query instance=prod method=trigger.get params='{"output":"extend","filter":{"value":"1"}}'
goclaw tools exec zabbix_query instance=prod method=item.get params='{"hostids":"10084","output":"extend","search":{"key_":"cpu"}}'

Common methods:

Method Purpose
host.get Host inventory and status
problem.get Current problems
trigger.get Trigger definitions and states
item.get Metric items
history.get Historical metric values
event.get Historical events
alert.get Recent alerts
hostgroup.get Host groups
event.acknowledge Acknowledge, comment on or close a problem; requires allow_write: true 
goclaw tools exec zabbix_query instance=prod method=event.acknowledge params='{"eventids":"12345","action":6,"message":"Investigating"}'

To have problems run a SOP, an admin asks the agent for an alert (zabbix_alert): see Zabbix Problems That Run a SOP.

Wazuh

Tool: wazuh_query

The current Wazuh tool accepts an API endpoint and handles JWT authentication internally. It is intentionally generic so callers can use any supported Wazuh endpoint.

goclaw tools exec wazuh_query instance=siem endpoint=/agents/summary/status
goclaw tools exec wazuh_query instance=siem endpoint=/agents
goclaw tools exec wazuh_query instance=siem endpoint='/syscheck/000?summary=yes'
goclaw tools exec wazuh_query instance=siem endpoint=/vulnerability/000

To have Wazuh alerts run a SOP, an admin asks the agent for an alert hook (wazuh_alert): see Wazuh Alerts That Run a SOP.

Benchmarking

Use goclaw tools bench to measure direct tool latency.

goclaw tools bench grafana_alerts instance=prod action=alerts --runs=10
goclaw tools bench graylog_query instance=prod action=search query='level:ERROR' --runs=5
goclaw tools bench zabbix_query instance=prod method=host.get params='{"output":"extend"}' --runs=10

Implementation Files