# SRE Observability Tools

GoClaw includes SRE tools that call observability APIs directly through Go HTTP clients. They do not require external CLIs.

Configured credentials are read from `tools.observability` in `config.yaml`, or, for an agent with its own `tools.observability` in its `agents.list` entry, from that: each agent's calls go to its own instances. Agents should call these tools directly and should not search memory for tokens.

The tools call their hosts through the [outbound HTTP policy](../tools_and_mcp.md#outbound-http).
An instance's own host needs no entry on `tools.http.allow`, private address included; a host on
`tools.http.deny` (or the agent's own deny list) is refused with `… is blocked by policy`. See
[Configuration Setup](index.md#2-configuration-setup).

## Config Shape

```yaml
tools:
  observability:
    grafana:
      prod:
        url: "https://grafana.example.com"
        token: "${secret:GRAFANA_TOKEN}"
        # user: "viewer"                       # basic auth, used when no token is set
        # password: "${secret:GRAFANA_PASSWORD}"
        # allow_write: true   # lets action=annotate write annotations, and grafana_alert_rule make rules; off by default
        description: "Production dashboards"

    graylog:
      prod:
        url: "https://graylog.example.com/api"
        token: "${secret:GRAYLOG_TOKEN}"
        # insecure: true      # skip TLS verification for a self-signed Graylog
        # allow_write: true   # lets graylog_alert create and remove alerts; off by default

    uptrace:
      main:
        url: "https://api.uptrace.dev"
        token: "${secret:UPTRACE_TOKEN}"
        project_id: "1"       # default for project-scoped actions
        # allow_write: true   # lets uptrace_monitor make monitors; off by default (annotations need no flag)
        # or, instead of url + token + project_id:
        # dsn: "https://<project_token>@api.uptrace.dev/<project_id>"

    zabbix:
      prod:
        url: "https://zabbix.example.com/api_jsonrpc.php"
        token: "${secret:ZABBIX_TOKEN}"       # API token (Zabbix 5.4+)
        # user: "api-user"                    # older Zabbix: user.login, used when no token is set
        # password: "${secret:ZABBIX_PASSWORD}"
        # allow_write: true   # lets event.acknowledge update problems, and zabbix_alert make alerts; off by default

    wazuh:
      siem:
        url: "https://wazuh.example.com:55000"
        user: "wazuh-api"
        password: "${secret:WAZUH_PASSWORD}"
        insecure: false
        # indexer_url: "https://wazuh.example.com:9200"   # the indexer, for wazuh_alert
        # indexer_user: "admin"
        # indexer_password: "${secret:WAZUH_INDEXER_PASSWORD}"
        # allow_write: true   # lets wazuh_alert make alert hooks in the indexer; off by default
```

Use `instance=<name>` when more than one instance is configured. If there is exactly one instance, most tools can infer it.

## Grafana

**Tool:** `grafana_alerts`

Despite the name, this tool supports several Grafana API actions.

| Action | Purpose |
|---|---|
| `alerts` | List active firing alerts |
| `dashboards` | Search/list dashboards |
| `annotate` | Create a graph annotation; requires `allow_write: true` |

```bash
goclaw tools exec grafana_alerts instance=prod action=alerts
goclaw tools exec grafana_alerts instance=prod action=dashboards
goclaw tools exec grafana_alerts instance=prod action=annotate text='Deployed v1.2.3' tags='["deploy","api"]'
```

To have an alert run a SOP, an admin asks the agent for an alert rule (`grafana_alert_rule`): see
[Grafana Alert Rules That Run a SOP](../automation/grafana-alert-rules.md).

## Graylog

**Tool:** `graylog_query`

The tool detects Graylog major versions and switches API strategy for v4, v5, v6, and v7.

| Action | Purpose |
|---|---|
| `search` | Search logs with a relative range in seconds |
| `search_absolute` | Search logs with explicit `from` and `to` timestamps |
| `terms_stats` | Count top values for a field |
| `streams_list` | List available streams |
| `version` | Show server version/system info |

```bash
goclaw tools exec graylog_query instance=prod action=search query='level:ERROR' range=3600 limit=50
goclaw tools exec graylog_query instance=prod action=search_absolute query='source:gateway' from=2026-06-20T00:00:00Z to=2026-06-20T12:00:00Z
goclaw tools exec graylog_query instance=prod action=terms_stats query='level:ERROR' field=source range=86400 limit=20
goclaw tools exec graylog_query instance=prod action=streams_list
```

Graylog query examples:

```text
level:ERROR
source:gateway AND level:WARN
(timeout OR connection) AND level:ERROR
message:*database*
response_time:[100 TO 500]
```

### Alerts that run a SOP

**Tool:** `graylog_alert`

A human admin can ask the agent, in a conversation, to register a Graylog alert: for example
*"run the incident-triage SOP when service payments logs more than 50 errors in 5 minutes"*. The
agent creates the event definition and the HTTP notification in Graylog and the webhook route in
GoClaw (`/goclaw/hook/graylog/<key>`). Each alert then runs the SOP. The agent shows a plan
first and acts only when the admin confirms it in a later message.

It needs:

- Graylog 5.0 or later;
- `allow_write: true` on the Graylog instance (without it the agent only searches);
- `gateway.ingress.url`, the address Graylog reaches the gateway at.

The tool is refused in SOP steps, scheduled jobs, webhook-started runs and subagents. The same
route can also be [set up by hand](../automation/graylog-alerts.md#connecting-graylog-without-the-agent)
on the Webhooks page, which does not need `allow_write`. See
[Graylog Alerts That Run a SOP](../automation/graylog-alerts.md).

## Uptrace

**Tool:** `uptrace_query`

Start with `projects`, then pass `project_id` to project-scoped actions. An instance with a
`project_id` in its config (or in its `dsn`) uses it when the call gives none.

| Area | Actions |
|---|---|
| Discovery | `projects` |
| Monitoring | `monitors`, `monitor`, `notifications`, `alerts` |
| Tracing | `spans`, `query`, `timeseries` |
| Performance | `percentiles`, `topology` |
| Dashboards | `dashboards`, `dashboard`, `dashboard_yaml` |
| Attributes | `attributes`, `systems` |
| Metrics | `metrics_timeseries` |
| Write | `annotate` |

```bash
goclaw tools exec uptrace_query instance=main action=projects
goclaw tools exec uptrace_query instance=main action=monitors project_id=1
goclaw tools exec uptrace_query instance=main action=query project_id=1 query='perMin(count()) | where _status = "error" | group by service.name'
goclaw tools exec uptrace_query instance=main action=spans project_id=1 trace_id=abc123defabc
goclaw tools exec uptrace_query instance=main action=topology project_id=1
```

Time parameters are `time_gte` and `time_lte`. They accept RFC3339, `YYYY-MM-DD`, or relative values such as `now-1h`, `now-30m`, `now-2d`, and `now`.

To have an alert run a SOP, an admin asks the agent for a monitor (`uptrace_monitor`): see
[Uptrace Monitors That Run a SOP](../automation/uptrace-monitors.md).

## Zabbix

**Tool:** `zabbix_query`

This tool is a JSON-RPC client. Use the Zabbix API method name and pass `params` as a JSON object. It accepts the read methods (any `<object>.get`, and `apiinfo.version`) and one write, `event.acknowledge`, which needs `allow_write: true` on the instance; any other method is refused.

It works with Zabbix 5.0 and later. It asks the API its version first (`apiinfo.version`, cached for 10 minutes) and sends the credentials the way that version takes them: the `Authorization: Bearer` header from 6.4, the request's `auth` field before. An API `token` needs Zabbix 5.4 or later; on 5.0 and 5.2 set `user` and `password`.

```bash
goclaw tools exec zabbix_query instance=prod method=host.get params='{"output":"extend"}'
goclaw tools exec zabbix_query instance=prod method=problem.get params='{"output":"extend","recent":true}'
goclaw tools exec zabbix_query instance=prod method=trigger.get params='{"output":"extend","filter":{"value":"1"}}'
goclaw tools exec zabbix_query instance=prod method=item.get params='{"hostids":"10084","output":"extend","search":{"key_":"cpu"}}'
```

Common methods:

| Method | Purpose |
|---|---|
| `host.get` | Host inventory and status |
| `problem.get` | Current problems |
| `trigger.get` | Trigger definitions and states |
| `item.get` | Metric items |
| `history.get` | Historical metric values |
| `event.get` | Historical events |
| `alert.get` | Recent alerts |
| `hostgroup.get` | Host groups |
| `event.acknowledge` | Acknowledge, comment on or close a problem; requires `allow_write: true` |

```bash
goclaw tools exec zabbix_query instance=prod method=event.acknowledge params='{"eventids":"12345","action":6,"message":"Investigating"}'
```

To have problems run a SOP, an admin asks the agent for an alert (`zabbix_alert`): see
[Zabbix Problems That Run a SOP](../automation/zabbix-alerts.md).

## Wazuh

**Tool:** `wazuh_query`

The current Wazuh tool accepts an API endpoint and handles JWT authentication internally. It is intentionally generic so callers can use any supported Wazuh endpoint.

```bash
goclaw tools exec wazuh_query instance=siem endpoint=/agents/summary/status
goclaw tools exec wazuh_query instance=siem endpoint=/agents
goclaw tools exec wazuh_query instance=siem endpoint='/syscheck/000?summary=yes'
goclaw tools exec wazuh_query instance=siem endpoint=/vulnerability/000
```

To have Wazuh alerts run a SOP, an admin asks the agent for an alert hook (`wazuh_alert`): see
[Wazuh Alerts That Run a SOP](../automation/wazuh-alerts.md).

## Benchmarking

Use `goclaw tools bench` to measure direct tool latency.

```bash
goclaw tools bench grafana_alerts instance=prod action=alerts --runs=10
goclaw tools bench graylog_query instance=prod action=search query='level:ERROR' --runs=5
goclaw tools bench zabbix_query instance=prod method=host.get params='{"output":"extend"}' --runs=10
```

## Implementation Files

- `internal/agent/tool_grafana.go`
- `internal/agent/tool_graylog.go`
- `internal/agent/tool_uptrace.go`
- `internal/agent/tool_zabbix.go`
- `internal/agent/tool_wazuh.go`
- `internal/config/config_types.go`