# Observability & Infrastructure Monitoring

GoClaw serves as a unified orchestration hub for infrastructure monitoring and observability. By connecting system agents to your SRE and DevOps monitoring platforms, agents can dynamically query metrics, logs, traces, security events, and hardware inventories to diagnose production incidents.

---

## 1. Core Principles

### Multi-Source Diagnosis
Instead of diagnosing incidents using a single metric, GoClaw agents can query multiple observability platforms in sequence to correlate events. For example, if a database response time spikes:
1. **Grafana Alerts:** Check active alerts on the system.
2. **Graylog Logs:** Search for database error traces.
3. **Zabbix Metrics:** Check host CPU, RAM, and Disk I/O utilization.
4. **Uptrace APM:** Trace request pathways to identify the bottleneck.
5. **Wazuh Security:** Verify if any unauthorized access or security events occurred.
6. **NetBox Asset Inventory:** Retrieve physical host locations, virtual machines, or rack configurations.

### Agent-Friendly Design
All integrated observability tools follow these key principles:
- **Named Instances:** Configure multiple profiles (e.g., `dev`, `staging`, `prod`) for the same tool.
- **Human-Readable Output:** Summarized findings instead of large, raw JSON payloads.
- **Implicit Defaults:** If only a single instance is configured, the agent queries it automatically without needing explicit instructions.
- **Read-Only by Default:** Write actions — a Grafana annotation or alert rule, a NetBox change, registering a Graylog, Zabbix or Wazuh alert or an Uptrace monitor — stay off until the instance sets `allow_write: true`. One exception: `uptrace_query` can always add an Uptrace annotation.
- **Secret Interpolation:** Authentication keys are kept out of plain-text config files using `${secret:KEY}` vault references.

---

## 2. Configuration Setup

All monitoring integrations are configured under the `tools.observability` key in `config.yaml`.

```yaml
tools:
  observability:
    # Grafana Dashboard & Alerts
    grafana:
      prod:
        url: "https://grafana.example.com"
        token: "${secret:GRAFANA_PROD_TOKEN}"
        description: "Production dashboards"

    # Centralized Log Analytics
    graylog:
      prod:
        url: "https://graylog.example.com"
        token: "${secret:GRAYLOG_PROD_TOKEN}"
        # allow_write: true   # lets graylog_alert create and remove alerts; off by default

    # Distributed Tracing (OTel)
    uptrace:
      main:
        dsn: "https://<project_token>@api.uptrace.dev/<project_id>"

    # Infrastructure Monitoring
    zabbix:
      prod:
        url: "https://zabbix.example.com/api_jsonrpc.php"
        token: "${secret:ZABBIX_PROD_TOKEN}"

    # SIEM & Security Events
    wazuh:
      siem:
        url: "https://wazuh.example.com:55000"
        user: "wazuh-api"
        password: "${secret:WAZUH_PASSWORD}"
        insecure: true

    # DCIM / IPAM Inventory
    netbox:
      prod:
        url: "https://netbox.example.com"
        token: "${secret:NETBOX_PROD_TOKEN}"
        allow_write: false
```

**Reaching each host.** The observability tools call their hosts through GoClaw's
[outbound HTTP policy](../tools_and_mcp.md#outbound-http), but the host of an instance's own
`url` (or of the Uptrace `dsn`) needs no entry on `tools.http.allow`, private address included:
you configured it, so each tool may reach it. That opens it to that tool only, not to
`http_request` or the browser. `tools.http.deny` still applies: a host on it is refused with
`domain "…" is blocked by policy` or `address … is blocked by policy`, and so is any other host
a call is sent on to, such as a redirect elsewhere, unless it is on `tools.http.allow`.

**Graylog alerts that run a SOP.** Graylog can also start work on its own: an alert in Graylog
calls the gateway's webhook (`/goclaw/hook/graylog/<key>`), and the gateway runs a SOP with the
alert and its matching messages. A human admin can ask the agent to set one up (tool
`graylog_alert`, Graylog 5.0 or later, `allow_write: true` on the instance), or
[connect Graylog by hand](../automation/graylog-alerts.md#connecting-graylog-without-the-agent) on
the Webhooks page. See [Graylog Alerts That Run a SOP](../automation/graylog-alerts.md).

---

## 3. Incident Response Workflow

When tasked with investigating an issue (e.g., *"Our production API is responding slowly. Please check."*), the agent follows a standardized incident workflow:

```mermaid
flowchart LR
    S1["<b>1. Grafana Alerts</b><br/>List all active system alarms"] --> S2["<b>2. Uptrace APM</b><br/>Trace slow endpoints &amp; topologies"]
    S2 --> S3["<b>3. Graylog Logs</b><br/>Locate stack traces &amp; error logs"]
    S3 --> S4["<b>4. Zabbix</b><br/>Check host resource consumption"]
    S4 --> S5["<b>5. Wazuh</b><br/>Verify if security alerts exist"]
    S5 --> S6["<b>6. NetBox</b><br/>Confirm asset location &amp; status"]
```

The agent correlates the collected data and provides a single, comprehensive incident summary with suggested remediation steps.

---

## 4. Offline Tool Testing (CLI)

Operators can verify monitoring connections directly from the CLI without running the main LLM loop. This is useful for testing API credentials and verifying connectivity:

```bash
# List all active Grafana alerts
goclaw tools exec grafana_alerts instance=prod action=alerts

# Search Graylog logs for system errors in the last 15 minutes
goclaw tools exec graylog_query instance=prod query='level:ERROR' range=900 limit=5

# Query Zabbix hosts directly via RPC methods
goclaw tools exec zabbix_query instance=prod method=host.get params='{"output":"extend"}'

# Search for active hardware devices in NetBox
goclaw tools exec netbox action=list resource_type=devices filters='{"status":"active"}'
```