# Zabbix Problems That Run a SOP Zabbix reports a problem when a trigger fires, and an action decides who is told. With a webhook media type that calls the gateway, a problem on the hosts you choose runs a SOP, with the problem's host, severity, operational data and tags as its input. ```mermaid flowchart LR TRG["Trigger Fires
host groups, severity, tag"] --> ACT["Action 'GoClaw <key>'
sends a message"] ACT --> USR["User goclaw-<key>
its only media"] USR --> MED["Webhook Media Type
script"] MED -->|"POST (X-Token)"| GW["/goclaw/hook/zabbix/<key>"] GW --> SOP["SOP Run on the Agent"] ``` There are two ways to set it up: - **Ask the agent.** An admin describes it in a conversation. The agent's `zabbix_alert` tool makes the Zabbix objects and the route in GoClaw, and removes them together. See [Asking for an alert](#asking-for-an-alert). - **Connect it yourself.** Make a `zabbix` route on the **Webhooks** page, and in Zabbix a webhook media type, a user with that media and an action. See [Connecting Zabbix without the agent](#connecting-zabbix-without-the-agent). --- ## Before you start | What | Where | | :--- | :--- | | Zabbix 5.0 or later, with `allow_write: true` on the instance (only when the agent makes alerts) | `tools.observability.zabbix.`: the **Integrations** page or `config.yaml` | | A **Super admin** user's API token (Zabbix 5.4 and later), or its user and password: only a Super admin may make media types, users and user groups | the instance's `token` (or `user` and `password`) | | An address the Zabbix **server** can reach the gateway at (the server runs the webhook, not the frontend) | `gateway.ingress.url`, and a listener there | | A SOP for the problems to run | the SOPs page, or the agent's `sop_save` tool ([Standard Operating Procedures](sop.md)) | ```yaml gateway: bind: lan ingress: url: "http://192.0.2.10:8080" # as the Zabbix server sees this gateway tools: observability: zabbix: prod: url: "https://zabbix.example.com" token: "${secret:ZABBIX_TOKEN}" allow_write: true # off by default: the agent cannot make or remove alerts ``` --- ## Asking for an alert ``` When a High or worse problem starts on the Linux servers host group, run the server-triage SOP. Run db-check for Average and above problems on Databases tagged service=payments, and again when they resolve. ``` The agent answers with a plan; nothing is created until you confirm it in your next message: ``` Plan 5d20e8a4 — nothing is created yet. Zabbix "prod" (version 7.0.12) gets, all named "GoClaw linux-servers-high": - an action: problems on host group(s) Linux servers at severity high or above - a webhook media type that calls http://192.0.2.10:8080/goclaw/hook/zabbix/linux-servers-high with a new secret only this route knows - a user group with read access to those host groups and no frontend access, and a user goclaw-linux-servers-high in it whose only media is that media type - no other user, group, media type or action is changed; Zabbix has no test call, so the first matching problem shows it working GoClaw: each problem runs SOP "server-triage" (Server triage) as agent main. ``` What the request can set: the host groups (one or more), the lowest severity (`not classified`, `information`, `warning`, `average`, `high` — the default — or `disaster`), an event tag and optionally its value, and whether the SOP also runs when the problem resolves (`on_resolved`). Why four objects: a Zabbix action sends messages to users, through their media, and only about hosts the user may read. So the alert has a user of its own, in a group of its own that may read the chosen host groups and cannot log in to the frontend, whose only media is the alert's webhook media type. Existing users, groups, media types and actions stay as they were. The plan is refused, with the reason, when a host group does not exist, Zabbix already has an object of one of those names, the route key is taken, the instance does not allow writes, or Zabbix is older than 5.0. A token that is not a Super admin's is refused by Zabbix when the first object is made, and nothing is left behind. The route's secret is in the media type's parameters, where Zabbix's Super admins can read it. The media type's script works on every version from 5.0: it uses `HttpRequest` (5.4 and later) or `CurlHttpRequest` (5.0 and 5.2), and each parameter holds a single macro, as Zabbix 7.4 and later require. --- ## What the SOP is given Each problem is the run's trigger data, in an untrusted section of every step. The agent is told: ``` Problem: High CPU on web-01 — PROBLEM Severity: High Host: web-01 (192.0.2.10) Operational data: CPU 97 % Tags: service:api-gateway, env:prod Started: 2026.10.09 10:00:00 Event: 4711 Trigger URL: https://zabbix.example.com/tr_events.php?triggerid=1&eventid=4711 ``` A recovery says `RESOLVED`, with the time it resolved and how long the problem lasted. A problem and its recovery keep one session. The route's events are `problem`, `recovery` and `update`. An alert made for problems filters on `problem`; one made with `on_resolved` lets `recovery` through too. Zabbix retries a message the gateway did not take (busy, unreachable); each retry carries the same `Idempotency-Key`, so a problem is run once. --- ## Listing, checking, removing - **list**: every alert this gateway registered, what it matches, whether its action is still there and enabled in Zabbix, Zabbix's last message for it, and how many problems the route received. - **test**: Zabbix has no test call. This reports Zabbix's last message for the alert (sent, failed and why, or still being retried) beside what the route received. In Zabbix, the media type's **Test** button sends a delivery with no event; the route answers it, counts it and runs nothing. - **delete**: planned and confirmed like a create; removes the action, the user, the user group, the media type and the route. Deleting the route on the **Webhooks** page removes the Zabbix objects too, when the instance allows writes; otherwise it says what to delete in Zabbix. Only a human admin can do this, in a conversation of their own: never a SOP, cron, webhook or subagent run. --- ## Connecting Zabbix without the agent 1. On the **Webhooks** page, create a route of type **Zabbix** with a key, the agent and the SOP. Set **Events** to `problem` to run only when a problem starts. Saving shows the URL, the secret, and the media type's script and parameters to paste. 2. In Zabbix, **Alerts › Media types › Create media type** (before 6.4: **Administration › Media types**): type **Webhook**, the script and parameters from the guide, with `goclaw_url` and `token` set to the route's URL and secret, and message templates for **Problem** and **Problem recovery** (any text: an action's default message needs them). 3. Give a user that media: a user of its own is best, in a group that may read the host groups. 4. Make a trigger action (**Alerts › Actions › Trigger actions**; before 6.4 **Configuration › Actions**) whose operation sends a message to that user through that media type, and, for resolved problems, a recovery operation that does the same. 5. The media type's **Test** button: the route answers it without running the SOP, and its **Hits** go up. A route made by hand with a script of your own still works: the SOP is told the `subject`, `message`, `host` and `severity` fields it sends, and its `event_type` (else `alert`) is what the route's **Events** filter on. ## Related - [Graylog Alerts That Run a SOP](graylog-alerts.md) — the same pattern for Graylog alerts - [Grafana Alert Rules That Run a SOP](grafana-alert-rules.md) — the same for Grafana - [SRE Tools](../observability/sre-tools.md#zabbix) — the `zabbix_query` tool and the instance's configuration - [Standard Operating Procedures](sop.md)