Zabbix Problems That Run a SOP

Zabbix reports a problem when a trigger fires, and an action decides who is told. With a webhook media type that calls the gateway, a problem on the hosts you choose runs a SOP, with the problem's host, severity, operational data and tags as its input.

flowchart LR
    TRG["<b>Trigger Fires</b><br/>host groups, severity, tag"] --> ACT["<b>Action 'GoClaw &lt;key&gt;'</b><br/>sends a message"]
    ACT --> USR["<b>User goclaw-&lt;key&gt;</b><br/>its only media"]
    USR --> MED["<b>Webhook Media Type</b><br/>script"]
    MED -->|"POST (X-Token)"| GW["<b>/goclaw/hook/zabbix/&lt;key&gt;</b>"]
    GW --> SOP["<b>SOP Run on the Agent</b>"]

There are two ways to set it up:

Ask the agent. An admin describes it in a conversation. The agent's zabbix_alert tool makes the Zabbix objects and the route in GoClaw, and removes them together. See Asking for an alert.
Connect it yourself. Make a zabbix route on the Webhooks page, and in Zabbix a webhook media type, a user with that media and an action. See Connecting Zabbix without the agent.

Before you start

What	Where
Zabbix 5.0 or later, with `allow_write: true` on the instance (only when the agent makes alerts)	`tools.observability.zabbix.<name>`: the Integrations page or `config.yaml`
A Super admin user's API token (Zabbix 5.4 and later), or its user and password: only a Super admin may make media types, users and user groups	the instance's `token` (or `user` and `password`)
An address the Zabbix server can reach the gateway at (the server runs the webhook, not the frontend)	`gateway.ingress.url`, and a listener there
A SOP for the problems to run	the SOPs page, or the agent's `sop_save` tool (Standard Operating Procedures)

gateway:
  bind: lan
  ingress:
    url: "http://192.0.2.10:8080"     # as the Zabbix server sees this gateway

tools:
  observability:
    zabbix:
      prod:
        url: "https://zabbix.example.com"
        token: "${secret:ZABBIX_TOKEN}"
        allow_write: true             # off by default: the agent cannot make or remove alerts

Asking for an alert

When a High or worse problem starts on the Linux servers host group, run the server-triage SOP.
Run db-check for Average and above problems on Databases tagged service=payments, and again when they resolve.

The agent answers with a plan; nothing is created until you confirm it in your next message:

Plan 5d20e8a4 — nothing is created yet.

Zabbix "prod" (version 7.0.12) gets, all named "GoClaw linux-servers-high":
- an action: problems on host group(s) Linux servers at severity high or above
- a webhook media type that calls http://192.0.2.10:8080/goclaw/hook/zabbix/linux-servers-high with a new secret only this route knows
- a user group with read access to those host groups and no frontend access, and a user goclaw-linux-servers-high in it whose only media is that media type
- no other user, group, media type or action is changed; Zabbix has no test call, so the first matching problem shows it working
GoClaw: each problem runs SOP "server-triage" (Server triage) as agent main.

What the request can set: the host groups (one or more), the lowest severity (not classified, information, warning, average, high — the default — or disaster), an event tag and optionally its value, and whether the SOP also runs when the problem resolves (on_resolved).

Why four objects: a Zabbix action sends messages to users, through their media, and only about hosts the user may read. So the alert has a user of its own, in a group of its own that may read the chosen host groups and cannot log in to the frontend, whose only media is the alert's webhook media type. Existing users, groups, media types and actions stay as they were.

The plan is refused, with the reason, when a host group does not exist, Zabbix already has an object of one of those names, the route key is taken, the instance does not allow writes, or Zabbix is older than 5.0. A token that is not a Super admin's is refused by Zabbix when the first object is made, and nothing is left behind.

The route's secret is in the media type's parameters, where Zabbix's Super admins can read it.

The media type's script works on every version from 5.0: it uses HttpRequest (5.4 and later) or CurlHttpRequest (5.0 and 5.2), and each parameter holds a single macro, as Zabbix 7.4 and later require.

What the SOP is given

Each problem is the run's trigger data, in an untrusted section of every step. The agent is told:

Problem: High CPU on web-01 — PROBLEM
Severity: High
Host: web-01 (192.0.2.10)
Operational data: CPU 97 %
Tags: service:api-gateway, env:prod
Started: 2026.10.09 10:00:00
Event: 4711
Trigger URL: https://zabbix.example.com/tr_events.php?triggerid=1&eventid=4711

A recovery says RESOLVED, with the time it resolved and how long the problem lasted. A problem and its recovery keep one session.

The route's events are problem, recovery and update. An alert made for problems filters on problem; one made with on_resolved lets recovery through too.

Zabbix retries a message the gateway did not take (busy, unreachable); each retry carries the same Idempotency-Key, so a problem is run once.

Listing, checking, removing

list: every alert this gateway registered, what it matches, whether its action is still there and enabled in Zabbix, Zabbix's last message for it, and how many problems the route received.
test: Zabbix has no test call. This reports Zabbix's last message for the alert (sent, failed and why, or still being retried) beside what the route received. In Zabbix, the media type's Test button sends a delivery with no event; the route answers it, counts it and runs nothing.
delete: planned and confirmed like a create; removes the action, the user, the user group, the media type and the route. Deleting the route on the Webhooks page removes the Zabbix objects too, when the instance allows writes; otherwise it says what to delete in Zabbix.

Only a human admin can do this, in a conversation of their own: never a SOP, cron, webhook or subagent run.

Connecting Zabbix without the agent

On the Webhooks page, create a route of type Zabbix with a key, the agent and the SOP. Set Events to problem to run only when a problem starts. Saving shows the URL, the secret, and the media type's script and parameters to paste.
In Zabbix, Alerts › Media types › Create media type (before 6.4: Administration › Media types): type Webhook, the script and parameters from the guide, with goclaw_url and token set to the route's URL and secret, and message templates for Problem and Problem recovery (any text: an action's default message needs them).
Give a user that media: a user of its own is best, in a group that may read the host groups.
Make a trigger action (Alerts › Actions › Trigger actions; before 6.4 Configuration › Actions) whose operation sends a message to that user through that media type, and, for resolved problems, a recovery operation that does the same.
The media type's Test button: the route answers it without running the SOP, and its Hits go up.

A route made by hand with a script of your own still works: the SOP is told the subject, message, host and severity fields it sends, and its event_type (else alert) is what the route's Events filter on.

Graylog Alerts That Run a SOP — the same pattern for Graylog alerts
Grafana Alert Rules That Run a SOP — the same for Grafana
SRE Tools — the zabbix_query tool and the instance's configuration
Standard Operating Procedures