Grafana Alert Rules That Run a SOP

Grafana evaluates a query on a schedule, and when the value crosses a threshold for long enough the alert fires. With a webhook contact point that calls the gateway, a firing alert runs a SOP, with the alert's labels, values and annotations as its input.

flowchart LR
    EVAL["<b>Grafana Evaluates the Rule</b><br/>every 60 s, for 5 min"] -->|fires| CP["<b>Rule Contact Point</b><br/>webhook"]
    CP -->|"POST (Bearer)"| GW["<b>/goclaw/hook/grafana/&lt;key&gt;</b>"]
    GW --> SOP["<b>SOP Run on the Agent</b>"]

There are two ways to set it up:

Ask the agent. An admin describes it in a conversation. The agent's grafana_alert_rule tool makes the alert rule, its own contact point and the route in GoClaw, and removes them together. See Asking for an alert rule.
Connect it yourself. Make a grafana route on the Webhooks page, and a webhook contact point in Grafana with the URL and secret the page shows; then send an alert rule to it. See Connecting Grafana without the agent.

Before you start

What	Where
Grafana 11.0 or later, with `allow_write: true` on the instance (only when the agent makes rules)	`tools.observability.grafana.<name>`: the Integrations page or `config.yaml`
A service account token with the Editor role (some older Grafana versions keep the alerting provisioning API from Editors; then Admin)	the instance's `token`
A Prometheus or Loki datasource for the rule's query	Grafana's Connections › Data sources
An address Grafana can reach the gateway at	`gateway.ingress.url`, and a listener there
A SOP for the alerts to run	the SOPs page, or the agent's `sop_save` tool (Standard Operating Procedures)

gateway:
  bind: lan
  ingress:
    url: "http://192.0.2.10:8080"     # as Grafana sees this gateway

tools:
  observability:
    grafana:
      prod:
        url: "https://grafana.example.com"
        token: "${secret:GRAFANA_TOKEN}"
        allow_write: true             # off by default: the agent cannot make or remove rules

The Grafana host needs no entry on tools.http.allow: the Grafana tools may reach the instance you configured.

Asking for an alert rule

When the API's 5xx rate stays above 5 per second, run the api-errors SOP.
  (sum(rate(http_requests_total{code=~"5.."}[5m])) on Prometheus)
Alert when free disk on the reports host drops below 10 GB for 15 minutes, and run disk-cleanup — also when it recovers.

The agent answers with a plan; nothing is created until you confirm it in your next message:

Plan 3e9b07c1 — nothing is created yet.

Grafana "prod" (version 12.2.1) gets an alert rule "API 5xx rate":
- every 60s, evaluate `sum(rate(http_requests_total{code=~"5.."}[5m]))` on prometheus datasource Prometheus
- fire when the value > 5 for 5 min; no data counts as OK, a failed evaluation fires a DatasourceError alert
- in folder "GoClaw" (made now), group "GoClaw api-5xx-rate", with its own contact point "GoClaw api-5xx-rate"
- Grafana calls http://192.0.2.10:8080/goclaw/hook/grafana/api-5xx-rate, with a new secret only this route knows; notification policies and other rules are not changed
GoClaw: when the rule fires, SOP "api-errors" (API errors) runs as agent main.

What the request can set: the query and its datasource (name or UID), the threshold and whether the value must be above (>, the default) or below (<) it, how long it must hold (for_minutes, default 5), how often Grafana evaluates it (every_seconds, default 60), and whether the SOP also runs when the alert resolves (on_resolved).

The rule is a new one, in a folder named GoClaw, and reaches its own contact point directly (the rule's Configure notifications, Grafana 11 and later). Notification policies, contact points and every other rule stay as they were. The folder is made when it is missing and left in place when a rule is removed.

The plan is refused, with the reason, when Grafana is older than 11.0, the datasource is not a Prometheus or Loki one the token can see, Grafana already has the rule or a contact point of the same name, the route key is taken, or the instance does not allow writes. A rule on another kind of query (SQL, Elasticsearch, a classic condition) is made in Grafana and sent to a contact point by hand, as below.

After creating it, the agent has Grafana send a test notification through the new contact point and says whether it reached the route.

What the SOP is given

Each notification is the run's trigger data, in an untrusted section of every step. The agent is told:

Alert: API 5xx rate — FIRING
Folder: GoClaw
- FIRING {service=api-gateway} values A=7.25, B=1
  Summary: API 5xx rate
  Since: 2026-10-09T02:00:00Z
  Rule: https://grafana.example.com/alerting/grafana/goclaw-api-5xx-rate/view

Up to five alerts of the notification are listed, each with its labels (those every alert of the rule carries are left out) and its values. One rule keeps one session.

The route's events are firing and resolved. A rule made to run on firing alerts filters on firing; one made with on_resolved lets both through.

Grafana repeats a firing alert's notification every repeat interval (4 hours by default). The repeat is the same delivery and is not run again; the alert firing again after it resolved is a new one.

Listing, checking, removing

list: every rule this gateway registered, its query, whether it is paused or gone in Grafana, and how many alerts the route received.
test: Grafana sends a test notification through the rule's contact point. The route answers it, counts it and runs nothing, so the SOP does not run; the agent says whether it arrived.
delete: planned and confirmed like a create; removes the rule, its contact point and the route. Deleting the route on the Webhooks page removes the rule and the contact point too, when the instance allows writes; otherwise it says what to delete in Grafana.

Only a human admin can do this, in a conversation of their own: never a SOP, cron, webhook or subagent run.

Connecting Grafana without the agent

On the Webhooks page, create a route of type Grafana with a key (for example the rule's name), the agent and the SOP. Set Events to firing to run only when the alert fires. Saving shows the URL and the secret once.
In Grafana, open Alerting › Contact points › Add contact point, choose the integration Webhook, and set URL to the route's URL. Under Optional Webhook settings, set Authorization Header - Scheme to Bearer and Authorization Header - Credentials to the route's secret.
Test: the route answers Grafana's test notification and runs nothing, and its Hits go up. Save the contact point.
Send the alert rule to it: in the rule's Configure notifications, choose the contact point, or add a notification policy that matches the rule's labels.

From Grafana 12 the contact point can sign each notification instead: put the secret under HMAC Signature › Secret and leave the header as it is (X-Grafana-Alerting-Signature). A notification whose token or signature does not match is refused with 401, and the reason is shown on the route.

Graylog Alerts That Run a SOP — the same pattern for Graylog alerts
SonarQube Quality Gates That Run a SOP — the same for SonarQube analyses
SRE Tools — the grafana_alerts tool and the instance's configuration
Standard Operating Procedures