# Grafana Alert Rules That Run a SOP

Grafana evaluates a query on a schedule, and when the value crosses a threshold for long enough
the alert fires. With a webhook contact point that calls the gateway, a firing alert runs a SOP,
with the alert's labels, values and annotations as its input.

```mermaid
flowchart LR
    EVAL["<b>Grafana Evaluates the Rule</b><br/>every 60 s, for 5 min"] -->|fires| CP["<b>Rule Contact Point</b><br/>webhook"]
    CP -->|"POST (Bearer)"| GW["<b>/goclaw/hook/grafana/&lt;key&gt;</b>"]
    GW --> SOP["<b>SOP Run on the Agent</b>"]
```

There are two ways to set it up:

- **Ask the agent.** An admin describes it in a conversation. The agent's `grafana_alert_rule`
  tool makes the alert rule, its own contact point and the route in GoClaw, and removes them
  together. See [Asking for an alert rule](#asking-for-an-alert-rule).
- **Connect it yourself.** Make a `grafana` route on the **Webhooks** page, and a webhook contact
  point in Grafana with the URL and secret the page shows; then send an alert rule to it. See
  [Connecting Grafana without the agent](#connecting-grafana-without-the-agent).

---

## Before you start

| What | Where |
| :--- | :--- |
| Grafana 11.0 or later, with `allow_write: true` on the instance (only when the agent makes rules) | `tools.observability.grafana.<name>`: the **Integrations** page or `config.yaml` |
| A service account token with the **Editor** role (some older Grafana versions keep the alerting provisioning API from Editors; then **Admin**) | the instance's `token` |
| A Prometheus or Loki datasource for the rule's query | Grafana's **Connections › Data sources** |
| An address Grafana can reach the gateway at | `gateway.ingress.url`, and a listener there |
| A SOP for the alerts to run | the SOPs page, or the agent's `sop_save` tool ([Standard Operating Procedures](sop.md)) |

```yaml
gateway:
  bind: lan
  ingress:
    url: "http://192.0.2.10:8080"     # as Grafana sees this gateway

tools:
  observability:
    grafana:
      prod:
        url: "https://grafana.example.com"
        token: "${secret:GRAFANA_TOKEN}"
        allow_write: true             # off by default: the agent cannot make or remove rules
```

The Grafana host needs no entry on `tools.http.allow`: the Grafana tools may reach the instance
you configured.

---

## Asking for an alert rule

```
When the API's 5xx rate stays above 5 per second, run the api-errors SOP.
  (sum(rate(http_requests_total{code=~"5.."}[5m])) on Prometheus)
Alert when free disk on the reports host drops below 10 GB for 15 minutes, and run disk-cleanup — also when it recovers.
```

The agent answers with a plan; nothing is created until you confirm it in your next message:

```
Plan 3e9b07c1 — nothing is created yet.

Grafana "prod" (version 12.2.1) gets an alert rule "API 5xx rate":
- every 60s, evaluate `sum(rate(http_requests_total{code=~"5.."}[5m]))` on prometheus datasource Prometheus
- fire when the value > 5 for 5 min; no data counts as OK, a failed evaluation fires a DatasourceError alert
- in folder "GoClaw" (made now), group "GoClaw api-5xx-rate", with its own contact point "GoClaw api-5xx-rate"
- Grafana calls http://192.0.2.10:8080/goclaw/hook/grafana/api-5xx-rate, with a new secret only this route knows; notification policies and other rules are not changed
GoClaw: when the rule fires, SOP "api-errors" (API errors) runs as agent main.
```

What the request can set: the query and its datasource (name or UID), the threshold and
whether the value must be above (`>`, the default) or below (`<`) it, how long it must hold
(`for_minutes`, default 5), how often Grafana evaluates it (`every_seconds`, default 60), and
whether the SOP also runs when the alert resolves (`on_resolved`).

The rule is a new one, in a folder named **GoClaw**, and reaches its own contact point directly
(the rule's **Configure notifications**, Grafana 11 and later). Notification policies, contact
points and every other rule stay as they were. The folder is made when it is missing and left in
place when a rule is removed.

The plan is refused, with the reason, when Grafana is older than 11.0, the datasource is not a
Prometheus or Loki one the token can see, Grafana already has the rule or a contact point of the
same name, the route key is taken, or the instance does not allow writes. A rule on another kind
of query (SQL, Elasticsearch, a classic condition) is made in Grafana and sent to a contact point
by hand, as [below](#connecting-grafana-without-the-agent).

After creating it, the agent has Grafana send a test notification through the new contact point
and says whether it reached the route.

---

## What the SOP is given

Each notification is the run's trigger data, in an untrusted section of every step. The agent is
told:

```
Alert: API 5xx rate — FIRING
Folder: GoClaw
- FIRING {service=api-gateway} values A=7.25, B=1
  Summary: API 5xx rate
  Since: 2026-10-09T02:00:00Z
  Rule: https://grafana.example.com/alerting/grafana/goclaw-api-5xx-rate/view
```

Up to five alerts of the notification are listed, each with its labels (those every alert of the
rule carries are left out) and its values. One rule keeps one session.

The route's events are `firing` and `resolved`. A rule made to run on firing alerts filters on
`firing`; one made with `on_resolved` lets both through.

Grafana repeats a firing alert's notification every repeat interval (4 hours by default). The
repeat is the same delivery and is not run again; the alert firing again after it resolved is a
new one.

---

## Listing, checking, removing

- **list**: every rule this gateway registered, its query, whether it is paused or gone in
  Grafana, and how many alerts the route received.
- **test**: Grafana sends a test notification through the rule's contact point. The route
  answers it, counts it and runs nothing, so the SOP does not run; the agent says whether it
  arrived.
- **delete**: planned and confirmed like a create; removes the rule, its contact point and the
  route. Deleting the route on the **Webhooks** page removes the rule and the contact point too,
  when the instance allows writes; otherwise it says what to delete in Grafana.

Only a human admin can do this, in a conversation of their own: never a SOP, cron, webhook or
subagent run.

---

## Connecting Grafana without the agent

1. On the **Webhooks** page, create a route of type **Grafana** with a key (for example the
   rule's name), the agent and the SOP. Set **Events** to `firing` to run only when the alert
   fires. Saving shows the URL and the secret once.
2. In Grafana, open **Alerting › Contact points › Add contact point**, choose the integration
   **Webhook**, and set **URL** to the route's URL. Under **Optional Webhook settings**, set
   **Authorization Header - Scheme** to `Bearer` and **Authorization Header - Credentials** to the
   route's secret.
3. **Test**: the route answers Grafana's test notification and runs nothing, and its **Hits** go
   up. Save the contact point.
4. Send the alert rule to it: in the rule's **Configure notifications**, choose the contact point,
   or add a notification policy that matches the rule's labels.

From Grafana 12 the contact point can sign each notification instead: put the secret under
**HMAC Signature › Secret** and leave the header as it is (`X-Grafana-Alerting-Signature`). A
notification whose token or signature does not match is refused with 401, and the reason is shown
on the route.

## Related

- [Graylog Alerts That Run a SOP](graylog-alerts.md) — the same pattern for Graylog alerts
- [SonarQube Quality Gates That Run a SOP](sonarqube-hooks.md) — the same for SonarQube analyses
- [SRE Tools](../observability/sre-tools.md#grafana) — the `grafana_alerts` tool and the instance's configuration
- [Standard Operating Procedures](sop.md)