# Graylog Alerts That Run a SOP

Graylog does the watching: every minute or so it counts the messages that match a query over a
time window. When the count crosses a threshold, Graylog calls the gateway, and the gateway runs
a SOP with the alert and its matching messages as input.

```mermaid
flowchart LR
    DEF["<b>Graylog Event Definition</b><br/>query, window, threshold"] -->|alert| NOTIF["<b>HTTP Notification</b>"]
    NOTIF -->|POST| GW["<b>/goclaw/hook/graylog/&lt;key&gt;</b>"]
    GW --> SOP["<b>SOP Run on the Agent</b>"]
```

There are two ways to set it up:

- **Ask the agent.** An admin describes the alert in a conversation. The agent's `graylog_alert`
  tool creates the event definition and the notification in Graylog and the webhook route in
  GoClaw, and removes all three together. See [Asking for an alert](#asking-for-an-alert).
- **Connect it yourself.** You make the route on the **Webhooks** page (or in `config.yaml`) and
  the HTTP Notification in Graylog. GoClaw then only receives alerts. See
  [Connecting Graylog without the agent](#connecting-graylog-without-the-agent).

Either way, Graylog must be able to reach the gateway, and the alert needs a SOP to run.

---

## Before you start

| What | Where |
| :--- | :--- |
| Graylog 5.0 or later | Older versions describe alerts differently; the tool refuses them. |
| The Graylog instance, with `allow_write: true` (only when the agent sets alerts up) | `tools.observability.graylog.<name>`: the **Integrations** page, `config.yaml`, or `goclaw config set` |
| An address Graylog can reach the gateway at | `gateway.ingress.url`, e.g. `http://192.0.2.10:8080`, and a listener there: `gateway.bind` or `gateway.ingress.port`/`bind` |
| Graylog let in, if the gateway restricts callers | `gateway.ingress.allow_ips` |
| A SOP for the alerts to run | the SOPs page, or the agent's `sop_save` tool ([Standard Operating Procedures](sop.md)) |

```yaml
gateway:
  bind: lan                           # listen on the network, not only on 127.0.0.1
  ingress:
    url: "http://192.0.2.10:8080"     # as Graylog sees this gateway

tools:
  observability:
    graylog:
      main:
        url: "http://graylog.internal:9000/api"
        token: "${secret:GRAYLOG_TOKEN}"
        allow_write: true             # off by default: the agent cannot create or remove alerts
```

### The Graylog instance

`allow_write` is off by default. The agent can then still search Graylog, and list and test the
alerts it registered, but it cannot create or remove one. Turn it on in one of three ways:

- on the **Integrations** page, edit the Graylog instance and tick **Allow write actions (create
  and remove alerts)**; the instance then shows the *write-enabled* tag;
- in `config.yaml`, as in the example above;
- `goclaw config set tools.observability.graylog.main.allow_write=true`.

The agent's next request uses it; no restart is needed.

The instance is read from the tools of the agent the alerts run as (by default, the agent you are
talking to). An agent with its own `tools.observability` block in its `agents.list` entry uses
that block instead of the global one, as a whole. The Integrations page edits the global block, so
for such an agent put the instance, with `allow_write`, into its own block. With more than one
instance, name the one you mean in your request ("on Graylog main").

The Graylog token needs these rights:

- read streams;
- create, read and delete event definitions;
- create, edit and delete notifications.

If the gateway's address is not yet on Graylog's URL allowlist (System › Configurations), the
tool adds it. That needs the right to change Graylog's configuration. Without it, a Graylog
admin adds the address and you ask the agent again.

### Letting the gateway reach Graylog

The agent calls Graylog's API through the gateway's outbound HTTP policy. The host of the Graylog
instance's own `url` needs no entry on `tools.http.allow`, private address included: the Graylog
tools may reach the instance you configured. Only `tools.http.deny` stops them (the global one,
or the agent's own in its `agents.list` entry), with `domain "graylog.internal" is blocked by
policy` or `address … is blocked by policy`. If `graylog_query` works for the agent, so does
this.

### Letting Graylog reach the gateway

`gateway.ingress.url` only tells Graylog where to call: the agent builds the alert's URL from it.
The gateway must also accept the call at that address.

- **The gateway listens on 127.0.0.1 by default** (`gateway.bind: loopback`), which a Graylog
  server on another host cannot reach. Set `gateway.bind: lan`, or put a reverse proxy on the
  gateway's host that forwards to it and use the proxy's address as `gateway.ingress.url`.
- **With `gateway.ingress.port` set**, webhooks are served only on that port, by a listener of
  their own bound by `gateway.ingress.bind` (127.0.0.1 when empty). Put that port in the URL and
  set `gateway.ingress.bind: lan`. A call to the Web UI port then gets `404 page not found`. This
  is also the way to let Graylog in without putting the Web UI on the network:

  ```yaml
  gateway:
    bind: loopback                  # the Web UI and API stay on 127.0.0.1
    ingress:
      url: "http://192.0.2.10:8081"
      port: "8081"                  # webhooks only
      bind: lan
  ```

- **If `gateway.ingress.allow_ips` or `allow_preset` is set**, any caller outside the list gets
  `403 Forbidden`. Add the address of every Graylog node that sends notifications to
  `allow_ips`. When `allow_ips` is set, it replaces the preset rather than adding to it: on a
  gateway with `allow_preset: microsoft-teams`, adding `allow_ips` drops the Teams ranges. Either
  list those ranges in `allow_ips` too, or have Graylog come in through a reverse proxy on the
  gateway's host: callers on 127.0.0.1 are always let in while `gateway.ingress.trust_proxy` is
  off.

Changes to `gateway.bind` and to the ingress port, bind and allow lists take effect after a
restart. The **Webhook URL Reference** on the Webhooks page shows which port webhooks are served
on. See [Public Ingress](../ingress.md) for proxies and ngrok.

---

## Asking for an alert

Talk to the agent as an admin, and say "Graylog alert":

> Create a Graylog alert that runs the incident-triage SOP when service payments logs more than
> 50 errors in 5 minutes, per host, in the Payments stream.

**Name the feature.** The agent is offered the tools that match what you write, and "Graylog
alert" is the surest way to get it `graylog_alert`. If admins manage alerts often, add the tool to
`tools.core` in `config.yaml`: the tools listed there are offered on every request.

```yaml
tools:
  core: [graylog_alert]
```

The agent checks everything and answers with a plan. Nothing is created yet:

```
Plan 3f9a1c2e — nothing is created yet.

Graylog "main" (version 6.1) gets an alert "Payments errors", priority normal:
- every 1 min, count the messages matching `service:payments AND level:ERROR` in streams
  Payments over the last 5 min, counted per host
- alert when the count > 50; the same alert is not repeated within 10 min
- each alert carries up to 5 matching messages
- Graylog calls http://192.0.2.10:8080/goclaw/hook/graylog/payments-errors, with a new
  secret only this route knows
GoClaw: each alert runs SOP "incident-triage" (Incident triage) as agent main.
```

Answer in your **next message**, for example "yes, create it". The agent then creates it in
Graylog and in GoClaw. It then has Graylog send its test notification, and says whether the test
reached the gateway. A plan cannot be carried out in the same turn it was made, nor from another
conversation. It expires after 30 minutes.

### What you can say

The agent turns your words into these settings; you can name any of them in the request.

| Setting | Default | |
| :--- | :--- | :--- |
| `title` | (required) | what the alert is, e.g. "Payments errors"; it also gives the alert's key (below) |
| `query` | (required) | the Graylog search the alert counts, e.g. `service:payments AND level:ERROR` |
| `threshold` | (required) | alert when the count crosses it |
| `comparator` | `>` | `>`, `>=`, `<`, `<=` or `==` against the threshold |
| `streams` | all streams | stream titles or IDs |
| `group_by` | none | fields counted separately, e.g. `host`: one alert per host that crosses the threshold |
| `window_minutes` | 5 | count over the last N minutes, 1 to 1440 |
| `every_minutes` | 1 | how often Graylog checks, 1 to the window |
| `grace_minutes` | 10 | the same alert, for the same group-by values, is not repeated sooner; 0 to 1440 |
| `backlog` | 5 | matching messages sent with each alert, 0 to 20 |
| `priority` | 2 | 1 low, 2 normal, 3 high |
| `sop_id` | (required) | the SOP each alert runs |
| `agent_id` | the agent you are talking to | the agent the SOP runs as, whose tools the Graylog instance is read from |
| `instance` | the only one | the Graylog instance, needed when there are several |
| `description` | empty | the event definition's description in Graylog |

**The key.** The title names the alert, and its key is the title in lower case with every run of
characters other than Latin letters and digits turned into one dash, at most 40 characters:
"Payments errors" gives `payments-errors`. The key is the end of the route's URL
(`/goclaw/hook/graylog/payments-errors`), and you name the alert by it to test or remove it. A
title with no Latin letters or digits, such as one written only in Cyrillic, gives no key and is
refused: add a Latin word or a number. A title whose key a route already has is refused too: an
alert's, one made on the Webhooks page, or one in `config.yaml`.

---

## What the SOP is given

Each alert starts one SOP run, as the agent the alert names, under the `webhook-system`
identity. Graylog's call is answered as soon as the run has been created, not when it ends. The
run's trigger shows as `webhook:graylog` in the SOP's History.

Every step of the run sees the alert as **data from outside**: in a section marked untrusted,
never as part of the SOP's own instructions. That section shows at most 16 KB per alert. It holds
the alert's title and message, its group-by values, and the matching messages Graylog sent with
it. A log line is text anyone who can make the service log can write: the SOP should treat it as
evidence, not as instructions.

Alerts of one event definition and one group-by value share a log session in GoClaw
(`webhook-sop:<agent>:graylog:<sop>:alert-<definition>-<value>`), which keeps its last 50
alerts.

Graylog's test notification only shows that the alert is connected. The route answers it and
counts it, and no SOP runs. An alert Graylog sends again (a retry, say) is known by its event's
ID for a week: the repeat is answered and runs nothing. That holds across a restart too: an alert
the gateway was taking when it stopped runs when Graylog sends it again, unless its SOP run had
begun a step.

The SOP must be able to run when the alert comes. A SOP that has been deleted or disabled, or
whose review date has passed, refuses every alert with `409`. Disabling the SOP (pausing it on the
SOPs page, or `sop_enable` with `enabled: false`) stops its alerts for a while; enable it again to
resume. A SOP overdue for review runs again once someone reviews it (ask the agent to review the
SOP; its `sop_review` tool sets the next review date). See
[What Graylog is told](#what-graylog-is-told).

---

## Listing, testing, removing

| Ask the agent to… | It… | Needs `allow_write` |
| :--- | :--- | :--- |
| list the Graylog alerts | shows each alert's key, title and query, its Graylog instance and state there, its SOP and agent, whether the route is enabled or paused, how many calls from Graylog the route took, and the last problem, if any | no |
| test alert `<key>` | has Graylog send the test notification again, and says whether it arrived within 10 seconds | no |
| remove alert `<key>` | answers with a plan of what it removes; after you confirm in your next message, it deletes the event definition and the notification in Graylog and the route in GoClaw | yes, both when it plans and when it removes |

The count in the list is the calls the route took, test notifications included: a new alert whose
creation test arrived shows 1. Duplicate and refused calls are not counted.

Without `allow_write`, removing is refused: the agent says to turn it on, or to delete the route
on the Webhooks page and the event definition and notification in Graylog by hand. If the Graylog
instance is no longer configured for the alert's agent at all, removing goes ahead: the route is
removed, and the agent says which event definition and notification to delete in Graylog.

These actions see only the alerts the agent registered. A route you made yourself is not listed,
tested or removed by them.

### The alert's route on the Webhooks page

The alert's route appears on **Automation › Webhooks** as `graylog/<key>`.

- **Delete** removes the Graylog side as well, or says what is left for a Graylog admin to delete.
  Without `allow_write` it changes nothing in Graylog.
- **Pause** stops the alerts: while the route is paused, Graylog's calls are answered `404` and
  run nothing, and the route's **Health** says a delivery arrived while it was paused. Graylog
  keeps alerting; whether it sends a refused call again is up to Graylog.
- **Edit**: do not change the type, the key or the secret. Graylog's notification keeps the
  address and secret it was created with, so after a new key every alert is answered `404` (or
  goes to a Graylog route without a key, if there is one), and after a new secret `401`. A
  changed agent runs the alerts as that agent, and list, test and remove then look for the
  Graylog instance in that agent's tools. A changed SOP applies to the next alert. To change what
  the alert counts, remove it and ask for a new one.

---

## Connecting Graylog without the agent

You can make the route and the Graylog notification yourself: for a Graylog instance without
`allow_write`, for an event definition that already exists, or because you would rather set it up
in Graylog. GoClaw then only receives alerts. It creates and removes nothing in Graylog,
`graylog_alert` does not list, test or remove the route, and deleting the route leaves Graylog as
it is. GoClaw makes no calls to Graylog here, so `tools.observability` is not needed; Graylog
must still reach the gateway (see
[Letting Graylog reach the gateway](#letting-graylog-reach-the-gateway)).

### 1. The route

**On the Webhooks page** (Automation › Webhooks › **New Webhook**):

- **Type**: Graylog.
- **Key**: a name for the alert, e.g. `checkout-errors`. It ends the URL.
- **Agent**: the agent the SOP runs as, e.g. `main`.
- **Secret**: leave it empty to have one generated.
- **Events**: leave empty: Graylog sends only alerts. (Its test notification is answered whatever
  the filter says.)
- **SOP ID**: the SOP each alert runs.

After you save, the page shows **Set up in Graylog** with the URL, the API Key and the API Secret
to enter in Graylog. A generated secret is shown only then: copy it before you close the window.
Check that Graylog can reach that URL: the page builds it from the ngrok tunnel's address, else
from `gateway.ingress.url`, and when neither is set from the address you opened the page at and
the port webhooks are served on (the page then says so).

**Or in `config.yaml`.** Store the secret in the vault first, so the route finds it:

```bash
goclaw secret gen GRAYLOG_HOOK_CHECKOUT     # prints the value; goclaw secret get shows it again
```

```yaml
webhooks:
  - type: graylog
    key: checkout-errors            # URL: <gateway.ingress.url>/goclaw/hook/graylog/checkout-errors
    secret: "${secret:GRAYLOG_HOOK_CHECKOUT}"
    agent: main                     # an agent ID, or "*" for every agent
    sop_id: incident-triage         # the SOP each alert runs
```

Put the shared value in `secret`, not `token`: `token` is compared only with a `?token=`
parameter, and Graylog 5.1 and later sends the value as a header. A route in `config.yaml` applies
to the next call without a restart. It is listed on the Webhooks page, but it records no hits and
no refusals there: the gateway log has them. The keys are explained under
[Webhook Routes](../configuration.md#webhook-routes).

`batch_mode: bulk` with `batch_window` (seconds, default 300) runs the SOP once for all the alerts
of a window instead of once per alert. Those runs share one log session for the route, and the
per-alert log sessions described under [What the SOP is given](#what-the-sop-is-given) do not
apply.

**Without `sop_id`**, each alert is a chat turn: the agent is given the alert's summary (the
matching messages are counted, not included), and its reply is saved in the session
`webhook:<agent>:graylog:alert-<definition>-<value>` and sent nowhere. Give the route a SOP when
the alert should lead to something.

**Use type Graylog, not Custom.** A Custom route accepts Graylog's calls the same way, but loses
what type Graylog does:

| | Graylog | Custom |
| :--- | :--- | :--- |
| Graylog's test notification | answered `{"status":"test"}`, counted, nothing runs | runs like an alert |
| An alert Graylog sends again | known by its event ID, runs once | runs again |
| What the session log and a chat turn read | a summary: alert, description, priority, message, group-by values, time, number of matching messages | up to five top-level text fields of the payload |
| Sessions | one per alert and group-by value | SOP runs: one for the whole route; chat turns: a new one per call |

### 2. The notification in Graylog

In Graylog, open **Alerts › Notifications**, create a notification and choose the type **HTTP
Notification**:

| Field | Value |
| :--- | :--- |
| URL | `<gateway.ingress.url>/goclaw/hook/graylog/<key>`. With `gateway.ingress.port` set, the URL uses that port. |
| API Key | `X-Token` |
| API Secret | the route's secret |
| The option that sends the API key as a header | on (Graylog 5.1 and later) |
| Basic authentication | leave empty: GoClaw does not read it |

Graylog 5.0 has no header option. Enter `token` as the API Key instead: Graylog then adds
`?token=<secret>` to the URL, which the route accepts too.

If Graylog refuses the URL, add it to Graylog's URL allowlist (System › Configurations › URL
Allowlist). Run the test (step 3), save the notification, and add it to the event definition that
should call it (**Alerts › Event Definitions**, its Notifications step). The definition's message
backlog decides how many matching messages each alert carries.

### 3. Test it

Use **Execute Test Notification** in Graylog. The route answers `200 {"status":"test"}` and runs
nothing. On a route made on the Webhooks page, **Hits** goes up by one. A route in `config.yaml`
counts nothing there; the gateway log shows `Graylog test notification reached route
graylog/<key>`.

A `401` says why in its body:

- `no credential sent — send this route's secret as X-Token, ?token=, or an X-Signature HMAC`:
  the API Key is not `X-Token` (or `token` on 5.0); or, on 5.1 and later, it is `X-Token` but the
  option that sends it as a header is off, so Graylog adds `?X-Token=` to the URL, which the route
  does not read; or only Basic authentication is filled in.
- `the X-Token / ?token= value does not match this route's secret`: the API Secret is not the
  route's secret.
- `this route has no secret, …` or `this route's secret is a ${secret:…} reference with no value
  in the vault`: the route itself has no usable secret.

For any other answer, see the next section.

---

## What Graylog is told

A call to the route's URL gets one of these answers. Graylog reports a failed call in its server
log, and **Execute Test Notification** shows the answer directly. On the GoClaw side, a route
made on the Webhooks page or by the agent keeps the reason for its last refusal in its **Health**
column (and in the agent's list of alerts, as the last problem), for example
`2026-10-07 14:03 409: SOP "incident-triage" is overdue for review. Please run sop_review.`
Answers given before the call reaches a route are only in the gateway log, and so is everything
for a route kept in `config.yaml`.

| Answer | Why | Where it shows |
| :--- | :--- | :--- |
| `200 {"status":"ok"}` | Taken: the SOP run was created (queued, for a bulk route), or the chat turn admitted. | **Hits** goes up |
| `301` or `405` | The URL ends with `/` (the key is missing: the page's **Base** button copies the URL without one), or the call is not a POST. | not counted |
| `200 {"status":"test"}` | Graylog's test notification. Nothing runs. | **Hits** goes up |
| `200 {"status":"duplicate"}` | The route took this event (same event ID) before, within the last week. Nothing runs. | not counted |
| `200 {"status":"filtered"}` | The route's **Events** leave this call out. Leave Events empty on a Graylog route. | not counted |
| `401` | The secret is missing or wrong, or the route has none. The body says which (see [Test it](#3-test-it)). After a route's secret was changed on the Webhooks page, Graylog still sends the old one. | Health |
| `403 Forbidden` | The caller's address is not in `gateway.ingress.allow_ips` or `allow_preset`. | gateway log only (`[ingress] denied`) |
| `404` | `no webhook route for graylog/<key>`: a wrong key, or a key changed on the Webhooks page. A plain `404 page not found` is the Web UI port, when `gateway.ingress.port` is set. | gateway log, with the keys that exist |
| `404` | `no active webhook route for graylog/<key>`: the route is paused. | Health |
| `409` | The route's SOP was deleted or disabled, or is overdue for review. Nothing runs until the SOP exists, is enabled and is reviewed. On a route without a SOP, the route's agent is not running on this gateway: restart the gateway after adding the agent to `config.yaml`, or look in its log for `[multi-agent] Failed to create agent`. | Health |
| `413` | The body is larger than 4 MB. Lower the backlog. | gateway log only |
| `429` | Over the gateway's rate limit: `gateway.rate_limit.per_minute` (default 60) calls a minute from one address. `Retry-After: 60`. | gateway log only |
| `500` | The SOP could not be read, or its run could not be started. | Health |
| `503` | Too many in progress: 256 webhook-started SOP runs already waiting or running (for a route without a SOP, 256 chat turns). `Retry-After: 30`. Or another request is taking this same alert right now (`Retry-After: 10`), which a resend settles. Or the alert's event ID could not be recorded (`Retry-After: 10`), so it was not run. | Health (too many in progress, not recorded) |

Whether Graylog sends a refused call again is up to Graylog. A refused call is not remembered, so
it runs if it comes again.

**If the agent's test fails**, look at what it says. Graylog's test is the same call as **Execute
Test Notification**:

- *Graylog's test notification failed*: the call did not get a `2xx`, and the error carries
  Graylog's own message with the answer it got. Find that answer in the table above. A refused
  connection or a timeout is the address or port
  ([Letting Graylog reach the gateway](#letting-graylog-reach-the-gateway)); a URL Graylog will
  not call is its URL allowlist.
- *Graylog sent its test notification and got an answer, but this route counted nothing*:
  something else at that address answered — another gateway or server, or a proxy that answers by
  itself. Check that `gateway.ingress.url` reaches this gateway.

---

## Who can do this

Only a human admin, in a conversation of their own, can ask the agent to create, list, test or
remove alerts. Creating and removing also need `allow_write` on the Graylog instance; listing and
testing do not. The tool is refused in SOP steps, scheduled jobs, webhook-started runs and
subagents.

These rules exist because the agent reads log lines: in search results, and in the alerts
themselves. A crafted log line could otherwise lead an agent to create, change or remove the
automation that reads it. The confirmation in a later message keeps a person's decision between
what the agent proposes and what changes.

---

## Related

- [Standard Operating Procedures](sop.md): writing the SOP an alert runs, and
  [runs started by a webhook](sop.md#runs-started-by-a-webhook)
- [Public Ingress](../ingress.md): `gateway.ingress`, reverse proxies, the caller allow list
- [Configuration › Webhook Routes](../configuration.md#webhook-routes): the `webhooks:` keys
- [SRE Tools › Graylog](../observability/sre-tools.md#graylog): the Graylog instance and
  `graylog_query`, the agent's log search
- [GitLab Webhook Integration](gitlab-webhook.md): the same webhook receiver, for GitLab events