Graylog does the watching: every minute or so it counts the messages that match a query over a time window. When the count crosses a threshold, Graylog calls the gateway, and the gateway runs a SOP with the alert and its matching messages as input.
flowchart LR
DEF["<b>Graylog Event Definition</b><br/>query, window, threshold"] -->|alert| NOTIF["<b>HTTP Notification</b>"]
NOTIF -->|POST| GW["<b>/goclaw/hook/graylog/<key></b>"]
GW --> SOP["<b>SOP Run on the Agent</b>"]
There are two ways to set it up:
graylog_alert
tool creates the event definition and the notification in Graylog and the webhook route in
GoClaw, and removes all three together. See Asking for an alert.config.yaml) and
the HTTP Notification in Graylog. GoClaw then only receives alerts. See
Connecting Graylog without the agent.Either way, Graylog must be able to reach the gateway, and the alert needs a SOP to run.
| What | Where |
|---|---|
| Graylog 5.0 or later | Older versions describe alerts differently; the tool refuses them. |
The Graylog instance, with allow_write: true (only when the agent sets alerts up) |
tools.observability.graylog.<name>: the Integrations page, config.yaml, or goclaw config set |
| An address Graylog can reach the gateway at | gateway.ingress.url, e.g. http://192.0.2.10:8080, and a listener there: gateway.bind or gateway.ingress.port/bind |
| Graylog let in, if the gateway restricts callers | gateway.ingress.allow_ips |
| A SOP for the alerts to run | the SOPs page, or the agent's sop_save tool (Standard Operating Procedures) |
gateway:
bind: lan # listen on the network, not only on 127.0.0.1
ingress:
url: "http://192.0.2.10:8080" # as Graylog sees this gateway
tools:
observability:
graylog:
main:
url: "http://graylog.internal:9000/api"
token: "${secret:GRAYLOG_TOKEN}"
allow_write: true # off by default: the agent cannot create or remove alerts
allow_write is off by default. The agent can then still search Graylog, and list and test the
alerts it registered, but it cannot create or remove one. Turn it on in one of three ways:
config.yaml, as in the example above;goclaw config set tools.observability.graylog.main.allow_write=true.The agent's next request uses it; no restart is needed.
The instance is read from the tools of the agent the alerts run as (by default, the agent you are
talking to). An agent with its own tools.observability block in its agents.list entry uses
that block instead of the global one, as a whole. The Integrations page edits the global block, so
for such an agent put the instance, with allow_write, into its own block. With more than one
instance, name the one you mean in your request ("on Graylog main").
The Graylog token needs these rights:
If the gateway's address is not yet on Graylog's URL allowlist (System › Configurations), the tool adds it. That needs the right to change Graylog's configuration. Without it, a Graylog admin adds the address and you ask the agent again.
The agent calls Graylog's API through the gateway's outbound HTTP policy. The host of the Graylog
instance's own url needs no entry on tools.http.allow, private address included: the Graylog
tools may reach the instance you configured. Only tools.http.deny stops them (the global one,
or the agent's own in its agents.list entry), with domain "graylog.internal" is blocked by policy or address … is blocked by policy. If graylog_query works for the agent, so does
this.
gateway.ingress.url only tells Graylog where to call: the agent builds the alert's URL from it.
The gateway must also accept the call at that address.
The gateway listens on 127.0.0.1 by default (gateway.bind: loopback), which a Graylog
server on another host cannot reach. Set gateway.bind: lan, or put a reverse proxy on the
gateway's host that forwards to it and use the proxy's address as gateway.ingress.url.
With gateway.ingress.port set, webhooks are served only on that port, by a listener of
their own bound by gateway.ingress.bind (127.0.0.1 when empty). Put that port in the URL and
set gateway.ingress.bind: lan. A call to the Web UI port then gets 404 page not found. This
is also the way to let Graylog in without putting the Web UI on the network:
gateway:
bind: loopback # the Web UI and API stay on 127.0.0.1
ingress:
url: "http://192.0.2.10:8081"
port: "8081" # webhooks only
bind: lan
If gateway.ingress.allow_ips or allow_preset is set, any caller outside the list gets
403 Forbidden. Add the address of every Graylog node that sends notifications to
allow_ips. When allow_ips is set, it replaces the preset rather than adding to it: on a
gateway with allow_preset: microsoft-teams, adding allow_ips drops the Teams ranges. Either
list those ranges in allow_ips too, or have Graylog come in through a reverse proxy on the
gateway's host: callers on 127.0.0.1 are always let in while gateway.ingress.trust_proxy is
off.
Changes to gateway.bind and to the ingress port, bind and allow lists take effect after a
restart. The Webhook URL Reference on the Webhooks page shows which port webhooks are served
on. See Public Ingress for proxies and ngrok.
Talk to the agent as an admin, and say "Graylog alert":
Create a Graylog alert that runs the incident-triage SOP when service payments logs more than 50 errors in 5 minutes, per host, in the Payments stream.
Name the feature. The agent is offered the tools that match what you write, and "Graylog
alert" is the surest way to get it graylog_alert. If admins manage alerts often, add the tool to
tools.core in config.yaml: the tools listed there are offered on every request.
tools:
core: [graylog_alert]
The agent checks everything and answers with a plan. Nothing is created yet:
Plan 3f9a1c2e — nothing is created yet.
Graylog "main" (version 6.1) gets an alert "Payments errors", priority normal:
- every 1 min, count the messages matching `service:payments AND level:ERROR` in streams
Payments over the last 5 min, counted per host
- alert when the count > 50; the same alert is not repeated within 10 min
- each alert carries up to 5 matching messages
- Graylog calls http://192.0.2.10:8080/goclaw/hook/graylog/payments-errors, with a new
secret only this route knows
GoClaw: each alert runs SOP "incident-triage" (Incident triage) as agent main.
Answer in your next message, for example "yes, create it". The agent then creates it in Graylog and in GoClaw. It then has Graylog send its test notification, and says whether the test reached the gateway. A plan cannot be carried out in the same turn it was made, nor from another conversation. It expires after 30 minutes.
The agent turns your words into these settings; you can name any of them in the request.
| Setting | Default | |
|---|---|---|
title |
(required) | what the alert is, e.g. "Payments errors"; it also gives the alert's key (below) |
query |
(required) | the Graylog search the alert counts, e.g. service:payments AND level:ERROR |
threshold |
(required) | alert when the count crosses it |
comparator |
> |
>, >=, <, <= or == against the threshold |
streams |
all streams | stream titles or IDs |
group_by |
none | fields counted separately, e.g. host: one alert per host that crosses the threshold |
window_minutes |
5 | count over the last N minutes, 1 to 1440 |
every_minutes |
1 | how often Graylog checks, 1 to the window |
grace_minutes |
10 | the same alert, for the same group-by values, is not repeated sooner; 0 to 1440 |
backlog |
5 | matching messages sent with each alert, 0 to 20 |
priority |
2 | 1 low, 2 normal, 3 high |
sop_id |
(required) | the SOP each alert runs |
agent_id |
the agent you are talking to | the agent the SOP runs as, whose tools the Graylog instance is read from |
instance |
the only one | the Graylog instance, needed when there are several |
description |
empty | the event definition's description in Graylog |
The key. The title names the alert, and its key is the title in lower case with every run of
characters other than Latin letters and digits turned into one dash, at most 40 characters:
"Payments errors" gives payments-errors. The key is the end of the route's URL
(/goclaw/hook/graylog/payments-errors), and you name the alert by it to test or remove it. A
title with no Latin letters or digits, such as one written only in Cyrillic, gives no key and is
refused: add a Latin word or a number. A title whose key a route already has is refused too: an
alert's, one made on the Webhooks page, or one in config.yaml.
Each alert starts one SOP run, as the agent the alert names, under the webhook-system
identity. Graylog's call is answered as soon as the run has been created, not when it ends. The
run's trigger shows as webhook:graylog in the SOP's History.
Every step of the run sees the alert as data from outside: in a section marked untrusted, never as part of the SOP's own instructions. That section shows at most 16 KB per alert. It holds the alert's title and message, its group-by values, and the matching messages Graylog sent with it. A log line is text anyone who can make the service log can write: the SOP should treat it as evidence, not as instructions.
Alerts of one event definition and one group-by value share a log session in GoClaw
(webhook-sop:<agent>:graylog:<sop>:alert-<definition>-<value>), which keeps its last 50
alerts.
Graylog's test notification only shows that the alert is connected. The route answers it and counts it, and no SOP runs. An alert Graylog sends again (a retry, say) is known by its event's ID for a week: the repeat is answered and runs nothing. That holds across a restart too: an alert the gateway was taking when it stopped runs when Graylog sends it again, unless its SOP run had begun a step.
The SOP must be able to run when the alert comes. A SOP that has been deleted or disabled, or
whose review date has passed, refuses every alert with 409. Disabling the SOP (pausing it on the
SOPs page, or sop_enable with enabled: false) stops its alerts for a while; enable it again to
resume. A SOP overdue for review runs again once someone reviews it (ask the agent to review the
SOP; its sop_review tool sets the next review date). See
What Graylog is told.
| Ask the agent to… | It… | Needs allow_write |
|---|---|---|
| list the Graylog alerts | shows each alert's key, title and query, its Graylog instance and state there, its SOP and agent, whether the route is enabled or paused, how many calls from Graylog the route took, and the last problem, if any | no |
test alert <key> |
has Graylog send the test notification again, and says whether it arrived within 10 seconds | no |
remove alert <key> |
answers with a plan of what it removes; after you confirm in your next message, it deletes the event definition and the notification in Graylog and the route in GoClaw | yes, both when it plans and when it removes |
The count in the list is the calls the route took, test notifications included: a new alert whose creation test arrived shows 1. Duplicate and refused calls are not counted.
Without allow_write, removing is refused: the agent says to turn it on, or to delete the route
on the Webhooks page and the event definition and notification in Graylog by hand. If the Graylog
instance is no longer configured for the alert's agent at all, removing goes ahead: the route is
removed, and the agent says which event definition and notification to delete in Graylog.
These actions see only the alerts the agent registered. A route you made yourself is not listed, tested or removed by them.
The alert's route appears on Automation › Webhooks as graylog/<key>.
allow_write it changes nothing in Graylog.404 and
run nothing, and the route's Health says a delivery arrived while it was paused. Graylog
keeps alerting; whether it sends a refused call again is up to Graylog.404 (or
goes to a Graylog route without a key, if there is one), and after a new secret 401. A
changed agent runs the alerts as that agent, and list, test and remove then look for the
Graylog instance in that agent's tools. A changed SOP applies to the next alert. To change what
the alert counts, remove it and ask for a new one.You can make the route and the Graylog notification yourself: for a Graylog instance without
allow_write, for an event definition that already exists, or because you would rather set it up
in Graylog. GoClaw then only receives alerts. It creates and removes nothing in Graylog,
graylog_alert does not list, test or remove the route, and deleting the route leaves Graylog as
it is. GoClaw makes no calls to Graylog here, so tools.observability is not needed; Graylog
must still reach the gateway (see
Letting Graylog reach the gateway).
On the Webhooks page (Automation › Webhooks › New Webhook):
checkout-errors. It ends the URL.main.After you save, the page shows Set up in Graylog with the URL, the API Key and the API Secret
to enter in Graylog. A generated secret is shown only then: copy it before you close the window.
Check that Graylog can reach that URL: the page builds it from the ngrok tunnel's address, else
from gateway.ingress.url, and when neither is set from the address you opened the page at and
the port webhooks are served on (the page then says so).
Or in config.yaml. Store the secret in the vault first, so the route finds it:
goclaw secret gen GRAYLOG_HOOK_CHECKOUT # prints the value; goclaw secret get shows it again
webhooks:
- type: graylog
key: checkout-errors # URL: <gateway.ingress.url>/goclaw/hook/graylog/checkout-errors
secret: "${secret:GRAYLOG_HOOK_CHECKOUT}"
agent: main # an agent ID, or "*" for every agent
sop_id: incident-triage # the SOP each alert runs
Put the shared value in secret, not token: token is compared only with a ?token=
parameter, and Graylog 5.1 and later sends the value as a header. A route in config.yaml applies
to the next call without a restart. It is listed on the Webhooks page, but it records no hits and
no refusals there: the gateway log has them. The keys are explained under
Webhook Routes.
batch_mode: bulk with batch_window (seconds, default 300) runs the SOP once for all the alerts
of a window instead of once per alert. Those runs share one log session for the route, and the
per-alert log sessions described under What the SOP is given do not
apply.
Without sop_id, each alert is a chat turn: the agent is given the alert's summary (the
matching messages are counted, not included), and its reply is saved in the session
webhook:<agent>:graylog:alert-<definition>-<value> and sent nowhere. Give the route a SOP when
the alert should lead to something.
Use type Graylog, not Custom. A Custom route accepts Graylog's calls the same way, but loses what type Graylog does:
| Graylog | Custom | |
|---|---|---|
| Graylog's test notification | answered {"status":"test"}, counted, nothing runs |
runs like an alert |
| An alert Graylog sends again | known by its event ID, runs once | runs again |
| What the session log and a chat turn read | a summary: alert, description, priority, message, group-by values, time, number of matching messages | up to five top-level text fields of the payload |
| Sessions | one per alert and group-by value | SOP runs: one for the whole route; chat turns: a new one per call |
In Graylog, open Alerts › Notifications, create a notification and choose the type HTTP Notification:
| Field | Value |
|---|---|
| URL | <gateway.ingress.url>/goclaw/hook/graylog/<key>. With gateway.ingress.port set, the URL uses that port. |
| API Key | X-Token |
| API Secret | the route's secret |
| The option that sends the API key as a header | on (Graylog 5.1 and later) |
| Basic authentication | leave empty: GoClaw does not read it |
Graylog 5.0 has no header option. Enter token as the API Key instead: Graylog then adds
?token=<secret> to the URL, which the route accepts too.
If Graylog refuses the URL, add it to Graylog's URL allowlist (System › Configurations › URL Allowlist). Run the test (step 3), save the notification, and add it to the event definition that should call it (Alerts › Event Definitions, its Notifications step). The definition's message backlog decides how many matching messages each alert carries.
Use Execute Test Notification in Graylog. The route answers 200 {"status":"test"} and runs
nothing. On a route made on the Webhooks page, Hits goes up by one. A route in config.yaml
counts nothing there; the gateway log shows Graylog test notification reached route graylog/<key>.
A 401 says why in its body:
no credential sent — send this route's secret as X-Token, ?token=, or an X-Signature HMAC:
the API Key is not X-Token (or token on 5.0); or, on 5.1 and later, it is X-Token but the
option that sends it as a header is off, so Graylog adds ?X-Token= to the URL, which the route
does not read; or only Basic authentication is filled in.the X-Token / ?token= value does not match this route's secret: the API Secret is not the
route's secret.this route has no secret, … or this route's secret is a ${secret:…} reference with no value in the vault: the route itself has no usable secret.For any other answer, see the next section.
A call to the route's URL gets one of these answers. Graylog reports a failed call in its server
log, and Execute Test Notification shows the answer directly. On the GoClaw side, a route
made on the Webhooks page or by the agent keeps the reason for its last refusal in its Health
column (and in the agent's list of alerts, as the last problem), for example
2026-10-07 14:03 409: SOP "incident-triage" is overdue for review. Please run sop_review.
Answers given before the call reaches a route are only in the gateway log, and so is everything
for a route kept in config.yaml.
| Answer | Why | Where it shows |
|---|---|---|
200 {"status":"ok"} |
Taken: the SOP run was created (queued, for a bulk route), or the chat turn admitted. | Hits goes up |
301 or 405 |
The URL ends with / (the key is missing: the page's Base button copies the URL without one), or the call is not a POST. |
not counted |
200 {"status":"test"} |
Graylog's test notification. Nothing runs. | Hits goes up |
200 {"status":"duplicate"} |
The route took this event (same event ID) before, within the last week. Nothing runs. | not counted |
200 {"status":"filtered"} |
The route's Events leave this call out. Leave Events empty on a Graylog route. | not counted |
401 |
The secret is missing or wrong, or the route has none. The body says which (see Test it). After a route's secret was changed on the Webhooks page, Graylog still sends the old one. | Health |
403 Forbidden |
The caller's address is not in gateway.ingress.allow_ips or allow_preset. |
gateway log only ([ingress] denied) |
404 |
no webhook route for graylog/<key>: a wrong key, or a key changed on the Webhooks page. A plain 404 page not found is the Web UI port, when gateway.ingress.port is set. |
gateway log, with the keys that exist |
404 |
no active webhook route for graylog/<key>: the route is paused. |
Health |
409 |
The route's SOP was deleted or disabled, or is overdue for review. Nothing runs until the SOP exists, is enabled and is reviewed. On a route without a SOP, the route's agent is not running on this gateway: restart the gateway after adding the agent to config.yaml, or look in its log for [multi-agent] Failed to create agent. |
Health |
413 |
The body is larger than 4 MB. Lower the backlog. | gateway log only |
429 |
Over the gateway's rate limit: gateway.rate_limit.per_minute (default 60) calls a minute from one address. Retry-After: 60. |
gateway log only |
500 |
The SOP could not be read, or its run could not be started. | Health |
503 |
Too many in progress: 256 webhook-started SOP runs already waiting or running (for a route without a SOP, 256 chat turns). Retry-After: 30. Or another request is taking this same alert right now (Retry-After: 10), which a resend settles. Or the alert's event ID could not be recorded (Retry-After: 10), so it was not run. |
Health (too many in progress, not recorded) |
Whether Graylog sends a refused call again is up to Graylog. A refused call is not remembered, so it runs if it comes again.
If the agent's test fails, look at what it says. Graylog's test is the same call as Execute Test Notification:
2xx, and the error carries
Graylog's own message with the answer it got. Find that answer in the table above. A refused
connection or a timeout is the address or port
(Letting Graylog reach the gateway); a URL Graylog will
not call is its URL allowlist.gateway.ingress.url reaches this gateway.Only a human admin, in a conversation of their own, can ask the agent to create, list, test or
remove alerts. Creating and removing also need allow_write on the Graylog instance; listing and
testing do not. The tool is refused in SOP steps, scheduled jobs, webhook-started runs and
subagents.
These rules exist because the agent reads log lines: in search results, and in the alerts themselves. A crafted log line could otherwise lead an agent to create, change or remove the automation that reads it. The confirmation in a later message keeps a person's decision between what the agent proposes and what changes.
gateway.ingress, reverse proxies, the caller allow listwebhooks: keysgraylog_query, the agent's log search