# Uptrace Monitors That Run a SOP
An Uptrace metric monitor checks a query every minute, and opens an alert when the value stays
beyond its bounds. With a webhook notification channel that calls the gateway, an alert runs a
SOP, with the alert's name, status and attributes as its input.
```mermaid
flowchart LR
MON["Uptrace Monitor 'GoClaw: <title>'
query, above/below, N minutes"] -->|"opens an alert"| CH["Webhook Channel 'GoClaw: <title>'
notifies only this monitor"]
CH -->|POST| GW["/goclaw/hook/uptrace/<key>?token=…"]
GW --> SOP["SOP Run on the Agent"]
```
There are two ways to set it up:
- **Ask the agent.** An admin describes it in a conversation. The agent's `uptrace_monitor` tool
makes the monitor and its channel in Uptrace and the route in GoClaw, and removes them
together. See [Asking for a monitor](#asking-for-a-monitor).
- **Connect it yourself.** Make an `uptrace` route on the **Webhooks** page, and in Uptrace a
webhook channel with the URL the page shows, which a monitor notifies. See
[Connecting Uptrace without the agent](#connecting-uptrace-without-the-agent).
---
## Before you start
| What | Where |
| :--- | :--- |
| The Uptrace instance with `url` and a **user** token, and `allow_write: true` (only when the agent makes monitors) | `tools.observability.uptrace.`: the **Integrations** page or `config.yaml` |
| The project the monitor belongs to | the instance's `project_id`, or named in the request |
| An address Uptrace can reach the gateway at | `gateway.ingress.url`, and a listener there |
| A SOP for the alerts to run | the SOPs page, or the agent's `sop_save` tool ([Standard Operating Procedures](sop.md)) |
```yaml
gateway:
bind: lan
ingress:
url: "http://192.0.2.10:8080" # as Uptrace sees this gateway
tools:
observability:
uptrace:
main:
url: "https://uptrace.example.com"
token: "${secret:UPTRACE_USER_TOKEN}" # a user's token, not a DSN's project token
project_id: "1"
allow_write: true # off by default: the agent cannot make or remove monitors
```
A DSN's project token sends data; it cannot manage monitors. The user token is in Uptrace's
user profile (2.x), or `auth.users[].auth_token` in Uptrace's own config (1.7).
**Uptrace versions.** Uptrace does not document the API its UI uses to manage monitors, and it
changed twice. The agent finds out which one the server speaks before it plans, and the plan
names it:
| Version | How sure | Notes |
| :--- | :--- | :--- |
| 1.7 | checked against Uptrace's source | queries write `per_min(…)`; saving a channel sends a test message |
| 2.0 | from a client tested against 2.0.2; not seen against a live server | no test call |
| 2.1 and later | from Uptrace's published OpenAPI | queries write `perMin(…)`; a **Test** call; webhooks to private addresses are refused unless the server's `outbound_http.ssrf_protection` allows them |
---
## Asking for a monitor
```
When the checkout service's error rate stays above 10 per minute for 5 minutes, run checkout-triage.
```
The agent looks the metric and query up with `uptrace_query` first, then answers with a plan;
nothing is created until you confirm it in your next message:
```
Plan 9a3c6e10 — nothing is created yet.
Uptrace "main" (API of version 1.7), project 1, gets a metric monitor "GoClaw: Checkout errors":
- metric uptrace_tracing_spans as $m, query `per_min(count($m{_status_code="error"})) as errors | where service_name = "checkout"`, column errors
- fires when errors is above 10 for 5 minute(s) in a row
- it notifies only its own webhook channel "GoClaw: Checkout errors", which calls http://192.0.2.10:8080/goclaw/hook/uptrace/checkout-errors with a new secret only this route knows (in the URL: Uptrace sends no header of its own)
- other monitors and channels are not changed
GoClaw: when an alert opens, SOP "checkout-triage" (Checkout triage) runs as agent main.
```
What the request can set: the metric and its alias in the query (`$m` by default), the query and
the column it checks, `above` and/or `below`, how many minutes in a row (`points`, default 5),
the project, and whether the SOP also runs when the alert resolves (`on_resolved`).
The plan is refused, with the reason, when the project does not exist for the token, the query
does not read the metric's alias or names no column, the bounds are missing or the wrong way
round, Uptrace already has a monitor or channel of that name, the route key is taken, the
instance has no user token or does not allow writes, or the server's version cannot be told.
After creating it, the agent has Uptrace send its test message where the version has one, and
says whether it reached the route.
---
## What the SOP is given
Each alert event is the run's trigger data, in an untrusted section of every step. The agent is
told:
```
Alert: GoClaw: Checkout errors — OPEN
Event: created
Type: metric
Since: 2026-10-09T01:55:00Z
Alert URL: https://uptrace.example.com/alerting/1/alerts/88
```
Uptrace 2.x adds the alert's attributes (`Attributes: service.name=checkout, …`). One alert keeps
one session.
The route's events are `alert` (it opened, or opened again), `resolved` (closed, resolved or
archived), `recurring` (still open; Uptrace repeats it), and Uptrace's own name for anything else.
A monitor made for alerts filters on `alert`; one made with `on_resolved` lets `resolved` through
too.
Uptrace retries a delivery the gateway did not take; each retry is the same alert event, which is
run once.
The route's secret is in the channel's URL, where anyone who can see the channel in Uptrace can
read it.
---
## Listing, checking, removing
- **list**: every monitor this gateway registered, its query, whether it is still in Uptrace, and
how many alerts the route received.
- **test**: Uptrace sends its test message to the channel (2.1: the test call; 1.7: saving the
channel again, which Uptrace refuses when the route does not answer). The route answers it and
runs nothing. 2.0 has no test call.
- **delete**: planned and confirmed like a create; removes the monitor, then its channel, and the
route. Deleting the route on the **Webhooks** page removes them too, when the instance allows
writes; otherwise it says what to delete in Uptrace.
Only a human admin can do this, in a conversation of their own: never a SOP, cron, webhook or
subagent run.
---
## Connecting Uptrace without the agent
1. On the **Webhooks** page, create a route of type **Uptrace** with a key, the agent and the
SOP. Set **Events** to `alert` to run only when an alert opens. Saving shows the URL with the
secret in it, once.
2. In Uptrace, add a **Webhook** notification channel to the project with that URL. On 2.1 and
later, untick **match all monitors** and choose the monitor, or every monitor's alerts reach
the route.
3. In the monitor, notify that channel.
4. Test the channel (2.1: **Test**; 1.7: save it again): the route answers without running the
SOP, and its **Hits** go up.
## Related
- [Grafana Alert Rules That Run a SOP](grafana-alert-rules.md) — the same pattern for Grafana
- [Zabbix Problems That Run a SOP](zabbix-alerts.md) — the same for Zabbix
- [SRE Tools](../observability/sre-tools.md#uptrace) — the `uptrace_query` tool and the instance's configuration
- [Standard Operating Procedures](sop.md)