Uptrace Monitors That Run a SOP

An Uptrace metric monitor checks a query every minute, and opens an alert when the value stays beyond its bounds. With a webhook notification channel that calls the gateway, an alert runs a SOP, with the alert's name, status and attributes as its input.

flowchart LR
    MON["<b>Uptrace Monitor 'GoClaw: &lt;title&gt;'</b><br/>query, above/below, N minutes"] -->|"opens an alert"| CH["<b>Webhook Channel 'GoClaw: &lt;title&gt;'</b><br/>notifies only this monitor"]
    CH -->|POST| GW["<b>/goclaw/hook/uptrace/&lt;key&gt;?token=…</b>"]
    GW --> SOP["<b>SOP Run on the Agent</b>"]

There are two ways to set it up:

Ask the agent. An admin describes it in a conversation. The agent's uptrace_monitor tool makes the monitor and its channel in Uptrace and the route in GoClaw, and removes them together. See Asking for a monitor.
Connect it yourself. Make an uptrace route on the Webhooks page, and in Uptrace a webhook channel with the URL the page shows, which a monitor notifies. See Connecting Uptrace without the agent.

Before you start

What	Where
The Uptrace instance with `url` and a user token, and `allow_write: true` (only when the agent makes monitors)	`tools.observability.uptrace.<name>`: the Integrations page or `config.yaml`
The project the monitor belongs to	the instance's `project_id`, or named in the request
An address Uptrace can reach the gateway at	`gateway.ingress.url`, and a listener there
A SOP for the alerts to run	the SOPs page, or the agent's `sop_save` tool (Standard Operating Procedures)

gateway:
  bind: lan
  ingress:
    url: "http://192.0.2.10:8080"     # as Uptrace sees this gateway

tools:
  observability:
    uptrace:
      main:
        url: "https://uptrace.example.com"
        token: "${secret:UPTRACE_USER_TOKEN}"   # a user's token, not a DSN's project token
        project_id: "1"
        allow_write: true                       # off by default: the agent cannot make or remove monitors

A DSN's project token sends data; it cannot manage monitors. The user token is in Uptrace's user profile (2.x), or auth.users[].auth_token in Uptrace's own config (1.7).

Uptrace versions. Uptrace does not document the API its UI uses to manage monitors, and it changed twice. The agent finds out which one the server speaks before it plans, and the plan names it:

Version	How sure	Notes
1.7	checked against Uptrace's source	queries write `per_min(…)`; saving a channel sends a test message
2.0	from a client tested against 2.0.2; not seen against a live server	no test call
2.1 and later	from Uptrace's published OpenAPI	queries write `perMin(…)`; a Test call; webhooks to private addresses are refused unless the server's `outbound_http.ssrf_protection` allows them

Asking for a monitor

When the checkout service's error rate stays above 10 per minute for 5 minutes, run checkout-triage.

The agent looks the metric and query up with uptrace_query first, then answers with a plan; nothing is created until you confirm it in your next message:

Plan 9a3c6e10 — nothing is created yet.

Uptrace "main" (API of version 1.7), project 1, gets a metric monitor "GoClaw: Checkout errors":
- metric uptrace_tracing_spans as $m, query `per_min(count($m{_status_code="error"})) as errors | where service_name = "checkout"`, column errors
- fires when errors is above 10 for 5 minute(s) in a row
- it notifies only its own webhook channel "GoClaw: Checkout errors", which calls http://192.0.2.10:8080/goclaw/hook/uptrace/checkout-errors with a new secret only this route knows (in the URL: Uptrace sends no header of its own)
- other monitors and channels are not changed
GoClaw: when an alert opens, SOP "checkout-triage" (Checkout triage) runs as agent main.

What the request can set: the metric and its alias in the query ($m by default), the query and the column it checks, above and/or below, how many minutes in a row (points, default 5), the project, and whether the SOP also runs when the alert resolves (on_resolved).

The plan is refused, with the reason, when the project does not exist for the token, the query does not read the metric's alias or names no column, the bounds are missing or the wrong way round, Uptrace already has a monitor or channel of that name, the route key is taken, the instance has no user token or does not allow writes, or the server's version cannot be told.

After creating it, the agent has Uptrace send its test message where the version has one, and says whether it reached the route.

What the SOP is given

Each alert event is the run's trigger data, in an untrusted section of every step. The agent is told:

Alert: GoClaw: Checkout errors — OPEN
Event: created
Type: metric
Since: 2026-10-09T01:55:00Z
Alert URL: https://uptrace.example.com/alerting/1/alerts/88

Uptrace 2.x adds the alert's attributes (Attributes: service.name=checkout, …). One alert keeps one session.

The route's events are alert (it opened, or opened again), resolved (closed, resolved or archived), recurring (still open; Uptrace repeats it), and Uptrace's own name for anything else. A monitor made for alerts filters on alert; one made with on_resolved lets resolved through too.

Uptrace retries a delivery the gateway did not take; each retry is the same alert event, which is run once.

The route's secret is in the channel's URL, where anyone who can see the channel in Uptrace can read it.

Listing, checking, removing

list: every monitor this gateway registered, its query, whether it is still in Uptrace, and how many alerts the route received.
test: Uptrace sends its test message to the channel (2.1: the test call; 1.7: saving the channel again, which Uptrace refuses when the route does not answer). The route answers it and runs nothing. 2.0 has no test call.
delete: planned and confirmed like a create; removes the monitor, then its channel, and the route. Deleting the route on the Webhooks page removes them too, when the instance allows writes; otherwise it says what to delete in Uptrace.

Only a human admin can do this, in a conversation of their own: never a SOP, cron, webhook or subagent run.

Connecting Uptrace without the agent

On the Webhooks page, create a route of type Uptrace with a key, the agent and the SOP. Set Events to alert to run only when an alert opens. Saving shows the URL with the secret in it, once.
In Uptrace, add a Webhook notification channel to the project with that URL. On 2.1 and later, untick match all monitors and choose the monitor, or every monitor's alerts reach the route.
In the monitor, notify that channel.
Test the channel (2.1: Test; 1.7: save it again): the route answers without running the SOP, and its Hits go up.

Grafana Alert Rules That Run a SOP — the same pattern for Grafana
Zabbix Problems That Run a SOP — the same for Zabbix
SRE Tools — the uptrace_query tool and the instance's configuration
Standard Operating Procedures