# Standard Operating Procedures (SOPs) SOPs are autonomous, multi-step workflows that allow an AI agent to perform complex tasks without human intervention. Unlike simple Cron Jobs, SOPs include planning, tool restrictions, and self-reflection to ensure reliability. ## Overview An SOP consists of multiple **Steps**. Each step defines: - **Instruction**: What the agent should do. - **Tools**: Which tools are allowed for this specific step. - **Success Criteria**: How to verify the step's output (e.g., `contains:OK` or `json`). - **Failure Action**: What to do if criteria aren't met (`retry`, `abort`, or `escalate`). ## Managing SOPs via WebUI The GoClaw Admin Dashboard provides a dedicated interface for managing SOPs. ### 1. Viewing SOPs Navigate to the **SOPs** tab in the sidebar. Each SOP is a card showing its name and ID, its purpose, the number of steps, its owner, and whether it is **Active** or **Paused**; badges show the linked cron job and how many steps require approval. Click a card to open the SOP: its screen has **Run**, pause/enable and delete buttons, and the tabs **Overview**, **Steps**, **Diagram**, **Channels**, **History** and **Edit**. ### 2. Creating a New SOP Click the **New SOP** button. A blank form opens on the **Edit** tab. - **ID (slug)**: A unique slug (e.g., `daily-report-generator`). - **Name**: A descriptive name for humans. - **Purpose**: A brief explanation for the AI to understand why this SOP exists. - **SOP-level tools**: Tools every step may call in addition to its own (see [Which tools a step may call](#which-tools-a-step-may-call)). - **Shared knowledge**: Markdown injected into every step's prompt (`context`). - **Owner agent**: The agent the run runs as: its model, its workspace and configuration, its core instruction, the hooks limited to it and its daily token limit. `default`, `main` or an empty owner is the default agent (unless an agent is configured under that ID), and so is an owner that names no configured agent. A run a webhook route starts runs as the route's agent instead, and one `sop_run` starts in a conversation as that conversation's agent, on that agent's model too. A run for an agent the gateway is not running (removed from `config.yaml`, added since the gateway started, or whose model failed to start) does not start: a webhook route is answered `409`, `sop_run` and **Run** are refused, and a scheduled run fails before its first step. - **Cron schedule**: When the SOP should run. Saving with a schedule creates a cron job for the SOP and links it, or moves the job it already has to the new schedule (see [Cron Jobs](cron-jobs.md)). Schedules have six fields, seconds first: `0 0 9 * * *` is daily at 09:00; a five-field `0 9 * * *` is read the same. Clearing the field leaves the job as it is: pause or delete it on the **Cron** page. A schedule that cannot be applied is reported after the save, which still goes through. - **Enabled**: A paused SOP is skipped by its schedule and refused by webhooks. - **Workflow steps**: **Add step** adds a step, with the fields described in the [Step Configuration Reference](#step-configuration-reference) below, except `timeout`, which a save keeps (see below). **Validate** checks the draft without saving it; **Save** saves it. ### 3. Editing an SOP Open the SOP and its **Edit** tab. The form has the same fields; the next run uses what you save. > **What a save keeps.** The form has no field for a step's `timeout` or the > SOP's `prerequisites`, `virtual_level` and review dates; a save keeps them as > they are. `POST /api/sop` keeps any field the request leaves out, at the top > level and in each step (matched by its `id`); a field it sends, even empty, > replaces the stored one. A save that changes the SOP's definition archives > the previous version, which `sop_rollback` restores, and raises the version; > pausing or resuming it does not. > > The agent's `sop_save` tool keeps the owner, timezone, SOP-level tools, shared > knowledge and prerequisites when an update leaves them out (an empty value > clears one), always keeps `virtual_level`, and keeps a step's `timeout`, `queries`, > `queries_shadow` and `mcp_call` when the step sends none. It archives the previous version too. ### 4. Running SOPs Manually Click **Run** on the SOP's screen to start a run immediately. This is useful for testing new procedures before scheduling them. The run goes on in the background, as one started with `sop_run`: it ends after 2 hours at most, and it waits for a free slot. Runs started with **Run**, `sop_run` or a webhook, and runs resumed after a restart, share 4 slots; scheduled runs do not take one. A paused SOP, or one overdue for review, is not run: the page says why. Resume it (**Resume** on its screen), or ask the agent to review it (its `sop_review` tool sets the next review date), first. ### 5. Viewing Execution History Open the **History** tab to see a detailed log of past runs. - **Status**: `completed` (every step passed, or a step's `end_run_if` ended the run early with nothing failed: the entry says which step and why; see [Ending a run early, or skipping a step](#ending-a-run-early-or-skipping-a-step)), `partial` (the run reached its last step, or ended early, but at least one step failed), `failed`, `escalated` (a step with `failure_mode: escalate` ended the run), `interrupted` (a restart cut the run off and it was not resumed; see [Time limits and restarts](#time-limits-and-restarts)), or `running`. A run is `failed` when a step with `failure_mode: abort` stopped it, when it could not start because a prerequisite or virtual level check failed, when one step was reached more than 3 times through `on_success_goto`/`on_failure_goto` jumps, or when an attempt did not stop after running out of time. - **Steps**: Shows how many steps passed (e.g., `3/4`). - **Trigger**: `manual` (**Run**, or the agent's `sop_run` tool), `cron`, or `webhook:` for a run a webhook started: `webhook:graylog` for a [Graylog alert](graylog-alerts.md), `webhook:gitlab` for a [GitLab event](gitlab-webhook.md). See [Runs started by a webhook](#runs-started-by-a-webhook). ## Runs started by a webhook A SOP also runs when an outside system calls the gateway. A webhook route with a `sop_id` runs that SOP for the event instead of a chat turn: a [Graylog alert](graylog-alerts.md) works this way, and so can a [GitLab event](gitlab-webhook.md) or anything else that can POST JSON. Routes are made on the **Webhooks** page, in [`config.yaml`](../configuration.md#webhook-routes), or, for a Graylog alert, by the agent. - **One run per event.** With `batch_mode: bulk` the route gathers its events for `batch_window` seconds (default 300) and runs the SOP once for them, at most 500 events per run. - **The runs of one object take turns.** An event about an object whose run is still in progress — an issue or merge request and its comments, one alert and group-by value — is answered `200`, queued, and waits for that run. When it ends, the events that came meanwhile start one run together, every payload in the order they came, so a person editing an issue field by field (GitLab sends an event per field) starts two runs rather than one per field, and the second sees the first one's comment. Queued events survive a restart and run once the gateway is back. Events about different objects run side by side, as do a GitLab project's pushes, jobs and deployments, which name no object of their own. More than 500 events waiting for one object are refused with `503`. - **The event is data, not instructions.** Every step sees it in a section headed *Inbound event* and marked as coming from outside: at most 16 KB of each event, and 48 KB for all the events of a bulk run. It is never added to the SOP's own context. Whoever can write into the payload (a log line, an MR title) writes into that section, so write the steps to treat it as evidence. - **It runs as `webhook-system`**, on the route's agent. That identity is not privileged: whatever tools a step declares, it cannot manage cron jobs or SOPs, write system files, or register Graylog alerts. A step that declares `db_write` or `metrics_store` can write to a datasource opened with `allow_write: true`, as in any other run: only an admin can save a SOP, so the step's tool list carries the author's grant. Every step sees the inbound event, so declare them only where the write is safe whatever the payload says. - **A restart does not resume it.** A webhook-started run that a restart cuts off is closed and reported to the admin (see [Time limits and restarts](#time-limits-and-restarts)). - **The sender is told whether the run started**: `409` when the SOP was deleted or disabled or is overdue for review, `503` when too many webhook-started runs are already waiting or running. A bulk route answers `200` once the event is queued. When its run later cannot start, nothing is sent back: the events are dropped when the SOP refuses to run (deleted, disabled, overdue for review) or the route was deleted, with the reason noted on the route on the Webhooks page and in the gateway log. When too many runs are in progress they stay queued for the next round without a note; after any other error they stay queued and the error is noted on the route and in the log. A paused route's events wait until it is enabled again. The full list is in [Gateway Endpoints](../../api-reference/gateway_endpoints.md#webhook-receiver). ## Step Configuration Reference Each step in the `steps` array has the following fields. These are the **exact JSON keys**. A key spelled any other way is not an error — it is simply ignored, and the step runs as if you had never set it. Copy the names from this table rather than from memory. | Field | Type | Description | | :--- | :--- | :--- | | `id` | string | **Required.** Unique ID for the step. | | `name` | string | Display name. | | `intent` | string | **Required, min 20 chars** (optional for pure deterministic query steps). One sentence saying why this step exists. | | `instruction` | string | **Required, min 80 chars** (optional for pure deterministic query steps). What the agent should do. Can use `{prev_output}` for the result of the step immediately before, or `{{steps..output}}` / `{{steps..file}}` for any earlier step. | | `output_format` | string | The shape the answer must take. Required in practice for any step that declares `outputs` — the Web UI warns when it does not mention JSON. | | `constraints` | array | Inline rules the step must obey, injected verbatim into its prompt. | | `context` | string | Step-level knowledge (markdown) injected into the step prompt. | | `inputs` | object | Hardcoded parameters, e.g. `{"project_id": "5"}`. | | `outputs` | array | Optional. Names this step publishes. Declaring them forces the step to answer with a JSON object keyed by exactly these names. | | `queries` | array | Optional. Deterministic [saved query](../saved-queries.md) bindings (`[{template, datasource, params, as, max_rows}]`). When `queries_shadow` is false, the step runs these queries directly through the worker pool with **zero LLM calls** and publishes `` (the Markdown table) and `_ref` (its `data_ref` ID for `spreadsheet`). Parameter values may be literals, `{{run.date±Nd}}`, `{{run.time±Nh\|m}}` (evaluated in the SOP's `timezone`, defaulting to the server's local timezone), or `{{steps..outputs.}}`. | | `queries_shadow` | bool | Optional. When `true`, `queries` run in shadow mode (recording version, row count, duration, and errors in `query_runs` on the step result) while the step still executes normally through the LLM. | | `for_each` | string | Optional. Collection to fan out over — the step runs once per item. Cannot be combined with non-shadow `queries`. | | `item_var` | string | Optional. Placeholder for the current item, default `item`. | | `max_items` | int | Optional. Fan-out cap, default 50, ceiling 500. | | `on_item_failure` | string | Optional. `abort` (default) or `continue`. | | `mcp_call` | object | Optional, with `for_each`. `{server, tool, arguments, timeout}`: the step calls that MCP tool once per item itself, with **zero LLM calls**, and publishes `results` and `results_ref`. See [Map steps that only call an MCP tool](#map-steps-that-only-call-an-mcp-tool-mcp_call). | | `uses_outputs` | array | Optional. Earlier results this step consumes — `"s1"` for a whole step, `"s1.staff_list"` for one published field. | | `tools` | array | Tool names this step may call. **An empty list does not mean "all"** — see [Which tools a step may call](#which-tools-a-step-may-call) below. | | `success_check` | string | Rule to validate output: `json`, `json_fields:a,b`, `contains:TEXT`, `not_contains:TEXT` (both ignore case), `equals:VALUE`, `min_len:N`, `max_len:N`, `regex:PATTERN`, or `expr:CONDITION`, a comparison of published values (see [Checking published values](#checking-published-values-expr)). Empty means no check. A rule GoClaw does not recognise (a misspelling such as `contain:`), a length that is not a number, or a regex that does not compile is refused when the SOP is saved. A SOP saved before that check runs as it did (an unknown rule, or `min_len` without a number, passes every output), and the gateway logs the rule each time; correct it at the next save. | | `failure_mode` | string | What to do when retries are exhausted: `retry` (the default: carry on to the next step), `abort` (fail the run and run the rollback), or `escalate` (end the run `escalated`, run the rollback, and send a notice to `sop.alerts` in `config.yaml`, or only to the gateway log when that is not set). **Jumping to another step is a separate field — `on_failure_goto`, not a value here.** | | `on_failure_goto` | string | Optional. Step ID to jump to when retries are exhausted. Takes precedence over `failure_mode`. | | `max_retries` | int | How many attempts the step gets in all, the first one included. Default 2 (one retry); `1` means no retry; cap 5. | | `timeout` | string | Optional. How long one attempt may run before it is cancelled and retried, e.g. `"20m"`; between `1m` and `2h`. Default: `sop.step_timeout` in `config.yaml`, 15 minutes when unset. | | `on_success_goto` | string | Optional. Jump to a specific `step_id` instead of the next sequential step (Conditional Branching). | | `end_run_if` | string | Optional. One comparison over published values, e.g. `"{{steps.s3.outputs.rows}} == 0"`, checked after the step passes: when it holds, the run ends there, `completed`, with no model call. See [Ending a run early, or skipping a step](#ending-a-run-early-or-skipping-a-step). | | `skip_if` | string | Optional. The same kind of comparison over earlier steps' values, checked before the step starts: when it holds, the step is skipped and the run goes on to the next one. | | `requires_approval` | bool | Optional. After the step answers, the run waits (`suspended` state) until someone approves or rejects the step through the API — see [Human-in-the-Loop](#human-in-the-loop-hitl). Cannot be combined with non-shadow `queries`. | | `rollback_command` | string | Optional. A shell command that undoes this step, run when a later step ends the run with `abort` or `escalate` (the newest step's first). It runs only if `tools.shell.allow` and the deny lists permit it, and each command is stopped after 60 seconds. | ## Which tools a step may call A step calls **exactly what the SOP declares for it, and nothing else**. Nothing is chosen for it at runtime by relevance ranking. The effective list is built like this: 1. the step's own `tools`, **unioned** with the SOP-level `tools` (declare shared tools such as `memory_write` or `notify` once at SOP level instead of repeating them on every step); 2. `"*"` in either list stands for **every registered tool**; 3. **nothing declared in either list means the step runs text-only** — it works from its inputs and produces an answer, and any tool call it attempts is refused; 4. a virtual level's tool grants **narrow** the result; they never widen it. Six side-effect-free internal tools are available to every step regardless of what it declares: `calc`, `datetime`, `json_transform`, `read_output`, `regex`, `text_utils`. > **A SOP sends only to its pinned targets.** Inside a run, `message` and > `notify` send to the conversation pinned for that SOP and channel, whatever > target the model names; with no pin for the channel, the call is refused > (*`no pinned discord target for SOP …`*) and the step fails. Pin the target on > the SOP's **Channels** tab, with `POST /api/sop/channels`, or by asking the > agent in the destination group to run `sop_pin_channel`. `notify` to > `channel: "webhook"` posts to the URL given and needs no pin. > **`message` and `notify` are not automatic.** A step that sends must declare > the sending tool itself. A step scoped to, say, `["db_query"]` that tries to > `message` is refused at runtime with > *`tool "message" exists but was not offered for this request`* — the report is > built and then never leaves the agent. Put `message` (or `notify`) on the step > whose job is to send, and only on that step: a metrics step that can also post > will eventually post its raw intermediate JSON to your channel. > **A send that goes nowhere fails the step** (from 0.0.86). A step passes on > its own answer, not on whether its tools worked — which is right for a query, > since "no rows" is a real answer, and wrong for a send. So delivery is checked > separately: if a step called `message` or `notify` at all and **every** one of > those calls failed, the step does not pass, whatever it wrote. A step that > never tried to send is unaffected, and one that failed on `message` but got > through on `notify` has delivered and still passes. > > The verdict is applied once, after the retries, not inside them: a channel > that has stopped accepting messages will not start because the step ran again, > so a retry would only repeat the queries and the tokens to reach the same > rejection. > > Every tool error a step hit is recorded on its result as `tool_failures` > (tool, calls, failures, last error) whether or not the step passed, so a step > that quietly got nothing back from a query is visible too. > **A retry does not do again what an earlier attempt did.** A step that fails > its check is retried to correct its answer, not to repeat its work: > > - When an earlier attempt's `message` or `notify` went out, the retry is offered > neither, and is told the message was delivered. > - When an earlier attempt's call that changes something went through, the > retry is not offered that tool, and is told what was already done. This > covers `memory_write`, a `metrics_store` record, `db_write`, `file_write`, > `email` and every other tool that is not read-only (see > [Time limits and restarts](#time-limits-and-restarts)). > - A tool that reads or writes depending on the call stays on the retry, for > its reads, and a write through it is refused. These are `http_request`, > `metrics_store`, and the Uptrace, Grafana and Zabbix tools. > - A call that failed did not go through, so the retry may make it again. > - Two tools stay available whatever they did. `mcp` may only have read, and a > retry may need the data again; `db_analyze` writes only its own schema cache. > - Every attempt starts a new conversation, so the retry is handed what the > earlier ones got, under *From earlier attempts*: the `data_ref` of every > result they stored (up to 20), which `read_output` reads without fetching > the data again, and what each call that went through returned (up to 10, > 400 characters each), so that its answer can say what was done. Tool names must exist in the agent's registry — check `/api/tools` for the live list. **MCP servers are not tools**: reach them through the `mcp` tool (or `db_query` for a datasource), naming the server in the call. From 0.0.86 the step's prompt is scoped the same way: the workspace `TOOLS.md` catalogue is narrowed to the tools that step may call, so it is no longer shown descriptions of tools it would be refused for using. If your own `TOOLS.md` does not use the `- **tool_name** — description` form, it is passed through whole instead of being filtered. > A step still publishing several messages is not blocked — a step may have more > than one thing to say — but a non-map step that publishes more than once logs a > warning, because one report split across posts arrives out of order. ## Passing data between steps A step does **not** automatically see what earlier steps produced. `{prev_output}` only ever means the step immediately before, so a step that needs results from two different places used to have no way to get them. The following fictional work-assignment SOP uses synthetic staff, jobs and categories: | Step | Produces | | :--- | :--- | | `s1` | The staff rostered on shift today | | `s2` | Pending jobs, grouped by category | | `s3` | Assigns `s2`'s example jobs to `s1`'s staff | By the time `s3` runs, `{prev_output}` holds only `s2`. Declare the dependency instead: ```json { "id": "s3", "name": "Assign work", "intent": "Distribute pending category jobs across the staff on shift today", "uses_outputs": ["s1", "s2"], "instruction": "Use the staff roster from step s1 and the pending job list from step s2. Assign the example jobs, matching each job's category to a staff member rostered for that category.", "tools": ["db_query", "file_read"], "output_format": "JSON array of {staff_id, job_id, category}" } ``` The executor injects both outputs under an **Upstream step outputs** heading in `s3`'s prompt. > Declaring `uses_outputs` **replaces** the implicit "Previous step output" > block. A step that asks for one small field should not also be handed the > whole previous answer — that is the cost it was avoiding. List the previous > step in `uses_outputs` when you want both. ## Named outputs A step publishes a single blob of text by default. Declare `outputs` and it publishes addressable fields instead: ```json { "id": "s1", "name": "Shift roster", "outputs": ["staff_list", "shift_date"], "output_format": "JSON object {staff_list: [{staff_id, name, category}], shift_date: \"YYYY-MM-DD\"}", "instruction": "Query the rota for today and return the staff on shift with the category each covers.", "tools": ["db_query"] } ``` The step must then answer with a JSON object carrying exactly those top-level keys. Markdown fences and a short preamble are tolerated; a missing key or unparseable JSON **fails the step** and feeds the reflection retry with a message naming the offending key — so the model is corrected instead of the SOP quietly continuing with nothing. Downstream, take one field rather than the whole answer: ```json { "id": "s3", "uses_outputs": ["s1.staff_list", "s2.jobs_by_category"] } ``` Referencing a name the producing step does not declare is rejected at save time, with the real list of names in the error. ### Inline references For finer control, name a value where you want it to land: | Reference | Expands to | | :--- | :--- | | `{{steps.s1.output}}` | That step's full captured answer, uncapped | | `{{steps.s1.file}}` | The artifact path of that answer, for `file_read` | | `{{steps.s1.outputs.staff_list}}` | One published field | | `{{steps.s1.outputs.staff_list.file}}` | That field's own artifact path | | `{prev_output}` | The previous step only (unchanged) | Single braces (`{steps.s1.output}`) work too. A reference that is also declared in `uses_outputs` is injected once, not twice. A misspelled selector produces an explicit marker in the prompt rather than passing through as literal text. ## Map steps (one call per item) A step that handles a whole collection does it in one LLM call: *"assign the pending jobs across every category"*. That holds until the collection grows. The model must then fit every category into one prompt and write every assignment into one answer — which is where long lists get silently truncated, and where one bad row fails the entire step. `for_each` turns that into one call per item: ```json { "id": "s3", "name": "Assign per category", "for_each": "{{steps.s2.outputs.categories}}", "item_var": "category", "on_item_failure": "continue", "uses_outputs": ["s1.staff_list"], "instruction": "Fetch the pending jobs for {{category}} with db_query and assign them to staff rostered in {{category}} today. Handle only this category.", "outputs": ["assignments", "unassigned_count"], "output_format": "JSON object {assignments: [{staff_id, job_id}], unassigned_count: N}", "tools": ["db_query"] } ``` Each iteration's prompt carries **one** category, so its size stays constant however many categories there are. Every iteration is also told, in its own header, that it is handling one item of N and must not touch the others. Map iterations do not automatically receive the previous step's whole answer. Declare `uses_outputs` or an explicit output placeholder when an iteration needs additional shared data. Retries keep the same resolved inputs and item scope, along with feedback from the last attempt. ### The collection `for_each` is expanded against the run before it is read, so it normally names an upstream value. A JSON array, one item per line, or a comma-separated line all parse — but have the producing step publish a **JSON array**, which is the only unambiguous form. An unresolvable reference fails the step rather than iterating once over the error text. `{{item_index}}` (1-based) and `{{item_count}}` are available inside every iteration alongside your `item_var`. ### Gathering the results Declared `outputs` are collected across iterations: - if every iteration returned a **JSON array** for a name, the published value is those arrays **concatenated** — one flat list of all assignments; - otherwise it is the **list of per-iteration values**, in item order. Original JSON types are preserved: a string ID such as `"123"` stays a string, and `null` retains its position in the result list. Failed iterations contribute nothing. The step's own `output` is a JSON document listing every iteration with its item, status and answer — useful for auditing, and what lands in the artifact file. ### Failure and limits | Setting | Behaviour | | :--- | :--- | | `on_item_failure: "abort"` (default) | The first failing item fails the step. | | `on_item_failure: "continue"` | Every item runs; the step passes if at least one did, and the aggregate records which failed. | | `max_items` | Default 50, ceiling 500. Extra items are not run, rather than failing the step, so one over-broad upstream query cannot become thousands of LLM calls. The step's output says how many were left out (`not_run_items` and a `note`), so the step after it knows the results are incomplete. | An empty collection is a **pass**, not a failure — there was simply nothing to do this run. Cancellation fails the map even if some iterations already succeeded; completed results remain available for inspection, and no further items are started. Iterations run **sequentially**. The gain is bounded prompt size and per-item error isolation, not wall-clock time. `for_each` cannot be combined with `requires_approval`; approve a summarising step before or after the map instead. ### Map steps that only call an MCP tool (`mcp_call`) Some map steps do nothing for an item but one MCP tool call whose arguments come straight from the item: add each ticket of a list to a work plan, create each row in another system. Run through the model, every item costs an LLM round trip carrying the whole step prompt. On one pilot agent such a step made 349 calls through about 280 LLM calls of ~45k prompt tokens each, 13.9M tokens a day, although the model added nothing to any call. Give such a step `mcp_call` instead of a brief, and the executor makes the calls itself: ```json { "id": "s3", "name": "Add tickets to the plan", "for_each": "{{steps.s2.outputs.tickets_ref}}", "item_var": "ticket", "max_items": 400, "on_item_failure": "continue", "tools": ["mcp"], "mcp_call": { "server": "planner", "tool": "add_item", "arguments": {"ticket_id": "{{ticket.id}}", "owner": "{{ticket.assignee}}", "day": "{{run.date}}"}, "timeout": "30s" } } ``` - **The rows.** `for_each` may name a `data_ref` (a [saved query step](../saved-queries.md)'s `_ref`, or a ref another step published): each row of its table is an object of its columns, named as `data_compute` names them (lower case, other characters as `_`: `Ticket ID` is `ticket_id`). It may also be a JSON array (objects keep their fields and types) or a plain list. - **The arguments.** A string value may be one whole placeholder: `{{ticket}}` (the row), `{{ticket.}}`, `{{item_index}}`, `{{item_count}}`, `{{run.date±Nd}}`, `{{run.time±Nh|m}}`, `{{steps..outputs.}}` or `{{steps..output}}`. A placeholder inside other text (`"ticket {{ticket.id}}"`) is refused when the SOP is saved: a row's value goes in as a whole argument, never spliced into a query or command the server would read. Table cells are text; the tool's input schema turns them into numbers or booleans where it asks for one. - **The server** is a configured name (`mcp.servers` in `config.yaml`, or an approved plugin server of the agent), never a URL, and the step must have `mcp` among its tools (or the SOP's), as an LLM step would. - **What it publishes.** `results` is a table with one line per row: its status (`ok`, `failed` or `not run`), the arguments sent, what the server answered and the error, each cell cut at 300 characters. `results_ref` is the same table as a `data_ref` with every cell whole, for `read_output`, `data_compute` or `spreadsheet`. The step's own output is the counts and the failed rows, for the step after it. `intent` and `instruction` are optional; `outputs` cannot be declared. Each call is the one the model would have made: `mcp` with `call_tool`, through the worker pool. The Hooks page's rules see every call with its row's arguments, every call is recorded in the run's effects journal before it is made, and each shows in the trace. The step makes **one** attempt: - more rows than `max_items` (default 50, ceiling 500) fails the step before the first call, instead of doing the first 50 of a list that is longer than expected; - `on_item_failure` works as for any map step: `abort` stops at the first failed row and marks the rest `not run`, `continue` calls every row and passes if one succeeded. A tool error the server answers with is a failed row; - `mcp_call.timeout` limits one call (default `1m`, between `1s` and `10m`); the step's `timeout` limits all of them together; - a step that has made a call is never run again in the same run: not retried, not resumed after a restart, and refused when an `on_failure_goto` leads back to it, since every call would be made twice; - it does not run in a run that acts for a person (a delegated cron or virtual-level run): nothing confines an MCP server's calls to what that person may see or change. ## Keeping prompts small Every value that moves between steps costs tokens twice: the producing step writes it, the consuming step reads it. Three mechanisms bound that, in increasing order of effectiveness: 0. **Fan out instead of batching.** A `for_each` step's prompt does not grow with the collection at all — it is the only option on this list that is bounded by construction. 1. **Take the field, not the answer.** `uses_outputs: ["s1.staff_list"]` instead of `["s1"]`. Typically halves the handoff, often much more. 2. **Hand over a reference.** Values past the inline limit (`sop.inline_output_chars`, 12,000 characters) are cut in the prompt; the step is told so and given a `data_ref` and the artifact path, and reads what it needs with `read_output` or pages through the file with `read_file_lines`. 3. **Do not move the rows at all.** The most effective option, and the one no handoff mechanism can substitute for: if a step must emit 5,000 rows, the model has to *write* 5,000 rows — slow, expensive, and where long lists get silently truncated or invented. Instead have the producing step publish a **reference** and let the consumer re-query: ```json { "id": "s2", "outputs": ["jobs_query", "job_count", "categories"], "instruction": "Do NOT return the rows. Return the SQL that selects them, the row count, and the distinct categories." } ``` `s3` then runs `jobs_query` itself with `db_query`. Rows never enter a prompt; only the query, a count, and a short category list do. Rule of thumb: **aggregates and identifiers travel through the prompt; rows do not.** A step whose output grows with the size of the dataset is a step that will eventually break. ### Large tool results inside a step A tool result over `tools.max_result_chars` (default 8,000) — and every `db_query` result regardless of size — is **spilled** out of the prompt into an in-memory store, and the step receives a `ref` plus a schema-only preview instead of the rows. It reads what it needs with `read_output(ref, …)`, or hands the ref straight to `spreadsheet(data_ref: …)` so the rows never enter a prompt at all. See [the tools engine](../../architecture/tools_engine.md) for the full mechanism. That store evicts after an hour, keeps only the newest 200 entries, and is lost when the gateway restarts — while a SOP may run for two hours, and a step waiting on an approval can sit idle for one. A ref produced by an early step and read by a later one could therefore expire mid-run. So inside a SOP the spilled result is **also written under the run's own directory**: ``` /sop_runs///tool-.json ``` `read_output` falls back to that file when the in-memory entry is gone, and puts it back in memory for the rest of the run. The file belongs to the execution that produced it — another run of the same SOP cannot resolve the ref — and it is swept with the rest of that run's artifacts under the same 20-run retention. One consequence worth knowing: a rehydrated entry carries no pre-generated CSV, because that file lives in the OS temp directory rather than in the run. A `spreadsheet(data_ref: …)` call after a rehydration parses the content instead of streaming the file — slower, same result. ### Artifact files Every output — and every named output — is written to a file: ``` /sop_runs///.md /sop_runs///..md ``` Each step's prompt carries an index of the artifacts written so far in that run, so a step can reach back past `uses_outputs` with `file_read` — without anyone having to predict a filename. > The index lists what this step's prompt does **not** already carry (from > 0.0.86). A value delivered in full through `uses_outputs`, through a > `{{steps...}}` placeholder, or as the previous step's output is not offered as > a file, because a step shown a path reads it: on the pilot fleet a reporting > step re-read its own upstream metrics in 37 of 53 runs, and another spent a > tool call fetching a 10-character date it had been handed three paragraphs > earlier. A **truncated** value keeps its entry — that path is the only way to > the rest of it — and so does every step this one never asked for, which is > what the index is for. The execution ID is part of the path, so concurrent runs of the same SOP never overwrite each other, and the files stay on disk afterwards for debugging. `sop_history` reports each step's `artifact_path`. The **20 most recent runs** of each SOP are retained; older run directories are removed when the SOP next executes. Automatically injected values longer than **12,000 characters** (`sop.inline_output_chars`) are cut in the prompt. A note before the value says how much of it is shown, the line the rest starts at, and where all of it is: a `data_ref` that `read_output` reads (`slice`, `search` or `sql`), and the artifact file. **Declare `file_read` on any step that may receive a long list** — the validator warns when a step consumes upstream output without it — and `read_file_lines` when it may need to page through one. A step without either gets the value whole, however long. An explicit `{{steps.s1.output}}` reference is never cut: naming a value by hand is taken as asking for all of it. ### Validation Output references are checked when the SOP is saved: - referencing a step that does not exist → rejected - referencing a step that runs **later** → rejected - referencing the step's own output → rejected - referencing a name the producing step does not publish → rejected - an output name that is not a plain identifier, or declared twice → rejected - `for_each` combined with `requires_approval` → rejected - `item_var` / `max_items` / `on_item_failure` set without `for_each` → rejected (the step would have run once, which is not what the author meant) - `max_items` above the ceiling, or an unknown `on_item_failure` → rejected - `mcp_call` without `for_each`, with a URL for `server`, without `mcp` in the step's or the SOP's tools, with `outputs`, or with a placeholder inside other text in its `arguments` → rejected - an `end_run_if` or `skip_if` that is not one comparison as described in [Ending a run early, or skipping a step](#ending-a-run-early-or-skipping-a-step), or that reads a step that runs later, a name that step does not publish, or (`skip_if`) the step's own outputs → rejected - a `success_check` of `expr:` with no condition, or one that is not one comparison, or that reads a step that runs later or a name that step does not publish → rejected The Web UI also warns when a map step's instruction never mentions its `item_var` (every iteration would get an identical brief), and when a map step lists the collection it iterates in `uses_outputs` as well (every iteration would carry the whole list on top of its own item). The Web UI additionally warns when a step declares `outputs` but its `output_format` does not mention JSON. Failed steps are still addressable by ID; the prompt marks their output as not having passed its success check. A step that never ran in this execution (a branch skipped it) yields an explicit "did not run" marker rather than an invented value. ## SOP v2: Advanced Features ### Conditional Branching (If/Else) Unlike simple linear scripts, GoClaw SOPs can conditionally skip or loop steps based on verification criteria. By setting `on_failure_goto: "remediation_step"`, an agent will autonomously detour to fix a failed constraint. If successful, `on_success_goto: "deploy_step"` can jump back to the standard flow. (`failure_mode` only chooses between `retry`, `abort` and `escalate` — the jump target is its own field.) ### Ending a run early, or skipping a step `on_success_goto` and `on_failure_goto` jump on pass or fail, so a SOP whose early step finds nothing to do still runs, and pays for, every step after it. On one pilot agent a daily allocation SOP's third step found no staff for the slot, and its instructions said to stop there. The run still went on through a map over zero items, a report and an audit: 766k tokens for a run that allocated nothing. Two step fields let the executor make that decision itself, with no model call: | Field | Checked | When its condition holds | | :--- | :--- | :--- | | `end_run_if` | after the step passes | The run ends there. No later step runs. | | `skip_if` | before the step starts | The step does not run, and the run goes on to the next step. | ```json { "id": "s3", "name": "Open requests", "outputs": ["requests", "rows"], "output_format": "JSON object {requests: [{id, title}], rows: N}", "end_run_if": "{{steps.s3.outputs.rows}} == 0" } ``` ```json { "id": "s5", "name": "Report", "skip_if": "{{steps.s4.outputs.assignments}} == []" } ``` A condition is **one comparison**: - Each side is a named output, `{{steps..outputs.}}`, or a value: a number (`0`, `-1.5`), a quoted string (`"changed"` or `'changed'`), `true`, `false`, or `[]`. - `==` and `!=` compare numbers when both sides are numbers, and text otherwise (the published value trimmed; `true` and `false` ignore case). `<`, `>`, `<=` and `>=` compare numbers only. - `[]` means empty: a blank value, `[]`, `{}`, `null`, or a saved query that returned no rows. It goes with `==` or `!=` only. - There is no `and`/`or` and no arithmetic, and a step's whole answer (`{{steps..output}}`) cannot be compared. A decision that needs more belongs in the step that publishes the value: have it publish, say, `nothing_to_do: true`. `end_run_if` reads the step's own outputs or an earlier step's; `skip_if` reads earlier steps only. Both are checked when the SOP is saved. A run that ends early is `completed` (or `partial` when an earlier step had failed, with the usual alert) and counts as passed in the SOP's statistics; no alert is sent for it. Its History entry says which step ended it and what held, for example `step "s3": end_run_if {{steps.s3.outputs.rows}} == 0 held (s3.rows = 0)`. So do its trace (the final output and the `sop_end_reason` metadata), the cron job's last output and `sop_history`. Only the steps that ran used tokens; a step that reaches the run's token limit and ends the run completes it rather than failing it. The decision is saved with the step, so a restart right after it does not go on to the steps it ended. A skipped step shows in the History as ○ with its reason. It publishes nothing: a later step that reads it gets the "did not run" marker, and `{prev_output}` stays the last step that ran. It is not a failure. A condition that cannot be decided does not hold: the step it reads did not run or did not pass, or a value that should be a number is not. The run then goes on as it would without the condition, and the step's record says why. ### Checking published values (`expr:`) The other `success_check` rules read the step's answer as text, so none of them can say "fail when these two numbers differ". On one pilot agent a daily allocation SOP's audit step published 253 assigned against 200 expected and passed; the report before it had said everyone got their share, and one person had got none. The audit found the mismatch, and nothing could fail on it. `expr:` takes one comparison, in the same language as `end_run_if`, and the step passes only when it holds. The executor decides it from the values the step published, with no model call: ```json { "id": "s4", "name": "Audit assignments", "intent": "Count today's assignments against the jobs that were due", "instruction": "Count the jobs assigned today and the jobs that were due today with db_query. Publish both counts as found.", "tools": ["db_query"], "outputs": ["assigned", "expected"], "output_format": "JSON object {assigned: N, expected: N}", "success_check": "expr:{{steps.s4.outputs.assigned}} == {{steps.s4.outputs.expected}}", "failure_mode": "escalate" } ``` - The step reads its own outputs as `{{steps..outputs.}}`, from the attempt being checked, and an earlier step's as usual. Each name it reads must be one the step publishes: declare it in `outputs` (or as a query's `as`). - A comparison that does not hold fails the step, and the message names the values: `success_check expr:{{steps.s4.outputs.assigned}} == {{steps.s4.outputs.expected}} does not hold (s4.assigned = 253, s4.expected = 200)`. The failure takes the usual path: the step's retries, then its `on_failure_goto` or `failure_mode`, and the admin alert carries the message. - A comparison that cannot be decided fails too, and says why: the value is not a number, a step it reads did not run or did not pass, the step did not publish the name. This is the opposite of `end_run_if`, where a condition that cannot be decided lets the run go on: a check that cannot tell is not a pass. - The retry prompt carries the values and tells the model to derive them again from the data, not to change them until the check holds. The step's own prompt says the same. A check that reads only earlier steps' outputs is not retried, since another attempt cannot change them. - On a map step the check runs on each iteration's own outputs; on a query or `mcp_call` step, on the values the step published, with no retry. Compare two published values, or one with a fixed value (`expr:{{steps.s4.outputs.unassigned}} == 0`). The check is only as good as the values: a count the step reads from the database is data, while a number the model writes from memory is its word. Have the step publish what it counted, with the tool that counted it. ### Human-in-the-Loop (HITL) Critical procedures (e.g., executing schema drops or sending broadcast emails) can be safeguarded by setting `requires_approval: true`. When the agent reaches this step: 1. The step runs once and its answer is held: the step's job enters a `suspended` state in the job queue, and the run waits. 2. A notice with the step's output and its job ID goes to `sop.alerts` in `config.yaml`, or only to the gateway log when that is not set. 3. A human administrator approves or rejects it. In a chat with the agent, ask it to approve (or reject) the SOP step with that job ID: its `sop_approve` tool does it, and refuses anyone who is not an administrator, and any SOP, cron or webhook run. Or call the API with the gateway token: ```bash curl -X POST https:///api/sop/approve \ -H "Authorization: Bearer $GOCLAW_TOKEN" -H "Content-Type: application/json" \ -d '{"job_id": "", "action": "approve"}' ``` To reject, send `"action": "reject"` and, optionally, a `"reason"`. 4. Approved, the step's `success_check` decides whether it passed, and the run goes on from there. Rejected, the step fails and its `on_failure_goto` or `failure_mode` applies. A step that nobody approves or rejects within 1 hour fails the same way. The wait for approval lives in the running gateway. A restart ends it, and the run is then treated like any other run a restart cut off (below). ## Time limits and restarts **Each attempt of a step has a time limit**, 15 minutes unless `sop.step_timeout` in `config.yaml` or the step's own `timeout` says otherwise. A whole run is still capped at 2 hours. An attempt that runs out of time is cancelled and counts as a failed attempt. What happens next depends on what it had done: - It had only read: it is retried, with a note to make fewer and smaller tool calls, and `failure_mode` decides what happens when the retries run out. - It had already called something that sends or changes (see below): it is not retried, so that is not done twice. - It did not stop within 10 seconds of being cancelled: it may still be running, so the run ends there, whatever `failure_mode` says, and the admin is told. The rollback is not run either, since the attempt may still be using what it would undo; the alert names the steps whose `rollback_command` is left to a person to decide. **A crash in the SOP executor ends the run, not the gateway.** The run ends `failed` (the log shows `[sop_v2] PANIC` and the stack): the steps that finished keep their results, the step it was on says `panic: …`, and the admin is told through `sop.alerts`. The rollback is not run, since what the run had done when it crashed is not known; the alert names the steps whose `rollback_command` is left to a person to decide. **A run is saved as it goes**: when it starts, and as each step starts and ends. When the gateway is stopped or restarted in the middle of a run, the next start finds it and marks it `interrupted`. It then does one of two things. - **Resume from the step it was on**, when all of these hold: - the run was started by a schedule (not by hand or by a webhook), and that schedule still exists, is not paused, and still runs as the same person. Several schedules can run one SOP for different people; a resume acts only as the one the run began for; - the SOP has not been edited since the run started (its purpose, steps, context, tools, prerequisites, owner and virtual level); the finished steps belong to the version the run began with; - it started at most `sop.resume_window` ago (10 minutes by default; `"0"` never resumes), no newer run of the SOP has started, and the SOP is enabled; - the run was not already ending: a step that failed with `failure_mode: abort` or `escalate`, a rollback, an attempt that did not stop; - the step it was on had made no call that sends or changes something. The steps that finished are not run again; their results are carried over. The run's history shows where it was interrupted. - **Otherwise close it and tell the admin** through `sop.alerts`, with the reason it was not resumed. Check what the step had done before running the SOP by hand: the alert names its calls. **What counts as a call that sends or changes something**: every tool call that is not known to be read-only. Read-only are database queries (`db_query`, which only runs SELECT, SHOW, DESCRIBE and EXPLAIN), Uptrace, Graylog, Wazuh and Grafana queries, Zabbix `*.get` methods, `metrics_store` `query` and `trend`, `http_request` GET and HEAD, memory and file reads, and the date, calculation and text tools. Everything else counts — `message`, `notify`, `db_write`, `shell_exec`, MCP tools, an Uptrace or Grafana annotation, and any tool not listed — whether the call finished or not: a message may or may not have arrived, a row may already be written. Such a call is recorded before it is made; if the record cannot be written (a locked database, a full disk), the call is refused and the step sees the error, rather than make a call that recovery would not know about. A refused call changed nothing, so it does not stop a timed-out attempt from being retried. ```yaml sop: alerts: { channel: msteams, target: "" } step_timeout: 15m resume_window: 10m max_run_tokens: 500000 heavy_run_tokens: 200000 inline_output_chars: 12000 max_output_tokens: 8192 ``` ## Token limit of a run **A run stops at 500,000 tokens** (`sop.max_run_tokens`), its steps, attempts and retries together. It is checked before every model call. A run that reaches it ends `failed`: it does not retry, does not go on to the next step, and its rollback runs. A cron job whose run stopped there is paused (`token_limit` on the Cron page), so it does not spend as much again at the next tick. A run past 200,000 tokens (`sop.heavy_run_tokens`) is marked heavy. A SOP whose early step can find nothing to do should stop there with [`end_run_if`](#ending-a-run-early-or-skipping-a-step) rather than run its later steps to say so. The run history shows what each run and step used. See [Token Budgeting](../token-budgeting.md#3-sop-run-limit) for what a normal run uses, and for raising or turning off the limit. ## How much of a step's result reaches the next step A step's result goes through several limits on its way to the steps after it. Each protects the run's token budget, and each says so when it cuts in: | What | Limit | Setting | When it cuts in | | :--- | :--- | :--- | :--- | | A step's answer | 8,192 output tokens a model call (a chat reply on OpenAI and Ollama: 4,096). Claude, DeepSeek and Gemini agents keep their own limit when it is higher: `ai.anthropic.max_output_tokens` and `ai.deepseek.max_output_tokens` (16,000 by default, thinking included), `ai.gemini.max_output_tokens` (16,384). | `sop.max_output_tokens`; 0 = the provider's own | The model stops mid-answer. The step is asked again, for a shorter answer that gives counts and the `data_ref` of the rows rather than the rows. If its last attempt is cut too, its output ends with *[This answer stopped at the model's output limit …]*, and a step that publishes `outputs` fails with that reason. | | An earlier step's output in a step's prompt | 12,000 characters, for a step with `file_read` or `read_file_lines` | `sop.inline_output_chars`; 0 = never cut | The value is cut, with a note before it naming its `data_ref` and file. See [Artifact files](#artifact-files). | | A saved-query step's output | 12,000 characters; 256K when every step that reads it has `file_read` or `read_file_lines` | `sop.inline_output_chars` | The step fails: *query step output is N characters, exceeding the M-character limit*. | | A tool result inside a step | 8,000 characters | `tools.max_result_chars` | The step sees a preview and a `ref`. See [Large tool results inside a step](#large-tool-results-inside-a-step). | | A step's older tool results | Results older than the newest 4 rounds, over 2,000 characters | `tools.full_result_rounds`; -1 = never | Each is cut to its first lines and the `ref` that reads it whole. | | A step's context | 80,000 tokens | `memory.auto_flush_tokens` | The step's older tool results leave the context; a note lists each with the `ref` that reads it whole. The step's own prompt, with the earlier steps' outputs, stays. | | `mcp_call` results | 300 characters a cell of `results` | — | `results_ref` holds every cell whole. | | A map step's items | `max_items`, 50 by default, ceiling 500 | the step's `max_items` | Items past it are not run, and the step's output says how many. | | A run | 500,000 tokens | `sop.max_run_tokens` | The run stops. See [Token limit of a run](#token-limit-of-a-run). | Every `data_ref` made in a run is also written under the run's directory, so it still reads after the in-memory store's hour, or a restart. The run's record in the database keeps every step's output whole: no column there is limited. **Every cut is shown.** A run in which a limit cut data says **⚠ data cut** in the SOP's **History** tab; opened, each step lists its cuts under it (what was cut, the setting, where the rest is), and the run's result keeps them as the step's `warnings`. On the **Traces** page the trace has the same tag and a ⚠ span under the step. See [Where a limit cut data](../../architecture/observability.md#where-a-limit-cut-data). ## What happens to SOPs you never touch > **One exception, from 0.0.85 — read this before upgrading.** A step that > declares no `tools`, in a SOP that declares none either, used to have its > tools chosen for it at runtime by relevance ranking. It now runs **text-only** > and every tool call it makes is refused. If such a step used to query a > database or send a message, it will stop doing so silently — the run can still > end `completed`. Give it the tools it needs, or `"*"`, before you upgrade. A step > that already declares its tools is unaffected. In the same release, `"*"` is > understood for the first time: a SOP saved through the `@sop:` directive > writes `tools: ["*"]`, and before 0.0.85 that meant no tools at all. Otherwise, taking the binary update without migrating anything is safe. The data-handoff fields are opt-in, and a SOP that declares none of them is executed the way it always was: - `{prev_output}` and the "Previous step output" block behave identically. - **Nothing is ever truncated that the step could not fetch back.** The 12,000 character cap only applies when the step declares `file_read` or `read_file_lines` — otherwise the value is passed through whole, however large, exactly as before. A step is never handed a pointer it cannot follow. - The artifact index is only shown to steps that can read files, so a legacy step's prompt gains nothing it cannot act on. - Validation only rejects the new fields. A SOP without them cannot fail a check it passed before, and validation does not run at execution time anyway. - Cron, the job queue and the scheduler are untouched. From 0.0.86 one more thing changes for every SOP: a run that reaches its last step with any step failed ends **`partial`** instead of `completed`, and `run_stats` counts it under Failed. Nothing about how the run executes changed — `failure_mode: retry` still carries on to the next step — only what the record says afterwards. A SOP that was quietly failing a step every run will start reporting it; that is the point, and the number it reports was always the true one. Two things do change for every SOP, migrated or not: - **Step outputs are written to disk.** Each step's result lands in `sop_runs///.md` inside the workspace. The newest 20 runs per SOP are kept and older ones are swept on the next execution, so growth is bounded, but a SOP with large outputs on a frequent schedule will use noticeably more disk than before. A failed write is logged and the run continues. - **A step whose LLM call fails outright now ends in a terminal job state.** Previously a stream or drain error on the final attempt left the step's row in the job queue stuck at `running` forever; it is now marked `failed`. ## Upgrading existing SOPs SOPs written before the handoff model still run unchanged — and still carry its failure. [sop-migration-prompt.md](sop-migration-prompt.md) holds a prompt you hand to an agent so it audits and upgrades the SOPs it owns, with a diagnosis checklist and the guardrails to keep it from rewriting what already works. ## Best Practices - **Atomic Steps**: Keep instructions focused. Instead of "Fix the code and deploy", use Step 1: "Find the bug", Step 2: "Write the fix", Step 3: "Run tests", Step 4: "Deploy". - **Strict Criteria**: Use `json` criteria for steps that should output structured data to ensure follow-up steps can parse it correctly. - **Declare Your Inputs**: If a step consumes an earlier step's result, list it in `uses_outputs`. Relying on `{prev_output}` alone silently loses everything older than one step. - **Publish Fields, Not Blobs**: Declare `outputs` on any step whose result is consumed in parts. A downstream step that needs one field should ask for one field. - **Never Page Rows Through the LLM**: If the row count grows with the data, publish a query or a table name instead of the rows themselves. - **Fan Out Over Groups**: If a step says "for each category / branch / customer", make it a `for_each` step. One call per group beats one call for all of them, and a failure is isolated to its own item. - **Least Privilege**: Only allow necessary tools for each step to minimize the risk of unintended actions during autonomous runs. - **Decouple Frequent Collection from Reporting (Paired SOPs)**: When monitoring systems 24/7, do not attempt to run a 24-hour analysis in a single run or flood channels hourly. Instead, pair an hourly collector (`_hourly`, e.g. `0 0 * * * *`) that records data via `metrics_store` with an evening summary SOP (`_daily_summary`, e.g. `0 0 22 * * *`) that queries the day's metrics and prepares executive digests.