# Subagents & the Job Queue

GoClaw lets a **parent agent delegate a scoped task to a child agent** ("subagent")
that runs **asynchronously** in its own isolated session. Every run is persisted to an
embedded SQLite `jobs` table, so delegation survives process restarts, is auditable, and
never leaks goroutines.

This document explains what a subagent is, how you trigger one, the guardrails that bound
it, and the runtime machinery (job queue, EventBus, workers) that executes it.

---

## 1. What is a subagent?

Three terms, used the same way in the Web UI, the tools and this document:

- An **Agent** is an AI employee of its own: its configuration, workspace, memory and
  channels.
- A **Subagent** belongs to one agent: a name, a description of when to use it,
  instructions, a narrowed set of tools and optionally its own model. It is a file in that
  agent's workspace, `agents/<name>.agent.md`, made on the Agents page or with the
  `subagents` tool. It runs *inside* its agent, under that agent's permissions.
- A **run** is one task handed either to one of the agent's own subagents or to another
  agent. Every surface names a run the same way: `Requester → Target (Type)`, for example
  `Support → code-reviewer (Subagent)` or `Support → Finance (Agent)`.

A **run** answers a single, well-scoped task. The child runs in a fresh session
(`agent:<parent>:subagent:<id>`), streams its answer, and reports back to the parent — all
without blocking the parent's own turn.

Use a subagent when the task is a distinct sub-problem that benefits from a different agent's
persona, model, or toolset (e.g. a general assistant delegating a code change to a coding
agent), or when you want fan-out/parallel work. Do **not** use one for a trivial single-tool
call the parent can do itself — a subagent spins up a full agent turn (LLM + tools) and has
real cost.

Key properties:

- **Isolated session** — the child gets its own conversation history; it does not see the
  parent's full context, only the `task` string it is handed.
- **Asynchronous** — with `wait` (the default) the tool waits up to 60 seconds and returns the
  answer; a run that takes longer, or one started with `wait: false`, returns a `run_id`. The
  parent polls, yields, or reads the result the next time it is invoked (see
  [§6](#6-completion-eventbus-yield-and-announcements)). The run goes on after the turn that
  started it ends, until it finishes, is stopped, or reaches its one-hour limit (see
  [§10](#10-limits--gotchas)).
- **RBAC-preserving** — the parent's identity (`GoClawRole`, `UnifiedUserID`, `Channel`) is
  forwarded into the child, so memory scoping and role-gated writes behave correctly inside
  the subagent. A subagent is not a privilege-escalation path.

---

## 2. How to trigger a subagent

There are two surfaces, both backed by the same `SpawnSubagent` entry point. The `sessions_spawn` and `sessions_send` tools start a run and send it a follow-up through the same functions (`SpawnSubagent`, `SendSubagentMessage`).

### 2.1 The `subagents` tool (LLM-facing)

A registered agent tool named **`subagents`** (`internal/agent/tool_subagents.go`) keeps
**registration** (which subagents an agent has) apart from **execution** (running tasks).
Registering never runs anything, and running never changes a registration.

| Action | Parameters | Effect |
|---|---|---|
| `create` | `name`, `description`, `instructions`, `tools`, `model`, `agent_id`, `draft` | Register a subagent. Admins only. Refuses a name that already exists; `draft: true` shows it without saving. |
| `update` | `name`, any of the above | Change a subagent. Admins only. |
| `delete` | `name`, `agent_id` | Remove a subagent. Admins only. |
| `list` | `agent_id` | List an agent's subagents. |
| `inspect` | `name`, `agent_id` | Show one subagent. Its instructions are shown to admins only. |
| `run` | `agent_id`, `task`, `target_type`, `wait` | Start a run (`spawn` still works). `agent_id` may be omitted → auto-routing to an agent (below). |
| `status` | `run_id` | Report one run's state, timing, and error. |
| `list_runs` | — | List this agent's active and recent runs. |
| `follow_up` | `run_id`, `message` | Send a further message to a run's session. |
| `yield` | `run_id` | Return the run's result if finished; otherwise report "still running". |
| `stop` | `run_id` | Cancel a queued/running run. |

For registration, `name` is the subagent and `agent_id` the agent it belongs to (default:
the agent of this conversation). Another agent's subagents can be listed, inspected or
changed by an administrator only. For `run`, `agent_id` is the target: one of the agent's
own subagents, or another agent's ID. If a subagent and a configured agent have the same
name, `target_type` (`"subagent"` or `"agent"`) must say which.

The execution actions only reach runs the calling agent started, and of those only the runs
the caller may touch: the person who started the run, a human admin, or the system acting for
no one (a cron or SOP run). A run that recorded no person belongs to the session that started
it. Any other run answers "not found", so its task and result stay with it. A follow-up acts
with the run's delegation: a caller acting for another person, or whose own grant does not
cover the run's, is refused.

Example (as the model would call it):

```json
{ "action": "run", "agent_id": "code-reviewer", "task": "Review merge request !42 for bugs" }
```

With `wait` (the default) the tool waits up to 60 seconds and returns the answer; a run that
takes longer keeps going while the turn that started it runs (see
[§10](#10-limits--gotchas)). `action` is validated at runtime (not by the
JSON schema), so an unknown action returns an error listing the valid ones.

If `agent_id` is omitted on `run`, GoClaw attempts **embedding-based auto-routing**
(`RouteToAgent`) to pick the best-matching agent for the task; an explicit `agent_id` always
wins. If no confident match is found, the tool asks the caller to pick one via `agents_list`
or `suggest_agent`.

### 2.2 The `/subagents` slash command (human/channel-facing)

The same operations are available as a chat command, parsed by `HandleSubagentCommand`
(`internal/agent/subagent_commands.go`) and wired into every message entry point — chat
channels, the WebSocket UI, the REST API, and the OpenAI-compatible endpoint:

```
/subagents spawn <agent_id> <task>
/subagents list
/subagents status <run_id>
/subagents stop <run_id>
```

---

## 3. Spawn policy — the guardrails

Before any run starts, `validateSubagentPolicy` enforces per-parent limits declared in the
parent agent's config, plus a system-wide safety cap. This is what prevents runaway
"explosive chaining" loops.

| Guardrail | Config key (per agent) | Default | Meaning |
|---|---|---|---|
| **Allow-list** | `subagents.allow_agents` | *(empty = allow all)* | Which other configured agents this parent may spawn. `"*"` = any. If set and the target is not listed, the spawn is denied. It does not apply to the agent's own subagents (`agents/<name>.agent.md`). |
| **Max concurrency** | `subagents.max_active` | `5` | Max simultaneously *queued + running* children for this parent, its own subagents included. |
| **Max depth** | `subagents.max_depth` | `1` | How deep the spawn chain may nest. Depth is counted from the number of `:subagent:` segments in the parent session ID, so a child cannot recurse past the limit. |
| **System-wide cap** | *(constant)* | `10` | `maxConcurrentSubagentsV2` — a global semaphore bounding *all* concurrently executing subagents across every parent (prevents goroutine leaks). |

Two details worth knowing: `max_active` is counted **per parent agent** (all of that agent's
sessions share the one budget), and the whole policy block is only applied when the parent's
config is found — if `config.GetAgentConfig(parent)` returns nil, **all checks are skipped**.

A separate switch governs **agent-to-agent messaging** (the "send a follow-up to a run"
path, `follow_up` and `SendSubagentMessage`) when the run's target is **another agent**:
`tools.agent_to_agent.enabled` must be `true` (it defaults to **off**), and
`tools.agent_to_agent.allow` (if non-empty) must list the participating agents. A follow-up to
one of the agent's own subagents does not need it. A follow-up runs the subagent with the same
instructions, tools and model as its first run, and fails if the subagent has been deleted
since. This gate is about *messaging* an existing run, not starting one.

> **Note:** these limits are what stands between a prompt-injected message and a spawn
> storm. Registering, changing and deleting subagents needs an administrator, but starting a
> run does not, so `allow_agents` is the real boundary for runs on other agents — keep it
> tight for agents exposed to untrusted channels. The agent's own subagents are bounded by
> `max_active` and by their narrowed tools.

---

## 4. Execution model

Runs are `SubagentManagerV2`'s (`subagent_v2.go`): each run is a row in the SQLite `jobs`
table, so it survives a restart and can be audited, its end is published on the EventBus and
announced to the parent, and it has its own trace.

`SpawnSubagent()` — the function both the tool and the slash command call — delegates to
`SpawnSubagentV2` and converts the resulting `Job` back into a `SubagentRun` for callers;
`GetSubagentRun`, `ListSubagentRuns` and `StopSubagentRun` read and stop jobs the same way.
`SubagentManager` (`subagent.go`) is what remains of the first, in-memory manager: it holds the
agent registry that a follow-up (`Send`, behind `follow_up` and `sessions_send`) runs on, and
finds the run to follow up in the `jobs` table.

---

## 5. Lifecycle & state machine

A subagent run moves through these states (`SubagentStatus`):

```mermaid
stateDiagram-v2
    direction LR
    [*] --> queued : spawn
    queued --> running : acquire slot
    queued --> failed : stop / cancel
    running --> done : finish
    running --> failed : finish (error) / stop / cancel
```

A stopped or cancelled run ends `failed`, usually with the error `killed` (a stop records
`killed by user` first; the run's own exit then overwrites it), or `cancelled waiting for
execution slot` when it had not started. A run that reaches its one-hour limit (queued time
included) ends the same way. The turn that started a run ending does not stop it. There is no
separate `killed` state.

Under the hood, `SpawnSubagentV2` performs these steps:

1. **Validate** the spawn against the policy in [§3](#3-spawn-policy--the-guardrails).
2. **Insert** a `Job` row (`job_type="subagent"`, `state="queued"`) with the parent/child
   context serialized into `context_json`.
3. **Launch** a bounded goroutine. It acquires a slot from the global semaphore
   (cap `10`); if the context is cancelled while waiting, the job is failed with
   *"cancelled waiting for execution slot"*.
4. **Run** — flips the job to `running`, forwards the parent's RBAC identity, and calls the
   child agent's `ProcessStream`, draining it to a final result (`DrainToResult`).
5. **Persist** the task + answer into the child's session history.
6. **Finish** — marks the job `done` (or `failed` on error or cancel), publishes an
   EventBus event, and, when it is `done`, **announces to the parent** (see next section).

> The `jobs` row for a subagent is a **ledger** (state, result, audit, recovery) — not a
> dispatch queue. Execution is driven directly by the spawning goroutine, not by a background
> poller. (Contrast this with SOP steps and memory jobs, [§7](#7-who-actually-runs-the-jobs).)

---

## 6. Completion: EventBus, yield, and announcements

When a run finishes, the parent finds out through either of two mechanisms:

- **Session injection** — a run that is `done` has its answer appended to the parent's session as
  `"[Support → code-reviewer (Subagent) done (Run: <id>)] …"`, so the parent sees it in
  history on its next turn. It waits for the parent's turn in progress to end (at most 15
  minutes) so neither write drops the other. A failed run is not announced; nothing is posted
  to a channel.
- **EventBus** — the run publishes to the in-process `EventBus` (`internal/agent/eventbus.go`)
  on topic `job.done` or `job.failed`.

The `yield` action (and `YieldSubagentV2`) uses the EventBus to wait efficiently: it
subscribes to `job.done`/`job.failed`, and — because publishing is non-blocking and *drops*
events to a full subscriber channel — also polls the DB every **3 seconds** as a safety net,
plus an overall timeout. This makes waiting correct even under bursty, high-concurrency
fan-out.

---

## 7. Who actually runs the jobs?

The `jobs` table is shared by several kinds of async work, but they are executed by **three
distinct mechanisms**. Keeping them straight is important — they have different concurrency
limits.

| Executor | Handles | Concurrency | Dispatch |
|---|---|---|---|
| **Subagent manager** (`subagent_v2.go`) | `subagent` jobs | Semaphore, cap **10** | In-process goroutine per spawn (no poller) |
| **MemoryWorker** (`memory_worker.go`) | `fact_extract`, `memory_consolidate`, `graph_extract`, `graph_sync`, `belief_revision` (an `auto_learn_error` row left by an older version is drained as a no-op) | **Sequential** (one at a time) | Polls `queued` jobs every **5 s** |
| **Tool WorkerPool** (`worker_pool.go`) | Tool calls (not jobs at all) | `tools.worker_pool_size` (**8**), queue `tools.worker_queue_size` (**64**) | Buffered channel with back-pressure |

Two clarifications that the older version of this doc got wrong:

- **The `8`-worker / `64`-queue pool is the *tool* execution pool**, not the subagent pool.
  Subagent concurrency is the semaphore of `10` plus the per-parent `max_active` limit.
- **Memory background work is deliberately serialized** by the `MemoryWorker` (one job at a
  time) to respect LLM rate limits and to avoid SQLite write contention on the `memories`
  table.

SOP steps also live in the `jobs` table (`job_type="sop_step"`) and use the extra
`suspended` state for human-in-the-loop approval, resuming via the EventBus and the
`parent_job_id` link. SOPs are covered in [Standard Operating Procedures](../user-guide/automation/sop.md).

### The `jobs` table

| Column | Purpose |
|---|---|
| `id` | Unique job ID (`job-<unixnano>-<seq>`). |
| `parent_job_id` | Links a child job to its parent (used by SOP resume). |
| `job_type` | `subagent` \| `sop_step` \| `fact_extract` \| `memory_consolidate` \| `graph_extract` \| `graph_sync` \| `belief_revision`; `auto_learn_error` only in rows left by an older version. |
| `agent_id` | The agent that executes the job. |
| `task` | The work description handed to the agent. |
| `state` | `queued` \| `running` \| `suspended` \| `done` \| `failed`. |
| `context_json` | Serialized context for resume (parent/child IDs, etc.). |
| `result_json` | Output captured on completion. |
| `error` | Failure reason, if any. |
| `created_at` / `updated_at` | Timestamps. |

---

## 8. Reliability & crash recovery

Because state lives in SQLite, a restart does not lose or double-run work:

- On startup, `InitSubagentV2` calls `RecoverPendingJobs` → `FailStaleJobs`, which marks any
  job left in `running` (interrupted mid-execution by the crash) as `failed` with a clear
  reason. Jobs still `queued` are left intact.
- Panics inside a subagent goroutine are recovered and turn into a `failed` job rather than
  crashing the process.
- Each subagent run records its own trace, so child execution is observable separately from
  the parent in the dashboard.

---

## 9. Configuration example

Enable subagents on a parent agent and allow it to delegate to two workers:

```yaml
agents:
  default: main
  list:
    - id: main
      subagents:
        allow_agents: [coding, research]  # who "main" may spawn ("*" = any)
        max_active: 5                      # concurrent children (default 5)
        max_depth: 1                       # nesting depth (default 1)
    - id: coding
    - id: research

tools:
  # Required to send a follow-up to a run on another agent (not to the agent's own subagents).
  agent_to_agent:
    enabled: true
    allow: [main, coding, research]

  # Tool-execution pool (separate from subagent concurrency).
  worker_pool_size: 8
  worker_queue_size: 64
```

---

## 10. Limits & gotchas

- **Depth is capped** (`max_depth`, default `1`): by default another agent running a task
  cannot itself spawn a run. Raise it deliberately — deep chains multiply cost. An agent's own
  subagents (`.agent.md`) never nest: they are never offered `subagents`, `sessions_spawn` or
  `sessions_send`, whatever their profile lists. The policy is checked only for agents listed
  in `agents.list`.
- **Global cap of 10**: even with generous per-agent `max_active`, no more than 10 subagents
  run at once process-wide.
- **A run may outlast the wait**: `run` waits up to 60 seconds (or not at all with
  `wait: false`) and then returns the `run_id`. Use `status`/`yield` or wait for the
  announcement — do not assume the result is ready synchronously.
- **A run outlasts its turn, up to an hour**: it keeps who it acts for and the scope of the
  turn that started it, but not that turn's cancellation, so a chat channel answering does not
  stop it. A run still going after an hour (queued time included) is cancelled and ends
  `failed` (`killed`).
- **A subagent is a full agent turn**: it consumes tokens and can call tools. Delegate whole
  sub-problems, not one-liners.
- **`allow_agents` is a security boundary**, not just organization: it is the guard against a
  prompt-injected channel message spawning arbitrary agents. Keep it tight on exposed agents.
- **Queued-at-crash jobs are orphaned**: startup recovery fails `running` jobs, but a subagent
  left `queued` at crash time is never re-dispatched (there is no queue poller for subagents).
  In practice the `queued` window is momentary, so this is rare.