Fleet Toolkit (Remote Management via SSH)

Whether you run GoClaw on a single remote machine (a Mac Mini, Linux VPS, or Raspberry Pi) or across dozens of hosts, the Fleet Toolkit lets you inspect, back up, audit, and upgrade them from your own laptop over SSH.

Even for a single host, the toolkit gives you one-command workflows that go beyond the on-host CLI:

make macos-check — Checks whether a headless Mac Mini will sleep, stop at the FileVault/login screen, or fail to auto-start after a power cut, and prints the exact pmset command to fix it.
make upgrade — Prints the plan for every host and the exact command that carries it out. With CONFIRM=yes it checks that no scheduled job or active turn is running, downloads a .gclaw backup to your laptop first, runs goclaw update, verifies the new process answers /health, and automatically rolls back to the previous binary if the new gateway fails to come up.
make report — Downloads a database snapshot and recent logs to your laptop and generates a 7-day Markdown health report (reports/report_<date>.md) covering request latency, tool failures, SOP runs, and log errors without exposing user chat text.
make teams_package — Reads your remote agent's Microsoft Teams settings and builds the ready-to-upload Teams app .zip on your computer.

It is a set of Ansible playbooks behind make commands. It reaches each host over SSH with your key and installs nothing on the host: the playbooks run the host's own goclaw binary and read the files it already wrote. macOS and Linux hosts can be mixed in one list.

1. Prerequisites

A. On your computer (Operator machine)

Choose the instructions for the operating system you work from:

macOS

Install Ansible (make and python3 are included with Xcode Command Line Tools or Homebrew):
```
brew install ansible
```
Generate an SSH key (if you do not have one) and copy it to the GoClaw host:
```
ssh-keygen -t ed25519
ssh-copy-id ops@192.0.2.10
```

Linux (Ubuntu / Debian / RHEL / Fedora)

Install ansible, make, python3, and curl:

# Ubuntu / Debian
sudo apt update && sudo apt install -y ansible make python3 curl

# RHEL / Fedora / Rocky
sudo dnf install -y ansible-core make python3 curl

Generate an SSH key and copy it to the GoClaw host:
```
ssh-keygen -t ed25519
ssh-copy-id ops@192.0.2.10
```

Windows (via WSL2)

Ansible requires a POSIX environment and does not run directly in PowerShell or CMD. Run the toolkit inside WSL2 (Ubuntu):

Open PowerShell as Administrator and install WSL (if not already installed):
```
wsl --install -d Ubuntu
```
Open the Ubuntu terminal from the Start menu and install the prerequisites:
```
sudo apt update && sudo apt install -y ansible make python3 curl
```
Inside the Ubuntu terminal, generate an SSH key and copy it to the GoClaw host:
```
ssh-keygen -t ed25519
ssh-copy-id ops@192.0.2.10
```

[!IMPORTANT] The toolkit connects with BatchMode=yes and never stops to ask for a password. Always test ssh ops@192.0.2.10 once from your terminal to confirm it logs in without a password prompt before running make ping.

B. On each Agent Host (Target machine running GoClaw)

The host only needs an existing GoClaw installation, python3, curl, and SSH enabled for the account that runs GoClaw.

macOS Agent Host (e.g., Mac Mini)

Enable SSH (Remote Login):
- Open System Settings → General → Sharing and turn on Remote Login (or run sudo systemsetup -setremotelogin on in Terminal).
Ensure python3 is installed:
- Run python3 --version on the Mac. If macOS prompts to install the Command Line Developer Tools, accept it (xcode-select --install), or install via Homebrew (brew install python).
Prevent sleep and login-screen hangs on a headless Mac:
- Disable sleep and enable automatic restart after power failure:
```
sudo pmset -a sleep 0 autorestart 1 womp 1
```
- If GoClaw runs as a user LaunchAgent (~/Library/LaunchAgents/goclaw.plist), turn FileVault OFF and enable Automatic Login (System Settings → Users & Groups → Automatically log in as…) so the Mac does not wait at the pre-boot unlock screen after a power cut. Run make macos-check from your computer to verify all of these settings.

Linux Agent Host (e.g., Ubuntu/Debian VPS, Raspberry Pi, RHEL)

Install SSH server, Python 3, and curl:

# Ubuntu / Debian / Raspberry Pi OS
sudo apt update && sudo apt install -y openssh-server python3 curl
sudo systemctl enable --now ssh

# RHEL / Fedora / Rocky
sudo dnf install -y openssh-server python3 curl
sudo systemctl enable --now sshd

Service mode (systemd --user vs --system):
- Per-user service (goclaw service install): Enable user lingering on the Linux host so the agent starts at boot and stays running after you log out of SSH:
```
sudo loginctl enable-linger $USER
```
- System-wide service (sudo goclaw service install --system): Read commands (make status, make fetch_logs, make fetch_db, make report) work without root. To run make upgrade or make restart remotely against a system service, add SUDO=yes (which requires passwordless sudo for the SSH user), or run sudo goclaw update directly on the host.
Binary in /usr/local/bin: install.sh puts the binary there, owned by root. Whatever the service mode, replacing it needs root, so make upgrade needs SUDO=yes and passwordless sudo for the SSH user. A binary in ~/bin or ~/.local/bin is replaced as the SSH user.

2. Download

Each release publishes the toolkit as goclaw-fleet-<version>.tar.gz:

v=$(curl -fsSL https://goclaw.ulab.mn/releases/latest.json | grep -o '"version"[[:space:]]*:[[:space:]]*"[^"]*"' | cut -d'"' -f4)
curl -fsSL "https://goclaw.ulab.mn/releases/$v/goclaw-fleet-$v.tar.gz" | tar -xz
cd goclaw-fleet

(Optional) Verify the archive against checksums.json before extracting:

curl -fsSLO "https://goclaw.ulab.mn/releases/$v/goclaw-fleet-$v.tar.gz"
curl -fsSL "https://goclaw.ulab.mn/releases/$v/checksums.json" \
  | python3 -c "import json,sys,hashlib; want=json.load(sys.stdin)['goclaw-fleet-$v.tar.gz']; got=hashlib.sha256(open('goclaw-fleet-$v.tar.gz','rb').read()).hexdigest(); print('checksum ok' if got==want else 'CHECKSUM MISMATCH')"
tar -xzf "goclaw-fleet-$v.tar.gz" && cd goclaw-fleet

3. Set up `hosts.ini`

cp inventory/hosts.ini.example inventory/hosts.ini
$EDITOR inventory/hosts.ini
make ping
make status

inventory/hosts.ini has one line per host (even if you only have one):

[goclaw_agents]
agent-ops      ansible_host=192.0.2.10  ansible_user=ops
agent-finance  ansible_host=192.0.2.11  ansible_user=finance

If your host uses a non-default SSH port or a specific key file, add ansible_port=2222 or ansible_ssh_private_key_file=~/.ssh/id_ed25519 on the same line.

The toolkit automatically detects on each host where the goclaw binary, its state directory, and its logs are, which port it answers on, and how its service runs (launchd or systemd, per-user or system-wide). The state directory is read from the service unit or plist, defaulting to ~/.goclaw. Every goclaw command the toolkit runs on the host uses that state directory.

Read commands

These change nothing on the agents.

Command	What it does
`make status`	Shows the version and process, whether the gateway answers, and the service state.
`make version`	Shows the version on every host.
`make doctor`	Runs the agent's own `goclaw doctor`, which also says whether its service starts at boot.
`make quiet_check`	Checks the agent is idle: no scheduled job running, and its log still for two minutes.
`make get-config`	Shows `config.yaml` with credential values masked on the agent before anything is sent.
`make macos-check`	Reports a macOS agent's sleep, firewall and login settings.
`make logs`	Shows the end of the gateway log and the service log (launchd file or systemd journal).
`make fetch_logs`	Downloads the gateway log to `fetched/<host>/gateway.log`: the last hour, unless `ARGS` says otherwise (below).
`make fetch_db`	Downloads a `.gclaw` database snapshot to `fetched/<host>/`.
`make fetch_md`	Downloads the workspace's Markdown files (`Memory.md`, `AGENTS.md` and the other bootstrap files) to `fetched/<host>/`.
`make backup`	Saves a snapshot with SOP runs to `backups/<host>/`.
`make memory_stats`	Shows memory counts and index health; it does not reindex.
`make report`	Downloads snapshots and logs, then writes `reports/report_<date>.md` (`report_<date>_<LIMIT>.md` with `LIMIT`).
`make teams_package`	Reads an agent's Teams settings and builds the Teams app `.zip` on your computer.

Every command takes LIMIT=<host|group> and ARGS="-e key=value". make help lists every command, and the Makefile documents each one's options above its target. The ones you will reach for most:

make fetch_logs LIMIT=agent-ops ARGS="-e hours=3"
make fetch_logs LIMIT=agent-ops ARGS="-e since=2026-06-01T09:00"   # the agent's local time, not UTC
make fetch_logs LIMIT=agent-ops ARGS="-e commit=<sha>"             # since that build was deployed
make fetch_db LIMIT=agent-ops ARGS="-e fetch_db_sessions=true -e fetch_db_usage=true"
make backup LIMIT=agent-ops ARGS="-e backup_sessions=true"

fetch_logs takes the first of commit, since and hours that is given. fetch_db adds tables with fetch_db_sessions, fetch_db_chunks, fetch_db_executions, fetch_db_usage, fetch_db_files and fetch_db_attachments, and fetch_db_agent_id=<id> keeps one agent's rows; backup takes backup_sessions, backup_chunks, backup_usage, backup_files and backup_attachments.

The report

make report (the last 7 days by default; DAYS=1 for one) opens with a table across the fleet. Each row gives a host's version, its requests, the share that failed, tool failures, SOP runs that did not complete, failing scheduled jobs, and the panics and errors in its log. Then, for each host:

its requests by channel and status, how long they took, and the tokens used;
the slowest requests and the failed or interrupted ones;
each tool's calls and how often they failed;
SOP runs per SOP: completed, partial, failed, interrupted;
the starts that failed: many of them usually mean a service restarting while a second gateway, started by hand, holds its port;
the most frequent error and warning messages in the log.

Requests are named by their trace id, so the agent's Traces page finds them. The report never shows what a person wrote. With LIMIT (hosts or a group from hosts.ini) the report covers those hosts and gets their name, so the day's report across the fleet is kept.

Upgrade and restart

make upgrade and make restart change the agents. Without CONFIRM=yes they only print the plan, and the last line they print is PLAN ONLY: nothing was upgraded on any host (or restarted):

make upgrade                         # the plan: each host's version → target, and what is in the way
make upgrade CONFIRM=yes             # the upgrade
make upgrade CONFIRM=yes SUDO=yes    # with sudo on the hosts that need root
make upgrade LIMIT=agent-ops CONFIRM=yes
make restart LIMIT=agent-ops CONFIRM=yes

The plan covers every host at once and ends with a summary: which hosts are ready, which need SUDO=yes, which cannot be upgraded from here, and the exact command that upgrades the rest. Run that command.

A host needs root when its binary is in a directory the SSH account cannot write (/usr/local/bin, where install.sh puts it) or its service is system-wide (goclaw service install --system). The plan checks whether the account has passwordless sudo there. If it has, SUDO=yes upgrades the host; if not, run sudo goclaw update on that host yourself, since nothing here can answer a password prompt. SUDO=yes uses sudo only on the hosts that need root, so one command covers a fleet that mixes ~/bin and /usr/local/bin installs, and a systemd --user service is still restarted as the SSH account.

With CONFIRM=yes, while one host in the run needs SUDO=yes, has no passwordless sudo or could not be reached, no host is upgraded: each fails with its reason, and the summary gives the command for the others. CONFIRM=true and other values than yes are refused.

An upgrade installs the latest release one host at a time; ARGS="-e target=vX.Y.Z" installs that version instead. For each host:

A host already on the target version is skipped.
A busy host (a scheduled job running, or its log written in the last two minutes) is refused unless you add ARGS="-e force=true".
A snapshot is saved to backups/<host>/…_before-<version>.gclaw.
goclaw update downloads the release, checks its checksum, swaps the binary and restarts the service. An agent that was not running is started, and the plan says so.
The playbook waits until a gateway started since step 4 answers /health, and checks the binary is the new version. An older gateway still answering means the service did not restart, and counts as a failure.

If step 4 or 5 fails and the upgrade replaced the binary, the one that ran before is put back and the service restarted. If the upgrade did not replace it (the download failed), nothing is put back, so an older goclaw.prev never downgrades the agent. No further host is touched.

A restart, too, refuses a busy host unless you add ARGS="-e force=true", which cuts off what the agent is doing. A system-wide service needs SUDO=yes.

Your data

fetched/, backups/ and reports/ hold the agents' real data. Treat them as you would the agents' own disks. The toolkit's .gitignore keeps them, and inventory/hosts.ini, out of any repository you keep the directory in.