# Fleet Toolkit (Remote Management via SSH) Whether you run GoClaw on **a single remote machine** (a Mac Mini, Linux VPS, or Raspberry Pi) or across **dozens of hosts**, the Fleet Toolkit lets you inspect, back up, audit, and upgrade them from your own laptop over SSH. Even for a single host, the toolkit gives you one-command workflows that go beyond the on-host CLI: - **`make macos-check`** — Checks whether a headless Mac Mini will sleep, stop at the FileVault/login screen, or fail to auto-start after a power cut, and prints the exact `pmset` command to fix it. - **`make upgrade`** — Prints the plan for every host and the exact command that carries it out. With `CONFIRM=yes` it checks that no scheduled job or active turn is running, downloads a `.gclaw` backup to your laptop first, runs `goclaw update`, verifies the new process answers `/health`, and **automatically rolls back** to the previous binary if the new gateway fails to come up. - **`make report`** — Downloads a database snapshot and recent logs to your laptop and generates a 7-day Markdown health report (`reports/report_.md`) covering request latency, tool failures, SOP runs, and log errors without exposing user chat text. - **`make teams_package`** — Reads your remote agent's Microsoft Teams settings and builds the ready-to-upload Teams app `.zip` on your computer. It is a set of Ansible playbooks behind `make` commands. It reaches each host over SSH with your key and installs nothing on the host: the playbooks run the host's own `goclaw` binary and read the files it already wrote. macOS and Linux hosts can be mixed in one list. ## 1. Prerequisites ### A. On your computer (Operator machine) Choose the instructions for the operating system you work from: #### macOS 1. Install Ansible (`make` and `python3` are included with Xcode Command Line Tools or Homebrew): ```bash brew install ansible ``` 2. Generate an SSH key (if you do not have one) and copy it to the GoClaw host: ```bash ssh-keygen -t ed25519 ssh-copy-id ops@192.0.2.10 ``` #### Linux (Ubuntu / Debian / RHEL / Fedora) 1. Install `ansible`, `make`, `python3`, and `curl`: ```bash # Ubuntu / Debian sudo apt update && sudo apt install -y ansible make python3 curl # RHEL / Fedora / Rocky sudo dnf install -y ansible-core make python3 curl ``` 2. Generate an SSH key and copy it to the GoClaw host: ```bash ssh-keygen -t ed25519 ssh-copy-id ops@192.0.2.10 ``` #### Windows (via WSL2) Ansible requires a POSIX environment and does not run directly in PowerShell or CMD. Run the toolkit inside **WSL2 (Ubuntu)**: 1. Open PowerShell as Administrator and install WSL (if not already installed): ```powershell wsl --install -d Ubuntu ``` 2. Open the **Ubuntu** terminal from the Start menu and install the prerequisites: ```bash sudo apt update && sudo apt install -y ansible make python3 curl ``` 3. Inside the Ubuntu terminal, generate an SSH key and copy it to the GoClaw host: ```bash ssh-keygen -t ed25519 ssh-copy-id ops@192.0.2.10 ``` > [!IMPORTANT] > The toolkit connects with `BatchMode=yes` and never stops to ask for a password. Always test `ssh ops@192.0.2.10` once from your terminal to confirm it logs in without a password prompt before running `make ping`. --- ### B. On each Agent Host (Target machine running GoClaw) The host only needs an existing GoClaw installation, `python3`, `curl`, and SSH enabled for the account that runs GoClaw. #### macOS Agent Host (e.g., Mac Mini) 1. **Enable SSH (Remote Login):** - Open **System Settings → General → Sharing** and turn on **Remote Login** (or run `sudo systemsetup -setremotelogin on` in Terminal). 2. **Ensure `python3` is installed:** - Run `python3 --version` on the Mac. If macOS prompts to install the Command Line Developer Tools, accept it (`xcode-select --install`), or install via Homebrew (`brew install python`). 3. **Prevent sleep and login-screen hangs on a headless Mac:** - Disable sleep and enable automatic restart after power failure: ```bash sudo pmset -a sleep 0 autorestart 1 womp 1 ``` - If GoClaw runs as a user LaunchAgent (`~/Library/LaunchAgents/goclaw.plist`), turn **FileVault OFF** and enable **Automatic Login** (**System Settings → Users & Groups → Automatically log in as…**) so the Mac does not wait at the pre-boot unlock screen after a power cut. Run `make macos-check` from your computer to verify all of these settings. #### Linux Agent Host (e.g., Ubuntu/Debian VPS, Raspberry Pi, RHEL) 1. **Install SSH server, Python 3, and curl:** ```bash # Ubuntu / Debian / Raspberry Pi OS sudo apt update && sudo apt install -y openssh-server python3 curl sudo systemctl enable --now ssh # RHEL / Fedora / Rocky sudo dnf install -y openssh-server python3 curl sudo systemctl enable --now sshd ``` 2. **Service mode (`systemd --user` vs `--system`):** - **Per-user service (`goclaw service install`):** Enable user lingering on the Linux host so the agent starts at boot and stays running after you log out of SSH: ```bash sudo loginctl enable-linger $USER ``` - **System-wide service (`sudo goclaw service install --system`):** Read commands (`make status`, `make fetch_logs`, `make fetch_db`, `make report`) work without root. To run `make upgrade` or `make restart` remotely against a system service, add `SUDO=yes` (which requires passwordless `sudo` for the SSH user), or run `sudo goclaw update` directly on the host. 3. **Binary in `/usr/local/bin`:** `install.sh` puts the binary there, owned by root. Whatever the service mode, replacing it needs root, so `make upgrade` needs `SUDO=yes` and passwordless `sudo` for the SSH user. A binary in `~/bin` or `~/.local/bin` is replaced as the SSH user. --- ## 2. Download Each release publishes the toolkit as `goclaw-fleet-.tar.gz`: ```bash v=$(curl -fsSL https://goclaw.ulab.mn/releases/latest.json | grep -o '"version"[[:space:]]*:[[:space:]]*"[^"]*"' | cut -d'"' -f4) curl -fsSL "https://goclaw.ulab.mn/releases/$v/goclaw-fleet-$v.tar.gz" | tar -xz cd goclaw-fleet ``` *(Optional)* Verify the archive against `checksums.json` before extracting: ```bash curl -fsSLO "https://goclaw.ulab.mn/releases/$v/goclaw-fleet-$v.tar.gz" curl -fsSL "https://goclaw.ulab.mn/releases/$v/checksums.json" \ | python3 -c "import json,sys,hashlib; want=json.load(sys.stdin)['goclaw-fleet-$v.tar.gz']; got=hashlib.sha256(open('goclaw-fleet-$v.tar.gz','rb').read()).hexdigest(); print('checksum ok' if got==want else 'CHECKSUM MISMATCH')" tar -xzf "goclaw-fleet-$v.tar.gz" && cd goclaw-fleet ``` ## 3. Set up `hosts.ini` ```bash cp inventory/hosts.ini.example inventory/hosts.ini $EDITOR inventory/hosts.ini make ping make status ``` `inventory/hosts.ini` has one line per host (even if you only have one): ```ini [goclaw_agents] agent-ops ansible_host=192.0.2.10 ansible_user=ops agent-finance ansible_host=192.0.2.11 ansible_user=finance ``` If your host uses a non-default SSH port or a specific key file, add `ansible_port=2222` or `ansible_ssh_private_key_file=~/.ssh/id_ed25519` on the same line. The toolkit automatically detects on each host where the `goclaw` binary, its state directory, and its logs are, which port it answers on, and how its service runs (`launchd` or `systemd`, per-user or system-wide). The state directory is read from the service unit or plist, defaulting to `~/.goclaw`. Every `goclaw` command the toolkit runs on the host uses that state directory. ## Read commands These change nothing on the agents. | Command | What it does | |---|---| | `make status` | Shows the version and process, whether the gateway answers, and the service state. | | `make version` | Shows the version on every host. | | `make doctor` | Runs the agent's own `goclaw doctor`, which also says whether its service starts at boot. | | `make quiet_check` | Checks the agent is idle: no scheduled job running, and its log still for two minutes. | | `make get-config` | Shows `config.yaml` with credential values masked on the agent before anything is sent. | | `make macos-check` | Reports a macOS agent's sleep, firewall and login settings. | | `make logs` | Shows the end of the gateway log and the service log (launchd file or systemd journal). | | `make fetch_logs` | Downloads the gateway log to `fetched//gateway.log`: the last hour, unless `ARGS` says otherwise (below). | | `make fetch_db` | Downloads a `.gclaw` database snapshot to `fetched//`. | | `make fetch_md` | Downloads the workspace's Markdown files (`Memory.md`, `AGENTS.md` and the other bootstrap files) to `fetched//`. | | `make backup` | Saves a snapshot with SOP runs to `backups//`. | | `make memory_stats` | Shows memory counts and index health; it does not reindex. | | `make report` | Downloads snapshots and logs, then writes `reports/report_.md` (`report__.md` with `LIMIT`). | | `make teams_package` | Reads an agent's Teams settings and builds the Teams app `.zip` on your computer. | Every command takes `LIMIT=` and `ARGS="-e key=value"`. `make help` lists every command, and the `Makefile` documents each one's options above its target. The ones you will reach for most: ```bash make fetch_logs LIMIT=agent-ops ARGS="-e hours=3" make fetch_logs LIMIT=agent-ops ARGS="-e since=2026-06-01T09:00" # the agent's local time, not UTC make fetch_logs LIMIT=agent-ops ARGS="-e commit=" # since that build was deployed make fetch_db LIMIT=agent-ops ARGS="-e fetch_db_sessions=true -e fetch_db_usage=true" make backup LIMIT=agent-ops ARGS="-e backup_sessions=true" ``` `fetch_logs` takes the first of `commit`, `since` and `hours` that is given. `fetch_db` adds tables with `fetch_db_sessions`, `fetch_db_chunks`, `fetch_db_executions`, `fetch_db_usage`, `fetch_db_files` and `fetch_db_attachments`, and `fetch_db_agent_id=` keeps one agent's rows; `backup` takes `backup_sessions`, `backup_chunks`, `backup_usage`, `backup_files` and `backup_attachments`. ## The report `make report` (the last 7 days by default; `DAYS=1` for one) opens with a table across the fleet. Each row gives a host's version, its requests, the share that failed, tool failures, SOP runs that did not complete, failing scheduled jobs, and the panics and errors in its log. Then, for each host: - its requests by channel and status, how long they took, and the tokens used; - the slowest requests and the failed or interrupted ones; - each tool's calls and how often they failed; - SOP runs per SOP: completed, partial, failed, interrupted; - the starts that failed: many of them usually mean a service restarting while a second gateway, started by hand, holds its port; - the most frequent error and warning messages in the log. Requests are named by their trace id, so the agent's **Traces** page finds them. The report never shows what a person wrote. With `LIMIT` (hosts or a group from `hosts.ini`) the report covers those hosts and gets their name, so the day's report across the fleet is kept. ## Upgrade and restart `make upgrade` and `make restart` change the agents. Without `CONFIRM=yes` they only print the plan, and the last line they print is `PLAN ONLY: nothing was upgraded on any host` (or restarted): ```bash make upgrade # the plan: each host's version → target, and what is in the way make upgrade CONFIRM=yes # the upgrade make upgrade CONFIRM=yes SUDO=yes # with sudo on the hosts that need root make upgrade LIMIT=agent-ops CONFIRM=yes make restart LIMIT=agent-ops CONFIRM=yes ``` The plan covers every host at once and ends with a summary: which hosts are ready, which need `SUDO=yes`, which cannot be upgraded from here, and the exact command that upgrades the rest. Run that command. A host needs root when its binary is in a directory the SSH account cannot write (`/usr/local/bin`, where `install.sh` puts it) or its service is system-wide (`goclaw service install --system`). The plan checks whether the account has passwordless `sudo` there. If it has, `SUDO=yes` upgrades the host; if not, run `sudo goclaw update` on that host yourself, since nothing here can answer a password prompt. `SUDO=yes` uses `sudo` only on the hosts that need root, so one command covers a fleet that mixes `~/bin` and `/usr/local/bin` installs, and a `systemd --user` service is still restarted as the SSH account. With `CONFIRM=yes`, while one host in the run needs `SUDO=yes`, has no passwordless `sudo` or could not be reached, no host is upgraded: each fails with its reason, and the summary gives the command for the others. `CONFIRM=true` and other values than `yes` are refused. An upgrade installs the latest release one host at a time; `ARGS="-e target=vX.Y.Z"` installs that version instead. For each host: 1. A host already on the target version is skipped. 2. A busy host (a scheduled job running, or its log written in the last two minutes) is refused unless you add `ARGS="-e force=true"`. 3. A snapshot is saved to `backups//…_before-.gclaw`. 4. `goclaw update` downloads the release, checks its checksum, swaps the binary and restarts the service. An agent that was not running is started, and the plan says so. 5. The playbook waits until a gateway started since step 4 answers `/health`, and checks the binary is the new version. An older gateway still answering means the service did not restart, and counts as a failure. If step 4 or 5 fails and the upgrade replaced the binary, the one that ran before is put back and the service restarted. If the upgrade did not replace it (the download failed), nothing is put back, so an older `goclaw.prev` never downgrades the agent. No further host is touched. A restart, too, refuses a busy host unless you add `ARGS="-e force=true"`, which cuts off what the agent is doing. A system-wide service needs `SUDO=yes`. ## Your data `fetched/`, `backups/` and `reports/` hold the agents' real data. Treat them as you would the agents' own disks. The toolkit's `.gitignore` keeps them, and `inventory/hosts.ini`, out of any repository you keep the directory in.