# Secrets Vault & Configuration Interpolation

GoClaw features a secure, built-in **Secrets Vault** and dynamic **Configuration Interpolation** engine. This system ensures that API keys, webhook credentials, database passwords, and other sensitive parameters are encrypted at rest and never stored in plain-text inside configuration files.

---

## 1. Secrets Vault Architecture

The Secrets Vault is designed to provide secure credentials management on local filesystems:

```mermaid
flowchart TD
    CLI["<b>goclaw CLI / API</b>"] --> ENG["<b>AES-256-GCM Engine</b><br/>• PBKDF2 Key Derivation (600K iterations)<br/>• Salt: random 32 bytes per vault"]
    ENG --> FILE["<b>~/.goclaw/credentials/secrets.enc</b><br/>(File permissions: 0600)"]
```

- **Encryption at Rest:** Secrets are stored in a single encrypted file, `credentials/secrets.enc` in the state directory (`~/.goclaw`, or `GOCLAW_STATE_DIR` when set), using **AES-256-GCM** encryption.
- **Key Derivation:** The encryption key is derived from a passphrase using **PBKDF2** (SHA-256) with **600,000 iterations** and a random 32-byte salt generated for each vault and stored in the file's header. A vault written by an older version with the fixed legacy salt is still read, and is rewritten in the new format on its next save.
- **Passphrase:** The passphrase is the `GOCLAW_SECRET_KEY` environment variable when it is set. Otherwise it is derived (SHA-256) from `goclaw-secrets`, the OS user name and a machine identifier (`/etc/machine-id` on Linux, `/var/db/SystemKey` on macOS, `COMPUTERNAME` on Windows, else the home directory), and the gateway logs a warning at startup. Those identifiers are readable by anyone with access to the host, so the derived passphrase protects little against a local attacker. It also ties the vault to one user on one machine: a vault copied to another host, or opened by another user (for example through `sudo`), fails with `decrypt failed (wrong passphrase?)`. In production, set `GOCLAW_SECRET_KEY` and keep it outside the state directory (an OS keychain, or a `0600` file the service reads).
- **File Permissions:** The `credentials/` directory is created `0700` and `secrets.enc` is written `0600` (read/write by the owner only), so other users on the host cannot read the file.
- **Masked Listing:** When listing credentials via the CLI or admin dashboard, values are masked to their first and last four characters (e.g., `AIza****xAPw`); a value of 8 characters or fewer is shown as `****`. `goclaw secret list --show-values` and `goclaw secret get` print the plaintext.

---

## 2. Configuration Interpolation

GoClaw supports configuration interpolation inside `config.yaml`. This allows you to reference environment variables, system files, or vault secrets from the fields that hold credentials.

### Supported Reference Patterns

| Pattern Syntax | Resolution Source | Example |
|---|---|---|
| `${secret:KEY}` | Looks up `KEY` inside the encrypted Secrets Vault. | `api_key: "${secret:GEMINI_API_KEY}"` |
| `${env:VAR}` | Fetches the value of the environment variable `VAR`. | `api_key: "${env:OPENAI_API_KEY}"` |
| `${file:/path}` | Reads the first line of the specified file, trimming whitespace. | `token: "${file:/etc/secrets/token}"` |
| `${VAR}` (Bare) | Fallback mode: tries environment first, then queries the Secrets Vault. | `token: "${DISCORD_TOKEN}"` |

### Which Fields Are Expanded

References are expanded only in fields that hold credentials. In any other
field (a port, a URL, a model name, a channel ID) a `${...}` reference stays
literal text. The expanded fields are:

- **AI providers:** `ai.openai.api_key`, `ai.gemini.api_key`, `ai.anthropic.api_key`, `ai.deepseek.api_key`.
- **Gateway:** `gateway.auth.token`, `gateway.ingress.authtoken`, `gateway.ngrok.authtoken`.
- **Databases:** `database.dsn`; `memory.graph.dsn` and `memory.graph.password`. The gateway opens the vault before the state database, so its DSN may name a vault secret.
- **Channels:** `channels.discord.token`; `channels.msteams.app_id`, `app_password`, `tenant_id`, `ngrok.authtoken`; `channels.facebook.page_token`, `app_secret`, `verify_token`; `channels.telegram.token`, `webhook_secret`; `channels.app_api.api_key`, `jwt_secret` and the `key` of each `api_keys` entry.
- **Datasources:** `password` and `dsn` of each entry under `datasources`.
- **Tools:** `tools.web_search.brave_api_key`, `serp_api_key`; the `token` of `tools.gitlab`, `tools.sonarqube` and each instance under `tools.gitlabs` / `tools.sonarqubes`; the credentials of each instance under `tools.observability` (Grafana `token`/`password`, Graylog `token`, Uptrace `token`/`dsn`, Zabbix `token`/`password`, Wazuh `password`, NetBox `token`); and the same GitLab, SonarQube and observability fields in an agent's own `tools` overrides.
- **MCP servers:** every value under `env` and `headers` of each `mcp.servers` entry.
- **Logging:** `logging.otel.dsn` and every value under `logging.otel.headers`.
- **Voice:** `voice.elevenlabs.api_key`; `voice.chimege.tts_token`, `stt_short_token`, `stt_long_token`.
- **Email:** `email.smtp.password`, `email.imap.password`.
- **Hooks:** the `url` and `secret` of each webhook hook under `hooks`.

A webhook route's `secret`, `token` and `agent`, whether set on the Webhooks page or under
`webhooks:` in `config.yaml`, may also be a `${secret:NAME}` reference; it is expanded on each
delivery.

### Configuration Example

Below is an example of a secure `config.yaml` utilizing all interpolation types:

```yaml
server:
  port: "8080"            # not a credential field: written as a plain value

gateway:
  bind: "loopback"
  auth:
    token: "${secret:GATEWAY_ACCESS_TOKEN}"

ai:
  provider: "gemini"
  gemini:
    api_key: "${secret:GEMINI_API_KEY}"
  openai:
    api_key: "${env:OPENAI_API_KEY}"
  anthropic:
    api_key: "${file:/run/secrets/anthropic_key}"

channels:
  discord:
    token: "${DISCORD_BOT_TOKEN}"
```

---

## 3. Managing Secrets via CLI

You can easily manage your vault credentials from the terminal. The CLI supports category tags and provider association to organize tokens.

### Common CLI Operations

- **Generate a Secret:**
  ```bash
  goclaw secret gen <KEY> --provider <PROVIDER>
  ```
  Produces a cryptographically random value, stores it, and prints it once so
  it can be handed to whatever will present it. Use this for the shared secrets
  GoClaw asks for but does not issue — `channels.app_api.api_key` and its
  `api_keys` entries, `jwt_secret`, webhook tokens — rather than inventing a
  value by hand.
  *Example:*
  ```bash
  goclaw secret gen APP_KEY_ERP --provider app_api
  ```
  Flags: `--length` (bytes, default 32, minimum 16), `--format hex|base64`,
  `--category`, `--provider`, and `--force` to replace an existing name.
  Replacing revokes whatever still presents the old value, so it is not the
  default.

- **Add or Update a Secret:**
  ```bash
  goclaw secret set <KEY> <VALUE> --category <CATEGORY> --provider <PROVIDER>
  ```
  *Example:*
  ```bash
  goclaw secret set GEMINI_API_KEY AIzaSyBEt... --category api_key --provider gemini
  ```

- **Retrieve a Decrypted Secret:**
  ```bash
  goclaw secret get GEMINI_API_KEY
  ```
  Prints the plaintext. The CLI has no role check of its own: anyone who can
  run `goclaw` as the gateway's OS user (the derived passphrase is the same),
  or who has `GOCLAW_SECRET_KEY`, can read every value.

- **List All Secrets (Masked):**
  ```bash
  goclaw secret list
  ```
  `--show-values` prints the values unmasked.

- **Delete a Secret:**
  ```bash
  goclaw secret delete GEMINI_API_KEY
  ```

### Agent Tools

The agent has four vault tools: `secret_set`, `secret_get`, `secret_list` and
`secret_delete`. Each one is refused unless a human admin is the caller; a
non-admin user, and SOP, cron and webhook runs, get an access-denied error.
`secret_list` masks values as the CLI does. `secret_get` returns the plaintext
value to the model, so it then appears in the conversation and its trace and
is sent to the LLM provider. Tools that need a credential do not need it: they
get it from `${secret:NAME}` expansion in their configuration.

The model can also name a vault entry without seeing it: a header it writes for
`http_request`, or for an `mcp` call, may hold `${secret:NAME}` (or `${env:VAR}`,
`${file:/path}`), which is expanded as the request is sent. Only an admin, or a
SOP or cron run an admin set up, may do this; for anyone else, a webhook-started
run included, the call is refused. A credential a regular user's requests need
belongs in the integration's own configuration instead.

---

## 4. Reload behaviour

The vault file is shared with the `goclaw secret` CLI, which writes it from a
separate process. The gateway notices:

- **The secrets API re-reads the vault on every request**, so a secret written
  from a terminal shows up in the Web UI immediately.
- **`${secret:NAME}` in `config.yaml` is re-expanded whenever the config is
  loaded** — at startup and on every hot reload. Storing a secret does not by
  itself re-resolve a reference that was already expanded, so after adding one
  that the config points at, touch `config.yaml` (or restart) to put it into
  effect.

The config watcher accepts timestamp-only changes (`touch`) and continues
watching when an editor saves by replacing the file. Reloads run in order.
If the vault changes again during a read, that newer version remains detectable
on the next reload.

A vault that cannot be decrypted — re-keyed elsewhere, or caught mid-write —
leaves the already-loaded credentials in place and logs a warning rather than
emptying a running gateway.

---

## 5. Troubleshooting

- **A secret written while the gateway is running does not appear.** The secrets
  page re-reads the vault per request, so this should not happen; if it does, the
  gateway logged a reload failure (a re-keyed vault, or one written under a
  different passphrase — see section 1). A `${secret:NAME}` that the config
  already expanded is a different case and is expected: it refreshes on the next
  config load, so touch `config.yaml`.

- **Unresolved Value Warns:** If an interpolation reference (e.g., `${secret:MISSING_KEY}`) cannot be resolved, GoClaw logs a warning (once per reference) and leaves the reference in the field as literal text. Credential checks treat such a value as missing, never as the token: an unresolved `gateway.auth.token` makes the gateway answer every request that needs the token with HTTP 500 (`gateway.auth.token could not be resolved`), while the health probes, the Web UI's own files and the app API still answer, an unresolved `channels.app_api` key or JWT secret refuses every app API caller, and an unresolved webhook route secret refuses every delivery to that route.
- **`decrypt failed (wrong passphrase?)`:** The vault was written under a different passphrase: another machine, another OS user, or a different `GOCLAW_SECRET_KEY` (see section 1). Run the gateway as the user that created the vault, or with the `GOCLAW_SECRET_KEY` it was created with.
- **Permission Denied Errors:** If GoClaw fails to read the secrets database, ensure the running user has ownership of the `~/.goclaw/` directory and that the credentials file has the correct `0600` permissions.