Provider Fallback & Cooldown System

Each agent is served by its own provider (ai.provider, or the agent's agents.list[].model) and, when that fails, by the providers of its fallback list:

ai:
  provider: gemini
  fallback:                       # tried in order; each needs its credentials under ai.<provider>
    - deepseek/deepseek-v4-flash
    - openai/gpt-4o-mini

agents:
  list:
    - id: ops
      model: claude/claude-sonnet-4-5
      fallback: [gemini/gemini-3-flash]   # replaces ai.fallback for this agent
    - id: local
      model: ollama/llama3
      fallback: []                        # no fallback

internal/agent/fallback_agent.go (the wrapper every agent is served through), with internal/agent/fallback.go (probe throttle, error classification) and internal/agent/provider_cooldown.go (cooldowns).

Before it moves on: the same provider is asked again

A model call that fails because the provider is busy is made again before the request fails or moves to the next provider, in both agent loops and for the partial summary at the tool-call limit (internal/agent/llm_retry.go):

Failure	Retried
Overload: 503 / `UNAVAILABLE` ("The service is currently unavailable", "high demand"), Anthropic's 529, a 429 that says the model is overloaded	twice, after about 2 s and 6 s (± a fifth); a longer `Retry-After` (Gemini's `RetryInfo`) is waited out, up to 20 s, and one longer than that is not retried
A network failure (refused, reset, i/o timeout)	once, after 0.5 s
Quota, billing, a rate limit, a key or permission error, anything else	no

Each retry logs one line ([LLM] gemini:<model>: provider overloaded; retry 1 in 2.1s: …). Text the failed call had streamed is discarded (EventBufferReset), and the per-call token checks are made again before the retry. When every try fails, the request fails, or moves on, with the error it failed with before.

When a request moves to the next provider

A request moves on when its provider fails before the reply has used a tool, for a reason on the provider's side:

Failure	Moves on
Rate limit (429), overload (503), timeout	yes
Key or permission (401, 403), billing or quota (402)	yes
Model not found (404)	yes
A network failure (refused, reset, DNS)	yes
The request itself: too long for the context window, malformed (400)	no
The caller: cancelled, over its token budget	no
Any failure after a tool started	no: the next provider would run the tool again

Text the failed provider had already streamed is discarded (EventBufferReset), so a streaming client shows only the answer that completes. The prompt is formatted for the provider that serves the request. A provider that fails is put in cooldown (below) and skipped while others remain; the agent's own provider is tried again now and then (the probe) to find out it is back. When every provider fails, the request fails with each one's error.

The list is read on each request, so a config reload applies to the next one. A fallback provider's client is built the first time it is needed; one that cannot be built (no key) is logged and skipped.

Feature Comparison

#	Feature	OpenClaw	GoClaw	Status
2.1	Multi-provider fallback chain	✅ Provider rotation	✅ `ai.fallback` / `agents.list[].fallback`, served through `fallbackAgent`; streaming, never after a tool ran	Done
2.2	Error classification (8 types)	Basic retry	✅ auth, billing, rate_limit, overloaded, timeout, model_missing, format, unknown	Done
2.3	Exponential backoff cooldown	✅	✅ `60s × 5^min(n-1, 3)`, max 1 hour	Done
2.4	Probe throttling	—	✅ Primary provider only, 30s min interval	GoClaw only
2.5	Context overflow detection	✅	✅ `isContextOverflow` — stops fallback chain	Done
2.6	Failure window reset	—	✅ 10min no-error window → counter reset	GoClaw only
2.7	Probe state TTL + eviction	—	✅ 24h TTL, max 256 keys	GoClaw only
2.8	Per-provider usage stats	—	✅ `ProviderUsageStats` — lastUsed, errorCount, failureCounts, disabled	GoClaw only
2.9	Disabled vs Cooldown distinction	—	`IsDisabled` checks `DisabledUntil`, which nothing sets; only the cooldown is ever applied	Unused
2.10	Fallback attempt tracking	—	✅ each move is logged (`[fallback] serving the request with …`), and the final error names every provider's failure	GoClaw only
2.11	Provider health dashboard	✅	—	TODO

Cooldown Backoff Formula

cooldown_ms = min(60000 × 5^min(errorCount-1, 3), 3600000)

Error 1: 1 min
Error 2: 5 min
Error 3: 25 min
Error 4+: 60 min (max)

Error Classification (`classifyError`)

Keyword Pattern	Failure Reason
401, 403, unauthorized	`auth`
402, billing, quota	`billing`
429, rate limit	`rate_limit`
503, overloaded	`overloaded`
timeout, deadline	`timeout`
404, model not found	`model_not_found`
400, invalid	`format`
(other)	`unknown`

Constants

Constant	Value	Description
`minProbeIntervalMs`	30,000	Min interval between probes
`probeMarginMs`	120,000	Probe allowed near cooldown expiry
`probeStateTTLMs`	86,400,000	24h — probe state eviction
`maxProbeKeys`	256	Max tracked throttle keys
`probeScopeDelimiter`	`::`	agentDir::provider key separator

Every agent is served through fallbackAgent, so this runs for every request; there is no provider health dashboard yet (2.11).