🏠 Home ← Docs Index
Open Beta 📖 Interactive Reader Raw .md llms.txt

Model Failover

Every agent is served through fallbackAgent (internal/agent/fallback_agent.go): its own provider first, then each provider/model of its fallback list in order, when the provider fails before the reply has used a tool. The list is ai.fallback, or the agent's own agents.list[].fallback, which replaces it ([] means none). It is read on each request, so a config reload applies to the next one. Provider Fallback has the configuration and the full rules; this page explains the moving parts.

Before a request moves on, the same provider is asked again: both agent loops retry an overloaded call twice and a network failure once, for every provider (internal/agent/llm_retry.go).

How It Works

flowchart TD
    REQ["Request"] --> P0["<b>Agent's Provider</b>"]
    P0 -->|Success| RES["Response"]
    P0 -->|"Provider-side failure<br/>(no tool has run)"| F1["<b>Fallback #1</b>"]
    F1 -->|Success| RES
    F1 -->|"Provider-side failure<br/>(no tool has run)"| F2["<b>Fallback #2</b>"]
    F2 -->|Success| RES
    F2 -->|"Provider-side failure<br/>(no tool has run)"| ERR["<b>Error</b><br/>Names every provider's failure"]

A failure after a tool started is not passed on: the next provider would run the tool again. Text the failed provider had streamed is discarded (EventBufferReset).

Cooldown System

When a provider fails, it enters a cooldown period. The cooldown duration uses exponential backoff:

Error Count Cooldown Duration
1st failure 1 minute
2nd failure 5 minutes
3rd failure 25 minutes
4th+ failure 1 hour (maximum)

Formula: 60s * 5^min(errorCount - 1, 3)

Every failure reason gets the same cooldown. A provider in cooldown is skipped while others remain; the last candidate is always tried.

Counter Reset

Error counters are reset in two scenarios:

  1. On success: When a provider call succeeds, all error state is cleared immediately
  2. On expiry: When both cooldownUntil and disabledUntil have expired, errorCount and failureCounts are reset to zero

This prevents infinite escalation — without reset, stale error counts from hours ago would cause disproportionately long cooldowns on the next failure.

Additionally, if no failures have occurred within the failure window (10 minutes) and the previous cooldown has expired, counters are reset before incrementing the new failure.

Probe Throttling

When the agent's own provider is in cooldown and fallbacks exist, GoClaw probes it now and then to find out whether it has recovered. Probes are throttled to avoid hammering a failing provider:

Parameter Value
Min probe interval 30 seconds
Probe margin 2 minutes before cooldown expiry
Probe state TTL 24 hours
Max probe keys 256

Probe Decision Logic

  1. Only the agent's own provider gets probe attempts (fallback providers in cooldown are skipped)
  2. The probe is allowed when now >= soonestCooldownExpiry - 2 minutes
  3. The probe throttle must be open (at least 30s since last probe for this provider)
  4. If both conditions are met, the provider is attempted; otherwise, it's skipped

Error Classification

Provider errors are classified into structured failure reasons. The last column says whether the request moves to the next provider:

Reason Trigger Patterns Moves on
auth 401, 403, authentication, unauthorized yes
billing 402, billing, quota exceeded yes
rate_limit 429, rate limit, too many requests yes
overloaded 503, overloaded, capacity yes
timeout timeout, deadline exceeded, ETIMEDOUT yes
model_not_found 404, model not found yes
format 400, invalid, malformed no
unknown Anything else only a network failure

Context Overflow

Context overflow errors (e.g., context_length_exceeded) never move to a fallback provider. These errors indicate the prompt is too large, and switching providers won't help. Neither does a cancelled request or one over its token budget.

Attempt Tracking

Each move to another provider is logged with the failures before it, and a fallback whose client cannot be built (no key) is logged and skipped:

[fallback] serving the request with deepseek/deepseek-v4-flash (gemini/gemini-3-flash: 429 Too Many Requests)
[fallback] openai/gpt-4o-mini cannot be used: <error>

When every provider fails, the request fails with every provider failed: …, naming each one's error. goclaw doctor shows an ai_fallback card per agent.

Related