# Model Failover

Every agent is served through `fallbackAgent` (`internal/agent/fallback_agent.go`): its own
provider first, then each `provider/model` of its fallback list in order, when the provider fails
before the reply has used a tool. The list is `ai.fallback`, or the agent's own
`agents.list[].fallback`, which replaces it (`[]` means none). It is read on each request, so a
config reload applies to the next one. [Provider Fallback](../provider_fallback.md) has the
configuration and the full rules; this page explains the moving parts.

Before a request moves on, the same provider is asked again: both agent loops retry an overloaded
call twice and a network failure once, for every provider (`internal/agent/llm_retry.go`).

## How It Works

```mermaid
flowchart TD
    REQ["Request"] --> P0["<b>Agent's Provider</b>"]
    P0 -->|Success| RES["Response"]
    P0 -->|"Provider-side failure<br/>(no tool has run)"| F1["<b>Fallback #1</b>"]
    F1 -->|Success| RES
    F1 -->|"Provider-side failure<br/>(no tool has run)"| F2["<b>Fallback #2</b>"]
    F2 -->|Success| RES
    F2 -->|"Provider-side failure<br/>(no tool has run)"| ERR["<b>Error</b><br/>Names every provider's failure"]
```

A failure after a tool started is not passed on: the next provider would run the tool again.
Text the failed provider had streamed is discarded (`EventBufferReset`).

## Cooldown System

When a provider fails, it enters a cooldown period. The cooldown duration uses exponential backoff:

| Error Count | Cooldown Duration |
|-------------|-------------------|
| 1st failure | 1 minute |
| 2nd failure | 5 minutes |
| 3rd failure | 25 minutes |
| 4th+ failure | 1 hour (maximum) |

**Formula**: `60s * 5^min(errorCount - 1, 3)`

Every failure reason gets the same cooldown. A provider in cooldown is skipped while others
remain; the last candidate is always tried.

## Counter Reset

Error counters are reset in two scenarios:

1. **On success**: When a provider call succeeds, all error state is cleared immediately
2. **On expiry**: When both `cooldownUntil` and `disabledUntil` have expired, `errorCount` and `failureCounts` are reset to zero

This prevents **infinite escalation** — without reset, stale error counts from hours ago would cause disproportionately long cooldowns on the next failure.

Additionally, if no failures have occurred within the **failure window** (10 minutes) and the previous cooldown has expired, counters are reset before incrementing the new failure.

## Probe Throttling

When the agent's own provider is in cooldown and fallbacks exist, GoClaw **probes** it now and then
to find out whether it has recovered. Probes are throttled to avoid hammering a failing provider:

| Parameter | Value |
|-----------|-------|
| Min probe interval | 30 seconds |
| Probe margin | 2 minutes before cooldown expiry |
| Probe state TTL | 24 hours |
| Max probe keys | 256 |

### Probe Decision Logic

1. Only the agent's **own provider** gets probe attempts (fallback providers in cooldown are skipped)
2. The probe is allowed when `now >= soonestCooldownExpiry - 2 minutes`
3. The probe throttle must be open (at least 30s since last probe for this provider)
4. If both conditions are met, the provider is attempted; otherwise, it's skipped

## Error Classification

Provider errors are classified into structured failure reasons. The last column says whether the
request moves to the next provider:

| Reason | Trigger Patterns | Moves on |
|--------|-----------------|----------|
| `auth` | 401, 403, authentication, unauthorized | yes |
| `billing` | 402, billing, quota exceeded | yes |
| `rate_limit` | 429, rate limit, too many requests | yes |
| `overloaded` | 503, overloaded, capacity | yes |
| `timeout` | timeout, deadline exceeded, ETIMEDOUT | yes |
| `model_not_found` | 404, model not found | yes |
| `format` | 400, invalid, malformed | no |
| `unknown` | Anything else | only a network failure |

## Context Overflow

Context overflow errors (e.g., `context_length_exceeded`) never move to a fallback provider. These
errors indicate the prompt is too large, and switching providers won't help. Neither does a
cancelled request or one over its token budget.

## Attempt Tracking

Each move to another provider is logged with the failures before it, and a fallback whose client
cannot be built (no key) is logged and skipped:

```
[fallback] serving the request with deepseek/deepseek-v4-flash (gemini/gemini-3-flash: 429 Too Many Requests)
[fallback] openai/gpt-4o-mini cannot be used: <error>
```

When every provider fails, the request fails with `every provider failed: …`, naming each one's
error. `goclaw doctor` shows an `ai_fallback` card per agent.

## Related

- [Provider Fallback](../provider_fallback.md) — Configuration and rules
- [Agent Runtime](agent.md) — How the agent processes messages
- [Configuration](../../user-guide/configuration.md) — Provider configuration