> ## Documentation Index > Fetch the complete documentation index at: https://gomodel-feat-audit-operation-filter.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Resilience > Tune retries and the per-provider circuit breaker, and learn when to reach for cross-model failover instead. GoModel wraps every upstream provider call with two resilience layers: 1. **Retry with exponential backoff** — repeats a failed request against the same provider with growing delays. 2. **Circuit breaker** — short-circuits calls to a provider that has been failing repeatedly, then probes once the timeout elapses. Retries apply to the selected target. Circuit breakers apply per provider by default, or per model with `scope: model`. They do not switch to a different model or provider on failure. For cross-model failover, see [Failover](/features/failover). This page covers the knobs. For what GoModel does when a provider changes its response shape, answers `200` with nothing usable, or dies mid-stream, see [Provider failure modes](/advanced/provider-failure-modes). ## Defaults The defaults are tuned to be safe for most deployments. Override only what you need. | Setting | Default | Notes | | --------------------- | -------------------------------- | -------------------------------------------------- | | `retry_on_statuses` | `[429, 502, 503, 504, 522, 524]` | HTTP statuses that trigger retries | | `failure_on_statuses` | `[429, 5xx]` | HTTP statuses that count as breaker failures | | `scope` | `provider` | Breaker isolation: `provider` or `model` | | `max_retries` | `3` | Maximum retry attempts per request | | `initial_backoff` | `1s` | First retry wait | | `max_backoff` | `30s` | Upper cap on retry wait | | `backoff_factor` | `2.0` | Exponential multiplier between retries | | `jitter_factor` | `0.1` | Random jitter as a fraction of the backoff | | `enabled` | `true` | Switches the circuit breaker on or off | | `failure_threshold` | `5` | Consecutive failures before the circuit opens | | `success_threshold` | `2` | Consecutive successes to close it again | | `timeout` | `30s` | How long the circuit stays open before probing | | `slow_call_threshold` | `0` (off) | Answers slower than this count as breaker failures | To disable retries, set `max_retries: 0` — every request gets exactly one attempt. To disable the circuit breaker, set `circuit_breaker.enabled: false` (or `CIRCUIT_BREAKER_ENABLED=false`); the thresholds are kept but never consulted, so flipping it back on restores your tuning. Setting `failure_threshold: 0` also disables the breaker. Both switches work globally or per provider, and a per-provider `enabled: true` re-enables the breaker for that provider when it is off globally. Turn the breaker off when something in front of the provider already sheds load (a provider-side load balancer, a service mesh) or when a short burst of `5xx` responses must never pause traffic to that provider. Keep it on otherwise: it is what stops a dead upstream from tying up every request for the full timeout. ## What counts as a failure **Retries** fire on transport errors (connection refused, resets, DNS failures) and, by default, on `429`, `502`, `503`, `504`, `522`, and `524` responses. Other statuses — including `500` — are returned to the caller without retrying. The following paths retry less than the table suggests: * **Local HTTP client timeouts** return immediately without retrying. An expired overall request context also prevents further attempts. * **Streaming** requests are never retried once dispatched, because partial data may already have been sent. * **Passthrough** requests are retried only when they are replay-safe: `GET`, `HEAD`, `OPTIONS`, `PUT`, or any request carrying an `Idempotency-Key` header. **Idempotency keys** reach the provider. When a client sends an `Idempotency-Key` header, GoModel forwards it unchanged on the provider call it makes for that request and on every retry of that call, so a provider that deduplicates by the key can recognize a replayed request. A failover attempt to another target sends a different request body, so it goes out without the key, and so do gateway-internal calls such as guardrail checks. Keys must be visible ASCII of at most 255 characters; any other value is ignored. **Errors hidden behind HTTP 200** are unmasked before any of this applies. Some providers (OpenRouter is the best-known) answer `200 OK` with a bare `{"error": ...}` JSON body. GoModel detects such payloads and handles them as the error they really are: a status embedded in `error.code` is preserved (so a hidden `429` behaves like a real one), anything else maps to `502`. The mapped status then drives retries, the circuit breaker, and failover exactly like a genuine error status, and the response is never cached as a success. Passthrough endpoints are exempt: they forward the provider's response byte-for-byte. **Streams that stop early** end with an explicit error instead of a silent EOF. When a provider stream closes or fails before its terminal event (`data: [DONE]` or a `finish_reason` for chat completions, a terminal `response.*` event for `/v1/responses`, `message_stop` for `/v1/messages`), GoModel relays everything received so far and then sends one error event in the endpoint's format: * chat completions: `data: {"error":{"code":"stream_incomplete",...}}` * `/v1/responses`: an `error` event with `"code":"stream_incomplete"` * `/v1/messages`: an Anthropic `error` event with `error.type` set to `api_error` This covers malformed chunks, a connection dropped in the middle of a tool call, and similar truncations, so a client never mistakes a partial answer for a complete one. A stream that already sent its terminal event stays complete even if the connection fails afterwards. The request log records a truncated stream as a `stream_error`. **The circuit breaker** counts transport errors and, by default, `429` and all `5xx` responses as failures. One exhausted retry sequence counts as one failure; individual HTTP retries do not increment the breaker. A successful sequence records success. Client cancellation is ignored; local client timeouts count as failures. Set `retry.retry_on_statuses` and `circuit_breaker.failure_on_statuses` independently. Both accept exact codes and classes such as `5xx`. Omitted lists use defaults; explicit lists replace them; `[]` disables status-based triggers while retaining transport-error handling. For example, `failure_on_statuses: [5xx]` excludes rate limits from breaker failures, including during recovery probes. Invalid codes or scopes fail configuration loading. **Sustained latency** can open the circuit too. Set `circuit_breaker.slow_call_threshold` (or `CIRCUIT_BREAKER_SLOW_CALL_THRESHOLD`) and any call the provider answers slower than that counts as a breaker failure, exactly like a `5xx`. The slow answer is still returned to its caller; after `failure_threshold` consecutive slow or failed calls the circuit opens, later requests fail fast with a failover-eligible `503`, and a slow recovery probe reopens it. See [Fail over from a slow provider](#fail-over-from-a-slow-provider). The clock covers the whole call including retries. For streaming requests it stops at the first data event, so the threshold bounds time to first token, not the length of the stream. It is off by default because normal latency depends on the workload: a non-streaming reasoning call can legitimately take minutes. Pick a value well above your slowest healthy request, and override it per provider when workloads differ. The threshold never cancels a request — that is `HTTP_TIMEOUT`'s job, and a call that hits it already counts as a failure. The default now includes `429` breaker failures. Deployments that previously relied on rate limits leaving the breaker closed can set `failure_on_statuses: [5xx]` to retain that behavior. While the circuit is open, requests fail fast with a `503` and the message `circuit breaker is open - provider temporarily unavailable`, where `` is the configured provider name (`openai-eu`, not its type). After `timeout` elapses, a single probe request is let through while concurrent requests keep failing fast; `success_threshold` consecutive successful probes close the circuit, and a failure matching the breaker policy reopens it. Probes make only one HTTP attempt, without retries. The circuit breaker is in-memory and per gateway process: each provider gets its own breaker by default. With `scope: model`, each model within a provider instance gets an independent breaker; requests without a model, such as discovery, use a separate provider breaker. Multipart audio uploads use their explicit model names. Each provider retains at most 1,024 model breakers. Idle closed entries expire after 10 minutes when a new model is looked up; the oldest idle closed entry is evicted sooner when the limit is reached. Active and open breakers are preserved. If every slot is protected, a new model receives a failover-eligible `503` until a slot becomes available. State resets on restart and nothing is shared between replicas. ## Environment Variables These set the global defaults that apply to every provider unless overridden in YAML. ### Retry | Variable | Type | Default | Description | | ----------------------- | -------------------- | ------------------------- | ------------------------------------------ | | `RETRY_ON_STATUSES` | comma-separated list | `429,502,503,504,522,524` | HTTP statuses that trigger retries | | `RETRY_MAX_RETRIES` | int | `3` | Maximum retry attempts per request | | `RETRY_INITIAL_BACKOFF` | duration | `1s` | First retry wait (e.g. `500ms`, `2s`) | | `RETRY_MAX_BACKOFF` | duration | `30s` | Upper cap on retry wait | | `RETRY_BACKOFF_FACTOR` | float | `2.0` | Exponential multiplier between retries | | `RETRY_JITTER_FACTOR` | float | `0.1` | Random jitter as a fraction of the backoff | ### Circuit Breaker | Variable | Type | Default | Description | | ------------------------------------- | -------------------- | ---------- | -------------------------------------------------------- | | `CIRCUIT_BREAKER_FAILURE_ON_STATUSES` | comma-separated list | `429,5xx` | HTTP statuses counted as failures | | `CIRCUIT_BREAKER_SCOPE` | string | `provider` | `provider` or `model` isolation | | `CIRCUIT_BREAKER_ENABLED` | bool | `true` | Set `false` to turn the circuit breaker off | | `CIRCUIT_BREAKER_FAILURE_THRESHOLD` | int | `5` | Consecutive failures before opening | | `CIRCUIT_BREAKER_SUCCESS_THRESHOLD` | int | `2` | Consecutive successes to close again | | `CIRCUIT_BREAKER_TIMEOUT` | duration | `30s` | How long the circuit stays open before probing | | `CIRCUIT_BREAKER_SLOW_CALL_THRESHOLD` | duration | `0` | Answers slower than this count as failures; `0` disables | ## YAML The same fields are available under the global `resilience:` block, and can be overridden per provider: ```yaml theme={null} resilience: retry: max_retries: 2 initial_backoff: 500ms max_backoff: 10s backoff_factor: 1.5 jitter_factor: 0.05 circuit_breaker: enabled: true failure_threshold: 3 success_threshold: 1 timeout: 15s providers: anthropic: type: anthropic api_key: ${ANTHROPIC_API_KEY} resilience: retry: max_retries: 5 # Anthropic supports long requests — allow more retries ollama: type: ollama base_url: ${OLLAMA_BASE_URL:-http://localhost:11434/v1} resilience: circuit_breaker: failure_threshold: 10 # local service — tolerate more transient failures timeout: 5s vllm: type: vllm base_url: ${VLLM_BASE_URL:-http://localhost:8000/v1} resilience: circuit_breaker: enabled: false # fronted by its own load balancer — never fail fast ``` Only fields explicitly listed under a provider's `resilience:` block are overridden. Everything else inherits from the global section, which in turn inherits from the built-in defaults. Per-provider tuning **must** come from YAML. Environment variables set global defaults only — `RETRY_MAX_RETRIES` cannot target a single provider. See [config.yaml gotchas](/advanced/config-yaml#gotchas). ### Worked example Given the YAML above, the effective per-provider settings are: | Provider | max\_retries | cb enabled | failure\_threshold | cb timeout | | --------- | ---------------- | -------------------- | ------------------ | -------------------- | | openai | 2 (global) | true (global) | 3 (global) | 15s (global) | | anthropic | **5** (override) | true (global) | 3 (global) | 15s (global) | | ollama | 2 (global) | true (global) | **10** (override) | **5s** (override) | | vllm | 2 (global) | **false** (override) | 3 (global, unused) | 15s (global, unused) | `anthropic`, `ollama`, and `vllm` inherit every field they did not explicitly override. With the breaker disabled, `vllm` also reports no breaker state on the dashboard or in the `gomodel_circuit_breaker_state` metric. ## Circuit breaker and dashboard provider health The dashboard's [provider status](/providers/overview#provider-status) combines two independent signals: * **Model discovery** — whether the provider's model inventory could last be fetched (details below). * **Request health** — a 10-minute sliding window of real request outcomes per provider and model, including the live circuit breaker state. The circuit breaker's state is shown on each provider card: expanding a card reveals a **Breaker State** chip, an open breaker turns the provider's status pill to `Circuit Open`, and a half-open breaker shows `Recovering`. With `scope: model`, the provider-level breaker display and metric reflect the breaker of the most recently completed request, rather than an aggregate of all model breakers. A model whose recent requests keep failing (at least 3 errors making up half or more of its windowed requests) marks the provider `Degraded` even while model discovery still succeeds — this catches upstreams that list models fine but fail real calls, e.g. with misreported `4xx` errors that deliberately never trip the breaker. Request-health signals only ever worsen the discovery-based status, never improve it, and the tracking is in-memory per gateway process. The breaker still recovers on its own within `timeout` (default 30s) once the provider is back; when [metrics](/advanced/configuration#metrics) are enabled its state is also exported as the `gomodel_circuit_breaker_state` gauge (0 = closed, 1 = half-open, 2 = open). Model discovery is re-checked: * at startup, * on every model registry refresh, controlled by `CACHE_REFRESH_INTERVAL` (seconds, default `3600` — hourly), * on the fast recheck loop, which re-probes **only** providers whose latest refresh failed, controlled by `PROVIDER_RECHECK_INTERVAL` (`cache.model.recheck_interval` in `config.yaml`; seconds, default `60`; `0` disables), and * on demand, when a request asks for a provider-qualified model (`provider/model`) that is missing from the registry. ## What happens while a provider is down When a provider's refresh fails, its previously discovered models are marked stale, and the dashboard shows the provider as Offline: * **Model listings** (`GET /v1/models` and the dashboard model list) hide the provider's models until it recovers, so clients are not offered models that cannot currently be served. * **Direct requests** to its models (`provider/model`) still resolve and are sent to the provider, so callers get an honest `502`/`503` (and a shadowing virtual model's failover chain can fire) instead of a misleading "model not found". * **Virtual-model redirects** skip the provider's targets, so a load-balanced redirect keeps working through its healthy targets. Exception: when **every** provider is failing at once (for example, a control-plane-only outage where model discovery is unreachable but inference still works), the previous inventory is kept as-is and stays listed — hiding everything would only turn provider errors into "model not found". The fast recheck loop re-probes the provider every `PROVIDER_RECHECK_INTERVAL` seconds, updating "Last checked" and restoring normal routing typically within a minute of the provider coming back. A provider that was already down at startup (nothing discovered yet) shows as Unhealthy until its first successful fetch. ## Failover vs. Resilience Retries stay on the selected target; the circuit breaker tracks the configured provider or model scope. If you also want GoModel to try a different model or provider when the primary keeps failing, give the model a virtual model with more than one target — the remaining targets are its failover chain. See [Failover](/features/failover). ## Retry Cloudflare timeouts, then switch models For non-streaming translated requests, combine retries with a failover virtual model. Merge the following settings into your existing configuration, using your configured provider and model names: ```yaml theme={null} providers: cloudflare: # Keep your existing type, base_url, and credentials here. resilience: retry: max_retries: 2 retry_on_statuses: [429, 502, 503, 504, 522, 524] circuit_breaker: scope: model failure_on_statuses: [429, 5xx] failure_threshold: 5 virtual_models: - source: resilient-chat strategy: failover targets: - model: cloudflare/model1 - model: cloudflare/model2 failover: enabled: true retry_on_statuses: [429, 5xx] ``` Send `model: resilient-chat`. While model1's breaker is closed, a `524` response triggers up to two retries against model1. If all three attempts fail, GoModel tries model2 with its own retry budget. Model1's open breaker skips its upstream calls without blocking model2. With the default `scope: provider`, both models share a breaker instead. The caller's deadline must allow time for the attempts and backoff. Streaming requests do not get this retry sequence; a partially delivered response cannot be restarted transparently. ## Fail over from a slow provider A provider that still answers, but takes minutes to do so, never produces the errors that open the circuit. `slow_call_threshold` makes latency count: ```yaml theme={null} providers: opencode: # Keep your existing type, base_url, and credentials here. resilience: circuit_breaker: scope: model # one slow model must not pause the whole provider slow_call_threshold: 60s failure_threshold: 3 virtual_models: - source: resilient-chat strategy: failover targets: - model: opencode/deepseek-v3 - model: openai/gpt-4o-mini failover: enabled: true ``` Once three calls in a row to `opencode/deepseek-v3` take longer than 60 seconds (or fail), its circuit opens and `resilient-chat` requests go straight to `openai/gpt-4o-mini` without waiting. Every `timeout` (default 30s) one probe request tries the primary again; traffic returns once probes answer within the threshold. A fast answer in between resets the count, so occasional slow calls never open the circuit. Prefer `scope: model` here: with the default `scope: provider`, fast models on the same provider keep resetting the count for the slow one.