RELIABILITY GUIDE · LAST REVIEWED JULY 15, 2026
LLM fallback routing: make failures recoverable, not invisible.
Short answer: retry transient failures, change route only when the task contract permits it, and record every decision. Never use fallback to hide invalid requests, authorization failures, or unsafe output.
What fallback routing solves
LLM applications depend on networks, providers, rate limits, model availability, and long-running requests. A fallback policy can prevent one unavailable route from becoming a complete user-facing outage. It is not a substitute for validation, quota management, or a clear product error state.
| Failure class | Recommended action | Do not do this |
|---|---|---|
| Network timeout or temporary 5xx | Retry a limited number of times with backoff; then use an equivalent fallback if the task permits. | Retry forever or send duplicate work without idempotency protection. |
| Rate limit or overload | Honor provider retry guidance; queue, delay, or move to an approved alternate route. | Immediately fan out the same request to every provider. |
| Invalid request, bad credentials, or insufficient permission | Return an actionable error and fix the request or account state. | Switch models in the hope that validation or authorization changes. |
| Safety, tool, or structured-output contract failure | Apply the product-specific recovery path and preserve the original failure reason. | Silently return an incompatible answer from a weaker route. |
A safe fallback policy
- Classify the error as transient, rate-related, permanent, or contract-related.
- Retry only transient errors and use bounded exponential backoff.
- Before switching, check that the alternate route supports the required modalities, tools, context, response format, and data policy.
- Log the original route, failure class, retry count, selected fallback, latency, and final outcome.
- Expose a clear degraded-mode message when quality or capability changes materially.
Metrics that reveal whether fallback helps
Track success rate after fallback, added latency, fallback frequency by provider and model, task-level quality regression, and duplicate or abandoned requests. A high fallback rate is a capacity or dependency signal—not simply a routing success metric.
How ELTEX approaches it
ELTEX is designed to make provider and model routing a policy layer. For production use, teams should still define which tasks can degrade, which require an exact model or tool contract, and what users see when no compatible route is available.
Explore ELTEX AI Router All guides
Primary source
Anthropic API error handling documents error classes, request IDs, and transient-failure retry behavior. Apply provider-specific documentation for every route you operate.