Skip to main content

How LLM Fallback and Failover Work

· 4 min read
DVARA Team
DVARA Team

Model providers are remarkably good and occasionally unavailable. They rate-limit you at the worst moment, degrade under load, and have real outages. If your application talks to a provider directly, every one of those events is a user-facing failure — and every team ends up writing its own retry loop, usually badly.

Resilience belongs in the same layer as governance, and for the same reason: it can't be trusted if it's scattered and invisible. When failover runs at a single governed control point — the DVARA LLM Gateway, the data plane of the AI governance platform — every retry and provider switch is policy-checked and written to the audit trail, so "we failed over to a different provider" is a recorded, reviewable event, not a silent mystery. LLM fallback and failover move resilience out of application code and onto the governed path.

Fallback vs. failover​

The terms get used interchangeably; the distinction is useful:

  • Failover — the primary provider fails (an error, a 429, a timeout, an open circuit) and the request is retried against an alternative that can serve it.
  • Fallback — the ordered list of alternatives to try. Your fallback chain is the plan; failover is executing it.

Both live naturally at a governed control point, because it's the one place that already knows every provider, their health, and their capabilities — and governs the switch.

The circuit breaker​

Retrying a provider that's clearly down just adds latency and load. A circuit breaker tracks failures per provider and, once they cross a threshold, "opens" — the Gateway stops sending traffic there for a cooldown window and fails straight over to an alternative. When the cooldown passes, it lets a probe through; if that succeeds, the circuit closes and normal traffic resumes.

This protects both sides: your users don't wait on a doomed call, and you don't hammer a struggling provider while it's trying to recover.

Capability-aware failover — the part that's easy to get wrong​

Here's the failure mode naive failover walks into. A request asks for response_format: json_schema. The primary — which supports structured outputs — is down. A naive gateway fails over to the next provider in the list… which doesn't support structured outputs. Now the request fails anyway, with a confusing error, and your "resilience" made things worse.

Capability-aware failover filters the fallback candidates by what the request actually needs before retrying. If the request needs structured output, only structured-output-capable providers are considered. If no capable fallback exists, the Gateway says so explicitly — a clear "no capable provider for this request" error — instead of failing over into a wall. This is the same capability filtering that governs normal multi-provider routing.

Upstream rate limits: shed before you fail​

The best failover is the one you never trigger. Providers publish rate-limit headers on every response — how much of your token and request budget remains. A control point that reads those can proactively shed or shift traffic as you approach a limit, rather than waiting for the 429. Under pressure it can honor workspace priority — protecting premium traffic and letting bulk traffic absorb the throttling first. You degrade deliberately instead of hitting a wall, and the shed is attributed and audited like any other governed decision.

Resilience you can prove​

Failover only earns trust if you can observe and account for it. Which providers are failing over, how often, to whom? What's the retry rate, the circuit-open rate, the fallback rate by provider pair? Those are Prometheus metrics and audit events, not guesswork — which is why resilience and observability are two sides of the same coin. A governance platform doesn't just recover from a provider outage; it can show you exactly what it did.

A checklist for evaluating failover​

  • Does it use a circuit breaker, or just blind retries?
  • Is failover capability-aware, or will it retry on a model that can't serve the request?
  • Does it react to upstream rate-limit headers before the 429, or only after?
  • Does it honor workspace priority when a shared credential is near its limit?
  • Are failover events observable and audited — metrics and a trail per provider pair, not just a log line?

Where DVARA fits​

DVARA is an AI governance platform whose LLM Gateway wraps each provider in a circuit breaker, fails over capability-aware (a request that needs structured output only retries on a provider that supports it, with a clear error if none can), and reads upstream rate-limit headers to shed or shift traffic — honoring workspace priority — before a 429 fires. Every retry, fallback, and shed is a governed, audited, observable event. Read the pillar guide, What Is an LLM Gateway?, for how resilience fits with routing and cost control.