Resilience and failover

DVARA wraps every provider with retry, circuit breaker, and timeout protection. When a provider fails, the gateway automatically attempts fallback providers.

Routing chooses the primary provider. When a request matched a route, failover stays inside that route's own fallbacks list. Read Routing and load balancing for primary selection and use this page as the source of truth for what happens after a provider failure.

Change these settings in Flightdeck, with no restart

From 1.10.0 the retry, circuit-breaker, timeout and failover settings on this page can be changed on Flightdeck's Resilience page, and every gateway applies them within a bundle refresh. See Change settings in Flightdeck. The values below are the starting point: what the gateway uses for any field the page does not set. Up to 1.9.x every value was read once at boot.

Retry with Exponential Backoff

Failed requests are automatically retried up to a configurable number of attempts. The wait between attempts starts at initial-backoff-ms, multiplies by backoff-multiplier on each retry, and is capped at max-backoff-ms:

dvara:
  llm-gateway:
    resilience:
      retry:
        max-attempts: 3          # total attempts including the first try
        initial-backoff-ms: 500  # wait before the first retry
        backoff-multiplier: 2.0  # delay grows: 500ms → 1s → 2s → …
        max-backoff-ms: 10000    # capped so a long chain can't explode

Or via environment variables:

DVARA_LLM_GATEWAY_RESILIENCE_RETRY_MAX_ATTEMPTS=3
DVARA_LLM_GATEWAY_RESILIENCE_RETRY_INITIAL_BACKOFF_MS=500
DVARA_LLM_GATEWAY_RESILIENCE_RETRY_BACKOFF_MULTIPLIER=2.0
DVARA_LLM_GATEWAY_RESILIENCE_RETRY_MAX_BACKOFF_MS=10000

Retry follows what the response says about provider health. A provider 429 is retried with backoff. A 408, a 5xx, or a connection failure is retried and counts against the circuit breaker. Other provider 4xx responses are returned immediately and do not count against the shared circuit. WORKSPACE_CREDENTIAL_REQUIRED and UNSUPPORTED_CAPABILITY never count against it. From 1.10.0 an error inside the gateway, such as a provider answer the gateway could not read or convert, does not count against the provider and is not retried either; only a network error or timeout anywhere in its causes, or an HTTP 5xx or 408, does. Up to 1.9.0 a few such gateway errors could open a healthy provider's circuit, and were retried, paying the provider again. A timed-out call is interrupted rather than continuing in the background.

This changes upgrade behaviour: a bad credential or a request the provider refuses no longer opens that provider's circuit for every workspace.

Circuit Breaker

Each provider has an independent circuit breaker that prevents cascading failures:

dvara:
  llm-gateway:
    resilience:
      circuit-breaker:
        failure-rate-threshold: 50              # percentage of failing calls in the window that opens the breaker
        sliding-window-size: 10                 # number of calls the breaker averages over
        minimum-number-of-calls: 5              # minimum calls in the window before the breaker considers tripping — suppresses early-startup noise
        wait-duration-in-open-state-ms: 30000   # time the breaker stays OPEN before probing upstream again
        permitted-calls-in-half-open: 3         # number of probe calls allowed in HALF_OPEN — if all succeed, the breaker closes

Or via environment variables:

DVARA_LLM_GATEWAY_RESILIENCE_CIRCUIT_BREAKER_FAILURE_RATE_THRESHOLD=50
DVARA_LLM_GATEWAY_RESILIENCE_CIRCUIT_BREAKER_SLIDING_WINDOW_SIZE=10
DVARA_LLM_GATEWAY_RESILIENCE_CIRCUIT_BREAKER_MINIMUM_NUMBER_OF_CALLS=5
DVARA_LLM_GATEWAY_RESILIENCE_CIRCUIT_BREAKER_WAIT_DURATION_IN_OPEN_STATE_MS=30000
DVARA_LLM_GATEWAY_RESILIENCE_CIRCUIT_BREAKER_PERMITTED_CALLS_IN_HALF_OPEN=3

State Transitions

CLOSED ──(failure rate ≥ 50%)──▶ OPEN ──(30s wait)──▶ HALF_OPEN ──(test calls)──▶ CLOSED
                                  │                                     │
                                  │                                     ▼
                                  ◀────────────(still failing)──────── OPEN

State	Behavior
CLOSED	Normal operation. Tracks success/failure in a sliding window.
OPEN	All requests fail immediately with PROVIDER_CIRCUIT_OPEN (503). No upstream calls.
HALF_OPEN	A limited number of test calls are allowed through. If successful, circuit closes. If not, circuit re-opens.

Mock Provider Exclusion

The mock provider is excluded from circuit breaker and retry wrapping. Simulated errors injected via error-rate are intended for testing client-side error handling and will never trip the circuit breaker or trigger retries. The mock provider is always considered healthy.

Timeout Configuration

Separate timeouts for non-streaming and streaming requests:

dvara:
  llm-gateway:
    resilience:
      timeout:
        chat-timeout-ms: 30000        # 30 seconds for non-streaming
        streaming-timeout-ms: 120000  # 120 seconds for streaming

Or via environment variables:

DVARA_LLM_GATEWAY_RESILIENCE_TIMEOUT_CHAT_TIMEOUT_MS=30000
DVARA_LLM_GATEWAY_RESILIENCE_TIMEOUT_STREAMING_TIMEOUT_MS=120000

For a non-streaming call, the timeout can trigger fallback when fallback is enabled and an eligible target is available (see Automatic Failover). Streaming has two timeout boundaries described below.

Streaming and the resilience layer

The resilience wrapper governs the attempt to open a provider stream. A separate controller lifetime bounds the complete SSE response. The boundary matters when you tune streaming:

• Retry covers the attempt to obtain the provider's stream iterator. If that attempt fails, a configured retry can run. Once DVARA receives the iterator and starts reading, a later failure does not trigger a retry.
• Circuit breaker uses the provider outcome rules above. A stream that ends without a finish reason or carries an in-band provider error is PROVIDER_ERROR; do not treat the partial answer as complete.
• Fallback is attempted only when opening the provider stream fails before an iterator is returned. After reading begins, DVARA does not reissue the prompt to a different provider or concatenate partial responses.
• Timeout has two scopes. The global streaming-timeout-ms value, 120 seconds by default, bounds the complete client-visible SSE lifetime. A provider-specific override bounds only the attempt to open that provider stream. Neither value is an idle timer that resets after each event.

Size the global value for the longest complete response your application accepts. Use provider-specific values to stop waiting for a provider that cannot open a stream in time. On global timeout or client cancellation, DVARA closes delivery and releases the upstream transport. See SSE streaming for the post-header error and partial-evidence behavior.

Automatic Failover

With resilience and fallback enabled (both default to true), eligible failures on live chat calls can trigger another provider attempt. This includes provider errors, an open circuit, and the provider-rate-limit signal.

1. Primary provider fails — error, timeout, or circuit breaker open.
2. Candidate resolution — this depends on whether the request matched a route:
   • The request matched a route. It fails over only along that route's own fallbacks list, in order, and nowhere else. A matched route with no fallbacks does not fail over: the caller gets the primary's error.
   • The request matched no route. The default resolver checks registered providers that serve the same model, leaving out the failed provider.
3. Capability filter — a target must serve the model it is asked for and support the requested response format, image input, and tool capabilities. Streaming tool calls need streaming-tool support.
4. Retry with fallback — the request is sent to the next available capable target.

Route-scoped fallback chain

A route names its recovery chain in fallbacks: up to 5 targets, each a provider and, optionally, the model to ask that provider for. Each target is tried as a fresh request. With no model, the target gets the request's own model, which suits a provider serving the same model (OpenAI and Azure OpenAI). A backup in another model family must name its model; the gateway never guesses one.

# gateway.yaml
routes:
  - id: chat
    model: "gpt-4o"
    provider: openai
    fallbacks:
      - provider: azure-openai            # same model name
      - provider: anthropic
        model: claude-sonnet-4-5          # cross-family backup names its model

The older single fallback: <provider> field is read as a one-target chain with the request's own model.

A target is skipped when its provider is not configured, does not serve the model, cannot take what the request needs, is paused by its circuit breaker, or a governance check refuses it. Two limits bound the chain:

Setting	Default	What it does
dvara.llm-gateway.resilience.fallback.max-attempts	3	Most fallback targets one request tries after its primary fails. Skipped targets do not count.
dvara.llm-gateway.resilience.fallback.deadline	60s	How long after the primary's first attempt a fallback may still start. Targets left when it passes are not tried.

Unhealthy candidates are skipped. If every eligible target is unhealthy or fails, the original primary error is returned. Without a route chain, the request keeps its resolved model name: a failed gpt-4o request is not rewritten to a Claude or Gemini model. Registering several providers alone does not guarantee a compatible alternative exists.

For streaming, fallback covers errors raised while opening the upstream stream. Once the stream iterator has been returned, errors while reading it do not trigger provider switching, even if the caller has not received a token yet.

Failover with Capability Matching

Model support is checked before capability matching. Supporting JSON schema does not make a provider eligible to receive a different provider's model name.

The error depends on where candidates are rejected. An empty result from the default resolver returns the original provider error. If candidates exist but all fail the response-format or tool-capability checks (on a route chain: if no target was tried and at least one was skipped for capability), the response is HTTP 503 with failover_capability_mismatch and the X-Gateway-Failover-Blocked: capability_mismatch header. Do not rely on that specific error for every case where no suitable fallback exists.

Disabling failover

Automatic failover is on by default. To keep the resilience wrappers (retry, circuit breaker, timeouts) but never attempt a fallback on failure, set:

dvara:
  llm-gateway:
    resilience:
      fallback:
        enabled: false

Or via environment variable:

DVARA_LLM_GATEWAY_RESILIENCE_FALLBACK_ENABLED=false
DVARA_LLM_GATEWAY_RESILIENCE_FALLBACK_MAX_ATTEMPTS=3
DVARA_LLM_GATEWAY_RESILIENCE_FALLBACK_DEADLINE=60s

With fallback.enabled: false, a failed primary provider returns the error straight to the client — no alternative provider is tried, on a route chain or otherwise.

Upstream provider rate-limit shedding

Upstream providers enforce their own rate limits. OpenAI publishes x-ratelimit-remaining-requests and x-ratelimit-remaining-tokens; Anthropic publishes anthropic-ratelimit-*; providers can also return Retry-After with HTTP 429. Left unhandled, a burst that exhausts a shared upstream key can affect every workspace using that key.

Enterprise only

Not included in DVARA Open Source.

The provider rate-limit tracker reads those headers from responses sent by credential-based providers and, on the next request, decides whether to forward, shed, or fail over before the call leaves the Gateway. Ollama and Bedrock do not use this tracker. The feature is off by default:

dvara:
  llm-gateway:
    provider-rate-limit:
      enabled: true            # master switch; when false the tracker short-circuits to "allow"
      token-headroom-pct: 15   # slack reserved over the estimated input tokens (token estimation is approximate)

DVARA_LLM_GATEWAY_PROVIDER_RATE_LIMIT_ENABLED=true
DVARA_LLM_GATEWAY_PROVIDER_RATE_LIMIT_TOKEN_HEADROOM_PCT=15

Keyed per credential, not per workspace

An upstream rate limit belongs to the API key that calls the provider, not to the workspace making the request. So the tracker keys its budget state on a salted hash of the resolved upstream credential — never the raw key, and never the workspace id:

• A shared platform key used by many workspaces is tracked as one budget window — the tracker sheds across all of them together, which is exactly what the upstream sees.
• A workspace's own key (BYOK) is an isolated window — one workspace burning down their own key never sheds another workspace's traffic.

In the Enterprise distribution, the provider-budget state uses the embedded cluster, so a budget spent on pod A is visible to pod B. This is separate from the optional external shared-cache rate limiter.

Tier-aware shedding

When a credential's remaining budget runs low, the tracker doesn't shed everyone at once — it sheds by workspace priority tier so bulk traffic yields the last of a shared key's budget to premium traffic. The tier is the same priority-tier workspace setting that drives priority admission — there's no separate knob. Each tier is shed once the remaining-budget fraction drops to or below its threshold:

dvara:
  llm-gateway:
    provider-rate-limit:
      tier:
        bulk-threshold-pct: 50      # bulk traffic sheds first — at 50% remaining
        standard-threshold-pct: 20  # standard sheds at 20% remaining
        premium-threshold-pct: 10   # premium retains the last ~10% of a shared key's budget

DVARA_LLM_GATEWAY_PROVIDER_RATE_LIMIT_TIER_BULK_THRESHOLD_PCT=50
DVARA_LLM_GATEWAY_PROVIDER_RATE_LIMIT_TIER_STANDARD_THRESHOLD_PCT=20
DVARA_LLM_GATEWAY_PROVIDER_RATE_LIMIT_TIER_PREMIUM_THRESHOLD_PCT=10

A workspace with no priority-tier set is treated as standard.

Workspace tiers apply without priority admission

Provider shedding uses the workspace's tier whether or not priority admission is on. With dvara.llm-gateway.routing.priority.enabled on, priority admission sets the tier when it admits the request. With it off, the gateway reads the tier from workspace metadata.

What a shed looks like

When the tracker decides to shed a request, the gateway tries the same automatic failover path: the matched route's fallbacks list, or compatible registered providers when no route matched. Only when no eligible alternative succeeds does the client see a terminal response:

HTTP/1.1 429 Too Many Requests
Retry-After: 12

with error code provider_rate_limited. The Retry-After value is derived from the upstream's own signal (its Retry-After header, or the window reset it advertised), so a client's backoff aligns with when the upstream budget actually refreshes.

Fail-open by design

The tracker never causes an outage. A cold start (no headers observed yet), the feature disabled, or any internal tracker error all resolve to allow — the request is forwarded normally. Shedding only engages once the Gateway has seen a fresh budget signal from the upstream and knows the remaining fraction has crossed a tier threshold. This fail-open rule is specific to the provider-budget tracker; other governance controls use their own documented failure action.

Not the same thing as the per-key rate limiter

This is distinct from the per-key rate limiter, which caps what a client may send to the gateway. This tracker governs what the gateway forwards to an upstream provider, reacting to that provider's own published limits. The two run independently — a request can pass the client limiter and still be shed here if the upstream key is exhausted.

Change settings in Flightdeck

From 1.10.0, open Traffic → Resilience in Flightdeck as a platform owner or policy-admin. The page shows each setting's effective value and where it comes from: set on the page, or the gateway's environment. It has the global retry, circuit-breaker, timeout and failover settings, and an override row per provider.

A value set on the page wins; an empty field keeps the gateway's environment value, including the environment's own per-provider overrides. Clearing every field puts the environment's settings back.

A save is a new version, audited as RESILIENCE_UPDATED. The page keeps the history, shows what changed between versions, and rolls back to any of them; a rollback is itself a new version, audited as RESILIENCE_ROLLED_BACK. Two people editing at once cannot overwrite each other: a save from an out-of-date page is refused.

The settings reach every gateway on the config bundle, usually within seconds:

• The next call to each provider uses the new values. A call already running keeps the values it started with.
• A provider's circuit breaker keeps its state when only its retry or timeout settings change. Changing the breaker's own settings starts it fresh.
• Failover reads its switch, attempt limit and deadline on each failover.

Each value has bounds, and the page refuses one outside them:

Setting	Allowed
Retry attempts	1 to 10
First backoff	0 to 60,000 ms
Backoff multiplier	1 to 10
Longest backoff	0 to 300,000 ms
Failure rate to open	1 to 100 %
Calls in the window, and calls before it can open	1 to 1,000
Time open	1 second to 1 hour
Trial calls when half open	1 to 100
Call timeout, and stream open timeout	100 to 900,000 ms
Failover attempts	0 to 10
Failover deadline	1 second to 10 minutes

Two settings stay in the environment and need a restart:

• Turning resilience on or off (dvara.llm-gateway.resilience.enabled).
• A stream's whole lifetime (streaming-timeout-ms). The page's streaming field sets only how long a stream may take to open; a value above the gateway's lifetime never takes effect.

Per-provider overrides

Every field on retry, circuit-breaker, and timeout can be overridden per provider under dvara.llm-gateway.resilience.providers.<name>. Anything you don't override inherits the top-level default. Use this to give a slow upstream a longer chat timeout, or a flaky upstream a more permissive circuit breaker, without changing the baseline for the rest of the pool:

dvara:
  llm-gateway:
    resilience:
      retry:
        max-attempts: 3
        initial-backoff-ms: 500
      circuit-breaker:
        failure-rate-threshold: 50
        sliding-window-size: 10

      providers:
        bedrock:
          timeout:
            chat-timeout-ms: 60000      # Bedrock can be slow on first call from a cold SigV4 signer
        ollama:
          retry:
            max-attempts: 1             # no point retrying a local Ollama — fail fast to the fallback
          circuit-breaker:
            failure-rate-threshold: 80  # Ollama flaps more in dev, don't trip the breaker as eagerly

The override keys are the same provider names the Providers page lists — openai, anthropic, gemini, bedrock, azure-openai, mistral, cohere, groq, qwen, deepseek, moonshot, chatglm, grok, ollama. (mock is accepted but ignored, because the mock provider has no resilience wrapper.) An unrecognized provider name is silently ignored at startup, so double-check spelling if your override appears not to apply.

Per-provider overrides require a config file, not environment variables

Spring Boot's environment-variable binding can't hydrate a Map<String, ...> of nested objects, so DVARA_LLM_GATEWAY_RESILIENCE_PROVIDERS_OPENAI_TIMEOUT_CHAT_TIMEOUT_MS does not work — it parses but is silently dropped. Use application.yml for per-provider overrides, or pass the structured config via SPRING_APPLICATION_JSON:

SPRING_APPLICATION_JSON='{"dvara":{"llm-gateway":{"resilience":{"providers":{"openai":{"timeout":{"chatTimeoutMs":60000}}}}}}}'

The top-level keys (retry.*, circuit-breaker.*, timeout.*, fallback.*) all support direct env-var binding as shown in the per-section examples above.

Error Responses

When the resilience layer surfaces a failure to the client — upstream provider error, open circuit breaker, or a failover blocked because no capable fallback exists — the response shape and the full HTTP-status / error-code mapping are documented once on Error Handling. The same page covers retry-after semantics, the X-Gateway-Failover-Blocked header, and example error bodies.

Full configuration example

Every resilience knob on the gateway, with defaults matching what you get out of the box — safe to paste into application.yml without changing anything and use it as a starting point for tuning:

dvara:
  llm-gateway:
    resilience:
      enabled: true                                 # master switch; set false only in tests
      retry:
        max-attempts: 3
        initial-backoff-ms: 500
        backoff-multiplier: 2.0
        max-backoff-ms: 10000
      circuit-breaker:
        failure-rate-threshold: 50
        sliding-window-size: 10
        minimum-number-of-calls: 5
        wait-duration-in-open-state-ms: 30000
        permitted-calls-in-half-open: 3
      timeout:
        chat-timeout-ms: 30000
        streaming-timeout-ms: 120000
      fallback:
        enabled: true
        max-attempts: 3
        deadline: 60s
      providers: {}                                 # per-provider overrides, see "Per-provider overrides" above
    provider-rate-limit:
      enabled: false                                # off by default; reacts to upstream provider rate-limit headers
      token-headroom-pct: 15
      tier:
        bulk-threshold-pct: 50
        standard-threshold-pct: 20
        premium-threshold-pct: 10