Resilience and failover

DVARA wraps every provider with retry, circuit breaker, and timeout protection. When a provider fails, the gateway automatically attempts fallback providers.

Routing chooses the primary provider. Failover is a separate recovery path and does not stay inside the matched route. Read Routing and load balancing for primary selection and use this page as the source of truth for what happens after a provider failure.

Configuration is static at startup

Every value on this page is read once at boot from application.yml (or the matching environment variables). Changing any value requires a deploy roll — there is no Flightdeck UI and no live-reload via the configuration-refresh pipeline that powers routes and policies. The values are baked into each provider's wrappers when the gateway starts and stay there for the life of the pod.

Retry with Exponential Backoff

Failed requests are automatically retried up to a configurable number of attempts. The wait between attempts starts at initial-backoff-ms, multiplies by backoff-multiplier on each retry, and is capped at max-backoff-ms:

dvara:
  llm-gateway:
    resilience:
      retry:
        max-attempts: 3          # total attempts including the first try
        initial-backoff-ms: 500  # wait before the first retry
        backoff-multiplier: 2.0  # delay grows: 500ms → 1s → 2s → …
        max-backoff-ms: 10000    # capped so a long chain can't explode

Or via environment variables:

DVARA_LLM_GATEWAY_RESILIENCE_RETRY_MAX_ATTEMPTS=3
DVARA_LLM_GATEWAY_RESILIENCE_RETRY_INITIAL_BACKOFF_MS=500
DVARA_LLM_GATEWAY_RESILIENCE_RETRY_BACKOFF_MULTIPLIER=2.0
DVARA_LLM_GATEWAY_RESILIENCE_RETRY_MAX_BACKOFF_MS=10000

Retry follows what the response says about provider health. A provider 429 is retried with backoff. A 408, a 5xx, or a connection failure is retried and counts against the circuit breaker. Other provider 4xx responses are returned immediately and do not count against the shared circuit. WORKSPACE_CREDENTIAL_REQUIRED and UNSUPPORTED_CAPABILITY never count against it. A timed-out call is interrupted rather than continuing in the background.

This changes upgrade behaviour: a bad credential or a request the provider refuses no longer opens that provider's circuit for every workspace.

Circuit Breaker

Each provider has an independent circuit breaker that prevents cascading failures:

dvara:
  llm-gateway:
    resilience:
      circuit-breaker:
        failure-rate-threshold: 50              # percentage of failing calls in the window that opens the breaker
        sliding-window-size: 10                 # number of calls the breaker averages over
        minimum-number-of-calls: 5              # minimum calls in the window before the breaker considers tripping — suppresses early-startup noise
        wait-duration-in-open-state-ms: 30000   # time the breaker stays OPEN before probing upstream again
        permitted-calls-in-half-open: 3         # number of probe calls allowed in HALF_OPEN — if all succeed, the breaker closes

Or via environment variables:

DVARA_LLM_GATEWAY_RESILIENCE_CIRCUIT_BREAKER_FAILURE_RATE_THRESHOLD=50
DVARA_LLM_GATEWAY_RESILIENCE_CIRCUIT_BREAKER_SLIDING_WINDOW_SIZE=10
DVARA_LLM_GATEWAY_RESILIENCE_CIRCUIT_BREAKER_MINIMUM_NUMBER_OF_CALLS=5
DVARA_LLM_GATEWAY_RESILIENCE_CIRCUIT_BREAKER_WAIT_DURATION_IN_OPEN_STATE_MS=30000
DVARA_LLM_GATEWAY_RESILIENCE_CIRCUIT_BREAKER_PERMITTED_CALLS_IN_HALF_OPEN=3

State Transitions

CLOSED ──(failure rate ≥ 50%)──▶ OPEN ──(30s wait)──▶ HALF_OPEN ──(test calls)──▶ CLOSED
                                  │                                     │
                                  │                                     ▼
                                  ◀────────────(still failing)──────── OPEN

State	Behavior
CLOSED	Normal operation. Tracks success/failure in a sliding window.
OPEN	All requests fail immediately with PROVIDER_CIRCUIT_OPEN (503). No upstream calls.
HALF_OPEN	A limited number of test calls are allowed through. If successful, circuit closes. If not, circuit re-opens.

Mock Provider Exclusion

The mock provider is excluded from circuit breaker and retry wrapping. Simulated errors injected via error-rate are intended for testing client-side error handling and will never trip the circuit breaker or trigger retries. The mock provider is always considered healthy.

Timeout Configuration

Separate timeouts for non-streaming and streaming requests:

dvara:
  llm-gateway:
    resilience:
      timeout:
        chat-timeout-ms: 30000        # 30 seconds for non-streaming
        streaming-timeout-ms: 120000  # 120 seconds for streaming

Or via environment variables:

DVARA_LLM_GATEWAY_RESILIENCE_TIMEOUT_CHAT_TIMEOUT_MS=30000
DVARA_LLM_GATEWAY_RESILIENCE_TIMEOUT_STREAMING_TIMEOUT_MS=120000

For a non-streaming call, the timeout can trigger fallback to a compatible registered provider when fallback is enabled and a candidate is available. Streaming has two timeout boundaries described below.

Streaming and the resilience layer

The resilience wrapper governs the attempt to open a provider stream. A separate controller lifetime bounds the complete SSE response. The boundary matters when you tune streaming:

• Retry covers the attempt to obtain the provider's stream iterator. If that attempt fails, a configured retry can run. Once DVARA receives the iterator and starts reading, a later failure does not trigger a retry.
• Circuit breaker uses the provider outcome rules above. A stream that ends without a finish reason or carries an in-band provider error is PROVIDER_ERROR; do not treat the partial answer as complete.
• Fallback is attempted only when opening the provider stream fails before an iterator is returned. After reading begins, DVARA does not reissue the prompt to a different provider or concatenate partial responses.
• Timeout has two scopes. The global streaming-timeout-ms value, 120 seconds by default, bounds the complete client-visible SSE lifetime. A provider-specific override bounds only the attempt to open that provider stream. Neither value is an idle timer that resets after each event.

Size the global value for the longest complete response your application accepts. Use provider-specific values to stop waiting for a provider that cannot open a stream in time. On global timeout or client cancellation, DVARA closes delivery and releases the upstream transport. See SSE streaming for the post-header error and partial-evidence behavior.

Automatic Failover

With resilience and fallback enabled (both default to true), eligible failures on live chat calls can trigger another provider attempt. This includes provider errors, an open circuit, and the provider-rate-limit signal.

1. Primary provider fails — error, timeout, or circuit breaker open
2. Candidate resolution — the default resolver checks registered providers that support the same model request, excluding the failed provider. It does not restrict candidates to the matched route or read its named fallback.
3. Capability filter — alternatives must support the requested response format, image input, and tool capabilities. Streaming tool calls need streaming-tool support.
4. Retry with fallback — the request is sent to the next available capable candidate

Route fallback fields do not control this resolver in 1.8

The 1.8 resolver does not read the matched route, its provider order, region entries, or a named fallback value. A route can choose the primary provider, but it cannot define an ordered or route-scoped recovery chain. The fallback attempt keeps the same resolved model identifier; DVARA does not translate an OpenAI or Anthropic model name into a Bedrock model ID.

Ordered route-scoped fallback and explicit destination-model mapping are tracked in dvarahq/dvara#7. They are not 1.8 capabilities.

Unhealthy candidates are skipped. If every eligible alternative is unhealthy or fails, the original primary error is returned. The request keeps its resolved model name: a failed gpt-4o request is not automatically rewritten to a Claude or Gemini model. Registering several providers alone does not guarantee a compatible alternative exists.

For streaming, fallback covers errors raised while opening the upstream stream. Once the stream iterator has been returned, errors while reading it do not trigger provider switching, even if the caller has not received a token yet.

Failover with Capability Matching

Model support is checked before capability matching. Supporting JSON schema does not make a provider eligible to receive a different provider's model name.

The error depends on where candidates are rejected. An empty result from the default resolver returns the original provider error. If candidates reach the dispatcher but all fail its response-format or tool-capability checks, the response is HTTP 503 with failover_capability_mismatch and the X-Gateway-Failover-Blocked: capability_mismatch header. Do not rely on that specific error for every case where no suitable fallback exists.

Disabling failover

Automatic failover is on by default. To keep the resilience wrappers (retry, circuit breaker, timeouts) but never attempt a fallback on failure, set:

dvara:
  llm-gateway:
    resilience:
      fallback:
        enabled: false

Or via environment variable:

DVARA_LLM_GATEWAY_RESILIENCE_FALLBACK_ENABLED=false

With fallback.enabled: false, a failed primary provider returns the error straight to the client — no alternative provider is tried.

Upstream provider rate-limit shedding

Upstream providers enforce their own rate limits. OpenAI publishes x-ratelimit-remaining-requests and x-ratelimit-remaining-tokens; Anthropic publishes anthropic-ratelimit-*; providers can also return Retry-After with HTTP 429. Left unhandled, a burst that exhausts a shared upstream key can affect every workspace using that key.

Enterprise only

Not included in DVARA Open Source.

The provider rate-limit tracker reads those headers from responses sent by credential-based providers and, on the next request, decides whether to forward, shed, or fail over before the call leaves the Gateway. Ollama and Bedrock do not use this tracker in 1.8. The feature is off by default:

dvara:
  llm-gateway:
    provider-rate-limit:
      enabled: true            # master switch; when false the tracker short-circuits to "allow"
      token-headroom-pct: 15   # slack reserved over the estimated input tokens (token estimation is approximate)

DVARA_LLM_GATEWAY_PROVIDER_RATE_LIMIT_ENABLED=true
DVARA_LLM_GATEWAY_PROVIDER_RATE_LIMIT_TOKEN_HEADROOM_PCT=15

Keyed per credential, not per workspace

An upstream rate limit belongs to the API key that calls the provider, not to the workspace making the request. So the tracker keys its budget state on a salted hash of the resolved upstream credential — never the raw key, and never the workspace id:

• A shared platform key used by many workspaces is tracked as one budget window — the tracker sheds across all of them together, which is exactly what the upstream sees.
• A workspace's own key (BYOK) is an isolated window — one workspace burning down their own key never sheds another workspace's traffic.

In the Enterprise distribution, the provider-budget state uses the embedded cluster, so a budget spent on pod A is visible to pod B. This is separate from the optional external shared-cache rate limiter.

Tier-aware shedding

When a credential's remaining budget runs low, the tracker doesn't shed everyone at once — it sheds by workspace priority tier so bulk traffic yields the last of a shared key's budget to premium traffic. The tier is the same priority-tier workspace setting that drives priority admission — there's no separate knob. Each tier is shed once the remaining-budget fraction drops to or below its threshold:

dvara:
  llm-gateway:
    provider-rate-limit:
      tier:
        bulk-threshold-pct: 50      # bulk traffic sheds first — at 50% remaining
        standard-threshold-pct: 20  # standard sheds at 20% remaining
        premium-threshold-pct: 10   # premium retains the last ~10% of a shared key's budget

DVARA_LLM_GATEWAY_PROVIDER_RATE_LIMIT_TIER_BULK_THRESHOLD_PCT=50
DVARA_LLM_GATEWAY_PROVIDER_RATE_LIMIT_TIER_STANDARD_THRESHOLD_PCT=20
DVARA_LLM_GATEWAY_PROVIDER_RATE_LIMIT_TIER_PREMIUM_THRESHOLD_PCT=10

A workspace with no priority-tier set is treated as standard.

Enable priority admission to preserve workspace tiers in 1.8

In 1.8, provider shedding receives the resolved workspace tier from priority admission. If dvara.llm-gateway.routing.priority.enabled remains false, provider shedding treats every request as standard, even when workspace metadata says premium or bulk. Enable priority admission when tier-aware shedding is required.

What a shed looks like

When the tracker decides to shed a request, the gateway tries the same automatic failover path using compatible registered providers. It does not use a route-scoped fallback list. Only when no eligible alternative succeeds does the client see a terminal response:

HTTP/1.1 429 Too Many Requests
Retry-After: 12

with error code provider_rate_limited. The Retry-After value is derived from the upstream's own signal (its Retry-After header, or the window reset it advertised), so a client's backoff aligns with when the upstream budget actually refreshes.

Fail-open by design

The tracker never causes an outage. A cold start (no headers observed yet), the feature disabled, or any internal tracker error all resolve to allow — the request is forwarded normally. Shedding only engages once the Gateway has seen a fresh budget signal from the upstream and knows the remaining fraction has crossed a tier threshold. This fail-open rule is specific to the provider-budget tracker; other governance controls use their own documented failure action.

Not the same thing as the per-key rate limiter

This is distinct from the per-key rate limiter, which caps what a client may send to the gateway. This tracker governs what the gateway forwards to an upstream provider, reacting to that provider's own published limits. The two run independently — a request can pass the client limiter and still be shed here if the upstream key is exhausted.

Per-provider overrides

Every field on retry, circuit-breaker, and timeout can be overridden per provider under dvara.llm-gateway.resilience.providers.<name>. Anything you don't override inherits the top-level default. Use this to give a slow upstream a longer chat timeout, or a flaky upstream a more permissive circuit breaker, without changing the baseline for the rest of the pool:

dvara:
  llm-gateway:
    resilience:
      retry:
        max-attempts: 3
        initial-backoff-ms: 500
      circuit-breaker:
        failure-rate-threshold: 50
        sliding-window-size: 10

      providers:
        bedrock:
          timeout:
            chat-timeout-ms: 60000      # Bedrock can be slow on first call from a cold SigV4 signer
        ollama:
          retry:
            max-attempts: 1             # no point retrying a local Ollama — fail fast to the fallback
          circuit-breaker:
            failure-rate-threshold: 80  # Ollama flaps more in dev, don't trip the breaker as eagerly

The override keys are the same provider names the Providers page lists — openai, anthropic, gemini, bedrock, azure-openai, mistral, cohere, groq, qwen, deepseek, moonshot, chatglm, grok, ollama, mock. An unrecognized provider name is silently ignored at startup, so double-check spelling if your override appears not to apply.

Per-provider overrides require a config file, not environment variables

Spring Boot's environment-variable binding can't hydrate a Map<String, ...> of nested objects, so DVARA_LLM_GATEWAY_RESILIENCE_PROVIDERS_OPENAI_TIMEOUT_CHAT_TIMEOUT_MS does not work — it parses but is silently dropped. Use application.yml for per-provider overrides, or pass the structured config via SPRING_APPLICATION_JSON:

SPRING_APPLICATION_JSON='{"dvara":{"llm-gateway":{"resilience":{"providers":{"openai":{"timeout":{"chatTimeoutMs":60000}}}}}}}'

The top-level keys (retry.*, circuit-breaker.*, timeout.*, fallback.enabled) all support direct env-var binding as shown in the per-section examples above.

Error Responses

When the resilience layer surfaces a failure to the client — upstream provider error, open circuit breaker, or a failover blocked because no capable fallback exists — the response shape and the full HTTP-status / error-code mapping are documented once on Error Handling. The same page covers retry-after semantics, the X-Gateway-Failover-Blocked header, and example error bodies.

Full configuration example

Every resilience knob on the gateway, with defaults matching what you get out of the box — safe to paste into application.yml without changing anything and use it as a starting point for tuning:

dvara:
  llm-gateway:
    resilience:
      enabled: true                                 # master switch; set false only in tests
      retry:
        max-attempts: 3
        initial-backoff-ms: 500
        backoff-multiplier: 2.0
        max-backoff-ms: 10000
      circuit-breaker:
        failure-rate-threshold: 50
        sliding-window-size: 10
        minimum-number-of-calls: 5
        wait-duration-in-open-state-ms: 30000
        permitted-calls-in-half-open: 3
      timeout:
        chat-timeout-ms: 30000
        streaming-timeout-ms: 120000
      fallback:
        enabled: true
      providers: {}                                 # per-provider overrides, see "Per-provider overrides" above
    provider-rate-limit:
      enabled: false                                # off by default; reacts to upstream provider rate-limit headers
      token-headroom-pct: 15
      tier:
        bulk-threshold-pct: 50
        standard-threshold-pct: 20
        premium-threshold-pct: 10