Routing and load balancing

Check the distribution

The Open Source distribution supports model-prefix, round-robin, weighted, and canary routing from gateway.yaml. The Enterprise platform adds managed latency-, cost-, geo-, and intelligent routing.

DVARA decides which provider handles each LLM request by walking a configurable list of routes, matching the request's model field against each route's pattern, and applying the route's strategy to pick a provider from its pool. By default, with no routes configured, the gateway falls back to matching on the model-name prefix — gpt-* goes to OpenAI, claude-* to Anthropic, gemini-* to Google, and so on. The Open Source distribution reads routes from gateway.yaml at startup. The Enterprise platform stores them centrally and propagates updates across gateway nodes without a restart.

This page starts with the Open Source distribution's local strategies, then covers Enterprise latency-, cost-, and region-aware routing and intelligent model selection. Each Enterprise-only section carries the standard marker. See Providers for provider-specific behavior and the capability matrix used to filter incompatible providers.

Default routing: model prefix

This is the fallback the gateway uses when no configured route matches. If you've created a route for a model pattern, that route runs first and the table below isn't consulted. Treat this as the bundled set of default behaviours; anything outside the table needs a route. To add a route, see Route configuration.

Model in request	Matched provider
gpt-4o, gpt-4.1, o1-preview, o3-mini, o4-mini, chatgpt-4o-latest	OpenAI
text-embedding-3-small, text-embedding-3-large	OpenAI (embeddings)
claude-sonnet-4-5, claude-3-5-haiku-20241022	Anthropic
gemini-2.0-flash, gemini-1.5-pro	Google Gemini
bedrock/anthropic.claude-3-sonnet-20240229-v1:0	AWS Bedrock
azure/gpt-4o	Azure OpenAI
mistral-large-latest, mistral-small-latest	Mistral
command-r-plus, command-r	Cohere
groq/llama-3.3-70b-versatile	Groq
ollama/llama3.2	Ollama
qwen2.5-72b-instruct, qwen-max	Alibaba Qwen (DashScope)
deepseek-chat, deepseek-reasoner	DeepSeek
moonshot-v1-128k, moonshot-v1-32k	Moonshot (Kimi)
glm-4, glm-4-air, glm-4v-plus	Zhipu ChatGLM
grok-2-1212, grok-2-vision-1212, grok-3-latest	xAI Grok
mock/test-model	Mock

OpenAI claims three prefix families on the chat path — classic gpt-*, the o-series reasoning models (o1-*, o3-*, o4-*), and the chatgpt-* alias models. Embeddings route under text-embedding-* separately. All four route to the same provider and the same OPENAI_API_KEY.

Models from OpenAI-compatible providers that aren't first-class — ERNIE (Baidu), Fireworks, Together AI, Perplexity, vLLM hosts, internal corporate gateways — aren't routed by default prefix matching. Configure a route or override an existing provider's base URL. See Additional providers.

Each provider registers at startup only if its credential trigger is set (OPENAI_API_KEY, ANTHROPIC_API_KEY, etc.; dvara.llm-gateway.providers.ollama.enabled=true for Ollama; BEDROCK_ENABLED=true for Bedrock — note Bedrock registers on the flag alone, and AWS credentials are checked per request by the SigV4 signer, not at startup). If no provider is registered for the matched prefix, the gateway returns HTTP 400 no_provider with an error message naming the environment variable to set. Per-workspace-scoped Flightdeck pages credentials override the startup credential per request but do not register a provider on their own — set a placeholder startup value if you plan to onboard workspaces who bring their own keys. See Credentials & BYOK.

Example request:

curl -X POST http://localhost:8080/v1/chat/completions \
  -H 'Authorization: Bearer $DVARA_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "claude-sonnet-4-5",
    "messages": [{"role": "user", "content": "Summarize this PR description in two bullets."}]
  }'

claude-sonnet-4-5 has no custom route, so the default prefix matcher runs. The claude prefix resolves to Anthropic and the gateway forwards the request upstream. You can confirm which provider handled the call by grepping the access log for the X-Trace-ID that came back on the response — the provider field of the structured JSON line shows anthropic.

Routing strategies

DVARA supports eight strategies. Pick one per route based on how you want traffic distributed across your provider pool. For when to choose each, see the multi-provider LLM routing guide.

The eight strategies:

Strategy	Description
model-prefix	Default. First provider whose prefix matches the model.
round-robin	Distributes requests evenly across listed providers in rotation.
weighted	Distributes requests by configured relative weights.
canary	Splits traffic between a baseline and candidate provider for A/B testing.
latency-aware	Routes to the lowest-latency healthy provider using a running EWMA of response times. Enterprise only.
cost-aware	Routes to the cheapest provider, optionally constrained by a latency SLA. Enterprise only.
geo-aware	Filters providers by region and prefers the gateway node's own region, subject to workspace data-residency constraints. Enterprise only.
intelligent	Classifies request complexity and picks the cheapest model tier that can handle it. Enterprise only.

Shadow traffic (Shadow Routing) is a property of any strategy above, not a strategy itself — it sends a copy of traffic to a secondary provider without affecting the client response, for side-by-side comparison.

Priority admission control (SLA-Aware Priority Routing) is independent of strategy selection — it governs whether a request is admitted at all, based on per-workspace priority tier under load.

Route configuration

In the Enterprise platform, create and update routes through DVARA Flightdeck at /routes. Changes propagate across Gateway nodes on the configured refresh interval (see Hot reload). From 1.8.4 Flightdeck keeps one route per model pattern: a second route for a pattern that already has one is refused on save, on rollback and on import. See Routes in the Flightdeck.

The dvara.llm-gateway.routes block is static startup configuration. It is useful for DVARA Open Source and simple single-node deployments. Editing the file requires a Gateway restart. In Enterprise, use Flightdeck for live route changes instead of maintaining the same route in both places.

Example YAML (static)

dvara:
  llm-gateway:
    routes:
      - id: load-balance-gpt
        model-pattern: "gpt*"
        strategy: round-robin
        providers:
          - provider: openai
          - provider: azure-openai

      - id: weighted-gpt
        model-pattern: "gpt*"
        strategy: weighted
        providers:
          - provider: openai
            weight: 70
          - provider: azure-openai
            weight: 30

Route fields

Field	Type	Required	Description
id	string	yes	Unique route identifier
model-pattern	string	yes	Prefix pattern. A trailing * matches any remainder (gpt* matches gpt-4o, gpt-4o-mini, gpt-5). Anything else is treated as a literal exact match — there is no support for ?, **, or regex.
strategy	string	no	One of model-prefix (default), round-robin, weighted, or canary in Open Source. Enterprise also accepts latency-aware, cost-aware, geo-aware, and intelligent.
model-tiers	map	no	Complexity → model mapping for intelligent routing (e.g., {SIMPLE: "gpt-4o-mini", MODERATE: "gpt-4o", COMPLEX: "claude-3-opus"})
cost-tolerance-pct	int	no	Cost tolerance percentage for latency/cost-aware routing (0–100, default: 0)
latency-sla-ms	long	no	Latency SLA in ms for cost-aware routing (providers exceeding the SLA are excluded; default: 0 = no SLA)
canary-config	object	no	Canary A/B test configuration (see Canary Routing)
shadow-config	object	no	Enterprise shadow-traffic configuration for parallel testing. Open Source refuses this block because it does not dispatch shadow traffic.
providers	list	yes*	Provider pool for this route (*required for every strategy except model-prefix)
providers[].provider	string	yes	Provider name: openai, anthropic, gemini, bedrock, azure-openai, mistral, cohere, groq, ollama, qwen, deepseek, moonshot, chatglm, grok, or mock
providers[].weight	int	no	Weight for weighted routing (default: 1)
providers[].region	string	no	Region affinity for this provider entry (used by geo-aware routing)
pinned-model-version	string	no	Present in configuration, but does not change the upstream request in 1.8. Do not rely on it for version pinning.

Every provider receives the same model identifier

A route chooses a provider; it does not map the request to a different model name for each provider. Every provider in a route must therefore accept the same request model. For example, an OpenAI and Azure OpenAI route works only when the Azure deployment name matches the model name DVARA sends. A Bedrock model ID is not interchangeable with an OpenAI or Anthropic model name. In 1.8, pinned-model-version does not change the model sent upstream, so it cannot be used as a route-wide or per-provider mapping.

Route-scoped provider/model fallback is tracked in dvarahq/dvara#7. Treat that issue as future work, not as configuration available in 1.8.

How a request reaches a provider

Every POST /v1/chat/completions follows four selection stages before the upstream call. A fifth, separate recovery stage can run after that call fails:

1. Match a route          → which route's pattern fits the model?
2. Filter by capability   → can each provider honor the requested format and tools?
3. Run the strategy       → which provider in the filtered pool wins?
4. Call the provider      → send the original request model upstream
5. Recover after failure  → try a compatible registered provider

Stages 1–4 form primary selection and dispatch. Stage 2 narrows the pool when the request needs structured output, JSON mode, function calling, or streamed function calling. Stage 5 runs only after an eligible provider failure and is not constrained to the matched route.

The rest of this section walks each stage, then follows one request through the primary path and explains when recovery would begin.

1. Match a route

The gateway walks the configured routes top-to-bottom and stops at the first one whose model-pattern matches the request's model. If nothing matches, the built-in model-prefix fallback picks a provider from the registered prefix table. Embeddings use their own embedding-capable provider selection rather than this chat-routing sequence.

Example. Two routes are configured: gpt-canary (pattern gpt-4o-mini, canary strategy) and gpt-default (pattern gpt*, round-robin). A request for gpt-4o-mini matches gpt-canary first and stops there. A request for gpt-4o walks past gpt-canary (pattern doesn't match — gpt-4o-mini is a literal exact match, not a prefix) and lands on gpt-default. A request for mistral-large-latest matches neither route, so the prefix fallback runs — the mistral prefix resolves to the Mistral provider.

2. Filter by capability

The route's provider pool is narrowed when the request needs structured output, JSON mode, function calling, or streamed function calling. Stage 3 runs against that filtered route pool. If the selected provider later fails, the resilience path performs its own capability checks across compatible registered providers.

response_format	Required capability
json_schema	Structured outputs
json_object	JSON mode
text / absent	No filtering

A request that offers tools, or replays an assistant tool call in its history, requires native tool-call support. When that request also streams, the provider must declare streamed-tool-call support. Vision is not a primary-route filter because its provider declarations can cover a family containing both vision and text-only models. Choose a known vision model explicitly; DVARA uses the vision declaration when considering failover.

If no provider on the route can honor the requested format or tool contract, DVARA returns HTTP 400 no_capable_provider before an upstream call. See Capability-aware filtering for the full rules.

Example. A route's pool is [openai, cohere]. A request with response_format: {"type": "json_schema", ...} filters Cohere out — Cohere doesn't support structured outputs — leaving only OpenAI for the strategy to pick from. A request with response_format: {"type": "text"}, or no response_format at all, leaves both providers in the pool.

3. Run the strategy

The route's strategy runs against the filtered pool and picks exactly one provider:

• round-robin rotates through the pool one request at a time
• weighted rolls a uniform random against the configured weights
• latency-aware picks the lowest current EWMA latency
• cost-aware picks the cheapest, optionally bounded by a latency SLA
• canary splits traffic on a configured percentage between baseline and candidate
• geo-aware filters by data-residency policy and prefers same-region providers
• intelligent classifies the request's complexity and picks the cheapest tier that fits
• model-prefix is the default — first provider whose registered prefix matches

Each strategy's payload semantics are documented in its own section below.

Example. Route gpt-weighted has [openai weight 70, azure-openai weight 30]. The strategy rolls a number in [0, 100): if it lands under 70, OpenAI wins; otherwise Azure OpenAI wins. This example assumes the Azure deployment has the same name as the model in the request.

4. Call the provider

DVARA sends the request's model identifier to the selected provider. Azure OpenAI uses it as the deployment name. Bedrock removes only a leading bedrock/ routing prefix before sending the remaining Bedrock model ID. DVARA 1.8 does not translate between provider model namespaces.

5. Recover after failure

If the chosen provider returns provider_error, provider_circuit_open, or provider_rate_limited, the resilience layer asks every registered provider whether it supports the resolved request, excluding the failed provider. It then applies response-format, image, and tool-capability checks and skips unhealthy candidates. The resolver does not read the matched route, its region entries, or its named fallback value. It also preserves the resolved model name rather than translating it for another provider. See Resilience for retry budgets and circuit-breaker tuning.

Example. The strategy picked OpenAI in stage 4. OpenAI's circuit breaker has just opened due to a string of 5xxs. DVARA checks other registered providers for one that accepts the same model name and requested capabilities. A provider listed on the route is not guaranteed to be the next attempt, and a compatible provider outside the route can be considered. If no resolver candidate exists, the original provider error is returned. HTTP 503 failover_capability_mismatch is returned only when fallback candidates exist but the dispatcher's response-format or tool checks reject all of them.

End-to-end: route one request and recover if needed

A route configured like this:

dvara:
  llm-gateway:
    routes:
      - id: gpt-split
        model-pattern: "gpt-4o-mini"
        strategy: weighted
        providers:
          - provider: openai
            weight: 70
          - provider: azure-openai
            weight: 30

And a client sending:

curl -X POST http://localhost:8080/v1/chat/completions \
  -H 'Authorization: Bearer $DVARA_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "gpt-4o-mini",
    "response_format": {"type": "json_object"},
    "messages": [{"role": "user", "content": "Reply with a JSON object containing keys ok and ts."}]
  }'

What the gateway does:

1. Match. gpt-4o-mini matches the literal route pattern, so DVARA selects gpt-split.
2. Filter. response_format: json_object requires JSON mode. Both configured providers declare it.
3. Select. The weighted roll lands at 0.42, so OpenAI wins.
4. Call. DVARA sends gpt-4o-mini to OpenAI.
5. Recover if needed. OpenAI returns the JSON object, so fallback does not run.

This configuration assumes the Azure deployment is named gpt-4o-mini. The audit and request metrics record the request model and selected provider. If OpenAI fails, automatic fallback is governed by the registered-provider rules on Resilience, not by the 70/30 route list.

Round-robin example

Distribute GPT requests evenly between OpenAI and an Azure OpenAI deployment that uses the same model name:

dvara:
  llm-gateway:
    routes:
      - id: gpt-round-robin
        model-pattern: "gpt*"
        strategy: round-robin
        providers:
          - provider: openai
          - provider: azure-openai

Requests cycle through OpenAI and Azure OpenAI. The counter is per-route, not global — two different routes with round-robin strategy rotate independently.

Example request:

curl -X POST http://localhost:8080/v1/chat/completions \
  -H 'Authorization: Bearer $DVARA_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

gpt-4o-mini matches gpt* on the gpt-round-robin route. The strategy alternates between the two providers, provided the Azure deployment is also named gpt-4o-mini. The counter advances by one per selection. Before the strategy runs, DVARA removes providers whose circuit is unavailable. If every capable provider is unavailable, it passes the full capable set to the strategy so the normal provider error is preserved. If a selected call then fails, the separate failover layer evaluates compatible registered providers.

Weighted routing example

Send 70% of GPT traffic to OpenAI and 30% to an identically named Azure OpenAI deployment:

dvara:
  llm-gateway:
    routes:
      - id: gpt-weighted
        model-pattern: "gpt*"
        strategy: weighted
        providers:
          - provider: openai
            weight: 70
          - provider: azure-openai
            weight: 30

Weights are relative, not percentages — weight: 7 and weight: 3 behave identically to 70/30. A provider with weight: 0 is included in the pool but never selected, as long as at least one other provider on the route has a positive weight. A weighted route where every provider has weight: 0 is rejected at construction time. DVARA filters unavailable providers before weighted selection unless every capable provider is unavailable.

Example request:

curl -X POST http://localhost:8080/v1/chat/completions \
  -H 'Authorization: Bearer $DVARA_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [{"role": "user", "content": "Write a short release note."}]
  }'

gpt-4o-mini matches gpt*, so the route rolls a per-request uniform random number against the cumulative weight ranges: [0, 70) → OpenAI, [70, 100) → Azure OpenAI. Over thousands of requests the distribution converges to 70/30, but short bursts can drift. If you set OpenAI's weight to 0, the next request skips it and Azure OpenAI receives all routed traffic. The Azure deployment must be named gpt-4o-mini; DVARA does not translate that name.

Model version pinning in 1.8

Do not use pinned-model-version as a production control in 1.8. The value is accepted in route configuration, but the ordinary routing path does not apply it to the request sent upstream. Send the exact provider model or deployment name in the client request until the product defect is fixed and documented in a later release. The correction is tracked in dvarahq/dvara#18.

Capability-aware filtering

Before applying the routing strategy, DVARA filters the provider pool for the response and function-call contract requested by the client:

response_format type	Required capability
json_schema	Structured outputs support
json_object	JSON mode support
text / absent	No filtering applied

Offering tools or replaying a tool call requires native tool-call support. A streaming request with tools also requires streamed-tool-call support. If the filtered primary pool is empty, DVARA returns HTTP 400 no_capable_provider. If a selected provider fails and registered fallback candidates exist but none can carry the required contract, DVARA returns HTTP 503 failover_capability_mismatch with X-Gateway-Failover-Blocked: capability_mismatch.

Vision does not use this primary filter because support can vary between models within one provider family. Use an explicit route containing known vision-capable models and providers. DVARA applies the provider's vision declaration when choosing a fallback, and an incapable provider must refuse an image request itself.

See Structured Outputs for per-provider details on which format mappings are native versus emulated, and the Capabilities Matrix on the Providers page.

Latency-aware routing

Enterprise only

Not included in DVARA Open Source.

DVARA Open Source refuses a route that names this strategy at startup.

The latency-aware strategy tracks live EWMA (exponentially weighted moving average) latency per provider-and-model pair and automatically routes each request to the fastest healthy provider. This is ideal for multi-provider setups where provider performance varies over time — regional congestion, provider-side load, or a new model deployment are all picked up within a handful of samples.

How it works

1. Latency recording — every successful provider call records its wall-clock response time. The running average is computed as ewma = alpha * sample + (1 - alpha) * previous, with alpha = 0.2 by default.
2. Provider selection — for each request, healthy providers are ranked by EWMA. The lowest-latency provider is selected.
3. Cold-start exploration — providers without enough samples (below min-samples, default 5) are bypassed for EWMA selection. A 10% exploration slice still routes to them so the EWMA has data to converge on. When every provider on the route is cold, the strategy falls back to round-robin until samples accumulate.
4. Time decay — stale entries (no samples for more than 60 seconds by default) get a decay penalty so the router does not keep preferring a provider it hasn't measured recently.

Configuration

dvara:
  llm-gateway:
    routes:
      - id: latency-gpt
        model-pattern: "gpt*"
        strategy: latency-aware
        providers:
          - provider: openai
          - provider: azure-openai

cost-tolerance-pct is read by the cost-aware strategy, not latency-aware — if you want latency and cost trade-off together, use cost-aware with a latency-sla-ms cap (Cost-aware routing).

Example request:

curl -X POST http://localhost:8080/v1/chat/completions \
  -H 'Authorization: Bearer $DVARA_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Generate a JSON response."}]
  }'

The strategy pulls the current EWMA latency for gpt-4o on both providers. If OpenAI is averaging 180 ms and Azure OpenAI is averaging 250 ms, OpenAI wins. This assumes the Azure deployment is named gpt-4o. When a fresh provider has fewer than dvara.llm-gateway.routing.latency.min-samples (5 by default), it is considered "cold" and receives a 10% exploration slice. If every provider on the route is cold, the strategy uses round-robin until samples accumulate.

EWMA tuning

These global properties apply to every latency-aware route:

Property	Default	Description
dvara.llm-gateway.routing.latency.alpha	0.2	EWMA smoothing factor (0.0–1.0). Higher = more weight on recent samples
dvara.llm-gateway.routing.latency.decay-threshold-ms	60000	Staleness threshold in ms. Entries older than this get a decay penalty
dvara.llm-gateway.routing.latency.decay-multiplier	0.5	Stale EWMA is divided by this value (lower = harsher penalty)
dvara.llm-gateway.routing.latency.min-samples	5	Minimum samples before EWMA is used for routing decisions
dvara.llm-gateway.routing.latency.snapshot-interval	100	Persist a latency snapshot every N samples

Monitoring latency data

Open the Flightdeck's Latency page (/latency). It lists every tracked (provider, model) pair with its sample count, EWMA latency, raw last sample and a freshness badge, and refreshes on a poll. Use this page to confirm that each provider has enough samples before you judge a routing change.

Cost-aware routing

Enterprise only

Not included in DVARA Open Source.

DVARA Open Source refuses a route that names this strategy at startup.

The cost-aware strategy selects the cheapest provider capable of handling the request, optionally constrained by a latency SLA. Manage per-provider pricing in DVARA Flightdeck — see Cost Management for how prices are added and rotated.

How it works

1. Filter to healthy + configured providers. Providers the resilience layer has marked unavailable are dropped. If the candidate pool is empty after this filter, DVARA returns no_provider.
2. SLA filter. When latency-sla-ms > 0, providers whose current EWMA latency exceeds the SLA are excluded. Providers without enough latency data pass through so they get a chance to prove themselves. If the SLA is so tight that every provider exceeds it, the filter is bypassed and the strategy continues with the full healthy pool — the gateway logs a debug line and picks the cheapest from the original candidates rather than returning no_provider. Set latency-sla-ms to a value you've actually measured under load — too-tight settings degrade silently to "cheapest", not to a hard error.
3. Cost estimation. For every remaining provider, the cost estimator looks up the pricing-table entry and computes estimated per-request cost from the request's estimated input + output token counts.
4. Split by cost-data availability. Providers with a pricing entry go to a warm pool; providers without a pricing entry go to a cold pool. (Note: this is about whether the provider has pricing data, not about how many samples it has — there is no min-samples knob on cost-aware. The latency-aware min-samples property is for the latency-aware strategy only.)
5. All cold → round-robin fallback. If no provider has cost data, the strategy falls back to round-robin so traffic still flows while pricing data accumulates.
6. 5% cold-start exploration. When warm + cold pools both exist, 5% of requests are routed to a cold provider so its cost data can accumulate. The other 95% go to the warm pool. Cold-start providers are not skipped — they're sampled at a small fixed rate.
7. Cheapest with cost-tolerance tiebreak. From the warm pool, the strategy picks the cheapest provider. If cost-tolerance-pct > 0, providers within the tolerance band — cost ≤ minCost × (1 + costTolerancePct/100) — are re-ranked by latency. So cost-tolerance-pct: 15 means "I'll accept anything within 15% of the cheapest; among those, pick the fastest."

dvara:
  llm-gateway:
    routes:
      - id: cost-optimized
        model-pattern: "gpt*"
        strategy: cost-aware
        latency-sla-ms: 500          # exclude providers with >500ms EWMA latency
        cost-tolerance-pct: 15       # within 15% of cheapest, prefer lowest latency
        providers:
          - provider: openai
          - provider: azure-openai

Example request:

curl -X POST http://localhost:8080/v1/chat/completions \
  -H 'Authorization: Bearer $DVARA_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Please analyze this paragraph for tone."}]
  }'

Walk the decision: the strategy estimates cost for each provider from the pricing table. Suppose the request is roughly 400 input tokens and 200 output tokens, and the configured OpenAI price produces a lower estimate than Azure OpenAI. OpenAI wins. If OpenAI's EWMA crosses the 500 ms SLA, it is excluded and Azure OpenAI wins. With cost-tolerance-pct: 15, candidates within 15% of the cheapest are ranked by latency. The Azure deployment must be named gpt-4o for this route.

Canary routing

The canary strategy splits traffic between a baseline and a candidate provider for A/B testing. Use it to validate a new provider on a fraction of real traffic before promoting it to the primary path.

DVARA Open Source loads canary routes and their canary-config from gateway.yaml at startup. Restart the gateway after changing the file. In the Enterprise platform, create and change canary routes in DVARA Flightdeck so the gateway fleet receives the update without a restart.

Configure a canary route

For DVARA Open Source, add the route to gateway.yaml:

dvara:
  llm-gateway:
    routes:
      - id: canary-test
        model-pattern: "gpt*"
        strategy: canary
        providers:
          - provider: openai
          - provider: azure-openai
        canary-config:
          baseline-provider: openai
          candidate-provider: azure-openai
          split-pct: 20
          workspace-scope: workspace-a
          test-name: azure-eval

For the Enterprise platform, open Routes in DVARA Flightdeck, create a canary route, and enter the same baseline, candidate, split, test name, and optional workspace scope.

Field	Type	Required	Description
baseline-provider	string	yes	Provider name for baseline traffic. Must also appear in the route's providers list — see the note below.
candidate-provider	string	yes	Provider name for candidate traffic. Must also appear in the route's providers list.
split-pct	int	yes	Percentage of traffic sent to the candidate (0–100)
workspace-scope	string	no	Restrict the canary to one workspace. Omit it to include every workspace. Requests from other workspaces use the baseline.
test-name	string	no	Human-readable test name shown in the comparison report

Keep providers aligned with both variants

If the variant the strategy picks isn't in the route's providers list, the gateway silently falls back to the other variant instead of erroring out. So if baseline-provider: openai but providers only includes bedrock, every OpenAI decision lands on Bedrock without an error — your 0% split still routes 100% to the candidate. Include both baseline-provider and candidate-provider explicitly in the providers list.

Example request against the canary route:

curl -X POST http://localhost:8080/v1/chat/completions \
  -H 'Authorization: Bearer $WORKSPACE_A_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Summarize this paragraph."}]
  }'

gpt-4o matches gpt* on the canary-test route. Because workspace-scope is workspace-a, only requests whose API key resolves to that workspace enter the split. Other workspaces use OpenAI. Within workspace-a, 20% of requests go to the identically named Azure OpenAI deployment and 80% go to OpenAI. The gateway_canary_requests_total metric counts decisions by route and variant.

Workspace scope comes from the API key

workspace-scope uses the workspace resolved from the API key, not a field in the request body. A client cannot opt itself into or out of a canary by changing the request payload.

Monitoring

Open the route's Canary dashboard (/routes/{id}/canary). It shows the live comparison — request count, error rate, latency, and cost per variant — and carries Reset to start a fresh window and a control to change the split percentage. A split change takes effect on the next request in the Enterprise platform. DVARA Open Source exposes the canary request counter through Prometheus; change its YAML configuration and restart to update the split.

Geo-aware routing

Enterprise only

Not included in DVARA Open Source.

DVARA Open Source refuses a route that names this strategy at startup.

The geo-aware strategy respects data residency policies by filtering providers to only those in allowed regions before selecting one. Use it when regulatory requirements forbid sending data outside a specific region (GDPR, India DPDP, sector-specific rules).

dvara:
  llm-gateway:
    routes:
      - id: geo-gpt
        model-pattern: "gpt*"
        strategy: geo-aware
        providers:
          - provider: openai
            region: us-east-1
          - provider: azure-openai
            region: eu-west-1

Region inputs

Each request is evaluated against two region signals:

• Current region — the calling gateway node's own region, set at gateway startup via DVARA_REGION_ID (or dvara.region.id in application.yml). This is the region the gateway is running in, not the workspace's home region.
• The calling workspace — resolved from the API key. DVARA reads data-residency.allowed-regions from that workspace. The caller supplies no region field in the request body.

Remove the legacy client region field

DVARA 1.8 ignores a client-supplied tenant_region. Remove it from request payloads. The workspace's allowed-region configuration is authoritative.

Allowed-region configuration

Data residency is driven by a per-workspace metadata key, not by route configuration. Set data-residency.allowed-regions on the workspace as a comma-separated list of region ids — only those regions are permitted upstream for that workspace's traffic. Set it from the Flightdeck's workspace form, or as a metadata entry on that workspace in the GitOps export document:

{ "workspaces": [ { "id": "acme-corp", "metadata": {
      "data-residency.allowed-regions": "eu-west-1,eu-central-1"
} } ] }

The residency filter still passes everything in the cases below. The first is the one to check when you expected a 403 and did not get one:

Bypass case	Effect
Workspace has no data-residency.allowed-regions set	All providers pass; the strategy selects purely on same-region preference
Route's provider entries lack region: on one or more entries	Those entries pass the residency filter unconditionally — set region: on every entry in a geo-aware route
The call has no workspace at all — an unauthenticated request, admin tooling, an internal probe	All providers pass. There is no workspace whose residency could be violated, so there is no question to answer

Selection order

For each request, the strategy walks the provider pool in this order:

1. Health filter — drop providers the resilience layer has marked unavailable. If every provider is unhealthy, the strategy falls back to the full pool (including unhealthy ones) rather than failing fast — health is preferred, not strictly enforced. The residency filter in step 2 still runs, so a disallowed region is never silently chosen even when health is degraded.
2. Data-residency filter — per-provider check against the workspace allow-list and provider region. Providers whose region is not in the allow-list are dropped. If this leaves the pool empty, DVARA returns HTTP 403 data_residency_violation and makes no upstream call. Subject to the three bypass cases above.
3. Same-region preference — if DVARA_REGION_ID is set, pick the first residency-compliant provider whose region matches the gateway node's own region (and whose region is currently healthy).
4. Cross-region fallback — no same-region provider? Pick the first residency-compliant provider whose region is currently healthy.
5. Last resort — still nothing? Return the first residency-compliant provider regardless of region health (logs a warning).

Step 2 runs before any provider call, so sensitive data never leaves an allowed region — provided the three bypass cases above are all closed.

Example request:

curl -X POST http://localhost:8080/v1/chat/completions \
  -H 'Authorization: Bearer $DVARA_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Classify this ticket."}]
  }'

Note there is no residency field in the body. The API key identifies the workspace, the gateway stamps it on the request, and the strategy reads that workspace's allow-list. If the workspace behind this key has "eu-west-1,eu-central-1" set, the OpenAI entry in us-east-1 is dropped and the Azure OpenAI entry in eu-west-1 is selected. If the gateway node also runs in eu-west-1, the same-region preference chooses Azure OpenAI when more than one compliant candidate exists. The Azure deployment must accept the same gpt-4o identifier.

Canary workspace scope works the same way: it uses the workspace resolved from the API key, not a value the caller sends.

Intelligent routing

Enterprise only

Not included in DVARA Open Source.

DVARA Open Source refuses a route that names this strategy at startup.

The intelligent strategy classifies request complexity and automatically picks the cheapest model tier that can handle it. Use it when you want a single endpoint that scales from cheap short-form answers to expensive complex reasoning without clients having to pick the right model themselves.

dvara:
  llm-gateway:
    routes:
      - id: auto-tier
        model-pattern: "*"
        strategy: intelligent
        model-tiers:
          SIMPLE: gpt-4o-mini          # factual Q&A, translation
          MODERATE: gpt-4o             # summarization, analysis
          COMPLEX: claude-3-opus       # creative writing, code generation
        providers:
          - provider: openai
          - provider: anthropic

Complexity scoring

The classifier scores each request on these factors:

Factor	Points
1 message	0
2–5 messages	1
6+ messages	2
Non-system message text ≤ 500 chars	0
Non-system message text 501–2000 chars	1
Non-system message text > 2000 chars	2
Tool use present in the conversation	2
System prompt ≤ 500 chars	0
System prompt 501–1500 chars	1
System prompt > 1500 chars	2
Complexity keywords (full list below)	0.5 each, max 2

The character counts on the second row are deliberately for non-system message text (user, assistant, and tool messages — anything except role: "system"); the system prompt is scored on its own row and is intentionally not added in. Boundary values fall in the lower band — exactly 500 chars scores 0, exactly 2000 scores 1, exactly 1500 (system prompt) scores 1.

The complexity-keyword list, in full: analyze, compare, create, design, architecture, refactor, debug, optimize, implement, algorithm, multi-step, reasoning, evaluate, synthesize, critique. Case-insensitive whole-word match, 0.5 points each, capped at 2 total. Useful to know up-front because common code-assist prompts like "implement and evaluate this algorithm" tip the score by +1.5 from keywords alone, often the difference between SIMPLE and MODERATE.

Thresholds: 0–2 = SIMPLE, 3–5 = MODERATE, 6+ = COMPLEX.

Fallback behavior

1. If the tier model's provider is unavailable, the router tries one lower-tier hop, not a full cascade. So COMPLEX falls back to MODERATE; MODERATE falls back to SIMPLE; SIMPLE has no further tier fallback. If the one-hop attempt also has no available provider, step 2 takes over.
2. Round-robin across the route's configured providers, regardless of tier. This is the safety net for "neither the target tier nor its immediate fallback can be served".
3. The resolved model overrides the request's model field before calling the provider, so your billing and audit trail show the actual model used, not the wildcard the client sent.

Worked example: a COMPLEX classification points at claude-3-opus. Anthropic is down. The router tries MODERATE → gpt-4o. OpenAI is also down. Rather than then try SIMPLE → gpt-4o-mini, the router goes straight to round-robin across the configured providers. If gpt-4o-mini happens to live on a third configured provider that is up, you get there by chance (round-robin walks the pool), not by tier cascade. The practical implication: if you want resilient COMPLEX traffic, make sure your MODERATE tier is on a different provider than COMPLEX — and ideally have at least one configured provider outside both tiers as a safety net.

Example requests

A short factual question classifies as SIMPLE (score 0 — one message, under 500 chars, no tool use, no keywords) and routes to gpt-4o-mini:

curl -X POST http://localhost:8080/v1/chat/completions \
  -H 'Authorization: Bearer $DVARA_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "auto",
    "messages": [{"role": "user", "content": "What is the capital of Japan?"}]
  }'

A long multi-turn conversation with a detailed system prompt, tool use, and complexity keywords classifies as COMPLEX (score ≥ 6) and routes to claude-3-opus:

curl -X POST http://localhost:8080/v1/chat/completions \
  -H 'Authorization: Bearer $DVARA_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "auto",
    "messages": [
      {"role": "system", "content": "You are a senior staff engineer. When asked to analyze, refactor, or design, walk through the problem systematically and cite the relevant sections of the code."},
      {"role": "user", "content": "Here is a 2000-line service. Analyze the race condition in the workspace-resolution path and propose a refactor with tests."},
      {"role": "assistant", "content": "..."},
      {"role": "tool", "content": "..."},
      {"role": "user", "content": "Now design a migration plan and implement the first two steps."}
    ]
  }'

Scoring breakdown for the second request: 5 messages (+1), non-system text well over 2000 chars (+2, the long user message alone clears the threshold — the system prompt isn't counted here, it's scored separately on the row below), tool use present (+2), system prompt 501–1500 chars (+1), multiple complexity keywords — analyze, refactor, design, implement — (+2, capped). Total 8, well above the COMPLEX threshold of 6.

The client's "model": "auto" is rewritten to the tier-resolved model (gpt-4o-mini or claude-3-opus) before the upstream call, so the provider bills for and the audit trail records the actual model, not the wildcard.

Shadow routing

Shadow routing sends a copy of production traffic to a shadow provider for comparison without affecting the primary response. The client always sees the primary response; the shadow is invoked asynchronously in the background and its output is compared for analysis.

Shadow is a property on top of any other strategy — a round-robin route can carry shadow traffic:

Enterprise only

Not included in DVARA Open Source.

DVARA Open Source refuses a static shadow-config at startup; it does not silently ignore the block.

Create the route on Flightdeck's Routes page. Choose the shadow provider, sample percentage, optional test name, and optional workspace scope. Shadow is a property on top of another strategy, not a strategy of its own.

providers lists the primary candidates only; the shadow provider does not need to be in that list. It must still be configured and available to the gateway. If it is unavailable, DVARA skips the shadow call and the primary path continues; check the gateway logs if the report remains empty.

Limit shadow traffic to a workspace

The optional workspace scope filters which requests get sampled, and its behaviour differs from canary in one case:

• Scope set, and the request's resolved workspace matches → shadow may be dispatched (subject to the sample percentage)
• Scope set, workspace resolved, does not match → shadow is skipped
• Scope set and there is no workspace at all → shadow is skipped. The scope must resolve to the named workspace before the copy is dispatched.

The shipped Gateway requires an API key on every /v1 request and resolves its workspace before routing.

Account for shadow cost and capacity

A sampled request makes a real second provider call, so the shadow provider bills it. DVARA records shadow token usage and cost separately for cost views, chargeback, and budget evaluation. Shadow dispatch also uses its configured concurrency bound, so a high sample percentage cannot create unbounded upstream work.

Monitoring

Open the route's Shadow dashboard (/routes/{id}/shadow). It polls a live comparison of the primary against the shadow provider, and carries a Reset action to start a fresh window.

The report includes total sampled request count, primary-vs-shadow average latency, and a response match rate — note that "match" here is a coarse word-level Jaccard similarity > 0.5 on the first text block, not exact equality or semantic equivalence. Two paraphrases with mostly-overlapping vocabulary count as a match; an LLM that reworded heavily but kept the meaning won't. Treat the match rate as a rough drift signal, not a regression-test pass/fail.

Example request against the shadow route:

curl -X POST http://localhost:8080/v1/chat/completions \
  -H 'Authorization: Bearer $DVARA_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Rewrite this paragraph in plain English."}]
  }'

The client gets the primary response from the provider selected by round-robin. On roughly 10% of requests, a copy goes to the configured shadow provider. That provider must accept the same resolved model identifier. The client never sees the shadow response; Flightdeck shows its latency, success status, and response-match score in the comparison report. An outage on the shadow provider does not change the primary response.

SLA-aware priority routing

Disabled by default

Priority admission control is off by default. Set dvara.llm-gateway.routing.priority.enabled=true to turn it on.

Enterprise only

Not included in DVARA Open Source.

Priority routing provides concurrency-based admission control with per-tier throttle thresholds. Premium workspaces are admitted until the observed load reaches full capacity; standard and bulk workspaces are throttled earlier, so high-value traffic receives preference under pressure.

Priority tiers

Tier	Default threshold	Behavior
premium	100%	Throttled when observed concurrent load reaches full capacity
standard	80%	Throttled when concurrent load reaches 80% of max capacity
bulk	50%	Throttled when concurrent load reaches 50% of max capacity

Rejected requests return HTTP 429 with error code priority_throttled.

Thresholds are not hard concurrency caps in 1.8

The 1.8 admission check reads the current count before it reserves a slot. During a simultaneous burst, more than one request can cross a threshold together. Use the tiers to express overload preference, not as an exact concurrency guarantee.

Per-workspace configuration

Set a workspace's priority tier through the workspace metadata priority-tier key, from the DVARA Flightdeck:

{ "workspaces": [ { "id": "acme-corp", "metadata": { "priority-tier": "premium" } } ] }

Set it on the workspace form, or as a metadata entry in the GitOps export document.

Workspaces without a priority-tier metadata key default to standard.

Configuration

dvara:
  llm-gateway:
    routing:
      priority:
        enabled: true                  # default: false
        max-concurrent-requests: 1000  # default: 1000
        tiers:
          premium:
            throttle-threshold-pct: 100
          standard:
            throttle-threshold-pct: 80
          bulk:
            throttle-threshold-pct: 50
        resolver-cache-ttl-seconds: 5  # default: 5

How it works

1. After policy and PII enforcement, DVARA resolves the workspace's priority tier from workspace metadata. A metadata change takes effect after the five-second cache expires.
2. An in-flight request counter tracks the current concurrent load. The load percentage is (current / maxConcurrent) * 100.
3. If the load percentage meets or exceeds the tier's threshold, the request is rejected with priority_throttled.
4. Admitted requests increment the counter; it is decremented after the request completes or fails. A request that never completes continues to hold its slot until its execution path ends.

Monitoring

Priority admission has two observability surfaces — pick the one that matches the workflow.

Prometheus metrics — primary. Live dashboards, alerting, and historical trend analysis all belong on your observability stack. DVARA emits the admission counters every request:

• gateway_priority_requests_total{workspace, tier} — admitted requests by workspace and tier
• gateway_priority_throttled_total{workspace, tier} — rejected requests by workspace and tier

Graph the ratio throttled / (admitted + throttled) per tier in Grafana to see the exact moments a tier hit its throttle threshold under load. The metrics carry per-workspace cardinality, so you can isolate a noisy workspace or cross-reference with that workspace's cost records.

On the dashboard. The Flightdeck's home page carries a priority-admission card — concurrent now, max concurrent, load percentage, and a per-tier table showing each tier's throttle threshold and whether it is currently admitting or throttling. It is rendered only when dvara.llm-gateway.routing.priority.enabled=true.

For scripting and alerting, use the Prometheus metrics above rather than a snapshot endpoint — gateway_priority_requests_total and gateway_priority_throttled_total carry the same information over time, which is what a runbook or an oncall alert actually wants.

There is no dedicated priority-statistics API or separate Flightdeck page. The Dashboard card is the live snapshot; use the Prometheus counters for history and alerts.

---

Hot reload

Enterprise only

Not included in DVARA Open Source.

DVARA Open Source reads gateway.yaml once at startup; its routes stay unchanged until restart, and it does not support dvara.config.poll-interval-ms.

Route configuration from DVARA Flightdeck refreshes across the Gateway fleet without a restart. Gateway nodes check for configuration changes every three seconds by default; set dvara.config.poll-interval-ms to change that interval. Allow at least one complete poll interval before treating a route change as active on every node.

What hot-reloads:

• Route pattern, strategy, provider pool, weights, pinned model version
• Canary config on a route (split percentage, workspace scope, test name)
• Shadow config on a route (sample percentage, shadow provider)
• Workspace priority-tier metadata — read through a 5-second TTL cache per workspace, so a workspace-metadata update propagates within 5 seconds of the next request

What does not hot-reload (these require a restart):

• The list of activated providers set at startup from environment variables or provider configuration
• Every property under dvara.llm-gateway.routing.* in application.yml — EWMA smoothing factor, decay thresholds, priority admission max concurrent requests, per-tier throttle thresholds, resolver TTLs. These are bound at gateway startup and aren't refreshable without a restart.
• The dvara.llm-gateway.routes: block itself — editing static route configuration requires a Gateway restart. Do not define the same Enterprise route in static configuration and Flightdeck.

Update routes from the DVARA Flightdeck at /routes. Route versions are tracked on every save — roll back from DVARA Flightdeck if a live change causes a regression.

Next steps

• Providers — provider-specific quirks, capability matrix, and BYOK credentials for every upstream DVARA can talk to.
• Structured Outputs — how json_schema and json_object interact with capability-aware filtering.
• Resilience — circuit breakers, retries, and failover between providers on a route.
• Cost Management — the pricing table cost-aware routing reads from, plus budgets, forecasts, and chargeback reports.
• Multi-Tenancy — how per-workspace priority tiers and region context feed priority and geo-aware routing.