Canary a provider change and roll back on error rate

Different operating paths

DVARA Open Source configures it in gateway.yaml and restarts; the Enterprise platform manages it in Flightdeck.

The problem

You want to move a workload from one provider to another without shifting all traffic at once. This example compares gpt-4o on OpenAI with an Azure OpenAI deployment also named gpt-4o, then uses error rate, latency, and cost to decide whether to promote or roll back.

The approach

DVARA's canary routing strategy splits a route between a baseline provider and a candidate provider with a configurable percentage. Both sides write metrics tagged with the variant so the canary report shows them side-by-side under the same route ID. Flipping the split back to 0% rolls back instantly.

Canary scope: provider-to-provider, not model-to-model

The canary strategy splits between two providers. It does not map one provider's model name to another, and it cannot compare two models on the same provider. Both providers must accept the model identifier in the client request. To test a model upgrade on one provider, split model names in the client or in an experiment system outside the 1.8 route configuration.

Prerequisites

• A running DVARA instance with a workspace and API key (Quickstart)
• Both providers registered (e.g. OPENAI_API_KEY for the baseline and AZURE_OPENAI_API_KEY + AZURE_OPENAI_BASE_URL for the candidate) — see Provider Setup
• An owner or policy-admin account on DVARA Flightdeck

The steps

1. Create a canary route

In the Enterprise platform, create the route in Flightdeck under Routing → Routes → New Route. In DVARA Open Source, define the canary route in gateway.yaml and restart the process. Then set

field	value
Model pattern	gpt-4o
Strategy	canary
Canary baseline	openai
Canary candidate	azure-openai
Split %	10
Test name	azure-eval (optional, labels the comparison)
Workspace scope	optional — confines the experiment to one workspace

This route matches gpt-4o and routes 90% to OpenAI and 10% to Azure OpenAI. The Azure deployment must also be named gpt-4o; DVARA does not translate it.

2. Send traffic

Applications continue sending model: "gpt-4o". The literal route pattern prevents other GPT model names from entering a split whose Azure deployment names have not been verified.

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "..."}],
)

3. Confirm the split is actually happening

Nothing on the response tells you which variant served it. There is no X-Gateway-Variant header in 1.8.0, so do not write client-side assertions against one — the split is observable on the gateway, not on the wire.

Where it is observable is the Prometheus counter:

curl -s -H "Authorization: Bearer $DVARA_ACTUATOR_METRICS_API_KEY" \
  https://dvara.internal.example.com/actuator/prometheus | grep gateway_canary_requests_total

# HELP gateway_canary_requests_total Requests served per canary variant
# TYPE gateway_canary_requests_total counter
gateway_canary_requests_total{route_id="rt_7f3a",variant="baseline",model="gpt-4o"} 903.0
gateway_canary_requests_total{route_id="rt_7f3a",variant="candidate",model="gpt-4o"} 97.0

That is the check that the split you configured is the split you are running — 90/10 here, against a configured 10%.

An empty result does not mean the canary is broken

The counter is created on first increment, so grep returns nothing until the route has served at least one request. A canary you have just configured and not yet driven traffic through looks identical to one that is not working. Send traffic first, then query.

4. Monitor canary metrics

Open the route's Canary dashboard (/routes/{id}/canary). It refreshes on a poll and shows per-variant counts, error rates, latency percentiles and cost, side by side under the same route.

Reset on that page starts a fresh comparison window — use it after changing the split, so the numbers describe the split you are actually running.

A typical decision threshold for promoting a canary:

Metric	Canary must be …
Error rate	Not meaningfully higher than baseline (within 1σ, or a fixed delta you set)
p95 latency	Within an acceptable budget of baseline (e.g. +20%)
Cost per request	Within budget, or the quality gain justifies the delta
Sample size	At least N thousand requests so the comparison is not noise

5a. Promote (if candidate is healthy)

Edit the route and make the candidate the sole provider: change the strategy to weighted and leave azure-openai as the only provider with a positive weight. Do not switch this route to model-prefix; that strategy selects from registered model prefixes rather than treating the route's provider list as a forced target. DVARA keeps the previous route version so you can still roll back.

5b. Roll back (if candidate is bad)

Set the split to 0% on the route's canary dashboard. The change propagates across the fleet on the config-version poll — a few seconds, no pod restart.

Or restore the previous route version outright: the route's edit page lists its version history with a diff against the current config, and a Rollback action for each.

All traffic goes back to OpenAI immediately. No redeploy, no app change.

Why this works

• Split is server-side — applications keep sending model: "gpt-4o"; the weight is enforced inside DVARA. No need to ship a feature flag to every service.
• Variants are separable in metrics — each canary request is tagged with the variant so gateway_canary_requests_total and the per-route error and latency histograms don't mix the two.
• Rollback is atomic — the save updates the route version, the config-version poll propagates the change to every data plane pod within the poll interval (a few seconds; see Architecture → Config propagation), and the next request on every pod uses the new split.
• Route history is preserved — the old version is retained, so Rollback restores it without you having to remember the old config.

Common mistakes

• Sampling too small a window before deciding — a 100-request canary shows noise, not signal. Pick a traffic volume and elapsed time that makes the delta detectable for your baseline error rate.
• Using different model namespaces — DVARA sends the same request model to both variants. An Anthropic model name is not a Bedrock model ID. Even OpenAI and Azure OpenAI work only when the Azure deployment name matches the request.
• Forgetting to clean up the canary route after promotion — once the candidate is the new baseline, use a weighted route containing only that provider. A stale canary config keeps the dashboard live with a stagnant split.

Next steps

• Routing — all the routing strategies (round-robin, weighted, latency-aware, cost-aware, geo-aware)
• Resilience — circuit breakers and failover that compose with canary splits
• Observability — the metrics and audit events that back the canary decision
• Routes and Policies (Flightdeck) — canary dashboard walkthrough in the UI