Canary a new model version and roll back on error rate

The problem

You're considering moving a workload from one provider to another — gpt-4o on OpenAI to the same model on Azure OpenAI for HA, claude-sonnet-4-5 on Anthropic to the same model via AWS Bedrock for data-residency, or any first-class provider you've added recently. You want to send a small percentage of traffic to the candidate while the rest stays on the production provider, get side-by-side metrics (error rate, latency, cost per request), and make the promote or roll back call in minutes — not days, and without a redeploy.

The approach

DVARA's canary routing strategy splits a route between a baseline provider and a candidate provider with a configurable percentage. Both sides write metrics tagged with the variant so the canary report shows them side-by-side under the same route ID. Flipping the split back to 0% rolls back instantly.

Canary scope: provider-to-provider, not model-to-model

The canary strategy splits between two providers. Canarying a new model on the same provider (e.g. gpt-4o → gpt-5-preview both on OpenAI) is not supported by route configuration alone — DVARA routes pick a provider, and the upstream model name comes from the request or pinned-model-version. To test a same-provider model upgrade, gate the client side with a feature flag and let it send the new model name to a separately-pinned route. The canary recipe below covers the more common case of swapping the provider underneath the same workload.

Prerequisites

• A running DVARA instance with a workspace and API key (Quickstart)
• Both providers registered (e.g. OPENAI_API_KEY for the baseline and AZURE_OPENAI_API_KEY + AZURE_OPENAI_BASE_URL for the candidate) — see Provider Setup
• An owner or policy-admin account on DVARA Flightdeck

The steps

1. Create a canary route

Canary configuration is not bindable from application.yml — the gateway.routes block does not bind canary configs at startup. Create the route in the Console instead: Routing → Routes → New Route, then set

field	value
Model pattern	gpt*
Strategy	canary
Canary baseline	openai
Canary candidate	azure-openai
Split %	10
Test name	azure-eval (optional, labels the comparison)
Workspace scope	optional — confines the experiment to one workspace

This route matches any gpt* model the client sends, routes 90% to OpenAI (baseline) and 10% to Azure OpenAI (candidate). DVARA tags each request with its variant so canary metrics stay separable from baseline metrics.

2. Send traffic

No client change needed. Applications continue sending model: "gpt-4o" (or any gpt* model — the route matches the pattern). DVARA handles the split server-side.

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "..."}],
)

3. Confirm the split is actually happening

Nothing on the response tells you which variant served it. There is no X-Gateway-Variant header in 1.7.0, so do not write client-side assertions against one — the split is observable on the gateway, not on the wire.

Where it is observable is the Prometheus counter:

curl -s -H "Authorization: Bearer $DVARA_ACTUATOR_METRICS_API_KEY" \
  https://dvara.internal.example.com/actuator/prometheus | grep gateway_canary_requests_total

# HELP gateway_canary_requests_total Requests served per canary variant
# TYPE gateway_canary_requests_total counter
gateway_canary_requests_total{route_id="rt_7f3a",variant="baseline",model="gpt-4o"} 903.0
gateway_canary_requests_total{route_id="rt_7f3a",variant="candidate",model="gpt-4o"} 97.0

That is the check that the split you configured is the split you are running — 90/10 here, against a configured 10%.

An empty result does not mean the canary is broken

The counter is created on first increment, so grep returns nothing until the route has served at least one request. A canary you have just configured and not yet driven traffic through looks identical to one that is not working. Send traffic first, then query.

4. Monitor canary metrics

Open the route's Canary dashboard (/routes/{id}/canary). It refreshes on a poll and shows per-variant counts, error rates, latency percentiles and cost, side by side under the same route.

Reset on that page starts a fresh comparison window — use it after changing the split, so the numbers describe the split you are actually running.

A typical decision threshold for promoting a canary:

Metric	Canary must be …
Error rate	Not meaningfully higher than baseline (within 1σ, or a fixed delta you set)
p95 latency	Within an acceptable budget of baseline (e.g. +20%)
Cost per request	Within budget, or the quality gain justifies the delta
Sample size	At least N thousand requests so the comparison is not noise

5a. Promote (if candidate is healthy)

Edit the route and make the candidate the sole provider: set the strategy to model-prefix and leave azure-openai as the only entry. DVARA keeps the previous route version, so you can still go back if something surprises you in production.

5b. Roll back (if candidate is bad)

Set the split to 0% on the route's canary dashboard. The change propagates across the fleet on the config-version poll — a few seconds, no pod restart.

Or restore the previous route version outright: the route's edit page lists its version history with a diff against the current config, and a Rollback action for each.

All traffic goes back to OpenAI immediately. No redeploy, no app change.

Why this works

• Split is server-side — applications keep sending model: "gpt-4o"; the weight is enforced inside DVARA. No need to ship a feature flag to every service.
• Variants are separable in metrics — each canary request is tagged with the variant so gateway_canary_requests_total and the per-route error and latency histograms don't mix the two.
• Rollback is atomic — the save updates the route version, the config-version poll propagates the change to every data plane pod within the poll interval (a few seconds; see Architecture → Config propagation), and the next request on every pod uses the new split.
• Route history is preserved — the old version is retained, so Rollback restores it without you having to remember the old config.

Common mistakes

• Sampling too small a window before deciding — a 100-request canary shows noise, not signal. Pick a traffic volume and elapsed time that makes the delta detectable for your baseline error rate.
• Picking providers that serve different model classes for the same workload — if the baseline is OpenAI's gpt-4o and the candidate is Anthropic's claude-sonnet-4-5, the canary report mixes model-quality variance with provider-infrastructure variance. Pick a candidate that exposes the same model name (e.g. azure-openai for gpt-4o, bedrock for claude-*) to keep the comparison clean.
• Forgetting to clean up the canary route after promotion — once the candidate is the new baseline, swap the route's strategy to model-prefix with a single-provider list. A stale canary config keeps the canary dashboard live but reports a stagnant 100/0 split.

Next steps

• Routing — all the routing strategies (round-robin, weighted, latency-aware, cost-aware, geo-aware)
• Resilience — circuit breakers and failover that compose with canary splits
• Observability — the metrics and audit events that back the canary decision
• Routes and Policies (Flightdeck) — canary dashboard walkthrough in the UI