4 posts tagged with "architecture"

"Which LLM gateway should we use?" doesn't have a universal answer, because teams optimize for different things — fastest to adopt, cheapest to run, most observable, or most governable. What is universal is the set of questions worth asking, and the order to ask them in.

This is a framework, not a leaderboard. And it's deliberately governance-first, because that's the axis where products differ most and demo videos reveal least. Routing and provider coverage are easy to compare and roughly commoditized; whether a platform can enforce a policy, prove what it did, and keep sensitive data in your perimeter is where the real divergence — and the real buying decision — lives. DVARA is one option in the field; the point of this post is to help you score the field, including us, on criteria that matter.

Model providers are remarkably good and occasionally unavailable. They rate-limit you at the worst moment, degrade under load, and have real outages. If your application talks to a provider directly, every one of those events is a user-facing failure — and every team ends up writing its own retry loop, usually badly.

Resilience belongs in the same layer as governance, and for the same reason: it can't be trusted if it's scattered and invisible. When failover runs at a single governed control point — the DVARA LLM Gateway, the data plane of the AI governance platform — every retry and provider switch is policy-checked and written to the audit trail, so "we failed over to a different provider" is a recorded, reviewable event, not a silent mystery. LLM fallback and failover move resilience out of application code and onto the governed path.

Once you're past a single model, a question shows up on every request: who should serve this one? Maybe cheap prompts go to a small model and hard ones to a frontier model. Maybe EU tenants must stay on EU providers. Maybe you're splitting load across two vendors for redundancy. Encode that in every application and you've spread the same brittle if/else — and the same ungoverned decision — across your whole codebase.

The reason to centralize routing isn't convenience; it's governance. The moment every model call flows through one control point, routing stops being scattered plumbing and becomes the place where policy is evaluated, cost is attributed, PII is scanned, and the decision is audited — before the request leaves your perimeter. Multi-provider routing is the mechanism; a single governed control plane is the point.

Your API gateway handles TLS termination, rate limiting, and request routing. It does these things well. But when your traffic carries prompts, tokens, tool calls, and model-specific payloads, an API gateway becomes a passthrough — it can see the HTTP envelope but not the AI semantics inside it.