6 posts tagged with "architecture"

How many LLM calls did your company make yesterday?

Most platform leads can't answer. Not roughly, not which providers, not which teams. That gap has a name — shadow AI — and it is a visibility problem long before it is a policy problem.

DVARA 1.6.0 made the free Community Edition a separate image that anyone could pull and evaluate. This post records how that distribution worked when 1.6 shipped; the update below explains what replaced it in 1.8.

"Which LLM gateway should we use?" doesn't have a universal answer, because teams optimize for different things — fastest to adopt, cheapest to run, most observable, or most governable. What is universal is the set of questions worth asking, and the order to ask them in.

This is a framework, not a leaderboard. And it's deliberately governance-first, because that's the axis where products differ most and demo videos reveal least. Routing and provider coverage are easy to compare and roughly commoditized; whether a platform can enforce a policy, prove what it did, and keep sensitive data in your perimeter is where the real divergence — and the real buying decision — lives. DVARA is one option in the field; the point of this post is to help you score the field, including us, on criteria that matter.

Model providers are remarkably good and occasionally unavailable. They rate-limit you at the worst moment, degrade under load, and have real outages. If your application talks to a provider directly, every one of those events is a user-facing failure — and every team ends up writing its own retry loop, usually badly.

Resilience belongs in the same layer as governance, and for the same reason: it can't be trusted if it's scattered and invisible. When failover runs at a single governed control point — the DVARA LLM Gateway, the data plane of the AI governance platform — every retry and provider switch is policy-checked and written to the audit trail, so "we failed over to a different provider" is a recorded, reviewable event, not a silent mystery. LLM fallback and failover move resilience out of application code and onto the governed path.

Multi-provider LLM routing sends each model request to an eligible provider according to a declared strategy while keeping policy, PII controls, rate limits, audit, and cost attribution on the same request path. It solves a platform problem: applications should not each implement their own provider selection and silently drift into different governance behavior.

DVARA is an AI governance platform; the DVARA LLM Gateway is the component that routes model traffic. The application keeps one OpenAI-compatible endpoint, while the Gateway chooses a provider only after governance checks and capability filtering.

Your API gateway handles TLS termination, rate limiting, and request routing. It does these things well. But when your traffic carries prompts, tokens, tool calls, and model-specific payloads, an API gateway becomes a passthrough — it can see the HTTP envelope but not the AI semantics inside it.