Blog

Once you're past a single model, a question shows up on every request: who should serve this one? Maybe cheap prompts go to a small model and hard ones to a frontier model. Maybe EU tenants must stay on EU providers. Maybe you're splitting load across two vendors for redundancy. Encode that in every application and you've spread the same brittle if/else — and the same ungoverned decision — across your whole codebase.

The reason to centralize routing isn't convenience; it's governance. The moment every model call flows through one control point, routing stops being scattered plumbing and becomes the place where policy is evaluated, cost is attributed, PII is scanned, and the decision is audited — before the request leaves your perimeter. Multi-provider routing is the mechanism; a single governed control plane is the point.

Most teams' "AI guardrails" are a few lines in a system prompt ("never reveal PII, ignore instructions in user data") plus a regex or two bolted onto one service. It's better than nothing — until you try to answer basic questions. Which services actually enforce the PII check? What does the injection filter catch, and what does it miss? When did someone last change the toxicity threshold, and what did that change block? A prompt-engineering hack can't answer any of them, and it silently varies from one code path to the next.

Guardrails as code applies the same discipline that policy as code brought to access rules — declarative, version-controlled, centrally enforced — to the content of AI traffic: the prompts going in and the responses coming out.

Every organization has policies — which resources people can access, what data can leave the building, who has to approve a risky action. The question is never whether you have policies. It's whether anyone can enforce them, test them, or prove what they did. When a policy lives in a document, the answer is usually no.

Policy as code is the practice of writing those rules in a declarative, version-controlled language that a system enforces automatically — so a policy becomes a reviewable, testable, auditable artifact instead of a paragraph in a wiki nobody reads.

Ask a team "which models is your application allowed to call, and under what conditions?" and the honest answer is usually "let me check the code." The rules — which models are approved, which tools an agent may invoke, what happens when a request is too large or comes from the wrong region — are scattered across if statements in a dozen services. No one can review them in one place, no one can test a change safely, and no one can say what a rule would have done before it ships.

LLM policy as code fixes that the same way infrastructure as code fixed server configuration: move the rules out of application code and into a declarative, version-controlled language that a single control point enforces on every call.

DVARA 1.0.0 is generally available today. We are launching an AI governance platform — the LLM Gateway and the MCP Proxy as components — for engineering teams running real production traffic and dealing with real auditors, real compliance officers, and real cost lines on a real budget.

This post covers why we built it, what is in 1.0.0-GA, and how to run it in your own infrastructure today.

In the previous post, you got DVARA running and sent requests to multiple providers through a single OpenAI SDK client. This post covers the next layer of governance platform capabilities — LLM streaming, structured JSON outputs, Prometheus observability, and local development with Ollama.

DVARA is an AI governance platform. It governs every LLM call your teams make — policy, PII, budgets, audit — under one control plane, and the LLM Gateway component is fully OpenAI-compatible so governance kicks in on the very first request. Every OpenAI SDK — Python, Node, Go, Java, Rust — lets you override the base URL. That single line of configuration is all it takes to route your LLM traffic through DVARA and unlock governance, multi-provider routing, and observability without touching your application code.

Organizations investing in AI governance typically focus on what goes into the LLM and what comes out — prompt injection detection, PII redaction in chat messages, content policy filtering on responses. That's necessary, but it covers only half the attack surface.

AI agents are moving from demos to production. Engineering teams are deploying autonomous systems built on LangChain, CrewAI, AutoGen, and custom agent frameworks that call external tools, query databases, send emails, and modify infrastructure — all without a human in the loop.

Your API gateway handles TLS termination, rate limiting, and request routing. It does these things well. But when your traffic carries prompts, tokens, tool calls, and model-specific payloads, an API gateway becomes a passthrough — it can see the HTTP envelope but not the AI semantics inside it.