18 posts tagged with "governance"

Model providers are remarkably good and occasionally unavailable. They rate-limit you at the worst moment, degrade under load, and have real outages. If your application talks to a provider directly, every one of those events is a user-facing failure — and every team ends up writing its own retry loop, usually badly.

Resilience belongs in the same layer as governance, and for the same reason: it can't be trusted if it's scattered and invisible. When failover runs at a single governed control point — the DVARA LLM Gateway, the data plane of the AI governance platform — every retry and provider switch is policy-checked and written to the audit trail, so "we failed over to a different provider" is a recorded, reviewable event, not a silent mystery. LLM fallback and failover move resilience out of application code and onto the governed path.

Once you're past a single model, a question shows up on every request: who should serve this one? Maybe cheap prompts go to a small model and hard ones to a frontier model. Maybe EU workspaces must stay on EU providers. Maybe you're splitting load across two vendors for redundancy. Encode that in every application and you've spread the same brittle if/else — and the same ungoverned decision — across your whole codebase.

The reason to centralize routing isn't convenience; it's governance. The moment every model call flows through one control point, routing stops being scattered plumbing and becomes the place where policy is evaluated, cost is attributed, PII is scanned, and the decision is audited — before the request leaves your perimeter. Multi-provider routing is the mechanism; a single governed control plane is the point.

Most teams' "AI guardrails" are a few lines in a system prompt ("never reveal PII, ignore instructions in user data") plus a regex or two bolted onto one service. It's better than nothing — until you try to answer basic questions. Which services actually enforce the PII check? What does the injection filter catch, and what does it miss? When did someone last change the toxicity threshold, and what did that change block? A prompt-engineering hack can't answer any of them, and it silently varies from one code path to the next.

Guardrails as code applies the same discipline that policy as code brought to access rules — declarative, version-controlled, centrally enforced — to the content of AI traffic: the prompts going in and the responses coming out.

Every organization has policies — which resources people can access, what data can leave the building, who has to approve a risky action. The question is never whether you have policies. It's whether anyone can enforce them, test them, or prove what they did. When a policy lives in a document, the answer is usually no.

Policy as code is the practice of writing those rules in a declarative, version-controlled language that a system enforces automatically — so a policy becomes a reviewable, testable, auditable artifact instead of a paragraph in a wiki nobody reads.

Ask a team "which models is your application allowed to call, and under what conditions?" and the honest answer is usually "let me check the code." The rules — which models are approved, which tools an agent may invoke, what happens when a request is too large or comes from the wrong region — are scattered across if statements in a dozen services. No one can review them in one place, no one can test a change safely, and no one can say what a rule would have done before it ships.

LLM policy as code fixes that the same way infrastructure as code fixed server configuration: move the rules out of application code and into a declarative, version-controlled language that a single control point enforces on every call.

DVARA 1.0.0 is generally available today. We are launching an AI governance platform — the LLM Gateway and the MCP Proxy as components — for engineering teams running real production traffic and dealing with real auditors, real compliance officers, and real cost lines on a real budget.

This post covers why we built it, what is in 1.0.0-GA, and how to run it in your own infrastructure today.

AI agents are moving from demos to production. Engineering teams are deploying autonomous systems built on LangChain, CrewAI, AutoGen, and custom agent frameworks that call external tools, query databases, send emails, and modify infrastructure — all without a human in the loop.

Your API gateway handles TLS termination, rate limiting, and request routing. It does these things well. But when your traffic carries prompts, tokens, tool calls, and model-specific payloads, an API gateway becomes a passthrough — it can see the HTTP envelope but not the AI semantics inside it.