Blog

In the previous post, you got DVARA running and sent requests to multiple providers through a single OpenAI SDK client. This post covers the next layer of governance platform capabilities — LLM streaming, structured JSON outputs, Prometheus observability, and local development with Ollama.

DVARA is an AI governance platform. It governs every LLM call your teams make — policy, PII, budgets, audit — under one control plane, and the LLM Gateway component is fully OpenAI-compatible so governance kicks in on the very first request. Every OpenAI SDK — Python, Node, Go, Java, Rust — lets you override the base URL. That single line of configuration is all it takes to route your LLM traffic through DVARA and unlock governance, multi-provider routing, and observability without touching your application code.

Organizations investing in AI governance typically focus on what goes into the LLM and what comes out — prompt injection detection, PII redaction in chat messages, content policy filtering on responses. That's necessary, but it covers only half the attack surface.

AI agents are moving from demos to production. Engineering teams are deploying autonomous systems built on LangChain, CrewAI, AutoGen, and custom agent frameworks that call external tools, query databases, send emails, and modify infrastructure — all without a human in the loop.

Your API gateway handles TLS termination, rate limiting, and request routing. It does these things well. But when your traffic carries prompts, tokens, tool calls, and model-specific payloads, an API gateway becomes a passthrough — it can see the HTTP envelope but not the AI semantics inside it.