How many LLM calls did your company make yesterday?
Most platform leads can't answer. Not roughly, not which providers, not which teams. That gap has a name — shadow AI — and it is a visibility problem long before it is a policy problem.
Ask a team running AI in production a few basic questions — what's our p95 latency to Anthropic? how many tokens did the support team burn last week? what's our error rate on gpt-4o? — and watch how long the answer takes. If it's a research project rather than a dashboard glance, you don't have LLM observability.
Observability isn't a nice-to-have next to governance; it's the layer underneath it. Policy enforcement, cost attribution, and audit are only as trustworthy as the telemetry that proves they ran. "We block disallowed models" needs a metric that shows the blocks; "we attribute every dollar" needs token counts per workspace; "we can prove what happened" needs traces and logs. In DVARA, the DVARA LLM Gateway emits that evidence on every request, which is what turns governance claims into governance you can audit.
This post was published against 1.0.0 and referred to the gateway image as
ghcr.io/dvarahq/dvara/dvara-llm-gateway. The package was renamed to
ghcr.io/dvarahq/dvara-gateway in 1.7.0, so the original command no longer pulls anything. The
commands below have been updated to the current image and tag; the rest of the post is unchanged.
In the previous post, you got DVARA running and sent requests to multiple providers through a single OpenAI SDK client. This post covers the next layer of governance platform capabilities — LLM streaming, structured JSON outputs, Prometheus observability, and local development with Ollama.