2 posts tagged with "observability"

Ask a team running AI in production a few basic questions — what's our p95 latency to Anthropic? how many tokens did the support team burn last week? what's our error rate on gpt-4o? — and watch how long the answer takes. If it's a research project rather than a dashboard glance, you don't have LLM observability.

Observability isn't a nice-to-have next to governance; it's the layer underneath it. Policy enforcement, cost attribution, and audit are only as trustworthy as the telemetry that proves they ran. "We block disallowed models" needs a metric that shows the blocks; "we attribute every dollar" needs token counts per tenant; "we can prove what happened" needs traces and logs. In DVARA, the DVARA LLM Gateway emits that evidence on every request, which is what turns governance claims into governance you can audit.

In the previous post, you got DVARA running and sent requests to multiple providers through a single OpenAI SDK client. This post covers the next layer of governance platform capabilities — LLM streaming, structured JSON outputs, Prometheus observability, and local development with Ollama.