Skip to main content

LLM Observability: The Telemetry Layer Under AI Governance

· 4 min read
DVARA Team
DVARA Team

Ask a team running AI in production a few basic questions — what's our p95 latency to Anthropic? how many tokens did the support team burn last week? what's our error rate on gpt-4o? — and watch how long the answer takes. If it's a research project rather than a dashboard glance, you don't have LLM observability.

Observability isn't a nice-to-have next to governance; it's the layer underneath it. Policy enforcement, cost attribution, and audit are only as trustworthy as the telemetry that proves they ran. "We block disallowed models" needs a metric that shows the blocks; "we attribute every dollar" needs token counts per tenant; "we can prove what happened" needs traces and logs. In DVARA, the DVARA LLM Gateway emits that evidence on every request, which is what turns governance claims into governance you can audit.

What LLM observability is​

LLM observability is the practice of instrumenting your model traffic so you can measure and diagnose it: request rate, latency distribution, token consumption, cost, cache behavior, and errors — broken down by the dimensions that matter (model, provider, tenant, status). It's ordinary observability applied to a new kind of traffic, plus a few AI-specific signals (tokens, cache hits, provider failovers) that generic app monitoring doesn't capture.

Because a governed control point sits on the path of every call, it's the natural place to emit this telemetry consistently — every request, every provider, one schema.

The three pillars, applied to LLMs​

Metrics​

The counters and histograms you actually watch on a dashboard:

  • Request rate and latency — requests/sec and p50/p95/p99 latency, tagged by model, provider, and tenant.
  • Token usage — input and output tokens, per model and tenant. This is the raw material for cost governance.
  • Errors — provider errors by code, so you see which provider is failing and how.
  • Resilience signals — retries, fallbacks (from-provider → to-provider), and rate-limit sheds, which tell you whether your failover is a quiet safety net or a daily tax.

Exposed in a standard format (Prometheus), these drop straight into the monitoring stack you already run.

Traces​

A single request often fans out — filter chain, routing, an upstream call, maybe a failover to a second provider. Distributed tracing stitches those into one timeline so you can see where the milliseconds and the failures actually went, with trace context propagated to the upstream call.

Logs​

One structured log line per request — trace ID, model, provider, status, latency, token counts, tenant, masked API key, cache status, error code — is the record you grep when something specific went wrong. Structured (JSON), not prose, so it's queryable.

The GenAI semantic conventions​

Here's the piece that's new and worth understanding. OpenTelemetry has standardized a set of GenAI semantic conventions — a common vocabulary for model telemetry, with attributes like gen_ai.system (which provider), gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, and gen_ai.response.model.

Why it matters: if your platform emits these standard attributes on its spans, your traces are portable. GenAI-aware backends — Langfuse, Arize Phoenix, and others — understand them without custom parsing. You're not locked into one vendor's bespoke schema; you're speaking the ecosystem's shared language. The DVARA LLM Gateway emits gen_ai.* alongside its own span attributes, so your governance telemetry travels to whatever backend you standardize on.

What to actually put on the dashboard​

  • Traffic: requests/sec by model and tenant.
  • Latency: p95 by provider (where failover decisions get validated).
  • Tokens: input/output per tenant per day (the cost driver).
  • Errors: provider error rate by code.
  • Resilience: fallback and retry rates by provider pair.
  • Cache: hit rate (ties to cost).

For the request-level view — streaming, structured outputs, and per-call telemetry together — see the deep dive on streaming, structured outputs & observability.

An evaluation checklist​

  • Are Prometheus metrics exposed for rate, latency, tokens, and errors — tagged by model/provider/tenant?
  • Is there distributed tracing with context propagated to upstream calls?
  • Does it emit the OpenTelemetry GenAI conventions (gen_ai.*), so traces are portable?
  • Is there one structured log line per request with the fields you'd need to debug?
  • Are resilience and cache signals (retries, fallbacks, hit rate) observable, not just request/latency?

Where DVARA fits​

DVARA is an AI governance platform, and observability is the evidence layer beneath it: its LLM Gateway exposes Prometheus metrics for request rate, latency percentiles, token usage, cost, and errors — tagged by tenant, model, and provider — and emits OpenTelemetry traces carrying the standard GenAI semantic-convention attributes (gen_ai.*) so they're portable, alongside one structured JSON log line per request. That telemetry is what lets you prove policy, cost, and audit ran. Read the pillar guide, What Is an LLM Gateway?, for how observability fits the rest.