Put model traffic behind one governed endpoint. One OpenAI-compatible API in front of your providers.
The DVARA LLM Gateway is the governed data plane for model calls. Your applications get one OpenAI-compatible endpoint instead of an integration per provider — and because every model call now runs through one place, budget, policy, inspection, routing and evidence become properties of the platform rather than code in each application.
The gateway boots with no licence key; the governance engines need one — see pricing.
Why doesn't integrating providers one by one scale?
Wire each application straight to each provider and you own an SDK per provider, hand-rolled retry and failover, no shared view of cost or tokens, and governance scattered across services with no single place to enforce it. An LLM gateway is the layer that consolidates that: one endpoint your applications call, which understands what is inside an AI request — models, tokens, prompts, tools — and can therefore route, budget, inspect and record on attributes an ordinary API gateway never sees.
What does your application actually talk to?
One endpoint speaking the OpenAI API shapes, in front of fourteen first-class providers and any other OpenAI-compatible endpoint you configure. Switching models is a string change; switching providers is a route change.
Point an OpenAI SDK at DVARA by changing the base URL. Chat completions, embeddings and models are there, and so are the Responses API, the Batch API and files — the same shapes your client already speaks.
OpenAI, Anthropic, Gemini, Bedrock, Azure OpenAI, Mistral, Cohere, Groq, Ollama, Qwen, DeepSeek, Moonshot, ChatGLM and Grok, plus a Mock provider for CI. Other OpenAI-compatible endpoints route through the same pipeline once you configure them.
SSE streaming, and response_format translated to whatever each provider actually implements — native where it exists, a tool-use rewrite where it does not, and a clear error where the provider cannot do it at all.
Bring your own provider keys. Credentials resolve per tenant first, then the platform default, then a vault, then the environment — and strict BYOK refuses the platform fallback so a tenant can only ever spend its own key.
How does DVARA choose a provider, and what happens when one fails?
A route matches the model and its strategy picks from the pool, with providers that cannot honour the request filtered out first. On failure the gateway moves to a healthy capable peer — and where none is capable, it says so rather than substituting one that is not.
Model-prefix and round-robin run keyless. Weighted, latency-aware, cost-aware, canary, geo-aware and intelligent routing are licensed. Routes are stored centrally and take effect on the next request without a restart.
A provider that cannot honour the requested response format is dropped from the candidate list before anything is chosen. If none on the route can, the call stops with a 400 rather than a degraded answer — no upstream hop.
Every provider is wrapped in retry, timeout and a circuit breaker. On failure the gateway moves to a healthy capable peer on the same route and skips candidates whose own circuit is open. These values are read at boot, so changing them is a deploy.
Failover re-applies the capability and region checks on every retry, so an incapable fallback is never quietly substituted. When no capable peer exists you get a 503 and a header saying failover was blocked — the honest failure, not a wrong answer.
What runs before a call reaches a provider?
Budget, policy, PII inspection and guardrails — in that order, all of them before egress, and each doing what that tenant configured it to do. Any of them can stop the call while stopping it is still free.
Budget, policy, PII scan, guardrails, then routing — in that order, before the provider hop. On the way back: schema validation, grounding, guardrails, PII again, then the audit record.
YAML DENY and WARN_AGENT rules with conditions on model, time of day, residency and budget use, or a CEL expression where the matchers cannot compose it. Shadow status runs a policy against live traffic before you make it active.
Checksum-validated regex detection runs keyless with log or block. Reversible redaction, tokenization and NER models are licensed. Guardrails — injection, jailbreak, prompt leakage, output sanitization — are licensed, and each tenant sets block, flag, or log.
Where region constraints are configured, a call that no allowed region can serve is refused with a residency violation rather than routed somewhere it should not go.
What do you get back from every call?
A priced, attributed, recorded call. Spend enforced rather than observed, the metrics and traces your stack already ingests, and an audit record of what was decided and what happened.
Hard and soft caps at the global, tenant and API-key level. A hard breach rejects the request; a soft breach alerts and downgrades the model. A per-call cost ceiling is available and is off by default.
Exact-match by default, with semantic matching as an opt-in that replaces it. PII is tokenized before the key is computed, so lookups and writes use the stripped form, and a tenant’s cache is bounded to that tenant. Measure hit rate on your own traffic before counting on it.
Tokens, cost, latency, provider and model recorded per call and attributable per tenant and key. Prometheus metrics and OpenTelemetry traces go to the stack you already run.
An append-only record of each request, response and admin action. Keyless, that is a local unsigned log held about a week. Licensed, records are HMAC-signed and hash-chained, a scheduled job re-verifies the chain, and they feed webhooks, SIEM export and compliance reports.
How does a request flow through the gateway?
Six stages, and where a stage sits is the point: everything through step three happens before the provider hop, so a call that fails one of them never leaves. Only the last two see a result.
- 1
Your application calls DVARA
An OpenAI-compatible request goes to the gateway on :8080 instead of the provider. It is authenticated, resolved to a tenant and key, and counted against that key’s rate limits.
- 2
Budget, policy, inspection
Budget is checked, policy is evaluated against the request, and the configured PII and guardrail stages inspect it. Any of them can stop the call here — before it costs anything and before it leaves.
- 3
A provider is selected
The route’s strategy picks from its pool, with providers that cannot honour the request filtered out first. Where caching is enabled, a valid hit is served here and the upstream hop never happens.
- 4
One upstream hop
DVARA translates the request to that provider’s API and makes a single call, using the credential resolved for this tenant. On failure it moves to a healthy capable peer and re-checks capability and region.
- 5
The response is inspected on the way back
Schema validation, grounding checks, guardrails, and a second PII pass run on the response — each doing what that tenant configured it to do.
- 6
The call is recorded
Tokens, cost, latency, provider, model, the decisions taken and the outcome are written to the audit record and exposed as metrics and a trace. A trace ID comes back on every response.
How does the LLM Gateway fit with the rest of DVARA?
Three governed data planes and one control plane, deployed and scaled separately. Operators change governance in the control plane; the data planes read the change and enforce it on the request path.
This page. The governed data plane for model calls: your applications in, one provider hop out.
The same governance model applied to tool calls — the same policy engine and filter stages, run before a tool executes rather than before a provider call.
An opt-in data plane for agent-to-agent hops, governed per hop with its own audit chain.
The control plane. Routes, policies, credentials, budgets and reports are changed here; the data planes pick the change up on their next config read, without a restart.
Where do you see the providers, routes, and spend?
In DVARA Flightdeck, the console that ships with the platform. It shows traffic per provider, which route each call took, and what it cost — without standing up a separate observability stack first.
How does DVARA compare with the gateways it replaces?
Against the three alternatives that score strongest on core gateway and routing in the matrix behind how DVARA compares — chosen by score, not by how well DVARA does against them. Each row is checkable against that product's own public documentation.
| Capability | DVARA | Portkey | LiteLLM | Bifrost |
|---|---|---|---|---|
| OpenAI-compatible unified API | ✓ | ✓ | ✓ | ✓ |
| SSE streaming (zero-copy pass-through) | ✓ | ✓ | ✓ | ✓ |
| Structured outputs across providers | ✓ | ∼ | ∼ | — |
| Multi-provider (14+: OpenAI, Anthropic, Gemini, Bedrock, Azure, Ollama, Mistral, Cohere, Groq, Qwen, DeepSeek, Moonshot, ChatGLM, Grok) | ✓ | ✓ | ✓ | ✓ |
| Weighted routing + failover | ✓ | ✓ | ✓ | ✓ |
| Latency-aware routing (EWMA P95) | ✓ | ∼ | ∼ | ∼ |
| Pricing model | Licensed, sized by request volume — no hard stop on overage | Subscription from $49/mo + per-request overage (source) | Usage-priced per call (Cloud); OSS proxy self-hosted (source) | See their pricing |
Capabilities last verified June 2026; pricing models August 2026. A stale mark is worse than no claim — every row is checkable against each product's own public documentation. For the full matrix across ten alternatives, see how DVARA compares.
What do you need to run the LLM Gateway?
The gateway itself needs nothing — it boots from public images with no key. What needs a licence is most of what makes it a governance gateway rather than a router. See what a licence covers.
The gateway and the console boot from public images with no key: the OpenAI-compatible API, the providers, model-prefix and round-robin routing, streaming, structured outputs, retry and failover, and multi-tenancy.
The governance engines: latency-, cost-, canary- and geo-aware routing, guardrails, reversible PII redaction and NER, budgets and FinOps, response caching, signed and chained audit with SIEM export and compliance reports, and SSO, mTLS and vault-backed credentials.
The DVARA MCP Gateway and A2A Gateway have no keyless tier at all. Their images are private and they refuse to start without a valid key.
Common questions about the DVARA LLM Gateway
What does the LLM Gateway govern?
Model calls made through it. Each one is authenticated to a tenant and key, checked against budget and policy, inspected by whichever PII and guardrail stages that tenant configured, routed to one provider, and recorded. A call your application makes straight to a provider is not in that path.
Is it OpenAI-compatible, and does my app need provider-specific code?
It exposes the OpenAI API shapes — chat completions, embeddings, models, the Responses API, and the Batch API with files — so an existing SDK works by changing the base URL. Your application does not integrate providers individually; DVARA translates to each provider’s own API and hands you back the OpenAI shape.
Which providers are supported?
Fourteen first-class: OpenAI, Anthropic, Gemini, Bedrock, Azure OpenAI, Mistral, Cohere, Groq, Ollama, Qwen, DeepSeek, Moonshot, ChatGLM and Grok — plus a Mock provider for CI. Any other OpenAI-compatible endpoint routes through the same pipeline once you configure a route or base URL for it. Providers differ in what they support, so the gateway declares each one’s capabilities and routes on them.
How do routing, fallback, and failover work?
A route matches the request’s model and applies one of eight strategies to pick from its pool. Model-prefix and round-robin run keyless; latency-, cost-, canary- and geo-aware routing are licensed. Providers that cannot honour the request are filtered out before selection, and on a failure or an open circuit the gateway moves to a healthy capable peer, re-checking capability and region. If no capable peer exists it returns a 503 saying so rather than substituting one that cannot do the job.
Can budgets apply per tenant, per key, or per call?
All three. Hard and soft caps sit at the global, tenant and API-key level: a hard breach rejects the request, a soft breach alerts and downgrades the model. A per-call cost ceiling exists as well and is off by default. Budgets are a licensed capability.
What happens to prompts and responses?
They pass through the gateway to the provider you routed to — that is the job, so they do leave your network on the upstream hop. What DVARA controls is what happens first: policy evaluation, and PII and guardrail inspection configured per tenant, which can block a call, redact values, or record the detection before egress. The response is inspected again on the way back.
Can it be self-hosted with our own provider keys?
Yes. You run it in your own environment and bring your own keys. Credentials resolve per tenant, then the platform default, then a vault, then the environment, and strict BYOK refuses the platform fallback so a tenant can only spend its own key. Self-hosting decides where the gateway runs and where your keys are held; it does not change the fact that the call to a hosted provider goes to that provider.
How does caching handle sensitive prompts?
PII is tokenized before the cache key is computed, so both the lookup and the write use the stripped form and two requests differing only in identifying values share an entry. A tenant’s cache is bounded to that tenant. Blocking PII short-circuits the request before the cache is reached at all. Caching is licensed, exact-match by default, with semantic matching as an opt-in — measure hit rate on your own traffic before relying on it.
How are model calls connected to agent sessions?
Send an X-Session-Id header and DVARA groups the requests sharing it into one agentic session, which is also what activates loop detection and session-level audit grouping. The same header on MCP tool calls ties a model turn and the tool calls under it together. Without the header the call is still recorded and traced, just not grouped.
Which edition includes which capability?
The gateway and console boot with no licence key: the OpenAI-compatible API, providers, model-prefix and round-robin routing, streaming, structured outputs, resilience, multi-tenancy, regex PII detection and a local unsigned audit log. A licence key adds the governance engines — advanced routing, guardrails, reversible redaction, budgets, caching, signed and chained audit, SIEM export, compliance reports and SSO. The MCP and A2A Gateways are licensed-only.
Guides on running an LLM gateway
Why API gateways like Kong and NGINX can’t govern LLM and agent traffic — and the six gaps a purpose-built AI gateway closes.
Beyond the basics — streaming responses, JSON-schema-enforced structured outputs, and per-call token/cost/latency telemetry.
Point the OpenAI SDK you already have at DVARA and call every provider through one endpoint — a five-minute switch.
One governed API in front of every model — model-prefix, weighted, and capability-aware routing, with policy, cost, and PII applied on the same path.
Governed resilience: circuit breakers and capability-aware failover that keep AI traffic serving — and audited — when a provider degrades.
Attribution and enforcement, not just dashboards — real-time cost attribution, hard budget caps, model downgrade, and chargeback.
Governance that runs inside your perimeter — policy, PII, and audit before egress, with BYOK and vault-backed credentials.
Serve semantically similar prompts for $0, PII-safe by default — the cost-and-latency win that stays a governance-safe surface.
The telemetry layer under governance — Prometheus metrics, OpenTelemetry traces with the GenAI semantic conventions, and per-request structured logs.
A governance-first comparison framework — the categories on the market and the criteria that matter, weighted toward enforcement over dashboards.
Put one application's model calls behind the gateway.
Run it with no key, point one application at it, and add the governance engines when you want policy, budgets and signed audit on the same path. Or read how routing and the request pipeline work.