How to Compare LLM Gateways
"Which LLM gateway should we use?" doesn't have a universal answer, because teams optimize for different things — fastest to adopt, cheapest to run, most observable, or most governable. What is universal is the set of questions worth asking, and the order to ask them in.
This is a framework, not a leaderboard. And it's deliberately governance-first, because that's the axis where products differ most and demo videos reveal least. Routing and provider coverage are easy to compare and roughly commoditized; whether a platform can enforce a policy, prove what it did, and keep sensitive data in your perimeter is where the real divergence — and the real buying decision — lives. DVARA is one option in the field; the point of this post is to help you score the field, including us, on criteria that matter.
The categories on the market
Broadly, the options fall into a few shapes, each with a different center of gravity:
- Open-source proxies / libraries (e.g. LiteLLM, Bifrost). Fast to start, free to run, huge provider coverage. You operate and secure them yourself, and governance depth — Policy-as-Code, tamper-evident audit, PII enforcement — is usually thin or DIY. Great when routing and provider abstraction are the whole ask.
- Cloud-vendor gateways (e.g. Cloudflare AI Gateway, Kong's AI Gateway add-on). If you're all-in on one ecosystem, the native integration is tight. The trade-off is gravity toward that vendor and weaker neutrality — and governance that's often coarse.
- Developer/observability-first platforms (e.g. Portkey, Helicone). Strong on tracing, evals, and prompt management, with a gateway attached. (Since this was written, Helicone was acquired by Mintlify in March 2026 and has moved to maintenance mode, with its experiments feature withdrawn — read it as a description of the category rather than a current recommendation.) A good fit if your primary pain is visibility; typically lighter on hard enforcement (blocking budget caps, policy denial, PII blocking) and on self-hosting for regulated data.
- Governance-native platforms. Built around Policy-as-Code, PII/guardrails, tamper-evident audit, agentic (MCP) governance, and self-hosting from the start, with routing and cost as table stakes. The fit when compliance and control are the point. DVARA sits here — and the honest framing is that most of the field started as a gateway and added governance, while a governance-native platform starts from the control requirements and ships the gateway as the mechanism.
No category is "best." They're answers to different questions.
This post previously described DVARA's audit trail as immutable, including in the comparison table below. That was wrong when it was written, and it is corrected throughout.
The mechanism is HMAC-SHA256 signing plus hash chaining, which makes alteration and removal detectable — it does not prevent them. The trail is append-only through the API; anyone with direct database access can still delete a row, and what the chain guarantees is that you will know. That stronger word would be a claim about storage and permissions, which is your deployment rather than our product. The correct word, used here now, is tamper-evident.
It mattered most in a comparison, which is why the note is on this post: a criterion you are asked to judge other products against has to be stated accurately about ours first.
The criteria that matter — governance first
Score any candidate on these, weighted by what your team is actually optimizing for. Governance rows lead deliberately:
| Criterion | What to ask |
|---|---|
| Policy enforcement | Policy-as-Code that can deny a request (model, tools, region, budget), or only observe? |
| Audit | Tamper-evident trail of every decision — or just request logs? |
| PII / guardrails | Prompt & response scanning that can block or redact, in-perimeter? |
| Agentic (MCP) governance | Can it govern tool calls — approval gates, loop detection, tool-poisoning defense — not just model calls? |
| Data residency / self-hosting | Runs inside your perimeter, or SaaS-only? |
| Cost control | Real-time attribution and hard budget caps, or dashboards only? |
| Caching | Semantic caching, and does a hit cost $0 and stay PII-safe? |
| Routing & failover | Model-prefix, weighted, and capability-aware failover — or blind retries? |
| Provider coverage | The models you use and the ones you might switch to? |
| Observability | Prometheus + OpenTelemetry (incl. GenAI conventions), or a closed dashboard? |
A few deserve emphasis because they're where products quietly differ:
- Enforcement vs. observation. "We show you your spend / your prompts / your traces" is not "we can stop a disallowed call." Hard caps that block and policies that deny are a different capability class than dashboards that inform. → Cost Management
- Where it runs. SaaS is fastest; self-hosting keeps prompts, policy enforcement, and audit in your perimeter. For regulated data this is often the first filter, not the last. → Self-Hosted LLM Gateway
- Model-aware governance. A repurposed API gateway can't see models, tokens, or PII. If governance matters, that gap is disqualifying. → Why an AI Gateway, Not an API Gateway
- Capability-aware failover. Blind failover can retry on a model that can't serve the request. Ask specifically. → Fallback & Failover
A note on MCP and agents
By 2026, governing agent traffic — not just model calls but the tool calls agents make via the Model Context Protocol — is a real axis of comparison, and most serious gateways now offer some MCP support. The differentiator isn't presence, it's depth: whether MCP is governed (approval gates, loop detection, tool-poisoning defense, per-call audit) or just proxied. If you're building agents, weight this heavily; if you're not yet, treat it as forward-looking rather than deciding.
How to actually choose
- Write down your top three constraints. Regulated data? Multi-cloud? Tiny ops team? These pick your category before you compare products.
- Disqualify on the hard requirements. SaaS-only is a non-starter for some; that filter shrinks the list fast.
- Trial two candidates on real traffic. Routing and coverage demo easily; enforcement, PII blocking, audit integrity, and self-hosting only reveal themselves under load.
- Weight enforcement over dashboards if control is why you're here.
Where DVARA fits
DVARA is a governance-native, self-hostable AI governance platform: Policy-as-Code that denies, tamper-evident HMAC-signed audit, PII redaction and guardrails, governed MCP for agent traffic, and data residency by self-hosting — with capability-aware routing and failover, real-time cost attribution and hard budget caps, and $0 PII-safe semantic caching as table stakes. It's the right fit when compliance, control, and sovereignty lead your criteria; if your top constraint is "free and OSS with the widest provider list," an open-source proxy may serve you better. Score it against your own weighted criteria using the pillar guide, What Is an LLM Gateway?, as the reference.