Skip to main content

Semantic Caching for LLMs: $0 Cache Hits, PII-Safe by Default

· 5 min read
DVARA Team
DVARA Team

Traditional caching keys on an exact match: same input, same cached output. That works for HTTP and falls apart for language, because two prompts can be worded completely differently and mean exactly the same thing. "What's your refund policy?" and "How do I get my money back?" would miss an exact-match cache every time — and you'd pay to generate the same answer twice.

Semantic caching keys on meaning instead of bytes. But a cache that stores prompts and responses is also a governance surface: it can quietly hold raw PII, and it can quietly keep charging you for calls that never happened. Both are failure modes a governance platform has to close by default — which is why, in DVARA, the semantic cache lives inside the same governed path as Policy-as-Code and PII redaction, not off to the side.

What semantic caching is​

Semantic caching stores prompt–response pairs and serves a cached response when a new prompt is semantically similar to a stored one — close in meaning, not identical in text. Instead of hashing the prompt, it embeds the prompt into a vector and compares that vector against cached prompts by cosine similarity. If the closest match is above a similarity threshold, it's a hit and you return the stored answer. Below the threshold, it's a miss and the call goes to the model.

How it works, step by step​

  1. Embed the incoming prompt into a vector.
  2. Compare it against the vectors of cached prompts (cosine similarity).
  3. Threshold — if the best match scores above your similarity threshold (say 0.9), serve the cached response.
  4. On a miss, call the model, then store the new prompt vector and response for next time.

The similarity threshold is the dial that matters. Too low and you serve near-misses that aren't really the same question (false hits). Too high and you behave like an exact-match cache and rarely hit. It's tunable per pattern and per tenant — an FAQ bot can be looser than a code assistant.

PII-safe by default​

Here's the governance failure mode a naive cache walks into: it becomes a store of raw sensitive data. If prompts or responses can contain PII, and you cache them as-is, your cache is now an unaudited copy of the very data your policies exist to protect.

In a governance platform, PII is stripped before anything is written to the cache — the redaction that runs on the request path also runs before the cache write, so the cache never holds raw sensitive data. Caching is a performance feature that must not become a compliance hole; closing that gap by default is the difference between a cache and a governed cache.

Why a cache hit must cost $0​

The other quiet failure: billing. When you serve a response from cache, no upstream API call happened — so that request should cost you nothing. A cache that still books the original call's cost against your budget on every hit erases the savings the cache exists to create.

Done right, a cache hit is recorded (you want the token volume for reporting) but billed at $0, because there was no upstream spend. That distinction — served volume vs. actual upstream cost — is what makes caching a real cost governance lever and not just a latency trick. On a workload with repetitive prompts, it's one of the largest cost reductions available.

The latency win​

Cost is the headline, but hits are also fast. A cached response skips the round trip to the provider entirely — sub-hundred-millisecond instead of multi-second. For high-repetition surfaces (support bots, autocomplete, templated queries), that's a materially better user experience on top of the savings.

Trade-offs to respect​

  • Staleness. A cached answer can go out of date. TTLs and per-pattern scoping keep it fresh.
  • False hits. Two prompts can be close without being the same. The similarity threshold, tuned per use case, is your control.
  • Personalization and correctness. Don't cache responses that are user-specific or must be freshly computed. Scope caching to surfaces where a shared answer is genuinely correct.

An evaluation checklist​

  • Does it match on meaning (embeddings + similarity), or only exact text?
  • Is the similarity threshold tunable per pattern/tenant?
  • Is PII stripped before caching, so the cache isn't a sensitive-data store?
  • Does a cache hit book $0 cost, or does it re-charge the original call?
  • Are there TTLs and per-pattern scoping to control staleness?

Where DVARA fits​

DVARA does per-tenant semantic caching inside its governed path: a tunable similarity threshold and TTLs, PII stripped before anything is cached, and a cache hit that records served token volume but books $0 cost — no upstream call, no charge. It's a cost-and-latency win that stays a governance-safe surface. See how it ties into spend in the LLM cost management deep dive, or the pillar, What Is an LLM Gateway?.