Response Caching

Runs unlicensed

Available with no license, in the Development posture. A license adds the MCP and A2A planes, production rights and support — not this feature.

DVARA is the AI governance platform for LLM and MCP traffic. Response caching is one of the cost-control mechanisms inside that platform: it short-circuits identical or near-identical prompts before they reach an upstream provider, so repeat traffic doesn't pay the same per-token cost twice.

Backends

Backend	Activation	What it serves on
Semantic (active by default)	dvara.llm-gateway.cache.semantic.enabled — defaults to true	An exact cache-key match, unless you also set similarity-serving: true
Exact-match	dvara.llm-gateway.cache.enabled=true	An exact cache-key match

You have a cache whether or not you configured one

The semantic backend's enabled flag defaults to true, so a stock install caches responses with no cache configuration at all. Because similarity-serving is off by default it behaves as an exact-match cache — a repeated identical request is served from it, with X-Cache: HIT and no upstream call.

That is easy to miss and it matters: a cached response is a reused response. It is re-scanned by today's output filters on every hit (see below), it books $0, and it is shared across pods. If you need every call to reach the provider, set dvara.llm-gateway.cache.semantic.enabled=false.

Only one backend is active at a time — the gateway either hashes the request for exact-match lookup or embeds it for similarity lookup, never both.

Both backends are shared across pods. Cached entries live in the same embedded distributed cache that carries rate-limit counters and API-key lookups, so a response cached by pod A is visible to pod B on the next call. No external cache infrastructure is required — pods auto-cluster across the fleet (multicast in dev, headless-Service discovery in Kubernetes; see Rate Limiting for the discovery setup).

Enabling Caching

Set dvara.llm-gateway.cache.enabled=true in application.yml:

dvara:
  llm-gateway:
    cache:
      enabled: true
      ttl-seconds: 3600        # cache entry time-to-live (default: 3600)
      max-size: 10000          # per-node max entries before LRU eviction (default: 10000)

Or via environment variables — Spring Boot's relaxed binding picks these up automatically:

DVARA_LLM_GATEWAY_CACHE_ENABLED=true
DVARA_LLM_GATEWAY_CACHE_TTL_SECONDS=3600
DVARA_LLM_GATEWAY_CACHE_MAX_SIZE=10000

Setting dvara.llm-gateway.cache.enabled=true activates the exact-match backend. Note this is not the switch that turns caching on — the semantic backend is already active by default (see the caution above). A no-op cache is injected only when both backends are off, and then all requests pass through to providers with zero overhead.

Confirm what your install is doing

Three calls settle it, and they need no configuration:

BODY='{"model":"gpt-4o","messages":[{"role":"user","content":"cache probe"}]}'

# 1 — first call: fetched from the provider, then cached
curl -si -X POST http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer $DVARA_API_KEY" -H "Content-Type: application/json" \
  -d "$BODY" | grep -i x-cache

# 2 — same body again: served from cache, no upstream call
curl -si -X POST http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer $DVARA_API_KEY" -H "Content-Type: application/json" \
  -d "$BODY" | grep -i x-cache

# 3 — same body, bypassing
curl -si -X POST http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer $DVARA_API_KEY" -H "Content-Type: application/json" \
  -H "X-Cache-Control: no-cache" -d "$BODY" | grep -i x-cache

X-Cache: MISS
X-Cache: HIT
X-Cache: MISS

Measured on 1.7.0 with no cache properties set at all, which is the point: if you expected three MISSes, the default is not what you thought. Add stream: true to the body and the header disappears entirely — streaming bypasses the cache.

Cache Key Derivation

The cache key is a SHA-256 hash of the canonical request:

Included in the hashed string: model, each message's role, each message's extracted text content (in order), temperature, maxTokens.

Excluded from the hashed string:

• stream — streaming requests bypass the cache entirely before the key is computed
• metadata — transient request-routing hints, not part of the cache identity
• responseFormat, tools, toolChoice — not currently included in the key. Two otherwise-identical requests with different response_format values hash to the same key and can share a cache entry. If this matters for your workload, disable caching per-request with X-Cache-Control: no-cache.
• Non-text content blocks — a non-text image content block is replaced with the placeholder [image:<mediaType>] in the hashed string, so two requests with different images of the same MIME type hash to the same key. Multi-modal callers that rely on cache correctness should send X-Cache-Control: no-cache.

PII stripping before cache lookup and write

Before the cache is consulted, the request is passed through DVARA's PII enforcer, which rewrites any PII tokens in the request text into workspace-specific stable placeholders. The stripped form is then used for both the cache lookup and the cache write. Two requests that differ only in their PII values therefore resolve to the same cache key and share a hit, regardless of whether the workspace's pii.action is LOG, REDACT, or any other non-BLOCK mode. (BLOCK short-circuits the request before it reaches the cache at all.)

This means PII-heavy workloads — the canonical "summarize this support ticket" pattern where hundreds of tickets differ only in identifying fields — collapse onto a single cache entry by default, with no extra knob to flip.

Per-workspace PII configuration controls which entity types get tokenized and how. See PII Detection for the enforcer configuration; the cache uses the same policies with no separate knob.

How It Works

1. A non-streaming request arrives at POST /v1/chat/completions.
2. If the X-Cache-Control: no-cache header is present, the cache lookup is skipped.
3. Otherwise, the gateway looks up the request in the cache.
4. Cache hit: the cached response is run through the output pipeline — response PII scanning, output guardrails, schema validation, grounding detection — and then returned with X-Cache: HIT. No provider call is made.
5. Cache miss: the request is forwarded to the provider. The response is stored in the cache and returned with X-Cache: MISS.
6. Streaming requests ("stream": true) bypass the cache entirely — no lookup, no storage, no X-Cache header.

Cached responses are governed by today's filters, not the ones that stored them

A hit runs the same output filters a miss runs, on the same bytes and at the same point in the pipeline. This matters because the alternative is silent: before 1.7.0 a hit returned before the output filters ran, so a response that passed once was replayed indefinitely — tighten a guardrail, add a PII pattern or activate an output schema, and every cached response went on bypassing all of it.

If the filters refuse a cached response, the entry is evicted as well as refused. Without that the cache would go on storing a response the current policy will never allow, re-scanning and re-refusing it on every request. Eviction is best-effort and never masks the refusal — the error propagates whether or not the entry could be removed.

The cache key does not yet carry a policy or guardrail version, so a filter change does not invalidate the cache wholesale; it is enforced on read, one entry at a time.

Response Headers

Header	Value	When
X-Cache	HIT	Response served from cache
X-Cache	MISS	Response fetched from provider, now cached
(absent)	—	Streaming request (cache bypassed)

Bypassing the Cache

Send the X-Cache-Control: no-cache header to force a fresh provider call:

curl -X POST http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer $DVARA_API_KEY" \
  -H "Content-Type: application/json" \
  -H "X-Cache-Control: no-cache" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "What time is it?"}]
  }'

The response will still be stored in the cache (with X-Cache: MISS), so subsequent requests without the bypass header will hit the cache.

Cache Metering

Cache hits are logged at INFO level with the number of tokens saved:

Cache HIT for key=a1b2c3d4e5f6 — tokens saved: 33

Token usage from cached responses is still recorded for rate-limiting purposes.

Semantic Cache

Validate cache hit rate on your own traffic

The semantic cache defaults to a character-trigram hash embedding, not a neural model. It catches close paraphrases (re-orderings, added pleasantries, minor word swaps) but is not semantically rich: two sentences that mean the same thing but share few characters — "summarize this paragraph" vs. "give me the gist" — will usually miss, and because the embedding is character-based, semantically-different prompts can occasionally collide. For meaning-based hits, opt into the neural embedding below.

Measure hit rate against your actual traffic with the stats endpoint and tune the similarity threshold before relying on it for cost savings. Keep the default default-threshold: 0.92 until you've measured. Workspace isolation is enforced — see Vector store — so a hit for workspace A is bounded to workspace A's cache; the hit-rate ceiling is governed by the embedding quality.

Serving a cached response to a different request is off, and it is a switch rather than a number

By default the semantic cache uses vectors as a cheap prefilter but serves only on an exact cache-key match — a safe near-duplicate and retry cache. Serving a stored response to a similar request requires dvara.llm-gateway.cache.semantic.similarity-serving=true.

It is off on measured evidence, not caution. Cosine similarity over sentence embeddings measures topical relatedness, not semantic equivalence — the right property for retrieval, the wrong one for a cache that serves one response as the answer to another. Measured over a 20-pair corpus with the bundled model, the three highest-scoring pairs were all ones that must not be served — two negations and a word-order swap — and the best genuine paraphrase only ranked fourth.

Precision never exceeds 0.38 at any threshold, and is 0.00 at 0.97. So there is no threshold at which turning it on is better than leaving it off, which is why it is a switch and not a number to tune.

Turn it on only where a topically-similar answer is acceptable, and never where a wrong answer is a correctness or safety problem.

Neural embedding (opt-in)

For meaning-based cache hits — where synonym-substituted prompts that share no characters still match — switch the embedder to the in-process all-MiniLM-L6-v2 model (384-dim, Apache-2.0, runs via ONNX Runtime, no external service):

dvara:
  llm-gateway:
    cache:
      semantic:
        embedding:
          provider: onnx        # default: simple-hash (trigram)
          # model-path: /opt/models/all-MiniLM-L6-v2   # optional; blank = the model bundled in the image

The neural embedder is opt-in so minimal / air-gapped deployments stay dependency-free by default. Switching the provider changes the vector space, so the gateway clears the semantic cache on the change (mixing embedding spaces would corrupt similarity) — expect a cold cache right after the switch. If the model can't be loaded, the gateway logs a warning and falls back to the trigram embedder rather than failing.

When the semantic cache is enabled, it replaces the exact-match backend for chat-completion caching. Instead of hashing the request into an exact-match key, the semantic cache embeds the request text and performs a cosine-similarity search against previously-cached embeddings, scoped to the workspace id from the request's resolved API key. A hit fires when the nearest neighbor's similarity exceeds the configured threshold.

Why fuzzy matching matters

Exact-match caching only hits when two requests produce the same bytes. Chat traffic rarely does — users rephrase, rearrange, add a pleasantry, or shift a word. A semantic cache aims to treat "summarize this paragraph" and "can you give me a short summary of this text" as the same query if the embeddings land close enough in vector space.

By default the embedding is a deterministic character-trigram hash, not a neural model: it works for surface paraphrases that share substrings but not for sentences that mean the same thing in different words. Opt into the neural embedding (all-MiniLM-L6-v2) to catch synonym-substituted queries too. Measure hit rate on your own traffic before relying on the semantic cache for cost savings.

Configuration

dvara:
  llm-gateway:
    cache:
      semantic:
        enabled: true              # default: true
        default-threshold: 0.92    # cosine similarity threshold; lower = more aggressive matching
        max-entries: 10000         # per-workspace cap before oldest-first eviction
        ttl-seconds: 3600          # per-entry TTL; same semantics as exact-match cache
        drift:
          check-schedule: ""       # cron expression for drift detection; blank = disabled
          default-threshold: 0.85  # drift detection threshold (separate from match threshold)

Or via environment variables:

DVARA_LLM_GATEWAY_CACHE_SEMANTIC_ENABLED=true
DVARA_LLM_GATEWAY_CACHE_SEMANTIC_DEFAULT_THRESHOLD=0.92
DVARA_LLM_GATEWAY_CACHE_SEMANTIC_MAX_ENTRIES=10000
DVARA_LLM_GATEWAY_CACHE_SEMANTIC_TTL_SECONDS=3600
DVARA_LLM_GATEWAY_CACHE_SEMANTIC_DRIFT_CHECK_SCHEDULE=""
DVARA_LLM_GATEWAY_CACHE_SEMANTIC_DRIFT_DEFAULT_THRESHOLD=0.85

Per-workspace thresholds

The match threshold is overridable per workspace from the Console's Cache page (/cache). A workspace that prefers precision over hit rate can set a higher threshold (say 0.96) while another workspace chasing cost savings can lower theirs (say 0.85). Threshold resolution walks: workspace-scoped config matching the model pattern → platform-global config matching the model pattern → default-threshold.

Vector store

Embeddings and cached response bodies live in one distributed map per workspace, named dvara-semantic-cache:{tenantId} (the platform-default workspace uses dvara-semantic-cache:_platform_). The store is shared across pods through the same embedded distributed cluster that carries rate-limit counters and API-key lookups, so a hit cached by pod A is visible to pod B on the next call. Workspace isolation is enforced by the per-workspace map name — there is no path for workspace A's lookup to consider workspace B's entries. The store evicts the oldest entry past max-entries and drops entries past ttl-seconds.

Search defaults to brute-force linear cosine over the workspace's map — fine while per-workspace entry counts stay under ~10K, but lookup latency then grows linearly with the entry count. For higher-scale workspaces (especially paired with the neural embedding, which lifts hit rates and entry counts), switch to the HNSW approximate-nearest-neighbor index for sub-linear lookup:

dvara:
  llm-gateway:
    cache:
      semantic:
        index: hnsw          # default: linear (brute force)
        hnsw:
          m: 16                  # graph fan-out
          ef-construction: 200
          ef-search: 50          # higher = better recall, slower query
          rebuild-threshold: 0.3 # rebuild once the index exceeds live capacity by this fraction (reclaims evicted phantoms)

HNSW does nothing unless similarity-serving is on

The index is only consulted when the cache is allowed to serve on similarity. With the default similarity-serving: false the lookup is an exact cache-key map read that returns before any vector search happens — so on a default install index: hnsw builds and maintains a graph that is never queried. Cost, no benefit.

Measured on 1.7.0: with index: hnsw and similarity serving left off, gateway_cache_index_size{workspace="…"} 4.0 is reported while gateway_cache_index_query_seconds is not emitted at all, because nothing ever queries the index.

Turn HNSW on when — and only when — you have set dvara.llm-gateway.cache.semantic.similarity-serving: true (see Serving a cached response to a different request, above, for why that is off by default) and have enough entries per workspace for the linear scan to hurt.

The HNSW index is per-pod, in-process and sits over the same per-workspace map, which stays the source of truth: each pod builds its index from the map on startup (so a rolling restart loses no state), an entry-listener propagates other pods' writes into the local index, and evicted entries are filtered out at query time (the map is authoritative). Recall is ≥ 0.95 vs. brute force on the defaults.

index=linear remains the default until you have validated recall and the per-workspace index metrics on your own traffic:

• gateway_cache_index_size{workspace} — gauge, entries in this pod's index
• gateway_cache_index_query_seconds{workspace} — histogram, only recorded when a vector query actually runs, i.e. with similarity-serving: true

Note the label is workspace on both, matching gateway_semantic_cache_hits_total{workspace}.

Stats

Hit rate is a gateway metric, not a Console one, and since 1.7.0 the Console says so rather than showing a number.

The counters live inside each gateway process. The Console talks to the shared store, so it can report cache size accurately and can clear the cache — but it cannot see another process's counters. It used to render a hit-rate card anyway, which was structurally 0% on every install; it now renders "— not measurable from the Console" for hit rate, average similarity and total lookups, and names where they are. A screenshot showing a Console hit-rate percentage predates 1.7.0.

Scrape them from the data plane instead. Two counters are emitted on every lookup:

• gateway_semantic_cache_hits_total{workspace} — incremented on each cache hit
• gateway_semantic_cache_misses_total{workspace} — incremented on each cache miss (including paths that bypass the embed step)

Measured on a 1.7.0 gateway after four identical prompts (one miss, three hits):

gateway_semantic_cache_hits_total{workspace="ws-demo-platform"} 3.0
gateway_semantic_cache_misses_total{workspace="ws-demo-platform"} 2.0

The label is workspace — the resolved workspace id, or _platform_ for calls with no workspace (admin tooling, internal probes). Graph the ratio per workspace to find the ones whose threshold needs tuning, and validate the default threshold against your own traffic before rolling out per-workspace overrides.

Note that a fleet has one of these per gateway process, so a fleet-wide hit rate is a sum() across instances — which is the other reason the Console does not attempt it.

Distributed cache for API-key lookups

Multi-instance deployments share API-key lookups through an embedded distributed cache that runs inside the gateway process — no external cache infrastructure required. Pods auto-cluster across the fleet, so an API key resolved once on pod A is sub-millisecond on pod B too. Reads check the distributed map first and fall back to PostgreSQL; writes update PostgreSQL and immediately evict the cached entry across the cluster so revoked or rotated keys are never served stale.

The same distributed cache also carries per-key rate-limit counters. Discovery, headless-Service setup, and the CACHE_SERVICE_NAME environment variable are documented once on Rate Limiting.