Version: 1.10.0-dev

Open-source gateway configuration

Reference for the open-source DVARA LLM Gateway, run from one gateway.yaml. The quickstart gets it running.

One file, read once at startup; change it and restart. ${VAR} and ${VAR:-default} resolve from the environment. The quickstart file, extended:

providers:
  - type: openai
    api_key: ${OPENAI_API_KEY}

workspaces:
  - id: default
    metadata:
      pii.action: REDACT
      guardrail.action: BLOCK

  - id: research
    credentials:
      provider.openai.api-key: ${RESEARCH_OPENAI_KEY}

routes:
  - id: default
    model: "gpt*"
    provider: openai

api_keys:
  - key_hash: sha256:9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08
    workspace: default

policies:
  - id: approved-models-only
    workspace: default
    status: ACTIVE
    dsl: |
      version: "1"
      rules:
        - id: deny-gpt-35
          conditions:
            model:
              denylist: [gpt-3.5-turbo]
          action: DENY
          deny_message: "gpt-3.5-turbo is not approved"

Without the two metadata keys, PII and guardrail detections are logged and the request goes through; REDACT and BLOCK are the strict choices. A route's model takes a trailing *; a policy allowlist or denylist matches model names exactly, so list each name. A max_tokens condition applies only to a request that sets max_tokens or max_completion_tokens. A rule's action is DENY or WARN_AGENT.

A policy applies to the model the gateway actually sends, not only the model the caller named. When a route sends another model (a strategy pins a version or picks a tier), the policy is checked again on that model. If it denies it, the caller gets the same 403 policy_denied and one POLICY_DENIED audit event, and no provider is called, unless a route backup the policy allows takes the request. This changed in 1.8.6: before, only the named model was checked, so a request the policy allows by name but whose route sends a model the policy denies now fails where it used to be served. Add the model the route sends to the allowlist, or take it off the denylist, to keep serving it.

Anything Spring Boot accepts works the same way here: --server.port=9090 on the command line, or SERVER_PORT=9090 in the environment.

Failover to another provider

A route can name where its requests go when the provider serving them fails: an ordered chain of provider and model. Say which model each backup is asked for; the gateway never picks one for you.

providers:
  - type: openai
    api_key: ${OPENAI_API_KEY}
  - type: anthropic
    api_key: ${ANTHROPIC_API_KEY}

routes:
  - id: support-assistant
    model: gpt-4o
    provider: openai
    fallbacks:
      - provider: anthropic
        model: claude-sonnet-4-5

The application keeps sending gpt-4o:

curl -s http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer $DVARA_KEY" -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o","messages":[{"role":"user","content":"Reply with the word ready."}]}'

If OpenAI fails with a server error, an open circuit or a rate limit, after its retries, the same request goes to Anthropic as claude-sonnet-4-5, on Anthropic's own key. The answer is an ordinary chat completion whose model is the model that served it (claude-sonnet-4-5). The gateway logs Route [support-assistant]: falling back from [openai] to anthropic/claude-sonnet-4-5, and counts it in gateway_fallbacks_total{from_provider="openai",to_provider="anthropic"}.

Only the route's own chain is tried, in order. A route without fallbacks does not fail over. A request that matches no route still falls back to another configured provider serving the same model.
A backup is skipped if it doesn't serve its model, can't take what the request needs (tools, images, a JSON schema), is paused by its circuit breaker, or the workspace's policy doesn't allow its model. If no backup can take the request, the answer is 503 failover_capability_mismatch; if every backup that was tried failed, the primary's error.
A request the provider rejected (400), a policy denial and an authentication failure never fail over.
A stream fails over only if it fails to open. Once it has started, an error ends it.
A target without model is asked for the request's own model, for a provider serving the same one (OpenAI and Azure OpenAI). fallback: <provider> is the same as a one-entry chain without a model.
At most dvara.llm-gateway.resilience.fallback.max-attempts backups are tried (default 3), none starting more than ...fallback.deadline after the first attempt (default 60s). A chain may name five. ...fallback.enabled: false turns failover off.
The workspace's policy also checks the model sent to the first provider (see above).
An application that embeds the gateway can check the first provider in other ways, with its own PrimaryTargetGuard bean. It is asked about the request with the model the route actually sends, which may not be the one the caller named (a strategy can pin a version or pick a tier). If it refuses, the route's chain is tried; if no backup takes the request, the caller gets the guard's error. The refused provider is never called.

Model context windows

Before a request is sent, the gateway estimates its size and compares it with the context window of the model that will serve it. At 90% of the window it refuses the request with 400 context_window_exceeded, or trims it if the workspace asks for that. Each provider declares one window for all its models (OpenAI declares 128,000 tokens), and many models accept more. Write a model's own window in model_limits, and the gateway uses it instead:

model_limits:
  - model: gpt-4.1          # exactly this model
    context_tokens: 1047576
  - model: "gpt-4.1-*"      # a trailing * matches every model with that prefix
    context_tokens: 1047576
  - model: claude-sonnet-4-5
    provider: anthropic     # optional: only when this provider serves it
    context_tokens: 1000000
    output_tokens: 64000    # optional: the most it can write in one answer

A model with no entry keeps its provider's window.
When several entries match, the most specific wins: an exact model over a prefix, a longer prefix over a shorter one, and an entry naming the provider over one that does not.
If a request could be served by more than one provider (failover), the smallest window among them applies.
An entry without model, or with neither a positive context_tokens nor a positive output_tokens, stops the gateway at startup. An entry may set either or both; one it leaves out comes from the next source.
The same list can be set as Spring properties, dvara.llm-gateway.model-limits[0].model and ...[0].context-tokens. An application that embeds the gateway can also supply its own ModelContextLimits bean, for example one that reads a model catalogue. Entries written here win over it.
GET /v1/models shows the same window for each model, in capabilities.max_context_tokens, and the model's output limit in capabilities.max_output_tokens where a source knows it. Providers declare no output limit, so a model no source knows is listed without one.
The guardrail's size limits apply first: by default a single message longer than 50,000 characters is refused with 413. The number of messages in a request has no cap by default, so an agent's long conversation is bounded by the model's window, not by a count. dvara.llm-gateway.guardrail.max-message-length and ...max-messages-per-request change them.

Counting tokens

The gateway counts a request's tokens itself, for the rate limits, the context-window check and the usage of a stream that reports none. It counts each model by its family:

OpenAI models with their own vocabulary: o200k_base for GPT-4o, GPT-4.1, GPT-5, the o-series and gpt-oss, cl100k_base for GPT-4 and GPT-3.5.
Every other family (Anthropic, Gemini, Mistral, DeepSeek, Llama, Qwen, Cohere, Grok) with cl100k_base as a stand-in, since their own tokenizers are not available in the gateway. A model no family claims counts the same way.

How close that is to what a provider bills was measured on 2026-10-16 over four samples (TokenCountAccuracyTest):

Provider	Plain text	A request with tools
OpenAI (gpt-4.1-mini)	Exact on prose, code and text in seven scripts	19% over: OpenAI encodes tool definitions more compactly than JSON
Anthropic (claude-haiku-4-5), raw stand-in count	From 21% under (code) to 11% over (non-Latin scripts)	61% under: Anthropic adds its own text for tools
Anthropic, with the measured correction	Within the same spread; under 1% over across the three texts together	Exact on the sample

So the gateway ships two measured defaults for Anthropic: its count is multiplied by 1.05, and a request with tools adds 540 tokens. Other families have no measurement yet and are counted as they are. An over-count is the safe direction for a limit; an under-count lets through a request that costs more than the limit allows.

The settings:

dvara:
  llm-gateway:
    token-count:
      corrections:            # multiply a family's count; replaces the measured default for that family
        anthropic: 1.05
      tool-overhead:          # tokens a request with tools adds; replaces the measured default for that family
        anthropic: 540
      vendor:
        enabled: false        # false, true (every request) or selected (the requests a selector picks)
        timeout: 300ms        # not waited for longer than this
        ttl: 60s              # a request's count is kept this long
        cache-size: 10000

A correction should come from a measurement, not a guess. TokenCountAccuracyTest in dvara-gateway-runtime compares the raw and corrected counts with each vendor's count and with what the provider billed, over samples in the repo: run it with -Ddvara.token-accuracy=true and the providers' API keys in the environment. It fails when a key is set and no call with it succeeds, so a wrong key or a retired model is not read as a result.
The vendor count is the exact one for Anthropic and Gemini: it asks the provider (count_tokens, countTokens), which bill nothing for it. Anthropic's count matched what it billed on every sample. It is a network call on the request path, so it is off by default, cached, and gives up after the timeout. On any failure, or for a provider with no counting endpoint, or before the request's workspace is known, the local count is used: a count never fails a request. It sends the request's text to the provider under the credential the call would use.
selected asks the vendor only for the requests a VendorTokenCountSelector bean picks, for example the ones checked against a cost limit before they are sent. Every other request keeps the local count and pays no network call. With no selector registered, no request is picked.
To count some family another way, register your own TokenizerResolver bean.

API keys

Every request under /v1 carries an API key, and a request without one is refused with 401. The key resolves to a workspace, and the workspace is what every control is scoped to: the PII action, the guardrail action, the policies, the rate-limit override, the provider credentials, and whose batch jobs and cached answers are whose. There is no setting that serves a keyless request; a gateway with no keys configured starts, and refuses every call until one is minted.

The gateway never generates a key at runtime. The operator mints one with --generate-key, which prints it once and stores nothing; the file holds only its SHA-256. The one path that takes no key is the webhook approval action, which carries its own signed token.

dvara.llm-gateway.data-plane.require-api-key, which once allowed keyless requests, was removed in 1.8.0. The gateway refuses to start if it is set to false, so a deployment that relied on it learns at startup rather than from its callers; set to true it starts and says the line can go.

Mint a key, fingerprint a key you already hold, and call the gateway with it

Mint a key, and the fingerprint that goes in the file:

java -jar …-app.jar --generate-key --name ci --workspace default

  Key — give this to whoever calls the gateway.
  It is not stored anywhere and cannot be recovered. If you lose it, mint another.

    gw_16df9718bd08f34d37e8d118db5bd815714249aa

  Add to gateway.yaml:

    api_keys:
      - key_hash: sha256:d11765e5979c1a3b76a659293ea6db8cd39cb65cc001998422c1d872419249dd
        name: "ci"
        workspace: "default"

The file holds the SHA-256 of the key, never the key. The key is printed once and stored nowhere; lose it and you mint another.

To fingerprint a key you already hold:

printf '%s' "$KEY" | java -jar …-app.jar --hash-key -
java -jar …-app.jar --hash-key-file /run/secrets/gateway-key

Give the key on standard input or name a file that holds it. --hash-key <key> as an argument works for one more release and warns. These commands print and exit without starting the gateway, so they work while it is running. Then call it with the key:

curl http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer gw_16df9718bd08f34d37e8d118db5bd815714249aa" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Hello"}]}'

Revoking a key is editing the file and restarting. A key from this file never expires.

Failover to another provider​

Model context windows​

Counting tokens​

API keys​

Failover to another provider

Model context windows

Counting tokens

API keys