Open-source gateway configuration

Reference for the open-source DVARA LLM Gateway, run from one gateway.yaml. The quickstart gets it running.

One file, read once at startup; change it and restart. ${VAR} and ${VAR:-default} resolve from the environment. The quickstart file, extended:

providers:
  - type: openai
    api_key: ${OPENAI_API_KEY}

workspaces:
  - id: default
    metadata:
      pii.action: REDACT
      guardrail.action: BLOCK

  - id: research
    credentials:
      provider.openai.api-key: ${RESEARCH_OPENAI_KEY}

routes:
  - id: default
    model: "gpt*"
    provider: openai

api_keys:
  - key_hash: sha256:9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08
    workspace: default

policies:
  - id: approved-models-only
    workspace: default
    status: ACTIVE
    dsl: |
      version: "1"
      rules:
        - id: deny-gpt-35
          conditions:
            model:
              denylist: [gpt-3.5-turbo]
          action: DENY
          deny_message: "gpt-3.5-turbo is not approved"

Without the two metadata keys, PII and guardrail detections are logged and the request goes through; REDACT and BLOCK are the strict choices. A route's model takes a trailing *; a policy allowlist or denylist matches model names exactly, so list each name. A max_tokens condition applies only to a request that sets max_tokens or max_completion_tokens. A rule's action is DENY or WARN_AGENT.

A policy applies to the model the gateway actually sends, not only the model the caller named. When a route sends another model (a strategy pins a version or picks a tier), the policy is checked again on that model. If it denies it, the caller gets the same 403 policy_denied and one POLICY_DENIED audit event, and no provider is called, unless a route backup the policy allows takes the request. This changed in 1.8.6: before, only the named model was checked, so a request the policy allows by name but whose route sends a model the policy denies now fails where it used to be served. Add the model the route sends to the allowlist, or take it off the denylist, to keep serving it.

Anything Spring Boot accepts works the same way here: --server.port=9090 on the command line, or SERVER_PORT=9090 in the environment.

Failover to another provider

A route can name where its requests go when the provider serving them fails: an ordered chain of provider and model. Say which model each backup is asked for; the gateway never picks one for you.

providers:
  - type: openai
    api_key: ${OPENAI_API_KEY}
  - type: anthropic
    api_key: ${ANTHROPIC_API_KEY}

routes:
  - id: support-assistant
    model: gpt-4o
    provider: openai
    fallbacks:
      - provider: anthropic
        model: claude-sonnet-4-5

The application keeps sending gpt-4o:

curl -s http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer $DVARA_KEY" -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o","messages":[{"role":"user","content":"Reply with the word ready."}]}'

If OpenAI fails with a server error, an open circuit or a rate limit, after its retries, the same request goes to Anthropic as claude-sonnet-4-5, on Anthropic's own key. The answer is an ordinary chat completion whose model is the model that served it (claude-sonnet-4-5). The gateway logs Route [support-assistant]: falling back from [openai] to anthropic/claude-sonnet-4-5, and counts it in gateway_fallbacks_total{from_provider="openai",to_provider="anthropic"}.

• Only the route's own chain is tried, in order. A route without fallbacks does not fail over. A request that matches no route still falls back to another configured provider serving the same model.
• A backup is skipped if it doesn't serve its model, can't take what the request needs (tools, images, a JSON schema), is paused by its circuit breaker, or the workspace's policy doesn't allow its model. If no backup can take the request, the answer is 503 failover_capability_mismatch; if every backup that was tried failed, the primary's error.
• A request the provider rejected (400), a policy denial and an authentication failure never fail over.
• A stream fails over only if it fails to open. Once it has started, an error ends it.
• A target without model is asked for the request's own model, for a provider serving the same one (OpenAI and Azure OpenAI). fallback: <provider> is the same as a one-entry chain without a model.
• At most dvara.llm-gateway.resilience.fallback.max-attempts backups are tried (default 3), none starting more than ...fallback.deadline after the first attempt (default 60s). A chain may name five. ...fallback.enabled: false turns failover off.
• The workspace's policy also checks the model sent to the first provider (see above).
• An application that embeds the gateway can check the first provider in other ways, with its own PrimaryTargetGuard bean. It is asked about the request with the model the route actually sends, which may not be the one the caller named (a strategy can pin a version or pick a tier). If it refuses, the route's chain is tried; if no backup takes the request, the caller gets the guard's error. The refused provider is never called.

Model context windows

Before a request is sent, the gateway estimates its size and compares it with the context window of the model that will serve it. At 90% of the window it refuses the request with 400 context_window_exceeded, or trims it if the workspace asks for that. Each provider declares one window for all its models (OpenAI declares 128,000 tokens), and many models accept more. Write a model's own window in model_limits, and the gateway uses it instead:

model_limits:
  - model: gpt-4.1          # exactly this model
    context_tokens: 1047576
  - model: "gpt-4.1-*"      # a trailing * matches every model with that prefix
    context_tokens: 1047576
  - model: claude-sonnet-4-5
    provider: anthropic     # optional: only when this provider serves it
    context_tokens: 1000000

• A model with no entry keeps its provider's window.
• When several entries match, the most specific wins: an exact model over a prefix, a longer prefix over a shorter one, and an entry naming the provider over one that does not.
• If a request could be served by more than one provider (failover), the smallest window among them applies.
• An entry without model or with no positive context_tokens stops the gateway at startup.
• The same list can be set as Spring properties, dvara.llm-gateway.model-limits[0].model and ...[0].context-tokens. An application that embeds the gateway can also supply its own ModelContextLimits bean, for example one that reads a model catalogue. Entries written here win over it.
• GET /v1/models shows the same window for each model, in capabilities.max_context_tokens.
• The guardrail's size limits apply first: by default a single message longer than 50,000 characters or a request with more than 100 messages is refused with 413. dvara.llm-gateway.guardrail.max-message-length and ...max-messages-per-request change them.

API keys

Every request under /v1 carries an API key, and a request without one is refused with 401. The key resolves to a workspace, and the workspace is what every control is scoped to: the PII action, the guardrail action, the policies, the rate-limit override, the provider credentials, and whose batch jobs and cached answers are whose. There is no setting that serves a keyless request; a gateway with no keys configured starts, and refuses every call until one is minted.

The gateway never generates a key at runtime. The operator mints one with --generate-key, which prints it once and stores nothing; the file holds only its SHA-256. The one path that takes no key is the webhook approval action, which carries its own signed token.

dvara.llm-gateway.data-plane.require-api-key, which once allowed keyless requests, was removed in 1.8.0. The gateway refuses to start if it is set to false, so a deployment that relied on it learns at startup rather than from its callers; set to true it starts and says the line can go.

Mint a key, fingerprint a key you already hold, and call the gateway with it

Mint a key, and the fingerprint that goes in the file:

java -jar …-app.jar --generate-key --name ci --workspace default

  Key — give this to whoever calls the gateway.
  It is not stored anywhere and cannot be recovered. If you lose it, mint another.

    gw_16df9718bd08f34d37e8d118db5bd815714249aa

  Add to gateway.yaml:

    api_keys:
      - key_hash: sha256:d11765e5979c1a3b76a659293ea6db8cd39cb65cc001998422c1d872419249dd
        name: "ci"
        workspace: "default"

The file holds the SHA-256 of the key, never the key. The key is printed once and stored nowhere; lose it and you mint another.

To fingerprint a key you already hold:

printf '%s' "$KEY" | java -jar …-app.jar --hash-key -
java -jar …-app.jar --hash-key-file /run/secrets/gateway-key

Give the key on standard input or name a file that holds it. --hash-key <key> as an argument works for one more release and warns. These commands print and exit without starting the gateway, so they work while it is running. Then call it with the key:

curl http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer gw_16df9718bd08f34d37e8d118db5bd815714249aa" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Hello"}]}'

Revoking a key is editing the file and restarting. A key from this file never expires.