Govern model calls through the LLM Gateway

Your applications should not have to implement policy, data protection, cost control, and audit separately for every model provider. The DVARA LLM Gateway is the model-traffic component of the DVARA AI governance platform. Put it on the request path, point an OpenAI-compatible client at it, and apply the same configured controls before and after each model call.

The LLM Gateway runs in both the Open Source distribution and the Enterprise platform. The Open Source distribution reads configuration from gateway.yaml and keeps local audit evidence when you configure an audit file. The Enterprise platform adds Flightdeck, centralized configuration, durable fleet operations, and advanced routing and cost controls. The Enterprise LLM path runs without a licence key only in the Development posture; Enterprise production use requires a licence.

How does DVARA govern a model call?

For a chat, Responses, completion, or text-embedding request, DVARA follows this path:

1. Identify the caller. When API-key enforcement is on, DVARA resolves the key to a workspace. It defaults to off in the Open Source distribution; an anonymous call does not receive workspace-specific overrides.
2. Evaluate configured controls. Policy, PII, guardrails, request limits, and rate limits can allow, change, or refuse the request. Enterprise cost and budget controls run when they are configured.
3. Choose a capable provider. DVARA matches a route, removes providers that cannot handle the requested capability, and applies the route's strategy.
4. Call the provider. Provider-specific authentication and request shapes stay behind the OpenAI-compatible DVARA API.
5. Govern the result. Configured response controls run before the client receives a non-streaming answer. Streaming has the boundaries described below.
6. Record the outcome. With audit storage configured, DVARA writes a GATEWAY_RESPONSE event. Usage and cost evidence depend on the deployment and the provider data available for that call.

The application or agent framework still owns the conversation and any tool execution loop. The LLM Gateway governs model calls; the MCP Gateway governs tool calls.

How do you make the first governed call?

Complete the Open Source distribution quickstart to run DVARA with the Mock provider and a local audit file. Then send this request:

curl -s http://localhost:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "mock/gpt-4",
    "messages": [
      {"role": "user", "content": "Summarize why runtime governance matters."}
    ]
  }'

The identifiers and timestamp vary, but the response has this shape:

{
  "id": "mock-6d0d5f08d3dd4aa5b536ba765cb0be63",
  "object": "chat.completion",
  "created": 1789092000,
  "model": "mock/gpt-4",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "This is a mock response"
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 12,
    "completion_tokens": 5,
    "total_tokens": 17
  }
}

Verify the effect by inspecting the configured audit file:

tail -n 1 var/audit.jsonl

The last line contains a GATEWAY_RESPONSE event with the model, provider, status, latency, sequence, previous hash, and HMAC. DVARA does not store the prompt or response text unless you separately configure content retention.

This local example is keyless because API-key enforcement defaults to off in the Open Source distribution. Set DVARA_LLM_GATEWAY_REQUIRE_API_KEY=true before using the runtime beyond local evaluation. The Enterprise quickstart creates a workspace key and sends it as Authorization: Bearer <your-api-key>.

What does each endpoint actually govern?

Every canonical /v1 operation below passes through data-plane API-key and request-rate checks. That shared outer boundary does not give every operation the live model-call pipeline.

Operation	Request-side work	Response and evidence boundary	Availability
POST /v1/chat/completions	For supported live calls: configured policy, PII, guardrails, limits, routing, and Enterprise budget/admission controls	Non-streaming response controls run before release. Streaming uses its configured enforcement mode. A completed call writes GATEWAY_RESPONSE when audit storage is configured.	Both distributions; individual controls depend on distribution and configuration
POST /v1/responses	Uses the same live model-call path after refusing unsupported fields, including tools and tool_choice	Returns a Responses object or typed SSE events; the same configured response controls and completion evidence apply	Both distributions
POST /v1/completions	Converts one string prompt to the governed chat path; refuses arrays, token arrays, streaming, n > 1, and best_of > 1	One non-streaming completion; configured chat response controls and completion evidence apply	Both distributions
POST /v1/embeddings	Text input uses the live request preamble, including configured policy, PII, guardrails, and Enterprise budget/admission checks. Token IDs cannot be text-scanned.	Returns vectors, so no model-text response scan runs. Usage is recorded when that deployment has a usage store.	Both distributions
GET /v1/models	API-key and request-rate checks only; queries each registered provider	Best-effort aggregation. A failing provider is omitted from the 200 result. No model-call policy, PII, guardrail, budget, or GATEWAY_RESPONSE event runs.	Both distributions
POST /v1/files	API-key and request-rate checks, then one PII pass over the complete JSONL upload before provider dispatch	Returns the provider file record. It does not run each JSONL line through chat policy, guardrails, or routing.	Both distributions when a configured provider declares batch support
GET /v1/files/{fileId}/content	API-key, scope, workspace-ownership, and request-rate checks	Returns provider content without response-side PII or guardrail enforcement.	Both distributions with tracked file ownership
POST /v1/batches	API-key and request-rate checks; Enterprise deployments also apply the available submit-time budget check	The provider executes the lines outside DVARA's live chat pipeline. DVARA does not apply per-line policy, PII, guardrails, or routing.	Both distributions when a configured provider declares batch support
GET /v1/batches	API-key, scope, workspace, and request-rate checks	Lists DVARA-tracked jobs and refreshes provider state; it does not govern batch output.	Both distributions; Open Source tracking is in memory
GET /v1/batches/{id}	API-key, scope, workspace, and request-rate checks	Polls provider status and settles completed usage once. It does not scan the output file.	Both distributions; Open Source tracking is in memory
POST /v1/batches/{id}/cancel	API-key, scope, workspace, and request-rate checks	Requests cancellation and settles completed work; provider charges can still apply to finished lines.	Both distributions; Open Source tracking is in memory
POST /v1/budget/estimate	API-key and request-rate checks, then pricing lookup and the current Enterprise budget decision	No provider call, model response, response scan, or GATEWAY_RESPONSE event	Enterprise platform only

Use the API specifications for request and response fields. The health endpoint is GET /actuator/health; it is an operator endpoint rather than part of the OpenAI-compatible /v1 surface.

POST /v1/webhooks/actions/{action} is also not an LLM operation. It is the Enterprise MCP approval callback used by signed approve/deny links. The signed approval token is its credential, so the endpoint is deliberately exempt from the DVARA API-key requirement.

How does API-key authentication behave?

dvara.llm-gateway.data-plane.require-api-key answers one question: may a request omit the bearer key? It defaults to false. A keyless request is then anonymous and has no workspace-specific policy, credential, budget, or limit overrides.

When a key repository is configured, a bearer value that does not resolve is refused even when key omission is allowed. A revoked or expired key is also always refused. An application that embeds the Open Source distribution can omit the key repository entirely; in that special case DVARA has nothing against which to validate a supplied bearer value. It passes a SHA-256 fingerprint downstream for rate limiting without assigning a workspace or retaining the plaintext.

Condition	HTTP	Error type	Error code
No bearer key and require-api-key=true	401	authentication_error	api_key_required
Presented key does not resolve	401	authentication_error	invalid_api_key
Presented key is revoked	401	authentication_error	api_key_revoked
Presented key is expired	401	authentication_error	api_key_expired
Valid key lacks the endpoint's scope	403	permission_error	api_key_scope
Non-canonical /v1 path	400	invalid_request_error	invalid_path

An empty scope list is unrestricted. A non-empty list is enforced by endpoint family. See Authenticate data-plane calls for the exact scope map and upgrade procedure.

Which OpenAI features have narrower support?

The OpenAI-compatible shape makes migration small, but it does not mean every OpenAI feature is implemented on every endpoint or provider.

Responses API

POST /v1/responses supports text and image input, instructions, metadata, structured output through text.format, top_p, and synchronous or streaming responses. It returns HTTP 400 with UNSUPPORTED_CAPABILITY instead of silently ignoring server-side conversation state, reusable prompt objects, tools and tool_choice, extra output through include, reasoning items, background mode, or file and audio input.

Use POST /v1/chat/completions when your agent needs function-call passthrough. DVARA governs the model request and scans tool-call arguments, but your agent framework executes the tool.

Legacy completions

Use POST /v1/completions only for an existing client that still sends the legacy text-completion shape. DVARA wraps one string prompt as a user message, then runs the same configured request and response controls, routing, cache, audit, usage, and cost work as a synchronous Chat Completions call.

Send one prompt with a key that carries completions:write:

curl -s https://dvara.internal.example.com/v1/completions \
  -H 'Authorization: Bearer <your-dvara-api-key>' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "gpt-3.5-turbo-instruct",
    "prompt": "Summarize the quarterly risk review in one sentence.",
    "max_tokens": 80,
    "temperature": 0.2,
    "top_p": 0.9,
    "n": 1
  }'

The response contains one text choice and provider-reported usage:

{
  "id": "cmpl_01K4V6A8PH28M8CW7D9K2R5TQZ",
  "object": "text_completion",
  "created": 1789257600,
  "model": "gpt-3.5-turbo-instruct",
  "choices": [
    {
      "text": "The review found that access controls are effective, with two remediation items due this quarter.",
      "index": 0,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 11,
    "completion_tokens": 18,
    "total_tokens": 29
  }
}

Confirm the call in Flightdeck or your configured audit output by matching the response's X-Trace-ID header. A completed call writes GATEWAY_RESPONSE when audit storage is configured. X-Cache is HIT or MISS according to the configured response cache.

The compatibility boundary is explicit:

Request field	Accepted value	Refused value
prompt	One string	A string array, token-ID array, or array of token-ID arrays
stream	Absent or false	true
n	Absent or 1	Any other integer
best_of	Absent or 1	Any other integer
user	Any string	Nothing; the value is accepted but ignored

Every refused form returns HTTP 400 with unsupported_capability before a provider call. For example, stream: true returns:

{
  "error": {
    "message": "stream is not supported on /v1/completions; use /v1/chat/completions with stream=true",
    "type": "invalid_request_error",
    "code": "unsupported_capability",
    "trace_id": "01K4V5R9X21MDN8W5A6E7ZQ3TC"
  }
}

The accepted sampling fields are max_tokens, temperature, and top_p. Use Chat Completions when you need streaming, multiple input prompts, token-ID input, tool calls, or structured output. The optional user field does not select a workspace or change attribution; DVARA uses the workspace API key.

How does embedding input get governed?

POST /v1/embeddings accepts one string, an array of strings, one token-ID array, or an array of token-ID arrays. DVARA 1.8 serves embeddings through the OpenAI integration; configuring a chat-only provider does not make this endpoint available.

For text, DVARA maps each input string to its own message and runs the live request preamble: configured policy, PII, guardrails, limits, and the Enterprise budget and admission controls available in that deployment. Separate array items stay separate while scanning. DVARA does not join the end of one item to the start of the next.

If PII action REDACT changes the text, the changed text is what OpenAI embeds. That changes the vector. Re-embed a corpus after changing from LOG to REDACT if old and new vectors must remain comparable.

Token IDs contain no readable text. The request controls still run, but PII and text guardrail scanners have no text to inspect, so DVARA passes the token IDs to OpenAI unchanged.

Send one governed text input:

curl -s https://dvara.internal.example.com/v1/embeddings \
  -H 'Authorization: Bearer <your-dvara-api-key>' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "text-embedding-3-small",
    "input": "How do I rotate an API key?",
    "dimensions": 256,
    "encoding_format": "float",
    "user": "search-indexer"
  }'

The response contains numeric vectors and input usage:

{
  "object": "list",
  "data": [
    {
      "object": "embedding",
      "embedding": [0.0124, -0.0318, 0.0071],
      "index": 0
    }
  ],
  "model": "text-embedding-3-small",
  "usage": {
    "prompt_tokens": 8,
    "total_tokens": 8
  }
}

dimensions is relayed to the provider. encoding_format defaults to float; base64 is refused with unsupported_capability because DVARA returns arrays of numbers. The optional user value is relayed to OpenAI for caller-side tracking, but DVARA attributes governance and usage from the API key.

Embedding requests reserve no estimated tokens at admission. After the call, DVARA reconciles the provider's reported total into the token-rate window and records the provider's input tokens and cost when those stores are configured. Vectors contain no generated text, so response PII, guardrail, grounding, and schema-validation stages do not run. Embeddings also bypass the response cache.

Failure	HTTP	Error code	What to change
Model or input is missing	400	validation_error	Supply both required fields.
No configured provider supports the model	400	no_provider	Configure OpenAI and use a text-embedding-* model.
encoding_format is base64	400	unsupported_capability	Omit it or send float.
Configured policy, PII, or guardrail blocks the text	403	Decision-specific	Change the input or the workspace control.
Upstream embedding call fails	502	provider_error	Check the OpenAI credential and upstream availability.

How do you check budget before a call?

Enterprise applications can call POST /v1/budget/estimate before sending a model request. The endpoint looks up the current workspace and API-key budget, but it does not call a model provider, reserve funds, or write usage and cost records.

Send the model and the same positive output limit you plan to use on the real request:

curl -s https://dvara.internal.example.com/v1/budget/estimate \
  -H 'Authorization: Bearer <your-dvara-api-key>' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "gpt-4o-mini",
    "max_tokens": 500
  }'

One possible response is:

{
  "estimated_cost_usd": 0.0003,
  "budget_remaining_usd": 42.18,
  "budget_remaining_tokens": 112480000,
  "budget_remaining_pct": 84,
  "would_exceed_budget": false,
  "pricing_known": true
}

estimated_cost_usd uses max_tokens and the matching output-token price; it does not estimate prompt tokens. budget_remaining_tokens is a separate approximation that uses the average of that pricing entry's input and output prices. The budget values come from the tightest budget that applies to the workspace and API key.

Always evaluate pricing_known before using the estimate. If it is false, no exact or glob price matched the model and estimated_cost_usd falls back to 0.0; that is not evidence that the request is free. The current budget check still runs, so would_exceed_budget is true when the hard cap is already exhausted even if pricing is unknown.

If no budget applies, budget_remaining_usd, budget_remaining_tokens, and budget_remaining_pct are null, while would_exceed_budget is false. An absent, zero, or negative max_tokens also produces a zero estimate even when pricing is known.

A key with a nonempty scope list needs completions:write; a key with no scopes is unrestricted. This route is available only in Enterprise. It returns a snapshot rather than a guarantee, so the subsequent model request still runs the normal budget enforcement against the spend current at that moment.

Batch jobs

DVARA scans an uploaded JSONL file for PII as one unit. On the Enterprise platform, submission also runs the available budget check. The provider then executes each line outside the live chat filters: DVARA does not apply per-line policy, guardrails, routing, or response PII controls, and downloaded results are raw provider output.

Uploads default to a 100 MB limit, configurable with DVARA_BATCH_MAX_FILE_SIZE; an oversized upload returns 413. OpenAI and Azure OpenAI declare production batch support in 1.8. The Mock provider also implements the lifecycle for local development and automated tests.

Use the same optional ?provider= value when you upload and submit so the file and batch remain on the same provider. Poll the batch, read its output_file_id, and download the result from GET /v1/files/{fileId}/content. There is no GET /v1/batches/{id}/results endpoint in 1.8.

The Open Source distribution tracks batches in memory. A restart loses its local ability to poll an outstanding job unless the deployment supplies persistent batch tracking.

How do you choose a provider and route?

Configure one or more providers, then route by model name or an explicit route. See Provider setup for credentials and provider-specific capabilities.

Deployment	Routing strategies
Open Source distribution	Model prefix, round robin, weighted, and canary
Enterprise platform	Open Source strategies plus latency-aware, cost-aware, geo-aware, and intelligent routing

Capability filtering happens before route selection. A request for vision, streaming, tool calls, structured output, JSON mode, batch, or streamed tool calls stays off a provider that does not declare that capability. See Routing and load balancing for route examples.

GET /v1/models queries registered providers on a best-effort basis. Its capabilities describe the provider's declaration for streaming, vision, tool calls, structured output, JSON mode, and context size. Treat them as routing metadata, not as an independent test of every model. The response can contain duplicate model IDs when two providers expose the same name.

What changes when you stream a response?

Set stream: true to receive Server-Sent Events. DVARA keeps the provider's OpenAI-compatible chunk shape and finishes the chat stream with data:[DONE].

The response-governance mode for a withholding action is Deferred: DVARA holds generated text and tool-call content, evaluates it as a whole, then releases or refuses it. Metadata-only chunks can pass while generated content remains held. Old rolling-window settings do not control this enforcement path. This gives the scanner complete context but removes the token-by-token latency benefit for that response action.

DVARA can try another provider only when opening the first provider stream fails before reading begins. It cannot fail over or resume after a stream has started. The global dvara.llm-gateway.resilience.timeout.streaming-timeout-ms setting bounds the complete client-visible lifetime, not the idle time between events, and defaults to 120000 milliseconds. A provider-specific override bounds only the wait to open that provider stream. A timeout or client disconnect closes delivery and releases the upstream transport.

Usage is exact when the provider supplies terminal streaming usage and is estimated otherwise. OpenAI-compatible providers and Mistral ask the upstream to include usage and preserve it when returned. Groq and Bedrock preserve their terminal provider metadata. Anthropic, Gemini, Cohere, and Ollama streams use an estimate in 1.8. The public SSE chunk does not add a DVARA-specific usage field.

On a normal completion, DVARA stores usage and cost before closing the stream. That finalization is best effort because content may already have reached the client. A pre-stream failure can return a normal HTTP error. After headers are committed, the Responses API emits response.failed for an upstream failure; an unhandled Chat Completions transport failure closes its SSE stream.

Native function calls use OpenAI-compatible delta.tool_calls[]. Assemble each call by index and concatenate its argument fragments until finish_reason: "tool_calls". A streamed tool request is routed only to a provider that declares both tool-call and streamed-tool support. DVARA includes generated function names and arguments in usage estimates and applies the configured streaming governance mode to the assembled arguments. See Function calling for the complete client and refusal contract.

See SSE streaming before enabling a withholding response action on latency-sensitive traffic.

What does a failed route look like?

Send a model that no configured provider serves:

curl -s -w '\nHTTP %{http_code}\n' http://localhost:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "unconfigured/model",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

DVARA refuses the request instead of silently changing the model or provider:

{
  "error": {
    "message": "No provider configured for model: unconfigured/model",
    "type": "invalid_request_error",
    "code": "no_provider",
    "trace_id": "01K4V5R9X21MDN8W5A6E7ZQ3TC"
  }
}
HTTP 400

Add a matching provider and route, restart an Open Source distribution after changing gateway.yaml, and repeat the request. On the Enterprise platform, publish the configuration change from Flightdeck instead.

What should you do before production?

• Require workspace API keys and send one in every /v1 request.
• Store provider credentials outside source control and test their rotation.
• Start policy, PII, and guardrails in their observation action before moving tested rules to a blocking or redacting action.
• Configure audit storage, protect its HMAC secret, and test verification and export.
• Test provider failure, capability mismatch, stream timeout, and budget refusal with the same request shapes your application sends.

Next, configure policy as code, PII governance, and guardrails on this request path.