Open-source gateway APIs and providers

Reference for the open-source DVARA LLM Gateway, run from one gateway.yaml. The quickstart gets it running.

Batch

the OpenAI Batch API through the gateway. The input file is PII-scanned as a whole and its cost is booked when the batch finishes. Its lines do not run the request pipeline, so a policy, guardrail or rate limit on direct requests does not apply to them. An application can refuse models in a batch with a BatchModelCheck bean: the gateway asks it about each model a file names when the file is uploaded and again when a batch is submitted, and refuses the file naming the line. A suspended workspace cannot upload a batch file or submit a batch; it can still read, download and cancel the batches it has. Jobs are tracked in memory, so a restart forgets them.

Anthropic Messages API

POST /v1/messages, for a client built for it such as Claude Code: set ANTHROPIC_BASE_URL to the gateway and ANTHROPIC_AUTH_TOKEN to a gateway API key. Governed like chat, on any provider. On a route to Anthropic the request goes through as the client sent it — every field, message role, content block and the anthropic-beta header — and the response comes back as Anthropic sent it, changed only where governance changes it (redacted text, the routed model). On another provider the request is translated: extended thinking is refused naming the provider, and what only tunes Anthropic (a field or block type the gateway does not model, cache_control) is left out and counted. POST /v1/messages/count_tokens counts tokens without calling a model. Server tools, mcp_servers and container are refused, and Claude Code signed in with a Claude subscription does not use a custom base URL, so it cannot be pointed here. metadata.user_id is the request's end user, as user is on chat, only when it is a plain id. A JSON object there, such as a client's own device and session ids, is not taken as the end user; on a route to Anthropic it still goes upstream as sent. GET /v1/models with an anthropic-version header answers in Anthropic's model-list shape and lists only the Claude models Anthropic serves, so a client built for Anthropic is never offered another provider's model.

OpenAI Responses API

POST /v1/responses, whole or streamed, governed like chat on any provider. Text, images, structured output (text.format) and your own tools:

• function tools (with strict, sent to OpenAI), tool_choice, and function_call and function_call_output items in input. A call comes back as a function_call output item, and streams with response.output_item.added, response.function_call_arguments.delta and .done, and response.output_item.done.
• custom tools, whose input is free text such as a patch, with custom_tool_call and custom_tool_call_output items. OpenAI takes them as they are; another provider gets a function with one string argument, input, and the grammar in its description.
• namespace tools, such as one MCP server's: each inner tool travels as namespace__name (cut to 64 characters with a hash) and its calls come back under their own name and namespace.
• A client-run tool_search ("execution": "client") is a function on a translated route, its calls coming back as tool_search_call; the tools an additional_tools item or a tool_search_output brings are read in. Any other tool type the gateway cannot translate (a shell, a tool_search the provider runs) goes as it came to OpenAI or Azure OpenAI, and a request with one is refused on any other provider.
• A tool's output can hold text and images. A provider that cannot take an image in a tool result (anything but Anthropic and Bedrock) gets it in a user message right after the results.

On a route to OpenAI or Azure OpenAI the request goes to the provider's own Responses API as you sent it: every field, tool and input item, including encrypted reasoning from an earlier answer (include: ["reasoning.encrypted_content"] and reasoning items), with only the edits governance makes (redacted text, the routed model, a token limit). The answer, whole or streamed event by event, comes back as OpenAI sent it, with redacted text written back in place, and its usage object exactly as sent. The access log adds tokens_cached and tokens_reasoning when the provider reports them. When the provider ends the stream with an error before any output (a spent quota, a rate limit, a bad request), the client gets the status that error stands for, such as 429, with the provider's code and message, instead of a stream that starts and then fails. After output has started, the provider's own response.failed or error event is sent as the provider sent it. When the streaming guard holds content back, the answer is rebuilt in the Responses shape and reasoning items are kept. On Azure the call goes to {base-url}/v1/responses with the deployment as model; dvara.llm-gateway.providers.azure-openai.responses-api-version adds an api-version (none by default). On any other provider the request is translated to its chat API, as on chat. The gateway runs no tool; your code does. Built-in tools (web_search, file_search, code_interpreter, computer_use, mcp and the rest) run at the provider, outside the gateway's checks, so they are refused; with Codex CLI, set web_search = "disabled". So are store: true, previous_response_id, background, a service_tier other than auto or default, file parts, and input item types the gateway does not read. On /v1/chat/completions, reasoning_effort goes to OpenAI and Azure OpenAI as sent, so a model's own refusal reaches the client. A chat request that has function tools and an effort other than none, which OpenAI's Chat Completions refuses on its newest models, is sent to the provider's Responses API instead and answered in the Chat Completions shape, whole or streamed: text, function and custom tool calls, finish reasons, usage with cached and reasoning tokens, and the provider's own error. stop, seed and the penalties have no place in the Responses API, so such a request with one of them is refused. Encrypted reasoning cannot come back over Chat Completions, so each chat turn starts without the reasoning of the turn before; a client that wants it kept uses /v1/responses. On Anthropic, and so on /v1/cursor with a -thinking-<effort> model, it turns on extended thinking with a budget of 1024 (low), 4096 (medium), 16384 (high) or 32000 (xhigh) tokens, added to max_tokens so the answer keeps its room; none and minimal ask for none. Thinking takes no temperature but 1 and no top_p below 0.95, so a request with one of those and an effort is refused. A Claude model without extended thinking, and any other provider, is not sent the effort, which is logged at DEBUG. Thinking tokens are the reasoning part of the usage. On /v1/chat/completions a custom tool ({"type": "custom", "custom": {name, description, format}}) is sent to OpenAI as it is and to any other provider as a function with one input string; its calls come back as {"type": "custom", "custom": {name, input}}, whole or, in a stream, with the input in one fragment before the finish. A tool of any other type is refused there, never left out. parallel_tool_calls: false goes to OpenAI as it is and to Anthropic as disable_parallel_tool_use; other providers are not sent it (logged at DEBUG). The same holds on /v1/chat/completions. On a translated route, reasoning.effort, text.verbosity and prompt_cache_key are sent to OpenAI-shaped chat only where the provider takes them (logged at DEBUG when left out), and what only asks for reasoning output is accepted with no reasoning in the answer: reasoning.summary, include: ["reasoning.encrypted_content"], stream_options, and reasoning items in input, which are left out. A developer message is a system message. client_metadata is accepted and not sent. When the provider reports usage, response.completed carries it.

Gemini generateContent API

for a client built for it such as Gemini CLI or the Google GenAI SDKs. Set GOOGLE_GEMINI_BASE_URL to the gateway's address followed by /v1/gemini (for example https://gateway.example.com/v1/gemini) and GEMINI_API_KEY to a gateway API key; the client then calls POST /v1/gemini/v1beta/models/{model}:generateContent, :streamGenerateContent?alt=sse and :countTokens, with the key in the x-goog-api-key header (Authorization: Bearer works too). A key in the URL (?key=) is not read, since URLs end up in logs. Gemini CLI 0.63 run headless (gemini -p) also needs "security": {"auth": {"selectedType": "gemini-api-key"}} in ~/.gemini/settings.json, or it exits with "Invalid auth method selected". Use a current Gemini CLI: very old releases (0.1.x) send a first request straight to Google with the key in the URL, whatever the base URL. Pick a model the route serves, such as gemini -m gemini-3.8-flash, or a route name for another provider. Governed like chat, on any provider, with the model in the path routed like any model name. On a route to Gemini the request goes through as the client sent it — every field and part, call ids, thought signatures, thinkingConfig and safetySettings included — and the response comes back as Gemini sent it, changed only where governance changes it. Thinking and cached tokens are metered from usageMetadata. On another provider the request is translated: systemInstruction is a system message, function declarations are tools, functionCall and functionResponse parts are tool calls and their results, and the answer, plain or streamed, is written back in Gemini's shape; what only tunes Gemini (thinkingConfig, topK, safetySettings, thought parts) is left out, and a provider that takes only one of temperature and topP (Anthropic) is sent temperature. When governance holds or rebuilds an answer, parts it does not read (executableCode, codeExecutionResult, inlineData) are still delivered. Built-in tools that Google runs itself (googleSearch, urlContext, codeExecution, googleSearchRetrieval and the like) are refused on every route, Gemini included, because what they fetch or run is never checked. Gemini CLI asks for them only from its web search and web fetch tools; deny those with a policy rule, for example in ~/.gemini/policies/gateway.toml: [[rule]], toolName = ["google_web_search", "web_fetch"], decision = "deny" (older releases used "tools": {"exclude": [...]} in settings.json, which 0.63 marks deprecated). cachedContent and more than one candidate are refused too. A refusal, including one a provider makes before its stream starts, comes back as an error status with JSON in Google's error shape and the provider's own reason. Gemini CLI sends no output limit, so the default response cap (dvara.llm-gateway.guardrail.default-max-response-tokens) becomes its maxOutputTokens; Gemini counts thinking inside that limit, so raise the cap if long agent turns stop at MAX_TOKENS. Gemini CLI signed in with a Google account does not use a custom base URL, so it cannot be pointed here.

Response cache

exact-match, in memory, per process. Off until dvara.llm-gateway.cache.in-memory.enabled is true.

Cursor (preview)

Point Cursor's "Override OpenAI Base URL" (Settings, Models) at the gateway's Cursor base URL and use a gateway API key as the OpenAI key:

• Base URL: https://<your-gateway>/v1/cursor. Cursor calls {base}/models to check the key and {base}/chat/completions for every request.
• Key: a gateway API key with the completions:write and models:read scopes.
• Models: add each as a custom model in Cursor under the name your routes serve. Cursor refuses a custom model named like one of its built-in models, so give your routes names of their own.
• Variants: a model picked as <name>-fast is routed as <name>, and <name>-thinking-<effort> (effort minimal, low, medium, high or xhigh) as <name> with that reasoning effort. A bare -thinking stays in the name, so a route can carry it.

Cursor sends its requests from its own servers, not from your machine, so the gateway must be reachable from the internet over HTTPS.

In chat mode Cursor sends a Chat Completions body. In agent mode it sends a Responses API body (input, instructions, flat tool definitions, reasoning) to the same path. The gateway reads each the way its own API reads it (agent mode as /v1/responses does, so OpenAI and Azure OpenAI are sent it as it came) and always answers in the Chat Completions shape, which is what Cursor reads. Every request gets the same checks, metering and audit as /v1/chat/completions.

This is a preview: it is tested against Cursor's request shapes as publicly documented from its traffic, not yet against a live Cursor session. Cursor's Azure OpenAI setting, which calls /openai/deployments/<deployment>/chat/completions, is not served.

Supported providers

Capabilities as each provider declares them; /v1/models reports the same for a running gateway, with each model's own context window where one is known (see Model context windows).

Provider	Streaming	Vision	Tool calls	Structured outputs	Batch
OpenAI	✅	✅	✅	✅	✅
Azure OpenAI	✅	✅	✅	✅	✅
Anthropic	✅	✅	✅	✅	—
Google Gemini	✅	✅	✅	✅	—
AWS Bedrock	✅	✅	✅	✅	—
xAI Grok	✅	✅	✅	✅	—
Mistral	✅	—	✅	✅	—
DeepSeek	✅	—	✅	—	—
Moonshot	✅	—	✅	—	—
ChatGLM (Zhipu)	✅	—	✅	—	—
Cohere	✅	—	—	—	—
Groq	✅	—	—	—	—
Qwen	✅	—	—	—	—
Ollama	✅	✅	✅	✅	—

A token limit can be sent as max_tokens or max_completion_tokens; the gateway holds it as one limit and sends each provider the field its model takes. OpenAI's reasoning models (the o-series and GPT-5 and later) refuse max_tokens, so OpenAI and Azure OpenAI are sent max_completion_tokens for them, and max_tokens for every other model. On Azure OpenAI the model is the deployment name, so name a reasoning deployment after its model (azure/gpt-5-mini). Sending both names with different values is refused with 400. The workspace's default response cap (dvara.llm-gateway.guardrail.default-max-response-tokens, 4096, 0 turns it off) is that same limit when the caller sets none; on a reasoning model it also counts the tokens spent reasoning.

When a provider refuses a request with a 4xx other than 401 and 403 (a bad parameter, a model that is no longer offered, a limit), the error the caller gets carries the provider's own reason after the gateway's message, on one line and at most 300 characters: OpenAI API error 400: unsupported_parameter: Unsupported parameter: 'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead. Nothing else of the provider's answer is returned, and 401, 403 and 5xx keep the gateway's message alone.

A mock provider is included for tests and demos; it needs no account. It also takes batches: an uploaded file reads back as what was uploaded, every batch completes on its first poll, and its output file holds two canned lines with usage. It keeps the newest 256 uploaded files in memory, so a restart forgets them; a file it does not hold is not found, as a provider answers.