Govern streamed LLM responses

DVARA governs streamed LLM responses while preserving incremental delivery when your controls do not need to withhold content. A blocking, redacting, tokenizing, or grounding control holds generated content before DVARA releases or refuses it. The Enterprise A2A Gateway has a separate, PII-only streaming contract.

Stream a governed response

Set "stream": true in your chat completion request:

curl -s -N -X POST http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer $DVARA_API_KEY" \
  -H "X-Trace-ID: stream-demo-1042" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [
      {"role": "user", "content": "Count to five slowly."}
    ],
    "stream": true
  }'

You receive content chunks followed by data:[DONE]. After the stream closes, verify the call in Flightdeck or your configured audit sink with trace ID stream-demo-1042. A governed stream records STREAMING_ENFORCEMENT_SUMMARY; usage remains server-side and is not added to the public SSE payload.

Read the SSE response

The response uses Content-Type: text/event-stream. Each event is a JSON chunk prefixed with data: — no space after the colon:

data:{"id":"chatcmpl-abc123","object":"chat.completion.chunk","model":"gpt-4o","choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null}]}

data:{"id":"chatcmpl-abc123","object":"chat.completion.chunk","model":"gpt-4o","choices":[{"index":0,"delta":{"content":"One"},"finish_reason":null}]}

data:{"id":"chatcmpl-abc123","object":"chat.completion.chunk","model":"gpt-4o","choices":[{"index":0,"delta":{"content":", two"},"finish_reason":null}]}

data:{"id":"chatcmpl-abc123","object":"chat.completion.chunk","model":"gpt-4o","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data:[DONE]

Read each chunk

Each chunk contains:

• id — Stable completion ID across all chunks
• object — Always "chat.completion.chunk"
• model — The model used
• choices[].delta — Incremental content (content for text, role for the first chunk)
• choices[].finish_reason — null during streaming, "stop" on the final content chunk

The stream terminates with data:[DONE].

DVARA accepts a stream as complete only after a provider finish reason. Any finish reason ends the answer. If the upstream closes without one, returns an empty body, or carries an in-band error, DVARA records PROVIDER_ERROR instead of presenting and billing the partial stream as a complete answer. If response headers have already been sent, the client sees the terminal stream error rather than a new HTTP status.

Match data:, not data:

The SSE specification makes the space after the colon optional and tells consumers to strip one if present. DVARA does not emit one — verified byte-for-byte on 1.7.0. Every mainstream SSE library handles this, but a hand-rolled parser that splits on the literal "data: " silently matches nothing.

Stream a structured response

Streaming works with response_format. From the client's point of view the experience is identical no matter which provider handles the call: JSON bytes arrive progressively as content deltas. The DVARA LLM Gateway absorbs the provider-specific streaming protocols. See Structured Outputs for the rewrite mechanics.

Provider group	Streaming behaviour with response_format
OpenAI, Azure OpenAI, Mistral, Gemini, Grok	Native. Standard OpenAI-format content deltas.
Ollama (from 1.8.4)	OpenAI's response_format is forwarded for json_object and json_schema. With OLLAMA_STRUCTURED_OUTPUTS=false, json_schema is routed away.
Anthropic, Bedrock	Tool-use rewrite for json_schema. The gateway flattens the upstream tool-call stream into normal content deltas — JSON bytes arrive progressively, no client-side tool-call handling required.
DeepSeek, Moonshot, ChatGLM, Groq	Native json_object. json_schema routed away by capability filter.
Qwen, Cohere	Filtered out for any response_format. A request on a route with only these providers fails fast with no_capable_provider (HTTP 400) before any streaming starts.
Mock	Streams the wrapped {"result": "..."} text one token at a time.

Bound the complete stream lifetime

Streaming requests have a separate, longer timeout than non-streaming requests:

dvara:
  llm-gateway:
    resilience:
      timeout:
        chat-timeout-ms: 30000        # non-streaming: 30 seconds
        streaming-timeout-ms: 120000  # streaming: 120 seconds

The global streaming-timeout-ms value starts when DVARA accepts the request and bounds the complete client-visible SSE lifetime. It is not an idle timeout that resets after each event. When it expires, DVARA closes the client stream and releases the upstream transport so a stalled read can stop.

A provider-specific streaming timeout has a narrower job: it bounds the wait to open that provider stream. It does not replace the global client-visible lifetime. Choose the global value for the longest response your application allows, then keep provider-specific open timeouts short enough to reject a provider that cannot start in time.

Know when caching is bypassed

Streaming requests bypass the response cache entirely — no cache lookup, no cache storage, no X-Cache header. This is by design, as streaming responses are typically unique and the cache cannot store partial streams.

Plan for the streaming boundaries

The streaming path differs from a non-streaming call in ways that affect client retries, usage reporting, and response latency.

• Usage on the wire only when you ask for it. By default the SSE response carries no token counts. With stream_options.include_usage: true, the last chunk has empty choices and the provider's usage block, as OpenAI sends it. DVARA sends that chunk only when the provider reported usage; an estimate is never presented as the provider's count. From 1.8.4 the block can include the cached-input, cache-write and reasoning breakdown. DVARA retains terminal provider usage for server-side metering when the provider supplies it; otherwise it estimates usage from the request and the delivered text and tool-call arguments. The estimated field on the stored usage record tells you which path applied.
• Tool calls are partial until the terminal chunk. Native function calls use OpenAI-compatible delta.tool_calls[] events. Join each call by its index and concatenate its function.arguments fragments in arrival order. Parse the arguments only after finish_reason: "tool_calls". See Function calling for a complete example and the governance behavior.
• No X-Gateway-Strict-Downgraded header on streams. That header is set on a non-streaming response; streamed responses don't carry it. If you need to detect strict downgrade for a streamed call, do the same call non-streaming first to read the header, then stream subsequent calls knowing the route's downgrade behavior.
• No resume after delivery starts. DVARA can try another provider only when opening the first provider stream fails before an iterator is returned. It does not fail over or resume after reading begins. Retrying a request after receiving content starts a new model call and can duplicate output and spend.

For server-side streaming usage, the current decoder behavior is:

Usage source	Providers
Terminal provider usage when supplied	OpenAI-compatible providers, Mistral, Groq, Bedrock, and, from 1.8.4, Anthropic and Gemini
DVARA estimate	Cohere and Ollama (and Anthropic and Gemini before 1.8.4)

Choose a delivery mode through your controls

Streaming uses one of two delivery modes, selected by the configured controls:

Mode	When it applies	What the client receives
Immediate	PII LOG, guardrail LOG or FLAG, and no withholding control	Provider events are relayed as they arrive while DVARA accumulates text for final evidence.
Deferred	PII BLOCK, REDACT, or TOKENIZE; guardrail or grounding BLOCK	DVARA holds all generated text and tool-call content, evaluates it once, then releases governed content or ends with finish_reason: content_filter. Metadata-only chunks can pass through while content remains held.

The action distinction remains important in Deferred mode. REDACT replaces new response PII irreversibly and stores no original value. TOKENIZE does the same for newly detected response PII: it does not mint a recoverable response token. Request-side tokenization and optional non-streaming detokenization are separate behaviors. LOG is the default PII action and changes no content.

Whole-response enforcement prevents a sensitive value split across provider chunks from escaping or being partially released. It also removes incremental delivery for withholding actions and uses memory proportional to the held response. The default maximum is 1,000,000 held characters; exceeding it stops the stream with STREAM_TOO_LARGE_TO_SCAN rather than releasing unchecked content.

Streamed native tool calls are governed after DVARA assembles each call's JSON argument fragments. One response can contain at most 256 tool calls and 512 argument values for enforcement. Invalid, duplicate-key, or over-limit argument JSON is refused under Deferred delivery with STREAM_TOOL_ARGUMENTS_UNENFORCEABLE; under Immediate delivery it is relayed and the summary records incomplete scanning. An upstream failure also prevents a Deferred grounding decision and produces STREAM_INCOMPLETE_UNGROUNDED.

An enforcement refusal ends a Chat Completions stream with finish_reason: "content_filter". A failure before streaming begins can still return a normal HTTP error. After the SSE response is committed, DVARA cannot replace its status: a Responses API stream emits response.failed, while an unhandled Chat Completions transport failure closes the stream. Do not treat every post-header failure as a governance refusal.

When the client disconnects, DVARA stops delivery and releases the upstream transport. Deferred mode discards content that was still being held. Immediate mode schedules final evidence and usage work for the partial response with a five-second deadline. That finalization is best effort because content has already left DVARA; the streaming enforcement summary records the stream as truncated when finalization succeeds.

Every governed stream writes STREAMING_ENFORCEMENT_SUMMARY. Treat its fields as separate facts: scan_incomplete means Immediate delivery exceeded a scan bound or contained unenforceable arguments; scan_failed means enforcement failed after Immediate content had already been delivered; truncated means the upstream ended abnormally or the client disconnected. Deferred delivery refuses instead of setting scan_incomplete and releasing content.

Monitor held-buffer utilization, overflows, and incomplete cancellation finalization with the bounded-label metrics and alert examples in Observability.

Usage and cost finalization also happens before a normally completed stream is closed. It is best effort because the client may already have received content. If the provider supplies terminal usage, DVARA stores it as exact even when a Deferred control refuses the response. Without terminal usage, DVARA estimates from the request and the content actually delivered. If neither provider usage nor delivered output is available, no usage row can be written and the initial token reservation remains charged.

The old streaming-scan-window-size and streaming-overlap-margin properties remain accepted for compatibility but do not control the 1.8 whole-response enforcement path. Use the supported scan-streaming-responses switches and held-response limit documented in Configuration.

Workspace setting	Default	Effect
pii.action	LOG	LOG, BLOCK, irreversible REDACT, or Enterprise request-side TOKENIZE
pii.scan-streaming-responses	true	Enables PII inspection of streamed responses
guardrail.action	LOG	LOG, FLAG, or BLOCK
guardrail.scan-streaming-responses	true	Enables guardrail inspection of streamed responses
grounding.action	LOG	LOG, FLAG, or BLOCK when grounding is configured

Call DVARA from an SDK

The minimal curl invocation:

curl -s -N -X POST http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer $DVARA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4-5",
    "messages": [{"role": "user", "content": "Write a haiku about programming."}],
    "stream": true
  }'

The same call from Python with the OpenAI SDK:

from openai import OpenAI

client = OpenAI(api_key="<your-dvara-api-key>", base_url="http://localhost:8080/v1")

stream = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Write a haiku about programming."}],
    stream=True,
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
print()

For end-to-end Node.js and Java consumers — including line-iterator parsing, partial-JSON accumulation for json_schema streams, and safe retry boundaries — see the Streaming end-to-end cookbook recipe.

Check provider-specific behavior

Every built-in provider supports streaming. The DVARA LLM Gateway normalizes each upstream protocol into the same OpenAI-compatible chunks. Governance and audit therefore see one shape, and the client does not branch by provider.

Provider	Upstream streaming protocol	Notes
OpenAI	data: {"choices":[{"delta":…}]} lines	Standard OpenAI-format deltas.
Azure OpenAI	Same shape as OpenAI	Same.
Mistral	OpenAI-compatible chunks	Same.
Grok	OpenAI-compatible chunks	Same. Supports json_schema natively.
Gemini	data: {"candidates":[{"content":…}]} lines	The gateway walks the candidates[].content.parts[] tree and concatenates text parts per chunk.
Anthropic	Event-based: message_start, content_block_delta, message_delta, message_stop	Each event becomes one content delta on the wire. With json_schema, the upstream's tool-use deltas are flattened into content text so the JSON arrives progressively.
Bedrock	Binary bedrock-runtime event stream	Decoded server-side; both normal text and the json_schema tool-use path produce progressive content deltas. The upstream tool_use stop reason is mapped to stop.
DeepSeek, Moonshot, ChatGLM	OpenAI-compatible chunks	Same as OpenAI for json_object; json_schema filtered out by capability routing.
Cohere	OpenAI-style chunks from Cohere's v2 chat API	No response_format support — filtered out when one is requested.
Groq	OpenAI-compatible chunks	Supports json_object streaming but not json_schema.
Qwen	OpenAI-compatible chunks	No response_format support — filtered out when one is requested.
Ollama	OpenAI-compatible chunks	From 1.8.4, json_object and json_schema stream as normal content deltas, and streamed tool calls arrive as delta.tool_calls[].
Mock	Simulated token-by-token emission with configurable delay	Splits the configured response text into tokens and emits one chunk per token, separated by dvara.llm-gateway.providers.mock.stream-token-delay-ms (default 20 ms). Useful for integration tests and for exercising the streaming governance path with deterministic content.