Prove DVARA governance is running

An UP process does not prove that an AI request was governed. For each test request, connect the response to its access log, metrics, trace, and audit evidence. Each signal answers a different question; none replaces the others.

Enterprise only

Audit search, exports, and compliance evidence are not included in DVARA Open Source.

MCP and A2A signals appear only when those planes receive traffic.

Know what each signal proves

Signal	Question it answers	Important boundary
Health	Should this process receive traffic?	Health does not prove that a provider call or governance decision succeeded
Structured log	What happened inside one request?	Logs are operational records, not the tamper-evident audit chain
Metric	Is behavior changing across many requests?	Counters and histograms do not explain one decision
Trace	Where did one request spend time?	Trace export is off until you configure a collector
Audit evidence	Which governed action and outcome did DVARA record?	Open Source evidence is a local file; Enterprise evidence is searchable and signed in the shared store
Alert	Does an operator need to act now?	A loaded rule still needs a tested notification destination

Check health without exposing internals

These Gateway endpoints are available without a bearer token so load balancers and orchestrators can use them:

Endpoint	Use it for
GET /actuator/health	Overall health summary
GET /actuator/health/liveness	Decide whether the process should be restarted
GET /actuator/health/readiness	Decide whether the process should receive new traffic
GET /actuator/info	Confirm the running build version and time

curl --fail --silent https://<dvara-host>/actuator/health/readiness

An anonymous healthy response gives the verdict without component details:

{
  "status": "UP"
}

Use DVARA_ACTUATOR_API_KEY for /actuator/gateway-status. On the Enterprise platform, the same bearer authorizes /actuator/license and component detail on health requests. Gateway status reports runtime metadata and warnings; licence status reports posture, licensed, licensee, licence ID, type, runtime status, expiry, days remaining, grace-period state, and the workspace and call limits that apply. Neither endpoint returns the raw licence, provider credentials, encryption material, or audit secret. The Open Source distribution does not expose /actuator/license.

The Gateway excludes high-risk diagnostic surfaces such as environment dumps, heap dumps, runtime mappings, and logger controls. Do not widen the exposed endpoint list to make routine monitoring easier.

Follow one request across the system

Send a normal governed request through the same public edge your application uses. Keep a stable session ID for a multi-turn or agent workflow, but omit a custom trace ID when trace export is active so the response returns the actual OpenTelemetry trace ID.

curl --include https://<dvara-host>/v1/chat/completions \
  --header 'Authorization: Bearer <your-dvara-key>' \
  --header 'Content-Type: application/json' \
  --header 'X-Session-Id: claims-review-2026-09-12-001' \
  --data '{
    "model": "<configured-model>",
    "messages": [
      {"role": "user", "content": "Summarize the policy for this test case."}
    ]
  }'

The response includes a correlation header and the provider response:

HTTP/2 200
X-Trace-ID: 4bf92f3577b34da6a3ce929d0e0e4736
Content-Type: application/json

{"id":"chatcmpl-8d31","object":"chat.completion","choices":[...]}

If the request supplies X-Trace-ID, DVARA echoes that value. Otherwise it uses the active trace ID, or generates a 32-character identifier when no trace is active. X-Session-Id is optional and is added to request-scoped logs and tracing attributes when present.

Find the request in structured logs

JSON logging is the default. Each /v1/* request produces one access record after the request completes; streaming requests are recorded after the stream finishes rather than when it opens.

Search for the returned trace ID in your log system. From a Kubernetes pod:

kubectl logs <gateway-pod> \
  | jq 'select(.trace_id == "4bf92f3577b34da6a3ce929d0e0e4736")'

A completed LLM access record can include the workspace, opaque API-key ID, model, selected provider, HTTP status, latency, cache result, stream flag, token counts, and error code. It never logs the bearer token. Fields that do not apply or were not known are absent.

Set SPRING_PROFILES_ACTIVE=log-plain only when a person needs readable local output. Plain mode keeps the trace ID in each request line but gives up the stable JSON shape expected by log pipelines.

Scrape metrics with the metrics key

Prometheus export is enabled by default at /actuator/prometheus, but every scrape requires the distinct DVARA_ACTUATOR_METRICS_API_KEY:

curl --fail --silent \
  --header "Authorization: Bearer <your-metrics-api-key>" \
  https://<dvara-host>/actuator/prometheus \
  | grep -E '^(gateway_requests_total|gateway_latency_seconds|gateway_tokens_total)'

The metrics key must differ from DVARA_ACTUATOR_API_KEY. If it is unset or does not match the value used by the scraper, the endpoint returns 401.

Start with the signals that describe the request path:

Metric	Type	Main labels	What it measures
gateway_requests_total	Counter	workspace, model, provider, status, region	Completed Gateway requests
gateway_latency_seconds	Timer/histogram	workspace, model, provider, status, region	End-to-end request latency
gateway_tokens_total	Counter	workspace, model, direction	Input and output tokens
gateway_provider_errors_total	Counter	provider, error_code	Upstream provider failures
gateway_retries_total	Counter	provider	Provider-call retries after a retryable failure
gateway_fallbacks_total	Counter	from_provider, to_provider	Successful provider fallback attempts; these are not necessarily confined to the matched route
gateway_cost_dollars_total	Counter	workspace, model, provider	Calculated model cost
gateway_guardrail_blocked_total	Counter	workspace, category	Requests refused by a guardrail
gateway_config_refresh_total	Counter	table, status	Enterprise configuration refreshes by type and outcome
gateway_config_refresh_duration_seconds	Timer/histogram	config_type, status	Enterprise configuration refresh latency by type and outcome

The 1.8 operational signals below are not optional examples; each diagnoses a specific loss of governance or fleet consistency.

Metric	Distribution	What a bad value means
gateway_spool_shed_total{class}	Enterprise	Records were dropped from the local spool; usage-class drops can lose billing evidence.
gateway_drain_lag_seconds	Enterprise	Age of the oldest unshipped record across every drained class; the persistent sink is not keeping up.
gateway_spool_depth{class}	Enterprise	Records waiting in the local spool, by class. class="a2a_audit" is the A2A audit backlog. From 1.8.5 /actuator/gateway-status also shows it as a2a_audit_depth, beside audit_depth for the LLM chain.
gateway_webhook_deliveries_dropped_total	Enterprise	The 1,000-job webhook queue filled and its oldest waiting delivery was discarded.
gateway_siem_export_total{exporter,outcome}	Enterprise	failure means delivery failed; dropped means the bounded queue discarded a waiting SIEM copy. The primary audit record remains stored.
gateway_tier_unresolved_ids	Enterprise	Workspaces reference tier IDs the active catalogue cannot resolve.
dvara_internal_mtls_rejections_total{reason}	Enterprise	An internal client certificate was refused; investigate expiry, trust, or probing.
gateway_hazelcast_cluster_members	Enterprise	A value below the gateway replica count means embedded-cluster counters and caches are split by pod.
gateway_unpriced_calls_total{model,provider}	Enterprise	Calls served for a model with no pricing row, so they cost nothing in every report and every budget. The same models are listed under unpricedModels on /actuator/gateway-status. New in 1.8.1; before it, an unpriced model was silent.

Do not alert on a metric until the feature that creates it is enabled and the series exists.

Diagnose whether governance is operating

Metric	Distribution	What to investigate
gateway_audit_chain_broken_total	Both	Any increase means chain verification found tampering or missing evidence.
gateway_deny_list_ever_contacted	Enterprise	0 means the session deny-list channel has never succeeded.
gateway_config_bundle_ever_contacted	Enterprise	0 means the Gateway has never received a signed bundle from Flightdeck.
gateway_accounts_without_tier	Enterprise	Accounts are operating without a resolved quota tier.
gateway_config_poller_up	Enterprise, Flightdeck only	0 after startup means Flightdeck's check for configuration changes is unhealthy, so no new bundle is published. From 1.8.2 Gateways do not register this or the three gauges below.
gateway_config_poller_polls_total	Enterprise	Gauge despite the _total suffix; read the value, do not assume counter reset semantics.
gateway_config_poller_errors_total	Enterprise	Gauge despite the _total suffix; a rising value shows failed polls.
gateway_config_poller_areas_refreshed_total	Enterprise	Gauge despite the _total suffix; compare change over an interval deliberately.
gateway_grounding_embedding_failures_total{stage}	Enterprise	From 1.8.4. A grounding check could not embed a source or a claim (stage is source or claim). That check counts as not grounded, so under BLOCK the response is refused. A rising value points at the embedding service, not at the model's answers.

| dvara_session_kill_enforced_total{plane,source} | Enterprise | From 1.8.5. Calls refused because their session was killed. plane is llm, mcp or a2a; source is local (this Gateway's own kill), relay (another Gateway's) or deny_list (the revocation list). | | dvara_session_kill_relay_seconds | Enterprise | From 1.8.5. Time from a kill on one Gateway to its first refusal on another. | | dvara_loop_detector_fallback_total{reason} | Enterprise | From 1.8.5. Loop checks that judged the call on the Gateway's own history because the shared history did not answer within 50 ms. A steady rise means loops spread over Gateways are caught later. Expect some during a rolling upgrade. |

Observe the Anthropic Messages endpoint

From 1.8.5 the Gateway serves the Anthropic Messages API at /v1/messages. Two counters show what it did not govern or did not forward:

Metric	Distribution	Meaning
gateway_anthropic_opaque_blocks_total{direction,type}	Both	Content blocks of a type the Gateway does not read, passed on unchanged on a route to Anthropic.
gateway_anthropic_fields_dropped_total{field}	Both	Anthropic-only request content, such as cache_control or an unmodelled field, left out because the call went to another provider.

Observe the MCP request path

Metric	Distribution	Meaning
mcp_requests_total	Enterprise	Every /mcp request by workspace, server, operation, status and error_code, refused requests included. error_code is the lower-case code from the refusal's body, or none when the request was not refused.
mcp_latency_seconds	Enterprise	End-to-end MCP request latency.
mcp_errors_total	Enterprise	One count per refusal, by server and the same lower-case error_code.
mcp_tool_calls_total	Enterprise	Tool calls by workspace, server, tool, and status.
mcp_tool_call_latency_seconds	Enterprise	Tool-call latency by server and tool.

Update MCP dashboards and alerts for 1.8.4

From 1.8.4 the MCP error codes in metrics are lower case, the same code the refusal's body carries. A query such as mcp_errors_total{error_code="MCP_SERVER_NOT_FOUND"} now matches nothing; write mcp_errors_total{error_code="mcp_server_not_found"}. The same applies to every code, for example mcp_policy_denied. mcp_requests_total also gains the error_code label. Refusals written by the MCP plane's error handlers, such as 400 mcp_invalid_json, 404 mcp_not_found and 403 mcp_session_killed, were not counted in mcp_errors_total before 1.8.4 and now are, so the error rate can rise after the upgrade without any change in traffic. The error_code field of the MCP access log is lower case too. The native /mcp endpoint's own 400, 401 and 403 refusals still count as none.

Observe the A2A request path

Metric	Distribution	Meaning
a2a_requests_total	Enterprise	Every /a2a request by workspace, operation, status and error_code (none when not refused).
a2a_request_duration_seconds	Enterprise	A2A request latency by operation and status.
a2a_hops_total	Enterprise	Hops by workspace, target agent, operation, outcome and error_code.
a2a_hop_latency_seconds	Enterprise	Latency of hops that reached a peer, by target agent and operation.
dvara_a2a_delegation_refused_total	Enterprise	Hops the delegation guard refused, by workspace, reason and mode. From 1.8.4 mode says what happened: every refused hop is enforce, including an invalid chain in an observe workspace, which is always refused. observe counts only a hop that would have been refused and was let through.
dvara_a2a_delegation_depth	Enterprise	The chain depth each checked hop reaches. Use it to choose a2a.delegation.max-depth.
dvara_a2a_delegation_guard_active{reason}	Enterprise	From 1.8.4. 1 when the pod can run the delegation guard; 0 when it runs as off because it has no usable signing key.
dvara_a2a_delegation_binding_lookup_failed_total	Enterprise	From 1.8.4. Hops whose API key's agent binding could not be read, by workspace.
dvara_a2a_hop_budget_unavailable_total{workspace,outcome}	Enterprise	From 1.8.5. Hops whose session hop count could not be read in time. outcome is refused in a hardened workspace and allowed elsewhere.

Alert on dvara_a2a_delegation_guard_active == 0: the guard is not checking cycles or depth on that pod.

The three rate-limiter counters do not have one stable label set: the in-process and shared-cache implementations emit untagged series, while the Enterprise embedded-cluster limiter adds reason. Write queries that tolerate both shapes.

From 1.8, a refused 400 invalid_path, a 401, a 403 api_key_scope, and a 429 all create an access-log record, a request metric, and an audit event. The refusal code is stored as error_code; earlier releases omitted these attempts. Alert on increases in gateway_requests_total{status=~"400|401|403|429"}, then group the matching audit evidence by error_code. This exposes alternate-path probing, stolen-key probing, and sustained throttling that were previously invisible.

For policy refresh, status="degraded" means one or more policies failed to compile: DVARA names and skips the bad policies while valid policies continue to enforce. status="failure" means the refresh itself failed and the previous policy set remains active. Both outcomes require investigation.

Useful starting queries are:

# Request rate by provider and status
sum by (provider, status) (rate(gateway_requests_total[5m]))

# Published P95 latency by provider
max by (provider) (gateway_latency_seconds{quantile="0.95"})

# Provider error rate
sum by (provider, error_code) (rate(gateway_provider_errors_total[5m]))

# Policy refreshes that did not load the full intended policy set
increase(gateway_config_refresh_total{
  table="policies",status=~"degraded|failure"
}[15m])

Workspace and model labels are useful but can produce many time series. Define retention and recording rules around the cardinality your fleet actually creates; never place trace or session IDs in metric labels.

Monitor streaming enforcement bounds

Streaming enforcement metrics tell you when a response approaches or crosses the memory and scan bounds that keep whole-response governance finite. LLM series are available. Series with plane="A2A" appear only when the Enterprise A2A plane handles streamed traffic. Prometheus creates a series lazily, so an absent overflow or cancellation series means no matching event has been recorded since that process started.

Published metric	Prometheus series	Labels	What it means
gateway_streaming_enforcement_buffer_utilization	gateway_streaming_enforcement_buffer_utilization_ratio_count, _ratio_sum, and _ratio_max	plane, mode, bound	Fraction of a configured streaming buffer retained as events arrive. The ratio segment comes from the metric's base unit. A final fragment can make the observed maximum slightly greater than 1 before Deferred mode refuses it.
gateway_streaming_enforcement_overflow_total	Same name	plane, mode, bound	A stream crossed a configured bound. Deferred mode refuses the held response. Immediate mode continues delivery, retains only the bounded prefix for its final scan, and records incomplete scanning.
gateway_streaming_enforcement_cancellation_incomplete_total	Same name	plane, reason	Finalization of a cancelled Immediate stream failed, exceeded its five-second deadline, or could not be scheduled. Content already delivered can be incompletely scanned or recorded. This counter does not increment for a successfully finalized cancellation.

The label values are deliberately bounded:

Label	Values
plane	LLM, A2A
mode	IMMEDIATE, DEFERRED
bound	LLM: HELD_CHARS, plus TOOL_CALLS on overflow. A2A: HELD_CHARS, HELD_BYTES, or HELD_EVENTS.
reason	FAILED, TIMEOUT, REJECTED

Start with these queries:

# Highest retained fraction currently reported by each enforcement path
max by (plane, mode, bound) (
  gateway_streaming_enforcement_buffer_utilization_ratio_max
)

# Deferred responses refused after crossing a bound
sum by (plane, bound) (
  increase(gateway_streaming_enforcement_overflow_total{mode="DEFERRED"}[10m])
)

# Immediate responses whose tail was delivered beyond the retained scan prefix
sum by (plane, bound) (
  increase(gateway_streaming_enforcement_overflow_total{mode="IMMEDIATE"}[10m])
)

# Cancelled Immediate streams whose final scan or evidence work did not finish
sum by (plane, reason) (
  increase(gateway_streaming_enforcement_cancellation_incomplete_total[10m])
)

Use increase() for the counters because they restart with the process. The utilization summary has no workspace label; use the audit trail and request logs to investigate individual calls without creating unbounded metric series.

Export traces only when a collector is configured

DVARA creates request and provider observations, but it does not configure an OTLP destination by default. Set the collector's base URL to opt in:

OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318

Do not append /v1/traces; the exporter adds that path. An empty value is not the same as an unset value and can prevent startup, so remove the environment variable completely when export is disabled.

TRACING_SAMPLING_PROBABILITY accepts 0.0 through 1.0 and defaults to 1.0. Set the rate from observed volume and the debugging value of complete traces. The gen_ai.* semantic-convention attributes are on by default; set management.tracing.genai-semconv.enabled=false to suppress those attributes without removing the native provider and model attributes.

The main LLM observations are gateway.provider.chat, gateway.provider.stream, and gateway.provider.embed. Provider and model are low-cardinality attributes; token counts and session ID are trace attributes, not metric tags.

Confirm the governance evidence

In the Enterprise platform, open Governance → Audit, choose the workspace and time range, and filter for GATEWAY_RESPONSE. The row shows model, status, and latency. Expand it to inspect the payload, including the request's trace_id.

If the request was refused, also look for the feature-specific event, such as POLICY_DENIED, PII_DETECTED, or the relevant guardrail event. One request can create more than one audit event because the decision and final HTTP outcome answer different questions.

The Open Source distribution writes evidence only when DVARA_AUDIT_FILE_PATH is set. Its GATEWAY_RESPONSE record contains request outcome metadata but does not currently carry the trace ID, so correlate it by time, workspace, model, and status. See Audit and compliance for signing, verification, export, retention, and prompt-storage boundaries.

Turn monitoring into actionable alerts

The Kubernetes chart can create Gateway and Flightdeck ServiceMonitors and a PrometheusRule, but all three are off by default:

llmGatewayServer:
  serviceMonitor:
    enabled: true

flightdeck:
  serviceMonitor:
    enabled: true

alerts:
  enabled: true

These resources require Prometheus Operator support. The scraper's bearer secret must contain the same metrics key as the applications. A scrape that quietly returns 401 leaves every higher-level dashboard and alert blind.

The chart's rules cover operator-impacting conditions including stale revocations, stale configuration bundles, audit-chain gaps or broken links, SIEM export failures, spool depth and drain lag, missing spend/quota history, and dead-lettered email. From 1.8.2 every Enterprise Gateway serves from a bundle and spools its records, so the bundle and spool alerts apply to every install.

Add streaming rules that match the risk of each mode. Deferred overflow is a safe refusal but still affects availability, so start with a warning. Immediate overflow or incomplete cancellation finalization means content was already delivered without complete final scanning or evidence, so treat it as critical:

groups:
  - name: dvara-streaming-enforcement
    rules:
      - alert: DvaraDeferredStreamOverflow
        expr: increase(gateway_streaming_enforcement_overflow_total{mode="DEFERRED"}[10m]) > 0
        for: 0m
        labels:
          severity: warning
          component: data-plane
        annotations:
          summary: DVARA refused a Deferred stream after it crossed a governance bound.
          description: Check the plane and bound, then compare normal response sizes with the configured limit.
          runbook_url: https://runbooks.internal.example.com/dvara/streaming-enforcement

      - alert: DvaraImmediateStreamScanIncomplete
        expr: increase(gateway_streaming_enforcement_overflow_total{mode="IMMEDIATE"}[10m]) > 0
        for: 0m
        labels:
          severity: critical
          component: data-plane
        annotations:
          summary: DVARA delivered an Immediate stream beyond the retained scan prefix.
          description: The response tail was delivered but was not included in the final governance scan.
          runbook_url: https://runbooks.internal.example.com/dvara/streaming-enforcement

      - alert: DvaraCancelledStreamFinalizationIncomplete
        expr: increase(gateway_streaming_enforcement_cancellation_incomplete_total[10m]) > 0
        for: 0m
        labels:
          severity: critical
          component: data-plane
        annotations:
          summary: DVARA could not complete final governance work for a cancelled Immediate stream.
          description: Check the reason label; delivered partial content can be incompletely scanned or recorded.
          runbook_url: https://runbooks.internal.example.com/dvara/streaming-enforcement

Route the first alert to the team that owns response-size limits and client expectations. Route the critical alerts to the governance operations team. Inspect plane, bound, and reason before changing a limit: a sustained high utilization ratio supports capacity tuning, while one malformed or unusually large response may call for a tighter application limit instead.

Connect the alert manager to an owned destination, assign a runbook, and fire a synthetic alert before production traffic arrives. Operator alerts are separate from workspace-facing webhooks and email; testing one does not test the other.

Use the prepared dashboards as a starting point

The public dvara-examples repository contains dashboards for Gateway traffic, cost and budgets, policy and routing, MCP activity, and infrastructure. Treat them as a starting point: remove panels for features you do not run, add your service objectives, and test every alert against the labels emitted by your environment.

Complete the production observability check

Before launch, prove all of these with one test request and one controlled failure:

1. Readiness removes an unhealthy replica from service.
2. The response returns an X-Trace-ID.
3. A structured log is searchable by that ID.
4. Request, latency, token, and error metrics are being scraped.
5. A trace reaches the configured collector, if tracing is required.
6. The Enterprise audit row or Open Source audit-file record is present.
7. A synthetic alert reaches the person or system expected to act.

If any link is missing, the system is observable only in theory. Fix the gap before increasing traffic or enabling a more complex topology.

Continue with the related guide

• Review audit evidence and compliance exports
• Configure DVARA safely
• Deploy DVARA
• Operate DVARA with Flightdeck
• How the gateway runs without a database