Shadow AI: you can't govern what you can't see

How many LLM calls did your company make yesterday?

Most platform leads can't answer. Not roughly, not which providers, not which teams. That gap has a name — shadow AI — and it is a visibility problem long before it is a policy problem.

How it happens

Nobody decides to have shadow AI. It arrives one reasonable decision at a time.

One team adds an OpenAI key to a service. Another wires up Anthropic for a different feature. A third is calling a model through a SaaS tool that nobody logged as an AI vendor. A fourth is running an agent that calls tools on its own.

Four sensible decisions. Four paths out of your network, four sets of credentials, four logging conventions. No single place the traffic passes through — so no single place to see it.

Four questions that measure it

Ask these about yesterday, not last quarter.

1. Which applications called a model, and which model did each one use?
2. What did they send? Would you know if customer data had left?
3. Who authorised it? For an agent: which tool did it call, on whose behalf?
4. Prove it. Produce a record you would hand an auditor.

If those take a week of Slack archaeology, the problem isn't that your policy is badly written. There is nothing in the request path that could enforce a policy even if it were perfect.

Why the usual answers don't close it

• A written policy describes intent. It doesn't change what happens on the next request.
• Your API gateway sees an HTTPS POST to a hostname. Not the model, not the tokens, not the prompt, not the tool call.
• Per-app logging is four formats in four places that stop the moment someone is in a hurry.
• Provider dashboards are organised around billing, one per provider, and know nothing about that SaaS tool.

Each is useful. None produces an inventory.

The fix is an old one

When services multiplied, we stopped asking every team to get retries and mTLS right and put a service mesh in the path. When APIs multiplied, we put a gateway in front.

Same move. Route model traffic through one point and visibility stops depending on whether every team remembered. It becomes a property of the architecture.

The control has to sit in the request path. Something that observes and reports is a monitoring tool — if it can't refuse a call, it can't enforce anything.

What it costs to adopt

Almost every provider now speaks an OpenAI-compatible API, so this is a base URL, not a rewrite.

Python — before. Each service holds its own key and makes its own way out.

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

After. Same SDK, same call sites.

client = OpenAI(
    base_url="https://dvara.internal/v1",
    api_key=os.environ["DVARA_API_KEY"],
)

Java, Spring AI. Configuration only — no Java changes.

spring:
  ai:
    openai:
      base-url: ${DVARA_URL:http://localhost:8080/v1}
      api-key: ${DVARA_API_KEY}

One gotcha worth an afternoon of your life: Spring AI 2.0 uses the official OpenAI SDK, which POSTs to ${base-url}/chat/completions. The base URL must include /v1 yourself — unlike Spring AI 1.x, the SDK does not prepend it.

Agents travel the same road. Point the MCP client at the governed endpoint and tool calls are governed too:

spring:
  ai:
    mcp:
      client:
        streamable-http:
          connections:
            dvara:
              url: ${DVARA_MCP_URL:http://localhost:8070/mcp}

That matters more than it looks. Most AI-governance conversations stop at the model call. But an agent that can call tools can read a database, hit an internal API, or move money. The model call is the least dangerous thing an agent does.

What this actually produced

Everything below is from a local run against the mock provider — no real provider traffic, so the model names are stand-ins. What is real is the record the gateway kept without any application being changed.

A hundred and twenty-four governed calls later, the inventory that did not exist:

Model	Calls	Avg latency
mock/gpt-4o-mini	27	44.9 ms
mock/gpt-4o	21	59.2 ms
mock/claude-3-haiku	18	60.3 ms
mock/claude-3-5-sonnet	16	59.7 ms
mock/gemini-1.5-pro	15	55.9 ms
mock/gpt-4o-expensive	14	60.4 ms
mock/mistral-large	13	44.5 ms

That table is a GROUP BY over the stream above it — every governed call, with the model it went to and the workspace it came from, in one place. Question one, answered by a query rather than a week of archaeology.

Two requests carrying a US Social Security number were detected on the way out and recorded as PII_DETECTED, with the entity type — before the call left the boundary.

And the audit trail is a hash chain rather than a log file that claims to be one. 429 signed envelopes, each carrying its own HMAC and its predecessor's hash.

The chain is per writing process, which is the part worth knowing before you check one: this install had four — the gateway, the console, and two more from restarts — so there are four chains, not one, and each numbers its own envelopes from 1. Checking each chain against itself:

• every chain runs 1 to N with no gaps (420, 4, 3 and 2 envelopes)
• zero envelopes, across all 429, whose previous_hash failed to match the preceding hmac

Compare envelopes across chains instead and you will get breaks that are not there — the sequence numbers interleave. It is the first thing to get wrong.

Remove or edit a record and the chain stops matching at that point. That is the difference between logs and evidence.

Finally, visibility is only worth having if you can act on it. A rule denying one model:

version: "1"
rules:
  - id: no-expensive-model
    conditions:
      model:
        denylist: [mock/gpt-4o-expensive]
    action: DENY

The refusal names the rule that fired, and it happens before the provider is called — nothing sent, nothing billed. Every other model keeps working.

The honest objection

One point in the path is one point of failure, and anyone who has run infrastructure will say so immediately. That has to be answered in the design, not the pitch: capability-aware failover so one provider outage doesn't take the path down, circuit breakers, and the ability to run inside your own perimeter so the control point isn't a new third party holding your prompts.

If the thing you added for governance becomes the reason you're down, you've made the system worse, and no audit trail compensates for that.

Where DVARA fits

This is what we build. DVARA is a self-hosted control plane that sits in the request path for model calls, MCP tool calls and agent-to-agent hops, applying one policy set and one audit trail across all three. Everything above is its actual configuration and its actual output.

We work across all three planes rather than the model call alone for the reason in the middle of this piece: by the time an agent is invoking tools, the model call is the least consequential hop in the chain.

If you can already answer the four questions at the top, you don't need us. If answering them would take a week, the gap is architectural — and it doesn't close by writing it down.