PII Detection and Redaction

Runs unlicensed

Runs unlicensed, in full. Regex detection, reversible redaction and tokenization, Presidio/GLiNER NER and the in-process embedded scanner are all available in the Development posture. Earlier releases withheld everything but regex; 1.7.0 removed that split.

What a licence adds is the MCP and A2A planes, so it extends PII enforcement to tool calls and agent hops rather than unlocking anything on the model-call path.

DVARA detects and enforces policies on Personally Identifiable Information (PII) and Protected Health Information (PHI) in prompts and LLM responses. This prevents sensitive data from being forwarded to external LLM providers without explicit authorization.

How It Works

PII enforcement runs at two points in the LLM request lifecycle:

1. Request scanning — After policy evaluation, before dispatching to the LLM provider. Detects PII in user messages, tool result blocks, and — on function-calling requests — assistant tool-call arguments (tool_calls[].arguments).
2. Response scanning — After receiving the LLM response, before returning to the client. Detects PII in assistant messages (output leak detection).

Additionally, PII is always stripped from requests before they are written to the response cache.

For MCP (Model Context Protocol) traffic, the DVARA MCP Gateway runs the same PII enforcement against tool calls:

1. Argument scanning — For tools/call operations, the MCP Gateway recursively scans every string value in the tool-call arguments before forwarding to the upstream MCP server.
2. Response scanning — After the MCP server responds (2xx only), the MCP Gateway recursively scans every string value in the response body for PII output leaks before returning to the agent.

MCP PII scanning uses the same configuration (global dvara.llm-gateway.pii.* and per-workspace pii.* metadata) and the same detection engine as LLM PII scanning.

Supported PII Types

Type	Label	Detection Method
Email addresses	email	Regex pattern
US phone numbers	phone_us	Regex (with optional +1)
International phone numbers	phone_intl	Regex (E.164 format)
Social Security Numbers	ssn	Regex (rejects 000/666/9xx prefixes)
Credit card numbers	credit_card	Regex + Luhn checksum validation
Dates of birth	dob	Regex (MM/DD/YYYY or MM-DD-YYYY)
IPv4 addresses	ipv4	Regex
US passport numbers	passport_us	Regex
Driver's license numbers	drivers_license	Keyword (driver's license) + alphanumeric ID
IBAN numbers	iban	Regex (international bank account)
Medical Record Numbers	mrn	Keyword (MRN/Medical Record) + digits
DEA numbers	dea	Format regex + DEA checksum validation
NPI numbers	npi	Context keyword "NPI" required + Luhn checksum
Person names	person_name	Salutation heuristic (Mr/Mrs/Dr/Prof), confidence 0.7
India Aadhaar	aadhaar	Regex + Verhoeff check-digit validation
India PAN	pan	Regex + format/structural validation (holder-type char; not a numeric checksum)

Aadhaar and PAN are validated so an arbitrary 12-digit number or AAAAA9999A-shaped string isn't flagged: Aadhaar's trailing digit is a Verhoeff check digit (a wrong or transposed digit is rejected), and PAN is validated structurally — 5 letters + 4 digits + 1 letter with a recognised holder-type 4th character (PAN carries no numeric checksum). A Verhoeff-validated Aadhaar outranks a looser phone-number match on the same digits.

Actions

Each PII detection can trigger one of three actions:

Action	Behavior	Audit Event
LOG	Log the detection, forward request unchanged	PII_DETECTED
BLOCK	Reject the request with HTTP 400, type: pii_violation, code: pii_detected	PII_DETECTED
REDACT	Replace PII with reversible tokens, forward modified request	PII_REDACTED

When REDACT is active, PII values are replaced with tokens like {{PII_EMAIL_a1b2c3d4}}. The original values are encrypted (AES-256-GCM) and held in a durable, workspace-scoped token store (survives restarts and is shared across replicas). Authorized admins can restore the original text on demand from the Console's PII page.

Transparent round-trip (opt-in). When pii.auto-detokenize-response=true for a workspace (default off), tokens that were minted from that request's PII and echoed back by the model are automatically restored in the response the client receives — a PII_RESPONSE_DETOKENIZED audit event records it (lengths/counts only, never the revealed text). Restoration is request-scoped (only tokens the request carried, so any PII the model newly leaks stays masked) and runs after the response is written to the semantic cache, so the cache only ever holds the tokenized form. Non-streaming /v1/chat/completions only.

Response scanning emits a PII_OUTPUT_LEAK audit event on every detection. The response body is rewritten with tokenized placeholders only when the workspace action is REDACT; on LOG the response is returned unchanged, and BLOCK is not applied to response scanning (the audit event is your signal).

Configuration

Global Configuration

Add to application.yml:

dvara:
  llm-gateway:
    pii:
      enabled: true                                          # enable PII scanning
      provider: regex                                        # regex (default); a Presidio endpoint alone enables layered NER (see below)
      default-action: LOG                                    # LOG, BLOCK, or REDACT
      scan-responses: true                                   # scan LLM responses for output leaks
      auto-detokenize-response: false                        # opt-in transparent round-trip (default off)
      strip-before-cache: true                               # always redact before caching
      max-tokens-per-workspace: 50000                        # max stored PII tokens per workspace
      token-retention-days: 30                               # auto-expire after N days

  # NOTE: the token-store master secret sits under `dvara.pii`, NOT
  # `dvara.llm-gateway.pii` — it is cross-cutting. The gateway and the MCP
  # gateway both mint tokens; Flightdeck detokenizes and purges them, so every
  # app must be given the same value or tokens minted by one are unreadable
  # by another.
  pii:
    token-encryption-master-password: ${DVARA_PII_TOKEN_ENCRYPTION_MASTER_PASSWORD:}

Embedded PII Scanner (In-Process, No Third-Party Required)

For broader filter coverage without deploying a sidecar, enable the embedded PII scanner. It runs entirely in-process (Apache-2.0 licensed) with sub-millisecond per-call latency, no network calls, no API keys, and no external service to operate.

There are two switches and they do different things. Neither is licence-gated; both are off by default, so the scanner does nothing until you pick one.

Setting	Default	What the scanner's hits become
dvara.llm-gateway.pii.embedded.enabled	false	PII-pipeline detections — redacted or tokenized, and they take part in the detokenize round-trip
dvara.llm-gateway.guardrail.embedded.enabled	false	Guardrail detections — flagged, blocked or logged, never redacted

dvara:
  llm-gateway:
    guardrail:
      embedded:
        enabled: true     # opt-in — flag-only, the guardrail pipeline

Do not set both. The same entity is then redacted by the PII pipeline and flagged by the guardrail pipeline, and the gateway logs a warning at startup saying so. Pick the pipeline that matches what you want to happen to the match. When enabled, the embedded scanner runs alongside the built-in regex detector as a complementary layer. The default policy enables 14 filter types — the deterministic filters the regex detector does not already cover — so the two layers don't both flag the same entity:

Category	Filters
Financial	BANK_ROUTING_NUMBER, BITCOIN_ADDRESS, CURRENCY
Contact	PHONE_NUMBER_EXTENSION
Network / device	MAC_ADDRESS, URL
Addresses	STREET_ADDRESS, ZIP_CODE, STATE_ABBREVIATION
Dates & demographics	DATE, AGE
Vehicle / shipping	VIN, TRACKING_NUMBER
PHI	PHYSICIAN_NAME

Regex-covered types are excluded from the default — SSN, PASSPORT_NUMBER, DRIVERS_LICENSE_NUMBER, CREDIT_CARD, IBAN_CODE, PHONE_NUMBER, EMAIL_ADDRESS, IP_ADDRESS are already handled (with checksums) by the always-on regex detector, so the embedded scanner leaves them alone to avoid double-detection. If you run the embedded scanner without the regex PII detector, opt them back in via embedded.enabled-filters.

Eight further filters ship in the catalog but are opt-in per workspace because they match against dictionaries (names, cities, counties, states, hospitals, medical conditions) that collide with ordinary English on free-form chat workloads — too many false positives to auto-enable: FIRST_NAME, SURNAME, LOCATION_CITY, LOCATION_STATE, LOCATION_COUNTY, HOSPITAL, MEDICAL_CONDITION, IDENTIFIER (the last over-matches long digit sequences without dictionary tuning).

Per-workspace filter restriction is editable from the Built-in PII Filters tab in the DVARA Flightdeck workspace form, or by setting the workspace metadata key embedded.enabled-filters to a comma-separated list of filter-type names.

Limitations. The embedded scanner is best-effort pattern matching with no Luhn validation. It can mis-classify long digit sequences (e.g. classify a credit card number as a drivers license number depending on filter ordering). For production credit card and NPI detection, rely on the always-on built-in regex detector (which Luhn-validates). Use the embedded scanner as an additive layer for the broader filter coverage it provides (MAC address, IBAN, bitcoin, VIN, etc.) that the curated regex set doesn't include.

Because the default embedded filter set is complementary (non-overlapping with the regex detector), a given entity is caught by exactly one layer — no duplicate detections/audits for the default configuration. (If you opt the regex-covered types back in, both layers may flag the same match; that's the trade-off for the extra coverage.)

Redact instead of flag — the embedded scanner in the PII pipeline

By default the embedded scanner is a guardrail-pipeline layer: its hits are governance detections (FLAG / BLOCK / LOG), but the matched values are not redacted — a detected MAC address or street address is flagged yet still forwarded to the upstream model. To make those broad structured-PII hits tokenize/redact like regular PII, promote the embedded scanner into the PII pipeline instead:

dvara:
  llm-gateway:
    pii:
      embedded:
        enabled: true     # the embedded scanner becomes a PII detector (redact/tokenize)

With pii.embedded.enabled: true, the embedded detector joins the regex (and optional Presidio) detectors in the same composite: matches are deduplicated by confidence, tokenized/redacted, and eligible for the detokenize round-trip. Only redactable filter types are mapped:

• Regex-covered structured PII (SSN, CREDIT_CARD, EMAIL_ADDRESS, …) → the matching first-class entity type. (The composite dedups the overlap with the regex detector, so you don't need the complementary-default exclusion here.)
• Names (FIRST_NAME, SURNAME, PHYSICIAN_NAME) → PERSON_NAME.
• Broad structured PII with no first-class type (MAC_ADDRESS, ZIP_CODE, BANK_ROUTING_NUMBER, VIN, BITCOIN_ADDRESS, TRACKING_NUMBER, STREET_ADDRESS, location) → CUSTOM, with the specific kind preserved in the detection label (e.g. mac_address).
• Non-redactable content signals (CURRENCY, AGE, DATE, URL, STATE_ABBREVIATION, MEDICAL_CONDITION, PHONE_NUMBER_EXTENSION) are not mapped — redacting a currency amount or URL would damage the prompt, so these stay flag-only in the guardrail pipeline.

The default PII-pipeline filter set is the complementary structured-PII types (MAC_ADDRESS, ZIP_CODE, BANK_ROUTING_NUMBER, VIN, BITCOIN_ADDRESS, TRACKING_NUMBER, STREET_ADDRESS); override it globally with dvara.llm-gateway.pii.embedded.filters. This is a global setting — per-workspace filter selection remains a guardrail-pipeline capability (embedded.enabled-filters), because the PII scan path is workspace-agnostic.

Pick one pipeline. Enabling both guardrail.embedded.enabled and pii.embedded.enabled makes the same entity get flagged (guardrail) and redacted (PII) — a double detection. The gateway logs a startup WARN if both are on. Choose the guardrail path when you want to observe/block, or the PII path when you want to redact.

Presidio Integration — layered NER (separate pod)

For industrial-grade PII detection using NER models, run Microsoft Presidio as a separate pod and point the gateway at it. Setting a non-blank presidio.endpoint alone enables layered regex + NER — you no longer need provider: presidio (it's kept as an explicit alias). An endpoint is the single toggle:

dvara:
  llm-gateway:
    pii:
      presidio:
        endpoint: http://presidio-analyzer:3000/analyze      # endpoint alone enables layered NER
        language: en                                         # analysis language
        score-threshold: 0.5                                 # minimum confidence
        timeout-seconds: 5                                   # HTTP timeout
        cache-max-size: 1000                                 # LRU cache entries (0 = disabled)
        cache-ttl-seconds: 300                               # cache TTL

The DVARA Helm chart ships an optional in-cluster Presidio pod (piiNer.enabled=true, default off) that deploys the analyzer and auto-wires this endpoint — the one Helm value that turns on layered detection. The pod's default image is DVARA's GLiNER-configured Presidio build (gliner_multi_pii-v1, Apache-2.0) for zero-shot PII accuracy on unstructured PII (names / locations / organisations); the model is baked into the image so pods start without a model download. The gateway's /analyze contract is identical whether the pod runs GLiNER or the stock spaCy analyzer — set piiNer.image.repository to mcr.microsoft.com/presidio-analyzer to fall back to spaCy.

With an endpoint set, the gateway merges results from the built-in regex detector (16 patterns + checksum/format validation) and Presidio (NER-based PERSON, LOCATION, ORGANIZATION, and more). Overlapping detections are deduplicated, keeping the higher-confidence result. The startup log states the active posture (regex or regex+presidio).

Detection layers at a glance. Regex (always-on, checksum-validated structured PII) → Presidio NER (optional pod, unstructured PII like names/locations, same tokenize/redact path) → Embedded scanner (optional in-process, rule-based broad patterns; runs either in the guardrail layer as flag-only, guardrail.embedded.enabled, or in the PII pipeline as redact/tokenize, pii.embedded.enabled — pick one). Layered NER is not on by default — the default is regex-only.

Property	Default	Description
dvara.llm-gateway.pii.provider	regex	Detection provider alias. A non-blank presidio.endpoint enables layered NER regardless of this value
dvara.llm-gateway.pii.auto-detokenize-response	false	Opt-in transparent round-trip — restore request-minted tokens in the response (non-streaming)
dvara.llm-gateway.pii.presidio.endpoint	—	Presidio analyzer endpoint URL — the single toggle for layered NER
dvara.llm-gateway.pii.presidio.language	en	Presidio analysis language
dvara.llm-gateway.pii.presidio.score-threshold	0.5	Minimum Presidio confidence score
dvara.llm-gateway.pii.presidio.timeout-seconds	5	HTTP call timeout
dvara.llm-gateway.pii.presidio.cache-max-size	1000	LRU cache max entries (0 = disabled)
dvara.llm-gateway.pii.presidio.cache-ttl-seconds	300	Cache entry TTL

Presidio is fail-open: if the sidecar is unreachable, detection falls back to regex-only results. Cached results use SHA-256 of the input text as cache key; delegate failures are cached with a 30-second TTL to prevent thundering herd.

Per-Workspace Configuration

Since 1.7.0 the typed settings win, and a metadata edit can do nothing

PII, guardrail, agentic and approval settings moved into typed tables, and the resolver reads typed store first, metadata second.

Once a workspace has a typed row, editing the matching metadata key has no effect and raises no error. Detection still runs; the action simply stays where the typed row put it. Nothing warns you, and the exported document still shows the metadata value you set — which is the part that makes this cost an afternoon rather than a minute.

Write through the Console's Data Protection page, which sets the typed store. Treat the metadata keys below as the second-precedence fallback they now are: they still apply to a workspace that has no typed row, which is why they are documented, but they are no longer the place to make a change.

The fastest way to override PII behavior for a workspace is the PII & DLP tab in the DVARA Flightdeck workspace form (Workspaces → Edit). Form-based controls cover pii.enabled, pii.action, and pii.scan-responses. Updates emit a WORKSPACE_METADATA_UPDATED audit event with a diff.

For Terraform, CI/CD and other programmatic tooling, set the same metadata keys through the Console's GitOps export/import — each workspace in the exported document carries a metadata object:

{ "workspaces": [ { "id": "acme-corp", "metadata": {
      "pii.enabled": "true",
      "pii.action": "REDACT"
} } ] }

pii.custom-patterns is currently API-only — the UI form does not yet expose a custom-pattern editor.

pii.custom-patterns must be a nested object, not a stringified JSON value

The value of pii.custom-patterns must be a JSON object (label → regex map), as shown above. Passing it as a stringified JSON value (e.g. "pii.custom-patterns": "{\"employee_id\": \"EMP-\\d{6}\"}") silently fails — the gateway's workspace-config resolver checks the value with instanceof Map, the string fails the check, custom patterns are dropped, and no error is logged. The workspace simply gets zero custom-pattern matches.

Metadata Key	Values	Description
pii.enabled	true / false	Override global PII detection
pii.action	BLOCK / REDACT / LOG	Override default action
pii.scan-responses	true / false	Override response scanning
pii.custom-patterns	JSON object (label → regex)	Add custom regex patterns. Must be a nested object, not a stringified JSON value.

Custom Patterns

Custom patterns are specified as a JSON object where keys are labels and values are regex strings. They are detected as CUSTOM entity type:

{
  "employee_id": "EMP-\\d{6}",
  "internal_project": "PROJ-[A-Z]{2,4}-\\d{4}"
}

Managing tokens

Detokenize

Restore original PII values from redacted text on the Console's PII page (/pii/detokenize), or from the workspace Portal at /portal/data-protection. Both are workspace-scoped, return no-store headers, and never write the recovered values to the audit payload — only a PII_DETOKENIZE_REQUESTED event recording that it happened.

Paste the tokenised text and the page returns it with the original values restored:

Contact user@example.com about account

Requires owner or policy-admin role on the Console; workspace admin on the Portal.

Purge Tokens

Remove all stored PII tokens for a workspace (irreversible):

Purge from the Console's **PII** page, or from the workspace Portal at
`/portal/data-protection`. Both require typing the workspace name to confirm, and emit a
`PII_TOKENS_PURGED` audit event. It is irreversible: the tokens become permanently
unresolvable.

Requires `owner` role.

## Audit Trail

All PII events are written to the audit trail. The audit payload includes entity types and counts but **never includes the actual PII values**.

### LLM Traffic

| Event Type | When |
|---|---|
| `PII_DETECTED` | PII found in request (action: LOG or BLOCK) |
| `PII_REDACTED` | PII redacted from request (action: REDACT) |
| `PII_OUTPUT_LEAK` | PII detected in LLM response |

### MCP Traffic

| Event Type | When |
|---|---|
| `MCP_PII_DETECTED` | PII found in tool call arguments (action: LOG or BLOCK) |
| `MCP_PII_REDACTED` | PII redacted from tool call arguments (action: REDACT) |
| `MCP_PII_OUTPUT_LEAK` | PII detected in MCP server response |

MCP PII audit events include `server_id`, `tool_name`, `entity_count`, `entity_types`, `source` (`request` / `response`), `action`, and `trace_id` — enough for auditors to filter PII activity by tool, by workspace, or by direction, and to correlate against the gateway trace for the originating LLM turn.

Example audit event payload (LLM PII path — the MCP path adds `server_id` / `tool_name` / `action` on top):

```json
{
  "eventType": "PII_REDACTED",
  "tenantId": "acme-corp",
  "payload": {
    "source": "request",
    "entity_count": 2,
    "entity_types": "EMAIL, SSN",
    "entity_type_counts": {"EMAIL": 1, "SSN": 1}
  }
}

A few notes on the wire format that aren't obvious from a single example:

• tenantId is on the envelope, not in payload. Filter audit-event queries by the envelope field, not by a payload field.
• entity_types is a comma-separated string (Java List.toString().joining(", ")), not a JSON array. Receivers parsing as an array would fail.
• entity_type_counts is the per-type breakdown — useful for dashboards that want to plot email vs ssn vs credit_card over time without re-parsing the string above.
• LLM PII payloads do not include an action field — the event type itself (PII_DETECTED vs PII_REDACTED) carries that signal. The MCP PII path does include action since the same MCP_PII_DETECTED event type is used for both LOG and BLOCK actions.

Access control

PII admin operations are gated by platform role — there are no granular PII permission scopes; access follows DVARA's standard six-role model.

Operation	Roles
Detokenize (Console PII → Detokenize)	owner, policy-admin
Purge a workspace's PII tokens (Console PII → Purge)	owner
Configure per-workspace PII settings	owner / policy-admin (Console); workspace admin (Portal → Data Protection)
View PII audit events	any admin role (results scoped to the caller's workspace)

Streaming PII Enforcement

When stream=true, the gateway wraps the SSE chunk iterator with buffered scanning that detects and redacts PII in-flight before forwarding to the client.

How it works:

1. Text deltas accumulate in a rolling buffer
2. When the buffer exceeds streaming-scan-window-size (default 256 chars), the safe region is scanned
3. An overlap margin (default 64 chars) catches PII spanning chunk boundaries
4. REDACT: PII tokens replaced in buffered text before forwarding
5. BLOCK: Stream terminates with finishReason=content_filter
6. LOG: Stream continues, summary audit event at end

Configuration:

Property	Default	Description
dvara.llm-gateway.pii.scan-streaming-responses	true	Enable PII scanning on streaming responses
dvara.llm-gateway.pii.streaming-scan-window-size	256	Chars buffered before scan trigger
dvara.llm-gateway.pii.streaming-overlap-margin	64	Chars retained between windows for boundary detection

Per-workspace override: set pii.scan-streaming-responses in the workspace metadata.

Security Considerations

• PII tokens are encrypted at rest using AES-256-GCM with the configured token-encryption-password
• Token storage is durable — encrypted at rest in PostgreSQL, workspace-scoped, and shared across replicas, so tokens survive gateway restarts and detokenize correctly on any pod
• Audit events never contain the actual PII values, only entity types and counts
• The BLOCK action rejects the entire request — no partial content is forwarded
• Cache stripping always uses REDACT behavior regardless of the workspace's configured action