Skip to main content
Version: 1.8.0

Control request and token throughput

DVARA rate limits stop one application from consuming all of your model capacity. You can cap requests and tokens per API key, then narrow those caps for a workspace, team, or account.

Rate limiting runs in the Development posture and is off by default. The Open Source distribution uses a separate token bucket in each Gateway process. The Enterprise platform can enforce one limit across the fleet through its embedded cluster or an external Valkey or Redis service.

Enable a local rate limit​

Add the following settings to application.yml:

dvara:
  llm-gateway:
    rate-limit:
      enabled: true
      per-key:
        requests-per-minute: 100
        tokens-per-minute: 100000
        requests-burst: 120
        tokens-burst: 120000

Or set the equivalent environment variables:

DVARA_LLM_GATEWAY_RATE_LIMIT_ENABLED=true
DVARA_LLM_GATEWAY_RATE_LIMIT_PER_KEY_REQUESTS_PER_MINUTE=100
DVARA_LLM_GATEWAY_RATE_LIMIT_PER_KEY_TOKENS_PER_MINUTE=100000
DVARA_LLM_GATEWAY_RATE_LIMIT_PER_KEY_REQUESTS_BURST=120
DVARA_LLM_GATEWAY_RATE_LIMIT_PER_KEY_TOKENS_BURST=120000

The two *-per-minute values are sustained refill rates. The two *-burst values are the most unused capacity a token bucket can accumulate. A burst value of 0, or an omitted burst value, uses the matching per-minute rate as its capacity. A zero per-minute rate disables that dimension in the shared-cache token buckets; negative values are refused.

These global settings are read at startup. Restart the Gateway after changing them.

Test the limit​

Replace <dvara-api-key> with a valid workspace API key, then send enough requests to exhaust its request bucket:

curl --request POST 'http://localhost:8080/v1/chat/completions' \
  --header 'Authorization: Bearer <dvara-api-key>' \
  --header 'Content-Type: application/json' \
  --data '{
    "model": "gpt-4o-mini",
    "messages": [{"role": "user", "content": "Summarize this quarter in one sentence."}]
  }'

The request that exceeds the active limit is refused before a provider call:

HTTP/1.1 429 Too Many Requests
Retry-After: 60
X-RateLimit-Retry-After-Seconds: 60
X-Trace-Id: a6783439db1f46a6bfed511a0011e955
Content-Type: application/json

DVARA records the 429 in the structured access log, request metrics, and audit evidence. Use the trace ID to correlate the refusal; no provider usage is created because the request stops at admission.

{
  "error": {
    "message": "Rate limit exceeded",
    "type": "rate_limit_error",
    "code": "rate_limit_exceeded",
    "trace_id": "a6783439db1f46a6bfed511a0011e955",
    "rate_limit": {
      "limited_resource": "requests",
      "limit_type": "requests_per_minute",
      "limit": 100,
      "remaining": 0,
      "retry_after_seconds": 60,
      "reset_at": "2026-09-12T15:31:00Z"
    }
  }
}

Verify the decision in gateway_rate_limit_denied_total{reason="request"}. A token-limit refusal uses limited_resource: tokens and limit_type: tokens_per_minute, and also returns X-RateLimit-Tokens-Limit, X-RateLimit-Tokens-Remaining, and X-RateLimit-Reset.

Understand what each limit counts​

Every canonical /v1/* request consumes one request from the bucket identified by its resolved API-key ID. Missing, invalid, revoked, or expired keys are refused before rate limiting.

Token accounting depends on the request path:

  • Chat completions reserve an estimate at admission, then settle the difference when actual or estimated final usage is known.
  • Responses and embeddings reserve no input estimate in the request filter, then settle their final usage.
  • Other /v1/* operations consume request capacity but do not add token usage through this request filter.

Settlement never charges the estimate and the complete final usage as two separate amounts. If final usage is lower, DVARA refunds the difference up to the bucket's burst capacity. If it is higher, DVARA charges the difference. For chat request bodies over 10 MiB, DVARA skips the admission estimate, forwards the complete body, and settles final usage later.

Rotating an API key creates a new bucket because the replacement has a new opaque key ID. Each key in a workspace receives the same effective ceiling, but the keys do not share one bucket. An embedding application with no key repository uses a SHA-256 fingerprint of the bearer value instead; DVARA does not retain the plaintext credential.

The external shared-cache backend stores that opaque key ID, or the SHA-256 fingerprint used by an embedding without a key repository. It never uses the plaintext bearer value as a cache key.

Choose the right backend​

Distribution and configurationAlgorithmScopeBurst settings
Open Source distributionToken bucketOne Gateway processApplied
Enterprise platform with external Valkey or RedisDistributed token bucketAll Gateways using the same storeApplied
Enterprise platform with its embedded cluster60-second sliding windowGateways in that clusterNot applied; the rate itself is the window cap

With the Open Source distribution, N replicas can admit about N times the configured rate because each process owns its counters. Use the Enterprise shared-cache backend when one fleet-wide token bucket is required.

Do not set the embedded request rate to zero in 1.8

The Enterprise embedded backend treats requests-per-minute: 0 as a zero-capacity window and refuses every request. Use a positive request rate, or disable rate limiting with rate-limit.enabled: false.

Connect an external shared cache​

Set the shared-cache URL in the Enterprise Gateway environment. The URL selects the backend; there is no second backend-selection switch.

dvara:
  llm-gateway:
    rate-limit:
      enabled: true
      shared-cache:
        url: redis://valkey.internal:6379
        failure-mode: OPEN

failure-mode accepts OPEN or CLOSED and defaults to OPEN. The Gateway starts without opening a cache connection. It connects on the first governed request:

  • OPEN allows that request when the store is unreachable and records a rate-limit error.
  • CLOSED refuses it when the store is unreachable.

Logs identify the configured host and port without printing credentials embedded in the URL.

Rename the old settings before upgrading

dvara.llm-gateway.rate-limit.resp.url and dvara.llm-gateway.rate-limit.resp.failure-mode are ignored in 1.8. Replace them with rate-limit.shared-cache.url and rate-limit.shared-cache.failure-mode. Leaving the old URL can put each replica back on its normal limiter instead of the intended shared backend.

Connect the cache deployed by Helm​

valkey.enabled or redis.enabled deploys a single-node evaluation cache. It does not connect the Gateway to that service. Until the chart exposes a dedicated rate-limit value, add the URL through llmGatewayServer.extraEnv:

valkey:
  enabled: true

llmGatewayServer:
  extraEnv:
    - name: DVARA_LLM_GATEWAY_RATE_LIMIT_SHARED_CACHE_URL
      value: redis://dvara-valkey:6379
    - name: DVARA_LLM_GATEWAY_RATE_LIMIT_ENABLED
      value: "true"

Confirm the Service name generated for your Helm release before applying the URL. Without the shared-cache URL, the cache pod is idle and the Gateway keeps using its normally selected limiter. The bundled cache is single-node and intended for evaluation; use a production cache topology for a production fleet.

The same boundary applies to Docker Compose: starting a Valkey or Redis service does not wire the Gateway. Set DVARA_LLM_GATEWAY_RATE_LIMIT_SHARED_CACHE_URL separately.

Workspace token overrides and shared-cache settlement

In 1.8, the shared-cache backend applies a workspace token override when it reserves capacity, but settles final usage against the global token settings. This can weaken a narrower workspace token cap after a completion settles. Use the embedded backend when a strict workspace token ceiling is required.

Narrow limits by workspace, team, or account​

The global values are the default for each API key. A workspace can override either dimension with these settings:

Workspace settingEffect
rate-limit.requests-per-minuteRequest rate for each API key in the workspace
rate-limit.tokens-per-minuteToken rate for each API key in the workspace

In the Open Source distribution, place the settings in workspaces[].metadata in gateway.yaml. In the Enterprise platform, use the workspace settings in DVARA Flightdeck. Blank workspace fields inherit the global value. Changes normally reach the Gateway after the workspace-setting cache expires; its default TTL is 5 seconds.

The Enterprise hierarchy resolves limits in this order:

Account → Team → Workspace

For each dimension, the smallest positive value wins. An account or team can narrow a workspace limit but cannot raise it.

To manage hierarchy limits in Flightdeck:

  1. Open Access → Teams at /teams. From 1.8.3 the account limit is also on Access → Accounts.
  2. Choose Team limit or Account limit.
  3. Enter requests per minute, tokens per minute, or both. Leave a field blank to inherit for that dimension.
  4. Select Save. Leaving both fields blank clears that level's stored limit.

Saving a scoped limit records SCOPED_RATE_LIMIT_UPDATED. Clearing it records SCOPED_RATE_LIMIT_CLEARED.

Do not use zero for a hierarchy limit

Use blank fields to inherit or clear a team or account limit. The 1.8 Flightdeck and hierarchy paths do not yet agree on the meaning of an entered zero, so do not rely on zero as an operational control.

Preserve capacity for higher-priority work​

Enterprise only

Not included in DVARA Open Source.

Priority admission control can refuse lower-priority chat requests as concurrent load rises. It is separate from request and token rate limiting and is off by default.

dvara:
  llm-gateway:
    routing:
      priority:
        enabled: true
        max-concurrent-requests: 1000
        tiers:
          premium:
            throttle-threshold-pct: 100
          standard:
            throttle-threshold-pct: 80
          bulk:
            throttle-threshold-pct: 50

Set a workspace's priority-tier metadata to premium, standard, or bulk; an unset value uses standard. At the defaults, bulk requests begin receiving HTTP 429 at 50% observed load, standard requests at 80%, and premium requests at 100%. A rejection returns priority_throttled and increments gateway_priority_throttled_total{workspace,tier}.

The 1.8 admission check does not reserve its counter slot atomically. A simultaneous burst can briefly admit more work than the configured threshold, so treat these percentages as overload preferences rather than hard concurrency limits. See Route across providers for the complete configuration and dashboard.

Protect an upstream provider budget​

The Enterprise platform can observe published rate-limit headers for credential-based providers and shed a later request before sending it upstream. It is off by default, fails open when no fresh budget signal is available, and is separate from the client rate limiter on this page. Ollama and Bedrock do not use this tracker in 1.8.

dvara:
  llm-gateway:
    provider-rate-limit:
      enabled: true
      token-headroom-pct: 15

Enable priority admission as well if provider shedding must honor premium and bulk workspace tiers. With priority admission off, provider shedding treats every request as standard. See Recover from provider failures for thresholds, failover, and the terminal 429 response.

Monitor the selected backend​

All backends publish the same counter names, but their labels differ:

MetricPer-process token bucketShared-cache token bucketEmbedded sliding window
gateway_rate_limit_allowed_totalNo reason labelNo reason labelreason="request"
gateway_rate_limit_denied_total`reason="requesttoken"``reason="request
gateway_rate_limit_errors_totalNo reason labelStore errors have no reason; token-settlement errors can use reason="token"`reason="request

Do not write one dashboard query that assumes every series has a reason label. Query denials by reason, and aggregate allowed/error counters without requiring that label.

Next steps​