AI runtime enforcement benchmarks 2026: honest p50/p95/p99 overhead

Real numbers for a production runtime enforcement gateway measured against a direct-to-OpenAI baseline: +11 ms p50, +14 ms p95, +21 ms p99. Why microsecond claims are honest but unrepresentative, and what matters more.

This post publishes honest overhead numbers for the ThinkNEO enforcement gateway (gw.thinkneo.ai) measured against production traffic in July 2026. It also puts Bifrost's widely-cited "11 microseconds" number in context — because the comparison, done correctly, is more useful than either headline.

What was measured

The workload: 1,000 concurrent chat completion requests against openai/gpt-4o-mini, average prompt 400 tokens, average completion 250 tokens. All requests routed through a single-replica gateway deployment on a c5.xlarge equivalent. BYOK provider key. Enforcement rules: budget check, quota check, model allow-list, project scope. No content guardrails in this run (we benchmark those separately).

Comparison baseline: same traffic sent directly to api.openai.com without the gateway, from the same source, same time window.

Headline numbers (p50 / p95 / p99)

Path	p50 latency	p95 latency	p99 latency
Direct to OpenAI (baseline)	1,142 ms	2,894 ms	4,201 ms
Through ThinkNEO Gateway	1,153 ms	2,908 ms	4,222 ms
Overhead (delta)	+11 ms	+14 ms	+21 ms

Overhead is dominated by the enforcement decision (2–4 ms) plus the network hop into and out of the gateway (~7 ms local, more on cross-region). The rest is the provider call itself, which the gateway does not affect.

Why not microseconds?

Bifrost's 11 microsecond number is a famous claim from Maxim AI's benchmarking. It is a real number for a specific measurement: the additional time the Bifrost process spends per request when Bifrost is co-located with the target and the target is a synthetic responder. That is a useful lower bound for the gateway process itself.

It is not the number you get in production. In production the gateway must:

Terminate TLS from the client (~1–2 ms on hot connections, ~30–80 ms on cold).
Read and validate the request payload.
Resolve the caller's identity from the key.
Look up policy for the project (~1 ms with cache, ~15 ms cold).
Check budget state atomically (~1 ms in Redis, ~5 ms in Postgres).
Establish a client to the real provider (~5–20 ms cold, near-zero warm).
Forward the request, stream the response, and write the metered event.

Any honest gateway benchmark in production will land in the low tens of milliseconds of added latency on the p50 chat completion path. If a vendor tells you microseconds and shows a chart of overhead in isolation, they are showing you a valid but unrepresentative number.

What matters more than milliseconds

Once your baseline is 1,000+ ms (which it is for any real LLM call), 11–20 ms of overhead is a rounding error. The decisions that actually move the P50 for user-perceived latency are:

Model choice (haiku vs sonnet, mini vs 4o).
Prompt length.
Streaming vs non-streaming.
Provider region.
Whether you cache prompt prefixes.

The gateway does not fight you on any of these. It should be transparent to them.

What the gateway must not do

An enforcement gateway that does the following has failed its benchmark:

Adds more than ~5 ms of enforcement decision time at p95 for a cached policy.
Serializes streaming responses (kills first-token latency).
Blocks writes to the audit ledger on the request path (should be async).
Loses request-level attribution under load.
Silently retries on 5xx without a policy directive (surprises the caller).

Streaming first-token latency

For streaming responses, the number that matters is time-to-first-token (TTFT):

Path	p50 TTFT	p95 TTFT
Direct to OpenAI	318 ms	612 ms
Through ThinkNEO Gateway	329 ms	626 ms
Overhead	+11 ms	+14 ms

The gateway does not buffer the stream. Tokens flow through as they arrive from the provider. Enforcement completes before the first byte is forwarded, so the added latency lives entirely in the pre-stream phase.

What the audit ledger costs

Zero on the request path. The gateway writes the decision + metered event to a bounded buffer in-process and flushes asynchronously to Postgres. If the flusher backs up, the buffer is bounded and old entries are dropped to a spill file, then replayed. The invariant: the request path never waits for durability.

How to reproduce these numbers

The benchmark harness (a small Go tool) is available on request. The methodology is intentionally boring:

1. Warm up the gateway with 100 requests.
2. Ramp to 1,000 concurrent clients over 60 seconds.
3. Hold for 5 minutes.
4. Record TTFT and full latency per request.
5. Compare against a parallel run direct-to-provider.

All numbers in this post come from a single such run. We will re-run monthly and publish the delta.

Bottom line

An honest overhead budget for a production enforcement gateway is 10–20 ms at p50, 15–25 ms at p95, on top of the ~1,000 ms LLM call. Anything faster is a synthetic benchmark; anything slower is a design bug. Optimize for what the user perceives, not what the microbenchmark measures.