How to implement AI runtime enforcement (BYOK reference architecture)

This is a build guide for engineers deploying an AI runtime enforcement plane in front of a BYOK (bring-your-own-key) LLM stack. The specific reference is ThinkNEO Gateway (gw.thinkneo.ai), but the pattern generalizes.

By the end of this post you will have a working request path where every call is enforced against your policies, cost is metered against real tokens, and no code change beyond a base-URL swap is required in your application.

The pattern in one paragraph

Your application uses the OpenAI SDK (or Anthropic SDK) as it always has. You point the SDK's base URL at the enforcement gateway. You authenticate to the gateway with a project-scoped API key you mint yourself. The gateway resolves policy, checks budget, forwards the call to the provider using the tenant's own BYOK provider key, streams the response back, and writes a metered event to the audit ledger. No code change beyond the base URL.

Step 1: mint a project-scoped key

In the ThinkNEO dashboard, create a project. Enable at least one model (for example, openai/gpt-4o-mini or anthropic/claude-haiku-4.5). Mint an API key scoped to that project. The key format is tnk_....

curl -X POST https://thinkneo.ai/api/v1/tenant/projects/my-project/keys \
     -H "Authorization: Bearer $TENANT_ADMIN_TOKEN" \
     -H "Content-Type: application/json" \
     -d '{"name": "backend-service", "scope": "project"}'

The response includes the raw key exactly once. Store it in your secret manager. From this point forward, the raw key is not retrievable — you can rotate but not recover.

Step 2: register BYOK provider keys

In the same dashboard, register your OpenAI and Anthropic (and Google, Mistral, etc.) provider keys. These are the keys that will be used to actually call the providers. The gateway stores them encrypted and reveals them only to itself at request time.

Real BYOK means the tokens are billed to you, not to the gateway operator. ThinkNEO does not resell tokens. Your OpenAI dashboard will show the usage. That is the point.

Step 3: point the SDK at the gateway

OpenAI SDK (Python):

from openai import OpenAI

client = OpenAI(
    base_url="https://gw.thinkneo.ai/v1",
    api_key="tnk_...",   # your ThinkNEO project key
)

response = client.chat.completions.create(
    model="openai/gpt-4o-mini",
    messages=[{"role": "user", "content": "Hello"}],
)

Anthropic SDK (Python):

from anthropic import Anthropic

client = Anthropic(
    base_url="https://gw.thinkneo.ai/anthropic",
    api_key="tnk_...",
)

response = client.messages.create(
    model="anthropic/claude-haiku-4.5",
    max_tokens=256,
    messages=[{"role": "user", "content": "Hello"}],
)

Note the model id: <provider>/<model>. The gateway rejects unqualified model names to prevent silent provider swaps. If you later need to run the same model against multiple provider keys, use the @key suffix (openai/gpt-4o-mini@primary, openai/gpt-4o-mini@fallback) to distinguish them in metering.

Step 4: set a budget

In the dashboard (or via admin API), set a monthly USD budget on the project. Choose the enforcement mode:

• monitor — the call proceeds; a warning event is emitted when the budget is breached.
• enforce — the call is denied with HTTP 402 when the budget is spent. This is the fail-CLOSED path for cost.

Trial workspaces default to enforce at end of trial. Paid workspaces default to enforce from day one. The choice is explicit and versioned; every change to enforcement mode is a row in the audit ledger.

Step 5: verify enforcement

The fastest way to prove enforcement is live: set the budget to 0.01 USD and make a single call. The first call succeeds; the second is denied with HTTP 402 and a body of the form:

{
  "error": {
    "type": "budget_hard_block",
    "code": 402,
    "message": "Project budget exhausted for period 2026-07.",
    "rule_id": "budget.hard_block.v1",
    "trace_id": "tnk_ev_..."
  }
}

Take the trace_id, open the audit ledger in the dashboard, and you will see the identical rule fire with full context: which project, which key, which model, which model call, which prior cost accumulated, which threshold triggered.

Step 6: metering (arithmetic, no fabrication)

Cost accounting on the gateway is deterministic: tokens_in × price_in + tokens_out × price_out, where prices come from the versioned pricing catalog. There is no rounding up, no minimum-charge, no fabricated tokens. If the provider returns usage.total_tokens = 0, the cost is 0.

This design choice was not free — an earlier release had a database trigger that filled in default token counts when the provider did not report them, and that trigger was retired after it produced a run of implausible USD-per-request numbers. Meter what you actually saw. If you didn't see it, don't invent it.

Failure modes you should exercise before shipping

1. Gateway unreachable → SDK exception; treat like any other provider outage.
2. Budget exhausted → HTTP 402 with typed error. Your application should downgrade gracefully (e.g., surface an in-app message).
3. Quota exhausted → HTTP 429 with Retry-After. Standard back-off.
4. Provider outage → gateway propagates the provider's HTTP status. No masking.
5. Policy rule change mid-flight → the next request uses the new rule; in-flight requests complete under the old rule. No mid-request re-eval.

What you have when you're done

• Every LLM call from every service in your organization goes through one enforcement point.
• Every call is metered honestly and attributed to a project, workspace, and (optionally) end user.
• Budgets are enforced fail-CLOSED; quotas degrade to monitor if the control plane hiccups.
• The audit ledger has one row per decision, immutable, exportable to your SIEM.
• Your provider dashboards (OpenAI, Anthropic) show the same usage the gateway shows — no drift.

That is a runtime enforcement plane. It is deployable in an afternoon. The hard part is not the software — it is deciding, as an organization, that "every call, enforced" is a business commitment you are willing to make.

Related reading: What is runtime enforcement · BYOK cost accounting · Pricing