LLM cost control is the discipline of knowing what every LLM call costs, attributing that cost to the right owner, and applying policy to prevent unbounded spend. In 2026 it is the number-one operational concern for enterprise AI teams, ahead of latency and even model quality.
The reason is arithmetic: model quality has plateaued in per-token pricing, but consumption has not. When a single agent loop can execute 200 model calls to answer one user question, and 500 developers can each spawn agents, the annual budget disappears in weeks.
The three questions cost control has to answer
- What did this call cost? A honest number, derived from real tokens and versioned prices. Not a fabricated default.
- Who is responsible for it? Which team, which project, which end user. Not "shared infra."
- Can we stop the next one? Before the money is spent. Not after the invoice arrives.
Any tool that answers only the first is a reporting system. Any tool that answers the first two is FinOps for AI. Only a tool that answers all three is cost control.
Why fabricated cost is worse than no cost
In an earlier ThinkNEO release, a database trigger silently filled in default token counts (50 in, 20 out, $0.05 estimated) for any usage row where the provider did not report usage. It was well-intentioned. It was also wrong: it made every zero-cost cache hit and every failed request look like a real spend, and the dashboards told a story that was not true.
We retired the trigger. Every prior row was flagged with a corrections table. Aggregations were rewired to a view that respects the corrections. The rule now: meter what you actually saw. If the provider reported zero tokens, the cost is zero. If usage is missing, the row is marked missing — not backfilled.
That commitment is not abstract. It changed a fake $94.82 aggregate for one demo tenant to a real $19.28. It changed a fake $579,000 anomaly (arising from the same trigger interacting with a Bradesco simulator) into a real $0.038.
Fabricated cost is worse than no cost because it triggers wrong actions: budget freezes on tenants that did not overspend, and complacency on tenants that did.
The five levers of cost control
| Lever | Where it lives | Effect |
|---|---|---|
| Model choice | Application code | 10-100x cost delta between haiku and opus for equivalent tasks. |
| Prompt engineering | Application code | 2-5x reduction from prompt trimming, structured output, few-shot vs zero-shot. |
| Prompt caching | Provider or gateway | Up to 90% reduction on repeated system prompts. |
| Per-key/team quotas | Gateway | Bounds blast radius per owner. |
| Hard spend budgets | Gateway | Absolute upper bound per period. |
Model choice and prompt engineering are application-team concerns. Caching, quotas, and hard budgets are platform-team concerns. Both must be in place.
Why the gateway is the cheapest place to control cost
The gateway is the only component every LLM call passes through. That is the definition of a control point. Compare the alternatives:
- In-application checks: every application must implement them. Any application that skips the check is uncontrolled. This does not scale.
- Provider-side budgets: only the account-level limit, not per-team or per-project. The blast radius is the entire organization.
- Post-hoc alerting: the money is already spent when the alert fires.
The gateway sees every call by definition. Enforcing at the gateway is enforcing at the single choke point that already exists.
Why BYOK matters for cost transparency
In BYOK (bring-your-own-key) architectures, the tenant supplies their own OpenAI, Anthropic, Google, or Mistral key. The gateway routes through those keys. Tokens bill to the tenant's provider account. There is no reseller markup, no minimum spend, no bundled pricing.
This makes the cost number checkable: the tenant's OpenAI dashboard should match the gateway's metered cost within a small drift window. If it doesn't, one of them is wrong, and the tenant can tell which because they see both.
Non-BYOK gateways (reseller model) obscure the number by design: you pay the gateway, the gateway pays the provider, the delta is the vendor's margin. This is a fine business model but it is not cost transparency.
The failure modes of "cost control" done wrong
- Fabrication: filling in default token counts when the provider reports zero. Cure: retire the default-fill; audit prior rows; expose corrections.
- Aggregation without attribution: knowing the total spend but not which project. Cure: attribution at write time, indexed for query.
- Warning without enforcement: alerts that fire but do not block. Cure: hard budgets in enforce mode, with documented rollback.
- Enforcement without warning: a call gets blocked at 100% but nobody was warned at 75%. Cure: multi-threshold alerts wired to the same rule engine.
- Attribution without granularity: knowing which workspace but not which end user. Cure:
userfield or documented header per request.
What good looks like
An enterprise with mature LLM cost control has, at minimum:
- A single gateway all LLM traffic flows through.
- Per-project USD budgets, enforced fail-CLOSED.
- Per-project quotas, enforced fail-NEUTRAL (degrade to monitor).
- An audit ledger showing every metered call with real tokens and real prices.
- Alerts at 50%, 75%, 90% of budget.
- Provider dashboards that match the gateway's numbers.
- Attribution to end user when the application can supply it.
None of this is exotic. All of it is table stakes in 2026.
Related reading: LLM cost control vs FinOps · Hard-block vs monitor · BYOK cost accounting