The invoice says your company spent $120,000 on AI last quarter. The uncomfortable question isn’t “is that too much?” — it’s “what did you actually buy?”. Learn more about the math of AI waste.

The honest answer, in most organisations, is that a large slice of that money paid for answers a much cheaper system could have produced. My rough read of the numbers we keep seeing in the field: somewhere between 40% and 70% of a typical generative-AI budget is burned on top-tier models running tasks that were never complex enough to need them. This isn’t a technology problem. It’s an efficiency problem — and efficiency is the only thing that makes AI economically scalable at all.

 

The silent waste

Here’s what waste actually looks like in practice. A support team routes every ticket through a flagship frontier model, because that’s the one the pilot ran on and nobody changed the setting. A developer sets “the big model” as the default in their tool and leaves it there. An internal agent, built to summarise a weekly report, calls an expensive API thousands of times a day, for a task a far smaller model handles flawlessly.

None of this is deliberate. It’s just gravity: once a model is in place, teams default to it for everything. The result is a single, expensive path for work that ranges from trivial to genuinely hard. You end up paying top price for the bottom 80% of your workload.

 

The feeling of paying for things twice

Here’s what I’ve learned watching teams that got this right: the ones spending sensibly didn’t find better models, they got sharper about which model handles which job, and they stopped treating every call to an API as if it were the first time the question had ever been asked. They finally understood the math of AI waste.
None of this requires betting on some unproven breakthrough. It’s engineering discipline applied to an expense that most companies are still treating as a mystery. When you apply it, the effect is not incremental. Cutting cost per call by 60–80%, removing entire categories of duplicated spend, and slashing the cost of context, this is where the “millions saved” figure actually comes from.

 

The good news: this is fixable with discipline, not new technology

Here’s what I’ve learned watching teams that got this right: the ones spending sensibly didn’t find better models — they got sharper about which model handles which job, and they stopped treating every call to an API as if it were the first time the question had ever been asked.

None of this requires betting on some unproven breakthrough. It’s engineering discipline applied to an expense that most companies are still treating as a mystery. When you apply it, the effect is not incremental. Cutting cost per call by 60–80%, removing entire categories of duplicated spend, and slashing the cost of context, this is where the “millions saved” figure actually comes from.

 

How to stop overpaying: three levers that move the number:

 

1 – Route: stop using one model for everything (dynamic model routing)

The single-model trap is the most common and most expensive error in AI today. One default model, usually the most capable and most costly, handles every request — from “summarise this paragraph” to “write this complex report.”

The fix is a gateway or router that sits in front of your model calls and decides, per task, which model is good enough. A routing layer (the kind of thing LiteLLM or Portkey provide) sends simple tasks to compact, cheap models — think GPT-4o-mini, Claude Haiku, a small open model — and reserves the frontier model for the minority of requests that genuinely need it.

The effect is predictable and large: 60% to 80% lower cost per call, with no measurable drop in output quality for the routed tasks. The frontier model becomes the exception, not the rule — and that
single inversion is where the biggest savings live.

 

2 – Don’t re-pay for answers you already have (semantic caching)

A surprising share of the “new” questions your systems answer every day are essentially the same question phrased slightly differently. In many workflows, around 30% of calls are near-duplicates. Paying full price for each one is like re-buying a report every time someone rephrases the question.

Semantic caching fixes this. Store responses in a vector index, Redis vector search and GPTCache are the tools people reach for — and when a new request lands within ~95% semantic proximity of one already answered, serve it from the cache instead of calling the model. Cost drops toward zero for those calls, and latency drops below 50ms, often faster than the model would have been anyway.

 

3 – Stop paying to re-send the same instruction (prompt pruning + prompt caching)

Long system prompts are quietly one of the biggest token sinks in modern AI. Teams bolt on instruction after instruction until a prompt runs to ~2,000 tokens — and that whole block is billed again on every single call, dwarfing the cost of the actual answer.

Two complementary fixes. First, prune: audit your input-to-output ratio and cut every prompt down to what is strictly functional. Second, use native prompt caching — Anthropic and OpenAI both let you cache a stable context prefix so you only pay for it once, not per call. Combined, teams routinely cut between 50% and 90% of their context cost.

 

The pattern: treat AI as infrastructure to manage, not magic to pay for

Step back and the theme is clear. This isn’t about any single clever technique. It’s a shift in mindset: stop treating “AI spend” as an opaque cost centre and start treating it as infrastructure, something you meter, route, cache and tune the way you would any other critical system.
The companies that win with AI at scale aren’t the ones using the flashiest models. They’re the ones who got the economics right first, and scaled from there. Capability was never the bottleneck. Cost discipline is. Then you will understand the math of AI waste.