How We Cut LLM Costs by 70%

The starting position

A client ran a document analysis product processing roughly 400,000 requests a month. Every request went to the largest available model with the full document in context. The monthly bill was about $18,000 and rising with usage, which made the unit economics of their pricing increasingly uncomfortable.

Nothing about the implementation was incompetent — it was built quickly to validate the product, and it worked. It simply had no cost architecture, because at prototype volume there was no reason to build one.

Lever one — model routing

Not every request needs the strongest model. Their traffic broke down into classification and routing steps, straightforward extraction, and genuinely complex analysis. Only the third category benefited measurably from the largest model.

We added a routing layer that classifies each request and sends it to the appropriate tier. The classifier itself is a small, fast model costing a fraction of a cent. Roughly 60% of traffic moved to cheaper models with no measurable quality change against the evaluation set.

Measured saving: approximately 38% of total spend

Lever two — semantic caching

Analysis of their query log showed substantial near-duplication: different users asking materially the same question about the same document, and the same user re-running a query after a minor edit.

A semantic cache embeds each incoming query and returns a stored response when a sufficiently similar query has already been answered against the same document version. The similarity threshold matters — set it too loose and users get subtly wrong answers, so we tuned it conservatively against the golden set. Cache hit rate settled around 22%.

Measured saving: approximately 18% of remaining spend

Lever three — prompt compression

Every request was passing the entire document, frequently 40 pages, when the relevant content was two or three sections. This is the most common source of avoidable token spend we encounter.

Introducing a retrieval step before generation — selecting relevant passages rather than passing everything — cut average input tokens by about 71%. Answer quality improved slightly, which is a consistent finding: models reason better over focused context than over long documents where the relevant passage is buried.

Measured saving: approximately 24% of remaining spend

Lever four — structured output

Outputs were free-form prose that downstream code parsed with regular expressions. Parsing failures triggered retries, and retries are pure duplicated cost.

Moving to enforced JSON schema output eliminated the failure mode. Retry rate fell from about 9% to under 1%, and as a secondary benefit the downstream code became considerably simpler.

Measured saving: approximately 7% of remaining spend

The result and the caveat

Combined, monthly spend fell from roughly $18,000 to $5,200 — about 71% — with evaluation scores holding steady and latency improving, since smaller models and shorter prompts both return faster.

The caveat worth stating plainly: none of this is safe without an evaluation harness. Every one of these levers trades a small amount of capability for cost, and without a golden set to score against you are guessing about whether quality held. Build the evaluation first; optimise second.

Frequently asked questions

Does routing to cheaper models hurt quality?

Not if it is measured. Route based on evaluation scores per request type rather than intuition. Some categories show no measurable difference between a small model and a large one; others degrade immediately. The evaluation set tells you which is which.

Is semantic caching risky?

It is if the similarity threshold is set loosely, because near-miss matches return confidently wrong answers. Tune conservatively against a golden set, scope cache keys to document version, and accept a lower hit rate in exchange for correctness.

Where should we start?

Prompt compression, almost always. Most systems pass far more context than needed, it is the easiest lever to implement, and it frequently improves answer quality as well as cost. Routing is the next-largest win but requires an evaluation set first.