About 47% lower LLM spend for one evening, with the same model
My first idea for a lower model bill was a cheaper model. On my own product, the cheaper models usually lost answers. The two changes that worked kept the model I already had.
The product is WhatIsThatBook, which I build and run. People describe a book they half remember, a model suggests candidates, and the app checks each one against a book catalog. A cheaper call is only useful if the reader still finds their book. All numbers below are from this product, not from client work.
The short answer
A shorter prompt. Average input went from about 2,683 tokens to 901. At the prices used in the test, estimated generation cost fell 33.5% per hundred calls, and first-result accuracy stayed at 69%.
A cheaper routing tier for the same model. Two runs over the same 171 test queries measured roughly 53% to 58% lower cost on the main model call. On the evening the change merged, spend was about 47% lower than the evening before, with almost the same number of identification outcomes (209 against 206). Slightly fewer searches returned results.
Most cheaper models lost answers. So did a cheap-first router and a search without the model.
Other models found fewer books
In August 2026, I compared four models on the same 100 test cases. The model in production put the right book first on 69 of them. The two cheaper models got 56 and 47. The third cost more per call and still got only 64.
Getting a model to return valid JSON was sometimes a job in itself. In this comparison, all four models completed every case. But a model can return clean JSON with five real titles and still miss the one book the reader wants.
Two other ideas promised savings:
Call the expensive model only when the cheap one looks unsure. On the cases I used to design the rule, it promised roughly 69% savings. On a separate set of 50 searches, the cheap model missed 14 that the production model solved. My limit of one expensive call for every five searches allowed 10 calls for those 14 misses, so even a perfect rule could not recover all of them.
Skip the model and search the catalog directly. On a production-like sample, this cut the cost per search by 99.96%. It also dropped the share of cases with the right book in the top five from 26% to 6%.
What worked: a shorter prompt
In August 2026, I cut the prompt of the model already in production. Average input went from about 2,683 tokens to 901. Output grew a little, from 361 to 416 tokens. At the prices used in that test, estimated generation cost fell from $0.1682 to $0.1118 per hundred calls, a 33.5% reduction. Soon after, the model's discount ended and each token cost more. The shorter prompt helped, but it did not protect me from that.
First-result accuracy stayed at 69%, and the share with the right book in the top five moved from 71% to 73%. With 100 cases per prompt, that doesn't prove an improvement. It was enough to keep the shorter version.
What also worked: a cheaper routing tier for the same model
My model calls go through a routing service. It offered a cheaper tier for the same model. I kept the production prompt and the model, and only changed the routing. If a call still returned nothing usable, the app made one more attempt through standard routing. A valid "no matches" answer did not trigger that extra call.
Two runs over the same 171 test queries measured roughly 53% to 58% lower cost on the main model call, before retries and fallbacks. Quality was close, with a small loss: across both runs, on the answerable cases, the cheaper tier found the right book in the top five 231 times, against 234 for standard routing. That is not proof of equal quality.
The change merged on September 7, 2026. That evening, spend was about 47% lower than the evening before, with almost the same number of identification outcomes (209 against 206). Slightly fewer searches returned results, and I don't know yet if those results were right. The median response time went from 7.57 seconds to 4.18 seconds.
One evening is not a month. Different people searched for different books on each evening, and I haven't verified exactly when the deploy reached users. The percentages in this post also come from different tests, so they don't add up.
On September 27, 2026, I moved every search to a different model, so that more searches return books. How I tested that switch.
The saving I didn't keep
One test showed about 70% savings with an even shorter prompt on the cheaper tier. Scored on the titles and authors the model wrote, the shorter prompt won 114 to 107 on 171 queries. Scored on the books the catalog actually found, it lost 122 to 123. I kept the old prompt.
That is the main lesson for a team that wants a lower bill. The score that counts is the one at the end of the full pipeline, where the user sees the result.
If you want the same for your AI feature
The method is not specific to books. Before a cost change reaches your users, I want three things in place:
A fixed set of real inputs with known right answers, run before and after the change.
The score taken at the end of the full pipeline, not on the raw model output.
The change behind a flag, with the old path one switch away.
A cheaper version of a feature you already run is one of the example scopes of my AI integration sprint. Cost and quality are measured before and after, on your real cases.
The technical long version, with every experiment and its notes, is on my personal site: What I learned trying to reduce my LLM costs.