A cheaper model looked fine on my 100-case test. I turned it off 14 minutes after it met real users
On my own product, a cheaper model looked fine on my test. It went live for 70% of users, and I turned it off 14 minutes later.
The product is WhatIsThatBook, which I build and run. A reader describes a book they half remember, a model returns up to five candidates, and the app checks them against a book catalog. All numbers below are from this product, not from client work.
The short answer
Test on real cases, with the exact output format your app sends.
Count the empty answers, not only the right ones.
Put the new model behind a flag for part of your users, and keep the way back one switch away.
The test looked fine
In September 2026, I screened seven cheaper models on my test set. My first runs had a gap: they sent a looser response format than the one production sends, and one provider rejected those requests before it generated anything. I fixed the test to send the exact schema the app sends.
With that fix, one model came close. On the 95 answerable cases of my 100-case test (the other 5 are too vague to name one book), it put the right book first 64 times. The model in production did it 66 times. The new model had zero failed calls, and the run of 100 cases cost $0.0202, against $0.0885 for the model in production.
It was slower: a median of 5.2 seconds per answer, against 2.8. I accepted that and put it behind a feature flag for 70% of users.
Fourteen minutes in production
Four of the first five production searches that reached the new model came back with no books. The answers were valid JSON. They just held a clarifying question instead of candidates.
I turned the flag off 14 minutes after I turned it on. Then I replayed ten real searches that the old model had answered with books. The new model returned books on three of them.
Later that night, I tried it again with the old model filling in empty answers. All 8 searches that reached the new model came back without candidates from it, and I turned it off again.
The flag is why this was one bad evening and not a bad week. Turning it off gave every user the old model again.
What the test missed
The evidence was already in the test report. On the 95 answerable cases, the new model returned no books 21 times. The old one did it twice. My summary showed accuracy scores only, and an accuracy score can't tell "guessed wrong" from "didn't guess".
For a user, those are very different. A wrong guess still gives them something to click. An empty page gives them nothing.
So the test harness I rebuilt after this counts empty answers and fails above a limit. Its default limit is 10%. On my real cases, the new model was at 22% and the old one at 2%.
The second switch: more books, slower answers
Two days later, I tested another model. It put the right book first on 66% of test cases, against 69% for the model in production, and it returned no books on 18 of 100. This time I checked that number first. A prompt written for this model brought empty answers on answerable cases down to zero and first-result accuracy up to 68%.
Then it went behind a flag for about 47 hours, from September 25 to 27, 2026. Counting first searches and leaving out refinements, the new model returned books on 688 of 770 searches (89%). The old one did it on 198 of 261 (76%).
It was not better on every measure. Its median model time was 7.4 seconds, against 4.4. Readers marked "none of these match" on 26% of its searches, against 22%. That feedback is voluntary, so it counts only the readers who clicked. Returning books was the goal of the change, so on September 27 I moved every search to the new model, with the old one as a fallback when the new one fails.
At list price, the new model cost more than the old one on the same 100 test cases, so this post makes no cost claim for that second switch. The cost detail is in the original post.
The steps, for your own model switch
A fixed set of real cases with known answers. Mine has 100.
The exact output format your production code sends.
A count of empty answers next to accuracy, with a limit that fails the test.
A spend cap on each test run. My 20-case screen ran under a $1 cap and used about four cents.
A feature flag for part of your users, and a rule for when to turn it off.
A fallback to the old model when the new one fails.
Moving an AI feature you already run to a different model fits my AI integration sprint: the test runs on your real cases first, and the change ships behind a flag.
The technical long version, with a test harness in TypeScript that you can run, is on my personal site: Benchmarking cheap LLMs before you switch models.