A 100-case test stopped two AI upgrades. One lost up to 12 points, the other never beat the baseline

Two upgrades to my book search felt like obvious wins. Re-rank the results. Ask the model for more versions of the question. I built both, and both made search worse on my test. I only know because the test said so before they shipped.

The product is WhatIsThatBook, which I build and run. A reader describes a book they half remember, and a model suggests up to five candidates. All numbers below are from this product, from runs I wrote up in May 2026, not from client work.

The short answer

Both upgrades were built as experiments and run against a fixed test before release. The first lowered the share of test cases with the right book first by 2 to 12 points. The second never beat the baseline. Neither went to production, and I estimate that each one saved a week of building.

The test

By "eval" I mean a fixed list of questions with known right answers. Mine has 100 cases, stored in the repo, so a pull request can review them and a CI gate can run them. About a third come from solved Reddit threads, a quarter from saved production searches, and I wrote the rest by hand. It took a weekend.

Upgrade 1: re-rank the results by plot similarity

Some candidates are real books that have nothing to do with what the reader described. The standard fix: fetch each candidate's plot summary, compare it with the reader's description using embeddings, and reorder or drop the weak matches. I built it and tried cutoffs from 0.55 to 0.70.

The share of test cases with the right book first dropped by 2 to 12 points, depending on the embedding model. Every setup was worse than the order the model had already given.

When the model was confident, its first pick was usually right. When it wasn't, the plot comparison was too weak to confirm or reject the pick. So the comparison added no new information.

Upgrade 2: ask the model for more versions of the question

The second idea: have the model also return three or four rewordings of the reader's description, search the catalog with each one, and merge the results. The hope was more books found. The cost: a few hundred extra tokens and three or four extra catalog searches each time.

The first run looked like a 7-point gain. Then I checked the baseline file. A model change a few days earlier had moved the baseline to 58% (right book in the top five), so the upgrade was 6 points behind, not 7 ahead.

The extra searches hit the catalog's rate limits. I added retries and a local cache, and ran both versions again. The baseline scored 58% on two different days. The upgrade scored between 51% and 57%. My own rule was +5 points to ship. It never got there.

The reason was the same as for the first upgrade: rewordings from the same model are the same guess in different words.

What I take from this for a client feature

• Write the pass rule before the test runs. Mine was +5 points.

• Run the baseline more than once, on different days, so you know how much the score moves by itself.

• Read the baseline file before you read the result.

• Before you build an upgrade, ask one question: when the new step disagrees with the model, is it more often right or wrong?

If your team has a list of AI improvements and no test to rank them, I build that test into your repo as part of my AI integration sprint, and it stays there after the handover.

The technical long version is on my personal site: The two upgrades my eval killed.