Just past midnight, I got the best numbers of the night.
Another late session, and I was not writing features. I was tuning retrieval, the part of Attrove that decides which messages actually matter when you ask it a question. Different strategies for planning a search, merging what came back, ranking it. Ten variations in nine hours, and after every change I re-ran the benchmark to see whether I had helped or hurt.
Version ten looked like the best of the lot. Average tokens: 589, down from 2,155. Runtime: 6.1 seconds, down from 21.
Roughly 70% leaner and faster. I was jazzed.
Then the eval score landed.
Zero.
What?
Fourteen of the twenty runs had crashed and returned nothing at all. The six that finished told me (politely and in complete sentences) that they had tried to check my messages but could not access them.
In engineering speak: a 401. Unauthorized.
The system was locked out of its own data and apologizing about it. Today's AI is friendly that way. It can be completely useless and extremely well mannered at the same time.
Of course it was fast. It barely did anything at all. Naturally the token count fell. It had almost nothing to say.
Failure is cheap, which is what makes it dangerous.
The averages were not wrong. They were something worse: mathematically correct and operationally meaningless. Fourteen dead runs dragged every efficiency number down and made a broken system look like the best work of the night. Among the runs that actually completed, the numbers came back at 1,964 tokens and 17.1 seconds. Basically unchanged from the version before it.
There was no breakthrough here. I had measured the absence of work and called it optimization.
The dashboard did not lie to me. It answered with the data it had available. It simply could not tell the difference between working and dead.
John Doerr has been banging the "measure what matters" drum for decades. That lesson usually gets pointed at running a business. It applies harder to systems that behave differently every time you run them.
Before I ask how fast it answered, I need to know whether it answered.
Before I ask what it cost, I need to know whether it did the work.
Before I compare two responses, I need to know whether either one could reach the evidence.
Otherwise you are declaring a patch of dirt structurally sound because you stomped on it a few times and it felt firm. Not a good indicator of a solid foundation.
Vibes don't hold up production systems. Or houses.
So I built an eval set for Attrove. Fifty pre-authored questions with their expected answers, difficulty labels, and notes on where the answer actually lives. All of it version controlled and locked per category in Git. Underneath sits a synthetic corpus of 372 artifacts: 192 Slack messages, 108 emails, 36 calendar events, 18 meeting transcripts, 18 notes. Twenty-two people across five project arcs, every one of them invented, so I can publish the numbers without publishing anyone's inbox.
The point of writing the answers first is to decide what "right" looks like before the model does its thing.
Did it flag the right person for raising the issue? Was the project status accurate? Did it find the deadline buried two thirds of the way into a meeting transcript? Did it notice when there was no answer at all?
That last one is where eval design gets hard.
Five of the fifty questions are deliberately unanswerable. The correct response is some version of "the data doesn't say." Models hate this. They are trained to be helpful, so hand them a thin picture and they will produce a confident answer just to appease the overlords.
On those five, refusing is the right answer.
Which is the trap. My broken system also refused.
One refusal means "I checked the evidence and it isn't there." The other means "I never reached the evidence." The sentences look nearly identical. The behavior underneath is opposite. An eval that only checks whether the system responded sensibly waves the broken one straight through.
A benchmark that only ever agrees with you isn't a benchmark.
Here is the part that stung. Every version ran head to head against a deliberately dumb baseline: raw provider APIs and an LLM, none of the retrieval work I had been grinding on. On the broken run, that baseline scored 73 out of 100. Mine scored zero. The simple thing kept answering while the sophisticated thing fell over.
Even on the best version of the night, Attrove won three of the five categories in that run and lost two. It was better at pulling commitments out of a thread, reconstructing why a decision got made, and spotting what was still unresolved. It was worse at stitching one answer across email and Slack, and worse at knowing when to say nothing. Structure helped in some places and got in the way in others, and I would not have known which was which without a scoreboard.
That is the whole point of building the thing. If it only confirms what you were already hoping, it’s not doing its job.
So: did my agents discover a world class improvement that made everything 70% more efficient in one shot? Or did I break an auth token?
Occam's razor remains undefeated.
Where does the benchmarking reflex come from? It doesn't fall far from the Apple tree.
I spent fourteen years there, most recently leading a team that productized wireless performance for iPhone, AirPods, Mac, et al. High volume, always on test software that helped decide which devices went to customers and which ones went back to get fixed. How do you quantify performance across roughly a million devices a day? Data, and a lot of it.
We measured everything. Boot time. Yield. Calibration time. Idle time between test steps. At that scale a few unnecessary seconds costs millions of dollars, so you get diligent fast. Establish a baseline. Do a control run. Change one thing. Run the experiment. Keep iterating until the evidence is strong enough to move the chains, then set a new baseline and do it again.
Manufacturing test at least gives you an honest failure. A crash. A timeout. A unit that won't calibrate. You know.
AI offers no such courtesy. It fails politely, fluently, and with total confidence.
The scariest number is the one that looks like a win.
Models change. Workflows change. The same prompt can behave differently from one run to the next. Without a baseline you cannot separate shipped and working from shipped and quietly worse, and quietly worse is the one that costs you a customer six weeks later.
Evals are not the glamorous part of building. I think of them as unit tests for the AI era. They don't make a demo sparkle, nobody asks to see them on a sales call, and they cost time and tokens to run.
Nearly every founder I talk to knows they need them. Nearly every one of them is behind.
Everyone has an AI enablement plan, a freshly vibe coded dashboard, and an ROI model somebody built in a spreadsheet. Ask about the baseline and the room gets quiet. Evals are this generation's DevOps: unglamorous, invisible when they work, and the only reason you can move fast without lying to yourself.
I built them for the side project too. Second Tell is a small thing I have been working on to help people tell better stories, and when I change the coaching analysis I have no way of knowing whether it actually got better without a fixed set of examples and a scoring rubric. It is in beta as a web app right now and feedback is the entire game, so if that sounds like your kind of thing, take it for a spin and tell me where it falls down.
I don't trust myself without the safety net. I am the person most motivated to believe my own improvement worked.
Evals are not a magic bullet either. Mine was good enough to catch a dead system and not good enough to catch it before I got excited about it. They cannot anticipate every strange way a non-deterministic system goes sideways. But they give you the one thing vibes can't: a baseline, and a way to say this got better, this got worse, and this never worked at all.
I still want that 589 token, 6.1 second run. But it needs to be earned.
Until the eval agrees, it isn't.
If you are running agents against anything that matters: what is your baseline, and when did you last re-run it?
Creators lock in holiday calendars 90 days out. Structure commissions, recruit creators, and optimize your creator affiliate strategy before the rush with Levanta's 90-Day Holiday Sprint. Get the Free Guide.
