Why I Stopped Trusting Benchmarks and Started Timing My Own Workflow
Model benchmarks kept telling me one thing was better. My own stopwatch, on my own actual tasks, kept telling me something else. Only one of those numbers pays my invoices.
I used to switch tools the week a new benchmark came out. A model would top a coding leaderboard and I'd migrate a client workflow to it within days. I don't do that anymore, because I started actually timing my own work, and the two numbers kept disagreeing.
The benchmark measures a task I don't do
Most public benchmarks score a narrow, self-contained task: solve this isolated coding problem, answer this exam question, pass this specific eval. My actual work is almost never isolated. It's twelve small decisions about someone else's half-finished codebase, made in a specific order, most of which depend on context the benchmark never has to hold. A model that's excellent at the narrow task can still be slower for me in practice, because the narrow task was never the bottleneck — the context-gathering around it was.
What I started measuring instead
Nothing sophisticated. I timed, honestly, how long a real recurring task took start to finish with each tool — not the AI's response time, the whole task including the time I spent correcting it. A tool that answers instantly but needs three corrections loses to a tool that takes longer and needs zero. That number, total wall-clock time to a result I'd actually ship, turned out to correlate weakly with published benchmark rankings and strongly with something benchmarks don't measure at all: how well the tool held onto context across a long session without me re-explaining things.
The place benchmarks are genuinely useful
This isn't an argument against benchmarks — they're the right tool for one specific question: is this model capable of the task category at all. If I'm evaluating something genuinely new, a published eval score tells me whether it's even in the right ballpark before I spend an afternoon testing it myself. What they don't tell me is which of three capable tools will actually save me time on my work, because that answer depends on my workflow, not the model in isolation.
What changed in practice
I now keep a running, unscientific log — a task name, the tool, and a real time. Not a spreadsheet, just enough to catch myself before I make a switching decision based on a leaderboard screenshot instead of evidence from my own week. The number that's actually predictive of whether a tool change will help me isn't how it scored on a public benchmark last month. It's how long the last five things I actually needed done took, start to finish, corrections included.
