I spent weeks optimizing an engine that runs in the browser, on your own GPU, with no server and nothing leaving the machine. I expected the hard part to be writing fast code. It wasn't. The hard part was realizing how often my own thermometer was lying to me.
That's what this is about. You don't need to be writing an engine for it to apply: it's for anyone who optimizes something and needs to know whether they actually improved it or just imagined they did.
The one I'm most embarrassed about is a noise meter. The idea was honest enough: run the control branch twice, compare the two runs against each other, and the difference tells you how much noise the benchmark harness has. I implemented it and got exactly 1.000. Every time. Zero noise, pristine environment, wonderful.
I was dividing the control branch by itself. The result couldn't have been anything but one.
It wasn't the only one. A correctness check divided by a quantity that in some cases was zero, and in those cases it passed without having compared anything at all. And the mirror image: a sanity assert — a condition that should always hold — had a formula that was impossible to satisfy, so it went red even when everything worked. I went hunting for a bug that didn't exist.
The conclusion is uncomfortable but clear: an instrument that can't fail is worse than no instrument at all. No instrument leaves you blind; an instrument that can't fail gives you false confidence, which is far more expensive.
The rule I took away: before you trust a new check, sabotage it on purpose. Feed it garbage, break the good branch, flip a sign. If it doesn't go red, the check is decorative and it can go.
Another painful one. I wanted to confirm an optimization had actually shipped, so I grabbed the file the site serves and searched for the name of the optimization inside it. There it was. Deployed, next.
It wasn't. The name showed up in an old copy that happened to mention it too. The presence of a word tells you nothing about which version sits behind it.
The correct check is to compare the hash — the fingerprint of the content — of the file you serve against the file you built: either they match or they don't, and there's no room for interpretation.
For weeks there was work that existed in the repo and never reached the product. Nobody lied. I just asked the wrong question.
Several times I wanted to know whether another program was using the GPU before I measured. And several times I counted how many foreign processes there were instead of listing them.
A number either reassures you or scares you, but you can't act on it. With the list you see who each one is. One of those times I went with the first process that showed up, declared the environment clean and kept measuring, ignoring a much bigger one that had been running for far longer. Everything I measured after that was worthless, and I didn't find out until much later.
This is the least intuitive effect of all. I repeated the same experiment with different amounts of GPU taken by another program, and the results looked nothing like each other.
The interesting part isn't that contention slows everything down — you'd expect that. The interesting part is the direction of the error: contention drags both branches toward the same bottleneck and compresses the difference between them. A real, large improvement can show up as a laughable gain.
So measuring with a noisy neighbor isn't "being conservative". It's being wrong in exactly the direction that makes you throw away good ideas.
I isolated an access pattern in a tiny benchmark to measure it cleanly, and it showed a huge improvement. I implemented it in the real kernel — the program that actually does the work — and what was left was a fraction of that.
It wasn't noise, and it took me a while to accept it. In the micro-benchmark that operation runs alone and its cost is fully exposed, so any saving shows up whole. In the real kernel it overlaps with everything else, and whatever you save gets eaten by work that was already happening in parallel.
A micro-benchmark isn't a prediction. It's a ceiling: it tells you at most how much you could gain, never how much you will gain.
I was measuring whether a change added overhead, and it looked like it did. Then I looked at how I was aggregating the runs: I was taking the median of two. With two runs, that median always picked the slower one.
For timings that's exactly backwards, because interference can only slow you down: nothing external makes your code run faster than it can. Everything above the floor is contamination. That's why the sensible estimator is the minimum, not the median and not the mean. Changing that turned an apparent overhead into essentially zero.
I'll end with the lesson that cost me the most. Something wouldn't compile, and I had at hand a reconstruction of what I believed was being compiled. I formed a hypothesis, refuted it, formed another. Eleven in a row, all of them reasoning about that hand-regenerated text.
What closed it was intercepting the exact text being compiled and dumping it verbatim. The problem was right there in plain sight, and it had never been in my reconstruction.
When something doesn't add up and you're three dead hypotheses deep, the problem isn't your reasoning anymore. It's that you're reasoning about a copy. Capture the real thing.
Part of a series on building a browser-native inference engine.