Engineering notes

Building a browser-native LLM engine

Four notes on running a large language model inside a browser tab, on the visitor’s own GPU, with no server: how it loads, how it stays cached, what actually made it faster, and how our own benchmarks lied to us.

Running a 27B LLM in the browser: WebGPU inference from scratchReading a GGUF by ranges, one dequantizer per format, and five bugs that kept every test green. Caching >4GB in a web browser: OPFS, sync access handles, and why Cache Storage isn’t enoughCache Storage gets evicted. OPFS needs a Worker on mobile. And the button that freed space didn’t. The bottleneck wasn’t the weights: activation traffic in a WGSL matmulTiling the weights turned off the three routes that attacked the real traffic — and we had to retract a win. Benchmarks that can’t fail: when your own instruments lieA noise gate dividing the control by itself, a grep that matched its own comment, and eleven refuted hypotheses.

Written while building the inference engine behind the Elffuss apps.