A language model running inside the browser comes with a non-negotiable entry fee: the whole thing has to come down before it says its first word. You pay for the first visit, and there's no trick around it. The second visit shouldn't cost anything. That's where the trouble starts.
The first thing anyone reaches for is Cache Storage, the store websites use to keep their own files around and work offline. It's the obvious answer: drop the model chunks in there and you're done.
It's no good. Cache Storage is evictable: when the browser needs space, it clears it out without asking and without telling anyone. For a stylesheet that's fine — it comes back down in a blink. For a large model it means the user pays for the full download again the next time they show up, and they have no idea why: it was fast yesterday and it isn't today.
The alternative is OPFS, the origin private file system: a small disk the browser hands your site, invisible from anywhere else, and not evicted casually. That's where the model has to live.
The detail that took us a while to work out is that on mobile there is no direct write from the main thread. The API that actually writes, the sync access handle, is only available inside a Worker, a separate thread from the one painting the page. Write from the main thread and on mobile nothing lands; and when nothing lands, the code falls back to the evictable store from before.
So: on exactly the device where the download is most expensive and the connection worst, the user was paying for the whole download on every visit. We moved the write into a Worker and the symptom disappeared.
The download goes by ranges: instead of asking for the file in one shot, you ask for pieces and write them as they arrive. It sounds like pointless complication until the wifi drops halfway through.
With a single whole-file download, a drop puts you back at the start. With ranges, you come back, look at what's already on disk, and ask only for the pieces that are missing. It's the difference between "try again from scratch" and "pick up where you left off", which at this file size is the difference between someone using it and someone closing the tab.
And here's the good part, which as always is the part where we were wrong.
For a while our test bench said the large model wasn't loading. It said so insistently, always the same way. And it was lying twice over.
The first lie came from a redirect. The site redirected to another domain, but the setting that picks which model to load was seeded on the old domain, before the jump. Result: the app booted with a different brain than the one we thought we were measuring. We were reading the results of some other experiment.
The second lie was dumber and more human: the machine went to sleep partway through the download. When the system suspends, the browser cuts the network and the request dies. With a small file you never see it. With a big one you land squarely inside that window, because the time it takes is exactly the time it takes you to go do something else.
Two believable causes, both false, identical symptom. The lesson wasn't fixing either one: it was that the measurement has to invalidate itself. If the machine went to sleep, the bench should say "this measurement isn't valid" instead of "it doesn't load". A bench that reports a failure that isn't there is worse than no bench at all, because it sends you off to fix things that already work.
Auditing the code, we found one of those holes that make you wince. Settings had a "free up space" button. It emptied Cache Storage. It never touched OPFS.
Since the model lives precisely in OPFS, the user saw their space taken up, pressed the button, and the space was still taken up. A function to delete the model from the private disk did exist — written, and with zero callers. Nobody called it from anywhere.
A huge file, orphaned, impossible to delete from the interface. The worst thing about this class of bug is that it almost never shows up as a ticket: the user doesn't report "your button doesn't work", they just decide your site eats their disk and never come back.
With the largest model we made the opposite call, and we think it's the right one.
That one brushes up against the quota ceiling the browser grants an origin. In practice the write was lost almost every time, near the end, and when it was lost it left the store poisoned for the next attempt: occupied, incomplete, useless. Caching it didn't just fail to help, it actively made the second visit worse.
So it isn't cached. It's fetched from the network by ranges every time and we eat the cost. It's an honest defeat, and it works better than a half-won victory.
Storing a model in the browser looks like a storage problem and turns out to be a problem of special cases: the right store, the right thread, resumption, a button that actually deletes, and knowing when to give up.
And above all: if your measuring instrument can lie to you, the first thing to fix is the instrument.
Part of a series on building a browser-native inference engine.