I built a free image-to-text tool where the image never leaves the browser

Most "image to text" sites work the same way: you upload a screenshot, their server runs OCR, you get text back. That is fine for a meme. It is not fine for a bank letter, a contract or a chat with someone's phone number in it, and most people never check where the picture goes.

So we built one that runs entirely on the visitor's device. No upload, no account, no server doing the reading. It is live at tofutoon.com/tools/image-to-text, free. This post is about what that took, what worked and what still doesn't.

The constraint: nothing leaves the tab
"Client-side" is easy to claim and easy to break. One analytics script, one library that quietly fetches a model from a CDN, and your "nothing is uploaded" line becomes a hope rather than a fact.

We set three rules before writing any code:

Every file the OCR needs is served from our own domain. That means the worker, the WebAssembly runtime and the models. OCR libraries love to pull their models from a public CDN by default, so each one had to be pointed at our own paths.
The browser enforces it, not us. The tool pages send a Content-Security-Policy that only allows network requests to our origin and the analytics hosts the site already uses. It started in report-only mode, so we could see violations before blocking anything. A compromised dependency could not post an image somewhere else without the browser complaining.
Nothing heavy loads until you ask for it. The OCR engine is imported only when you press the button. A language model is fetched only when you pick that language.
The third rule matters more than it sounds. Reader files are big: about 18 MB for English and most languages, 25 MB for Chinese and Japanese. Shipping that to everyone who opens the page to read the FAQ would be rude. After the first run, the browser keeps the files, so the second visit starts in a moment.

One engine wasn't enough
We started with Tesseract.js, the obvious choice: mature, well documented, WebAssembly, no server. On clean screenshots it is excellent.

Then we built a proper test set: 90 images covering posters, comic panels, tilted JPEG photos and screenshots in 20 languages. Tesseract got 87% of words right overall. The averages hid the real story:

posters with outlined titles: 76%;
comic panels with speech bubbles: 69%.
That is where people actually need OCR, because you can't select text on a poster.

So we added PaddleOCR's PP-OCRv5 mobile models, running on ONNX Runtime's WebAssembly build inside a worker. On the same 90 images:

Image type Tesseract With PaddleOCR
All 90 images 87.0% 96.8%
Posters 76% 94%
Comic panels 69% 94%
On a separate set of 15 hard English images, 99.6% of characters came out exactly right, capitals and punctuation included, against 94.3% before.

We didn't drop Tesseract, though. None of the Paddle models we tried handles Vietnamese tone marks, and Paddle read clean Arabic worse than Tesseract did. So the tool picks the engine by language:

PaddleOCR for 18 languages;
Tesseract for Vietnamese and Arabic.
If Paddle can't start on a device, Tesseract takes over. Two engines are more work to maintain, but each language now runs on the engine that reads it best.

Small things that made a big difference
Enlarging small screenshots. A screenshot of 14 px desktop text is tiny for an OCR model. Scaling images under 2,000 px up 2× before reading took that case from 88% to 100% of words. No model change, just bigger letters.

Reading order. An OCR engine gives you boxes, not a document. Two columns, a receipt, a chat with bubbles on both sides, right-to-left text: if you just sort the boxes top to bottom, the output reads like a ransom note. We wrote our own ordering step on top of the boxes. Getting it right was more work than plugging in either engine.

Choosing the language matters. Reading a Vietnamese screenshot with the English setting got about a quarter of the characters wrong, mostly the accented ones. So the UI asks for the language up front instead of guessing, and has an "Also English words" option for mixed text.

Translation without a server. Chrome and Edge on desktop now ship a built-in Translator API that runs on the device. We use it to translate the extracted text, so the "nothing leaves the tab" rule still holds. Firefox, Safari and phones don't have it yet, so there the Translate button is off. We chose not to fall back to an online translator, because that would break the promise the page makes.

What still doesn't work
Handwriting. These models are trained on printed text.
Vertical Japanese and Chinese. Top-to-bottom text isn't read.
Decorative lettering in Vietnamese and Arabic, the two languages on Tesseract.
Whole comic pages. A single bubble reads well, but the bubbles of a full page don't always come out in story order. That needs knowing the panel layout, which a general OCR tool doesn't.
Phones. The page works on a phone, but we haven't timed it on real devices yet. On a recent Mac it takes about 2 seconds per image once the reader is loaded.
What I'd tell anyone building something similar
Build the test set before choosing the engine. Our first engine looked great on clean screenshots and poor on posters and comics. Scores per image type told us far more than any average.
Make the privacy claim checkable. A CSP header turns "we don't upload your files" from a sentence into something the browser enforces, and anyone can open DevTools and check it.
Lazy-load everything heavy, and say how big it is. People accept an 18 MB first download when the page tells them up front and it only happens once.
If you try it on something nasty (a dark-mode screenshot, a wrinkled receipt, a meme with outlined text), I'd like to hear what it got wrong. The tool is free at tofutoon.com/tools/image-to-text, along with a few other in-browser converters we've built for ebooks and comics.