Show HN: Gemma 4 26B at 100 tok/s on a Mac, using the Neural Engine and 2GB RAM

After checking out Turbo Fieldfare (a very interesting project that streamed experts from SSD on Apple Silicon Macs to run Gemma 4 26B with very little RAM usage), I tried to make this project much faster and succeeded!

This project gets up to 100+ tokens/s inference on Gemma 4 26B (Turbo Fieldfare runs at 20-30 on similar hardware) and doesn't have insanely slow prefill times due to the ANE helping (I had to reverse engineer Apple's APIs to get this working).

I also implemented speculative decoding, and while Turbo steered away from using mmap, this repo utilizes it so the OS file cache serves as the expert cache.

---

Comments URL: https://news.ycombinator.com/item?id=50174998

Points: 1

# Comments: 0