Why I chose on-device inference for a writing assistant

When I built DraftKey, the biggest decision was running the model locally instead of calling a cloud API. Here are the tradeoffs.

The upside: people's text never leaves their Mac, it works offline, and there are no per-request API costs.

The downside: the app needs a model download of about 1.1 GB, at least 8 GB of RAM, and an Apple Silicon Mac, and a small model has less raw capability than the frontier ones.

For short rewrites and autocomplete, a small local model turned out to be good enough, and privacy ended up being the feature people notice most. Curious whether others building local-AI apps landed in the same place. getdraftkey.com