I trained a small language model in TypeScript. Here is the path that worked.

2026-11-17来源 dev.to TypeScriptEnglish

TypeScript ML examples often stop at calling an existing model. I wanted to check the other path: define a decoder, train it on text, evaluate it, and save its state, all from a TypeScript application.

I used Affon's decoder app. It defines embeddings, causal attention, and six decoder layers. The bundled WikiText-2 text becomes 256-token windows. The training workflow evaluates batches, checks validation loss, and writes a checkpoint.

Train with a smaller tokenizer

With Affon 0.4.2 installed on macOS arm64, this runs on Metal from a source checkout:

git clone https://github.com/neucine/affon.git cd affon mkdir -p apps/decoder-lm/artifacts/wikitext-8k RUNTIME_PACKAGE_PATH="$PWD/packages" \ AFFON_TRAIN_CONFIG=apps/decoder-lm/configs/train-decoder-lm-wikitext-8k.config.json \ affon run apps/decoder-lm/train.ts 

This configuration uses an 8,192-token tokenizer trained on the WikiText-2 training split. It runs a six-layer decoder for 256 training batches, then evaluates up to 32 validation batches. In the saved run, validation loss moved from 9.070 to 6.579, final training loss was 7.093, and the app saved first-run-epoch-1. The generated sample still repeated words and symbols. This capped run shows learning on those batches; it does not establish useful generated text. Loss from a different tokenizer is not directly comparable.

Why the model definition matters

Affon calls the computation a Program. The decoder Program names its input shape and parameters, including the token embedding table and the layers that use it. A training Program is derived from the model, a cross-entropy loss, and an optimizer. The app compiles those Programs for a Session and keeps model state explicit.

That structure gives me something to inspect before execution: which operations and tensor shapes the model asks for, and where parameters belong in the nested decoder. It also lets the same model definition serve training and inference. The model source and workflow show the full path.

The complete walkthrough covers installation, output files, and the training configuration. If you work on TypeScript ML, I would value feedback on the Program boundary: what would you need to inspect before trusting a larger training run?

原文:https://www.a8q9.cn/en/article/i-trained-a-small-language-model-in-typescript-here-is-the-path-that-worked-c5e229.html

相关文章