I Built a Tiny Japanese GPT from Scratch in PyTorch and Tried a Toy 'RSI' Loop: Here's What It Does and Doesn't Show
Introduction
I'm a second-year student at ZEN University studying intelligent informatics (online), and I build AI agents as a personal project. I have no work experience in machine learning.
I use LLMs through APIs and agents every day, but I wanted to see what happens inside one with my own hands. So I wrote a small Japanese GPT using only basic PyTorch operations: no pretrained weights, no high-level model libraries. Then I tried a toy version of "RSI" (recursive self-improvement): the model generates several answers, grades them itself, and retrains on the best ones.
Up front: this is a tiny educational model trained on a few KB of text. Don't expect it to be useful, and please don't read the RSI numbers as "the model got smarter." I'll explain why below. This is not AGI or anything close to it.
What I built
A character-level, decoder-only Transformer with about 0.9M parameters (the pretraining version has 898,425, which I also checked by hand from the layer sizes). It has three stages:
-
mini_gpt.py: character tokenizer + Transformer, pretrained on short excerpts of modern Japanese literature to continue text. -
chat_gpt.py: the same architecture, fine-tuned onユーザー: ...\nAI: ...◁dialogues so it behaves like a chatbot. -
rsi_gpt.py: a best-of-N self-training loop.
There is also a local web UI (web_ui.py, standard-library http.server only) that shows the top-5 probabilities for the first generated character and lets you edit the dialogue data and retrain in the browser.
How it works
1. A minimal Transformer
Hyperparameters in mini_gpt.py:
BLOCK_SIZE = 64 # max context length N_EMBD = 128 # embedding dimension N_HEAD = 4 # attention heads N_LAYER = 4 # Transformer blocks Causal self-attention masks out future characters with a lower-triangular mask before the softmax:
att = (q @ k.transpose(-2, -1)) * (1.0 / math.sqrt(head_dim)) att = att.masked_fill(self.mask[:, :, :T, :T] == 0, float("-inf")) att = F.softmax(att, dim=-1) Each block is Pre-LayerNorm with two residual connections:
def forward(self, x): x = x + self.attn(self.ln1(x)) # residual + attention x = x + self.mlp(self.ln2(x)) # residual + feed-forward return x The tokenizer just numbers the characters that appear in the training text:
chars = sorted(list(set(text))) stoi = {ch: i for i, ch in enumerate(chars)} 2. Turning it into a chatbot
I didn't change the architecture. Two things changed:
- The input is formatted as
"ユーザー: {question}\nAI: ". - Every AI answer in the training data ends with a stop character
◁, and generation stops when it is sampled.
if token_id == stop_token_id: break # stop when the end marker appears Without the stop character the model keeps writing the next ユーザー: ... turn by itself. I call that role leakage below.
3. The RSI loop
This is a small-scale best-of-N rejection sampling loop, in the spirit of STaR / ReST:
- For each of 7 prompts, generate N=6 candidate answers.
- Score each one out of 100 with a rule-based function.
- If the best candidate scores 75 or more, keep that (prompt, answer) pair as synthetic data.
- Add the synthetic data to the dialogue data and retrain for 350 steps. Repeat for 3 generations.
The score is the sum of four parts (evaluate_candidate):
| Part | Points | Meaning |
|---|---|---|
| Completion | 25 | ended naturally with ◁ |
| Format | 25 | no "ユーザー"/"AI:" leakage, no newline, no 4+ repeated characters, at least 12 characters |
| Relevance | up to 30 | contains keywords I hand-picked per prompt (15 points each) |
| Self-confidence | up to 20 | the model's own average probability of the characters it sampled, times 20 |
matched_kw = [kw for kw in prompt_info["keywords"] if kw in reply] kw_score = min(30.0, len(matched_kw) * 15.0) ... conf_score = round(candidate["avg_confidence"] * 20.0, 1) The exploration temperature is lowered each generation: max(0.45, 0.85 - (gen - 1) * 0.12), i.e. 0.85, 0.73, 0.61.
Results
Sources: PAPER.md and rsi_history.json. I recomputed the RSI table from rsi_history.json.
Pretraining (1,678 characters of Japanese literature excerpts, 377 distinct characters, 800 steps): loss went from 6.1098 at step 1 to 0.0683 at step 800. Prompted with "吾輩は", it produces the opening of I Am a Cat.
Dialogue fine-tuning (27 Q&A pairs, 1,575 characters, 1,000 steps): loss went from 5.5833 at step 1 to 0.0387 at step 1000.
RSI (7 prompts, N=6, threshold 75, 350 steps per generation):
| Generation | Temperature | Avg best score | Avg worst score | Prompts with worst < 75 | Loss after retraining |
|---|---|---|---|---|---|
| Gen 1 | 0.85 | 97.56 | 80.77 | 4/7 | 0.0635 |
| Gen 2 | 0.73 | 97.84 | 97.80 | 0/7 | 0.0490 |
| Gen 3 | 0.61 | 97.84 | 97.80 | 0/7 | 0.0639 |
Gen 1 failures looked like this:
- For "こんにちは!元気?" ("Hello! How are you?"), one candidate answered "こんばんは!…" ("Good evening!…") (69.8).
- For "LLMとAttentionについて教えて", one candidate wrote "Attentionって何?\nAI: …" inside its own answer, i.e. role leakage (74.9).
The average worst-case score went from 80.77 to 97.80, and prompts with a failing candidate went from 4/7 to 0/7. But please don't read this as the model getting smarter. Here is why.
Limitations (please read this part)
- Very little data. Pretraining text is 1,678 characters (about 5 KB); the dialogue set is 27 pairs. Losses of 0.04 to 0.07 mostly mean the model has memorized its data. I never held out a validation set, so there is no generalization number anywhere in this project.
- The evaluation prompts are the training prompts. The 7 accepted synthetic pairs are answers to exactly the 7 prompts used for scoring (they sit at the end of
chat_data.txt). Gen 2 and Gen 3 are being asked questions whose answers they saw in training. Higher scores are expected and say nothing about unseen questions. - The grader is the model itself plus my rules. The score comes from rules I wrote, keywords I chose, and the model's own confidence. Self-grading is biased toward whatever fits the rules, and the confidence term tends to reward memorized answers.
- A confound. I also lowered the sampling temperature each generation, which by itself should reduce garbage output. I did not separate the effect of RSI from the effect of temperature.
- One recorded run. I don't have repeated runs with different seeds, or a comparison against the same setup without RSI.
- Not useful. Ask about anything outside the training data and you get an off-topic reply. The README says the same.
So what actually happened is: a tiny model picked its best-scoring answers to 7 questions and re-learned them.
Running it
pip install -r requirements.txt # torch>=2.0.0 python mini_gpt.py # pretrain and generate python chat_gpt.py # train on dialogues (saves chat_model.pt) python chat_gpt.py --chat # chat with the saved model python web_ui.py # local web UI (http://localhost:8765) python rsi_gpt.py # run 3 RSI generations Note that rsi_gpt.py rewrites chat_data.txt, so back it up first. If you only want to poke at it, the demo Space below is easier.
Takeaways
- A ~0.9M-parameter Japanese GPT fits in plain PyTorch;
mini_gpt.pyis 175 lines including comments. - A prompt format plus a stop character was enough to turn a text-continuation model into a (very small) chat model, with no architecture change.
- The RSI numbers come from a toy setup with self-grading and evaluation prompts leaked into training, so they are not evidence of general self-improvement. If I do this again, I would split off held-out prompts, use a grader other than my own rules, and fix the temperature.
Links
- GitHub: https://github.com/koheip/mini-llm
- Model on Hugging Face: https://huggingface.co/koheip/mini-gpt-ja
- Demo Space: https://huggingface.co/spaces/koheip/mini-gpt-ja-demo