ai-explained

build · hands-on · pytorch

Build your own tiny LLM from scratch.

A code-along walkthrough — open the article in one tab, your editor in another. By the end you've written every line of a working 10M-parameter transformer that generates coherent toy text. Every concept cross-references the matching article and interactive demo, so the math, the visualization, and the code are one click apart.

17
steps
17
ready to read
5h 31m
of reading
7h 7m
of hands-on
the destination

What you end up with

the model
10M-param GPT
A decoder-only transformer with multi-head attention, layer norm, and a learned tokenizer. Yours. Runs in your browser by step 15.
the training
TinyStories quality
Trained on the TinyStories corpus to write coherent simple stories. Not ChatGPT — but every line of how it works, you wrote.
the understanding
Full-stack mental model
Tokenization, attention, gradients, sampling, KV cache, fine-tuning — every concept in modern AI, implemented and validated against the demos you've already explored.
the path

The 17-step curriculum

Three phases, each a coherent run. Live steps render now; stubs mark what's coming. Numbers are stable — you can bookmark step 05 today and it'll still be step 05 next month.

Foundations

math + tokenizer + data

steps 00–03
  1. 00
    Set the table
    What we'll build, why this curriculum, and how to get your environment ready.
    10 min + 10 min hands-on
  2. 01
    The math you actually need
    Five ideas — vectors, matmul, softmax, cross-entropy, gradients — and where each one shows up in our code.
    12 min
  3. 02
    Build a BPE tokenizer
    Train your own byte-pair encoding from scratch — no external library, ~90 lines of Python.
    25 min + 35 min hands-on
  4. 03
    Tokenize TinyStories, build batches
    Download the dataset, train the BPE on it, save token IDs once, and yield (input, target) batches the model can learn from.
    18 min + 25 min hands-on

The Model

attention → transformer → GPT

steps 04–09
  1. 04
    Embeddings and positional encoding
    Two lookup tables — one for token, one for position — added together to start the residual stream.
    16 min + 20 min hands-on
  2. 05
    Scaled dot-product attention, from scratch
    The keystone of every transformer, in 30 lines of PyTorch.
    22 min + 25 min hands-on
  3. 06
    Multi-head attention
    Run n_heads attentions in parallel — efficiently, with one big projection matrix and a reshape.
    20 min + 25 min hands-on
  4. 07
    The transformer block
    LayerNorm → MultiHead → residual → LayerNorm → MLP → residual. The pattern stacked N times to make a deep transformer.
    20 min + 22 min hands-on
  5. 08
    Assemble the GPT class
    Embeddings + N transformer blocks + final norm + LM head + tied weights. The whole model in 60 lines.
    18 min + 25 min hands-on
  6. 09
    The training loop
    AdamW with weight-decay groups, warmup + cosine LR schedule, gradient clipping, periodic eval, checkpoints.
    28 min + 1h hands-on

Make It Real

sampling, scaling, tuning, eval, ship

steps 10–16
  1. 10
    Sampling and decoding
    Greedy, temperature, top-k, top-p (nucleus). Same model, four very different writers.
    18 min + 20 min hands-on
  2. 11
    Scale it up: 1M → 10M → 100M params
    Same architecture, three sizes. What changes (and what doesn't) when you grow the model. Chinchilla, in practice.
    22 min + 30 min hands-on
  3. 12
    Fine-tune with LoRA
    Adapt your trained base model to a new behavior — with 800× fewer trainable parameters than full fine-tuning.
    24 min + 35 min hands-on
  4. 13
    Evaluate honestly
    Three lenses on model quality — perplexity, generation samples, LLM-as-judge. None complete on its own; together they let you make decisions instead of vibes.
    22 min + 25 min hands-on
  5. 14
    Inference: KV cache + ONNX export
    Two operations that make generation production-fast: cache K and V across steps, then export the whole model to a portable ONNX file.
    24 min + 35 min hands-on
  6. 15
    Run it in your browser
    Capstone — load your trained ONNX model into onnxruntime-web and generate text from a single HTML page. No Python, no server.
    20 min + 35 min hands-on
  7. 16
    Where to go from here
    Twelve threads to pull on, ranked by leverage. The base model is no longer mysterious; the rest of the field is the next ~dozen weekends.
    12 min

How this fits the rest of the site

Every step cross-references the matching theory article and interactive demo. When you implement attention in step 05, the page links to the math derivation and the Attention Inspector. The site becomes a multi-modal study environment: code in your editor, math in one tab, real-model visualization in another.

You're not expected to have read the theory articles first. The build track is self-contained — but if anything in a step feels shaky, the matching theory + demo are exactly one click away.

Status: 17 live · 0 wip · 0 stubbed. The full ~3-month roadmap is set; articles ship in three releases (foundations → model → make it real).