Transformers and LLMs
Take a transformer apart. Compute attention by hand, see how position is encoded, trace a token through every layer, run a real language model in your browser to inspect its predictions, clean a web crawl into training data, and see exactly what a chat assistant reads.
- Level
- Advanced
- Length
- 13 lessons, 4 hr 2 min
- Assumes
- Neural networks and embeddings, or equivalent background.
- Builds on
- Neural networks and Language as vectors
- 1Predicting the next tokenAt every step a language model scores every token in its vocabulary. Softmax turns the scores into probabilities, and a sampling rule picks one. Temperature, top-k, and top-p shape that choice.16 min
- 2Before transformers: recurrent networksRecurrent networks read a sequence one step at a time and carry a hidden state forward. See why their gradients vanish or explode, how gates help, and why attention replaced the loop.20 min
- 3Attention, step by stepAttention lets every token gather information from the other tokens in its context. Work through it one query at a time, from scores to weights to a mixed output.20 min
- 4Many heads and the causal maskTransformers run several attention heads side by side, each free to learn its own pattern, and language models hide the future from every token with a causal mask.18 min
- 5Where words areAttention ignores word order unless the model is told where each token sits. Sinusoidal encodings add position as waves; rotary embeddings rotate queries and keys so attention sees relative distance.18 min
- 6The transformer blockA transformer is one block repeated many times. Each block reads from a shared residual stream, mixes information across tokens with attention, transforms each token with an MLP, and adds the results back.18 min
- 7Inside a real language modelRun SmolLM2-135M in your browser. Watch it split your text into tokens, score every token in its vocabulary, and write one token at a time.15 min
- 8How LLMs are trainedPretraining on trillions of tokens builds a base model; fine-tuning and preference tuning turn it into an assistant. Scaling laws tell you how to split a compute budget between model size and data.20 min
- 9Where training data comes fromWeb crawls, books, code, synthetic text, and human feedback, and the filtering and deduplication that turn a raw crawl into training tokens. Run a miniature pipeline yourself.25 min
- 10Fine-tuning and LoRAFine-tuning adapts a pretrained model to a new task. LoRA freezes the original weights and learns a small low-rank change instead, cutting the numbers you train by orders of magnitude.18 min
- 11Generating fast: the KV cacheEach new token needs the keys and values of every token before it. Storing them instead of recomputing them makes generation fast, and the store grows with every token.16 min
- 12Quantization: smaller numbersStoring weights in 8 or 4 bits instead of 16 cuts a model's memory by half or more. See how scales, groups, and outliers decide what that costs in accuracy.18 min
- 13How a chat assistant worksA chat assistant is a language model continuing one long text. See the chat template, the system prompt, a conversation sent again every turn, a context window filling up, and why the model needs today's date.20 min
Labs in this track
- Sampling the next tokenReshape a next-token distribution with temperature, top-k, and top-p, then draw from it and compare the counts.
- A hidden state through timeWatch a one-number recurrent network carry the subject of a sentence word by word, and see its gradient vanish or explode through time.
- Attention calculatorDrag queries, keys, and values on a plane and watch scaled dot-product attention compute scores, weights, and outputs for every token.
- Attention heads and the causal maskCompare hand-built versions of five attention patterns seen in trained transformers, switch the causal mask on and off, and see how heads split the model width.
- Position encodingsExplore the sine and cosine waves of the original transformer's position encodings, then rotate queries and keys with RoPE and watch their dot product depend on position only through the offset.
- Transformer anatomySelect any part of a decoder-only transformer to see its job, its tensor shapes, and its parameter count, for models from GPT-2 to Llama 3.
- A live language modelRun SmolLM2-135M in your browser, see how it tokenizes your text, read its real next-token probabilities, and generate one token at a time.
- Compute and scaling lawsChoose a model size and a number of training tokens, see the compute it costs, and compare it with the compute-optimal split under two fitted scaling laws.
- Clean a web crawlRun a miniature pretraining-data pipeline on a sample crawl. Switch on quality rules, a word blocklist, and exact and near-duplicate removal, and see what survives.
- How low can the rank go?Choose a LoRA rank and see how much of a target weight change B times A can capture, from the singular values or by training from B = 0.
- The KV cacheStep through generation with and without a key-value cache, then size the cache for real models, context lengths, and batches.
- Squeeze the weightsQuantize 128 weights to 8, 4, or 3 bits with absmax or zero-point scales, add an outlier, and see what rounding costs.
- Same text, two modelsGive SmolLM2-135M and SmolLM2-135M-Instruct exactly the same tokens, as plain text or in the chat template, and compare what each writes.
- A chat, as the model reads itChat with SmolLM2-135M-Instruct in your browser and see the one long text it reads each turn, the system prompt, and a context window filling up.