Frontiers
How image generators turn noise into pictures, how one model learns both images and text, how mixture of experts adds parameters without adding compute per token, how language models use tools and think longer to solve harder problems, how to read a benchmark score, what the harness around a coding agent does, what training and serving cost in compute and energy, and how well models use long inputs.
- Level
- Advanced
- Length
- 12 lessons, 5 hr 5 min
- Assumes
- The transformers track.
- Builds on
- Transformers and LLMs
- 1Diffusion: from noise to dataImage generators start from pure noise and remove it a little at a time. Watch the exact reverse process turn random dots into a dataset, and see what a real model has to learn.26 min
- 2Diffusion on imagesA small diffusion model trained on handwritten digits runs in your browser. Watch a digit emerge from noise, compare DDPM with DDIM, and steer it with classifier-free guidance.30 min
- 3Pictures, one token at a timeJanus-Pro writes a picture the way a chatbot writes a sentence, as 576 image tokens in reading order. Run it on your GPU and compare it with diffusion.27 min
- 4One space for images and textCLIP learned to place a picture and a sentence that describes it at nearly the same point in one embedding space. Train a tiny version and see why that enables zero-shot classification.22 min
- 5Mixture of expertsSome of the largest language models run only a fraction of their weights for each token. See how a router picks experts, and why total and active parameter counts differ so much.24 min
- 6Thinking longerLanguage models often answer better when they write out intermediate steps, sample several attempts, or are trained to reason at length. See exactly when voting over attempts helps and when it hurts.27 min
- 7Models that use toolsA language model only produces text, yet it can check the weather, search, or run code. Step through the loop that connects a model to tools, and see where it fails and where safety controls belong.22 min
- 8Evaluation: what a score meansBenchmarks, error bars, contamination, pass@k, and arena ratings. Learn how much a model score can move by chance, and how to estimate and compare scores honestly.35 min
- 9Inside an agent harnessIn a coding agent, the model is one part. Step through a harness that reads, edits, and tests a real project in your browser, with permissions, a sandbox, a token budget, and compaction.28 min
- 10Compute and energyWhat it takes to train a model and to answer a prompt, in operations, GPU hours, electricity, and water, with the published figures, the estimates around them, and why they disagree.27 min
- 11Long context in practiceWhat fills a model's context window in a real product, what a longer context costs in memory, compute, and money, how well models really use it, and when retrieval is the better choice.25 min
- 12Where multimodal models are headingFrom language models with an image encoder attached to single networks that read and write text, pictures, and sound. Four designs, what their papers report, and what is still hard.12 min
Labs in this track
- Diffusion from noise to dataWatch noise turn into a 2D dataset using the exact reverse diffusion process, with the score field drawn as arrows.
- Noising a digitMix a real handwritten digit with noise using DDPM's schedule, and see what a trained denoising network guesses at every step: the noise, and the clean digit.
- Generating a digitA small class-conditional diffusion model, trained on MNIST and running in your browser: watch a digit emerge from noise, compare DDPM with DDIM, and turn up classifier-free guidance.
- Text to image, one token at a timeMake pictures from your own prompts with Janus-Pro-1B on your GPU, and watch it write each picture as 576 codes in reading order before a decoder turns them into pixels.
- Contrastive learning, CLIP styleTrain an image encoder and a text encoder together until each picture lands next to its caption in one shared space.
- Mixture of expertsSee a router send each token to its top experts, explore real routing recorded from OLMoE-1B-7B, then compare total and active parameter counts for a Mixtral-shaped model.
- Voting over sampled answersCompute exactly how majority voting over n sampled answers changes accuracy, then explore real reasoning traces from a small open reasoning model at thinking budgets from 0 to 4,096 tokens.
- The tool-use loopStep through how an application and a model trade messages to use tools, switch on failures to see how the loop copes, then replay real runs recorded from three sizes of an open model.
- Which model is better?Two models with known accuracy take the same benchmark. See how often the weaker one wins, and how many questions it takes to tell them apart.
- Estimating pass@kEstimate the chance that one of k attempts passes, from n samples per problem. Compare the unbiased estimator with the biased plug-in shortcut.
- A pairwise arenaSimulated voters compare models two at a time. Turn votes into online Elo or Bradley-Terry ratings and watch the ratings and their intervals settle.
- Inside a coding agent's harnessA scripted coding agent fixes a real bug in a tiny project in your browser. Change permissions, the sandbox, and the context window, and watch what the harness does.
- Training and serving energyTurn a model's size, its training tokens, and the hardware into operations, GPU hours, days, and electricity, then add a year of answering prompts. Anchored on published runs.
- Long context, cost and positionFill a context window with the pieces a real product sends and see the tokens, memory, and compute. Then see where in a long input real models find answers, from recorded results.