How LLMs work, part 7: RLHF, PPO, GRPO, DPO and reasoning

Fine-tuning teaches a model to copy good answers. Reinforcement learning teaches it which answers are actually better, and it is what made reasoning models like o1 and DeepSeek-R1 possible. This is the final part of our series on how LLMs work. You will learn how reinforcement learning works, how RLHF aligns a model with what people want, how the three main algorithms (PPO, GRPO and DPO) differ, and why reasoning is now the main way models get smarter.