GRPO is a reinforcement learning algorithm for post-training large language models, introduced in the DeepSeekMath paper (2024) and prominently used to train the DeepSeek-R1 reasoning model. It replaces PPO's learned critic network with a group-based Monte Carlo baseline: the model generates multiple completions per prompt, scores them, and normalizes those scores relative to the group to compute advantages. The key insight is that a critic model is unnecessary when you can estimate the baseline directly from a group of parallel rollouts — cutting training memory roughly in half while maintaining stable policy gradients. Understanding GRPO requires keeping one mental model front of mind: every advantage is relative, not absolute — the algorithm never asks "is this response good?" but only "is this response better than the average for this prompt?"
What This Cheat Sheet Covers
This topic spans 14 focused tables and 91 indexed concepts, 80 flashcards. Below is a complete table-by-table outline of this topic, spanning foundational concepts through advanced details.
A jump-to index of every table row in this cheat sheet.
An interactive map of every table and concept in this topic.
Table 1: Core Concepts and Terminology
GRPO's vocabulary differs subtly from standard RL: "group" replaces "episode batch," "advantage whitening" replaces critic subtraction, and "verifiable reward" replaces reward-model scoring. Getting these definitions precise before reading the math prevents most common misunderstandings.
| Concept | Example | Description | |
|---|---|---|---|
G = 8 responses generated per prompt | • For each prompt $q$, the policy $\pi_{\theta_\text{old}}$ generates $G$ independent completions ${o_1, \ldots, o_G}$ • these form the "group" used for advantage estimation | ||
$\hat{A}_i = \frac{r_i - \mu_G}{\sigma_G}$ | • Each completion's reward is z-score normalized within its group: subtract the group mean $\mu_G$ and divide by group std $\sigma_G$ • no critic or value function needed | ||
Correct boxed math answer → r=1; wrong → r=0 | A rule-based, deterministic reward function (e.g., regex match, compiler pass/fail) that eliminates reward-model training and reduces reward hacking compared to neural reward models. | ||
π_ref = frozen pre-trained or SFT model | • A frozen checkpoint used exclusively for KL regularization • prevents the training policy from drifting too far from its starting distribution | ||
$\beta D_\text{KL}(\pi_\theta \Vert \pi_\text{ref})$ | • Added directly to the loss (not subtracted from the reward as in PPO) • coefficient $\beta$ controls how tightly the policy stays near the reference | ||
$r_{i,t}(\theta) = \frac{\pi_\theta(o_{i,t} \mid q, o_{i,<t})}{\pi_{\theta_\text{old}}(o_{i,t} \mid q, o_{i,<t})}$ | • Per-token importance weight measuring how much the current policy has shifted relative to the rollout policy • clipped to prevent large updates | ||
Single scalar at end of full completion | • Reward assigned once per full response, shared uniformly across all tokens in that response • contrasts with process rewards that score intermediate steps | ||
Step-level scores $r^{(1)}, r^{(2)}, \ldots$ per reasoning step | • Reward assigned after each reasoning step • GRPO can use step-level rewards by computing return-to-go advantages $\hat{A}{i,t} = \sum{j \geq t} \tilde{r}^{(j)}$. | ||
Math correctness check, code compilation | • The paradigm of applying GRPO with programmatic verifiers instead of trained reward models • pioneered by DeepSeek-R1. |